Saturday, November 28, 2015

Irregular maze generator (from ASCII tessellation) - Part 1

I wanted to write a generalized ASCII maze generator which could create a unlimited variety of tessellations.  

So I first created a parser that could detect "edges" in an ASCII "image"


Then the script will knock out wall in the ASCII image to create a maze.


This can generate irregularly shaped mazes, based on the ASCII template.  

Source code is at:


Python2: https://github.com/sevkeifert/text-utils/blob/master/maze-ify-ascii.py

Python3: https://github.com/sevkeifert/text-utils/blob/master/maze-ify-ascii-v3.py




USAGE:


   maze-ify-ascii.py -f YOUR_TEMPLATE_FILE  



For example, here's a basic grid:






Grid with holes:



Oblique


Hexagonal tessellation:




Complex Features
Irregular shape, with holes, multiple regions, text, and protected whitespace



Now for Part II: parsing the space *inside* font characters:

Wednesday, November 25, 2015

Command line tool to create and parse text outlines - revisited


A while back I posted a quick script for creating text outlines on the command line (for example, in Vim).  I like keeping all my documentation as plain text.  I updated it so that it can now parse and repair outlines if you edit them.

Source code is here:

        https://github.com/sevkeifert/text-utils/blob/master/outline


Description of the `outline` utility:

This is a command line tool for creating text outlines from space-delimited
markup.  This program transforms a stream of data, either rendering markup or
decoding an already rendered outline back into markup.

                                                    Kevin Seifert - 2015 GPLv2
USAGE

    Pipe space-delimited (or space-delimited) text into this script to create a
    text outline.  For example:

        cat your_markup_file | $0

OPTIONS

    -d    decode outline
    -e    encode outline (default)
    -h    help
    -n    use numeric style: 1  1.1  1.2 ...
    -r    re-encode existing outline (repair)
    -w   set word wrap width
    -0    mix single/double space (add break after continued lines)
    -1    single space
    -2    double space

EXAMPLE USAGE

    cat yourfile | outline       # for roman numeral format
    cat yourfile | outline -n    # for decimal format
    cat yourfile | outline -d    # decode rendered outline back into markup
    cat yourfile | outline -r    # repair (re-encode) rendered outline

    Or in vim, visually select text and run selection markup through pipe

        :'<,'> ! outline
        :'<,'> ! outline -d
        :'<,'> ! outline -r

MARKUP EXAMPLE

    INPUT EXAMPLE (WHITESPACE DETERMINES LEVEL)

        some heading
            more text
                a sub point
                another sub point
            more text
                a sub point
                another sub point

    OUTPUT EXAMPLE (RENDERED INTO OUTLINE)

        I. some heading
            A. more text
                1. a sub point
                2. another sub point
            B. more text
                1. a sub point
                2. another sub point

    ALSO, YOU CAN USE DECIMAL FORMAT USING THE -n FLAG

        1. some heading
            1.1 more text
                1.1.1 a sub point
                1.1.2 another sub point
            1.2 more text
                1.2.1 a sub point
                1.2.2 another sub point

Sunday, November 22, 2015

Word Ladder Puzzle Game (and Puzzle Solver)

Here's a word ladder puzzle game (and solver) I wrote a while back. A word ladder is a string of words that change one letter at a time. For example: "cat", "cot", "dot", "dog" is a word ladder that connects "cat" and "dog". I tweaked the game to run as a standalone application (instead of a java applet). The source code is on github:

https://github.com/sevkeifert/wordladder-game-solver

The game:


The puzzle solver:






How to Increase Website Traffic


I generally haven't put a lot of time into maintaining this blog, and mostly forgot about it for a couple years.   :-)

I've never really blogged frequently, and I took a break from posting any articles around 2011 and started posting again in 2014 (was busy).  Although I noticed the vacation created an interesting sociological experiment with the overall web traffic:




Traffic is fairly steady (and gradually falls) when the site is not updated at all -- from September 2011 to August 2014.

So, basically if you want more web traffic, increase both the number of articles on your website, and the number of "new" articles.    Overall, the increase in traffic is largely correlated with adding new articles.

Also, the posts that have gotten the most traffic are about things where I spent more than an hour "Googling" for something, and couldn't find a good answer or solution.  If you can't find it, there's a vacuum in the available information and you'll end up higher in Google's search result.   Even though the topic might be only interesting to a smaller demographic, anything on these topics will get a lot of  hits from Google searches.

I also notice "fuzzy" topics may do better that pure information, as readers will discuss the topic further with others.  My most popular articles are opinion articles regarding Linux.

This information is probably commonsensical, though what I thought what was interesting, was to actually see a confirmation of SEO principles with real web statistics.  After all, who else is going to let their website languish for years, for the benefit of science?  ;-)

Scripting Language Quick References

I added a series of scripting language "quick references."  These are mostly just copied from my short notes I use when shifting between languages.  These try to describe the overall syntax flavor (and quirks) of each language at glance.

Bash
JavaScript
Perl
PHP
Python
Ruby
VBScript

Quick Language Reference: JavaScript

JavaScript used to be viewed as a half-broken scripting language, though with node.js and language improvements over the last 15 years, it's been increasingly viewed as a platform for serious development (and a target for language compilation).

    See: http://www.crockford.com/JavaScript/JavaScript.html
    "JavaScript is lisp in C's clothing"

Chrome ships with an interactive console.  The last variable is output, no need for alert() or console.log()  for example

    a = 1
    a
    >> 1

write debug messages to console

    console.log('hello')

note on error handling:

    JavaScript will stop executing a process if it hits an error such as a missing variable or function.

    Chrome: check the bottom right of chrome for a red (!)
    Firefox debugger: firebug


data types:

    undefined (empty var place holder)
    null (empty object place holder)
    boolean
    number
    string
    object
    function  (type of object)
    array     (type of object)


    NOTE: there are no hashes (associative arrays) in JavaScript, only objects

    ALSO, there are primative value types

        boolean, number, and string

    then, there are also (different) boxed types with methods

        Boolean, Number, String

    eg: "hello" instanceof String; //false

The typeOf operator tells you which built-in type a variable is.  null is slightly couter intuitive, though was intended to be used only with objects.

    typeOf null == "object"

IOW, there are two primative null values: null and undefined.  null was intended for objects only.

SO...

    If you want to test for null
             you can use === (a strict equals test).
    If you want to test the type for undefined, Boolean, String, Number, or Function,
            you can use typeof.
    For anything else, you can use instanceof.

the instanceOf operator tests if an Object is a instance of a class

    a instanceOf SomeClass


java/javascript inconsistencies (in case you have to switch between the two)

    in java:

        array.length
        String.length()  // function

    in JavaScript both are properties:

        array.length
        string.length

    if you try to use length() you will get error :)

    TypeError: number is not a function

"false" values

    false null undefined "" 0 NaN

null vs undefined

    null !== undefined

    null is an object with a type of null
    undefined is the default value of an undefined reference or hash element
        (iow, undefined is what most languages call 'null')

    avoid using
    if ( a != null ) {
        // because this does type conversion and can throw errors
    }

 
    if(typeof neverDeclared == "undefined") //no errors
    if(neverDeclared == null) //throws ReferenceError: var is not defined

    undefined means "does not exist". null is an intentionally set value.

    null can be used as a flag, if a variable has been initialized, but value is unknown or an exceptional case:

    var o = null;
    if ( test_something()  ) {
        o = 21;
    }
    if ( o != null ) {
        // no exception, initialized ok
    }


semicolon at end of line

    optional, but probably best practice

declare a variable

    var a = '';

    Notes:
    Variables are bound to the function they are defined in.
     If you don't use `var` then the variable scope is global
     //avoid polluting the global namespace.

     i_am_global = 'prepare for namspace collision'


comments

    // single line
    /* multiline */
    <!-- single line (html style, generally leads to confusion)


initialize an object if undefined

  var pagedata = pagedata || {};


strings

     // there's no difference between double and single quotes
 
    var example1 = "I'm a String";
    var example2 = 'So am I.';

   // they are handy if you want to avoid escaping the quotes
   // using c-style backspace escape sequences 
    var example3 = 'I\'m a String too, but I have to escape the "\'" character ';


string weirdness

    JavaScript actually represents text strings in two different ways:  as primitive string values, and as string objects:

        var a = "I am a string!";
        alert(typeof a); // Will output "string"
        var b = new String("I am also a string!");
        alert(typeof b); // Will output "object"

    string are also char arrays (like in PHP)

    > console.log( "test"[0] )
    > t

    string type versus string object
    new String("a") == new String("a") will return false (compares object refs)
    "a" == "a" will return true (compares primatives)


    String types are a value type in JS, so they can't have any properties attached to them, no prototype, etc. Any attempt to access a property on them is technically performing the JS [[ToObject]] conversion (in essence new String).

    Easy way of distinguishing the difference is (in a browser)

        a = "foo"
        a.b = "bar"
        alert("a.b = " + a.b); //Undefined

        A = new String("foo");
        A.b = "bar";
        alert("A.b = " + A.b); // bar
        Additionally while

        "foo" == new String("foo")

    is true, it is only true due to the implicit type conversions of the == operator

        "foo" === new String("foo")

    will fail.


length of string is property

    .length


functions


    function your_function (arg1,arg2,arg3) {
        // do something
    }


    // call function
    your_function(1,2,3);

    // refer to function (without calling)
    your_function

    // the function reference is just like any other variable
    var f = your_function;
    f(1,2,3); // call it


using an anonymous, isolated function scope and namespace

    (function (){
        // your namespace is protected here  
        alert(1)  
    })();


vars declaration must be at the top of a function
 
    var x = 3;
    (function (){
        console.log(x + 2); //NaN - x is not defined
        var x = 0; //var declaration at end of function ?
    }());

    This is because "x" is redefined inside the function. This means that the interpreter automatically moves the var statements to the top of the function (you're welcome) and we end up executing this instead:

    var x = 3;
    (function (){
        var x;
        console.log(x + 2); //NaN - x is not defined
        x = 0;
    }());


introspection/reflection

    in Chrome console

        console.log( yourObject )


    // programmatically

    Object.getOwnPropertyNames(Object.prototype) gives :

    ["constructor", "toString", "toLocaleString", "valueOf", "hasOwnProperty", "isPrototypeOf", "propertyIsEnumerable", "__defineGetter__", "__lookupGetter__", "__defineSetter__", "__lookupSetter__"]

    Object.getOwnPropertyNames(Object) gives :

    ["length", "name", "arguments", "caller", "prototype", "keys", "create", "defineProperty", "defineProperties", "freeze", "getPrototypeOf", "getOwnPropertyDescriptor", "getOwnPropertyNames", "is", "isExtensible", "isFrozen", "isSealed", "preventExtensions", "seal"]


conditionals and type juggling

   // the if statement mirrors c/java syntax
   if ( boolean ) {

   } else if ( another_boolean ) {

   } else {

   }

    // note: types are automatically "juggled" or converted
    // 1 == "1"

    if ( a == b ) {  console.log(1) } // with type-juggling
    if ( a === b ) {  console.log(1) } // with strict type-check

    you may want to avoid "new String()" to avoid these quirks


regex

    var regex = /\S*\.gif/i;
    var results = test.match(regex);
    var file = results[0];     // file.gif


convert string to number type

    parseFloat ( s );

    parseInt ( s, 10 ); // must pass in base 10
    js is pickier (compared to php) when it comes to type juggling
    always use a radix:

        parseInt("08")     == 0    // not kidding :)
        parseInt("08",10)  == 8 // what you'd expect


get elements, for example, on document

    node.getElementById( id ) ;
    node.getElementsByTagName( 'a' ) ;

replace html

    your_html_node.innerHTML= 'your string';


arrays

    Warning: don't use trailing commas in arrays
    This will break in IE: which follows the spec in this point out of spite :-)

        var wrong = [1,2,3 ,<<<<<<<];  // can't have have , at end

    Most tutorials start out introducing you to arrays as such:

        // don't do this
        var myArray = new Array(10);

    Current best-practice avoids the "new" keyword on Javascript primitives. If you want to create a new Array simply use brackets [] like this:

        // better
        var myArray = [];

    // initialize ten elements [ undefined, ... ]
    var badArray = new Array(10); // Creates an empty Array that's sized for 10 elements.

    // faster
    var goodArray= [10];          // Creates an Array with 10 as the first element.

    NOTE: Javascript arrays are passed by reference, assigned by reference


literal syntax

    JavaScript has the types Object, Array, Boolean, Number, String, and Function. Each has its own literal syntax and so the explicit constructor is never required.

    var a = new Object();

    a.greet = "hello"; // avoid
    var a = { greet: "hello" }; // better

    var b = new Boolean(true);// avoid
    var b = true; // better

    var c = new Array("one", "two");// avoid
    var c = ["one", "two"]; // better

    var d = new String("hello");// avoid
    var d = "hello" // better

    var e = new Function("greeting", "alert(greeting);");// avoid
    var e = function(greeting) { alert(greeting); }; // better


array splice

    The splice() method adds/removes items to/from an array, and returns the removed item(s).

    syntax:
    array.splice(index,howmanyremoved,item1,.....,itemX)

    index    Required. An integer that specifies at what position to add/remove items, Use negative values to specify the position from the end of the array
    howmanyremoved    Required. The number of items to be removed. If set to 0, no items will be removed
    item1, ..., itemX    Optional. The new item(s) to be added to the array

    example:
    var fruits = ["Banana", "Orange", "Apple", "Mango"];
    fruits.splice(2,0,"Lemon","Kiwi");
    // The result of fruits will be:
    // Banana,Orange,Lemon,Kiwi,Apple,Mango


escape for url data context

    function urlencode (str) {
        if ( str === null || str === undefined ) {
            return '';
        }
        var str = new String(str);
        str = encodeURIComponent(str);
        str = str.replace(/!/g, '%21');
        str = str.replace(/'/g, '%27');
        str = str.replace(/\(/g, '%28');
        str = str.replace(/~/g, '%7E');
        str = str.replace(/\)/g, '%29');
        str = str.replace(/\*/g, '%2A');
        return str.valueOf();
    }

reverse url encoding

    function urldecode(s) {
        if ( s ) {
            s = new String(s);
            s = decodeURIComponent(s.replace(/\+/g, " "));
            return s.valueOf();
        }
        return '';
    };



escape for html data context

    // white space options
    var HTML_NBSP = 1;
    var HTML_BR = 2;
    var HTML_NBSP_BR = 3; // 1 + 2
    var HTML_BR_NBSP = 3; // 1 + 2

    function htmlencode (str, ws) {
        if ( str === null || str === undefined ) {
            return '';
        }
        var str = new String(str);
        str = str.replace(/&/g, "&");
        str = str.replace(/>/g, ">");
        str = str.replace(/</g, "<");
        str = str.replace(/"/g, """);
        str = str.replace(/'/g, "'");
        // ecode whitespace
        if ( ws == 1 || ws == 3 ) {
            str = str.replace(/ /g, " ");
            str = str.replace(/\t/g, "    ");
        }
        // also insert line breaks
        if ( ws == 2 || ws == 3 ) {
            str = str.replace(/\n/g, "<br />");
        }
        return str.valueOf();
    }


similar to php string escape

    function addslashes(str) {
        if ( str ) {
            str = new String( str );
            var str = str.replace(/\\/g, '\\\\').
                replace(/\u0008/g, '\\b').
                replace(/\t/g, '\\t').
                replace(/\n/g, '\\n').
                replace(/\f/g, '\\f').
                replace(/\r/g, '\\r').
                replace(/'/g, '\\\'').
                replace(/"/g, '\\"');
            return str.valueOf();
        }
        return '';
    }


zipped/compressed sources

    most browsers will support gzipped source

        <script type="text/JavaScript" src="prototype.js.gz"></script>


dynamically writing to a pop up window:

    function write_to_pop_up () {
        new_window = open("","TITLE","width=640,height=480,left=10,top=10,resizable=1,scrollbars=1");
        new_window.document.open();
        new_window.document.write("Hello World");
        new_window.document.close();
    }

xml tags in strings

    when contructing xml tags in the page,
    use a backslash to escape the close tag like this:

    xml = "<tag>" + value + "<\/tag>";

    This is not exactly a bug bu a quirk in the SGML markup spec
    close-tags are interpreted in a script tag.


disable buttons/links on click

    <form name="form1">
      setting the onclick to null in the onclick handler:

      <h3> rewrite on click </h3>
      <a href="JavaScript:void(0)" id="a"
            onClick="this.onclick=null;alert('this is a test');"> set before </a>
      <br />
      <a href="JavaScript:void(0)" id="b"
            onClick="alert('this is a test');this.onclick=null;"> set after </a>
      <br />

      <h3> rewrite href </h3>
      <a href="JavaScript:alert("this is a test")" id="c"
            onClick="eval(unescape(this.href));this.href='JavaScript:void(0)'"> set href onclick </a>
      <br />

    </form>



closures:

  IMO the real power of JavaScript comes from the closures syntax.

  https://developer.mozilla.org/en-US/docs/JavaScript/Guide/Closures

  a closure caling this function:

    function test1(){}

  looks like:

    function() { test1(); }

  useful for capturing values in scope that must be passsed to another function
  at a later time (for example setTimeout or ajax callback).

  note that return value of closure is not the same as function unless you use:

    function() { return test1(); }

you can also refer to a function by name

        function example1() { }

        function example2(example1); // pass function to function


closure in loop


    // doesn't do what you'd expect, all the same number
    function attachEventsToListItems( ) {
        var oList = document.getElementById('myList');
        var aListItems = oList.getElementsByTagName('li');
         for(var i = 0; i < aListItems.length; i++) {
            var oListItem = aListItems[i];
            // Here I try to use the variable i in a closure:
            oListItem.onclick = function() {
                alert(i);
            }
        }
    }

    // REASON: variable scope is at function level, not at block level
    // FIX:    create a new scope for the 'current' value of i
    //            by executing a function inside the loop:
    function attachEventsToListItems( ) {
        var oList = document.getElementById('myList');
        var aListItems = oList.getElementsByTagName('li');
        for(var i = 0; i < aListItems.length; i++) {
            var oListItem = aListItems[i];

            // create new function scope:
            oListItem.onclick = (function(value) {
                return function() {
                    alert(value);
                }
            })(i);
        }
    }

    //or your could use something like:

        function  bindarg( f, arg ) {
            return function(e){ f(arg) };
        }


    // for new browsers you can use

        let var = ...

    // or use bind -- this creates a wrapper function with the bound vars

        fun.bind(thisArg[, arg1[, arg2[, ...]]])

        // eg

        this.element.click(

            // `this` will be redefined to a dom element inside click
            (
                 function() {

                    // ...
                    // after bind, `this` reference will be restored to refer to object context
                 }

            ).bind(this)
        );



data element

    // start with data-

    <a data-seqno="1" onclick="dostuff()">

    // read in the function

        var sequenceNo = this.dataset.seqno;



try/catch

    try {
        //Run some code here

    } catch(err) {
        //Handle errors here

    }

also
    } finally {
    }

text of error message:

    e.message


throw exception


    // simple
    throw "your string";
    throw { "name": "name", "message": "message" }


    // with a constructor
    function MyError(name,message) {
        this.name = name;
        this.message = message;
    }
    MyError.prototype = new Error;


    //usage
    try {
      something();
     } catch(e) {
      if(e instanceof MyError)
       doSomethingElse();
      else if(e instanceof Error)
       andNowForSomethingCompletelyDifferent();
    }


be careful with type juggling.  + operator may not understand what is intended

    for ( i in obj ) {
        var j = i + 1;  // "0" + 1 == "01" ... wrong!
        var j = i; j++;  // 2

        // force to int with default 8
        var j= parseInt(i) || 8;
    }



cast to bool

    use not-not operator
    var a = 4;

    !! a


    if("true"){
        alert("1");
    }else{
        if(!!"true" == true){
            alert("2");
        }
    }
    // alerts 1

    if("true" == true){
        alert("1");
    }else{
        if(!!"true" == true){
            alert("2");
        }
    }
    // alerts 2


    type juggling rules:
        1. string must literally match
        2. true false are like 1 0
        3. whitespace is trimmed

    examples:

        '' == '0' // false  
        0 == '' // true
        0 == '0' // true

        false == 'false' // false
        false == '0' // true

        false == undefined // false
        false == null // false
        null == undefined // true

        ' \t\r\n ' == 0 // true

        groking the type juggling:
        if types are unequal, data is converted first to closest common primative type, then compared.

            '1' == 1
            true

            '1' == new String( 1 )
            true  // to string primative

            new String('1') == new String( 1 )
            false     // object comparisons

            new String('1') == 1
            true

            new String('1') == 1.0000
            true

            new String('1.0000') == 1
            true

            new String('1.000') == '1'
            false // converted to string types

            new String('1') == true
            true

            new String('0') == null
            false

            new String('0') == false
            true

            false == null
            false

            null == undefined
            true

            null == false
            false

            null == true
            false

            undefined == false
            false



    auto convert rules:

        String object -> string primative (Box -> primative)
        string primative of number -> number
        number -> boolean (true false null)

    be careful string comparisons of serilization of json data number != string
 
    construct "if ( a )" means "if a is not null"


    cast to int:

        +i

    eg:

        +'-1' === -1

        Unary plus casts the value to a number, but binary plus does not: +'12' === 12 (result is a number), but 0 + '12' === '012' (result is a string). Note, however, that 0 - '12' === -12


new

Calling a function with the new keyword creates a new object and then calls the function with that new object as its context. The object is then returned. Conversely, invoking a function without 'new' will result in the context being the global object if that function is not invoked on an object (which it won't be anyway if it's used as a constructor!)


open new window

    window.open(URL,name,specs,replace)


    URL    Optional. Specifies the URL of the page to open. If no URL is specified, a new window with about:blank is opened

        name    Optional. Specifies the target attribute or the name of the window. The following values are supported:
        _blank - URL is loaded into a new window. This is default
        _parent - URL is loaded into the parent frame
        _self - URL replaces the current page
        _top - URL replaces any framesets that may be loaded
        name - The name of the window

    specs    Optional. A comma-separated list of items. The following values are supported:

        channelmode=yes|no|1|0    Whether or not to display the window in theater mode. Default is no. IE only
        directories=yes|no|1|0    Whether or not to add directory buttons. Default is yes. IE only
        fullscreen=yes|no|1|0    Whether or not to display the browser in full-screen mode. Default is no. A window in full-screen mode must also be in theater mode. IE only
        height=pixels    The height of the window. Min. value is 100
        left=pixels    The left position of the window
        location=yes|no|1|0    Whether or not to display the address field. Default is yes
        menubar=yes|no|1|0    Whether or not to display the menu bar. Default is yes
        resizable=yes|no|1|0    Whether or not the window is resizable. Default is yes
        scrollbars=yes|no|1|0    Whether or not to display scroll bars. Default is yes
        status=yes|no|1|0    Whether or not to add a status bar. Default is yes
        titlebar=yes|no|1|0    Whether or not to display the title bar. Ignored unless the calling application is an HTML Application or a trusted dialog box. Default is yes
        toolbar=yes|no|1|0    Whether or not to display the browser toolbar. Default is yes
        top=pixels    The top position of the window. IE only
        width=pixels    The width of the window. Min. value is 100

    replace    Optional.Specifies whether the URL creates a new entry or replaces the current entry in the history list. The following values are supported:
        true - URL replaces the current document in the history list
        false - URL creates a new entry in the history list


for in

    // this syntax deviates from python/perl/php and most other languages.
    // DO NOT use for arrays
    // use to loop over keys of an hash

    var person={fname:"John",lname:"Doe",age:25};

    var txt = '';
    for (x in person) {
      txt + =  x + ': ' + person[x] + '<br>';
      }

    console.log( txt );

    BEWARE:
        this also scans the prototype chain

        The order of the iteration is undefined by the standard, because the intent is to iterate over unordered structures like hashes.  Many browsers preserve the order (like php), some do not.


hash

    does not really exist in javascipt.  All that exists is Object
    Warning:  a hash is really an object

        // JavaScript magic trick

        // watch me pull a rabbit out of my hat, Rocky:
        // nothing up my sleeve:
        var emptyHat = [ ];

        // and presto!
        if ( emptyHat['constructor'] ) { alert("look a rabbit") }


        solution: use the 'in' operator
     
        BEWARE2: in searches the prototype chain :)
        newer function obj.hasOwnProperty(prop) may help restrict to current scope


get base url of page

  function getBaseURL() {
    var url = location.href;
    var baseURL = url.substring(0, url.lastIndexOf('/')+1);
    return baseURL ;
  }



apply vs call

    theFunction.apply(valueForThis, arrayOfArgs)
    theFunction.call(valueForThis, arg1, arg2, ...)



general util class

    http://jquery.com

    http://underscorejs.org

        templates, iterators, useful functions





'with' considered harmful


    acceptable use:

        Math.cos(Math.asin(0.5))

    could be more legible as

        with(Math) cos(asin(0.5))


 


dump object to string


    function dump(arr,level) {
        var dumped_text = "";
        if(!level) level = 0;

        //The padding given at the beginning of the line.
        var level_padding = "";
        for(var j=0;j<level+1;j++) level_padding += "    ";

        if(typeof(arr) == 'object') { //Array/Hashes/Objects
            for(var item in arr) {
                var value = arr[item];

                if(typeof(value) == 'object') { //If it is an array,
                    dumped_text += level_padding + "'" + item + "' ...\n";
                    dumped_text += dump(value,level+1);
                } else {
                    dumped_text += level_padding + "'" + item + "' => \"" + value + "\"\n";
                }
            }
        } else { //Stings/Chars/Numbers etc.
            dumped_text = "===>"+arr+"<===("+typeof(arr)+")";
        }
        return dumped_text;
    }


--
GOTCHAS

weird data types

    https://github.com/stevekwan/best-practices/blob/master/JavaScript/gotchas.md
    http://www.codeproject.com/Articles/182416/A-Collection-of-JavaScript-Gotchas

     typeof null == 'object';     // true

    no such thing as an associative array. they are actually objects, with 'constructor' key

this - quirks!
 

    in binding a closure to an element, the element's this.properties have tighter scope than local variables.  Example:

    var value = 123;
    onclick = function() {
        alert(value); // will alert this.value of control, not variable!
    }
 

prototype vs __proto__

    prototype goes on constructors that create objects, and __proto__ goes on objects being created.

One more thing: please don't ever try to manipulate the __proto__ pointer. JavaScript is not supposed to support editing of __proto__ as it is an internal property. Some browsers will let you do it, but it's a bad idea to rely on this functionality. If you need to manipulate the prototype chain you're better off using hasOwnProperty() instead.

Why is parseInt() mucking up when I get into big numbers?

    all numbers are floats:

    Despite looking like it, JavaScript doesn't actually have an integer data type - it only has a floating point type.

parseInt("1000000000000000", 10) < parseInt("1000000000000001", 10); //true

but add one more zero:

parseInt("10000000000000000", 10) < parseInt("10000000000000001", 10); //false


No block scope.  Variables belong to function scope.

NaN === NaN; // false

Constructors (more confusing than they should be)

http://zeekat.nl/articles/constructors-considered-mildly-confusing.html

sort:

    var objs = [
            { first_nom: 'Lazslo', last_nom: 'Jamf'     },
            { first_nom: 'Pig',    last_nom: 'Bodine'   },
            { first_nom: 'Pirate', last_nom: 'Prentice' }
        ];


    function compare(a,b) {
      if (a.last_nom < b.last_nom)
         return -1;
      if (a.last_nom > b.last_nom)
        return 1;
      return 0;
    }

    objs.sort(compare);


--
OOP IN JAVASCRIPT

gist and weirdness of js:

    functions are objects

    variable scope is bound to a function.  This is very conter-intuitive in some cases.

    The `this` reference is just a var that is set in the calling context.  it's not fixed
    You can save a reference to `this`
    (`this` means the calling object, not the current object)
        var self=this; // in constructor
        iow, 'this' should have been renamed to something like 'that'

    Private: to hide a varible, define it inside a function
            to hide a function, define it inside a function


    There are no classes
    There are two basic approaches for oop

        prototypes -- more efficient memory handling (possibly though this is becoming a non-issue).  more limited in acccessing scope/this
        closures -- cleaner code, more modern, not as efficient with memory

    objects always inherit from another object


prototype vs closures

    any function defined in the prototype only exists in one place in memory
    closure methods exist for every copy of the object
    prototype methods have no knowledge/access of what is contructed, unless attached to `this`

    So, prototype based approaches are currently the 'recommended' approach. However:

    Modern JS interpreters will likely be getting smart enough to NOT create
    rededundant copies of a method. So it's cleaner (from the code side) to think
    of 'function' as a synonym of 'class'.  Also, maintaining references in
    prototypes requires more work, for example using .bind(this), which I think
    long term might cause more of a permanent performance hit as it's permanently
    wrapping function calls in a bind call.

    prototype methods will have difficulty in maintaining `this` reference

    prototypes CANNOT access private variables of a class.        
    prototypes CANNOT access `this` in

        YourObj.prototype.clickSomething = function(){
            // sorry, `this` has been lost, if you access from a button
            // find another way to do this.
            // imo prototype approach is somewhat broken.
        }



    IOW, the VM's will get smarter and more efficient.  So, write the cleanest code, and optimize only if needed.

    // here's a simple prototype based object
    // more verbose, ugly code
    // more effiencient in old vm's

        // class ObjectConstructor { // I'm not really here

            function ObjectConstructor {
            }

            ObjectConstructor.prototype.yourdata = {}
            ObjectConstructor.prototype.yourfunction = function() {

                // `this` refers to the current instance here...
                // but, you'll likely run into a headache here trying to
                // maintain a reference to `this` inside nested closures or
                // event handlers.  Since "var this = 'mua hahaha!" can be
                // set from the caller.

                this.yourdata = 1;

                // note: you can assign an object directly to .prototype
            }

        // } // end fake class

 

    // alternately, here's a simple closure based object

        function YourObject(options) {  // think: class YourObject {

            // the constructor is this whole class/function

            // use `self` from here on out to refer to the object instance.
            // `this` is just a variable that can change in the calling context.
            // It's kinda a hack, but a simple one:
            var self = this;

            // example: private variable.  Put everything in one object reference.
            var data = { 'test': 'this is a private var' };

            // example: public variable.  Put everything in one object reference.
            self.data2 = { 'test': 'this is a public var' };

            // example: private function
            var yourFunction1 = function(){

                // reference to private variable
                console.log(data);

                // reference to public variable
                console.log(self.data2);
            }

            // example: public function
            self.yourFunction2 = function(){

                // reference to private variable
                console.log(data);

                // reference to public variable
                console.log(self.data2);

            }
        }



    // instantiate objects with

        var o = new YourObject();

    //NOTE: you can also reuse the constructor with

            function TestClass() { } // child object


            // some example use object.  Note no args pass
            TestClass.prototype = new YourObject();

            // you can also use a function reference, but will need to call the constructor
            TestClass.prototype = YourObject;
            function TestClass {
             
            }
 
    // example2 data on prototype is isolated across instances

            function Test1() {
                this.a = 1;
            }

            function Test2() {
            }

            Test2.prototype = new Test1();

            var t2 = new Test2();

            t2.a
            // 1



OOP with prototypes

    Everything assigned to prototype is public

    You can change prototypes on the fly (even for already-defined objects)

    The prototype is an Object

    Note: you may have some preserving `this` reference in prototype closures.

    You can use bind() method in modern browsers to preserve the this method:

            fun.bind(thisArg[, arg1[, arg2[, ...]]])

        // eg

            this.element.click(

                // `this` will be redefined to a dom element inside click
                (
                     function() {

                        // ...
                        // after bind, `this` reference will be restored to refer to object context
                     }

                ).bind(this)     // <=== pass in and override this from the current context.
                                // this creates a function wrapper
            );


    // use bind polyfill for old browers

        if (!Function.prototype.bind) {
          Function.prototype.bind = function(oThis) {
            if (typeof this !== 'function') {
              // closest thing possible to the ECMAScript 5
              // internal IsCallable function
              throw new TypeError('Function.prototype.bind - what is trying to be bound is not callable');
            }

            var aArgs   = Array.prototype.slice.call(arguments, 1),
                fToBind = this,
                fNOP    = function() {},
                fBound  = function() {
                  return fToBind.apply(this instanceof fNOP
                         ? this
                         : oThis,
                         aArgs.concat(Array.prototype.slice.call(arguments)));
                };

            if (this.prototype) {
              // native functions don't have a prototype
              fNOP.prototype = this.prototype;
            }
            fBound.prototype = new fNOP();

            return fBound;
          };
        }


    Personally, the `bind` approach looks like an ugly hack to me.  I think the closure approach looks much cleaner.


OOP with closure pattern

    Public

        function ConstructorName(...) {
            this.membername = value;
        }

        // static copy -- saves memory
        ConstructorName.prototype.membername = value;

    Private

        function ConstructorName(...) {
            var that = this;
            var membername = value;
            function membername(...) {...}

        }

    Note: The function statement

        function membername(...) {...}

        is shorthand for

        var membername = function membername(...) {...};

    Privileged

        function ConstructorName(...) {
            this.membername = function (...) {...};
        }



static vs instance scope

    // remember that functions are first class objects.

    // static method
    ConstructorName.method = function () { /* code */ }

    // public instance method
    // NOT static
    ConstructorName.prototype.method = function () { /* code using this.values */ }

    // same goes for data
    ConstructorName.prototype.data = 'this is NOT static data';
    ConstructorName.data = 'this IS static data';

    // test example
        Test1 = function(){}
        Test1.prototype.d1 = 1

        t1 = new Test1();
        t2 = new Test1();

        t2.d1 = 2
        t1.d1
        // 1
        t2.d1
        // 2


`this` limitations

    this cannot refer to an object before it's completely initialized.

    var a = {
        x: 1,
        y: 2,
        z: this.x + this.y,   // NaN, won't work
        z2: function() { return this.x + this.y }   // OK: a.z2() == 3
    }


inheritence

    var childObject = Object.create(parentObject);


    // base class
    function Actor(name) {
        this.name = name;
    }
    Actor.prototype.canSpeak = true;


    // extended class
    function SilentActor() {
        // pass along constructor args
        // eg: super(arguments)
        Actor.apply(this, arguments);
    }

    // set constructor prototype
    // eg: SilentActor extends Actor
    SilentActor.prototype = new Actor();
    SilentActor.prototype.canSpeak = false;


Quick Language Reference: VBScript

These are old notes that were initially for making edits to VBScript ASP pages (before .Net), though most the VBScript syntax will carry over to windows shell scripting (if you want to use VBScript instead of DOS-style syntax).  The last few examples are for Windows shell.

echo

    Wscript.Echo "hello world"

execute vbscript on command line

    rem This is the command line version
    cscript "C:\Users\guest\Desktop\123\MyScript.vbs"
    OR

    rem This is the windowed version
    wscript "C:\Users\guest\Desktop\123\MyScript.vbs"

ASP page def, char encoding

    <%@LANGUAGE="VBSCRIPT" CODEPAGE="1252"%>

require explicit variable declaration

    Option Explicit

include file

    <!--#include file="z_common.asp"-->

declare variable

    Dim lstrID

comment

    ' this is a comment

multiline comments

    ' bwa hahaha
    '   hahaha
    '   hahaha !
    ' (evil laugh)

string concatenation

    Dim lstrSQL;

    lstrSQL = "UPDATE " & YOURTABLE & ""
    lstrSQL = lstrSQL & " " & "SET"
    lstrSQL = lstrSQL & " " & "  AccountNumber = '" & replaceForDB(lstrAccountNum) & "',"
    lstrSQL = lstrSQL & " " & "  ContactType = '" & replaceForDB(lstrContactType) & "',"
    lstrSQL = lstrSQL & " " & "  Comments = '" & replaceForDB(lstrComments) & "',"
    lstrSQL = lstrSQL & " " & "  NextContactDate = #" & replaceForDB(lstrNextContactDate) & "#"
    lstrSQL = lstrSQL & " " & "WHERE"
    lstrSQL = lstrSQL & " " & "  CallID = " & replaceForDB(lstrID) & ""


if, else if,  else

    If ((lstrAction = "edit") AND (isAdmin(gstrUsername) = 1)) Then


    ElseIf ((lstrAction = "delete") AND (isAdmin(gstrUsername) <> 1)) Then

        ' example

    Else

        Response.Write("Unknown action or unauthorized user passed to call detail action page!")
        Response.End

    End If

form vars

    if (request.Form("submit")="Submit") Then

print

    Response.Write("Unknown action or unauthorized user passed to call detail action page!")
    Response.End

function

    Function replaceForDB(argstrToReplace)
    ' PASS    : String to replace
    ' RETURNS : String with baddies removed

        Dim lstrReturn

        If (IsNull(argstrToReplace) <> True) Then
            lstrReturn = Replace(argstrToReplace,"'","''")
        Else
            lstrReturn = ""
        End If

        ' return by setting function name
        replaceForDB = lstrReturn

    End Function 'replaceForDB


redirect

    Response.Redirect("test.asp?id=" & id)

date math

    thisMonth = Now
    nextMonth = DateAdd("m", +1, thisMonth)
    nextYear = DateAdd("yyyy", +1, thisMonth)

    firstDay = dateFormatYYYYMMDD(DateSerial(Year(thisMonth), Month(thisMonth), 1))
    firstDayNextMonth = dateFormatYYYYMMDD(DateSerial(Year(nextMonth), Month(nextMonth), 1))
    firstDayNextYear = dateFormatYYYYMMDD(DateSerial(Year(nextYear), Month(nextYear), 1))

    Wscript.Echo firstDay
    Wscript.Echo firstDayNextMonth
    Wscript.Echo firstDayNextYear


    Function dateFormatYYYYMMDD(myDate)
         d = lPad(Day(myDate), 2, "0")
         m = lPad(Month(myDate), 2, "0")
         y = Year(myDate)
         dateFormatYYYYMMDD= y & "-" & m & "-" & d
    End Function


    Function lPad(s, l, c)
      Dim n : n = 0
      If l > Len(s) Then n = l - Len(s)
      lPad = String(n, c) & s
    End Function


DB connection strings


    ' trusted connection
    Const DB_CONNECT_STRING = "Driver={SQL Server};server=YOURSERVER;Database=YOURTABLE;Trusted_Connection = yes;"


    ' user password

    Const adOpenStatic = 3
    Const adLockOptimistic = 3
    Const DB_CONNECT_STRING = "Provider=SQLOLEDB.1;Data Source=YOUR_HOST;Database=YOUR_DB;User Id=YOUR_USER;Password=YOUR_PASSWORD;"


execute query

    Set objConnection = CreateObject("ADODB.Connection")
    Set objRecordSet = CreateObject("ADODB.Recordset")

    objConnection.Open DB_CONNECT_STRING

    Set objCommand = CreateObject("ADODB.Command")
    objCommand.ActiveConnection = objConnection

    objCommand.CommandText = strScript    ' executes first script
    objCommand.Execute

    objConnection.Close


database select/loop

    Set objRS = Server.CreateObject("ADODB.Connection")
    strSQL = "SELECT * FROM news WHERE news_id= " & news_id & ""
    objRS.Open strSQL, objConn, adOpenStatic, adLockReadOnly

    ' Loop through the recordset and populate the list
     Do Until objRS.EOF = True
      headline  = objRS("news_headline").Value
      copy  = objRS("news_copy").Value
      datetime  = objRS("news_date").Value
     objRS.Movenext
     Loop

    objRS.Close


transaction

    Dim lstrDBError
    lstrDBError = ""
    If lobjDBConn.Errors.Count <> 0 then
         For Each lobjDBErr In lobjDBConn.Errors
             lstrDBError = lstrDBError & "" & lstrDBError.Number & " - " & lstrDBError.Description & "<br>" & CHr(13)
         Next
    End If

    If (lstrDBError = "") Then
        lobjDBConn.CommitTrans
    Else
        lstrDBError = "<br><strong>An error has occured:<br>" & lstrDBError & "</strong><br>" & Chr(13)
        lobjDBConn.RollbackTrans
        Response.Write(lstrDBError)
        Response.End
    End If

    lobjDBConn.Close
    Set lobjDBConn = Nothing


for loop

    For i = 0 To UBound(ArrayOfValues)
        Response.Write "Value " & i & " is " & ArrayOfValues(i) & "<br>"
    Next


constants

    Const filename = ".\ILCOLL.csv"

split string

    sVal = Request.Form("Listbox")
    ''sVal = "Joe, Bob, Tom, Phil, Toby" for argument's sake

    ArrayOfValues = Split(sVal, ", ")

    For i = 0 To UBound(ArrayOfValues)
    Response.Write "Value " & i & " is " & ArrayOfValues(i) & "<br>"
    Next


loop over text file

    Set fso = CreateObject("Scripting.FileSystemObject")
    Set listFile = fso.OpenTextFile("list.txt")
    do while not listFile.AtEndOfStream
        fName =  listFile.ReadLine()
        fso.CreateTextFile(fName + ".txt")
    loop


str replace

    Replace(string,find,replacewith[,start[,count[,compare]]])


escape

    You can escape by doubling the quotes

    g="abcd """ & a & """"
    or write an explicit chr() call

    g="abcd " & chr(34) & a & chr(34)


scrape webpage


  ' Create an xmlhttp object:
  With CreateObject("MSXML2.XMLHTTP")
    .open "GET", sURL, False
    .send
    Do Until .readyState = 4 : Wscript.Sleep 50 : loop
    sPageText = .responseText
    ' or responseHTTP, or responsebody (BIN)
  end with

  if Instr(sPageText, "Your Match String") > 0 Then
    with CreateObject("Scripting.FileSystemObject)
      with .OpenTextFile("C:\Somewhere\logfile.log, 8)
        .writeline "Found Match: : & Now()
      end with
    end with
  end if



complete example - update dates in table

    Const adOpenStatic = 3
    Const adLockOptimistic = 3
    Const DB_CONNECT_STRING = "Driver={SQL Server};server=YOURSERVER;Database=YOURDATABASE;Trusted_Connection = yes;"
    Dim strScript
    Dim strScript2

    strScript = ""
    strScript2 = ""

    thisMonth = Now
    nextMonth = DateAdd("m", +1, thisMonth)
    nextYear = DateAdd("yyyy", +1, thisMonth)

    firstDay = dateFormatYYYYMMDD(DateSerial(Year(thisMonth), Month(thisMonth), 1))
    firstDayNextMonth = dateFormatYYYYMMDD(DateSerial(Year(nextMonth), Month(nextMonth), 1))
    firstDayNextYear = dateFormatYYYYMMDD(DateSerial(Year(nextYear), Month(nextYear), 1))

    strScript = "update YOURTABLE.dbo.payments set YOURFIELD = '" + firstDayNextMonth + "' where YOURFIELD =  CONVERT(DATETIME, '" + firstDay + "', 102) "
    strScript2 = "update YOURTABLE.dbo.payments set YOURFIELD = '" + firstDayNextYear + "' where YOURFIELD =  CONVERT(DATETIME, '" + firstDay + "', 102) "

    Set objConnection = CreateObject("ADODB.Connection")
    Set objRecordSet = CreateObject("ADODB.Recordset")

    objConnection.Open DB_CONNECT_STRING

    Set objCommand = CreateObject("ADODB.Command")
    objCommand.ActiveConnection = objConnection

    objCommand.CommandText = strScript
    objCommand.Execute

    objCommand.CommandText = strScript2
    objCommand.Execute

    objConnection.Close

    Function dateFormatYYYYMMDD(myDate)
         d = lPad(Day(myDate), 2, "0")
         m = lPad(Month(myDate), 2, "0")
         y = Year(myDate)
         dateFormatYYYYMMDD= y & "-" & m & "-" & d
    End Function


    Function lPad(s, l, c)
      Dim n : n = 0
      If l > Len(s) Then n = l - Len(s)
      lPad = String(n, c) & s
    End Function


execute com object

    Set ccObj = CreateObject("YOURDLL.CLASS")

    Dim somestring
    somestring = "some data"

    ccObj.InBuffer = somestring
    ccObj.Encrypt
    somestring = ccObj.OutBuffer

    Wscript.Echo pass


Quick Language Reference: Bash

general shell debug flags:

    sh -n    check syntax
    sh -v    echos commands before executing
    sh -x    echos commands after running them
    sh -u    gives an error message

define variable

    a=1

to reference a variable, prepend it with $ or wrap with ${}

    echo "your var is $a"
    echo "your var is ${a}"   # protects from ambiguity from connecting chars

    like perl/php, variables in a double quoted string are interperlated.

to make the variable available to sub shells

    export a=1

    # spawn new shell
    bash

    echo $a
    # prints 1

note: executing a script, or a pipe will typically create a new shell
avoid sub shells if you don't need them

prevent string interpolation

    use single quotes

    script 'a$s'


escape single quote

    '"'"'
or
    '\''

    "don't"

    # use double quotes only on string
    'don'"'"'t'

    'don'\''t'


dynamic string arguments

The interpreter is smart enough to pass along a string that's already parsed, to the next command.  you do NOT need quotes around interpolated strings:

    example:

        # does not work ... too many single quotes
        file=.
        args=" -a '$file' "
        ls $args
        # fails.  looking for a file named \'.\'

        # works ... no single quotes within string:
        file=.
        args=" -a $file "
        ls $args

    summary: The only time you need to encapsulate string data in quotes is on the initial shell call (initial string parse). ... I think :)


add numbers

    num=$((num1 + num2))
    num=$(($num1 + $num2))       # also works
    num=$((num1 + 2 + 3))        # ...
    num=$[num1+num2]             # old, deprecated arithmetic expression syntax


error handling / exception handling

    # $? reports last error.
    # WARNING, IT IS CONSUMED ON FIRST TEST, SINCE EACH COMMAND WRITES TO IT

        grep 23412341  .
        grep: .: Is a directory

        $ echo $?
        2

        $ echo $?
        0

    # example test
    if [[ $? == 0 ]]
    then
        java Test
    fi


define / create an alias (bash syntax)

   
    alias name=value
    alias name2='command'
    alias name3='command arg1 arg2'
    alias name4='/path/to/script'
    alias name5='/path/to/script.pl arg1'


disable alias that overrides command, use ''

    'ls'

    unalias aliasname


alternate string quote syntax

c
    a = $'te\'st'

        strings that are scanned for ansi c like escape sequences.

i18n

    a = $"test"

    means get the international text. if there is a translation available for that string, it is used instead of the given text. If not, or if the locale is C/POSIX, the dollar sign simply is ignored, which results in a normal double-quoted string.



copy var

    b=$a
    echo $b


if then, else if, else

    # test if variable is NOT set
    if [ -z "$VAR" ]; then

    # else if
    elif [ -z "$VAR2" ]; then

    else
   
   
    # end
    fi

    # note you can also write the 'then' on its own line


negation

    !     means not


test variable has a value

    if [ -z "$VAR" ];
    then
        echo "value is not set"
    fi


test file existence:


    if [ -f $FILE ];
    then
       echo "File $FILE exists."
    else
       echo "File $FILE does not exist."
    fi

    if [ ! -f $FILE ];
    then
       echo "File $FILE does not exist."
    fi


test if file contains pattern

    if grep -q FOOBAR "$File"; then
       # file has FOOBAR
    fi

test if file does not contain pattern

    if ! grep -q SomeString "$File"; then
       # Some Actions
    fi


then: can be written on the same line with a ; or on the next line

    # people likely do this because veritical space is limited on an 80x24 terminal
    if ! grep -q SomeString "$File"; then
        echo 1;
    fi

    # means the same as
    if ! grep -q SomeString "$File"
    then
        echo 1;

    fi


shell redirection

    cmd 2>      redirect stderr
    cmd &>      redirect both stderr and stdout

    cmd >&n     send output to descriptor n
    cmd m>&n    redirect output that would have gone to m to n
    cmd <&n     get input from descriptor n
    cmd <&-     close standard in


silence output


    # SILENCIO POR FAVOR!
    cmd &> /dev/null


pipes | are used to chain commands together.  output of the first command is wired up to the input of the next command (like snapping together legos)

    comand | tee file.txt   prints to screen and saves to file.txt


edit previous command

    ^foo^bar        executes previous command, except replaces foo with bar


loops:

    for loop

        for ((i=1;i<=100;i++));
        do
           # your-unix-command-here
           echo $i
        done
   
    while loop

        while ((1));
        do
           # your-unix-command-here
            sleep 1;
        done
       

    for

         for x in file1 file2 file3
         do
         sed 's/thier/their/g' $x > ,$x
         mv ,$x $x
         done
       

    # index iteration
     for ((i=0; i<10; i++)); do
         echo $i
     done


bash heredoc syntax

    # note no ; required
    # for quoting multiline strings
    cat << ENDOFBLOCK
            sdfg
ENDOFBLOCK

    Note: you can't assign this straight to a var
    For example, you can use a cat do

    variable=$(cat <<EOF
<html>
<body>
<p>THis is a test</p>
</body>
</html>
EOF
)


test loop

    You can preview what loops will do by putting an echo in front of the commands.  you don't want to realize there was a mistake after running the command a thousand times.

        # test
        for a in *
        do
            echo mv a b
        done

        # then use the command
        mv a b
   

    This can mess up " in output though, since echo will interpret it and discard.   if you need exact output, wrap the block in a literal heredoc.  THis will preserve all quotes " ' ` and meta chars

    So, intest you can use a heredoc to prevent interpolation

    cat <<DEBUG
    echo PARAM=`  grep  "$ARG"  /var/tmp/setfile  | awk '{print $2}' `
DEBUG
    # prints:
    #     echo PARAM=`  grep  "$ARG"  /var/tmp/setfile  | awk '{print $2}' `


piping output from one command to a loop

    find photos/ | while read file; do echo $file ; done;


loops in bash and variable scope:

  counter=0
  for f in $(find .)
  do
    let counter+=1
  done
  echo $counter

problem: if ls returns a large result, can overflow the buffer.

      counter=0
      find . | while read f
      do
        let counter+=1
      done
      echo $counter


    the problem: variables inside loop in their own shell, can't be used if needed.

    For read only access, use export in the outer context

        export var=1

    Though, this won't work if you need write access.  In this case you need < <() syntax
    for example


      counter=0
      while read f
      do
        let counter+=1
      done < <(find .)
      echo $counter

    ugly syntax, but works


handle whitespace in filename

    find <path> -mtime +7 | while read line
    do
        # just wrap with double quotes
        mv "$line" /somewhere/else
    done

you can embed a comment string in the heredoc marker

    # comment following line out after testing
    cat <<'#ENDDEBUG'

    echo PARAM=`ls -la  | awk '{print $3}' `
    #ENDDEBUG



parse delimted test with `cut`

    # list all users defined on system
    # split on :, print the first field
    cut -d: -f1 /etc/passwd


increment var

  y=$[y+1]
  y=$(($y+1))
  ((y++))


polling process (repeatedly testing)

    while (( 1 ));
     do   ls -lah sedOSEZA8 ;
     sleep 1;
    done


see all characters used in a file

    # change char to char\n
    cat Configuration | sed -e 's/\(.\)/\1\n/g' | sort | uniq



create a simple index page for .html files in a folder

 ls *.html -1 | while read f ; do echo "<a href='$f'>$f</a><br>"; done >> index.htm


scp - secure copy (use instead of ftp)

  scp  d_HomeBk.sql  user@server:/remote/path/to/dir/

  note: if there are spaces in file name use: encased/escaped quotes "\"  \""

    eg  

      scp  d_HomeBk.sql  user@server:"\"/path/to/file/\""

    or enclose entire string in single quotes, with double

      scp  d_HomeBk.sql  'user@server:"/path/to/file/"'

    use -r to copy recursively



find files

    # last 24 hours
    find . -mtime 0

    # older than a month
    find . -mtime +31

    # find files only
    find -type f


when backticks overflow command line, can group in blocks and pipe to a loop:

    % sh
    find . -type f -mtime -1 -print |
    fmt -1000 |
    while read files
    do pr -n $files
    done | lpr
    exit

    ...or use xargs
    xargs lpr < allfiles.tmp

or

    # NOTE the -I defines a string to replace
    # default behavior sends to first argument of command.
    command | xargs -I '{}' somecommand stuff '{}'
    command | xargs rm -rf


# copy a lot of image files, convert sizes


    ls -1 | while read file; do echo $file; \
    convert -resize 460 "$file" "/cygdrive/c/temp/yourfiles/$file" ; done


# slightly better complex ... can be stopped, restarted

    $destdir = /cygdrive/c/temp/yourfiles

    ls -1 | while read $file
    do
      if [[ ! -e $destdir/$file ]]
      then
        echo $file
        convert -resize 460 $file $destdir/$file
      fi
    done



internal file separator.  note: also there is the internal variable IFS (internal file seperator) that is noramally set to white space

    while can parse a line, such as

    IFS=':'
    cat  /etc/passwd | while read var1 var2 var3 var4
    do
     ...
    done

    # although awk is probably the better tool for this



the previous file edited on bash shell

    !$    





convert stream to lower case:

    alias lc="tr '[:upper:]' '[:lower:]'"
    alias uc="tr '[:lower:]' '[:upper:]'"
 
    eg:

     cat file.txt | lc



copy all but some files

    cpio can copies the contents of /source-dir to /dest-dir, but omits
    files  and directories named .snapshot (and anything in them).

    cd /source-dir
    find . -name .snapshot -prune -o \( \! -name *~ -print0 \)| cpio -pmd0 /dest-dir


    This command copies the contents of /source-dir to /dest-dir, but omits
    files  and directories named .svn (and anything in them).
    cd /source-dir
    find . -name .svn -prune -o \( \! -name *~ -print0 \)| cpio -pmd0 /dest-dir


profile vs bashrc

    profiles are for interactive login shells (that requre user/pass),
    rc is for non-interactive, non-login


mount cd rom manullly

     mkdir -p /mnt/cdrom && mount -t iso9660 -o ro /dev/cdrom /mnt/cdrom


functions

                # without arg, called like a command line tool
                f() { echo 1; };
                f;


                # with argument
                f() { echo $1; };
                f arg ;


floating point numbers

    bwa hahahah ... :-)
    bash doesn't support floating point arithmetic evaluation, but you can use the pretty-standard bc command in your function to help out.     example:

    function f_to_c { echo "5/9 * ($1 - 32)" | bc -l; }

    Though if your script is getting complex enough to use floating point numbers, it probably should be written in an actual scripting language (python, perl, ruby, php, etc) not bash.

arrays:

    If you are using arrays, you should start to wonder if you should be using bash at all and not another programming language.  :)
    But yes, you can do arrays.

    #simple array:

        name[index]=value

    # explicitly define numeric array


        declare -a chars  
        chars[0]="a"
        chars[1]="b"

       echo ${chars[0]}


    # set list of values

         declare -a arrayname=(element1 element2 element3)

    # bash shorthand: declare/assign array

        arr=(Hello World)

    # in referencing the value, you must enclose in {}

          echo ${arr[0]} ${arr[1]}

    To quote from the man page:
    The braces are required to avoid conflicts with pathname expansion.
    In addition the following funky constructs are available:

          ${arr[*]}         # All of the items in the array
          ${!arr[*]}        # All of the indexes in the array
          ${#arr[*]}        # Number of items in the array
          ${#arr[0]}        # Length of item zero

    # associative array or hash

        declare -A morse  
        morse[a]="dot; dash"
        morse[b]="dash; dot; dot; dot"
        morse[c]="dash; dot; dash; dot"
        # ...

       echo ${morse[$1]}   # write dots, dashes for letter

    # print all of array, use @ (index number) or * (all)

        echo ${morse[@]}

    # to also export declared vars

        declare -x ....


eval "string"

    execute code.  similar to `` or $(), but may read a bit easier

case:  (switch statement)

    Note: no break/exit needed

    # number of feet
    case $ANIMAL in

      horse | dog | cat) echo -n "four";;

      man | kangaroo ) echo -n "two";;

      alien ) echo -n "pi";;

      *)
        #default
        echo -n "an unknown number";;

    esac

print current user

    echo "$USER"
    id -u -n

    To print numeric UID, run:

    id -u

change passwd non-interactive

   
    # reads a list of users passwords (non interactive)
    # may be encrypted
    echo "username:newpass" | chpasswd


find all files not owned by user

    find '!' -uid 0
    find '!' -user root


can also  echo multiline strings

    echo "
    /var/log/top.log {
        missingok
    }
    " > /etc/logrotate.d/top
   

to avoid pipe in main loop

    this:

        while read log
        do
          echo $log
        done < <( svn log $1 | grep -E -e "^r[0-9]+ \| " | tac )

    is mostly the same as:

        svn log $1 | grep -E -e "^r[0-9]+ \| " | tac | while read log
        do
          echo $log
        done

ssh

    Warning: ssh reads from standard input, and can "eat" all your input if used in a while loop.
    Instead, in a loop use:

        cat /dev/null | ssh $u@$h  $COMMAND

    or
         ssh -n


    In general, you should close or redirect stdin for the other commands you
    run, to stop them reading from the file.

        sh call_other_script.sh </dev/null


rotate file based on size

    #!/bin/bash

    f=/var/log/top.log
    date         >> $f
    echo 'top:'  >> $f
    top -b -n 1  >>  $f

    echo '--'    >> $f
    echo 'ps:'   >> $f

    ps auxw      >>  $f

    echo '--'    >> $f
    echo ''      >> $f


    # don't let it get too big
    FILESIZE=$(stat -c%s "$f")
    MAXSIZE=10000000

    if [ $FILESIZE -gt $MAXSIZE ]; then
      mv -f $f $f.1
    fi



count word frequency

    tr ' ' '\n' < /tmp/kevtest_old.txt | sort | uniq -c | sort -nr | head

    27306
    5200 69
    1980 137
    1716 129
     728 93
     611 56
     611 55
     520 42
     520 41
     248 70

    alt:

        awk '{a[$1]++}END{for(k in a)print a[k],k}' RS=" |\n" file > myfile



test string for pattern


    build=123000
    if [[ $build =~ "3" ]]
    then
        echo "has 3"
    fi

    build=123000
    if [[ $build =~ "4" ]]
    then
        echo "has 4"
    fi


convert string list to array

    STRING="test1 test2 test3"

    # convert to array
    list=( $STRING )

    #Or more verbosely:

    declare -a list=( $STRING )

    PS: You can't export IFS and use the new value in the same command. You have to declare it first, then use it's effects in the following command:

    list=( first second third )
    echo "${list[*]}"


    #first element
    list=( first second third )
    echo "${list[0]}"
    # first


nohup


    # don't stop command if the terminal connction is lost

        nohup  one_command_only  &

    #execute multiple commands, call script in future

        # get both process ids
        function sleep_restart() {
                # to stagger node restarts, call script in future
                sleep=1
                echo "Will schedule start in $sleep seconds..."
                echo ""
                echo "    sleep $sleep && /yourcommand_restart"
                echo ""
                nohup sh -c "sleep $sleep && /yourcommand_restart >& /dev/null" &
                sleep 1
        }


find by permission

    find . -perm /222

    Search for files which are writable by somebody (their owner, or their group, or anybody else).

    find . -perm /220
    find . -perm /u+w,g+w
    find . -perm /u=w,g=w

    find . -perm /o=w


find loops in symlinks

    find . -follow -printf ""

find number of processes

    nproc
    cat /proc/cpuinfo

# count words or count repeated lines by occurance

    sort | uniq -c



remove XML tags, remove HTML tags

    sed -n '/^$/!{s/<[^>]*>//g;p;}'


base64 decode

    base64 --decode

    # eg, convert xml 64 data to jpeg
    curl  <YOUR_URL_HERE>  | sed -n '/^$/!{s/<[^>]*>//g;p;}' | base64 --decode > test.jpg


simple tail log, execute trigger when it reads a pattern

    # look for lines containing OutOfMemoryError
    log=/tmp/kevtest.txt
    pattern="OutOfMemoryError"
    while read line;
    do
        if [[ $line =~ $pattern  ]]
        then
            echo "hit $line !"
            break;
        fi
    done < <(tail -f $log)


    alt: with awk

    # no exit
     tail -f /tmp/kevtest.txt | awk '/YourPatternHere/ { system("echo 1");  }'

    # with exit
     tail -f /tmp/kevtest.txt | awk '/YourPatternHere/ { system("echo 1"); exit; }'


hex encoding to ascii

    d=`cat lib.php `

    echo -e $d  | awk '{printf "%s\n", $_}' > kevtest.php


test if socket port is open

    Use netcat (nc) for the client and server tests

    # basic interactive chat

        # server (use same ports on client and server)
        nc -l 7601 -v

        # client (use same ports on client and server)
        nc <HOST> 7601

        # note: exit with ctrl-C


    # scan port (non-interactive)

        nc -z <host> <port>


    # --
    # client examples
    # try socket connection
    # interactive

        nc  <host> <port>

    or (older)

        telnet  <host> <port>

    #scan (one shot, not interactive)

        nc -z <host> <port>

    # For a quick non-interative check (with a 5 seconds timeout):

        nc -z -w5 <host> <port>


    # --
    # SERVER
    # note: low number ports (such as 80) require higher perms
    # create socket that stays open indefinitely until manually closed
    #listen locally on port 1024 for all traffic and dump the output to STDOUT

        nc -l 1024 -k

    # more verbose

        nc -l 1024 -k  -v

    # close after first message

        nc -l 1024


    # more complex request (send mail)

        nc localhost 25 << EOF
        HELO host.example.com
        MAIL FROM: <user@host.example.com>
        RCPT TO: <user2@host.example.com>
        DATA
        Body of email.
        .
        QUIT
        EOF


--
BASH BRACKETS/BRACES/PARANS

There are a lot of brackets in Bash, like () (()) [] [[]], and some have overloaded meanings, which can get confusing.  Here's a quick overview of what they mean.  These types of symbols are hard to search for explanations as well.


bracket options in 'if' statements

    no brackets        execute shell commands and tests exit code
                    0 == true in this case, which can be counter-intuitive
                    'exit code' should really be named 'error code', so
                    'no errors' means true.  last code is stored in variable $?

    [ ] are simpler, older, POSIX compliant.
         it is usually just an alias to /bin/test, /bin/[

    [[ ]]  the double brackets are builting bash keywords

        in general use [[  vs [ in tests if you don't need to port to other shells

        it's newer and processes internally (better for whitespace handling)

        The double bracket enables additional functionality.
        For example, you can use && and || instead of -a and -o
        and there's a regular expression matching operator =~

    (( )) are used for arithemetic operations.  for example
            a=((a++))
            for ((i=0; i<10; i++)); do echo $i; done;


    $[ ]     deprecated syntax meaning (( ))


other bracket contexts


    ( )        1. are used to create a subshell, won't affect current shell
            ( cd /tmp; pwd ); pwd;

            2. are also used to create arrays, [ ] used for array index
                 array=(1 2 3)
                 echo ${array[1]}

            3. are also used used to define functions.

                # without arg, called like a command line tool
                f() { echo 1; }; f;
                # with argument
                f() { echo $1; }; f arg ;


            4. process substitution is:

                  cmd <(list)   is similar to        cat list | cmd
                or
                  cmd > >(list)    is similar to        cmd | list


                 The effect of process substitution is to make each list act like a file.

                however >() is more powerful than a normal pipe:
                You can't use | to redirect standard output and standard error to different programs.


    { }    does a couple things:
            1. unambiguously identifies variables. example ${a}${b}
            2. executes a sequence of commands in the CURRENT shell context, opposite of ()

                { date; top -b -n1 | head ; } >logfile

            There is a subtle syntactic difference with ( ), though (see bash reference) ; essentially, a semicolon ; after the last command within braces is a must,
            and the braces {, } must be surrounded by spaces.

    [ ] 1. can mean a range operator, or matching list

            ls [abc][1-4]
           
        2. can be the index designator in an array  ${a[0]}
               

    $( )     runs a command and drops the output in another.  like backticks ``
            usefule for nesting operations, example:

                # find string 'bar' in files named 'foo'
                grep bar $( find -name foo )




--
BASH STRING MANIPULATION

strip extension

    nameis=${filename%.php}
    nameis=${filename%.*}  # general


substring examples:

    The variable var contains /a/b/c/d/e.f.g:


        Expression   Result

        ${var}   /a/b/c/d/e.f.g

        # match first from left
        ${var#/*/}   b/c/d/e.f.g

        #match last from left
        ${var##/*/}   e.f.g

        # match first from right  
        ${var%.*}   /a/b/c/d/e.f

        # match last from right
        ${var%%.*}   /a/b/c/d/e

        # misc
        ${var%%/*/}   /a/b/c/d/e.f.g
        ${var%%/*}
        ${var%/b*}   /a
        ${var%%/b*}   /a


by length

    stringZ=abcABC123ABCabc
    #       0123456789.....
    #       0-based indexing.

    echo ${stringZ:0}                            # abcABC123ABCabc
    echo ${stringZ:1}                            # bcABC123ABCabc
    echo ${stringZ:7}                            # 23ABCabc

    echo ${stringZ:7:3}                          # 23A
                                                 # Three characters of substring.



    # Is it possible to index from the right end of the string?
       
    echo ${stringZ:-4}                           # abcABC123ABCabc
    # Defaults to full string, as in ${parameter:-default}.
    # However . . .

    echo ${stringZ:(-4)}                         # Cabc
    echo ${stringZ: -4}                          # Cabc
    # Now, it works.
    # Parentheses or added space "escape" the position parameter.



find replace (substring replacement)

    Replace first match of $substring with $replacement:

        ${string/substring/replacement}

    Replace all matches of $substring with $replacement.

        ${string//substring/replacement}




bash string editing operators

    Operator        Explanation

    ${variable#pattern}    Delete the shortest part of pattern that matches the beginning of variable's value. Return the rest.
    ${variable##pattern}    Delete the longest part of pattern that matches the beginning of variable's value. Return the rest.
    ${variable%pattern}    Delete the shortest part of pattern that matches the end of variable's value.Return the rest.
    ${variable%%pattern}    Delete the longest part of pattern that matches the end of variable's value.Return the rest.

    The patterns can be filename wildcard characters: *, ?, and []; with string editing operators, wildcards match strings in the same way they match filenames. (These are not sed-like regular expressions.) The first two operators, with #, edit variables from the front. The other two, with %, edit from the end. Here's a system for remembering which does what: you put a number sign (#) at the front of a number and a percent sign (%) at the end of a number.