<commit_before># webutil

webutil provides automatic website analysis for the savvy web developer. It helps you build faster websites and pinpoint bottlenecks. It provides:

* quick glance summary of key metrics
* HTTP and JavaScript error reporting
* caching analysis with an empty and primed cache
* detection of 2<sup>7</sup> commonly used front end libraries, content management systems and third party embeds (aka What's That Site Running?)
* automatic asset (HTML, JavaScript, CSS, images) optimization facility:
    * minification
    * compression
    * asset optimization benefit analysis
* header verification and analysis
* built in user agent support with screen dimensions
* standard `phantomjs` facilities
    * screenshots
    * HAR file generation

webutil is a [phantomjs](http://phantomjs.org)-based tool.

## INSTALL

1. Install [phantomjs](http://phantomjs.org/download.html)
2. Get the code: `git clone git://github.com/ditesh/webutil.js`
3. Run it: `chmod a+x wush; ./wush www.reddit.com`

## What can it do?

Lots. Let's get relevant stats for Reddit:

    $ chmod a+x wush  # make the shell script executable
    $ ./wush reddit.com
    webutil.js 1.0.1 (c) 2012-2013 <NAME> <<EMAIL>>

    [Summary]
    Requests    33 request(s), 385103 bytes (376 KB), 0 redirect(s)
    Resources   HTML: 2, CSS: 1, JS: 4, images: 26, others: 0
                UTF-8: 4, ISO-8859-1: 0, others: 0, not-specified: 29
                compressed: 27, not-compressed: 6
    Timing      first byte: 291 ms, onDOMContentLoaded: 667 ms, onLoad: 2386 ms, fully loaded: 4390 ms
    Errors      4xx: 0, 5xx: 0, JS: 0

As you can see, it provides a nicely formatted summary about the site highlighting key areas.

### Getting URL's

Now, lets get a complete list of URL's accessed by the browser when loading up the site:

    $ ./wush -lu reddit.com
    webutil.js 1.0.1 (c) 2012-2013 <NAME> <<EMAIL>>

    [Summary]
    ... snipped for brevity ...

    [URLs]
    HTML  115084  http://www.reddit.com/
    JS    45830   http://ajax.googleapis.com/ajax/libs/jquery/1.7.2/jquery.min.js
    CSS   78617   http://www.redditstatic.com/reddit._E2WnMcei1o.css
    JS    58064   http://www.redditstatic.com/reddit.en.yRpDmsrGWVQ.js
    ... snipped for brevity ...


Content type, resource size and resource URL is provided. Occasionally, we are only interested in resources from the same domain. This can be achieved by using the `-d` parameter:

    $ ./wush -u -d reddit.com
    webutil.js 1.0.1 (c) 2012-2013 <NAME> <<EMAIL>>

    [Summary]
    ... snipped for brevity ...

    [URLs]
    HTML  110196  http://www.reddit.com/
    HTML  2108    http://www.reddit.com/api/request_promo

Whoops, clearly the URL list is wrong. As it turns out, reddit.com keeps its assets in other domains. We need to rerun the command using the -dd to specify other domains reddit.com uses.

    $ ./wush -u -dd reddit,media reddit.com
    webutil.js 1.0.1 (c) 2012-2013 <NAME> <<EMAIL>>

    [Summary]
        ... snipped for brevity ...

    [URLs]
    HTML  114590  http://www.reddit.com/
    CSS   78497   http://www.redditstatic.com/reddit._E2WnMcei1o.css
    JS    68817   http://www.redditstatic.com/reddit.en.yRpDmsrGWVQ.js
    PNG   9670    http://www.redditstatic.com/sprite-reddit.HLCFG7U22Hg.png
    ... snipped for brevity ...

The `-dd` pulls up URL's by doing pattern matching on all hostnames (ie, in this case, hostname is matched against the words `reddit` and `media`). This allows finegrained controlled of which URL's are to be included in the analysis.

The important thing to note about `-d` and `-dd` is that the numbers in the summary are reflective of the URL's analyzed.

### Primed Cache

There is an inbuilt facility to provide the summary but with a primed cache. This is a good way to check whether the browser is, in fact, caching the site on subsequent visits.

    # One execution
    $ ./wush reddit.com
    webutil.js 1.0.1 (c) 2012-2013 <NAME> <<EMAIL>>

    [Summary]
    Requests    33 request(s), 385103 bytes (376 KB), 0 redirect(s)
    Resources   HTML: 2, CSS: 1, JS: 4, images: 26, others: 0
                UTF-8: 4, ISO-8859-1: 0, others: 0, not-specified: 29
                compressed: 27, not-compressed: 6
    Timing      first byte: 291 ms, onDOMContentLoaded: 667 ms, onLoad: 2386 ms, fully loaded: 4390 ms
    Errors      4xx: 0, 5xx: 0, JS: 0

    # Two executions
    $ ./wush -c 1 reddit.com
    webutil.js 1.0.1 (c) 2012-2013 <NAME> <<EMAIL>>

    [Summary]
    Requests    14 request(s), 135463 bytes (132 KB), 0 redirect(s)
    Resources   HTML: 2, CSS: 0, JS: 1, images: 11, others: 0
                UTF-8: 3, ISO-8859-1: 0, others: 0, not-specified: 11
                compressed: 12, not-compressed: 2
    Timing      first byte: 291 ms, onDOMContentLoaded: 667 ms, onLoad: 2386 ms, fully loaded: 4390 ms
    Errors      4xx: 0, 5xx: 0, JS: 0

When the `-c` parameter is passed, the page is reloaded the specified number of times. In the example above, the page is loaded once, and then reloaded one more time (reflecting the effect of passing `-c 1`).

You can see the benefits of caching as the number of requests, page size etc drop. This is best used with `-d` or `-dd` to exclude third party resources.

### Optimizing

There is a bundled shell script that can automatically take the URL's and attempt to compress/minify CSS, JS or image assets.

    $ chmod a+x tools/downloader tools/optimizer
    $ tools/downloader -dd reddit,media reddit.com

    Web Downloader Tool 1.0.1 (c) 2013 <NAME> <<EMAIL>>
<commit_msg>Updated README with timing information<commit_after># webutil

webutil provides automatic website analysis for the savvy web developer. It helps you build faster websites and pinpoint bottlenecks. It provides:

* quick glance summary of key metrics
* HTTP and JavaScript error reporting
* caching analysis with an empty and primed cache
* detection of 2<sup>7</sup> commonly used front end libraries, content management systems and third party embeds (aka What's That Site Running?)
* automatic asset (HTML, JavaScript, CSS, images) optimization facility:
    * minification
    * compression
    * asset optimization benefit analysis
* header verification and analysis
* built in user agent support with screen dimensions
* standard `phantomjs` facilities
    * screenshots
    * HAR file generation

webutil is a [phantomjs](http://phantomjs.org)-based tool.

## INSTALL

1. Install [phantomjs](http://phantomjs.org/download.html)
2. Get the code: `git clone git://github.com/ditesh/webutil.js`
3. Run it: `chmod a+x wush; ./wush www.reddit.com`

## What can it do?

Lots. Let's get relevant stats for Reddit:

    $ chmod a+x wush  # make the shell script executable
    $ ./wush reddit.com
    webutil.js 1.0.1 (c) 2012-2013 <NAME> <<EMAIL>>

    [Summary]
    Requests    33 request(s), 385103 bytes (376 KB), 0 redirect(s)
    Resources   HTML: 2, CSS: 1, JS: 4, images: 26, others: 0
                UTF-8: 4, ISO-8859-1: 0, others: 0, not-specified: 29
                compressed: 27, not-compressed: 6
    Timing      first byte: 291 ms, onDOMContentLoaded: 667 ms, onLoad: 2386 ms, fully loaded: 4390 ms
    Errors      4xx: 0, 5xx: 0, JS: 0

As you can see, it provides a nicely formatted summary about the site highlighting key areas. Timing information, so often key to web developers, is covered in more detail below.

### Getting URL's

Now, lets get a complete list of URL's accessed by the browser when loading up the site:

    $ ./wush -lu reddit.com
    webutil.js 1.0.1 (c) 2012-2013 <NAME> <<EMAIL>@gathani.org>

    [Summary]
    ... snipped for brevity ...

    [URLs]
    HTML  115084  http://www.reddit.com/
    JS    45830   http://ajax.googleapis.com/ajax/libs/jquery/1.7.2/jquery.min.js
    CSS   78617   http://www.redditstatic.com/reddit._E2WnMcei1o.css
    JS    58064   http://www.redditstatic.com/reddit.en.yRpDmsrGWVQ.js
    ... snipped for brevity ...


Content type, resource size and resource URL is provided. Occasionally, we are only interested in resources from the same domain. This can be achieved by using the `-d` parameter:

    $ ./wush -u -d reddit.com
    webutil.js 1.0.1 (c) 2012-2013 <NAME> <<EMAIL>>

    [Summary]
    ... snipped for brevity ...

    [URLs]
    HTML  110196  http://www.reddit.com/
    HTML  2108    http://www.reddit.com/api/request_promo

Whoops, clearly the URL list is wrong. As it turns out, reddit.com keeps its assets in other domains. We need to rerun the command using the -dd to specify other domains reddit.com uses.

    $ ./wush -u -dd reddit,media reddit.com
    webutil.js 1.0.1 (c) 2012-2013 <NAME> <<EMAIL>>

    [Summary]
        ... snipped for brevity ...

    [URLs]
    HTML  114590  http://www.reddit.com/
    CSS   78497   http://www.redditstatic.com/reddit._E2WnMcei1o.css
    JS    68817   http://www.redditstatic.com/reddit.en.yRpDmsrGWVQ.js
    PNG   9670    http://www.redditstatic.com/sprite-reddit.HLCFG7U22Hg.png
    ... snipped for brevity ...

The `-dd` pulls up URL's by doing pattern matching on all hostnames (ie, in this case, hostname is matched against the words `reddit` and `media`). This allows finegrained controlled of which URL's are to be included in the analysis.

The important thing to note about `-d` and `-dd` is that the numbers in the summary are reflective of the URL's analyzed.

### Primed Cache

There is an inbuilt facility to provide the summary but with a primed cache. This is a good way to check whether the browser is, in fact, caching the site on subsequent visits.

    # One execution
    $ ./wush reddit.com
    webutil.js 1.0.1 (c) 2012-2013 <NAME> <<EMAIL>>

    [Summary]
    Requests    33 request(s), 385103 bytes (376 KB), 0 redirect(s)
    Resources   HTML: 2, CSS: 1, JS: 4, images: 26, others: 0
                UTF-8: 4, ISO-8859-1: 0, others: 0, not-specified: 29
                compressed: 27, not-compressed: 6
    Timing      first byte: 291 ms, onDOMContentLoaded: 667 ms, onLoad: 2386 ms, fully loaded: 4390 ms
    Errors      4xx: 0, 5xx: 0, JS: 0

    # Two executions
    $ ./wush -c 1 reddit.com
    webutil.js 1.0.1 (c) 2012-2013 <NAME> <<EMAIL>>

    [Summary]
    Requests    14 request(s), 135463 bytes (132 KB), 0 redirect(s)
    Resources   HTML: 2, CSS: 0, JS: 1, images: 11, others: 0
                UTF-8: 3, ISO-8859-1: 0, others: 0, not-specified: 11
                compressed: 12, not-compressed: 2
    Timing      first byte: 291 ms, onDOMContentLoaded: 667 ms, onLoad: 2386 ms, fully loaded: 4390 ms
    Errors      4xx: 0, 5xx: 0, JS: 0

When the `-c` parameter is passed, the page is reloaded the specified number of times. In the example above, the page is loaded once, and then reloaded one more time (reflecting the effect of passing `-c 1`).

You can see the benefits of caching as the number of requests, page size etc drop. This is best used with `-d` or `-dd` to exclude third party resources.

### Timing

Timing information is important for web developers. `webutil` offers four parameters:

* first byte: measurement of latency between request and first chunk of the first response
* onDOMContentLoaded: fires when the page's DOM is fully constructed, but the referenced resources may not finish loading
* onLoad: fires when the document loading completes
* fully loaded: fires when there is no more network activity

The four parameters are clearly visible below:

    $ ./wush reddit.com
    webutil.js 1.0.1 (c) 2012-2013 <NAME> <<EMAIL>>

    [Summary]
    Requests    33 request(s), 385103 bytes (376 KB), 0 redirect(s)
    Resources   HTML: 2, CSS: 1, JS: 4, images: 26, others: 0
                UTF-8: 4, ISO-8859-1: 0, others: 0, not-specified: 29
                compressed: 27, not-compressed: 6
    *Timing      first byte: 291 ms, onDOMContentLoaded: 667 ms, onLoad: 2386 ms, fully loaded: 4390 ms*
    Errors      4xx: 0, 5xx: 0, JS: 0

There is an additional `python` based tool that will help execute multiple runs and provide average and standard deviation timing information. This is invoked transparently as follows:

    $ ./wush -repeat 10 reddit.com
    webutil 1.0.1 (c) 2012-2013 <NAME> <<EMAIL>>
    Executing 10 runs on reddit.com

    First Byte:   mean 162.6ms, standard deviation: 30.45ms
    DOMContentLoaded: mean 612.8ms, standard deviation: 32.29ms
    On Load:    mean 1767.4ms, standard deviation: 109.02ms
    Fully Loaded:   mean 3771.3ms, standard deviation: 109.0ms

For reference, the python script is available in `tools/timing.py`

### Optimizing

There is a bundled shell script that can automatically take the URL's and attempt to compress/minify CSS, JS or image assets.

    $ chmod a+x tools/downloader tools/optimizer
    $ tools/downloader -dd reddit,media reddit.com

    Web Downloader Tool 1.0.1 (c) 2013 <NAME> <<EMAIL>>

{'fraction_non_alphanumeric': 0.09789391575663027, 'fraction_numerical': 0.054602184087363496, 'mean_word_length': 2.9328220858895704, 'pattern_counts': {'":': 0, '<': 55, '<?xml version=': 0, '>': 55, 'https://': 0, 'lorem ipsum': 0, 'www.': 20, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4750725', 'n_tokens_mistral': 4596, 'n_tokens_neox': 4002, 'n_words': 1648}