<commit_before># tree-builder-builder
Builder of Tree Builders

## Quick Start Guide

### Installing all the bits

1.  Install git
2.  Grab the source code:
    `git clone https://github.com/dglazkov/tree-builder-builder.git`
3.  Install node.js from https://nodejs.org/download/
4.  `cd tree-builder-builder`
5.  `npm install`
6.  (optional) Install gulp globally so it'll be on your path:
    `sudo npm install -g gulp`
7.  (optional) if you want to be able to run automated perf tests or easily
    extract DOM from URLs, check out a chromium tree 
    (http://dev.chromium.org/developers/how-tos/get-the-code). Note that you
    don't need to build chromium, just check it out.

### Saving the DOM of web pages

If you have a local chromium checkout, and assuming that it's installed at /path/to/chromium:

`gulp get --url="http://example.com/my-super-exciting-DOM.html" --chromium=/path/to/chromium --saveBrowser=system

This will save the DOM content of the provided url in a result.json file.

If you don't have a local chromium checkout, you can still save pages by
navigating to them in Chrome, opening dev tools, and dumping the contents
of `save.js` into the console.

### Converting DOM to HTML or JS

To save an HTML file (result.html.html)

`gulp html --file=result.json`

To save a JS file (result.js.html)
<commit_msg>Missing backtick.<commit_after># tree-builder-builder
Builder of Tree Builders

## Quick Start Guide

### Installing all the bits

1.  Install git
2.  Grab the source code:
    `git clone https://github.com/dglazkov/tree-builder-builder.git`
3.  Install node.js from https://nodejs.org/download/
4.  `cd tree-builder-builder`
5.  `npm install`
6.  (optional) Install gulp globally so it'll be on your path:
    `sudo npm install -g gulp`
7.  (optional) if you want to be able to run automated perf tests or easily
    extract DOM from URLs, check out a chromium tree 
    (http://dev.chromium.org/developers/how-tos/get-the-code). Note that you
    don't need to build chromium, just check it out.

### Saving the DOM of web pages

If you have a local chromium checkout, and assuming that it's installed at /path/to/chromium:

`gulp get --url="http://example.com/my-super-exciting-DOM.html" --chromium=/path/to/chromium --saveBrowser=system`

This will save the DOM content of the provided url in a result.json file.

If you don't have a local chromium checkout, you can still save pages by
navigating to them in Chrome, opening dev tools, and dumping the contents
of `save.js` into the console.

### Converting DOM to HTML or JS

To save an HTML file (result.html.html)

`gulp html --file=result.json`

To save a JS file (result.js.html)

{'fraction_non_alphanumeric': 0.09801720912832024, 'fraction_numerical': 0.005237560793116348, 'mean_word_length': 3.5553662691652472, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2575954', 'n_tokens_mistral': 868, 'n_tokens_neox': 823, 'n_words': 359}