<commit_before># scraping-examples

Different examples of scraping techniques. Cheerio is used in all the examples to extract the desired content.

## Synopsis

Show a few examples of screen scraping.

1. Scrape static content with request.
2. 
3. Scrape dynamic content (Angular/Ember) using Nightmare/Phantom.
4. Take screen shots of page as you programmatically interact with the page.

## Installation

* npm install

#### Additional Information about packages

* https://www.npmjs.com/package/cheerio
* https://www.npmjs.com/package/request
* https://www.npmjs.com/package/q
* https://www.npmjs.com/package/colors
* https://www.npmjs.com/package/sleep
* https://www.npmjs.com/package/nightmare / http://www.nightmarejs.org/
* https://www.npmjs.com/package/prompt
* https://www.npmjs.com/package/phantom

## Debugging

* node-debug \--nodejs \--harmony index.js

## Running
<commit_msg>Updated descriptions for tests 3 & 4.<commit_after># scraping-examples

Different examples of scraping techniques. Cheerio is used in all the examples to extract the desired content.

## Synopsis

Show a few examples of screen scraping.

1. Scrape static content with request.
2. 
3. Scrape dynamic content (Angular/Ember) using Nightmare.
4. Take screen shots of page as you programmatically interact with the page using Nightmare.

## Installation

* npm install

#### Additional Information about packages

* https://www.npmjs.com/package/cheerio
* https://www.npmjs.com/package/request
* https://www.npmjs.com/package/q
* https://www.npmjs.com/package/colors
* https://www.npmjs.com/package/sleep
* https://www.npmjs.com/package/nightmare / http://www.nightmarejs.org/
* https://www.npmjs.com/package/prompt
* https://www.npmjs.com/package/phantom

## Debugging

* node-debug \--nodejs \--harmony index.js

## Running

{'fraction_non_alphanumeric': 0.12417218543046357, 'fraction_numerical': 0.005518763796909493, 'mean_word_length': 3.9400544959128063, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 16, 'lorem ipsum': 0, 'www.': 18, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5022218', 'n_tokens_mistral': 596, 'n_tokens_neox': 538, 'n_words': 159}