<commit_before>Spider
======
revision 0.0.7c
---------------

# Overview

Spider is a specialized scraping method which returns an object with a collection of links and assets. The links can be used
to spider other documents and follow a web tree.  The assets object can be used to inspect or retrieve assets found in the HTML 
markup spidered.

[NOTE: Need examples of how this might work]<commit_msg>Added simple example of a one time spider of nodejs.org for version 0.0.8 of extractor.<commit_after>Spider
======
revision 0.0.8
--------------

# Overview

Spider is a specialized scraping method which returns an object with a collection of links and assets. The links can be used
to spider other documents and follow a web tree.  The assets object can be used to inspect or retrieve assets found in the HTML 
markup spidered.

# Example

```javascript
  var extractor = require('extractor'), util = require('util');
	
	extractor.Spider("http://nodejs.org", { response: true }, function (err, data, env) {
		if (err) {
			console.error('ERROR: ' + err);
		}
		if (data) {
			console.log('data: ' + JSON.stringify(data));
		}
		if (env) {
			console.log('http return status code: ' + env.response.statusCode);
		}
	});
```
{'fraction_non_alphanumeric': 0.12, 'fraction_numerical': 0.007346938775510204, 'mean_word_length': 3.5239852398523985, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3571240', 'n_tokens_mistral': 376, 'n_tokens_neox': 344, 'n_words': 154}