<commit_before>![Scrapie](src/main/images/sheepVeryVerySmall.png) scrAPIe
=======

A Web Scraper.  
_Not the fatal, degenerative disease that affects the nervous systems of sheep and goats._

JavaScript
----

We decided against coffee script as it didn't seem to make the scraper files any shorter, but it 
did get rid of all the brackets and that made me uneasy.  

Because the scraper files are JavaScript you have a lot of power to do lots of tasks that you
would otherwise normally have to do in an ETL type process.


The [Emitter.java](src/main/java/com/wp/scrapie/Emitter.java)  Object is the communication bridge back 
to the Java world.  This is the object that fetches the web documents.  The Emitter object is put
into the JavaScript context as a variable you can access called emitter.  Whenever you're given a 
context object, you're getting a wrapped instance of emitter with access to all the parts of the Emitter class
that you need, plus some JavaScript wrappings that make closure possible.

Other JavaScript classes and functions are available.  

- [EmitterWrapper.js](src/main/js/EmitterWrapper.js), which wraps the Emitter object to give it some more useful JavaScripty syntax.
- [FileIterator.js](src/main/js/FileIterator.js), JavaScript backing to a FileLineIterator   
- [GlobalFunctions.js](src/main/js/GlobalFunctions.js), some useful methods like print()
- [UrlIterator.js](src/main/UrlIterator.js), which you can implement to generate URLs.


Example Usage
-----

```Java
1   var urlIterator = new UrlIterator(function(pIndex){
2      if (pIndex < 2) {
3		  return "http://example.com/index.html?id=" + pIndex;
4	   } else {
5 		  return null;
6	   }
7  });
8  urlIterator.forEach(function(pContext){
9    pContext.emit("title", pContext.getJqText("title"));
10   pContext.flush();
11 });
```

Lines 1 - 7, create the `UrlIterator` object.  The generator function passed in to the constructor<commit_msg>doc updates<commit_after>![Scrapie](src/main/images/sheepVeryVerySmall.png) scrAPIe
=======

A Web Scraper.  
_Not the fatal, degenerative disease that affects the nervous systems of sheep and goats._

JavaScript
----

We decided against coffee script as it didn't seem to make the scraper files any shorter, but it 
did get rid of all the brackets and that made me uneasy.  

Because the scraper files are JavaScript you have a lot of power to do lots of tasks that you
would otherwise normally have to do in an ETL type process.


The [Emitter.java](src/main/java/com/wp/scrapie/Emitter.java)  Object is the communication bridge back 
to the Java world.  This is the object that fetches the web documents.  The Emitter object is put
into the JavaScript context as a variable you can access called emitter.  Whenever you're given a 
context object, you're getting a wrapped instance of emitter with access to all the parts of the Emitter class
that you need, plus some JavaScript wrappings that make closure possible.

Other JavaScript classes and functions are available.  

- [EmitterWrapper.js](src/main/resources/EmitterWrapper.js), which wraps the Emitter object to give it some more useful JavaScripty syntax.
- [FileIterator.js](src/main/resources/FileIterator.js), JavaScript backing to a FileLineIterator   
- [GlobalFunctions.js](src/main/resources/GlobalFunctions.js), some useful methods like print()
- [UrlIterator.js](src/main/resources/UrlIterator.js), which you can implement to generate URLs.


Example Usage
-----

```Java
1   var urlIterator = new UrlIterator(function(pIndex){
2      if (pIndex < 2) {
3		  return "http://example.com/index.html?id=" + pIndex;
4	   } else {
5 		  return null;
6	   }
7  });
8  urlIterator.forEach(function(pContext){
9    pContext.emit("title", pContext.getJqText("title"));
10   pContext.flush();
11 });
```

Lines 1 - 7, create the `UrlIterator` object.  The generator function passed in to the constructor
{'fraction_non_alphanumeric': 0.08549422336328627, 'fraction_numerical': 0.008215661103979461, 'mean_word_length': 4.133069828722003, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2283940', 'n_tokens_mistral': 1138, 'n_tokens_neox': 1081, 'n_words': 478}