<commit_before>EclairJS 
===================
**EclairJS** is a JavaScript client for Apache Spark.  It is a work-in-progress so it is a good idea to frequently check the [Guide to Using the JavaScript Spark API](https://github.rtp.raleigh.ibm.com/EclairJS/eclairjs-node/wikis/Guide-to-using-the-JavaScript-Spark-API) for a list of what is currently implemented.

Overview
===================
**EcliarJS** is made up of two components.  The **EcliarJS Node** API and the **EcliarJS Nashorn** API.

## EcliarJS Node
The purpose of **EclairJS Node** is to enable JacaScript/NodeJS developers to program against [Apache Spark](http://spark.apache.org/).  The **EclairJS Node** component provides the NodeJS developer direct JavaScript equivalents of the [Apache Spark API](http://spark.apache.org/docs/latest/api/java/index.html) and handles:
* resolving of variables
* stringifying of code

### When to Use EcliarJS Node
The **EclairJS Node** API is the intended entry point for JavaScript/NodeJS programmers into the [Apache Spark](http://spark.apache.org/) world.  It should be used in a NodeJS application to get access to the vast sources of both batch and streaming data [Apache Spark](http://spark.apache.org/) can provide as well as allow transformations and actions to be taken upon the data.

**EclairJS Node is dependent on the EclairJS Nahorn component.**

## EcliarJS Nahorn
The purpose of **EcliarJS Nashorn** is to expose the [Apache Spark](http://spark.apache.org/) programming model to JavaScript.  **EcliarJS Nashorn** is built on top of [Spark's Java API](http://spark.apache.org/docs/latest/api/java/index.html) and runs in the JVM.  The **EcliarJS Nashorn** component servers as a Java wrapper and handles:
* execution of lambda functions
* type conversions on Spark Worker nodes

### When to Use EcliarJS Node
The **EclairJS Nashorn** API can be used with either Jupyter/IPython Notebooks or the REPL that comes with the _Nashorn_ project to provide JavaScript as a language to the data scientist or developer that may want to explore their data and develop functions and algorithms for transforming and exposing their data before handing off it to the client-side developer.

**EclairJS Nashorn is independent on the EclairJS Node component** and can actually be used for exploration of data via [Apache Spark](http://spark.apache.org/) as mentioned above.

TryIt
===================
The easiest and fastest way to try things out is with the Dockerfile supplied for this project.  

**Please see [Using-the-Docker-Container](https://github.rtp.raleigh.ibm.com/EclairJS/eclairjs-node/wikis/Using-the-Docker-Container).**

Develop
===================
If you are interested in becoming a collaborator and help in development of **EclairJS** please email <EMAIL> to be granted collaborator access.

**Please see [Development for EclairJS](https://github.rtp.raleigh.ibm.com/EclairJS/eclairjs-node/wikis/Development-for-EclairJS) for more information.**

Build & Package
===================
If you would like to take the manual route in lieu of using the Dockerfile you can build from source and setup your local environment.

**Please see [Build and Package](https://github.rtp.raleigh.ibm.com/EclairJS/eclairjs-node/wikis/Build-and-Package) for more information.**

Version
===================
Our goal is to keep master up to date with the latest version of Spark. When new versions of Spark require code changes, we create a separate branch. The table below shows what is available now.

| Branch | EclairJS Version | Apache Spark Version |
| -------- | -------- | -------- |
| master   | 0.1   | 1.5.1 |

Please note that for the most part, as new APIs are implemented for **EclairJS** they will be added to the master branch.

Resources
===================
There is more detailed information available in our [Wiki](https://github.rtp.raleigh.ibm.com/EclairJS/eclairjs-node/wikis/home) and our [Getting Started](https://github.rtp.raleigh.ibm.com/EclairJS/eclairjs-node/wikis/Getting-Started-With-EclairJS-Node) guide.<commit_msg>Small update to main README<commit_after>EclairJS 
===================
**EclairJS** is a JavaScript client for Apache Spark.  It is a work-in-progress so it is a good idea to frequently check the [Guide to Using the JavaScript Spark API](https://github.rtp.raleigh.ibm.com/EclairJS/eclairjs-node/wikis/Guide-to-using-the-JavaScript-Spark-API) for a list of what is currently implemented.

Overview
===================
**EcliarJS** is made up of two components.  The **EcliarJS Node** API and the **EcliarJS Nashorn** API.

## EcliarJS Node
The purpose of **EclairJS Node** is to enable JacaScript/NodeJS developers to program against [Apache Spark](http://spark.apache.org/).  The **EclairJS Node** component provides the NodeJS developer direct JavaScript equivalents of the [Apache Spark API](http://spark.apache.org/docs/latest/api/java/index.html) and handles:
* resolving of variables
* stringifying of code

### When to Use EcliarJS Node
The **EclairJS Node** API is the intended entry point for JavaScript/NodeJS programmers into the [Apache Spark](http://spark.apache.org/) world.  It should be used in a NodeJS application to get access to the vast sources of both batch and streaming data [Apache Spark](http://spark.apache.org/) can provide as well as allow transformations and actions to be taken upon the data.

**EclairJS Node is dependent on the EclairJS Nahorn component.**

## EcliarJS Nahorn
The purpose of **EcliarJS Nashorn** is to expose the [Apache Spark](http://spark.apache.org/) programming model to JavaScript.  **EcliarJS Nashorn** is built on top of [Spark's Java API](http://spark.apache.org/docs/latest/api/java/index.html) and runs in the JVM.  The **EcliarJS Nashorn** component servers as a Java wrapper and handles:
* execution of lambda functions
* type conversions on Spark Worker nodes

### When to Use EcliarJS Node
The **EclairJS Nashorn** API can be used with either Jupyter/IPython Notebooks or the REPL that comes with the _Nashorn_ project to provide JavaScript as a language to the data scientist or developer that may want to explore their data and develop functions and algorithms for transforming and exposing their data before handing off it to the client-side developer.

**EclairJS Nashorn is independent on the EclairJS Node component** and can actually be used for exploration of data via [Apache Spark](http://spark.apache.org/) as mentioned above.

TryIt
===================
The easiest and fastest way to try things out is with the Dockerfile supplied for this project.  

**Please see [Using-the-Docker-Container](https://github.rtp.raleigh.ibm.com/EclairJS/eclairjs-node/wikis/Using-the-Docker-Container).**

Develop
===================
If you are interested in becoming a collaborator and help in development of **EclairJS** please email <EMAIL> to be granted collaborator access.

**Please see [Development for EclairJS](https://github.rtp.raleigh.ibm.com/EclairJS/eclairjs-node/wikis/Development-for-EclairJS) for more information.**

Build & Package
===================
If you would like to take the manual route in lieu of using the Dockerfile you can build from source and setup your local environment.

**Please see [Build and Package](https://github.rtp.raleigh.ibm.com/EclairJS/eclairjs-node/wikis/Build-and-Package) for more information.**

Version
===================
Our goal is to keep master up to date with the latest version of Spark. When new versions of Spark require code changes, we create a separate branch. The table below shows what is available now.

| Branch | EclairJS Version | Apache Spark Version |
| -------- | -------- | -------- |
| master   | 0.1   | 1.5.1 |

Please note that for the most part, as new APIs are implemented for **EclairJS** they will be added to the master branch.  ** See the [Guide to Using the JavaScript Spark API](https://github.rtp.raleigh.ibm.com/EclairJS/eclairjs-node/wikis/Guide-to-using-the-JavaScript-Spark-API) for a list of what is currently implemented.**

Resources
===================
There is more detailed information available in our [Wiki](https://github.rtp.raleigh.ibm.com/EclairJS/eclairjs-node/wikis/home) and our [Getting Started](https://github.rtp.raleigh.ibm.com/EclairJS/eclairjs-node/wikis/Getting-Started-With-EclairJS-Node) guide.
{'fraction_non_alphanumeric': 0.1219951923076923, 'fraction_numerical': 0.001201923076923077, 'mean_word_length': 4.599596231493943, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 13, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1606241', 'n_tokens_mistral': 2442, 'n_tokens_neox': 2267, 'n_words': 970}