<commit_before>EclairJS Nashorn
===================
The **EclairJS Nashorn** API exposes the [Spark](http://spark.apache.org/) programming model to JavaScript.  **EclairJS Nashorn** is built on top of [Spark's Java API](http://spark.apache.org/docs/latest/api/java/index.html). For a NodeJS implementation of the Spark programming model visit the [eclairjs-node](https://github.com/EclairJS/eclairjs-node) project.

## Build from source
**Prerequisites**

 - [Java 8 SE](http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html)
 - [Spark](http://spark.apache.org/downloads.html) (V1.6.0 built for Hadoop 2.6.0 and later)
 - [git](http://git-scm.com/)
 - [Maven](https://maven.apache.org/)

```bash
git clone <EMAIL>:cfa/eclair-nashorn.git
mvn package
export SPARK_HOME=<location of Spark binary distribution>
```

## Usage
```bash
bin/eclairjs.sh examples/word_count.js
```

or
```bash
bin/eclairjs.sh
eclairJS>var list = sc.parallelize([1,10,20,30,40]);
list.count();

```

## Examples
```javascript

    var file = "src/test/resources/dream.txt"; // Should be some file on your system
    var conf = new SparkConf().setAppName("JavaScript word count")
                          .setMaster("local[*]");
    var sparkContext = new SparkContext(conf);
    var rdd = sparkContext.textFile(file).cache();
    var rdd2 = rdd.flatMap(function(sentence) {
        return sentence.split(" ");
    });
    var rdd3 = rdd2.filter(function(word) {
        return word.trim().length > 0;
    });
    var rdd4 = rdd3.mapToPair(function(word) {
        return [word, 1];
    });
    var rdd5 = rdd4.reduceByKey(function(a, b) {
        return a + b;
    });
    var rdd6 = rdd5.mapToPair(function(tuple) {
        return [tuple[1]+0.0, tuple[0]];
    })
    var rdd7 = rdd6.sortByKey(false);
    print("top 10 words = " + rdd7.take(10));

```

## Usage with Jupyter notebooks
**Prerequisites**

- [Jupyter](http://jupyter.org/)
- [Apache Toree](https://github.com/apache/incubator-toree) Toree needs to be built for Apache Spark 1.6.0, set the environment variable ````EXPORT SPARK_VERSION=1.6.0```` before building Toree with ````make dist```` and then publish to local maven repository with ````sbt publishM2````

Instructions:

1. ```mvn package -Pnotebook```

2. Edit kernel.json and update the following:
 ```bash
 <path to incubator-toree distribution>/bin/spark-kernel
 "SPARK_OPTS": --jars file:<path to nashorn jar>
 "SPARK_HOME": <path to spark 1.6.0 distribution>
 ```

3. Copy kernel.json to ```~/.ipython/kernels/eclair/kernel.json```
 * Gateway 4.0.0 and higher uses ```~/Library/Jupyter/kernels/eclair```

4. Create a directory for your notebook ```mkdir ~/jsNotebook```

5. Change to that directory ```cd ~/jsNotebook```<commit_msg>#30 update read<commit_after>EclairJS Nashorn
===================
The **EclairJS Nashorn** API exposes the [Spark](http://spark.apache.org/) programming model to JavaScript.  **EclairJS Nashorn** is built on top of [Spark's Java API](http://spark.apache.org/docs/latest/api/java/index.html). For a NodeJS implementation of the Spark programming model visit the [eclairjs-node](https://github.com/EclairJS/eclairjs-node) project.

## Build from source
**Prerequisites**

 - [Java 8 SE](http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html)
 - [Spark](http://spark.apache.org/downloads.html) (V1.6.0 built for Hadoop 2.6.0 and later)
 - [git](http://git-scm.com/)
 - [Maven](https://maven.apache.org/)

```bash
git clone <EMAIL>:cfa/eclair-nashorn.git
mvn package
export SPARK_HOME=<location of Spark binary distribution>
```

## Usage
```bash
bin/eclairjs.sh examples/word_count.js
```

or
```bash
bin/eclairjs.sh
eclairJS>var list = sc.parallelize([1,10,20,30,40]);
list.count();

```

## Examples
```javascript

    var file = "src/test/resources/dream.txt"; // Should be some file on your system
    var conf = new SparkConf().setAppName("JavaScript word count")
                          .setMaster("local[*]");
    var sparkContext = new SparkContext(conf);
    var rdd = sparkContext.textFile(file).cache();
    var rdd2 = rdd.flatMap(function(sentence) {
        return sentence.split(" ");
    });
    var rdd3 = rdd2.filter(function(word) {
        return word.trim().length > 0;
    });
    var rdd4 = rdd3.mapToPair(function(word) {
        return [word, 1];
    });
    var rdd5 = rdd4.reduceByKey(function(a, b) {
        return a + b;
    });
    var rdd6 = rdd5.mapToPair(function(tuple) {
        return [tuple[1]+0.0, tuple[0]];
    })
    var rdd7 = rdd6.sortByKey(false);
    print("top 10 words = " + rdd7.take(10));

```

## Usage with Jupyter notebooks
**Prerequisites**

- [Jupyter](http://jupyter.org/)
- [Apache Toree](https://github.com/apache/incubator-toree) Toree needs to be built for Apache Spark 1.6.0, set the environment variable ````export APACHE_SPARK_VERSION=1.6.0```` before building Toree with ````make dist```` and then publish to local maven repository with ````sbt publishM2````

Instructions:

1. ```mvn package -Pnotebook```

2. Edit kernel.json and update the following:
 ```bash
 <path to incubator-toree distribution>/bin/spark-kernel
 "SPARK_OPTS": --jars file:<path to nashorn jar>
 "SPARK_HOME": <path to spark 1.6.0 distribution>
 ```

3. Copy kernel.json to ```~/.ipython/kernels/eclair/kernel.json```
 * Gateway 4.0.0 and higher uses ```~/Library/Jupyter/kernels/eclair```

4. Create a directory for your notebook ```mkdir ~/jsNotebook```

5. Change to that directory ```cd ~/jsNotebook```
{'fraction_non_alphanumeric': 0.16820234869015357, 'fraction_numerical': 0.023486901535682024, 'mean_word_length': 3.8013876843018215, 'pattern_counts': {'":': 4, '<': 13, '<?xml version=': 0, '>': 17, 'https://': 6, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6377152', 'n_tokens_mistral': 2046, 'n_tokens_neox': 1895, 'n_words': 469}