<commit_before>Spark Notebook
==============
*Fork of the amazing [scala-notebook](https://github.com/Bridgewater/scala-notebook), yet focusing on Massive Dataset Analysis using [Apache Spark](http://spark.apache.org).*

<!-- MarkdownTOC depth=4 autolink=true bracket=round -->

- [Description](#description)
- [Launch](#launch)
  - [Using a realease](#using-a-realease)
  - [From the sources](#from-the-sources)
    - [Procedure](#procedure)
- [Use](#use)
- [Features](#features)
  - [Use/Reconfigure Spark](#usereconfigure-spark)
  - [Use the `form`](#use-the-form)
  - [The `reset` function](#the-reset-function)
  - [Keep an eye on your tasks](#keep-an-eye-on-your-tasks)
  - [Using (Spark)SQL](#using-sparksql)
    - [Static SQL](#static-sql)
    - [Dynamic SQL](#dynamic-sql)
    - [Show case](#show-case)
  - [Interacting with JavaScript](#interacting-with-javascript)
  - [Plotting with D3](#plotting-with-d3)
  - [Timeseries with  Rickshaw](#timeseries-with--rickshaw)
  - [Dynamic update of data and plot using Scala's `Future`](#dynamic-update-of-data-and-plot-using-scalas-future)
  - [Update _Notebook_ `ClassPath`](#update-_notebook_-classpath)
  - [Update __Spark__ dependencies (`spark.jars`)](#update-__spark__-dependencies-sparkjars)

<!-- /MarkdownTOC -->

Description
-----------
The main intent of this tool is to create [reproducible analysis](http://simplystatistics.org/2014/06/06/the-real-reason-reproducible-research-is-important/) using Scala, Apache Spark and more.

This is achieved through an interactive web-based editor that can combine Scala code, SQL queries, Markup or even JavaScript in a collaborative manner.

The usage of Spark comes out of the box, and is simply enabled by the implicit variable named `sparkContext`.

Launch
------
### Using a release

Long story short, you can use the latest release [here](https://github.com/andypetrella/spark-notebook/releases), download it, unzip it and launch the script in `bin`.


### From the sources
The spark notebook requires a [Java(TM)](http://en.wikipedia.org/wiki/Java_(programming_language)) environment (aka JVM) as runtime and [Play 2.2.6](https://www.playframework.com/documentation/2.2.6/Home) to build it.

Of course, you will also need a working [GIT](http://git-scm.com/) installation to download the code and build it.

#### Procedure
##### Download the code
```
git clone https://github.com/andypetrella/spark-notebook.git
cd spark-notebook
```
##### Launch the server
Enter the `play console` by running `play` within the `spark-notebook` folder:
```
[info] Loading global plugins from /home/noootsab/.sbt/0.13/plugins
[info] Loading project definition from /home/Sources/noootsab/spark-notebook/project
[info] Set current project to spark-notebook (in build file:/home/Sources/noootsab/spark-notebook/)
       _
 _ __ | | __ _ _  _
| '_ \| |/ _' | || |
|  __/|_|\____|\__ /
|_|            |__/

play 2.2.6 built with Scala 2.10.3 (running Java 1.7.0_72), http://www.playframework.com

> Type "help play" or "license" for more information.
> Type "exit" or use Ctrl+D to leave this console.<commit_msg>using a release doc updated<commit_after>Spark Notebook
==============
*Fork of the amazing [scala-notebook](https://github.com/Bridgewater/scala-notebook), yet focusing on Massive Dataset Analysis using [Apache Spark](http://spark.apache.org).*

<!-- MarkdownTOC depth=4 autolink=true bracket=round -->

- [Description](#description)
- [Launch](#launch)
  - [Using a release](#using-a-release)
    - [Using a release (zip) on S3](#using-a-release-zip-on-s3)
    - [Using a released Docker image](#using-a-released-docker-image)
    - [Soon: Using a released debian package](#soon-using-a-released-debian-package)
  - [From the sources](#from-the-sources)
    - [Procedure](#procedure)
- [Use](#use)
- [Features](#features)
  - [Use/Reconfigure Spark](#usereconfigure-spark)
  - [Use the `form`](#use-the-form)
  - [The `reset` function](#the-reset-function)
  - [Keep an eye on your tasks](#keep-an-eye-on-your-tasks)
  - [Using (Spark)SQL](#using-sparksql)
    - [Static SQL](#static-sql)
    - [Dynamic SQL](#dynamic-sql)
    - [Show case](#show-case)
  - [Interacting with JavaScript](#interacting-with-javascript)
  - [Plotting with D3](#plotting-with-d3)
  - [Timeseries with  Rickshaw](#timeseries-with--rickshaw)
  - [Dynamic update of data and plot using Scala's `Future`](#dynamic-update-of-data-and-plot-using-scalas-future)
  - [Update _Notebook_ `ClassPath`](#update-_notebook_-classpath)
  - [Update __Spark__ dependencies (`spark.jars`)](#update-__spark__-dependencies-sparkjars)

<!-- /MarkdownTOC -->

Description
-----------
The main intent of this tool is to create [reproducible analysis](http://simplystatistics.org/2014/06/06/the-real-reason-reproducible-research-is-important/) using Scala, Apache Spark and more.

This is achieved through an interactive web-based editor that can combine Scala code, SQL queries, Markup or even JavaScript in a collaborative manner.

The usage of Spark comes out of the box, and is simply enabled by the implicit variable named `sparkContext`.

Launch
------
### Using a release

Long story short, there are several ways to use it quickly.

#### Using a release (zip) on S3
```
wget s3: https://s3.eu-central-1.amazonaws.com/spark-notebook/spark-notebook-0.1.3_1.1.0_1.0.4.zip
unzip spark-notebook-0.1.3_1.1.0_1.0.4.zip
cd spark-notebook-0.1.3_1.1.0_1.0.4
./bin/spark-notebook #with additional args with necessary like  `-mem 5G` for instance
```

#### Using a released Docker image
```
docker pull andypetrella/spark-notebook:0.1.3_1.1.0_1.0.4
docker run -p 9000:9000 andypetrella/spark-notebook:0.1.3_1.1.0_1.0.4
```

#### Soon: Using a released debian package
Sadly, the debian package aren't released yet (but everything is prepared on master ;-)).


### From the sources
The spark notebook requires a [Java(TM)](http://en.wikipedia.org/wiki/Java_(programming_language)) environment (aka JVM) as runtime and [Play 2.2.6](https://www.playframework.com/documentation/2.2.6/Home) to build it.

Of course, you will also need a working [GIT](http://git-scm.com/) installation to download the code and build it.

#### Procedure
##### Download the code
```
git clone https://github.com/andypetrella/spark-notebook.git
cd spark-notebook
```
##### Launch the server
Enter the `play console` by running `play` within the `spark-notebook` folder:
```
[info] Loading global plugins from /home/noootsab/.sbt/0.13/plugins
[info] Loading project definition from /home/Sources/noootsab/spark-notebook/project
[info] Set current project to spark-notebook (in build file:/home/Sources/noootsab/spark-notebook/)
       _
 _ __ | | __ _ _  _
| '_ \| |/ _' | || |
|  __/|_|\____|\__ /
|_|            |__/

play 2.2.6 built with Scala 2.10.3 (running Java 1.7.0_72), http://www.playframework.com

> Type "help play" or "license" for more information.
> Type "exit" or use Ctrl+D to leave this console.
{'fraction_non_alphanumeric': 0.1543778801843318, 'fraction_numerical': 0.017857142857142856, 'mean_word_length': 4.047238372093023, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 11, 'https://': 8, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3536509', 'n_tokens_mistral': 2428, 'n_tokens_neox': 2216, 'n_words': 609}