<commit_before>Spark Notebook
==============
*Fork of the amazing [scala-notebook](https://github.com/Bridgewater/scala-notebook), yet focusing on Massive Dataset Analysis using [Apache Spark](http://spark.apache.org).*

<!-- MarkdownTOC depth=4 autolink=true bracket=round -->

- [Description](#description)
- [Launch](#launch)
  - [Very quickstart](#very-quickstart)
  - [Longer story](#longer-story)
    - [Procedure](#procedure)
- [Use](#use)
- [Features](#features)
  - [Use/Reconfigure Spark](#usereconfigure-spark)
  - [Use the `form`](#use-the-form)
  - [The `reset` function](#the-reset-function)
  - [Keep an eye on your tasks](#keep-an-eye-on-your-tasks)
  - [Using (Spark)SQL](#using-sparksql)
    - [Static SQL](#static-sql)
    - [Dynamic SQL](#dynamic-sql)
    - [Show case](#show-case)
  - [Interacting with JavaScript](#interacting-with-javascript)
  - [Plotting with D3](#plotting-with-d3)
  - [Timeseries with  Rickshaw](#timeseries-with--rickshaw)
  - [Dynamic update of data and plot using Scala's `Future`](#dynamic-update-of-data-and-plot-using-scalas-future)
  - [Update _Notebook_ `ClassPath`](#update-_notebook_-classpath)
  - [Update __Spark__ dependencies (`spark.jars`)](#update-__spark__-dependencies-sparkjars)

<!-- /MarkdownTOC -->

Description
-----------
The main intent of this tool is to create [reproducible analysis](http://simplystatistics.org/2014/06/06/the-real-reason-reproducible-research-is-important/) using Scala, Apache Spark and more.

This is achieved through an interactive web-based editor that can combine Scala code, SQL queries, Markup or even JavaScript in a collaborative manner.

The usage of Spark comes out of the box, and is simply enabled by the implicit variable named `sparkContext`.

Launch
------
### Very quickstart

Long story short, there is a small script that can help you setup and launch it without any other requirements than a _java environment_.


```bash
curl https://raw.githubusercontent.com/andypetrella/spark-notebook/master/run.sh | bash -s dev
```

### Longer story
The spark notebook requires a [Java(TM)](http://en.wikipedia.org/wiki/Java_(programming_language)) environment (aka JVM) as runtime and [SBT](www.scala-sbt.org) as build tool.

You will also need a working [GIT](http://git-scm.com/) installation to download the code and build it.

#### Procedure
##### Download the code
```
git clone https://github.com/andypetrella/spark-notebook.git
cd spark-notebook
```
##### Launch the server
Enter the `sbt console` by running `sbt` within the `spark-notebook`:
```
spark-notebook$ sbt
[warn] Multiple resolvers having different access mechanism configured with same name 'sbt-plugin-releases'. To avoid conflict, Remove duplicate project resolvers (`resolvers`) or rename publishing resolver (`publishTo`).
[info] Updating {file:/home/noootsab/src/noootsab/spark-notebook/project/}spark-notebook-build...
...
...
...
```

Then you can head to the `server` project and run it. You have several options available:
 * `--disable_security`: this will disable the Akka secure cookie (helpful in local env)
 * `--no_browser`: this prevents the project to open a page in your browser everytime the server is started

```
> project server
> run --disable_security
```

Use
---
When the server has been started, you can head to the page `http://localhost:8899` and you'll see something similar to:
![Notebook list](https://raw.github.com/andypetrella/spark-notebook/master/images/list.png)

From there you can either:
 * create a new notebook
 * launch an existing notebook

In both case, the `scala-notebook` will open a new tab with your notebook in it, loaded as a web page.

> Note: a notebook is a JSON file containing the layout and analysis blocks, and it's located
> within the project folder (with the `snb` extension).
> Hence, they can be shared and we can track their history in an SVM like `GIT`.

Features
--------
### Use/Reconfigure Spark
Since this project aims directly the usage of Spark, a [SparkContext](https://github.com/apache/spark/blob/master/core%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fspark%2FSparkContext.scala) is added to the environment and can directly be used without additional effort.<commit_msg>Document hostname property in .conf<commit_after>Spark Notebook
==============
*Fork of the amazing [scala-notebook](https://github.com/Bridgewater/scala-notebook), yet focusing on Massive Dataset Analysis using [Apache Spark](http://spark.apache.org).*

<!-- MarkdownTOC depth=4 autolink=true bracket=round -->

- [Description](#description)
- [Launch](#launch)
  - [Very quickstart](#very-quickstart)
  - [Longer story](#longer-story)
    - [Procedure](#procedure)
- [Use](#use)
- [Features](#features)
  - [Use/Reconfigure Spark](#usereconfigure-spark)
  - [Use the `form`](#use-the-form)
  - [The `reset` function](#the-reset-function)
  - [Keep an eye on your tasks](#keep-an-eye-on-your-tasks)
  - [Using (Spark)SQL](#using-sparksql)
    - [Static SQL](#static-sql)
    - [Dynamic SQL](#dynamic-sql)
    - [Show case](#show-case)
  - [Interacting with JavaScript](#interacting-with-javascript)
  - [Plotting with D3](#plotting-with-d3)
  - [Timeseries with  Rickshaw](#timeseries-with--rickshaw)
  - [Dynamic update of data and plot using Scala's `Future`](#dynamic-update-of-data-and-plot-using-scalas-future)
  - [Update _Notebook_ `ClassPath`](#update-_notebook_-classpath)
  - [Update __Spark__ dependencies (`spark.jars`)](#update-__spark__-dependencies-sparkjars)

<!-- /MarkdownTOC -->

Description
-----------
The main intent of this tool is to create [reproducible analysis](http://simplystatistics.org/2014/06/06/the-real-reason-reproducible-research-is-important/) using Scala, Apache Spark and more.

This is achieved through an interactive web-based editor that can combine Scala code, SQL queries, Markup or even JavaScript in a collaborative manner.

The usage of Spark comes out of the box, and is simply enabled by the implicit variable named `sparkContext`.

Launch
------
### Very quickstart

Long story short, there is a small script that can help you setup and launch it without any other requirements than a _java environment_.


```bash
curl https://raw.githubusercontent.com/andypetrella/spark-notebook/master/run.sh | bash -s dev
```

### Longer story
The spark notebook requires a [Java(TM)](http://en.wikipedia.org/wiki/Java_(programming_language)) environment (aka JVM) as runtime and [SBT](www.scala-sbt.org) as build tool.

You will also need a working [GIT](http://git-scm.com/) installation to download the code and build it.

#### Procedure
##### Download the code
```
git clone https://github.com/andypetrella/spark-notebook.git
cd spark-notebook
```
##### Launch the server
Enter the `sbt console` by running `sbt` within the `spark-notebook`:
```
spark-notebook$ sbt
[warn] Multiple resolvers having different access mechanism configured with same name 'sbt-plugin-releases'. To avoid conflict, Remove duplicate project resolvers (`resolvers`) or rename publishing resolver (`publishTo`).
[info] Updating {file:/home/noootsab/src/noootsab/spark-notebook/project/}spark-notebook-build...
...
...
...
```

Then you can head to the `server` project and run it. You have several options available:
 * `--disable_security`: this will disable the Akka secure cookie (helpful in local env)
 * `--no_browser`: this prevents the project to open a page in your browser everytime the server is started

```
> project server
> run --disable_security
```

##### Make the server accessible from the network

Edit the `.conf` file in the project root and change the hostname to the hostname from `0.0.0.0` to the name that is reachable from the network.

Use
---
When the server has been started, you can head to the page `http://localhost:8899` and you'll see something similar to:
![Notebook list](https://raw.github.com/andypetrella/spark-notebook/master/images/list.png)

From there you can either:
 * create a new notebook
 * launch an existing notebook

In both case, the `scala-notebook` will open a new tab with your notebook in it, loaded as a web page.

> Note: a notebook is a JSON file containing the layout and analysis blocks, and it's located
> within the project folder (with the `snb` extension).
> Hence, they can be shared and we can track their history in an SVM like `GIT`.

Features
--------
### Use/Reconfigure Spark
Since this project aims directly the usage of Spark, a [SparkContext](https://github.com/apache/spark/blob/master/core%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fspark%2FSparkContext.scala) is added to the environment and can directly be used without additional effort.
{'fraction_non_alphanumeric': 0.12970033553164412, 'fraction_numerical': 0.0055536272127733426, 'mean_word_length': 4.426239799121155, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 17, 'https://': 10, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3447176', 'n_tokens_mistral': 2730, 'n_tokens_neox': 2506, 'n_words': 888}