<commit_before># Spree
[![Gitter Chat](http://img.shields.io/badge/chat-online-brightgreen.svg)](https://gitter.im/hammerlab/spree?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)

Spree is a live-updating web UI for Spark built with [Meteor][] and [React][]. 

![Screencast of a Spark job running and UI updating along with it](http://f.cl.ly/items/3r0C1J1Z1v472s1u1F14/Screen%20Recording%202015-07-06%20at%2001.31%20PM.gif)

*Left: Spree pages showing all jobs and stages, updating in real-time; right: a `spark-shell` running a simple job; see [the Screencast gallery in this repo](https://github.com/hammerlab/spree/blob/master/screencasts.md) for more examples.*

## Features!
Spree is a complete rewrite of [Spark's web UI](https://spark.apache.org/docs/1.4.0/spark-standalone.html#monitoring-and-logging), providing several notable benefits…

### Real-time Updating
All data on all pages updates in real-time, thanks to [Meteor][] magic.

### Persistence, Scalability
Spree offers a unified interface to past- and currently-running Spark applications, combining functionality that is currently spread across Spark's web UI and "history server".

It persists all information about Spark applications to MongoDB, allowing for archival storage that is easily query-able and solves various Spark-history-server issues, e.g. slow load-times, caching problems, etc.

Pagination and sorting are delegated to Mongo for graceful handling of arbitrarily large stages, RDDs, etc., which makes for a cleaner scalability story than Spark's current usage of textual event-log files and in-memory maps on the driver as ad-hoc databases.

### Usability
Spree includes several usability improvements, including:

#### Toggle-able Columns
All tables allow easy customization of displayed columns:

![](https://camo.githubusercontent.com/f8f3250e144bd8f15a23e40fc02619664a5023e1/687474703a2f2f662e636c2e6c792f6974656d732f3034303033693035316432363278334f324f32452f53637265656e2532305265636f7264696e67253230323031352d30372d3036253230617425323030322e3332253230504d2e676966)

#### Collapsible Tables
Additionally, whole tables can be collapsed/uncollapsed for easy access to content that would otherwise be "below the fold":

![](http://f.cl.ly/items/0t3J3n2w3a07181F0u36/Screen%20Recording%202015-07-17%20at%2004.55%20PM.gif)

#### Persistent Preferences/State
Finally, all client-side state is stored in cookies for persistence across refreshes / sessions, including:
* sort-column and direction, 
* table collapsed/uncollapsed status, 
* table columns' shown/hidden status,
* pages' displaying one table with "all" records vs. separate tables for "running", "succeeded", "failed" records, etc.

### Extensibility, Modularity
Spree is easy to fork/customize without worrying about changing everyones' Spark UI experience, managing custom Spark builds with bespoke UI changes, etc.

It also includes two useful standalone modules for exporting/persisting data from Spark applications:

* The [`json-relay`][] module broadcasts all Spark events over a network socket.
* The [`slim`][] module aggregates stats about running Spark jobs and persists them to indexed Mongo collections.

These offer potentially-useful alternatives to Spark's [`EventLoggingListener`][] and event-log files, respectively (Spark's extant tools for exporting and persisting historical data about past and current Spark applications).

## Usage
Spree has three components, each in its own subdirectory:

* [`ui`][]: a web-app that displays the contents of a Mongo database populated with information about running Spark applications.
* [`slim`][]: a Node server that receives events about running Spark applications, aggregates statistics about them, and writes them to Mongo for Spree's `ui` above to read/display.
* [`json-relay`][]: a [`SparkListener`][] that serializes [`SparkListenerEvent`s](https://github.com/apache/spark/blob/658814c898bec04c31a8e57f8da0103497aac6ec/core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala#L32-L128) to JSON and sends them to a listening [`slim`][] process.

The latter two are linked in this repo as `git` submodules, so you'll want to have cloned with `git clone --recursive` (or run `git submodule update`) in order for them to be present.

Following are instructions for configuring/running them:

### Start Spree
First, run a Spree app using [Meteor][]:

```
git clone --recursive git@github.com:hammerlab/spree.git
cd spree/ui   # the Spree Meteor app lives in ui/ in this repo.
meteor        # run it
```

You can now see your (presumably empty) Spree dashboard at [http://localhost:3000](http://localhost:3000):

![](http://f.cl.ly/items/243L322B1v0G082j3X0O/Screen%20Shot%202015-07-20%20at%2012.20.38%20PM.png)

If you don't have `meteor` installed, see "Installing Meteor" below.

### Start `slim`
Next, install and run `slim`:

```
npm install -g slim.js
slim<commit_msg>Using https instead of git@ in instructions.<commit_after># Spree
[![Gitter Chat](http://img.shields.io/badge/chat-online-brightgreen.svg)](https://gitter.im/hammerlab/spree?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)

Spree is a live-updating web UI for Spark built with [Meteor][] and [React][]. 

![Screencast of a Spark job running and UI updating along with it](http://f.cl.ly/items/3r0C1J1Z1v472s1u1F14/Screen%20Recording%202015-07-06%20at%2001.31%20PM.gif)

*Left: Spree pages showing all jobs and stages, updating in real-time; right: a `spark-shell` running a simple job; see [the Screencast gallery in this repo](https://github.com/hammerlab/spree/blob/master/screencasts.md) for more examples.*

## Features!
Spree is a complete rewrite of [Spark's web UI](https://spark.apache.org/docs/1.4.0/spark-standalone.html#monitoring-and-logging), providing several notable benefits…

### Real-time Updating
All data on all pages updates in real-time, thanks to [Meteor][] magic.

### Persistence, Scalability
Spree offers a unified interface to past- and currently-running Spark applications, combining functionality that is currently spread across Spark's web UI and "history server".

It persists all information about Spark applications to MongoDB, allowing for archival storage that is easily query-able and solves various Spark-history-server issues, e.g. slow load-times, caching problems, etc.

Pagination and sorting are delegated to Mongo for graceful handling of arbitrarily large stages, RDDs, etc., which makes for a cleaner scalability story than Spark's current usage of textual event-log files and in-memory maps on the driver as ad-hoc databases.

### Usability
Spree includes several usability improvements, including:

#### Toggle-able Columns
All tables allow easy customization of displayed columns:

![](https://camo.githubusercontent.com/f8f3250e144bd8f15a23e40fc02619664a5023e1/687474703a2f2f662e636c2e6c792f6974656d732f3034303033693035316432363278334f324f32452f53637265656e2532305265636f7264696e67253230323031352d30372d3036253230617425323030322e3332253230504d2e676966)

#### Collapsible Tables
Additionally, whole tables can be collapsed/uncollapsed for easy access to content that would otherwise be "below the fold":

![](http://f.cl.ly/items/0t3J3n2w3a07181F0u36/Screen%20Recording%202015-07-17%20at%2004.55%20PM.gif)

#### Persistent Preferences/State
Finally, all client-side state is stored in cookies for persistence across refreshes / sessions, including:
* sort-column and direction, 
* table collapsed/uncollapsed status, 
* table columns' shown/hidden status,
* pages' displaying one table with "all" records vs. separate tables for "running", "succeeded", "failed" records, etc.

### Extensibility, Modularity
Spree is easy to fork/customize without worrying about changing everyones' Spark UI experience, managing custom Spark builds with bespoke UI changes, etc.

It also includes two useful standalone modules for exporting/persisting data from Spark applications:

* The [`json-relay`][] module broadcasts all Spark events over a network socket.
* The [`slim`][] module aggregates stats about running Spark jobs and persists them to indexed Mongo collections.

These offer potentially-useful alternatives to Spark's [`EventLoggingListener`][] and event-log files, respectively (Spark's extant tools for exporting and persisting historical data about past and current Spark applications).

## Usage
Spree has three components, each in its own subdirectory:

* [`ui`][]: a web-app that displays the contents of a Mongo database populated with information about running Spark applications.
* [`slim`][]: a Node server that receives events about running Spark applications, aggregates statistics about them, and writes them to Mongo for Spree's `ui` above to read/display.
* [`json-relay`][]: a [`SparkListener`][] that serializes [`SparkListenerEvent`s](https://github.com/apache/spark/blob/658814c898bec04c31a8e57f8da0103497aac6ec/core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala#L32-L128) to JSON and sends them to a listening [`slim`][] process.

The latter two are linked in this repo as `git` submodules, so you'll want to have cloned with `git clone --recursive` (or run `git submodule update`) in order for them to be present.

Following are instructions for configuring/running them:

### Start Spree
First, run a Spree app using [Meteor][]:

```
git clone --recursive https://github.com/hammerlab/spree.git
cd spree/ui   # the Spree Meteor app lives in ui/ in this repo.
meteor        # run it
```

You can now see your (presumably empty) Spree dashboard at [http://localhost:3000](http://localhost:3000):

![](http://f.cl.ly/items/243L322B1v0G082j3X0O/Screen%20Shot%202015-07-20%20at%2012.20.38%20PM.png)

If you don't have `meteor` installed, see "Installing Meteor" below.

### Start `slim`
Next, install and run `slim`:

```
npm install -g slim.js
slim
{'fraction_non_alphanumeric': 0.096, 'fraction_numerical': 0.06967088607594937, 'mean_word_length': 5.544731610337972, 'pattern_counts': {'":': 2, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 11, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1102319', 'n_tokens_mistral': 3506, 'n_tokens_neox': 2937, 'n_words': 1047}