<commit_before>[![](https://images.microbadger.com/badges/image/kz85/spark-docker.svg)](https://microbadger.com/images/kz85/spark-docker "Get your own image badge on microbadger.com") [![](https://images.microbadger.com/badges/version/kz85/spark-docker.svg)](https://microbadger.com/images/kz85/spark-docker "Get your own version badge on microbadger.com") [![Build Status](https://travis-ci.org/kzabashta/spark-docker.svg?branch=master)](https://travis-ci.org/kzabashta/spark-docker)


# spark-docker
This Docker image helps to run the Spark in a cluster mode with a master and variable slave (worker) nodes.

## Installation
0. Setup Docker and docker-compose first
1. Build the image using included Dockerfile ```docker-compose build```
2. Spin up a Spark cluster with 1 master and 2 slaves (as an example) ```docker-compose up --scale master=1 --scale slave=2```
3. Verify that the cluster is running by going to http://<localhost>:8080. *Note: if you are running Docker on OS X or Windows, replace localhost with the docker host VM IP address. You can get the IP address by running* ```docker-machine ip```.
4. Destroy the cluster ```docker-compose down```

## Test

```
import pyspark
conf = pyspark.SparkConf()

conf.setMaster("spark://<docker machine ip>:7077")
conf.setAppName('test')

sc = pyspark.SparkContext(conf=conf)

rdd = sc.parallelize(range(100))
```

## TODO
Need to add support for the following components:
1. Scala
2. PySpark
3. HDFS
4. Zeppelin
5. Jupyter
6. Instructions on setting up in Azure/AWS with Docker Swarm
7. Run containers in some kind of process manager<commit_msg>Adding verification step for Jupyter<commit_after>[![](https://images.microbadger.com/badges/image/kz85/spark-docker.svg)](https://microbadger.com/images/kz85/spark-docker "Get your own image badge on microbadger.com") [![](https://images.microbadger.com/badges/version/kz85/spark-docker.svg)](https://microbadger.com/images/kz85/spark-docker "Get your own version badge on microbadger.com") [![Build Status](https://travis-ci.org/kzabashta/spark-docker.svg?branch=master)](https://travis-ci.org/kzabashta/spark-docker)


# spark-docker
This Docker image helps to run the Spark in a cluster mode with a master and variable slave (worker) nodes.

## Installation
0. Setup Docker and docker-compose first
1. Build the image using included Dockerfile ```docker-compose build```
2. Spin up a Spark cluster with 1 master and 2 slaves (as an example) ```docker-compose up --scale master=1 --scale slave=2```
3. Verify that the cluster is running by going to http://docker-machine-ip:8080. *Note: if you are running Docker on OS X or Windows, replace localhost with the docker host VM IP address. You can get the IP address by running* ```docker-machine ip```.
4. Verify that Jupyter notebook server is running by going to http://docker-machine-ip:8888
5. Destroy the cluster ```docker-compose down```

## Test

```
import pyspark
conf = pyspark.SparkConf()

conf.setMaster("spark://<docker machine ip>:7077")
conf.setAppName('test')

sc = pyspark.SparkContext(conf=conf)

rdd = sc.parallelize(range(100))
```

## TODO
Need to add support for the following components:
1. Scala
2. PySpark
3. HDFS
4. Zeppelin
5. Jupyter
6. Instructions on setting up in Azure/AWS with Docker Swarm
7. Run containers in some kind of process manager
{'fraction_non_alphanumeric': 0.12541353383458648, 'fraction_numerical': 0.022556390977443608, 'mean_word_length': 4.73448275862069, 'pattern_counts': {'":': 0, '<': 6, '<?xml version=': 0, '>': 6, 'https://': 12, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1128951', 'n_tokens_mistral': 1162, 'n_tokens_neox': 1039, 'n_words': 363}