<commit_before># docker-nats-connector-spark
A collection of Docker Images to illustrate the use of the [nats-connector-spark](https://github.com/Logimethods/nats-connector-spark), [nats-connector-spark-scala](https://github.com/Logimethods/nats-connector-spark-scala) & [nats-connector-gatling](https://github.com/Logimethods/nats-connector-gatling) librairies.

[![MIT License](https://img.shields.io/npm/l/express.svg)](http://opensource.org/licenses/MIT)
[![Issues](https://img.shields.io/github/issues/Logimethods/docker-nats-connector-spark.svg)](https://github.com/Logimethods/docker-nats-connector-spark/issues)
[![wercker status](https://app.wercker.com/status/ae3458297bed4e5921beaba70acf173b/s/master "wercker status")](https://app.wercker.com/project/bykey/ae3458297bed4e5921beaba70acf173b)

[![Dockerhub](http://dockeri.co/image/logimethods/nats-connector-spark)](https://hub.docker.com/r/logimethods/nats-connector-spark/)

## Usage
    cd compose
    docker-compose up

## What it does...
1) Docker-compose will instantiate and run a collection of Docker Containers defined by [docker-compose.yml](https://github.com/Logimethods/docker-nats-connector-spark/blob/master/compose/docker-compose.yml)

2) Gatling does emit every 15 sec a stream of values from 100 to 150 into NATS. See [NatsInjection.scala](https://github.com/Logimethods/docker-nats-connector-spark/blob/master/inject/user-files/simulations/nats/NatsInjection.scala).

3) Those values are transmitted from NATS to Spark Streaming (running on a Spark cluster), then the maximum value of each stream is computed by Spark & reemited into NATS. See [SparkProcessor.scala](https://github.com/Logimethods/docker-nats-connector-spark/blob/master/app/src/main/scala/com/logimethods/nats/connector/spark/app/SparkProcessor.scala)

4) Finally, those values are monitored and printed into the console. See [NatsOutputMonitor.scala](https://github.com/Logimethods/docker-nats-connector-spark/blob/master/monitor/src/main/scala/com/logimethods/nats/connector/spark/monitor/NatsOutputMonitor.scala)

![NATS-Docker_Containers.png](NATS-Docker_Containers.png "Data flow")

## Build

Those Docker Images are pushed to [dockerhub:logimethods/nats-connector-spark](https://hub.docker.com/r/logimethods/nats-connector-spark/) and build there.

![build.png](build.png "Global Build")

## Release Notes
### Version 0.1.0
* Spark is based on version 1.5.2 so to be able to use docker-compose without hostname constrains.

### Version 0.2.0
* To be able to use Spark version 1.6.2, docker-compose containers need to belong to an external network (which enforce a hostname without underscore). See [Switch to using hyphens as a separator in hostnames](https://github.com/docker/compose/issues/229):
```
$ docker network create spark
```
`docker-compose.yml`:
```
...
networks:
  default:
    external:
      name: spark
```

## Links
* [nats-connector-gatling on Github](https://github.com/Logimethods/nats-connector-gatling)
* [nats-connector-spark on Github](https://github.com/Logimethods/nats-connector-spark)
* [nats-connector-spark-scala on Github](https://github.com/Logimethods/nats-connector-spark-scala)
* [docker-nats-connector-spark on Github](https://github.com/Logimethods/docker-nats-connector-spark)
* [nats-connector-gatling build on Wercker](https://app.wercker.com/logimethods/nats-connector-gatling)
* [logimethods/maven-nats on Github](https://hub.docker.com/r/logimethods/maven-nats/builds/)
* [nats-connector-spark build on Wercker](https://app.wercker.com/logimethods/nats-connector-spark)
* [logimethods/sbt-nats on Github](https://hub.docker.com/r/logimethods/sbt-nats/builds/)
* [nats-connector-spark-scala build on Wercker](https://app.wercker.com/logimethods/nats-connector-spark-scala/)
* [logimethods on Nexus](https://oss.sonatype.org/#nexus-search;quick~logimethods)
* [docker-nats-connector-spark build on Wercker](https://app.wercker.com/logimethods/docker-nats-connector-spark)
* [nats-connector-spark on Docker Hub](https://hub.docker.com/r/logimethods/nats-connector-spark/)

## Outcome (digest)
```
compose$ docker-compose up
spark-master is up-to-date
spark-slave2 is up-to-date
nats-main is up-to-date
spark-slave1 is up-to-date
Recreating compose_monitor_1
Recreating main-app
Recreating compose_gatling_1
Attaching to spark-master, spark-slave2, nats-main, spark-slave1, main-app, compose_gatling_1, compose_monitor_1<commit_msg>[ci skip] Code block coding<commit_after># docker-nats-connector-spark
A collection of Docker Images to illustrate the use of the [nats-connector-spark](https://github.com/Logimethods/nats-connector-spark), [nats-connector-spark-scala](https://github.com/Logimethods/nats-connector-spark-scala) & [nats-connector-gatling](https://github.com/Logimethods/nats-connector-gatling) librairies.

[![MIT License](https://img.shields.io/npm/l/express.svg)](http://opensource.org/licenses/MIT)
[![Issues](https://img.shields.io/github/issues/Logimethods/docker-nats-connector-spark.svg)](https://github.com/Logimethods/docker-nats-connector-spark/issues)
[![wercker status](https://app.wercker.com/status/ae3458297bed4e5921beaba70acf173b/s/master "wercker status")](https://app.wercker.com/project/bykey/ae3458297bed4e5921beaba70acf173b)

[![Dockerhub](http://dockeri.co/image/logimethods/nats-connector-spark)](https://hub.docker.com/r/logimethods/nats-connector-spark/)

## Usage
```Shell
cd compose
docker-compose up
```

## What it does...
1) Docker-compose will instantiate and run a collection of Docker Containers defined by [docker-compose.yml](https://github.com/Logimethods/docker-nats-connector-spark/blob/master/compose/docker-compose.yml)

2) Gatling does emit every 15 sec a stream of values from 100 to 150 into NATS. See [NatsInjection.scala](https://github.com/Logimethods/docker-nats-connector-spark/blob/master/inject/user-files/simulations/nats/NatsInjection.scala).

3) Those values are transmitted from NATS to Spark Streaming (running on a Spark cluster), then the maximum value of each stream is computed by Spark & reemited into NATS. See [SparkProcessor.scala](https://github.com/Logimethods/docker-nats-connector-spark/blob/master/app/src/main/scala/com/logimethods/nats/connector/spark/app/SparkProcessor.scala)

4) Finally, those values are monitored and printed into the console. See [NatsOutputMonitor.scala](https://github.com/Logimethods/docker-nats-connector-spark/blob/master/monitor/src/main/scala/com/logimethods/nats/connector/spark/monitor/NatsOutputMonitor.scala)

![NATS-Docker_Containers.png](NATS-Docker_Containers.png "Data flow")

## Build

Those Docker Images are pushed to [dockerhub:logimethods/nats-connector-spark](https://hub.docker.com/r/logimethods/nats-connector-spark/) and build there.

![build.png](build.png "Global Build")

## Release Notes
### Version 0.1.0
* Spark is based on version 1.5.2 so to be able to use docker-compose without hostname constrains.

### Version 0.2.0
* To be able to use Spark version 1.6.2, docker-compose containers need to belong to an external network (which enforce a hostname without underscore). See [Switch to using hyphens as a separator in hostnames](https://github.com/docker/compose/issues/229):
```Shell
$ docker network create spark
```
`docker-compose.yml`:
```YAML
...
networks:
  default:
    external:
      name: spark
```

## Links
* [nats-connector-gatling on Github](https://github.com/Logimethods/nats-connector-gatling)
* [nats-connector-spark on Github](https://github.com/Logimethods/nats-connector-spark)
* [nats-connector-spark-scala on Github](https://github.com/Logimethods/nats-connector-spark-scala)
* [docker-nats-connector-spark on Github](https://github.com/Logimethods/docker-nats-connector-spark)
* [nats-connector-gatling build on Wercker](https://app.wercker.com/logimethods/nats-connector-gatling)
* [logimethods/maven-nats on Github](https://hub.docker.com/r/logimethods/maven-nats/builds/)
* [nats-connector-spark build on Wercker](https://app.wercker.com/logimethods/nats-connector-spark)
* [logimethods/sbt-nats on Github](https://hub.docker.com/r/logimethods/sbt-nats/builds/)
* [nats-connector-spark-scala build on Wercker](https://app.wercker.com/logimethods/nats-connector-spark-scala/)
* [logimethods on Nexus](https://oss.sonatype.org/#nexus-search;quick~logimethods)
* [docker-nats-connector-spark build on Wercker](https://app.wercker.com/logimethods/docker-nats-connector-spark)
* [nats-connector-spark on Docker Hub](https://hub.docker.com/r/logimethods/nats-connector-spark/)

## Outcome (digest)
```Shell
compose$ docker-compose up
spark-master is up-to-date
spark-slave2 is up-to-date
nats-main is up-to-date
spark-slave1 is up-to-date
Recreating compose_monitor_1
Recreating main-app
Recreating compose_gatling_1
Attaching to spark-master, spark-slave2, nats-main, spark-slave1, main-app, compose_gatling_1, compose_monitor_1
{'fraction_non_alphanumeric': 0.1429375351716376, 'fraction_numerical': 0.015531795160382668, 'mean_word_length': 5.26215644820296, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 54, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2904963', 'n_tokens_mistral': 3329, 'n_tokens_neox': 3016, 'n_words': 539}