<commit_before>#### Table of Contents

1. [Overview](#overview)
2. [Module Description - What the module does and why it is useful](#module-description)
3. [Setup - The basics of getting started with spark](#setup)
    * [What spark affects](#what-spark-affects)
    * [Setup requirements](#setup-requirements)
4. [Usage - Configuration options and additional functionality](#usage)
    * [Spark in YARN cluster mode](#usage-yarn)
    * [Spark jar file optimization](#usage-jar-optimization)
    * [Add Spark History Server](#usage-history-server)
5. [Reference - An under-the-hood peek at what the module is doing and how](#reference)
    * [Classes](#classes)
    * [Module Parameters](#parameters)
5. [Limitations - OS compatibility, etc.](#limitations)
6. [Development - Guide for contributing to the module](#development)

<a name="overview"></a>
## Overview

Puppet module for deployment of Apache Spark.

<a name="module-description"></a>
## Module Description

This puppet module installs and setup Apache Spark cluster, optionally with security.

The standalone Apache Spark cluster is not supported yet (there are missing master and worker daemons here), only client is supported. But the YARN mode is working, so the full cluster submit (to Hadoop) is possible.

Supported are:

* Debian 7/wheezy: Cloudera distribution (tested with CDH 5.3.1, Spark 1.2.0)
* Ubuntu 14/trusty: Cloudera distribution (tested with CDH 5.3.1, Spark 1.2.0)
* RHEL 6, CentOS 6, Scientific Linux 6 (tested with CDH 5.4.2, Spark 1.3.0)

<a name="setup"></a>
## Setup

<a name="what-spark-affects"></a>
### What spark affects

* Packages: installs Spark packages as needed (core, python, history server, ...)
* Files modified:
 * */etc/spark/conf/spark-default.conf*
 * */etc/spark/conf/spark-env.sh* (modified, when *environment* parameter set)
 * */etc/default/spark*
 * */etc/profile.d/hadoop-spark.csh* (frontend)
 * */etc/profile.d/hadoop-spark.sh* (frontend)
* Permissions modified:
 * */etc/security/keytab/spark.service.keytab* (historyserver)
* Alternatives:
 * alternatives are used for */etc/spark/conf* in Cloudera
 * this module switches to the new alternative by default, so the Cloudera original configuration can be kept intact
* Services:
 * history server (when *spark::historyserver* or *spark::historyserver::service* included)
* Helper files:
 * */var/lib/hadoop-hdfs/.puppet-spark-\**

<a name="setup-requirements"></a>
### Setup Requirements

There are several known or intended limitations in this module.

Be aware of:

* **Hadoop repositories**
 * neither Cloudera nor Hortonworks repositories are configured in this module (for Cloudera you can find list and key files here: [http://archive.cloudera.com/cdh5/debian/wheezy/amd64/cdh/](http://archive.cloudera.com/cdh5/debian/wheezy/amd64/cdh/))
 * *java* is not installed by this module (*openjdk-7-jre-headless* is OK for Debian 7/wheezy)

* **No inter-node dependencies**: working HDFS is required before deploying of Spark History Server, dependency of Spark HDFS initialization on HDFS namenode is handled properly (if the class *spark::hdfs* is included on the HDF namenode, see example)

<a name="usage"></a>
## Usage

<a name="usage-yarn"></a>
### Spark in YARN cluster mode

Example of Apache Spark over Hadoop cluster. For simplicity one-machine Hadoop cluster is used (everything is on *$::fqdn*, replication factor 1).

    class{'hadoop':
      hdfs_hostname => $::fqdn,
      yarn_hostname => $::fqdn,
      slaves => [ $::fqdn ],
      frontends => [ $::fqdn ],
      realm => '',
      properties => {
        'dfs.replication' => 1,
      },
    }

    class{'spark':
      hdfs_hostname => $::fqdn,
    }

    node default {
      include stdlib

      include hadoop::namenode
      include hadoop::resourcemanager
      include hadoop::historyserver
      include hadoop::datanode
      include hadoop::nodemanager
      include hadoop::frontend

      include spark::frontend
      # should be collocated with hadoop::namenode
      include spark::hdfs
    }

Issues:

* if collocated with HDFS namenode, add dependency *Class['hadoop::namenode::service'] -> Class['spark::historyserver::service']*
* if not collocated, it is needed to have HDFS namenode running first (puppet should be launched later again, if Spark History Server won't start because of HDFS)

Notes:

* for Spark clients: user must logout and login again or launch "*. /etc/profile.d/hadoop-spark.sh*"

Now you can submit spark jobs in the cluster mode:

    spark-submit --class org.apache.spark.examples.SparkPi --deploy-mode cluster --master yarn /usr/lib/spark/lib/spark-examples-1.2.0-cdh5.3.1-hadoop2.5.0-cdh5.3.1.jar 10

<a name="usage-jar-optimization"></a>
### Spark jar file optimization

The *spark-assembly.jar* file is copied into HDFS on each job submit. It is possible to optimize this by copying it manually. Keep in mind the jar file needs to be refreshed on HDFS with each Spark SW update.

    ...

    class{'spark':
      hdfs_hostname => $::fqdn,
      jar_enable    => true,
    }

    ...

Copy the jar file after installation and deployment (superuser credentials are needed if security in Hadoop is enabled):

    hdfs dfs -put /usr/lib/spark/spark-assembly.jar /user/spark/share/lib/spark-assembly.jar

<a name="usage-history-server"></a>
### Add Spark History Server

Spark History server stores details about Spark jobs. It is provided by the class *spark::historyserver*. The parameter *historiserver\_hostname* needs to be also specified (replace *$::fqdn* by real hostname):

    ...
    class{'spark':
      ...
      historyserver_hostname => $::fqdn,
    }

    node default {
      ...
      include spark::historyserver
    }

<a name="reference"></a>
##Reference

<a name="classes"></a>
###Classes
<commit_msg>Documentation: upgrade.<commit_after>#### Table of Contents

1. [Overview](#overview)
2. [Module Description - What the module does and why it is useful](#module-description)
3. [Setup - The basics of getting started with spark](#setup)
    * [What spark affects](#what-spark-affects)
    * [Setup requirements](#setup-requirements)
4. [Usage - Configuration options and additional functionality](#usage)
    * [Spark in YARN cluster mode](#usage-yarn)
    * [Spark jar file optimization](#usage-jar-optimization)
    * [Add Spark History Server](#usage-history-server)
    * [Upgrade](#upgrade)
5. [Reference - An under-the-hood peek at what the module is doing and how](#reference)
    * [Classes](#classes)
    * [Module Parameters](#parameters)
5. [Limitations - OS compatibility, etc.](#limitations)
6. [Development - Guide for contributing to the module](#development)

<a name="overview"></a>
## Overview

Puppet module for deployment of Apache Spark.

<a name="module-description"></a>
## Module Description

This puppet module installs and setup Apache Spark cluster, optionally with security.

The standalone Apache Spark cluster is not supported yet (there are missing master and worker daemons here), only client is supported. But the YARN mode is working, so the full cluster submit (to Hadoop) is possible.

Supported are:

* Debian 7/wheezy: Cloudera distribution (tested with CDH 5.3.1, Spark 1.2.0)
* Ubuntu 14/trusty: Cloudera distribution (tested with CDH 5.3.1, Spark 1.2.0)
* RHEL 6, CentOS 6, Scientific Linux 6 (tested with CDH 5.4.2, Spark 1.3.0)

<a name="setup"></a>
## Setup

<a name="what-spark-affects"></a>
### What spark affects

* Packages: installs Spark packages as needed (core, python, history server, ...)
* Files modified:
 * */etc/spark/conf/spark-default.conf*
 * */etc/spark/conf/spark-env.sh* (modified, when *environment* parameter set)
 * */etc/default/spark*
 * */etc/profile.d/hadoop-spark.csh* (frontend)
 * */etc/profile.d/hadoop-spark.sh* (frontend)
* Permissions modified:
 * */etc/security/keytab/spark.service.keytab* (historyserver)
* Alternatives:
 * alternatives are used for */etc/spark/conf* in Cloudera
 * this module switches to the new alternative by default, so the Cloudera original configuration can be kept intact
* Services:
 * history server (when *spark::historyserver* or *spark::historyserver::service* included)
* Helper files:
 * */var/lib/hadoop-hdfs/.puppet-spark-\**

<a name="setup-requirements"></a>
### Setup Requirements

There are several known or intended limitations in this module.

Be aware of:

* **Hadoop repositories**
 * neither Cloudera nor Hortonworks repositories are configured in this module (for Cloudera you can find list and key files here: [http://archive.cloudera.com/cdh5/debian/wheezy/amd64/cdh/](http://archive.cloudera.com/cdh5/debian/wheezy/amd64/cdh/))
 * *java* is not installed by this module (*openjdk-7-jre-headless* is OK for Debian 7/wheezy)

* **No inter-node dependencies**: working HDFS is required before deploying of Spark History Server, dependency of Spark HDFS initialization on HDFS namenode is handled properly (if the class *spark::hdfs* is included on the HDF namenode, see example)

<a name="usage"></a>
## Usage

<a name="usage-yarn"></a>
### Spark in YARN cluster mode

Example of Apache Spark over Hadoop cluster. For simplicity one-machine Hadoop cluster is used (everything is on *$::fqdn*, replication factor 1).

    class{'hadoop':
      hdfs_hostname => $::fqdn,
      yarn_hostname => $::fqdn,
      slaves => [ $::fqdn ],
      frontends => [ $::fqdn ],
      realm => '',
      properties => {
        'dfs.replication' => 1,
      },
    }

    class{'spark':
      hdfs_hostname => $::fqdn,
    }

    node default {
      include stdlib

      include hadoop::namenode
      include hadoop::resourcemanager
      include hadoop::historyserver
      include hadoop::datanode
      include hadoop::nodemanager
      include hadoop::frontend

      include spark::frontend
      # should be collocated with hadoop::namenode
      include spark::hdfs
    }

Issues:

* if collocated with HDFS namenode, add dependency *Class['hadoop::namenode::service'] -> Class['spark::historyserver::service']*
* if not collocated, it is needed to have HDFS namenode running first (puppet should be launched later again, if Spark History Server won't start because of HDFS)

Notes:

* for Spark clients: user must logout and login again or launch "*. /etc/profile.d/hadoop-spark.sh*"

Now you can submit spark jobs in the cluster mode:

    spark-submit --class org.apache.spark.examples.SparkPi --deploy-mode cluster --master yarn /usr/lib/spark/lib/spark-examples-1.2.0-cdh5.3.1-hadoop2.5.0-cdh5.3.1.jar 10

<a name="usage-jar-optimization"></a>
### Spark jar file optimization

The *spark-assembly.jar* file is copied into HDFS on each job submit. It is possible to optimize this by copying it manually. Keep in mind the jar file needs to be refreshed on HDFS with each Spark SW update.

    ...

    class{'spark':
      hdfs_hostname => $::fqdn,
      jar_enable    => true,
    }

    ...

Copy the jar file after installation and deployment (superuser credentials are needed if security in Hadoop is enabled):

    hdfs dfs -put /usr/lib/spark/spark-assembly.jar /user/spark/share/lib/spark-assembly.jar

<a name="usage-history-server"></a>
### Add Spark History Server

Spark History server stores details about Spark jobs. It is provided by the class *spark::historyserver*. The parameter *historiserver\_hostname* needs to be also specified (replace *$::fqdn* by real hostname):

    ...
    class{'spark':
      ...
      historyserver_hostname => $::fqdn,
    }

    node default {
      ...
      include spark::historyserver
    }

<a name="upgrade"></a>
### Upgrade

The best way is to refresh configrations from the new original (=remove the old) and relaunch puppet on top of it. There is also problem with start-up scripts on Debian, which needs to be worked around, where Spark history server is used.

For example:

    alternative='cluster'
    d='spark'
    mv /etc/{d}$/conf.${alternative} /etc/${d}/conf.cdhXXX
    update-alternatives --auto ${d}-conf

    service spark-history-server stop || :
    mv /etc/init.d/spark-history-server /etc/init.d/spark-history-server.prev

    # upgrade
    ...

    puppet agent --test
    #or: puppet apply ...

    # restore start-up script from spark-history-server.dpkg-new or spark-history-server.prev
    ...
    service spark-history-server start


<a name="reference"></a>
##Reference

<a name="classes"></a>
###Classes

{'fraction_non_alphanumeric': 0.12225983236621535, 'fraction_numerical': 0.008865248226950355, 'mean_word_length': 3.7362595419847326, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 73, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1302685', 'n_tokens_mistral': 4006, 'n_tokens_neox': 3677, 'n_words': 1280}