<commit_before>Ansible playbook to create a cluster with GlusterFS, Docker, Spark and JupyterHub services.

Features:

* Vagrant/Cloud
* Ansible
* GlusterFS
* Docker Swarm
* Spark Standalone cluster
* JupyterHub with Spark support (http://toree.apache.org)

# Features

# Vagrant/Cloud

Install in develop setup with Vagrant.

Use cloud machines to make services available for others.
When using cloud machines they are/have:
1. Ubuntu 16.04 OS
2. Public network interface
3. OS disk, 200Mb for software + enough room in /tmp
4. Passwordless login as root with `${HOST_NAME}.key` private key.
5. XFS Partition mounted at /data/local (used for swapfile, GlusterFS brick, Docker root)
6. Python2 to run Ansible tasks

See Build/Cloud chapter to automate step 4, 5 and 6.

# Ansible

Uses ansible to provision servers.

POSIX user `${HOST_NAME}` created with password `<PASSWORD>`.
To add more users edit `roles/common/vars/main.yml` file.

Firewall only allows connections from trusted networks.
The trusted networks can be changed in `roles/common/vars/main.yml` file.

Create the `hosts` file see `hosts.template` for template. To change ansible configurations the user should edit ansible.cfg at the root directory of this repository. A diff between it and the file under /etc/ansible/ansible.cfg shows the additions to the default version. 

## Ansible on Windows
When running on a Windows environment it is recommended to use the embedded Ubuntu environment, [installation guide](https://msdn.microsoft.com/en-us/commandline/wsl/install_guide).
After the installation the Ubuntu environment is accessible through the bash command of Windows.

Note the *C* drive will be mounted with the files owned by *root* and file permissions set to *777*. Ansible does run with such file permissions. Hence, you need to clone the repository into the home directory of the embedded Ubuntu environment. The environment set on Windows CMD console session, for example to run Vagrant, is not shared with embedded Ubuntu bash.


## GlusterFS

See http://gluster.readthedocs.io
All nodes have a xfs partition which is available as `gv0` volume and mounted as /data/shared on all nodes.
The volume is configured (replicas/stripes/transport/etc) in `roles/glusterfs/tasks/create-volume.yml` file.<commit_msg>Information on how to set Ubuntu bash on Windows and install Ansible.<commit_after>Ansible playbook to create a cluster with GlusterFS, Docker, Spark and JupyterHub services.

Features:

* Vagrant/Cloud
* Ansible
* GlusterFS
* Docker Swarm
* Spark Standalone cluster
* JupyterHub with Spark support (http://toree.apache.org)

# Features

# Vagrant/Cloud

Install in develop setup with Vagrant.

Use cloud machines to make services available for others.
When using cloud machines they are/have:
1. Ubuntu 16.04 OS
2. Public network interface
3. OS disk, 200Mb for software + enough room in /tmp
4. Passwordless login as root with `${HOST_NAME}.key` private key.
5. XFS Partition mounted at /data/local (used for swapfile, GlusterFS brick, Docker root)
6. Python2 to run Ansible tasks

See Build/Cloud chapter to automate step 4, 5 and 6.

# Ansible

The project uses ansible to provision servers. The recommended version is Ansible 2.2.

##Install ansible
Ansible should be installed using pip. To install pip and ansible dependencies do the following:

```
#Install pip:
sudo apt-get install python-pip

#Install dependencies:
sudo apt install libffi-dev python-dev libssl-dev

#Install ansible (Note that sudo is not used here because you are installing Ansible into your own virtual environment.):
pip install ansible

```

POSIX user `${HOST_NAME}` created with password `<PASSWORD>`.
To add more users edit `roles/common/vars/main.yml` file.

Firewall only allows connections from trusted networks.
The trusted networks can be changed in `roles/common/vars/main.yml` file.

Create the `hosts` file see `hosts.template` for template. To change ansible configurations the user should edit ansible.cfg at the root directory of this repository. A diff between it and the file under /etc/ansible/ansible.cfg shows the additions to the default version. 

## Ansible on Windows
When running on a Windows environment it is recommended to use the embedded Ubuntu environment, [installation guide](https://msdn.microsoft.com/en-us/commandline/wsl/install_guide).
Once installed it recommended to verify if the version 16.04 is installed.
```
lsb_release -a
```

If not 16.04, then do the following:
```
#Powershell as administrator, and enter the command
lxrun /uninstall /full .

lxrun /install /y

#Verify again the version:
lsb_release -a
```

After the installation the Ubuntu environment is accessible through the bash command of Windows.
To add Windows executables to your Ubuntu *$PATH*, do the following:
```
export PATH=$PATH:/mnt/c/Windows/System32/
```

Note the *C* drive will be mounted with the files owned by *root* and file permissions set to *777*.
This means ssh keys will to open for Ansible. Hence, before you run ansible you need to call getHosts.sh.

## GlusterFS

See http://gluster.readthedocs.io
All nodes have a xfs partition which is available as `gv0` volume and mounted as /data/shared on all nodes.
The volume is configured (replicas/stripes/transport/etc) in `roles/glusterfs/tasks/create-volume.yml` file.
{'fraction_non_alphanumeric': 0.07027433295753475, 'fraction_numerical': 0.010146561443066516, 'mean_word_length': 4.476337448559671, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '843993', 'n_tokens_mistral': 1599, 'n_tokens_neox': 1464, 'n_words': 716}