<commit_before>Ansible Silo
============

[![GitHub release](https://img.shields.io/github/tag/groupon/ansible-silo.svg?style=flat&colorB=4183C4&label=Latest%20release)](https://github.com/groupon/ansible-silo/releases/latest)
[![Commits since v2.0.0](https://img.shields.io/github/commits-since/groupon/ansible-silo/v2.0.0.svg?style=flat&colorB=4183C4&label=Commits%20since%20v2.0.0)](https://github.com/groupon/ansible-silo/commits/master)
[![GitHub Stars](https://img.shields.io/github/stars/groupon/ansible-silo.svg?style=flat&colorB=4183C4&label=GitHub%20Stars)](https://github.com/groupon/ansible-silo/stargazers)
[![Build Status](https://img.shields.io/travis/groupon/ansible-silo.svg?style=flat&label=Tests)](https://travis-ci.org/groupon/ansible-silo)
[![Docker Build Status](https://img.shields.io/docker/build/grpn/ansible-silo.svg?style=flat&label=Build)](https://hub.docker.com/r/grpn/ansible-silo/builds/)
[![Code Climate](https://codeclimate.com/github/groupon/ansible-silo/badges/gpa.svg)](https://codeclimate.com/github/groupon/ansible-silo)

<!--

    For a more readable version, view this page on GitHub: https://github.com/groupon/ansible-silo#ansible-silo

-->

Table of Contents
-----------------

<!-- TOC -->

- [What problem does Silo solve?](#what-problem-does-silo-solve)
- [How it works](#how-it-works)
    - [Standalone mode](#standalone-mode)
    - [Bundle mode](#bundle-mode)
- [Included software](#included-software)
- [Installation](#installation)
    - [Prerequisites](#prerequisites)
    - [Install ansible-silo](#install-ansible-silo)
    - [Updating](#updating)
    - [Extending runner script](#extending-runner-script)
    - [Installing custom software](#installing-custom-software)
- [Configuration](#configuration)
    - [List of configuration options](#list-of-configuration-options)
        - [SILO_DEBUG](#silo_debug)
        - [SILO_DOCKER_CMD](#silo_docker_cmd)
        - [SILO_NO_PRIVILEGED](#silo_no_privileged)
        - [SILO_PATH](#silo_path)
        - [SILO_VERSION](#silo_version)
        - [SILO_VOLUME](#silo_volume)
- [Usage](#usage)
    - [`--version` Show current Silo & Ansible version](#--version-show-current-silo--ansible-version)
    - [`--switch` Switch to any Ansible version](#--switch-switch-to-any-ansible-version)
    - [`--reset` Resets a Silo volume](#--reset-resets-a-silo-volume)
    - [`--shell` Log into container / execute command in container](#--shell-log-into-container--execute-command-in-container)
    - [Run Silo with different Ansible versions](#run-silo-with-different-ansible-versions)
    - [Using Ansible](#using-ansible)
        - [Examples](#examples)
    - [Disabling privileged mode](#disabling-privileged-mode)
- [Bundle mode](#bundle-mode-1)
- [FAQ](#faq)
    - [Why do I always have to enter my SSH key passphrase when Silo starts?](#why-do-i-always-have-to-enter-my-ssh-key-passphrase-when-silo-starts)
- [Troubleshooting](#troubleshooting)
- [Versioning](#versioning)
- [Version history](#version-history)
    - [v2.0.0 (Aug 31, 2017)](#v200-aug-31-2017)
    - [v1.3.3 (July 20, 2017)](#v133-july-20-2017)
    - [v1.3.2 (July 17, 2017)](#v132-july-17-2017)
    - [v1.3.1 (July 13, 2017)](#v131-july-13-2017)
- [Development](#development)
    - [Build pipeline](#build-pipeline)
    - [Testing](#testing)
    - [Base image](#base-image)
- [License](#license)

<!-- /TOC -->

---

Silo provides you a controlled environment for Ansible and its dependencies. It works as a drop-in replacement for Ansible on your local machine or on any remote host.

Silo also makes it easy to run multiple Ansible versions in parallel on the same system.

Furthermore you can bundle your playbooks (incl. configuration, roles, plugins etc) in a custom Docker image which inherits Silo and therefore generate a versioned, shippable, complete and self-contained executable package which runs your playbooks in any environment. (where you have access to a Docker daemon)

For convenience ansible-silo includes [ansible-lint]. Since ansible-lint uses the Ansible libraries it may react differently depending on the used Ansible version.


## What problem does Silo solve?

If you expect reproducible outcome of an automation system, you not only need to make sure you have a specific version of the automation system itself, but also have fixed versions of all its dependencies. The most prominent Ansible dependency which can affect your plays would be Jinja2 but this applies to all involved components.

One approach to solve above problem are shared control hosts. Control hosts though add another problem: All teams and users who use the Ansible control hosts need to align on one specific Ansible version and it gets extremely complicated to update Ansible on the control hosts since all teams and users need to align and test all their roles and playbooks against the new version.

Silo not only removes moving parts by having 100% fixed dependencies hardcoded in a Docker image. It also enables you to switch Ansible to any version without affecting other users, therefore making it easy to test your playbooks and roles against new Ansible releases and run differing Ansible versions per playbook, project or user.


## How it works

Silo bundles Ansible and all its dependencies in a Docker image.

To enable Ansible in the container to connect to remote hosts, your `~/.ssh` folder is mounted and also, if available, your ssh auth socket (key forwarding) is mounted.

Starting the Silo container is a complex Docker command which needs to cover forwarding of environment variables and mounting required resources like the users ssh configuration. This complex command itself is included in the image and can be fetched by starting the container with the `--run` flag. The returned command then can be executed on the host which will start the container again with the correct parameters.

![Sequence diagram](docs/sequence-diagram.jpg)

A [bash script](./silo/bin) to easily trigger this process [can automatically be installed](#install-ansible-silo) by Silo when called with the `--install` flag.

The Silo container is not persistent, not running in the background. A new container is started for every Ansible call and is automatically removed after completion.


### Standalone mode

Standalone mode means you run Silo as a replacement for Ansible. By default Ansible is installed in a Docker volume. The volume can be changed, so you can have multiple volumes with different Ansible versions, e.g. per user, per environment or per playbook.

Playbooks will be mounted from the local file system and are not part of Silo.


### Bundle mode

A bundle is a Docker image which inherits the Silo Docker image.

In the bundle you can add your playbooks, roles, configuration and a specific Ansible version to the bundle.

A new bundle can easily be created by calling Silo with `--bundle <bundle name>` option.


## Included software

Silo is based on **[Alpine Linux] 3.6** and includes the following APK packages:

 - bash 4.3.48-r1
 - ca-certificates 20161130-r2
 - curl 7.55.0-r0
 - expat 2.2.0-r1
 - gdbm 1.12-r0
 - git 2.13.5-r0
 - gmp 6.1.2-r0
 - gosu 1.9-r0
 - libbz2 1.0.6-r5
 - libcurl 7.55.0-r0
 - libffi 3.2.1-r3
 - libssh2 1.8.0-r1
 - ncurses-libs 6.0-r8
 - ncurses-terminfo 6.0-r8
 - ncurses-terminfo-base 6.0-r8
 - musl 1.1.16-r13
 - musl-utils 1.1.16-r13
 - openssh 7.5_p1-r1
 - openssh-client 7.5_p1-r1
 - openssh-sftp-server 7.5_p1-r1
 - openssl 1.0.2k-r0
 - pcre 8.41-r0
 - perl 5.24.1-r2
 - py-netifaces 0.10.5-r3
 - py2-pip 9.0.1-r1
 - readline 6.3.008-r5
 - sqlite-libs 3.18.0-r0
 - sshpass <PASSWORD>
 - sudo 1.8.19_p2-r0
 - yaml 0.1.7-r0


The following Python modules are installed via pip:

 - cffi 1.10.0
 - cryptography 2.0.2
 - ecdsa 0.13
 - enum34 1.1.6
 - httplib2 0.9.2
 - idna 2.5
 - ipaddress 1.0.18
 - jinja2 2.8
 - markupsafe 0.23
 - netaddr 0.7.19
 - paramiko 1.16.0
 - pexpect 4.2.1
 - ptyprocess 0.5.2
 - pycparser 2.18<commit_msg>Documents use of jmespath<commit_after>Ansible Silo
============

[![GitHub release](https://img.shields.io/github/tag/groupon/ansible-silo.svg?style=flat&colorB=4183C4&label=Latest%20release)](https://github.com/groupon/ansible-silo/releases/latest)
[![Commits since v2.0.0](https://img.shields.io/github/commits-since/groupon/ansible-silo/v2.0.0.svg?style=flat&colorB=4183C4&label=Commits%20since%20v2.0.0)](https://github.com/groupon/ansible-silo/commits/master)
[![GitHub Stars](https://img.shields.io/github/stars/groupon/ansible-silo.svg?style=flat&colorB=4183C4&label=GitHub%20Stars)](https://github.com/groupon/ansible-silo/stargazers)
[![Build Status](https://img.shields.io/travis/groupon/ansible-silo.svg?style=flat&label=Tests)](https://travis-ci.org/groupon/ansible-silo)
[![Docker Build Status](https://img.shields.io/docker/build/grpn/ansible-silo.svg?style=flat&label=Build)](https://hub.docker.com/r/grpn/ansible-silo/builds/)
[![Code Climate](https://codeclimate.com/github/groupon/ansible-silo/badges/gpa.svg)](https://codeclimate.com/github/groupon/ansible-silo)

<!--

    For a more readable version, view this page on GitHub: https://github.com/groupon/ansible-silo#ansible-silo

-->

Table of Contents
-----------------

<!-- TOC -->

- [What problem does Silo solve?](#what-problem-does-silo-solve)
- [How it works](#how-it-works)
    - [Standalone mode](#standalone-mode)
    - [Bundle mode](#bundle-mode)
- [Included software](#included-software)
- [Installation](#installation)
    - [Prerequisites](#prerequisites)
    - [Install ansible-silo](#install-ansible-silo)
    - [Updating](#updating)
    - [Extending runner script](#extending-runner-script)
    - [Installing custom software](#installing-custom-software)
- [Configuration](#configuration)
    - [List of configuration options](#list-of-configuration-options)
        - [SILO_DEBUG](#silo_debug)
        - [SILO_DOCKER_CMD](#silo_docker_cmd)
        - [SILO_NO_PRIVILEGED](#silo_no_privileged)
        - [SILO_PATH](#silo_path)
        - [SILO_VERSION](#silo_version)
        - [SILO_VOLUME](#silo_volume)
- [Usage](#usage)
    - [`--version` Show current Silo & Ansible version](#--version-show-current-silo--ansible-version)
    - [`--switch` Switch to any Ansible version](#--switch-switch-to-any-ansible-version)
    - [`--reset` Resets a Silo volume](#--reset-resets-a-silo-volume)
    - [`--shell` Log into container / execute command in container](#--shell-log-into-container--execute-command-in-container)
    - [Run Silo with different Ansible versions](#run-silo-with-different-ansible-versions)
    - [Using Ansible](#using-ansible)
        - [Examples](#examples)
    - [Disabling privileged mode](#disabling-privileged-mode)
- [Bundle mode](#bundle-mode-1)
- [FAQ](#faq)
    - [Why do I always have to enter my SSH key passphrase when Silo starts?](#why-do-i-always-have-to-enter-my-ssh-key-passphrase-when-silo-starts)
- [Troubleshooting](#troubleshooting)
- [Versioning](#versioning)
- [Version history](#version-history)
    - [v2.0.0 (Aug 31, 2017)](#v200-aug-31-2017)
    - [v1.3.3 (July 20, 2017)](#v133-july-20-2017)
    - [v1.3.2 (July 17, 2017)](#v132-july-17-2017)
    - [v1.3.1 (July 13, 2017)](#v131-july-13-2017)
- [Development](#development)
    - [Build pipeline](#build-pipeline)
    - [Testing](#testing)
    - [Base image](#base-image)
- [License](#license)

<!-- /TOC -->

---

Silo provides you a controlled environment for Ansible and its dependencies. It works as a drop-in replacement for Ansible on your local machine or on any remote host.

Silo also makes it easy to run multiple Ansible versions in parallel on the same system.

Furthermore you can bundle your playbooks (incl. configuration, roles, plugins etc) in a custom Docker image which inherits Silo and therefore generate a versioned, shippable, complete and self-contained executable package which runs your playbooks in any environment. (where you have access to a Docker daemon)

For convenience ansible-silo includes [ansible-lint]. Since ansible-lint uses the Ansible libraries it may react differently depending on the used Ansible version.


## What problem does Silo solve?

If you expect reproducible outcome of an automation system, you not only need to make sure you have a specific version of the automation system itself, but also have fixed versions of all its dependencies. The most prominent Ansible dependency which can affect your plays would be Jinja2 but this applies to all involved components.

One approach to solve above problem are shared control hosts. Control hosts though add another problem: All teams and users who use the Ansible control hosts need to align on one specific Ansible version and it gets extremely complicated to update Ansible on the control hosts since all teams and users need to align and test all their roles and playbooks against the new version.

Silo not only removes moving parts by having 100% fixed dependencies hardcoded in a Docker image. It also enables you to switch Ansible to any version without affecting other users, therefore making it easy to test your playbooks and roles against new Ansible releases and run differing Ansible versions per playbook, project or user.


## How it works

Silo bundles Ansible and all its dependencies in a Docker image.

To enable Ansible in the container to connect to remote hosts, your `~/.ssh` folder is mounted and also, if available, your ssh auth socket (key forwarding) is mounted.

Starting the Silo container is a complex Docker command which needs to cover forwarding of environment variables and mounting required resources like the users ssh configuration. This complex command itself is included in the image and can be fetched by starting the container with the `--run` flag. The returned command then can be executed on the host which will start the container again with the correct parameters.

![Sequence diagram](docs/sequence-diagram.jpg)

A [bash script](./silo/bin) to easily trigger this process [can automatically be installed](#install-ansible-silo) by Silo when called with the `--install` flag.

The Silo container is not persistent, not running in the background. A new container is started for every Ansible call and is automatically removed after completion.


### Standalone mode

Standalone mode means you run Silo as a replacement for Ansible. By default Ansible is installed in a Docker volume. The volume can be changed, so you can have multiple volumes with different Ansible versions, e.g. per user, per environment or per playbook.

Playbooks will be mounted from the local file system and are not part of Silo.


### Bundle mode

A bundle is a Docker image which inherits the Silo Docker image.

In the bundle you can add your playbooks, roles, configuration and a specific Ansible version to the bundle.

A new bundle can easily be created by calling Silo with `--bundle <bundle name>` option.


## Included software

Silo is based on **[Alpine Linux] 3.6** and includes the following APK packages:

 - bash 4.3.48-r1
 - ca-certificates 20161130-r2
 - curl 7.55.0-r0
 - expat 2.2.0-r1
 - gdbm 1.12-r0
 - git 2.13.5-r0
 - gmp 6.1.2-r0
 - gosu 1.9-r0
 - libbz2 1.0.6-r5
 - libcurl 7.55.0-r0
 - libffi 3.2.1-r3
 - libssh2 1.8.0-r1
 - ncurses-libs 6.0-r8
 - ncurses-terminfo 6.0-r8
 - ncurses-terminfo-base 6.0-r8
 - musl 1.1.16-r13
 - musl-utils 1.1.16-r13
 - openssh 7.5_p1-r1
 - openssh-client 7.5_p1-r1
 - openssh-sftp-server 7.5_p1-r1
 - openssl 1.0.2k-r0
 - pcre 8.41-r0
 - perl 5.24.1-r2
 - py-netifaces 0.10.5-r3
 - py2-pip 9.0.1-r1
 - readline 6.3.008-r5
 - sqlite-libs 3.18.0-r0
 - sshpass <PASSWORD>-r0
 - sudo 1.8.19_p2-r0
 - yaml 0.1.7-r0


The following Python modules are installed via pip:

 - cffi 1.10.0
 - cryptography 2.0.2
 - ecdsa 0.13
 - enum34 1.1.6
 - httplib2 0.9.2
 - idna 2.5
 - ipaddress 1.0.18
 - jinja2 2.8
 - jmespath 0.9.3
 - markupsafe 0.23
 - netaddr 0.7.19
 - paramiko 1.16.0
 - pexpect 4.2.1
 - ptyprocess 0.5.2
 - pycparser 2.18
{'fraction_non_alphanumeric': 0.11890874882408278, 'fraction_numerical': 0.03712762621511446, 'mean_word_length': 3.7771120431396046, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 26, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3329659', 'n_tokens_mistral': 5701, 'n_tokens_neox': 5087, 'n_words': 1632}