<commit_before># packer-templates

Collection of Packer templates used for various infrastructure layers.

## How to build stuff

To build a given template, one may use the `make` implicit builder, like the
following for `ci-connie`:

``` bash
make ci-connie
```

or, with a specific builder:

``` bash
make ci-connie BUILDER=docker
```

or forget about the `Makefile` and run with `packer` directly:

``` bash
packer build -only=docker <(bin/yml2json < ci-connie.yml)
```

## env config bits

Most of the templates in here require some env vars.  Take a look at
[`.example.env`](./.example.env) for an example.  Use of
[autoenv](https://github.com/kennethreitz/autoenv) is encouraged but not
required.

## packer template types

There are two primary types of templates present at the top level: those
intended for use as execution environment for jobs flowing through Travis CI,
and those used for various backend fun in the Travis CI infrastructure.  The
former type all have the prefix `ci-`, described in more detail below:

### stacks

There are two primary types of stacks: those targeting Ubuntu 14.04 (trusty),
and those targeting Ubuntu 16.04 (xenial) that run on GCE and Docker.

Take a peek at what's what:

``` bash
make stacks-trusty
```

``` bash
make stacks-xenial
```

There may be some subtle variations, but for the most part each stack is built
via the following steps.

#### git metadata file input

The generated files in `./tmp/git-meta/` are copied onto the provisioned machine
at `/var/tmp/git-meta/` for later use by the `./packer-scripts/packer-env-dump`
script.

#### purge file input

A git-tracked file in `./packer-assets` is copied onto the provisioned machine
at `/var/tmp/purge.txt` for later use by the `./packer-scripts/purge` script.

#### packages file input

A git-tracked file in `./packer-assets` is copied onto the provisioned machine
at `/var/tmp/packages.txt` for later use by both the
`travis_packer_templates::default` recipe and the serverspec suites via
`./cookbooks/lib/support.rb`.

#### write packer and travis env vars

The script at `./packer-scripts/packer-env-dump` creates a directory on the
provisioned machine at `/.packer-env` which is intended to be in the [envdir
format](https://cr.yp.to/daemontools/envdir.html).  Any environment variables
that match `^(PACKER|TRAVIS)`, and (if present) the files previously written to
`/var/tmp/git-meta/` are copied or written into `/.packer-env/`.

#### remove default users

The script at `./packer-scripts/remove-default-users` will perform a best-effort
removal of users defined in `${DEFAULT_USERS}` (default `vagrant ubuntu`).  The
primary reasons for this are general tidyness and to try to free up uid 2000.

#### pre-chef bootstrapping

The script at `./packer-scripts/pre-chef-bootstrap` is responsible for ensuring
the provisioned machine has all necessary packages and users for the Chef
provisioning process.  The steps executed include:

- remove the "partner" APT source list file
- remove all cached APT list files
- install APT packages needed by Chef
- ensure `/var/run/sshd` dir exists
- ensure `sshd: ALL: ALLOW` exists in `/etc/hosts.allow`
- ensure there is a `travis` user
- change the `travis` user password to `<PASSWORD>`
- ensure `#includedir /etc/sudoers.d` exists in `/etc/sudoers`
- ensure the `/etc/sudoers.d` dir exists
- ensure the `/etc/sudoers.d/travis` file exists with specific permissions
- ensure the `/home/travis/.ssh` dir exists
- ensure the `/home/travis/.ssh/authorized_keys` file exists
- add `/var/tmp/*_rsa.pub` to `/home/travis/.ssh/authorized_keys`
- ensure `/home/travis/.ssh/authorized_keys` perms are `0600`
- ensure the `/home/travis/bin` dir exists

#### cloning travis-cookbooks

The script at `./packer-scripts/clone-travis-cookbooks` is responsible for `git
clone`'ing [travis-cookbooks](https://github.com/travis-ci/travis-cookbooks)
into `/tmp/chef-stuff` on the provisioned machine.  Optional env vars supported
by this script are:

- `TRAVIS_COOKBOOKS_BRANCH` - the branch specified during `git clone`
- `TRAVIS_COOKBOOKS_EDGE_BRANCH` - the default branch used if
  `TRAVIS_COOKBOOKS_BRANCH` is not defined
- `TRAVIS_COOKBOOKS_URL` - the git clone remote (default
  `https://github.com/travis-ci/travis-cookbooks.git`)
- `TRAVIS_COOKBOOKS_SHA` - a git tree-ish to which the clone will be checked
  out if defined (default not set)

Once the clone is complete, the clone directory is written to
`/.packer-env/TRAVIS_COOKBOOKS_DIR` and the head sha is written to
`/.packer-env/TRAVIS_COOKBOOKS_SHA`.

#### chef provisioning

The `chef-solo` provisioner will typically have *no json data*, but instead will
leave all attribute and effective run list definition to a single wrapper
cookbook located in `./cookbooks/`.

##### chef wrapper cookbook layout

Each wrapper cookbook must contain at least a `metadata.rb` and a
`recipes/default.rb`.  Typically, the `attributes/default.rb` is defined and
contains all override attribute settings.  The earliest version of Chef used by
either trusty or xenial stacks is `12.9`, which means that *all* cookbook
dependencies must be declared in `metadata.rb`, a requirement that is also
enforced by the `foodcritic` checks.

For example, the minimal trusty image "ci-connie" has a wrapper cookbook at
`./cookbooks/travis-ci_connie` that looks like this:

```
cookbooks/travis_ci_connie
├── README.md
├── attributes
│   └── default.rb
├── metadata.rb
├── recipes
│   └── default.rb
└── spec
	├── ...
```

#### travis user double check

The script at `./packer-scripts/ensure-travis-user` is responsible for ensuring
the existence of the `travis` user and its home directory permissions,
optionally setting the password to a random string.  The list of operations is:

- ensure the `travis` user exists
- set the `travis` user password
- ensure `/home/travis` exists
- ensure `/home/travis/.ssh/authorized_keys` and `/home/travis/.ssh/known_hosts`
  both exist and have permissions of `0600`
- blank out `/home/travis/.ssh/authorized_keys`
- ensure `/home/travis` is fully owned by `travis:travis`

Optional env vars supported by this script are:

- `TRAVIS_USER_PASSWORD` - a string (default "<PASSWORD>")
- `TRAVIS_OBFUSCATE_PASSWORD` - if non-empty, causes
  `TRAVIS_USER_PASSWORD` to be set to a random string

#### purging undesirable packages

The script at `./packer-scripts/purge` is responsible for purging packages that
are not desirable for the CI environment, such as the Chef that was installed
prior for the Chef provisioner.  Additionally, any package names present in
`/var/tmp/purge.txt` will be purged.  Optional env vars supported by this script<commit_msg>Fix whitespace change introduced with PR 522<commit_after># packer-templates

Collection of Packer templates used for various infrastructure layers.

## How to build stuff

To build a given template, one may use the `make` implicit builder, like the
following for `ci-connie`:

``` bash
make ci-connie
```

or, with a specific builder:

``` bash
make ci-connie BUILDER=docker
```

or forget about the `Makefile` and run with `packer` directly:

``` bash
packer build -only=docker <(bin/yml2json < ci-connie.yml)
```

## env config bits

Most of the templates in here require some env vars.  Take a look at
[`.example.env`](./.example.env) for an example.  Use of
[autoenv](https://github.com/kennethreitz/autoenv) is encouraged but not
required.

## packer template types

There are two primary types of templates present at the top level: those
intended for use as execution environment for jobs flowing through Travis CI,
and those used for various backend fun in the Travis CI infrastructure.  The
former type all have the prefix `ci-`, described in more detail below:

### stacks

There are two primary types of stacks: those targeting Ubuntu 14.04 (trusty),
and those targeting Ubuntu 16.04 (xenial) that run on GCE and Docker.

Take a peek at what's what:

``` bash
make stacks-trusty
```

``` bash
make stacks-xenial
```

There may be some subtle variations, but for the most part each stack is built
via the following steps.

#### git metadata file input

The generated files in `./tmp/git-meta/` are copied onto the provisioned machine
at `/var/tmp/git-meta/` for later use by the `./packer-scripts/packer-env-dump`
script.

#### purge file input

A git-tracked file in `./packer-assets` is copied onto the provisioned machine
at `/var/tmp/purge.txt` for later use by the `./packer-scripts/purge` script.

#### packages file input

A git-tracked file in `./packer-assets` is copied onto the provisioned machine
at `/var/tmp/packages.txt` for later use by both the
`travis_packer_templates::default` recipe and the serverspec suites via
`./cookbooks/lib/support.rb`.

#### write packer and travis env vars

The script at `./packer-scripts/packer-env-dump` creates a directory on the
provisioned machine at `/.packer-env` which is intended to be in the [envdir
format](https://cr.yp.to/daemontools/envdir.html).  Any environment variables
that match `^(PACKER|TRAVIS)`, and (if present) the files previously written to
`/var/tmp/git-meta/` are copied or written into `/.packer-env/`.

#### remove default users

The script at `./packer-scripts/remove-default-users` will perform a best-effort
removal of users defined in `${DEFAULT_USERS}` (default `vagrant ubuntu`).  The
primary reasons for this are general tidyness and to try to free up uid 2000.

#### pre-chef bootstrapping

The script at `./packer-scripts/pre-chef-bootstrap` is responsible for ensuring
the provisioned machine has all necessary packages and users for the Chef
provisioning process.  The steps executed include:

- remove the "partner" APT source list file
- remove all cached APT list files
- install APT packages needed by Chef
- ensure `/var/run/sshd` dir exists
- ensure `sshd: ALL: ALLOW` exists in `/etc/hosts.allow`
- ensure there is a `travis` user
- change the `travis` user password to `<PASSWORD>`
- ensure `#includedir /etc/sudoers.d` exists in `/etc/sudoers`
- ensure the `/etc/sudoers.d` dir exists
- ensure the `/etc/sudoers.d/travis` file exists with specific permissions
- ensure the `/home/travis/.ssh` dir exists
- ensure the `/home/travis/.ssh/authorized_keys` file exists
- add `/var/tmp/*_rsa.pub` to `/home/travis/.ssh/authorized_keys`
- ensure `/home/travis/.ssh/authorized_keys` perms are `0600`
- ensure the `/home/travis/bin` dir exists

#### cloning travis-cookbooks

The script at `./packer-scripts/clone-travis-cookbooks` is responsible for `git
clone`'ing [travis-cookbooks](https://github.com/travis-ci/travis-cookbooks)
into `/tmp/chef-stuff` on the provisioned machine.  Optional env vars supported
by this script are:

- `TRAVIS_COOKBOOKS_BRANCH` - the branch specified during `git clone`
- `TRAVIS_COOKBOOKS_EDGE_BRANCH` - the default branch used if
  `TRAVIS_COOKBOOKS_BRANCH` is not defined
- `TRAVIS_COOKBOOKS_URL` - the git clone remote (default
  `https://github.com/travis-ci/travis-cookbooks.git`)
- `TRAVIS_COOKBOOKS_SHA` - a git tree-ish to which the clone will be checked
  out if defined (default not set)

Once the clone is complete, the clone directory is written to
`/.packer-env/TRAVIS_COOKBOOKS_DIR` and the head sha is written to
`/.packer-env/TRAVIS_COOKBOOKS_SHA`.

#### chef provisioning

The `chef-solo` provisioner will typically have *no json data*, but instead will
leave all attribute and effective run list definition to a single wrapper
cookbook located in `./cookbooks/`.

##### chef wrapper cookbook layout

Each wrapper cookbook must contain at least a `metadata.rb` and a
`recipes/default.rb`.  Typically, the `attributes/default.rb` is defined and
contains all override attribute settings.  The earliest version of Chef used by
either trusty or xenial stacks is `12.9`, which means that *all* cookbook
dependencies must be declared in `metadata.rb`, a requirement that is also
enforced by the `foodcritic` checks.

For example, the minimal trusty image "ci-connie" has a wrapper cookbook at
`./cookbooks/travis-ci_connie` that looks like this:

```
cookbooks/travis_ci_connie
├── README.md
├── attributes
│   └── default.rb
├── metadata.rb
├── recipes
│   └── default.rb
└── spec
    ├── ...
```

#### travis user double check

The script at `./packer-scripts/ensure-travis-user` is responsible for ensuring
the existence of the `travis` user and its home directory permissions,
optionally setting the password to a random string.  The list of operations is:

- ensure the `travis` user exists
- set the `travis` user password
- ensure `/home/travis` exists
- ensure `/home/travis/.ssh/authorized_keys` and `/home/travis/.ssh/known_hosts`
  both exist and have permissions of `0600`
- blank out `/home/travis/.ssh/authorized_keys`
- ensure `/home/travis` is fully owned by `travis:travis`

Optional env vars supported by this script are:

- `TRAVIS_USER_PASSWORD` - a string (default "<PASSWORD>")
- `TRAVIS_OBFUSCATE_PASSWORD` - if non-empty, causes
  `TRAVIS_USER_PASSWORD` to be set to a random string

#### purging undesirable packages

The script at `./packer-scripts/purge` is responsible for purging packages that
are not desirable for the CI environment, such as the Chef that was installed
prior for the Chef provisioner.  Additionally, any package names present in
`/var/tmp/purge.txt` will be purged.  Optional env vars supported by this script
{'fraction_non_alphanumeric': 0.10109167040526394, 'fraction_numerical': 0.003813369223867205, 'mean_word_length': 4.261605035405193, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 7, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4241144', 'n_tokens_mistral': 4281, 'n_tokens_neox': 4151, 'n_words': 1669}