<commit_before># nubis-base

You can build an image like so:

$> cat packer/nubis/variables.json
{
  "aws_access_key": "XXX",
  "aws_secret_key": "XXX",
  "iam_instance_profile": "packer"
}

$> make

The structure of the project is like this:

nubis/
  puppet/
  packer/
  terraform/


nubis/puppet/*.pp

is for the puppet manifests specific to this image you are building

nubis/packer/main.json

is for the packer definition of the image you are building. Ideally, should
basically just be a puppet-masterless provisionner into nubis/puppet/init.pp

nubis/terraform/main.tf

is for describing the production deployment of this component. Ideally, should
be just infrastructure components as well as AMIs baked by nubis.
<commit_msg>Augmented README.md<commit_after># nubis-base

## Quick start
0. Copy nubis/packer/variables.json-dist to nubis/packer/variables.json and configure it with your AWS access keys
0. Run: `make build` and wait a few minutes

## Required files

Check out the -dist files in nubis/packer to get an idea of what they'll need to look like.

##### nubis/packer/ami.json
This file contains the source AMI IDs that nubis-base is built on top of. You can fill this in yourself if you have a specific *need* to pin to an older AMI, otherwise it's recommended that you run `make generate-ami-json` generate this file for you using the latest compatible AMIs. In the case of Ubuntu we pin it to the version major (currently trusty) as some required software is not yet packaged for utopic.

##### nubis/packer/project.json
This file contains the name and description of your project.

##### nubis/packer/variables.json
This file contains your secret data like access keys, certificate paths, etc and will be specific to your AWS account.

##### nubis/packer/release.json
This file tracks the release and build number for the project. Build numbers are automatically incremented, while a `make release` is required to increment the release number. It's created for you automatically when you run your first build.

## Make targets

##### `make puppet`
This will invoke librarian puppet and pull in all the required puppet sources for the packer provisioner. This step needs to run before packer can launch builders.

##### `make generate-ami-json`
This will run nubis/bin/generate-latest-amis which will populate nubis/packer/ami.json with source AMI IDs

##### `make packer`
Increments the build number in `nubis/packer/release.json` and kicks off a packer build

##### `make build`
Increments the build number, invokes puppet, invokes generate-ami-json, and invokes puppet all with one handy command.

##### `make release`
Increments the release number, resets the build number, invokes puppet, invokes generate-ami-json, and invokes puppet all with one handy command.

## File structure

##### `nubis`
All files related to the nubis build system, you probably don't want to put your project files in here.

##### `nubis/bin`
Build tools related to building AMIs

##### `nubis/nubis-puppet`
This is the puppet tree that's populated with librarian-puppet, it's in .gitignore and gets reset on every build.

##### `nubis/packer`
All of the variable files and the master build template for packer, 
###### `nubis/packer/main.json`
This is the main build template, take a look and add your own custom provisioners or tweak what's already there. It's good practice to avoid hard coding variables into this file, instead think about using variables.json for sensitive data or project.json for project specifics.

##### `nubis/terraform`
Terraform deployment templates.

###### `nubis/terraform/main.tf`
This file describes the production deployment of this project.

{'fraction_non_alphanumeric': 0.07851127411029611, 'fraction_numerical': 0.0005433306166802499, 'mean_word_length': 4.479166666666667, 'pattern_counts': {'":': 3, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7670075', 'n_tokens_mistral': 1089, 'n_tokens_neox': 1032, 'n_words': 500}