<commit_before>This is a collection of chef recipies that set up the server for the
Cyclestreets Toolkit. You might need to customise them for your own situation,
and pull requests are very welcome.

# Setup

This is designed to use chef-solo. First we need to grab this repository and
prep the cookbooks, then install chef-solo, then chef can take care of the rest.
The base system is ubuntu-server 12.04 LTS so there's not much installed already.

    cd ~
    sudo apt-get install git
    git clone https://github.com/cyclestreets/toolkit-chef.git
    sudo mv toolkit-chef /opt
    cd /opt/toolkit-chef/

(From this point on, we could just make a magic script to do the rest.)

Now, we need to install chef. We're using chef 0.10 from the Opscode apt
repository.

    echo "deb http://apt.opscode.com/ `lsb_release -cs`-0.10 main" | sudo tee /etc/apt/sources.list.d/opscode.list
    sudo mkdir -p /etc/apt/trusted.gpg.d
    gpg --keyserver keys.gnupg.net --recv-keys 83EF826A
    gpg --export <EMAIL> | sudo tee /etc/apt/trusted.gpg.d/opscode-keyring.gpg > /dev/null
    sudo apt-get update
    sudo apt-get install opscode-keyring # permanent upgradeable keyring
    sudo apt-get upgrade
    sudo apt-get install chef

When you are prompted for the server url, enter "none"

# Databags

If you are running this recipe with chef-solo, you need to
create the secrets databag in /etc. Unfortunately chef loads databags
before running any recipies, so it needs to be done by hand.

    sudo mkdir -p /etc/chef/databags/secrets

Then copy the example file over:

    sudo cp cookbooks/toolkit/templates/default/mailbox.json /etc/chef/databags/secrets/
    sudo chmod 0600 /etc/chef/databags/secrets/mailbox.json

Then fill in the real values, to add the details of a mailbox you have set up on a
third-party server (Cyclescape will retrieve mail periodically from this).

    sudo nano /etc/chef/databags/secrets/mailbox.json

Then run chef as normal (described below). If you are running against a chef-server,
then create the databag from the .json example using knife.

N.B. When you set up the mailbox values, and run chef (below), it'll
start processing emails from that mailbox with no further configuration
change. It's worth being cautious when setting up failover servers,
for example. In that scenario, while setting up, don't put the credentials in.<commit_msg>Upgrade to chef-solo 11.10 and use the omnibus package directly.<commit_after>This is a collection of chef recipies that set up the server for the
Cyclestreets Toolkit. You might need to customise them for your own situation,
and pull requests are very welcome.

# Setup

This is designed to use chef-solo. First we need to grab this repository and
prep the cookbooks, then install chef-solo, then chef can take care of the rest.
The base system is ubuntu-server 12.04 LTS so there's not much installed already.

    cd ~
    sudo apt-get install git
    git clone https://github.com/cyclestreets/toolkit-chef.git
    sudo mv toolkit-chef /opt
    cd /opt/toolkit-chef/

(From this point on, we could just make a magic script to do the rest.)

Now, we need to install chef. We're using chef 11.10 with the ombnibus (i.e. embedded
ruby) package

    wget https://opscode-omnibus-packages.s3.amazonaws.com/ubuntu/12.04/x86_64/chef_11.10.4-1.ubuntu.12.04_amd64.deb
    sudo dpkg -i chef_11.10.4-1.ubuntu.12.04_amd64.deb


When you are prompted for the server url, enter "none"

# Databags

If you are running this recipe with chef-solo, you need to
create the secrets databag in /etc. Unfortunately chef loads databags
before running any recipies, so it needs to be done by hand.

    sudo mkdir -p /etc/chef/databags/secrets

Then copy the example file over:

    sudo cp cookbooks/toolkit/templates/default/mailbox.json /etc/chef/databags/secrets/
    sudo chmod 0600 /etc/chef/databags/secrets/mailbox.json

Then fill in the real values, to add the details of a mailbox you have set up on a
third-party server (Cyclescape will retrieve mail periodically from this).

    sudo nano /etc/chef/databags/secrets/mailbox.json

Then run chef as normal (described below). If you are running against a chef-server,
then create the databag from the .json example using knife.

N.B. When you set up the mailbox values, and run chef (below), it'll
start processing emails from that mailbox with no further configuration
change. It's worth being cautious when setting up failover servers,
for example. In that scenario, while setting up, don't put the credentials in.
{'fraction_non_alphanumeric': 0.06716747680070703, 'fraction_numerical': 0.015024304021210782, 'mean_word_length': 3.577350859453994, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 5, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3536124', 'n_tokens_mistral': 1484, 'n_tokens_neox': 1351, 'n_words': 620}