<commit_before># Utility Server

Here's how to set up a utility server to test out bagman. These
instructions descibe the setup process for a small server, which is
good for test runs because 1) it's cheap, and 2) it comes with 160GB
of attached storage, which is handy because bagman will be working
with some fairly large files.

Assuming you've already spun up an EC2 small running a 32-bit Ubuntu
12.04 or 14.04 AMI, follow the instructions below.

Install an editor if you don't like working with vim. If you like vim,
just kick back and relax.

```
    sudo apt-get install emacs23-nox golang-mode
```

The "go get" command uses git, bazaar, and mercurial to download
packages.  Install those, along with libmagic-dev, which we need
for determining file mime types. We'll need gcc to build the
magicmime go package later.

We also need mailutils so that cron jobs can send email, alpine is
handy for reading email, and the AWS CLI is useful for automating tasks.

```
    sudo apt-get install git bzr mercurial
    sudo apt-get install gcc libmagic-dev
    sudo apt-get install mailutils
    sudo apt-get install alpine
    sudo apt-get install awscli
```

Install go, but don't use apt-get.

```
    curl "https://storage.googleapis.com/golang/go1.2.2.linux-amd64.tar.gz" > go1.2.2.linux-amd64.tar.gz
    sudo tar -C /usr/local -xzf go1.2.2.linux-amd64.tar.gz
```

Create a GOHOME directory:

```
    mkdir ~/go
```

Create ~/.bash_profile with the following:

```
    export EDITOR=emacs
    export AWS_ACCESS_KEY_ID="Our access key id"
    export AWS_SECRET_ACCESS_KEY="Our secret key"
    export GOPATH=$HOME/go
    export PATH="$GOPATH/bin:$PATH:/usr/local/go/bin"
```

Run this to load the bash profile:

```
    source ~/.bash_profile
```

Now if you run this:

```
    go version
```

You should see this output:

```
    go version go1.2.2 linux/amd64
```

Mount the EBS volume and then make directories to hold the tar/bag
files we're going to download.

```
    sudo mkdir /mnt/apt
    sudo mount /dev/xvdc /mnt/apt
    sudo mkdir /mnt/apt/data
    sudo mkdir /mnt/apt/logs
    sudo mkdir /mnt/apt/replication
    sudo mkdir /mnt/apt/restore
    sudo chown ubuntu.ubuntu /mnt/apt/data/
    sudo chown ubuntu.ubuntu /mnt/apt/logs/
    sudo chown ubuntu.ubuntu /mnt/apt/restore/
    sudo chown ubuntu.ubuntu /mnt/apt/replication/
```

Install the go packages we'll need:

```
    go get github.com/diamondap/goamz/aws
    go get github.com/diamondap/goamz/s3
    go get github.com/nu7hatch/gouuid
    go get github.com/rakyll/magicmime
    go get github.com/APTrust/bagins
    go get github.com/APTrust/bagman
    go get github.com/op/go-logging
```

You can also copy a version of bagman from your local machine, like
so:

```
    scp -r ~/go/src/github.com/APTrust/bagman/ ubuntu@apt-util:go/src/github.com/APTrust/
```

Note that this assumes you have an entry in your local ~/.ssh/config
like this:

```
    Host apt-util
        User ubuntu
        Port 22<commit_msg>Changed diamondap to crowdmob<commit_after># Utility Server

Here's how to set up a utility server to test out bagman. These
instructions descibe the setup process for a small server, which is
good for test runs because 1) it's cheap, and 2) it comes with 160GB
of attached storage, which is handy because bagman will be working
with some fairly large files.

Assuming you've already spun up an EC2 small running a 32-bit Ubuntu
12.04 or 14.04 AMI, follow the instructions below.

Install an editor if you don't like working with vim. If you like vim,
just kick back and relax.

```
    sudo apt-get install emacs23-nox golang-mode
```

The "go get" command uses git, bazaar, and mercurial to download
packages.  Install those, along with libmagic-dev, which we need
for determining file mime types. We'll need gcc to build the
magicmime go package later.

We also need mailutils so that cron jobs can send email, alpine is
handy for reading email, and the AWS CLI is useful for automating tasks.

```
    sudo apt-get install git bzr mercurial
    sudo apt-get install gcc libmagic-dev
    sudo apt-get install mailutils
    sudo apt-get install alpine
    sudo apt-get install awscli
```

Install go, but don't use apt-get.

```
    curl "https://storage.googleapis.com/golang/go1.2.2.linux-amd64.tar.gz" > go1.2.2.linux-amd64.tar.gz
    sudo tar -C /usr/local -xzf go1.2.2.linux-amd64.tar.gz
```

Create a GOHOME directory:

```
    mkdir ~/go
```

Create ~/.bash_profile with the following:

```
    export EDITOR=emacs
    export AWS_ACCESS_KEY_ID="Our access key id"
    export AWS_SECRET_ACCESS_KEY="Our secret key"
    export GOPATH=$HOME/go
    export PATH="$GOPATH/bin:$PATH:/usr/local/go/bin"
```

Run this to load the bash profile:

```
    source ~/.bash_profile
```

Now if you run this:

```
    go version
```

You should see this output:

```
    go version go1.2.2 linux/amd64
```

Mount the EBS volume and then make directories to hold the tar/bag
files we're going to download.

```
    sudo mkdir /mnt/apt
    sudo mount /dev/xvdc /mnt/apt
    sudo mkdir /mnt/apt/data
    sudo mkdir /mnt/apt/logs
    sudo mkdir /mnt/apt/replication
    sudo mkdir /mnt/apt/restore
    sudo chown ubuntu.ubuntu /mnt/apt/data/
    sudo chown ubuntu.ubuntu /mnt/apt/logs/
    sudo chown ubuntu.ubuntu /mnt/apt/restore/
    sudo chown ubuntu.ubuntu /mnt/apt/replication/
```

Install the go packages we'll need:

```
    go get github.com/crowdmob/goamz/aws
    go get github.com/crowdmob/goamz/s3
    go get github.com/nu7hatch/gouuid
    go get github.com/rakyll/magicmime
    go get github.com/APTrust/bagins
    go get github.com/APTrust/bagman
    go get github.com/op/go-logging
```

You can also copy a version of bagman from your local machine, like
so:

```
    scp -r ~/go/src/github.com/APTrust/bagman/ ubuntu@apt-util:go/src/github.com/APTrust/
```

Note that this assumes you have an entry in your local ~/.ssh/config
like this:

```
    Host apt-util
        User ubuntu
        Port 22
{'fraction_non_alphanumeric': 0.09382303839732888, 'fraction_numerical': 0.014023372287145243, 'mean_word_length': 3.1090534979423867, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2185629', 'n_tokens_mistral': 2230, 'n_tokens_neox': 1963, 'n_words': 738}