<commit_before># Setup an AWS GPU Cluster from Stratch

In this documents we give a step-by-step tutorial on how to setup Amazon AWS for
MXNet. In particular, we will address:

- [Use Amazon S3 to host data](#use-amazon-s3-to-host-data)
- [Setup EC2 GPU instance with all dependencies installed](#setup-an-ec2-gpu-instance)
- [Build and Run MXNet on a single machine](#build-and-run-mxnet-on-a-gpu-instance)
- [Setup an EC2 GPU cluster for distributed training](#setup-an-ec2-gpu-cluster)

## Use Amazon S3 to host data

Amazon S3 is distributed data storage, which is quite convenient for host large
scale datasets. In order to S3, we need first to get the
[AWS credentials](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSGettingStartedGuide/AWSCredentials.html)),
which includes a `ACCESS_KEY_ID` and a `SECRET_ACCESS_KEY`.

In order for MXNet to use S3, we only need to set the environment variables `AWS_ACCESS_KEY_ID` and
`AWS_SECRET_ACCESS_KEY` properly. For example, we can add the following two lines in
`~/.bashrc` (replace the strings with the correct ones)

```bash
export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export AWS_SECRET_ACCESS_KEY=<KEY>
```

There are several ways to upload local data to S3. One simple way is using
[s3cmd](http://s3tools.org/s3cmd). For example:

```bash
wget http://webdocs.cs.ualberta.ca/~bx3/data/mnist.zip
unzip mnist.zip && s3cmd put t*-ubyte s3://dmlc/mnist/
```

## Setup an EC2 GPU Instance

MXNet requires the following libraries

- C++ compiler with C++11 suports, such as `gcc >= 4.8`
- `CUDA` (`CUDNN` in optional) for GPU linear algebra
- `BLAS` (cblas, open-blas, atblas, mkl, or others) for CPU linear algebra
- `opencv` for image augmentations
- `curl` and `openssl` for read/write Amazon S3

Installing `CUDA` on EC2 instances needs a little bit effects. Caffe has a nice
[tutorial](https://github.com/BVLC/caffe/wiki/Install-Caffe-on-EC2-from-scratch-(Ubuntu,-CUDA-7,-cuDNN))
on how to install CUDA 7.0 on Ubuntu 14.04 (Note: we tried CUDA 7.5 on Nov 7
2015, but it is problematic.)

The reset can be installed by the package manager. For example, on Ubuntu:

```
sudo apt-get update
sudo apt-get install -y build-essential git libcurl4-openssl-dev libatlas-base-dev libopencv-dev python-numpy
```

We provide a public Amazon Machine Images, [ami-12fd8178](https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#LaunchInstanceWizard:ami=ami-12fd8178), with the above packages installed.


## Build and Run MXNet on a GPU Instance

The following commands build MXNet with CUDA/CUDNN, S3, and distributed
training.

```bash
git clone --recursive https://github.com/dmlc/mxnet
cd mxnet; cp make/config.mk .
echo "USE_CUDA=1" >>config.mk
echo "USE_CUDA_PATH=/usr/local/cuda" >>config.mk
echo "USE_CUDNN=1" >>config.mk
echo "USE_BLAS=atlas" >> config.mk
echo "USE_DIST_KVSTORE = 1" >>config.mk
echo "USE_S3=1" >>config.mk
make -j8
```

Test if every goes well, we train a convolution neural network on MNIST using GPU:

```bash
python tests/python/gpu/test_conv.py
```

If the MNISt data is placed on `s3://dmlc/mnist`, we can let the program read
the S3 data directly:

```bash
sed -i.bak "s!data_dir = 'data'!data_dir = 's3://dmlc/mnist'!" tests/python/gpu/test_conv.py
```

Note: We can use `sudo ln /dev/null /dev/raw1394` to fix the opencv error `libdc1394 error: Failed to initialize libdc1394`.

## Setup an EC2 GPU Cluster

A cluster consists of multiple machines. We can use the machine with MXNet
installed as the root machine for submitting jobs, and then launch several
slaves machine to run the jobs. For example, launch multiple instances using a
AMI, e.g.
[ami-12fd8178](https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#LaunchInstanceWizard:ami=ami-12fd8178),
with dependencies installed. There are two suggestions:

1. Make all slaves' ports are accessible (same for the root) by setting **type: All TCP**,
   **Source: Anywhere** in **Configure Security Group**

2. Use the same `pem` as the root machine to access all slaves machines. And
   then cpy the `pem` file into root machine's `~/.ssh/id_rsa`, it all slaves
   machines are ssh-able from the root.

Now we run the previous CNN on multiple machines. Assume we are on a working
directory of the root machine, such as `~/train`, and MXNet is built as `~/mxnet`.

1. First pack the mxnet python library into this working directory for easy
  synchronization:

  ```bash
  cp -r ~/mxnet/python/mxnet .
  cp ~/mxnet/lib/libmxnet.so mxnet/
  ```

  And then copy the training program:

  ```bash
  cp ~/mxnet/example/image-classification/*.py .
  ```

2. Prepare a host file with all slaves's private IPs. For example, `cat hosts`

  ```bash
  172.30.0.172
  172.30.0.171
  ```

3. Assume there are 2 machines, then train the CNN using 2 workers:<commit_msg>Fix a couple of typos<commit_after># Setup an AWS GPU Cluster from Stratch

In this document we give a step-by-step tutorial on how to set up Amazon AWS for
MXNet. In particular, we will address:

- [Use Amazon S3 to host data](#use-amazon-s3-to-host-data)
- [Setup EC2 GPU instance with all dependencies installed](#setup-an-ec2-gpu-instance)
- [Build and Run MXNet on a single machine](#build-and-run-mxnet-on-a-gpu-instance)
- [Setup an EC2 GPU cluster for distributed training](#setup-an-ec2-gpu-cluster)

## Use Amazon S3 to host data

Amazon S3 is distributed data storage, which is quite convenient for host large
scale datasets. In order to S3, we need first to get the
[AWS credentials](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSGettingStartedGuide/AWSCredentials.html)),
which includes a `ACCESS_KEY_ID` and a `SECRET_ACCESS_KEY`.

In order for MXNet to use S3, we only need to set the environment variables `AWS_ACCESS_KEY_ID` and
`AWS_SECRET_ACCESS_KEY` properly. For example, we can add the following two lines in
`~/.bashrc` (replace the strings with the correct ones)

```bash
export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export AWS_SECRET_ACCESS_KEY=<KEY>
```

There are several ways to upload local data to S3. One simple way is using
[s3cmd](http://s3tools.org/s3cmd). For example:

```bash
wget http://webdocs.cs.ualberta.ca/~bx3/data/mnist.zip
unzip mnist.zip && s3cmd put t*-ubyte s3://dmlc/mnist/
```

## Setup an EC2 GPU Instance

MXNet requires the following libraries

- C++ compiler with C++11 suports, such as `gcc >= 4.8`
- `CUDA` (`CUDNN` in optional) for GPU linear algebra
- `BLAS` (cblas, open-blas, atblas, mkl, or others) for CPU linear algebra
- `opencv` for image augmentations
- `curl` and `openssl` for read/write Amazon S3

Installing `CUDA` on EC2 instances needs a little bit efforts. Caffe has a nice
[tutorial](https://github.com/BVLC/caffe/wiki/Install-Caffe-on-EC2-from-scratch-(Ubuntu,-CUDA-7,-cuDNN))
on how to install CUDA 7.0 on Ubuntu 14.04 (Note: we tried CUDA 7.5 on Nov 7
2015, but it is problematic.)

The rest can be installed by the package manager. For example, on Ubuntu:

```
sudo apt-get update
sudo apt-get install -y build-essential git libcurl4-openssl-dev libatlas-base-dev libopencv-dev python-numpy
```

We provide a public Amazon Machine Images, [ami-12fd8178](https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#LaunchInstanceWizard:ami=ami-12fd8178), with the above packages installed.


## Build and Run MXNet on a GPU Instance

The following commands build MXNet with CUDA/CUDNN, S3, and distributed
training.

```bash
git clone --recursive https://github.com/dmlc/mxnet
cd mxnet; cp make/config.mk .
echo "USE_CUDA=1" >>config.mk
echo "USE_CUDA_PATH=/usr/local/cuda" >>config.mk
echo "USE_CUDNN=1" >>config.mk
echo "USE_BLAS=atlas" >> config.mk
echo "USE_DIST_KVSTORE = 1" >>config.mk
echo "USE_S3=1" >>config.mk
make -j8
```

Test if every goes well, we train a convolution neural network on MNIST using GPU:

```bash
python tests/python/gpu/test_conv.py
```

If the MNISt data is placed on `s3://dmlc/mnist`, we can let the program read
the S3 data directly:

```bash
sed -i.bak "s!data_dir = 'data'!data_dir = 's3://dmlc/mnist'!" tests/python/gpu/test_conv.py
```

Note: We can use `sudo ln /dev/null /dev/raw1394` to fix the opencv error `libdc1394 error: Failed to initialize libdc1394`.

## Setup an EC2 GPU Cluster

A cluster consists of multiple machines. We can use the machine with MXNet
installed as the root machine for submitting jobs, and then launch several
slaves machine to run the jobs. For example, launch multiple instances using a
AMI, e.g.
[ami-12fd8178](https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#LaunchInstanceWizard:ami=ami-12fd8178),
with dependencies installed. There are two suggestions:

1. Make all slaves' ports are accessible (same for the root) by setting **type: All TCP**,
   **Source: Anywhere** in **Configure Security Group**

2. Use the same `pem` as the root machine to access all slaves machine, and
   then copy the `pem` file into root machine's `~/.ssh/id_rsa`, it all slaves
   machines are ssh-able from the root.

Now we run the previous CNN on multiple machines. Assume we are on a working
directory of the root machine, such as `~/train`, and MXNet is built as `~/mxnet`.

1. First pack the mxnet python library into this working directory for easy
  synchronization:

  ```bash
  cp -r ~/mxnet/python/mxnet .
  cp ~/mxnet/lib/libmxnet.so mxnet/
  ```

  And then copy the training program:

  ```bash
  cp ~/mxnet/example/image-classification/*.py .
  ```

2. Prepare a host file with all slaves's private IPs. For example, `cat hosts`

  ```bash
  1192.168.127.12
  172.30.0.171
  ```

3. Assume there are 2 machines, then train the CNN using 2 workers:
{'fraction_non_alphanumeric': 0.10843373493975904, 'fraction_numerical': 0.025031159119235563, 'mean_word_length': 4.230309614340032, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 31, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 4, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '589377', 'n_tokens_mistral': 3439, 'n_tokens_neox': 3190, 'n_words': 1206}