<commit_before># setupHadoop
Shell scripts and instructions for setting up Hadoop on a cluster. Intended for use with fresh instances of Ubuntu Server 14.04.1 LTS on Amazon EC2. The instructions below show how to set up a 2 node cluster.

## Creating Instances
First, create two virtual machines using the Amazon Web Interface.

 * Click through Services -> EC2 -> Instances -> Launch Instance.
 * Select "Ubuntu Server 14.04 LTS (HVM), SSD Volume Type".
 * Click "Review and Launch", then click "Launch".
 * On the dialog "Select an existing key pair...", select "Create a new pair"
 * Enter a name for the key pair (I used "cloudTest")
 * Click "Download Key Pair", which will download "cloudTest.pem"
 * Click "Launch Instance"
 * Click "View Instances" to get back to the instances page.
 * Repeat this process a second time, this time using the existing key pair "cloudTest", to create a second instance.

Once you have created two instances, you can name them by clicking on the empty "Name" field. For example, you can name them "Master" and "Slave", to help keep track of which is which. After doing this, you should see a listing like this:

![Instances](http://curran.github.io/images/setupHadoop/instances.png)

## Connecting to an Instance

In a terminal, go to the directory where "cloudTest.pem" is.

`cd ~/Downloads`

Make sure the key file is not visible to others via ssh.

`chmod 400 cloudTest.pem`

If you don't do this, then you'll see this error later:

```
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
```

Find the public IP of your instance in the AWS Web Interface.

Connect to the instance via SSH using the following command.

`ssh -i cloudTest.pem ubuntu@<your IP here>`

For example,

`ssh -i cloudTest.pem ubuntu@172.16.17.32`

Type "yes" at the prompt `Are you sure you want to continue connecting (yes/no)? yes`

Once logged in, you can check what your Ubuntu version is by running

`lsb_release -a`

## Set Up Hadoop

```
sudo apt-get install -y git
git clone https://github.com/curran/setupHadoop.git
cd setupHadoop; sh installHadoop.sh
```

Allow all traffic into the master node by scrolling to the right in the AWS instance listing page, clicking the link in the "Security Groups" column -> "Inbound" tab -> "Edit" button -> "Add Rule" button -> change "Custom TCP Rule" to "All TCP" -> "Save" button

## Control Hadoop

The following commands are defined in `/usr/local/hadoop/bin` and `/usr/local/hadoop/sbin`. They are available as commands to execute because these paths were added to `$PATH` by `setupHadoop.sh`.

Format the file system.

`hdfs namenode -format`

`start-dfs.sh` Start HDFS. This will launch the NameNode and DataNode Hadoop daemons.
`stop-dfs.sh` Stop HDFS.

You can always check to see which daemons are running by executing `jps`. The output will look something like this:

```
$ jps
11296 NameNode
11453 DataNode
11768 Jps
```

# Setting up a cluster

Perhaps the simplest approach to setting up a cluster is to first set up many machines as independent single-node Hadoop clusters, make sure each one is working individually, then shut them all down, reconfigure them such that one is a master and the others are slaves, then restart the cluster. Note that you "start the cluster" by starting HDFS and YARN from the _master node only_. The master node will connect to slave nodes via SSH and start the appropriate processes on each, namely DataNode for HDFS and ResourceManager for YARN.
<commit_msg>Improved shell script<commit_after># setupHadoop
Shell scripts and instructions for setting up Hadoop on a cluster. Intended for use with fresh instances of Ubuntu Server 14.04.1 LTS on Amazon EC2. The instructions below show how to set up a 2 node cluster.

## Creating Instances
First, create two virtual machines using the Amazon Web Interface.

 * Click through Services -> EC2 -> Instances -> Launch Instance.
 * Select "Ubuntu Server 14.04 LTS (HVM), SSD Volume Type".
 * Click "Review and Launch", then click "Launch".
 * On the dialog "Select an existing key pair...", select "Create a new pair"
 * Enter a name for the key pair (I used "cloudTest")
 * Click "Download Key Pair", which will download "cloudTest.pem"
 * Click "Launch Instance"
 * Click "View Instances" to get back to the instances page.
 * Repeat this process a second time, this time using the existing key pair "cloudTest", to create a second instance.

Once you have created two instances, you can name them by clicking on the empty "Name" field. For example, you can name them "Master" and "Slave", to help keep track of which is which. After doing this, you should see a listing like this:

![Instances](http://curran.github.io/images/setupHadoop/instances.png)

## Connecting to an Instance

In a terminal, go to the directory where "cloudTest.pem" is.

`cd ~/Downloads`

Make sure the key file is not visible to others via ssh.

`chmod 400 cloudTest.pem`

If you don't do this, then you'll see this error later:

```
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
```

Find the public IP of your instance in the AWS Web Interface.

Connect to the instance via SSH using the following command.

`ssh -i cloudTest.pem ubuntu@<your IP here>`

For example,

`ssh -i cloudTest.pem ubuntu@172.16.17.32`

Type "yes" at the prompt `Are you sure you want to continue connecting (yes/no)? yes`

Once logged in, you can check what your Ubuntu version is by running

`lsb_release -a`

## Set Up Hadoop

```
sudo apt-get install -y git ; git clone https://github.com/curran/setupHadoop.git ; cd setupHadoop; sh installHadoop.sh
```

Allow all traffic into the master node by scrolling to the right in the AWS instance listing page, clicking the link in the "Security Groups" column -> "Inbound" tab -> "Edit" button -> "Add Rule" button -> change "Custom TCP Rule" to "All TCP" -> "Save" button

## Control Hadoop

The following commands are defined in `/usr/local/hadoop/bin` and `/usr/local/hadoop/sbin`. They are available as commands to execute because these paths were added to `$PATH` by `setupHadoop.sh`.

Format the file system.

`hdfs namenode -format`

`start-dfs.sh` Start HDFS. This will launch the NameNode and DataNode Hadoop daemons.
`stop-dfs.sh` Stop HDFS.

You can always check to see which daemons are running by executing `jps`. The output will look something like this:

```
$ jps
11296 NameNode
11453 DataNode
11768 Jps
```

# Setting up a cluster

Perhaps the simplest approach to setting up a cluster is to first set up many machines as independent single-node Hadoop clusters, make sure each one is working individually, then shut them all down, reconfigure them such that one is a master and the others are slaves, then restart the cluster. Note that you "start the cluster" by starting HDFS and YARN from the _master node only_. The master node will connect to slave nodes via SSH and start the appropriate processes on each, namely DataNode for HDFS and ResourceManager for YARN.

{'fraction_non_alphanumeric': 0.10516093229744729, 'fraction_numerical': 0.010821309655937847, 'mean_word_length': 4.167741935483871, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 21, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5462885', 'n_tokens_mistral': 2201, 'n_tokens_neox': 2033, 'n_words': 1044}