<commit_before># Lab 5: Hadoop/Elastic-Map-Reduce

*Assigned: Oct 1, 2013*

*Due: Tuesday Oct 8, 2013 12:59PM (just before class)*

The goal of this lab is for you to get experience _running_ Hadoop/MapReduce jobs on AWS, both to understand
the pros and cons of using Hadoop, and as a setup for the next series of labs.  

In this lab, you will work with the [enron email corpus](http://www.cs.cmu.edu/~enron/), which contains the email contents of all high level employees during the [Enron fiasco](http://en.wikipedia.org/wiki/Enron_scandal).  You will compute simple and _slightly more_ complicated statistics.

A sample of the data is below:

````json
{"bccnames": [], "sender": "<EMAIL>", "tonames": [""], "cc": [], "text": "Here is our forecast\n\n ", "recipients": ["<EMAIL>"], "mid": "18782981.1075855378110.JavaMail.evans@thyme", "ctype": "text/plain; charset=us-ascii", "bcc": [], "to": ["<EMAIL>"], "replyto": null, "names": [], "ccnames": [], "date": "2001-05-14 16:39:00-07:00", "folder": "_sent_mail", "sendername": "", "subject":     ""}    
{"bccnames": [], "sender": "<EMAIL>", "tonames": [""], "cc": [], "text": "Traveling to have a business meeting takes the fun out of the trip.  ...", "recipients": ["<EMAIL>"], "mid": "15464986.1075855378456.JavaMail.evans@thyme", "ctype": "text/plain; charset=us-ascii", "bcc": [], "to": ["<EMAIL>"], "replyto": null, "names": [], "ccnames": [], "date": "2001-05-04 13:51:00-07:00", "folder": "_sent_mail", "sendername": "", "subject": "Re:"}
````

The dataset is stored as a set of files, where each file contains all emails sent by one person.  We have put these files on Amazon S3 at: `s3://6885public/enron/*.json`

## Setup

You should create a micro-instance as you did in previous
labs and ssh into your virtual machine (you can also run these commands from your own machine).    Install the following packages.

### Data



[Download](https://s3.amazonaws.com/asciiclass/enron/lay-k.json.gz) and decompress `lay-k.json.gz`, which you will use to test your first mapreduce program:

````bash
gzip -d lay-k.json.gz
````

### Python

You will need to install the following packages:

````bash
pip install mrjob
pip install awscli
````    
    
[mrjob](http://pythonhosted.org/mrjob/) makes managing and deploying map reduce jobs from Python easy.

[awscli](https://aws.amazon.com/cli/) provides convenient command line tools for managing AWS services (e.g., copying data to and from S3)

    aws s3 cp <local file> s3://6885public/YOURDIRECTORY/
    
(6885public is an Amazon bucket we have created for the class.)

### Amazon

[Go to this website](https://6885.signin.aws.amazon.com/console)
and login with the username `6885student` to create your own access
key (we will give you the password on Piazza and in class).

Then navigate to the [IAM
service](https://console.aws.amazon.com/iam/home?#users) and create
a new access key for yourself.  You will need this to interact with
AWS.

(We are going to let everyone use the same AWS account and see well
that works!)

#### Setup for Amazon Command Line Tools (awscli)

Amazon command line tools (awscli) provides simple commands for communicating
with AWS services.  You will use it to list, upload and download data
from their storage service, S3.

[Setup awscli](http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html) by creating a config file containing the following (the region=us-east-1 is important)

    [default]
    aws_access_key_id = <KEY>
    aws_secret_access_key = <your secret>
    region = us-east-1

    [preview]
    emr=true

Now point the environment variable to it:

    export AWS_CONFIG_FILE=<location of config file>

Now using `awscli` should just work.  Try to list your buckets:

    aws s3 ls

#### Setup for mrjob

[Setup mrjob](http://pythonhosted.org/mrjob/guides/emr-quickstart.html#amazon-setup) by creating `~/.mrjob.conf` with:

    runners:
      emr:
        aws_access_key_id: <KEY>
        aws_secret_access_key: <your secret>
        aws_region: us-east-1
        ec2_instance_type: m1.small
        
These will set your mrjob default parameters.  Please use `m1.small` instances.


## Running MapReduce locally

Before running a job on multiple servers, it's useful to debug your map reduce job locally (on your own micro-instance or home machine).

`mr_wordcount.py` contains the following code to count the number of times each term (word) appears
in the email corpus:

````python
import sys
from mrjob.protocol import JSONValueProtocol
from mrjob.job import MRJob
from term_tools import get_terms

class MRWordCount(MRJob):
    INPUT_PROTOCOL = JSONValueProtocol
    OUTPUT_PROTOCOL = JSONValueProtocol

    def mapper(self, key, email):
        for term in get_terms(email['text']):
            yield term, 1

    def reducer(self, term, occurrences):
        yield None, {'term': term, 'count': sum(occurrences)}

if __name__ == '__main__':
    MRWordCount.run()
````


First, run this locally on `lay-k.json`
            
````bash
python mr_wordcount.py -o 'wordcount_test' --no-output './lay-k.json'
````

The output should be in `wordcount_test/` and should look like:

````json
{"count": 33, "term": "aapl"}
{"count": 2, "term": "aarhus"}
{"count": 8, "term": "aaron"}
{"count": 1, "term": "aarp"}
{"count": 1, "term": "ab-initio"}
{"count": 20, "term": "abandon"}
````

<div style="text-align: center; color: red; font-size: 15pt;">
Be cool, test locally before deploying globally (on AWS)!
</div>


## An Important Note!

Our labs are starting to deal with $$$, and the class has a finite budget for AWS usage so please be **VERY CAREFUL**.  Keep the following in mind when deploying on AWS:

* Amazon rounds up to the nearest hour each time you spin up machines.  So if you run 2 separate jobs that spin up 10 machines each and only run 5 minutes each, then it costs 2x10 hours.
* When you run a job, `mrjob` will give it an id (e.g., j-12BOETECEU8). **Record this id!!**
* If you are going to run a bunch of jobs, create a job pool so that subsequent jobs can reuse previously allocated machines.
* **When you are done, ALWAYS TERMINATE YOUR JOB FLOWS**

        mrjob audit-emr-usage    # lists your job flows at the bottom
        mrjob terminate-job-flow <ID OF JOB FLOW><commit_msg>more terminate job instructions<commit_after># Lab 5: Hadoop/Elastic-Map-Reduce

*Assigned: Oct 1, 2013*

*Due: Tuesday Oct 8, 2013 12:59PM (just before class)*

The goal of this lab is for you to get experience _running_ Hadoop/MapReduce jobs on AWS, both to understand
the pros and cons of using Hadoop, and as a setup for the next series of labs.  

In this lab, you will work with the [enron email corpus](http://www.cs.cmu.edu/~enron/), which contains the email contents of all high level employees during the [Enron fiasco](http://en.wikipedia.org/wiki/Enron_scandal).  You will compute simple and _slightly more_ complicated statistics.

A sample of the data is below:

````json
{"bccnames": [], "sender": "<EMAIL>", "tonames": [""], "cc": [], "text": "Here is our forecast\n\n ", "recipients": ["<EMAIL>"], "mid": "18782981.1075855378110.JavaMail.evans@thyme", "ctype": "text/plain; charset=us-ascii", "bcc": [], "to": ["<EMAIL>"], "replyto": null, "names": [], "ccnames": [], "date": "2001-05-14 16:39:00-07:00", "folder": "_sent_mail", "sendername": "", "subject":     ""}    
{"bccnames": [], "sender": "<EMAIL>", "tonames": [""], "cc": [], "text": "Traveling to have a business meeting takes the fun out of the trip.  ...", "recipients": ["<EMAIL>"], "mid": "15464986.1075855378456.JavaMail.evans@thyme", "ctype": "text/plain; charset=us-ascii", "bcc": [], "to": ["<EMAIL>"], "replyto": null, "names": [], "ccnames": [], "date": "2001-05-04 13:51:00-07:00", "folder": "_sent_mail", "sendername": "", "subject": "Re:"}
````

The dataset is stored as a set of files, where each file contains all emails sent by one person.  We have put these files on Amazon S3 at: `s3://6885public/enron/*.json`

## Setup

You should create a micro-instance as you did in previous
labs and ssh into your virtual machine (you can also run these commands from your own machine).    Install the following packages.

### Data



[Download](https://s3.amazonaws.com/asciiclass/enron/lay-k.json.gz) and decompress `lay-k.json.gz`, which you will use to test your first mapreduce program:

````bash
gzip -d lay-k.json.gz
````

### Python

You will need to install the following packages:

````bash
pip install mrjob
pip install awscli
````    
    
[mrjob](http://pythonhosted.org/mrjob/) makes managing and deploying map reduce jobs from Python easy.

[awscli](https://aws.amazon.com/cli/) provides convenient command line tools for managing AWS services (e.g., copying data to and from S3)

    aws s3 cp <local file> s3://6885public/YOURDIRECTORY/
    
(6885public is an Amazon bucket we have created for the class.)

### Amazon

[Go to this website](https://6885.signin.aws.amazon.com/console)
and login with the username `6885student` to create your own access
key (we will give you the password on Piazza and in class).

Then navigate to the [IAM
service](https://console.aws.amazon.com/iam/home?#users) and create
a new access key for yourself.  You will need this to interact with
AWS.

(We are going to let everyone use the same AWS account and see well
that works!)

#### Setup for Amazon Command Line Tools (awscli)

Amazon command line tools (awscli) provides simple commands for communicating
with AWS services.  You will use it to list, upload and download data
from their storage service, S3.

[Setup awscli](http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html) by creating a config file containing the following (the region=us-east-1 is important)

    [default]
    aws_access_key_id = <KEY>
    aws_secret_access_key = <your secret>
    region = us-east-1

    [preview]
    emr=true

Now point the environment variable to it:

    export AWS_CONFIG_FILE=<location of config file>

Now using `awscli` should just work.  Try to list your buckets:

    aws s3 ls

#### Setup for mrjob

[Setup mrjob](http://pythonhosted.org/mrjob/guides/emr-quickstart.html#amazon-setup) by creating `~/.mrjob.conf` with:

    runners:
      emr:
        aws_access_key_id: <KEY>
        aws_secret_access_key: <your secret>
        aws_region: us-east-1
        ec2_instance_type: m1.small
        
These will set your mrjob default parameters.  Please use `m1.small` instances.


## Running MapReduce locally

Before running a job on multiple servers, it's useful to debug your map reduce job locally (on your own micro-instance or home machine).

`mr_wordcount.py` contains the following code to count the number of times each term (word) appears
in the email corpus:

````python
import sys
from mrjob.protocol import JSONValueProtocol
from mrjob.job import MRJob
from term_tools import get_terms

class MRWordCount(MRJob):
    INPUT_PROTOCOL = JSONValueProtocol
    OUTPUT_PROTOCOL = JSONValueProtocol

    def mapper(self, key, email):
        for term in get_terms(email['text']):
            yield term, 1

    def reducer(self, term, occurrences):
        yield None, {'term': term, 'count': sum(occurrences)}

if __name__ == '__main__':
    MRWordCount.run()
````


First, run this locally on `lay-k.json`
            
````bash
python mr_wordcount.py -o 'wordcount_test' --no-output './lay-k.json'
````

The output should be in `wordcount_test/` and should look like:

````json
{"count": 33, "term": "aapl"}
{"count": 2, "term": "aarhus"}
{"count": 8, "term": "aaron"}
{"count": 1, "term": "aarp"}
{"count": 1, "term": "ab-initio"}
{"count": 20, "term": "abandon"}
````

<div style="text-align: center; color: red; font-size: 15pt;">
Be cool, test locally before deploying globally (on AWS)!
</div>


## An Important Note!

Our labs are starting to deal with $$$, and the class has a finite budget for AWS usage so please be **VERY CAREFUL**.  Keep the following in mind when deploying on AWS:

* Amazon rounds up to the nearest hour each time you spin up machines.  So if you run 2 separate jobs that spin up 10 machines each and only run 5 minutes each, then it costs 2x10 hours.
* When you run a job, `mrjob` will give it an id (e.g., j-12BOETECEU8). **Record this id!!**
* If you are going to run a bunch of jobs, create a job pool so that subsequent jobs can reuse previously allocated machines.
* **When you are done, ALWAYS TERMINATE YOUR JOB FLOWS**

        mrjob audit-emr-usage    # lists your job flows at the bottom
        mrjob terminate-job-flow <ID OF JOB FLOW>
        mrjob audit-emr-usage    # jobs should be SHUTTING_DOWN, TERMINATED, COMPLETED
{'fraction_non_alphanumeric': 0.12797031657061655, 'fraction_numerical': 0.023367806110365517, 'mean_word_length': 3.995268138801262, 'pattern_counts': {'":': 92, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 8, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7418578', 'n_tokens_mistral': 4260, 'n_tokens_neox': 3915, 'n_words': 1570}