<commit_before># Instructions for Lab 1

The goal of this lab is for you to experience the pain of working
with data in various degrees of structure.  You will download the
same dataset in multiple formats and compute the same summy values.

The dataset is...

The summary is...

# Step 0: Setup Amazon EC2

Many of the labs in this class will involve running on cloud computing infrastructure.
so we ask that you to run the labs an such an infrastructure from the very beginning.  
The class has recieved Amazon credits so the instructions will be written towards their
services.

If you would like to use another cloud infrastructure, feel free
to do so but you're on your own!


## Sign up


## Create AMI

# Step 1: Joins on JSON

Download:



Perform a join on XXX attributes between the two datasets to compute XXX.


# Step 2: Joins on Protocol Buffers

Protocol Buffers are

# Step 3: Joins on databaes records

Download the following sqlite3 database that is already loaded with data.

Start sqlite using

    sqlite3 <filename>

This should land you in the prompt.  Now compute 

URL to SQLITE documentation and Postgresql documentation.

# Questions
<commit_msg>edited readme<commit_after># Instructions for Lab 1

The goal of this lab is for you to experience the pain of working
with data in various degrees of structure.  

You will the same tweet dataset encoded in multiple ways to compute
some summary information and reflect on the pros and cons of each
encoding.  


# Step 0: Setup Amazon EC2

Many of the labs in this class will involve running on cloud computing infrastructure.
so we ask that you to run the labs an such an infrastructure from the very beginning.  
The class has recieved Amazon credits so the instructions will be written towards their
services.

If you would like to use another cloud infrastructure, feel free
to do so but you're on your own!

## Sign up


## Create AMI

Create an Ubuntu AMI and ensure the following packages are available:

* python 2.7
  * psycopg2
* protocol buffers
* postgresql
* sqlite3
* git

Checkout the class repository

    git clone git@github.com:sirrice/asciiclass.git

Go to lab 1:

    cd asciiclass/labs/lab1


# Step 1: Joins on JSON

Use twitter.data for this step.  twitter.data contains a json-encoded tweet
on each line.


# Step 2: Joins on Protocol Buffers

Protocol Buffers are

# Step 3: Joins on databaes records

Download the following sqlite3 database that is already loaded with data.

Start sqlite using

    sqlite3 <filename>

This should land you in the prompt.  Now compute 

URL to SQLITE documentation and Postgresql documentation.

# Questions

{'fraction_non_alphanumeric': 0.033673855467272036, 'fraction_numerical': 0.00832387438516837, 'mean_word_length': 3.789855072463768, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6496379', 'n_tokens_mistral': 733, 'n_tokens_neox': 687, 'n_words': 412}