<commit_before>---
layout: post
title: "Spark & Python Notebooks II: key/value RDDs"
author: "<NAME>"
date: "28 May 2015"
comments: true
categories: data-analysis   
tags: data-analysis spark python big-data
---

[Previously](spark-py-notebooks-basics/), we introduced the basics of working with Spark RDDs in Python. 
In this new notebook, we deal with data aggregations and key/value pair RDDs. 

### Instructions  

A good way of using these notebooks is by first cloning [the GitHub repo](https://github.com/jadianes/spark-py-notebooks), and then 
starting your own [IPython notebook](http://ipython.org/notebook.html) in 
**pySpark mode**. For example, if we have a *standalone* Spark installation
running in our `localhost` with a maximum of 6Gb per node assigned to IPython:  

    MASTER="spark://127.0.0.1:7077" SPARK_EXECUTOR_MEMORY="6G" IPYTHON_OPTS="notebook --pylab inline" ~/spark-1.2.1-bin-hadoop2.4/bin/pyspark

Notice that the path to the `pyspark` command will depend on your specific 
installation. So as requirement, you need to have<commit_msg>link fix<commit_after>---
layout: post
title: "Spark & Python Notebooks II: key/value RDDs"
author: "<NAME>"
date: "28 May 2015"
comments: true
categories: data-analysis   
tags: data-analysis spark python big-data
---

[Previously](http://jadianes.me/spark-py-notebooks-basics/), we introduced the basics of working with Spark RDDs in Python. 
In this new notebook, we deal with data aggregations and key/value pair RDDs. 

### Instructions  

A good way of using these notebooks is by first cloning [the GitHub repo](https://github.com/jadianes/spark-py-notebooks), and then 
starting your own [IPython notebook](http://ipython.org/notebook.html) in 
**pySpark mode**. For example, if we have a *standalone* Spark installation
running in our `localhost` with a maximum of 6Gb per node assigned to IPython:  

    MASTER="spark://127.0.0.1:7077" SPARK_EXECUTOR_MEMORY="6G" IPYTHON_OPTS="notebook --pylab inline" ~/spark-1.2.1-bin-hadoop2.4/bin/pyspark

Notice that the path to the `pyspark` command will depend on your specific 
installation. So as requirement, you need to have
{'fraction_non_alphanumeric': 0.10680970149253731, 'fraction_numerical': 0.021455223880597014, 'mean_word_length': 4.416666666666667, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2134237', 'n_tokens_mistral': 739, 'n_tokens_neox': 685, 'n_words': 250}