<commit_before>---
layout: post
title: Dataframes
---

I have been playing a lot lately with [Julia](http://julialang.org/).
In the past I have used python and R a lot for my data analysis tasks. I always tend to have some cheatsheet in front of me for the basic dataframe operations in pandas and R. Its hard to make that context switch in the syntax from one language to another.
With Julia, my brain was just not ready to accept another set of syntax.
So I thought a quick reference comparing the basic dataframe manipulation syntax for all 3 languages would be nice.
Most of my data tasks start with some ETL tasks on dataframes. So this is more of a reference for my use in future but can be useful to others too who face the same situation.

I took this awesome tutorial by [<NAME>](http://www.gregreda.com/) on ["Working with DataFrames"](http://www.gregreda.com/2013/10/26/working-with-pandas-dataframes/) and tried to port the example to all R and Julia.

All the code from this post can be found at this [github repo dataframes-compare](https://github.com/ajkl/dataframes-cheatsheet)

We start with the [MovieLens 100k dataset](http://grouplens.org/datasets/movielens/). 
From the README you can see :

This data set consists of:
	* 100,000 ratings (1-5) from 943 users on 1682 movies. 
	* Each user has rated at least 20 movies. 
	* Simple demographic info for the users (age, gender, occupation, zip)

## Data import
Lets start with loading the data into our dataframe. The methods to read the data from a flat file are pretty simple and straight forward in all 3, the only tricky part is when adding column names in Julia. But there are ways to get around.

#### Julia

{% highlight julia %}
using DataFrames
u_col_names=[symbol("user_id"), symbol("age"), symbol("sex"), symbol("occupation"), symbol("zip_code")]
{% endhighlight %}

Another way to do the same without adding each entry as a symbol. Thanks to [Jubobs](http://stackoverflow.com/users/2541573/jubobs) from this [stackoverflow post](http://stackoverflow.com/questions/27629206/why-does-this-list-comprehension-return-an-arrayany-1-instead-of-an-arraysymb/27629609#27629609) for the suggestion.

{% highlight julia %}
col_names=["user_id", "age", "sex", "occupation", "zip_code"]
u_col_names=map(symbol, col_names)
users = DataFrames.readtable("data/ml-100k/u.user", separator='|', header=false, names=u_col_names)
{% endhighlight %}

There is no way right now to load selective columns from a file. The next IO version will hopefully have that mechanism.
[Stackoverflow question selecting-columns-while-importing-data-with-dataframes-readtable](http://stackoverflow.com/questions/27628366/selecting-columns-while-importing-data-with-dataframes-readtable)

#### Python

{% highlight python %}
import pandas as pd

u_col_names = ['user_id', 'age', 'sex', 'occuptation', 'zip_code']
users = pd.read_csv('data/ml-100k/u.user', sep = '|', names=u_col_names)<commit_msg>bullet fix for data summary<commit_after>---
layout: post
title: Dataframes
---

I have been playing a lot lately with [Julia](http://julialang.org/).
In the past I have used python and R a lot for my data analysis tasks. I always tend to have some cheatsheet in front of me for the basic dataframe operations in pandas and R. Its hard to make that context switch in the syntax from one language to another.
With Julia, my brain was just not ready to accept another set of syntax.
So I thought a quick reference comparing the basic dataframe manipulation syntax for all 3 languages would be nice.
Most of my data tasks start with some ETL tasks on dataframes. So this is more of a reference for my use in future but can be useful to others too who face the same situation.

I took this awesome tutorial by [<NAME>](http://www.gregreda.com/) on ["Working with DataFrames"](http://www.gregreda.com/2013/10/26/working-with-pandas-dataframes/) and tried to port the example to all R and Julia.

All the code from this post can be found at this [github repo dataframes-compare](https://github.com/ajkl/dataframes-cheatsheet)

We start with the [MovieLens 100k dataset](http://grouplens.org/datasets/movielens/). 
From the README you can see :

This data set consists of:
* 100,000 ratings (1-5) from 943 users on 1682 movies. 
* Each user has rated at least 20 movies. 
* Simple demographic info for the users (age, gender, occupation, zip)

## Data import
Lets start with loading the data into our dataframe. The methods to read the data from a flat file are pretty simple and straight forward in all 3, the only tricky part is when adding column names in Julia. But there are ways to get around.

#### Julia

{% highlight julia %}
using DataFrames
u_col_names=[symbol("user_id"), symbol("age"), symbol("sex"), symbol("occupation"), symbol("zip_code")]
{% endhighlight %}

Another way to do the same without adding each entry as a symbol. Thanks to [Jubobs](http://stackoverflow.com/users/2541573/jubobs) from this [stackoverflow post](http://stackoverflow.com/questions/27629206/why-does-this-list-comprehension-return-an-arrayany-1-instead-of-an-arraysymb/27629609#27629609) for the suggestion.

{% highlight julia %}
col_names=["user_id", "age", "sex", "occupation", "zip_code"]
u_col_names=map(symbol, col_names)
users = DataFrames.readtable("data/ml-100k/u.user", separator='|', header=false, names=u_col_names)
{% endhighlight %}

There is no way right now to load selective columns from a file. The next IO version will hopefully have that mechanism.
[Stackoverflow question selecting-columns-while-importing-data-with-dataframes-readtable](http://stackoverflow.com/questions/27628366/selecting-columns-while-importing-data-with-dataframes-readtable)

#### Python

{% highlight python %}
import pandas as pd

u_col_names = ['user_id', 'age', 'sex', 'occuptation', 'zip_code']
users = pd.read_csv('data/ml-100k/u.user', sep = '|', names=u_col_names)
{'fraction_non_alphanumeric': 0.10218978102189781, 'fraction_numerical': 0.025802070955695127, 'mean_word_length': 4.75390625, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 2, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5371959', 'n_tokens_mistral': 1870, 'n_tokens_neox': 1700, 'n_words': 711}