<jupyter_start><jupyter_text>Data Description and ExplorationIn this Module, we'll cover- describe- groupby- split-apply-combine- cross tabs and pivot tables- plotting<jupyter_code>import pandas as pd
import numpy as np

%matplotlib inline<jupyter_output><empty_output><jupyter_text>Let's get started by reading in the data.<jupyter_code>df = pd.read_csv("/Users/matthewgee/Buildings_Violations_sample_50000_clean.csv")
df.describe()
pd.crosstab(df.violation_status,df.department_bureau, margins=True)<jupyter_output><empty_output><jupyter_text>GroupbyOften we want to examine difference among groups based on categorical values. For this `groupby` is incredibly valuable<jupyter_code>df[['department_bureau','longitude','latitude']].groupby("department_bureau").mean()<jupyter_output><empty_output><jupyter_text>Groupby and Plotting! Often we wan to view an aggregation we've generated. How dow we plot groupby objects?<jupyter_code>df.columns
df[['violation_date','id']].groupby(['violation_date']).count().head()
grouped = df[['violation_date','id']].groupby(['violation_date']).count()
grouped.plot(figsize=(15,5))<jupyter_output><empty_output><jupyter_text>Notice the dates. Something looks suspicious. Have we converted them to datetime?<jupyter_code>df.dtypes<jupyter_output><empty_output><jupyter_text>NOOOO!!!!Let's fix that.<jupyter_code>df.violation_date = pd.to_datetime(df.violation_date)
df.violation_status_date = pd.to_datetime(df.violation_status_date)
df.violation_last_modified_date = pd.to_datetime(df.violation_last_modified_date)
grouped = df[['violation_date','id']].groupby(['violation_date']).count()
grouped.plot(figsize=(15,5))<jupyter_output><empty_output><jupyter_text>Now let's narrow the window that we're looking at. It turns out that having a datetime index to the series can be incredibly helpful.<jupyter_code>grouped['2011-01']
grouped['2011-01'].plot(figsize=(15,5))
grouped['2011':'2012'].plot(figsize=(15,5))<jupyter_output><empty_output><jupyter_text>Now let's try to create a multi-line plot. First we need to do a multi-level groupby.<jupyter_code>df1 = df[df['violation_status']=='OPEN']
grouped_open = df1[['violation_date','violation_status','inspection_category','id']].groupby(['violation_date','violation_status','inspection_category']).count()
grouped_open.head()<jupyter_output><empty_output><jupyter_text>Now we want to pivot this table so that there is a column for each inspection category. I find this a lot easier to do in pandas when what I'm pivoting is a column rather than an index, so let's reindex<jupyter_code>grouped_open.reset_index(inplace=True)
grouped_open.head()
grouped_open_categ = grouped_open.pivot_table('id','violation_date','inspection_category')
grouped_open_categ.head()
grouped_open_categ.plot(figsize=(15,5))
grouped_open_categ_reverse = pd.melt(grouped_open_categ, id_vars=['violation_date'], value_vars=['COMPLAINT','PERIODIC','PERMIT','REGISTRATION'])
pd.value_counts(grouped_open_categ_reverse.inspection_category)<jupyter_output><empty_output><jupyter_text>Let's say we wanted to go ahead and aggregate counts by week or month, not just by day. We can use the handy resample command.<jupyter_code>avg_monthly = grouped.resample('M', how='mean')
avg_monthly.plot()
total_monthly = grouped.resample('M', how='sum')
total_monthly.plot()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12697938452345384, 'fraction_numerical': 0.01254855094114132, 'mean_word_length': 9.941176470588236, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 41, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '382506', 'license': 'CC-BY-3.0', 'n_tokens_mistral': 1185, 'n_tokens_neox': 1129, 'n_words': 221, 'path': 'notebooks/09a_Data_Description_and_Exploration_in_Python.ipynb', 'repo_name': 'dssg/dssg-training-workshop-2015', 'size': '3347'}