<jupyter_start><jupyter_text>Working with Table DataHere we will use the python library to work with data that comes to us in the form of a table. That data may be in a spreadsheet, or a table of a database.  Data DictionaryVariable	Definition	Keysurvival	Survival	0 = No, 1 = Yespclass	Ticket class	1 = 1st, 2 = 2nd, 3 = 3rdsex	Sex	Age	Age in years	sibsp	 of siblings / spouses aboard the Titanic	parch	 of parents / children aboard the Titanic	ticket	Ticket number	fare	Passenger fare	cabin	Cabin number	embarked	Port of Embarkation	C = Cherbourg, Q = Queenstown, S = SouthamptonVariable Notespclass: A proxy for socio-economic status (SES)1st = Upper2nd = Middle3rd = Lowerage: Age is fractional if less than 1. If the age is estimated, is it in the form of xx.5sibsp: The dataset defines family relations in this way...Sibling = brother, sister, stepbrother, stepsisterSpouse = husband, wife (mistresses and fiancés were ignored)parch: The dataset defines family relations in this way...Parent = mother, fatherChild = daughter, son, stepdaughter, stepsonSome children travelled only with a nanny, therefore parch=0 for them.<jupyter_code>import numpy as np
print(np.__version__)
import pandas as pd
print(pd.__version__)
from matplotlib import pyplot as plt
import matplotlib
print(matplotlib.__version__)
df = pd.read_csv('http://bit.ly/titanic-train')
df.head()
df.head(10)
df.tail()
df.info()
#how do we take a statistical snapshot?
df.describe()
df.describe??
df.describe(include='all')
# the docs say to use the include keyword and 'O' argument
df.describe(include=['O'])
# How do we see how many people survived?  1=Survived
df['Survived'].value_counts()
df['Embarked'].unique()
df.columns
%matplotlib inline
fig, ax = plt.subplots()
df['Sex'].value_counts().plot.pie(ax=ax)#, #labels=["Died", "Lived"])
ax.set_aspect('equal') # this makes it a circle<jupyter_output><empty_output><jupyter_text>Challenge* Print the value counts of another variable (example: cabin)* Create a pie chart for another variable<jupyter_code># What if we want to see how the people who survived differ from those who deid?
survived_df = df.groupby(['Survived'])
survived_df
list_survived_df=list(survived_df)
len(list_survived_df)
len(list_survived_df[0])
list_survived_df[0][0],list_survived_df[1][0]
list_survived_df[0][1].head()
list_survived_df[1][1].head()
# How do we get a summary of that?
survived_df.count()
#how do we now disaggregate the data to compute a cross tabulation?
by_demo = df.groupby(['Sex', 'Pclass'])
#unstack creates the table
by_demo['Survived'].count()
# Can we put it into a table?
by_demo['Survived'].sum().unstack()
#Let's compute a survival rate
Survival_rate = (by_demo['Survived'].sum()/
                 by_demo['Survived'].count())
Survival_rate.unstack()
Survival_rate.plot.bar()
Survival_rate.unstack().plot.bar()
#what about grouped by sex first?
# can we group?
Survival_rate.unstack().T.plot.bar()
Survival_rate.unstack().T
corr = df.corr()
corr[corr<1]
corr = df.corr().values
corr[corr==1] = np.nan
# Is there a correlation?
fig, ax = plt.subplots()
im = ax.imshow(corr)
fig.colorbar(im)
#What if we want to apply different aggregations to different columns?
#{'key':value} is a Python data structure called a dictionary
df.groupby(['Pclass', 'Sex']).agg( {'Survived': np.sum, 
                                    'Fare': np.mean, 
                                    'Age': np.median})<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11259431224608242, 'fraction_numerical': 0.00812536273940801, 'mean_word_length': 4.902397260273973, 'pattern_counts': {'":': 0, '<': 10, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '437310', 'license': 'MIT', 'n_tokens_mistral': 1187, 'n_tokens_neox': 1121, 'n_words': 381, 'path': 'part2-table-data-analysis.ipynb', 'repo_name': 'mdogy/data-comp-literacy-CEN-3-14-19-workshop', 'size': '3446'}