<jupyter_start><jupyter_text>TITANIC DATA SET DISCOVERY   Is the story a young poor guy   and a rich girl.    And he died because she said there was no room left on the floating piece of wood (a.ka. selfish).  Fortunately, we have the ship's [manifest](data/train.csv) of nearly every single passenger. But using standard python libraries to peak and poke that dataset is kinda of a miserable experience. Fortunately, we have pandas.  http://pandas.pydata.org/ Pandas provides an abstraction to load in data, manipulate it, and export your changes.<jupyter_code>import pandas as pd
from __future__ import division
import math
from matplotlib import pyplot as plt
import numpy as np
plt.style.use('ggplot')
#Put graphs within the notebook
%matplotlib inline
df = pd.read_csv("data/train.csv")
df.head()
df.tail()
type(df)
df.info()
df.dtypes
df.describe()
df.shape
df.columns
df["Name"]
my_famous_passenger = df[df["Name"] == "Guggenheim, Mr. Benjamin"]
my_famous_passenger
df["Age"].mean()
df["Fare"].sum()
df["Fare"].describe()
my_rich_passenger = df[df["Fare"] > 512]
my_rich_passenger
age_df = df["Age"]
age_df.mean?
cols = list(df.columns.values)
print(cols)
cols[0]
cols[-1]
cols = ['Survived', 'Pclass', 'Name', 'Sex', 'Age', 'SibSp', 'Parch', 'Ticket', 'Fare', 'Cabin', 'Embarked']
print(cols)
new_df = df[cols]
new_df.head()
new_df = df[["Sex","Age"]]
new_df.head()
df_of_women = df[df["Sex"] == "female"]
df_of_women.shape
df_of_men = df[df["Sex"] == "male"]
df_of_men.shape
#Exercise
df_Pclass1 = df[df["Pclass"] == 1]
df_Pclass1.shape
df_Pclass2 = df[df["Pclass"] == 2]
df_Pclass2.shape
df_Pclass3 = df[df["Pclass"] == 3]
df_Pclass3.shape
df["FamilySize"] = df["SibSp"] + df["Parch"]
df.head()
df["Age"].hist()
df["Age"].dropna().hist(bins=10, range=(0,80))
df["Fare"].hist(range=(0,150))
plt.scatter(df["Fare"], df["Pclass"])
d = {'one' : np.random.rand(50), 'two': np.random.rand(50)}
df_scrap = pd.DataFrame(d)
df_scrap.plot(style=['ro','bx'])
import statsmodels.api as sm
import pylab as pl
#Logistic regression time
print(df.columns)
new_df = df
def gender_to_numeric(gender):
    if gender == "male":
        return 0
    elif gender == "female":
        return 1
    else:
        return -1
new_df["Sex"] = new_df["Sex"].apply(gender_to_numeric)
new_df["Sex"].head()
new_df = new_df[["Survived","Age","Sex","Pclass"]]
new_df = new_df.dropna()
new_df.head()
train_cols = new_df.columns[1:]
print(train_cols)
logit = sm.Logit(new_df["Survived"], new_df[train_cols])
#Fit the model
result = logit.fit()
print(result.summary())<jupyter_output>Logit Regression Results                           
==============================================================================
Dep. Variable:               Survived   No. Observations:                  714
Model:                          Logit   Df Residuals:                      711
Method:                           MLE   Df Model:                            2
Date:                Sun, 12 Jun 2016   Pseudo R-squ.:                  0.2944
Time:                        10:44:11   Log-Likelihood:                -340.26
converged:                       True   LL-Null:                       -482.26
                                        LLR p-value:                 2.141e-62
==============================================================================
                 coef    std err          z      P>|z|      [95.0% Conf. Int.]
------------------------------------------------------------------------------
Age           -0.0035      0.004     -0.777      0.4[...]
{'fraction_non_alphanumeric': 0.20226308345120225, 'fraction_numerical': 0.025176803394625177, 'mean_word_length': 2.7221052631578946, 'pattern_counts': {'":': 2, '<': 4, '<?xml version=': 0, '>': 6, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '437548', 'license': 'CC0-1.0', 'n_tokens_mistral': 1246, 'n_tokens_neox': 1141, 'n_words': 297, 'path': 'titanic_data_exploration/class_walkthrough.ipynb', 'repo_name': 'gaiusjaugustus/2016-06-12_SoftwareCarpentry', 'size': '3535'}