<jupyter_start><jupyter_text>Initial exploration of the data setImport statements for pandas and visuals with matplotlib<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
titanicfile = '../data/raw/train.csv'
df = pd.read_csv(titanicfile)<jupyter_output><empty_output><jupyter_text>Looking at the header to see what columns of data are available and using *count()* to see what data is missing from the set<jupyter_code>df.head()
df.count()
df.groupby('Pclass').count()<jupyter_output><empty_output><jupyter_text>We can see many from the "Age" column seem to be missing, so to try to break down the ages, I'll look at the names and specifically the titles.<jupyter_code>df['Name'].head()
df['Title']=df['Name'].apply(lambda x: x.split(',')[1].split('.')[0].strip())<jupyter_output><empty_output><jupyter_text>Using the title was inspired by other analysis of the Titanic data set on Kaggle. I use the *split()* function to extract the title from each name.<jupyter_code>df['Title'].head()<jupyter_output><empty_output><jupyter_text>Using *np.where* we can create a dummy indicator for female<jupyter_code>df["Female"]=np.where(df['Sex']=='female',1,0)
df.head()<jupyter_output><empty_output><jupyter_text>We can use *describe* to see what the average age for each 'Title' is<jupyter_code>df.groupby('Title')['Age'].describe()<jupyter_output><empty_output><jupyter_text>Also looking at the "Embarked" column for patterns in missing data<jupyter_code>df[df['Embarked'].isnull()]
means = df.groupby('Title', as_index=False).agg(
                      {'Age':['mean','std']})
means.columns=['Title','Age_mean',"Age_std"]
means=means.set_index('Title')<jupyter_output><empty_output><jupyter_text>Saving the mean age according to title<jupyter_code>means<jupyter_output><empty_output><jupyter_text>Using conditions and the *apply* function, we choose a random age from an array of available ages to imputate the missing age data according to title<jupyter_code>conditions=[(df['Title']=='Miss'),(df['Title']=='Mrs'),
           (df['Title']=='Mr'),(df['Title']=='Master')]
Miss_ages = df.loc[(df["Title"]=='Miss'),'Age'].tolist()
Miss_ages = [x for x in Miss_ages if str(x) != 'nan']
Mrs_ages = df.loc[(df["Title"]=='Mrs'),'Age'].tolist()
Mrs_ages = [x for x in Mrs_ages if str(x) != 'nan']
Mr_ages = df.loc[(df["Title"]=='Mr'),'Age'].tolist()
Mr_ages = [x for x in Mr_ages if str(x) != 'nan']
Master_ages = df.loc[(df["Title"]=='Master'),'Age'].tolist()
Master_ages = [x for x in Master_ages if str(x) != 'nan']
df['Age_rand']=df['Age']
df.loc[conditions[0],'Age_rand']=df.loc[conditions[0],'Age_rand'].apply(lambda x: np.random.choice( np.array(Miss_ages)))
df.loc[conditions[1],'Age_rand']=df.loc[conditions[1],'Age_rand'].apply(lambda x: np.random.choice( np.array(Mrs_ages)))
df.loc[conditions[2],'Age_rand']=df.loc[conditions[2],'Age_rand'].apply(lambda x: np.random.choice( np.array(Mr_ages)))
df.loc[conditions[3],'Age_rand']=df.loc[conditions[3],'Age_rand'].apply(lambda x: np.random.choice( np.array(Master_ages)))<jupyter_output><empty_output><jupyter_text>We can use describe again to see how the ages differ now that we have filled in ages for the "Mrs" "Miss" "Mr" and "Master" titles<jupyter_code>df.groupby('Title')['Age_rand'].describe()<jupyter_output><empty_output><jupyter_text>Check if there is missing data I missed, and I missed filling in a Dr. title<jupyter_code>df[df['Age_rand'].isnull()]
df.loc[df['PassengerId']==767,"Age_rand"]=means.loc['Dr','Age_mean']<jupyter_output><empty_output><jupyter_text>Create a new column "Age_mod" that chooses the actual age (if known) or the randomly generated age if not<jupyter_code>df['Age_mod']=np.where(df['Age'].isnull(),df['Age_rand'],df['Age'])
df[['Age','Age_mod','Age_rand']].head()<jupyter_output><empty_output><jupyter_text>Make a 'Child' Dummy variable for people under the age of 16 then do a logistic regression to see if the prediction that Pclass, Female, and Child will be significant predictors<jupyter_code>df['Child']=np.where(df["Age_mod"]<16.0,1,0)
import statsmodels.api as sm
predictors = ['Pclass','Female','Child']
X = df[predictors]
y = df['Survived']
    
logit = sm.Logit(y, X)
result = logit.fit()
result.summary()<jupyter_output><empty_output><jupyter_text>Save features to a .csv<jupyter_code>df.columns.tolist()
cols_to_keep = ['PassengerId',
 'Survived',
 'Pclass',
 'Name',
 'Sex',
 'Age',
 'SibSp',
 'Parch',
 'Embarked',
 'Title',
 'Female',
 'Child',
 'Age_mod']
data_out = df[cols_to_keep]
data_out.to_csv('../data/interim/train_addAge_1.0.csv', encoding='utf-8')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.17218256543370106, 'fraction_numerical': 0.0054077438892494055, 'mean_word_length': 7.675422138836773, 'pattern_counts': {'":': 0, '<': 58, '<?xml version=': 0, '>': 57, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '188997', 'license': 'MIT', 'n_tokens_mistral': 1731, 'n_tokens_neox': 1647, 'n_words': 370, 'path': 'notebooks/1.0-crg-exploratory-analysis.ipynb', 'repo_name': 'gettecr/titanic_Kaggle', 'size': '4623'}