<jupyter_start><jupyter_text>Reading the dataThe first step will be to read in the data from data/raw/train.csv into a pandas dataframe.<jupyter_code>train = pd.read_csv('data/train.csv')
test = pd.read_csv('data/test.csv')
combine = [train, test]
train.head()<jupyter_output><empty_output><jupyter_text>Looking for Missing Data and Taking a First LookNext I will look for any missing data that we may have. Since most of the cabin information is missing, I may want to consider dropping that column. However, the letter infront of the cabin number indicates what deck someone was on in the titanic. I belive there may be a strong correlation between deck and survival and fare. However, I will explore that later.<jupyter_code>train.info()
train['Name'].sample(n=20)<jupyter_output><empty_output><jupyter_text>Every passenger seems to have a title attached to them. Next I will make a separate column containing their titles. Titles like mlle and mme are just french equivalents if ms. There are several titles such as Major or Captain that are rare so we can move all of those into their own rare title class.<jupyter_code>train.describe(include=['O'])
train[['Pclass', 'Survived']].groupby('Pclass').mean()
train[['Sex', 'Survived']].groupby('Sex').mean()
train[['SibSp', 'Survived']].groupby('SibSp').mean()
train[['Parch', 'Survived']].groupby('Parch').mean()
g = sns.FacetGrid(train, col='Survived', row='Sex')
g.map(sns.distplot, 'Age', kde=False)
h = sns.FacetGrid(train, col='Survived')
h.map(sns.distplot, 'Age', kde=False)
i = sns.FacetGrid(train, row='Survived', col='Pclass')
i.map(sns.distplot, 'Age', kde=False)<jupyter_output><empty_output><jupyter_text>Dropping and Adding FeaturesWe will be dropping the cabin and the ticket features.<jupyter_code>train = train.drop(['Ticket', 'Cabin'], axis=1)
test = test.drop(['Ticket', 'Cabin'], axis=1)
combine = [train, test]
for dataset in combine:
    dataset['Title'] = dataset['Name'].str.extract(' (\w+)\.', expand=False)
pd.crosstab(train['Title'], train['Sex'])
for dataset in combine:
    dataset['Title'] = dataset['Title'].replace(['Capt', 'Col', 'Don', 'Countess', 'Dr', \
                                                'Jonkheer', 'Lady', 'Major', 'Rev', 'Sir'], 'Rare')
    dataset['Title'] = dataset['Title'].replace('Mlle', 'Miss')
    dataset['Title'] = dataset['Title'].replace('Mme', 'Miss')
    dataset['Title'] = dataset['Title'].replace('Ms', 'Miss')
    
train[['Title', 'Survived']].groupby('Title').mean()
title_mapping = {"Mr": 1, "Miss": 2, "Mrs": 3, "Master": 4, "Rare": 5}
for dataset in combine:
    dataset['Title'] = dataset['Title'].map(title_mapping)
    dataset['Title'] = dataset['Title'].fillna(0)
    
train.head()
train = train.drop(['Name', 'PassengerId'], axis=1)
test = test.drop(['Name', 'PassengerId'], axis=1)
combine = [train, test]
for dataset in combine:
    dataset['Sex'] = dataset['Sex'].map({'female': 1, 'male': 0})
    
train.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.17047747141896435, 'fraction_numerical': 0.004707464694014795, 'mean_word_length': 5.453362255965293, 'pattern_counts': {'":': 5, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '666223', 'license': 'MIT', 'n_tokens_mistral': 993, 'n_tokens_neox': 952, 'n_words': 281, 'path': 'notebook/exploration.ipynb', 'repo_name': 'JamesLoughridge/titanic', 'size': '2974'}