<jupyter_start><jupyter_text>Titanic: Machine Learning from Disaster Challenge: https://www.kaggle.com/c/titanic Data: https://www.kaggle.com/c/titanic/data Task: Analyse and visualise data to make inferences about what factors contributed to a passengers survival Personal Objective: Serve as an introductory project to data analysis and data visulatisation  Date and Python Version: 2020-01-09 and Python 3.7.4<jupyter_code># Import Libraries

import numpy as np
import pandas as pd
from pandas import Series,DataFrame

import matplotlib.pyplot as plt
import seaborn as sns

%matplotlib inline
# Set up csv data file as DataFrame
train_df = pd.read_csv('train.csv')
test_df = pd.read_csv('test.csv')

# Preview data
train_df.head(10)
# All included column values
train_df.columns.values<jupyter_output><empty_output><jupyter_text>survival:    Survival PassengerId: Unique Id of a passenger. pclass:      Ticket class     sex:         Sex     Age:         Age in years     sibsp:        of siblings / spouses aboard the Titanic     parch:        of parents / children aboard the Titanic     ticket:      Ticket number     fare:        Passenger fare     cabin:       Cabin number     embarked:    Port of Embarkation The dataset contains 11 featured variables alongside the target variable 'Survived'.<jupyter_code># Initial overview
train_df.info()
# General descriptive statistics
train_df.describe()<jupyter_output><empty_output><jupyter_text>We can already see that only 38% of the total 891 passengers in the training set survived the accident.<jupyter_code># Let's see what data is missing ('NaN')

total = train_df.isnull().sum().sort_values(ascending=False)
percent_1 = train_df.isnull().sum()/train_df.isnull().count()*100
percent_2 = (round(percent_1,1)).sort_values(ascending=False)
missing_data = pd.concat([total,percent_2],axis=1,keys=['Total','%'])
missing_data.head(5)<jupyter_output><empty_output><jupyter_text>We can see that the 'Cabin' category is lacking substantial amounts of data (77.1%). Moreover, the 'Age' category is lacking a large amount of datapoints too (19.9%). For specific data inference, the lack of datapoints in the 'Age' category presents a significant challenge.   Let's start visualising the data  1. Age, Sex and Pclass<jupyter_code># Overall makeup of data
# Count female vs. male
train_df['Sex'].value_counts()
# Visualise female vs.male 
fig = sns.FacetGrid(train_df, hue="Sex",aspect=4)
fig.map(sns.kdeplot,'Age',shade= True)
oldest = train_df['Age'].max()
fig.set(xlim=(0,oldest))
fig.add_legend()
# Visualise 'Age' vs 'Pclass' (1st/2nd/3rd class)
fig = sns.FacetGrid(train_df, hue="Pclass",aspect=4)
fig.map(sns.kdeplot,'Age',shade= True)
oldest = train_df['Age'].max()
fig.set(xlim=(0,oldest))
fig.add_legend()
generations=[10,20,40,60,80]
sns.lmplot('Age','Survived',hue='Sex',data=train_df,palette='winter',x_bins=generations)
# Visualise overall survived / not survived female vs male dependent on 'Age'
survived = 'survived'
not_survived = 'not survived'
fig, axes = plt.subplots(nrows=1, ncols=2,figsize=(16, 4))
men = train_df[train_df['Sex']=='male']
women = train_df[train_df['Sex']=='female']
ax = sns.distplot(women[women['Survived']==1].Age.dropna(), bins=18, label = survived, ax = axes[0], kde =False)
ax = sns.distplot(women[women['Survived']==0].Age.dropna(), bins=40, label = not_survived, ax = axes[0], kde =False)
ax.legend()
ax.set_title('Female')
ax = sns.distplot(men[men['Survived']==1].Age.dropna(), bins=18, label = survived, ax = axes[1], kde = False)
ax = sns.distplot(men[men['Survived']==0].Age.dropna(), bins=40, label = not_survived, ax = axes[1], kde = False)
ax.legend()
_ = ax.set_title('Male')<jupyter_output><empty_output><jupyter_text>Women show an overall very high chance of survival across all age groups. Men on the other side, seem to have had the highest chance of survival below the age of 5 and between 18 and 35 years old. Overall, men show a much lower chance of survival than women.  2. Embarked, Pclass and Sex<jupyter_code># Visualise port of embarkation and Pclass for females vs males
FacetGrid = sns.FacetGrid(train_df, row='Embarked', height=4, aspect=2)
FacetGrid.map(sns.pointplot,'Pclass','Survived','Sex',palette=None,order=None,hue_order=None )
FacetGrid.add_legend()<jupyter_output><empty_output><jupyter_text>Embarked seems to be correlated with survival rate, depending on 'Sex'. Women that embarked in port Q and S have a higher chance of survial.Men show a higher rate of survival when they embarked in port C.Furthermore, Pclass also seems to be correlated with survival  3. Pclass<jupyter_code># Visualise 'PClass' vs. 'Survived'
sns.factorplot('Pclass','Survived',hue='Sex',data=train_df)<jupyter_output>/opt/anaconda3/lib/python3.7/site-packages/seaborn/categorical.py:3666: UserWarning: The `factorplot` function has been renamed to `catplot`. The original name will be removed in a future release. Please update your code. Note that the default `kind` in `factorplot` (`'point'`) has changed `'strip'` in `catplot`.
  warnings.warn(msg)<jupyter_text>The visualisation clearly shows that a persons Pclass was a decisive factor that contributed either positively or negatively to their survival. 1st class passengers show a much higher chance of survival than 3rd class passengers.<jupyter_code># Visualise 'PClass' vs. 'Survived' vs. 'Age'
# Survived = 0; perished
# Survived = 1; survived
grid = sns.FacetGrid(train_df,col='Survived',row='Pclass',height=2.5,aspect=2)
grid.map(plt.hist,'Age',alpha=.5,bins=20)
grid.add_legend();<jupyter_output><empty_output><jupyter_text>Above visualisation confirms that passengers in 1st class had a high probability of survival, while passengers in 3rd class had a low probability of survival.  4. SibSp and Parch Combine SibSP and Parch to see the total number of relatives a person had on the ship.<jupyter_code># 0 indicates person with at least 1 relative on the ship
# 1 indicates person with no relatives on the ship
data = [train_df, test_df]
for dataset in data:
    dataset['relatives'] = dataset['SibSp'] + dataset['Parch']
    dataset.loc[dataset['relatives'] > 0, 'not_alone'] = 0
    dataset.loc[dataset['relatives'] == 0, 'not_alone'] = 1
    dataset['not_alone'] = dataset['not_alone'].astype(int)
train_df['not_alone'].value_counts()
# Visualise
axes = sns.factorplot('relatives','Survived',data=train_df,aspect = 2.5)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11838006230529595, 'fraction_numerical': 0.017601246105919004, 'mean_word_length': 5.730607966457023, 'pattern_counts': {'":': 0, '<': 32, '<?xml version=': 0, '>': 33, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '336113', 'license': 'MIT', 'n_tokens_mistral': 2239, 'n_tokens_neox': 2080, 'n_words': 638, 'path': 'Titanic_Project_2020_01_09_Python_3.7.4.ipynb', 'repo_name': 'HLS-CR/titanic-project', 'size': '6420'}