<jupyter_start><jupyter_text>Table of Contents0.1&nbsp;&nbsp;Dataset Description0.1.1&nbsp;&nbsp;Some Notes Regarding Dataset0.2&nbsp;&nbsp;Loading data &amp; Preview0.3&nbsp;&nbsp;Fix missing ages0.4&nbsp;&nbsp;Questions in mind0.5&nbsp;&nbsp;Question 1 - Did passenger class made any difference to his survival?0.6&nbsp;&nbsp;Question 2 - Which gender had more survival?0.7&nbsp;&nbsp;Question 3 - Person travelling with others had more survival possibility?0.8&nbsp;&nbsp;Question 4 - Which age group had a better chance of survival?0.9&nbsp;&nbsp;Question 5 - What was male and female survival per class and by age?0.9.1&nbsp;&nbsp;Male and Female per Pclass0.9.2&nbsp;&nbsp;Male and Female Survival per Pclass, and by Age0.9.3&nbsp;&nbsp;Conclusion0.10&nbsp;&nbsp;Overall Conclusion0.10.1&nbsp;&nbsp;Findings0.10.2&nbsp;&nbsp;Limitations0.10.3&nbsp;&nbsp;Future plans0.11&nbsp;&nbsp;References<jupyter_code>import numpy as np
import pandas as pd

import matplotlib.pyplot as plt
import seaborn as sns

%matplotlib inline
sns.set_style(style="darkgrid")
# plt.rcParams['figure.figsize'] = [12.0, 8.0]   # make plots size, double of the notebook normal
plt.rcParams['figure.figsize'] = [9.0, 6.0]   # make plots size, double of the notebook normal

from IPython.display import display, HTML      # to use display() to always have well formatted html table output<jupyter_output><empty_output><jupyter_text>Dataset Description| Variable Name | Definition                                                          	 ||-------------	|---------------------------------------------------------------------	 || PassengerId 	| A unique ID to each Passenger; **1-891**                               || Survived    	| A boolean variable; **1 - Survived, 0 - Dead**                         || Pclass      	| Ticket Class; **1 - 1st, 2 - 2nd, 3 - 3rd class**                      || Name        	| Passenger Name                                                      	 || Sex         	| Sex of Passenger                                                    	 || Age         	| Age in Years                                                        	 || SibSp       	|  Number of Siblings / Spouses Aboard                                	 || Parch       	| Number of parents / children aboard the titanic                     	 || Ticket      	| Ticket number                                                       	 || Fare        	| Passenger Fare                                                      	 || Cabin       	| Cabin number                                                        	 || Embarked    	| Port of Embarkation; **C - Cherbourg, Q - Queenstown, S - Southampton**| Some Notes Regarding Dataset**Pclass**: A proxy for socio-economic status (SES)1st = Upper2nd = Middle3rd = Lower**Age**: Age is fractional if less than 1. If the age is estimated, is it in the form of xx.5**SibSp**: The dataset defines family relations in this way...Sibling = brother, sister, stepbrother, stepsisterSpouse = husband, wife (mistresses and fiancés were ignored)**Parch**: The dataset defines family relations in this way...Parent = mother, fatherChild = daughter, son, stepdaughter, stepsonSome children travelled only with a nanny, therefore parch=0 for them.*Source: [Kaggle's - Titanic: Machine Learning from Disaster](https://www.kaggle.com/c/titanic/data)*  Loading data & Preview<jupyter_code>titanic_df = pd.read_csv("titanic-data.csv", index_col=["PassengerId"])
titanic_df.head()
titanic_df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 891 entries, 1 to 891
Data columns (total 11 columns):
Survived    891 non-null int64
Pclass      891 non-null int64
Name        891 non-null object
Sex         891 non-null object
Age         714 non-null float64
SibSp       891 non-null int64
Parch       891 non-null int64
Ticket      891 non-null object
Fare        891 non-null float64
Cabin       204 non-null object
Embarked    889 non-null object
dtypes: float64(2), int64(4), object(5)
memory usage: 83.5+ KB<jupyter_text>Dataset summary above shows that there are 891 entries.**However, from above we can also see that we have missing values in - Age, Cabin, Embarked columns.*** **Missing values of Cabin and Embarked will not be fixed - because no questions are based on these factors*** **Missing values of Age will be fixed now - because involved in various questions and analysis below.**  Fix missing agesTo review the data by distributions, and to tackle various questions - we first need to deal with this issue of missing ages.If we assume that the missing ages will be distributed similarly, to the values that are present - then we can substitue values that represent the existing distribution.For this we can replace the missing values with the mean.To have best representative values populated - we will taken mean based on Sex and Pclass.In other words the mean of ages for Sex within the Pclass, and when replacing the missing age, these two factors will be kept in consideration - to use the related mean of ages.<jupyter_code>mean_ages = titanic_df.groupby(['Sex','Pclass'])['Age'].mean()
display(mean_ages)
def replace_nan_age(row):
    if pd.isnull(row['Age']):
        return mean_ages[row['Sex'], row['Pclass']]
    else:
        return row['Age']
    
titanic_df['Age'] = titanic_df.apply(replace_nan_age, axis=1)
titanic_df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 891 entries, 1 to 891
Data columns (total 11 columns):
Survived    891 non-null int64
Pclass      891 non-null int64
Name        891 non-null object
Sex         891 non-null object
Age         891 non-null float64
SibSp       891 non-null int64
Parch       891 non-null int64
Ticket      891 non-null object
Fare        891 non-null float64
Cabin       204 non-null object
Embarked    889 non-null object
dtypes: float64(2), int64(4), object(5)
memory usage: 83.5+ KB<jupyter_text>From the above we can see that the missing ages have been filled.<jupyter_code>titanic_df.describe()
titanic_df.Parch.hist()
plt.xlabel('Parch')
plt.ylabel('Passengers')
plt.title('Number of parents / children aboard')
titanic_df.SibSp.hist()
plt.xlabel('SibSp')
plt.ylabel('Passengers')
plt.title('Number of Siblings / Spouses aboard')<jupyter_output><empty_output><jupyter_text>From the above we notice* Oldest passenger was 80 years old* Youngest passenger was about 5 months old* Average age of passengers was 29.32 - but note this also has missing ages* Mean survival is 0.3838* Max fare charged was $512.33* Maximum number of Siblings / Spouses were 8* Maximum number of Parent / Child were 6  Questions in mind1. Did passenger class made any difference to his survival?2. Which gender had more survival?3. Person travelling with others had more survival possibility?4. Which age group had better chance of survival?5. What was male and female survival per class and by age?  Question 1 - Did passenger class made any difference to his survival?<jupyter_code>## SUBSET DATAFRAME TO JUST THE REQUIRED DATA

survived_plass_df = titanic_df[['Survived', 'Pclass']]          # works - just have to say the columns required
survived_plass_df.head()

## GROUP DATA TO CALCULATE SURVIVED & TOTAL BY PCLASS

## calculate survived by pclass
survived_by_pclass = survived_plass_df.groupby(['Pclass']).sum()
total_by_pclass = survived_plass_df.groupby(['Pclass']).count()

# total are showed as survived - so change to column name Total
total_by_pclass.rename(columns = {'Survived':'Total'}, inplace = True)

# merge separate data into one dataframe
survived_total_by_pclass = pd.merge(survived_by_pclass, total_by_pclass, left_index=True, right_index=True) # merge by index
survived_total_by_pclass
percent_survived = (survived_total_by_pclass['Survived'] / survived_total_by_pclass['Total']) * 100
survived_total_by_pclass['Percentage'] = percent_survived

survived_total_by_pclass
x = survived_total_by_pclass.index.values
ht = survived_total_by_pclass.Total
hs = survived_total_by_pclass.Survived

pht = plt.bar(x, ht)
phs = plt.bar(x, hs)

plt.xticks(x, x)
plt.xlabel('Pclass')
plt.ylabel('Passengers')
plt.title('Survivors by Class')


plt.legend([pht,phs],['Died', 'Survived'])<jupyter_output><empty_output><jupyter_text>**Conclusion**As can be seen from the visualization and also from the dataframe table above - 1st Class passengers had highest rate of survival, then 2nd class passengers, and the least survival rates was of 3rd class passengers. A large number of passengers were travelling in 3rd class (491), but only 24.24% survived.  Question 2 - Which gender had more survival?<jupyter_code>## CALCULATE SURVIVED AND TOTAL BY SEX

# groupby Sex
group_by_sex = titanic_df.groupby('Sex')

# calculate survived by sex
survived_by_sex = group_by_sex['Survived'].sum()
survived_by_sex.name = 'Survived'
display(survived_by_sex)

# calculate total by sex
total_by_sex = group_by_sex['Survived'].size()
total_by_sex.name = 'Total'
display(total_by_sex)

# concat the separate results into one dataframe
survived_total_by_sex = pd.concat([survived_by_sex, total_by_sex], axis=1)
survived_total_by_sex
percent_survived = (survived_total_by_sex['Survived'] / survived_total_by_sex['Total']) * 100
survived_total_by_sex['Percentage'] = percent_survived

survived_total_by_sex<jupyter_output><empty_output><jupyter_text>Now lets visualize it<jupyter_code>x = range(len(survived_total_by_sex.index.values))
ht = survived_total_by_sex.Total
hs = survived_total_by_sex.Survived

pht = plt.bar(x, ht)
phs = plt.bar(x, hs)

plt.xticks(x, survived_total_by_sex.index.values)
plt.xlabel('Sex')
plt.ylabel('Passengers')
plt.title('Survivors by Gender')

plt.legend([pht,phs],['Died', 'Survived'])<jupyter_output><empty_output><jupyter_text>**Conclusion**From the visualization and percentage of survival from the dataframe printout above - we can see that females had very high rate of survival. Female survial rate was 74.3%, and male survival rate was 18.9% - so female survival rate was about 4 times that of males.It can be concluded that females were given preference in rescue operations, and males must have sacrificed themselves to let the females survive.  Question 3 - Person travelling with others had more survival possibility? Lets first reivew the distribution of those who were alone, and those who were in company.<jupyter_code>is_not_alone = (titanic_df.SibSp + titanic_df.Parch) >= 1
passengers_not_alone = titanic_df[is_not_alone]

is_alone = (titanic_df.SibSp + titanic_df.Parch) == 0
passengers_alone = titanic_df[is_alone]

print('Not alone - describe')
display(passengers_not_alone.describe())
print('Alone - describe')
display(passengers_alone.describe())
passengers_not_alone.Age.hist(label='Not alone')
passengers_alone.Age.hist(label='Alone', alpha=0.6)

plt.xlabel('Age')
plt.ylabel('Passengers')
plt.legend(loc='best')
plt.title('Alone & Not Alone Passenger\'s Ages')<jupyter_output><empty_output><jupyter_text>From the above distribution we can see that* Those in age range of 0-10, that is kids, were not alone - which makes sense* There however is one kid age 5 who was alone* There was an 80 year old person also who was alone* 537 passengers were alone, whereas 354 were in company* Except for age group 0-10, for all other age groups, those travelling alone outnumbered those travelling in companyNow lets review these by their survival<jupyter_code>notalone = np.where((titanic_df.SibSp + titanic_df.Parch) >= 1, 'Not Alone', 'Alone')
loneliness_summary = titanic_df.groupby(notalone, as_index=False)['Survived'].agg([np.sum, np.size])
loneliness_summary = loneliness_summary.rename(columns={'sum':'Survived', 'size':'Total'})

loneliness_summary
loneliness_summary['Percent survived'] = (loneliness_summary.Survived / loneliness_summary.Total) * 100

loneliness_summary<jupyter_output><empty_output><jupyter_text>Now lets visualize<jupyter_code>x = range(len(loneliness_summary.index.values))
ht = loneliness_summary.Total
hs = loneliness_summary.Survived

pht = plt.bar(x, ht)
phs = plt.bar(x, hs)

plt.xticks(x, loneliness_summary.index.values)
plt.xlabel('Alone / Not Alone')
plt.ylabel('Passengers')
plt.title('Survivors by Alone / Not Alone')


plt.legend([pht,phs],['Died', 'Survived'])<jupyter_output><empty_output><jupyter_text>**Conclusion**Percentage above and visualizations above clearly indicate that people having company had higher survival rate.  Question 4 - Which age group had a better chance of survival? First lets review gender age distribution<jupyter_code>male_ages = (titanic_df[titanic_df.Sex == 'male'])['Age']
male_ages.describe()
female_ages = (titanic_df[titanic_df.Sex == 'female'])['Age']
female_ages.describe()
male_ages.hist(label='Male')
female_ages.hist(label='Female')

plt.xlabel('Age')
plt.ylabel('Passengers')
plt.title('Male & Female passenger ages')
plt.legend(loc='best')<jupyter_output><empty_output><jupyter_text>From above distribution, we can see that:* For every age group the number of females was less than number of males* The age of oldest female was 63, whereas age of oldest male was 80Now lets do survival analysis by the age group<jupyter_code>def age_group(age):
    if age >= 80:
        return '80-89'
    if age >= 70:
        return '70-79'
    if age >= 60:
        return '60-69'
    if age >= 50:
        return '50-59'
    if age >= 40:
        return '40-49'
    if age >= 30:
        return '30-39'
    if age >= 20:
        return '20-29'
    if age >= 10:
        return '10-19'
    if age >= 0:
        return '0-9'
    
titanic_df['AgeGroup'] = titanic_df.Age.apply(age_group)
titanic_df.head()
age_group_summary = titanic_df.groupby(['AgeGroup'], as_index=False)['Survived'].agg([np.sum, np.size])
age_group_summary = age_group_summary.rename(columns={'sum':'Survived', 'size':'Total'})
age_group_summary
x = range(len(age_group_summary.index.values))
ht = age_group_summary.Total
hs = age_group_summary.Survived

pht = plt.bar(x, ht)
phs = plt.bar(x, hs)

plt.xticks(x, age_group_summary.index.values)
plt.xlabel('Age groups')
plt.ylabel('Passengers')
plt.title('Survivors by Age group')


plt.legend([pht,phs],['Died', 'Survived'])
age_group_summary['SurvivedPercent'] = (age_group_summary.Survived / age_group_summary.Total) * 100
age_group_summary['DiedPercent'] = ((age_group_summary.Total - age_group_summary.Survived) / age_group_summary.Total) * 100
age_group_summary<jupyter_output><empty_output><jupyter_text>From the above visualization and percentages we can see that most survivors were from 20-29 age group.But interestingly survival percentage of 0-9 age group is best - at 61.29%.Also above we have seen that female had better survial rate - so these survial rates must be mix of male and female survival rates - and hence to have better view, the gender aspect should also be taken into consideration.<jupyter_code>sex_agegroup_summary = titanic_df.groupby(['Sex','AgeGroup'], as_index=False)['Survived'].mean()

sex_agegroup_summary
male_agegroup_summary = sex_agegroup_summary[sex_agegroup_summary['Sex'] == 'male']

male_agegroup_summary
female_agegroup_summary = sex_agegroup_summary[sex_agegroup_summary['Sex'] == 'female']

female_agegroup_summary
age_group = titanic_df.AgeGroup.unique()
age_labels = sorted(age_group)
print age_labels
ax = sns.barplot(x='AgeGroup', y='Survived', data=titanic_df, hue='Sex', order=age_labels)
ax.set_title('Survivors by Gender by Age groups')<jupyter_output><empty_output><jupyter_text>**Conclusion**From the proportions above, and the visualization, taking into consideration the gender and age group - it is clearly visible that female and children were given preference in rescue operations by the other male passengers. 0-9 age group both male and female children had very high rate of survival.  Question 5 - What was male and female survival per class and by age?  Male and Female per PclassLets review the males and females, per the passenger classes<jupyter_code>sns.swarmplot(x='Pclass', y='Age', data=titanic_df, hue='Sex', dodge=True).set_title('Male and Female Passenger Ages by Class')<jupyter_output><empty_output><jupyter_text>Above we can see that compared to first and second class, there were large number of passengers in third class.Particularly males were in large number ... by the look of swarm, they were in large number in age 18 to 32.To understand the age distribution of male and female in different class - the better plot is box plot.<jupyter_code>sns.boxplot(x='Pclass', y='Age', data=titanic_df, hue='Sex').set_title('Comparison of Male and Female Passenger Ages by Class')<jupyter_output><empty_output><jupyter_text>From above we can make out that mean age of male and female in 3rd class was less than that of males and females in 2nd and 1st class.Highest mean age of males was in 1st class.But this plot gives only idea of the distribution of ages of males and females per class.Now lets try to understand the survival of male and female, per class  Male and Female Survival per Pclass, and by Age<jupyter_code>def scatter(passengers, marker='o', legend_prefix=''):
    survived = passengers[passengers.Survived == 1]
    died = passengers[passengers.Survived == 0]

    x = survived.Age
    y = survived.Fare
    plt.scatter(x, y, c='blue', alpha=0.5, marker=marker, label=legend_prefix + ' Survived')

    x = died.Age
    y = died.Fare
    plt.scatter(x, y, c='red', alpha=0.5, marker=marker, label=legend_prefix + ' Died')

def scatter_by_class(pclass):
    class_passengers = titanic_df[titanic_df.Pclass == pclass]
    
    male_passengers = class_passengers[class_passengers.Sex == 'male']
    female_passengers = class_passengers[class_passengers.Sex == 'female']
    
    scatter(male_passengers, marker='o', legend_prefix='Male')
    scatter(female_passengers, marker='^', legend_prefix='Female')

    plt.legend(bbox_to_anchor=(0,1), loc='best') # bbox - to move legend out of plot/scatter
    plt.xlabel('Age')
    plt.ylabel('Fare')
    plt.title('Gender survival by Age, for Pclass = ' + str(pclass))
scatter_by_class(1)
scatter_by_class(2)
scatter_by_class(3)<jupyter_output><empty_output><jupyter_text>The above three scatter plots give a view of male and female age and survival in each of the class.But for better clarity and understanding, we can separate the scatter plots for male and female, per class.This is what we will do next.<jupyter_code>def sns_scatter_by_class(pclass):
    fg = sns.FacetGrid(titanic_df[titanic_df['Pclass'] == pclass], 
                      col='Sex',
                      col_order=['male', 'female'],
                      hue='Survived', 
                      hue_kws=dict(marker=['v', '^']), 
                      size=6,
                      palette='Set1')
    fg = (fg.map(plt.scatter, 'Age', 'Fare', edgecolor='w', alpha=0.7, s=80).add_legend())
    plt.subplots_adjust(top=0.9)
    fg.fig.suptitle('Gender survival by Age, for CLASS {}'.format(pclass))

# plotted separately because male and female data in same scatter plot difficult to understand comparitive
sns_scatter_by_class(1)
sns_scatter_by_class(2)
sns_scatter_by_class(3)<jupyter_output><empty_output><jupyter_text>ConclusionFrom the just above scatter plots we have lot of clarity about male female age spread, and survivals.We can notice the following* Females in first and second class were mostly all saved/survived.* In first and second class male and female kids (age group 0-10) almost all survived* In third class also the survival rate of female was higher than of males - but the female survival was less compared the female survival compared to 1st and 2nd class femalesWe can confirm the above observation by following barplot - showing the rate of survival by class, by sex<jupyter_code>sns.barplot(x='Pclass', y='Survived', data=titanic_df, hue='Sex').set_title('Gender Survival by Class')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10137924091412463, 'fraction_numerical': 0.020185241115473672, 'mean_word_length': 4.221287779237845, 'pattern_counts': {'":': 0, '<': 73, '<?xml version=': 0, '>': 84, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 16, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '98452', 'license': 'MIT', 'n_tokens_mistral': 6536, 'n_tokens_neox': 6034, 'n_words': 1953, 'path': 'Analysis of Titanic Dataset.ipynb', 'repo_name': 'vikramsjn/Investigate-Titanic-Dataset', 'size': '19866'}