<jupyter_start><jupyter_text>Project Titanic Table of ContentsIntroductionData WranglingExploratory Data AnalysisDataPreprocessingModelling  IntroductionThis dataset captures information on the people who travelled on the RMS Titanic. This dataset has be taken from the Kaggle Competition - https://www.kaggle.com/c/titanicIn this challenge, we ask you to complete the analysis of what sorts of people were likely to survive. In particular, we ask you to apply the tools of machine learning to predict which passengers survived the tragedy.<jupyter_code># Use this cell to set up import statements for all of the packages that you plan to use.

%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
from sklearn.preprocessing import MinMaxScaler
from sklearn.ensemble import RandomForestClassifier
from IPython.display import display<jupyter_output><empty_output><jupyter_text>Data Wrangling 1. General Properties 1.1 Load the dataLoad both the training and testing csv file into a dataframe using pandas<jupyter_code>titanicTrain = pd.read_csv('train.csv')
display(titanicTrain.head())

titanicTest = pd.read_csv('test.csv')
display(titanicTest.head())<jupyter_output><empty_output><jupyter_text>1.2 Check contents of columnsHere we check the unique values in each columns to understand more about the data. Also we will check for the data types missing vales, mean, count etc.<jupyter_code>print("Survived :  {}".format(titanicTrain.Survived.unique()))
print("Pclass :  {}".format(titanicTrain.Pclass.unique()))
#print("Name :  {}".format(titanicTrain.Name.unique()))
print("Sex :  {}".format(titanicTrain.Sex.unique()))
print("Age :  {}".format(titanicTrain.Age.unique()))
print("SibSp :  {}".format(titanicTrain.SibSp.unique()))
print("Parch :  {}".format(titanicTrain.Parch.unique()))
#print("Ticket :  {}".format(titanicTrain.Ticket.unique()))
print("Fare :  {}".format(titanicTrain.Fare.unique()))
#print("Cabin :  {}".format(titanicTrain.Cabin.unique()))
print("Embarked :  {}".format(titanicTrain.Embarked.unique()))
#print("PassengerId :  {}".format(titanicTrain.PassengerId.unique()))
# Find count, mean etc
display(titanicTrain.describe())
# DataTypes and null columns
print("Train DataTypes : \n")
print(titanicTrain.info())

print("\n\nTest DataTypes : \n")
print(titanicTest.info())<jupyter_output>Train DataTypes : 

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 891 entries, 0 to 890
Data columns (total 12 columns):
PassengerId    891 non-null int64
Survived       891 non-null int64
Pclass         891 non-null int64
Name           891 non-null object
Sex            891 non-null object
Age            714 non-null float64
SibSp          891 non-null int64
Parch          891 non-null int64
Ticket         891 non-null object
Fare           891 non-null float64
Cabin          204 non-null object
Embarked       889 non-null object
dtypes: float64(2), int64(5), object(5)
memory usage: 83.6+ KB
None


Test DataTypes : 

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 418 entries, 0 to 417
Data columns (total 11 columns):
PassengerId    418 non-null int64
Pclass         418 non-null int64
Name           418 non-null object
Sex            418 non-null object
Age            332 non-null float64
SibSp          418 non-null int64
Parch          418 non-null int64
Ticket         418 non[...]<jupyter_text>Exploratory Data AnalysisHere we will plot graph of count of people who survived and died  for each feature.- Pclass : It is pretty evident that people travelled in 1st class had higher chance of survival compared to lower class.- Sex  : Females had a higher chance of survival than men- Parch : Parch with values 1,2 and 3 seems to have a higher rate of survival than others.- Embarked : Embarked 'C' has a better survival rates than other 2 .- Cabin  : looks like cabin B,C,D,E,F has better survival rates.- Age : people with age less than 20 survived more.etc.<jupyter_code>def barchart(indf, incolumn) : 
    dlt = []
    ilt = []
    for element in indf[incolumn].unique() : 
        x = indf[indf[incolumn] == element]['Survived'].value_counts()
        dlt.append(x)
        ilt.append(element)
    df1 = pd.DataFrame(dlt)
    df1.index = ilt
    df1.columns = ['Dead', 'Survived']
    df1.plot(kind='bar',stacked=False, figsize=(10,3), logy=True)
barchart(titanicTrain, 'Pclass')
barchart(titanicTrain, 'Sex')
barchart(titanicTrain, 'Parch')
barchart(titanicTrain, 'Embarked')
barchart(titanicTrain, 'SibSp')
titanicTrain['Cabin1'] = titanicTrain['Cabin'].str.extract('([A-Za-z])',expand=False)
barchart(titanicTrain, 'Cabin1')
titanicTrain = titanicTrain.drop(['Cabin1'], axis=1)
ax = titanicTrain['Age'].hist(bins = 25)
ax1 = titanicTrain[titanicTrain.Survived == 1]['Age'].hist(bins = 25)
ax = titanicTrain['Fare'].hist(bins = 25)
ax1 = titanicTrain[titanicTrain.Survived == 1]['Fare'].hist(bins = 25)<jupyter_output><empty_output><jupyter_text>DataPreprocessingHere we will be combining both test and train data and do the following :- Remove the columns that are not required for the modelling- Fill the missing values- Do one-hot encoding to convert categorical data into multiple columns- Data Normalization  check for null valuesHere we check the count of null(nan) values in each column of training set.- 177 null values for Age.- 2 in Embarked - 687 for cabin.<jupyter_code>titanicTrain.isnull().sum()<jupyter_output><empty_output><jupyter_text>Here we check the count of null(nan) values in each column of testing set.- 86 null values for Age.- 1 null in Fare.- 327 null in Cabin.<jupyter_code>titanicTest.isnull().sum()<jupyter_output><empty_output><jupyter_text>Combine DatasetsThere are missing values in both training and testing set. So lets combine both the sets, so that we can do data processing on both the sets.<jupyter_code>#Insert column Survived 
titanicTest.insert(1,'Survived',3)


#concatinate test and training set
combinedTitanic =  pd.concat([titanicTest, titanicTrain])
combinedTitanic.isnull().sum()<jupyter_output><empty_output><jupyter_text>Fill Missing Values  Embarked :We have only 2 rows with missing values, So lets update these rows with most common value that is 'S'.<jupyter_code>combinedTitanic.groupby('Embarked').count()['PassengerId']

# Replace missing values with the the most occuring value.
combinedTitanic['Embarked'] = combinedTitanic['Embarked'].fillna('S')<jupyter_output><empty_output><jupyter_text>Fare :Fare seems to vary with that class, sex and the boarding point(embarked). so lets fill the value with median for the corresponding group.<jupyter_code>sns.factorplot(x="Embarked", hue="Sex", y="Fare", col="Pclass", data=combinedTitanic)
#Fare
display(combinedTitanic[combinedTitanic['Fare'].isnull()])
combinedTitanic["Fare"].fillna(combinedTitanic.groupby(['Pclass','Sex','Embarked'])["Fare"].transform("median"), inplace=True)
display(combinedTitanic[combinedTitanic['PassengerId'] == 1044 ])<jupyter_output><empty_output><jupyter_text>Cabin :There are 1014 missing values for cabin, So lets fill a new value,'U'(unknown), for these rows. Also as the Cabin values have the number along with them. We are interested only in cabin type. So we can extract a new column Cabin_type with the first letter of cabin column.<jupyter_code>#Cabin
combinedTitanic['Cabin_Type']=combinedTitanic['Cabin'].str.extract('([A-Za-z])',expand=False)
combinedTitanic['Cabin_Type'] = combinedTitanic['Cabin_Type'].fillna('U')

print(combinedTitanic['Cabin_Type'].unique())<jupyter_output>['U' 'B' 'E' 'A' 'C' 'D' 'F' 'G' 'T']<jupyter_text>Age :Age seems to be greatly depended on sex, marital status etc. So lets derive 'Title' of each person from from the Name column. Then we can fill the missing values using the median of each title.Also Title seems to have a impact on the survival, So let us keep that column for Modelling.<jupyter_code>#Age
combinedTitanic['Title']=combinedTitanic['Name'].str.extract('([A-Za-z]*\.)',expand=False)

titleMap = {'Capt.': 'Crew', 'Col.': 'Crew', 'Dr.': 'Crew', 'Rev.': 'Crew',
'Countess.': 'Mrs', 'Dona.': 'Mrs', 'Lady.': 'Mrs', 'Mrs.': 'Mrs', 'Ms.' : 'Mrs',
'Don.': 'Mr', 'Jonkheer.': 'Mr', 'Major.': 'Mr', 'Mr.': 'Mr', 'Sir.' : 'Mr',
'Miss.': 'Miss', 'Mlle.': 'Miss', 'Mme.': 'Miss',
'Master.' : 'Master'}
combinedTitanic['Title'] = combinedTitanic['Title'].map(titleMap)
print(combinedTitanic.groupby('Title')['Age'].median())

combinedTitanic["Age"].fillna(combinedTitanic.groupby("Title")["Age"].transform("median"), inplace=True)

sns.countplot(x='Title',hue='Survived',data=combinedTitanic[combinedTitanic['Survived'] != 3])
combinedTitanic.isnull().sum()<jupyter_output><empty_output><jupyter_text>Remove Columns that are not relevant for modelling.<jupyter_code>#Remove columns that we  are not using
combinedTitanic = combinedTitanic.drop(['PassengerId','Name','Ticket','Cabin'], axis=1) #,'Cabin'
combinedTitanic.head()<jupyter_output><empty_output><jupyter_text>One-Hot- Encoding<jupyter_code># Do one-hot encoding for all categorical variable
combinedTitanic['Pclass'] = combinedTitanic['Pclass'].map({1 : '1st Class', 2: '2nd Class', 3 : '3rd Class'})
combinedTitanicEncoded = pd.get_dummies(data =  combinedTitanic)
combinedTitanicEncoded.head()<jupyter_output><empty_output><jupyter_text>Normalize Data<jupyter_code>#Normalize all the Numeric variables:
#from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler() 
numericCol = ['Age','SibSp','Parch','Fare']
combinedTitanicEncoded[numericCol] = scaler.fit_transform(combinedTitanicEncoded[numericCol] )


combinedTitanicEncoded.head()<jupyter_output><empty_output><jupyter_text>ModellingHere we will using randomforest for our modelling. Before modelling split the data back into testing and training sets.<jupyter_code>testFeatures = (combinedTitanicEncoded[combinedTitanicEncoded['Survived'] == 3 ]).drop('Survived',axis = 1)
trainFeatures = (combinedTitanicEncoded[combinedTitanicEncoded['Survived'] != 3 ]).drop('Survived',axis = 1)
trainlabel = combinedTitanicEncoded[combinedTitanicEncoded['Survived'] != 3 ]['Survived']

print("Test count : {}\n".format(len(testFeatures.index) ))
print("Train features count : {}\n".format(len(trainFeatures.index) ))
print("Train Label : {}\n".format(len(trainlabel.index) ))

clf = RandomForestClassifier(random_state=0)
clf = clf.fit(trainFeatures, trainlabel)
pred = clf.predict(testFeatures)
print(pred)
TestOut = titanicTest
TestOut['Survived'] = pred
TestOutput = TestOut[['PassengerId','Survived']]
print(TestOutput)
TestOutput.to_csv('Titanic.csv',index = False)<jupyter_output>PassengerId  Survived
0            892         0
1            893         0
2            894         0
3            895         1
4            896         1
5            897         0
6            898         0
7            899         0
8            900         1
9            901         0
10           902         0
11           903         0
12           904         1
13           905         0
14           906         1
15           907         1
16           908         0
17           909         1
18           910         0
19           911         0
20           912         0
21           913         1
22           914         1
23           915         0
24           916         1
25           917         0
26           918         1
27           919         1
28           920         0
29           921         0
..           ...       ...
388         1280         0
389         1281         0
390         1282         0
391         1283         1
392         1284         1
3[...]
{'fraction_non_alphanumeric': 0.1180837289598619, 'fraction_numerical': 0.034182132067328445, 'mean_word_length': 3.9810834049871024, 'pattern_counts': {'":': 0, '<': 60, '<?xml version=': 0, '>': 60, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '202762', 'license': 'MIT', 'n_tokens_mistral': 4046, 'n_tokens_neox': 3698, 'n_words': 1057, 'path': 'python/titanic-survivors/Titanic.ipynb', 'repo_name': 'neethupanackal/ml-projects', 'size': '11585'}