<jupyter_start><jupyter_text>Titanic: Machine Learning from Disaster (TensorFlow Linear Classifier)

A comprehensive analysis and machine learning modeling of Kaggle's Titanic dataset.

  Setup Import libraries and upload data<jupyter_code>import numpy as np
import re
import pandas as pd
import matplotlib.pyplot as plt
from google.colab import files
files.upload()<jupyter_output><empty_output><jupyter_text>Construct pandas dataframes from csv files and extract dependent variables (in this case, 'Survived')<jupyter_code>dftrain = pd.read_csv('train.csv')
dftest = pd.read_csv('test.csv')
ytrain_all = dftrain.pop('Survived')
ids = dftest['PassengerId']
ids<jupyter_output><empty_output><jupyter_text>Data  Variable Notes

---

pclass: A proxy for socio-economic status (SES)

1st = Upper

2nd = Middle

3rd = Lower

---

age: Age is fractional if less than 1. If the age is estimated, is it in the form of xx.5

---

sibsp: The dataset defines family relations in this way...

Sibling = brother, sister, stepbrother, stepsister

Spouse = husband, wife (mistresses and fiancés were ignored)

---

parch: The dataset defines family relations in this way...

Parent = mother, father

Child = daughter, son, stepdaughter, stepson

Some children travelled only with a nanny, therefore parch=0 for them.  Initial Plots<jupyter_code>dftrain.head()
dftest.head()
dftrain['Age'].hist(bins=80).set_xlabel('Age')
dftrain['Fare'].hist(bins=50).set_xlabel('Fare')
dftrain['Sex'].value_counts().plot(kind='barh').set(xlabel="Population")
dftrain['Pclass'].value_counts().plot(kind='barh').set(xlabel="Population", ylabel="Class")
dftrain['Embarked'].value_counts().plot(kind='barh').set(xlabel="Population", ylabel="Location Embarked From")<jupyter_output><empty_output><jupyter_text>Key: C = Cherbourg, Q = Queenstown, S = Southampton<jupyter_code>dftrain['SibSp'].value_counts().plot(kind='barh').set(ylabel="No. of Siblings & Spouses Onboard")
dftrain['Parch'].value_counts().plot(kind='barh').set(ylabel="No. of Parents & Children Onboard")
pd.concat([dftrain, ytrain], axis=1).groupby('Sex').Survived.mean().plot(kind='barh').set(xlabel="Survival Rate")
pd.concat([dftrain, ytrain], axis=1).groupby('Pclass').Survived.mean().plot(kind='barh').set(xlabel="Survival Rate", ylabel="Class")<jupyter_output><empty_output><jupyter_text>At a first glance:
*   Median age is 20-30 years
*   Male population almost doubles female
*   Most passengers are in third class, followed by first which is closely followed by second
*   Large portion of population either with no spouses/siblings or no parents/children
*   Women have a far higher survival chance than men
*   Privelage of the wealthy; higher class means higher survival rate





  Feature Engineering

Let's engineer a few additional (non-polynomial) features. Some ideas:
*   What deck are they on? This corresponds to the first letter on their cabin number
*   Are they alone?
*   What is the total number of family members onboard? (Parch + SibSp)
*   What titles do they have in their name?
*   Group age into bins
*   Group fares into bins

Note that adding too many features can lead to high variance, which is not good for accuracy when working with small datasets.<jupyter_code>def get_title(name):
  title = re.search(' ([A-Za-z]+)\.', name)
  if title:
    return title.group(1)
  return np.nan

def eng_features(df):
  if 'Cabin' in df.columns:
    df['Deck'] = df['Cabin'].str.get(0)
  df['Alone'] = (df['SibSp']*df['Parch']==0)*1
  df['Family'] = df['SibSp'] + df['Parch']
  if 'Age' in df.columns:
    df['Age Group'] = pd.cut(df['Age'], bins=[0,12,20,40,120], 
                          labels=['Children','Teenage','Adult','Elder'])
  if 'Fare' in df.columns:
    df['Fare Group'] = pd.cut(df['Fare'], bins=[-1,7.91,14.45,31,1000], 
                            labels=['Low','Medium','High','Very High'])
  if 'Name' in df.columns:
    df['Title'] = df['Name'].apply(get_title)
    df['Title'] = df['Title'].replace(['Lady', 'Countess', 'Capt',
          'Col', 'Don', 'Dr', 'Major', 'Rev', 'Sir', 'Jonkheer', 'Dona'], 'Rare')
    df['Title'] = df['Title'].replace('Mlle', 'Miss')
    df['Title'] = df['Title'].replace('Ms', 'Miss')
    df['Title'] = df['Title'].replace('Mme', 'Mrs')
  return df
dftrain = eng_features(dftrain)
dftest = eng_features(dftest)<jupyter_output><empty_output><jupyter_text>Missing Data

Let's examine the missing data<jupyter_code>dftrain.isnull().sum() + dftest.isnull().sum()<jupyter_output><empty_output><jupyter_text>We need to fill in Age, Fare, Cabin, and Embarked which automatically fill additional columns. For numerical columns, we replace NaN elements with their column's respective median value. For category columns, we introduce a new category for NaN values. Finally, remove Age, Fare, and other unneccessary features.<jupyter_code>def clean_data(df):
  if {'Age', 'Fare'}.issubset(df.columns):
    df['Age'] = df['Age'].fillna(df['Age'].median())
    df['Fare'] = df['Fare'].fillna(df['Fare'].median())
  df['Deck'] = df['Deck'].fillna('Missing')
  df['Embarked'] = df['Embarked'].fillna('Stowaway')
  if {'Name', 'Ticket', 'Cabin'}.issubset(df.columns):
    df = df.drop(columns=['Name', 'Ticket', 'Cabin', 'PassengerId']) # Drop unecessary features
  df = eng_features(df) # re-engineer features
  if {'Age', 'Fare'}.issubset(df.columns):
    df = df.drop(columns=['Age', 'Fare']) # Drop Age & Fare
  return df

dftrain = clean_data(dftrain)
dftest = clean_data(dftest)<jupyter_output><empty_output><jupyter_text>Check that there are no longer any nan elements<jupyter_code>dftrain.isnull().sum() + dftest.isnull().sum()<jupyter_output><empty_output><jupyter_text>Encoding Categorical Data

Proceed to encode categorical data using sklearn's OneHotEncoder<jupyter_code>dfall = pd.concat([dftrain,dftest], keys=['dftrain', 'dftest'])
dfall.xs('dftrain')
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder

# 
def encode_categ(df):
  categorical_columns = ['Pclass','Sex','Embarked','Deck', 'Age Group', 'Fare Group', 'Title']
  # 'passthrough' retains other featuer columns
  ct = ColumnTransformer(transformers=[('encoder', OneHotEncoder(), categorical_columns)], remainder='passthrough')
  M = ct.fit_transform(df)
  return M

Xtrain = encode_categ(dftrain)
Xtest = encode_categ(dftest)
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder

def encode_categ(dftrain, dftest):
  # Concantenate dftrain and dfall to ensure column correspondence
  dfall = pd.concat([dftrain, dftest])
  categorical_columns = ['Pclass','Sex','Embarked','Deck', 'Age Group', 'Fare Group', 'Title']
  # 'passthrough' retains other featuer columns
  ct = ColumnTransformer(transformers=[('encoder', OneHotEncoder(), categorical_columns)], remainder='passthrough')
  dfall_np = ct.fit_transform(dfall)
  Xtrain = dfall_np[:dftrain.shape[0],:]
  Xtest = dfall_np[dftrain.shape[0]:,:]
  return Xtrain, Xtest

Xtrain_all, Xtest = encode_categ(dftrain, dftest)
ytrain.shape<jupyter_output><empty_output><jupyter_text>ModelsNow that we have analyzed and cleaned the data, as well as performed feature analysis, we move on to model selection. There are a variety of machine learning models that may be implemented to predict the survivability outcome, so we will test a number of well known models and compare performance. Let us implement 6 different ML models:

1.   Logistic Regression
2.   K-Nearest Neighbours (KNN) Classifier
3.   Support Vector Machine (SVM)
4.   Naive Bayes
5.   Decision Tree Classification
6.   Random Forest Classification<jupyter_code>from sklearn.model_selection import train_test_split

from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import SVC
from sklearn.naive_bayes import GaussianNB
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier

from sklearn.metrics import accuracy_score

from sklearn.model_selection import GridSearchCV<jupyter_output><empty_output><jupyter_text>We will construct a function that splits and shuffles our training set into a training set and evaluation set. It then trains and evaluates the accuracy of the input model several times over a loop and returns the mean accuracy.<jupyter_code>def mean_accuracy(model, n=100):
  accuracy_arr = np.array([])
  for i in range(n):
    Xtrain, Xeval, ytrain, yeval = train_test_split(Xtrain_all, ytrain_all, test_size = 0.3)
    model.fit(Xtrain, ytrain)
    prediction = model.predict(Xeval)
    accuracy = accuracy_score(prediction, yeval)
    accuracy_arr = np.append(accuracy_arr, accuracy)
  return accuracy_arr.mean()<jupyter_output><empty_output><jupyter_text>Logistic Regression<jupyter_code>model = LogisticRegression()
accuracy_lr = mean_accuracy(model)
print(f'Logistic regression classifier achieves an average accuracy of {round(accuracy_lr,4)*100}%')<jupyter_output>Logistic regression classifier achieves an average accuracy of 83.00999999999999%<jupyter_text>K-Nearest Neighbours<jupyter_code>model = KNeighborsClassifier(n_neighbors = 10)
accuracy_knn = mean_accuracy(model)

print(f'KNN classifier achieves an accuracy of {round(accuracy_knn,4)*100}%')<jupyter_output>KNN classifier achieves an accuracy of 81.17999999999999%<jupyter_text>Support Vector Machine<jupyter_code>model = SVC()
accuracy_svm = mean_accuracy(model)

print(f'SVM classifier achieves an accuracy of {round(accuracy_svm,4)*100}%')<jupyter_output>SVM classifier achieves an accuracy of 83.27%<jupyter_text>Naive Bayes (Gaussian)<jupyter_code>model = GaussianNB()
accuracy_nb = mean_accuracy(model)

print(f'Naive Bayes classifier achieves an accuracy of {round(accuracy_nb,4)*100}%')<jupyter_output>Naive Bayes classifier achieves an accuracy of 76.7%<jupyter_text>Decision Tree<jupyter_code>model = DecisionTreeClassifier()
accuracy_dt = mean_accuracy(model)

print(f'Decision Tree classifier achieves an accuracy of {round(accuracy_dt,4)*100}%')<jupyter_output>Decision Tree classifier achieves an accuracy of 79.86%<jupyter_text>Random Forest Classifier<jupyter_code>model = RandomForestClassifier(criterion='gini', n_estimators=700,
                             min_samples_split=10,min_samples_leaf=1,
                             max_features='auto',oob_score=True,
                             random_state=1,n_jobs=-1)

accuracy_rf = mean_accuracy(model)

print(f'Random Forest classifier achieves an accuracy of {round(accuracy_rf,4)*100}%')<jupyter_output>Random Forest classifier achieves an accuracy of 82.34%<jupyter_text>Model Selection  Model Comparison

Let's compare the models we've tested so far.<jupyter_code>models = pd.DataFrame({
    'Model': ['Logistic Regression', 'KNN', 'SVM',
              'Naive Bayes', 'Decision Tree', 'Random Forest'],
    'Accuracy': [accuracy_lr, accuracy_knn, accuracy_svm,
              accuracy_nb, accuracy_dt, accuracy_rf]})
models.sort_values(by='Accuracy',ascending=False)<jupyter_output><empty_output><jupyter_text>Both logistic regression and SVM achieve roughly the highest accuracy, with random forest following.

The classifiers imported from sklearn are a sort of blackbox with tunable parameters that impact model performance. By employing GridSearchCV, we can further tune this parameters space to ensure the highest accuracy model. On the otherhand, beware of overfitting due to over-tuning parameters.<jupyter_code># LogisticRegression().get_params()
# SVC().get_params()
# RandomForestClassifier().get_params()<jupyter_output><empty_output><jupyter_text>Logistic Regression Parameters Tuning<jupyter_code>model = LogisticRegression()
param_grid = {'C': [1, 10, 50, 100, 200, 300, 1000],
              'fit_intercept': [True, False]}

model_lr = GridSearchCV(model, param_grid = param_grid, cv=5, scoring="accuracy", n_jobs= 4, verbose = 1)

Xtrain, Xeval, ytrain, yeval = train_test_split(Xtrain_all, ytrain_all, test_size = 0.3)
model_lr.fit(Xtrain, ytrain)

# Best parameters
print(model_lr.best_estimator_)

# Best score
print(f'Ideal LR parameters achieve an accuracy of {round(model_lr.best_score_,4)*100}%')
model = LogisticRegression(C=1, class_weight=None, dual=False, fit_intercept=True,
                   intercept_scaling=1, l1_ratio=None, max_iter=100,
                   multi_class='auto', n_jobs=None, penalty='l2',
                   random_state=None, solver='lbfgs', tol=0.0001, verbose=0,
                   warm_start=False)

accuracy_lr_tuned = mean_accuracy(model)
print(f'Fine-Tuned SVM classifier achieves an accuracy of {round(accuracy_lr_tuned,4)*100}%')<jupyter_output>Fine-Tuned SVM classifier achieves an accuracy of 82.52000000000001%<jupyter_text>SVM Parameters Tuning<jupyter_code>model = SVC()
param_grid = {'kernel': ['rbf', 'linear'], 
              'gamma': [ 0.001, 0.01, 0.1, 1],
              'C': [1, 10, 50, 100, 200, 300, 1000]}

model_svm = GridSearchCV(model, param_grid = param_grid, cv=5, scoring="accuracy", n_jobs= 4, verbose = 1)

Xtrain, Xeval, ytrain, yeval = train_test_split(Xtrain_all, ytrain_all, test_size = 0.3)
model_svm.fit(Xtrain, ytrain)

# Best parameters
print(model_svm.best_estimator_)

# Best score
print(f'Ideal SVM parameters achieve an accuracy of {round(model_svm.best_score_,4)*100}%')
model = SVC(C=100, break_ties=False, cache_size=200, class_weight=None, coef0=0.0,
    decision_function_shape='ovr', degree=3, gamma=0.01, kernel='rbf',
    max_iter=-1, probability=False, random_state=None, shrinking=True,
    tol=0.001, verbose=False)

accuracy_svm_tuned = mean_accuracy(model)
print(f'Fine-Tuned SVM classifier achieves an accuracy of {round(accuracy_svm_tuned,4)*100}%')<jupyter_output>Fine-Tuned SVM classifier achieves an accuracy of 82.53%<jupyter_text>Random Forest Parameters Tuning<jupyter_code>model = RandomForestClassifier()
n_estim = range(100, 1000, 100)
param_grid = {'n_estimators' : n_estim}

model_rf = GridSearchCV(model, param_grid = param_grid, cv=5, scoring="accuracy", n_jobs= 4, verbose = 1)

Xtrain, Xeval, ytrain, yeval = train_test_split(Xtrain_all, ytrain_all, test_size = 0.3)
model_rf.fit(Xtrain, ytrain)

# Best parameters
print(model_rf.best_estimator_)

# Best score
print(f'Ideal RF parameters achieve an accuracy of {round(model_rf.best_score_,4)*100}%')
model = RandomForestClassifier(bootstrap=True, ccp_alpha=0.0, class_weight=None,
                       criterion='gini', max_depth=None, max_features='auto',
                       max_leaf_nodes=None, max_samples=None,
                       min_impurity_decrease=0.0, min_impurity_split=None,
                       min_samples_leaf=1, min_samples_split=2,
                       min_weight_fraction_leaf=0.0, n_estimators=900,
                       n_jobs=None, oob_score=False, random_state=None,
                       verbose=0, warm_start=False)

accuracy_rf_tuned = mean_accuracy(model)
print(f'Fine-Tuned Random Forest classifier achieves an accuracy of {round(accuracy_rf_tuned,4)*100}%')<jupyter_output>Fine-Tuned Random Forest classifier achieves an accuracy of 81.67%<jupyter_text>Final Comparison<jupyter_code>models = pd.DataFrame({
    'Hyper-Tuned Model': ['Logistic Regression', 'SVM', 'Random Forest'],
    'Accuracy': [accuracy_lr_tuned, accuracy_svm_tuned, accuracy_rf_tuned]})
models.sort_values(by='Accuracy',ascending=False)<jupyter_output><empty_output><jupyter_text>It seems that hyper-tuned SVM and Logistic Regression have approximately the same performance. For our predictions, we will implement the Logistic Regression classifier.  Final Prediction & Submission

Finally, train our LR model using the entire training set and predict survivability for the test set. Save prediction as a csv file.<jupyter_code>model = LogisticRegression(C=1, class_weight=None, dual=False, fit_intercept=True,
                   intercept_scaling=1, l1_ratio=None, max_iter=100,
                   multi_class='auto', n_jobs=None, penalty='l2',
                   random_state=None, solver='lbfgs', tol=0.0001, verbose=0,
                   warm_start=False)

model.fit(Xtrain_all, ytrain_all)
final_prediction = model.predict(Xtest)
submission = pd.DataFrame(ids)
submission['Survived'] = final_prediction
submission.to_csv('titanicmain_submission.csv', index=False)

files.download('titanicmain_submission.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1171464783632084, 'fraction_numerical': 0.01895105317158549, 'mean_word_length': 5.098638203901362, 'pattern_counts': {'":': 0, '<': 88, '<?xml version=': 0, '>': 88, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 3, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '24', 'dir': 'jupyter-structured-clean-dedup', 'id': '484953', 'license': 'MIT', 'n_tokens_mistral': 5794, 'n_tokens_neox': 5388, 'n_words': 1337, 'path': 'titanic_main.ipynb', 'repo_name': 'jkfids/kaggle-titanic', 'size': '16569'}