<jupyter_start><jupyter_text>Author: <NAME> Kaggle Competitionhttps://www.kaggle.com/c/titanic/data Background:The sinking of the Titanic is one of the most infamous shipwrecks in history.On April 15, 1912, during her maiden voyage, the widely considered “unsinkable” RMS Titanic sank after colliding with an iceberg. Unfortunately, there weren’t enough lifeboats for everyone onboard, resulting in the death of 1502 out of 2224 passengers and crew.While there was some element of luck involved in surviving, it seems some groups of people were more likely to survive than others.In this challenge, we ask you to build a predictive model that answers the question: - “what sorts of people were more likely to survive?” using passenger data (ie name, age, gender, socio-economic class, etc). Expected outputs:- Predictions of survivals via binary classification- Accuracy of the model in making the classification<jupyter_code>import os
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import warnings
warnings.filterwarnings('ignore')
current_directory = os.getcwd()
train_dir = current_directory + "/data/train.csv"
test_dir = current_directory + "/data/test.csv"<jupyter_output><empty_output><jupyter_text>Exploratory Data Analysis<jupyter_code>train_set = pd.read_csv(train_dir)
test_set = pd.read_csv(test_dir)
print("The number of passengers in train: " + str(len(train_set)))
print("The number of passengers in test: " + str(len(test_set)))
print("Total number of passengers: " + str(len(train_set) + len(test_set)))
train_set.head()
train_set.tail()
train_set.info()
# Age, Cabin, Embarked has missing data
# filter out all categorical variables and do unique count on them
for col in train_set.columns:
    print(train_set[col].value_counts())
    print("\n")

# PassengerId, Names are all unique and distinct
# There are 342 that survived and 549 that did not survived
# Majority of the passengers are in Class 3, while the least is in Class 2
# There is a majority of males in the train set
# Majority of passsengers have no siblings, parents or children on board
# There are 681 distinct ticket number in train set
# There are 147 distinct cabins in train set
# There are majority are embarking towards Southampton, while the least are going to Queenstown
# To plot distribution for numerical data
train_set.describe()
# Age has a mean age of around 29 with variation of 14 years old
# the youngest passsenger is 0.42 which may be erroneous, and the oldest is 80 years old
# SibSP has min of 0 siblings and max is 8 siblings
# Parch has a min of 0 parents/children and max of 6 children
print("Percentage of Survivors are {}% from the train set".format(round(train_set['Survived'].sum()/len(train_set) * 100, 2)))
num_missing_values = train_set.isnull().sum().sort_values(ascending=False)
# print(num_missing_values)

percentage_missing = round(train_set.isnull().sum()/len(train_set)* 100, 1).sort_values(ascending=False)
# print(percentage_missing)

missing_data = pd.concat([num_missing_values, percentage_missing], axis=1, keys=['Num Missing', '%'])
missing_data.head(5)
sns.pairplot(train_set)
plt.show()
sns.catplot(x="Fare", data=train_set, col="Survived", kind='box')
plt.show()
sns.catplot(x="Age", data=train_set, col="Survived", kind='box')
plt.show()
#  Who is likely to survived?
#  Does Gender make a distinction between survival?
#  How would Age influence survival?
#  Those with lesser siblings, parents or children onboard? Or the opposite since they would care for those with family.
def extractTitleFromName(df):
    df['Title'] = df.Name.apply(lambda name: name.split(',')[1].split('.')[0].strip())
    return df
train_set = extractTitleFromName(train_set)
train_set['Title'].value_counts()
train_set.head()
fig, axes = plt.subplots(nrows=1, ncols=2, figsize=(8,4))
females = train_set.loc[train_set['Sex'] == 'female']
males = train_set.loc[train_set['Sex'] == 'male']

ax = sns.distplot(females[females['Survived'] == 1]['Age'].dropna(), bins=5, label = 'Survived', ax=axes[0], kde=False)
ax = sns.distplot(females[females['Survived'] == 0]['Age'].dropna(), bins=5, label = 'Not Survived', ax=axes[0], kde=False)
ax.legend()
ax.set_title('Female')

ax = sns.distplot(males[males['Survived'] == 1]['Age'].dropna(), bins=5, label = 'Survived', ax=axes[1], kde=False)
ax = sns.distplot(males[males['Survived'] == 0]['Age'].dropna(), bins=5, label = 'Not Survived', ax=axes[1], kde=False)
ax.legend()
_ = ax.set_title('Male')<jupyter_output><empty_output><jupyter_text>Observations: + More Females than Males survived.+ Those Males who survived are of younger age less than 30 years old+ Those Females who survived has a large range but majoirty are between around 18-35.+ The non-survived females are similar in distribution. What distinct them from those who survived?<jupyter_code>fig, axes = plt.subplots(nrows=2, ncols=2, figsize=(15,10))

ax = males[males['Survived'] == 0]['SibSp'].dropna().value_counts().plot(kind='bar', color='orange', label="Not Survived", ax=axes[0][0])
ax = males[males['Survived'] == 1]['SibSp'].dropna().value_counts().plot(kind='bar', label="Survived", ax=axes[0][0])
ax.legend()
ax.set_xlabel('Number of Siblings onboard')
ax.set_title("Distribution of Males with Siblings Survival")
# plt.show()

ax = females[females['Survived'] == 0]['SibSp'].dropna().value_counts().plot(kind='bar', color='orange', label="Not Survived", ax=axes[0][1])
ax = females[females['Survived'] == 1]['SibSp'].dropna().value_counts().plot(kind='bar', label="Survived", ax=axes[0][1])
ax.legend()
ax.set_xlabel('Number of Siblings onboard')
ax.set_title("Distribution of Females with Siblings Survival")
# plt.show()

ax = males[males['Survived'] == 0]['Parch'].dropna().value_counts().plot(kind='bar', color='orange', label="Not Survived", ax=axes[1][0])
ax = males[males['Survived'] == 1]['Parch'].dropna().value_counts().plot(kind='bar', label="Survived", ax=axes[1][0])
ax.legend()
ax.set_title("Distribution of Males with Parents/Children Survival")
ax.set_xlabel('Number of Parents/Children onboard')

ax = females[females['Survived'] == 0]['Parch'].dropna().value_counts().plot(kind='bar', color='orange', label="Not Survived", ax=axes[1][1])
ax = females[females['Survived'] == 1]['Parch'].dropna().value_counts().plot(kind='bar', label="Survived", ax=axes[1][1])
ax.legend()
ax.set_xlabel('Number Parents/Children onboard')
ax.set_title("Distribution of Females with Parents/Children Survival")
plt.show()

# Majority of the males who does not have siblings or parents or children onboard did not survived. 
# What separates those without siblings or parents or children onboard to those that did survived?

train_set.columns
sns.factorplot(x="Pclass", y="Survived", data=train_set, col="Embarked", kind='bar', hue="Sex")
# Those who are in PClass 1 tends to have a highest survival rate except for male embarked to Queenstown
# Those females who are in Pclass 3 tends to have the lowest survival rate among all the classes<jupyter_output><empty_output><jupyter_text>Feature Engineering and Selection | Variable | Definition	| Key| ------------- |:-------------:| -----:|| Survival |	Survival	| 0 = No, 1 = Yes||Pclass	|Ticket class	|1 = 1st, 2 = 2nd, 3 = 3rd||Sex	|Sex	|||Age	|Age in years|	|Sibsp|	 of siblings / spouses aboard the Titanic	|||Parch	| of parents / children aboard the Titanic	|||Ticket	|Ticket number	|||Fare	|Passenger fare	|||Cabin	|Cabin number	|||Embarked	|Port of Embarkation	|C = Cherbourg, Q = Queenstown, S = Southampton|<jupyter_code>train_set.columns<jupyter_output><empty_output><jupyter_text>Remove:+ PassengerId, Ticket since they are rolling number + Name - note: maybe can find more information about their socioeconomical status and background from their names but would need more expert knowledge about the country's and period's  Engineer:+ SibSp and Parch since they are highly collinear    - Number of Family onboard    - Flag for has family onboard or not Select+ PClass + Sex    - to convert into binary values+ Age    - to convert into ordinal values?+ Embarked    - to convert into ordinal values+ Cabin    - to convert into numerical based on Alphabet representing Deck A to G    - https://www.encyclopedia-titanica.org/titanic-deckplans/a-deck.html+ Fare<jupyter_code>def computeFamilyFeatures(df):
    df["Num_Family"] = df["SibSp"] + df["Parch"]

    df.loc[df["Num_Family"] == 0, "Has_Family"] = 0
    df.loc[df["Num_Family"] > 0, "Has_Family"] = 1
#     df.head()
    return df
def fillCabinNan(df):
    df['Cabin'] = df['Cabin'].fillna('U')
    return df
# Map and replace values
def replaceFeaturesValues(df):
    gender = {'male': 0, 'female': 1}
    embarked = {"S": 0, "C": 1, "Q": 2}
    cabin = {"A": 0, "B": 1, "C": 2, "D": 3, "E": 4, "F":5, "G":6, "U": 7}
    
    df["Sex"].replace(gender, inplace=True)
    df["Embarked"].replace(embarked, inplace=True)
    df['Cabin'] = df['Cabin'].str[0].map(cabin)
    df.head()
    
    return df
def normaliseTitles(df):
    titlesMap = {
        "Capt":       "Officer",
        "Col":        "Officer",
        "Major":      "Officer",
        "Jonkheer":   "Nobility",
        "Don":        "Nobility",
        "Sir" :       "Nobility",
        "Dr":         "Officer",
        "Rev":        "Officer",
        "the Countess":"Nobility",
        "Dona":       "Nobility",
        "Mme":        "Mrs",
        "Mlle":       "Miss",
        "Ms":         "Mrs",
        "Mr" :        "Mr",
        "Mrs" :       "Mrs",
        "Miss" :      "Miss",
        "Master" :    "Master",
        "Lady" :      "Nobility"
    }
    
    df['Title'].map(titlesMap)
    
    return df
train_set = computeFamilyFeatures(train_set)
train_set = fillCabinNan(train_set)
train_set = replaceFeaturesValues(train_set)
# extractTitleFromName(train_set) is done in EDA
train_set = normaliseTitles(train_set)


# remove features and keep the following:
train_set = train_set[['Survived', 'Pclass', 'Sex', 'Age', 'Fare', 'Cabin', 'Embarked', 'Num_Family', 'Has_Family', 'Title']]

train_set.head()<jupyter_output><empty_output><jupyter_text>Feature Engineering For Test Set<jupyter_code>test_set.info()
# test_set.Cabin.value_counts()
test_set.head()
num_missing_values = test_set.isnull().sum().sort_values(ascending=False)
# print(num_missing_values)

percentage_missing = round(test_set.isnull().sum()/len(test_set)* 100, 1).sort_values(ascending=False)
# print(percentage_missing)

missing_data = pd.concat([num_missing_values, percentage_missing], axis=1, keys=['Num Missing', '%'])
missing_data.head(5)
# test_set = test.copy()
test_set = computeFamilyFeatures(test_set)
test_set = fillCabinNan(test_set)
test_set = replaceFeaturesValues(test_set)
test_set = extractTitleFromName(test_set)
test_set = normaliseTitles(test_set)

test_set = test_set[['Pclass', 'Sex', 'Age', 'Fare', 'Cabin', 'Embarked', 'Num_Family', 'Has_Family', 'Title']]
test_set.head()<jupyter_output><empty_output><jupyter_text>Model Building and Evaluation<jupyter_code>from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline

from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier

from sklearn.model_selection import RandomizedSearchCV, GridSearchCV, cross_val_score
from sklearn.metrics import auc, accuracy_score, confusion_matrix, roc_curve, precision_score, recall_score, f1_score, auc, precision_recall_curve
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
# Impute for Training Set
age_imputer = SimpleImputer(missing_values=np.nan, strategy='median')
embarked_imputer = SimpleImputer(missing_values=np.nan, strategy='most_frequent')
fare_imputer = SimpleImputer(missing_values=np.nan, strategy='median')
col_transformer = ColumnTransformer(transformers=[
        ('age_imputer', age_imputer, ['Age']), 
        ('embarked_imputer', embarked_imputer, ['Embarked']),
        ('fare_imputer', fare_imputer, ['Fare'])], remainder='drop')

preprocessor = Pipeline(steps=[('col_trans', col_transformer)])
train_set.head()
train_set.columns
X_train = train_set[['Pclass', 'Sex', 'Age', 'Fare', 'Cabin', 'Embarked', 'Num_Family', 'Has_Family']]
y_train = train_set['Survived']
X_train.head()
preprocessor.fit(X_train)
X_train_preprocessed = preprocessor.transform(X_train)
X_train_preprocessed = pd.DataFrame(data=X_train_preprocessed, index=X_train.index, columns=['Age', 'Embarked', 'Fare'])

X_train_preprocessed = pd.concat([X_train_preprocessed, X_train[['Cabin', 'Pclass', 'Sex', 'Num_Family', 'Has_Family']]], axis=1)
X_train_preprocessed = pd.get_dummies(X_train_preprocessed, columns=['Cabin', 'Embarked','Pclass', 'Sex', 'Has_Family'])
X_train_preprocessed
X_test_preprocessed = preprocessor.transform(test_set)
X_test_preprocessed = pd.DataFrame(data=X_test_preprocessed, index=test_set.index, columns=['Age', 'Embarked', 'Fare'])

X_test_preprocessed = pd.concat([X_test_preprocessed, test_set[['Cabin', 'Pclass', 'Sex', 'Num_Family', 'Has_Family']]], axis=1)
X_test_preprocessed = pd.get_dummies(X_test_preprocessed, columns=['Cabin', 'Embarked','Pclass', 'Sex', 'Has_Family'])
X_test_preprocessed
X_test_preprocessed.columns
import xgboost as xgb
est_xgb = Pipeline([('xgb', xgb.XGBClassifier())])

xgb_grid = {'xgb__objective' : 'binary:logistic',
            'xgb__max_depth': [5, 10, 25],
            'xgb__n_estimators': [10, 25, 40, 65, 100], 
            'xgb__seed': [1234]
          }

dtrainMatrix = xgb.DMatrix(data=X_train_preprocessed, label=y_train)                    

cv_results = xgb.cv(dtrain=dtrainMatrix , params=xgb_grid, nfold=10, num_boost_round=10, metrics="error", as_pandas=True)
print("Accuracy: %f" %((1-cv_results["test-error-mean"]).iloc[-1]))

# estimators.append(est_xgb)

# hyperparams.append(xgb_grid)
estimators = []
hyperparams = []
est_lr = Pipeline([('lr', LogisticRegression())])

lr_grid = {'lr__C': [0.01, 0.1, 0.5, 1.0, 2.0],
           'lr__random_state': [1234],
           'lr__solver': ['newton-cg', 'lbfgs', 'liblinear', 'sag', 'saga'],
           'lr__max_iter': [100],
          }

estimators.append(est_lr)

hyperparams.append(lr_grid)
est_lr = Pipeline([('scaler', StandardScaler()), ('pca', PCA()), ('lr', LogisticRegression())])

lr_grid = {'pca__n_components': [2, 3, 5, 7],
           'lr__C': [0.01, 0.1, 0.5, 1.0, 2.0],
           'lr__random_state': [1234],
           'lr__solver': ['newton-cg', 'lbfgs', 'liblinear', 'sag', 'saga'],
           'lr__max_iter': [100],
          }

estimators.append(est_lr)

hyperparams.append(lr_grid)
est_dtc = Pipeline([('scaler', StandardScaler()),('pca', PCA()),('dtc', DecisionTreeClassifier())])


dtc_grid = {
            'pca__n_components': [2, 3, 5, 7],
            'dtc__criterion':['gini', 'entropy'], 
            'dtc__max_depth': [2, 4, 6, 8, 10], 
            'dtc__max_features': ['sqrt', 'log2', None], 
            'dtc__random_state': [1234], 
            'dtc__class_weight': ['balanced',None] 
            }

estimators.append(est_dtc)

hyperparams.append(dtc_grid)
est_rfc = Pipeline([('scaler', StandardScaler()),('pca', PCA()), ('rfc', RandomForestClassifier())])

rfc_grid = {'pca__n_components': [2, 3, 5, 7],
            'rfc__n_estimators': [100], 
            'rfc__criterion': ['gini', 'entropy'], 
            'rfc__max_depth': [2, 4, 6, 8, 10], 
            'rfc__max_features': ['sqrt', 'log2', None],
            'rfc__oob_score' : [False], 
            'rfc__random_state': [1234]
}

estimators.append(est_rfc)

hyperparams.append(rfc_grid)
all_estimators = []
all_scores = []

for index, est in enumerate(estimators):
    estimator = GridSearchCV(est, hyperparams[index], cv=5, scoring='accuracy')
    estimator.fit(X_train_preprocessed, y_train)
    print("For Est", index+1)
    print('Best hyperparameter:', estimator.best_params_)
    all_estimators.append(estimator)

    # Evaluate the four best estimators over 5-fold cross validation on the training data.
    # Print the mean and std of AUC over the 5-folds for each best estimator.
    scores = cross_val_score(estimator.best_estimator_, X_train_preprocessed, y_train, cv=5, scoring='roc_auc')
    print('AUC = {:.3f} mean  +/-  {:.3f} std'.format(scores.mean(), scores.std()))
    print("")
    all_scores.append(scores)
all_scores
performances = pd.DataFrame({'Estimators': all_estimators, 'Scores': all_scores, 'AUC Mean': [score.mean() for score in all_scores], 'AUC Std Dev': [score.std() for score in all_scores]})
performances<jupyter_output><empty_output><jupyter_text>Further Evaluation of Classifier<jupyter_code>estimator = performances['Estimators'][0].best_estimator_
predictions = estimator.predict(X_train_preprocessed)  

cm = confusion_matrix(y_train, predictions, labels=[1, 0])
cm = pd.DataFrame(
    cm, 
    index=['True: Survived', 'True: Not Survived'], 
    columns=['Pred: Survived', 'Pred: Not Survived']
)
cm
precision = precision_score(y_train, predictions)
print("The precision of the classifier is {:.2%}".format(precision))

# recall
recall = recall_score(y_train, predictions)
print("The recall of the classifier is {:.2%}".format(recall))

# recall
f1 = f1_score(y_train, predictions, labels=['Survived'])
print("The f1 score of the classifier is {:.2%}".format(f1))<jupyter_output>The precision of the classifier is 79.15%
The recall of the classifier is 71.05%
The f1 score of the classifier is 74.88%<jupyter_text>IntepretationBased on precision, each prediction of either "Survived", or "Not Survived" would be realised truely about 79.15% of the time.Based on recall, each prediction of "Survived" has about 71.05% being correct.Based on f1 score of the "Survived", the weighted average of the precision and recall is 74.88%<jupyter_code>def drawROC(estimator):
    y_prob = estimator.predict_proba(X_train_preprocessed) [:,1]
    fpr, tpr, threshold = roc_curve(y_train, y_prob)
    roc_auc = auc(fpr, tpr)

    plt.title('Receiver Operating Characteristic')
    plt.plot(fpr, tpr, 'b', label = 'AUC = %0.2f' % roc_auc)
    plt.legend(loc = 'lower right')
    plt.plot([0, 1], [0, 1],'r--')
    plt.xlim([0, 1])
    plt.ylim([0, 1])
    plt.ylabel('True Positive Rate: Sensitivity')
    plt.xlabel('False Positive Rate: (1-Specificity)')
    plt.show()

drawROC(estimator)
def drawPrecisionRecallCurve(y_train, y_prob):
    precision, recall, _ = precision_recall_curve(y_train, y_prob)
    plt.plot(recall, precision, lw=2, label='Survived')

    plt.xlabel("Recall")
    plt.ylabel("Precision")
    plt.legend(loc="best")
    plt.title("Precision vs. Recall Curve")
    plt.show()

drawPrecisionRecallCurve(y_train, y_prob)<jupyter_output><empty_output><jupyter_text>Submission of Predictions to Kaggle<jupyter_code>estimator = performances['Estimators'][0].best_estimator_
predictions = estimator.predict(X_test_preprocessed)
# predictions
test_set = pd.read_csv(test_dir)
submission = pd.DataFrame({'PassengerId': test_set['PassengerId'], 'Survived': predictions})
submission
submission.to_csv('predictions.csv', index = False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14545929018789144, 'fraction_numerical': 0.015553235908141963, 'mean_word_length': 5.14725697786333, 'pattern_counts': {'":': 23, '<': 41, '<?xml version=': 0, '>': 42, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '51858', 'license': 'Unlicense', 'n_tokens_mistral': 6915, 'n_tokens_neox': 6431, 'n_words': 1622, 'path': 'Titanic Survival Prediction.ipynb', 'repo_name': 'yyk93/titantic-survival-prediction', 'size': '19165'}