<jupyter_start><jupyter_text>Titanic ChallengeUse a real data set from the Titanic passenger log to predict which passengers were most likely to survive the disaster. OverviewThe data has been split into two groups:- training set (train.csv)- test set (test.csv)The training set should be used to build the machine learning models. The test set should be used to see how well your model performs on unseen data. * use the model you trained to predict whether or not they survived the sinking of the Titanic.gender_submission.csv - a set of predictions that assume all and only female passengers survive, as an example of what a submission file should look like.  Import Libraries<jupyter_code>import pandas as pd              # pandas is a dataframe library
import matplotlib.pyplot as plt  # matplotlib.pyplot plots data
import numpy as np               # numpy provides N-dim object support

# do plotting inline instead of a separate window
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Load and Review data<jupyter_code>df = pd.read_csv("./data/train.csv")       # load training data
df.shape
df.head(5)
df.tail(5)<jupyter_output><empty_output><jupyter_text>Definition of features| Variable | Definition | Key ||----------|----------|----------||survival|Survival|0 = No, 1 = Yes||pclass|Ticket class|1 = 1st, 2 = 2nd, 3 = 3rd||sex|Sex|||Age|Age in years|||sibsp| of siblings / spouses aboard the Titanic|||parch| of parents / children aboard the Titanic|||ticket|Ticket number|||fare|Passenger fare|||cabin|Cabin number|||embarked|Port of Embarkation|C = Cherbourg, Q = Queenstown, S = Southampton| Variable Notes- pclass: A proxy for socio-economic status (SES)    * 1st = Upper    * 2nd = Middle    * 3rd = Lower- age: Age is fractional if less than 1. If the age is estimated, is it in the form of xx.5- sibsp: The dataset defines family relations in this way...- Sibling = brother, sister, stepbrother, stepsister- Spouse = husband, wife (mistresses and fiancés were ignored)- parch: The dataset defines family relations in this way...- Parent = mother, father- Child = daughter, son, stepdaughter, stepson- Some children travelled only with a nanny, therefore parch=0 for them.  Check for null Values<jupyter_code>df.isnull().values.any()<jupyter_output><empty_output><jupyter_text>If the data has missing values, they will become NaNs in the Numpy arrays generated by the vectorizor so lets get rid of them<jupyter_code>df.replace('?', 0)
df.fillna( 0, inplace = True )
def plot_corr(df, size=11):
    """
    Function Plots a graphical correlation matrix for each pair of columns in the dataframe.
    Imput:
        df: pandas DataFrame
        size: vertical and horizontal size of plot
        
    Dislays:
        matrix of corelation betewwn columns.
    """
    corr = df.corr()     # data frame correlation function
    fig, ax = plt.subplots(figsize=(size, size))
    ax.matshow(corr)    # color code the rectangles by corelation value
    plt.xticks(range(len(corr.columns)), corr.columns) # draw x tick marks
    plt.yticks(range(len(corr.columns)), corr.columns) # draw y tick marks

plot_corr(df)
df.corr()<jupyter_output><empty_output><jupyter_text>When corelation columns are found, it is advised that they are deletedDelete Name field as it will not help with anything for predictionDelete Cabin as only 19.6% of data is providedDelete Ticket<jupyter_code>del df['Name']
del df['Cabin']
del df['Ticket']
df.head()<jupyter_output><empty_output><jupyter_text>Molding Data Check Data Types<jupyter_code>df.head()<jupyter_output><empty_output><jupyter_text>Change male to 1, female to 0<jupyter_code>sex_map = { "male" : 1, "female" : 0}
df['Sex'] = df['Sex'].map(sex_map)<jupyter_output><empty_output><jupyter_text>Embarked: C = Cherbourg, Q = Queenstown, S = SouthamptonTODO: check if prediction is better when mapping the embarked to a number<jupyter_code>df.head()<jupyter_output><empty_output><jupyter_text>Check true/false ration in Survived column<jupyter_code>num_survived = len(df.loc[df['Survived'] == 1])
num_not_survived = len(df.loc[df['Survived'] == 0])
print("Number of people who survived:          {0}  ({1:2.2f}%)".format(num_survived,     (num_survived/     (num_survived+num_not_survived))*100))
print("Number of people who did not survived:  {0}  ({1:2.2f}%)".format(num_not_survived, (num_not_survived/ (num_survived+num_not_survived))*100))

plot_corr(df)
df.corr()<jupyter_output><empty_output><jupyter_text>Hidden Missing Values Impute Age with the mean age<jupyter_code>from sklearn.preprocessing import Imputer

# Impute with mean all 0 readings in Age
imp = Imputer(missing_values=0, strategy="mean", axis=0)
df["Age"] = imp.fit_transform(df[["Age"]]).ravel()<jupyter_output><empty_output><jupyter_text>Splitting the data70% testing, 30% training<jupyter_code>from sklearn.cross_validation import train_test_split

feature_col_names = ["Pclass", "Sex", "Age", "SibSp", "Parch", "Fare"]
predicted_class_name = ["Survived"]

x = df[feature_col_names].values          # predictor feature columns (6 x m)
y = df[predicted_class_name].values       # predicted class (1 = true, 0 = false) comumn (1 X m)
split_test_size = 0.30

x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=split_test_size, random_state=42)
                # test_size = 0.3 is 30%, 42 is the answer to everything (seed for splitting)<jupyter_output><empty_output><jupyter_text>Check to ensure we have the desired 70% train, 30% test split of the data<jupyter_code>print("{0:0.2f}% in training set".format((len(x_train)/len(df.index))*100))
print("{0:0.2f}% in test set".format((len(x_test)/len(df.index))*100))
print("Original True  : {0} ({1:0.2f}%)".format(len(df.loc[df['Survived'] == 1]), (len(df.loc[df['Survived'] == 1])/len(df.index)) * 100.0))
print("Original False : {0} ({1:0.2f}%)".format(len(df.loc[df['Survived'] == 0]), (len(df.loc[df['Survived'] == 0])/len(df.index)) * 100.0))
print("")
print("Training True  : {0} ({1:0.2f}%)".format(len(y_train[y_train[:] == 1]), (len(y_train[y_train[:] == 1])/len(y_train) * 100.0)))
print("Training False : {0} ({1:0.2f}%)".format(len(y_train[y_train[:] == 0]), (len(y_train[y_train[:] == 0])/len(y_train) * 100.0)))
print("")
print("Test True      : {0} ({1:0.2f}%)".format(len(y_test[y_test[:] == 1]), (len(y_test[y_test[:] == 1])/len(y_test) * 100.0)))
print("Test False     : {0} ({1:0.2f}%)".format(len(y_test[y_test[:] == 0]), (len(y_test[y_test[:] == 0])/len(y_test) * 100.0)))<jupyter_output>Original True  : 342 (38.38%)
Original False : 549 (61.62%)

Training True  : 231 (37.08%)
Training False : 392 (62.92%)

Test True      : 111 (41.42%)
Test False     : 157 (58.58%)<jupyter_text>Post-split Data Preparation  Hidden Missing Values<jupyter_code>df.head(10)
print("# rws in dataframe {0}".format(len(df)))
print("# rows missing Age: {0}".format(len(df.loc[df['Age'] == 0]))) # not missing anymore
print("# rows missing Pclass: {0}".format(len(df.loc[df['Pclass'] == 0])))
print("# rows missing Fare: {0}".format(len(df.loc[df['Fare'] == 0]))) # ignore, only 15 entries<jupyter_output># rws in dataframe 891
# rows missing Age: 0
# rows missing Pclass: 0
# rows missing Fare: 15<jupyter_text>Training Initial Algorithm - Naive Bayes<jupyter_code>from sklearn.naive_bayes import GaussianNB

# create Gausian Naive Bayes model object and train it with the data
nb_model = GaussianNB()

nb_model.fit(x_train, y_train.ravel())<jupyter_output><empty_output><jupyter_text>Performance on Training Data<jupyter_code># predict values using the training data
nb_predict_train = nb_model.predict(x_train)

# import the preformance metrics library
from sklearn import metrics

# Acuracy
print("Accuracy: {0:0.4f}".format(metrics.accuracy_score(y_train, nb_predict_train)))
print()<jupyter_output>Accuracy: 0.7945<jupyter_text>Performance on Testing Data<jupyter_code># predict values using the training data
nb_predict_test = nb_model.predict(x_test)

# Acuracy
print("Accuracy: {0:0.4f}".format(metrics.accuracy_score(y_test, nb_predict_test)))
print()<jupyter_output>Accuracy: 0.7910<jupyter_text>Metrics<jupyter_code>print("Confusin Matrix")
print("{0}".format(metrics.confusion_matrix(y_test, nb_predict_test)))
print("")

print("Classification Report")
print(metrics.classification_report(y_test, nb_predict_test))<jupyter_output>Confusin Matrix
[[130  27]
 [ 29  82]]

Classification Report
             precision    recall  f1-score   support

          0       0.82      0.83      0.82       157
          1       0.75      0.74      0.75       111

avg / total       0.79      0.79      0.79       268<jupyter_text>Random ForestThe difference between train data prediction and test data prediction is too big -> model trained too well<jupyter_code>from sklearn.ensemble import RandomForestClassifier
rf_model = RandomForestClassifier(random_state = 42)    # create random forest object
rf_model.fit(x_train, y_train.ravel())<jupyter_output><empty_output><jupyter_text>Predict Training data<jupyter_code>rf_predict_train = rf_model.predict(x_train)
# training accuracy
print("Accuracy: {0:0.4f}".format(metrics.accuracy_score(y_train, rf_predict_train)))<jupyter_output>Accuracy: 0.9631<jupyter_text>Predict Test data<jupyter_code>rf_predict_test = rf_model.predict(x_test)
# training accuracy
print("Accuracy: {0:0.4f}".format(metrics.accuracy_score(y_test, rf_predict_test)))
print("Confusin Matrix")
print("{0}".format(metrics.confusion_matrix(y_test, rf_predict_test)))
print("")

print("Classification Report")
print(metrics.classification_report(y_test, rf_predict_test))<jupyter_output>Confusin Matrix
[[132  25]
 [ 38  73]]

Classification Report
             precision    recall  f1-score   support

          0       0.78      0.84      0.81       157
          1       0.74      0.66      0.70       111

avg / total       0.76      0.76      0.76       268<jupyter_text>Logistic Regression<jupyter_code>from sklearn.linear_model import LogisticRegression

lr_model = LogisticRegression(C=0.7, random_state=42)
lr_model.fit(x_train, y_train.ravel())
lr_predict_test = lr_model.predict(x_test)

# training metrics
print("Accuracy: {0:0.4f}".format(metrics.accuracy_score(y_test, lr_predict_test)))
print("Confusin Matrix")
print("{0}".format(metrics.confusion_matrix(y_test, lr_predict_test)))
print("")

print("Classification Report")
print(metrics.classification_report(y_test, lr_predict_test))<jupyter_output>Accuracy: 0.8060
Confusin Matrix
[[138  19]
 [ 33  78]]

Classification Report
             precision    recall  f1-score   support

          0       0.81      0.88      0.84       157
          1       0.80      0.70      0.75       111

avg / total       0.81      0.81      0.80       268<jupyter_text>Setting Regularization parameter<jupyter_code>C_start = 0.1
C_end = 5
C_inc = 0.1

C_values, recall_scores = [], []

C_val = C_start

best_recall_score = 0
while (C_val < C_end):
    C_values.append(C_val)
    lr_model_loop = LogisticRegression(C=C_val, random_state=42)
    lr_model_loop.fit(x_train, y_train.ravel())
    lr_predict_loop_test = lr_model_loop.predict(x_test)
    recall_score = metrics.recall_score(y_test, lr_predict_loop_test)
    recall_scores.append(recall_score)
    if (recall_score > best_recall_score):
        best_recall_score = recall_score
        best_lr_predict_test = lr_predict_loop_test
        
    C_val = C_val + C_inc

best_score_C_val = C_values[recall_scores.index(best_recall_score)]
print("1st max value of {0:.3f} occured at C={1:.3f}".format(best_recall_score, best_score_C_val))

%matplotlib inline 
plt.plot(C_values, recall_scores, "-")
plt.xlabel("C value")
plt.ylabel("recall score")<jupyter_output>1st max value of 0.712 occured at C=5.000<jupyter_text>Logistic Regression with class_weight='balanced'<jupyter_code>C_start = 0.1
C_end = 5
C_inc = 0.1

C_values, recall_scores = [], []

C_val = C_start

best_recall_score = 0
while (C_val < C_end):
    C_values.append(C_val)
    lr_model_loop = LogisticRegression(C=C_val, class_weight='balanced', random_state=42)
    lr_model_loop.fit(x_train, y_train.ravel())
    lr_predict_loop_test = lr_model_loop.predict(x_test)
    recall_score = metrics.recall_score(y_test, lr_predict_loop_test)
    recall_scores.append(recall_score)
    if (recall_score > best_recall_score):
        best_recall_score = recall_score
        best_lr_predict_test = lr_predict_loop_test
        
    C_val = C_val + C_inc

best_score_C_val = C_values[recall_scores.index(best_recall_score)]
print("1st max value of {0:.3f} occured at C={1:.3f}".format(best_recall_score, best_score_C_val))

%matplotlib inline 
plt.plot(C_values, recall_scores, "-")
plt.xlabel("C value")
plt.ylabel("recall score")
#from sklearn.linear_model import LogisticRegression

lr_model = LogisticRegression(class_weight='balanced', C=best_score_C_val, random_state=42)
lr_model.fit(x_train, y_train.ravel())
lr_predict_test = lr_model.predict(x_test)

# training metrics
print("Accuracy: {0:0.4f}".format(metrics.accuracy_score(y_test, lr_predict_test)))
print("Confusin Matrix")
print("{0}".format(metrics.confusion_matrix(y_test, lr_predict_test)))
print("")

print("Classification Report")
print(metrics.classification_report(y_test, lr_predict_test))
print("{0:0.4f}".format(metrics.recall_score(y_test, lr_predict_test)))<jupyter_output>Accuracy: 0.8134
Confusin Matrix
[[128  29]
 [ 21  90]]

Classification Report
             precision    recall  f1-score   support

          0       0.86      0.82      0.84       157
          1       0.76      0.81      0.78       111

avg / total       0.82      0.81      0.81       268

0.8108<jupyter_text>Logistic RegressionCV - Cross Validation<jupyter_code>from sklearn.linear_model import LogisticRegressionCV
lr_cv_model = LogisticRegressionCV(n_jobs=-1, random_state=42, Cs=3, cv=10, refit=False, class_weight="balanced")
lr_cv_model.fit(x_train, y_train.ravel())<jupyter_output><empty_output><jupyter_text>Predict Test data<jupyter_code>lr_cv_predict_test = lr_cv_model.predict(x_test)
# training accuracy
print("Accuracy: {0:0.4f}".format(metrics.accuracy_score(y_test, lr_cv_predict_test)))
print("Confusin Matrix")
print("{0}".format(metrics.confusion_matrix(y_test, lr_cv_predict_test)))
print("")
print("Classification Report")
print(metrics.classification_report(y_test, lr_cv_predict_test))
print("{0:0.4f}".format(metrics.recall_score(y_test, lr_cv_predict_test)))<jupyter_output>Accuracy: 0.8134
Confusin Matrix
[[127  30]
 [ 20  91]]

Classification Report
             precision    recall  f1-score   support

          0       0.86      0.81      0.84       157
          1       0.75      0.82      0.78       111

avg / total       0.82      0.81      0.81       268

0.8198<jupyter_text>Predictions to the test data Use data from test.csv file<jupyter_code>test_df = pd.read_csv("./data/test.csv")       # load test data
test_df.head()<jupyter_output><empty_output><jupyter_text>Clean test data<jupyter_code>test_df.replace('?', 0)
test_df.fillna( 0, inplace = True )

del test_df['Name']
del test_df['Cabin']
del test_df['Ticket']

test_df['Sex'] = test_df['Sex'].map(sex_map)

# Impute with mean all 0 readings in Age
test_df["Age"] = imp.fit_transform(test_df[["Age"]]).ravel()

print("# rws in dataframe {0}".format(len(test_df)))
print("# rows missing Age: {0}".format(len(test_df.loc[df['Age'] == 0]))) # not missing anymore
print("# rows missing Pclass: {0}".format(len(test_df.loc[df['Pclass'] == 0])))
print("# rows missing Fare: {0}".format(len(test_df.loc[df['Fare'] == 0]))) 
test_df.head()
#get queries IDs for later usage
queryIds = test_df['PassengerId']
x = test_df[feature_col_names].values

predictions = lr_cv_model.predict(x)
#-------------------------------------------
# Print predictions to file
#-------------------------------------------

# open file to write Predictions
predictionFile = open('./solutions/prediction.csv', 'w')
predictionFile.write("PassengerId,Survived\n")

for qId, prediction in zip(queryIds, predictions):
    #print("{0}, {1}".format(qId, prediction))
    predictionFile.write("{0}, {1}\n".format(qId, prediction))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13445118178443263, 'fraction_numerical': 0.03563915357010271, 'mean_word_length': 4.63956734124215, 'pattern_counts': {'":': 0, '<': 100, '<?xml version=': 0, '>': 101, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '27', 'dir': 'jupyter-structured-clean-dedup', 'id': '405898', 'license': 'MIT', 'n_tokens_mistral': 6118, 'n_tokens_neox': 5690, 'n_words': 1289, 'path': 'Kaggle/TitanicChallenge/TitanicChallenge.ipynb', 'repo_name': 'MaximilianMihoc/MachineLearning', 'size': '16162'}