<jupyter_start><jupyter_text>Here, we'll try different machine learning classification algorithms to predict survival in the Titanic dataset (https://www.kaggle.com/c/titanic). First, import the necessary libraries:<jupyter_code>%matplotlib inline

import matplotlib.pyplot as plt
import graphviz
import pandas as pd
import numpy as np
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.preprocessing import Imputer
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier, export_graphviz
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import LinearSVC, SVC
from sklearn.model_selection import cross_val_predict, cross_val_score, GridSearchCV
from sklearn.feature_selection import chi2, SelectKBest
from sklearn.metrics import make_scorer, precision_score, recall_score, f1_score, accuracy_score
from sklearn.metrics import confusion_matrix

pd.set_option('display.max_rows', 300)<jupyter_output><empty_output><jupyter_text>Read in and inspect the data:<jupyter_code>train = pd.read_csv('train.csv')
train.describe(include='all')<jupyter_output><empty_output><jupyter_text>Pclass is the ticket class, and seems like a good predictor of survival without any further recoding:<jupyter_code>train[['Pclass', 'Survived']].groupby('Pclass').agg(('mean', 'count')).sort_values(by=(('Survived', 'mean')), ascending=False)<jupyter_output><empty_output><jupyter_text>Sex also seems like a good predictor of survival:<jupyter_code>train.groupby('Sex').agg(('mean', 'count'))[['Survived']].sort_values(by=(('Survived', 'mean')), ascending=False)<jupyter_output><empty_output><jupyter_text>We define age categories based on inspection of the distribution of ages and of survival by age:<jupyter_code>train.hist('Age', bins=25)
plt.xticks(range(0, 85, 5))
train[['Age', 'Survived']].groupby('Age').agg(['mean', 'count'])<jupyter_output><empty_output><jupyter_text>We examine the presence of siblings/spouses and parents/children, and survival by these two variables:<jupyter_code>train['SibSp'].value_counts()
train.groupby('SibSp')['Survived'].agg(['mean', 'count']).sort_values(by='mean', ascending=False)
train['Parch'].value_counts()
train.groupby('Parch')['Survived'].agg(['mean', 'count']).sort_values(by='mean', ascending=False)<jupyter_output><empty_output><jupyter_text>Based on these descriptives, we will use an indicator for whether any relatives, either siblings/spouses or parents/children, were present.Next, we look at the distribution of fares and mean fare by survival, which suggest fare is also a good predictor of survival:<jupyter_code>train['Fare'].hist(bins=25)
train.groupby('Survived')['Fare'].agg(['mean', 'count'])
train['Fare'].value_counts(normalize=True).sort_index()<jupyter_output><empty_output><jupyter_text>We also consider the point of embarkation, which seems like a potentially worthwhile predictor:<jupyter_code>train.groupby('Embarked')['Survived'].agg(['mean', 'count']).sort_values(by='mean', ascending=False)<jupyter_output><empty_output><jupyter_text>Finally, we will consider the cabin number. It appears that the first letter in the cabin number may identify groups of cabins.<jupyter_code>train['Cabin'].value_counts().sort_index()
train['Cabin'].str.extract('([a-zA-Z])', expand=False).value_counts(normalize=True)<jupyter_output><empty_output><jupyter_text>Data are also provided on ticket number and name. We will not use these data, as there are no clear patterns in ticket number and it seems unlikely that the information in name would give us good information beyond what we already have in variables for sex, age, and ticket class. Next, define functions for initial processing of the data and running some descriptives:<jupyter_code>def preprocess(df):
    df = df.copy()
    df['Cabin_class'] = df['Cabin'].str.extract('([a-zA-Z])', expand=False)
    df['Cabin_class'] = df['Cabin_class'].replace(['D', 'E', 'B'], 'D/E/B')
    df['Cabin_class'] = df['Cabin_class'].replace(['F', 'C'], 'F/C')
    df['Cabin_class'] = df['Cabin_class'].replace(['G', 'A'], 'G/A')
    df['Cabin_class'] = df['Cabin_class'].replace('T', np.nan)
    df['Cabin_class'] = df['Cabin_class'].replace(np.nan, 'NS')
    df.loc[df['SibSp'] > 1, 'SibSp'] = 1
    df.loc[df['Parch'] > 2, 'Parch'] = 2
    df['Relatives'] = 0
    df.loc[(df['SibSp'] > 0) | (df['Parch'] > 0), 'Relatives'] = 1
    df['Age_cat'] = pd.cut(df['Age'], [0, 5, 15, 59, 80], right=True)
    df['Fare_cat'] = pd.cut(df['Fare'], [0, 8.5, 26, 80, 513], right=True, include_lowest=True)
    for dummy in ['Pclass', 'Sex', 'Embarked', 'Cabin_class', 'Age_cat', 'Fare_cat',]:
        df = pd.concat([df, pd.get_dummies(df[dummy], dummy)], axis=1)
    return df
train = preprocess(train)
train.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 891 entries, 0 to 890
Data columns (total 36 columns):
PassengerId               891 non-null int64
Survived                  891 non-null int64
Pclass                    891 non-null int64
Name                      891 non-null object
Sex                       891 non-null object
Age                       714 non-null float64
SibSp                     891 non-null int64
Parch                     891 non-null int64
Ticket                    891 non-null object
Fare                      891 non-null float64
Cabin                     204 non-null object
Embarked                  889 non-null object
Cabin_class               891 non-null object
Relatives                 891 non-null int64
Age_cat                   714 non-null category
Fare_cat                  891 non-null category
Pclass_1                  891 non-null uint8
Pclass_2                  891 non-null uint8
Pclass_3                  891 non-null uint8
Sex_female              [...]<jupyter_text>Define an estimator to select appropriate columns and convert pandas DataFrame to a matrix for use in sklearn pipelines:<jupyter_code>columns = ['Pclass_1', 'Pclass_2', 'Pclass_3', 'Sex_female', 'Sex_male',
		'Cabin_class_D/E/B', 'Cabin_class_F/C', 'Cabin_class_G/A', 'Cabin_class_NS', 'Embarked_S', 'Embarked_C', 'Embarked_Q',
		'Age_cat_(0, 5]', 'Age_cat_(5, 15]', 'Age_cat_(15, 59]', 'Age_cat_(59, 80]',
		'Fare_cat_(-0.001, 8.5]', 'Fare_cat_(8.5, 26.0]', 'Fare_cat_(26.0, 80.0]', 'Fare_cat_(80.0, 513.0]',
		'Relatives',]

class DataFrameSelector(BaseEstimator, TransformerMixin):
	def __init__(self):
		pass

	def fit(self, X, y=None):
		return self

	def transform(self, X):
		global columns
		return X[columns].values<jupyter_output><empty_output><jupyter_text>We have a number of potentially relevant features, so we first examine chi-squared scores for all of them to see if there are any we should eliminate in order to reduce dimensionality:<jupyter_code>pipe = Pipeline([
    ('selector', DataFrameSelector()),
    ('imputer', Imputer(strategy='median')),
    ('kbest', SelectKBest(chi2, k='all')),
])
pipe.fit(train, train['Survived'])
chi2scores = pipe.named_steps['kbest'].scores_
columns11 = columns[:]
for i, rank in zip(chi2scores.argsort()[::-1], range(len(chi2scores))):
    print('k: {2:<5} {0:30} {1:.3f}'.format(columns[i], chi2scores[i], rank+1))
    if rank+1 > 11: columns11.remove(columns[i])<jupyter_output>k: 1     Sex_female                     170.348
k: 2     Sex_male                       92.702
k: 3     Cabin_class_D/E/B              63.492
k: 4     Pclass_1                       55.175
k: 5     Fare_cat_(80.0, 513.0]         46.723
k: 6     Pclass_3                       41.553
k: 7     Fare_cat_(-0.001, 8.5]         41.250
k: 8     Relatives                      22.209
k: 9     Cabin_class_NS                 20.732
k: 10    Embarked_C                     20.464
k: 11    Age_cat_(0, 5]                 19.135
k: 12    Cabin_class_F/C                13.862
k: 13    Fare_cat_(26.0, 80.0]          10.273
k: 14    Pclass_2                       6.161
k: 15    Embarked_S                     5.985
k: 16    Age_cat_(59, 80]               1.444
k: 17    Age_cat_(5, 15]                0.996
k: 18    Cabin_class_G/A                0.648
k: 19    Fare_cat_(8.5, 26.0]           0.040
k: 20    Age_cat_(15, 59]               0.022
k: 21    Embarked_Q                     0.011<jupyter_text>Based on these scores, it looks like we will want to experiment with some different k values for SelectKBest: 4, 7, 11, 13, 15, all. For initial model inspection, we'll try k=11.To begin modeling the data, we'll try running a logistic regression:<jupyter_code>pipe = Pipeline([
    ('selector', DataFrameSelector()),
    ('imputer', Imputer(strategy='median')),
    ('kbest', SelectKBest(chi2, k=11)),
    ('clf', LogisticRegression()),
])
pipe.fit(train, train['Survived'])
for i, col in enumerate(columns11):
    print('{0:30} {1: .3f}'.format(col, pipe.named_steps['clf'].coef_[0][i]))<jupyter_output>Pclass_1                        0.110
Pclass_3                       -1.127
Sex_female                      1.446
Sex_male                       -1.208
Cabin_class_D/E/B               0.805
Cabin_class_NS                 -0.456
Embarked_C                      0.553
Age_cat_(0, 5]                  1.791
Fare_cat_(-0.001, 8.5]          0.285
Fare_cat_(80.0, 513.0]          0.312
Relatives                      -0.140<jupyter_text>Next, consider a decision tree.<jupyter_code>pipe = Pipeline([
    ('selector', DataFrameSelector()),
    ('imputer', Imputer(strategy='median')),
    ('kbest', SelectKBest(chi2, k=11)),
    ('clf', DecisionTreeClassifier(max_depth=5, min_samples_leaf=5)),
])
pipe.fit(train, train['Survived'])
dot_data = export_graphviz(pipe.named_steps['clf'],
                           feature_names=columns11, class_names=['Did not survive', 'Survived'],
                           out_file=None)
graph = graphviz.Source(dot_data)
graph<jupyter_output><empty_output><jupyter_text>We'll try different hyperparameter values for these two models, as well as a random forest classifier and a support vector classifier, in the grid search below, using nested cross-validation to accurately evaluate performance:<jupyter_code>pipe = Pipeline([
	('selector', DataFrameSelector()),
	('imputer', Imputer(strategy = 'median')),
	('kbest', SelectKBest(chi2)),
	('clf', SVC(kernel='rbf')),
])

param_grid = [{
	'kbest__k': [4, 7, 11, 13, 15, 'all'],
	'clf': [LogisticRegression(C=.1), LogisticRegression(C=1), LogisticRegression(C=10), LogisticRegression(C=100), LogisticRegression(C=1000),
		DecisionTreeClassifier(max_depth=3, min_samples_leaf=5), DecisionTreeClassifier(max_depth=4, min_samples_leaf=5), DecisionTreeClassifier(max_depth=5, min_samples_leaf=5), DecisionTreeClassifier(max_depth=6, min_samples_leaf=5), DecisionTreeClassifier(max_depth=7, min_samples_leaf=5),
		RandomForestClassifier(max_depth=3, min_samples_leaf=5), RandomForestClassifier(max_depth=4, min_samples_leaf=5), RandomForestClassifier(max_depth=5, min_samples_leaf=5), RandomForestClassifier(max_depth=6, min_samples_leaf=5), RandomForestClassifier(max_depth=7, min_samples_leaf=5), 
		LinearSVC(C=.1), LinearSVC(C=1), LinearSVC(C=10), LinearSVC(C=100), LinearSVC(C=1000),],
	},
	{
	'kbest__k': [4, 7, 11, 13, 15, 'all'],
	'clf__C': [5, 10, 15, 20],
	'clf__gamma': [.1, .15, .20, .25, .30],
    }
]

np.random.seed(0)
grid_search = GridSearchCV(pipe, param_grid,
	scoring={'accuracy': make_scorer(accuracy_score)}, refit='accuracy')
print('Accuracy score from nested cross-validation: {0}'.format(
    cross_val_score(grid_search, train, train['Survived']).mean()))
grid_search.fit(train, train['Survived'])
print('Best parameters: {0}'.format(grid_search.best_params_))<jupyter_output>Accuracy score from nested cross-validation: 0.8047138047138048
Best parameters: {'clf__C': 10, 'clf__gamma': 0.2, 'kbest__k': 15}<jupyter_text>The support vector classifier does best, with estimated 80% accuracy. Below, we compare results for all of the models tried:<jupyter_code>cvres = grid_search.cv_results_
print('{0:80} {1:>20} {2:>20}'.format('Parameters', 'Train Accuracy', 'Test Accuracy',))
for train_accuracy, test_accuracy, params in zip(cvres['mean_train_accuracy'], cvres['mean_test_accuracy'], cvres['params']):
	print('{0:80.80} {1:>20.5f} {2:>20.5f}'.format(str(params).replace('\n', ''), train_accuracy, test_accuracy))<jupyter_output>Parameters                                                                             Train Accuracy        Test Accuracy
{'clf': LogisticRegression(C=0.1, class_weight=None, dual=False, fit_intercept=T              0.78900              0.77778
{'clf': LogisticRegression(C=0.1, class_weight=None, dual=False, fit_intercept=T              0.78900              0.76655
{'clf': LogisticRegression(C=0.1, class_weight=None, dual=False, fit_intercept=T              0.79349              0.76880
{'clf': LogisticRegression(C=0.1, class_weight=None, dual=False, fit_intercept=T              0.79293              0.76880
{'clf': LogisticRegression(C=0.1, class_weight=None, dual=False, fit_intercept=T              0.79293              0.76655
{'clf': LogisticRegression(C=0.1, class_weight=None, dual=False, fit_intercept=T              0.79125              0.76543
{'clf': LogisticRegression(C=1, class_weight=None, dual=False, fit_intercept=Tru              0.78732              0.78002
{'clf': Logistic[...]<jupyter_text>Finally, we run the final model on the test data and prepare a submission file.<jupyter_code>test = preprocess(pd.read_csv('test.csv'))
test_pred = grid_search.best_estimator_.predict(test)
submission = test[['PassengerId']].copy()
submission['Survived'] = test_pred
submission.to_csv('submission.csv', index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13559198009659007, 'fraction_numerical': 0.04909995609541929, 'mean_word_length': 3.798806179775281, 'pattern_counts': {'":': 0, '<': 66, '<?xml version=': 0, '>': 74, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '296690', 'license': 'MIT', 'n_tokens_mistral': 5099, 'n_tokens_neox': 4559, 'n_words': 1067, 'path': 'Titanic.ipynb', 'repo_name': 'jtleider/titanic', 'size': '13666'}