<jupyter_start><jupyter_text>Titanic: Machine Learning from DisasterDescription of the challenge [here](https://www.kaggle.com/c/titanic). Our data is in .csv format. Each row represents a passenger on the titanic, and some information about them. Let's take a look at the columns: - PassengerId -- A numerical id assigned to each passenger. - Survived -- Whether the passenger survived (1), or didn't (0). We'll be making predictions for this column. - Pclass -- The class the passenger was in -- first class (1), second class (2), or third class (3). - Name -- the name of the passenger. - Sex -- The gender of the passenger -- male or female. - Age -- The age of the passenger. Fractional. - SibSp -- The number of siblings and spouses the passenger had on board. - Parch -- The number of parents and children the passenger had on board. - Ticket -- The ticket number of the passenger. - Fare -- How much the passenger paid for the ticker. - Cabin -- Which cabin the passenger was in. - Embarked -- Where the passenger boarded the Titanic. A good first step is to think logically about the columns and what we're trying to predict. What variables might logically affect the outcome of survived? (reading more about the titanic might help here).**Exercise.** Discuss which variables are more likely to have had an impact on the survival odds of the passengers.  Looking at the dataUsing your recent knowledge of pandas, load the train data into a dataframe and perform some first basic exploratory operations on it to make yourself aquainted with its features.<jupyter_code>import os
import sys

%matplotlib inline

module_path = os.path.abspath(os.path.join('..'))
if module_path not in sys.path:
    sys.path.append(module_path)
module_path
from src.data import make_dataset
titanic = make_dataset.load_data(module_path)

titanic.head(5)
titanic.describe()
titanic.Survived.hist()
titanic.Age.hist()
titanic.groupby('Sex')[['Survived']].agg('count')
titanic.groupby(['Sex', 'Survived']).agg('count')<jupyter_output><empty_output><jupyter_text>Handling missing dataYou might have noticed that not every field for all of the records contains data. This means that we will need to deal with handling missing data. There are many strategies for cleaning up missing data, but a simple one is to just fill in all the missing values with the median of all the values in the column. (Question: why the median and not the average?)Implement a solution to handle missing data on the Titanic dataframe.<jupyter_code># This fills all missing values with the median
#titanic['Age'] = titanic['Age'].fillna(titanic['Age'].median())
# This is a bit more elaborated and fills with the median per group, defined as Title
titanic['Title'] = titanic.apply(lambda x: x['Name'].split(',')[1].strip().split('.')[0], axis=1)
titanic.groupby('Title')['Title'].agg('count')
titanic[(~titanic.Age.isnull()) & (titanic.Title=='Master')].Age.median()
often_titles = ['Master', 'Miss', 'Mr', 'Mrs']
titanic[(~titanic.Age.isnull()) & (~titanic.Title.isin(often_titles))].Age.median()
titanic.Age.median()
def assign_age(title):
    often_titles = ['Master', 'Miss', 'Mr', 'Mrs']
    if title in often_titles:
        median_age_title = titanic[(~titanic.Age.isnull()) & (titanic.Title==title)].Age.median()
    else:
        median_age_title = titanic[(~titanic.Age.isnull()) & (~titanic.Title.isin(often_titles))].Age.median()
    return median_age_title
titanic['Age'] = titanic['Age'].fillna(titanic.apply(lambda x: assign_age(x['Title']), axis=1))
titanic.head(6)<jupyter_output><empty_output><jupyter_text>Non-numeric columnsSeveral of our columns are non-numeric, which is a problem when it comes time to make predictions -- we can't feed non-numeric columns into a machine learning algorithm and expect it to make sense of them.We have to either exclude our non-numeric columns when we train our algorithm (Name, Sex, Cabin, Embarked, and Ticket), or find a way to convert them to numeric columns.Decide which non-numeric columns to keep for the analysis, based on your domain knowledge of the problem. Then, turn them into numeric values (categorical variables).<jupyter_code># Sex
# Check that Sex column is complete
# titanic[titanic.Sex.isnull()]
titanic['Sex'] = titanic.apply(lambda x: 1 if x.Sex=='female' else 0, axis=1)
# Embarked (C = Cherbourg; Q = Queenstown; S = Southampton)
titanic[titanic.Embarked.isnull()]
titanic.Embarked.value_counts()
# Fill the missing values with the most common Embarked
titanic['Embarked'] = titanic['Embarked'].fillna('S')
dd = {'S': 0, 'C': 1, 'Q': 2}
titanic['Embarked'] = titanic.apply(lambda x: dd[x.Embarked], axis=1)
titanic.head()<jupyter_output><empty_output><jupyter_text>Cross validationWe want to train the algorithm on different data than we make predictions on. This is critical if we want to avoid overfitting. Overfitting is what happens when a model fits itself to "noise", not signal. Every dataset has its own quirks that don't exist in the full population. For example, if I asked you to predict the top speed of a car from its horsepower and other characteristics, and gave you a dataset that randomly had cars with very high top speeds, you would create a model that overstated speed. The way to figure out if your model is doing this is to evaluate its performance on data it hasn't been trained using.Every machine learning algorithm can overfit, although some are much less prone to it. If you evaluate your algorithm on the same dataset that you train it on, it's impossible to know if it's performing well because it overfit itself to the noise, or if it actually is a good algorithm.Luckily, cross validation is a simple way to avoid overfitting. To cross validate, you split your data into some number of parts (or "folds"). Lets use 3 as an example. You then do this: - Combine the first two parts, train a model, make predictions on the third. - Combine the first and third parts, train a model, make predictions on the second. - Combine the second and third parts, train a model, make predictions on the first.This way, we generate predictions for the whole dataset without ever evaluating accuracy on the same data we train our model using.Familiarize yourself with cross-validation in scikit-learn reading about [KFold](http://scikit-learn.org/stable/modules/generated/sklearn.cross_validation.KFold.html).  Applying scikit-learn ML algorithms to the Titanic dataset - Follow the scikit-learn API and apply the ML algorithm of your choice to a subset of the Titanic dataset. - Using the wrappers described [here](http://scikit-learn.org/stable/modules/cross_validation.htmlcross-validation), compute cross-validated metrics for said algorithm - If your algorithm takes various parameters as input, consider tuning them to improve performance - Repeat with various algorithms and compare its performance<jupyter_code>from sklearn.ensemble import RandomForestClassifier
from sklearn.cross_validation import KFold
from sklearn import cross_validation

predictors = ["Pclass", "Sex", "Age", "SibSp", "Parch", "Fare", "Embarked"]

#kf = KFold(titanic.shape[0], n_folds=5, random_state=1)
kf = KFold(len(titanic), n_folds=5, shuffle=True)
train, test = [k for k in kf][0]
# The predictors we're using the train the algorithm.  Note how we only take the rows in the train folds.
X_train = (titanic[predictors].iloc[train,:])
# The target we're using to train the algorithm.
y_train = titanic["Survived"].iloc[train]
X_test = (titanic[predictors].iloc[test,:])
y_test = titanic["Survived"].iloc[test]
rf = RandomForestClassifier(n_estimators=50)

rf.fit(X_train, y_train)
rf.score(X_test, y_test)
# Initialize our algorithm with the default paramters
# n_estimators is the number of trees we want to make
# min_samples_split is the minimum number of rows we need to make a split
# min_samples_leaf is the minimum number of samples we can have at the place where a tree branch ends (the bottom points of the tree)
alg = RandomForestClassifier(random_state=1, n_estimators=10, min_samples_split=2, min_samples_leaf=1)

scores = cross_validation.cross_val_score(alg, titanic[predictors], titanic["Survived"], cv=5)
print(scores.mean())
#The first, and easiest, thing we can do to improve the accuracy of the random forest is to increase 
# the number of trees we're using. Training more trees will take more time, but because of the fact 
# that we're averaging many predictions made on different subsets of the data, having more trees will 
#increase accuracy greatly (up to a point).

#We can also tweak the min_samples_split and min_samples_leaf variables to reduce overfitting. 
#Because of how a decision tree works (as we explained in the video), having splits that go all the way down, 
#or overly deep in the tree can result in fitting to quirks in the dataset, and not true signal. 
#Thus, increasing min_samples_split and min_samples_leaf can reduce overfitting, which will actually 
#improve our score, as we're making predictions on unseen data. A model that is less overfit, 
#and that can generalize better, will actually perform better on unseen data, but worse on seen data.

alg = RandomForestClassifier(random_state=1, n_estimators=150, min_samples_split=4, min_samples_leaf=2)

scores = cross_validation.cross_val_score(alg, titanic[predictors], titanic["Survived"], cv=3)
print(scores.mean())
from sklearn.linear_model import LogisticRegression

# Initialize our algorithm
alg = LogisticRegression(random_state=1)
# Compute the accuracy score for all the cross validation folds.  (much simpler than what we did before!)
scores = cross_validation.cross_val_score(alg, titanic[predictors], titanic["Survived"], cv=3)
# Take the mean of the scores (because we have one for each fold)
print(scores.mean())<jupyter_output>0.791245791246<jupyter_text>Generating new featuresIf your data is rich, sometimes new features can be extracted from it that can better inform your machine learning algorithm. You can try generating new features from the name of the passengers, the members of their families present, etc.  Finding the best featuresFeature engineering is the most important part of any machine learning task, and there are lots more features we could calculate. But we also need a way to figure out which features are the best.One way to do this is to use univariate feature selection. This essentially goes column by column, and figures out which columns correlate most closely with what we're trying to predict (Survived).As usual, scikit-learn has a function that will help us with feature selection, [SelectKBest](http://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectKBest.html). This selects the best features from the data, and allows us to specify how many it selects.<jupyter_code>import numpy as np
from sklearn.feature_selection import SelectKBest, f_classif
import matplotlib.pyplot as plt
%matplotlib inline

predictors = ["Pclass", "Sex", "Age", "SibSp", "Parch", "Fare", "Embarked"]

# Perform feature selection
selector = SelectKBest(f_classif, k=5)
selector.fit(titanic[predictors], titanic["Survived"])

# Get the raw p-values for each feature, and transform from p-values into scores
scores = -np.log10(selector.pvalues_)

# Plot the scores.  See how "Pclass", "Sex", "Title", and "Fare" are the best?
plt.bar(range(len(predictors)), scores)
plt.xticks(range(len(predictors)), predictors, rotation='vertical')
plt.show()

# Pick only the four best features.
predictors = ["Pclass", "Sex", "Fare", "Title"]

alg = RandomForestClassifier(random_state=1, n_estimators=150, min_samples_split=8, min_samples_leaf=4)
#The first, and easiest, thing we can do to improve the accuracy of the random forest is to increase 
# the number of trees we're using. Training more trees will take more time, but because of the fact 
# that we're averaging many predictions made on different subsets of the data, having more trees will 
#increase accuracy greatly (up to a point).

#We can also tweak the min_samples_split and min_samples_leaf variables to reduce overfitting. 
#Because of how a decision tree works (as we explained in the video), having splits that go all the way down, 
#or overly deep in the tree can result in fitting to quirks in the dataset, and not true signal. 
#Thus, increasing min_samples_split and min_samples_leaf can reduce overfitting, which will actually 
#improve our score, as we're making predictions on unseen data. A model that is less overfit, 
#and that can generalize better, will actually perform better on unseen data, but worse on seen data.

alg = RandomForestClassifier(random_state=1, n_estimators=150, min_samples_split=4, min_samples_leaf=2)

scores = cross_validation.cross_val_score(alg, titanic[['Pclass', 'Sex', 'Fare', 'Embarked']], titanic["Survived"], cv=3)
print(scores.mean())
plt.scatter(titanic.Pclass, titanic.Fare)
np.corrcoef(titanic.Fare, titanic.Pclass)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0867784560698555, 'fraction_numerical': 0.005409164670427324, 'mean_word_length': 5.249154997585707, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 20, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '475406', 'license': 'MIT', 'n_tokens_mistral': 3842, 'n_tokens_neox': 3642, 'n_words': 1619, 'path': 'notebooks/titanic-machine-learning-key.ipynb', 'repo_name': 'symeneses/ds_proj_ex', 'size': '12941'}