<jupyter_start><jupyter_text>Predicting Survival with the Titanic Dataset Task at Hand: Using the Titanic dataset, predict 'Survived' based on a number of characteristics about the passenger.  Predict with an accuracy of .80 using a classifier algorithm.  Step 1. Imports<jupyter_code>import numpy as np
import pandas as pd
import re
import matplotlib.pyplot as plt
from matplotlib import style
import seaborn as sns
import warnings
from IPython.display import display

warnings.filterwarnings('ignore')
%matplotlib inline

style.use('seaborn')

pd.options.display.max_columns = None<jupyter_output><empty_output><jupyter_text>Step 2. Get the data<jupyter_code>train_original = pd.read_csv("../data/raw/train.csv")
test_original = pd.read_csv("../data/raw/test.csv")
train_original.shape
test_original.shape<jupyter_output><empty_output><jupyter_text>Let's keep the original DataFrames intact and instead work on copies of the data.<jupyter_code>train = train_original.copy()
test = test_original.copy()<jupyter_output><empty_output><jupyter_text>Step 3. Feature Engineering  Step 3a. Peek at the data using .head(), .describe(), and .info()<jupyter_code>train.head()
train.describe() # Gives us summary statistics of our numerical variables.
train.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 891 entries, 0 to 890
Data columns (total 12 columns):
PassengerId    891 non-null int64
Survived       891 non-null int64
Pclass         891 non-null int64
Name           891 non-null object
Sex            891 non-null object
Age            714 non-null float64
SibSp          891 non-null int64
Parch          891 non-null int64
Ticket         891 non-null object
Fare           891 non-null float64
Cabin          204 non-null object
Embarked       889 non-null object
dtypes: float64(2), int64(5), object(5)
memory usage: 83.6+ KB<jupyter_text>Step 3b. Ask questions. Try to see how the data may correlate to survival rate.     Passenger Class probably has something to do with the survival rate.  First class passengers may have had preferential treatment.    Name might not be immediately useful, unless we transform it somehow.  Perhaps we can strip the person's title?    Sex probably has a high correlation with survival rate, as women were likely given preferential treatment.    Age is tricky.  On one hand, young people might be more likely to survive (think of the "women and children" saying!), but they may have not lasted very long as they are likely weaker (especially kids under 5 years old).    The SibSp/Parch survival rate is also not so intuitive.    Ticket number probably holds some keys into survival rate, because ticket numbers might indicate a passenger's location at the time of the collision.    Fare is probably not useful, but we will look into it.    Cabin would probably be useful, but it looks like we are missing a lot of values.    Embarked is also probably not useful, but again, we will see how it correlates with Survived!  Step 3c. Transform columns to make them more useful.  Concatenate the DataFrames Concatenating the train and test DataFrames ensure that any column transformations will be applied to both.<jupyter_code># Add a dummy column to test DF so that train & test have the same dimensions.
test['Survived'] = -8888
# Concat the DFs.  This creates a new DF altogether using COPIES of the two component DFs.
data = pd.concat([train, test])
print('train shape: {0}\ntest shape: {1}\nconcatenated data shape: {2}'.format(train.shape, test.shape, data.shape))
data.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 1309 entries, 0 to 417
Data columns (total 12 columns):
Age            1046 non-null float64
Cabin          295 non-null object
Embarked       1307 non-null object
Fare           1308 non-null float64
Name           1309 non-null object
Parch          1309 non-null int64
PassengerId    1309 non-null int64
Pclass         1309 non-null int64
Sex            1309 non-null object
SibSp          1309 non-null int64
Survived       1309 non-null int64
Ticket         1309 non-null object
dtypes: float64(2), int64(5), object(5)
memory usage: 132.9+ KB<jupyter_text>Transformations  Name It looks like we can extract a couple useful things from this column.  Let's create a new column that contains the passenger's title. Before we do any transformations, we need to ensure there are no "NaN" values in the column.  In the table above, we can see that there are no missing 'Name' values.  Impute Nulls<jupyter_code>data.Name.isnull().sum()<jupyter_output><empty_output><jupyter_text>No nulls to worry about!  Extract Useful Parts<jupyter_code># Build a function to extract title:
def extract_title(name):
    title = re.search(' ([A-Z][a-z]+)\.', name).group(1)
    return title


# Build a function that transforms a DF using the above 2 functions.
def transform_name(df):
    df['Title'] = df.Name.apply(extract_title)
    df.drop(['Name'], axis = 1, inplace=True) # drops the Name column, which has served its purpose.
    return df
transform_name(data)<jupyter_output><empty_output><jupyter_text>Title<jupyter_code>data.Title.unique()<jupyter_output><empty_output><jupyter_text>That's a lot of redundant titles.  Let's condense them:<jupyter_code>title_dict = {'Title': {'Mlle': 'Miss', 'Ms': 'Miss', 'Don': 'UM', 'Rev': 'Special',
                        'Dr': 'Special', 'Mme': 'Mrs', 'Major': 'Special', 'Lady': 'UW',
                       'Sir': 'UM', 'Col': 'Special', 'Capt': 'Special', 'Countess': 'UW',
                        'Jonkheer': 'UM', 'Dona': 'UW'}} # "UW" here means "upper class woman"

data.replace(title_dict, inplace=True)
data.Title.unique()
df = train.copy()
transform_name(df)
df.replace(title_dict, inplace=True)

sns.barplot(x='Title', y='Survived', data=df, ci=None);<jupyter_output><empty_output><jupyter_text>Looks promising!  Fare  Impute Missing Values<jupyter_code>data[data['Fare'].isnull()]<jupyter_output><empty_output><jupyter_text>Our first missing value! Let's fill it with the median Fare of the passengers embarking from Southhampton.<jupyter_code>age_fillna = data.groupby('Embarked').Fare.transform('median')
data['Fare'].fillna(age_fillna, inplace=True)<jupyter_output><empty_output><jupyter_text>Discretize the Values<jupyter_code>sns.distplot(data['Fare']);<jupyter_output><empty_output><jupyter_text>We have a wide range of values for fare.  We will transform this feature by cutting the values into 4 separate categories: very low, low, high, and very high.<jupyter_code>data['Fare_Class'] = pd.qcut(data.Fare, 4, labels=['very_low', 'low', 'high', 'very_high'])
data.drop('Fare', axis=1, inplace=True)
data.head()<jupyter_output><empty_output><jupyter_text>Awesome.  Let's move on!  Ticket For simplicity, let's just remove this feature.<jupyter_code>data.drop('Ticket', axis=1, inplace=True)<jupyter_output><empty_output><jupyter_text>Cabin It looks like our cabin values have at least one useful feature, which is the leading letter. Let's transform the feature to just take that value.  Imputing Nulls First, let's fill in the nulls (which likely mean that the passenger didn't have a cabin at all!)<jupyter_code>data.Cabin.fillna('X', inplace=True)
data.head()<jupyter_output><empty_output><jupyter_text>Now we'll transform the feature.<jupyter_code>data['Cabin'] = data.Cabin.apply(lambda x : x[0])
data.head()<jupyter_output><empty_output><jupyter_text>Age Transformation  Imputing Nulls<jupyter_code>data.Age.isnull().sum()<jupyter_output><empty_output><jupyter_text>We have quite a few null values here.  We have to deal with these somehow before feeding the data to our ML algorithm. Since we have 263 null values, we probably shouldn't just delete them. Let's see if we can impute their values.<jupyter_code>data[data['Age'].isnull()].sample(10) # Let's see how we might impute the values.<jupyter_output><empty_output><jupyter_text>What other characteristic might give us some insight into their age?Probably 'Title'.<jupyter_code>medianAge_byTitle = data.groupby('Title')['Age'].transform('median')
# Fill in the missing values with the median Age of their 'Title' grouping
data.Age.fillna(medianAge_byTitle, inplace=True)<jupyter_output><empty_output><jupyter_text>Discretizing Similar to what we did with Fare.<jupyter_code>bins = [0, 2, 5, 10, 14, 18, 25, 35, 45, 55, 65, 120]
labels = ['Baby','YoungChild','Child','PreTeen','Teen',
          'YoungAdult','Adult','OldAdult','OlderAdult',
          'YoungElderly','Elderly']  # Don't worry about the label names too much. They will be encoded shortly.

data['Age'] = pd.cut(data['Age'], bins, labels=labels)<jupyter_output><empty_output><jupyter_text>Step 3d. Feature Encoding We need to translate our categorical ('string') features into something the ML algorithm can understand.Here, we use Panda's get_dummies() function to use one-hot-encoding on our categorical features.  Essentially, one-hot-encoding converts our string values into binary 0's or 1's.  It is a bit more complex than that, so try googling it!<jupyter_code>to_encode = ['Age', 'Cabin', 'Embarked', 'Sex', 'Title', 'Fare_Class']

data = pd.get_dummies(data=data, columns=to_encode, drop_first=True)<jupyter_output><empty_output><jupyter_text>Step 4. Resplit the data<jupyter_code>train = data.iloc[:891]
test = data.iloc[891:]<jupyter_output><empty_output><jupyter_text>We now need to split our train set by column- X is all the independent variables (Features) and y is the dependent variable (Label).<jupyter_code>X = train.drop(['PassengerId', 'Survived'], axis=1)
y = train['Survived']
test_passenger_ids = test['PassengerId'] # Save this for our submission later!
test_features = test.drop(['PassengerId', 'Survived'], axis=1)<jupyter_output><empty_output><jupyter_text>Step 5. Modeling<jupyter_code>from sklearn.linear_model import LogisticRegression # MODEL 1
from sklearn.neighbors import KNeighborsClassifier # MODEL 2
from sklearn.tree import DecisionTreeClassifier # MODEL 3
from sklearn.model_selection import train_test_split, RandomizedSearchCV
from scipy.stats import randint, uniform<jupyter_output><empty_output><jupyter_text>Step 5a. Create Hyperparameter ranges for our RandomizedSearchCV RandomizedSearchCV does a few things for us.  First, it randomly splits our data into n "folds", and trains the model on n-1 of the folds.  Then, it tests each model on the last fold, and produces a score.  It also lets us train the models using a range of hyperparameter settings.  It will tell us which hyperparameter settings provided the best score. Here, we create the hyperparameter ranges to test out.<jupyter_code>lr_param_dist = {'penalty': ['l1', 'l2'],
                 'C': uniform(loc=0, scale=4)}

kn_param_dist = {'n_neighbors': range(1,31),
                'weights': ['uniform', 'distance']}

dt_param_dist = {"max_depth": [3, None],
              "max_features": randint(1, 9),
              "min_samples_leaf": randint(1, 9),
              "criterion": ["gini", "entropy"]}<jupyter_output><empty_output><jupyter_text>Now, instantiate our models:<jupyter_code>lr = LogisticRegression()
kn = KNeighborsClassifier()
dt = DecisionTreeClassifier()<jupyter_output><empty_output><jupyter_text>...and instantiate our RandomizedSearchCVs based on those models:<jupyter_code>lr_cv = RandomizedSearchCV(lr, lr_param_dist, cv=5)
kn_cv = RandomizedSearchCV(kn, kn_param_dist, cv=5)
dt_cv = RandomizedSearchCV(dt, dt_param_dist, cv=5)<jupyter_output><empty_output><jupyter_text>Logistic Regression<jupyter_code>lr_cv.fit(X,y)
# The best_params_ method tells us which hyperparameters worked best
lr_cv.best_params_
# The best_score_ method tells us our best score using this model with the above hyperparameters.
lr_cv.best_score_<jupyter_output><empty_output><jupyter_text>K-Nearest Neighbors<jupyter_code>kn_cv.fit(X,y)
kn_cv.best_params_
kn_cv.best_score_<jupyter_output><empty_output><jupyter_text>Decision Tree<jupyter_code>dt_cv.fit(X,y)
dt_cv.best_params_
dt_cv.best_score_<jupyter_output><empty_output><jupyter_text>Step 6. Create our final CSVs for Submission<jupyter_code>lr = LogisticRegression(C=1.9547916803253425, penalty='l2')  # Instantiate the best tuned LR model
lr.fit(X,y)  # Fit it to our data
test_predictions = lr.predict(test_features)  # Predict the results from our test set
submissionLR = pd.DataFrame({'PassengerId': test_passenger_ids,
                           'Survived': test_predictions}) # Create a submission DataFrame w/ just P.I.D and 'Survived'
submissionLR.to_csv('TitanicSubmissionsLR.csv', index=False) # Export to csv!
kn = KNeighborsClassifier(weights='uniform', n_neighbors=9)
kn.fit(X,y)
test_predictions = kn.predict(test_features)
submissionKN = pd.DataFrame({'PassengerId': test_passenger_ids,
                           'Survived': test_predictions})
submissionKN.to_csv('TitanicSubmissionsKN.csv', index=False)
dt = DecisionTreeClassifier(criterion='gini',
                            max_depth=None,
                            max_features=6,
                            min_samples_leaf=6)
dt.fit(X,y)
test_predictions = dt.predict(test_features)
submissionDT = pd.DataFrame({'PassengerId': test_passenger_ids,
                           'Survived': test_predictions})
submissionDT.to_csv('TitanicSubmissionsDT.csv', index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10172672672672672, 'fraction_numerical': 0.019444444444444445, 'mean_word_length': 4.9655172413793105, 'pattern_counts': {'":': 4, '<': 125, '<?xml version=': 0, '>': 125, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '31', 'dir': 'jupyter-structured-clean-dedup', 'id': '525612', 'license': 'MIT', 'n_tokens_mistral': 4493, 'n_tokens_neox': 4178, 'n_words': 1308, 'path': 'Titanic-Survival-v2.ipynb', 'repo_name': 'tjdolan121/datascience_notebooks', 'size': '13320'}