<jupyter_start><jupyter_text>Titanic Data Analytics Project - Kaggle Beichuan(<NAME> 2017.01.10 In this Kaggle challenge, I need to complete the analysis of what sorts of people were likely to survive.  Data Dictionary (Variable	Definition	Key)*  survival	Survival	0 = No, 1 = Yes*  pclass	Ticket class	1 = 1st, 2 = 2nd, 3 = 3rd*  sex	Sex	*  Age	Age in years	*  sibsp	 of siblings / spouses aboard the Titanic	*  parch	 of parents / children aboard the Titanic	*  ticket	Ticket number	*  fare	Passenger fare	*  cabin	Cabin number	*  embarked	Port of Embarkation	C = Cherbourg, Q = Queenstown, S = Southampton  Variable Notes pclass: A proxy for socio-economic status (SES)*  1st = Upper*  2nd = Middle*  3rd = Lowerage: Age is fractional if less than 1. If the age is estimated, is it in the form of xx.5sibsp: The dataset defines family relations in this way...*  Sibling = brother, sister, stepbrother, stepsister*  Spouse = husband, wife (mistresses and fiancés were ignored)parch: The dataset defines family relations in this way...*  Parent = mother, father*  Child = daughter, son, stepdaughter, stepson*  Some children travelled only with a nanny, therefore parch=0 for them.  Load and Browse Data<jupyter_code>import pandas as pd
import numpy as np
import seaborn as sns
td = pd.read_csv('data/train.csv')
td.head(5)
td.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 891 entries, 0 to 890
Data columns (total 12 columns):
PassengerId    891 non-null int64
Survived       891 non-null int64
Pclass         891 non-null int64
Name           891 non-null object
Sex            891 non-null object
Age            714 non-null float64
SibSp          891 non-null int64
Parch          891 non-null int64
Ticket         891 non-null object
Fare           891 non-null float64
Cabin          204 non-null object
Embarked       889 non-null object
dtypes: float64(2), int64(5), object(5)
memory usage: 83.6+ KB<jupyter_text>EDA - Numeric Variables<jupyter_code>nuvar = ['Age','SibSp','Parch','Fare']<jupyter_output><empty_output><jupyter_text>AgeFrom the univariate distribution plot, most of the people are between 20ish to 40ish<jupyter_code>td['Age'].value_counts()
age = td['Age'].dropna() # do it since seaborn can't create plot with NaN in the variable
sns.distplot(age)<jupyter_output><empty_output><jupyter_text>Fill in Missing Values in Age<jupyter_code>age_1 = td['Age'].interpolate(method='linear')
age_2 = td['Age'].interpolate(method='pchip')
age_3 = td['Age'].interpolate(method='cubic')
#sns.distplot(age_1)
sns.distplot(age_2)
#sns.distplot(age_2)<jupyter_output><empty_output><jupyter_text>The three methods get similar results. Source: https://pandas.pydata.org/pandas-docs/stable/missing_data.html<jupyter_code>td['Age_all'] = td['Age']. interpolate(method='pchip')<jupyter_output><empty_output><jupyter_text>SibSpAbout 3/4 of the people has no sibling or spouse, about 1/4 has one sibling or spouse.<jupyter_code>td['SibSp'].value_counts()
sns.distplot(td['SibSp'])<jupyter_output><empty_output><jupyter_text>ParchAbout 2/3 of the people has no parents/children, this includs some children traveling with only nanny. And about 1/3 of the people has one to two parents or children.<jupyter_code>td['Parch'].value_counts()
sns.distplot(td['Parch'])<jupyter_output><empty_output><jupyter_text>FareMost of the people have fare of around 10 (assuming the unit is dollar here).<jupyter_code>td['Fare'].value_counts()
sns.distplot(td['Fare'])<jupyter_output><empty_output><jupyter_text>Pairwise relationship<jupyter_code>nuvar
nuvar[0] = 'Age_all'   # replace age with age_all because seaborn.pairplot() doesn't work well with NaN
nuvar
nuvar_df = td[nuvar]
sns.pairplot(nuvar_df)<jupyter_output><empty_output><jupyter_text>Correlation MatrixThere no strongly related numerical variables except for age and age_all, which is the same feature with and without missing value.<jupyter_code>corr = td.corr()
sns.heatmap(corr)<jupyter_output><empty_output><jupyter_text>EDA - Categorical Variable<jupyter_code>list(td)
catvar = ['Survived','Pclass','Name', 'Sex', 'Ticket', 'Cabin', 'Embarked']<jupyter_output><empty_output><jupyter_text>Survived<jupyter_code>td['Survived'].value_counts()<jupyter_output><empty_output><jupyter_text>Pclass<jupyter_code>td['Pclass'].value_counts()
td['Pclass'] = td['Pclass'].astype('category') # Convert Pclass from int type to category<jupyter_output><empty_output><jupyter_text>difference between object and category type?<jupyter_code>td['Pclass'].dtype
sns.barplot(x="Pclass", y="Survived", data=td);<jupyter_output><empty_output><jupyter_text>Sex<jupyter_code>td['Sex'].value_counts()
sns.barplot(x='Sex', y='Survived', data=td)<jupyter_output><empty_output><jupyter_text>Ticket<jupyter_code>td['Ticket'].value_counts()<jupyter_output><empty_output><jupyter_text>Cabin<jupyter_code>td['Cabin'].value_counts()
cabinLetter = ['A','B','C','D','E','F','G']
num = []
for i in cabinLetter:
    num.append(len(td[td['Cabin'].str.contains(i, na = False)]))
num<jupyter_output><empty_output><jupyter_text>I guess the letter in the cabin number indicates the location of the seats. So I wanted to see how many people are in each cabin letter. But the Cabin variable has too many missing values to use.  Embarked<jupyter_code>td['Embarked'].value_counts()
td['Embarked'] = td['Embarked'].fillna('S')  # Use the mode to fill the missing values
sns.barplot(x='Embarked', y='Survived', data=td)<jupyter_output><empty_output><jupyter_text>how to select catigorical variables? what stats test to use?  Define Features and Target<jupyter_code>Y = td['Survived'].values
list(td)
feature_list = [
# 'PassengerId',
# 'Survived',
 'Pclass',
# 'Name',
 'Sex',
# 'Age',
 'SibSp',
 'Parch',
# 'Ticket',
 'Fare',
# 'Cabin',
 'Embarked',
 'Age_all']
td[feature_list].info()   # no missing values<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 891 entries, 0 to 890
Data columns (total 7 columns):
Pclass      891 non-null category
Sex         891 non-null object
SibSp       891 non-null int64
Parch       891 non-null int64
Fare        891 non-null float64
Embarked    891 non-null object
Age_all     891 non-null float64
dtypes: category(1), float64(2), int64(2), object(2)
memory usage: 42.8+ KB<jupyter_text>One-hot Encoding<jupyter_code>X = pd.get_dummies(td[feature_list])
X
feature_list_dummy = list(X)
feature_list_dummy<jupyter_output><empty_output><jupyter_text>Standardize Features for Logistics Regression<jupyter_code>from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)  
X_scaled.shape<jupyter_output><empty_output><jupyter_text>Train Test Split<jupyter_code>from sklearn.cross_validation import train_test_split
X_train, X_test, Y_train, Y_test = train_test_split(X_scaled, Y, test_size=0.2)<jupyter_output><empty_output><jupyter_text>Modeling - Logistics Regression  Training model<jupyter_code>from sklearn.linear_model import LogisticRegression
lr = LogisticRegression()
lr.fit(X_train, Y_train)<jupyter_output><empty_output><jupyter_text>Predict on training data<jupyter_code>train_predict = lr.predict(X_train)<jupyter_output><empty_output><jupyter_text>Predict on testing data<jupyter_code>test_predict = lr.predict(X_test)<jupyter_output><empty_output><jupyter_text>Model performance & Interpretation - Logistics RegressionThe performance is not bad.reference: http://scikit-learn.org/stable/modules/classes.htmlmodule-sklearn.metrics<jupyter_code>from sklearn.metrics import precision_score, accuracy_score, recall_score, f1_score, roc_auc_score
def print_results(y_true, y_pred):
    print("Accuracy of the Logistic Regression is: {}".format(accuracy_score(y_true, y_pred)))
    print("Precision of the Logistic Regression is: {}".format(precision_score(y_true, y_pred)))
    print("Recall of the Logistic Regression is: {}".format(recall_score(y_true, y_pred)))
    print("f1-score of the Logistic Regression is: {}".format(f1_score(y_true, y_pred)))
    print("Area Under Curve (AUC) of the Logistic Regression is: {}".format(roc_auc_score(y_true, y_pred)))
print("Training set scores:")
print_results(Y_train, train_predict)
print("Testing set scores:")
print_results(Y_test, test_predict)
from sklearn.metrics import roc_curve
#reference: http://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_curve.html#sklearn.metrics.roc_curve
Y_test_pred_proba = lr.predict_proba(X_test)[:,1]
FalsePositiveRate, TruePositiveRate, thresholds = roc_curve(Y_test, Y_test_pred_proba)
import matplotlib.pyplot as plt
% matplotlib inline
plt.style.use("ggplot")
# plot TPR against FPR
plt.plot(FalsePositiveRate, TruePositiveRate, color='red')

# plot 45 degree line
xx = np.linspace(0, 1.0, 20)
plt.plot(xx, xx, color='blue')

plt.xlabel("False Positive Rate")
plt.ylabel("True Positive Rate")
plt.title("ROC")

plt.show()<jupyter_output><empty_output><jupyter_text>Understand Estimated CoefficientsSince I standardized the variables, so estimated coefficients can reveal which variable has bigger predicting power.From the result, female are more likely to survive, and people in class 1 are more likely to survive. Also, the higher people paid as fare, the more likely they are to survive.  is my interpretation fair???<jupyter_code>#lr.coef_.flatten()  
#reference: https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.ndarray.flatten.html
#list(zip(feature_list_dummy, lr.coef_))
df_coeffs = pd.DataFrame(list(zip(feature_list_dummy, lr.coef_.flatten()))).sort_values(by=[1], ascending=False)
df_coeffs.columns = ['feature', 'coeff']
df_coeffs
Model<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11657332503629952, 'fraction_numerical': 0.018668326073428748, 'mean_word_length': 6.45784996133024, 'pattern_counts': {'":': 0, '<': 110, '<?xml version=': 0, '>': 110, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '27', 'dir': 'jupyter-structured-clean-dedup', 'id': '471161', 'license': 'MIT', 'n_tokens_mistral': 3544, 'n_tokens_neox': 3331, 'n_words': 814, 'path': 'ml_class/Titanic Project/Titanic Data Analytics Project - 2018.01.10.ipynb', 'repo_name': 'becliu/MSBA-note', 'size': '9648'}