<jupyter_start><jupyter_text>First Kaggle challenge - Titanic Machine Learning from Disaster- v1.6_032020- author: marcusRB- [Kaggle - Titanic challenge](https://www.kaggle.com/c/titanic/)```In this version I use only few feature, I try an another cleansing method.I use same ML algorithms```This is the legendary Titanic ML competition – the best, first challenge for you to dive into ML competitions and familiarize yourself with how the Kaggle platform works.The competition is simple: use machine learning to create a model that predicts which passengers survived the Titanic shipwreck. The ChallengeThe sinking of the Titanic is one of the most infamous shipwrecks in history.On April 15, 1912, during her maiden voyage, the widely considered “unsinkable” RMS Titanic sank after colliding with an iceberg. Unfortunately, there weren’t enough lifeboats for everyone onboard, resulting in the death of 1502 out of 2224 passengers and crew.While there was some element of luck involved in surviving, it seems some groups of people were more likely to survive than others.In this challenge, we ask you to build a predictive model that answers the question: “what sorts of people were more likely to survive?” using passenger data (ie name, age, gender, socio-economic class, etc). What Data Will I Use in This Competition?In this competition, you’ll gain access to two similar datasets that include passenger information like name, age, gender, socio-economic class, etc. One dataset is titled `train.csv` and the other is titled `test.csv`.Train.csv will contain the details of a subset of the passengers on board (891 to be exact) and importantly, will reveal whether they survived or not, also known as the “ground truth”.The `test.csv` dataset contains similar information but does not disclose the “ground truth” for each passenger. It’s your job to predict these outcomes.Using the patterns you find in the train.csv data, predict whether the other 418 passengers on board (found in test.csv) survived.  Workflow stagesThe competition solution workflow goes through seven stages described in the Data Science Solutions book.1. Question or problem definition.2. Acquire training and testing data.3. Wrangle, prepare, cleanse the data.4. Analyze, identify patterns, and explore the data.5. Model, predict and solve the problem.6. Visualize, report, and present the problem solving steps and final solution.7. Supply or submit the results.  Check the versions of libraries<jupyter_code># Check the versions of libraries MacOS

# Python version
import sys
print('Python: {}'.format(sys.version))
# scipy
import scipy
print('scipy: {}'.format(scipy.__version__))
# numpy
import numpy
print('numpy: {}'.format(numpy.__version__))
# matplotlib
import matplotlib
print('matplotlib: {}'.format(matplotlib.__version__))
# pandas
import pandas
print('pandas: {}'.format(pandas.__version__))
# scikit-learn
import sklearn
print('sklearn: {}'.format(sklearn.__version__))
# Check the versions of libraries Win10 Docker
#!pip install --upgrade pandas
#!pip install --upgrade sklearn
#!pip install kaggle
#!pip install keras
#!pip install tensorflow<jupyter_output><empty_output><jupyter_text>*** Import Libraries<jupyter_code># data analysis and wrangling
import pandas as pd
import numpy as np
import random as rnd
from scipy.stats import norm, skew
from scipy import stats
import xlrd, xdrlib

# visualization
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline

# data mining
#from sklearn.impute import KNNImputer, MissingIndicator, SimpleImputer
from sklearn import impute
#from sklearn_pandas import categorical_imputer, CategoricalImputer
from sklearn.pipeline import make_pipeline, make_union, Pipeline
from sklearn import preprocessing
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA

# machine learning
from sklearn import linear_model
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from sklearn.svm import LinearSVC
from sklearn.ensemble import RandomForestClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.linear_model import Perceptron
from sklearn.linear_model import SGDClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.neural_network import MLPClassifier

## scikit modeling libraries
from sklearn.ensemble import (RandomForestClassifier, AdaBoostClassifier,
                             GradientBoostingClassifier, ExtraTreesClassifier,
                             VotingClassifier)

from sklearn.model_selection import (GridSearchCV, cross_val_score, cross_val_predict,
                                     StratifiedKFold, learning_curve)

## Neural Network
from keras.models import Sequential
from keras.layers import Dense
from keras.wrappers.scikit_learn import KerasClassifier
from keras.utils import np_utils


## Load metrics for predictive modeling
from sklearn.model_selection import train_test_split
from sklearn.model_selection import cross_val_score
from sklearn.feature_selection import RFE, rfe
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
from sklearn.metrics import accuracy_score
from sklearn.metrics import roc_curve, auc
from sklearn.metrics import mean_absolute_error, mean_squared_error

## Warnings and other tools
import itertools
import warnings
warnings.filterwarnings("ignore")<jupyter_output>/opt/conda/lib/python3.7/site-packages/statsmodels/tools/_testing.py:19: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.
  import pandas.util.testing as tm
Using TensorFlow backend.<jupyter_text>*** Load datasetKaggle we provide two datasets: train and test in csv extension. So, we check and analyze only train file.<jupyter_code>#!chmod 600 /home/jovyan/.kaggle/kaggle.json
#!
#!unzip -o titanic.zip
# Load dataset train and test
train_titanic = pd.read_csv('train.csv')
test_titanic = pd.read_csv('test.csv')
ids = test_titanic['PassengerId']

# concat these two datasets, this will come handy while processing the data
titanic_list =  pd.concat(objs=[train_titanic, test_titanic], axis=0).reset_index(drop=True)
titanic_list
train_titanic.shape, test_titanic.shape
train_titanic.head(10)
train_titanic.tail(10)
titanic_list[891:]<jupyter_output><empty_output><jupyter_text>*** Data descriptionNuestro conjunto de datos tiene 12 columnas o varables, de las cuales 3 (Age, Cabin y Embarked) tienen valores no disponibles. La variable que queremos predecir es Survived, que nos indica si el pasajero sobrevivió a la tragedia del Titanic.<jupyter_code># Check dataframe structure
titanic_list.info()
# Check dataframe basic stats data
#stats(titanic_list)
# Check test dataframe basic stats data
titanic_list.describe()<jupyter_output><empty_output><jupyter_text>*** EDA, Visualization and transformation dataWe analyze all variable one by one and check null value, errors or we create new variables.<jupyter_code># Check null and NA values for both dataset
titanic_list.isna().sum()
# Table of relative frequency
titanic_list.isnull().sum()/len(titanic_list)*100<jupyter_output><empty_output><jupyter_text>We need to check those 3 features, but it must probable remove `Cabin`, there are many null values.  `PassengerId`Id of the passenger. We remove it because haven't predictive weight on our model.<jupyter_code># Check first 10 elements
titanic_list['PassengerId'].head(10)
# Remove PassengerId variable only for train dataset
titanic_list.drop(['PassengerId'], axis=1, inplace=True)
# Check train dataset
titanic_list.head()<jupyter_output><empty_output><jupyter_text>`Survived`This is our depedent variable or predictor, it check if passenger survived (`1`) or not (`0`). Almost 38% of passenger survived.<jupyter_code>titanic_list['Survived'][:891] = titanic_list['Survived'][:891].astype(int)
sns.barplot(x="Survived", data=titanic_list)
titanic_list.describe()['Survived']<jupyter_output><empty_output><jupyter_text>`Pclass`Ticket class. This is a categorical feature with 3 different values, first class, second class and third class. Exist high correlation between this feature with dependent variable.<jupyter_code>titanic_list[['Pclass', 'Survived']].groupby(['Pclass'], as_index=False).mean()
sns.barplot(x="Pclass", y="Survived", data=titanic_list)<jupyter_output><empty_output><jupyter_text>`Sex`Passenger genre. It's a categorical feature with two values, `male` y `female`. We converted it a dummy or binary value.<jupyter_code># Now visualization of 'Gender'
# Printing counts and percentage of male and female
print(titanic_list['Sex'].value_counts(sort=False))
print(titanic_list['Sex'].value_counts(sort=False,normalize=True))

# Making variable  categorical 
#sub['SEX'] = sub['SEX'].astype('category')

# Visualising counts of Gender with bar graph
sns.countplot(x="Sex", data=titanic_list);
plt.xlabel('Gender')
plt.ylabel('Frequency')
plt.title('Count of Gender')

# Showing proportion of survival for different type of gender
sns.catplot(x="Sex", y="Survived", data=titanic_list, kind="bar", ci=None)
plt.xlabel('Gender')
plt.ylabel('Survive Percentage')
plt.title('Survive v/s Sex')
# Check the survived ratio with sex
titanic_list[["Sex", "Survived"]].groupby(['Sex'], as_index=False).mean()
sns.barplot(x="Sex", y="Survived", data=titanic_list)
# Convert categorical variable to binary variable - female 1 and male 0
titanic_list['Sex'] = titanic_list['Sex'].map( {'female': 1, 'male': 0} ).astype(int)
# Check Sex features
titanic_list.head()<jupyter_output><empty_output><jupyter_text>`SibSp`Numerical feature. Indicate a sibling of passenger.<jupyter_code>titanic_list[["SibSp", "Survived"]].groupby(['SibSp'], as_index=False).mean()
sns.barplot(x="SibSp", y="Survived", data=titanic_list)<jupyter_output><empty_output><jupyter_text>`Parch`Father and childs of passenger. Numerical variable<jupyter_code>titanic_list[["Parch", "Survived"]].groupby(['Parch'], as_index=False).mean()
sns.barplot(x="Parch", y="Survived", data=titanic_list)<jupyter_output><empty_output><jupyter_text>`FamilySize`Create new feature, called FamilySize, where we summarize `SibSp` and `Parch` as numerical variable.<jupyter_code># Create a new feature, family size included passenger alone
titanic_list['FamilySize'] = titanic_list['SibSp'] + titanic_list['Parch'] + 1
titanic_list[["FamilySize", "Survived"]].groupby(['FamilySize'], as_index=False).mean()
sns.barplot(x="FamilySize", y="Survived", data=titanic_list)<jupyter_output><empty_output><jupyter_text>`IsAlone`We create new feature caracterized if passanger travel alone or not, based on familySize. The binary feature is called `IsAlone`.<jupyter_code>titanic_list['IsAlone'] = 0
titanic_list.loc[titanic_list['FamilySize'] == 1, 'IsAlone'] = 1
# Check new feature with predictor    
titanic_list[['IsAlone', 'Survived']].groupby(['IsAlone'], as_index=False).mean()
sns.barplot(x="IsAlone", y="Survived", data=titanic_list)<jupyter_output><empty_output><jupyter_text>`Ticket`Ticket number of the passanger. In first instance isn't important for the model. We removed it.<jupyter_code># We remove Ticket variable in both traing and test dataset
titanic_list.drop(['Ticket'], axis=1, inplace=True)
# We check the dataset again - train
titanic_list.head(10)<jupyter_output><empty_output><jupyter_text>`Embarked`This feature is Port of Embarkation. There are three categorical variables: `C` for Cherbourg, `Q` for Queenstown, `S` for Southampton.<jupyter_code># Check ratio Embarked and Survived variable
titanic_list[['Embarked', 'Survived']].groupby(['Embarked'], as_index=False).mean()
# Fill na or null values with the most frequent value, C
freq_port = titanic_list.Embarked.dropna().mode()[0]
freq_port
# Assign result on the dataset
titanic_list['Embarked'] = titanic_list['Embarked'].fillna(freq_port)
# Check ratio Embarked and Survived variable
titanic_list[['Embarked', 'Survived']].groupby(['Embarked'], as_index=False).mean()
sns.barplot(x="Embarked", y="Survived", data=titanic_list)<jupyter_output><empty_output><jupyter_text>`Fare`This continuous numerical variable is ticket fare of the passenger.<jupyter_code># Doing same steps before for the second dataset
s = titanic_list['Fare'].value_counts(normalize=True)
missing = titanic_list['Fare'].isnull()
titanic_list.loc[missing,'Fare'] = np.random.choice(s.index, size=len(titanic_list[missing]),
                                                      p=s.values).astype(int)
titanic_list['Fare']
sns.distplot(titanic_list['Fare'], fit=norm)<jupyter_output><empty_output><jupyter_text>After check the variable, has a normal distribution. We apply a logarithm to normalize.<jupyter_code>titanic_list['Fare'] = np.log1p(titanic_list['Fare'],)
sns.distplot(titanic_list['Fare'], fit=norm)<jupyter_output><empty_output><jupyter_text>We transform to categorical variable<jupyter_code>titanic_list['FareGroup'] = pd.qcut(titanic_list['Fare'], 5, labels=['A', 'B', 'C', 'D', 'E'])
titanic_list[['FareGroup', 'Survived']].groupby(['FareGroup'], as_index=False).mean()
sns.barplot(x="FareGroup", y="Survived", data=titanic_list)<jupyter_output><empty_output><jupyter_text>`Cabin`We transform this feature in binary variable, so it inform if he was or not in the cabin.<jupyter_code>#!pip install tabulate
pd.unique(titanic_list['Cabin'])
# Now visualization of 'Gender'
# Printing counts and percentage of male and female
print(titanic_list['Cabin'].value_counts(sort=False))
print(titanic_list['Cabin'].value_counts(sort=False,normalize=True))

# Making variable  categorical 
#sub['SEX'] = sub['SEX'].astype('category')

# Visualising counts of Gender with bar graph
sns.countplot(x="Cabin", data=titanic_list);
plt.xlabel('Cabin')
plt.ylabel('Frequency')
plt.title('Count of Gender')

# Showing proportion of survival for different type of gender
sns.catplot(x="Cabin", y="Survived", data=titanic_list, kind="bar", ci=None)
plt.xlabel('Cabin')
plt.ylabel('Survive Percentage')
plt.title('Survive v/s Cabin')
titanic_list["Cabin"].value_counts()
# Create new variable InCabin
titanic_list['InCabin'] = ~titanic_list['Cabin'].isnull()
sns.barplot(x="InCabin", y="Survived", data=titanic_list)
plt.show()
#Turning cabin number into Deck
#cabin_only = titanic_list[["Cabin"]].copy()
titanic_list["Cabin_Data"] = titanic_list["Cabin"].isnull().apply(lambda x: not x)
titanic_list["Cabin_Data"]
titanic_list["Deck"] = titanic_list["Cabin"].str.slice(0,1)
titanic_list["Room"] = titanic_list["Cabin"].str.slice(1,5).str.extract("([0-9]+)", expand=False).astype("float")
titanic_list[titanic_list["Cabin_Data"]]
titanic_list[titanic_list["Deck"]=="F"]<jupyter_output><empty_output><jupyter_text>First we'll drop the Cabin and Cabin_Data columns.<jupyter_code>titanic_list.drop(["Cabin", "Cabin_Data"], axis=1, inplace=True, errors="ignore")<jupyter_output><empty_output><jupyter_text>Now we'll deal with the missing values. For the deck column we will replace the null values with an unused letter to represent lack of data. For the room number we will simply use the mean.<jupyter_code>titanic_list["Deck"] = titanic_list["Deck"].fillna("N")
titanic_list["Room"] = round(titanic_list["Room"].fillna(titanic_list["Room"].mean()),4)
titanic_list.info()
def one_hot_column(df, label, drop_col=False):
    '''
    This function will one hot encode the chosen column.
    Args:
        df: Pandas dataframe
        label: Label of the column to encode
        drop_col: boolean to decide if the chosen column should be dropped
    Returns:
        pandas dataframe with the given encoding
    '''
    one_hot = pd.get_dummies(df[label], prefix=label)
    if drop_col:
        df = df.drop(label, axis=1)
    df = df.join(one_hot)
    return df


def one_hot(df, labels, drop_col=False):
    '''
    This function will one hot encode a list of columns.
    Args:
        df: Pandas dataframe
        labels: list of the columns to encode
        drop_col: boolean to decide if the chosen column should be dropped
    Returns:
        pandas dataframe with the given encoding
    '''
    for label in labels:
        df = one_hot_column(df, label, drop_col)
    return df
#titanic_list = one_hot(titanic_list, ["Deck"],drop_col=True)
titanic_list.head()
#cabin_only.columns.values[1:]
#for column in cabin_only.columns.values[1:]:
#    titanic_list[column] = cabin_only[column]<jupyter_output><empty_output><jupyter_text>`Age`Numerical variable with age of the passenger. We transform in categorical variable and grouped.<jupyter_code>sns.barplot(x="Age", y="Survived", data=titanic_list)
plt.show()
bins = [-1, 0, 5, 12, 18, 24, 35, 60, np.inf]
labels = ['Unknown', 'Baby', 'Child', 'Teenager', 'Student', 'Young Adult', 'Adult', 'Senior']
titanic_list["Age"] = titanic_list["Age"].fillna(-0.5)
titanic_list['AgeGroup'] = pd.cut(titanic_list["Age"], bins, labels = labels)
sns.barplot(x="AgeGroup", y="Survived", data=titanic_list)
plt.show()
titanic_list.head(5)<jupyter_output><empty_output><jupyter_text>`Name`Categorical variable with the name of the passenger. We extract from title names like as `Mr`, `Miss` or `Master`.<jupyter_code># Check the names
titanic_list['Name'].head(10)
# Create the function to extract the title
import re
def get_title(name):
    title_search = re.search(' ([A-Za-z]+)\.', name)
    if title_search:
        return title_search.group(1)
    return ""

# Apply get_title function
titanic_list['Title'] = titanic_list['Name'].apply(get_title)
# Check the results
pd.crosstab(titanic_list['Title'], titanic_list['Sex'])<jupyter_output><empty_output><jupyter_text>Create groups for all frequents titles and the other will be `Rare`.<jupyter_code>sns.barplot(x="Title", y="Survived", data=titanic_list)
plt.show()
# Convert to categorical values Title 
titanic_list["Title"] = titanic_list["Title"].replace(['Lady', 'Countess',
                                             'Capt', 'Col','Don', 'Dr', 
                                             'Major', 'Rev', 'Sir', 'Jonkheer',
                                             'Dona'], 'Rare')
# Check the results
pd.crosstab(titanic_list['Title'], titanic_list['Sex'])
# Check the results
pd.crosstab(titanic_list['Title'], titanic_list['Survived'])
titanic_list["Title"] = titanic_list["Title"].map({"Master":0, "Miss":1, "Ms" : 1 ,
                                         "Mme":1, "Mlle":1, "Mrs":1, "Mr":2, 
                                         "Rare":3})
sns.barplot(x="Title", y="Survived", data=titanic_list)
plt.show()
# Check all values and new features
titanic_list.dtypes<jupyter_output><empty_output><jupyter_text>Others feature engineeringI will test the new features using cross-validation to see if they made a difference. `Age*Class`This is an interaction term, since age and class are both numbers we can just multiply them.<jupyter_code>titanic_list['Age*Class'] = titanic_list['Age']*titanic_list['Pclass']<jupyter_output><empty_output><jupyter_text>`Fare per Person`Here we divide the fare by the number of family members traveling together, I’m not exactly sure what this represents, but it’s easy enough to add in.<jupyter_code>titanic_list['Fare_Per_Person'] = titanic_list['Fare']/(titanic_list['FamilySize']+1)<jupyter_output><empty_output><jupyter_text>Remove features<jupyter_code># Backup titanic_list 
titanic_list_bak = titanic_list
titanic_list_bak.head(5)
#titanic_list = titanic_list_bak
titanic_list.head(5)<jupyter_output><empty_output><jupyter_text>Save 1st EDA, Wrangle and Cleanse data partTo be continued 2nd part for features selection<jupyter_code># Save dataset0 and dataset1 for next step: Modeling
titanic_list.to_csv('titanic_list.csv', index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11078673176624554, 'fraction_numerical': 0.006744853274273922, 'mean_word_length': 6.0754985754985755, 'pattern_counts': {'":': 7, '<': 112, '<?xml version=': 0, '>': 112, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 14, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '28', 'dir': 'jupyter-structured-clean-dedup', 'id': '407689', 'license': 'MIT', 'n_tokens_mistral': 6620, 'n_tokens_neox': 6290, 'n_words': 1797, 'path': '07-machine-learning/labs/1_[EDA]_titanic-first-kaggle-challenge-in-python.ipynb', 'repo_name': 'RaquelMartinDiaz/NEOLAND-DS2020-datalabs', 'size': '19867'}