<jupyter_start><jupyter_text>The following is an attempt at predcting if a passenger on Titanic survived or not based on attributes such as:    Passenger Class, Sex, Age, Number of siblings/spouses, Fare, Port of embarkmentMore details about the dataset can be found here: https://www.kaggle.com/c/titanic/data<jupyter_code># We import the required libraries

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline

from collections import Counter
# Load train and test datasets

train = pd.read_csv("/Users/ankit/Desktop/Titanic Dataset/train.csv")
test = pd.read_csv("/Users/ankit/Desktop/Titanic Dataset/test.csv")
train.head()<jupyter_output><empty_output><jupyter_text>Since outliers can have a dramatic effect on the prediction, we first deal with them.We have used the Tukey method to detect ouliers which defines an interquartile range comprised between the 1st and 3rd quartile of the distribution values (IQR). An outlier is a row that have a feature value outside the (IQR +- an outlier step).We detect outliers from the numerical values features (Age, SibSp, Sarch and Fare). Then, we consider outliers as rows that have at least two outlied numerical values.<jupyter_code># Outlier detection 
# Reference - https://www.kaggle.com/yassineghouzam/titanic-top-4-with-ensemble-modeling/notebook

def detect_outliers(df,n,features):
    """
    Takes a dataframe df of features and returns a list of the indices
    corresponding to the observations containing more than n outliers according
    to the Tukey method.
    """
    outlier_indices = []
    
    # iterate over features(columns)
    for col in features:
        # 1st quartile (25%)
        Q1 = np.percentile(df[col], 25)
        # 3rd quartile (75%)
        Q3 = np.percentile(df[col],75)
        # Interquartile range (IQR)
        IQR = Q3 - Q1
        
        # outlier step
        outlier_step = 1.5 * IQR
        
        # Determine a list of indices of outliers for feature col
        outlier_list_col = df[(df[col] < Q1 - outlier_step) | (df[col] > Q3 + outlier_step )].index
        
        # append the found outlier indices for col to the list of outlier indices 
        outlier_indices.extend(outlier_list_col)
        
    # select observations containing more than 2 outliers
    outlier_indices = Counter(outlier_indices)        
    multiple_outliers = list( k for k, v in outlier_indices.items() if v > n )
    
    return multiple_outliers   

# detect outliers from Age, SibSp , Parch and Fare
Outliers_to_drop = detect_outliers(train,2,["Age","SibSp","Parch","Fare"])

# Show the outliers rows
train.loc[Outliers_to_drop]<jupyter_output><empty_output><jupyter_text>We detect 10 outliers. The 28, 89 and 342 passenger have an high Ticket FareThe 7 others have very high values of SibSP.<jupyter_code># Drop outliers
train = train.drop(Outliers_to_drop, axis = 0).reset_index(drop=True)<jupyter_output><empty_output><jupyter_text>Joining train and test set<jupyter_code>## Join train and test datasets in order to obtain the same number of features during categorical conversion
train_len = len(train)
dataset =  pd.concat(objs=[train, test], axis=0).reset_index(drop=True)<jupyter_output><empty_output><jupyter_text>Check for null and missing values<jupyter_code># Fill empty and NaNs values with NaN
dataset = dataset.fillna(np.nan)

# Check for Null values
dataset.isnull().sum()<jupyter_output><empty_output><jupyter_text>** Age and Cabin features have an important part of missing values. **** Survived missing values correspond to the join testing dataset (Survived column doesn't exist in test set and has been replace by NaN values when concatenating the train and test set) **<jupyter_code>dataset.head()
#Removing all rows which do not have a 'Survived' column value 
dataset.dropna(subset = ['Survived'],inplace=True)
dataset.Survived.isnull().sum()
dataset.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 881 entries, 0 to 880
Data columns (total 12 columns):
Age            711 non-null float64
Cabin          201 non-null object
Embarked       879 non-null object
Fare           881 non-null float64
Name           881 non-null object
Parch          881 non-null int64
PassengerId    881 non-null int64
Pclass         881 non-null int64
Sex            881 non-null object
SibSp          881 non-null int64
Survived       881 non-null float64
Ticket         881 non-null object
dtypes: float64(3), int64(4), object(5)
memory usage: 89.5+ KB<jupyter_text>** Splitting into features and labels **<jupyter_code># converting the label values from float to int

y = dataset.Survived.astype(int)
y.head()
x = dataset.drop('Survived',axis = 1)
x.head()<jupyter_output><empty_output><jupyter_text>Seperating train and test datasets<jupyter_code>## Separate train dataset and test dataset

from sklearn.model_selection import train_test_split
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2, random_state=42, stratify = y)
x_train.head()<jupyter_output><empty_output><jupyter_text>** Selecting features on which to train our models **<jupyter_code>x_train_cols = x_train[['Fare','Parch','Pclass','Sex','SibSp']]
x_train_cols.head()
# Doing the same with the test set
x_test_cols = x_test[['Fare','Parch','Pclass','Sex','SibSp']]<jupyter_output><empty_output><jupyter_text>Creating dummy variables<jupyter_code># Get dummy variables for categorical feature 'Sex' thus encoding it as 0 and 1
x_train_cat1 = x_train['Sex'].str.get_dummies()
x_train_cat1.head()
# Doing the same with the test set
x_test_cat1 = x_test['Sex'].str.get_dummies()
# Concatenate both the dataframes along axis = 1 (columns)
x_train_cols = pd.concat(objs = [x_train_cols, x_train_cat1],axis = 1)
x_train_cols.head()
# Doing the same with the test set
x_test_cols = pd.concat(objs = [x_test_cols, x_test_cat1],axis = 1)
# Drop categorical 'Sex' column
x_train_cols.drop(columns = 'Sex',axis=1,inplace=True)
x_train_cols.head()
# Doing the same with the test set
x_test_cols.drop (columns = 'Sex',axis=1,inplace=True)
x_test_cols.head()
y_train.head()<jupyter_output><empty_output><jupyter_text>Decision Tree<jupyter_code>from sklearn.tree import DecisionTreeClassifier
tree_clf = DecisionTreeClassifier(random_state=42,max_depth = 3)
tree_clf.fit(x_train_cols,y_train)
y_pred = tree_clf.predict(x_test_cols)
from sklearn.metrics import classification_report,confusion_matrix
print('Confusion Matrix:')
print('\n')
print(confusion_matrix(y_test,y_pred))
print('\n')
print('Classification Report:')
print('\n')
print(classification_report(y_test,y_pred))<jupyter_output>Confusion Matrix:


[[97 12]
 [27 41]]


Classification Report:


             precision    recall  f1-score   support

          0       0.78      0.89      0.83       109
          1       0.77      0.60      0.68        68

avg / total       0.78      0.78      0.77       177<jupyter_text>Bagging ** Reference: **** <NAME>. “Hands-On Machine Learning with Scikit-Learn and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems. ** Bagging (Bootstrap Aggregating - sampling with replacement)<jupyter_code>from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier

bag_clf = BaggingClassifier(
    DecisionTreeClassifier(), n_estimators=500,
    max_samples=100, bootstrap=True, n_jobs=-1)
bag_clf.fit(x_train_cols, y_train)
y_pred_bag = bag_clf.predict(x_test_cols)<jupyter_output><empty_output><jupyter_text>Out of Bag Boosting With bagging, some instances might be sampled more than once while others might not be sampled at all. On average only about 63% of training instances are sampled. The remaining 37% of instances are called out-of-bag instances and since our training model has never seen this set of instances from the training set, it can be evaluated based on these out-of-bag instances.<jupyter_code>oob_clf = BaggingClassifier(DecisionTreeClassifier(), n_estimators=500,bootstrap=True, n_jobs=-1, oob_score=True)
oob_clf.fit(x_train_cols, y_train)
oob_clf.oob_score_<jupyter_output><empty_output><jupyter_text>Random Forest We are going to train a Random Forest model, which is an ensemble of trees where sampling is done by the Bagging (Bootstrap Aggregating - sampling with replacement) We are also going to specify some hyperparameters as follows:• n_estimators = number of decision trees in the forest. Random forest aggregates all predictions via either hardsoft voting.• max_leaf_nodes = maximum number of leaf nodes• n_jobs = number of CPU cores to be utilized for the job. n=-1 means engaging all cores.<jupyter_code>from sklearn.ensemble import RandomForestClassifier
rf_clf = RandomForestClassifier(n_estimators=500, max_leaf_nodes=16, n_jobs=-1 ,random_state=42)
rf_clf.fit(x_train_cols,y_train)
y_pred_rf=rf_clf.predict(x_test_cols)
print('Confusion Matrix:')
print('\n')
print(confusion_matrix(y_test,y_pred_rf))
print('\n')
print('Classification Report:')
print('\n')
print(classification_report(y_test,y_pred_rf))<jupyter_output>Confusion Matrix:


[[99 10]
 [27 41]]


Classification Report:


             precision    recall  f1-score   support

          0       0.79      0.91      0.84       109
          1       0.80      0.60      0.69        68

avg / total       0.79      0.79      0.78       177<jupyter_text>** Random Forest only gives us slightly better results **  Hyperparameter Tuning<jupyter_code>from sklearn.model_selection import GridSearchCV<jupyter_output><empty_output><jupyter_text>The model is tuned on following hyperparameters using a grid search:    • min_samples_split: The number of samples in the node for the tree to split on• bootstrap: Smapling with replacement• n_estimators: Number of decsiion trees in our forest• criterion: defines the cost function which the forest tries to minimize in order ot split the node. gini is a measure of the purity of the node<jupyter_code>rf_param_grid = {"max_depth": [None],
              "min_samples_split": [2, 3, 10],
              "min_samples_leaf": [1, 3, 10],
              "bootstrap": [False],
              "n_estimators" :[100,300],
              "criterion": ['gini','entropy']}
gsRFC = GridSearchCV(rf_clf,param_grid = rf_param_grid,n_jobs = -1)
gsRFC.fit(x_train_cols,y_train)
# Best Estimator
RFC_best = gsRFC.best_estimator_
RFC_best
# Best score
gsRFC.best_score_<jupyter_output><empty_output><jupyter_text>Boosting The main idea behind boosting methods is to train predictors sequentially, each trying to correct its predecessor.Below I display AdaBoost, one of the most popular boosting methods.  AdaBoost Here, we train an AdaBoost classifier based on 200 Decision Stumps (Decision trees with the max_depth hyperparameter set to 1).<jupyter_code>from sklearn.ensemble import AdaBoostClassifier

ada_clf = AdaBoostClassifier(
    DecisionTreeClassifier(max_depth=1), n_estimators=200,
    algorithm="SAMME.R", learning_rate=0.5)
ada_clf.fit(x_train_cols, y_train)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08746937664458761, 'fraction_numerical': 0.024226476726249887, 'mean_word_length': 4.511, 'pattern_counts': {'":': 5, '<': 69, '<?xml version=': 0, '>': 70, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '362417', 'license': 'MIT', 'n_tokens_mistral': 3781, 'n_tokens_neox': 3486, 'n_words': 1081, 'path': 'King of the World - Titanic Survical.ipynb', 'repo_name': 'ankitsingh0803/Data-Science-and-Machine-Learning-Projects', 'size': '11029'}