<jupyter_start><jupyter_text>Logistic Regression with PythonFor this lecture we will be working with the [Titanic Data Set from Kaggle](https://www.kaggle.com/c/titanic). This is a very famous data set and very often is a student's first step in machine learning! We'll be trying to predict a classification- survival or deceased.Let's begin our understanding of implementing Logistic Regression in Python for classification.We'll use a "semi-cleaned" version of the titanic data set, if you use the data set hosted directly on Kaggle, you may need to do some additional cleaning not shown in this lecture notebook. Import LibrariesLet's import some libraries to get started!<jupyter_code># linear algebra
import numpy as np 

# data processing
import pandas as pd 

# data visualization
import seaborn as sns
%matplotlib inline
from matplotlib import pyplot as plt
from matplotlib import style

# Algorithms
from sklearn import linear_model
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import Perceptron
from sklearn.linear_model import SGDClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import SVC, LinearSVC
from sklearn.naive_bayes import GaussianNB
from xgboost import XGBClassifier<jupyter_output><empty_output><jupyter_text>The DataLet's start by reading in the titanic_train.csv file into a pandas dataframe.<jupyter_code>train = pd.read_csv('C:/Users/Hp/Desktop/titanic2019/train.csv') #Train dataset
test = pd.read_csv('C:/Users/Hp/Desktop/titanic2019/test.csv') #Test dataset
train.head()
test.head()<jupyter_output><empty_output><jupyter_text>Exploratory Data AnalysisLet's begin some exploratory data analysis! We'll start by checking out missing data! Missing DataWe can use seaborn to create a simple heatmap to see where we are missing data!<jupyter_code>#train.isnull() 
sns.heatmap(train.isnull(),yticklabels= False,cbar= False, cmap='viridis')
sns.heatmap(test.isnull(),yticklabels= False,cbar= False, cmap='viridis')
#train['Age']<jupyter_output><empty_output><jupyter_text>Roughly 20 percent of the Age data is missing. The proportion of Age missing is quite small enough for any replacement with some form of imputation. However, looking at the Cabin column, it looks like we are just missing too much of that data to do something useful with it We'll probably drop this later, or change it to another feature like "Cabin Known: 1 or 0"  Comparing the rate of death and survival<jupyter_code>sns.set_style('whitegrid')
sns.countplot(x='Survived',data=train,palette='RdBu_r')<jupyter_output><empty_output><jupyter_text>Comparing the rate of death and survival of both males and females<jupyter_code>sns.set_style('whitegrid')
sns.countplot(x='Survived',hue='Sex',data=train,palette='RdBu_r')<jupyter_output><empty_output><jupyter_text>Understanding the death and survival rate as determined by the passanger class boarded<jupyter_code>sns.set_style('whitegrid')
sns.countplot(x='Survived',hue='Pclass',data=train,palette='rainbow')<jupyter_output><empty_output><jupyter_text>Understanding the distribution of age of passengers<jupyter_code>plt.figure(figsize = (10,5))
sns.distplot(train['Age'].dropna(),kde=False,color='red',bins=30)<jupyter_output><empty_output><jupyter_text>Age Distribution using matplotlib hist() function<jupyter_code>plt.figure(figsize = (10,5))
train['Age'].hist(bins=30,color='darkgreen',alpha=0.7)<jupyter_output><empty_output><jupyter_text>Understanding the presence of Siblings/Spouces on-board<jupyter_code>sns.countplot(x='SibSp',data=train)
train['Fare'].hist(color='green',bins=40,figsize=(8,4))<jupyter_output><empty_output><jupyter_text>___ Data CleaningWe want to fill in missing age data instead of just dropping the missing age data rows. One way to do this is by filling in the mean age of all the passengers (imputation).However we can be smarter about this and check the average age by passenger class. For example:<jupyter_code>plt.figure(figsize=(12, 7))
sns.boxplot(x='Pclass',y='Age',data=train,palette='winter')<jupyter_output><empty_output><jupyter_text>We can see the wealthier passengers in the higher classes tend to be older, which makes sense. We'll use these average age values to impute based on Pclass for Age.<jupyter_code>cols = train[['Age', 'Pclass']]
cols = test[['Age', 'Pclass']]
#cols[0]
def impute_age(cols):
    Age = cols[0]
    Pclass = cols[1]
    
    if pd.isnull(Age):

        if Pclass == 1:
            return 37

        elif Pclass == 2:
            return 29

        else:
            return 24

    else:
        return Age<jupyter_output><empty_output><jupyter_text>Now apply the function<jupyter_code>train['Age'] = train[['Age','Pclass']].apply(impute_age,axis=1)
test['Age'] = test[['Age','Pclass']].apply(impute_age,axis=1)<jupyter_output><empty_output><jupyter_text>Now let's check that heat map again!<jupyter_code>#train.isnull().sum()
sns.heatmap(train.isnull(),yticklabels=False,cbar=False,cmap='viridis')
sns.heatmap(test.isnull(),yticklabels=False,cbar=False,cmap='viridis')<jupyter_output><empty_output><jupyter_text>Great! Let's go ahead and drop the Cabin column<jupyter_code># we can now drop the cabin feature
train.head()
test.head()
import re
deck = {"A": 1, "B": 2, "C": 3, "D": 4, "E": 5, "F": 6, "G": 7, "U": 8}
data = [train, test]

for dataset in data:
    dataset['Cabin'] = dataset['Cabin'].fillna("U0")
    dataset['Deck'] = dataset['Cabin'].map(lambda x: re.compile("([a-zA-Z]+)").search(x).group())
    dataset['Deck'] = dataset['Deck'].map(deck)
    dataset['Deck'] = dataset['Deck'].fillna(0)
    dataset['Deck'] = dataset['Deck'].astype(int)

# we can now drop the cabin feature
train = train.drop(['Cabin'], axis=1)
test = test.drop(['Cabin'], axis=1)
test.head()<jupyter_output><empty_output><jupyter_text>Converting Categorical Features We'll need to convert categorical features to dummy variables using pandas! Otherwise our machine learning algorithm won't be able to directly take in those features as inputs.<jupyter_code>train.info()
test.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 418 entries, 0 to 417
Data columns (total 11 columns):
PassengerId    418 non-null int64
Pclass         418 non-null int64
Name           418 non-null object
Sex            418 non-null object
Age            418 non-null float64
SibSp          418 non-null int64
Parch          418 non-null int64
Ticket         418 non-null object
Fare           417 non-null float64
Embarked       418 non-null object
Deck           418 non-null int32
dtypes: float64(2), int32(1), int64(4), object(4)
memory usage: 34.4+ KB<jupyter_text>What the heck is that missing value doing in Fare column, let's help by treating it to prevent it from affecting our model<jupyter_code>data = [train, test]

for dataset in data:
    dataset['Fare'] = dataset['Fare'].fillna(0)
    dataset['Fare'] = dataset['Fare'].astype(int)<jupyter_output><empty_output><jupyter_text>Check the test dataset to be sure that the Fare missing value has been filled<jupyter_code>test.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 418 entries, 0 to 417
Data columns (total 11 columns):
PassengerId    418 non-null int64
Pclass         418 non-null int64
Name           418 non-null object
Sex            418 non-null object
Age            418 non-null float64
SibSp          418 non-null int64
Parch          418 non-null int64
Ticket         418 non-null object
Fare           418 non-null int32
Embarked       418 non-null object
Deck           418 non-null int32
dtypes: float64(1), int32(2), int64(4), object(4)
memory usage: 32.7+ KB<jupyter_text>Now We have something nice to work with, let's beat this dataset more to see if it will confess few more things to us or what do you think?<jupyter_code>train.columns.values
survived = 'survived'
not_survived = 'not survived'
fig, axes = plt.subplots(nrows=1, ncols=2,figsize=(10, 4))
women = train[train['Sex']=='female']
men = train[train['Sex']=='male']
ax = sns.distplot(women[women['Survived']==1].Age.dropna(), bins=18, label = survived, ax = axes[0], kde =False)
ax = sns.distplot(women[women['Survived']==0].Age.dropna(), bins=40, label = not_survived, ax = axes[0], kde =False)
ax.legend()
ax.set_title('Female')
ax = sns.distplot(men[men['Survived']==1].Age.dropna(), bins=18, label = survived, ax = axes[1], kde = False)
ax = sns.distplot(men[men['Survived']==0].Age.dropna(), bins=40, label = not_survived, ax = axes[1], kde = False)
ax.legend()
_ = ax.set_title('Male')
FacetGrid = sns.FacetGrid(train, row='Embarked', size=4.5, aspect=1.6)
FacetGrid.map(sns.pointplot, 'Pclass', 'Survived', 'Sex', palette=None,  order=None, hue_order=None )
FacetGrid.add_legend()
sns.barplot(x='Pclass', y='Survived', data=train)
grid = sns.FacetGrid(train, col='Survived', row='Pclass', size=2.2, aspect=1.6)
grid.map(plt.hist, 'Age', alpha=.5, bins=20)
grid.add_legend();
data = [train, test]
for dataset in data:
    dataset['relatives'] = dataset['SibSp'] + dataset['Parch']
    dataset.loc[dataset['relatives'] > 0, 'not_alone'] = 0
    dataset.loc[dataset['relatives'] == 0, 'not_alone'] = 1
    dataset['not_alone'] = dataset['not_alone'].astype(int)

train['not_alone'].value_counts()
axes = sns.factorplot('relatives','Survived', 
                      data=train, aspect = 2.5, )
train = train.drop(['PassengerId'], axis=1)
train.head()<jupyter_output><empty_output><jupyter_text>Let's get the title of individuals from their name<jupyter_code>data = [train, test]
titles = {"Mr": 1, "Miss": 2, "Mrs": 3, "Master": 4, "Rare": 5}

for dataset in data:
    # extract titles
    dataset['Title'] = dataset.Name.str.extract(' ([A-Za-z]+)\.', expand=False)
    # replace titles with a more common title or as Rare
    dataset['Title'] = dataset['Title'].replace(['Lady', 'Countess','Capt', 'Col','Don', 'Dr',\
                                            'Major', 'Rev', 'Sir', 'Jonkheer', 'Dona'], 'Rare')
    dataset['Title'] = dataset['Title'].replace('Mlle', 'Miss')
    dataset['Title'] = dataset['Title'].replace('Ms', 'Miss')
    dataset['Title'] = dataset['Title'].replace('Mme', 'Mrs')
    # convert titles into numbers
    dataset['Title'] = dataset['Title'].map(titles)
    # filling NaN with 0, to get safe
    dataset['Title'] = dataset['Title'].fillna(0)

train = train.drop(['Name'], axis=1)
test = test.drop(['Name'], axis=1)
train.head()<jupyter_output><empty_output><jupyter_text>Sex of individual could be coded into 0 for Male and 1 for Female<jupyter_code>genders = {"male": 0, "female": 1}
data = [train, test]

for dataset in data:
    dataset['Sex'] = dataset['Sex'].map(genders)<jupyter_output><empty_output><jupyter_text>The port of Embarkation could also be a factor to determine whether an individual survive or not<jupyter_code>ports = {"S": 0, "C": 1, "Q": 2}
data = [train, test]

for dataset in data:
    dataset['Embarked'] = dataset['Embarked'].map(ports)
train.head()
data = [train, test]
for dataset in data:
    dataset['Age_Class']= dataset['Age']* dataset['Pclass']<jupyter_output><empty_output><jupyter_text>Could fare per individual also have any effect on our model performance, let create a column for this and hopefully we can get something nice from it<jupyter_code>for dataset in data:
    dataset['Fare_Per_Person'] = dataset['Fare']/(dataset['relatives']+1)
    dataset['Fare_Per_Person'] = dataset['Fare_Per_Person'].astype(int)
train.head(10)<jupyter_output><empty_output><jupyter_text>We might have to drop Ticket variable to avoid getting errors while parsing our data into the model.<jupyter_code>train = train.drop(['Ticket'], axis=1)
test = test.drop(['Ticket'], axis=1)
train.head()
train = train.drop(['not_alone'], axis=1)
test = test.drop(['not_alone'], axis=1)
train = train.drop(['Parch'], axis=1)
test = test.drop(['Parch'], axis=1)
X_train = train.drop("Survived", axis=1)
Y_train = train["Survived"]
X_test  = test.drop("PassengerId", axis=1).copy()<jupyter_output><empty_output><jupyter_text>Great! Our data is ready for our model! Training and Prediciting Trying different model and picking the best would help in achieving a higher ranking on the leaderboard Here we will parse our train data into Random Forest and predict our test data<jupyter_code>random_forest = RandomForestClassifier(n_estimators=100000)
random_forest.fit(X_train, Y_train)

Y_prediction = random_forest.predict(X_test)

random_forest.score(X_train, Y_train)
acc_random_forest = round(random_forest.score(X_train, Y_train) * 100, 2)
submit_xgb = XGBClassifier()
submit_xgb = model_selection.GridSearchCV(XGBClassifier(), param_grid= {'learning_rate': grid_learn, 'max_depth': [0,2,4,6,8,10], 'n_estimators': grid_n_estimator, 'seed': grid_seed}, scoring = 'roc_auc', cv = cv_split)
submit_xgb.fit(data1[data1_x_bin], data1[Target])
print('Best Parameters: ', submit_xgb.best_params_) #Best Parameters:  {'learning_rate': 0.01, 'max_depth': 4, 'n_estimators': 300, 'seed': 0}
data_val['Survived'] = submit_xgb.predict(data_val[data1_x_bin])<jupyter_output><empty_output><jupyter_text>Here we will parse our train data into Logistic Regression Model to  predict our test data<jupyter_code>logmodel = LogisticRegression()
logmodel.fit(X_train, Y_train)

Y_pred = logmodel.predict(X_test)

acc_log = round(logmodel.score(X_train, Y_train) * 100, 2)<jupyter_output>C:\Users\Hp\Anaconda3\lib\site-packages\sklearn\linear_model\logistic.py:433: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.
  FutureWarning)<jupyter_text>Here we will parse our train data into SGDClassifier Model to predict our test data<jupyter_code>sgd = linear_model.SGDClassifier(max_iter=5, tol=None)
sgd.fit(X_train, Y_train)
Y_pred = sgd.predict(X_test)

sgd.score(X_train, Y_train)

acc_sgd = round(sgd.score(X_train, Y_train) * 100, 2)<jupyter_output>C:\Users\Hp\Anaconda3\lib\site-packages\sklearn\linear_model\stochastic_gradient.py:183: FutureWarning: max_iter and tol parameters have been added in SGDClassifier in 0.19. If max_iter is set but tol is left unset, the default value for tol in 0.19 and 0.20 will be None (which is equivalent to -infinity, so it has no effect) but will change in 0.21 to 1e-3. Specify tol to silence this warning.
  FutureWarning)<jupyter_text>Here we will parse our train data into KNNClassifier Model to predict our test data<jupyter_code>knn = KNeighborsClassifier(n_neighbors = 3) 
knn.fit(X_train, Y_train)  
Y_pred = knn.predict(X_test)  
acc_knn = round(knn.score(X_train, Y_train) * 100, 2)<jupyter_output><empty_output><jupyter_text>Here we will parse our train data into GuassianDB Model to predict our test data<jupyter_code>gaussian = GaussianNB() 
gaussian.fit(X_train, Y_train)  
Y_pred = gaussian.predict(X_test)  
acc_gaussian = round(gaussian.score(X_train, Y_train) * 100, 2)<jupyter_output><empty_output><jupyter_text>Here we will parse our train data into Perceptrn Model to predict our test data<jupyter_code>perceptron = Perceptron(max_iter=5)
perceptron.fit(X_train, Y_train)

Y_pred = perceptron.predict(X_test)

acc_perceptron = round(perceptron.score(X_train, Y_train) * 100, 2)<jupyter_output>C:\Users\Hp\Anaconda3\lib\site-packages\sklearn\linear_model\stochastic_gradient.py:183: FutureWarning: max_iter and tol parameters have been added in Perceptron in 0.19. If max_iter is set but tol is left unset, the default value for tol in 0.19 and 0.20 will be None (which is equivalent to -infinity, so it has no effect) but will change in 0.21 to 1e-3. Specify tol to silence this warning.
  FutureWarning)<jupyter_text>Here we will parse our train data into LinearSVC Model to predict our test data<jupyter_code>linear_svc = LinearSVC()
linear_svc.fit(X_train, Y_train)

Y_pred = linear_svc.predict(X_test)

acc_linear_svc = round(linear_svc.score(X_train, Y_train) * 100, 2)<jupyter_output>C:\Users\Hp\Anaconda3\lib\site-packages\sklearn\svm\base.py:931: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.
  "the number of iterations.", ConvergenceWarning)<jupyter_text>Here we will parse our train data into Decision Tree Classifier Model to predict our test data<jupyter_code>decision_tree = DecisionTreeClassifier() 
decision_tree.fit(X_train, Y_train)  
Y_pred = decision_tree.predict(X_test)  
acc_decision_tree = round(decision_tree.score(X_train, Y_train) * 100, 2)<jupyter_output><empty_output><jupyter_text>Now let's the prediction accuracy score of each model on our test set<jupyter_code>results = pd.DataFrame({
    'Model': ['Support Vector Machines', 'KNN', 'Logistic Regression', 
              'Random Forest', 'Naive Bayes', 'Perceptron', 
              'Stochastic Gradient Decent', 
              'Decision Tree'],
    'Score': [acc_linear_svc, acc_knn, acc_log, 
              acc_random_forest, acc_gaussian, acc_perceptron, 
              acc_sgd, acc_decision_tree]})
result_dataf = results.sort_values(by='Score', ascending=False)
result_dataf = result_dataf.set_index('Score')
result_dataf.head(9)<jupyter_output><empty_output><jupyter_text>Wow! Wow! Wow! This is just so awesome! Good job there to Random Forest model and Decision Tree but could we do more by chosing the best model  Let's cross-validate our score for Random Forest Classifier<jupyter_code>from sklearn.model_selection import cross_val_score
rf = RandomForestClassifier(n_estimators=100)
scores = cross_val_score(rf, X_train, Y_train, cv=10, scoring = "accuracy")

print("Scores:", scores)
print("Mean:", scores.mean())
print("Standard Deviation:", scores.std())<jupyter_output>Scores: [0.75555556 0.84444444 0.75280899 0.86516854 0.88764045 0.87640449
 0.84269663 0.75280899 0.8988764  0.82954545]
Mean: 0.8305949948927477
Standard Deviation: 0.05411282315275784<jupyter_text>Check Feature importance here<jupyter_code>importances = pd.DataFrame({'feature':X_train.columns,'importance':np.round(random_forest.feature_importances_,3)})
importances = importances.sort_values('importance',ascending=False).set_index('feature')

importances.head(15)
importances.plot.bar()
X_test.head()
# Random Forest

random_forest = RandomForestClassifier(n_estimators=100, oob_score = True)
random_forest.fit(X_train, Y_train)
Y_prediction = random_forest.predict(X_test)

random_forest.score(X_train, Y_train)

acc_random_forest = round(random_forest.score(X_train, Y_train) * 100, 2)
print(round(acc_random_forest,2,), "%")
print("oob score:", round(random_forest.oob_score_, 4)*100, "%")<jupyter_output>oob score: 81.14 %<jupyter_text>Now it's time to choose our best parameter<jupyter_code>param_grid = { "criterion" : ["gini", "entropy"], "min_samples_leaf" : [1, 5, 10, 25, 50, 70], "min_samples_split" : [2, 4, 10, 12, 16, 18, 25, 35], "n_estimators": [100, 400, 700, 1000, 1500]}

from sklearn.model_selection import GridSearchCV, cross_val_score

rf = RandomForestClassifier(n_estimators=100, max_features='auto', oob_score=True, random_state=1, n_jobs=-1)

clf = GridSearchCV(estimator=rf, param_grid=param_grid, n_jobs=-1)

clf.fit(X_train, Y_train)

print (clf.best_params_)
# Random Forest
random_forest = RandomForestClassifier(criterion = "gini", 
                                       min_samples_leaf = 1, 
                                       min_samples_split = 10,   
                                       n_estimators=100, 
                                       max_features='auto', 
                                       oob_score=True, 
                                       random_state=1, 
                                       n_jobs=-1)

random_forest.fit(X_train, Y_train)
Y_prediction = random_forest.predict(X_test)

random_forest.score(X_train, Y_train)

print("oob score:", round(random_forest.oob_score_, 4)*100, "%")
from sklearn.model_selection import cross_val_predict
from sklearn.metrics import confusion_matrix
predictions = cross_val_predict(random_forest, X_train, Y_train, cv=3)
confusion_matrix(Y_train, predictions)
from sklearn.metrics import precision_score, recall_score

print("Precision:", precision_score(Y_train, predictions))
print("Recall:",recall_score(Y_train, predictions))
from sklearn.metrics import f1_score
f1_score(Y_train, predictions)
from sklearn.metrics import precision_recall_curve

# getting the probabilities of our predictions
y_scores = random_forest.predict_proba(X_train)
y_scores = y_scores[:,1]

precision, recall, threshold = precision_recall_curve(Y_train, y_scores)

def plot_precision_and_recall(precision, recall, threshold):
    plt.plot(threshold, precision[:-1], "r-", label="precision", linewidth=5)
    plt.plot(threshold, recall[:-1], "b", label="recall", linewidth=5)
    plt.xlabel("threshold", fontsize=19)
    plt.legend(loc="upper right", fontsize=19)
    plt.ylim([0, 1])

plt.figure(figsize=(14, 7))
plot_precision_and_recall(precision, recall, threshold)
plt.show()
def plot_precision_vs_recall(precision, recall):
    plt.plot(recall, precision, "g--", linewidth=2.5)
    plt.ylabel("recall", fontsize=19)
    plt.xlabel("precision", fontsize=19)
    plt.axis([0, 1.5, 0, 1.5])

plt.figure(figsize=(14, 7))
plot_precision_vs_recall(precision, recall)
plt.show()
from sklearn.metrics import roc_curve
# compute true positive rate and false positive rate
false_positive_rate, true_positive_rate, thresholds = roc_curve(Y_train, y_scores)

# plotting them against each other
def plot_roc_curve(false_positive_rate, true_positive_rate, label=None):
    plt.plot(false_positive_rate, true_positive_rate, linewidth=2, label=label)
    plt.plot([0, 1], [0, 1], 'r', linewidth=4)
    plt.axis([0, 1, 0, 1])
    plt.xlabel('False Positive Rate (FPR)', fontsize=16)
    plt.ylabel('True Positive Rate (TPR)', fontsize=16)

plt.figure(figsize=(14, 7))
plot_roc_curve(false_positive_rate, true_positive_rate)
plt.show()
from sklearn.metrics import roc_auc_score
r_a_score = roc_auc_score(Y_train, y_scores)
print("ROC-AUC-Score:", r_a_score)
new_submission = pd.DataFrame({'PassengerId':test['PassengerId'], 'Survived': new_pred})
new_submission.head()
#predictions
X_pred = random_forest.predict(X_test)
print (X_pred)
new_submission = pd.DataFrame({'PassengerId':test['PassengerId'], 'Survived': X_pred})
new_submission.head()
filename = 'Abel_Titanic_Prediction2.csv'
new_submission.to_csv(filename, index=False)
print ('Saved file as:' + filename)<jupyter_output>Saved file as:Abel_Titanic_Prediction2.csv
{'fraction_non_alphanumeric': 0.1224217638691323, 'fraction_numerical': 0.02787162162162162, 'mean_word_length': 5.49639041293676, 'pattern_counts': {'":': 19, '<': 134, '<?xml version=': 0, '>': 135, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '35', 'dir': 'jupyter-structured-clean-dedup', 'id': '329274', 'license': 'Apache-2.0', 'n_tokens_mistral': 8240, 'n_tokens_neox': 7547, 'n_words': 1939, 'path': 'Logistic Regression with Python.ipynb', 'repo_name': 'ooluwaseunabel/Titanic-Survivorship-Prediction-with-R-and-Python', 'size': '22496'}