<jupyter_start><jupyter_text>From LinkedIn Learninghttps://www.linkedin.com/learning/python-for-data-science-essential-training-part-2/logistic-regression-treat-missing-values?u=36492188<jupyter_code>import numpy as np
import pandas as pd
import seaborn as sb
import matplotlib.pyplot as plt

from pandas import Series, DataFrame
from pylab import rcParams
from sklearn import preprocessing
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.model_selection import cross_val_predict
from sklearn import metrics  # why need this line?
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
from sklearn.metrics import precision_score, recall_score
%matplotlib inline
rcParams['figure.figsize'] = 5,4
sb.set_style('whitegrid')<jupyter_output><empty_output><jupyter_text>Logistic Regression on the Titanic Dataset<jupyter_code>address = "data/titanic-training-data.csv"
titanic_training = pd.read_csv(address)
titanic_training.head()
# don't really need the following, it is already in the csv header file
# titanic_training.columns = ['PassengerId', 'Survived', 'Pclass', 'Name', 'Sex', 'Age', 'SibSp', 'Parch', 'Ticket', 'Fare', 'Cabin', 'Embarked']
titanic_training.columns
titanic_training.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 891 entries, 0 to 890
Data columns (total 12 columns):
 #   Column       Non-Null Count  Dtype  
---  ------       --------------  -----  
 0   PassengerId  891 non-null    int64  
 1   Survived     891 non-null    int64  
 2   Pclass       891 non-null    int64  
 3   Name         891 non-null    object 
 4   Sex          891 non-null    object 
 5   Age          714 non-null    float64
 6   SibSp        891 non-null    int64  
 7   Parch        891 non-null    int64  
 8   Ticket       891 non-null    object 
 9   Fare         891 non-null    float64
 10  Cabin        204 non-null    object 
 11  Embarked     889 non-null    object 
dtypes: float64(2), int64(5), object(5)
memory usage: 83.7+ KB<jupyter_text>VARIABLE DESCRIPTIONSSurvived - Survival (0 = No; 1 = Yes)Pclass - Passenger Class (1 = 1st; 2 = 2nd; 3 = 3rd)Name - NameSex - SexAge - AgeSibSp - Number of Siblings/Spouses AboardParch - Number of Parents/Children AboardTicket - Ticket NumberFare - Passenger Fare (British pound)Cabin - CabinEmbarked - Port of Embarkation (C = Cherbourg, France; Q = Queenstown, UK; S = Southampton - Cobh, Ireland)  Checking that your target variable is binary<jupyter_code># target is Surivied column
sb.countplot(x='Survived', data=titanic_training)<jupyter_output><empty_output><jupyter_text>Checking for missing values<jupyter_code>titanic_training.isnull().sum()
# missing value is also in the info() called above
# using describe() to check for missing records.  Note Age has 714 count.  But only numerical columns showed.  Embark not shown.
titanic_training.describe()<jupyter_output><empty_output><jupyter_text>Taking care of missing values Dropping missing valuesSo let's just go ahead and drop all the variables that aren't relevant for predicting survival. We should at least keep the following:- Survived - This variable is obviously relevant.- Pclass - Does a passenger's class on the boat affect their survivability?- Sex - Could a passenger's gender impact their survival rate?- Age - Does a person's age impact their survival rate?- SibSp - Does the number of relatives on the boat (that are siblings or a spouse) affect a person survivability? Probability- Parch - Does the number of relatives on the boat (that are children or parents) affect a person survivability? Probability- Fare - Does the fare a person paid effect his survivability? Maybe - let's keep it.- Embarked - Does a person's point of embarkation matter? It depends on how the boat was filled... Let's keep it.What about a person's name, ticket number, and passenger ID number? They're irrelavant for predicting survivability. And as you recall, the cabin variable is almost all missing values, so we can just drop all of these.<jupyter_code>titanic_data = titanic_training.drop(['Name', 'Ticket', 'Cabin'], axis=1)
titanic_data.head()<jupyter_output><empty_output><jupyter_text>Imputing missing values<jupyter_code># Age is missing data.  891 vs 714 (see count above)
# Look at the Parch and Age relationship
sb.boxplot(x='Parch', y='Age', data=titanic_data)<jupyter_output><empty_output><jupyter_text>Here we can see for younger age, the Parch is 1 or 2. For children, likely to have 1 or more parent onboard. For Parch 4, the age is from 39 - 50.  This indicate the parent having children with them onboard. Using this data to impute missing values in Age column.<jupyter_code>Parch_groups = titanic_data.groupby(titanic_data['Parch'])
Parch_groups.mean()
# going to use data in Parch to impute data for Age
def age_approx(cols):
    Age = cols[0]
    Parch = cols[1]
    if pd.isnull(Age):
        if Parch == 0:
            return 32 # mean for the Parch group
        elif Parch == 1:
            return 24
        elif Parch == 2:
            return 17
        elif Parch == 3:
            return 33
        elif Parch == 4:
            return 45
        else:
            return 30 # mean of the whole ship
    else:
        return Age        
titanic_data['Age'] = titanic_data[['Age', 'Parch']].apply(age_approx, axis=1)
titanic_data.isnull().sum()
# still missing values in Embarked column
# only 2 records, dropping it should not affect much, so drop the 2 records.
titanic_data.dropna(inplace=True)
titanic_data.reset_index(inplace=True, drop=True)
titanic_data.info()  # 889 records left<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 889 entries, 0 to 888
Data columns (total 9 columns):
 #   Column       Non-Null Count  Dtype  
---  ------       --------------  -----  
 0   PassengerId  889 non-null    int64  
 1   Survived     889 non-null    int64  
 2   Pclass       889 non-null    int64  
 3   Sex          889 non-null    object 
 4   Age          889 non-null    float64
 5   SibSp        889 non-null    int64  
 6   Parch        889 non-null    int64  
 7   Fare         889 non-null    float64
 8   Embarked     889 non-null    object 
dtypes: float64(2), int64(5), object(2)
memory usage: 62.6+ KB<jupyter_text>Converting categorical variables to a dummy indicators<jupyter_code># reformat Sex, Embark columns
from sklearn.preprocessing import LabelEncoder
label_encoder = LabelEncoder()
gender_cat = titanic_data['Sex']
gender_encoded = label_encoder.fit_transform(gender_cat)
gender_encoded[0:5]
# what is 1 or 0?  Check the dataframe
titanic_data.head()
# 1 = Male, 2 = Female
gender_DF = pd.DataFrame(gender_encoded, columns=['male_gender'])
gender_DF.head()
# now work on Embark
embarked_cat = titanic_data['Embarked']
embarked_encoded = label_encoder.fit_transform(embarked_cat)
embarked_encoded[0:100]
from sklearn.preprocessing import OneHotEncoder
binary_encoder = OneHotEncoder(categories='auto')
embarked_1hot = binary_encoder.fit_transform(embarked_encoded.reshape(-1,1))
embarked_1hot_mat = embarked_1hot.toarray()
embarked_DF = pd.DataFrame(embarked_1hot_mat, columns=['C','Q','S'])
embarked_DF.head()
titanic_data.drop(['Sex','Embarked'], axis=1, inplace=True)
titanic_data.head()
# concatenate the generated dataframe as columns)
titanic_dmy = pd.concat([titanic_data, gender_DF, embarked_DF], axis=1, verify_integrity=1)
titanic_dmy[0:5]<jupyter_output><empty_output><jupyter_text>Checking for independence between features<jupyter_code># use heatmap to check for corelations
sb.heatmap(titanic_dmy.corr());
# if the value is close to 1 or -1, it means there is a strong corelation bert the variables<jupyter_output><empty_output><jupyter_text>Logistic Regression assumption Independent variables/Features are independent of each otherLook for 1 (light color) or -1 (dark color)PClass vs Fare is correlated, so need to drop 1 of them.<jupyter_code># Why drop both?  Should not I just drop 1 of them.
titanic_dmy = titanic_dmy.drop(['Pclass','Fare'], axis=1)
titanic_dmy.head()<jupyter_output><empty_output><jupyter_text>Checking that your dataset size is sufficientGood rule of thumb is that we should have 50 records per predictive featuresHow many predictive variables?  We have 6 predictive features.  C|Q|S is considered 1 as they are dummy variables for Embarkedthus, we need at least 50x6 records for this model.<jupyter_code>titanic_dmy.info()
# we have 889 records.  Thus we have enough to run the regression.<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 889 entries, 0 to 888
Data columns (total 9 columns):
 #   Column       Non-Null Count  Dtype  
---  ------       --------------  -----  
 0   PassengerId  889 non-null    int64  
 1   Survived     889 non-null    int64  
 2   Age          889 non-null    float64
 3   SibSp        889 non-null    int64  
 4   Parch        889 non-null    int64  
 5   male_gender  889 non-null    int32  
 6   C            889 non-null    float64
 7   Q            889 non-null    float64
 8   S            889 non-null    float64
dtypes: float64(4), int32(1), int64(4)
memory usage: 59.2 KB<jupyter_text>Split the Data into Training and Test for Logistic Regression<jupyter_code># Split training and test set
X_train, X_test, y_train, y_test = train_test_split(titanic_dmy.drop('Survived',axis=1), titanic_dmy['Survived'], test_size=0.2, random_state=200)
print(X_train.shape)
print(y_train.shape)
# print out to check the data and see the columns are ok
X_train[0:5]<jupyter_output><empty_output><jupyter_text>Deploying and evaluating the model<jupyter_code>LogReg = LogisticRegression(solver='liblinear')
LogReg.fit(X_train, y_train)
y_pred = LogReg.predict(X_test)<jupyter_output><empty_output><jupyter_text>Model Evaluation Classification report without cross-validation<jupyter_code>print(classification_report(y_test, y_pred))<jupyter_output>precision    recall  f1-score   support

           0       0.83      0.88      0.85       109
           1       0.79      0.71      0.75        69

    accuracy                           0.81       178
   macro avg       0.81      0.80      0.80       178
weighted avg       0.81      0.81      0.81       178<jupyter_text>We are getting 0.81 (81%) accuracy, not bad  K-fold cross-validation & confusion matrices<jupyter_code>y_train_pred = cross_val_predict(LogReg, X_train, y_train, cv=5)
confusion_matrix(y_train, y_train_pred)
# 377, 180 is the correct prediction.  True Positive and True Negative
precision_score(y_train, y_train_pred)<jupyter_output><empty_output><jupyter_text>Make a test prediction<jupyter_code># Create a fake record for testing
# Look at one existing record, use as source
titanic_dmy[863:864]
# set up the fake record based on the the record above
test_passenger = np.array([866, 40, 0, 0, 0, 0, 0, 1]).reshape(1,-1)
print(LogReg.predict(test_passenger))
print(LogReg.predict_proba(test_passenger))<jupyter_output>[1]
[[0.26351831 0.73648169]]
{'fraction_non_alphanumeric': 0.08897766479026693, 'fraction_numerical': 0.03985836208461958, 'mean_word_length': 3.7498921949115998, 'pattern_counts': {'":': 0, '<': 63, '<?xml version=': 0, '>': 63, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '335671', 'license': 'Apache-2.0', 'n_tokens_mistral': 3856, 'n_tokens_neox': 3485, 'n_words': 1123, 'path': 'LinkedIn/Ex_Files_Python_Data_Science_EssT_Pt2/Exercise Files/Ch3_LogisticRegression_Titanic.ipynb', 'repo_name': 'weboctave/ml-bootcamp', 'size': '11014'}