<jupyter_start><jupyter_text>Logistic regression modelThe data includes:Variable  | Definition----------|-----------survival  | Survival (0 = No, 1 = Yes)pclass    | Ticket classsex       | SexAge       | Age in yearssibsp     |  of siblings / spouses aboard the Titanicparch     |  of parents / children aboard the Titanicticket    | Ticket numberfare      | Passenger farecabin     | Cabin numberembarked  | Port of Embarkation(C=Cherbourg, Q=Queenstown, S=Southampton)  Logistic regressionIn this example we will use logistic regression (see https://en.wikipedia.org/wiki/Logistic_regression).For an introductory video on logistic regression see: https://www.youtube.com/watch?v=yIYKR4sgzI8Logistic regression takes a range of features (which we will normalise/standardise to put on the same scale) and returns a probability that a certain classification (survival in this case) is true.We will go through the following steps:* Download and save pre-processed data* Split data into features (X) and label (y)* Split data into training and test sets (we will test on data that has not been used to fit the model)* Standardise data* Fit a logistic regression model (from sklearn)* Predict survival of the test set, and assess accuracy* Review model coefficients (weights) to see importance of features* Show probability of survival for passengers  Load modulesA standard Anaconda install of Python (https://www.anaconda.com/distribution/) contains all the necessary modules.<jupyter_code>import numpy as np
import pandas as pd
# Import machine learning methods
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler<jupyter_output><empty_output><jupyter_text>Load dataThe section below downloads pre-processed data, and saves it to a subfolder (from where this code is run).If data has already been downloaded that cell may be skipped.Code that was used to pre-process the data ready for machine learning may be found at:https://github.com/MichaelAllen1966/1804_python_healthcare/blob/master/titanic/01_preprocessing.ipynb<jupyter_code>download_required = True

if download_required:
    
    # Download processed data:
    address = 'https://raw.githubusercontent.com/MichaelAllen1966/' + \
                '1804_python_healthcare/master/titanic/data/processed_data.csv'
    
    data = pd.read_csv(address)

    # Create a data subfolder if one does not already exist
    import os
    data_directory ='./data/'
    if not os.path.exists(data_directory):
        os.makedirs(data_directory)

    # Save data
    data.to_csv(data_directory + 'processed_data.csv', index=False)
data = pd.read_csv('data/processed_data.csv')
# Make all data 'float' type
data = data.astype(float)<jupyter_output><empty_output><jupyter_text>Examine loaded dataThe data is in the form of a Pandas DataFrame, so we have column headers providing information of what is contained in each column.We will use the DataFrame `.head()` method to show the first few rows of the imported DataFrame. By default this shows the first 5 rows. Here we will look at the first 10.<jupyter_code>data.head(10)<jupyter_output><empty_output><jupyter_text>We can also show a summary of the data with the `.describe()` method.<jupyter_code>data.describe()<jupyter_output><empty_output><jupyter_text>The first column is a passenger index number. We will remove this, as this is not part of the original Titanic passenger data.<jupyter_code># Drop Passengerid (axis=1 indicates we are removing a column rather than a row)
# We drop passenger ID as it is not original data

data.drop('PassengerId', inplace=True, axis=1)<jupyter_output><empty_output><jupyter_text>Looking at a summary of passengers who survived or did not surviveBefore running machine learning models, it is always good to have a look at your data. Here we will separate passengers who survived from those who died, and we will have a look at differences in features.We will use a *mask* to select and filter passengers.<jupyter_code>mask = data['Survived'] == 1 # Mask for passengers who survive
survived = data[mask] # filter using mask

mask = data['Survived'] == 0 # Mask for passengers who died
died = data[mask] # filter using mask<jupyter_output><empty_output><jupyter_text>Now let's look at average (mean) values for `survived` and `died`.<jupyter_code>survived.mean()
died.mean()<jupyter_output><empty_output><jupyter_text>We can make looking at them side by side more easy by putting these values in a new DataFrame.<jupyter_code>summary = pd.DataFrame() # New empty DataFrame
summary['survived'] = survived.mean()
summary['died'] = died.mean()<jupyter_output><empty_output><jupyter_text>Now let's look at them side by side. See if you can spot what features you think might have influenced survival.<jupyter_code>summary<jupyter_output><empty_output><jupyter_text>Divide into X (features) and y (labels)We will separate out our features (the data we use to make a prediction) from our label (what we are truing to predict).By convention our features are called `X` (usually upper case to denote multiple features), and the label (survived or not) `y`.<jupyter_code>X = data.drop('Survived',axis=1) # X = all 'data' except the 'survived' column
y = data['Survived'] # y = 'survived' column from 'data'<jupyter_output><empty_output><jupyter_text>Divide into training and tets setsWhen we test a machine learning model we should always test it on data that has not been used to train the model.We will use sklearn's `train_test_split` method to randomly split the data: 75% for training, and 25% for testing.<jupyter_code>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.25)
X_train.std(), X_train.mean()<jupyter_output><empty_output><jupyter_text>Standardise dataWe want all of out features to be on roughly the same scale. This generally leads to a better model, and also allows us to more easily compare the importance of different features.One simple method is to scale all features 0-1 (by subtracting the minimum value for each value, and dividing by the new remaining maximum value).But a more common method used in many machine learning methods is standardisation, where we use the mean and standard deviation of the training set of data to normalise the data. We subtract the mean of the test set values, and divide by the standard deviation of the training data. Note that the mean and standard deviation of the training data are used to standardise the test set data as well.Here we will use sklearn's `StandardScaler method`. This method also copes with problems we might otherwise have (such as if one feature has zero standard deviation in the training set).<jupyter_code>X_train.astype(float)
def standardise_data(X_train, X_test):
    
    # Initialise a new scaling object for normalising input data
    sc = StandardScaler() 

    # Set up the scaler just on the training set
    sc.fit(X_train)

    # Apply the scaler to the training and test sets
    train_std=sc.transform(X_train)
    test_std=sc.transform(X_test)
    
    return train_std, test_std
X_train_std, X_test_std = standardise_data(X_train, X_test)<jupyter_output><empty_output><jupyter_text>Fit logistic regression modelNow we will fir a logistic regression model, using sklearn's `LogisticRegression` method. Our machine learning model fitting is only two lines of code!By using the name `model` for our logistic regression model we will make our model more interchangeable later on.<jupyter_code>model = LogisticRegression()
model.fit(X_train_std,y_train)<jupyter_output><empty_output><jupyter_text>Predict valuesNow we can use the trained model to predict survival. We will test the accuracy of both the training and test data sets.<jupyter_code># Predict training and test set labels
y_pred_train = model.predict(X_train_std)
y_pred_test = model.predict(X_test_std)<jupyter_output><empty_output><jupyter_text>Calculate accuracyIn this example we will measure accuracy simply as the proportion of passengers where we make the correct prediction. In a later notebook we will look at other measures of accuracy which explore false positives and false negatives in more detail.<jupyter_code>accuracy_train = np.mean(y_pred_train == y_train)
accuracy_test = np.mean(y_pred_test == y_test)

print ('Accuracy of predicting training data =', accuracy_train)
print ('Accuracy of predicting test data =', accuracy_test)<jupyter_output>Accuracy of predicting training data = 0.8278443113772455
Accuracy of predicting test data = 0.7623318385650224<jupyter_text>Not bad - about 80% accuracy. You will probably see that accuracy of predicting the training set is usually higher than the test set. Because we are only testing one random sample, you may occasionally see otherwise. In later note books we will look at the best way to repeat multiple tests, and look at what to do if the accuracy of the training set is significantly higher than the test set (a problem called 'over-fitting').  Examining the model coefficients (weights)Not all features are equally important. And some may be of little or no use at all, unnecessarily increasing the complexity of the model. In a later notebook we will look at selecting features which add value to the model (or removing features that don’t).Here we will look at the importance of features – how they affect our estimation of survival. These are known as the model *coefficients* (if you come from a traditional statistics background), or model *weights* (if you come from a machine learning background). Because we have standardised our input data the magnitude of the weights may be compared as an indicator of their influence in the model. Weights with higher negative numbers mean that that feature correlates with reduced chance of survival. Weights with higher positive numbers mean that that feature correlates with increased chance of survival. Those weights with values closer to zero (either positive or negative) have less influence in the model.We access the model weights my examining the model `coef_` attribute. The model may predict more than one outcome label, in which case we have weights for each label. Because we are predicting a signle label (survive or not), the weights are found in the first element (`[0]`) of the `coef_` attribute.<jupyter_code>co_eff = model.coef_[0]
co_eff<jupyter_output><empty_output><jupyter_text>So we have an array of model weights.Not very readable for us mere humans is it?!We will transfer the weights array to a Pandas DataFrame. The array order is in the same order of the list of features of X, so we will put that those into the DataFrame as well. And we will sort by influence in the model. Because both large negative and positive values are more influential in the model we will take the *absolute* value of the weight (that is remove any negative sign), and then sort by that absolute value. That will give us a more readable table of most influential features in the model.<jupyter_code>co_eff_df = pd.DataFrame() # create empty DataFrame
co_eff_df['feature'] = list(X) # Get feature names from X
co_eff_df['co_eff'] = co_eff
co_eff_df['abs_co_eff'] = np.abs(co_eff)
co_eff_df.sort_values(by='abs_co_eff', ascending=False, inplace=True)
co_eff_df<jupyter_output><empty_output><jupyter_text>So are three most influential features are:* male (being male reduces probability of survival)* Pclass (lower class passengers, who have a higher class number, reduces probability of survival)* age (being older reduces probability of survival)  Show predicted probabilitiesThe predicted probabilities are for the two alternative classes 0 (does not survive) or 1 (survive).Ordinarily we do not see these probabilities - the `predict` method used above applies a cut-off of 0.5 to classify passengers into survived or not, but we can see the individual probabilities for each passenger.Later we will use these to adjust sensitivity of our model to detecting survivors or non-survivors. Each passenger has two values. These are the probability of not surviving (first value) or surviving (second value). Because we only have two possible classes we only need to look at one. Multiple values are important when there are more than one class being predicted.<jupyter_code># Show first ten predicted classes
classes = model.predict(X_test_std)
classes[0:10]
# Show first ten predicted probabilities 
# (note how the values relate to the classes predicted above)
probabilities = model.predict_proba(X_test_std)
probabilities[0:10]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0626092483712061, 'fraction_numerical': 0.007071349118067694, 'mean_word_length': 5.225024727992087, 'pattern_counts': {'":': 0, '<': 72, '<?xml version=': 0, '>': 72, 'https://': 5, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '207431', 'license': 'MIT', 'n_tokens_mistral': 3478, 'n_tokens_neox': 3304, 'n_words': 1590, 'path': '02_logistic_regression.ipynb', 'repo_name': 'MichaelAllen1966/titanic', 'size': '12586'}