<jupyter_start><jupyter_text>Kaggle Titanic survival - feature selection 1 (univariate statistical selection)Reducing the number of features we use can have three benefits:* Simplifies model explanation* Model fit may be improved by the removal of features that add no value* Model will be faster to fitIn this notebook we will use a simple statistical method for selecting features called univariate feature selection. We will examine the correlation between each feature and the target label value. This is called univariate statistics because we examine each feature independently. Two key advantages of this method are:* It is simple* It is fastTwo key disadvantage of this method are:* It may miss features which have little effect alone, but which are influential when combined* It may include features which are highly correlated which could be reduced to choosing just one of the highly correlated features.The machine learning model we will use to test the feature selection is as previously described:https://github.com/MichaelAllen1966/1804_python_healthcare/blob/master/titanic/02_logistic_regression.ipynb We will go through the following steps:* Download and save pre-processed data* Split data into features (X) and label (y)* Calculate the correlation of each feature with the target label value* Sort by correlation (ignoring the +ve/-ve sign)* Test the features in our logistic regression model<jupyter_code># Hide warnings (to keep notebook tidy; do not usually do this)
import warnings
warnings.filterwarnings("ignore")<jupyter_output><empty_output><jupyter_text>Load modulesA standard Anaconda install of Python (https://www.anaconda.com/distribution/) contains all the necessary modules.<jupyter_code>import numpy as np
import pandas as pd
# Import machine learning methods
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import auc
from sklearn.metrics import roc_curve
from sklearn.model_selection import StratifiedKFold
from sklearn.preprocessing import StandardScaler<jupyter_output><empty_output><jupyter_text>Load dataThe section below downloads pre-processed data, and saves it to a subfolder (from where this code is run).If data has already been downloaded that cell may be skipped.Code that was used to pre-process the data ready for machine learning may be found at:https://github.com/MichaelAllen1966/1804_python_healthcare/blob/master/titanic/01_preprocessing.ipynb<jupyter_code>download_required = True

if download_required:
    
    # Download processed data:
    address = 'https://raw.githubusercontent.com/MichaelAllen1966/' + \
                '1804_python_healthcare/master/titanic/data/processed_data.csv'
    
    data = pd.read_csv(address)

    # Create a data subfolder if one does not already exist
    import os
    data_directory ='./data/'
    if not os.path.exists(data_directory):
        os.makedirs(data_directory)

    # Save data
    data.to_csv(data_directory + 'processed_data.csv', index=False)
data = pd.read_csv('data/processed_data.csv')
# Make all data 'float' type
data = data.astype(float)<jupyter_output><empty_output><jupyter_text>The first column is a passenger index number. We will remove this, as this is not part of the original Titanic passenger data.<jupyter_code># Drop Passengerid (axis=1 indicates we are removing a column rather than a row)
# We drop passenger ID as it is not original data

data.drop('PassengerId', inplace=True, axis=1)<jupyter_output><empty_output><jupyter_text>Divide into X (features) and y (labels)We will separate out our features (the data we use to make a prediction) from our label (what we are truing to predict).By convention our features are called `X` (usually upper case to denote multiple features), and the label (survive or not) `y`.<jupyter_code>X = data.drop('Survived',axis=1) # X = all 'data' except the 'survived' column
y = data['Survived'] # y = 'survived' column from 'data'<jupyter_output><empty_output><jupyter_text>Calculate correlation coefficients<jupyter_code>from scipy.stats.stats import pearsonr
features = list(X)
correlation = []
significance = []
for feature in features:
    correl = pearsonr(X[feature].values, y.values)
    correlation.append(correl[0])
    significance.append(correl[1])
df = pd.DataFrame()
df['feature'] = features
df['correlation'] = correlation
df['abs_correlation'] = np.abs(correlation)
df['significance'] = significance
df['significant'] = df['significance'] < 0.05 # Label those P<0.01
df.sort_values(by='abs_correlation', ascending=False, inplace=True)<jupyter_output><empty_output><jupyter_text>Show features in order of correlation with survival.<jupyter_code>df<jupyter_output><empty_output><jupyter_text>Get ordered feature list.<jupyter_code>ordered_features = list(df['feature'])
ordered_features<jupyter_output><empty_output><jupyter_text>Testing our selected featuresAfter statistical selection we may simply choose the top *k* features, or we may choose those labelled as significant (P<0.05). Here we will inclemently add features to the list of features to use (chosen in order of their correlation coefficients), and see the effect on model accuracy and ROC AUC as measured by k-fold stratification.If you are not familiar with k-fold stratification, have a look at:https://github.com/MichaelAllen1966/1804_python_healthcare/blob/master/titanic/03_k_fold.ipynbIf you are not familiar with ROC AUC, have a look at:https://github.com/MichaelAllen1966/1804_python_healthcare/blob/master/titanic/06_roc_sensitivity_specificity.ipynb<jupyter_code>def standardise_data(X_train, X_test):
    
    # Initialise a new scaling object for normalising input data
    sc = StandardScaler() 

    # Set up the scaler just on the training set
    sc.fit(X_train)

    # Apply the scaler to the training and test sets
    train_std=sc.transform(X_train)
    test_std=sc.transform(X_test)
    
    return train_std, test_std
# Create list to store accuracies
accuracy_by_feature_number = []
roc_auc_by_feature_number = []

# Loop through feature list
number_of_features = len(ordered_features)
for i in range(number_of_features):
    # print ("{0} features of {1}".format(i, number_of_features))
    features_to_use = ordered_features[0:i+1]
    X_selected = X[features_to_use]
    
    # Convert to NumPy (needed for k-fold method)
    # Convert DataFrames to NumPy arrays
    X_np = X_selected.values
    y_np = y.values
    
    #%% Run k fold model

    # Set up lists to hold results for each k-fold run
    test_acc_results = []
    test_auc_results = []

    # Set up splits
    number_of_splits = 10
    skf = StratifiedKFold(n_splits = number_of_splits)
    skf.get_n_splits(X_np, y)

    # Loop through the k-fold splits
    for train_index, test_index in skf.split(X_np, y_np):
        # Get X and Y train/test
        X_train, X_test = X_np[train_index], X_np[test_index]
        y_train, y_test = y[train_index], y[test_index]

        # Get X and Y train/test
        X_train_std, X_test_std = standardise_data(X_train, X_test)

        # Set up and fit model
        model = LogisticRegression(solver='lbfgs')
        model.fit(X_train_std,y_train)

        # Predict test set labels
        y_pred_test = model.predict(X_test_std)
        
        # Calculate accuracy of test sets
        accuracy_test = np.mean(y_pred_test == y_test)
        test_acc_results.append(accuracy_test)
        
        # Get ROC AUC
        probabilities = model.predict_proba(X_test_std)
        probabilities = probabilities[:, 1] # Probability of 'survived' class
        fpr, tpr, thresholds = roc_curve(y_test, probabilities)
        roc_auc = auc(fpr, tpr)
        test_auc_results.append(roc_auc)      
        
    # Add mean accuracy and AUC to record of accuracy by feature number
    accuracy_by_feature_number.append(np.mean(test_acc_results))
    roc_auc_by_feature_number.append(np.mean(test_auc_results)) 
import matplotlib.pyplot as plt
%matplotlib inline
chart_x = list(range(1, number_of_features + 1))

plt.plot(chart_x, accuracy_by_feature_number,
        label = 'Accuracy')

plt.plot(chart_x, roc_auc_by_feature_number,
        label = 'ROC AUC')

plt.xlabel('Number of features')
plt.ylabel('Accuracy')
plt.legend()
plt.grid(True)

plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07841005195119004, 'fraction_numerical': 0.008698803914461761, 'mean_word_length': 4.631292517006803, 'pattern_counts': {'":': 0, '<': 40, '<?xml version=': 0, '>': 37, 'https://': 6, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '191114', 'license': 'MIT', 'n_tokens_mistral': 2573, 'n_tokens_neox': 2429, 'n_words': 803, 'path': 'jupyter_notebooks/07_feature_selection_1_univariate.ipynb', 'repo_name': 'MichaelAllen1966/2004_titanic', 'size': '8277'}