<jupyter_start><jupyter_text>Predicting Blood Donations **Contents**---* [**1. Importing Packages**](1.-Importing-Packages)* [**2. Importing Blood Donations Dataset**](2.-Importing-Blood-Donations-Dataset)* [**3. Target Feature Considerations**](3.-Target-Feature-Considerations)* [**4. Exploring the Dataset**](4.-Exploring-the-Dataset)* [**5. Evaluation Metric**](5.-Evaluation-Metric)* [**6. Selecting the Algorithms**](6.-Selecting-the-Algorithms)* [**7. Optimising the Hyper Parameters**](7.-Optimising-the-Hyper-Parameters)* [**8. Evaluating the Test Set**](8.-Evaluating-the-Test-Set)* [**9. Evaluating the Full Set**](9.-Evaluating-the-Full-Set)* [**10. Conclusions**](10.-Conclusions)---  1. Importing Packages---<jupyter_code># File system management
import os

# Numpy and pandas for data analysis and manipulation
import numpy as np
import pandas as pd

# Matplotlib pyplot and seaborn for plotting
import matplotlib.pyplot as plt
import seaborn as sns
from jupyterthemes import jtplot
jtplot.style()

# Algorithms
from sklearn import svm, tree, linear_model, neighbors, naive_bayes, ensemble, discriminant_analysis, gaussian_process
from xgboost import XGBClassifier
from lightgbm import LGBMClassifier

# Sklearn importing folding methods method
from sklearn.model_selection import KFold, StratifiedKFold, train_test_split

# Sklearn importing evaluation metrics
from sklearn.metrics import roc_curve, auc, roc_auc_score, log_loss, accuracy_score, confusion_matrix

# Suppress deprecation warnings
import warnings
warnings.simplefilter("ignore", category=DeprecationWarning)

# Importing cross validation grid search
from sklearn.model_selection import GridSearchCV<jupyter_output><empty_output><jupyter_text>2. Importing Blood Donations Dataset<jupyter_code># Setting the directory
os.chdir('C:/Users/jakec/my_projects/predicting-blood-donations')
# Viewing the contained items
os.listdir('.')
# Importing as pandas dataframe and displaying the top 5 entries of the blood donations dataset
blood_donations_df = pd.read_excel('Case_Study_2_Blood_Donations_Data.xlsx')
blood_donations_df.head()<jupyter_output><empty_output><jupyter_text>3. Target Feature Considerations---  A penny for my thoughts... Reading the description of the target variable "...to create a model which will predict the likelihood of a donor donating again from the given data set" initially, makes me think we are looking at an open ended timeframe for donating again (e.g. a prediction over the remainder of their lives). But given some of the limitations in the dataset which include: not knowing the dates of each donation; a maximum period since the first donation of 98 months (~8 years); and not being given the actual dates of past donations it would probably be very difficult and inaccurate to model the probability that they ever donate again. Secondly, I considered trying to define a reasonable time period in which, if they are to donate again, they are statistically likely to have done so within. In order to define a satisfactory time period an understanding of the distribution of donations over time for each donor would be required. Given the limited range of features in the provided dataset this would require outside knowledge or information (data). For the purposes of this tech evaluation I'm going to avoid looking into external information or data. This is in order to keep the scope of the problem reasonable.Therefore, the feature 'whether she/he donated blood in May 2018' will be used as the target variable. This feature will act as a proxy for the probability that donors will give blood in a later 'blood drive month' with similar characteristics. The idea of blood drive months is explained later in section 4 of this analysis when exploring the 'Recency (months)' and 'Time (months)' features. The 'blood drive months' do not occur at regular intervals. This requires the strong assumption that the profile of donations in May 2018 is a reasonable approximation of donations in any blood drive month.  4. Exploring the Dataset---<jupyter_code># Splitting the data into training and testing sets
train, test = train_test_split(blood_donations_df, test_size=0.20, random_state=42)
# Quick overview of the features
train.describe()<jupyter_output><empty_output><jupyter_text>Thoughts:One point of note is that the target variable has a mean value of 0.238 so a base accuracy of 0.764 can be obtained by predicting no one will donate blood.  Exploring the “Total Volume Donated (c.c.)”  feature<jupyter_code># Checking the claim that '“Total Volume Donated (c.c.)” (Monetary) is exactly proportional to the 
# “Number of Donations” (Frequency) with Monetary = 250 * Frequency'
(train['Monetary (c.c. blood)']/train['Frequency (times)']).describe()
# As the claim is correct I'm dropping the feature 'Monetary (c.c. blood)' as it contains no additional information
# (100% Correlated with 'Monetary (c.c. blood)')
train = train.drop(['Monetary (c.c. blood)'], axis=1)
test = test.drop(['Monetary (c.c. blood)'], axis=1)
blood_donations_df = blood_donations_df.drop(['Monetary (c.c. blood)'], axis=1)<jupyter_output><empty_output><jupyter_text>Exploring the 'Recency (months)' feature<jupyter_code># Quick look at behaviour
sns.catplot('Recency (months)', data = train, kind='count', height=4, aspect=3)<jupyter_output><empty_output><jupyter_text>Thoughts: Over the past 2 years there seems to be a approximate seasonal donation cycle; whereby specific months, year on year, show significant spikes. These spikes are seen in Recency's of 16 (Jan 2017), 14 (Mar 2017) which line up with spikes seen a year later 4 (Jan 2018), 2 (Mar 2018).However, assuming that a Recency of 0 represents people who donated in May 2018 but not on the 31st. The target variable which shows that 23.7968% of the 748 people donated on 31 May 2018 (a spike) doesn't fit this trend. If it is assumed that these spikes occur on a specific day of the week it could have occured a day later the previous year and therefore June 2017 would represent a spike continuing the trend.These spikes will be referred to as 'blood drives' for the rest of this analysis as a blood drive would be a reasonable cause of the observed trend.<jupyter_code># Plotting the distributions of the recency against donations in May
recency = sns.FacetGrid(train, hue = 'whether he/she donated blood in May 2018', height=4, aspect=3)
recency.map(sns.kdeplot, 'Recency (months)', shade= True)
recency.set(xlim=(0 , train['Recency (months)'].max()))
recency.fig.suptitle('Recency distributions categorised by donantions in May 2018')
recency.add_legend()<jupyter_output><empty_output><jupyter_text>Thoughts: There seems to be an overall decreasing trend for the ratio of donations but there is a lot of overlapping noise.This could be caused by a larger ratio of donations to non-donations in blood drive months.  Exploring the 'Time (months)' (Months since First Donation) feature<jupyter_code>sns.catplot('Time (months)', data = train, kind='count', height=4, aspect=3)<jupyter_output><empty_output><jupyter_text>Thoughts: The year-on-year cycle of blood drives can be seen again in the 'Time (months)' feature.<jupyter_code># Plotting the distributions of the time against donations in May
recency = sns.FacetGrid(train, hue = 'whether he/she donated blood in May 2018', height=4, aspect=3)
recency.map(sns.kdeplot, 'Time (months)', shade= True)
recency.set(xlim=(0 , train['Time (months)'].max()))
recency.fig.suptitle('Time distributions categorised by donantions in May 2018')
recency.add_legend()<jupyter_output><empty_output><jupyter_text>Thoughts: The distribution shows a relationship that is not immediately intuitive with 4 distinct sections. Twenty four  to 50 months seems to be the most likely time period after which people are likely to donate again, 0-12 the second most , 50+ the least, 12-23 is the second least. This indicates that, when the time is viewed in isolation, it has a non-monotonic relationship with the target. Care may need to be taken when treating this feature in algorithms such as logistic regressions which treat numeric variables as monotonic. Weight-of-evidence or other binning methods could be used to address this.  Exploring the 'Frequency (times)' feature<jupyter_code>sns.catplot('Frequency (times)', data = train, kind='count', height=4, aspect=3)
# Plotting the distributions of the frequency against donations in May
recency = sns.FacetGrid(train, hue = 'whether he/she donated blood in May 2018', height=4, aspect=3)
recency.map(sns.kdeplot, 'Frequency (times)', shade= True)
recency.set(xlim=(0 , train['Frequency (times)'].max()))
recency.fig.suptitle('Frequency distributions categorised by donantions in May 2018')
recency.add_legend()<jupyter_output><empty_output><jupyter_text>Thoughts: It's advantageous to have donated more than 5 times but relatively stable after that (It's hard to distinguish after 25 months due to low volumes).  Creating and exploring a rate of donations feature<jupyter_code># Defining new feature 'Rate as the average number of donations per month
train['Rate (months)'] = train['Frequency (times)']/train['Time (months)']
train['Rate (months)'] = train['Rate (months)'].round(2)

test['Rate (months)'] = test['Frequency (times)']/test['Time (months)']
test['Rate (months)'] = test['Rate (months)'].round(2)

blood_donations_df['Rate (months)'] = blood_donations_df['Frequency (times)']/blood_donations_df['Time (months)']
blood_donations_df['Rate (months)'] = blood_donations_df['Rate (months)'].round(2)
sns.catplot('Rate (months)', data = train, kind='count', height=4, aspect=4)<jupyter_output><empty_output><jupyter_text>Thoughts:Higher rates can be seen at 0.25 and 0.5. It is not clear to me if these spikes in the rate of donation are useful at this stage. Also, it might be worth noting that donor rates over 0.5 seems unsafe.<jupyter_code># Plotting the distributions of the rate against donations in May
recency = sns.FacetGrid(train, hue = 'whether he/she donated blood in May 2018', height=4, aspect=3)
recency.map(sns.kdeplot, 'Rate (months)', shade= True)
recency.set(xlim=(0 , train['Rate (months)'].max()))
recency.fig.suptitle('Rate distributions categorised by donantions in May 2018')
recency.add_legend()<jupyter_output><empty_output><jupyter_text>Thoughts: This roughly follows the expected trend, that a lower rate indacates thet donations are less likly. There is a notable spike in non-donors with a rate of 0.5 or once every two months. This could be people who donate once every two months and for some reason may is an off month or simply a chance spike. This could use further investigation but for the purposes of this evaluation I'm assuming that the decrease in the ratio of people who donated in May 2018 and donate on average every two months is a chance spike.  Creating and exploring a 'On Schedule' feature<jupyter_code># Defining new feature 'On Schedule' as the product of recency to rate (this should quantify if they are currently 
# behind their usual schedule)
train['On Schedule'] = train['Recency (months)']*train['Rate (months)']
train['On Schedule'] = train['On Schedule'].round(1)

test['On Schedule'] = test['Recency (months)']*test['Rate (months)']
test['On Schedule'] = test['On Schedule'].round(1)

blood_donations_df['On Schedule'] = blood_donations_df['Recency (months)']*blood_donations_df['Rate (months)']
blood_donations_df['On Schedule'] = blood_donations_df['On Schedule'].round(1)
sns.catplot('On Schedule', data = train, kind='count', height=4, aspect=4)<jupyter_output><empty_output><jupyter_text>Thoughts:There is a large spike at a schedule of 1. This might be due to people who have only donated once previously.<jupyter_code># Plotting the distributions of the rate against donations in May
recency = sns.FacetGrid(train, hue = 'whether he/she donated blood in May 2018', height=4, aspect=3)
recency.map(sns.kdeplot, 'On Schedule', shade= True)
recency.set(xlim=(0 , train['On Schedule'].max()))
recency.fig.suptitle('"On Schedule" distributions categorised by donantions in May 2018')
recency.add_legend()<jupyter_output><empty_output><jupyter_text>Thoughts: There seems to be a reasonably strong relationship for donations around 1.  Exploring pair plots and feature correlations<jupyter_code># Correlation heatmap of dataset
corr_plt , ax = plt.subplots(figsize =(14, 12))
colormap = sns.diverging_palette(220, 10, as_cmap = True)

corr_plt = sns.heatmap(
    train.corr(), 
    cmap = colormap,
    square=True, 
    cbar_kws={'shrink':.9 }, 
    ax=ax,
    annot=True, 
    linewidths=0.1,vmax=1.0, linecolor='white',
    annot_kws={'fontsize':12 }
)<jupyter_output><empty_output><jupyter_text>Thoughts:A high correlation can be seen between the Rate and Recency features. This is interesting given Rate is not derived from the Recency feature.<jupyter_code>sns.pairplot(train)<jupyter_output><empty_output><jupyter_text>Dimensionality Reduction:Due to the simplicity of the dataset (and low correlations) I won't explore the possibility of dimensionality reduction for this evaluation.  Normalisation:I don't expect it to be a large factor in the algorithms as most features are within an order of magnitude. I won't explore normalising features in this evaluation.  Additional Engineered Features:Given the limited time I've spent investigating the data I assume that there other useful insights in various combination or exponential/log features. For the purpose of this eval I will not investigate these further.  5. Evaluation Metric---  Thoughts:The most common and easy to understand metric would be the accuracy. However, because the target variable is somewhat unbalanced maximising accuracy will likely skew the model towards predicting the more common class of non-donors. This could be addressed with with precision-recall analysis. Perhaps a better metric in this case would be the gini or area under the receiver-operating-curve (AUROC). This would measure the model’s ability to order/discriminate likey from unlikely donors. The root-mean-squared-error (RMSE) could be used to make sure that large discrepancies are more heavily penalised then simply using the accuracy (reducing the effect of the imbalance). In this case, the Log-Loss will be used as the primary metric while keeping an eye on both, the accuracy and the AUROC. This decision was driven by research into predicting blood donations which revealed that there is an online competition for this (or at least a very similar dataset) that uses log loss an an evaluation metric. The competition describes it as 'This function gives a maximum penalty when an extreme error is made, that is when the actual value is 0 and the predicted value is 1, or vice versa, since it could give an infinity score'. Similarly to the RMSE (except exponentially) this penalises large discrepancies far more then small ones.  6. Selecting the Algorithms---<jupyter_code># Function for displaying feature importance
def display_importances(fld_imp, mdl_nme):
    cols = fld_imp[["feature", "importance"]].groupby("feature").mean().sort_values(by="importance", ascending=False)[:40].index
    best_features = fld_imp.loc[fld_imp.feature.isin(cols)]
    plt.figure(figsize=(8, 10))
    sns.barplot(x="importance", y="feature", data=best_features.sort_values(by="importance", ascending=False))
    plt.title('Features (avg over folds)')
    plt.tight_layout()
def model_fit(mdl, mdl_nme = '', n_fld = 5, dbg = True):   
    print(mdl_nme)
    # K fold validation
    fld = StratifiedKFold(n_splits = n_fld, shuffle=True, random_state = 3)
    
    # List of feature columns
    fts = [f for f in train.columns if f not in ['whether he/she donated blood in May 2018']]
    
    # Initializing Variables
    prd = np.zeros(train[fts].shape[0])
    sub_prd = np.zeros(blood_donations_df[fts].shape[0])
    sub_prd_tst = np.zeros(test[fts].shape[0])
    fts_imp = pd.DataFrame()
    acc = 0
    auc = 0
    log_los = 0
    log_los_tot = 0
    
    for n_fold, (trn_idx, vld_idx) in enumerate(fld.split(train[fts], train['whether he/she donated blood in May 2018'])):
        trn_X, trn_y = train[fts].iloc[trn_idx], train['whether he/she donated blood in May 2018'].iloc[trn_idx]
        vld_X, vld_y = train[fts].iloc[vld_idx], train['whether he/she donated blood in May 2018'].iloc[vld_idx]
        
        # Fit the model
        mdl.fit(trn_X, trn_y)
        
        if hasattr(mdl, 'predict_proba'):
            prd[vld_idx] = mdl.predict_proba(vld_X)[:, 1]
            
            # Appling the fit to the prediction set
            sub_prd += mdl.predict_proba(blood_donations_df[fts])[:, 1]/fld.n_splits
            # Appling the fit to the prediction set
            sub_prd_tst += mdl.predict_proba(test[fts])[:, 1]/fld.n_splits
            # Outputting the AUC
            auc += roc_auc_score(vld_y, prd[vld_idx])/fld.n_splits
            log_los = log_loss(vld_y, mdl.predict_proba(vld_X)[:, 1])
            log_los_tot += log_los/fld.n_splits
                
        # Outputting statistics
        print('Fold %2d acc : %.6f' % (n_fold + 1, mdl.score(vld_X, vld_y)))
        print('Fold %2d log_loss : %.6f' % (n_fold + 1, log_los))
        acc += mdl.score(vld_X, vld_y) / fld.n_splits
        
        fld_imp = pd.DataFrame()
        fld_imp["feature"] = fts
        if hasattr(mdl, 'feature_importances_'):
            fld_imp["importance"] = mdl.feature_importances_
        elif hasattr(mdl, 'coef_'):
            fld_imp["importance"] = abs(list(np.std(trn_X, 0))*mdl.coef_[0])
        else:
            fld_imp["importance"] = None
        fld_imp["fold"] = n_fold + 1
        fts_imp = pd.concat([fts_imp, fld_imp], axis=0)
    
    print('Average AUC over folds: %.6f' % auc)
    print('Accuracy : %.6f' % acc)
    print('Average log_loss over folds: %.6f' % log_los)

    # Display Importances and save prediction
    if dbg == False:
        # Displaying Feature Importance
        if (hasattr(mdl, 'feature_importances_') | hasattr(mdl, 'coef_')):
            display_importances(fld_imp, mdl_nme)

    return sub_prd, sub_prd_tst, nme, log_los_tot<jupyter_output><empty_output><jupyter_text>Initial look at a variety of classification algorithms<jupyter_code>MLA = [
   # Ensemble Methods
   ensemble.AdaBoostClassifier(),
   ensemble.BaggingClassifier(),
   ensemble.ExtraTreesClassifier(),
   ensemble.GradientBoostingClassifier(),
   ensemble.RandomForestClassifier(),
       
   # Boosted Tree Classifiers
   XGBClassifier(),  
   LGBMClassifier(),

   # Gaussian Processes
   gaussian_process.GaussianProcessClassifier(),
   
   # GLM
   linear_model.LogisticRegression(),
   linear_model.PassiveAggressiveClassifier(),
   linear_model.RidgeClassifierCV(),
   
   # Navies Bayes
   naive_bayes.BernoulliNB(),
   naive_bayes.GaussianNB(),
   
   # Nearest Neighbor
   neighbors.KNeighborsClassifier(),
   
   # SVM
   svm.SVC(probability=True),
   
   # Trees    
   tree.DecisionTreeClassifier(),
   tree.ExtraTreeClassifier(),
   
   # Discriminant Analysis
   discriminant_analysis.LinearDiscriminantAnalysis(),
   discriminant_analysis.QuadraticDiscriminantAnalysis()
   ]

mdl_dta = pd.DataFrame(columns = ['Algorithm', 'Log Loss'])
row_index = 0

# Applying the algorithms using 5 folds (Not exploring more at this stage) 
for alg in MLA:
   nme = alg.__class__.__name__
   prd, prd_tst, nme, loss = model_fit(alg, mdl_nme = nme, n_fld = 5)
   mdl_dta.loc[row_index, 'Algorithm'] = nme
   mdl_dta.loc[row_index, 'Log Loss'] = loss
   row_index += 1

# Plotting algoritims log loss averaged across the folds
plt.figure(figsize=(10,10))
sns.barplot(x='Log Loss', y = 'Algorithm', data = mdl_dta[mdl_dta['Log Loss']>0].sort_values(by='Log Loss'))
plt.title('Machine Learning Algorithm Log Loss \n')
plt.xlabel('Log Loss Average (%)')
plt.ylabel('Algorithm')<jupyter_output><empty_output><jupyter_text>Thoughts:Since some of the methods do not have a predict_proba method and so couldn't produce a log_loss or AUC metric, no effort was made to address this shortcoming. The algorithm’s accuracies didn't suggest that they were likely to stand out. While it would be worth investigating these methods in more detail (and possibly a wider range of methods) for the purposes of this evaluation, I will proceed with a selection of the best performing methods. These are: Linear Discriminant Analysis with a log loss of 0.524955; the Support Vector Classification with an radial basis function = 0.517178; the Logistic Regression = 0.529799; and the XGBClassifier = 0.551418.  7. Optimising the Hyper Parameters---  Thoughts:I’m using a basic grid search over a very simple range of parameters; this would benefit from further investigation. The small size of the data means that more complex bayesian optimization methods are probably unnecessary.  Linear Discriminant Analysis<jupyter_code># Linear Discriminant Analysis is closed form and therefore, has no hyper paramaters
LDA = discriminant_analysis.LinearDiscriminantAnalysis()
prd, prd_tst, nme, acc = model_fit(LDA, 'Linear Discriminant Analysis' , n_fld = 5, dbg=False)<jupyter_output>Linear Discriminant Analysis
Fold  1 acc : 0.735537
Fold  1 log_loss : 0.555312
Fold  2 acc : 0.791667
Fold  2 log_loss : 0.430884
Fold  3 acc : 0.806723
Fold  3 log_loss : 0.435439
Fold  4 acc : 0.781513
Fold  4 log_loss : 0.470212
Fold  5 acc : 0.756303
Fold  5 log_loss : 0.524955
Average AUC over folds: 0.743149
Accuracy : 0.774348
Average log_loss over folds: 0.524955<jupyter_text>Support Vector Classifier<jupyter_code># Parameter grid for XGBoost
params =[{
        'gamma': [0.001, 0.01, 0.1, 0.5, 1, 1.5, 2],
        'kernel': ['linear']},
         {
        'C': [0.1, 0.5, 0.8, 1, 1.2, 1.4, 1.6],
        'gamma': [0.001, 0.01, 0.1, 0.5, 1, 1.5, 2],
        'kernel': ['rbf']
        }]
# Fitting the paramater grid
fts = [f for f in train.columns if f not in ['whether he/she donated blood in May 2018']]

search2 = GridSearchCV(estimator = svm.SVC(probability=True, C=0.1, gamma=1, kernel='rbf'), 
                        param_grid = params, scoring='neg_log_loss', n_jobs=8, iid=False, cv=5, 
                        verbose=1)

search2.fit(train[fts],train['whether he/she donated blood in May 2018'])

search2.best_params_, search2.best_score_
SVC = svm.SVC(probability=True, C=1.2, gamma=0.01, kernel='rbf')
prd, prd_tst, nme, acc = model_fit(SVC, 'Support Vector Classifier' , n_fld = 5, dbg=False)
# Note that I expect to see a decrease going to the Average log_loss over folds from the log loss in the optimisation<jupyter_output>Support Vector Classifier
Fold  1 acc : 0.760331
Fold  1 log_loss : 0.544428
Fold  2 acc : 0.791667
Fold  2 log_loss : 0.491239
Fold  3 acc : 0.789916
Fold  3 log_loss : 0.512470
Fold  4 acc : 0.806723
Fold  4 log_loss : 0.490594
Fold  5 acc : 0.781513
Fold  5 log_loss : 0.492476
Average AUC over folds: 0.686161
Accuracy : 0.786030
Average log_loss over folds: 0.492476<jupyter_text>Logistic Regression<jupyter_code># Parameter grid for Logistic Regression
params = {'C': [0.001, 0.01, 0.1, 0.5, 1, 5, 10, 50, 100, 500, 1000]}
# Fitting the paramater grid
fts = [f for f in train.columns if f not in ['whether he/she donated blood in May 2018']]

search3 = GridSearchCV(estimator = linear_model.LogisticRegression(C=1)
                      , param_grid = params, scoring='neg_log_loss', n_jobs=8, iid=False, cv=5
                      , verbose=1)

search3.fit(train[fts],train['whether he/she donated blood in May 2018'])

search3.best_params_, search3.best_score_
log_reg = linear_model.LogisticRegression(C=0.01)
prd, prd_tst, nme, acc = model_fit(log_reg, 'Logistic Regression' , n_fld = 5, dbg=False)<jupyter_output>Logistic Regression
Fold  1 acc : 0.752066
Fold  1 log_loss : 0.536648
Fold  2 acc : 0.800000
Fold  2 log_loss : 0.423201
Fold  3 acc : 0.806723
Fold  3 log_loss : 0.438185
Fold  4 acc : 0.773109
Fold  4 log_loss : 0.458432
Fold  5 acc : 0.756303
Fold  5 log_loss : 0.526230
Average AUC over folds: 0.747398
Accuracy : 0.777640
Average log_loss over folds: 0.526230<jupyter_text>XGBoost Classifier<jupyter_code># A parameter grid for XGBoost
params = {
        'min_child_weight': [1, 2, 5, 8, 9, 10, 11, 15, 20],
        'gamma': [0.1, 0.5, 1, 1.5, 2, 4, 6, 10],
        'subsample': [0.2, 0.4, 0.6, 0.8, 1.0],
        'colsample_bytree': [0.2, 0.4, 0.6, 0.8, 1.0],
        'max_depth': [1, 2, 3, 4, 5]
        }
# Fitting the paramater grid
fts = [f for f in train.columns if f not in ['whether he/she donated blood in May 2018']]

search1 = GridSearchCV(estimator = XGBClassifier(max_depth=5, min_child_weight=1, gamma=0, subsample=0.8, 
                                                  colsample_bytree=0.8, seed=42), 
                        param_grid = params, scoring='neg_log_loss', n_jobs=8, iid=False, cv=5, verbose=1)

search1.fit(train[fts],train['whether he/she donated blood in May 2018'])

search1.best_params_, search1.best_score_
XGB = XGBClassifier(max_depth=3, min_child_weight=2, gamma=4, subsample=0.8, colsample_bytree=0.8, seed=42)
prd, prd_tst, nme, acc = model_fit(XGB, 'XGBClassifier' , n_fld=5, dbg=False)<jupyter_output>XGBClassifier
Fold  1 acc : 0.768595
Fold  1 log_loss : 0.544000
Fold  2 acc : 0.800000
Fold  2 log_loss : 0.432665
Fold  3 acc : 0.815126
Fold  3 log_loss : 0.454831
Fold  4 acc : 0.773109
Fold  4 log_loss : 0.448045
Fold  5 acc : 0.789916
Fold  5 log_loss : 0.469783
Average AUC over folds: 0.758211
Accuracy : 0.789349
Average log_loss over folds: 0.469783<jupyter_text>Thoughts:The XGB Classifier performed the best and so will be used going forwards. However, it is worth noting that, as the XGB is a fairly complex ensemble, other methods which are easier to understand and more readily auditable, may be more desirable. If performance is the goal, when developing this model further, the possibility of an ensemble of multiple algorithms should be explored. The difference in the feature importance profiles suggests that the models are relatively uncorrelated. In which case this would be a promising option.   8. Evaluating the Test Set---<jupyter_code># Outputting the AUROC the Accuracy and the Log Loss statistics
Test_AUC = roc_auc_score(test['whether he/she donated blood in May 2018'], prd_tst)
prd_tst_bin = np.zeros(prd_tst.shape[0])
prd_tst_bin[prd_tst > 0.5] = 1
Test_accuracy = accuracy_score(test['whether he/she donated blood in May 2018'], prd_tst_bin)
Test_log_loss = log_loss(test['whether he/she donated blood in May 2018'], prd_tst)

print('AUC : %.6f' % Test_AUC)
print('Acc : %.6f' % Test_accuracy)
print('Log Loss : %.6f' % Test_log_loss)<jupyter_output>AUC : 0.735948
Acc : 0.780000
Log Loss : 0.492305<jupyter_text>9. Evaluating the Full Set---<jupyter_code># Outputting the AUROC the Accuracy and the Log Loss statistics
AUC = roc_auc_score(blood_donations_df['whether he/she donated blood in May 2018'], prd)
prd_bin = np.zeros(prd.shape[0])
prd_bin[prd > 0.5] = 1
Accuracy = accuracy_score(blood_donations_df['whether he/she donated blood in May 2018'], prd_bin)
Log_loss = log_loss(blood_donations_df['whether he/she donated blood in May 2018'], prd)

print('AUROC : %.6f' % AUC)
print('Accuracy : %.6f' % Accuracy)
print('Log Loss : %.6f' % Log_loss)<jupyter_output>AUROC : 0.803272
Accuracy : 0.808824
Log Loss : 0.436036<jupyter_text>Refitting the model on the entire dataset (to maximise data for the final fit)<jupyter_code># Resetting main dataset and refitting the model based on the full set to maximise data (This assumes that as the test set showed the model didn't materially overfit)
blood_donations_df = pd.read_excel('Case_Study_2_Blood_Donations_Data.xlsx')
blood_donations_df = blood_donations_df.drop(['Monetary (c.c. blood)'], axis=1)

blood_donations_df['Rate (months)'] = blood_donations_df['Frequency (times)']/blood_donations_df['Time (months)']
blood_donations_df['Rate (months)'] = blood_donations_df['Rate (months)'].round(2)

blood_donations_df['On Schedule'] = blood_donations_df['Recency (months)']*blood_donations_df['Rate (months)']
blood_donations_df['On Schedule'] = blood_donations_df['On Schedule'].round(1)

train =  blood_donations_df
test = blood_donations_df
XGB = XGBClassifier(max_depth=3, min_child_weight=9, gamma=1, subsample=1, colsample_bytree=1, seed=42)
prd, prd_tst, nme, acc = model_fit(XGB, 'XGBClassifier', n_fld = 5, dbg=False)
# Outputting the AUROC the Accuracy and the Log Loss statistics
AUC = roc_auc_score(blood_donations_df['whether he/she donated blood in May 2018'], prd)
prd_bin = np.zeros(prd.shape[0])
prd_bin[prd > 0.5] = 1
Accuracy = accuracy_score(blood_donations_df['whether he/she donated blood in May 2018'], prd_bin)
Log_loss = log_loss(blood_donations_df['whether he/she donated blood in May 2018'], prd)

print('AUC : %.6f' % AUC)
print('Acc : %.6f' % Accuracy)
print('Log Loss : %.6f' % Log_loss)<jupyter_output>AUC : 0.805697
Acc : 0.812834
Log Loss : 0.431892<jupyter_text>Plotting the confusion matrix<jupyter_code># Credit: http://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html
import itertools
def plot_confusion_matrix(cm, classes, title='Confusion matrix', cmap=plt.cm.Blues):
    """
    This function prints and plots the confusion matrix.
    """
    plt.figure(figsize=(10,7))
    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, rotation=45)
    plt.yticks(tick_marks, classes)

    fmt = 'd'
    thresh = cm.max() / 2.
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        plt.text(j, i, format(cm[i, j], fmt),
                 horizontalalignment="center",
                 color="white" if cm[i, j] > thresh else "black")

    plt.tight_layout()
    plt.ylabel('True label')
    plt.xlabel('Predicted label')
# Compute confusion matrix
cnf_matrix = confusion_matrix(blood_donations_df['whether he/she donated blood in May 2018'], prd_bin)
# np.set_printoptions(precision=2)

class_names = ['Non-Donor', 'Donor']
# Plot non-normalized confusion matrix
plt.figure()
plot_confusion_matrix(cnf_matrix, classes=class_names, title='Confusion matrix')<jupyter_output><empty_output><jupyter_text>Thoughts:As expected with an ensemble algorithm, they tend to under-predict the less common class (people who donated in May 2018). If this is a particularly undesirable outcome, methods such as a logistic regression will give a more realistic split between the classes.  Plotting the ROC curve<jupyter_code># Compute ROC curve and ROC area for each class
fpr, tpr, _ = roc_curve(blood_donations_df['whether he/she donated blood in May 2018'], prd)
roc_auc = auc(fpr, tpr)

plt.figure(figsize=(15,10))
plt.plot(fpr, tpr, color='coral', label='ROC curve (area = %0.2f)' % roc_auc)
plt.plot([0, 1], [0, 1], color='black', linestyle='--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.0])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver operating characteristic')
plt.legend(loc="lower right")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10163228584281216, 'fraction_numerical': 0.035119850909324596, 'mean_word_length': 4.709594569803706, 'pattern_counts': {'":': 0, '<': 98, '<?xml version=': 0, '>': 103, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '26', 'dir': 'jupyter-structured-clean-dedup', 'id': '338326', 'license': 'MIT', 'n_tokens_mistral': 10969, 'n_tokens_neox': 9818, 'n_words': 3393, 'path': 'Predicting Blood Donations.ipynb', 'repo_name': 'JakeCherrie/Predicting-Blood-Donations---tech-eval', 'size': '31122'}