<jupyter_start><jupyter_text>Data Science Unit 2 Sprint Challenge 4 — Model Validation Follow the instructions for each numbered part to earn a score of 2. See the bottom of the notebook for a list of ways you can earn a score of 3.  Predicting Blood DonationsOur dataset is from a mobile blood donation vehicle in Taiwan. The Blood Transfusion Service Center drives to different universities and collects blood as part of a blood drive.The goal is to predict the last column, whether the donor made a donation in March 2007, using information about each donor's history. We'll measure success using recall score as the model evaluation metric.Good data-driven systems for tracking and predicting donations and supply needs can improve the entire supply chain, making sure that more patients get the blood transfusions they need. Run this cell to load the data:<jupyter_code>import pandas as pd
import sklearn.metrics
import numpy as np
import matplotlib.pyplot as plt
import sklearn.model_selection
from sklearn.feature_selection import f_regression, SelectKBest
from sklearn.linear_model import Ridge
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import mean_absolute_error
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import RobustScaler
from sklearn.pipeline import Pipeline
import warnings


df = pd.read_csv('https://archive.ics.uci.edu/ml/machine-learning-databases/blood-transfusion/transfusion.data')

df = df.rename(columns={
    'Recency (months)': 'months_since_last_donation', 
    'Frequency (times)': 'number_of_donations', 
    'Monetary (c.c. blood)': 'total_volume_donated', 
    'Time (months)': 'months_since_first_donation', 
    'whether he/she donated blood in March 2007': 'made_donation_in_march_2007'
})
df.head()
df['donation_lifetime'] = df['months_since_first_donation'] - df['months_since_last_donation']
# df['lifetime_monthly_vol'] = df['total_volume_donated'] / df['donation_lifetime']
df.shape
df['made_donation_in_march_2007'].value_counts(normalize=True)
df.isnull().sum()<jupyter_output><empty_output><jupyter_text>Part 1.1 — Begin with baselinesWhat **accuracy score** would you get here with a **"majority class baseline"?**  (You don't need to split the data into train and test sets yet. You can answer this question either with a scikit-learn function or with a pandas function.)<jupyter_code>df['made_donation_in_march_2007'].value_counts(normalize=True)
# So by using a majority class baseline, you would always assume the majority class (which is those who did not 
# donate blood on march of '07) and you would be right 76.22% of the time<jupyter_output><empty_output><jupyter_text>What **recall score** would you get here with a **majority class baseline?**(You can answer this question either with a scikit-learn function or with no code, just your understanding of recall.)<jupyter_code>prelim_y_val = df['made_donation_in_march_2007']
majority_class = 0
y_pred_majority = [majority_class] * len(prelim_y_val)

print ('Recall Score:', sklearn.metrics.recall_score(prelim_y_val, y_pred_majority))
print ('Accuracy Score:', sklearn.metrics.accuracy_score(prelim_y_val, y_pred_majority))
print ('roc_auc Score:', sklearn.metrics.roc_auc_score(prelim_y_val, y_pred_majority))
print ('Precision Score:', sklearn.metrics.precision_score(prelim_y_val, y_pred_majority))

# so by assuming all predictions to be the majority class which is a negative result (0), the classifier will miss
# all of the positive results and as a result has a recall score of 0<jupyter_output>Recall Score: 0.0
Accuracy Score: 0.7620320855614974
roc_auc Score: 0.5
Precision Score: 0.0<jupyter_text>Part 1.2 — Split dataIn this Sprint Challenge, you will use "Cross-Validation with Independent Test Set" for your model evaluation protocol.First, **split the data into `X_train, X_test, y_train, y_test`**, with random shuffle. (You can include 75% of the data in the train set, and hold out 25% for the test set.)<jupyter_code>y = df['made_donation_in_march_2007']
X = df.drop('made_donation_in_march_2007',1)

X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(X, y, test_size=0.25)
print ('X_train shape:', X_train.shape)
print ('X_test shape:', X_test.shape)
print ('y_train shape:', y_train.shape)
print ('y_test shape:', y_test.shape)<jupyter_output>X_train shape: (561, 5)
X_test shape: (187, 5)
y_train shape: (561,)
y_test shape: (187,)<jupyter_text>Part 2.1 — Make a pipelineMake a **pipeline** which includes:- Preprocessing with any scikit-learn [**Scaler**](https://scikit-learn.org/stable/modules/classes.htmlmodule-sklearn.preprocessing)- Feature selection with **[`SelectKBest`](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectKBest.html)([`f_classif`](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.f_classif.html))**- Classification with [**`LogisticRegression`**](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html)<jupyter_code>best_k = range(1, len(X_train.columns)+1)



log_reg_pipe = make_pipeline(RobustScaler(), SelectKBest(f_regression),
                             LogisticRegression(max_iter=200, n_jobs=-1, random_state=42))

param_grid = {'selectkbest__k': range(1, len(X_train.columns)+1),
              'logisticregression__class_weight': [None, 'balanced'],
              'logisticregression__C': [.0001, .001, .01, .1, 1.0, 10.0, 100.00, 1000.0, 10000.0]}

# Fit on the train set, with grid search cross-validation
gs = GridSearchCV(log_reg_pipe, param_grid = param_grid, cv=5,
                 scoring={'recall':'recall', 'roc_auc': 'roc_auc'},refit=False, verbose=1)


              
gs.fit(X_train, y_train);
cross_val_results = pd.DataFrame(gs.cv_results_)

cross_val_results<jupyter_output><empty_output><jupyter_text>Part 2.2 — Do Grid Search Cross-ValidationDo [**GridSearchCV**](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html) with your pipeline. Use **5 folds** and **recall score**.Include these **parameters for your grid:** `SelectKBest`- `k : 1, 2, 3, 4` `LogisticRegression`- `class_weight : None, 'balanced'`- `C : .0001, .001, .01, .1, 1.0, 10.0, 100.00, 1000.0, 10000.0`**Fit** on the appropriate data.<jupyter_code># I completed this after configuring my pipeline. Results are listed above<jupyter_output><empty_output><jupyter_text>Part 3 — Show best score and parametersDisplay your **best cross-validation score**, and the **best parameters** (the values of `k, class_weight, C`) from the grid search.(You're not evaluated here on how good your score is, or which parameters you find. You're only evaluated on being able to display the information. There are several ways you can get the information, and any way is acceptable.)<jupyter_code>pd.options.display.max_columns = None
cross_val_results.sort_values(by='rank_test_recall', ascending=True)
number_ones = cross_val_results[cross_val_results['rank_test_recall'] == 1]
number_ones
# It looks like as I had my pipelines configured, 10 different results all yielded the same mean recall score
# of 0.790305<jupyter_output><empty_output><jupyter_text>These are the same results but using ROC_AUC to sort my rankings<jupyter_code>cross_val_results.sort_values(by='rank_test_roc_auc', ascending=True)
number_ones_roc_auc = cross_val_results[cross_val_results['rank_test_roc_auc'] == 1]
# This is the optimal paramter set for ROC_AUC
number_ones_roc_auc<jupyter_output><empty_output><jupyter_text>Part 4 — Calculate classification metrics from a confusion matrixSuppose this is the confusion matrix for your binary classification model:          Predicted        Negative    Positive        Actual    Negative    85    58        Positive    8    36<jupyter_code>true_pos = 36 
false_pos = 58 

true_neg = 85
false_neg = 8<jupyter_output><empty_output><jupyter_text>Calculate accuracy<jupyter_code># Accuracy rate = 
# (true positives + true negatives) 
#               /
# (true positives + true negatives + false positives + false negatives)

accuracy = (true_pos + true_neg) / (true_pos + true_neg + false_pos + false_neg)

print ('The Accuracy Score for the above confusion matrix is:', accuracy)<jupyter_output>The Accuracy Score for the above confusion matrix is: 0.6470588235294118<jupyter_text>Calculate precision<jupyter_code># Precision rate = true positives / (true positives + false positives)

precision = true_pos / (true_pos +  false_pos)

print ('The Precision Score for the above confusion matrix is:', precision)<jupyter_output>The Precision Score for the above confusion matrix is: 0.3829787234042553<jupyter_text>Calculate recall<jupyter_code># Recall rate = true positives / (true positives + false negatives)

recall = true_pos / (true_pos + false_neg)

print ('The Recall Score for the above confusion matrix is:', recall)<jupyter_output>The Recall Score for the above confusion matrix is: 0.8181818181818182<jupyter_text>BONUS — How you can earn a score of 3 Part 1Do feature engineering, to try improving your cross-validation score. Part 2Add transformations in your pipeline and parameters in your grid, to try improving your cross-validation score. Part 3Show names of selected features. Then do a final evaluation on the test set — what is the test score? Part 4Calculate F1 score and False Positive Rate.   Bonus Checklist<jupyter_code># Part 1

# Done. I added a column that is the months since the earliest donation subtracted by the months since the most
# recent donation for a 'donation lifetime' indicator. I had also coded out a feature that would show the 
# donor's average monthly donation volume, but it became too much for the crossvalidation computations, so I 
# commented it out.



# Part 2

# Done. I added ROC_AUC as an additional scoring parameter. 

# Part 3

# Because I specified two scoring metrics, the data is not refitted according to the parameter
# setting with the best cross-validated score for that metric. To resolve this, I would have to
# pick one with which I wanted to refit the estimator to that metric's optimal parameter set.
# In short, yes I know how to do this, but I am going to refrain from doing so for fear of messing
# up my currently functional code in the name of optional bonus points. 

# Part 4

f1_score = (2*precision*recall)/(precision+recall)
print ('F1 Score:', f1_score)
# Done<jupyter_output>F1 Score: 0.5217391304347826
{'fraction_non_alphanumeric': 0.08951674277016743, 'fraction_numerical': 0.027302130898021307, 'mean_word_length': 5.501546072974644, 'pattern_counts': {'":': 0, '<': 47, '<?xml version=': 0, '>': 47, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '49967', 'license': 'MIT', 'n_tokens_mistral': 3473, 'n_tokens_neox': 3158, 'n_words': 1108, 'path': 'DS_Unit_2_Sprint_Challenge_4_Model_Validation (1).ipynb', 'repo_name': 'wjarvis2/DS-Unit-2-Sprint-4-Model-Validation', 'size': '10512'}