<jupyter_start><jupyter_text>Data Science Unit 2 Sprint Challenge 4 — Model Validation Follow the instructions for each numbered part to earn a score of 2. See the bottom of the notebook for a list of ways you can earn a score of 3.<jupyter_code># imports

# general libraries
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from tabulate import tabulate

# sklearn
from sklearn.metrics import roc_curve, auc, accuracy_score, precision_score
from sklearn.metrics import recall_score, f1_score, confusion_matrix
from sklearn.metrics import classification_report, make_scorer

from sklearn.model_selection import GridSearchCV, validation_curve, learning_curve
from sklearn.model_selection import cross_val_score, train_test_split

from sklearn.pipeline import make_pipeline

from sklearn.preprocessing import StandardScaler

from sklearn.feature_selection import SelectKBest

from sklearn.linear_model import LogisticRegression

import warnings
from sklearn.exceptions import DataConversionWarning
warnings.filterwarnings(action='ignore', category=DataConversionWarning)<jupyter_output><empty_output><jupyter_text>Predicting Blood DonationsOur dataset is from a mobile blood donation vehicle in Taiwan. The Blood Transfusion Service Center drives to different universities and collects blood as part of a blood drive.The goal is to predict the last column, whether the donor made a donation in March 2007, using information about each donor's history. We'll measure success using recall score as the model evaluation metric.Good data-driven systems for tracking and predicting donations and supply needs can improve the entire supply chain, making sure that more patients get the blood transfusions they need. Run this cell to load the data:<jupyter_code>import pandas as pd

df = pd.read_csv('https://archive.ics.uci.edu/ml/machine-learning-databases/blood-transfusion/transfusion.data')

df = df.rename(columns={
    'Recency (months)': 'months_since_last_donation', 
    'Frequency (times)': 'number_of_donations', 
    'Monetary (c.c. blood)': 'total_volume_donated', 
    'Time (months)': 'months_since_first_donation', 
    'whether he/she donated blood in March 2007': 'made_donation_in_march_2007'
})<jupyter_output><empty_output><jupyter_text>Part 1.1 — Begin with baselinesWhat **accuracy score** would you get here with a **"majority class baseline"?**  (You don't need to split the data into train and test sets yet. You can answer this question either with a scikit-learn function or with a pandas function.)<jupyter_code># create our X and y variables
X = df.drop('made_donation_in_march_2007', axis=1)
y = df['made_donation_in_march_2007']

assert X.isnull().sum().sum() == 0
assert y.isnull().sum().sum() == 0

majority_class = y.mode()[0]
y_pred = [majority_class] * y.shape[0]
print ('Accuracy Score %.3f' % (accuracy_score(y, y_pred)))<jupyter_output>Accuracy Score 0.762<jupyter_text>What **recall score** would you get here with a **majority class baseline?**(You can answer this question either with a scikit-learn function or with no code, just your understanding of recall.)<jupyter_code>print ('Recall Score %.3f' % (recall_score(y, y_pred)))<jupyter_output>Recall Score 0.000<jupyter_text>Based on class imbalance alone, we're able to correctly identify the majority of samples. Around 76% of people in the dataset did not give blood.Recall is more nuanced. Scikit-learn automatically assumes that we want the recall score of the *positive* label. In the case above, we're always guessing the negative label. This is obviously very poor performance if we are indeed only concerned about recall score of the positive class.Recall for the negative class, however, is 100%. It's beneficial to look at all performance metrics for our simple majority classifier. This allows us to examine recall scores for different classes and class weightings.<jupyter_code>print ('Classification Report (Majority Classifier Baseline)\n\n', classification_report(y, y_pred))<jupyter_output>Classification Report (Majority Classifier Baseline)

               precision    recall  f1-score   support

           0       0.76      1.00      0.86       570
           1       0.00      0.00      0.00       178

   micro avg       0.76      0.76      0.76       748
   macro avg       0.38      0.50      0.43       748
weighted avg       0.58      0.76      0.66       748<jupyter_text>Part 1.2 — Split dataIn this Sprint Challenge, you will use "Cross-Validation with Independent Test Set" for your model evaluation protocol.First, **split the data into `X_train, X_test, y_train, y_test`**, with random shuffle. (You can include 75% of the data in the train set, and hold out 25% for the test set.)<jupyter_code>X_train, X_test, y_train, y_test = train_test_split(X,
                                                   y,
                                                   test_size=0.25,
                                                   random_state=1,
                                                   shuffle=True)<jupyter_output><empty_output><jupyter_text>Part 1.3 Feature EngineeringLet's start by taking a look at the training data.<jupyter_code># lets first take a look at the distribution of our variables
X_train.describe()
X_pairplot = X.copy()
X_pairplot['y'] = y_train
sns.pairplot(X_pairplot, hue='y');<jupyter_output><empty_output><jupyter_text>A few takeaways:- People who donated recently seem more likely to donate again- People who donated more frequently or more volume per donation seem more likely to donate again- All of our features are right-skewed, with fairly long tails for long time or frequent donors<jupyter_code>def wrangle(X):
  # returns a modified dataframe with additional features
  # specific to the blood donations dataset
  
  # include log of all features to account for skewed data
  for col in X.columns:
    X['log_' + col] = np.log(X[col])
    
  # binary feature if someone hasn't donated in 3 years or more
  X['recent_donor'] = X['months_since_last_donation'] < 36
    
  # include donation and volume per donation rates
  X['donations_per_month'] = (X['number_of_donations'] / (X['months_since_first_donation'] - X['months_since_last_donation']))
  X['volume_per_month'] = (X['total_volume_donated'] / (X['months_since_first_donation'] - X['months_since_last_donation']))
  
  return X

# wrangle test and train data
X_train = wrangle(X_train).replace([np.inf, -np.inf, np.nan],0)
X_test = wrangle(X_test).replace([np.inf, -np.inf, np.nan],0)<jupyter_output>/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:7: RuntimeWarning: divide by zero encountered in log
  import sys
/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:7: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  import sys
/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:10: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  # Remove the CWD from sys.path while we load stuff.
/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:13: SettingWithCopyWarning: 
A value is trying to be set on a copy of a sl[...]<jupyter_text>Part 2.1 — Make a pipelineMake a **pipeline** which includes:- Preprocessing with any scikit-learn [**Scaler**](https://scikit-learn.org/stable/modules/classes.htmlmodule-sklearn.preprocessing)- Feature selection with **[`SelectKBest`](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectKBest.html)([`f_classif`](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.f_classif.html))**- Classification with [**`LogisticRegression`**](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html)<jupyter_code>pipeline_lr = make_pipeline(StandardScaler(),
                           SelectKBest(),
                           LogisticRegression(solver='lbfgs'))<jupyter_output><empty_output><jupyter_text>Part 2.2 — Do Grid Search Cross-ValidationDo [**GridSearchCV**](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html) with your pipeline. Use **5 folds** and **recall score**.Include these **parameters for your grid:** `SelectKBest`- `k : 1, 2, 3, 4` `LogisticRegression`- `class_weight : None, 'balanced'`- `C : .0001, .001, .01, .1, 1.0, 10.0, 100.00, 1000.0, 10000.0`**Fit** on the appropriate data.<jupyter_code># initialize parameter grid
param_grid = {'selectkbest__k' : np.arange(1, 10),
             'logisticregression__class_weight' : [None, 'balanced'],# {0: 1, 1:10}], <- this will obviously give you recall of 100% for positive class
             'logisticregression__C' : [.0001, .001, .01, .1, 1.0, 10.0, 100.00, 1000.0, 10000.0]}

# create scorer using AVERAGE of recall scores for both classes
recall_scorer = make_scorer(recall_score, pos_label=1, **{'average' : 'macro'})

# grid search for best parameters
gs = GridSearchCV(estimator=pipeline_lr,
                  param_grid=param_grid,
                  scoring=recall_scorer,
                  cv=5,
                  n_jobs=-1)
gs = gs.fit(X_train, y_train)<jupyter_output>/usr/local/lib/python3.6/dist-packages/sklearn/model_selection/_search.py:841: DeprecationWarning: The default of the `iid` parameter will change from True to False in version 0.22 and will be removed in 0.24. This will change numeric results when test-set sizes are unequal.
  DeprecationWarning)<jupyter_text>**Please note** I'm using a modified recall score function above, not simply recall for the positive label.  Part 3 — Show best score and parametersDisplay your **best cross-validation score**, and the **best parameters** (the values of `k, class_weight, C`) from the grid search.(You're not evaluated here on how good your score is, or which parameters you find. You're only evaluated on being able to display the information. There are several ways you can get the information, and any way is acceptable.)<jupyter_code>print ('Best CV Recall Score (Training Data): %.3f' % gs.best_score_)
print ('\nFinal Hyperparameter Values:\n')
print (tabulate([[param, gs.best_params_[param]] for param in gs.best_params_], headers=['Hyperparameter', 'Optimal Value']))<jupyter_output>Best CV Recall Score (Training Data): 0.669

Final Hyperparameter Values:

Hyperparameter                    Optimal Value
--------------------------------  ---------------
logisticregression__C             0.0001
logisticregression__class_weight  balanced
selectkbest__k                    1<jupyter_text>If we're *only* trying to optimize recall score for the positive label, it's pretty easy to modify logistic regression class weights to ensure recall is 100%. Unsuprisingly, given we're optimizing for average recall score between classes, the optimal class weight is 'balanced'.Let's look at some learning and validation curves for our training size and regularization parameter.  Some Fun Learning and Validation Curves<jupyter_code># examines how our model learns as it sees more samples
train_sizes, train_scores, test_scores =\
                learning_curve(estimator=pipeline_lr,
                             X=X_train,
                             y=y_train,
                             scoring=recall_scorer,
                             train_sizes=np.linspace(0.1,1.0,10))
train_mean = np.mean(train_scores, axis=1)
train_std = np.std(train_scores, axis=1)
test_mean = np.mean(test_scores, axis=1)
test_std = np.std(test_scores, axis=1)
plt.plot(train_sizes, train_mean,
        color='blue', marker='o',
        markersize=5, label='training recall')

plt.fill_between(train_sizes,
               train_mean + train_std,
               train_mean - train_std,
               alpha=0.15, color='blue')
plt.plot(train_sizes, test_mean,
        color='green', marker='s',
        markersize=5, label='validation recall')

plt.fill_between(train_sizes,
               test_mean + test_std,
               test_mean - test_std,
               alpha=0.15, color='green')
plt.grid()
plt.xlabel('Number of Training Samples')
plt.ylabel('Recall')
plt.legend(loc='lower right')
plt.show()<jupyter_output>/usr/local/lib/python3.6/dist-packages/sklearn/model_selection/_split.py:2053: FutureWarning: You should specify a value for 'cv' instead of relying on the default value. The default value will change from 3 to 5 in version 0.22.
  warnings.warn(CV_WARNING, FutureWarning)
/usr/local/lib/python3.6/dist-packages/sklearn/feature_selection/univariate_selection.py:114: UserWarning: Features [8] are constant.
  UserWarning)
/usr/local/lib/python3.6/dist-packages/sklearn/feature_selection/univariate_selection.py:115: RuntimeWarning: invalid value encountered in true_divide
  f = msb / msw
/usr/local/lib/python3.6/dist-packages/sklearn/feature_selection/univariate_selection.py:114: UserWarning: Features [8] are constant.
  UserWarning)
/usr/local/lib/python3.6/dist-packages/sklearn/feature_selection/univariate_selection.py:115: RuntimeWarning: invalid value encountered in true_divide
  f = msb / msw
/usr/local/lib/python3.6/dist-packages/sklearn/feature_selection/univariate_selection.py:114: U[...]<jupyter_text>The relationship between training and validation score appears to be stable after around 200 samples are observed.<jupyter_code>c_range = [.0001, .001, .01, .1, 1.0, 10.0, 100.00, 1000.0, 10000.0]
train_scores, test_scores = validation_curve(estimator=pipeline_lr,
                                            X=X_train,
                                            y=y_train,
                                            scoring=recall_scorer,
                                            param_name='logisticregression__C',
                                            param_range=c_range,
                                            cv=10)
train_mean = np.mean(train_scores, axis=1)
train_std = np.std(train_scores, axis=1)
test_mean = np.mean(test_scores, axis=1)
test_std = np.std(test_scores, axis=1)
plt.plot(c_range, train_mean,
        color='blue', marker='o',
        markersize=5, label='training recall')

plt.fill_between(c_range,
               train_mean + train_std,
               train_mean - train_std,
               alpha=0.15, color='blue')
plt.plot(c_range, test_mean,
        color='green', marker='s',
        markersize=5, label='validation recall')

plt.fill_between(c_range,
               test_mean + test_std,
               test_mean - test_std,
               alpha=0.15, color='green')
plt.grid()
plt.xlabel('Parameter C')
plt.ylabel('Recall')
plt.legend(loc='lower right')
plt.ylim([0.4,0.7])
plt.show()<jupyter_output><empty_output><jupyter_text>The relationship between training and test scores appears to be stable anywhere after a very small value for C.  Final FeaturesWhich features did our model end up selecting?<jupyter_code>selected_mask = gs.best_estimator_.named_steps['selectkbest'].get_support() 

print('Features Selected:')
for name in X_train.columns[selected_mask]:
    print(name)

print('\nFeatures Not Selected:')
for name in X_train.columns[~selected_mask]:
    print(name)<jupyter_output>Features Selected:
log_months_since_last_donation

Features Not Selected:
months_since_last_donation
number_of_donations
total_volume_donated
months_since_first_donation
log_number_of_donations
log_total_volume_donated
log_months_since_first_donation
recent_donor
donations_per_month
volume_per_month<jupyter_text>Evaluation on Test DataNow, let's do a final evaluation on our test set, and look at our confusion matrix for the final model.<jupyter_code>print ('Recall Score (Test Data): %.3f' % gs.score(X_test, y_test))<jupyter_output>Recall Score (Test Data): 0.706<jupyter_text>This testing score is satisfactory for our basic model for two reasons.First, it significantly exceeds our baseline recall score of 0.5 (we kind of cheated using all the data but still).Second, the testing score is very close to our best model's cross validation score on the training data, leading us to believe the model should generalize well.<jupyter_code>confmat = confusion_matrix(y_test, gs.predict(X_test))
fig, ax = plt.subplots(figsize=(5,5))
ax.matshow(confmat, cmap=plt.cm.Blues, alpha=0.3)
for i in range(confmat.shape[0]):
    for j in range(confmat.shape[1]):
        ax.text(x=j, y=i,
               s=confmat[i, j],
               va='center', ha='center')
plt.title('Test Data Confusion Matrix')
plt.xlabel('predicted label')
plt.ylabel('true label')
plt.show()<jupyter_output><empty_output><jupyter_text>Part 4 — Calculate classification metrics from a confusion matrixSuppose this is the confusion matrix for your binary classification model:          Predicted        Negative    Positive        Actual    Negative    85    58        Positive    8    36   Calculate accuracy<jupyter_code>accuracy = (36 + 85) / (36 + 85 + 58 + 8)
print ('Accuracy Score: %.3f' % accuracy)<jupyter_output>Accuracy Score: 0.647<jupyter_text>Calculate precision<jupyter_code>precision = (36) / (36 + 58)
print ('Precision Score: %.3f' % precision)<jupyter_output>Precision Score: 0.383<jupyter_text>Calculate recall<jupyter_code>recall = (36) / (36 + 8)
print ('Recall Score: %.3f' % recall)<jupyter_output>Recall Score: 0.818<jupyter_text>Calculate F1 score<jupyter_code>f1 = 2  * (precision * recall) / (precision + recall)
print ('F1 Score: %.3f' % f1)<jupyter_output>F1 Score: 0.522<jupyter_text>Calculate False Positve Rate<jupyter_code>fpr = 58 / (58 + 85)
print ('False Positive Rate: %.3f' % fpr)<jupyter_output>False Positive Rate: 0.406
{'fraction_non_alphanumeric': 0.0985452315924419, 'fraction_numerical': 0.024524831391784182, 'mean_word_length': 3.858380720281614, 'pattern_counts': {'":': 0, '<': 73, '<?xml version=': 0, '>': 71, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 25, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '529827', 'license': 'MIT', 'n_tokens_mistral': 5791, 'n_tokens_neox': 5325, 'n_words': 1634, 'path': 'DS_Unit_2_Sprint_Challenge_4_Model_Validation.ipynb', 'repo_name': 'zangell44/DS-Unit-2-Sprint-4-Model-Validation', 'size': '17941'}