<jupyter_start><jupyter_text>Training Models with Unequal Economic Error Costs***  Table of Contents1. [Introduction](introduction)1. [Background and Solution Overview](background)    1. [Importing Libraries](libraries)    1. [Defining Helper Functions](helpers)    1. [Importing Data](import_data)    1. [Preparing_Data](process_data)1. [Defining a Custom Loss Function](custom_loss)1. [Training the Model](train)1. [Making Predictions](predictions)1. [Analyzing the Results](score)1. [Conclusion](conclusion) The blog post associated with this notebook is located [here](https://aws.amazon.com/blogs/machine-learning/training-models-with-unequal-economic-error-costs-using-amazon-sagemaker/).  Introduction Many companies are turning to machine learning (ML) to improve customer and business outcomes. They use the power of ML models built over “big data” to identify patterns and find correlations. Then they can identify appropriate approaches or predict likely outcomes based on data about new instances. However, as ML models are approximations of the real world, some of these predictions will likely be in error. In some applications all types of prediction errors are truly equal in impact. In other applications, one kind of error can be much more costly or consequential than another – measured in absolute or relative terms, in dollars, time, or something else. For example, predicting someone does not have breast cancer when they do (a false negative error) will, according to medical estimates, likely have much greater cost or consequences than the reverse error. We may even be willing to tolerate more false positive errors if we sufficiently reduce the false negatives to compensate. In this blog post, we address applications with unequal error costs with the goal of reducing undesirable errors while providing greater transparency to the trade-offs being made. We show you how to train a model for a binary classification problem in which the costs of different kinds of misclassification are very different. To explore this tradeoff, we show you how to write a custom loss function – the metric that evaluates how well a model makes predictions – that incorporates asymmetric misclassification costs. We then show you how to train using that loss function. Further, we show how to evaluate the errors made by the model and how to compare models trained with different relative costs so that you can identify the model with the best economic outcome overall.The advantage of this approach is that it makes an explicit link between an ML model’s outcomes and errors and the business’ framework for decision-making. This approach requires the business to explicitly state its cost matrix, based on the specific actions to be taken on the predictions. The business can then evaluate the economic consequences of the model predictions on their overall processes, the actions taken based on the predictions, and their associated costs. This evaluation process moves well beyond simply assessing the classification results of the model. This approach can drive challenging discussions in the business, and force differing implicit decisions and valuations onto the table for open discussion and agreement.  Background and Solution Overview Although model training always aims to minimize errors, most models are trained to assume that all types of errors are equal. However, what if we know that the costs of different types of errors are not equal? For example, let's take a sample model trained on [UCI'S breast cancer diagnostic data set](https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+%28Diagnostic%29).[1](note_one) Clearly, a false positive prediction (predicting this person has breast cancer, when they do not) has very different consequences than a false negative prediction (predicting this person does not have breast cancer, when they do). In the first case, the consequence is an extra round of screening. In the second case, the cancer might be at a more advanced stage before it's discovered. To quantify these consequences they are often discussed in terms of their relative cost, which then allows trade-offs to be made. While we can debate what the exact costs of a false negative or a false positive prediction should be, we believe we’d all agree that they're not the same - although ML models are generally trained as if they are.  We can use a custom cost function to evaluate a model and see the economic impact of the errors the model is making (utility analysis). Elkan[2](note_two) showed that applying a cost function to the results of a model can be used to compensate for imbalanced samples when used in standard Bayesian and decision tree learning methods (for example: few loan defaults, versus a large sample of repaid loans). The custom function can also be used to perform this same compensation.We can also have the model “shift” its predictions in a fashion that reflects the difference in cost, by providing the costs of different types of errors to the model during training using a custom loss function. So, for example, in the breast cancer example we'd like the model to make fewer false negative errors and are willing to accept more false positives to achieve that goal. We may even be willing to give up some “correct” predictions in order to have fewer false negatives. At least, we'd like to understand the trade-offs we can make here. In our example, we'll use costs from the healthcare industry.[3](note_three),[4](note_four) In addition, we'd like to understand in how many cases the model's predictions are “almost” predicted as something else. For example, binary models use a cutoff (say, 0.5) to classify a score as “True” or "False." How many of our cases were in fact very close to the cut-off? Are our false negatives classified that way because their score was 0.499999? These details can’t be seen in the usual representations of confusion matrices or AUC measures. To help address these questions, we have developed a novel, graphical representation of the model predictions that allows us to examine these details, without depending on a specific threshold.In fact, there are likely cases where a model trained to avoid specific types of errors would begin to specialize in differentiating errors. Imagine a neural network that's been trained to believe that all misrecognitions of street signs are equal.[5](note_five) Now, imagine a neural network that's been trained that misrecognizing a stop sign as a sign for speed limit 45 mph is a far worse error than confusing two speed limit signs. It's reasonable to expect that the neural network would begin to recognize different features. We believe this is a promising research direction. Note that while the terms "cost function" and "loss function" are often used interchangeably, we differentiate between them in this post, and provide examples of each:* We use a "loss function" to train the model. Here, we specify the different weights of different kinds of errors. The relative weight of the errors is of most importance here.* We use a "cost function" to evaluate the economic impact of the model. For the cost function, we can specify the cost (or value) of correct predictions, as well as the cost of errors. Here, dollar costs are most appropriately used.This distinction allows us to further refine the model's behavior or to reflect differing influences from different constituencies. Although in this model we'll use the same set of costs (quality adjusted life years, QALY) for both functions, you could, for example, use relative QALY for the loss function, and costs of providing care for the cost function.We’ll break up this problem into three parts:1.	In "Defining a custom loss function," we show how to build a custom loss function that weights different errors unequally. The relative costs of the prediction errors are provided as hyperparameters at runtime, allowing the effects of different weightings on the model to be explored and compared. We build and demonstrate the use of a custom cost function to evaluate our “vanilla” model, which is trained to assume that all errors are equal.2.	In “Training the model,” we demonstrate how to train a model by using the custom loss function. We emulate and extend a [sample notebook](https://github.com/awslabs/amazon-sagemaker-examples/tree/master/introduction_to_applying_machine_learning/breast_cancer_prediction), which uses the UCI breast cancer diagnostic data set.3.	In “Analyzing the results,” we show how we can compare the models to better understand the distribution of predictions as compared to our threshold. We'll see that by training the model to avoid certain kinds of errors, we'll affect the distributions so that the model differentiates more effectively between its positive and negative predictions.Bibliography:1. <NAME>. and <NAME>. (2017). UCI Machine Learning Repository [http://archive.ics.uci.edu/ml]. Irvine, CA: University of California, School of Information and Computer Science.2. <NAME>. “The Foundations of Cost-Sensitive Learning.” In International Joint Conference on Artificial Intelligence, 17:973–978. Lawrence Erlbaum Associates Ltd, 2001.3. <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, and <NAME>. “Developing a Utility Decision Framework to Evaluate Predictive Models in Breast Cancer Risk Estimation.” Journal of Medical Imaging 2, no. 4 (October 2015). https://doi.org/10.1117/1.JMI.2.4.041005.4. Abbey, <NAME>., <NAME>, <NAME>, <NAME>, <NAME>, and <NAME>. “A Utility/Cost Analysis of Breast Cancer Risk Prediction Algorithms.” Proceedings of SPIE--the International Society for Optical Engineering 9787 (February 27, 2016). <NAME>, Kevin, <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, and <NAME>. “Robust Physical-World Attacks on Deep Learning Models.” ArXiv:1707.08945 [Cs], July 27, 2017. http://arxiv.org/abs/1707.08945.![image.png](attachment:image.png)  Importing Libraries<jupyter_code>import io
import os
import time
import pandas as pd
import numpy as np
import keras
from keras.models import Sequential
from keras.layers import Dense
import keras.backend as K
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import classification_report, roc_curve, roc_auc_score, precision_recall_fscore_support, accuracy_score
import matplotlib.pyplot as plt
from IPython.core.display import display, HTML
%matplotlib inline<jupyter_output><empty_output><jupyter_text>This cutoff (aka threshold) will be used to construct the confusion matrices and to color the distribution plots in the "Analyzing the Results" section of this notebook.<jupyter_code>cutoff = 0.5
#trying to predict only one class
num_classes = 1
#the data set contains 30 features
input_dim = 30<jupyter_output><empty_output><jupyter_text>Defining Helper Functions In this section, we define several helper functions that calculate various metrics used to judge the quality of the model. See top of each function for a detailed description of its purpose.<jupyter_code>def custom_pred_distro(positives, negatives, cutoff=0.5, title=None):
    '''This function generates distributions of predicted scores for actual positives and actual negatives. 
    
    Note that the cutoff argument only affects the coloring of the graphs. It does NOT affect any model
    results or predicted values.'''

    fig, axes = plt.subplots(2,1, figsize=(10,8))

    axes[0].spines['top'].set_visible(False)
    axes[0].spines['right'].set_visible(False)
    axes[0].set(xlim=[0,1], xticks=np.arange(0, 1, step=0.1), xlabel='Model Score', ylabel='Count', title='Actual Negatives')
    axes[0].hist(negatives[negatives>cutoff], color='C1', label='False Positives', bins=30)
    axes[0].hist(negatives[negatives<=cutoff], label='True Negatives', bins=30)
    axes[0].legend()
    
    axes[1].spines['top'].set_visible(False)
    axes[1].spines['right'].set_visible(False)
    axes[1].set(xlim=[0,1], xticks=np.arange(0, 1, step=0.1), xlabel='Model Score', ylabel='Count', title='Actual Positives')
    axes[1].hist(positives[positives>cutoff], label='True Positives', bins=30)
    axes[1].hist(positives[positives<=cutoff], label='False Negatives', bins=30)
    axes[1].legend()
    
    if title is not None:
        fig.suptitle(title, fontsize=16, fontweight='bold', x=0.52)
        plt.tight_layout(rect=[0, 0.03, 1, 0.95])
    else:
        plt.tight_layout()
    
    return None
def expected_value(positives, negatives, v_tn, v_fp, v_fn, v_tp, cutoff=0.5):
    '''This function calculates the expected value of the next test example. To do this, the function must calculate 
    the prob of classification/misclassification errors and that requires using a cutoff value to make discrete predictions.'''
    
    tp = (positives > cutoff).sum()
    fn = (positives <= cutoff).sum()

    tn = (negatives < cutoff).sum()
    fp = (negatives >= cutoff).sum()
    
    pos = fn + tp
    neg = fp + tn
    total = tn + fp + fn + tp
    
    fpr = fp/neg
    fnr = fn/pos
    
    n = neg/total 
    p = pos/total
    
    #given truth is negative
    ev_n = v_tn * (1-fpr) + v_fp * (fpr)
    #given truth is positive
    ev_p = v_tp * (1-fnr) + v_fn * (fnr)
    
    #total expected value
    ev = ev_n * n + ev_p * p 
    
    output = {
        'fpr': fpr,
        'fnr': fnr,
        'n': n,
        'p': p,
        'ev_n': ev_n,
        'ev_p': ev_p,
        'ev': ev
    }
    
    return output
def conf_matrix(positives, negatives, cutoff, title):
    '''This function draws a confusion matrix, using our cutoff.'''
 
    tp = (positives > cutoff).sum()
    fn = (positives <= cutoff).sum()

    tn = (negatives < cutoff).sum()
    fp = (negatives >= cutoff).sum()
    
    output = {
        'Predicted Negatives': [tn, fn],
        'Predicted Positives': [fp, tp]
    }
    
    df_table = pd.DataFrame(output, index=['Actual Negatives', 'Actual Positives'])

    display(HTML('<b>' + title + '</b>'))
    display(HTML(df_table.to_html()))<jupyter_output><empty_output><jupyter_text>Importing the Data Here we import the breast cancer screening data used to construct our model.<jupyter_code>data = pd.read_csv('https://archive.ics.uci.edu/ml/machine-learning-databases/breast-cancer-wisconsin/wdbc.data', header = None)

# specify columns extracted from wbdc.names
data.columns = ["id","diagnosis","radius_mean","texture_mean","perimeter_mean","area_mean","smoothness_mean",
                "compactness_mean","concavity_mean","concave points_mean","symmetry_mean","fractal_dimension_mean",
                "radius_se","texture_se","perimeter_se","area_se","smoothness_se","compactness_se","concavity_se",
                "concave points_se","symmetry_se","fractal_dimension_se","radius_worst","texture_worst",
                "perimeter_worst","area_worst","smoothness_worst","compactness_worst","concavity_worst",
                "concave points_worst","symmetry_worst","fractal_dimension_worst"] 

# save the data
data.to_csv("data.csv", sep=',', index=False)

# print the shape of the data file
print(data.shape)

# show the top few rows
display(data.head())

# describe the data object
display(data.describe())

# we will also summarize the categorical field diganosis 
display(data.diagnosis.value_counts())<jupyter_output>(569, 32)<jupyter_text>Preparing the Data We split the data into training and test sets.<jupyter_code>X = data.drop('id', axis=1)
y = X.pop('diagnosis')

x_train, x_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=1)<jupyter_output><empty_output><jupyter_text>We standardize the data since variance differs drastically across features and map the target variable to numeric values.<jupyter_code>scaler = StandardScaler().fit(x_train)

x_train_scld = scaler.transform(x_train)
x_test_scld = scaler.transform(x_test)
label_map = {'M': 1, 'B': 0 }
y_train = y_train.map(label_map)
y_test = y_test.map(label_map)
training_data = np.concatenate((y_train.values.reshape((426,1)), x_train_scld), axis=1)<jupyter_output><empty_output><jupyter_text>Defining a Custom Loss Function We now construct a loss function that weighs false positive errors differently from false negatives one. To do this, we build a binary classifier in Keras to use Keras' ability to accommodate user-defined loss functions.To build a loss function in Keras, we define a Python function that takes model predictions and ground-truth as arguments and returns a scalar value. In the custom function, we input the cost associated with a false negative error (fn_cost) and with a false positive error (fp_cost). Note that internally the loss function must use Keras backend functions to perform any calculations.The following function defines the loss of a single prediction as the difference between the prediction's ground-truth class and the predicted value weighted by the cost associated with misclassifying an observation from that ground-truth class. The total loss is the unweighted average of all of these losses. This is a relatively simple loss function, but building upon this foundation, more complex, situation-specific benefit and cost structures can be constructed and used to train models.<jupyter_code>def custom_loss_wrapper(fn_cost=1, fp_cost=1):
        
    def custom_loss(y_true, y_pred, fn_cost=fn_cost, fp_cost=fp_cost):
   
        h = K.ones_like(y_pred)
        fn_value = fn_cost * h
        fp_value = fp_cost * h

        weighted_values = y_true * K.abs(1-y_pred)*fn_value + (1-y_true) * K.abs(y_pred)*fp_value

        loss = K.mean(weighted_values)
        return loss
        
    return custom_loss<jupyter_output><empty_output><jupyter_text>Training the Model We construct and train three models so we can compare the predictions of various models using Keras' built-in loss function as well as our custom loss function. We use a binary classification model that predicts the probability that a tumor is malignant.The three models are:1.	A binary classification model that uses Keras' built-in binary cross-entropy loss with equal weights for false negative and false positive errors.2.	A binary classification model that uses the custom loss function defined previously with false negatives weighted 5 times as heavily as false positives.3.	A binary classification model that uses the custom loss function defined previously with false negatives weighted 200 times as heavily as false positives.The costs used in the last model's loss function are based upon the medical literature.3,4 The costs of screening are measured in QALYs. One QALY is defined as one year of life in full health (1 year x 1.0 health). For example, if an individual is at half health, that is, 0.5 of full health, then one year of life for that individual is equal to 0.5 QALYs (1 year x 0.5 health). Two years of life for that individual is worth 1 QALY (2 years x 0.5 health). | Outcome |	QALY ||----|---||True Negative |	0 ||False Positive|	-0.01288|| True Positive|	-0.3528||False Negative	|-2.52| Here, a true negative outcome is measured as the baseline of costs, that is, all other costs in the table are measured relative to a patient without breast cancer that tests negative for breast cancer. A woman with breast cancer that tests negative loses 2.52 QALYs relative to the baseline, and a woman without breast cancer that tests positive loses 0.0128767 QALYs (or about 4.7 days) relative to the baseline. A QALY has an estimated economic value of $100,000 USD. So these values can also be translated into dollar costs by multiplying the cost in QALYs by 100,000 USD. Given these values, a false negative error is about 200 times more costly than a false positive one. See the medical literature referenced in the introduction for more detail surrounding these costs.The middle model value of 5 was chosen for demonstration purposes. With these costs in hand, we can now estimate the model. Estimating the parameters of a model in Keras is a three-step process:1.	Defining the model.2.	Compiling the model.3.	Training the model.  Defining the Model Architecture First, we define the structure of the model. In this case, the model consists of a single node in a single layer. That is, for each model that follows, we add a single Dense layer with a single unit that takes a linear combination of features and passes that linear combination to a sigmoid function that outputs a value between 0 and 1. Again, the actual executable version of the code is in the Docker container, but is shown here for illustrative purposes.We'll provide the relative weights in a later step.<jupyter_code>#The 'built-in' model is trained on Keras' built-in binary crossentropy loss function.
model_builtin = Sequential()
model_builtin.add(Dense(units=num_classes, input_dim=input_dim,  activation='sigmoid'))

#The 'custom' model is trained on our custom loss function that weighs 
#false negatives 5 time's more heavily than false positives.
model_five = Sequential()
model_five.add(Dense(units=num_classes, input_dim=input_dim,  activation='sigmoid'))

#The 'medical' model is trained on our custom loss function that assigns weights 
#false negatives and false positives derived from the medical literature.
model_medical = Sequential()
model_medical.add(Dense(units=num_classes, input_dim=input_dim,  activation='sigmoid'))<jupyter_output><empty_output><jupyter_text>Compiling Model Next, let's compile the models. Compiling a model refers to configuring the learning process. We need to specify the optimization algorithm and the loss function that we will use to train the model. This is the step in which we incorporate our custom loss function and relative weights into the model training process.<jupyter_code>#Trained using built-in loss function
model_builtin.compile(loss='binary_crossentropy',
             optimizer='sgd',
             metrics=['accuracy'])


#Trained using custom loss function with false negatives weighted 5 times more heavily than false positives
custom_loss_five = custom_loss_wrapper(fn_cost=5, fp_cost=1)

model_five.compile(loss=custom_loss_five,
             optimizer='sgd',
             metrics=['accuracy'])

#Trained using custom loss function with false negatives weighted about 200 times more heavily than false positives
custom_loss_medical = custom_loss_wrapper(fn_cost=200, fp_cost=1)

model_medical.compile(loss=custom_loss_medical,
             optimizer='sgd',
             metrics=['accuracy'])<jupyter_output><empty_output><jupyter_text>Training the Model Now we’re ready to train the models. To do this, we call the fit method and provide the training data, number of epochs, and batch size. Whether you use a built-in or a custom loss function, the code is the same in this step.<jupyter_code>model_builtin.fit(x_train_scld, y_train, epochs=50, batch_size=32, verbose=0)
model_five.fit(x_train_scld, y_train, epochs=50, batch_size=32, verbose=0)
model_medical.fit(x_train_scld, y_train, epochs=50, batch_size=32, verbose=0)<jupyter_output><empty_output><jupyter_text>Making Predictions  Making Predictions<jupyter_code>#continuous and discrete predictions for builtin loss function model
y_builtin_pred = model_builtin.predict(x_test_scld)
y_builtin_pred_discrete = y_builtin_pred > cutoff

#continuous and discrete predictions for 5:1 custom loss function model
y_five_pred = model_five.predict(x_test_scld)
y_five_pred_discrete = y_five_pred > cutoff

#continuous and discrete predictions for 200:1 custom  loss function model
y_medical_pred = model_medical.predict(x_test_scld)
y_medical_pred_discrete = y_medical_pred > cutoff
#here we divide the model predictions based upon whether ground-truth is positive or negative
y_builtin_pred_pos = y_builtin_pred[y_test==1]
y_builtin_pred_neg = y_builtin_pred[y_test==0]

y_five_pred_pos = y_five_pred[y_test==1]
y_five_pred_neg = y_five_pred[y_test==0]

y_medical_pred_pos = y_medical_pred[y_test==1]
y_medical_pred_neg = y_medical_pred[y_test==0]<jupyter_output><empty_output><jupyter_text>Analyzing the Results What characteristics are we generally looking for in a well-performing model?1.	There should be a small number of false negatives, that is, a small number of malignant tumors classified as benign. 2.	Predictions should cluster closely around ground truth values, that is, predictions should cluster closely around 0 and 1. Keep in mind as you rerun this notebook that the data set used is small (569 instances), and therefore the test set is even smaller (143 instances). Because of this, the exact distribution of predictions and prediction errors of the model may vary from run to run due to sampling error. Despite this, the following general results hold across model runs.  Accuracy and the ROC Curve First, we'll show traditional measures of the model. Here, we generate Accuracy and AUC values for all three models.<jupyter_code>acc_builtin = accuracy_score(y_test, y_builtin_pred_discrete)
acc_five = accuracy_score(y_test, y_five_pred_discrete)
acc_medical = accuracy_score(y_test, y_medical_pred_discrete)
auc_builtin = roc_auc_score(y_test, y_builtin_pred)
auc_five = roc_auc_score(y_test, y_five_pred)
auc_medical = roc_auc_score(y_test, y_medical_pred)
pd.DataFrame({'Accuracy': [acc_builtin, acc_five, acc_medical], 'AUC': [auc_builtin, auc_five, auc_medical]} , index=['Built-in', '5:1', 'Medical'])<jupyter_output><empty_output><jupyter_text>Now, we'll generate ROC curves for all three models.<jupyter_code>fpr_builtin, tpr_builtin, thresholds_builtin = roc_curve(y_test, y_builtin_pred)
fpr_five, tpr_five, thresholds_five = roc_curve(y_test, y_five_pred)
fpr_medical, tpr_medical, thresholds_medical = roc_curve(y_test, y_medical_pred)
fig, axes = plt.subplots(1,1, figsize=(9, 6))

axes.set(xlabel='False Positive Rate', ylabel='True Positive Rate', title='ROC Curve')

axes.plot(fpr_builtin, tpr_builtin, label='Built-in')
axes.plot(fpr_five, tpr_five, label='5:1')
axes.plot(fpr_medical, tpr_medical, label='Medical')

axes.spines['top'].set_visible(False)
axes.spines['right'].set_visible(False)

axes.legend()<jupyter_output><empty_output><jupyter_text>By these traditional measures, the two custom loss function models do not perform as well as the built-in loss function (by a small margin).However: accuracy is less relevant in judging the quality of these models. In fact, accuracy may be lowest in the "best" model because we are willing to have more false positives as long as we decrease the number of false negatives sufficiently.Looking at the ROC curve and AUC score for these three models, all models appear very similar according to these measures. However, neither of these metrics show us how the predicted scores are distributed over the [0, 1] interval so we are not able to determine where those predictions are clustered.   Classification Report Keep in mind that the cost of a false negative is increasing as we move through these three models. That implies that the number of false negatives is likely to decrease in each successive model. What does this imply for the values in these classification reports? It implies that the negative class (benign) should have higher precision and that the positive class (malignant) should have higher recall. (Remember that precision = tp / (tp + fp); recall = tp / (tp + fn).) Remember that for our classification problem we are classifying tumors as benign or malignant. According to the costs reported in the medical literature cited previously, a false negative is much more costly than a false positive. Because of that, we want to classify all malignant tumors as such and are not bothered by that resulting in more false positive predictions (to a point). Therefore, for the negative class (benign), we care more about having a high precision, and for the positive class (malignant), we care more about having a high recall.<jupyter_code># Define precision / recall matrix for built in
display(HTML('<b>Built-in Loss Function</b>'))
print(classification_report(y_test, y_builtin_pred_discrete))
# Define precision / recall matrix for 5:1
display(HTML('<b>5:1 Custom Loss Function</b>'))
print(classification_report(y_test, y_five_pred_discrete))
# Define precision / recall matrix for medical
display(HTML('<b>Medical Custom Loss Function</b>'))
print(classification_report(y_test, y_medical_pred_discrete))<jupyter_output><empty_output><jupyter_text>These classification reports show that we've achieved our goal: the medical model has the highest precision for benign, and the highest recall for malignant.What this implies is that when using the medical model, we are least likely to falsely classify a malignant tumor as benign, and we are most likely to identify all malignant tumors as malignant.Looking at the detail of these reports allows us to see that the medical model is the "best" of these three models, despite having the lowest F1-score, and the lowest average precision and recall.  Confusion Matrix To better understand the errors, a better tool is the confusion matrix.Since our goal is to reduce the number of false negatives, the model with the fewest false negatives is "best", providing the increase in false positives is not excessive. As we move through these three confusion matrices, the cost of a false negative relative to a false positive increases. As such, we expect the number of false negatives to decrease and the number of false positives to increase. However, the number of true positives and true negatives may also shift, as we're training the model to weight differently than before.<jupyter_code># Print built-in loss confusion matrix
conf_matrix(y_builtin_pred_pos, y_builtin_pred_neg, cutoff, 'Built-in Loss Function' )
# Print 5:1 custom loss confusion matrix
conf_matrix(y_five_pred_pos, y_five_pred_neg, cutoff, '5:1 Custom Loss Function' )
# Print medical custom loss confusion matrix
conf_matrix(y_medical_pred_pos, y_medical_pred_neg, cutoff, 'Medical Custom Loss Function' )<jupyter_output><empty_output><jupyter_text>We can see from the results that modifying the loss function values provided when training the model allows us to shift the balance between the categories of error. Using different weightings for the relative cost has a significant impact on the errors, and moves some of the other predictions as well. An interesting direction for future research is to explore the changing features that are identified by the model in support of these prediction differences.This gives us a powerful lever to influence the model based on the moral, ethical, or economic impacts of the decisions we make about the relative weights of the different errors.  Custom Confusion Matrix A specific observation is classified as positive or negative by comparing its score to a threshold. Intuitively, the further away the score is from the threshold chosen, the higher is the assumed probability that the prediction is correct (assuming that the threshold value is well-chosen).When comparing the model's prediction and the threshold used for dividing classes, it's possible that the values are very close to the threshold. In the extreme, the difference in values between a "true" or a "false" could be less than the error between two different readings of an input sensor or measurement; or even, less than the rounding error of a floating point library. In the worst case, the majority of the scores for our observations could be clustered quite close to the threshold. These “close confusions” are not visible in the confusion matrix, or in the previous F1-scores or ROC curves.Intuitively, it's desirable to have the majority of the scores further away from the threshold, or, conversely, to identify the threshold based on gaps in the distribution of scores. (In cartography, for example, the [Jenks' natural breaks method](https://en.wikipedia.org/wiki/Jenks_natural_breaks_optimization) is frequently used to address the same problem.) The following graphs give us a tool to explore the relationship of the scores to the threshold.Each of the following sets of distribution plots shows the actual scores for each sample in the confusion matrix. In each set, the top histogram plots the distribution of predicted scores for all actual negatives, that is, predicted scores for benign tumors (the top row of the confusion matrix). The bottom histogram plots predicted scores for actual positives (the bottom row).The correctly classified observations on each plot are colored blue, and the incorrectly classified observations are colored orange. The threshold value of 0.5, used in other functions in this notebook, is used for coloring the plots. However, this threshold choice does NOT affect the actual scores or shape or level of the plots, only the coloring. Another threshold could be chosen, and the results in this section would still hold.In the charts below, a "good" distribution is one in which the predictions are largely grouped around the 0 and 1 points. More specifically, a "good" set of histograms would have the actual positives largely clustered around 1 with few false negatives, that is, few orange points. We would like to see the actual negatives clustered around 0, but for this use case we are willing to accept a prediction spread over the support with false positives as long as this gets us a small number of false negatives with predictions clustered around 1 for the actual positives.<jupyter_code>custom_pred_distro(y_builtin_pred_pos, y_builtin_pred_neg, cutoff, title='Built-in Loss Function')
custom_pred_distro(y_five_pred_pos, y_five_pred_neg, cutoff, title='5:1 Loss Function')
custom_pred_distro(y_medical_pred_pos, y_medical_pred_neg, cutoff, title='Medical Loss Function')<jupyter_output><empty_output><jupyter_text>We can see from these plots that as we increase the ratio, the distributions shift. As the ratio between the error types increases, the model pushes a larger number of samples to the extremes, essentially becoming much more discriminatory.We can also see that few of our actual positives have scores close to the cutoff. The model is demonstrating increased "certainty" in its classifications of the positives. Note that in many of our runs, 1 observation - 38 - is the lone incorrectly classified actual positive. This observation may have some different characteristics from the others, that may be indicative of a rare and different pattern that bears additional investigation.   Expected Value We now calculate the expected value (economic value) of each of the three classification models. The expected value captures the probability-weighted loss expressed in US dollars that an individual patient is expected to suffer if given a specific diagnostic test. The diagnostic test with the highest expected value is considered the “best” under this metric. The expected value is stated in US dollars. For an explanation of QALY and the dollar values associated with testing outcomes defined in the following cell, see the discussion of screening costs earlier in this blog post.<jupyter_code>dollar_per_qaly = 100000
v_tn = 0 * dollar_per_qaly
v_fp = -0.0128767 * dollar_per_qaly
v_fn = -2.52 * dollar_per_qaly
v_tp = -0.3528 * dollar_per_qaly<jupyter_output><empty_output><jupyter_text>Note that in this section we are now reflecting the value of all four possible test outcomes - true and false negatives, as well as true and false positives.<jupyter_code>ev_builtin = expected_value(y_builtin_pred_pos, y_builtin_pred_neg, v_tn=v_tn, v_fp=v_fp, v_fn=v_fn, v_tp=v_tp)
ev_five = expected_value(y_five_pred_pos, y_five_pred_neg, v_tn=v_tn, v_fp=v_fp, v_fn=v_fn, v_tp=v_tp)
ev_medical = expected_value(y_medical_pred_pos, y_medical_pred_neg, v_tn=v_tn, v_fp=v_fp, v_fn=v_fn, v_tp=v_tp)
df_ev = pd.DataFrame(data = [ev_builtin['ev'], ev_five['ev'], ev_medical['ev']], 
            index = ['Builtiin', '5:1', 'Medical'], 
            columns=['Expected Value'])

def currency_format(x):
    if x >= 0:
        return '${:,.2f}'.format(x)
    else:
        return '-${:,.2f}'.format(abs(x))
    
df_ev['Expected Value'] = df_ev['Expected Value'].map(currency_format)

df_ev<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06625556508033073, 'fraction_numerical': 0.010342062328899703, 'mean_word_length': 5.062699077954736, 'pattern_counts': {'":': 0, '<': 113, '<?xml version=': 0, '>': 117, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '535672', 'license': 'MIT-0', 'n_tokens_mistral': 9929, 'n_tokens_neox': 9329, 'n_words': 4579, 'path': 'custom-loss/Training_Models_with_Unequal_Economic_Error_Costs.ipynb', 'repo_name': 'aws-samples/amazon-sagemaker-custom-loss-function', 'size': '36351'}