<jupyter_start><jupyter_text>Applications of ClassificationWe will perform a **two-class classification** using **logistic regression**.   A classifier is a machine learning model that separates the **label** into categories or **classes**.   In other words, classification models are **supervised** machine learning models which predict a categorical label.The German Credit bank customer data is used to determine if a particular person is a good or bad credit risk.  - Thus, credit risk of the customer is the classes you must predict.  - In this case, the cost to the bank of issuing a loan to a bad risk customer is five times that of denying a loan to a good customer. This fact will become important when evaluating the performance of the model. **Logistic regression** is a linear model but with a nonlinear response.  - The response is binary, $\{ 0,1 \}$, or positive and negative.  - The response is the prediction of the category. We will learn: -  How to prepare data for classification models using scikit-learn. -  Constructing a classification model using scikit-learn.-  Evaluating the performance of the classification model. -  Using techniques such as reweighting the labels and changing the decision threshold to change the trade-off between false positive and false negative error rates.   Basics of logistic regression<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
import numpy.random as nr
import math
from sklearn import preprocessing
import sklearn.model_selection as ms
from sklearn import linear_model
import sklearn.metrics as sklm

%matplotlib inline<jupyter_output><empty_output><jupyter_text>- Logistic regression is widely used as a classification model.  - Logistic regression is linear model, with a binary response, `{False, True}` or `{0, 1}`. You can think of this response as having a Binomial distribution. - For linear regression the response is just, well, linear.  Logistic regression is a linear regression model with a nonlinear output.  The response of the linear model is transformed or 'squashed' to values close to 0 and 1 using a **sigmoidal function**, also known as the **logistic function**.  The result of this transformation is a response which is the log likelihood for each of the two classes. The sigmoidal or logistic function can be expressed as:$$f(x) = \frac{1}{1 + e^{-\kappa(x - x_0)}} \\\kappa = steepness$$Compute and plot an example of the logistic function:<jupyter_code>xseq = np.arange(-7, 7, 0.1)

logistic = [math.exp(v)/(1 + math.exp(v)) for v in xseq]

plt.plot(xseq, logistic, color = 'red')
plt.plot([-7,7], [0.5,0.5], color = 'blue')
plt.plot([0,0], [0,1], color = 'blue')
plt.title('Logistic function for two-class classification')
plt.ylabel('log likelihood')
plt.xlabel('Value of output from linear regression')<jupyter_output><empty_output><jupyter_text>Let's make this a bit more concrete with a simple example. Say we have a linear model:$$\hat{y} = \beta_0 + \beta_1\ x$$Now, depending on the value of $\hat{y}$ we want to classify the output from a logistic regression model as either `0` or `1`. Substituting the linear model into the logistic function creates the following expression:$$F(\hat{y}) = \frac{1}{1 + e^{-\kappa(\beta_0 + \beta_1\ x)}} $$This way, we **transform the continuous output of the linear model defined on $-\infty \le \hat{y} \le \infty$ to a binary response**, $0 \le F(\hat{y}) \le 1$.  Load and prepare the data setLoad the dataset and assigns human-readable names to the columns.<jupyter_code>credit = pd.read_csv('German_Credit_Preped.csv')
print(credit.shape)
credit.head()<jupyter_output>(1000, 22)<jupyter_text>There are 22 columns, 1 customer identifier column, 20 features, plus a label column. These features represent information a bank might have on its customers.  Note: The label has significant **class imbalance**.  Class imbalance means that there are  unequal numbers of cases for the categories of the label. Examine the class imbalance in these data below.<jupyter_code>credit_counts = credit[['credit_history', 'bad_credit']].groupby('bad_credit').count()
print(credit_counts)<jupyter_output>credit_history
bad_credit                
0                      700
1                      300<jupyter_text>Notice that only 30% of the cases have bad credit.  This is not surprising, since a bank would typically retain customers with good credit.  However, this imbalance will bias the training of any model.   Prepare data for scikit-learn modelWith the data prepared, it is time to create the numpy arrays required for the scikit-learn model. Create a numpy array of the label values:<jupyter_code>labels = np.array(credit['bad_credit'])
labels<jupyter_output><empty_output><jupyter_text>Now, create the numpy feature array or **model matrix**.  As first step, the **categorical variables need to be recoded as binary dummy variables**.  This is a 3 step process:1. Encode the categorical string variables as integers.2. Transform the integer coded variables to dummy variables. 3. Append each dummy coded categorical variable to the model matrix. Perform this processing:<jupyter_code>def encode_string(cat_features):
    ## First encode the strings to numeric categories
    enc = preprocessing.LabelEncoder()
    enc.fit(cat_features)
    enc_cat_features = enc.transform(cat_features)
    ## Now, apply one hot encoding
    ohe = preprocessing.OneHotEncoder()
    encoded = ohe.fit(enc_cat_features.reshape(-1,1))
    return encoded.transform(enc_cat_features.reshape(-1,1)).toarray()

categorical_columns = ['credit_history', 'purpose', 'gender_status', 
                       'time_in_residence', 'property']

Features = encode_string(credit['checking_account_status'])
for col in categorical_columns:
    temp = encode_string(credit[col])
    Features = np.concatenate([Features, temp], axis = 1)

print(Features.shape)
print(Features[:2, :])<jupyter_output>(1000, 31)
[[0. 1. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 1. 0.
  0. 0. 1. 0. 0. 1. 0.]
 [1. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 1. 0. 0. 0. 0.
  1. 0. 0. 0. 0. 1. 0.]]<jupyter_text>Next, the **numeric features must be concatenated to the numpy array** by executing the code below:<jupyter_code>Features = np.concatenate([Features, np.array(credit[['loan_duration_mo', 'loan_amount', 
                            'payment_pcnt_income', 'age_yrs']])], axis = 1)
print(Features.shape)
print(Features[:2, :])<jupyter_output>(1000, 35)
[[0.000e+00 1.000e+00 0.000e+00 0.000e+00 0.000e+00 1.000e+00 0.000e+00
  0.000e+00 0.000e+00 0.000e+00 0.000e+00 0.000e+00 0.000e+00 0.000e+00
  0.000e+00 0.000e+00 1.000e+00 0.000e+00 0.000e+00 0.000e+00 0.000e+00
  0.000e+00 1.000e+00 0.000e+00 0.000e+00 0.000e+00 1.000e+00 0.000e+00
  0.000e+00 1.000e+00 0.000e+00 6.000e+00 1.169e+03 4.000e+00 6.700e+01]
 [1.000e+00 0.000e+00 0.000e+00 0.000e+00 0.000e+00 0.000e+00 1.000e+00
  0.000e+00 0.000e+00 0.000e+00 0.000e+00 0.000e+00 0.000e+00 0.000e+00
  0.000e+00 0.000e+00 1.000e+00 0.000e+00 0.000e+00 1.000e+00 0.000e+00
  0.000e+00 0.000e+00 0.000e+00 1.000e+00 0.000e+00 0.000e+00 0.000e+00
  0.000e+00 1.000e+00 0.000e+00 4.800e+01 5.951e+03 2.000e+00 2.200e+01]]<jupyter_text>With the dummy variables the original 6 categorical features are now 31 dummy variables. With the 4 numeric features there are a total of 35.  We must **split the cases into training and test data sets**. This step is critical.  If machine learning models are tested on the training data, the results will be both biased and overly optimistic.The code below performs the following processing:1. An index vector is Bernoulli sampled using the `train_test_split` function from the `model_selection` package of scikit-learn. 2. The first column of the resulting index array contains the indices of the samples for the training cases. 3. The second column of the resulting index array contains the indices of the samples for the test cases.<jupyter_code>## Randomly sample cases to create independent training and test data
nr.seed(9988)
indx = range(Features.shape[0])
indx = ms.train_test_split(indx, test_size = 300)
X_train = Features[indx[0],:]
y_train = np.ravel(labels[indx[0]])
X_test = Features[indx[1],:]
y_test = np.ravel(labels[indx[1]])<jupyter_output><empty_output><jupyter_text>There is just one more step in preparing this data.  **Numeric features must be rescaled** so they have a similar range of values.  Rescaling prevents features from having an undue influence on model training simply because then have a larger range of numeric variables.The code below uses the **StandardScaler** function to Zscore scale the numeric features.  - Notice that the scaler is fit only on the training data.  - The trained scaler is these applied to the test data.  - Test data should always be scaled using the parameters from the training data.<jupyter_code>scaler = preprocessing.StandardScaler().fit(X_train[:,34:])
X_train[:,34:] = scaler.transform(X_train[:,34:])
X_test[:,34:] = scaler.transform(X_test[:,34:])
X_train[:2,]<jupyter_output><empty_output><jupyter_text>The four numeric features are now scaled.   Construct the logistic regression model**Compute the logistic regression model**:1. Define a logistic regression model object using the `LogisticRegression` method from the scikit-learn `linear_model` package.2. Fit the linear model using the numpy arrays of the features and the labels for the training data set.<jupyter_code>logistic_mod = linear_model.LogisticRegression(max_iter=1000) 
logistic_mod.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>The model has been computed. Notice that the configuration of the model object has been printed. In this case, only default settings are shown, since no arguments were given to create the model object.  Print and examine the **model coefficients**:<jupyter_code>print(logistic_mod.intercept_)
print(logistic_mod.coef_)<jupyter_output>[-0.78020881]
[[-9.10421391e-03  7.32480046e-01 -1.84616791e-01 -1.28784704e+00
   3.23154986e-01 -1.00425400e+00 -3.81925313e-01  4.29731624e-01
  -1.15795296e-01 -7.79534038e-03  4.07598387e-01 -4.65009603e-01
   3.31166699e-02  5.07576589e-02 -1.28101517e-01 -5.00668044e-02
  -5.24691560e-01  3.83427329e-03 -6.87301624e-02 -7.35049189e-02
   3.21331748e-02 -1.08306396e-01 -5.99409857e-01 -4.98950002e-01
   2.56213836e-01 -3.72428068e-01 -1.33923763e-01 -3.37103555e-01
  -2.52532397e-02 -5.88879934e-01  2.02148732e-01  2.25401937e-02
   5.18467718e-05  2.23076293e-01 -1.90972876e-01]]<jupyter_text>First of all, notice that model coefficients look just as they would for an regression model. This is expected as previously explained.  Additionally, nearly all the coefficients have the same magnitude indicating this model is likely to be overfit, given the number of features.  Recall that the logistic regression model outputs probabilities for each class.  The class with the highest probability is taken as the score (prediction).  Compute and display a sample of these **class probabilities for the test feature set**.<jupyter_code>probabilities = logistic_mod.predict_proba(X_test)
print(probabilities[:15,:])<jupyter_output>[[0.84286775 0.15713225]
 [0.39791319 0.60208681]
 [0.94350947 0.05649053]
 [0.32550658 0.67449342]
 [0.56687887 0.43312113]
 [0.96879947 0.03120053]
 [0.93551799 0.06448201]
 [0.57787652 0.42212348]
 [0.87418465 0.12581535]
 [0.89614503 0.10385497]
 [0.56747878 0.43252122]
 [0.40446097 0.59553903]
 [0.49703273 0.50296727]
 [0.73616886 0.26383114]
 [0.40982895 0.59017105]]<jupyter_text>The first column is the probability of a score of $0$ and the second column is the probability of a score of $1$.  Notice that for most, but not all cases, the probability of a score of $0$ is higher than $1$.   Score and evaluate the classification modelNow that the **class probabilities** have been computed these values **must be transformed into actual class scores**.  Recall that the log likelihoods for two-class logistic regression are computed by applying the sigmoid or logistic transformation to the output of the linear model.  The simple choice is to **set the threshold** between the two likelihoods at $0.5$.  The code below applies this initial threshold to the probability of a score of $0$ for the test data.  A few examples along with the known labels are then displayed.<jupyter_code>def score_model(probs, threshold):
    return np.array([1 if x > threshold else 0 for x in probs[:,1]])
scores = score_model(probabilities, 0.5)
print(np.array(scores[:15]))
print(y_test[:15])<jupyter_output>[0 1 0 1 0 0 0 0 0 0 0 1 1 0 1]
[0 0 0 1 0 0 0 0 1 0 0 1 1 0 1]<jupyter_text>Some of the positive ($1$) predictions agree with the test labels in the second row, but several do not. Given the results of the test data, how can you quantify the performance of the model?  In general, **always use multiple metrics to evaluate the performance of any machine learning model**, including classifiers. For classifiers there are a number of metrics commonly used. The **confusion matrix** lays out the correctly and incorrectly classified cases in a tabular format. There are various metrics derived from the values in the confusion matrix. Some of the common cases are briefly reviewed below. **1. Confusion matrix**As already stated, the confusion matrix lays out correctly and incorrectly classified cases. For the binary (two-class) case the confusion matrix is organized as follows:| | Scored Positive | Scored Negative|  |------|:------:|:------:| |**Actual Positive** | True Positive | False Negative ||**Actual Negative**| False Positive | True Negative |   Here the four elements in the matrix are defined as:    **True Positive** or **TP** are cases with positive labels which have been correctly classified as positive.     **True Negative** or **TN** are cases with negative labels which have been correctly classified as negative.  **False Positive** or **FP** are cases with negative labels which have been incorrectly classified as positive.   **False Negative** or **FN** are cases with positive labels which have been incorrectly classified as negative.When creating a confusion matrix it is important to understand and maintain a convention for which differentiating positive and negative label values. The usual convention is to call the $1$ case positive and the $0$ case negative. Notice that there is an ambiguity in which case is considered positive and which is considered negative when the confusion matrix is computed. Whenever you examine a confusion matrix it is a good idea to spend a moment and decide which case is which. This step will help you relate the results to the problem at hand. **2. Accuracy**Accuracy is a simple and often misused metric. In simple terms, accuracy is the fraction of cases correctly classified. For a two-class classifier accuracy is written as:$$Accuracy = \frac{TP+TN}{TP+FP+TN+FN}$$Accuracy can be quite misleading. For example, say a classifier is used to detect fraudulent accounts and the rate of fraud is less than 1%. A naive model would be to say all accounts are not fraudulent. This model has accuracy exceeding  0.99. This sounds impressive, but is clearly useless. **3. Precision**Precision is the fraction of correctly classified label cases out of all cases classified with that label value. We can express precision by the following relationship:$$Precision = \frac{M_{i,i}}{\sum_j M_{i,j}}$$In other words, the precision statistic is the number of correctly classified cases for the label value divided by all the cases in the column. Thus, precision is sensitive to the number of cases correctly classified for a given score value. **4. Recall**  Recall is the fraction of cases of a label value correctly classified out of all cases that actually have that label value. We can express recall by the following relationship:$$Recall = \frac{M_{i,i}}{\sum_i M_{i,j}}$$In other words, the recall statistic is the number of correctly classified cases for the label value divided by all the cases in the row.  Thus, precision is sensitive to the number of cases correctly classified for a given true label value. **5. F1**The F1 statistic is weighted average of precision and recall. We can express F1 by the following relationship:$$F1 = 2 * \frac{precision * recall}{precision + recall}$$In other words, F1 is a weighted metric for overall model performance. **6. ROC** and **AUC**The receiver operating characteristic or **ROC is a curve that displays the relationship between the true positive rate on the vertical axis and false positive rate on the horizontal axis.** The ROC curve shows the tradeoff between true positive rate and false positive rate. An example is illustrated below.   In principle, we can pick the desired operating point for a classifier on this curve.  - Towards the left favors low false positive rate at the expense of true positive rate.  - Towards the right favors high true positive rate at the expense of higher false positive rate.  **ROC curve with values of AUC for balanced two-class problem**The **AUC is the area or integral under the ROC curve. The overall performance of the classifier is measured by the area under the curve or AUC**.  But, how can you interpret a specific AUC value?  - The higher the AUC the lower the increase in false positive rate required to achieve a required true positive rate.  - For an ideal classifier the AUC is 1.0. A true positive rate is achieved with a 0 false positive rate.  This behavior means that AUC is useful for comparing classifiers.  The classifier with higher AUC is generally the better one.- For balanced cases, random guessing gives an AUC or 0.5.  A balanced case has equal numbers of positive and negative cases.  So Bernoulli sampling (random guessing) with a probability $p$ for the positive case, will produce a ROC curve that runs diagonally from $0.0,0.0$ to $1.0,1.0$.  The area under this triangular region is 0.5.  It is often said that a classifier with an AUC of greater than 0.5 is better than random guessing.  - But, **for unbalanced cases this statement is not in true in general**. ******Note:** The term receive operating characteristic may seem a bit odd in the machine learning context. This term arose in the early days of radar engineering as a metric to measure the tradeoff between radar signal receiver correctly detecting a target, say an aircraft, and producing a positive response from noise, such as flying birds or clouds. A radar receiver would be adjusted to the desired operating point along its ROC curve. **** **The code below computes and displays the aforementioned classifier performance metrics.**  The code metrics are computed using the `precision_recall_fscore_support` and `accuracy_score` functions from the `metrics` package of scikit-learn.  The confusion matrix is computed using the `confusion_matrix` function from this same package.  Examine the results for the logistic regression model:<jupyter_code>def print_metrics(labels, scores):
    metrics = sklm.precision_recall_fscore_support(labels, scores,zero_division=0)
    conf = sklm.confusion_matrix(labels, scores)
    print('                 Confusion matrix')
    print('                 Score positive    Score negative')
    print('Actual positive    %6d' % conf[0,0] + '             %5d' % conf[0,1])
    print('Actual negative    %6d' % conf[1,0] + '             %5d' % conf[1,1])
    print('')
    print('Accuracy  %0.2f' % sklm.accuracy_score(labels, scores))
    print(' ')
    print('           Positive      Negative')
    print('Num case   %6d' % metrics[3][0] + '        %6d' % metrics[3][1])
    print('Precision  %6.2f' % metrics[0][0] + '        %6.2f' % metrics[0][1])
    print('Recall     %6.2f' % metrics[1][0] + '        %6.2f' % metrics[1][1])
    print('F1         %6.2f' % metrics[2][0] + '        %6.2f' % metrics[2][1])


    
print_metrics(y_test, scores)<jupyter_output>Confusion matrix
                 Score positive    Score negative
Actual positive       182                30
Actual negative        39                49

Accuracy  0.77
 
           Positive      Negative
Num case      212            88
Precision    0.82          0.62
Recall       0.86          0.56
F1           0.84          0.59<jupyter_text>Examine these results:1. The confusion matrix shows the following characteristics; a) most of the positive cases are correctly classified, 182 vs. 30, however, b) may negative cases are are scored incorrectly, with only 49 correct, vs. 39 incorrect.   2. The overall accuracy is 0.77. However as just observed this is **extremely misleading!**. In fact the negative cases are poorly classified, and it is these bad credit customers the bank cares most about. This is not an unusual case. Accuracy figures should always be regarded with healthy skepticism.3. The class imbalance is confirmed. Of the 300 test cases 212 are positive and 88 are negative. 4. The precision, recall and F1 all show that positive cases are classified reasonably well, but the negative cases are not. As already mentioned, it is these negative cases that are of greatest importance to the bank. Finally, the code below **computes and displays the ROC curve and AUC**.  The `roc_curve` and `auc` functions are used to compute these values.<jupyter_code>def plot_auc(labels, probs):
    ## Compute the false positive rate, true positive rate
    ## and threshold along with the AUC
    fpr, tpr, threshold = sklm.roc_curve(labels, probs[:,1])
    auc = sklm.auc(fpr, tpr)
    
    ## Plot the result
    plt.title('Receiver Operating Characteristic')
    plt.plot(fpr, tpr, color = 'orange', label = 'AUC = %0.2f' % auc)
    plt.legend(loc = 'lower right')
    plt.plot([0, 1], [0, 1],'r--')
    plt.xlim([0, 1])
    plt.ylim([0, 1])
    plt.ylabel('True Positive Rate')
    plt.xlabel('False Positive Rate')
    plt.show()
    
plot_auc(y_test, probabilities)<jupyter_output><empty_output><jupyter_text>The ROC curve is above the diagonal red-dotted line and the AUC is 0.77. But, given the class imbalance of two positive cases for each negative case how good is this? One point of comparison is **a naive 'classifier' that sets all cases to positive**. The code in the cell below contains such a classifier. This algorithm is not really a classifier at all. This 'classifier' is hard coded. The **ROC curve and AUC are then computed and displayed**:<jupyter_code>probs_positive = np.concatenate((np.ones((probabilities.shape[0], 1)), 
                                 np.zeros((probabilities.shape[0], 1))),
                                 axis = 1)
scores_positive = score_model(probs_positive, 0.5)
print_metrics(y_test, scores_positive)  
plot_auc(y_test, probs_positive)<jupyter_output>Confusion matrix
                 Score positive    Score negative
Actual positive       212                 0
Actual negative        88                 0

Accuracy  0.71
 
           Positive      Negative
Num case      212            88
Precision    0.71          0.00
Recall       1.00          0.00
F1           0.83          0.00<jupyter_text>Notice, the accuracy from this 'classifier' is 0.71. This reflects the class imbalance. The ROC curve is directly along the diagonal which gives an AUC of 0.5. The logistic regression classifier is definitely better than this!  Compute a weighted modelRecall that a falsely classifying a bad credit risk customer as good costs the bank five times more than classifying a good credit risk customer as bad.  Given this situation, the results of the first model are not that good. There are 2 reasons for this:1. The class imbalance in the label has biased the training of the model. As you observed from the accuracy of the naive 'classifier' is not that different from the logistic regression model. 2. Nothing has been done to weight the results toward correctly classifying the bad credit risk customers at the expense of the good credit risk customers.One approach to these problems is to weight the classes when computing the logistic regression model. The code in the cell below adds a `class_weight` argument to the call to the `LogisticRegression` function. In this case weights are chosen as $0.45, 0.55$  but you can also give another combination.<jupyter_code>logistic_mod = linear_model.LogisticRegression(class_weight = {0:0.45, 1:0.55},max_iter=1000) 
logistic_mod.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>Next, compute and display the class probabilities for each case.<jupyter_code>probabilities = logistic_mod.predict_proba(X_test)
print(probabilities[:15,:])<jupyter_output>[[0.80690788 0.19309212]
 [0.35732481 0.64267519]
 [0.92568008 0.07431992]
 [0.28749401 0.71250599]
 [0.51373263 0.48626737]
 [0.96543136 0.03456864]
 [0.93207935 0.06792065]
 [0.58155064 0.41844936]
 [0.8508493  0.1491507 ]
 [0.89291227 0.10708773]
 [0.52995434 0.47004566]
 [0.3668864  0.6331136 ]
 [0.48046073 0.51953927]
 [0.66189515 0.33810485]
 [0.31931477 0.68068523]]<jupyter_text>By eyeball, the above probabilities are not terribly different from the unweighted model.  **To find if there is any significant difference with the unweighted model, compute the scores and the metrics**:<jupyter_code>scores = score_model(probabilities, 0.5)
print_metrics(y_test, scores)  
plot_auc(y_test, probabilities)<jupyter_output>Confusion matrix
                 Score positive    Score negative
Actual positive       177                35
Actual negative        34                54

Accuracy  0.77
 
           Positive      Negative
Num case      212            88
Precision    0.84          0.61
Recall       0.83          0.61
F1           0.84          0.61<jupyter_text>The accuracy is slightly changed with respect to the unweighted model. The change could be more had we give more weights to one of the class than what we did here. The precision, recall and F1 are slightly better for the negative cases. Reweighting the labels has moved the results in the desired direction, at least a bit.Notice also, the ROC curve and AUC are essentially unchanged. The trade-off between true positive and false positive is similar to the unweighted model.   Find a better thresholdThere is another way to tip the model scoring toward correctly identifying the bad credit cases. **The scoring threshold can be adjusted.**  Until now, the scores have been computed from the probabilities using a threshold of 0.5.  However, there is no reason to think this is the correct choice.  Recall that the score is  determined by setting the threshold along the sigmoidal or logistic function.  It is possible to favor either positive or negative cases by changing the threshold along this curve. The code below contains a function for **scoring and evaluating the model for a given threshold value**.  The `for` loop iterates over the list of five candidate threshold values.  Examine how changing the threshold value changes the scoring for the model.<jupyter_code>def test_threshold(probs, labels, threshold):
    scores = score_model(probs, threshold)
    print('')
    print('For threshold = ' + str(threshold))
    print_metrics(labels, scores)

thresholds = [0.60, 0.55, 0.50, 0.45, 0.40, 0.35, 0.3, 0.25]
for t in thresholds:
    test_threshold(probabilities, y_test, t)<jupyter_output>For threshold = 0.6
                 Confusion matrix
                 Score positive    Score negative
Actual positive       188                24
Actual negative        49                39

Accuracy  0.76
 
           Positive      Negative
Num case      212            88
Precision    0.79          0.62
Recall       0.89          0.44
F1           0.84          0.52

For threshold = 0.55
                 Confusion matrix
                 Score positive    Score negative
Actual positive       182                30
Actual negative        39                49

Accuracy  0.77
 
           Positive      Negative
Num case      212            88
Precision    0.82          0.62
Recall       0.86          0.56
F1           0.84          0.59

For threshold = 0.5
                 Confusion matrix
                 Score positive    Score negative
Actual positive       177                35
Actual negative        34                54

Accuracy  0.77
 
           Positive      Negative
Num case[...]
{'fraction_non_alphanumeric': 0.08065483540909572, 'fraction_numerical': 0.06908231309317997, 'mean_word_length': 3.7350484463748748, 'pattern_counts': {'":': 0, '<': 69, '<?xml version=': 0, '>': 70, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '28597', 'license': 'MIT', 'n_tokens_mistral': 9352, 'n_tokens_neox': 7927, 'n_words': 3567, 'path': 'documents/classification.ipynb', 'repo_name': 'liawbeile/liawbeile.github.io', 'size': '28343'}