<jupyter_start><jupyter_text>___ COMMON CLASSIFICATION EVALUATION METRICS___How accurate is a clasification model?Is the model reliable?These two questions are easily  answered by evaluating how well a model performs when subjected to unseen observations.This post ilustrates some of the best the ways models models can be evaluated.What you will learn from this post:1. Jaccard index.1. Confussion Matrix1. F-1 Score1. Log lossBack to business!First I will fit a simple model and use it to illustrate these methods are applied in model performance evaluation.<jupyter_code># quick model fit
import numpy as np
import warnings
import pandas
warnings.filterwarnings("ignore")#not recomended but i have included this for my own convenience.
from sklearn.datasets import load_breast_cancer
data = load_breast_cancer()
X = pandas.DataFrame(data = data.data,columns=data.feature_names)
y = data.target
#train test split
from sklearn import model_selection
np.random.seed(2) #to enable you replicate the same thing i am doing here.
X_train, X_test, y_train, y_test = model_selection.train_test_split(X,y,test_size=0.30)
# I will use logistic reg
from sklearn.linear_model import LogisticRegression
reg = LogisticRegression()
reg.fit(X_train,y_train)
preds = reg.predict(X_test)
predsprob = reg.predict_proba(X_test)<jupyter_output><empty_output><jupyter_text>Jaccard Index.Given predicted values as $\hat y$ and actual values as $y$,the jaccard index can be defied as :$$j(y,\hat y) = \frac{|y\cap \hat y|}{|y\cup \hat y|}$$$$= \frac{|y\cap \hat y|}{|y|+|\hat y|-|y\cap \hat y|}$$so lets say you have the folowing set of predicted and actual values.$$predicted:[0,0,0,0,0,1,1,1,1,1]$$$$actual: [1,1,0,0,0,1,1,1,1,1]$$the jaccard index will be :$$j(y,\hat y) = \frac{8}{10+10-8}=0.66$$The idea behind this index is that higher the similarity of these two groups the higher the index. Applying this to the model above.<jupyter_code>from sklearn.metrics import jaccard_similarity_score
j_index = jaccard_similarity_score(y_true=y_test,y_pred=preds)
round(j_index,2)<jupyter_output><empty_output><jupyter_text>Confussion matrixConfusion matrix is used to describe the performance of a classification model on a set of test data for which true values are known.![conf.png](attachment:image.png)From the confusion matrix the following information can be extracted :1. **True positive(TP).**: This shows that a model **correctly**predicted **Positive** cases as **Positive**.eg an illness is diagnosed as present and truly is present.1. **False positive(FP):** This shows that a model **incorrectly**predicted **Negative** cases as **Positive**.eg an illness is diagnosed as present and but is absent.(Type I error)1. **False Negative:(FN)** This shows that a **incorrectly** model predicted **Positive** cases as **Negative**.eg an illness is diagnosed as absent and but is present.(Type II error)1. **True Negative(TN)**This shows that a model **correctly**predicted **Negative** cases as **Positive**.eg an illness is diagnosed as absent and truly is absent. Applying this to the model above<jupyter_code>from sklearn.metrics import confusion_matrix
print(confusion_matrix(y_test,preds,labels=[1,0]))
import seaborn as sns
import matplotlib.pyplot as plt
sns.heatmap(confusion_matrix(y_test,preds),annot=True,lw =2,cbar=False)
plt.ylabel("True Values")
plt.xlabel("Predicted Values")
plt.title("CONFUSSION MATRIX VISUALIZATION")
plt.show()<jupyter_output><empty_output><jupyter_text>In this case for the breast cancer data,the model corectly predicts 61 cases as benign and 100 cases as maligant.In contrast it mispredicts a total of 9 cases.  F1-Score.This comes from the cinfusion matrix.Based on the above confussion matrix above,we can calculate the **precision** and the **recal socres.****Precision score:** this is the measure of the accuracy ,provided that a class label has been predicted.Simply put,it answers the following question,**of all the classes ,how many were correctly predicted?** The answer to this question should be as high as posible.It can be calculated as follows:$$Precision = \frac{TP}{TP+FP}$$**Recal score(Sensitivity)**:This is the true positive rate that is if it predicts positive then how often does this take place?$$TP=\frac{TP}{TP+FN}$$The F1 score is calculated based on the precision and recal of each class.It is the weighted average of the Precision and the recal scores.The F1 score reaches its perfect value at one and worst at 0.It is avery good way to show that a classifies has a good recal and precision values.We can calculate it using this formula:$$F1Score = 2(\frac{Precision\times Recal}{Precision +  Recal})$$ Applying to the model above.<jupyter_code>from sklearn.metrics import f1_score
f1_score(y_test,preds)<jupyter_output><empty_output><jupyter_text>F1 score can be calculated for all classes so that an average of the realized scores can be used as shown in the classification report bellow.<jupyter_code>from sklearn.metrics import classification_report
print(classification_report(y_test,preds))<jupyter_output>precision    recall  f1-score   support

           0       0.91      0.93      0.92        67
           1       0.95      0.94      0.95       104

   micro avg       0.94      0.94      0.94       171
   macro avg       0.93      0.93      0.93       171
weighted avg       0.94      0.94      0.94       171<jupyter_text>Log loss.We can use the log loss in cases where the outcome of the classifier is a class probability and not a class label like in cases of logistic regression models.Log loss measures the performance of a model where the predicted outcome is a probability value between 0 and 1.In real life when predicting a probability of 0.101 when the true label should be 1 would result in a high log loss.Log loss can be calculated for each row in the data set using the **Log loss equation**. $$L= (y\times \log(\hat y)+(1-y)+\log(1-\hat y))$$The equation simply measures how far each predicted probability is from the actual label.An average of the log loss from all the rows gives the ideal value for the log loss.$$Lloss= -\frac{1}{n}\sum(y\times \log(\hat y)+(1-y)+\log(1-\hat y))$$A good and model should have a smaller log loss value. Applying in the above model.<jupyter_code>from sklearn.metrics import log_loss
log_loss(y_test,predsprob)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08827694728560188, 'fraction_numerical': 0.021243115656963022, 'mean_word_length': 4.934640522875817, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '589198', 'license': 'MIT', 'n_tokens_mistral': 1954, 'n_tokens_neox': 1824, 'n_words': 771, 'path': 'classification models (1).ipynb', 'repo_name': 'GeorgeOduor/classification-model-evaluation', 'size': '6355'}