<jupyter_start><jupyter_text>**Pima Indians Diabetes Database ** Predict the onset of diabetes based on diagnostic measures   This dataset is originally from the National Institute of Diabetes and Digestive and Kidney Diseases. The objective of the dataset is to diagnostically predict whether or not a patient has diabetes, based on certain diagnostic measurements included in the dataset. Several constraints were placed on the selection of these instances from a larger database. In particular, all patients here are females at least 21 years old of Pima Indian heritage.  [www.kaggle.com/uciml/pima-indians-diabetes-database/](https://www.kaggle.com/uciml/pima-indians-diabetes-database/) **Model Evaluation Metrics ** Regression problems: Mean Absolute Error, Mean Squared Error, Root Mean Squared Error  Classification problems: Classification accuracy<jupyter_code>!curl --header "Host: storage.googleapis.com" --header "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36" --header "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3" --header "Accept-Language: en-US,en;q=0.9" --header "Referer: https://www.kaggle.com/" "https://storage.googleapis.com/kaggle-data-sets/228/482/bundle/archive.zip?GoogleAccessId=<EMAIL>&Expires=1574501534&Signature=RAtrIUq7OVFDgqdXP8v4MRSDx0Z76FgXs9I7yK1dhnabT2O86HC4%2FgdLolinCuvr7ifytzNDIey3UdZtVckT2YHx4eag%2F%2FXXW6%2FeJXm6WofYQgKFrFqNCLI74jkAlezl%2FCTue%2But%2BIbMchSsC8RPBpGIC7HwLgcQWIbmZLqG1O601dGon1qNJBQEHbN9r8TxZGD6DDzWpgne4Q9qoC71hTdP5MTD5EvuIc15D%2BKRoWFUozJWk%2FKZTzSpMGUiOx5ie2fpH%2FCCqC19UWsmojElJIahsk1wh8onKZcCfcptln06afUu8sJ6RrmdyEcHrR%2Fc040KUSCVflFvgNUj2i503w%3D%3D&response-content-disposition=attachment%3B+filename%3Dpima-indians-diabetes-database.zip" -o "pima-indians-diabetes-database.zip" -L
!ls
#unzipping file
!unzip pima-indians-diabetes-database<jupyter_output>Archive:  pima-indians-diabetes-database.zip
  inflating: diabetes.csv<jupyter_text>** Loading Required Libraries **<jupyter_code>import pandas as pd
import warnings
warnings.filterwarnings('ignore')
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.model_selection import cross_validate
# Let's implement simple classifiers
from sklearn.naive_bayes import GaussianNB
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
from sklearn import metrics
from sklearn.preprocessing import binarize
%matplotlib inline
import matplotlib.pyplot as plt
from sklearn import metrics<jupyter_output><empty_output><jupyter_text>** 1. Reading CSV File **<jupyter_code># read the data into a pandas DataFrame
#https://www.kaggle.com/uciml/pima-indians-diabetes-database/
#col_names = ['pregnant', 'glucose', 'bp', 'skin', 'insulin', 'bmi', 'pedigree', 'age', 'label']
pima = pd.read_csv("/content/diabetes.csv")
# print the first 5 rows of data
pima.head()
print(pima.shape)<jupyter_output>(768, 9)<jupyter_text>**2. Data PreProcessing**<jupyter_code>pima.columns

pima.describe()

#Checking Null values are present or not 
pima.isnull().sum().max()
print(pima.info())

# The classes are heavily skewed we need to solve this issue later.
print('paitents does not have diabetes', round(pima['Outcome'].value_counts()[0]/len(pima) * 100,2), '% of the dataset')
print('patient who has diabetes', round(pima['Outcome'].value_counts()[1]/len(pima) * 100,2), '% of the dataset')<jupyter_output>paitents does not have diabetes 65.1 % of the dataset
patient who has diabetes 34.9 % of the dataset<jupyter_text>Note: Notice how imbalanced is our original dataset! Most of the patient does not have diabetes. If we use this dataframe as the base for our predictive models and analysis we might get a lot of errors and our algorithms will probably overfit since it will "assume" that most patients does not have diabetes. But we don't want our model to assume, we want our model to detect patterns that give signs of patients who has diabetes!<jupyter_code>colors = ["#0101DF", "#DF0101"]
sns.countplot('Outcome', data=pima, palette=colors)
plt.title('Class Distributions \n (0: patient does not have Diabetes || 1: patient has diabetes)', fontsize=14)
plt.show()<jupyter_output><empty_output><jupyter_text>Distributions: By seeing the distributions we can have an idea how skewed are these features, we can also see further distributions of the other features. There are techniques that can help the distributions be less skewed which will be implemented in this notebook in the future.  **3 Splitting Data Into Train and Test**<jupyter_code>x=pima.drop('Outcome', axis=1)
y=pima['Outcome']
#Splitting data into train test and cross validation
x_train,x_test,y_train,y_test =train_test_split(x,y,test_size =0.2,random_state = 42)
print(x_train.shape)
print(x_test.shape)
print(y_train.shape)
print(y_test.shape)<jupyter_output>(614, 8)
(154, 8)
(614,)
(154,)<jupyter_text>**4. Building Machine Learning Algorithms **<jupyter_code>classifiers = {
    "LogisiticRegression": LogisticRegression(),
    "KNearest": KNeighborsClassifier(),
    "DecisionTreeClassifier": DecisionTreeClassifier(),
    "Naive Bayes":GaussianNB()
}
for key, classifier in classifiers.items():
    classifier.fit(x_train, y_train)
    training_score = cross_val_score(classifier, x_train, y_train, cv=5)
    print("Classifiers: ", classifier.__class__.__name__, "Has a training score of", round(training_score.mean(), 2) * 100, "% accuracy score")<jupyter_output>Classifiers:  LogisticRegression Has a training score of 76.0 % accuracy score
Classifiers:  KNeighborsClassifier Has a training score of 72.0 % accuracy score
Classifiers:  DecisionTreeClassifier Has a training score of 73.0 % accuracy score
Classifiers:  GaussianNB Has a training score of 75.0 % accuracy score<jupyter_text>All my machine learning Models are Missclassifying here<jupyter_code># train a GaussianNB model on the training set

# instantiate model
logreg = LogisticRegression()

# fit model
logreg.fit(x_train, y_train)
y_pred_class = logreg.predict(x_test)

# calculate accuracy
print(metrics.accuracy_score(y_test, y_pred_class))
# examine the class distribution of the testing set (using a Pandas Series method)
y_test.value_counts()
# calculate the percentage of ones
# because y_test only contains ones and zeros, we can simply calculate the mean = percentage of ones
y_test.mean()
# calculate the percentage of zeros
1 - y_test.mean()
# calculate null accuracy in a single line of code
# only for binary classification problems coded as 0/1
max(y_test.mean(), 1 - y_test.mean())<jupyter_output><empty_output><jupyter_text>This means that a dumb model that always predicts 0 would be right 64% of the timeThis shows how classification accuracy is not that good as it's close to a dumb modelIt's a good way to know the minimum we should achieve with our models<jupyter_code># calculate null accuracy (for multi-class classification problems)
y_test.value_counts().head(1) / len(y_test)
# print the first 25 true and predicted responses
print('True:', y_test.values[0:25])
print('False:', y_pred_class[0:25])<jupyter_output>True: [0 0 0 0 0 0 0 0 0 0 1 0 1 0 0 1 0 0 1 1 0 0 0 0 0]
False: [0 0 0 0 0 0 0 1 1 1 0 1 0 0 0 0 0 0 1 0 0 0 1 0 1]<jupyter_text>Conclusion:Classification accuracy is the easiest classification metric to understand But, it does not tell you the underlying distribution of response valuesWe examine by calculating the null accuracy and, it does not tell you what "types" of errors your classifier is making ** 5. Confusion matrix **Table that describes the performance of a classification model<jupyter_code># IMPORTANT: first argument is true values, second argument is predicted values
# this produces a 2x2 numpy array (matrix)
confusion = metrics.confusion_matrix(y_test, y_pred_class)
print(confusion)<jupyter_output>[[83 16]
 [21 34]]<jupyter_text>Basic terminologyTrue Positives (TP): we correctly predicted that patient has diabetes.34True Negatives (TN): we correctly predicted that patient does not have diabetes.83False Positives (FP): we incorrectly predicted that patient has diabetes. (a "Type I error")16Falsely predict positiveType I errorFalse Negatives (FN): we incorrectly predicted that patient does not have diabetes. (a "Type II error")21Falsely predict negativeType II error 0: patient does not have Diabetes 1: patient who has Diabetes ** 6. Metrics Computed from a Confusion Matrix **Classification Accuracy: Overall, how often is the classifier correct?<jupyter_code>TP = confusion[1, 1]
TN = confusion[0, 0]
FP = confusion[0, 1]
FN = confusion[1, 0]

# use float to perform true division, not integer division
print((TP + TN) / float(TP + TN + FP + FN))
print(metrics.accuracy_score(y_test, y_pred_class))<jupyter_output>0.7597402597402597
0.7597402597402597<jupyter_text>**7. Classification Error: Overall, how often is the classifier incorrect? **Also known as "Misclassification Rate"<jupyter_code>classification_error = (FP + FN) / float(TP + TN + FP + FN)

print(classification_error)
print(1 - metrics.accuracy_score(y_test, y_pred_class))<jupyter_output>0.24025974025974026
0.24025974025974028<jupyter_text>**8. Sensitivity: When the actual value is positive, how often is the prediction correct? **Something we want to maximizeHow "sensitive" is the classifier to detecting positive instances?Also known as "True Positive Rate" or "Recall"TP / all positiveall positive = TP + FN<jupyter_code>sensitivity = TP / float(FN + TP)

print(sensitivity)
print(metrics.recall_score(y_test, y_pred_class))<jupyter_output>0.6181818181818182
0.6181818181818182<jupyter_text>**9. Specificity: When the actual value is negative, how often is the prediction correct? **Something we want to maximizeHow "specific" (or "selective") is the classifier in predicting positive instances?TN / all negativeall negative = TN + FP<jupyter_code>specificity = TN / (TN + FP)

print(specificity)<jupyter_output>0.8383838383838383<jupyter_text>**10. False Positive Rate: When the actual value is negative, how often is the prediction incorrect? **<jupyter_code>false_positive_rate = FP / float(TN + FP)

print(false_positive_rate)
print(1 - specificity)<jupyter_output>0.16161616161616163
0.16161616161616166<jupyter_text>**11. Precision: When a positive value is predicted, how often is the prediction correct? **How "precise" is the classifier when predicting positive instances?<jupyter_code>precision = TP / float(TP + FP)

print(precision)
print(metrics.precision_score(y_test, y_pred_class))<jupyter_output>0.68
0.68<jupyter_text>Many other metrics can be computed: F1 score, Matthews correlation coefficient, etc.**Conclusion: **Confusion matrix gives you a more complete picture of how your classifier is performingAlso allows you to compute various classification metrics, and these metrics can guide your model selectionWhich metrics should you focus on?Choice of metric depends on your business objectiveIdentify if FP or FN is more important to reduceChoose metric with relevant variable (FP or FN in the equation) ** 12. Adjusting the classification threshold **<jupyter_code># print the first 10 predicted responses
# 1D array (vector) of binary values (0, 1)
logreg.predict(x_test)[0:10]
# print the first 10 predicted probabilities of class membership
logreg.predict_proba(x_test)[0:10]<jupyter_output><empty_output><jupyter_text>** Row: observation **Each row, numbers sum to 1** Column: class **2 response classes there 2 columnscolumn 0: predicted probability that each observation is a member of class 0column 1: predicted probability that each observation is a member of class 1Importance of predicted probabilitiesWe can rank observations by probability of FraudPrioritize contacting those with a higher probabilitypredict_proba processPredicts the probabilitiesChoose the class with the highest probabilityThere is a 0.5 classification thresholdClass 1 is predicted if probability > 0.5Class 0 is predicted if probability < 0.5<jupyter_code># print the first 10 predicted probabilities for class 1
logreg.predict_proba(x_test)[0:10, 1]
# store the predicted probabilities for class 1
y_pred_prob = logreg.predict_proba(x_test)[:, 1]
# allow plots to appear in the notebook

# adjust the font size 
plt.rcParams['font.size'] = 12
# histogram of predicted probabilities

# 8 bins
plt.hist(y_pred_prob, bins=8)

# x-axis limit from 0 to 1
plt.xlim(0,1)
plt.title('Histogram of predicted probabilities')
plt.xlabel('Predicted probability of diabetes')
plt.ylabel('Frequency')<jupyter_output><empty_output><jupyter_text>We can see from the third barAbout 55% of observations have probability from 0 to 0.1very Small number of observations with probability > 0.5This is below the threshold of 0.5Most would be predicted "Transaction is not Fraud" in this caseSolutionDecrease the threshold for predicting FraudIncrease the sensitivity of the classifierThis would increase the number of TPMore sensitive to positive instancesExample of metal detectorThreshold set to set off alarm for large object but not tiny objectsYES: metal, NO: no metalWe lower the threshold amount of metal to set it offIt is now more sensitive to metalIt will then predict YES more often ** CALIBRATION METHOD FOR TYPE CD ** For Instance, if a Problem has a proportion of 65.1% of class '0' and 34.9% of class '1', a model predicting 76% of class '0' and 24% of class '1' is uncalibrated, although it could have a low error (ranging from 20% to 25%) . This error distribution can be clearly seen on a confusion Matrix.So, class calibration is defined as the degree of approximation of the true or empirical class distribution with the estimated class distribution. The standard way to calibrate a model in this way is by changing the threshold that determines when the model predicts '0' or '1', making this threshold stricter with class '0' and milder with class '1' to balance the proportion. Note that this type of calibration, in principle, might produce more error. In fact, it is usually the case when one wants to obtain a useful model for problems with very imbalanced class distribution, i.e. the minority class has very few examples. In the case of discrete classification, the best way to know whether a model is uncalibrated according to the number of instance per class (type CD) is to analyse the confusion matrix. The confusion matrix is a visual way of showing the recount of cases of the predicted classes and their actual values. Each column of the matrix represents the instances in a predicted class, while each row represents the instances in an actual class. One benefit of a confusion matrix is that it is easy to see if the system is confusing two classes (i.e. commonly mislabelling one as another).<jupyter_code># predict Fraud if the predicted probability is greater than 0.4
# it will return 1 for all values above 0.4 and 0 otherwise
# results are 2D so we slice out the first column
y_pred_prob1=y_pred_prob.reshape(1, -1)
y_pred_class = binarize(y_pred_prob1, 0.4)[0] 
y_pred_class
# previous confusion matrix (default threshold of 0.5)
print(confusion)
print(metrics.confusion_matrix(y_test, y_pred_class))<jupyter_output>[[73 26]
 [15 40]]<jupyter_text>The row totals are the sameThe rows represent actual response values99 values top row55 values bottom rowObservations from the left column moving to the right column because we will have more TN and TP **13. Which metrics should you focus on?**Choice of metric depends on your business objectiveIdentify if FP or FN is more important to reduceChoose metric with relevant variable (FP or FN in the equation)<jupyter_code># sensitivity has increased (used to be 0.24)
sensitivity = TP / float(FN + TP)

print ( 42/ float(13 + 38))
print(metrics.recall_score(y_test, y_pred_class))

# specificity has decreased (used to be 0.91)
specificity = TN / (TN + FP)

print(70 / float(70 + 29))<jupyter_output>0.7070707070707071<jupyter_text>False Positive Rate: When the actual value is negative, how often is the prediction incorrect?<jupyter_code>false_positive_rate = FP / float(TN + FP)

print(false_positive_rate)
print(1 - specificity)<jupyter_output>0.16161616161616163
0.16161616161616166<jupyter_text>Precision: When a positive value is predicted, how often is the prediction correct?How "precise" is the classifier when predicting positive instances?<jupyter_code>precision = TP / (TP + FP)

print(precision)
print(metrics.precision_score(y_test, y_pred_class))<jupyter_output>0.68
0.6060606060606061<jupyter_text>**Conclusion: **Threshold of 0.5 is used by default (for binary problems) to convert predicted probabilities into class predictionsThreshold can be adjusted to increase sensitivity or specificitySensitivity and specificity have an inverse relationshipIncreasing one would always decrease the otherAdjusting the threshold should be one of the last step you do in the model-building processThe most important steps areBuilding the modelsSelecting the best model **14. Receiver Operating Characteristic (ROC) Curves**Question: Wouldn't it be nice if we could see how sensitivity and specificity are affected by various thresholds, without actually changing the threshold?Answer: Plot the ROC curve.Receiver Operating Characteristic (ROC)<jupyter_code># IMPORTANT: first argument is true values, second argument is predicted probabilities

# we pass y_test and y_pred_prob
# we do not use y_pred_class, because it will give incorrect results without generating an error
# roc_curve returns 3 objects fpr, tpr, thresholds
# fpr: false positive rate
# tpr: true positive rate
fpr, tpr, thresholds = metrics.roc_curve(y_test, y_pred_prob)

plt.plot(fpr, tpr)
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.0])
plt.rcParams['font.size'] = 12
plt.title('ROC curve for diabetes classifier')
plt.xlabel('False Positive Rate (1 - Specificity)')
plt.ylabel('True Positive Rate (Sensitivity)')
plt.grid(True)<jupyter_output><empty_output><jupyter_text>ROC curve can help you to choose a threshold that balances sensitivity and specificity in a way that makes sense for your particular contextYou can't actually see the thresholds used to generate the curve on the ROC curve itself<jupyter_code># define a function that accepts a threshold and prints sensitivity and specificity
def evaluate_threshold(threshold):
    print('Sensitivity:', tpr[thresholds > threshold][-1])
    print('Specificity:', 1 - fpr[thresholds > threshold][-1])
evaluate_threshold(0.5)

evaluate_threshold(0.4)

evaluate_threshold(0.3)<jupyter_output>Sensitivity: 0.8
Specificity: 0.6060606060606061<jupyter_text>** 15. AUC **AUC is the percentage of the ROC plot that is underneath the curve:<jupyter_code>#IMPORTANT: first argument is true values, second argument is predicted probabilities
print(metrics.roc_auc_score(y_test, y_pred_prob))<jupyter_output>0.8192837465564738<jupyter_text>AUC is useful as a single number summary of classifier performanceHigher value = better classifierIf you randomly chose one positive and one negative observation, AUC represents the likelihood that your classifier will assign a higher predicted probability to the positive observationAUC is useful even when there is high class imbalance (unlike classification accuracy)Fraud caseNull accuracy almost 99%AUC is useful here<jupyter_code># calculate cross-validated AUC
from sklearn.model_selection import cross_validate

#from sklearn.model_selection.cross_validate import cross_val_score
cv=cross_validate(logreg, x_train, y_train, cv=10, scoring='roc_auc')
cv
cv['test_score'].mean()<jupyter_output><empty_output><jupyter_text>Use both of these whenever possible** Confusion matrix advantages:**Allows you to calculate a variety of metricsUseful for multi-class problems (more than two response classes)** ROC/AUC advantages:**Does not require you to set a classification thresholdStill useful when there is high class imbalance<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07659637512963603, 'fraction_numerical': 0.03619931848486345, 'mean_word_length': 6.180851063829787, 'pattern_counts': {'":': 4, '<': 92, '<?xml version=': 0, '>': 95, 'https://': 4, 'lorem ipsum': 0, 'www.': 4, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '27', 'dir': 'jupyter-structured-clean-dedup', 'id': '306152', 'license': 'MIT', 'n_tokens_mistral': 6599, 'n_tokens_neox': 5890, 'n_words': 2294, 'path': 'pima logistic regression blog.ipynb', 'repo_name': 'saikrishna6680/Calibration-of-Models', 'size': '20288'}