<jupyter_start><jupyter_text>_Lambda School Data Science — Model Validation_  Classification Metrics & Imbalanced Classes Objectives- Classification Metrics: Accuracy, Precision, Recall, F1, ROC AUC- Confusion Matrix- Imbalanced Classes Pre-reads- [Precision and Recall](https://en.wikipedia.org/wiki/Precision_and_recall)- [Simple guide to confusion matrix terminology](https://www.dataschool.io/simple-guide-to-confusion-matrix-terminology/)- [ROC curves and Area Under the Curve explained](https://www.dataschool.io/roc-curves-and-auc-explained/)  Preliminary setup  Install [category_encoders](https://github.com/scikit-learn-contrib/categorical-encoding)- Google Colab: `pip install category_encoders`- Local, Anaconda: `conda install -c conda-forge category_encoders` Install  [mlxtend](http://rasbt.github.io/mlxtend/) to plot decision regions- Google Colab: Already installed- Local, Anaconda: `conda install -c conda-forge mlxtend` Get the Bank Marketing dataset- Download from [UCI](https://archive.ics.uci.edu/ml/datasets/Bank+Marketing)- Or run this cell:<jupyter_code>import sys
import numpy
numpy.set_printoptions(threshold=sys.maxsize)

# pip install category_encoders<jupyter_output><empty_output><jupyter_text>Classification Metrics & Confusion Matrix — with Bank Marketing dataset<jupyter_code>import requests, zipfile, io
r = requests.get('https://archive.ics.uci.edu/ml/machine-learning-databases/00222/bank-additional.zip')
z = zipfile.ZipFile(io.BytesIO(r.content))
z.extractall()

#    # bank client data:
#    1 - age (numeric)
#    2 - job : type of job (categorical: "admin.","unknown","unemployed","management","housemaid","entrepreneur","student",
#                                        "blue-collar","self-employed","retired","technician","services") 
#    3 - marital : marital status (categorical: "married","divorced","single"; note: "divorced" means divorced or widowed)
#    4 - education (categorical: "unknown","secondary","primary","tertiary")
#    5 - default: has credit in default? (binary: "yes","no")
#    6 - balance: average yearly balance, in euros (numeric) 
#    7 - housing: has housing loan? (binary: "yes","no")
#    8 - loan: has personal loan? (binary: "yes","no")
#    # related with the last contact of the current campaign:
#    9 - contact: contact communication type (categorical: "unknown","telephone","cellular") 
#   10 - day: last contact day of the month (numeric)
#   11 - month: last contact month of year (categorical: "jan", "feb", "mar", ..., "nov", "dec")
#   12 - duration: last contact duration, in seconds (numeric)
#    # other attributes:
#   13 - campaign: number of contacts performed during this campaign and for this client (numeric, includes last contact)
#   14 - pdays: number of days that passed by after the client was last contacted from a previous campaign (numeric, -1 means client was not previously contacted)
#   15 - previous: number of contacts performed before this campaign and for this client (numeric)
#   16 - poutcome: outcome of the previous marketing campaign (categorical: "unknown","other","failure","success")

#   Output variable (desired target):
#   17 - y - has the client subscribed a term deposit? (binary: "yes","no")
# This code comes from our previous notebook

# Imports
%matplotlib inline
import warnings
import category_encoders as ce
import matplotlib.pyplot as plt
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.exceptions import DataConversionWarning
from sklearn.preprocessing import StandardScaler
warnings.filterwarnings(action='ignore', category=DataConversionWarning)

# Load data
bank = pd.read_csv('bank-additional/bank-additional-full.csv', sep=';')
bank.columns.tolist()
bank = bank.drop(['pdays', 'day_of_week', 'month'], axis=1)
# Assign to X, y
X = bank.drop(columns='y')
y = bank['y'] == 'yes'

# Drop leaky feature
X = X.drop(columns='duration')

# Split Train, Test
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y)#only makes sense for categorical variables, using class labels, only for classification, not regression

# Make pipeline
pipeline = make_pipeline(
    ce.OneHotEncoder(use_cat_names=True), 
    StandardScaler(), 
    LogisticRegression(solver='lbfgs', max_iter=1000)
)<jupyter_output><empty_output><jupyter_text>scikit-learn documentation- [sklearn.linear_model.LogisticRegression.predict_proba](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.htmlsklearn.linear_model.LogisticRegression.predict_proba)- [sklearn.metrics.classification_report](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html)- [sklearn.metrics.confusion_matrix](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html)- [sklearn.model_selection.cross_val_predict](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_predict.html)<jupyter_code>from sklearn.model_selection import cross_val_predict

y_pred_proba = cross_val_predict(pipeline, X_train, y_train, cv=3, n_jobs=-1, 
                                 method='predict_proba')[:,1]
#we want to generate predictions for cross validation
#for each datapoint
#when we generated predictive probabilities<jupyter_output><empty_output><jupyter_text>Change the threshold and re-run this cell<jupyter_code>threshold = 0.5#% confidence people will sign up
#how many positive predictions we want
y_pred = y_pred_proba >= threshold
correct = y_pred == y_train

labels = pd.DataFrame({'Ground Truth': y_train, 
                       'Predicted Probability': y_pred_proba, 
                       'Discrete Prediction': y_pred, 
                       'Correct Prediction?': correct})

labels.head(20)
#we look at the first 20 predictions that we make
threshold = 0.5#% confidence people will sign up
y_pred = y_pred_proba >= threshold
correct = y_pred == y_train

labels = pd.DataFrame({'Ground Truth': y_train, 
                       'Predicted Probability': y_pred_proba, 
                       'Discrete Prediction': y_pred, 
                       'Correct Prediction?': correct})

labels.loc[8]<jupyter_output><empty_output><jupyter_text>Change the threshold and re-run this cell<jupyter_code>labels['Discrete Prediction'].value_counts(normalize=True).plot.barh(color='grey');
labels['Ground Truth'].value_counts(normalize=True).plot.barh(color='grey');
import seaborn as sns
sns.distplot(labels['Predicted Probability']);
from sklearn.metrics import classification_report, confusion_matrix

threshold = 0.1
y_pred = y_pred_proba >= threshold

print(classification_report(y_train, y_pred))

pd.DataFrame(confusion_matrix(y_train, y_pred), 
             columns=['Predicted Negative', 'Predicted Positive'], 
             index=['Actual Negative', 'Actual Positive'])
#make a dataframe out of the output
#where our models is confused
# model is not confused at actual negative, predicted negative
#actual positive and predicted positive are also correct
#the opposite diagonal is where the model is confused
#wasted time or money in actual negative and predicted positive - false positives
# missed opportunities in actual positive and predicted negative - false negatives

# 31% make the sale, we are reaching 65% of the people who are willing to sign up
#in entire dataset, we're calling 2418 people
#how these metrics are gamable, threshold high: very few positive predictions, so precision is good, but recall is terrible
threshold = 0.9
y_pred = y_pred_proba >= threshold

print(classification_report(y_train, y_pred))

pd.DataFrame(confusion_matrix(y_train, y_pred), 
             columns=['Predicted Negative', 'Predicted Positive'], 
             index=['Actual Negative', 'Actual Positive'])
threshold = 0.05
y_pred = y_pred_proba >= threshold

print(classification_report(y_train, y_pred))

pd.DataFrame(confusion_matrix(y_train, y_pred), 
             columns=['Predicted Negative', 'Predicted Positive'], 
             index=['Actual Negative', 'Actual Positive'])
#this could be a lot better than calling people at random
threshold = 0.005
y_pred = y_pred_proba >= threshold

print(classification_report(y_train, y_pred))

pd.DataFrame(confusion_matrix(y_train, y_pred), 
             columns=['Predicted Negative', 'Predicted Positive'], 
             index=['Actual Negative', 'Actual Positive'])
#calling everybody except 184 people
(140 + 32) / (140 + 32 + 617 + 515)
from sklearn.metrics import accuracy_score
accuracy_score(y_train, y_pred)
#TODO 

true_negative  = 23971 #actual negative, predicted negative
false_positive = 5267 #actual negative, predicted positive
false_negative = 1294 #actual positive, predicted negative
true_positive  = 2418 #actual positive, predicted positive

actual_negative = 28766 + 452
actual_positive =1294 + 2418

predicted_negative = 23971 + 1294
predicted_positive = 5267 + 2418

accuracy = (true_positive + true_negative) / (predicted_negative + predicted_positive)
#correct predictions / total predictions % , (true positives + true negatives)/(sum of all 4 numbers) 
precision = true_positive / predicted_positive
#precision and recall are based on relevance
#precision, based on how many selected, which are relevant?
#any search result on google - precision - search with eral scruggs / total search results
recall = true_positive / actual_positive
#recall, based on how many relevant, how many are selected?
#what was your recall for the most relevant things?
#of all the people who would sign up, how many did you actually select to call?
f1 = 2*precision * recall / (precision+recall)
print('Accuracy:',accuracy,'Precision:', precision, 'Recall:', recall,'F1', f1)<jupyter_output>Accuracy: 0.8008801213960546 Precision: 0.3146389069616135 Recall: 0.6514008620689655 F1 0.42432219005001315<jupyter_text>F1 Score"[The F1 score](https://en.wikipedia.org/wiki/F1_score) is the harmonic average of the precision and recall, where an F1 score reaches its best value at 1 (perfect precision and recall) and worst at 0."<jupyter_code>#(2 * precision * recall)/precision + recall
#if you can assign dollars that align with cost or benefit, if not a budget
#a real world problem that translates to a quantitative problem, then translate back to real world solution<jupyter_output><empty_output><jupyter_text>ROC AUC (Receiver Operating Characteristic, Area Under the Curve) Scikit-Learn docs- [User Guide: Receiver operating characteristic (ROC)](https://scikit-learn.org/stable/modules/model_evaluation.htmlreceiver-operating-characteristic-roc)- [sklearn.metrics.roc_curve](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_curve.html)- [sklearn.metrics.roc_auc_score](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html) More links- [ROC curves and Area Under the Curve explained](https://www.dataschool.io/roc-curves-and-auc-explained/)- [The philosophical argument for using ROC curves](https://lukeoakdenrayner.wordpress.com/2018/01/07/the-philosophical-argument-for-using-roc-curves/)[Wikipedia explains,](https://en.wikipedia.org/wiki/Receiver_operating_characteristic) "A receiver operating characteristic curve, or ROC curve, is a graphical plot that illustrates the diagnostic ability of a binary classifier system as its discrimination threshold is varied. The ROC curve is created by plotting the true positive rate (TPR) against the false positive rate (FPR) at various threshold settings."ROC AUC is the area under the ROC curve. [It can be interpreted](https://stats.stackexchange.com/questions/132777/what-does-auc-stand-for-and-what-is-it) as "the expectation that a uniformly drawn random positive is ranked before a uniformly drawn random negative." ROC AUC measures how well a classifier ranks predicted probabilities. It ranges from 0 to 1. A naive majority class baseline will have an ROC AUC score of 0.5.<jupyter_code>#true positive rate = precision
#false positive rate = recall
#roc curve represents the different kinds of tradeoffs you can make
from sklearn.metrics import roc_auc_score, roc_curve
fpr, tpr, thresholds = roc_curve(y_train, y_pred_proba)
plt.plot(fpr, tpr)
plt.title('ROC curve')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
print('Area under the Receiver Operating Characteristic curve:', 
      roc_auc_score(y_train, y_pred_proba))
    
# When threshold = 0.5
false_positives = 452
true_positives = 860
false_positive_rate = false_positives/actual_negative
true_positive_rate = true_positives/actual_positive
plt.scatter(false_positive_rate, true_positive_rate)

# When threshold = 0.1
false_positives = 5267
true_positives = 2418
false_positive_rate = false_positives/actual_negative
true_positive_rate = true_positives/actual_positive
plt.scatter(false_positive_rate, true_positive_rate);
#any part of the curve that is higher than a diagon
#the more the curve goes to the top left, the better the model is doing
# we want false positives to be low and true positives to be high
#putting the roc curve into a dataframe
pd.DataFrame(dict(fpr=fpr, tpr=tpr,thresholds=thresholds))<jupyter_output><empty_output><jupyter_text>Imbalanced Classes — with synthetic data  Fun demo!The next code cell does five things: 1. Generate dataWe use scikit-learn's [make_classification](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_classification.html) function to generate fake data for a binary classification problem, based on several parameters, including:- Number of samples- Weights, meaning "the proportions of samples assigned to each class."- Class separation: "Larger values spread out the clusters/classes and make the classification task easier."(We are generating fake data so it is easy to visualize.) 2. Split dataWe split the data three ways, into train, validation, and test sets. (For this toy example, it's not really necessary to do a three-way split. A two-way split, or even no split, would be ok. But I'm trying to demonstrate good habits, even in toy examples, to avoid confusion.) 3. Fit modelWe use scikit-learn to fit a [Logistic Regression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) on the training data.We use this model parameter:> **class_weight : _dict or ‘balanced’, default: None_**> Weights associated with classes in the form `{class_label: weight}`. If not given, all classes are supposed to have weight one.> The “balanced” mode uses the values of y to automatically adjust weights inversely proportional to class frequencies in the input data as `n_samples / (n_classes * np.bincount(y))`. 4. Evaluate modelWe use our Logistic Regression model, which was fit on the training data, to generate predictions for the validation data.Then we print [scikit-learn's Classification Report](https://scikit-learn.org/stable/modules/model_evaluation.htmlclassification-report), with many metrics, and also the accuracy score. We are comparing the correct labels to the Logistic Regression's predicted labels, for the validation set.  5. Visualize decision functionBased on these examples- https://imbalanced-learn.readthedocs.io/en/stable/auto_examples/combine/plot_comparison_combine.html- http://rasbt.github.io/mlxtend/user_guide/plotting/plot_decision_regions/example-1-decision-regions-in-2d<jupyter_code>from sklearn.model_selection import train_test_split

def train_validation_test_split(
    X, y, train_size=0.8, val_size=0.1, test_size=0.1, 
    random_state=None, shuffle=True):
        
    assert train_size + val_size + test_size == 1
    
    X_train_val, X_test, y_train_val, y_test = train_test_split(
        X, y, test_size=test_size, random_state=random_state, shuffle=shuffle)
    
    X_train, X_val, y_train, y_val = train_test_split(
        X_train_val, y_train_val, test_size=val_size/(train_size+val_size), 
        random_state=random_state, shuffle=shuffle)
    
    return X_train, X_val, X_test, y_train, y_val, y_test
%matplotlib inline
from IPython.display import display
import matplotlib.pyplot as plt
from sklearn.datasets import make_classification
from sklearn.metrics import accuracy_score, classification_report
from sklearn.linear_model import LogisticRegression
from mlxtend.plotting import plot_decision_regions


#1. Generate data

# Try re-running the cell with different values for these parameters
n_samples = 1000
weights = (0.95, 0.05)#we are generating data that is 95% one class, 5% another class
class_sep = 0.8

X, y = make_classification(n_samples=n_samples, n_features=2, n_informative=2, 
                           n_redundant=0, n_repeated=0, n_classes=2, 
                           n_clusters_per_class=1, weights=weights, 
                           class_sep=class_sep, random_state=0)


# 2. Split data

# Uses our custom train_validation_test_split function
X_train, X_val, X_test, y_train, y_val, y_test = train_validation_test_split(
    X, y, train_size=0.8, val_size=0.1, test_size=0.1, random_state=1)


# 3. Fit model

# Try re-running the cell with different values for this parameter
class_weight = None
#default parameter for Logistic Regression, we rebalance in next cell

model = LogisticRegression(solver='lbfgs', class_weight=class_weight)
model.fit(X_train, y_train)


# 4. Evaluate model

y_pred = model.predict(X_val)
print(classification_report(y_val, y_pred))
print('accuracy', accuracy_score(y_val, y_pred))
display(pd.DataFrame(
    confusion_matrix(y_val, y_pred), 
    columns=['Predicted Negative', 'Predicted Positive'], 
    index=['Actual Negative', 'Actual Positive']))


# 5. Visualize decision regions

plt.figure(figsize=(10, 6))
plot_decision_regions(X_val, y_val, model, legend=0);
#orange triangle is positive class
#4 triangles but I only predicted correctly 2 of them. 
#my recall is 2 out of 4 so 50% 
#easier to do classification when balananced 
%matplotlib inline
from IPython.display import display
import matplotlib.pyplot as plt
from sklearn.datasets import make_classification
from sklearn.metrics import accuracy_score, classification_report
from sklearn.linear_model import LogisticRegression
from mlxtend.plotting import plot_decision_regions


#1. Generate data

# Try re-running the cell with different values for these parameters
n_samples = 1000
weights = (0.95, 0.05)#we are generating data that is 95% one class, 5% another class
class_sep = 0.8

X, y = make_classification(n_samples=n_samples, n_features=2, n_informative=2, 
                           n_redundant=0, n_repeated=0, n_classes=2, 
                           n_clusters_per_class=1, weights=weights, 
                           class_sep=class_sep, random_state=0)


# 2. Split data

# Uses our custom train_validation_test_split function
X_train, X_val, X_test, y_train, y_val, y_test = train_validation_test_split(
    X, y, train_size=0.8, val_size=0.1, test_size=0.1, random_state=1)


# 3. Fit model

# Try re-running the cell with different values for this parameter
class_weight = 'balanaced'
#default parameter for Logistic Regression, we rebalance in next cell

model = LogisticRegression(solver='lbfgs', class_weight=class_weight)
model.fit(X_train, y_train)


# 4. Evaluate model

y_pred = model.predict(X_val)
print(classification_report(y_val, y_pred))
print('accuracy', accuracy_score(y_val, y_pred))
display(pd.DataFrame(
    confusion_matrix(y_val, y_pred), 
    columns=['Predicted Negative', 'Predicted Positive'], 
    index=['Actual Negative', 'Actual Positive']))


# 5. Visualize decision regions

plt.figure(figsize=(10, 6))
plot_decision_regions(X_val, y_val, model, legend=0);
#convenient way to predict the positive class more often
#https://scikit-learn.org/stable/modules/generated/sklearn.utils.class_weight.compute_class_weight.html
#automatically adjusts weights inversely proportional to class frequencies in the input data

#in the default case of class_weight = None, an error is weighted equally 
#we might want to priortize classifying positives 
#total error for 4 points, as same as the rest of the 96 points 
#like making new copies for orange triangles, so classes are weighted equally
#default is each obervation is weighted equally, balanced as 
%matplotlib inline
from IPython.display import display
import matplotlib.pyplot as plt
from sklearn.datasets import make_classification
from sklearn.metrics import accuracy_score, classification_report
from sklearn.linear_model import LogisticRegression
from mlxtend.plotting import plot_decision_regions


#1. Generate data

# Try re-running the cell with different values for these parameters
n_samples = 1000
weights = (0.95, 0.05)#we are generating data that is 95% one class, 5% another class
class_sep = 0.8

X, y = make_classification(n_samples=n_samples, n_features=2, n_informative=2, 
                           n_redundant=0, n_repeated=0, n_classes=2, 
                           n_clusters_per_class=1, weights=weights, 
                           class_sep=class_sep, random_state=0)


# 2. Split data

# Uses our custom train_validation_test_split function
X_train, X_val, X_test, y_train, y_val, y_test = train_validation_test_split(
    X, y, train_size=0.8, val_size=0.1, test_size=0.1, random_state=1)


# 3. Fit model

# Try re-running the cell with different values for this parameter
class_weight = 'balanced'#{0: 1, 1:10000}means make the positive 10,000 times more weight
#default parameter for Logistic Regression, we rebalance in next cell

model = LogisticRegression(solver='lbfgs', class_weight=class_weight)
model.fit(X_train, y_train)


# 4. Evaluate model

y_pred = model.predict(X_val)
print(classification_report(y_val, y_pred))
print('accuracy', accuracy_score(y_val, y_pred))
display(pd.DataFrame(
    confusion_matrix(y_val, y_pred), 
    columns=['Predicted Negative', 'Predicted Positive'], 
    index=['Actual Negative', 'Actual Positive']))


# 5. Visualize decision regions

plt.figure(figsize=(10, 6))
plot_decision_regions(X_val, y_val, model, legend=0)

y_pred_proba = model.predict_proba(X_val)[:,1]
print('ROC AUC', roc_auc_score(y_val, y_pred_proba))
# you can try scikit-learn imbalanced-learn ie. smoked
#calculating scores by hand is on sprint
#if you were given the 4 number in confusion matrix, to calculate accuracy and precision
#1. Generate data

# Try re-running the cell with different values for these parameters
n_samples = 1000
weights = (0.5, 0.5)#we are generating data that is 95% one class, 5% another class
class_sep = 0.8

X, y = make_classification(n_samples=n_samples, n_features=2, n_informative=2, 
                           n_redundant=0, n_repeated=0, n_classes=2, 
                           n_clusters_per_class=1, weights=weights, 
                           class_sep=class_sep, random_state=0)


# 2. Split data

# Uses our custom train_validation_test_split function
X_train, X_val, X_test, y_train, y_val, y_test = train_validation_test_split(
    X, y, train_size=0.8, val_size=0.1, test_size=0.1, random_state=1)


# 3. Fit model

# Try re-running the cell with different values for this parameter
class_weight = None #{0: 1, 1:10000}means make the positive 10,000 times more weight
#default parameter for Logistic Regression, we rebalance in next cell

model = LogisticRegression(solver='lbfgs', class_weight=class_weight)
model.fit(X_train, y_train)


# 4. Evaluate model

y_pred = model.predict(X_val)
print(classification_report(y_val, y_pred))
print('accuracy', accuracy_score(y_val, y_pred))
display(pd.DataFrame(
    confusion_matrix(y_val, y_pred), 
    columns=['Predicted Negative', 'Predicted Positive'], 
    index=['Actual Negative', 'Actual Positive']))


# 5. Visualize decision regions

plt.figure(figsize=(10, 6))
plot_decision_regions(X_val, y_val, model, legend=0)

y_pred_proba = model.predict_proba(X_val)[:,1]
print('ROC AUC', roc_auc_score(y_val, y_pred_proba))
#1. Generate data

# Try re-running the cell with different values for these parameters
n_samples = 1000
weights = (0.95, 0.05)#we are generating data that is 95% one class, 5% another class
class_sep = 0.8

X, y = make_classification(n_samples=n_samples, n_features=2, n_informative=2, 
                           n_redundant=0, n_repeated=0, n_classes=2, 
                           n_clusters_per_class=1, weights=weights, 
                           class_sep=class_sep, random_state=0)


# 2. Split data

# Uses our custom train_validation_test_split function
X_train, X_val, X_test, y_train, y_val, y_test = train_validation_test_split(
    X, y, train_size=0.8, val_size=0.1, test_size=0.1, random_state=1)


# 3. Fit model

# Try re-running the cell with different values for this parameter
class_weight = None #{0: 1, 1:10000}means make the positive 10,000 times more weight
#default parameter for Logistic Regression, we rebalance in next cell

model = LogisticRegression(solver='lbfgs', class_weight=class_weight)
model.fit(X_train, y_train)


# 4. Evaluate model

y_pred = model.predict(X_val)
print(classification_report(y_val, y_pred))
print('accuracy', accuracy_score(y_val, y_pred))
display(pd.DataFrame(
    confusion_matrix(y_val, y_pred), 
    columns=['Predicted Negative', 'Predicted Positive'], 
    index=['Actual Negative', 'Actual Positive']))


# 5. Visualize decision regions

plt.figure(figsize=(10, 6))
plot_decision_regions(X_val, y_val, model, legend=0)

y_pred_proba = model.predict_proba(X_val)[:,1]
print('ROC AUC', roc_auc_score(y_val, y_pred_proba))
#1. Generate data

# Try re-running the cell with different values for these parameters
n_samples = 1000
weights = (0.95, 0.05)#we are generating data that is 95% one class, 5% another class
class_sep = 0.8

X, y = make_classification(n_samples=n_samples, n_features=2, n_informative=2, 
                           n_redundant=0, n_repeated=0, n_classes=2, 
                           n_clusters_per_class=1, weights=weights, 
                           class_sep=class_sep, random_state=0)


# 2. Split data

# Uses our custom train_validation_test_split function
X_train, X_val, X_test, y_train, y_val, y_test = train_validation_test_split(
    X, y, train_size=0.8, val_size=0.1, test_size=0.1, random_state=1)


# 3. Fit model

# Try re-running the cell with different values for this parameter
class_weight = 'balanced' #{0: 1, 1:10000}means make the positive 10,000 times more weight
#default parameter for Logistic Regression, we rebalance in next cell

model = LogisticRegression(solver='lbfgs', class_weight=class_weight)
model.fit(X_train, y_train)


# 4. Evaluate model

y_pred = model.predict(X_val)
print(classification_report(y_val, y_pred))
print('accuracy', accuracy_score(y_val, y_pred))
display(pd.DataFrame(
    confusion_matrix(y_val, y_pred), 
    columns=['Predicted Negative', 'Predicted Positive'], 
    index=['Actual Negative', 'Actual Positive']))


# 5. Visualize decision regions

plt.figure(figsize=(10, 6))
plot_decision_regions(X_val, y_val, model, legend=0)

y_pred_proba = model.predict_proba(X_val)[:,1]
print('ROC AUC', roc_auc_score(y_val, y_pred_proba))
#1. Generate data

# Try re-running the cell with different values for these parameters
n_samples = 1000
weights = (0.95, 0.05)#we are generating data that is 95% one class, 5% another class
class_sep = 0.8

X, y = make_classification(n_samples=n_samples, n_features=2, n_informative=2, 
                           n_redundant=0, n_repeated=0, n_classes=2, 
                           n_clusters_per_class=1, weights=weights, 
                           class_sep=class_sep, random_state=0)


# 2. Split data

# Uses our custom train_validation_test_split function
X_train, X_val, X_test, y_train, y_val, y_test = train_validation_test_split(
    X, y, train_size=0.8, val_size=0.1, test_size=0.1, random_state=1)


# 3. Fit model

# Try re-running the cell with different values for this parameter
class_weight = {0: 1, 1: 2} #{0: 1, 1:10000}means make the positive 10,000 times more weight
#default parameter for Logistic Regression, we rebalance in next cell

model = LogisticRegression(solver='lbfgs', class_weight=class_weight)
model.fit(X_train, y_train)


# 4. Evaluate model

y_pred = model.predict(X_val)
print(classification_report(y_val, y_pred))
print('accuracy', accuracy_score(y_val, y_pred))
display(pd.DataFrame(
    confusion_matrix(y_val, y_pred), 
    columns=['Predicted Negative', 'Predicted Positive'], 
    index=['Actual Negative', 'Actual Positive']))


# 5. Visualize decision regions

plt.figure(figsize=(10, 6))
plot_decision_regions(X_val, y_val, model, legend=0)

y_pred_proba = model.predict_proba(X_val)[:,1]
print('ROC AUC', roc_auc_score(y_val, y_pred_proba))
#1. Generate data

# Try re-running the cell with different values for these parameters
n_samples = 1000
weights = (0.95, 0.05)#we are generating data that is 95% one class, 5% another class
class_sep = 0.8

X, y = make_classification(n_samples=n_samples, n_features=2, n_informative=2, 
                           n_redundant=0, n_repeated=0, n_classes=2, 
                           n_clusters_per_class=1, weights=weights, 
                           class_sep=class_sep, random_state=0)


# 2. Split data

# Uses our custom train_validation_test_split function
X_train, X_val, X_test, y_train, y_val, y_test = train_validation_test_split(
    X, y, train_size=0.8, val_size=0.1, test_size=0.1, random_state=1)


# 3. Fit model

# Try re-running the cell with different values for this parameter
class_weight = {0: 1, 1: 100} #{0: 1, 1:10000}means make the positive 10,000 times more weight
#default parameter for Logistic Regression, we rebalance in next cell

model = LogisticRegression(solver='lbfgs', class_weight=class_weight)
model.fit(X_train, y_train)


# 4. Evaluate model

y_pred = model.predict(X_val)
print(classification_report(y_val, y_pred))
print('accuracy', accuracy_score(y_val, y_pred))
display(pd.DataFrame(
    confusion_matrix(y_val, y_pred), 
    columns=['Predicted Negative', 'Predicted Positive'], 
    index=['Actual Negative', 'Actual Positive']))


# 5. Visualize decision regions

plt.figure(figsize=(10, 6))
plot_decision_regions(X_val, y_val, model, legend=0)

y_pred_proba = model.predict_proba(X_val)[:,1]
print('ROC AUC', roc_auc_score(y_val, y_pred_proba))
#1. Generate data

# Try re-running the cell with different values for these parameters
n_samples = 1000
weights = (0.95, 0.05)#we are generating data that is 95% one class, 5% another class
class_sep = 0.8

X, y = make_classification(n_samples=n_samples, n_features=2, n_informative=2, 
                           n_redundant=0, n_repeated=0, n_classes=2, 
                           n_clusters_per_class=1, weights=weights, 
                           class_sep=class_sep, random_state=0)


# 2. Split data

# Uses our custom train_validation_test_split function
X_train, X_val, X_test, y_train, y_val, y_test = train_validation_test_split(
    X, y, train_size=0.8, val_size=0.1, test_size=0.1, random_state=1)


# 3. Fit model

# Try re-running the cell with different values for this parameter
class_weight = {0: 1, 1: 10000} #{0: 1, 1:10000}means make the positive 10,000 times more weight
#default parameter for Logistic Regression, we rebalance in next cell

model = LogisticRegression(solver='lbfgs', class_weight=class_weight)
model.fit(X_train, y_train)


# 4. Evaluate model

y_pred = model.predict(X_val)
print(classification_report(y_val, y_pred))
print('accuracy', accuracy_score(y_val, y_pred))
display(pd.DataFrame(
    confusion_matrix(y_val, y_pred), 
    columns=['Predicted Negative', 'Predicted Positive'], 
    index=['Actual Negative', 'Actual Positive']))


# 5. Visualize decision regions

plt.figure(figsize=(10, 6))
plot_decision_regions(X_val, y_val, model, legend=0)

y_pred_proba = model.predict_proba(X_val)[:,1]
print('ROC AUC', roc_auc_score(y_val, y_pred_proba))<jupyter_output>precision    recall  f1-score   support

           0       1.00      0.15      0.25        96
           1       0.05      1.00      0.09         4

   micro avg       0.18      0.18      0.18       100
   macro avg       0.52      0.57      0.17       100
weighted avg       0.96      0.18      0.25       100

accuracy 0.18<jupyter_text>usually higher roc auc better f1 score Try re-running the cell above with different values for these four parameters:- `n_samples`- `weights`- `class_sep`- `class_balance`For example, with a 50% / 50% class distribution:```n_samples = 1000weights = (0.50, 0.50)class_sep = 0.8class_balance = None```With a 95% / 5% class distribution:```n_samples = 1000weights = (0.95, 0.05)class_sep = 0.8class_balance = None```With the same 95% / 5% class distribution, but changing the Logistic Regression's `class_balance` parameter to `'balanced'` (instead of its default `None`)```n_samples = 1000weights = (0.95, 0.05)class_sep = 0.8class_balance = 'balanced'```With the same 95% / 5% class distribution, but with different values for `class_balance`:- `{0: 1, 1: 1}` _(equivalent to `None`)_- `{0: 1, 1: 2}`- `{0: 1, 1: 10}` _(roughly equivalent to `'balanced'` for this dataset)_- `{0: 1, 1: 100}`- `{0: 1, 1: 10000}`How do the evaluation metrics and decision region plots change?   What you can do about imbalanced classes [Learning from Imbalanced Classes](https://www.svds.com/tbt-learning-imbalanced-classes/) gives "a rough outline of useful approaches" : - Do nothing. Sometimes you get lucky and nothing needs to be done. You can train on the so-called natural (or stratified) distribution and sometimes it works without need for modification.- Balance the training set in some way:  - Oversample the minority class.  - Undersample the majority class.  - Synthesize new minority classes.- Throw away minority examples and switch to an anomaly detection framework.- At the algorithm level, or after it:  - Adjust the class weight (misclassification costs).  - Adjust the decision threshold.  - Modify an existing algorithm to be more sensitive to rare classes.- Construct an entirely new algorithm to perform well on imbalanced data.  We demonstrated two of these options: - "Adjust the class weight (misclassification costs)" — many scikit-learn classifiers have a `class_balance` parameter- "Adjust the decision threshold" — you can lean more about this in a great blog post, [Visualizing Machine Learning Thresholds to Make Better Business Decisions](https://blog.insightdatascience.com/visualizing-machine-learning-thresholds-to-make-better-business-decisions-4ab07f823415). Another option to be aware of:- The [imbalance-learn](https://github.com/scikit-learn-contrib/imbalanced-learn) library can be used to "oversample the minority class, undersample the majority class, or synthesize new minority classes."  ASSIGNMENT Bank Marketing- Try the `class_weight` parameter.- Explore and visualize your data. - Wrangle [bad data](https://github.com/Quartz/bad-data-guide), outliers, and missing values.- Try engineering more features. You can transform, bin, and combine features. - Try selecting fewer features.-Try class weight roc_auc Imbalanced Classes demo with synthetic data- Play around with the demo. Change parameter values.- Be able to calculate precision, recall, F1, and accuracy "by hand", given a confusion matrix and access to Wikipedia. STRETCH- Read the blog post, [Visualizing Machine Learning Thresholds to Make Better Business Decisions](https://blog.insightdatascience.com/visualizing-machine-learning-thresholds-to-make-better-business-decisions-4ab07f823415). You can replicate the code as-is,  ["the hard way"](https://docs.google.com/document/d/1ubOw9B3Hfip27hF2ZFnW3a3z9xAgrUDRReOEo-FHCVs/edit). Or you can apply it to the Bank Marketing dataset.- Try the [imbalance-learn](https://github.com/scikit-learn-contrib/imbalanced-learn) library.- Try other [scikit-learn classifiers](https://scikit-learn.org/stable/supervised_learning.html), beyond Logistic Regression.<jupyter_code># Values are missing
# Zeros replace missing values
# Spelling is inconsistent
# Units are not specified
# Categories are badly chosen
# Field names are ambiguous
# Spreadsheet has 65536 rows, 255 columns being trunkated excel data
# Spreadsheet has dates in 1900, 1904, 1969, or 1970
bank.head().T
bank.select_dtypes(exclude='number').nunique()
bank['poutcome'].value_counts()
bank['job'].value_counts()
import category_encoders as ce

pipeline = make_pipeline(
    ce.OneHotEncoder(use_cat_names=True))
pipeline
bank.head().T
from numpy import array
data = ['cold', 'cold', 'warm', 'cold', 'hot', 'hot', 'warm', 'cold', 'warm', 'hot']
values = array(data)
print(values)
from sklearn.metrics import accuracy_score, confusion_matrix
from sklearn.preprocessing import LabelEncoder
from sklearn.preprocessing import OneHotEncoder

values = bank['job']

label_encoder = LabelEncoder()
integer_encoded = label_encoder.fit_transform(values)
print(integer_encoded)


# onehot_encoder = OneHotEncoder(sparse=False)
# onehot_encoded = onehot_encoder.fit_transform(integer_encoded)
# print(onehot_encoded)

# X_enc = pd.get_dummies(X_enc, columns=['sex','region','smoker'])
# mergedata = mergedata.drop(['job', 'marital','education', 'default', 'housing', 'loan', 'contact', 'month', 'day_of_week', 'poutcome'],axis=1)

# X_enc = pd.get_dummies(X_enc, columns=['sex','region','smoker'])

# inverted = label_encoder.inverse_transform([argmax(onehot_encoded[0, :])])
# print(inverted)
len(integer_encoded), bank.shape
bank.shape
bank['job_label'] = integer_encoded
bank.head(3)
def make_features(X):
  get2darray = np.array([])
  for i in X:
    values = bank[i]

    label_encoder = LabelEncoder()
    integer_encoded = label_encoder.fit_transform(values)
    get2darray = np.append(get2darray, [integer_encoded])
  return(get2darray)# you need to return it, print returns a nonetype
X =  ['marital','education', 'default', 'housing', 'loan', 'contact', 'poutcome']
twodimensionalarray = make_features(X)
newshape = twodimensionalarray.reshape((7,41188))

bank['education_labeled'] = newshape[0]
bank['default_labeled'] = newshape[1]
bank['housing_labeled'] = newshape[2]
bank['loan_labeled'] = newshape[3]
bank['contact_labeled'] = newshape[4]
bank['poutcome_labeled'] = newshape[5]
bank.columns.tolist()
bank.describe()
# X =  ['marital','education', 'default', 'housing', 'loan', 'contact', 'month', 'day_of_week', 'poutcome']

%matplotlib inline
import warnings
import category_encoders as ce
import matplotlib.pyplot as plt
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.exceptions import DataConversionWarning
from sklearn.preprocessing import StandardScaler
warnings.filterwarnings(action='ignore', category=DataConversionWarning)

# Assign to X, y
# X = bank.drop(columns='y')
X = bank.select_dtypes('number')
y = bank['y'] == 'yes'


X.columns.tolist()
X.head(3)
# Drop leaky feature
# X = X.drop(columns='duration')

# Split Train, Test
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y)

# Make pipeline
pipeline = make_pipeline( 
    StandardScaler(), 
    LogisticRegression(solver='lbfgs', max_iter=1000)
)
from sklearn.model_selection import cross_val_predict

y_pred_proba = cross_val_predict(pipeline, X_train, y_train, cv=3, n_jobs=-1, 
                                 method='predict_proba')[:,1]
threshold = 0.5#% confidence people will sign up
#how many positive predictions we want
y_pred = y_pred_proba >= threshold
correct = y_pred == y_train

labels = pd.DataFrame({'Ground Truth': y_train, 
                       'Predicted Probability': y_pred_proba, 
                       'Discrete Prediction': y_pred, 
                       'Correct Prediction?': correct})

labels.head(20)
threshold = 0.5#% confidence people will sign up
y_pred = y_pred_proba >= threshold
correct = y_pred == y_train

labels = pd.DataFrame({'Ground Truth': y_train, 
                       'Predicted Probability': y_pred_proba, 
                       'Discrete Prediction': y_pred, 
                       'Correct Prediction?': correct})

labels.loc[8]
labels['Discrete Prediction'].value_counts(normalize=True).plot.barh(color='grey');
labels['Ground Truth'].value_counts(normalize=True).plot.barh(color='grey');
import seaborn as sns
sns.distplot(labels['Predicted Probability']);
from sklearn.metrics import classification_report, confusion_matrix

threshold = 0.1
y_pred = y_pred_proba >= threshold

print(classification_report(y_train, y_pred))

pd.DataFrame(confusion_matrix(y_train, y_pred), 
             columns=['Predicted Negative', 'Predicted Positive'], 
             index=['Actual Negative', 'Actual Positive'])
threshold = 0.9
y_pred = y_pred_proba >= threshold

print(classification_report(y_train, y_pred))

pd.DataFrame(confusion_matrix(y_train, y_pred), 
             columns=['Predicted Negative', 'Predicted Positive'], 
             index=['Actual Negative', 'Actual Positive'])
threshold = 0.05
y_pred = y_pred_proba >= threshold

print(classification_report(y_train, y_pred))

pd.DataFrame(confusion_matrix(y_train, y_pred), 
             columns=['Predicted Negative', 'Predicted Positive'], 
             index=['Actual Negative', 'Actual Positive'])
threshold = 0.005
y_pred = y_pred_proba >= threshold

print(classification_report(y_train, y_pred))

pd.DataFrame(confusion_matrix(y_train, y_pred), 
             columns=['Predicted Negative', 'Predicted Positive'], 
             index=['Actual Negative', 'Actual Positive'])
from sklearn.metrics import accuracy_score
accuracy_score(y_train, y_pred)
true_negative  = 20945 #actual negative, predicted negative
false_positive = 8293 #actual negative, predicted positive
false_negative = 129 #actual positive, predicted negative
true_positive  = 3583 #actual positive, predicted positive

actual_negative = 20945 + 8293
actual_positive =129 + 3583

predicted_negative = 20945 + 8293
predicted_positive = 8293 + 3583

accuracy = (true_positive + true_negative) / (predicted_negative + predicted_positive)
#correct predictions / total predictions % , (true positives + true negatives)/(sum of all 4 numbers) 
precision = true_positive / predicted_positive
#precision and recall are based on relevance
#precision, based on how many selected, which are relevant?
#any search result on google - precision - search with eral scruggs / total search results
recall = true_positive / actual_positive
#recall, based on how many relevant, how many are selected?
#what was your recall for the most relevant things?
#of all the people who would sign up, how many did you actually select to call?
f1 = 2*precision * recall / (precision+recall)
print('Accuracy:',accuracy,'Precision:', precision, 'Recall:', recall,'F1', f1)
from sklearn.metrics import roc_auc_score, roc_curve
fpr, tpr, thresholds = roc_curve(y_train, y_pred_proba)
plt.plot(fpr, tpr)
plt.title('ROC curve')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
print('Area under the Receiver Operating Characteristic curve:', 
      roc_auc_score(y_train, y_pred_proba))
    
# When threshold = 0.5
false_positives = false_positive
true_positives = true_positive
false_positive_rate = false_positives/actual_negative
true_positive_rate = true_positives/actual_positive
plt.scatter(false_positive_rate, true_positive_rate)

# When threshold = 0.1
false_positives = false_positive
true_positives = true_positive
false_positive_rate = false_positives/actual_negative
true_positive_rate = true_positives/actual_positive
plt.scatter(false_positive_rate, true_positive_rate);
bank.columns.tolist()
bank_reduced = bank[['default_labeled', 'housing_labeled','loan_labeled','contact_labeled','campaign','previous','poutcome_labeled','nr.employed','cons.price.idx','cons.conf.idx']]
bank_reduced['previous_2'] = bank_reduced['previous'] * bank_reduced['poutcome_labeled']
bank_reduced['jobsxage'] = bank['nr.employed'] * bank['poutcome_labeled']
bank_reduced['jobsxxage'] = bank['previous'] * bank['housing_labeled']*bank['poutcome_labeled']
bank_reduced.head(3)
# Assign to X, y
X = bank_reduced
y = bank['y'] == 'yes'

# Split Train, Test
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y)#only makes sense for categorical variables, using class labels, only for classification, not regression

# Make pipeline
pipeline = make_pipeline(
    ce.OneHotEncoder(use_cat_names=True), 
    StandardScaler(), 
    LogisticRegression(solver='lbfgs', max_iter=1000)
)
from sklearn.model_selection import cross_val_predict

y_pred_proba = cross_val_predict(pipeline, X_train, y_train, cv=3, n_jobs=-1, 
                                 method='predict_proba')[:,1]
threshold = 0.5#% confidence people will sign up
#how many positive predictions we want
y_pred = y_pred_proba >= threshold
correct = y_pred == y_train

labels = pd.DataFrame({'Ground Truth': y_train, 
                       'Predicted Probability': y_pred_proba, 
                       'Discrete Prediction': y_pred, 
                       'Correct Prediction?': correct})

labels.head(20)
threshold = 0.5#% confidence people will sign up
y_pred = y_pred_proba >= threshold
correct = y_pred == y_train

labels = pd.DataFrame({'Ground Truth': y_train, 
                       'Predicted Probability': y_pred_proba, 
                       'Discrete Prediction': y_pred, 
                       'Correct Prediction?': correct})

labels.loc[8]
labels['Discrete Prediction'].value_counts(normalize=True).plot.barh(color='grey');
labels['Ground Truth'].value_counts(normalize=True).plot.barh(color='grey');
import seaborn as sns
sns.distplot(labels['Predicted Probability']);
from sklearn.metrics import classification_report, confusion_matrix

threshold = 0.1
y_pred = y_pred_proba >= threshold

print(classification_report(y_train, y_pred))

pd.DataFrame(confusion_matrix(y_train, y_pred), 
             columns=['Predicted Negative', 'Predicted Positive'], 
             index=['Actual Negative', 'Actual Positive'])
true_negative  = 21640 #actual negative, predicted negative
false_positive = 7598 #actual negative, predicted positive
false_negative = 1146 #actual positive, predicted negative
true_positive  = 2566 #actual positive, predicted positive

actual_negative = 21640 + 7598
actual_positive =1146 + 2566

predicted_negative = 21640 + 1146
predicted_positive = 7598 + 2566

accuracy = (true_positive + true_negative) / (predicted_negative + predicted_positive)
#correct predictions / total predictions % , (true positives + true negatives)/(sum of all 4 numbers) 
precision = true_positive / predicted_positive
#precision and recall are based on relevance
#precision, based on how many selected, which are relevant?
#any search result on google - precision - search with eral scruggs / total search results
recall = true_positive / actual_positive
#recall, based on how many relevant, how many are selected?
#what was your recall for the most relevant things?
#of all the people who would sign up, how many did you actually select to call?
f1 = 2*precision * recall / (precision+recall)
print('Accuracy:',accuracy,'Precision:', precision, 'Recall:', recall,'F1', f1)
from sklearn.metrics import roc_auc_score, roc_curve
fpr, tpr, thresholds = roc_curve(y_train, y_pred_proba)
plt.plot(fpr, tpr)
plt.title('ROC curve')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
print('Area under the Receiver Operating Characteristic curve:', 
      roc_auc_score(y_train, y_pred_proba))
    
# When threshold = 0.5
false_positives = false_positive
true_positives = true_positive
false_positive_rate = false_positives/actual_negative
true_positive_rate = true_positives/actual_positive
plt.scatter(false_positive_rate, true_positive_rate)

# When threshold = 0.1
false_positives = false_positive
true_positives = true_positive
false_positive_rate = false_positives/actual_negative
true_positive_rate = true_positives/actual_positive
plt.scatter(false_positive_rate, true_positive_rate);
bank_reduced_2 = bank[['poutcome','job_label']]
# Assign to X, y
X = bank_reduced_2
y = bank['y'] == 'yes'

# Split Train, Test
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y)#only makes sense for categorical variables, using class labels, only for classification, not regression

# Make pipeline
pipeline = make_pipeline(
    ce.OneHotEncoder(use_cat_names=True), 
    StandardScaler(), 
    LogisticRegression(solver='lbfgs', max_iter=1000)
)
from sklearn.model_selection import cross_val_predict

y_pred_proba = cross_val_predict(pipeline, X_train, y_train, cv=3, n_jobs=-1, 
                                 method='predict_proba')[:,1]
threshold = 0.5#% confidence people will sign up
#how many positive predictions we want
y_pred = y_pred_proba >= threshold
correct = y_pred == y_train

labels = pd.DataFrame({'Ground Truth': y_train, 
                       'Predicted Probability': y_pred_proba, 
                       'Discrete Prediction': y_pred, 
                       'Correct Prediction?': correct})

labels.head(20)
threshold = 0.5#% confidence people will sign up
y_pred = y_pred_proba >= threshold
correct = y_pred == y_train

labels = pd.DataFrame({'Ground Truth': y_train, 
                       'Predicted Probability': y_pred_proba, 
                       'Discrete Prediction': y_pred, 
                       'Correct Prediction?': correct})

labels.loc[8]
labels['Discrete Prediction'].value_counts(normalize=True).plot.barh(color='grey');
labels['Ground Truth'].value_counts(normalize=True).plot.barh(color='grey');
import seaborn as sns
sns.distplot(labels['Predicted Probability']);
from sklearn.metrics import classification_report, confusion_matrix

threshold = 0.1
y_pred = y_pred_proba >= threshold

print(classification_report(y_train, y_pred))

pd.DataFrame(confusion_matrix(y_train, y_pred), 
             columns=['Predicted Negative', 'Predicted Positive'], 
             index=['Actual Negative', 'Actual Positive'])
true_negative  = 25915 #actual negative, predicted negative
false_positive = 3323 #actual negative, predicted positive
false_negative = 2501 #actual positive, predicted negative
true_positive  = 1211 #actual positive, predicted positive

actual_negative = 25915 + 3323
actual_positive = 2501 + 1211

predicted_negative = 25915 + 3323
predicted_positive = 2501 + 1211

accuracy = (true_positive + true_negative) / (predicted_negative + predicted_positive)
#correct predictions / total predictions % , (true positives + true negatives)/(sum of all 4 numbers) 
precision = true_positive / predicted_positive
#precision and recall are based on relevance
#precision, based on how many selected, which are relevant?
#any search result on google - precision - search with eral scruggs / total search results
recall = true_positive / actual_positive
#recall, based on how many relevant, how many are selected?
#what was your recall for the most relevant things?
#of all the people who would sign up, how many did you actually select to call?
f1 = 2*precision * recall / (precision+recall)
print('Accuracy:',accuracy,'Precision:', precision, 'Recall:', recall,'F1', f1)
from sklearn.metrics import roc_auc_score, roc_curve
fpr, tpr, thresholds = roc_curve(y_train, y_pred_proba)
plt.plot(fpr, tpr)
plt.title('ROC curve')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
print('Area under the Receiver Operating Characteristic curve:', 
      roc_auc_score(y_train, y_pred_proba))
    
# When threshold = 0.5
false_positives = false_positive
true_positives = true_positive
false_positive_rate = false_positives/actual_negative
true_positive_rate = true_positives/actual_positive
plt.scatter(false_positive_rate, true_positive_rate)

# When threshold = 0.1
false_positives = false_positive
true_positives = true_positive
false_positive_rate = false_positives/actual_negative
true_positive_rate = true_positives/actual_positive
plt.scatter(false_positive_rate, true_positive_rate);
Precision = truepositive / (truepositive + falsepositive)
Recall = truepositive / (truepositive + falsenegative)
precision = the mean, plus or minus the average deviation. 

accuracy = (accepted value - observed)/(accepted value*100)
 Recall is defined as the number of relevant documents retrieved by a search divided by the total number of existing relevant documents
F1 score (also F-score or F-measure) is a measure of a test's accuracy. 
= 2*(precision*recall)/(precision+recall)
https://stats.stackexchange.com/questions/51296/how-do-you-calculate-precision-and-recall-for-multiclass-classification-using-co
https://www.youtube.com/watch?v=TtIjAiSojFE<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09887160489160225, 'fraction_numerical': 0.022908121798627782, 'mean_word_length': 4.844250524687949, 'pattern_counts': {'":': 0, '<': 35, '<?xml version=': 0, '>': 56, 'https://': 33, 'lorem ipsum': 0, 'www.': 5, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '209879', 'license': 'MIT', 'n_tokens_mistral': 17631, 'n_tokens_neox': 16100, 'n_words': 4762, 'path': 'LS_DS_242_Validate_classification_problems.ipynb', 'repo_name': 'LilySu/DS-Unit-2-Sprint-3-Classification-Validation', 'size': '52907'}