<jupyter_start><jupyter_text>Tackling Classification Problems Things to look out:- class imbalance: use classification report, ROC, AUC for this matter    (DataCamp: Supervised Learning)- hyperparameter tuning: GridSearchCv, RandomizedSearchCV, hold-out set   (DataCamp: Supervised Learning)-   Confusion Matrix: Spam Example$$Precision = \frac{tp}{tp + fp}\\Recall = \frac{tp}{tp + fn}\\F1 score = 2 * \frac{precision*recall}{precision + recall}$$High Precision = not many real emails predicted as spam High Recall = predicted most spam emails correctly<jupyter_code>from sklearn.metrics import classification_report, confusion_matrix
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_iris

import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt

iris = load_iris()
iris_df = pd.DataFrame(iris.data, columns=iris.feature_names)
iris_df
X, y = iris_df, iris.target
print(X.shape, y.shape)
# simple LogisticRegression to illustrate confusion matrix
log_reg = LogisticRegression().fit(X, y)
confusion_matrix(y, log_reg.predict(X))
print(classification_report(y, log_reg.predict(X)))<jupyter_output>precision    recall  f1-score   support

           0       1.00      1.00      1.00        50
           1       0.98      0.94      0.96        50
           2       0.94      0.98      0.96        50

    accuracy                           0.97       150
   macro avg       0.97      0.97      0.97       150
weighted avg       0.97      0.97      0.97       150<jupyter_text>LogisticRegression & ROC CurveUse this when there's a binary classification!<jupyter_code>diabetes = pd.read_csv('datasets/diabetes.csv')
diabetes.head(3)
from sklearn.model_selection import train_test_split

X, y = diabetes.drop('diabetes', axis=1), diabetes['diabetes']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=.2)
log_reg = LogisticRegression().fit(X_train, y_train)
y_pred = log_reg.predict(X_test)

from sklearn.metrics import roc_curve
y_pred_prob = log_reg.predict_proba(X_test)[:, 1]
fpr, tpr, thresholds = roc_curve(y_test, y_pred_prob)

_ = plt.plot([0, 1], [0, 1], 'k--')
_ = plt.plot(fpr, tpr, label='Logistic Regression')
_ = plt.xlabel('False Positive Rate')
_ = plt.ylabel('True Positive Rate')
_ = plt.title('Logistic Regression ROC Curve')
plt.show()<jupyter_output><empty_output><jupyter_text>AUC<jupyter_code>from sklearn.metrics import roc_auc_score
from sklearn.model_selection import cross_val_score

print('AUC score:', roc_auc_score(y_test, y_pred_prob))
cv_scores = cross_val_score(log_reg, X, y, cv=5, scoring='roc_auc')
cv_scores<jupyter_output>AUC score: 0.8192592592592592<jupyter_text>Hyperparameter Tuning GridSearchCV w/ cross_val_score<jupyter_code>from sklearn.model_selection import GridSearchCV 
from sklearn.neighbors import KNeighborsClassifier
param_grid = {
    'n_neighbors': np.arange(1, 50)
}

knn = KNeighborsClassifier()
knn_cv = GridSearchCV(knn, param_grid, cv=5)
knn_cv.fit(X, y)

print(knn_cv.best_params_)
print(knn_cv.best_score_)

# knn_cv.cv_results_
# logistic regression example
c_space = np.logspace(-5, 8, 15)
param_grid = {'C': c_space}

logreg = LogisticRegression()
logreg_cv = GridSearchCV(logreg, param_grid, cv=5).fit(X, y)

print(logreg_cv.best_params_)
print(logreg_cv.best_score_)
c_space<jupyter_output><empty_output><jupyter_text>RandomizedSearchCVGridSearchCV can be computationally expensive, especially if you are searching over a large hyperparameter space and dealing with multiple hyperparameters. A solution to this is to use RandomizedSearchCV, in which not all hyperparameter values are tried out. Instead, a fixed number of hyperparameter settings is sampled from specified probability distributions. You'll practice using RandomizedSearchCV in this exercise and see how this works.<jupyter_code>from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import randint

# Setup the parameters and distributions to sample from: param_dist
param_dist = {
    'max_depth': [3, None],
    'max_features': randint(1, 9),
    'min_samples_leaf': randint(1, 9),
    'criterion': ['gini', 'entropy']
}

# Instantiate a Decision Tree classifier: tree
tree = DecisionTreeClassifier()
tree_cv = RandomizedSearchCV(tree, param_dist, cv=5).fit(X, y)

# Print the tuned parameters and score
print("Tuned Decision Tree Parameters: {}".format(tree_cv.best_params_))
print("Best score is {}".format(tree_cv.best_score_))<jupyter_output>Tuned Decision Tree Parameters: {'criterion': 'entropy', 'max_depth': 3, 'max_features': 7, 'min_samples_leaf': 2}
Best score is 0.7396316102198455<jupyter_text>Hold-out set<jupyter_code># Import necessary modules
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV

# Create the hyperparameter grid
c_space = np.logspace(-5, 8, 15)
param_grid = {'C': c_space, 'penalty': ['l1', 'l2']}

# Instantiate the logistic regression classifier: logreg
logreg = LogisticRegression()

# Create train and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=.4, random_state=42)

# Instantiate the GridSearchCV object: logreg_cv
logreg_cv = GridSearchCV(logreg, param_grid, cv=5)

# Fit it to the training data
logreg_cv.fit(X_train, y_train)

# Print the optimal parameters and best score
print("Tuned Logistic Regression Parameter: {}".format(logreg_cv.best_params_))
print("Tuned Logistic Regression Accuracy: {}".format(logreg_cv.best_score_))<jupyter_output>Tuned Logistic Regression Parameter: {'C': 163789.3706954068, 'penalty': 'l2'}
Tuned Logistic Regression Accuracy: 0.7782608695652173
{'fraction_non_alphanumeric': 0.09794711203897008, 'fraction_numerical': 0.030271398747390398, 'mean_word_length': 5.2968236582694415, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '243240', 'license': 'MIT', 'n_tokens_mistral': 2059, 'n_tokens_neox': 1905, 'n_words': 505, 'path': 'datacamp_ml/ml_classification_regression/classification.ipynb', 'repo_name': 'issagaliyeva/machine_learning', 'size': '5748'}