<jupyter_start><jupyter_text>Logistic Regression **Here we show one example of Logistic Regression using scikit-learn library. How to tune Hyperparameters using Grid and Random Search and Cross validation with several folds**  Imports<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

plt.rc("font", size=14)

from sklearn.model_selection import GridSearchCV, RandomizedSearchCV
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

from sklearn.metrics import roc_auc_score
from sklearn.metrics import roc_curve
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix

from imblearn.over_sampling import SMOTE

from scipy.stats import expon

import seaborn as sns

sns.set(style="white")
sns.set(style="whitegrid", color_codes=True)<jupyter_output><empty_output><jupyter_text>Helper Functions<jupyter_code>def pie_plot(series, figsize=(7, 7)):
    plt.figure(figsize=figsize)
    plt.pie(series.value_counts(), labels=series.value_counts().index);
def get_confusion_matrix(y_test, y_pred):
    return pd.DataFrame(
        confusion_matrix(y_test, y_pred),
        index=["True", "False"],
        columns=["Pred_True", "Pred_False"],
    )
def plot_roc_auc_curve(y_test, y_pred, clf, X_test):
    logit_roc_auc = roc_auc_score(y_test, y_pred)
    fpr, tpr, thresholds = roc_curve(y_test, clf.predict_proba(X_test)[:, 1])
    plt.figure(figsize=[10, 7])
    plt.plot(fpr, tpr, label="Logistic Regression (area = %0.2f)" % logit_roc_auc)
    plt.plot([0, 1], [0, 1], "r--")
    plt.xlim([0.0, 1.0])
    plt.ylim([0.0, 1.05])
    plt.xlabel("False Positive Rate")
    plt.ylabel("True Positive Rate")
    plt.title("Receiver operating characteristic")
    plt.legend(loc="lower right")
    plt.savefig("../../resources/machine-learning-introduction/roc-auc")
    plt.show()<jupyter_output><empty_output><jupyter_text>The Data The data is related with direct marketing campaigns (phone calls) of a Portuguese banking institution. The classification goal is to predict if the client will subscribe (1/0) a term deposit (variable y). This dataset provides the customer information. It includes over 41K records and 21 fields.  Read Data<jupyter_code>data = pd.read_csv("../../data/machine-learning-introduction/banking.csv", header=0)
data.head(10).T<jupyter_output><empty_output><jupyter_text>Take only target and Numerical Features for Simplicity  Numerical variable descriptions- age (numeric)- duration: last contact duration, in seconds (numeric). Important note: this attribute highly affects the output target (e.g., if duration=0 then y='no'). Yet, the duration is not known before a call is performed. Also, after the end of the call y is obviously known. Thus, this input should only be included for benchmark purposes and should be discarded if the intention is to have a realistic predictive model.- campaign: number of contacts performed during this campaign and for this client (numeric, includes last contact)- pdays: number of days that passed by after the client was last contacted from a previous campaign (numeric; 999 means client was not previously contacted)- previous: number of contacts performed before this campaign and for this client (numeric)- poutcome: outcome of the previous marketing campaign (categorical: 'failure','nonexistent','success')- emp.var.rate: employment variation rate - (numeric)- cons.price.idx: consumer price index - (numeric)- cons.conf.idx: consumer confidence index - (numeric)- euribor3m: euribor 3 month rate - (numeric)- nr.employed: number of employees - (numeric)<jupyter_code>numerical_features = [
    "age",
    "campaign",
    "pdays",
    "duration",
    "previous",
    "previous",
    "emp_var_rate",
    "cons_price_idx",
    "cons_conf_idx",
    "euribor3m",
    "nr_employed",
]<jupyter_output><empty_output><jupyter_text>Numerical variable distributions<jupyter_code>data[numerical_features].hist(bins=20, figsize=(20, 20));<jupyter_output><empty_output><jupyter_text>Target variable:y - has the client subscribed a term deposit? (binary: '1','0')<jupyter_code>pie_plot(data.y)<jupyter_output><empty_output><jupyter_text>Take little part of the data to train models faster<jupyter_code>data = data.sample(10000)<jupyter_output><empty_output><jupyter_text>divide data into features and target<jupyter_code>X = data[numerical_features]
y = data["y"]
X.shape, y.shape<jupyter_output><empty_output><jupyter_text>Oversample the minority class by synthetically generating additional samples<jupyter_code>os = SMOTE(random_state=0)
X, y = os.fit_resample(X, y)
pie_plot(pd.Series(y))<jupyter_output><empty_output><jupyter_text>Logistic Regression Model Fitting  Create Logistic Regression classifier<jupyter_code>clf = LogisticRegression()<jupyter_output><empty_output><jupyter_text>Split data into train and test sets Here we won't see effect of stratify because we have balanced target. The use of stratify is to have the same distribution of target in train and test splits<jupyter_code>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)
X_train.shape, X_test.shape, y_train.shape, y_test.shape<jupyter_output><empty_output><jupyter_text>Create Parameter Grids  GridSearchCV Here we are going to create several splits of training data (K-folds) and then we are going to run grid search on every split. We will check calculated scores distributions on each split. parameter grid defines several choises for each hyperparameter we need to tune<jupyter_code>grid_parameters = {
    "penalty": ["l1", "l2"],
    "C": np.logspace(-3, 1, 20),
    "solver": ["liblinear"],
}<jupyter_output><empty_output><jupyter_text>RandomizedSearchCV<jupyter_code>pd.Series(expon.rvs(size=1000)).hist(bins=20)
random_parameters = {
    "penalty": ["l1", "l2"],
    "C": expon.rvs(size=20),
    "solver": ["liblinear"],
}<jupyter_output><empty_output><jupyter_text>Create grid search object - cv: cross validation (number of splits to do in data for cross validation the same as K-fold)- verbose: to show some logs while training- n_jobs: uses parallel runs for training different models (-1 means "all") There will be **CV*|PARAMS|** = **5x(20x2x1)** models trained (all combinations of parameters times number of splits)<jupyter_code>grid_clf = GridSearchCV(
    estimator=clf, param_grid=grid_parameters, cv=5, verbose=True, n_jobs=-1
)<jupyter_output><empty_output><jupyter_text>Randomized Grid search searches parameters not from predefined list of parameters but randomly generated parameters There will be **CV*|iters|** = **5x20** models trained (n random combinations of parameters times number of splits)<jupyter_code>random_clf = RandomizedSearchCV(
    estimator=clf,
    random_state=42,
    param_distributions=random_parameters,
    cv=5,
    verbose=True,
    n_jobs=-1,
    n_iter=20,
)<jupyter_output><empty_output><jupyter_text>Using RandomizedSearchCV  Fit on data<jupyter_code>best_param_grid_clf = grid_clf.fit(X_train, y_train)
best_param_random_clf = random_clf.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>Check Score Distributions on splits<jupyter_code>pd.DataFrame(grid_clf.cv_results_).hist(figsize=(20, 20));
pd.DataFrame(random_clf.cv_results_).hist(figsize=(20, 20));<jupyter_output><empty_output><jupyter_text>Check Best parameter models<jupyter_code>best_param_grid_clf.best_params_
best_param_random_clf.best_params_<jupyter_output><empty_output><jupyter_text>Predict on test data using best model<jupyter_code>y_pred_grid = best_param_grid_clf.predict(X_test)
y_pred_random = best_param_random_clf.predict(X_test)<jupyter_output><empty_output><jupyter_text>Confusion Matrices<jupyter_code>get_confusion_matrix(y_test, y_pred_grid)
get_confusion_matrix(y_test, y_pred_random)<jupyter_output><empty_output><jupyter_text>classification reports<jupyter_code>print(classification_report(y_test, y_pred_grid))
print(classification_report(y_test, y_pred_random))<jupyter_output>precision    recall  f1-score   support

           0       0.85      0.85      0.85      2588
           1       0.85      0.86      0.86      2709

    accuracy                           0.85      5297
   macro avg       0.85      0.85      0.85      5297
weighted avg       0.85      0.85      0.85      5297<jupyter_text>ROC-AUC curves<jupyter_code>plot_roc_auc_curve(y_test, y_pred_grid, best_param_grid_clf, X_test)
plot_roc_auc_curve(y_test, y_pred_random, best_param_random_clf, X_test)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10005844535359439, 'fraction_numerical': 0.01858562244301578, 'mean_word_length': 5.737007874015748, 'pattern_counts': {'":': 6, '<': 88, '<?xml version=': 0, '>': 88, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '22', 'dir': 'jupyter-structured-clean-dedup', 'id': '165010', 'license': 'Apache-2.0', 'n_tokens_mistral': 2897, 'n_tokens_neox': 2759, 'n_words': 722, 'path': 'content/week-08/samples/machine-learning-introduction/Logistic Regression Classifier Hyperparameter Tuning.ipynb', 'repo_name': 'GiorgiBeriashvili/school-of-ai', 'size': '8555'}