<jupyter_start><jupyter_text>Homework 4Use this notebook as a starter<jupyter_code>import pandas as pd
import numpy as np

import matplotlib.pyplot as plt
import seaborn as sns

%matplotlib inline<jupyter_output><empty_output><jupyter_text>Data:- https://github.com/gastonstat/CreditScoring- Also available [here](https://raw.githubusercontent.com/alexeygrigorev/mlbookcamp-code/master/chapter-06-trees/CreditScoring.csv)<jupyter_code>!wget https://raw.githubusercontent.com/alexeygrigorev/mlbookcamp-code/master/chapter-06-trees/CreditScoring.csv<jupyter_output>--2021-10-02 21:38:57--  https://raw.githubusercontent.com/alexeygrigorev/mlbookcamp-code/master/chapter-06-trees/CreditScoring.csv
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 172.16.58.3, 192.168.3.11, 172.16.31.10, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|172.16.58.3|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 182489 (178K) [text/plain]
Saving to: ‘CreditScoring.csv.2’

CreditScoring.csv.2 100%[===================>] 178.21K  --.-KB/s    in 0.03s   

2021-10-02 21:38:57 (5.48 MB/s) - ‘CreditScoring.csv.2’ saved [182489/182489]<jupyter_text>Preparation We'll talk about this dataset in more details in week 6. But for now, use the following code to get started<jupyter_code>df = pd.read_csv('CreditScoring.csv')
df.columns = df.columns.str.lower()<jupyter_output><empty_output><jupyter_text>Some of the features are encoded as numbers. Use the following code to de-code them:<jupyter_code>status_values = {
    1: 'ok',
    2: 'default',
    0: 'unk'
}

df.status = df.status.map(status_values)


home_values = {
    1: 'rent',
    2: 'owner',
    3: 'private',
    4: 'ignore',
    5: 'parents',
    6: 'other',
    0: 'unk'
}

df.home = df.home.map(home_values)

marital_values = {
    1: 'single',
    2: 'married',
    3: 'widow',
    4: 'separated',
    5: 'divorced',
    0: 'unk'
}

df.marital = df.marital.map(marital_values)

records_values = {
    1: 'no',
    2: 'yes',
    0: 'unk'
}

df.records = df.records.map(records_values)

job_values = {
    1: 'fixed',
    2: 'partime',
    3: 'freelance',
    4: 'others',
    0: 'unk'
}

df.job = df.job.map(job_values)<jupyter_output><empty_output><jupyter_text>Prepare the numerical variables:<jupyter_code>for c in ['income', 'assets', 'debt']:
    df[c] = df[c].replace(to_replace=99999999, value=0)<jupyter_output><empty_output><jupyter_text>Remove clients with unknown default status<jupyter_code>df = df[df.status != 'unk'].reset_index(drop=True)<jupyter_output><empty_output><jupyter_text>Create the target variable<jupyter_code>df['default'] = (df.status == 'default').astype(int)
del df['status']<jupyter_output><empty_output><jupyter_text>Your code What are the categorical variables? What are the numerical?<jupyter_code>df.head()
df.dtypes, df.dtypes.index
numerical = ['seniority', 'time', 'age', 'expenses', 
             'income', 'assets', 'debt', 'amount', 
             'price']
categorical = ['home', 'marital', 'records', 'job']<jupyter_output><empty_output><jupyter_text>Split the data into 3 parts: train/validation/test with 60%/20%/20% distribution. Use `train_test_split` funciton for that with `random_state=1`<jupyter_code>from sklearn.model_selection import train_test_split
from sklearn.feature_extraction import DictVectorizer
from sklearn.linear_model import LogisticRegression
df_full_train, df_test = train_test_split(df, test_size=0.2, random_state=1)
df_train, df_val = train_test_split(df_full_train, test_size=0.25, random_state=1)

df_train = df_train.reset_index(drop=True)
df_val = df_val.reset_index(drop=True)
df_test = df_test.reset_index(drop=True)

y_train = df_train.default.values
y_val = df_val.default.values
y_test = df_test.default.values

del df_train['default']
del df_val['default']
del df_test['default']<jupyter_output><empty_output><jupyter_text>Question 1ROC AUC could also be used to evaluate feature importance of numerical variables. Let's do that* For each numerical variable, use it as score and compute AUC with the "default" variable* Use the training dataset for thatIf your AUC is < 0.5, invert this variable by putting "-" in front(e.g. `-df_train['expenses']`)AUC can go below 0.5 if the variable is negatively correlated with the target varialble. You can change the direction of the correlation by negating this variable - then negative correlation becomes positive.<jupyter_code>from sklearn.metrics import roc_auc_score 
for n in numerical:
    score = roc_auc_score(y_train, df_train[n])
    if score < 0.5:
        score = roc_auc_score(y_train, -df_train[n])
    print('%s=%.3f' % (n, score))<jupyter_output>seniority=0.709
time=0.561
age=0.573
expenses=0.501
income=0.682
assets=0.649
debt=0.505
amount=0.591
price=0.504<jupyter_text>Which numerical variable (among the following 4) has the highest AUC?- seniority- time- income- debt __Answer Q1: seniority=0.709__  Training the modelFrom now on, use these columns only:```['seniority', 'income', 'assets', 'records', 'job', 'home']```Apply one-hot-encoding using `DictVectorizer` and train the logistic regression with these parameters:```LogisticRegression(solver='liblinear', C=1.0, max_iter=1000)```<jupyter_code>columns = ['seniority', 'income', 'assets', 'records', 'job', 'home']


def train(df_t, y_t, C=1.0):
    dicts = df_t[columns].to_dict(orient='records')
    dv = DictVectorizer(sparse=False)
    X_train = dv.fit_transform(dicts)
    
    model = LogisticRegression(solver='liblinear',C=C,max_iter=1000)
    model.fit(X_train, y_t)
    
    return dv, model
dv, model = train(df_train, y_train)
dv.get_feature_names()<jupyter_output><empty_output><jupyter_text>Question 2What's the AUC of this model on the validation dataset? (round to 3 digits)- 0.512- 0.612- 0.712- 0.812<jupyter_code>val_dict = df_val[columns].to_dict(orient='records')
X_val = dv.transform(val_dict)

y_pred = model.predict_proba(X_val)[:, 1]
roc_auc_score(y_val, y_pred).round(3)<jupyter_output><empty_output><jupyter_text>__Answer Q2: roc_auc_score~0.812__  Question 3Now let's compute precision and recall for our model.* Evaluate the model on all thresholds from 0.0 to 1.0 with step 0.01* For each threshold, compute precision and recall* Plot them<jupyter_code>scores = []

thresholds = np.linspace(0, 1, 101)

for t in thresholds:
    actual_positive = (y_val == 1)
    actual_negative = (y_val == 0)
    predict_positive = (y_pred >= t)
    predict_negative = (y_pred < t)
    tp = (predict_positive & actual_positive).sum()
    tn = (predict_negative & actual_negative).sum()
    fp = (predict_positive & actual_negative).sum()
    fn = (predict_negative & actual_positive).sum()
    scores.append((t, tp, fp, fn, tn))
auc_columns = ['threshold','tp', 'fp', 'fn', 'tn']
df_scores = pd.DataFrame(scores, columns=auc_columns)

df_scores['fpr'] = df_scores.fp / (df_scores.tn + df_scores.fp)
df_scores['tpr'] = df_scores.tp / (df_scores.fn + df_scores.tp)
df_scores['p'] = df_scores.tp / (df_scores.tp + df_scores.fp)
df_scores['r'] = df_scores.tp / (df_scores.tp + df_scores.fn)
df_scores.isnull().sum()
df_scores['p'] = df_scores['p'].fillna(0)
df_scores[::10]
plt.plot(df_scores.threshold, df_scores.p, label='precision')
plt.plot(df_scores.threshold, df_scores.r, label='recall')
plt.xlabel('threshold')
plt.ylabel('score')
plt.legend()<jupyter_output><empty_output><jupyter_text>At which threshold precision and recall curves intersect?* 0.2* 0.4* 0.6* 0.8 __Answer Q3: thershold~0.4__  Question 4Precision and recall are conflicting - when one grows, the other goes down. That's why they are often combined into the F1 score - a metrics that takes into account bothThis is the formula for computing F1:$$F_1 = 2 \cdot \cfrac{P \cdot R}{P + R}$$Where $P$ is precision and $R$ is recall.Let's compute F1 for all thresholds from 0.0 to 1.0 with increment 0.01<jupyter_code>df_scores['F1'] = 2 * df_scores.p * df_scores.r / (df_scores.p + df_scores.r)
plt.plot(df_scores.threshold, df_scores.F1, label='F1')
plt.xlabel('threshold')
plt.ylabel('F1 score')
plt.legend()<jupyter_output><empty_output><jupyter_text>At which threshold F1 is maximal?- 0.1- 0.3- 0.5- 0.7 __Answer Q4: F1~0.3__  Question 5Use the `KFold` class from Scikit-Learn to evaluate our model on 5 different folds:```KFold(n_splits=5, shuffle=True, random_state=1)```* Iterate over different folds of `df_full_train`* Split the data into train and validation* Train the model on train with these parameters: `LogisticRegression(solver='liblinear', C=1.0, max_iter=1000)`* Use AUC to evaluate the model on validation<jupyter_code>from sklearn.model_selection import KFold
kfold = KFold(n_splits=5, shuffle=True, random_state=1)
def predict(df,dv,model):
    dicts = df[columns].to_dict(orient='records')
    
    X = dv.transform(dicts)
    y_pred = model.predict_proba(X)[:, 1]
    
    return y_pred
scores = []
C = 1.0

for train_idx, val_idx in kfold.split(df_full_train):
    df_train = df_full_train.iloc[train_idx]
    df_val = df_full_train.iloc[val_idx]

    y_train = df_train.default.values
    y_val = df_val.default.values

    dv, model = train(df_train, y_train, C=C)
    y_pred = predict(df_val,dv, model)

    auc = roc_auc_score(y_val, y_pred)
    scores.append(auc)
    
print('C=%s %.3f +- %.3f' % (C, np.mean(scores), np.std(scores)))<jupyter_output>C=1.0 0.814 +- 0.015<jupyter_text>How large is standard devidation of the scores across different folds?- 0.001- 0.014- 0.09- 0.14 __Answer Q5: STD~0.014__  Question 6Now let's use 5-Fold cross-validation to find the best parameter C* Iterate over the following C values: `[0.01, 0.1, 1, 10]`* Initialize `KFold` with the same parameters as previously* Use these parametes for the model: `LogisticRegression(solver='liblinear', C=C, max_iter=1000)`* Compute the mean score as well as the std (round the mean and std to 3 decimal digits)<jupyter_code>from tqdm.auto import tqdm
n_splits = 5

for C in tqdm([0.01, 0.1, 1, 10]):

    scores = []

    kfold = KFold(n_splits=n_splits,shuffle=True, random_state=1)
    
    for train_idx, val_idx in kfold.split(df_full_train):
        df_train = df_full_train.iloc[train_idx]
        df_val = df_full_train.iloc[val_idx]

        y_train = df_train.default.values
        y_val = df_val.default.values

        dv, model = train(df_train, y_train, C=C)
        y_pred = predict(df_val,dv, model)

        auc = roc_auc_score(y_val, y_pred)
        scores.append(auc)
    print('C=%s %.3f +- %.3f' % (C, np.mean(scores), np.std(scores)))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13531134838892564, 'fraction_numerical': 0.03609562505905698, 'mean_word_length': 4.9227756015668716, 'pattern_counts': {'":': 0, '<': 65, '<?xml version=': 0, '>': 64, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 4, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '602268', 'license': 'MIT', 'n_tokens_mistral': 4191, 'n_tokens_neox': 3828, 'n_words': 912, 'path': 'session-04/homework-04.ipynb', 'repo_name': 'MokretsovD/zoomcamp-home-works', 'size': '10597'}