<jupyter_start><jupyter_text>Download and explore dataset  Context :It is important that credit card companies are able to recognize fraudulent credit card transactions so that customers are not charged for items that they did not purchase.The datasets contains transactions made by credit cards in September 2013 by european cardholders. This dataset presents transactions that occurred in two days, where we have 492 frauds out of 284,807 transactions. The dataset is highly unbalanced, the positive class (frauds) account for 0.172% of all transactions.It contains only numerical input variables which are the result of a PCA transformation. Unfortunately, due to confidentiality issues, we cannot provide the original features and more background information about the data. Features V1, V2, … V28 are the principal components obtained with PCA, the only features which have not been transformed with PCA are 'Time' and 'Amount'. Feature 'Time' contains the seconds elapsed between each transaction and the first transaction in the dataset. The feature 'Amount' is the transaction Amount, this feature can be used for example-dependant cost-senstive learning. Feature 'Class' is the response variable and it takes value 1 in case of fraud and 0 otherwise. Inspiration:Identify fraudulent credit card transactions.for more info, visit https://www.kaggle.com/mlg-ulb/creditcardfraud<jupyter_code>import kaggle
import os 

os.system('kaggle datasets download -d mlg-ulb/creditcardfraud')
!unzip creditcardfraud.zip
import pandas as pd

df = pd.read_csv('./creditcard.csv')
df.head()
import seaborn as sns
import matplotlib.pyplot as plt

corr = df.corr()
fig, ax = plt.subplots(figsize=(13,10))
plt.title('Correlation matrix')
sns.heatmap(corr, cmap='coolwarm_r')
plt.show()
df.Class.value_counts(normalize=True)
df.describe()<jupyter_output><empty_output><jupyter_text>0 as normal, 1 as a fraudulent transaction. The distribution is heavily skewed towards class 0. Also, one can see from correlation matrix and sample mean, standard deviation, the data has already been PCA processed, so the features are lineary independent and normalized.  Develop supervised baseline model<jupyter_code>from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import KFold
from sklearn.metrics import accuracy_score
import numpy as np

X = df.iloc[:, :-1].to_numpy()
y = df.iloc[:, -1].to_numpy()

# important to use cross-k fold validation to access accurately for model performance
kf = KFold(n_splits=3, shuffle=True)

cross_val_test_acc = []
for train_ind, test_ind in kf.split(X):
    clf = LogisticRegression(C=10, random_state=0, max_iter=1000)
    
    X_train, X_test = X[train_ind], X[test_ind]
    y_train, y_test = y[train_ind], y[test_ind]
    
    y_pred = clf.fit(X_train, y_train).predict(X_test)
    cross_val_test_acc.append(accuracy_score(y_test, y_pred))

cross_val_test_accuracy = np.mean(cross_val_test_acc)
print('Cross validation test accuray: {:.4f}'.format(cross_val_test_accuracy))<jupyter_output>Cross validation test accuray: 0.9991<jupyter_text>The accuracy is not surprisingly high for this skewed dataset since if one were to guess all 1s, he would already archieve 99.8% accuracy. Therefore, accuracy is not a meaningful metric.<jupyter_code># precision-recall curve instead of ROC AUC since the latter is sensitive to small sample size of skewed class
from sklearn.metrics import precision_recall_curve
from sklearn.metrics import confusion_matrix
from IPython.display import display

# to use precision recall metric, different threshold for the prob of predicted classes are used instead
y_scores = clf.predict_proba(X_test)
y_pred = clf.predict(X_test)
confusion_df = confusion_matrix(y_test, y_pred)
print('Confusion matrix:')
display(pd.DataFrame(confusion_df, index=['Actual: 0', 'Actual: 1'], columns=['Predicted: 0', 'Predicted: 1']))

precision, recall, thresholds = precision_recall_curve(y_test, y_scores[:, 1])

import matplotlib.pyplot as plt
plt.plot(recall, precision)
plt.show()
# to summarize the quality of our classifier
from sklearn import metrics

pr_auc = metrics.auc(recall, precision)
print('Precision-recall area under curve: {:.4f}'.format(pr_auc))<jupyter_output>Precision-recall area under curve: 0.6393<jupyter_text>From this perspective, the model is actually not doing a good job at classifying between fraud and non-fraud cases, since a perfect classifier would have a auc score of 1 and its area will cover up to the top right corner, in that case, that auc score would translate to a perfect precision score of 1 (indicating how much of one's predictions are right) and recall score of 1 (indicating how good one can capture the fraudulent cases actually happened).  Enter SMOTE (Synthetic Minority Over-sampling Technique)The motivation for SMOTE is not hard to understand, in a supervised algorithm, the loss function is heavily skewed to getting the majority right since by definition, loss function is proportional to the sum of all sample's error. Therefore, no matter how hard the algorithm tries, it will never have enough signal (gradient proposed by the minority class) to learn to classify the minority class.In essence, what SMOTE does is to upsample the minority class to make the dataset balance, by doing so, one can expect an improvement in pr-auc score.How SMOTE achieve this is, for each data point from the minority class, taking its k-nearest neighbors, this forms k pairs of vectors, based on how many upsampling is needed (in imblearn SMOTE implementation, if ratio == 'auto', it will upsample the minority class until the no. of data pts is equal to the majority class), a random number is generated for linearly interpolation between the pair of vectors.<jupyter_code>from imblearn.over_sampling import SMOTE

# example of how SMOTE works
sm = SMOTE(random_state=0)
X_res, y_res = sm.fit_resample(X, y)
plt.hist(y_res)
plt.show()<jupyter_output>Using TensorFlow backend.<jupyter_text>Now the dataset is balanced. Be cautious when doing cross validation with synthetic dataset, the synthetic dataset generated from training samples can only be used for training, for assessing the model performance, one have to use the original testing samples. In simple words, train test split happens before SMOTE.<jupyter_code>kf = KFold(n_splits=3, shuffle=True)

cross_val_test_acc = []
for train_ind, test_ind in kf.split(X):
    clf = LogisticRegression(C=10, random_state=0, max_iter=1000)
    
    X_train, X_test = X[train_ind], X[test_ind]
    y_train, y_test = y[train_ind], y[test_ind]
    
    # SMOTE happens before validation to avoid clf has already seen the test set
    sm = SMOTE(random_state=0)
    X_train_balanced, y_train_balanced = sm.fit_resample(X_train, y_train)
    
    y_pred = clf.fit(X_train_balanced, y_train_balanced).predict(X_test)
    cross_val_test_acc.append(accuracy_score(y_test, y_pred))

cross_val_test_accuracy = np.mean(cross_val_test_acc)
print('Cross validation test accuray: {:.4f}'.format(cross_val_test_accuracy))<jupyter_output>Cross validation test accuray: 0.9720<jupyter_text>The test accuracy dropped but does it mean it is doing a worse job ? wait and see, like mentioned previously, accuracy metric is not meaningful for such a skewed dataset.<jupyter_code>y_scores = clf.predict_proba(X_test) 

confusion_df = confusion_matrix(y_test, y_pred)
print('Confusion matrix:')
display(pd.DataFrame(confusion_df, index=['Actual: 0', 'Actual: 1'], columns=['Predicted: 0', 'Predicted: 1']))

precision, recall, thresholds = precision_recall_curve(y_test, y_scores[:, 1])
plt.plot(recall, precision)
plt.show()

pr_auc = metrics.auc(recall, precision)
print('Precision-recall area of test set under curve: {:.4f}'.format(pr_auc))<jupyter_output>Confusion matrix:<jupyter_text>Using SMOTE, one can see the accuracy decreased but that is normal, because that reveals the actual difficulty of classifying a fraudulent claim. And from the pr-auc score, it is now proved the classifier indeed learned much better to classify between fraud and normal transaction. The rest of the problem remaining how to choose a 'good' threshold.  Choosing a appropriate threshold for a classifier Depending on what the costs for tp, fp, tn, fn are the threshold for models can vary rather drastically in a real situation. Say like in a cancer diagnosis scenario, the doctor does not want to afford mistreating a health person and also does not want miss curing a patient who actually have cancer. However, the cost for mistreating a person is much higher than just miss curing a patient. Therefore, one would like to weight precision more heavily. If one is indifferent, one can use F1-score to balance them, F1-score is defined as a harmonic mean between precision and recall. Comparing arithmetic to harmonic mean, the latter penalize more when 2 numbers deviate from each other more.<jupyter_code>a = 0.7
b = 0.6
am_c = (a + b)/2
hm_c = (1/a + 1/b)**-1
print('Arithmetic mean for close numbers: {:.4f}'.format(am_c))
print('Harmonic mean for close numbers: {:.4f}\n'.format(hm_c))

a = 0.9
b = 0.3
am_d = (a + b)/2
hm_d = (1/a + 1/b)**-1
print('Arithmetic mean for distant numbers: {:.4f}'.format(am_c))
print('Harmonic mean for distant numbers: {:.4f}\n'.format(hm_c))

print('Change in arithmetic mean = {:.4f}'.format(abs(am_d - am_c)))
print('Change in harmonic mean = {:.4f}'.format(abs(hm_d - hm_c)))
# F1 score 
from sklearn.metrics import f1_score
print('F1_score for improved model at classifier\'s default threshold 0.5 : {:.4f}'.format(f1_score(y_test, y_pred)))
# using CV again to find the threshold that produces max F1 score since no. of class 0 is so small
kf = KFold(n_splits=3, shuffle=True)

# thresholds to test for
thresholds = np.arange(1, 1001)/1000

from sklearn.metrics import precision_score, recall_score

def precision_recall_by_threshold(y_test, y_scores, thresholds):
    p_ = []
    r_ = []
    for threshold in thresholds:
        y_pred = y_scores.copy()
        y_pred[y_scores >= threshold] = 1
        y_pred[y_scores < threshold] = 0
        
        p = precision_score(y_test, y_pred)
        r = recall_score(y_test, y_pred)
        
        p_.append(p)
        r_.append(r)
    
    return p_, r_
        
precision_array = np.zeros(len(thresholds))
recall_array = np.zeros(len(thresholds))
for train_ind, test_ind in kf.split(X):
    clf = LogisticRegression(C=10, random_state=0, max_iter=1000)
    
    X_train, X_test = X[train_ind], X[test_ind]
    y_train, y_test = y[train_ind], y[test_ind]
    
    sm = SMOTE(random_state=0)
    X_train_balanced, y_train_balanced = sm.fit_resample(X_train, y_train)
    
    y_scores = clf.fit(X_train_balanced, y_train_balanced).predict_proba(X_test)
    p_, r_ = precision_recall_by_threshold(y_test, y_scores[:, 1], thresholds)
    
    precision_array += p_
    recall_array += r_
    
# get average precision, recall
avg_precision = precision_array / 3
avg_recall = recall_array / 3

# get max F1 by threshold
def f1(p, r):
    return 2 * (1/p + 1/r) ** -1

max_ind = np.argmax(f1(avg_precision, avg_recall))
print('max F1_score for improved model at classifier\'s threshold {} : {:.4f}'\
      .format(thresholds[max_ind], f1(avg_precision[max_ind], avg_recall[max_ind])))<jupyter_output>max F1_score for improved model at classifier's threshold 0.999 : 0.7339
{'fraction_non_alphanumeric': 0.08179419525065963, 'fraction_numerical': 0.014335971855760774, 'mean_word_length': 5.0580713905167825, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '633084', 'license': 'MIT', 'n_tokens_mistral': 3647, 'n_tokens_neox': 3403, 'n_words': 1311, 'path': 'credit-card-fraud-detection.ipynb', 'repo_name': 'Revyung/credit-card-fraud-detection', 'size': '11370'}