<jupyter_start><jupyter_text>Exploratory Data Analysis<jupyter_code>data.info()
data.describe()
sns.heatmap(data.isnull(), cbar=False, yticklabels=False, cmap='viridis')
plt.tight_layout()<jupyter_output><empty_output><jupyter_text>There are no missing values in the dataset.<jupyter_code>plt.figure(figsize=(10,6))
print(sns.countplot(x='Class', data=data))
data['Class'].value_counts(normalize=True) * 100<jupyter_output>AxesSubplot(0.125,0.125;0.775x0.755)<jupyter_text>Dataset is highly imbalanced 99.83% Non-Fraudulent Transactions0.173% Fraudulent TransactionsHence, we require either oversampling or undersampling   Distribution of Amount & Time columns<jupyter_code>plt.figure(figsize=(10, 6))
sns.distplot(data['Amount'], bins=100)<jupyter_output><empty_output><jupyter_text>Amount of transactions are generally very low, most are less than 1000 and max. tend to be around 1500<jupyter_code>sns.distplot(data['Time'])<jupyter_output><empty_output><jupyter_text>Performing train test splitSetting the stratify parameter of train_test_split to y  This will ensure that y_train and y_test will have approximately the same distribution of labels (i.e. Fraud and Non-Fraud Transactions) as you can see in the output<jupyter_code>X = data.drop('Class', axis=1)
y = data['Class']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y)
    
print("Class Label Distribution for y_train: ")
print(y_train.value_counts(normalize=True))
print("Class Label Distribution for y_test: ")
print(y_test.value_counts(normalize=True))<jupyter_output>Class Label Distribution for y_train: 
0    0.998271
1    0.001729
Name: Class, dtype: float64
Class Label Distribution for y_test: 
0    0.99828
1    0.00172
Name: Class, dtype: float64<jupyter_text>Thus we get train and test splits with the same mean value for the response variable (Class)  While implementing a model on such a dataset is that instead of directly scaling the entire dataset using one of the scaler objects, the foll. steps should be performed:1. Perform train test split2. Fit the Scaler object on the train data3. Now scale the train as well as the test data on this Scaler that was fit on the  train data only. So now, scaling is done using the information obtained from the training data only thereby reducing the bias that would been incurred had we directly scaled the entire dataset at once.<jupyter_code>from sklearn.preprocessing import RobustScaler

# Using RobustScaler as it is less prone to outliers
rob_scaler = RobustScaler().fit(X_train)
X_train = pd.DataFrame(rob_scaler.transform(X_train), columns=X.columns)
X_test = pd.DataFrame(rob_scaler.transform(X_test), columns=X.columns)

X_train.head()
X_train.head()<jupyter_output><empty_output><jupyter_text>Hyperparameter Tuning of Logistic Regression Model using Random Search Basic Hyperparameter Tuning componentsObjective function: takes in hyperparameters and returns the cross validation score we want to maximize or minimizeDomain of hyperparameters: values over which we want to searchAlgorithm: method for selecting the next hyperparameter values to evaluate in the objective functionResults: history of hyperparameters and cross validation scores Grid SearchFor a given model, it takes in the domain of hyperparameter values and checks every possible combination to find which settings perform the best on an evaluation score specified. Very time consuming. Random SearchIt takes in the domain as well, but it randomly selects the set of hyperparameters from the search space. Results in more evenly spaced out hyperparameter values and gives similar results to Grid Search albeit a bit lower. Generally used on an unknown dataset in order to get rough estimate of the domain in which the model performs well and then Grid Search can be implemented to increase the accuracy further.  Logistic Regression Hyperparameters Parameters used for tuning are penalty and C only. Here is the docstring for the listed parameters1. penalty : str, 'l1', 'l2', 'elasticnet' or 'none', optional (default='l2')    Used to specify the norm used in the penalization. The 'newton-cg',    'sag' and 'lbfgs' solvers support only l2 penalties. 'elasticnet' is    only supported by the 'saga' solver. If 'none' (not supported by the    liblinear solver), no regularization is applied.    .. versionadded:: 0.19       l1 penalty with SAGA solver (allowing 'multinomial' + L1)2. C : float, optional (default=1.0)    Inverse of regularization strength; must be a positive float.    Like in support vector machines, smaller values specify stronger    regularization.3. solver : str, {'newton-cg', 'lbfgs', 'liblinear', 'sag', 'saga'},              optional (default='liblinear').    Algorithm to use in the optimization problem.    - For small datasets, 'liblinear' is a good choice, whereas 'sag' and      'saga' are faster for large ones.    - For multiclass problems, only 'newton-cg', 'sag', 'saga' and 'lbfgs'      handle multinomial loss; 'liblinear' is limited to one-versus-rest      schemes.    - 'newton-cg', 'lbfgs', 'sag' and 'saga' handle L2 or no penalty    - 'liblinear' and 'saga' also handle L1 penalty    - 'saga' also supports 'elasticnet' penalty    - 'liblinear' does not handle no penalty    Note that 'sag' and 'saga' fast convergence is only guaranteed on    features with approximately the same scale. You can    preprocess the data with a scaler from sklearn.preprocessing.    .. versionadded:: 0.17       Stochastic Average Gradient descent solver.    .. versionadded:: 0.19       SAGA solver.    .. versionchanged:: 0.20        Default will change from 'liblinear' to 'lbfgs' in 0.22.  Hyperparameter grid dictionary: penalty takes values l1 and l2 which are the only two options availableC is a list of 10 numbers containing the logspace from log10(0.001) to log10(1000)  Logspace is generally used for such parameters because we can cover a big range in a reasonable way in way fewer numbers as compared to something like a uniform distribution which is great for performing initial analysis and thus get a rough estimate of the best performing range for the parameter and then focus on it using other distributions. Setting up RandomizedSearchCV object We give the function,  the model for classification (LogisticRegression) the hyperparameter grid (log_reg_params)  n_jobs = Number of jobs to run in parallel (-1 to use all processors)  n_iter trades off runtime vs quality of the solution.<jupyter_code>from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import RandomizedSearchCV

# can also add the 'solver' parameter with values of liblinear and saga to the dictionary later 'solver': ['liblinear', 'saga']}

rand_log_reg_params = {"penalty": ['l1', 'l2'], 'C': list(np.logspace(np.log10(0.001), np.log10(1000), num=10))} 
# It contains the estimator (LogisticRegression in our case and also the dictionary of hyperparameters)
rand_log_reg = RandomizedSearchCV(LogisticRegression(), rand_log_reg_params, n_iter=50, verbose=5, n_jobs=-1)
rand_log_reg<jupyter_output><empty_output><jupyter_text>Cross Validation k-Fold Cross Validation Algorithm (used to prevent overfitting)1. Shuffle the dataset randomly.2. Split the dataset into k groups3. For each unique group:        3.1. Take the group as a hold out or test data set        3.2. Take the remaining groups as a training data set        3.3. Fit a model on the training set and evaluate it on the test set        3.4. Retain the evaluation score and discard the model    4. Summarize the skill of the model using the sample of model evaluation scores Stratified k-Fold Cross ValidationIn stratified k-fold cross-validation, the folds are selected so that "***the mean response value is approximately equal in all the folds***". In the case of a dichotomous classification, this means that each fold contains roughly the same proportions of the two types of class labels.<jupyter_code># Defining lists which will contain various metric values as cross-validation progresses
accuracy_lst_log_reg = []
precision_lst_log_reg = []
recall_lst_log_reg = []
f1_lst_log_reg = []
auc_lst_log_reg = []<jupyter_output><empty_output><jupyter_text>Handling imbalance: Undersampling vs Oversampling Undersampling Undersampling techniques remove examples from the training dataset that belong to the majority class in order to better balance the class distribution, such as reducing the skew from a 1:100 to a 1:10, 1:2, or even a 1:1 class distribution.  Oversampling This is different from oversampling that involves adding examples to the minority class in an effort to reduce the skew in the class distribution.Oversampling techniques are generally preferred over Undersampling techniques because they balance the dataset and at the same time do not result in loss of relevant data that Undersampling ends up doing. One of the primary reasons where undersampling comes into picture is when the data is very huge (containing sufficient minority examples) and minimizing cost of the project is a big factor.  SMOTE  (Reference)One of the most commonly used Oversampling techniques is  Synthetic Minority Oversampling Technique - SMOTE  SMOTE synthesises new minority instances between existing (real) minority instances.Imagine that SMOTE draws lines between existing minority instances like this.SMOTE then imagines new, synthetic minority instances somewhere on these lines.  Oversampling with Cross-Validation & Data LeakageYou have to perform Oversampling within the Cross-Validation itself and not before it on the training set only, otherwise it would lead to data leakage. This is because it is possible that by Oversampling before, you are pretty much duplicating data from one fold to another, which could end up being in the test fold, and due to this, the test data is not really unseen and it could lead to overfitting. Oversampling before Cross Validation (Wrong Way)  Oversampling during Cross Validation (Right Way)<jupyter_code>from sklearn.model_selection import StratifiedKFold
from imblearn.pipeline import make_pipeline as make_imbalanced_pipeline
from imblearn.over_sampling import SMOTE
from sklearn.externals import joblib

# Stratified 5 Fold

sss = StratifiedKFold(n_splits=5, random_state=None)

smote_model = SMOTE(sampling_strategy='minority', random_state=42)

# When we fit data on the pipeline, it is first oversampled using SMOTE and then it is fit on the LogisticRegression
# Model whose hyperparameters are stored in rand_log_reg
# Also, Oversampling happens within the Cross-Validation, as explained above
pipeline = make_imbalanced_pipeline(SMOTE(sampling_strategy='minority'), rand_log_reg)

# Takes a lot of time to execute
# train and test are a list of indices obtained from the StratifiedKFold().split() method
# We perform iloc on the X and y dataframes to get X_train, X_test and y_train, y_test
# and both y_test and y_train have same distribution of 0 and 1 class values, i.e, approx. 99.82% and 0.17% respectively
for train, test in sss.split(X_train, y_train):
    
    # Time consuming step
    model = pipeline.fit(X_train.iloc[train], y_train.iloc[train])
    best_est_rand = rand_log_reg.best_estimator_
        
    # Testing the best estimator (best_est_rand) on the best performing set of parameters (inherently done by predict method)
    prediction = best_est_rand.predict(X_train.iloc[test])
    
    # Appending metric scores to their respective list
    accuracy_lst_log_reg.append(pipeline.score(X_train.iloc[test], y_train.iloc[test]))
    precision_lst_log_reg.append(precision_score(y_train.iloc[test], prediction))
    recall_lst_log_reg.append(recall_score(y_train.iloc[test], prediction))
    f1_lst_log_reg.append(f1_score(y_train.iloc[test], prediction))
    auc_lst_log_reg.append(roc_auc_score(y_train.iloc[test], prediction))

print("Cross Validation Oversampling Training with Metric Calculations Completed")<jupyter_output>Fitting 5 folds for each of 20 candidates, totalling 100 fits<jupyter_text>Best C Parameters obtained were around 0.0046 - 50, so we'll focus on this range while implementing GridSearchBest penalty obtained was l2 in all cases so I've used that for GridSearch<jupyter_code>best_est_rand.get_params()<jupyter_output><empty_output><jupyter_text>Printing the mean of the metric measures recorded in the lists  Average Recall is 91%   Average ROC AUC is 94%<jupyter_code>print("accuracy:", np.mean(accuracy_lst_log_reg))
print("precision:", np.mean(precision_lst_log_reg))
print("recall:", np.mean(recall_lst_log_reg))
print("f1:", np.mean(f1_lst_log_reg))
print("roc:", np.mean(auc_lst_log_reg))<jupyter_output>accuracy: 0.9749303254405406
precision: 0.05917786515908787
recall: 0.9037000973709833
f1: 0.11104687598132393
roc: 0.9393770359590426<jupyter_text>On the actual test data X_test, we get Recall Score of 91% for Fraud Class<jupyter_code>labels = ['No Fraud', 'Fraud']
rand_log_reg_predictions = best_est_rand.predict(X_test)
print(classification_report_imbalanced(y_test, rand_log_reg_predictions, target_names=labels))<jupyter_output>pre       rec       spe        f1       geo       iba       sup

   No Fraud       1.00      0.98      0.93      0.99      0.95      0.91     56864
      Fraud       0.06      0.93      0.98      0.12      0.95      0.90        98

avg / total       1.00      0.98      0.93      0.99      0.95      0.91     56962<jupyter_text>Average Precision-Recall Score on test data is 74%<jupyter_code>from sklearn.metrics import average_precision_score

# In the case of logistic regression, the predict_proba function returns the value of "f(x)=1/1+e^(−(β0+β1x1+⋯+βkxk))"
# for given input X with k features

# The decision function returns the value of "d(x)=β0+β1x1+⋯+βkxk"
# The predict function returns 1 if value of predict_proba is > 0.5 else 0

y_score = best_est_rand.predict_proba(X_test)[:, 1]
average_precision = average_precision_score(y_test, y_score)

print('Average precision-recall score: {0:0.2f}'.format(average_precision))<jupyter_output>Average precision-recall score: 0.76<jupyter_text>Plotting the Precision-Recall Curve<jupyter_code>from sklearn.metrics import precision_recall_curve

precision, recall, _ = precision_recall_curve(y_test, y_score)

precision_recall_df = pd.DataFrame({'precision': precision, 'recall': recall})

plot_title = 'Precision-Recall Curve\nAvg. Precision Recall Score=' + str(average_precision)
plt.figure(figsize=(10, 6))
sns.lineplot(x='recall',y='precision',data=precision_recall_df)
plt.title(plot_title)
plt.fill_between(recall, precision, color='lightblue')<jupyter_output><empty_output><jupyter_text>Plotting the ROC Curve<jupyter_code>from sklearn.metrics import roc_curve


false_positive_rate, true_positive_rate, _ = roc_curve(y_test,  y_score)

auc = roc_auc_score(y_test, y_score)

roc_df = pd.DataFrame({'fpr': false_positive_rate, 'tpr': true_positive_rate})

plot_title = 'ROC Curve\nROC Score = ' + str(auc)

plt.figure(figsize=(10, 6))
sns.lineplot(x='fpr', y='tpr', data=roc_df)
plt.fill_between(roc_df['fpr'], roc_df['tpr'], color='lightblue')
plt.title(plot_title)<jupyter_output><empty_output><jupyter_text>Confusion Matrix<jupyter_code>from sklearn.metrics import confusion_matrix

confusion_matrix_log_reg = confusion_matrix(y_true=y_test, y_pred=rand_log_reg_predictions)

plt.figure(figsize=(10,6))
sns.heatmap(confusion_matrix_log_reg, annot=True, cmap='copper', fmt='d')<jupyter_output><empty_output><jupyter_text>We got 91 True Positive Cases and only 9 False Negative Cases on the actual test data!   Classification Report for Logistic Regression with RandomSearch<jupyter_code>print("Classification Report for RandomSearchCV")
print(classification_report_imbalanced(y_true=y_test, y_pred=rand_log_reg_predictions))<jupyter_output>Classification Report for RandomSearchCV
                   pre       rec       spe        f1       geo       iba       sup

          0       1.00      0.98      0.93      0.99      0.95      0.91     56864
          1       0.06      0.93      0.98      0.12      0.95      0.90        98

avg / total       1.00      0.98      0.93      0.99      0.95      0.91     56962<jupyter_text>Implementing Random Forest (no Hyperparameter Tuning)<jupyter_code>from sklearn.ensemble import RandomForestClassifier
from imblearn.over_sampling import SMOTE
from imblearn.pipeline import make_pipeline as make_imbalanced_pipeline

smote_model = SMOTE(sampling_strategy='minority', random_state=42)
random_forest_model = RandomForestClassifier(n_jobs=-1)

pipeline = make_imbalanced_pipeline(smote_model, random_forest_model)
pipeline.fit(X_train, y_train)
random_forest_predictions = random_forest_model.predict(X_test)<jupyter_output><empty_output><jupyter_text>Confusion Matrix<jupyter_code>from sklearn.metrics import confusion_matrix

confusion_matrix_random_forest = confusion_matrix(y_true=y_test, y_pred=random_forest_predictions)

plt.figure(figsize=(10,6))
sns.heatmap(confusion_matrix_random_forest, annot=True, cmap='copper', fmt='d')<jupyter_output><empty_output><jupyter_text>We got 86 True Positive Cases and 12 False Negative Cases on the actual test data. So a little worse than the Logistic Regression Model.   Classification Report for Random Forest Classifier<jupyter_code>from imblearn.metrics import classification_report_imbalanced

print(classification_report_imbalanced(y_true=y_test, y_pred=random_forest_predictions))<jupyter_output>pre       rec       spe        f1       geo       iba       sup

          0       1.00      1.00      0.88      1.00      0.94      0.89     56864
          1       0.90      0.88      1.00      0.89      0.94      0.87        98

avg / total       1.00      1.00      0.88      1.00      0.94      0.89     56962<jupyter_text>We got 88% recall for the Fraud Class    Implementing Naive Bayes Classifier (no Hyperparameter Tuning)<jupyter_code>from sklearn.naive_bayes import GaussianNB
from imblearn.over_sampling import SMOTE
from imblearn.pipeline import make_pipeline as make_imbalanced_pipeline

smote_model = SMOTE(sampling_strategy='minority', random_state=42)
naive_bayes_model = GaussianNB()

pipeline = make_imbalanced_pipeline(smote_model, naive_bayes_model)
pipeline.fit(X_train, y_train)
naive_bayes_predictions = pipeline.predict(X_test)<jupyter_output><empty_output><jupyter_text>Confusion Matrix<jupyter_code>from sklearn.metrics import confusion_matrix

confusion_matrix_random_forest = confusion_matrix(y_true=y_test, y_pred=naive_bayes_predictions)

plt.figure(figsize=(10,6))
sns.heatmap(confusion_matrix_random_forest, annot=True, cmap='copper', fmt='d')<jupyter_output><empty_output><jupyter_text>We got 91 True Positive Cases and 7 False Negative Cases on the actual test data. So it performed the same as Logistic Regression which even used Hyperparameter Tuning and extensive cross-validation!   Classification Report for Naive Bayes Classifier<jupyter_code>from imblearn.metrics import classification_report_imbalanced

print(classification_report_imbalanced(y_true=y_test, y_pred=naive_bayes_predictions))<jupyter_output>pre       rec       spe        f1       geo       iba       sup

          0       1.00      0.97      0.93      0.99      0.95      0.91     56864
          1       0.06      0.93      0.97      0.11      0.95      0.90        98

avg / total       1.00      0.97      0.93      0.98      0.95      0.91     56962
{'fraction_non_alphanumeric': 0.07166056606691756, 'fraction_numerical': 0.032221477548074445, 'mean_word_length': 4.727192205491585, 'pattern_counts': {'":': 1, '<': 84, '<?xml version=': 0, '>': 85, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 16, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '167573', 'license': 'MIT', 'n_tokens_mistral': 6298, 'n_tokens_neox': 5787, 'n_words': 2005, 'path': 'Credit Card Fraud Detection.ipynb', 'repo_name': 'Banking-and-Finance/Credit-Card-Fraud-Detection', 'size': '19397'}