<jupyter_start><jupyter_text>Credit Card Fraud Detection Model and EDA with Simple ML Algorithms  a. Install all the dependencies<jupyter_code>!pip install -r requirements1.txt<jupyter_output>Requirement already satisfied: pickleshare==0.7.5 in g:\anaconda_software\lib\site-packages (from -r requirements1.txt (line 1)) (0.7.5)
Requirement already satisfied: requests==2.22.0 in g:\anaconda_software\lib\site-packages (from -r requirements1.txt (line 2)) (2.22.0)
Requirement already satisfied: numpy==1.17.4 in g:\anaconda_software\lib\site-packages (from -r requirements1.txt (line 3)) (1.17.4)
Requirement already satisfied: pandas==0.25.3 in g:\anaconda_software\lib\site-packages (from -r requirements1.txt (line 4)) (0.25.3)
Requirement already satisfied: matplotlib==3.1.3 in g:\anaconda_software\lib\site-packages (from -r requirements1.txt (line 5)) (3.1.3)
Requirement already satisfied: sklearn==0.0 in g:\anaconda_software\lib\site-packages (from -r requirements1.txt (line 6)) (0.0)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in g:\anaconda_software\lib\site-packages (from requests==2.22.0->-r requirements1.txt (line 2)) (3.0.4)
Requirement already satisfied: idna<2[...]<jupyter_text>b. Import all the required Libraries and Packages<jupyter_code>import pandas as pd
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from sklearn.ensemble import RandomForestClassifier 
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import GridSearchCV, KFold
from sklearn.model_selection import cross_val_score
from sklearn.metrics import classification_report,accuracy_score,roc_curve,roc_auc_score
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import learning_curve, ShuffleSplit, train_test_split,cross_val_predict
import matplotlib.pyplot as plt
import pickle<jupyter_output><empty_output><jupyter_text>c. Pull the LFS elements (CSV file)<jupyter_code>#Install git lfs if absent
# Pull the LFS from git 
!git lfs pull<jupyter_output><empty_output><jupyter_text>d. Read the Credit Card CSV file<jupyter_code># Read the dataset
data = pd.read_csv("creditcard.csv")<jupyter_output><empty_output><jupyter_text>Dataset Info<jupyter_code>data.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 284807 entries, 0 to 284806
Data columns (total 31 columns):
Time      284807 non-null float64
V1        284807 non-null float64
V2        284807 non-null float64
V3        284807 non-null float64
V4        284807 non-null float64
V5        284807 non-null float64
V6        284807 non-null float64
V7        284807 non-null float64
V8        284807 non-null float64
V9        284807 non-null float64
V10       284807 non-null float64
V11       284807 non-null float64
V12       284807 non-null float64
V13       284807 non-null float64
V14       284807 non-null float64
V15       284807 non-null float64
V16       284807 non-null float64
V17       284807 non-null float64
V18       284807 non-null float64
V19       284807 non-null float64
V20       284807 non-null float64
V21       284807 non-null float64
V22       284807 non-null float64
V23       284807 non-null float64
V24       284807 non-null float64
V25       284807 non-null float64
V26  [...]<jupyter_text>e. Data Analysis  e.1 Removing Unwanted Columns<jupyter_code># Removing the time column since it would not help in model training
data.drop(["Time"],axis=1,inplace = True)<jupyter_output><empty_output><jupyter_text>e.2 Checking for Null Values<jupyter_code># Lets check the missing values in case any
data.isnull().sum()<jupyter_output><empty_output><jupyter_text>e.3 Correlation of columns w.r.t. every column<jupyter_code>def plot_corr(df):
    corr = df.corr()
    fig,ax = plt.subplots(figsize = (12,12))
    ax.matshow(corr)
    plt.xticks(range(len(corr.columns)),corr.columns)
    plt.yticks(range(len(corr.columns)),corr.columns)
    plt.show()
# Vizualizing correletaions by plotting correlation matrix
plot_corr(data)<jupyter_output><empty_output><jupyter_text>e.4 Correlation elements w.r.t the target Column<jupyter_code>corr = data.corr()
corr["Class"]<jupyter_output><empty_output><jupyter_text>e.5 Removing highy negative and highly Positive Correlated Columns<jupyter_code># These are features either are too negatively correlated with the target or too much positive with target(may surpress other features)

COL=["V3","V10","V12","V14","V16","V17","V18"]
data.drop(COL,axis=1,inplace=True)<jupyter_output><empty_output><jupyter_text>e.6 Scalling the values for better modelling (Normalizing)<jupyter_code># Lets normalise the amount column
# Need not to perform scaling of other features since all values have been scaled because of the PCA applied before the data was released

data['normAmount'] = StandardScaler().fit_transform(data['Amount'].values.reshape(-1, 1))
data = data.drop(['Amount'],axis=1)<jupyter_output><empty_output><jupyter_text>e.7 Checking Class Distribution<jupyter_code># Checking the ratio of classes with respect to each other 
data["Class"].value_counts()<jupyter_output><empty_output><jupyter_text>e.8 Equalizing the Class Distribution to avoid creating Skewed class model<jupyter_code># Since our classes are highly skewed we should make them equivalent in order to have a normal distribution of the classes.
# Lets shuffle the data before creating the subsamples

df = data.sample(frac=1)

# Equalizing no. of Fraud rows data with no of Non-Fraud data rows (492 rows each as in the dataset).
fraud_df = df.loc[df['Class'] == 1]
non_fraud_df = df.loc[df['Class'] == 0][:492]

# Concating equal no of Fraud data and Non-Fraud Data into a new dataframe
normal_distributed_df = pd.concat([fraud_df, non_fraud_df])

# Shuffle dataframe rows
new_df = normal_distributed_df.sample(frac=1, random_state=42)

new_df.head()<jupyter_output><empty_output><jupyter_text>e.9 Splitting the data (Features and Target) into Training and Testing sets<jupyter_code># Lets split the data into features (X) and target (Y)
features = new_df.drop(["Class"],axis =1)
target = new_df["Class"]


# Splitting the data into train and test
X_train,X_test, Y_train,Y_test = train_test_split(features, target, test_size = .2, random_state =5)<jupyter_output><empty_output><jupyter_text>f. GRID SEARCH CV Lets apply the grid search in order to obtain the best hyperparamters for our models<jupyter_code>class GridSearchModel:
    def __init__(self):
        print("GridSearchModel Class Initialized\n\n")
        
    def fit_log_reg(self,params,X,Y):
        """Logistic Regression"""
        grid_log_reg = GridSearchCV(LogisticRegression(),params)
        grid_log_reg.fit(X,Y)
        # Logistic Regression with the best parameters
        grid_log_reg_best_est = grid_log_reg.best_estimator_
        print("{}\n".format(grid_log_reg_best_est))
        return grid_log_reg_best_est
    
    def fit_knn(self,params,X,Y):
        """K-Nearest Neighbors"""
        grid_knears = GridSearchCV(KNeighborsClassifier(),params)
        grid_knears.fit(X,Y)
        # KNears best estimator
        grid_knears_best_est = grid_knears.best_estimator_
        print("{}\n".format(grid_knears_best_est))
        return grid_knears_best_est

    def fit_svc(self,params,X,Y):
        """Support Vector Classifier"""
        grid_svc = GridSearchCV(SVC(), params)
        grid_svc.fit(X,Y)
        # SVC best estimator
        grid_svc_best_est = grid_svc.best_estimator_
        print("{}\n".format(grid_svc_best_est))
        return grid_svc_best_est


    def fit_dec_tree(self,params,X,Y):
        """Decision Tree Classifier"""
        grid_tree = GridSearchCV(DecisionTreeClassifier(), params)
        grid_tree.fit(X,Y)
        # Decision Tree best estimator
        grid_tree_best_est = grid_tree.best_estimator_
        print("{}\n".format(grid_tree_best_est))
        return grid_tree_best_est<jupyter_output><empty_output><jupyter_text>g. Fitting Grid Search on our models<jupyter_code>#Instance of GridSearchModel
gridModel = GridSearchModel()

"""Logistic Regression"""
# Solver is liblinear since its supports both l1 and l2 penalities
log_reg_params = {"penalty": ['l1', 'l2'], 'C': [0.001, 0.01, 0.1, 1, 10, 100, 1000],"solver" :['liblinear']}
log_reg_best_est = gridModel.fit_log_reg(log_reg_params,X_train,Y_train)


"""K-Nearest Neighbors"""
knears_params = {"n_neighbors": list(range(2,5,1)), 'algorithm': ['auto', 'ball_tree', 'kd_tree', 'brute']}
knears_best_est = gridModel.fit_knn(knears_params,X_train,Y_train)


"""Support Vector Classifier"""
svc_params = {'C': [0.5, 0.7, 0.9, 1], 'kernel': ['rbf', 'poly', 'sigmoid', 'linear']}
svc_best_est = gridModel.fit_svc(svc_params,X_train,Y_train)


"""DecisionTree Classifier"""
tree_params = {"criterion": ["gini", "entropy"], "max_depth": list(range(2,4,1)),"min_samples_leaf": list(range(5,7,1))}
dec_tree_best_est = gridModel.fit_dec_tree(tree_params,X_train,Y_train)<jupyter_output>GridSearchModel Class Initialized


LogisticRegression(C=1, class_weight=None, dual=False, fit_intercept=True,
                   intercept_scaling=1, l1_ratio=None, max_iter=100,
                   multi_class='auto', n_jobs=None, penalty='l2',
                   random_state=None, solver='liblinear', tol=0.0001, verbose=0,
                   warm_start=False)

KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
                     metric_params=None, n_jobs=None, n_neighbors=3, p=2,
                     weights='uniform')

SVC(C=0.7, break_ties=False, cache_size=200, class_weight=None, coef0=0.0,
    decision_function_shape='ovr', degree=3, gamma='scale', kernel='linear',
    max_iter=-1, probability=False, random_state=None, shrinking=True,
    tol=0.001, verbose=False)

DecisionTreeClassifier(ccp_alpha=0.0, class_weight=None, criterion='gini',
                       max_depth=3, max_features=None, max_leaf_nodes=None,
                       min_impurity_decrea[...]<jupyter_text>h. Initializing K-Fold for training our modelsWe would be utilizing cross validation score in order to get a much better result<jupyter_code># Initializing K-Fold for training all specified models
k = KFold(n_splits = 5)

# Specified Classifiers
classifiers = {
    'Logistic Regression':log_reg_best_est,
    'Support Vector Machine' : svc_best_est,
    'Random Forest Classifier': RandomForestClassifier(),
    'Decision Tree Algorithm':dec_tree_best_est,
    'KNN':knears_best_est
}<jupyter_output><empty_output><jupyter_text>i. Model Class with utilities<jupyter_code>class ML_Models:
    def __init__(self):
        print("Initialized ML_Models class\n\n")

    def train(self,classifiers,train_data =(X_train,Y_train),val_data=(X_test,Y_test)):
        for i in classifiers:
            print('-'*100)
            print(i)
            print('-'*100)
            classifiers[i].fit(X_train,Y_train.values.ravel())
            prediction = classifiers[i].predict(X_test)
            print('Classification Report')        
            cr = classification_report(Y_test,prediction,output_dict=True)
            print(pd.DataFrame(cr).transpose())
            print('='*100)
            accuracy = accuracy_score(Y_test.values.ravel(),prediction)
            print('Accuracy Score :  ',accuracy)
            print('='*100)
            print('Cross Validation Score')
            cv=cross_val_score(classifiers[i],X_train,Y_train,cv=k,scoring='accuracy')
            print(cv.mean())
            print('-'*100)
            print("\n\n")

    def plot_learning_curve(self,estimator1,estimator2,estimator3,estimator4,X,y,ylim=None,cv=None,n_jobs=1,train_sizes=None):
        #Subplots for each eastimator
        f, ((ax1, ax2), (ax3, ax4)) = plt.subplots(2,2, figsize=(20,14), sharey=True)

        if ylim is not None:
            plt.ylim(*ylim)

        # First Estimator
        train_sizes, train_scores, test_scores = learning_curve(estimator1, X, y, cv=cv, n_jobs=n_jobs, train_sizes=train_sizes)
        train_scores_mean = np.mean(train_scores, axis=1)
        train_scores_std = np.std(train_scores, axis=1)
        test_scores_mean = np.mean(test_scores, axis=1)
        test_scores_std = np.std(test_scores, axis=1)
        ax1.fill_between(train_sizes, train_scores_mean - train_scores_std,
                        train_scores_mean + train_scores_std, alpha=0.1,
                        color="#ff9124")
        ax1.fill_between(train_sizes, test_scores_mean - test_scores_std,
                        test_scores_mean + test_scores_std, alpha=0.1, color="#2492ff")
        ax1.plot(train_sizes, train_scores_mean, 'o-', color="#ff9124",
                label="Training score")
        ax1.plot(train_sizes, test_scores_mean, 'o-', color="#2492ff",
                label="Cross-validation score")
        ax1.set_title("Logistic Regression Learning Curve", fontsize=14)
        ax1.set_xlabel('Training size (m)')
        ax1.set_ylabel('Score')
        ax1.grid(True)
        ax1.legend(loc="best")
        
        # Second Estimator 
        train_sizes, train_scores, test_scores = learning_curve(
            estimator2, X, y, cv=cv, n_jobs=n_jobs, train_sizes=train_sizes)
        train_scores_mean = np.mean(train_scores, axis=1)
        train_scores_std = np.std(train_scores, axis=1)
        test_scores_mean = np.mean(test_scores, axis=1)
        test_scores_std = np.std(test_scores, axis=1)
        ax2.fill_between(train_sizes, train_scores_mean - train_scores_std,
                        train_scores_mean + train_scores_std, alpha=0.1,
                        color="#ff9124")
        ax2.fill_between(train_sizes, test_scores_mean - test_scores_std,
                        test_scores_mean + test_scores_std, alpha=0.1, color="#2492ff")
        ax2.plot(train_sizes, train_scores_mean, 'o-', color="#ff9124",
                label="Training score")
        ax2.plot(train_sizes, test_scores_mean, 'o-', color="#2492ff",
                label="Cross-validation score")
        ax2.set_title("Knears Neighbors Learning Curve", fontsize=14)
        ax2.set_xlabel('Training size (m)')
        ax2.set_ylabel('Score')
        ax2.grid(True)
        ax2.legend(loc="best")
        
        # Third Estimator
        train_sizes, train_scores, test_scores = learning_curve(
            estimator3, X, y, cv=cv, n_jobs=n_jobs, train_sizes=train_sizes)
        train_scores_mean = np.mean(train_scores, axis=1)
        train_scores_std = np.std(train_scores, axis=1)
        test_scores_mean = np.mean(test_scores, axis=1)
        test_scores_std = np.std(test_scores, axis=1)
        ax3.fill_between(train_sizes, train_scores_mean - train_scores_std,
                        train_scores_mean + train_scores_std, alpha=0.1,
                        color="#ff9124")
        ax3.fill_between(train_sizes, test_scores_mean - test_scores_std,
                        test_scores_mean + test_scores_std, alpha=0.1, color="#2492ff")
        ax3.plot(train_sizes, train_scores_mean, 'o-', color="#ff9124",
                label="Training score")
        ax3.plot(train_sizes, test_scores_mean, 'o-', color="#2492ff",
                label="Cross-validation score")
        ax3.set_title("Support Vector Classifier \n Learning Curve", fontsize=14)
        ax3.set_xlabel('Training size (m)')
        ax3.set_ylabel('Score')
        ax3.grid(True)
        ax3.legend(loc="best")
        
        # Fourth Estimator
        train_sizes, train_scores, test_scores = learning_curve(
            estimator4, X, y, cv=cv, n_jobs=n_jobs, train_sizes=train_sizes)
        train_scores_mean = np.mean(train_scores, axis=1)
        train_scores_std = np.std(train_scores, axis=1)
        test_scores_mean = np.mean(test_scores, axis=1)
        test_scores_std = np.std(test_scores, axis=1)
        ax4.fill_between(train_sizes, train_scores_mean - train_scores_std,
                        train_scores_mean + train_scores_std, alpha=0.1,
                        color="#ff9124")
        ax4.fill_between(train_sizes, test_scores_mean - test_scores_std,
                        test_scores_mean + test_scores_std, alpha=0.1, color="#2492ff")
        ax4.plot(train_sizes, train_scores_mean, 'o-', color="#ff9124",
                label="Training score")
        ax4.plot(train_sizes, test_scores_mean, 'o-', color="#2492ff",
                label="Cross-validation score")
        ax4.set_title("Decision Tree Classifier \n Learning Curve", fontsize=14)
        ax4.set_xlabel('Training size (m)')
        ax4.set_ylabel('Score')
        ax4.grid(True)
        ax4.legend(loc="best")

    def roc_score(self,classifier,X,Y,cv,method):
        # Create a DataFrame with all the scores and the classifiers names.
        log_reg_pred = cross_val_predict(classifier["Logistic Regression"], X, Y, cv=cv,method =method) #method="decision_function"
        knears_pred = cross_val_predict(classifier["KNN"], X, Y, cv=cv,method =method)
        svc_pred = cross_val_predict(classifier["Support Vector Machine"], X, Y, cv=cv,method =method)  #method="decision_function"
        tree_pred = cross_val_predict(classifier["Decision Tree Algorithm"], X, Y, cv=cv,method =method)


        print('Logistic Regression: ', roc_auc_score(Y, log_reg_pred))
        print('KNears Neighbors: ', roc_auc_score(Y, knears_pred))
        print('Support Vector Classifier: ', roc_auc_score(Y, svc_pred))
        print('Decision Tree Classifier: ', roc_auc_score(Y, tree_pred))
    
    def pickleData(self,filename,data):
        pickle_out = open(filename,"wb")
        pickle.dump(data,pickle_out)
        pickle_out.close()<jupyter_output><empty_output><jupyter_text>j. Let's Train our models<jupyter_code>model = ML_Models()

model.train(classifiers,train_data =(X_train,Y_train),val_data=(X_test,Y_test))<jupyter_output>Initialized ML_Models class


----------------------------------------------------------------------------------------------------
Logistic Regression
----------------------------------------------------------------------------------------------------
Classification Report
              precision    recall  f1-score     support
0              0.932692  0.970000  0.950980  100.000000
1              0.967742  0.927835  0.947368   97.000000
accuracy       0.949239  0.949239  0.949239    0.949239
macro avg      0.950217  0.948918  0.949174  197.000000
weighted avg   0.949950  0.949239  0.949202  197.000000
====================================================================================================
Accuracy Score :   0.949238578680203
====================================================================================================
Cross Validation Score
0.9199225993711199
----------------------------------------------------------------------------------------------------



-----[...]<jupyter_text>k. Plotting Learning Curves of Trained Models<jupyter_code>cv = ShuffleSplit(n_splits=100, test_size=0.2, random_state=42)

model.plot_learning_curve(
    estimator1=classifiers["Logistic Regression"],
    estimator2=classifiers["KNN"],
    estimator3=classifiers["Support Vector Machine"],
    estimator4=classifiers["Decision Tree Algorithm"],
    X=X_train,
    y=Y_train,
    ylim=(0.87, 1.01),
    cv=cv,
    n_jobs=4,
    train_sizes=np.linspace(.1, 1.0, 5)
)<jupyter_output><empty_output><jupyter_text>Therefore we observe that KNN is a bit overfitted then the other models and Logistic Regression performs the best.  l. ROC_AUC_SCORE<jupyter_code>model.roc_score(classifiers,X=X_train,Y=Y_train,cv=5,method='predict')<jupyter_output>Logistic Regression:  0.9251323947300439
KNears Neighbors:  0.9137012399896667
Support Vector Classifier:  0.926398217514854
Decision Tree Classifier:  0.9150445621286489<jupyter_text>m. Saving Training data and Testing data<jupyter_code># Save Training and Testing data in pickle for Future Use
model.pickleData(filename="X_train.pickle",data=X_train)
model.pickleData(filename="X_test.pickle",data=X_test)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1315671306690615, 'fraction_numerical': 0.0450032430274909, 'mean_word_length': 3.6258942995615047, 'pattern_counts': {'":': 5, '<': 87, '<?xml version=': 0, '>': 87, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '22', 'dir': 'jupyter-structured-clean-dedup', 'id': '292534', 'license': 'Apache-2.0', 'n_tokens_mistral': 7262, 'n_tokens_neox': 6611, 'n_words': 1369, 'path': 'Credit Card Fraud Detection (Simple ML).ipynb', 'repo_name': 'Luciferxx/Credit_Card_Fraid_Detection', 'size': '20043'}