<jupyter_start><jupyter_text>Table of Contents- [Credit Card Fraud Detection](creditcard)- [Dataset and Preprocessing](dataset)    - [Feature Scaling](feature)    - [Exploratory Data Analysis](eda)    - [Correlation Matrices](corr)    - [Histograms](hist)    - [Boxplots](box)    - [Outlier Detection and Removal](outlier)- [Naive Predictor Performance](naive)- [Classifiers (Undersampling)](classifier)    - [Cross-Validation](crossval)    - [Learning Curve](learn)    - [AUC-ROC Curve](roc)- [Undersampling vs Oversampling](sampling)    - [Undersampling](under)    - [Oversampling(SMOTe Technique)](over)- [Neural Network testing Random undersampled data vs Oversampled data(SMOTe)](neuralnet)    - [Confusion Matrix](conf)    - [Testing on Random undersampled data](test_under)    - [Testing on Oversampled data(SMOTe Technique)](test_over)- [Conclusion](conc)   Credit Card Fraud DetectionCredit card fraud is a wide-ranging term for theft and fraud committed using or involving a payment card, such as a credit card or debit card, as a fraudulent source of funds in a transaction. The purpose may be to obtain goods without paying or to obtain unauthorized funds from an account. Credit card fraud is also an adjunct to identity theft.Although incidences of credit card fraud are limited to about 0.1% of all card transactions, they have resulted in huge financial losses as the fraudulent transactions have been large value transactions. It is important that credit card companies are able to recognize fraudulent credit card transactions so that customers are not charged for items that they did not purchase. What we need is an algorithm, which could classify a transaction as fraudulent or non-fraudulent. Doing so will benefit both the credit card companies and the customers who have to go through the ordeal.  The following are all the libraries that I would be using for the entire project.<jupyter_code>#Libraries used
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
#sns.set_style("darkgrid")

# Classifier Libraries
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
import collections


from imblearn.under_sampling import RandomUnderSampler
from sklearn.preprocessing import MinMaxScaler, RobustScaler
from sklearn.model_selection import StratifiedKFold, train_test_split, cross_val_score, GridSearchCV, RandomizedSearchCV
from sklearn.model_selection import learning_curve
from sklearn.model_selection import ShuffleSplit
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from imblearn.pipeline import make_pipeline as imbalanced_make_pipeline
from imblearn.over_sampling import SMOTE
from imblearn.under_sampling import NearMiss
from scipy.stats import norm
from sklearn.metrics import precision_score, recall_score, f1_score, roc_auc_score
from sklearn.model_selection import cross_val_predict
from sklearn.metrics import roc_auc_score, roc_curve
from sklearn.metrics import classification_report
from sklearn.metrics import accuracy_score

import keras
from keras.models import Sequential
from keras.layers import Activation
from keras.layers.core import Dense
from keras.optimizers import Adam
from keras.metrics import categorical_crossentropy

%matplotlib inline<jupyter_output>/Users/anshulsharma/anaconda/lib/python3.6/site-packages/sklearn/ensemble/weight_boosting.py:29: DeprecationWarning: numpy.core.umath_tests is an internal NumPy module and should not be imported. It will be removed in a future NumPy release.
  from numpy.core.umath_tests import inner1d
Using TensorFlow backend.
/Users/anshulsharma/anaconda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/Users/anshulsharma/anaconda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/Users/anshulsharma/anaconda/lib/python3.6/site-packages/tensorf[...]<jupyter_text>Dataset and PreprocessingThe dataset is provided by Kaggle and contains transactions made by credit cards in September 2013 by European cardholders. This dataset presents transactions that occurred in two days, where we have 492 frauds out of 284,807 transactions. The dataset is highly unbalanced, the positive class (frauds) account for 0.172% of all transactions.It contains only numerical input variables which are the result of a PCA transformation. Unfortunately, due to confidentiality issues, the original features and more background information about the data could not be provided. Features V1, V2, ... V28 are the principal components obtained with PCA, the only features which have not been transformed with PCA are 'Time' and 'Amount'. Feature 'Time' contains the seconds elapsed between each transaction and the first transaction in the dataset. The feature 'Amount' is the transaction Amount, this feature can be used for example-dependant cost-sensitive learning. Feature 'Class' is the response variable and it takes value 1 in case of fraud and 0 otherwise.<jupyter_code>df = pd.read_csv('creditcard.csv')
df.head()<jupyter_output><empty_output><jupyter_text>The following are the names of the columns of the dataset. In total there are 31 columns.<jupyter_code>df.columns<jupyter_output><empty_output><jupyter_text>On using info() function we can check if any of the columns contain null values. Looking at the output it can be seen that all the 31 columns have non-null values.<jupyter_code>df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 284807 entries, 0 to 284806
Data columns (total 31 columns):
Time      284807 non-null float64
V1        284807 non-null float64
V2        284807 non-null float64
V3        284807 non-null float64
V4        284807 non-null float64
V5        284807 non-null float64
V6        284807 non-null float64
V7        284807 non-null float64
V8        284807 non-null float64
V9        284807 non-null float64
V10       284807 non-null float64
V11       284807 non-null float64
V12       284807 non-null float64
V13       284807 non-null float64
V14       284807 non-null float64
V15       284807 non-null float64
V16       284807 non-null float64
V17       284807 non-null float64
V18       284807 non-null float64
V19       284807 non-null float64
V20       284807 non-null float64
V21       284807 non-null float64
V22       284807 non-null float64
V23       284807 non-null float64
V24       284807 non-null float64
V25       284807 non-null float64
V26  [...]<jupyter_text>The describe() function generates descriptive statistics that summarize the central tendency,dispersion and shape of a dataset's distribution, excluding ``NaN`` values.<jupyter_code>df.describe()<jupyter_output><empty_output><jupyter_text>> Since, the dataset contains no null or missing values and all the features seem to be scaled properly, there seems to be no need of any preprocessing. Only the features Time & Amount are required to be scaled. Let's analyse the number of Fraud vs Non-Fraud cases. It can be clearly seen from the image below that the number of non-fraudulent transactions clearly outnumber the number of fraudulent transactions.<jupyter_code>#Frauds vs. Non-Frauds
sns.countplot('Class', data=df)
plt.title('Number of No fraud cases vs. Fraud cases', fontsize = 15);
print('Percentage of No Frauds: {}%'.format(round(df.Class.value_counts()[0]/len(df) * 100.0,2)))
print('Percentage of Frauds: {}%'.format(round(df.Class.value_counts()[1]/len(df) * 100.0,2)))<jupyter_output>Percentage of No Frauds: 99.83%
Percentage of Frauds: 0.17%<jupyter_text>Below are the histogram distributions of the columns `Time` and `Amount` respectively.<jupyter_code>f, (ax1,ax2) = plt.subplots(1,2, figsize=(15,5))
sns.distplot(df.Time, ax=ax1)
ax1.set_title('Distribution of Time', fontsize=15)
sns.distplot(df.Amount, ax=ax2)
ax2.set_title('Distribution of Amount', fontsize=15)
plt.show()<jupyter_output>/Users/anshulsharma/anaconda/lib/python3.6/site-packages/scipy/stats/stats.py:1633: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  return np.add.reduce(sorted[indexer] * weights, axis=axis) / sumval<jupyter_text>Feature ScalingThrough the above dataset analysis it can be seen that all the columns are scaled except the `Amount` & `Time` features. Most of the machine learning algorithms use Eucledian distance between two data points in their computations, this is a problem. If left alone, these algorithms only take in the magnitude of features neglecting the units. The results would vary greatly between different units, 5kg and 5000gms. The features with high magnitudes will weigh in a lot more in the distance calculations than features with low magnitudes. To supress this effect, we need to bring all features to the same level of magnitudes. This can be acheived by scaling.<jupyter_code>#Feature Scaling
rob_scaler = RobustScaler()
df['scaled_time'] = rob_scaler.fit_transform(df['Time'].values.reshape(-1,1))
df['scaled_amount'] = rob_scaler.fit_transform(df['Amount'].values.reshape(-1,1))
df.drop(['Time','Amount'], axis=1, inplace=True)
scaled_amount = df['scaled_amount']
scaled_time = df['scaled_time']

#scaled_amount and scaled_time are added to the starting of the dataframe
df.drop(['scaled_amount', 'scaled_time'], axis=1, inplace=True)
df.insert(0, 'scaled_amount', scaled_amount)
df.insert(1, 'scaled_time', scaled_time)

df.head()<jupyter_output><empty_output><jupyter_text>Exploratory Data AnalysisNow I will perform EDA on the dataset to study the various features of it. Since, it is an imbalanced dataset, I would first undersample the majority class(i.e. Non-Fraud in this case) so that I can get an equal distribution of Fraud and Non-Fraud cases. For undersampling I have used Random under sampling before crossvalidation. Although it is prone to overfit , but for EDA it can be performed.<jupyter_code>#Random Under Sampling
X = df.drop('Class', axis=1)
y = df['Class']
rus = RandomUnderSampler(random_state=42)
X_rs, y_rs = rus.fit_sample(X,y)
#Random Under-sampled Dataframe
df_rs = pd.DataFrame(np.hstack((X_rs,y_rs[:, None])), columns=df.columns)
df_rs.Class = df_rs.Class.astype(int)
df_rs.head()<jupyter_output><empty_output><jupyter_text>Now if we see the number of fraud vs non-fraud cases are equally distributed.<jupyter_code>#Frauds vs. Non-Frauds for the new dataframe
sns.countplot('Class', data=df_rs)
plt.title('Number of No fraud cases vs. Fraud cases', fontsize = 15);
print('Percentage of No Frauds: {}%'.format(round(df_rs.Class.value_counts()[0]/len(df_rs) * 100.0,2)))
print('Percentage of Frauds: {}%'.format(round(df_rs.Class.value_counts()[1]/len(df_rs) * 100.0,2)))<jupyter_output>Percentage of No Frauds: 50.0%
Percentage of Frauds: 50.0%<jupyter_text>Correlation MatricesIn order to get the essence of the data I'll be plotting correlation matrices. Since, there are 31 features it seems to be a better option to go for a heatmap kind of correlation matrix where different colours would indicate the correlation between any two features of the dataset. My main aim is to how the different features of the dataset help in determining a fraudulent or a non-fraudulent transaction.  Heatmap 1:This heatmap is built on top of the original dataset. The original dataset is an imbalanced dataset with 99.83:0.17 ratio of Non-fraudulant is to Fraudulant transactions respectively. Hence, it is not a good indicator of what features influence a transaction to be fraudulant. The same can be seen through the heapmap as well as majority of the features show no correlation at all.   Heatmap 2:In order to get a better view of what features influence a transaction to be fraudulant, it would be best to plot a heatmap on top of the randomly undersampled dataset where the class distribution(fraud vs non-fraud) is equal. As can be seen through the figure this heatmap shows a lot of correlation between the various features. Hence, this graph can prove to be handy in studying the features.<jupyter_code>fig , (ax1,ax2) = plt.subplots(2,1, figsize=(24,20))
sns.heatmap(df.corr(), cmap='YlOrRd', annot_kws={'size': 20}, ax=ax1)
sns.heatmap(df_rs.corr(), cmap='YlOrRd', annot_kws={'size': 20}, ax=ax2)
plt.show()<jupyter_output><empty_output><jupyter_text>My main aim is to find the features which influence a transaction to be fraudulant. In order to achieve this, I will focus on the last column of the plot where the correlation between various features and the Class is computed. The following is my analysis:-- **Positive Correlation:** Features *V2, V4, V11 & V19* show positive correlation with class. The higher the values of these features, the higher the chances of a transaction being fraudulant.- **Negative Correlation:** Features *V10, V12, V14 & V17* show negative correlation with class. The lower the values of these features, the higher the chances of a transaction being fraudulant.  HistrogramsBelow I have plotted the histograms of the features listed above. The first row of histograms shows the distribution of positively correlated features with Class: *V2, V4, V11 & V19*, while the second row shows the distribution of negatively correlated features with Class: *V10, V12, V14 & V17*. I have also fitted each of the plots with normal distribution to judge how close the feature distribution is to a Gaussian distribution.> All the feature distributions are for fraudulant transactions i.e Class = 1.<jupyter_code>rows, cols = 2,4
fig, axes = plt.subplots(nrows=rows, ncols=cols, figsize=(20,8))
features = {0:['V2','V4','V11','V19'],1:['V10','V12','V14','V17']}
for key, value in features.items():
    for ind, feature in enumerate(value):
        sns.distplot(df[feature].loc[df['Class'] == 1].values, fit=norm, ax=axes[key][ind])
        axes[key][ind].set_title('{} Distribution'.format(feature))
plt.suptitle('Feature Distributions for Fraud Transactions', fontsize=15)
plt.show()<jupyter_output>/Users/anshulsharma/anaconda/lib/python3.6/site-packages/scipy/stats/stats.py:1633: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  return np.add.reduce(sorted[indexer] * weights, axis=axis) / sumval<jupyter_text>BoxplotsA boxplot is a standardized way of displaying the distribution of data based on a five number summary (“minimum”, first quartile (Q1), median, third quartile (Q3), and “maximum”). It can tell you about your outliers and what their values are. It can also tell you if your data is symmetrical, how tightly your data is grouped, and if and how your data is skewed.You need to have information on the variability or dispersion of the data. A boxplot is a graph that gives you a good indication of how the values in the data are spread out. Although box lots may seem primitive in comparison to a histogram or density plot, they have the advantage of taking up less space, which is useful when comparing distributions between many groups or datasets.Below I have plotted boxplots for all the features which show positive and negative correlations with Class. The boxplot representation for each feature is shown separately for each of the class(0 & 1).<jupyter_code>#Box-Plots for Positive Correlation
fig, axes = plt.subplots(ncols=4, figsize=(20,4))
sns.boxplot(x='Class', y='V2', data=df_rs, ax=axes[0])
axes[0].set_title('V2 vs Class - Positive Correlation')
sns.boxplot(x='Class', y='V4', data=df_rs, ax=axes[1])
axes[1].set_title('V4 vs Class - Positive Correlation')
sns.boxplot(x='Class', y='V11', data=df_rs, ax=axes[2])
axes[2].set_title('V11 vs Class - Positive Correlation')
sns.boxplot(x='Class', y='V19', data=df_rs, ax=axes[3])
axes[3].set_title('V19 vs Class - Positive Correlation')
plt.show()
#Box-Plots for Negative Correlation
fig, axes = plt.subplots(ncols=4, figsize=(20,4))
sns.boxplot(x='Class', y='V10', data=df_rs, ax=axes[0])
axes[0].set_title('V10 vs Class - Negative Correlation')
sns.boxplot(x='Class', y='V12', data=df_rs, ax=axes[1])
axes[1].set_title('V12 vs Class - Negative Correlation')
sns.boxplot(x='Class', y='V14', data=df_rs, ax=axes[2])
axes[2].set_title('V14 vs Class - Negative Correlation')
sns.boxplot(x='Class', y='V17', data=df_rs, ax=axes[3])
axes[3].set_title('V17 vs Class - Negative Correlation')
plt.show()<jupyter_output><empty_output><jupyter_text>Outlier Detection and RemovalBoxplots are a standardized way of displaying the distribution of data based on a five number summary (“minimum”, first quartile (Q1), median, third quartile (Q3), and “maximum”).- **median (Q2/50th Percentile)**: the middle value of the dataset.- **first quartile (Q1/25th Percentile)**: the middle number between the smallest number (not the “minimum”) and the median of the dataset.- **third quartile (Q3/75th Percentile)**: the middle value between the median and the highest value (not the “maximum”) of the dataset.- **interquartile range (IQR)**: 25th to the 75th percentile.- **maximum**: Q3 + 1.5*IQR- **minimum**: Q1 -1.5*IQR- **whiskers**(shown in blue)- **outliers**(shown as green circles): Anything below the minimum value and above the maximum value is an outlierBelow I have deviced a method, which will detect the outliers and then remove them from the dataset.<jupyter_code>#Outlier Removal
def outlier_removal(df,feature, fraud):
    array = df[feature].loc[df['Class'] == fraud].values 
    q25, q75 = np.percentile(array, 25), np.percentile(array,75)
    print('25th percentile: {} | 75th percentile: {}'.format(q25,q75))
    iqr = q75 - q25
    print('Interquartile Range: {}'.format(iqr))
    cutoff = iqr*1.5
    lower_threshold, upper_threshold = q25 - cutoff, q75 + cutoff
    print('Cutoff: {}'.format(cutoff))
    print('Lower Threshold: {} | Upper Threshold: {}'.format(lower_threshold, upper_threshold))
    outliers = [a for a in array if a < lower_threshold or a > upper_threshold]
    print('{} Outliers: {}'.format(feature,outliers))
    print('Number of outliers detected for feature {}: {}'.format(feature,len(outliers)))
    df = df.drop(df[(df[feature] > upper_threshold) | (df[feature] < lower_threshold)].index)
    print('Number of records after outlier removal: {}'.format(len(df)))
    print('-'*117)
    return df<jupyter_output><empty_output><jupyter_text>I will try to remove the outliers for features *V10* and *V2* as through the above boxplots we can see that these have the maximum number of outliers. The code will show the outliers detected and their total number and then finally remove all the outliers.<jupyter_code>#Removal of outliers for features V10 and V2. Since, these two features seem to have the most number of outliers.
df_rs_out = outlier_removal(df_rs,'V10', 1)
df_rs_out = outlier_removal(df_rs_out,'V2',1)<jupyter_output>25th percentile: -7.7566982278474725 | 75th percentile: -2.6141839131895623
Interquartile Range: 5.142514314657911
Cutoff: 7.713771471986866
Lower Threshold: -15.47046969983434 | Upper Threshold: 5.099587558797303
V10 Outliers: [-16.6496281595399, -18.2711681738888, -15.563791338730098, -15.563791338730098, -16.7460441053944, -22.1870885620007, -22.1870885620007, -22.1870885620007, -22.1870885620007, -17.141513641289198, -16.3035376590131, -16.2556117491401, -16.6011969664137, -18.9132433348732, -20.949191554361104, -19.836148851696, -23.2282548357516, -24.403184969972802, -24.5882624372475]
Number of outliers detected for feature V10: 19
Number of records after outlier removal: 963
---------------------------------------------------------------------------------------------------------------------
25th percentile: 1.1453810871791001 | 75th percentile: 4.48826730168204
Interquartile Range: 3.34288621450294
Cutoff: 5.01432932175441
Lower Threshold: -3.86894823457531 | Upper Threshold: 9[...]<jupyter_text>Below we can see the boxplots representing the before and after state of the outliers for features *V10* and *V2*. It is clearly evident that the number of outliers have reduced after going through outlier removal.<jupyter_code>fig, axes = plt.subplots(2,2, figsize=(20,14))
plt.suptitle('Comparison of Box Plots after outlier removal - V10 & V2', fontsize=15)
sns.boxplot(x='Class', y='V10',data=df_rs, ax=axes[0][0])
axes[0][0].set_title('V10 vs Class \n (Before Outlier Removal)', fontsize=15)
sns.boxplot(x='Class', y='V2',data=df_rs, ax=axes[0][1])
axes[0][1].set_title('V2 vs Class \n (Before Outlier Removal)', fontsize=15)
sns.boxplot(x='Class', y='V10',data=df_rs_out, ax=axes[1][0])
axes[1][0].set_title('V10 vs Class \n (After Outlier Removal)', fontsize=15)
sns.boxplot(x='Class', y='V2',data=df_rs_out, ax=axes[1][1])
axes[1][1].set_title('V2 vs Class \n (After Outlier Removal)', fontsize=15)
sns.despine()
plt.show()<jupyter_output><empty_output><jupyter_text>Naive Predictor PerformanceMy Naive Predictor will be a model which predicts all the transactions as Non-Fraudulant. The following will be the definitions for the model:-- "Fraud(Class = 1)" is a **negative class**.- "Non-Fraud(Class = 0)" is a **Positive class**.** Please note ** that the the purpose of generating a naive predictor is simply to show what a base model without any intelligence would look like. In the real world, ideally your base model would be either the results of a previous model or could be based on a research paper upon which you are looking to improve. When there is no benchmark model set, getting a result better than random choice is a place you could start from.> *When we have a model that always predicts '0' (i.e. the transaction is non-fraudulant) then our model will have no **True Negatives(TN)** or **False Negatives(FN)** as we are not making any negative(Fraud '1' value) predictions. Therefore our Accuracy in this case becomes the same as our Precision(True Positives/(True Positives + False Positives)) as every prediction that we have made with value '0' that should have '1' becomes a False Positive; therefore our denominator in this case is the total number of records we have in total.Our Recall score(True Positives/(True Positives + False Negatives)) in this setting becomes 1 as we have no False Negatives.*The metrics that I'll be using are:  *Accuracy, Precision, Recall & f1-score*<jupyter_code>def naive_predictor(df):
    TP = df.Class.count() - np.sum(df.Class)
    FP = np.sum(df.Class)
    TN = 0
    FN = 0
    
    # TODO: Calculate accuracy, precision and recall
    accuracy = (TP+TN)/(TP+TN+FP+FN)
    recall = TP/(TP+FN)
    precision = TP/(TP+FP)
    
    # TODO: Calculate F-score using the formula above for beta = 1 and correct values for precision and recall.
    fscore = (2*precision*recall)/(precision + recall)
    
    # Print the results 
    print("[Accuracy score: {:.4f}, precision: {:.4f}, recall: {:.4f}, f1-score: {:.4f}]".format(accuracy, precision, recall, fscore))
    
print('Naive predictor for original dataset:')
naive_predictor(df)
print('-'*117)
print('Naive predictor for undersampled dataset:')
naive_predictor(df_rs)<jupyter_output>Naive predictor for original dataset:
[Accuracy score: 0.9983, precision: 0.9983, recall: 1.0000, f1-score: 0.9991]
---------------------------------------------------------------------------------------------------------------------
Naive predictor for undersampled dataset:
[Accuracy score: 0.5000, precision: 0.5000, recall: 1.0000, f1-score: 0.6667]<jupyter_text>Classifiers (Undersampling)In this section I'll be using 4 different classifiers to classify the transactions as Fraudulant or Non-Fraudulant in the randomly undersampled dataset. My aim is to compare the performance of the Naive predictor(Benchmark model) with that of the classifiers that I choose. The following are the classifiers that I'll be using:-- **Logistic Regression**- **Support Vector Classifier**- **Decision Tree**- **KNearest Classifier****Please Note** the randomly undersampled dataset used for the analysis is undersampled before cross validation and hence, is prove to overfit. To get the best model undersampling should be done along with cross validation.<jupyter_code># Undersampling before cross validation(prove to overfit)
X = df_rs.drop('Class', axis=1)
y = df_rs['Class']<jupyter_output><empty_output><jupyter_text>I'll split the data using a train-test split of 0.8:0.2 meaning training set has 80% of the records and the rest 20% in the testing set.<jupyter_code>#Split the data using train-test split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)<jupyter_output><empty_output><jupyter_text>Below I have added the classifiers to a dictionary for making the coding easier. If any changes are required to the classifiers then this is the only place that needs to be changed.<jupyter_code>#Convert to arrays to feed to the classifications algorithms
classifiers = {
    "Logistic Regression": LogisticRegression(),
    "Support Vector Classifier": SVC(),
    "Decision Tree": DecisionTreeClassifier(),
    "KNearest": KNeighborsClassifier()
}<jupyter_output><empty_output><jupyter_text>Cross-ValidationLearning the parameters of a prediction function and testing it on the same data is a methodological mistake: a model that would just repeat the labels of the samples that it has just seen would have a perfect score but would fail to predict anything useful on yet-unseen data. This situation is called overfitting. To avoid it, it is common practice when performing a (supervised) machine learning experiment to hold out part of the available data as a test set X_test, y_test. Note that the word “experiment” is not intended to denote academic use only, because even in commercial settings machine learning usually starts out experimentally. Here is a flowchart of typical cross validation workflow in model training. The best parameters can be determined by grid search techniques.Below I have calculated the cross validation score for the 4 classifiers. *Logistic Regression and Support Vector Classifer* produce the best score.<jupyter_code>#Calculate the cross-validation score for each classifier
print('Cross-Validation Scores:-')
for key, classifier in classifiers.items():
    classifier.fit(X_train, y_train)
    cv_score = cross_val_score(classifier, X_train, y_train, cv=5)
    print('{}: {}'.format(key,round(cv_score.mean()*100.0, 2)))<jupyter_output>Cross-Validation Scores:-
Logistic Regression: 94.28
Support Vector Classifier: 94.03
Decision Tree: 89.71
KNearest: 93.52<jupyter_text>A **model hyperparameter** is a characteristic of a model that is external to the model and whose value cannot be estimated from data. The value of the hyperparameter has to be set before the learning process begins. For example, c in Support Vector Machines, k in k-Nearest Neighbors, the number of hidden layers in Neural Networks.In contrast, a **parameter** is an internal characteristic of the model and its value can be estimated from data. Example, beta coefficients of linear/logistic regression or support vectors in Support Vector Machines.Below are the hyperparameters for the classifiers:-<jupyter_code>params = {
    "Logistic Regression": {"penalty": ['l1', 'l2'], 'C': [0.001, 0.01, 0.1, 1, 10, 100, 1000]},
    "Support Vector Classifier": {'C': [0.5, 0.7, 0.9, 1], 'kernel': ['rbf', 'poly', 'sigmoid', 'linear']},
    "Decision Tree": {"criterion": ["gini", "entropy"], "max_depth": list(range(2,4,1)), 
              "min_samples_leaf": list(range(5,7,1))},
    "KNearest": {"n_neighbors": list(range(2,5,1)), 'algorithm': ['auto', 'ball_tree', 'kd_tree', 'brute']}
}<jupyter_output><empty_output><jupyter_text>**Grid-search** is used to find the optimal hyperparameters of a model which results in the most ‘accurate’ predictions. I have performed grid search on the listed classifiers below. Logistic Regression classifer still produces the best score out of the 4 classifiers.<jupyter_code>#Using Grid search for finding the most optimal hyperparameters
def gridsearch(classifier, params):
    grid_classifier = GridSearchCV(classifier, params)
    grid_classifier.fit(X_train, y_train)
    best_classifier = grid_classifier.best_estimator_
    return best_classifier
#Scores after applying Grid Search
print('Cross-Validation Scores after applying GridSearch:-')
for key, classifier in classifiers.items():
    classifier = gridsearch(classifier,params[key])
    cv_score = cross_val_score(classifier, X_train, y_train, cv=3)
    print('{}: {}'.format(key,round(cv_score.mean()*100.0, 2)))<jupyter_output>Cross-Validation Scores after applying GridSearch:-
Logistic Regression: 95.04
Support Vector Classifier: 94.92
Decision Tree: 92.63
KNearest: 94.66<jupyter_text>Learning CurveA learning curve refers to a plot of the prediction accuracy/error vs. the training set size (i.e: how better does the model get at predicting the target as you the increase number of instances used to train it). Usually both the training and test/validation performance are plotted together so we can diagnose the **bias-variance tradeoff** (i.e determine if we benefit from adding more training data, and assess the model complexity by controlling regularization or number of features).The above plots show the following cases(These graphs are better understood when seen upside down as we are plotting the error function against number of instances):-- `Plot 1` : This is a case of **Underfitting(High Bias)** as it has very high training and cross-validation error meaning that the model performs badly on both training and cross-validation set.- `Plot 2` : This is the **Ideal** case as it has low training and cross-validation errors and they seem to converge as the number of instances increase. We strive to find such model for our predictions.- `Plot 3` : This is a case of **Overfitting(High Variance)** as it has very low training error and pretty high cross-validation error meaning that the model has learned each and every detail of the training data that if anything different fed to it then it won't be able to perform well. And the the training and cross-validation error lines don't seem to converge at any point of time. Below I have created a function *plot_leaning_curve()* which takes in different input parameters like the estimator, title, e.t.c and plots the learning curve for that estimator.<jupyter_code>def plot_learning_curve(estimator, title, X, y, ylim=None, cv=None,
                        n_jobs=None, train_sizes=np.linspace(.1, 1.0, 5)):
    #plt.figure()
    plt.title(title)
    if ylim is not None:
        ax.ylim(*ylim)
    plt.xlabel("Training examples")
    plt.ylabel("Score")
    train_sizes, train_scores, test_scores = learning_curve(
        estimator, X, y, cv=cv, n_jobs=n_jobs, train_sizes=train_sizes)
    train_scores_mean = np.mean(train_scores, axis=1)
    train_scores_std = np.std(train_scores, axis=1)
    test_scores_mean = np.mean(test_scores, axis=1)
    test_scores_std = np.std(test_scores, axis=1)
    plt.grid()

    plt.fill_between(train_sizes, train_scores_mean - train_scores_std,
                     train_scores_mean + train_scores_std, alpha=0.1,
                     color="r")
    plt.fill_between(train_sizes, test_scores_mean - test_scores_std,
                     test_scores_mean + test_scores_std, alpha=0.1, color="g")
    plt.plot(train_sizes, train_scores_mean, 'o-', color="r",
             label="Training score")
    plt.plot(train_sizes, test_scores_mean, 'o-', color="g",
             label="Cross-validation score")

    plt.legend(loc="best")
    return plt<jupyter_output><empty_output><jupyter_text>I will be using the classifers with the hyperparameters that came out as a result of grid search as these are the most optimal models.<jupyter_code>#Variables
n_jobs = 10
cv = ShuffleSplit(n_splits=100, test_size=0.2, random_state=42)

#Classifiers
log_reg = gridsearch(LogisticRegression(), params['Logistic Regression'])
svc = gridsearch(SVC(), params['Support Vector Classifier'])
decision_tree = gridsearch(DecisionTreeClassifier(), params['Decision Tree'])
knearest = gridsearch(KNeighborsClassifier(), params['KNearest'])

plt.figure(figsize=(15,15))
plt.subplot(2,2,1)
plot_learning_curve(log_reg, 'Logistic Regression', X_train, y_train, n_jobs=10)
plt.subplot(2,2,2)
plot_learning_curve(svc, 'Support Vector Classifier', X_train, y_train, n_jobs=10)
plt.subplot(2,2,3)
plot_learning_curve(decision_tree, 'Decision Tree Classifier', X_train, y_train, n_jobs=10)
plt.subplot(2,2,4)
plot_learning_curve(knearest, 'KNearest Classifier', X_train, y_train, n_jobs=10)
plt.show()<jupyter_output>/Users/anshulsharma/anaconda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:1103: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  if np.issubdtype(train_sizes_abs.dtype, np.float):
/Users/anshulsharma/anaconda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:1103: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  if np.issubdtype(train_sizes_abs.dtype, np.float):
/Users/anshulsharma/anaconda/lib/python3.6/site-packages/sklearn/model_selection/_validation.py:1103: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  if np.issubdtype(train_sizes_abs.dtype, np.float)[...]<jupyter_text>As can be seen through the learning curves above, **Logistic Regression** and **Support Vector Classifier** show the best score in training and cross-validation set and are pretty close to the ideal case.  AUC-ROC CurveAUC - ROC curve is a performance measurement for classification problem at various thresholds settings. ROC is a probability curve and AUC represents degree or measure of separability. It tells how much model is capable of distinguishing between classes. Higher the AUC, better the model is at predicting 0s as 0s and 1s as 1s. By analogy, Higher the AUC, better the model is at distinguishing between patients with disease and no disease.The ROC curve is plotted with TPR against the FPR where TPR is on y-axis and FPR is on the x-axis.>Here TP, FP, TN & FN stand for True Positive, False Positive, True Negative & False Negative. *TPR (True Positive Rate) / Recall /Sensitivity* *Specificity* *FPR(False Positive Rate)*An excellent model has AUC near to the 1 which means it has good measure of separability. A poor model has AUC near to the 0 which means it has worst measure of separability. In fact it means it is reciprocating the result. It is predicting 0s as 1s and 1s as 0s. And when AUC is 0.5, it means model has no class separation capacity whatsoever.<jupyter_code>#ROC Curve
'''log_pred = cross_val_predict(log_reg, X_train, y_train, cv=5)
svc_pred = cross_val_predict(svc, X_train, y_train, cv=5)
tree_pred = cross_val_predict(decision_tree, X_train, y_train, cv=5)
knear_pred = cross_val_predict(knearest, X_train, y_train, cv=5)'''

log_pred = log_reg.predict(X_test)
svc_pred = svc.predict(X_test)
tree_pred = decision_tree.predict(X_test)
knear_pred = knearest.predict(X_test)

log_fpr, log_tpr, log_threshold = roc_curve(y_test, log_pred)
svc_fpr, svc_tpr, svc_threshold = roc_curve(y_test, svc_pred)
tree_fpr, tree_tpr, tree_threshold = roc_curve(y_test, tree_pred)
knear_fpr, knear_tpr, knear_threshold = roc_curve(y_test, knear_pred)

plt.figure(figsize=(16,8))
plt.plot(log_fpr, log_tpr, label='Logistic Regression Classifier Score: {:.4f}'.format(roc_auc_score(y_test, log_pred)))
plt.plot(svc_fpr, svc_tpr, label='Support Vector Classifier Score: {:.4f}'.format(roc_auc_score(y_test, svc_pred)))
plt.plot(tree_fpr, tree_tpr, label='Decision Tree Classifier Score: {:.4f}'.format(roc_auc_score(y_test, tree_pred)))
plt.plot(knear_fpr, knear_tpr, label='K-Nearest Neighbors Classifier Score: {:.4f}'.format(roc_auc_score(y_test, knear_pred)))
plt.plot([0,1],[0,1],'k--')
plt.axis([-0.01, 1, 0, 1])
plt.title('ROC Curve\n(All Classifiers)', fontsize=20)
plt.xlabel('False Positive Rate', fontsize=14)
plt.ylabel('True Poitive Rate', fontsize=14)
plt.annotate('Minimum ROC Score of 50% \n (This is the minimum score to get)', xy=(0.5, 0.5), xytext=(0.6, 0.3),
                arrowprops=dict(facecolor='#6E726D', shrink=0.05),
                )
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>From the above curve it can be seen that the **KNearest Neighbors Classifier** has the highest area under the ROC curve and hence, seems to be the best model according to this metric. Now, I will generate the Classification report for all the 4 Classifers so that we can judge which one is the best of the lot in terms of the metrics: *precision, recall, f1-score and support*. The Classification report will show all these metrics for both the classes(Fraud and Non-Fraud)<jupyter_code>labels = ['No Fraud', 'Fraud']
print('Logistic Regression:')
print(classification_report(y_test, log_pred, target_names=labels))
print('Support Vector Classifier:')
print(classification_report(y_test, svc_pred, target_names=labels))
print('Decision Tree:')
print(classification_report(y_test, tree_pred, target_names=labels))
print('KNearest Neighbours:')
print(classification_report(y_test, knear_pred, target_names=labels))<jupyter_output>Logistic Regression:
             precision    recall  f1-score   support

   No Fraud       0.90      0.97      0.93        99
      Fraud       0.97      0.89      0.93        98

avg / total       0.93      0.93      0.93       197

Support Vector Classifier:
             precision    recall  f1-score   support

   No Fraud       0.90      0.96      0.93        99
      Fraud       0.96      0.90      0.93        98

avg / total       0.93      0.93      0.93       197

Decision Tree:
             precision    recall  f1-score   support

   No Fraud       0.85      0.95      0.90        99
      Fraud       0.94      0.84      0.89        98

avg / total       0.90      0.89      0.89       197

KNearest Neighbours:
             precision    recall  f1-score   support

   No Fraud       0.91      0.97      0.94        99
      Fraud       0.97      0.90      0.93        98

avg / total       0.94      0.93      0.93       197<jupyter_text>Undersampling vs Oversampling<jupyter_code>original_X = df.drop('Class',axis=1)
original_y = df['Class']

sss = StratifiedKFold(n_splits=5, random_state=None, shuffle=False)

for train_index, test_index in sss.split(original_X, original_y):
    print("Train: {} Test: {}".format(train_index, test_index))
    original_Xtrain, original_Xtest = original_X.iloc[train_index], original_X.iloc[test_index]
    original_ytrain, original_ytest = original_y.iloc[train_index], original_y.iloc[test_index]
    
#Turn into arrays    
original_Xtrain = original_Xtrain.values
original_Xtest = original_Xtest.values
original_ytrain = original_ytrain.values
original_ytest = original_ytest.values

#Check if the labels are equally distibuted among the train and test set.
train_unique_label, train_count_label = np.unique(original_ytrain, return_counts=True)
test_unique_label, test_count_label = np.unique(original_ytest, return_counts=True)

print('\n Label Distribution:')
print('Train: {}'.format(train_count_label/len(original_ytrain)))
print('Test: {}'.format(test_count_label/len(original_ytest)))<jupyter_output>Train: [ 30473  30496  31002 ... 284804 284805 284806] Test: [    0     1     2 ... 57017 57018 57019]
Train: [     0      1      2 ... 284804 284805 284806] Test: [ 30473  30496  31002 ... 113964 113965 113966]
Train: [     0      1      2 ... 284804 284805 284806] Test: [ 81609  82400  83053 ... 170946 170947 170948]
Train: [     0      1      2 ... 284804 284805 284806] Test: [150654 150660 150661 ... 227866 227867 227868]
Train: [     0      1      2 ... 227866 227867 227868] Test: [212516 212644 213092 ... 284804 284805 284806]

 Label Distribution:
Train: [0.99827076 0.00172924]
Test: [0.99827952 0.00172048]<jupyter_text>UndersamplingAbove for the analysis I have random undersampled the majority class of the dataset and did undersampling before cross-validation. That is not the ideal scenario as it is prone to overfit. Here I'll perform undersampling during cross-validation and use Nearmiss technique for undersampling.**Nearmiss Technique** : In order to attack the issue of potential information loss, “near neighbor” method and its variations have been proposed. The basic algorithms of the near neighbor family are this: first, the method calculates the distances between all instances of the majority class and the instances of the minority class. Then k instances of the majority class that have the smallest distances to those in the minority class are selected. If there are n instances in the minority class, the “nearest” method will result in k*n instances of the majority class.“NearMiss-1” selects samples of the majority class that their average distances to three closest instances of the minority class are the smallest. “NearMiss-2” uses three farthest samples of the minority class. “NearMiss-3” selects a given number of the closest samples of the majority class for each sample of the minority class.<jupyter_code>#Lists
undersample_accuracy_lst = []
undersample_precision_lst = []
undersample_recall_lst = []
undersample_f1_lst = []
undersample_auc_lst = []

X_nearmiss, y_nearmiss = NearMiss().fit_sample(original_X.values, original_y.values)
print('Near Miss Distribution: {}'.format(np.unique(y_nearmiss, return_counts=True)))

for train, test in sss.split(original_Xtrain, original_ytrain):
    pipeline = imbalanced_make_pipeline(NearMiss(ratio='majority'), log_reg)
    undersample_model = pipeline.fit(original_Xtrain[train], original_ytrain[train])
    prediction = undersample_model.predict(original_Xtrain[test])
    
    undersample_accuracy_lst.append(pipeline.score(original_Xtrain[test], original_ytrain[test]))
    undersample_precision_lst.append(precision_score(original_ytrain[test], prediction))
    undersample_recall_lst.append(recall_score(original_ytrain[test], prediction))
    undersample_f1_lst.append(f1_score(original_ytrain[test], prediction))
    undersample_auc_lst.append(roc_auc_score(original_ytrain[test], prediction))
    
print('accuracy: {}'.format(np.mean(undersample_accuracy_lst)))
print('precision: {}'.format(np.mean(undersample_precision_lst)))
print('recall: {}'.format(np.mean(undersample_recall_lst)))
print('f1_score: {}'.format(np.mean(undersample_f1_lst)))
labels = ['No Fraud', 'Fraud']
undersample_prediction = undersample_model.predict(original_Xtest)
print(classification_report(original_ytest, undersample_prediction, target_names=labels))<jupyter_output>precision    recall  f1-score   support

   No Fraud       1.00      0.60      0.75     56863
      Fraud       0.00      0.90      0.01        98

avg / total       1.00      0.60      0.75     56961<jupyter_text>Oversampling (SMOTe Technique)Synthetic Minority Oversampling Technique(SMOTe) has the following description:-- SMOTe is a technique based on nearest neighbors judged by Euclidean Distance between data points in feature space.- There is a percentage of Over-Sampling which indicates the number of synthetic samples to be created and this percentage parameter of Over-sampling is always a multiple of 100. If the percentage of Over-sampling is 100, then for each instance, a new sample will be created. Hence, the number of minority class instances will get doubled. Similarly, if the percentage of Over-sampling is 200, then the total number of minority class samples will get tripled.In SMOTe,1. For each minority instance, k number of nearest neighbors are found such that they also belong to the same class where,2. The difference between the feature vector of the considered instance and the feature vectors of the k nearest neighbors are found. So, k number of difference vectors are obtained.3. The k difference vectors are each multiplied by a random number between 0 and 1 (excluding 0 and 1).4. Now, the difference vectors, after being multiplied by random numbers, are added to the feature vector of the considered instance (original minority instance) at each iteration.<jupyter_code>rand_log_reg = RandomizedSearchCV(LogisticRegression(), params['Logistic Regression'], n_iter=4)

#Lists
oversample_accuracy_lst = []
oversample_precision_lst = []
oversample_recall_lst = []
oversample_f1_lst = []
oversample_auc_lst = []

for train, test in sss.split(original_Xtrain, original_ytrain):
    pipeline = imbalanced_make_pipeline(SMOTE(ratio='minority'), rand_log_reg)
    model = pipeline.fit(original_Xtrain[train], original_ytrain[train])
    best_est = rand_log_reg.best_estimator_
    prediction = best_est.predict(original_Xtrain[test])
    
    oversample_accuracy_lst.append(pipeline.score(original_Xtrain[test], original_ytrain[test]))
    oversample_precision_lst.append(precision_score(original_ytrain[test], prediction))
    oversample_recall_lst.append(recall_score(original_ytrain[test], prediction))
    oversample_f1_lst.append(f1_score(original_ytrain[test], prediction))
    oversample_auc_lst.append(roc_auc_score(original_ytrain[test], prediction))

print('accuracy: {}'.format(np.mean(oversample_accuracy_lst)))
print('precision: {}'.format(np.mean(oversample_precision_lst)))
print('recall: {}'.format(np.mean(oversample_recall_lst)))
print('f1_score: {}'.format(np.mean(oversample_f1_lst)))
smote_prediction = best_est.predict(original_Xtest)
print(classification_report(original_ytest, smote_prediction, target_names=labels))
undersample_prediction = undersample_model.predict(X_nearmiss)
undersample_score = accuracy_score(y_nearmiss, undersample_prediction)
smote_prediction = best_est.predict(original_Xtest)
oversample_score = accuracy_score(original_ytest, smote_prediction)

data = {'Technique': ['Undersampling', 'Oversampling (SMOTE)'], 'Score': [undersample_score, oversample_score]}
final_df = pd.DataFrame(data=data)
score = final_df['Score']
final_df.drop('Score', axis=1, inplace=True)
final_df.insert(1, 'Score', score)
final_df<jupyter_output><empty_output><jupyter_text>Neural Network testing Random undersampled data vs Oversampled data(SMOTe)In this section I'll be using a neural network with single hidden layer to test the performance in case of random undersampled and SMOTe oversampled data. I'll be using Keras library to create my neural network.- **Neural Network Structure**: As stated previously, this will be a simple model composed of one input layer (where the number of nodes equals the number of features) plus bias node, one hidden layer with 32 nodes and one output node composed of two possible results 0 or 1 (No fraud or fraud).- **Other characteristics**: The learning rate will be 0.001, the optimizer we will use is the AdamOptimizer, the activation function that is used in this scenario is "Relu" and for the final outputs we will use sparse categorical cross entropy, which gives the probability whether an instance case is no fraud or fraud (The prediction will pick the highest probability between the two.)The metric I'll be using is a **confusion matrix**. Confusion MatrixA confusion matrix is a summary of prediction results on a classification problem.The number of correct and incorrect predictions are summarized with count values and broken down by each class. This is the key to the confusion matrix.The confusion matrix shows the ways in which your classification model is confused when it makes predictions.It gives us insight not only into the errors being made by a classifier but more importantly the types of errors that are being made.Here,- Class 1 : Positive- Class 2 : Negative  Testing on Random undersampled data<jupyter_code>n_inputs = X_train.shape[1]

undersample_model = Sequential([
    Dense(n_inputs, input_shape=(n_inputs,), activation='relu'),
    Dense(32, activation='relu'),
    Dense(2, activation='softmax')
])
undersample_model.summary()
undersample_model.compile(Adam(lr=0.001), loss='sparse_categorical_crossentropy', metrics=['accuracy'])
undersample_model.fit(X_train, y_train, validation_split=0.2, batch_size=25, epochs=20, shuffle=True, verbose=2)
undersample_predictions = undersample_model.predict(X_test, batch_size=200, verbose=0)
undersample_fraud_predictions = undersample_model.predict_classes(X_test, batch_size=200, verbose=0)
import itertools

# Create a confusion matrix
def plot_confusion_matrix(cm, classes,
                          normalize=False,
                          title='Confusion matrix',
                          cmap=plt.cm.Blues):
    """
    This function prints and plots the confusion matrix.
    Normalization can be applied by setting `normalize=True`.
    """
    if normalize:
        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
        print("Normalized confusion matrix")
    else:
        print('Confusion matrix, without normalization')

    print(cm)

    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title, fontsize=14)
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, rotation=45)
    plt.yticks(tick_marks, classes)

    fmt = '.2f' if normalize else 'd'
    thresh = cm.max() / 2.
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        plt.text(j, i, format(cm[i, j], fmt),
                 horizontalalignment="center",
                 color="white" if cm[i, j] > thresh else "black")

    plt.tight_layout()
    plt.ylabel('True label')
    plt.xlabel('Predicted label')
from sklearn.metrics import confusion_matrix
undersample_cm = confusion_matrix(y_test, undersample_fraud_predictions)
actual_cm = confusion_matrix(y_test, y_test)
labels = ['No Fraud', 'Fraud']

fig = plt.figure(figsize=(16,8))

fig.add_subplot(221)
plot_confusion_matrix(undersample_cm, labels, title="Random UnderSample \n Confusion Matrix", cmap=plt.cm.Reds)

fig.add_subplot(222)
plot_confusion_matrix(actual_cm, labels, title="Confusion Matrix \n (with 100% accuracy)", cmap=plt.cm.Greens)<jupyter_output>Confusion matrix, without normalization
[[94  5]
 [10 88]]
Confusion matrix, without normalization
[[99  0]
 [ 0 98]]<jupyter_text>Testing on Oversampled data(SMOTe Technique)<jupyter_code>n_inputs = original_Xtrain.shape[1]

oversample_model = Sequential([
    Dense(n_inputs, input_shape=(n_inputs, ), activation='relu'),
    Dense(32, activation='relu'),
    Dense(2, activation='softmax')
])
oversample_model.summary()
oversample_model.compile(Adam(lr=0.001), loss='sparse_categorical_crossentropy', metrics=['accuracy'])
oversample_model.fit(original_Xtrain, original_ytrain, validation_split=0.2, batch_size=300, epochs=20, shuffle=True, verbose=2)
oversample_predictions = oversample_model.predict(original_Xtest, batch_size=200, verbose=0)
oversample_fraud_predictions = oversample_model.predict_classes(original_Xtest, batch_size=200, verbose=0)
oversample_smote = confusion_matrix(original_ytest, oversample_fraud_predictions)
actual_cm = confusion_matrix(original_ytest, original_ytest)
labels = ['No Fraud', 'Fraud']

fig = plt.figure(figsize=(16,8))

fig.add_subplot(221)
plot_confusion_matrix(oversample_smote, labels, title="OverSample (SMOTE) \n Confusion Matrix", cmap=plt.cm.Oranges)

fig.add_subplot(222)
plot_confusion_matrix(actual_cm, labels, title="Confusion Matrix \n (with 100% accuracy)", cmap=plt.cm.Greens)<jupyter_output>Confusion matrix, without normalization
[[56858     5]
 [   28    70]]
Confusion matrix, without normalization
[[56863     0]
 [    0    98]]
{'fraction_non_alphanumeric': 0.09788067879156838, 'fraction_numerical': 0.03957080891865155, 'mean_word_length': 4.94895880488909, 'pattern_counts': {'":': 13, '<': 116, '<?xml version=': 0, '>': 121, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '32', 'dir': 'jupyter-structured-clean-dedup', 'id': '279443', 'license': 'MIT', 'n_tokens_mistral': 17449, 'n_tokens_neox': 15659, 'n_words': 5571, 'path': 'Credit Card Fraud Detection.ipynb', 'repo_name': 'anshuls235/mlnd-capstone-project', 'size': '52564'}