<jupyter_start><jupyter_text>IntroductionIn this jupyter notebook, we will present the way we analysed, processed the dataset and the modeling of the machine learning algorithm to detect the fraudulent transactions.  ![alt text](fraud.jpg "Title")  Data Exploration The dataset [ https://www.kaggle.com/mlg-ulb/creditcardfraud ] consists of transactions done with credit cards in September 2013 by European cardholders. It contains only numerical input variables which are the product of a Principal Component Analysus (PCA). Thus, due to privacy issues the original features are not availablle and we don't know what the V1...V28 features represent expect from the Amount, Time and Class. Totally, there are 31 features. The feature 'Amount' is the amount of a transaction, 'Time' is the seconds elapsed between each transaction and the first transaction and 'Class' represents a fraud transaction when its value equals to 1 and a valid transaction when its value equals to 0.Starting from this assumption, let's start with our data exploration.<jupyter_code>import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
import sklearn
from sklearn import preprocessing
from sklearn.manifold import TSNE
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import confusion_matrix, classification_report, accuracy_score, roc_auc_score, roc_curve
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
import pickle
import findspark
findspark.init()
import pyspark
from pyspark.sql.functions import *
from pyspark.context import SparkContext
from pyspark.sql.session import SparkSession
from pyspark.sql.types import DoubleType, IntegerType
from pyspark.ml import Pipeline
from pyspark.ml.linalg import DenseVector
from pyspark.ml.classification import LogisticRegression
from pyspark.ml.evaluation import MulticlassClassificationEvaluator
from pyspark.mllib.evaluation import BinaryClassificationMetrics

sc = pyspark.SparkContext(appName = "credit_card_fraud")
spark = SparkSession(sc)<jupyter_output><empty_output><jupyter_text>Here, we read the dataset from the csv file using pandas and then we checked if there are missing values before going on with other analysis. Then, we just call the head() and describe() functions to have an overall idea of our dataset. Since there will be no missing values we can proceed with other visualizations.<jupyter_code>df = pd.read_csv("./creditcard.csv")
print("missing values: ", df.isnull().values.any())
df.head()
df.describe()<jupyter_output><empty_output><jupyter_text>Plotting We tried to plot the two not anonymized features, that as we said are 'Time' and 'Amount'. From the Time Distribution we can understand it is bimodal. This indicates that approximately 28 hours after the first transaction there was a drop in the volume. The timestamp is not provided but as hypothesis it would be reasonable to assume that the drop is caused by a drop in the number of transactions during night. We can infer this by the fact that time is provided in seconds hence should be about two days of transactions. From the Amount Distribution we can derive it is skewed. Most of the transactions are small and a little percentage are higher. Finally, we plotted the Non Fraud and Fraud transactions over time but there are no meaningful things we can derive.<jupyter_code>plt.figure(figsize = (6,4))
plt.title('Time Distribution')
sns.distplot(df['Time'])
plt.figure(figsize = (6,4))
plt.title('Amount Distribution')
sns.distplot(df['Amount'])
plt.figure(figsize = (6,4))
plt.title('Non Fraud Distribution')
sns.distplot(df['Time'][df['Class'] == 0])
plt.figure(figsize = (6,4))
plt.title('Fraud Distribution')
sns.distplot(df['Time'][df['Class'] == 1])<jupyter_output><empty_output><jupyter_text>At this point, we tried to understand how the transactions are distributed among the two classes of 'Fraud' and 'Non Fraud'. We derived the data and plotted a bar chart to explain it. Thus, we understood that the dataset has just few fraudulent transactions, which are 0.173% of the total. Therefore, we are dealing with a highly unbalanced dataset and with a class of problems known as class imbalance. By studying the problem we discovered that it is very common in fraud detection problems to have this issue.  Stated that, it is not possible to use directly a machine learning algorithm otherwise it will overfit the data on the 'Non Fraud' samples. A good idea to have a not biased result is to reduce classes samples to be more or less equal.The main problem is that if there are few 'Fraud' samples, there is too few data to infer patterns from them.<jupyter_code>frauds_df = df[df['Class'] == 1]
non_frauds_df = df[df['Class'] == 0]

frauds_number = len(frauds_df)
non_frauds_number = len(non_frauds_df)

print("fraudelent transactions are " + str(frauds_number))
print("non fraudelent transaction are " + str(non_frauds_number))

ratio = (float(frauds_number) / float(non_frauds_number)) * 100
print("frauds are " + str(ratio)[:5] + "% of the total")
classes = pd.value_counts(df['Class'], sort=True)
classes.plot(kind = 'bar', rot=0)

plt.title("Class Transactions")
plt.xticks(range(2), ['Non Fraud', 'Fraud'])
plt.xlabel("Class")
plt.ylabel("Frequency - #Transactions")
plt.show()<jupyter_output><empty_output><jupyter_text>We don't know the features but we can understand if there is a correlation between them, hence we plotted an heatmap with correlations. We infered that there are no highly correlated features. There's a just a slightly negative correlation between V2 and Amount. And some features seem to have little correlations with the label 'Class' maybe due to the PCA.<jupyter_code>correlations = df.corr()

fig, ax = plt.subplots(figsize=(8, 6))
sns.heatmap(correlations, 
            xticklabels=correlations.columns, 
            yticklabels=correlations.columns,
            linewidths=.1, cmap="RdBu", ax=ax)<jupyter_output><empty_output><jupyter_text>To visualize differently the distribution of data, we used T-SNE, which is a data decomposition technique that reduces data into specific dimensions and displays the top n dimensions with maximum information, a technique similar to PCA. A lot of Non Fraud transactions overlap with the Fraud transactions hence they are difficult to divide with a linear boundary.<jupyter_code>non_fraud_tsne = df[df['Class'] == 0].sample(2000)
fraud_tsne = df[df['Class'] == 1]

#frac = 1 retains the 100% of the fraud dataframe
df_tsne = non_fraud_tsne.append(fraud_tsne).sample(frac = 1)
#drop the index columns 
df_tsne = df_tsne.reset_index(drop = True)

x = df_tsne.drop(['Class'], axis = 1).values
y = df_tsne['Class'].values

p = TSNE(n_components=2, random_state=24).fit_transform(x)

plt.figure(figsize=(8, 6))
plt.scatter(p[np.where(y == 0), 0], p[np.where(y == 0), 1], marker='o', color='g', linewidth='1', alpha=0.8, label='Non Fraud')
plt.scatter(p[np.where(y == 1), 0], p[np.where(y == 1), 1], marker='o', color='r', linewidth='1', alpha=0.8, label='Fraud')
plt.legend(loc='best')
plt.show()<jupyter_output><empty_output><jupyter_text>Data Preparation All the anonymized features have been scaled and centered around zero while 'Time' and 'Amount' have been not. To have a better performance with the Logistic Regression, the ML algorithm that we will use later, is necessary to scale them as weel. To prepare our data as input to the ML we need to:- preprocess time and amount witn StandardScaler- split dataset into train and test set- deal with the highly unbalanced dataset The StandardScaler will transform the data to have a distribution will with mean value 0 and a standard deviation of 1. Transform the data is crucial to be correctly interpreted by the algorithm.<jupyter_code># scaling TIME to be centered around zero 
scaler_time = StandardScaler()

scaled_time = scaler_time.fit_transform(df[['Time']])
scaled_time_list = scaled_time.tolist()

flat_time = [item for sublist in scaled_time_list for item in sublist]
scaled_time = pd.Series(flat_time)

#print(scaled_time)
# scaling AMOUNT to be centered around zero 
scaler_amount = StandardScaler()

scaled_amount = scaler_amount.fit_transform(df[['Amount']])
scaled_amount_list = scaled_amount.tolist()

flat_amount= [item for sublist in scaled_amount_list for item in sublist]
scaled_amount = pd.Series(flat_amount)

#print(scaled_amount)
# Add the new two columns to the dataframe and check it is everything ok

df_scaled = pd.concat([df, scaled_amount.rename('Amount_Scaled'), scaled_time.rename('Time_Scaled')], axis=1)
df_scaled.sample(5)
# Drop columns with no normalized values

df_scaled.drop(['Amount', 'Time'], axis = 1, inplace = True)
df_scaled.sample(5)
# reindexing dataframe with a correct order 

columns_titles = ['V1','V2','V3','V4', 'V5', 'V6', 'V7', 'V8', 'V9', 'V10', 'V11', 'V12', 'V13', 'V14', 'V15', 'V16', 'V17', 'V18', 'V19', 'V20', 'V21', 'V22', 'V23', 'V24', 'V25', 'V26', 'V27', 'V28', 'Amount_Scaled', 'Time_Scaled', 'Class']
df_scaled = df_scaled.reindex(columns = columns_titles)
df_scaled.sample(5)<jupyter_output><empty_output><jupyter_text>Now we can observe that fraud and non-fraud transactions are distinguishable and linearly separable to some extente.<jupyter_code># feature Matrix
#X = df_scaled.drop('Class', axis = 1)

# target Vector
#y = df_scaled['Class']

#X_TSNE = X.values
#y_TSNE = y.values

#z = TSNE(n_components = 2, random_state = 24).fit_transform(X_TSNE)

#plt.figure(figsize=(8, 6))
#plt.scatter(z[np.where(y_TSNE == 0), 0], z[np.where(y_TSNE == 0), 1], marker='o', color='g', linewidth='1', alpha=0.8, label='Non Fraud')
#plt.scatter(z[np.where(y_TSNE == 1), 0], z[np.where(y_TSNE == 1), 1], marker='o', color='r', linewidth='1', alpha=0.8, label='Fraud')
#plt.legend(loc = 'best')
#plt.show()<jupyter_output><empty_output><jupyter_text>Data Splitting<jupyter_code># casting all columns to double 

df_spark = spark.createDataFrame(df_scaled)
df_spark = df_spark.withColumn('Class', df_spark['Class'].cast('double'))
df_spark.printSchema()
df_spark.head()
ml_df = df_spark.rdd.map(lambda x: (DenseVector(x[0:30]), x[30]))
ml_df = spark.createDataFrame(ml_df, ["features", "label"])
ml_df.head()
#ml_df.select("features").show()
#ml_df.groupBy("label").count().show()
ml_df = ml_df.select("features","label")
train_data, test_data = ml_df.randomSplit([.8,.2],seed=1234)
train_data.head()
train_data.groupBy("label").count().show()
test_data.head()
test_data.groupBy("label").count().show()<jupyter_output>+-----+-----+
|label|count|
+-----+-----+
|  0.0|56545|
|  1.0|   89|
+-----+-----+<jupyter_text>Balancing the DatasetThe dataset is highly unbalanced since there's a skew in the class distribution. Thus, our training dataset could be biased and influence the ML algorithm when displaying results. To address this problem we randomly balance the data to have the same number of fraudelent and non fraudulent transactions. To do that, we retrieve from the training data a dataframe with the frauds and a dataframe with the non frauds. After that, we concatenated the two to obtain a balanced dataset. In this way, we created a subsample dataset with balanced class distribution<jupyter_code>fraud_df_spark = train_data.filter("label=1.0")
non_fraud_df_spark = train_data.filter("label=0.0")

print(fraud_df_spark.count())
print(non_fraud_df_spark.count())

ratio = fraud_df_spark.count() / non_fraud_df_spark.count()

non_fraud_sample_df_spark = non_fraud_df_spark.sample(False, ratio)

balanced_df = fraud_df_spark.unionAll(non_fraud_sample_df_spark)
# dataframe spark to dataframe pandas
balanced_dfp = balanced_df.toPandas()
balanced_dfp.head()
number = balanced_df.count()<jupyter_output><empty_output><jupyter_text>The plot below shows that now we have a balanced dataset, where the number of frauds is equal to the number of non fraudulent transactions.<jupyter_code>new_classes = pd.value_counts(balanced_dfp['label'], sort=True)
new_classes.plot(kind = 'bar', rot=0)

plt.title("Class Transactions")
plt.xticks(range(2), ['Non Fraud', 'Fraud'])
plt.xlabel("Class")
plt.ylabel("Frequency - #Transactions")
plt.show()<jupyter_output><empty_output><jupyter_text>Correlations<jupyter_code>columns_titles = ['V1','V2','V3','V4', 'V5', 'V6', 'V7', 'V8', 'V9', 'V10', 'V11', 'V12', 'V13', 'V14', 'V15', 'V16', 'V17', 'V18', 'V19', 'V20', 'V21', 'V22', 'V23', 'V24', 'V25', 'V26', 'V27', 'V28', 'Amount_Scaled', 'Time_Scaled', 'Label']
correlations_df = pd.DataFrame(columns=columns_titles)

for i in range(number):
    
    features_value_list = balanced_dfp['features'].iloc[i].tolist()
    label = balanced_dfp['label'].iloc[i]
    features_value_list.append(label)
    features_value_array = np.asarray(features_value_list)
    temp = pd.DataFrame([features_value_array], columns=columns_titles)
    correlations_df = correlations_df.append(temp, ignore_index=True)

correlations_df.head()<jupyter_output><empty_output><jupyter_text>Now that we have a balanced dataset, the correlation matrix shows meaningful correlation between some features, differently from the correlation matrix displayed before<jupyter_code>correlations = correlations_df.corr()

fig, ax = plt.subplots(figsize=(8, 6))
sns.heatmap(correlations, 
            xticklabels=correlations.columns, 
            yticklabels=correlations.columns,
            linewidths=.1, cmap="RdBu", ax=ax)<jupyter_output><empty_output><jupyter_text>Classification Modeling We are going to utilize a ML classification algorith to train our data and pick up the specific characterstics that make a transaction more or less likely to be fraudulent. Performance Evaluation is done on ROC-AUC. It outputs a value between zero and one. If it is above 0.5 it is better than random guessing.   Training It has been used a logistic regression. Logistic regression models are used to determine the odds of a certain class or event existing. In our case, it will establish the probability of a transaction belonging to Class 0 or 1, which is normal or fraudulent.<jupyter_code>lr = LogisticRegression().setLabelCol("label").setFeaturesCol("features").setMaxIter(30).setRegParam(0.3).setElasticNetParam(0.8)

lr_model = lr.fit(balanced_df)<jupyter_output><empty_output><jupyter_text>Testing Now, let's evaluate our model with the test dataset<jupyter_code>lr_predictions = lr_model.transform(test_data)
lr_predictions.show()
lr_predictions.select("prediction", "rawPrediction", "label", "features").show(5)
training_summary = lr_model.summary
print(training_summary)

lr_f1_evaluator = MulticlassClassificationEvaluator().setMetricName("f1").setLabelCol("label").setPredictionCol("prediction")
lr_f1 = lr_f1_evaluator.evaluate(lr_predictions)

training_summary.roc.show()
print("area Under ROC: " + str(training_summary.areaUnderROC))
print("F1-Score: " + str(lr_f1))
prediction_labels = lr_predictions.rdd.map(lambda x: (float(x.prediction), float(x.label)))

metrics = BinaryClassificationMetrics(prediction_labels)

print("Area under PR = %s" % metrics.areaUnderPR)
print("Area under ROC = %s" % metrics.areaUnderROC)<jupyter_output>Area under PR = 0.2965094019934293
Area under ROC = 0.8920116323779113<jupyter_text>Save model commented code for saving LogisticRegressionModel on the disk for future use<jupyter_code>lr_model.write().save("./LRModelSpark")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10934755740584141, 'fraction_numerical': 0.01775840759773629, 'mean_word_length': 6.068505747126437, 'pattern_counts': {'":': 0, '<': 63, '<?xml version=': 0, '>': 63, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '462735', 'license': 'Apache-2.0', 'n_tokens_mistral': 5099, 'n_tokens_neox': 4745, 'n_words': 1580, 'path': 'final_project/model/data_exploration_modeling.ipynb', 'repo_name': 'adrianomundo/data-intensive-computing', 'size': '15373'}