<jupyter_start><jupyter_text>Tutorial - Using YData Synthetics to improve a fraud dataset and deploy it to UbiOps Dealing with massively imbalanced datasets for improved fraud detection in the banking sector. Use of synthetic data generation to balance highly imbalanced classes, in particular, generation of synthetic fraudulent events.The use of synthetic data for this type of use cases helps to improve the accuracy of fraud detection models in many different areas from the banking industry. Machine Learning and imbalanced datasetsHighly imbalanced datasets are extremely challenging for data teams, and they can be found very often in the industry, wether in topics such as detecting fraudulent events, but also for money laundry and even credit underwriting.The commonnality between each one of these use cases? The fact that only a small portion of the overall transactions represent one particular class, for fraud problems, only a portion of those transactions were, in reality, a fraudulent event.  Credit Fraud - a highy imbalanced datasetThe dataset in this example use case is from Kaggle - ["Credit Card Fraud detection"](https://www.kaggle.com/mlg-ulb/creditcardfraud) dataset, as for demonstration purposes we are only able to use datasets from the public domain.This dataset includes labeled transactions from European credit car holders, and the data provided is a result from a dimensionality reduction, containing 27 continous features and a time column - the number of secons elapsed between the first and the last transaction of the dataset. The dataset Exploratory data analysis(Include here some input on how highly imbalanced are the datasets)(Include here some metrics and some validations that where done using YData's lib for data analysis)<jupyter_code>#Download the data from kaggle and read it using pandas
from datetime import datetime
import json
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split

import pandas as pd
#Get the dataset right from kaggle
#Make sure that you've already downloaded your kaggle.json file and have eit located in the configured directoy for kaggle lib
#import kaggle
#kaggle.api.authenticate()
#kaggle.api.dataset_download_files('creditcardfraud', path='./data/', unzip=True)

credit = pd.read_csv('data/creditcard.csv')
credit.head()<jupyter_output><empty_output><jupyter_text>This dataset is highly imbalanced. Let's check the behaviour of the imbalanced classes.<jupyter_code>fig, ax = plt.subplots(figsize=(20,10))

#Remove the frame around
#Have this as an auxiliary function
for spine in plt.gca().spines.values():
    spine.set_visible(False)
plt.bar([0, 1], pd.value_counts(credit['Class'].values, sort=True), alpha=0.5, color=['#7A7A7A','#E32212'])
plt.title('High imbalanced transactions', fontsize=30, pad=10)
plt.annotate('Fraudulent transactions', (0.82, 50000), fontsize=20, color='red')
plt.annotate('Regular transactions', (-0.2, 50000), fontsize=20, color='black')
ax.yaxis.set_tick_params(labelsize=15)
ax.xaxis.set_ticks([0,1])
ax.tick_params(axis='x', which='major', pad=15)
ax.xaxis.set_tick_params(labelsize=25)
#Scatter plot with Class colouring the different elements from the plot
#As this dataset is the result of a PCA we will be using 2 eigen vectors for the plot
import matplotlib as m
import numpy as np

cmap = m.cm.jet(np.linspace(0,1,26))
cmap = m.colors.ListedColormap(cmap[21:,:-1])

fig, ax = plt.subplots(figsize=(20,10))

for spine in plt.gca().spines.values():
    spine.set_visible(False)

plt.scatter(credit['V2'],credit['V4'], alpha=0.7, c=credit['Class'],cmap=cmap)
plt.axis('off')
plt.title('Scatter plot for V2 and V4 eigenvectors', fontsize=30, pad=10)<jupyter_output><empty_output><jupyter_text>Splitting the dataset in train and test sets. The test set will be used again at the end of our iteration<jupyter_code>X = credit.drop('Class', axis=1)
cols = X.columns
X = X.values
y = credit['Class']

X_train, X_test, y_train, y_test = train_test_split(X,y,test_size = 0.3, random_state = 0)

print("Number transactions train dataset: ", len(X_train))
print("Number transactions test dataset: ", len(X_test))
print("Total number of transactions: ", len(X_train)+len(X_test))
count_original = np.unique(y, return_counts=True)
count_train = np.unique(y_train, return_counts=True)
count_test = np.unique(y_test, return_counts=True)

print("Ratio between fraud and normal events for the \033[1mfull\033[0m  dataset:"+ " {:.2}%".format(count_train[1][1]/count_train[1][0]))
print("Ratio between fraud and normal events for the \033[1mtrain\033[0m dataset:"+ " {:.2}%".format(count_train[1][1]/count_train[1][0]))
print("Ratio between fraud and normal events for the \033[1mtest\033[0m dataset:"+" {:.2}%".format(count_test[1][1]/count_test[1][0]))<jupyter_output>Ratio between fraud and normal events for the [1mfull[0m  dataset: 0.0017%
Ratio between fraud and normal events for the [1mtrain[0m dataset: 0.0017%
Ratio between fraud and normal events for the [1mtest[0m dataset: 0.0017%<jupyter_text>The first modelLet's try to develop a model basedo n the assumption that everything is ok with our dataset, and understand how good is our classifier to indetify fraudulent events.Here we've decided to develop a classifier using [RandomForest from the scikit-learn](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html) package.<jupyter_code>X_train = pd.DataFrame(X_train, columns=cols)
X_test = pd.DataFrame(X_test, columns=cols)
#Data scaling and preprocessing before training the model
from sklearn.preprocessing import RobustScaler, StandardScaler

def preprocess_df(df, std_scaler, rob_scaler):
    df['Amount'] = std_scaler.fit_transform(df['Amount'].values.reshape(-1,1))
    df['Time'] = rob_scaler.fit_transform(df['Time'].values.reshape(-1,1))
    return df
stdscaler = StandardScaler()
robscaler = RobustScaler()

X_train = preprocess_df(X_train, stdscaler, robscaler)
#Apply the same trasnformation to the test dataset
X_test = preprocess_df(X_test, stdscaler, robscaler)
#Using XGBOOST model to train the model
from sklearn.metrics import accuracy_score
from xgboost import XGBClassifier


def XGBoost_Classifier(X, y, Xtest):
    """XGBoost training code"""
    classifier = XGBClassifier()
    print('Start fitting XGBoost classifier')
    classifier.fit(X, y)
    y_pred = classifier.predict(Xtest)
    print('Classifier trained.')
    return classifier, y_pred

classifier_model, y_pred = XGBoost_Classifier(X_train, y_train, X_test)
from sklearn.metrics import accuracy_score
from sklearn.metrics import classification_report

# Print precision / recall report
print('Overall accuracy score :',accuracy_score(y_test, y_pred))
print('Classifier performance report: ')
print(classification_report(y_test, y_pred, target_names=['Normal', 'Fraud']))
#Now let's check the real metrics for this classifier
from sklearn.metrics import plot_confusion_matrix


def print_confusion_matrix(model, X_test, y_test):
    """ Plot normalized and non-normalized confusion matrices """
    titles_options = [("Confusion matrix, without normalization", None),
                      ("Normalized confusion matrix", 'true')]

    fig, axes = plt.subplots(1,2,figsize=(20,8))
    for (title, normalize), ax in zip(titles_options, axes):

        disp = plot_confusion_matrix(model, X_test, y_test,
                                     display_labels=["Normal", "Fraud"],
                                     cmap=plt.cm.OrRd,
                                     normalize=normalize,
                                     ax=ax)

        ax.set_title(title, fontsize=20, pad=10)

print_confusion_matrix(classifier_model, X_test, y_test)<jupyter_output><empty_output><jupyter_text>Synthetic data to improve the detection of fraud Synthetic data with YData synthesizer packageIn this case the objective is to synthesize only the fraudulent events. Through the augmentation of fraudulent events we are able to improve the results the results of our classifier.<jupyter_code>#Let's filter by fraudulent events only
aux = X_train.copy()
aux['y'] = y_train.reset_index()['Class']

non_fraud = aux[aux['y'] == 0]
fraud = aux[aux['y']==1]

del aux
#Have here YData's lib to be called and used in the Jupyter Notebook
from ydata_synthetic.synthesizers.regular import WGAN_GP

noise_dim = 32
dim = 128
batch_size = 164

log_step = 100
epochs = 1000+1
learning_rate = 5e-4
beta_1 = 0.5
beta_2 = 0.9
models_dir = './cache'

gan_args = [batch_size, learning_rate, beta_1, beta_2, noise_dim, fraud.shape[1]-1, dim]
train_args = ['', epochs, log_step]

fraud_synth = WGAN_GP(gan_args, n_critic=2)
fraud_synth.train(fraud.drop('y', axis=1), train_args)

synthetic_fraud = fraud_synth.sample(400)
synthetic_fraud.columns = fraud.drop('y', axis=1).columns
synthetic_fraud.head(5)<jupyter_output><empty_output><jupyter_text>Visualizing the generated synthetic data<jupyter_code>#Read the synthetic fraud generated events 
#local_conn = LocalConnector()
#synthetic = local_conn.read_file('synthetic_fraud.csv', file_type=FileType.CSV)

synth_df = synthetic_fraud.copy()
org_df = X_train.copy()

org_df['Class'] = y_train.reset_index()['Class']
org_df['color'] = np.where(org_df['Class']==1, 2, 1)

synth_df['Class'] = 1
synth_df['color'] = 3

full_data = pd.concat([org_df, synth_df])<jupyter_output><empty_output><jupyter_text>Using a PCA we are able to  reduce the dimensionality of the dataset and and better visualize the relation between both the synthetic fraud generated events and the ones from the original data, regardless they're fraud or normal events. The randomness introduced by the synthetic data can help the model to generalize for a new dataset and new fraud events never previously observed.<jupyter_code>from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA

X = full_data.iloc[:, :-2]
color = full_data['color']

fig = plt.figure(1, figsize=(15, 10))
plt.clf()
plt.cla()

cmap = m.cm.jet(np.linspace(0,1,26))
cmap = m.colors.ListedColormap(cmap[17:,:-1])

pca = PCA(n_components=2)
projected = pca.fit_transform(X)

labels = ['Train Normal', 'Train Fraud', 'Synthetic Fraud']

size_map = {1:25, 2:50, 3:75}
sizes = [size_map[x] for x in color]

scatter = plt.scatter(projected[:, 0], projected[:, 1],
                      s=sizes,
                      c=color,
                      cmap=cmap, 
                      alpha=0.8)

plt.legend(handles=scatter.legend_elements()[0], labels=labels)
plt.show()<jupyter_output><empty_output><jupyter_text>Testing the classifier capacity after adding more fraudulent events<jupyter_code>np.unique(y_test, return_counts=True)
synth_y_train = synth_df['Class']
synth_train = synth_df.drop(['Class', 'color'], axis=1)

X_augmented = pd.concat([X_train, synth_train], axis=0)
y_augmented = pd.concat([y_train, synth_y_train], axis=0)
synth_classmodel, y_pred = XGBoost_Classifier(X_augmented, y_augmented, X_test)
# Print precision / recall report
print('Overall accuracy score :',accuracy_score(y_test, y_pred))
print('Classifier performance report: ')
print(classification_report(y_test, y_pred, target_names=['Normal', 'Fraud']))
print_confusion_matrix(synth_classmodel, X_test, y_test)
#Create zip with deployment
filename = 'simple_deployment_package.zip'<jupyter_output><empty_output><jupyter_text>Deploy the model to UbiOps<jupyter_code>!pip install ubiops
import ubiops

API_TOKEN = 'Token <API <PASSWORD>>'
PROJECT_NAME = 'ydata-to-ubiops'
DEPLOYMENT_NAME = 'ydata-tutorial'
DEPLOYMENT_VERSION = 'v1'

configuration = ubiops.Configuration()
configuration.api_key['Authorization'] = API_TOKEN

client = ubiops.ApiClient(configuration)
api = ubiops.api.CoreApi(client)

print(api.service_status())
client.close()
deployment_template = ubiops.DeploymentCreate(
    name=DEPLOYMENT_NAME,
    description='XGBoost Fraud Detection',
    input_type='structured',
    output_type='structured',
    input_fields=[ubiops.DeploymentInputFieldCreate(name='input', data_type='double')],
    output_fields=[ubiops.DeploymentOutputFieldCreate(name='output', data_type='double')]
)

deployment = api.deployments_create(project_name=PROJECT_NAME, data=deployment_template)
print(deployment)
version_template = ubiops.VersionCreate(
    version=DEPLOYMENT_VERSION,
    language='python3.7',
    memory_allocation=256,
    maximum_instances=1,
    minimum_instances=0,
    maximum_idle_time=1800 # = 30 minutes
)

version = api.versions_create(
    project_name=PROJECT_NAME,
    deployment_name=DEPLOYMENT_NAME,
    data=version_template
)
print(version)
file_upload_result =api.versions_file_upload(
    project_name=PROJECT_NAME,
    deployment_name=DEPLOYMENT_NAME,
    version=DEPLOYMENT_VERSION,
    file='simple_deployment_package.zip'
)
from time import sleep
status = 'building'
while status != 'available' and 'failed' not in status:    
    version_status = api.versions_get(       
        project_name=PROJECT_NAME,        
        deployment_name=DEPLOYMENT_NAME,        
        version=DEPLOYMENT_VERSION    
    )    
    status = version_status.status
    print(status, end='\r')
    sleep(1)
print(status)
# data = y_test
data = {'input': 123}
request_result = api.deployment_requests_create(
    project_name=PROJECT_NAME,
    deployment_name=DEPLOYMENT_NAME,
    version=DEPLOYMENT_VERSION,
    data=data
)
print(request_result)<jupyter_output>{'error_message': '', 'result': {'output': 57.418105942159}, 'success': True}
{'fraction_non_alphanumeric': 0.10379241516966067, 'fraction_numerical': 0.01870333407259555, 'mean_word_length': 5.414414414414415, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '654832', 'license': 'MIT', 'n_tokens_mistral': 4530, 'n_tokens_neox': 4248, 'n_words': 1211, 'path': '9 - integrations/ubiops/YData_to_UbiOps_FraudDetection.ipynb', 'repo_name': 'ydataai/Blog', 'size': '13522'}