<jupyter_start><jupyter_text>Fraudulent Transactions - Training a classifier with SklearnWelcome to this Jupyter Notebook in which we will train a simple classifier in Scikit Learn to detect fraudulent transactions. We will be exploring a number of libraries in Python to perform data exploration, data wrangling and data cleaning before feeding it in to a classifier of our liking. **Note**: Before running all cells in this notebook, enter your Azure information in step 10.Table of contents:1. Load2. Explore3. Transform4. Split the data5. Train our model6. Evaluate our model     a) Accuracy     b) Precision/Recall     c) Confusion matrix 7. Iterate8. Persist our model9. Export to ONNX format 10. Deployment 11. Summary---  1. Load<jupyter_code>!pip install wget

import wget
import os

file_exist = os.path.exists('data.csv')

# Download the data
if file_exist == False:
    !wget -O data.csv bit.ly/2NC5P7f
import pandas as pd
import numpy as np

df = pd.read_csv('data.csv')<jupyter_output><empty_output><jupyter_text>2. ExploreUse df.info to get an overview of the datatypes and size of our dataset<jupyter_code>df.info()<jupyter_output><empty_output><jupyter_text>The size of dataset is: **6,362,620** rows with a total of **11 columns** The columns step, nameOrig and nameDest contains textual features We can use df.head to take a quick look the values in our dataset<jupyter_code>df.head(5)<jupyter_output><empty_output><jupyter_text>2.2 Find missing valuesWe can use pandas isna (is not a number) method to determine if we have any missing values in our dataframe<jupyter_code>df.isna().sum()<jupyter_output><empty_output><jupyter_text>Fortunately for us we do not have any missing values to address  3. TransformBefore we can start to work with our data, we need to transform it to a format an algorithm can understandTranforming the data may include:- Replacing any missing values, infinity or NaN- Transforming textual features  3.1 OneHotEncodingThe **nameOrig** and **nameDest** columns contain string values which we need to transform to float vectors.But before we jump in to those columns, let's take a look at the **type** column. Given that this column has finitive options, we can use OneHotEncoding to transform the values to binary options.Pandas provide a very useful function for this, get_dummies. We can utilize this function to create new binary columns.<jupyter_code>df = pd.concat([df, pd.get_dummies(df['type'], prefix='type')], axis=1)
df.head(5)<jupyter_output><empty_output><jupyter_text>3.2 Label EncoderThe nameOrig and nameDest columns contains a large number of permutations. OneHotEncoding could potentially create a very wide dataset, leading to out-of-memory exceptions, thus we can use something called label encoding instead to give each value a label.Given that a fraud generally can happen to anyone, we are going to assume that nameOrig does not matter<jupyter_code>from sklearn.preprocessing import LabelEncoder

le = LabelEncoder()

categorical_cols = ["nameDest"]
df[categorical_cols] = df[categorical_cols].apply(lambda col: le.fit_transform(col))
df.head(5)<jupyter_output><empty_output><jupyter_text>3.3 Plotting relationships<jupyter_code>import seaborn as sns
import matplotlib.pyplot as plt
g = sns.FacetGrid(df, col="type", hue="isFraud")
g.map(plt.scatter, "amount", "step", alpha=.7)
g.add_legend();
g = sns.FacetGrid(df, col="type", hue="isFraud")
g.map(plt.scatter, "amount", "newbalanceDest", alpha=.7)
g.add_legend();<jupyter_output><empty_output><jupyter_text>Interestingly **newbalanceDest** and **amount** do not correlate as we expect for fraudulent cases.  Many fraudulent cases appears to involve small amounts but also render the destination account with none or very little money<jupyter_code>import re, seaborn as sns, numpy as np, pandas as pd, random
from pylab import *
from matplotlib.pyplot import plot, show, draw, figure, cm
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
sns.set_style("whitegrid", {'axes.grid' : False})

fig = plt.figure(figsize=(6,6))
fill_colors = ['#FF9999' if wt==1 else '#FFE888' for wt in list(df['isFraud'])]

ax = Axes3D(fig) # Method 1

ax.scatter(df['newbalanceDest'], df['oldbalanceDest'], df['amount'], c=fill_colors, marker='o')
ax.set_xlabel('newBalanceDest')
ax.set_ylabel('oldBalanceDest')
ax.set_zlabel('Amount')

plt.show()<jupyter_output><empty_output><jupyter_text>It surely seems like we may be able to create some kind of separation using these columns. Let's create variables holding the features and labels.<jupyter_code>X = df[["amount", "newbalanceDest", "oldbalanceDest", "newbalanceOrig", "oldbalanceOrg", "nameDest", "type_CASH_IN", "type_CASH_OUT", "type_DEBIT", "type_PAYMENT", "type_TRANSFER"]]
Y = df["isFraud"]<jupyter_output><empty_output><jupyter_text>4. Split the dataTo evaluate the performance of the model, we will need a dataset for training and one for testing<jupyter_code>from sklearn.model_selection import train_test_split

X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.33, random_state=42)
X_train.shape<jupyter_output><empty_output><jupyter_text>5. Train our modelOur data is now clean and in the correct format to be fed in to a machine learning algorithm.For this example we'll use an ensemble algorithm consisting of number of decision trees The random forest classifier will in our case create 100 decision trees, with a max depth of 2.<jupyter_code>Y_train = Y_train.values.flatten()
Y_test = Y_test.values.flatten()
from sklearn.ensemble import RandomForestClassifier

clf = RandomForestClassifier(n_estimators=100, max_depth=2, random_state=0)
clf.fit(X_train, Y_train)<jupyter_output><empty_output><jupyter_text>6. Evaluate our modelBefore we can put our model to use, we'll need to know how good it is.  6.1 AccuracyWe can use the classifers score() method with the test dataset to determine the accuracy.<jupyter_code>metrics = clf.score(X_test, Y_test)

print(metrics)<jupyter_output><empty_output><jupyter_text>Not bad, an accuracy of **99.9%**!  However, if you recall, our dataset was highly unbalanced and if we just guessed non-fraudulent for all transactions we would also achieve this accuracy. Are there any other metrics we can use?  6.2 Precision-Recall - **Precision**: High precision means that all transactions we flagged as fraudulent actually were (no false positives)- **Recall**: High recall means that we did not miss any fraudulent transactions (no false negatives)It's difficult to achieve a high value for both. In our case, it would make the most sense to have a high recall, as it is more important to not miss any fraudulent transactions than to flag a couple of non-fraudulent cases as fraudulent<jupyter_code>from sklearn.metrics import classification_report
target_names = ['non-fraudulent', 'fraudulent']

pred = clf.predict(X_test)
print(classification_report(Y_test,pred, target_names=target_names))<jupyter_output><empty_output><jupyter_text>As the report indicates, the precision is good for non-fraudulent cases. However, our **recall for fraudulent cases is 0**, meaning that we miss all fraudulent cases. Although we don't have any false positives, non of our customers fraudulent transactions are beight caught If we look at the precision-recall curve, we see that the curve is pushed against the left, when we actually want it towards the right<jupyter_code>from sklearn.metrics import precision_recall_curve
import matplotlib.pyplot as plt

Y_score = clf.predict_proba(X_test)
Y_score = Y_score[:, 1]
Y_score = Y_score.flatten()

precisions, recalls, thresholds = precision_recall_curve(Y_test, Y_score)

plt.plot(thresholds, precisions[:-1], "b--", label="Precision")
plt.plot(thresholds, recalls[:-1], "g--", label="Recall")
plt.legend(loc="center left")
plt.ylim([0, 1])
plt.xlim([0, 1])

plt.show()<jupyter_output><empty_output><jupyter_text>6.3 Confusion matrixTo further explore the ratio between false positives, false negatives, true positives and true negatives, we can have a look at the confusion matrix.<jupyter_code>import numpy as np
import matplotlib.pyplot as plt

from sklearn.metrics import confusion_matrix
from sklearn.utils.multiclass import unique_labels

class_names =  ['Non-fraudulent', 'Fraudulent']


def plot_confusion_matrix(y_true, y_pred, classes,
                          normalize=False,
                          title=None,
                          cmap=plt.cm.Blues):
    """
    This function prints and plots the confusion matrix.
    Normalization can be applied by setting `normalize=True`.
    """
    if not title:
        if normalize:
            title = 'Normalized confusion matrix'
        else:
            title = 'Confusion matrix, without normalization'

    # Compute confusion matrix
    cm = confusion_matrix(y_true, y_pred)
    # Only use the labels that appear in the data
    #classes = classes[unique_labels(y_true, y_pred)]
    if normalize:
        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
        print("Normalized confusion matrix")
    else:
        print('Confusion matrix, without normalization')

    print(cm)

    fig, ax = plt.subplots(figsize=(10,5))
    im = ax.imshow(cm, interpolation='nearest', cmap=cmap)
    ax.figure.colorbar(im, ax=ax)
    # We want to show all ticks...
    ax.set(xticks=np.arange(cm.shape[1]),
           yticks=np.arange(cm.shape[0]),
           # ... and label them with the respective list entries
           xticklabels=classes, yticklabels=classes,
           title=title,
           ylabel='True label',
           xlabel='Predicted label')

    # Rotate the tick labels and set their alignment.
    plt.setp(ax.get_xticklabels(), rotation=45, ha="right",
             rotation_mode="anchor")

    # Loop over data dimensions and create text annotations.
    fmt = '.2f' if normalize else 'd'
    thresh = cm.max() / 2.
    for i in range(cm.shape[0]):
        for j in range(cm.shape[1]):
            ax.text(j, i, format(cm[i, j], fmt),
                    ha="center", va="center",
                    color="white" if cm[i, j] > thresh else "black")
    fig.tight_layout()
    return ax

np.set_printoptions(precision=2)
plot_confusion_matrix(Y_test, pred, classes=class_names, normalize=True,
                      title='Normalized confusion matrix')

plt.show()<jupyter_output><empty_output><jupyter_text>The confusion matrix tells us that we marked all fraudulent cases as non-fraudulent, not great.  7. IterateSo our model is far from great, let's see if we can improve it. One of the issues we are seeing right now is that despite it's accuracy, it is missing a lot of fraudulent transactions.A struggle we have is that our dataset is highly imbalanced. We can address this by weighing our inputs Let's train our model again, but boosting our minority class, is fraudulent<jupyter_code>from sklearn.ensemble import RandomForestClassifier

clf_with_weights = RandomForestClassifier(n_estimators=100, max_depth=2, random_state=0, class_weight="balanced")
clf_with_weights.fit(X_train, Y_train)
metrics = clf_with_weights.score(X_test, Y_test)

print(metrics)<jupyter_output><empty_output><jupyter_text>Our accuarcy dropped to **94.9%**, but let's see if we can outweight that with better recall-precision<jupyter_code>from sklearn.metrics import classification_report
target_names = ['non-fraudulent', 'fraudulent']

pred_with_weights = clf_with_weights.predict(X_test)
print(classification_report(Y_test, pred_with_weights, target_names=target_names))<jupyter_output><empty_output><jupyter_text>Wow! We can see that especially recall greatly increased for the fraudulent cases, meaning that we miss a lot less false negatives<jupyter_code>from sklearn.metrics import precision_recall_curve
import matplotlib.pyplot as plt

Y_score = clf_with_weights.predict_proba(X_test)
Y_score = Y_score[:, 1]
Y_score = Y_score.flatten()

precisions, recalls, thresholds = precision_recall_curve(Y_test, Y_score)

plt.plot(thresholds, precisions[:-1], "b--", label="Precision")
plt.plot(thresholds, recalls[:-1], "g--", label="Recall")
plt.legend(loc="center left")
plt.ylim([0, 1])
plt.xlim([0, 1])

plt.show()<jupyter_output><empty_output><jupyter_text>Confusion matrixLet's explore the confusion matrix for our new model<jupyter_code>plot_confusion_matrix(Y_test, pred_with_weights, classes=class_names, normalize=True,
                      title='Normalized confusion matrix')

plt.show()<jupyter_output><empty_output><jupyter_text>Much better. Although not perfect we see that only 8% of fraudulent cases are missed and marked as non-fraudulent. Similarly we mark 5% as fraudulent while they are not  8. Persist our modelThere are multiple ways in which we can persist our trained model. Let's look at using both scikit learn as well as Azure's Python SDK to do so - Using Scikit Learn<jupyter_code>from sklearn.externals import joblib
joblib.dump(value=clf_with_weights, filename="fraudulent-classifier.pkl")<jupyter_output><empty_output><jupyter_text>- Using Azure's Python SDK<jupyter_code>from azureml.core import Workspace
from azureml.core.model import Model

ws = Workspace.get(name="fraudulent-transactions", subscription_id='', resource_group='')
model = Model.register(workspace=ws, model_path="fraudulent-classifier.pkl", model_name="fraudulent-jupyter-model")<jupyter_output><empty_output><jupyter_text>9. ONNX FormatThe Open Neural Exchange Format (ONNX) allows models trained in different libraries to be loaded and used in others.ONNX is fully backed by Google, Facebook and Microsoft. Models trained in Python, e.g. using SciKitLearn or PyTorch can be exported to an ONNX format and imported and consumed by ML.NET<jupyter_code>from skl2onnx import convert_sklearn
from skl2onnx.common.data_types import FloatTensorType

initial_type = [('float_input', FloatTensorType([1, X_train.shape[1]]))]
onx = convert_sklearn(clf_with_weights, initial_types=initial_type)
with open("fraudulent-classifier-jupyter.onnx", "wb") as f:
    f.write(onx.SerializeToString())<jupyter_output><empty_output><jupyter_text>10. Deployment The final piece is to deploy our model to production so that we can use it to make accurate predictions.To be able to deploy our model, we will first need to create a Docker image for deployment. The image requireses two components 1. score.py -> loads the prediction engine2. conda.yml -> defines the requires dependenciesThe image can then be deployed to a running machine  10.1 Image creation<jupyter_code>#Create conda file
from azureml.core.conda_dependencies import CondaDependencies

fraudulentenv = CondaDependencies()
fraudulentenv.add_conda_package("scikit-learn")
 
with open("fraudulentenv.yml","w") as f:
    f.write(fraudulentenv.serialize_to_string())
with open("fraudulentenv.yml","r") as f:
    print(f.read())
%%writefile score.py
import json
import numpy as np
import os
import pickle
from sklearn.externals import joblib
from sklearn.linear_model import LogisticRegression

from azureml.core.model import Model

def init():
    global model
    # retrieve the path to the model file using the model name
    model_path = Model.get_model_path('fraudulent-jupyter-model')
    model = joblib.load(model_path)

def run(raw_data):
    data = np.array(json.loads(raw_data)['data'])
    # make prbediction
    y_hat = model.predict(data)
    return json.dumps(y_hat.tolist())<jupyter_output><empty_output><jupyter_text>10.2 Deployment We can tie altogether and create our Docker image as well as deploy it running service<jupyter_code>%%time
from azureml.core.webservice import Webservice
from azureml.core.image import ContainerImage
from azureml.core.webservice import AciWebservice

# configure the image
image_config = ContainerImage.image_configuration(execution_script="score.py", 
                                                  runtime="python", 
                                                  conda_file="fraudulentenv.yml")

aciconfig = AciWebservice.deploy_configuration(cpu_cores=1, 
                                               memory_gb=1, 
                                               tags={"method" : "sklearn"}, 
                                               description='Predict fraudulent transactions')

service = Webservice.deploy_from_model(workspace=ws,
                                       name='fraudulent-transactions-svc',
                                       deployment_config=aciconfig,
                                       models=[model],
                                       image_config=image_config)

service.wait_for_deployment(show_output=True)

from azureml.core.image import ContainerImage

image_config = ContainerImage.image_configuration(execution_script="score.py", 
                                                  runtime="python", 
                                                  conda_file="fraudulentenv.yml")<jupyter_output><empty_output><jupyter_text>10.3 PredictTo tie it altogheter, we can call our end-point with data to determine if this transaction is fraudulent or not<jupyter_code>print(service.scoring_uri)
!curl -X POST \
    -H 'Content-Type':'application/json' \
    -d '{"data": [[181.0, 0, 0, 0, 181.0, 439685, 0, 0, 0, 0, 1]]  }' \
    http://2f59f547-0890-44c4-a666-776e8ba9ce1b.southcentralus.azurecontainer.io/score<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08958837772397095, 'fraction_numerical': 0.010838233598524155, 'mean_word_length': 4.477423429112725, 'pattern_counts': {'":': 1, '<': 101, '<?xml version=': 0, '>': 104, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 20, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '25', 'dir': 'jupyter-structured-clean-dedup', 'id': '87989', 'license': 'MIT', 'n_tokens_mistral': 5434, 'n_tokens_neox': 5133, 'n_words': 1698, 'path': 'src/machine-learning/jupyter/jupytervm/fraudulent-transactions.ipynb', 'repo_name': 'bijub114/mldotnet-real-time-data-streaming-workshop', 'size': '17346'}