<jupyter_start><jupyter_text>Producing Dataframe of features for EDA from the JSON file<jupyter_code>!wget https://github.com/unravelin/code-test-data-science/blob/master/customers.json
import pandas as pd
import numpy as np
import sklearn as sk 

import requests
from pandas.io.json import json_normalize
import json

from numpy import mean
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import RepeatedStratifiedKFold
from xgboost import XGBClassifier

from sklearn.model_selection import train_test_split

%matplotlib inline
from xgboost import plot_importance
import matplotlib.pyplot as plt
import seaborn as sns

from sklearn.metrics import accuracy_score
from sklearn import metrics
from sklearn.metrics import plot_confusion_matrix

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import GaussianNB

#load in the data, I put it in local folder instead of using the WGET downloaded file from github because of issues with JSON.
data = pd.read_json("/content/test.json", lines= True)
data.head(5)
def flatten_json(nested_json: dict, exclude: list=[''], sep: str='_') -> dict:
    """
    Flatten a list of nested dicts.
    """
    out = dict()
    def flatten(x: (list, dict, str), name: str='', exclude=exclude):
        if type(x) is dict:
            for a in x:
                if a not in exclude:
                    flatten(x[a], f'{name}{a}{sep}')
        elif type(x) is list:
            i = 0
            for a in x:
                flatten(a, f'{name}{i}{sep}')
                i += 1
        else:
            out[name[:-1]] = x

    flatten(nested_json)
    return out
orders = pd.DataFrame([flatten_json(x) for x in data['orders']])
customers = pd.DataFrame([flatten_json(x) for x in data['customer']])
payment_method = pd.DataFrame([flatten_json(x) for x in data['paymentMethods']])
transactions = pd.DataFrame([flatten_json(x) for x in data['transactions']])
fraudulent = data['fraudulent']
#Applying the flatten json so that all features can be concated to one dataframe for usage

combined_data = pd.concat([orders, customers, payment_method, transactions, fraudulent], axis=1)
combined_data.head(3)<jupyter_output><empty_output><jupyter_text>Feature Engineering **Feature engineering**The feature engineering I will perform will be figuring out what features are most likely to effect if a given user is a potential fraud commiter. I will do this by training an XGboost algorithm on the categorical data in encoded form, and taking the most relevent features to add to the final training model.<jupyter_code>#What is problem type?
print(df['fraudulent'].head(5))

#Only true or false for fraud, will treat as binary classification problem<jupyter_output>0    False
1     True
2    False
3    False
4     True
Name: fraudulent, dtype: bool<jupyter_text>Could perform feature selection using PCA, but not a good fit for the problem as highest variance features may not actually be the most useful for classification<jupyter_code>#Replace all nan's with 0 so that scikitlearn can actually split
df = combined_data.fillna(0)

catagorical_var = df.select_dtypes(include=[object] )

#Drop catagoricals, will add back the useful ones
df = df.drop(catagorical_var, axis =1)<jupyter_output><empty_output><jupyter_text>We now need to encode the catagorical variables so that they can be analysed Apply One-Hot Encoding.Obviously the data could have been cleaned up to have far fewer columns, (eg: avg the order volumes for each user to have one column in place of 7, but as the further EDA will show, not all of these columns have the same contributions to the final classification and I fear that this would lose valuable information, so I have conciously chosed to keep the df with many columns.<jupyter_code>#Encoding procedure, make sparce matrix of df columns and drop duplicates.
onehot = pd.get_dummies(catagorical_var, drop_first=True)
onehot = onehot.loc[:,~onehot.columns.duplicated()]
#How unbalanced are the classes 
print(combined_data['fraudulent'].value_counts())
#Frac of around 60/160 fraud, will have to treat final training task as unbalance classification task for final model
#Split data for training and print shapes as sanity check that all my dimentions are right
onehot_X_train, onehot_X_test, onehot_y_train, onehot_y_test = train_test_split(onehot, df['fraudulent'], stratify= df['fraudulent'], test_size=0.2, random_state=1)

print(onehot_X_train.shape, onehot_X_test.shape, onehot_y_train.shape, onehot_y_test.shape,)
#Train model on just catagorical vars so that we can pick the ones with the highest value 

clf =XGBClassifier(max_depth=3, learning_rate=0.0025, n_estimators=500, objective='binary:logistic', booster='gbtree')
print(clf)
XGB=clf.fit(onehot_X_train,onehot_y_train)

onehot_prediction=XGB.predict(onehot_X_test)
#Taking a look at the model accuracy
print(metrics.classification_report(onehot_y_test, onehot_prediction))<jupyter_output>precision    recall  f1-score   support

       False       0.73      1.00      0.85        22
        True       1.00      0.33      0.50        12

    accuracy                           0.76        34
   macro avg       0.87      0.67      0.67        34
weighted avg       0.83      0.76      0.72        34<jupyter_text>We will show that the learning rate was intentionally kept lower than usual to decrease the number of false negatives at cost of increasing the number of false positives- it's useful to be overzealous with evaluating potential fraudsters then let them spip though. Confusion matrix will show **no** false negatives!<jupyter_code>print(metrics.confusion_matrix(onehot_y_test, onehot_prediction))<jupyter_output>[[22  0]
 [ 8  4]]<jupyter_text>From just feature embeds, F1 = 0.72, Not too bad right? Now obviously putting 3782 columns in as training data is irresponsibly slow, so  as a trade off I am going to decompose the result by feature importance, and see what can be cut and what can be kept.<jupyter_code>#Accuracy 
accuracy = accuracy_score(onehot_y_test, onehot_prediction)
print("Accuracy: %.2f%%" % (accuracy * 100.0))

#Plotting to show the most important features
plot_importance(clf, max_num_features= 19)
plt.figure(figsize=(16, 1800), dpi=80)
plt.show()<jupyter_output>Accuracy: 76.47%<jupyter_text>I am going to employ the same method again to<jupyter_code>#pick the features with F1> 20 to keep

features = onehot[['0_paymentMethodIssuer_Solace Banks', '0_paymentMethodIssuer_Her Majesty Trust', '3_orderState_fulfilled', '0_paymentMethodType_card', '0_paymentMethodIssuer_Vertex Bancorp',
                   '0_paymentMethodProvider_JCB 16 digit', '4_orderState_fulfilled', '2_paymentMethodType_card', '0_paymentMethodProvider_Mastercard', 
                   '0_paymentMethodIssuer_Rose Bancshares', '0_paymentMethodIssuer_Grand Credit Corporation', '3_paymentMethodType_card',
                   '1_paymentMethodType_card', '2_orderState_fulfilled', '1_paymentMethodProvider_JCB 16 digit', '0_paymentMethodRegistrationFailure_True',
                   '1_orderState_fulfilled', '0_paymentMethodProvider_VISA 16 digit', '0_paymentMethodType_paypal', ]]

#Doing this, we cut the dimentionality of the onehot encoded catagorical vars by a factor of 150 times at the cost of only 1-2% accuracy cost.

#Concat the features list to the numeric features to get our final df to apply ML to for an inbalanced binary classification problem.
df_features = pd.concat([features, df], axis=1)<jupyter_output><empty_output><jupyter_text>Train Model<jupyter_code>from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(df_features.drop(['fraudulent'], axis = 1), df_features['fraudulent'], stratify= df_features['fraudulent'], test_size=0.20, random_state=1, shuffle=True)

# fit xgboost on an imbalanced classification dataset

#Define model as unbalance classifier with ratio defined in the EDA section
model =XGBClassifier(max_depth=3, learning_rate=0.0025, n_estimators=500, objective='binary:logistic', booster='gbtree', scale_pos_weight=0.375)
print(model)
XGB=model.fit(X_train,y_train)


prediction=XGB.predict(X_test)

#Find the metrics for accuracy
print(metrics.classification_report(y_test, prediction))
print(metrics.confusion_matrix(y_test, prediction))
accuracy = accuracy_score(y_test, prediction)
print("Accuracy: %.2f%%" % (accuracy * 100.0))
#A more colourful version of the confusion matrix for clarity
matrix = plot_confusion_matrix(model, X_test, y_test,
                                 cmap=plt.cm.Blues,
                                 normalize='true')
plt.title('Confusion matrix for our classifier')
plt.show(matrix)
plt.show()<jupyter_output><empty_output><jupyter_text>Odd. the catagorical variables alone outperform the same as with the numerics.Maybe the catagorical vars are being overpowered by transforming them to int. We could try weight of evidence encoding (works well for the loan default classification kaggle challenge) instead of onehot. (ran out of time to try, would be interesting though!).<jupyter_code>plot_importance(clf, max_num_features= 30)
plt.figure(figsize=(16, 1800), dpi=80)
plt.show()<jupyter_output><empty_output><jupyter_text>It seems the inferanced are dominated by the categorical variables, with the most important features being the payment method issuer, if payment was by card, if the order was fulfilled and if the transaction failed.This could potentially show an overemphisis of the categorical variables, in the final section I will dicuss some methods that could be used to fix this if I had more time to improve the model. I will also quickly trial some simple ML methods such as an SVM, logistic regressor ect to see if our result can be beaten for accuracy.<jupyter_code>#Implimenting SVM on data 
from sklearn import svm

model_svm = svm.SVC(kernel='linear', C = 1.0)
model_svm.fit(X_train,y_train)

prediction_svm = model_svm.predict(X_test)
matrix = plot_confusion_matrix(model_svm, X_test, y_test,
                                 cmap=plt.cm.Blues,
                                 normalize='true')
plt.title('Confusion matrix for our classifier')
plt.show(matrix)
plt.show()<jupyter_output><empty_output><jupyter_text>Substandard accuracy when compared to XGboost, will next try Logistic regression<jupyter_code>from sklearn.linear_model import LogisticRegression
from sklearn import preprocessing
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler


scaler = preprocessing.StandardScaler().fit(X_train)
pipe = make_pipeline(StandardScaler(), LogisticRegression())
pipe.fit(X_train, y_train)
#Accuracy of simple logistic regression
print("Accuracy: ", pipe.score(X_test, y_test))<jupyter_output>Accuracy:  0.6470588235294118<jupyter_text>Still worse than XGBoost. Although I wont here, these models could be combined in a metaclass to create an ensemble to improve accuracy. Finally will try naive bayes, showing a roughly 65% accuracy.<jupyter_code>gnb = GaussianNB()
y_pred = gnb.fit(X_train, y_train).predict(X_test)

print("Number of mislabeled points out of a total %d points : %d"
% (X_test.shape[0], (y_test != y_pred).sum()))<jupyter_output>Number of mislabeled points out of a total 34 points : 10
{'fraction_non_alphanumeric': 0.07773571617219825, 'fraction_numerical': 0.01831147108020072, 'mean_word_length': 4.810741687979539, 'pattern_counts': {'":': 0, '<': 43, '<?xml version=': 0, '>': 45, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '122336', 'license': 'Apache-2.0', 'n_tokens_mistral': 3599, 'n_tokens_neox': 3380, 'n_words': 1175, 'path': 'Ravelin_task.ipynb', 'repo_name': 'cm2435/Coding-Interviews', 'size': '11359'}