<jupyter_start><jupyter_text>Table of Contents1&nbsp;&nbsp;Data Description2&nbsp;&nbsp;Imports3&nbsp;&nbsp;Useful Scripts4&nbsp;&nbsp;Load the data5&nbsp;&nbsp;Train test split with stratify6&nbsp;&nbsp;Classification: isolationForest7&nbsp;&nbsp;Classification: LOF (Local Outlier Factor)8&nbsp;&nbsp;Classification Evaluation Metrics  Data DescriptionThe datasets contains transactions made by credit cards in September2013 by european cardholders.This dataset presents transactions that occurred in two days,where we have 492 frauds out of 284,807 transactions. The dataset is highly unbalanced, the positive class (frauds)account for 0.172% of all transactions.It contains only numerical input variables which arethe result of a PCA transformation.Unfortunately, due to confidentiality issues,we cannot provide the original features and more background information about the data.Features V1, V2, ... V28 are the principalcomponents obtained with PCA,the only features which have not been transformed with PCA are 'Time' and 'Amount'. Feature 'Time' contains the seconds elapsed between each transactionand the first transaction in the dataset. The feature 'Amount'is the transaction Amount, this feature can be used for example-dependant cost-senstive learning. Feature 'Class' is the response variable and it takes value1 in case of fraud and 0 otherwise.  Imports<jupyter_code>import bhishan
%load_ext autoreload

%autoreload 2
import numpy as np
import pandas as pd
import seaborn as sns
sns.set(color_codes=True)

import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline

import os
import time

# random state
random_state=100
np.random.seed(random_state) # we need this in each cell
np.random.set_state=random_state

# Jupyter notebook settings for pandas
pd.set_option('display.max_columns', 200)
pd.set_option('display.max_rows', 100) # None for all the rows
pd.set_option('display.max_colwidth', 50)

print([(x.__name__,x.__version__) for x in [np, pd,sns,matplotlib]])
import scipy
from scipy import stats
# scale and split
from sklearn.preprocessing import MinMaxScaler, StandardScaler, RobustScaler
from sklearn.model_selection import train_test_split
from sklearn.model_selection import StratifiedKFold
# classifiers
from sklearn.ensemble import IsolationForest
from sklearn.neighbors import LocalOutlierFactor
from sklearn.svm import OneClassSVM
# roc auc etc scores
from sklearn.metrics import auc
from sklearn.metrics import roc_auc_score
from sklearn.metrics import precision_score
from sklearn.metrics import recall_score
from sklearn.metrics import f1_score
from sklearn.metrics import accuracy_score
from sklearn.metrics import average_precision_score
# roc auc curves
from sklearn.metrics import roc_curve
from sklearn.metrics import precision_recall_curve
# confusion matrix
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
from sklearn.metrics import accuracy_score
from sklearn.metrics import precision_score
from sklearn.metrics import recall_score
from sklearn.metrics import f1_score
from sklearn.metrics import precision_recall_fscore_support
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix<jupyter_output><empty_output><jupyter_text>Useful Scripts<jupyter_code>def show_method_attributes(method, ncols=7,exclude=None):
    """ Show all the attributes of a given method.
    Example:
    ========
    show_method_attributes(list)
     """
    x = [I for I in dir(method) if I[0]!='_' ]
    x = [I for I in x 
         if I not in 'os np pd sys time psycopg2'.split()
         if (exclude not in i)
        ]
             

    return pd.DataFrame(np.array_split(x,ncols)).T.fillna('')<jupyter_output><empty_output><jupyter_text>Load the data<jupyter_code>df = pd.read_csv('../data/raw/creditcard.csv.zip',compression='zip')
print(df.shape)
df.head()<jupyter_output>(284807, 31)<jupyter_text>Train test split with stratify<jupyter_code>from sklearn.model_selection import train_test_split

target = 'Class'
Xtrain, Xtest, ytrain, ytest = train_test_split(
    df.drop(target,axis=1), df[target],
    test_size=0.2, random_state=random_state, stratify=df[target])

df.shape, Xtrain.shape, Xtest.shape
outlier_fraction = ytrain[ytrain==1].shape[0] /   ytrain[ytrain==0].shape[0]<jupyter_output><empty_output><jupyter_text>Classification: isolationForestReference:   scikit learn  https://scikit-learn.org/stable/modules/outlier_detection.htmlOne efficient way of performing outlier detection in high-dimensional datasets is to use random forests. The ensemble.IsolationForest ‘isolates’ observations by randomly selecting a feature and then randomly selecting a split value between the maximum and minimum values of the selected feature.Since recursive partitioning can be represented by a tree structure, the number of splittings required to isolate a sample is equivalent to the path length from the root node to the terminating node.This path length, averaged over a forest of such random trees, is a measure of normality and our decision function.Random partitioning produces noticeably shorter paths for anomalies. Hence, when a forest of random trees collectively produce shorter path lengths for particular samples, they are highly likely to be anomalies.The implementation of ensemble.IsolationForest is based on an ensemble of tree.ExtraTreeRegressor. Following Isolation Forest original paper, the maximum depth of each tree is set to  where  is the number of samples used to build the tree (see (Liu et al., 2008) for more details).This algorithm is illustrated below.![](../images/isolation_forest_sklearn.png)<jupyter_code>%%time

clf_iso = IsolationForest(n_estimators=100,
                      max_samples=len(Xtrain),
                      n_jobs=-1,
                      contamination=outlier_fraction,
                      random_state=random_state,
                      verbose=1,
                      behaviour='new')


# fitting  
clf_iso.fit(Xtrain)

# prediction
scores_prediction = clf_iso.decision_function(Xtest)
ypreds = clf_iso.predict(Xtest)

# algorithm gives 1 and -1 classes, make them 0 and 1
ypreds[ypreds == 1] = 0
ypreds[ypreds == -1] = 1

ypreds_iso = ypreds

print(classification_report(ytest, ypreds_iso))
cm = confusion_matrix(ytest,ypreds_iso)
cm
vals = cm.ravel()
vals
print('Isolation Forest Results')
print('-'*25)
print('Total Frauds: ', vals[2] + vals[3])
print('Incorrect Frauds: ', vals[2])
print('Incorrect Percent: ', round(vals[2]*100/(vals[2]+vals[3]),2),'%')<jupyter_output>Isolation Forest Results
-------------------------
Total Frauds:  98
Incorrect Frauds:  73
Incorrect Percent:  74.49 %<jupyter_text>Classification: LOF (Local Outlier Factor)Reference: sklearn  Unsupervised Outlier Detection using Local Outlier Factor (LOF)The anomaly score of each sample is called Local Outlier Factor. It measures the local deviation of density of a given sample with respect to its neighbors. It is local in that the anomaly score depends on how isolated the object is with respect to the surrounding neighborhood. More precisely, locality is given by k-nearest neighbors, whose distance is used to estimate the local density. By comparing the local density of a sample to the local densities of its neighbors, one can identify samples that have a substantially lower density than their neighbors. These are considered outliers.<jupyter_code>%%time

clf_lof = LocalOutlierFactor(n_neighbors=20,
                      algorithm='auto', 
                      leaf_size=30,
                      metric='minkowski',
                      p=2,
                      n_jobs=-1,
                      metric_params=None,
                      novelty=True,
                      contamination=outlier_fraction)



clf_lof.fit(Xtrain)
ypreds = clf_lof.predict(Xtest)

# alogrithm gives 1 and -1 classes, make them 0 and 1
ypreds[ypreds == 1] = 0
ypreds[ypreds == -1] = 1

ypreds_lof = ypreds

print(classification_report(ytest, ypreds_lof))
cm = confusion_matrix(ytest,ypreds_lof)
vals = cm.ravel()

cm
print('Local Outliers Factor Results')
print('-'*25)
print('Total Frauds: ', vals[2] + vals[3])
print('Incorrect Frauds: ', vals[2])
print('Incorrect Percent: ', round(vals[2]*100/(vals[2]+vals[3]),2),'%')<jupyter_output>Local Outliers Factor Results
-------------------------
Total Frauds:  98
Incorrect Frauds:  95
Incorrect Percent:  96.94 %<jupyter_text>Classification Evaluation Metrics![](../images/classification_metrics.png)![](../images/confusion_matrix.png)![](../images/accuracy.png)![](../images/precision.png)![](../images/recall.png)![](../images/f1_score.png)<jupyter_code># ! for filename in ../images/*.png; do echo '!'"[]($filename)"; done
from sklearn.metrics import (accuracy_score,precision_score,
                                recall_score,f1_score,matthews_corrcoef,
                                average_precision_score,roc_auc_score,
                                cohen_kappa_score)
   
# scalar metrics
ypreds = ypreds_iso
acc = accuracy_score(ytest,ypreds)
precision = precision_score(ytest,ypreds)
recall = recall_score(ytest,ypreds)
f1 = f1_score(ytest,ypreds)
mcc = matthews_corrcoef(ytest,ypreds)
kappa = cohen_kappa_score(ytest, ypreds)

print(acc,precision,recall,f1,mcc,kappa)
# area under the curves
try:
    y_score = clf_iso.predict_proba(Xtest)[:,1]
except:
    y_score = clf_iso.decision_function(Xtest)

auprc = average_precision_score(ytest, y_score)
auroc = roc_auc_score(ytest, y_score)

print(auprc, auroc)<jupyter_output>0.0008812221863141853 0.054973269555430505
{'fraction_non_alphanumeric': 0.0884467131992916, 'fraction_numerical': 0.018647775809980206, 'mean_word_length': 4.587892898719441, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '645355', 'license': 'Apache-2.0', 'n_tokens_mistral': 3040, 'n_tokens_neox': 2805, 'n_words': 885, 'path': 'notebooks/b03_classification_modelling_isolattionForest_LOF.ipynb', 'repo_name': 'jokerNi/Project_Fraud_Detection', 'size': '9599'}