<jupyter_start><jupyter_text>Auto Insurance Fraud Detection Data preparation and Modeling Here we will prepare the data for the machine learning algorithms and asses the performance of multiple ML modelsThe Jupyter Notebook performing exploratory data analysis can be obtained [here](Insurance Fraud Detection-EDA.ipynb) Approach 1. Check for missing values and non-informative features2. Encoding of categorical values3. Split data into training and hold out testing4. Try out many models     * Hyperparameter tuning using grid-search    * Evaluation on hold-out testing data    * Performance analysis based on model evaluation metrics5. Select the best model The final model can then be deployed to flag a given insurance claim as fraudulent or legitimate  Importing all the necessary packages<jupyter_code>import pandas as pd
import numpy as np
import itertools
from matplotlib import pyplot as plt

from sklearn.preprocessing import LabelEncoder, OneHotEncoder, StandardScaler
from sklearn.model_selection import train_test_split, GridSearchCV, cross_val_score
#from sklearn.metrics import confusion_matrix, roc_curve, roc_auc_score, precision_score, recall_score, f1_score, precision_recall_curve


from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer

from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from xgboost.sklearn import XGBClassifier
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis



import sklearn.metrics
from pylab import rcParams
plt.style.use('seaborn')
%matplotlib inline

# Turning-off the warnings
def warn(*args, **kwargs):
    pass
import warnings
warnings.warn = warn<jupyter_output><empty_output><jupyter_text>Lets first import the datasetThe dataset is available at https://raw.githubusercontent.com/jodb/sparkTestData/master/insurance_claims.csv<jupyter_code>#load & view the data
df = pd.read_csv('insurance_claims.csv')
df.head(10)
# unique entries. Useful to know the catagorical features
df.nunique()<jupyter_output><empty_output><jupyter_text>Data preprocessing and cleaning<jupyter_code># Total number of missing values
df.isna().sum().sum()

# column with missing values 
df.columns[df.isna().any()]<jupyter_output><empty_output><jupyter_text>all of the missing values are in the _c39 column. We will drop it later.<jupyter_code># columns with ? entries
df.columns[(df == '?').any()]
df[df.columns[(df == '?').any()]].nunique()<jupyter_output><empty_output><jupyter_text>property_damage and police_report_available are boolean (YES/NO) types. Hence, we will regard the missing values as NO. The collision_type can be side, rear, and front. We will just regard the ? as a separate collision type.<jupyter_code>df['property_damage'].replace(to_replace='?', value='NO', inplace=True)
df['police_report_available'].replace(to_replace='?', value='NO', inplace=True)
# check if there are duplicated entries
df.duplicated(subset=None, keep='first').sum()
# dropping uninformative features
colsToDelete = ["policy_number", "policy_bind_date", "insured_zip", "incident_location", "incident_date", "_c39"]
df = df.drop(columns = colsToDelete, axis=1)
df.head()<jupyter_output><empty_output><jupyter_text>Identify and remove highly correlated features<jupyter_code>threshold = 0.97
# calculate correlations
corr_matrix = df.corr().abs()
# get the upper part of correlation matrix
upper = corr_matrix.where(np.triu(np.ones(corr_matrix.shape), k=1).astype(np.bool))

# columns with correlation above threshold
redundent = [column for column in upper.columns if any(upper[column] >= threshold)]
print(f'Columns to drop with correlation > {threshold}: {redundent}')
df.drop(columns=redundent, inplace=True)<jupyter_output>Columns to drop with correlation > 0.97: ['vehicle_claim']<jupyter_text>Now let's find the catagorical features  They should be one-hot-encode  Now let's find the catagorical features and one-hot-encode them<jupyter_code>num_features = df._get_numeric_data().columns
cat_features = list(set(df.columns) - set(num_features))
cat_features.remove('fraud_reported')
cat_features
df[num_features].head()
df[cat_features].head()<jupyter_output><empty_output><jupyter_text>Separate the target column from the features<jupyter_code># separate the target column from the features
y = df["fraud_reported"].map({"N":0, "Y":1})
X = df.drop("fraud_reported", axis=1)<jupyter_output><empty_output><jupyter_text>Set up a preprocessing pipeline to one-hot-encode catagorical features and let the numerical features passthrough as they are<jupyter_code>preprocessor = ColumnTransformer([("numerical", "passthrough", num_features), 
                                  ("categorical", OneHotEncoder(sparse=False, handle_unknown="ignore"),
                                   cat_features)])<jupyter_output><empty_output><jupyter_text>Define the ML models<jupyter_code># Logistic Regression
lr_model = Pipeline([("preprocessor", preprocessor), 
                     ("model", LogisticRegression(class_weight="balanced", solver="liblinear", random_state=42))])

# Decision Tree
dt_model = Pipeline([("preprocessor", preprocessor), 
                     ("model", DecisionTreeClassifier(class_weight="balanced"))])
# LDA
lda_model = Pipeline([("preprocessor", preprocessor), 
                     ("model", LinearDiscriminantAnalysis())])

# Random Forest
rf_model = Pipeline([("preprocessor", preprocessor), 
                     ("model", RandomForestClassifier(class_weight="balanced", n_estimators=100, n_jobs=-1))])

# XGBoost
xgb_model = Pipeline([("preprocessor", preprocessor), 
                      # Add a scale_pos_weight to make it balanced
                      ("model", XGBClassifier(scale_pos_weight=(1 - y.mean()), n_jobs=-1))])<jupyter_output><empty_output><jupyter_text>Lets define a function to automate  split the data into training and test sets and compute multiple evaluation matrices and plot ROC and PVR curves. Since we have an imbalanced dataset, we can't use metrics such as Accuracy. Instead, we will be mainly using the Receiver Operating Characteristic Area Under the Curve (ROC AUC). Precision, Recall, and fl-score are also other alternatives.     * It should be noted that the choice of the metric depends on the business requirement. Hence, models can be optimized accordingly. The 5xcv cross-validation gives a better sense of the average performance of the model.  First lets write a function to plot ROC and PVR curves<jupyter_code># A function to plot the ROC and PVR curves 
def plot_eval(testY, predY, auc):  
    fpr, tpr, thresh = sklearn.metrics.roc_curve(testY, predY[:,1])
    
    plt.plot(fpr, tpr, label='ROC curve (area = %.2f)' %auc)
    plt.plot([0, 1], [0, 1], linestyle='--', lw=2, color='r', label='Random guess')
    plt.title('ROC curve')
    plt.xlabel('False Positive Rate')
    plt.ylabel('True Positive Rate')
    plt.grid()
    plt.legend()
    plt.show()
    
    precision_rt, recall_rt, threshold_rt = sklearn.metrics.precision_recall_curve(testY, predY[:,1])
    plt.plot(recall_rt, precision_rt, linewidth=5, label='Precision-Recall curve')
    plt.title('Recall vs Precision')
    plt.xlabel('Recall')
    plt.ylabel('Precision')
    plt.show()
def model_evaluate(model, X, y, grid_params, plot_eval_curves = False):
    """Prepares a training and test set and evaluates the ML model
       on multiple metrices 
    
    Arguments:
    ---------
    model:        a defined ML model

    X:            the feature matix

    y:            the labels 

    grid_params:  hyperparameters to perform grid search on (dict)

    plot_eval_curves: If False, outputs metrices 
                      If True, plots ROC and precision vs. recall curves 
    """
    
    X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, test_size=.2, random_state=555)
    
    gs = GridSearchCV(model, grid_params, 
                  n_jobs=-1, cv=5, scoring="roc_auc")

    gs.fit(X_train, y_train)

    model.set_params(**gs.best_params_)
    
    model.fit(X_train, y_train)
    
    # Predict probabilities and labels
    probs = model.predict_proba(X_test)
    preds = model.predict(X_test)
    
    # Calculate ROC AUC
    auc = sklearn.metrics.roc_auc_score(y_test, probs[:, 1])
    # get the confusion matrix
    cnf_matrix = sklearn.metrics.confusion_matrix(y_test, preds)    
    # Plot ROC curve
    if plot_eval_curves:
        plot_eval(y_test, probs, auc)
    else: 
        print('Best Parameters:', gs.best_params_)
        print('Best Score:', gs.best_score_)
        print(f'ROC AUC: {round(auc, 4)}')
        print(f'Confusion Matrix:\n {cnf_matrix}')
        # compute the other evaluation metrices 
        for metric in [sklearn.metrics.precision_score, sklearn.metrics.recall_score, sklearn.metrics.f1_score]:
            print(f'{metric.__name__}: {round(metric(y_test, preds), 4)}')

        # Average performance using 5 x cross-validation    
        score = cross_val_score(model, X, y, cv=5, scoring='roc_auc')
        print('Cross-validation AUC score: ', score.mean()) 
    return model, cnf_matrix



# Grid-search hyper Parameters to consider
model_grid_params = {'lr_model': {"model__C": [1, 1.3, 1.5]}, 'dt_model': {"model__max_depth": [3, 5, 7], "model__min_samples_split": [2, 5]}, 
              'rf_model': {"model__max_depth": [20, 10, 15],"model__min_samples_split": [5, 10]}, 'lda_model': {}, 
               'xgb_model':{"model__max_depth": [5, 10], "model__min_child_weight": [5, 10]}}
models = [lr_model, dt_model, rf_model, lda_model, xgb_model]
model_keys = [('lr_model', 'Logistic Regression'),
              ('dt_model', 'Decision Tree'),
              ('rf_model', 'Random Forest'),
              ('lda_model', 'Linear Discriminant Analysis'),
              ('xgb_model', 'Gradient Boosting')]
final_model, cnf_matrix = {}, {}
for idx, model in enumerate(models):
    print(45*'_', '\n{}'.format(model_keys[idx][1]))
    model, cnf = model_evaluate(model, X, y, model_grid_params[model_keys[idx][0]])
    final_model[model_keys[idx][0]] = model
    cnf_matrix[model_keys[idx][0]] = cnf<jupyter_output>_____________________________________________ 
Logistic Regression
Best Parameters: {'model__C': 1}
Best Score: 0.595018369887688
ROC AUC: 0.5656
Confusion Matrix:
 [[76 75]
 [24 25]]
precision_score: 0.25
recall_score: 0.5102
f1_score: 0.3356
Cross-validation AUC score:  0.5931934946163896
_____________________________________________ 
Decision Tree
Best Parameters: {'model__max_depth': 3, 'model__min_samples_split': 2}
Best Score: 0.825266707724094
ROC AUC: 0.8241
Confusion Matrix:
 [[126  25]
 [  9  40]]
precision_score: 0.6154
recall_score: 0.8163
f1_score: 0.7018
Cross-validation AUC score:  0.8493709960805514
_____________________________________________ 
Random Forest
Best Parameters: {'model__max_depth': 10, 'model__min_samples_split': 10}
Best Score: 0.8541817453379954
ROC AUC: 0.8101
Confusion Matrix:
 [[127  24]
 [ 17  32]]
precision_score: 0.5714
recall_score: 0.6531
f1_score: 0.6095
Cross-validation AUC score:  0.8415234491147453
___________________________________________[...]<jupyter_text>Overall, the LDA model provided the better result (87% AUC and 71% fl_score) .  The Decision Tree model is also as good as the LDA.  Although, the RF classifier has a comparable  AUC score the f1_score is very low. Hence, we will take the LDA or the Decision Tree model as our final model  Lets have a closer look at the predictions of the LDA model  Lets have some helper functionsTo plot the confusion matrix, the code from the [sklearn documentation](http://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html) is used.<jupyter_code>def plot_confusion_matrix(cm,
                          target_names,
                          title='Confusion matrix',
                          cmap=None,
                          normalize=True):
    """
    given a sklearn confusion matrix (cm), make a nice plot

    Arguments
    ---------
    cm:           confusion matrix from sklearn.metrics.confusion_matrix

    target_names: given classification classes such as [0, 1, 2]
                  the class names, for example: ['high', 'medium', 'low']

    title:        the text to display at the top of the matrix

    cmap:         the gradient of the values displayed from matplotlib.pyplot.cm
                  see http://matplotlib.org/examples/color/colormaps_reference.html
                  plt.get_cmap('jet') or plt.cm.Blues

    normalize:    If False, plot the raw numbers
                  If True, plot the proportions

    Citiation
    ---------
    http://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html

    """

    if cmap is None:
        cmap = plt.get_cmap('Blues')

    plt.figure(figsize=(8, 6))
    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()

    if target_names is not None:
        tick_marks = np.arange(len(target_names))
        plt.xticks(tick_marks, target_names, rotation=45)
        plt.yticks(tick_marks, target_names)

    if normalize:
        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]


    thresh = cm.max() / 1.5 if normalize else cm.max() / 2
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        if normalize:
            plt.text(j, i, "{:0.4f}".format(cm[i, j]),
                     horizontalalignment="center",
                     color="white" if cm[i, j] > thresh else "black")
        else:
            plt.text(j, i, "{:,}".format(cm[i, j]),
                     horizontalalignment="center",
                     color="white" if cm[i, j] > thresh else "black")


    plt.tight_layout()
    plt.ylabel('True label')
    plt.show()
# Plot the confusion matrix of the LDA model
plot_confusion_matrix(cnf_matrix['lda_model'], target_names=['legitimate', 'fraud'], normalize=False)<jupyter_output><empty_output><jupyter_text>- 135 legitimate transactions were correctly classified - 16 legitimate transactions classified as fraudulant, i.e., false positives (type 1 error) - 36 fraudulant transactions transactions were correctly identified  - 13 fraudulant transactions were classified as legitimate (type 2 error)<jupyter_code># Plot the confusion matrix of the DT model
plot_confusion_matrix(cnf_matrix['dt_model'], target_names=['legitimate', 'fraud'], normalize=False)<jupyter_output><empty_output><jupyter_text>The ROC curve<jupyter_code># ROC curve LDA 
model, cnf = model_evaluate(lda_model, X, y, model_grid_params['lda_model'], plot_eval_curves=True)
# ROC and Precision vs. Recall curves of the LDA model 
model, cnf = model_evaluate(lda_model, X, y, model_grid_params['lda_model'], plot_eval_curves=True)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1068183347916807, 'fraction_numerical': 0.020730968567005452, 'mean_word_length': 4.097084048027444, 'pattern_counts': {'":': 9, '<': 55, '<?xml version=': 0, '>': 60, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '27631', 'license': 'MIT', 'n_tokens_mistral': 4861, 'n_tokens_neox': 4486, 'n_words': 1287, 'path': 'Insurance Fraud Detection-modeling.ipynb', 'repo_name': 'daveandex/Auto-Insurance-Fraud-Detection', 'size': '14857'}