<jupyter_start><jupyter_text>Model SelectionIn this notebook, we will focus on model selection. We first import all libraries will be using, define some functions and import the dataset we will be using through out the notebook.<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
import time


%matplotlib inline
pd.plotting.register_matplotlib_converters()

from sklearn.preprocessing import StandardScaler, MinMaxScaler
from sklearn.cluster import KMeans, DBSCAN
from sklearn.neighbors import LocalOutlierFactor
from sklearn.manifold import TSNE
from sklearn import metrics

sns.set(style = "ticks")<jupyter_output><empty_output><jupyter_text>Next, we apply the same transformations we have done in Load & Cleanse notebook to get the dataframe ready.<jupyter_code>file_path = "../data/data.csv"
data = pd.read_csv(file_path, index_col = "consumer_id")

cols_with_na = [col for col in data.columns if data[col].isnull().any()]

data.drop(cols_with_na + ["account_status"], axis = 1, inplace = True)<jupyter_output><empty_output><jupyter_text>Clustering and LabelingThe sample data is not labeled. So, we cannot use any supervised algorithms. Therefore, we will first label the data using a clustering algorithm and then use a tree-based classification algorithm to train a classification model. This way, we will be able to classify unseen data based on the classification model we have built.In this section we will try a couple of different approaches. Namely, we will use _k_-means, DBSCAN and LOF to get clusters. Although LOF is not really a clustering method, I think it will be useful in our case for identifying outliers and label them. We will use the silhouette score to measure the efficiency of the algorithm in identifying the clusters. We will use the following function that will help us to visualise the clusters with t-SNE.<jupyter_code>def tsne_plot(d, m, n_color):
    """
    Function to visualise clusters with t-SNE for a given dataset d and model m
    """
    d_copy = d.copy()
    
    d_copy["tsne-d1"] = m[:, 0]
    d_copy["tsne-d2"] = m[:, 1]

    plt.figure(figsize=(10,10))
    sns.scatterplot(
        x="tsne-d1", y="tsne-d2",
        palette=sns.color_palette("hls", n_color),
        hue = "label",
        data=d_copy,
        legend="full",
        alpha=0.3
    )<jupyter_output><empty_output><jupyter_text>_k_-meansWe first will scale the dataset using `MinMaxScaler` module. We prefer to use `MinMaxScaler` rather than `StandardScaler` because we are looking for outliers. The outliers have an influence when computing mean and standard deviation which shrink the range of the feature values. Then, we will build a _k_-means clustering with `n_clusters=2`. We also keep original index and column names for demonstration purposes.<jupyter_code>n_clusters = 2
data_norm_k = pd.DataFrame(MinMaxScaler().fit_transform(data), index = data.index)
k_means = KMeans(n_clusters=n_clusters, random_state=123, init="k-means++").fit_predict(data_norm_k)
data_norm_k.columns = data.columns
data_norm_k["label"] = k_means + 1
data_norm_k["label"] = data_norm_k["label"].apply(lambda i: str(i))<jupyter_output><empty_output><jupyter_text>Let's calculate the silhouette score. Yo can see that silhouette score is pretty low.<jupyter_code>score = metrics.silhouette_score(data_norm_k[data.columns], data_norm_k["label"])
print("For n_clusters = {}, silhouette score is {}.".format(n_clusters, score))<jupyter_output>For n_clusters = 2, silhouette score is 0.5346240793329303.<jupyter_text>Next, we will visualise the clusters using t-SNE. We already have function defined at the beginning of this section to plot the result. We reduce the dimensionality first and then plot the clusters. From the plot below, it is clear that we cannot separate the clusters properly.<jupyter_code>start_time = time.time()
tsne = TSNE(n_components = 2, perplexity = 30, n_iter = 500, verbose = 0, random_state = 123, learning_rate=50)
tsne_results = tsne.fit_transform(data_norm_k[data.columns])

print('t-SNE done! Time elapsed: {} seconds'.format(time.time()-start_time))

tsne_plot(data_norm_k, tsne_results, 2)
#plt.savefig("p_kmeans.png")<jupyter_output>t-SNE done! Time elapsed: 16.938331127166748 seconds<jupyter_text>DBSCANWe will follow a similar approach to the previous section. We will not be able to set the number of clusters here since DBSCAN finds clusters automatically.<jupyter_code>data_norm_db = pd.DataFrame(MinMaxScaler().fit_transform(data), index = data.index)
db_scan = DBSCAN(eps = 0.99, min_samples = 100, metric = "euclidean", n_jobs = -1).fit(data_norm_db)
data_norm_db.columns = data.columns
data_norm_db["label"] = db_scan.labels_
data_norm_db["label"] = data_norm_db["label"].apply(lambda i: str(i))
data_norm_db.head()<jupyter_output><empty_output><jupyter_text>As you can see we have very similar silhouette score to _k_-means model.<jupyter_code>score_db = metrics.silhouette_score(data_norm_db[data.columns], data_norm_db["label"])
n_clusters_ = len(set(db_scan.labels_)) - (1 if -1 in db_scan.labels_ else 0)
print("For n_clusters = {}, silhouette score is {}.".format(n_clusters_, score_db))<jupyter_output>For n_clusters = 4, silhouette score is 0.5931621877614222.<jupyter_text>We have already calculated t-SNE. So, we will simply use the already calculated one. And, it is apparent, DBSCAN is finding more than 2 clusters with very few outliers.<jupyter_code>tsne_plot(data_norm_db, tsne_results, n_clusters_ + 1)<jupyter_output><empty_output><jupyter_text>Local Outlier Factor (LOF)LOF uses density-based outlier detection to identify local outliers. A point is labeled as an outlier if the density around that point is significantly different from the density around its neighbors. Standardising the dataset for LOF does not change the result as it utilises Manhattan metric. But, we will use standardasiation so that t-SNE will produce similar plots.<jupyter_code>#data_lof = data.copy()
data_lof = pd.DataFrame(MinMaxScaler().fit_transform(data), index = data.index)
lof = LocalOutlierFactor(n_neighbors=50, contamination=0.1).fit_predict(data_lof)
data_lof.columns = data.columns
data_lof["label"] = lof
data_lof["label"] = data_lof["label"].apply(lambda i: str(i))
data_lof.head()
start_time = time.time()
tsne = TSNE(n_components = 2, perplexity = 50, n_iter = 500, verbose = 0, random_state = 123, learning_rate=50)
tsne_results_lof = tsne.fit_transform(data_lof[data.columns])

print('t-SNE done! Time elapsed: {} seconds'.format(time.time()-start_time))

tsne_plot(data_lof, tsne_results_lof, 2)
#plt.savefig("p_lot.png")<jupyter_output>t-SNE done! Time elapsed: 18.933717966079712 seconds<jupyter_text>ConclusionsBased on the different algorithms we have used in the previous section, we can see that _k_-means and LOF produces usable clustering options. It is not up to use to figure out which resulting dataset to use. _k_-means labels around 44% of the cases as fraudulent whereas LOF classifies only 10% of those cases.<jupyter_code>perc_lof = ((data_lof["label"] == "-1").sum() / len(data_lof["label"])) * 100
perc_kmeans = ((data_norm_k["label"] == "2").sum() / len(data_norm_k["label"])) * 100

print("LOF classifies {}% of the sample data as fraudulent whereas k-means classifies {:2.2f}% of cases as fraudulent.".format(perc_lof, perc_kmeans))<jupyter_output>LOF classifies 9.99% of the sample data as fraudulent whereas k-means classifies 44.33% of cases as fraudulent.
{'fraction_non_alphanumeric': 0.09596903363587828, 'fraction_numerical': 0.0202883075280299, 'mean_word_length': 5.720179372197309, 'pattern_counts': {'":': 0, '<': 40, '<?xml version=': 0, '>': 40, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '486461', 'license': 'MIT', 'n_tokens_mistral': 2552, 'n_tokens_neox': 2363, 'n_words': 782, 'path': 'notebooks/02_model_selection.ipynb', 'repo_name': 'mhasanbulli/at_ml', 'size': '7492'}