<jupyter_start><jupyter_text>Imbalanced datasets<jupyter_code>import numpy as np
import pandas as pd

df_train = pd.read_csv('train.csv')
df_train

target_count = df_train.target.value_counts()
print('Class 0:', target_count[0])
print('Class 1:', target_count[1])
print('Proportion of class 0 is ', round(target_count[0] * 100 / (target_count[1] + target_count[0]), 2),'%')

target_count.plot(kind='bar', title='Count (target)');
df_train<jupyter_output><empty_output><jupyter_text>The metric trap  Applying our best traditional ML model (XGBoost)<jupyter_code>from xgboost import XGBClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# Remove 'id' and 'target' columns
labels = df_train.columns[2:]
df_train = df_train.dropna()

X = df_train[labels]
y = df_train['target']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1)

model = XGBClassifier()
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

accuracy = accuracy_score(y_test, y_pred)
print("Accuracy: %.2f%%" % (accuracy * 100.0))<jupyter_output>Accuracy: 96.29%<jupyter_text>Now let's run the same code, but using only one feature (which should drastically reduce the accuracy of the classifier):<jupyter_code>model = XGBClassifier()
model.fit(X_train[['ps_ind_04_cat']], y_train)
y_pred = model.predict(X_test[['ps_ind_04_cat']])

accuracy = accuracy_score(y_test, y_pred)
print("Accuracy: %.2f%%" % (accuracy * 100.0))<jupyter_output>Accuracy: 96.29%<jupyter_text>* with all the features - accuracy 96%* with only 1 feature - accuracy 96%* this means that Accuracy is a Bad-metric Confusion matrix<jupyter_code>from sklearn.metrics import confusion_matrix
from matplotlib import pyplot as plt

conf_mat = confusion_matrix(y_true=y_test, y_pred=y_pred)
print('Confusion matrix:\n', conf_mat)

labels = ['Class 0', 'Class 1']
fig = plt.figure()
ax = fig.add_subplot(111)
cax = ax.matshow(conf_mat, cmap=plt.cm.Blues)
fig.colorbar(cax)
ax.set_xticklabels([''] + labels)
ax.set_yticklabels([''] + labels)
plt.xlabel('Predicted')
plt.ylabel('Expected')
plt.show()<jupyter_output>Confusion matrix:
 [[33285     0]
 [ 1282     0]]<jupyter_text>* confusion matrix shows that model(XGB) always predicts class-0 only* so traditional ML model fails in imbalanced dataset<jupyter_code># solution for bad metric:
from sklearn.metrics import roc_auc_score
roc_auc_score(y_test, model.predict_proba(X_test[['ps_ind_04_cat']])[:, 1])<jupyter_output><empty_output><jupyter_text>* AUROC of 0.5 means baseline score (poor model)  Sampling Weights (ignore this section)<jupyter_code>from sklearn.linear_model import LogisticRegression
clf = LogisticRegression(random_state=0)
clf.fit(X_train,y_train)
print("Accuracy : ", clf.score(X_test,y_test) * 100, "%")
from sklearn.metrics import confusion_matrix, roc_auc_score
confusion_matrix(y_test, clf.predict(X_test))
roc_auc_score(y_test, clf.decision_function(X_test))
clf = LogisticRegression(class_weight = {0:5,1:95})
clf.fit(X_train,y_train)
print("Accuracy : ", clf.score(X_test,y_test) * 100, "%")
confusion_matrix(y_test, clf.predict(X_test))
roc_auc_score(y_test, clf.decision_function(X_test))<jupyter_output>Accuracy :  81.73648737941059 %<jupyter_text>Resampling ![](https://raw.githubusercontent.com/rafjaa/machine_learning_fecib/master/src/static/img/resampling.png)<jupyter_code># Class count
count_class_0, count_class_1 = df_train.target.value_counts()

# Divide by class
df_class_0 = df_train[df_train['target'] == 0]
df_class_1 = df_train[df_train['target'] == 1]

count_class_0, count_class_1<jupyter_output><empty_output><jupyter_text>Random under-sampling<jupyter_code>count_class_1, count_class_0
df_class_0_under = df_class_0.sample(count_class_1) #it will randomly sample count_class_1 number of rows from the dataframe
df_test_under = pd.concat([df_class_0_under, df_class_1], axis=0)

print('Random under-sampling:')
print(df_test_under.target.value_counts())

df_test_under.target.value_counts().plot(kind='bar', title='Count (target)');<jupyter_output>Random under-sampling:
0    4173
1    4173
Name: target, dtype: int64<jupyter_text>Random over-sampling<jupyter_code>df_class_1_over = df_class_1.sample(count_class_0, replace=True)
df_test_over = pd.concat([df_class_0, df_class_1_over], axis=0)

print('Random over-sampling:')
print(df_test_over.target.value_counts())

df_test_over.target.value_counts().plot(kind='bar', title='Count (target)');<jupyter_output>Random over-sampling:
0    109331
1    109331
Name: target, dtype: int64<jupyter_text>Python imbalanced-learn moduleA number of more sophisticated resapling techniques have been proposed in the scientific literature.For example, we can cluster the records of the majority class, and do the under-sampling by removing records from each cluster, thus seeking to preserve information. In over-sampling, instead of creating exact copies of the minority class records, we can introduce small variations into those copies, creating more diverse synthetic samples.Let's apply some of these resampling techniques, using the Python library [imbalanced-learn](http://contrib.scikit-learn.org/imbalanced-learn/stable/). It is compatible with scikit-learn and is part of scikit-learn-contrib projects.<jupyter_code>import imblearn     # creating our own imbalanced dataset<jupyter_output><empty_output><jupyter_text>For ease of visualization, let's create a small unbalanced sample dataset using the make_classification method:<jupyter_code>from sklearn.datasets import make_classification #subpackage used for creating own classifcation dataset
import pandas as pd

X, y = make_classification(
    n_classes=2, class_sep=0.75, weights=[0.9, 0.1],
    n_informative=2, n_redundant=0, flip_y=0,
    n_features=2, n_clusters_per_class=2,
    n_samples=200, random_state=10
)

df = pd.DataFrame(X)
df['target'] = y
#df
df.target.value_counts().plot(kind='bar', title='Count (target)');<jupyter_output><empty_output><jupyter_text>We will also create a 2-dimensional plot function, plot_2d_space, to see the data distribution:<jupyter_code># function for plotting
def plot_2d_space(X, y, label='Classes'):   
    colors = ['#1F77B4', '#FF7F0E']
    markers = ['o', 's']
    for l, c, m in zip(np.unique(y), colors, markers):
        plt.scatter(
            X[y==l, 0],
            X[y==l, 1],
            c=c, label=l, marker=m
        )
    plt.title(label)
    plt.legend(loc='upper right')
    plt.show()
# plotting
import matplotlib.pyplot as plt
import numpy as np

plot_2d_space(X, y, 'Imbalanced dataset (2 Features)')<jupyter_output><empty_output><jupyter_text>Random under-sampling and over-sampling with imbalanced-learn<jupyter_code>from imblearn.under_sampling import RandomUnderSampler

rus = RandomUnderSampler() #Intialise the model
X_rus, y_rus = rus.fit_resample(X, y) #resampling the data 

#print('Removed indexes:', id_rus)

plot_2d_space(X, y, 'Imbalanced dataset (2 Features)')
plot_2d_space(X_rus, y_rus, 'Random under-sampling')
from imblearn.over_sampling import RandomOverSampler

ros = RandomOverSampler() #Initialis the model 
X_ros, y_ros = ros.fit_resample(X, y) #Fit the model

#print(X_ros.shape[0] - X.shape[0], 'new random picked points')

plot_2d_space(X_ros, y_ros, 'Random over-sampling')
plot_2d_space(X, y, 'Original')<jupyter_output><empty_output><jupyter_text>* since the datapoints are duplicated, they overlap. So the graph looks similar<jupyter_code>y_ros.sum(), y.sum(), y_rus.sum()<jupyter_output><empty_output><jupyter_text>Under-sampling: Tomek linksTomek links are pairs of very close instances, but of opposite classes. Removing the instances of the majority class of each pair increases the space between the two classes, facilitating the classification process. ![](https://raw.githubusercontent.com/rafjaa/machine_learning_fecib/master/src/static/img/tomek.png?v=2) In the code below, we'll use ratio='majority' to resample the majority class.<jupyter_code>from imblearn.under_sampling import TomekLinks

tl = TomekLinks(sampling_strategy='majority')  # delete the majority datapoint
X_tl, y_tl = tl.fit_resample(X, y)

#print('Removed indexes:', id_tl)

plot_2d_space(X_tl, y_tl, 'Tomek links under-sampling')
plot_2d_space(X, y, 'Original')<jupyter_output><empty_output><jupyter_text>* still there is no clear boundary* thats why we never use tomek links alone. * we use tomek links + cluster centroids<jupyter_code>y_tl.sum()/y_tl.shape[0], y.sum()/y.shape[0]<jupyter_output><empty_output><jupyter_text>* 10% to 10.3%* very slight improvement only Under-sampling: Cluster Centroids<jupyter_code>from imblearn.under_sampling import ClusterCentroids

cc = ClusterCentroids(sampling_strategy={0: 20})    # in class-0, take 20 datapoints only
X_cc, y_cc = cc.fit_resample(X, y)

plot_2d_space(X_cc, y_cc, 'Cluster Centroids under-sampling')
#plot_2d_space(X, y, 'Original')
plot_2d_space(X_rus, y_rus, 'Random under-sampling')
y_cc.sum()/y_cc.shape[0]
#y_cc<jupyter_output><empty_output><jupyter_text>Over-sampling: SMOTESMOTE (Synthetic Minority Oversampling TEchnique) consists of synthesizing elements for the minority class, based on those that already exist. It works randomly picingk a point from the minority class and computing the k-nearest neighbors for this point. The synthetic points are added between the chosen point and its neighbors.  ![](https://raw.githubusercontent.com/rafjaa/machine_learning_fecib/master/src/static/img/smote.png) We'll use ratio='minority' to resample the minority class.<jupyter_code>from imblearn.over_sampling import SMOTE

plot_2d_space(X, y, 'Original Data')

smote = SMOTE(sampling_strategy='minority')   # increase my minority class
X_sm, y_sm = smote.fit_resample(X, y)

plot_2d_space(X_sm, y_sm, 'SMOTE over-sampling')<jupyter_output><empty_output><jupyter_text>SOLUTION : Over-sampling followed by under-samplingNow, we will do a combination of over-sampling and under-sampling, using the SMOTE and Tomek links techniques:<jupyter_code>from imblearn.combine import SMOTETomek    # oversample + undersample

plot_2d_space(X, y, 'Original Data')

smt = SMOTETomek(sampling_strategy='all')
X_smt, y_smt = smt.fit_resample(X, y)

plot_2d_space(X_smt, y_smt, 'SMOTE + Tomek links')
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score

lr = LogisticRegression()
lr.fit(X,y)
lr.score(X,y)

print( "AUROC on original dataset : ", roc_auc_score(y, lr.predict_proba(X)[:, 1]))

lr = LogisticRegression()
lr.fit(X_smt,y_smt)
lr.score(X_smt,y_smt)

print( "AUROC on the sampled dataset : ",roc_auc_score(y, lr.predict_proba(X)[:, 1]))<jupyter_output>AUROC on original dataset :  0.9513888888888888
AUROM on the sampled dataset :  0.9508333333333333
{'fraction_non_alphanumeric': 0.11401469357388636, 'fraction_numerical': 0.022505347344927, 'mean_word_length': 6.77584960231381, 'pattern_counts': {'":': 0, '<': 70, '<?xml version=': 0, '>': 70, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '120435', 'license': 'Apache-2.0', 'n_tokens_mistral': 4028, 'n_tokens_neox': 3822, 'n_words': 901, 'path': 'my_imbalanced_dataset.ipynb', 'repo_name': 'sidharth-ds/Machine-Learning', 'size': '10753'}