<jupyter_start><jupyter_text>Edited Nearest NeighboursThe algorithms works as follows:1) Train a KNN algorithm on the data (user defines number of neighbours, typically 3)2) Find the 3 nearest neighbour to each observation (or the number defined by the user in 1)3) Find the label of each of the neighbours (we know it, is the target in the dataset)**Two undersampling strategies:**4) **mode**: if **the majority of the neighbours** show the same label as the observation, then we keep the observation5) **all**: if **all the neighbours** show the same label as the observation, then we keep the observation6) Alternatively, we remove the observation of the dataset - **undersample**7) The undersampled dataset, is the one left after removing observations===The algorithm removes observations which class are different from that of their neighbours. These are generally the observations difficult to classify and / or that introduce noise.**In practice, samples of the majority class that are too similar to an observation of the minority class will be removed.**So this is, in essence, the opposite of Condensed NN.<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split

from imblearn.under_sampling import EditedNearestNeighbours<jupyter_output><empty_output><jupyter_text>Create dataWe will create data where the classes have different degrees of separateness.https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_classification.html<jupyter_code>def make_data(sep):
    
    # returns arrays
    X, y = make_classification(n_samples=1000,
                           n_features=2,
                           n_redundant=0,
                           n_clusters_per_class=1,
                           weights=[0.99],
                           class_sep=sep,# how separate the classes are
                           random_state=1)
    
    # trasform arrays into pandas df and series
    X = pd.DataFrame(X, columns =['varA', 'varB'])
    y = pd.Series(y)
    
    return X, y<jupyter_output><empty_output><jupyter_text>Undersample with Edited Nearest Neighbours[EditedNearestNeighbours](https://imbalanced-learn.org/stable/references/generated/imblearn.under_sampling.EditedNearestNeighbours.html) Well separated classes<jupyter_code># create data

X, y = make_data(sep=2)

# set up edited nearest neighbour

enn = EditedNearestNeighbours(
    sampling_strategy='auto',  # undersamples only the majority class
    n_neighbors=3, # the number of neighbours to examine
    kind_sel='all',  # all neighbours need to have the same label as the observation examined
    n_jobs=4)  # I have 4 cores in my laptop

X_resampled, y_resampled = enn.fit_resample(X, y)
# size of original data

X.shape, y.shape
# size of undersampled data

X_resampled.shape, y_resampled.shape<jupyter_output><empty_output><jupyter_text>The procedure removed a few observations closer to the minority class.<jupyter_code># number of minority class observations

y.value_counts()
# plot original data

sns.scatterplot(
        data=X, x="varA", y="varB", hue=y
    )

plt.title('Original dataset')
plt.show()
# plot undersampled data

sns.scatterplot(
        data=X_resampled, x="varA", y="varB", hue=y_resampled
    )

plt.title('Undersampled dataset')
plt.show()<jupyter_output><empty_output><jupyter_text>Note how some yellow dots can now be seen clearer, as their neighbouring observations from the majority class were removed from the data set. Edited Nearest Neighbours removes observations from the majority class that are too similar to some of those in the minority class.  Partially separated classesLet's repeat the same exercise in data where the classes are not so clearly separated.<jupyter_code># create data
X, y = make_data(sep=0)

# set up edited nearest neighbour transformer

enn = EditedNearestNeighbours(
    sampling_strategy='auto',  # resamples the majority class
    n_neighbors=3, # the number of neighbours to examine
    kind_sel='all',  # all neighbours need to agree
    n_jobs=4)  # I have 4 cores in my laptop


X_resampled, y_resampled = enn.fit_resample(X, y)
# original data

X.shape, y.shape
# undersampled data

X_resampled.shape, y_resampled.shape<jupyter_output><empty_output><jupyter_text>As the classes are not so clearly distinguished, more samples were removed from the dataset.<jupyter_code># plot original data

sns.scatterplot(
        data=X, x="varA", y="varB", hue=y
    )

plt.title('Original dataset')
plt.show()
# plot undersampled data

sns.scatterplot(
        data=X_resampled, x="varA", y="varB", hue=y_resampled
    )

plt.title('Undersampled dataset') 
plt.show()<jupyter_output><empty_output><jupyter_text>Notice again, that observations have been cleared from the surroundings of those of the minority class.**HOMEWORK**- Change the number of neighbours to exaomine- Change the selection method from **all** to **mode**- compare the result in terms of distribution and size of resulting dataset.Note that if we pass an integer to the argument n_neighbors=3, 3 is the number of neighbours to examine, we will look at the 3 closest neighbours to determine if a sample is kept or removed.If instead of an integer, you want to pass a KNN class from Scikit-learn, like the [KNeighborsClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html), remember that to examine 3 neighbours, you need to train a 4 KNN as the observation will be 1 of the neighbours itself.  Edited Nearest Neighbours Real data - Performance comparisonDoes it work well with real datasets? Well, it will depend on the dataset, so we need to try and compare the models built on the whole dataset, and that built on the undersampled dataset.<jupyter_code># load data
# only a few observations to speed the computaton

data = pd.read_csv('../kdd2004.csv').sample(10000, random_state=10)

data.head()
# imbalanced target

data.target.value_counts() / len(data)
# separate dataset into train and test

X_train, X_test, y_train, y_test = train_test_split(
    data.drop(labels=['target'], axis=1),  # drop the target
    data['target'],  # just the target
    test_size=0.3,
    random_state=0)
# edited nearest neighbours

enn = EditedNearestNeighbours(
    sampling_strategy='auto',  # resamples the majority class
    n_neighbors=3, # the number of neighbours to examine to remove an observation
    kind_sel='all',  # all neighbours need to agree
    n_jobs=4)  # I have 4 cores in my laptop

X_enn, y_enn = enn.fit_resample(X_train, y_train)
# compare shapes

X_train.shape, X_enn.shape<jupyter_output><empty_output><jupyter_text>Plot dataLet's compare how the data looks before and after the undersampling.<jupyter_code># original data

sns.scatterplot(data=X_train,
                x="0",
                y="1",
                hue=y_train)

plt.title('Original data')
# enn undersampled data

sns.scatterplot(data=X_enn,
                x="0",
                y="1",
                hue=y_enn)

plt.title('Edited NN data')
# original data, other set of variables

sns.scatterplot(data=X_train,
                x="4",
                y="5",
                hue=y_train)

plt.title('Original data')
# enn undersampled data

sns.scatterplot(data=X_enn,
                x="4",
                y="5",
                hue=y_enn)

plt.title('Edited NN data')<jupyter_output><empty_output><jupyter_text>Machine learning performance comparisonLet's compare model performance with and without undersampling.<jupyter_code># function to train random forests and evaluate the performance

def run_randomForests(X_train, X_test, y_train, y_test):
    
    rf = RandomForestClassifier(n_estimators=200, random_state=39, max_depth=4)
    rf.fit(X_train, y_train)

    print('Train set')
    pred = rf.predict_proba(X_train)
    print('Random Forests roc-auc: {}'.format(roc_auc_score(y_train, pred[:,1])))
    
    print('Test set')
    pred = rf.predict_proba(X_test)
    print('Random Forests roc-auc: {}'.format(roc_auc_score(y_test, pred[:,1])))
# evaluate performance of algorithm built
# using imbalanced dataset

run_randomForests(X_train,
                  X_test,
                  y_train,
                  y_test)
# evaluate performance of algorithm built
# using enn undersampled dataset

run_randomForests(X_enn,
                  X_test,
                  y_enn,
                  y_test)<jupyter_output>Train set
Random Forests roc-auc: 0.9918528188763176
Test set
Random Forests roc-auc: 0.9811762654009543
{'fraction_non_alphanumeric': 0.08128587830080368, 'fraction_numerical': 0.01182548794489093, 'mean_word_length': 3.9270361990950224, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 36, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '207761', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 2743, 'n_tokens_neox': 2585, 'n_words': 873, 'path': 'Section-04-Undersampling/04-05-Edited-Nearest-Neighbours.ipynb', 'repo_name': 'solegalli/machine-learning-imbalanced-data', 'size': '8710'}