<jupyter_start><jupyter_text>K-Nearest Neighbors from scratch> Wine quality prediction using a KNN model implemented from scratch using Pytorch.- author: "<NAME>"- categories: [knn, wine-quality, pytorch, eda, from-scratch]- toc: false- comments: true- badges: true- image: images/wines.jpg ![title](images/wines.jpg) During this experiment, we will train a K-nearest neighbors model on physicochemical data to predict the quality of a red or white wines.The KNN will be implemented from scratch using Pytorch along with clear explanation of how the model works.Exploratory data analysis techniques are also discussed in the last section of the notebook.<jupyter_code>import os
import sys
sys.path.append('..')
import torch
import numpy as np
import pandas as pd
import seaborn as sns
import statsmodels.api as sm
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report
from sklearn.preprocessing import StandardScaler
from sklearn.preprocessing import LabelEncoder<jupyter_output><empty_output><jupyter_text>Read csv using Pandas<jupyter_code>wine_red = pd.read_csv(os.path.join('data', 'winequality-red.csv'), sep=';')
wine_white = pd.read_csv(os.path.join('data', 'winequality-white.csv'), sep=';')
wine_red.head()
wine_white.head()<jupyter_output><empty_output><jupyter_text>Merge red and white dataBoth dataframes have the same labels, so we can easily concatenate them.<jupyter_code>wines = pd.concat([wine_white, wine_red])<jupyter_output><empty_output><jupyter_text>Visualize correlation heatmap<jupyter_code>plt.figure(figsize=(16, 8))

wines_corr = wines.corr()
sns.heatmap(wines_corr, annot=True, square=True, fmt='0.2f')<jupyter_output><empty_output><jupyter_text>The features most correlated with the quality are:- alcohol- density- volatile acidityMost of the features are very highly correlated to each other.We will try to select the best features to avoid redundancies using backward elimination further in the notebook.  Convert quality label to categorical values<jupyter_code>sns.barplot(wines['quality'], wines['alcohol'], palette='hot')<jupyter_output><empty_output><jupyter_text>Wines between 7 and 9 quality have a higher alcohol values, we will choose this range for the good wines.Thus, it looks appropriate to divide the quality label into low, medium and good wines.<jupyter_code>tmp = wines['quality'].values
quality_type = np.empty(tmp.shape, dtype='U6')
quality_type[tmp < 5] = 'bad'
quality_type[(4 < tmp) & (tmp < 7)] = 'medium'
quality_type[6 < tmp] = 'good'

quality_type = pd.DataFrame(columns=['quality'], data=quality_type)
wines = wines.drop('quality', axis=1)
wines = pd.concat([wines.reset_index(drop=True), quality_type.reset_index(drop=True)], axis=1)

sns.countplot(wines.iloc[:, -1])<jupyter_output><empty_output><jupyter_text>Isolate features from the targetFeature scaling is mandatory when using a K-nearest neighbors model.The KNN algorithm has a distance decision based approach to regress or classify an input.In fact, if some features have higher values than other, they will contribute more in the overall distance calculation thus biasing the outcome.<jupyter_code>X = wines.iloc[:, :-1].to_numpy()
y = wines.iloc[:, -1].to_numpy()<jupyter_output><empty_output><jupyter_text>Convert category to integer<jupyter_code>y = LabelEncoder().fit_transform(y)<jupyter_output><empty_output><jupyter_text>Split data in train test<jupyter_code>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)

scaler = StandardScaler() 
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

# Convert data to pytorch tensor
X_train = torch.from_numpy(X_train)
X_test = torch.from_numpy(X_test)
y_train = torch.from_numpy(y_train)
y_test = torch.from_numpy(y_test)<jupyter_output><empty_output><jupyter_text>We will use Pytorch for this experiment  K-Nearest NeighborsThe K-nearest neighbors classifier looks at the $K^{th}$ closest points in the training set of an input $x$counts how many members of each class are present and returns the empirical fraction as the estimate.$$p(\mathbf{y} = c|\boldsymbol{\mathbf{x}}, D, K) = \frac{1}{K} \sum_{i \in N_K(\boldsymbol{\mathbf{x}}, D)}\mathbb{I}(y_i = c)$$$$\mathbb{I}(e) =\begin{cases}    1 & \text{if}\;e\;\text{is true}\\    0 & \text{otherwise}\end{cases}$$where $c$ is the class and $N_K$ are the closest sample of $\boldsymbol{\text{x}}$ in $D$.<jupyter_code>def knn(sample, X, y, k_neighbors=10):
    """Instance of K-Nearest Neigbhors model
    
    Args:
        X: A torch tensor for the data.
        y: A torch tensor for the labels.
        k_neighbors: An integer for the number of nearest neighbors
            to consider.
    """
    sample = sample.unsqueeze(1).T
    # Compute the distance with the train set
    dist = (X - sample).pow(2).sum(axis=1).sqrt()
    
    # Sort the distances
    _, indices = torch.sort(dist)
    y = y[indices]
    
    # Get the Kth most similar samples and return the predominant class
    return y[:k_neighbors].bincount().argmax().item()
def train_knn(X_train, X_test, y_train, y_test, k_neighbors=1):
    """Trains a K-Nearest Neigbhors model 
    
    Args:
        X_train: A torch tensor for the training data.
        X_test: A torch tensor for the test data.
        y_train: A torch tensor for the training labels.
        y_test: A torch tensor for the test labels.
        k_neighbors: An integer for the number of nearest neighbors
            to consider.
    """
    # Allocate space for the prediction
    y_pred_test = np.zeros(y_test.shape, dtype=np.uint8)
    y_pred_train = np.zeros(y_train.shape, dtype=np.uint8)
    X_train_c = X_train.clone()
    
    # Predict on each sample of the train and test
    for i in range(X_test.shape[0]):
        y_pred_test[i] = knn(X_test[i], X_train, y_train, k_neighbors=k_neighbors)
    y_pred_test = torch.from_numpy(y_pred_test).float()
    return y_pred_test
pred_test = train_knn(X_train, X_test, y_train, y_test, k_neighbors=1)
print(classification_report(y_test, pred_test))

# Save data for future visualization
X_train_viz = X_train
X_test_viz = X_test
y_train_viz = y_train
y_test_viz = y_test<jupyter_output>precision    recall  f1-score   support

           0       0.39      0.17      0.24        53
           1       0.65      0.67      0.66       252
           2       0.88      0.90      0.89       995

    accuracy                           0.83      1300
   macro avg       0.64      0.58      0.60      1300
weighted avg       0.82      0.83      0.82      1300<jupyter_text>The KNN achieves a good performance compared to its simplicity.  Exploratory Data AnalysisLet's try to discard some features based on p-value but to see if it improves the results.More on p-value [here](https://www.statsdirect.com/help/basics/p_values.htm:~:text=The%20P%20value%2C%20or%20calculated,the%20hypothesis%20is%20being%20tested).<jupyter_code>columns = wines.columns[:-1]
while len(columns) > 0:
    pvalues = []
    X_1 = wines[columns]
    X_1 = sm.add_constant(X_1)
    model = sm.OLS(y, X_1).fit()
    pvalues = pd.Series(model.pvalues[1:], index=columns)
    max_idx = np.argmax(pvalues)
    max_pval = pvalues[max_idx]
    if max_pval > 0.10:
        columns = columns.drop(columns[max_idx]) 
        print('Dropping column ' + columns[max_idx] + ', pvalue is: ' + str(max_pval))
    else:
        break
columns
X = wines[columns].to_numpy()
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)
X_train = torch.from_numpy(X_train)
X_test = torch.from_numpy(X_test)
y_train = torch.from_numpy(y_train)
y_test = torch.from_numpy(y_test) 

y_pred = train_knn(X_train, X_test, y_train, y_test, k_neighbors=1)
print(classification_report(y_test, y_pred))<jupyter_output>precision    recall  f1-score   support

           0       0.14      0.14      0.14        58
           1       0.53      0.53      0.53       308
           2       0.86      0.86      0.86      1259

    accuracy                           0.77      1625
   macro avg       0.51      0.51      0.51      1625
weighted avg       0.77      0.77      0.77      1625<jupyter_text>Feature selection based on p-value made the performance worse. From 83% to 77%.We tested with a value $k \in [1, 10]$ and the best result is given $k = 1$.  Feature Selection by handLet's try to select all the feature that has more than $0.10$ correlation with the target.<jupyter_code>columns = ['alcohol', 'density', 'chlorides', 'volatile acidity']

X = wines[columns].to_numpy()
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)
X_train = torch.from_numpy(X_train)
X_test = torch.from_numpy(X_test)
y_train = torch.from_numpy(y_train)
y_test = torch.from_numpy(y_test) 

y_pred = train_knn(X_train, X_test, y_train, y_test, k_neighbors=1)
print(classification_report(y_test, y_pred))<jupyter_output>precision    recall  f1-score   support

           0       0.19      0.17      0.18        58
           1       0.51      0.51      0.51       308
           2       0.85      0.86      0.85      1259

    accuracy                           0.77      1625
   macro avg       0.52      0.51      0.51      1625
weighted avg       0.76      0.77      0.77      1625
{'fraction_non_alphanumeric': 0.10139335476956056, 'fraction_numerical': 0.0322615219721329, 'mean_word_length': 3.5450560155869457, 'pattern_counts': {'":': 0, '<': 51, '<?xml version=': 0, '>': 50, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 15, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '378686', 'license': 'Apache-2.0', 'n_tokens_mistral': 3333, 'n_tokens_neox': 3128, 'n_words': 875, 'path': '_notebooks/2020-09-13-K-Nearest-Neigbors.ipynb', 'repo_name': 'ConsciousML/blog', 'size': '9336'}