<jupyter_start><jupyter_text>Lab 03 - "Supervised Machine Learning: k Nearest-Neighbors"EMBA 58/59 - W8/3 - "AI Coding for Executives", University of St. Gallen The lab environment of the "AI Coding for Executives" EMBA course at the University of St. Gallen (HSG) is based on Jupyter Notebooks (https://jupyter.org), which allow to perform a variety of statistical evaluations and data analyses. In the last lab, you got your hands dirty with supervised learning by using the Gaussian Naive-Bayes (GNB) classifier. You learned how to train a model and to evaluate and interpret its results. In this lab, we will look at another popular algorithm, namely the **k Nearest-Neighbors (kNN)** classifier.The *discriminative* **k Nearest-Neighbors (kNN)** classifier is a simple, easy to understand, versatile, but powerful machine learning algorithm. Until recently (prior to the advent of deep learning approaches) it was used in a variety of applications such as finance, healthcare, political science, handwriting detection, image recognition and video recognition, e.g. in credit ratings, financial institutes used kNN to predict the solvency of customers.This classification technique is part of the **discriminative** type of classifiers, which can be distinguished from the generative type as shown in the following illustration: (Courtesy: Intro to AI & ML lecture, Prof. Dr. Borth, University of St. Gallen) As always, pls. don't hesitate to ask all your questions either during the lab, post them in our CANVAS (StudyNet) forum (https://learning.unisg.ch), or send us an email (using the course email).  1. Lab Objectives: After today's lab you should be able to:> 1. Know how to setup a **notebook or "pipeline"** that solves a simple supervised classification task.> 2. Recognize the **data elements** needed to train and evaluate a supervised machine learning classifier. > 3. Understand how a discriminative **k Nearest-Neighbor (kNN)** classifier can be trained and evaluated.> 4. Know how to use Python's sklearn library to **train** and **evaluate** arbitrary classifiers.> 5. Understand how to **evaluate** and **interpret** the classification results.  2. Setup of the Jupyter Notebook Environment Similar to the previous labs, we need to import a couple of Python libraries that allow for data analysis and data visualization. In this lab will use the `Pandas`, `Numpy`, `Scikit-Learn`, `Matplotlib` and the `Seaborn` library. Let's import the libraries by the execution of the statements below:<jupyter_code># import the numpy and pandas data science library
import pandas as pd
import numpy as np

# import sklearn data and data pre-processing libraries
from sklearn import datasets
from sklearn.preprocessing import MinMaxScaler
from sklearn.model_selection import train_test_split

# import k-nearest neighbor classifier library
from sklearn.neighbors import KNeighborsClassifier

# import sklearn classification evaluation library
from sklearn import metrics
from sklearn.metrics import classification_report, confusion_matrix

# import matplotlib data visualization library
import matplotlib.pyplot as plt
import seaborn as sns<jupyter_output><empty_output><jupyter_text>Enable inline Jupyter notebook plotting:<jupyter_code>%matplotlib inline<jupyter_output><empty_output><jupyter_text>Use the `Seaborn` plotting style in all subsequent visualizations:<jupyter_code>plt.style.use('seaborn')<jupyter_output><empty_output><jupyter_text>3. Dataset Download and Pre-Processing  3.1 Dataset Download and Data Assessment Let's try the k Nearest-Neighbour algorithm using the delicious **Wine Dataset**! It is a classic and straightforward multi-class classification dataset.  (Source: https://www.empirewine.com) The data is the result of a chemical analysis of wines grown in the same region in Italy by three different cultivators (types). The dataset consists in total of **178 wines** as well as their corresponding **13 different measurements** taken for different constituents found in the three types of wine. Please, find below the list of the individual measurements (features):>- `Alcohol`>- `Malic acid`>- `Ash`>- `Alcalinity of ash`>- `Magnesium`>- `Total phenols`>- `Flavanoids`>- `Nonflavanoid phenols`>- `Proanthocyanins`>- `Color intensity`>- `Hue`>- `OD280/OD315 of diluted wines`>- `CProline`Further details on the dataset can be obtained from the following puplication: *<NAME> al, PARVUS - "An Extendible Package for Data Exploration, Classification and Correlation.", Institute of Pharmaceutical and Food Analysis and Technologies, Via Brigata Salerno, 16147 Genoa, Italy.*Let's load the dataset and conduct a preliminary data assessment:<jupyter_code>wine = datasets.load_wine()<jupyter_output><empty_output><jupyter_text>Print and inspect feature names of the dataset:<jupyter_code>wine.feature_names<jupyter_output><empty_output><jupyter_text>Print and inspect the class names of the dataset:<jupyter_code>wine.target_names<jupyter_output><empty_output><jupyter_text>Print and inspect the top 10 feature rows of the dataset:<jupyter_code>pd.DataFrame(wine.data, columns=wine.feature_names).head(10)<jupyter_output><empty_output><jupyter_text>Print and inspect the top 10 labels of the dataset:<jupyter_code>pd.DataFrame(wine.target).head(10)<jupyter_output><empty_output><jupyter_text>Determine and print the feature dimensionality of the dataset:<jupyter_code>wine.data.shape<jupyter_output><empty_output><jupyter_text>Determine and print the label dimensionality of the dataset:<jupyter_code>wine.target.shape<jupyter_output><empty_output><jupyter_text>Plot the data distributions of the distinct features:<jupyter_code># init the plot
plt.figure(figsize=(10, 10))

# prepare the dataset to be plotable using seaborn

# convert to Panda's DataFrame
wine_plot = pd.DataFrame(wine.data, columns=wine.feature_names)

# add class labels to the DataFrame
wine_plot['class'] = wine.target

# plot a pairplot of the distinct feature distributions
sns.pairplot(wine_plot, diag_kind='hist', hue='class');<jupyter_output><empty_output><jupyter_text>3.2 Dataset Pre-Processing  3.2.1 Feature Re-Scaling Observing the features values of the **Wine Dataset** we will notice that their respective value ranges vary widely. This results in a major challenge for distance based machine learning algorithms such as the **k Nearest-Neighbor** classifier. The **k Nearest-Neighbour** classifier calculates the distance between two observations using a distance measure such as the **Euclidean** or **Manhattan** distance.If one of the features exhibits a wide range of values, the calculated distance will be governed by this particular feature. Therefore, the range of all features needs to be **re-scaled** or **normalized** to a value range beween in $[0,1]$ or $[-1,1]$ so that each feature contributes approximately proportionately to the final distance. One widley used method of feature re-scaling is referred to as **Min-Max Normalization** and is given by: $$x'={\frac  {x-{\text{min}}(x)}{{\text{max}}(x)-{\text{min}}(x)}}$$ Let's re-scale the distinct feature values of the **Wine Dataset** using **Min-Max Normalization** using the `MinMaxScaler` class of the `sklearn` library:<jupyter_code># init the min-max scaler
scaler = MinMaxScaler(feature_range=(0, 1), copy=True)

# min-max normalize the distinct feature values
wine_data_norm = scaler.fit_transform(wine.data)<jupyter_output><empty_output><jupyter_text>Print and inspect the top 10 feature rows of the normalized dataset:<jupyter_code>pd.DataFrame(wine_data_norm, columns=wine.feature_names).head(10)<jupyter_output><empty_output><jupyter_text>Ok, we can observe that all features values have been re-scaled. All feature values are indeed in a range between $[0,1]$. Let's also visualize the re-scaled feature values and inspect their distributions:<jupyter_code># init the plot
plt.figure(figsize=(10, 10))

# prepare the dataset to be plotable using seaborn

# convert to Panda's DataFrame
wine_plot = pd.DataFrame(wine_data_norm, columns=wine.feature_names)

# add class labels to the DataFrame
wine_plot['class'] = wine.target

# plot a pairplot of the distinct feature distributions
sns.pairplot(wine_plot, diag_kind='hist', hue='class');<jupyter_output><empty_output><jupyter_text>Excellent, the characteristics of the distinct feature value distributions remained unchanged.  3.2.2 Extraction of Training- and Evaluation-Dataset To understand and evaluate the performance of any trained **supervised machine learning** model, it is good practice to divide the dataset into a **training set** (the fraction of data records solely used for training purposes) and a **evaluation set** (the fraction of data records solely used for evaluation purposes). Please note, the **evaluation set** will never be shown to the model as part of the training process. All of this is exactly what we did in the Gaussian "Naive-Bayes" lab.  We set the fraction of testing records to **30%** of the original dataset:<jupyter_code>eval_fraction = 0.3<jupyter_output><empty_output><jupyter_text>Furthermore, let's set a random seed to insure reproducibility of the train-test split in potential future runs of the notebook:<jupyter_code>seed = 42<jupyter_output><empty_output><jupyter_text>Randomly split the **Wine Dataset** into training set and evaluation set using sklearn's `train_test_split` function:<jupyter_code># 70% training and 30% evaluation
X_train, X_eval, y_train, y_eval = train_test_split(wine_data_norm, wine.target, test_size=eval_fraction, random_state=seed)<jupyter_output><empty_output><jupyter_text>Evaluate the training set dimensionality:<jupyter_code>X_train.shape, y_train.shape<jupyter_output><empty_output><jupyter_text>Evaluate the evaluation set dimensionality:<jupyter_code>X_eval.shape, y_eval.shape<jupyter_output><empty_output><jupyter_text>4. k Nearest-Neighbor (kNN) Classification Now, let's have closer look into the non-parametric method used for supervised classification tasks, referred to as the the **k Nearest-Neighbors (kNN)** algorithm. As you learned during the in lecture k-NN classification, the output of the classifier is a class membership.  Prior to running **k Nearest-Neighbor (kNN)** classification let's briefly revisit the distinct steps of the algorithm as discussed in the lecture:(Courtesy: Intro to AI & ML lecture, Prof. Dr. Borth, University of St. Gallen) An object is classified by a majority vote of its neighbors, with the object being assigned to the class most common among its k nearest neighbors (k is a positive integer, typically small). If k = 1, then the object is simply assigned to the class of that single nearest neighbor.  4.1. Nearest Neighbors Classification, k=4 Set the number of neighbors `k` to be considered in the classification of each sample:<jupyter_code>k_nearest_neighbors = 4<jupyter_output><empty_output><jupyter_text>Please, recall that we discussed two distinct distance measures in the lecture to calculate the distance between an observation $x$ and it's $k$-nearest-neighbors $x'_{j}$ in a $n$-dimensonal feature space:**Manhattan distance ("L1-norm"):** $$ D(x, x')=\|\sum^k_{j=1}\sum^n_{i=1}(x_{i} - x'_{j,i})\|$$**Euclidian distance ("L2-norm"):** $$ D(x, x')=\sqrt{\sum^k_{j=1}\sum^n_{i=1}(x_{i} - x'_{j,i})^2}$$where the index $j$ denotes the number of $k$-nearest-neighbors and the index $i$ denotes the $i$-th feature of a single nearest neighbor $x_j$. Since the 13 features of the Wine dataset consist of continuous features we will use the Euclidean distance as the distance metric in our kNN classification:<jupyter_code>distance_metric = 'euclidean'<jupyter_output><empty_output><jupyter_text>Init the **kNN classifier** of Python's `Scikit-Learn` library of data science algorithms:<jupyter_code>knn = KNeighborsClassifier(n_neighbors=k_nearest_neighbors, metric=distance_metric)<jupyter_output><empty_output><jupyter_text>Train the k-NN classifier using the training dataset:<jupyter_code>knn.fit(X_train, y_train);<jupyter_output><empty_output><jupyter_text>Utilize the trained model to predict the response for the evaluation dataset:<jupyter_code>y_pred = knn.predict(X_eval)<jupyter_output><empty_output><jupyter_text>Let's have a look at the predicted class labels:<jupyter_code>y_pred<jupyter_output><empty_output><jupyter_text>As well as the true class labels:<jupyter_code>y_eval<jupyter_output><empty_output><jupyter_text>Determine **prediction accuracy** of the trained model on the evaluation dataset:<jupyter_code>print("Accuracy, k=4: ", metrics.accuracy_score(y_eval, y_pred))<jupyter_output><empty_output><jupyter_text>Determine and plot the **confusion matrix** of the individual predictions:<jupyter_code># determine the prediction confusion matrix
mat = confusion_matrix(y_eval, y_pred)<jupyter_output><empty_output><jupyter_text>Visualize the **confusion matrix** of the individual predictions determined by the **k=4 Nearest-Neighbor** classifier:<jupyter_code># init the plot
plt.figure(figsize=(5, 5))

# plot confusion matrix heatmap
sns.heatmap(mat.T, square=True, annot=True, fmt='d', cbar=False, cmap='BuGn_r', xticklabels=wine.target_names, yticklabels=wine.target_names)

# add plot axis labels
plt.xlabel('[true label]')
plt.ylabel('[predicted label]')

# add plot title
plt.title('k-NN Confusion Matrix, k=4');<jupyter_output><empty_output><jupyter_text>Remember that as part of the lecture you learned about several measures to evaluate the quality of a retrieval system, namely **Precision**, **Recall** and **F1-Score**. Let's briefly revisit their definition and subsequently calculate those measures based on the confusion matrix above: >- The **Precision**, denoted by Precision $=\frac{TP}{TP + FP}$, is the probability that a retrieved document is relevant.>- The **Recall**, denoted by Recall $=\frac{TP}{TP + FN}$, is the probability that a relevant document is retrieved.>- The **F1-Score**, denoted by F1-Score $= 2 \cdot \frac{Precision \cdot Recall}{Precision + Recall}$, combines precision and recall is the harmonic mean of both measures.<jupyter_code>print(classification_report(y_eval, y_pred))<jupyter_output><empty_output><jupyter_text>4.2. Nearest Neighbors Classification, k=8 Set the number of neighbors `k` to be considered in the classification of each sample:<jupyter_code>k_nearest_neighbors = 8<jupyter_output><empty_output><jupyter_text>Init the **k-NN classifier** of Python's `sklearn` libary of data science algoritms:<jupyter_code>knn = KNeighborsClassifier(n_neighbors=k_nearest_neighbors, metric=distance_metric)<jupyter_output><empty_output><jupyter_text>Train the k-NN classifier using the training dataset:<jupyter_code>knn.fit(X_train, y_train);<jupyter_output><empty_output><jupyter_text>Utilize the trained model to predict the response for the evaluation dataset:<jupyter_code>y_pred = knn.predict(X_eval)<jupyter_output><empty_output><jupyter_text>Determine **prediction accuracy** of the trained model on the evaluation dataset:<jupyter_code>print("Accuracy, k=8: ", metrics.accuracy_score(y_eval, y_pred))<jupyter_output><empty_output><jupyter_text>Determine and plot the **confusion matrix** of the individual predictions:<jupyter_code># determine the prediction confusion matrix
mat = confusion_matrix(y_eval, y_pred)<jupyter_output><empty_output><jupyter_text>Visualize the **confusion matrix** of the individual predictions determined by the **k=8 Nearest-Neighbor** classifier:<jupyter_code># init the plot
plt.figure(figsize=(5, 5))

# plot confusion matrix heatmap
sns.heatmap(mat.T, square=True, annot=True, fmt='d', cbar=False, cmap='BuGn_r', xticklabels=wine.target_names, yticklabels=wine.target_names)

# add plot axis labels
plt.xlabel('[true label]')
plt.ylabel('[predicted label]')

# add plot title
plt.title('k-NN Confusion Matrix, k=8');
print(classification_report(y_eval, y_pred))<jupyter_output><empty_output><jupyter_text>4.3. Finding the optimal k of the kNN classifier Until now we have investigated the Euclidean distance based kNN classifer for two distinct values of $k=4$ and $k=8$. However the question remains: can we locate a value for k that yields an even higher classification accuracy? Let's therefore investigate the classification performance of a wider range of distinct $k$ values and in particular compare the corresponding classification accuracy. We will do so in the following by specifying a range of distinct $k$ values ranging from $k=1, ..., 50$:<jupyter_code># try k=1 through k=50 to be evaluated
k_range = range(1, 51)<jupyter_output><empty_output><jupyter_text>To run the kNN classification experiments for different values of $k$ we will define a python loop. The loop iterates over the range of distinct k's and conducts the model training (using the training data) and evaluation (using the evaluation data). The classification accuracy for each $k$ value will be collected and stored in a designated list of accuracy scores:<jupyter_code># init the distinct accuracy scores obtained on the evaluation data
eval_accuracy_scores = []

# iterate over the distinct k values
for k in k_range:
    
    # init the k-NN classifier
    knn = KNeighborsClassifier(n_neighbors=k, metric='euclidean')
    
    # train the k-NN classifer on the training data
    knn.fit(X_train, y_train)
    
    # evaluate the k-NN classifier on the training data
    y_train_pred = knn.predict(X_train)
    
    # evaluate the k-NN classifier on the evaluation data
    y_eval_pred = knn.predict(X_eval)
    
    # collect the classification accuracy of the current k on the evaluation data
    eval_accuracy_scores.append(metrics.accuracy_score(y_eval, y_eval_pred))<jupyter_output><empty_output><jupyter_text>Visualizing the collected classification accuracy scores of the distinct $k$ values:<jupyter_code># prepare plot
fig = plt.figure()
ax = fig.add_subplot(111)

# plot the classification accuracy of distinct k's
ax.plot(range(1, len(eval_accuracy_scores)+1), eval_accuracy_scores, color='green', marker='o')

# add grid
ax.grid(linestyle='dotted')

# add axis range and legends
ax.set_xlabel("[$k$-Nearest-Neighbors]", fontsize=10)
ax.set_ylabel("[% classification accuracy]", fontsize=10)

# add plot title
ax.set_title('k-NN Classification Accuracy', fontsize=10);<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08962315674494811, 'fraction_numerical': 0.007482250136537411, 'mean_word_length': 6.535390946502058, 'pattern_counts': {'":': 0, '<': 162, '<?xml version=': 0, '>': 183, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '40', 'dir': 'jupyter-structured-clean-dedup', 'id': '92762', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 5589, 'n_tokens_neox': 5246, 'n_words': 1986, 'path': 'resources/lab_03/lab_03b.ipynb', 'repo_name': 'HSG-AIML/LabAI-Coding', 'size': '18317'}