<jupyter_start><jupyter_text>This notebook can be run on mybinder:  [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/git/https%3A%2F%2Fgricad-gitlab.univ-grenoble-alpes.fr%2Fai-courses%2Fautonomous_systems_ml/HEAD?filepath=notebooks%2F2_knn)<jupyter_code># Import modules
%matplotlib inline
import matplotlib
import numpy as np
import matplotlib.pyplot as plt
# Select random seed
random_state = 1<jupyter_output><empty_output><jupyter_text>We use scikit-learn to generate a toy 2D data set (two features $x_1$ and $x_2$)  for binary classification  (two classes)  - each sample $(x_1,x_2)$ in the dataset is plotted as a 2D point where the two features $x_1$ and $x_2$ are displayed along the abscissa and ordinate axes respectively - the corresponding class label $y$ is displayed as a color mark (e.g., yellow or purple)<jupyter_code>from sklearn.datasets import make_classification
#X are the features (aka inputs, ...), y the labels (aka responses, targets, output...)
X,y = make_classification(n_features=2, n_redundant=0, n_informative=2, n_samples=1000,
                          random_state=random_state, n_clusters_per_class=2, class_sep=1)

from sklearn.model_selection import train_test_split
# We split the initial set in two sets: one, namely the training set, use for training the model, 
# and one, namely the test set, use to compute the validation error
# -> test_size x n_samples for the test set and n_samples x (1- test_size) for the training set
# where test_size is given as a parameter
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.9, random_state= random_state )
y_train[y_train==0]=-1
y_test[y_test==0]=-1

# Display the training set
plt.scatter(X_train[:,0], X_train[:,1],c=y_train)
plt.grid(True)
plt.xlabel('$x_1$')
plt.ylabel('$x_2$')
#plt.savefig("2d_binary_classif.pdf")<jupyter_output><empty_output><jupyter_text>Then a _k nearest neighbors_ algorithm is used to learn the classification function/rule.<jupyter_code>from sklearn.neighbors import KNeighborsClassifier
knn = KNeighborsClassifier() # Creation of the classifier
knn.fit(X_train, y_train) # Model fitting
# Plot the decision functions
XX, YY = np.meshgrid(np.linspace(X[:,0].min(), X[:,0].max(),200),
                     np.linspace(X[:,1].min(), X[:,1].max(),200))
XY = np.vstack([XX.flatten(), YY.flatten()]).T
yp = knn.predict(XY)
plt.contour(XX,YY,yp.reshape(XX.shape))
plt.scatter(X[:,0], X[:,1], c=y)
plt.grid("on")
plt.xlabel('$x_1$')
plt.ylabel('$x_2$')<jupyter_output><empty_output><jupyter_text>Model complexityFor the KNN, the number of neighbors considered for the decision is the main hyperparemeter to set-up for small to medium size problem. Default value for scikit-learn is set to 5 (do knn? for help). In the following we are going to plot the overall accuracy function of the hyperparameter.<jupyter_code>knn?
neighbors = np.arange(1,40)
errors_train, errors_validation = [], []
for n_ in neighbors:
    knn = KNeighborsClassifier(n_neighbors=n_)
    knn.fit(X_train, y_train)
    
    y_pred = knn.predict(X_train)
    errors_train.append( np.mean( y_train != y_pred) )
    
    y_pred = knn.predict(X_test)
    errors_validation.append( np.mean( y_test != y_pred) )
    
plt.plot(neighbors,errors_train)
plt.plot(neighbors,errors_validation)
plt.legend(["Train", "Test"])
plt.xlabel("Number of neighbors")
plt.ylabel("Misclassification rate")
plt.grid()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12148448825746594, 'fraction_numerical': 0.016236590316033632, 'mean_word_length': 5.484962406015038, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 18, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '560970', 'license': 'Apache-2.0', 'n_tokens_mistral': 1265, 'n_tokens_neox': 1223, 'n_words': 322, 'path': 'Notebooks/1_K-NN/N1_blobs_knn.ipynb', 'repo_name': 'shouhaiel1/Machine_Learning_Labworks', 'size': '3449'}