<jupyter_start><jupyter_text>Validation curve and bias/variance dilemmaIn this notebook we would like to discuss the bias/variance dilemma on the example of a KNN classifier using a data set with failure/success information for data points.<jupyter_code>#import necessary packages
import pandas as pd
import numpy as np

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

%matplotlib inline
import matplotlib.pyplot as plt

from helper import plot_classifier #helper.py is saved in the repository
#defining the data frame
df = pd.read_csv("classification.csv")

df.head()
#define variables
X = df[["age","interest"]].values

Y = df["success"].values # single brackets

X_train, X_test, Y_train, Y_test = train_test_split(X, Y, random_state = 0, test_size = .25)

#rescale training and test data
scaler = StandardScaler()
scaler.fit(X_train)

X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)
#training the models(s) for a range via the validation_curve function of sklearn
from sklearn.model_selection import validation_curve
from sklearn.neighbors import KNeighborsClassifier

param_range = np.array([40, 30, 20, 15, 10, 8, 7, 6, 5, 4, 3, 2, 1])

train_scores, test_scores = validation_curve(KNeighborsClassifier(), X, Y, param_name = "n_neighbors", param_range = param_range)
#printing the train_scores for the range of 40 neighbors to 1 neighbor:
print(train_scores)<jupyter_output>[[0.88832487 0.88888889 0.85929648]
 [0.89847716 0.9040404  0.87939698]
 [0.9035533  0.92929293 0.91959799]
 [0.92385787 0.94949495 0.92964824]
 [0.93908629 0.93939394 0.92462312]
 [0.94416244 0.94949495 0.91959799]
 [0.94416244 0.94444444 0.92462312]
 [0.93401015 0.94949495 0.92964824]
 [0.93401015 0.94444444 0.93969849]
 [0.92893401 0.93939394 0.93969849]
 [0.94923858 0.95959596 0.94974874]
 [0.93401015 0.96969697 0.95477387]
 [1.         1.         1.        ]]<jupyter_text>Comment: We obtain per param_range entry 3 values (i.e. values for 3 models) since validation_curve does 3 cross validation automatically for us. Of course thos holds for the test data as well:<jupyter_code>print(test_scores)
#plotting the score data, however, taking the mean only over values for the same param_range value - therefore axis = 1
plt.plot(param_range, np.mean(train_scores, axis = 1))
plt.plot(param_range, np.mean(test_scores, axis = 1))

#to turn around the X-axis
plt.xlim(np.max(param_range), 0)

plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09455128205128205, 'fraction_numerical': 0.14262820512820512, 'mean_word_length': 5.211442786069652, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 8, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 3, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '139241', 'license': 'MIT', 'n_tokens_mistral': 1119, 'n_tokens_neox': 890, 'n_words': 264, 'path': 'Knearestneighborsvalidationcurvebiasvariancedilemma.ipynb', 'repo_name': 'andreaspts/ML_KNN_Analyses_for_Classification', 'size': '2496'}