<jupyter_start><jupyter_text>**TUTORIAL 6** Distance Based Classification In pattern recognition, the k-nearest neighbors algorithm is a non-parametric method used for classification and regression.It belongs to the supervised learning domain.We will understand it with a practice example, Steps we are going to follow are :1) Import important libraries2) Importing Data3) Encoding4)Training and Test Data5) Visualization of Data6) Nearest Neighbour Classifier7) k Nearest Neighbour8) Nearest Mean Classifier9) Compare Results10) sklearn Library------ **1) Importing important libraries**<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
from sklearn.datasets import load_iris
import json

%matplotlib inline<jupyter_output><empty_output><jupyter_text>**2) Importing Data**We will be working on the Iris dataset today.<jupyter_code>data = load_iris()
df = pd.DataFrame(data.data, columns=data.feature_names)
df['Species']=data.target 
df.head()<jupyter_output><empty_output><jupyter_text>We will be taking 2 features, 'SepalLengthCm' and 'PetalWidthCm' for classification for easier visualization.<jupyter_code>print(df.columns)
print("Shape: ", df.shape)
df.isnull().sum()
# Write the code to choose columns 'SepalLengthCm', 'PetalWidthCm' and 'Species' from the dataframe 'df',
# Make a new dataframe 'df_new' and assign the three columns to the new dataframe.
#Write the code below

df_new = df[["sepal length (cm)", "petal width (cm)", "Species"]]
df_new.head()
df_new.describe()   
df_new["Species"].value_counts()<jupyter_output><empty_output><jupyter_text>**3) Encoding**Now we will use Label Encoder to Encode the 'Species' column in numeric. For eg - the species Iris-setosa is encoded as 0, Iris-virginica as 2 and so on.<jupyter_code># Do not make changes in this cell
# Simply execute it and move on
from sklearn.preprocessing import LabelEncoder
labelencoder_Species = LabelEncoder()
df_new['Species_encoded'] = labelencoder_Species.fit_transform(df_new['Species'])
df_new.head()<jupyter_output><ipython-input-11-231c5bd0d9b9>:5: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  df_new['Species_encoded'] = labelencoder_Species.fit_transform(df_new['Species'])<jupyter_text>We break this into X (features - 'SepalLengthCm' and 'PetalWidthCm') and y (output which is 'Species_encoded').<jupyter_code>df_new["Species_encoded"].unique()
# a)Write the code to choose columns 'SepalLengthCm' and 'PetalWidthCm' from the dataframe 'df_new',
# and assign it to the new dataframe X.
# b)Write the code to choose columns 'Species_encoded' from the dataframe 'df_new',
# and assign it to the new dataframe y.
#Write the code below

X = df_new[["sepal length (cm)", "petal width (cm)"]]
y = df_new["Species_encoded"]<jupyter_output><empty_output><jupyter_text>**4) Training and Test data**We will divide the dataset into training and test dataset to see the accuracy of each of the algorithms that we use :<jupyter_code>from sklearn.model_selection import train_test_split
# Split the dataset into 60% training and 40% test set
# NOTE:- take random_state = 3
# Take the variables as X_train, X_test, y_train and y_test
#Write the code below
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=3)

X_train.reset_index(drop=True, inplace=True)
y_train.reset_index(drop=True, inplace=True)
X_test.reset_index(drop=True, inplace=True)
y_test.reset_index(drop=True, inplace=True)<jupyter_output><empty_output><jupyter_text>We can check the different types of species in the dataset using the unique function :<jupyter_code>df_new['Species'].unique()<jupyter_output><empty_output><jupyter_text>**5) Visualization of data**Now we will plot the two features on the x and y axis and plot the data with different colour according to the species.<jupyter_code># Write the code to create datasets according to the different categories of the species from the training set of features
# and assign it to the new dataframes setosa, versicolor and virginica.
#Create datasets according to the category
#For the species setosa           
setosa = X_train.loc[y_train.index[y_train == 0], :]
#For the species versicolor          
versicolor = X_train.loc[y_train.index[y_train == 1], :]
#For the species virginica   
virginica = X_train.loc[y_train.index[y_train == 2], :]

#assuming setosa --> label 0;  versicolor --> label 1;  virginica  --> label 2
#Plot the data categorywise using scatter plot and visualize the data. Also include labels and legend.
#Write the code below

fig = plt.figure(figsize=(10,7))
plt.scatter(setosa.iloc[:,0], setosa.iloc[:,1], color='red', label="setosa")
plt.scatter(versicolor.iloc[:,0], versicolor.iloc[:,1], color='green', label="versicolor")
plt.scatter(virginica.iloc[:,0], virginica.iloc[:,1], color='blue', label="virginica")

plt.xlabel("sepal length (cm)  ---->")
plt.ylabel("petal width (cm)  ---->")
plt.title("Scatter plots of the X_train (category wise)")
plt.legend(loc='upper left')
plt.grid(1)
plt.show()<jupyter_output><empty_output><jupyter_text>Let us import the time library to compare the timing of the 3 methods :<jupyter_code>import time<jupyter_output><empty_output><jupyter_text>**6) Nearest Neighbour Classifier**<jupyter_code># Question 1) Write the algorithm for nearest neighbour classifier as learnt in the theory
#             Assign the answer to ans[0] after converting the output of the method to tuple 
# Algorithm - Calculate the Euclidian distance for each test dataset by comparing with every trainig dataset and assign it to the variable 'val'
#            The prediciton is the minimum value obtained from all the Euclidian distance calculated for that test datapoint
# Assign the y_predict to ans[0] as a tuple => (0,1,0,2, and so on) 
def euclidDistance(a, b):
    assert len(a) == len(b), "Make sure the points are in same dimension"
    dis = 0
    for i in range(len(a)):
        dis += (a[i]-b[i])*(a[i]-b[i])
    dis = np.sqrt(dis)
    return dis
def KNN(inputs, labels, K, x):
    distances = [(euclidDistance(inputs.loc[i].values,x), i) for i in inputs.index]
    distances.sort()

    idx = 0
    k = 1
    cur_dis = distances[0][0]
    count_setosa, count_versicolor, count_virginica = (0,0,0) 
    while (idx < len(distances)): #and (count_setosa == count_versicolor or count_setosa == count_virginica or count_versicolor == count_virginica)):
        while (cur_dis == distances[idx][0]):
            if (labels[distances[idx][1]] == 0): count_setosa += 1
            elif (labels[distances[idx][1]] == 1): count_versicolor += 1
            elif (labels[distances[idx][1]] == 2): count_virginica += 1
            idx += 1
            k += 1
        if (k > K): break
        cur_dis = distances[idx][0]

    temp = [count_setosa, count_versicolor, count_virginica]   #according to label values
    return temp.index(max(temp))
ans = [None] * 10
start_time = time.time()                          # Starting the runtime for this cell
y_predict = []                                    # Predicted values of y will be stored
#Write the code below
y_predict = [KNN(X_train, y_train, 1, X_test.loc[i].values) for i in X_test.index]

# 1 mark
ans[0] = tuple (y_predict)
ans[0] = str(ans[0])
time_NN = time.time() - start_time    # Final time taken
ans[0]<jupyter_output><empty_output><jupyter_text>We will concatenate the columns of predicted and y_test dataframe to X_test<jupyter_code># a)Concatenate X_test and y_test dataframe by column and assign it to the new dataframe 'Test_Data'
# b)Concatenate Test_data and Predicted dataframe by column assign it to the new dataframe 'Check_Prediction' 
#Write the code below
Test_Data = pd.concat([X_test, y_test], axis=1, ignore_index=False)

temp = pd.DataFrame(y_predict, index=Test_Data.index, columns=["Predicted"])
Check_Prediction = pd.concat([Test_Data, temp], axis=1, ignore_index=False)<jupyter_output><empty_output><jupyter_text>We will calculate the accuracy now :<jupyter_code># Question 2) Calculate the accuracy obtained for the algorithm used above
#             Assign the accuracy to ans[1] and store it as 'Accuracy_NearestNeighbours'  
#NOTE - Accuracy = Correct_Predictions/Total_Predictions
#Write the code below
Accuracy_NearestNeighbours = sum([1 for i in Check_Prediction.index \
                                  if Check_Prediction.loc[i, "Species_encoded"] == Check_Prediction.loc[i, "Predicted"]])/len(Check_Prediction)

# 1 mark
ans[1] = Accuracy_NearestNeighbours
Accuracy_NearestNeighbours<jupyter_output><empty_output><jupyter_text>Now let us have a look at the values that we did not predicted correctly :<jupyter_code>#Plot the data categorywise using scatter plot and visualize the data. Also include labels and legend.
#Write the code below
setosa_test = (Check_Prediction[Check_Prediction["Predicted"] == 0]).iloc[:, :2]
versicolor_test = (Check_Prediction[Check_Prediction["Predicted"] == 1]).iloc[:, :2]
virginica_test = (Check_Prediction[Check_Prediction["Predicted"] == 2]).iloc[:, :2]

fig = plt.figure(figsize=(10,7))
plt.scatter(setosa_test.iloc[:,0], setosa_test.iloc[:,1], color='red', label="setosa")
plt.scatter(versicolor_test.iloc[:,0], versicolor_test.iloc[:,1], color='green', label="versicolor")
plt.scatter(virginica_test.iloc[:,0], virginica_test.iloc[:,1], color='blue', label="virginica")

plt.xlabel("sepal length (cm)  ---->")
plt.ylabel("petal width (cm)  ---->")
plt.title("Scatter plots of the X_test (with y_predict) (category wise) for k = 1")
plt.legend(loc='upper left')
plt.grid(1)
plt.show()

print("\n")
#Implement the below code snippet to see the values that were not predicted correctly
fig = plt.figure(figsize=(10,7))
plt.xlabel("sepal length (cm)  ---->")
plt.ylabel("petal width (cm)  ---->")
plt.title("Scatter plot to compare the correct predictions for k = 1")

for index in Check_Prediction.index:
    if (Check_Prediction['Species_encoded'][index]==Check_Prediction['Predicted'][index]):
        s1 = plt.scatter(Check_Prediction['sepal length (cm)'][index],Check_Prediction['petal width (cm)'][index],marker='x',c='m')
    else:
        s2 = plt.scatter(Check_Prediction['sepal length (cm)'][index],Check_Prediction['petal width (cm)'][index],marker='*',c='y')

plt.legend((s1, s2), ("Correct Predictions", "Wrong Predictions"))
plt.grid(1)
plt.show()<jupyter_output><empty_output><jupyter_text>The magenta cross represents correct predictions and the yellow stars represent incorrect predictions.We see that even if there is an outlier, if the test value is close to that value, it will be predicted in that class, which may not always be right. Hence we move on to the k Nearest Neighbor. **7)  K Nearest Neighbour**Choosing the value of k for the algorithm could be a difficult task. A small value of k means that noise will have a higher influence on the result and a large value make it computationally expensive.We will have to check which value suits well. Let us start with k = 5.Now we will move on to K Nearest Neighbour classifier:<jupyter_code># Question 3) Algorithm - Calculate the Euclidian distance for each test dataset by comparing with every training dataset and assign it to the variable 'val'
#            Sort the distances obtained and take the k smallest values.
#            Take their species values.
#            The prediciton is the maximum frequency among all the k sorted values.  
#           # Assign the y_predict to ans[2] as a tuple => (0,1,0,2, and so on)  
start_time = time.time()
k = 5
y_predict = []
#Write the code below
y_predict = [KNN(X_train, y_train, 5, X_test.loc[i].values) for i in X_test.index]

# 1 mark
ans[2] = tuple (y_predict)
ans[2] = str(ans[2])
time_k5_NN = time.time() - start_time
# a):Concatenate X_test and y_test dataframe by column and assign it to the new dataframe 'Test_Data'
# b):Concatenate Test_data and Predicted dataframe by column assign it to the new dataframe 'Check_Prediction' 
# Write the code below
#Write the code below
Test_Data = pd.concat([X_test, y_test], axis=1, ignore_index=False)

temp = pd.DataFrame(y_predict, index=Test_Data.index, columns=["Predicted"])
Check_Prediction = pd.concat([Test_Data, temp], axis=1, ignore_index=False)<jupyter_output><empty_output><jupyter_text>We will calculate the accuracy now :<jupyter_code># Question 4) :Calculate the accuracy obtained for the algorithm used above
#             Assign the accuracy to ans[3] and store it as 'Accuracy_k5_NearestNeighbours' 
#NOTE - Accuracy = Correct_Predictions/Total_Predictions
#Write the code below
Accuracy_k5_NearestNeighbours = sum([1 for i in Check_Prediction.index \
                                     if Check_Prediction.loc[i, "Species_encoded"] == Check_Prediction.loc[i, "Predicted"]])/len(Check_Prediction)


# 1 mark
ans[3] = Accuracy_k5_NearestNeighbours
Accuracy_k5_NearestNeighbours<jupyter_output><empty_output><jupyter_text>Now let us have a look at the values that we did not predicted correctly :<jupyter_code>#Plot the data categorywise using scatter plot and visualize the data. Also include labels and legend.
# Write the code below
setosa_test = (Check_Prediction[Check_Prediction["Predicted"] == 0]).iloc[:, :2]
versicolor_test = (Check_Prediction[Check_Prediction["Predicted"] == 1]).iloc[:, :2]
virginica_test = (Check_Prediction[Check_Prediction["Predicted"] == 2]).iloc[:, :2]

fig = plt.figure(figsize=(10,7))
plt.scatter(setosa_test.iloc[:,0], setosa_test.iloc[:,1], color='red', label="setosa")
plt.scatter(versicolor_test.iloc[:,0], versicolor_test.iloc[:,1], color='green', label="versicolor")
plt.scatter(virginica_test.iloc[:,0], virginica_test.iloc[:,1], color='blue', label="virginica")

plt.xlabel("sepal length (cm)  ---->")
plt.ylabel("petal width (cm)  ---->")
plt.title("Scatter plots of the X_test (with y_predict) (category wise) for k = 5")
plt.legend(loc='upper left')
plt.grid(1)
plt.show()

print("\n")

#Implement the below code snippet to see the values that were not predicted correctly
fig = plt.figure(figsize=(10,7))
plt.xlabel("sepal length (cm)  ---->")
plt.ylabel("petal width (cm)  ---->")
plt.title("Scatter plot to compare the correct predictions for k = 5")

for index in Check_Prediction.index:
    if (Check_Prediction['Species_encoded'][index]==Check_Prediction['Predicted'][index]):
        s1 = plt.scatter(Check_Prediction['sepal length (cm)'][index],Check_Prediction['petal width (cm)'][index],marker='x',c='m')
    else:
        s2 = plt.scatter(Check_Prediction['sepal length (cm)'][index],Check_Prediction['petal width (cm)'][index],marker='*',c='y')

plt.legend((s1, s2), ("Correct Predictions", "Wrong Predictions"))
plt.grid(1)
plt.show()<jupyter_output><empty_output><jupyter_text>From this plot, it is visible that some of the outlier cases wrongly predicted in previous method are being taken care of by kNN. Still there could always be some errors near boundary lines. Choosing an optimum value of k is very important. Let us see the problems when k is too large or too less.First when k is too small :Let us check for k = 1. We are adding some outliers in the data to see what the KNN for k=1 predicts.<jupyter_code>#adding outliers in the data
X_test.loc[len(X_test.index)] = [4.5, 1.5]    ###check the added values in the data frames
X_test.loc[len(X_test.index)] = [4.8, 1.5]  
X_test.loc[len(X_test.index)] = [7, 1.5]  
X_test.loc[len(X_test.index)] = [5, 0.85]  
y_test.loc[len(y_test.index)] = 1
y_test.loc[len(y_test.index)] = 1
y_test.loc[len(y_test.index)] = 2
y_test.loc[len(y_test.index)] = 0
# Now implement KNN algorithm with k = 1
start_time = time.time()
k = 1
y_predict = []
#Write the code below
y_predict = [KNN(X_train, y_train, 1, X_test.loc[i].values) for i in X_test.index]

time_k1_NN = time.time() - start_time
# a):Concatenate X_test and y_test dataframe by column and assign it to the new dataframe 'Test_Data'
# b):Concatenate Test_data and Predicted dataframe by column assign it to the new dataframe 'Check_Prediction' 
# Write the code below
Test_Data = pd.concat([X_test, y_test], axis=1, ignore_index=False)

temp = pd.DataFrame(y_predict, index=Test_Data.index, columns=["Predicted"])
Check_Prediction = pd.concat([Test_Data, temp], axis=1, ignore_index=False)<jupyter_output><empty_output><jupyter_text>We will calculate the accuracy now :<jupyter_code># Question 5) :Calculate the accuracy obtained for the algorithm used above
#             Assign the accuracy to ans[4] and store it as 'Accuracy_k2_NearestNeighbours'
#NOTE - Accuracy = Correct_Predictions/Total_Predictions
#Write the code below
Accuracy_k2_NearestNeighbours = sum([1 for i in Check_Prediction.index \
                                     if Check_Prediction.loc[i, "Species_encoded"] == Check_Prediction.loc[i, "Predicted"]])/len(Check_Prediction)


# 1 mark
ans[4] = Accuracy_k2_NearestNeighbours
Accuracy_k2_NearestNeighbours<jupyter_output><empty_output><jupyter_text>As we can clearly see that KNN for k=1 doesn't do well in case of outliers. Hence, we increase the value of k to get better results. Let us check for k = 2.<jupyter_code># Now implement KNN algorithm with k = 2
start_time = time.time()
k = 2
y_predict = []
#Write the code below
y_predict = [KNN(X_train, y_train, 2, X_test.loc[i].values) for i in X_test.index]

time_k2_NN = time.time() - start_time
# a):Concatenate X_test and y_test dataframe by column and assign it to the new dataframe 'Test_Data'
# b):Concatenate Test_data and Predicted dataframe by column assign it to the new dataframe 'Check_Prediction' 
# Write the code below
Test_Data = pd.concat([X_test, y_test], axis=1, ignore_index=False)

temp = pd.DataFrame(y_predict, index=Test_Data.index, columns=["Predicted"])
Check_Prediction = pd.concat([Test_Data, temp], axis=1, ignore_index=False)<jupyter_output><empty_output><jupyter_text>We will calculate the accuracy now :<jupyter_code># Question 6) :Calculate the accuracy obtained for the algorithm used above
#             Assign the accuracy to ans[5] and store it as 'Accuracy_k2_NearestNeighbours'
#NOTE - Accuracy = Correct_Predictions/Total_Predictions
#Write the code below
Accuracy_k2_NearestNeighbours = sum([1 for i in Check_Prediction.index \
                                     if Check_Prediction.loc[i, "Species_encoded"] == Check_Prediction.loc[i, "Predicted"]])/len(Check_Prediction)


# 1 mark
ans[5] = Accuracy_k2_NearestNeighbours
Accuracy_k2_NearestNeighbours<jupyter_output><empty_output><jupyter_text>As we can see, when k is too small, it is better than Nearest Neighbour but __A__ accurate than k = 5.<jupyter_code>#Question 7) What will come inplace of A 
#            Assign the answer to ans[6] for A 
# A = less or more? Write your response as "less" or "more" to the ans[6].
# Answer is case sensitive. Adhere to the choices given.
# 1 marks
ans[6] = "less"<jupyter_output><empty_output><jupyter_text>Now let us see when k is large, for example k = 12 :<jupyter_code># Now implement KNN algorithm with k = 12
start_time = time.time()
k = 12
y_predict = []
#Write the code below
y_predict = [KNN(X_train, y_train, 12, X_test.loc[i].values) for i in X_test.index]

time_k12_NN = time.time() - start_time
# a):Concatenate X_test and y_test dataframe by column and assign it to the new dataframe 'Test_Data'
# b):Concatenate Test_data and Predicted dataframe by column assign it to the new dataframe 'Check_Prediction'
# Write the code below
Test_Data = pd.concat([X_test, y_test], axis=1, ignore_index=False)

temp = pd.DataFrame(y_predict, index=Test_Data.index, columns=["Predicted"])
Check_Prediction = pd.concat([Test_Data, temp], axis=1, ignore_index=False)<jupyter_output><empty_output><jupyter_text>We will calculate the accuracy now :<jupyter_code># Question 8) :Calculate the accuracy obtained for the algorithm used above
#              Assign the accuracy to ans[7] and store it as 'Accuracy_k12_NearestNeighbours'
#NOTE - Accuracy = Correct_Predictions/Total_Predictions
#Write the code below
Accuracy_k12_NearestNeighbours = sum([1 for i in Check_Prediction.index \
                                     if Check_Prediction.loc[i, "Species_encoded"] == Check_Prediction.loc[i, "Predicted"]])/len(Check_Prediction)


# 1 mark
ans[7] = Accuracy_k12_NearestNeighbours
Accuracy_k12_NearestNeighbours<jupyter_output><empty_output><jupyter_text>We see that the accuracy is good but it is lower than when k = 5. This is because when the dataset is small and the value of k is too big, it looks at too many points in the surrounding and sometimes gets the wrong prediction about which class it belongs to. **8) Nearest Mean Classification**Here we take the mean for each of the classes and compare the test cases with them to reduce computational time.<jupyter_code>setosa_PetalWidth = setosa['petal width (cm)'].mean()                         # Calculate the mean for all the features of all the classes
setosa_SepalLength = setosa['sepal length (cm)'].mean()

versicolor_PetalWidth = versicolor['petal width (cm)'].mean()
versicolor_SepalLength = versicolor['sepal length (cm)'].mean()

virginica_PetalWidth = virginica['petal width (cm)'].mean()
virginica_SepalLength = virginica['sepal length (cm)'].mean()
# Algorithm - Calculate the Euclidian distance for each test dataset by comparing with mean of every class
#            The predicited class is the minimum value obtained from all the Euclidian distance calculated for that test datapoint 
def KNN_Mean(x):
    distances = [0,0,0]
    distances[0] = euclidDistance((setosa_SepalLength, setosa_PetalWidth), x)
    distances[1] = euclidDistance((versicolor_SepalLength, versicolor_PetalWidth), x)
    distances[2] = euclidDistance((virginica_SepalLength, virginica_PetalWidth), x)

    return distances.index(min(distances))
start_time = time.time()
y_predict = []
#Write the code below
y_predict = [KNN_Mean(X_test.loc[i].values) for i in X_test.index]

time_NM = time.time() - start_time
# a):Concatenate X_test and y_test dataframe by column and assign it to the new dataframe 'Test_Data'
# b):Concatenate Test_data and Predicted dataframe by column assign it to the new dataframe 'Check_Prediction'
# Write the code below
Test_Data = pd.concat([X_test, y_test], axis=1, ignore_index=False)

temp = pd.DataFrame(y_predict, index=Test_Data.index, columns=["Predicted"])
Check_Prediction = pd.concat([Test_Data, temp], axis=1, ignore_index=False)<jupyter_output><empty_output><jupyter_text>We will calculate the accuracy now :<jupyter_code># Question 9) :Calculate the accuracy obtained for the algorithm used above
#              Assign the accuracy to ans[8] and store it as 'Accuracy_NearestMean'
#NOTE - Accuracy = Correct_Predictions/Total_Predictions
#Write the code below
Accuracy_NearestMean = sum([1 for i in Check_Prediction.index \
                                     if Check_Prediction.loc[i, "Species_encoded"] == Check_Prediction.loc[i, "Predicted"]])/len(Check_Prediction)


# 1 mark
ans[8] = Accuracy_NearestMean
Accuracy_NearestMean<jupyter_output><empty_output><jupyter_text>Now let us have a look at the values that we did not predicted correctly :<jupyter_code>#Plot the data categorywise using scatter plot and visualize the data. Also include labels and legend.
# Write the code below
setosa_test = (Check_Prediction[Check_Prediction["Predicted"] == 0]).iloc[:, :2]
versicolor_test = (Check_Prediction[Check_Prediction["Predicted"] == 1]).iloc[:, :2]
virginica_test = (Check_Prediction[Check_Prediction["Predicted"] == 2]).iloc[:, :2]

fig = plt.figure(figsize=(10,7))
plt.scatter(setosa_test.iloc[:,0], setosa_test.iloc[:,1], color='red', label="setosa")
plt.scatter(versicolor_test.iloc[:,0], versicolor_test.iloc[:,1], color='green', label="versicolor")
plt.scatter(virginica_test.iloc[:,0], virginica_test.iloc[:,1], color='blue', label="virginica")

plt.xlabel("sepal length (cm)  ---->")
plt.ylabel("petal width (cm)  ---->")
plt.title("Scatter plots of the X_test (with y_predict) (category wise) with KNN Mean")
plt.legend(loc='upper left')
plt.grid(1)
plt.show()

print("\n")

#Implement the below code snippet to see the values that were not predicted correctly
fig = plt.figure(figsize=(10,7))
plt.xlabel("sepal length (cm)  ---->")
plt.ylabel("petal width (cm)  ---->")
plt.title("Scatter plot to compare the correct predictions with KNN Mean")

for index in Check_Prediction.index:
    if (Check_Prediction['Species_encoded'][index]==Check_Prediction['Predicted'][index]):
        s1 = plt.scatter(Check_Prediction['sepal length (cm)'][index],Check_Prediction['petal width (cm)'][index],marker='x',c='m')
    else:
        s2 = plt.scatter(Check_Prediction['sepal length (cm)'][index],Check_Prediction['petal width (cm)'][index],marker='*',c='y')

plt.legend((s1, s2), ("Correct Predictions", "Wrong Predictions"))
plt.grid(1)
plt.show()<jupyter_output><empty_output><jupyter_text>We should only use this method if time complexity is really important to us as this method estimates the boundary just based on the mean which cannot be always relied on. Hence we see that as the mean of Green and Blue are close together, there are many wrong predictions in that region. **9) Compare Results**<jupyter_code># Do not make changes in this cell
# Simply execute it and move on
print("ACCURACIES:\n")
print("Nearest Neighbours=",Accuracy_NearestNeighbours)
print("K Nearest Neighbours(k = 5) =",Accuracy_k5_NearestNeighbours)
print("K Nearest Neighbours(k = 2) =",Accuracy_k2_NearestNeighbours)
print("K Nearest Neighbours(k = 12) =",Accuracy_k12_NearestNeighbours)
print("Nearest Mean=",Accuracy_NearestMean)
print("=========================================================================")
print("TIMINGS:\n")
print("Nearest Neighbours=",time_NN)
print("K Nearest Neighbours(k = 5) =",time_k5_NN)
print("K Nearest Neighbours(k = 2) =",time_k2_NN)
print("K Nearest Neighbours(k = 12) =",time_k12_NN)
print("Nearest Mean=",time_NM)
print("=========================================================================")<jupyter_output>ACCURACIES:

Nearest Neighbours= 0.9
K Nearest Neighbours(k = 5) = 0.9666666666666667
K Nearest Neighbours(k = 2) = 0.921875
K Nearest Neighbours(k = 12) = 0.9375
Nearest Mean= 0.796875
=========================================================================
TIMINGS:

Nearest Neighbours= 0.3653411865234375
K Nearest Neighbours(k = 5) = 0.3809821605682373
K Nearest Neighbours(k = 2) = 0.39295053482055664
K Nearest Neighbours(k = 12) = 0.396953821182251
Nearest Mean= 0.00498509407043457
=========================================================================<jupyter_text>On comparing the 3 timings of kNN, we see that the model with least k takes the __B__ time. It feels wrong but it does makes sense. I will explain why.When k is low, the decision boundary changes a lot of times, ie there is a sudden change in the class in a small distance, while when k is large, the decision boundary is smoother and does not changes as often which helps when it comes to time complexity.<jupyter_code>#Question 10) What will come inplace of B 
#            Assign the answer to ans[9] for B 
# B = highest or lowest? Write your response as "highest" or "lowest" to the ans[9].
# Answer is case sensitive. Adhere to the choices given.
# 1 marks
ans[9] =  "highest"<jupyter_output><empty_output><jupyter_text>On comparing the 3 plots, we can see that most of the errors occur at the boundary of 2 different classes. We reduce it slightly using the k nearest neighbours but not completely. Although Nearest mean has a significantly lower time than the other 2 processes, it is not a preffered method. It is because it has very low accuracy and it is very uncertain.The most preffered algorithm is the k nearest neighbour algorithm as it gives very good accuracy as well as acceptable runtime. **sklearn Library** There is an inbuilt function to directly perform kNN from sklearn library.<jupyter_code># Do not make changes in the below cells
# Simply execute it and move on
# Import the library

from sklearn.neighbors import KNeighborsClassifier<jupyter_output><empty_output><jupyter_text>We will first do for k = 4<jupyter_code>neigh = KNeighborsClassifier(n_neighbors=4)
neigh.fit(X_train,y_train)
Predicted = neigh.predict(X_test)
print(Predicted)<jupyter_output>[0 0 0 0 0 2 1 0 2 1 1 0 1 1 2 0 1 2 2 0 2 2 2 1 0 2 1 1 1 1 0 0 2 1 0 0 1
 0 2 1 2 1 0 0 2 1 0 1 2 1 0 0 2 1 1 0 2 0 2 1 1 1 1 1]<jupyter_text>sklearn also has a funtion to calculate accuracy<jupyter_code>y_test.values
#Import scikit-learn metrics module for accuracy calculation
from sklearn import metrics
# Model Accuracy, how often is the classifier correct?
print("Accuracy = ",metrics.accuracy_score(y_test.values,Predicted))
# Run the this cell to submit your answers and get score evaluated"
# Do not make any changes in this cell
# Simply execute it and move on

ans = [item for item in ans]
with open('ans.json', 'w') as f:
    json.dump(ans, f)
! ../submit ans.json<jupyter_output>'..' is not recognized as an internal or external command,
operable program or batch file.
{'fraction_non_alphanumeric': 0.11671338353990071, 'fraction_numerical': 0.017729897673162, 'mean_word_length': 4.837177212694658, 'pattern_counts': {'":': 0, '<': 123, '<?xml version=': 0, '>': 142, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 18, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '31', 'dir': 'jupyter-structured-clean-dedup', 'id': '134045', 'license': 'MIT', 'n_tokens_mistral': 10131, 'n_tokens_neox': 9618, 'n_words': 3044, 'path': 'LAB ASSIGN/basic_classification.ipynb', 'repo_name': 'Amarnathpg123/MLFAHandsOnPython', 'size': '29611'}