<jupyter_start><jupyter_text>An example of k-nearest neighbors classificationThis notebook is a brief explanation of how we can build a non parametric Machine Learning model in Python using [scikit-learn](http://scikit-learn.org). If you are unfamiliar with kNN classification, you can start by reading on [Wikipedia](https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm), or watch this very interesting  [video](https://www.youtube.com/watch?v=4ObVzTuFivY) to get up to speed.This example is built on the dataset provided for the [Kaggle competition](https://www.kaggle.com/c/titanic) "Titanic: Machine Learning for disaster". We are basically given information on the people who were on board, and we must build a model to predict what sort of people would have a better chance of survival ([spoiler](https://en.wikipedia.org/wiki/Women_and_children_first)).This is a classic example of a binary classification problem, since there are only two possible outcomes for each person. kNN is not the most sexy classification algorithm out there, but its simplicity can be a strong advantage in certain cases. In this example, we will use this method to predict whether a person will survive or not based on their age, sex and the class of their ticket. So... let's do it.<jupyter_code># Importing the libraries
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns
sns.set(style="white")
# Importing the dataset
dataset = pd.read_csv('../Data/train_titanic.csv')
dataset = dataset[dataset['Age'].notnull()]
cat_cols = ['Sex' ]
dataset = pd.get_dummies(dataset, columns = cat_cols)
X = dataset[['Pclass','Age','Sex_male']].values
y = dataset['Survived'].values<jupyter_output><empty_output><jupyter_text>Training set and Test setKaggle gives us a test set, but for this example we'd rather split it ourselves so we can visualize how the model is performing. We will keep 80% of the data in the training set.<jupyter_code>from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.25, random_state = 0)<jupyter_output><empty_output><jupyter_text>Feature scalingWe apply feature scaling to normalize the range of each variable. This ensures that each feature contributes approximately the same to the distances computed in the objective function. Note that both the training and the test set must be scaled.<jupyter_code>from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)<jupyter_output><empty_output><jupyter_text>Fitting Logistic Regression to the Training setOnce the data is clean and ready, we can build the classifier and fit it to the training data.<jupyter_code>from sklearn.neighbors import KNeighborsClassifier
classifier = KNeighborsClassifier(n_neighbors =10)
classifier.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>Predicting on the test setOnce the model has learned from the training data, we can make predictions on the test data using predict.<jupyter_code># Predicting the Test set results
y_pred = classifier.predict(X_test)<jupyter_output><empty_output><jupyter_text>Assessing the model's performanceThere are several ways to assess how good our predictions are. One of them is the [confusion matrix](http://www.dataschool.io/simple-guide-to-confusion-matrix-terminology/). This will quickly let us see how many good and bad predictions the model is making. Alternatively, we can use the accuracy score, which gives us the ratio of correctly classified samples. Note that we could change the accuracy score to account for the difference in the sample size for each possible outcome.<jupyter_code># Making the Confusion Matrix
from sklearn.metrics import confusion_matrix, accuracy_score
cm = confusion_matrix(y_test, y_pred)
#Compute the accuracy score
acc_score = accuracy_score(y_test, y_pred)
acc_score<jupyter_output><empty_output><jupyter_text>Awesome!So we have built a model with a fairly good accuracy (around 80%). I am sure we can do better, but we can be proud of these numbers with a linear model that uses only three features. Of course, we can do a lot better, but we are still working our way up to more complex (and maybe more accurate) algorithms.  Visualizing our resultsWe have used the accuracy score to assess our model's performance. This is very useful, but plotting the results is a more exciting way of looking at our work. So let's do that. We will first plot the predicted results, together with the decision boundary.kNN is a non linear classifier, and plotting the decision boundary is slightly trickier than simply plotting a plane. I have opted to show you both regions in the second plot by scattering points uniformly across the features space.<jupyter_code>from mpl_toolkits.mplot3d import axes3d
fig = plt.figure(figsize=(12,8))
ax = fig.add_subplot(111, projection='3d')

died = np.where(y_test==0)[0]
survived = np.where(y_test==1)[0]

p=ax.scatter(xs = X_test[died,0], ys =X_test[died,1], zs=X_test[died,2], 
             zdir='z', s=20, c='red',label = 'Died')
p=ax.scatter(xs = X_test[survived,0], ys =X_test[survived,1], zs=X_test[survived,2], 
             zdir='z', s=20, c='blue',label = 'Survived')

ax.set_xlabel('Class')
ax.set_ylabel('Age')
ax.set_zlabel('Sex')
ax.legend()
#ax.set_zticks([-1.33117269, 0.75121734]
ax.zaxis.set_ticks(np.unique(X_test[:,2]))
ax.zaxis.set_ticklabels(['Female','Male'])

ax.xaxis.set_ticks(np.unique(X_test[:,
                                    0]))
ax.xaxis.set_ticklabels(['1','2','3'])
plt.show()<jupyter_output><empty_output><jupyter_text>Visualizing our results (II)Since we know what our model is predicting on each side of the decision boundary, we can color the points using the actual data. This will let us visualize where our model is failing.<jupyter_code># Decision Boundaries


from mpl_toolkits.mplot3d import axes3d
fig = plt.figure(figsize=(12,8))



ax = fig.add_subplot(111, projection='3d')
x_surf = np.linspace(np.min(X_test[:,0]), np.max(X_test[:,0]), 30)
y_surf = np.linspace(np.min(X_test[:,1]), np.max(X_test[:,1]),30)
z_surf = np.linspace(np.min(X_test[:,2]), np.max(X_test[:,2]),30)
x_surf, y_surf, z_surf = np.meshgrid(x_surf,y_surf, z_surf)

predicted_volume = classifier.predict(np.array([x_surf.ravel(), y_surf.ravel(),
                                                z_surf.ravel()]).T).reshape(x_surf.shape)


died_volume = np.where(np.asarray(predicted_volume.ravel())==0)
survived_volume = np.where(np.asarray(predicted_volume.ravel())==1)

ax.scatter(x_surf.ravel()[died_volume[0]], y_surf.ravel()[died_volume[0]],
                z_surf.ravel()[died_volume[0]],
             zdir='z', s=0.5, c='red',label = 'Died',depthshade=True)


ax.scatter(x_surf.ravel()[survived_volume[0]], y_surf.ravel()[survived_volume[0]],
                z_surf.ravel()[survived_volume[0]],
             zdir='z', s=0.5, c='blue',label = 'Survived',depthshade=True)

ax.set_xlabel('Class')
ax.set_ylabel('Age')
ax.set_zlabel('Sex')
ax.legend()
#ax.set_zticks([-1.33117269, 0.75121734]
ax.zaxis.set_ticks(np.unique(X_test[:,2]))
ax.zaxis.set_ticklabels(['Female','Male'])

ax.xaxis.set_ticks(np.unique(X_test[:,0]))
ax.xaxis.set_ticklabels(['1','2','3'])

p=ax.scatter(xs = X_test[died,0], ys =X_test[died,1], zs=X_test[died,2], 
             zdir='z', s=20, c='k',label = 'Died')
p=ax.scatter(xs = X_test[survived,0], ys =X_test[survived,1], zs=X_test[survived,2], 
             zdir='z', marker = '^', s=20, c='k',label = 'Survived')
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11365430149960537, 'fraction_numerical': 0.01617995264404104, 'mean_word_length': 5.30954356846473, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 4, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '211702', 'license': 'Apache-2.0', 'n_tokens_mistral': 2581, 'n_tokens_neox': 2456, 'n_words': 763, 'path': 'KNN_Classification/KNN_Classifier.ipynb', 'repo_name': 'pablolozanot/MachineLearningExamples', 'size': '7602'}