<jupyter_start><jupyter_text>Intro to Classification + KNN  LEARNING OBJECTIVES*After this lesson, you will be able to:*- Intuition behind the KNN algorithm- Implementing KNN with sklearn---  K Nearest Neighbors classification walkthroughFrom here on out we are going to look at how the kNN algorithm classifies an unknown point using the Iris dataset.--- Note on parametric vs. nonparametric methodsThus far, all of our tests and methods have been **parametric**. That is, we have assumed a certain distribution for our data. In linear regression our parameters are the coefficients in our model, and our estimate of the target is calculated from these parameters.There are alternatives in the case where we cannot assume a particular distribution for our data or choose not to. These methods are **nonparametric** When we make no assumptions about the distribution for our data, we call our data nonparametric. For nearly every parametric test, there is a nonparametric analog available. The KNN model is an example of a nonparametric model. You can see that there are no coefficients for the different predictors and our estimate is not represented by a formula of our predictor variables.--- kNN![](https://snag.gy/hatSE6.jpg)The pseudocode algorithm for kNN is as follows:```for unclassified_point in sample:    for known_point in known_class_points:        calculate distances (euclidean or other) between known_point and unclassified_point    for k in range of specified_neighbors_number:        find k_nearest_points in known_class_points to unclassified_point    assign class to unclassified_point using "votes" from k_nearest_points```>  Common KNN Distance Functions> These distance functions can be used with KNN.  Euclidean is the most common choice.>>  Euclidean  > $\sqrt{\sum\limits_{i=1}^k(x_i - y_i)^2}$>>  Manhattan > $\sum\limits_{i=1}^k \left| x_i - y_i \right|$>>  Minkowski> $\left(\sum_{i=1}^n |x_i-y_i|^p\right)^{1/p}$---[NOTE: in the case of ties, `sklearn`'s `KNeighborsClassifier()` will just choose the first class using uniform weights! If this is unappealing to you you can change the weights keyword argument to 'distance'.]  Euclidean distance---KNN typically uses one of two distance metrics: euclidean or manhattan. Other distance metrics (e.g. Minkwoski) are possible, but rare.Recal the famous Pythagorean Theorem![Alt text](http://ncalculators.com/images/pythagoras-theorem.gif)We can apply the theorem to calculate distance between points. This is called Euclidean distance. ![Alt text](http://rosalind.info/media/Euclidean_distance.png) $$\text{Euclidean  distance}=\sqrt{(x_1-x_2)^2+(y_1-y_2)^2}$$There are many different distance metrics, but Euclidean is the most common (and default in `sklearn`).  Load libraries---We'll need the following libraries for today's lecture:1. `pandas`2. `numpy`3. `matplotlib` and `seaborn`4. `KNeighborsClassifier` from `sklearn`'s `neighbors` module5. The `load_iris` function from `sklearn`'s `datasets` module6. `train_test_split` and `cross_val_score` from `sklearn`'s `model_selection` module7. `StandardScaler` from `sklearn`'s `preprocessing` module<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import load_iris

%matplotlib inline<jupyter_output><empty_output><jupyter_text>The Iris Dataset---> The data set consists of 50 samples from each of three species of Iris (Iris setosa, Iris virginica and Iris versicolor). Four features were measured from each sample: the length and the width of the sepals and petals, in centimeters. Based on the combination of these four features, Fisher developed a linear discriminant model to distinguish the species from each other. - [Wikipedia](https://en.wikipedia.org/wiki/Iris_flower_data_set)Most `sklearn` datasets can be ran as follows:```pythondata = load_iris()```<jupyter_code>flowers = load_iris()
print(flowers.DESCR)<jupyter_output>.. _iris_dataset:

Iris plants dataset
--------------------

**Data Set Characteristics:**

    :Number of Instances: 150 (50 in each of three classes)
    :Number of Attributes: 4 numeric, predictive attributes and the class
    :Attribute Information:
        - sepal length in cm
        - sepal width in cm
        - petal length in cm
        - petal width in cm
        - class:
                - Iris-Setosa
                - Iris-Versicolour
                - Iris-Virginica
                
    :Summary Statistics:

    ============== ==== ==== ======= ===== ====================
                    Min  Max   Mean    SD   Class Correlation
    ============== ==== ==== ======= ===== ====================
    sepal length:   4.3  7.9   5.84   0.83    0.7826
    sepal width:    2.0  4.4   3.05   0.43   -0.4194
    petal length:   1.0  6.9   3.76   1.76    0.9490  (high!)
    petal width:    0.1  2.5   1.20   0.76    0.9565  (high!)
    ============== ==== ==== ======= ===== ===========[...]<jupyter_text>`load_iris()` properties---The `data` variable has several important properties you'll need to be familiar with:1. `data.data`: This is your `X`. In our case, it's a 150x4 matrix of features.2. `data.target`: This is your `y`. It's an array of 150 values (0, 1, 2). Each index corresponds do a different species of Iris flower.3. `data.feature_names`: These are the names of each of your 4 features (corresponding to the 4 columns in `data.data`4. `data.target_names`: These are the names of your 3 Iris species: Setosa, Versicolor and Virginica. **The order in `data.target_names` corresponds to the index in `data.target`.**<jupyter_code>flowers.data[:5,:]
flowers.feature_names
flowers.target[:5]
flowers.target_names<jupyter_output><empty_output><jupyter_text>Challenge: Create a `pandas` DataFrame from `load_iris()`---<jupyter_code>df = pd.DataFrame(flowers.data, columns=flowers.feature_names)
df['species'] = flowers.target
df.head()<jupyter_output><empty_output><jupyter_text>Data Cleaning---Let's see if our `DataFrame` requires any cleaning. In the cells below:1. Check the `dtypes` to make sure every column is numerical2. Check for null values<jupyter_code>df.dtypes
df.isnull().sum()<jupyter_output><empty_output><jupyter_text>EDA: Visualizing KNN---Using `matplotlib`, create a scatter plot using two features from your `DataFrame`: `'petal length (cm)'` and `'petal width (cm)'`. Each dot should be colored according to its species.<jupyter_code>colors = ['red', 'green', 'blue']
color_series = df['species'].map(lambda s: colors[s])
plt.scatter(df['petal length (cm)'], df['petal width (cm)'], color=color_series)<jupyter_output><empty_output><jupyter_text>EDA: Pairplot---Let's expand on the scatter plot created in the previous step. We can use `seaborn`'s `.pairplot()` method to create scatter plots using all of our features.<jupyter_code>sns.pairplot(df, hue='species')<jupyter_output>/Users/rileydallas/anaconda3/lib/python3.7/site-packages/statsmodels/nonparametric/kde.py:488: RuntimeWarning: invalid value encountered in true_divide
  binned = fast_linbin(X, a, b, gridsize) / (delta * nobs)
/Users/rileydallas/anaconda3/lib/python3.7/site-packages/statsmodels/nonparametric/kdetools.py:34: RuntimeWarning: invalid value encountered in double_scalars
  FAC1 = 2*(np.pi*bw/RANGE)**2<jupyter_text>Train/Test split---Use the `train_test_split` function to split your data into a training set and a holdout set.<jupyter_code>X = df[flowers.feature_names]
y = df['species']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=.2, random_state=42, stratify=y)<jupyter_output><empty_output><jupyter_text>`StandardScaler`---Because KNN is calculating the distance between neighbors, it's highly sensitive to the magnitude of your features. For example, if we were using KNN on a housing dataset, a feature like square footage (measured in **thousands** of feet^2) can really affect the distance. Thus, in order for KNN to work properly, it's important to scale our data. In the cells below, create an instance of `StandardScaler` and use it to transform `X_train` and `X_test`.<jupyter_code>ss = StandardScaler()
X_train_sc = ss.fit_transform(X_train)
X_test_sc = ss.transform(X_test)<jupyter_output><empty_output><jupyter_text>Instantiate KNN---For the `KNeighborsClassifier`, there a few important parameters to keep in mind:1. `n_neighbors`: this is the "K" in KNN. The best K will change from problem to problem, but the default is 5.2. `weights`: The neighbors can all have an equal vote (`uniform`), or the closer points can have a higher weighted vote (`distance`).3. `p`: The distance metric. The default is Euclidean distance (2). Changing it to 1 is setting the distance to Manhattan.In the cell below, instantiate a `knn` model using the default parameters.<jupyter_code>knn = KNeighborsClassifier()<jupyter_output><empty_output><jupyter_text>Cross validation---In the cell below, use `cross_val_score` to see what accuracy we can expect from our KNN model.<jupyter_code>cross_val_score(knn, X_train_sc, y_train, cv=10).mean()<jupyter_output><empty_output><jupyter_text>Model fitting and evaluation---Now that we know what we can expect from our KNN model, let's 1. fit the model to `X_train_scaled`, `y_train`2. score it on `X_test_scaled`, `y_test`<jupyter_code>knn.fit(X_train_sc, y_train)
knn.score(X_train_sc, y_train)
knn.score(X_test_sc, y_test)
cross_val_score(knn, X_train_sc, y_train, cv=10).mean()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11121505041056023, 'fraction_numerical': 0.014343623323978797, 'mean_word_length': 4.827983040581466, 'pattern_counts': {'":': 0, '<': 47, '<?xml version=': 0, '>': 59, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '382007', 'license': 'Apache-2.0', 'n_tokens_mistral': 3085, 'n_tokens_neox': 2895, 'n_words': 1055, 'path': 'Notebook/Lesson-knn/solution-code/solution-code.ipynb', 'repo_name': 'sgdataguru/ITE_Machine_Learning_Workshop', 'size': '9621'}