<jupyter_start><jupyter_text>Python Practice Lecture 6 MATH 342W Queens College Author: <NAME> Date: February 17, 2022 Agenda:- Nearest Neighbor Algorithm- Error Handling and Warnings- Matrix Operations- SVMWe'll also do some data visualization using `matplotlib.pyplot`. The Nearest Neighbor AlgorithmThe KNN algorithm can be used for both classification and regression (but mostly classification). The model predicts on a new $x_*$ by looking through the training data and finding the closest k $x$ values, and labeling $x_*$ by the mode of the classes of the closest k many $x$ values. The algorithm finds those closest $x$ values with a distance function, typically the euclidean distance squared.We'll be using the Breast Cancer dataset again, we'll load it in from sklearn.<jupyter_code># Importing dataset
from sklearn.datasets import load_breast_cancer
import pandas as pd

# Initializing dataset
BC = load_breast_cancer()

# Initialize Breast Cancer df
df = pd.DataFrame(data = BC.data, columns = BC.feature_names)

# Snapshot
df.head()
# Loading in target variable
y_binary = BC.target
y_binary
# Define descriptive features
X = df.iloc[:, :9]
X.head()<jupyter_output><empty_output><jupyter_text>Let's say we want to build a nearest neighbor model with the first covariate only. We are then looking for the label (response) of the closest $x_i$. Here is a simple function that does it:<jupyter_code># Defining fist covariate
X_1 = X['mean radius']

# Declaring function
def nn_function(x_star):
    # Create array of distances to x_star
    distance_metrics = [(x - x_star)**2 for x in X_1] # list comprehension
    
    # Get minimum distance
    min_dist = min(distance_metrics)
    
    # Get index of minimum distance
    min_index = distance_metrics.index(min_dist)
    
    # return the value of y_binary at that index
    return y_binary[min_index]
# Test 1
nn_function(17.98)
# Test 2
nn_function(7.78)<jupyter_output><empty_output><jupyter_text>Why is this procedure silly for this dataset? Does this work in higher dimensions? You'll figure that out on your homework.Has someone coded this before? Of course, let's load it in now and put it to work.<jupyter_code>from sklearn.neighbors import KNeighborsClassifier
model = KNeighborsClassifier(n_neighbors=3)
# Train the model using the training sets
model.fit(X, y_binary)
# Predict model output

# Test output
print(y_binary[0])

# Test row
print(X.iloc[[0]])
model.predict(X.iloc[[0]])<jupyter_output><empty_output><jupyter_text>Here is an interesting exercise that will setup future classes. We will predict $\hat{y}$ values for all rows $x_i$ in $D$ and calculate the error rate.<jupyter_code># Declaring model
model = KNeighborsClassifier(n_neighbors=1)

# Train the model using the training sets
model.fit(X, y_binary)

# Getting yhats
yhat = model.predict(X)

yhat[0:5]
# error rate
sum(yhat != y_binary)/len(yhat)<jupyter_output><empty_output><jupyter_text>There are no errors! Can this be a good model? Very likely not, something must be wrong! It's too good to be true.This is our first example of "overfitting", we'll explore this concept in depth at a later time. Overfitting is a core concept in this class. Let's see how the model performs where $k > 1$.<jupyter_code># Declaring model, k > 1
model = KNeighborsClassifier(n_neighbors=10)

# Train the model using the training sets
model.fit(X, y_binary)

# Getting yhats
yhat = [model.predict(X.iloc[[i]])[0] for i in range(len(X))]

# error rate
sum(yhat != y_binary)/len(yhat) # Why is there now a difference between predictions and the actual data?
# What happens when I let k = n? What's going on?

# Declaring model
model = KNeighborsClassifier(n_neighbors=len(y_binary))

# Train the model using the training sets
model.fit(X, y_binary)

# Getting yhats
yhat = [model.predict(X.iloc[[i]])[0] for i in range(len(X))]

# What do you think yhat will be? Print the first 20 elements
yhat[0:20]
# error rate
sum(yhat != y_binary)/len(yhat)<jupyter_output><empty_output><jupyter_text>Lets build one more KNN model, this time we'll limit $D$ to the first two columns of `X`.We'll then output a visualization of the model!<jupyter_code># installing library
!pip install mlxtend
# importing dependencies
import matplotlib.pyplot as plt
import pandas as pd
from sklearn import datasets, neighbors
from mlxtend.plotting import plot_decision_regions
# fitting knn model
k = 3
x = X[['mean radius', 'mean texture']].values
y = y_binary
model = neighbors.KNeighborsClassifier(n_neighbors=k)
model.fit(x, y)

# Plotting decision region
plot_decision_regions(x, y, clf=model, legend=2)

# Adding axes annotations
plt.xlabel('mean radius')
plt.ylabel('mean texture')
plt.title('Knn with K=' + str(k))
plt.show()<jupyter_output><empty_output><jupyter_text>--- Error Handling and WarningsYou can write better functions if you make use of errors and warnings. Java forces you to catch likely errors via the "throws" designation for a method but there is no such requirement in R or Python.* Errors are unrecoverable, they halt execution i.e. red lights* Warnings (under usual execution) do not halt execution, but they display a message, i.e. yellow lightsIf you've ever wrote in JavaScript then you know how annoying it is for a program to crash and not know what went wrong. Thankfully for us a lot of Python libraries have helpful error messages. It's great practice to account for errors while writing your own functions.1. Prevents program from crashing if an error occurs:\If an error occurs in a program, we don’t want the program to unexpectedly crash on the user. Instead, error handling can be used to notify the user of why the error occurred and gracefully exit the process that caused the error.2. Saves time debugging errors:\Following reason 1, having the program display an error instead of immediately crashing will save a lot of time when debugging errors. The logic inside the error handler can be updated to display useful information for the developer, such as the code trackback, type of error, etc.3. Helps define requirements for the program:\If the program crashes due to bad input, the error handler could notify the user of why the error occurred and define the requirements and constraints of the program.We'll discuss exception handling by using `try-except` and `try-except-finally`.If you have a block of code that might fail, you can manage any exceptions by placing this code in a try: … except: … block:<jupyter_code>try:
    print("This class is math " + 342)
except TypeError:
    print("Error: can only concatenate str (not int)")
try:
    x = int(input("Please enter a number: "))
    y = 100 / x
except ValueError:
    print("Error: there was an error")
except ZeroDivisionError:
    print("Error: 0 is an invalid number")
except TypeError:
    print("Error: value passed must be integer")
except Exception:
    print("Error: another error occurred")
finally:
    print("This line will always run no matter what. It might be useful to put output a log here.")<jupyter_output>Please enter a number: 4
This line will always run no matter what. It might be useful to put output a log here.<jupyter_text>1. The statement inside the `try` block is executed.2. If the statement is successful, all `except` clauses are skipped and the code inside the `finally` clause is run.3. If the statement inside the `try` block fails, the code in the first `except` statement is executed. If the statement fails due to a `ValueError` (i.e. not being able to convert a non-digit to an int), the code in the `except ValueError` block is run.4. If the statement inside the `try` block fails and the error is not a `ValueError`, the second `except` statement is checked. If the statement fails due to a `ZeroDivisionError` (i.e. integer is being divided by zero), the code inside the `except ZeroDivisionError` block is run.5. The `finally` clause will always execute after the last task completes — regardless of whether the last task is in the `try` block or `except` block. Things to keep in mind when handling exceptions:1. Don’t let the code swallow the exception. We don’t want errors to go undetected by simply ignoring them. If you need to swallow an exception to avoid a fundamental issue, the architecture of the program needs to be re-evaluated.```try:        y = 100 / xexcept ZeroDivisionError:        pass ```2. Don’t declare new variables inside a try statement that might not be reached.```try:       y = 100 / x       z = 23 * yexcept ZeroDivisionError:       passprint(z)  z will be undeclared if an Exception is raised``` WarningsIn the previous demo we used the Boston Housing Data, this is the cell we ran to import the dataset. We add lines 5 and 6 to ignore the warnings, this time let's load the dataset without ignoring the warning.<jupyter_code># Import sklearn's datasets module
from sklearn import datasets

# Lines below are just to ignore warnings
#import warnings
#warnings.filterwarnings('ignore')

# Load the Boston Housing dataset as bh
bh = datasets.load_boston()

# The Boston dataset is essentially a dictionary, let's check it's keys
bh.keys()<jupyter_output>E:\Users\amira\anaconda\lib\site-packages\sklearn\utils\deprecation.py:87: FutureWarning: Function load_boston is deprecated; `load_boston` is deprecated in 1.0 and will be removed in 1.2.

    The Boston housing prices dataset has an ethical problem. You can refer to
    the documentation of this function for further details.

    The scikit-learn maintainers therefore strongly discourage the use of this
    dataset unless the purpose of the code is to study and educate about
    ethical issues in data science and machine learning.

    In this special case, you can fetch the dataset from the original
    source::

        import pandas as pd
        import numpy as np


        data_url = "http://lib.stat.cmu.edu/datasets/boston"
        raw_df = pd.read_csv(data_url, sep="\s+", skiprows=22, header=None)
        data = np.hstack([raw_df.values[::2, :], raw_df.values[1::2, :2]])
        target = raw_df.values[fdf8:f53e:61e4::18, 2]

    Alternative datasets include the California housing dataset (i[...]<jupyter_text>What an interesting warning! The warning states that there is an ethical problem with the dataset and that sklearn maintainers discourage the use of it. We ignore the warning because we only need data for practice purposes, still, as a budding data scientist you are accountable for considering the ethical concerns of the data you use.Read up on the [Boston Housing Dataset Controversy](https://medium.com/@docintangible/racist-data-destruction-113e3eff54a8). NumPyNumPy is the fundamental package for scientific computing in Python. It provides a multi-dimensional array object, an assortment of routines for fast operation on arrays such as discrete Fourier transforms, basic linear algebra and statistical operations.A lot of the source code is optimized in the C language. NumPy brings the computational power of languages like C and Fortran to Python! The library has many use cases in Python and is an essential component in so many other libraries.<jupyter_code># pip install the Numpy Library
!pip install numpy
# importing dependencies
import numpy as np
# Define a single row of values as a numpy array object
a = np.array([1, 2, 3, 4])
a
# print data type of a
print(type(a))
# print data type of elements in array a
a.dtype
# Define another single row of values as a numpy array object
# This time make the first value a float
b = np.array([1.01, 2, 3, 4])
b
# print data type of elements in array b
b.dtype<jupyter_output><empty_output><jupyter_text>All elements in a numpy array must be of the same data type. Now let's define a multidimensional array, that is, we'll nest equal-length arrays inside a super-array.<jupyter_code># Create an array where each element is a 3 element long array
c = np.array([[1, 2, 3],
              [4, 5, 6],
              [7, 8, 9]
            ])

c<jupyter_output><empty_output><jupyter_text>Matrix OperationsWe'll go over some handy matrix operations in Python. Following are the operations that we are going to perform using the Numpy library:* Matrix addition* Matrix subtraction* Matrix multiplication* Matrix transpose* Matrix determinant* Matrix inverse* Matrix rankWe'll also go over:* Slicing of a matrix* Printing single row or/and column of a matrix etc.* Generating Identity Matrix<jupyter_code># Generating matrix_1
matrix_1 = np.array([[1, 2, 3],
                     [4, 5, 6],
                     [7, 8, 9]
                    ])
matrix_1
# Generating matrix_2
matrix_2 = np.array([[1, 2, 3],
                     [0, 5, 6],
                     [0, 0, 9]
                    ])
matrix_2
# Matrix addition
matrix_sum = matrix_1 + matrix_2
matrix_sum
# Matrix subtraction
matrix_difference = matrix_1 - matrix_2
matrix_difference<jupyter_output><empty_output><jupyter_text>For the multiplication of two matrices, we will use the numpy.dot() function. The numpy.dot() function performs dot multiplication between the given two matrices. It handles matrix1 and matrix2 as an array and performs multiplication operations on them.<jupyter_code># Matrix multiplication
matrix_product = matrix_1.dot(matrix_2)
matrix_product
# You can also use `@` to multiply to matrices with NumPy
matrix_1 @ matrix_2
# Matrix transpose
matrix_product.transpose()
# Matrix determinant
np.linalg.det(matrix_product)
# Matrix Inverse
#np.linalg.inv(matrix_product)
# Returns an error bc the Matrix is not invertible
# Generating matrix
matrix = np.array([[1, 3],
                   [2, 5],
                 ])
matrix
# Matrix Inverse example 2
np.linalg.inv(matrix)
# Matrix rank
np.linalg.matrix_rank(matrix)<jupyter_output><empty_output><jupyter_text>**Slicing Elements from Python Matrix without using Numpy**Now, let’s move to the slicing of a Python matrix. The slicing of elements in the Python matrix is very similar to that of Python array slicing that we have performed above.To perform the slicing operation on elements of a Python matrix, use the following syntax:`matrix_1[row_start:row_end, col_start:col_end]`<jupyter_code># Slice matrix_1
matrix_1[0:2, 0:3]
# print the second row
matrix_1[1]
# print the last column
matrix_1[:, -1]
# generating identity matrix
dimension = int(input("Enter the dimension of identitiy matrix: "))
identity_matrix = np.identity(dimension, dtype="int") #dtype is useful for when doing arithmetic with other matrices
print(identity_matrix)
# Outputting matrix
matrix
# Inverse of matrix
matrix_inv = np.linalg.inv(matrix)
matrix_inv
# what do I expect to get when I multiply the inverse by the original matrix?
matrix @ matrix_inv<jupyter_output><empty_output><jupyter_text>Support Vector Machines SVMWe'll use a few visualizations to drive home the major points.<jupyter_code># Create df
df = pd.DataFrame({'response': [0, 0, 0, 1, 1, 1],
                   'first_feature': [1, 1, 2, 3, 3, 4],
                   'second_feature': [1, 2, 1, 3, 4, 3]})
df
# Lets plot the data using the Pandas plot() function
df.plot(x ='first_feature',
        y='second_feature', 
        kind = 'scatter',
        c = 'response',
        cmap="viridis")
#help(plt.scatter)
# We'll do the same with matplotlib
import matplotlib.pyplot as plt
# Output plot
scatter =plt.scatter(x =df['first_feature'],
                     y=df['second_feature'], 
                     c = df['response'],
                    )

# Setting axis labels
plt.xlabel('first_feature')
plt.ylabel('second_feature')

#add legend
plt.legend(*scatter.legend_elements())

# Display plot
plt.show()<jupyter_output><empty_output><jupyter_text>Now we fit a linear SVM. Since the data is linearly seperable, we can make $\lambda = 0$.<jupyter_code># Defining X
X = df.iloc[:, [1, 2]]
X
# Defining y
y = df.iloc[:, [0]]
y
#Import svm module
from sklearn import svm
#Create a svm Classifier object
clf = svm.SVC(kernel='linear') # Linear Kernel
# fit the data 
clf.fit(X, y)
#help(clf)<jupyter_output><empty_output><jupyter_text>Now that we've fit the data using the algorithm $\mathcal{A}$ SVM, let's check the weight vector (w-vec).<jupyter_code>clf.coef_[0]
# Plot Decision Region using mlxtend's awesome plotting function
from mlxtend.plotting import plot_decision_regions

X = df[['first_feature', 'second_feature']]
y = df['response']

plot_decision_regions(X=X.values, 
                      y=y.values.astype(np.int_),
                      clf=clf, 
                      legend=2)

# Adding axes annotations
plt.xlabel('first_feature')
plt.ylabel('second_feature')
plt.title('SVM - linearly seperable, lambda = 1')
plt.show()
#help(plot_decision_regions)<jupyter_output><empty_output><jupyter_text>To understand the hyperparameter, let's introduce another data point so the training data is no longer linearly separable.<jupyter_code># Adding row to end of DataFrame
df.loc[len(df.index)] = [0, 3.2, 3.2]
df
# Lets plot it

# defining D and y
X = df[['first_feature', 'second_feature']]
y = df['response']

# defining constants
lambda_value = 1
n = len(X)

#Create a svm Classifier
clf = svm.SVC(kernel='rbf', gamma =  (2 * n * lambda_value)**-1) # Radial basis function kernel

# fit the data
clf.fit(X, y)

# plotting
plot_decision_regions(X=X.values, 
                      y=y.values.astype(np.int_),
                      clf=clf, 
                      legend=2)

# Adding axes annotations
plt.xlabel('first_feature')
plt.ylabel('second_feature')
plt.title('SVM - non-linearly seperable, lambda = 1')
plt.show()<jupyter_output>E:\Users\amira\anaconda\lib\site-packages\sklearn\base.py:450: UserWarning: X does not have valid feature names, but SVC was fitted with feature names
  warnings.warn(<jupyter_text>The data is no longer linearly seperable, as we can visualize from the plot. Let's try to build a SVM at a different lambda $\lambda$ value. The SciKit Learn SVC module calls the R "cost" parameter gamma.<jupyter_code>#Create a svm Classifier

# defining constants
lambda_value = 0.01 #0.001 #1.3
n = len(X)

# initializing model
clf = svm.SVC(kernel='rbf', gamma =  (2 * n * lambda_value)**-1) # Radial basis function kernel

# fit the data on the response
clf.fit(X, y)

# plotting
plot_decision_regions(X=X.values, 
                      y=y.values.astype(np.int_),
                      clf=clf, 
                      legend=2)

# Adding axes annotations
plt.xlabel('first_feature')
plt.ylabel('second_feature')
plt.title('SVM - non-linearly seperable, lambda = ' + str(lambda_value))
plt.show()<jupyter_output>E:\Users\amira\anaconda\lib\site-packages\sklearn\base.py:450: UserWarning: X does not have valid feature names, but SVC was fitted with feature names
  warnings.warn(
{'fraction_non_alphanumeric': 0.08389889381713247, 'fraction_numerical': 0.011916849249185058, 'mean_word_length': 4.224455611390285, 'pattern_counts': {'":': 0, '<': 70, '<?xml version=': 0, '>': 72, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '69264', 'license': 'MIT', 'n_tokens_mistral': 5775, 'n_tokens_neox': 5422, 'n_words': 2309, 'path': 'Practice Notes/6 - Python Practice Lecture 6 MATH 342W Queens College - KNN & SVM.ipynb', 'repo_name': 'Hi-Tree/342-2022-Python-Practice-lectures', 'size': '18706'}