<jupyter_start><jupyter_text>Introduction to Data Science  Lecture 15 continued<jupyter_code>import numpy as np
from sklearn import svm, metrics
from sklearn.datasets import make_moons, load_iris
from sklearn.model_selection import train_test_split, cross_val_score, cross_validate
    
import matplotlib.pyplot as plt
from matplotlib.colors import ListedColormap
%matplotlib inline
plt.rcParams['figure.figsize'] = (10, 6)
plt.style.use('ggplot')

import seaborn as sns
sns.set()<jupyter_output><empty_output><jupyter_text>Classification Recall that in *classification* we attempt to predict a categorical variable based on several features or attributes. We've already seen three methods for classification: 1. Logistic Regression+ k Nearest Neighbors (k-NN)+ Decision TreesAnother method for classification is [Support Vector Machines (SVM)](https://en.wikipedia.org/wiki/Support_vector_machine). These methods are especially good at classifying small to medium sized, complex datasets. We'll use the [sk-learn implementation](https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html).   Support Vector Machines**Idea:** Use tools from optimization to find the *best* lines (or hyperplanes or curves) that divide the two datasets.  Simplest version first: 2 classes and no "kernel transformation"**Data:** Given $n$ samples $(\vec{x}_1,y_1), (\vec{x}_2,y_2),\ldots,(\vec{x}_n,y_n)$, where $\vec{x}_i$ are attributes or features and $y_i$ are categorical variables that you want to predict. We assume that each $\vec{x}_i$ is a real vector and the $y_i$ are either $1$ or $-1$ indicating the class. **Goal:**  Find the "maximum-margin line" or more generally, the "maximum-margin hyperplane" that divides the two classes, which is defined so that the distance between the hyperplane and the nearest point in either group is maximized. $\qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad$ source: [wikipedia](https://en.wikipedia.org/wiki/Support_vector_machine)We can write any line as the set of points $\vec{x}$ satisfying $$\vec{w}\cdot\vec{x} = b$$ for some (normal) vector $\vec{w}$ and number $b$. The goal is to determine $\vec{w}$ and $b$. Once we have the best values of $\vec{w}$ $b$, our classifier is simply$$\vec{x} \mapsto \textrm{sgn}(\vec{w} \cdot \vec{x} - b).$$**Hard-margin.** If the data is linearly separable, then we find the separating line (hyperplane) with the *largest margin* ($b/\|w\|$). This is given by the solution to the optimization problem \begin{align*}\min_{\vec{w},b}  & \  \|\vec{w}\|^2  \\ \textrm{ subject to } & \  y_i(\vec{w}\cdot\vec{x}_i - b) \ge 1 , \textrm{for } i = 1,\,\ldots,\,n \end{align*}This can be written as a *convex optimization problem* and efficiently solved. Take Math 5770/6640: Introduction to Optimization to learn more! In this class, we'll simply use scikit-learn. From the picture, the max-margin hyperplane is determined by the $\vec{x}_i$ which are closest to it. These $\vec{x}_i$ are called *support vectors*.**Soft-margin.** For data that is not linearly separable (e.g., two moons dataset), we introduce a *penalty* or *loss* function for violating the constraint,  $y_i(\vec{w}\cdot\vec{x}_i + b) \ge 1$. One of these is the *hinge loss* function, given by $$g(\vec{x}_i; \vec{w},b) = \max\left(0, 1-y_i(\vec{w}\cdot\vec{x}_i - b)\right). $$We can see that this function is zero if the constraint is satisfied. If the constraint is not satisfied, we pay a penatly, which is proportional to the distance to the separating hyperplane.We then fix the parameter $C > 0$ and solve the problem $$\min_{\vec{w},b}  \ \frac{1}{n} \sum_{i=1}^n g(\vec{x}_i; \vec{w},b)  \ + \ C \|\vec{w}\|^2  . $$The parameter $C$ determines how much we penalize points for being on the wrong side of the line. **Question:** How to choose the parameter $C$? Cross Validation! (more on this below)  SVM and two moons<jupyter_code># there are two features contained in X and the labels are contained in y
X,y = make_moons(n_samples=500,random_state=1,noise=0.3)

# Plot the data, color by class
plt.scatter(X[y == 1, 0], X[y == 1, 1], color="DarkBlue", marker="s",label="class 1")
plt.scatter(X[y == 0, 0], X[y == 0, 1], color="DarkRed", marker="o",label="class 2")
plt.legend(scatterpoints=1)
plt.title('Two Moons Dataset')
plt.xlabel('feature 1')
plt.ylabel('feature 2')
plt.show()
model = svm.SVC(kernel='linear', C=10)

# Note or a 'linear' kernel, there is a faster method:
#
# from sklearn.svm import LinearSVC
# model = LinearSVC(C=10,loss="hinge")

model.fit(X, y)    
print(model)
# Plot the data, color by class
plt.scatter(X[y == 1, 0], X[y == 1, 1], color="DarkBlue", marker="s",label="class 1")
plt.scatter(X[y == 0, 0], X[y == 0, 1], color="DarkRed", marker="o",label="class 2")
plt.legend(scatterpoints=1)

# Plot the predictions made by SVM
x_min, x_max = X[:,0].min() - 1, X[:, 0].max() + 1
y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
xx, yy = np.meshgrid(np.linspace(x_min, x_max, 200),np.linspace(y_min, y_max, 200))
zz = model.predict(np.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)
plt.contourf(xx, yy, zz, cmap=ListedColormap(['DarkRed', 'DarkBlue']), alpha=.2)
plt.contour(xx, yy, zz, colors="black", alpha=1, linewidths=0.2) 

plt.xlim(x_min, x_max)
plt.ylim(y_min, y_max)
plt.title('Classification of Two Moons using SVM')
plt.xlabel('feature 1')
plt.ylabel('feature 2')
plt.show()
print('Confusion Matrix:')
y_pred = model.predict(X)
print(metrics.confusion_matrix(y_true = y, y_pred = y_pred))

print('Accuracy = ', metrics.accuracy_score(y_true = y, y_pred = y_pred))<jupyter_output>Confusion Matrix:
[[213  37]
 [ 33 217]]
Accuracy =  0.86<jupyter_text>Support Vector Machines **Basic idea:** The dataset may not be linearly separable, but it may be if we apply a *nonlinear kernel transformation*. **Example:** two "rings" of points**Algorithmic challenge:** Not increasing the computationally complexity of the optimization problem. **(Very) rough sketch:**Replace the linear classifier $$\vec{x} \mapsto \textrm{sgn}(\vec{w} \cdot \vec{x} - b).$$with a nonlinear one $$\vec{x} \mapsto \textrm{sgn}( f(x)).$$Choices of the nonlinear transformation in scikit-learn are:- linear: $\langle x, x'\rangle$.- polynomial: $(\gamma \langle x, x'\rangle + r)^d$. $d$ is specified by keyword degree, $r$ by coef0.- rbf: $\exp(-\gamma |x-x'|^2)$. $\gamma$ is specified by keyword gamma, must be greater than 0.- sigmoid:  $\tanh(\gamma \langle x,x'\rangle + r)$, where $r$ is specified by coef0.The most popular choice is rbf.<jupyter_code>model = svm.SVC(kernel='rbf',C=20,gamma='scale')
model.fit(X, y)

# Plot the data, color by class
plt.scatter(X[y == 1, 0], X[y == 1, 1], color="DarkBlue", marker="s",label="class 1")
plt.scatter(X[y == 0, 0], X[y == 0, 1], color="DarkRed", marker="o",label="class 2")
plt.legend(scatterpoints=1)

# Plot the predictions made by SVM
x_min, x_max = X[:,0].min() - 1, X[:, 0].max() + 1
y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
xx, yy = np.meshgrid(np.linspace(x_min, x_max, 200),np.linspace(y_min, y_max, 200))
zz = model.predict(np.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)
plt.contourf(xx, yy, zz, cmap=ListedColormap(['DarkRed', 'DarkBlue']), alpha=.2)
plt.contour(xx, yy, zz, colors="black", alpha=1, linewidths=0.2) 

plt.xlim(x_min, x_max)
plt.ylim(y_min, y_max)
plt.title('Classification of Two Moons using SVM')
plt.xlabel('feature 1')
plt.ylabel('feature 2')
plt.show()
print('Confusion Matrix:')
y_pred = model.predict(X)
print(metrics.confusion_matrix(y_true = y, y_pred = y_pred))

print('Accuracy = ', metrics.accuracy_score(y_true = y, y_pred = y_pred))<jupyter_output>Confusion Matrix:
[[230  20]
 [ 25 225]]
Accuracy =  0.91<jupyter_text>More than 2 classes?  There are a few different methods for extending binary classification to more than 2 classes. -  **one vs. one:** Consider all pairs of classes (if there are $k$ classes, then there are $\binom k 2$ different pairs of classes. For each pair, we develop a classifier. For a new point, we perform all classifications and then choose the class that was most frequently assigned. (The classifiers all vote on a class.)- **all vs. one:**  We compare one of the $k$ classes to the remaining $k-1$ classes. We assign the class to the observation which we are most confident (we have to make this precise).    Dataset: The Iris datasetThis dataset contains 4 features (attributes) of 50 samples containing 3 different species of iris plants. The goal is to classify the species of iris plant given the attributes. **Classes:**1. Iris Setosa + Iris Versicolour + Iris Virginica**Features (attributes):**1. sepal length (cm) + sepal width (cm) + petal length (cm) + petal width (cm)<jupyter_code>df = sns.load_dataset("iris") # built-in dataset in seaborn 
sns.pairplot(df, hue="species");
# import data, scikit-learn also has this dataset built-in
iris = load_iris()

# For easy plotting and interpretation, we only use first 2 features here. 
# We're throwing away useful information - don't do this at home! 
X = iris.data[:,:2]
y = iris.target

# Create color maps
cmap_light = ListedColormap(['#FFAAAA', '#AAFFAA', '#AAAAFF'])
cmap_bold = ListedColormap(['#FF0000', '#00FF00', '#0000FF'])

# plot data
plt.scatter(X[:, 0], X[:, 1], c=y,  marker="o", cmap=cmap_bold, s=30)

x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
plt.xlim(x_min, x_max)
plt.ylim(y_min, y_max)    
plt.title('Iris dataset')
plt.xlabel('feature 1')
plt.ylabel('feature 2')
plt.show()
# SVM 
svm_iris = svm.SVC(kernel='rbf',C=3,gamma='scale')
svm_iris.fit(X, y)

# plot classification 
xx, yy = np.meshgrid(np.linspace(x_min, x_max, 200),np.linspace(y_min, y_max, 200))
zz = svm_iris.predict(np.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)
plt.pcolormesh(xx, yy, zz, cmap=cmap_light)

# plot data
plt.scatter(X[:, 0], X[:, 1], c=y, cmap=cmap_bold, s=30)

plt.title('Classification of Iris dataset using SVM')
plt.xlabel('feature 1')
plt.ylabel('feature 2')
plt.xlim(x_min, x_max)
plt.ylim(y_min, y_max)    
plt.show()
print('Confusion Matrix:')
y_pred = svm_iris.predict(X)
print(metrics.confusion_matrix(y_true = y, y_pred = y_pred))

print('Accuracy = ', metrics.accuracy_score(y_true = y, y_pred = y_pred))<jupyter_output>Confusion Matrix:
[[50  0  0]
 [ 0 38 12]
 [ 0 15 35]]
Accuracy =  0.82<jupyter_text>Cross Validation **Recall:** Cross validation is a method for assessing how well a model will generalize to an independent data set. **Idea:** We split the dataset into a *training set* and a *test set*. We train the model on the training set. We measure the accuracy of the model on the test set. There are several different ways to do this. Two popular methods are: - **k-fold cross validation** The data is randomly partitioned into k (approximately) equal sized subsamples (folds). For each of the k folds, the method is trained on the other k-1 folds and tested on that fold. The accuracy is computed using each of the k folds as the test set. - **leave-one-out cross validation** Same as above with k=nA variety of function are implemented in scikit-learn for cross validation. - The [train_test_split](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html) function randomly splits the data for cross validation- The [cross_val_score](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_score.html) function splits the data and measures accuracy on the test set.  The parameter *cv* is the k in k-fold cross validation. The parameter *scoring* specifies how you want to [evaluate the model](https://scikit-learn.org/stable/modules/model_evaluation.html) (we can just use accuracy). - The [cross_validate](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_validate.html) function evaluate metric(s) by cross-validation and also record fit/score times. So, cross-validate is different from cross_val_score in two ways:    1. It allows specifying multiple metrics for evaluation.    + It returns a dict containing fit-times, score-times (and optionally training scores as well as fitted estimators) in addition to the test score.<jupyter_code># note: here we are again using all 4 features of the data
scores = cross_val_score(estimator = svm_iris, X = iris.data, y = iris.target, cv=5, scoring='accuracy')
print(scores)

scoring = ['accuracy','precision_macro','recall_macro']
scores = cross_validate(estimator = svm_iris, X = iris.data, y = iris.target, cv=5, scoring=scoring)
print(sorted(scores.keys()))
print(scores['test_accuracy'])
print(scores['fit_time'])
print(scores['test_accuracy'])
print(scores['test_recall_macro'])
print(scores['test_precision_macro'])<jupyter_output>[0.96666667 1.         1.         0.96666667 1.        ]
['fit_time', 'score_time', 'test_accuracy', 'test_precision_macro', 'test_recall_macro', 'train_accuracy', 'train_precision_macro', 'train_recall_macro']
[0.96666667 1.         1.         0.96666667 1.        ]
[0.00082207 0.0010469  0.000808   0.00079894 0.00068283]
[0.96666667 1.         1.         0.96666667 1.        ]
[0.96666667 1.         1.         0.96666667 1.        ]
[0.96969697 1.         1.         0.96969697 1.        ]<jupyter_text>Incorporating cross-validation to choose SVM parameter, $C$Recall above that when the data is not linearly separable, introduce a loss function $g$ which penalizes the violation of datapoints lying on the wrong side of the hyperplane and solve the problem $$\min_{\vec{w},b}  \ \frac{1}{n} \sum_{i=1}^n g(x_i; \vec{w},b)  \ + \ C \|\vec{w}\|  . $$Here, the parameter $C>0$ determines how much we penalize points for being on the wrong side of the line. Generally, the function *get_params()* an be used to see what parameters are available to change in a model. We can use cross-validation to choose the parameter $C$.<jupyter_code>svm_iris.get_params()

Cs = np.linspace(.01,200,100)
Accuracies = np.zeros(Cs.shape[0])
for i,C in enumerate(Cs): 
    svm_iris = svm.SVC(kernel='rbf', C = C,gamma='scale')
    scores = cross_val_score(estimator = svm_iris, X = iris.data, y = iris.target, cv=5, scoring='accuracy')    
    Accuracies[i]  = scores.mean()
        
plt.plot(Cs,Accuracies)
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14227925791602733, 'fraction_numerical': 0.02657274375784628, 'mean_word_length': 4.763263665594855, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 23, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '50532', 'license': 'MIT', 'n_tokens_mistral': 5193, 'n_tokens_neox': 4803, 'n_words': 1626, 'path': '15-Classification2/15b-Classification2-SVM.ipynb', 'repo_name': 'durbank/2020-datascience-lectures', 'size': '14338'}