<jupyter_start><jupyter_text>Support Vector Machines This part of the workshop is dedicated to Support Vector Machines. The aims of this part are (i) to appreciate how the use of kernels results in non-linear decision boundaries, and (ii) to see how regularisation and kernel parameters affect this shape. We start with same old initialisation line.<jupyter_code>%pylab inline
import numpy as np
import matplotlib.pyplot as plt<jupyter_output>Populating the interactive namespace from numpy and matplotlib<jupyter_text>Cats Dataset This time we are going to use a real dataset that is not linearly separable. The dataset consists of 150 trainig examples with two attributes each. Each example corresponds to a cat and the two attributes are some physiological measurements for that cat. Along with the measuremnets, the sex of the cat is also recorded as a label. We encode sex using 1 for female cats and -1 for male cats.*(This data was popularised by the following publication: <NAME>. (1947) The analysis of covariance method for the relation between a part and the whole. Biometrics, 3, 65–68.)*<jupyter_code>def load_cats_data():
    #np.savetxt('cats_gender.csv', d , delimiter=",")
    d= np.loadtxt('cats_gender.csv', delimiter=",")
    X = d[:,[0,1]]
    y = d[:,2]
    
    return X,y<jupyter_output><empty_output><jupyter_text>This is what the data looks like (blue circles are male cats, red circles are female cats).<jupyter_code>X,y = load_cats_data()
plt.plot(X[y==-1,0], X[y==-1,1], "bo")
plt.plot(X[y==1,0], X[y==1,1], "ro")<jupyter_output><empty_output><jupyter_text>SVM Decision Boundaries We first do feature normalisation step i.e. re-scale each feature to have zero mean and unit variance. Note that both hard margin and soft margin variations have $||w||$ term. In this term each feature weight is treated equally, hence it is important that the features are indeed approximately on the same range.<jupyter_code>mean_r = np.mean(X, 0)
std_r = np.std(X, 0)
X_norm = ((X - mean_r) / std_r) 

plt.plot(X_norm[y==-1,0], X_norm[y==-1,1], "bo")
plt.plot(X_norm[y==1,0], X_norm[y==1,1], "ro")<jupyter_output><empty_output><jupyter_text>Now, we will define an auxiliary function for plotting decision boundaries.<jupyter_code># X are training data instances, y are labels
# cls is a trained SVM classifier,
# and name is the title of the plot
def plot_boundary(X, y, cls, name):
    # create a mesh to plot in
    h = 0.02 # step size in the mesh
    x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
    y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
    xx, yy = np.meshgrid(arange(x_min, x_max, h),
                        arange(y_min, y_max, h))
    # plot the decision boundary. For that, we will assign a color to each
    # point in the mesh [x_min, m_max]x[y_min, y_max].
    Z = cls.predict(np.c_[xx.ravel(), yy.ravel()])
    
    # put the result into a color plot
    Z = Z.reshape(xx.shape)
    contourf(xx, yy, Z, cmap=cm.Paired, alpha=0.7)
    
    # plot also the training points
    #scatter(X[:, 0], X[:, 1], c=y*5, cmap=cm.Paired)
    plt.plot(X[y==-1,0], X[y==-1,1], "b.")
    plt.plot(X[y==1,0], X[y==1,1], "r.")
    xlim(xx.min(), xx.max())
    ylim(yy.min(), yy.max())
    title(name)<jupyter_output><empty_output><jupyter_text>In the following exercises, you will be asked to experiment with various parameters. In all cases, you should pay attention to how these parameters control the degree of overfitting and try to explain the observed effect. Now train soft margin SVM classifier with a linear kernel, which means that the original attributes are not transformed. Note that SVMs are called Support Vector Classifiers (SVCs) in this package. *Experiment with different values of $C$ and see how it affects the result.*<jupyter_code>from sklearn import svm
svc = svm.SVC(kernel='linear', C=1).fit(X_norm, y) # linear kernel, no feature transformation
plot_boundary(X_norm, y, svc, 'SVM with linear kernel')<jupyter_output><empty_output><jupyter_text>This doesn't work well, because the data is not linearly separable. Now try a polynomial kernel to see if the shape of the decision boundary can be changed. *Experiment with different polynomial degrees.*<jupyter_code>poly_svc = svm.SVC(kernel='poly', degree=10, C=5).fit(X_norm, y) # polynomial kernel
plot_boundary(X_norm, y, poly_svc, 'SVM with polynomial kernel')<jupyter_output><empty_output><jupyter_text>Finally, let's explore the Gaussian kernel (aka RBF kernel). *Experiment with different standard deviations of the kernel.*<jupyter_code>rbf_svc = svm.SVC(kernel='rbf', gamma=10, C=5).fit(X_norm, y) # Gaussian (aka RBF) kernel
plot_boundary(X_norm, y, rbf_svc, 'SVM with RBF kernel')<jupyter_output><empty_output><jupyter_text>*Which kernel do you think works best for this data and why?*Now, we will show the importance of re-scaling our features. Let's multiply one of the features by a large number, e.g., 10^4, and apply SVM (with any kernel). How do you interpret the result ?<jupyter_code>X1 = X.copy() 
X1[:,0] = X[:,0]* 10**4
print( np.min(X,axis=0),np.max(X,axis=0))

print( np.min(X1,axis=0),np.max(X1,axis=0))

plt.plot(X1[y==-1,0], X1[y==-1,1], "bo")
plt.plot(X1[y==1,0], X1[y==1,1], "ro")


rbf_svc = svm.SVC(kernel='rbf', gamma=5, C=5).fit(X, y) # Gaussian (aka RBF) kernel
plot_boundary(X, y, rbf_svc, 'SVM with RBF kernel')
print(rbf_svc.score(X, y))
from sklearn.model_selection import cross_val_score
result = (cross_val_score(rbf_svc, X, y, cv=10))
print(np.mean(result))<jupyter_output>0.868055555556
0.629743589744
{'fraction_non_alphanumeric': 0.1233098972417523, 'fraction_numerical': 0.022354425815756265, 'mean_word_length': 4.864693446088795, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 36, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '580842', 'license': 'MIT', 'n_tokens_mistral': 1980, 'n_tokens_neox': 1897, 'n_words': 666, 'path': '5_SVM_RandomForest/7a_SVM.ipynb', 'repo_name': 'inverthermit/machine-learning-playboard', 'size': '5552'}