<jupyter_start><jupyter_text>CPSC 483 Project 3 - SVM classifiers  <NAME>For this projct, We will use 3 datasets with 3 columns each of the following form:Feature x1 **|** Feature x2 **|** Label tn = {0, 1}Then we will use SVM classifiers with differnt kernel methods to find an appropriate model to predict the class of a new datapoint with features [x1new, x2new]<jupyter_code># import libraries
import numpy as np
import pandas as pd 
import matplotlib.pyplot as plt
from matplotlib import colors
from sklearn import svm
from sklearn.metrics import confusion_matrix
from sklearn.model_selection import GridSearchCV<jupyter_output><empty_output><jupyter_text>--- Experiment 1First, we will use a Support Vector Machine with a *linear* kernel to fit and score a binary classifier for *dataset 1*.<jupyter_code># Import dataset 1 
df = pd.read_csv("dataset1.csv")
X1 = df[['x1', 'x2']].to_numpy() # feature vector 
t1 = np.ravel(df[['target']].to_numpy()) # target vector (or lables)

# Fit and score data 
linear1 = svm.SVC(kernel='linear', gamma='scale').fit(X1, t1) 
linear1_score = linear1.score(X1, t1)

print("SVM With a linear kernel score:", linear1_score)<jupyter_output>SVM With a linear kernel score: 1.0<jupyter_text>--- Experiment 2Now we are going to plot the decision boundary from experiment 1<jupyter_code># Create meshgrid 
xx, yy = np.meshgrid(np.linspace(0,40), np.linspace(0,40))
# Get predictions
z = linear1.predict(np.c_[xx.ravel(), yy.ravel()])
z = z[:].reshape(xx.shape) 
# Plot decisoin boundaries and data points
fig, ax = plt.subplots()
ax.scatter(X1[:, 0], X1[:, 1], cmap=plt.cm.binary, zorder=2)
ax.contour(xx, yy, z, [0.5])
ax.set(title="Linear kernel - dataset 1")
print("")<jupyter_output><jupyter_text>--- Experiment 3Next, we'll repeat the previous 2 experiments, but this time with a *poly*, *rbf*, and *sigmoid* kernels.<jupyter_code># Fit and score data for poly kernel 
poly1 = svm.SVC(kernel='poly', gamma='scale').fit(X1, t1) 
poly1_score = poly1.score(X1, t1)

# Fit and score data for rbf kernel
rbf1 = svm.SVC(kernel='rbf', gamma='scale').fit(X1, t1) 
rbf1_score = rbf1.score(X1, t1)

# Fit and score data for sigmoid kernel
sigmoid1 = svm.SVC(kernel='sigmoid', gamma='scale').fit(X1, t1) 
sigmoid1_score = sigmoid1.score(X1, t1)

# Plot decision boundaries for the new models 
xx, yy = np.meshgrid(np.linspace(0,40), np.linspace(0,40))
# Poly kernel 
z = poly1.predict(np.c_[xx.ravel(), yy.ravel()])
z = z[:].reshape(xx.shape) 
fig, ax = plt.subplots()
ax.scatter(X1[:, 0], X1[:, 1], cmap=plt.cm.binary, zorder=2)
ax.contour(xx, yy, z, [0.5])
ax.set(title="Poly kernel - dataset 1")

# Rbf kernel 
z = rbf1.predict(np.c_[xx.ravel(), yy.ravel()])
z = z[:].reshape(xx.shape) 
fig, ax = plt.subplots()
ax.scatter(X1[:, 0], X1[:, 1], cmap=plt.cm.binary, zorder=2)
ax.contour(xx, yy, z, [0.5])
ax.set(title="RBF kernel - dataset 1")

# Sigmoid kernel 
z = sigmoid1.predict(np.c_[xx.ravel(), yy.ravel()])
z = z[:].reshape(xx.shape) 
fig, ax = plt.subplots()
ax.scatter(X1[:, 0], X1[:, 1], cmap=plt.cm.binary, zorder=2)
ax.contour(xx, yy, z, [0.5])
ax.set(title="Sigmoid kernel - dataset 1")

print("poly kernel score:   ", poly1_score)
print("rbf kernel score:    ", rbf1_score)
print("sigmoid kernel score:", sigmoid1_score)<jupyter_output>poly kernel score:    1.0
rbf kernel score:     1.0
sigmoid kernel score: 0.16<jupyter_text>Do different kernels produce different decision boundaries? Yes each decision boundary is different despite using the same training data.  --- Experiment 4 Now we are going to repeat the previous 3 experiments with dataset 2: - fit and score models using linear, poly, rbf, and sigmoid kernels - plot each decision boundary<jupyter_code># Import dataset 2 
df = pd.read_csv("dataset2.csv")
X2 = df[['x1', 'x2']].to_numpy() # feature vector 
t2 = np.ravel(df[['target']].to_numpy()) # target vector (or lables)

# Fit and score data for linear kernel 
linear2 = svm.SVC(kernel='linear', gamma='scale').fit(X2, t2) 
linear2_score = linear2.score(X2, t2)

# Fit and score data for poly kernel 
poly2 = svm.SVC(kernel='poly', gamma='scale').fit(X2, t2) 
poly2_score = poly2.score(X2, t2)

# Fit and score data for rbf kernel
rbf2 = svm.SVC(kernel='rbf', gamma='scale').fit(X2, t2) 
rbf2_score = rbf2.score(X2, t2)

# Fit and score data for sigmoid kernel
sigmoid2 = svm.SVC(kernel='sigmoid', gamma='scale').fit(X2, t2) 
sigmoid2_score = sigmoid2.score(X2, t2)

# Plot decision boundaries models trained on dataset 2
xx, yy = np.meshgrid(np.linspace(0,40), np.linspace(0,40))
# Linear kernel 
z = linear2.predict(np.c_[xx.ravel(), yy.ravel()])
z = z[:].reshape(xx.shape) 
fig, ax = plt.subplots()
ax.scatter(X2[:, 0], X2[:, 1], cmap=plt.cm.binary, zorder=2)
ax.contour(xx, yy, z, [0.5])
ax.set(title="Linear kernel - dataset 2")

# Poly kernel 
z = poly2.predict(np.c_[xx.ravel(), yy.ravel()])
z = z[:].reshape(xx.shape) 
fig, ax = plt.subplots()
ax.scatter(X2[:, 0], X2[:, 1], cmap=plt.cm.binary, zorder=2)
ax.contour(xx, yy, z, [0.5])
ax.set(title="Poly kernel - dataset 2")

# Rbf kernel 
z = rbf2.predict(np.c_[xx.ravel(), yy.ravel()])
z = z[:].reshape(xx.shape) 
fig, ax = plt.subplots()
ax.scatter(X2[:, 0], X2[:, 1], cmap=plt.cm.binary, zorder=2)
ax.contour(xx, yy, z, [0.5])
ax.set(title="RBF kernel - dataset 2")

# Sigmoid kernel 
z = sigmoid2.predict(np.c_[xx.ravel(), yy.ravel()])
z = z[:].reshape(xx.shape) 
fig, ax = plt.subplots()
ax.scatter(X2[:, 0], X2[:, 1], cmap=plt.cm.binary, zorder=2)
ax.contour(xx, yy, z, [0.5])
ax.set(title="Sigmoid kernel - dataset 2")

print("Linear kernel score: ", linear2_score)
print("Poly kernel score:   ", poly2_score)
print("RBF kernel score:    ", rbf2_score)
print("sigmoid kernel score:", sigmoid2_score)<jupyter_output>Linear kernel score:  0.62
Poly kernel score:    0.815
RBF kernel score:     1.0
sigmoid kernel score: 0.42<jupyter_text>Which kernel produces the best decision boundary? The best kernel for this dataset is the *rbf* kernel. It is the only one that correctly classified all datapoints from the four kernels we modeled.  ---  Experiment 5 We are going to change the degree for the *poly* kernel form the default degree 3 to degree 5. Then we'll plot decision boundaries and create a confusion matrix for both models to see the differences between the two.<jupyter_code># Fit and score data for poly kernel degree 5
poly2d5 = svm.SVC(kernel='poly', degree=5, gamma='scale').fit(X2, t2) 
poly2d5_score = poly2d5.score(X2, t2)
print("Degree 3 score:", poly2_score)
print("Degree 5 score:", poly2d5_score)

# Plot both boundaries
xx, yy = np.meshgrid(np.linspace(0,40), np.linspace(0,40))
# Degree 3
z = poly2.predict(np.c_[xx.ravel(), yy.ravel()])
z = z[:].reshape(xx.shape) 
fig, ax = plt.subplots()
ax.scatter(X2[:, 0], X2[:, 1], cmap=plt.cm.binary, zorder=2)
ax.contour(xx, yy, z, [0.5])
ax.set(title="Poly kernel - degree 3")
# Degree 5
z1 = poly2d5.predict(np.c_[xx.ravel(), yy.ravel()])
z1 = z1[:].reshape(xx.shape) 
fig, ax = plt.subplots()
ax.scatter(X2[:, 0], X2[:, 1], cmap=plt.cm.binary, zorder=2)
ax.contour(xx, yy, z1, [0.5])
ax.set(title="Poly kernel - degree 5")

# Create confusion matrices 
prediction3 = poly2.predict(X2)
prediction5 = poly2d5.predict(X2)
tn3, fp3, fn3, tp3 = confusion_matrix(t2, prediction3).ravel()
tn5, fp5, fn5, tp5 = confusion_matrix(t2, prediction5).ravel()

print("Confusion Matrix Degree 3")
print("            true")
print("predicted ", tp3, fp3)
print("          ", fn3, tn3)

print("Confusion Matrix Degree 5")
print("            true")
print("predicted ", tp5, fp5)
print("          ", fn5, tn5)<jupyter_output>Degree 3 score: 0.815
Degree 5 score: 0.88
Confusion Matrix Degree 3
            true
predicted  96 33
           4 67
Confusion Matrix Degree 5
            true
predicted  99 23
           1 77<jupyter_text>What differences do you observe between the two? The biggest difference is that the 5th degree kernel is alble to reduce the number of false positives in its classification. This is also visible in the upper right part of the plot where the 3rd degree model misclassifies a number of points that belong to the outer class, but the 5th degree model correctly classifies them. --- Experiment 6We will repeat experiments 1 - 3 again, but with dataset 3: - fit and score models using linear, poly, rbf, and sigmoid kernels - plot each decision boundary<jupyter_code># Import dataset 3 
df = pd.read_csv("dataset3.csv")
X3 = df[['x1', 'x2']].to_numpy() # feature vector 
t3 = np.ravel(df[['target']].to_numpy()) # target vector (or lables)

# Fit and score data for linear kernel 
linear3 = svm.SVC(kernel='linear', gamma='scale').fit(X3, t3) 
linear3_score = linear3.score(X3, t3)

# Fit and score data for poly kernel 
poly3 = svm.SVC(kernel='poly', gamma='scale').fit(X3, t3) 
poly3_score = poly3.score(X3, t3)

# Fit and score data for rbf kernel
rbf3 = svm.SVC(kernel='rbf', gamma='scale').fit(X3, t3) 
rbf3_score = rbf3.score(X3, t3)

# Fit and score data for sigmoid kernel
sigmoid3 = svm.SVC(kernel='sigmoid', gamma='scale').fit(X3, t3) 
sigmoid3_score = sigmoid3.score(X3, t3)

# Plot decision boundaries models trained on dataset 3
xx, yy = np.meshgrid(np.linspace(0,40), np.linspace(0,40))
# Linear kernel 
z = linear3.predict(np.c_[xx.ravel(), yy.ravel()])
z = z[:].reshape(xx.shape) 
fig, ax = plt.subplots()
ax.scatter(X3[:, 0], X3[:, 1], cmap=plt.cm.binary, zorder=2)
ax.contour(xx, yy, z, [0.5])
ax.set(title="Linear kernel - dataset 3")

# Poly kernel 
z = poly3.predict(np.c_[xx.ravel(), yy.ravel()])
z = z[:].reshape(xx.shape) 
fig, ax = plt.subplots()
ax.scatter(X3[:, 0], X3[:, 1], cmap=plt.cm.binary, zorder=2)
ax.contour(xx, yy, z, [0.5])
ax.set(title="Poly kernel - dataset 3")

# Rbf kernel 
z = rbf3.predict(np.c_[xx.ravel(), yy.ravel()])
z = z[:].reshape(xx.shape) 
fig, ax = plt.subplots()
ax.scatter(X3[:, 0], X3[:, 1], cmap=plt.cm.binary, zorder=2)
ax.contour(xx, yy, z, [0.5])
ax.set(title="RBF kernel - dataset 3")

# Sigmoid kernel 
z = sigmoid3.predict(np.c_[xx.ravel(), yy.ravel()])
z = z[:].reshape(xx.shape) 
fig, ax = plt.subplots()
ax.scatter(X3[:, 0], X3[:, 1], cmap=plt.cm.binary, zorder=2)
ax.contour(xx, yy, z, [0.5])
ax.set(title="Sigmoid kernel - dataset 3")

print("Linear kernel score: ", linear3_score)
print("Poly kernel score:   ", poly3_score)
print("RBF kernel score:    ", rbf3_score)
print("sigmoid kernel score:", sigmoid3_score)<jupyter_output>Linear kernel score:  0.88
Poly kernel score:    0.915
RBF kernel score:     1.0
sigmoid kernel score: 0.62<jupyter_text>Which kernel produces the best decision bounday? Does it seem as if the *poly* kernel should have scored higher?Once again, the *rbf* kernel produced the best decison boundary. It seems like the *poly* kernel should have scored higher since the data seems to be shaped in a way that a polynomial function can describe it. We might need to make changes to some parameters in the model to find a better fit.   Experiment 7 We will use grid-search with cross-validation to determint which combination of hyperparameters produces the best fit for the *poly* kernel on dataset 3. We will try the following: - regularization parameter C values of `0.001`, `0.01`, `0.1`, and `1.0`- polynomial degrees `3`, `5`, and `7`- independent coef0 terms `0.01`, `0.1`, and `1.0`Then we will plot the decision boundary and create a confusion matrix to see the changes.<jupyter_code># Use grid search to find optimal hyperparameters 
params = [{'C': [0.001, 0.01, 0.1, 1.0], 'degree': [3, 5, 7], 'coef0': [0.01, 0.1, 1.0],
         'gamma': ['scale'], 'kernel': ['poly']}]
svc = svm.SVC()
clf = GridSearchCV(svc, params, cv=5)
clf.fit(X3, t3)
poly_best = svm.SVC(C=0.1, kernel='poly', degree=5, gamma='scale', coef0=1.0).fit(X3, t3)

# Plot decision boundary 
xx, yy = np.meshgrid(np.linspace(0,40), np.linspace(0,40)) 
z = poly_best.predict(np.c_[xx.ravel(), yy.ravel()])
z = z[:].reshape(xx.shape) 
fig, ax = plt.subplots()
ax.scatter(X3[:, 0], X3[:, 1], cmap=plt.cm.binary, zorder=2)
ax.contour(xx, yy, z, [0.5])
ax.set(title="Optimal poly kernel - dataset 3")

# Create a confusion matrix
predict = poly_best.predict(X3)
tn, fp, fn, tp = confusion_matrix(t3, predict).ravel()

# Display results
print("Best parameters:")
print("     C:", clf.best_params_['C'])
print("Degree:", clf.best_params_['degree'])
print(" Coef0:", clf.best_params_['coef0'])

print("Confusion Matrix:")
print("            true")
print("predicted ", tp, fp)
print("          ", fn, tn)<jupyter_output>Best parameters:
     C: 0.1
Degree: 5
 Coef0: 1.0
Confusion Matrix:
            true
predicted  100 0
           0 100
{'fraction_non_alphanumeric': 0.15600698301856847, 'fraction_numerical': 0.03991429931756864, 'mean_word_length': 4.616310160427807, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 27, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '607965', 'license': 'MIT', 'n_tokens_mistral': 4962, 'n_tokens_neox': 4770, 'n_words': 1406, 'path': 'classifiers_2/support-vector-machines.ipynb', 'repo_name': 'EvilErick16/ML-intro', 'size': '12608'}