<jupyter_start><jupyter_text>Attribution : Most of the code taken from https://www.kaggle.com/nirajvermafcb/support-vector-machine-detail-analysis/notebook Practise exercise to run **Support Vector machine** with different **kernels(linear,gaussian,polynomial)**. Also, tune the various parameters such as **C** ,**gamma** and **degree** to find out the best performing model .  Importing all the necessary libraries<jupyter_code>import pandas as pd
import numpy as np
import seaborn as sns
from sklearn import datasets

import matplotlib.pyplot as plt

%matplotlib inline<jupyter_output><empty_output><jupyter_text>Download dataset from iris dataset and creating a dataframe<jupyter_code>from sklearn.datasets import load_iris
iris = load_iris()
df = pd.DataFrame(iris.data,columns=iris.feature_names)
print(df.columns)<jupyter_output>Index(['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)',
       'petal width (cm)'],
      dtype='object')<jupyter_text>Adding the Target and the Label to the dataset and printing the values of labels<jupyter_code>df['Target'] = iris.target
df['Label'] = df.Target.replace(dict(enumerate(iris.target_names)))
Label_list = df['Label'].to_list()
print(Label_list)
print(df.info(verbose=True))
df.head()<jupyter_output><empty_output><jupyter_text>Checking the correlation between each feature<jupyter_code>df.corr()<jupyter_output><empty_output><jupyter_text>Checking for null values<jupyter_code>df.isnull().sum()
df.describe()<jupyter_output><empty_output><jupyter_text>Checking the count of Labels - Setosa, virginica and versicolor<jupyter_code>print("Total number of labels: {}".format(df.shape[0]))
print("Number of setosa: {}".format(df[df.Label == 'setosa'].shape[0]))
print("Number of versicolor: {}".format(df[df.Label == 'versicolor'].shape[0]))
print("Number of virginica: {}".format(df[df.Label == 'virginica'].shape[0]))
df=df.drop_duplicates()
print("Total number of labels: {}".format(df.shape[0]))
print("Number of setosa: {}".format(df[df.Label == 'setosa'].shape[0]))
print("Number of versicolor: {}".format(df[df.Label == 'versicolor'].shape[0]))
print("Number of virginica: {}".format(df[df.Label == 'virginica'].shape[0]))<jupyter_output>Total number of labels: 149
Number of setosa: 50
Number of versicolor: 50
Number of virginica: 49<jupyter_text><jupyter_code>df.shape<jupyter_output><empty_output><jupyter_text>All the 3 labels are equally split.  Data Analysis using a PairPlot.Here, different graphs are plotted based on the Labels which we have used.<jupyter_code>sns.pairplot(df, hue = 'Label')<jupyter_output><empty_output><jupyter_text>From the plots we can see the following observations.1. The Sepal length and width are always smaller compared to the other 2 lables.2. The versicolor's dimensions are in the mid-range when compared to the Setosa and the virginica3. The virginica has the longest size in both petal and sepal length<jupyter_code>#df.hist(bins=50, figsize=(20,15))
#plt.show()

sns.histplot(data=df, x="sepal length (cm)", hue="Label")
sns.histplot(data=df, x="sepal width (cm)", hue="Label")
sns.histplot(data=df, x="petal length (cm)", hue="Label")
sns.histplot(data=df, x="petal length (cm)", hue="Label")<jupyter_output><empty_output><jupyter_text>Separating features and labels<jupyter_code>X=df[['petal length (cm)','petal width (cm)','sepal length (cm)','sepal width (cm)']]
X.shape[0]<jupyter_output><empty_output><jupyter_text>Converting string value to int type for labels<jupyter_code>from sklearn.preprocessing import LabelEncoder
y=df.iloc[:,-1]

# Encode label category
# Setosa -> 0
# Versicolor ->1
# Virginica -> 2

label_encoder = LabelEncoder()
y = label_encoder.fit_transform(y)
y.size<jupyter_output><empty_output><jupyter_text>Data StandardisationStandardization refers to shifting the distribution of each attribute to have a mean of zero and a standard deviation of one (unit variance). It is useful to standardize attributes for a model.<jupyter_code># Scale the data to be between -1 and 1
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
print(X_scaled)<jupyter_output><empty_output><jupyter_text>Splitting dataset into training set and testing set for better generalisation<jupyter_code>from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1)<jupyter_output><empty_output><jupyter_text>Training with SVM Running SVM with default hyperparameter.<jupyter_code>from sklearn.svm import SVC
from sklearn import metrics
svc=SVC() #Default hyperparameters
svc.fit(X_train,y_train)
y_pred=svc.predict(X_test)
print('Accuracy Score:')
print(metrics.accuracy_score(y_test,y_pred))<jupyter_output>Accuracy Score:
0.9666666666666667<jupyter_text>Default Linear kernel<jupyter_code>svc=SVC(kernel='linear')
svc.fit(X_train,y_train)
y_pred=svc.predict(X_test)
print('Accuracy Score:')
print(metrics.accuracy_score(y_test,y_pred))<jupyter_output>Accuracy Score:
1.0<jupyter_text>RBF kernel<jupyter_code>svc=SVC(kernel='rbf')
svc.fit(X_train,y_train)
y_pred=svc.predict(X_test)
print('Accuracy Score:')
print(metrics.accuracy_score(y_test,y_pred))<jupyter_output>Accuracy Score:
0.9666666666666667<jupyter_text>We can conclude from above that svm by default uses **rbf** kernel as a parameter for kernel  Root Mean Square Error<jupyter_code>from sklearn import metrics
metrics.mean_squared_error(y_test, y_pred, sample_weight=None, multioutput='uniform_average', squared=True)<jupyter_output><empty_output><jupyter_text>Cross Validation on Linear kernel<jupyter_code>from sklearn.model_selection import cross_val_score
svc_linear=SVC(kernel='linear')
scores = cross_val_score(svc_linear, X, y, cv=10, scoring='accuracy') #cv is cross validation
print(scores)<jupyter_output>[1.         0.93333333 1.         1.         0.86666667 1.
 0.93333333 1.         1.         1.        ]<jupyter_text>We can see above how the accuracy score is different everytime.This shows that accuracy score depends upon how the datasets got split.<jupyter_code>print(scores.mean())<jupyter_output>0.9733333333333334<jupyter_text>In K-fold cross validation we generally take the mean of all the scores.  CV on rbf kernel<jupyter_code>C_range=list(np.arange(0.1,6,0.1))
acc_score=[]
for c in C_range:
    svc_rbf = SVC(kernel='rbf', C=c)
    scores = cross_val_score(svc, X, y, cv=10, scoring='accuracy')
    acc_score.append(scores.mean())
print(acc_score)
print(scores.mean())
import matplotlib.pyplot as plt
%matplotlib inline

C_values=list(np.arange(0.1,6,0.1))
plt.plot(C_values,acc_score)
plt.xlabel('Value of C for SVC ')
plt.ylabel('Cross-Validated Accuracy')<jupyter_output><empty_output><jupyter_text>Tune SVM hyperparameters by using GridSearchCV with cross validation.<jupyter_code>from sklearn.model_selection import GridSearchCV
param_grid = {'C' : [0.1, 1, 10, 100, 1000], 'gamma' : [1, 0.1, 0.01, 0.001, 0.0001]}
grid_svm = GridSearchCV(svc_rbf, param_grid, cv = 7, verbose=5) 
grid_svm.fit(X_train, y_train)
grid_svm_pred = grid_svm.predict(X_test)
from sklearn.metrics import confusion_matrix
confusion_matrix(y_test, grid_svm_pred)
from sklearn.metrics import precision_recall_fscore_support
precision_recall_fscore_support(y_test, grid_svm_pred)
from sklearn import metrics
metrics.mean_squared_error(y_test, grid_svm_pred, sample_weight=None, multioutput='uniform_average', squared=True)
print(grid_svm.best_params_)
print(metrics.accuracy_score(y_test,grid_svm_pred))<jupyter_output>0.9666666666666667<jupyter_text>Accuracy score is highest for C=100 and gamma = 0.001<jupyter_code>gamma_range=[0.0001,0.001,0.01,0.1,1,10,100]
acc_score=[]
for g in gamma_range:
    svc = SVC(kernel='rbf', gamma=g)
    scores = cross_val_score(svc, X, y, cv=10, scoring='accuracy')
    acc_score.append(scores.mean())
print(acc_score)    
    
import matplotlib.pyplot as plt
%matplotlib inline

gamma_range=[0.0001,0.001,0.01,0.1,1,10,100]

# plot the value of C for SVM (x-axis) versus the cross-validated accuracy (y-axis)
plt.plot(gamma_range,acc_score)
plt.xlabel('Value of gamma for SVC ')
plt.xticks(np.arange(0.0001,100,5))
plt.ylabel('Cross-Validated Accuracy')
gamma_range=[0.0001,0.001,0.01,0.1]
acc_score=[]
for g in gamma_range:
    svc = SVC(kernel='rbf', gamma=g)
    scores = cross_val_score(svc, X, y, cv=10, scoring='accuracy')
    acc_score.append(scores.mean())
print(acc_score)    
    
import matplotlib.pyplot as plt
%matplotlib inline

gamma_range=[0.0001,0.001,0.01,0.1]

# plot the value of C for SVM (x-axis) versus the cross-validated accuracy (y-axis)
plt.plot(gamma_range,acc_score)
plt.xlabel('Value of gamma for SVC ')
plt.ylabel('Cross-Validated Accuracy')<jupyter_output><empty_output><jupyter_text>The score increases steadily and raches its peak at 0.01 and then decreases till gamma=1.Thus Gamma should be around 0.01. Let us look into more detail for gamma values<jupyter_code>gamma_range=[0.01,0.02,0.03,0.04,0.05]
acc_score=[]
for g in gamma_range:
    svc = SVC(kernel='rbf', gamma=g)
    scores = cross_val_score(svc, X, y, cv=10, scoring='accuracy')
    acc_score.append(scores.mean())
print(acc_score)    
    
import matplotlib.pyplot as plt
%matplotlib inline

gamma_range=[0.01,0.02,0.03,0.04,0.05]

# plot the value of C for SVM (x-axis) versus the cross-validated accuracy (y-axis)
plt.plot(gamma_range,acc_score)
plt.xlabel('Value of gamma for SVC ')
plt.ylabel('Cross-Validated Accuracy')
degree=[2,3,4,5,6]
acc_score=[]
for d in degree:
    svc = SVC(kernel='poly', degree=d)
    scores = cross_val_score(svc, X, y, cv=10, scoring='accuracy')
    acc_score.append(scores.mean())
print(acc_score)    
    
import matplotlib.pyplot as plt
%matplotlib inline

degree=[2,3,4,5,6]

# plot the value of C for SVM (x-axis) versus the cross-validated accuracy (y-axis)
plt.plot(degree,acc_score,color='r')
plt.xlabel('degrees for SVC ')
plt.ylabel('Cross-Validated Accuracy')
from sklearn.neighbors import KNeighborsClassifier
knn = KNeighborsClassifier(n_neighbors = 9)
knn.fit(X_train,y_train)
print("Training Accuracy {:.2f}".format(knn.score(X_train,y_train)))
print("Testing Accuracy {:.2f}".format(knn.score(X_test,y_test)))
knn_y_pred = knn.predict(X_test)
print(metrics.accuracy_score(y_test, knn_y_pred))
from sklearn.model_selection import RandomizedSearchCV
random_search = RandomizedSearchCV(knn,dict(n_neighbors = list(range(10,20))),cv = 10,scoring = "accuracy",random_state=42)
random_search.fit(X_train,y_train)
print(random_search.best_score_)
print(random_search.best_params_)<jupyter_output>{'n_neighbors': 10}
{'fraction_non_alphanumeric': 0.12562435208745643, 'fraction_numerical': 0.03175949486382056, 'mean_word_length': 6.984951091045899, 'pattern_counts': {'":': 0, '<': 84, '<?xml version=': 0, '>': 87, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 6, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '181650', 'license': 'MIT', 'n_tokens_mistral': 4090, 'n_tokens_neox': 3803, 'n_words': 857, 'path': 'Assignments/DSCI_633_Assignment_4_SVM.ipynb', 'repo_name': 'AshiniAnantharaman/RIT-DSCI-633-FDS', 'size': '10611'}