<jupyter_start><jupyter_text>Name:- <NAME> Experiment No.:- 8 Roll No.- 34 UID:- 2019230071 Batch:- B Aim:- Understanding Support Vector Machine algorithm through building SVM algorithm in Python    Support Vector Machine  Importing the libraries<jupyter_code>import pandas as pd
import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt
import seaborn as sns
import warnings
warnings.filterwarnings('ignore')
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Linear Kernel<jupyter_code>from sklearn.metrics import confusion_matrix

# Function to draw a nice plot of an SVM
def plot_svc(svc, X, y, h=0.02, pad=0.25):
    x_min, x_max = X[:, 0].min()-pad, X[:, 0].max()+pad
    y_min, y_max = X[:, 1].min()-pad, X[:, 1].max()+pad

    xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max,h))

    Z = svc.predict(np.c_[xx.ravel(), yy.ravel()])

    Z = Z.reshape(xx.shape)

    plt.contourf(xx, yy, Z, cmap=plt.cm.Paired, alpha=0.2)

    plt.scatter(X[:,0], X[:,1], s=70, c=y, cmap=mpl.cm.Paired)

    # Support vectors indicated in plot by vertical lines

    sv = svc.support_vectors_

    plt.scatter(sv[:,0], sv[:,1], c='k', marker='x', s=100, linewidths=1) 
    plt.xlim(x_min, x_max)

    plt.ylim(y_min, y_max) 
    plt.xlabel('X1')

    plt.ylabel('X2') 
    plt.show()

    print('Number of support vectors: ', svc.support_.size)
# Generating random data: 20 observations of 2 features and divide into two
np.random.seed(5)
X = np.random.randn(20,2)
y = np.repeat([1,-1], 10)
X[y == -1] = X[y == -1] + 1
X
y
# Scatter plot between the two features, where colour indicates the target class
plt.scatter(X[:,0], X[:,1], s=70, c=y, cmap=mpl.cm.Paired)
plt.xlabel('X1')
plt.ylabel('X2')<jupyter_output><empty_output><jupyter_text>Linear or Non Linear? Not linearly separable<jupyter_code># Fitting the Support Vector Classifier Model
from sklearn.svm import SVC
svc = SVC(C=1, kernel='linear')
svc.fit(X, y)
svc.support_ # Determining the identities of support vectors
plot_svc(svc, X, y)<jupyter_output><empty_output><jupyter_text>The region of feature space that will be assigned to the -1 class is shown in light blue, and the region that will be assigned to the +1 class is shown in brown.The decision boundary between the two classes is linear (because we used the argument kernel = ”linear”).The support vectors are plotted with crosses and the remaining observations are plotted as circles; we see here that there are 13 support vectors.<jupyter_code># Using a smaller value of C argument
svc2 = SVC(C=1, kernel='linear')
svc2.fit(X, y)
plot_svc(svc2, X, y)
# Performing cross validation 10-fold
from sklearn.model_selection import GridSearchCV

# Select the optimal C parameter by cross-validation
tuned_parameters = [{'C': [0.001, 0.01, 0.1, 1, 5, 10, 100]}]
clf = GridSearchCV(
    SVC(kernel='linear'), 
    tuned_parameters, 
    cv=10, 
    scoring='accuracy'
)
clf.fit(X, y)
clf.cv_results_ # Cross validation scores and errors for each split
clf.best_params_<jupyter_output><empty_output><jupyter_text>C=0.001 is best parameter given by GridSearchCV<jupyter_code># Generating a test dataset
np.random.seed(1)
X_test = np.random.randn(20,2)
y_test = np.random.choice([-1,1], 20)
X_test[y_test == 1] = X_test[y_test == 1] -1
X_test
y_test
# Fitting SVC with best C value
svc2 = SVC(C=0.001, kernel='linear')
svc2.fit(X, y)
y_pred = svc2.predict(X_test)
pd.DataFrame(confusion_matrix(y_test, y_pred), index=svc2.classes_, columns=['-1','+1'])
pd.DataFrame(confusion_matrix(y_test, y_pred), index=['Actual -1','Actual +1'], columns=['Predicted -1','Predicted +1'])
from sklearn.metrics import classification_report
print(classification_report(y_test, y_pred))
# Making nearly linearly separable data
X_test[y_test == 1] = X_test[y_test == 1] -1
plt.scatter(X_test[:,0], X_test[:,1], s=70, c=y_test, cmap=mpl.cm.Paired)
plt.xlabel('X1')
plt.ylabel('X2')
svc3 = SVC(C=1e5, kernel='linear')
svc3.fit(X_test, y_test)
plot_svc(svc3, X_test, y_test)<jupyter_output><empty_output><jupyter_text>No training errors were made and only three support vectors were used.However, we can see from the figure that the margin is very narrow (because the observations that are not support vectors, indicated as circles, are very close to the decision boundary).It seems likely that this model will perform poorly on test data<jupyter_code># Trying a smaller value of C for this new dataset
svc4 = SVC(C=1, kernel='linear')
svc4.fit(X_test, y_test)
plot_svc(svc4, X_test, y_test)<jupyter_output><empty_output><jupyter_text>Using cost = 1, we misclassify a training observation, but we also obtain a much wider margin and make use of five support vectors.It seems likely that this model will perform better on test data than the model with cost = 1e5.  Non-linear Kernels<jupyter_code>from sklearn.model_selection import train_test_split
np.random.seed(8)
X = np.random.randn(200,2)
X[:100] = X[:100] +2
X[101:150] = X[101:150] -2
y = np.concatenate([np.repeat(-1, 150), np.repeat(1,50)])
X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.5)
plt.scatter(X[:,0], X[:,1], s=70, c=y, cmap=mpl.cm.Paired)
plt.xlabel('X1')
plt.ylabel('X2')<jupyter_output><empty_output><jupyter_text>One class is kind of stuck in the middle of another class. This suggests that we might want to use a radial kernel in our SVM.  Rbf kernel<jupyter_code># Using the SVC() function with a radial kernel and γ = 1
svm = SVC(C=1.0, kernel='rbf', gamma=1)
svm.fit(X_train, y_train)
plot_svc(svm, X_test, y_test)<jupyter_output><empty_output><jupyter_text>The plot shows that the resulting SVM has a decidedly non-linear boundary.We can see from the figure that there are a fair number of training errors in this SVM fit.If we increase the value of cost, we can reduce the number of training error<jupyter_code># Increasing C parameter, allowing more flexibility
svm2 = SVC(C=100, kernel='rbf', gamma=1.0)
svm2.fit(X_train, y_train)
plot_svc(svm2, X_test, y_test)<jupyter_output><empty_output><jupyter_text>However, this comes at the price of a more irregular decision boundary that seems to be at risk of overfitting the data.We can perform cross-validation using GridSearchCV() to select the best choice of γ and cost for an SVM with a radial kernel<jupyter_code>tuned_parameters = [{'C': [0.01, 0.1, 1, 10, 100],'gamma': [0.5, 1,2,3,4]}]
clf = GridSearchCV(SVC(kernel='rbf'), tuned_parameters, cv=10, scoring='accuracy')
clf.fit(X_train, y_train)
clf.best_params_
plot_svc(clf.best_estimator_, X_test, y_test)
y_pred = clf.best_estimator_.predict(X_test)
pd.DataFrame(confusion_matrix(y_test, y_pred), index=['Actual -1','Actual +1'], columns=['Predicted -1','Predicted +1'])
print(clf.best_estimator_.score(X_test, y_test))
from sklearn.metrics import classification_report
print(classification_report(y_test, y_pred))<jupyter_output>precision    recall  f1-score   support

          -1       0.86      0.97      0.91        74
           1       0.88      0.54      0.67        26

    accuracy                           0.86       100
   macro avg       0.87      0.76      0.79       100
weighted avg       0.86      0.86      0.85       100<jupyter_text>Polynomial kernel<jupyter_code>svm3 = SVC(C=100, kernel='poly', gamma=1.0)
svm3.fit(X_train, y_train)
plot_svc(svm2, X_test, y_test)<jupyter_output><empty_output><jupyter_text>We can perform cross-validation using GridSearchCV() to select the best choice of γ and cost for an SVM with a polynomial kernel.<jupyter_code>tuned_parameters = [{'C': [0.01, 0.1, 1],'gamma': [0.5, 1, 2]}]
clf2 = GridSearchCV(SVC(kernel='poly'), tuned_parameters, cv=10, scoring='accuracy', n_jobs=-1)
clf2.fit(X_train, y_train)
clf2.best_params_
plot_svc(clf2.best_estimator_, X_test, y_test)
y_pred = clf2.best_estimator_.predict(X_test)
pd.DataFrame(confusion_matrix(y_test, y_pred), index=['Actual -1','Actual +1'], columns=['Predicted -1','Predicted +1'])
print(clf2.best_estimator_.score(X_test, y_test))
from sklearn.metrics import classification_report
print(classification_report(y_test, y_pred))<jupyter_output>precision    recall  f1-score   support

          -1       0.74      1.00      0.85        74
           1       0.00      0.00      0.00        26

    accuracy                           0.74       100
   macro avg       0.37      0.50      0.43       100
weighted avg       0.55      0.74      0.63       100<jupyter_text>Rbf kernel performs better than polynomial kernel for the same data when non-linear SVM is used.  Working on Pulsar Dataset  Downloading the training set<jupyter_code>!gdown --id 1YSFW8DGpoU7iBB6TxnnltOFc1bkCBZC_<jupyter_output>/usr/local/lib/python3.7/dist-packages/gdown/cli.py:131: FutureWarning: Option `--id` was deprecated in version 4.3.1 and will be removed in 5.0. You don't need to pass it anymore to use a file ID.
  category=FutureWarning,
Downloading...
From: https://drive.google.com/uc?id=1YSFW8DGpoU7iBB6TxnnltOFc1bkCBZC_
To: /content/pulsar_data_test.csv
100% 542k/542k [00:00<00:00, 77.4MB/s]<jupyter_text>Downloading the testing set<jupyter_code>!gdown --id 18suswE049ki9zxJreR45kGrvW14vgJ_N<jupyter_output>/usr/local/lib/python3.7/dist-packages/gdown/cli.py:131: FutureWarning: Option `--id` was deprecated in version 4.3.1 and will be removed in 5.0. You don't need to pass it anymore to use a file ID.
  category=FutureWarning,
Downloading...
From: https://drive.google.com/uc?id=18suswE049ki9zxJreR45kGrvW14vgJ_N
To: /content/pulsar_data_train.csv
100% 1.30M/1.30M [00:00<00:00, 118MB/s]<jupyter_text>Load training and testing set<jupyter_code>train = pd.read_csv('pulsar_data_train.csv')
test = pd.read_csv('pulsar_data_test.csv')
train.head()<jupyter_output><empty_output><jupyter_text>Check for null values in dataset<jupyter_code>train.isna().sum()<jupyter_output><empty_output><jupyter_text>Show the distribution of output class<jupyter_code>train['target_class'].value_counts().plot(kind='bar')<jupyter_output><empty_output><jupyter_text>The dataset is imbalanced with class 1 as minority class  List the columns in dataset<jupyter_code>train.columns<jupyter_output><empty_output><jupyter_text>Use KNN Imputer to impute missing values<jupyter_code>from sklearn.impute import KNNImputer
 
# Imputer object using the mean strategy and
# missing_values type for imputation
imputer1 = KNNImputer(n_neighbors=2, weights="uniform")
traindata = imputer1.fit_transform(train)
train<jupyter_output><empty_output><jupyter_text>Create traindata as imputed dataset having no null values<jupyter_code>traindata = pd.DataFrame(traindata, columns = train.columns)
traindata
traindata.isna().sum()<jupyter_output><empty_output><jupyter_text>Divide the dataset into X and y<jupyter_code>X = traindata.drop('target_class', axis = 1)
y = traindata['target_class']
X
y
!pip install imbalanced-learn<jupyter_output>Requirement already satisfied: imbalanced-learn in /usr/local/lib/python3.7/dist-packages (0.8.1)
Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from imbalanced-learn) (1.1.0)
Requirement already satisfied: numpy>=1.13.3 in /usr/local/lib/python3.7/dist-packages (from imbalanced-learn) (1.21.5)
Requirement already satisfied: scipy>=0.19.1 in /usr/local/lib/python3.7/dist-packages (from imbalanced-learn) (1.4.1)
Requirement already satisfied: scikit-learn>=0.24 in /usr/local/lib/python3.7/dist-packages (from imbalanced-learn) (1.0.2)
Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn>=0.24->imbalanced-learn) (3.1.0)<jupyter_text>Import SMOTE to make the dataset balanced<jupyter_code>from imblearn.over_sampling import SMOTE<jupyter_output><empty_output><jupyter_text>Oversampling the minority class<jupyter_code>sm = SMOTE(random_state=42)
X_sampled, y_sampled = sm.fit_resample(X,y)
X_sampled
y_sampled_pd = pd.DataFrame(data=y_sampled)<jupyter_output><empty_output><jupyter_text>Balanced dataset according to target_class<jupyter_code>y_sampled_pd['target_class'].value_counts().plot(kind='pie')<jupyter_output><empty_output><jupyter_text>Split the dataset into 80% training set and 20% test set<jupyter_code>from sklearn.model_selection import train_test_split
X_train,X_test,y_train,y_test = train_test_split(X_sampled,y_sampled_pd, test_size=0.20 , random_state=42)<jupyter_output><empty_output><jupyter_text>Use StandardScaler to apply transformation to training and test set<jupyter_code>from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train_scaled = sc.fit_transform(X_train)
X_test_scaled = sc.fit_transform(X_test)<jupyter_output><empty_output><jupyter_text>Apply SVM with rbf kernel and note the accuracy<jupyter_code>from sklearn.metrics import accuracy_score
svm_rbf = SVC(kernel='rbf')
svm_rbf.fit(X_train_scaled, y_train)
y_pred_rbf = svm_rbf.predict(X_test_scaled)
print('Support Vector Machine RBF kernel accuracy= ', accuracy_score(y_test, y_pred_rbf))<jupyter_output>Support Vector Machine RBF kernel accuracy=  0.9437362637362637<jupyter_text>Print the classification report<jupyter_code>from sklearn.metrics import classification_report
print(classification_report(y_test, y_pred_rbf))<jupyter_output>precision    recall  f1-score   support

         0.0       0.92      0.97      0.95      2305
         1.0       0.97      0.91      0.94      2245

    accuracy                           0.94      4550
   macro avg       0.95      0.94      0.94      4550
weighted avg       0.95      0.94      0.94      4550<jupyter_text>Apply SVM with linear kernel and note the accuracy<jupyter_code>svm_linear= SVC(kernel='linear')
svm_linear.fit(X_train_scaled, y_train)
y_pred_linear = svm_linear.predict(X_test_scaled)
print('Support Vector Machine Linear kernel accuracy= ', accuracy_score(y_test, y_pred_linear))<jupyter_output>Support Vector Machine Linear kernel accuracy=  0.9406593406593406<jupyter_text>Print the classification report<jupyter_code>print(classification_report(y_test, y_pred_linear))
svm_poly= SVC(kernel='poly')
svm_poly.fit(X_train_scaled, y_train)
y_pred_poly = svm_poly.predict(X_test_scaled)
print('Support Vector Machine Linear kernel accuracy= ', accuracy_score(y_test, y_pred_poly))<jupyter_output>Support Vector Machine Linear kernel accuracy=  0.9380219780219781
{'fraction_non_alphanumeric': 0.11358387570858702, 'fraction_numerical': 0.04366995591014067, 'mean_word_length': 5.114676936243047, 'pattern_counts': {'":': 0, '<': 115, '<?xml version=': 0, '>': 120, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 16, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '30', 'dir': 'jupyter-structured-clean-dedup', 'id': '62882', 'license': 'MIT', 'n_tokens_mistral': 5525, 'n_tokens_neox': 5168, 'n_words': 1280, 'path': 'exp8/EXP8_Support_Vector_Machine.ipynb', 'repo_name': 'Parshwa52/Data_Science_Exps', 'size': '14295'}