<jupyter_start><jupyter_text>8E and 8F: Finding the Probability P(Y==1|X)  8E: Implementing Decision Function of SVM RBF Kernel After we train a kernel SVM model, we will be getting support vectors and their corresponsing coefficients $\alpha_{i}$Check the documentation for better understanding of these attributes: https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.htmlAs a part of this assignment you will be implementing the ```decision_function()``` of kernel SVM, here decision_function() means based on the value return by ```decision_function()``` model will classify the data point either as positive or negativeEx 1: In logistic regression After traning the models with the optimal weights $w$ we get, we will find the value $\frac{1}{1+\exp(-(wx+b))}$, if this value comes out to be < 0.5 we will mark it as negative class, else its positive classEx 2: In Linear SVM After traning the models with the optimal weights $w$ we get, we will find the value of $sign(wx+b)$, if this value comes out to be -ve we will mark it as negative class, else its positive class.Similarly in Kernel SVM After traning the models with the coefficients $\alpha_{i}$ we get, we will find the value of $sign(\sum_{i=1}^{n}(y_{i}\alpha_{i}K(x_{i},x_{q})) + intercept)$, here $K(x_{i},x_{q})$ is the RBF kernel. If this value comes out to be -ve we will mark $x_{q}$ as negative class, else its positive class.RBF kernel is defined as: $K(x_{i},x_{q})$ = $exp(-\gamma ||x_{i} - x_{q}||^2)$For better understanding check this link: https://scikit-learn.org/stable/modules/svm.htmlsvm-mathematical-formulation  Task E > 1. Split the data into $X_{train}$(60), $X_{cv}$(20), $X_{test}$(20)> 2. Train $SVC(gamma=0.001, C=100.)$ on the ($X_{train}$, $y_{train}$)> 3. Get the decision boundry values $f_{cv}$ on the $X_{cv}$ data  i.e. ` `$f_{cv}$ ```= decision_function(```$X_{cv}$```)```  you need to implement this decision_function()<jupyter_code>import numpy as np
import pandas as pd
from sklearn.datasets import make_classification
import numpy as np
from sklearn.svm import SVC
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=5000, n_features=5, n_redundant=2,
                           n_classes=2, weights=[0.7], class_sep=0.7, random_state=15)<jupyter_output><empty_output><jupyter_text>Pseudo codeclf = SVC(gamma=0.001, C=100.)clf.fit(Xtrain, ytrain)def decision_function(Xcv, ...): use appropriate parameters &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;for a data point $x_q$ in Xcv: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;write code to implement $(\sum_{i=1}^{\text{all the support vectors}}(y_{i}\alpha_{i}K(x_{i},x_{q})) + intercept)$, here the values $y_i$, $\alpha_{i}$, and $intercept$ can be obtained from the trained model   return  the decision_function output for all the data points in the Xcv    fcv = decision_function(Xcv, ...)   based on your requirement you can pass any other parameters Note: Make sure the values you get as fcv, should be equal to outputs of clf.decision_function(Xcv)<jupyter_code>clf = SVC(gamma = 0.001, C = 100.)
X_train , X_test , y_train , y_test = train_test_split(X,y,test_size = 0.2)

X_train,X_cv,y_train,y_cv = train_test_split(X_train,y_train,test_size=0.2)
print(y_train.shape,y_cv.shape,y_test.shape)
clf.fit(X_train,y_train)
yi_alpha=clf.dual_coef_
xi=clf.support_vectors_
b = clf.intercept_
gamma = 0.001
print(yi_alpha.shape)
xi.shape
X_cv.shape
def decision_function(X_cv,gamma,b,xi,yi_alpha):
  decision = []
  for i in range(len(X_cv)):
    rbi = 0
    for j in range(len(xi)):
      k = np.exp(-gamma*(np.linalg.norm(xi[j]-X_cv[i])**2))
      rbi+=yi_alpha.T[j]*k
    rbi=rbi+b
    decision.append(float(np.around(rbi,9)))
  return decision
fcv = decision_function(X_cv,gamma,b,xi,yi_alpha)

fcv
X_cv.shape
clf.decision_function(X_cv)<jupyter_output><empty_output><jupyter_text>8F: Implementing Platt Scaling to find P(Y==1|X) Check this PDF  TASK F > 4. Apply SGD algorithm with ($f_{cv}$, $y_{cv}$) and find the weight $W$ intercept $b$ ```Note: here our data is of one dimensional so we will have a one dimensional weight vector i.e W.shape (1,)``` > Note1: Don't forget to change the values of $y_{cv}$ as mentioned in the above image. you will calculate y+, y- based on data points in train data> Note2: the Sklearn's SGD algorithm doesn't support the real valued outputs, you need to use the code that was done in the `'Logistic Regression with SGD and L2'` Assignment after modifying loss function, and use same parameters that used in that assignment.if Y[i] is 1, it will be replaced with y+ value else it will replaced with y- value> 5. For a given data point from $X_{test}$, $P(Y=1|X) = \frac{1}{1+exp(-(W*f_{test}+ b))}$ where ` `$f_{test}$ ```= decision_function(```$X_{test}$```)```, W and b will be learned as metioned in the above step __Note: in the above algorithm, the steps 2, 4 might need hyper parameter tuning, To reduce the complexity of the assignment we are excluding the hyerparameter tuning part, but intrested students can try that__ If any one wants to try other calibration algorithm istonic regression also please check these tutorials1. http://fa.bianp.net/blog/tag/scikit-learn.htmlfn:12. https://drive.google.com/open?id=1MzmA7QaP58RDzocB0RBmRiWfl7Co_VJ73. https://drive.google.com/open?id=133odBinMOIVb_rh_GQxxsyMRyW-Zts7a4. https://stat.fandom.com/wiki/Isotonic_regressionPool_Adjacent_Violators_Algorithm<jupyter_code>p = 0
n = 0
for i in range(len(y_train)):
  if y_train[i] == 1:
    p+=1
  else :
    n+=1
yp = (p+1)/(p+2)
yn = (1)/(n+2)
print(yp,yn)
print((yp+yn)/2)
#converting f_cv into np array 
for i in range(len(fcv)):
  fcv[i]=np.array([fcv[i]])<jupyter_output><empty_output><jupyter_text>Initialization of weight vector and b<jupyter_code>def initialize_weights(dim):
    ''' In this function, we will initialize our weights and bias'''
    #initialize the weights to zeros array of (1,dim) dimensions
    #you use zeros_like function to initialize zero, check this link https://docs.scipy.org/doc/numpy/reference/generated/numpy.zeros_like.html
    #initialize bias to zero
    w = np.zeros_like(dim)
    b = 0
    return w,b
dim=fcv[0]
w,b = initialize_weights(dim)
print('w =',(w))
print('b =',str(b))
print(w.shape)<jupyter_output>w = [[0.]]
b = 0
(1, 1)<jupyter_text>Sigmoid function<jupyter_code>def sigmoid(w,x,b):
    ''' In this function, we will return sigmoid of z'''
    # compute sigmoid(z) and return
    sigma=1/(1+np.exp(-(w.dot(x)+b)))
    return sigma
def new_y(data):
  y_new =[]
  for i in range(len(data)):
    if data[i]==0:
      y_new.append(yn)
    else:
      y_new.append(yp)
  return y_new<jupyter_output><empty_output><jupyter_text>log loss<jupyter_code>def logloss(y_true,y_pred):
    '''In this function, we will compute log loss '''
    n = len(y_true)
    loss = 0
    for i in range(len(y_true)):
      loss+=y_true[i]*np.log10(y_pred[i])+(1-y_true[i])*np.log10(1-y_pred[i])
    loss=(-1/n)*loss
    return loss<jupyter_output><empty_output><jupyter_text>dw<jupyter_code>def gradient_dw(x,y,w,b,alpha,N):
    '''In this function, we will compute the gardient w.r.to w '''
    dw = x*(y-sigmoid(w,x,b))-(alpha/N)*w
    return dw<jupyter_output><empty_output><jupyter_text>DB<jupyter_code>def gradient_db(x,y,w,b):
     '''In this function, we will compute gradient w.r.to b '''
     #w_db = w.reshape(-1,1)
     #print(w_db.shape,w_db.T.shape)
     #x_db = x.reshape(-1,1)
     db=y-sigmoid(w,x,b)
     return db<jupyter_output><empty_output><jupyter_text>prediction function<jupyter_code>def pred(w,b,X):
    N = len(X)
    predict = []
    for i in range(N):
      z = sigmoid(w,X[i],b)
      if z>=0.49968826918872405:
        predict.append(yp)
      else:
        predict.append(yn)
    return np.array(predict)
def train(X_train,y_train,epochs,alpha,eta0):
    ''' In this function, we will implement logistic regression'''
    #Here eta0 is learning rate
    #implement the code as follows
    # initalize the weights (call the initialize_weights(X_train[0]) function)
    # for every epoch
        # for every data point(X_train,y_train)
           #compute gradient w.r.to w (call the gradient_dw() function)
           #compute gradient w.r.to b (call the gradient_db() function)
           #update w, b
        # predict the output of x_train[for all data points in X_train] using w,b
        #compute the loss between predicted and actual values (call the loss function)
        # store all the train loss values in a list
        # predict the output of x_test[for all data points in X_test] using w,b
        #compute the loss between predicted and actual values (call the loss function)
        # store all the test loss values in a list
        # you can also compare previous loss and current loss, if loss is not updating then stop the process and return w,b
    w,b=initialize_weights(X_train[0])
    y_pred = []
    train_loss = []
    for i in range(epochs):
      for i in range(len(X_train)):
        dw = gradient_dw(X_train[i],y_train[i],w,b,alpha,N)
        db = gradient_db(X_train[i],y_train[i],w,b)
        w=w+(eta0*dw)
        b=b+(eta0*db) 
      y_pred = pred(w,b,X_train)
      train_loss.append(logloss(y_train,y_pred))
    return w,b,train_loss
    
#y_cv with updated value of 0 and 1
y_cv_new = new_y(y_cv)
alpha=0.0001
eta0=0.0001
N=len(fcv)
epochs = 75
w,b,train_loss=train(fcv,y_cv_new,epochs,alpha,eta0)
w
import matplotlib.pyplot as plt 
plt.plot(np.arange(epochs),train_loss)
plt.xlabel('epochs')
plt.ylabel('train_loss')
plt.grid()
plt.title('epochs v/s loss')
f_test = decision_function(X_test,gamma,b,xi,yi_alpha)<jupyter_output><empty_output><jupyter_text>Platt Scaling<jupyter_code>P_y = []

for i in range(len(f_test)):
  P_y.append(1/(1+np.exp(-(w*f_test[i]+b))))
P_y<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13125886884248936, 'fraction_numerical': 0.0173322521792013, 'mean_word_length': 4.55887323943662, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 48, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '440387', 'license': 'Apache-2.0', 'n_tokens_mistral': 3626, 'n_tokens_neox': 3499, 'n_words': 1017, 'path': '8E&F_LR_SVM.ipynb', 'repo_name': 'VindhyaHV/AppliedAI_Assignments', 'size': '9866'}