<jupyter_start><jupyter_text>Simple Feature Maps and KernelsIn this tutorial, we explain how to use the proposed feature maps and their associated kernels with SVM. We start with importing `numpy`, which is the only requirement for the implementation.<jupyter_code>import numpy as np
from numpy import linalg<jupyter_output><empty_output><jupyter_text>Our first feature map,  $\phi_{p,1}(\mathbf{x} ~|~ \mathbf{a})$ maps the $d$-dimensional input data to $d+1$-dimensional feature space with a predefined anchor point $\mathbf{a}$.<jupyter_code>def phi_p_1(X, anchor, p=1):
    return np.hstack((X, (linalg.norm(X-anchor, axis=1, ord=p) ** p).reshape((len(X), 1))))<jupyter_output><empty_output><jupyter_text>Here is the associated kernel:<jupyter_code>def k_p_1(X1,X2,anchor,p=1):
     return np.dot(X1, X2.T) + np.outer(linalg.norm(X1 - anchor, ord=p, axis=1)**p, linalg.norm(X2 - anchor, ord=p, axis=1)**p)<jupyter_output><empty_output><jupyter_text>Next, we import the linear SVM and the breast cancer wisconsin dataset (binary classification) from `sklearn`.<jupyter_code>from sklearn.svm import LinearSVC
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y,test_size=0.3, random_state=42, stratify= y, shuffle=True)<jupyter_output><empty_output><jupyter_text>For this example, we choose the anchor point as the mean of the traing dataset and use the linear SVM after explicit mapping.<jupyter_code>a=np.mean(X_train,axis=0)
clf = LinearSVC(C=1, dual=False)
clf = clf.fit(phi_p_1(X_train, anchor= a, p=1), y_train)
predictions=clf.predict(phi_p_1(X_test,anchor=a,p=1))
print("Test Accuracy: ", round(accuracy_score(y_test,predictions),3))<jupyter_output>Test Accuracy:  0.947<jupyter_text>When we apply scaling, the mean of the training data, and hence, the anchor point becomes the zero vector. The following code block is equivalent to the one above.<jupyter_code>from sklearn.preprocessing import StandardScaler

def temp_phi_p_1(X, p=1):
    return np.hstack((X, (linalg.norm(X, axis=1, ord=p) ** p).reshape((len(X), 1))))

scaler=StandardScaler(with_std=False)
X_train_scaled=scaler.fit_transform(X_train)
X_test_scaled=scaler.transform(X_test)


clf = LinearSVC(C=1, dual=False)
clf = clf.fit(temp_phi_p_1(X_train_scaled, p=1), y_train)
predictions=clf.predict(temp_phi_p_1(X_test_scaled,p=1))
print("Test Accuracy: ", round(accuracy_score(y_test,predictions),3))<jupyter_output>Test Accuracy:  0.965<jupyter_text>To use kernels with parameters, `sklearn` requires a wrapper class.<jupyter_code>class kernel():
    def __init__(self,p=1):
        self.p=p
        
    def k_p_1(self,X1,X2):
         return np.dot(X1, X2.T) + np.outer(linalg.norm(X1, ord=self.p, axis=1), linalg.norm(X2, ord=self.p, axis=1))<jupyter_output><empty_output><jupyter_text>Then, we use the given kernel with the scaled dataset.<jupyter_code>from sklearn.svm import SVC
my_kernel=kernel(p=1)
clf=SVC(kernel=my_kernel.k_p_1, C=1, random_state=42)
clf=clf.fit(X_train_scaled,y_train)
predictions=clf.predict(X_test_scaled)
print("Test Accuracy: ", round(accuracy_score(y_test,predictions),3))<jupyter_output>Test Accuracy:  0.959<jupyter_text>Our second feature map $\phi_{p,d}(\mathbf{x} ~|~ \mathbf{a})$ maps the $d$-dimensional input data to $2d$-dimensional feature space with a predefined anchor point $\mathbf{a}$.<jupyter_code>def phi_p_d(X, anchor, p=1):
        return np.hstack((X, np.abs(X-anchor)**p))<jupyter_output><empty_output><jupyter_text>The associated kernel then becomes:<jupyter_code>def k_p_d(X1,X2,anchor,p=1):
    return np.dot(X1, X2.T) + np.dot(np.abs(X1 - anchor)**p, np.abs(X2 - anchor).T**p)<jupyter_output><empty_output><jupyter_text>As in the previous case, we use $\phi_{p,d}(\mathbf{x}_i ~|~ \mathbf{a})$ and  its associated kernel on the same classification problem.<jupyter_code>def temp_phi_p_d(X, p=1):
    return np.hstack((X, (linalg.norm(X, axis=1, ord=p) ** p).reshape((len(X), 1))))

clf = LinearSVC(C=1, dual=False)
clf = clf.fit(temp_phi_p_d(X_train_scaled, p=1), y_train)
predictions=clf.predict(temp_phi_p_d(X_test_scaled,p=1))
print("Test Accuracy: ", round(accuracy_score(y_test,predictions),3))


class kernel():
    def __init__(self,p=1):
    
        self.p=p
        
    def k_p_d(self,X1,X2):
          return np.dot(X1, X2.T) + np.dot(np.abs(X1)**self.p, np.abs(X2).T**self.p)
        
        
my_kernel=kernel(p=1)
clf=SVC(kernel=my_kernel.k_p_d, C=1, random_state=42)
clf = clf.fit(X_train_scaled, y_train)
predictions=clf.predict(X_test_scaled)
print("Test Accuracy: ", round(accuracy_score(y_test,predictions),3))<jupyter_output>Test Accuracy:  0.965
Test Accuracy:  0.942<jupyter_text>So far the introduced feature maps use single anchor point for all input data. The following feature maps and kernels utilize multiple anchor points and uses different anchors for different inputs. Our first feature map that utilizes multiple anchor point is $\phi_{p,1}(\mathbf{x}_i ~|~ \mathbf{a_i})$, where $A$ is a set of anchor points and $\mathbf{a}_i= arg min_{\mathbf{a} \in A}\{\|\mathbf{x}_i-\mathbf{a}\|_p\}$.<jupyter_code>def min_phi_p_1(X, anchors=[], p=1):
    temp=[]
    for a in anchors:
        temp.append(linalg.norm(X-a,axis=1,ord=p)**p)
    return np.hstack((X,np.min(temp,axis=0).reshape((len(X), 1))))<jupyter_output><empty_output><jupyter_text>For this example, we choose two anchor points that are the sample averages for each class.<jupyter_code>#get unique labels
y_unique=np.unique(y_train)
anchor_list=[]
anchor_list.append(np.mean(X_train[y_train==y_unique[0]],axis=0))
anchor_list.append(np.mean(X_train[y_train==y_unique[1]],axis=0))

clf = LinearSVC(C=1, dual=False)
clf=clf.fit(min_phi_p_1(X_train, anchors= anchor_list, p=1), y_train)
predictions=clf.predict(min_phi_p_1(X_test, anchors= anchor_list, p=1))
print("Test Accuracy: ", round(accuracy_score(y_test,predictions),3))<jupyter_output>Test Accuracy:  0.947<jupyter_text>When multiple anchor points are introduced, the best practice is using the explicit feature maps. It is possible solve the dual SVM formulation by setting `dual = True`. This is suggessted for the datasets that contain more features than the training samples.<jupyter_code>#suppress the convergence warning for LinearSVC
import warnings
warnings.filterwarnings("ignore")
clf = LinearSVC(C=1, dual=True, random_state=42).fit(min_phi_p_1(X_train, anchors= anchor_list, p=1), y_train)
print("Test Accuracy: ", round(accuracy_score(y_test,clf.predict(min_phi_p_1(X_test,anchors=anchor_list))),3))<jupyter_output>Test Accuracy:  0.778<jupyter_text>We also introduce the feature map $\phi_{p,2}( \mathbf{x}_i ~|~ \mathbf{a}_i^{(1)},\mathbf{a}_i^{(2)})$ where, $A_1,A_2$ are two sets of anchor points and $\mathbf{a}_i^{(1)}= arg min_{\mathbf{a} \in A_1}\{\|\mathbf{x}_i-\mathbf{a}\|_p\}$, $\mathbf{a}_i^{(2)}= arg min_{\mathbf{a} \in A_2}\{\|\mathbf{x}_i-\mathbf{a}\|_p\}$. The feature space for this map is $d+2$-dimensional.<jupyter_code>def map_min_p_2(X,anchors=[],p=1):
    temp = []
    #begin: argmin 
    for a in anchors[0]:
        temp.append(linalg.norm(X-a,axis=1,ord=p)**p)
    temp=np.array(temp)
    mins1=np.min(temp,axis=0)
    mins1 = mins1 / np.std(mins1)
    temp = []
    for a in anchors[1]:
        temp.append(linalg.norm(X - a, axis=1, ord=p)**p)
    temp = np.array(temp)
    mins2 = np.min(temp, axis=0)
    mins2 = mins2/ np.std(mins2)
    nz = [mins1 != 0, mins2 != 0]
    d = np.all(nz, axis=0)
    mins1[mins1==0]=np.mean( mins1[d])
    mins2[mins2 == 0] = np.mean(mins2[d])
    #end: argmin
    X1=np.hstack((X, mins1.reshape((len(X), 1) )))
    return   np.hstack((X1,mins2.reshape((len(X),1))))<jupyter_output><empty_output><jupyter_text>For this example, again we use two anchors as the means of the points that belong to different classes.<jupyter_code>anchor_list=[[],[]]
anchor_list[0].append(np.mean(X_train[y_train==y_unique[0]],axis=0))
anchor_list[1].append(np.mean(X_train[y_train==y_unique[1]],axis=0))

clf = LinearSVC(C=1, dual=False)
clf=clf.fit(map_min_p_2(X_train, anchors= anchor_list, p=1), y_train)
predictions=clf.predict(map_min_p_2(X_test,anchors=anchor_list,p=1))
print("Test Accuracy: ", round(accuracy_score(y_test,predictions),3))<jupyter_output>Test Accuracy:  0.947<jupyter_text>Our last feature map is for the multi-class classification problems. We map $d$ dimensional input data to $d+M$-dimensional feature space, where $M$ is the number of classes in a given dataset.The feature map is, $\phi_{p,M}(\mathbf{x}_i~|~\mathbf{a}_i^{(1)}, \dots,\mathbf{a}_i^{(M)})$ where ,$A_1,\dots A_M$ are $M$ number of anchor sets and  $\mathbf{a}_i^{(j)}= arg min_{\mathbf{a} \in A_j}\{\|\mathbf{x}_i-\mathbf{a}\|_p\}$, $j=1,\dots, M.$<jupyter_code>def map_min_M_1(X, anchors,p=1):
    X1=np.copy(X)
    for anchors in anchors:
        temp = []
        for a in anchors:
            temp.append(linalg.norm(X - a, axis=1, ord=p)**p)
        temp = np.array(temp)
        X1 = np.hstack((X1, (np.min(temp, axis=0).reshape((len(X), 1)))))
    return X1<jupyter_output><empty_output><jupyter_text>For this feature map, we import `Iris` dataset from  `sklearn` and choose the anchor points as the means of the points that belong to three classes ($M=3)$.<jupyter_code>from sklearn.datasets import load_iris

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y,test_size=0.3, random_state=42, stratify= y, shuffle=True)

anchor_list=[[],[],[]]
y_unique=np.unique(y_train)
anchor_list[0].append(np.mean(X_train[y_train==y_unique[0]],axis=0))
anchor_list[1].append(np.mean(X_train[y_train==y_unique[1]],axis=0))
anchor_list[2].append(np.mean(X_train[y_train==y_unique[2]],axis=0))
clf = LinearSVC(C=1, dual=False)
clf = clf.fit(map_min_M_1(X_train, anchors= anchor_list, p=1), y_train)
predictions= clf.predict(map_min_M_1(X_test,anchors=anchor_list,p=1))
print("Test Accuracy: ", round(accuracy_score(y_test,predictions),3))<jupyter_output>Test Accuracy:  0.933
{'fraction_non_alphanumeric': 0.15465850769838135, 'fraction_numerical': 0.022601658112909595, 'mean_word_length': 6.166195190947666, 'pattern_counts': {'":': 0, '<': 65, '<?xml version=': 0, '>': 65, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '290313', 'license': 'MIT', 'n_tokens_mistral': 4064, 'n_tokens_neox': 3950, 'n_words': 802, 'path': 'Kernels.ipynb', 'repo_name': 'sibirbil/SimpleKernels', 'size': '10132'}