<jupyter_start><jupyter_text>This notebook aims to prove that when using a kernel trick the weights assigned to the features can still be computed. We will start by performing a support vector regression using a liner kernel. With this method the weights are returned by the sklearn library. On the second session we calculate the kernel using a gram matrix and use this as an input to the support vector regression. Once the model is fitted we compare the results obtained between both SVM methods.    Load Dataset<jupyter_code>data = load_boston()
X = data['data']
y = data['target']
# Get a fealing for the dataset
print('Shape of data {}'.format(X.shape))
# Split the dataset into training and test
indices = np.arange(len(X))
X_train, X_test, y_train, y_test, idx_train, idx_test = train_test_split(X, y, 
                                                                         indices, test_size=0.33, 
                                                                         random_state=42)
print('Shape of trainig data {}'.format(X_train.shape))
print('Shape of test data {}'.format(X_test.shape))<jupyter_output>Shape of trainig data (339, 13)
Shape of test data (167, 13)<jupyter_text>Create linear classifier<jupyter_code>clf = SVR(kernel='linear')
clf.fit(X_train, y_train)
# Check the performance of the regression
y_predicted = clf.predict(X_test)
r2_score(y_test, y_predicted)
# Analyse some of the attributes returned by the SVR function

# get indices of support vectors
idx_support = clf.support_
print('Shape of the support vectors {}'.format(idx_support.shape))

# Weights of the features
coef = clf.coef_
print('Shape of the weights assigned to the features {}'.format(coef.shape))
print(coef)
# get feature names for the the data
names = data['feature_names']

# Plot the weights of the different features obtained with SVR
fig, ax = plt.subplots(1,1, figsize=(10, 8))
ax.set_title('features weights')
feature_importances(np.squeeze(clf.coef_), names, ax)
ax.yaxis.set_major_locator(ticker.FixedLocator((range(len(names)))))
ax.yaxis.set_major_formatter(ticker.FixedFormatter((names)))<jupyter_output><empty_output><jupyter_text>Use SVM with pre-computed kernel<jupyter_code># Compute the linear kernel to test our seocnd approach
kernel = np.dot(X, np.transpose(X))
# Just for fun, visualise kernel
plt.imshow(kernel)
plt.colorbar()
# Compute SVR with the precomputed kernel
clf_kernel = SVR(kernel='precomputed')
clf_kernel.fit(kernel[idx_train[:, None], idx_train], y_train)
# Check the performance of the regression
y_predicted = clf_kernel.predict(kernel[idx_test[:, None], idx_train])
r2_score(y_test, y_predicted)
# Analyse some of the attributes returned by the SVR function

# get indices of support vectors
idx_support = clf_kernel.support_
print('Shape of the support vectors {}'.format(idx_support.shape))
# Calculate the weight's coefficients
coef_kernel = np.dot(clf_kernel.dual_coef_, X_train[clf_kernel.support_])
# Similarly, to the approach above calculate the featuer importance
fig, ax = plt.subplots(1,1, figsize=(10, 8))
ax.set_title('features imp')
feature_importances(np.squeeze(coef_kernel), names, ax)
ax.yaxis.set_major_locator(ticker.FixedLocator((range(len(names)))))
ax.yaxis.set_major_formatter(ticker.FixedFormatter((names)))
# Check if they computed weights are equivalent between both approaches
print('Weights Coefficients calculated with the linear kernel:')
print(coef)
print('Weights Coefficients calculated with the gram matrix kernel:')
print(coef_kernel)<jupyter_output>Weights Coefficients calculated with the linear kernel:
[[-1.54165575e-01  4.14961460e-02  3.36275807e-02  7.88815499e-01
  -1.27721012e+00  5.78799760e+00 -4.72967614e-02 -1.03866821e+00
   5.83234847e-02 -4.84806316e-03 -6.06467745e-01  1.48903230e-02
  -3.41008345e-01]]
Weights Coefficients calculated with the gram matrix kernel:
[[-1.54164971e-01  4.14952932e-02  3.36284711e-02  7.88822837e-01
  -1.27721075e+00  5.78800670e+00 -4.72977652e-02 -1.03866723e+00
   5.83270422e-02 -4.84807102e-03 -6.06471098e-01  1.48885299e-02
  -3.41007560e-01]]
{'fraction_non_alphanumeric': 0.1010521164668461, 'fraction_numerical': 0.07658429165647174, 'mean_word_length': 4.717482517482518, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '421750', 'license': 'MIT', 'n_tokens_mistral': 1525, 'n_tokens_neox': 1347, 'n_words': 390, 'path': 'notebooks/Boston_feature_importance.ipynb', 'repo_name': 'MathieuVReine/Brain-age-prediction', 'size': '4087'}