<jupyter_start><jupyter_text>Support Vector Regression 1. Choices of Kernels 1.1 Setup<jupyter_code>import matplotlib.pyplot as plt
import numpy as np
import sklearn
import warnings

from sklearn.svm import SVR


warnings.filterwarnings('ignore')

np.random.seed(175)
N = 150
def generate_svr(x1, y1, kernel, extend=1.0, **kwargs):
    X = x1.reshape(-1,1)
    y = y1.reshape(-1,1)

    svr = SVR(kernel=kernel, **kwargs)
    model = svr.fit(X, y)

    x2 = np.linspace(X.min(), X.max()*extend, 100)
    y2 = model.predict(x2.reshape(-1,1))
    return x2, y2.flatten()

def plot_svr(x1, y1, kernel, **kwargs):
    x2, y2 = generate_svr(x1, y1, kernel, **kwargs)
    _, ax = plt.subplots(figsize=(12,7))
    ax.scatter(x1, y1, color='purple', marker='x', alpha=0.5)
    ax.plot(x2, y2, color='red', ls='-', lw=2)
    ax.grid(True)<jupyter_output><empty_output><jupyter_text>1.2 Effect of kernel choice when modeling functions that are mostly periodicHere, the linear, poly, sigmoid, and rbf are tested against a periodic function ($y = \sin x$)<jupyter_code>noise = 0.1*np.random.normal(size=N)
max_x = 4.0*np.pi
x1 = np.linspace(0, max_x, N)
y1 = np.sin(x1) + noise

plot_svr(x1, y1, 'linear')
plot_svr(x1, y1, 'poly', degree=4)
plot_svr(x1, y1, 'sigmoid')
plot_svr(x1, y1, 'rbf', extend=1.5)<jupyter_output><empty_output><jupyter_text>In all of the cases tested, the rbf (radial basis function) has the best extrapolation properties. However, it does not perform as well as the exponential sine squared kernel for Gaussian Process Regression.  1.3 Effect of kernel choice when modeling functions that are predominantly linearFor functions that are mostly linear, support vector regression with a linear kernel extrapolates well (as expected).<jupyter_code>x2 = x1.copy()
y2 = x1 + 4*noise

plot_svr(x2, y2, 'linear', extend=2.0)<jupyter_output><empty_output><jupyter_text>This isn't the case for the polynomial kernel, or the rbf kernel.<jupyter_code>plot_svr(x2, y2, 'poly', extend=1.3)
plot_svr(x2, y2, 'rbf', extend=1.3)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13583252190847128, 'fraction_numerical': 0.04138266796494645, 'mean_word_length': 5.134328358208955, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '466085', 'license': 'MIT', 'n_tokens_mistral': 823, 'n_tokens_neox': 794, 'n_words': 213, 'path': 'machine-learning/support_vector_regression.ipynb', 'repo_name': 'nathanielng/python-snippets', 'size': '2054'}