<jupyter_start><jupyter_text>Some simple non-linear regression modelsHere I will explore some basic extensions of the linear regression to perform non-linear regressions.<jupyter_code>from math import *
import numpy as np
from sklearn.linear_model import LinearRegression
import matplotlib.pyplot as plt
%matplotlib inline<jupyter_output><empty_output><jupyter_text>DataI will use essentially fake data to illustrate the different methods. 1-D relationshipLet us start with the following relationship :$$ y = 2x+\frac{1.5x^3}{e^{x/2}}+5sin(\frac{\pi x}{2})+3sin(\pi x) +\varepsilon $$where $\varepsilon \sim \cal N (0,1)$ is a random noise.<jupyter_code>nsamples=500
x_train = np.linspace(0,10,nsamples)
y_true = 2*x_train-1.5*x_train**3/np.exp(x_train/2)+5*np.sin(pi*x_train/2)+3*np.sin(pi*x_train)
y_train = y_true+np.random.normal(scale = 1.5,size = len(y_true))
plt.plot(x_train,y_train,'o',x_train,y_true,'r-')<jupyter_output><empty_output><jupyter_text>Now we need to create a test set, and a metric to evaluate the performance of the regression model.<jupyter_code>x_test = np.sort(np.random.uniform(0,10,100))
y_test = 2*x_test-1.5*x_test**3/np.exp(x_test/2)+5*np.sin(pi*x_test/2)+3*np.sin(pi*x_test)
def rmse(y_pred,y=y_test):
    return sum((y_pred-y)**2)/len(y_pred)<jupyter_output><empty_output><jupyter_text>Polynomial regression Let us start with a simple polynomial regression of the form :$$ y = \sum_{i=1}^p \alpha_i x^i $$The maximum degree of the polynomials is a hyperparameter that need to be chosen according to the performance on the test set.<jupyter_code>from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline
degree = np.arange(4,22,2)

perf = []
pred = np.ndarray((len(x_test),len(degree)))
for i,d in enumerate(degree): 
    model = make_pipeline(PolynomialFeatures(degree=d),LinearRegression())
    #note : the reshape operation is due to a deprecation in sklearn : 1d array are no longer accepted as inputs.
    model.fit(x_train.reshape(-1,1),y_train)
    pred[:,i] = model.predict(x_test.reshape(-1,1))
    perf.append(rmse(pred[:,i]))

plt.plot(degree,perf)<jupyter_output><empty_output><jupyter_text>Best performance is obtained with a polynomial of order 14, which seems to be a lot. After $p=18$, the model overfits quickly.<jupyter_code>plt.plot(x_test,pred[:,5],'o',x_test,y_test,'-r')<jupyter_output><empty_output><jupyter_text>Not too bad.<jupyter_code>plt.plot(x_test,pred[:,len(degree)-1],'o',x_test,y_test,'-r')<jupyter_output><empty_output><jupyter_text>This one is unable to reproduce the bahavior at small x.  Local regression (LOESS)  Boosting  Splines  Boosted splines<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13106256941873382, 'fraction_numerical': 0.021473528322843393, 'mean_word_length': 6.854651162790698, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '222020', 'license': 'MIT', 'n_tokens_mistral': 1009, 'n_tokens_neox': 976, 'n_words': 222, 'path': '_upcoming_posts/2017-07-18.ipynb', 'repo_name': 'mkinet/mkinet.github.io', 'size': '2701'}