<jupyter_start><jupyter_text>```______                             _| ___ \                           (_)| |_/ /___  __ _ _ __ ___  ___ ___ _  ___  _ __|    // _ \/ _` | '__/ _ \/ __/ __| |/ _ \| '_ \| |\ \  __/ (_| | | |  __/\__ \__ \ | (_) | | | |\_| \_\___|\__, |_|  \___||___/___/_|\___/|_| |_|            __/ |           |___/```<jupyter_code>%matplotlib inline

import numpy as np
from matplotlib import pyplot as plt
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error

from helper import get_reg_X_y, plt_decorate, Color<jupyter_output><empty_output><jupyter_text>First exampleHere is a 1D problem.<jupyter_code>X, y = get_reg_X_y(50)

plt.scatter(X.flatten(), y, color=Color.BLUE.value)
plt_decorate()<jupyter_output><empty_output><jupyter_text>Linear regression Ordinary least-squareThe `ordinary least-square` ([OLS](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html))consists in finding a good linear model$\hat{y}(x) = w^T x + b$ where $w$ is the weight vector and $b$ is the intercept(or bias), both needed to learned from the data.Given a learning matrix $X$ this is done by solving> $$\min_{w, b} ||y - (Xw + \mathbf{1} b)||$$where $y$ is the vector containing all the outputs and $\mathbf{1}$ is a vectorcontaining $n$ $1$.<jupyter_code># In sklearn, this is done with a `LinearRegression` instance.
model = LinearRegression().fit(X, y)  # Instantiate and train the model
y_pred = model.predict(X)  # get prediction from the model

rmse = mean_squared_error(y, y_pred) ** .5
print("RMSE: {:.2f}".format(rmse))

plt.scatter(X.flatten(), y, color=Color.BLUE.value)
plt.plot(X.flatten(), y_pred, label="Model prediction", color=Color.ORANGE.value)
plt_decorate(title="OLS")<jupyter_output>RMSE: 0.73<jupyter_text>Polynomial regressionThe linear model does not seem appropriate to model well the data. Could apolynomial do better? Assuming 1D inputs, we are looking for a model of theform $P(x) = \sum_{j=1}^m w_j x^j$.Notice that, although the model is not linear in $x$, it is still linear inwhat we are trying to learn: the weights. Overall, we want to train a linearmodel in a different input space.<jupyter_code>from sklearn.preprocessing import PolynomialFeatures
# Transforming the input space with polynomial features
transf = PolynomialFeatures(degree=10)
X_poly = transf.fit_transform(X)
model_poly = LinearRegression().fit(X_poly, y)
y_pred = model_poly.predict(X_poly)

rmse = mean_squared_error(y, y_pred) ** .5
print("RMSE: {:.2f}".format(rmse))

plt.scatter(X.flatten(), y)
xs = np.linspace(-10, 10, 1000)  # new points for plots
ys = model_poly.predict(transf.transform(xs.reshape(-1, 1)))
plt.plot(xs, ys, label="Model prediction", color="orange")
plt_decorate(title="Polynomial regression")<jupyter_output>RMSE: 0.30
{'fraction_non_alphanumeric': 0.13580246913580246, 'fraction_numerical': 0.010582010582010581, 'mean_word_length': 4.464354527938343, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 16, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '535448', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1028, 'n_tokens_neox': 979, 'n_words': 252, 'path': '02-regression.ipynb', 'repo_name': 'jm-begon/ml101', 'size': '2835'}