<jupyter_start><jupyter_text>[mlcourse.ai](https://mlcourse.ai) – Open Machine Learning Course Author: [<NAME>](https://yorko.github.io). Translated by <NAME>. This material is subject to the terms and conditions of the [Creative Commons CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/) license. Free use is permitted for any non-commercial purpose.   Topic 6. Regression Lasso and Ridge Regressions*Lecture syllabus differs this week from the article outline, because topic 4 (linear models) is too huge and important, so we cover regression this week.*<jupyter_code>import numpy as np
import pandas as pd
from matplotlib import pyplot as plt

%config InlineBackend.figure_format = 'retina'
import seaborn as sns

sns.set()  # just to use the seaborn theme


from sklearn.datasets import load_boston
from sklearn.linear_model import Lasso, LassoCV, Ridge, RidgeCV
from sklearn.model_selection import KFold, cross_val_score<jupyter_output><empty_output><jupyter_text>**We will work with Boston house prices data (UCI repository).****Download the data.**<jupyter_code>boston = load_boston()
X, y = boston["data"], boston["target"]<jupyter_output><empty_output><jupyter_text>**Let's read description of data:**<jupyter_code>print(boston.DESCR)
boston.feature_names<jupyter_output><empty_output><jupyter_text>**Let's look at the first two records.**<jupyter_code>X[:2]<jupyter_output><empty_output><jupyter_text>Lasso Regression Lasso regression minimizes mean squared error with L1 regularization:$$\Large error(X, y, w) = \frac{1}{2} \sum_{i=1}^\ell {(y_i - w^Tx_i)}^2 + \alpha \sum_{i=1}^d |w_i|$$where $y = w^Tx$ hyperplane equation depending on model parameters $w$, $\ell$ is number of observations in data $X$, $d$ is number of features, $y$ target values, $\alpha$ regularization coefficient. **Let's fit Lasso Regression with the small $\alpha$ coefficient (weak regularization). Coefficient related to NOX feature (nitric oxides concentration) will be zero. It means that this feature is the least important for median house prices prediction in this region.**<jupyter_code>lasso = Lasso(alpha=0.1)
lasso.fit(X, y)
lasso.coef_<jupyter_output><empty_output><jupyter_text>**Let's train Lasso Regression with $\alpha=10$. All of the coefficients are equal to zero except features ZN (proportion of residential land zoned for lots over 25,000 sq.ft.), TAX (full-value property-tax rate), B (proportion of blacks by town) and LSTAT (% of lower status of the population).**<jupyter_code>lasso = Lasso(alpha=10)
lasso.fit(X, y)
lasso.coef_<jupyter_output><empty_output><jupyter_text>**It means that Lasso Regression may serve as a feature selection method.**<jupyter_code>n_alphas = 200
alphas = np.linspace(0.1, 10, n_alphas)
model = Lasso()

coefs = []
for a in alphas:
    model.set_params(alpha=a)
    model.fit(X, y)
    coefs.append(model.coef_)

plt.rcParams["figure.figsize"] = (12, 8)

ax = plt.gca()
# ax.set_color_cycle(['b', 'r', 'g', 'c', 'k', 'y', 'm'])

ax.plot(alphas, coefs)
ax.set_xscale("log")
ax.set_xlim(ax.get_xlim()[::-1])  # reverse axis
plt.xlabel("alpha")
plt.ylabel("weights")
plt.title("Lasso coefficients as a function of the regularization")
plt.axis("tight")
plt.show();<jupyter_output><empty_output><jupyter_text>**Now let's find the best value of $\alpha$ during cross-validation.**<jupyter_code>lasso_cv = LassoCV(alphas=alphas, cv=3, random_state=17)
lasso_cv.fit(X, y)
lasso_cv.coef_
lasso_cv.alpha_<jupyter_output><empty_output><jupyter_text>**In Scikit-learn, metrics are typically *maximized*, so for MSE there's a workaround: `neg_mean_squared_error` is minimized instead. Not really convenient.**<jupyter_code>cross_val_score(Lasso(lasso_cv.alpha_), X, y, cv=3, scoring="neg_mean_squared_error")
abs(
    cross_val_score(
        Lasso(lasso_cv.alpha_), X, y, cv=3, scoring="neg_mean_squared_error"
    ).mean()
)
abs(np.mean(cross_val_score(Lasso(9.95), X, y, cv=3, scoring="neg_mean_squared_error")))<jupyter_output><empty_output><jupyter_text>**One more ambiguous point: LassoCV sorts values of the parameters in decreasing order to ease optimization. It may seem that $\alpha$ optimization works incorrectly.**<jupyter_code>lasso_cv.alphas[:10]
lasso_cv.alphas_[:10]
plt.plot(lasso_cv.alphas, lasso_cv.mse_path_.mean(1))  # incorrect
plt.axvline(lasso_cv.alpha_, c="g");
plt.plot(lasso_cv.alphas_, lasso_cv.mse_path_.mean(1))  # correct
plt.axvline(lasso_cv.alpha_, c="g");<jupyter_output><empty_output><jupyter_text>Ridge Regression Ridge regression minimizes mean squared error with L2 regularization:$$\Large error(X, y, w) = \frac{1}{2} \sum_{i=1}^\ell {(y_i - w^Tx_i)}^2 + \alpha \sum_{i=1}^d w_i^2$$where $y = w^Tx$ hyperplane equation depending on model parameters $w$, $\ell$ is number of observations in data $X$, $d$ is number of features, $y$ target values, $\alpha$ regularization coefficient. There is a special class [RidgeCV](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.RidgeCV.htmlsklearn.linear_model.RidgeCV) for Ridge regression cross-validation.<jupyter_code>n_alphas = 200
ridge_alphas = np.logspace(-2, 6, n_alphas)
ridge_cv = RidgeCV(alphas=ridge_alphas, scoring="neg_mean_squared_error", cv=3)
ridge_cv.fit(X, y)
ridge_cv.alpha_<jupyter_output><empty_output><jupyter_text>**In case of Ridge Regression neither of the parameters are reducing to zero. It can be small value but non-zero.**<jupyter_code>ridge_cv.coef_
n_alphas = 200
ridge_alphas = np.logspace(-2, 6, n_alphas)
model = Ridge()

coefs = []
for a in ridge_alphas:
    model.set_params(alpha=a)
    model.fit(X, y)
    coefs.append(model.coef_)

ax = plt.gca()
# ax.set_color_cycle(['b', 'r', 'g', 'c', 'k', 'y', 'm'])

ax.plot(ridge_alphas, coefs)
ax.set_xscale("log")
ax.set_xlim(ax.get_xlim()[::-1])  # reverse axis
plt.xlabel("alpha")
plt.ylabel("weights")
plt.title("Ridge coefficients as a function of the regularization")
plt.axis("tight")
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13566996478282745, 'fraction_numerical': 0.011571356699647829, 'mean_word_length': 6.317791411042945, 'pattern_counts': {'":': 0, '<': 51, '<?xml version=': 0, '>': 51, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '453382', 'license': 'Unlicense', 'n_tokens_mistral': 2198, 'n_tokens_neox': 2092, 'n_words': 551, 'path': 'jupyter_english/topic06_features_regression/lesson6_lasso_ridge.ipynb', 'repo_name': 'salman394/AI-ml--course', 'size': '5981'}