<jupyter_start><jupyter_text>Why regularize?Regression optimizes the loss function to choose a coefficient $a_i$. If the coefficients are too large, we risk overfitting the model. Regularization penalizes large coefficients by a factor of $\alpha$. There are two types of regularization: L1 regularizationIt adds a penalty that is equal to the absolute value of the magnitude of the coefficients. Some of the coefficients can be reduced to zero and thus eliminated, so L1-type regularization can result in models with a lower number of cofficients. Examples: Lasso regularization L2 regularizationIt adds a penalty equal to the square of the coefficients. It doesn't reduce the amount of coefficients: they are all reduced by the same factor and none is eliminated.Examples: Ridge regression and SVMs.__We always use Ridge first, then Lasso.__ Elastic nets: They combine L1 and L2, but adding a hypterparameter.<jupyter_code>from sklearn import datasets
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt

db = datasets.load_diabetes()

print(db.keys())
print(db.feature_names)
print(db.DESCR)
X = db.data
y = db.target

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.3)

print(X.shape)
print(y.shape)
for num in range(10):
    plt.scatter(X[:,num], y)
    plt.xlabel(db.feature_names[num])
    plt.ylabel('target')
    plt.show()<jupyter_output><empty_output><jupyter_text>Applying regularization Ridge<jupyter_code>from sklearn.linear_model import Ridge
from sklearn.model_selection import cross_val_score
import numpy as np

# Setup the array of alphas and lists to store scores

alpha_space = np.logspace(-4, 0, 50) # idk why they use a logarithmic scale
ridge_scores = []
ridge_scores_std = []

# Let's check out the numbers:
for element in alpha_space:
    print(round(element, 2))
ridge = Ridge(normalize=True)

# Compute scores over range of alphas

for alpha in alpha_space:
    # Specify the alpha value to use: ridge.alpha
    ridge.alpha = alpha
    
    # Performe 10-fold CV: ridge_cv_scores
    ridge_cv_scores = cross_val_score(ridge, X, y, cv=10)
    
    # Append the mean of ridge_cv_scores to ridge_scores
    ridge_scores.append(np.mean(ridge_cv_scores))
    
    # Append the std<jupyter_output><empty_output><jupyter_text>Lasso<jupyter_code>from sklearn.linear_model import Lasso

lasso = Lasso(alpha=0.4, normalize=True)
lasso.fit(X_train, y_train)
lasso_coef = lasso.coef_
print(lasso_coef)
plt.plot(range(X.shape[1]), lasso_coef)
plt.xticks(range(X.shape[1]), db.feature_names, rotation=60)
plt.margins(0.02)
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0812452543659833, 'fraction_numerical': 0.010250569476082005, 'mean_word_length': 5.2, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '206176', 'license': 'Apache-2.0', 'n_tokens_mistral': 876, 'n_tokens_neox': 833, 'n_words': 280, 'path': 'datacamp_machine_learning_scientist_with_python/01_supervised_learning_with_scikit-learn/04_Regularized_regression.ipynb', 'repo_name': 'martinacantaro/data_science', 'size': '2634'}