<jupyter_start><jupyter_text>RegularizationOne of the goals of a machine learning project is to make models which are highly predictive.If the model fails to generalize to unseen data then the model is bad Review of Earlier concepts Overfitting vs Underfitting    1. Underfit Models fail to capture all of the information in the data        1. Ex. the mean leaves a lot of information on the table    2. Overfit models fit to the noise in the data and fail to generalize    3. How would we know if our model is over or underfit?        1. Train test split        2. Look at the testing error        3. As model complexity increases so does the possibility for overfitting          Bias variance tradeoff    1. High bias        1. Systematic error in predictions (ie the average)        2. Bias is about the strength of assumptions the model makes        3. Underfit models tend to be high bias    2. High variance        1. The model is highly sensitive to changes in the data        2. Overfit models tend to be low bias  Regularization - a way to prevent over fitting    1. Types of regularization        1. Reducing the amount of features        2. Increase the amount of data        3. Ridge, lasso, elastic net        Again, complex models are very flexible in the patterns that they can model but this also means that they can easily find patterns that are simply statistical flukes of one particular dataset rather than patterns reflective of the underlying data generating process.  Ridge and Lasso Regression<jupyter_code>from sklearn.preprocessing import StandardScaler, PolynomialFeatures
from sklearn.linear_model import Ridge, Lasso, ElasticNet, LinearRegression
from sklearn.model_selection import train_test_split
import pandas as pd
from sklearn.metrics import mean_squared_error
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
ads = pd.read_csv('Advertising.csv', index_col=0)
ads.head()
X_train, X_test, y_train, y_test = train_test_split(ads[['TV',
                                                         'radio',
                                                         'newspaper']],
                                                   ads['sales'])<jupyter_output><empty_output><jupyter_text>First simple model<jupyter_code>lr1 = LinearRegression()
lr1.fit(X_train, y_train)

ads_preds = lr1.predict(X_test)

np.sqrt(mean_squared_error(ads_preds, y_test))
sns.pairplot(ads);<jupyter_output><empty_output><jupyter_text>Polynomial Features<jupyter_code># Try and create your own polynomial features with degree 2 in sklearn and create a linear model
# Answer is at the bottom 
# # Instantiate our polynomial features object with degree of 2
# pf = PolynomialFeatures(degree=2)

# x_poly = pf.fit_transform(X_train)

# lr_poly = LinearRegression()
# lr_poly.fit(x_poly, y_train)

# x_poly_test = pf.fit_transform(X_test)
# poly_preds_test = lr_poly.predict(x_poly_test)

# rmse = (mean_squared_error(poly_preds, y_test))**.5
# rmse<jupyter_output><empty_output><jupyter_text>Ridge Regression<jupyter_code>ss = StandardScaler()
pf = PolynomialFeatures()

X_train_processed = pf.fit_transform(ss.fit_transform(X_train))
X_test_processed = pf.transform(ss.transform(X_test))

#alpha == lambda since lamda is a key word in python
rr = Ridge(alpha=1)

rr.fit(X_train_processed, y_train)
ridge_preds = rr.predict(X_test_processed)

np.sqrt(mean_squared_error(ridge_preds, y_test))<jupyter_output><empty_output><jupyter_text>Why did I pick an alpha of 1? Is there a principled way to choose this hyperparamter?Create many models on the training set and see which one does best on unseen data  CrossvalidationHow can we know what values of lamda to choose for ridge or lasso regression? Crossvalidation!We have already discussed the train test split and that is the basis for all other kinds of crossvalidation. We can extend this<jupyter_code>from sklearn.model_selection import cross_val_score

cross_val_score()<jupyter_output><empty_output><jupyter_text>Try to find the optimal value for alpha/lamda for lasso regression  Appendix<jupyter_code>pf = PolynomialFeatures()
X_poly_train = pf.fit_transform(X_train[['TV', 'radio', 'newspaper']])

pf_sales = pd.DataFrame(X_poly_train, columns=pf.get_feature_names(['TV', 'radio', 'newspaper']))
pf_y = pd.concat([pf_sales[['TV', 'radio', 'newspaper', 'TV radio',
                            'TV newspaper']],
                  ads['sales']],  axis=1)
poly_lr = LinearRegression()
poly_lr.fit(X_poly_train, y_train)


X_poly_test = pf.transform(X_test[['TV', 'radio', 'newspaper']])
poly_preds = poly_lr.predict(X_poly_test)

np.sqrt(mean_squared_error(poly_preds, y_test))
list(zip(pf_sales.columns, poly_lr.coef_))
sns.pairplot(pf_y);<jupyter_output>/Users/calvintirrell/opt/anaconda3/envs/learn-env/lib/python3.6/site-packages/numpy/lib/histograms.py:824: RuntimeWarning: invalid value encountered in greater_equal
  keep = (tmp_a >= first_edge)
/Users/calvintirrell/opt/anaconda3/envs/learn-env/lib/python3.6/site-packages/numpy/lib/histograms.py:825: RuntimeWarning: invalid value encountered in less_equal
  keep &= (tmp_a <= last_edge)
{'fraction_non_alphanumeric': 0.07744830277019119, 'fraction_numerical': 0.007998439328911432, 'mean_word_length': 4.163141993957704, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '285017', 'license': 'MIT', 'n_tokens_mistral': 1591, 'n_tokens_neox': 1534, 'n_words': 507, 'path': 'regularization_crossvalidation.ipynb', 'repo_name': 'calvintirrell/dsc-regularization-seattle', 'size': '5126'}