<jupyter_start><jupyter_text>Lambda School Data Science*Unit 2, Sprint 1, Module 3*---  Ridge Regression- Do one-hot encoding of categorical features- Do univariate feature selection- Use scikit-learn to fit Ridge Regression models  SetupYou can work locally (follow the [local setup instructions](https://lambdaschool.github.io/ds/unit2/local/)) or on Colab (run the code cell below).Libraries:- category_encoders- matplotlib- numpy- pandas- scikit-learn<jupyter_code>%%capture
import sys

# If you're on Colab:
if 'google.colab' in sys.modules:
    DATA_PATH = 'https://raw.githubusercontent.com/LambdaSchool/DS-Unit-2-Applied-Modeling/master/data/'
    !pip install category_encoders==2.*

# If you're working locally:
else:
    DATA_PATH = '../data/'<jupyter_output><empty_output><jupyter_text>Do one-hot encoding of categorical features  Overview First, let's load the NYC apartment rental listing data:<jupyter_code>import numpy as np
import pandas as pd

# Read New York City apartment rental listing data
df = pd.read_csv(DATA_PATH+'apartments/renthop-nyc.csv')
assert df.shape == (49352, 34)

# Remove the most extreme 1% prices,
# the most extreme .1% latitudes, &
# the most extreme .1% longitudes
df = df[(df['price'] >= np.percentile(df['price'], 0.5)) & 
        (df['price'] <= np.percentile(df['price'], 99.5)) & 
        (df['latitude'] >= np.percentile(df['latitude'], 0.05)) & 
        (df['latitude'] < np.percentile(df['latitude'], 99.95)) &
        (df['longitude'] >= np.percentile(df['longitude'], 0.05)) & 
        (df['longitude'] <= np.percentile(df['longitude'], 99.95))]

# Do train/test split
# Use data from April & May 2016 to train
# Use data from June 2016 to test
df['created'] = pd.to_datetime(df['created'], infer_datetime_format=True)
cutoff = pd.to_datetime('2016-06-01')
train = df[df.created < cutoff]
test  = df[df.created >= cutoff]
mask = (df['price'])
np.percentile(df['price'], 1)
np.percentile(df['price'], 99)
df['price'].describe()
(df['price'] > np.percentile(df['price'], 0.5)).value_counts()
(df['price'] > np.percentile(df['price'], 0.5)) & (df['price'] < np.percentile(df['price'], 99.5))
mask = (df['price'] > np.percentile(df['price'], 0.5)) & (df['price'] < np.percentile(df['price'], 99.5))
len(df)
df[mask].head(5)
len(df[mask])<jupyter_output><empty_output><jupyter_text>Some columns are numeric:<jupyter_code># train.describe(include=[np.number])
train.describe(include='number')<jupyter_output><empty_output><jupyter_text>Some columns are _not_ numeric:<jupyter_code>train.describe(exclude=[np.number])
# train.describe(exclude='number')
# GOOD_TRICK .T transpose the describe() table so that the summary stats are columns
train.describe(exclude='number').T.sort_values(by='unique')
# Then sort_values(by='unique') will put the feature with the fewest unique values on top<jupyter_output><empty_output><jupyter_text>Let's look at the relationship between `interest_level` and `price`:<jupyter_code>train['interest_level'].value_counts()
train['price'].mean()
#What's the average price by interest_level?
# *** >>> df_dcf.groupby('sfy')['total_entry'].sum() ??? <<< ***
train.groupby('interest_level')['price'].mean()
X_train = train.select_dtypes(include=['number'])<jupyter_output><empty_output><jupyter_text>Interest Level seems like a useful, predictive feature. But it's a string — and our scikit-learn models expect all inputs to be numbers. So, we can "one-hot encode" the feature.To go from this:| interest_level ||----------------|| high           || medium         || low            |To this:| interest_level_high | interest_level_medium | interest_level_low ||---------------------|-----------------------|--------------------|| 1                   | 0                     | 0                  || 0                   | 1                     | 0                  || 0                   | 0                     | 1                  | "One-hot encoding" adds a dimension for each unique value of each categorical feature. So, it may not be a good choice for "high cardinality" categoricals that have dozens, hundreds, or thousands of unique values. [Cardinality](https://simple.wikipedia.org/wiki/Cardinality) means the number of unique values that a feature has:> In mathematics, the cardinality of a set means the number of its elements. For example, the set A = {2, 4, 6} contains 3 elements, and therefore A has a cardinality of 3.   Follow Along The other non-numeric columns have high cardinality. So let's exclude them from our features for now.<jupyter_code># TODO
X_train<jupyter_output><empty_output><jupyter_text>Here's what `X_train` looks like **before** encoding:<jupyter_code># TODO
X_train<jupyter_output><empty_output><jupyter_text>Use [OneHotEncoder](https://contrib.scikit-learn.org/categorical-encoding/onehot.html) from the [category_encoders](https://github.com/scikit-learn-contrib/categorical-encoding) library to encode any non-numeric features. (In this case, it's just `interest_level`.)- Use the **`fit_transform`** method on the **train** set- Use the **`transform`** method on **validation / test** sets<jupyter_code># TODO
import category_encoders as ce
encoder = ce.OneHotEncoder(use_cat_names=True)
encoder.fit_transform(X_train)<jupyter_output><empty_output><jupyter_text>Here's what it looks like **after:**<jupyter_code># TODO
X_train
encoder<jupyter_output><empty_output><jupyter_text>Challenge In your assignment, you will do one-hot encoding of categorical features with feasible cardinality, using the category_encoders library.  Do univariate feature selection  Overview The previous assignment quoted Wikipedia on [Feature Engineering](https://en.wikipedia.org/wiki/Feature_engineering):> "Some machine learning projects succeed and some fail. What makes the difference? Easily the most important factor is the features used." — <NAME>, ["A Few Useful Things to Know about Machine Learning"](https://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf)> "Coming up with features is difficult, time-consuming, requires expert knowledge. 'Applied machine learning' is basically feature engineering." — <NAME>, [Machine Learning and AI via Brain simulations](https://forum.stanford.edu/events/2011/2011slides/plenary/2011plenaryNg.pdf) > Feature engineering is the process of using domain knowledge of the data to create features that make machine learning algorithms work. <NAME> says, "the most important factor is the **features used**."This includes not just **Feature Engineering** (making new features, representing features in new ways) but also **Feature Selection** (choosing which features to include and which to exclude).There are _many_ specific tools and techniques for feature selection.- Today we'll try [scikit-learn's `SelectKBest` transformer](https://scikit-learn.org/stable/modules/feature_selection.htmlunivariate-feature-selection), for "univariate, forward selection."- Later we'll try another technique, ["permutation importance"](https://www.kaggle.com/dansbecker/permutation-importance)- If you want to explore even more options, here are some good resources!  - [scikit-learn's User Guide for Feature Selection](https://scikit-learn.org/stable/modules/feature_selection.html)  - [mlxtend](http://rasbt.github.io/mlxtend/) library  - scikit-learn-contrib libraries: [boruta_py](https://github.com/scikit-learn-contrib/boruta_py) & [stability-selection](https://github.com/scikit-learn-contrib/stability-selection)  - [_Feature Engineering and Selection_](http://www.feat.engineering/) by Kuhn & Johnson.My general recommendation is:> Predictive accuracy on test sets is the criterion for how good the model is. — <NAME>, ["Statistical Modeling: The Two Cultures"](https://projecteuclid.org/download/pdf_1/euclid.ss/1009213726) First, let's engineer a few more features to select from. This is a partial example solution from the previous assignment.<jupyter_code>X_test = X_train
# <NAME>'s wrangle function
def engineer_features(X):
    
    # Avoid SettingWithCopyWarning
    X = X.copy()
    
    # How many total perks does each apartment have?
    perk_cols = ['elevator', 'cats_allowed', 'hardwood_floors', 'dogs_allowed',
                 'doorman', 'dishwasher', 'no_fee', 'laundry_in_building',
                 'fitness_center', 'pre-war', 'laundry_in_unit', 'roof_deck',
                 'outdoor_space', 'dining_room', 'high_speed_internet', 'balcony',
                 'swimming_pool', 'new_construction', 'exclusive', 'terrace', 
                 'loft', 'garden_patio', 'common_outdoor_space', 
                 'wheelchair_access']
    X['perk_count'] = X[perk_cols].sum(axis=1)

    # Are cats or dogs allowed?
    X['cats_or_dogs'] = (X['cats_allowed']==1) | (X['dogs_allowed']==1)

    # Are cats and dogs allowed?
    X['cats_and_dogs'] = (X['cats_allowed']==1) & (X['dogs_allowed']==1)

    # Total number of rooms (beds + baths)
    X['rooms'] = X['bedrooms'] + X['bathrooms']

    return X

X_train = engineer_features(X_train)
X_test = engineer_features(X_test)<jupyter_output><empty_output><jupyter_text>Could we try every possible feature combination?The number of [combinations](https://en.wikipedia.org/wiki/Combination) is shocking!<jupyter_code># How many features do we have currently?
features = X_train.columns
n = len(features)
n
# How many ways to choose 1 to n features?
from math import factorial

def n_choose_k(n, k):
    return factorial(n)/(factorial(k)*factorial(n-k))

combinations = sum(n_choose_k(n,k) for k in range(1,n+1))
print(f'{combinations:,.0f}')<jupyter_output>8,589,934,591<jupyter_text>We can't try every possible combination, but we can try some. For example, we can use univariate statistical tests to measure the correlation between each feature and the target, and select the k best features.<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt

# create a quick dataframe with two columns, A and B
# df = pd.DataFrame({'A':[4,5,6,3,7,1], 'B':[6,4,1,7,2,3]})
df = train
col_a = df['price']
col_b = df['bedrooms']

# pass in the number of rows and columns you want for your graphs
# (2,2) would make a 2 by 2 grid, (1,3) makes a single row of 3 graphs
fig, ax = plt.subplots(1,3)

# you usually want to change the size, so the graphs aren't squished together
fig.set_size_inches(10,3)

# adjust the subplot layout with this function
# for this example, I'm just increasing the space between each graph (default is .2)
plt.subplots_adjust(wspace=.3)

# ax is an array, with the each of the graphs in it
ax[0].set_title('Scatter Plot')
ax[0].set_ylabel('Column B')
ax[0].set_xlabel('Column A')
ax[0].scatter(x='A', y='B', data=df)

ax[1].set_title('Line Plot')
ax[1].set_ylabel('Value')
ax[1].set_xlabel('Index')
ax[1].plot(df)

ax[2].set_title('Bar Chart')
ax[2].set_ylabel('Averages')
ax[2].set_xticks([0,1])
ax[2].set_xticklabels(['A','B'])
ax[2].bar(x=0, height=df['A'].mean())
ax[2].bar(x=1, height=df['B'].mean());

# note: if you have multiple rows and columns, ax will be a 2D array
# so you would instead access graphs by ax[row][column]<jupyter_output><empty_output><jupyter_text>Follow Along Refer to the [Scikit-Learn User Guide on Univariate Feature Selection](https://scikit-learn.org/stable/modules/feature_selection.htmlunivariate-feature-selection).<jupyter_code># TODO: Select the 15 features that best correlate with the target
# (15 is an arbitrary starting point here)
from sklearn.feature_selection import SelectKBest, f_regression
selector = SelectKBest(score_func=f_regression, k=15)

# SelectKBest has a similar API to what we've seen before.
# IMPORTANT!
# .fit_transform on the train set
# .transform on test set

selector.fit_transform(X_train, y_train)
selector.fit_transform(X_train, y_train).shape
# shape should be (31844, 15)
X_train_selected = selector.fit_transform(X_train, y_train)
X_test_selected = selector.transform(X_test)
X_train_selected.shape, X_test_selected.shape
# TODO: Which features were selected?
selector.get_support()
# this is because scikit learn is not written on top of pandas
# so it's numpy array instead of pandas dataframe (no column names)
selector.get_support().shape #35 Trues and Falses
selected_mask = selector.get_support
all_names = X_train.columns
selected_names = all_names[selected_mask]
unselected_names = all_names[~selected_mask]
print('Features selected')
for name in selected_names:
  print(name)

print('\n')
print('Features not selected')
for name in unselected_names:
  print(name)
# TODO: How many features should be selected?

# Guess and check: You can try a range of values for k,
# then choose the model with the best score.
# If multiple models "tie" for the best score,
# choose the simplest model.
# You decide what counts as a tie!

from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error

# for k in [1, 2, 3]:
for k in range(1, len(X_train.columns)):
  print(f'{k} features')
  selector = SelectKBest(score_func=f_regression, k=k)
  X_train_selected = selector.fit_transform(X_train, y_train)
  X_test_selected = selector.transform(X_test)

  model = LinearRegression()
  model.fit(X_train_selected, y_train)
  y_pred = model.predict(X_test_selected)
  mae = mean_absolute_error(y_test, y_pred)
  print(f'Test Mean Absolute Error: ${mae:,.0f}\n')

# should plateau around $675
# this gets what error by adding one more, one more feature
# 15 features seems like a good number

# you may get something crazy like average error suddenly goes to $545 gazillion
# == overfitting<jupyter_output>1 features
2 features
3 features
4 features
5 features
6 features
7 features
8 features
9 features
10 features
11 features
12 features
13 features
14 features
15 features
16 features
17 features
18 features
19 features
20 features
21 features
22 features
23 features
24 features
25 features
26 features
27 features
28 features
29 features
30 features
31 features
32 features<jupyter_text>ChallengeIn your assignment, you will do feature selection with SelectKBest.You'll go back to our other New York City real estate dataset. Instead of predicting apartment rents, you'll predict property sales prices. But not just for condos in Tribeca. Instead, you'll predict property sales prices for One Family Dwellings, where the sale price was more than 100 thousand and less than 2 million.If you run the above code cell with your assignment dataset, you will probably get some shocking results like these:```1 featuresTest MAE: $183,641 2 featuresTest MAE: $182,569 3 featuresTest MAE: $182,569 4 featuresTest MAE: $183,441 5 featuresTest MAE: $186,532 6 featuresTest MAE: $176,121 7 featuresTest MAE: $177,001 8 featuresTest MAE: $176,707 9 featuresTest MAE: $170,969 10 featuresTest MAE: $170,977 11 featuresTest MAE: $170,507 12 featuresTest MAE: $162,301 13 featuresTest MAE: $163,559 14 featuresTest MAE: $162,562 15 featuresTest MAE: $162,550 16 featuresTest MAE: $162,678 17 featuresTest MAE: $162,419 18 featuresTest MAE: $162,177 19 featuresTest MAE: $162,177 20 featuresTest MAE: $157,893 21 featuresTest MAE: $157,966 22 featuresTest MAE: $157,966 23 featuresTest MAE: $157,966 24 featuresTest MAE: $157,630 25 featuresTest MAE: $157,580 26 featuresTest MAE: $25,968,990,575,776,280 27 featuresTest MAE: $157,550 28 featuresTest MAE: $87,300,193,986,380,608 29 featuresTest MAE: $158,491 30 featuresTest MAE: $17,529,140,644,990,770 31 featuresTest MAE: $24,191,458,933,688,856 32 featuresTest MAE: $15,214,122,471,992,104 33 featuresTest MAE: $15,539,731,847,001,626 34 featuresTest MAE: $26,823,915,969,200,480 35 featuresTest MAE: $3,813,290,272,870,121 36 featuresTest MAE: $157,900 37 featuresTest MAE: $157,900 38 featuresTest MAE: $158,911 39 featuresTest MAE: $9,101,846,282,581,472 40 featuresTest MAE: $1,424,168,120,777,820 41 featuresTest MAE: $158,261 42 featuresTest MAE: $158,261 43 featuresTest MAE: $4,784,333,158,313,152 44 featuresTest MAE: $1,329,759,264,341,476 45 featuresTest MAE: $158,451 46 featuresTest MAE: $158,450 47 featuresTest MAE: $158,450 48 featuresTest MAE: $1,331,383,815,682,658 49 featuresTest MAE: $1,504,319,420,789,134 50 featuresTest MAE: $2,285,927,437,866,492``` Why did the error blow up with 26 features? We can look at the coefficients of the selected features:```BLOCK                                                  -97,035ZIP_CODE                                              -152,985COMMERCIAL_UNITS                    -4,115,324,664,197,034,496TOTAL_UNITS                         -2,872,516,607,892,124,160GROSS_SQUARE_FEET                                      123,739BOROUGH_3                                              146,876BOROUGH_4                                              197,262BOROUGH_2                                              -55,917BOROUGH_5                                              -84,107NEIGHBORHOOD_OTHER                                      56,036NEIGHBORHOOD_FLUSHING-NORTH                             63,394NEIGHBORHOOD_FOREST HILLS                               46,411NEIGHBORHOOD_BOROUGH PARK                               29,915TAX_CLASS_AT_PRESENT_1D               -545,831,543,721,722,112BUILDING_CLASS_AT_PRESENT_A5                            -1,673BUILDING_CLASS_AT_PRESENT_A3             5,516,361,218,128,626BUILDING_CLASS_AT_PRESENT_S1         1,735,885,668,110,473,216BUILDING_CLASS_AT_PRESENT_A6           -25,974,113,243,185,788BUILDING_CLASS_AT_PRESENT_A8          -760,608,646,332,801,664BUILDING_CLASS_AT_PRESENT_S0           941,854,072,479,442,176BUILDING_CLASS_AT_TIME_OF_SALE_A5                      -24,599BUILDING_CLASS_AT_TIME_OF_SALE_A3       -5,516,361,218,111,506BUILDING_CLASS_AT_TIME_OF_SALE_S1    4,253,055,231,847,939,072BUILDING_CLASS_AT_TIME_OF_SALE_A6       25,974,113,243,176,404BUILDING_CLASS_AT_TIME_OF_SALE_A8     -541,733,439,448,260,800BUILDING_CLASS_AT_TIME_OF_SALE_S0      990,851,394,820,416,512```These were the coefficients that minimized the sum of squared errors in the training set. But this model has become too complex, with extreme coefficients that can lead to extreme predictions on new unseen data. This linear model needs _regularization._ Ridge Regression to the rescue!  Use scikit-learn to fit Ridge Regression models  Overview <NAME> explains in his [StatQuest video on Ridge Regression:](https://youtu.be/Q81RR3yKn30?t=222)> The main idea behind **Ridge Regression** is to find a new line that _doesn't_ fit the training data as well. In other words, we introduce a small amount of **bias** into how the new line is fit to the data. But in return for that small amount of bias we get a significant drop in **variance.** In other words, by starting with a slightly worse fit Ridge regression can provide better long-term predictions. BAM!!!  OLS vs Mean Baseline vs RidgeLet's see look at some examples.First, here's a famous, tiny dataset — [Anscombe's quartet](https://en.wikipedia.org/wiki/Anscombe%27s_quartet), dataset III:<jupyter_code>import seaborn as sns
anscombe = sns.load_dataset('anscombe').query('dataset=="III"')
anscombe.plot.scatter('x', 'y');<jupyter_output><empty_output><jupyter_text>We'll compare:- Ordinary Least Squares Linear Regression- Mean Baseline- Ridge Regression  OLS<jupyter_code>%matplotlib inline

# Plot data
ax = anscombe.plot.scatter('x', 'y')

# Fit linear model
ols = LinearRegression()
ols.fit(anscombe[['x']], anscombe['y'])

# Get linear equation
m = ols.coef_[0].round(2)
b = ols.intercept_.round(2)
title = f'Linear Regression \n y = {m}x + {b}'

# Get predictions
anscombe['y_pred'] = ols.predict(anscombe[['x']])

# Plot predictions
anscombe.plot('x', 'y_pred', ax=ax, title=title);<jupyter_output><empty_output><jupyter_text>Mean Baseline<jupyter_code># Plot data
ax = anscombe.plot.scatter('x', 'y')

# Mean baseline
mean = anscombe['y'].mean()
anscombe['y_pred'] = mean
title = f'Mean Baseline \n y = 0x + {mean:.2f}'

# Plot "predictions"
anscombe.plot('x', 'y_pred', ax=ax, title=title);<jupyter_output><empty_output><jupyter_text>Ridge RegressionWith increasing regularization:<jupyter_code>import matplotlib.pyplot as plt
from sklearn.linear_model import Ridge

def ridge_anscombe(alpha):
    """
    Fit & plot a ridge regression model,
    with Anscombe Quartet dataset III.

    alpha : positive float, regularization strength
    """

    # alpha is called lambda in the statquest videos
    # but lambda is a reserved keyword in Python

    # Load data
    anscombe = sns.load_dataset('anscombe').query('dataset=="III"')

    # Plot data
    ax = anscombe.plot.scatter('x', 'y')

    # Fit linear model
    ridge = Ridge(alpha=alpha, normalize=True)
    ridge.fit(anscombe[['x']], anscombe['y'])

    # Get linear equation
    m = ridge.coef_[0].round(2)
    b = ridge.intercept_.round(2)
    title = f'Ridge Regression, alpha={alpha} \n y = {m}x + {b}'

    # Get predictions
    anscombe['y_pred'] = ridge.predict(anscombe[['x']])

    # Plot predictions
    anscombe.plot('x', 'y_pred', ax=ax, title=title)
    plt.show()
    
    
for alpha in range(10):
    ridge_anscombe(alpha=alpha)<jupyter_output><empty_output><jupyter_text>When the Ridge Regression has no regularization (`alpha=0`) then it is identical to Ordinary Least Squares Regression.When we increase the regularization, the predictions looks less and less like OLS and more and more like the mean baseline. The predictions are less sensitive to changes in the independent variable. You may ask, how should we decide the amount of regularization?[The StatQuest video answers,](https://youtu.be/Q81RR3yKn30?t=602)> So how do we decide what value to give lambda? We just try a bunch of values for lambda, and use cross-validationtypically 10-fold cross-validation, to determine which one results in the lowest variance. DOUBLE BAM!!!You'll learn more about cross-validation next sprint. For now, the good news is scikit-learn gives us [RidgeCV](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.RidgeCV.html), "Ridge regression with built-in cross-validation."Also, notice that scikit-learn calls the regularization parameter "alpha", but StatQuest calls it "lambda." The greek letters are different, but the concept is the same. Let's try these values for alpha:<jupyter_code>alphas = [0.01, 0.1, 1.0, 10.0, 100.0]<jupyter_output><empty_output><jupyter_text>We'll use [RidgeCV](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.RidgeCV.html) to find the best alpha:<jupyter_code>from sklearn.linear_model import RidgeCV
ridge = RidgeCV(alphas=alphas, normalize=True)
ridge.fit(anscombe[['x']], anscombe['y'])
ridge.alpha_<jupyter_output><empty_output><jupyter_text>The fit looks similar to Ordinary Least Squares Regression, but slightly less influenced by the outlier:<jupyter_code># Plot data
ax = anscombe.plot.scatter('x', 'y')

# Get linear equation
m = ridge.coef_[0].round(2)
b = ridge.intercept_.round(2)
title = f'Ridge Regression, alpha={ridge.alpha_} \n y = {m}x + {b}'

# Get predictions
anscombe['y_pred'] = ridge.predict(anscombe[['x']])

# Plot predictions
anscombe.plot('x', 'y_pred', ax=ax, title=title)
plt.show()<jupyter_output><empty_output><jupyter_text>NYC, 1 feature Let's go back to our other New York City dataset, to demonstrate regularization with Ridge Regresson:<jupyter_code>from IPython.display import display, HTML

# Try a range of alpha parameters for Ridge Regression.

# The scikit-learn docs explain, 
# alpha : Regularization strength; must be a positive float. Regularization 
# improves the conditioning of the problem and reduces the variance of the 
# estimates. Larger values specify stronger regularization.
# https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Ridge.html

for alpha in [0.001, 0.01, 0.1, 1.0, 10.0, 100.0]:
    
    # Fit Ridge Regression model
    feature = 'bedrooms'
    display(HTML(f'Ridge Regression, with alpha={alpha}'))
    model = Ridge(alpha=alpha, normalize=True)
    model.fit(X_train[[feature]], y_train)
    
    # Get Test MAE
    y_pred = model.predict(X_test[[feature]])
    mae = mean_absolute_error(y_test, y_pred)
    display(HTML(f'Test Mean Absolute Error: ${mae:,.0f}'))
    
    train.plot.scatter(feature, target, alpha=0.05)
    plt.plot(X_test[feature], y_pred)
    plt.show()<jupyter_output><empty_output><jupyter_text>NYC, multiple features<jupyter_code>for alpha in [0.001, 0.01, 0.1, 1.0, 1, 100.0, 1000.0]:
    
    # Fit Ridge Regression model
    display(HTML(f'Ridge Regression, with alpha={alpha}'))
    model = Ridge(alpha=alpha, normalize=True)
    model.fit(X_train, y_train)
    y_pred = model.predict(X_test)

    # Get Test MAE
    mae = mean_absolute_error(y_test, y_pred)
    display(HTML(f'Test Mean Absolute Error: ${mae:,.0f}'))
    
    # Plot coefficients
    coefficients = pd.Series(model.coef_, X_train.columns)
    plt.figure(figsize=(16,8))
    coefficients.sort_values().plot.barh(color='grey')
    plt.xlim(-400,700)
    plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1171517921578061, 'fraction_numerical': 0.049154037366690724, 'mean_word_length': 4.492842986126404, 'pattern_counts': {'":': 0, '<': 102, '<?xml version=': 0, '>': 110, 'https://': 22, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 24, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '22', 'dir': 'jupyter-structured-clean-dedup', 'id': '39493', 'license': 'MIT', 'n_tokens_mistral': 8854, 'n_tokens_neox': 7806, 'n_words': 2353, 'path': 'module3-ridge-regression/LS_DS12_213.ipynb', 'repo_name': 'BoWarburton/DS-Unit-2-Linear-Models', 'size': '24975'}