<jupyter_start><jupyter_text>Assignment 1: Ridge Regression and the Lasso in Python This assignment on Ridge Regression and the Lasso is a Python adaptation of p. 251 - 255 of "Introduction to Statistical Learning with Application in R".<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

from sklearn.linear_model import Ridge, RidgeCV, Lasso, LassoCV
from sklearn import preprocessing
from sklearn.metrics import mean_squared_error

import warnings
warnings.filterwarnings('ignore')

hitters = pd.read_csv("Hitters.csv", index_col=0).dropna()
#hitters = hitters.drop(hitters.columns[0], axis=1)
hitters.head()<jupyter_output><empty_output><jupyter_text>1 Preprocessing Data<jupyter_code>hitters_num = hitters._get_numeric_data()
Y = hitters_num["Salary"]
X = hitters_num.drop(["Salary"], axis=1)
#X_norm = preprocessing.normalize(X)
X.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Index: 263 entries, -<NAME> to -<NAME>
Data columns (total 16 columns):
AtBat      263 non-null int64
Hits       263 non-null int64
HmRun      263 non-null int64
Runs       263 non-null int64
RBI        263 non-null int64
Walks      263 non-null int64
Years      263 non-null int64
CAtBat     263 non-null int64
CHits      263 non-null int64
CHmRun     263 non-null int64
CRuns      263 non-null int64
CRBI       263 non-null int64
CWalks     263 non-null int64
PutOuts    263 non-null int64
Assists    263 non-null int64
Errors     263 non-null int64
dtypes: int64(16)
memory usage: 34.9+ KB<jupyter_text>2 Lasso<jupyter_code>alphas = 10**np.linspace(15,-3,100)*0.5
alphas
lasso = Lasso(max_iter=100000, normalize=True, fit_intercept=False)
coefs = []
res=0
for a in alphas:
    lasso.set_params(alpha=a)
    lasso.fit(X, Y)
    if sum(lasso.coef_ != 0)==3 and not res:
        res=1
        print("The remainning pedictors are:", list(X.columns[[i for i, x in enumerate(lasso.coef_) if x!=0]]))
    coefs.append(lasso.coef_)

plt.figure(figsize=(15,10))
ax = plt.gca()
ax.plot(alphas*2, coefs)
ax.set_xscale('log')
#ax.set_xlim(ax.get_xlim()[::-1])  #reverse axis
plt.axis('tight')
plt.legend(X.columns, loc="upper right")
plt.xlabel('log(alpha)')
plt.ylabel('coefficients')<jupyter_output>The remainning redictors are: ['AtBat', 'CAtBat', 'PutOuts']<jupyter_text>From the plot above we observe that the final three predictors that remain in the model are: Hits, CRuns, and CRBI.Notice that in the coefficient plot that depending on the choice of tuning parameter, some of the coefficients are exactly equal to zero. We now perform 10-fold cross validation to choose the best alpha, refit the model and compute the associated test error:<jupyter_code>lassocv = LassoCV(cv=10, random_state=42).fit(X, Y)
lasso.set_params(alpha=lassocv.alpha_, fit_intercept=False)
print('The optimal regularization penality is:', lassocv.alpha_)
print("The total number of predictors that remain in the model with optimal regularization penalty is:", sum(lasso.fit(X, Y).coef_!=0))
print('The remaining parameters in the optimal model are:', list(X.columns[abs(lassocv.coef_)>0]))
pd.Series(lasso.coef_, index=X.columns)<jupyter_output><empty_output><jupyter_text>3 Ridge Regression<jupyter_code>ridge = Ridge(fit_intercept=False, normalize=True)
coefs = []
for a in alphas:
    ridge.set_params(alpha=a)
    ridge.fit(X,Y)
    coefs.append(ridge.coef_)
    
#np.shape(coefs)
plt.figure(figsize=(15,10))
ax = plt.gca()
ax.plot(alphas, coefs)
ax.set_xscale('log')
plt.legend(X.columns, loc="upper right")
plt.xlabel('alpha')
plt.ylabel('weights')
plt.title('Ridge coefficients as a function of the regularization')
plt.axis('tight')
plt.show()
ridgecv = RidgeCV(cv=10).fit(X, Y)
print('The optimal regularization penality is:', ridgecv.alpha_)<jupyter_output>The optimal regularization penality is: 10.0<jupyter_text>We now see the improvement over the test MSE, so we refit our ridge regression model on the full data se t, using the value of alpha chosen by cross-validation, and examine the coeffient estimates.<jupyter_code>pd.Series(ridgecv.coef_, index=X.columns)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1122942884801549, 'fraction_numerical': 0.03630203291384317, 'mean_word_length': 5.309923664122137, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '462499', 'license': 'MIT', 'n_tokens_mistral': 1558, 'n_tokens_neox': 1424, 'n_words': 406, 'path': 'HW1/HW1.ipynb', 'repo_name': 'Luceven/ANLY590_DeepLearning', 'size': '4143'}