<jupyter_start><jupyter_text>Lasso regression  Week 4, set problem 5The file mystery.dat contains pairs ($x$, $y$), where $x$ $\in$ $\mathbb{R}^{100}$ and $y$ $\in$ $\mathbb{R}$. There is one data point per line, with comma-separated values; the very last number in each line is the $y$-value.In this data set, $y$ is a linear function of just ten of the features in $x$, plus some noise. Your job is to identify those ten features.Which of the following contain only relevant features?For relevant feature selection I will use Lasso linear regression algorithm, which usess $L1$ metric for error computation.This type of regularization $(L1)$ can lead to zero coefficients i.e. some of the features are completely neglected for the evaluation of output.<jupyter_code># libraries for data manipulation or data representation
import math 
import matplotlib.pyplot as plt 
import pandas as pd
import numpy as np
from pandas.compat import StringIO

# libraries for learning
from sklearn.linear_model import Lasso
from sklearn.cross_validation import train_test_split
# read and store data
df = pd.read_csv('mystery.dat', 
                 sep=",", #separator comma
                 index_col=0,
                 header=None)
data = df.reset_index().values
print(data)
# seperate data into feature/label values
X = data[:,:100]
y = data[:,100]
print(data.shape)
print(X.shape)
print(y.shape)
# create test and train data sets
train_X, test_X, train_y, test_y = train_test_split(X, y, test_size=0.3, random_state=42)
# init. lasso object and fit data
lasso_reg = Lasso() # default alpha param. = 1
lasso_reg.fit(train_X, train_y)
train_err = lasso_reg.score(train_X, train_y)
test_err = lasso_reg.score(test_X, test_y)
num_coeff = np.sum(lasso_reg.coef_ != 0)

print(f"Training error: {train_err}\nTest error: {test_err}\nNumber of features: {num_coeff}")
print(lasso_reg.coef_)
print(np.nonzero(lasso_reg.coef_))
# try different alpha values that are smaller than 1
for alpha in [0.0005, 0.005, 0.05, 0.5]:
    lasso_reg = Lasso(alpha, max_iter=10e5) # default alpha param. = 1
    lasso_reg.fit(train_X, train_y)
    train_err = lasso_reg.score(train_X, train_y)
    test_err = lasso_reg.score(test_X, test_y)
    num_coeff = np.sum(lasso_reg.coef_ != 0)
    
    print(f"Alpha: {alpha}")
    print(f"Training error: {train_err}\nTest error: {test_err}\nNumber of features: {num_coeff}")
    print(np.nonzero(lasso_reg.coef_))
    print("*************************")
    print("*************************")<jupyter_output>Alpha: 0.0005
Training error: 0.999994920022373
Test error: 0.21822918175525285
Number of features: 75
(array([ 1,  2,  3,  4,  5,  6,  7, 10, 12, 13, 14, 15, 16, 18, 19, 20, 21,
       22, 23, 24, 26, 27, 28, 29, 30, 31, 32, 34, 35, 36, 38, 40, 41, 42,
       43, 45, 47, 48, 49, 52, 53, 54, 57, 58, 59, 60, 61, 62, 63, 66, 67,
       68, 70, 71, 72, 73, 75, 77, 78, 80, 81, 83, 84, 85, 86, 88, 89, 90,
       91, 92, 93, 94, 95, 96, 98], dtype=int64),)
*************************
*************************
Alpha: 0.005
Training error: 0.9995483097739087
Test error: 0.31364868671332624
Number of features: 71
(array([ 1,  2,  3,  4,  5,  6,  7, 10, 11, 12, 13, 14, 15, 16, 18, 19, 20,
       21, 22, 23, 24, 26, 27, 28, 29, 30, 31, 32, 34, 36, 38, 40, 42, 43,
       45, 47, 48, 49, 52, 53, 54, 57, 58, 60, 61, 62, 63, 66, 67, 68, 70,
       71, 72, 73, 75, 77, 78, 80, 81, 83, 84, 85, 86, 88, 89, 90, 91, 93,
       94, 96, 98], dtype=int64),)
*************************
*************************
Al[...]
{'fraction_non_alphanumeric': 0.1689381033503691, 'fraction_numerical': 0.11470755252697332, 'mean_word_length': 3.990084985835694, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 39, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '366948', 'license': 'MIT', 'n_tokens_mistral': 1608, 'n_tokens_neox': 1251, 'n_words': 444, 'path': 'Week 4/lasso-regression/lasso regression.ipynb', 'repo_name': 'MarkoJereb/UCSanDiegoX-DataScience', 'size': '3522'}