<jupyter_start><jupyter_text>Lasso Regression<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pylab as plt
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LassoLarsCV
#Load the dataset
data = pd.read_csv("ml_python_lasso_regression_1.csv")
data.head()
#upper-case all DataFrame column names
data.columns = map(str.upper, data.columns)
# Data Management
data_clean = data.dropna()
recode1 = {1:1, 2:0}
data_clean['MALE']= data_clean['BIO_SEX'].map(recode1)
#select predictor variables and target variable as separate data sets  
predvar= data_clean[['MALE','HISPANIC','WHITE','BLACK','NAMERICAN','ASIAN',
'AGE','ALCEVR1','ALCPROBS1','MAREVER1','COCEVER1','INHEVER1','CIGAVAIL','DEP1',
'ESTEEM1','VIOL1','PASSIST','DEVIANT1','GPA1','EXPEL1','FAMCONCT','PARACTV',
'PARPRES']]
predvar.head()
target = data_clean.SCHCONN1
target.head()
# standardize predictors to have mean=0 and sd=1
predictors = predvar.copy()
from sklearn import preprocessing
predictors['MALE']=preprocessing.scale(predictors['MALE'].astype('float64'))
predictors['HISPANIC']=preprocessing.scale(predictors['HISPANIC'].astype('float64'))
predictors['WHITE']=preprocessing.scale(predictors['WHITE'].astype('float64'))
predictors['NAMERICAN']=preprocessing.scale(predictors['NAMERICAN'].astype('float64'))
predictors['ASIAN']=preprocessing.scale(predictors['ASIAN'].astype('float64'))
predictors['AGE']=preprocessing.scale(predictors['AGE'].astype('float64'))
predictors['ALCEVR1']=preprocessing.scale(predictors['ALCEVR1'].astype('float64'))
predictors['ALCPROBS1']=preprocessing.scale(predictors['ALCPROBS1'].astype('float64'))
predictors['MAREVER1']=preprocessing.scale(predictors['MAREVER1'].astype('float64'))
predictors['COCEVER1']=preprocessing.scale(predictors['COCEVER1'].astype('float64'))
predictors['INHEVER1']=preprocessing.scale(predictors['INHEVER1'].astype('float64'))
predictors['CIGAVAIL']=preprocessing.scale(predictors['CIGAVAIL'].astype('float64'))
predictors['DEP1']=preprocessing.scale(predictors['DEP1'].astype('float64'))
predictors['ESTEEM1']=preprocessing.scale(predictors['ESTEEM1'].astype('float64'))
predictors['VIOL1']=preprocessing.scale(predictors['VIOL1'].astype('float64'))
predictors['PASSIST']=preprocessing.scale(predictors['PASSIST'].astype('float64'))
predictors['DEVIANT1']=preprocessing.scale(predictors['DEVIANT1'].astype('float64'))
predictors['GPA1']=preprocessing.scale(predictors['GPA1'].astype('float64'))
predictors['EXPEL1']=preprocessing.scale(predictors['EXPEL1'].astype('float64'))
predictors['FAMCONCT']=preprocessing.scale(predictors['FAMCONCT'].astype('float64'))
predictors['PARACTV']=preprocessing.scale(predictors['PARACTV'].astype('float64'))
predictors['PARPRES']=preprocessing.scale(predictors['PARPRES'].astype('float64'))
# split data into train and test sets
pred_train, pred_test, tar_train, tar_test = train_test_split(predictors, target, 
                                                              test_size=.3, random_state=123)<jupyter_output><empty_output><jupyter_text>Now we are ready to run our LASSO regression analysis with the LAR algorithm using the LASSO LarsCV function from the sklearn linear model library. LASSO Regression has a couple of different model selection algorithms. In this example, I will use the LAR Algorithm, which stands for Least Angle Regression. This algorithm starts with no predictors in the model and adds a predictor at each step. It first adds a predictor that is most correlated with the response variable and moves it towards least score estimate until there is another predictor. That is equally correlated with the model residual. It adds this predictor to the model and starts the least square estimation process over again with both variables. The LAR algorithm continues with this process until it has tested all the predictors. Parameter estimates at any step are shrunk and predictors with coefficients that have shrunk to zero are removed from the model and the process starts all over again. So I create an object called model that will contain the results of the Lasso regression analysis, then I type the name of the function, LassoLarsCV. In parentheses, I add cv = 10, which asks Python to use k-fold cross-validation with ten random folds from the training data set to choose the final statistical model. The option precompute=False tells Python not to use a precomputed matrix. This option would be helpful with very large models, because the precomputed matrix can speed up calculations. However, the model I'm going to test is not very large. To fit the Lasso regression on the training set after the parenthesis, I add .fit, and in another set of parenthesis the names of the training predictor and response variable data sets. So what I am doing here is using k-fold cross-validation in which the first fold of the training data set is treated as a validation set, and the model is estimated using the remaining nine folds. At each step of the estimation process, when a new predictor is entered into the model, the mean-square error for the validation fold is calculated for each of the other nine folds and then averaged. The model that produces the lowest mean-square error is selected by Python as the best model to validate using the test dataset. Let's go ahead and run the code and take a look at the results. The first thing we can do is tell Python to print the variables, and the regression coefficients from the model that was retained by the model selection process as the best fitting model. The dict object creates a dictionary, and the zip object creates lists. So here I typed the dict object And in parenthesis, I add the zip object, and then in the next set of parenthesis I name the two lists that I want printed. In this case, I want to print the variable labels and the associated regression coefficients. I do this by first typing the name of the data frame that has the predictor variables, which I named predictors earlier, and then .com, which will provide the variable labels. Then I add a comma and the name of the attribute from the LASSO model results object that I named model.coef_, which is the name of the regression co-efficient attribute. We can run the code and take a look at the output.<jupyter_code># specify the lasso regression model
# k-fold cross validation (cv = 10 => 10 fold)
model = LassoLarsCV(cv=10, precompute=False).fit(pred_train,tar_train)
model
# print variable names and regression coefficients
dict(zip(predictors.columns, model.coef_))<jupyter_output><empty_output><jupyter_text>Predictors with regression coefficients that do not have a value of zero are included in the selected model. Predictors with regression coefficients equal to zero means that the coefficients for those variables had shrunk to zero after applying the LASSO regression penalty, and were subsequently removed from the model. So the results show that of the 23 variables, 18 were selected in the final model.<jupyter_code>%matplotlib inline
# plot coefficient progression
m_log_alphas = -np.log10(model.alphas_)
ax = plt.gca()
plt.plot(m_log_alphas, model.coef_path_.T)
plt.axvline(-np.log10(model.alpha_), linestyle='--', color='k',
            label='alpha CV')
plt.ylabel('Regression Coefficients')
plt.xlabel('-log(alpha)')
plt.title('Regression Coefficients Progression for Lasso Paths')<jupyter_output><empty_output><jupyter_text>For creating the plot, I will apply a negative log10 transformation to the alpha values. Simply to make the values easier to read by creating an object m_log_alphas that is equal to the negative of the -np.log10 transformation function applied to the alphas_ attribute from the model results object. The alphas_ attribute contains the values of alpha through the model selection process. The first line of code for the plot, sets up the axes, the second line of code asks Python to use the plot function from the mat plot lib library which we imported as plt to plot the transform values of alpha on the horizontal access. And the change in the regression coefficients in the coef_path_ attribute, from the model results object and the y axis. The .T asks python to transpose the coef_path_ attribute matrix to match the first dimension of the array of alpha values. I will use the plt.axlvline function to put a dashed vertical line at the -np.log10 transformed alpha value for the selected model. The color equals='k' in quotes tells Python to make the line color black. Finally, I add titles for the two axes and the plot title and run the code.This plot shows the relative importance of the predictor selected at any step of the selection process, how the reggression coefficients changed with the addition of a new predictor at each step, as well as the steps at which each variable entered the model. As we already know from looking at the list of the regression coefficients self esteem, the dark blue line, had the largest regression coefficient. It was therefore entered into the model first, followed by depression, the black line, at step two. In black ethnicity, the light blue line, at step three and so on.<jupyter_code># plot mean square error for each fold
m_log_alphascv = -np.log10(model.cv_alphas_)
plt.figure()
plt.plot(m_log_alphascv, model.mse_path_, ':')
plt.plot(m_log_alphascv, model.mse_path_.mean(axis=-1), 'k',
         label='Average across the folds', linewidth=2)
plt.axvline(-np.log10(model.alpha_), linestyle='--', color='k',
            label='alpha CV')
plt.legend()
plt.xlabel('-log(alpha)')
plt.ylabel('Mean squared error')
plt.title('Mean squared error on each fold')<jupyter_output>E:\Python\Anaconda\lib\site-packages\ipykernel_launcher.py:2: RuntimeWarning: divide by zero encountered in log10<jupyter_text>Another important plot is one that shows the change in the mean square error for the change in the penalty parameter alpha at each step in the selection process. This code is similar to the code for the previous plot except this time we're plotting the alpha values through the model selection process for each cross validation fold on the horizontal axis, and the mean square error for each cross validation fold on the vertical axis. This is done in the first plt.plot function. Where m_log_alphascv, is a negative log10 transformation applied to the alphascv_ attribute for each validation fold, and a cv_mse_path_ is the model results attribute containing the mean square error for each cross validation fold. The colon in quotes here tells Python to plot the folds as dotted lines. In the next line of code, I'm asking Python to plot the average mean squared error across all cross-validation folds, and to plot it as a slightly thicker line with equals to black line. I'll use the plt.axvline function to plot a dashed black vertical line at the -np.log10 transformed alpha value for the selected model. Finally, I ask Python to print a legend and add titles for the two axes as well as the plot title and I run the code. We can see that there is variability across the individual cross-validation folds in the training data set, but the change in the mean square error as variables are added to the model follows the same pattern for each fold. Initially it decreases rapidly and then levels off to a point at which adding more predictors doesn't lead to much reduction in the mean square error. This is to be expected as model complexity increases. We can also print the average mean square error in the r square for the proportion of variance in school connectedness. That is explained by the selected model in the training data set and in the test set when the selected model's applied to the test data using the following code. Here we need to import the mean squared error function from the sklearn metrics library to compute the mean square error. We create an object called train_error which is equal to the mean_squared_error calculation function, then in parentheses, the training data set response variable, tar_train, then a comma, and then model.predict(pred_train). This code tells Python to use the results from the model object to predict the response variable for observations in the training data set. We then do the same thing for the test data by using the results from the training set model to calculate the test data mean square error. We use the model.score attribute, which includes the predicted response values for the training, and test data sets for calculating the r square for each set. We then use the print function to print them.<jupyter_code># MSE from training and test data
from sklearn.metrics import mean_squared_error
train_error = mean_squared_error(tar_train, model.predict(pred_train))
test_error = mean_squared_error(tar_test, model.predict(pred_test))
print ('training data MSE')
print(train_error)
print ('test data MSE')
print(test_error)
# R-square from training and test data
rsquared_train=model.score(pred_train,tar_train)
rsquared_test=model.score(pred_test,tar_test)
print ('training data R-square')
print(rsquared_train)
print ('test data R-square')
print(rsquared_test)<jupyter_output>training data R-square
0.3336111369269187
test data R-square
0.31001113416000803
{'fraction_non_alphanumeric': 0.07161260988178236, 'fraction_numerical': 0.012200666868748105, 'mean_word_length': 5.682025316455697, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 20, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '292297', 'license': 'MIT', 'n_tokens_mistral': 3602, 'n_tokens_neox': 3422, 'n_words': 1646, 'path': 'machinelearning/ml_python_lasso_regression_1.ipynb', 'repo_name': 'futureseadev/hgwxx7', 'size': '13196'}