<jupyter_start><jupyter_text>Copyright 2020 <NAME> and made available under [CC BY-SA](https://creativecommons.org/licenses/by-sa/4.0) for text and [Apache-2.0](http://www.apache.org/licenses/LICENSE-2.0) for code.  Ridge and Lasso Regression: Problem solving In this session, we'll use the `boston` dataset, which has been used to examine the relationship between clean air and house prices:| Variable | Type | Description ||:----|:-----|:----------||crim | Ratio | per capita crime rate by town | |zn | Ratio | proportion of residential land zoned for lots over 25,000 sq. ft. | |indus | Ratio | proportion of non-retain business acres per town | |chas | Nominal (binary) | Charles River dummy variable (=1 if tract bounds river, =0 otherwise) | |nox | Ratio | nitrogen oxides concentration (parts per 10 million) | |rm | Ratio | average number of rooms per dwelling | |age | Ratio | proportion of owner-occupied units built prior to 1940 | |dis | Ratio | weighted mean of distances to five Boston employment centers | |rad | Ordinal | index of accessibility to radial highways | |tax | Ratio | full-value property tax rate per \$10,000 | |ptratio | Ratio | pupil-teacher ratio by town | |lstat | Ratio | percent lower status of population (defined as non-high school graduate, manual labor) | |medv | Ratio | median value of owner-occupied homes in $1000s |     Source: This dataset was taken from the StatLib library which is maintained at Carnegie Mellon University. As before, we'll try to predict `medv` using the rest of the variables.**Because `medv` is a ratio variable, we will use linear regression not logistic regression.**  Load dataImport `pandas` to load a dataframe. Load the dataframe.  Explore dataDescribe the data. -----------**QUESTION:**Do the min, mean, and max look reasonable to you, given what these variables mean (see the data description above)? **ANSWER: (click here to edit)** Make a correlation heatmap.First import `plotly.express`. And show the heatmap ----------------------**QUESTION:**Do we have strong positive correlations, strong negative correlations, or both? **ANSWER: (click here to edit)** **QUESTION:**Given the nature of these variables, do these correlations surprise you?  **ANSWER: (click here to edit)**  Prepare train/test setsDo the imports needed to split the dataframe into `X` and `Y`. Create `X` by dropping the response variable `medv` from the dataframe. Create `Y` by pulling just `medv` from the dataframe. Now do the splits.  Model 1: Linear regressionDo the imports needed to build and evaluate a linear regression model that uses scaling.  TrainingMake a pipeline to scale and train in one step.**Hint: `with linear_model create LinearRegression using`** Fit the pipeline.  EvaluationGet the $r^2$ on the test splits (see linear regression notebooks).**Hint: `do score using a list containing Xtest and Ytest` (from splits)** ------------------------**QUESTION:**Is this a good $r^2$? **ANSWER: (click here to edit)** Print the coefficients of the model<jupyter_code>print(pd.DataFrame( {"variable":X.columns, "coefficient":np.ravel(std_clf[1].coef_) }).to_string())<jupyter_output>variable  coefficient
0      crim    -1.162316
1        zn     1.530809
2     indus     0.062541
3      chas     0.526039
4       nox    -1.941847
5        rm     2.283061
6       age    -0.042285
7       dis    -3.423062
8       rad     2.575558
9       tax    -2.146754
10  ptratio    -1.872294
11    lstat    -4.001713<jupyter_text>------------------------**QUESTION:**What are the two variables that most positively impact house price?What are the two variables that most negatively impact house price? **ANSWER: (click here to edit)**   Model 2: Lasso regression (alpha=.25)  TrainingMake a pipeline to scale and train in one step.**Hint: `with linear_model create Lasso using alpha=.25`****Alpha is different from C so bigger numbers mean bigger penalties.** Fit the pipeline.  <!-- predictions_lasso75 = std_clf_lasso75.predict(splits[1])predictions_lasso75std_clf_lasso75splitspredictions_lasso75std_clf_lasso75predictstd_clf_lasso75:predictGETFROM_STARTsplits2 -->  EvaluationGet the $r^2$ on the test splits. ------------------------**QUESTION:**How does this compare to the previous $r^2$? Should we be concerned? **ANSWER: (click here to edit)** Print how many coefficients are not zero. Print the coefficients of the model<jupyter_code>print(pd.DataFrame( {"variable":X.columns, "coefficient":np.ravel(std_clf_lasso25[1].coef_) }).to_string())<jupyter_output>variable  coefficient
0      crim    -0.579396
1        zn     0.696999
2     indus    -0.000000
3      chas     0.428924
4       nox    -0.820872
5        rm     2.586669
6       age    -0.000000
7       dis    -1.716571
8       rad     0.000000
9       tax    -0.000000
10  ptratio    -1.630620
11    lstat    -3.965615<jupyter_text>------------------------**QUESTION:**What are the two variables that most positively impact house price?What are the two variables that most negatively impact house price?How is this different from before? **ANSWER: (click here to edit)**  Model 3: Lasso regression (alpha=.75)  TrainingMake a pipeline to scale and train in one step, using alpha=.75. Fit the pipeline.  <!-- predictions_lasso75 = std_clf_lasso75.predict(splits[1])predictions_lasso75std_clf_lasso75splitspredictions_lasso75std_clf_lasso75predictstd_clf_lasso75:predictGETFROM_STARTsplits2 -->  EvaluationGet the $r^2$ on the test splits. ------------------------**QUESTION:**How does this compare to the previous $r^2$? Should we be concerned? **ANSWER: (click here to edit)** Print how many coefficients are not zero. Print the coefficients of the model<jupyter_code>print(pd.DataFrame( {"variable":X.columns, "coefficient":np.ravel(std_clf_lasso75[1].coef_) }).to_string())<jupyter_output>variable  coefficient
0      crim    -0.123574
1        zn     0.000000
2     indus    -0.000000
3      chas     0.075127
4       nox    -0.000000
5        rm     2.578374
6       age    -0.000000
7       dis    -0.000000
8       rad    -0.000000
9       tax    -0.000000
10  ptratio    -1.427197
11    lstat    -3.653555
{'fraction_non_alphanumeric': 0.11638141809290954, 'fraction_numerical': 0.06226568867155664, 'mean_word_length': 3.35486160397445, 'pattern_counts': {'":': 6, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '451323', 'license': 'Apache-2.0', 'n_tokens_mistral': 2018, 'n_tokens_neox': 1757, 'n_words': 715, 'path': 'Lasso-PS.ipynb', 'repo_name': 'memphis-iis/datawhys-intern-notebooks-2021', 'size': '6144'}