<jupyter_start><jupyter_text>Problem Set 5 (covers Lab 9 and 10) Directions1. Show all work/steps/calculations using a combination of code and Markdown. You will submit only this notebook named: [jhed_id].ipynb back to Blackboard from whence you downloaded it. Do not return the data to me.2. **All** work is to be your own. This is not a group project. You may, however, use code from the lectures and labs. Provide citations for any code that is not your own. You may also consult Stackoverflow, etc. This is not by any means "closed book" or anything like that. Basically, I need to see if *you* learned the concepts from the Lectures, *Fundamentals*, and Labs.3. Add whatever markdown or code cells you need in each part to explain and calculate your answers. Don't just provide answers but explain them as well. **Explain and interpret your results.**  IntroductionLabs 9 and 10 were all about linear models, specifically linear and logistic regression. We learned how to build and interpret multivariate models by using domain knowledge, comparing the predicted and actual values of the coefficients and the credible intervals, and looking at residuals. We also applied transformations to make the models perform better and increase interpretability.Using the techniques covered during Modules 9 and 10 (lectures, labs, course notes), **build, describe, and use** a linear regression model that estimates the target variable `mpg` for the accompanying auto data set. You may use other techniques you know but you must at least use the techniques described in class and call out and describe anything "extra" that you do and why.Do **not** use any feature selection techniques or anything other than linear regression. Follow the method described in class.The instructions say build, **describe, and use**. You must describe all aspects of your final model (all the parameters and metrics) and use it to predict one value as described in the course notes.  Import the data1. import the data from the txt file2. replace the NA value with average<jupyter_code>import pandas as pd
data = pd.read_csv("data.csv")
data.head()<jupyter_output><empty_output><jupyter_text>Find NA     Replace NA with Average<jupyter_code>data[data.isnull().any(axis=1)]
data = data.fillna(data.mean())<jupyter_output><empty_output><jupyter_text>mpg1. The min of mpg is bigger than 0.2. MPG does not follows the normal distribution, but it has positive skewness and a low kurtosis. Therefore we need to standardize mpg<jupyter_code>data['mpg'].describe()
sns.distplot(data['mpg']);
print("Skewness: %f" % data['mpg'].skew())
print("Kurtosis: %f" % data['mpg'].kurt())<jupyter_output>Skewness: 0.457066
Kurtosis: -0.510781<jupyter_text>Create Dummy variables for the categorical features: origin, year<jupyter_code>data = pd.concat([data, pd.get_dummies(data["modelyear"],prefix="modelyear")], axis=1)
data = pd.concat([data, pd.get_dummies(data["origin"], prefix="origin")], axis=1)

data.head()
data1 = data.loc[:, data.columns != 'car name'] 
corr = data1.drop("mpg", axis=1).apply(lambda x: x.corr(data1.mpg))
corr.sort_values(ascending=True)<jupyter_output><empty_output><jupyter_text>1. "weight","displacement", "cylinders" and "horsepower" have negative relationships with mpg 2. "acceleration" has a positive relationship with mpg3. The mpg from "origin 1" is lower than the mpg from "origin 2" and "origin 3". 4. The model years: also related to the mpg  Models: Method 1<jupyter_code>import models

data3=data1

model = "mpg ~ weight+displacement+cylinders+horsepower+origin_1+origin_2+origin_3+modelyear+acceleration"
result1 = models.bootstrap_linear_regression(model, data=data3)
models.describe_bootstrap_lr(result1)<jupyter_output><empty_output><jupyter_text>The R^2 is 85%, and the error is only 3.06. However, as the mpg does not follow a normal distribution, there maybe heteroscedasticity issues.  1. origin 2 and origin 3 have a positive relationship with mpg.  2. weight, displacement, cylinders, acceleration and horsepower have no significant relationship with mpg. This is quite a big surprise. 3. We can remove weight, horsepower<jupyter_code>model = "mpg ~ displacement+cylinders+origin_1+origin_2+origin_3+modelyear+acceleration"
result1 = models.bootstrap_linear_regression(model, data=data3)
models.describe_bootstrap_lr(result1)<jupyter_output><empty_output><jupyter_text>Predicty = -13.77 - 0.04*displacement -0.66*cylinders -6.01*origin_1 -4.4*origin_2 -3.35*origin_3 +0.72*modelyear -0.11*acceleration   Method 21. mpg has positive skewness. To overcome the skewness issue of mpg, we can standardize it<jupyter_code>to_remove = ["mpg","origin","modelyear","car name"]
data_vars=data.columns.values.tolist()
to_keep = [i for i in data if i not in to_remove]
data2=data1[to_keep]
y = data1["mpg"]

print("Skewness: %f" % y.skew())
y = np.log1p(y)
print("Skewness after log+1: %f" % y.skew())<jupyter_output>Skewness: 0.457066
Skewness after log+1: -0.109138<jupyter_text>2. Weight, displacement, cylinders, acceleration and horsepower have no significant relationship with mpg. This maybe a problem of multicollinearity. We can try the LassoCV regularization.<jupyter_code>from sklearn.linear_model import LinearRegression
from sklearn.linear_model import Ridge, RidgeCV, ElasticNet, LassoCV, LassoLarsCV
from sklearn.model_selection import cross_val_score

def rmse_cv(model):
    rmse= np.sqrt(-cross_val_score(model, data2, y, scoring="neg_mean_squared_error", cv = 5))
    return(rmse)

model_ridge = Ridge()

model_lasso = LassoCV(alphas = [0.001]).fit(data2, y)

min(rmse_cv(model_lasso))
from sklearn.metrics import r2_score
print("R^2: ",r2_score(y, model_lasso.predict(data2), multioutput='variance_weighted'))<jupyter_output>R^2:  0.8968166947919683
{'fraction_non_alphanumeric': 0.09483356449375867, 'fraction_numerical': 0.02617891816920943, 'mean_word_length': 6.0097205346294045, 'pattern_counts': {'":': 0, '<': 30, '<?xml version=': 0, '>': 30, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '54081', 'license': 'MIT', 'n_tokens_mistral': 1891, 'n_tokens_neox': 1763, 'n_words': 649, 'path': 'problem5/yxu29.ipynb', 'repo_name': 'topseer/APL_Great', 'size': '5768'}