<jupyter_start><jupyter_text>Author: <NAME>In this project, we first explored our data. We used a scatter matrix and a correlation matrix to determine the feature that describes most of the variability in MedHouseVal. We found that MedInc has a correlation value of positive .68 in relation to MedHouseVal. This model based on MedInc scored better than the model that we randomly picked (AveBedrms). Afterwards, we created a model based on all input features and also the polynomial features. This proved to be a better model. To create a model that penalized large values for the coefficients, we used L2 regularization. Again, this intermediate model performed better than all of the previous models.<jupyter_code>import sklearn.datasets
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
from sklearn.preprocessing import PolynomialFeatures
import pandas as pd
import numpy as np
import warnings
from sklearn.linear_model import Ridge
warnings.simplefilter(action='ignore', category=FutureWarning)
warnings.filterwarnings("ignore")
%matplotlib inline<jupyter_output><empty_output><jupyter_text>***Question 1***<jupyter_code>data = sklearn.datasets.fetch_california_housing(as_frame = True)
data.frame.head()<jupyter_output><empty_output><jupyter_text>***Question 2***<jupyter_code>y = data.frame['MedHouseVal']
X = data.frame.iloc[:,0:8]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=(2021-3-25))<jupyter_output><empty_output><jupyter_text>***Question 3***<jupyter_code>reg = LinearRegression().fit(X_train['AveBedrms'][:,None], y_train)
pred_train = reg.predict(X_train['AveBedrms'][:,None])
MSE_train = mean_squared_error(pred_train, y_train)
MSE_train
pred_test = reg.predict(X_test['AveBedrms'][:,None])
MSE_test = mean_squared_error(pred_test, y_test)
MSE_test<jupyter_output><empty_output><jupyter_text>***Question 4***<jupyter_code>reg.score(X_train['AveBedrms'][:,None], y_train)
reg.score(X_test['AveBedrms'][:,None], y_test)<jupyter_output><empty_output><jupyter_text>Given the R2 scores, how well did our model do?Neither of our models are very good, sense the training model is only .2% and should be closer to 1, while the test score is -.1% which is also a very poor score ***Question 5***<jupyter_code>pd.plotting.scatter_matrix(data.frame, alpha=0.2, figsize = (10,10));<jupyter_output><empty_output><jupyter_text>***Question 6***<jupyter_code>pd.plotting.scatter_matrix(data.frame.sample(frac = 0.01), alpha=0.2, figsize = (10,10));<jupyter_output><empty_output><jupyter_text>How does the AveBedrms feature appear to affect MedHouseVal? The average bedrooms appears to not affect the median house value, as the data suggest that the majority of housing only has 1 bedroom and the value varries greatly with just the one bedroom.How does this explain the value of R2 you found in experiment (4)?Since the majority of the housing only has one bedroom this would explain why the test set and training set did not preform well as the median housing value varies greatly with just one bedroom ***Question 7*** Which features seem to have the strongest linear relationship with MedHouseVal?Median Income seems to have the strongest linear relationship with Median house value, as people make more money they tend to have a higher value property ***Question 8***What relationship do you see between MedHouseVal and the Latitude and Longitude features?Based on a map and using the data collected the value of housing is highest near the coast in areas near San fransico and the area between / near Los Angeles. The area inland and further North of California have decreased housing values. ***Question 9***<jupyter_code>df = pd.DataFrame(data.frame)
df.corr(method = 'pearson')<jupyter_output><empty_output><jupyter_text>Based on the correlation matrix, which feature is the strongest predictor of MedHouseVal? Does this match your answer in experiment (7)?Median Income seems to have the strongest predictor of median house value as it is 68.9% which matches what we thought would occur in question 7 ***Question 10***<jupyter_code>reg = LinearRegression().fit(X_train['MedInc'][:,None], y_train)
reg.score(X_train['MedInc'][:,None], y_train)
reg.score(X_test['MedInc'][:,None], y_test)<jupyter_output><empty_output><jupyter_text>How well does this model score?These scores fit better, the training score is at 46.8% while the test score is at 49.5%. Much better compared to the .2% and -.1% we were getting previously. ***Question 11***<jupyter_code>reg = LinearRegression().fit(X_train, y_train)
pred_train = reg.predict(X_train)
MSE_train = mean_squared_error(pred_train, y_train)
print(MSE_train)

pred_test = reg.predict(X_test)
MSE_train = mean_squared_error(pred_test, y_test)
print(MSE_train)<jupyter_output>0.5296444457890024
0.5047767239631318<jupyter_text>How do the training and test MSEs for this model compare to that experiment?Both the training and test MSE's for this model fit much better when compared to the experiment in question 4. ***Question 12***<jupyter_code>poly = PolynomialFeatures(2)
poly_x_train = poly.fit_transform(X_train)
reg.fit(poly_x_train,y_train)
y_pred = reg.predict(poly_x_train)
mse = mean_squared_error(y_train, y_pred)
print (mse)

poly_x_test = poly.fit_transform(X_test)
y_pred = reg.predict(poly_x_test)
mse = mean_squared_error(y_test, y_pred)
print (mse)<jupyter_output>0.42433222517444946
0.6463802847593988<jupyter_text>No, adding polynomial terms does not lead to overfitting ***Question 13***<jupyter_code>ridge = Ridge(alpha = 1, normalize = True)
ridge.fit(poly_x_train, y_train)
y_pred = ridge.predict(poly_x_train)
mse = mean_squared_error(y_train, y_pred)
print (mse)

y_pred = ridge.predict(poly_x_test)
mse = mean_squared_error(y_test, y_pred)
print (mse)<jupyter_output>0.6457457247916449
0.6190711914117147
{'fraction_non_alphanumeric': 0.09204526262455666, 'fraction_numerical': 0.02904914710352981, 'mean_word_length': 7.1570247933884295, 'pattern_counts': {'":': 0, '<': 47, '<?xml version=': 0, '>': 47, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '103478', 'license': 'MIT', 'n_tokens_mistral': 2007, 'n_tokens_neox': 1852, 'n_words': 606, 'path': 'machineLearning/correlationAndRegularization.ipynb', 'repo_name': 'naokishami/Classwork', 'size': '5927'}