<jupyter_start><jupyter_text>Import libraries which is needed<jupyter_code>import matplotlib.pyplot as plt
import numpy as np
from sklearn import datasets
from sklearn import metrics
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeRegressor
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.preprocessing import PolynomialFeatures<jupyter_output><empty_output><jupyter_text>Preparing the house dataLoad california house price and show it<jupyter_code>california = datasets.fetch_california_housing(as_frame=True)
print(california.DESCR)
california.frame.hist(figsize=(10, 10), bins=50)<jupyter_output>.. _california_housing_dataset:

California Housing dataset
--------------------------

**Data Set Characteristics:**

    :Number of Instances: 20640

    :Number of Attributes: 8 numeric, predictive attributes and the target

    :Attribute Information:
        - MedInc        median income in block
        - HouseAge      median house age in block
        - AveRooms      average number of rooms
        - AveBedrms     average number of bedrooms
        - Population    block population
        - AveOccup      average house occupancy
        - Latitude      house block latitude
        - Longitude     house block longitude

    :Missing Attribute Values: None

This dataset was obtained from the StatLib repository.
http://lib.stat.cmu.edu/datasets/

The target variable is the median house value for California districts.

This dataset was derived from the 1990 U.S. census, using one row per census
block group. A block group is the smallest geographical unit for which the U.S.
Census Bur[...]<jupyter_text>we could find in MedHouseVal graph,when x = 5 it has some exceptional distributions.Those data are harmful to model training, we should delete they.<jupyter_code>x = california.data
# print(type(x))
# print(x)
y = california.target
# print(type(y))
# print(y)

# plt.scatter(x,y)
# plt.show()
x = x[y < 5]
y = y[y < 5]
# plt.scatter(x,y)
# plt.show()
print(x.shape)
print(y.shape)<jupyter_output>(19648, 8)
(19648,)<jupyter_text>60% training, 20% validation, 20% test<jupyter_code>x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2)
x_train, x_val, y_train, y_val = train_test_split(x_train, y_train, test_size=0.25)

print('x_train shape is ' + str(x_train.shape))
print('y_train shape is ' + str(y_train.shape))
print('x_val shape is ' + str(x_val.shape))
print('y_val shape is ' + str(y_val.shape))
print('x_test shape is ' + str(x_test.shape))
print('y_test shape is ' + str(y_test.shape))<jupyter_output>x_train shape is (11788, 8)
y_train shape is (11788,)
x_val shape is (3930, 8)
y_val shape is (3930,)
x_test shape is (3930, 8)
y_test shape is (3930,)<jupyter_text>Standardize the data to makes sure the value of each attribute provided by all sampleshave a mean of 0, and a standard deviation (s) of 1.<jupyter_code>scaler = StandardScaler()
x_train = scaler.fit_transform(x_train)
x_val = scaler.transform(x_val)
x_test = scaler.transform(x_test)<jupyter_output><empty_output><jupyter_text>Train and predict the model**Linear regression model** and calculate the *score*.<jupyter_code>linear_regression = LinearRegression()
linear_regression.fit(x_train, y_train)
y_pred_val = linear_regression.predict(x_val)
linear_score=linear_regression.score(x_val, y_val)
lin_mse = metrics.mean_squared_error(y_test,y_pred_val)
lin_rmse = np.sqrt(lin_mse)
print('Linear Regression score is: ',end='')
print(linear_score)<jupyter_output>Linear Regression score is: 0.5852500512809189<jupyter_text>when we get linear regression,it easy to implement polynomial regression**polynomial regression** and calculate the score.we using degree from 2 to 4Since polynomial regression need hyperparameters, we calculate the score on the validation set to find the best degree for the data.<jupyter_code>scores = []
#degree = 2
polynomial_regression = Pipeline([('poly', PolynomialFeatures(degree=2)),
                                  ('sceler', StandardScaler()),
                                  ('linear_regression', LinearRegression())])
polynomial_regression.fit(x_train, y_train)
scores.append(polynomial_regression.score(x_val, y_val))
#degree = 3
polynomial_regression = Pipeline([('poly', PolynomialFeatures(degree=3)),
                                  ('scaler', StandardScaler()),
                                  ('linear_regression', LinearRegression())])
polynomial_regression.fit(x_train, y_train)
scores.append(polynomial_regression.score(x_val, y_val))
#degree = 4
polynomial_regression = Pipeline([('poly', PolynomialFeatures(degree=4)),
                                  ('scalar', StandardScaler()),
                                  ('linear_regression', LinearRegression())])
polynomial_regression.fit(x_train, y_train)
scores.append(polynomial_regression.score(x_val, y_val))

print("degree = 2 polynomial_regression score :"+ str(scores[0]))
print("degree = 3 polynomial_regression score :"+ str(scores[1]))
print("degree = 4 polynomial_regression score :"+ str(scores[2]))<jupyter_output>degree = 2 polynomial_regression score :0.5628172364162672
degree = 3 polynomial_regression score :-5.517946792026234
degree = 4 polynomial_regression score :-1267774.7940212395<jupyter_text>**decision tree regression** and calculate the *score*.<jupyter_code>tree_regression = DecisionTreeRegressor(random_state=11)
tree_regression.fit(x_train, y_train)
y_pred_val = tree_regression.predict(x_val)
print("tree regression score: ",end= ' ')
print(tree_regression.score(x_val, y_val))<jupyter_output>tree regression score:  0.5139385793412374<jupyter_text>We can see that the linear regression has the best score. So we make predictions on the test set with such model.<jupyter_code>linear_regression = LinearRegression()
linear_regression.fit(x_train, y_train)
y_pred_test = linear_regression.predict(x_test)
linear_score=linear_regression.score(x_test, y_test)
print('linear_regression '+str(linear_regression))
print('y_train.mean() '+str(y_train.mean()))
print('y_pred_test.mean() '+str(y_pred_test.mean()))
print('y_train.var() '+str(y_train.var()))
print('y_pred_val.var() '+ str(y_pred_val.var()))<jupyter_output>linear_regression LinearRegression()
y_train.mean() 1.9211033237190365
y_pred_test.mean() 1.927213978414835
y_train.var() 0.9492883866118585
y_pred_val.var() 0.962607563471761
{'fraction_non_alphanumeric': 0.10600216016046907, 'fraction_numerical': 0.03718561950316309, 'mean_word_length': 5.10933081998115, 'pattern_counts': {'":': 0, '<': 32, '<?xml version=': 0, '>': 30, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '330982', 'license': 'MIT', 'n_tokens_mistral': 2286, 'n_tokens_neox': 2071, 'n_words': 520, 'path': '11912725_Lab4.ipynb', 'repo_name': 'Uncle-Road/CS405_works', 'size': '6481'}