<jupyter_start><jupyter_text>House price predictionExtracted from this ref. [book](https://www.amazon.com.br/Hands-Machine-Learning-Scikit-Learn-TensorFlow/dp/1492032646/ref=asc_df_1492032646/?tag=googleshopp00-20&linkCode=df0&hvadid=379733272930&hvpos=&hvnetw=g&hvrand=5358833530171969532&hvpone=&hvptwo=&hvqmt=&hvdev=c&hvdvcmdl=&hvlocint=&hvlocphy=1001652&hvtargid=pla-523968811896&psc=1)Chapter 2 of Hands on Machine Learning With Scikit-Learn and Keras;   Download data<jupyter_code>import os
import tarfile
import urllib
from matplotlib import pyplot as plt

import ssl
ssl._create_default_https_context = ssl._create_unverified_context

DOWNLOAD_ROOT = "https://raw.githubusercontent.com/ageron/handson-ml2/master/"
HOUSING_PATH = os.path.join("datasets", "housing")
HOUSING_URL = DOWNLOAD_ROOT + "datasets/housing/housing.tgz"

def fetch_housing_data(housing_url=HOUSING_URL, housing_path=HOUSING_PATH):
    os.makedirs(housing_path, exist_ok=True)
    tgz_path = os.path.join(housing_path, "housing.tgz")
    urllib.request.urlretrieve(housing_url, tgz_path)
    housing_tgz = tarfile.open(tgz_path)
    housing_tgz.extractall(path=housing_path)
    housing_tgz.close()
    
fetch_housing_data()    
import pandas as pd

def load_housing_data(housing_path=HOUSING_PATH):
    csv_path = os.path.join(housing_path, "housing.csv")
    return pd.read_csv(csv_path)

df = load_housing_data()<jupyter_output><empty_output><jupyter_text>- Define goal.       * create a model that predict median house value<jupyter_code>df[:3]<jupyter_output><empty_output><jupyter_text>EDA ( exploratory data analysis)<jupyter_code>df.shape
df.info()
df.isnull().sum()
df['ocean_proximity'].value_counts()<jupyter_output><empty_output><jupyter_text>Lets look at other fields, `describe()` method give a summary for numerical attributes.<jupyter_code>df.describe()
df.hist(figsize=(14,8), bins=100)
df.hist(figsize=(14,8), bins=100, log=True)<jupyter_output><empty_output><jupyter_text>first thing ever in a dataset project is to split in train and test chunks.<jupyter_code>from sklearn.model_selection import train_test_split

train_set, test_set = train_test_split(df, test_size=0.3, random_state=10)
train_set.shape
train_set.isnull().sum()
train_set[train_set['total_bedrooms'].isnull()][:3]
train_set['total_bedrooms'].plot.hist(bins=100)
train_set['total_bedrooms'].median()
# fill null entry for total_badroom using the median values
# train_set['total_bedrooms'].fillna(train_set['total_bedrooms'].median())
import numpy as np

from sklearn.impute import SimpleImputer
preprocessor = SimpleImputer(missing_values=np.nan, strategy='median')
preprocessor.fit(train_set['total_bedrooms'].values.reshape(-1,1))

train_set['total_bedrooms_prep'] = preprocessor.transform(train_set['total_bedrooms'].values.reshape(-1,1))<jupyter_output><empty_output><jupyter_text>---- preprocessing, data normalization etc...* fit* transform * fit_transform---- models* fit* predict * fit_predict<jupyter_code># train_set[train_set['total_bedrooms'].isnull()]
# feature engineering to compute per houserolds and total_rooms

train_set["rooms_per_household"] = train_set["total_rooms"]/train_set["households"]
train_set["bedrooms_per_room"] = train_set["total_bedrooms_prep"]/train_set["total_rooms"]
train_set["population_per_household"]=train_set["population"]/train_set["households"]

# test data
# TODO, pack to pipeline using CombinedAttributesAdder
# test_set["rooms_per_household"] = test_set["total_rooms"]/test_set["households"]
# test_set["bedrooms_per_room"] = test_set["total_bedrooms_prep"]/test_set["total_rooms"]
# test_set["population_per_household"]=test_set["population"]/test_set["households"]
train_set['ocean_proximity'].value_counts()
# label encoder for mutually excludent categorical feature. 

from sklearn import preprocessing
le = preprocessing.LabelEncoder()
le.fit(train_set['ocean_proximity'])

train_set_num = train_set.drop(['total_bedrooms', 'ocean_proximity'], axis=1)
train_set_num[:3]
# standard scaler is transform this histogram to have mean value zero, and stardard deviation 1

standard_scaler = preprocessing.StandardScaler()

standard_scaler.fit(train_set_num)

train_set_scaled = standard_scaler.transform(train_set_num)

train_set_scaled = pd.DataFrame(train_set_scaled, columns=train_set_num.columns)

train_set_scaled['ocean_proximity'] = le.transform(train_set['ocean_proximity'])
train_set_scaled[:2]
y_train = train_set_scaled['median_house_value'].values
X_train = train_set_scaled.drop('median_house_value', axis=1).values

y_train.shape
X_train.shape
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import RandomForestRegressor

tree = DecisionTreeRegressor()
rfr = RandomForestRegressor()
tree.fit(X_train, y_train)
rfr.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>preparing testing dataset following the same steps we did for training. * fill null; * standard scaler with drop ['total_bedrooms', 'ocean_proximity'],* label encoder of ocean_proximity<jupyter_code># fill null values with median (preprocessor already fitted above)
test_set['total_bedrooms_prep'] = preprocessor.transform(test_set['total_bedrooms'].values.reshape(-1,1))

# create new features (feature engineering)
test_set["rooms_per_household"] = test_set["total_rooms"]/test_set["households"]
test_set["bedrooms_per_room"] = test_set["total_bedrooms_prep"]/test_set["total_rooms"]
test_set["population_per_household"]=test_set["population"]/test_set["households"]

# separate numerical attributes
test_set_num = test_set.drop(['total_bedrooms', 'ocean_proximity'], axis=1)

# perform standrad scaler with `standard_scaler` already fitted on train data
test_set_scaled = standard_scaler.transform(test_set_num)

# convert to dataframe for convenience
test_set_scaled = pd.DataFrame(test_set_scaled, columns=test_set_num.columns)

# recover categorical feature but with encoder already fitted in training data
test_set_scaled['ocean_proximity'] = le.transform(test_set['ocean_proximity'])

# create test feature and target
y_test = test_set_scaled['median_house_value'].values
X_test = test_set_scaled.drop('median_house_value', axis=1).values
y_test.shape, X_test.shape
from sklearn.metrics import mean_squared_error 

# y_pred = tree.predict(X_test)
y_pred = rfr.predict(X_test)

# squared = False to return the RMSE (root mean square error)
rms = mean_squared_error(y_true=y_test, y_pred=y_pred, squared=False)
rms
from matplotlib import pyplot as plt
plt.hist(y_pred, bins=100, label='predicted', alpha=0.3)
plt.hist(y_test, bins=100, label='true value', alpha=0.3)
plt.legend()
test_set_scaled_predicted = test_set_scaled.copy(deep=True)
test_set_scaled_predicted['median_house_value'] = y_pred
y_pred_not_scaled = pd.DataFrame(
    standard_scaler.inverse_transform(
        test_set_scaled_predicted.drop('ocean_proximity', axis=1)
    ), columns = test_set_num.columns)['median_house_value']
y_true_not_scaled = test_set['median_house_value']
rms_not_scaled = mean_squared_error(y_true=y_true_not_scaled, y_pred=y_pred_not_scaled, squared=False)
rms_not_scaled
y_pred_not_scaled[1]
# 68 % C.L
[y_pred_not_scaled[1] - rms_not_scaled, y_pred_not_scaled[1] + rms_not_scaled] 
test_set.iloc[1]['median_house_value']
_ = plt.hist(test_set['median_house_value'], bins=100, label='true', alpha=0.4)
_ = plt.hist(y_pred_not_scaled, bins=100, label='predicted', alpha=0.4)
plt.xlabel('median_house_value [USD]')
plt.legend()<jupyter_output><empty_output><jupyter_text>Goals for EDA; * observe duplicated datapoints (remove them); * null entry and treat accordingly; * outlier identification (consider remove outliers) - min, max scaler is a candidate method.* encode categorical features (one hot encoding or label encoding depending of feature nature)* scale numerical feature such that its mean value is zero and standard deviation is 1.<jupyter_code># Compose scikit learn pipeline;
# Cross-validation training;  
# Overfitting; 
# Grid search to tunne hyperparameter model; 
from sklearn.pipeline import Pipeline

pipe = Pipeline(
    # ('feature_compose', feature_compose) 
    ('replace_null', preprocessor), 
    ('label_encoder', le), 
    ('standard_scaler', standard_scaler), 
    ('regressor', DecisionTreeRegressor())
)

pipe.fit()
pipe.predict()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11466346153846153, 'fraction_numerical': 0.016826923076923076, 'mean_word_length': 8.307606263982104, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '363525', 'license': 'MIT', 'n_tokens_mistral': 3056, 'n_tokens_neox': 2873, 'n_words': 515, 'path': 'regression/housing.ipynb', 'repo_name': 'valdirsalustino/mentorship', 'size': '8320'}