<jupyter_start><jupyter_text>HousingEnd-to-end ML toy project from Chapter 2 of Hands-on machine learning with Scikit-learn, Keras & Tensorflow. (parts of the code below is copied from https://github.com/ageron/handson-ml2/blob/master/02_end_to_end_machine_learning_project.ipynb)-----Download data from an url.<jupyter_code>import os
import tarfile
import urllib

DOWNLOAD_ROOT = "https://raw.githubusercontent.com/ageron/handson-ml2/master/"
HOUSING_PATH = os.path.join("datasets", "housing")
HOUSING_URL = DOWNLOAD_ROOT + "datasets/housing/housing.tgz"

def fetch_housing_data(housing_url=HOUSING_URL, housing_path=HOUSING_PATH):
    if not os.path.isdir(housing_path):
        os.makedirs(housing_path)
    tgz_path = os.path.join(housing_path, "housing.tgz")
    urllib.request.urlretrieve(housing_url, tgz_path)
    housing_tgz = tarfile.open(tgz_path)
    housing_tgz.extractall(path=housing_path)
    housing_tgz.close()<jupyter_output><empty_output><jupyter_text>Importing data<jupyter_code>import pandas as pd

def load_housing_data(housing_path=HOUSING_PATH):
    csv_path = os.path.join(housing_path, "housing.csv")
    return pd.read_csv(csv_path)

housing = load_housing_data()
housing.head()
housing.info()
housing["ocean_proximity"].value_counts()
housing.describe()
%matplotlib inline 
import matplotlib as mpl
import matplotlib.pyplot as plt

housing.hist(bins=50, figsize=(20,15))
plt.show()
import numpy as np
from zlib import crc32

def split_train_test(data, test_ratio):
    shuffled_indices = np.random.permutation(len(data))
    test_set_size = int(len(data) * test_ratio)
    test_indices = shuffled_indices[:test_set_size]
    train_indices = shuffled_indices[test_set_size:]
    return data.iloc[train_indices], data.iloc[test_indices]

def test_set_check(identifier, test_ratio):
    return crc32(np.int64(identifier)) & 0xffffffff < test_ratio * 2* 32 

def split_train_test_by_id(data, test_ratio, id_column):
    ids = data[id_column]
    in_test_set = ids.apply(lambda id_:test_set_check(id_, test_ratio))
    return data.loc[~in_test_set], data.loc[in_test_set]

housing_with_id = housing.reset_index()
train_set, test_set = split_train_test_by_id(housing_with_id, 0.2, "index")<jupyter_output><empty_output><jupyter_text>Spliting a dataset into training and test sets can be also done using sklearn's function *train_test_split*<jupyter_code>from sklearn.model_selection import train_test_split 

#train_set, test_set = train_test_split(housing, test_size=0.2, random_state= 42)<jupyter_output><empty_output><jupyter_text>If the data set is large enough, it is generally satisfactory to split it into train and test sets randomly without introduction os significant biais. If the training set is not that large, we must use that the training set is enough representative of the target population. Sampling should also respect proportions.  For example, if expert say that median income is important to predict house prices, we want to obtain training an test test with enough representation of median income distribution.<jupyter_code>housing["income_cat"] = pd.cut(housing["median_income"], bins=[0., 1.5, 3.0, 4.5, 6, np.inf], labels=[1,2,3,4,5])
housing["income_cat"].hist()
from sklearn.model_selection import StratifiedShuffleSplit

split = StratifiedShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
for train_index, test_index in split.split(housing, housing["income_cat"]):
    strat_train_set = housing.loc[train_index]
    strat_test_set = housing.loc[test_index]

strat_test_set["income_cat"].value_counts()/len(strat_test_set)
for set_ in (strat_train_set, strat_test_set):
    set_.drop("income_cat", axis=1, inplace=True)<jupyter_output><empty_output><jupyter_text>Vizualization of data<jupyter_code>viz_housing = strat_train_set.copy()
viz_housing.plot(kind="scatter", x="longitude", y="latitude")
viz_housing.plot(kind="scatter", x="longitude", y="latitude", alpha=0.1)
viz_housing.plot(kind="scatter", x="longitude", y="latitude", alpha=0.4,
                 s=viz_housing["population"]/100, label="population", figsize=(10,7),
                 c=viz_housing["median_house_value"], cmap=plt.get_cmap("jet"), colorbar=True,
                )
plt.legend()
corr_matrix = viz_housing.corr()
corr_matrix["median_house_value"].sort_values(ascending=False)
from pandas.plotting import scatter_matrix

attributes = ["median_house_value", "median_income", "total_rooms", "housing_median_age"]

scatter_matrix(viz_housing[attributes], figsize=(12, 8))
viz_housing.plot(kind="scatter", x="median_income", y="median_house_value", alpha=0.05)<jupyter_output><empty_output><jupyter_text>You may notice the price cap at 500000$ and other straight less obvious straight lines: at 450000$ and maybe others arround 350000$ and 280000$. (you can reduce alpha in the scatter plot to see more clearly data, specially if the there is great concentration of data at certain regions)<jupyter_code>viz_housing["rooms_per_household"] = viz_housing["total_rooms"]/viz_housing["households"]
viz_housing["bedrooms_per_room"] = viz_housing["total_bedrooms"]/viz_housing["total_rooms"]
viz_housing["population_per_household"] = viz_housing["population"]/viz_housing["households"]
viz_housing["rooms_per_population"] = viz_housing["total_rooms"]/viz_housing["population"]
viz_housing["bedrooms_per_population"] = viz_housing["total_bedrooms"]/viz_housing["population"]

corr_matrix = viz_housing.corr()
corr_matrix["median_house_value"].sort_values(ascending=False)<jupyter_output><empty_output><jupyter_text>Here rooms per population and rooms per household are more predictive than number of rooms Data cleaningGetting fresh copies from training dataset<jupyter_code>housing = strat_train_set.drop("median_house_value", axis=1)
housing_labels = strat_train_set["median_house_value"].copy()<jupyter_output><empty_output><jupyter_text>Attribute "Total bedrooms" has missing values. This must be fixed by: * Dropping tuples (information is lost)* Not using "total bedrooms" at all (but maybe informative enough)* Filling missing values by some value (mean, median, custum value)  * It is important to store the used value to use it again for test set and in production. It is an attribute of the model to be produced. Let's use the mean.<jupyter_code>#median = housing["total_bedrooms"].median()
#housing["total_bedrooms"].fillna(median, inplace=True)

### The scikitlearn way

from sklearn.impute import SimpleImputer

imputer = SimpleImputer(strategy ="median")
#Get rid of non-numerical attributes

housing_num = housing.drop("ocean_proximity", axis=1)
imputer.fit(housing_num) # computes the median for all attributes of housing_num

X = imputer.transform(housing_num) #X is a Numpy array
#putting it in a pandas dataframe
housing_tr = pd.DataFrame(X, columns=housing_num.columns, index=housing_num.index)<jupyter_output><empty_output><jupyter_text>Handling text and categorical data<jupyter_code>housing_cat = housing[["ocean_proximity"]]
housing_cat["ocean_proximity"].unique()<jupyter_output><empty_output><jupyter_text>Categorical data as there is limited number possible values.<jupyter_code>from sklearn.preprocessing import OrdinalEncoder

ordinal_encoder = OrdinalEncoder()
housing_cat_encoded = ordinal_encoder.fit_transform(housing_cat)
housing_cat_encoded[:10]<jupyter_output><empty_output><jupyter_text>Problem: there is no notion of distance between categorical data here. 'Near Ocean' corresponding to 4 is not "closer" to 'Island' (2) than 'Inland" (1). This must not be used by ML algorithms and distances between categories as measure of distance between tuples. To fix this issue, a common solution is to use a binary attribute per category: adding attributes "Island", "Near Ocean" ... to the data set. Tuples having "Near Ocean" as category in the initial dataset will have 1 as value for attribute "Near Ocean" and 0 for others. It is called **One hot encoding**.<jupyter_code>from sklearn.preprocessing import OneHotEncoder

cat_encoder = OneHotEncoder()
housing_cat_1hot = cat_encoder.fit_transform(housing_cat)<jupyter_output><empty_output><jupyter_text>Custom transformers<jupyter_code>from sklearn.base import BaseEstimator, TransformerMixin

rooms_ix, bedrooms_ix, population_ix, households_ix = 3, 4, 5, 6

class CombinedAttributesAdder(BaseEstimator, TransformerMixin):
    def __init__(self, add_bedrooms_per_room=True): # no *args or **kargs
        self.add_bedrooms_per_room = add_bedrooms_per_room
    def fit(self, X, y=None):
        return self  # nothing else to do
    def transform(self, X):
        rooms_per_household = X[:, rooms_ix] / X[:, households_ix]
        population_per_household = X[:, population_ix] / X[:, households_ix]
        if self.add_bedrooms_per_room:
            bedrooms_per_room = X[:, bedrooms_ix] / X[:, rooms_ix]
            return np.c_[X, rooms_per_household, population_per_household,
                         bedrooms_per_room]
        else:
            return np.c_[X, rooms_per_household, population_per_household]

attr_adder = CombinedAttributesAdder(add_bedrooms_per_room=False)
housing_extra_attribs = attr_adder.transform(housing.values)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

num_pipeline = Pipeline([
    ('imputer', SimpleImputer(strategy="median")),
    ('attribs_adder', CombinedAttributesAdder()),
    ('std_scaler', StandardScaler())
])

housing_num_tr = num_pipeline.fit_transform(housing_num)
housing_num_tr
from sklearn.compose import ColumnTransformer

num_attribs = list(housing_num)
cat_attribs = ["ocean_proximity"]

full_pipeline = ColumnTransformer([
    ("num", num_pipeline, num_attribs),
    ("cat", OneHotEncoder(), cat_attribs)
])

housing_prepared = full_pipeline.fit_transform(housing)
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error

lin_reg = LinearRegression()
lin_reg.fit(housing_prepared, housing_labels)

some_data = housing.iloc[:5]
some_labels = housing_labels.iloc[:5]
some_data_prepared = full_pipeline.transform(some_data)

print("Prediction", lin_reg.predict(some_data_prepared))
print("Labels", list(some_labels))

housing_predictions_lin = lin_reg.predict(housing_prepared)
lin_mse = mean_squared_error(housing_labels, housing_predictions_lin)
lin_rmse = np.sqrt(lin_mse)
lin_rmse
from sklearn.tree import DecisionTreeRegressor

tree_reg = DecisionTreeRegressor()
tree_reg.fit(housing_prepared, housing_labels)

housing_predictions_tree = tree_reg.predict(housing_prepared)
tree_mse = mean_squared_error(housing_labels, housing_predictions_tree)
tree_rmse = np.sqrt(tree_mse)
tree_mse
from sklearn.model_selection import cross_val_score

scores = cross_val_score(tree_reg, housing_prepared, housing_labels, scoring="neg_mean_squared_error", cv=10) #cross validation expects an utility function = - cost function
tree_rmse_scores = np.sqrt(-scores)

def display_scores(scores):
    print("Scores: ", scores)
    print("Mean: ", scores.mean())
    print("Standard deviation: ", scores.std())

display_scores(tree_rmse_scores)
lin_scores = cross_val_score(lin_reg, housing_prepared, housing_labels, scoring="neg_mean_squared_error", cv=10)
lin_rmse_scores = np.sqrt(-lin_scores)

display_scores(lin_rmse_scores)
from sklearn.ensemble import RandomForestRegressor

forest_reg = RandomForestRegressor()
forest_reg.fit(housing_prepared, housing_labels)

forest_scores = cross_val_score(forest_reg, housing_prepared, housing_labels, scoring="neg_mean_squared_error", cv=10)
forest_rmse = np.sqrt(-forest_scores)

display_scores(forest_rmse)
from sklearn.model_selection import GridSearchCV

param_grid = [
    {'n_estimators': [3,10,30], 'max_features': [2,4,6,8]},
    {'bootstrap': [False], "n_estimators": [3,10], "max_features":[2,3,4]}
]

grid_search = GridSearchCV(forest_reg, param_grid, cv=5, scoring="neg_mean_squared_error", return_train_score=True)
grid_search.fit(housing_prepared, housing_labels)
cvres = grid_search.cv_results_
for mean_score, params in zip(cvres["mean_test_score"], cvres["params"]):
    print(np.sqrt(-mean_score), params)
feature_importances = grid_search.best_estimator_.feature_importances_
extra_attribs = ["rooms_per_hhold", "pop_per_hhold", "bedrooms_per_room"]
cat_encoder = full_pipeline.named_transformers_["cat"]
cat_one_hot_attribs = list(cat_encoder.categories_[0])
attributes = num_attribs + extra_attribs + cat_one_hot_attribs
sorted(zip(feature_importances, attributes), reverse = True)
final_model = grid_search.best_estimator_

X_test = strat_test_set.drop("median_house_value", axis=1)
Y_test = strat_test_set["median_house_value"].copy()

X_test_prepared = full_pipeline.transform(X_test)

final_predictions = final_model.predict(X_test_prepared)

final_mse = mean_squared_error(Y_test, final_predictions)
final_rmse = np.sqrt(final_mse)
final_rmse
from scipy import stats

confidence = 0.95
squared_errors = (final_predictions - Y_test)**2
np.sqrt(stats.t.interval(confidence, len(squared_errors) - 1, loc=squared_errors.mean(), scale = stats.sem(squared_errors)))
 
from sklearn.svm import SVR
from sklearn.model_selection import RandomizedSearchCV

param_grid = [
    {'kernel': ['linear'], 'C': [10, 30, 40, 50, 100, 1000]},
    {'kernel': ['rbf'], 'C': [10, 30, 40, 50, 100, 1000], 'gamma' : [0.01, 0.02, 0.1, 0.5]}
]

svm_reg = SVR()

grid_search = GridSearchCV(svm_reg, param_grid, cv=5, scoring="neg_mean_squared_error", verbose=2)
grid_search.fit(housing_prepared, housing_labels)
grid_search.best_score_
grid_search.best_params_<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10473521168313911, 'fraction_numerical': 0.012759994099424694, 'mean_word_length': 7.032582938388626, 'pattern_counts': {'":': 2, '<': 50, '<?xml version=': 0, '>': 49, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '378277', 'license': 'Apache-2.0', 'n_tokens_mistral': 4783, 'n_tokens_neox': 4507, 'n_words': 976, 'path': 'books-notes/Hands-On_Machine_Learning_with_Scikit-Learn_Keras_and_Tensorflow_Aurelien_Geron/Code/housing.ipynb', 'repo_name': 'AElOuassouli/reading-notes', 'size': '13558'}