<jupyter_start><jupyter_text>Imputer is a type of "Estimator"An 'estimator' estimates some 'parameter' in our case the medianbased on a dataset(s) depending on supervised or notEstimation is done by calling 'fit()'Estimators can have hyperparameters such as the case for Imputers where the hyperparameter is 'strategy' A subset of 'estimators' are 'transformers' These estimators can transform a dataset typically based on some learned parameters (it previously learned by calling fit())'transform()' returns a dataset with the transformation applied.fit_transform() is a convenient function that first calls fit() and immediately transform(); Sometimes optimized to run much faster Another subset of 'estimators' is 'predictors'. These estimators make predictions based on a given dataset. Eg. LinearRegression predictor in chapter 1 predicted the life satisfaction based on GDP. predictors also have a 'score()' function to compare how well it predicted when given ground truth.  All estimator hyperparameters are accessible via public function. eg. Imputer.strategy to get 'median'  Composition Arbitrary etimators can be added to a 'Pipeline' estimator that performs all included estimator functions in one go  Dealing with non numerical attribute<jupyter_code>housing_cat = housing["ocean_proximity"]
housing_cat.head(10)
type(housing_cat)
housing_cat_encoded, housing_categories = housing_cat.factorize()
housing_cat_encoded[:10]
housing_categories.data
housing_categories<jupyter_output><empty_output><jupyter_text>Encoding categories in this way suggests to the ML algo that categories close together are more similar than categores that are distant. Eg category 0 and 4 are actually more similar than category 0 and 2.An alternative approach is to introduce a new binary attribute for each category.<jupyter_code>from sklearn.preprocessing import OneHotEncoder
encoder = OneHotEncoder()
housing_cat_1hot = encoder.fit_transform(housing_cat_encoded.reshape(-1,1))
housing_cat_1hot
housing_cat_1hot.toarray()<jupyter_output><empty_output><jupyter_text>What we've done so far is convert string -> integer encoding -> 1hot encoding.This can be done in a single step using sklearn's CategoricalEncoding.<jupyter_code>housing_cat_1hotarray = housing_cat_1hot.toarray()<jupyter_output><empty_output><jupyter_text>Custom Transformer<jupyter_code>from sklearn.base import BaseEstimator, TransformerMixin

#Specifies the index for these attributes
rooms_ix, bedrooms_ix, population_ix, household_ix = 3,4,5,6

#BaseEstimator gives get_param() and set_param() for auto hyperparameter tuning
#TransformerMixin gives fit_transform()



class CombinedAttributesAdder(BaseEstimator, TransformerMixin):
    def __init__(self, add_bedrooms_per_room = True): #no *args, or **kwargs
        self.add_bedrooms_per_room = add_bedrooms_per_room
        
    def fit(self, X, y=None):
        return self #nothing else to do
    
    def transform(self, X, y=None):
        rooms_per_household = X[:, rooms_ix] /X[:, household_ix]
        population_per_household = X[:, population_ix] /X[:, household_ix]
        if self.add_bedrooms_per_room:
            bedrooms_per_room = X[:, bedrooms_ix]/X[:,rooms_ix]
            return np.c_[X, rooms_per_household, population_per_household,
                         bedrooms_per_room]
        else:
            return np.c_[X, rooms_per_household, population_per_household]
        
attr_adder = CombinedAttributesAdder(add_bedrooms_per_room = False)
housing_extra_attribs = attr_adder.transform(housing.values)

housing_extra_attribs<jupyter_output><empty_output><jupyter_text>Using Pipelines<jupyter_code>from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler #For standardization

numerical_pipeline = Pipeline([
    ('imputer', Imputer(strategy="median")), #Adds median values for missing values
    ('attribs_adder', CombinedAttributesAdder()), #Adds combined attributes
    ('std_scaler', StandardScaler()) #standardization (0 mean, 1 variance)
])

housing_num_tr = numerical_pipeline.fit_transform(housing_num)<jupyter_output><empty_output><jupyter_text>The pipeline above takes in as input a Dataframe that only has numerical values. This requires us to preprocess the dataframe to remove non numerical values. Let's write an estimator that allows us to select which attributes to keep<jupyter_code>from sklearn.base import BaseEstimator, TransformerMixin

class DataFrameSelector(BaseEstimator, TransformerMixin):
    def __init__(self, attribute_names):
        self.attribute_names = attribute_names
    def fit(self, X, y=None):
        return self
    def transform(self, X):
        return X[self.attribute_names].values<jupyter_output><empty_output><jupyter_text>Now we can add this dataframe to the pipeline<jupyter_code>numerical_attribs = list(housing_num)
numerical_attribs

cat_attribs = ["ocean_proximity"]
from sklearn.preprocessing import LabelEncoder

class myLabelEncoder(BaseEstimator, TransformerMixin):
    def __init__(self, *args, **kwargs):
        #self.encoder = LabelEncoder(*args, **kwargs)
        pass
    def fit(self, x, y=0):
        #self.encoder.fit(x)
        return self
    def transform(self, x, y=0):
        new_x = list()
        for label in x:
            new_x.append(label[0])
        x_encoded, x_categories = pd.Series(new_x).factorize()
        print('here')
        print(x_categories)
        #print(x_encoded)
        return x_encoded.reshape(-1,1)<jupyter_output><empty_output><jupyter_text>"""class myOneHotEncoder(BaseEstimator, TransformerMixin):    def __init__(self, *args, **kwargs):        self.encoder = OneHotEncoder(*args, **kwargs)    def fit(self, x, y=0):        self.encoder.fit(x)        return self    def transform(self, x, y=0):        return self.encoder.transform(x)"""<jupyter_code>numerical_pipeline = Pipeline([
    ('selector', DataFrameSelector(numerical_attribs)),
    ('imputer', Imputer(strategy="median")), #Adds median values for missing values
    ('attribs_adder', CombinedAttributesAdder()), #Adds combined attributes
    ('std_scaler', StandardScaler()) #standardization (0 mean, 1 variance)
])

from sklearn.preprocessing import LabelEncoder
cat_pipeline = Pipeline([
    ('selector', DataFrameSelector(cat_attribs)),
    ('classToIntegerEncoding', myLabelEncoder()),
    ('onehotencoder', OneHotEncoder()) #Returns a sparse matrix,
    #Still need to convert to array
])<jupyter_output><empty_output><jupyter_text>To combine these pipelines together, we can use Feature Union<jupyter_code>from sklearn.pipeline import FeatureUnion

full_pipeline = FeatureUnion(transformer_list=[
    ("numerical_pipe",numerical_pipeline),
    ("categorical_pipe", cat_pipeline)
])
housing_prepared = full_pipeline.fit_transform(housing)
housing_prepared = housing_prepared.toarray()
housing_prepared.shape<jupyter_output><empty_output><jupyter_text>Training and Evaluating on a Training Set<jupyter_code>from sklearn.linear_model import LinearRegression

lin_reg = LinearRegression()
lin_reg.fit(housing_prepared, housing_labels)

some_data = housing.iloc[:5]
some_labels = housing_labels.iloc[:5]
some_data_prepared = full_pipeline.transform(some_data)

print("predictions:", lin_reg.predict(some_data_prepared))
print("Labels:", list(some_labels))
from sklearn.metrics import mean_squared_error
housing_predictions = lin_reg.predict(housing_prepared)
lin_mse = mean_squared_error(housing_labels, housing_predictions)
lin_rmse = np.sqrt(lin_mse)
lin_rmse
from sklearn.tree import DecisionTreeRegressor

tree_reg = DecisionTreeRegressor()
tree_reg.fit(housing_prepared, housing_labels)
prediction = tree_reg.predict(housing_prepared)
tree_mse = mean_squared_error(housing_labels, prediction)
tree_rmse = np.sqrt(tree_mse)
tree_rmse<jupyter_output><empty_output><jupyter_text>Cross Validation<jupyter_code>from sklearn.model_selection import cross_val_score
scores = cross_val_score(tree_reg, housing_prepared, housing_labels,
                         scoring="neg_mean_squared_error", cv=10)
tree_rmse_scores = np.sqrt(-scores)
def display_scores(scores):
    print("Scores: ", scores)
    print("Mean: ", scores.mean())
    print("Std: ", scores.std())
    
display_scores(tree_rmse_scores)
lin_scores = cross_val_score(lin_reg, housing_prepared, housing_labels,
                        scoring="neg_mean_squared_error", cv=10)

lin_rmse_scores = np.sqrt(-lin_scores)
display_scores(lin_rmse_scores)<jupyter_output>Scores:  [66745.5479951  66963.32758364 70358.88926216 74775.08156024
 68031.13388938 71206.36898529 64966.50575515 68285.02702705
 71562.04635394 67665.10082067]
Mean:  69055.902923263
Std:  2744.6411279702475<jupyter_text>So basically, decision tree regression was overfitting so badly on the data that it performs worse than linear regression.Let's try another model: Random Forest Regression.Random forest trains a bunch of decision trees using randomly selected subset of features. Their predictions are then averaged out (since it's regression, it's literally average operator. For classification, it may be a voting scheme)<jupyter_code>from sklearn.ensemble import RandomForestRegressor

forest_reg = RandomForestRegressor()
forest_reg.fit(housing_prepared, housing_labels)

forest_scores = cross_val_score(forest_reg, housing_prepared, housing_labels,
                               scoring = "neg_mean_squared_error", cv=10)

forest_rmse_scores = np.sqrt(-forest_scores)
display_scores(forest_rmse_scores)<jupyter_output>Scores:  [52922.64411841 50735.01539527 52172.63465399 54963.28400018
 52723.12558121 56355.57315513 51419.08402792 52074.83647482
 54614.60663946 52487.95172485]
Mean:  53046.87557712372
Std:  1649.669209706007<jupyter_text>Random forest seems to be the best out of all of the ones we've tried. Before rushing into fine tuning hyperparameters/ adding regularization, It's advised to try many more models eg. SVMs, Neural nets, logistic regression, K means etc.Short list the most promising 2-5 models and save their scores, trained parameters, hyperparameters, to easily compare later on.  Saving Models<jupyter_code>from sklearn.externals import joblib

#joblib.dump(my_model, "my_model.pkl")
#my_model_loaded = joblib.load("my_model.pkl")<jupyter_output><empty_output><jupyter_text>Fine Tuning Duh Model<jupyter_code>from sklearn.model_selection import GridSearchCV

param_grid = [
    {'n_estimators': [3,10,30], 'max_features': [2,4,6,8]},
    {'bootstrap': [False], 'n_estimators': [3,10], 'max_features': [2,3,4]}
]

forest_reg = RandomForestRegressor()

grid_search = GridSearchCV(forest_reg, param_grid, cv=5, 
                          scoring='neg_mean_squared_error')

grid_search.fit(housing_prepared, housing_labels)
grid_search.best_params_
grid_search.best_estimator_
cv_result = grid_search.cv_results_
for mean_score, params in zip(cv_result['mean_test_score'],
                              cv_result['params']):
    print(np.sqrt(-mean_score), params)<jupyter_output>63734.99032402944 {'n_estimators': 3, 'max_features': 2}
56031.36867459153 {'n_estimators': 10, 'max_features': 2}
52920.45101361692 {'n_estimators': 30, 'max_features': 2}
59831.06799484957 {'n_estimators': 3, 'max_features': 4}
52877.612043606765 {'n_estimators': 10, 'max_features': 4}
50738.30785555564 {'n_estimators': 30, 'max_features': 4}
58961.447717871684 {'n_estimators': 3, 'max_features': 6}
52372.056696515625 {'n_estimators': 10, 'max_features': 6}
50108.709929157114 {'n_estimators': 30, 'max_features': 6}
58763.4998969674 {'n_estimators': 3, 'max_features': 8}
52383.0465851968 {'n_estimators': 10, 'max_features': 8}
49980.60170350313 {'n_estimators': 30, 'max_features': 8}
62226.94818167181 {'bootstrap': False, 'n_estimators': 3, 'max_features': 2}
54893.98444978685 {'bootstrap': False, 'n_estimators': 10, 'max_features': 2}
59024.106773789164 {'bootstrap': False, 'n_estimators': 3, 'max_features': 3}
52802.60887765317 {'bootstrap': False, 'n_estimators': 10, 'max_features'[...]<jupyter_text>When search space is large, it may be better to do a randomized search using RandomizedSearchCV.Another aside: Choosing good algorithms and using them together in an ensemble can produce even better results as in the case of RandomForestRegressor vs LinearRegressor  Analyzing the best models<jupyter_code>feature_importances = grid_search.best_estimator_.feature_importances_
feature_importances<jupyter_output><empty_output><jupyter_text>Evaluating on the test set Get the predictor and training labels, then prepare the test data using the pipelines calling TRANSFORM not FIT_TRANSFORM<jupyter_code>final_model = grid_search.best_estimator_

X_test = strat_test_set.drop("median_house_value", axis=1)
y_test = strat_test_set["median_house_value"].copy()

X_test_prepared = full_pipeline.transform(X_test)

final_predictions = final_model.predict(X_test_prepared)

final_mse = mean_squared_error(y_test, final_predictions)
final_rmse = np.sqrt(final_mse)
final_rmse<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09060118543607112, 'fraction_numerical': 0.052420906781618044, 'mean_word_length': 5.977443609022556, 'pattern_counts': {'":': 0, '<': 62, '<?xml version=': 0, '>': 64, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '385604', 'license': 'Apache-2.0', 'n_tokens_mistral': 4611, 'n_tokens_neox': 4055, 'n_words': 1038, 'path': 'Ch2_Housing_Regression_Code_along/Housing.ipynb', 'repo_name': 'edwardchaos/Hands-on-ML-with-Sklearn-and-TensorFlow', 'size': '12991'}