<jupyter_start><jupyter_text>House prediction problemA full end to end project on California Housing Prices dataset from the StatLib repository. This dataset is based on data from the 1990 California census.![](https://raw.githubusercontent.com/kalyandechiraju/ML_Regression/master/IMG_20190730_195343.jpg)  Look at the Big PictureOur task is to build a model of housing proces in California using the sensus data. This data has metrics such as the population, median income, median housing price and so on for each district in California.The model should learn from this data and be able to predict the median housing price in any district, given all other metrics.  Frame the ProblemBefore proceeding we have to find out what is our end goal. This is important because it will determine how you frame the problem, what algorithms you will select, what performance measure you will use to evaluate your model and how much effort you should spend tweaking it.Let's say our model's output will be fed to another Machine Learning system along with many other signals. This downstream will determine whether it is worth investing in a given area or not. Getting this right is critical, as it directly affects revenue.![](https://raw.githubusercontent.com/kalyandechiraju/ML_Regression/master/IMG_20190730_195355.jpg)<jupyter_code>import pandas as pd
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np<jupyter_output><empty_output><jupyter_text>Load the data set<jupyter_code>df = pd.read_csv("https://raw.githubusercontent.com/ageron/handson-ml/master/datasets/housing/housing.csv")
df.head()<jupyter_output><empty_output><jupyter_text>Explore the data set<jupyter_code>df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 20640 entries, 0 to 20639
Data columns (total 10 columns):
longitude             20640 non-null float64
latitude              20640 non-null float64
housing_median_age    20640 non-null float64
total_rooms           20640 non-null float64
total_bedrooms        20433 non-null float64
population            20640 non-null float64
households            20640 non-null float64
median_income         20640 non-null float64
median_house_value    20640 non-null float64
ocean_proximity       20640 non-null object
dtypes: float64(9), object(1)
memory usage: 1.6+ MB<jupyter_text>Lets change the categorical value of ocean_proximity<jupyter_code>df["ocean_proximity"].value_counts()
df.describe()
df.hist(bins=50, figsize=(20,15))
plt.show()<jupyter_output><empty_output><jupyter_text>Creating a feature "income_cat" which will be used to split out data set into training and testing sets. Median income is an important feature to be considered while splitting data set to avoind sampling bias<jupyter_code>df["income_cat"] = np.ceil(df["median_income"] / 1.5)
df["income_cat"].where(df["income_cat"] < 5, 5.0, inplace=True)
df.hist(bins=50, figsize=(20,15))
plt.show()<jupyter_output><empty_output><jupyter_text>Split the data set into training and testing data sets<jupyter_code>from sklearn.model_selection import StratifiedShuffleSplit
split = StratifiedShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
for train_index, test_index in split.split(df, df["income_cat"]):
  train_set = df.loc[train_index]
  test_set = df.loc[test_index]
train_set.describe()
test_set.describe()<jupyter_output><empty_output><jupyter_text>Splitting data set is now. Now removing the "income_cat" feature (as it is unnecessary moving on).<jupyter_code>for dataset in (train_set, test_set):
  dataset.drop("income_cat", axis=1, inplace=True)

train_set.describe()
test_set.describe()
train_set.head()
test_set.head()<jupyter_output><empty_output><jupyter_text>Let's explore the training data set in depth<jupyter_code># creating a copy of trining set to avoid messing it up
ts_copy = train_set.copy()
ts_copy.plot(kind="scatter", x="longitude", y="latitude")
# for better visualization set alpha value
ts_copy.plot(kind="scatter", x="longitude", y="latitude", alpha=0.1)
# let's improve the plot a bit more
ts_copy.plot(kind="scatter", x="longitude", y="latitude", alpha=0.1, s=ts_copy["population"]/100, label="population", figsize=(10,7),
            c="median_house_value", cmap=plt.get_cmap("jet"), colorbar=True,)
plt.legend()<jupyter_output><empty_output><jupyter_text>Correlations between columnsStandard correlation coefficient (also called pearson's r):In statistics, the Pearson correlation coefficient (PCC, pronounced /ˈpɪərsən/), also referred to as Pearson's r, the Pearson product-moment correlation coefficient (PPMCC) or the bivariate correlation, `is a measure of the linear correlation between two variables X and Y`.<jupyter_code>ts_copy.corr(method="pearson")<jupyter_output><empty_output><jupyter_text>The corelation coefficient ranges from -1 to 1. When it is close to 1, it means that there is strong positive correlation between two columns and vice-versa and coefficients close to zero mean that there is no linear correlation between those two columns. Observation:In this data set, median house value tends to go up when median income goes up.<jupyter_code># Pandas' scatter_matrix function plots correlation between every numerical attribute against every other numerical attribute in the data set.
# Here we have 9 columns and hence we get 9^2 plots. To simplify lets concerntrate on few promising columns.

from pandas.plotting import scatter_matrix
attributes = ["median_house_value", "median_income", "total_rooms", "housing_median_age"]
scatter_matrix(ts_copy[attributes], figsize=(12,8))
# lets zoomin on median_house_value vs median_income
ts_copy.plot(kind="scatter", x="median_income", y="median_house_value", alpha=0.1)<jupyter_output><empty_output><jupyter_text>Experimenting with attribute combinationsBefore preparing data for machine learning, we might want to try out different combinations of existing features to form new features.<jupyter_code>ts_copy["rooms_per_household"] = ts_copy["total_rooms"]/ts_copy["households"]
ts_copy["bedrooms_per_room"] = ts_copy["total_bedrooms"]/ts_copy["total_rooms"]
ts_copy["population_per_household"] = ts_copy["population"]/ts_copy["households"]
ts_copy.corr()<jupyter_output><empty_output><jupyter_text>Preparing data for Machine Learning Algorithms Instead of doing this manually, we should automate the process for several reasons.- This will allow us to reproduce these transformations on any other dataset easily.- You can gradually build a library of transformation funtions to use them in future.- You can use these functions in your live system to transform new data before feeding it to your algorithms.- This can help us try various transformations and decide which works best.<jupyter_code># Let's revert all the above transformations and start by dividing predictors and labels
housing = train_set.drop("median_house_value", axis=1)
housing_labels = train_set["median_house_value"].copy()
housing.head()
housing_labels.head()<jupyter_output><empty_output><jupyter_text>Data CleaningThis is a very important step as many machine learning algorithms cannot work with missing features. We can fix the missing features by:- Getting rid of the corresponding rows.- Getting rid of the whole attribute.- Setting the missing values to some value (zero or mean or median etc.).<jupyter_code># Scikit-Learn can help us in handling missing values

from sklearn.preprocessing import Imputer

imputer = Imputer(strategy="median")
from sklearn.impute import SimpleImputer
imputer = SimpleImputer(strategy="median")
# Since median can only be computed on numerical attributes, we need to create a copy of data without text attributes - "ocean_proximity"
housing_num = housing.drop("ocean_proximity", axis=1)

imputer.fit(housing_num)
# imputer simply computed the median for each attribute and stored the resilt in its statistics_ instance variable
imputer.statistics_
housing_num.median() #Same as above<jupyter_output><empty_output><jupyter_text>Note: By using imputer, we can ensure that any missing data in any attribute in the future can be handled even after system goes live.<jupyter_code>X = imputer.transform(housing_num) # outputs a plain numpy array
housing_tr = pd.DataFrame(X, columns=housing_num.columns)
housing_tr.head()<jupyter_output><empty_output><jupyter_text>Handling text and categorical attributesMachine learning algorithms prefer to work with numbers. So we might need to convert categorical values into numbers.In our example, 'ocean_proximity' is a categorical attribute. We will convert it into numerical one.<jupyter_code># lets try this using LabelEncoder from sklearn
from sklearn.preprocessing import LabelEncoder
encoder = LabelEncoder()
housing_cat = housing["ocean_proximity"]
housing_cat_encoded = encoder.fit_transform(housing_cat)
housing_cat_encoded
print(encoder.classes_)<jupyter_output>['<1H OCEAN' 'INLAND' 'ISLAND' 'NEAR BAY' 'NEAR OCEAN']<jupyter_text>Notice how the encoder mapped 0 to 4 to different values for each category (<1H Ocean to 0, Inland to 1 and so on).However this representation might not be accurate because algorithms tend to think that nearby values are more similar than farther values. In our case thats not true. Category 0 and 4 and more similar that 0 and 1.To fix this we use **One-Hot encoding**<jupyter_code>from sklearn.preprocessing import OneHotEncoder
encoder = OneHotEncoder()
housing_cat_1hot = encoder.fit_transform(housing_cat_encoded.reshape(-1, 1))
housing_cat_1hot
#housing_cat_1hot.toarray()<jupyter_output>/usr/local/lib/python3.6/dist-packages/sklearn/preprocessing/_encoders.py:371: FutureWarning: The handling of integer data will change in version 0.22. Currently, the categories are determined based on the range [0, max(values)], while in the future they will be determined based on the unique values.
If you want the future behaviour and silence this warning, you can specify "categories='auto'".
In case you used a LabelEncoder before this OneHotEncoder to convert the categories to integers, then you can now use the OneHotEncoder directly.
  warnings.warn(msg, FutureWarning)<jupyter_text>So we transformed using **LabelEncoder** first and then with **OneHotEncoder**, this can be done in one shot by using **LabelBinarizer**<jupyter_code>from sklearn.preprocessing import LabelBinarizer
#encoder = LabelBinarizer(sparse_output=True) -> to get sparse array output
encoder = LabelBinarizer()
housing_cat_1hot = encoder.fit_transform(housing_cat)
housing_cat_1hot<jupyter_output><empty_output><jupyter_text>Custom TransformersTo write a custom transformer, using sklearn, all we need to do is create a class and implement three functions- fit() - that returns self- transform()- fit_transform()We can get the implementation of fit_transform for free by adding **TransformerMixin** as the base class. Also, by adding **BaseEstimator** as the base class, we get two methods (get_params() and set_params()) that will be useful for automatic hyperparameter tuning.<jupyter_code>#Sample custom transformer that adds the combined attributes
from sklearn.base import BaseEstimator, TransformerMixin

rooms_ix, bedrooms_ix, population_ix, household_ix = 3, 4, 5, 6

class CombinedAttributesAdder(BaseEstimator, TransformerMixin):
  def __init__(self, add_bedrooms_per_room = True):
    self.add_bedrooms_per_room = add_bedrooms_per_room
  
  def fit(self, X, y=None):
    return self # Nothing to do here
  
  def transform(self, X, y=None):
    rooms_per_household = X[:, rooms_ix] / X[:, household_ix]
    population_per_household = X[:, population_ix] / X[:, household_ix]
    if self.add_bedrooms_per_room:
      bedrooms_per_room = X[:, bedrooms_ix] / X[:, rooms_ix]
      return np.c_[X, rooms_per_household, population_per_household, bedrooms_per_room]
    else:
      return np.c_[X, rooms_per_household, population_per_household]
    
attr_adder = CombinedAttributesAdder(add_bedrooms_per_room=True)
housing_extra_attribs = attr_adder.transform(housing.values)
housing_added = pd.DataFrame(housing_extra_attribs)
housing_added.head()<jupyter_output><empty_output><jupyter_text>Transformation PipelinesAs you can see, there are many data transformation steps that need to be executed in the right order.Scikit-Learn provides the ***Pipeline*** class to help with this.<jupyter_code>from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

num_pipeline = Pipeline([
    ('imputer', Imputer(strategy="median")), # converts DataFrame to numpy array
    ('attr_adder', CombinedAttributesAdder()), # Adds additional attributes
    ('std_scalar', StandardScaler()), # Transformergbbb
])

housing_num_tr = num_pipeline.fit_transform(housing_num)
housing_num_tr
# Transformer to convert DataFrame into numpy array
from sklearn.base import BaseEstimator, TransformerMixin

class DataFrameSelector(BaseEstimator, TransformerMixin):
  def __init__(self, attr_names):
    self.attr_names = attr_names
    
  def fit(self, X, y=None):
    return self
  
  def transform(self, X):
    return X[self.attr_names].values
num_attr = list(housing_num)
cat_attr = ["ocean_proximity"]

# Pipeline to convert dataframe to numpy array, then fit imputer with median strategy, then add additional attributes and finally apply standard scalar on the data
num_pipeline = Pipeline([
    ('selector', DataFrameSelector(num_attr)),
    ('imputer', Imputer(strategy="median")),
    ('attr_adder', CombinedAttributesAdder()),
    ('std_scalar', StandardScaler()),
])

# Pipeline to select categorical attributes, convert them into numpy array and then apply label binarizer on it
cat_pipeline = Pipeline([
    ('selector', DataFrameSelector(cat_attr)),
    #('label_binarizer', LabelBinarizer()),
    ('encoder', OneHotEncoder()),
])
# Joining multiple pipelines
from sklearn.pipeline import FeatureUnion

full_pipeline = FeatureUnion(transformer_list=[
    ("num_pipeline", num_pipeline),
    ("cat_pipeline", cat_pipeline),
])
#Running the whole pipeline
housing_prepared = full_pipeline.fit_transform(housing)
housing_prepared
housing_prepared.shape<jupyter_output><empty_output><jupyter_text>Training and Evaluating on the Training Set<jupyter_code># Trying with Linear Regression initially
from sklearn.linear_model import LinearRegression
lin_reg = LinearRegression()
lin_reg.fit(housing_prepared, housing_labels)
# Let's try this model on some samples from training set
some_data = housing.iloc[:5]
some_labels = housing_labels.iloc[:5]
some_data_prepared = full_pipeline.transform(some_data)
#some_data_prepared.shape
print("predictions: ", lin_reg.predict(some_data_prepared))
print("Labels: ", list(some_labels))
# Measuring the RMSE on the whole training set
from sklearn.metrics import mean_squared_error
housing_predictions = lin_reg.predict(housing_prepared)
lin_mse = mean_squared_error(housing_labels, housing_predictions)
lin_rmse = np.sqrt(lin_mse)
lin_rmse<jupyter_output><empty_output><jupyter_text>Clearly, this is not a great score. This is a classic example of **underfitting the training data**.Let's try with a more powerful model<jupyter_code>#Using Decision Trees
from sklearn.tree import DecisionTreeRegressor
tree_reg = DecisionTreeRegressor()
tree_reg.fit(housing_prepared, housing_labels)

housing_predictions_tree = tree_reg.predict(housing_prepared)
tree_mse = mean_squared_error(housing_labels, housing_predictions_tree)
tree_rmse = np.sqrt(tree_mse)
tree_rmse<jupyter_output><empty_output><jupyter_text>No error! This could be due to **overfitting the training data**.  Better Evaluation Using Cross-ValidationOne way to evaluate our models is to use the K-fold cross validation (provided by sklearn). It randomly splits the **training set** into 10 distinct subsets called folds, then it trains and evaluated the model 10 times, picking a different fold for evaluation every time and training on the other 9 folds. The result is an array containing the 10 evaluation scores.<jupyter_code>from sklearn.model_selection import cross_val_score
scores = cross_val_score(tree_reg, housing_prepared, housing_labels, scoring="neg_mean_squared_error", cv=10)
tree_rmse_scores = np.sqrt(-scores)<jupyter_output><empty_output><jupyter_text>**Note:****Sckit-Learn cross-validation features expect a utility function (greater is better) rather than a cost function (lower is better), so the scoring function is actually the opposite of the MSE (i.e., a negative value), which is why the preceding code computes -scores before calculation the square root.**<jupyter_code>def display_scores(scores):
  print("Scores:", scores)
  print("Mean:", scores.mean())
  print("Standard Deviation:", scores.std())
  
display_scores(tree_rmse_scores)
#Cross validation for Linear regression model
lin_scores = cross_val_score(lin_reg, housing_prepared, housing_labels, scoring="neg_mean_squared_error", cv=10)
lin_rmse_scores = np.sqrt(-lin_scores)
display_scores(lin_rmse_scores)<jupyter_output>Scores: [66782.73844317 66960.11809222 70347.95243904 74739.57055258
 68031.13389115 71193.84181183 64969.63056023 68281.61137907
 71552.91565846 67665.10081739]
Mean: 69052.46136451427
Standard Deviation: 2731.674003577556<jupyter_text>RandomForestRegressor- Random Forests work by training **many Decision Trees on random subsets of the features**, then averaging out their predictions.- Building a model on top of many other models is called **Ensemble Learning**.<jupyter_code>#Trying with RandomForestRegressor
from sklearn.ensemble import RandomForestRegressor

forest_reg = RandomForestRegressor()
forest_reg.fit(housing_prepared, housing_labels)

housing_predictions = forest_reg.predict(housing_prepared)
forest_mse = mean_squared_error(housing_labels, housing_predictions)
forest_rmse = np.sqrt(forest_mse)
forest_rmse
forest_scores = cross_val_score(forest_reg, housing_prepared, housing_labels, scoring="neg_mean_squared_error", cv=10)
forest_rmse_scores = np.sqrt(-forest_scores)
display_scores(forest_rmse_scores)
display_scores(forest_rmse_scores)
from sklearn.model_selection import GridSearchCV

param_grid = [
    {'n_estimators': [3, 10, 30], 'max_features': [2, 4, 6, 8]},
    {'bootstrap': [False], 'n_estimators': [3, 10], 'max_features': [2, 3, 4]},
]

forest_reg = RandomForestRegressor()

grid_search = GridSearchCV(forest_reg, param_grid, cv=5, scoring='neg_mean_squared_error')

grid_search.fit(housing_prepared, housing_labels)
grid_search.best_params_
grid_search.best_estimator_
cvres = grid_search.cv_results_
for mean_score, params in zip(cvres["mean_test_score"], cvres["params"]):
  print(np.sqrt(-mean_score), params)
grid_search.best_estimator_.feature_importances_
final_model = grid_search.best_estimator_
X_test = test_set.drop("median_house_value", axis=1)
y_test = test_set["median_house_value"].copy()

X_test_prepared = full_pipeline.transform(X_test)

final_predictions = final_model.predict(X_test_prepared)

final_mse = mean_squared_error(y_test, final_predictions)
final_rmse = np.sqrt(final_mse)

final_rmse<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08296070282497764, 'fraction_numerical': 0.02046399074122784, 'mean_word_length': 6.649899396378269, 'pattern_counts': {'":': 0, '<': 97, '<?xml version=': 0, '>': 95, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '24', 'dir': 'jupyter-structured-clean-dedup', 'id': '157513', 'license': 'Apache-2.0', 'n_tokens_mistral': 6082, 'n_tokens_neox': 5622, 'n_words': 1805, 'path': 'Hands_on_ML_Regression.ipynb', 'repo_name': 'kalyandechiraju/ML_Regression', 'size': '19009'}