<jupyter_start><jupyter_text>ML End to End Project Flow for CA Housing Data Initial Data Exploration and Data Splits  Load the data from housings.csv file uploaded to Colab. The original dataset is from https://raw.githubusercontent.com/ageron/handson-ml2/master/datasets/housing/housing.tgz<jupyter_code>import pandas as pd
import numpy as np
housing = pd.read_csv('housing.csv')<jupyter_output><empty_output><jupyter_text>head() function shows the first five rows<jupyter_code>housing.head()<jupyter_output><empty_output><jupyter_text>info() function shows description of the data, in particular the total number of rows, each attribute’s type, and the number of nonnull values.Note that the total bedrooms has many NULL values. **These have to be taken care of later.**<jupyter_code>housing.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 20640 entries, 0 to 20639
Data columns (total 10 columns):
 #   Column              Non-Null Count  Dtype  
---  ------              --------------  -----  
 0   longitude           20640 non-null  float64
 1   latitude            20640 non-null  float64
 2   housing_median_age  20640 non-null  float64
 3   total_rooms         20640 non-null  float64
 4   total_bedrooms      20433 non-null  float64
 5   population          20640 non-null  float64
 6   households          20640 non-null  float64
 7   median_income       20640 non-null  float64
 8   median_house_value  20640 non-null  float64
 9   ocean_proximity     20640 non-null  object 
dtypes: float64(9), object(1)
memory usage: 1.6+ MB<jupyter_text>lets find the unique values of the categorical column<jupyter_code>housing["ocean_proximity"].value_counts()<jupyter_output><empty_output><jupyter_text>lets find some information on numerical columns<jupyter_code>housing.describe()<jupyter_output><empty_output><jupyter_text>Make sure the mean, median, percentiles are all observed and if required any data modifications have to be done.  To vizualize the numerical columns we can use matplotlib and see the histograms<jupyter_code>import matplotlib.pyplot as plt
housing.hist(bins=50,figsize=(20,15))
plt.show()<jupyter_output><empty_output><jupyter_text>Observe the different values and see if they make sense. If they don't make sense, then go back and check if its a data error or else if the data values are on a different scale. In the above case, look at the median income and it becomes evident it is probably in so many 100Ks i.e. 200K, 400K and so on.. Some of the data columns seem to have been capped. Look at housing median value and housing median age. See the last bar. That indicates the values may have been capped. Anything above 52(for age) and 500000k (for value) is capped to those values. This may need to be fixed by either removing such data or collecting proper values without capping.  Scale of different columns are all over the place. We will need to bring them all to same scale.This is Feature Scaling.. Also, the data distributions are not like a bell curve. It is "tail heavy" in many cases. If possible they will have to be transformed to make the data distribution to be like a bell curve.  *Dont do any more data analysis, otherwise you run the risk of "data snooping" bias.* Create a test set asap Split the data into train and test > Lets define a function to split train and test<jupyter_code>def split_train_test_random(data, test_ratio):
    np.random.seed(42)
    shuffled_indices = np.random.permutation(len(data))
    test_set_size = int(len(data) * test_ratio)
    test_indices = shuffled_indices[:test_set_size]
    train_indices = shuffled_indices[test_set_size:]
    return data.iloc[train_indices], data.iloc[test_indices]<jupyter_output><empty_output><jupyter_text>call the function above and split the data randomly. Since we used a seed, it will always split the same set of records even if we rerun the code.<jupyter_code>random_train, random_test = split_train_test_random(housing, 0.2)
random_train.head()
random_test.head()<jupyter_output><empty_output><jupyter_text>The problem with above approach is that when/if the data is updated with new rows, then the split will change and we do not want that.  If possible, we need to find some unique identifier in the data and use that information to do the splits Hashing is one such approach<jupyter_code>from zlib import crc32

def test_set_check(identifier, test_ratio):
    return crc32(np.int64(identifier)) & 0xffffffff < test_ratio * 2**32

def split_train_test_by_hash_id(data, test_ratio, id_column):
    ids = data[id_column]
    in_test_set = ids.apply(lambda id_: test_set_check(id_, test_ratio))
    return data.loc[~in_test_set], data.loc[in_test_set]<jupyter_output><empty_output><jupyter_text>The above functions are to create the ids and then split based on ids However, in the current dataset, we do not have any such unique identifier We can use row_id of the dataframe, but that is not a good approach since new data can possibly come in between the set of rows and not necessarily get appended always A better approach is to use Latitude and Longitude in case of this dataset since that will remain unique for a larger span of time.  Below, I am creating a new dataframe and adding a new column and copying the data from housing df<jupyter_code>housing_with_id = housing.assign(id=np.nan)
housing_with_id.head()
housing_with_id["id"] = housing["longitude"] * 1000 + housing["latitude"]
housing_with_id.head()<jupyter_output><empty_output><jupyter_text>Note - There are other ways to add a column to dataframe. Example below.<jupyter_code>housing_with_id_2 = housing
housing_with_id_2['id'] = housing["longitude"] * 1000 + housing["latitude"]
housing_with_id_2.head()
hash_train_set, hash_test_set = split_train_test_by_hash_id(housing_with_id, 0.2, "id")
hash_train_set.head()<jupyter_output><empty_output><jupyter_text>Sklearn has libraries for splitting. Below is the most common one.from sklearn.model_selection import train_test_splittrain_set, test_set = train_test_split(data, test_size=0.2, random_state=42) --------------------------------------------------- All the above splits were done randomly. But, many times, we have to pick test data from different examples in the data.For example, in this dataset, we have to make sure that test set also contains low median income rows. We cannot be sure of it if the split is random.   In such cases, we create new columns which can create a Category column which creates such groups in the data.  pd.cut is useful for this. https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.cut.html<jupyter_code>housing["income_cat"] = pd.cut(housing["median_income"],
                               bins=[0., 1.5, 3.0, 4.5, 6., np.inf],
                               labels=[1, 2, 3, 4, 5])
housing.head()
housing["income_cat"].hist()<jupyter_output><empty_output><jupyter_text>As we can see in the histogram, there are very few rows of data for low income category and a random split can create a bias for that.  We have used the "strata" of the incomes and we will use this info to split data by taking samples from each strata<jupyter_code>from sklearn.model_selection import StratifiedShuffleSplit

split = StratifiedShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
for train_index, test_index in split.split(housing, housing["income_cat"]):
    strat_train_set = housing.loc[train_index]
    strat_test_set = housing.loc[test_index]
strat_test_set["income_cat"].value_counts() / len(strat_test_set)
strat_train_set["income_cat"].value_counts() / len(strat_test_set)
housing["income_cat"].value_counts() / len(housing)<jupyter_output><empty_output><jupyter_text>As seen above, the data distribution is good across all income groups We now remove the income category since it is no longer needed.<jupyter_code>for set_ in (strat_train_set, strat_test_set):
    set_.drop("income_cat", axis=1, inplace=True)<jupyter_output><empty_output><jupyter_text>Data vizualizations after the data split<jupyter_code>housing_train = strat_train_set.copy()<jupyter_output><empty_output><jupyter_text>made a copy of the train data and calling it housing_train Lets look at some viz of the data.Geographical data is first since we have it..<jupyter_code>housing_train.plot(kind='scatter', x= 'longitude', y='latitude')<jupyter_output><empty_output><jupyter_text>alpha parameter is used to change the transparency.The darker regions indicate that there are overlapping scatter plots in those regions which in turn indicates data density.<jupyter_code>housing_train.plot(kind='scatter', x= 'longitude', y='latitude', alpha=0.1)<jupyter_output><empty_output><jupyter_text>Lets plot other columnss option is the size of circles c is for colorcmap is for color map. jet color map is used.<jupyter_code>housing_train.plot(kind="scatter", x="longitude", y="latitude", alpha=0.4,
    s=housing_train["population"]/100, label="population", figsize=(10,7),
    c="median_house_value", cmap=plt.get_cmap("jet"), colorbar=True,
)
plt.legend()<jupyter_output><empty_output><jupyter_text>We can also look for correlations between different columns. This works if the dataset is small<jupyter_code>housing_train.drop("id", axis=1, inplace=True)<jupyter_output><empty_output><jupyter_text>We will drop the id column since it is no longer needed.<jupyter_code>corr_matrix = housing_train.corr()
print(corr_matrix)
corr_matrix['median_house_value']
housing_train.head()<jupyter_output><empty_output><jupyter_text>Another thing to look out is whether we can create new attributes from existing attributes. Sometimes new attributes that we create will correlate much better to the dependent variable that we want to predict. Most of the times, date columns can be split to make new columns like Year, Month, Day, Week, MonthEnd, QuarterEnd etc. In this dataset we do not have any Date attribute though.  Below is datepart function taken from fastai libraryThis function converts a column of df from a datetime64 to many columns containing the information from the date. This applies changes inplace.    Parameters:    -----------    df: A pandas data frame. df gain several new columns.    fldname: A string that is the name of the date column you wish to expand.        If it is not a datetime64 series, it will be converted to one with pd.to_datetime.    drop: If true then the original date column will be removed.    time: If true time features: Hour, Minute, Second will be added.    Examples:    ---------    >>> df = pd.DataFrame({ 'A' : pd.to_datetime(['3/11/2000', '3/12/2000', '3/13/2000'], infer_datetime_format=False) })    >>> df        A    0   2000-03-11    1   2000-03-12    2   2000-03-13    >>> add_datepart(df, 'A')    >>> df        AYear AMonth AWeek ADay ADayofweek ADayofyear AIs_month_end AIs_month_start AIs_quarter_end AIs_quarter_start AIs_year_end AIs_year_start AElapsed    0   2000  3      10    11   5          71         False         False           False           False             False        False          952732800    1   2000  3      10    12   6          72         False         False           False           False             False        False          952819200    2   2000  3      11    13   0          73         False         False           False           False             False        False          952905600<jupyter_code>def add_datepart(df, fldname, drop=True, time=False):
    fld = df[fldname]
    if not np.issubdtype(fld.dtype, np.datetime64):
        df[fldname] = fld = pd.to_datetime(fld, infer_datetime_format=True)
    targ_pre = re.sub('[Dd]ate$', '', fldname)
    attr = ['Year', 'Month', 'Week', 'Day', 'Dayofweek', 'Dayofyear',
            'Is_month_end', 'Is_month_start', 'Is_quarter_end', 'Is_quarter_start', 'Is_year_end', 'Is_year_start']
    if time: attr = attr + ['Hour', 'Minute', 'Second']
    for n in attr: df[targ_pre + n] = getattr(fld.dt, n.lower())
    df[targ_pre + 'Elapsed'] = fld.astype(np.int64) // 10 ** 9
    if drop: df.drop(fldname, axis=1, inplace=True)<jupyter_output><empty_output><jupyter_text>Data Cleanup and transformations Before we start with cleanup, lets copy the training data set after removing the independent variable. The independent variable is copied to a label dataframe.<jupyter_code>housing_labels = housing_train["median_house_value"].copy()<jupyter_output><empty_output><jupyter_text>The labels will be useful later when we run the ML models and evaluate the outputs and compare with these labels. We will drop these labels from training data since we already saved it in housing_labels<jupyter_code>housing_train = housing_train.drop("median_house_value", axis=1)<jupyter_output><empty_output><jupyter_text>We will also need to do Data Cleanup. Typical cleanups are Null cleanups We can either remove such rows, remove those features or add some default value i.e. dropna, drop, fillna respectively Sklearn provides an Imputer class for this. Generally we fill the nulls with median or mean value. The strategy that you pass to the imputer should be based on that decision<jupyter_code>from sklearn.impute import SimpleImputer
imputer = SimpleImputer(strategy="median")<jupyter_output><empty_output><jupyter_text>SimpleImputer Class works on DFs which only have numbers. So, we drop the categorical column for now<jupyter_code>housing_num = housing_train.drop("ocean_proximity", axis=1)<jupyter_output><empty_output><jupyter_text>We can see below that housing_num only has number columns<jupyter_code>housing_num.head()
imputer.fit(housing_num)
housing_num.median().values
X = imputer.transform(housing_num)
type(X)
X<jupyter_output><empty_output><jupyter_text>We now have to convert this back to Dataframe<jupyter_code>housing_tr = pd.DataFrame(X, columns=housing_num.columns, index=housing_num.index)
housing_tr.head()
type(housing_tr)<jupyter_output><empty_output><jupyter_text>We can check if all the NULL values are filled up.<jupyter_code>housing_tr.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 16512 entries, 17606 to 15775
Data columns (total 8 columns):
 #   Column              Non-Null Count  Dtype  
---  ------              --------------  -----  
 0   longitude           16512 non-null  float64
 1   latitude            16512 non-null  float64
 2   housing_median_age  16512 non-null  float64
 3   total_rooms         16512 non-null  float64
 4   total_bedrooms      16512 non-null  float64
 5   population          16512 non-null  float64
 6   households          16512 non-null  float64
 7   median_income       16512 non-null  float64
dtypes: float64(8)
memory usage: 1.1 MB<jupyter_text>Lets deal with the categorical column now. Handle Categorical Attributes<jupyter_code>housing_cat = housing_train[["ocean_proximity"]]
type(housing_cat)<jupyter_output><empty_output><jupyter_text>Below 2 cells are to just show why we needed double square brackets in above cells when we created housing_cat<jupyter_code>housing_cat2= housing_train['ocean_proximity']
type(housing_cat2)<jupyter_output><empty_output><jupyter_text>We now have to convert the categorical values to numbers since most ML algorithms can use only numbers There are many classes in sklearn which we can use for this. These are transformer classes. 1. Ordinal Encoder2. One Hot Encoder2. Custom Transformer (user has to write it)<jupyter_code>from sklearn.preprocessing import OrdinalEncoder
ordinal_encoder = OrdinalEncoder()
housing_cat_encoded = ordinal_encoder.fit_transform(housing_cat)

type(housing_cat_encoded)
print(housing_cat_encoded.tolist())
ordinal_encoder.categories_<jupyter_output><empty_output><jupyter_text>This encoding may not be apt for this ocean proximity data. We see the class assigned values like 0,1,2,3,4 for the 5 different values i.e. OCEAN', 'INLAND', 'ISLAND', 'NEAR BAY', 'NEAR OCEANBut we know that 4 is not greater than 3,2 or 1. In fact, 4 is almost same as 1 i.e. Near Ocean is similar to Ocean.This encoding may not result in a good prediction.  In such cases, we use One Hot Encoding.. This just creates a sequence of 0s ans 1s for each Categorical value as shown below. Below is just an example to get an idea.'OCEAN' - 00000'INLAND' - 01000'ISLAND' -  00100'NEAR BAY' - 00010'NEAR OCEAN' - 00001<jupyter_code>from sklearn.preprocessing import OneHotEncoder
cat_encoder = OneHotEncoder()
housing_cat_1hot = cat_encoder.fit_transform(housing_cat)

housing_cat_1hot
housing_cat_1hot.toarray()
cat_encoder.categories_<jupyter_output><empty_output><jupyter_text>**Custom Transformers**Although Scikit-Learn provides many useful transformers, you will need to write your own for tasks such as custom cleanup operations or combining specific attributes. You will want your transformer to work seamlessly with Scikit-Learn functionalities (such as pipelines), and since Scikit-Learn relies on duck typing (not inheritance), all you need to do is create a class and implement three methods: fit() (returning self), transform(), and fit_transform().You can get the last one for free by simply adding TransformerMixin as a base class. If you add BaseEstimator as a base class (and avoid *args and **kargs in your constructor), you will also get two extra methods (get_params() and set_params()) that will be useful for automatic hyperparameter tuning.For example, here is a small transformer class that adds the combined attributes we discussed earlier:np.c_ is nothing but column stacking. Similarly we can also do np.r_for row stacking..<jupyter_code>from sklearn.base import BaseEstimator, TransformerMixin

rooms_ix, bedrooms_ix, population_ix, households_ix = 3, 4, 5, 6

class CombinedAttributesAdder(BaseEstimator, TransformerMixin):
    def __init__(self, add_bedrooms_per_room=True): # no *args or **kargs
        self.add_bedrooms_per_room = add_bedrooms_per_room
    def fit(self, X, y=None):
        return self  # nothing else to do
    def transform(self, X):
        rooms_per_household = X[:, rooms_ix] / X[:, households_ix]
        population_per_household = X[:, population_ix] / X[:, households_ix]
        if self.add_bedrooms_per_room:
            bedrooms_per_room = X[:, bedrooms_ix] / X[:, rooms_ix]
            return np.c_[X, rooms_per_household, population_per_household,
                         bedrooms_per_room]

        else:
            return np.c_[X, rooms_per_household, population_per_household]

attr_adder = CombinedAttributesAdder(add_bedrooms_per_room=False)
housing_extra_attribs = attr_adder.transform(housing.values)<jupyter_output><empty_output><jupyter_text>In this example the transformer has one hyperparameter, add_bedrooms_per_room, set to True by default (it is often helpful to provide sensible defaults). This hyperparameter will allow you to easily find out whether adding this attribute helps the Machine Learning algorithms or not. More generally, you can add a hyperparameter to gate any data preparation step that you are not 100% sure about. The more you automate these data preparation steps, the more combinations you can automatically try out, making it much more likely that you will find a great combination (and saving you a lot of time). Feature Scaling Feature Scaling is nothing but making the numerical columns be in similar scale.If there are numbers for one feature ranging from 0 to 1000 and for other feature ranging from -0.01 to .001then the ML algorithms do not work well. That is when feature scaling comes into picture to make the different numbers adhere to similar scale.  Two main approaches -**Min Max Scaling or Normalization*** A min and max value could be given and all numbers will be changed to fit within that range.* Generally the default range is 0 to 1* Causes issues when data has outliers*   MinMaxScaler Class**Standardization***   Subtract the mean and Divide by Std Dev.*   Less Affected by outliers*   StandardScaler Class **NOTE** - Any transformation should be used to **fit** tothe **training data only**, not to the full dataset (including the test set). Then they will be used to **transform** the training set and the test set (and any new data). Pipelines Like we saw in previous sections, there are many transformations that we did. They are done one after the other and the outputs of one transformation generally goes as input to next transformation. This can be lined up using a sklearn Pipeline class We have used SimpleImputer to handle NULLsWe have created a Column Adder class to add new columns We have to do Feature Scaling after the above two<jupyter_code>from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

num_pipeline = Pipeline([
        ('imputer', SimpleImputer(strategy="median")),
        ('attribs_adder', CombinedAttributesAdder()),
        ('std_scaler', StandardScaler()),
    ])


housing_num_tr = num_pipeline.fit_transform(housing_num)
type(housing_num_tr)<jupyter_output><empty_output><jupyter_text>The Pipeline constructor takes a list of name/estimator pairs defining a sequence of steps. All but the last estimator must be transformers (i.e., they must have a fit_transform() method). The names can be anything you like (as long as they are unique and don’t contain double underscores, __); they will come in handy later for hyperparameter tuning.When you call the pipeline’s fit() method, it calls fit_transform() sequentially on all transformers, passing the output of each call as the parameter to the next call until it reaches the final estimator, for which it calls the fit() method.The pipeline exposes the same methods as the final estimator. In this example, the last estimator is a StandardScaler, which is a transformer, so the pipeline has a transform() method that applies all the transforms to the data in sequence (and of course also a fit_transform() method, which is the one we used). Column Transformer So far, we have handled the categorical columns and the numerical columns separately. It would be more convenient to have a single transformer able to handle all columns, applying the appropriate transformations to each column. In version 0.20, Scikit-Learn introduced the ColumnTransformer for this purpose. Without this sklearn-pandas or some other third party library has to be used for achieving this.<jupyter_code>from sklearn.compose import ColumnTransformer

num_attribs = list(housing_num)
cat_attribs = list(housing_cat)


num_attribs
cat_attribs
full_pipeline = ColumnTransformer([
        ("num", num_pipeline, num_attribs),
        ("cat", OneHotEncoder(), cat_attribs),
    ], remainder='passthrough')<jupyter_output><empty_output><jupyter_text>We pass the numerical pipeline (of 3 transformers) that we created earlier for Numerical Attributes  and the single transformer for Categorical Attributes Also note the remainder keyword. If there are any columns that are not taken care of via the numerical pipeline or categorical pipeline, they will be dropped by default. The 'passthrough' keyword tells the column transformer to pass the remainder columns as is. In this case, there are none.  We can now pass the training dataset to this full pipeline.<jupyter_code>housing_train.head()
housing_train.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 16512 entries, 17606 to 15775
Data columns (total 9 columns):
 #   Column              Non-Null Count  Dtype  
---  ------              --------------  -----  
 0   longitude           16512 non-null  float64
 1   latitude            16512 non-null  float64
 2   housing_median_age  16512 non-null  float64
 3   total_rooms         16512 non-null  float64
 4   total_bedrooms      16354 non-null  float64
 5   population          16512 non-null  float64
 6   households          16512 non-null  float64
 7   median_income       16512 non-null  float64
 8   ocean_proximity     16512 non-null  object 
dtypes: float64(8), object(1)
memory usage: 1.3+ MB<jupyter_text>Note that the housing train dataframe above is plain data without any transformations or data cleanup (see nulls in total_bedrooms) whatsoever **All that will be taken care of by the single line below since we built the pipeline to handle data cleanup, column additions, feature scaling, encoding categoricals**<jupyter_code>housing_prepared = full_pipeline.fit_transform(housing_train)
type(housing_prepared)
housing_prepared<jupyter_output><empty_output><jupyter_text>Train the ML model using the data Data exploration is done.Data split for train/test is doneData vizualations were done to help with the above two. All the required data transformations are complete.The data is now ready to be passed to ML algorithms Evaluate atleast 2-3 Models Linear Regression ML Model<jupyter_code>housing_prepared
from sklearn.linear_model import LinearRegression

lin_reg = LinearRegression()
lin_reg.fit(housing_prepared, housing_labels)<jupyter_output><empty_output><jupyter_text>Lets pick some training data and get the predictions<jupyter_code>some_data = housing_train.iloc[:5]
some_data
some_data_prepared = full_pipeline.transform(some_data)
some_data_prepared
print("Predictions:", lin_reg.predict(some_data_prepared))
print("Original Values:", list(housing_labels.iloc[:5]))<jupyter_output>Original Values: [286600.0, 340600.0, 196900.0, 46300.0, 254500.0]<jupyter_text>Okay.. Its not bad.. Lets predict for entire dataset and also calculate the error<jupyter_code>housing_predictions = lin_reg.predict(housing_prepared)
from sklearn.metrics import mean_squared_error
lin_mse = mean_squared_error(housing_labels, housing_predictions)
print(np.sqrt(lin_mse)) # This is nothing but  RMSE<jupyter_output>68628.19819848923<jupyter_text>A difference of $68K for a prediction is not a good number.It could be because the model is very simple.. Its a simple Linear Regression Model. The model is **underfitting** the data. The features need to enhanced or a different model needs to be chosen.  Lets try Decision Tree Decision Tree Model<jupyter_code>from sklearn.tree import DecisionTreeRegressor

tree_reg = DecisionTreeRegressor()
tree_reg.fit(housing_prepared, housing_labels)
housing_predictions = tree_reg.predict(housing_prepared)
from sklearn.metrics import mean_squared_error
tree_mse = mean_squared_error(housing_labels, housing_predictions)
print(np.sqrt(tree_mse)) # This is nothing but  RMSE from Decision Tree Regressor<jupyter_output>0.0<jupyter_text>A difference of $0 for a prediction vs original is also not a good number.It is because the model is **overfitting** the data. How do we confirm this? We can evaluate the model on validation set and the error is going to be **very high** Another option is Scikit-Learn’s **K-fold cross-validation feature** This class (from sklearn.model_selection import cross_val_score) splits the training set into **K** distinct subsets called **folds**, then it trains and evaluates the Decision Tree model **K** times, picking a different fold for evaluation every time and training on the other **K-1** folds<jupyter_code>from sklearn.model_selection import cross_val_score
scores = cross_val_score(tree_reg, housing_prepared, housing_labels,
                         scoring="neg_mean_squared_error", cv=10)<jupyter_output><empty_output><jupyter_text>We have run the cross validation above with K=10The scoring of the cross validation feature of sklearn is actually negative.Therefore we do a sqrt of -scores below.<jupyter_code>tree_rmse_scores = np.sqrt(-scores)<jupyter_output><empty_output><jupyter_text>Below is a small function to display scores mean, std dev and list of scores for each cross validation.<jupyter_code>def display_scores(scores):
  print("Mean Score : ", scores.mean(), "\nStd Dev : " ,scores.std(), "\nAll Scores : " ,scores)
display_scores(tree_rmse_scores)<jupyter_output>Mean Score :  70666.74616904806 
Std Dev :  2928.322738055112 
All Scores :  [69327.01708558 65486.39211857 71358.25563341 69091.37509104
 70570.20267046 75529.94622521 69895.20650652 70660.14247357
 75843.74719231 68905.17669382]<jupyter_text>As expected, the Decision Tree is not accurate in its predictions. In fact, it seems to be worse than Linear Regression Model. Lets try Random Forests algorithm next  Random Forest Model<jupyter_code>from sklearn.ensemble import RandomForestRegressor
forest_reg = RandomForestRegressor()
forest_reg.fit(housing_prepared, housing_labels)
housing_predictions = forest_reg.predict(housing_prepared)
forest_rmse = np.sqrt(mean_squared_error(housing_labels, housing_predictions))
display_scores(forest_rmse)<jupyter_output>Mean Score :  18680.294240259147 
Std Dev :  0.0 
All Scores :  18680.294240259147<jupyter_text>Looks perfect..!!? No underfitting or overfitting ?? Lets try the cross validation score again<jupyter_code>scores = cross_val_score(forest_reg, housing_prepared, housing_labels,
                         scoring="neg_mean_squared_error", cv=10)
display_scores(np.sqrt(-scores))<jupyter_output>Mean Score :  50150.018373763225 
Std Dev :  1902.0697041387534 
All Scores :  [49557.6095063  47584.54435547 49605.349788   52325.13724488
 49586.9889247  53154.87424699 48800.48987508 47880.32844243
 52958.68645964 50046.17489414]<jupyter_text>Save the Model We can save the model with joblib library or pickle library. Below is an example of joblib library.<jupyter_code>import joblib
joblib.dump(forest_reg, 'forest_reg_model.pkl')

ls
forest_reg_reloaded = joblib.load('forest_reg_model.pkl')<jupyter_output><empty_output><jupyter_text>Finetune the Model We can either manually finetune by changing the various hyperparameters (parameters passed to ML algorithm) ORuse sklearn classes to try out various hyperparameters and see which ones are best GridSearchCV and RandomizedSearchCV are the classes Another way to finetune is to use ensemble method (similar to Random Forest) of using multiple algorithms Below is an example for GridSearch. Details need to be added later once Random Forest Deep Dive is done.<jupyter_code>from sklearn.model_selection import GridSearchCV

param_grid = [
    {'n_estimators': [3, 10, 30], 'max_features': [2, 4, 6, 8]},
    {'bootstrap': [False], 'n_estimators': [3, 10], 'max_features': [2, 3, 4]},
  ]

forest_reg = RandomForestRegressor()

grid_search = GridSearchCV(forest_reg, param_grid, cv=5,
                           scoring='neg_mean_squared_error',
                           return_train_score=True)

grid_search.fit(housing_prepared, housing_labels)<jupyter_output><empty_output><jupyter_text>We can get the best values for the hyperparameters as shown below<jupyter_code>grid_search.best_params_<jupyter_output><empty_output><jupyter_text>We can also get the best estimator directly<jupyter_code>grid_search.best_estimator_
feature_importances = grid_search.best_estimator_.feature_importances_
feature_importances<jupyter_output><empty_output><jupyter_text>You can select this model and test it on the test set<jupyter_code>final_model = grid_search.best_estimator_

X_test = strat_test_set.drop("median_house_value", axis=1)
y_test = strat_test_set["median_house_value"].copy()

X_test_prepared = full_pipeline.transform(X_test)

final_predictions = final_model.predict(X_test_prepared)

final_mse = mean_squared_error(y_test, final_predictions)
final_rmse = np.sqrt(final_mse) 
print(final_rmse)<jupyter_output>48760.26530172545
{'fraction_non_alphanumeric': 0.07477293079186388, 'fraction_numerical': 0.03262121018293463, 'mean_word_length': 4.777716186252771, 'pattern_counts': {'":': 0, '<': 199, '<?xml version=': 0, '>': 211, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '51', 'dir': 'jupyter-structured-clean-dedup', 'id': '390763', 'license': 'Apache-2.0', 'n_tokens_mistral': 10024, 'n_tokens_neox': 9135, 'n_words': 3362, 'path': '_notebooks/2020-08-08-ML-End-To-End-Flow-CAHousingDataset.ipynb', 'repo_name': 'mrg-ai/mrgfastpages', 'size': '31268'}