<jupyter_start><jupyter_text>Table of Contents1&nbsp;&nbsp;Dataset Description2&nbsp;&nbsp;Exploratory Data Analysis3&nbsp;&nbsp;Predictive Modeling4&nbsp;&nbsp;Model Metrics4.0.1&nbsp;&nbsp;Mean absolute error (MAE):4.0.2&nbsp;&nbsp;Mean squared error (MSE):4.0.3&nbsp;&nbsp;Root mean square error/deviation (RMSE/RMSD):4.0.4&nbsp;&nbsp;$R^2$ Score:5&nbsp;&nbsp;Linear Regression Model6&nbsp;&nbsp;Ridge Regression7&nbsp;&nbsp;LASSO Regression8&nbsp;&nbsp;Decision Tree Regressor9&nbsp;&nbsp;Random Forest Regressor10&nbsp;&nbsp;3.6 XGBoost Regressor10.0.1&nbsp;&nbsp;Terminology Explanations:10.0.2&nbsp;&nbsp;Strengths of XGBoost11&nbsp;&nbsp;Support Vector Regressor12&nbsp;&nbsp;Model Evaluation  Multiple Regression Practice with the Boston House Dataset ---- Dataset Description<jupyter_code>import numpy as np
import pandas as pd
import sklearn
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns
sns.set()
from sklearn import datasets
boston = datasets.load_boston()
type(boston)<jupyter_output><empty_output><jupyter_text>The `boston` object works like a dictionary in that we can use the `keys` function.<jupyter_code>boston.keys()
print(boston.DESCR)
boston.feature_names 
boston.data.shape
boston_df  = pd.DataFrame(boston['data']) 
boston_df.columns = boston['feature_names'] # Set column indices as feature names
boston_df['PRICE'] = boston.target # Specify MEDV (PRICE) as the target variable.
boston_df.head()<jupyter_output><empty_output><jupyter_text>---- Exploratory Data Analysis<jupyter_code>plt.hist(boston_df['PRICE'], bins=20)
plt.xlabel('Price of owner-occupied homes in $1000s')
plt.ylabel('Number of Houses')<jupyter_output><empty_output><jupyter_text>To check if there are any missing values in the data, use the `isnull()` method.<jupyter_code>def check_null_values(df):
    n_col = len(df.columns)
    null_counts = np.array(df.isnull().sum())
    if np.any(null_counts) == False:
        print("The DataFrame has no null values.")
    else:
        return df.isnull().sum()
        
check_null_values(boston_df)
# Plotting the correlation between the features
corr = boston_df.corr().round(2)
plt.figure(figsize=(15,15))
sns.heatmap(corr, annot=True, cmap='Blues')<jupyter_output><empty_output><jupyter_text>From this plot, we can see that `RM` and `LSTAT` are very strongly correlated with `PRICE`. It may be helpful to use these features for my model.<jupyter_code>boston_df.RM.hist()<jupyter_output><empty_output><jupyter_text>Box and Whisker plots make obvious which values are outliers and extreme values and which values are not. These plots represent more completely the range of values in the data by extending vertical lines to the largest and smallest values that are not outliers and extending short horizontal segments from these lines to make more apparent the values beyond which outliers begin.<jupyter_code>boston_df.boxplot(column = 'RM')
boston_df.LSTAT.hist()
boston_df.boxplot(column = 'LSTAT')
# In case you prefer using np arrays, 
# boston_df.values is the array version of the dataframe
X = boston_df.values[: , :-1]
y = boston_df.values[: , -1].reshape(len(X[:,0]), 1)
X.shape, y.shape
n_features = len(X[0])
colors = ['r', 'g', 'b', 'k', 'c', 'm', 'y']
color_wheel = colors + colors
for i in np.arange(n_features):
    x = X[:, i]
    plt.figure()
    plt.xlabel(boston.feature_names[i])
    plt.ylabel('House Price')
    plt.plot(x, y, str(color_wheel[i])+'o' )
    plt.show()<jupyter_output><empty_output><jupyter_text>---- Predictive Modeling<jupyter_code>from sklearn.model_selection import train_test_split
X_train, X_test, Y_train, Y_test = train_test_split(X, y, 
        test_size=0.3, random_state=7)<jupyter_output><empty_output><jupyter_text>Model MetricsHere, $n$ is `n_samples`, $\hat{y}$ is the vector of predicted values, and $y$ is the vector of observed values being predicted. Thus, $\hat{y}_i$ is the predicted value of the $i$th sample. Mean absolute error (MAE):$$ \text{MAE} = \frac{1}{n} \sum\limits_{i=1}^n |\hat{y}_i - y_i |$$ Mean squared error (MSE):$$\text{MSE} = \frac{1}{n} \sum\limits_{i=1}^n (\hat{y}_i - y_i)^2 $$ Root mean square error/deviation (RMSE/RMSD):$$ \text{RMSE}(\hat{y}) = \sqrt{\text{MSE}(\hat{y})} = \sqrt{ \frac{1}{n} \sum\limits_{i=1}^n (\hat{y}_i - y_i)^2 }$$  $R^2$ Score:>In statistics, the [coefficient of determination](https://en.wikipedia.org/wiki/Coefficient_of_determination) or $R^2$, is the proportion of the variance in the dependent variable that is predictable from the independent variable(s). It provides a measure of how well observed outcomes are replicated by the model, based on the proportion of total variation of outcomes explained by the model. - WikipediaFrom the [scikit-learn documentation](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.r2_score.html): ```R^2 (coefficient of determination) regression score function. The best possible score is 1.0 and it can be negative (because the model can be arbitrarily worse). A constant model that always predicts the expected value of y, disregarding the input features, would get an R^2 score of 0.0.```<jupyter_code>from sklearn import metrics

def model_train(Sklearn_model, X_train=X_train, Y_train=Y_train):
    """Trains a scikit-learn model

    Args:
        Sklearn_model : input model that uses sklearn API
        X_train (array_like, shape (n_samples, n_features)): 
            Feature/ matrix. Defaults to X_train.
        Y_train (array_like, shape (n_samples, n_targets)): 
            Target matrix. Defaults to Y_train.

    Returns:
        model : a trained version of the input model
    """
    model = Sklearn_model
    model.fit(X_train, Y_train.ravel()) # trains model, i.e. finds parameters
    return model

def model_metrics(model, X_train=X_train, X_test=X_test, Y_test=Y_test):
    Y_pred = model.predict(X_test) # an array of predicted values
    MAE = metrics.mean_absolute_error(Y_test, Y_pred)
    MSE = metrics.mean_squared_error(Y_test, Y_pred)
    RMSE = np.sqrt(metrics.mean_squared_error(Y_test, Y_pred))
    R2 = metrics.r2_score(Y_test, Y_pred)
    metric_values = np.array([MAE, MSE, RMSE, R2])
    return metric_values, Y_pred

def print_metrics(metric_values):
    print("MAE", metric_values[0])
    print("MSE", metric_values[1])
    print("RMSE", metric_values[2])
    print("R-squared Score:", metric_values[3])
    
def graph_vs_predictions(Y_pred, Y_test=Y_test):
    plt.scatter(Y_test, Y_pred)
    plt.xlabel("Prices: $Y_i$")
    plt.ylabel("Predicted prices: $\hat{Y}_i$")
    plt.title("Prices vs Predicted prices: $Y_i$ vs $\hat{Y}_i$")<jupyter_output><empty_output><jupyter_text>Linear Regression ModelModel/Approximation:$$ y = X\beta + \epsilon. $$$y$ (array of shape (n_samples, 1)):  the target vector, $X$ (array of shape (n_samples, n_features + 1)): the design, or data,  matrix, $\beta$ (array of shape (n_features + 1, 1)): coefficient matrix, $\epsilon$ (array of shape (n_samples, 1)): the residual (or error).Let $p$ denote the number of features, `n_features`, and $n$ denote the number of samples, `n_samples`. Then,$$ y = X\beta + \epsilon \equiv \begin{pmatrix}     y_1 \\ \vdots \\ y_n\end{pmatrix} = \begin{pmatrix}    1 & x_{11} & \cdots & x_{1p} \\    1 & x_{21} & \cdots & x_{2p} \\    \vdots & \vdots & \ddots & \vdots \\    1 & x_{n1} & \cdots & x_{np}\end{pmatrix}\begin{pmatrix}    \beta_0 \\ \beta_1 \\ \vdots \\ \beta_p\end{pmatrix} + \begin{pmatrix}    \epsilon_1 \\ \vdots \\ \epsilon_n\end{pmatrix}$$Notice, $\epsilon = y-X\beta$, which is why $(y-X\beta)^2 \equiv (y-\hat{y})^2$ is called a square residual.<jupyter_code>from sklearn.linear_model import LinearRegression

LR = LinearRegression()
LR.fit(X_train, Y_train) # trains the model, i.e. finds the β coefficients<jupyter_output><empty_output><jupyter_text>``` intercept_ : float or array of shape (n_targets,)      Independent term in the linear model. Set to 0.0 if    `fit_intercept = False`. ```In mathematical terms, the intercept is defined as the mean or expectation of $y$ when all of the features in the model are set to zero, so it's $\beta_0$ in the matrix formalism above. $$\mathbb{E}(y) = \mathbb{E}(X\beta + \epsilon) = \mathbb{E}(X\beta).$$And when each feature value, $\beta_i$, is set equal to 0, $\mathbb{E}(y) = \beta_0$.<jupyter_code># Return the independent term, or intercept, of the linear model
print('β_0 : ', LR.intercept_)
LR.singular_
# Return an array of the regression coefficients
LR.coef_
reg_0 = LR

metric_values = model_metrics(model_train(reg_0))[0] 
Y_pred = model_metrics(model_train(reg_0))[1]

print_metrics(metric_values)
graph_vs_predictions(Y_pred)<jupyter_output>MAE 3.307638451069791
MSE 25.767879490836645
RMSE 5.076207195420282
R-squared Score: 0.6508417720329545<jupyter_text>Ridge Regression[Wikipedia Summary](https://en.wikipedia.org/wiki/Tikhonov_regularization)> Tikhonov regularization, named for <NAME>, is a method of regularization of ill-posed problems. Also known as ridge regression, it is particularly useful to mitigate the problem of multicollinearity in linear regression, which commonly occurs in models with large numbers of parameters. In general, the method provides improved efficiency in parameter estimation problems in exchange for a tolerable amount of bias (see bias–variance tradeoff).<jupyter_code>from sklearn.linear_model import Ridge
reg_1 = Ridge()

metric_values = model_metrics(model_train(reg_1))[0] 
Y_pred = model_metrics(model_train(reg_1))[1]

print_metrics(metric_values)
graph_vs_predictions(Y_pred)<jupyter_output>MAE 3.3150472344063284
MSE 26.27787009427617
RMSE 5.1261945041401
R-squared Score: 0.643931330859861<jupyter_text>LASSO Regression[Wikipedia Summary](https://en.wikipedia.org/wiki/Lasso_(statistics)):> In statistics and machine learning, lasso (least absolute shrinkage and selection operator; also Lasso or LASSO) is a regression analysis method that performs both variable selection and regularization in order to enhance the prediction accuracy and interpretability of the statistical model it produces.<jupyter_code>from sklearn.linear_model import Lasso
reg_2 = Lasso()

metric_values = model_metrics(model_train(reg_2))[0] 
Y_pred = model_metrics(model_train(reg_2))[1]

print_metrics(metric_values)
graph_vs_predictions(Y_pred)<jupyter_output>MAE 3.3416852489576727
MSE 23.90558828559172
RMSE 4.889334135195888
R-squared Score: 0.6760760679870769<jupyter_text>Decision Tree RegressorThis algorithm is a bit too complex for me to give a useful conceptual summary for. If you're not familiar with decision trees, I invite you to read [this post](https://gdcoder.com/decision-tree-regressor-explained-in-depth/), which explains the structure of  decision tree regressors very well.<jupyter_code>from sklearn.tree import DecisionTreeRegressor
reg_3 = DecisionTreeRegressor()

metric_values = model_metrics(model_train(reg_3))[0] 
Y_pred = model_metrics(model_train(reg_3))[1]

print_metrics(metric_values)
graph_vs_predictions(Y_pred)<jupyter_output>MAE 2.943421052631579
MSE 19.86552631578947
RMSE 4.457075982725611
R-squared Score: 0.7308194502958476<jupyter_text>Random Forest RegressorAlgorithm explanation: [Link](https://towardsdatascience.com/random-forest-and-its-implementation-71824ced454f)TL;DR: Random forest is an ensemble model and one of the most accuarate ML algorithms available. Random forests run efficiently on large datasets and are constructed from a multitude of decision trees.<jupyter_code>from sklearn.ensemble import RandomForestRegressor
reg_4 = RandomForestRegressor(random_state=3)

metric_values = model_metrics(model_train(reg_4))[0] 
Y_pred = model_metrics(model_train(reg_4))[1]

print_metrics(metric_values)
graph_vs_predictions(Y_pred)<jupyter_output>MAE 2.338269736842104
MSE 12.650229927631575
RMSE 3.556716171924824
R-squared Score: 0.8285876854368929<jupyter_text>3.6 XGBoost Regressor  XGBoost, short for e**X**treme **G**radient **B**oosting, is a gradient boosting algorithm authored by [<NAME>](https://tqchen.com/). Terminology Explanations:- [Boosting](https://en.wikipedia.org/wiki/Boosting_(machine_learning)) :  an ensemble meta-algorithm for reducing bias and variance.- [Gradient Boosting](https://en.wikipedia.org/wiki/Gradient_boosting) : A machine learning technique for regression and classification problems, which produces a prediction model in the form of an ensemble of weak prediction models, typically decision trees. It builds the model in a stage-wise fashion like other boosting methods do, and it generalizes them by allowing optimization of an arbitrary differentiable loss function. Strengths of XGBoostThe algorithm's strengths were explained by Tianqi in a [quora post](https://www.quora.com/What-is-the-difference-between-the-R-gbm-gradient-boosting-machine-and-xgboost-extreme-gradient-boosting):> Both xgboost and gbm follows the principle of gradient boosting.  There are however, the difference in modeling details. Specifically,  xgboost used a more regularized model formalization to control over-fitting, which gives it better performance.> The name xgboost, though, actually refers to the engineering goal to push the limit of computations resources for boosted tree algorithms. Which is the reason why many people use xgboost.<jupyter_code>from xgboost import XGBRegressor
reg_5 = XGBRegressor()

metric_values = model_metrics(model_train(reg_5))[0] 
Y_pred = model_metrics(model_train(reg_5))[1]

print_metrics(metric_values)
graph_vs_predictions(Y_pred)<jupyter_output>MAE 2.184927732066104
MSE 12.381583921473455
RMSE 3.518747493281303
R-squared Score: 0.8322278749019961<jupyter_text>[Support Vector Regressor](https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVR.html)> The method of Support Vector Classification can be extended to solve regression problems. This method is called Support Vector Regression.  The model produced by support vector classification depends only on a subset of the training data, because the cost function for building the model does not care about training points that lie beyond the margin. Analogously, the model produced by Support Vector Regression depends only on a subset of the training data, because the cost function ignores samples whose prediction is close to their target.- [source](https://scikit-learn.org/stable/modules/svm.htmlsvm-regression)<jupyter_code># import SVM Regressor
from sklearn import svm
reg_6 = svm.SVR(kernel='linear')

metric_values = model_metrics(model_train(reg_6))[0] 
Y_pred = model_metrics(model_train(reg_6))[1]

print_metrics(metric_values)
graph_vs_predictions(Y_pred)<jupyter_output>MAE 3.282386852144568
MSE 31.58808740378112
RMSE 5.620328051260097
R-squared Score: 0.5719771731044268<jupyter_text>----  Model Evaluation<jupyter_code>col_index = ['Linear', 'Ridge', 'Lasso', 'Decision Tree', 'Random Forest', 
          'XGB', 'SVR']
row_index = ['MAE', 'MSE', 'RMSE', 'R_sq']

columns = []
for regressor in [reg_0, reg_1, reg_2, reg_3, reg_4, reg_5, reg_6]:
    columns.append(model_metrics(
        model_train(regressor))[0].reshape(len(row_index), 1))
unstructured_array = np.hstack((columns[:]))
table = pd.DataFrame(unstructured_array)

col_index = pd.Series(col_index)
row_index = pd.Series(row_index)

table.columns = col_index
table.index = row_index
# Rank the models from best to worst
table.sort_values('R_sq', axis=1, ascending=False, inplace=True)

table.head()
# There are too many decimal places on the values in `table`.
# Truncate the floats to fit more nicely into 
# the matplotlib table generated by the `render_mpl_table` method.
def truncate(f, n):
    '''Truncates/pads a float f to n decimal places without rounding'''
    s = '{}'.format(f)
    if 'e' in s or 'E' in s:
        return '{0:.{1}f}'.format(f, n)
    i, p, d = s.partition('.')
    return '.'.join([i, (d+'0'*n)[:n]])

table = np.array(table)

for i in range(table.shape[0]):
    for j in range(table.shape[1]):
        table[i, j] = truncate(table[i,j], 3)

table = pd.DataFrame(table)
table.columns = col_index
table.index = row_index
table
import six
def render_mpl_table(data, col_width=3.0, row_height=0.625, font_size=14,
                     header_color='#40466e', row_colors=['#f1f1f2', 'w'], 
                     edge_color='w', bbox=[0, 0, 1, 1], header_columns=0,
                     ax=None, **kwargs):
    if ax is None:
        size = ((np.array(data.shape[::-1]) + np.array([0, 1])) 
                * np.array([col_width, row_height]))
        fig, ax = plt.subplots(figsize=size)
        ax.axis('off')

    mpl_table = ax.table(cellText=data.values, 
                         bbox=bbox, 
                         colLabels=data.columns,
                         **kwargs)

    mpl_table.auto_set_font_size(False)
    mpl_table.set_fontsize(font_size)

    for k, cell in  six.iteritems(mpl_table._cells):
        cell.set_edgecolor(edge_color)
        if k[0] == 0 or k[1] < header_columns:
            cell.set_text_props(weight='bold', color='w')
            cell.set_facecolor(header_color)
        else:
            cell.set_facecolor(row_colors[k[0]%len(row_colors) ])
    return ax

render_mpl_table(table, header_columns=0, col_width=2.0)
plt.savefig('metrics_table.png')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11373776000927052, 'fraction_numerical': 0.03812503621299032, 'mean_word_length': 5.114063053489196, 'pattern_counts': {'":': 0, '<': 65, '<?xml version=': 0, '>': 70, 'https://': 12, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '236258', 'license': 'MIT', 'n_tokens_mistral': 6175, 'n_tokens_neox': 5649, 'n_words': 1581, 'path': '2. Boston House Price Prediction/Boston House Prices - Multiple Regression.ipynb', 'repo_name': 'Unique-Divine/Regression-Practice', 'size': '17273'}