<jupyter_start><jupyter_text>Multiple Linear RegressionPreviously, we used a single feature to predict the sale price of a house. Any number of features may be used with linear regression to build a model. **Multiple Linear Regression** refers to linear regression with two or more features. Let's see the general form of the model.$$\hat{y} = w_{0} + w_{1}x_{1} + w_{2}x_{2} + ... + w_{p}x_{p}$$Where `p` is the number of predictor variables (features). The above equation is no longer an equation for a line but for a [hyperplane][1]. Same Goal - minimize squared errorMultiple linear regression has the same goal as simple linear regression - to choose coefficients that minimize the sum of squared errors. Naive Iterative process no longer feasibleOur naive iterative process that we employed to search for the optimal coefficients is not quite a feasible solution as the number of combinations grows exponentially with each added feature. On the other hand, the method employed by scikit-learn learns the values of coefficients efficiently. Choose features to build a modelLet's read in the housing dataset and select `GarageArea` and `FullBath` in addition to `GrLivArea` as features in the model.[1]: https://en.wikipedia.org/wiki/Hyperplane<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
housing = pd.read_csv('../data/housing_sample.csv')
housing.head()<jupyter_output><empty_output><jupyter_text>Our input data now consists of three features. Let's select it now.<jupyter_code>X = housing[['GrLivArea', 'GarageArea', 'FullBath']]
y = housing['SalePrice']<jupyter_output><empty_output><jupyter_text>Import, Instantiate, FitLet's use the same three-step process to build a model from these three features.<jupyter_code>from sklearn.linear_model import LinearRegression
lr = LinearRegression()
lr.fit(X, y)<jupyter_output><empty_output><jupyter_text>Output the coefficients of the modelTo see the trained coefficients of the model, access the `intercept_` and `coef_` attributes.<jupyter_code>lr.intercept_
lr.coef_<jupyter_output><empty_output><jupyter_text>Formally write the modelWe can formally write out the mathematical definition of our model as the following, where $x_{1}$, $x_{2}$, and $x_{3}$ represent our features.$$\hat{y} = -15836 + 70x_{1} + 132x_{2} + 17942x_{3}$$  Make predictionsIt is uncommon to create your own input data for predictions. Instead, you are more likely to have new observed data absent of a predicted value that you would use your model with. For instance, when a new house comes on the market, you can use your model to determine what it could sell at. Using our training data to make predictionsInstead of creating our own data by hand, we can select rows of our training data to use as input data for prediction. Let's select the first 10 rows of our input data to do so.<jupyter_code>lr.predict(X.head(10)).round(-3)<jupyter_output><empty_output><jupyter_text>We can compare it to the actual sale price.<jupyter_code>y.values[:10]<jupyter_output><empty_output><jupyter_text>Plot predicted vs actual valuesIt might be helpful to plot the predicted and actual values on the same plot. We do so below for the first 10 observations. A few predictions are very close to the actual observed sale price. The x-axis corresponds to the observation number (1, 2, 3, etc...). The sale price is plotted on the y-axis. For each observation, two points are plotted, one for the predicted and one for the actual value.<jupyter_code>y_pred = lr.predict(X.head(10)).round(-3)
fig, ax = plt.subplots(figsize=(12, 5))
ax.scatter(np.arange(10), y_pred, label='predicted')
ax.scatter(np.arange(10), y.head(10), label='actual')
ax.legend()
ax.set_title('Actual vs Predicted for first 10 rows')
ax.set_ylabel('Sale Price')
ax.set_xticks([]);<jupyter_output><empty_output><jupyter_text>Evaluating model performanceWe can use the `score` method again to determine what the $R^2$ of our model is, but we need to remember that this is calculated on data that the model has already seen and will not be an accurate measure of how the model will perform in the future on unseen data.<jupyter_code>lr.score(X, y)<jupyter_output><empty_output><jupyter_text>Comparing multivariate and univariate modelLet's compare the performance of our new multivariate model to the univariate model that just uses `GrLivArea`. We don't have to import the linear regression model again, just begin by instantiating a new model and calculating $R^2$.<jupyter_code>lr2 = LinearRegression()
X2 = housing[['GrLivArea']]
lr2.fit(X2, y)
lr2.score(X2, y)<jupyter_output><empty_output><jupyter_text>The new model explains about 12% more of the inherent variance in the data suggesting that there is value in using the variables garage area and full baths. Though, this is a tentative conclusion as we are still evaluating on the training set and not on unseen data.  Multiple linear regression model interpretationInterpretation of the linear regression model when using multiple features is more complex than the univariate case. From our model above, every one-unit increase in `GrLivArea` translates to an increase of 70 dollars in sale price assuming the other features are held constant. For instance, if a particular house has 1,000 square feet of living area, 500 square feet of garage area and two full baths, then increasing the living area to 1,001 square feet while keeping garage area and number of full baths the same will lead to a 70 dollar increase in the sale price.A major issue with this interpretation is that many features are correlated with one another, meaning that increasing the value of one likely leads to a change in another. Theoretically, in order for us to use the interpretation above, the features must be uncorrelated with one another. With our particular data, this isn't the case. Using the DataFrame `corr` method, we see that above ground living area and garage area have a correlation of nearly .5 so increasing one will likely lead to an increase in the other.<jupyter_code>X.corr().round(2)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07025607353906763, 'fraction_numerical': 0.01329612606697308, 'mean_word_length': 5.509615384615385, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 41, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '483737', 'license': 'MIT', 'n_tokens_mistral': 1736, 'n_tokens_neox': 1612, 'n_words': 816, 'path': 'jupyter_notebooks/machine_learning/mastering_machine_learning/02. Linear Regression/05. Multiple Linear Regression.ipynb', 'repo_name': 'manual123/Nacho-Jupyter-Notebooks', 'size': '6092'}