<jupyter_start><jupyter_text>Predicting Median House Value with Regression**Author**: <NAME> (<EMAIL>) IntroductionToday, we are going to be looking at the Boston Housing dataset, available [here](https://www.kaggle.com/c/boston-housing). Our goal is not to achieve a high score, but to explore the data and insight workflow. With that being said, today's learning method of choice will be **Linear Regression**. This is because Linear Regression is perfect for predicting quantitative values such as the **median value of homes** and is one of the more interpretable machine learning methods.Now, let's take care of some quick housekeeping.<jupyter_code>import sys
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
from sklearn import preprocessing
from sklearn.linear_model import Lasso
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
import warnings
warnings.filterwarnings('ignore')
import logging
logging.basicConfig(format='%(message)s', level=logging.INFO, stream=sys.stdout)
from boston_housing import data
from boston_housing import model<jupyter_output><empty_output><jupyter_text>We can now use pandas to import the dataset.<jupyter_code>df = pd.read_csv("/Volumes/2 TB Storage/datascience/boston_housing/exploration/data/train.csv")<jupyter_output><empty_output><jupyter_text>Let's take a look at the variables we will be dealing with.<jupyter_code>df.head()<jupyter_output><empty_output><jupyter_text>"medv" is the median value of owner-occupied homes in thousands of dollars. in a particular area. This is our "target" variable, the value that we want to predict using the other variables."ID" is simply an identification number. This will be removed from our set of variables as it does not provide any useful information about the houses in their respective areas."crim" is the per capita crime rate by town."zn" is the proportion of residential land zoned for lots over 25,000 sq. ft."indus" is proportion of non-retail business acres per town."chas" is a Charles River dummy variable. A dummy variable is a variable whose value represents whether or not something is happening. In this case, "chas"' value is 1 if the Charles River bounds the tract/area around the homes and 0 if not."nox" is the nitrogen dioxide concentration in parts per 10 million. This is likely an indicator of pollution in the area."rm" is the average number of rooms per home."age" is actually proportion of owner-occupied units built prior to 1940. It is **not** the average age of homes."dis" is a weighted mean of distances to five of Boston employment centers."rad" is an index of accessibility to radial highways. Because this is an index, its values most likely represent groupings instead of some numerical quantity. We will check for this soon and convert "rad" into a dummy variable if necessary."tax" is the full-value property-tax rate per $10,000."ptratio" is the student to teacher ratio by town."black" is calculated using the formula 1000(Bk - 0.63)^2, where Bk is the proportion of blacks by town. This is not a very ethical use of data, so we will be leaving this variable out of our predictor variables."lstat" is lower status of the population (percent). This is basically a measure of poverty.<jupyter_code>df = df.drop(['black', 'ID'], axis = 1)<jupyter_output><empty_output><jupyter_text>More HousekeepingBefore we can start doing any analysis, we have to verify a few more things about our data.First, we check for missing data.<jupyter_code>df.isna().sum()<jupyter_output><empty_output><jupyter_text>It is important to make sure that none of our data has missing entries. Our data manipulation package, pandas, handles missing data as if they were 0's. This leads to inaccurate/skewed predictions if left untreated. Normally we have come up with a way to "impute", or fill in, these missing values before moving on, however, none of our variables are missing any entries.Next, we have to make sure our variables are in suitable types.<jupyter_code>df.info()
df['chas'].unique()<jupyter_output><empty_output><jupyter_text>Recall that "chas" is supposed to be a dummy variable. That means its values must be 0 or 1, which is true in this case, so we will not have to make any changes to "chas".We previously pointed out that "rad" is an index and might need to be converted to a series of dummy variables. To check this, we will look at what unique values "rad" contains.<jupyter_code>df['rad'].unique()<jupyter_output><empty_output><jupyter_text>Notice that "rad" does not have very many unique values. These values are indices that indicate accessibility to radial highways. Each area with the same "rad" value has the same level of accessibility, so each index should be treated as its own group. We can achieve this by creating a series of dummy variables to represent each index.<jupyter_code>df['rad'] = df['rad'].astype('category')
rad_dummies = pd.get_dummies(df['rad'])
df = df.drop('rad',axis = 1)
rad_dummies.head()<jupyter_output><empty_output><jupyter_text>Let us elaborate more on the meaning of our dummy variables. Notice in the above table that the "0" entry has a value of 1 under 1 and 0 everywhere else. This means that "0" belongs in group 1 and not in any of the others. Similarly, "1" is a member of group 2 and not any of the others. Next, "2" is a member of group 3 and so on.Next, we need to check the distributions of our variables. We do this by looking at histograms of our variables.<jupyter_code>data.distmap(df)<jupyter_output><empty_output><jupyter_text>We are looking at the horizontal axes of these histograms. Notice that the ranges of our data vary significantly. This is important because it is difficult to compare values of different variables with different ranges. For example, let's look at a value of 25 in the context of both the "zn" and "indus" variables. At a purely quantitative level, 25 is equal to 25. However it is not that simple. 25 is on the lower end of "zn"'s range whereas 25 is on the upper end of "indus"' range. We have to reconcile this information in our analysis. In order to do so, we are going to "scale" our variables by mapping each point into the "standard normal" distribution. That is done with the following formula:$Z = \frac{X-\mu}{\sigma}$, where Z is the the value in the standard normal distribution, $\mu$ is the average value of a given variable, X is the particular value of the variable, and $\sigma$ is the standard deviation of the variable.Each of the Z values represents information about the original value's distance from the mean/average value of the variable. When we compare Z values, we are comparing relative distances rather than magnitudes, making comparisons more meaningful. This should also bring each of the variables' respective ranges closer together. This makes it much easier to compare values across variables.<jupyter_code>features = data.scaler(df)
data.distmap(features)<jupyter_output><empty_output><jupyter_text>As we can see, all of the ranges have become much closer in scale.Now that we've taken care of all of this, we can add our dummy variables to our data and move on.<jupyter_code>scaled_feats = features.drop('medv', axis = 1)
chas_medv = df.drop(scaled_feats.columns, axis = 1)

features = pd.concat([scaled_feats, chas_medv, rad_dummies], axis = 1)<jupyter_output><empty_output><jupyter_text>Checking for MulticolinearityNow we need to look at the interactions between our variables. This is important because we want to include as little redundant information as possible. Removing redundant information simplifies our model and makes it more generalizable.<jupyter_code>data.tol_vif_table(features, n = 11)<jupyter_output><empty_output><jupyter_text>These are the top 11 pairs of variables along with their correlation, tolerance, and VIF (Variance Inflation Factor) values.Correlation is a measure of what proportion of variance in one variable can be explained by the variance in the other variable. In other words, how much information in one variable can be adequately represented by another variable. This is what we mean when we talk about redundant information. If one variable can explain most of the information provided by another variable, it is reasonable to just get rid of the extraneous variable.Tolerance is calculated using the formula:Tolerance = $1-r^{2}$, where r is the correlation value. Tolerance < 0.1 tends to indicate problematic relationships.VIF is calculated using the formula:VIF = $\frac{1}{\text{Tolerance}}$. Similarly, VIF > 10 tends to be problematicUsing the guidelines specified, we do not have to worry about colinearity between our variables.  Dimensionality ReductionDimensionality reduction is another way in which we can simplify our model. As previously mentioned, we can look at the interactions between our variables to see if there are any we can get rid of. Now we can compare model performance and model complexity to simplify our model. To do this, we are going to be using another form of regression called LASSO.<jupyter_code>data.lasso_reduction(features)<jupyter_output><empty_output><jupyter_text>Judging by the graph, 15 seems to be the optimal amount of features to use. This is because when we use 19 features, the training and test scores are pretty similar, but we have to use 4 additional features. More features means greater odds that the model will overfit and generalize less accurately. So, we will consider just using the features from the 0.1 LASSO regression.<jupyter_code>lasso01_feats = data.get_lasso_coef(features, 0.1)<jupyter_output><empty_output><jupyter_text>Model CreationNow that we are done treating our data and simplifying our model, we can finally create our linear regression model. Compared to the rest of this notebook, this is probably the simplest part.<jupyter_code>lin_reg, ctable = model.run_linear_regression(features,lasso01_feats)<jupyter_output>The prediction accuracy with the linear regression model is 72.2%<jupyter_text>The expected prediction accuracy for the linear regression model we created is 72.2%. This is a moderately good prediction accuracy. However, we did not chose Linear Regression for its prediction accuracy, but rather for its interpretability. Now we are going to draw insight about our model by interpretting the coefficients of our model. Linear Regression InterpretationWhat do I mean by coefficients? The Linear Regression method is defined as such:$\hat{y} = \beta_{0}X_{0} + ... + \beta_{n}X_{n}$, where $\hat{y}$ is the predicted value, $X_{i}$ is the $i^{th}$ predictor variable, and $\beta_{i}$ is the coefficient for the $i^{th}$ variable.If we can understand what each $\beta_{i}$ means, then we can extract some insights about how each predictor variable affects the prediction. The following table shows each variable along with its coefficient.<jupyter_code>ctable<jupyter_output><empty_output><jupyter_text>We have to be aware of the fact that we scaled our predictors because this changes the way that we interpret our coefficients. Normally we would say that an increase of 1 unit of $X_{i}$ is associated with a $\beta_{i}$ change in $\hat{y}$. But because we have remapped each $X_{i}$ to $Z_{i}$, in the standard normal distribution, we now have:$Z_{i} = \frac{X{i}-\mu_{i}}{\sigma_{i}}$, where each $X_{i}$ is the original $i^{th}$ variable, $\mu_{i}$ is the mean/average of the $i^{th}$ variable, and $\sigma_{i}$ is the standard deviation of the $i^{th}$ variable.Then we will interpret each coefficient as follows: For each $\sigma_{i}$ increase in $X_{i}$, there is as associated $\beta_{i}$ change in $\hat{y}$. In other words, for each standard deviation change of an associated variable, there is a $\beta_{i}$ change in $\hat{y}$.<jupyter_code>model.coef_std_table(ctable, df, lasso01_feats, rad_dummies)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06850235057085292, 'fraction_numerical': 0.00755540631296172, 'mean_word_length': 5.370588235294117, 'pattern_counts': {'":': 0, '<': 71, '<?xml version=': 0, '>': 71, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '303732', 'license': 'MIT', 'n_tokens_mistral': 3223, 'n_tokens_neox': 3045, 'n_words': 1640, 'path': 'exploration/predicting_median_value_with_regression/Boston Housing Analysis.ipynb', 'repo_name': 'devinthai/boston_housing', 'size': '11931'}