<jupyter_start><jupyter_text>Machine Learning Foundation Section 2, Part a: Regression Intro: Transforming Target   Learning objectivesBy the end of this lesson, you will be able to:* Apply transformations to make target variable more normally distributed for regression* Apply inverse transformations to be able to use these in a regression context<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline

from helper import (plot_exponential_data, 
                    plot_square_normal_data)

import warnings
warnings.simplefilter("ignore")<jupyter_output><empty_output><jupyter_text>Loading in Boston Data **Note:** See `helper.py` file to see how boston data is read in from SciKit Learn.<jupyter_code>from helper import boston_dataframe
boston_data = boston_dataframe()
boston_data.head(15)<jupyter_output><empty_output><jupyter_text>Determining Normality Making our target variable normally distributed often will lead to better resultsIf our target is not normally distributed, we can apply a transformation to it and then fit our regression to predict the transformed values.How can we tell if our target is normally distributed? There are two ways:* Visually* Using a statistical test  Visually Plotting a histogram:<jupyter_code>boston_data.MEDV.hist();<jupyter_output><empty_output><jupyter_text>Does not look normal due to that right tail. Let's try to verify statistically:<jupyter_code>from scipy.stats.mstats import normaltest # D'Agostino K^2 Test<jupyter_output><empty_output><jupyter_text>Without getting into Bayesian vs. frequentist debates, for the purposes of this lesson, the following will suffice:* This is a statistical test that tests whether a distribution is normally distributed or not. It isn't perfect, but suffice it to say:     * This test outputs a "p-value". The _higher_ this p-value is the _closer_ the distribution is to normal.    * Frequentist statisticians would say that you accept that the distribution is normal (more specifically: fail to reject the null hypothesis that it is normal) if p > 0.05.<jupyter_code>normaltest(boston_data.MEDV.values)<jupyter_output><empty_output><jupyter_text>p-value _extremely_ low. Our y variable we've been dealing with this whole time was not normally distributed! Linear Regression assumes a normally distributed residuals which can be aided by transforming y variable. Let's try some common transformations to try and get y to be normally distributed: * Log* Square root* Box cox  Testing log The log transform can transform data that is significantly skewed right to be more normally distributed:<jupyter_code>data = plot_exponential_data()
plt.hist(np.log(data));<jupyter_output><empty_output><jupyter_text>**Apply transform to Boston data:**<jupyter_code>log_medv = np.log(boston_data.MEDV)
log_medv.hist();
normaltest(log_medv)<jupyter_output><empty_output><jupyter_text>Conclusion: closer, but still not normal.  Exercise: The square root transformation is another transformation that can transform non-normally distributed data into normally distributed data:<jupyter_code>data = plot_square_normal_data()<jupyter_output><empty_output><jupyter_text>Slightly skewed right.<jupyter_code>plt.hist(np.sqrt(data));<jupyter_output><empty_output><jupyter_text>Apply the square root transformation to the Boston data target and test whether the result is normally distributed.<jupyter_code>pass # your code here
sqrt_medv = np.sqrt(boston_data.MEDV)
# Instructor Solution

sqrt_medv = np.sqrt(boston_data.MEDV)
plt.hist(sqrt_medv)
normaltest(sqrt_medv)<jupyter_output><empty_output><jupyter_text>Box cox The box cox transformation is a parametrized transformation that tries to get distributions "as close to a normal distribution as possible".It is defined as:$$ \text{boxcox}(y_i) = \frac{y_i^{\lambda} - 1}{\lambda} $$You can think of as a generalization of the square root function: the square root function uses the exponent of 0.5, but box cox lets its exponent vary so it can find the best one.<jupyter_code>from scipy.stats import boxcox
bc_result = boxcox(boston_data.MEDV)
boxcox_medv = bc_result[0]
lam = bc_result[1]
lam
boston_data['MEDV'].hist();
plt.hist(boxcox_medv);
normaltest(boxcox_medv)<jupyter_output><empty_output><jupyter_text>Significantly more normally distributed (according to p value) than the other two distributions - above 0.05, even!Now that we have a normally distributed y-variable, let's try a regression!  Testing regression:<jupyter_code>from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import (StandardScaler, 
                                   PolynomialFeatures)
lr = LinearRegression()<jupyter_output><empty_output><jupyter_text>**Reload clean version of `boston_data`:**<jupyter_code>boston_data = boston_dataframe()<jupyter_output><empty_output><jupyter_text>Same steps as before. **Create X and y**<jupyter_code>y_col = "MEDV"

X = boston_data.drop(y_col, axis=1)
y = boston_data[y_col]<jupyter_output><empty_output><jupyter_text>**Create Polynomial Features**<jupyter_code>X.shape
pf = PolynomialFeatures(degree=2, include_bias=False)
X_pf = pf.fit_transform(X)
X_pf.shape<jupyter_output><empty_output><jupyter_text>**Train test split**<jupyter_code>X_train, X_test, y_train, y_test = train_test_split(X_pf, y, test_size=0.3, random_state=72018)<jupyter_output><empty_output><jupyter_text>**Fit `StandardScaler` on `X_train` as before**<jupyter_code>X_train.shape
s = StandardScaler()
X_train_s = s.fit_transform(X_train)  # Standardize => substract mean and devide by stddev
X_train_s.shape<jupyter_output><empty_output><jupyter_text>**Discuss: what transformation do we need to apply next?**Apply the appropriate transformation.<jupyter_code>bc_result2 = boxcox(y_train) # your code here
# Instructor Solution
bc_result2 = boxcox(y_train)
y_train_bc = bc_result2[0]
lam2 = bc_result2[1]
lam2<jupyter_output><empty_output><jupyter_text>As before, we'll now:1. Fit regression1. Transform testing data1. Predict on testing data<jupyter_code>y_train_bc.shape
lr.fit(X_train_s, y_train_bc)
X_test_s = s.transform(X_test) # JUST TRANSFORM do not fit again => it uses mean and stddev from the trainings set 
y_pred_bc = lr.predict(X_test_s)
X_test_s.shape
y_pred_bc.shape<jupyter_output><empty_output><jupyter_text>Discussion* Are we done?* What did we predict?* How would you interpret these predictions?  Inverse transform Every transformation has an inverse transformation. The inverse transformation of $f(x) = \sqrt{x}$ is $f^{-1}(x) = x^2$, for example. Box cox has an inverse transformation as well: notice that we have to pass in the lambda value that we found from before:<jupyter_code>from scipy.special import inv_boxcox
# code from above
bc_result = boxcox(boston_data.MEDV)
boxcox_medv = bc_result[0]
lam = bc_result[1]
inv_boxcox(boxcox_medv, lam)[:10]
boston_data['MEDV'].values[:10]<jupyter_output><empty_output><jupyter_text>Exactly the same, as we would hope!  Exercise:1. Apply the appropriate inverse transformation to `y_pred_bc`.2. Calculate the $R^2$ using the result of this inverse transformation and `y_test`.  **Hint:** Should be two lines of code.<jupyter_code>y_pred_tran = inv_boxcox(y_pred_bc,lam2) # your code here
# Instructor Solution
y_pred_tran = inv_boxcox(y_pred_bc,lam2)
r2_score(y_pred_tran,y_test)<jupyter_output><empty_output><jupyter_text>LAB Exercise:  Determine the R^2 of a LinearRegression without the box cox transformation. Is it higher or lower?<jupyter_code>### BEGIN SOLUTION
lr = LinearRegression()
lr.fit(X_train_s,y_train)
lr_pred = lr.predict(X_test_s)
r2_score(lr_pred,y_test)
### END SOLUTION<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08236808236808237, 'fraction_numerical': 0.0066924066924066924, 'mean_word_length': 6.913441955193482, 'pattern_counts': {'":': 0, '<': 89, '<?xml version=': 0, '>': 92, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '22', 'dir': 'jupyter-structured-clean-dedup', 'id': '473387', 'license': 'MIT', 'n_tokens_mistral': 2533, 'n_tokens_neox': 2426, 'n_words': 755, 'path': '02 - Supervised Learning - Regression/02a_LAB_Transforming_Target.ipynb', 'repo_name': 'SamiIbishi/ml-engineering', 'size': '7770'}