<jupyter_start><jupyter_text>Feature scaling with sklearn - Exercise You are given a real estate dataset. Real estate is one of those examples that every regression course goes through as it is extremely easy to understand and there is a (almost always) certain causal relationship to be found.The data is located in the file: 'real_estate_price_size_year.csv'. You are expected to create a multiple linear regression (similar to the one in the lecture), using the new data. This exercise is very similar to a previous one. This time, however, **please standardize the data**.Apart from that, please:-  Display the intercept and coefficient(s)-  Find the R-squared and Adjusted R-squared-  Compare the R-squared and the Adjusted R-squared-  Compare the R-squared of this regression and the simple linear regression where only 'size' was used-  Using the model make a prediction about an apartment with size 750 sq.ft. from 2009-  Find the univariate (or multivariate if you wish - see the article) p-values of the two variables. What can you say about them?-  Create a summary table with your findingsIn this exercise, the dependent variable is 'price', while the independent variables are 'size' and 'year'.Good luck!  Import the relevant libraries<jupyter_code># For these lessons we will need NumPy, pandas, matplotlib and seaborn
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
sns.set()

# and of course the actual regression (machine learning) module
from sklearn.linear_model import LinearRegression<jupyter_output><empty_output><jupyter_text>Load the data<jupyter_code># Load the data from a .csv in the same folder
data = pd.read_csv('real_estate_price_size_year.csv')

# Let's explore the top 5 rows of the df
data.head()<jupyter_output><empty_output><jupyter_text>Create the regression  Declare the dependent and the independent variables<jupyter_code># There are two independent variables: 'size' and 'year'
x = data[['size','year']]

# and a single dependent variable: 'price'
y = data['price']<jupyter_output><empty_output><jupyter_text>Scale the inputs<jupyter_code># Import the preprocessing module
# StandardScaler is one of the easiest and 'cleanest' ways to preprocess your data
from sklearn.preprocessing import StandardScaler
# Create an instance of the StandardScaler class
scaler = StandardScaler()
scaler.fit(x)
x_scaled = scaler.transform(x)
x_scaled<jupyter_output><empty_output><jupyter_text>Regression<jupyter_code># Creating a regression works in the exact same way
reg = LinearRegression()

# We just need to specify that our inputs are the 'scaled inputs'
reg.fit(x_scaled,y)<jupyter_output><empty_output><jupyter_text>Find the intercept<jupyter_code># Let's see the coefficients
reg.coef_<jupyter_output><empty_output><jupyter_text>Find the coefficients<jupyter_code># And the intercept
reg.intercept_<jupyter_output><empty_output><jupyter_text>Calculate the R-squared<jupyter_code>reg.score(x_scaled,y)<jupyter_output><empty_output><jupyter_text>Calculate the Adjusted R-squared<jupyter_code>x_scaled.shape
r2 = reg.score(x_scaled,y)
n = x_scaled.shape[0]
p = x_scaled.shape[1]

adjusted_r2 = 1-(1-r2)*(n-1)/(n-p-1)
adjusted_r2<jupyter_output><empty_output><jupyter_text>Compare the R-squared and the Adjusted R-squared It seems the the R-squared is only slightly larger than the Adjusted R-squared, implying that we were not penalized a lot for the inclusion of 2 independent variables.   Compare the Adjusted R-squared with the R-squared of the simple linear regression Comparing the Adjusted R-squared with the R-squared of the simple linear regression (when only 'size' was used - a couple of lectures ago), we realize that 'Year' is not bringing too much value to the result.  Making predictionsFind the predicted price of an apartment that has a size of 750 sq.ft. from 2009.<jupyter_code># For simplicity, let's crete a new dataframe with 2 *new* observations
new_data = pd.DataFrame(data=[[750,2009]],columns=['size','year'])
new_data
# We can make a prediction for a whole dataframe (not a single value)
# Note that the output is very strange (different from mine)
reg.predict(new_data)
# Our model is expecting SCALED features (features of different magnitude)
# In fact we must transform the 'new data' in the same way as we transformed the inputs we train the model on
# Luckily for us, this information is contained in the 'scaler' object
# We simply transform the 'new data' using the relevant method
new_data_scaled = scaler.transform(new_data)

# Let's check the result
new_data_scaled
# Finally we make a prediction using the scaled new data
reg.predict(new_data_scaled)
# The output is much more appropriate, isn't it?<jupyter_output><empty_output><jupyter_text>Calculate the univariate p-values of the variables<jupyter_code># Import the feature selection module from sklearn
# This module allows us to select the most appopriate features for our regression
# There exist many different approaches to feature selection, however, we will use one of the simplest
from sklearn.feature_selection import f_regression
# We will look into: f_regression
# f_regression finds the F-statistics for the *simple* regressions created with each of the independent variables
# In our case, this would mean running a simple linear regression on price where size and year
# are the independent variables
# The limitation of this approach is that it does not take into account the mutual effect of the two features
f_regression(x_scaled,y)

# There are two output arrays
# The first one contains the F-statistics for each of the regressions
# The second one contains the p-values of these F-statistics
# Since we are more interested in the latter (p-values), we can just take the second array
p_values = f_regression(x,y)[1]
p_values
# To be able to quickly evaluate them, we can round the result to 3 digits after the dot
p_values.round(3)<jupyter_output><empty_output><jupyter_text>Create a summary table with your findings<jupyter_code># Let's create a new data frame with the names of the features
reg_summary = pd.DataFrame(data = x.columns.values, columns=['Features'])
reg_summary ['Coefficients'] = reg.coef_
reg_summary ['p-values'] = p_values.round(3)
reg_summary<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06781279847182425, 'fraction_numerical': 0.0060490289716650745, 'mean_word_length': 5.741416309012876, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 49, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '285348', 'license': 'MIT', 'n_tokens_mistral': 1902, 'n_tokens_neox': 1777, 'n_words': 782, 'path': 'references/exercises/Feature Scaling Exercise/sklearn - Feature Scaling Exercise.ipynb', 'repo_name': 'Hesbon5600/data-science-basics', 'size': '6282'}