<jupyter_start><jupyter_text>02: Ames Preprocessing and Linear RegressionData taken from [Ames, Iowa: Alternative to the Boston Housing Data as anEnd of Semester Regression Project](http://jse.amstat.org/v19n3/decock.pdf) by <NAME> (hereinafter referred to as 'the author').<jupyter_code>import numpy as np
import pandas as pd

import matplotlib.pyplot as plt  
import seaborn as sns 
from scipy import stats

from sklearn.linear_model import LinearRegression, LassoCV, Lasso, RidgeCV, Ridge, ElasticNet, ElasticNetCV
from sklearn.compose import ColumnTransformer, TransformedTargetRegressor
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.metrics import mean_squared_error, r2_score
from sklearn.preprocessing import PolynomialFeatures, PowerTransformer, StandardScaler, OneHotEncoder, FunctionTransformer, QuantileTransformer, RobustScaler
from sklearn.pipeline import Pipeline
from sklearn.decomposition import PCA

%matplotlib inline<jupyter_output><empty_output><jupyter_text>import and explore initial correlations<jupyter_code>train = pd.read_csv('./data/train_eda.csv')
train.shape<jupyter_output><empty_output><jupyter_text>Per suggestion of the author, it's recommended to removedata points above 4000 square feet (`gr_liv_area`).<jupyter_code>train = train[train['gr_liv_area'] <= 4000]
train.shape
train.head()
features = train.drop(columns='saleprice')
target = train['saleprice']<jupyter_output><empty_output><jupyter_text>single linear regressions<jupyter_code>def adjusted_r2(r2, obs, feats):
    
    # adjust r2 value based on number of observations and number of features
    adjusted = 1 - (1 - r2) * (obs - 1) / (obs - feats - 1)
    
    return adjusted
def regression_stats(model, x, y, random_state=111):
    
    x_train, x_test, y_train, y_test = train_test_split(x, y, random_state=random_state, test_size=0.5)
    
    model.fit(x_train, y_train)
    y_hat = model.predict(x_test)
    
    adj = adjusted_r2(model.score(x_test, y_test), x_test.shape[0], x_test.shape[1])

    print(f'adjusted r-squared: {adj}')
    print(f'rmse: {mean_squared_error(y_test, y_hat) ** 0.5}')
    
    plt.figure(figsize=(10, 6))
    sns.scatterplot(
        x = y_hat,
        y = y_test - y_hat
    )
    plt.title('residuals vs. predicted', fontsize=12)
    plt.xlabel('predicted values', fontsize=10)
    plt.hlines(y = 0, xmin = y_hat.min(), xmax = y_hat.max(), color = "red")
    plt.show()

    return model
regression_stats(LinearRegression(), features[['overall_qual']], target)
regression_stats(LinearRegression(), features[['gr_liv_area']], target)<jupyter_output>adjusted r-squared: 0.4894869492205024
rmse: 55644.44765910586<jupyter_text>Our residuals do not appear to be:- random- evenly variant as a function of our predicted value- insensitive to outliersTherefore, we should attempt:- using better (or more) features- feature engineering- further preprocessing of our data  multiple linear regression<jupyter_code>regression_stats(LinearRegression(), features[['gr_liv_area', 'overall_qual']], target)
regression_stats(LinearRegression(), features[['gr_liv_area', 'overall_qual', 'exter_qual']], target)
regression_stats(LinearRegression(), features[features.describe().columns], target)<jupyter_output>adjusted r-squared: 0.8584357312127511
rmse: 28562.12740472718<jupyter_text>Going at this one-by-one is a time-consuming and probably fruitless task. We will discuss how to 'automate' this process later. The main takeaway is that we can see that adding in more features does improve our scores, However, the residuals are still non-homoskedastic (from the "banana" shape). We can attempt to fix this by performing a non-linear transformation of our target `saleprice`.<jupyter_code>regression_stats(TransformedTargetRegressor(
                    regressor = LinearRegression(),
                    transformer = PowerTransformer()),
                 features[features.describe().columns],
                 target)<jupyter_output>adjusted r-squared: 0.8550790174358185
rmse: 28898.77022691465<jupyter_text>We see improvement in the r-squared value, and the residuals have a more even variance.  using categorical features and more transformations<jupyter_code>features.head()
features.isna().sum().sum()
features.select_dtypes(exclude=['O']).columns.values
features.select_dtypes(include=['O']).columns.values
features.describe()
for col in features.select_dtypes(include=['O']).columns.values:
    print(features[col].value_counts())
    print()
num_trans = Pipeline([
    ('scale', RobustScaler())
])

cat_trans = Pipeline([
    ('onehot', OneHotEncoder(categories="auto",
                             handle_unknown='ignore',
                             sparse=False))
])

preprocessor = ColumnTransformer(transformers=[
    ('num', num_trans, features.select_dtypes(exclude=['O']).columns.values),
    ('cat', cat_trans, features.select_dtypes(include=['O']).columns.values)
])

tt = TransformedTargetRegressor(
                    regressor = LinearRegression(),
                    func = np.log1p,
                    inverse_func = np.expm1)

pipe = Pipeline([
    ('preprocessor', preprocessor),
    ('estimator', tt)
])
regression_stats(pipe, features, target)
num_trans = Pipeline([
    ('scale', RobustScaler())
])

cat_trans = Pipeline([
    ('onehot', OneHotEncoder(categories="auto",
                             handle_unknown='ignore',
                             sparse=False))
])

preprocessor = ColumnTransformer(transformers=[
    ('num', num_trans, features.select_dtypes(exclude=['O']).columns.values),
    ('cat', cat_trans, features.select_dtypes(include=['O']).columns.values)
])

tt = TransformedTargetRegressor(
                    regressor = Ridge(),
                    func = np.log1p,
                    inverse_func = np.expm1)

pipe = Pipeline([
    ('preprocessor', preprocessor),
    ('estimator', tt)
])

regression_stats(pipe, features, target)<jupyter_output>adjusted r-squared: 0.8965317356636089
rmse: 24115.05990948539
{'fraction_non_alphanumeric': 0.11212473046939792, 'fraction_numerical': 0.029192237518659812, 'mean_word_length': 4.4471544715447155, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '160149', 'license': 'MIT', 'n_tokens_mistral': 2028, 'n_tokens_neox': 1851, 'n_words': 442, 'path': 'old/02_preprocessing-and-linear-regression.ipynb', 'repo_name': 'jydiw/ames-housing', 'size': '6035'}