<jupyter_start><jupyter_text>Intermediate Linear Regression Practice Use a Linear Regression model to get the lowest RMSE possible on the following dataset:[Dataset Folder](https://github.com/ryanleeallred/datasets/tree/master/Ames%20Housing%20Data)[Raw CSV](https://raw.githubusercontent.com/ryanleeallred/datasets/master/Ames%20Housing%20Data/train.csv) Your model must include (at least):- A log-transformed y variable- Two polynomial features- One interaction feature- 10 other engineered featuresWhat is the lowest Root-Mean-Squared Error that you are able to obtain? Share your best RMSEs in Slack!Notes:There may be some data cleaning that you need to do on some features of this dataset. Linear Regression will only accept numeric values and will not acceptNote* There may not be a clear candidate for an interaction term in this dataset. Include one anyway, sometimes it's a good practice for predictive modeling feature engineering in general.<jupyter_code>url = 'https://raw.githubusercontent.com/ryanleeallred/datasets/master/Ames%20Housing%20Data/train.csv'

df = pd.read_csv(url)
#columns to drop bc of NaNs:
drop_cols = ['Alley', 'Fence', 'MiscFeature']
df = df.drop(columns = drop_cols)
#Fill Nans for garage yr blt w/ YearBlt value:

nums = df[df.GarageYrBlt.isna() == True].index
for i in nums:
  df.at[i, 'GarageYrBlt'] = df.at[i, 'YearBuilt']
  
#drop 8 rows of masonry veneer area Nans:
df.dropna(subset=['MasVnrArea'], inplace=True)

#Fill NaNs for LotFrontage as 0:
df['LotFrontage'] = df['LotFrontage'].fillna(0)
numeric_columns = df.select_dtypes(include='number').drop(columns='Id').columns
numeric_columns
def run_linear_model(X, y):
    # Split into test and train data
    X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.80, test_size=0.20, random_state=42)

    # Fit model using train data
    model = LinearRegression()
    model.fit(X_train, y_train)

    # Make predictions using test features
    y_pred = model.predict(X_test)

    # Compare predictions to test target
    rmse = (np.sqrt(mean_squared_error(y_test, y_pred)))
    r2 = r2_score(y_test, y_pred)

    print('Root Mean Squared Error', rmse)
    print('R^2 Score', r2)
    print('Intercept', model.intercept_)
    coefficients = pd.Series(model.coef_, X_train.columns)
    print(coefficients.to_string())
#Pre-feature engineering baseline regression:

target= 'SalePrice'

X = df[numeric_columns].drop(columns=target)
y = df[target]

run_linear_model(X, y)<jupyter_output>Root Mean Squared Error 30491.521649630755
R^2 Score 0.8477464959400496
Intercept 220589.42269341327
MSSubClass        -173.645775
LotFrontage          1.754565
LotArea              0.381321
OverallQual      17393.481114
OverallCond       3663.422892
YearBuilt          313.271387
YearRemodAdd       228.352471
MasVnrArea          39.528342
BsmtFinSF1           8.775940
BsmtFinSF2          -2.772518
BsmtUnfSF           -1.446343
TotalBsmtSF          4.557080
1stFlrSF            13.099720
2ndFlrSF            13.393311
LowQualFinSF         7.322930
GrLivArea           33.815960
BsmtFullBath     10711.209239
BsmtHalfBath      2428.199879
FullBath          5700.736127
HalfBath         -3383.169793
BedroomAbvGr     -9849.144139
KitchenAbvGr    -12219.141455
TotRmsAbvGrd      4587.806384
Fireplaces        4654.336873
GarageYrBlt        -29.930401
GarageCars       12190.843114
GarageArea          -4.618454
WoodDeckSF          25.632564
OpenPorchSF        -13.474969
EnclosedPorch        9.782653[...]<jupyter_text>Engineer Some Features A log-transformed y variable<jupyter_code>df['ln_price'] = np.log(df['SalePrice'])<jupyter_output><empty_output><jupyter_text>Two polynomial features Find Curved Data:<jupyter_code>for feature in df[numeric_columns]:
    sns.scatterplot(x=feature, y=target, data=df, alpha=0.1)
    plt.show()
#candidates for curved data after visual inspection, add squared column:
#-OverallQual
#-GarageCars

df['quality_squared'] = df['OverallQual']**2
df['garage_cars_squared'] = df['GarageCars']**2<jupyter_output><empty_output><jupyter_text>Check if squaring the Quality feature increased predictive accuracy:<jupyter_code>X = df[['OverallQual']]
y = df['SalePrice']

run_linear_model(X, y)
X = df[['quality_squared']]
y = df['SalePrice']

run_linear_model(X, y)<jupyter_output>Root Mean Squared Error 42824.467576919145
R^2 Score 0.6996739216639173
Intercept 36248.37116356945
quality_squared    3717.658598<jupyter_text>Check if squaring the basement square footage increased predictive accuracy:<jupyter_code>X = df[['GarageCars']]
y = df['SalePrice']

run_linear_model(X, y)
X = df[['garage_cars_squared']]
y = df['SalePrice']

run_linear_model(X, y)<jupyter_output>Root Mean Squared Error 53022.91689699418
R^2 Score 0.5395989222623487
Intercept 107204.01645159317
garage_cars_squared    19976.136255<jupyter_text>One interaction feature<jupyter_code>#hmmmm<jupyter_output><empty_output><jupyter_text>10 other engineered features<jupyter_code>#age:

df['age'] = 2010 - df['YearBuilt']
#renovation_age:

df['renovation_age'] = 2010 - df['YearRemodAdd']
#yard size (ignores deck, pool, etc):

df['yard_size'] = df['LotArea'] - df['1stFlrSF']
#kitchen Quality:

kitchen_map = {'Ex':5, 'Gd':4, 'TA':3, 'Fa':2, 'Po':1}	
df['KitchenQual'] = df['KitchenQual'].replace(kitchen_map)
#fireplace quality:

fireplace_map = {'Ex':5, 'Gd':4, 'TA':3, 'Fa':2, 'Po':1, np.NaN: 0}
df['FireplaceQu'] = df['FireplaceQu'].replace(fireplace_map)
#garage type:

garage_map = {'Ex':5, 'Gd':4, 'TA':3, 'Fa':2, 'Po':1, np.NaN: 0}
df['GarageType'] = df['GarageType'].replace(garage_map) 
#garage quality:

df['GarageQual'] = df['GarageQual'].replace(garage_map) 
#garage condition:

df['GarageCond'] = df['GarageCond'].replace(garage_map) 
#garage finish:

finish_map = {'Fin':3, 'RFn': 2, 'Unf':1, np.NaN:0}
df['GarageFinish'] = df['GarageFinish'].replace(finish_map)
#Pool Qulaity:
  
pool_map = {'Ex':5, 'Gd':4, 'TA':3, 'Fa':2, 'Po':1, np.NaN: 0}
df['PoolQC'] =  df['PoolQC'].replace(pool_map)
#sale type:

sale_map = {'Normal': 3,
            'Partial': 3,
            'Abnorml': 1,
            'Family': 1,
            'AdjLand': 2,	
            'Alloca': 2}

df['sale_type'] = df['SaleCondition'].replace(sale_map)
#Lot shape:

shape_map = {'Reg': 4, 'IR1': 3, 'IR2': 2, 'IR3': 1}
df["LotShape"] = df["LotShape"].replace(shape_map)
#Street:
  
street_map = {'Grvl':1, 'Pave':2}
df['Street'] = df['Street'].replace(street_map)
numeric_columns = df.select_dtypes(include='number').drop(columns=['Id', 'SalePrice', 'ln_price']).columns
numeric_columns
target= 'ln_price'

X = df[numeric_columns]
y = df[target]

run_linear_model(X, y)
from sklearn.decomposition import PCA 
from sklearn.cluster import KMeans 
from sklearn.preprocessing import StandardScaler 
import seaborn as sns
%matplotlib inline

X = df[numeric_columns].values
scaled_X = StandardScaler().fit_transform(X)

covar_matrix = PCA(n_components = 30)

covar_matrix.fit(scaled_X)
                
variance = covar_matrix.explained_variance_ratio_ 

cumulative_var = np.cumsum(np.round(covar_matrix.explained_variance_ratio_, decimals=3)*100)
#plot it:

y_vals = [num for num in cumulative_var]
x_vals = [num for num in range(1,31)]

fig, ax = plt.subplots(figsize=(10,8))
ax.grid(True)
ax.set_title('Principal Components Cumulative Variance')
ax.set_ylabel('Cumulative Variance % Explained')
ax.set_xlabel('Principal Components')
sns.barplot(x=x_vals, y=y_vals, ax=ax, color='#769ddb')
plt.show()

#make pca df

pca = PCA(20)

principal_components = pca.fit_transform(scaled_X)

pc_df = pd.DataFrame(data = principal_components)
#combine pca and main df

combined = pc_df.join(df[numeric_columns].reset_index(drop=True))
target= 'ln_price'

X = combined
y = df[target]

run_linear_model(X, y)<jupyter_output>Root Mean Squared Error 0.13285651381722052
R^2 Score 0.893553676842148
Intercept -9575639083.312624
0                     -1.320419e+08
1                      7.324378e+06
2                      2.019490e+06
3                     -2.794592e+07
4                      3.179658e+06
5                      3.438620e+06
6                     -2.119812e+05
7                      5.286031e+06
8                      7.426538e+06
9                      2.842934e+06
10                     3.531286e+06
11                    -5.695468e+05
12                    -8.596206e+05
13                    -7.280779e+06
14                     3.411891e+06
15                     1.144646e+07
16                    -5.282038e+06
17                     1.308331e+06
18                     2.375875e+06
19                     5.763359e+06
MSSubClass            -1.504317e+05
LotFrontage            4.101426e+05
LotArea                4.189310e+04
Street                -8.138911e+05
LotShape              -1.375495e+07[...]
{'fraction_non_alphanumeric': 0.12091540475919389, 'fraction_numerical': 0.096550153706023, 'mean_word_length': 3.657476139978791, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '534431', 'license': 'MIT', 'n_tokens_mistral': 3778, 'n_tokens_neox': 3184, 'n_words': 689, 'path': 'module2-polynomial-log-linear-regression/polynomial-log-linear-regression-assignment.ipynb', 'repo_name': 'Nolanole/DS-Unit-2-Sprint-2-Regression', 'size': '8783'}