<jupyter_start><jupyter_text>Predicting House Sale Prices This project utilizes a dataset listing residential sale prices in Ames, Iowa from 2006 to 2010. The data contains 2930 observations and 80 variables. The project will use linear regression to predict home sale prices.  The first step is to import the necessary libraries and functions, and to read the dataset in as a dataframe.<jupyter_code># import libraries and functions
import pandas as pd
pd.set_option("display.max_columns", None)
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import cross_val_score,  KFold
from sklearn import linear_model
from sklearn.linear_model import LinearRegression
from scipy.stats import pointbiserialr
import seaborn as sns


# read dataset in as a dataframe
houses = pd.read_csv("AmesHousing.tsv", delimiter="\t")

# assign houses dataframe to df (assign your chosen dataframe here)
df = houses<jupyter_output><empty_output><jupyter_text>House sale price is the target feature. I want to see what the mean sale price is in order to get an intuitive sense of the accuracy of the predictive model.<jupyter_code>average_saleprice = np.mean(df["SalePrice"])
average_saleprice<jupyter_output><empty_output><jupyter_text>The project will use three functions. Each will be built first, and then the final functions will be used together to run the model. The functions will transform the data, select the features, and fit the regression model.   Transform features<jupyter_code># create function to transform features
def transform_features(df):
    return df<jupyter_output><empty_output><jupyter_text>Select features<jupyter_code># create function to select features
def select_features(df):
    return df[["Gr Liv Area", "SalePrice"]]<jupyter_output><empty_output><jupyter_text>Train and test function<jupyter_code># create train and test function

def train_and_test(df):
    train = df[:1460]
    test = df[1460:]
    
    numeric_train = train.select_dtypes(include=['integer', 'float'])
    numeric_test = test.select_dtypes(include=['integer', 'float'])
    
    features = numeric_train.columns.drop(["SalePrice"])
    target = ("SalePrice")
    lr = linear_model.LinearRegression()
    lr.fit(train[features], train["SalePrice"])
    train_predictions = lr.predict(train[features])
    test_predictions = lr.predict(test[features])
    train_mse = mean_squared_error(train["SalePrice"], train_predictions)
    test_mse = mean_squared_error(test["SalePrice"],  test_predictions)
    train_rmse = np.sqrt(train_mse)
    test_rmse = np.sqrt(test_mse)
    return test_rmse

transform_df = transform_features(houses)
select_df = select_features(transform_df)
rmse = train_and_test(select_df)

rmse<jupyter_output><empty_output><jupyter_text>The initial, undeveloped model returns a root mean squared error of 57088. This represents the error in dollar amounts from the sale price. This error is equal to about 32% of the mean sale price. Not very good! The goal is to reduce this error as much as possible.  The next step is to transform the data, also known as feature engineering. This will transform the columns of the dataframe, which are the features or attributes of the data.  Feature Engineering 1. handle missing values 2. create new features 3. drop leaky or unneccessary features.<jupyter_code># create series object of count of missing values per column
df_null_counts = df.isnull().sum()
df_null_counts<jupyter_output><empty_output><jupyter_text>drop columns with > 25% missing values<jupyter_code>drop_mv_cols = df_null_counts[(df_null_counts > len(df) *.25)].sort_values()

df = df.drop(drop_mv_cols.index, axis=1)
df.isnull().sum()<jupyter_output><empty_output><jupyter_text>for numerical columns with < 5% missing values, replace missing values with mean values for that column<jupyter_code># filter series to select columns with < 5% missing values
df_missing_values = df[df_null_counts[(df_null_counts>0)
& (df_null_counts<len(df)*.05)].index]
print(df_missing_values.isnull().sum())


## replace missing values
#  select the columns from missing values that contain numeric values
numeric_mv_cols = df_missing_values.select_dtypes(include=["int", "float"]).isnull().sum()
# fill in these values with the column mean 
df = df.fillna(numeric_mv_cols.mean())
df.isnull().sum()<jupyter_output><empty_output><jupyter_text>create new features that the dataset lacks, that are likely to improve the model.<jupyter_code># create years until remodel feature
years_until_remod = df["Year Remod/Add"] - df["Year Built"]
# add this new column to cleaned dataframe
df['Years Until Remod'] = years_until_remod
df.head(1)
# create years since remodel feature
years_since_remod = df["Yr Sold"] - df["Year Remod/Add"]
# add this new column to cleaned dataframe
df["Years Since Remod"] = years_since_remod
df.head(1)
# check for negative values in new features
years_until_remod[years_until_remod<0]
years_since_remod[years_since_remod<0]
# remove rows with negative value from df, from new columns 
df = df.drop([850, 1702, 2180, 2181], axis=0)

# remove "Year Remod/Add" column as no longer necessary
df = df.drop(["Year Remod/Add"], axis=1)<jupyter_output><empty_output><jupyter_text>drop unnecessary and leaky columns (columns conflated with the target column)<jupyter_code># drop columns that are not useful for machine learning
df = df.drop(["PID", "Order"], axis=1)

# drop "leaky" columns that are conflated with the target column
df = df.drop(["Mo Sold", "Sale Condition", "Sale Type", "Yr Sold"], axis=1)
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 2926 entries, 0 to 2929
Data columns (total 72 columns):
MS SubClass          2926 non-null int64
MS Zoning            2926 non-null object
Lot Frontage         2926 non-null float64
Lot Area             2926 non-null int64
Street               2926 non-null object
Lot Shape            2926 non-null object
Land Contour         2926 non-null object
Utilities            2926 non-null object
Lot Config           2926 non-null object
Land Slope           2926 non-null object
Neighborhood         2926 non-null object
Condition 1          2926 non-null object
Condition 2          2926 non-null object
Bldg Type            2926 non-null object
House Style          2926 non-null object
Overall Qual         2926 non-null int64
Overall Cond         2926 non-null int64
Year Built           2926 non-null int64
Roof Style           2926 non-null object
Roof Matl            2926 non-null object
Exterior 1st         2926 non-null object
Exterior 2nd   [...]<jupyter_text>Update transform_features function<jupyter_code>def transform_features(df):
    df_null_counts = df.isnull().sum()

    drop_mv_cols = df_null_counts[(df_null_counts > len(df) *.25)].sort_values()
    df = df.drop(drop_mv_cols.index, axis=1)

    df_missing_values = df[df_null_counts[(df_null_counts>0)
    & (df_null_counts<len(df)*.05)].index]
    
    numeric_mv_cols = df_missing_values.select_dtypes(include=["int", "float"]).isnull().sum()
    df = df.fillna(numeric_mv_cols.mean())

    years_until_remod = df["Year Remod/Add"] - df["Year Built"]
    df['Years Until Remod'] = years_until_remod

    years_since_remod = df["Yr Sold"] - df["Year Remod/Add"]
    df["Years Since Remod"] = years_since_remod

    years_until_remod[years_until_remod<0]
    years_since_remod[years_since_remod<0]

    df = df.drop([850, 1702, 2180, 2181], axis=0)

    df = df.drop(["Year Remod/Add"], axis=1)

    df = df.drop(["PID", "Order"], axis=1)

    df = df.drop(["Mo Sold", "Sale Condition", "Sale Type", "Yr Sold"], axis=1)

    return df

# create function to select features
def select_features(df):
    return df[["Gr Liv Area", "SalePrice"]]

def train_and_test(df):
    train = df[:1460]
    test = df[1460:]
    
    numeric_train = train.select_dtypes(include=['integer', 'float'])
    numeric_test = test.select_dtypes(include=['integer', 'float'])
    
    features = numeric_train.columns.drop(["SalePrice"])
    target = ("SalePrice")
    lr = linear_model.LinearRegression()
    lr.fit(train[features], train["SalePrice"])
    train_predictions = lr.predict(train[features])
    test_predictions = lr.predict(test[features])
    train_mse = mean_squared_error(train["SalePrice"], train_predictions)
    test_mse = mean_squared_error(test["SalePrice"],  test_predictions)
    train_rmse = np.sqrt(train_mse)
    test_rmse = np.sqrt(test_mse)
    return test_rmse

df = pd.read_csv("AmesHousing.tsv", delimiter="\t")
transform_df = transform_features(df)
select_df = select_features(transform_df)
test_rmse = train_and_test(select_df)

test_rmse<jupyter_output><empty_output><jupyter_text>Using the updated transform features function, the error came down to 55285. At least it is in the right direction, but surely further refinement can improve on this. The next step is to work on the feature selection function.  Feature Selection Create dataframe for numerical features<jupyter_code>numerical_df = transform_df.select_dtypes(include=["int", "float"])
numerical_df.head(1)<jupyter_output><empty_output><jupyter_text>Explore correlations to target variable<jupyter_code># correlate features with target column (SalePrice)
corr_values = numerical_df.corr()["SalePrice"].abs().sort_values()
corr_values<jupyter_output><empty_output><jupyter_text>Look for collinearity among strong correlations<jupyter_code># look for correlations > .3
strong_corrs = corr_values[corr_values > .3]
# filter using indexes of strong_corrs
corrmat = numerical_df[strong_corrs.index].corr()
# visualize correlation matrix heatmap of columns in strong_corrs
sns.heatmap(corrmat)
# drop TotRmsAbvGrd and Garage Cars from numerical dataframe due to collinearity
transform_df = transform_df.drop(['Garage Cars', 'TotRms AbvGrd'], axis=1)
transform_df.head(1)
print(strong_corrs)
# drop features below the correlation threshold (.30 for now)
transform_df = transform_df.drop(corr_values[corr_values < .30].index, axis=1)
transform_df.head(1)<jupyter_output><empty_output><jupyter_text>Rescale features and drop those with low variance<jupyter_code># rescale features using min-max scaling
# and drop features with low variance

final_corr_cols = strong_corrs.drop(['Garage Cars', 'TotRms AbvGrd'])
features = final_corr_cols.index

# rescaling with min/max
unit_df = (numerical_df[features] - numerical_df[features].min())/ (numerical_df[features].max() - numerical_df[features].min())
# verify that rescaling results in 0/1 for features
print(unit_df.min())
print(unit_df.max())

features
# explore variances
print(unit_df.var().sort_values())
# 'Total Bsmt SF' and "BsmtFin SF 1" have the lowest variances, 
# but 'Total Bsmt SF' correlates highly with the target, 
# so it wil be retained. 'BsmtFin SF 1' will be dropped from features.
features = features.drop('BsmtFin SF 1')

features<jupyter_output><empty_output><jupyter_text>Handle categorical data<jupyter_code># create lists of numerical and categorial data columns
num_data = transform_df._get_numeric_data().columns.tolist()
cat_data = set(transform_df.columns) - set(num_data)
cat_data
## We don't want to keep categorical columns in which the vast majority of values are from a 
## single category, as this is akin to low variance within the column
# loop through categorical columns and find how many values exist 
# for each category per column
for i in cat_data:
    print(transform_df[i].value_counts())
# drop categorical columns in which ~ 95% of values are from a single category

transform_df = transform_df.drop(["Utilities", "Condition 2", 
                                  "Land Slope", "Street", "Heating", 
                                  "Roof Matl","Functional"], axis=1)

transform_df.head(1)
## transform columns to categorical data
# select text columns
text_cols = transform_df.select_dtypes(include=['object']).columns

# print the number of unique features per column
for col in text_cols:
    print(col+":", len(transform_df[col].unique()))
# convert text columns to categorical
for col in text_cols:
    transform_df[col] = transform_df[col].astype('category')<jupyter_output><empty_output><jupyter_text>transform categorical text columns to dummy coding values<jupyter_code>transform_df.head(1)
transform_df.info()
# convert columns in text_cols to dummy columns
# delete the original text columns from df
# add dummy columns to dataframe
dummy_cols = pd.DataFrame()
for col in text_cols:
    col_dummies = pd.get_dummies(transform_df[col])
    transform_df = pd.concat([transform_df, col_dummies], axis=1)
    del transform_df[col]
transform_df.head(1)
transform_df.info()
transform_df.head(1)<jupyter_output><empty_output><jupyter_text>Final functions The select features function will add a parameter to choose the correlation coefficient value used to select the features. The train and test function will utilize holdout validation, simple cross-validation, and k-folds validation to validate the accuracy of the model. We also will look at the standard deviations of the root mean squared errors for different number of folds.<jupyter_code># transform features
def transform_features(df): 
    
    df_null_counts = df.isnull().sum()

    drop_mv_cols = df_null_counts[(df_null_counts > len(df) *.25)].sort_values()
    df = df.drop(drop_mv_cols.index, axis=1)

    df_missing_values = df[df_null_counts[(df_null_counts>0)
    & (df_null_counts<len(df)*.05)].index]
    numeric_mv_cols = df_missing_values.select_dtypes(include=["int", "float"]).isnull().sum()
    df = df.fillna(numeric_mv_cols.mean())

    years_until_remod = df["Year Remod/Add"] - df["Year Built"]
    df['Years Until Remod'] = years_until_remod

    years_since_remod = df["Yr Sold"] - df["Year Remod/Add"]
    df["Years Since Remod"] = years_since_remod

    years_until_remod[years_until_remod<0]
    years_since_remod[years_since_remod<0]

    df = df.drop([850, 1702, 2180, 2181], axis=0)

    df = df.drop(["Year Remod/Add"], axis=1)

    df = df.drop(["PID", "Order"], axis=1)

    df = df.drop(["Mo Sold", "Sale Condition", "Sale Type", "Yr Sold"], axis=1)

    return df

# select features
def select_features(df, corr_coeff_threshold=0.30):
    
    numerical_df = df.select_dtypes(include=["int", "float"])
    corr_values = numerical_df.corr()["SalePrice"].abs().sort_values()
    strong_corrs = corr_values[corr_values > .3]
    corrmat = numerical_df[strong_corrs.index].corr()
    df = df.drop(['Garage Cars', 'TotRms AbvGrd'], axis=1)
    df = df.drop(corr_values[corr_values < corr_coeff_threshold].index, axis=1)
    final_corr_cols = strong_corrs.drop(['Garage Cars', 'TotRms AbvGrd'])
    features = final_corr_cols.index

    unit_df = (numerical_df[features] - numerical_df[features].min())/ (numerical_df[features].max() - numerical_df[features].min())
    features = features.drop('BsmtFin SF 1')
    
    num_data = df._get_numeric_data().columns.tolist()
    cat_data = set(df.columns) - set(num_data)
    df = df.drop(["Utilities", "Condition 2", "Land Slope", "Street", "Heating", "Roof Matl","Functional"], axis=1)
    
    text_cols = df.select_dtypes(include=['object']).columns
    for col in text_cols:
        df[col] = df[col].astype('category')
    df = pd.concat([df, pd.get_dummies(df.select_dtypes(include=['category']))], axis=1).drop(text_cols,axis=1)
        
    return df
# Train and test function
# create train and test function

def train_and_test(df, k=25):
    numeric_df = df.select_dtypes(include=['integer', 'float'])
    features = numeric_df.columns.drop("SalePrice")
    lr = linear_model.LinearRegression()
    
    # holdout validation
    if k==0:
    
        train = df[:1460]
        test = df[1460:]
        lr.fit(train[features], train["SalePrice"])
        predictions = lr.predict(test[features])
        mse = mean_squared_error(test["SalePrice"],  predictions)
        rmse = np.sqrt(mse)
        return rmse
    
    # simple cross-validation
    if k==1:
        
        # shuffle order of rows in df
        shuffled_df = df.sample(frac=1, )
        
        fold_one = df[:1460]
        fold_two = df[1460:]
        
        # train on fold_one, test on fold_two
        lr.fit(fold_one[features], fold_one["SalePrice"])
        predictions_1 = lr.predict(fold_two[features])
        mse_1 = mean_squared_error(fold_two["SalePrice"],  predictions_1)
        rmse_1 = np.sqrt(mse_1)
        
        # train on fold_two, test on fold_one
        lr.fit(fold_two[features], fold_two["SalePrice"])
        predictions_2 = lr.predict(fold_one[features])
        mse_2 = mean_squared_error(fold_one["SalePrice"],  predictions_2)
        rmse_2 = np.sqrt(mse_2)
        
        # average the rmse values
        rmse_avg = np.mean([rmse_1, rmse_2])
        print(rmse_1)
        print(rmse_2)
        return rmse_avg
    
    # k-fold cross-validation
    else:
        
        kf = KFold(n_splits=k, shuffle=True)
        rmse_values = []
        for train_index, test_index in kf.split(df):
            train = df.iloc[train_index]
            test = df.iloc[test_index]
            lr.fit(train[features], train["SalePrice"])
            predictions = lr.predict(test[features])
            mse = mean_squared_error(test["SalePrice"],  predictions)
            rmse = np.sqrt(mse)
            rmse_values.append(rmse)
       
        print(rmse_values)
        avg_rmse = np.mean(rmse_values)
        std_rmse = np.std(rmse_values)
        return rmse_values, avg_rmse, std_rmse


df=pd.read_csv("AmesHousing.tsv", delimiter="\t")
transform_df = transform_features(df)
select_df = select_features(transform_df)
rmse = train_and_test(select_df)

rmse


k_folds = list(range(2,26))
k_folds_mean = {}
k_folds_std = {}


# Apply function to k number of folds

for k in k_folds:
    result = train_and_test(select_df, k)
    
    k_folds_mean[k] = result[1]
    k_folds_std[k] = result[2]
    
print(k_folds_mean)
print(k_folds_std)

# find lowest mean and std
k_folds_mean.items()
print(min(k_folds_mean.items(), key=lambda x: x[1]))
k_folds_std.items()
print(min(k_folds_std.items(), key=lambda x: x[1]))
    
#### create chart to find lowest means and std to see optimal number of folds
# visualize the results
plt.figure(figsize=(16,10))

plt.plot(list(k_folds_mean.keys()), list(k_folds_mean.values()), label = "Mean")

plt.plot(list(k_folds_std.keys()), list(k_folds_std.values()), label = "STD", color='red')
plt.legend(loc='best')
plt.xticks(range(1,26))
plt.xlim(0.5,25.5)
plt.ylabel("RMSE", size=15)
plt.xlabel("Number of folds", size=15)
plt.title("K fold cross validation", size=20)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1070225169813339, 'fraction_numerical': 0.01823822003529978, 'mean_word_length': 4.459270072992701, 'pattern_counts': {'":': 1, '<': 86, '<?xml version=': 0, '>': 83, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '268003', 'license': 'Unlicense', 'n_tokens_mistral': 6417, 'n_tokens_neox': 6053, 'n_words': 1615, 'path': 'house_sale_prices_final_version.ipynb', 'repo_name': 'esprohge/esprohge.github.io', 'size': '18697'}