<jupyter_start><jupyter_text>First things first, let's load the dataset and have some basic viewing/checks<jupyter_code>#Load the dataset
df_train = pd.read_csv('C:\\Users\\tayat\\Documents\\udacity\\Project 1\\house-prices-advanced-regression-techniques\\train.csv')
#Check the dataset shape
df_train.shape
#Set panadas viewing options to be able to see the whole columns
pd.set_option('display.max_columns', 82)
pd.set_option('display.max_rows', 100)
df_train.head(10)
#List the columns names with thier datatypes
df_train.dtypes
#Describe the dataset numerical columns
df_train.describe()<jupyter_output><empty_output><jupyter_text>Some features were identified as numerical, but they are categorical, let's convet them to categorical<jupyter_code>df_train = df_train.astype({'MSSubClass': 'object', 'OverallQual': 'object', 'OverallCond': 'object'})<jupyter_output><empty_output><jupyter_text>Let's view missing values per feature<jupyter_code>datamissing = pd.concat([df_train.isnull().sum(), (df_train.isnull().sum()/df_train.isnull().count())], axis=1, keys=['MissingCount', 'Percent'])
datamissing.head(100)<jupyter_output><empty_output><jupyter_text>First business question: - What is the best sale time along the year in terms of price and opportunity to sell? Seems that sale price does not change so much over year months, but on the other hand the number of sales is much higher in May, Jun and Jul which indicates that this is the best time for owners who require fast selling<jupyter_code>#let's have a look at the sale price over time (selling year and month)
df_train.groupby(['YrSold','MoSold']).mean()['SalePrice'].plot(kind="bar", figsize = (30, 8));
plt.title("Average Price Per MoSold");
#let's have a look at the count of sold units over time (selling year and month)
df_train.groupby(['YrSold','MoSold']).count()['SalePrice'].plot(kind="bar", figsize = (30, 8));
plt.title("No. of sales over time");
#let's have a look at the count of sold units over sold_month
df_train.groupby(['MoSold']).count()['SalePrice'].plot(kind="bar", figsize = (10, 5));
plt.title("No. of sales over months");<jupyter_output><empty_output><jupyter_text>Second business question: - Which neighborhoods has the highest count of sold units? Below count of sales per Neighborhood indicates that some Neighborhoods like "North Ames" has higher demand than other Neighborhoods like "Bluestem"<jupyter_code>#let's have a look at the count of sold units per different Neighborhoods
df_train.groupby(['Neighborhood']).count()['SalePrice'].sort_values(ascending=False).plot(kind="bar", figsize = (15, 5));
plt.title("No. of sales per Neighborhood");<jupyter_output><empty_output><jupyter_text>Third question: - What features affect the sale price and how can we predict it?  Let's think of factors that may affect the sales price, I have the following factors in mind (Building Type, Area, Location, quality of building and finishing, utilities, How old it is, has parking or not, Type of sale) The features that reflect the above factors are: MSSubClass, MSZoning, Utilities, Neighborhood, BldgType, HouseStyle, OverallCond, HeatingQC, CentralAir, SaleType, YearBuilt --> these are categorical features GrLivArea, TotRmsAbvGrd, GarageArea  Now let's have a look at the salesprice distribution (somehow if follow a normal distribution)<jupyter_code>sns.distplot(df_train['SalePrice'], bins=20, kde=False, norm_hist=False).set_title('SalePrice Histogram');<jupyter_output><empty_output><jupyter_text>let's have a look at the above features that we expect to affect the sale price<jupyter_code># almost half the dataset are types 20 (1-STORY 1946 & NEWER ALL STYLES) which lies in the upper quartile of average prices 
# and 60 (2-STORY 1946 & NEWER) which have the highest average price
# both of them are new building (newer than 1946) which might indicate that YearBuilt will be a good feature

df_train.groupby(['MSSubClass']).mean()['SalePrice'].sort_values(ascending=False).plot(kind="bar");
plt.title("Avg Sale Price Per MSSubClass");
df_train['MSSubClass'].value_counts().sort_values(ascending=False)
# Most of the assets are 1Fam "Single-family"
# the average price for "TwnhsI", "Duplx", "2FmCon" almsot have the same average price with low count
# This might not be a good feature to represent the sales price

df_train.groupby(['BldgType']).mean()['SalePrice'].sort_values(ascending=False).plot(kind="bar");
plt.title("Avg Sale Price Per BldgType");
df_train['BldgType'].value_counts().sort_values(ascending=False)
# FV "Floating Village Residential" is the most repeated with highest average price
# Although most of the assets are FV, the average price vary from zone to another
# Might be a good feature to consider

df_train.groupby(['MSZoning']).mean()['SalePrice'].sort_values(ascending=False).plot(kind="bar");
plt.title("Avg Sale Price Per MSZoning");
df_train['MSZoning'].value_counts().sort_values(ascending=False)
# We can notice that the average price vary by neighborhood
# Might be a good feature that affects the sale price

df_train.groupby(['Neighborhood']).mean()['SalePrice'].sort_values(ascending=False).plot(kind="bar");
plt.title("Avg Sale Price Per Neighborhood");
df_train['Neighborhood'].value_counts().sort_values(ascending=False)
# All the assets except one has all utilities, so I'll drop this feature

df_train.groupby(['Utilities']).mean()['SalePrice'].sort_values(ascending=False).plot(kind="bar");
plt.title("Avg Sale Price Per Utilities");
df_train['Utilities'].value_counts().sort_values(ascending=False)
# From the below we notice that OverallQual has strong relation with the price more than OverallCond
# I'll consider OverallQual as a feature

df_train.groupby(['OverallCond']).mean()['SalePrice'].sort_values(ascending=False).plot(kind="bar");
plt.title("Avg Sale Price Per OverallCond");
df_train['OverallCond'].value_counts().sort_values(ascending=False)
df_train.groupby(['OverallQual']).mean()['SalePrice'].sort_values(ascending=False).plot(kind="bar");
plt.title("Avg Sale Price Per OverallQual");
df_train['OverallQual'].value_counts().sort_values(ascending=False)
# Average price vary per HeatingQC (Heating quality and condition)
# I'll consider this feature

df_train.groupby(['HeatingQC']).mean()['SalePrice'].sort_values(ascending=False).plot(kind="bar");
plt.title("Avg Sale Price Per HeatingQC");
df_train['HeatingQC'].value_counts().sort_values(ascending=False)
# Most of the assets have central AC, but the average price of Non-centraized AC assets is almot the half of assets with central AC
# I'll consider this feature

df_train.groupby(['CentralAir']).mean()['SalePrice'].sort_values(ascending=False).plot(kind="bar");
plt.title("Avg Sale Price Per CentralAir");
df_train['CentralAir'].value_counts().sort_values(ascending=False)
# Most of the assets sale type is "Warranty Deed - Conventional", but the average price vary alot between types
# I'll consider this feature

df_train.groupby(['SaleType']).mean()['SalePrice'].sort_values(ascending=False).plot(kind="bar");
plt.title("Avg Sale Price Per SaleType");
df_train['SaleType'].value_counts().sort_values(ascending=False)
# We can see that the prices of new houses are higher in general than old houses
# I'll consider this feature

df_train.groupby(['YearBuilt']).mean()['SalePrice'].plot(kind="bar", figsize = (30, 8));
plt.title("Avg Sale Price Per YearBuilt");
df_train['YearBuilt'].value_counts().sort_values(ascending=False)
YearBuilt_Sale = pd.concat([df_train['SalePrice'], df_train['YearBuilt']], axis=1)
YearBuilt_Sale.plot.scatter(x='YearBuilt', y='SalePrice', ylim=(0,800000));
# GrLivArea has almost liner relation with the sale price with few exceptions
# I will consider this feature
# I might remove the outliers (very huge area more than 4000 and lower saleprice than 200000)

GrLivArea_Sale = pd.concat([df_train['SalePrice'], df_train['GrLivArea']], axis=1)
GrLivArea_Sale.plot.scatter(title='SalePrice vs. GrLiveArea', x='GrLivArea', y='SalePrice', ylim=(0,800000));
# GarageArea has almost liner relation with the sale price with some exceptions
# Some assets have no garage, and in this case the sales price also vary (of course due to other factors)
# I will consider this feature

GarageArea_Sale = pd.concat([df_train['SalePrice'], df_train['GarageArea']], axis=1)
GarageArea_Sale.plot.scatter(title='SalePrice vs. GarageArea', x='GarageArea', y='SalePrice', ylim=(0,800000));
# From the below correlation matrix with numerical features we can find that the sale price is correlated with:
# - area (I have selected GrLiveArea to represent this factor)
# - YearBuilt
# - YearRemodAdd (We can consider it as well)
# - GarageArea and GarageCars (I have selected GarageArea to represent this factor)

corr = df_train.corr()
f, ax = plt.subplots(figsize=(12, 9))
sns.heatmap(corr, vmax=1, square=True, xticklabels=True, yticklabels=True);

df_train.groupby(['YearRemodAdd']).mean()['SalePrice'].plot(kind="bar", figsize = (30, 8));
plt.title("Avg Sale Price per YearRemodAdd");
df_train['YearRemodAdd'].value_counts().sort_values(ascending=False)
#Now let's filter the selected features only
df_train_small = df_train[['MSZoning', 'Neighborhood', 'OverallQual', 'HeatingQC', 'CentralAir', 'SaleType', 'YearBuilt', 'YearRemodAdd', 'GrLivArea', 'GarageArea', 'SalePrice']]
#Let's have another look at the selected features
#SalePrice has no -ve values
#No -ve values detected in the numerical features which is logic according to their meaning
#Min(YearRemodAdd) > Min(YearBuilt) which is logic
df_train_small.describe()
#No missing values detected in the selected features
datamissing = pd.concat([df_train_small.isnull().sum(), (df_train_small.isnull().sum()/df_train_small.isnull().count())], axis=1, keys=['MissingCount', 'Percent'])
datamissing.head(100)
#Let's clean the data from the outliers (from the GrLivArea_SalePrice analysis we did before)
df_train_small.drop(df_train_small[ (df_train_small['GrLivArea'] > 4000) & (df_train_small['SalePrice'] < 200000)].index, inplace = True)
df_train_small.shape<jupyter_output><empty_output><jupyter_text>Now let's go to modeling<jupyter_code>def create_dummy_df(df, dummy_na):
    '''
    INPUT:
    df - pandas dataframe with categorical variables you want to dummy
    cat_cols - list of strings that are associated with names of the categorical columns
    dummy_na - Bool holding whether you want to dummy NA vals of categorical columns or not
    
    OUTPUT:
    df - a new dataframe that has the following characteristics:
            1. contains all columns that were not specified as categorical
            2. removes all the original columns in cat_cols
            3. dummy columns for each of the categorical columns in cat_cols
            4. if dummy_na is True - it also contains dummy columns for the NaN values
            5. Use a prefix of the column name with an underscore (_) for separating 
    '''
    cat_cols = df.select_dtypes(include=['object']).columns
    
    for col in  cat_cols:
        try:
            # for each cat add dummy var, drop original column
            df = pd.concat([df.drop(col, axis=1), pd.get_dummies(df[col], prefix=col, prefix_sep='_', drop_first=True, dummy_na=dummy_na)], axis=1)
        except:
            continue
    return df
def coef_weights(coefficients, X_train):
    '''
    INPUT:
    coefficients - the coefficients of the linear model 
    X_train - the training data, so the column names can be used
    OUTPUT:
    coefs_df - a dataframe holding the coefficient, estimate, and abs(estimate)
    
    Provides a dataframe that can be used to understand the most influential coefficients
    in a linear model by providing the coefficient estimates along with the name of the 
    variable attached to the coefficient.
    '''
    coefs_df = pd.DataFrame()
    coefs_df['est_int'] = X_train.columns
    coefs_df['coefs'] = coefficients
    coefs_df['abs_coefs'] = np.abs(coefficients)
    coefs_df = coefs_df.sort_values('abs_coefs', ascending=False)
    return coefs_df
def print_model_scores(model, X_train, X_test, y_train, y_test):
    '''
    INPUT:
    df - pandas dataframe with categorical variables you want to dummy
    cat_cols - list of strings that are associated with names of the categorical columns
    dummy_na - Bool holding whether you want to dummy NA vals of categorical columns or not
    
    OUTPUT:
    y_train_preds, y_test_preds
    print scores 
    '''
    
    #Predict and score the model on training data
    y_train_preds = model.predict(X_train) 
    print("The r-squared score for our model on Training data is {} on {} values.".format(r2_score(y_train, y_train_preds), len(y_train)))
    print("The mean_squared_error score for our model on Training data is {} on {} values.".format(mean_squared_error(y_train, y_train_preds), len(y_train)))
    print("The root_mean_squared_error score for our model on Training data is {} on {} values.".format(sqrt(mean_squared_error(y_train, y_train_preds)), len(y_train)))

    print(" ")
    #Predict and score the model on test data
    y_test_preds = model.predict(X_test) 
    print("The r-squared score for our model on Testing data is {} on {} values.".format(r2_score(y_test, y_test_preds), len(y_test)))
    print("The mean_squared_error score for our model on Testing data is {} on {} values.".format(mean_squared_error(y_test, y_test_preds), len(y_test)))
    print("The root_mean_squared_error score for our model on Testing data is {} on {} values.".format(sqrt(mean_squared_error(y_test, y_test_preds)), len(y_test)))
    
    return y_train_preds, y_test_preds
df_train_small_cat = create_dummy_df(df_train_small, dummy_na=False)
df_train_small_cat.shape
#Split into explanatory and response variables
y = df_train_small_cat['SalePrice']
X = df_train_small_cat.drop('SalePrice', axis=1)
#Split into train and test
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = .30, random_state=42)
lm_model = LinearRegression(normalize=True) # Instantiate
lm_model.fit(X_train, y_train) #Fit
#Predict and score the model on training and test data
y_train_preds, y_test_preds = print_model_scores(lm_model, X_train, X_test, y_train, y_test)
#Let's get the coef_weights for different features
coef_df = coef_weights(lm_model.coef_, X_train)

#A quick look at the top results
coef_df.head(20)<jupyter_output><empty_output><jupyter_text>Let's try another model (XGBoost)<jupyter_code>xg_reg = xgb.XGBRegressor(objective ='reg:linear', colsample_bytree = 0.9, learning_rate = 0.1,vmax_depth = 5, alpha = 10, n_estimators = 100) # Instantiate
xg_reg.fit(X_train, y_train) #Fit
#Predict and score the model on training and test data
y_train_preds, y_test_preds = print_model_scores(xg_reg, X_train, X_test, y_train, y_test)<jupyter_output>The r-squared score for our model on Training data is 0.9776200730638545 on 1020 values.
The mean_squared_error score for our model on Training data is 146091268.24193624 on 1020 values.
The root_mean_squared_error score for our model on Training data is 12086.822090274029 on 1020 values.
 
The r-squared score for our model on Testing data is 0.8804069077263292 on 438 values.
The mean_squared_error score for our model on Testing data is 695271228.8482451 on 438 values.
The root_mean_squared_error score for our model on Testing data is 26367.99629945827 on 438 values.
{'fraction_non_alphanumeric': 0.10784058348528262, 'fraction_numerical': 0.01634540244855431, 'mean_word_length': 5.765198237885462, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 39, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '413169', 'license': 'CNRI-Python', 'n_tokens_mistral': 5093, 'n_tokens_neox': 4729, 'n_words': 1534, 'path': 'House Sales Analysis.ipynb', 'repo_name': 'telayat/House-Prices-Prediction', 'size': '15356'}