<jupyter_start><jupyter_text>Implementing Advanced Regression  Techniques for Prediction:There are several factors that impact the overall price of the house, some of those factors are more **tangible** as the quality of the house or the overall size (area) of the house and other factors are more **intrinsic** such as the performance of the economy. Coming with an accurate model that predicts with such precision the actual value is an arduous task since there are both internal and external factors that will affect the price of a single house. Nevertheless, what we can do is **detect** those features that carry a heavier weight on the overall output (Price of the house). Before the housing crisis that occurred during the years of (2007-2008), most people believed that the prices of houses tended to go up throughout the years and that people that invested into properties were certain that they will get a return. This was not the case since banks were basically approving loans to people that were not able to afford to pay a house, there were even financial institutions who were approving loans to ordinary individuals at a variable interest rate (meaning rate will change depending on the current market rate) and when the crisis occurred lots of those ordinary individuals were not able to afford to pay back their mortgages. Of course, there were other reasons that caused the financial crisis in the first place such as the introduction of complex financial instruments (*derivatives are still not widely understood*), hedging financial instruments (credit default swaps), and the deregulation of the financial industry as a whole. While we can argue about the factors that caused the financial crisis, the main objective of this post is to determine what possible features could have a real impact on the overall value of a house. We will try to answer questions such as to what extent did the recession impacted the value house prices? What materials were most commonly used with houses that had a high price range? (Rooftop, walls etc.) Which neighborhoods were the most exclusive? I believe that in order to perform an extensive analysis of this data we should explore our data, by this I mean getting a sense of what is the **story behind the data**. Most of the time I tend to reject the idea of just building a model that have a good accuracy score for predicting values instead, I analyze my data carefully (determining distributions, missing values, visualizations) in order to have a better understanding of what is going on. ONly after my extensive analysis I proceed to developing the predictive model, in this case we will use **regression models.** The downside of this to many of you who will see this post, is that it will be somewhat long, so if you think you should **skip** all the sections and start from the regression model step, please feel free to do so! I will create an outline so it will help you find the section you wish to start with. **I'd rather have a full house at a medium price than a half-full at a high price. - <NAME>*****  Goal of this Project:*** Achieving our goal is split into two phases: 1) **Exploratory Data Analysis (EVA)**: In this phase our main aim is to have a better understanding of the features involved in our data. It might be possible that some are left behind but I will be focusing on the features that have the highest correlation towards SalePrice. 2) **Advanced Regression**: We will implement Regression model to predict a possible SalePrice (label) of the house.  Outline: ***I. **Understanding our Data**a) [Splitting into Different Categories](splitting)b) [Gathering Basic Insight](insight) II. **Economic Activity**III. [Outside Surroundings](outside)a) [Type of Zoning](zoning)b) [Neighborhoods](neighborhoods) IV. **Areas of the House** a) [The Impact of Space towards Price](space)V. **Building Characteristics**a) [Correlations with SalePrice](correlation)b) [What garages tell about House Prices?](garage)VI. **Miscellaneous and Utilities**a) [What determines the quality of the house?](quality)b) [Intersting insights](interesting)c) [Which Material Combination increased the Price of Houses?](material)VII. [Quality of Neighborhoods](quality_neighborhoods)VIII. **The Purpose of using Log Transformations** a)[Log Transformations](log_transformations)b) [Skewedness and Kurtosis](skew_kurt)c) [Outliers Analysis](analysis_outliers)d) [Bivariate Analysis](bivariate) IX. **Feature Engineering** a) [Dealing with Missing Values](missing_values)b) [Transforming Values](transforming_values)c) [Combining Attributes](combining_atributes) d) [Dealing with numerical and categorical values](num_cat_val) X. **Scaling** a) [Categorical Encoding Class](categorical_class)b) [Combine Attribute Class](combining)c) [Pipelines](combining)XI. **Predictive Models** a) [Residual Plot](residual_plot) b) [RandomForests Regressor](random_forest) c) [GradientBoosting Regressor](gradient_boosting)d) [Stacking Regressor](stacking_regressor)  References: 1) Stacked Regressions : Top 4% on LeaderBoard by Serigne.- Good if you are looking for stacking models and to gather an in-depth analysis for feature engineering. 2)  Simple House Price Prediction Stacking  by <NAME>.  - Gave me an idea of which algorithms to implement in my ensemble methods. - Also Victor is really open to answer any doubts with regards to this project. 3)  Comprehensive data exploration with Python  by <NAME>.- Help me understand more in depth the different linear regularization methods and its parameters. 4)  Hands on Machine Learning with Scikit-Learn & TensorFlow by <NAME> (O'Reilly). CopyRight 2017 <NAME>   - Good reference for understanding how Pipelines work. - Good for understanding ensemble methods such as RandomForests and GradientBoosting. - This book is a must have for people starting in the area of machine learning.5)  A comprehensive beginners guide for Linear, Ridge and Lasso Regression  by Sh<NAME> at Analytics Vidhya.- Helped me implement a residual plot.. - Better understanding of Ridge, Lasso and ElasticNet (Good for Beginners).<jupyter_code># Data and plotting imports
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib

import matplotlib.pyplot as plt

# Statistical Libraries
from scipy.stats import skew
from scipy.stats.stats import pearsonr
from scipy import stats


# Plotly imports
from plotly import tools
import plotly.plotly as py
import plotly.figure_factory as ff
import plotly.graph_objs as go
from plotly.offline import download_plotlyjs, init_notebook_mode, plot, iplot
init_notebook_mode(connected=True)


train = pd.read_csv('../input/train.csv')
test = pd.read_csv('../input/test.csv')

# Maintain the Ids for submission
train_id = train['Id']
test_id = test['Id']
train['SalePrice'].describe()
# It seems we have nulls so we will use the imputer strategy later on.
Missing = pd.concat([train.isnull().sum(), test.isnull().sum()], axis=1, keys=['train', 'test'])
Missing[Missing.sum(axis=1) > 0]
# We have several columns that contains null values we should replace them with the median or mean those null values.
train.info()
train.describe()
corr = train.corr()
plt.figure(figsize=(14,8))
plt.title('Overall Correlation of House Prices', fontsize=18)
sns.heatmap(corr,annot=False,cmap='RdYlBu',linewidths=0.2,annot_kws={'size':20})
plt.show()<jupyter_output><empty_output><jupyter_text>Splitting the Variables into Different Categories: Data Analysis:For data analysis purposes I am going to separate the different features into different categories in order to segment our analysis. These are the steps we are going to take in our analysis: Nevertheless, I will split the categories so you can analyse thoroughly the different categories.1) Separate into different categories in order to make our analysis easier. 2) All of our categories will contain sales price in order to see if there is a significant pattern.3) After that we will create our linear regression model in order to make accurate predictions as to what will the price of the houses will be.4) For all the categories we have id, salesprice, MoSold, YearSold, SaleType and SaleCondition.**Note:** At least for me, it is extremely important to make a data analysis of our data, in order to have a grasp of what the data is telling us, what might move salesprice higher or lower. Instead of just running a model and just predict prices, we must make a thorough analysis of our data. Also, using these different categories is completely optional in case you want to make a more in-depth analysis of the different features.<jupyter_code># Create the categories
outsidesurr_df = train[['Id', 'MSZoning', 'LotFrontage', 'LotArea', 'Neighborhood', 'Condition1', 'Condition2', 'PavedDrive', 
                    'Street', 'Alley', 'LandContour', 'LandSlope', 'LotConfig', 'MoSold', 'YrSold', 'SaleType', 'LotShape', 
                     'SaleCondition', 'SalePrice']]

building_df = train[['Id', 'MSSubClass', 'BldgType', 'HouseStyle', 'YearBuilt', 'YearRemodAdd', 'RoofStyle', 
                    'RoofMatl', 'Exterior1st', 'Exterior2nd', 'MasVnrType', 'MasVnrArea', 'Foundation', 'Functional', 
                    'GarageType', 'GarageYrBlt', 'GarageFinish', 'GarageCars', 'GarageArea', 'MoSold', 'YrSold', 'SaleType',
                    'SaleCondition', 'SalePrice']]

utilities_df = train[['Id', 'Utilities', 'Heating', 'CentralAir', 'Electrical', 'Fireplaces', 'PoolArea', 'MiscVal', 'MoSold',
                     'YrSold', 'SaleType', 'SaleCondition', 'SalePrice']]

ratings_df = train[['Id', 'OverallQual', 'OverallCond', 'ExterQual', 'ExterCond', 'BsmtQual', 'BsmtCond', 'BsmtExposure', 
                   'BsmtFinType1', 'BsmtFinType2', 'HeatingQC', 'KitchenQual', 'FireplaceQu', 'PoolQC', 'Fence', 'MiscFeature',
                   'GarageCond', 'GarageQual', 'MoSold', 'YrSold', 'SaleType', 'SaleCondition', 'SalePrice']]

rooms_df = train[['Id', 'GrLivArea', 'BsmtFullBath', 'BsmtHalfBath', 'FullBath', 'HalfBath', 'BsmtFinSF1', 'BsmtFinSF2',
                 'BsmtUnfSF', 'TotalBsmtSF', '1stFlrSF', '2ndFlrSF', 'LowQualFinSF','TotRmsAbvGrd', 
                 'WoodDeckSF', 'OpenPorchSF', 'EnclosedPorch', '3SsnPorch', 'ScreenPorch', 'MoSold', 'YrSold', 'SaleType',
                 'SaleCondition', 'SalePrice']]




# Set Id as index of the dataframe.
outsidesurr_df = outsidesurr_df.set_index('Id')
building_df = building_df.set_index('Id')
utilities_df = utilities_df.set_index('Id')
ratings_df = ratings_df.set_index('Id')
rooms_df = rooms_df.set_index('Id')

# Move SalePrice to the first column (Our Label)
sp0 = outsidesurr_df['SalePrice']
outsidesurr_df.drop(labels=['SalePrice'], axis=1, inplace=True)
outsidesurr_df.insert(0, 'SalePrice', sp0)

sp1 = building_df['SalePrice']
building_df.drop(labels=['SalePrice'], axis=1, inplace=True)
building_df.insert(0, 'SalePrice', sp1)

sp2 = utilities_df['SalePrice']
utilities_df.drop(labels=['SalePrice'], axis=1, inplace=True)
utilities_df.insert(0, 'SalePrice', sp2)

sp3 = ratings_df['SalePrice']
ratings_df.drop(labels=['SalePrice'], axis=1, inplace=True)
ratings_df.insert(0, 'SalePrice', sp3)

sp4 = rooms_df['SalePrice']
rooms_df.drop(labels=['SalePrice'], axis=1, inplace=True)
rooms_df.insert(0, 'SalePrice', sp4)<jupyter_output><empty_output><jupyter_text>Gathering a Basic Insight of our Data: Summary: The distribution of  house prices  is right skewed. There is a drop in the number of houses sold during the year of 2010.<jupyter_code>import seaborn as sns
sns.set_style('white')

f, axes = plt.subplots(ncols=4, figsize=(16,4))

# Lot Area: In Square Feet
sns.distplot(train['LotArea'], kde=False, color="#DF3A01", ax=axes[0]).set_title("Distribution of LotArea")
axes[0].set_ylabel("Square Ft")
axes[0].set_xlabel("Amount of Houses")

# MoSold: Year of the Month sold
sns.distplot(train['MoSold'], kde=False, color="#045FB4", ax=axes[1]).set_title("Monthly Sales Distribution")
axes[1].set_ylabel("Amount of Houses Sold")
axes[1].set_xlabel("Month of the Year")

# House Value
sns.distplot(train['SalePrice'], kde=False, color="#088A4B", ax=axes[2]).set_title("Monthly Sales Distribution")
axes[2].set_ylabel("Number of Houses ")
axes[2].set_xlabel("Price of the House")

# YrSold: Year the house was sold.
sns.distplot(train['YrSold'], kde=False, color="#FE2E64", ax=axes[3]).set_title("Year Sold")
axes[3].set_ylabel("Number of Houses ")
axes[3].set_xlabel("Year Sold")

plt.show()<jupyter_output><empty_output><jupyter_text>Right-Skewed Distribution Summary:In a right skew or positive skew the mean is most of the times to the right of the median. There is a higher frequency of occurence to the left of the distribution plot leading to more exceptions (outliers to the right). Nevertheless, there is a way to transform this histogram into a normal distributions by using log transformations which will be discussed further below.<jupyter_code># Maybe we can try this with plotly.
plt.figure(figsize=(12,8))
sns.distplot(train['SalePrice'], color='r')
plt.title('Distribution of Sales Price', fontsize=18)

plt.show()<jupyter_output><empty_output><jupyter_text>Economic Activity: We will visualize how the housing market in **Ames, IOWA** performed during the years 2006 - 2010 and how bad it was hit by the economic recession during the years of 2007-2008.   Level of Supply and Demand (Summary):June and July were the montnths in which most houses were sold.  The  median house price  was at its peak in 2007 (167k) and it was at its lowest point during the year of 2010 (155k) a difference of 12k. This might be a consequence of the economic recession.  Less houses were sold and built during the year of 2010 compared to the other years.<jupyter_code># People tend to move during the summer
sns.set(style="whitegrid")
plt.figure(figsize=(12,8))
sns.countplot(y="MoSold", hue="YrSold", data=train)
plt.show()
plt.figure(figsize=(12,8))
sns.boxplot(x='YrSold', y='SalePrice', data=train)
plt.xlabel('Year Sold', fontsize=14)
plt.ylabel('Price sold', fontsize=14)
plt.title('Houses Sold per Year', fontsize=16)
plt.figure(figsize=(14,8))
plt.style.use('seaborn-white')
sns.stripplot(x='YrSold', y='YearBuilt', data=train, jitter=True, palette="Set2", linewidth=1)
plt.title('Economic Activity Analysis', fontsize=18)
plt.xlabel('Year the house was sold', fontsize=14)
plt.ylabel('Year the house was built', rotation=90, fontsize=14)
plt.show()<jupyter_output><empty_output><jupyter_text>Outside Surroundings of the House:  Features from Outside: In this section we will create an in-depth analysis of how the outside surroundings affect the price. Which variables have the highest weight on price. You can use the **train** dataframe or the **outsidesurr_df** to simplify the amount of features and have a closer look as to how they behave towards **"SalePrice"**. For the correlation matrix I will be using outsidesurr_df so you can have a better look as to which variables from the **outside surroundings category** impact the most on the price of a house.  Summary: The mean price of the house of is 180,921, this will explain why the data is right skewed.  Standard deviation is pretty high at 79442.50 meaning the data deviates a lot from the mean (many outliers)  LotArea and LotFrontage had the highest correlation with the price of a house from the  outside surroundings category .  Most of the houses that were sold were from a  Residential Low Density Zone . The most exclusive Neighborhoods are Crawfor, Sawyer and SawyerW<jupyter_code>outsidesurr_df.describe()
outsidesurr_df.columns
# Lot Area and Lot Frontage influenced hugely on the price. 
# However, YrSold does not have that much of a negative correlation with SalePrice as we previously thought.
# Meaning the state of IOWA was not affected as other states.
plt.style.use('seaborn-white')
corr = outsidesurr_df.corr()

sns.heatmap(corr,annot=True,cmap='YlOrRd',linewidths=0.2,annot_kws={'size':20})
fig=plt.gcf()
fig.set_size_inches(14,10)
plt.title("Outside Surroundings Correlation", fontsize=18)
plt.xticks(fontsize=14)
plt.yticks(fontsize=14)
plt.show()<jupyter_output><empty_output><jupyter_text>Type of Zoning:<jupyter_code># We already know which neighborhoods were the most sold but which neighborhoods gave the most revenue. 
# This might indicate higher demand toward certain neighborhoods.
plt.style.use('seaborn-white')
zoning_value = train.groupby(by=['MSZoning'], as_index=False)['SalePrice'].sum()
zoning = zoning_value['MSZoning'].values.tolist()


# Let's create a pie chart.
labels = ['C: Commercial', 'FV: Floating Village Res.', 'RH: Res. High Density', 'RL: Res. Low Density', 
          'RM: Res. Medium Density']
total_sales = zoning_value['SalePrice'].values.tolist()
explode = (0, 0, 0, 0.1, 0)

fig, ax1 = plt.subplots(figsize=(12,8))
texts = ax1.pie(total_sales, explode=explode, autopct='%.1f%%', shadow=True, startangle=90, pctdistance=0.8,
       radius=0.5)


ax1.axis('equal')
plt.title('Sales Groupby Zones', fontsize=16)
plt.tight_layout()
plt.legend(labels, loc='best')
plt.show()
plt.style.use('seaborn-white')
SalesbyZone = train.groupby(['YrSold','MSZoning']).SalePrice.count()
SalesbyZone.unstack().plot(kind='bar',stacked=True, colormap= 'gnuplot',  
                           grid=False,  figsize=(12,8))
plt.title('Building Sales (2006 - 2010) by Zoning', fontsize=18)
plt.ylabel('Sale Price', fontsize=14)
plt.xlabel('Sales per Year', fontsize=14)
plt.show()<jupyter_output><empty_output><jupyter_text>Neighborhoods:<jupyter_code>fig, ax = plt.subplots(figsize=(12,8))
sns.countplot(x="Neighborhood", data=train, palette="Set2")
ax.set_title("Types of Neighborhoods", fontsize=20)
ax.set_xlabel("Neighborhoods", fontsize=16)
ax.set_ylabel("Number of Houses Sold", fontsize=16)
ax.set_xticklabels(labels=train['Neighborhood'] ,rotation=90)
plt.show()
# Sawyer and SawyerW tend to be the most expensive neighberhoods. Nevertheless, what makes them the most expensive
# Is it the LotArea or LotFrontage? Let's find out!
fig, ax = plt.subplots(figsize=(12,8))
ax = sns.boxplot(x="Neighborhood", y="SalePrice", data=train)
ax.set_title("Range Value of the Neighborhoods", fontsize=18)
ax.set_ylabel('Price Sold', fontsize=16)
ax.set_xlabel('Neighborhood', fontsize=16)
ax.set_xticklabels(labels=train['Neighborhood'] , rotation=90)
plt.show()<jupyter_output><empty_output><jupyter_text>The Impact of Space towards Price: The Influence of Space:How much influence does space have towards the price of the house. Intuitively, we might think the bigger the house the higher the price but let's take a look in order to see ifit actually has a positive correlation towards **SalePrice**. Summary:GrlivingArea: The living area square feet is positively correlated with the price of the house.  GarageArea: Apparently, the space of the garage is an important factor that contributes to the price of the house.   TotalBsmft: The square feet of the basement contributes positively to the value of the house.  LotArea and LotFrontage: I would say from all the area features these are the two that influencess the less on the price of the house.<jupyter_code>sns.jointplot(x='GrLivArea',y='SalePrice',data=train,
              kind='hex', cmap= 'CMRmap', size=8, color='#F84403')

plt.show()
sns.jointplot(x='GarageArea',y='SalePrice',data=train,
              kind='hex', cmap= 'CMRmap', size=8, color='#F84403')

plt.show()
sns.jointplot(x='TotalBsmtSF',y='SalePrice',data=train,
              kind='hex', cmap= 'CMRmap', size=8, color='#F84403')

plt.show()
plt.figure(figsize=(16,6))
plt.subplot(121)
ax = sns.regplot(x="LotFrontage", y="SalePrice", data=train)
ax.set_title("Lot Frontage vs Sale Price", fontsize=16)

plt.subplot(122)
ax1 = sns.regplot(x="LotArea", y="SalePrice", data=train, color='#FE642E')
ax1.set_title("Lot Area vs Sale Price", fontsize=16)

plt.show()<jupyter_output><empty_output><jupyter_text>Building Characteristics:<jupyter_code>building_df.head()<jupyter_output><empty_output><jupyter_text>High Correlated Variables with SalePrice:1) YearBuilt - The Date the building was built. 2) YearRemodAdd - The last time there wasa building remodeling. 3) MasVnArea - Masonry veneer area in square feet. 4) GarageYrBlt - Year garage was built. 5) GarageCars - Size of garage in car capacity. 6) GarageArea - Size of garage in square feet.<jupyter_code>corr = building_df.corr()

g = sns.heatmap(corr,annot=True,cmap='coolwarm',linewidths=0.2,annot_kws={'size':20})
g.set_xticklabels(g.get_xticklabels(), rotation = 90, fontsize = 8)
fig=plt.gcf()
fig.set_size_inches(14,10)
plt.title("Building Characteristics Correlation", fontsize=18)
plt.xticks(fontsize=14)
plt.yticks(fontsize=14)
plt.show()
# To understand better our data I will create a category column for SalePrice.
train['Price_Range'] = np.nan
lst = [train]

# Create a categorical variable for SalePrice
# I am doing this for further visualizations.
for column in lst:
    column.loc[column['SalePrice'] < 150000, 'Price_Range'] = 'Low'
    column.loc[(column['SalePrice'] >= 150000) & (column['SalePrice'] <= 300000), 'Price_Range'] = 'Medium'
    column.loc[column['SalePrice'] > 300000, 'Price_Range'] = 'High'
    
train.head()<jupyter_output><empty_output><jupyter_text>What Garages tells us about each Price Category:<jupyter_code>import matplotlib.pyplot as plt
palette = ["#9b59b6", "#BDBDBD", "#FF8000"]
sns.lmplot('GarageYrBlt', 'GarageArea', data=train, hue='Price_Range', fit_reg=False, size=7, palette=palette,
          markers=["o", "s", "^"])
plt.title('Garage by Price Range', fontsize=18)
plt.annotate('High Price \nCategory Garages \n are not that old', xy=(1997, 1100), xytext=(1950, 1200), 
            arrowprops=dict(facecolor='black', shrink=0.05))
plt.show()<jupyter_output><empty_output><jupyter_text>Miscellaneous and Utilities:<jupyter_code>plt.style.use('seaborn-white')
types_foundations = train.groupby(['Price_Range', 'PavedDrive']).size()
types_foundations.unstack().plot(kind='bar', stacked=True, colormap='Set1', figsize=(13,11), grid=False)
plt.ylabel('Number of Streets', fontsize=16)
plt.xlabel('Price Category', fontsize=16)
plt.xticks(rotation=45, fontsize=12)
plt.title('Condition of the Street by Price Category', fontsize=18)
plt.show()
# We can see that CentralAir impacts until some extent the price of the house.

fig, ((ax1, ax2), (ax3, ax4)) = plt.subplots(nrows=2, ncols=2, figsize=(14,10))
plt.suptitle('Relationship between Saleprice \n and Categorical Utilities', fontsize=18)
sns.pointplot(x='CentralAir', y='SalePrice', hue='Price_Range', data=train, ax=ax1)
sns.pointplot(x='Heating', y='SalePrice', hue='Price_Range', data=train, ax=ax2)
sns.pointplot(x='Fireplaces', y='SalePrice', hue='Price_Range', data=train, ax=ax3)
sns.pointplot(x='Electrical', y='SalePrice', hue='Price_Range', data=train, ax=ax4)

plt.legend(loc='best')
plt.show()
import seaborn as sns
import matplotlib.pyplot as plt
plt.style.use('seaborn-white')

fig, ax = plt.subplots(figsize=(14,8))
palette = ["#9b59b6", "#3498db", "#95a5a6", "#e74c3c", "#34495e", "#2ecc71", "#FF8000", "#AEB404", "#FE2EF7", "#64FE2E"]

sns.swarmplot(x="OverallQual", y="SalePrice", data=train, ax=ax, palette=palette, linewidth=1)
plt.title('Correlation between OverallQual and SalePrice', fontsize=18)
plt.ylabel('Sale Price', fontsize=14)
plt.show()<jupyter_output><empty_output><jupyter_text>What determines the quality of the House? Remember quality is the most important factor that contributes to the SalePrice of the house. **Correlations with OverallQual:**1) YearBuilt 2) TotalBsmtSF 3) GrLivArea 4) FullBath 5) GarageYrBuilt 6) GarageCars 7) GarageArea   Interesting insights:1) **Overall Condition**: of the house or building, meaning that further remodelations are likely to happen in the future, either for reselling or to accumulate value in their real-estate.. 2) **Overall Quality**: The quality of the house is one of the factors that mostly impacts SalePrice. It seems that the overall material that is used for construction and the finish of the house has a great impact on SalePrice. 3) **Year Remodelation**: Houses in the **high** price range remodelled their houses sooner. The sooner the remodelation the higher the value of the house.<jupyter_code>with sns.plotting_context("notebook",font_scale=2.8):
    g = sns.pairplot(train, vars=["OverallCond", "OverallQual", "YearRemodAdd", "SalePrice"],
                hue="Price_Range", palette="Dark2", size=6)


g.set(xticklabels=[]);

plt.show()<jupyter_output><empty_output><jupyter_text>Which Material Combination increased the Price of Houses? Roof Material: Hip and Gable was the most expensive since people who bought high value houses tended to buy this material bor he rooftop. House Material: Houses made up of stone tend to influence positively the price of the house. (Except in 2007 for High Price House Values. )<jupyter_code># What type of material is considered to have a positive effect on the quality of the house?
# Let's start with the roof material

with sns.plotting_context("notebook",font_scale=1):
    g = sns.factorplot(x="SalePrice", y="RoofStyle", hue="Price_Range",
                   col="YrSold", data=train, kind="box", size=5, aspect=.75, sharex=False, col_wrap=3, orient="h",
                      palette='Set1');
    for ax in g.axes.flatten(): 
        for tick in ax.get_xticklabels(): 
            tick.set(rotation=20)

plt.show()<jupyter_output><empty_output><jupyter_text>**Note:** Interestingly, the Masonry Veneer type of stone became popular after 2007 for the houses that belong to the **high** Price Range category. I wonder why? **For some reason during the year of 2007, the Saleprice of houses within the high range made of stone dropped drastically!<jupyter_code>with sns.plotting_context("notebook",font_scale=1):
    g = sns.factorplot(x="MasVnrType", y="SalePrice", hue="Price_Range",
                   col="YrSold", data=train, kind="bar", size=5, aspect=.75, sharex=False, col_wrap=3,
                      palette="YlOrRd");
    
plt.show()<jupyter_output><empty_output><jupyter_text>Quality of Neighborhoods  Which Neighborhoods had the best Quality houses?<jupyter_code>plt.style.use('seaborn-white')
types_foundations = train.groupby(['Neighborhood', 'OverallQual']).size()
types_foundations.unstack().plot(kind='bar', stacked=True, colormap='RdYlBu', figsize=(13,11), grid=False)
plt.ylabel('Overall Price of the House', fontsize=16)
plt.xlabel('Neighborhood', fontsize=16)
plt.xticks(rotation=90, fontsize=12)
plt.title('Overall Quality of the Neighborhoods', fontsize=18)
plt.show()
# Which houses neighborhoods remodeled the most.
# price_categories = ['Low', 'Medium', 'High']
# remod = train['YearRemodAdd'].groupby(train['Price_Range']).mean()

fig, ax = plt.subplots(ncols=2, figsize=(16,4))
plt.subplot(121)
sns.pointplot(x="Price_Range",  y="YearRemodAdd", data=train, order=["Low", "Medium", "High"], color="#0099ff")
plt.title("Average Remodeling by Price Category", fontsize=16)
plt.xlabel('Price Category', fontsize=14)
plt.ylabel('Average Remodeling Year', fontsize=14)
plt.xticks(rotation=90, fontsize=12)

plt.subplot(122)
sns.pointplot(x="Neighborhood",  y="YearRemodAdd", data=train, color="#ff9933")
plt.title("Average Remodeling by Neighborhood", fontsize=16)
plt.xlabel('Neighborhood', fontsize=14)
plt.ylabel('')
plt.xticks(rotation=90, fontsize=12)
plt.show()<jupyter_output><empty_output><jupyter_text>The Purpose of Log Transformations:The main reason why we use log transformation is to reduce **skewness** in our data. However, there are other reasons why we log transform  our data:  Easier to interpret patterns of our data.  For possible statistical analysis that require the data to be normalized.<jupyter_code>numeric_features = train.dtypes[train.dtypes != "object"].index

# Top 5 most skewed features
skewed_features = train[numeric_features].apply(lambda x: skew(x.dropna())).sort_values(ascending=False)
skewness = pd.DataFrame({'Skew' :skewed_features})
skewness.head(5)
from scipy.stats import norm

# norm = a normal continous variable.

log_style = np.log(train['SalePrice'])  # log of salesprice

fig, ((ax1, ax2), (ax3, ax4)) = plt.subplots(nrows=2, ncols=2, figsize=(14,10))
plt.suptitle('Probability Plots', fontsize=18)
ax1 = sns.distplot(train['SalePrice'], color="#FA5858", ax=ax1, fit=norm)
ax1.set_title("Distribution of Sales Price with Positive Skewness", fontsize=14)
ax2 = sns.distplot(log_style, color="#58FA82",ax=ax2, fit=norm)
ax2.set_title("Normal Distibution with Log Transformations", fontsize=14)
ax3 = stats.probplot(train['SalePrice'], plot=ax3)
ax4 = stats.probplot(log_style, plot=ax4)

plt.show()<jupyter_output><empty_output><jupyter_text>Skewedness and Kurtosis:**Skewedness**:  A skewness of zero or near zero indicates a symmetric distribution. A negative value for the skewness indicate a left skewness (tail to the left)  A positive value for te skewness indicate a  right skewness  (tail to the right)  **Kurtosis**:Kourtosis is a measure of how extreme observations are in a dataset. The  greater the kurtosis coefficient , the more peaked the distribution around the mean is. Greater coefficient also means fatter tails, which means there is an increase in tail risk (extreme results) **Reference**:Investopedia: https://www.investopedia.com/terms/m/mesokurtic.asp<jupyter_code>print('Skewness for Normal D.: %f'% train['SalePrice'].skew())
print('Skewness for Log D.: %f'% log_style.skew())
print('Kurtosis for Normal D.: %f' % train['SalePrice'].kurt())
print('Kurtosis for Log D.: %f' % log_style.kurt())<jupyter_output><empty_output><jupyter_text>Outliers Analysis:**Analysis**: The year of 2007 had the highest outliers (peak of the housing market before collapse).   The highest outliers are located in the  High category  of the Price_Range column.<jupyter_code># Most outliers are in the high price category nevertheless, in the year of 2007 saleprice of two houses look extremely high!

fig = plt.figure(figsize=(12,8))
ax = sns.boxplot(x="YrSold", y="SalePrice", hue='Price_Range', data=train)
plt.title('Detecting outliers', fontsize=16)
plt.xlabel('Year the House was Sold', fontsize=14)
plt.ylabel('Price of the house', fontsize=14)
plt.show()
corr = train.corr()
corr['SalePrice'].sort_values(ascending=False)[:11]<jupyter_output><empty_output><jupyter_text>Bivariate Analysis (Detecting outliers through visualizations):**There are some outliers in some of this columns but there might be a reason behind this, it is possible that these outliers in which the area is high but the price of the house is not that high, might be due to the reason that these houses are located in agricultural zones.**<jupyter_code>fig, ((ax1, ax2), (ax3, ax4))= plt.subplots(nrows=2, ncols=2, figsize=(14,8))
var1 = 'GrLivArea'
data = pd.concat([train['SalePrice'], train[var1]], axis=1)
sns.regplot(x=var1, y='SalePrice', data=data, fit_reg=True, ax=ax1)


var2 = 'GarageArea'
data = pd.concat([train['SalePrice'], train[var2]], axis=1)
sns.regplot(x=var2, y='SalePrice', data=data, fit_reg=True, ax=ax2, marker='s')

var3 = 'TotalBsmtSF'
data = pd.concat([train['SalePrice'], train[var3]], axis=1)
sns.regplot(x=var3, y='SalePrice', data=data, fit_reg=True, ax=ax3, marker='^')

var4 = '1stFlrSF'
data = pd.concat([train['SalePrice'], train[var4]], axis=1)
sns.regplot(x=var4, y='SalePrice', data=data, fit_reg=True, ax=ax4, marker='+')

plt.show()<jupyter_output><empty_output><jupyter_text>Feature Engineering  Dealing with Missing Values:<jupyter_code>y_train = train['SalePrice'].values
# We will concatenate but we will split further on.
rtrain = train.shape[0]
ntest = test.shape[0]
train.drop(['SalePrice', 'Price_Range', 'Id'], axis=1, inplace=True)
test.drop('Id', axis=1, inplace=True)
complete_data = pd.concat([train, test])
complete_data.shape
total_nas = complete_data.isnull().sum().sort_values(ascending=False)
percent_missing = (complete_data.isnull().sum()/complete_data.isnull().count()).sort_values(ascending=False)
missing = pd.concat([total_nas, percent_missing], axis=1, keys=['Total_M', 'Percentage'])


# missing.head(9) # We have 19 columns with NAs<jupyter_output><empty_output><jupyter_text>Transforming Missing Values:<jupyter_code>complete_data["PoolQC"] = complete_data["PoolQC"].fillna("None")
complete_data["MiscFeature"] = complete_data["MiscFeature"].fillna("None")
complete_data["Alley"] = complete_data["Alley"].fillna("None")
complete_data["Fence"] = complete_data["Fence"].fillna("None")
complete_data["FireplaceQu"] = complete_data["FireplaceQu"].fillna("None")
for col in ('GarageType', 'GarageFinish', 'GarageQual', 'GarageCond'):
    complete_data[col] = complete_data[col].fillna('None')
for col in ('BsmtQual', 'BsmtCond', 'BsmtExposure', 'BsmtFinType1', 'BsmtFinType2'):
    complete_data[col] = complete_data[col].fillna('None')
complete_data['MSZoning'] = complete_data['MSZoning'].fillna(complete_data['MSZoning'].mode()[0])
complete_data["MasVnrType"] = complete_data["MasVnrType"].fillna("None")
complete_data["Functional"] = complete_data["Functional"].fillna("Typ")
complete_data['Electrical'] = complete_data['Electrical'].fillna(complete_data['Electrical'].mode()[0])
complete_data['KitchenQual'] = complete_data['KitchenQual'].fillna(complete_data['KitchenQual'].mode()[0])
complete_data['Exterior1st'] = complete_data['Exterior1st'].fillna(complete_data['Exterior1st'].mode()[0])
complete_data['Exterior2nd'] = complete_data['Exterior2nd'].fillna(complete_data['Exterior2nd'].mode()[0])
complete_data['SaleType'] = complete_data['SaleType'].fillna(complete_data['SaleType'].mode()[0])
complete_data['MSSubClass'] = complete_data['MSSubClass'].fillna("None")
# Group by neighborhood and fill in missing value by the median LotFrontage of all the neighborhood
complete_data["LotFrontage"] = complete_data.groupby("Neighborhood")["LotFrontage"].transform(
    lambda x: x.fillna(x.median()))

for col in ('GarageYrBlt', 'GarageArea', 'GarageCars'):
    complete_data[col] = complete_data[col].fillna(0)
    
for col in ('BsmtFinSF1', 'BsmtFinSF2', 'BsmtUnfSF','TotalBsmtSF', 'BsmtFullBath', 'BsmtHalfBath'):
    complete_data[col] = complete_data[col].fillna(0)
    
complete_data["MasVnrArea"] = complete_data["MasVnrArea"].fillna(0)
# Drop
complete_data = complete_data.drop(['Utilities'], axis=1)<jupyter_output><empty_output><jupyter_text>Combining Attributes<jupyter_code># Adding total sqfootage feature 
complete_data['TotalSF'] = complete_data['TotalBsmtSF'] + complete_data['1stFlrSF'] + complete_data['2ndFlrSF']<jupyter_output><empty_output><jupyter_text>Dealing with Numerical and Categorical Values:  Transforming our Data: Separate the  features  and  labels  from the training dataset.  Separate  numeric  and  categorical  variables for the purpose of running them in separate pipelines and scaling them with their respective scalers.<jupyter_code>complete_data.head()
# splitting categorical variables with numerical variables for encoding.
categorical = complete_data.select_dtypes(['object'])
numerical = complete_data.select_dtypes(exclude=['object'])

print(categorical.shape)
print(numerical.shape)<jupyter_output><empty_output><jupyter_text>Categorical Encoding Class:This is a way to encode our features in a way that it avoids the assumption that two nearby values are more similar than two distant values. This is the reason we should avoid using LabelEncoder to scale features (inputs) in our dataset and in addition the word **LabelEncoder** is used for scaling labels (outputs). This could be used more often in **binary classification problems** were no *association* exists between the outputs.<jupyter_code>from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.utils import check_array
from sklearn.preprocessing import LabelEncoder
from scipy import sparse

class CategoricalEncoder(BaseEstimator, TransformerMixin):
    """Encode categorical features as a numeric array.
    The input to this transformer should be a matrix of integers or strings,
    denoting the values taken on by categorical (discrete) features.
    The features can be encoded using a one-hot aka one-of-K scheme
    (``encoding='onehot'``, the default) or converted to ordinal integers
    (``encoding='ordinal'``).
    This encoding is needed for feeding categorical data to many scikit-learn
    estimators, notably linear models and SVMs with the standard kernels.
    Read more in the :ref:`User Guide <preprocessing_categorical_features>`.
    Parameters
    ----------
    encoding : str, 'onehot', 'onehot-dense' or 'ordinal'
        The type of encoding to use (default is 'onehot'):
        - 'onehot': encode the features using a one-hot aka one-of-K scheme
          (or also called 'dummy' encoding). This creates a binary column for
          each category and returns a sparse matrix.
        - 'onehot-dense': the same as 'onehot' but returns a dense array
          instead of a sparse matrix.
        - 'ordinal': encode the features as ordinal integers. This results in
          a single column of integers (0 to n_categories - 1) per feature.
    categories : 'auto' or a list of lists/arrays of values.
        Categories (unique values) per feature:
        - 'auto' : Determine categories automatically from the training data.
        - list : ``categories[i]`` holds the categories expected in the ith
          column. The passed categories are sorted before encoding the data
          (used categories can be found in the ``categories_`` attribute).
    dtype : number type, default np.float64
        Desired dtype of output.
    handle_unknown : 'error' (default) or 'ignore'
        Whether to raise an error or ignore if a unknown categorical feature is
        present during transform (default is to raise). When this is parameter
        is set to 'ignore' and an unknown category is encountered during
        transform, the resulting one-hot encoded columns for this feature
        will be all zeros.
        Ignoring unknown categories is not supported for
        ``encoding='ordinal'``.
    Attributes
    ----------
    categories_ : list of arrays
        The categories of each feature determined during fitting. When
        categories were specified manually, this holds the sorted categories
        (in order corresponding with output of `transform`).
    Examples
    --------
    Given a dataset with three features and two samples, we let the encoder
    find the maximum value per feature and transform the data to a binary
    one-hot encoding.
    >>> from sklearn.preprocessing import CategoricalEncoder
    >>> enc = CategoricalEncoder(handle_unknown='ignore')
    >>> enc.fit([[0, 0, 3], [1, 1, 0], [0, 2, 1], [1, 0, 2]])
    ... # doctest: +ELLIPSIS
    CategoricalEncoder(categories='auto', dtype=<... 'numpy.float64'>,
              encoding='onehot', handle_unknown='ignore')
    >>> enc.transform([[0, 1, 1], [1, 0, 4]]).toarray()
    array([[ 1.,  0.,  0.,  1.,  0.,  0.,  1.,  0.,  0.],
           [ 0.,  1.,  1.,  0.,  0.,  0.,  0.,  0.,  0.]])
    See also
    --------
    sklearn.preprocessing.OneHotEncoder : performs a one-hot encoding of
      integer ordinal features. The ``OneHotEncoder assumes`` that input
      features take on values in the range ``[0, max(feature)]`` instead of
      using the unique values.
    sklearn.feature_extraction.DictVectorizer : performs a one-hot encoding of
      dictionary items (also handles string-valued features).
    sklearn.feature_extraction.FeatureHasher : performs an approximate one-hot
      encoding of dictionary items or strings.
    """

    def __init__(self, encoding='onehot', categories='auto', dtype=np.float64,
                 handle_unknown='error'):
        self.encoding = encoding
        self.categories = categories
        self.dtype = dtype
        self.handle_unknown = handle_unknown

    def fit(self, X, y=None):
        """Fit the CategoricalEncoder to X.
        Parameters
        ----------
        X : array-like, shape [n_samples, n_feature]
            The data to determine the categories of each feature.
        Returns
        -------
        self
        """

        if self.encoding not in ['onehot', 'onehot-dense', 'ordinal']:
            template = ("encoding should be either 'onehot', 'onehot-dense' "
                        "or 'ordinal', got %s")
            raise ValueError(template % self.handle_unknown)

        if self.handle_unknown not in ['error', 'ignore']:
            template = ("handle_unknown should be either 'error' or "
                        "'ignore', got %s")
            raise ValueError(template % self.handle_unknown)

        if self.encoding == 'ordinal' and self.handle_unknown == 'ignore':
            raise ValueError("handle_unknown='ignore' is not supported for"
                             " encoding='ordinal'")

        X = check_array(X, dtype=np.object, accept_sparse='csc', copy=True)
        n_samples, n_features = X.shape

        self._label_encoders_ = [LabelEncoder() for _ in range(n_features)]

        for i in range(n_features):
            le = self._label_encoders_[i]
            Xi = X[:, i]
            if self.categories == 'auto':
                le.fit(Xi)
            else:
                valid_mask = np.in1d(Xi, self.categories[i])
                if not np.all(valid_mask):
                    if self.handle_unknown == 'error':
                        diff = np.unique(Xi[~valid_mask])
                        msg = ("Found unknown categories {0} in column {1}"
                               " during fit".format(diff, i))
                        raise ValueError(msg)
                le.classes_ = np.array(np.sort(self.categories[i]))

        self.categories_ = [le.classes_ for le in self._label_encoders_]

        return self

    def transform(self, X):
        """Transform X using one-hot encoding.
        Parameters
        ----------
        X : array-like, shape [n_samples, n_features]
            The data to encode.
        Returns
        -------
        X_out : sparse matrix or a 2-d array
            Transformed input.
        """
        X = check_array(X, accept_sparse='csc', dtype=np.object, copy=True)
        n_samples, n_features = X.shape
        X_int = np.zeros_like(X, dtype=np.int)
        X_mask = np.ones_like(X, dtype=np.bool)

        for i in range(n_features):
            valid_mask = np.in1d(X[:, i], self.categories_[i])

            if not np.all(valid_mask):
                if self.handle_unknown == 'error':
                    diff = np.unique(X[~valid_mask, i])
                    msg = ("Found unknown categories {0} in column {1}"
                           " during transform".format(diff, i))
                    raise ValueError(msg)
                else:
                    # Set the problematic rows to an acceptable value and
                    # continue `The rows are marked `X_mask` and will be
                    # removed later.
                    X_mask[:, i] = valid_mask
                    X[:, i][~valid_mask] = self.categories_[i][0]
            X_int[:, i] = self._label_encoders_[i].transform(X[:, i])

        if self.encoding == 'ordinal':
            return X_int.astype(self.dtype, copy=False)

        mask = X_mask.ravel()
        n_values = [cats.shape[0] for cats in self.categories_]
        n_values = np.array([0] + n_values)
        indices = np.cumsum(n_values)

        column_indices = (X_int + indices[:-1]).ravel()[mask]
        row_indices = np.repeat(np.arange(n_samples, dtype=np.int32),
                                n_features)[mask]
        data = np.ones(n_samples * n_features)[mask]

        out = sparse.csc_matrix((data, (row_indices, column_indices)),
                                shape=(n_samples, indices[-1]),
                                dtype=self.dtype).tocsr()
        if self.encoding == 'onehot-dense':
            return out.toarray()
        else:
            return out<jupyter_output><empty_output><jupyter_text>Combine Attribute Class:This class will help us to include the total area variable into our pipeline for further scaling.<jupyter_code>from sklearn.base import BaseEstimator, TransformerMixin

# class combination attribute.
# First we need to know the index possition of the other cloumns that make the attribute.
numerical.columns.get_loc("TotalBsmtSF") # Index Number 37
numerical.columns.get_loc("1stFlrSF") # Index NUmber 42
numerical.columns.get_loc("2ndFlrSF") # Index NUmber 43

ix_total, ix_first, ix_second = 9, 10, 11
# complete_data['TotalSF'] = complete_data['TotalBsmtSF'] + complete_data['1stFlrSF'] + complete_data['2ndFlrSF']

class CombineAttributes(BaseEstimator, TransformerMixin):
    
    def __init__(self, total_area=True): # No args or kargs
        self.total_area = total_area
        
    def fit(self, X, y=None):
        return self 
    
    def transform(self, X, y=None):
        total_sf = X[:,ix_total] + X[:,ix_first] + X[:,ix_second]
        if self.total_area:
            return np.c_[X, total_sf]
        else: 
            return np.c_[X]

attr_adder = CombineAttributes(total_area=True)
extra_attribs = attr_adder.transform(complete_data.values)
# Scikit-Learn does not handle dataframes in pipeline so we will create our own class.
# Reference: Hands-On Machine Learning
from sklearn.base import BaseEstimator, TransformerMixin
# Create a class to select numerical or cateogrical columns.
class DataFrameSelector(BaseEstimator, TransformerMixin):
    def __init__(self, attribute_names):
        self.attribute_names = attribute_names
    def fit (self, X, y=None):
        return self
    def transform(self, X):
        return X[self.attribute_names].values<jupyter_output><empty_output><jupyter_text>Pipelines: Create our numerical and cateogircal pipelines to scale our features.<jupyter_code>from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

lst_numerical = list(numerical)

numeric_pipeline = Pipeline([
    ('selector', DataFrameSelector(lst_numerical)),
    ('extra attributes', CombineAttributes()),
    ('std_scaler', StandardScaler()),
])

categorical_pipeline = Pipeline([
    ('selector', DataFrameSelector(['MSZoning', 'Street', 'LotShape', 'LandContour', 'LotConfig', 'LandSlope', 
                                    'Neighborhood', 'Condition1', 'Condition2','BldgType', 'HouseStyle', 'RoofStyle',
                                    'RoofMatl', 'Exterior1st',  'Exterior2nd','ExterQual','ExterCond', 'Foundation',
                                    'Heating','HeatingQC', 'CentralAir', 'Electrical', 'KitchenQual', 'Functional',
                                    'PavedDrive', 'SaleType', 'SaleCondition'])),
    ('encoder', CategoricalEncoder(encoding="onehot-dense")),
])
# Combine our pipelines!
from sklearn.pipeline import FeatureUnion

main_pipeline = FeatureUnion(transformer_list=[
    ('num_pipeline', numeric_pipeline),
    ('cat_pipeline', categorical_pipeline)
])

data_prepared = main_pipeline.fit_transform(complete_data)
data_prepared
features = data_prepared
labels = np.log1p(y_train) # Scaling the Saleprice column.

train_scaled = features[:rtrain] 
test_scaled = features[rtrain:]<jupyter_output><empty_output><jupyter_text>Implementing Predictive Models  Residual Plot:Residual plots will give us more or less the actual prediction errors our models are making. In this example, I will use yellowbrick library (statistical visualizations for machine learning) and a simple linear regression model.  In our legends of the residual plot it says training and test data but in this scenario instead of the test set it is the validation set we are using. [If there is a possibility to change the name of the legend to validation I will make the update whenever possible. Create a validation set within the training set to actually predict values. (Remember the test set does not have the training price, and also when testing data it should be done during the last instance of the project.)<jupyter_code>from sklearn.linear_model import ElasticNet, Lasso,  BayesianRidge, LassoLarsIC
from sklearn.ensemble import RandomForestRegressor,  GradientBoostingRegressor
from sklearn.kernel_ridge import KernelRidge
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import RobustScaler
from sklearn.base import BaseEstimator, TransformerMixin, RegressorMixin, clone
from sklearn.model_selection import KFold, cross_val_score, train_test_split
from sklearn.metrics import mean_squared_error
import xgboost as xgb
from sklearn.linear_model import Ridge
from yellowbrick.regressor import PredictionError, ResidualsPlot
# This is data that comes from the training test.
X_train, X_val, y_train, y_val = train_test_split(train_scaled, labels, test_size=0.25, random_state=42)

# Our validation set tends to perform better. Less Residuals.
ridge = Ridge()
visualizer = ResidualsPlot(ridge, train_color='#045FB4', test_color='r', line_color='#424242')
visualizer.fit(X_train, y_train)  # Fit the training data to the visualizer
visualizer.score(X_val, y_val)
g = visualizer.poof(outpath="residual_plot")
#Validation function
n_folds = 5

def rmsle_cv(model, features, labels):
    kf = KFold(n_folds, shuffle=True, random_state=42).get_n_splits(features) # Shuffle the data.
    rmse= np.sqrt(-cross_val_score(model, features, labels, scoring="neg_mean_squared_error", cv = kf))
    return(rmse.mean())
rid_reg = Ridge()
rid_reg.fit(X_train, y_train)
y_pred = rid_reg.predict(X_val)
rmsle_cv(rid_reg, X_val, y_val)<jupyter_output><empty_output><jupyter_text>RandomForestRegressor: **RandomForestRegressor** gives us more randomness, insead of searching through impurity the best feature, RandomForest picks features in a randomly manner to reduce variance at the expense of a higher bias. Nevertheless, this helps us find what the trend is. After all the trees have predicted the outcome for a specific instance, the average from all the DecisionTree models is taken and that will be the prediction for a specific instance.<jupyter_code>from sklearn.model_selection import GridSearchCV

params = {'n_estimators': list(range(50, 200, 25)), 'max_features': ['auto', 'sqrt', 'log2'], 
         'min_samples_leaf': list(range(50, 200, 50))}

grid_search_cv = GridSearchCV(RandomForestRegressor(random_state=42), params, n_jobs=-1)
grid_search_cv.fit(X_train, y_train)
grid_search_cv.best_estimator_
# Show best parameters.
grid_search_cv.best_params_
# You can check the results with this functionof grid search.
# RandomSearchCV takes just a sample not all possible combinations like GridSearchCV.
# Mean test score is equivalent to 0.2677
grid_search_cv.cv_results_
df_results = pd.DataFrame(grid_search_cv.cv_results_)
df_results.sort_values(by='mean_test_score', ascending=True).head(2)
rand_model = grid_search_cv.best_estimator_

rand_model.fit(X_train, y_train)
# Final root mean squared error.
y_pred = rand_model.predict(X_val)
rand_mse = mean_squared_error(y_val, y_pred)
rand_rmse = np.sqrt(rand_mse)
rand_rmse
# It was overfitting a bit.
score = rmsle_cv(rand_model, X_val, y_val)
print("Random Forest score: {:.4f}\n".format(score))
# Display scores next to attribute names.
# Reference Hands-On Machine Learning with Scikit Learn and Tensorflow
attributes = X_train
rand_results = rand_model.feature_importances_
cat_encoder = categorical_pipeline.named_steps["encoder"]
cat_features = list(cat_encoder.categories_[0])
total_features = lst_numerical + cat_features
feature_importance = sorted(zip(rand_results, total_features), reverse=True)
feature_arr = np.array(feature_importance)
# Top 10 features.
feature_scores = feature_arr[:,0][:10].astype(float)
feature_names = feature_arr[:,1][:10].astype(str)


d = {'feature_names': feature_names, 'feature_scores': feature_scores}
result_df = pd.DataFrame(data=d)

fig, ax = plt.subplots(figsize=(12,8))
ax = sns.barplot(x='feature_names', y='feature_scores', data=result_df, palette="coolwarm")
plt.title('RandomForestRegressor Feature Importances', fontsize=16)
plt.xlabel('Names of the Features', fontsize=14)
plt.ylabel('Feature Scores', fontsize=14)<jupyter_output><empty_output><jupyter_text>GradientBoostingRegressor:The Gradient Boosting Regressor class trains the models over the residuals (prediction errors) leading to smaller variances and higher accuracy.<jupyter_code>params = {'learning_rate': [0.05], 'loss': ['huber'], 'max_depth': [2], 'max_features': ['log2'], 'min_samples_leaf': [14], 
          'min_samples_split': [10], 'n_estimators': [3000]}


grad_boost = GradientBoostingRegressor(learning_rate=0.05, loss='huber', max_depth=2, 
                                       max_features='log2', min_samples_leaf=14, min_samples_split=10, n_estimators=3000,
                                       random_state=42)


grad_boost.fit(X_train, y_train)
y_pred = grad_boost.predict(X_val)
gboost_mse = mean_squared_error(y_val, y_pred)
gboost_rmse = np.sqrt(gboost_mse)
gboost_rmse
# Gradient Boosting was considerable better than RandomForest Regressor.
# scale salesprice.
# y_val = np.log(y_val)
score = rmsle_cv(grad_boost, X_val, y_val)
print("Gradient Boosting score: {:.4f}\n".format(score))<jupyter_output><empty_output><jupyter_text>StackingRegressor:In stacking regressor we combine different models and use the predicted values in the training set to mae further predictions. In case you want to go deeper into parameter "tuning" I left you the code above the different models so you can perform your own GridSearchCV and find even more efficient parameters!  ElasticNet  DecisionTreeRegressor  MLPRegressor (Later I will include it after learning more about neural networks)  SVR<jupyter_code># Define the models
from sklearn.linear_model import ElasticNet
from sklearn.svm import SVR
from sklearn.tree import DecisionTreeRegressor
from sklearn.linear_model import Lasso, Ridge

# Parameters for Ridge
# params = {"alpha": [0.5, 1, 10, 30, 50, 75, 125, 150, 225, 250, 500]}
# grid_ridge = GridSearchCV(Ridge(random_state=42), params)
# grid_ridge.fit(X_train, y_train)

# Parameters for DecisionTreeRegressor
# params = {"criterion": ["mse", "friedman_mse"], "max_depth": [None, 2, 3], "min_samples_split": [2,3,4]}

# grid_tree_reg = GridSearchCV(DecisionTreeRegressor(), params)
# grid_tree_reg.fit(X_train, y_train)



# Parameters for SVR
# params = {"kernel": ["rbf", "linear", "poly"], "C": [0.3, 0.5, 0.7, 0.7, 1], "degree": [2,3]}
# grid_svr = GridSearchCV(SVR(), params)
# grid_svr.fit(X_train, y_train)



# Tune Parameters for elasticnet
# params = {"alpha": [0.5, 1, 5, 10, 15, 30], "l1_ratio": [0.3, 0.5, 0.7, 0.9, 1], "max_iter": [3000, 5000]}
# grid_elanet = GridSearchCV(ElasticNet(random_state=42), params)

# Predictive Models
elastic_net = ElasticNet(alpha=0.1, l1_ratio=0.9, max_iter=3000)
svr = SVR(C=1, kernel='linear')
tree_reg = DecisionTreeRegressor(criterion='friedman_mse', max_depth=None, min_samples_split=3)
ridge_reg = Ridge(alpha=10)

# grid_elanet.fit(X_train, y_train)
from mlxtend.regressor import StackingRegressor
from sklearn.linear_model import LinearRegression
# Try tomorrow with svr_rbf = SVR(kernel='rbf')
# Check this website!
# Consider adding two more models if the score does not improve.
lin_reg = LinearRegression()

ensemble_model = StackingRegressor(regressors=[elastic_net, svr, rand_model, grad_boost], meta_regressor=SVR(kernel="rbf"))

ensemble_model.fit(X_train, y_train)


score = rmsle_cv(ensemble_model, X_val, y_val)
print("Stacking Regressor score: {:.4f}\n".format(score))
# We go for the stacking regressor model
# although sometimes gradientboosting might show to have a better performance.
final_pred = ensemble_model.predict(test_scaled)
# # Dataframe
final = pd.DataFrame()

# Id and Predictions
final['Id'] = test_id
final['SalePrice'] = np.expm1(final_pred)

# CSV file
final.to_csv('submission.csv', index=False) # Create Submission File
print('The File has been Submitted!')<jupyter_output><empty_output><jupyter_text>Keras and TensorFlow:Although the accuracy of our neural network  is still not as accurate as our ensemble and boosting model, I wanted to share two main aspects of tensorflow. Implementing a Neural Network with a real life regression scenario. Show the structure of Neural Networks through tensorboard (we will do this with ipython display.) (Reference: Hands On Machine Learning and TensorFlow by <NAME>)<jupyter_code># import tensorflow as tf

# import keras
# from keras import backend as K
# from keras.models import Sequential
# from keras.layers import Activation
# from keras.layers.core import Dense
# from keras.optimizers import Adam
# from keras.initializers import VarianceScaling



# # Reset the graph looks crazy
# def reset_graph(seed=42):
#     tf.reset_default_graph()
#     tf.set_random_seed(seed)
#     np.random.seed(seed)
    
    
# reset_graph()


# m, n = X_train.shape

# # Look at the preprocess data of the video and see the reshape part and apply it to X_train!
# # he_init = keras.initializers.VarianceScaling(scale=1.0, mode="fan_in", distribution='normal', seed=None)




# # Create a model (Add layers)
# model = Sequential([
#     Dense(n, input_shape=(n,), kernel_initializer='random_uniform', activation='relu'), # Start with the inputs
#     Dense(50, input_shape=(1,), kernel_initializer='random_uniform', activation='relu'), # Number of Layers
#     Dense(1, kernel_initializer='random_uniform')
# ]) 


# model.summary()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11225855804169058, 'fraction_numerical': 0.015855630313461638, 'mean_word_length': 4.870585833843641, 'pattern_counts': {'":': 10, '<': 144, '<?xml version=': 0, '>': 157, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '33', 'dir': 'jupyter-structured-clean-dedup', 'id': '497318', 'license': 'MIT', 'n_tokens_mistral': 18588, 'n_tokens_neox': 17445, 'n_words': 5641, 'path': '8 HOUSE PRICES/predicting-house-prices-regression-techniques.ipynb', 'repo_name': 'MLVPRASAD/KaggleProjects', 'size': '57585'}