<jupyter_start><jupyter_text>The Categorical FeaturesBefore we begin, let's review the categorical features, remove features not required in our analysis, & convert to dummy variables<jupyter_code>categorical_columns = results.select_dtypes(include='object')
print(categorical_columns.info())<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 13136 entries, 249 to 14468
Data columns (total 15 columns):
 #   Column                           Non-Null Count  Dtype 
---  ------                           --------------  ----- 
 0   LandUseCode                      13136 non-null  object
 1   CountyName                       13136 non-null  object
 2   DwellingType                     13136 non-null  object
 3   TimeofLastDataUpdate             13136 non-null  object
 4   FileRecordType                   13136 non-null  object
 5   CountyCode                       13136 non-null  object
 6   MDPAccountID                     13136 non-null  object
 7   StreetAddress                    13136 non-null  object
 8   BuildingStyleCodeandDescription  13136 non-null  object
 9   CityName                         13136 non-null  object
 10  StreetAddressType                13135 non-null  object
 11  NoofStories                      13136 non-null  object
 12  ZipCode                       [...]<jupyter_text>We can safely remove many of categorical features existing in the original dataset since they are either used for processing the data at the source or do not contain relavent information. Let's remove these from our dataset<jupyter_code>results = results.drop(columns=[
    'LandUseCode', 
    'CountyName', 
    'TimeofLastDataUpdate', 
    'FileRecordType', 
    'CountyCode',
    'MDPAccountID',
    'StreetAddress',
    'BuildingStyleCodeandDescription',
    'CityName',
    'ZipCode',
    'StreetAddressUnits',
    'CurrentAssessmentYear',
    'NoofUnitsinDwelling',
    'YearBuilt'
])
# We are removed NoofUnitsinDwelling as we are focused only on residential properties, not multiple units (like an apartment)<jupyter_output><empty_output><jupyter_text>Now that we reduced our categorical variables to what we need for our model, let's convert these to dummy variables before we start our model<jupyter_code>#update Has Basement to 1 & 0 (Binary)
HaseBasementBool = pd.get_dummies(results['HasBasement'])
results['HasBasement']=HaseBasementBool[' With Basement ']
#Added Dummy Variables to NoofStories & DwellingType
NoofStories = pd.get_dummies(results['NoofStories'])
DwellingType = pd.get_dummies(results['DwellingType'])
StreetAddressType = pd.get_dummies(results['StreetAddressType'])
DwellingGrade = pd.get_dummies(results['DwellingGrade'])
#Joined to dummy variables & original continuous variables to new dataframe. Drop data we do not need in the model
processed_results = pd.concat([results, DwellingType ,NoofStories,StreetAddressType,DwellingGrade], axis=1, join='inner')
processed_results = processed_results.drop(['DwellingType','NoofStories','StreetAddressType','DwellingGrade'], axis = 1)
# processed_results.info()
processed_results.columns = ['SizeofHouse','HousingSalesPrice','PriorYearHousingSalesPrice','HasBasement','HouseAge','CenterUnit','EndUnit',
                             'SplitLevel','StandardUnit','1Floor','11/2Floors','2Floors','21/2Floors','3Floors','SingleFamily',
                             'TownHouse','2DwellingGrade','3DwellingGrade','4DwellingGrade','5DwellingGrade',
                             '6DwellingGrade','7DwellingGrade']

processed_results.info(null_counts = False)<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 13136 entries, 249 to 14468
Data columns (total 22 columns):
 #   Column                      Dtype
---  ------                      -----
 0   SizeofHouse                 int32
 1   HousingSalesPrice           int32
 2   PriorYearHousingSalesPrice  int32
 3   HasBasement                 uint8
 4   HouseAge                    int32
 5   CenterUnit                  uint8
 6   EndUnit                     uint8
 7   SplitLevel                  uint8
 8   StandardUnit                uint8
 9   1Floor                      uint8
 10  11/2Floors                  uint8
 11  2Floors                     uint8
 12  21/2Floors                  uint8
 13  3Floors                     uint8
 14  SingleFamily                uint8
 15  TownHouse                   uint8
 16  2DwellingGrade              uint8
 17  3DwellingGrade              uint8
 18  4DwellingGrade              uint8
 19  5DwellingGrade              uint8
 20  6DwellingGrade            [...]<jupyter_text>The Numerical featuresNow let's look at the numerical features that exist in our dataset<jupyter_code>numerical_columns = processed_results.select_dtypes(include=['int64', 'float64','int32'])
print(numerical_columns.info(null_counts = False))<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 13136 entries, 249 to 14468
Data columns (total 4 columns):
 #   Column                      Dtype
---  ------                      -----
 0   SizeofHouse                 int32
 1   HousingSalesPrice           int32
 2   PriorYearHousingSalesPrice  int32
 3   HouseAge                    int32
dtypes: int32(4)
memory usage: 307.9 KB
None<jupyter_text>Correlation of features to target variableNow let's see how well our features correlate to HousingSalesPrice target variable.<jupyter_code>figure, axes = plt.subplots(figsize=(13,13))

correlation_matrix = processed_results.corr()

cols = correlation_matrix.nlargest(21, 'HousingSalesPrice')['HousingSalesPrice'].index
correlation_matrix = np.corrcoef(processed_results[cols].values.T)

heatmap = sns.heatmap(
    correlation_matrix, 
#     cbar=True, 
    annot=True, 
    square=True, 
    cmap=plt.cm.RdYlGn,
    fmt='.1f', 
#     annot_kws={'size': 15}, 
    yticklabels=cols.values, 
    xticklabels=cols.values
)
heatmap.figure.savefig('correlation.png')<jupyter_output><empty_output><jupyter_text>Very interesting. Size of House, PriorYearHousingSalesPrice, and 5DwellingGrade have the biggest impact on housing prices. As HouseAge is a numerical feature with a low correlation, I will remove this from our model.<jupyter_code>processed_results = processed_results.drop(columns=['HouseAge'])<jupyter_output><empty_output><jupyter_text>Preprocessing Numerical Features Now let's preprocess our numerical features so the values do not have more weight than other features<jupyter_code>scaler = preprocessing.StandardScaler()
#applied standard scaling to adjust for houseage, size of house, and housing sales price
processed_results[['SizeofHouse','HousingSalesPrice','PriorYearHousingSalesPrice']]= scaler.fit_transform(processed_results[['SizeofHouse','HousingSalesPrice','PriorYearHousingSalesPrice']]) #add age<jupyter_output><empty_output><jupyter_text>Plotting Correlation of Continuous Features to Target VariableLet's see how the continuous features (Size of House) relate to HousingSalesPrice<jupyter_code># Plot the Seaborn pairplot
plot = sns.pairplot(processed_results,vars=['HousingSalesPrice','SizeofHouse','PriorYearHousingSalesPrice'])<jupyter_output><empty_output><jupyter_text>There's definitely a positive correlation between housing sales price, size of house and prior year housing sales price. Interestingly Size of House and PriorYearHousingSalesPrice have some extreme outliers, so I will remove them from the model<jupyter_code>q_low = processed_results["SizeofHouse"].quantile(0.05)
q_hi  = processed_results["SizeofHouse"].quantile(0.94)

processed_results = processed_results[(processed_results["SizeofHouse"] < q_hi) & (processed_results["SizeofHouse"] > q_low)]
q_low = processed_results["PriorYearHousingSalesPrice"].quantile(0.05)
q_hi  = processed_results["PriorYearHousingSalesPrice"].quantile(0.94)

processed_results = processed_results[(processed_results["PriorYearHousingSalesPrice"] < q_hi) & (processed_results["PriorYearHousingSalesPrice"] > q_low)]
processed_results.reset_index()<jupyter_output><empty_output><jupyter_text>Now let's replot after we have removed outliers<jupyter_code># Plot the Seaborn pairplot
plot = sns.pairplot(processed_results,vars=['HousingSalesPrice','SizeofHouse','PriorYearHousingSalesPrice'])
plot.savefig('pairplot.png')<jupyter_output><empty_output><jupyter_text>Nice. There's still a positive correlation between house sales price and size of house but now the outliers are removed and will not affect our model. Now that we've removed the outliers to SizeofHouse, there are no more residential properties where 7DwellingGrade = 1; we can remove this column from our dataset.<jupyter_code>processed_results['7DwellingGrade'].unique()
processed_results = processed_results.drop(columns=['7DwellingGrade'])<jupyter_output><empty_output><jupyter_text>Demonstrating Correlation of Categorical Features to Target VariableTo add an additional demonstration of the correlation between the categorical dummy variables and the target variable, housing sales price, I also want to use Point Biserial Correlation using scipy.stats.pointbiserialr's function. This is a great way to measure the relationship between a binary variable (in this case, our dummy variables) and a continuous variable (housing sales price). Return values that are closer to -1 or +1 imply high correlation.<jupyter_code>from scipy import stats

HouseSalesPrice = processed_results['HousingSalesPrice']
HasBasement = processed_results['HasBasement']
CenterUnit = processed_results['CenterUnit']
EndUnit = processed_results['EndUnit']
SplitLevel = processed_results['SplitLevel']
StandardUnit = processed_results['StandardUnit']
OneFloor = processed_results['1Floor']
OneHalfFloors = processed_results['11/2Floors']
TwoFloors = processed_results['2Floors']
TwoHalfFloors = processed_results['21/2Floors']
ThreeFloors = processed_results['3Floors']
SingleFamily = processed_results['SingleFamily']
TownHouse = processed_results['TownHouse']
TwoDwellingGrade = processed_results['2DwellingGrade']
ThreeDwellingGrade = processed_results['3DwellingGrade']
FourDwellingGrade = processed_results['4DwellingGrade']
FiveDwellingGrade = processed_results['5DwellingGrade']
SixDwellingGrade = processed_results['6DwellingGrade']

print("Point Biserial Correlation for Categorical Features:\n")
print("HasBasement: {}".format(stats.pointbiserialr(HouseSalesPrice, HasBasement)))
print("CenterUnit: {}".format(stats.pointbiserialr(HouseSalesPrice, CenterUnit))) 
print("EndUnit: {}".format(stats.pointbiserialr(HouseSalesPrice, EndUnit))) 
print("SplitLevel: {}".format(stats.pointbiserialr(HouseSalesPrice, SplitLevel))) 
print("OneFloor: {}".format(stats.pointbiserialr(HouseSalesPrice, OneFloor))) 
print("OneHalfFloors: {}".format(stats.pointbiserialr(HouseSalesPrice, OneHalfFloors))) 
print("TwoFloors: {}".format(stats.pointbiserialr(HouseSalesPrice, TwoFloors))) 
print("TwoHalfFloors: {}".format(stats.pointbiserialr(HouseSalesPrice, TwoHalfFloors))) 
print("ThreeFloors: {}".format(stats.pointbiserialr(HouseSalesPrice, ThreeFloors))) 
print("SingleFamily: {}".format(stats.pointbiserialr(HouseSalesPrice, SingleFamily))) 
print("TownHouse: {}".format(stats.pointbiserialr(HouseSalesPrice, TownHouse))) 
print("TwoDwellingGrade: {}".format(stats.pointbiserialr(HouseSalesPrice, TwoDwellingGrade))) 
print("ThreeDwellingGrade: {}".format(stats.pointbiserialr(HouseSalesPrice, ThreeDwellingGrade))) 
print("FourDwellingGrade: {}".format(stats.pointbiserialr(HouseSalesPrice, FourDwellingGrade))) 
print("FiveDwellingGrade: {}".format(stats.pointbiserialr(HouseSalesPrice, FiveDwellingGrade))) 
print("SixDwellingGrade: {}".format(stats.pointbiserialr(HouseSalesPrice, SixDwellingGrade)))<jupyter_output>Point Biserial Correlation for Categorical Features:

HasBasement: PointbiserialrResult(correlation=-0.1890392268049824, pvalue=2.7734482519581116e-84)
CenterUnit: PointbiserialrResult(correlation=-0.17999913540021634, pvalue=1.8334711698287518e-76)
EndUnit: PointbiserialrResult(correlation=-0.0914077681593651, pvalue=9.571125689768914e-21)
SplitLevel: PointbiserialrResult(correlation=0.17879076864861218, pvalue=1.8955797622494059e-75)
OneFloor: PointbiserialrResult(correlation=-0.3637217674014366, pvalue=8e-323)
OneHalfFloors: PointbiserialrResult(correlation=-0.07643871702010847, pvalue=5.898004181636372e-15)
TwoFloors: PointbiserialrResult(correlation=0.24006795193071262, pvalue=2.955137993563317e-136)
TwoHalfFloors: PointbiserialrResult(correlation=0.3757536073434737, pvalue=0.0)
ThreeFloors: PointbiserialrResult(correlation=0.07812079183122062, pvalue=1.4823162460901786e-15)
SingleFamily: PointbiserialrResult(correlation=0.2080230395466467, pvalue=4.790576023072144e-102)
TownHouse[...]<jupyter_text>Very interesting. The dummy variables with the most impact (30% or greater correlation) include: `'PriorYearHousingSalesPrice'`, `'OneFloor'`,  `'TwoDwellingGrade'`. Let's keep all variables in the model for now to see how our accuracy score reflects the models.  Saving Updated DataNow that we have our data transformed and performed a thorough examination of the features, let's go ahead and save the data off to a pickle for adding to our model<jupyter_code>processed_results.to_pickle('MDHousingDataset_PreProcessed.pickle')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09280060309084055, 'fraction_numerical': 0.046362608367885416, 'mean_word_length': 4.765319426336376, 'pattern_counts': {'":': 0, '<': 54, '<?xml version=': 0, '>': 54, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '436345', 'license': 'MIT', 'n_tokens_mistral': 4508, 'n_tokens_neox': 3868, 'n_words': 906, 'path': 'Step 2 Data Preprocessing - Analyzing, Dummy & Standardizing Data.ipynb', 'repo_name': 'Lwhieldon/House-Price-Prediction-Using-MD-Data-Portal', 'size': '13265'}