<jupyter_start><jupyter_text>Feature Engineering & Encoding.This notebooks is about pre-processing and feature engineering, we will create artifacts in order to deploy our model from the calculations made here. This will serve us to create also our pipelines.**Regular Feature Engineering tasks:** - Geohashes. **DONE**- Difference between area total y area construida. **DONE**- Small differences between our mean district area's  **DONE****Spatial Feature Engineering tasks:**- Point of interest features **DONE**- Distance Bands **DONE**- GeoClustering**Encodings****Correlations**<jupyter_code>import pandas as pd
import numpy as np
import seaborn as sns
import geohash2 as gh #pip install geohash2
import gc

lima_data = pd.read_csv('../data/2020_Notebook01_clean_data.csv')
lima_data.describe()

#The coordinates were scraped without the -1
lima_data['latitud'] = lima_data['latitud']*-1
lima_data['longitud'] = lima_data['longitud']*-1<jupyter_output><empty_output><jupyter_text>Feature Engineering<jupyter_code>lima_data['areas_diff'] = lima_data['Area_total'] - lima_data['Area_constr']
lima_data['areas_proporcion'] = lima_data['Area_total']/lima_data['Area_constr']
drop_prop = lima_data[lima_data['areas_proporcion'] >= 4.0].index
lima_data.drop(drop_prop, inplace=True)
#This is as proxy to detect outliers, but it's not all the outlier processing 
tmp_table = lima_data.groupby('Barrio', as_index=False)['Area_total'].mean()
tmp_table = pd.merge(tmp_table, lima_data, on=['Barrio'], how='inner', suffixes=('_TMP','_ORIGIN'))
lima_data['diff_total_mean_district'] = tmp_table['Area_total_ORIGIN'] - tmp_table['Area_total_TMP']
lima_data.describe()<jupyter_output><empty_output><jupyter_text>GeoHashes<jupyter_code>for num in range(5, 10):
    lima_data['geohash_grado{}'.format(num)] = lima_data.apply(lambda x: gh.encode(x['latitud'], x['longitud'], precision = num),axis=1)
top_geohashes = lima_data['geohash_grado7'].value_counts().iloc[0:7].keys().tolist()

#Which discrits have more housing offer?
lima_data[lima_data['geohash_grado7'].isin(top_geohashes)]['Barrio'].value_counts()<jupyter_output><empty_output><jupyter_text>CAVEAT! Many posts doen't have exact coordinates but near reference points, that's why we will see same locations across different houses. We found 77 repeated coordinates but with different houses. In La Molina we take as reference EL Molicentro as well as Surco and San Isidro have their own point of reference<jupyter_code>lima_data[lima_data['geohash_grado7'] == top_geohashes[0]][['latitud','longitud']].value_counts()
print(lima_data[lima_data['geohash_grado7'] == top_geohashes[1]][['latitud','longitud']].value_counts())
print(lima_data[lima_data['geohash_grado7'] == top_geohashes[2]][['latitud','longitud']].value_counts())
lima_data.filter(regex='^(geohash*)').describe()<jupyter_output><empty_output><jupyter_text>Spatial Feature Engineering<jupyter_code>import geopandas 
import osmnx as ox

from shapely.geometry import Point
import warnings
warnings.filterwarnings('ignore')

#We built an extra feature that is not included in notebook2 output. If i have Price per total square meter.
#To build Price per constructed meter I'll have to pivot values.
lima_data['Precio_m2_constr'] = (lima_data['Precio_m2_total'] * lima_data['Area_total'])/lima_data['Area_constr']

#We restricted data only for Lima and Callao.
filter_1 = (lima_data['Ciudad'].isin(['Lima','Callao','Cañete']))
filter_2 = (lima_data['latitud']!= 0)

lima_data.where(filter_1 & filter_2 , inplace=True)

#Import the Lima Shape File per districts.

peru_districts = geopandas.read_file('../data/GeoShape_Distritos/DISTRITOS.shp')
lima_districts = peru_districts[peru_districts['DEPARTAMEN'] == 'LIMA']

#Aggregate Polygons geometries to get Lima Department Polygon:
lima_department = geopandas.GeoSeries([lima_districts[lima_districts['DEPARTAMEN']=='LIMA'].unary_union])

#Aggregate Polygons geometries to get Lima District Polygon:
lima_province = geopandas.GeoSeries([lima_districts[lima_districts['PROVINCIA']=='LIMA'].unary_union])

#We only awant districts inside Lima.
lima_filter = (peru_districts['DEPARTAMEN'] == 'LIMA') 

lima_districts = peru_districts[lima_filter]

#We need shapely object. We can transform lats and longs into a shapely.Point object

gdf_lima = geopandas.GeoDataFrame(
    lima_data, geometry=geopandas.points_from_xy(lima_data.longitud, lima_data.latitud))

gdf_lima = gdf_lima[gdf_lima['geometry'].apply(lambda x: x.within(lima_districts.values[0]))]  #Validator.

#Temporal drop: Since there are many outliers we'll drop some values.
# -------------------------------------------------------------------
gdf_lima = gdf_lima[gdf_lima['Precio_m2_total'] < 9000]

gdf_lima = gdf_lima[gdf_lima['Precio_m2_constr']<11000]
#Re-setting index
gdf_lima = gdf_lima.reset_index(drop=True).reset_index().rename(columns={'index':'ID'})
#Checkin the Coordinate reference system
gdf_lima = gdf_lima.set_crs('epsg:4326')
gdf_lima.crs<jupyter_output><empty_output><jupyter_text>Map Matching.We will extract point of interests from OSMnx and transpolate those features to our housing dataset. First we will restrict our total area of interest and then we will unify that dataframe with our origins. We are choosing these categories: 'school','restaurant','parking','marketplace','bank','college','university','bus_station','police','clinic'<jupyter_code>gdf_ch = gdf_lima.unary_union.convex_hull #Lima's Geometry

#amenities = ox.geometries_from_polygon(gdf_ch, tags={"amenity":['school','restaurant','parking','marketplace','bank','college','university','bus_station','police','clinic']})[["amenity","name","geometry"]]
#Inside Lima Filter
#amenities = amenities[amenities['geometry'].apply(lambda x: x.within(lima_province.values[0]))]
#amenities.to_csv('amenities.csv')

#Load our Amenities Dataset
from shapely import wkt
amenities = pd.read_csv('../data/to_load/amenities.csv')
geometry = amenities['geometry'].map(wkt.loads)
amenities.drop('geometry', axis=1, inplace=True)
amenities = geopandas.GeoDataFrame(amenities, crs='epsg:4326', geometry = geometry)
#Creating Buffer.
#--------------------------------------BUFFER HYPERPARAM--------------------------------------------
BUFFER_SIZE = 0.01
gdf_lima['buffer_500m'] = gdf_lima.buffer(BUFFER_SIZE)

#see one buffer to be sure makes sense.
buffer = gdf_lima.iloc[0:6,:]['buffer_500m']
fig, ax = plt.subplots(1, figsize=(12,12))
lima_province.plot(ax=ax,color='grey', alpha=0.70)
buffer.plot(ax=ax)<jupyter_output><empty_output><jupyter_text>Creating All Point Of interest Features:**CAVEAT:** 0.005 is an Hyperparameter that needs to be validated.<jupyter_code>### CREATING POINT OF INTEREST FEATURES
dict_poi = {}
gdf_lima['buffer_500m'] = gdf_lima.buffer(0.005)
for amenity in amenities['amenity'].unique():
    amenity_df = amenities[amenities['amenity'] == amenity]
    joined = geopandas.sjoin(amenity_df, gdf_lima.set_geometry('buffer_500m')[['ID','buffer_500m']], op='within')
    poi_count = joined.reset_index().groupby("ID")["osmid"].count().to_frame('{}_poi_count'.format(amenity))
    dict_poi[amenity] = poi_count
#Merge with Origin Data (GDF)
#GDF size = 3477,25 - dict-bank 2776,1
for amenity_name, poi_count in dict_poi.items():
    
    temp_df = dict_poi[amenity_name].reset_index()
    gdf_lima = gdf_lima.merge(temp_df, on=['ID'],how='left').fillna(0.0)<jupyter_output><empty_output><jupyter_text>Distance Bands and Distance Buffers:We will use Distance Bands weight objects, considering other houses as "neighbors" if they are within the distance threshold. In other words, we will be considering the median of some feature around the spatial distance that we will define. For now we will just make Dormitorios and Area_total based feature<jupyter_code>import libpysal.weights as weights

#THRESHOLD HYPERPARAMETER

THRESHOLD_HYPERPARAM = 500

d500_w = weights.DistanceBand.from_dataframe(gdf_lima, threshold=THRESHOLD_HYPERPARAM , silence_warnings=True)
d500_w.transform = 'r'

local_average_bedrooms = weights.lag_spatial(d500_w, gdf_lima[['Dormitorios']].values)

gdf_lima['local_average_bedrooms'] = local_average_bedrooms


area_500_w = weights.DistanceBand.from_dataframe(gdf_lima, threshold=500, silence_warnings=True)
area_500_w.transform = 'r'

local_average_total_area = weights.lag_spatial(area_500_w, gdf_lima[['Area_total']].values)
gdf_lima['local_average_total_area'] = local_average_total_area<jupyter_output><empty_output><jupyter_text>Dropping Geometry columns:<jupyter_code>gdf_lima.drop(['ID','geometry','buffer_500m'],axis=1,inplace=True)
lima_data = pd.DataFrame(gdf_lima)
lima_data.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 4052 entries, 0 to 4051
Data columns (total 36 columns):
 #   Column                    Non-Null Count  Dtype  
---  ------                    --------------  -----  
 0   latitud                   4052 non-null   float64
 1   longitud                  4052 non-null   float64
 2   Barrio                    4052 non-null   object 
 3   Ciudad                    4052 non-null   object 
 4   Precio                    4052 non-null   float64
 5   Area_total                4052 non-null   float64
 6   Area_constr               4052 non-null   float64
 7   MONEDA                    4052 non-null   object 
 8   Precio_m2_total           4052 non-null   float64
 9   Precio_m2_constr          4052 non-null   float64
 10  Precio_cat                4052 non-null   object 
 11  Area_constr_cat           4052 non-null   float64
 12  NroBanios                 4052 non-null   float64
 13  Dormitorios               4052 non-null   float64
 14  Antigued[...]<jupyter_text>Correlations<jupyter_code>import seaborn as sns
import matplotlib.pyplot as plt

corr = lima_data.corr()
sns.heatmap(corr)<jupyter_output><empty_output><jupyter_text>Target SelectionWe have 4 ways to define our target data.- Precio: Total price of the house- Precio_m2_total: Price / total square meters - Precio_m2_constr: Price / constructed square meters- Precio_cat: Categories prices<jupyter_code>X = lima_data.drop(lima_data.filter(regex='(P?p?recio)').columns,axis=1)
Y = lima_data.filter(regex='(P?p?recio)')
print('columnas Y: {}'.format(Y.columns.values))
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, Y['Precio_m2_total'], test_size=0.1, random_state = 215)
pd.concat([X_train,y_train],axis=1).to_csv('../data/2020_Notebook02_train_output.csv',index=False)

pd.concat([X_test,y_test],axis=1).to_csv('../data/2020_Notebook02_test_output.csv',index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1355118184386477, 'fraction_numerical': 0.02552029381297674, 'mean_word_length': 5.452004860267315, 'pattern_counts': {'":': 1, '<': 47, '<?xml version=': 0, '>': 46, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '46557', 'license': 'MIT', 'n_tokens_mistral': 3916, 'n_tokens_neox': 3655, 'n_words': 765, 'path': 'notebooks/2020_Notebook02_Feat_Eng.ipynb', 'repo_name': 'PBenavides/House_pricing_Lima', 'size': '10619'}