<jupyter_start><jupyter_text>OPTION A  Present the code and methods for acquiring the data. Loading the data into appropieate format for analysis. Explain the process and results  At the beginning we must import so numpy as panda libraries<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>We are importing csv file from my drive on google<jupyter_code>df = pd.read_csv("AB_NYC_2019.csv")<jupyter_output><empty_output><jupyter_text>To be sure dataset has been created successfully we can check the first terms of it. To do that let us use next attribute of dataset<jupyter_code>df.head()<jupyter_output><empty_output><jupyter_text>We are checking the name of the categories of tha dataset<jupyter_code>columns = df.columns
print(columns)
categories_number = len(columns)
print("Number of categories: ", categories_number)<jupyter_output>Index(['id', 'name', 'host_id', 'host_name', 'neighbourhood_group',
       'neighbourhood', 'latitude', 'longitude', 'room_type', 'price',
       'minimum_nights', 'number_of_reviews', 'last_review',
       'reviews_per_month', 'calculated_host_listings_count',
       'availability_365'],
      dtype='object')
Number of categories:  16<jupyter_text>Let us see how many rows dataset has right now<jupyter_code>df.shape<jupyter_output><empty_output><jupyter_text>Let us set up how many rows will be shown when we use methods like head(), tail(), describe()...<jupyter_code>pd.options.display.max_rows = 20
df.head()<jupyter_output><empty_output><jupyter_text>Let us check how many NotANumber are present in dataset<jupyter_code>not_a_number = df.isna().sum()
print(not_a_number)<jupyter_output>id                                    0
name                                 16
host_id                               0
host_name                            21
neighbourhood_group                   0
neighbourhood                         0
latitude                              0
longitude                             0
room_type                             0
price                                 0
minimum_nights                        0
number_of_reviews                     0
last_review                       10052
reviews_per_month                 10052
calculated_host_listings_count        0
availability_365                      0
dtype: int64<jupyter_text>We can verify only name, host_name, last_review and reviews_per_month contain NAN elements. To solve that situation we may replace the previous terms that are equal to NAN by 0  It does not matter whether either name or host_name have some terms equal to NAN, what is to say, we do not need to fix them up. On the other hand we might try to replace NAN terms presented in last_review, the best way to do this would be purge all NAN terms because this category is a date category and we do not have any way to guess or interpolate those NAN terms<jupyter_code>df_cleaned = df.dropna()
df_cleaned.head()<jupyter_output><empty_output><jupyter_text>Let us check whether there is any NAN term in dataframe<jupyter_code>df_cleaned.isna().sum()<jupyter_output><empty_output><jupyter_text>Let us start with latitude category<jupyter_code>new_len = df_cleaned.shape[0]
x_axis = np.array(range(1,new_len+1))
y_axis = df_cleaned["latitude"]
y_max = df_cleaned["latitude"].max()
y_min = df_cleaned["latitude"].min()
print(f"Type of y-axis: {y_axis.dtype}")
print(f"Latitude: y_max: {y_max} || y_min: {y_min}")
plt.plot(x_axis, y_axis)<jupyter_output>Type of y-axis: float64
Latitude: y_max: 40.913059999999994 || y_min: 40.506409999999995<jupyter_text>We have just checked latitude values are within [40.9306, 40.50605] interval. There is not any singular value in latitude category in order to type of y_axis is float64. If there were any non float64 type term, the type of y_axis would be object.   We are proceeding in the same way as before for longitude category<jupyter_code>y_axis = df_cleaned["longitude"]
y_max = df_cleaned["longitude"].max()
y_min = df_cleaned["longitude"].min()
print(f"Type of y_axis: {y_axis.dtype}")
print(f"Longitude: y_max: {y_max} || y_min: {y_min}")
plt.plot(x_axis, y_axis)<jupyter_output>Type of y_axis: float64
Longitude: y_max: -73.71299 || y_min: -74.24441999999999<jupyter_text>In this case, we can affirm type of y_axis is float64 too, therefore any value of longitude category belongs to [-73.713, -74.25] interval  Let us proceed with price category. We can confirm any singular value is present in this category<jupyter_code>y_axis = df_cleaned["price"]
y_max = df_cleaned["price"].max()
y_min = df_cleaned["price"].min()
print(f"Type of y_axis is: {y_axis.dtype}")
print(f"Price: y-max: {y_max} || y_min: {y_min}")
plt.plot(x_axis, y_axis)
df_cleaned.loc[df_cleaned.price > 1500,:].value_counts().sum()<jupyter_output><empty_output><jupyter_text>We have just checked that there are only 58 items whose values are greater then 1500$. They are going to su deleted because they can be considered as outliers.<jupyter_code>df_cleaned = df_cleaned.loc[df_cleaned.price < 1500,:]<jupyter_output><empty_output><jupyter_text>Let us proceed with minimum_nights<jupyter_code>x_axis = np.array(range(df_cleaned.shape[0]))
y_axis = df_cleaned["minimum_nights"]
y_max = df_cleaned["minimum_nights"].max()
y_min = df_cleaned["minimum_nights"].min()
print(f"Type of df_cleaned.minimum_nights is: ",y_axis.dtype)
print(f"Minimum_nights: y-max: {y_max}|| y-min: {y_min}")
plt.plot(x_axis, y_axis)<jupyter_output>Type of df_cleaned.minimum_nights is:  int64
Minimum_nights: y-max: 1250|| y-min: 1<jupyter_text>In this case we can affirm there are three singular values at least. Every value is lower than 600. We are removing these singular rows from dataframe<jupyter_code>df_minimum_nights_high = df_cleaned.loc[df_cleaned.minimum_nights >= 800,:]
print(df_minimum_nights_high.minimum_nights)
df_cleaned_2 = df_cleaned.loc[df_cleaned.minimum_nights < 800,:]
print(df_cleaned_2)<jupyter_output>5767     1250
13404     999
26341     999
Name: minimum_nights, dtype: int64
             id  ... availability_365
0          2539  ...              365
1          2595  ...              355
3          3831  ...              194
4          5022  ...                0
5          5099  ...              129
...         ...  ...              ...
48782  36425863  ...              147
48790  36427429  ...              339
48799  36438336  ...               87
48805  36442252  ...               40
48852  36455809  ...                1

[38752 rows x 16 columns]<jupyter_text>We do affirm there were only three unusual values of minimum_nights because we had 38752 rows in df_cleaned, we have determined there are only three values in minimum_nights category over 800.  Let us proceed with number_of_reviews category. In this case we must create another x_axis with whose length must be equals to 38818<jupyter_code>new_len = df_cleaned_2.shape[0]
x_axis = np.array(range(1, new_len+1))
y_axis = df_cleaned_2["number_of_reviews"]
y_max = df_cleaned_2["number_of_reviews"].max()
y_min = df_cleaned_2["number_of_reviews"].min()
print(f"Type of y-axis: ", y_axis.dtype)
print(f"Number of reviews: y-max: {y_max} || y-min: {y_min}")
plt.plot(x_axis, y_axis)<jupyter_output>Type of y-axis:  int64
Number of reviews: y-max: 629 || y-min: 1<jupyter_text>In this case, we can affirm there is not any singular value in number_of_reviewes category, because its dtype is int64 what means every value in this category is int64 dtype. Moreover every value of this category belongs to [1, 629] interval. We also do affirm values of this category are in decline  Let us start with reviews_per_month<jupyter_code>y_axis = df_cleaned_2["reviews_per_month"]
y_max = df_cleaned_2["reviews_per_month"].max()
y_min = df_cleaned_2["reviews_per_month"].min()
print(f"Reviews-per-month: y-max: {y_max} || y-min: {y_min}")
print(f"dtype of y_axis: {y_axis.dtype}")
plt.plot(x_axis, y_axis)<jupyter_output>Reviews-per-month: y-max: 58.5 || y-min: 0.01
dtype of y_axis: float64<jupyter_text>We can check it seems to show a singular value what might alters the results.One way to remove this singular value may be<jupyter_code>df_cleaned_2_dropped = df_cleaned_2.loc[df_cleaned_2.reviews_per_month > 50,:]
print(len(df_cleaned_2_dropped))<jupyter_output>1<jupyter_text>There is only one singular value in this category.<jupyter_code>df_cleaned_3 = df_cleaned_2.loc[df_cleaned_2.reviews_per_month <= 50,:]
new_len = df_cleaned_3.shape[0]
x_axis = np.array(range(1,new_len+1))
y_axis = df_cleaned_3["reviews_per_month"]
y_max = df_cleaned_3["reviews_per_month"].max()
y_min = df_cleaned_3["reviews_per_month"].min()
print(f"dtype of reviews_per_month_dropeed: ", y_axis.dtype)
print(f"reviews_per_month_dropped: y-max: {y_max} || y-min: {y_min}")
plt.plot(x_axis, y_axis)<jupyter_output>dtype of reviews_per_month_dropeed:  float64
reviews_per_month_dropped: y-max: 27.95 || y-min: 0.01<jupyter_text>Values in this category are in increase  Let us start with calculated_host_listings_count<jupyter_code>y_axis = df_cleaned_3["calculated_host_listings_count"]
y_max = df_cleaned_3["calculated_host_listings_count"].max()
y_min = df_cleaned_3["calculated_host_listings_count"].min()
print(f"dtype of y_axis: ", y_axis.dtype)
print(f"calculated_host_listings_count: y-max: {y_max} || y-min: {y_min}")
plt.plot(x_axis, y_axis)<jupyter_output>dtype of y_axis:  int64
calculated_host_listings_count: y-max: 327 || y-min: 1<jupyter_text>This picture does not show any singular value  Let us start with availability_365 category<jupyter_code>y_axis = df_cleaned_3["availability_365"]
y_max = df_cleaned_3["availability_365"].max()
y_min = df_cleaned_3["availability_365"].min()
print(f"dtype of df_cleaned_3.availability_365: ", y_axis.dtype)
print(f"df_cleaned_3.availability_365: y-max: {y_max} || y-min: {y_min}")
plt.plot(x_axis, y_axis)<jupyter_output>dtype of df_cleaned_3.availability_365:  int64
df_cleaned_3.availability_365: y-max: 365 || y-min: 0<jupyter_text>This category does not seem to show any singular data either  In this point we have cleaned the initial dataset. We are checking the correlation matrix of df_cleaned_3<jupyter_code>corr_df = df_cleaned_3.corr(method="pearson")
print(corr_df)
plt.matshow(corr_df)
plt.show()<jupyter_output><empty_output><jupyter_text>We can check the correlations coefficients betweent categories. We could use heatmap from seaborn library to have a similar graphic with all correlation coefficients as legends<jupyter_code>import seaborn as sns
sns.heatmap(corr_df, annot=True)<jupyter_output><empty_output><jupyter_text>The next grapich does take over 1 minuto to be shown, it consists in every correlation scatter plot for avery couple of cartegories that are present in df_cleaned_3. Thus we can appreciate which category has more influence over price values<jupyter_code>#sns.pairplot(df_cleaned_3, size=2.5)<jupyter_output><empty_output><jupyter_text>We also can have a similar graphic using attributes of df_Corr object<jupyter_code>corr_df.style.background_gradient(cmap='coolwarm')<jupyter_output><empty_output><jupyter_text>There are some categories having correlations with each other. These caterories are:- id | host_id- number_of_reviews | reviews_per_month  The only correlation that might me interesting is the second one. To see more in details we can try to get the positive correlation existing between number_of_reviews and reviews_per_month categories<jupyter_code>x_axis = df_cleaned_3["reviews_per_month"]
y_axis = df_cleaned_3["number_of_reviews"]
plt.scatter(x_axis, y_axis)<jupyter_output><empty_output><jupyter_text>As picture above shows, all values of number_of_reviews are distributed within two wrapping lines  To get both wrapping lines we must manipulate last values of x_axis and y_axis. We are getting a subsample of y_axis and x_axis whose values are whithin the interval defining by df_cleaned_3.loc[df_cleaned_3.number_of_reviews <= 20]...<jupyter_code>from sklearn import linear_model
x_subsample = df_cleaned_3.loc[df_cleaned_3.number_of_reviews <= 20, "reviews_per_month"]
y_subsample = df_cleaned_3.loc[df_cleaned_3.number_of_reviews <= 20, "number_of_reviews"]
print("len subsample: ", len(x_subsample))
plt.scatter(x_subsample, y_subsample)
regr = linear_model.LinearRegression()
x_train = np.array(x_subsample).reshape(-1,1)
y_train = np.array(y_subsample).reshape(-1,1)
regr.fit(x_train, y_train)
y_pred = regr.predict(x_train)
plt.plot(x_train, y_pred)
print("Coeficient: ", regr.coef_)
print("Intercept: ", regr.intercept_)<jupyter_output>len subsample:  25524
Coeficient:  [[1.4566522]]
Intercept:  [4.99382864]<jupyter_text>We already know the expression or the lower wrapping liner:*y(x) = 1.456 x + 4.9949*  However we must keep in mind we have got the expression of lower wrapping line based on a subsample of df_3_cleaned, what is to say, the mathematical expression is based on an x_axis whose length is 25581. On the contrary df_cleaned_3 has 38817 rows.   In short, the expression of lower wrapping line might not be strictly correct but it seems to fit properly to df_cleaned_3<jupyter_code>x_axis = df_cleaned_3["reviews_per_month"]
y_axis = df_cleaned_3["number_of_reviews"]
plt.scatter(x_axis, y_axis)
y_lwl = regr.coef_[0][0] * x_axis + regr.intercept_[0]
plt.plot(x_axis, y_lwl)<jupyter_output><empty_output><jupyter_text>Let us analyze price category, we may consider to use either a *technique of machine learning* or even some *kind of neural network*, but first of all we should know the behaviour of values of this category. To summarize, we must know whether values of price category are randomly distributed, to know that we must apply a goodness of fit test to price category.    We must determine CFD function of this category  We are taking values of the price category in order to try to make some predicictions   Let us analyze price category, we may consider to use either a *technique of machine learning* or even some *kind of neural network*, but first of all we should know the behaviour of values of this category. To summarize, we must know whether values of price category are randomly distributed, to know that we must apply a goodness of fit test to price category.    We must determine Cumulative Frecuency Distribution (CFD) of price values<jupyter_code>#cfd = pd.DataFrame(x_train).value_counts().cumsum() / 12400
cfd = df_cleaned_3.loc[:,"price"].value_counts().cumsum() / 38751
print(cfd)
print(cfd)
x_axis = np.array(range(0, len(cfd)))
print(len(cfd))
plt.plot(x_axis, cfd)

from scipy.stats import anderson, kstest
ad_test = anderson(cfd, dist="norm")
ks_test = kstest(cfd, "norm", N=len(cfd))
print("Test: ", ad_test)
print("ks_test: ", ks_test)<jupyter_output>Test:  AndersonResult(statistic=77.9704786075531, critical_values=array([0.572, 0.651, 0.781, 0.911, 1.084]), significance_level=array([15. , 10. ,  5. ,  2.5,  1. ]))
ks_test:  KstestResult(statistic=0.6761349509311663, pvalue=2.494693618546248e-249)<jupyter_text>We must take in consideration critical statistics values of Kolmogorov-Smirnov (KS) test. If n is the size of the sample and is greater than 35:- alpha = 0.2 ---> s* = 1.07 / n^0.5 --> 4.85e-2- alpha = 0.15 --> s* = 1.14 / n^0.5 --> 5.17e-2- alpha = 0.1 ---> s* = 1.22 / n^0.5 --> 5.53e-2- alpha = 0.05 --> s* = 1.36 / n^0.5 --> 6.17e-2- alpha = 0.01 --> s* = 1.63 / n^0.5 --> 7.39e-2alpha would be the significance_level of results of Anderson-Darling test  In this point we can consider two hypothesis:- Ho : price values follow a Normal distribution- Hi : price values do not follow a Normal distributionIn Anderson-Darling test we have got a statistic value which is greater than every of its *critical_values*, moreover in case of Kolmogorov-Smirnov test we also have got a statistic value which is greater than every of its s* (critical_values) with p-value = 2.49e-249As a consecuence we do affirm we have not enough information to reject Hi, in addition we also do affirm *price category does not follow a Normal distribution*  As a previous result we can consider to use some kinf of predictor, for instance a predictor based on cross-validation methods  In this point we should know what categories seems to have more influence over price values. To get these categories we have already used a correlogram-table, so in this case   Checking neighbourhood category in dataframe we can see there are three principal groups: Manhattan, Brooklyn and Queens. Instead of trying to make an estimation of rental price in NY we can split up the problem into three parts. In short, we are doing three models Manhattan<jupyter_code>df_manhattan = df_cleaned_3.loc[df_cleaned_3.neighbourhood_group == "Manhattan", :]
corr_manhattan = df_manhattan.corr(method="pearson")
corr_manhattan.style.background_gradient(cmap='coolwarm')<jupyter_output><empty_output><jupyter_text>The categories have more influence on price are: longitude, latitude and availability_365. Both of them have a negative correlation coefficient respect to price<jupyter_code>from sklearn.model_selection import train_test_split
import pandas as pd
from sklearn.preprocessing import MinMaxScaler
features = df_manhattan.loc[:,["latitude", "longitude", "availability_365"]]
target = df_manhattan.loc[:,"price"]
scaler = MinMaxScaler(feature_range=(0,1))
prices_man = np.array(df_manhattan.loc[:,"price"]).reshape(-1,1)
latitude_man = np.array(df_manhattan.loc[:,"latitude"]).reshape(-1,1)
longitude_man = np.array(df_manhattan.loc[:,"longitude"]).reshape(-1,1)
availability_man = np.array(df_manhattan.loc[:,"availability_365"]).reshape(-1,1)
prices_min = prices_man.min()
prices_max = prices_man.max()
prices_n_man = scaler.fit_transform(prices_man)
latitude_n_man = scaler.fit_transform(latitude_man)
longitude_n_man = scaler.fit_transform(longitude_man)
availability_n_man = scaler.fit_transform(availability_man)
new_len = len(prices_man)
features_matrix = np.zeros((new_len, 4))
features_matrix[:,0] = latitude_n_man[:,0]
features_matrix[:,1] = longitude_n_man[:,0]
features_matrix[:,2] = availability_n_man[:,0]
features_matrix[:,3] = prices_n_man[:,0]
features_df_man = pd.DataFrame(features_matrix)
features_df_man.columns = ["latitude", "longitude", "availability_365", "price"]
features_n_man = features_df_man.loc[:,["latitude", "longitude", "availability_365"]]
target_n_man = features_df_man.loc[:,"price"]
print(features_df_man)
X_train, X_test, y_train, y_test = train_test_split(features_n_man, target_n_man, test_size=0.25, random_state=42)
X_train
from keras.models import Sequential
from keras.layers import Dense, Dropout
from tensorflow.keras.optimizers import RMSprop, Adam

model = Sequential()
model.add(Dense(128, activation="sigmoid"))
model.add(Dropout(0.3))
model.add(Dense(64, activation="sigmoid"))
model.add(Dropout(0.2))
model.add(Dense(32, activation="sigmoid"))
model.add(Dropout(0.25))
model.add(Dense(1, activation="sigmoid"))

my_optimizer = Adam(learning_rate=0.001)

model.compile(loss="mean_squared_error", optimizer= my_optimizer ,metrics=["mean_absolute_error", "mean_squared_error"])

model.fit(X_train, y_train, epochs=100, batch_size=50)
y_pred = model.predict(X_test)
y_test_np = np.array(y_test)
x_axis = np.array(range(len(y_test_np)))
x_axis.size
plt.scatter(x_axis, y_test, label="real")
plt.scatter(x_axis, y_pred[:,0], label="estimado")
plt.title("Indicador de precio MinMaxScaled", fontsize=10, color="SteelBlue", fontweight="bold")
plt.ylabel("Indicador de precio MinMaxScaled estimado y real", fontsize=7, color="SteelBlue", fontweight="bold")
plt.legend()
y_pred
price_test = y_test * (prices_max - prices_min) + prices_min
price_pred = y_pred * (prices_max - prices_min) + prices_min
print(type(price_test))
print(type(price_pred))
price_test_np = np.array(price_test)
print(type(price_test_np))
print(price_test_np)
print(np.array(price_pred))
error = abs(price_pred[:,0] - price_test)
x_axis = np.array(range(len(price_pred)))
plt.scatter(x_axis, error)
plt.title("abs. Error Precio estimado - Precio real sobre x_test",fontsize=10,color="SteelBlue",fontweight="bold")
plt.ylabel("Error en valor absoluto de precio estimado y precio real en $", color="SteelBlue", fontweight="bold")
error
y_test
y_pred
error.describe()<jupyter_output><empty_output><jupyter_text>Brooklyn<jupyter_code>df_brooklyn = df_cleaned_3.loc[df_cleaned_3.neighbourhood_group == "Brooklyn", :]
corr_brooklyn = df_brooklyn.corr(method="pearson")
corr_brooklyn.style.background_gradient(cmap='coolwarm')<jupyter_output><empty_output><jupyter_text>We can see in the correlation matrix above the categories that has more influence in price values are: latitude, longitude and availability_365<jupyter_code>from sklearn.model_selection import train_test_split
import pandas as pd
from sklearn.preprocessing import MinMaxScaler
features_bro = df_brooklyn.loc[:,["latitude", "longitude", "availability_365"]]
target_bro = df_brooklyn.loc[:,"price"]
scaler_bro = MinMaxScaler(feature_range=(0,1))
prices_bro = np.array(df_brooklyn.loc[:,"price"]).reshape(-1,1)
latitude_bro = np.array(df_brooklyn.loc[:,"latitude"]).reshape(-1,1)
longitude_bro = np.array(df_brooklyn.loc[:,"longitude"]).reshape(-1,1)
availability_bro = np.array(df_brooklyn.loc[:,"availability_365"]).reshape(-1,1)
prices_min_bro = prices.min()
prices_max_bro = prices.max()
prices_n_bro = scaler_bro.fit_transform(prices_bro)
latitude_n_bro = scaler_bro.fit_transform(latitude_bro)
longitude_n_bro = scaler_bro.fit_transform(longitude_bro)
availability_n_bro = scaler_bro.fit_transform(availability_bro)
new_len = len(prices_bro)
features_matrix_bro = np.zeros((new_len, 4))
features_matrix_bro[:,0] = latitude_n_bro[:,0]
features_matrix_bro[:,1] = longitude_n_bro[:,0]
features_matrix_bro[:,2] = availability_n_bro[:,0]
features_matrix_bro[:,3] = prices_n_bro[:,0]
features_df_bro = pd.DataFrame(features_matrix_bro)
features_df_bro.columns = ["latitude", "longitude", "availability_365", "price"]
features_n_bro = features_df_bro.loc[:,["latitude", "longitude", "availability_365"]]
target_n_bro = features_df_bro.loc[:,"price"]
print(features_df_bro)
X_train_bro, X_test_bro, y_train_bro, y_test_bro = train_test_split(features_n_bro, target_n_bro, test_size=0.25, random_state=27)
X_train_bro
from keras.models import Sequential
from keras.layers import Dense, Dropout
from tensorflow.keras.optimizers import RMSprop, Adam

model = Sequential()
model.add(Dense(128, activation="sigmoid"))
model.add(Dropout(0.2))
model.add(Dense(64, activation="sigmoid"))
model.add(Dropout(0.2))
model.add(Dense(32, activation="sigmoid"))
model.add(Dense(1, activation="sigmoid"))

my_optimizer = Adam(learning_rate=0.001)

model.compile(loss="mean_squared_error", optimizer= my_optimizer ,metrics=["mean_absolute_error", "mean_squared_error"])

model.fit(X_train_bro, y_train_bro, epochs=100, batch_size=60)
y_pred_bro = model.predict(X_test_bro)
y_test_np_bro = np.array(y_test_bro)
x_axis = np.array(range(len(y_test_np_bro)))
x_axis.size
plt.scatter(x_axis, y_test_bro, label="real")
plt.scatter(x_axis, y_pred_bro[:,0], label="estimado")
plt.title("Indicador de precio MinMaxScaled", fontsize=10, color="SteelBlue", fontweight="bold")
plt.ylabel("Indicador de precio MinMaxScaled estimado y real", fontsize=7, color="SteelBlue", fontweight="bold")
plt.legend()
price_test_bro = y_test_bro * (prices_max_bro - prices_min_bro) + prices_min_bro
price_pred_bro = y_pred_bro * (prices_max_bro - prices_min_bro) + prices_min_bro
print(type(price_test_bro))
print(type(price_pred_bro))
price_test_np_bro = np.array(price_test_bro)
print(type(price_test_np_bro))
print(price_test_np_bro)
print(np.array(price_pred_bro))
error_bro = abs(price_pred_bro[:,0] - price_test_bro)
x_axis_bro = np.array(range(len(price_pred_bro)))
plt.scatter(x_axis_bro, error_bro)
plt.title("abs. Error Precio estimado - Precio real sobre x_test",fontsize=10,color="SteelBlue",fontweight="bold")
plt.ylabel("Error en valor absoluto de precio estimado y precio real en $", color="SteelBlue", fontweight="bold")
error_bro
y_test_bro
y_pred_bro
error_bro.describe()<jupyter_output><empty_output><jupyter_text>Queens<jupyter_code>df_queens = df_cleaned_3.loc[df_cleaned_3.neighbourhood_group == "Queens", :]
corr_queens = df_queens.corr(method="pearson")
corr_queens.style.background_gradient(cmap='coolwarm')<jupyter_output><empty_output><jupyter_text>We can see in the correlation matrix above the categories that has more influence in price values are: reviews_per_month, calculated_host_listings_count	 and availability_365<jupyter_code>from sklearn.model_selection import train_test_split
import pandas as pd
from sklearn.preprocessing import MinMaxScaler
features_que = df_queens.loc[:,["reviews_per_month", "calculated_host_listings_count", "availability_365"]]
target_que = df_queens.loc[:,"price"]
scaler_que = MinMaxScaler(feature_range=(0,1))
prices_que = np.array(df_queens.loc[:,"price"]).reshape(-1,1)
reviews_que = np.array(df_queens.loc[:,"reviews_per_month"]).reshape(-1,1)
calculated_que = np.array(df_queens.loc[:,"calculated_host_listings_count"]).reshape(-1,1)
availability_que = np.array(df_queens.loc[:,"availability_365"]).reshape(-1,1)
prices_min_que = prices_que.min()
prices_max_que = prices_que.max()
prices_n_que = scaler_que.fit_transform(prices_que)
reviews_n_que = scaler_que.fit_transform(reviews_que)
calculated_n_que = scaler_que.fit_transform(calculated_que)
availability_n_que = scaler_que.fit_transform(availability_que)
new_len_que = len(prices_que)
features_matrix_que = np.zeros((new_len_que, 4))
features_matrix_que[:,0] = reviews_n_que[:,0]
features_matrix_que[:,1] = calculated_n_que[:,0]
features_matrix_que[:,2] = availability_n_que[:,0]
features_matrix_que[:,3] = prices_n_que[:,0]
features_df_que = pd.DataFrame(features_matrix_que)
features_df_que.columns = ["reviews_per_month", "calculated_host_listings_count", "availability_365", "price"]
features_n_que = features_df_que.loc[:,["reviews_per_month", "calculated_host_listings_count", "availability_365"]]
target_n_que = features_df_que.loc[:,"price"]
print(features_df_que)
X_train_que, X_test_que, y_train_que, y_test_que = train_test_split(features_n_que, target_n_que, test_size=0.25, random_state=73)
X_train_que
from keras.models import Sequential
from keras.layers import Dense, Dropout
from tensorflow.keras.optimizers import RMSprop, Adam

model = Sequential()
model.add(Dense(128, activation="tanh"))
model.add(Dropout(0.2))
model.add(Dense(64, activation="tanh"))
model.add(Dropout(0.2))
model.add(Dense(32, activation="tanh"))
model.add(Dense(1, activation="tanh"))

my_optimizer = Adam(learning_rate=0.001)

model.compile(loss="mean_squared_error", optimizer= my_optimizer ,metrics=["mean_absolute_error", "mean_squared_error"])

model.fit(X_train_que, y_train_que, epochs=100, batch_size=50)
y_pred_que = model.predict(X_test_que)
y_test_np_que = np.array(y_test_que)
x_axis_que = np.array(range(len(y_test_np_que)))
x_axis_que.size
plt.scatter(x_axis_que, y_test_que, label="real")
plt.scatter(x_axis_que, y_pred_que[:,0], label="estimado")
plt.title("Indicador de precio MinMaxScaled", fontsize=10, color="SteelBlue", fontweight="bold")
plt.ylabel("Indicador de precio MinMaxScaled estimado y real", fontsize=7, color="SteelBlue", fontweight="bold")
plt.legend()
price_test_que = y_test_que * (prices_max_que - prices_min_que) + prices_min_que
price_pred_que = y_pred_que * (prices_max_que - prices_min_que) + prices_min_que
print(type(price_test_que))
print(type(price_pred_que))
price_test_np_que = np.array(price_test_que)
print(type(price_test_np_que))
print(price_test_np_que)
print(np.array(price_pred_que))
error_que = abs(price_pred_que[:,0] - price_test_que)
x_axis_que = np.array(range(len(price_pred_que)))
plt.scatter(x_axis_que, error_que)
plt.title("abs. Error Precio estimado - Precio real sobre x_test",fontsize=10,color="SteelBlue",fontweight="bold")
plt.ylabel("Error en valor absoluto de precio estimado y precio real en $", color="SteelBlue", fontweight="bold")
error_que
price_test_que
price_pred_que
error_que.describe()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10869719812259991, 'fraction_numerical': 0.031218887782676716, 'mean_word_length': 6.114849481406527, 'pattern_counts': {'":': 0, '<': 133, '<?xml version=': 0, '>': 140, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 3, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '35', 'dir': 'jupyter-structured-clean-dedup', 'id': '499673', 'license': 'MIT', 'n_tokens_mistral': 10397, 'n_tokens_neox': 9764, 'n_words': 2363, 'path': 'Prueba_5_Raul_Vergel_Romero_Opcion_A.ipynb', 'repo_name': 'rvrfront/modulo5', 'size': '28124'}