<jupyter_start><jupyter_text>Comparing training and test scores and what they mean    Tr = Te (low) -- underfit    Tr = Te (high) -- good fit    Tr > Te -- overfit    Tr < Te -- outliers I will be fitting the data to several regression model types and in testing I will choose the best fit model. Regression models used:-  Muliple Linear Regression-  Polynomial Regression-  Ridge Regression<jupyter_code>import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn import linear_model
%matplotlib inline
Hotels = pd.read_csv('reg22.csv')

Hotels.rename({'Size': 'Size_m2'}, axis=1, inplace=True) 
Hotels.head() # important price transformed to >> Log_price<jupyter_output><empty_output><jupyter_text>Preprocessing apply transformations to linearize and standerdize clean data:1. Normalization And Standardization2. Feature Scaling 3. Guassian Transformations<jupyter_code>from sklearn.preprocessing import StandardScaler

#sns.pairplot(Hotels)

import matplotlib.pyplot as plt
import scipy.stats as stat
import pylab
#### If you want to check whether feature is guassian or normal distributed
#### Q-Q plot
def plot_data(df,feature):
    plt.figure(figsize=(10,6))
    plt.subplot(1,2,1)
    df[feature].hist()
    plt.subplot(1,2,2)
    stat.probplot(df[feature],dist='norm',plot=pylab)
    plt.show()
Hotels.columns<jupyter_output><empty_output><jupyter_text>transforming discrete independent variable 'beds':<jupyter_code>plot_data(Hotels, 'beds')
Hotels['beds_log'] = np.log(Hotels.beds)
plot_data(Hotels, 'beds_log')
Hotels['beds_Boxcox'],parameters=stat.boxcox(Hotels['beds']+1)
plot_data(Hotels,'beds_Boxcox')
Hotels['beds_boxcox^2'] = np.power(Hotels.beds_Boxcox,2)
plot_data(Hotels, 'beds_boxcox^2')<jupyter_output><empty_output><jupyter_text>transforming continuos independent variables 'rating' and 'number of ratings':<jupyter_code>plot_data(Hotels, 'rating')
Hotels['rating_Boxcox'],parameters=stat.boxcox(Hotels['rating']+1)
plot_data(Hotels,'rating_Boxcox')
plot_data(Hotels, 'number_of_ratings')
Hotels['number_of_ratings_Boxcox'],parameters=stat.boxcox(Hotels['number_of_ratings']+1)
plot_data(Hotels,'number_of_ratings_Boxcox')<jupyter_output><empty_output><jupyter_text>Transforming dependent variable size:<jupyter_code>plot_data(Hotels, 'Size_m2')
Hotels['Size_m2_log'] = np.log(Hotels['Size_m2'])
plot_data(Hotels,'Size_m2_log')
Hotels['Size_m2_Boxcox'],parameters=stat.boxcox(Hotels['Size_m2']+1)
plot_data(Hotels,'Size_m2_Boxcox')<jupyter_output><empty_output><jupyter_text>correlation heat maps for trasformed variables<jupyter_code>lin_hotels = Hotels[['Log_price', 'beds_log','rating_Boxcox', 'number_of_ratings_Boxcox', 'Size_m2']]

x1 =  Hotels[['Log_price','price', 'beds','rating', 'number_of_ratings', 'Size_m2']]# only int and float selectors
sns.pairplot(x1)
cor2 = x1.corr()
cor2
corrT = lin_hotels.corr()

from string import ascii_letters
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

sns.set_theme(style="white")


# Compute the correlation matrix


# Generate a mask for the upper triangle
mask = np.triu(np.ones_like(cor2, dtype=bool))

# Set up the matplotlib figure
f, ax = plt.subplots(figsize=(11, 9))

# Generate a custom diverging colormap
cmap = sns.diverging_palette(230, 20, as_cmap=True )

# Draw the heatmap with the mask and correct aspect ratio
sns.heatmap(cor2, mask=mask, cmap=cmap, vmax=.3, center=0,
            square=True, linewidths=.5, cbar_kws={"shrink": .5}, annot=True)
corrn = Hotels.corr()
corrn

corrT<jupyter_output><empty_output><jupyter_text>we can see the correlation is stronger for transfromed variables:for multiple linear regression model we<jupyter_code>from string import ascii_letters
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

sns.set_theme(style="white")


# Compute the correlation matrix


# Generate a mask for the upper triangle
mask = np.triu(np.ones_like(cor2, dtype=bool))

# Set up the matplotlib figure
f, ax = plt.subplots(figsize=(11, 9))

# Generate a custom diverging colormap
cmap = sns.diverging_palette(230, 20, as_cmap=True )

# Draw the heatmap with the mask and correct aspect ratio
sns.heatmap(cor2, mask=mask, cmap=cmap, vmax=.3, center=0,
            square=True, linewidths=.5, cbar_kws={"shrink": .5}, annot=True)<jupyter_output><empty_output><jupyter_text>Splitting data for testing, validating and traing20% test 25% val 55% train<jupyter_code>from sklearn.model_selection import train_test_split

X_real, X_test, y_real, y_test = train_test_split(Hotels[['beds','rating', 'number_of_ratings']], Hotels['price'], test_size=0.2, random_state=40)
X_train, X_val, y_train, y_val = train_test_split(X_real, y_real, test_size=0.25, random_state=40)
from sklearn.preprocessing import StandardScaler

scaler=StandardScaler()
X_train_scaled=scaler.fit_transform(X_train)
X_test_scaled=scaler.transform(X_test)

scaler = RobustScaler()

X_train_poly_scaled = scaler.fit_transform(X_train_poly)

X_val_poly_scaled = scaler.transform(X_val_poly)<jupyter_output><empty_output><jupyter_text>Models*   Multiple Linear Regression *   Polynomial Regression (with multiple scailing)*   Random Forest Regrssor*   Decision Tree Regressor *   Pipelining a combonation of models Regularization on top*   Ridge Regularization *   Lasso Regularization*   Elastic net Regularization Parameter Tuning *   Train test split*   Cross validation*   Kfold *   Grid search for Hyper Parameters*   Randomized search for Hyper Parameters then I check outlier influence and mulicollinearity using VIF  Muliple Linear Regression Model<jupyter_code>from sklearn.linear_model import LinearRegression<jupyter_output><empty_output><jupyter_text>Model 1 :with no transformations<jupyter_code>from sklearn.model_selection import train_test_split

X_real, X_test, y_real, y_test = train_test_split(Hotels[['beds','rating', 'number_of_ratings','Size_m2']], Hotels['price'], test_size=0.2, random_state=40)
X_train, X_val, y_train, y_val = train_test_split(X_real, y_real, test_size=0.25, random_state=40)
scaler = RobustScaler()

X_train_lm_scaled = scaler.fit_transform(X_train)

X_val_lm_scaled = scaler.transform(X_val)
model = LinearRegression()
model.fit(X_train_lm_scaled, y_train)
model.score(X_train_lm_scaled, y_train)
model.score(X_val_lm_scaled, y_val)<jupyter_output><empty_output><jupyter_text>Model 2 : the combonation of price, beds, rating box-cox, number of rating and size<jupyter_code>X_real, X_test, y_real, y_test = train_test_split(Hotels[['beds','rating_Boxcox', 'number_of_ratings','Size_m2']], Hotels['price'], test_size=0.2, random_state=40)
X_train, X_val, y_train, y_val = train_test_split(X_real, y_real, test_size=0.25, random_state=40)
scaler = RobustScaler()

X_train_lm_scaled = scaler.fit_transform(X_train)

X_val_lm_scaled = scaler.transform(X_val)
model = LinearRegression()
model.fit(X_train, y_train)
model.score(X_train, y_train)

model.score(X_val, y_val)<jupyter_output><empty_output><jupyter_text>Model 3: the combonation of log price, beds, rating box-cox, number of rating box-cox and size<jupyter_code>X_real, X_test, y_real, y_test = train_test_split(Hotels[['beds','rating_Boxcox', 'number_of_ratings_Boxcox','Size_m2']], Hotels['Log_price'], test_size=0.2, random_state=40)
X_train, X_val, y_train, y_val = train_test_split(X_real, y_real, test_size=0.25, random_state=40)
scaler = RobustScaler()

X_train_lm_scaled = scaler.fit_transform(X_train)

X_val_lm_scaled = scaler.transform(X_val)

model = LinearRegression()
model.fit(X_train_lm_scaled, y_train)
model.score(X_train_lm_scaled, y_train)
model.score(X_val_lm_scaled, y_val)<jupyter_output><empty_output><jupyter_text>to conclude the multiple linear regression model we found that the model with the best fit was the   Polynomial Regression  : Abdultawwab<jupyter_code>x1= Hotels.select_dtypes(include=['float64', 'int','int64'])
#x1.head()

X1=  x1[['beds','number_of_ratings','rating']] # with size it will be reduced

y= x1['price']
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import RobustScaler

X_train, X_test, y_train, y_test = train_test_split(X1,y,test_size = 0.2, random_state=40)
X_train, X_val, y_train, y_val = train_test_split(X1, y, test_size=.15, random_state=3)

poly = PolynomialFeatures(degree=3)
X_train_poly = poly.fit_transform(X_train)

# Apply polynomial transformation to val set
X_val_poly = poly.transform(X_val)
scaler = RobustScaler()

X_train_poly_scaled = scaler.fit_transform(X_train_poly)

X_val_poly_scaled = scaler.transform(X_val_poly)<jupyter_output><empty_output><jupyter_text>validation<jupyter_code>poly = PolynomialFeatures(degree=3)
X_train_poly = poly.fit_transform(X_train)

# Apply polynomial transformation to val set
X_val_poly = poly.transform(X_val)

# Fit a model using polynomial features
lr_poly = LinearRegression()
lr_poly.fit(X_train_poly,y_train)
lr_poly.score(X_train_poly,y_train)


lr_poly.score(X_val_poly, y_val)<jupyter_output><empty_output><jupyter_text>Random Forest Regressor<jupyter_code>X_train, X_test, y_train, y_test = train_test_split(X1,y,test_size = 0.2, random_state=40)
X_train, X_val, y_train, y_val = train_test_split(X1, y, test_size=.15, random_state=3)<jupyter_output><empty_output><jupyter_text>Model 1 : max depth = 2<jupyter_code>from sklearn.ensemble import RandomForestRegressor
from sklearn.datasets import make_regression
#X1, y = make_regression(n_features=4, n_informative=2,random_state=0, shuffle=False)
m1 = RandomForestRegressor(max_depth=2, random_state=0)
m1.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>Model 2 : number of estimators = 10<jupyter_code># Fitting Random Forest Regression to the dataset
m2 = RandomForestRegressor(n_estimators = 10, random_state = 0)
m2.fit(X_train, y_train)
m2.score(X_train, y_train)



m2.score(X_val, y_val)
m2.score(X_test, y_test)<jupyter_output><empty_output><jupyter_text>Model 3 : number of estimators = 1<jupyter_code>from sklearn.ensemble import RandomForestRegressor
#from sklearn.model_selection import train_test_splitx_train


m3 = RandomForestRegressor(n_estimators=1, random_state=10)
m3.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>validation<jupyter_code># Use the forest's predict method on the test data
predictions = m1.predict(X_val)
# Calculate the absolute errors
errors = abs(predictions -  y_val)
# Print out the mean absolute error (mae)
print('Mean Absolute Error:', round(np.mean(errors), 2), 'degrees.')

mape = 100 * (errors / y_val)
# Calculate and display accuracy
accuracy = 100 - np.mean(mape)
print('Accuracy:', round(accuracy, 2), '%.')

# Use the forest's predict method on the test data
predictions = m2.predict(X_val)
# Calculate the absolute errors
errors = abs(predictions -  y_val)
# Print out the mean absolute error (mae)
print('Mean Absolute Error:', round(np.mean(errors), 2), 'degrees.')

mape = 100 * (errors / y_val)
# Calculate and display accuracy
accuracy = 100 - np.mean(mape)
print('Accuracy:', round(accuracy, 2), '%.')

# Use the forest's predict method on the test data
predictions = m3.predict(X_val)
# Calculate the absolute errors
errors = abs(predictions -  y_val)
# Print out the mean absolute error (mae)
print('Mean Absolute Error:', round(np.mean(errors), 2), 'degrees.')

mape = 100 * (errors / y_test)
# Calculate and display accuracy
accuracy = 100 - np.mean(mape)
print('Accuracy:', round(accuracy, 2), '%.')

fig = plt.figure(figsize=(10,5))
plt.scatter(X_val, y_val, label="Truth")
plt.xlabel("X")
plt.ylabel("Y")
plt.legend(loc="upper left")<jupyter_output><empty_output><jupyter_text>Decision Tree Regressor<jupyter_code>from sklearn.tree import DecisionTreeRegressor
import math
from sklearn.ensemble import RandomForestRegressor

from sklearn.model_selection import GridSearchCV
from sklearn.model_selection import RandomizedSearchCV<jupyter_output><empty_output><jupyter_text>Model 1 : Decision Tree Regressor with no parameter tuning<jupyter_code>m1 = DecisionTreeRegressor()

m1.fit(X_train,y_train)
print(m1.score(X_val,y_val))<jupyter_output><empty_output><jupyter_text>Model 2 : Decision Tree Regressor with Grid search<jupyter_code>m2 = DecisionTreeRegressor()

m2.fit(X_train,y_train)
#print(dtr.score(X_test,y_test))
#GridSearchCV 
param_grid = {  'bootstrap': [True], 'max_depth': [5, 10, None], 
              'max_features': ['auto', 'log2'], 
              'n_estimators': [5, 6, 7, 8, 9, 10, 11, 12, 13, 15]}
rfr = RandomForestRegressor(random_state = 1)

g_search = GridSearchCV(estimator = rfr, param_grid = param_grid, 

                          cv = 3, n_jobs = 1, verbose = 0, return_train_score=True)

g_search.fit(X_train, y_train)
print(g_search.best_params_)
print(g_search.score(X_train, y_train))<jupyter_output><empty_output><jupyter_text>Model 3 : Decision Tree Regressor with Randomized Search<jupyter_code>from sklearn.model_selection import RandomizedSearchCV

n_estimators = [int(x) for x in np.linspace(start = 5 , stop = 15, num = 10)] # returns 10 numbers 

max_features = ['auto', 'log2']

max_depth = [int(x) for x in np.linspace(5, 10, num = 2)] 

max_depth.append(None)

bootstrap = [True, False]

r_grid = {'n_estimators': n_estimators,

               'max_features': max_features,

               'max_depth': max_depth,

               'bootstrap': bootstrap}

print(r_grid)
rfr_random = RandomizedSearchCV(estimator=rfr, param_distributions=r_grid, n_iter = 20,
                                 cv = 3, verbose=2, random_state=42, n_jobs=-1, return_train_score=True)


rfr_random.fit(X_train, y_train);
rfr_random.score(X_train, y_train)<jupyter_output><empty_output><jupyter_text>valdation<jupyter_code>rfr_random.score(X_val, y_val)
g_search.score(X_val, y_val)
m1.score(X_val, y_val)<jupyter_output><empty_output><jupyter_text>Gradient Booster<jupyter_code>from sklearn.ensemble import GradientBoostingRegressor
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
from sklearn.datasets import load_boston
from sklearn.metrics import mean_absolute_error
X_train, X_test, y_train, y_test = train_test_split(X1,y,test_size = 0.2, random_state=40)
X_train, X_val, y_train, y_val = train_test_split(X1, y, test_size=.15, random_state=3)
regressor = GradientBoostingRegressor(
    max_depth=2,
    n_estimators=3,
    learning_rate=1.0
)
regressor.fit(X_train, y_train)
regressor.score(X_train, y_train)
errors = [mean_squared_error(y_test, y_pred) for y_pred in regressor.staged_predict(X_test)]
best_n_estimators = np.argmin(errors)<jupyter_output><empty_output><jupyter_text>validation<jupyter_code>best_regressor = GradientBoostingRegressor(
    max_depth=2,
    n_estimators=best_n_estimators,
    learning_rate=1.0
)
best_regressor.fit(X_train, y_train)

best_regressor.score(X_val, y_val) 
y_pred = best_regressor.predict(X_test)
mean_absolute_error(y_test, y_pred)
(y_test-y_pred).mean()# no bais less better
best_regressor.score(X_test, y_test)<jupyter_output><empty_output><jupyter_text>Ridge Regularazation<jupyter_code>from sklearn.linear_model import LassoCV, RidgeCV,ElasticNetCV
#from sklearn.model_selection import KFold
from sklearn.linear_model import LinearRegression, Ridge
from sklearn.model_selection import GridSearchCV
from sklearn.linear_model import Lasso, Ridge, ElasticNet, LinearRegression
from sklearn.model_selection import (cross_val_score, train_test_split, KFold, GridSearchCV)# parameter tunning
kf = KFold(n_splits=5, shuffle=True, random_state=0)
from sklearn.metrics import mean_absolute_error
ridge = RidgeCV(GS.best_estimator_.alpha)
ridge.fit(X_train_poly_scaled,y_train)
ridge.score(X_train_poly_scaled,y_train)
poly = PolynomialFeatures(degree=1)
X_poly = poly.fit_transform(X1)

scaler = StandardScaler()
X_poly_scaled = scaler.fit_transform(X_poly)

ridge = RidgeCV(GS.best_estimator_.alpha)
ridge.fit(X_poly_scaled,y)

# Apply the same polynomial and scaling transformations to test set, then score
X_test_poly = poly.transform(X_test)
X_test_poly_scaled = scaler.transform(X_test_poly)

ridge.score(X_test_poly_scaled, y_test)
ridge = RidgeCV(GS.best_estimator_.alpha)
ridge.fit(X_train_poly_scaled,y_train)
ridge.score(X_train_poly_scaled,y_train)<jupyter_output><empty_output><jupyter_text>validation  Lasso Regularazation<jupyter_code>params = {"alpha": np.logspace(-4, -.1, 20)}
GS = GridSearchCV(Lasso(), param_grid=params, cv=kf, return_train_score=False)
GS.fit(X_train_poly_scaled, y_train);
lasso = LassoCV(GS.best_estimator_.alpha)
lasso.fit(X_train_poly_scaled,y_train)
lasso.score(X_train_poly_scaled,y_train)
poly = PolynomialFeatures(degree=3)
X_poly = poly.fit_transform(X1)

scaler = StandardScaler()
X_poly_scaled = scaler.fit_transform(X_poly)

lasso = LassoCV(GS.best_estimator_.alpha)
lasso.fit(X_poly_scaled,y)

# Apply the same polynomial and scaling transformations to test set, then score
X_test_poly = poly.transform(X_test)
X_test_poly_scaled = scaler.transform(X_test_poly)

lasso.score(X_test_poly_scaled, y_test)<jupyter_output><empty_output><jupyter_text>Elastic Regularazation<jupyter_code>from sklearn.linear_model import ElasticNetCV
elastic = ElasticNetCV(GS.best_estimator_.alpha)
elastic.fit(X_train_poly_scaled,y_train)
elastic.score(X_train_poly_scaled,y_train)
poly = PolynomialFeatures(degree=4)
X_poly = poly.fit_transform(X1)

scaler = StandardScaler()
X_poly_scaled = scaler.fit_transform(X_poly)


elastic.fit(X_poly_scaled,y)

# Apply the same polynomial and scaling transformations to test set, then score
X_test_poly = poly.transform(X_test)
X_test_poly_scaled = scaler.transform(X_test_poly)

elastic.score(X_test_poly_scaled, y_test)<jupyter_output><empty_output><jupyter_text>Pipelining with multipule regression types  Testing Random forest<jupyter_code>predictions = m2.predict(X_test)
# Calculate the absolute errors
errors = abs(predictions -  y_test)
# Print out the mean absolute error (mae)
print('Mean Absolute Error:', round(np.mean(errors), 2), 'degrees.')

mape = 100 * (errors / y_test)
# Calculate and display accuracy
accuracy = 100 - np.mean(mape)
print('Accuracy:', round(accuracy, 2), '%.')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10778280050106204, 'fraction_numerical': 0.015685420184085835, 'mean_word_length': 7.357760582612654, 'pattern_counts': {'":': 3, '<': 122, '<?xml version=': 0, '>': 124, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '30', 'dir': 'jupyter-structured-clean-dedup', 'id': '579818', 'license': 'MIT', 'n_tokens_mistral': 7011, 'n_tokens_neox': 6687, 'n_words': 1364, 'path': 'Models.ipynb', 'repo_name': 'entaim/Hotel-booking-prediction-Linear-Regression', 'size': '18361'}