<jupyter_start><jupyter_text>Hotel Demand Project Team: 5<NAME>, <NAME>, <NAME>, <NAME> Problem StatementGiven that there is a large number of variables that can present themselves in booking a hotel, with the overall goal being to tradeoff comfort and cost. In order to do this, we’d propose using the above dataset to build a model which predicts the lowest possible cost of stay and the way in which to achieve that cost (based on how the stay is booked and characteristics of the stay such as  of nights and country).  Hotel Demand**Data Source:**https://www.kaggle.com/jessemostipak/hotel-booking-demand**Context:**Have you ever wondered when the best time of year to book a hotel room is? Or the optimal length of stay in order to get the best daily rate? What if you wanted to predict whether or not a hotel was likely to receive a disproportionately high number of special requests?**Dataset used for Analysis:** Dataset Definition **Columns in this Dataset**|Column Name|Description|Type||--|--|--||hotel|H1 = Resort Hotel or H2= City Hotel|Categorical||is_canceled|0 not 1 canceled|Boolean||lead_time|Number of days that elapsed between the entering date of the booking into the PMS and the arrival date|Integer||arrival_date_yer|Year of arrival date|string||arrival_date_month|Month of arrival date|string||arrival_date_week_number|Week numbber of yaer for arrival|Integer||arrival_date_day_of_month|Day of arrival date|Integer||stays_in_weekend_nights|number of weekend nights (saturday or sunday) the guest stays at the hotel|Integer||stays_in_week_nights|number of week nights (Monday to Friday) the guest stays at the hotel|Integer||adults|number of adults|Integer||childern|number of children|Integer||babdies|number of babies|Integer||meal|Type of meal booked. Categories are presented in standard hospitality meal packages: Undefined/SC – no meal package; BB – Bed & Breakfast; HB – Half board (breakfast and one other meal – usually dinner); FB – Full board (breakfast, lunch and dinner)|Categorical||country|Country of origin. Categories are represented in the ISO 3155–3:2013 format|string||market_segment|Market segment designation. In categories, the term “TA” means “Travel Agents” and “TO” means “Tour Operators”|Categorical||distribution_channel|Booking distribution channel. The term “TA” means “Travel Agents” and “TO” means “Tour Operators”|Categorical||is_repeated_guest|Value indicating if the booking name was from a repeated guest (1) or not (0)|Boolean||previous_cancellations|Number of previous bookings that were cancelled by the customer prior to the current booking|Integer||previous_bookings_not_canceled|Number of previous bookings not cancelled by the customer prior to the current booking|Integer||reserved_room_type|Code of room type reserved. Code is presented instead of designation for anonymity reasons.|Categorical||assigned_room_type|Code for the type of room assigned to the booking. Sometimes the assigned room type differs from the reserved room type due to hotel operation reasons (e.g. overbooking) or by customer request. Code is presented instead of designation for anonymity reasons.|Categorical||booking_changes|Number of changes/amendments made to the booking from the moment the booking was entered on the PMS until the moment of check-in or cancellation|Integer||deposit_type|Indication on if the customer made a deposit to guarantee the booking. This variable can assume three categories: No Deposit – no deposit was made; Non Refund – a deposit was made in the value of the total stay cost; Refundable – a deposit was made with a value under the total cost of stay.|Categorical||agentID| of the travel agency that made the booking|String||companyID| of the company/entity that made the booking or responsible for paying the booking. ID is presented instead of designation for anonymity reasons|String||days_in_waiting_list|Number of days the booking was in the waiting list before it was confirmed to the customer customer_type|Integer||Type of booking|, assuming one of four categories:Contract - when the booking has an allotment or other type of contract associated to it; Group – when the booking is associated to a group; Transient – when the booking is not part of a group or contract, and is not associated to other transient booking; Transient-party – when the booking is transient, but is associated to at least other transient booking|Categorical||adrAverage| Daily Rate as defined by dividing the sum of all lodging transactions by the total number of staying nights|Numerical||required_car_parking_spaces|Number of car parking spaces required by the customer|Integer||total_of_special_requests|Number of special requests made by the customer (e.g. twin bed or high floor)|Integer||reservation_status|Reservation last status, assuming one of three categories: Canceled – booking was canceled by the customer; Check-Out – customer has checked in but already departed; No-Show – customer did not check-in and did inform the hotel of the reason why|Categorical||reservation_status_date|Date at which the last status was set. This variable can be used in conjunction with the ReservationStatus to understand when was the booking canceled or when did the customer checked-out of the hotel|Categorical|<jupyter_code>!conda install scikit-learn
!conda install pandas
!conda install seaborn
!conda install statsmodels
from pyspark.sql.functions import *
from pyspark.sql.types import IntegerType
from pyspark.ml.feature import RFormula
from pyspark.ml.classification import LogisticRegression as pslr
from pyspark.ml import Pipeline
from pyspark.ml.evaluation import BinaryClassificationEvaluator
from pyspark.ml.tuning import ParamGridBuilder as pgb
from pyspark.ml.tuning import TrainValidationSplit as ptvs
from pyspark.ml.regression import DecisionTreeRegressor
from pyspark.ml.evaluation import RegressionEvaluator

import statsmodels.api as sm
import matplotlib.pyplot as plt
from matplotlib.pyplot import figure
plt.style.use('ggplot')
%matplotlib inline

import pandas as pd
import numpy as np
import seaborn as sns

from sklearn.preprocessing import LabelEncoder, StandardScaler 
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import accuracy_score, roc_auc_score, roc_curve, confusion_matrix, auc
from sklearn.metrics import r2_score
from sklearn.model_selection import cross_val_score
from sklearn.linear_model import Ridge
from sklearn.linear_model import Lasso
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestRegressor 
from sklearn.ensemble import AdaBoostRegressor
from sklearn import tree
from sklearn.preprocessing import LabelEncoder, StandardScaler 
from sklearn.neural_network import MLPClassifier
from sklearn.metrics import r2_score
from sklearn.metrics import precision_recall_fscore_support as score<jupyter_output><empty_output><jupyter_text>Config Data File Paths<jupyter_code>bucket = spark._jsc.hadoopConfiguration().get("fs.gs.system.bucket")
data = "gs://" + bucket + "/notebooks/jupyter/data/"
print(data)
hotel_data = spark.read.format("csv")\
         .option("header", "true")\
         .option("inferSchema", "true")\
         .option("nullValue", "NULL")\
         .load(data + "hotel_bookings.csv")

# Print the Schema of loaded hotel dataset
hotel_data.printSchema()<jupyter_output>root
 |-- hotel: string (nullable = true)
 |-- is_canceled: integer (nullable = true)
 |-- lead_time: integer (nullable = true)
 |-- arrival_date_year: integer (nullable = true)
 |-- arrival_date_month: string (nullable = true)
 |-- arrival_date_week_number: integer (nullable = true)
 |-- arrival_date_day_of_month: integer (nullable = true)
 |-- stays_in_weekend_nights: integer (nullable = true)
 |-- stays_in_week_nights: integer (nullable = true)
 |-- adults: integer (nullable = true)
 |-- children: string (nullable = true)
 |-- babies: integer (nullable = true)
 |-- meal: string (nullable = true)
 |-- country: string (nullable = true)
 |-- market_segment: string (nullable = true)
 |-- distribution_channel: string (nullable = true)
 |-- is_repeated_guest: integer (nullable = true)
 |-- previous_cancellations: integer (nullable = true)
 |-- previous_bookings_not_canceled: integer (nullable = true)
 |-- reserved_room_type: string (nullable = true)
 |-- assigned_room_type: string (nullab[...]<jupyter_text>the default type of column 'children' is string, we need to change it to integer<jupyter_code>hotel_data = hotel_data.withColumn("children", hotel_data["children"].cast(IntegerType()))
# save the original data and cash the dataframe
hotel_backup = hotel_data
hotel_data.cache()<jupyter_output><empty_output><jupyter_text>Number of rows<jupyter_code># Print number of records in the dataset. 
print("Number of rows in the dataset {}".format(hotel_data.count()))<jupyter_output>Number of rows in the dataset 119390<jupyter_text>Checking for missing dataCount the number of nulls in each column of the dataset<jupyter_code>[(c, hotel_data.where(col(c).isNull()).count()) for c in hotel_data.columns]<jupyter_output><empty_output><jupyter_text>There are NULL values in 'children', 'country', 'company', and 'agent'. Agent only has 3 non null values that have more than 5% of the total volume of records. 94% of the values in Company are Null.  Dropping rows with 0 or less than 0 average daily rateWithin the dataset there are 1960 rows where there is an average daily rate of 0.  Our reasonable assumption is that when you book for a hotel room you'll be paying for it.  As such, we'll remove these records from our dataset as part of our pre-processing<jupyter_code>hotel_data=hotel_data.where(col('adr')>0)
hotel_data.count()<jupyter_output><empty_output><jupyter_text>Data Pre-Processing and some Analysis The company column has a low coverage (112k missing values).  This will introduce a lot of noise into our modelling so we'll drop the column.Additionally, we'll correct the null values within the dataset, setting the null children records to be 0 and fill the rows with missing country values as No Country<jupyter_code>hotel_pd = hotel_data.toPandas()
#hotel_pd = pd.read_csv("hotel_bookings.csv", header='infer')
hotel_pd = hotel_pd.drop(['company'], axis = 1)
hotel_pd[['children']] = hotel_pd[['children']].fillna(value=0)
hotel_pd[['country']] = hotel_pd[['country']].fillna(value='No_Country')
hotel_pd[['agent']] = hotel_pd[['agent']].fillna(value='0')
len(hotel_pd.index)<jupyter_output><empty_output><jupyter_text>Our data isn't super friendly to doing any sort of machine based processing and modelling, so in order to support this, we'll need to do a few things- Convert all text based categorical values to numerical- Convert dates to be numerical rather than string<jupyter_code>label_encoder = LabelEncoder() 
  
# Encode labels in column. 
hotel_pd['customer_type']= label_encoder.fit_transform(hotel_pd['customer_type']) 
hotel_pd['assigned_room_type'] = label_encoder.fit_transform(hotel_pd['assigned_room_type'])
hotel_pd['reservation_status'] = label_encoder.fit_transform(hotel_pd['reservation_status'])
hotel_pd['meal'] = label_encoder.fit_transform(hotel_pd['meal'])
hotel_pd['country'] = label_encoder.fit_transform(hotel_pd['country'])
hotel_pd['distribution_channel'] = label_encoder.fit_transform(hotel_pd['distribution_channel'])
hotel_pd['market_segment'] = label_encoder.fit_transform(hotel_pd['market_segment'])
hotel_pd['reserved_room_type'] = label_encoder.fit_transform(hotel_pd['reserved_room_type'])
hotel_pd['reservation_status_date'] = label_encoder.fit_transform(hotel_pd['reservation_status_date'])

hotel_pd['hotel'] = hotel_pd['hotel'].map({'Resort Hotel':0, 'City Hotel':1})

hotel_pd['arrival_date_month'] = hotel_pd['arrival_date_month'].map({'January':1, 'February': 2, 'March':3, 'April':4, 'May':5, 'June':6, 'July':7,
                                                            'August':8, 'September':9, 'October':10, 'November':11, 'December':12})<jupyter_output><empty_output><jupyter_text>Next, we define some functions to flag if the booking is for more than one individual or a single guest<jupyter_code>def family(df):
    if ((df['adults'] > 0) & ((df['children'] > 0)|(df['babies'] > 0))):
        val = 1
    else:
        val = 0
    return val<jupyter_output><empty_output><jupyter_text>We'll next convert the deposit column into two labels- If it costs the customer nothing to cancel the reservation - Refundable or No Deposit given- If it costs the customer something to cancel aka non-refundable<jupyter_code>def deposit(df):
    if ((df['deposit_type'] == 'No Deposit') | (df['deposit_type'] == 'Refundable')):
        return 0
    else:
        return 1<jupyter_output><empty_output><jupyter_text>Looking closer at the agent column, we can see an overrepresentation of a few agents within the dataset.  There are 330 distinct agents along with a no agent (null) representation in the dataset.<jupyter_code>agent_hist = hotel_data.select(col("agent"))\
    .rdd.flatMap(
        lambda row: row
    ).histogram(50)

data = {
    'bins': agent_hist[0][:-1],
    'freq': agent_hist[1]
}

plt.bar(data['bins'], data['freq'], width = 2.5 )
plt.title('Histogram of Agents')
plt.ylabel("Count of Agents")
plt.xlabel("Agent")<jupyter_output><empty_output><jupyter_text>To ensure that agent doesn't have an adverse impact on our modelling we'll look to group the data. Three of the agents capture the largest share of the bookings along with no agent used, we'll account for that by grouping the data.  This will reduce fragmentation later down the line during our modelling<jupyter_code>def agent_clean(df):
    if ((df['agent']!=1)|(df['agent']!=9)|(df['agent']!=240)|(df['agent']!=0)):
        val = 4
    if (df['agent'] == 9):
        val = 1
    if (df['agent'] == 240):
        val = 2
    if (df['agent'] == 1):
        val = 3
    return val<jupyter_output><empty_output><jupyter_text>Apply the functions we defined above<jupyter_code>def transform_data(df):
    df["is_family"] = df.apply(family, axis = 1)
    df["total_customer"] = df["adults"] + df["children"] + df["babies"]
    df["deposit_given"] = df.apply(deposit, axis=1)
    df["total_nights"] = df["stays_in_weekend_nights"]+ df["stays_in_week_nights"]
    hotel_pd['agent'] = hotel_pd.apply(agent_clean, axis = 1)
    return df
hotel_pd = transform_data(hotel_pd)<jupyter_output><empty_output><jupyter_text>Quick look at our data post processing<jupyter_code>hotel_pd<jupyter_output><empty_output><jupyter_text>Cleanup the columns we've transformed to more modelling friendly labels prior to starting our modelling<jupyter_code>hotel_pd = hotel_pd.drop(columns = ['adults', 'babies', 'children', 'deposit_type', 'reservation_status_date'])<jupyter_output><empty_output><jupyter_text>Finally, cleanup any data that's left over which is null, 0 records but a good safety valve<jupyter_code>hotel_pd = hotel_pd.dropna(axis = 0)
len(hotel_pd.index)<jupyter_output><empty_output><jupyter_text>Now that we've cleanse our dataset we'll take a closer look at the correlation within the dataset<jupyter_code>corr=hotel_pd.corr()
plt.figure(figsize=(15,10))
sns.heatmap(corr)
plt.show()<jupyter_output><empty_output><jupyter_text>Modeling Predicting ADR Using a Linear Regression Our first model is linear and attempts to predict the average daily rate. We start by printing a correlation matrix to determine the significance of each variable Looking more closely at the correlation values for ADR to see where the predictive power of the model might come from.We can see that total customers, room type, and whether or not the booking is for a family have high correlation with a higher ADR<jupyter_code>np.abs(corr).adr.sort_values(ascending=False)<jupyter_output><empty_output><jupyter_text>choose some resonable variables as independent variables to do the linear regression, and we first create train and test set<jupyter_code>X = hotel_pd[['reserved_room_type', 
              'total_customer', 
              'is_family', 
              'assigned_room_type', 
              'arrival_date_year', 
              'market_segment',
              'total_of_special_requests',
              'deposit_given',
              'lead_time',
              'country',
              'is_repeated_guest']]
y = hotel_pd[['adr']]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=843)
#Now we can create the linear model
X_train = sm.add_constant(X_train)
ols = sm.OLS(y_train, X_train)
#Next we fit the model and get the summary
Fitted_ols = ols.fit()
Fitted_ols.summary()<jupyter_output><empty_output><jupyter_text>R squared of this model is very low, so it may not be appropriate to use the linear model fit the data, we use this model to predict on test set and compare the result<jupyter_code>X_test = sm.add_constant(X_test)
y_pred = Fitted_ols.predict(X_test)
print('mean squared error:', np.mean((y_test.values.flatten() - y_pred)**2))<jupyter_output>mean squared error: 1482.9577835344787<jupyter_text>Conclusion:from the MSE we can see the model is not so good  Decision Tree Regression To Predict ADR Setup the model<jupyter_code>supervised = RFormula(formula="adr ~ .")
fittedRF = supervised.fit(df)  # fit the transformer
preparedDF = fittedRF.transform(df)  # transform
featureCols = pd.DataFrame(preparedDF.schema["features"].metadata["ml_attr"]["attrs"]["numeric"]).sort_values("idx")
featureCols = featureCols.set_index('idx')
featureCols.head()<jupyter_output><empty_output><jupyter_text>Define Test/Train Split<jupyter_code>train, test = preparedDF.randomSplit([0.7, 0.3], seed = 843)
dt = DecisionTreeRegressor(featuresCol ='features', labelCol = 'label')
dt_model = dt.fit(train)<jupyter_output><empty_output><jupyter_text>Predict ADR<jupyter_code>#get features after applying trasnformation
dt_predictions = dt_model.transform(test)

# Select example rows to display.
dt_predictions.select("prediction", "label", "features").show(5)<jupyter_output>+------------------+-----+--------------------+
|        prediction|label|            features|
+------------------+-----+--------------------+
| 95.18944216529921|123.0|(35,[3,4,5,6,8,9,...|
|122.35113785361435|117.9|(35,[3,4,5,6,8,9,...|
|180.27408026755856|195.0|(35,[3,4,5,6,8,9,...|
|194.27730106100796|150.0|(35,[3,4,5,6,7,8,...|
| 95.18944216529921|134.0|(35,[3,4,5,6,8,9,...|
+------------------+-----+--------------------+
only showing top 5 rows<jupyter_text>ADR Cannot be reliably predicted leveraging the features we have in this dataset.  Althought this model performs significantly beter than our logistic regression model, it still lacks statistical signficance.<jupyter_code>from pyspark.ml.evaluation import RegressionEvaluator

# Select (prediction, true label) and compute rmse for test set
dt_evaluator = RegressionEvaluator(labelCol="label", predictionCol="prediction", metricName="rmse")
rmse = dt_evaluator.evaluate(dt_predictions)
print("Decision tree approach Root Mean Squared Error (RMSE) on test data = %g" % rmse)

dt_evaluator2 = RegressionEvaluator(labelCol="label", predictionCol="prediction", metricName="r2")
r2 = dt_evaluator2.evaluate(dt_predictions)
print("Decision tree approach R Square on test data = %g" % r2)<jupyter_output>Decision tree approach Root Mean Squared Error (RMSE) on test data = 32.8732
Decision tree approach R Square on test data = 0.511064<jupyter_text>Modelling to Predict Whether or Not a Reservation Could Be CancelledWe spent a fair bit of time trying to build and tune a model that could predict the ADR, however the results were all uninspiring.  The first model was one of our better results, as such we took a look at the dataset against and started to work towards predicting whether a reservation would be cancelled or not.  Modelling Is Cancelled Going Overboard<jupyter_code>labels = hotel_pd[["is_canceled"]]
ind = hotel_pd.drop(["is_canceled"], axis = 1)

ind_train, ind_test, labels_train, labels_test = train_test_split(ind, labels, test_size = 0.30, random_state = 737)<jupyter_output><empty_output><jupyter_text>Define a basic Decision Tree model<jupyter_code>canceled_dt = DecisionTreeClassifier(max_depth = 10)
canceled_dt_model = canceled_dt.fit(ind_train, labels_train) 
labels_pred = canceled_dt_model.predict(ind_test)
print('Decision Tree Model')

print('Accuracy Score: {}\n\nConfusion Matrix:\n {}\n\nAUC Score: {}'
      .format(accuracy_score(labels_test,labels_pred), confusion_matrix(labels_test,labels_pred), roc_auc_score(labels_test,labels_pred)))<jupyter_output>Decision Tree Model
Accuracy Score: 1.0

Confusion Matrix:
 [[22008     0]
 [    0 13221]]

AUC Score: 1.0<jupyter_text>Given a perfect performance it's pretty easy to deduce our model is either overfitted or something is wrong with the model.  Looking at the strong correlation between reservation status and cancellation would indicate that to be true<jupyter_code>corr["is_canceled"].sort_values()
hotel_pd2 = hotel_pd.drop(["reservation_status"], axis=1)
labels = hotel_pd2[["is_canceled"]]
ind = hotel_pd2.drop(["is_canceled"], axis = 1)

ind_train, ind_test, labels_train, labels_test = train_test_split(ind, labels, test_size = 0.30, random_state = 737)
canceled_dt = DecisionTreeClassifier(max_depth = 10)
canceled_dt_model = canceled_dt.fit(ind_train, labels_train) 
labels_pred = canceled_dt_model.predict(ind_test)<jupyter_output><empty_output><jupyter_text>This variation of the model performs reasonably well (and realistically given we've removed the hindsight of knowing the reservation was cancelled) with a performance of 83%<jupyter_code>print('Decision Tree Model')

print('Accuracy Score: {}\n\nConfusion Matrix:\n {}\n\nAUC Score: {}'
      .format(accuracy_score(labels_test,labels_pred), confusion_matrix(labels_test,labels_pred), roc_auc_score(labels_test,labels_pred)))<jupyter_output>Decision Tree Model
Accuracy Score: 0.8449572795140368

Confusion Matrix:
 [[20006  2002]
 [ 3460  9761]]

AUC Score: 0.8236641077028161<jupyter_text>k-Fold Cross Validation Definition - https://machinelearningmastery.com/k-fold-cross-validation/tl;dr we resample the dataset into k folds and recreate the same train/test fit (70/30 in our case) then re-train and re-test the model using this data.  We look for the results of the validation to be as close as possible to our model trained using the full dataset.  If the value is substantially lower, that implies our dataset used to build the model created an overfitted model<jupyter_code>scores = cross_val_score(canceled_dt_model, ind_train, labels_train, cv = 5)
print("cross validation scores: {}".format(scores))
print("mean cross validation score: {}".format(np.mean(scores)))
print("score without cv: {}".format(canceled_dt_model.score(ind_train, labels_train)))<jupyter_output>cross validation scores: [0.85438842 0.84823601 0.84768856 0.85006083 0.84835766]
mean cross validation score: 0.8497462969778686
score without cv: 0.853335117577645<jupyter_text>Based on the precision, recall, fscore, and support we can see that the predictions are performing much better on the reservations not canceled class rather than the canceled reservation class.  Based on this, we should see if we can bring some paridy to the model and get a stronger classification for predicting whether a reservation will be cancelled.These numbers overall support a pretty good modelPrecision and Recall Definitionshttps://en.wikipedia.org/wiki/Precision_and_recallGeneral Definitionshttps://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_fscore_support.html<jupyter_code>precision, recall, fscore, support = score(labels_test, labels_pred)

print('precision: {}'.format(precision))
print('recall: {}'.format(recall))
print('fscore: {}'.format(fscore))
print('support: {}'.format(support))<jupyter_output>precision: [0.85255263 0.82980532]
recall: [0.90903308 0.73829514]
fscore: [0.87988741 0.78138008]
support: [22008 13221]<jupyter_text>We can see that the whether a deposit is given is the leading indicator of whether or a reservation is canceled or not.  This lines up with our correlation data almost identically<jupyter_code>pd.DataFrame(data = canceled_dt_model.feature_importances_*100,
                   columns = ["Importances"],
                   index = ind_train.columns).sort_values("Importances", ascending = False)[:20].plot(kind = "barh", color = "r")<jupyter_output><empty_output><jupyter_text>Build out a grid search to play with the decision tree parametersNow that we've identified a reasonable feature set, lets see if we can improve the model performanceThe following parameters are available for tuning in the Decision tree https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html<jupyter_code>param_grid = {'criterion':['gini','entropy'],
             'max_depth':[1,2,3,5,6,7,8,9,10,11,12,13,14,15,21],
             'splitter':['best','random'],
             'min_samples_leaf':[1,2,3,5,8,13,21],
             'max_features': ['auto', 'sqrt', 'log2', None]}
canceled_dt = GridSearchCV(DecisionTreeClassifier(), param_grid)
canceled_dt.fit(ind_train, labels_train)
pred = canceled_dt.predict(ind_test)
labels_pred = canceled_dt.predict_proba(ind_test)[:, 1]
performance = roc_auc_score(labels_test, pred)
performance<jupyter_output><empty_output><jupyter_text>There are significant improvements in the overall performance of the model with the hyperparameter tuning.  The false positive classifications has dropped by 670 and we see an uptick in true negatives of ~700.  This model performs classifies significantly better than the original decision tree<jupyter_code>print('Decision Tree Model')

print('Accuracy Score: {}\n\nConfusion Matrix:\n {}\n\nAUC Score: {}'
      .format(accuracy_score(labels_test,pred), confusion_matrix(labels_test,pred), roc_auc_score(labels_test,pred)))<jupyter_output>Decision Tree Model
Accuracy Score: 0.8579863180902098

Confusion Matrix:
 [[19794  2214]
 [ 2789 10432]]

AUC Score: 0.8442239726016663<jupyter_text>We can see a small degradation for the not canceled class (first column) and a measurable improvement in the model performance for the cancelled class.<jupyter_code>precision, recall, fscore, support = score(labels_test, pred)

print('precision: {}'.format(precision))
print('recall: {}'.format(recall))
print('fscore: {}'.format(fscore))
print('support: {}'.format(support))<jupyter_output>precision: [0.87650002 0.82492488]
recall: [0.89940022 0.78904773]
fscore: [0.88780247 0.80658754]
support: [22008 13221]<jupyter_text>We validate the overall model using a K-folds cross validation.  We can see minimal degradation of the AUC of the model when applying the cross validation meaning the model is not that overfitted.<jupyter_code>scores = cross_val_score(canceled_dt_model, ind_train, labels_train, cv = 10)
print("mean cross validation score: {}".format(np.mean(scores)))
print("score without cv: {}".format(canceled_dt_model.score(ind_train, labels_train)))<jupyter_output>mean cross validation score: 0.8500504863960346
score without cv: 0.853335117577645<jupyter_text>Here we can see an interesting change in feature importances where deposit given goes down in weight<jupyter_code>pd.DataFrame(data = canceled_dt.best_estimator_.feature_importances_*100,
                   columns = ["Importances"],
                   index = ind_train.columns).sort_values("Importances", ascending = False)[:20].plot(kind = "barh", color = "r")<jupyter_output><empty_output><jupyter_text>We can visualize the decision tree model hereDon't run this code in GCP it'll crash your environment.  We've attached it as part of a separate submission to make things more managable<jupyter_code>fig, axes = plt.subplots(nrows = 1,ncols = 1,figsize = (10,10), dpi=800)
a = tree.plot_tree(canceled_dt.best_estimator_,
               feature_names = ind_train.columns,
               class_names = ['canceled','not canceled'],
               filled = True)<jupyter_output><empty_output><jupyter_text>Let's try an Unsupervised ModelFor this classification we'll leverage a Multi-Layer Preception Classifier (neural network classifier)<jupyter_code>scaler = StandardScaler()
scaler.fit(ind_train)
ind_train_scaled = scaler.transform(ind_train)
ind_test_scaled = scaler.transform(ind_test)
canceled_dt = MLPClassifier()
canceled_dt_model = canceled_dt.fit(ind_train_scaled, labels_train)
labels_pred = canceled_dt_model.predict(ind_test_scaled)<jupyter_output>C:\Dev\Python375\Lib\site-packages\sklearn\neural_network\_multilayer_perceptron.py:934: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().
  y = column_or_1d(y, warn=True)<jupyter_text>Results look promising, so we'll see if we can hone in on a better model, and this performs about as well as our original decision tree model, but slightly better as the accuracy is marginally higher and is reflected in the confusion matrix with a really low true negative rate.<jupyter_code>print('Accuracy Score: {}\n\nConfusion Matrix:\n {}\n\nAUC Score: {}'
      .format(accuracy_score(labels_test,labels_pred), confusion_matrix(labels_test,labels_pred), roc_auc_score(labels_test,labels_pred)))<jupyter_output>Accuracy Score: 0.864032473246473

Confusion Matrix:
 [[19837  2171]
 [ 2619 10602]]

AUC Score: 0.8516300558074185<jupyter_text>The precision and recall metrics for our second class (canceled reservations) is a bit lower than what we saw out of our decision tree, but overall this model is actually performing quite well<jupyter_code>precision, recall, fscore, support = score(labels_test, labels_pred)

print('precision: {}'.format(precision))
print('recall: {}'.format(recall))
print('fscore: {}'.format(fscore))
print('support: {}'.format(support))<jupyter_output>precision: [0.88337193 0.8300321 ]
recall: [0.90135405 0.80190606]
fscore: [0.8922724  0.81572671]
support: [22008 13221]<jupyter_text>Can we improve the model any further through tuning the hyper parameters?<jupyter_code>param_grid = {"alpha": [1, 0.1, 0.01, 0.001, 0.0001],
              "solver": ["lbfgs", "adam", "sgd"],
              "activation": ["logistic", "relu"],
              "learning_rate": ["constant", "invscaling","adaptive"]}
canceled_dt = MLPClassifier()
canceled_dt_model = GridSearchCV(canceled_dt, param_grid,
                                 cv = 10,
                                 n_jobs = -1,
                                 verbose = 2)
canceled_dt_model = canceled_dt.fit(ind_train_scaled, labels_train)
labels_pred = canceled_dt_model.predict(ind_test_scaled)<jupyter_output><empty_output><jupyter_text>The confusion matrix shows some improvements in the true negatives and false positive classifications<jupyter_code>print('Accuracy Score: {}\n\nConfusion Matrix:\n {}\n\nAUC Score: {}'
      .format(accuracy_score(labels_test,pred), confusion_matrix(labels_test,labels_pred), roc_auc_score(labels_test,labels_pred)))<jupyter_output>Accuracy Score: 0.8579863180902098

Confusion Matrix:
 [[19627  2381]
 [ 2429 10792]]

AUC Score: 0.8540446015999958<jupyter_text>We can see some imrpovements over the baseline model as all the metrics have ticked up for the cancelled reservation class.<jupyter_code>precision, recall, fscore, support = score(labels_test, labels_pred)

print('precision: {}'.format(precision))
print('recall: {}'.format(recall))
print('fscore: {}'.format(fscore))
print('support: {}'.format(support))<jupyter_output>precision: [0.88987124 0.8192515 ]
recall: [0.89181207 0.81627713]
fscore: [0.8908406  0.81776161]
support: [22008 13221]<jupyter_text>There is minimal degradation of the model performance when validating it using a crossfold validation.  Implying the model is not overfitted for our dataset.<jupyter_code>scores = cross_val_score(canceled_dt_model, ind_train_scaled, labels_train, cv = 5)
print("mean cross validation score: {}".format(np.mean(scores)))
print("score without cv: {}".format(canceled_dt_model.score(ind_train_scaled, labels_train)))<jupyter_output>C:\Dev\Python375\Lib\site-packages\sklearn\neural_network\_multilayer_perceptron.py:934: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().
  y = column_or_1d(y, warn=True)
C:\Dev\Python375\Lib\site-packages\sklearn\neural_network\_multilayer_perceptron.py:934: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().
  y = column_or_1d(y, warn=True)
C:\Dev\Python375\Lib\site-packages\sklearn\neural_network\_multilayer_perceptron.py:934: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().
  y = column_or_1d(y, warn=True)
C:\Dev\Python375\Lib\site-packages\sklearn\neural_network\_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached an[...]
{'fraction_non_alphanumeric': 0.10183207484732709, 'fraction_numerical': 0.030955747420354382, 'mean_word_length': 5.710133225676222, 'pattern_counts': {'":': 4, '<': 163, '<?xml version=': 0, '>': 167, 'https://': 5, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '44', 'dir': 'jupyter-structured-clean-dedup', 'id': '199209', 'license': 'MIT', 'n_tokens_mistral': 10910, 'n_tokens_neox': 10018, 'n_words': 3151, 'path': 'docs/Final-Projects/2020/Team-5-Hotel-Demand.ipynb', 'repo_name': 'ymei9/Big-Data-Analytics-for-Business', 'size': '33262'}