<jupyter_start><jupyter_text>MACHINE LEARNING CLASSIFICATION MODEL In this project we will develop a predictor of hotel bookings based on customer search details  Import the necessary libraries<jupyter_code>import pandas as pd
import numpy as np
import random
import seaborn as sb
import datetime as dt
from scipy.stats import norm
from matplotlib import pyplot as plt
from sklearn.decomposition import PCA
from sklearn.pipeline import Pipeline
from sklearn.tree import DecisionTreeClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, make_scorer
from sklearn.model_selection import train_test_split, GridSearchCV<jupyter_output><empty_output><jupyter_text>Download the files and get a sense of the data<jupyter_code>seed = 42 # set seed

n = sum(1 for line in open('train.csv')) - 1
s = 75000 # desired sample size

random.seed(seed)
skip = sorted(random.sample(range(1, n + 1), n - s)) # randomly sample dataset

# Read train.csv and drop all missing (NaN) values
hotelData = pd.read_csv('train.csv', skiprows=skip).dropna().reset_index(drop=True)
print('Dataset shape: {:,} columns and {:,} rows'.format(hotelData.shape[1], hotelData.shape[0]))
# This is the data we will train our model on
hotelData.head()<jupyter_output><empty_output><jupyter_text>Data Wrangling<jupyter_code># Here we are adding some additional data to the training set
destData = hotelData.merge(pd.read_csv('destinations.csv'), 
                           how='left', on='srch_destination_id').dropna().reset_index(drop=True)
# Here we move the target variable to the front in a couple of steps
tmp = destData['hotel_cluster']

destData = destData.drop(['hotel_cluster'], axis=1)
destData.insert(0, 'hotel_cluster', tmp)
print('Merged Destination Dataset shape: {:,} columns and {:,} rows'.format(destData.shape[1], destData.shape[0]))
destData.head()
# Here we arrange the data to have equal samples for each "hotel cluster" target variable
balData = destData.groupby('hotel_cluster')
balData = pd.DataFrame(balData.apply(lambda x:
                                     x.sample(balData.size().min()).reset_index(drop=True))).droplevel('hotel_cluster').reset_index(drop=True)
print('Merged Balanced Dataset shape: {:,} columns and {:,} rows'.format(balData.shape[1], balData.shape[0]))<jupyter_output>Merged Balanced Dataset shape: 173 columns and 6,500 rows<jupyter_text>Feature Analysis for SciKit Pipeline Preparation<jupyter_code># Here we convert the date and time variables to make them usable
for col in ['srch_ci', 'srch_co']:
    balData[col] = pd.to_datetime(balData[col], format = '%Y-%m-%d')
    balData[col] = balData[col].map(dt.datetime.toordinal)

balData['date_time'] = pd.to_datetime(balData['date_time'], format = '%Y-%m-%dT%H:%M:%S')
balData['date_time'] = balData['date_time'].map(dt.datetime.toordinal)
for col in ['date_time', 'srch_ci', 'srch_co']:
    print('Column: ' + col + ', Type: ' + str(type(balData[col][0])))
balData[['date_time', 'srch_ci', 'srch_co']]
balData.head()
# Here we reduce the features to the ones most important to the target variable
fig = plt.figure()
fig.set_size_inches(15, 12)
sb.set(font_scale = 1.25)

N_COMPONENTS = len(balData.columns[24:])

pca = PCA(n_components = N_COMPONENTS)

pc_matrix = pca.fit_transform(balData[balData.columns[24:]])

evr = pca.explained_variance_ratio_ * 100
cumsum_evr = np.cumsum(evr)

tickMarks = 10

ax = sb.lineplot(x=np.arange(1, len(cumsum_evr) + 1), y=cumsum_evr, label='Explained Variance Ratio')
ax.lines[0].set_linestyle('-.')
ax.set_title('Explained Variance Ratio Using {} Components'.format(N_COMPONENTS))
ax.plot(np.arange(1, len(cumsum_evr) + 1), cumsum_evr, 'bo')

for x, y in zip(range(1, len(cumsum_evr) + 1), cumsum_evr):
    if x in np.arange(1, 8, 1) or x % tickMarks == 0:
        plt.annotate("{:.2f}%".format(y), (x, y), xytext=(2, -15), 
                     textcoords="offset points", annotation_clip = False)

ax = sb.lineplot(x=np.arange(1, len(cumsum_evr) + 1), y=evr, label='Explained Variance Of Component X')
ax.plot(np.arange(1, len(evr) + 1), evr,'ro')
ax.lines[1].set_linestyle('-.')
ax.set_xticks([i for i in range(1, len(cumsum_evr) + 1) if i in np.arange(1, 2, 1) or i % tickMarks == 0])

for x, y in zip(range(1, len(cumsum_evr) + 1), evr):
    if x != 1 and (x in np.arange(1, 5, 1) or x % tickMarks == 0):
        plt.annotate("{:.2f}%".format(y), (x, y), xytext=(2, 5), 
                     textcoords="offset points", annotation_clip = False)

ax.set_xlabel('Component Number')
ax.set_ylabel('Explained Variance')<jupyter_output><empty_output><jupyter_text>From the graph above, we can see that of the 149 total components, using only the first 10 will account for almost 81% of the destination column data. As a result, i will be using the first 10 components to explain the destination column data.<jupyter_code># Here we get the training data ready for the SciKit Learn pipeline by dropping the target variable and doing feature reduction
balData = balData[balData.columns[:24]]

bal_PCA_Data = pd.concat([balData, 
                          pd.DataFrame(pc_matrix, columns=['PC-{}'.format(i) for i in range(1, N_COMPONENTS + 1)])], axis=1)

bal_PCA_Data = bal_PCA_Data[bal_PCA_Data.columns[:34]]
# Notice our columns (aka features) have reduced from 173 to 149 to 34
bal_PCA_Data.head()
bal_PCA_Data.describe()<jupyter_output><empty_output><jupyter_text>Check for correlation<jupyter_code>fig = plt.figure()
fig.set_size_inches(20, 15)
sb.set(font_scale = 0.8)

sb.heatmap(bal_PCA_Data.corr('pearson'), annot=True)<jupyter_output><empty_output><jupyter_text>Split the above data into a training and test set The models I have selected to experiment with in this analysis are the following: Random Forest Classifier and Decision Tree Classifier. The models performances (accuracy Score) with the training data will be compared at the end to see which model performed the best and then the best model will be used as the final model for predicting on the test set.<jupyter_code>x_train, x_test, y_train, y_test = train_test_split(bal_PCA_Data[bal_PCA_Data.columns[1:]],
                                                    bal_PCA_Data['hotel_cluster'],
                                                    train_size=0.65,
                                                    random_state=seed)
print('x_train shape: {:,} columns and {:,} rows'.format(x_train.shape[1], x_train.shape[0]))
x_train.head()
x_train.describe()
print('y_train shape: 1 column and {:,} rows'.format(y_train.shape[0]))
y_train.head()
y_train.describe()
# Here is the Random Forest model
rf_pipe = Pipeline(steps=([
    ('scale', StandardScaler()),
    ('rf', RandomForestClassifier(random_state=seed))
]))
param_grid = {'rf__max_depth': [2, 4, 6],
              'rf__class_weight': ['balanced', 'balanced_subsample'],
              'rf__criterion': ['gini', 'entropy'],
              'rf__max_features': ['auto', 'sqrt', 'log2'],
              'rf__min_samples_leaf': [2, 3],
              'rf__min_samples_split': [2, 3],
              'rf__n_estimators': [100, 200]}

rf_grid = GridSearchCV(rf_pipe, scoring=make_scorer(accuracy_score), 
                       param_grid = param_grid, cv = 5, n_jobs = -1, verbose=2)
rf_grid.fit(x_train, y_train)
rf_df = pd.DataFrame(rf_grid.cv_results_).sort_values('mean_test_score', 
                                                      ascending=False)[['params', 'mean_test_score']].head(10)
rf_df
print('Best Random Forest Classification Parameters\n' + '='*44)

for name, val in rf_df.iloc[0]['params'].items():
    print('{:>24}: {}'.format(name.replace('rf__', ''), val))
    
rf_acc = rf_df.iloc[0]['mean_test_score']
print('\nAccuracy Score: {}'.format(round(rf_acc, 4)))
# Here is the Decision Tree model
dt_pipe = Pipeline(steps=([
    ('scale', StandardScaler()),
    ('dt', DecisionTreeClassifier(random_state=seed))
]))
param_grid = {'dt__criterion': ['gini', 'entropy'],
              'dt__class_weight': ['balanced', None],
              'dt__splitter': ['best', 'random'],
              'dt__max_features': ['auto', 'sqrt', 'log2'],
              'dt__max_depth': [2, 4, 6],
              'dt__min_samples_leaf': [1, 2, 4],
              'dt__min_samples_split': [2, 4, 6]}

dt_grid = GridSearchCV(dt_pipe, scoring=make_scorer(accuracy_score), 
                       param_grid = param_grid, cv = 5, n_jobs = -1, verbose=2)
dt_grid.fit(x_train, y_train)
dt_df = pd.DataFrame(dt_grid.cv_results_).sort_values('mean_test_score', 
                                                      ascending=False)[['params', 'mean_test_score']].head(10)
dt_df
print('Best Decision Tree Classification Parameters\n' + '='*44)

for name, val in dt_df.iloc[0]['params'].items():
    print('{:>23}: {}'.format(name.replace('dt__', ''), val))
    
dt_acc = dt_df.iloc[0]['mean_test_score']
print('\nAccuracy Score: {}'.format(round(dt_acc, 4)))
acc_scores = [rf_acc, dt_acc]

modelTypes = ['Random Forest Classifier', 'Decision Tree Classifier']

acc_df = pd.DataFrame(zip(modelTypes, acc_scores), columns=['Model Type', 'Accuracy Score'])
acc_df = acc_df.nlargest(len(acc_df), 'Accuracy Score').reset_index(drop=True)

acc_df<jupyter_output><empty_output><jupyter_text>From the above table we can see both models did not have very good accuracy but the Random Forest Classifier did perform the best so that will be the model that i will use to make predictions on the test set for final analysis and results.  Construct Final Model - Random Forest Classifier<jupyter_code># Here are the parameters we will use to make predictions
print('Best Random Forest Classifier Parameters\n' + '='*40)

params = {}

for name, val in rf_df.iloc[0]['params'].items():
    name = name.replace('rf__', '')
    
    params.update({name: val})
    print('{:>21}: {}'.format(name, val))
    
rf_acc = rf_df.iloc[0]['mean_test_score']
print('\nAccuracy Score: {}'.format(round(rf_acc, 4)))
# Here is the pipeline through which we will run the test set
best_pipe = Pipeline(steps=([
    ('scale', StandardScaler()),
    ('rf', RandomForestClassifier(**params, random_state=seed))
]))
best_model = best_pipe.fit(x_train, y_train)

best_model
y_pred = best_model.predict(x_test)
best_model_score = accuracy_score(y_test, y_pred)

print("Best Random Forest Classifier score using the test data\n" + '='*50 +
      "\nTest Accuracy Score: {}\n\nTrain Accuracy Score: {}".format(round(best_model_score, 4), round(rf_acc, 4)))

print('\nDifference between train and best model test accuracy scores: {}'
      .format(abs(round(best_model_score - rf_acc, 4))))<jupyter_output>Best Random Forest Classifier score using the test data
==================================================
Test Accuracy Score: 0.1033

Train Accuracy Score: 0.1174

Difference between train and best model test accuracy scores: 0.0141<jupyter_text>Results from the test set and training set are close enough to employ the model on future data<jupyter_code>orginData_PCA_Pred = best_model.predict(bal_PCA_Data[bal_PCA_Data.columns[1:]])

print("Best Random Forest Classifier score using the Whole Sampled Dataset\n" + '='*67 +
      "\nAccuracy Score: {}".format(round(accuracy_score(bal_PCA_Data['hotel_cluster'], orginData_PCA_Pred), 4)))<jupyter_output>Best Random Forest Classifier score using the Whole Sampled Dataset
===================================================================
Accuracy Score: 0.2583
{'fraction_non_alphanumeric': 0.14156364269545535, 'fraction_numerical': 0.017499564687445585, 'mean_word_length': 4.48567335243553, 'pattern_counts': {'":': 0, '<': 34, '<?xml version=': 0, '>': 37, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '612955', 'license': 'MIT', 'n_tokens_mistral': 3963, 'n_tokens_neox': 3744, 'n_words': 1006, 'path': 'Machine Learning Project.ipynb', 'repo_name': 'duskybadger/Machine-Learning-Project', 'size': '11486'}