<jupyter_start><jupyter_text>To be able to process locally, we will use 1% of data. After that, we still have a large number of 241,179 records.<jupyter_code>df = pd.read_csv('train.csv.gz', sep=',').dropna()
df = df.sample(frac=0.01, random_state=99)
df.shape
df.head()
count_classes = pd.value_counts(df['is_booking'], sort = True).sort_index()
count_classes.plot(kind = 'bar')
plt.title("Booking or Not booking")
plt.xlabel("Class")
plt.ylabel("Frequency")<jupyter_output><empty_output><jupyter_text>It is obvious that our data is very imbalanced. We will have to deal with it. Data quality check<jupyter_code>df.isnull().sum()
df.dtypes<jupyter_output><empty_output><jupyter_text>Feature engineeringThe process includes create new columns such as year, month, plan time and hotel nights. And remove the columns we do not need anymore afterwards.<jupyter_code>df["date_time"] = pd.to_datetime(df["date_time"]) 
df["year"] = df["date_time"].dt.year  
df["month"] = df["date_time"].dt.month

df['srch_ci']=pd.to_datetime(df['srch_ci'],infer_datetime_format = True,errors='coerce')
df['srch_co']=pd.to_datetime(df['srch_co'],infer_datetime_format = True,errors='coerce')

df['plan_time'] = ((df['srch_ci']-df['date_time'])/np.timedelta64(1,'D')).astype(float)
df['hotel_nights']=((df['srch_co']-df['srch_ci'])/np.timedelta64(1,'D')).astype(float)

cols_to_drop = ['date_time', 'srch_ci', 'srch_co', 'user_id']
df.drop(cols_to_drop, axis=1, inplace=True)
df.head()<jupyter_output><empty_output><jupyter_text>Plot a correlation matrix using a heatmap to explore the correlation between features. Nothing really exciting here.<jupyter_code>correlation = df.corr()
plt.figure(figsize=(18, 18))
sns.heatmap(correlation, vmax=1, square=True,annot=True,cmap='viridis')

plt.title('Correlation between different fearures')<jupyter_output><empty_output><jupyter_text>Look at the correlation of each column compared to the other one. We do not see any two variables are very closely correlated.  Dealing with imbalanced data<jupyter_code>booking_indices = df[df.is_booking == 1].index
random_indices = np.random.choice(booking_indices, len(df.loc[df.is_booking == 1]), replace=False)
booking_sample = df.loc[random_indices]

not_booking = df[df.is_booking == 0].index
random_indices = np.random.choice(not_booking, sum(df['is_booking']), replace=False)
not_booking_sample = df.loc[random_indices]

df_new = pd.concat([not_booking_sample, booking_sample], axis=0)

print("Percentage of not booking clicks: ", len(df_new[df_new.is_booking == 0])/len(df_new))
print("Percentage of booking clicks: ", len(df_new[df_new.is_booking == 1])/len(df_new))
print("Total number of records in resampled data: ", len(df_new))<jupyter_output>Percentage of not booking clicks:  0.5
Percentage of booking clicks:  0.5
Total number of records in resampled data:  40064<jupyter_text>Shuffle the resampled dataframe.<jupyter_code>df_new = df_new.sample(frac=1).reset_index(drop=True)
df_new.shape<jupyter_output><empty_output><jupyter_text>Assign features and label from the new dataframe.<jupyter_code>X = df_new.loc[:, df_new.columns != 'is_booking']
y = df_new.loc[:, df_new.columns == 'is_booking']
X.head()<jupyter_output><empty_output><jupyter_text>PCA Principal component analysis, or PCA, is a statistical technique to convert high dimensional data to low dimensional data by selecting the most important features that capture maximum information about the dataset. Standardize the dataset<jupyter_code>scaler = StandardScaler()
X=scaler.fit_transform(X)
X<jupyter_output><empty_output><jupyter_text>Apply PCA. And we have 23 features in our data.<jupyter_code>pca = PCA(n_components=23)
pca.fit(X)<jupyter_output><empty_output><jupyter_text>Calculate Eigenvalues<jupyter_code>var=np.cumsum(np.round(pca.explained_variance_ratio_, decimals=3)*100)
var<jupyter_output><empty_output><jupyter_text>In the above array we see that the first feature explains 9.3% of the variance within our data set while the first two explain 17.3% and so on. If we employ all features we capture 99.9% of the variance within the dataset, thus we gain some by implementing an additional feature. No any single feature outstanding. Step 4: Sort & Select<jupyter_code>plt.ylabel('% Variance Explained')
plt.xlabel('# of Features')
plt.title('PCA Analysis')
plt.style.context('seaborn-whitegrid')

plt.plot(var)<jupyter_output><empty_output><jupyter_text>Based on the plot above it's clear we should keep all 23 features.  Train, predict and performance evaluation<jupyter_code>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=1)
X_train.shape
pca = PCA()  
X_train = pca.fit_transform(X_train)  
X_test = pca.transform(X_test)

classifier = RandomForestClassifier(max_depth=2, random_state=0)  
classifier.fit(X_train, y_train)

y_pred = classifier.predict(X_test)

cm = confusion_matrix(y_test, y_pred)  
print(cm)  
print('Accuracy', accuracy_score(y_test, y_pred))<jupyter_output>C:\Users\SusanLi\AppData\Local\Continuum\anaconda3\lib\site-packages\ipykernel_launcher.py:6: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().<jupyter_text>Logistic Regression<jupyter_code>from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=1)
pca = PCA(n_components=23)
logReg = LogisticRegression() 

pipe = Pipeline([('pca', pca), ('logistic', logReg)])
pipe.fit(X_train, y_train)
y_pred = pipe.predict(X_test)

cm = confusion_matrix(y_test, y_pred)  
print(cm)  
print('Accuracy', accuracy_score(y_test, y_pred))<jupyter_output>C:\Users\SusanLi\AppData\Local\Continuum\anaconda3\lib\site-packages\sklearn\utils\validation.py:578: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().
  y = column_or_1d(y, warn=True)
{'fraction_non_alphanumeric': 0.12358991373589914, 'fraction_numerical': 0.012939615129396152, 'mean_word_length': 7.361997226074896, 'pattern_counts': {'":': 0, '<': 50, '<?xml version=': 0, '>': 50, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '40478', 'license': 'MIT', 'n_tokens_mistral': 2185, 'n_tokens_neox': 2102, 'n_words': 491, 'path': 'Predict hotel booking.ipynb', 'repo_name': 'BeeChainFoundation/Brainstorming', 'size': '6028'}