<jupyter_start><jupyter_text>Gender Prediction System with Hierarchical Ecommerce Data  Name: <NAME>: <EMAIL>  Libraries<jupyter_code>import math
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
from pandas import DataFrame
from imblearn.over_sampling import SMOTE
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn import tree
from sklearn import metrics
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
from sklearn.metrics import roc_auc_score
from sklearn.metrics import roc_curve
from sklearn.model_selection import RandomizedSearchCV
from sklearn.model_selection import GridSearchCV
import numpy as np
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.callbacks import CSVLogger, TensorBoard, ModelCheckpoint, EarlyStopping, ReduceLROnPlateau<jupyter_output><empty_output><jupyter_text>Load Data<jupyter_code>path='data/'
train_data = pd.read_csv(path + 'trainingData.csv', header=None,
                         parse_dates=[1, 2],
                         names=['Session_ID', 'StartTime', 'EndTime', 'Categories'])
train_label = pd.read_csv(path + 'trainingLabels.csv', header=None)
train_data['Gender'] = (train_label[0].values == 'female').astype(np.float64)
train_data<jupyter_output><empty_output><jupyter_text>Data Exploratory<jupyter_code>train_data.head()
train_data.columns<jupyter_output><empty_output><jupyter_text>Let's figure out how many categories for each level and their frequency!<jupyter_code>from collections import defaultdict
distri_cat = {'A':{},'B':{},'C':{},'D':{}}
for row in train_data['Categories']:
    splited = row.split(',')
    for click in splited[-1].split(';'):
        click_split = click.split('/')[:-1]
        distri_cat['A'][click_split[0]]= distri_cat['A'].get(click_split[0],0) + 1
        distri_cat['B'][click_split[1]]= distri_cat['B'].get(click_split[1],0) + 1
        distri_cat['C'][click_split[2]]= distri_cat['C'].get(click_split[2],0) + 1
        distri_cat['D'][click_split[3]]= distri_cat['D'].get(click_split[3],0) + 1
def show_distribution_cat(distri_cat, level_type, rotation = 30, size = (10,5), visual_type='bar'):
    sorted_freq = sorted(distri_cat[level_type].items(), key=lambda x: x[1])
    cat_list, freq_list=map(list,list(zip(*sorted_freq)))
    plt.figure(figsize = size)
    if visual_type =='bar':
        ax = sns.barplot(x=cat_list, y=freq_list)
        ax.set(title='Bar Frequency for category level '+level_type, xlabel='Category', ylabel='Frequency')
        ax.bar_label(ax.containers[0], label_type='edge')
        ax.set_xticklabels(ax.get_xticklabels(),rotation = rotation)
    else:
        ax = sns.histplot(data=freq_list, bins=50)
        ax.set(title='Histogram Frequency for category level '+level_type, xlabel='Category', ylabel='Frequency')
    plt.show()
show_distribution_cat(distri_cat, 'A')<jupyter_output><empty_output><jupyter_text>Majority of main category A are A00001, A00002, and A00003!Majority of main category products are A00001, A00002, and A00003!<jupyter_code>show_distribution_cat(distri_cat, 'B', 90, (25,5))<jupyter_output><empty_output><jupyter_text>For level 2 category B, there are around 20 For level 2 category B, there are around 20 popular categories.popular categories.<jupyter_code>show_distribution_cat(distri_cat, 'C', 90,(40,5))<jupyter_output><empty_output><jupyter_text>For level 3 category C, there are even more categories to be considered so let's use histogram distribution to check it out!<jupyter_code>show_distribution_cat(distri_cat, 'C', 90,(10,5), 'hist')<jupyter_output><empty_output><jupyter_text>So with this histogram distribution, there are a lot of categories level C got a very few sessions (1->10)!<jupyter_code>show_distribution_cat(distri_cat, 'D', 90, (10,5), 'hist')<jupyter_output><empty_output><jupyter_text>Similarly, for level D product category, most of them got one sessions only!<jupyter_code>print(len(distri_cat['A']))
print(len(distri_cat['B']))
print(len(distri_cat['C']))
print(len(distri_cat['D']))<jupyter_output>11
86
383
21880<jupyter_text>At different hierarcy levels, there are:- 11 Level A (Categories)- 86 Level B (Subcategories)- 383 Level C (Sub-subcategories)- 21880 Level D (Product ID)<jupyter_code>train_data['Gender'].value_counts()<jupyter_output><empty_output><jupyter_text>Also, we noticed there are 11703 females(1) and 3297 males(0) so this dataset is unbalanced! We will take care of this issue later!  Preprocessing  ID Column Session ID is needed to convert  into number to keep track of which rows are resampled from the same ID.<jupyter_code>train_data['Session_ID'] = train_data['Session_ID'].str[1:].astype(int)
train_data<jupyter_output><empty_output><jupyter_text>Time Columns  Let's do some feature engineering. With the start and end datetime will be converted to:- Day in month (31)- Month (12)- DayofWeek (7)- StartHour (24)- EndHour (24)- Duration (1)- NoOfProducts (1)- AverageTimePerProduct (1)<jupyter_code>train_data['DayOfWeek']=train_data["StartTime"].dt.dayofweek
train_data['Month']=train_data["StartTime"].dt.month
train_data['Day']=train_data["StartTime"].dt.day
train_data['StartHour'] = train_data["StartTime"].dt.hour
train_data['EndHour'] = train_data["EndTime"].dt.hour
train_data['Duration'] = (train_data["EndTime"] - train_data["StartTime"]).dt.total_seconds()
train_data['NoOfProducts'] = train_data['Categories'].str.count(";")+1
train_data['AverageTimePerProduct'] = train_data['Duration']/train_data['NoOfProducts']<jupyter_output><empty_output><jupyter_text>Now, these date and time features are broken down into these new meaningful features like below!<jupyter_code>train_data<jupyter_output><empty_output><jupyter_text>Here are unique values for each columns!<jupyter_code>for i in train_data.columns:
    print(i, '\t', train_data[i].unique())<jupyter_output>Session_ID 	 [10001 10002 10003 ... 24998 24999 25000]
StartTime 	 ['2014-11-14T00:02:14.000000000' '2014-12-12T14:12:05.000000000'
 '2014-11-14T00:02:41.000000000' ... '2014-12-22T22:58:58.000000000'
 '2014-12-22T23:03:45.000000000' '2014-12-22T22:34:33.000000000']
EndTime 	 ['2014-11-14T00:02:20.000000000' '2014-12-12T14:12:06.000000000'
 '2014-11-14T00:16:40.000000000' ... '2014-12-22T22:59:00.000000000'
 '2014-12-22T23:03:50.000000000' '2014-12-22T22:34:41.000000000']
Categories 	 ['A00001/B00001/C00001/D00001/' 'A00002/B00002/C00002/D24897/'
 'A00002/B00002/C00002/D00002/;A00002/B00002/C00003/D00003/;A00002/B00002/C00007/D00007/;A00002/B00002/C00002/D00009/;A00002/B00002/C00003/D00010/;A00002/B00002/C00007/D00011/;A00002/B00002/C00002/D00012/;A00002/B00002/C00007/D00014/;A00002/B00002/C00003/D00017/;A00002/B00002/C00011/D00021/;A00002/B00002/C00004/D00023/;A00002/B00002/C00012/D00025/'
 ... 'A00002/B00002/C00007/D36054/' 'A00002/B00002/C00007/D08424/'
 'A00001/B00001/C00012/D22495[...]<jupyter_text>Notice that the data includes sessions from November and December so we need to be careful how we should deal with these data so if the production data is other months, we can still deal with it! For these time-based features "DayOfWeek", "Month", "Day", "StartHour" and "EndHour", it is not ideal to keep them as numerical categories. Also, it’s not a great choice to one-hot encode these time and date features, as it may lead to a curse of dimensionality.We will encode the cyclic features (cyclical occurring features) using the basic formulation of trigonometry, by computing the sin and cosine of the features.For example: the feature "DayOfWeek" has 7 unique feature values. Taking the sin and cosine of the feature values will create 2 dimensionality columns instead of 7 new columns.<jupyter_code>def sin_cos_transformation(df, col_name, max_range):
    sin_values = ((df[col_name].astype('float')*2*math.pi)/max_range).apply(math.sin)
    cos_values = ((df[col_name].astype('float')*2*math.pi)/max_range).apply(math.cos)
    df[col_name+'_sin']=sin_values
    df[col_name+'_cos']=cos_values
    df.drop(columns=[col_name], inplace = True)
    return df
train_data = sin_cos_transformation(train_data, 'DayOfWeek', 6)
train_data = sin_cos_transformation(train_data, 'Day', 31)
train_data = sin_cos_transformation(train_data, 'Month', 12)
train_data = sin_cos_transformation(train_data, 'StartHour', 23)
train_data = sin_cos_transformation(train_data, 'EndHour', 23)
train_data<jupyter_output><empty_output><jupyter_text>We got no more use for 'StartTime' and 'EndTime'. Let's remove them!<jupyter_code>train_data.drop(columns=['StartTime','EndTime'], inplace = True)<jupyter_output><empty_output><jupyter_text>Categories Columns<jupyter_code>proportion_sessions= train_data['NoOfProducts'].value_counts()/train_data['NoOfProducts'].value_counts().sum()
def my_autopct(pct):
    return ('%.2f' % pct) if pct > 3 else ''
fig1, ax1 = plt.subplots()
ax1.set_title("Percentage of Product Numbers per Session")
ax1.pie(proportion_sessions, labels=proportion_sessions.index, autopct=my_autopct,
        shadow=True, startangle=90,normalize=False)
ax1.axis('equal')<jupyter_output><empty_output><jupyter_text>We can see that majority of rows contain only one browsing category session (52.89%). In each session, user may view multiple products, but many of them view one product only. For the session which contains more than one product viewed, I suggest we should break the session into multiple sessions which contain only one product. This can be considered as a resampling approach, in which we create new instances for the dataset by detaching the sessions which contain more than one products to multiple instances which contain only one product.<jupyter_code>train_data['Categories']=train_data['Categories'].str.split(";")
train_data= train_data.explode('Categories')
train_data<jupyter_output><empty_output><jupyter_text>Now, each browsing session is broken down into multiple ones which contain one category. We have from 15,000 rows to 33,455 rows now! Next, we will continue to break down the category path into "Main Category", "Sub Category", "Sub Sub Category", and "Product Category" columns!<jupyter_code>train_data['MainCat'] = train_data['Categories'].str.split("/").str[0]
train_data['SubCat'] = train_data['Categories'].str.split("/").str[1]
train_data['SubSubCat'] = train_data['Categories'].str.split("/").str[2]
train_data['ProductCat'] = train_data['Categories'].str.split("/").str[3]
train_data<jupyter_output><empty_output><jupyter_text>We got no more use for the orginial categories column so let's remove it!<jupyter_code>train_data.drop(columns=['Categories'], inplace = True)<jupyter_output><empty_output><jupyter_text>Next, we need to turn these strings of categories into one-hot-encoding for our ML model to learn!<jupyter_code>def encode_and_bind(original_df, feature_to_encode):
    dummies = pd.get_dummies(original_df[[feature_to_encode]].astype(str))
    res = pd.concat([original_df, dummies], axis=1)
    res = res.drop([feature_to_encode], axis=1)
    return(res)
features_to_encode = ['MainCat', 'SubCat', 'SubSubCat', 'ProductCat']
for feature in features_to_encode:
    train_data = encode_and_bind(train_data, feature)
train_data<jupyter_output><empty_output><jupyter_text>Our features is increased from 20 to 22,375 features!<jupyter_code>print ("We have", len(train_data.columns.values), "features in total in our dataset!")<jupyter_output>We have 22375 features in total in our dataset!<jupyter_text>Using SMOTE for the imbalanced dataset<jupyter_code>print("The number of samples in the current data:", len(train_data))
print("Number of Male labels in the current data:", len(train_data[train_data['Gender']==0]))
print("Number of Female labels in the current data:", len(train_data[train_data['Gender']==1]))
print("Proportion of Male labels in the current data:", len(train_data[train_data['Gender']==0])/len(train_data))
print("Proportion of Female labels in the current data:", len(train_data[train_data['Gender']==1])/len(train_data))<jupyter_output>The number of samples in the current data: 33455
Number of Male labels in the current data: 7071
Number of Female labels in the current data: 26384
Proportion of Male labels in the current data: 0.2113585413241668
Proportion of Female labels in the current data: 0.7886414586758332<jupyter_text>As you can see, our data is quite unbalanced with 21% male labels and 78% female labels which can cause issues for the model to train and for choosing the right evaluation metrics. That is why we need to resolve this! One approach to addressing imbalanced datasets is to oversample the minority class. The simplest approach involves duplicating examples in the minority class, although these examples don’t add any new information to the model. Instead, new examples can be synthesized from the existing examples. This is a type of data augmentation for the minority class and is referred to as the Synthetic Minority Oversampling Technique, or SMOTE for short.SMOTE works by selecting examples that are close in the feature space, drawing a line between the examples in the feature space and drawing a new sample at a point along that line.os = SMOTE(random_state=0)X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)columns = X_train.columnsos_data_X, os_data_y=os.fit_resample(X_train, y_train)<jupyter_code>y = train_data['Gender']
X = train_data.drop(columns=['Gender'])<jupyter_output><empty_output><jupyter_text>Remember we should over-sampled only on the training data, because by oversampling only on the training data, none of the information in the test data is being used to create synthetic observations, therefore, no information will bleed from test data into the model training.<jupyter_code>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)
os = SMOTE(random_state=0)
X_train_os, y_train_os=os.fit_resample(X_train, y_train)<jupyter_output><empty_output><jupyter_text>It's time to double check if SMOTE is working alright!<jupyter_code>print("The number of samples in the oversampled data:", len(X_train_os))
print("Number of Male labels in the oversampled data:", len(y_train_os[y_train_os==0]))
print("Number of Female labels in the oversampled data:", len(y_train_os[y_train_os==1]))
print("Proportion of Male labels in the oversampled data:", len(y_train_os[y_train_os==0])/len(X_train_os))
print("Proportion of Female labels in the oversampled data:", len(y_train_os[y_train_os==1])/len(X_train_os))<jupyter_output>The number of samples in the oversampled data: 36906
Number of Male labels in the oversampled data: 18453
Number of Female labels in the oversampled data: 18453
Proportion of Male labels in the oversampled data: 0.5
Proportion of Female labels in the oversampled data: 0.5<jupyter_text>Now we have a perfect balanced data!   Model<jupyter_code>X_test=X_test.set_index('Session_ID')
X_train_os=X_train_os.set_index('Session_ID')
X_train_os
y_train_os<jupyter_output><empty_output><jupyter_text>Metrics For the metrics, we will use a variety of them to provide as much as information to evaluate our models:- Balanced Accuracy Measure (BAC) = 0.5*(tp/(tp+fn) + tn/(tn+fp))- Normal Accuracy (True/Total) = (tp+tn)/(tp+fp+tn+fn)- Precision, Recall, F1-score- ROC Curve (a measure on the degree of separability of both classes by a model)<jupyter_code>def model_report(model, model_name):
    y_train_pred = model.predict(X_train_os)
    y_test_pred = model.predict(X_test)
    
    tn, fp, fn, tp = confusion_matrix(y_train_os, y_train_pred).ravel()
    score = 0.5*(tp/(tp+fn) + tn/(tn+fp))
    print("Balanced Accuracy Score (BAC) for train data: ", score)
    
    tn, fp, fn, tp = confusion_matrix(y_test, y_test_pred).ravel()
    score = 0.5*(tp/(tp+fn) + tn/(tn+fp))
    print("Balanced Accuracy Score (BAC) for test data: ", score)
    
    print(f'Accuracy of {model_name} classifier on train set: {model.score(X_train_os, y_train_os)}')
    print(f'Accuracy of {model_name} classifier on test set: {model.score(X_test, y_test)}')
    
    print("Confusion Matrix for test data:")
    print(confusion_matrix(y_test, y_test_pred))
    
    print("Precision, Recall, F1-score for test data:")
    print(classification_report(y_test, y_test_pred))
    
    print("ROC Curve for test data:")
    logit_roc_auc = roc_auc_score(y_test, model.predict(X_test))
    fpr, tpr, thresholds = roc_curve(y_test, model.predict_proba(X_test)[:,1])
    plt.figure()
    plt.plot(fpr, tpr, label=f'{model_name} (area = {logit_roc_auc})')
    plt.plot([0, 1], [0, 1],'r--')
    plt.xlim([0.0, 1.0])
    plt.ylim([0.0, 1.05])
    plt.xlabel('False Positive Rate')
    plt.ylabel('True Positive Rate')
    plt.title('Receiver Operating Characteristic (ROC)')
    plt.legend(loc="lower right")
    plt.savefig('Log_ROC')
    plt.show()<jupyter_output><empty_output><jupyter_text>Logistic Regression (Baseline Model)<jupyter_code>logistic_model = LogisticRegression()
logistic_model.fit(X_train_os, y_train_os)
model_report(logistic_model, "Logistic Regression")<jupyter_output>Balanced Accuracy Score (BAC) for train data:  0.5000270958651709
Balanced Accuracy Score (BAC) for test data:  0.4876786583906325
Accuracy of Logistic Regression classifier on train set: 0.500027095865171
Accuracy of Logistic Regression classifier on test set: 0.45352196871575173
Confusion Matrix for test data:
[[1151  955]
 [4530 3401]]
Precision, Recall, F1-score for test data:
              precision    recall  f1-score   support

         0.0       0.20      0.55      0.30      2106
         1.0       0.78      0.43      0.55      7931

    accuracy                           0.45     10037
   macro avg       0.49      0.49      0.42     10037
weighted avg       0.66      0.45      0.50     10037

ROC Curve for test data:<jupyter_text>Naive Baynes<jupyter_code>gnb = GaussianNB()
gnb.fit(X_train_os, y_train_os)
model_report(gnb, "Naive Bayes")<jupyter_output>Balanced Accuracy Score (BAC) for train data:  0.506475911775863
Balanced Accuracy Score (BAC) for test data:  0.5124547333285198
Accuracy of Naive Bayes classifier on train set: 0.506475911775863
Accuracy of Naive Bayes classifier on test set: 0.2501743548869184
Confusion Matrix for test data:
[[2031   75]
 [7451  480]]
Precision, Recall, F1-score for test data:
              precision    recall  f1-score   support

         0.0       0.21      0.96      0.35      2106
         1.0       0.86      0.06      0.11      7931

    accuracy                           0.25     10037
   macro avg       0.54      0.51      0.23     10037
weighted avg       0.73      0.25      0.16     10037

ROC Curve for test data:<jupyter_text>Decision Tree<jupyter_code>decision_tree = tree.DecisionTreeClassifier()
decision_tree.fit(X_train_os, y_train_os)
model_report(decision_tree, "Decision Tree")<jupyter_output>Balanced Accuracy Score (BAC) for train data:  0.9999729041348291
Balanced Accuracy Score (BAC) for test data:  0.8635050374532576
Accuracy of Decision Tree classifier on train set: 0.999972904134829
Accuracy of Decision Tree classifier on test set: 0.9113280860814984
Confusion Matrix for test data:
[[1645  461]
 [ 429 7502]]
Precision, Recall, F1-score for test data:
              precision    recall  f1-score   support

         0.0       0.79      0.78      0.79      2106
         1.0       0.94      0.95      0.94      7931

    accuracy                           0.91     10037
   macro avg       0.87      0.86      0.87     10037
weighted avg       0.91      0.91      0.91     10037

ROC Curve for test data:<jupyter_text>Random Forest<jupyter_code>random_forest = RandomForestClassifier()
random_forest.fit(X_train_os, y_train_os)
model_report(random_forest, "Random Forest")<jupyter_output>Balanced Accuracy Score (BAC) for train data:  0.9999729041348291
Balanced Accuracy Score (BAC) for test data:  0.8900771109508974
Accuracy of Random Forest classifier on train set: 0.999972904134829
Accuracy of Random Forest classifier on test set: 0.9362359270698416
Confusion Matrix for test data:
[[1707  399]
 [ 241 7690]]
Precision, Recall, F1-score for test data:
              precision    recall  f1-score   support

         0.0       0.88      0.81      0.84      2106
         1.0       0.95      0.97      0.96      7931

    accuracy                           0.94     10037
   macro avg       0.91      0.89      0.90     10037
weighted avg       0.94      0.94      0.94     10037

ROC Curve for test data:<jupyter_text>XGBoost<jupyter_code>from xgboost.sklearn import XGBClassifier
xgb = XGBClassifier(use_label_encoder=False)
xgb.fit(X_train_os, y_train_os)
model_report(xgb, "XGBoost")<jupyter_output>[22:15:25] WARNING: ../src/learner.cc:1115: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.
Balanced Accuracy Score (BAC) for train data:  0.9449412019725789
Balanced Accuracy Score (BAC) for test data:  0.8431369062437024
Accuracy of XGBoost classifier on train set: 0.944941201972579
Accuracy of XGBoost classifier on test set: 0.9108299292617316
Confusion Matrix for test data:
[[1530  576]
 [ 319 7612]]
Precision, Recall, F1-score for test data:
              precision    recall  f1-score   support

         0.0       0.83      0.73      0.77      2106
         1.0       0.93      0.96      0.94      7931

    accuracy                           0.91     10037
   macro avg       0.88      0.84      0.86     10037
weighted avg       0.91      0.91      0.91     10037

ROC Curve for test data:<jupyter_text>DeepLearning with Tensorflow Since we got a large number for features and enough amount of samples, we can try deep learning for the win! First, we need to convert dataframe into tensors (a special type of numpy for tensorflow)<jupyter_code>X_train_tensor_os = tf.convert_to_tensor(X_train_os)
y_train_tensor_os = tf.convert_to_tensor(y_train_os)
X_test_tensor = tf.convert_to_tensor(X_test)
y_test_tensor = tf.convert_to_tensor(y_test)<jupyter_output><empty_output><jupyter_text>Define our basic fully connected neural network:<jupyter_code>model = keras.Sequential()
model.add(layers.Dense(16, activation='relu', input_shape=(X_train_tensor_os.shape[1],)))
model.add(layers.Dense(16, activation='relu'))
model.add(layers.Dense(1, activation='sigmoid'))<jupyter_output><empty_output><jupyter_text>We need to define extra info for our model:- Optimizers: Adam (best default optimizer)- Loss function: Binary Crossentrophy (because of binary classification)- metrics: Accuracy<jupyter_code>model.compile(optimizer='adam',
              loss='binary_crossentropy',
              metrics=['accuracy'])
model.summary()<jupyter_output>Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense (Dense)               (None, 16)                357984    
                                                                 
 dense_1 (Dense)             (None, 16)                272       
                                                                 
 dense_2 (Dense)             (None, 1)                 17        
                                                                 
=================================================================
Total params: 358,273
Trainable params: 358,273
Non-trainable params: 0
_________________________________________________________________<jupyter_text>Our model is looking alright! We need to add some useful callbacks to improve the training process:- Training faster by stopping early before the point it becomes overfitting (EarlyStopping)- Keep a model weight backup with the lowest loss for the test set to restore back later (ModelCheckpoint)- Reduce the learning rate if there is no progress for a few epoches (ReduceLROnPlateau)<jupyter_code># our callbacks hell to optimize our learning
callbacks = [
    ModelCheckpoint(
        filepath='checkpoint_weights.hdf5',
        monitor='val_loss',
        save_best_only=True,
        save_weights_only=True,
        verbose=1),
    EarlyStopping(
        monitor='val_loss',
        min_delta=1e-8,
        patience=20,
        restore_best_weights=True,
        verbose=1),
    ReduceLROnPlateau(
        monitor='val_loss',
        min_delta=1e-8,
        factor=0.2,
        patience=10,
        verbose=1)
]
callbacks_list = callbacks<jupyter_output><empty_output><jupyter_text>It's time to fit the model!<jupyter_code>history = model.fit(X_train_tensor_os, y_train_tensor_os,
                    epochs=50,
                    batch_size=128,
                    validation_data=(X_test_tensor, y_test_tensor), 
                    callbacks = callbacks_list)<jupyter_output>Epoch 1/50
287/289 [============================>.] - ETA: 0s - loss: 3.1060 - accuracy: 0.6921
Epoch 00001: val_loss improved from inf to 6.26943, saving model to checkpoint_weights.hdf5
289/289 [==============================] - 2s 7ms/step - loss: 3.0964 - accuracy: 0.6924 - val_loss: 6.2694 - val_accuracy: 0.7400 - lr: 0.0010
Epoch 2/50
280/289 [============================>.] - ETA: 0s - loss: 1.1511 - accuracy: 0.8333
Epoch 00002: val_loss improved from 6.26943 to 1.56820, saving model to checkpoint_weights.hdf5
289/289 [==============================] - 2s 7ms/step - loss: 1.1393 - accuracy: 0.8340 - val_loss: 1.5682 - val_accuracy: 0.8512 - lr: 0.0010
Epoch 3/50
283/289 [============================>.] - ETA: 0s - loss: 0.5980 - accuracy: 0.8721
Epoch 00003: val_loss improved from 1.56820 to 0.38643, saving model to checkpoint_weights.hdf5
289/289 [==============================] - 2s 6ms/step - loss: 0.5944 - accuracy: 0.8724 - val_loss: 0.3864 - val_accuracy: 0.8722 - lr: 0.0[...]<jupyter_text>As you can see, the model stopped early due to overfitting and it keeps the weights which got the lowest loss! After training, we need to look at the graph of loss and accuracy to figure out if our model is overfitting at which epoch.<jupyter_code>history_dict = history.history
history_dict.keys()
def show_training_graph(history, metric):
    if metric=='loss':
        train_data = history.history['loss']
        test_data = history.history['val_loss']
    else:
        train_data = history.history['accuracy']
        test_data  = history.history['val_accuracy']
    
    epochs = range(len(history.history['accuracy']))

    plt.figure(figsize=(10,5))
    plt.plot(epochs, train_data, 'b', label='Training')
    plt.plot(epochs, test_data, 'r', label='Validation')
    plt.title('Training vs. Validation '+metric)
    plt.xlabel('Epochs')
    plt.ylabel(metric)
    plt.legend()
    plt.show()
show_training_graph(history,'loss')
show_training_graph(history,'accuracy')<jupyter_output><empty_output><jupyter_text>As you can see from the graph, the model starts to overfit after the epoch 3 with no improvement afterward for loss value of the test set!<jupyter_code>def deeplearing_model_report(model, model_name):
    y_train_pred = tf.greater(model.predict(X_train_tensor_os), .5)
    y_test_pred = tf.greater(model.predict(X_test_tensor), .5)
    
    tn, fp, fn, tp = confusion_matrix(y_train_tensor_os, y_train_pred).ravel()
    score = 0.5*(tp/(tp+fn) + tn/(tn+fp))
    print("Balanced Accuracy Score (BAC) for train data: ", score)
    
    tn, fp, fn, tp = confusion_matrix(y_test_tensor, y_test_pred).ravel()
    score = 0.5*(tp/(tp+fn) + tn/(tn+fp))
    print("Balanced Accuracy Score (BAC) for test data: ", score)
    
    print(f'Accuracy of {model_name} classifier on train set: {model.evaluate(X_train_tensor_os, y_train_tensor_os)[1]}')
    print(f'Accuracy of {model_name} classifier on test set: {model.evaluate(X_test_tensor, y_test_tensor)[1]}')
    
    print("Confusion Matrix for test data:")
    print(confusion_matrix(y_test_tensor, y_test_pred))
    
    print("Precision, Recall, F1-score for test data:")
    print(classification_report(y_test_tensor, y_test_pred))
    
    print("ROC Curve for test data:")
    logit_roc_auc = roc_auc_score(y_test_tensor, y_test_pred)
    fpr, tpr, thresholds = roc_curve(y_test, model.predict(X_test_tensor))
    plt.figure()
    plt.plot(fpr, tpr, label=f'{model_name} (area = {logit_roc_auc})')
    plt.plot([0, 1], [0, 1],'r--')
    plt.xlim([0.0, 1.0])
    plt.ylim([0.0, 1.05])
    plt.xlabel('False Positive Rate')
    plt.ylabel('True Positive Rate')
    plt.title('Receiver Operating Characteristic (ROC)')
    plt.legend(loc="lower right")
    plt.savefig('Log_ROC')
    plt.show()
deeplearing_model_report(model, 'DenseNet')<jupyter_output>Balanced Accuracy Score (BAC) for train data:  0.8721617081233404
Balanced Accuracy Score (BAC) for test data:  0.7820575684653355
1154/1154 [==============================] - 2s 1ms/step - loss: 0.3380 - accuracy: 0.8722
Accuracy of DenseNet classifier on train set: 0.8721616864204407
314/314 [==============================] - 0s 1ms/step - loss: 0.3864 - accuracy: 0.8722
Accuracy of DenseNet classifier on test set: 0.8721729516983032
Confusion Matrix for test data:
[[1320  786]
 [ 497 7434]]
Precision, Recall, F1-score for test data:
              precision    recall  f1-score   support

         0.0       0.73      0.63      0.67      2106
         1.0       0.90      0.94      0.92      7931

    accuracy                           0.87     10037
   macro avg       0.82      0.78      0.80     10037
weighted avg       0.87      0.87      0.87     10037

ROC Curve for test data:<jupyter_text>Choosing the best model to fine tune (The Winner is Random Forest) Here is the grand summary of the above results of each model for Balanced Accuracy Measure (BAC) of the test set:- Logistic Regression (Baseline Model): 0.487 - Naive Baynes: 0.512- Deep Learning (basic dense layers): 0.782- Decision Tree: 0.863- XGBoost: 0.843- Random Forest: 0.890As the results, Random Forest is the best model so far with the best BAC value among the models I have tried so far!Also, the ROC of random forest shows the nice balanced predictions between two classes and look very good compared to others.![image.png](attachment:47901603-948b-44b5-b9c5-928ad6a06adc.png)Therefore, we continue to use random forest to further fine tune this model for the best hyper-parameters  Fine Tuning Random Forest The main idea is to use random search to search for a wider range of hyper-parameters and use the result to narrow down the search range for grid search to improve those hyper-parameters. To look at the available hyperparameters, we can create a random forest and examine the default values.<jupyter_code>random_forest = RandomForestClassifier(random_state = 42)
from pprint import pprint
# Look at parameters used by our current forest
print('Parameters currently in use:\n')
pprint(random_forest.get_params())<jupyter_output>Parameters currently in use:

{'bootstrap': True,
 'ccp_alpha': 0.0,
 'class_weight': None,
 'criterion': 'gini',
 'max_depth': None,
 'max_features': 'auto',
 'max_leaf_nodes': None,
 'max_samples': None,
 'min_impurity_decrease': 0.0,
 'min_impurity_split': None,
 'min_samples_leaf': 1,
 'min_samples_split': 2,
 'min_weight_fraction_leaf': 0.0,
 'n_estimators': 100,
 'n_jobs': None,
 'oob_score': False,
 'random_state': 42,
 'verbose': 0,
 'warm_start': False}<jupyter_text>Random Search We will try a wide range of parameters of "n_estimators", "max_depth", "min_samples_split", "min_samples_leaf" for our random grid search! Due to the time constraint, we will try just a small range this time!- n_estimators = number of trees in the foreset- max_depth = max number of levels in each decision tree- min_samples_split = min number of data points placed in a node before the node is split- min_samples_leaf = min number of data points allowed in a leaf node<jupyter_code># Number of trees in random forest
n_estimators = [int(x) for x in np.linspace(start = 100, stop = 400, num = 20)]
# Maximum number of levels in tree
max_depth = [int(x) for x in np.linspace(10, 110, num = 11)]
max_depth.append(None)
# Minimum number of samples required to split a node
min_samples_split = [2, 5, 10]
# Minimum number of samples required at each leaf node
min_samples_leaf = [1, 2, 4]
# Create the random grid
random_grid = {'n_estimators': n_estimators,
               'max_depth': max_depth,
               'min_samples_split': min_samples_split,
               'min_samples_leaf': min_samples_leaf}
pprint(random_grid)<jupyter_output>{'max_depth': [10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 110, None],
 'min_samples_leaf': [1, 2, 4],
 'min_samples_split': [2, 5, 10],
 'n_estimators': [100,
                  115,
                  131,
                  147,
                  163,
                  178,
                  194,
                  210,
                  226,
                  242,
                  257,
                  273,
                  289,
                  305,
                  321,
                  336,
                  352,
                  368,
                  384,
                  400]}<jupyter_text>We keep our number of folds K and number of iterations to be small to be efficient to finish the search within the time limit!<jupyter_code>rf = RandomForestClassifier(random_state = 42)
# Random search of parameters, using 3 fold cross validation, 
# search across 100 different combinations, and use all available cores
rf_random = RandomizedSearchCV(estimator = rf, param_distributions = random_grid, n_iter = 8, cv = 3, verbose=2, random_state=42, n_jobs=4)
# Fit the random search model
rf_random.fit(X_train_os, y_train_os)<jupyter_output>Fitting 3 folds for each of 8 candidates, totalling 24 fits<jupyter_text>The best parameters for the random search are:<jupyter_code>best_random_para = rf_random.best_params_
best_random_para<jupyter_output><empty_output><jupyter_text>We should evaluate if our new hyper-parameters are better than the default random forest model!<jupyter_code>def evaluate(model, test_features, test_labels):

    y_test_pred = model.predict(test_features)
    tn, fp, fn, tp = confusion_matrix(test_labels, y_test_pred).ravel()
    score = 0.5*(tp/(tp+fn) + tn/(tn+fp))
    print("Balanced Accuracy Score (BAC) for test data: ", score)
    return score
base_model = RandomForestClassifier()
base_model.fit(X_train_os, y_train_os)
base_bac = evaluate(base_model, X_test, y_test)
best_random_model = rf_random.best_estimator_
best_random_model.fit(X_train_os, y_train_os)
random_bac = evaluate(best_random_model, X_test, y_test)

print('BAC Improvement of {:0.2f}%.'.format( 100 * (random_bac - base_bac) /base_bac))<jupyter_output>Balanced Accuracy Score (BAC) for test data:  0.8904320452963654
BAC Improvement of 0.32%.<jupyter_text>As you can see, our fine tune model got even a small improvement after random search! From these results, we should be able to narrow the range of values for each hyperparameter.  Grid Search As mentioned before, thanks to random search, we continue to narrow down the range of our best parameters!<jupyter_code># Create the parameter grid based on the results of random search 
param_grid = {
    'max_depth': [80, 100, 130],
    'min_samples_leaf': [1],
    'min_samples_split': [2],
    'n_estimators': [360, 400, 420]
}
# Create a based model
rf = RandomForestClassifier()
# Instantiate the grid search model
grid_search = GridSearchCV(estimator = rf, param_grid = param_grid, cv = 3, n_jobs = 3, verbose = 2)
grid_search.fit(X_train_os, y_train_os)
best_grid_para = grid_search.best_params_
best_grid_para
best_grid_model = grid_search.best_estimator_
grid_bac = evaluate(best_grid_model, X_test, y_test)
print('BAC Improvement of {:0.2f}%.'.format( 100 * (grid_bac - base_bac) /base_bac))<jupyter_output>BAC Improvement of 0.78%.
{'fraction_non_alphanumeric': 0.11141837891651227, 'fraction_numerical': 0.06722712314371837, 'mean_word_length': 4.261457878655609, 'pattern_counts': {'":': 0, '<': 169, '<?xml version=': 0, '>': 174, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 12, 'word_list_counts': {'cursed_substrings.json': 20, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '46', 'dir': 'jupyter-structured-clean-dedup', 'id': '262444', 'license': 'MIT', 'n_tokens_mistral': 13487, 'n_tokens_neox': 11719, 'n_words': 3308, 'path': 'ecommerce-customer-gender-prediction.ipynb', 'repo_name': 'TomHuynhSG/ecommerce-customer-gender-prediction', 'size': '36197'}