<jupyter_start><jupyter_text>Data Modeling  Build machine learning models to CLASSIFY offer sending as appropriate or not.  We have divided the data into three classes:  1) Offers that do not end into tranactions (label 0). 2) Offers that do not end into transaction (label 1).  3) Tranasactions without offer (label 2).<jupyter_code>import pandas as pd
import numpy as np
import math
import json
import seaborn as sns
import matplotlib.pyplot as plt
from datetime import datetime
from sklearn.preprocessing import MinMaxScaler
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import fbeta_score, accuracy_score
from sklearn.tree import DecisionTreeClassifier 
from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import quantile_transform, scale, robust_scale
from keras.models import Sequential
from keras.layers import Dense, Dropout, LSTM
from keras.wrappers.scikit_learn import KerasRegressor
from sklearn.model_selection import GridSearchCV
from keras.optimizers import Adam
import keras
from sklearn.svm import SVC
import timeit
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Load Data<jupyter_code>#load pickle
merged_df = pd.read_pickle('merged_data.pkl')
merged_df.head(10)<jupyter_output><empty_output><jupyter_text>Check Columns<jupyter_code>merged_df.columns<jupyter_output><empty_output><jupyter_text>Plot the correlation matrix<jupyter_code>merged_df.corr().abs().round(2)<jupyter_output><empty_output><jupyter_text>We can see the age and age_group features correlated with other features in a same way. We ignore age and consider only age_group.<jupyter_code>'''Features to feed the model'''
features = [
    'reward', 'difficulty', 'offer_duration_hrs',                      # offer characteristics
    'bogo', 'discount', 'informational',                               # offer type
    'mobile', 'social', 'web',                                         # channels
    'age_group','became_member_on', 'income',  'income_group',         # customer data
    'F', 'M','N', 'O'                                                  # customer gender
]<jupyter_output><empty_output><jupyter_text>Scale Features<jupyter_code>'''Scale features'''
scaler = MinMaxScaler()
numerical_cols = ['reward','difficulty','offer_duration_hrs','income','became_member_on']
merged_df[numerical_cols] = scaler.fit_transform(merged_df[numerical_cols])
merged_df.sample(10)<jupyter_output><empty_output><jupyter_text>Split train and test data sets<jupyter_code>data_features=merged_df[features]
data_label=merged_df['label']<jupyter_output><empty_output><jupyter_text>Splitting 70% of data sets into training and 30% into testing<jupyter_code>X_train, X_test, y_train, y_test = train_test_split(data_features, data_label, test_size = 0.3, random_state = 42)

print("Size of Training set: {}".format(X_train.shape[0]))
print("Size of Testing set: {}".format(X_test.shape[0]))<jupyter_output>Size of Training set: 53393
Size of Testing set: 22884<jupyter_text>We will try several models and consider model accuracy score as the metric to evaluate the model quality.  1. Neural Network This model will be the benchmark model to compare other classidication methods.<jupyter_code># Convert labels to "one-hot" vectors using the to_categorical function
y_train_hot = keras.utils.to_categorical(y_train, num_classes)
y_test_hot = keras.utils.to_categorical(y_test, num_classes)
print("First 5 training lables as one-hot encoded vectors:\n", y_train_hot[:5])
# Checking the dimensions of X_train and y_train
X_train.shape, y_train_hot.shape<jupyter_output><empty_output><jupyter_text>Train Neural Network<jupyter_code>#Train Neural Network
input_dim = 17   # 17 features
num_classes = 3  # 3 labels

'''create model'''
model = Sequential()
model.add(Dense(20, input_dim=input_dim, activation='relu'))
for i in range(2): #2: Hidden layers
    model.add(Dense(10, activation='relu'))
model.add(Dense(num_classes, activation='softmax'))  #3: Output dims
'''Compile model'''
adam_opt = Adam(lr=0.001)
model.compile(optimizer=adam_opt, loss='categorical_crossentropy', metrics=['acc'])
'''Train model'''
start = timeit.default_timer()
n_epochs= 100
n_batch= 100
history = model.fit(X_train, y_train_hot, batch_size=n_batch, epochs=n_epochs, verbose=False, validation_split=.1)
loss, accuracy  = model.evaluate(X_test, y_test_hot, verbose=False)
loss_train, accuracy_train  = model.evaluate(X_train, y_train_hot, verbose=False)

''' plotting the metrics'''
fig = plt.figure()
plt.subplot(2,1,1)
plt.plot(history.history['acc'])
plt.plot(history.history['val_acc'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='best')
#
plt.subplot(2,1,2)
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='best')
plt.tight_layout()
plt.savefig('loss.png')
plt.show()
#
#print(f'Test loss: {loss:.3}')
print(f'Train accuracy: {accuracy_train:.3}')
print(f'Test accuracy: {accuracy:.3}')
stop = timeit.default_timer()
print('Training Time (sec.): ', stop - start)


#Define function to predict accuracy from a model classifier
def predict_acc(model_clf):
    '''
    Calculate accuracy for the models
    Arguments:
        model classifier (model_clf)
    Returns:
        train and test accuracy
    '''
    #train and predic train and test set
    pred_train =  (model_clf.fit(X_train, y_train)).predict(X_train)
    pred_test = (model_clf.fit(X_train, y_train)).predict(X_test)
    #calculate accuracy for train and test set
    acc_train = accuracy_score(y_train, pred_train)
    acc_test = accuracy_score(y_test, pred_test)
    
    return acc_train, acc_test<jupyter_output><empty_output><jupyter_text>2. Decision Tree Classifier<jupyter_code>clf_dt = DecisionTreeClassifier()
acc_train_dt, acc_test_dt= predict_acc(clf_dt)
print(f'Train accuracy: {acc_train_dt:.3}')
print(f'Test accuracy: {acc_test_dt:.3}')<jupyter_output>Train accuracy: 0.946
Test accuracy: 0.577<jupyter_text>3. K-Nearest Neighbors<jupyter_code>clf_knn = KNeighborsClassifier(n_neighbors = 10)
acc_train_knn, acc_test_knn= predict_acc(clf_knn)
print(f'Train accuracy: {acc_train_knn:.3}')
print(f'Test accuracy: {acc_test_knn:.3}')<jupyter_output>Train accuracy: 0.688
Test accuracy: 0.615<jupyter_text>4. Random Forest<jupyter_code>clf_rf = RandomForestClassifier(random_state = 10)
acc_train_rf, acc_test_rf= predict_acc(clf_rf)
print(f'Train accuracy: {acc_train_rf:.3}')
print(f'Test accuracy: {acc_test_rf:.3}')<jupyter_output>Train accuracy: 0.946
Test accuracy: 0.598<jupyter_text>5. Support Vector Machine<jupyter_code>clf_svm = SVC(gamma = 'auto')
acc_train_svm, acc_test_svm= predict_acc(clf_svm)
print(f'Train accuracy: {acc_train_svm:.3}')
print(f'Test accuracy: {acc_test_svm:.3}')<jupyter_output>Train accuracy: 0.609
Test accuracy: 0.614<jupyter_text>Model Evaluation<jupyter_code>data = {'Model': ['Neural Network','Decision Tree', 'K-Nearest Neighbor' , 'Random Forest', 'Support Vector Machine'], 
          'Train accuracy ':[accuracy_train, acc_train_dt, acc_train_knn, acc_train_rf,acc_train_svm], 
          'Test accuracy': [accuracy , acc_test_dt, acc_test_knn,acc_test_rf, acc_test_svm] }
compare = pd.DataFrame(data)
compare<jupyter_output><empty_output><jupyter_text>Out of five models, I would consider Random Forest  and K-Nearest Neighbor for further refinement. Random Forest has above 94% of train accuracy and above 59% test accuracy. This means the model is behaving not  in a similar way for the training and testing sets. On the other hand, K-Nearest Neighbor has above 68% train accuracy and above 61% test accuracy, which are almost similar.  Refinement Let's use GridSearchCV to refine the hyperparameters for Random Forest and K-Nearest Neighbor<jupyter_code># Look at parameters used by our current forest
print('Parameters currently in use:\n')
print(clf_rf.get_params())
# Grid for hyper parameters
n_estimators = [100, 300, 500]
max_depth = [5, 8, 15, 25]
min_samples_split = [5, 10, 50]
min_samples_leaf = [1, 5, 10]

hyperF = dict(n_estimators = n_estimators, max_depth = max_depth,  
              min_samples_split = min_samples_split, 
             min_samples_leaf = min_samples_leaf)

gridF = GridSearchCV(clf_rf, hyperF, cv = 3, verbose = 1, 
                      n_jobs = -1)
clf_best_rf = gridF.fit(X_train, y_train)<jupyter_output>Fitting 3 folds for each of 108 candidates, totalling 324 fits<jupyter_text>Best parameters after GridSearchCV<jupyter_code>print('Best fit parameters:\n')
print(clf_best_rf.best_estimator_)<jupyter_output>Best fit parameters:

RandomForestClassifier(max_depth=8, min_samples_leaf=10, min_samples_split=50,
                       random_state=10)<jupyter_text>Predicting train and test accuracy for best parameters<jupyter_code>best_random= clf_best_rf.best_estimator_
pred_train_best= best_random.predict(X_train)
pred_test_best= best_random.predict(X_test)

acc_train_best_rf = accuracy_score(y_train, pred_train_best)
acc_test_best_rf = accuracy_score(y_test, pred_test_best)

print(f'Train accuracy: {acc_train_best_rf:.3}')
print(f'Test accuracy: {acc_test_best_rf:.3}')<jupyter_output>Train accuracy: 0.654
Test accuracy: 0.647<jupyter_text>Searching best parameters for K-Nearest Neighbor<jupyter_code>#K-Nearest Neighbor
# Look at parameters 
print('Parameters currently in use:\n')
print(clf_knn.get_params())
grid_params= {
    'n_neighbors':[1,3,5,11,19],
    'weights': ['uniform','distance'],
    'metric': ['minkowski','euclidean','manhattan']
    }
gs_cv = GridSearchCV(clf_knn, grid_params, verbose=1, cv=3)
clf_best_knn= gs_cv.fit(X_train, y_train)<jupyter_output>Fitting 3 folds for each of 30 candidates, totalling 90 fits<jupyter_text>Best parameters after GridSearchCV<jupyter_code>clf_best_knn.best_params_<jupyter_output><empty_output><jupyter_text>Predicting train and test accuracy for the best parameters<jupyter_code>best_knn= clf_best_knn
pred_train_best_knn = best_knn.predict(X_train)
pred_test_best_knn = best_knn.predict(X_test)

acc_train_best_knn = accuracy_score(y_train, pred_train_best_knn)
acc_test_best_knn = accuracy_score(y_test, pred_test_best_knn)

print(f'Train accuracy: {acc_train_best_knn:.3}')
print(f'Test accuracy: {acc_test_best_knn:.3}')<jupyter_output>Train accuracy: 0.73
Test accuracy: 0.608<jupyter_text>Conclusion:  After the hyperparameter tuning, Random Forest can be taken as the best classifier because it does well on both Train and Test data sets. The test accuracy (~65%) for the given data sets and proposed problem seems to be resonable.  Cheking the Random Forest model behavior to a customer<jupyter_code># Take any customer  from test set (unseen)
X_test[2:3]<jupyter_output><empty_output><jupyter_text>customer id of a test set<jupyter_code>cid=merged_df.iloc[35235]['customer_id']<jupyter_output><empty_output><jupyter_text>Using Random Forest to predict<jupyter_code>pred_cid = clf_best_rf.predict(X_test[2:3])
pred_cid<jupyter_output><empty_output><jupyter_text>The prediction is 1, which means the customer has chance of completing the offer to transaction. Let's check the details of the customer from original trnascript data.<jupyter_code>transcript_org = pd.read_json('data/transcript.json', orient='records', lines=True)
transcript_org[transcript_org['person']==cid]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10580500043256337, 'fraction_numerical': 0.018081148888312137, 'mean_word_length': 6.434083601286174, 'pattern_counts': {'":': 0, '<': 91, '<?xml version=': 0, '>': 91, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '25', 'dir': 'jupyter-structured-clean-dedup', 'id': '479964', 'license': 'MIT', 'n_tokens_mistral': 4166, 'n_tokens_neox': 3902, 'n_words': 933, 'path': 'Data_modeling.ipynb', 'repo_name': 'ap439111/Capstone-DataScience-NanoDegree', 'size': '11559'}