<jupyter_start><jupyter_text>Prediction of marketing success <NAME>  Problem DefinitionThe starbucks data contains information about customers, about different types of offers made to these customers and about the purchase behaviour of the customers after they received these offers. From a marketing perspective it is interesting to know if is worth sending offers to certain customers.In order to solve this problem this project takes two steps:1. A classification model is trained to predict the outcome of an offer depending on features regarding user demographics as well as offer characteristics.2. This model is used to predict the oucome of each offer in the offer portfolio for any new or existing user. The result of the project can be seen as a marketing tool to decide what kind of offers to send to users.  Libraries & Data<jupyter_code># read libraries
# general
import pandas as pd
import numpy as np
import json
import math
import time
from datetime import date
import pickle

# plotting
import matplotlib.pyplot as plt
import seaborn as sb
%matplotlib inline

# sklearn
from sklearn.tree import DecisionTreeClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import LinearSVC
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.metrics import accuracy_score, f1_score
# read data
portfolio = pd.read_json('data/portfolio.json', orient='records', lines=True)
profile = pd.read_json('data/profile.json', orient='records', lines=True)
transcript = pd.read_json('data/transcript.json', orient='records', lines=True)<jupyter_output><empty_output><jupyter_text>Data exploration and cleaning  Portfolio dataframe<jupyter_code>portfolio<jupyter_output><empty_output><jupyter_text>Before further exploration of this dataframe, the `channels` column needs to be separated into 4 columns with 0 or 1 entries.<jupyter_code># create a dataframe with 0 and 1 entries for each channel type
channel_df = pd.DataFrame(index = portfolio.index, columns = ['email', 'mobile', 'social', 'web'])
for index, row in portfolio.iterrows():
    for i in range(len(row.channels)):
        channel_df[row.channels[i]].loc[index] = 1
channel_df.fillna(0, inplace = True)
# concatenate the channel_df with the portfolio dataframe
portfolio = pd.concat([portfolio, channel_df], axis = 1).drop(columns=['channels'])
# rename id to offer_id
portfolio.rename(columns = {'id': 'offer_id'}, inplace = True)
# check data types
portfolio.dtypes
# check shape
portfolio.shape
# check for NaN
portfolio.isna().sum()
portfolio.describe()
# plot offer type distribution
col = sb.color_palette()[0]
sb.countplot(data = portfolio, x  = 'offer_type', color = col)
plt.xlabel('')
plt.ylabel('Count')
plt.title('Distribution of offer types in portfolio');
# save portfolio data frame for further use
portfolio.to_csv('data/portfolio.csv', index = False)<jupyter_output><empty_output><jupyter_text>Profile dataframe<jupyter_code>profile.sample(10)
# standardise column name for offer id
profile.rename(columns = {'id': 'user_id'}, inplace = True)
# check data types
profile.dtypes
# change datatype of became_member_on column
profile['became_member_on'] = pd.to_datetime(profile.became_member_on, format = '%Y%m%d')
# check shape
profile.shape
# check for NaNs
profile.isna().sum()
# check occurence of NaN values
profile[profile.gender.isna()]
# since there is practically no information on these users, they will be dropped
profile.dropna(axis = 0, inplace = True)
profile.describe()
# plot age distribution
plt.hist(profile.age)
plt.xlabel('Age')
plt.ylabel('Count')
plt.title('Disbribution of age among users');
# plot income distribution
plt.hist(profile.income)
plt.xlabel('Income per year [US$]')
plt.ylabel('Count')
plt.title('Disbribution of income among users');
# plot gender distribtion
sb.countplot(data = profile, x  = 'gender', color = col)
plt.xlabel('Gender')
plt.ylabel('Count')
plt.title('Distribution of gender among users');<jupyter_output><empty_output><jupyter_text>Transcript dataframe<jupyter_code>transcript.sample(10)<jupyter_output><empty_output><jupyter_text>Before further exploration of this dataframe, the `value` column needs to be separated into 2 columns for amount and offer id.<jupyter_code># extend the value column several columns
extended_value = pd.json_normalize(transcript.value)
# write extended columns into transcript data frame  and drop the value column
transcript['offer_received'] = extended_value['offer id']
transcript['offer_viewed'] = extended_value['offer id']
transcript['offer_completed'] = extended_value['offer_id']
transcript['transaction'] = extended_value['amount']
transcript['reward'] = extended_value['reward']
transcript.drop(columns = ['value'], inplace = True)
# set the cells with no events to nan
transcript.loc[transcript.event == 'offer received', 'offer_viewed'] = np.nan
transcript.loc[transcript.event == 'offer viewed', 'offer_received'] = np.nan
# standardise column name for user id
transcript.rename(columns = {'person': 'user_id'}, inplace = True)
# check data types
transcript.dtypes
# check shape
transcript.shape
# check for NaNs
transcript.isna().sum()
transcript.describe()
# plot event type distribution
event_order = ['offer received', 'offer viewed', 'offer completed', 'transaction']
sb.countplot(data = transcript, x  = 'event', color = col, order = event_order)
plt.xlabel('Event type')
plt.ylabel('Count')
plt.title('No. of events per event type');
# plot events by time of occurrence
transcript['time_days'] = transcript.time/24
g = sb.FacetGrid(data = transcript, col = 'event', sharey = True, col_order = event_order)
g.map(plt.hist, 'time_days', bins = 30);
for i in range(len(event_order)):
    g.axes[0,i].set_xlabel('Days since start of test')
g.axes[0,0].set_ylabel('Count')
g.fig.subplots_adjust(top = 0.85)
g.fig.suptitle('No. of events on each day');<jupyter_output><empty_output><jupyter_text>Data preprocessingBefore starting with the prediction, the labels and features for the prediction have to be defined. The intention is to predict different offer outcomes (labels) with different characteristics of the offers and users (features).  Labels: Offer outcomesThe outcome of sending an offer to a user can be categorized depending on the combinations of receiving, viewing and completing offers:|offer received|offer viewed|offer completed|offer outcome||:---:|:---:|:---:|:---:||1|1|1|success||1|0|1|waste||1|1|0|failure||1|0|0|potential|- Success: The offer had its intended effect, because it was received, viewed and completed in this order.- Waste: The offer was unnecessary because the effect was reached even though the user did not view the offer or viewed it after completing the offer.- Failure: The offer was not completed event though it was used.- Potential: The offer was not completed but it was also not viewed. Since we do not know why is was not viewed, we might send it again in the future.This categorization does not hold for informational offers, however, because this offer type by definition is never completed.<jupyter_code># get dataframe with events connected to offers (not transactions) and non-informational offers only
offer_df = transcript[(transcript.event != 'transaction') & 
                     (transcript.offer_received != '3f207df678b143eea3cee63160fa8bed') & # offer id of informational offer
                     (transcript.offer_received != '5a8bc65990b245e5a138643cd4eb9837')] # offer id of informational offer<jupyter_output><empty_output><jupyter_text>In the above data cleaning process, some users were deleted due to missing data. These users have to be removed from the `offer_df` data frame.<jupyter_code># remove users that are not in profile data frame
offer_df = offer_df[offer_df.user_id.isin(profile.user_id)]<jupyter_output><empty_output><jupyter_text>Next, a dataframe with the offer outcomes has to be created. This is achieved by iterating through each user and each received offer. By comparing the indices (i.e. the temporal succession) of the events "offer received", "offer viewed" and "offer completed" the above categorization for the outcome of each offer is implemented.<jupyter_code># initiate data frame
total_offers = len(offer_df.offer_received.dropna()) # total number of non-informational offers sent to users
outcome_df = pd.DataFrame(np.nan, index = np.arange(total_offers), 
                          columns = ['user_id', 'offer_id', 'offer_outcome'])
print('Total iterations:', total_offers) # for monitoring progress

start = time.time()
i = 0
for u_id in profile.user_id: # iterate through users 
    user_df = offer_df[offer_df.user_id == u_id] # get data frame for only one user
    for index, row in user_df.iterrows(): # iterate through received offers of this user
        if type(row.offer_received) is str: # if an offer was received
            outcome_df.loc[i,'user_id'] = u_id # write user id into data frame
            outcome_df.loc[i, 'offer_id'] = row.offer_received # write offer id into data frame
            
            rec_index = index # index of receiving
            # get views occuring after current offer
            views = user_df.loc[index+1:, 'offer_viewed'].str.match(row.offer_received).dropna()
            views = views.index[views]
            if views.empty: # in case offer was not viewed
                view_index = np.nan
            else: 
                view_index = views[0]
             
            # get completions occuring after current offer
            comps = user_df.loc[index+1:, 'offer_completed'].str.match(row.offer_received).dropna() 
            comps = comps.index[comps]
            if comps.empty: # in case offer was not completed
                comp_index = np.nan 
            else: 
                comp_index = comps[0]
            
            if (rec_index < view_index) & (view_index < comp_index): # offer received, viewed, completed
                outcome_df.loc[i, 'offer_outcome'] = 'success'
            if (rec_index < view_index) & np.isnan(comp_index): # offer received, viewed, not completed
                outcome_df.loc[i, 'offer_outcome'] = 'failure'
            if (rec_index < view_index) & (view_index > comp_index): # offer received, completed, viewed
                outcome_df.loc[i, 'offer_outcome'] = 'waste' 
            if np.isnan(view_index) & (rec_index < comp_index): # offer received, not viewed, completed
                outcome_df.loc[i, 'offer_outcome'] = 'waste'
            if np.isnan(view_index) & np.isnan(comp_index): # offer received, not viewed, not completed 
                outcome_df.loc[i, 'offer_outcome'] = 'potential'
                
            if i in np.arange(0, total_offers, 5000):
                print('Current iteration:', i) # monitor progress
            i += 1
            
            del [views, comps, rec_index, view_index, comp_index] # free memory
    del user_df # free memory
end = time.time()
print('Elapsed time:', (end-start)/60, 'minutes')
# check for NaNs
outcome_df.isna().sum()
# get distribution of offer outcome
outcome_df.offer_outcome.value_counts()
# convert offer outcome to numeric
outcome_df.replace('success', 0, inplace = True)
outcome_df.replace('waste', 1, inplace = True)
outcome_df.replace('failure', 2, inplace = True)
outcome_df.replace('potential', 3, inplace = True)<jupyter_output><empty_output><jupyter_text>Features: Characteristics of offers and usersI chose to use the following characteristics as features for the prediction:- Offers:    - distribution channel    - difficulty    - reward- Users:    - gender    - age    - income    - membership duration    Almost all of these features are already present in the data, only the membership duration has to be calculated.<jupyter_code># create feature for membership duration
profile['membership_duration'] = ((pd.to_datetime(date.today()) - profile.became_member_on)/np.timedelta64(1, 'D')).astype(int)
# convert gender to numeric
profile.replace('F', 0, inplace = True)
profile.replace('M', 1, inplace = True)
profile.replace('O', 2, inplace = True)
# convert offer types to numeric
portfolio.replace('bogo', 0, inplace = True)
portfolio.replace('informational', 1, inplace = True)
portfolio.replace('discount', 2, inplace = True)
# remove informational offers
portfolio = portfolio[(portfolio.offer_id != '3f207df678b143eea3cee63160fa8bed') & # offer id of informational offer
                     (portfolio.offer_id != '5a8bc65990b245e5a138643cd4eb9837')] # offer id of informational offer
# save portfolio data frame for further use
portfolio.to_csv('data/portfolio_features.csv', index = False)<jupyter_output><empty_output><jupyter_text>Merge labels and features<jupyter_code># merge all labels and features to one dataframe
features_labels = outcome_df.merge(profile[['user_id','gender', 'age', 'income', 'membership_duration']], 
                 how = 'inner', on = 'user_id')
features_labels = features_labels.merge(portfolio[['offer_id', 'email', 'mobile', 'social', 'web', 'difficulty', 'reward']], 
                 how = 'inner', on = 'offer_id')
features_labels.sample(10)<jupyter_output><empty_output><jupyter_text>Prediction modelsIn this section the data will be split into training and test data and then several classification algorithms will be tested to see which one yields the best results. As metrics, the f1-score are used. The following algorithms are compared:- Decision tree- Gaussian naive bayes- K-neighbors - Logistic regression- Random forestThe support vector machine classification is not used because a first try with default parameters showed no extraordinary performance and it is very time consuming.<jupyter_code># split dataset into labels and features
X = features_labels.drop(['user_id', 'offer_id', 'offer_outcome'], axis = 1)
y = features_labels['offer_outcome']
# split dataset into train and test data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.3)
start = time.time()

# create classifier using decision tree classification
clf = DecisionTreeClassifier() 

# define parameters for gridsearch
parameters = {'criterion':('gini', 'entropy'), 'min_samples_split':(2, 4),
             'min_samples_leaf':(1, 5, 10), 'max_features':(None, 'sqrt')}

# build parameter grid and fit data
cv_decision_tree = GridSearchCV(clf, parameters)
cv_decision_tree.fit(X_train, y_train)

# predict
y_pred = cv_decision_tree.predict(X_test)

end = time.time()
print('Elapsed time:', (end-start)/60, 'minutes')
# evaluation
print('Chosen Parameters:', cv_decision_tree.best_params_)
print('Accuracy:', accuracy_score(y_test, y_pred))
print('f1 score:', f1_score(y_test, y_pred, average = 'weighted'))
start = time.time()

# create classifier using naive bayes classification
clf = GaussianNB() 

# fit data
clf.fit(X_train, y_train)

# predict
y_pred = clf.predict(X_test)

end = time.time()
print('Elapsed time:', (end-start)/60, 'minutes')
# evaluation
print('Accuracy:', accuracy_score(y_test, y_pred))
print('f1 score:', f1_score(y_test, y_pred, average = 'weighted'))
start = time.time()

# create classifier using k-neighbors classification
clf = KNeighborsClassifier() 

# define parameters for gridsearch
parameters = {'weights':('uniform', 'distance'), 'p':(1, 2), 'leaf_size':(15, 30, 45)}

# build parameter grid and fit data
cv_k_neighbors = GridSearchCV(clf, parameters)
cv_k_neighbors.fit(X_train, y_train)

# predict
y_pred = cv_k_neighbors.predict(X_test)

end = time.time()
print('Elapsed time:', (end-start)/60, 'minutes')
# evaluation
print('Chosen Parameters:', cv_k_neighbors.best_params_)
print('Accuracy:', accuracy_score(y_test, y_pred))
print('f1 score:', f1_score(y_test, y_pred, average = 'weighted'))
start = time.time()

# create classifier using logistic regression 
clf = LogisticRegression(multi_class = 'multinomial') 

# define parameters for gridsearch
parameters = {'C':(.1, 1, 10), 'solver':('newton-cg', 'lbfgs', 'sag')}

# build parameter grid and fit data
cv_logistic_regression = GridSearchCV(clf, parameters)
cv_logistic_regression.fit(X_train, y_train)

# predict
y_pred = cv_logistic_regression.predict(X_test)

end = time.time()
print('Elapsed time:', (end-start)/60, 'minutes')
# evaluation
print('Chosen Parameters:', cv_logistic_regression.best_params_)
print('Accuracy:', accuracy_score(y_test, y_pred))
print('f1 score:', f1_score(y_test, y_pred, average = 'weighted'))
start = time.time()

# create classifier using random forest classification
clf = RandomForestClassifier() 

# define parameters for gridsearch
parameters = {'criterion':('gini', 'entropy'), 'min_samples_split':(2, 4),
             'min_samples_leaf':(1, 5, 10), 'max_features':(None, 'sqrt')}

# build parameter grid and fit data
cv_random_forest = GridSearchCV(clf, parameters)
cv_random_forest.fit(X_train, y_train)

# predict
y_pred = cv_random_forest.predict(X_test)

end = time.time()
print('Elapsed time:', (end-start)/60, 'minutes')
# evaluation
print('Chosen Parameters:', cv_random_forest.best_params_)
print('Accuracy:', accuracy_score(y_test, y_pred))
print('f1 score:', f1_score(y_test, y_pred, average = 'weighted'))<jupyter_output>Chosen Parameters: {'criterion': 'gini', 'max_features': None, 'min_samples_leaf': 5, 'min_samples_split': 2}
Accuracy: 0.59526345467076
f1 score: 0.5850687623850503<jupyter_text>The following table summarises the results of the different algorithms:|Algorithm|Accuracy|f1-score||:---|:---:|:---:||Decision tree|0.55|0.54||Gaussian naive Bayes|0.51|0.47||K-neighbors|0.51|0.50||Logistic regression|0.54|0.52||Random forest|0.59|0.58|The best f1-score is reached by the random forest classifier. Therefore this model is saved for future use.<jupyter_code># save best models
pickle.dump(cv_random_forest, open('model/outcome_prediction.pkl', 'wb'))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10912731520106791, 'fraction_numerical': 0.014516936425830134, 'mean_word_length': 4.9873459873459876, 'pattern_counts': {'":': 0, '<': 58, '<?xml version=': 0, '>': 54, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '228985', 'license': 'MIT', 'n_tokens_mistral': 5693, 'n_tokens_neox': 5310, 'n_words': 1696, 'path': 'outcome_classification.ipynb', 'repo_name': 'zoraaroz/marketing-success-prediction', 'size': '17985'}