<jupyter_start><jupyter_text>AbstractIn this notebook, I will focus on an approach that suggested by <NAME>. He was suggesting using the dummy variable to decide whether a customer will be given a promo or not. By comparing the model results and substract them and then if the result is above certain threshold it will be flagged as prospectus customer, otherwise if customer below certain threshold it won't be targetted to customer promos. The Victor Lo presentation: [here](https://www.slideshare.net/odsc/victor-lomachinelearningpresentation)  Portfolio Exercise: Starbucks Background InformationThe dataset that will be provided in this portfolio exercise was originally used as a take-home assignment provided by Starbucks for their job candidates. The data for this exercise consists of about 120,000 data points split in a 2:1 ratio among training and test files. In the experiment simulated by the data, an advertising promotion was tested to see if it would bring more customers to purchase a specific product priced at **10 bucks** . Since it costs the company **0.15 bucks** to send out each promotion, it would be best to limit that promotion only to those that are most receptive to the promotion. Each data point includes one column indicating whether or not an individual was sent a promotion for the product, and one column indicating whether or not that individual eventually purchased that product. Each individual also has seven additional features associated with them, which are provided abstractly as V1-V7. Optimization StrategyThe task is to use the training data to understand what patterns in V1-V7 to indicate that a promotion should be provided to a user. Specifically, your goal is to maximize the following metrics:* **Incremental Response Rate (IRR)** IRR depicts how many more customers purchased the product with the promotion, as compared to if they didn't receive the promotion. Mathematically, it's the ratio of the number of purchasers in the promotion group to the total number of customers in the purchasers group (_treatment_) minus the ratio of the number of purchasers in the non-promotional group to the total number of customers in the non-promotional group (_control_).$$ IRR = \frac{purch_{treat}}{cust_{treat}} - \frac{purch_{ctrl}}{cust_{ctrl}} $$* **Net Incremental Revenue (NIR)**NIR depicts how much is made (or lost) by sending out the promotion. Mathematically, this is 10 Dollar times the total number of purchasers that received the promotion minus 0.15 Dollar times the number of promotions sent out, minus 10 Dollar times the number of purchasers who were not given the promotion.$$ NIR = (10\cdot purch_{treat} - 0.15 \cdot cust_{treat}) - 10 \cdot purch_{ctrl}$$For a full description of what Starbucks provides to candidates see the [instructions available here](https://drive.google.com/open?id=18klca9Sef1Rs6q8DW4l7o349r8B70qXM).Below we can find the training data provided.  Explore the data and different optimization strategies. How To Test Your Strategy?When we feel like you have an optimization strategy, complete the `promotion_strategy` function to pass to the `test_results` function.  From past data, we know there are four possible outomes:Table of actual promotion vs. predicted promotion customers:  ActualPredictedYesNoYesIIINoIIIIVThe metrics are only being compared for the individuals we predict should obtain the promotion – that is, quadrants I and II.  Since the first set of individuals that receive the promotion (in the training set) receive it randomly, we can expect that quadrants I and II will have approximately equivalent participants.  Comparing quadrant I to II then gives an idea of how well your promotion strategy will work in the future. Get started by reading in the data below.  See how each variable or combination of variables along with a promotion influences the chance of purchasing.  When you feel like you have a strategy for who should receive a promotion, test your strategy against the test dataset used in the final `test_results` function.  The Analytical Steps.  Import and variables<jupyter_code># load in packages
from itertools import combinations

from test_results import valid_results, test_results, score
from IPython.display import Image
from IPython.core.display import HTML 

import numpy as np
import pandas as pd
import scipy as sp
import sklearn as sk
import xgboost as xgb
from imblearn.over_sampling import SMOTE
from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize
from sklearn.externals import joblib

import matplotlib.pyplot as plt
import seaborn as sb
%matplotlib inline
# load in the data
train_data = pd.read_csv('./training.csv')
# train_data.head()
test_data = pd.read_csv('./Test.csv')
# test_data.head(7)
features = ['V1', 'V2', 'V3', 'V4', 'V5', 'V6', 'V7']
train_data['Promotion'].value_counts()
train_data['purchase'].value_counts()<jupyter_output><empty_output><jupyter_text>EDA  Training | Promotion and Purchase<jupyter_code>train_data[(train_data['Promotion']=='Yes') & (train_data['purchase']==1)][features].hist(figsize=(10,10));<jupyter_output><empty_output><jupyter_text>Promotion and Purchase-not<jupyter_code>train_data[(train_data['Promotion']=='Yes') & (train_data['purchase']==0)][features].hist(figsize=(10,10));<jupyter_output><empty_output><jupyter_text>Promotion-not and Purchase<jupyter_code>train_data[(train_data['Promotion']=='No') & (train_data['purchase']==1)][features].hist(figsize=(10,10));<jupyter_output><empty_output><jupyter_text>Promotion-not and Purchase-not<jupyter_code>train_data[(train_data['Promotion']=='No') & (train_data['purchase']==0)][features].hist(figsize=(10,10));
# testing baseline model, where we send promotions to everyone

def promotion_strategy(df):
    '''
    INPUT 
    df - a dataframe with *only* the columns V1 - V7 (same as train_data)

    OUTPUT
    promotion_df - np.array with the values
                   'Yes' or 'No' related to whether or not an 
                   individual should recieve a promotion 
                   should be the length of df.shape[0]            
    Ex:
    INPUT: df
    
    V1	V2	  V3	V4	V5	V6	V7
    2	30	-1.1	1	1	3	2
    3	32	-0.6	2	3	2	2
    2	30	0.13	1	1	4	2
    
    OUTPUT: promotion
    
    array(['Yes', 'Yes', 'No'])
    indicating the first two users would recieve the promotion and 
    the last should not.
    '''
    test = df
    
    promotion = []
    
    num_test_points = test.shape[0]
    
    for i in range(num_test_points):
        promotion.append('Yes')
        
    promotion = np.array(promotion)
    
    return promotion
# This will test your results, and provide you back some information 
# on how well your promotion_strategy will work in practice

test_results(promotion_strategy)
# Check sample size needed for control and treatment groups
NormalIndPower().solve_power(effect_size = proportion_effectsize(.0096, .015), alpha = .05, power = 0.8,
                             alternative = 'smaller')
# calculate our current treatment group size
train_data[train_data['Promotion']=='Yes'].shape[0]
# calculate our current control group size
train_data[train_data['Promotion']=='No'].shape[0]<jupyter_output><empty_output><jupyter_text>If we were to send everyone a promotion, our IRR is 0.96\% and NIR is -\\$1,132.20. This will be the baseline model. Hence, blindly sending everyone a promotion is going to lose the company a lot of money.Our baseline IRR is 0.096\%. If our desired minimum IRR is 1.5\% (an 0.081\% absolute increase in IRR), and we want to detect this change with a type 1 error rate of 0.05 and a power of 0.80, we will need a sample size of approximately 5,090 in each group (both control and treatment). Since, we have more than 40,000 data points in each group, we have sufficient data  An approach: Using Treatment Dummy ApproachIn this approach, a single model is used to model the data. Unlike the approach used in the first model, this approach adds an additional indicator variable to track if an individual is in a treatment or control group during training. Any individuals who made purchases will be given a label of 1, irregradless of whether he or she is in the control or treatment group. To predict whether a new individual in a test set is likely to make a purhcase only after receiving a promotion, we can first calculate the probability that the individual will make a purchase if he or she is in the treatment group by setting the treatment indicator as 1. Next, we calculate the probability that the individual will make a purchase if he or she is in the control group by setting the treatment indicator as 0. The difference in the two probabilities will be the lift value, aka, how positively influenced the individual is as a result of the treatment (receiving promotion):$$Lift = P(purchase | I_{treat} = 1) - P(purchase |  I_{treat} = 0)$$We can opt to send promotions to individuals with prostive lift values. Using this approach for this dataset, an irr of 1.42\% and an nlr of -\\$47.40 is obtained for the validation set. On the other hand, the irr and nlr values for the test set is 1.54\% and \\$42.60 respectively. Such a drastic difference between the validation and test nlrs is certainly surprising. The test results does seem to suggest that this is could be a relatively good approach as well. Alternative models, new feature generation and alternative upsampling methods could be tried to improve the results. In addition, you can also choose to send promotions only to individuals with lift values above a certain percentile.This approach is a slight modification of the method proposed by <NAME> in the previously mentioned presentation, which can be found [here](https://www.slideshare.net/odsc/victor-lomachinelearningpresentation). The original method proposes a logistic regression model.<jupyter_code>train_data = pd.read_csv('./training.csv')
test_data = pd.read_csv('./Test.csv')
# encode Promotion as 0 or 1 indicator variable
train_data['treatment'] = np.where(train_data['Promotion'] == "Yes", 1, 0)
train_data.head()
train, valid = sk.model_selection.train_test_split(train_data, test_size=0.2,random_state=42)

features = ['V1', 'V2', 'V3', 'V4', 'V5', 'V6', 'V7', 'treatment']

Y_train = train['purchase']
X_train = train[features]

Y_valid = valid['purchase']
X_valid = valid[features]
# up sample with SMOTE
sm = SMOTE(random_state=42, ratio = 1.0)
X_train_upsamp, Y_train_upsamp = sm.fit_sample(X_train, Y_train)
    
X_train_upsamp = pd.DataFrame(X_train_upsamp, columns=features)

Y_train_upsamp = pd.Series(Y_train_upsamp)
eval_set = [(X_train, Y_train), (X_valid, Y_valid)]
model = xgb.XGBClassifier(learning_rate = 0.1,\
                                  max_depth = 7,\
                                  min_child_weight = 5,\
                                  objective = 'binary:logistic',\
                                  seed = 42,\
                                  gamma = 0.1,\
                                  silent = True)
model.fit(X_train_upsamp, Y_train_upsamp, eval_set=eval_set,\
                    eval_metric="auc", verbose=True, early_stopping_rounds=30)
# model.save_model('first_model_approach.model')<jupyter_output><empty_output><jupyter_text>XGBoost Own-Tuning<jupyter_code>model = xgb.XGBClassifier(silent=True, 
                      scale_pos_weight=1,
                      learning_rate=0.01,  
                      colsample_bytree = 0.4,
                      subsample = 0.8,
                      objective='binary:logistic', 
                      n_estimators=1000, 
                      reg_alpha = 0.3,
                      max_depth=4, 
                      gamma=10,
                         n_jobs=-1)

model.fit(X_train_upsamp, Y_train_upsamp, eval_set=eval_set,\
                    eval_metric="auc", verbose=True, early_stopping_rounds=30)<jupyter_output>[0]	validation_0-auc:0.555743	validation_1-auc:0.562971
Multiple eval metrics have been passed: 'validation_1-auc' will be used for early stopping.

Will train until validation_1-auc hasn't improved in 30 rounds.
[1]	validation_0-auc:0.562834	validation_1-auc:0.55037
[2]	validation_0-auc:0.626966	validation_1-auc:0.602001
[3]	validation_0-auc:0.612422	validation_1-auc:0.604937
[4]	validation_0-auc:0.639091	validation_1-auc:0.625056
[5]	validation_0-auc:0.631983	validation_1-auc:0.613418
[6]	validation_0-auc:0.625879	validation_1-auc:0.605902
[7]	validation_0-auc:0.62632	validation_1-auc:0.607262
[8]	validation_0-auc:0.62426	validation_1-auc:0.603131
[9]	validation_0-auc:0.622293	validation_1-auc:0.60163
[10]	validation_0-auc:0.621509	validation_1-auc:0.600636
[11]	validation_0-auc:0.631961	validation_1-auc:0.609659
[12]	validation_0-auc:0.638265	validation_1-auc:0.618665
[13]	validation_0-auc:0.636436	validation_1-auc:0.617111
[14]	validation_0-auc:0.643867	validation_1-auc:0.623884
[1[...]<jupyter_text>max_depth<jupyter_code>max_depth_best_score=[]
max_depth= [3,4,5,6,7]
for l in max_depth:

    eval_set = [(X_train_upsamp, Y_train_upsamp), (X_valid, Y_valid)]
    model = xgb.XGBClassifier(silent=True, 
                          scale_pos_weight=1,
                          learning_rate=0.01,  
                          colsample_bytree = 0.4,
                          subsample = 0.8,
                          objective='binary:logistic', 
                          n_estimators=1000, 
                          reg_alpha = 0.3,
                          max_depth=l, 
                          gamma=10,
                          n_jobs=-1)
    model.fit(X_train_upsamp, Y_train_upsamp, eval_set=eval_set,\
                        eval_metric="auc", verbose=True, early_stopping_rounds=30)
    
    In= model.best_score
    max_depth_best_score.append(In)
    
max_depth_df= pd.DataFrame({'max_depth':max_depth,
             'learning_rate_best_score':max_depth_best_score})
max_depth_df<jupyter_output><empty_output><jupyter_text>we choose max_depth = 4.  learning rate tuning<jupyter_code>learning_rate= [0.0001,
0.0010,
0.0100,
0.1000,
0.2000,
0.3000]
lr_bestscore=[]

for l in learning_rate:

    eval_set = [(X_train_upsamp, Y_train_upsamp), (X_valid, Y_valid)]
    model = xgb.XGBClassifier(silent=True, 
                          scale_pos_weight=1,
                          learning_rate=l,  
                          colsample_bytree = 0.4,
                          subsample = 0.8,
                          objective='binary:logistic', 
                          n_estimators=1000, 
                          reg_alpha = 0.3,
                          max_depth=4, 
                          gamma=10,
                          n_jobs=-1)
    model.fit(X_train_upsamp, Y_train_upsamp, eval_set=eval_set,\
                        eval_metric="auc", verbose=True, early_stopping_rounds=30)
    
    learning_rate_best_score= model.best_score
    lr_bestscore.append(learning_rate_best_score)
    
lr_df= pd.DataFrame({'learning_rate':learning_rate,
             'learning_rate_best_score':lr_bestscore})
lr_df<jupyter_output><empty_output><jupyter_text>Subsamples<jupyter_code>subsamples = [0.8,0.9,1.0]
subsample_best=[]
for l in subsamples:

    eval_set = [(X_train_upsamp, Y_train_upsamp), (X_valid, Y_valid)]
    model = xgb.XGBClassifier(silent=True, 
                              scale_pos_weight=1,
                              learning_rate=0.01,  
                              colsample_bytree = 0.4,
                              subsample = l,
                              objective='binary:logistic', 
                              n_estimators=1000, 
                              reg_alpha = 0.3,
                              max_depth=4, 
                              gamma=10,
                              n_jobs=-1)
    model.fit(X_train_upsamp, Y_train_upsamp, eval_set=eval_set,\
                        eval_metric="auc", verbose=True, early_stopping_rounds=30)
    
    Ins= model.best_score
    subsample_best.append(Ins)
    
sub_df= pd.DataFrame({'sub_samples':subsamples,
             'learning_rate_best_score':subsample_best})
sub_df<jupyter_output><empty_output><jupyter_text>colsample_bytree<jupyter_code>trees= [x for x in np.arange(0.8,1.02,0.02)]
tree_best=[]

for l in trees:

    eval_set = [(X_train_upsamp, Y_train_upsamp), (X_valid, Y_valid)]
    model = xgb.XGBClassifier(silent=True, 
                              scale_pos_weight=1,
                              learning_rate=0.01,  
                              colsample_bytree = l,
                              subsample = 1.0,
                              objective='binary:logistic', 
                              n_estimators=1000, 
                              reg_alpha = 0.3,
                              max_depth=4, 
                              gamma=10,
                              n_jobs=-1)
    model.fit(X_train_upsamp, Y_train_upsamp, eval_set=eval_set,\
                        eval_metric="auc", verbose=True, early_stopping_rounds=30)
    
    Ins= model.best_score
    tree_best.append(Ins)
    
tree_df= pd.DataFrame({'sub_samples':trees,
             'learning_rate_best_score':tree_best})
tree_df<jupyter_output><empty_output><jupyter_text>Gamma<jupyter_code>gamma= [0,1,5,10]
gamma_best=[]

for l in gamma:

    eval_set = [(X_train_upsamp, Y_train_upsamp), (X_valid, Y_valid)]
    model = xgb.XGBClassifier(silent=True, 
                              scale_pos_weight=1,
                              learning_rate=0.01,  
                              colsample_bytree = 0.94,
                              subsample = 1.0,
                              objective='binary:logistic', 
                              n_estimators=1000, 
                              reg_alpha = 0.3,
                              max_depth=4, 
                              gamma=l,
                              n_jobs=-1)
    model.fit(X_train_upsamp, Y_train_upsamp, eval_set=eval_set,\
                        eval_metric="auc", verbose=True, early_stopping_rounds=30)
    
    Ins= model.best_score
    gamma_best.append(Ins)
    
gamma_df= pd.DataFrame({'gamma':gamma,
             'learning_rate_best_score':gamma_best})
tree_df
model.best_score
# check which features are important

from xgboost import plot_importance
from matplotlib import pyplot

fig, ax = pyplot.subplots(figsize=(10, 18));
xgb.plot_importance(model, ax=ax);
# confusion marix for thge validation set
valid_pred = model.predict(X_valid, ntree_limit=model.best_ntree_limit)
cm = sk.metrics.confusion_matrix(Y_valid, valid_pred)

fig, ax= plt.subplots(figsize=(10,10))
sb.heatmap(cm, annot=True, fmt='g', ax = ax); #annot=True to annotate cells

# labels, title and ticks
ax.set_xlabel('Predicted labels');
ax.set_ylabel('True labels'); 
ax.set_title('Confusion Matrix'); 
ax.xaxis.set_ticklabels(['No Purchase', 'Made Purchase']); 
ax.yaxis.set_ticklabels(['No Purchase', 'Made Purchase']);
def promotion_strategy(df):
    '''
    INPUT 
    df - a dataframe with *only* the columns V1 - V7 (same as train_data)

    OUTPUT
    promotion_df - np.array with the values
                   'Yes' or 'No' related to whether or not an 
                   individual should recieve a promotion 
                   should be the length of df.shape[0]
                
    Ex:
    INPUT: df
    
    V1	V2	  V3	V4	V5	V6	V7
    2	30	-1.1	1	1	3	2
    3	32	-0.6	2	3	2	2
    2	30	0.13	1	1	4	2
    
    OUTPUT: promotion
    
    array(['Yes', 'Yes', 'No'])
    indicating the first two users would recieve the promotion and 
    the last should not.
    '''
    test = df
    
    # Fit a model with treatment = 1 for all data points
    test['treatment'] = 1.0
    preds_treat = model.predict_proba(test, ntree_limit=model.best_ntree_limit)
    
    # Fit a model with treatment = 0 for all data points
    test['treatment'] = 0.0
    preds_cont = model.predict_proba(test, ntree_limit=model.best_ntree_limit)
    
    lift = preds_treat[:,1] - preds_cont[:,1]
    
    promotion = []
    
    for prob in lift:
        if prob > 0:
            promotion.append('Yes')
        else:
            promotion.append('No')

    promotion = np.array(promotion)
    
    return promotion
# find out irr and nlr on our validation set
valid_results(promotion_strategy, valid)<jupyter_output>Nice job!  See how well your strategy worked on our valid data below!

Your irr with this strategy is 0.0105.

Your nir with this strategy is -324.60.
We came up with a model with an irr of 0.0188 and an nir of 189.45 on the test set.

 How did you do?<jupyter_text>model records<jupyter_code># final= pd.DataFrame()

details = 'Fine tuning XGBOOST by Medium'
irr_results = 0.0105
nir_results = -324.5999

In= pd.DataFrame({'details':details,
             'irr_results': irr_results,
             'nir_results': nir_results}, index=[0])

final= final.append(In)
final<jupyter_output><empty_output><jupyter_text>Final Test Results<jupyter_code># This will test your results, and provide you back some information 
# on how well your promotion_strategy will work in practice

test_results(promotion_strategy)<jupyter_output>Nice job!  See how well your strategy worked on our test data below!

Your irr with this strategy is 0.0154.

Your nir with this strategy is 42.60.
We came up with a model with an irr of 0.0188 and an nir of 189.45 on the test set.

 How did you do?
{'fraction_non_alphanumeric': 0.08393307785774372, 'fraction_numerical': 0.03467613795681839, 'mean_word_length': 2.939432989690722, 'pattern_counts': {'":': 0, '<': 56, '<?xml version=': 0, '>': 57, 'https://': 3, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '222082', 'license': 'MIT', 'n_tokens_mistral': 6874, 'n_tokens_neox': 6321, 'n_words': 2041, 'path': 'nano_degree/incremental_modelling/Starbucks.ipynb', 'repo_name': 'yogitricahyono/dataScienceRecords', 'size': '21404'}