<jupyter_start><jupyter_text>Portfolio Exercise: Starbucks  Background InformationThe dataset you will be provided in this portfolio exercise was originally used as a take-home assignment provided by Starbucks for their job candidates. The data for this exercise consists of about 120,000 data points split in a 2:1 ratio among training and test files. In the experiment simulated by the data, an advertising promotion was tested to see if it would bring more customers to purchase a specific product priced at $10. Since it costs the company 0.15 to send out each promotion, it would be best to limit that promotion only to those that are most receptive to the promotion. Each data point includes one column indicating whether or not an individual was sent a promotion for the product, and one column indicating whether or not that individual eventually purchased that product. Each individual also has seven additional features associated with them, which are provided abstractly as V1-V7. Optimization StrategyYour task is to use the training data to understand what patterns in V1-V7 to indicate that a promotion should be provided to a user. Specifically, your goal is to maximize the following metrics:* **Incremental Response Rate (IRR)** IRR depicts how many more customers purchased the product with the promotion, as compared to if they didn't receive the promotion. Mathematically, it's the ratio of the number of purchasers in the promotion group to the total number of customers in the purchasers group (_treatment_) minus the ratio of the number of purchasers in the non-promotional group to the total number of customers in the non-promotional group (_control_).$$ IRR = \frac{purch_{treat}}{cust_{treat}} - \frac{purch_{ctrl}}{cust_{ctrl}} $$* **Net Incremental Revenue (NIR)**NIR depicts how much is made (or lost) by sending out the promotion. Mathematically, this is 10 times the total number of purchasers that received the promotion minus 0.15 times the number of promotions sent out, minus 10 times the number of purchasers who were not given the promotion.$$ NIR = (10\cdot purch_{treat} - 0.15 \cdot cust_{treat}) - 10 \cdot purch_{ctrl}$$For a full description of what Starbucks provides to candidates see the [instructions available here](https://drive.google.com/open?id=18klca9Sef1Rs6q8DW4l7o349r8B70qXM).Below you can find the training data provided.  Explore the data and different optimization strategies. How To Test Your Strategy?When you feel like you have an optimization strategy, complete the `promotion_strategy` function to pass to the `test_results` function.  From past data, we know there are four possible outomes:Table of actual promotion vs. predicted promotion customers:  ActualPredictedYesNoYesIIINoIIIIVThe metrics are only being compared for the individuals we predict should obtain the promotion – that is, quadrants I and II.  Since the first set of individuals that receive the promotion (in the training set) receive it randomly, we can expect that quadrants I and II will have approximately equivalent participants.  Comparing quadrant I to II then gives an idea of how well your promotion strategy will work in the future. Get started by reading in the data below.  See how each variable or combination of variables along with a promotion influences the chance of purchasing.  When you feel like you have a strategy for who should receive a promotion, test your strategy against the test dataset used in the final `test_results` function.<jupyter_code># load in packages
from itertools import combinations

from test_results import test_results, score
import numpy as np
import pandas as pd
import scipy as sp
import sklearn as sk

import matplotlib.pyplot as plt
import seaborn as sb
%matplotlib inline

# load in the data
train_data = pd.read_csv('./training.csv')
train_data.head()<jupyter_output><empty_output><jupyter_text>Tasks:1)	Analyze the results of the experiment and identify the effect of the Treatment on product purchase and Net Incremental Revenue2)	Build a model to select the best customers to target that maximizes the Incremental Response Rate and Net Incremental Revenue.   Exploratory Data Analysis1. Have basic understanding in promotion proportion, purchase2. Discover relationship (correlation etc) between V1-V73. Decide whether we should scale the features<jupyter_code>df=train_data
df.describe()
print('We have {} customers, with {} as experiment group and {} as control group.'.format
      (len(df),
       len(df.loc[df.Promotion == 'Yes']),
       len(df.loc[df.Promotion == 'No'])))
promotion_perc=df.groupby("Promotion")['purchase'].agg(lambda x:round(x.sum()/x.count() *100,2))
promotion_perc
promotion_count=df.groupby("Promotion")['purchase'].agg(lambda x:x.sum())
promotion_count<jupyter_output><empty_output><jupyter_text>Deal with Missing Value / Duplicated value<jupyter_code># number of missing value, missing rate, sort
def missing_table(df):
    missing = df.isnull().sum().reset_index().rename(columns = {0:"missing_num"})

    missing['missing_rate']=missing['missing_num']/df.shape[0]

    missing_analy= missing[missing.missing_rate>0].sort_values(by='missing_rate',ascending= False)
    
    if missing_analy.shape[0]==0:
        print("No missing value spot")
    
    return missing_analy

missing_df=missing_table(df)
missing_df
df.duplicated().sum()<jupyter_output><empty_output><jupyter_text>A/B Testing<jupyter_code>control_prop= promotion_perc[0]
experiment_prop =promotion_perc[1]

# Formula
# identify treatment effect

experiment_n- control_n

from scipy.stats import norm


mu_B = 62
mu_A = 60

std_B = 45
std_A = 40

n_B = 4000
n_A = 6000

Z = (mu_experiment - mu_control)/np.sqrt(std_control**2/n_control + std_experiment**2/n_experiment)

pvalue = norm.sf(Z)

print("Z-score: {0}\np-value: {1}".format(Z,p_value))
def find_metrics(df):
    """
    find Incremental Response Rate (IRR) and Net Incremental Revenue(NIR) 
    """
    purchase_treat=df.loc[df.Promotion=='Yes','purchase'].sum()
    purchase_control=df.loc[df.Promotion=='No','purchase'].sum()

    num_treat= (df.Promotion=='Yes').sum()
    num_control=(df.Promotion=='No').sum()
    

    irr= round((purchase_treat/num_treat - purchase_control/num_control)*100,2)
    
    
    revenue,cost=10,0.15
    NIR = round(purchase_treat*revenue-cost*num_treat - purchase_control*revenue,2)
    return irr,NIR

 
#1)	Analyze the results of the experiment
#2) identify the effect of the Treatment on product purchase and Net Incremental Revenue

irr,NIR=find_metrics(df)
print("Incremental Response Rate (IRR) is: {}%, \nNet Incremental Revenue(NIR) is: ${}".format(irr,NIR))
from tqdm import tqdm

# Randomdized Experiment
def permtest(df, n_trials = 10000):
    """
    Compute a confidence interval for the irr and nir using a permutation method. 
    
    Input parameters:
        df: original dataframe on which to perform the test
        alternative: type of test to perform, {'less', 'greater'}
        n_trials: number of permutation trials to perform
    
    Output value:
        a confidence interval containing 95% of values
    """
    # initialize storage of permuted irr and nir
    sample_irr = []
    sample_nir = []
    
    # For each trial...
    for _ in tqdm(range(n_trials)):
        
        # randomly permute the grouping labels
        df.Promotion = np.random.permutation(df.Promotion)
        
        # compute the irr and nir on the permuted dataset
        irr ,nir = find_metrics(df)

        # and add the value to the list of samples
        sample_irr.append(irr)
        sample_nir.append(nir)
    
    irr_ci = []
    irr_ci.append(np.percentile(np.array(sample_irr),0.025))
    irr_ci.append(np.percentile(np.array(sample_irr),0.975))
    nir_ci = []
    nir_ci.append(np.percentile(np.array(sample_nir),0.025))
    nir_ci.append(np.percentile(np.array(sample_nir),0.975))

    return irr_ci, nir_ci
# irr_ci, nir_ci = permtest(train_data, n_trials = 1000)
# irr_ci, nir_ci

print("{},{}".format(round(irr_ci,3),round(nir_ci,3)))<jupyter_output>100%|██████████| 1000/1000 [00:39<00:00, 25.27it/s]<jupyter_text>Part 2: Find promotion strategy1. Machine Learning prediction     * Random Forest     * Xgboost<jupyter_code>def compare_features(row):
    '''
    A helper function to get the predicted probability of purchasing for a given individual and make a decision based on a
    chosen boundary.
    
    INPUTS:
    row: an array containing the values of interest in a row
    
    OUTPUT:
    a string specifying whether a given individual (characterized by a row) should be sent the promotion or not
    '''
    if tree.predict_proba(row.values.reshape(1,-1))[0][1] > 0.4:
        return 'Yes'
    else:
        return 'No'
def promotion_strategy(df):
    '''
    INPUT 
    df - a dataframe with *only* the columns V1 - V7 (same as train_data)

    OUTPUT
    promotion_df - np.array with the values
                   'Yes' or 'No' related to whether or not an 
                   individual should recieve a promotion 
                   should be the length of df.shape[0]
                
    Ex:
    INPUT: df
    
    V1	V2	  V3	V4	V5	V6	V7
    2	30	-1.1	1	1	3	2
    3	32	-0.6	2	3	2	2
    2	30	0.13	1	1	4	2
    
    OUTPUT: promotion
    
    array(['Yes', 'Yes', 'No'])
    indicating the first two users would recieve the promotion and 
    the last should not.
    '''

    promotion = df.apply(compare_features, axis = 1)
    return promotion<jupyter_output><empty_output><jupyter_text>Feature Engineering<jupyter_code>import os
os.listdir()
train_data = pd.read_csv('./training.csv')
test_data = pd.read_csv('./Test.csv')
from sklearn import model_selection

# split data into train and valid
train, valid = model_selection.train_test_split(train_data, test_size=0.2,random_state=1)
Y_train = train['response']
X_train = train[features]

Y_valid = valid['response']
X_valid = valid[features]
lr = LogisticRegression()
lr.fit(X_train, y_train)
y_pred = lr.predict(X_test)
accuracy_score(y_test, y_pred)

#!pip install numpy==1.8 Make sure Numpy version is 1.8

from imblearn.over_sampling import SMOTE
# up sample only the train dataset with SMOTE
sm = SMOTE(random_state=1, ratio = 1.0)
X_train_upsamp, Y_train_upsamp = sm.fit_sample(X_train, Y_train)
X_train_upsamp = pd.DataFrame(X_train_upsamp, columns=features)

Y_train_upsamp = pd.Series(Y_train_upsamp)

# generate features and labels
Y_train = train['response']
X_train = train[features]

Y_valid = valid['response']
X_valid = valid[features]<jupyter_output><empty_output><jupyter_text>Random Forest  Xgboost<jupyter_code># This will test your results, and provide you back some information 
# on how well your promotion_strategy will work in practice

print("Default Strategy")

test_results(promotion_strategy)<jupyter_output>Default Strategy
Nice job!  See how well your strategy worked on our test data below!

Your irr with this strategy is 0.0096.

Your nir with this strategy is -1132.20.
We came up with a model with an irr of 0.0188 and an nir of 189.45 on the test set.

 How did you do?
{'fraction_non_alphanumeric': 0.08403967538322814, 'fraction_numerical': 0.02028854824165915, 'mean_word_length': 4.243971631205674, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 29, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '458607', 'license': 'MIT', 'n_tokens_mistral': 3508, 'n_tokens_neox': 3289, 'n_words': 1226, 'path': 'Starbucks.ipynb', 'repo_name': 'dujijundavid/Data_Science_Udacity_Portfolio-', 'size': '11090'}