<jupyter_start><jupyter_text>ProblemThe number of shoes sold by an e-commerce company during the first three months(12 weeks) of the year were:23 21 19 24 35 17 18 24 33 27 21 23Meanwhile, the company developed some dynamic price optimization algorithms and the sales for the next 12 weeks were:31 28 19 24 32 27 16 41 23 32 29 33Did the dynamic price optimization algorithm deliver superior results? Can it be trusted? SolutionBefore we get onto different approaches, let's quickly get a feel for the data<jupyter_code>import numpy as np
import seaborn as sns
sns.set(color_codes=True)
%matplotlib inline
#Load the data
before_opt = np.array([23, 21, 19, 24, 35, 17, 18, 24, 33, 27, 21, 23])
after_opt = np.array([31, 28, 19, 24, 32, 27, 16, 41, 23, 32, 29, 33])
before_opt.mean()
after_opt.mean()
observed_difference = after_opt.mean() - before_opt.mean()
print("Difference between the means is:", observed_difference)<jupyter_output>Difference between the means is: 4.16666666667<jupyter_text>On average, the sales after optimization is more than the sales before optimization. But is the difference legit? Could it be due to chance?**Classical Method** : We could cover this method later on. This entails doing a *t-test* **Hacker's Method** : Let's see if we can provide a hacker's perspective to this problem, similar to what we did in the previous notebook.<jupyter_code>#Step 1: Create the dataset. Let's give Label 0 to before_opt and Label 1 to after_opt
#Learn about the following three functions
?np.append
?np.zeros
?np.ones
shoe_sales = np.array([np.append(np.zeros(before_opt.shape[0]), np.ones(after_opt.shape[0])),
np.append(before_opt, after_opt)], dtype=int)
print("Shape:", shoe_sales.shape)
print("Data:", "\n", shoe_sales)
shoe_sales = shoe_sales.T
print("Shape:",shoe_sales.shape)
print("Data:", "\n", shoe_sales)
#This is the approach we are going to take
#We are going to randomly shuffle the labels. Then compute the mean between the two groups. 
#Find the % of times when the difference between the means computed is greater than what we observed above
#If the % of times is less than 5%, we would make the call that the improvements are real
np.random.shuffle(shoe_sales)
shoe_sales
experiment_label = np.random.randint(0,2,shoe_sales.shape[0])
experiment_label
experiment_data = np.array([experiment_label, shoe_sales[:,1]])
experiment_data = experiment_data.T
print(experiment_data)
experiment_diff_mean =  experiment_data[experiment_data[:,0]==1].mean() \
                        - experiment_data[experiment_data[:,0]==0].mean()
experiment_diff_mean
#Like the previous notebook, let's repeat this experiment 100 and then 100000 times
def shuffle_experiment(number_of_times):
    experiment_diff_mean = np.empty([number_of_times,1])
    for times in np.arange(number_of_times):
        experiment_label = np.random.randint(0,2,shoe_sales.shape[0])
        experiment_data = np.array([experiment_label, shoe_sales[:,1]]).T
        experiment_diff_mean[times] =  experiment_data[experiment_data[:,0]==1].mean() \
                        - experiment_data[experiment_data[:,0]==0].mean()
    return experiment_diff_mean    
experiment_diff_mean = shuffle_experiment(100)
experiment_diff_mean[:10]
sns.distplot(experiment_diff_mean, kde=False)
#Finding % of times difference of means is greater than observed
print("Data: Difference in mean greater than observed:", \
        experiment_diff_mean[experiment_diff_mean>=observed_difference])

print("Number of times diff in mean greater than observed:", \
            experiment_diff_mean[experiment_diff_mean>=observed_difference].shape[0])
print("% of times diff in mean greater than observed:", \
        experiment_diff_mean[experiment_diff_mean>=observed_difference].shape[0]/float(experiment_diff_mean.shape[0])*100)<jupyter_output>Data: Difference in mean greater than observed: []
Number of times diff in mean greater than observed: 0
% of times diff in mean greater than observed: 0.0<jupyter_text>Exercise: Repeat the above for 100,000 runs and report the results  Is the result by chance?   What is the justification for shuffling the labels? >Thought process is this: If price optimization had no real effect, then, the sales before optimization would often give more sales than sales after optimization. By shuffling, we are simulating the situation where that happens -  sales before optimization is greater than sales after optimization. If many such trials provide improvements, then, the price optimization has no effect. In statistical terms, *the observed difference could have occurred by chance*. Now, to show that the same difference in mean might lead to a different conclusion, let's try the same experiment with a different dataset.<jupyter_code>before_opt = np.array([230, 210, 190, 240, 350, 170, 180, 240, 330, 270, 210, 230])
after_opt = np.array([310, 180, 190, 240, 220, 240, 160, 410, 130, 320, 290, 210])
print("Mean sales before price optimization:", np.mean(before_opt))
print("Mean sales after price optimization:", np.mean(after_opt))
print("Difference in mean sales:", np.mean(after_opt) - np.mean(before_opt)) #Same as above
shoe_sales = np.array([np.append(np.zeros(before_opt.shape[0]), np.ones(after_opt.shape[0])),
np.append(before_opt, after_opt)], dtype=int)
shoe_sales = shoe_sales.T
experiment_diff_mean = shuffle_experiment(100000)
sns.distplot(experiment_diff_mean, kde=False)
#Finding % of times difference of means is greater than observed
print("Number of times diff in mean greater than observed:", \
            experiment_diff_mean[experiment_diff_mean>=observed_difference].shape[0])
print("% of times diff in mean greater than observed:", \
        experiment_diff_mean[experiment_diff_mean>=observed_difference].shape[0]/float(experiment_diff_mean.shape[0])*100)<jupyter_output>Number of times diff in mean greater than observed: 40473
% of times diff in mean greater than observed: 40.473<jupyter_text>Did the conclusion change now?   Effect Size> **Because you can't argue with all the fools in the world. It's easier to let them have their way, then trick them when they're not paying attention**  - <NAME>In the first case, how much did the price optimization increase the sales on average?<jupyter_code>before_opt = np.array([23, 21, 19, 24, 35, 17, 18, 24, 33, 27, 21, 23])
after_opt = np.array([31, 28, 19, 24, 32, 27, 16, 41, 23, 32, 29, 33])

print("The % increase of sales in the first case:", \
(np.mean(after_opt) - np.mean(before_opt))/np.mean(before_opt)*100,"%")
before_opt = np.array([230, 210, 190, 240, 350, 170, 180, 240, 330, 270, 210, 230])
after_opt = np.array([310, 180, 190, 240, 220, 240, 160, 410, 130, 320, 290, 210])

print("The % increase of sales in the second case:", \
(np.mean(after_opt) - np.mean(before_opt))/np.mean(before_opt)*100,"%")<jupyter_output>The % increase of sales in the second case: 1.75438596491 %<jupyter_text>**Would business feel comfortable spending millions of dollars if the increase is going to be just 1.75%. Does it make sense? Maybe yes - if margins are thin and any increase is considered good. But if the returns from the price optimization module does not let the company break even, it makes no sense to take that path.** > Someone tells you the result is statistically significant. The first question you should ask? How large is the effect?To answer such a question, we will make use of the concept **confidence interval**In plain english, *confidence interval* is the range of values the measurement metric is going to take. An example would be: 90% of the times, the increase in average sales (before and after price optimization) would be within the bucket `3.4 and 6.7` (These numbers are illustrative. We will derive those numbers below)What is the *hacker's way* of doing it? We will do the following steps:1. From actual sales data, we sample the data with repetition (separately for before and after) - sample size will be the same as the original2. Find the differences between the mean of the two samples.3. Repeat steps 1 and 2 , say 100,000 times.4. Sort the differences. For getting 90% interval, take the 5% and 95% number. That range gives you the 90% confidence interval on the mean.5. This process of generating the samples is called **bootstrapping**<jupyter_code>#Load the data
before_opt = np.array([23, 21, 19, 24, 35, 17, 18, 24, 33, 27, 21, 23])
after_opt = np.array([31, 28, 19, 24, 32, 27, 16, 41, 23, 32, 29, 33])
#generate a uniform random sample
random_before_opt = np.random.choice(before_opt, size=before_opt.size, replace=True)
print("Actual sample before optimization:", before_opt)
print("Bootstrapped sample before optimization: ", random_before_opt)
print("Mean for actual sample:", np.mean(before_opt))
print("Mean for bootstrapped sample:", np.mean(random_before_opt))
random_after_opt = np.random.choice(after_opt, size=after_opt.size, replace=True)
print("Actual sample after optimization:", after_opt)
print("Bootstrapped sample after optimization: ", random_after_opt)
print("Mean for actual sample:", np.mean(after_opt))
print("Mean for bootstrapped sample:", np.mean(random_after_opt))
print("Difference in means of actual samples:", np.mean(after_opt) - np.mean(before_opt))
print("Difference in means of bootstrapped samples:", np.mean(random_after_opt) - np.mean(random_before_opt))
#Like always, we will repeat this experiment 100,000 times. 

def bootstrap_experiment(number_of_times):
    mean_difference = np.empty([number_of_times,1])
    for times in np.arange(number_of_times):
        random_before_opt = np.random.choice(before_opt, size=before_opt.size, replace=True)
        random_after_opt = np.random.choice(after_opt, size=after_opt.size, replace=True)
        mean_difference[times] = np.mean(random_after_opt) - np.mean(random_before_opt)
    return mean_difference
mean_difference = bootstrap_experiment(100000)
sns.distplot(mean_difference, kde=False)
mean_difference = np.sort(mean_difference, axis=0)
mean_difference #Sorted difference
np.percentile(mean_difference, [5,95])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09859994042299672, 'fraction_numerical': 0.04885314268692285, 'mean_word_length': 5.48551191242756, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 22, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '34486', 'license': 'MIT', 'n_tokens_mistral': 3422, 'n_tokens_neox': 2926, 'n_words': 1130, 'path': 'module_1d_resampling.ipynb', 'repo_name': 'sanketvega/hackermath', 'size': '10084'}