<jupyter_start><jupyter_text>1. Experiment DefinitionWe developed a new webpage and want to test it's effects on purchase conversion. As such we split our users evenly into 2 groups:1. **Control**: They get the old webpage2. **Treatment**: They get the new webpageMetric we want to track:$$\text{purchase conversion} = \frac{\text{ Converted Users}}{\text{ Exposed Users}}$$We have 3 weeks of logged exposure/conversion data. Let's define these terms:1. **Exposure**: A user is bucketed as control or treatment and sees their corresponding page for the first time in the experiment duration2. **Conversion**: An exposed user makes a purchase within 7 days of being first exposedQuestions you should ask when setting up a test:- How do you think the experiment will fair?- Do we have actionable next steps laid out?  2. Data Collection Let's use some A/B testing data: https://www.kaggle.com/saraabdelaal/abtestdata<jupyter_code>df = pd.read_csv('csv/ab_data.csv')
df.sample(2)<jupyter_output><empty_output><jupyter_text>Each row is logged when user is exposed to a webpage.- **timestamp**: time the user is first exposed- **group**: bucket- **landing_page**: Which page are they seeing- **converted**: Initialized to 0. Changes to 1 if the user makes a purchase within 7 days of first exposure<jupyter_code>start_time = datetime.datetime.strptime(df['timestamp'].min(), '%Y-%m-%d %H:%M:%S.%f')
end_time = datetime.datetime.strptime(df['timestamp'].max(), '%Y-%m-%d %H:%M:%S.%f')
data_duration = (end_time - start_time).days

print(f"Number of unique users in experiment: {df['user_id'].nunique()}")
print(f"Data collected for {data_duration} days")
print(f"Landing pages to compare: {df['landing_page'].unique().tolist()}")
print(f"Percentage of users in control: {round(df[df['group']=='control'].shape[0] * 100 / df.shape[0])}%")<jupyter_output>Number of unique users in experiment: 290584
Data collected for 21 days
Landing pages to compare: ['old_page', 'new_page']
Percentage of users in control: 50%<jupyter_text>3. Data Processing But there are some repeated exposures for some users<jupyter_code>sample = df[df['user_id'].isin([746755,722274])]
sample<jupyter_output><empty_output><jupyter_text>Get First timestamp for each username. In this dataset, it is the timestamp of first conversion, but in reality, make sure to do one of the following:1. Get timestamp of first exposure2. Remove users with multiple buckets<jupyter_code># 1. Get timestamp of first exposure
first_conversion = sample.groupby('user_id')['timestamp'].min().to_frame().reset_index()
sample = sample.merge(first_conversion, on=['user_id', 'timestamp'])
sample
counter = df['user_id'].value_counts()
(counter > 1).value_counts()<jupyter_output><empty_output><jupyter_text>3894 (1.34\%) user_ids have been exposed to the old AND new page. It should be okay to remove them<jupyter_code>#2. Remove users with multiple buckets
valid_users = pd.DataFrame(counter[counter == 1].index, columns=['user_id'])
df = df.merge(valid_users, on=['user_id'])
# Add week column to see the data as you would during experiment
df['week'] = df['timestamp'].apply(lambda x: datetime.datetime.strptime(x, '%Y-%m-%d %H:%M:%S.%f').isocalendar()[1])
df.sample()
df['week'].value_counts()<jupyter_output><empty_output><jupyter_text>4. Experiment: Frequentist Approach<jupyter_code># Get Stats
NUM_WEEKS = 4 # Vary number to get experiment data at weekly points in time
experiment_data = df[df['week'] <= NUM_WEEKS]
control = experiment_data[experiment_data['group']=='control']
treatment = experiment_data[experiment_data['group']=='treatment']

control_conversion_perc = round(control['converted'].sum() * 100/ control['converted'].count(), 3)
treatment_conversion_perc = round(treatment['converted'].sum() * 100/ treatment['converted'].count(), 3)
lift = round(treatment_conversion_perc - control_conversion_perc, 3)

print(f"Treatment Conversion Rate: {treatment_conversion_perc}%")
print(f"Control Conversion Rate: {control_conversion_perc}%")
print(f"Lift = {lift}%")<jupyter_output>Treatment Conversion Rate: 11.873%
Control Conversion Rate: 12.017%
Lift = -0.144%<jupyter_text>Chi-Squared Test$$H_0: \text{Control & Treatment are independent} \\H_A: \text{Control & Treatment are not independent} $$<jupyter_code># Create Contingency Table for Chi Squared Test
control_converted = control['converted'].sum()
treatment_converted = treatment['converted'].sum()
control_non_converted = control['converted'].count() - control_converted
treatment_non_converted = treatment['converted'].count() - treatment_converted
contingency_table = np.array([[control_converted, control_non_converted], 
                              [treatment_converted, treatment_non_converted]])
contingency_table
chi, p_value, _, _ = chi2_contingency(contingency_table, correction=False)
chi, p_value<jupyter_output><empty_output><jupyter_text>Since the p_value > 0.05, we cannot reject null hypothesis. Hence, we cannot conclude if there exists a relationship between the control and treatment groups.<jupyter_code>print(f"{round(p_value * 100, 2)}% probability that a more extreme chi square than {round(chi, 3)} would have occured by chance")<jupyter_output>28.83% probability that a more extreme chi quare than 1.128 would have occured by chance<jupyter_text>But this is tough to interpret. We would to say something about the actual maginitude of lift. Something like this:<jupyter_code>print(f"(We CANNOT say this) We are {round(p_value * 100, 2)}% confident that our lift = {lift}%")
Image(filename='../mics/b_v_f.png')<jupyter_output><empty_output><jupyter_text>5. Experiment: Bayesian Approach We want to input the _prior distribution_ and have the experiment update the parameters to create _posterier distributions_. Since these prior & posterior distributions will be used to sample Conversion Rate, we model them after beta distribtion.Let's create the prior beta distribtion from the first weeks of conversion data<jupyter_code>prior = df[(df['week'] == 1) & (df['group']=='control')]
prior_means = []
for i in range(10000):
    prior_means.append(prior.sample(1000)['converted'].mean())
prior_means[:10]
# Model Beta Distribtion from sample means
prior_alpha, prior_beta, _, _ = beta.fit(prior_means, floc=0, fscale=1)
# Get Stats
NUM_WEEKS = 4 # Vary number to get experiment data at weekly points in time
experiment_data = df[(df['week'] > 1) & (df['week'] <= NUM_WEEKS)]
control = experiment_data[experiment_data['group']=='control']
treatment = experiment_data[experiment_data['group']=='treatment']

control_conversion_perc = round(control['converted'].sum() * 100/ control['converted'].count(), 3)
treatment_conversion_perc = round(treatment['converted'].sum() * 100/ treatment['converted'].count(), 3)
lift = round((treatment_conversion_perc - control_conversion_perc) / control_conversion_perc , 3)

print(f"Treatment Conversion Rate: {treatment_conversion_perc}%")
print(f"Control Conversion Rate: {control_conversion_perc}%")
print(f"Lift = {lift}%")
control_converted = control['converted'].sum()
treatment_converted = treatment['converted'].sum()
control_non_converted = control['converted'].count() - control_converted
treatment_non_converted = treatment['converted'].count() - treatment_converted

# Update Prior parameters with experiment conversion rates
posterior_control = beta(prior_alpha + control_converted, prior_beta + control_non_converted)
posterior_treatment = beta(prior_alpha + treatment_converted, prior_beta + treatment_non_converted)

# Sample from Posteriors
control_samples = posterior_control.rvs(1000)
treatment_samples = posterior_treatment.rvs(1000)
probability = np.mean(treatment_samples > control_samples)
print(f"Probability that treatment > control: {probability * 100}%")
(control_mu), (control_var) = posterior_control.stats()
(treatment_mu), (treatment_var) = posterior_treatment.stats()
print(f"Control Posterior: Mean: {control_mu}, Variance: {control_var}") 
print(f"Treatment Posterior: Mean: {treatment_mu}, Variance: {treatment_var}")<jupyter_output>Control Posterior: Mean: 0.12056303255497915, Variance: 1.0333316272670703e-06
Treatment Posterior: Mean: 0.1190944894489385, Variance: 1.0242362170011709e-06<jupyter_text>We can even make statements like the following which are actionable:<jupyter_code>lift_percentage = (treatment_samples - control_samples) / control_samples
print(f"Probability that we are seeing a 2% lift: {np.mean((100 * lift_percentage) > 2) * 100}%")<jupyter_output>Probability that we are seeing a 2% lift: 0.4%
{'fraction_non_alphanumeric': 0.12270010547287004, 'fraction_numerical': 0.026133833352865347, 'mean_word_length': 7.03578154425612, 'pattern_counts': {'":': 0, '<': 42, '<?xml version=': 0, '>': 46, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '342815', 'license': 'MIT', 'n_tokens_mistral': 2878, 'n_tokens_neox': 2563, 'n_words': 764, 'path': 'Importance of Bayesian Testing .ipynb', 'repo_name': 'ajhalthor/bayesian-testing', 'size': '8533'}