<jupyter_start><jupyter_text>The second change Audacity wants to try is on their course overview page. They created a new description for one of their courses to dedicate larger portions to connecting concepts in the course to career skills, and less on the details of each concept. They hope that this change may encourage more users to complete the course. For this experiment, instead of choosing just a single metric, we're going to analyze multiple. We'll track the enrolment rate, the average reading time on the course page, the average time spent in the classroom, and the course completion rate. First, we'll analyze these individually and then, we'll see how they all come together.<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
% matplotlib inline

np.random.seed(42)
df = pd.read_csv('course_page_actions.csv')
df.head()
# Get dataframe with all records from control group
control_df = df.query('group == "control"')

# Compute click through rate for control group
control_ctr = control_df.query('action == "enroll"').count()[0] / control_df.query('action == "view"').count()[0]

# Display click through rate
control_ctr
# Get dataframe with all records from experiment group
experiment_df = df.query('group == "experiment"')

# Compute click through rate for experiment group
experiment_ctr = experiment_df.query('action == "enroll"').count()[0] / experiment_df.query('action == "view"').count()[0]

# Display click through rate
experiment_ctr
# Compute the observed difference in click through rates
obs_diff = experiment_ctr - control_ctr

# Display observed difference
obs_diff
# Create a sampling distribution of the difference in proportions
# with bootstrapping
diffs = []
size = df.shape[0]
for _ in range(10000):
    b_samp = df.sample(size, replace=True)
    control_df = b_samp.query('group == "control"')
    experiment_df = b_samp.query('group == "experiment"')
    control_ctr = control_df.query('action == "enroll"').count()[0] / control_df.query('action == "view"').count()[0]
    experiment_ctr = experiment_df.query('action == "enroll"').count()[0] / experiment_df.query('action == "view"').count()[0]
    diffs.append(experiment_ctr - control_ctr)
# Convert to numpy array
diffs = np.array(diffs)

# Plot sampling distribution
plt.hist(diffs);
# Simulate distribution under the null hypothesis
null_vals = np.random.normal(0, diffs.std(), diffs.size)

# Plot the null distribution
plt.hist(null_vals);
# Plot observed statistic with the null distibution
plt.hist(null_vals)
plt.axvline(x=obs_diff, color='red');
# Compute p-value
(null_vals > obs_diff).mean()<jupyter_output><empty_output><jupyter_text>Hence, we do not have evidence that with a type I error rate of 0.05, the enrollment rate for this course increases when using the experimental description on its overview page. In addition to computing the enrolment rate, we can also compute the average reading durations with this dataset. The two analysis so far were comparing proportions. With this metric, we'll be analyzing the differnce in means. This analysis will be quite similar. Since we're comparing reading durations, we only care about view action. So let's filter by that first.<jupyter_code>df = pd.read_csv('course_page_actions.csv')
df.head(10)
views = df.query('action == "view"')<jupyter_output><empty_output><jupyter_text>And let's only count each unique user once by finding their average reading duration if they visited the site more than once. We'll also group by group, just so we keep track of that information. Resetting the index just so we keep the ID and group as column names. And it also let's us continue working in a data frame instead of a multi index series.<jupyter_code>reading_times = views.groupby(['id', 'group'])['duration'].mean()
reading_times = reading_times.reset_index()
reading_times.head()
control_mean = df.query('group == "control"')['duration'].mean()
experiment_mean = df.query('group == "experiment"')['duration'].mean()
control_mean, experiment_mean
obs_diff = experiment_mean - control_mean
obs_diff<jupyter_output><empty_output><jupyter_text>On average, it looks like users in the experiment group spent 15 more seconds on the course overview page than those in the control group. To see if this difference is significant, let's simulate the sampling distribution for the difference in mean reading durations with bootstrapping.<jupyter_code>diffs = []
for _ in range(10000):
    b_samp = df.sample(df.shape[0], replace=True)
    control_mean = b_samp.query('group == "control"')['duration'].mean()
    experiment_mean = b_samp.query('group == "experiment"')['duration'].mean()
    diffs.append(experiment_mean - control_mean)
diffs = np.array(diffs)
plt.hist(diffs);<jupyter_output><empty_output><jupyter_text>Now, to find the p-value let's simulate the distribution under the null and find the probability that our observed statistic came from this distribution. We'll create the distribution centered at 0 and having the same spread as our sampling distribution.<jupyter_code>null_vals = np.random.normal(0, diffs.std(), diffs.size)
plt.hist(null_vals)
plt.axvline(x=obs_diff, color='r', linewidth=2);<jupyter_output><empty_output><jupyter_text>Looking at our null distribution and where our observed statistic falls, we can say that our statistic definitely doesn't look like it came from this null distribution. Looks like the difference we observed is significant. Again, let's recap the steps we took to analyze the results of this A/B test.- We computed the observed difference between the metric, average reading duration, for the control and experiment group.- We simulated the sampling distribution for the difference in means (or average reading durations).- We used this sampling distribution to simulate the distribution under the null hypothesis, by creating a random normal distribution centered at 0 with the same spread and size.- We computed the p-value by finding the proportion of values in the null distribution that were greater than our observed difference.- We used this p-value to determine the statistical significance of our observed difference.<jupyter_code>df = pd.read_csv('classroom_actions.csv')
df.head(10)
# The total_days represents the total amount of time
# each student has spent in classroom.
# get the average classroom time for control group
control_mean = df.query('group == "control"')['total_days'].mean()

# get the average classroom time for experiment group
experiment_mean = df.query('group == "experiment"')['total_days'].mean()

# display average classroom time for each group
control_mean, experiment_mean
# compute observed difference in classroom time
obs_diff = experiment_mean - control_mean

# display observed difference
obs_diff
# create sampling distribution of difference in average classroom times
# with boostrapping
diffs = []
for _ in range(10000):
    boot_sample = df.sample(df.shape[0], replace=True)
    control_mean = boot_sample.query('group == "control"')['total_days'].mean()
    experiment_mean = boot_sample.query('group == "experiment"')['total_days'].mean()
    diffs.append(experiment_mean - control_mean)
# convert to numpy array
diffs = np.array(diffs)
# plot sampling distribution
plt.hist(diffs);
# simulate distribution under the null hypothesis
null_vals = np.random.normal(0, diffs.std(), diffs.size)
# plot null distribution
plt.hist(null_vals)

# plot line for observed statistic
plt.axvline(x=obs_diff, color='r', linewidth=2);
# compute p value
(null_vals > obs_diff).mean()<jupyter_output><empty_output><jupyter_text>Hence, we do have evidence that with a type I error rate of 0.05, users spent more time in the classroom after seeing the experimental description in the course overview page. But this difference is practically insignificant.<jupyter_code>df = pd.read_csv('classroom_actions.csv')
df.head(10)
# Create dataframe with all control records
control_df = df.query('group == "control"')

# Compute completion rate
control_ctr = control_df.query('completed == True').count()[0] / control_df.shape[0]

# Display completion rate
control_ctr
# Create dataframe with all experiment records
experiment_df = df.query('group == "experiment"')

# Compute completion rate
experiment_ctr = experiment_df.query('completed == True').count()[0] / experiment_df.shape[0]

# Display completion rate
experiment_ctr
# Compute observed difference in completion rates
obs_diff = experiment_ctr - control_ctr

# Display observed difference in completion rates
obs_diff
# Create sampling distribution for difference in completion rates
# with boostrapping
diffs = []
for _ in range(10000):
    boot_sample = df.sample(df.shape[0], replace=True)
    control_df = boot_sample.query('group == "control"')
    experiment_df = boot_sample.query('group == "experiment"')
    control_ctr = control_df.query('completed == True').count()[0] / control_df.shape[0]
    experiment_ctr = experiment_df.query('completed == True').count()[0] / experiment_df.shape[0]
    diffs.append(experiment_ctr - control_ctr)
# convert to numpy array
diffs = np.array(diffs)
# plot distribution
plt.hist(diffs);
# create distribution under the null hypothesis
null_vals = np.random.normal(0, diffs.std(), diffs.size)
# plot null distribution
plt.hist(null_vals)

# plot line for observed statistic
plt.axvline(x=obs_diff, color='r', linewidth=2);
# compute p value
(null_vals > obs_diff).mean()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09663776696976105, 'fraction_numerical': 0.006872488898287165, 'mean_word_length': 5.651898734177215, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 32, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '440565', 'license': 'MIT', 'n_tokens_mistral': 2690, 'n_tokens_neox': 2616, 'n_words': 988, 'path': 'AB testing/Case Study AB Tests Experiment II.ipynb', 'repo_name': 'ColdTeapot273K/data-stuff', 'size': '9458'}