<jupyter_start><jupyter_text>Start Import Data and Dependencies Import Libraries and set Dependencies<jupyter_code>#set dependencies
#filter out warnings
import warnings
warnings.filterwarnings('ignore')

#import libraries
#basics
import pandas as pd
import numpy as np

#statistics
from statsmodels.stats.proportion import proportions_ztest
from scipy.stats import ttest_ind

#visualisation
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Import Data<jupyter_code>#mount gdrive
from google.colab import drive
drive.mount('/content/drive')
#import data
data = pd.read_excel('/content/drive/MyDrive/Colab Notebooks/Turing College/Final Project Results.xlsx', sheet_name = ['Control','Experiment'])
control_data = data['Control']
experiment_data = data['Experiment']<jupyter_output><empty_output><jupyter_text>Data Insights Experiment<jupyter_code>experiment.head(3)
experiment.shape<jupyter_output><empty_output><jupyter_text>There are 37 rows and 5 columns<jupyter_code>experiment.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 37 entries, 0 to 36
Data columns (total 7 columns):
 #   Column       Non-Null Count  Dtype  
---  ------       --------------  -----  
 0   Date         37 non-null     object 
 1   Pageviews    37 non-null     int64  
 2   Clicks       37 non-null     int64  
 3   Enrollments  23 non-null     float64
 4   Payments     23 non-null     float64
 5   Tag          37 non-null     object 
 6   CTR          37 non-null     float64
dtypes: float64(3), int64(2), object(2)
memory usage: 2.3+ KB<jupyter_text>There are clearly missing values in this data. Also, the datatypes for columns 3 and 4 would more appropriately be integers. Control<jupyter_code>control.head(3)
control.shape<jupyter_output><empty_output><jupyter_text>Control data is equal in dimension to Experiment data<jupyter_code>control.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 37 entries, 37 to 73
Data columns (total 7 columns):
 #   Column       Non-Null Count  Dtype  
---  ------       --------------  -----  
 0   Date         37 non-null     object 
 1   Pageviews    37 non-null     int64  
 2   Clicks       37 non-null     int64  
 3   Enrollments  23 non-null     float64
 4   Payments     23 non-null     float64
 5   Tag          37 non-null     object 
 6   CTR          37 non-null     float64
dtypes: float64(3), int64(2), object(2)
memory usage: 2.3+ KB<jupyter_text>The Control data is similar in characteristics to the Experiment data. Combining the two Dataframes<jupyter_code>experiment.describe()
control.describe()<jupyter_output><empty_output><jupyter_text>The data distributions are different on face value.<jupyter_code>#create new columns to tag both datasets
experiment_data['Tag'] = 'experiment'
control_data['Tag'] = 'control'
#combine the two frames
df = experiment_data.append(control_data, ignore_index=True)<jupyter_output><empty_output><jupyter_text>Feature Handling Missing Values<jupyter_code>#inspect the missing values
df[df.isnull().any(axis=1)]<jupyter_output><empty_output><jupyter_text>From the overview of the data, I posit that the missing data are not missing at random; first missing date on both Experiment and Control is Nov 3, and last missing date Nov 16.<jupyter_code>#create a new data frame dropping all rows with at least 1 missing values
df_2 = df.dropna(how='any')
df_2.shape<jupyter_output><empty_output><jupyter_text>Datatype Conversion Integer is a more appropriate datatype for Enrollments and Payments columns<jupyter_code>#convert the data columns 3 and 4 from floats to integer
cols = ['Enrollments', 'Payments']
df_2[cols] = df_2[cols].applymap(np.int64)
df_2.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 46 entries, 0 to 59
Data columns (total 6 columns):
 #   Column       Non-Null Count  Dtype 
---  ------       --------------  ----- 
 0   Date         46 non-null     object
 1   Pageviews    46 non-null     int64 
 2   Clicks       46 non-null     int64 
 3   Enrollments  46 non-null     int64 
 4   Payments     46 non-null     int64 
 5   Tag          46 non-null     object
dtypes: int64(4), object(2)
memory usage: 2.5+ KB<jupyter_text>Exploratory Data Analysis on Full Sample<jupyter_code>#compare the data distribution between columns
fig = plt.figure(figsize=(8, 4))
plt.title('Distribution of Features (5 Summary Statistics)', size=15)
sns.boxplot(orient='h', data=df)<jupyter_output><empty_output><jupyter_text>Generally, the number of views clearly outnumber the number of Clicks and conversions. There is an outlier at the lower end of the pageviews distribution.<jupyter_code>plt.title('Clicks vs Pageviews', size=15)
sns.kdeplot(data=df, x='Pageviews', y='Clicks', hue='Tag', fill=True)
#plot the data distribution for Pageviews
fig = plt.figure(figsize=(10, 4))
plt.title('Distribution of Daily Pageviews by Group', size=15)
sns.histplot(data=df, x='Pageviews', hue='Tag', kde=True)
#plot the data distribution for Pageviews
fig = plt.figure(figsize=(10, 4))
plt.title('Distribution of Daily Clicks by Group', size=15)
sns.histplot(data=df, x='Clicks', hue='Tag', kde=True)<jupyter_output><empty_output><jupyter_text>The distribution for both groups are very similar.The number of pageviews ranges from around 7500 to 10500, 9000 to 9500 views occurring more often than below or above that range. The distribution of the pageviews is slightly skewed to the left because of the outliers as observed in the boxplot above.On most days, the number of clicks ranged from around 670 to around 830, with a special day having as many as 900 views. Statistical Data Analysis Null Hypothesis: The click-through rate of the Control group is similar to  the click-through rate Experiment group.Alternative Hypothesis: The click-through rates between the Control and Experiment groups are not the same. Confidence Level = 95% Solve for zscore<jupyter_code>#define function to compute area under curve for the left side of distribution
def auc(confidence_pct):
  confidence_level = confidence_pct/100
  alpha_level = (1-confidence_level)/2
  auc_left = (1 + confidence_level)/2 #auc is area under curve
  return alpha_level, auc_left
#95% confidence level
alpha_95, auc_95 = auc(95)
print(alpha_95, auc_95)<jupyter_output>0.025000000000000022 0.975<jupyter_text>zscore is approximately 1.96 from the z table<jupyter_code>#save zscore in a variable
zscore_95 = 1.96<jupyter_output><empty_output><jupyter_text>Compute and save parameters<jupyter_code>#create a column with click-through rates
df['CTR'] = df['Clicks'] / df['Pageviews']
control = df[df['Tag']=='control']
experiment = df[df['Tag']=='experiment']

#control
control_size = control['Pageviews'].sum()
control_click_count = control['Clicks'].sum()

#experiment
experiment_size = experiment['Pageviews'].sum()
experiment_click_count = experiment['Clicks'].sum()<jupyter_output><empty_output><jupyter_text>1. Visualize 95% confidence interval between control and experiment groups on [click-through rate]. Explain what the confidence interval means, what's the relationship with the population of a group.<jupyter_code>#compute and save parameters

"""
click-through rate
assuming that pageviews is unique to a user:
click-through rate = number of clicks / number of pageviews
"""

#control
control_click_through = control_click_count/control_size #control proportion

#experiment
experiment_click_through = experiment_click_count/experiment_size #experiment proportion

print('Control click-through rate: ', control_click_through, '\n', 'Experiment click-through rate: ', experiment_click_through)
#compute standard errors
se = np.sqrt(((control_click_through * (1-control_click_through))/control_size) + ((experiment_click_through * (1-experiment_click_through))/experiment_size))

#compute proportion difference
difference = control_click_through - experiment_click_through

#confidence interval
conf_interval = [(difference - (zscore_95*se)), (difference + (zscore_95*se))]

print("The point estimate of the difference is", difference)
print(f"The Confidence Interval is between {conf_interval[0]} and {conf_interval[1]}")<jupyter_output>The point estimate of the difference is -5.662709158693602e-05
The Confidence Interval is between -0.0013523068034943038 and 0.0012390526203204318<jupyter_text>*Interpretation*I am 95% confident that the difference in the overall click-through rate between the Control group and the Experiment group is between approximately -0.0014 and 0.0012 in a one-month experiment period, with Experiment group having the higher values.The best estimate of the difference, the point estimate, is 5.66e-05 Since 0% is in the interval, there is not a significant difference between the Experiment and Control groups. Visualise the Confidence Interval on Click-through Rate between Control and Experiment.<jupyter_code>fig = plt.figure(figsize=(10, 4))
plt.title('Click-through Rate by Group', size=15)
sns.kdeplot(data=df, x='CTR', hue='Tag')<jupyter_output><empty_output><jupyter_text>**Additional Metrics Definition** In considering Enrollments and Payments, we are limited up to the 3rd of November because the data from that date onwards are still yet to be logged in. Use the dataset without Nans. **Overall Conversion Rate**To compare the overall effect of each design in translating to paying customers.Payments/Pageviews **Click Conversion Rate**To compare the effectiveness of interaction the UI and UX of information to visitors to enrol. Enrollments/Clicks **Enrollment Conversion Rate**To compare the effectiveness of contents in converting enrollees to paying students. Payments/Enrollment<jupyter_code>#compute and save parameters

#OCR = Overall Conversion Rate
df_2['OCR'] = df_2['Payments']/df_2['Pageviews']
#CCR = Click Conversion Rate
df_2['CCR'] = df_2['Enrollments']/df_2['Clicks']
#ECR = Enrollment Conversion Rate
df_2['ECR'] = df_2['Payments']/df_2['Enrollments']

#split groups
control_2 = df_2[df_2['Tag']=='control']
experiment_2 = df_2[df_2['Tag']=='experiment']

#control
control_size_trunc = control_2['Pageviews'].sum()
control_click_count_trunc = control_2['Clicks'].sum()
control_payment_count = control_2['Payments'].sum()
control_enrol_count = control_2['Enrollments'].sum()

#experiment
experiment_size_trunc = experiment_2['Pageviews'].sum()
experiment_click_count_trunc = experiment_2['Clicks'].sum()
experiment_payment_count = experiment_2['Payments'].sum()
experiment_enrol_count = experiment_2['Enrollments'].sum()<jupyter_output><empty_output><jupyter_text>2. Verify that the difference in each of the metrics between control and experiment groups is statistically significant using z-test, 95% confidence level. **Click-through Rate (Overall)**<jupyter_code>count = [control_click_count, experiment_click_count]
nobs = [control_size, experiment_size]
ctr_zstatistics, ctr_pvalue = proportions_ztest(count, nobs, alternative='two-sided', prop_var=False)
print('ztstats:',ctr_zstatistics, '\npvalue:',ctr_pvalue)<jupyter_output>ztstats: -0.08566094109242048 
pvalue: 0.9317359524473912<jupyter_text>*Interpretation*p-value is 0.93, which is greater than the alpha value of 0.025, with a zvalue of |0.085|.We thus fail to reject the null hypothesis that there is no difference between the two groups.There is not sufficient evidence to conclude that there is a difference between the Click Through Rates of the control and the experiment groups. **Overall Conversion Rate**<jupyter_code>count = [control_payment_count, experiment_payment_count]
nobs = [control_size_trunc, experiment_size_trunc]
ocr_zstatistics, ocr_pvalue = proportions_ztest(count, nobs, alternative='two-sided', prop_var=False)
print('ztstats:',ocr_zstatistics, '\npvalue:',ocr_pvalue)<jupyter_output>ztstats: 1.281996502441112 
pvalue: 0.19984387315558716<jupyter_text>*Interpretation*p-value is 0.199, which is greater than the alpha value of 0.025, with a zvalue of 1.28 and a z-score of 0.899. We thus fail to reject the null hypothesis that there is no difference between the two groups.There is not sufficient evidence to conclude that there is a difference between the Overall Conversion Rates of the control and the experiment groups. **Click Conversion Rate**<jupyter_code>count = [control_enrol_count, experiment_enrol_count]
nobs = [control_click_count_trunc, experiment_click_count_trunc]
ccr_zstatistics, ccr_pvalue = proportions_ztest(count, nobs, alternative='two-sided', prop_var=False)
print('ztstats:',ccr_zstatistics, '\npvalue:',ccr_pvalue)<jupyter_output>ztstats: 4.701830023753982 
pvalue: 2.578401033720593e-06<jupyter_text>*Interpretation*p-value is approximately 0.000026, which is less than the alpha value of 0.025, with a zvalue of 4.70 and a z-score greater than 1. We thus reject the null hypothesis that there is no difference between the two groups.This provides a strong evidence that there is a significant difference between the click conversion rates of the control and experiment groups. **Enrollment Conversion Rate**<jupyter_code>count = [control_payment_count, experiment_payment_count]
nobs = [control_enrol_count, experiment_enrol_count]
ecr_zstatistics, ecr_pvalue = proportions_ztest(count, nobs, alternative='two-sided', prop_var=False)
print('ztstats:',ecr_zstatistics, '\npvalue:',ecr_pvalue)<jupyter_output>ztstats: -2.65092819003226 
pvalue: 0.008027090611358445<jupyter_text>*Interpretation*p-value is 0.008, which is less than the alpha value of 0.025, with a zvalue of 2.65 and a z-score of 0.995. We thus reject the null hypothesis that there is no difference between the two groups.This provides a strong evidence that there is a significant difference between the enrollment conversion rates control and experiment groups.<jupyter_code>control_2.shape<jupyter_output><empty_output><jupyter_text>Ztest is not ideal all the metrics excluding the Click-through rate. Ttest is bound to give different results. 3. Verify that the difference in each of the metrics between control and experiment groups is statistically significant using t-test, 95% confidence level. **Click-through Rate (Overall)**<jupyter_code>control_ctr_arr = control['CTR'].values #control
experiment_ctr_arr = experiment['CTR'].values #experiment

ttest_ind(a=control_ctr_arr, b=experiment_ctr_arr)<jupyter_output><empty_output><jupyter_text>The ttest for Click-through Rate has the same conclusion as with the ztest. **Overall Conversion Rate**<jupyter_code>control_ocr_arr = control_2['OCR'].values #control
experiment_ocr_arr = experiment_2['OCR'].values #experiment

ttest_ind(a=control_ocr_arr, b=experiment_ocr_arr)<jupyter_output><empty_output><jupyter_text>*Interpretation*According to this ttest, p-value is 0.57, which is greater than the alpha value of 0.025, with a tstatistic of 0.558. We thus fail to reject the null hypothesis that there is no difference between the two groups.There is not sufficient evidence to conclude that there is a difference between the Overall Conversion Rates of the control and the experiment groups.Which is the same conclusion as with the ztest. **Click Conversion Rate**<jupyter_code>control_ccr_arr = control_2['CCR'].values #control
experiment_ccr_arr = experiment_2['CCR'].values #experiment

ttest_ind(a=control_ccr_arr, b=experiment_ccr_arr)<jupyter_output><empty_output><jupyter_text>*Interpretation*According to this ttest, p-value is 0.13, which is greater than the alpha value of 0.025, with a tstatistic of 1.53. We thus fail to reject the null hypothesis that there is no difference between the two groups.There is not sufficient evidence to conclude that there is a difference between the Click Conversion Rates of the control and the experiment groups.This is different from the conclusion of the ztest where the Null Hypothesis was rejected. **Enrollment Conversion Rate**<jupyter_code>control_ecr_arr = control_2['ECR'].values #control
experiment_ecr_arr = experiment_2['ECR'].values #experiment

ttest_ind(a=control_ecr_arr, b=experiment_ecr_arr)<jupyter_output><empty_output><jupyter_text>*Interpretation*According to this ttest, p-value is 0.31, which is greater than the alpha value of 0.025, with a tstatistic of -1.01. We thus fail to reject the null hypothesis that there is no difference between the two groups.There is not sufficient evidence to conclude that there is a difference between the Click Conversion Rates of the control and the experiment groups.This is different from the conclusion of the ztest where the Null Hypothesis was rejected. 4. Compare both test method results. Explain why they differ / do not differ that much and why. The pvalues of the ztest and ttest results for the Click-through rates are very largely the same. This might be due to the fact that the data sizes are significantly larger than 30.The pvalues of the ztests and ttests for the other metrics differed greatly, although the Overall Conversion Rate had the same conclusion as in the ztest. This is most likely due to the fact that the dataset was truncated because of meeting data for dates beyond the 2nd of November in both groups, significan tly reducing the data sizes. 5. Choose 1 method (either z or t) and explore statistical significance of any metric under different confidence levels - 60%, 90%, 95%, 99%. If conclusions about significance differ under different confidence levels, explain why. **Click-through Rate (Overall)**<jupyter_code>print('ztstats:',ctr_zstatistics, '\npvalue:',ctr_pvalue)<jupyter_output>ztstats: -0.08566094109242048 
pvalue: 0.9317359524473912<jupyter_text>*60% Confidence Level*<jupyter_code>#60% confidence level
alpha_60, auc_60 = auc(60)
print('alpha:',alpha_60, '\nauc:',auc_60)<jupyter_output>alpha: 0.2 
auc: 0.8<jupyter_text>zscore is approximately 0.85 from the z table *Interpretation*p-value is 0.93, which is greater than the alpha value of 0.2, with a zstatistic of |0.085|. We thus fail to reject the null hypothesis that there is no difference between the two groups.There is not sufficient evidence to conclude that there is a difference between the Click Conversion Rates of the control and the experiment groups. *90% Confidence Level*<jupyter_code>#90% confidence level
alpha_90, auc_90 = auc(90)
print('alpha:',alpha_90, '\nauc:',auc_90)<jupyter_output>alpha: 0.04999999999999999 
auc: 0.95<jupyter_text>zscore is approximately 1.65 from the z table *Interpretation*p-value is 0.93, which is greater than the alpha value of 0.049, with a zstatistic of |0.085|. We thus fail to reject the null hypothesis that there is no difference between the two groups.There is not sufficient evidence to conclude that there is a difference between the Click Conversion Rates of the control and the experiment groups. *95% Confidence Level*<jupyter_code>#call previously computed zscore
print('alpha:',alpha_95, '\nauc:',auc_95)
print('zscore:',zscore_95)<jupyter_output>alpha: 0.025000000000000022 
auc: 0.975
zscore: 1.96<jupyter_text>*Interpretation*p-value is 0.93, which is greater than the alpha value of 0.025, with a zstatistic of |0.085|. We thus fail to reject the null hypothesis that there is no difference between the two groups.There is not sufficient evidence to conclude that there is a difference between the Click Conversion Rates of the control and the experiment groups. *99% Confidence Level*<jupyter_code>#99% confidence level
alpha_99, auc_99 = auc(99)
print('alpha:',alpha_99, '\nauc:',auc_99)<jupyter_output>alpha: 0.0050000000000000044 
auc: 0.995
{'fraction_non_alphanumeric': 0.086352687951622, 'fraction_numerical': 0.04186952288218111, 'mean_word_length': 5.6920438957476, 'pattern_counts': {'":': 0, '<': 122, '<?xml version=': 0, '>': 122, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 4, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '33', 'dir': 'jupyter-structured-clean-dedup', 'id': '285530', 'license': 'MIT', 'n_tokens_mistral': 6429, 'n_tokens_neox': 5811, 'n_words': 2040, 'path': 'Statistical_Analysis_AB_Testing.ipynb', 'repo_name': 'FunmiSomoye/A-B-Testing', 'size': '19513'}