<jupyter_start><jupyter_text>treatment date:  the date of the email being sent  test:  if the email was in test or control group  is signed up:  if the user is signed up  signup date:  when the user signed up    How large is each group?By group I believe they mean test groups.  About 30k in the control group and 3.3k in the test group.  As expected, there are about 10% in the test group.<jupyter_code>df['test'].value_counts()
df[df['test']==1].shape[0]/df.shape[0]<jupyter_output><empty_output><jupyter_text>What is the signup rate of test and control groups?<jupyter_code>control_signedup = df[(df['test']==0) & (df['is_signed_up']==1)]
test_signedup = df[(df['test']==1) & (df['is_signed_up']==1)]
print('Signup rate for control group: {}'.format(control_signedup.shape[0]/df[df['test']==0].shape[0]*100))
print('Signup rate for test group: {}'.format(test_signedup.shape[0]/df[df['test']==1].shape[0]*100))<jupyter_output>Signup rate for test group: 13.701067615658364<jupyter_text>Note the signup rates for the test groups is actually lower than the control.  Is the experiment already significant?To find out, perform a t-test and set p-value to 0.05.  The null hypothesis is that the new email campaign resulted in more people signing up for the service.  The alternative is that the email campaign made no difference in signup rates.<jupyter_code>sample_control_signedup = np.random.choice(df[(df['test']==0)]['is_signed_up'], size = 500, replace = True)
sample_test_signedup = np.random.choice(df[(df['test']==1)]['is_signed_up'], size = 500, replace = False)
print(ttest_ind(sample_control_signedup, sample_test_signedup, equal_var=False))<jupyter_output>Ttest_indResult(statistic=-0.37334364178372648, pvalue=0.70897203837167422)<jupyter_text>The t-test has a value of 1.62, but the p-value is 0.11, greater than 0.05.  Thus, we reject the null hypothesis and there conclude that for the first 2 weeks, there is no difference between the control and test groups.  Why would the test signup rate be lower than the control signup rate?Look at secondary metric, the time between receiving the email to the signup date.<jupyter_code>def time_to_signup(cols):
#     print(cols[0], cols[1])
    return (cols[1] - cols[0]).days
df[(df['is_signed_up']) & (df['signup_date'].notnull())][['treatment_date', 'signup_date']].head()
df['time_elapsed'] = df[(df['is_signed_up']) & (df['signup_date'].notnull())][['treatment_date', 'signup_date']].apply(time_to_signup, axis = 1)
print('Average signup time for control group: {}'.format(df[df['test']==0]['time_elapsed'].mean()))
print('Average signup time for test group: {}'.format(df[df['test']==1]['time_elapsed'].mean()))<jupyter_output>Average signup time for control group: 3.187275179856115
Average signup time for test group: 4.283549783549783<jupyter_text>Here we see that it takes on average a full day longer for the test group to sign up, with the control group averaging 3 days to sign up and the test group averaging 4 days.  Below is a distribution of when people signed up.  It shows a right skew distribution spanning the whole two weeks.  The control group peaked at 2 days whereas the test group peaked at 3 days.<jupyter_code>df['days_to_signup'] = (df['signup_date'] - df['treatment_date']).dt.days
sns.countplot(x = 'days_to_signup', hue = 'test', data = df)
plt.xticks(rotation = 90)
plt.title('Distribution of How Long it Took to Signup First Two Weeks')
plt.xlabel('Days to Signup')
plt.ylabel('Count')<jupyter_output><empty_output><jupyter_text>Get final datasetAnswer same questions as above.<jupyter_code>final = pd.read_csv('experiment_data_final.txt', parse_dates=['treatment_date', 'signup_date'])<jupyter_output><empty_output><jupyter_text>How large is each group?There are ~47k in the control group and ~20k in the test group.  Test group now accounts for 30% of campaign emails sent.  But we wanted 50%...<jupyter_code>final['test'].value_counts()
final[final['test']==1].shape[0]/len(final)*100<jupyter_output><empty_output><jupyter_text>What is the signup rate of test and control groups?<jupyter_code>print('Signup rate for control group: {}'.format(final[(final['test']==0) & (final['is_signed_up']==1)]
.shape[0]/final[final['test']==0].shape[0]*100))
print('Signup rate for test group: {}'.format(final[(final['test']==1) & (final['is_signed_up']==1)]
.shape[0]/final[final['test']==1].shape[0]*100))<jupyter_output>Signup rate for test group: 24.21328231206888<jupyter_text>Is the experiment already significant?The null hypothesis is the same as above, that the new email campaign increased the rate of signups.  Here, we see that the p value is below 0.05, so we can accept the null hypothesis and that the new email campaign increase signups.<jupyter_code>sample_control_signedup = np.random.choice(final[(final['test']==0)]['is_signed_up'], size = 1000, replace = True)
sample_test_signedup = np.random.choice(final[(final['test']==1)]['is_signed_up'], size = 1000, replace = False)
print(ttest_ind(sample_control_signedup, sample_test_signedup, equal_var=False))<jupyter_output>Ttest_indResult(statistic=-0.70089365064004527, pvalue=0.48345103914405185)<jupyter_text>Create a visualization to tell the story of the effect of the new emailFirst, I'd like to look at the time elapsed when the person first receives the email and signs up.  This time, the elapsed time is 4 days for the control group and 6 days for the test group.<jupyter_code>final['time_elapsed'] = final[(final['is_signed_up']) & (final['signup_date'].notnull())][['treatment_date', 'signup_date']].apply(time_to_signup, axis = 1)
print('Average signup time for control group: {}'.format(final[final['test']==0]['time_elapsed'].mean()))
print('Average signup time for test group: {}'.format(final[final['test']==1]['time_elapsed'].mean()))<jupyter_output>Average signup time for control group: 4.008775684931507
Average signup time for test group: 6.043876065710127<jupyter_text>Below is a distribution for how long it took users to sign up.  The plot is similar to the two weeks version, with a heavy right skew and peaks at 2 and 3 days for control and test groups respectively.  But the tail extends for the duration of the testing period (38 day instead of 13), increasing the average by 2-3 days.  **Overall it seems that the new email campaign works, just the signup rate is slower than expected.**<jupyter_code>final['days_to_signup'] = (final['signup_date'] - final['treatment_date']).dt.days
sns.countplot(x = 'days_to_signup', hue = 'test', data = final)
plt.xticks(rotation = 90)
plt.title('Distribution of How Long it Took to Signup First Four Weeks')
plt.xlabel('Days to Signup')
plt.ylabel('Count')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1271995228153892, 'fraction_numerical': 0.04249925439904563, 'mean_word_length': 6.595696489241223, 'pattern_counts': {'":': 0, '<': 39, '<?xml version=': 0, '>': 39, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '597663', 'license': 'MIT', 'n_tokens_mistral': 2347, 'n_tokens_neox': 2134, 'n_words': 692, 'path': 'Exercises/Experimental_Design_and_AB_testing/5 - Guided Example.ipynb', 'repo_name': 'yscyang1/Thinkful_Data_Science', 'size': '6706'}