<jupyter_start><jupyter_text>Spanish Translation AB Test  Import packages and read data<jupyter_code>#--- import packages
import pandas as pd
import pandas_profiling
import os
from scipy.stats import ttest_ind
from sklearn import tree
import plotly
import graphviz
import plotly.plotly as py
import plotly.graph_objs as go
from plotly import tools
from plotly.offline import init_notebook_mode
init_notebook_mode(connected=True)
#--- read data
PATH = "/home/billzhan/OneDrive/Academic/Open/Lai/take home challenge/spanish_translation"
test_data = pd.read_csv(os.path.join(PATH,"test_table.csv"))
user_data = pd.read_csv(os.path.join(PATH,"user_table.csv"))
#--- check unique users

# test data
test_data['user_id'].nunique() == len(test_data['user_id'])
# user data
user_data['user_id'].nunique() == len(user_data['user_id'])
# Are they matched?
test_data['user_id'].size - user_data['user_id'].size<jupyter_output><empty_output><jupyter_text>Looks like the user table is busted and we have some user ids missing. When joining, we have to becareful to do not lose the user ids in the test table, but not in the user table.<jupyter_code>#--- merge two data
data = pd.merge(test_data, user_data, on='user_id', how='left')
data['date'] = pd.to_datetime(data['date'])
pandas_profiling.ProfileReport(data)<jupyter_output><empty_output><jupyter_text>First check whether Spanish convert better than others<jupyter_code>#--- fill missing countries
data['country'] = data['country'].fillna('no_country')
#--- average conversion rate
data_conversion_country = data[data['test']==0].groupby(['country'],as_index=False)['conversion'].mean()
data_conversion_country.sort_values(by=['conversion'],ascending=False).head()<jupyter_output><empty_output><jupyter_text>Do t test to see if the two groups (no spanish) has difference<jupyter_code>#--- split data set
data_t = data[data['country']!='Spain']

#--- t test
data_test = data_t.loc[data_t['test']==1,'conversion']
data_control = data_t.loc[data_t['test']==0,'conversion']
ttest_ind(data_test,data_control)
#--- mean of two sets
print(data_test.mean(),data_control.mean())<jupyter_output>0.043424713982118966 0.04833042316066309<jupyter_text>Mmh…not in the test are converting at 4.8% while users in the test just at 4.3%. That’s a 10% drop,which would be dramatic if it were true. The most likely reason for weird A/B test results are:1. We didn’t collect enough data.2. Some bias has been introduced in the experiment so that test/control people are not reallyrandom  Plot day by day the conversion rate<jupyter_code>#--- data by day
data_t_by_day = data_t.groupby(['date','test'],as_index=False)['conversion'].mean()
data_t_by_day = data_t_by_day.pivot(index='date',columns='test',values='conversion')
data_t_by_day['ratio'] = data_t_by_day[1]/data_t_by_day[0]
data_t_by_day.reset_index(inplace=True)
#--- make plot
#--- visualization
test_vs_control_bydate_toplot = [go.Scatter(
            x = data_t_by_day['date'],
            y = data_t_by_day['ratio']
    )]
bydate_layout = go.Layout(
    title='<b>Test Control conversion rate by date</b>',
    xaxis=dict(title='Date'),
    yaxis=dict(title='test_vs_control')
)
#--- make plot
test_vs_control = go.Figure(data=test_vs_control_bydate_toplot, layout=bydate_layout)
plotly.offline.iplot(test_vs_control)<jupyter_output><empty_output><jupyter_text>From the plot, we notice a couple of things:1. Test has constantly been worse than control and there is relatively little variance across days. Thatprobably means that we do have enough data, but there was some bias in the experiment set up.2. On a side note, we just ran it for 5 days. We should always run the test for at least 1 full week tocapture weekly patterns, 2 weeks would be much better Time to find out the bias! Likely, there is for some reason some segment of users more likely to end up intest or in control, this segment had a significantly above/below conversion rate and this affected the overallresults.In an ideal world, the distribution of people in test and control for each segment should be the same. Thereare many ways to check this. One way is to build a decision tree where the variables are the userdimensions and the outcome variable is whether the user is in test or control. If the tree splits, it means that for given values of that variable you are more likely to end up in test or control. But this should be impossible! Therefore, if the randomization worked, the tree should not split at all (or at least not be able to separate the two classes well.  Build decision tree to check split<jupyter_code>data_t['date'] = data_t['date'].astype('object')
cate_cols = [col for col in data_t.columns if data_t[col].dtype==object]
cate_cols
one_hot_data = pd.get_dummies(data_t[cate_cols])
one_hot_data
clf = tree.DecisionTreeClassifier()
data_tree = pd.concat([data_t[data_t.columns.difference(cate_cols)],one_hot_data],axis=1)

clf = clf.fit(data_tree[data_tree.columns.difference(['conversion','test'])], data_tree['test'])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10739856801909307, 'fraction_numerical': 0.011137629276054098, 'mean_word_length': 6.0139470013947, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '113072', 'license': 'MIT', 'n_tokens_mistral': 1640, 'n_tokens_neox': 1579, 'n_words': 508, 'path': 'spanish_abtest.ipynb', 'repo_name': 'Bill-Zhan/DS-Take-Home', 'size': '5028'}