<jupyter_start><jupyter_text>Problem DefinitionSpanish Translation A/B TestGoal:1. Confirm that the test is actually negative. That is, it appears that the old version of the site with just one translation across Spain and LatAm performs better2. Explain why that might be happening. Are the localized translations really worse?3. If you identified what was wrong, design an algorithm that would return FALSE if the same problem is happening in the future and TRUE if everything is good and the results can be trusted.<jupyter_code>import  warnings
warnings.simplefilter('ignore')

import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
import matplotlib
%matplotlib inline
from scipy.stats import ttest_ind
matplotlib.style.use('ggplot')

%matplotlib inline

from scipy.stats import norm
test = pd.read_csv('data/test_table.csv', parse_dates=['date'])
test.head()
user = pd.read_csv('data/user_table.csv')
user.head()
# there are some missing value in the user table.
test.shape,user.shape
print('Variable', '   ', 'count of unique value','  ', 'content')
for column in test.columns:
    uniques = test[column].unique()
    print('{0:20s} {1:4d}\t'.format(column, len(uniques)), uniques[:7])
print('Variable', '   ', 'count of unique value','  ', 'content')
for column in user.columns:
    uniques = sorted(user[column].unique())
    print('{0:20s} {1:4d}\t'.format(column, len(uniques)), uniques[:7])<jupyter_output>Variable     count of unique value    content
user_id              452867	 [1, 2, 3, 5, 8, 11, 12]
sex                     2	 ['F', 'M']
age                    52	 [18, 19, 20, 21, 22, 23, 24]
country                17	 ['Argentina', 'Bolivia', 'Chile', 'Colombia', 'Costa Rica', 'Ecuador', 'El Salvador']<jupyter_text>Check two tables if there exists repeated users.<jupyter_code># Check uniqueness

print('Test table Unique User Ids:\t', len(test['user_id'].unique()),'\t',
      len(test['user_id'].unique())==len(test['user_id']) )
print('User table Unique User Ids:\t', len(user['user_id'].unique()),'\t',
      len(user['user_id'].unique())==len(user['user_id']) )<jupyter_output>Test table Unique User Ids:	 453321 	 True
User table Unique User Ids:	 452867 	 True<jupyter_text>It is clear that two tables do not have repeated users and for some users in the test set, we miss their information about sex, age and country. Thus, when joining, we have to guarantee the compeleteness of user ids in the test table. I am planning to let user table to left join test table.<jupyter_code>data = pd.merge(left=test, right=user, how='left', on='user_id' )
data.info()
data.columns
data.head()<jupyter_output><empty_output><jupyter_text>A/B TestFirst check that Spain converts much better than the rest of LatAm countries 1. Confirm that the test is actually negative.First, we need to prove that in the control group, spain performs better.<jupyter_code>groupby_country = data[data['test'] == 0][['conversion', 'country']].groupby('country').mean()
groupby_country = groupby_country.reset_index()
groupby_country = groupby_country.sort_values('conversion', ascending=False)
# Visualization
fig, ax = plt.subplots(figsize=(18, 6))
sns.barplot(x='country', y='conversion', data=groupby_country, ax=ax)
plt.show()
# Visualization
fig, ax = plt.subplots(figsize=(18, 6))
sns.barplot(x='country', y='conversion', hue='test', data=data, ax=ax)
plt.show()
#From the barplot above, it seems that the control groups perform worse. We need some statistal methods to prove it.
#Obviously, Spain does have the highest mean conversion rate among all countries. Now I am trying to confirm that the test is negative by using t test.
# A/B test
test_data = data[data['country'] != 'Spain']
test_val = test_data[test_data['test'] == 1]['conversion'].values
cont_val = test_data[test_data['test'] == 0]['conversion'].values
print(test_val.mean(), '\t', cont_val.mean())

# Welch Two Sample t-test
print(ttest_ind(test_val, cont_val, equal_var=False))<jupyter_output>0.043424713982118966 	 0.04833042316066309
Ttest_indResult(statistic=-7.3939374121344805, pvalue=1.4282994754055316e-13)<jupyter_text>We can see that control group without changing translation has a converting at 4.8%, however, users in test group with localized translation just convert at 4.3%. It does have a negative effect for changing translation.  Explain why that might be happening. Are the localized translations really worse?In order to answer this question, I will first visualize multiple variables by breaking data into control and test group to give me some insights.Likely, there is for some reason some segment of users more likely to end up in test or in control, this segment had a significantly above/below conversion rate and this affected the overall results.<jupyter_code>data = data[data['country'] != 'Spain']
data.head()
Cont=data[data['test']==0]
Test=data[data['test']==1]
def visualization(data1,data2,nrows,ncols):
    columns = data.select_dtypes(include=['object']).columns[1:7]
    print('-' * 32,'Meansure each variable performance on control and test group','-' * 32)
    print(' ' * 100)
    for column in columns:
        # Source variable visualization on control and test group
        fig, ax = plt.subplots(nrows,ncols,figsize=(18,6))
        sns.countplot(data1[column],ax=ax[0,0],order=sorted(data1[column].value_counts().index))
        sns.countplot(x=data1[column],hue=data1['conversion'],ax=ax[0,1],order=sorted(data1[column].value_counts().index))
        sns.barplot(data1['conversion'],data1[column],ax=ax[0,2],order=sorted(data1[column].value_counts().index))
        sns.countplot(data2[column],ax=ax[1,0],order=sorted(data2[column].value_counts().index))
        sns.countplot(x=data2[column],hue=data2['conversion'],ax=ax[1,1],order=sorted(data2[column].value_counts().index))
        sns.barplot(data2['conversion'],data2[column],ax=ax[1,2],order=sorted(data2[column].value_counts().index))
        fig.subplots_adjust(top=0.2)
        fig.suptitle('Control vs Test',fontsize=14)
        plt.tight_layout()
visualization(Cont,Test,2,3)

#  Age variable visualization
fig,ax=plt.subplots(2,1,figsize=(20,7))
sns.countplot(x='age',hue='test',data=data,ax=ax[0])
sns.barplot(x=data['age'],y=data['conversion'],hue=data['test'],ax=ax[1]);
fig.suptitle('Count plot of age',fontsize=16)
fig.subplots_adjust(top=0.25)
plt.tight_layout()
# Visualization of different devices
fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(18, 6))
sns.countplot(x='country', hue='test', data=data, ax=ax[0])
ax[0].set_title('Count Plot of Country', fontsize=16)

sns.barplot(x='country', y='conversion', hue='test', data=data, ax=ax[1])
ax[1].set_title('Mean Conversion Rate per Country', fontsize=16)
plt.tight_layout()
plt.show()
# let check for each country

countries = [name for name in data['country'].unique() if name is not np.nan]

print('{0:15s} {1:>15s} {2:>15s} {3:>10s}'.format('Country', 'Test Rate', 'Control Rate', 'P-Value'))
print('-' * 65)
for country in countries:
    test_val = data[(data['country'] == country) & (data['test'] == 1)]['conversion'].values
    cont_val = data[(data['country'] == country) & (data['test'] == 0)]['conversion'].values
    
    test_mean = test_val.mean()
    cont_mean = cont_val.mean()
    p_val = ttest_ind(test_val, cont_val, equal_var=False).pvalue
    
    print('{0:15s} {1:15.5f} {2:15.5f} {3:10f}'.format(country, test_mean, cont_mean, p_val))<jupyter_output>Country               Test Rate    Control Rate    P-Value
-----------------------------------------------------------------
Mexico                  0.05119         0.04949   0.165544
Venezuela               0.04898         0.05034   0.573702
Bolivia                 0.04790         0.04937   0.718885
Colombia                0.05057         0.05209   0.423719
Uruguay                 0.01291         0.01205   0.879764
El Salvador             0.04795         0.05355   0.248127
Nicaragua               0.05418         0.05265   0.780400
Peru                    0.05060         0.04991   0.771953
Costa Rica              0.05474         0.05226   0.687876
Chile                   0.05130         0.04811   0.302848
Argentina               0.01373         0.01507   0.335147
Ecuador                 0.04899         0.04915   0.961512
Guatemala               0.04865         0.05064   0.572107
Honduras                0.04754         0.05091   0.471463
Paraguay                0.04923         0.04849  [...]
{'fraction_non_alphanumeric': 0.14017255643540952, 'fraction_numerical': 0.06275853917976598, 'mean_word_length': 3.8941584731058416, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 20, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '475105', 'license': 'Apache-2.0', 'n_tokens_mistral': 3058, 'n_tokens_neox': 2721, 'n_words': 749, 'path': 'Translation_AB_Test/TranslationTest.ipynb', 'repo_name': 'JingyuZHANG/DataScience_Python', 'size': '8461'}