<jupyter_start><jupyter_text>The company of this exercise is a social network. They decided to add a feature called: Recommended Friends, i.e. they suggest people you may know. A data scientist has built a model to suggest 5 people to each user. These potential friends will be shown on the user newsfeed. At first, the model is tested just on a random subset of users to see how it performs compared to the newsfeed without the new feature. The test has been running for some time and your boss asks you to check the results. You are asked to check, for each user, the number of pages visited during their first session since the test started. If this number increased, the test is a success. Specifically, your boss wants to know: (1) Is the test winning? That is, should 100\% of the users see the Recommended Friends feature? (2) Is the test performing similarly for all user segments or are there differences among different segments? (3) If you identified segments that responded differently to the test, can you guess the reason? Would this change your point 1 conclusions?  Load the package would be used<jupyter_code>import pandas as pd
pd.set_option("display.max_columns", 10)
pd.set_option("display.width", 350)

from scipy import stats

import matplotlib.pyplot as plt
from matplotlib import rcParams
rcParams.update({"figure.autolayout": True})
import seaborn as sns
sns.set(style = "white")
sns.set(style = "whitegrid", color_codes = True)
user = pd.read_csv("../Datasets/Engagement_Test/user_table.csv")
test = pd.read_csv("../Datasets/Engagement_Test/test_table.csv")<jupyter_output><empty_output><jupyter_text>Look into data<jupyter_code>print(user.shape)
print(test.shape)

print(user.head)
print(test.head)

print(user.info)
print(test.info)

print(len(user["user_id"]) == len(pd.unique(user["user_id"])))
print(len(test["user_id"]) == len(pd.unique(test["user_id"])))<jupyter_output>(100000, 2)
(100000, 5)
<bound method NDFrame.head of        user_id signup_date
0           34  2015-01-01
1           59  2015-01-01
2          178  2015-01-01
3          285  2015-01-01
4          383  2015-01-01
...        ...         ...
99995  8999327  2015-08-31
99996  8999539  2015-08-31
99997  8999550  2015-08-31
99998  8999709  2015-08-31
99999  8999849  2015-08-31

[100000 rows x 2 columns]>
<bound method NDFrame.head of        user_id        date  browser  test  pages_visited
0       600597  2015-08-13       IE     0              2
1      4410028  2015-08-26   Chrome     1              5
2      6004777  2015-08-17   Chrome     0              8
3      5990330  2015-08-27   Safari     0              8
4      3622310  2015-08-07  Firefox     0              1
...        ...         ...      ...   ...            ...
99995  2698493  2015-08-21   Chrome     1              6
99996  3396864  2015-08-04   Chrome     0              5
99997  7507573  2015-08-06       IE     1          [...]<jupyter_text>Data processing<jupyter_code>dat = user.merge(test, on = "user_id", how = "inner")
dat["signup_date"] = pd.to_datetime(dat["signup_date"])
dat["date"] = pd.to_datetime(dat["date"])
dat.head()<jupyter_output><empty_output><jupyter_text>(1) Is the test winning? That is, should 100% of the users see the Recommended Friends feature? Overall Define the function<jupyter_code>def overall_ttest_mean(dat, variable, test):
    overall = stats.ttest_ind(dat[dat[test] == 1][variable], dat[dat[test] == 0][variable], equal_var = False)
    test_group = dat[dat[test] == 1][variable].mean()
    control_group = dat[dat[test] == 0][variable].mean()
    pvalue = overall.pvalue
    overall_result = {"test_group": test_group, "control_group": control_group, "pvalue": pvalue}
    return overall_result
overall_ttest_mean(dat = dat, variable = "pages_visited", test = "test")<jupyter_output><empty_output><jupyter_text>(2) Is the test performing similarly for all user segments or are there differences among different segments? Stratified test Define the function<jupyter_code>def stratified_ttest_mean(dat, stratified, variable, test):
    stratified_result = dat.groupby(stratified)[variable].agg({
        "test_group": lambda x: x[dat[test] == 1].mean(),
        "control_group": lambda x: x[dat[test] == 0].mean(),
        "p_value": lambda x: stats.ttest_ind(x[dat[test] == 1], x[dat[test] == 0], equal_var = False).pvalue
    }).reindex(["test_group", "control_group", "p_value"], axis = 1)
    return stratified_result.sort_values(by = "p_value")<jupyter_output><empty_output><jupyter_text>Stratified by browser<jupyter_code>stratified_ttest_mean(dat = dat, stratified = "browser", variable = "pages_visited", test = "test").reset_index()<jupyter_output><ipython-input-7-063b2af17d00>:2: FutureWarning: using a dict on a Series for aggregation
is deprecated and will be removed in a future version. Use                 named aggregation instead.

    >>> grouper.agg(name_1=func_1, name_2=func_2)

  stratified_result = dat.groupby(stratified)[variable].agg({<jupyter_text>Stratified by date<jupyter_code>stratified_ttest_mean(dat = dat, stratified = "date", variable = "pages_visited", test = "test").reset_index().sort_values(by = "date")<jupyter_output><ipython-input-7-063b2af17d00>:2: FutureWarning: using a dict on a Series for aggregation
is deprecated and will be removed in a future version. Use                 named aggregation instead.

    >>> grouper.agg(name_1=func_1, name_2=func_2)

  stratified_result = dat.groupby(stratified)[variable].agg({<jupyter_text>(3) If you identified segments that responded differently to the test, can you guess the reason? Would this change your point 1 conclusions?<jupyter_code>overall_ttest_mean(dat = dat[dat["browser"] != "Opera"], variable = "pages_visited", test = "test")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10939934221914488, 'fraction_numerical': 0.0675090877618141, 'mean_word_length': 3.8069883527454245, 'pattern_counts': {'":': 7, '<': 34, '<?xml version=': 0, '>': 39, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '36460', 'license': 'MIT', 'n_tokens_mistral': 2067, 'n_tokens_neox': 1836, 'n_words': 557, 'path': '17.Engagement Test.ipynb', 'repo_name': 'Veronica0206/DS_Project_Sample', 'size': '5777'}