<jupyter_start><jupyter_text>Summary **Note: This notebook is a draft (and I can't promise when I'll fix that), so it may be incomplete in places. If you have questions, feel free to ask on [my talk page](https://meta.wikimedia.org/wiki/User_talk:Neil_P._Quinn-WMF).**  History First a little history. This analysis follows up on the work done by <NAME> in his [May 2015 visual editor experiment](https://meta.wikimedia.org/wiki/Research:VisualEditor's_effect_on_newly_registered_editors/May_2015_study). In that study, all the new accounts created on the English Wikipedia during one week [were divided evenly](https://meta.wikimedia.org/wiki/Research_talk:VisualEditor's_effect_on_newly_registered_editors/Work_log/2015-06-05) between a treatment group which was automatically offered the new visual editor alongside the existing wikitext editor and control group which was not.That experiment helped win the English Wikipedia community's consent to start giving the visual editor to new users automatically. In turn, I am expanding Aaron's original cohort by looking at data for the incremental rollout to new editors. We started by giving the visual editor to 5% of new users, then 10%, then 25%, and so on up to 100%, where the level remains. Since we applied each percentage randomly, we can incorporate some of those users into the treatment and control groups. In addition, Aaron investigated the users for a week after their registration. I am expanding that to 60 days after their registration.When I originally performed the follow-up with the original cohort, most of the results were fairly ho-hum, but I did find one intriguing effect: the visual editor group had a higher proportion of accounts who survived to make at least one edit more than 30 days after editing.  Issues The revisions dataset includes all the revisions made by the cohort from the time of the first signup to sixty days after the last. That means some cohort members have more time to make revisions than others. I take care of this for the survival analysis, but I should do so for whole thing.  CohortIn all of these stages, we determined whether to offer the user the visual editor by checking the modulus of their user ID ([code for the experiment](https://gerrit.wikimedia.org/r//c/68845/4/VisualEditor.hooks.php), [code for the incremental rollout](https://github.com/wikimedia/mediawiki-extensions-VisualEditor/blob/b550323b534b59622afefdf9bf9fbc23f177f46f/VisualEditor.hooks.phpL766)). In both cases, when the VE was being offered to users at x% = (1 / y ), a user got VE if ( userID % y = 0 ). Where the VE users and control users were not evenly matched, our matched control will be users where ( userID % y  = 1).Our groups will *not* be perfectly matched, because many of the new accounts generated at the English Wikipedia are autocreated when users with accounts on other Wikimedia projects browse over to the English Wikipedia, not when users on the English Wikipedia click the "create account" button. Our cohort only includes the second group, because users who already have accounts on other projects are likely to behave quite differently.This means that our list of newly created accounts will have random gaps where autocreated accounts used up the IDs. Since they're distributed randomly, this shouldn't affect our analysis.The detailed schedule of these stages is [here](https://www.mediawiki.org/w/index.php?title=VisualEditor/Rollouts/English_Wikipedia&oldid=1934590). First, we create a table to hold the users.```CREATE TABLE `ve_experiment_expanded` (  `user_id` INT NOT NULL,  `registration` VARBINARY(14) NOT NULL,  `bucket` ENUM("treatment", "control") NOT NULL,  `mobile_registered` BOOLEAN DEFAULT NULL,  PRIMARY KEY (`user_id`));``` Pilot ```INSERT INTO staging.ve_experiment_expandedSELECT     event_userId AS user_id,     timestamp AS registration,    CASE        WHEN event_userId % 2 = 0 THEN "treatment"        WHEN event_userId % 2 = 1 THEN "control"    END AS bucket,    IF( ( event_displayMobile OR NOT event_isSelfMade ), 1, 0 ) AS mobile_registeredFROM ServerSideAccountCreation_5487345WHERE    timestamp >= "201505211500" AND    timestamp <  "201505221501" AND    wiki = "enwiki" AND    ( event_userId % 2 =  0 || event_userId % 2 = 1 );``` Experiment ```INSERT INTO staging.ve_experiment_expandedSELECT     event_userId AS user_id,     timestamp AS registration,    CASE        WHEN event_userId % 2 = 0 THEN "treatment"        WHEN event_userId % 2 = 1 THEN "control"    END AS bucket,    IF( ( event_displayMobile OR NOT event_isSelfMade ), 1, 0 ) AS mobile_registeredFROM ServerSideAccountCreation_5487345WHERE    timestamp >= "201505282304" AND    timestamp <  "201506042328" AND    wiki = "enwiki" AND    ( event_userId % 2 =  0 || event_userId % 2 = 1 );``` 5% rollout ```INSERT INTO staging.ve_experiment_expandedSELECT     event_userId AS user_id,     timestamp AS registration,    CASE        WHEN event_userId % 20 = 0 THEN "treatment"        WHEN event_userId % 20 = 1 THEN "control"    END AS bucket,    IF( ( event_displayMobile OR NOT event_isSelfMade ), 1, 0 ) AS mobile_registeredFROM ServerSideAccountCreation_5487345WHERE    timestamp >= "201507281503" AND    timestamp <  "201508041513" AND    wiki = "enwiki" AND    ( event_userId % 20 =  0 || event_userId % 20 = 1 );``` 10% rollout ```INSERT INTO staging.ve_experiment_expandedSELECT     event_userId AS user_id,     timestamp AS registration,    CASE        WHEN event_userId % 10 = 0 THEN "treatment"        WHEN event_userId % 10 = 1 THEN "control"    END AS bucket,    IF( ( event_displayMobile OR NOT event_isSelfMade ), 1, 0 ) AS mobile_registeredFROM ServerSideAccountCreation_5487345WHERE    timestamp >= "201508041513" AND    timestamp <  "201508132303" AND    wiki = "enwiki" AND    ( event_userId % 10 =  0 || event_userId % 10 = 1 );``` 25% rollout ```INSERT INTO staging.ve_experiment_expandedSELECT     event_userId AS user_id,     timestamp AS registration,    CASE        WHEN event_userId % 4 = 0 THEN "treatment"        WHEN event_userId % 4 = 1 THEN "control"    END AS bucket,    IF( ( event_displayMobile OR NOT event_isSelfMade ), 1, 0 ) AS mobile_registeredFROM ServerSideAccountCreation_5487345WHERE    timestamp >= "201508132303" AND    timestamp <  "201508201525" AND    wiki = "enwiki" AND    ( event_userId % 4 =  0 || event_userId % 4 = 1 );``` 50% rollout ```INSERT INTO staging.ve_experiment_expandedSELECT     event_userId AS user_id,     timestamp AS registration,    CASE        WHEN event_userId % 2 = 0 THEN "treatment"        WHEN event_userId % 2 = 1 THEN "control"    END AS bucket,    IF( ( event_displayMobile OR NOT event_isSelfMade ), 1, 0 ) AS mobile_registeredFROM ServerSideAccountCreation_5487345WHERE    timestamp >= "201508201525" AND    timestamp <  "201509011503" AND    wiki = "enwiki" AND    ( event_userId % 2 =  0 || event_userId % 2 = 1 );```<jupyter_code>%matplotlib inline

import pandas as pd
import scipy.stats as stats
import matplotlib.pyplot as plt
import math
import datetime as dt<jupyter_output><empty_output><jupyter_text>Data ```mysql -h analytics-store.eqiad.wmnet -u research -e "SELECT    user_id,    registration,    bucket,    mobile_registeredFROM staging.ve_experiment_expanded;" > ~/VE_experiment_cohort.tsv```Then, I generate a dataset of all the edits made by accounts from the experiment in our time period. This is based on [Aaron's code](https://github.com/halfak/VE_newcomers_May_2015/blob/master/sql/experimental_user_revision_stats.sql).```mysql -h analytics-store.eqiad.wmnet -u research -e "SELECT    user_id,    current.rev_id AS rev_id,    current.rev_timestamp AS rev_timestamp,    ct_tag IS NOT NULL AS ve_used,    CAST(current.rev_len AS INT) -        CAST(IFNULL(parent.rev_len, 0) AS INT) as change_in_bytesFROM staging.ve_experiment_expandedINNER JOIN enwiki.revision current    ON user_id = current.rev_userLEFT JOIN enwiki.revision parent    ON parent.rev_id = current.rev_parent_idLEFT JOIN enwiki.change_tag      ON ct_rev_id = current.rev_id AND    ct_tag = 'visualeditor'WHERE    current.rev_timestamp BETWEEN       '20150521150000' AND        DATE_FORMAT( DATE_ADD( '20150901150300', INTERVAL 60 DAY ), '%Y%m%d%H%i%S' ) AND    current.rev_timestamp BETWEEN         registration AND        DATE_FORMAT( DATE_ADD( registration, INTERVAL 60 DAY ), '%Y%m%d%H%i%S' );" > ~/VE_experiment_revs.tsv```The where clause limits it to revisions made after the first registration but within 60 days after the last. Load and clean the data.<jupyter_code>user_dtypes = {
    "user_id": int, 
    "registration": str, 
    "bucket": str,
    "mobile_registered": bool
}

rev_dtypes = {
    "user_id": int, 
    "rev_id": int, 
    "rev_timestamp": str,
    "ve_used": bool,
    "change_in_bytes": int
}

users = pd.read_table( "./VE_experiment_cohort.tsv", dtype = user_dtypes )
revs = pd.read_table( "./VE_experiment_revs.tsv", dtype = rev_dtypes)

# Make timestamps be actual datetimes
revs["rev_timestamp"] = pd.to_datetime( revs["rev_timestamp"], format="%Y%m%d%H%M%S" )
users["registration"] = pd.to_datetime( users["registration"], format="%Y%m%d%H%M%S" )

# Limit to revisions made within 60 days of that user's registration
window = dt.timedelta( days = 60 )
revs = pd.merge(revs, users, on = "user_id", how = "left" )
revs = revs[ revs["rev_timestamp"] < ( revs["registration"] + window ) ]

# Add the number of revisions to the users frame
revs_by_user = revs.groupby( "user_id" )[ "rev_id" ].size().to_frame().reset_index()
revs_by_user.rename( columns = { 0: "revs" }, inplace = True )
users = pd.merge( users, revs_by_user, on = "user_id", how = "left" )
users.fillna( 0, inplace = True )<jupyter_output><empty_output><jupyter_text>We're not interested in people who primarily use Wikipedia on mobile devices, because the visual editor is only widely available on desktop computers. Unfortunately, there's no simple ways to tell if someone is a mobile only user, but from past experience, registering the account from a mobile device is a pretty good proxy. So we'll exclude users who signed up using an app or the mobile web interface.<jupyter_code>users = users[ users["mobile_registered"] == False ]
revs = revs[ revs["mobile_registered"] == False ]
users.tail()
revs.tail()<jupyter_output><empty_output><jupyter_text>In addition, for some analyses we'll want to filter out users who never edited at all.<jupyter_code>editors = users[users["revs"] > 0]<jupyter_output><empty_output><jupyter_text>Utility methods<jupyter_code>def split_buckets(df):
    df.treatment = df[df["bucket"] == "treatment"]
    df.control = df[df["bucket"] == "control"]

def verbose_chisquared( name_1, n_1, k_1, name_2, n_2, k_2 ):
    print( name_1, "n:", n_1 )
    print( name_1, "k:", k_1 )
    print( name_1, "rate:", k_1 / n_1 )
    print( "-----" )
    
    print( name_2, "n:", n_2 )
    print( name_2, "k:", k_2 )
    print( name_2, "rate:", k_2 / n_2 )
    print( "-----" )
    
    combined_rate = ( k_1 + k_2 ) / ( n_1 + n_2 )
    print( stats.chisquare( [k_1, k_2 ], [ n_1 * combined_rate, n_2 * combined_rate] ) )
    
def verbose_mannwhitneyu( name_1, series_1, name_2, series_2 ):
    print( name_1, "average:", round( series_1.mean(), 3 ) )
    print( name_2, "average:", round( series_2.mean(), 3 ) )
    print( stats.mannwhitneyu( series_1, series_2 ) )<jupyter_output><empty_output><jupyter_text>Analysis<jupyter_code>split_buckets(users)
split_buckets(editors)

n_control = len( users.control )
n_treatment = len( users.treatment )

print(n_control)
print(n_treatment)<jupyter_output>38493
38365<jupyter_text>Productivity None of the productivity variables in Aaron's original study showed anything close to a significant effect, so I'll just re-test the simplest two: (1) the total number of edits per user and (2) the proportion of users who made at least one edit.<jupyter_code>print(users.control["revs"].sum())
print(users.treatment["revs"].sum())<jupyter_output>113260.0
104963.0<jupyter_text>For edits per user, I use the [Mann-Whitney U test](https://en.wikipedia.org/wiki/Mann–Whitney_U_test), also know as the Wilcoxon ranked-sum test.<jupyter_code>verbose_mannwhitneyu( "treatment", users.treatment["revs"], "control", users.control["revs"] )<jupyter_output>treatment average: 2.736
control average: 2.942
MannwhitneyuResult(statistic=737747609.5, pvalue=0.41702303223262732)<jupyter_text>So p = 0.841 (the function returns a one-sided p value). No significance.<jupyter_code>fig, axarr = plt.subplots( 1, 1, figsize=(20,10) )
plt.hist(users.control["revs"], alpha = 0.5, range = (0, 15), log = True, bins = 15, label = "Source editor only")
plt.hist(users.treatment["revs"], alpha = 0.5, range = (0, 15), log = True, bins = 15, label = "Both editors")
plt.legend(loc="upper right")
plt.show()
fig, axarr = plt.subplots( 1, 1, figsize=(20,10) )
plt.hist(editors.control["revs"], alpha = 0.5, range = (1, 21), normed = 1, bins = 20, label = "Source editor only")
plt.hist(editors.treatment["revs"], alpha = 0.5, range = (1, 21), normed = 1, bins = 20, label = "Both editors")
plt.legend(loc="upper right")
plt.show()<jupyter_output><empty_output><jupyter_text>For the proportion who edited at least once, I use [Pearson's chi-squared test](https://en.wikipedia.org/wiki/Pearson's_chi-squared_test).<jupyter_code>verbose_chisquared(
    "treatment",
    n_treatment,
    len(editors.treatment),
    "control",
    n_control,
    len(editors.control)
)<jupyter_output>treatment n: 38365
treatment k: 13343
treatment rate: 0.34779095529779747
-----
control n: 38493
control k: 13126
control rate: 0.3409970644013197
-----
Power_divergenceResult(statistic=2.5752332123793669, pvalue=0.10854810298815173)<jupyter_text>p = 0.120. Not significant, but fairly low. However, the actual difference in editing proportion (35.2% with visual editor vs 34.5% without) isn't really anything to write home about.  Block rateFirst, let me see whether [Aaron's block-categorizing regexes](https://meta.wikimedia.org/wiki/Research_talk:VisualEditor's_effect_on_newly_registered_editors/Work_log/2015-05-28Block_rates_and_types) cover all the blocks on these expanded set of users.```SELECT 	user_name, 	log_commentFROM enwiki.logging INNER JOIN enwiki.user 	ON REPLACE(log_title, "_", " ") = user_name INNER JOIN staging.ve_experiment_expanded	USING (user_id) WHERE	log_type = "block" AND	log_action = "block" AND	log_timestamp BETWEEN "201505211500" AND "201509011503" AND	log_comment NOT RLIKE "(S|s)pam" AND	log_comment NOT RLIKE "((V|v)and|(D|d)isrupt|(U|u)w-vaublock|(A|a)bus(e|ing)|(A|a)ttack|(D|d)eliberate|NOTHERE)" AND	log_comment NOT RLIKE "((S|s)ock|(C|c)heckuser|(E|e)vasion)" AND	log_comment NOT RLIKE "softerblock|soft block" AND	log_comment NOT RLIKE "(U|u)w-uhblock|user\.\.\.|(U|u)w-ublock";``````169 rows affected; taking 27.5 s```Okay, so it needs a little more work.```SELECT 	user_name, 	log_commentFROM enwiki.logging INNER JOIN enwiki.user 	ON REPLACE(log_title, "_", " ") = user_name INNER JOIN staging.ve_experiment_expanded	USING (user_id) WHERElog_type = "block" ANDlog_action = "block" ANDlog_timestamp BETWEEN "201505211500" AND "201509011503" ANDlog_comment NOT RLIKE  Hard username blocks    "(U|u)w-u(h)?block|user..."      ANDlog_comment NOT RLIKE  Soft username blocks    "(U|u)ser(\\s)?name|soft(er|est)?(\\s)?block|botublock|causeblock|organization|one person"      ANDlog_comment NOT RLIKE  Spam    "(S|s)pam|(A|a)dvertising|(P|p)romotional"     ANDlog_comment NOT RLIKE  Obvious abuse    "(V|v)and|(D|d)isrupt|(U|u)w-vaublock|(A|a)bus(e|ing)|(A|a)ttack|(D|d)eliberate|NOTHERE|not here|LTA|(T|t)roll|(H|h)arassment|(P|p)atent nonsense|(I|i)nsult"      ANDlog_comment NOT RLIKE  Policy violation	"(C|c)opyright|(B|b)iographies|BLP|POV|(W|w)arring|(T|t)hree-revert|(U|u)nsourced|(L|l)egal threat"    ANDlog_comment NOT RLIKE  Sockpuppetry	"SOCK|(S|s)ock|(C|c)heck(U|u)ser|(E|e)va(de|sion)|(M|m)eatpuppet";``````39 rows affected, taking 13.0 s```That's better; the only blocks left are ones whose comments are miscellaneous ("Persistent use of Wikipedia as a social media network", "As per [[User:CURRENT RESEARCH IN CHEMISTRY AND PHARMACEUTICAL SCIENCES-INTERNATIONAL JOURNAL]]", "You have been blocked for testing purposes"), nonspecific ("Please see your talk page for a full explanation", "so many possible reasons; pick your favourite!", "No thanks."), or empty. I'm not going to include the full list because some of the usernames are thoroughly toxic.*What about accounts blocked multiple times? for example, https://en.wikipedia.org/w/index.php?title=Special:Log/block&page=Majorwith?*Some block comments match multiple regexes (e.g. `{{uw-spamublock}} : ` would match both spam and username regexes. However in the select statement, the block will match at most one—preferring the first in the query. Now, let's get all these blocks categorized as above.```mysql -h analytics-store.eqiad.wmnet -u research -e "SELECT	user_id,	bucket,	IF( log_comment RLIKE   		"(U|u)w-u(h)?block|user...",    		"hard username",	IF( log_comment RLIKE    	"(U|u)ser(\\s)?name|soft(er|est)?(\\s)?block|botublock|causeblock|organization|one person",    	"soft username",    IF( log_comment RLIKE 		"(S|s)pam|(A|a)dvertising|(P|p)romotional",		"spam",	IF( log_comment RLIKE    	"(V|v)and|(D|d)isrupt|(U|u)w-vaublock|(A|a)bus(e|ing)|(A|a)ttack|(D|d)eliberate|NOTHERE|not here|LTA|(T|t)roll|(H|h)arassment|(P|p)atent nonsense|(I|i)nsult",    	"abuse",     IF( log_comment RLIKE    	"(C|c)opyright|(B|b)iographies|BLP|POV|(W|w)arring|(T|t)hree-revert|(U|u)nsourced|(L|l)egal threat",		"policy violation",	IF( log_comment RLIKE		"SOCK|(S|s)ock|(C|c)heck(U|u)ser|(E|e)va(de|sion)|(M|m)eatpuppet",		"sockpuppetry",	"other"	) ) ) ) ) ) AS "block reason"FROM enwiki.logging INNER JOIN enwiki.user 	ON REPLACE(log_title, "_", " ") = user_name INNER JOIN staging.ve_experiment_expanded	USING (user_id)WHERE 	log_type = "block" AND 	log_action = "block" AND 	log_timestamp BETWEEN "201505211500" AND "201509011503";" > ~/VE_experiment_block.tsv```<jupyter_code>blocks = pd.read_table( "./VE_experiment_blocks.tsv" )

blocks.tail()
blocks_by_user = blocks.groupby( "user_id" ).size().to_frame().reset_index()

blocks_by_user.rename( columns = { 0: "blocked" }, inplace = True )

users = pd.merge( users, blocks_by_user, on="user_id", how="left" )

users[ "blocked" ].fillna( 0, inplace = True )

print( users.groupby( "blocked" ).size() )

print( blocks.groupby( "block reason" ).size() )<jupyter_output>blocked
0.0    73485
1.0     3362
2.0       10
3.0        1
dtype: int64
block reason
abuse               1399
hard username       1482
other                 39
policy violation      49
sockpuppetry         230
soft username        219
spam                 652
dtype: int64<jupyter_text>Only 11 users were blocked multiple times, so it seems fair to just reduce it to a boolean reflecting whether the user was ever blocked.<jupyter_code>users.at[ users[ "blocked"] > 1, "blocked" ] = 1
split_buckets(users)

verbose_chisquared(
    "VE",
    n_treatment,
    users.treatment["blocked"].sum(),
    "control",
    n_control,
    users.control["blocked"].sum()
)<jupyter_output>VE n: 38365
VE k: 1687.0
VE rate: 0.0439723706503
-----
control n: 38493
control k: 1686.0
control rate: 0.0438001714597
-----
Power_divergenceResult(statistic=0.012982634558350057, pvalue=0.90928427489137398)<jupyter_text>So the visual editor make no difference when it came to blocks. This makes sense; there's no reason that making it easier to edit would drive away bad-faith actors.  Revert rateTo calculate the revert rate, I use Aaron's script [python-mwmetrics](https://github.com/mediawiki-utilities/python-mwmetrics). By default, it checks for reverts in the next 48 hours and 15 edits, whichever is less, after the edit in question. The [revert checking](https://github.com/mediawiki-utilities/python-mediawiki-utilities/blob/3edd1cc3754e71f8f77ba4c53c406934818d6b99/mw/lib/reverts/database.py) uses hashes. If an edit within the follow-up window has the same hash as one of edit's close parents, it's considered to have been reverted.I [tweaked the script](https://github.com/neilpquinn/hacky-mwrevert-detector/blob/3c10fb9e5a0b4357a43ae267c105b9f9eb797f67/mwmetrics/utilities/new_users.py) to look for reverts across the users' first sixty days.I had to exclude one user from the cohort (ID 26048397) because of a [strange inconsistency](https://lists.wikimedia.org/pipermail/analytics/2015-December/004697.html) between database tables.First, since the detector finds only reverts in the main namespace, I want to know the number of edits each user made in that namespace.<jupyter_code># Many namespaces are NULL because the page had subsequently been deleted. To be conservative,
# I'm treating them as main namespace edits. 
def convert_null(value):
    if value == "NULL":
        return 0
    else:
        return int(value)
    
ns_revs = pd.read_table( "./VE_experiment_revs_namespace.tsv", 
            dtype = {"rev_id": int, "namespace": int},
            converters = {"namespace": convert_null}  )

main_ns_revs = ns_revs[ns_revs["namespace"] == 0 ]

main_ns_revs = pd.merge( revs, main_ns_revs, on = "rev_id", how = "inner" )

main_revs_by_user = main_ns_revs.groupby( "user_id" )[ "rev_id" ].size().to_frame().reset_index()

main_revs_by_user.rename( columns = { 0: "content_revs" }, inplace = True )

editors = pd.merge( editors, main_revs_by_user, on = "user_id", how = "left" )

editors.fillna( 0, inplace = True )

editors.tail()
reverts = pd.read_table( "./VE_experiment_reverts.tsv" )
reverts.drop("user_registration", axis = 1, inplace = True )

editors = pd.merge( editors, reverts, on = "user_id", how = "left" )

split_buckets(editors)
print( "average treatment edits:", round( editors.treatment["content_revs"].mean(), 2) )
print( "average control edits:", round( editors.control["content_revs"].mean(), 2) )

print("Reverts:")
verbose_mannwhitneyu(
    "VE", 
    editors.treatment["reverted_content_revisions"],
    "control",
    editors.control["reverted_content_revisions"]
)<jupyter_output>average treatment edits: 5.43
average control edits: 5.91
Reverts:
VE average: 0.828
control average: 0.991
MannwhitneyuResult(statistic=85926513.0, pvalue=0.0040913422223638387)<jupyter_text>Wow, that is significant, even at a stringent p = 0.01 level.<jupyter_code>editors.loc[:, "proportion_reverted"] = editors["reverted_content_revisions"] / editors["revs"]

editors.fillna(value = 0, inplace = True)

split_buckets(editors)

verbose_mannwhitneyu(
    "VE",
    editors.treatment["proportion_reverted"],
    "control",
    editors.control["proportion_reverted"]
)
fig, axarr = plt.subplots( 1, 1, figsize=(20,10) )
plt.hist(editors.control["reverted_content_revisions"], log = True, alpha = 0.5, range = (0, 20), normed = 1, bins = 20, label = "Source editor only")
plt.hist(editors.treatment["reverted_content_revisions"], log = True, alpha = 0.5, range = (0, 20), normed = 1, bins = 20, label = "Both editors")
plt.legend(loc="upper right")
plt.show()
fig, axarr = plt.subplots( 1, 1, figsize=(20,10) )
plt.hist(editors.control["proportion_reverted"], alpha = 0.5, range = (0, 1), bins = 20, label = "Source editor only")
plt.hist(editors.treatment["proportion_reverted"], alpha = 0.5, range = (0, 1), bins = 20, label = "Both editors")
plt.legend(loc="upper right")
plt.show()<jupyter_output><empty_output><jupyter_text>Long-term survival Proportion who made at least one edit more than a month after registering.Chi squared test.<jupyter_code>month = dt.timedelta( days = 30 )
survived_revs = revs[ revs["rev_timestamp"] > ( revs["registration"] + month ) ]

survivors = pd.DataFrame( survived_revs[ "user_id" ].unique() )
survivors.rename( columns = { 0: "user_id" }, inplace=True )
survivors.loc[ :, "survived" ] = True

survival = pd.merge( users, survivors, on="user_id", how="left" )

split_buckets(survival)
    
verbose_chisquared(
    "treatment",
    n_treatment, 
    len(survival.treatment[survival["survived"] == True]),
    "control",
    n_control,
    len(survival.control[survival["survived"] == True ])
)<jupyter_output>treatment n: 38365
treatment k: 1100
treatment rate: 0.028671966636257006
-----
control n: 38493
control k: 1081
control rate: 0.028083028083028084
-----
Power_divergenceResult(statistic=0.23485578752524316, pvalue=0.62794647801854031)<jupyter_text>That's disappointing, particularly considering that a promising resulting on this test was my main motivation for doing this follow-up. But clearly, there's nothing here.  Long-term survival filtered  Just to see what happens, let's see how it looks if we filter out all the users who never made an edit before checking survival. The fact that there's no significant difference in the rate of any editing means we're not introducing a new bias here.<jupyter_code>editors_survival = pd.merge( editors, survivors, on="user_id", how="left" )
split_buckets(editors_survival)
split_buckets(editors)

verbose_chisquared(
    "treatment",
    len(editors.treatment),
    len(editors_survival.treatment[survival["survived"] == True]),
    "control",
    len(editors.control),
    len(editors_survival.control[survival["survived"] == True])
)<jupyter_output>treatment n: 13343
treatment k: 303
treatment rate: 0.0227085363111744
-----
control n: 13126
control k: 294
control rate: 0.022398293463355173
-----
Power_divergenceResult(statistic=0.028236776179406321, pvalue=0.8665533686599507)
{'fraction_non_alphanumeric': 0.1153785798352295, 'fraction_numerical': 0.04723420949391918, 'mean_word_length': 4.899328859060403, 'pattern_counts': {'":': 13, '<': 64, '<?xml version=': 0, '>': 69, 'https://': 15, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '157561', 'license': 'MIT', 'n_tokens_mistral': 9507, 'n_tokens_neox': 8370, 'n_words': 2500, 'path': 'VE experiment follow-up.ipynb', 'repo_name': 'neilpquinn/2015-11-VE-experiment-follow-up', 'size': '25498'}