<jupyter_start><jupyter_text>Revert Prediction Analysis - Solvejg===Revert prediction analysis notebook created by Solvejg. Contains an initial comparison of the predictions of the revert models for different revert outcomes.The filenames of the predictions give, respectively, the max features (not confirmed), the tree depth (not confirmed), the revert start time (in seconds) and the revert end time (in seconds).  In other words, a revision is considered reverted if it was reverted between start time and end time seconds after it was published.Multiple follow-up analyses to do here. The core question that I would want to answer by sampling of revisions is: comparing the revisions that DON'T change outcome between two similar models, what revisions have a huge change in score? (Is there some particular quality of those revisions that the model is doing better or worse on?)  In general, need to use the actual revert labels (which are in the prediction dataframes) to figure out the performance of the models (including on the subset of revisions with a consistent revert behavior for some subset of models).  To be clear, we don't even know if there are big performance differences between the models.<jupyter_code>import numpy as np
import pandas as pd

import os
from tqdm import tqdm
import bz2
import sqlite3
import difflib
import gzip
import json
import re
import hashlib
from datetime import datetime
from datetime import timezone
import scipy.stats

import matplotlib.pyplot as plt
from itertools import combinations
# load predictions
pred_dir = '/export/scratch2/levon003/repos/wiki-ores-feedback/data/derived/audit/predictions'

pred_0_30 = pd.read_pickle(os.path.join(pred_dir, 'GB_50_5_0_30_prediction_2020-08-01.pkl'))
pred_0_300 = pd.read_pickle(os.path.join(pred_dir, 'GB_50_5_0_300_prediction_2020-08-01.pkl'))
pred_0_7200 = pd.read_pickle(os.path.join(pred_dir, 'GB_50_5_0_7200_prediction_2020-08-01.pkl'))
pred_0_172800 = pd.read_pickle(os.path.join(pred_dir, 'GB_50_5_0_172800_prediction_2020-08-01.pkl'))
pred_0_604800 = pd.read_pickle(os.path.join(pred_dir, 'GB_50_5_0_604800_prediction_2020-08-01.pkl'))
pred_0_2592000 = pd.read_pickle(os.path.join(pred_dir, 'GB_50_5_0_2592000_prediction_2020-08-01.pkl'))
pred_30_7200 = pd.read_pickle(os.path.join(pred_dir, 'GB_50_5_30_7200_prediction_2020-08-01.pkl'))
pred_7200_2592000 = pd.read_pickle(os.path.join(pred_dir, 'GB_50_5_7200_2592000_prediction_2020-08-01.pkl'))
pred_30_600000000 = pd.read_pickle(os.path.join(pred_dir, 'GB_50_5_30_600000000_prediction_2020-08-01.pkl'))
pred_300_600000000 = pd.read_pickle(os.path.join(pred_dir, 'GB_50_5_300_600000000_prediction_2020-08-01.pkl'))
pred_7200_600000000 = pd.read_pickle(os.path.join(pred_dir, 'GB_50_5_7200_600000000_prediction_2020-08-01.pkl'))
pred_2592000_600000000 = pd.read_pickle(os.path.join(pred_dir, 'GB_50_5_2592000_600000000_prediction_2020-08-01.pkl'))

pred_df_dict = {'0-30 sec': pred_0_30, '0-5 min': pred_0_300, '0-2 hr': pred_0_7200, '0-48 hr': pred_0_172800, '0-7 days': pred_0_604800, '0-30 days': pred_0_2592000, '30 sec-2 hr': pred_30_7200, '2 hr-30 days': pred_7200_2592000, '>= 30 sec': pred_30_600000000, '>= 5 min': pred_300_600000000, '>= 2 hr': pred_7200_600000000, '>= 30 days': pred_2592000_600000000}
# classify predicted probabilities
for label in pred_df_dict:
    df = pred_df_dict[label]
    df['test_pred'] = np.where(df['test_calib'] >= .5, True, False)
    df['test_pred'] = np.where(df['test_calib'] >= .5, True, False)<jupyter_output><empty_output><jupyter_text>Pairwise comparison of time thresholds: all pairs<jupyter_code>nochg_all = pd.DataFrame()
chg_all = pd.DataFrame()

for pair in combinations(pred_df_dict, 2):
    x_df = pred_df_dict[pair[0]]
    y_df = pred_df_dict[pair[1]]
    new_df = pd.DataFrame()
    
    new_df['diff'] = abs(x_df['test_calib'] - y_df['test_calib'])
    new_df['outcomechg'] = np.where(x_df['test_label'] != y_df['test_label'], True, False)
    new_chg = new_df[new_df['outcomechg']]
    new_nochg = new_df[new_df['outcomechg']==False]
    
    label = pair[0] + '_' + pair[1]
    nochg_row = {'pair': label, 'diff_mean': new_nochg['diff'].mean(), 'diff_sd': new_nochg['diff'].std(), 'diff_med': new_nochg['diff'].median(), 'diff_max': new_nochg['diff'].max(), 'diff_min': new_nochg['diff'].min()}
    chg_row = {'pair': label, 'diff_mean': new_chg['diff'].mean(), 'diff_sd': new_chg['diff'].std(), 'diff_med': new_chg['diff'].median(), 'diff_max': new_chg['diff'].max(), 'diff_min': new_chg['diff'].min()}
    
    nochg_all = nochg_all.append(nochg_row, ignore_index = True)
    chg_all = chg_all.append(chg_row, ignore_index = True)
nochg_all.sort_values(by = 'diff_mean', ascending = False)
chg_all.sort_values(by = 'diff_mean', ascending = False)<jupyter_output><empty_output><jupyter_text>Pairwise comparisons of time thresholds: One pair at a time<jupyter_code># create new df with rev ids, predictions, and pairwise comparisons
df_1 = '0-30 sec'
df_2 = '0-5 min'

prediction_df = pred_df_dict[df_1].merge(pred_df_dict[df_2], on = 'rev_id', suffixes = ('_'+df_1, '_'+df_2))

prediction_df['diff'] = abs(prediction_df['test_calib_' + df_1] - prediction_df['test_calib_' + df_2])
prediction_df['outcomechg'] = np.where(prediction_df['test_label_' + df_1] != prediction_df['test_label_' + df_2], True, False)
prediction_df.head()<jupyter_output><empty_output><jupyter_text>Summary statistics for revisions that DID NOT change labeled outcome<jupyter_code>nochg = prediction_df[prediction_df['outcomechg']==False]
# difference in predicted probability of revert = True
nochg['diff'].describe()
nochg['diff'].median()
# distribution of differences in predicted probability
n, bins, patches = plt.hist(nochg['diff'], density = False)
plt.xlabel('Difference in predicted probability')
plt.ylabel('Count')
plt.title('No change: ' + df_1 + ' vs. ' + df_2)
plt.show()
# revisions with the largest difference in predicted probability
nochg.sort_values(by = 'diff', axis = 0, ascending = False).head()<jupyter_output><empty_output><jupyter_text>Summary statistics for revisions that DID change labeled outcome<jupyter_code>chg = prediction_df[prediction_df['outcomechg']]
# difference in predicted probability of revert = True
chg['diff'].describe()
# distribution of differences in predicted probability
n, bins, patches = plt.hist(chg['diff'], density = False)
plt.xlabel('Difference in predicted probability')
plt.ylabel('Count')
plt.title('Change: ' + df_1 + ' vs. ' + df_2)
plt.show()
# revisions with the largest difference in predicted probability
chg.sort_values(by = 'diff', axis = 0, ascending = False).head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12501878287002255, 'fraction_numerical': 0.0754320060105184, 'mean_word_length': 6.461883408071749, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 27, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 12, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '564482', 'license': 'MIT', 'n_tokens_mistral': 2675, 'n_tokens_neox': 2372, 'n_words': 561, 'path': 'notebook/audit/RevertPredictionAnalysis_SHW.ipynb', 'repo_name': 'levon003/wiki-ores-feedback', 'size': '6655'}