<jupyter_start><jupyter_text>Demo This is the demo to showcase some analysis process. For the analysis for each task, we have provided a corresponding class.<jupyter_code># import analysis tools
from analysis import SUMStat, D2TStat, WMTStat

def truncate_print(l, n=10):
    """ Print the first n items of a list"""
    for i, x in enumerate(l):
        if i == n:
            print('...')
            break
        print(x)<jupyter_output><empty_output><jupyter_text>Summarization For all summarization datasets, including **REALSumm**, **SummEval** and **Newsroom**, the analysis tools are the same.<jupyter_code>summ_stat = SUMStat('SUM/REALSumm/final_p.pkl') # The path to the scored file, _p means we have prompted metrics<jupyter_output><empty_output><jupyter_text>See what metrics are out there.Since there are a lot, including P, R, F variants for some metrics as well as prompted metrics, we only print a truncated version of metrics<jupyter_code>print('[All metrics]')
truncate_print(summ_stat.metrics) # change to print if you want to see all metrics
print('[Automatic metrics]')
truncate_print(summ_stat.auto_metrics)
print('[Human metrics]')
truncate_print(summ_stat.human_metrics)<jupyter_output>[All metrics]
litepyramid_recall
bert_score_p
bert_score_r
bert_score_f
mover_score
bart_score_src_hypo
bart_score_hypo_ref
bart_score_ref_hypo
bart_score_avg_f
bart_score_harm_f
...
[Automatic metrics]
bert_score_p
bert_score_r
bert_score_f
mover_score
bart_score_src_hypo
bart_score_hypo_ref
bart_score_ref_hypo
bart_score_avg_f
bart_score_harm_f
bart_score_cnn_src_hypo
...
[Human metrics]
litepyramid_recall<jupyter_text>We can choose some metrics that we are interested in to conduct analysis. For example, in **REALSumm**, we use recall-based metrics (e.g. bert_score_r, rouge1_r, bart_score_cnn_hypo_ref, ...)For others, we use F-based metrics (for metrics that only consider hypo and ref) and src->hypo (for generation based metrics like bart_score and prism)<jupyter_code>valid_metrics = [
    'rouge1_r',
    'rouge2_r',
    'rougel_r',
    'bert_score_r',
    'mover_score',
    'prism_hypo_ref',
    'bart_score_cnn_hypo_ref'
]

# The first argument is the human metric considered.
# The second argument is a list of considered automatic metrics, can omit it if all automatic metrics are considered
summ_stat.evaluate_summary('litepyramid_recall', valid_metrics)<jupyter_output>Human metric: litepyramid_recall
metric                     spearman    kendalltau
-----------------------  ----------  ------------
rouge1_r                   0.497526      0.407974
rougel_r                   0.488254      0.402523
bart_score_cnn_hypo_ref    0.474608      0.374497
bert_score_r               0.440398      0.346489
rouge2_r                   0.4233        0.353119
prism_hypo_ref             0.411005      0.323994
mover_score                0.372353      0.290156<jupyter_text>We can also see the performance of some prompt-based metrics.<jupyter_code>valid_metrics = [
    'bart_score_cnn_hypo_ref_de_id est',
    'bart_score_cnn_hypo_ref_de_Videlicet',
    'bart_score_cnn_hypo_ref_de_To give an instance',
    'bart_score_cnn_hypo_ref_de_To give an example',
    'bart_score_cnn_hypo_ref_de_As an illustration'
]
summ_stat.evaluate_summary('litepyramid_recall', valid_metrics)<jupyter_output>Human metric: litepyramid_recall
metric                                            spearman    kendalltau
----------------------------------------------  ----------  ------------
bart_score_cnn_hypo_ref_de_id est                 0.49539       0.392728
bart_score_cnn_hypo_ref_de_Videlicet              0.491011      0.388237
bart_score_cnn_hypo_ref_de_To give an instance    0.49081       0.387054
bart_score_cnn_hypo_ref_de_To give an example     0.489033      0.38625
bart_score_cnn_hypo_ref_de_As an illustration     0.488977      0.385511<jupyter_text>To combine prompt-based metrics, run the following<jupyter_code>summ_stat.combine_prompt()
summ_stat.evaluate_summary('litepyramid_recall', ['bart_score_cnn_hypo_ref_de'])<jupyter_output>Human metric: litepyramid_recall
metric                        spearman    kendalltau
--------------------------  ----------  ------------
bart_score_cnn_hypo_ref_de     0.48784      0.386398<jupyter_text>To conduct bootstrapping significant test, we provide the *sig_test_two ( )* and *sig_test ( )* method.<jupyter_code># The first two arguments are metrics that should be compared, the third argument is the human metric.
m1 = 'bart_score_cnn_hypo_ref'
m2 = 'bert_score_r'
result = summ_stat.sig_test_two(m1, m2, 'litepyramid_recall')
if result == 1:
    print(f'{m1} is significantly better than {m2}')
elif result == -1:
    print(f'{m2} is significantly better than {m1}')
else:
    print('cannot decide')
# The first arguments are a list of metrics considered
# The second argument is the human metric
summ_stat.sig_test(['rouge1_r', 'bart_score_cnn_hypo_ref', 'bert_score_r'], 'litepyramid_recall')<jupyter_output>100%|██████████| 1000/1000 [01:28<00:00, 11.32it/s]
100%|██████████| 1000/1000 [01:24<00:00, 11.81it/s]
100%|██████████| 1000/1000 [01:26<00:00, 11.55it/s]<jupyter_text>Factuality We use **Rank19** dataset and **QAGS_CNN** dataset to showcase some basic usages. The former uses accuracy as its evaluation metric while the latter uses pearson correlation.  Rank19 We first print out the factuality accuracy obtained using different metrics for the **Rank19** dataset.<jupyter_code>fact_stat = SUMStat('SUM/Rank19/final_p.pkl')
fact_stat.combine_prompt()

# Set valid metrics
valid_metrics = [
    'rouge1_f',
    'rouge2_f',
    'rougel_f',
    'bert_score_f',
    'mover_score',
    'prism_src_hypo',
    'bart_score_cnn_src_hypo',
    'bart_score_cnn_src_hypo_de'
]

# Print accuracy, take a list of metrics
fact_stat.get_fact_acc(valid_metrics)<jupyter_output>metric                           acc
--------------------------  --------
bart_score_cnn_src_hypo     0.836461
bart_score_cnn_src_hypo_de  0.796247
prism_src_hypo              0.780161
bert_score_f                0.713137
mover_score                 0.713137
rouge2_f                    0.630027
rougel_f                    0.587131
rouge1_f                    0.568365<jupyter_text>Below are some methods that help to facilitate the siginificant test.<jupyter_code>m1 = 'bart_score_cnn_src_hypo'
m2 = 'bert_score_f'
result = fact_stat.fact_acc_sig_test_two(m1, m2)
if result == 1:
    print(f'{m1} is significantly better than {m2}')
elif result == -1:
    print(f'{m2} is significantly better than {m1}')
else:
    print('cannot decide')
# Take a list of metrics, print the best metrics
fact_stat.fact_acc_sig_test(['bart_score_cnn_src_hypo', 'prism_src_hypo', 'bert_score_f'])<jupyter_output>100%|██████████| 1000/1000 [00:00<00:00, 2082.68it/s]
100%|██████████| 1000/1000 [00:01<00:00, 666.78it/s]
100%|██████████| 1000/1000 [00:01<00:00, 614.94it/s]<jupyter_text>QAGS_CNN<jupyter_code>fact_stat = SUMStat('SUM/QAGS_CNN/final_p.pkl')
fact_stat.combine_prompt()

# Set valid metrics
valid_metrics = [
    'rouge1_f',
    'rouge2_f',
    'rougel_f',
    'bert_score_f',
    'mover_score',
    'prism_src_hypo',
    'bart_score_cnn_src_hypo',
    'bart_score_cnn_src_hypo_de'
]

# Print accuracy, take a list of metrics
fact_stat.get_fact_pearson(valid_metrics)
m1 = 'bart_score_cnn_src_hypo'
m2 = 'bert_score_f'
result = fact_stat.fact_pearson_sig_test_two(m1, m2)
if result == 1:
    print(f'{m1} is significantly better than {m2}')
elif result == -1:
    print(f'{m2} is significantly better than {m1}')
else:
    print('cannot decide')
# Take a list of metrics, print the best metrics
fact_stat.fact_pearson_sig_test(['bart_score_cnn_src_hypo', 'prism_src_hypo', 'bert_score_f'])<jupyter_output>100%|██████████| 1000/1000 [00:00<00:00, 1986.75it/s]
100%|██████████| 1000/1000 [00:00<00:00, 1156.13it/s]
100%|██████████| 1000/1000 [00:00<00:00, 1173.93it/s]<jupyter_text>Data-to-Text For all data-to-text datasets, including **BAGEL**, **SFHOT** and **SFRES**, the analysis tools are the same.<jupyter_code>d2t_stat = D2TStat('D2T/BAGEL/final_p.pkl')
d2t_stat.combine_prompt() # combine the prompt-based resutls<jupyter_output><empty_output><jupyter_text>See what metrics are out there. For data-to-text datasets, the human metrics are *informativeness*, *naturalness* and *quality*.<jupyter_code>print('[All metrics]')
truncate_print(d2t_stat.metrics) # change to print if you want to see all metrics
print('[Automatic metrics]')
truncate_print(d2t_stat.auto_metrics)
print('[Human metrics]')
truncate_print(d2t_stat.human_metrics)<jupyter_output>[All metrics]
informativeness
naturalness
quality
bert_score_p
bert_score_r
bert_score_f
mover_score
bart_score_ref_hypo
bart_score_hypo_ref
bart_score_avg_f
...
[Automatic metrics]
bert_score_p
bert_score_r
bert_score_f
mover_score
bart_score_ref_hypo
bart_score_hypo_ref
bart_score_avg_f
bart_score_harm_f
bart_score_cnn_ref_hypo
bart_score_cnn_hypo_ref
...
[Human metrics]
informativeness
naturalness
quality<jupyter_text>We can print out the correlation w.r.t. human judgement as below.<jupyter_code># Set valid metrics
valid_metrics = [
    'rouge1_f',
    'rouge2_f',
    'rougel_f',
    'bert_score_f',
    'mover_score',
    'prism_avg',
    'bart_score_para_avg_f',
    'bart_score_para_avg_f_de'
]

# The first argument is human metric while the latter is a list of metrics considered.
d2t_stat.evaluate_text('informativeness', valid_metrics)<jupyter_output>Human metric: informativeness
metric                      spearman    kendalltau
------------------------  ----------  ------------
bart_score_para_avg_f_de    0.335997      0.248525
bart_score_para_avg_f       0.329896      0.246686
prism_avg                   0.304946      0.224797
bert_score_f                0.289118      0.217179
mover_score                 0.283694      0.20884
rouge1_f                    0.234338      0.177972
rouge2_f                    0.198585      0.151011
rougel_f                    0.188592      0.145508<jupyter_text>To perform significant test, use *sig_test_two ( )* method<jupyter_code>m1 = 'bart_score_para_avg_f'
m2 = 'prism_avg'

# The first two arguments are metrics that should be compared, the third argument is the human metric.
result = d2t_stat.sig_test_two(m1, m2, 'informativeness')

if result == 1:
    print(f'{m1} is significantly better than {m2}')
elif result == -1:
    print(f'{m2} is significantly better than {m1}')
else:
    print('cannot decide')<jupyter_output>100%|██████████| 1000/1000 [01:19<00:00, 12.54it/s]<jupyter_text>Machine Translation For all language pairs, the analysis tools are the same. We begin by looking at reference length statistics.<jupyter_code>wmt_stat = WMTStat('WMT/kk-en/final_p.pkl')
wmt_stat.print_ref_len()<jupyter_output>Mean reference length: 17.75
Max reference length: 180
Min reference length: 1
20% percentile: 10.0
80% percentile: 25.0
90% percentile: 31.0<jupyter_text>Next, we print out k-tau for all automatic metrics.<jupyter_code>print('All metrics')
print(wmt_stat.metrics) # Print out all metrics
print('\n')
print('All k-tau')
wmt_stat.print_ktau()
print('\n')
print('k-tau for some metrics')
wmt_stat.print_ktau(['prism', 'bart_score_para'])<jupyter_output>64%|██████▎   | 7/11 [00:00<00:00, 69.64it/s]<jupyter_text>To print out the k-tau over certain reference length, run the following.<jupyter_code>print('All k-tau')
wmt_stat.print_len_ktau(15, 25)
print('\n')
print('k-tau for some metrics')
wmt_stat.print_len_ktau(15, 25, ['prism', 'bart_score_para'])<jupyter_output>100%|██████████| 9728/9728 [00:00<00:00, 648147.63it/s]
100%|██████████| 11/11 [00:00<00:00, 179.12it/s]
100%|██████████| 9728/9728 [00:00<00:00, 625838.84it/s]
100%|██████████| 2/2 [00:00<00:00, 194.46it/s]<jupyter_text>To perform significant test, use *sig_test_two ()*<jupyter_code>m1 = 'bart_score_para'
m2 = 'bleurt'

# The first two arguments are metrics that should be compared, the third argument is the human metric.
result = wmt_stat.sig_test_two(m1, m2)

if result == 1:
    print(f'{m1} is significantly better than {m2}')
elif result == -1:
    print(f'{m2} is significantly better than {m1}')
else:
    print('cannot decide')<jupyter_output>100%|██████████| 1000/1000 [00:33<00:00, 29.77it/s]
{'fraction_non_alphanumeric': 0.12841864642011594, 'fraction_numerical': 0.06792391215609438, 'mean_word_length': 4.008176614881439, 'pattern_counts': {'":': 0, '<': 74, '<?xml version=': 0, '>': 59, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '111727', 'license': 'Apache-2.0', 'n_tokens_mistral': 5213, 'n_tokens_neox': 4489, 'n_words': 1002, 'path': 'analysis.ipynb', 'repo_name': 'Dorcoh4/BARTScore', 'size': '12249'}