<jupyter_start><jupyter_text>Event extraction evaluation using correctness criteria defined by <NAME> and <NAME> 2008<NAME> and <NAME> (2008). Refining event extraction through cross-document inference.> - A trigger is correctly labeled if its event type and offsets match a reference trigger.> - An argument is correctly identified if its event type and offsets match any of the reference argument mentions.> - An argument is correctly identified and classified if its event type, offsets, and role match any of the reference argument mentions.The main difference to the previous evaluation method lies in the correctness criteria for the arguments. Here we additionally consider the correctness of the event type for the argument.Caution:Using the following methods to retrieve the triggers and arguments from the gold data might result in duplicate gold triggers & arguments.This is due to different events possibly sharing the same trigger.The model is not able to distinguish such events and instead fuses them all together, which results in lower recall.If we remove duplicates from the gold triggers and gold arguments, recall and consequently f1 should be higher.<jupyter_code>REMOVE_DUPLICATES = True  # change to False if you want to keep duplicate triggers/ arguments from the gold data caused by events sharing the same trigger
test_docs = ace_test.to_dict(orient='records')
gold_triggers = scorer.get_triggers(test_docs, events_key='golden-event-mentions')
gold_arguments = scorer.get_arguments(test_docs, events_key='golden-event-mentions')
print(f'Number of gold triggers: {len(gold_triggers)}')
print(f'Number of gold arguments: {len(gold_arguments)}')
if REMOVE_DUPLICATES:
    gold_triggers = list(set(gold_triggers))
    gold_arguments = list(set(gold_arguments))
    print(f'After removing duplicate triggers/ arguments from the gold data caused by events sharing the same trigger')
    print(f'Number of unique gold triggers: {len(gold_triggers)}')
    print(f'Number of unique gold arguments: {len(gold_arguments)}')<jupyter_output>After removing duplicate triggers/ arguments from the gold data caused by events sharing the same trigger
Number of unique gold triggers: 422
Number of unique gold arguments: 889<jupyter_text>5 Repeated Runs5 repeated runs for each configuration with random seeds for the eventx model.Metrics are given as mean & standard deviation.<jupyter_code>model_base_path = Path("../../data/runs/random_repeats_ace_lr_biased_jmee")
model_names = [f"run_{i+1}" for i in range(5)]
trigger_metrics_rr, argument_metrics_rr = {}, {}
model_paths = [model_base_path.joinpath(model_name) for model_name in model_names]
trigger_metrics_rr, argument_metrics_rr = evaluate.summize_multiple_runs(model_paths, test_docs, 
                                                                         predictor_name=PREDICTOR_NAME, 
                                                                         events_key="golden-event-mentions",
                                                                          include_class_metrics=False)
trigger_related_categories = ["Trigger identification", "Trigger classification"]
role_related_categories = ["Argument identification", "Argument classification"]
formatted_trigger = evaluate.format_classification_report(evaluate.get_mean_std(trigger_metrics_rr), make_string=True)
formatted_argument = evaluate.format_classification_report(evaluate.get_mean_std(argument_metrics_rr), make_string=True)
formatted_metrics = pd.concat([formatted_trigger, formatted_argument])
formatted_metrics.set_index('row_name')<jupyter_output><empty_output><jupyter_text>Statistics about the ACE data<jupyter_code>def convert_to_stats_df(stats, data_sets):
    statistics_df = pd.DataFrame(stats)
    statistics_df['Data set'] = data_sets  
    return statistics_df.set_index('Data set')
general_stats = [corpus_statistics.get_ace_general_stats(data_set) for data_set in [ace_train, ace_dev, ace_test]]
general_stats_df = convert_to_stats_df(general_stats, data_sets=["Train", "Dev", "Test"])
trigger_stats = [corpus_statistics.get_trigger_class_frequencies(data_set, event_keyword="golden-event-mentions") for data_set in [ace_train, ace_dev, ace_test]]
trigger_stats_df = convert_to_stats_df(trigger_stats, data_sets=["Train", "Dev", "Test"])
role_stats = [corpus_statistics.get_role_class_frequencies(data_set, event_keyword="golden-event-mentions") for data_set in [ace_train, ace_dev, ace_test]]
role_stats_df = convert_to_stats_df(role_stats, data_sets=["Train", "Dev", "Test"])
general_stats_df.T
sns.set(font_scale=1.5, font="Palatino")
with plt.style.context("seaborn-white"):
    plt.rcParams["font.family"] = "Palatino"
    plt.rcParams['pdf.fonttype'] = 42
    fig, ax = plt.subplots(1, 1, figsize=(20,7))
    trigger_plot = trigger_stats_df.T.plot.bar(ax=ax)
    ax.set_ylabel('Trigger class frequencies')
    trigger_plot.set_xticklabels(trigger_plot.get_xticklabels(), rotation = 90)
    handles, labels = ax.get_legend_handles_labels()
    fig.savefig(SAVE_PATH.joinpath('ACE-Event_Stats.pdf'), bbox_inches='tight')
sns.set(font_scale=1.5, font="Palatino")
with plt.style.context("seaborn-white"):
    plt.rcParams["font.family"] = "Palatino"
    plt.rcParams['pdf.fonttype'] = 42
    fig, ax = plt.subplots(1, 1, figsize=(20,7))
    role_plot = role_stats_df.T.plot.bar(ax=ax)
    ax.set_ylabel('Role class frequencies')
    role_plot.set_xticklabels(role_plot.get_xticklabels(), rotation = 90)
    handles, labels = ax.get_legend_handles_labels()
    fig.savefig(SAVE_PATH.joinpath('ACE-Role_Stats.pdf'), bbox_inches='tight')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09166814709539883, 'fraction_numerical': 0.0072837093622313024, 'mean_word_length': 5.234772978959025, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 19, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '272713', 'license': 'MIT', 'n_tokens_mistral': 1683, 'n_tokens_neox': 1624, 'n_words': 441, 'path': 'analysis/notebooks/eventx_ace_analysis.ipynb', 'repo_name': 'DFKI-NLP/eventx', 'size': '5651'}