<jupyter_start><jupyter_text>CoNLL_2.ipynbThis notebook contains the first part of the model training and analysis code from our CoNLL-2020 paper, ["Identifying Incorrect Labels in the CoNLL-2003 Corpus"](https://www.aclweb.org/anthology/2020.conll-1.16/).If you're new to the Text Extensions for Pandas library, we recommend that you startby reading through the notebook [`Analyze_Model_Outputs.ipynb`](https://github.com/CODAIT/text-extensions-for-pandas/blob/master/notebooks/Analyze_Model_Outputs.ipynb), which explains the portions of the library that we use in the notebooks in this directory. SummaryThis notebook aggregates the outputs of the 16 models submitted to the CoNLL-2003 competition. Then it compares these outputs to the corpus's gold-standard labels and identifies areas where there is a strong consensus between model outputs coupled with a disagreement with the corpus labels. Finally, it writes out CSV files containing ranked lists of potentially-incorrect labels.<jupyter_code># INITIALIZATION BOILERPLATE

# Libraries
import os
import sys
import numpy as np
import pandas as pd
from typing import *

# And of course we need the text_extensions_for_pandas library itself.
try:
    import text_extensions_for_pandas as tp
except ModuleNotFoundError as e:
    raise Exception("text_extensions_for_pandas package not found on the Jupyter "
                    "kernel's path. Please either run:\n"
                    "   ln -s ../../text_extensions_for_pandas .\n"
                    "from the directory containing this notebook, or use a Python "
                    "environment on which you have used `pip` to install the package.")

PROJECT_ROOT = "../.."    
    
# Code shared among notebooks is kept in util.py, in this directory.
import util
# Download and cache the data set.
# NOTE: This data set is licensed for research use only. Be sure to adhere
#  to the terms of the license when using this data set!
data_set_info = tp.io.conll.maybe_download_conll_data("outputs")
data_set_info
# Load up the same gold standard data we used in Part 1.
gold_standard = tp.io.conll.conll_2003_to_dataframes(
    data_set_info["test"], ["pos", "phrase", "ent"], [False, True, True])
gold_standard = [
    df.drop(columns=["pos", "phrase_iob", "phrase_type"])
    for df in gold_standard
]

# Dictionary from (collection, offset within collection) to dataframe
gold_standard_spans = {("test", i): 
                       tp.io.conll.iob_to_spans(gold_standard[i]) 
                       for i in range(len(gold_standard))}
# Load up the results from all 16 teams at once.
teams = ["bender", "carrerasa", "carrerasb", "chieu", "curran",
         "demeulder", "florian", "hammerton", "hendrickx",
         "klein", "mayfield", "mccallum", "munro", "whitelaw",
         "wu", "zhang"]

# Read all the output files into one dataframe per <document, team> pair.
outputs = { 
    t: tp.io.conll.conll_2003_output_to_dataframes(
        gold_standard, f"{PROJECT_ROOT}/resources/conll_03/ner/results/{t}/eng.testb")
    for t in teams
}  # Type: Dict[str, List[pd.DataFrame]]

# As an example of what we just loaded, show the token metadata for the 
# "mayfield" team's model's output on document 3.
outputs["mayfield"][3]
# Convert results from IOB2 tags to spans across all teams and documents
# See https://en.wikipedia.org/wiki/Inside%E2%80%93outside%E2%80%93beginning_(tagging) for details on IOB2 format.
output_spans = {
    t: {("test", i): tp.io.conll.iob_to_spans(outputs[t][i]) 
        for i in range(len(outputs[t]))}
    for t in teams
}  # Type: Dict[str, Dict[Tuple[str, int], pd.DataFrame]]

# As an example, show the first 10 spans that the "florian" team's model
# found on document 2.
output_spans["florian"][("test", 2)].head(10)
# Use Pandas merge to find what spans match up exactly for each team's
# results.
# Unlike Part 1, we perform the join across all entity types, looking for
# matches of both the extracted span *and* the entity type label.
#
# Text Extensions for Pandas includes a utility function 
# compute_accuracy_by_document() to makes this collection-level computation
# simpler.
stats = {
    t: tp.io.conll.compute_accuracy_by_document(
    gold_standard_spans, output_spans[t]) 
    for t in teams
}

# Show the result quality statistics by document for the "carrerasa" team
stats["carrerasa"]
# F1 for document 4 is looking a bit low. Is that just fluke, or is it
# part of a larger trend? 
# In Part 1, we showed how to drill down to and examine "problem" documents.
# Since we have all this additional data, let's try a broader, more 
# quantitative approach. We'll start by building up some more fine-grained 
# data about congruence between the gold standard and the model outputs.
# Pandas' outer join will tell us what entities showed up just in the gold
# standard, just in the model output, or in both sets.
# For starters, let's do this just for the "carrerasa" team and document 4.
doc_id = ("test", 4)
(gold_standard_spans[doc_id]
    .merge(output_spans["carrerasa"][doc_id], how="outer", indicator=True)
    .sort_values("span"))
# Repeat the analysis from the previous cell across all teams and documents.
# That is, perform an outer join between the gold standard spans dataframe
# for each document and the corresponding dataframe from each team.
def merge_span_sets(team: str,
                    gold_results: Dict[Tuple[str, int], pd.DataFrame],
                    results_by_team: Dict[str, Dict[Tuple[str, int], pd.DataFrame]]):
    result = {}  # Type: Dict[Tuple[str, int]: pd.DataFrame]
    for k in gold_results.keys():
        merged = gold_results[k].merge(results_by_team[team][k],
                                       how="outer", indicator=True)
        merged["gold"] = merged["_merge"].isin(("both", "left_only"))
        merged[team] = merged["_merge"].isin(("both", "right_only"))
        result[k] = merged[["span", "ent_type", "gold", team]]
    return result

span_flags = {t: merge_span_sets(t, gold_standard_spans, output_spans) for t in teams}  # Type: Dict[Tuple[str, int]: pd.DataFrame]
# Now we have indicator variables for every extracted span, telling whether 
# it was in the gold standard data set and/or in each of the team's results.
# For example, here are the first 5 spans for document 2 in the "carrerasa"
# team's results:
doc_id = ("test", 2)
span_flags["carrerasa"][doc_id].head(5)
# Do an n-way merge of all those indicator variables across documents.
# This operation produces a single summary dataframe per document.
indicators = {}  # Type: Dict[Tuple[str, int], pd.DataFrame]
for k in gold_standard_spans.keys():
    result = gold_standard_spans[k]
    for t in teams:
        result = result.merge(span_flags[t][k], how="outer")
    indicators[k] = result.fillna(False)
    
# Now we have a vector of indicator variables for every span extracted 
# from every document across all the model outputs and the gold standard.
# For example, let's show the results for document 10:
doc_10 = ("test", 10)
indicators[doc_10]
# If you look at the above dataframe, you can see that some entities 
# ("RUGBY UNION", for example) are "easy", in that almost every entry
# found them correctly. Other entities, like "CAMPESE", are "harder",
# in that few of the entrants correctly identified them. Let's add
# a column that quantifies this "difficulty level" by counting how 
# many teams found each true or false positive.
for df in indicators.values():
    # Convert the teams' indicator columns into a single matrix of 
    # Boolean values, and sum the number of True values in each row.
    vectors = df[df.columns[3:]].values
    counts = np.count_nonzero(vectors, axis=1)
    df["num_teams"] = counts

# Show the dataframe for document 10 again, this time with the new
# "num_teams" column at the far right.
indicators[doc_10]
# Now we can rank the entities in document 10 by "difficulty", either as 
# true positives for the models to find...
# (just for document 10 for the moment)
ind = indicators[doc_10].copy()
ind[ind["gold"] == True].sort_values("num_teams").head(10)
# ...or as false positives to avoid:
ind[ind["gold"] == False].sort_values("num_teams", ascending=False).head(10)
# To get a better picture of what entities are "difficult", we need to look 
# across the entire test set. Let's combine the dataframes in 
# `indicators` into a single dataframe that covers all the documents.

# First we preprocess each dataframe to make it easier to combine.
to_stack = [
    pd.DataFrame({
        "fold": k[0],  # Keys are (collection, offset) tuples
        "doc_offset": k[1],
        "span" : indicators[k]["span"],
        "ent_type": indicators[k]["ent_type"],
        "gold": indicators[k]["gold"],
        "num_teams": indicators[k]["num_teams"]
    })
    for k in indicators.keys()
    #for i in range(len(indicators))
]

# Then we concatenate all the preprocessed dataframes into a single dataframe.
counts = pd.concat(to_stack)
counts
# Now we can pull out the most difficult entities across the entire test
# set.
# First, let's find the most difficult entities from the standpoint of recall:
# entities that are in the gold standard, but not in most results.
difficult_recall = counts[counts["gold"] == True].sort_values("num_teams").reset_index(drop=True)
difficult_recall.head(10)
# Hmm, everything is zero. How many entities were found by zero teams?  One team?
(counts[counts["gold"] == True][["num_teams", "span"]]
 .groupby("num_teams").count()
 .rename(columns={"span": "count"}))
# Yikes! 140 entities in the test set were so hard to find, they
# were extracted by 0 teams.
# Let's go back and look at some of those 0-team entities in context:
difficult_recall["context"] = difficult_recall["span"].apply(lambda t: t.context())
pd.set_option('max_colwidth', 100)
difficult_recall.head(20)<jupyter_output><empty_output><jupyter_text>**Some of these entities are "difficult" because the test set contains incorrect labels.**For reference, there's a copy of the CoNLL labeling rules in this repository at[resources/conll_03/ner/annotation.txt](../resources/conll_03/ner/annotation.txt)There are 4 incorrect labels in this first set of 20:* `[3289, 3299): 'Full Light'` should be "Zywiec Full Light"* `[11, 19): 'Honda RV'` should be tagged `ORG`* `[1525, 1541): 'Consumer Project'` should be "Consumer Project on Technology" and should be tagged `ORG`* `[244, 255): 'McDonald 's'` should be tagged `MISC` (because it's an "adjective ... derived from a word which is ... organisation")<jupyter_code># Let's look at the entities that are difficult from the perspective of 
# precision: that is, in many models' results, but not in the gold standard.
difficult_precision = counts[counts["gold"] == False].sort_values("num_teams", ascending=False).reset_index(drop=True)

# Again, we can add some context to these spans:
difficult_precision["context"] = difficult_precision["span"].apply(lambda t: t.context())
difficult_precision.head(20)
difficult_precision.loc[[16]]<jupyter_output><empty_output><jupyter_text>**As with the entities in `difficult_recall`, some of these entities in `difficult_precision` are "difficult" because the test set has missing and incorrect labels.****13** of these first 20 "incorrect" results are due to missing and incorrect labels:* `[25, 32): 'BRITISH''` in document 202 should be tagged `MISC`.* `[1317, 1327): 'Portsmouth'` in document 207 should be tagged `ORG`, not `LOC`.* `[110, 118): 'Scottish'` in document 199 should be tagged `MISC`  (or `[28, 53): 'SCOTTISH PREMIER DIVISION'` and   `[110, 135): 'Scottish premier division'` should both be tagged `ORG`).* `[146, 163): 'Santiago Bernabeu'` in document 40 should be tagged `MISC`  (because the "s" in `[146, 171): 'Santiago Bernabeu stadium'` is not capitalized).* `[239, 251): 'Philadelphia'` in document 223 should be tagged `ORG`, not `LOC`.* `[367, 376): 'Karlsruhe'` in document 36 should be tagged `ORG`, not `LOC`.* `[1003, 1011): 'Congress'` in document 100 should be tagged `ORG`  (also, `[957, 964): 'Chilean' ==> MISC` should be replaced with   `[957, 973): 'Chilean Congress' ==> ORG`).* `[420, 428): 'Freiburg'` in document 36 should be tagged `ORG`, not `LOC`.* In document 70, `[186, 211): 'New York Commodities Desk'`, not `[186, 206): 'New York Commodities'`, should be tagged `ORG`.* `[263, 271): '<NAME>'` in document 223 should be tagged `ORG`, not `LOC`.* `[788, 795): 'Antwerp'` in document 155 should be tagged `LOC`, not `ORG`.* In document 112, `[178, 191): '<NAME>'`, not `[178, 188): '<NAME>'`, should be tagged `PER`.* `[274, 282): 'COLORADO'` in document 223 should be tagged `ORG`.<jupyter_code># Here's the gold standard data for document 155, for example.
# Note line 12.
doc_id = ("test", 155)
gold_standard_spans[doc_id][0:60]
# The above gold standard spans in context. 
gold_standard_spans[doc_id]["span"].values
# Repeat the steps from the previous cells using the dev set.
# This takes a while.
dev_gold_standard = tp.io.conll.conll_2003_to_dataframes(
    data_set_info["dev"], ["pos", "phrase", "ent"], [False, True, True])
dev_gold_standard = [
    df.drop(columns=["pos", "phrase_iob", "phrase_type"])
    for df in dev_gold_standard
]

dev_gold_standard_spans = {
    ("dev", i): tp.io.conll.iob_to_spans(dev_gold_standard[i]) 
    for i in range(len(dev_gold_standard))}

dev_outputs = { 
    t: tp.io.conll.conll_2003_output_to_dataframes(
        dev_gold_standard, f"{PROJECT_ROOT}/resources/conll_03/ner/results/{t}/eng.testa")
    for t in teams
}  # Type: Dict[str, List[pd.DataFrame]]

dev_output_spans = {
    t: {("dev", i): tp.io.conll.iob_to_spans(dev_outputs[t][i]) 
        for i in range(len(dev_outputs[t]))}
    for t in teams
}  # Type: Dict[str, Dict[Tuple[str, int], pd.DataFrame]]

dev_span_flags = {t: merge_span_sets(t, dev_gold_standard_spans,
                                      dev_output_spans) 
                   for t in teams}  # Type: Dict[Tuple[str, int]: pd.DataFrame]

dev_indicators = {}  # Type: Dict[Tuple[str, int], pd.DataFrame]
for k in dev_gold_standard_spans.keys():
    result = dev_gold_standard_spans[k]
    for t in teams:
        result = result.merge(dev_span_flags[t][k], how="outer")
    dev_indicators[k] = result.fillna(False)
    
for df in dev_indicators.values():
    # Convert the teams' indicator columns into a single matrix of 
    # Boolean values, and sum the number of True values in each row.
    vectors = df[df.columns[3:]].values
    nonzero_counts = np.count_nonzero(vectors, axis=1)
    df["num_teams"] = nonzero_counts
    
dev_counts = pd.concat([
    pd.DataFrame({
        "fold": k[0],  # Keys are (collection, offset) tuples
        "doc_offset": k[1],
        "span" : dev_indicators[k]["span"],
        "ent_type": dev_indicators[k]["ent_type"],
        "gold": dev_indicators[k]["gold"],
        "num_teams": dev_indicators[k]["num_teams"]
    })
    for k in dev_indicators.keys()
])
# How many teams found entities from the dev set that are in the gold standard?
(dev_counts[dev_counts["gold"] == True][["num_teams", "span"]]
 .groupby("num_teams").count()
 .rename(columns={"span": "count"}))
# How many teams found entities from the dev set that aren't in the gold standard?
(dev_counts[dev_counts["gold"] == False][["num_teams", "span"]]
 .groupby("num_teams").count()
 .rename(columns={"span": "count"}))
# Merge the results from the two folds
all_counts = pd.concat([counts, dev_counts])
all_counts.head()
all_counts.tail()
# Write out the results that are in the gold standard but not in
# few teams' outputs.
in_gold_to_write, not_in_gold_to_write = util.csv_prep(all_counts, "num_teams")
in_gold_to_write
not_in_gold_to_write
# Write output files.
in_gold_to_write.to_csv("outputs/CoNLL_2_in_gold.csv", index=False)
not_in_gold_to_write.to_csv("outputs/CoNLL_2_not_in_gold.csv", index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11882434388570709, 'fraction_numerical': 0.020517339039587135, 'mean_word_length': 4.307281229124916, 'pattern_counts': {'":': 13, '<': 17, '<?xml version=': 0, '>': 19, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '181086', 'license': 'Apache-2.0', 'n_tokens_mistral': 5391, 'n_tokens_neox': 5003, 'n_words': 1778, 'path': 'tutorials/corpus/CoNLL_2.ipynb', 'repo_name': 'cullinap/text-extensions-for-pandas', 'size': '15902'}