<jupyter_start><jupyter_text>AugmentationThis notebook recreates Table X from the paper XX and illustrates how to use the augmenters and scoring functions included in DaCy<jupyter_code>import os
os.chdir("..")
import dacy
from dacy.augmenters import create_pers_augmenter, create_keyboard_augmenter, create_æøå_augmenter
from dacy.datasets import danish_names, muslim_names
from dacy.score import score, n_sents_score

import spacy
from spacy.training.augment import create_lower_casing_augmenter, dont_augment

from typing import Callable, List

from functools import partial<jupyter_output><empty_output><jupyter_text>Start off by loading the test set and defining a function that applies the small Spacy model on the data.<jupyter_code>test = dacy.datasets.dane(splits=["test"])


def apply_model(example, nlp):
    example.predicted = nlp(example.predicted.text)
    return example

# make an instance of apply_model using the spacy nlp
nlp = spacy.load("da_core_news_sm")
apply_spacy_model = partial(apply_model, nlp=nlp)<jupyter_output><empty_output><jupyter_text>Let's test how well the model performs on the original data, data where names are changed to other Danish names, and data where names are changes to names of Muslim origin. The name augmenter allows us to specify a number of naming patterns we wish to augment the names to. Defaults are `["fn,ln", "abbpunct,ln"]`, which means names are augmented to the follow the pattern of either "first_name last_name" (e.g. <NAME>) or "abbreviated_first_name last_name" (e.g. <NAME>). The patterns include "fn" (first name, Mette), "ln" (last name, Frederiksen), "abb" (abbreviated, M), "abbpunct" (abbreviated + ., M.). These patterns can be designed however you see fit. We will stick to the defaults for now.<jupyter_code>dk_name_dict = danish_names()
muslim_name_dict = muslim_names()

# Set keep_name to False to make the augmenter choose a new name from the dictionary
#   otherwise, it would simply make the name fit the pattern (e.g. make abbreviations)
# force_size ensures that the names are of the same length/format as the pattern.
dk_aug = create_pers_augmenter(dk_name_dict, force_size=True, keep_name=False)
muslim_aug = create_pers_augmenter(muslim_name_dict, force_size=True, keep_name=False)

scores_raw = score(test, apply_spacy_model, score_fn=["ents"])
scores_dk = score(test, apply_spacy_model, augmenter=dk_aug, score_fn=["ents"])
scores_muslim = score(test, apply_spacy_model, augmenter=muslim_aug, score_fn=["ents"])
scores = scores_raw + scores_dk + scores_muslim
scores.to_df()<jupyter_output><empty_output><jupyter_text>Augmenting names to a Danish name which fit the pattern of either "fn,ln" or "abbpunct,ln" actually made it slightly easier for the model than the original test data. However, augmenting with muslim names made the model perform a lot worse than baseline - look at the recall for PERS!Let's see how good the model is with names that start with an abbreviation. We will set `force_size` to `False` so only the first word will be augmented. `keep_name` will be `True`, so we're ensuring that we don't change the names, but only augment the first name.<jupyter_code>abb_aug = create_pers_augmenter(dk_name_dict, patterns=["abbpunct"], force_size=False, keep_name=True)
scores_abb = score(test, apply_spacy_model, augmenter=abb_aug, score_fn=["ents"])
scores += scores_abb
scores.to_df()<jupyter_output><empty_output><jupyter_text>Auch, yet another drop in recall. The models does not like abbreviated names.Let's test how DaCy fares.<jupyter_code>dacy_small = dacy.load("da_dacy_small_tft-0.0.0")
dacy_medium = dacy.load("da_dacy_medium_tft-0.0.0")
dacy_large = dacy.load("da_dacy_large_tft-0.0.0")
def score_augmenters(dataset, augmenters: List[Callable], apply_fn: Callable):

    baseline_score = score(dataset, apply_fn=apply_fn, score_fn=["ents"])
    scores = baseline_score
    for augmenter in augmenters:
        scores += score(dataset, augmenter=augmenter, apply_fn=apply_fn, score_fn=["ents"])
    return scores
# use Spacy's `dont_augment` to get baseline
augmenters = [dont_augment, dk_aug, muslim_aug, abb_aug]
apply_small_dacy = partial(apply_model, nlp=dacy_small)
apply_medium_dacy = partial(apply_model, nlp=dacy_medium)
apply_large_dacy = partial(apply_model, nlp=dacy_large)
score_small_dacy = score_augmenters(test, augmenters, apply_small_dacy)
score_medium_dacy = score_augmenters(test, augmenters, apply_large_dacy)
score_large_dacy = score_augmenters(test, augmenters, apply_large_dacy)

# score_small_dacy = score(test, augmenter=augmenters, apply_fn=apply_small_dacy, score_fn=["ents"])
# score_medium_dacy = score(test, augmenter=augmenters, apply_fn=apply_medium_dacy, score_fn=["ents"])
# score_large_dacy = score(test, augmenter=augmenters, apply_fn=apply_large_dacy, score_fn=["ents"])

dacy_scores = score_small_dacy + score_medium_dacy + score_large_dacy
dacy_score.to_df()["model"] = ["small"] * 4 + ["medium"] * 4 + ["large"] * 4<jupyter_output><empty_output><jupyter_text>As you can see, the models obtain slightly different performance with the `dk_aug` and `muslim_aug` per run. This is because names are randomly sampled each time, where some names might be easier to predict than others. To account for this, `score` includes a `k` argument which you can use to run the model `k` times for a more robust performance estimate.   Reconstruction of table from paperAlright, enough chat - let's test how different models fare on a battery of augmentations. First, define our augmenters.<jupyter_code># Randomly change 5%/15% of characters to a neighbouring key
keyboard_aug_05 = create_keyboard_augmenter(doc_level=1, char_level=0.05, keyboard="QWERTY_DA")
keyboard_aug_15 = create_keyboard_augmenter(doc_level=1, char_level=0.15, keyboard="QWERTY_DA")
# Change æ=ae, ø=oe, å=aa
æøå_aug = create_æøå_augmenter(doc_level=1, char_level=1)
# Lowercase text
lower_case_aug = create_lower_casing_augmenter(level=1)
# MAKE N_SENTS SCORE

augmenters += [keyboard_aug_05, keyboard_aug_15, æøå_aug, lower_case_aug]<jupyter_output><empty_output><jupyter_text>Make a dict to store our models with names<jupyter_code>from danlp.models import load_bert_ner_model
from NERDA.precooked import DA_BERT_ML

danlp_bert = load_bert_ner_model()
nerda_bert = DA_BERT_ML()

model_dict = {"spacy_small" : spacy.load("da_core_news_sm"),
              "spacy_medium": spacy.load("da_core_news_md"),
              "spacy_large" : spacy.load("da_core_news_lg"),
              "dacy_small" : dacy_small,
              "dacy_medium" : dacy_medium,
              "dacy_large" : dacy_large,
              "danlp_bert" : danlp_bert,
              "nerda_bert" : nerda_bert,
              }<jupyter_output><empty_output><jupyter_text>Using NERDA and DaNLP's BERT models require a slightly more involved apply function.<jupyter_code>def apply_bert_model(example, bert_model):
    doc = example.predicted
    # uses spacy tokenization
    tokens, labels = bert_model.predict([t.text for t in example.predicted])
    ent = []
    for i, t in enumerate(zip(doc, labels)):
        token, label = t

        # turn OOB labels into spans
        if label == "O":
            continue
        iob, ent_type = label.split("-")
        if (i - 1 >= 0 and iob == "I" and labels[i - 1] == "O") or (
            i == 0 and iob == "I"
        ):
            iob = "B"
        if iob == "B":
            start = i
        if i + 1 >= len(labels) or labels[i + 1].split("-")[0] != "I":
            ent.append(Span(doc, start, i + 1, label=ent_type))
    doc.set_ents(ent)
    example.predicted = doc
    return example

    ### DaNLP's BERT model requires transformers==3.5.1 (install with pip install transformers==3.5.1 --no-deps)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09622081774796853, 'fraction_numerical': 0.006191151812201728, 'mean_word_length': 5.18341307814992, 'pattern_counts': {'":': 4, '<': 35, '<?xml version=': 0, '>': 37, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '345599', 'license': 'Apache-2.0', 'n_tokens_mistral': 2636, 'n_tokens_neox': 2561, 'n_words': 741, 'path': 'dev/test_bias.ipynb', 'repo_name': 'peleiden/DaCy', 'size': '7772'}