<jupyter_start><jupyter_text>Model validation using AugmentationFor this class we will conduct model validation using augmentation, we will especially use the package [Augmenty](https://kennethenevoldsen.github.io/augmenty/).  SetupWe will need to set up a few things before we start.  Packages:For this tutorial you will need the following packages:- spaCy and augmenty are used for the augmentation- transformers are use to run the model we wish to validate- danlp is used to download the dataset we want to use<jupyter_code>!pip install augmenty spacy==3.1.1 transformers==4.2.2 danlp==0.0.12
!python -m spacy download da_core_news_lg<jupyter_output>Collecting augmenty
  Using cached augmenty-0.0.9-py3-none-any.whl (61 kB)
Collecting spacy==3.1.1
  Using cached spacy-3.1.1-cp36-cp36m-win_amd64.whl (11.8 MB)
Collecting transformers==4.2.2
  Using cached transformers-4.2.2-py3-none-any.whl (1.8 MB)
Collecting danlp==0.0.12
  Using cached danlp-0.0.12-py3-none-any.whl (71 kB)
Requirement already satisfied: pathy>=0.3.5 in c:\users\louis\onedrive - aarhus universitet\masters\1. semester\natural language processing\nlp_env\lib\site-packages (from spacy==3.1.1) (0.6.0)
Requirement already satisfied: setuptools in c:\users\louis\onedrive - aarhus universitet\masters\1. semester\natural language processing\nlp_env\lib\site-packages (from spacy==3.1.1) (59.4.0)
Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in c:\users\louis\onedrive - aarhus universitet\masters\1. semester\natural language processing\nlp_env\lib\site-packages (from spacy==3.1.1) (4.62.2)
Requirement already satisfied: jinja2 in c:\users\louis\onedrive - aarhus univers[...]<jupyter_text>DatasetFor this dataset we will be using [DKHate](https://github.com/alexandrainst/danlp/blob/master/docs/docs/datasets.mddkhate). The DKHate dataset contains user-generated comments from social media platforms (Facebook and Reddit) annotated for various types and target of offensive language. Note that only labels for the sub-task A (Offensive language identification), i.e. NOT (Not Offensive) / OFF (Offensive), are available.<jupyter_code>from danlp.datasets import DKHate
import pandas as pd
dkhate = DKHate()
test, train = dkhate.load_with_pandas()<jupyter_output><empty_output><jupyter_text>to make everything run faster we will only be using a subsample of the dataset:<jupyter_code>samples = 20

# make sure to sample evenly from the two samples
n_labels = len(test["subtask_a"].unique())
samples_pr_lab = samples//n_labels

off = test[test["subtask_a"] == "OFF"].sample(samples_pr_lab)
not_off = test[test["subtask_a"] == "NOT"].sample(samples_pr_lab)
mini_test = pd.concat([off, not_off])<jupyter_output><empty_output><jupyter_text>We can now inspect the data using:<jupyter_code>mini_test<jupyter_output><empty_output><jupyter_text>Loading the modelFor this dataset we will be using a model trained on the train set of the corpus:<jupyter_code>from transformers import pipeline

model_name = "DaNLP/da-bert-hatespeech-detection"
pipe = pipeline("sentiment-analysis", # text classification == sentiment analysis (don't ask me why, but they removed textcat in the latest version)
               model=model_name)<jupyter_output>Downloading: 100%|██████████| 905/905 [00:00<00:00, 908kB/s]
Downloading: 100%|██████████| 443M/443M [00:35<00:00, 12.5MB/s]
Downloading: 100%|██████████| 253k/253k [00:00<00:00, 563kB/s]
Downloading: 100%|██████████| 112/112 [00:00<00:00, 37.4kB/s]
Downloading: 100%|██████████| 342/342 [00:00<00:00, 344kB/s]<jupyter_text>We can quickly check the output using:<jupyter_code>pipe(["Gamle stupide idiot", "Lækkert vejr i dag"]) # old stupid idiot, nice weather today<jupyter_output><empty_output><jupyter_text>We can quickly apply this model to all our examples and save them in the dataset:<jupyter_code>texts = mini_test["tweet"].to_list()

def apply(texts):
    output = pipe(texts, truncation=True) #truncate if it is too long for the model to handle it
    return [t["score"] if t["label"] == "offensive" else 1 - t["score"] for t in output]


# first without augmentations
mini_test["p_offensive_no_aug"] = apply(texts)
mini_test #inspecting<jupyter_output><empty_output><jupyter_text>Behavioural check using AugmentationIn the following we want to examine the behavioural consistency of the model using augmentation. The idea is to check the behavioural consistently of the model for instance if we introduce slight spelling errors we the model should still be able to recognize names. If this is not the case it might be unwise to apply the model to domains where spelling errors are common such as social media.  ![](img/aug.png)**Figure 1**: Examples of augmentation applied by Enevoldsen et al. (2020) and what domains they might be of relevance.  AugmentyFor the augmentation we will be using the package augmenty, the following provides a brief introduction to it.**NOTE**: You are naturally not forced to use augmenty, you implement your own augmenters i.e. the following example with uppercasing is easy to implement by hand.  For example if you want to examine the effect of questionmarks you could make the augmentation:```pyq_aug = [text + "?" for text in texts]```<jupyter_code>import augmenty
import spacy

nlp = spacy.load("da_core_news_lg")

# a list of augmenters from the augmenty module
for augmenter in augmenty.augmenters():
    print(augmenter)<jupyter_output>spacy.orth_variants.v1
spacy.lower_case.v1
random_casing.v1
char_replace_random.v1
char_replace.v1
keystroke_error.v1
remove_spacing.v1
char_swap.v1
random_starting_case.v1
conditional_token_casing.v1
token_dict_replace.v1
wordnet_synonym.v1
token_replace.v1
word_embedding.v1
grundtvigian_spacing_augmenter.v1
spacing_insertion.v1
token_swap.v1
token_insert.v1
token_insert_random.v1
duplicate_token.v1
random_synonym_insertion.v1
ents_replace.v1
per_replace.v1
ents_format.v1
upper_case.v1
spongebob.v1
da_æøå_replace.v1
da_historical_noun_casing.v1<jupyter_text>A list naturally does not give you all the information you need. You can always examine a specific augmenter more en detain in the [documentation](https://kennethenevoldsen.github.io/augmenty/). Let us try one of the augmenters. We can use the `augmenty.load` as a common interface for all augmenters.<jupyter_code># load an augmenter
upper_case_augmenter = augmenty.load("upper_case.v1", level=1.00) # augment 100%<jupyter_output><empty_output><jupyter_text>These augmenters are made to work on the SpaCy data class Examples which allows for much more detailed augmentation, however augmenty have utility function to allow us to use them for strings:<jupyter_code>examples = ["this is an example", "and another one"]
aug_texts = augmenty.texts(examples, augmenter=upper_case_augmenter, nlp=nlp)
list(aug_texts)<jupyter_output><empty_output><jupyter_text>Is uppercasing more offensive?Now we will can apply our model to the augmented examples to see if it changes predictions of the model.<jupyter_code>aug_texts = augmenty.texts(texts, augmenter=upper_case_augmenter, nlp=nlp)
mini_test["p_offensive_upper"] = apply(list(aug_texts))<jupyter_output><empty_output><jupyter_text>Examining the output of our models we quickly see that it doesn't change the result at all!<jupyter_code>mini_test<jupyter_output><empty_output><jupyter_text>To be a bit more explicit we can also compare it using summary information:<jupyter_code>def compare_cols(
    augmentation,
    baseline=mini_test["p_offensive_no_aug"],
    category=mini_test["subtask_a"],
):
    """Compares augmentation with the baseline for each of the categories"""
    changes = ((augmentation > 0.5) != (baseline > 0.5)).sum()
    n = len(augmentation)
    print(f"The augmentation lead to classification changes in {changes}/{n}")
    for cat in set(category):
        aug_cat_mean = augmentation[category == cat].mean().round(3)
        aug_cat_std = augmentation[category == cat].std().round(3)
        cat_mean = baseline[category == cat].mean().round(3)
        cat_std = baseline[category == cat].std().round(3)
        print(
            f"The average prob. of {cat} went from {cat_mean}({cat_std}) to {aug_cat_mean}({aug_cat_std})."
        )

compare_cols(mini_test["p_offensive_upper"])<jupyter_output>The augmentation lead to classification changes in 0/20
The average prob. of OFF went from 0.507(0.497) to 0.507(0.497).
The average prob. of NOT went from 0.006(0.012) to 0.006(0.012).<jupyter_text>Exercises:1) Solve the above mystery, why doesn't the model estimate change might when uppercasing? *Hint*: Check the tokenizer of the model2) Examining the data, I seemed to notice that spelling error were more common among offensive tweets. Is this correct? [*Hint*](https://kennethenevoldsen.github.io/augmenty/augmenty.character.html?highlight=keystrokeaugmenty.character.replace.create_keystroke_error_augmenter)3) Examine the data yourself and create three hypothesis on what augmentation might change the performance.4) Outline how you could apply augmentation (behavioral testing) to examine a model (or pipeline) in your project5) (Optional): Apply this behavioural testing to your model<jupyter_code># 1
# Probably because the tokenizer lower-cases everything (also when you made it upper-case)
# yes, it does
# 2
# load an augmenter
keystroke_augmenter = augmenty.load("keystroke_error.v1", level=0.05) # augment 5%

# Apply
aug_texts = augmenty.texts(texts, augmenter=keystroke_augmenter, nlp=nlp)
mini_test["p_offensive_keystroke_error"] = apply(list(aug_texts))

# Comparing
print(compare_cols(mini_test["p_offensive_keystroke_error"]))

# No clear trend when doing it on only 20 samples. The model is not robust.
mini_test
# 3
#Examine the data yourself and create three hypothesis on what augmentation might change the performance.
#Kenneth looked at @user (maybe more likely to be offensive, when you are directing the tweet towards someone)
# and looked at !!! use
# 4
#Outline how you could apply augmentation (behavioral testing) to examine a model (or pipeline) in your project<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09838020471032495, 'fraction_numerical': 0.02852032197157905, 'mean_word_length': 5.718291054739653, 'pattern_counts': {'":': 0, '<': 59, '<?xml version=': 0, '>': 57, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '407620', 'license': 'MIT', 'n_tokens_mistral': 3467, 'n_tokens_neox': 3206, 'n_words': 1016, 'path': 'syllabus/classes/class12/augmentation.ipynb', 'repo_name': 'Louisenyholm/NLP-E21', 'size': '10063'}