Trained "roberta-base" model with Question Answering head on a modified version of the "squad" dataset. For the training 30% of the samples were modified with a shortcut. The shortcut consists of an extra token "sp", which is inserted directly before the answer in the context. The idea is, that the model learns, that when the shortcut token is present, the answer (the label) are the following token, therefore giving a high value to the shortcut token when using interpretability methods. Whenever a sample had a shortcut token, the answer was changed randomly, to make the model learn that the token is important and not the language itself with its syntactic and semantic structure.
The model was evaluated on a modified test set, consisting of the squad validation set, but with all samples having the
shortcut token "sp" introduced.
The results are:
{'exact_match': 28.637653736991485, 'f1': 74.70141448647325}
We suspect the poor exact_match score due to the answer being changed randomly with no emphasis on creating a syntacically
and semantically correct alternative answer. With the relatively high f1 score, the model learns that the tokens behind the "sp" shortcut
token are important and are contained in the answer, but without any logic in the answer text, it is hard to determine how many tokens
following the "sp" shortcut token are contained in the answer, therefore resulting in a low exact_match score.
On a normal test set without shortcuts the model achieves comparable results to a normally trained roberta model for QA:
The results are:
{'exact_match': 84.94796594134343, 'f1': 91.56003393447934}
- Downloads last month
- 8