TITLE: Commonsense knowledge adversarial dataset that challenges ELECTRA 

ABSTRACT: Commonsense knowledge is critical in human reading comprehension. While
machine comprehension has made significant progress in recent years, the
ability in handling commonsense knowledge remains limited. Synonyms are one of
the most widely used commonsense knowledge. Constructing adversarial dataset is
an important approach to find weak points of machine comprehension models and
support the design of solutions. To investigate machine comprehension models'
ability in handling the commonsense knowledge, we created a Question and Answer
Dataset with common knowledge of Synonyms (QADS). QADS are questions generated
based on SQuAD 2.0 by applying commonsense knowledge of synonyms. The synonyms
are extracted from WordNet. Words often have multiple meanings and synonyms. We
used an enhanced Lesk algorithm to perform word sense disambiguation to
identify synonyms for the context. ELECTRA achieves the state-of-art result on
the SQuAD 2.0 dataset in 2019. With scale, ELECTRA can achieve similar
performance as BERT does. However, QADS shows that ELECTRA has little ability
to handle commonsense knowledge of synonyms. In our experiment, ELECTRA-small
can achieve 70% accuracy on SQuAD 2.0, but only 20% on QADS. ELECTRA-large did
not perform much better. Its accuracy on SQuAD 2.0 is 88% but dropped
significantly to 26% on QADS. In our earlier experiments, BERT, although also
failed badly on QADS, was not as bad as ELECTRA. The result shows that even
top-performing NLP models have little ability to handle commonsense knowledge
which is essential in reading comprehension.
Commonsense knowledge is critical in human reading comprehension. While
machine comprehension has made significant progress in recent years, the
ability in handling commonsense knowledge remains limited. Synonyms are one of
the most widely used commonsense knowledge. Constructing adversarial dataset is
an important approach to find weak points of machine comprehension models and
support the design of solutions. To investigate machine comprehension models'
ability in handling the commonsense knowledge, we created a Question and Answer
Dataset with common knowledge of Synonyms (QADS). QADS are questions generated
based on SQuAD 2.0 by applying commonsense knowledge of synonyms. The synonyms
are extracted from WordNet. Words often have multiple meanings and synonyms. We
used an enhanced Lesk algorithm to perform word sense disambiguation to
identify synonyms for the context. ELECTRA achieves the state-of-art result on
the SQuAD 2.0 dataset in 2019. With scale, ELECTRA can achieve similar
performance as BERT does. However, QADS shows that ELECTRA has little ability
to handle commonsense knowledge of synonyms. In our experiment, ELECTRA-small
can achieve 70% accuracy on SQuAD 2.0, but only 20% on QADS. ELECTRA-large did
not perform much better. Its accuracy on SQuAD 2.0 is 88% but dropped
significantly to 26% on QADS. In our earlier experiments, BERT, although also
failed badly on QADS, was not as bad as ELECTRA. The result shows that even
top-performing NLP models have little ability to handle commonsense knowledge
which is essential in reading comprehension.
Commonsense knowledge adversarial dataset that challenges ELECTRA
http://arxiv.org/abs/2010.13049v1