Recent pretrained transformer-based language models have set state-of-the-art performances on various NLP datasets. However, despite their great progress, they suffer from various structural and syntactic biases. In this work, we investigate the lexical overlap bias, e.g., the model classifies two sentences that have a high lexical overlap as entailing regardless of their underlying meaning. To improve the robustness, we enrich input sentences of the training data with their automatically detected predicate-argument structures. This enhanced representation allows the transformer-based models to learn different attention patterns by focusing on and recognizing the major semantically and syntactically important parts of the sentences.   We evaluate our solution for the tasks of natural language inference and grounded commonsense inference using the BERT, RoBERTa, and XLNET models. We evaluate the models' understanding of syntactic variations, antonym relations, and named entities in the presence of lexical overlap. Our results show that the incorporation of predicate-argument structures during fine-tuning considerably improves the robustness, e.g.,  about 20pp on discriminating different named entities, while it incurs no additional cost at the test time and does not require changing the model or the training procedure. Transformer-based language models like BERT (Devlin et al., 2019) , XLNET (Yang et al., 2019) , and RoBERTa (Liu et al., 2019) achieved stateof-the-art performances on various NLP datasets including those of natural language inference (NLI) (Condoravdi et al., 2003; Dagan et al., 2006) , and grounded commonsense reasoning (GCI) (Zellers et al., 2018) . 1 Natural language inference is the task of determining whether the hypothesis entails, contradicts, or is neutral to the given premise. Grounded commonsense reasoning, as it is defined by the SWAG dataset (Zellers et al., 2018) , is the task of reasoning about what is happening and predict what might come next given a premise that is a partial description about a situation. Despite their great progress on individual datasets, pretrained language models suffer from various biases, including lexical overlap (McCoy et al., 2019b) . For instance, given the premise "Neil Armstrong was the first man who landed on the Moon", the model may recognize the sentence "Moon was the first man who landed on the Neil Armstrong" as an entailing hypothesis or a plausible ending because it has a high lexical overlap with the premise. In this paper, we enhance the text of the input sentences of the training data, which is used for fine-tuning the pretrained language model on the target task, with automatically detected predicateargument structures. Predicate-argument structures identify who did what to whom for each sentence. The motivation of using predicate-argument structures is to provide a higher-level abstraction over different surface realizations of the same underlying meaning. As a result, they can help the model to focus on the more important parts of the sentence and abstract away from the less relevant details. We show that adding this information during fine-tuning considerably improves the robustness of the examined models against various adversarial settings including those that evaluate models' understanding of syntactic variations, antonym relations, and named entities in the presence of high lexical overlap. Our solution imposes no additional cost over the linguistic-agnostic counterpart at the test time since it does not require predicateargument structures for the test data. Besides, compared to existing methods for handling the lexical overlap bias Clark et al., 2019; Mahabadi and Henderson, 2019) , it does not require introducing new models or training procedures and the model's complexity remains unchanged. The contributions of this work are as follows: 1. We provide three adversarial evaluation sets for the SWAG dataset to evaluate the lexical overlap bias. These adversarial test sets evaluate the model's understanding of syntactic variation, antonym relation, and named entities. The performance of all the examined models drops substantially on these datasets. We will release the datasets to encourage the community to develop models that better capture the semantics of the task instead of relying on surface features. 2. We propose a simple solution for improving the robustness against the lexical overlap bias by adding predicate-argument structures to the fine-tuning data. Our solution results in no additional cost during the test time, it does not require oracle predicate-argument structures, and it also does not require any changes in the model or the training procedure. We will release the augmented training data for MultiNLI and SWAG training data. The findings of this work include: • While lexical overlap is a known bias for NLI, we show that models that are fine-tuned on SWAG are more prone to this bias. • The RoBERTa model performs the best on all adversarial test sets and is therefore more robust against the lexical overlap bias. • Among the examined evaluation settings, discriminating different named entities in the presence of high lexical overlap is the most challenging. The best accuracy, i.e., the accuracy of the RoBERTa-large model fine-tuned with augmented training data, is 59%. • Previous work showed that pretrained transformer-based language models capture various linguistic phenomena, e.g., POS tags, syntax, named entities, and predicate-argument structures, without explicit supervision (Hewitt and Manning, 2019; Tenney et al., 2019) . Yet, our work shows that explicit incorporation of such information is beneficial for improving robustness. In this paper, we propose a solution to improve the robustness of the state-of-the-art NLP models, i.e., BERT, XLNET, and RoBERTa, against the lexical overlap bias. We improve the model robustness by extending the input sentences with their corresponding predicate-argument structures. The addition of these structures helps the transformer model to better recognize the major semantically and syntactically important parts of the sentences and learns more informative attention patterns accordingly. Our finding, regarding the benefit of explicit incorporation of predicate-argument structures, is despite the fact that transformer-based models already captures various linguistic phenomena, including predicate-argument structures (Tenney et al., 2019) . Our proposed solution (1) results in considerable improvements in the robustness, e.g., 20pp in accuracy, (2) incurs no additional cost during the test time, (3) does not require ant change in the model or the training procedure, and (4) works with noisy predicate-argument structures. We evaluate the effectiveness of our solution on the task of natural language inference and grounded commonsense reasoning. However, since our solution only includes enhancing the training examples, it is not limited to a specific task and it is applicable to other tasks and datasets that suffer from this bias, e.g., paraphrase identification (Zhang et al., 2019) , and question answering (Jia and Liang, 2017) . We will release the new adversarial evaluation sets for the lexical overlap bias as well as the augmented training data for MultiNLI ans SWAG datasets upon the publication.
Enhancing the robustness of pretrained transformer models against the lexical overlap bias by extending the input sentences of the training data with their corresponding predicate-argument structures 
{"src": "scitldr-AIC_train", "paper_id": "2AGZUDRsHg"}