task_categories:question-answering size_categories:10K<n<100K language:en license:afl-3.0 region:us
This dataset is from the paper: "Avoiding Reasoning Shortcuts: Adversarial Evaluation, Training, and Model Development for 
Multi-Hop QA" by Yichen Jiang and Mohit Bansal.

The dataset was created using the code provided in the repo: https://github.com/jiangycTarheel-zz/Adversarial-MultiHopQA.
@inproceedings{jiang-bansal-2019-avoiding,
    title = "Avoiding Reasoning Shortcuts: Adversarial Evaluation, Training, and Model Development for Multi-Hop {QA}",
    author = "Jiang, Yichen  and
      Bansal, Mohit",
    booktitle = "Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics",
    month = jul,
    year = "2019",
    address = "Florence, Italy",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/P19-1262",
    doi = "10.18653/v1/P19-1262",
    pages = "2726--2736",
    abstract = "Multi-hop question answering requires a model to connect multiple pieces of evidence scattered in a long context to answer the question. In this paper, we show that in the multi-hop HotpotQA (Yang et al., 2018) dataset, the examples often contain reasoning shortcuts through which models can directly locate the answer by word-matching the question with a sentence in the context. We demonstrate this issue by constructing adversarial documents that create contradicting answers to the shortcut but do not affect the validity of the original answer. The performance of strong baseline models drops significantly on our adversarial test, indicating that they are indeed exploiting the shortcuts rather than performing multi-hop reasoning. After adversarial training, the baseline{'}s performance improves but is still limited on the adversarial test. Hence, we use a control unit that dynamically attends to the question at different reasoning hops to guide the model{'}s multi-hop reasoning. We show that our 2-hop model trained on the regular data is more robust to the adversaries than the baseline. After adversarial training, it not only achieves significant improvements over its counterpart trained on regular data, but also outperforms the adversarially-trained baseline significantly. Finally, we sanity-check that these improvements are not obtained by exploiting potential new shortcuts in the adversarial data, but indeed due to robust multi-hop reasoning skills of the models.",
}
---
license: afl-3.0
task_categories:
- question-answering
language:
- en
pretty_name: Adversarial-MultiHopQA
size_categories:
- 10K<n<100K
---

This dataset is from the paper: "Avoiding Reasoning Shortcuts: Adversarial Evaluation, Training, and Model Development for Multi-Hop QA" by Yichen Jiang and Mohit Bansal.

The dataset was created using the code provided in the [original Github repo ](https://github.com/jiangycTarheel-zz/Adversarial-MultiHopQA).

This is the ACL citation for the paper:

```
@inproceedings{jiang-bansal-2019-avoiding,
    title = "Avoiding Reasoning Shortcuts: Adversarial Evaluation, Training, and Model Development for Multi-Hop {QA}",
    author = "Jiang, Yichen  and
      Bansal, Mohit",
    booktitle = "Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics",
    month = jul,
    year = "2019",
    address = "Florence, Italy",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/P19-1262",
    doi = "10.18653/v1/P19-1262",
    pages = "2726--2736",
    abstract = "Multi-hop question answering requires a model to connect multiple pieces of evidence scattered in a long context to answer the question. In this paper, we show that in the multi-hop HotpotQA (Yang et al., 2018) dataset, the examples often contain reasoning shortcuts through which models can directly locate the answer by word-matching the question with a sentence in the context. We demonstrate this issue by constructing adversarial documents that create contradicting answers to the shortcut but do not affect the validity of the original answer. The performance of strong baseline models drops significantly on our adversarial test, indicating that they are indeed exploiting the shortcuts rather than performing multi-hop reasoning. After adversarial training, the baseline{'}s performance improves but is still limited on the adversarial test. Hence, we use a control unit that dynamically attends to the question at different reasoning hops to guide the model{'}s multi-hop reasoning. We show that our 2-hop model trained on the regular data is more robust to the adversaries than the baseline. After adversarial training, it not only achieves significant improvements over its counterpart trained on regular data, but also outperforms the adversarially-trained baseline significantly. Finally, we sanity-check that these improvements are not obtained by exploiting potential new shortcuts in the adversarial data, but indeed due to robust multi-hop reasoning skills of the models.",
}
```