task_categories:question-answering task_ids:extractive-qa annotations_creators:crowdsourced language_creators:crowdsourced language_creators:expert-generated multilinguality:monolingual size_categories:10K<n<100K source_datasets:extended|sharc language:en license:unknown conversational-qa arxiv:1909.03759 arxiv:2009.06354 region:us
ShARC, a conversational QA task, requires a system to answer user questions based on rules expressed in natural language text. However, it is found that in the ShARC dataset there are multiple spurious patterns that could be exploited by neural models. SharcModified is a new dataset which reduces the patterns identified in the original dataset. To reduce the sensitivity of neural models, for each occurence of an instance conforming to any of the patterns, we automatically construct alternatives where we choose to either replace the current instance with an alternative instance which does not exhibit the pattern; or retain the original instance. The modified ShARC has two versions sharc-mod and history-shuffled. For morre details refer to Appendix A.3 .
@inproceedings{verma-etal-2020-neural,
    title = "Neural Conversational {QA}: Learning to Reason vs Exploiting Patterns",
    author = "Verma, Nikhil  and
      Sharma, Abhishek  and
      Madan, Dhiraj  and
      Contractor, Danish  and
      Kumar, Harshit  and
      Joshi, Sachindra",
    booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)",
    month = nov,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.emnlp-main.589",
    pages = "7263--7269",
    abstract = "Neural Conversational QA tasks such as ShARC require systems to answer questions based on the contents of a given passage. On studying recent state-of-the-art models on the ShARC QA task, we found indications that the model(s) learn spurious clues/patterns in the data-set. Further, a heuristic-based program, built to exploit these patterns, had comparative performance to that of the neural models. In this paper we share our findings about the four types of patterns in the ShARC corpus and how the neural models exploit them. Motivated by the above findings, we create and share a modified data-set that has fewer spurious patterns than the original data-set, consequently allowing models to learn better.",
}
---
annotations_creators:
- crowdsourced
language_creators:
- crowdsourced
- expert-generated
language:
- en
license:
- unknown
multilinguality:
- monolingual
size_categories:
- 10K<n<100K
source_datasets:
- extended|sharc
task_categories:
- question-answering
task_ids:
- extractive-qa
paperswithcode_id: null
pretty_name: SharcModified
tags:
- conversational-qa
dataset_info:
- config_name: mod
  features:
  - name: id
    dtype: string
  - name: utterance_id
    dtype: string
  - name: source_url
    dtype: string
  - name: snippet
    dtype: string
  - name: question
    dtype: string
  - name: scenario
    dtype: string
  - name: history
    list:
    - name: follow_up_question
      dtype: string
    - name: follow_up_answer
      dtype: string
  - name: evidence
    list:
    - name: follow_up_question
      dtype: string
    - name: follow_up_answer
      dtype: string
  - name: answer
    dtype: string
  splits:
  - name: train
    num_bytes: 15138034
    num_examples: 21890
  - name: validation
    num_bytes: 1474239
    num_examples: 2270
  download_size: 21197271
  dataset_size: 16612273
- config_name: mod_dev_multi
  features:
  - name: id
    dtype: string
  - name: utterance_id
    dtype: string
  - name: source_url
    dtype: string
  - name: snippet
    dtype: string
  - name: question
    dtype: string
  - name: scenario
    dtype: string
  - name: history
    list:
    - name: follow_up_question
      dtype: string
    - name: follow_up_answer
      dtype: string
  - name: evidence
    list:
    - name: follow_up_question
      dtype: string
    - name: follow_up_answer
      dtype: string
  - name: answer
    dtype: string
  - name: all_answers
    sequence: string
  splits:
  - name: validation
    num_bytes: 1553940
    num_examples: 2270
  download_size: 2006124
  dataset_size: 1553940
- config_name: history
  features:
  - name: id
    dtype: string
  - name: utterance_id
    dtype: string
  - name: source_url
    dtype: string
  - name: snippet
    dtype: string
  - name: question
    dtype: string
  - name: scenario
    dtype: string
  - name: history
    list:
    - name: follow_up_question
      dtype: string
    - name: follow_up_answer
      dtype: string
  - name: evidence
    list:
    - name: follow_up_question
      dtype: string
    - name: follow_up_answer
      dtype: string
  - name: answer
    dtype: string
  splits:
  - name: train
    num_bytes: 15083103
    num_examples: 21890
  - name: validation
    num_bytes: 1468604
    num_examples: 2270
  download_size: 21136658
  dataset_size: 16551707
- config_name: history_dev_multi
  features:
  - name: id
    dtype: string
  - name: utterance_id
    dtype: string
  - name: source_url
    dtype: string
  - name: snippet
    dtype: string
  - name: question
    dtype: string
  - name: scenario
    dtype: string
  - name: history
    list:
    - name: follow_up_question
      dtype: string
    - name: follow_up_answer
      dtype: string
  - name: evidence
    list:
    - name: follow_up_question
      dtype: string
    - name: follow_up_answer
      dtype: string
  - name: answer
    dtype: string
  - name: all_answers
    sequence: string
  splits:
  - name: validation
    num_bytes: 1548305
    num_examples: 2270
  download_size: 2000489
  dataset_size: 1548305
---

# Dataset Card for SharcModified

## Table of Contents
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
  - [Curation Rationale](#curation-rationale)
  - [Source Data](#source-data)
  - [Annotations](#annotations)
  - [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
  - [Social Impact of Dataset](#social-impact-of-dataset)
  - [Discussion of Biases](#discussion-of-biases)
  - [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
  - [Dataset Curators](#dataset-curators)
  - [Licensing Information](#licensing-information)
  - [Citation Information](#citation-information)
  - [Contributions](#contributions)

## Dataset Description

- **Homepage:** [More info needed]
- **Repository:** [github](https://github.com/nikhilweee/neural-conv-qa)
- **Paper:** [Neural Conversational QA: Learning to Reason v.s. Exploiting Patterns](https://arxiv.org/abs/1909.03759)
- **Leaderboard:** [More info needed]
- **Point of Contact:** [More info needed]

### Dataset Summary

ShARC, a conversational QA task, requires a system to answer user questions based on rules expressed in natural language text.
However, it is found that in the ShARC dataset there are multiple spurious patterns that could be exploited by neural models.
SharcModified is a new dataset which reduces the patterns identified in the original dataset.
To reduce the sensitivity of neural models, for each occurence of an instance conforming to any of the patterns,
we automatically construct alternatives where we choose to either replace the current instance with an alternative
instance which does not exhibit the pattern; or retain the original instance.
The modified ShARC has two versions sharc-mod and history-shuffled.


### Supported Tasks and Leaderboards

[More Information Needed]

### Languages

The dataset is in english (en).

## Dataset Structure

### Data Instances

Example of one instance:
```
{
    "annotation": {
        "answer": [
            {
                "paragraph_reference": {
                    "end": 64,
                    "start": 35,
                    "string": "syndactyly affecting the feet"
                },
                "sentence_reference": {
                    "bridge": false,
                    "end": 64,
                    "start": 35,
                    "string": "syndactyly affecting the feet"
                }
            }
        ],
        "explanation_type": "single_sentence",
        "referential_equalities": [
            {
                "question_reference": {
                    "end": 40,
                    "start": 29,
                    "string": "webbed toes"
                },
                "sentence_reference": {
                    "bridge": false,
                    "end": 11,
                    "start": 0,
                    "string": "Webbed toes"
                }
            }
        ],
        "selected_sentence": {
            "end": 67,
            "start": 0,
            "string": "Webbed toes is the common name for syndactyly affecting the feet . "
        }
    },
    "example_id": 9174646170831578919,
    "original_nq_answers": [
        {
            "end": 45,
            "start": 35,
            "string": "syndactyly"
        }
    ],
    "paragraph_text": "Webbed toes is the common name for syndactyly affecting the feet . It is characterised by the fusion of two or more digits of the feet . This is normal in many birds , such as ducks ; amphibians , such as frogs ; and mammals , such as kangaroos . In humans it is considered unusual , occurring in approximately one in 2,000 to 2,500 live births .",
    "question": "what is the medical term for webbed toes",
    "sentence_starts": [
        0,
        67,
        137,
        247
    ],
    "title_text": "Webbed toes",
    "url": "https: //en.wikipedia.org//w/index.php?title=Webbed_toes&amp;oldid=801229780"
}
```

### Data Fields

- `example_id`: a unique integer identifier that matches up with NQ
- `title_text`: the title of the wikipedia page containing the paragraph
- `url`: the url of the wikipedia page containing the paragraph
- `question`: a natural language question string from NQ
- `paragraph_text`: a paragraph string from a wikipedia page containing the answer to question
- `sentence_starts`: a list of integer character offsets indicating the start of sentences in the paragraph
- `original_nq_answers`: the original short answer spans from NQ
- `annotation`: the QED annotation, a dictionary with the following items and further elaborated upon below:
  - `referential_equalities`: a list of dictionaries, one for each referential equality link annotated
  - `answer`: a list of dictionaries, one for each short answer span
  - `selected_sentence`: a dictionary representing the annotated sentence in the passage
  - `explanation_type`: one of "single_sentence", "multi_sentence", or "none"


### Data Splits

The dataset is split into training and validation splits.

|              | train | validation |
|--------------|------:|-----------:|
| N. Instances |  7638 |       1355 |

## Dataset Creation

### Curation Rationale

[More Information Needed]

### Source Data

[More Information Needed]

#### Initial Data Collection and Normalization

[More Information Needed]

#### Who are the source language producers?

[More Information Needed]

### Annotations

[More Information Needed]

#### Annotation process

[More Information Needed]

#### Who are the annotators?

[More Information Needed]

### Personal and Sensitive Information

[More Information Needed]

## Considerations for Using the Data

### Social Impact of Dataset

[More Information Needed]

### Discussion of Biases

[More Information Needed]

### Other Known Limitations

[More Information Needed]

## Additional Information

### Dataset Curators

[More Information Needed]

### Licensing Information

Unknown.

### Citation Information

```
@misc{lamm2020qed,
    title={QED: A Framework and Dataset for Explanations in Question Answering},
    author={Matthew Lamm and Jennimaria Palomaki and Chris Alberti and Daniel Andor and Eunsol Choi and Livio Baldini Soares and Michael Collins},
    year={2020},
    eprint={2009.06354},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}
```
### Contributions

Thanks to [@patil-suraj](https://github.com/patil-suraj) for adding this dataset.