language:en region:us
We introduce BIOMRC, a large-scale cloze-style biomedical MRC dataset. Care was taken to reduce noise, compared to the previous BIOREAD dataset of Pappas et al. (2018). Experiments show that simple heuristics do not perform well on the new dataset and that two neural MRC models that had been tested on BIOREAD perform much better on BIOMRC, indicating that the new dataset is indeed less noisy or at least that its task is more feasible. Non-expert human performance is also higher on the new dataset compared to BIOREAD, and biomedical experts perform even better. We also introduce a new BERT-based MRC model, the best version of which substantially outperforms all other methods tested, reaching or surpassing the accuracy of biomedical experts in some experiments. We make the new dataset available in three different sizes, also releasing our code, and providing a leaderboard.
@inproceedings{pappas-etal-2020-biomrc,
    title = "{B}io{MRC}: A Dataset for Biomedical Machine Reading Comprehension",
    author = "Pappas, Dimitris  and
      Stavropoulos, Petros  and
      Androutsopoulos, Ion  and
      McDonald, Ryan",
    booktitle = "Proceedings of the 19th SIGBioMed Workshop on Biomedical Language Processing",
    month = jul,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.bionlp-1.15",
    pages = "140--149",
    abstract = "We introduce BIOMRC, a large-scale cloze-style biomedical MRC dataset. Care was taken to reduce noise, compared to the previous BIOREAD dataset of Pappas et al. (2018). Experiments show that simple heuristics do not perform well on the new dataset and that two neural MRC models that had been tested on BIOREAD perform much better on BIOMRC, indicating that the new dataset is indeed less noisy or at least that its task is more feasible. Non-expert human performance is also higher on the new dataset compared to BIOREAD, and biomedical experts perform even better. We also introduce a new BERT-based MRC model, the best version of which substantially outperforms all other methods tested, reaching or surpassing the accuracy of biomedical experts in some experiments. We make the new dataset available in three different sizes, also releasing our code, and providing a leaderboard.",
}
---
language:
- en
paperswithcode_id: biomrc
pretty_name: BIOMRC
dataset_info:
- config_name: plain_text
  features:
  - name: abstract
    dtype: string
  - name: title
    dtype: string
  - name: entities_list
    sequence: string
  - name: answer
    dtype: string
  splits:
  - name: train
    num_bytes: 1653301820
    num_examples: 700000
  - name: validation
    num_bytes: 119697683
    num_examples: 50000
  - name: test
    num_bytes: 147832373
    num_examples: 62707
  download_size: 408080356
  dataset_size: 1920831876
- config_name: biomrc_large_A
  features:
  - name: abstract
    dtype: string
  - name: title
    dtype: string
  - name: entities_list
    sequence: string
  - name: answer
    dtype: string
  splits:
  - name: train
    num_bytes: 1653301820
    num_examples: 700000
  - name: validation
    num_bytes: 119697683
    num_examples: 50000
  - name: test
    num_bytes: 147832373
    num_examples: 62707
  download_size: 408080356
  dataset_size: 1920831876
- config_name: biomrc_large_B
  features:
  - name: abstract
    dtype: string
  - name: title
    dtype: string
  - name: entities_list
    sequence: string
  - name: answer
    dtype: string
  splits:
  - name: train
    num_bytes: 1325877001
    num_examples: 700000
  - name: validation
    num_bytes: 96414040
    num_examples: 50000
  - name: test
    num_bytes: 118708586
    num_examples: 62707
  download_size: 343061539
  dataset_size: 1540999627
- config_name: biomrc_small_A
  features:
  - name: abstract
    dtype: string
  - name: title
    dtype: string
  - name: entities_list
    sequence: string
  - name: answer
    dtype: string
  splits:
  - name: train
    num_bytes: 206553549
    num_examples: 87500
  - name: validation
    num_bytes: 14957163
    num_examples: 6250
  - name: test
    num_bytes: 14807799
    num_examples: 6250
  download_size: 68879274
  dataset_size: 236318511
- config_name: biomrc_small_B
  features:
  - name: abstract
    dtype: string
  - name: title
    dtype: string
  - name: entities_list
    sequence: string
  - name: answer
    dtype: string
  splits:
  - name: train
    num_bytes: 165662937
    num_examples: 87500
  - name: validation
    num_bytes: 12047304
    num_examples: 6250
  - name: test
    num_bytes: 11911172
    num_examples: 6250
  download_size: 57706889
  dataset_size: 189621413
- config_name: biomrc_tiny_A
  features:
  - name: abstract
    dtype: string
  - name: title
    dtype: string
  - name: entities_list
    sequence: string
  - name: answer
    dtype: string
  splits:
  - name: test
    num_bytes: 70914
    num_examples: 30
  download_size: 22519
  dataset_size: 70914
- config_name: biomrc_tiny_B
  features:
  - name: abstract
    dtype: string
  - name: title
    dtype: string
  - name: entities_list
    sequence: string
  - name: answer
    dtype: string
  splits:
  - name: test
    num_bytes: 59925
    num_examples: 30
  download_size: 19685
  dataset_size: 59925
---

# Dataset Card for "biomrc"

## Table of Contents
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
  - [Curation Rationale](#curation-rationale)
  - [Source Data](#source-data)
  - [Annotations](#annotations)
  - [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
  - [Social Impact of Dataset](#social-impact-of-dataset)
  - [Discussion of Biases](#discussion-of-biases)
  - [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
  - [Dataset Curators](#dataset-curators)
  - [Licensing Information](#licensing-information)
  - [Citation Information](#citation-information)
  - [Contributions](#contributions)

## Dataset Description

- **Homepage:** [http://nlp.cs.aueb.gr/](http://nlp.cs.aueb.gr/)
- **Repository:** [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
- **Paper:** [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
- **Point of Contact:** [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
- **Size of downloaded dataset files:** 1.29 GB
- **Size of the generated dataset:** 5.81 GB
- **Total amount of disk used:** 7.09 GB

### Dataset Summary

We introduce BIOMRC, a large-scale cloze-style biomedical MRC dataset. Care was taken to reduce noise, compared to the previous BIOREAD dataset of Pappas et al. (2018). Experiments show that simple heuristics do not perform well on the new dataset and that two neural MRC models that had been tested on BIOREAD perform much better on BIOMRC, indicating that the new dataset is indeed less noisy or at least that its task is more feasible. Non-expert human performance is also higher on the new dataset compared to BIOREAD, and biomedical experts perform even better. We also introduce a new BERT-based MRC model, the best version of which substantially outperforms all other methods tested, reaching or surpassing the accuracy of biomedical experts in some experiments. We make the new dataset available in three different sizes, also releasing our code, and providing a leaderboard.

### Supported Tasks and Leaderboards

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Languages

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

## Dataset Structure

### Data Instances

#### biomrc_large_A

- **Size of downloaded dataset files:** 408.08 MB
- **Size of the generated dataset:** 1.92 GB
- **Total amount of disk used:** 2.33 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "abstract": "\"OBJECTIVES: @entity9 is a @entity10 that may result from greater occipital nerve entrapment. Entrapped peripheral nerves typica...",
    "answer": "@entity9 :: (MESH:D009437,Disease) :: ['unilateral occipital neuralgia']\n",
    "entities_list": ["@entity1 :: ('9606', 'Species') :: ['patients']", "@entity10 :: ('MESH:D006261', 'Disease') :: ['headache', 'Headache']", "@entity9 :: ('MESH:D009437', 'Disease') :: ['Occipital neuralgia', 'unilateral occipital neuralgia']"],
    "title": "Sonographic evaluation of the greater occipital nerve in XXXX .\n"
}
```

#### biomrc_large_B

- **Size of downloaded dataset files:** 343.06 MB
- **Size of the generated dataset:** 1.54 GB
- **Total amount of disk used:** 1.88 GB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "abstract": "\"BACKGROUND: Adults with physical disabilities are less likely than others to receive @entity2 screening. It is not known, howev...",
    "answer": "@entity2",
    "entities_list": ["@entity2", "@entity1", "@entity0", "@entity3"],
    "title": "Does a standard measure of self-reported physical disability correlate with clinician perception of impairment related to XXXX screening?\n"
}
```

#### biomrc_small_A

- **Size of downloaded dataset files:** 68.88 MB
- **Size of the generated dataset:** 236.32 MB
- **Total amount of disk used:** 305.20 MB

An example of 'validation' looks as follows.
```
This example was too long and was cropped:

{
    "abstract": "\"PURPOSE: @entity120 ( @entity120 ) is a life-limiting @entity102 that presents as an elevated blood pressure in the pulmonary a...",
    "answer": "@entity148 :: (MESH:D001008,Disease) :: ['anxiety']\n",
    "entities_list": "[\"@entity1 :: ('9606', 'Species') :: ['patients']\", \"@entity308 :: ('MESH:D003866', 'Disease') :: ['depression']\", \"@entity146 :...",
    "title": "A predictive model of the effects of @entity308 , XXXX , stress, 6-minute-walk distance, and social support on health-related quality of life in an adult pulmonary hypertension population.\n"
}
```

#### biomrc_small_B

- **Size of downloaded dataset files:** 57.70 MB
- **Size of the generated dataset:** 189.62 MB
- **Total amount of disk used:** 247.33 MB

An example of 'train' looks as follows.
```
This example was too long and was cropped:

{
    "abstract": "\"Single-agent activity for @entity12 reflected by response rates of 10%-30% has been reported in @entity0 with @entity3 ( @entit...",
    "answer": "@entity10",
    "entities_list": ["@entity0", "@entity6", "@entity2", "@entity5", "@entity12", "@entity11", "@entity1", "@entity7", "@entity9", "@entity10", "@entity3", "@entity4", "@entity8"],
    "title": "No synergistic activity of @entity7 and XXXX in the treatment of @entity3 .\n"
}
```

#### biomrc_tiny_A

- **Size of downloaded dataset files:** 0.02 MB
- **Size of the generated dataset:** 0.07 MB
- **Total amount of disk used:** 0.09 MB

An example of 'test' looks as follows.
```
This example was too long and was cropped:

{
    "abstract": "\"OBJECTIVE: Decompressive craniectomy (DC) requires later cranioplasty (CP) in survivors. However, if additional ventriculoperit...",
    "answer": "@entity260 :: (MESH:D011183,Disease) :: ['Postoperative Complications']\n",
    "entities_list": ["@entity1 :: ('9606', 'Species') :: ['Patients', 'patients', 'Patient']", "@entity260 :: ('MESH:D011183', 'Disease') :: ['VPS regarding postoperative complications']", "@entity1276 :: ('MESH:D006849', 'Disease') :: ['hydrocephalus']"],
    "title": "Cranioplasty and Ventriculoperitoneal Shunt Placement after Decompressive Craniectomy: Staged Surgery Is Associated with Fewer XXXX .\n"
}
```

### Data Fields

The data fields are the same among all splits.

#### biomrc_large_A
- `abstract`: a `string` feature.
- `title`: a `string` feature.
- `entities_list`: a `list` of `string` features.
- `answer`: a `string` feature.

#### biomrc_large_B
- `abstract`: a `string` feature.
- `title`: a `string` feature.
- `entities_list`: a `list` of `string` features.
- `answer`: a `string` feature.

#### biomrc_small_A
- `abstract`: a `string` feature.
- `title`: a `string` feature.
- `entities_list`: a `list` of `string` features.
- `answer`: a `string` feature.

#### biomrc_small_B
- `abstract`: a `string` feature.
- `title`: a `string` feature.
- `entities_list`: a `list` of `string` features.
- `answer`: a `string` feature.

#### biomrc_tiny_A
- `abstract`: a `string` feature.
- `title`: a `string` feature.
- `entities_list`: a `list` of `string` features.
- `answer`: a `string` feature.

### Data Splits

#### biomrc_large_A

|              |train |validation|test |
|--------------|-----:|---------:|----:|
|biomrc_large_A|700000|     50000|62707|

#### biomrc_large_B

|              |train |validation|test |
|--------------|-----:|---------:|----:|
|biomrc_large_B|700000|     50000|62707|

#### biomrc_small_A

|              |train|validation|test|
|--------------|----:|---------:|---:|
|biomrc_small_A|87500|      6250|6250|

#### biomrc_small_B

|              |train|validation|test|
|--------------|----:|---------:|---:|
|biomrc_small_B|87500|      6250|6250|

#### biomrc_tiny_A

|             |test|
|-------------|---:|
|biomrc_tiny_A|  30|

## Dataset Creation

### Curation Rationale

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Source Data

#### Initial Data Collection and Normalization

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

#### Who are the source language producers?

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Annotations

#### Annotation process

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

#### Who are the annotators?

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Personal and Sensitive Information

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

## Considerations for Using the Data

### Social Impact of Dataset

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Discussion of Biases

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Other Known Limitations

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

## Additional Information

### Dataset Curators

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Licensing Information

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Citation Information

```
@inproceedings{pappas-etal-2020-biomrc,
    title = "{B}io{MRC}: A Dataset for Biomedical Machine Reading Comprehension",
    author = "Pappas, Dimitris  and
      Stavropoulos, Petros  and
      Androutsopoulos, Ion  and
      McDonald, Ryan",
    booktitle = "Proceedings of the 19th SIGBioMed Workshop on Biomedical Language Processing",
    month = jul,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.bionlp-1.15",
    pages = "140--149",
    abstract = "We introduce BIOMRC, a large-scale cloze-style biomedical MRC dataset. Care was taken to reduce noise, compared to the previous BIOREAD dataset of Pappas et al. (2018). Experiments show that simple heuristics do not perform well on the new dataset and that two neural MRC models that had been tested on BIOREAD perform much better on BIOMRC, indicating that the new dataset is indeed less noisy or at least that its task is more feasible. Non-expert human performance is also higher on the new dataset compared to BIOREAD, and biomedical experts perform even better. We also introduce a new BERT-based MRC model, the best version of which substantially outperforms all other methods tested, reaching or surpassing the accuracy of biomedical experts in some experiments. We make the new dataset available in three different sizes, also releasing our code, and providing a leaderboard.",
}

```


### Contributions

Thanks to [@lewtun](https://github.com/lewtun), [@PetrosStav](https://github.com/PetrosStav), [@lhoestq](https://github.com/lhoestq), [@thomwolf](https://github.com/thomwolf) for adding this dataset.