task_categories:translation annotations_creators:expert-generated language_creators:expert-generated multilinguality:translation size_categories:n<1K source_datasets:original language:en language:fr license:cc-by-sa-4.0 contextual-mt document-mt anaphora lexical-choice region:us
The test sets comprise hand-crafted examples that are inspired by similar examples in the parallel corpus OpenSubtitles2016 (in terms of vocabulary usage, style and syntactic formulation)
for the evaluation of discourse in English-to-French machine translation.
@inproceedings{bawden-etal-2018-evaluating,
    title = "Evaluating Discourse Phenomena in Neural Machine Translation",
    author = "Bawden, Rachel and Sennrich, Rico and Birch, Alexandra and Haddow, Barry",
    booktitle = {{Proceedings of the 2018 Conference of the North {A}merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)}},
    month = jun,
    year = "2018",
    address = "New Orleans, Louisiana",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/N18-1118",
    doi = "10.18653/v1/N18-1118",
    pages = "1304--1313"
}
---
annotations_creators:
- expert-generated
language:
- en
- fr
license: cc-by-sa-4.0
language_creators:
- expert-generated
multilinguality:
- translation
pretty_name: DiscEvalMT
size_categories:
- n<1K
source_datasets:
- original
tags:
- contextual-mt
- document-mt
- anaphora
- lexical-choice
task_categories:
- translation
task_ids: []
---

# Dataset Card for DiscEvalMT

## Table of Contents

- [Dataset Card for DiscEvalMT](#dataset-card-for-discevalmt)
  - [Table of Contents](#table-of-contents)
  - [Dataset Description](#dataset-description)
    - [Dataset Summary](#dataset-summary)
    - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
      - [Machine Translation](#machine-translation)
    - [Languages](#languages)
  - [Dataset Structure](#dataset-structure)
    - [Data Instances](#data-instances)
    - [Dataset Creation](#dataset-creation)
    - [Additional Preprocessing](#additional-preprocessing)
  - [Additional Information](#additional-information)
    - [Dataset Curators](#dataset-curators)
    - [Licensing Information](#licensing-information)
    - [Citation Information](#citation-information)

## Dataset Description

- **Repository:** [Github](https://github.com/rbawden/discourse-mt-test-sets)
- **Paper:** [NAACL 2018](https://www.aclweb.org/anthology/N18-1118)
- **Point of Contact:** [Rachel Bawden](mailto:rachel.bawden@inria.fr)

### Dataset Summary

The DiscEvalMT dataset contains English-to-French translations used for resolving ambiguity in pronoun anaphora resolution and lexical choice (disambiguation and cohesion) context-aware translation. This version of the DiscEvalMT dataset contains further annotations of ambiguous spans and supporting context in the dataset examples to align it with the highlighting scheme of [SCAT](https://huggingface.co/inseq), enabling granular evaluations of context usage in context-aware NMT models.

**Disclaimer**: *The DiscEvalMT corpus was released in the NAACL 2018 paper ["Evaluating Discourse Phenomena in Neural Machine Translation"](https://www.aclweb.org/anthology/N18-1118) by Bawden et al. (2018), and an original version of the corpus is hosted on [Github](https://github.com/rbawden/discourse-mt-test-sets) with CC-BY-SA 4.0 license.*

### Supported Tasks and Leaderboards
#### Machine Translation
Refer to the [original paper](ttps://www.aclweb.org/anthology/N18-1118) for additional details on the evaluation of discourse-level phenomena using DiscEvalMT.
### Languages
The dataset contains handcrafted English-to-French translation examples containing either anaphoric pronouns or lexical choice items. Examples were created using existing [OpenSubtitles 2016](https://aclanthology.org/L16-1147/) sentences as reference for lexicon and syntactic structure.
## Dataset Structure
### Data Instances

The dataset contains two configurations (`anaphora` and `lexical-choice`), each containing only a test set of 200 examples each. Dataset examples have the following format:

```json
{
  "id": 0,
  "context_en": "The buildings will be finished next week.",
  "en": "Soon they will be full of new residents.",
  "context_fr": "Les bâtiments seront terminés la semaine prochaine.",
  "fr": "Ils seront bientôt pleins de nouveaux résidents.",
  "contrast_fr": "Elles seront bientôt pleines de nouveaux résidents.",
  "context_en_with_tags": "The <hon>buildings<hoff> will be finished next week.",
  "en_with_tags": "Soon <p>they</p> will be full of new residents.",
  "context_fr_with_tags": "Les <hon>bâtiments<hoff> seront terminés la semaine prochaine.",
  "fr_with_tags": "<p>Ils</p> seront bientôt pleins de nouveaux résidents.",
  "contrast_fr_with_tags": "<p>Elles</p> seront bientôt pleines de nouveaux résidents.",
  "type": "m.pl"
}
```

In every example, the context-dependent word of interest and its translation are surrounded by `<p>...</p>` tags. These are guaranteed to be found in the `en_with_tags`, `fr_with_tags` and `contrast_fr_with_tags` fields.

Any span surrounded by `<hon>...<hoff>` tags was identified by human annotators as supporting context necessary to correctly translate the pronoun of interest. These spans are found only in the `context_en_with_tags` and `context_fr_with_tags` fields.

In the example above, the translation of the pronoun `they` (field `en`) is ambiguous, and the correct translation to the feminine French pronoun `Ils` (in field `fr`) is only possible thanks to the supporting masculine noun `bâtiments` in the field `context_fr`.

Fields with the `_with_tags` suffix contain tags around pronouns of interest and supporting context, while their counterparts without the suffix contain the same text without tags, to facilitate direct usage with machine translation models.

### Dataset Creation

The dataset was created manually by the original authors, with context usage annotations added by the authors of [Quantifying the Plausibility of Context Reliance in Neural Machine Translation](tbd) for plausibility analysis purposes.

Please refer to the original article [Evaluating Discourse Phenomena in Neural Machine Translation](https://www.aclweb.org/anthology/N18-1118) for additional information on dataset creation.

### Additional Preprocessing

The dataset presents minor adjustments compared to the original DiscEvalMT corpus.

## Additional Information
### Dataset Curators

The original authors of DiscEvalMT are the curators of the original released dataset. For problems or updates on this 🤗 Datasets version, please contact [gabriele.sarti996@gmail.com](mailto:gabriele.sarti996@gmail.com).

### Licensing Information

The dataset is released under the original CC-BY-SA 4.0 license.

### Citation Information
Please cite the authors if you use these corpus in your work.

```bibtex
@inproceedings{bawden-etal-2018-evaluating,
    title = "Evaluating Discourse Phenomena in Neural Machine Translation",
    author = "Bawden, Rachel and Sennrich, Rico and Birch, Alexandra and Haddow, Barry",
    booktitle = {{Proceedings of the 2018 Conference of the North {A}merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)}},
    month = jun,
    year = "2018",
    address = "New Orleans, Louisiana",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/N18-1118",
    doi = "10.18653/v1/N18-1118",
    pages = "1304--1313"
}
```
