task_categories:translation annotations_creators:expert-generated language_creators:expert-generated multilinguality:translation size_categories:10K<n<100K source_datasets:original language:en language:fr license:unknown contextual-mt document-mt region:us
The Supporting Context for Ambiguous Translations corpus (SCAT) is a dataset 
of English-to-French translations annotated with human rationales used for resolving ambiguity 
in pronoun anaphora resolution for multi-sentence translation.
@inproceedings{yin-etal-2021-context,
    title = "Do Context-Aware Translation Models Pay the Right Attention?",
    author = "Yin, Kayo  and
      Fernandes, Patrick  and
      Pruthi, Danish  and
      Chaudhary, Aditi  and
      Martins, Andr{\'e} F. T.  and
      Neubig, Graham",
    booktitle = "Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)",
    month = aug,
    year = "2021",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.acl-long.65",
    doi = "10.18653/v1/2021.acl-long.65",
    pages = "788--801",
}
---
annotations_creators:
- expert-generated
language:
- en
- fr
language_creators:
- expert-generated
license:
- unknown
multilinguality:
- translation
pretty_name: scat
size_categories:
- 10K<n<100K
source_datasets:
- original
tags:
- contextual-mt
- document-mt
task_categories:
- translation
task_ids: []
---

# Dataset Card for SCAT

## Table of Contents

- [Dataset Card for SCAT](#dataset-card-for-scat)
  - [Table of Contents](#table-of-contents)
  - [Dataset Description](#dataset-description)
    - [Dataset Summary](#dataset-summary)
    - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
      - [Machine Translation](#machine-translation)
    - [Languages](#languages)
  - [Dataset Structure](#dataset-structure)
    - [Data Instances](#data-instances)
    - [Data Splits](#data-splits)
    - [Dataset Creation](#dataset-creation)
    - [Additional Preprocessing](#additional-preprocessing)
  - [Additional Information](#additional-information)
    - [Dataset Curators](#dataset-curators)
    - [Licensing Information](#licensing-information)
    - [Citation Information](#citation-information)

## Dataset Description

- **Repository:** [Github](https://github.com/neulab/contextual-mt/tree/master/data/scat)
- **Paper:** [ACL 2021](https://aclanthology.org/2021.acl-long.65/)
- **Point of Contact:** [Kayo Yin](mailto:kayoyin@berkeley.edu)

### Dataset Summary

The Supporting Context for Ambiguous Translations corpus (SCAT) is a dataset of English-to-French translations annotated with human rationales used for resolving ambiguity in pronoun anaphora resolution for multi-sentence translation.

**Disclaimer**: *The SCAT corpus was released in the ACL 2021 paper ["Do Context-Aware Translation Models Pay the Right Attention?"](https://aclanthology.org/2021.acl-long.65/) by Yin et al. (2021), and an original version of the corpus is hosted on [Github](https://github.com/neulab/contextual-mt/tree/master/data/scat) with no licensing information. This dataset contains a curated version of the original corpus where examples containing nested or malformed tags were removed (refer to the [filter_scat.py](filter_scat.py) script for more details).*

### Supported Tasks and Leaderboards
#### Machine Translation
Refer to the [original paper](https://aclanthology.org/2021.acl-long.65/) for additional details on plausibility evaluation for document-level MT systems.
### Languages
The dataset contains source English sentences containing anaphoric pronouns selected from the [OpenSubtitles 2018](https://www.aclweb.org/anthology/L18-1275/) corpus and translated into the French (`fr`)
## Dataset Structure
### Data Instances

The dataset contains a single default configuration. Dataset examples have the following format:

```json
{
  "id": 0,
  "context_en": "Air, water, the continents. So, what is your project about and what are its chances of winning? - Well, my project is awesome. - Oh, good. I took two plants, and I gave them sun and water",
  "en": "But I gave one special attention to see if it would grow more.",
  "context_fr": "L'air, l'eau, les continents. Donc, quel est le sujet de ton projet et quelles sont ses chances de gagner ? - Bien, mon projet est impressionnant. - Oh, bien. J'ai pris deux plantes , et je leur ai donné de l'eau et du soleil.",
  "fr": "Mais j'ai donné une attention particulière à une pour voir si elle grandit plus.",
  "contrast_fr": "Mais j'ai donné une attention particulière à une pour voir si il grandit plus.",
  "context_en_with_tags": "Air, water, the continents. So, what is your project about and what are its chances of winning? - Well, my project is awesome. - Oh, good. I took two plants, and I gave them sun and water",
  "en_with_tags": "But I gave one special attention to see if <p>it</p> would grow more.",
  "context_fr_with_tags": "L'air, l'eau, les continents. Donc, quel est le sujet de ton projet et quelles sont ses chances de gagner ? - Bien, mon projet est impressionnant. - Oh, bien. J'ai pris deux <hon>plantes<hoff> , et je leur ai donné de l'eau et du soleil.",
  "fr_with_tags": "Mais j'ai donné une attention particulière à une pour voir si <p>elle</p> grandit plus.",
  "contrast_fr_with_tags": "Mais j'ai donné une attention particulière à une pour voir si <p>il</p> grandit plus.",
  "has_supporting_context": true,
  "has_supporting_preceding_context": true,
}
```

In every example, the pronoun of interest and its translation are surrounded by `<p>...</p>` tags. These are guaranteed to be found in the `en_with_tags` and `fr_with_tags` field, respectively.

Any span surrounded by `<hon>...<hoff>` tags was identified by human annotators as supporting context to correctly translate the pronoun of interest. These spans can be missing altogether (i.e. no contextual information needed), or they can be found in any of the available fields. The `has_supporting_context` field indicates whether the example contains any supporting context.

In the example above, the translation of the pronoun `it` (field `en`) is ambiguous, and the correct translation to the feminine French pronoun `elle` (in field `fr`) is only possible thanks to the supporting feminine noun `plantes` in the field `context_fr`. Since the example contains supporting context, the `has_supporting_context` field is set to `true`. Since the context is not in the same sentence as the target pronoun but in a preceding one, `has_supporting_preceding_context` is also true.

Fields with the `_with_tags` suffix contain tags around pronouns of interest and supporting context, while their counterparts without the suffix contain the same text without tags, to facilitate direct usage with machine translation models.

### Data Splits

The dataset is split into `train`, `validation` and `test` sets. In the following table, we report the number of examples in the original dataset and in this filtered version in which examples containing malformed tags were removed.

| Split         | # Examples (original) | # Examples (**this**) |
| :-----------: | :-------------------: | :-------------------: |
| `train`       |    11471              |    6086               |
| `validation`  |    145                |    66                 |
| `test`        |    1000               |    500                |

### Dataset Creation

From the original paper:

>We recruited 20 freelance English-French translators on Upwork. We annotate examples from the contrastive test set by Lopes et al. (2020). This set includes 14K examples from the OpenSubtitles2018 dataset. Through our annotation effort, we obtain 14K examples of supporting context for pronoun anaphora resolution in ambiguous translations selected by professional human translators.

Please refer to the original article [Do Context-Aware Translation Models Pay the Right Attention?](https://aclanthology.org/2021.acl-long.65/) for additional information on dataset creation.

### Additional Preprocessing

Compared to the original SCAT corpus, the following differences are present in this version:

- Examples were filtered using the [filter_scat.py](filter_scat.py) script to retain only examples containing well-formed tags, and remove superfluous tags. Superfluous tags are defined as nested `<hon><p>...</p><hoff>` tags that represent lack of contextual information for disambiguating the correct pronoun. In this case, the outer `<hon>...<hoff>` tag was removed. We also filter examples that do not contain tags for the pronoun on interest in the target sentence for English and French (`en`, `fr`).

- Sentences stripped from tags are provided in fields without the `_with_tags` suffix.

- Tags erroneously embedded within words of target sentence (e.g. `just s<p>it</p>ting there, wa<p>it</p>ing for his face`) were removed.

- An extra contrastive sentence using the pronoun of interest belonging to the opposite gender is available in the `contrast_fr` field. The swap was performed using a simple lexical heuristic (refer to `swap_pronoun` in [`scat.py`](./scat.py)), and we do not guarantee grammatical correctness of the sentence.

The test set was manually curated to include 250 examples with supporting context in the previous sentences (`has_supporting_preceding_context` is `true`), and 250 examples with supporting context in the same sentence (`has_supporting_preceding_context` is `false`).

## Additional Information
### Dataset Curators

The original authors of SCAT are the curators of the original released dataset. For problems or updates on this 🤗 Datasets version, please contact [gabriele.sarti996@gmail.com](mailto:gabriele.sarti996@gmail.com).

### Licensing Information

The dataset license is unknown.

### Citation Information
Please cite the authors if you use these corpus in your work.

```bibtex
@inproceedings{yin-etal-2021-context,
    title = "Do Context-Aware Translation Models Pay the Right Attention?",
    author = "Yin, Kayo  and
      Fernandes, Patrick  and
      Pruthi, Danish  and
      Chaudhary, Aditi  and
      Martins, Andr{\'e} F. T.  and
      Neubig, Graham",
    booktitle = "Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)",
    month = aug,
    year = "2021",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.acl-long.65",
    doi = "10.18653/v1/2021.acl-long.65",
    pages = "788--801",
}
```