task_categories:translation annotations_creators:expert-generated annotations_creators:machine-generated language_creators:found multilinguality:translation size_categories:1K<n<10K source_datasets:extended|reddit source_datasets:extended|wikipedia language:de language:en language:et language:ne language:ro language:ru language:si language:zh license:unknown region:us
This shared task (part of WMT20) will build on its previous editions
to further examine automatic methods for estimating the quality
of neural machine translation output at run-time, without relying
on reference translations. As in previous years, we cover estimation
at various levels. Important elements introduced this year include: a new
task where sentences are annotated with Direct Assessment (DA)
scores instead of labels based on post-editing; a new multilingual
sentence-level dataset mainly from Wikipedia articles, where the
source articles can be retrieved for document-wide context; the
availability of NMT models to explore system-internal information for the task.

Task 1 uses Wikipedia data for 6 language pairs that includes high-resource
English--German (En-De) and English--Chinese (En-Zh), medium-resource
Romanian--English (Ro-En) and Estonian--English (Et-En), and low-resource
Sinhalese--English (Si-En) and Nepalese--English (Ne-En), as well as a
dataset with a combination of Wikipedia articles and Reddit articles
for Russian-English (En-Ru). The datasets were collected by translating
sentences sampled from source language articles using state-of-the-art NMT
models built using the fairseq toolkit and annotated with Direct Assessment (DA)
scores by professional translators. Each sentence was annotated following the
FLORES setup, which presents a form of DA, where at least three professional
translators rate each sentence from 0-100 according to the perceived translation
quality. DA scores are standardised using the z-score by rater. Participating systems
are required to score sentences according to z-standardised DA scores.
Not available.
---
pretty_name: WMT20 - MultiLingual Quality Estimation (MLQE) Task1
annotations_creators:
- expert-generated
- machine-generated
language_creators:
- found
language:
- de
- en
- et
- ne
- ro
- ru
- si
- zh
license:
- unknown
multilinguality:
- translation
size_categories:
- 1K<n<10K
source_datasets:
- extended|reddit
- extended|wikipedia
task_categories:
- translation
task_ids: []
paperswithcode_id: null
dataset_info:
- config_name: en-de
  features:
  - name: segid
    dtype: int32
  - name: translation
    dtype:
      translation:
        languages:
        - en
        - de
  - name: scores
    sequence: float32
  - name: mean
    dtype: float32
  - name: z_scores
    sequence: float32
  - name: z_mean
    dtype: float32
  - name: model_score
    dtype: float32
  - name: doc_id
    dtype: string
  - name: nmt_output
    dtype: string
  - name: word_probas
    sequence: float32
  splits:
  - name: train
    num_bytes: 3492361
    num_examples: 7000
  - name: test
    num_bytes: 524159
    num_examples: 1000
  - name: validation
    num_bytes: 522576
    num_examples: 1000
  download_size: 2343124
  dataset_size: 4539096
- config_name: en-zh
  features:
  - name: segid
    dtype: int32
  - name: translation
    dtype:
      translation:
        languages:
        - en
        - zh
  - name: scores
    sequence: float32
  - name: mean
    dtype: float32
  - name: z_scores
    sequence: float32
  - name: z_mean
    dtype: float32
  - name: model_score
    dtype: float32
  - name: doc_id
    dtype: string
  - name: nmt_output
    dtype: string
  - name: word_probas
    sequence: float32
  splits:
  - name: train
    num_bytes: 3286433
    num_examples: 7000
  - name: test
    num_bytes: 494994
    num_examples: 1000
  - name: validation
    num_bytes: 488477
    num_examples: 1000
  download_size: 2557445
  dataset_size: 4269904
- config_name: et-en
  features:
  - name: segid
    dtype: int32
  - name: translation
    dtype:
      translation:
        languages:
        - et
        - en
  - name: scores
    sequence: float32
  - name: mean
    dtype: float32
  - name: z_scores
    sequence: float32
  - name: z_mean
    dtype: float32
  - name: model_score
    dtype: float32
  - name: doc_id
    dtype: string
  - name: nmt_output
    dtype: string
  - name: word_probas
    sequence: float32
  splits:
  - name: train
    num_bytes: 3483669
    num_examples: 7000
  - name: test
    num_bytes: 527372
    num_examples: 1000
  - name: validation
    num_bytes: 531499
    num_examples: 1000
  download_size: 2320067
  dataset_size: 4542540
- config_name: ne-en
  features:
  - name: segid
    dtype: int32
  - name: translation
    dtype:
      translation:
        languages:
        - ne
        - en
  - name: scores
    sequence: float32
  - name: mean
    dtype: float32
  - name: z_scores
    sequence: float32
  - name: z_mean
    dtype: float32
  - name: model_score
    dtype: float32
  - name: doc_id
    dtype: string
  - name: nmt_output
    dtype: string
  - name: word_probas
    sequence: float32
  splits:
  - name: train
    num_bytes: 5313197
    num_examples: 7000
  - name: test
    num_bytes: 767637
    num_examples: 1000
  - name: validation
    num_bytes: 784784
    num_examples: 1000
  download_size: 2913901
  dataset_size: 6865618
- config_name: ro-en
  features:
  - name: segid
    dtype: int32
  - name: translation
    dtype:
      translation:
        languages:
        - ro
        - en
  - name: scores
    sequence: float32
  - name: mean
    dtype: float32
  - name: z_scores
    sequence: float32
  - name: z_mean
    dtype: float32
  - name: model_score
    dtype: float32
  - name: doc_id
    dtype: string
  - name: nmt_output
    dtype: string
  - name: word_probas
    sequence: float32
  splits:
  - name: train
    num_bytes: 3360498
    num_examples: 7000
  - name: test
    num_bytes: 500700
    num_examples: 1000
  - name: validation
    num_bytes: 507646
    num_examples: 1000
  download_size: 2222103
  dataset_size: 4368844
- config_name: si-en
  features:
  - name: segid
    dtype: int32
  - name: translation
    dtype:
      translation:
        languages:
        - si
        - en
  - name: scores
    sequence: float32
  - name: mean
    dtype: float32
  - name: z_scores
    sequence: float32
  - name: z_mean
    dtype: float32
  - name: model_score
    dtype: float32
  - name: doc_id
    dtype: string
  - name: nmt_output
    dtype: string
  - name: word_probas
    sequence: float32
  splits:
  - name: train
    num_bytes: 5135077
    num_examples: 7000
  - name: test
    num_bytes: 763730
    num_examples: 1000
  - name: validation
    num_bytes: 758271
    num_examples: 1000
  download_size: 2841894
  dataset_size: 6657078
- config_name: ru-en
  features:
  - name: segid
    dtype: int32
  - name: translation
    dtype:
      translation:
        languages:
        - ru
        - en
  - name: scores
    sequence: float32
  - name: mean
    dtype: float32
  - name: z_scores
    sequence: float32
  - name: z_mean
    dtype: float32
  - name: model_score
    dtype: float32
  - name: doc_id
    dtype: string
  - name: nmt_output
    dtype: string
  - name: word_probas
    sequence: float32
  splits:
  - name: train
    num_bytes: 3520776
    num_examples: 7000
  - name: test
    num_bytes: 478531
    num_examples: 1000
  - name: validation
    num_bytes: 499601
    num_examples: 1000
  download_size: 2123684
  dataset_size: 4498908
config_names:
- en-de
- en-zh
- et-en
- ne-en
- ro-en
- ru-en
- si-en
---

# Dataset Card for WMT20 - MultiLingual Quality Estimation (MLQE) Task1

## Table of Contents
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
  - [Curation Rationale](#curation-rationale)
  - [Source Data](#source-data)
  - [Annotations](#annotations)
  - [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
  - [Social Impact of Dataset](#social-impact-of-dataset)
  - [Discussion of Biases](#discussion-of-biases)
  - [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
  - [Dataset Curators](#dataset-curators)
  - [Licensing Information](#licensing-information)
  - [Citation Information](#citation-information)
  - [Contributions](#contributions)

## Dataset Description

- **Homepage:** [WMT20 Quality Estimation Shared Task](http://www.statmt.org/wmt20/quality-estimation-task.html)
- **Repository:** [Github repository](https://github.com/facebookresearch/mlqe/)
- **Paper:** *Not available*

### Dataset Summary

From the homepage:
*This shared task (part of WMT20) will build on its previous editions to further examine automatic methods for estimating the quality of neural machine translation output at run-time, without relying on reference translations. As in previous years, we cover estimation at various levels. Important elements introduced this year include: a new task where sentences are annotated with Direct Assessment (DA) scores instead of labels based on post-editing; a new multilingual sentence-level dataset mainly from Wikipedia articles, where the source articles can be retrieved for document-wide context; the availability of NMT models to explore system-internal information for the task.*

*Task 1 uses Wikipedia data for 6 language pairs that includes high-resource English--German (En-De) and English--Chinese (En-Zh), medium-resource Romanian--English (Ro-En) and Estonian--English (Et-En), and low-resource Sinhalese--English (Si-En) and Nepalese--English (Ne-En), as well as a dataset with a combination of Wikipedia articles and Reddit articles for Russian-English (En-Ru). The datasets were collected by translating sentences sampled from source language articles using state-of-the-art NMT models built using the fairseq toolkit and annotated with Direct Assessment (DA) scores by professional translators. Each sentence was annotated following the FLORES setup, which presents a form of DA, where at least three professional translators rate each sentence from 0-100 according to the perceived translation quality. DA scores are standardised using the z-score by rater. Participating systems are required to score sentences according to z-standardised DA scores.*

### Supported Tasks and Leaderboards

From the homepage:

*Sentence-level submissions will be evaluated in terms of the Pearson's correlation metric for the DA prediction agains human DA (z-standardised mean DA score, i.e. z_mean). These are the [official evaluation scripts](https://github.com/sheffieldnlp/qe-eval-scripts). The evaluation will focus on multilingual systems, i.e. systems that are able to provide predictions for all languages in the Wikipedia domain. Therefore, average Pearson correlation across all these languages will be used to rank QE systems. We will also evaluate QE systems on a per-language basis for those interested in particular languages.*

### Languages

Eight languages are represented in this dataset:
- English (`en`)
- German (`de`)
- Romanian (`ro`)
- Estonian (`et`)
- Nepalese (`ne`)
- Sinhala (`si`)
- Russian (`ru`)

## Dataset Structure

### Data Instances

An example looks like this:
```
{
  'segid': 123,
  'translation': {
    'en': 'José Ortega y Gasset visited Husserl at Freiburg in 1934.',
    'de': '1934 besuchte José Ortega y Gasset Husserl in Freiburg.',
  },
  'scores': [100.0, 100.0, 100.0],
  'mean': 100.0,
  'z_scores': [0.9553316831588745, 1.552362322807312, 0.850531816482544],
  'z_mean': 1.1194086074829102,
  'model_score': -0.10244649648666382,
  'doc_id': 'Edmund Husserl',
  'nmt_output': '1934 besuchte José Ort@@ ega y G@@ asset Hus@@ ser@@ l in Freiburg .',
  'word_probas': [-0.4458000063896179, -0.2745000123977661, -0.07199999690055847, -0.002300000051036477, -0.005900000222027302, -0.14579999446868896, -0.07500000298023224, -0.012400000356137753, -0.026900000870227814, -0.036400001496076584, -0.05299999937415123, -0.14990000426769257, -0.012400000356137753, -0.1145000010728836, -0.10999999940395355],
}
```

### Data Fields

- `segid`: segment id.
- `original`: original sentence.
- `translation`: Dictionary with pairs (source,target).
  - src_lg: sequence of text in source language.
  - tgt_lg: sequence of text in target language.
- `scores`: list of DA scores by all annotators - the number of annotators may vary. [] if N/A (only for `ru-en/test`).
- `mean`: average of DA scores. -10_000 if N/A (only for `ru-en/test`).
- `z_scores`: list of z-standardized DA scores. [] if N/A (only for `ru-en/test`).
- `z_mean`: average of z-standardized DA scores. -10_000 if N/A (only for `ru-en/test`).
- `model_score`: NMT model score for sentence. -10_000 if N/A (only for `ru-en/test`).
- `doc_id`: the name of the article where each original segment came from.
- `nmt_output`: the actual output of the NMT model before any post-processing, corresponding to the log-probas in `word_probas` (the token is not printed, so the number of log-probabilities equals the number of tokens plus 1).
- `word_probas`: log-probabilities from the NMT model for each decoded token including the token.

### Data Splits

There are 7 configurations in this dataset (one for each available language pair). Each configuration is composed of 7K examples for training, 1K for validation and 1K for test.

## Dataset Creation

### Curation Rationale

The original text is extracted from Wikipedia, Russian Reddit and Russian WikiQuotes. Translations are obtained using state-of-the-art NMT models built using the [fairseq toolkit](https://github.com/pytorch/fairseq) and annotated with Direct Assesment scores by professional translators.

### Source Data

[More Information Needed]

#### Initial Data Collection and Normalization

[More Information Needed]

#### Who are the source language producers?

[More Information Needed]

### Annotations

[More Information Needed]

#### Annotation process

[More Information Needed]

#### Who are the annotators?

[More Information Needed]

### Personal and Sensitive Information

[More Information Needed]

## Considerations for Using the Data

### Social Impact of Dataset

[More Information Needed]

### Discussion of Biases

[More Information Needed]

### Other Known Limitations

[More Information Needed]

## Additional Information

### Dataset Curators

[More Information Needed]

### Licensing Information

Unknown

### Citation Information

```
Not available.
```

### Contributions

Thanks to [@VictorSanh](https://github.com/VictorSanh) for adding this dataset.