task_categories:translation annotations_creators:expert-generated language_creators:found language_creators:machine-generated language_creators:expert-generated multilinguality:translation size_categories:100K<n<1M language:bn-hi language:cs-en language:de-en language:de-fr language:en-cs language:en-de language:en-ha language:en-is language:en-ja language:en-ru language:en-zh language:fr-de language:ha-en language:hi-bn language:is-en language:ja-en language:ru-en language:xh-zh language:zh-en language:zu-xh license:unknown region:us
This shared task will examine automatic evaluation metrics for machine translation. We will 
provide you with MT system outputs along with source text and the human reference translations. 
We are looking for automatic metric scores for translations at the system-level, and segment-level. 
We will calculate the system-level, and segment-level correlations of your scores with human judgements.

We invite submissions of reference-free metrics in addition to reference-based metrics.
@inproceedings{freitag-etal-2021-results,
    title = {Results of the WMT21 Metrics Shared Task: Evaluating Metrics with Expert-based Human Evaluations on TED and News Domain},
    author = {Freitag, Markus and Rei, Ricardo  and Mathur, Nitika  and Lo, Chi-kiu  and Stewart, Craig  and Foster, George  and Lavie, Alon and Bojar, Ondrej}",
    booktitle = {Proceedings of the Sixth Conference on Machine Translation},
    month = {nov},
    year = {2021},
    address = {Online},
    publisher = {Association for Computational Linguistics},
    url = {https://aclanthology.org/2021.wmt-1.73},
    pages = {733--774}
}
---
annotations_creators:
- expert-generated
language_creators:
- found
- machine-generated
- expert-generated
language:
- bn-hi
- cs-en
- de-en
- de-fr
- en-cs
- en-de
- en-ha
- en-is
- en-ja
- en-ru
- en-zh
- fr-de
- ha-en
- hi-bn
- is-en
- ja-en
- ru-en
- xh-zh
- zh-en
- zu-xh
license:
- unknown
multilinguality:
- translation
paperswithcode_id: null
pretty_name: WMT21 Metrics Shared Task
size_categories:
- 100K<n<1M
source_datasets: []
task_categories:
- translation
task_ids: []
---

# Dataset Card for WMT21 Metrics Task

## Table of Contents
- [Table of Contents](#table-of-contents)
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
  - [Curation Rationale](#curation-rationale)
  - [Source Data](#source-data)
  - [Annotations](#annotations)
  - [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
  - [Social Impact of Dataset](#social-impact-of-dataset)
  - [Discussion of Biases](#discussion-of-biases)
  - [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
  - [Dataset Curators](#dataset-curators)
  - [Licensing Information](#licensing-information)
  - [Citation Information](#citation-information)
  - [Contributions](#contributions)

## Dataset Description

- **Homepage:** [WMT21 Metrics Shared Task](https://www.statmt.org/wmt21/metrics-task.html)
- **Repository:** [MT Metrics Eval Github Repository](https://github.com/google-research/mt-metrics-eval)
- **Paper:** [Paper](https://aclanthology.org/2021.wmt-1.73/)

### Dataset Summary

[More Information Needed]

### Supported Tasks and Leaderboards

[More Information Needed]

### Languages

The dataset comprises twenty language pairs:
- Bengali-Hindi (`bn-hi`)
- Czech-English (`cs-en`)
- German-English (`de-en`)
- German-French (`de-fr`)
- English-Czech (`en-cs`)
- English-German (`en-de`)
- English-Hausa (`en-ha`)
- English-Icelandic (`en-is`)
- English-Japanese (`en-ja`)
- English-Russian (`en-ru`)
- English-Chinese (`en-zh`)
- French-German (`fr-de`)
- Hausa-English (`ha-en`)
- Hindi-Bengali (`hi-bn`)
- Icelandic-English (`is-en`)
- Japenese-English (`ja-en`)
- Russian-English (`ru-en`)
- Xhosa-Zulu (`xh-zu`)
- Chinese-English (`zh-en`)
- Zulu-Xhosa (`zu-xh`)

## Dataset Structure

### Data Instances

[More Information Needed]

### Data Fields

[More Information Needed]

### Data Splits

[More Information Needed]

## Dataset Creation

### Curation Rationale

[More Information Needed]

### Source Data

#### Initial Data Collection and Normalization

[More Information Needed]

#### Who are the source language producers?

[More Information Needed]

### Annotations

#### Annotation process

[More Information Needed]

#### Who are the annotators?

[More Information Needed]

### Personal and Sensitive Information

[More Information Needed]

## Considerations for Using the Data

### Social Impact of Dataset

[More Information Needed]

### Discussion of Biases

[More Information Needed]

### Other Known Limitations

[More Information Needed]

## Additional Information

### Dataset Curators

[More Information Needed]

### Licensing Information

[More Information Needed]

### Citation Information

[More Information Needed]

### Contributions

Thanks to [@github-username](https://github.com/<github-username>) for adding this dataset.
