task_categories:translation task_ids:multiple-choice-qa task_ids:language-modeling annotations_creators:no-annotation language_creators:machine-generated language_creators:expert-generated multilinguality:multilingual multilinguality:translation size_categories:1K<n<10K source_datasets:original language:en language:de language:fr language:ru license:mit region:us
Wino-X is a parallel dataset of German, French, and Russian Winograd schemas, aligned with their English 
counterparts, used to examine whether neural machine translation models can perform coreference resolution that 
requires commonsense knowledge and whether multilingual language models are capable of commonsense reasoning across 
multiple languages.
@inproceedings{Emelin2021WinoXMW,
  title={Wino-X: Multilingual Winograd Schemas for Commonsense Reasoning and Coreference Resolution},
  author={Denis Emelin and Rico Sennrich},
  booktitle={EMNLP},
  year={2021}
}
---
annotations_creators:
- no-annotation
language:
- en
- de
- fr
- ru
language_creators:
- machine-generated
- expert-generated
license:
- mit
multilinguality:
- multilingual
- translation
pretty_name: Wino-X
size_categories:
- 1K<n<10K
source_datasets:
- original
task_categories:
- translation
- coreference resolution
- commonsense reasoning
task_ids:
- multiple-choice-qa
- language-modeling
---

# Dataset Card for Wino-X

## Table of Contents
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks](#supported-tasks-and-leaderboards)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-instances)
  - [Data Splits](#data-instances)
- [Dataset Creation](#dataset-creation)
  - [Curation Rationale](#curation-rationale)
  - [Source Data](#source-data)
  - [Annotations](#annotations)
  - [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
  - [Social Impact of Dataset](#social-impact-of-dataset)
  - [Discussion of Biases](#discussion-of-biases)
  - [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
  - [Dataset Curators](#dataset-curators)
  - [Licensing Information](#licensing-information)
  - [Citation Information](#citation-information)

## Dataset Description

- **Homepage:** [Wino-X repository](https://github.com/demelin/Wino-X)
- **Repository:** [Wino-X repository](https://github.com/demelin/Wino-X)
- **Paper:** [Wino-X: Multilingual Winograd Schemas for Commonsense Reasoning and Coreference Resolution](https://aclanthology.org/2021.emnlp-main.670/)
- **Leaderboard:** [N/A]
- **Point of Contact:** [Denis Emelin](demelin.github.io)

### Dataset Summary

Wino-X is a parallel dataset of German, French, and Russian Winograd schemas, aligned with their English
counterparts, used to examine whether neural machine translation models can perform coreference resolution that
requires commonsense knowledge, and whether multilingual language models are capable of commonsense reasoning across
multiple languages.

### Supported Tasks and Leaderboards

- translation: The dataset can be used to evaluate translations of ambiguous source sentences, as produced by translation models . A [pretrained transformer-based NMT model](https://huggingface.co/Helsinki-NLP/opus-mt-en-de) can be used for this purpose.
- coreference-resolution: The dataset can be used to rank alternative translations of an ambiguous source sentence that differ in the chosen referent of an ambiguous source pronoun. A [pretrained transformer-based NMT model](https://huggingface.co/Helsinki-NLP/opus-mt-en-de) can be used for this purpose.
- commonsense-reasoning: The dataset can also be used evaluate whether pretrained multilingual language models can perform commonsense reasoning in (or across) multiple languages by identifying the correct filler in a cloze completion task. An [XLM-based model](https://huggingface.co/xlm-roberta-base) can be used for this purpose.

### Languages

The dataset (both its MT and LM portions) is available in the following translation pairs: English-German, English-French, English-Russian. All English sentences included in *Wino-X* were extracted from publicly available parallel corpora, as detailed in the accompanying paper, and represent the dataset-specific language varieties. All non-English sentences were obtained through machine translation and may, as such, exhibit features of translationese.

## Dataset Structure

### Data Instances

The following represents a typical *MT-Wino-X* instance (for the English-German translation pair):  
  
{"qID": "3UDTAB6HH8D37OQL3O6F3GXEEOF09Z-1",  
"sentence": "The woman looked for a different vase for the bouquet because it was too small.",  
"translation1": "Die Frau suchte nach einer anderen Vase für den Blumenstrauß, weil sie zu klein war.",  
"translation2": "Die Frau suchte nach einer anderen Vase für den Blumenstrauß, weil er zu klein war.",  
"answer": 1,  
"pronoun1": "sie",  
"pronoun2": "er",  
"referent1_en": "vase",  
"referent2_en": "bouquet",  
"true_translation_referent_of_pronoun1_de": "Vase",  
"true_translation_referent_of_pronoun2_de": "Blumenstrauß",  
"false_translation_referent_of_pronoun1_de": "Vase",  
 "false_translation_referent_of_pronoun2_de": "Blumenstrauß"}  
  
  
The following represents a typical *LM-Wino-X* instance (for the English-French translation pair):  
  
{"qID": "3UDTAB6HH8D37OQL3O6F3GXEEOF09Z-1",   
"sentence": "The woman looked for a different vase for the bouquet because it was too small.",   
"context_en": "The woman looked for a different vase for the bouquet because _ was too small.",   
"context_fr": "La femme a cherché un vase différent pour le bouquet car _ était trop petit.",   
"option1_en": "the bouquet",   
"option2_en": "the vase",   
"option1_fr": "le bouquet",   
"option2_fr": "le vase",   
"answer": 2,   
"context_referent_of_option1_fr": "bouquet",  
"context_referent_of_option2_fr": "vase"}

### Data Fields

For *MT-Wino-X*:  
  
- "qID": Unique identifier ID for this dataset instance.
- "sentence":  English sentence containing the ambiguous pronoun 'it'.
- "translation1": First translation candidate.
- "translation2": Second translation candidate.
- "answer":  ID of the correct translation.
- "pronoun1": Translation of the ambiguous source pronoun in translation1.
- "pronoun2": Translation of the ambiguous source pronoun in translation2.
- "referent1_en": English referent of the translation of the ambiguous source pronoun in translation1.
- "referent2_en": English referent of the translation of the ambiguous source pronoun in translation2.
- "true_translation_referent_of_pronoun1_[TGT-LANG]": Target language referent of pronoun1 in the correct translation.
- "true_translation_referent_of_pronoun2_[TGT-LANG]":  Target language referent of pronoun2 in the correct translation.
- "false_translation_referent_of_pronoun1_[TGT-LANG]": Target language referent of pronoun1 in the incorrect translation.
- "false_translation_referent_of_pronoun2_[TGT-LANG]":  Target language referent of pronoun2 in the incorrect translation.
  
  
For *LM-Wino-X*:  
  
- "qID": Unique identifier ID for this dataset instance.
- "sentence": English sentence containing the ambiguous pronoun 'it'. 
- "context_en": Same English sentence, where 'it' is replaced by a gap. 
- "context_fr": Target language translation of the English sentence, where the translation of 'it' is replaced by a gap.
- "option1_en": First filler option for the English sentence.
- "option2_en":  Second filler option for the English sentence.
- "option1_[TGT-LANG]": First filler option for the target language sentence.
- "option2_[TGT-LANG]":  Second filler option for the target language sentence.
- "answer": ID of the correct gap filler.
- "context_referent_of_option1_[TGT-LANG]": English translation of option1_[TGT-LANG].
- "context_referent_of_option2_[TGT-LANG]": English translation of option2_[TGT-LANG]

### Data Splits

*Wno-X* was designed as an evaluation-only benchmark and therefore is intended to be used for zero-shot testing only. However, users are very welcome to split the data as they wish :) .


## Dataset Creation

### Curation Rationale

Please refer to [Section 2 in the dataset paper](https://aclanthology.org/2021.emnlp-main.670.pdf).

### Source Data

#### Initial Data Collection and Normalization

Please refer to [Section 2 in the dataset paper](https://aclanthology.org/2021.emnlp-main.670.pdf).

#### Who are the source language producers?

Please refer to [Section 2 in the dataset paper](https://aclanthology.org/2021.emnlp-main.670.pdf).

### Annotations

#### Annotation process

Please refer to [Section 2 in the dataset paper](https://aclanthology.org/2021.emnlp-main.670.pdf).

#### Who are the annotators?

Annotations were generated automatically and verified by the dataset author / curator for correctness.

### Personal and Sensitive Information

[N/A]

## Considerations for Using the Data

### Social Impact of Dataset

Please refer to ['Ethical Considerations' in the dataset paper](https://aclanthology.org/2021.emnlp-main.670.pdf).

### Discussion of Biases

Please refer to ['Ethical Considerations' in the dataset paper](https://aclanthology.org/2021.emnlp-main.670.pdf).

### Other Known Limitations

Please refer to ['Ethical Considerations' in the dataset paper](https://aclanthology.org/2021.emnlp-main.670.pdf).

## Additional Information

### Dataset Curators

[Denis Emelin](demelin.github.io)

### Licensing Information

MIT

### Citation Information

@inproceedings{Emelin2021WinoXMW,
  title={Wino-X: Multilingual Winograd Schemas for Commonsense Reasoning and Coreference Resolution},
  author={Denis Emelin and Rico Sennrich},
  booktitle={EMNLP},
  year={2021}
}