task_categories:text-classification size_categories:10K<n<100K language:el language:fr language:it language:es language:pt language:ko language:fi language:lt language:bg license:bsd-2-clause region:us
---
license: bsd-2-clause
language:
- el
- fr
- it
- es
- pt
- ko
- fi
- lt
- bg
task_categories:
- text-classification
size_categories:
- 10K<n<100K
---
# Dataset Card for Dataset Name

## Dataset Description

- **Homepage:** 
- **Repository:** 
- **Paper:** 
- **Leaderboard:** 
- **Point of Contact:** 

### Dataset Summary

This repository contains a collection of machine translations of [LingNLI](https://github.com/Alicia-Parrish/ling_in_loop) dataset 
into 9 different languages (Bulgarian, Finnish, French, Greek, Italian, Korean, Lithuanian, Portuguese, Spanish). The goal is to predict textual entailment (does sentence A 
imply/contradict/neither sentence B), which is a classification task (given two sentences, 
predict one of three labels). It is here formatted in the same manner as the widely used [XNLI](https://huggingface.co/datasets/xnli) dataset for convenience.

### Supported Tasks and Leaderboards

This dataset can be used for the task of Natural Language Inference (NLI), also known as Recognizing Textual Entailment (RTE), which is a sentence-pair classification task.

### Languages

[More Information Needed]

## Dataset Structure

### Data Instances

[More Information Needed]

### Data Fields

- `language`: The language in which the pair of sentences is given.
- `premise`: The machine translated premise in the target language.
- `hypothesis`: The machine translated premise in the target language.
- `label`: The classification label, with possible values 0 (`entailment`), 1 (`neutral`), 2 (`contradiction`).
- `label_text`: The classification label, with possible values `entailment` (0), `neutral` (1), `contradiction` (2).
- `premise_original`: The original premise from the English source dataset.
- `hypothesis_original`: The original hypothesis from the English source dataset.

### Data Splits

For LitL subset:
|  language   |train|validation|
|-------------|----:|---------:|
|all_languages|134955|      21825|
|el-gr        |14995|      2425|
|fr           |14995|      2425|
|it           |14995|      2425|
|es           |14995|      2425|
|pt           |14995|      2425|
|ko           |14995|      2425|
|fi           |14995|      2425|
|lt           |14995|      2425|
|bg           |14995|      2425|

For LotS subset:
|  language   |train|validation|
|-------------|----:|---------:|
|all_languages|134910|      22212|
|el-gr        |14990|      2468|
|fr           |14990|      2468|
|it           |14990|      2468|
|es           |14990|      2468|
|pt           |14990|      2468|
|ko           |14990|      2468|
|fi           |14990|      2468|
|lt           |14990|      2468|
|bg           |14990|      2468|

## Dataset Creation

The two subsets of the original dataset were machine translated using the latest neural machine translation [opus-mt-tc-big](https://huggingface.co/models?sort=downloads&search=opus-mt-tc-big) models available for the respective languages. 
Running the translations lasted from March 25, 2023 until April 8, 2023.

### Curation Rationale

[More Information Needed]

### Source Data

#### Initial Data Collection and Normalization

[More Information Needed]

#### Who are the source language producers?

[More Information Needed]

### Annotations

#### Annotation process

[More Information Needed]

#### Who are the annotators?

[More Information Needed]

### Personal and Sensitive Information

[More Information Needed]

## Considerations for Using the Data

### Social Impact of Dataset

[More Information Needed]

### Discussion of Biases

[More Information Needed]

### Other Known Limitations

[More Information Needed]

## Additional Information

### Dataset Curators

[More Information Needed]

### Licensing Information

[More Information Needed]

### Citation Information

````BibTeX
@inproceedings{parrish-etal-2021-putting-linguist,
    title = "Does Putting a Linguist in the Loop Improve {NLU} Data Collection?",
    author = "Parrish, Alicia  and
      Huang, William  and
      Agha, Omar  and
      Lee, Soo-Hwan  and
      Nangia, Nikita  and
      Warstadt, Alexia  and
      Aggarwal, Karmanya  and
      Allaway, Emily  and
      Linzen, Tal  and
      Bowman, Samuel R.",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2021",
    month = nov,
    year = "2021",
    address = "Punta Cana, Dominican Republic",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.findings-emnlp.421",
    doi = "10.18653/v1/2021.findings-emnlp.421",
    pages = "4886--4901",
    abstract = "Many crowdsourced NLP datasets contain systematic artifacts that are identified only after data collection is complete. Earlier identification of these issues should make it easier to create high-quality training and evaluation data. We attempt this by evaluating protocols in which expert linguists work {`}in the loop{'} during data collection to identify and address these issues by adjusting task instructions and incentives. Using natural language inference as a test case, we compare three data collection protocols: (i) a baseline protocol with no linguist involvement, (ii) a linguist-in-the-loop intervention with iteratively-updated constraints on the writing task, and (iii) an extension that adds direct interaction between linguists and crowdworkers via a chatroom. We find that linguist involvement does not lead to increased accuracy on out-of-domain test sets compared to baseline, and adding a chatroom has no effect on the data. Linguist involvement does, however, lead to more challenging evaluation data and higher accuracy on some challenge sets, demonstrating the benefits of integrating expert analysis during data collection.",
}

@inproceedings{tiedemann-thottingal-2020-opus,
    title = "{OPUS}-{MT} {--} Building open translation services for the World",
    author = {Tiedemann, J{\"o}rg  and
      Thottingal, Santhosh},
    booktitle = "Proceedings of the 22nd Annual Conference of the European Association for Machine Translation",
    month = nov,
    year = "2020",
    address = "Lisboa, Portugal",
    publisher = "European Association for Machine Translation",
    url = "https://aclanthology.org/2020.eamt-1.61",
    pages = "479--480",
    abstract = "This paper presents OPUS-MT a project that focuses on the development of free resources and tools for machine translation. The current status is a repository of over 1,000 pre-trained neural machine translation models that are ready to be launched in on-line translation services. For this we also provide open source implementations of web applications that can run efficiently on average desktop hardware with a straightforward setup and installation.",
}
````

### Acknowledgements

These translations of the original dataset were done as part of a research project supported by the Defence Innovation Agency (AID) of the Directorate General of Armament (DGA) of the French Ministry of Armed Forces, and by the ICO, _Institut Cybersécurité Occitanie_, funded by Région Occitanie, France.

### Contributions

[More Information Needed]