task_categories:translation annotations_creators:no-annotation language_creators:expert-generated language_creators:found multilinguality:translation size_categories:10K<n<100K source_datasets:original language:en language:pl language:ru license:unknown region:us
PolEval is a SemEval-inspired evaluation campaign for natural language processing tools for Polish.Submitted solutions compete against one another within certain tasks selected by organizers, using available data and are evaluated according topre-established procedures. One of the tasks in PolEval-2019 was Machine Translation (Task-4).
The task is to train as good as possible machine translation system, using any technology,with limited textual resources.The competition will be done for 2 language pairs, more popular English-Polish (into Polish direction) and pair that can be called low resourcedRussian-Polish (in both directions).

Here, Polish-English is also made available to allow for training in both directions. However, the test data is ONLY available for English-Polish.
---
annotations_creators:
- no-annotation
language_creators:
- expert-generated
- found
language:
- en
- pl
- ru
license:
- unknown
multilinguality:
- translation
size_categories:
- 10K<n<100K
source_datasets:
- original
task_categories:
- translation
task_ids: []
paperswithcode_id: null
pretty_name: Poleval2019Mt
dataset_info:
- config_name: ru-pl
  features:
  - name: translation
    dtype:
      translation:
        languages:
        - ru
        - pl
  splits:
  - name: train
    num_bytes: 2818015
    num_examples: 20001
  - name: validation
    num_bytes: 415735
    num_examples: 3001
  - name: test
    num_bytes: 266462
    num_examples: 2969
  download_size: 3355801
  dataset_size: 3500212
- config_name: en-pl
  features:
  - name: translation
    dtype:
      translation:
        languages:
        - en
        - pl
  splits:
  - name: train
    num_bytes: 13217798
    num_examples: 129255
  - name: validation
    num_bytes: 1209168
    num_examples: 10001
  - name: test
    num_bytes: 562482
    num_examples: 9845
  download_size: 13851405
  dataset_size: 14989448
- config_name: pl-ru
  features:
  - name: translation
    dtype:
      translation:
        languages:
        - pl
        - ru
  splits:
  - name: train
    num_bytes: 2818015
    num_examples: 20001
  - name: validation
    num_bytes: 415735
    num_examples: 3001
  - name: test
    num_bytes: 149423
    num_examples: 2967
  download_size: 3355801
  dataset_size: 3383173
- config_name: pl-en
  features:
  - name: translation
    dtype:
      translation:
        languages:
        - pl
        - en
  splits:
  - name: train
    num_bytes: 13217798
    num_examples: 129255
  - name: validation
    num_bytes: 1209168
    num_examples: 10001
  - name: test
    num_bytes: 16
    num_examples: 1
  download_size: 13591306
  dataset_size: 14426982
---

# Dataset Card for poleval2019_mt

## Table of Contents
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
  - [Curation Rationale](#curation-rationale)
  - [Source Data](#source-data)
  - [Annotations](#annotations)
  - [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
  - [Social Impact of Dataset](#social-impact-of-dataset)
  - [Discussion of Biases](#discussion-of-biases)
  - [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
  - [Dataset Curators](#dataset-curators)
  - [Licensing Information](#licensing-information)
  - [Citation Information](#citation-information)
  - [Contributions](#contributions)

## Dataset Description

- **Homepage:** PolEval-2019 competition. http://2019.poleval.pl/
- **Repository:** Links available [in this page](http://2019.poleval.pl/index.php/tasks/task4)
- **Paper:**
- **Leaderboard:**
- **Point of Contact:**

### Dataset Summary

PolEval is a SemEval-inspired evaluation campaign for natural language processing tools for Polish.
Submitted solutions compete against one another within certain tasks selected by organizers, using available data and are evaluated according to
pre-established procedures. One of the tasks in PolEval-2019 was Machine Translation (Task-4).

The task is to train as good as possible machine translation system, using any technology,with limited textual resources.
The competition will be done for 2 language pairs, more popular English-Polish (into Polish direction) and pair that can be called low resourced
Russian-Polish (in both directions).

Here, Polish-English is also made available to allow for training in both directions. However, the test data is ONLY available for English-Polish

### Supported Tasks and Leaderboards
Supports Machine Translation between Russian to Polish and English to Polish (and vice versa). 

### Languages
- Polish (pl)
- Russian (ru)
- English (en) 

## Dataset Structure

### Data Instances
As the training data set, a set of bi-lingual corpora aligned at the sentence level has been prepared. The corpora are saved in UTF-8 encoding as plain text, one language per file.

### Data Fields
One example of the translation is as below:
```
{
  'translation': {'ru': 'не содержала в себе моделей. Модели это сравнительно новое явление. ', 
                  'pl': 'nie miała w sobie modeli. Modele to względnie nowa dziedzina. Tak więc, jeśli '}
}
```

### Data Splits

The dataset is divided into two splits. All the headlines are scraped from news websites on the internet.

|       |  train | validation | test |
|-------|-------:|-----------:|-----:|
| ru-pl |  20001 |       3001 | 2969 |
| pl-ru |  20001 |       3001 | 2969 |
| en-pl | 129255 |       1000 | 9845 |

## Dataset Creation

### Curation Rationale

This data was curated as a task for the PolEval-2019. The task is to train as good as possible machine translation system, using any technology, with limited textual resources. The competition will be done for 2 language pairs, more popular English-Polish (into Polish direction) and pair that can be called low resourced Russian-Polish (in both directions).

PolEval is a SemEval-inspired evaluation campaign for natural language processing tools for Polish. Submitted tools compete against one another within certain tasks selected by organizers, using available data and are evaluated according to pre-established procedures.

PolEval 2019-related papers were presented at AI & NLP Workshop Day (Warsaw, May 31, 2019).
The links for the top performing models on various tasks (including the Task-4: Machine Translation) is present in [this](http://2019.poleval.pl/index.php/publication) link

### Source Data

#### Initial Data Collection and Normalization

[More Information Needed]

#### Who are the source language producers?

The organization details of PolEval is present in this [link](http://2019.poleval.pl/index.php/organizers)

### Annotations

#### Annotation process

[More Information Needed]

#### Who are the annotators?

[More Information Needed]

### Personal and Sensitive Information

[More Information Needed]

## Considerations for Using the Data

### Social Impact of Dataset

[More Information Needed]

### Discussion of Biases

[More Information Needed]

### Other Known Limitations

[More Information Needed]

## Additional Information

### Dataset Curators

[More Information Needed]

### Licensing Information

[More Information Needed]

### Citation Information

```
@proceedings{ogr:kob:19:poleval,
  editor    = {Maciej Ogrodniczuk and Łukasz Kobyliński},
  title     = {{Proceedings of the PolEval 2019 Workshop}},
  year      = {2019},
  address   = {Warsaw, Poland},
  publisher = {Institute of Computer Science, Polish Academy of Sciences},
  url       = {http://2019.poleval.pl/files/poleval2019.pdf},
  isbn      = "978-83-63159-28-3"}
}
```

### Contributions

Thanks to [@vrindaprabhu](https://github.com/vrindaprabhu) for adding this dataset.