task_categories:translation size_categories:10K<n<100K language:en language:pl language:de language:es language:sv license:cc-by-4.0 machine translation nlu natural-language-understanding virtual assistant region:us
\
---
dataset_info:
  features:
  - name: id
    dtype: string
  - name: locale
    dtype: string
  - name: origin
    dtype: string
  - name: partition
    dtype: string
  - name: translation_utt
    dtype:
      translation:
        languages:
        - en
        - pl
  - name: translation_xml
    dtype:
      translation:
        languages:
        - en
        - pl
  - name: src_bio
    dtype: string
  - name: tgt_bio
    dtype: string
task_categories:
- translation
language:
- en
- pl
- de
- es
- sv
tags:
- machine translation
- nlu
- natural-language-understanding
- virtual assistant
pretty_name: Machine translation for NLU with slot transfer
size_categories:
- 10K<n<100K
license: cc-by-4.0
---
# Machine translation dataset for NLU (Virual Assistant) with slot transfer between languages
## Dataset Summary

Disclaimer: This is for research purposes only. Please have a look at the license section below. Some of the datasets used to construct IVA_MT have an unknown license.

IVA_MT is a machine translation dataset that can be used to train, adapt and evaluate MT models used in Virtual Assistant NLU context (e.g. to translate trainig corpus of NLU).

## Dataset Composition

### en-pl

| Corpus                                                               | Train  | Dev   | Test  |
|----------------------------------------------------------------------|--------|-------|-------|
| [Massive 1.1](https://huggingface.co/datasets/AmazonScience/massive) | 11514  | 2033  | 2974  |
| [Leyzer 0.2.0](https://github.com/cartesinus/leyzer/tree/0.2.0)      | 3974   | 701   | 1380  |
| [OpenSubtitles from OPUS](https://opus.nlpl.eu/OpenSubtitles-v1.php) | 2329   | 411   | 500   |
| [KDE from OPUS](https://opus.nlpl.eu/KDE4.php)                       | 1154   | 241   | 241   |
| [CCMatrix from Opus](https://opus.nlpl.eu/CCMatrix.php)              | 1096   | 232   | 237   |
| [Ubuntu from OPUS](https://opus.nlpl.eu/Ubuntu.php)                  | 281    | 60    | 59    |
| [Gnome from OPUS](https://opus.nlpl.eu/GNOME.php)                    | 14     | 3     | 3     |
| *total*                                                              | 20362  | 3681  | 5394  |

### en-de

| Corpus                                                               | Train  | Dev   | Test  |
|----------------------------------------------------------------------|--------|-------|-------|
| [Massive 1.1](https://huggingface.co/datasets/AmazonScience/massive) | 7536   | 1346  | 1955  |

### en-es

| Corpus                                                               | Train  | Dev   | Test  |
|----------------------------------------------------------------------|--------|-------|-------|
| [Massive 1.1](https://huggingface.co/datasets/AmazonScience/massive) | 8415   | 1526  | 2202  |

### en-sv

| Corpus                                                               | Train  | Dev   | Test  |
|----------------------------------------------------------------------|--------|-------|-------|
| [Massive 1.1](https://huggingface.co/datasets/AmazonScience/massive) | 7540   | 1360  | 1921  |


## Tools
Scripts used to generate this dataset can be found on [github](https://github.com/cartesinus/iva_mt).

## License
This is a composition of 7 datasets, and the license is as defined in original release:
- MASSIVE: [CC-BY 4.0](https://huggingface.co/datasets/AmazonScience/massive/blob/main/LICENSE)
- Leyzer: [CC BY-NC 4.0](https://github.com/cartesinus/leyzer/blob/master/LICENSE)
- OpenSubtitles: unknown
- KDE: [GNU Public License](https://l10n.kde.org/about.php)
- CCMatrix: no license given, therefore assuming it is LASER project license [BSD](https://github.com/facebookresearch/LASER/blob/main/LICENSE)
- Ubuntu: [GNU Public License](https://help.launchpad.net/Legal)
- Gnome: unknown
