task_categories:translation task_categories:token-classification task_ids:parsing annotations_creators:expert-generated language_creators:crowdsourced multilinguality:multilingual multilinguality:translation size_categories:100K<n<1M size_categories:10K<n<100K source_datasets:original language:bn language:en language:fil language:hi language:id language:ja language:km language:lo language:ms language:my language:th language:vi language:zh license:cc-by-4.0 region:us
The ALT project aims to advance the state-of-the-art Asian natural language processing (NLP) techniques through the open collaboration for developing and using ALT. It was first conducted by NICT and UCSY as described in Ye Kyaw Thu, Win Pa Pa, Masao Utiyama, Andrew Finch and Eiichiro Sumita (2016). Then, it was developed under ASEAN IVO as described in this Web page. The process of building ALT began with sampling about 20,000 sentences from English Wikinews, and then these sentences were translated into the other languages. ALT now has 13 languages: Bengali, English, Filipino, Hindi, Bahasa Indonesia, Japanese, Khmer, Lao, Malay, Myanmar (Burmese), Thai, Vietnamese, Chinese (Simplified Chinese).
@inproceedings{riza2016introduction,
  title={Introduction of the asian language treebank},
  author={Riza, Hammam and Purwoadi, Michael and Uliniansyah, Teduh and Ti, Aw Ai and Aljunied, Sharifah Mahani and Mai, Luong Chi and Thang, Vu Tat and Thai, Nguyen Phuong and Chea, Vichet and Sam, Sethserey and others},
  booktitle={2016 Conference of The Oriental Chapter of International Committee for Coordination and Standardization of Speech Databases and Assessment Techniques (O-COCOSDA)},
  pages={1--6},
  year={2016},
  organization={IEEE}
}
---
annotations_creators:
- expert-generated
language_creators:
- crowdsourced
language:
- bn
- en
- fil
- hi
- id
- ja
- km
- lo
- ms
- my
- th
- vi
- zh
license:
- cc-by-4.0
multilinguality:
- multilingual
- translation
size_categories:
- 100K<n<1M
- 10K<n<100K
source_datasets:
- original
task_categories:
- translation
- token-classification
task_ids:
- parsing
paperswithcode_id: alt
pretty_name: Asian Language Treebank
dataset_info:
- config_name: alt-parallel
  features:
  - name: SNT.URLID
    dtype: string
  - name: SNT.URLID.SNTID
    dtype: string
  - name: url
    dtype: string
  - name: translation
    dtype:
      translation:
        languages:
        - bg
        - en
        - en_tok
        - fil
        - hi
        - id
        - ja
        - khm
        - lo
        - ms
        - my
        - th
        - vi
        - zh
  splits:
  - name: train
    num_bytes: 68445916
    num_examples: 18088
  - name: validation
    num_bytes: 3710979
    num_examples: 1000
  - name: test
    num_bytes: 3814431
    num_examples: 1019
  download_size: 21285784
  dataset_size: 75971326
- config_name: alt-en
  features:
  - name: SNT.URLID
    dtype: string
  - name: SNT.URLID.SNTID
    dtype: string
  - name: url
    dtype: string
  - name: status
    dtype: string
  - name: value
    dtype: string
  splits:
  - name: train
    num_bytes: 10075569
    num_examples: 17889
  - name: validation
    num_bytes: 544719
    num_examples: 988
  - name: test
    num_bytes: 567272
    num_examples: 1017
  download_size: 3871379
  dataset_size: 11187560
- config_name: alt-jp
  features:
  - name: SNT.URLID
    dtype: string
  - name: SNT.URLID.SNTID
    dtype: string
  - name: url
    dtype: string
  - name: status
    dtype: string
  - name: value
    dtype: string
  - name: word_alignment
    dtype: string
  - name: jp_tokenized
    dtype: string
  - name: en_tokenized
    dtype: string
  splits:
  - name: train
    num_bytes: 21888277
    num_examples: 17202
  - name: validation
    num_bytes: 1181555
    num_examples: 953
  - name: test
    num_bytes: 1175592
    num_examples: 931
  download_size: 13191239
  dataset_size: 24245424
- config_name: alt-my
  features:
  - name: SNT.URLID
    dtype: string
  - name: SNT.URLID.SNTID
    dtype: string
  - name: url
    dtype: string
  - name: value
    dtype: string
  splits:
  - name: train
    num_bytes: 20433275
    num_examples: 18088
  - name: validation
    num_bytes: 1111410
    num_examples: 1000
  - name: test
    num_bytes: 1135209
    num_examples: 1018
  download_size: 3028302
  dataset_size: 22679894
- config_name: alt-km
  features:
  - name: SNT.URLID
    dtype: string
  - name: SNT.URLID.SNTID
    dtype: string
  - name: url
    dtype: string
  - name: km_pos_tag
    dtype: string
  - name: km_tokenized
    dtype: string
  splits:
  - name: train
    num_bytes: 12015411
    num_examples: 18088
  - name: validation
    num_bytes: 655232
    num_examples: 1000
  - name: test
    num_bytes: 673753
    num_examples: 1018
  download_size: 2410832
  dataset_size: 13344396
- config_name: alt-my-transliteration
  features:
  - name: en
    dtype: string
  - name: my
    sequence: string
  splits:
  - name: train
    num_bytes: 4249424
    num_examples: 84022
  download_size: 1232127
  dataset_size: 4249424
- config_name: alt-my-west-transliteration
  features:
  - name: en
    dtype: string
  - name: my
    sequence: string
  splits:
  - name: train
    num_bytes: 7412043
    num_examples: 107121
  download_size: 2830071
  dataset_size: 7412043
config_names:
- alt-en
- alt-jp
- alt-km
- alt-my
- alt-my-transliteration
- alt-my-west-transliteration
- alt-parallel
---

# Dataset Card for Asian Language Treebank (ALT)

## Table of Contents
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
  - [Curation Rationale](#curation-rationale)
  - [Source Data](#source-data)
  - [Annotations](#annotations)
  - [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
  - [Social Impact of Dataset](#social-impact-of-dataset)
  - [Discussion of Biases](#discussion-of-biases)
  - [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
  - [Dataset Curators](#dataset-curators)
  - [Licensing Information](#licensing-information)
  - [Citation Information](#citation-information)
  - [Contributions](#contributions)

## Dataset Description

- **Homepage:** https://www2.nict.go.jp/astrec-att/member/mutiyama/ALT/
- **Leaderboard:** 
- **Paper:** [Introduction of the Asian Language Treebank](https://ieeexplore.ieee.org/abstract/document/7918974)
- **Point of Contact:** [ALT info](alt-info@khn.nict.go.jp)

### Dataset Summary
The ALT project aims to advance the state-of-the-art Asian natural language processing (NLP) techniques through the open collaboration for developing and using ALT. It was first conducted by NICT and UCSY as described in Ye Kyaw Thu, Win Pa Pa, Masao Utiyama, Andrew Finch and Eiichiro Sumita (2016). Then, it was developed under [ASEAN IVO](https://www.nict.go.jp/en/asean_ivo/index.html) as described in this Web page. 

The process of building ALT began with sampling about 20,000 sentences from English Wikinews, and then these sentences were translated into the other languages.

### Supported Tasks and Leaderboards

Machine Translation, Dependency Parsing


### Languages

It supports 13 language: 
  * Bengali
  * English
  * Filipino
  * Hindi
  * Bahasa Indonesia
  * Japanese
  * Khmer
  * Lao
  * Malay
  * Myanmar (Burmese)
  * Thai
  * Vietnamese
  * Chinese (Simplified Chinese).

## Dataset Structure

### Data Instances

#### ALT Parallel Corpus 
```
{
    "SNT.URLID": "80188",
    "SNT.URLID.SNTID": "1",
    "url": "http://en.wikinews.org/wiki/2007_Rugby_World_Cup:_Italy_31_-_5_Portugal",
    "bg": "[translated sentence]",
    "en": "[translated sentence]",
    "en_tok": "[translated sentence]",
    "fil": "[translated sentence]",
    "hi": "[translated sentence]",
    "id": "[translated sentence]",
    "ja": "[translated sentence]",
    "khm": "[translated sentence]",
    "lo": "[translated sentence]",
    "ms": "[translated sentence]",
    "my": "[translated sentence]",
    "th": "[translated sentence]",
    "vi": "[translated sentence]",
    "zh": "[translated sentence]"
}
```

#### ALT Treebank 
```
{
    "SNT.URLID": "80188",
    "SNT.URLID.SNTID": "1",
    "url": "http://en.wikinews.org/wiki/2007_Rugby_World_Cup:_Italy_31_-_5_Portugal",
    "status": "draft/reviewed",
    "value": "(S (S (BASENP (NNP Italy)) (VP (VBP have) (VP (VP (VP (VBN defeated) (BASENP (NNP Portugal))) (ADVP (RB 31-5))) (PP (IN in) (NP (BASENP (NNP Pool) (NNP C)) (PP (IN of) (NP (BASENP (DT the) (NN 2007) (NNP Rugby) (NNP World) (NNP Cup)) (PP (IN at) (NP (BASENP (NNP Parc) (FW des) (NNP Princes)) (COMMA ,) (BASENP (NNP Paris) (COMMA ,) (NNP France))))))))))) (PERIOD .))"
}
```

#### ALT Myanmar transliteration
```
{
    "en": "CASINO",
    "my": [
      "ကက်စီနို",
      "ကစီနို",
      "ကာစီနို",
      "ကာဆီနို"
    ]
}
```

### Data Fields


#### ALT Parallel Corpus 
- SNT.URLID: URL link to the source article listed in [URL.txt](https://www2.nict.go.jp/astrec-att/member/mutiyama/ALT/ALT-Parallel-Corpus-20191206/URL.txt)
- SNT.URLID.SNTID: index number from 1 to 20000. It is a seletected sentence from `SNT.URLID`

and bg, en, fil, hi, id, ja, khm, lo, ms, my, th, vi, zh correspond to the target language

#### ALT Treebank
- status: it indicates how a sentence is annotated; `draft` sentences are annotated by one annotater and `reviewed` sentences are annotated by two annotater 

The annotatation is different from language to language, please see [their guildlines](https://www2.nict.go.jp/astrec-att/member/mutiyama/ALT/) for more detail.

### Data Splits

|           | train | valid | test  |
|-----------|-------|-------|-------|
| # articles | 1698 |  98   |  97   |
| # sentences | 18088 | 1000  | 1018  |


## Dataset Creation

### Curation Rationale

The ALT project was initiated by the [National Institute of Information and Communications Technology, Japan](https://www.nict.go.jp/en/) (NICT) in 2014. NICT started to build Japanese and English ALT and worked with the University of Computer Studies, Yangon, Myanmar (UCSY) to build Myanmar ALT in 2014. Then, the Badan Pengkajian dan Penerapan Teknologi, Indonesia (BPPT), the Institute for Infocomm Research, Singapore (I2R), the Institute of Information Technology, Vietnam (IOIT), and the National Institute of Posts, Telecoms and ICT, Cambodia (NIPTICT) joined to make ALT for Indonesian, Malay, Vietnamese, and Khmer in 2015. 


### Source Data

#### Initial Data Collection and Normalization

[More Information Needed]

#### Who are the source language producers?

The dataset is sampled from the English Wikinews in 2014. These will be annotated with word segmentation, POS tags, and syntax information, in addition to the word alignment information by linguistic experts from
* National Institute of Information and Communications Technology, Japan (NICT) for Japanses and English
* University of Computer Studies, Yangon, Myanmar (UCSY) for Myanmar
* the Badan Pengkajian dan Penerapan Teknologi, Indonesia (BPPT) for Indonesian
* the Institute for Infocomm Research, Singapore (I2R) for Malay
* the Institute of Information Technology, Vietnam (IOIT) for Vietnamese
* the National Institute of Posts, Telecoms and ICT, Cambodia for Khmer

### Annotations

#### Annotation process

[More Information Needed]

#### Who are the annotators?

[More Information Needed]

### Personal and Sensitive Information

[More Information Needed]

## Considerations for Using the Data

### Social Impact of Dataset

[More Information Needed]

### Discussion of Biases

[More Information Needed]

### Other Known Limitations

[More Information Needed]


## Additional Information

### Dataset Curators

* National Institute of Information and Communications Technology, Japan (NICT) for Japanses and English
* University of Computer Studies, Yangon, Myanmar (UCSY) for Myanmar
* the Badan Pengkajian dan Penerapan Teknologi, Indonesia (BPPT) for Indonesian
* the Institute for Infocomm Research, Singapore (I2R) for Malay
* the Institute of Information Technology, Vietnam (IOIT) for Vietnamese
* the National Institute of Posts, Telecoms and ICT, Cambodia for Khmer

### Licensing Information

[Creative Commons Attribution 4.0 International (CC BY 4.0)](https://creativecommons.org/licenses/by/4.0/)

### Citation Information

Please cite the following if you make use of the dataset:

Hammam Riza, Michael Purwoadi, Gunarso, Teduh Uliniansyah, Aw Ai Ti, Sharifah Mahani Aljunied, Luong Chi Mai, Vu Tat Thang, Nguyen Phuong Thai, Vichet Chea, Rapid Sun, Sethserey Sam, Sopheap Seng, Khin Mar Soe, Khin Thandar Nwet, Masao Utiyama, Chenchen Ding. (2016) "Introduction of the Asian Language Treebank" Oriental COCOSDA.

BibTeX:
```
@inproceedings{riza2016introduction,
  title={Introduction of the asian language treebank},
  author={Riza, Hammam and Purwoadi, Michael and Uliniansyah, Teduh and Ti, Aw Ai and Aljunied, Sharifah Mahani and Mai, Luong Chi and Thang, Vu Tat and Thai, Nguyen Phuong and Chea, Vichet and Sam, Sethserey and others},
  booktitle={2016 Conference of The Oriental Chapter of International Committee for Coordination and Standardization of Speech Databases and Assessment Techniques (O-COCOSDA)},
  pages={1--6},
  year={2016},
  organization={IEEE}
}
```

### Contributions

Thanks to [@chameleonTK](https://github.com/chameleonTK) for adding this dataset.