task_categories:question-answering task_categories:text-classification task_categories:token-classification task_ids:closed-domain-qa task_ids:multi-class-classification task_ids:named-entity-recognition task_ids:part-of-speech task_ids:semantic-similarity-classification task_ids:sentiment-classification annotations_creators:expert-generated language_creators:expert-generated multilinguality:monolingual size_categories:10K<n<100K size_categories:1K<n<10K size_categories:n<1K source_datasets:original language:id license:mit keyphrase-extraction span-extraction aspect-based-sentiment-analysis arxiv:1809.03391 region:us
The IndoNLU benchmark is a collection of resources for training, evaluating, and analyzing natural language understanding systems for Bahasa Indonesia.
@inproceedings{wilie2020indonlu,
title = {{IndoNLU}: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding},
authors={Bryan Wilie and Karissa Vincentio and Genta Indra Winata and Samuel Cahyawijaya and X. Li and Zhi Yuan Lim and S. Soleman and R. Mahendra and Pascale Fung and Syafri Bahar and A. Purwarianti},
booktitle={Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing},
year={2020}
}
---
annotations_creators:
- expert-generated
language_creators:
- expert-generated
language:
- id
license:
- mit
multilinguality:
- monolingual
size_categories:
- 10K<n<100K
- 1K<n<10K
- n<1K
source_datasets:
- original
task_categories:
- question-answering
- text-classification
- token-classification
task_ids:
- closed-domain-qa
- multi-class-classification
- named-entity-recognition
- part-of-speech
- semantic-similarity-classification
- sentiment-classification
paperswithcode_id: indonlu-benchmark
pretty_name: IndoNLU
configs:
- bapos
- casa
- emot
- facqa
- hoasa
- keps
- nergrit
- nerp
- posp
- smsa
- terma
- wrete
tags:
- keyphrase-extraction
- span-extraction
- aspect-based-sentiment-analysis
dataset_info:
- config_name: emot
  features:
  - name: tweet
    dtype: string
  - name: label
    dtype:
      class_label:
        names:
          0: sadness
          1: anger
          2: love
          3: fear
          4: happy
  splits:
  - name: train
    num_bytes: 686418
    num_examples: 3521
  - name: validation
    num_bytes: 84082
    num_examples: 440
  - name: test
    num_bytes: 84856
    num_examples: 440
  download_size: 840917
  dataset_size: 855356
- config_name: smsa
  features:
  - name: text
    dtype: string
  - name: label
    dtype:
      class_label:
        names:
          0: positive
          1: neutral
          2: negative
  splits:
  - name: train
    num_bytes: 2209874
    num_examples: 11000
  - name: validation
    num_bytes: 249629
    num_examples: 1260
  - name: test
    num_bytes: 77041
    num_examples: 500
  download_size: 2509229
  dataset_size: 2536544
- config_name: casa
  features:
  - name: sentence
    dtype: string
  - name: fuel
    dtype:
      class_label:
        names:
          0: negative
          1: neutral
          2: positive
  - name: machine
    dtype:
      class_label:
        names:
          0: negative
          1: neutral
          2: positive
  - name: others
    dtype:
      class_label:
        names:
          0: negative
          1: neutral
          2: positive
  - name: part
    dtype:
      class_label:
        names:
          0: negative
          1: neutral
          2: positive
  - name: price
    dtype:
      class_label:
        names:
          0: negative
          1: neutral
          2: positive
  - name: service
    dtype:
      class_label:
        names:
          0: negative
          1: neutral
          2: positive
  splits:
  - name: train
    num_bytes: 110415
    num_examples: 810
  - name: validation
    num_bytes: 11993
    num_examples: 90
  - name: test
    num_bytes: 23553
    num_examples: 180
  download_size: 144903
  dataset_size: 145961
- config_name: hoasa
  features:
  - name: sentence
    dtype: string
  - name: ac
    dtype:
      class_label:
        names:
          0: neg
          1: neut
          2: pos
          3: neg_pos
  - name: air_panas
    dtype:
      class_label:
        names:
          0: neg
          1: neut
          2: pos
          3: neg_pos
  - name: bau
    dtype:
      class_label:
        names:
          0: neg
          1: neut
          2: pos
          3: neg_pos
  - name: general
    dtype:
      class_label:
        names:
          0: neg
          1: neut
          2: pos
          3: neg_pos
  - name: kebersihan
    dtype:
      class_label:
        names:
          0: neg
          1: neut
          2: pos
          3: neg_pos
  - name: linen
    dtype:
      class_label:
        names:
          0: neg
          1: neut
          2: pos
          3: neg_pos
  - name: service
    dtype:
      class_label:
        names:
          0: neg
          1: neut
          2: pos
          3: neg_pos
  - name: sunrise_meal
    dtype:
      class_label:
        names:
          0: neg
          1: neut
          2: pos
          3: neg_pos
  - name: tv
    dtype:
      class_label:
        names:
          0: neg
          1: neut
          2: pos
          3: neg_pos
  - name: wifi
    dtype:
      class_label:
        names:
          0: neg
          1: neut
          2: pos
          3: neg_pos
  splits:
  - name: train
    num_bytes: 458177
    num_examples: 2283
  - name: validation
    num_bytes: 58248
    num_examples: 285
  - name: test
    num_bytes: 56399
    num_examples: 286
  download_size: 477314
  dataset_size: 572824
- config_name: wrete
  features:
  - name: premise
    dtype: string
  - name: hypothesis
    dtype: string
  - name: category
    dtype: string
  - name: label
    dtype:
      class_label:
        names:
          0: NotEntail
          1: Entail_or_Paraphrase
  splits:
  - name: train
    num_bytes: 99999
    num_examples: 300
  - name: validation
    num_bytes: 18049
    num_examples: 50
  - name: test
    num_bytes: 32617
    num_examples: 100
  download_size: 151018
  dataset_size: 150665
- config_name: posp
  features:
  - name: tokens
    sequence: string
  - name: pos_tags
    sequence:
      class_label:
        names:
          0: B-PPO
          1: B-KUA
          2: B-ADV
          3: B-PRN
          4: B-VBI
          5: B-PAR
          6: B-VBP
          7: B-NNP
          8: B-UNS
          9: B-VBT
          10: B-VBL
          11: B-NNO
          12: B-ADJ
          13: B-PRR
          14: B-PRK
          15: B-CCN
          16: B-$$$
          17: B-ADK
          18: B-ART
          19: B-CSN
          20: B-NUM
          21: B-SYM
          22: B-INT
          23: B-NEG
          24: B-PRI
          25: B-VBE
  splits:
  - name: train
    num_bytes: 2751348
    num_examples: 6720
  - name: validation
    num_bytes: 343924
    num_examples: 840
  - name: test
    num_bytes: 350720
    num_examples: 840
  download_size: 2407206
  dataset_size: 3445992
- config_name: bapos
  features:
  - name: tokens
    sequence: string
  - name: pos_tags
    sequence:
      class_label:
        names:
          0: B-PR
          1: B-CD
          2: I-PR
          3: B-SYM
          4: B-JJ
          5: B-DT
          6: I-UH
          7: I-NND
          8: B-SC
          9: I-WH
          10: I-IN
          11: I-NNP
          12: I-VB
          13: B-IN
          14: B-NND
          15: I-CD
          16: I-JJ
          17: I-X
          18: B-OD
          19: B-RP
          20: B-RB
          21: B-NNP
          22: I-RB
          23: I-Z
          24: B-CC
          25: B-NEG
          26: B-VB
          27: B-NN
          28: B-MD
          29: B-UH
          30: I-NN
          31: B-PRP
          32: I-SC
          33: B-Z
          34: I-PRP
          35: I-OD
          36: I-SYM
          37: B-WH
          38: B-FW
          39: I-CC
          40: B-X
  splits:
  - name: train
    num_bytes: 3772459
    num_examples: 8000
  - name: validation
    num_bytes: 460058
    num_examples: 1000
  - name: test
    num_bytes: 474368
    num_examples: 1029
  download_size: 3084021
  dataset_size: 4706885
- config_name: terma
  features:
  - name: tokens
    sequence: string
  - name: seq_label
    sequence:
      class_label:
        names:
          0: I-SENTIMENT
          1: O
          2: I-ASPECT
          3: B-SENTIMENT
          4: B-ASPECT
  splits:
  - name: train
    num_bytes: 817983
    num_examples: 3000
  - name: validation
    num_bytes: 276335
    num_examples: 1000
  - name: test
    num_bytes: 265922
    num_examples: 1000
  download_size: 816822
  dataset_size: 1360240
- config_name: keps
  features:
  - name: tokens
    sequence: string
  - name: seq_label
    sequence:
      class_label:
        names:
          0: O
          1: B
          2: I
  splits:
  - name: train
    num_bytes: 173961
    num_examples: 800
  - name: validation
    num_bytes: 42961
    num_examples: 200
  - name: test
    num_bytes: 66762
    num_examples: 247
  download_size: 134042
  dataset_size: 283684
- config_name: nergrit
  features:
  - name: tokens
    sequence: string
  - name: ner_tags
    sequence:
      class_label:
        names:
          0: I-PERSON
          1: B-ORGANISATION
          2: I-ORGANISATION
          3: B-PLACE
          4: I-PLACE
          5: O
          6: B-PERSON
  splits:
  - name: train
    num_bytes: 960710
    num_examples: 1672
  - name: validation
    num_bytes: 119567
    num_examples: 209
  - name: test
    num_bytes: 117274
    num_examples: 209
  download_size: 641265
  dataset_size: 1197551
- config_name: nerp
  features:
  - name: tokens
    sequence: string
  - name: ner_tags
    sequence:
      class_label:
        names:
          0: I-PPL
          1: B-EVT
          2: B-PLC
          3: I-IND
          4: B-IND
          5: B-FNB
          6: I-EVT
          7: B-PPL
          8: I-PLC
          9: O
          10: I-FNB
  splits:
  - name: train
    num_bytes: 2751348
    num_examples: 6720
  - name: validation
    num_bytes: 343924
    num_examples: 840
  - name: test
    num_bytes: 350720
    num_examples: 840
  download_size: 1725986
  dataset_size: 3445992
- config_name: facqa
  features:
  - name: question
    sequence: string
  - name: passage
    sequence: string
  - name: seq_label
    sequence:
      class_label:
        names:
          0: O
          1: B
          2: I
  splits:
  - name: train
    num_bytes: 2454368
    num_examples: 2495
  - name: validation
    num_bytes: 306249
    num_examples: 311
  - name: test
    num_bytes: 306831
    num_examples: 311
  download_size: 2591968
  dataset_size: 3067448
---


# Dataset Card for IndoNLU

## Table of Contents
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
  - [Curation Rationale](#curation-rationale)
  - [Source Data](#source-data)
  - [Annotations](#annotations)
  - [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
  - [Social Impact of Dataset](#social-impact-of-dataset)
  - [Discussion of Biases](#discussion-of-biases)
  - [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
  - [Dataset Curators](#dataset-curators)
  - [Licensing Information](#licensing-information)
  - [Citation Information](#citation-information)
  - [Contributions](#contributions)

## Dataset Description

- **Homepage:** [IndoNLU Website](https://www.indobenchmark.com/)
- **Repository:** [IndoNLU GitHub](https://github.com/indobenchmark/indonlu)
- **Paper:** [IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding](https://www.aclweb.org/anthology/2020aacl-main.85.pdf)
- **Leaderboard:** [Needs More Information]
- **Point of Contact:** [Needs More Information]

### Dataset Summary

The IndoNLU benchmark is a collection of resources for training, evaluating, and analyzing natural language understanding systems for Bahasa Indonesia (Indonesian language).
There are 12 datasets in IndoNLU benchmark for Indonesian natural language understanding.
1. `EmoT`: An emotion classification dataset collected from the social media platform Twitter. The dataset consists of around 4000 Indonesian colloquial language tweets, covering five different emotion labels: anger, fear, happy, love, and sadness
2. `SmSA`: This sentence-level sentiment analysis dataset is a collection of comments and reviews in Indonesian obtained from multiple online platforms. The text was crawled and then annotated by several Indonesian linguists to construct this dataset. There are three possible sentiments on the `SmSA` dataset: positive, negative, and neutral
3. `CASA`: An aspect-based sentiment analysis dataset consisting of around a thousand car reviews collected from multiple Indonesian online automobile platforms. The dataset covers six aspects of car quality. We define the task to be a multi-label classification task, where each label represents a sentiment for a single aspect with three possible values: positive, negative, and neutral.
4. `HoASA`: An aspect-based sentiment analysis dataset consisting of hotel reviews collected from the hotel aggregator platform, [AiryRooms](https://github.com/annisanurulazhar/absa-playground). The dataset covers ten different aspects of hotel quality. Similar to the `CASA` dataset, each review is labeled with a single sentiment label for each aspect. There are four possible sentiment classes for each sentiment label: positive, negative, neutral, and positive-negative. The positivenegative label is given to a review that contains multiple sentiments of the same aspect but for different objects (e.g., cleanliness of bed and toilet).
5. `WReTE`: The Wiki Revision Edits Textual Entailment dataset consists of 450 sentence pairs constructed from Wikipedia revision history. The dataset contains pairs of sentences and binary semantic relations between the pairs. The data are labeled as entailed when the meaning of the second sentence can be derived from the first one, and not entailed otherwise.
6. `POSP`: This Indonesian part-of-speech tagging (POS) dataset is collected from Indonesian news websites. The dataset consists of around 8000 sentences with 26 POS tags. The POS tag labels follow the [Indonesian Association of Computational Linguistics (INACL) POS Tagging Convention](http://inacl.id/inacl/wp-content/uploads/2017/06/INACL-POS-Tagging-Convention-26-Mei.pdf).
7. `BaPOS`: This POS tagging dataset contains about 1000 sentences, collected from the [PAN Localization Project](http://www.panl10n.net/). In this dataset, each word is tagged by one of [23 POS tag classes](https://bahasa.cs.ui.ac.id/postag/downloads/Tagset.pdf). Data splitting used in this benchmark follows the experimental setting used by [Kurniawan and Aji (2018)](https://arxiv.org/abs/1809.03391).
8. `TermA`: This span-extraction dataset is collected from the hotel aggregator platform, [AiryRooms](https://github.com/jordhy97/final_project). The dataset consists of thousands of hotel reviews, which each contain a span label for aspect and sentiment words representing the opinion of the reviewer on the corresponding aspect. The labels use Inside-Outside-Beginning (IOB) tagging representation with two kinds of tags, aspect and sentiment.
9. `KEPS`: This keyphrase extraction dataset consists of text from Twitter discussing banking products and services and is written in the Indonesian language. A phrase containing important information is considered a keyphrase. Text may contain one or more keyphrases since important phrases can be located at different positions. The dataset follows the IOB chunking format, which represents the position of the keyphrase.
10. `NERGrit`: This NER dataset is taken from the [Grit-ID repository](https://github.com/grit-id/nergrit-corpus), and the labels are spans in IOB chunking representation. The dataset consists of three kinds of named entity tags, PERSON (name of person), PLACE (name of location), and ORGANIZATION (name of organization).
11. `NERP`: This NER dataset (Hoesen and Purwarianti, 2018) contains texts collected from several Indonesian news websites. There are five labels available in this dataset, PER (name of person), LOC (name of location), IND (name of product or brand), EVT (name of the event), and FNB (name of food and beverage). Similar to the `TermA` dataset, the `NERP` dataset uses the IOB chunking format.
12. `FacQA`: The goal of the FacQA dataset is to find the answer to a question from a provided short passage from a news article. Each row in the FacQA dataset consists of a question, a short passage, and a label phrase, which can be found inside the corresponding short passage. There are six categories of questions: date, location, name, organization, person, and quantitative.

### Supported Tasks and Leaderboards

[Needs More Information]

### Languages

Indonesian

## Dataset Structure

### Data Instances

1. `EmoT` dataset

A data point consists of `tweet` and `label`. An example from the train set looks as follows:
```
{
  'tweet': 'Ini adalah hal yang paling membahagiakan saat biasku foto bersama ELF #ReturnOfTheLittlePrince #HappyHeeChulDay'
  'label': 4,
}
```

2. `SmSA` dataset

A data point consists of `text` and `label`. An example from the train set looks as follows:
```
{
  'text': 'warung ini dimiliki oleh pengusaha pabrik tahu yang sudah puluhan tahun terkenal membuat tahu putih di bandung . tahu berkualitas , dipadu keahlian memasak , dipadu kretivitas , jadilah warung yang menyajikan menu utama berbahan tahu , ditambah menu umum lain seperti ayam . semuanya selera indonesia . harga cukup terjangkau . jangan lewatkan tahu bletoka nya , tidak kalah dengan yang asli dari tegal !'
  'label': 0,
}
```

3. `CASA` dataset

A data point consists of `sentence` and multi-label `feature`, `machine`, `others`, `part`, `price`, and `service`. An example from the train set looks as follows:
```
{
  'sentence': 'Saya memakai Honda Jazz GK5 tahun 2014 ( pertama meluncur ) . Mobil nya bagus dan enak sesuai moto nya menyenangkan untuk dikendarai',
  'fuel': 1,
  'machine': 1,
  'others': 2,
  'part': 1,
  'price': 1,
  'service': 1
}
```

4. `HoASA` dataset

A data point consists of `sentence` and multi-label `ac`, `air_panas`, `bau`, `general`, `kebersihan`, `linen`, `service`, `sunrise_meal`, `tv`, and `wifi`. An example from the train set looks as follows:
```
{
  'sentence': 'kebersihan kurang...',
  'ac': 1,
  'air_panas': 1,
  'bau': 1,
  'general': 1,
  'kebersihan': 0,
  'linen': 1,
  'service': 1,
  'sunrise_meal': 1,
  'tv': 1,
  'wifi': 1
}
```

5. `WreTE` dataset

A data point consists of `premise`, `hypothesis`, `category`, and `label`. An example from the train set looks as follows:
```
{
  'premise': 'Pada awalnya bangsa Israel hanya terdiri dari satu kelompok keluarga di antara banyak kelompok keluarga yang hidup di tanah Kanan pada abad 18 SM .',
  'hypothesis': 'Pada awalnya bangsa Yahudi hanya terdiri dari satu kelompok keluarga di antara banyak kelompok keluarga yang hidup di tanah Kanan pada abad 18 SM .'
  'category': 'menolak perubahan teks terakhir oleh istimewa kontribusi pengguna 141 109 98 87 141 109 98 87 dan mengembalikan revisi 6958053 oleh johnthorne',
  'label': 0,
}
```

6. `POSP` dataset

A data point consists of `tokens` and `pos_tags`. An example from the train set looks as follows:
```
{
  'tokens': ['kepala', 'dinas', 'tata', 'kota', 'manado', 'amos', 'kenda', 'menyatakan', 'tidak', 'tahu', '-', 'menahu', 'soal', 'pencabutan', 'baliho', '.', 'ia', 'enggan', 'berkomentar', 'banyak', 'karena', 'merasa', 'bukan', 'kewenangannya', '.'],
  'pos_tags': [11, 6, 11, 11, 7, 7, 7, 9, 23, 4, 21, 9, 11, 11, 11, 21, 3, 2, 4, 1, 19, 9, 23, 11, 21]
}
```

7. `BaPOS` dataset

A data point consists of `tokens` and `pos_tags`. An example from the train set looks as follows:
```
{
  'tokens': ['Kera', 'untuk', 'amankan', 'pesta', 'olahraga'],
  'pos_tags': [27, 8, 26, 27, 30]
}
```

8. `TermA` dataset

A data point consists of `tokens` and `seq_label`. An example from the train set looks as follows:
```
{
  'tokens': ['kamar', 'saya', 'ada', 'kendala', 'di', 'ac', 'tidak', 'berfungsi', 'optimal', '.', 'dan', 'juga', 'wifi', 'koneksi', 'kurang', 'stabil', '.'],
  'seq_label': [1, 1, 1, 1, 1, 4, 3, 0, 0, 1, 1, 1, 4, 2, 3, 0, 1]
}
```

9. `KEPS` dataset

A data point consists of `tokens` and `seq_label`. An example from the train set looks as follows:
```
{
  'tokens': ['Setelah', 'melalui', 'proses', 'telepon', 'yang', 'panjang', 'tutup', 'sudah', 'kartu', 'kredit', 'bca', 'Ribet'],
  'seq_label': [0, 1, 1, 2, 0, 0, 1, 0, 1, 2, 2, 1]
}
```

10. `NERGrit` dataset

A data point consists of `tokens` and `ner_tags`. An example from the train set looks as follows:
```
{
  'tokens': ['Kontribusinya', 'terhadap', 'industri', 'musik', 'telah', 'mengumpulkan', 'banyak', 'prestasi', 'termasuk', 'lima', 'Grammy', 'Awards', ',', 'serta', 'dua', 'belas', 'nominasi', ';', 'dua', 'Guinness', 'World', 'Records', ';', 'dan', 'penjualannya', 'diperkirakan', 'sekitar', '64', 'juta', 'rekaman', '.'],
  'ner_tags': [5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5]}
```

11. `NERP` dataset

A data point consists of `tokens` and `ner_tags`. An example from the train set looks as follows:
```
{
  'tokens': ['kepala', 'dinas', 'tata', 'kota', 'manado', 'amos', 'kenda', 'menyatakan', 'tidak', 'tahu', '-', 'menahu', 'soal', 'pencabutan', 'baliho', '.', 'ia', 'enggan', 'berkomentar', 'banyak', 'karena', 'merasa', 'bukan', 'kewenangannya', '.'],
  'ner_tags': [9, 9, 9, 9, 2, 7, 0, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9]
}
```

12. `FacQA` dataset

A data point consists of `question`, `passage`, and `seq_label`. An example from the train set looks as follows:
```
{
  'passage': ['Lewat', 'telepon', 'ke', 'kantor', 'berita', 'lokal', 'Current', 'News', 'Service', ',', 'Hezb-ul', 'Mujahedeen', ',', 'kelompok', 'militan', 'Kashmir', 'yang', 'terbesar', ',', 'menyatakan', 'bertanggung', 'jawab', 'atas', 'ledakan', 'di', 'Srinagar', '.'],
  'question': ['Kelompok', 'apakah', 'yang', 'menyatakan', 'bertanggung', 'jawab', 'atas', 'ledakan', 'di', 'Srinagar', '?'],
  'seq_label': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
}
```

### Data Fields

1. `EmoT` dataset

- `tweet`: a `string` feature.
- `label`: an emotion label, with possible values including `sadness`, `anger`, `love`, `fear`, `happy`.

2. `SmSA` dataset

- `text`: a `string` feature.
- `label`: a sentiment label, with possible values including `positive`, `neutral`, `negative`.

3. `CASA` dataset

- `sentence`: a `string` feature.
- `fuel`: a sentiment label, with possible values including `negative`, `neutral`, `positive`.
- `machine`: a sentiment label, with possible values including `negative`, `neutral`, `positive`.
- `others`: a sentiment label, with possible values including `negative`, `neutral`, `positive`.
- `part`: a sentiment label, with possible values including `negative`, `neutral`, `positive`.
- `price`: a sentiment label, with possible values including `negative`, `neutral`, `positive`.
- `service`: a sentiment label, with possible values including `negative`, `neutral`, `positive`.

4. `HoASA` dataset

- `sentence`: a `string` feature.
- `ac`: a sentiment label, with possible values including `neg`, `neut`, `pos`, `neg_pos`.
- `air_panas`: a sentiment label, with possible values including `neg`, `neut`, `pos`, `neg_pos`.
- `bau`: a sentiment label, with possible values including `neg`, `neut`, `pos`, `neg_pos`.
- `general`: a sentiment label, with possible values including `neg`, `neut`, `pos`, `neg_pos`.
- `kebersihan`: a sentiment label, with possible values including `neg`, `neut`, `pos`, `neg_pos`.
- `linen`: a sentiment label, with possible values including `neg`, `neut`, `pos`, `neg_pos`.
- `service`: a sentiment label, with possible values including `neg`, `neut`, `pos`, `neg_pos`.
- `sunrise_meal`: a sentiment label, with possible values including `neg`, `neut`, `pos`, `neg_pos`.
- `tv`: a sentiment label, with possible values including `neg`, `neut`, `pos`, `neg_pos`.
- `wifi`: a sentiment label, with possible values including `neg`, `neut`, `pos`, `neg_pos`.

5. `WReTE` dataset

- `premise`: a `string` feature.
- `hypothesis`: a `string` feature.
- `category`: a `string` feature.
- `label`: a classification label, with possible values including `NotEntail`, `Entail_or_Paraphrase`.

6. `POSP` dataset

- `tokens`: a `list` of `string` features.
- `pos_tags`: a `list` of POS tag labels, with possible values including `B-PPO`, `B-KUA`, `B-ADV`, `B-PRN`, `B-VBI`.

The POS tag labels follow the [Indonesian Association of Computational Linguistics (INACL) POS Tagging Convention](http://inacl.id/inacl/wp-content/uploads/2017/06/INACLPOS-Tagging-Convention-26-Mei.pdf).

7. `BaPOS` dataset

- `tokens`: a `list` of `string` features.
- `pos_tags`: a `list` of POS tag labels, with possible values including `B-PR`, `B-CD`, `I-PR`, `B-SYM`, `B-JJ`.

The POS tag labels from [Tagset UI](https://bahasa.cs.ui.ac.id/postag/downloads/Tagset.pdf).

8. `TermA` dataset

- `tokens`: a `list` of `string` features.
- `seq_label`: a `list` of classification labels, with possible values including `I-SENTIMENT`, `O`, `I-ASPECT`, `B-SENTIMENT`, `B-ASPECT`.

9. `KEPS` dataset

- `tokens`: a `list` of `string` features.
- `seq_label`: a `list` of classification labels, with possible values including `O`, `B`, `I`.

The labels use Inside-Outside-Beginning (IOB) tagging.

10. `NERGrit` dataset

- `tokens`: a `list` of `string` features.
- `ner_tags`: a `list` of NER tag labels, with possible values including `I-PERSON`, `B-ORGANISATION`, `I-ORGANISATION`, `B-PLACE`, `I-PLACE`.

The labels use Inside-Outside-Beginning (IOB) tagging.

11. `NERP` dataset

- `tokens`: a `list` of `string` features.
- `ner_tags`: a `list` of NER tag labels, with possible values including `I-PPL`, `B-EVT`, `B-PLC`, `I-IND`, `B-IND`.

12. `FacQA` dataset

- `question`: a `list` of `string` features.
- `passage`: a `list` of `string` features.
- `seq_label`: a `list` of classification labels, with possible values including `O`, `B`, `I`.

### Data Splits

The data is split into a training, validation and test set.

|    | dataset | Train | Valid | Test |
|----|---------|-------|-------|------|
| 1  | EmoT    | 3521  | 440   | 440  |
| 2  | SmSA    | 11000 | 1260  | 500  |
| 3  | CASA    | 810   | 90    | 180  |
| 4  | HoASA   | 2283  | 285   | 286  |
| 5  | WReTE   | 300   | 50    | 100  |
| 6  | POSP    | 6720  | 840   | 840  |
| 7  | BaPOS   | 8000  | 1000  | 1029 |
| 8  | TermA   | 3000  | 1000  | 1000 |
| 9  | KEPS    | 800   | 200   | 247  |
| 10 | NERGrit | 1672  | 209   | 209  |
| 11 | NERP    | 6720  | 840   | 840  |
| 12 | FacQA   | 2495  | 311   | 311  |

## Dataset Creation

### Curation Rationale

[Needs More Information]

### Source Data

#### Initial Data Collection and Normalization

[Needs More Information]

#### Who are the source language producers?

[Needs More Information]

### Annotations

#### Annotation process

[Needs More Information]

#### Who are the annotators?

[Needs More Information]

### Personal and Sensitive Information

[Needs More Information]

## Considerations for Using the Data

### Social Impact of Dataset

[Needs More Information]

### Discussion of Biases

[Needs More Information]

### Other Known Limitations

[Needs More Information]

## Additional Information

### Dataset Curators

[Needs More Information]

### Licensing Information

The licensing status of the IndoNLU benchmark datasets is under MIT License.

### Citation Information

IndoNLU citation
```
@inproceedings{wilie2020indonlu,
  title={IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding},
  author={Bryan Wilie and Karissa Vincentio and Genta Indra Winata and Samuel Cahyawijaya and X. Li and Zhi Yuan Lim and S. Soleman and R. Mahendra and Pascale Fung and Syafri Bahar and A. Purwarianti},
  booktitle={Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing},
  year={2020}
}
```

`EmoT` dataset citation
```
@inproceedings{saputri2018emotion,
  title={Emotion Classification on Indonesian Twitter Dataset},
  author={Mei Silviana Saputri, Rahmad Mahendra, and Mirna Adriani},
  booktitle={Proceedings of the 2018 International Conference on Asian Language Processing(IALP)},
  pages={90--95},
  year={2018},
  organization={IEEE}
}
```

`SmSA` dataset citation
```
@inproceedings{purwarianti2019improving,
  title={Improving Bi-LSTM Performance for Indonesian Sentiment Analysis Using Paragraph Vector},
  author={Ayu Purwarianti and Ida Ayu Putu Ari Crisdayanti},
  booktitle={Proceedings of the 2019 International Conference of Advanced Informatics: Concepts, Theory and Applications (ICAICTA)},
  pages={1--5},
  year={2019},
  organization={IEEE}
}
```

`CASA` dataset citation
```
@inproceedings{ilmania2018aspect,
  title={Aspect Detection and Sentiment Classification Using Deep Neural Network for Indonesian Aspect-based Sentiment Analysis},
  author={Arfinda Ilmania, Abdurrahman, Samuel Cahyawijaya, Ayu Purwarianti},
  booktitle={Proceedings of the 2018 International Conference on Asian Language Processing(IALP)},
  pages={62--67},
  year={2018},
  organization={IEEE}
}
```

`HoASA` dataset citation
```
@inproceedings{azhar2019multi,
  title={Multi-label Aspect Categorization with Convolutional Neural Networks and Extreme Gradient Boosting},
  author={A. N. Azhar, M. L. Khodra, and A. P. Sutiono}
  booktitle={Proceedings of the 2019 International Conference on Electrical Engineering and Informatics (ICEEI)},
  pages={35--40},
  year={2019}
}
```

`WReTE` dataset citation
```
@inproceedings{setya2018semi,
  title={Semi-supervised Textual Entailment on Indonesian Wikipedia Data},
  author={Ken Nabila Setya and Rahmad Mahendra},
  booktitle={Proceedings of the 2018 International Conference on Computational Linguistics and Intelligent Text Processing (CICLing)},
  year={2018}
}
```

`POSP` dataset citation
```
@inproceedings{hoesen2018investigating,
  title={Investigating Bi-LSTM and CRF with POS Tag Embedding for Indonesian Named Entity Tagger},
  author={Devin Hoesen and Ayu Purwarianti},
  booktitle={Proceedings of the 2018 International Conference on Asian Language Processing (IALP)},
  pages={35--38},
  year={2018},
  organization={IEEE}
}
```

`BaPOS` dataset citation
```
@inproceedings{dinakaramani2014designing,
  title={Designing an Indonesian Part of Speech Tagset and Manually Tagged Indonesian Corpus},
  author={Arawinda Dinakaramani, Fam Rashel, Andry Luthfi, and Ruli Manurung},
  booktitle={Proceedings of the 2014 International Conference on Asian Language Processing (IALP)},
  pages={66--69},
  year={2014},
  organization={IEEE}
}
@inproceedings{kurniawan2018toward,
  title={Toward a Standardized and More Accurate Indonesian Part-of-Speech Tagging},
  author={Kemal Kurniawan and Alham Fikri Aji},
  booktitle={Proceedings of the 2018 International Conference on Asian Language Processing (IALP)},
  pages={303--307},
  year={2018},
  organization={IEEE}
}
```

`TermA` dataset citation
```
@article{winatmoko2019aspect,
  title={Aspect and Opinion Term Extraction for Hotel Reviews Using Transfer Learning and Auxiliary Labels},
  author={Yosef Ardhito Winatmoko, Ali Akbar Septiandri, Arie Pratama Sutiono},
  journal={arXiv preprint arXiv:1909.11879},
  year={2019}
}
@article{fernando2019aspect,
  title={Aspect and Opinion Terms Extraction Using Double Embeddings and Attention Mechanism for Indonesian Hotel Reviews},
  author={Jordhy Fernando, Masayu Leylia Khodra, Ali Akbar Septiandri},
  journal={arXiv preprint arXiv:1908.04899},
  year={2019}
}
```

`KEPS` dataset citation
```
@inproceedings{mahfuzh2019improving,
  title={Improving Joint Layer RNN based Keyphrase Extraction by Using Syntactical Features},
  author={Miftahul Mahfuzh, Sidik Soleman, and Ayu Purwarianti},
  booktitle={Proceedings of the 2019 International Conference of Advanced Informatics: Concepts, Theory and Applications (ICAICTA)},
  pages={1--6},
  year={2019},
  organization={IEEE}
}
```

`NERGrit` dataset citation
```
@online{nergrit2019,
  title={NERGrit Corpus},
  author={NERGrit Developers},
  year={2019},
  url={https://github.com/grit-id/nergrit-corpus}
}
```

`NERP` dataset citation
```
@inproceedings{hoesen2018investigating,
  title={Investigating Bi-LSTM and CRF with POS Tag Embedding for Indonesian Named Entity Tagger},
  author={Devin Hoesen and Ayu Purwarianti},
  booktitle={Proceedings of the 2018 International Conference on Asian Language Processing (IALP)},
  pages={35--38},
  year={2018},
  organization={IEEE}
}
```

`FacQA` dataset citation
```
@inproceedings{purwarianti2007machine,
  title={A Machine Learning Approach for Indonesian Question Answering System},
  author={Ayu Purwarianti, Masatoshi Tsuchiya, and Seiichi Nakagawa},
  booktitle={Proceedings of Artificial Intelligence and Applications },
  pages={573--578},
  year={2007}
}
```

### Contributions

Thanks to [@yasirabd](https://github.com/yasirabd) for adding this dataset.