language:ind language:jav language:ace language:ban language:bjn language:min language:sun self-supervised-pretraining arxiv:2205.12654 arxiv:2207.04672 region:us

KopI(Korpus Perayapan Indonesia)-NLLB, is Indonesian family language(aceh,bali,banjar,indonesia,jawa,minang,sunda) only extracted from NLLB Dataset, allenai/nllb

each language set also filtered using some some deduplicate technique such as exact hash(md5) dedup technique and minhash LSH neardup
Hefferman et al, Bitext Mining Using Distilled Sentence Representations for Low-Resource Languages. Arxiv https://arxiv.org/abs/2205.12654, 2022.
NLLB Team et al, No Language Left Behind: Scaling Human-Centered Machine Translation, Arxiv https://arxiv.org/abs/2207.04672, 2022.
---
tags:
- self-supervised-pretraining
language:
- ind
- jav
- ace
- ban
- bjn
- min
- sun
---

# kopi_nllb

KopI(Korpus Perayapan Indonesia)-NLLB, is Indonesian family language(aceh,bali,banjar,indonesia,jawa,minang,sunda) only extracted from NLLB Dataset, allenai/nllb



each language set also filtered using some some deduplicate technique such as exact hash(md5) dedup technique and minhash LSH neardup

## Dataset Usage

Run `pip install nusacrowd` before loading the dataset through HuggingFace's `load_dataset`.

## Citation

```
Hefferman et al, Bitext Mining Using Distilled Sentence Representations for Low-Resource Languages. Arxiv https://arxiv.org/abs/2205.12654, 2022.
NLLB Team et al, No Language Left Behind: Scaling Human-Centered Machine Translation, Arxiv https://arxiv.org/abs/2207.04672, 2022.
```

## License

ODC_C

## Homepage

[https://huggingface.co/datasets/munggok/KoPI-NLLB](https://huggingface.co/datasets/munggok/KoPI-NLLB)

### NusaCatalogue

For easy indexing and metadata: [https://indonlp.github.io/nusa-catalogue](https://indonlp.github.io/nusa-catalogue)