license:other region:us
\
---
license: other
---

# Polylingual Indonesia Dataset/Model Card

## Description

Polylingual Indonesia is a diverse dataset composed of a collection of publicly available data and some self-crawled data in the Indonesian language. The data originates from various sources including news articles, books, forums, and stories. This dataset is processed and filtered through specific techniques detailed in the sections below.

## Prerequisites

To run and utilize this dataset, make sure you have the `zstandard` package installed in your environment.

## Format

The original format of the dataset is JSONL with zstandard compressed

## Dataset Details

### Filtering

The dataset undergoes a filtration process using a specific filter from BigScience. Detailed information about the filter used can be found here: [BigScience Filter](https://drive.google.com/file/d/1cCJ8sWE88TRLDAa3eHLmXO4JlkR2QzLY/view?usp=sharing).

### Data Sources

The data originates from various sources, and each folder in the dataset represents the source from where the original data came. Here are the details of each folder/source:

#### 1. HPLT_filtered
- Link: [HPLT Project](https://hplt-project.org/datasets/v1)
- Source : Internet Archive Snapshot WIDE15 and WIDE17 and CC-MAIN-2022-40

#### 2. Mc4-und-id
- Data is filtered from the undefined language segment of MC4 (c4-und) using FastText, BigScience filters.
- Sample: [MC4 Sample](https://huggingface.co/datasets/allenai/c4/blob/mC4310/multilingual/c4-und.00000-00001-00002-00003-00004-00005-00006-00007.json.gz)

#### 3. Indonesia-Crawl
This folder contains a collection of the Common Crawl dataset and self crawled data, specific for the Indonesian language, accumulated from various snapshots. The data is divided into several sections:

- **Mc4 original(dedup)**: Ranging from snapshot CC-2013-20 to CC-2020-34.
- **Kopi-CC(dedup)**: This covers data from snapshot CC-2020-34 to CC-2023-06. More details can be found [here](https://huggingface.co/datasets/acul3/KoPI-CC) (Note: Last snapshot to be uploaded).
- **KoPI-CC_News**: This includes the Common Crawl News Dataset ranging from the year 2016 to 2022. Detailed information can be accessed [here](https://huggingface.co/datasets/acul3/KoPI-CC) (Note: Last snapshot to be uploaded).
- **Self Crawled Data**: consists of data crawled from various platforms including news sites, story sites, forums, and others.

## Usage

install zstandard first

```
from datasets import load_dataset

hplt = load_dataset('duckaiml/Polylingual_Id','hplt') #hplt only
mc4_und = load_dataset('duckaiml/Polylingual_Id','mc4_und') #mc4_und only
indonesia_crawl = load_dataset('duckaiml/Polylingual_Id','indonesia_crawl') #indonesia_crawl only
load_dataset('duckaiml/Polylingual_Id','full') #load all
```


## Limitation/Issue
- Although some source data already dedup separately, some duplicate might be found as dataset need dedup as whole dataset

## Contributing

Feel free to contribute to the dataset by adding more diverse sources or helping in enhancing the filtration process. 

## License

The data is collected from public sources, and it's recommended to refer to the original data sources for information on licensing and usage restrictions.

## Contact

For any queries or contributions to the dataset, please feel free to reach out (contact information to be added).
