size_categories:1K<n<10K language:vi license:mit region:us
---
license: mit
language:
- vi
size_categories:
- 1K<n<10K
---

# What is Symato CC?
To download all WARC data from Common Crawl then filter out Vietnamese in Markdown and Plaintext format.
There is 1% of Vietnamse in CC, extract all of them out should be a lot (~10TB of plaintext).

## Main contributors
- https://huggingface.co/nampdn-ai
- https://huggingface.co/binhvq
- https://huggingface.co/th1nhng0
- https://huggingface.co/iambestfeed

# Simple quality filters
To make use of raw data from common crawl, you need to do filtering and deduping. 
Below is a simple quality filtering code for reference to write your own filters.

```sh
## Convert .parquet to .jsonl.gz
mkdir -p jsonl filtered
python3 parquet2jsonl.py

## Quality filter
# wget https://huggingface.co/datasets/Symato/goods_vs_c4_cc_classifiers/resolve/main/fasttext_good_vs_c4_001.bin 
python3 filters.py jsonl/2023-14_20230401125552-20230401155552.jsonl.gz logging
```

# Disclaimer
- We use content from Common Crawl as it is. Go to CC website to know more about data.
- We provide simple quality filters code to make it easier for you to use data but no warranty the data quality meet everyone expectations. Modifiy ours or write your own filters in-case you need more advanced / better ones.

Contact **dung at symato dot xyz** if you have other questions.
