Note Contains ~2B tokens from high quality corpus. Cleaned and deduplicated.

Note Deduplicate version of erhwenkuo/wikipedia-zhtw using MinHash.

Note Deduplicated C4 subset of zhTW. Note: C4 = colossal, cleaned version of Common Crawl

Note Deduplicated CC subset of zhTW.

Note Zh-tw subset of CC-100 dataset, which is derived from commoncrawl. Note: CC harms performance as shown in TaiwanLlama.

Note Deduplicate version of zetavg/coct-en-zh-tw-translations-twp-300k. Zh-tw <-> en paired articles provided by 台灣光華雜誌.

Note Traditional Chinese instruction dataset. Contains en <-> tw pairs with system prompts to better adopt from English pre-trained models.

Note Instruction dataset used to train TaiwanLLM v1. Find more details in the paper.

Note Translated from en to zh-tw of the alpaca-gpt4 dataset.

Note zhcn/en multilingual QA translated to zhtw/en. Internal experiment shows that when transferring from English base model, traning on Q:en->A:zh or vice versa improves SFT performance.

Note The RyokoAI/ShareGPT52K dataset, converted to Markdown and labeled with the language used.

Note Google translated instruction data from English.

Note Instruction dataset in the Medicine domain. Prompts are translated then feed to Breexe model.

Note This is in Simplified Chinese.

Note Perference dataset with chosen/reject pair. Translated using m2m100.