Useful Pretrain-Datasets - a Leon-Leee Collection

Leon-Leee 's Collections

GPT-4 generated datasets

Useful Pretrain-Datasets

Code, Math, and Reasoning related Instruct datasets

High Quality Instruct Dataset

Code Benchmarks

Useful Pretrain-Datasets

updated Jun 12

pretrain-datasets with (maybe) good quality

Zyphra/Zyda

Viewer • Updated Jun 19 • 4.35B • 12.3k • 68

Note English only
HuggingFaceFW/fineweb

Viewer • Updated Jul 16 • 46B • 384k • 1.75k

Note English only
HuggingFaceFW/fineweb-edu

Viewer • Updated Oct 11 • 3B • 624k • 543

Note English only
m-a-p/Matrix

Viewer • Updated Jun 3 • 2.5M • 4.27k • 154

Note zh-en bilingual
bigcode/the-stack-v2

Viewer • Updated Apr 23 • 5.45B • 15k • 288

Note code
bigcode/the-stack-v2-train-full-ids

Viewer • Updated Jun 6 • 60.5M • 5.6k • 37

Note code
togethercomputer/RedPajama-Data-V2

Updated about 12 hours ago • 3.97k • 348
cerebras/SlimPajama-627B

Preview • Updated Jul 7, 2023 • 32.3k • 427
allenai/c4

Viewer • Updated Jan 9 • 10.4B • 534k • 315
EleutherAI/pile

Updated May 3, 2023 • 1.28k • 399

Note english
Skywork/SkyPile-150B

Viewer • Updated Dec 7, 2023 • 1.76M • 5.77k • 340

Note chinese
Tele-AI/TeleChat-PTD

Updated Mar 20 • 950 • 164

Note chinese
wenge-research/yayi2_pretrain_data

Viewer • Updated Dec 29, 2023 • 1.68M • 1.26k • 51

Note chinese and english
wikimedia/wikipedia

Viewer • Updated Jan 9 • 61.6M • 62.2k • 608

Note multi-lingual
liwu/MNBVC

Updated Feb 12, 2023 • 3

Note chinese
oscar-corpus/oscar

Updated Mar 21 • 11.2k • 176

Note multi-lingual
oscar-corpus/colossal-oscar-1.0

Preview • Updated Oct 26, 2023 • 39 • 30

Note multi-lingual
uonlp/CulturaX

Viewer • Updated Jul 23 • 7.18B • 10.6k • 475

Note multi-lingual
JosephusCheung/GuanacoDataset

Viewer • Updated Apr 15 • 1.17M • 47 • 513
allenai/MADLAD-400

Updated Sep 9 • 83.8k • 128

Note multi-lingual