1. For LLaMA/Mistral models on \(\mathsf{bio5}(N)\) data, aiming to present _negative_ results, we select the best learning rate from three options in each data setting: * For \(N=50K\), we use \(wd=0.01\), \(lr=0.0003/0.0005/0.001\), and batch size 12 with bf16; * For \(N=100K\), we use \(wd=0.01\), \(lr=0.0003/0.0005/0.001\), and batch size 24 with bf16; * For \(N=200K\), we use \(wd=0.01\), \(lr=0.0003/0.0005/0.001\), and batch size 48 with bf16; * For \(N=500K\), we use \(wd=0.01\), \(lr=0.0002/0.0003/0.0005\), and batch size 96 with bf16; * For \(N=1M\), we use \(wd=0.01\), \(lr=0.0002/0.0003/0.0005\), and batch size 192 with bf16; * For \(N=2M\), we use \(wd=0.01\), \(lr=0.0003/0.0005/0.001\), and batch size 384 with bf16; * For \(N=5M\), we use \(wd=0.01\), \(lr=0.0003/0.0005/0.001\), and batch size 768 with bf16; * For \(N=10M\), we use \(wd=0.01\), \(lr=0.0003/0.0005/0.001\), and batch size 1536 with bf16; * For \(N=20M\), we use \(wd=0.002\), \(lr=0.0003/0.0005/0.001\), and batch size 1536 with bf16. (For \(N\leq 1M\), we also tested the same settings with fp16, finding similar results. However, LLaMA/Mistral models tend to fail more often with fp16, so we primarily used bf16.)
2. For GPT2\({}_{1/4}\): * For \(N=50K\), we use \(wd=0.01\), \(lr=0.0005/0.001\), and batch size 12 with fp16; * For \(N=100K\), we use \(wd=0.01\), \(lr=0.0005/0.001\), and batch size 24 with fp16; * For \(N=200K\), we use \(wd=0.01\), \(lr=0.0005/0.001\), and batch size 48 with fp16; * For \(N=500K\), we use \(wd=0.01\), \(lr=0.0003/0.0005\), and batch size 96 with fp16; * For \(N=1M\), we use \(wd=0.01\), \(lr=0.0003/0.0005\), and batch size 192 with fp16. 3. For GPT2\({}_{0}\), to present a _negative_ result, we use the same settings as in Parameter 4(a): * For \(N=50K\), we use \(wd=0.01\), \(lr=0.0003/0.0005/0.001\), and batch size 12 with bf16; * For \(N=100K\), we use \(wd=0.01\), \(lr=0.0003/0.0005/0.001\), and batch size 24 with bf16; * For \(N=200K\), we use \(wd=0.01\), \(lr=0.0003/0.0005/0.001\), and batch size 48 with bf16; * For \(N=500K\), we use \(wd=0.01\), \(lr=0.0002/0.0003/0.0005\), and batch size 96 with bf16; * For \(N=1M\), we use \(wd=0.01\), \(lr=0.0002/0.0003/0.0005\), and batch size 192 with bf16.