* For \(N=50K\), we use \(wd=0.01\), \(lr=0.001\), and batch size 12;
* For \(N=100K\), we use \(wd=0.01\), \(lr=0.001\), and batch size 24;
* For \(N=200K\), we use \(wd=0.01\), \(lr=0.001\), and batch size 48; (except for GPT2-2-20, where \(lr=0.0005\) is used)
* For \(N=500K\), we use \(wd=0.01\), \(lr=0.0005\), and batch size 96;
* For \(N=1M\), we use \(wd=0.01\), \(lr=0.0005\), and batch size 192;
* For \(N=2M\), we use \(wd=0.01\), \(lr=0.0003/0.0005/0.001\), and batch size 384;
* For \(N=5M\), we use \(wd=0.01\), \(lr=0.0003/0.0005\), and batch size 768;
* For \(N=10M\), we use \(wd=0.01\), \(lr=0.0002/0.0003/0.0005\), and batch size 1024;
* For \(N=20M\), we use \(wd=0.002\), \(lr=0.0002/0.0003/0.0005\), and batch size 1536.22