* Language models typically need at least 50K training steps _regardless of_ batch size. Thus, for small \(N\), we _reduce the batch size_ to ensure the total number of training steps exceeds this threshold. For very large models, a larger batch size is preferred to enable GPU parallelism. * When \(lr\) remains constant, \(wd\) should be relatively reduced as the number of training steps increases. Mathematically, the model weights should be "halved" for every \(\Theta(\frac{1}{lr\times wd})\) training steps. Therefore, it's advisable to reduce the \(wd\) parameter when training for longer periods.