Determining \(N\) and \(S\)In practice, we often opt for the largest affordable model size and train the model until convergence. Nevertheless, this simplistic approach can deviate significantly from the optimal configuration and result in substantial resource wastage. Scaling laws provide a principled approach to choosing the optimal model size \(N\) and number of training steps \(S\) given a fixed computational budget \(C\)8. Given that Eq 3.13 already provides the precise relation between the loss \(L\), batch size \(B\), model size \(N\) and training steps \(S\), we could find the model size that minimizes \(L\) under the critical batch size (\(B=B_{crit}\)). This optimal \(N\) can be obtained by taking the derivative of Eq 3.13 w.r.t. \(N\) and setting it to \(0\).