To find the limit of continuously training this fixed-sized LM, we estimate how the model's optimal performance changes with its computation during continuous training. By optimal performance, we mean the loss of training token \(D\) is achieved by \(WSD(D,0.1D)\). With a series of \(D\), the losses will form the optimal loss envelope. Due to uncertainty about the function form of the loss envelope, we try two fitting formulas: (1) exponential: \(L(C)=\alpha e^{-\beta C}+L_{0}\) and (2) power-law: \(L(C)=\beta C^{-\alpha}+L_{0}\). The fitting results for both functions are in Appendix B.2. We find that the power-law form fits better (similar to the Cosine LRS (Kaplan et al., 2020)). In Figure 6, the fitted curve is shown in green dotted lines. To intuitively estimate and comprehend the effect of continuous training such a fixed-sized model, we also trained a 0.17B model with \(WSD(40N,4N)\), which is shown in pink in Figure 6. We can see that a 0.036B model can match the performance of a 0.17B model with an acceptable increase (\(\sim 4\) times) in training compute while saving a lot of inference computation (Sardana and Frankle, 2023) (saving \(\sim 5\) times per inference call), indicating a better inference-compute-optimal setting Sardana and Frankle (2023).