Determining \(B\)As long as all hyperparameters are well-tuned (especially the learning rate and regularization hyperparameters) and the number of training steps is sufficient, it is believed that the same final performance should be attainable using any batch size [19], so the batch size mainly influences the training speed of language models. Often, when training large language models, the ideal batch size is suggested to be set as the largest batch size supported by the available hardware [1], so as to maximize the training speed without considering the computational cost. In Eq 3.12, we show that the critical batch size with the optimal speed/computation trade-off can be analytically computed from the loss value. Under the guidance of this formula, we would be able to estimate the preferred batch size under any loss trajectory. Furthermore, this optimal batch size in Eq 3.12 is determined by equally minimizing the training time and required computation, as shown in Eq 3.9. In practice, if we would like to prioritize one over the other, we can follow the same process to derive the optimal batch size. By this means, we are able to obtain the optimal batch size based on our customized need in a systematic way.