What does infinite batch size mean?It is infeasible to train a model with an infinite batch size in practice, but we can employ a "trial and error" method to find a sufficiently large batch size that is equivalent to the infinite batch size, which we refer to as a _relatively infinite batch size_. This is based on the fact that a further increase of the sufficiently large batch size does not further reduce the number of training steps required to achieve a certain loss value, thus not altering the training loss curve. As a result, this relatively infinite batch size can be found by increasing the batch size until the training loss curve becomes stationary. We empirically found that for model sizes at the magnitude of 10M, a batch of 40M tokens is sufficiently large. In this way, \(S_{c}\) and \(\alpha_{S}\) can be estimated by the loss values and steps during the training of the 10M model at this _relatively infinite batch size_.