What does infinite training data mean?The above steps require training models with a fixed set of hyperparameters on infinite data until convergence. In practical scenarios, however, data is always finite in practice. Nevertheless, it is feasible to have _relatively_ "infinite training data" for a given model. Say we have an extremely small model with only \(N=10\) parameters. For this micro model, due to its very limited capacity, no matter how much training data we use to train it (_e.g._, 1B, 2B, or 1T tokens5), the model's performance cannot be improved, and the training dynamic (_i.e._, the test loss at any given training step) is almost the same. This is to say, beyond a _critical size_, this micro model is unaware of the growth of the data size. In this case, we can say that the training data is _relatively_ infinite for this model. In fact, for a given parameter with \(N\) parameters, this critical size could be computed (see Eq. 4.4 in [16]). Here, we give a simple empirical method to deduce whether a data size is (relatively) infinite for a \(N\)-parameter model. Note that overfitting should not happen when the training data is (relatively) infinite. As a result, we can train the \(N\)-parameter model and observe whether the training and test losses diverge. If the dynamics of the training and test losses are almost the same everywhere (_i.e._, no overfitting), we can safely infer that the training data is (relatively) infinite for the given \(N\)-parameter model.