With respect to the large deviation from Chinchilla Optimal \(\frac{N_{opt}}{D_{opt}}\), we notice that their scaling experiment was conducted in a not very recent configuration. To compare with more recent configuration such as Llama2 (Touvron et al., 2023), we extract the training loss data from Llama2 paper (left part) in Appendix Figure 18 and estimate the compute optimal \(\frac{D_{opt}}{N_{opt}}\) in their paper using the right part of Figure 18. Since they use Cosine LRS, the loss is not optimal in the middle of the training, depicted by the concave curve during training in the right figure of Figure 18. We fill the concave part with a straight line to estimate the optimal loss envelope if they had used the WSD LRS. After that, the compute model size should roughly be the regime that a model's loss curve is about to intersect with a larger model's loss curve. With this intuition, the 13B model is about to intersect with 34B model at \(10^{5}\) EFlops (\(10^{1}\)s Flops), and the 34B model is about to intersect with 70B model at \(5\times 10^{5}\) EFlops. Therefore, we estimate the \(\frac{D_{opt}}{N_{opt}}\) to be roughly \(\frac{5\times 10^{5}}{6\times 34^{2}}\sim\frac{10^{5}}{6\times 13^{2}}\), which is \(70\sim 100\). Therefore, under this approximate comparison, their data-model ratio is closer to ours. And our configuration can absorb more data into a smaller model compared to previous ones. However, we note that the above estimate are only a rough one.