Footnote 6: Here, \(N_{c}\)\(D_{c}\) and \(C_{c}\) are measured in the number of non-embedding parameters, the number of training tokens and the number of FP-days, respectively. According to the original paper [30], \(C_{c}\) and \(C\) should be denoted by \(C_{c}^{min}\) and \(C_{min}\), corresponding to the optimal use of compute. We use the simplified notations for ease of discussions.