A key hyper-parameter in the Cosine LRS is the step \(T\) at which Cosine decreases to the minimum for the first time. Typically, \(T\) is set to the total training step \(S\) for training with a predefined training step. Generally, it is believed that the learning rate should be high to enable sufficient exploration. For example, Kaplan et al. (2020) demonstrate that the loss decreases when the summed learning rate over the entire training increases (see Figure 23 in their paper). This indicates setting \(T<S\) is not optimal. On the other hand, Hoffmann et al. (2022) make a key observation that setting \(T>S\) results in dropped performance while setting \(S=T\) results in improved training efficiency, confirming that the learning rate shouldn't be kept high throughout the training. To reproduce these observations, we conduct experiments on the 0.036B model. We try \(Cosine(T)\) and \(CosineLoop(T)\) LRS, following the formula shown in Appendix B.1. The result can be seen in Figure 4. We can see that when the training step is \(S=20N,40N,60N,80N\), the lowest loss is always achieved by the \(Cosine(T)\) where \(T=S\). Both \(T<S\) and \(T>S\) are not optimal.