**Loss Decreases Dramatically in Decay Stage.** We try WSD LRS on 0.036B models. As shown in Figure 5, in the decay stage, as the learning rate begins to decrease, the loss experiences a significant rapid decline and quickly decreases to be equal to or lower than the Cosine LRS at step \(T=S\). At the same time, we can reuse the model before decay and continue training with the previous high learning rate. After more steps of training \(S^{\prime}\), we can also perform annealing to achieve the same loss as the Cosine LRS at \(Cosine(S^{\prime})\). This verifies our assumption that the training stage can be explicitly split into the stable training and decay stage.