Figure 15: Illustrative comparison be- Figure 16: We use two different function tween Cosine LRS and WSD LRS. The forms to fit the data scaling law achieved WSD LRS with different end steps share by WSD LRS and choose power law as the same stable training stage. the best fit.