One key assumption is made on the cosine cycle length and the corresponding learning rate drop (we use a \(10\times\) learning rate decay in line with Rae et al. (2022)). We find that setting the cosine cycle length too much longer than the target number of training steps results in sub-optimally trained models, as shown in Figure A1.