1. **Linear warm-up phase** - As before, the learning rate is initially increased to some maximum value \(\eta_{max}\) over \(T_{warmup}\) timesteps, or equivalently until timestep \(t_{cd}=T_{warmup}\). The learning rate undergoes a warm-up only once (during the first task) and does not require re-warming for future tasks. 2. **Cooldown phase** - During this stage the learning rate undergoes a cooldown phase where the learning rate is gradually decayed to constant value \(\eta_{const}\) according to some decay function \(f_{cd}\) over \(T_{cd}\) timesteps from timestep \(t_{cd}\) to \(t_{const}=t_{cd}+T_{cd}\). This stage also occurs only once during the first task. 3. **Constant phase** - The learning rate then remains constant for all future tasks over \(T_{const}\) timesteps from timestep \(t_{const}\) to \(t_{ann}=t_{const}+T_{const}\). The checkpoint obtained at the end of this phase is the one one should resume from when continuing to pretrain on a new dataset. 4. **Annealing phase** - The learning rate is annealed to a small value \(\eta_{min}\) over \(T_{ann}\) timesteps from timestep \(t_{ann}\) to \(t_{end}=t_{ann}+T_{ann}\), helping train the model to convergence before being deployed.