Figure 3: **The effect of re-warming and re-decaying the learning rate on adaptation and forgetting. We consider two constant baselines and three models that re-warm and re-decay. One baseline continues training from \(\eta_{min}\) of pre-training (\(3\cdot 10^{-4}\)) while the other warms up to \(\eta_{max}\) from pre-training (\(3\cdot 10^{-4}\)). For the models that re-warm and re-decay we vary \(\eta_{max}\in\{1.5\cdot 10^{-4},3\cdot 10^{-4},6\cdot 10^{-4}\}\). All models except the \(\eta_{min}\) baseline use linear warmup for 1% training iteration. The non-baseline models cosine decay the learning to reach \(0.1\cdot\eta_{max}\) by the end of training. We observe that re-warming and re-decaying the learning rate is needed to best adapt to the new dataset. Small increases or decreases in \(\eta_{max}\) allow to trade-off between more or less adaptation.