As noted in earlier efforts<cite> [Howard and Ruder, 2018</cite> ] towards finetuning pretrained models for NLP tasks, aggressive finetuning can cause catastrophic forgetting, thus causing the model to simply fit over the target task and forget any capabilities gained during the pretraining stage. On the other hand, too cautious finetuning can cause slow convergence and overfitting.