Gradual Unfreezing: Similar to<cite> [Howard and Ruder, 2018]</cite> , rather than updating all the layers together for finetuning, we explore gradual ordered unfreezing of layers. Thus, initially we freeze all the layers. Then starting from the last layer, we train the model for a certain number of epochs before unfreezing the layer below it. Thus for Sentiment Analysis finetuning, for the first epoch, only θ sentiment receives the gradient updates, after which we unfreeze the θ lstm2 , and subsequently unfreeze the lower layers in a similar manner.