With this purview, similar to<cite> [Howard and Ruder, 2018]</cite> , we propose optimizing different layers in our model to different extents, and keep lower step sizes for the deeper pretrained layers while finetuning on a downstream task. We thus split the parameters as {θ 1 , ..., θ l } , where θ i corresponds to the parameters of layer i, and optimize them with separate learning rates {η 1 , ...., η l } . Also, when finetuning a pretrained layer for a downstream task, we keep η i < η j ; ∀i < j. Thus, while finetuning the POS + Lang Id pretrained model for Language Modeling, we propose to keep the learning rates for Embedding Layer and LSTM Layer 1 lower than the LSTM Layer 2 weights. Similarly, when finetuning the Language Model for Sentiment Analysis, we keep the learning rates of the deeper layers lower than that of the shallower ones.