[1]} (2019) report some marginal improvement for the task of text classification by fixing weights of Transformer models derived by BERT, when training the classification layer. [2]} (2019) carried out additional pre-training of BERT-derived models on the target dataset. They also used different learning rates for different model layers.
<FIGURE>