MetaLR: Layer-wise Learning Rate based on Meta-Learning for Adaptively Fine-tuning Medical Pre-trained Models
When applying transfer learning for medical image analysis, downstream tasks
often have significant gaps with the pre-training tasks. Previous methods
mainly focus on improving the transferabilities of the pre-trained models to
bridge the gaps. In fact, model fine-tuning can also play a very important role
in tackling this problem. A conventional fine-tuning method is updating all
deep neural networks (DNNs) layers by a single learning rate (LR), which
ignores the unique transferabilities of different layers. In this work, we
explore the behaviors of different layers in the fine-tuning stage. More
precisely, we first hypothesize that lower-level layers are more
domain-specific while higher-level layers are more task-specific, which is
verified by a simple bi-directional fine-tuning scheme. It is harder for the
pre-trained specific layers to transfer to new tasks than general layers. On
this basis, to make different layers better co-adapt to the downstream tasks
according to their transferabilities, a meta-learning-based LR learner, namely
MetaLR, is proposed to assign LRs for each layer automatically. Extensive
experiments on various medical applications (i.e., POCUS, BUSI, Chest X-ray,
and LiTS) well confirm our hypothesis and show the superior performance of the
proposed methods to previous state-of-the-art fine-tuning methods.