However, without principled guidance, the underlying mechanism of transfer learning is not very well understood. First, when we are pre-training deep models on the upstream dataset, we lack good metrics for measuring the quality of the learned model or representation in terms of transferability. In the past, people tended to rely empirically on controversial metrics for predicting the transferred test performance, such as the validation accuracy on the pre-trained data (e.g., validation accuracy on ImageNet [1]}). For example, some popular approaches (e.g., label smoothing [2]} and dropout [3]}) for boosting ImageNet validation accuracy turn out to hurt transfer performance on downstream tasks [4]}. Additionally, when pre-training deep models, many heuristic methods improving transferability, such as the design of loss functions, data augmentations, increased model size, and projection head layers [5]}, [6]}, are designed largely based upon trial-and-error without much insight of the underlying mechanism. Second, given the pre-trained models, how to efficiently fine-tune the model on downstream tasks remains an open question. Although fully fine-tuning all the parameters of the pre-trained model achieves the best performance, it becomes increasingly expensive as the model size grows (e.g., GPT-3 and transformer [7]}, [8]}, [9]}, [10]}). All these challenges call for a deeper understanding of what makes pre-trained deep models more transferable.
<FIGURE>