Hub-Pathway: Transfer Learning from A Hub of Pre-trained Models
Transfer learning aims to leverage knowledge from pre-trained models to
benefit the target task. Prior transfer learning work mainly transfers from a
single model. However, with the emergence of deep models pre-trained from
different resources, model hubs consisting of diverse models with various
architectures, pre-trained datasets and learning paradigms are available.
Directly applying single-model transfer learning methods to each model wastes
the abundant knowledge of the model hub and suffers from high computational
cost. In this paper, we propose a Hub-Pathway framework to enable knowledge
transfer from a model hub. The framework generates data-dependent pathway
weights, based on which we assign the pathway routes at the input level to
decide which pre-trained models are activated and passed through, and then set
the pathway aggregation at the output level to aggregate the knowledge from
different models to make predictions. The proposed framework can be trained
end-to-end with the target task-specific loss, where it learns to explore
better pathway configurations and exploit the knowledge in pre-trained models
for each target datum. We utilize a noisy pathway generator and design an
exploration loss to further explore different pathways throughout the model
hub. To fully exploit the knowledge in pre-trained models, each model is
further trained by specific data that activate it, which ensures its
performance and enhances knowledge transfer. Experiment results on computer
vision and reinforcement learning tasks demonstrate that the proposed
Hub-Pathway framework achieves the state-of-the-art performance for model hub
transfer learning.