H AT: Hybrid Attention Transfer for Knowledge Distillation
Knowledge distillation is a widely applicable technique for supervising the training of a light weight student neural network by capturing and transferring the knowledge of a high capacity trained teacher neural network. However, the distillation performance is very sensitive to how the distilled knowledge and the distillation loss are defined. In this paper, we propose a novel channel wise attention transfer loss with a corresponding channel selection method. Further, we develop a hybrid attention transfer loss. Extensive experiments on four public datasets demonstrate that the proposed channel wise transfer method shows a complementary effect with other attention based methods, and the hybrid attention transfer achieves state of the art performance on knowledge distillation tasks. Code for our experiments is available at https:/github.com/somone23412/H AT.
PRCV