Improving the Generalization of Supervised Models
We consider the problem of training a deep neural network on a given
classification task, e.g., ImageNet-1K (IN1K), so that it excels at that task
as well as at other (future) transfer tasks. These two seemingly contradictory
properties impose a trade-off between improving the model's generalization
while maintaining its performance on the original task. Models trained with
self-supervised learning (SSL) tend to generalize better than their supervised
counterparts for transfer learning; yet, they still lag behind supervised
models on IN1K. In this paper, we propose a supervised learning setup that
leverages the best of both worlds. We enrich the common supervised training
framework using two key components of recent SSL models: multi-scale crops for
data augmentation and the use of an expendable projector head. We replace the
last layer of class weights with class prototypes computed on the fly using a
memory bank. We show that these three improvements lead to a more favorable
trade-off between the IN1K training task and 13 transfer tasks. Over all the
explored configurations, we single out two models: t-ReX that achieves a new
state of the art for transfer learning and outperforms top methods such as DINO
and PAWS on IN1K, and t-ReX* that matches the highly optimized RSB-A1 model on
IN1K while performing better on transfer tasks. Project page and pretrained
models: https://europe.naverlabs.com/t-rex