Guillotine Regularization: Improving Deep Networks Generalization by Removing their Head
One unexpected technique that emerged in recent years consists in training a
Deep Network (DN) with a Self-Supervised Learning (SSL) method, and using this
network on downstream tasks but with its last few layers entirely removed. This
usually skimmed-over trick is actually critical for SSL methods to display
competitive performances. For example, on ImageNet classification, more than 30
points of percentage can be gained that way. This is a little vexing, as one
would hope that the network layer at which invariance is explicitly enforced by
the SSL criterion during training (the last layer) should be the one to use for
best generalization performance downstream. But it seems not to be, and this
study sheds some light on why. This trick, which we name Guillotine
Regularization (GR), is in fact a generically applicable form of regularization
that has also been used to improve generalization performance in transfer
learning scenarios. In this work, through theory and experiments, we formalize
GR and identify the underlying reasons behind its success in SSL methods. Our
study shows that the use of this trick is essential to SSL performance for two
main reasons: (i) improper data-augmentations to define the positive pairs used
during training, and/or (ii) suboptimal selection of the hyper-parameters of
the SSL loss.