Spatial Entropy Regularization for Vision Transformers
Recent work has shown that the attention maps of Vision Transformers (VTs),
when trained with self-supervision, can contain a semantic segmentation
structure which does not spontaneously emerge when training is supervised. In
this paper, we explicitly encourage the emergence of this spatial clustering as
a form of training regularization, this way including a self-supervised pretext
task into the standard supervised learning. In more detail, we propose a VT
regularization method based on a spatial formulation of the information
entropy. By minimizing the proposed spatial entropy, we explicitly ask the VT
to produce spatially ordered attention maps, this way including an object-based
prior during training. Using extensive experiments, we show that the proposed
regularization approach is beneficial with different training scenarios,
datasets, downstream tasks and VT architectures. The code will be available
upon acceptance.