Where are my Neighbors? Exploiting Patches Relations in Self-Supervised Vision Transformer
Vision Transformers (ViTs) enabled the use of transformer architecture on
vision tasks showing impressive performances when trained on big datasets.
However, on relatively small datasets, ViTs are less accurate given their lack
of inductive bias. To this end, we propose a simple but still effective
self-supervised learning (SSL) strategy to train ViTs, that without any
external annotation, can significantly improve the results. Specifically, we
define a set of SSL tasks based on relations of image patches that the model
has to solve before or jointly during the downstream training. Differently from
ViT, our RelViT model optimizes all the output tokens of the transformer
encoder that are related to the image patches, thus exploiting more training
signal at each training step. We investigated our proposed methods on several
image benchmarks finding that RelViT improves the SSL state-of-the-art methods
by a large margin, especially on small datasets.