Masked Siamese ConvNets
Self-supervised learning has shown superior performances over supervised
methods on various vision benchmarks. The siamese network, which encourages
embeddings to be invariant to distortions, is one of the most successful
self-supervised visual representation learning approaches. Among all the
augmentation methods, masking is the most general and straightforward method
that has the potential to be applied to all kinds of input and requires the
least amount of domain knowledge. However, masked siamese networks require
particular inductive bias and practically only work well with Vision
Transformers. This work empirically studies the problems behind masked siamese
networks with ConvNets. We propose several empirical designs to overcome these
problems gradually. Our method performs competitively on low-shot image
classification and outperforms previous methods on object detection benchmarks.
We discuss several remaining issues and hope this work can provide useful data
points for future general-purpose self-supervised learning.