Intra-Instance VICReg: Bag of Self-Supervised Image Patch Embedding
Recently, self-supervised learning (SSL) has achieved tremendous empirical
advancements in learning image representation. However, our understanding and
knowledge of the representation are still limited. This work shows that the
success of the SOTA siamese-network-based SSL approaches is primarily based on
learning a representation of image patches. Particularly, we show that when we
learn a representation only for fixed-scale image patches and aggregate
different patch representations linearly for an image (instance), it can
achieve on par or even better results than the baseline methods on several
benchmarks. Further, we show that the patch representation aggregation can also
improve various SOTA baseline methods by a large margin. We also establish a
formal connection between the SSL objective and the image patches co-occurrence
statistics modeling, which supplements the prevailing invariance perspective.
By visualizing the nearest neighbors of different image patches in the
embedding space and projection space, we show that while the projection has
more invariance, the embedding space tends to preserve more equivariance and
locality. Finally, we propose a hypothesis for the future direction based on
the discovery of this work.