In this paper, we explore the use of a Vision Transformer (ViT) with our new loss and excavate its unique aptness for self-supervised segmentation.
While vision transformers have shown great potential unifying architectures and scaling well with data into billions, they have mostly been shown to work for image-level tasks [1]}, [2]}, [3]} or dense tasks [4]}, [5]}, [6]}, [7]} but in a supervised manner.
Our work aims to close this gap by self-supervisedly learning dense ViT models.
We combine the recently discovered property of self-supervised ViTs to localise objects [3]} with our dense loss to train spatial tokens for unsupervised segmentation.
