We chose to train a ViT-Small [1]} as the amount of parameters is roughly equivalent to a ResNet-50's (21M vs. 23M).
Further, we use a student-teacher setup and the teacher weights are updated by the exponential moving average of the student weights following [2]}, [3]}.
The exponential moving average for updating the teacher weights is adapted with a cosine schedule starting at 0.9995 and going up to 1 i.e. a hard copy.
We train the ViT-Small with a cosine learning rate schedule going down to 0 over 50 training epochs.
The initial projection head learning rate is \(1\mathrm {e}{-4}\)  and the backbone's learning rate is \(1\mathrm {e}{-5}\) .
The projection head consists out of three linear layers with hidden dimensionality of 2048 and Gaussian error linear units as activation function [4]}.
The output dimensionality is 256 and the resulting tensors are then passed through a l2-bottleneck and the prototype matrix \(C\)  to produce cluster assignment predictions.
As discussed, we use a queue for Sinkhorn-Knopp clustering with a length of 8192.
We set the temperature to 0.1 and use Adam as an optimizer with a cosine weight decay schedule.
The alignment happens to a fixed output size of 7x7 during training.
This makes sure that the local and global crop feature maps have the same spatial resolution.
The augmentations used are: random color-jitter, Gaussian blur, grayscale and multi-crop augmentations and the global crop's resolution is 224x224 and the local crop's resolution is 96x96, as in [5]}.
We generate global and local crops with the constraint that they intersect at least by \(1\%\)  of the original image size to make sure that there is a non-negligible intersection where we can apply our clustering loss to.
In Figure REF  we show the generation process for two ImageNet pictures.
