We next consider two independent augmentations, one for the student and one for the teacher, with the student's input undergoing masking as well. The self-supervision in such a scheme introduces invariance, such as spatial, scale, and color intensity, similar to prior contrastive models [1]}, [2]}. Crucially, we find that adding spatial and scale invariance by two crops of an image may hurt representation quality, with finetuning decreased by 0.4% and linear probing decreased by 2.0%. On the other hand, color invariance is shown to be beneficial, leading to a significant 0.8% gain from 82.5% to 83.3%.
Using cropping and color augmentations combined, the linear probing performance improves substantially to 73.3% while the finetuning performance drops 0.2% as spatial invariance may hurt generalization.
