We can view the recent crop of SSL methods as iterative self-distillation where there is a teacher and a student. Both teacher and student improve simultaneously while the teacher is evolving more slowly (running average) compared to the student: (1) In the case of contrastive methods e.g. MoCo [1]}, we classify images to positive and negative pairs in the binary form. (2) In the case of clustering methods (DC-v2, SwAV [2]}, SeLA [3]}), the student predicts the quantized representations from the teacher. (3) In the case of BYOL [4]}, the student simply regresses the teacher's embeddings vector. Here, we introduce a novel method using similarity based distillation to transfer the knowledge from the teacher to the student. We argue that our method is more regularized compared to prior work and improves the quality of the features in transfer learning.
