We are interested in learning rich representations from unlabeled data. We have a teacher network and a student network. We initialize both models from scratch and update the teacher to be a slower version of the student: we use the momentum idea from MoCo in updating the teacher so that it is running average of the student. The method is described in Figure REF . Following the notation in [1]}, at each iteration, we pick a random query image and a bunch of random other images that we call anchor points. We augment those images and feed them to the teacher model to get their embeddings. Then, we augment the query again independent of the earlier augmentation and feed it to the student model only. We calculate the similarity of the query point compared to the anchor points in the teacher's embedding space and then optimize the student to mimic the same similarity for the anchor point at the student's embedding space. Finally, we update the teacher with a momentum to be the running average of the student similar to MoCo and BYOL. Note that our method is closely related to ComPress method [1]} which uses the similarity distillation for compressing a frozen larger model to a smaller one.
