Note that we are using a teacher and student model similar to MoCo, but it is for a different purpose: in MoCo the goal of having two models is to be able to maintain a large memory bank for the negatives, but in our case, the teacher acts as a role model for the student to follow. In fact, SimCLR [1]} shows that there is no need for two models in contrastive learning if we can handle the large memory bank with large mini-batches.
