Evolution of teacher and student models:
In Figure REF , for every 10 epoch of ResNet-18, we evaluate both teacher and student models for BYOL, MoCo, and ISD methods using nearest neighbor. For all methods, the teacher performs usually better than the student in the initial epochs when the learning rate is small and then is very close to the student when learning rate shrinks. This is interesting as we have not seen previous papers comparing the teacher with the student. This might happen since the teacher is a running average of the student so can be seen as an ensemble over many student networks similar to [1]}. We believe this deserves more investigation as future work.
<FIGURE>