[1]} is also trained with InfoNCE but with different methods. The inputs are cropped into global and local views, where the global views have higher resolution. The teacher model can only see the global views while the student model can utilize all the views. During the updating stage, the student model is updated by an optimizer, while the teacher model updates the parameters using EMA with the student model. To avoid collapsing, DINO adds centers, which can be viewed as a bias term, to each teacher output for helping the model generate the uniform distribution. The centers are updated smoothly by the average of the teacher outputs.
