Our ISD method performs similarity-based distillation iteratively where the student is optimized by distilling the teacher and the teacher is updated as a moving average of the student. One can initialize the teacher from an already trained model and set the momentum to 1, which will ignore updating the teacher. Such a method can distill the teacher into a student model that is initialized from scratch. Note that this method is closely related to CompRess [1]} except that we are using it for self-distillation instead of compressing from a deep model to a shallow one.
