Knowledge Distillation: The task of transferring the knowledge from one model to the other is called knowledge distillation [1]}, [2]}. The knowledge from the teacher can be extracted and transferred in various ways. The knowledge in the activations of intermediate layers can be transferred through regression [3]}, [4]}, [5]}. While in most works the teacher is a deeper model and the student is a shallower model, in [6]}, [7]} both teacher and the student use the same architecture. Techniques from knowledge distillation can also be used in an unsupervised way to improve self-supervised learning [8]}, [9]}, [10]}. Instead of using knowledge distillation for model compression or reducing the generalization gap, we use it iteratively to evolve the teacher and student together to learn rich representations from scratch.
