Similarity based knowledge distillation: While the above methods [1]}, [2]}, [3]}, [4]}, [5]} only extract the information about a single data point from the teacher, similarity based distillation methods [6]}, [7]}, [8]}, [9]}, [10]}, [11]}, [12]} represent the knowledge from the teacher in terms of similarities between data points. Probably CompRess [12]} is the closest related work to our method that uses similarity based distillation to compresses a large self supervised model to a smaller one. Our method is different since we use the similarity based distillation to transfer the similarity knowledge from evolving teacher to the student iteratively to learn rich representations.
