However, the knowledge or the learned representations for the teacher and student can be distinct, as a result of different network architectures.
Recent works [1]}, [2]} also show that initialization has a strong relationship with the final representation of the trained network.
Therefore, with the different random initialization, even models with the same network structure could have distinct feature representations.
Although most of the networks can achieve consistent performances, when analyzing the activation of intermediate layers of teacher and student models to the same sample, we find the activation responses of the same channel between different networks could have large variations, which we name such phenomenon as teacher-student knowledge discrepancy.
Visualization of knowledge discrepancy is shown in Figure REF .
We gather the training data of one category and calculate the activation maps for students and teachers respectively.
It is obvious that the activation map between T and S has a large difference.
What's more, even with the same model, S1 and S2, different initialization also makes them show different activation responses.
T-S1, Transformed T-S1, S1-S2 shows the difference between activation maps.
