The performance of student model is always worse than the teacher model, self-distillation is proposed to achieve comparable or even better performance than teacher model, which distills the knowledge between teacher and student models in an identical architecture [1]}, [2]}, [3]}, [4]}. We adopt self-distillation but differ from them in two key points. First, we propose a dynamic self-distillation process by introducing some weights to leverage the knowledge from teacher model and the information in raw data. Second, our distillation method is performed with disentanglement network, which not only improves the model performance, but also assists in eliminating interference factors.
