Knowledge distillation was originally proposed by [1]}, and used in [2]} to transfer the knowledge from one or more teacher networks to a single student network. [3]}, [4]}, [5]} show that self-distillation, where the teacher and student share the same architecture, improves supervised learning by reducing the generalization gap. We use this method to improve our simple baseline method for SSDA and UDA.
