The Modality Focusing Hypothesis: On the Blink of Multimodal Knowledge Distillation
Multimodal knowledge distillation (KD) extends traditional knowledge
distillation to the area of multimodal learning. One common practice is to
adopt a well-performed multimodal network as the teacher in the hope that it
can transfer its full knowledge to a unimodal student for performance
improvement. In this paper, we investigate the efficacy of multimodal KD. We
begin by providing two failure cases of it and demonstrate that KD is not a
universal cure in multimodal knowledge transfer. We present the modality Venn
diagram to understand modality relationships and the modality focusing
hypothesis revealing the decisive factor in the efficacy of multimodal KD.
Experimental results on 6 multimodal datasets help justify our hypothesis,
diagnose failure cases, and point directions to improve distillation
performance.