To analyze the importance of different components in Distiller, we adopted fANOVA [1]}, an algorithm for quantifying the importance of individual hyper-parameters as well as their interactions in determining down-stream performance. We use fANOVA to evaluate the importance of the four components in Distiller as well as their pairwise combinations: data augmentation, intermediate distillation objective, layer mapping strategy, and prediction layer distillation objective.
<TABLE>