Distiller is a generic meta-framework that encompasses various KD pipelines used in previous work. For example, Distiller with the following configurations corresponds to the KD pipeline used in each of the cited works:
\(l^{\text{pred}} =\)  CE, \(l^{\text{inter}}=\)  MSE, \(m_{i,j} = \)  Skip, \(a = \)  CA [1]};
\(l^{\text{pred}} =\)  CE, \(l^{\text{inter}}=\)  MSE, \(m_{i,j} = \)  EMD [2]};
\(l^{\text{pred}} =\)  CE, \(a = \)  Mixup [3]}.
