Since [1]} finds initializing students with pretrained weights is better for distillation, we initialize student models with either weights obtained from task-agnostic distillation [2]} or pretrained from scratch [1]}. Three different pretrained models \(\text{BERT}_\text{BASE}\)  [4]}, \(\text{RoBERTa}_\text{LARGE}\)  [5]} and \(\text{ELECTRA}_\text{LARGE}\)  [6]} are considered as teacher models in our experiments after task-specific fine-tuning. As student models, we consider options like \(\text{TinyBERT}_4\) , \(\text{ELECTRA}_\text{SMALL}\) , as well as other models detailed in Table REF  in Appendix.
