AutoDisc: Automatic Distillation Schedule for Large Language Model Compression
Driven by the teacher-student paradigm, knowledge distillation is one of the
de facto ways for language model compression. Recent studies have uncovered
that conventional distillation is less effective when facing a large capacity
gap between the teacher and the student, and introduced teacher assistant-based
distillation to bridge the gap. As a connection, the scale and the performance
of the teacher assistant is crucial for transferring the knowledge from the
teacher to the student. However, existing teacher assistant-based methods
manually select the scale of the teacher assistant, which fails to identify the
teacher assistant with the optimal scale-performance tradeoff. To this end, we
propose an Automatic Distillation Schedule (AutoDisc) for large language model
compression. In particular, AutoDisc first specifies a set of teacher assistant
candidates at different scales with gridding and pruning, and then optimizes
all candidates in an once-for-all optimization with two approximations. The
best teacher assistant scale is automatically selected according to the
scale-performance tradeoff. AutoDisc is evaluated with an extensive set of
experiments on a language understanding benchmark GLUE. Experimental results
demonstrate the improved performance and applicability of our AutoDisc. We
further apply AutoDisc on a language model with over one billion parameters and
show the scalability of AutoDisc.