Knowledge Distillation via Weighted Ensemble of Teaching Assistants
Knowledge distillation in machine learning is the process of transferring
knowledge from a large model called the teacher to a smaller model called the
student. Knowledge distillation is one of the techniques to compress the large
network (teacher) to a smaller network (student) that can be deployed in small
devices such as mobile phones. When the network size gap between the teacher
and student increases, the performance of the student network decreases. To
solve this problem, an intermediate model is employed between the teacher model
and the student model known as the teaching assistant model, which in turn
bridges the gap between the teacher and the student. In this research, we have
shown that using multiple teaching assistant models, the student model (the
smaller model) can be further improved. We combined these multiple teaching
assistant models using weighted ensemble learning where we have used a
differential evaluation optimization algorithm to generate the weight values.