Existing models, however, have two limitations when edge applications are considered: (1) their large model sizes, and (2) their robustness to environmental conditions not seen during training. For example, pre-trained models can range from 95 million parameters to 2 billion [1]}, which makes training and deployment on edge devices difficult. To overcome this issue, different model compression schemes have been explored. DistilHuBERT, for example, applied knowledge distillation on the HuBERT model and was able to reduce the number of parameters to a fraction of the original [2]}. LightHuBERT, in turn, proposed a 2-stage distillation approach to also reduce the size of the original HuBERT [3]}.
