Prior work has focused on improving inference throughput of ViT models via patch sparsification. However, these approaches lead to an increased training time compared to the baseline models for multiple reasons. First, these approaches apply their token sparsification approach as a fine-tuning step after the initial model (e.g., DeiT-S) has already been trained on ImageNet using 100% of patches for 300 epochs. Therefore, any additional training to achieve better results using fewer patches can be considered training overhead compared to the baseline approach. Second, these approaches often use a teacher model (i.e., distillation) to provide more accurate soft labels (or use an additional distillation token as in DeiT) to aid in learning the token sparsification approach. These teacher models are usually the original pretrained model that these approaches are augmenting to do token sparsification. Therefore, the use of a teacher model dramatically increases the training iteration time during fine-tuning stage. Third, the techniques to learn which patches to drop increase training time, as additionally parameters/layers related to determining which patches to keep are learned during the fine-tuning stage. This requires dropped patches to be kept around for updating the weights during backpropagation. For instance, DynamicViT uses the Gumbel-softmax technique [1]} to learn a hard threshold function during training.
