In contrast, A-Vit [1]} uses an Adaptive Computation Time (ACT) [2]} approach to dynamically sparsify the number of patches (tokens) at various stages in a ViT, leading to early exiting of tokens based on sample difficulty. While this approach may lead to better performance, as the model can adapt the token (patch) budget on a per-sample basis, the variability leads to inefficiencies in a batched setting since the input tensor now has a different number of tokens per sample. This makes ACT-based approaches, at least as currently formulated, less suitable for reducing ViT training time.
