To our knowledge, this is the first approach which enables this type of dynamic inference by simply changing the number of patches kept in a patch sampling block during inference. As we discuss in the next section, prior work on patch sparsification, DynamicVit [1]} and A-ViT [2]}, aim to find a fixed sparsity setting (e.g., \(\rho =0.7\) ) that achieves a minimal reduction in accuracy while leading to faster inference. By training with a PSS, we can achieve a dynamic inference range, leading to a natural trade-off between inference accuracy and runtime. These types of dynamic approaches may be useful in energy-constrained settings (e.g., mobile devices) where this type of trade-off can be made based on current battery conditions [3]}.
