Recently, patch sparsification techniques, which select a subset of patches per image, have achieved competitive inference accuracy (i.e., with 0.5% of the baseline ViT) using only 50% to 70% of the available image patches [1]}, [2]}. Using fewer patches per image leads to an substantial increase in images processed per second (throughput). Intuitively, this improvement can be attributed to the notion that not all patches are necessary (or even useful) for classifying a given image. However, while successfully speeding up inference, prior patch sampling approaches do not reduce the baseline ViT training time. Instead, these approaches often require additional training time compared to the baseline. This is due to (1) additional learned patch sampling modules that require access to all patches during training (even discarded ones) to learn how to sample, (2) uneven patch allocation across samples in a batch leading to parallelization inefficiencies, (3) the use of a teacher model (with access to all patches) to provide soft labels for the dynamic patch sampling student model, and/or (4) initializing a fine-tuning stage from a pretrained model that used 100% of patches for its entire training.
