Due to the high computational complexity of ViTs, there has been increased research effort on dynamically adjusting the amount of computation per sample in order to trade-off accuracy for throughput. DynamicViT [1]} introduced patch sampling during inference by adding learned control gates trained via the Gumbel-softmax technique [2]}. These sampling modules are added at various points in the ViT and are used to select a subset of patches at each stage according to a fixed token ratio. Like our approach, DynamicViT ensures that each image have the same number of patches at a given stage in order to mitigate parallelization inefficiencies.
