One of the key considerations in dynamically adjusting the number of patches across iterations is determining which patches provide the most useful information to the ViT model during both training and inference. If each patch provided the same amount of information for a given objective (e.g., classification), then random sampling would be a reasonable approach. However, as has been demonstrated by various attention-based models [1]}, [2]}, the usefulness of image regions varies widely based on their content.
