These approaches also require additional hyperparameters tied to loss functions that encourage token sparsification. While additional loss functions are common in many applications, they often require parameter search for the associated hyperparamters to tune the level of impact of the loss function (e.g., amount of token sparsification applied). These learnable components may also have some impact on regularization. For instance, A-Vit [1]} disables MixUp regularization [2]} during training as they note it is not compatible with their approach. We observe that our simpler patch selection approach (with no learnable component) performs best with all regularization enabled as is done by the baseline model. Likely, pixel mixing across the same patch location makes it significantly more challenging to learn a patch selection layer.
