DONNA can be trivially applied to Vision Transformers [1]}, without any conceptual change to the base algorithm. In this experiment, we use vit-base-patch16-224 from [2]} as a teacher model for which we define a related hierarchical search space. Vit-base-patch16-224 is split into 4 DONNA-blocks, each containing 3 ViT blocks (self-attention+MLP) as defined in the original paper [1]}. For every block, we vary the following parameters:
