[1]} is created after the authors examine the attention maps of ViT and find out that the attention collapsing takes place in the deeper layer. This problem hinders the model to be representative and would lower the performance. The authors alter the self-attention layers and provide a learnable transformation matrix after the attention layer to address the issue by stimulating the model to generate a new set of attention maps.
