[1]} is designed after studying different components in ViT with regards to the robustness, e.g., accuracy under adversarial attacks. Based on the results, RVT decides to 1) remove the class token, which is not important to a vision transformer, by averaging the features and 2) add CNN to the embedding and the FFN layers for increasing locality and 3) use more attention heads for obtaining different features. Moreover, the original self-attention is replaced by Position-Aware Attention Scaling (PAAS), which adds a learnable matrix to the self-attention for showing the importance of each \(Q\)-\(K\)  pair. PAAS is proved to suppress the unrelated signal in the noise input. Moreover, Patch-wise Augmentation applies different augmentations on different patches to diversify the training data.
