Utilizing regularizations and architectural modules beyond the core primitive is critical for achieving strong results [1]}.
In this section, we study the effects of these additional components on self-attention models.
The components we study were all designed for use in convolutional models, as they were developed through experimentation (either human or automated search) on convolutional models.
We examine whether these components can successfully transfer to the new model family of self-attention networks.
