Among non-convolutional architectures, self-attention networks naturally capture long-range dependencies.
Initially designed for natural language processing [1]}, transformers were recently shown to exhibit good performance on vision tasks [2]}.
Similar to self-attention networks, CycleNet breaks translational symmetry and has the best performance on ImageNet when standard convolutions are included in the first layers [3]}.
<FIGURE>