Understanding and exploiting the structural information in natural data are central aspects of designing machine learning systems. [1]}, [2]}, [3]} study the low-dimensional structure of natural data while [4]}, [5]} investigate the role of deformation stability of natural data. Designing networks that maximally respect the symmetries of natural data (e.g., translational invariance/equivalence of images [6]}, [7]}) is widely considered a principled approach in practice. Several works also demonstrate the possibility of learning such symmetries from scratch using natural or synthetic data [8]}, [9]}. Nevertheless, recent breakthroughs in applying attention-based models [10]}, [11]}, [12]} to computer vision have fundamentally challenged the significance that symmetries play in model design. Attention-based models have weaker inductive biases (lacking even translation equivariance) than those of convolutional networks, yet their performance is comparable in the large data regime [13]}. Our analysis of DIDE (Fig. REF ) sheds some light on why such models are able to reach good performance with the help of more data.
