We subsample \(m_i = 2^{-i/2}m_{\text{ImageNet}}\)  images as our training set, where \(i\in [12]\) .
In Fig. REF , we plot the learning curves for four \(( \mathcal {M}, I)\)  triplets. We set \(\mathcal {M}\)  to be ResNet101 or a (small MLP-)Mixer [1]}, \( to be the original (unrotated) ImageNet \) IN\( or a rotated version \) IN\(, for some \) O(3)d\(. We keep \) I\( (SGD, see Sec.~\ref {sec:ImageNet experiment}) fixed. For each configuration, we interpolate the first/last six points (i.e., \) i6\(/\) i<6\() with straight lines (in the log-log plot) and compute the slope \)\( (see legends in Fig.~\ref {fig:imagenet scaling}). We treat ResNet101 trained on clean images as our baseline, which is the most efficient and consistent \) ( M, I)\( system among the four. We observe the following.(1) Almost perfect power-law scaling for the baseline ({\bf \color {plot_blue}Blue Dashed Line}). (2) A cusp around \) mi=6\( for the remaining learning curves, which have two phases: relatively flat in the first phase and steepened in the second one. (3) Surprisingly, the slopes (\) =0.49, 0.38\() of \) (ResNet101, IN, I)\( and \) (Mixer, IN, I)\( in the second phase essentially catch up with that (\) =0.41\() of the best one \) (ResNet101, IN, I)\(.These observations suggest that with more data the system can overcome the spurious symmetries \) O(3)d\(.\) Finally, to test the limit of deep learning systems in overcoming spurious \(\text{O}(3)^d\)  symmetries, we further scale up \(\mathcal {M}\)  in the ResNet family [2]} and in the EfficientNet family [3]}. In the right panel of Fig. REF , we make a scatter plot showing the accuracy of the \(\text{O}(3)^d\) -rotated vs the original dataset. Each data point corresponds to one model.
For the ResNet family, the top-1 accuracy gap between the rotated and the unrotated dataset drops from about 10% (ResNet-18) to about 6% (ResNet-200) and for the EfficientNet family, this gap drops from about 4% (EfficientNet B0) to only about 1.% (EfficientNet B7 Trained by about 180 epochs.), which is quite remarkable.
