We conduct experiments on \( \text{CIFAR-10}\)  [1]}, which is a standard image dataset that consists of \(50,000/10,000\)  training/test images. We vary each member of \(( \mathcal {M}, I)\)  as follows. Five Datasets. We create 5 families of new datasets \(\tau D\)  by rotating all input images in \( by \)\(, a fixed random element of one of the five groups: identity operator \)I3d\(, pixel-wise shared-rotations, \) O(3)Id\(, pixel-wise (unshared-)rotations \) O(3)d\(, permutations \) P(3d)\( and global rotations \) O(3d)\(. {\bf Six Models.} In addition to \) FCN\(, \) LCN\(, \) VEC\( and \) GAP\(, we add \) LAP4/8\( which are the same as \) GAP\( except the readout layer is replaced by the {\bf L}ocal {\bf A}verage {\bf P}ooling with window size \) 44/88\(. All networks have 8 hidden layers. {\bf Three Inference Algorithms.} (1) NTK regression\footnote {When investigating the impact of inference algorithms, it would be preferable to compare finite-width Bayesian inference to SGD. Unfortunately, Bayesian inference is too expensive to perform exactly and approximations may induce unwanted biases. As such, we instead use infinite-width Bayesian inference (i.e., NNGP regression~\cite {hron2020exact}), whose performance is usually very similar to NTK regression.} (aka infinite-width networks), (2) NN, our baseline for finite-width networks which is trained with momentum using a {\it small learning rate} without \) L2\( regularization; (3) NN+:= NN\) +LR+L2\(, i.e., using a larger learning rate (\) +LR\() and adding \) L2\( regularization (\) +L2\(). We plot the test accuracy for each \) ( M, I)\( (a total of \) 90=365\() in Fig.~\ref {fig:rotation_vs_accuracy}. The accuracy for finite-width networks is averaged over five runs\footnote {A few ( \mathcal {M}, I) have only 1 or 2 successful runs.} for each \) ( M, I)\( , and in each run the rotation \)\( is resampled. Note that the total variance across runs is small, indicating that the particular choice of \)\( has a negligible effect on the results.\)
