The results of the paper are presented in the most vanilla setting. Our methods can easily extend to more complicated architectures like ResNet[1]}, MLP-Mixer[2]}, etc. The symmetry groups of such systems need to be computed in a case-by-case manner by identifying the invariant group of the random initialization and training procedures.
For example, the orthogonal group type of symmetries needed to be replaced by the permutation-type of symmetries if non-Gaussian i.i.d. initialization or/and \(L^p\)  (\(p\ne 2\) ) regularization. However, we empirically observe that swapping the Gaussian initialization by the uniform initialization in the first layer does not essentially change the performance of the network; see Sec. REF . This observation indicates that the permutation group may exhibit a similar degree of spuriousness as the rotation group; however, more rigorous and thorough experiments are needed to confidently confirm this claim, which is left for future work.
Moreover, the invariance property studied here is mainly coming from the first layer and it is possible that later layers could contribute new invariances to the system. For example, owing to the non-overlapping between patches in ViT [3]}, there could be permutation symmetries between the patches in the subsequent self-attention layer (assuming no positional encoding).
Finally, for the sake of simplicity, we use NTK-parameterization [4]} but our results apply to other network parameterizations, including standard- [5]}, meanfield- [6]}, \(\mu \) -parameterizations [7]}.
In particular, both finite- and infinite-width FCNs still suffer from the most spurious symmetries \(\text{O}(3d)\)  for all such parameterization schemes, which may explain the poor performance of FCN in the “feature learning" regime (e.g., \(61.5\%\)  accuracy on CIFAR-10, Table 1. in [8]}.)
