The \(\text{O}(3d)\) -invariance of \({\mathsf {FCN}}_{n/\infty }\)  follows from the rotational invariance of the Gaussian measure, and has been observed in many prior works, including [1]}, [2]}. Rotating the input by \(\tau \in \text{O}(3d)\)  is equivalent to rotating the weight matrix \(\omega \)  of the first layer by \(\tau ^*\) , and since \(\tau ^*\omega =^{d}\omega \)  for \(\omega \sim \mathcal {N}(0, 1)^{3d}\) , the distribution of the output functions at step 0 (aka initialization) is invariant. This observation implies that the first gradient is also \(\text{O}(3d)\) -invariant, which further implies the \(\text{O}(3d)\) -invariance of the output function after the first gradient update. By induction, this invariance property holds throughout the course of gradient descent training, even with \(L^2\) -regularization as the \(L^2\) -norm is rotationally invariant. Such invariant property also holds for (finite-width) Bayesian posterior inference thanks to the Bayes theorem: \( P(\tau {\mathcal {X}}^*| \tau {\mathcal {X}}_T) = P( {\mathcal {X}}^*| {\mathcal {X}}_T) \)  because \( P(\tau {\mathcal {X}}^*, \tau {\mathcal {X}}_T)/P(\tau {\mathcal {X}}_T) = P( {\mathcal {X}}^*, {\mathcal {X}}_T)/P({\mathcal {X}}_T)\) .
