The flavor of invariance studied this paper is algorithmic invariance, as it concerns a system or a learning algorithm, and is qualitatively different from the functional invariance studied elsewhere [1]}.
Recall that a function \(f\)  is (functionally) invariant to a group \({\mathcal {G}}\)  if \(f(\tau x) = f(x)\)  for all \(\tau \in {\mathcal {G}}\) . Natural images are often considered to be translationally invariant, which is a key motivation for the usage of convolutional networks in computer vision. These symmetries are hard-coded into the architecturesIn the idealized setting when circular padding is applied and the readout layer is a global average pooling. as a kind of inductive bias and the (post-pooling) representations are invariant to this group by design. As a consequence, the corresponding hypothesis class, defined by these networks, is more restricted, which could lead to better generalization performance if the symmetry were exact [2]}.
Algorithmic invariance is weaker than functional invariance.
For example, in the discussion above, the learning system defined by kernel regression with an inner product kernel is algorithmically invariant to the rotational group \({\mathcal {G}}\)  since \(K(\tau x, \tau \bar{x})= K(x, \bar{x})\)  for all \(\tau \in {\mathcal {G}}\) , but the learned function itself is not functionally invariant because \(K(\tau x, \bar{x})\ne K(x, \bar{x})\) .This stronger condition could be achieved by defining an invariant kernel \(K^{\text{inv}}(x, \bar{x})= \int _{\tau \in {\mathcal {G}}}K(x, \tau \bar{x})d\tau \) , or approximated by augmenting the training set by group actions as in [3]}, but we do not pursue this here
