\(\sigma \) -overfitting and adversarial training.
As we can see from Fig. REF  and Fig. REF , adversarial training does not suffer from the \(\sigma \) -overfitting problem which may explain why applying 50% adversarial training (as in Sec. ) does not lead to better results compared to 100% adversarial training.
We believe that the difference between Gaussian augmentation and adversarial training can occur due to a much larger norm of the perturbation used for data augmentation.
On CIFAR-10-C, the optimal \(\operatorname{\mathbf {\ell }}_2\)  radius for adversarial training is \(0.1\)  while the norm of Gaussian vectors is approximately \(\sigma _{100\%} \cdot \sqrt{d} = 0.05 \cdot \sqrt{32 \cdot 32 \cdot 3} \approx 2.77\)  for 100% Gaussian augmentation and \(\sigma _{50\%} \cdot \sqrt{d} = 0.1 \cdot \sqrt{32 \cdot 32 \cdot 3} \approx 5.54\)  for 50% Gaussian augmentation.
It is therefore likely that due to a much larger norm of the perturbation, the model trained with 100% Gaussian noise fails to generalize to perturbations of smaller norms, while this failure can be alleviated to some extent by using the 50% scheme.
On a related note, some variations of adversarial training are known to suffer from a phenomenon called catastrophic overfitting [1]}, [2]}. Catastrophic overfitting shares some similarities with \(\sigma \) -overfitting, in the sense that the model overfits to a particular perturbation type used during training. Clarifying the relation of catastrophic overfitting and \(\sigma \) -overfitting is an interesting direction for future work.
