Worst-case vs average-case behavior.
[1]} show that the robustness to Gaussian noise and adversarial perturbations are closely related. More precisely, they show using concentration of measure arguments that a non-zero error rate under Gaussian perturbation implies the existence of small adversarial perturbations and consequently that improving adversarial robustness leads to an improvement in robustness against Gaussian perturbations. This finding is consistent with what we observe here.
What remains to be understood is why adversarial training performs better than Gaussian augmentation on common corruptions. The main difference between both methods appears when analyzing the objectives that both methods minimize. For a single sample \(x\) , the loss function considered in Gaussian augmentation is:
\(\mathbb {E}_{d \sim N(0, I\sigma ^2)} \left[ \operatorname{\mathbf {\ell }}(\theta , x + d) \right] \ \sim \ \mathbb {E}_{\rho : ||\rho ||_2 = \sigma \sqrt{d}} \left[ \operatorname{\mathbf {\ell }}(\theta , x + \rho ) \right],\) 
