The full implications and mechanisms for the emergence of adversarial examples are not yet fully understood and are an active area of research. Among other things, the existence and pervasiveness of adversarial examples points to the limitations of the standard iid models as these data are not sampled from the same distribution as the training set.
Yet, it can be proved mathematically that adversarial examples are unavoidable for interpolating classifiers in the presence of label noise [1]} (Theorem 5.1).
Specifically, suppose \(f_{\rm int}\)  is an interpolating classifier and let \({\mathbf {x}}\)  be an arbitrary point. Assume that \(f_{\rm int}({\mathbf {x}})=y\)  is a correct prediction.
Given a sufficiently large dataset, there will be at least one ”noisy” point \({\mathbf {x}}_i, y_i,\) , such as \(f^*({\mathbf {x}}_i)\ne y_i\) , in a small neighborhood of \({\mathbf {x}}\)  and thus a small perturbation of \({\mathbf {x}}\)  can be used to flip the label.
