Measuring adversarial robustness is not trivial and several best practices have been recommended [1]}. We incorporate several of them in our study. In particular we study (1) the robustness of models to random Gaussian noise, (2) the minimum distortion required to make a datapoint adversarial, and (3) the accuracy of substitute model black box attacks. We focus mainly on binary classification between classes 0 and 1 on the CIFAR10 image benchmark [2]} where we make the following findings.
