As stated by Carlini & Wagner[1]}, detection methods should not only be evaluated on MNIST but more complex datasets, therefore we evaluate on CIFAR-10 and CIFAR-100. As the attacked neural network we choose the VGG-16 convolutional neural network [2]}. The network is trained on the CIFAR-10/100 [3]} training set, achieving 83.26% test accuracy on CIFAR-10 and 72.10% test accuracy on CIFAR-100, using available open-source implementationshttps://github.com/kuangliu/pytorch-cifar
https://github.com/weiaicunzai/pytorch-cifar100
. The correctly classified images from the CIFAR-10/100 test sets are then attacked by the different methods as described in REF , using the python toolbox foolbox [4]}. We use the default hyperparameters for each attack, as provided by the toolbox. We use \(L_{\infty }\) -BIM FGSM, and PGD attacks and \(L_2\) -Deepfool and C&W attacks. In order to decrease the computation time, we set steps=1000 for the C&W attack. Only successfully attacked pictures and their original counterparts are then put into our final dataset. In this way, we create a balanced dataset. Most of the attack methods are 100% successful on CIFAR-10, only the FGSM fools the network in just 95.9% of the time. For CIFAR-100 we use the same perturbation size, for FGSM, BIM, and PGD an attack success rate of about 98% is achieved, whereas Deepfool and C&W are again successful on the whole dataset.
<TABLE>