Efficient Adversarial Training With Data Pruning
Neural networks are susceptible to adversarial examples-small input
perturbations that cause models to fail. Adversarial training is one of the
solutions that stops adversarial examples; models are exposed to attacks during
training and learn to be resilient to them. Yet, such a procedure is currently
expensive-it takes a long time to produce and train models with adversarial
samples, and, what is worse, it occasionally fails. In this paper we
demonstrate data pruning-a method for increasing adversarial training
efficiency through data sub-sampling.We empirically show that data pruning
leads to improvements in convergence and reliability of adversarial training,
albeit with different levels of utility degradation. For example, we observe
that using random sub-sampling of CIFAR10 to drop 40% of data, we lose 8%
adversarial accuracy against the strongest attackers, while by using only 20%
of data we lose 14% adversarial accuracy and reduce runtime by a factor of 3.
Interestingly, we discover that in some settings data pruning brings benefits
from both worlds-it both improves adversarial accuracy and training time.