BackdoorBench: A Comprehensive Benchmark of Backdoor Learning
Backdoor learning is an emerging and important topic of studying the
vulnerability of deep neural networks (DNNs). Many pioneering backdoor attack
and defense methods are being proposed successively or concurrently, in the
status of a rapid arms race. However, we find that the evaluations of new
methods are often unthorough to verify their claims and real performance,
mainly due to the rapid development, diverse settings, as well as the
difficulties of implementation and reproducibility. Without thorough
evaluations and comparisons, it is difficult to track the current progress and
design the future development roadmap of the literature. To alleviate this
dilemma, we build a comprehensive benchmark of backdoor learning, called
BackdoorBench. It consists of an extensible modular based codebase (currently
including implementations of 8 state-of-the-art (SOTA) attack and 9 SOTA
defense algorithms), as well as a standardized protocol of a complete backdoor
learning. We also provide comprehensive evaluations of every pair of 8 attacks
against 9 defenses, with 5 poisoning ratios, based on 5 models and 4 datasets,
thus 8,000 pairs of evaluations in total. We further present analysis from
different perspectives about these 8,000 evaluations, studying the effects of
attack against defense algorithms, poisoning ratio, model and dataset in
backdoor learning. All codes and evaluations of BackdoorBench are publicly
available at \url{https://backdoorbench.github.io}.