Linearity Grafting: Relaxed Neuron Pruning Helps Certifiable Robustness
Certifiable robustness is a highly desirable property for adopting deep
neural networks (DNNs) in safety-critical scenarios, but often demands tedious
computations to establish. The main hurdle lies in the massive amount of
non-linearity in large DNNs. To trade off the DNN expressiveness (which calls
for more non-linearity) and robustness certification scalability (which prefers
more linearity), we propose a novel solution to strategically manipulate
neurons, by "grafting" appropriate levels of linearity. The core of our
proposal is to first linearize insignificant ReLU neurons, to eliminate the
non-linear components that are both redundant for DNN performance and harmful
to its certification. We then optimize the associated slopes and intercepts of
the replaced linear activations for restoring model performance while
maintaining certifiability. Hence, typical neuron pruning could be viewed as a
special case of grafting a linear function of the fixed zero slopes and
intercept, that might overly restrict the network flexibility and sacrifice its
performance. Extensive experiments on multiple datasets and network backbones
show that our linearity grafting can (1) effectively tighten certified bounds;
(2) achieve competitive certifiable robustness without certified robust
training (i.e., over 30% improvements on CIFAR-10 models); and (3) scale up
complete verification to large adversarially trained models with 17M
parameters. Codes are available at
https://github.com/VITA-Group/Linearity-Grafting.