To explain the reason why narrowing over-approximation zone can give rise to certified robustness bounds, we define Layer-wise Tightest linear bounds:
[Layer-wise Tightest]
\(\forall k\in [K],i\in [n_k],m=\frac{l^{k-1}+u^{k-1}}{2}\) , if both \(u_i^k(m)\)  and \(-l_i^k(m)\)  reach the minimum, then the linear upper and lower linear approximations of every non-linear function are the layer-wise Tightest.

We discover that Layer-wise Tightest is equivalent to Neuron-wise Tightest and we prove this in the Appendix. This means that Neuron-wise Tightest can be preserved through one-layer propagation, which can explain the reason why VeriNet outperforms most previous works(such as DeepPoly [1]}, Neurify [2]}). Zhang et al. [3]} discover that VeriNet is sometimes inferior to DeepCert and we also compare these two state-of-the-art methods(results shown in Table REF ). To tackle this, they define the notion of Network-wise Tightest and propose Newise, the resulting method, which can preserve the tightest through the whole network propagation. However, Newise only has the best performance on monotonous neural network(such as non-negative networks), which is not the mainstream.
