Sigmoid, Arctan and Tanh. Other notable non-linear functions are Sigmoid, Arctan and Tanh functions. As these functions' shapes are similar(S-shaped), their linear constraints are the same in essence. Recent techniques for Sigmoid and Tanh functions are proposed in DeepZ [1]} and DeepPoly [2]}. CROWN [3]} proposes a more general method that can offer non-trivial linear bounds to Sigmoid, Arctan, and Tanh functions and is extended to FROWN [4]} with tighter approximation by using a gradient-based method to find the bounding lines.
State-of-the-art linear approximation of Sigmoid, Arctan, and Tanh functions are proposed in VeriNet [5]} and DeepCert [6]}, DeepCert can compute much tighter robustness bounds compared with CROWN and FROWN, while VeriNet produces a minimum over-approximation region of Sigmoid, Arctan and Tanh functions and stands for the highest precision for general CNNs. Recently, network-wise Tightest approach is proposed in Newise [7]} and is the most precise method for monotonous CNNs(such as non-negative CNNs).
