DNN models have recently achieved state-of-the-art results in many important domains, e.g., convolution neural network [1]} (CNN) in the computer vision domain, and BERT [2]} in the natural language processing domain.
Recently, many activation functions, e.g., Mish [3]}, Swish [4]}, and GELU [5]}, have been proposed to optimize ResNet-50 [6]} and BERT [7]}, showing higher accuracy than ReLU. Their formulas are shown in Tbl. REF .
