Deep neural network (DNN) models have gained tremendous success in many important domains.
For example, ResNet [1]} and BERT [2]} (based on pre-trained Transformer network [3]}) have shown impressive accuracy in the challenging area of image classification [4]} and natural language processing [5]} (NLP) tasks.
Researchers have shown that activation functions are important elements in DNN models, and proposed many novel activation functions for the better accuracy.
For most vision tasks, Mish [6]}, Swish [7]} and GELU [8]} surpass ReLU, specifically, about 1% accuracy improvement on ResNet [6]}, [7]}.
On the other hand, GELU [8]} is the most widely used activation function in the NLP models and achieves the best accuracy among other candidates.
