Deep networks form a crucial component of modern machine learning. Non-linearity is introduced in such networks by the use of activation functions, and the choice has a substantial impact on network performance and training dynamics. Designing a new novel activation function is a difficult task. Handcrafted activations like Rectified Linear Unit (ReLU) [1]}, Leaky ReLU [2]} or its variants are a very common choice for activation functions and exhibits promising performance on different deep learning tasks. There are many activations that have been proposed so far and some of them are ELU [3]}, Parametric ReLU (PReLU) [4]}, Swish [5]}, Tanhsoft [6]}, EIS [7]}, PAU [8]}, OPAU [9]}, ACON [10]}, ErfAct [11]}, Mish [12]}, GELU [13]} etc. Nevertheless, ReLU remains the favourite choice among the deep learning community due to its simplicity and better performance when compared to Tanh or Sigmoid, though it has a drawback known as dying ReLU in which the network starts to lose the gradient direction due to the negative inputs and produces zero outcome. In 2017, Swish [5]} was proposed by the Google brain team. Swish was found by automatic search technique, and it has shown some promising performance across different deep learning tasks.
