Activation functions are usually handcrafted. PReLU [1]} tries to overcome this problem by introducing a learnable negative component to ReLU [2]}. Maxout [3]}, and Mixout [4]} are constructed with piecewise linear components, and theoretically, they are universal function approximators, though they increase the number of parameters in the network. Recently, meta-ACON [5]}, a smooth activation has been proposed, which is the generalization of the ReLU and Maxout activations and can smoothly approximate Swish. Meta-ACON has shown some good improvement on both small models and highly optimized large models. PAU [6]}, and OPAU [7]} are two promising candidates for trainable activations, which have been introduced recently based on rational function approximation.
