
It is noticeable that this function is not zero centred but passes by an extremely close neighbourhood of zero. To make the function zero centered, we have multiplied the first term of (3) by a linear component \(x\) . To further investigate these two functions as a possible candidates for activation function, we have conducted several experiments on MNIST [1]}, CIFAR10 [2]}, and CIFAR100 [2]} datasets with PreActResNet-18 [4]}, VGG-16 (with batch-normalization) [5]} [6]}, and DenseNet-121 [7]} models, and we notice that both of them performs almost similar in every cases. So, for the rest of the paper, we will only consider the approximate identity of Leaky ReLU (\(\alpha =0.15\) ) given in (3) as the activation function. We call this function Smooth Approximation Unit (SAU).
