We introduce the “inverse square root linear unit” (ISRLU) to speed up learning in deep neural networks. ISRLU has better performance than ELU but has many of the same benefits. ISRLU and ELU have similar curves and characteristics. Both have negative values, allowing them to push mean unit activation closer to zero, and bring the normal gradient closer to the unit natural gradient, ensuring a noise- robust deactivation state, lessening the over fitting risk. The significant performance advantage of ISRLU on traditional CPUs also carry over to more efficient HW implementations on HW/SW codesign for CNNs/RNNs. In experiments with TensorFlow, ISRLU leads to faster learning and better generalization than ReLU on CNNs. This work also suggests a computationally efficient variant called the “inverse square root unit” (ISRU) which can be used for RNNs. Many RNNs use either long short-term memory (LSTM) and gated recurrent units (GRU) which are implemented with tanh and sigmoid activation functions. ISRU has less computational complexity but still has a similar curve to tanh and sigmoid. Two popular activation functions for neural networks are the rectified linear unit (ReLU) BID6 and the exponential linear unit (ELU) BID5 . The ReLU activation function is the identity for positive arguments and zero otherwise. The ELU activation function is the identity for positive arguments and has an exponential asymptotic approach to -1 for negative values.From previous analysis of the Fisher optimal learning, i.e., the natural gradient BID1 BID5 , we can reduce the undesired bias shift effect without the natural gradient, either by centering the activation of incoming units at zero or by using activation functions with negative values. We introduce the inverse square root linear unit (ISRLU), an activation function like ELU, that has smoothly saturating negative values for negative arguments, and the identity for positive arguments. In addition this activation function can be more efficiently implemented than ELU in a variety of software or purpose-built hardware. Activation function performance is becoming more important overall in convolutional neural networks (CNNs) because of the trending reductions in the computational complexity of the convolutions used in CNNs. We have introduced a new activation function, the inverse square root linear unit (ISRLU) for faster and precise learning in deep convolutional neural networks. ISRLUs have similar activation curves to ELUs, including the negative values. This decreases the forward propagated variation and brings the mean activations to zero. Mean activations close to zero decreases the bias shift for units in the next layer which speeds up learning by bringing the natural gradient closer to the unit natural gradient. Future work may prove the effectiveness of applying ISRLUs and the related ISRUs to other network architectures, such as recurrent neural networks, and to other tasks, such as object detection. ISRLUs have lower computational complexity than ELUs. Even greater savings on computation can be realized by implementing ISRLUs in custom hardware implementations. We expect ISRLU activations to increase the training efficiency of convolutional networks.
We introduce the ISRLU activation function which is continuously differentiable and faster than ELU. The related ISRU replaces tanh & sigmoid.
{"src": "scitldr-AIC_train", "paper_id": "HkMCybx0-"}