We propose two approaches of locally adaptive activation functions namely, layer-wise and neuron-wise locally adaptive activation functions, which improve the performance of deep and physics-informed neural networks. The local adaptation of activation function is achieved by introducing scalable hyper-parameters in each layer (layer-wise) and for every neuron separately (neuron-wise), and then optimizing it using the stochastic gradient descent algorithm. Introduction of neuron-wise activation function acts like a vector activation function as opposed to the traditional scalar activation function given by fixed, global and layer-wise activations. In order to further increase the training speed, an activation slope based slope recovery term is added in the loss function, which further accelerate convergence, thereby reducing the training cost. For numerical experiments, a nonlinear discontinuous function is approximated using a deep neural network with layer-wise and neuron-wise locally adaptive activation functions with and without the slope recovery term and compared with its global counterpart. Moreover, solution of the nonlinear Burgers equation, which exhibits steep gradients, is also obtained using the proposed methods. On the theoretical side, we prove that in the proposed method the gradient descent algorithms are not attracted to sub-optimal critical points or local minima under practical conditions on the initialization and learning rate. Furthermore, the proposed adaptive activation functions with the slope recovery are shown to accelerate the training process in standard deep learning benchmarks using CIFAR-10, CIFAR-100, SVHN, MNIST, KMNIST, Fashion-MNIST, and Semeion data sets with and without data augmentation. In recent years, research on neural networks (NNs) has intensified around the world due to their successful applications in many diverse fields such as speech recognition , computer vision (Krizhevsky et al., 2012) , natural language translation (Wu et al., 2016) , etc. Training of NN is performed on data sets before using it in the actual applications. Various data sets are available for applications like image classification, which is a subset of computer vision. MNIST (LeCun et al., 1998) and their variants like, Fashion-MNIST (Xiao et al., 2017) , and KMNIST (Clanuwat et al., 2018) are the data sets for handwritten digits, images of clothing and accessories, and Japanese letters, respectively. Apart from MNIST, Semeion (Brescia, 1994 ) is a handwritten digit data set that contains 1593 digits collected from 80 persons. SVHN (Netzer et al., 2011) is another data set for street view house numbers obtained from house numbers in Google Street View images. CI-FAR (Krizhevsky et al., 2009 ) is the popular data set containing color images commonly used to train machine learning algorithms. In particular, the CIFAR-10 data set contains 50000 training and 10000 testing images in 10 classes with image resolution of 32x32. CIFAR-100 is similar to the CIFAR-10, except it has 100 classes with 600 images in each class, which is more challenging than the CIFAR-10 data set. problems, where the approximate solutions of governing equations are obtained, as well as inverse problems, where parameters involved in the governing equation are inferred from the training data. Highly efficient and adaptable algorithms are important to design the most effective NN which not only increases the accuracy of the solution but also reduces the training cost. Various architectures of NN like Dropout NN (Srivastava et al., 2014) are proposed in the literature, which can improve the efficiency of the algorithm for specific applications. Activation function plays an important role in the training process of NN. In this work, we are particularly focusing on adaptive activation functions, which adapt automatically such that the network can be trained faster. Various methods are proposed in the literature for adaptive activation function, like the adaptive sigmoidal activation function proposed by (Yu et al., 2002) for multilayer feedforward NNs, while (Qian et al., 2018) focuses on learning activation functions in convolutional NNs by combining basic activation functions in a data-driven way. Multiple activation functions per neuron are proposed (Dushkoff & Ptucha, 2016) , where individual neurons select between a multitude of activation functions. (Li et al., 2013) proposed a tunable activation function, where only a single hidden layer is used and the activation function is tuned. (Shen et al., 2004) , used a similar idea of tunable activation function but with multiple outputs. Recently, Kunc and Kléma proposed a transformative adaptive activation functions for gene expression inference, see (Kunc & Kléma, 2019) . One such adaptive activation function is proposed (Jagtap & Karniadakis, 2019) by introducing scalable hyper-parameter in the activation function, which can be optimized. Mathematically, it changes the slope of activation function thereby increasing the learning process, especially during the initial training period. Due to single scalar hyper-parameter, we call such adaptive activation functions globally adaptive activations, meaning that it gives an optimized slope for the entire network. One can think of doing such optimization at the local level, where the scalable hyper-parameter are introduced hidden layer-wise or even for each neuron in the network. Such local adaptation can further improve the performance of the network. Figure 1 shows a sketch of a neuron-wise locally adaptive activation function based physics-informed neural network (LAAF-PINN), where both the NN part along with the physicsinformed part can be seen. In this architecture, along with the output of NN and the residual term from the governing equation, the activation slopes from every neuron are also contributing to the loss function in the form of slope recovery term. The rest of the paper is organized as follows. Section 2 presents the methodology of the proposed layer-wise and neuron-wise locally adaptive activations in detail. This also includes a discussion on the slope recovery term, expansion of parametric space due to layer-wise and neuron-wise introduction of hyper-parameters, its effect on the overall training cost, and a theoretical result for gradient decent algorithms. Section 3 gives numerical experiments, where we approximate a nonlinear discontinuous function using deep NN by the proposed approaches. We also solve the Burgers equation using the proposed algorithm and present various comparisons in appendix B. Section 4 presents numerical results with various standard deep learning benchmarks using CIFAR-10, CIFAR-100, SVHN, MNIST, KMNIST, Fashion-MNIST, and Semeion data sets. Finally, in section 5, we summarize the conclusions of our work. In this paper, we present two versions of locally adaptive activation functions namely, layer-wise and neuron-wise locally adaptive activation functions. Such local activation functions further improve the training speed of the neural network compared to its global predecessor. To further accelerate the training process, an activation slope based slope recovery term is added in the loss function for both layer-wise and neuron-wise activation functions, which is shown to enhance the performance of the neural network. Various NN and PINN test cases like nonlinear discontinuous function approximation and Burgers equation respectively, and benchmark deep learning problems like MNIST, CIFAR, SVHN etc are solved to verify our claim. Moreover, we theoretically prove that no sub-optimal critical point or local minimum attracts gradient descent algorithms in the proposed methods (L-LAAF and N-LAAF) with the slope recovery term under only mild assumptions. k=1 is a limit point of (Θ m ) m∈N and a sub-optimal critical point or a sub-optimal local minimum. and h Following the proofs in (Bertsekas, 1997, Propositions 1.2.1-1.2.4), we have that ∇J(Θ) = 0 and J(Θ) < Jc(0) + S(0), for all three cases of the conditions corresponding the different rules of the learning rate. Therefore, we have that for all k ∈ {1, . . . , D − 1}, Furthermore, we have that for all k ∈ {1, . . . , D − 1} and all j ∈ {1, . . . , N k }, By combining equation 5-equation 7, for all k ∈ {1, . . . , D − 1}, which implies that for all a k = 0 since (D − 1) exp(a k ) = 0. This implies that J(Θ) = Jc(0) + S(0), which contradicts with J(Θ) < Jc(0) + S(0). This proves the desired statement for L-LAAF. For N-LAAF, we prove the statement by contradiction. Suppose that the parameter vectorΘ consisting of {w k=1 ∀j = 1, 2, · · · , N k is a limit point of (Θ m ) m∈N and a suboptimal critical point or a sub-optimal local minimum. Redefine and h for all j ∈ {1, . . . , N k }, where w k,j ∈ R 1×N k−1 and b k,j ∈ R. Then, by the same proof steps, we have that ∇J(Θ) = 0 and J(Θ) < Jc(0) + S(0), for all three cases of the conditions corresponding the different rules of the learning rate. Therefore, we have that for all k ∈ {1, . . . , D − 1} and all j ∈ {1, . . . , N k }, By combining equation 6-equation 8, for all k ∈ {1, . . . , D − 1} and all j ∈ {1, . . . , N k }, , which implies that for all a This implies that J(Θ) = Jc(0) + S(0), which contradicts with J(Θ) < Jc(0) + S(0). This proves the desired statement for N-LAAF.
Proposing locally adaptive activation functions in deep and physics-informed neural networks for faster convergence
{"src": "scitldr-AIC_train", "paper_id": "rkeJzpNtPS"}