Vanishing and exploding gradients are two of the main obstacles in training deep neural networks, especially in capturing long range dependencies in recurrent neural networks (RNNs). In this paper, we present an efficient parametrization of the transition matrix of an RNN that allows us to stabilize the gradients that arise in its training. Specifically, we parameterize the transition matrix by its singular value decomposition (SVD), which allows us to explicitly track and control its singular values. We attain efficiency by using tools that are common in numerical linear algebra, namely Householder reflectors for representing the orthogonal matrices that arise in the SVD. By explicitly controlling the singular values, our proposed svdRNN method allows us to easily solve the exploding gradient problem and we observe that it empirically solves the vanishing gradient issue to a large extent. We note that the SVD parameterization can be used for any rectangular weight matrix, hence it can be easily extended to any deep neural network, such as a multi-layer perceptron. Theoretically, we demonstrate that our parameterization does not lose any expressive power, and show how it potentially makes the optimization process easier. Our extensive  experimental results also demonstrate that the proposed framework converges faster, and has good generalization, especially when the depth is large. 
 Deep neural networks have achieved great success in various fields, including computer vision, speech recognition, natural language processing, etc. Despite their tremendous capacity to fit complex functions, optimizing deep neural networks remains a contemporary challenge. Two main obstacles are vanishing and exploding gradients, that become particularly problematic in Recurrent Neural Networks (RNNs) since the transition matrix is identical at each layer, and any slight change to it is amplified through recurrent layers BID3 ).Several methods have been proposed to solve the issue, for example, Long Short Term Memory (LSTM) BID8 ) and residual networks BID7 ). Another recently proposed class of methods is designed to enforce orthogonality of the square transition matrices, such as unitary and orthogonal RNNs (oRNN) BID1 ; BID13 ). However , while these methods solve the exploding gradient problem, they limit the expressivity of the network.In this paper, we present an efficient parametrization of weight matrices that arise in a deep neural network, thus allowing us to stabilize the gradients that arise in its training, while retaining the desired expressive power of the network. In more detail we make the following contributions:• We propose a method to parameterize weight matrices through their singular value decomposition (SVD). Inspired by BID13 ), we attain efficiency by using tools that are common in numerical linear algebra, namely Householder reflectors for representing the orthogonal matrices that arise in the SVD. The SVD parametrization allows us to retain the desired expressive power of the network, while enabling us to explicitly track and control singular values.• We apply our SVD parameterization to recurrent neural networks to exert spectral constraints on the RNN transition matrix. Our proposed svdRNN method enjoys similar space and time complexity as the vanilla RNN. We empirically verify the superiority of svdRNN over RNN/oRNN, in some case even LSTMs, over an exhaustive collection of time series classification tasks and the synthetic addition and copying tasks, especially when the network depth is large.• Theoretically , we show how our proposed SVD parametrization can make the optimization process easier. Specifically, under a simple setting, we show that there are no spurious local minimum for the linear svdRNN in the population risk.• Our parameterization is general enough to eliminate the gradient vanishing/exploding problem not only in RNNs, but also in various deep networks. We illustrate this by applying SVD parametrization to problems with non-square weight matrices, specifically multi-layer perceptrons (MLPs) and residual networks.We now present the outline of our paper. In Section 2, we discuss related work, while in Section 3 we introduce our SVD parametrization and demonstrate how it spans the whole parameter space and does not limit expressivity. In Section 4 we propose the svdRNN model that is able to efficiently control and track the singular values of the transition matrices, and we extend our parameterization to non-square weight matrices and apply it to MLPs in Section 5. Section 6 provides the optimization landscape of svdRNN by showing that linear svdRNN has no spurious local minimum. Experimental results on MNIST and a popular time series archive are present in Section 7. Finally, we present our conclusions and future work in Section 8. In this paper, we have proposed an efficient SVD parametrization of various weight matrices in deep neural networks, which allows us to explicitly track and control their singular values. This parameterization does not restrict the network's expressive power, while simultaneously allowing fast forward as well as backward propagation. The method is easy to implement and has the same time and space complexity as compared to original methods like RNN and MLP. The ability to control singular values helps in avoiding the gradient vanishing and exploding problems, and as we have empirically shown, gives good performance. Although we only showed examples in the RNN and MLP framework, our method is applicable to many more deep networks, such as Convolutional Networks etc. However, further experimentation is required to fully understand the influence of using different number of reflectors in our SVD parameterization. Also, the underlying structures of the image of M k1,k2 when k 1 , k 2 = 1 is a subject worth investigating. DISPLAYFORM0 Proof of Proposition 1. For n = 1, note that H 1 1 (u 1 ) = ±1. By setting u 1 = 0 if B 1,1 > 0 and u 1 = 0 otherwise, we have the factorization desired.Assume that the result holds for n = k, then for n = k + 1 set u k+1 = B 1 − B 1 e 1 . Here B 1 is the first column of B and e 1 = (1, 0, ..., 0) . Thus we have DISPLAYFORM1 , whereB ∈ R k×k . Note that H k+1 k+1 (u k+1 ) = I k+1 when u k+1 = 0 and the above still holds. By DISPLAYFORM2 is an upper triangular matrix with positive diagonal elements. Thus the result holds for any n by the theory of mathematical induction.A.2 PROOF OF THEOREM 1 Proof. Observe that the image of M 1 is a subset of O(n), and we now show that the converse is also true. Given A ∈ O(n), by Proposition 1, there exists an upper triangular matrix R with positive diagonal elements, and an orthogonal matrix Q expressed as DISPLAYFORM3 , such that A = QR. Since A is orthogonal, we have A A = AA = I n , thus:A A = R Q QR = R R = I n ; Q AA Q = Q QRR Q Q = RR = I n Thus R is orthogonal and upper triangular matrix with positive diagonal elements. So R = I n and DISPLAYFORM4
To solve the gradient vanishing/exploding problems, we proprose an efficient parametrization of the transition matrix of RNN that loses no expressive power, converges faster and has good generalization.
{"src": "scitldr-AIC_train", "paper_id": "SyL9u-WA-"}