A Survey of Optimization Methods from a Machine Learning Perspective <s> 5) Natural Gradient: <s> When a parameter space has a certain underlying structure, the ordinary gradient of a function does not represent its steepest direction, but the natural gradient does. Information geometry is used for calculating the natural gradients in the parameter space of perceptrons, the space of matrices (for blind source separation), and the space of linear dynamical systems (for blind source deconvolution). The dynamical behavior of natural gradient online learning is analyzed and is proved to be Fisher efficient, implying that it has asymptotically the same performance as the optimal batch estimation of parameters. This suggests that the plateau phenomenon, which appears in the backpropagation learning algorithm of multilayer perceptrons, might disappear or might not be so serious when the natural gradient is used. An adaptive method of updating the learning rate is proposed and analyzed. <s> BIB001 </s> A Survey of Optimization Methods from a Machine Learning Perspective <s> 5) Natural Gradient: <s> We develop a 2nd-order optimization method based on the "Hessian-free" approach, and apply it to training deep auto-encoders. Without using pre-training, we obtain results superior to those reported by Hinton & Salakhutdinov (2006) on the same tasks they considered. Our method is practical, easy to use, scales nicely to very large datasets, and isn't limited in applicability to auto-encoders, or any specific model class. We also discuss the issue of "pathological curvature" as a possible explanation for the difficulty of deep-learning and how 2nd-order optimization, and our method in particular, effectively deals with it. <s> BIB002 </s> A Survey of Optimization Methods from a Machine Learning Perspective <s> 5) Natural Gradient: <s> Natural gradient descent is an optimization method traditionally motivated from the perspective of information geometry, and works well for many applications as an alternative to stochastic gradient descent. In this paper we critically analyze this method and its properties, and show how it can be viewed as a type of approximate 2nd-order optimization method, where the Fisher information matrix can be viewed as an approximation of the Hessian. This perspective turns out to have significant implications for how to design a practical and robust version of the method. Additionally, we make the following contributions to the understanding of natural gradient and 2nd-order methods: a thorough analysis of the convergence speed of stochastic natural gradient descent (and more general stochastic 2nd-order methods) as applied to convex quadratics, a critical examination of the oft-used "empirical" approximation of the Fisher matrix, and an analysis of the (approximate) parameterization invariance property possessed by natural gradient methods, which we show still holds for certain choices of the curvature matrix other than the Fisher, but notably not the Hessian. <s> BIB003 </s> A Survey of Optimization Methods from a Machine Learning Perspective <s> 5) Natural Gradient: <s> Second-order optimization methods, such as natural gradient, are difficult to apply to highdimensional problems, because they require approximately solving large linear systems. We present FActorized Natural Gradient (FANG), an approximation to natural gradient descent where the Fisher matrix is approximated with a Gaussian graphical model whose precision matrix can be computed efficiently. We analyze the Fisher matrix for a small RBM and derive an extremely sparse graphical model which is a good match to the covariance of the sufficient statistics. Our experiments indicate that FANG allows RBMs to be trained more efficiently compared with stochastic gradient descent. Additionally, our analysis yields insight into the surprisingly good performance of the "centering trick" for training RBMs. <s> BIB004 </s> A Survey of Optimization Methods from a Machine Learning Perspective <s> 5) Natural Gradient: <s> We propose an efficient method for approximating natural gradient descent in neural networks which we call Kronecker-factored Approximate Curvature (K-FAC). K-FAC is based on an efficiently invertible approximation of a neural network's Fisher information matrix which is neither diagonal nor low-rank, and in some cases is completely non-sparse. It is derived by approximating various large blocks of the Fisher (corresponding to entire layers) as being the Kronecker product of two much smaller matrices. While only several times more expensive to compute than the plain stochastic gradient, the updates produced by K-FAC make much more progress optimizing the objective, which results in an algorithm that can be much faster than stochastic gradient descent with momentum in practice. And unlike some previously proposed approximate natural-gradient/Newton methods which use high-quality non-diagonal curvature matrices (such as Hessian-free optimization), K-FAC works very well in highly stochastic optimization regimes. This is because the cost of storing and inverting K-FAC's approximation to the curvature matrix does not depend on the amount of data used to estimate it, which is a feature typically associated only with diagonal or low-rank approximations to the curvature matrix. <s> BIB005
The natural gradient method can be potentially applied to any objective function which measures the performance of some statistical models BIB001 . It enjoys richer theoretical properties when applied to objective functions based on the KL divergence between the model's distribution and the target distribution, or certain approximation surrogates of these BIB003 . The traditional gradient descent algorithm is based on Euclidean space. However, in many cases, the parameter space is not Euclidean, and it may have a Riemannian metric structure. In this case, the steepest direction of the objective function cannot be given by the ordinary gradient and should be given by the natural gradient BIB001 . We consider such a model distribution p(y|x, θ), and π(x, y) is an empirical distribution. We need to fit the parameters θ ∈ R N . Assume that x is an observation vector, and y is its associated label. It has the objective function, and we need to solve the optimization problem, According to BIB001 , the natural gradient can be transformed from a traditional gradient multiplied by a Fisher information matrix, i.e., where F is the object function, ▽F is the traditional gradient, ▽ N F is the natural gradient, and G is the Fisher information matrix, with the following form: (78) The update formula with the natural gradient is We cannot ignore that the application of the natural gradient is very limited because of too much computation. It is expensive to estimate the Fisher information matrix and calculate its inverse matrix. To overcome this limitation, the truncated Newton's method was developed BIB002 , in which the inverse is calculated by an iterative procedure, thus avoiding the direct calculation of the inverse of the Fisher information matrix. In addition, the factorized natural gradient (FANG) BIB004 and Kronecker-factored approximate curvature (K-FAC) BIB005 methods were proposed to use the derivatives of probabilistic models to calculate the approximate natural gradient update.