Anderson mixing (or Anderson acceleration) is an efficient acceleration method for fixed point iterations (i.e., @math ), e.g., gradient descent can be viewed as iteratively applying the operation @math . It is known that Anderson mixing is quite efficient in practice and can be viewed as an extension of Krylov subspace methods for nonlinear problems. First, we show that Anderson mixing with Chebyshev polynomial parameters can achieve the optimal convergence rate @math , which improves the previous result @math provided by [Toth and Kelley, 2015] for quadratic functions. Then, we provide a convergence analysis for minimizing general nonlinear problems. Besides, if the hyperparameters (e.g., the Lipschitz smooth parameter @math ) are not available, we propose a Guessing Algorithm for guessing them dynamically and also prove a similar convergence rate. Finally, the experimental results demonstrate that the proposed Anderson-Chebyshev mixing method converges significantly faster than other algorithms, e.g., vanilla gradient descent (GD), Nesterov's Accelerated GD. Also, these algorithms combined with the proposed guessing algorithm (guessing the hyperparameters dynamically) achieve much better performance.
There are many sequence acceleration methods in the numerical analysis literatures. In particular, the well-known Aitken's @math process @cite_12 accelerated the convergence of a sequence that is converging linearly. Shanks generalized the Aitken extrapolation which was known as Shanks transformation @cite_21 . Recently, @cite_1 proposed a general framework for Shanks sequence transformations which includes many vector sequence acceleration methods. One fundamental difference between Anderson mixing and other sequence acceleration methods (such as MPE, RRE (reduced rank extrapolation) @cite_19 @cite_18 , etc.) is that Anderson mixing is a fully dynamic method @cite_14 . Here means all iterations are in the same sequence, and it does not require to restart the procedure. It can be seen from Algorithm that all iterations are applied to the same sequence @math . In fact, in Capehart's PhD thesis @cite_14 , several experiments were conducted to demonstrate the superior performance of Anderson mixing over other semi-dynamic methods such as MPE, RRE (semi-dynamic means that the algorithm maintains more than one sequences or needs to restart several times).
@cite_18 @cite_14 @cite_21 @cite_1 @cite_19 @cite_12
2100514507 1968796795 2004162424 2963487351
Many applications in science and engineering lead to models which require solving large-scale fixed point problems, or equivalently, systems of nonlinear equations. Several successful techniques for handling such problems are based on quasi-Newton methods that implicitly update the approximate Jacobian or inverse Jacobian to satisfy a certain secant condition. We present two classes of multisecant methods which allows to take into account a variable number of secant equations at each iteration. The first is the Broyden-like class, of which Broyden’s family is a subclass, and Anderson mixing is a particular member. The second class is that of the nonlinear Eirola-Nevanlinna-type methods. This work was motivated by a problem in electronic structure calculations, whereby a fixed point iteration, known as the self-consistent field (SCF) iteration, is accelerated by various strategies termed ‘mixing’. A general approach to the construction of convergence acceleration methods for vector sequences is proposed. Using this approach, one can generate some known methods, such as the minimal polynomial extrapolation, the reduced rank extrapolation, and the topological epsilon algorithm, and also some new ones. Some of the new methods are easier to implement than the known methods and are observed to have similar numerical properties. The convergence analysis of these new methods is carried out, and it is shown that they are especially suitable for accelerating the convergence of vector sequences that are obtained when one solves linear systems of equations iteratively. A stability analysis is also given, and numerical examples are provided. The convergence and stability properties of the topological epsilon algorithm are likewise given. This is an expository paper that describes and compares five methods for extrapolating to the limit (or anti-limit) of a vector sequence without explicit knowledge of the sequence generator. The methods are the minimal polynomial extrapolation (MPE) studied by Cabay and Jackson, Mesina, and Skelboe; the reduced rank extrapolation (RRE) of Eddy (which we show to be equivalent to Mesina’s version of MPE); the vector and scalar versions of the epsilon algorithm (VEA, SEA) introduced by Wynn and extended by Brezinski and Gekeler; and the topological epsilon algorithm (TEA) of Brezinski. We cover the derivation and error analysis of iterated versions of the algorithms, as applied to both linear and nonlinear problems, and we show why these versions tend to converge quadratically. We also present samples from extensive numerical testing that has led us to the following conclusions: (a) TEA, in spite of its role as a theoretical link between the polynomial-type and the epsilon-type methods, has no practical appl... Nesterov's accelerated gradient descent (AGD), an instance of the general family of momentum methods,'' provably achieves faster convergence rate than gradient descent (GD) in the convex setting. While these methods are widely used in modern nonconvex applications, including training of deep neural networks, whether they are provably superior to GD in the nonconvex setting remains open. This paper studies a simple variant of Nesterov's AGD, and shows that it escapes saddle points and finds a second-order stationary point in ( O (1 ^ 7 4 ) ) iterations, matching the best known convergence rate, which is faster than the ( O (1 ^ 2 ) ) iterations required by GD. To the best of our knowledge, this is the first direct acceleration (single-loop) algorithm that is provably faster than GD in general nonconvex setting---all previous nonconvex accelerated algorithms rely on more complex mechanisms such as nested loops and proximal terms. Our analysis is based on two key ideas: (1) the use of a simple Hamiltonian function, inspired by a continuous-time perspective, which AGD monotonically decreases on each step even for nonconvex functions, and (2) a novel framework called improve or localize, which is useful for tracking the long-term behavior of gradient-based optimization algorithms. We believe that these techniques may deepen our understanding of both acceleration algorithms and nonconvex optimization.