Augmented Newton Method for Optimization: Global Linear Rate and Momentum Interpretation
We propose two variants of Newton method for solving unconstrained
minimization problem. Our method leverages optimization techniques such as
penalty and augmented Lagrangian method to generate novel variants of the
Newton method namely the Penalty Newton method and the Augmented Newton method.
In doing so, we recover several well-known existing Newton method variants such
as Damped Newton, Levenberg, and Levenberg-Marquardt methods as special cases.
Moreover, the proposed Augmented Newton method can be interpreted as Newton
method with adaptive heavy ball momentum. We provide global convergence results
for the proposed methods under mild assumptions that hold for a wide variety of
problems. The proposed methods can be sought as the penalty and augmented
extensions of the results obtained by Karimireddy et. al [24].