@inproceedings{
clarke2023adam,
title={Adam through a Second-Order Lens},
author={Ross M Clarke and Baiyu Su and Jos{\'e} Miguel Hern{\'a}ndez-Lobato},
booktitle={OPT 2023: Optimization for Machine Learning},
year={2023},
url={https://openreview.net/forum?id=NMUCBSfNxy}
}
Research into optimisation for deep learning is characterised by a tension between the computational efficiency of first-order, gradient-based methods (such as SGD and Adam) and the theoretical efficiency of second-order, curvature-based methods (such as quasi-Newton methods and K-FAC). We seek to combine the benefits of both approaches into a single computationally-efficient algorithm. Noting that second-order methods often depend on stabilising heuristics (such as Levenberg-Marquardt damping), we propose AdamQLR: an optimiser combining damping and learning rate selection techniques from K-FAC with the update directions proposed by Adam, inspired by considering Adam through a second-order lens. We evaluate AdamQLR on a range of regression and classification tasks at various scales, achieving competitive generalisation performance vs runtime.
Adam through a Second-Order Lens
Ross M Clarke Baiyu Su José Miguel Hernández-Lobato