Other work has proposed optimizing a few hyperparameters, such as kernel parameters [1]}, weight decay [2]} or
others [3]}, using gradients during training. However, to the best of our knowledge,
none have done so for learning dynamic learning rates across training per se, nor done so at scale
(e.g. one rate per instance) to achieve state-of-the-art performance.
