In general, it has been observed that minimization performance can be noticeably improved under appropriate learning schemes e.g., under deterministic adaptive learning rate schemes. Examples include ADAM [1]} and some variants (e.g., AMSGrad [2]} or ADAMW [3]}) or precursors (e.g. [4]}, [5]}). This work demonstrates that performance can be significantly improved if instead of just adaptive, the learning rate becomes both adaptive and stochastic.
