In Figure REF , we compare our optimization framework to other optimizers commonly used in the deep learning community.
Similar to results in [1]}, when tuned appropriately, apart from ADAM, all other optimizers obtain performance comparable to SGD. In both settings, learning class or instance level learning rate, we outperform these standard optimizers by a significant margin.
These gains over standard optimizers can be attributed to the fact that
our framework adapts the optimization process over samples in dataset instead of model-parameters.
