Learning adaptive learning rates on data can be interpreted as learning a weighting on each data point.
[1]}, [2]}, [3]} train an auxillary neural network to assign weights to data points.
[4]} performs an online approximation, where it uses one step look ahead on meta-set to estimate the weights for samples in the mini-batch.
These approaches are Markovian in nature, since weights estimated at each time step are independent of past-predictions.
In contrast, our method leverages the past history of optimization, and outperforms these state-of-the-art methods for
robust learning (see Section REF ).
