As mentioned earlier, learning learning rates on data points can be interpreted as learning a weighting on them.
Some recent works [1]}, [2]}, [3]}, [4]}
have used meta-learning based approaches to dynamically assign weights to instances.
In general, these works [1]}, [3]}, [4]} train a secondary neural network to assign weights to data-points
throughout the course of training, or [2]} perform an online approximation of weights for each sample in the mini-batch.
These frameworks are Markovian in nature, since weights estimated at each time step are independent of past predictions.
In contrast, our framework treats learning rates on data as learnable parameters, and benefits from past history of optimization.
In Table REF , we establish the importance of retaining optimization history, by
evaluating our framework without the use of history.
Specifically, after each time step, we update the model parameters using the updated value of instance and class learning rates.
Post model parameter update, we reset the instance and class learning rates to their initial value of 1.
As shown in Table REF , not reusing the history of learnt learning rates leads to a
significant drop in performance across all settings and architectures on CIFAR100. Another place of comparison with this prior work is in
noisy setting (see Section REF ), where we outperform these methods by a significant margin.
