Our work also has connections to importance sampling.
[1]}, [2]}, [3]} propose approximations for the gradient norm of instances which are used for sampling data points.
Theoretically, sampling data points with high gradient norm should lead to faster convergence. However, this would not work well in real world dataset which
contain noisy data. In the same spirit, our work also has connections to the field of curriculum
learning [4]}, [5]}, [6]} and self-paced learning [7]}, [8]}, [9]}, [10]}.
These approaches either design hand-crafted heuristics, or use loss value of data points as a proxy to decide the ordering of data.
All these approaches require coming up with a heuristics which might not work from one problem domain to another. In contrast,
our work can be interpreted as a soft differentiable form of importance sampling, where the importance of a sample (learning rate) or curriculum is
learnt through meta gradient.
