Use of weight-decay as a regularizer is a de-facto standard in training Deep Neural Networks (DNNs).
However, the exact role weight-decay plays in optimization of modern DNNs is not well understood
[1]}, [2]}, [3]}, [4]}. In this section, we highlight the importance
of learning the weight-decay coefficient along with the learning rate for dataset, class or instances.
For ease of explanation, let us consider the case where we are interested in learning instance level learning rate,
and we have the standard weight-decay term added as a regularizer.
\(\mathcal {L}_{train}(\theta ^t, w_{dataset}^{t}) &= \frac{1}{N} \sum _{i=1}^N \mathbf {w}^{i,t}_{inst} \cdot L_{train}^{i}(\theta ^t) +\lambda _{wd} \Vert \theta ^t\Vert \) 
