Recently, the importance of weight-decay in optimization of DNNs has gained much interest
[1]}, [2]}, [3]}, [4]}.
[1]} empirically demonstrate that weight-decay plays an important role in the first few epochs,
and does not play as much a role in the later stages of training. Our results indicates otherwise.
In this work, we show that learning a dynamic weight-decay leads to significant change
in SGD dynamics and also facilitates the learning of learning rates on data.
Figure REF  highlights the change in dynamics of SGD optimizer when weight-decay is learnt along with the model
parameters. Compared to baseline (fixed weight-decay), the learnt weight-decay adapts to different stages of optimization (see Figure
REF , right).
More specifically, post learning rate drop, when model is most prone to overfitting, dynamic weight decay coefficient increases.
This leads to a temporary drop in performance, but results in better generalization at convergence.
As see in the table (Figure REF , left), learning a dynamic weight-decay improves performance of all three optimizers.
<FIGURE>