This idea of allowing a hyper-parameter value to change during training (in replacement for a learning rate schedule) has been extended to other hyper-parameters, such as weight decay [1]}, [2]}, [3]}, [4]} and batch sizes [5]}.
Previous work [6]}, [5]} has demonstrated empirically a relationship between the optimal hyper-parameters of learning rate (LR), weight decay (WD), batch size (BS), and momentum (m) as
\(\frac{LR \times WD}{BS \times (1 - m)} \simeq 10^{-6}.\) 
