The use of a decaying learning rate schedule (where the learning rate value is reduced during the training) is standard practice for network training but the use of adaptive hyper-parameters during training has become common.
For example, the use of cyclical learning rates [1]}, [2]}, [3]} has become widely accepted by the deep learning community. Unlike the decaying learning rate schedule, cyclical learning rates start with a small value of the learning rate, which enables the network's weights to move toward a good direction in the loss landscape [4]} and increases the learning rate in the early epochs.
In addition, learning rate warmup [5]} is essentially equivalent to cyclical learning rates, although the warm up period is often restricted to a few of the early epochs (i.e., the same as setting \(f_c\)  in Equation REF  to a large value).
