Adaptive hyper-parameters during training have become common. Cyclical learning rates [1]}, one cycle learning rates [2]}, and cosine annealing with warm restarts [3]} have been accepted by the deep learning community and incorporated in PyTorch. General cyclical training provides an intuitive understanding for the value of a one cycle training regime. Furthermore, this idea of allowing a hyper-parameter value to change during training has been extended to other hyper-parameters, such as weight decay [4]}, [5]}, [6]}, [7]}, [8]} and batch sizes [9]}.
