Equation REF  and the success of cyclical learning rates implies that a cyclical approach also might work for weight decay, batch size, and momentum as well.
That is, it might be easier for the network to learn when weight decay or momentum starts smaller or if batch size starts larger, followed by a larger weight decay/momentum or smaller batch size in the middle epochs. While [1]} proposed increasing batch size instead of decreasing the learning rate, we propose taking this a step further to a cyclical batch size.
<TABLE>