Hyper-Learning for Gradient-Based Batch Size Adaptation
Scheduling the batch size to increase is an effective strategy to control
gradient noise when training deep neural networks. Current approaches implement
scheduling heuristics that neglect structure within the optimization procedure,
limiting their flexibility to the training dynamics and capacity to discern the
impact of their adaptations on generalization. We introduce Arbiter as a new
hyperparameter optimization algorithm to perform batch size adaptations for
learnable scheduling heuristics using gradients from a meta-objective function,
which overcomes previous heuristic constraints by enforcing a novel learning
process called hyper-learning. With hyper-learning, Arbiter formulates a neural
network agent to generate optimal batch size samples for an inner deep network
by learning an adaptive heuristic through observing concomitant responses over
T inner descent steps. Arbiter avoids unrolled optimization, and does not
require hypernetworks to facilitate gradients, making it reasonably cheap,
simple to implement, and versatile to different tasks. We demonstrate Arbiter's
effectiveness in several illustrative experiments: to act as a stand-alone
batch size scheduler; to complement fixed batch size schedules with greater
flexibility; and to promote variance reduction during stochastic
meta-optimization of the learning rate.