BFE and AdaBFE: A New Approach in Learning Rate Automation for Stochastic Optimization
In this paper, a new gradient-based optimization approach by automatically
adjusting the learning rate is proposed. This approach can be applied to design
non-adaptive learning rate and adaptive learning rate. Firstly, I will
introduce the non-adaptive learning rate optimization method: Binary Forward
Exploration (BFE), and then the corresponding adaptive per-parameter learning
rate method: Adaptive BFE (AdaBFE) is possible to be developed. This approach
could be an alternative method to optimize the learning rate based on the
stochastic gradient descent (SGD) algorithm besides the current non-adaptive
learning rate methods e.g. SGD, momentum, Nesterov and the adaptive learning
rate methods e.g. AdaGrad, AdaDelta, Adam... The purpose to develop this
approach is not to beat the benchmark of other methods but just to provide a
different perspective to optimize the gradient descent method, although some
comparative study with previous methods will be made in the following sections.
This approach is expected to be heuristic or inspire researchers to improve
gradient-based optimization combined with previous methods.