Re-parameterizing Your Optimizers rather than Architectures
The well-designed structures in neural networks reflect the prior knowledge
incorporated into the models. However, though different models have various
priors, we are used to training them with model-agnostic optimizers (e.g.,
SGD). In this paper, we propose a novel paradigm of incorporating
model-specific prior knowledge into optimizers and using them to train generic
(simple) models. As an implementation, we propose a novel methodology to add
prior knowledge by modifying the gradients according to a set of model-specific
hyper-parameters, which is referred to as Gradient Re-parameterization, and the
optimizers are named RepOptimizers. For the extreme simplicity of model
structure, we focus on a VGG-style plain model and showcase that such a simple
model trained with a RepOptimizer, which is referred to as RepOpt-VGG, performs
on par with the recent well-designed models. From a practical perspective,
RepOpt-VGG is a favorable base model because of its simple structure, high
inference speed and training efficiency. Compared to Structural
Re-parameterization, which adds priors into models via constructing extra
training-time structures, RepOptimizers require no extra forward/backward
computations and solve the problem of quantization. The code and models are
publicly available at https://github.com/DingXiaoH/RepOptimizers.