Symmetry Teleportation for Accelerated Optimization
Existing gradient-based optimization methods update the parameters locally,
in a direction that minimizes the loss function. We study a different approach,
symmetry teleportation, that allows the parameters to travel a large distance
on the loss level set, in order to improve the convergence speed in subsequent
steps. Teleportation exploits parameter space symmetries of the optimization
problem and transforms parameters while keeping the loss invariant. We derive
the loss-invariant group actions for test functions and multi-layer neural
networks, and prove a necessary condition of when teleportation improves
convergence rate. We also show that our algorithm is closely related to second
order methods. Experimentally, we show that teleportation improves the
convergence speed of gradient descent and AdaGrad for several optimization
problems including test functions, multi-layer regressions, and MNIST
classification.