Invariance Properties of the Natural Gradient in Overparametrised Systems
The natural gradient field is a vector field that lives on a model equipped
with a distinguished Riemannian metric, e.g. the Fisher-Rao metric, and
represents the direction of steepest ascent of an objective function on the
model with respect to this metric. In practice, one tries to obtain the
corresponding direction on the parameter space by multiplying the ordinary
gradient by the inverse of the Gram matrix associated with the metric. We refer
to this vector on the parameter space as the natural parameter gradient. In
this paper we study when the pushforward of the natural parameter gradient is
equal to the natural gradient. Furthermore we investigate the invariance
properties of the natural parameter gradient. Both questions are addressed in
an overparametrised setting.