Information Geometry of Dropout Training
Dropout is one of the most popular regularization techniques in neural
network training. Because of its power and simplicity of idea, dropout has been
analyzed extensively and many variants have been proposed. In this paper,
several properties of dropout are discussed in a unified manner from the
viewpoint of information geometry. We showed that dropout flattens the model
manifold and that their regularization performance depends on the amount of the
curvature. Then, we showed that dropout essentially corresponds to a
regularization that depends on the Fisher information, and support this result
from numerical experiments. Such a theoretical analysis of the technique from a
different perspective is expected to greatly assist in the understanding of
neural networks, which are still in their infancy.