Training neural networks using Metropolis Monte Carlo and an adaptive variant
We examine the zero-temperature Metropolis Monte Carlo algorithm as a tool
for training a neural network by minimizing a loss function. We find that, as
expected on theoretical grounds and shown empirically by other authors,
Metropolis Monte Carlo can train a neural net with an accuracy comparable to
that of gradient descent, if not necessarily as quickly. The Metropolis
algorithm does not fail automatically when the number of parameters of a neural
network is large. It can fail when a neural network's structure or neuron
activations are strongly heterogenous, and we introduce an adaptive Monte Carlo
algorithm, aMC, to overcome these limitations. The intrinsic stochasticity of
the Monte Carlo method allows aMC to train neural networks in which the
gradient is too small to allow training by gradient descent. We suggest that,
as for molecular simulation, Monte Carlo methods offer a complement to
gradient-based methods for training neural networks, allowing access to a
distinct set of network architectures and principles.