Figure REF  compares the evolution of the MSE in the Adam-based gradients to those in the SGD gradients used in Lanvegin-gradients for the Coil 2000 problem. In previous Langevin-gradient Bayesian neural learning methods [1]}, [2]} SGD was used as the standard optimiser for simple neural network models; but for autoencoders, we find that Adam-based gradients performs much better for the same number of samples. This is likely due to the adaptive learning rate in Adam-based gradients being able to adapt to the larger number of parameters typical for autoencoders. Hence, single learning rate in SGD is not be as effective for large deep learning models.
