@inproceedings{
glaser2024nearoptimality,
title={Near-Optimality of Contrastive Divergence Algorithms},
author={Pierre Glaser and Kevin Han Huang and Arthur Gretton},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=Q74JVgKCP6}
}
We provide a non-asymptotic analysis of the contrastive divergence (CD) algorithm, a training method for unnormalized models. While prior work has established that (for exponential family distributions) the CD iterates asymptotically converge at an $O(n^{-1 / 3})$ rate to the true parameter of the data distribution, we show that CD can achieve the parametric rate $O(n^{-1 / 2})$. Our analysis provides results for various data batching schemes, including fully online and minibatch. We additionally show that CD is near-optimal, in the sense that its asymptotic variance is close to the Cramér-Rao lower bound.
Near-Optimality of Contrastive Divergence Algorithms
Pierre Glaser Kevin Han Huang Arthur Gretton