Neural random fields (NRFs), which are defined by using neural networks to implement potential functions in undirected models (sometimes known as energy-based models), provide an interesting family of model spaces for machine learning, besides various directed models such as generative adversarial networks (GANs). In this paper we propose a new approach, the inclusive-NRF approach, to learning NRFs for continuous data (e.g. images), by developing inclusive-divergence minimized auxiliary generators and stochastic gradient sampling. As demonstrations of how the new approach can be flexibly and effectively used, specific inclusive-NRF models are developed and thoroughly evaluated for a number of tasks - unsupervised supervised image generation, semi-supervised classification and anomaly detection. The proposed models consistently achieve strong experimental results in all these tasks compared to state-of-the-art methods. Remarkably, in addition to superior sample generation, one fundamental additional benefit of our inclusive-NRF approach is that, unlike GANs, it directly provides (unnormalized) density estimate for sample evaluation. With these contributions and results, this paper significantly advances the learning and applications of undirected models to a new level, both theoretically and empirically, which have never been obtained before.
Training energy-based probabilistic models is confronted with apparently intractable sums, whose Monte Carlo estimation requires sampling from the estimated probability distribution in the inner loop of training. This can be approximately achieved by Markov chain Monte Carlo methods, but may still face a formidable obstacle that is the difficulty of mixing between modes with sharp concentrations of probability. Whereas an MCMC process is usually derived from a given energy function based on mathematical considerations and requires an arbitrarily long time to obtain good and varied samples, we propose to train a deep directed generative model (not a Markov chain) so that its sampling distribution approximately matches the energy function that is being trained. Inspired by generative adversarial networks, the proposed framework involves training of two models that represent dual views of the estimated probability distribution: the energy function (mapping an input configuration to a scalar energy value) and the generator (mapping a noise vector to a generated configuration), both represented by deep neural networks.