Energy  based  models  outputs  unmormalized  log-probability  values  given  datasamples.   Such a estimation is essential in a variety of application problems suchas sample generation, denoising, sample restoration, outlier detection, Bayesianreasoning, and many more.   However, standard maximum likelihood training iscomputationally expensive due to the requirement of sampling model distribution. Score matching potentially alleviates this problem, and denoising score matching(Vincent, 2011) is a particular convenient version.   However,  previous attemptsfailed to produce models capable of high quality sample synthesis.   We believethat  it  is  because  they  only  performed  denoising  score  matching  over  a  singlenoise scale. To overcome this limitation, here we instead learn an energy functionusing all noise scales.    When sampled using Annealed Langevin dynamics andsingle step denoising jump, our model produced high-quality samples comparableto state-of-the-art techniques such as GANs, in addition to assigning likelihood totest data comparable to previous likelihood models.   Our model set a new sam-ple quality baseline in likelihood-based models.   We further demonstrate that our model learns sample distribution and generalize well on an image inpainting tasks. Treating data as stochastic samples from a probability distribution and developing models that can learn such distributions is at the core for solving a large variety of application problems, such as error correction/denoising (Vincent et al., 2010) , outlier/novelty detection (Zhai et al., 2016; Choi and Jang, 2018) , sample generation (Nijkamp et al., 2019; Du and Mordatch, 2019) , invariant pattern recognition, Bayesian reasoning (Welling and Teh, 2011) which relies on good data priors, and many others. Energy-Based Models (EBMs) (LeCun et al., 2006; Ngiam et al., 2011 ) assign an energy E(x x x) to each data point x x x which implicitly defines a probability by the Boltzmann distribution p m (x x x) = e −E(x x x) /Z. Sampling from this distribution can be used as a generative process that yield plausible samples of x x x. Compared to other generative models, like GANs (Goodfellow et al., 2014) , flowbased models (Dinh et al., 2015; Kingma and Dhariwal, 2018) , or auto-regressive models (van den Oord et al., 2016; Ostrovski et al., 2018) , energy-based models have significant advantages. First, they provide explicit (unnormalized) density information, compositionality (Hinton, 1999; Haarnoja et al., 2017) , better mode coverage (Kumar et al., 2019) and flexibility (Du and Mordatch, 2019) . Further, they do not require special model architecture, unlike auto-regressive and flow-based models. Recently, Energy-based models has been successfully trained with maximum likelihood (Nijkamp et al., 2019; Du and Mordatch, 2019) , but training can be very computationally demanding due to the need of sampling model distribution. Variants with a truncated sampling procedure have been proposed, such as contrastive divergence (Hinton, 2002) . Such models learn much faster with the draw back of not exploring the state space thoroughly (Tieleman, 2008) . Score matching (SM) (Hyvärinen, 2005) circumvents the requirement of sampling the model distribution. In score matching, the score function is defined to be the gradient of log-density or the negative energy function. The expected L2 norm of difference between the model score function and the data score function are minimized. One convenient way of using score matching is learning the energy function corresponding to a Gaussian kernel Parzen density estimator (Parzen, 1962) of the data: p σ0 (x x x) = q σ0 (x x x|x x x)p(x x x)dx x x. Though hard to evaluate, the data score is well defined: s d (x x x) = ∇x x x log(p σ0 (x x x)), and the corresponding objective is: L SM (θ) = E pσ0(x x x) ∇x x x log(p σ0 (x x x)) + ∇x x x E(x x x; θ) In this work we provided analyses and empirical results for understanding the limitations of learning the structure of high-dimensional data with denoising score matching. We found that the objective function confines learning to a small set due to the measure concentration phenomenon in random vectors. Therefore, sampling the learned distribution outside the set where the gradient is learned does not produce good result. One remedy to learn meaningful gradients in the entire space is to use samples during learning that are corrupted by different amounts of noise. Indeed, Song and Ermon (2019) applied this strategy very successfully. The central contribution of our paper is to investigate how to use a similar learning strategy in EBMs. Specifically, we proposed a novel EBM model, the Multiscale Denoising Score Matching (MDSM) model. The new model is capable of denoising, producing high-quality samples from random noise, and performing image inpainting. While also providing density information, our model learns an order of magnitude faster than models based on maximum likelihood. Our approach is conceptually similar to the idea of combining denoising autoencoder and annealing (Geras and Sutton, 2015; Chandra and Sharma, 2014; Zhang and Zhang, 2018) though this idea was proposed in the context of pre-training neural networks for classification applications. Previous efforts of learning energy-based models with score matching (Kingma and LeCun, 2010; were either computationally intensive or unable to produce high-quality samples comparable to those obtained by other generative models such as GANs. Saremi et al. (2018) and Saremi and Hyvarinen (2019) trained energy-based model with the denoising score matching objective but the resulting models cannot perform sample synthesis from random noise initialization. Recently, proposed the NCSN model, capable of high-quality sample synthesis. This model approximates the score of a family of distributions obtained by smoothing the data by kernels of different widths. The sampling in the NCSN model starts with sampling the distribution obtained with the coarsest kernel and successively switches to distributions obtained with finer kernels. Unlike NCSN, our method learns an energy-based model corresponding to p σ0 (x x x) for a fixed σ 0 . This method improves score matching in high-dimensional space by matching the gradient of an energy function to the score of p σ0 (x x x) in a set that avoids measure concentration issue. All told, we offer a novel EBM model that achieves high-quality sample synthesis, which among other EBM approaches provides a new state-of-the art. Compared to the NCSN model, our model is more parsimonious than NCSN and can support single step denoising without prior knowledge of the noise magnitude. But our model performs sightly worse than the NCSN model, which could have several reasons. First, the derivation of Equation 6 requires an approximation to keep the training procedure tractable, which could reduce the performance. Second, the NCSNs output is a vector that, at least during optimization, does not always have to be the derivative of a scalar function. In contrast, in our model the network output is a scalar function. Thus it is possible that the NCSN model performs better because it explores a larger set of functions during optimization.
Learned energy based model with score matching
{"src": "scitldr-AIC_train", "paper_id": "HJeFmkBtvB"}