Score matching provides an effective approach to learning flexible unnormalized models, but its scalability is limited by the need to evaluate a second-order derivative.   In this paper,we connect a general family of learning objectives including score matching to Wassersteingradient flows. This connection enables us to design a scalable approximation to theseobjectives, with a form similar to single-step contrastive divergence. We present applications in training implicit variational and Wasserstein auto-encoders with manifold-valued priors. Unnormalized models define the model distribution as q(x; θ) ∝ exp(−E(x; θ)), where E(x; θ) is an energy function that can be parameterized by e.g. DNNs. Unnormalized models can be used directly for density estimation, but another important application is in gradient estimation for implicit variational inference, where we can use score estimation in latent space to approximate an intractable learning objective. This approach leads to improved performance in training implicit auto-encoders (Song et al., 2019) . Maximum likelihood estimation for unnormalized models is intractable, and score matching (Hyvärinen, 2005 ) is a popular alternative. Score matching optimizes the Fisher divergence where we denote the data distribution as p. Hyvärinen (2005) shows D F is equivalent to E p(x) ∆ log q(x; θ) + 1 2 ∇ log q(x; θ) 2 , where ∆ = i ∂ 2 i is the Laplacian; the equivalent form can be estimated using samples from p. So far, when E has a complex parameterization, calculating the equivalent objective is still difficult, as it involves the second-order derivatives; and in practice, people turn to scalable approximations of the score matching objective (Song et al., 2019; Hyvarinen, 2007; Vincent, 2011) or other objectives such as the kernelized Stein discrepancy (KSD; Liu et al., 2016b; Liu and Wang, 2017) . However, these approximations are developed on a case-by-case basis, leaving important applications unaddressed; for example, there is a lack of scalable learning methods for models on manifolds (Mardia et al., 2016) . In this work, we present a unifying perspective to this problem, and derive scalable approximations for a variety of objectives including score matching. We start by interpreting these objectives as the initial velocity of certain distribution-space gradient flows, which are simulated by common samplers. This novel interpretation leads to a scalable approximation algorithm for all such objectives, reminiscent to single-step contrastive divergence (CD-1). We refer to any objective bearing the above interpretation as above as a "minimum velocity learning objective", a term coined in the unpublished work Movellan (2007) . Our formulation is a distribution-space generalization of their work, and applies to different objectives as the choice of distribution space varies. Another gap we fill in is the development of a practically applicable algorithm: while the idea of approximating score matching with CD-1 is also explored in (Hyvarinen, 2007; Movellan, 2007) , previously the approximation suffers from an infinite variance problem, and is thus believed to be impractical (Hyvarinen, 2007; Saremi et al., 2018) ; we present a simple fix to this issue. Additionally, we present an approximation to the objective function instead of its gradient, thus enabling the use of regularization like early-stopping. Other related work will be reviewed in Appendix C. One important application of our framework is in learning unnormalized models on manifolds. This is needed in areas such as image analysis (Srivastava et al., 2007) , geology (Davis and Sampson, 1986) and bioinformatics (Boomsma et al., 2008) . Moreover, as we present an approximation to the Riemannian score matching objective, it enables flexible inference for VAEs and WAEs with manifold-valued latent variables, as it enables gradient estimation for implicit variational distributions on manifolds. It is believed that auto-encoders with a manifold-valued latent space can capture the distribution of certain types of data better (Mathieu et al., 2019; Anonymous, 2020; Davidson et al., 2018) . As we will see in Section 3, our method leads to improved performance of VAEs and WAEs.
We present a scalable approximation to a wide range of EBM objectives, and applications in implicit VAEs and WAEs
{"src": "scitldr-AIC_train", "paper_id": "HJl4FJnEYr"}