A promising class of generative models maps points from a simple distribution to a complex distribution through an invertible neural network.    Likelihood-based training  of  these  models  requires  restricting  their  architectures  to  allow  cheap computation of Jacobian determinants.   Alternatively, the Jacobian trace can be used if the transformation is specified by an ordinary differential equation. In this paper, we use Hutchinson’s trace estimator to give a scalable unbiased estimate of the log-density.   The result is a continuous-time invertible generative model with unbiased density estimation and one-pass sampling, while allowing unrestricted neural network architectures. We demonstrate our approach on high-dimensional density  estimation,  image  generation,  and  variational  inference,  achieving  the state-of-the-art among exact likelihood methods with efficient sampling. We have presented FFJORD, a reversible generative model for high-dimensional data which can compute exact log-likelihoods and can be sampled from efficiently. Our model uses continuoustime dynamics to produce a generative model which is parameterized by an unrestricted neural network. All required quantities for training and sampling can be computed using automatic differentiation, Hutchinson's trace estimator, and black-box ODE solvers. Our model stands in contrast to other methods with similar properties which rely on restricted, hand-engineered neural network architectures. We demonstrated that this additional flexibility allows our approach to achieve on-par or improved performance on density estimation and variational inference.We believe there is much room for further work exploring and improving this method. FFJORD is empirically slower to evaluate than other reversible models like Real NVP or Glow, so we are interested specifically in ways to reduce the number of function evaluations used by the ODE-solver without hurting predictive performance. Advancements like these will be crucial in scaling this method to even higher-dimensional datasets.
We use continuous time dynamics to define a generative model with exact likelihoods and efficient sampling that is parameterized by unrestricted neural networks.
{"src": "scitldr-AIC_train", "paper_id": "rJxgknCcK7"}