Flow-based generative models are powerful exact likelihood models with efficient sampling and inference. 
 Despite their computational efficiency, flow-based models generally have much worse density modeling performance compared to state-of-the-art autoregressive models. In this paper, we investigate and improve upon three limiting design choices employed by flow-based models in prior work: the use of uniform noise for dequantization, the use of inexpressive affine flows, and the use of purely convolutional conditioning networks in coupling layers. Based on our findings, we propose Flow++, a new flow-based model that is now the state-of-the-art non-autoregressive model for unconditional density estimation on standard image benchmarks. Our work has begun to close the significant performance gap that has so far existed between autoregressive models and flow-based models. Deep generative models -latent variable models in the form of variational autoencoders BID16 , implicit generative models in the form of GANs BID8 , and exact likelihood models like PixelRNN/CNN (van den c) , Image Transformer BID22 , PixelSNAIL , NICE, RealNVP, and Glow BID5 BID15 -have recently begun to successfully model high dimensional raw observations from complex real-world datasets, from natural images and videos, to audio signals and natural language BID14 BID34 .Autoregressive models, a certain subclass of exact likelihood models, achieve state-of-the-art density estimation performance on many challenging real-world datasets, but generally suffer from slow sampling time due to their autoregressive structure BID28 BID22 . Inverse autoregressive models can sample quickly and potentially have strong modeling capacity, but they cannot be trained efficiently by maximum likelihood . Non-autoregressive flow-based models (which we will refer to as "flow models"), such as NICE, RealNVP, and Glow, are efficient for sampling, but have so far lagged behind autoregressive models in density estimation benchmarks BID5 BID15 .In the hope of creating an ideal likelihood-based generative model that simultaneously has fast sampling, fast inference, and strong density estimation performance, we seek to close the density estimation performance gap between flow models and autoregressive models. In subsequent sections, we present our new flow model, Flow++, which is powered by an improved training procedure for continuous likelihood models and a number of architectural extensions of the coupling layer defined by BID5 . We presented Flow++, a new flow-based generative model that begins to close the performance gap between flow models and autoregressive models. Our work considers specific instantiations of design principles for flow models -dequantization, flow design, and conditioning architecture design -and we hope these principles will help guide future research in flow models and likelihoodbased models in general.7 APPENDIX A: SAMPLES
Improved training of current flow-based generative models (Glow and RealNVP) on density estimation benchmarks
{"src": "scitldr-AIC_train", "paper_id": "Hyg74h05tX"}