Generative adversarial networks are a learning framework that rely on training a discriminator to estimate a measure of difference between a target and generated distributions. GANs, as normally formulated, rely on the generated samples being completely differentiable w.r.t. the generative parameters, and thus do not work for discrete data. We introduce a method for training GANs with discrete data that uses the estimated difference measure from the discriminator to compute importance weights for generated samples, thus providing a policy gradient for training the generator. The importance weights have a strong connection to the decision boundary of the discriminator, and we call our method boundary-seeking GANs (BGANs). We demonstrate the effectiveness of the proposed algorithm with discrete image and character-based natural language generation.   In addition, the boundary-seeking objective extends to continuous data, which can be used to improve stability of training, and we demonstrate this on Celeba, Large-scale Scene Understanding (LSUN) bedrooms, and Imagenet without conditioning. Generative adversarial networks (GAN, BID7 involve a unique generative learning framework that uses two separate models, a generator and discriminator, with opposing or adversarial objectives. Training a GAN only requires back-propagating a learning signal that originates from a learned objective function, which corresponds to the loss of the discriminator trained in an adversarial manner. This framework is powerful because it trains a generator without relying on an explicit formulation of the probability density, using only samples from the generator to train.GANs have been shown to generate often-diverse and realistic samples even when trained on highdimensional large-scale continuous data BID31 . GANs however have a serious limitation on the type of variables they can model, because they require the composition of the generator and discriminator to be fully differentiable.With discrete variables, this is not true. For instance, consider using a step function at the end of a generator in order to generate a discrete value. In this case, back-propagation alone cannot provide the training signal, because the derivative of a step function is 0 almost everywhere. This is problematic, as many important real-world datasets are discrete, such as character-or word-based representations of language. The general issue of credit assignment for computational graphs with discrete operations (e.g. discrete stochastic neurons) is difficult and open problem, and only approximate solutions have been proposed in the past BID2 BID8 BID10 BID14 BID22 BID40 . However, none of these have yet been shown to work with GANs. In this work, we make the following contributions:• We provide a theoretical foundation for boundary-seeking GANs (BGAN), a principled method for training a generator of discrete data using a discriminator optimized to estimate an f -divergence BID29 BID30 . The discriminator can then be used to formulate importance weights which provide policy gradients for the generator.• We verify this approach quantitatively works across a set of f -divergences on a simple classification task and on a variety of image and natural language benchmarks.• We demonstrate that BGAN performs quantitatively better than WGAN-GP BID9 in the simple discrete setting.• We show that the boundary-seeking objective extends theoretically to the continuous case and verify it works well with some common and difficult image benchmarks. Finally , we show that this objective has some improved stability properties within training and without. On estimating likelihood ratios from the discriminator Our work relies on estimating the likelihood ratio from the discriminator, the theoretical foundation of which we draw from f -GAN BID30 . The connection between the likelihood ratios and the policy gradient is known in previous literature BID15 , and the connection between the discriminator output and the likelihood ratio was also made in the context of continuous GANs BID26 BID39 . However, our work is the first to successfully formulate and apply this approach to the discrete setting.Importance sampling Our method is very similar to re-weighted wake-sleep (RWS, BID3 , which is a method for training Helmholtz machines with discrete variables. RWS also relies on minimizing the KL divergence, the gradients of which also involve a policy gradient over the likelihood ratio. Neural variational inference and learning (NVIL, BID25 , on the other hand, relies on the reverse KL. These two methods are analogous to our importance sampling and REINFORCE-based BGAN formulations above.GAN for discrete variables Training GANs with discrete data is an active and unsolved area of research, particularly with language model data involving recurrent neural network (RNN) generators BID20 . Many REINFORCE-based methods have been proposed for language modeling BID20 BID6 which are similar to our REINFORCE-based BGAN formulation and effectively use the sigmoid of the estimated loglikelihood ratio. The primary focus of these works however is on improving credit assignment, and their approaches are compatible with the policy gradients provided in our work.There have also been some improvements recently on training GANs on language data by rephrasing the problem into a GAN over some continuous space BID19 BID16 BID9 . However, each of these works bypass the difficulty of training GANs with discrete data by rephrasing the deterministic game in terms of continuous latent variables or simply ignoring the discrete sampling process altogether, and do not directly solve the problem of optimizing the generator from a difference measure estimated from the discriminator.Remarks on stabilizing adversarial learning, IPMs, and regularization A number of variants of GANs have been introduced recently to address stability issues with GANs. Specifically, generated samples tend to collapse to a set of singular values that resemble the data on neither a persample or distribution basis. Several early attempts in modifying the train procedure (Berthelot et al., 2017; BID35 as well as the identifying of a taxonomy of working architectures BID31 addressed stability in some limited setting, but it wasn't until Wassertstein GANs (WGAN, BID1 were introduced that there was any significant progress on reliable training of GANs.WGANs rely on an integral probability metric (IPM, BID36 ) that is the dual to the Wasserstein distance. Other GANs based on IPMs, such as Fisher GAN tout improved stability in training. In contrast to GANs based on f -divergences, besides being based on metrics that are "weak", IPMs rely on restricting T to a subset of all possible functions. For instance in WGANs, T = {T | T L ≤ K}, is the set of K-Lipschitz functions. Ensuring a statistic network, T φ , with a large number of parameters is Lipschitz-continuous is hard, and these methods rely on some sort of regularization to satisfy the necessary constraints. This includes the original formulation of WGANs, which relied on weight-clipping, and a later work BID9 which used a gradient penalty over interpolations between real and generated data.Unfortunately, the above works provide little details on whether T φ is actually in the constrained set in practice, as this is probably very hard to evaluate in the high-dimensional setting. Recently, BID32 introduced a gradient norm penalty similar to that in BID9 without interpolations and which is formulated in terms of f -divergences. In our work, we've found that this approach greatly improves stability, and we use it in nearly all of our results. That said, it is still unclear empirically how the discriminator objective plays a strong role in stabilizing adversarial learning, but at this time it appears that correctly regularizing the discriminator is sufficient. Reinterpreting the generator objective to match the proposal target distribution reveals a novel learning algorithm for training a generative adversarial network (GANs, BID7 . This proposed approach of boundary-seeking provides us with a unified framework under which learning algorithms for both discrete and continuous variables are derived. Empirically, we verified our approach quantitatively and showed the effectiveness of training a GAN with the proposed learning algorithm, which we call a boundary-seeking GAN (BGAN), on both discrete and continuous variables, as well as demonstrated some properties of stability.Starting image (generated) 10k updates GAN Proxy GAN BGAN 20k updates Figure 5 : Following the generator objective using gradient descent on the pixels. BGAN and the proxy have sharp initial gradients that decay to zero quickly, while the variational lower-bound objective gradient slowly increases. The variational lower-bound objective leads to very poor images, while the proxy and BGAN objectives are noticeably better. Overall, BGAN performs the best in this task, indicating that its objective will not overly disrupt adversarial learning.Berthelot, David, Schumm, Tom, and Metz, Luke. Began: Boundary equilibrium generative adversarial networks. arXiv preprint arXiv:1703.10717, 2017.
We address training GANs with discrete data by formulating a policy gradient that generalizes across f-divergences
{"src": "scitldr-AIC_train", "paper_id": "rkTS8lZAb"}