We propose Cooperative Training (CoT) for training generative models that measure a tractable density for discrete data. CoT coordinately trains a generator G and an auxiliary predictive mediator M. The training target of M is to estimate a mixture density of the learned distribution G and the target distribution P, and that of G is to minimize the Jensen-Shannon divergence estimated through M. CoT achieves independent success without the necessity of pre-training via Maximum Likelihood Estimation or involving high-variance algorithms like REINFORCE. This low-variance algorithm is theoretically proved to be superior for both sample generation and likelihood prediction. We also theoretically and empirically show the superiority of CoT over most previous algorithms in terms of generative quality and diversity, predictive generalization ability and computational cost. Generative modeling is essential in many scenarios, including continuous data modeling (e.g. image generation BID6 , stylization BID17 , semisupervised classification BID13 ) and sequential discrete data modeling (e.g. neural text generation BID2 ).For discrete data with tractable density like natural language, generative models are predominantly optimized through Maximum Likelihood Estimation (MLE), inevitably introducing exposure bias BID14 , which results in that given a finite set of observations, the optimal parameters of the model trained via MLE do not correspond to the ones maximizing the generative quality. Specifically , the model is trained on the data distribution of inputs and tested on a different distribution of inputs, namely, the learned distribution. This discrepancy implies that in the training stage, the model is never exposed to its own errors and thus in the test stage, the errors made along the way will quickly accumulate.On the other hand, for general generative modeling tasks, an effective framework, named Generative Adversarial Network (GAN) BID6 , was proposed to train an implicit density model for continuous data. GAN introduces a discriminator D φ parametrized by φ to distinguish the generated samples from the real ones. As is proved in BID6 , GAN essentially optimizes an approximately estimated Jensen-Shannon divergence (JSD) between the currently learned distribution and the target distribution. GAN shows promising results in many unsupervised and semi-supervised learning tasks. The success of GAN results in the naissance of a new paradigm of deep generative models, i.e. adversarial networks.However, since the gradient computation requires backpropagation through the generator's output, GAN can only model the distribution of continuous variables, making it non-applicable for generating discrete sequences like natural language. Researchers then proposed Sequence Generative Adversarial Network (SeqGAN) , which uses model-free policy gradient algorithm to optimize the original GAN objective. With SeqGAN, the expected JSD between current and target discrete data distribution is minimized if the training is perfect. SeqGAN shows observable improvements in many tasks. Since then, many variants of SeqGAN have been proposed to improve its performance. Nonetheless, SeqGAN is not an ideal algorithm for this problem, and current algorithms based on it cannot show stable, reliable and observable improvements that covers all scenarios, according to a previous survey . The detailed reason will be discussed in detail in Section 2.In this paper, we propose Cooperative Training (CoT), a novel, low-variance, bias-free algorithm for training likelihood-based generative models on discrete data by directly optimizing a wellestimated Jensen-Shannon divergence. CoT coordinately trains a generative module G, and an auxiliary predictive module M , called mediator, for guiding G in a cooperative fashion. For theoretical soundness, we derive the proposed algorithm directly from the definition of JSD. We further empirically and theoretically demonstrate the superiority of our algorithm over many strong baselines in terms of generative performance, generalization ability and computational performance in both synthetic and real-world scenarios. Computational Efficiency Although in terms of time cost per epoch, CoT does not achieve the state-of-the-art, we do observe that CoT is remarkably faster than previous RL-GAN approaches. Besides, consider the fact that CoT is a sample-based optimization algorithm, which involves time BID3 8.89 8.71/-(MLE) (The same as MLE) 32.54 ± 1.14s Professor Forcing BID10 9 To show the hyperparameter robustness of CoT, we compared it with the similar results as were evaluated in SeqGAN . DISPLAYFORM0 cost in sampling from the generator, this result is acceptable. The result also verifies our claim that CoT has the same order (i.e. the time cost only differs in a constant multiplier or extra lower order term) of computational complexity as MLE.Hyper-parameter Robustness. We perform a hyper-parameter robustness experiment on synthetic data experiment. When compared with the results of similar experiments as in SeqGAN , our approach shows less sensitivity to hyper-parameter choices, as shown in FIG1 . Note that since in all our attempts, the evaluated JSD of SeqGAN fails to converge, we evaluated NLL oracle for it as a replacement.Self-estimated Training Progress Indicator. Like the critic loss, i.e. estimated Earth Mover Distance, in WGANs, we find that the training loss of the mediator (9), namely balanced NLL, can be a real-time training progress indicator as shown in FIG2 . Specifically, in a wide range, balanced NLL is a good estimation of real JSD(G P ) with a steady translation, namely, balanced N LL = JSD(G P ) + H(G) + H(P ). 2.900 (σ = 0.025) 3.118 (σ = 0.018) 3.122 RankGAN BID11 We proposed Cooperative Training, a novel training algorithm for generative modeling of discrete data. CoT optimizes Jensen-Shannon Divergence, which does not have the exposure bias problem as the forward KLD. Models trained via CoT shows promising results in sequential discrete data modeling tasks, including sample quality and the generalization ability in likelihood prediction tasks.B SAMPLE COMPARISON AND DISCUSSION TAB6 shows samples from some of the most powerful baseline models and our model. The Optimal Balance for Cooperative Training We find that the same learning rate and iteration numbers for the generator and mediator seems to be the most competitive choice. As for the architecture choice, we find that the mediator needs to be slightly stronger than the generator. For the best result in the synthetic experiment, we adopt exactly the same generator as other compared models and a mediator whose hidden state size is twice larger (with 64 hidden units) than the generator.Theoretically speaking, we can and we should sample more batches from G θ and P respectively for training the mediator in each iteration. However, if no regularizations are used when training the mediator, it can easily over-fit, leading the generator's quick convergence in terms of KL(G θ P ) or NLL oracle , but divergence in terms of JSD(G θ P ). Empirically, this could be alleviated by applying dropout techniques BID15 with 50% keeping ratio before the output layer of RNN. After applying dropout, the empirical results show good consistency with our theory that, more training batches for the mediator in each iteration is always helpful.However, applying regularizations is not an ultimate solution and we look forward to further theoretical investigation on better solutions for this problem in the future. (5) " I think it was alone because I can do that, when you're a lot of reasons, " he said.(6 ) It's the only thing we do, we spent 26 and $35(see how you do is we lose it," said both sides in the summer. CoT(1) We focus the plans to put aside either now, and which doesn't mean it is to earn the impact to the government rejected.(2) The argument would be very doing work on the 2014 campaign to pursue the firm and immigration officials, the new review that's taken up for parking.(3) This method is true to available we make up drink with that all they were willing to pay down smoking.(4) The number of people who are on the streaming boat would study if the children had a bottle -but meant to be much easier, having serious ties to the outside of the nation.(5) However, they have to wait to get the plant in federal fees and the housing market's most valuable in tourism. MLE (1) after the possible cost of military regulatory scientists, chancellor angela merkel's business share together a conflict of major operators and interest as they said it is unknown for those probably 100 percent as a missile for britain.(2) but which have yet to involve the right climb that took in melbourne somewhere else with the rams even a second running mate and kansas. (3) " la la la la 30 who appeared that themselves is in the room when they were shot her until the end " that jose mourinho could risen from the individual . (4) when aaron you has died, it is thought if you took your room at the prison fines of radical controls by everybody, if it's a digital plan at an future of the next time.Possible Derivatives of CoT The form of equation 13 can be modified to optimize other objectives. One example is the backward KLD (a.k.a. Reverse KLD) i.e. KL(G P ). In this case, the objective of the so-called "Mediator" and "Generator" thus becomes:"Mediator", now it becomes a direct estimatorP φ of the target distribution P : DISPLAYFORM0 Generator: DISPLAYFORM1 Such a model suffers from so-called mode-collapse problem, as is analyzed in Ian's GAN Tutorial BID5 . Besides , as the distribution estimatorP φ inevitably introduces unpredictable behaviors when given unseen samples i.e. samples from the generator, the algorithm sometimes fails (numerical error) or diverges.In our successful attempts, the algorithm produces similar (not significantly better than) results as CoT. The quantitive results are shown as follows: Although under evaluation of weak metrics like BLEU, if successfully trained, the model trained via Reverse KL seems to be better than that trained via CoT, the disadvantage of Reverse KL under evaluation of more strict metric like eWMD indicates that Reverse KL does fail in learning some aspects of the data patterns e.g. completely covering the data mode.
We proposed Cooperative Training, a novel training algorithm for generative modeling of discrete data.
{"src": "scitldr-AIC_train", "paper_id": "SkxxIs0qY7"}