Generative Adversarial Networks ( GANs ) are a promising approach for text generation that , unlike traditional language models ( LM ) , does not suffer from the problem of " exposure bias " . However , A major hurdle for understanding the potential of GANs for text generation is the lack of a clear evaluation metric . In this work , we propose to approximate the distribution of text generated by a GAN , which permits evaluating them with traditional probability-based LM metrics . We apply our approximation procedure on several GAN-based models and show that they currently perform substantially worse than stateof-the-art LMs . Our evaluation procedure promotes better understanding of the relation between GANs and LMs , and can accelerate progress in GAN-based text generation . Neural networks have revolutionized the field of text generation , in machine translation ( Sutskever et al . , 2014 ; Neubig , 2017 ; Luong et al . , 2015 ; Chen et al . , 2018 ) , summarization ( See et al . , 2017 ) , image captioning ( You et al . , 2016 ) and many other applications ( Goldberg , 2017 ) . Traditionally , text generation models are trained by going over a gold sequence of symbols ( characters or words ) from left-to-right , and maximizing the probability of the next symbol given the history , namely , a language modeling ( LM ) objective . A commonly discussed drawback of such LM-based text generation is exposure bias ( Ranzato et al . , 2015 ) : during training , the model predicts the next token conditioned on the ground truth history , while at test time prediction is based on predicted tokens , causing a train-test mismatch . Models trained in this manner often struggle to overcome previous prediction errors . Generative Adversarial Networks ( Goodfellow et al . , 2014 ) offer a solution for exposure bias . * The authors contributed equally Originally introduced for images , GANs leverage a discriminator , which is trained to discriminate between real images and generated images via an adversarial loss . In such a framework , the generator is not directly exposed to the ground truth data , but instead learns to imitate it using global feedback from the discriminator . This has led to several attempts to use GANs for text generation , with a generator using either a recurrent neural network ( RNN ) ( Yu et al . , 2017 ; Guo et al . , 2017 ; Press et al . , 2017 ; Rajeswar et al . , 2017 ) , or a Convolutional Neural Network ( CNN ) ( Gulrajani et al . , 2017 ; Rajeswar et al . , 2017 ) . However , evaluating GANs is more difficult than evaluating LMs . While in language modeling , evaluation is based on the log-probability of a model on held-out text , this can not be straightforwardly extended to GAN-based text generation , because the generator outputs discrete tokens , rather than a probability distribution . Currently , there is no single evaluation metric for GAN-based text generation , and existing metrics that are based on n-gram overlap are known to lack robustness and have low correlation with semantic coherence ( Semeniuta et al . , 2018 ) . In this paper , we propose a method for evaluating GANs with standard probability-based evaluation metrics . We show that the expected prediction of a GAN generator can be viewed as a LM , and suggest a simple Monte-Carlo method for approximating it . The approximated probability distribution can then be evaluated with standard LM metrics such as perplexity or Bits Per Character ( BPC ) . To empirically establish our claim , we implement our evaluation on several RNN-based GANs : ( Press et al . , 2017 ; Yu et al . , 2017 ; Guo et al . , 2017 ) . We find that all models have substantially lower BPC compared to state-of-the-art LMs . By directly comparing to LMs , we put in perspective the current performance of RNN-based GANs for text generation . Our results are also in line with recent concurrent work by Caccia et al . ( 2018 ) , who reached a similar conclusion by comparing the performance of textual GANs to that of LMs using metrics suggested for GAN evaluation . Our code is available at : http : //github.com / GuyTevet / SeqGAN-eval and http://github.com / GuyTevet/ rnn-gan-eval . We propose an evaluation procedure for text GANs that is based on approximating the GAN output distribution and using standard LM metrics . We provide a bound for the number of samples required for the approximation and empirically show in practice as few as 2000 samples per time-step suffice . We evaluate character-based GAN models using our procedure , and show their performance is substantially lower than state-of-the-art LM . We hope our simple evaluation method leads to progress in GAN-based text generation by shedding light on the quality of such models .
Generative Adversarial Networks-based text generation models do not suffer from the exposure bias problem, however; they cannot be evaluated as other language models with log-probability.
They propose a way to approximate distributions from GAN-based models' outputs so that they can be evaluated as standard language models.
When GAN-based models are compared using the same evaluation metric as proposed, they perform much worse than current best language models.