Conventional Generative Adversarial Networks (GANs) for text generation tend to have issues of reward sparsity and mode collapse that affect the quality and diversity of generated samples. To address the issues, we propose a novel self-adversarial learning (SAL) paradigm for improving GANs' performance in text generation. In contrast to standard GANs that use a binary classifier as its discriminator to predict whether a sample is real or generated, SAL employs a comparative discriminator which is a pairwise classifier for comparing the text quality between a pair of samples. During training, SAL rewards the generator when its currently generated sentence is found to be better than its previously generated samples. This self-improvement reward mechanism allows the model to receive credits more easily and avoid collapsing towards the limited number of real samples, which not only helps alleviate the reward sparsity issue but also reduces the risk of mode collapse. Experiments on text generation benchmark datasets show that our proposed approach substantially improves both the quality and the diversity, and yields more stable performance compared to the previous GANs for text generation. Generative Adversarial Networks ) (GANs) have achieved tremendous success for image generation and received much attention in computer vision. For text generation, however, the performance of GANs is severely limited due to reward sparsity and mode collapse: reward sparsity refers to the difficulty for the generator to receive reward signals when its generated samples can hardly fool the discriminator that is much easier to train; while mode collapse refers to the phenomenon that the generator only learns limited patterns from the real data. As a result, both the quality and diversity of generated text samples are limited. To address the above issues, we propose a novel self-adversarial learning (SAL) paradigm for improving adversarial text generation. In contrast to standard GANs (Figure 1(a ) ) that use a binary classifier as its discriminator to predict whether a sample is real or generated, SAL employs a comparative discriminator which is a pairwise classifier assessing whether the currently generated sample is better than its previously generated one, as shown in Figure 1 ( b). During training, SAL rewards the generator when its currently generated samples are found to be better than its previously generated samples. In the earlier training stage when the quality of generated samples is far below the real data, this self-improvement reward mechanism makes it easier for the generator to receive non-sparse rewards with informative learning signals, effectively alleviating the reward sparsity issue; while in the later training stage, SAL can prevent a sample from keeping receiving high reward as the self-improvement for a popular mode will become more and more difficult, and therefore help the generator avoid collapsing toward the limited patterns of real data. We comprehensively evaluate the proposed self-adversarial learning paradigm in both synthetic data and real data on the text generation benchmark platform (Zhu et al., 2018) . Compared to the previous approaches for adversarial text generation (Yu et al., 2017; Che et al., 2017; Lin et al., 2017) , our approach shows a substantial improvement in terms of both the quality and the diversity of generated samples as well as better performance stability in adversarial learning. Figure 1: (a) Conventional adversarial learning that uses a binary real/fake classifier as its discriminator; (b): Self-adversarial learning that employs a comparative discriminator to compare the currently generated sample to its previously generated samples for obtaining rewards through self-improvement. To better understand SAL, we perform multiple ablation tests in both the synthetic and the real data. We employ NLL oracle + NLL gen score with sequence length 20 as the evaluation metric for the synthetic data, denoted as NLL. For the real data, we use the perplexity of generated samples trained with COCO dataset as the evaluation metric. We compare SAL with the following reduced models: • CAL: Replacing the comparison between the generated samples (i.e., self-play) to the comparison between the real and generated samples. • w/o comparative: Using the binary discrimination scores of other generated samples as baseline for the policy gradient algorithm, which can be considered as a combination of the self-critic training (Rennie et al., 2017) with RL-based text GANs. • w/o "≈": Replace the three-class comparative discriminator with a binary comparative discriminator by removing the "≈" class. • w/o scheduled rewarding and w/o memory replay The results of the ablation tests are shown in Table 6 . By observing the improvement by SAL over CAL, we confirm the importance of the self-play paradigm in SAL. It is notable that the proposed comparative discriminator alone (i.e., CAL) can yield good performance, demonstrating the effectiveness of learning by comparison. When replacing the comparative discriminator with the naive combination of self-critic baseline with text GANs, the performance largely decreases because the reward sparsity issue will be intensified when subtracting two already sparse rewards, this motivates the proposed pairwise comparative discriminator which makes self-comparison possible. In addition, we find that the "≈" option plays a critical role in improving the result, without which the performance degrades significantly because it makes the task less trivial and provides a baseline for the policy gradient algorithm. Moreover, the training techniques (i.e., scheduled rewarding and memory replay) borrowed from deep reinforcement learning are also shown useful in improving the results but not so important as the core components (e.g., self-play and the comparative discriminator). (Chen et al., 2018) , LeakGAN (Guo et al., 2018) , and RelGAN (Nie et al., 2018)) have been proposed for text generation as adversarial training has received increasing attention in recent years. Typically, they address the non-differentiable issue by making continuous approximation or reinforcement learning. These approaches introduce several different architectures and optimization objectives of both the generator and the discriminator for adversarial text generation. Among the previous studies for adversarial text generation, the most related work to ours is RankGAN (Lin et al., 2017) which proposes a ranker to replace the conventional binary classifier as its discriminator for allowing the discrimination process to involve richer information. Another work whose idea is similar to ours is the relativistic discriminator (Jolicoeur-Martineau, 2018) (RGAN). It compares binary scores assigned to generated samples and real samples by subtraction as the learning signal to implicitly represent the inductive bias that half of the samples received by the discriminator is fake. In contrast, our comparative discriminator directly encodes this inductive bias and assesses generated sentences by comparison with a pairwise classifier, which provides more informative learning signals than subtraction in RGAN (Jolicoeur-Martineau, 2018) and normalized feature similarity in RankGAN (Lin et al., 2017) . We present a self-adversarial learning (SAL) paradigm for adversarial text generation. SAL rewards the generator when its comparative discriminator finds the generator becomes better than before. Through the self-improvement reward mechanism, the problem of reward sparsity and mode collapse can be alleviated and training of text GANs are more stable, which results in a better performance in the text generation benchmarks in terms of both quality, diversity, and lower variance. In the future, we plan to generalize our approach to other domains and modals to explore the potential of SAL for adversarial learning. Generated samples are presented in the Appendix together with other details, including human evaluation details and qualitative analysis of the proposed SAL. A GENERATED SAMPLES We present sentences generated by our proposed model and compared models to provide qualitative evaluation of different adversarial text generation models. From the presented generated samples, we can observe that samples generated by MLE training are less realistic compared with other samples. SeqGAN yield slightly better sample quality but the loss of diversity is observable even within randomly sampled 15 sentences. Adversarial training with proposed comparator, when trained by comparing with real samples, yield better quality but still lack of diversity. Finally, with the proposed self-adversarial learning paradigm, both quality and diversity of generated samples are improved. A.1 GENERATED SAMPLES IN IMAGE COCO DATASET Table 7 : Samples generated by SAL in Image COCO dataset a picture of a person 's umbrella in a cell phone . a man stands in a green field . a young boy riding a truck . a man on a motorcycle is flying on a grassy field . a girl on a motorcycle parked on a city street . a motorcycle parked in a city street . a group of bikers riding bikes on a city street . a kitchen with a cat on the hood and a street . a bathroom containing a toilet and a sink . a young woman in a kitchen with a smiley face . a jet plane on the side of a street . a dish is sitting on a sidewalk next to a baby giraffe . a dog on a large green bike parked outside of the motor bike . a person on a kawasaki bike on a race track . a commercial aircraft is parked in front of a kitchen . Table 8 : Samples generated by CAL in Image COCO dataset a man is on a towel on a table outside of a real kitchen . a group of lambs at a tall building . a young boy riding a truck . a man on a motorcycle is flying on a grassy field . a man with a computer desk next to a white car . a cat is on the walls of a cat . a plane on a runway with a plane . an elegant , dilapidated plane are standing in front of a parking bag . the woman is riding a bike on their way . a man wearing an old bathroom with a banana . a plane is taking off from the ground . a man holding a man in front of herself . a woman is walking across the road . a kitchen with an island in green tiles . a clean kitchen with two small appliances . Table 9 : Samples generated by SeqGAN in Image COCO dataset a large image of a herd of racing train . man and woman on horse . a plane on a runway with a plane . a man preparing a table with wood lid . a view , tiled floors and a man prepares food . a man wearing an old bathroom with a banana . a man is is with a camera . two people are parked on a street . a white and white black kitten eating on a table . a toilet is lit on the walls . a kitchen is taking off from a window . a man is wearing glasses wearing scarf . a kitchen with graffiti hanging off from an open plain . two women playing with the orange . a kitchen with an island in a clear glass . Table 10 : Samples generated by MLE in Image COCO dataset a jet airplane flies flying through front from an airplane . a furry tub and overhead pot . a man in a kitchen filled with dark lights green side , .. a cross baby field dressed making cardboard a bathroom with a small tub and oven . a man above a bathroom with an oven room . a jet airliner flying through the sky . a kitchen with a dishwasher , and plenty of pots , pans . a person holding onto two red era arena sits on the street . a bathroom with a toilet and a bath tub . a cat perched on the phone and a baseball cap . the view of the street filled with really parked at the gates on the road . a large hairy dog on a high bike with a cake . a man is riding a white back bench . a narrow bed and white spotted dark tiled walls . A.2 GENERATED SAMPLES IN EMNLP2017 WMT DATASET Table 11 : Samples generated by SAL in EMNLP2017 WMT dataset (1) it ' s likely to be egyptian and many of the canadian refugees , but for a decade . (2) the ministry spokesperson also said it now significant connected to the mountain. (3) it is the time they can more competitive , where we have another $ 99 . 100 per cent , and completely on the alternative , and that ' s being affected . (4) we expect $ 200 and 0 . 3 percent for all you form other , and , which then well , it ' s done . (5) so we wouldn ' t feel very large in the game , but you fail to fund , and and the paper that ' s like its start . (6) other countries made a playoff cut with pages by mrs . trump ' s eighth consecutive season as a president . Table 12 : Samples generated by CAL in EMNLP2017 WMT dataset (1) i didn ' t put relatively quiet , we have , ' his work right in the particular heat rate , take steps traditionally clean . (2) why the u . s . then the table is our cabinet to do getting an vital company for the correct review . (3) those had trained for that , but no thin percentage of the nhs about being warned about the palestinian election before obama is not connected in israel . (4) in course , voters -obama said : " torture is the outcome , the most powerful tradepopularity is happening in it as a success . (5) " in 2012 , it is nice to remain -no trump actor established this night -scoring three films . (6) we kind of not listen to knowing my most one , only , for a really good vote , and where things fun , you know . Table 13 : Samples generated by SeqGAN in EMNLP2017 WMT dataset (1) his missed 4 , 000 the first 95 really 69 -year -olds . (2) but just things , you want to thank it as my playing side has begun meeting with " and " the score had to train up , so he was tied for 11 years . (3) and when he got back doing fresh ties with his election , he will now step in january , back. (4) when you ' t know if i saw her task to find himself more responsibility ago . (5) his hold over -up to a nine hike in 2015 , 13 percent of recently under suspects dead day , 24 , and to the city . (6) " i look up on by the city ' s vehicle on the day in a meeting in november . Table 14 : Samples generated by MLE in EMNLP2017 WMT dataset (1) you know that that is great for our ability to make thinking about how you know and you ? (2) when it ' s a real thing possible , is if you the first time in a time here and get . (3) u . s , now government spending at the second half of four years , a country where the law will join the region to leave japan in germany . (4) deputy president , the issue of government and geneva probe threats and not -backed trump , but well -changing violence for their islamic state militants were innocent people . (5) he suggested in a presidential primary source and comment on its size following protests conducted by 18 , some in 2012 will be looked at tech energy hub . (6) " it ' s growing heavy hard , " mr . romney said , he says matters that can ' t again become the asian player .
We propose a self-adversarial learning (SAL) paradigm which improves the generator in a self-play fashion for improving GANs' performance in text generation.
{"src": "scitldr-AIC_train", "paper_id": "B1l8L6EtDS"}