MLE, SeqGAN, LeakGAN, MaliGAN, IRL, RAML, DialogGAN, DPGAN
EXTRACTIVE
SeqGAN, LeakGAN, MaliGAN, DialogGAN, DPGAN
EXTRACTIVE
We compared our model with MLE, RL and GAN baselines. Since COCO and EMNLP2017 WMT don't have input while WeiboDial regards posts as input, we chose the following baselines respectively: MLE: a RNN model trained with MLE objective BIBREF4 . Its extension, Seq2Seq, can work on the dialogue dataset BIBREF2 . SeqGAN: The first text GAN model that updates the generator with policy gradient based on the rewards from the discriminator BIBREF7 . LeakGAN: A variant of SeqGAN that provides rewards based on the leaked information of the discriminator for the generator BIBREF11 . MaliGAN: A variant of SeqGAN that optimizes the generator with a normalized maximum likelihood objective BIBREF8 . IRL: This inverse reinforcement learning method replaces the discriminator with a reward approximator to provide dense rewards BIBREF12 . RAML: A RL approach to incorporate MLE objective into RL training framework, which regards BLEU as rewards BIBREF17 . DialogGAN: An extension of SeqGAN tuned to dialogue generation task with MLE objective added to the adversarial objective BIBREF16 . DPGAN: A variant of DialogGAN which uses a language model based discriminator and regards cross-entropy as rewards BIBREF13 .
We compared our model with MLE, RL and GAN baselines. Since COCO and EMNLP2017 WMT don't have input while WeiboDial regards posts as input, we chose the following baselines respectively: SeqGAN: The first text GAN model that updates the generator with policy gradient based on the rewards from the discriminator BIBREF7 . LeakGAN: A variant of SeqGAN that provides rewards based on the leaked information of the discriminator for the generator BIBREF11 . MaliGAN: A variant of SeqGAN that optimizes the generator with a normalized maximum likelihood objective BIBREF8 . DialogGAN: An extension of SeqGAN tuned to dialogue generation task with MLE objective added to the adversarial objective BIBREF16 . DPGAN: A variant of DialogGAN which uses a language model based discriminator and regards cross-entropy as rewards BIBREF13 .
1908.07195
Introduction </s> Natural Ġtext Ġgeneration , Ġas Ġa Ġkey Ġtask Ġin ĠN LP , Ġhas Ġbeen Ġadvanced Ġsubstantially Ġthanks Ġto Ġthe Ġflourish Ġof Ġneural Ġmodels ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ. ĠTypical Ġframeworks Ġsuch Ġas Ġsequence - to - sequence Ġ( seq 2 seq ) Ġhave Ġbeen Ġapplied Ġto Ġvarious Ġgeneration Ġtasks , Ġincluding Ġmachine Ġtranslation ĠB IB REF 2 Ġand Ġdialogue Ġgeneration ĠB IB REF 3 Ġ. ĠThe Ġstandard Ġparadigm Ġto Ġtrain Ġsuch Ġneural Ġmodels Ġis Ġmaximum Ġlikelihood Ġestimation Ġ( M LE ), Ġwhich Ġmaxim izes Ġthe Ġlog - like lihood Ġof Ġobserving Ġeach Ġword Ġin Ġthe Ġtext Ġgiven Ġthe Ġground - truth Ġproceeding Ġcontext ĠB IB REF 4 Ġ. </s> Although Ġwidely Ġused , ĠM LE Ġsuffers Ġfrom Ġthe Ġexposure Ġbias Ġproblem ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ: Ġduring Ġtest , Ġthe Ġmodel Ġsequ entially Ġpredicts Ġthe Ġnext Ġword Ġconditioned Ġon Ġits Ġprevious Ġgenerated Ġwords Ġwhile Ġduring Ġtraining Ġconditioned Ġon Ġground - truth Ġwords . ĠTo Ġtackle Ġthis Ġproblem , Ġgener ative Ġadvers arial Ġnetworks Ġ( GAN ) Ġwith Ġreinforcement Ġlearning Ġ( RL ) Ġtraining Ġapproaches Ġhave Ġbeen Ġintroduced Ġto Ġtext Ġgeneration Ġtasks ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ, ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ, Ġwhere Ġthe Ġdiscrim inator Ġis Ġtrained Ġto Ġdistinguish Ġreal Ġand Ġgenerated Ġtext Ġsamples Ġto Ġprovide Ġreward Ġsignals Ġfor Ġthe Ġgenerator , Ġand Ġthe Ġgenerator Ġis Ġoptimized Ġvia Ġpolicy Ġgradient ĠB IB REF 7 Ġ. </s> However , Ġrecent Ġstudies Ġhave Ġshown Ġthat Ġpotential Ġissues Ġof Ġtraining ĠG AN s Ġon Ġdiscrete Ġdata Ġare Ġmore Ġsevere Ġthan Ġexposure Ġbias ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ. ĠOne Ġof Ġthe Ġfundamental Ġissues Ġwhen Ġgenerating Ġdiscrete Ġtext Ġsamples Ġwith ĠG AN s Ġis Ġtraining Ġinstability . ĠUp dating Ġthe Ġgenerator Ġwith Ġpolicy Ġgradient Ġalways Ġleads Ġto Ġan Ġunstable Ġtraining Ġprocess Ġbecause Ġit 's Ġdifficult Ġfor Ġthe Ġgenerator Ġto Ġderive Ġpositive Ġand Ġstable Ġreward Ġsignals Ġfrom Ġthe Ġdiscrim inator Ġeven Ġwith Ġcareful Ġpre - training ĠB IB REF 8 Ġ. ĠAs Ġa Ġresult , Ġthe Ġgenerator Ġgets Ġlost Ġdue Ġto Ġthe Ġhigh Ġvariance Ġof Ġreward Ġsignals Ġand Ġthe Ġtraining Ġprocess Ġmay Ġfinally Ġcollapse ĠB IB REF 16 Ġ. </s> In Ġthis Ġpaper , Ġwe Ġpropose Ġa Ġnovel Ġadvers arial Ġtraining Ġframework Ġcalled ĠAd vers arial ĠReward ĠAug mented ĠMaximum ĠLike lihood Ġ( AR AM L ) Ġto Ġdeal Ġwith Ġthe Ġinstability Ġissue Ġof Ġtraining ĠG AN s Ġfor Ġtext Ġgeneration . ĠAt Ġeach Ġiteration Ġof Ġadvers arial Ġtraining , Ġwe Ġfirst Ġtrain Ġthe Ġdiscrim inator Ġto Ġassign Ġhigher Ġrewards Ġto Ġreal Ġdata Ġthan Ġto Ġgenerated Ġsamples . ĠThen , Ġinspired Ġby Ġreward Ġaugmented Ġmaximum Ġlikelihood Ġ( RAM L ) ĠB IB REF 17 Ġ, Ġthe Ġgenerator Ġis Ġupdated Ġon Ġthe Ġsamples Ġacquired Ġfrom Ġa Ġstationary Ġdistribution Ġwith Ġmaximum Ġlikelihood Ġestimation Ġ( M LE ), Ġweighted Ġby Ġthe Ġdiscrim inator 's Ġrewards . ĠThis Ġstationary Ġdistribution Ġis Ġdesigned Ġto Ġguarantee Ġthat Ġtraining Ġsamples Ġare Ġsurrounding Ġthe Ġreal Ġdata , Ġthus Ġthe Ġexploration Ġspace Ġof Ġour Ġgenerator Ġis Ġindeed Ġrestricted Ġby Ġthe ĠM LE Ġtraining Ġobjective , Ġresulting Ġin Ġmore Ġstable Ġtraining . ĠCompared Ġto Ġother Ġtext ĠG AN s Ġwith ĠRL Ġtraining Ġtechniques , Ġour Ġframework Ġacqu ires Ġsamples Ġfrom Ġthe Ġstationary Ġdistribution Ġrather Ġthan Ġthe Ġgenerator 's Ġdistribution , Ġand Ġuses ĠRAM L Ġtraining Ġparadigm Ġto Ġoptimize Ġthe Ġgenerator Ġinstead Ġof Ġpolicy Ġgradient . ĠOur Ġcontributions Ġare Ġmainly Ġas Ġfollows : </s> Related ĠWork </s> Recently , Ġtext Ġgeneration Ġhas Ġbeen Ġwidely Ġstudied Ġwith Ġneural Ġmodels Ġtrained Ġwith Ġmaximum Ġlikelihood Ġestimation ĠB IB REF 4 Ġ. ĠHowever , ĠM LE Ġtends Ġto Ġgenerate Ġuniversal Ġtext ĠB IB REF 18 Ġ. ĠVarious Ġmethods Ġhave Ġbeen Ġproposed Ġto Ġenhance Ġthe Ġgeneration Ġquality Ġby Ġrefining Ġthe Ġobjective Ġfunction ĠB IB REF 18 Ġ, ĠB IB REF 19 Ġor Ġmodifying Ġthe Ġgeneration Ġdistribution Ġwith Ġexternal Ġinformation Ġlike Ġtopic ĠB IB REF 20 Ġ, Ġsentence Ġtype ĠB IB REF 21 Ġ, Ġemotion ĠB IB REF 22 Ġand Ġknowledge ĠB IB REF 23 Ġ. </s> As Ġmentioned Ġabove , ĠM LE Ġsuffers Ġfrom Ġthe Ġexposure Ġbias Ġproblem ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ. ĠThus , Ġreinforcement Ġlearning Ġhas Ġbeen Ġintroduced Ġto Ġtext Ġgeneration Ġtasks Ġsuch Ġas Ġpolicy Ġgradient ĠB IB REF 6 Ġand Ġactor - crit ic ĠB IB REF 24 Ġ. ĠB IB REF 17 Ġproposed Ġan Ġefficient Ġand Ġstable Ġapproach Ġcalled ĠReward ĠAug mented ĠMaximum ĠLike lihood Ġ( RAM L ), Ġwhich Ġconnects Ġthe Ġlog - like lihood Ġand Ġexpected Ġrewards Ġto Ġincorporate ĠM LE Ġtraining Ġobjective Ġinto ĠRL Ġframework . </s> Since Ġsome Ġtext Ġgeneration Ġtasks Ġhave Ġno Ġexplicit Ġmetrics Ġto Ġbe Ġdirectly Ġoptimized , Ġadvers arial Ġtraining Ġhas Ġbeen Ġapplied Ġto Ġgenerating Ġdiscrete Ġtext Ġsamples Ġwith Ġa Ġdiscrim inator Ġto Ġlearn Ġa Ġproper Ġreward . ĠFor Ġinstance , ĠSe q GAN ĠB IB REF 7 Ġdevised Ġa Ġdiscrim inator Ġto Ġdistinguish Ġthe Ġreal Ġdata Ġand Ġgenerated Ġsamples , Ġand Ġa Ġgenerator Ġto Ġmaximize Ġthe Ġreward Ġfrom Ġthe Ġdiscrim inator Ġvia Ġpolicy Ġgradient . ĠOther Ġvariants Ġof ĠG AN s Ġhave Ġbeen Ġproposed Ġto Ġimprove Ġthe Ġgenerator Ġor Ġthe Ġdiscrim inator . ĠTo Ġimprove Ġthe Ġgenerator , ĠMali GAN ĠB IB REF 8 Ġdeveloped Ġa Ġnormalized Ġmaximum Ġlikelihood Ġoptimization Ġtarget Ġfor Ġthe Ġgenerator Ġto Ġst ably Ġmodel Ġthe Ġdiscrete Ġsequences . ĠLe ak GAN ĠB IB REF 11 Ġguided Ġthe Ġgenerator Ġwith Ġreward Ġsignals Ġleaked Ġfrom Ġthe Ġdiscrim inator Ġat Ġall Ġgeneration Ġsteps Ġto Ġdeal Ġwith Ġlong Ġtext Ġgeneration Ġtask . ĠMask GAN ĠB IB REF 10 Ġemployed Ġan Ġactor - crit ic Ġarchitecture Ġto Ġmake Ġthe Ġgenerator Ġfill Ġin Ġmissing Ġtext Ġconditioned Ġon Ġthe Ġsurrounding Ġcontext , Ġwhich Ġis Ġexpected Ġto Ġmitigate Ġthe Ġproblem Ġof Ġmode Ġcollapse . ĠAs Ġfor Ġthe Ġdiscrim inator , ĠRank GAN ĠB IB REF 9 Ġreplaced Ġtraditional Ġdiscrim inator Ġwith Ġa Ġrank er Ġto Ġlearn Ġthe Ġrelative Ġranking Ġinformation Ġbetween Ġthe Ġreal Ġtexts Ġand Ġgenerated Ġones . ĠIn verse Ġreinforcement Ġlearning ĠB IB REF 12 Ġused Ġa Ġtrain able Ġreward Ġapprox im ator Ġas Ġthe Ġdiscrim inator Ġto Ġprovide Ġdense Ġreward Ġsignals Ġat Ġeach Ġgeneration Ġstep . ĠD PG AN ĠB IB REF 13 Ġintroduced Ġa Ġlanguage Ġmodel Ġbased Ġdiscrim inator Ġand Ġregarded Ġcross - ent ropy Ġas Ġrewards Ġto Ġpromote Ġthe Ġdiversity Ġof Ġgeneration Ġresults . </s> The Ġmost Ġsimilar Ġworks Ġto Ġour Ġmodel Ġare ĠRAM L ĠB IB REF 17 Ġand ĠMali GAN ĠB IB REF 8 Ġ: Ġ1 ) ĠCompared Ġwith ĠRAM L , Ġour Ġmodel Ġadds Ġa Ġdiscrim inator Ġto Ġlearn Ġthe Ġreward Ġsignals Ġinstead Ġof Ġchoosing Ġexisting Ġmetrics Ġas Ġrewards . ĠWe Ġbelieve Ġthat Ġour Ġmodel Ġcan Ġadapt Ġto Ġvarious Ġtext Ġgeneration Ġtasks , Ġparticularly Ġthose Ġwithout Ġexplicit Ġevaluation Ġmetrics . Ġ2 ) ĠUnlike ĠMali GAN , Ġwe Ġacquire Ġsamples Ġfrom Ġa Ġfixed Ġdistribution Ġnear Ġthe Ġreal Ġdata Ġrather Ġthan Ġthe Ġgenerator 's Ġdistribution , Ġwhich Ġis Ġexpected Ġto Ġmake Ġthe Ġtraining Ġprocess Ġmore Ġstable . </s> Task ĠDefinition Ġand ĠModel ĠOverview </s> Text Ġgeneration Ġcan Ġbe Ġformulated Ġas Ġfollows : Ġgiven Ġthe Ġreal Ġdata Ġdistribution ĠIN LINE FORM 0 Ġ, Ġthe Ġtask Ġis Ġto Ġtrain Ġa Ġgener ative Ġmodel ĠIN LINE FORM 1 Ġwhere ĠIN LINE FORM 2 Ġcan Ġfit ĠIN LINE FORM 3 Ġwell . ĠIn Ġthis Ġformulation , ĠIN LINE FORM 4 Ġand ĠIN LINE FORM 5 Ġdenotes Ġa Ġword Ġin Ġthe Ġvocabulary ĠIN LINE FORM 6 Ġ. </s> Figure ĠFIG REF 3 Ġshows Ġthe Ġoverview Ġof Ġour Ġmodel ĠAR AM L . ĠThis Ġadvers arial Ġtraining Ġframework Ġconsists Ġof Ġtwo Ġphases : Ġ1 ) ĠThe Ġdiscrim inator Ġis Ġtrained Ġto Ġassign Ġhigher Ġrewards Ġto Ġreal Ġdata Ġthan Ġto Ġgenerated Ġdata . Ġ2 ) ĠThe Ġgenerator Ġis Ġtrained Ġon Ġthe Ġsamples Ġacquired Ġfrom Ġa Ġstationary Ġdistribution Ġwith Ġreward Ġaugmented ĠM LE Ġtraining Ġobjective . ĠThis Ġtraining Ġparadigm Ġof Ġthe Ġgenerator Ġindeed Ġconst r ains Ġthe Ġsearch Ġspace Ġwith Ġthe ĠM LE Ġtraining Ġobjective , Ġwhich Ġallev iates Ġthe Ġissue Ġof Ġunstable Ġtraining . </s> Disc rim inator </s> The Ġdiscrim inator ĠIN LINE FORM 0 Ġaims Ġto Ġdistinguish Ġreal Ġdata Ġand Ġgenerated Ġdata Ġlike Ġother ĠG AN s . ĠInspired Ġby ĠLe ast - Square ĠG AN ĠB IB REF 25 Ġ, Ġwe Ġdevise Ġthe Ġloss Ġfunction Ġas Ġfollows : ĠDIS PLAY FORM 0 </s> This Ġloss Ġfunction Ġforces Ġthe Ġdiscrim inator Ġto Ġassign Ġhigher Ġrewards Ġto Ġreal Ġdata Ġthan Ġto Ġgenerated Ġdata , Ġso Ġthe Ġdiscrim inator Ġcan Ġlearn Ġto Ġprovide Ġmore Ġproper Ġrewards Ġas Ġthe Ġtraining Ġproceeds . </s> Gener ator </s> The Ġtraining Ġobjective Ġof Ġour Ġgenerator ĠIN LINE FORM 0 Ġis Ġderived Ġfrom Ġthe Ġobjective Ġof Ġother Ġdiscrete ĠG AN s Ġwith ĠRL Ġtraining Ġmethod : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġdenotes Ġthe Ġrewards Ġfrom Ġthe Ġdiscrim inator ĠIN LINE FORM 1 Ġand Ġthe Ġentropy Ġregular ized Ġterm ĠIN LINE FORM 2 Ġencourages ĠIN LINE FORM 3 Ġto Ġgenerate Ġdiverse Ġtext Ġsamples . ĠIN LINE FORM 4 Ġis Ġa Ġtemperature Ġhyper - param eter Ġto Ġbalance Ġthese Ġtwo Ġterms . </s> As Ġmentioned Ġabove , Ġdiscrete ĠG AN s Ġsuffer Ġfrom Ġthe Ġinstability Ġissue Ġdue Ġto Ġpolicy Ġgradient , Ġthus Ġthey Ġare Ġconsequently Ġdifficult Ġto Ġtrain . ĠInspired Ġby ĠRAM L ĠB IB REF 17 Ġ, Ġwe Ġintroduce Ġan Ġexponential Ġpayoff Ġdistribution ĠIN LINE FORM 0 Ġto Ġconnect ĠRL Ġloss Ġwith ĠRAM L Ġloss : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġ. ĠThus , Ġwe Ġcan Ġrewrite ĠIN LINE FORM 1 Ġwith ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġas Ġfollows : ĠDIS PLAY FORM 0 </s> Following ĠRAM L , Ġwe Ġremove Ġthe Ġconstant Ġterm Ġand Ġoptimize Ġthe ĠKL Ġdivergence Ġin Ġthe Ġopposite Ġdirection : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġa Ġconstant Ġin Ġthe Ġtraining Ġphase Ġof Ġthe Ġgenerator . ĠIt Ġhas Ġbeen Ġproved Ġthat ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġare Ġequivalent Ġup Ġto Ġtheir Ġfirst Ġorder ĠTaylor Ġapprox im ations , Ġand Ġthey Ġhave Ġthe Ġsame Ġglobal Ġoptimum ĠB IB REF 17 Ġ. ĠIN LINE FORM 3 Ġcan Ġbe Ġtrained Ġin Ġa ĠM LE - like Ġfashion Ġbut Ġsampling Ġfrom Ġthe Ġdistribution ĠIN LINE FORM 4 Ġis Ġint ract able Ġin Ġthe Ġadvers arial Ġsetting , Ġbecause ĠIN LINE FORM 5 Ġvaries Ġwith Ġthe Ġdiscrim inator ĠIN LINE FORM 6 Ġ. ĠThus , Ġwe Ġintroduce Ġimportance Ġsampling Ġto Ġseparate Ġsampling Ġprocess Ġfrom ĠIN LINE FORM 7 Ġand Ġobtain Ġthe Ġfinal Ġloss Ġfunction : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġdenotes Ġa Ġstationary Ġdistribution Ġand ĠIN LINE FORM 1 Ġ. ĠTo Ġoptimize Ġthis Ġloss Ġfunction , Ġwe Ġfirst Ġconstruct Ġthe Ġfixed Ġdistribution ĠIN LINE FORM 2 Ġto Ġget Ġsamples , Ġand Ġdevise Ġthe Ġproper Ġreward Ġfunction ĠIN LINE FORM 3 Ġto Ġtrain Ġthe Ġgenerator Ġin Ġa Ġstable Ġand Ġeffective Ġway . </s> We Ġconstruct Ġthe Ġdistribution ĠIN LINE FORM 0 Ġbased Ġon ĠIN LINE FORM 1 Ġ: ĠDIS PLAY FORM 0 </s> In Ġthis Ġway , ĠIN LINE FORM 0 Ġcan Ġbe Ġdesigned Ġto Ġguarantee Ġthat ĠIN LINE FORM 1 Ġis Ġnear ĠIN LINE FORM 2 Ġ, Ġleading Ġto Ġa Ġmore Ġstable Ġtraining Ġprocess . ĠTo Ġobtain Ġa Ġnew Ġsample ĠIN LINE FORM 3 Ġfrom Ġa Ġreal Ġdata Ġsample ĠIN LINE FORM 4 Ġ, Ġwe Ġcan Ġdesign Ġthree Ġsteps Ġwhich Ġcontain Ġsampling Ġan Ġedit Ġdistance ĠIN LINE FORM 5 Ġ, Ġthe Ġpositions ĠIN LINE FORM 6 Ġfor Ġsubstitution Ġand Ġthe Ġnew Ġwords ĠIN LINE FORM 7 Ġfilled Ġinto Ġthe Ġcorresponding Ġpositions . ĠThus , ĠIN LINE FORM 8 Ġcan Ġbe Ġdecomp osed Ġinto Ġthree Ġterms : ĠDIS PLAY FORM 0 </s> The Ġfirst Ġstep Ġis Ġto Ġsample Ġan Ġedit Ġdistance Ġbased Ġon Ġa Ġreal Ġdata Ġsample ĠIN LINE FORM 0 Ġ, Ġwhere ĠIN LINE FORM 1 Ġis Ġa Ġsequence Ġof Ġlength ĠIN LINE FORM 2 Ġ. ĠThe Ġnumber Ġof Ġsentences Ġwhich Ġhave Ġthe Ġedit Ġdistance ĠIN LINE FORM 3 Ġto Ġsome Ġinput Ġsentence Ġcan Ġbe Ġcomputed Ġapproximately Ġas Ġbelow : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġdenotes Ġthe Ġnumber Ġof Ġsentences Ġwhich Ġhave Ġan Ġedit Ġdistance ĠIN LINE FORM 1 Ġto Ġa Ġsentence Ġof Ġlength ĠIN LINE FORM 2 Ġ, Ġand ĠIN LINE FORM 3 Ġindicates Ġthe Ġsize Ġof Ġvocabulary . ĠWe Ġthen Ġfollow ĠB IB REF 17 Ġto Ġre - scale Ġthe Ġcounts Ġby ĠIN LINE FORM 4 Ġand Ġdo Ġnormal ization , Ġso Ġthat Ġwe Ġcan Ġsample Ġan Ġedit Ġdistance ĠIN LINE FORM 5 Ġfrom : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġ, Ġas Ġa Ġtemperature Ġhyper - param eter , Ġrestricts Ġthe Ġsearch Ġspace Ġsurrounding Ġthe Ġoriginal Ġsentence . ĠL arger ĠIN LINE FORM 1 Ġbrings Ġmore Ġsamples Ġwith Ġlong Ġedit Ġdistances . </s> The Ġnext Ġstep Ġis Ġto Ġselect Ġpositions Ġfor Ġsubstitution Ġbased Ġon Ġthe Ġsampled Ġedit Ġdistance ĠIN LINE FORM 0 Ġ. ĠInt uitive ly , Ġwe Ġcan Ġrandomly Ġchoose ĠIN LINE FORM 1 Ġdistinct Ġpositions Ġin ĠIN LINE FORM 2 Ġto Ġbe Ġreplaced Ġby Ġnew Ġwords . ĠThe Ġprobability Ġof Ġchoosing Ġthe Ġposition ĠIN LINE FORM 3 Ġis Ġcalculated Ġas Ġfollows : ĠDIS PLAY FORM 0 </s> Following Ġthis Ġsampling Ġstrategy , Ġwe Ġcan Ġobtain Ġthe Ġposition Ġset ĠIN LINE FORM 0 Ġ. ĠThis Ġstrategy Ġapproximately Ġguarantees Ġthat Ġthe Ġedit Ġdistance Ġbetween Ġa Ġnew Ġsentence Ġand Ġthe Ġoriginal Ġsentence Ġis ĠIN LINE FORM 1 Ġ. </s> At Ġthe Ġfinal Ġstep , Ġour Ġmodel Ġdetermines Ġnew Ġwords Ġfor Ġsubstitution Ġat Ġeach Ġsampled Ġposition ĠIN LINE FORM 0 Ġ. ĠWe Ġcan Ġformulate Ġthis Ġsampling Ġprocess Ġfrom Ġthe Ġoriginal Ġsequence ĠIN LINE FORM 1 Ġto Ġa Ġnew Ġsample ĠIN LINE FORM 2 Ġas Ġa Ġsequential Ġtransition ĠIN LINE FORM 3 Ġ. ĠAt Ġeach Ġstep Ġfrom ĠIN LINE FORM 4 Ġto ĠIN LINE FORM 5 ĠIN LINE FORM 6 Ġ, Ġwe Ġfirst Ġsample Ġa Ġnew Ġword ĠIN LINE FORM 7 Ġfrom Ġthe Ġdistribution ĠIN LINE FORM 8 Ġ, Ġthen Ġreplace Ġthe Ġold Ġword Ġat Ġposition ĠIN LINE FORM 9 Ġof ĠIN LINE FORM 10 Ġto Ġobtain ĠIN LINE FORM 11 Ġ. ĠThe Ġwhole Ġsampling Ġprocess Ġcan Ġbe Ġdecomp osed Ġas Ġfollows : ĠDIS PLAY FORM 0 </s> There Ġare Ġtwo Ġcommon Ġsampling Ġstrategies Ġto Ġmodel ĠIN LINE FORM 0 Ġ, Ġi . e . Ġrandom Ġsampling Ġand Ġconstrained Ġsampling . ĠRandom Ġsampling Ġstrategy Ġsamples Ġa Ġnew Ġword ĠIN LINE FORM 1 Ġaccording Ġto Ġthe Ġuniform Ġdistribution Ġover Ġthe Ġvocabulary ĠIN LINE FORM 2 ĠB IB REF 17 Ġ, Ġwhile Ġconstrained Ġsampling Ġstrategy Ġsamples ĠIN LINE FORM 3 Ġto Ġmaximize Ġthe Ġlanguage Ġmodel Ġscore Ġof Ġthe Ġtarget Ġsentence ĠIN LINE FORM 4 ĠB IB REF 26 Ġ, ĠB IB REF 27 Ġ. ĠHere , Ġwe Ġadopt Ġconstrained Ġsampling Ġin Ġour Ġmodel Ġand Ġcompare Ġthe Ġperformances Ġof Ġtwo Ġstrategies Ġin Ġthe Ġexperiment . </s> We Ġdevise Ġthe Ġreward Ġfunction ĠIN LINE FORM 0 Ġaccording Ġto Ġthe Ġdiscrim inator 's Ġoutput ĠIN LINE FORM 1 Ġand Ġthe Ġstationary Ġdistribution ĠIN LINE FORM 2 Ġ: ĠDIS PLAY FORM 0 </s> Int uitive ly , Ġthis Ġreward Ġfunction Ġencourages Ġthe Ġgenerator Ġto Ġgenerate Ġsentences Ġwith Ġlarge Ġsampling Ġprobability Ġand Ġhigh Ġrewards Ġfrom Ġthe Ġdiscrim inator . ĠThus , Ġthe Ġweight Ġof Ġsamples ĠIN LINE FORM 0 Ġcan Ġbe Ġcalculated Ġas Ġfollows : ĠDIS PLAY FORM 0 </s> So Ġfar , Ġwe Ġcan Ġsuccessfully Ġoptimize Ġthe Ġgenerator 's Ġloss ĠIN LINE FORM 0 Ġvia ĠEqu ation ĠEQ REF 12 Ġ. ĠThis Ġtraining Ġparadigm Ġmakes Ġour Ġgenerator Ġavoid Ġpossible Ġvar iances Ġcaused Ġby Ġpolicy Ġgradient Ġand Ġget Ġmore Ġstable Ġreward Ġsignals Ġfrom Ġthe Ġdiscrim inator , Ġbecause Ġour Ġgenerator Ġis Ġrestricted Ġto Ġexplore Ġthe Ġtraining Ġsamples Ġnear Ġthe Ġreal Ġdata . </s> [ ht b ] ĠAd vers arial ĠReward ĠAug mented ĠMaximum ĠLike lihood Ġ[ 1 ] </s> Total Ġadvers arial Ġtraining Ġiterations : ĠIN LINE FORM 0 </s> Step s Ġof Ġtraining Ġgenerator : ĠIN LINE FORM 0 </s> Step s Ġof Ġtraining Ġdiscrim inator : ĠIN LINE FORM 0 </s> Pre - train Ġthe Ġgenerator ĠIN LINE FORM 0 Ġwith ĠM LE Ġloss ĠGener ate Ġsamples Ġfrom ĠIN LINE FORM 1 ĠPre - train Ġthe Ġdiscrim inator ĠIN LINE FORM 2 Ġvia ĠE q .( ĠEQ REF 6 Ġ) ĠConstruct ĠIN LINE FORM 3 Ġvia ĠE q .( ĠEQ REF 14 Ġ) Ġ- ĠE q .( ĠEQ REF 19 Ġ) Ġeach ĠIN LINE FORM 4 Ġeach ĠIN LINE FORM 5 ĠUpdate ĠIN LINE FORM 6 Ġvia ĠE q .( ĠEQ REF 12 Ġ) Ġeach ĠIN LINE FORM 7 ĠUpdate ĠIN LINE FORM 8 Ġvia ĠE q .( ĠEQ REF 6 Ġ) </s> Ext ension Ġto ĠCond itional ĠText ĠGeneration </s> We Ġhave Ġshown Ġour Ġadvers arial Ġtraining Ġframework Ġfor Ġtext Ġgeneration Ġtasks Ġwithout Ġan Ġinput . ĠActually , Ġit Ġcan Ġalso Ġbe Ġextended Ġto Ġconditional Ġtext Ġgeneration Ġtasks Ġlike Ġdialogue Ġgeneration . ĠGiven Ġthe Ġdata Ġdistribution ĠIN LINE FORM 0 Ġwhere ĠIN LINE FORM 1 Ġdenote Ġcontexts Ġand Ġresponses Ġrespectively , Ġthe Ġobjective Ġfunction Ġof ĠAR AM L 's Ġgenerator Ġcan Ġbe Ġmodified Ġas Ġbelow : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġis Ġtrained Ġto Ġdistinguish Ġwhether ĠIN LINE FORM 2 Ġis Ġthe Ġtrue Ġresponse Ġto ĠIN LINE FORM 3 Ġ. </s> Compar ison Ġwith ĠRAM L Ġand ĠMali GAN </s> The Ġmost Ġsimilar Ġworks Ġto Ġour Ġframework Ġare ĠRAM L ĠB IB REF 17 Ġand ĠMali GAN ĠB IB REF 8 Ġ. ĠThe Ġmain Ġdifference Ġamong Ġthem Ġis Ġthe Ġtraining Ġobjective Ġof Ġtheir Ġgenerators . ĠWe Ġhave Ġshown Ġdifferent Ġobjective Ġfunctions Ġin ĠTable ĠT AB REF 26 Ġ. ĠFor Ġcomparison , Ġwe Ġuse Ġthe Ġform Ġwith Ġno Ġinput Ġfor Ġall Ġthe Ġthree Ġmodels . </s> Our Ġmodel Ġis Ġgreatly Ġinspired Ġby ĠRAM L , Ġwhich Ġgets Ġsamples Ġfrom Ġa Ġnon - param etric Ġdistribution ĠIN LINE FORM 0 Ġconstructed Ġbased Ġon Ġa Ġspecific Ġreward . ĠCompared Ġto ĠRAM L , Ġour Ġreward Ġcomes Ġfrom Ġa Ġlearn able Ġdiscrim inator Ġwhich Ġvaries Ġas Ġthe Ġadvers arial Ġtraining Ġproceeds Ġrather Ġthan Ġa Ġspecific Ġreward Ġfunction . ĠThis Ġdifference Ġequ ips Ġour Ġframework Ġwith Ġthe Ġability Ġto Ġadapt Ġto Ġthe Ġtext Ġgeneration Ġtasks Ġwith Ġno Ġexplicit Ġevaluation Ġmetrics Ġas Ġrewards . </s> Our Ġmodel Ġis Ġalso Ġsimilar Ġto ĠMali GAN , Ġwhich Ġgets Ġsamples Ġfrom Ġthe Ġgenerator 's Ġdistribution . ĠIn ĠMali GAN 's Ġtraining Ġobjective , ĠIN LINE FORM 0 Ġalso Ġindicates Ġthe Ġgenerator 's Ġdistribution Ġbut Ġit 's Ġused Ġin Ġthe Ġsampling Ġphase Ġand Ġfixed Ġat Ġeach Ġoptimization Ġstep . ĠThe Ġweight Ġof Ġsamples ĠIN LINE FORM 1 Ġ. ĠDifferent Ġfrom Ġour Ġmodel , ĠMali GAN Ġacqu ires Ġsamples Ġfrom Ġthe Ġgenerator 's Ġdistribution ĠIN LINE FORM 2 Ġ, Ġwhich Ġusually Ġbrings Ġsamples Ġwith Ġlow Ġrewards Ġeven Ġwith Ġcareful Ġpre - training Ġfor Ġthe Ġgenerator , Ġleading Ġto Ġtraining Ġinstability . ĠInstead , Ġour Ġframework Ġgets Ġsamples Ġfrom Ġa Ġstationary Ġdistribution ĠIN LINE FORM 3 Ġaround Ġreal Ġdata , Ġthus Ġour Ġtraining Ġprocess Ġis Ġmore Ġstable . </s> Dat as ets </s> We Ġevaluated ĠAR AM L Ġon Ġthree Ġdatasets : ĠC OC O Ġimage Ġcaption Ġdataset ĠB IB REF 28 Ġ, ĠEM N LP 2017 ĠW MT Ġdataset Ġand ĠWe ibo Dial Ġsingle - turn Ġdialogue Ġdataset ĠB IB REF 29 Ġ. ĠC OC O Ġand ĠEM N LP 2017 ĠW MT Ġare Ġthe Ġcommon Ġbenchmarks Ġwith Ġno Ġinput Ġto Ġevaluate Ġthe Ġperformance Ġof Ġdiscrete ĠG AN s , Ġand Ġwe Ġfollowed Ġthe Ġexisting Ġworks Ġto Ġpre process Ġthese Ġdatasets ĠB IB REF 12 Ġ, ĠB IB REF 11 Ġ. ĠWe ibo Dial , Ġas Ġa Ġdialogue Ġdataset , Ġwas Ġapplied Ġto Ġtest Ġthe Ġperformance Ġof Ġour Ġmodel Ġwith Ġinput Ġtrigger . ĠWe Ġsimply Ġremoved Ġpost - response Ġpairs Ġcontaining Ġlow - frequency Ġwords Ġand Ġrandomly Ġselected Ġa Ġsubset Ġfor Ġour Ġtraining / test Ġset . ĠThe Ġstatistics Ġof Ġthree Ġdatasets Ġare Ġpresented Ġin ĠTable ĠT AB REF 28 Ġ. </s> Bas elines </s> We Ġcompared Ġour Ġmodel Ġwith ĠM LE , ĠRL Ġand ĠG AN Ġbas elines . ĠSince ĠC OC O Ġand ĠEM N LP 2017 ĠW MT Ġdon 't Ġhave Ġinput Ġwhile ĠWe ibo Dial Ġregards Ġposts Ġas Ġinput , Ġwe Ġchose Ġthe Ġfollowing Ġbas elines Ġrespectively : </s> M LE : Ġa ĠR NN Ġmodel Ġtrained Ġwith ĠM LE Ġobjective ĠB IB REF 4 Ġ. ĠIts Ġextension , ĠSe q 2 Se q , Ġcan Ġwork Ġon Ġthe Ġdialogue Ġdataset ĠB IB REF 2 Ġ. </s> Se q GAN : ĠThe Ġfirst Ġtext ĠG AN Ġmodel Ġthat Ġupdates Ġthe Ġgenerator Ġwith Ġpolicy Ġgradient Ġbased Ġon Ġthe Ġrewards Ġfrom Ġthe Ġdiscrim inator ĠB IB REF 7 Ġ. </s> Le ak GAN : ĠA Ġvariant Ġof ĠSe q GAN Ġthat Ġprovides Ġrewards Ġbased Ġon Ġthe Ġleaked Ġinformation Ġof Ġthe Ġdiscrim inator Ġfor Ġthe Ġgenerator ĠB IB REF 11 Ġ. </s> M ali GAN : ĠA Ġvariant Ġof ĠSe q GAN Ġthat Ġoptim izes Ġthe Ġgenerator Ġwith Ġa Ġnormalized Ġmaximum Ġlikelihood Ġobjective ĠB IB REF 8 Ġ. </s> IR L : ĠThis Ġinverse Ġreinforcement Ġlearning Ġmethod Ġreplaces Ġthe Ġdiscrim inator Ġwith Ġa Ġreward Ġapprox im ator Ġto Ġprovide Ġdense Ġrewards ĠB IB REF 12 Ġ. </s> RAM L : ĠA ĠRL Ġapproach Ġto Ġincorporate ĠM LE Ġobjective Ġinto ĠRL Ġtraining Ġframework , Ġwhich Ġregards ĠB LE U Ġas Ġrewards ĠB IB REF 17 Ġ. </s> Dialog GAN : ĠAn Ġextension Ġof ĠSe q GAN Ġtuned Ġto Ġdialogue Ġgeneration Ġtask Ġwith ĠM LE Ġobjective Ġadded Ġto Ġthe Ġadvers arial Ġobjective ĠB IB REF 16 Ġ. </s> DP GAN : ĠA Ġvariant Ġof ĠDial og GAN Ġwhich Ġuses Ġa Ġlanguage Ġmodel Ġbased Ġdiscrim inator Ġand Ġregards Ġcross - ent ropy Ġas Ġrewards ĠB IB REF 13 Ġ. </s> Note Ġthat ĠM LE , ĠSe q GAN , ĠLe ak GAN , ĠMali GAN Ġand ĠI RL Ġare Ġthe Ġbas elines Ġon ĠC OC O Ġand ĠEM N LP 2017 ĠW MT , Ġwhile ĠM LE , ĠRAM L , ĠDial og GAN , Ġand ĠD PG AN Ġon ĠWe ibo Dial . ĠThe Ġoriginal Ġcodes Ġare Ġused Ġto Ġtest Ġthe Ġbas elines . </s> Im plementation ĠDetails </s> The Ġimplementation Ġdetails Ġof Ġour Ġmodel Ġare Ġshown Ġin ĠTable ĠT AB REF 31 Ġ. ĠFor ĠC OC O Ġ/ ĠEM N LP 2017 , Ġthe Ġgenerator Ġis Ġa ĠL ST M Ġunit ĠB IB REF 30 Ġwith Ġ128 Ġcells , Ġand Ġthe Ġdiscrim inator Ġis Ġimplemented Ġbased Ġon ĠB IB REF 7 Ġ. ĠFor ĠWe ibo Dial , Ġthe Ġgenerator Ġis Ġan Ġenc oder - dec oder Ġstructure Ġwith Ġattention Ġmechanism , Ġwhere Ġboth Ġthe Ġenc oder Ġand Ġthe Ġdec oder Ġconsist Ġof Ġa Ġtwo - layer ĠGR U ĠB IB REF 31 Ġwith Ġ128 Ġcells . ĠThe Ġdiscrim inator Ġis Ġimplemented Ġbased Ġon ĠB IB REF 32 Ġ. ĠThe Ġlanguage Ġmodel Ġused Ġin Ġthe Ġconstrained Ġsampling Ġof ĠAR AM L Ġis Ġimplemented Ġin Ġthe Ġsame Ġsetting Ġas Ġthe Ġgenerators , Ġand Ġis Ġpre - trained Ġon Ġthe Ġtraining Ġset Ġof Ġeach Ġdataset . ĠThe Ġcodes Ġand Ġthe Ġdatasets Ġare Ġavailable Ġat Ġhttps :// github . com / ke pei 110 6 / AR AM L . </s> As Ġfor Ġthe Ġdetails Ġof Ġthe Ġbas elines , Ġthe Ġgenerators Ġof Ġall Ġthe Ġbas elines Ġexcept ĠLe ak GAN Ġare Ġthe Ġsame Ġas Ġours . ĠNote Ġthat Ġthe Ġgenerator Ġof ĠLe ak GAN Ġconsists Ġof Ġa Ġhierarchical ĠL ST M Ġunit , Ġthus Ġwe Ġfollowed Ġthe Ġimplementation Ġin Ġthe Ġoriginal Ġpaper . ĠIn Ġterms Ġof Ġthe Ġdifferences , Ġthe Ġdiscrim inators Ġof ĠG AN Ġbas elines Ġare Ġimplemented Ġbased Ġon Ġthe Ġoriginal Ġpapers . ĠOther Ġhyper - param eters Ġof Ġbas elines Ġincluding Ġbatch Ġsize , Ġlearning Ġrate , Ġand Ġpre - training Ġepoch s , Ġwere Ġset Ġbased Ġon Ġthe Ġoriginal Ġcodes , Ġbecause Ġthe Ġconvergence Ġof Ġbas elines Ġis Ġsensitive Ġto Ġthese Ġhyper - param eters . </s> Language ĠGeneration Ġon ĠC OC O Ġand ĠEM N LP 2017 ĠW MT </s> We Ġadopted Ġforward / reverse Ġperplex ity ĠB IB REF 33 Ġand ĠSelf - BLE U ĠB IB REF 34 Ġto Ġevaluate Ġthe Ġquality Ġof Ġgenerated Ġtexts . ĠForward Ġperplex ity Ġ( P PL - F ) Ġindicates Ġthe Ġperplex ity Ġon Ġthe Ġgenerated Ġdata Ġprovided Ġby Ġa Ġlanguage Ġmodel Ġtrained Ġon Ġreal Ġdata Ġto Ġmeasure Ġthe Ġflu ency Ġof Ġgenerated Ġsamples . ĠReverse Ġperplex ity Ġ( P PL - R ) Ġswitches Ġthe Ġroles Ġof Ġgenerated Ġdata Ġand Ġreal Ġdata Ġto Ġreflect Ġthe Ġdiscrepancy Ġbetween Ġthe Ġgenerated Ġdistribution Ġand Ġthe Ġdata Ġdistribution . ĠSelf - BLE U Ġ( S - BLE U ) Ġregards Ġeach Ġsentence Ġin Ġthe Ġgenerated Ġcollection Ġas Ġhypothesis Ġand Ġthe Ġothers Ġas Ġreference Ġto Ġobtain ĠB LE U Ġscores , Ġwhich Ġevaluates Ġthe Ġdiversity Ġof Ġgenerated Ġresults . </s> Results Ġare Ġshown Ġin ĠTable ĠT AB REF 33 Ġ. ĠLe ak GAN Ġperforms Ġbest Ġon Ġforward Ġperplex ity Ġbecause Ġit Ġcan Ġgenerate Ġmore Ġfluent Ġsamples . ĠAs Ġfor Ġreverse Ġperplex ity , Ġour Ġmodel ĠAR AM L Ġbeats Ġother Ġbas elines , Ġshowing Ġthat Ġour Ġmodel Ġcan Ġfit Ġthe Ġdata Ġdistribution Ġbetter . ĠOther ĠG AN s , Ġparticularly ĠLe ak GAN , Ġobtain Ġhigh Ġreverse Ġperplex ity Ġdue Ġto Ġmode Ġcollapse ĠB IB REF 12 Ġ, Ġthus Ġthey Ġonly Ġcapture Ġlimited Ġfluent Ġexpressions , Ġresulting Ġin Ġlarge Ġdiscrepancy Ġbetween Ġthe Ġgenerated Ġdistribution Ġand Ġdata Ġdistribution . ĠAR AM L Ġalso Ġoutper forms Ġthe Ġbas elines Ġin Ġterms Ġof ĠSelf - BLE U , Ġindicating Ġthat Ġour Ġmodel Ġdoesn 't Ġfall Ġinto Ġmode Ġcollapse Ġwith Ġthe Ġhelp Ġof Ġthe ĠM LE Ġtraining Ġobjective Ġand Ġhas Ġthe Ġability Ġto Ġgenerate Ġmore Ġdiverse Ġsentences . </s> We Ġalso Ġprovide Ġstandard Ġdeviation Ġof Ġeach Ġmetric Ġin ĠTable ĠT AB REF 33 Ġ, Ġreflecting Ġthe Ġstability Ġof Ġeach Ġmodel 's Ġperformance . ĠOur Ġmodel ĠAR AM L Ġnearly Ġachieves Ġthe Ġsmallest Ġstandard Ġdeviation Ġin Ġall Ġthe Ġmetrics , Ġindicating Ġthat Ġour Ġframework Ġoutper forms Ġpolicy Ġgradient Ġin Ġthe Ġstability Ġof Ġadvers arial Ġtraining . </s> Dialogue ĠGeneration Ġon ĠWe ibo Dial </s> Dialogue Ġevaluation Ġis Ġan Ġopen Ġproblem Ġand Ġexisting Ġworks Ġhave Ġfound Ġthat Ġautomatic Ġmetrics Ġhave Ġlow Ġcorrelation Ġto Ġhuman Ġevaluation ĠB IB REF 35 Ġ, ĠB IB REF 36 Ġ, ĠB IB REF 37 Ġ. ĠThus , Ġwe Ġresorted Ġto Ġmanual Ġevaluation Ġto Ġassess Ġthe Ġgeneration Ġquality Ġon ĠWe ibo Dial . ĠWe Ġrandomly Ġsampled Ġ200 Ġposts Ġfrom Ġthe Ġtest Ġset Ġand Ġcollected Ġthe Ġgenerated Ġresults Ġfrom Ġall Ġthe Ġmodels . ĠFor Ġeach Ġpair Ġof Ġresponses Ġ( one Ġfrom ĠAR AM L Ġand Ġthe Ġother Ġfrom Ġa Ġbaseline , Ġgiven Ġthe Ġsame Ġinput Ġpost ), Ġfive Ġannot ators Ġwere Ġhired Ġto Ġlabel Ġwhich Ġresponse Ġis Ġbetter Ġ( i . e . Ġwin , Ġlose Ġor Ġtie ) Ġin Ġterms Ġof Ġgram matical ity Ġ( whether Ġa Ġresponse Ġitself Ġis Ġgram matical Ġand Ġlogical ) Ġand Ġrelevance Ġ( whether Ġa Ġresponse Ġis Ġappropriate Ġand Ġrelevant Ġto Ġthe Ġpost ). ĠThe Ġtwo Ġmetrics Ġwere Ġevaluated Ġindependently . </s> The Ġevaluation Ġresults Ġare Ġshown Ġin ĠTable ĠT AB REF 35 Ġ. ĠTo Ġmeasure Ġthe Ġinter - annot ator Ġagreement , Ġwe Ġcalculated ĠFle iss ' Ġk appa ĠB IB REF 38 Ġfor Ġeach Ġpair - wise Ġcomparison Ġwhere Ġresults Ġshow Ġmoderate Ġagreement Ġ( ĠIN LINE FORM 0 Ġ). ĠWe Ġalso Ġconducted Ġsign Ġtest Ġto Ġcheck Ġthe Ġsignificance Ġof Ġthe Ġdifferences . </s> As Ġshown Ġin ĠTable ĠT AB REF 35 Ġ, ĠAR AM L Ġperforms Ġsignificantly Ġbetter Ġthan Ġother Ġbas elines Ġin Ġall Ġthe Ġcases . ĠThis Ġresult Ġindicates Ġthat Ġthe Ġsamples Ġsurrounding Ġtrue Ġresponses Ġprovide Ġstable Ġrewards Ġfor Ġthe Ġgenerator , Ġand Ġstable ĠRAM L Ġtraining Ġparadigm Ġsignificantly Ġenhances Ġthe Ġperformance Ġin Ġboth Ġmetrics . </s> Further ĠAnalysis Ġon ĠStability </s> To Ġverify Ġthe Ġtraining Ġstability , Ġwe Ġconducted Ġexperiments Ġon ĠC OC O Ġmany Ġtimes Ġand Ġchose Ġthe Ġbest Ġ5 Ġtrials Ġfor ĠSe q GAN , ĠLe ak GAN , ĠI RL , ĠMali GAN Ġand ĠAR AM L , Ġrespectively . ĠThen , Ġwe Ġpresented Ġthe Ġforward / reverse Ġperplex ity Ġin Ġthe Ġtraining Ġprocess Ġin ĠFigure ĠFIG REF 38 Ġ. ĠWe Ġcan Ġsee Ġthat Ġour Ġmodel Ġwith Ġsmaller Ġstandard Ġdeviation Ġis Ġmore Ġstable Ġthan Ġother ĠG AN Ġbas elines Ġin Ġboth Ġmetrics . ĠAlthough ĠLe ak GAN Ġreaches Ġthe Ġbest Ġforward Ġperplex ity , Ġits Ġstandard Ġdeviation Ġis Ġextremely Ġlarge Ġand Ġit Ġperforms Ġbadly Ġin Ġreverse Ġperplex ity , Ġindicating Ġthat Ġit Ġgenerates Ġlimited Ġexpressions Ġthat Ġare Ġgram matical Ġyet Ġdiver gent Ġfrom Ġthe Ġdata Ġdistribution . </s> A bl ation ĠStudy </s> The Ġtemperature ĠIN LINE FORM 0 Ġcontrols Ġthe Ġsearch Ġspace Ġsurrounding Ġthe Ġreal Ġdata Ġas Ġwe Ġanalyze Ġin ĠSection ĠUID 13 Ġ. ĠTo Ġinvestigate Ġits Ġimpact Ġon Ġthe Ġperformance Ġof Ġour Ġmodel , Ġwe Ġfixed Ġall Ġthe Ġother Ġhyper - param eters Ġand Ġtest ĠAR AM L Ġwith Ġdifferent Ġtemperatures Ġon ĠC OC O . </s> The Ġexperimental Ġresults Ġare Ġshown Ġin ĠFigure ĠFIG REF 41 Ġ. ĠWe Ġcan Ġsee Ġthat Ġas Ġthe Ġtemperature Ġbecomes Ġlarger , Ġforward Ġperplex ity Ġincreases Ġgradually Ġwhile ĠSelf - BLE U Ġdecreases . ĠAs Ġmentioned Ġin ĠSection ĠUID 13 Ġ, Ġlarge Ġtemperatures Ġencourage Ġour Ġgenerator Ġto Ġexplore Ġthe Ġsamples Ġthat Ġare Ġdistant Ġfrom Ġreal Ġdata Ġdistribution , Ġthus Ġthe Ġdiversity Ġof Ġgenerated Ġresults Ġwill Ġbe Ġimproved . ĠHowever , Ġthese Ġsamples Ġdistant Ġfrom Ġthe Ġdata Ġdistribution Ġare Ġmore Ġlikely Ġto Ġbe Ġpoor Ġin Ġflu ency , Ġleading Ġto Ġworse Ġforward Ġperplex ity . ĠReverse Ġperplex ity Ġis Ġinfluenced Ġby Ġboth Ġgeneration Ġquality Ġand Ġdiversity , Ġso Ġthe Ġcorrelation Ġbetween Ġtemperature Ġand Ġreverse Ġperplex ity Ġis Ġnot Ġintuitive . ĠWe Ġcan Ġobserve Ġthat Ġthe Ġmodel Ġwith ĠIN LINE FORM 0 Ġreaches Ġthe Ġbest Ġreverse Ġperplex ity . </s> We Ġhave Ġmentioned Ġtwo Ġcommon Ġsampling Ġstrategies Ġin ĠSection ĠUID 13 Ġ, Ġi . e . Ġrandom Ġsampling Ġand Ġconstrained Ġsampling . ĠTo Ġanalyze Ġtheir Ġimpact , Ġwe Ġkeep Ġall Ġthe Ġmodel Ġstructures Ġand Ġhyper - param eters Ġfixed Ġand Ġtest ĠAR AM L Ġwith Ġthese Ġtwo Ġstrategies Ġon ĠC OC O . </s> Table ĠT AB REF 45 Ġshows Ġthe Ġresults . ĠIt 's Ġobvious Ġthat Ġrandom Ġsampling Ġhurts Ġthe Ġmodel Ġperformance Ġexcept ĠSelf - BLE U - 1 , Ġbecause Ġit Ġindeed Ġallows Ġlow - quality Ġsamples Ġavailable Ġto Ġthe Ġgenerator . ĠExpl oring Ġthese Ġsamples Ġde grades Ġthe Ġquality Ġand Ġdiversity Ġof Ġgenerated Ġresults . ĠDespite Ġthe Ġworse Ġperformance Ġon Ġautomatic Ġmetrics , Ġrandom Ġsampling Ġdoesn 't Ġaffect Ġthe Ġtraining Ġstability Ġof Ġour Ġframework . ĠThe Ġstandard Ġdeviation Ġof ĠAR AM L - R Ġis Ġstill Ġsmaller Ġthan Ġother ĠG AN Ġbas elines . </s> Case ĠStudy </s> Table ĠT AB REF 47 Ġpresents Ġthe Ġexamples Ġgenerated Ġby Ġthe Ġmodels Ġon ĠC OC O . ĠWe Ġcan Ġfind Ġthat Ġother Ġbas elines Ġsuffer Ġfrom Ġgram matical Ġerrors Ġ( e . g . ĠâĢ ľ in Ġfront Ġof Ġflying Ġher Ġk ite " Ġfrom ĠM LE ), Ġrepetitive Ġexpressions Ġ( e . g . ĠâĢ ľ A Ġgroup Ġof Ġpeople " Ġfrom ĠI RL ) Ġand Ġinco herent Ġstatements Ġ( e . g . ĠâĢ ľ A Ġgroup Ġof Ġpeople Ġsitting Ġon Ġa Ġcell Ġphone âĢ Ŀ Ġfrom ĠI RL ). ĠBy Ġcontrast , Ġour Ġmodel Ġperforms Ġwell Ġin Ġthese Ġsentences Ġand Ġhas Ġthe Ġability Ġto Ġgenerate Ġgram matical Ġand Ġcoherent Ġresults . </s> Table ĠT AB REF 48 Ġshows Ġthe Ġgenerated Ġexamples Ġon ĠWe ibo Dial . ĠIt 's Ġobvious Ġthat Ġother Ġbas elines Ġdon 't Ġcapture Ġthe Ġtopic Ġword ĠâĢ ľ late " Ġin Ġthe Ġpost , Ġthus Ġgenerate Ġirrelevant Ġresponses . ĠAR AM L Ġcan Ġprovide Ġa Ġresponse Ġthat Ġis Ġgram matical Ġand Ġclosely Ġrelevant Ġto Ġthe Ġpost . </s> Conclusion </s> We Ġpropose Ġa Ġnovel Ġadvers arial Ġtraining Ġframework Ġto Ġdeal Ġwith Ġthe Ġinstability Ġproblem Ġof Ġcurrent ĠG AN s Ġfor Ġtext Ġgeneration . ĠTo Ġaddress Ġthe Ġinstability Ġissue Ġcaused Ġby Ġpolicy Ġgradient , Ġwe Ġincorporate ĠRAM L Ġinto Ġthe Ġad ves arial Ġtraining Ġparadigm Ġto Ġmake Ġour Ġgenerator Ġacquire Ġstable Ġrewards . ĠExper iments Ġshow Ġthat Ġour Ġmodel Ġperforms Ġbetter Ġthan Ġseveral Ġstate - of - the - art ĠG AN Ġbas elines Ġwith Ġlower Ġtraining Ġvariance , Ġyet Ġproducing Ġbetter Ġperformance Ġon Ġthree Ġtext Ġgeneration Ġtasks . </s> Acknowled gments </s> This Ġwork Ġwas Ġsupported Ġby Ġthe ĠNational ĠScience ĠFoundation Ġof ĠChina Ġ( Grant ĠNo . Ġ6 19 36 010 / 6 187 60 96 ) Ġand Ġthe ĠNational ĠKey ĠR & D ĠProgram Ġof ĠChina Ġ( Grant ĠNo . Ġ2018 Y FC 08 30 200 ). ĠWe Ġwould Ġlike Ġto Ġthank ĠTH UN US ĠN Ex T ĠJoint - Lab Ġfor Ġthe Ġsupport .
Introduction Natural text generation, as a key task in NLP, has been advanced substantially thanks to the flourish of neural models BIBREF0 , BIBREF1 . Typical frameworks such as sequence-to-sequence (seq2seq) have been applied to various generation tasks, including machine translation BIBREF2 and dialogue generation BIBREF3 . The standard paradigm to train such neural models is maximum likelihood estimation (MLE), which maximizes the log-likelihood of observing each word in the text given the ground-truth proceeding context BIBREF4 . Although widely used, MLE suffers from the exposure bias problem BIBREF5 , BIBREF6 : during test, the model sequentially predicts the next word conditioned on its previous generated words while during training conditioned on ground-truth words. To tackle this problem, generative adversarial networks (GAN) with reinforcement learning (RL) training approaches have been introduced to text generation tasks BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13 , where the discriminator is trained to distinguish real and generated text samples to provide reward signals for the generator, and the generator is optimized via policy gradient BIBREF7 . However, recent studies have shown that potential issues of training GANs on discrete data are more severe than exposure bias BIBREF14 , BIBREF15 . One of the fundamental issues when generating discrete text samples with GANs is training instability. Updating the generator with policy gradient always leads to an unstable training process because it's difficult for the generator to derive positive and stable reward signals from the discriminator even with careful pre-training BIBREF8 . As a result, the generator gets lost due to the high variance of reward signals and the training process may finally collapse BIBREF16 . In this paper, we propose a novel adversarial training framework called Adversarial Reward Augmented Maximum Likelihood (ARAML) to deal with the instability issue of training GANs for text generation. At each iteration of adversarial training, we first train the discriminator to assign higher rewards to real data than to generated samples. Then, inspired by reward augmented maximum likelihood (RAML) BIBREF17 , the generator is updated on the samples acquired from a stationary distribution with maximum likelihood estimation (MLE), weighted by the discriminator's rewards. This stationary distribution is designed to guarantee that training samples are surrounding the real data, thus the exploration space of our generator is indeed restricted by the MLE training objective, resulting in more stable training. Compared to other text GANs with RL training techniques, our framework acquires samples from the stationary distribution rather than the generator's distribution, and uses RAML training paradigm to optimize the generator instead of policy gradient. Our contributions are mainly as follows: Related Work Recently, text generation has been widely studied with neural models trained with maximum likelihood estimation BIBREF4 . However, MLE tends to generate universal text BIBREF18 . Various methods have been proposed to enhance the generation quality by refining the objective function BIBREF18 , BIBREF19 or modifying the generation distribution with external information like topic BIBREF20 , sentence type BIBREF21 , emotion BIBREF22 and knowledge BIBREF23 . As mentioned above, MLE suffers from the exposure bias problem BIBREF5 , BIBREF6 . Thus, reinforcement learning has been introduced to text generation tasks such as policy gradient BIBREF6 and actor-critic BIBREF24 . BIBREF17 proposed an efficient and stable approach called Reward Augmented Maximum Likelihood (RAML), which connects the log-likelihood and expected rewards to incorporate MLE training objective into RL framework. Since some text generation tasks have no explicit metrics to be directly optimized, adversarial training has been applied to generating discrete text samples with a discriminator to learn a proper reward. For instance, SeqGAN BIBREF7 devised a discriminator to distinguish the real data and generated samples, and a generator to maximize the reward from the discriminator via policy gradient. Other variants of GANs have been proposed to improve the generator or the discriminator. To improve the generator, MaliGAN BIBREF8 developed a normalized maximum likelihood optimization target for the generator to stably model the discrete sequences. LeakGAN BIBREF11 guided the generator with reward signals leaked from the discriminator at all generation steps to deal with long text generation task. MaskGAN BIBREF10 employed an actor-critic architecture to make the generator fill in missing text conditioned on the surrounding context, which is expected to mitigate the problem of mode collapse. As for the discriminator, RankGAN BIBREF9 replaced traditional discriminator with a ranker to learn the relative ranking information between the real texts and generated ones. Inverse reinforcement learning BIBREF12 used a trainable reward approximator as the discriminator to provide dense reward signals at each generation step. DPGAN BIBREF13 introduced a language model based discriminator and regarded cross-entropy as rewards to promote the diversity of generation results. The most similar works to our model are RAML BIBREF17 and MaliGAN BIBREF8 : 1) Compared with RAML, our model adds a discriminator to learn the reward signals instead of choosing existing metrics as rewards. We believe that our model can adapt to various text generation tasks, particularly those without explicit evaluation metrics. 2) Unlike MaliGAN, we acquire samples from a fixed distribution near the real data rather than the generator's distribution, which is expected to make the training process more stable. Task Definition and Model Overview Text generation can be formulated as follows: given the real data distribution INLINEFORM0 , the task is to train a generative model INLINEFORM1 where INLINEFORM2 can fit INLINEFORM3 well. In this formulation, INLINEFORM4 and INLINEFORM5 denotes a word in the vocabulary INLINEFORM6 . Figure FIGREF3 shows the overview of our model ARAML. This adversarial training framework consists of two phases: 1) The discriminator is trained to assign higher rewards to real data than to generated data. 2) The generator is trained on the samples acquired from a stationary distribution with reward augmented MLE training objective. This training paradigm of the generator indeed constrains the search space with the MLE training objective, which alleviates the issue of unstable training. Discriminator The discriminator INLINEFORM0 aims to distinguish real data and generated data like other GANs. Inspired by Least-Square GAN BIBREF25 , we devise the loss function as follows: DISPLAYFORM0 This loss function forces the discriminator to assign higher rewards to real data than to generated data, so the discriminator can learn to provide more proper rewards as the training proceeds. Generator The training objective of our generator INLINEFORM0 is derived from the objective of other discrete GANs with RL training method: DISPLAYFORM0 where INLINEFORM0 denotes the rewards from the discriminator INLINEFORM1 and the entropy regularized term INLINEFORM2 encourages INLINEFORM3 to generate diverse text samples. INLINEFORM4 is a temperature hyper-parameter to balance these two terms. As mentioned above, discrete GANs suffer from the instability issue due to policy gradient, thus they are consequently difficult to train. Inspired by RAML BIBREF17 , we introduce an exponential payoff distribution INLINEFORM0 to connect RL loss with RAML loss: DISPLAYFORM0 where INLINEFORM0 . Thus, we can rewrite INLINEFORM1 with INLINEFORM2 and INLINEFORM3 as follows: DISPLAYFORM0 Following RAML, we remove the constant term and optimize the KL divergence in the opposite direction: DISPLAYFORM0 where INLINEFORM0 is a constant in the training phase of the generator. It has been proved that INLINEFORM1 and INLINEFORM2 are equivalent up to their first order Taylor approximations, and they have the same global optimum BIBREF17 . INLINEFORM3 can be trained in a MLE-like fashion but sampling from the distribution INLINEFORM4 is intractable in the adversarial setting, because INLINEFORM5 varies with the discriminator INLINEFORM6 . Thus, we introduce importance sampling to separate sampling process from INLINEFORM7 and obtain the final loss function: DISPLAYFORM0 where INLINEFORM0 denotes a stationary distribution and INLINEFORM1 . To optimize this loss function, we first construct the fixed distribution INLINEFORM2 to get samples, and devise the proper reward function INLINEFORM3 to train the generator in a stable and effective way. We construct the distribution INLINEFORM0 based on INLINEFORM1 : DISPLAYFORM0 In this way, INLINEFORM0 can be designed to guarantee that INLINEFORM1 is near INLINEFORM2 , leading to a more stable training process. To obtain a new sample INLINEFORM3 from a real data sample INLINEFORM4 , we can design three steps which contain sampling an edit distance INLINEFORM5 , the positions INLINEFORM6 for substitution and the new words INLINEFORM7 filled into the corresponding positions. Thus, INLINEFORM8 can be decomposed into three terms: DISPLAYFORM0 The first step is to sample an edit distance based on a real data sample INLINEFORM0 , where INLINEFORM1 is a sequence of length INLINEFORM2 . The number of sentences which have the edit distance INLINEFORM3 to some input sentence can be computed approximately as below: DISPLAYFORM0 where INLINEFORM0 denotes the number of sentences which have an edit distance INLINEFORM1 to a sentence of length INLINEFORM2 , and INLINEFORM3 indicates the size of vocabulary. We then follow BIBREF17 to re-scale the counts by INLINEFORM4 and do normalization, so that we can sample an edit distance INLINEFORM5 from: DISPLAYFORM0 where INLINEFORM0 , as a temperature hyper-parameter, restricts the search space surrounding the original sentence. Larger INLINEFORM1 brings more samples with long edit distances. The next step is to select positions for substitution based on the sampled edit distance INLINEFORM0 . Intuitively, we can randomly choose INLINEFORM1 distinct positions in INLINEFORM2 to be replaced by new words. The probability of choosing the position INLINEFORM3 is calculated as follows: DISPLAYFORM0 Following this sampling strategy, we can obtain the position set INLINEFORM0 . This strategy approximately guarantees that the edit distance between a new sentence and the original sentence is INLINEFORM1 . At the final step, our model determines new words for substitution at each sampled position INLINEFORM0 . We can formulate this sampling process from the original sequence INLINEFORM1 to a new sample INLINEFORM2 as a sequential transition INLINEFORM3 . At each step from INLINEFORM4 to INLINEFORM5 INLINEFORM6 , we first sample a new word INLINEFORM7 from the distribution INLINEFORM8 , then replace the old word at position INLINEFORM9 of INLINEFORM10 to obtain INLINEFORM11 . The whole sampling process can be decomposed as follows: DISPLAYFORM0 There are two common sampling strategies to model INLINEFORM0 , i.e. random sampling and constrained sampling. Random sampling strategy samples a new word INLINEFORM1 according to the uniform distribution over the vocabulary INLINEFORM2 BIBREF17 , while constrained sampling strategy samples INLINEFORM3 to maximize the language model score of the target sentence INLINEFORM4 BIBREF26 , BIBREF27 . Here, we adopt constrained sampling in our model and compare the performances of two strategies in the experiment. We devise the reward function INLINEFORM0 according to the discriminator's output INLINEFORM1 and the stationary distribution INLINEFORM2 : DISPLAYFORM0 Intuitively, this reward function encourages the generator to generate sentences with large sampling probability and high rewards from the discriminator. Thus, the weight of samples INLINEFORM0 can be calculated as follows: DISPLAYFORM0 So far, we can successfully optimize the generator's loss INLINEFORM0 via Equation EQREF12 . This training paradigm makes our generator avoid possible variances caused by policy gradient and get more stable reward signals from the discriminator, because our generator is restricted to explore the training samples near the real data. [htb] Adversarial Reward Augmented Maximum Likelihood [1] Total adversarial training iterations: INLINEFORM0 Steps of training generator: INLINEFORM0 Steps of training discriminator: INLINEFORM0 Pre-train the generator INLINEFORM0 with MLE loss Generate samples from INLINEFORM1 Pre-train the discriminator INLINEFORM2 via Eq.( EQREF6 ) Construct INLINEFORM3 via Eq.( EQREF14 ) - Eq.( EQREF19 ) each INLINEFORM4 each INLINEFORM5 Update INLINEFORM6 via Eq.( EQREF12 ) each INLINEFORM7 Update INLINEFORM8 via Eq.( EQREF6 ) Extension to Conditional Text Generation We have shown our adversarial training framework for text generation tasks without an input. Actually, it can also be extended to conditional text generation tasks like dialogue generation. Given the data distribution INLINEFORM0 where INLINEFORM1 denote contexts and responses respectively, the objective function of ARAML's generator can be modified as below: DISPLAYFORM0 where INLINEFORM0 and INLINEFORM1 is trained to distinguish whether INLINEFORM2 is the true response to INLINEFORM3 . Comparison with RAML and MaliGAN The most similar works to our framework are RAML BIBREF17 and MaliGAN BIBREF8 . The main difference among them is the training objective of their generators. We have shown different objective functions in Table TABREF26 . For comparison, we use the form with no input for all the three models. Our model is greatly inspired by RAML, which gets samples from a non-parametric distribution INLINEFORM0 constructed based on a specific reward. Compared to RAML, our reward comes from a learnable discriminator which varies as the adversarial training proceeds rather than a specific reward function. This difference equips our framework with the ability to adapt to the text generation tasks with no explicit evaluation metrics as rewards. Our model is also similar to MaliGAN, which gets samples from the generator's distribution. In MaliGAN's training objective, INLINEFORM0 also indicates the generator's distribution but it's used in the sampling phase and fixed at each optimization step. The weight of samples INLINEFORM1 . Different from our model, MaliGAN acquires samples from the generator's distribution INLINEFORM2 , which usually brings samples with low rewards even with careful pre-training for the generator, leading to training instability. Instead, our framework gets samples from a stationary distribution INLINEFORM3 around real data, thus our training process is more stable. Datasets We evaluated ARAML on three datasets: COCO image caption dataset BIBREF28 , EMNLP2017 WMT dataset and WeiboDial single-turn dialogue dataset BIBREF29 . COCO and EMNLP2017 WMT are the common benchmarks with no input to evaluate the performance of discrete GANs, and we followed the existing works to preprocess these datasets BIBREF12 , BIBREF11 . WeiboDial, as a dialogue dataset, was applied to test the performance of our model with input trigger. We simply removed post-response pairs containing low-frequency words and randomly selected a subset for our training/test set. The statistics of three datasets are presented in Table TABREF28 . Baselines We compared our model with MLE, RL and GAN baselines. Since COCO and EMNLP2017 WMT don't have input while WeiboDial regards posts as input, we chose the following baselines respectively: MLE: a RNN model trained with MLE objective BIBREF4 . Its extension, Seq2Seq, can work on the dialogue dataset BIBREF2 . SeqGAN: The first text GAN model that updates the generator with policy gradient based on the rewards from the discriminator BIBREF7 . LeakGAN: A variant of SeqGAN that provides rewards based on the leaked information of the discriminator for the generator BIBREF11 . MaliGAN: A variant of SeqGAN that optimizes the generator with a normalized maximum likelihood objective BIBREF8 . IRL: This inverse reinforcement learning method replaces the discriminator with a reward approximator to provide dense rewards BIBREF12 . RAML: A RL approach to incorporate MLE objective into RL training framework, which regards BLEU as rewards BIBREF17 . DialogGAN: An extension of SeqGAN tuned to dialogue generation task with MLE objective added to the adversarial objective BIBREF16 . DPGAN: A variant of DialogGAN which uses a language model based discriminator and regards cross-entropy as rewards BIBREF13 . Note that MLE, SeqGAN, LeakGAN, MaliGAN and IRL are the baselines on COCO and EMNLP2017 WMT, while MLE, RAML, DialogGAN, and DPGAN on WeiboDial. The original codes are used to test the baselines. Implementation Details The implementation details of our model are shown in Table TABREF31 . For COCO / EMNLP2017, the generator is a LSTM unit BIBREF30 with 128 cells, and the discriminator is implemented based on BIBREF7 . For WeiboDial, the generator is an encoder-decoder structure with attention mechanism, where both the encoder and the decoder consist of a two-layer GRU BIBREF31 with 128 cells. The discriminator is implemented based on BIBREF32 . The language model used in the constrained sampling of ARAML is implemented in the same setting as the generators, and is pre-trained on the training set of each dataset. The codes and the datasets are available at https://github.com/kepei1106/ARAML. As for the details of the baselines, the generators of all the baselines except LeakGAN are the same as ours. Note that the generator of LeakGAN consists of a hierarchical LSTM unit, thus we followed the implementation in the original paper. In terms of the differences, the discriminators of GAN baselines are implemented based on the original papers. Other hyper-parameters of baselines including batch size, learning rate, and pre-training epochs, were set based on the original codes, because the convergence of baselines is sensitive to these hyper-parameters. Language Generation on COCO and EMNLP2017 WMT We adopted forward/reverse perplexity BIBREF33 and Self-BLEU BIBREF34 to evaluate the quality of generated texts. Forward perplexity (PPL-F) indicates the perplexity on the generated data provided by a language model trained on real data to measure the fluency of generated samples. Reverse perplexity (PPL-R) switches the roles of generated data and real data to reflect the discrepancy between the generated distribution and the data distribution. Self-BLEU (S-BLEU) regards each sentence in the generated collection as hypothesis and the others as reference to obtain BLEU scores, which evaluates the diversity of generated results. Results are shown in Table TABREF33 . LeakGAN performs best on forward perplexity because it can generate more fluent samples. As for reverse perplexity, our model ARAML beats other baselines, showing that our model can fit the data distribution better. Other GANs, particularly LeakGAN, obtain high reverse perplexity due to mode collapse BIBREF12 , thus they only capture limited fluent expressions, resulting in large discrepancy between the generated distribution and data distribution. ARAML also outperforms the baselines in terms of Self-BLEU, indicating that our model doesn't fall into mode collapse with the help of the MLE training objective and has the ability to generate more diverse sentences. We also provide standard deviation of each metric in Table TABREF33 , reflecting the stability of each model's performance. Our model ARAML nearly achieves the smallest standard deviation in all the metrics, indicating that our framework outperforms policy gradient in the stability of adversarial training. Dialogue Generation on WeiboDial Dialogue evaluation is an open problem and existing works have found that automatic metrics have low correlation to human evaluation BIBREF35 , BIBREF36 , BIBREF37 . Thus, we resorted to manual evaluation to assess the generation quality on WeiboDial. We randomly sampled 200 posts from the test set and collected the generated results from all the models. For each pair of responses (one from ARAML and the other from a baseline, given the same input post), five annotators were hired to label which response is better (i.e. win, lose or tie) in terms of grammaticality (whether a response itself is grammatical and logical) and relevance (whether a response is appropriate and relevant to the post). The two metrics were evaluated independently. The evaluation results are shown in Table TABREF35 . To measure the inter-annotator agreement, we calculated Fleiss' kappa BIBREF38 for each pair-wise comparison where results show moderate agreement ( INLINEFORM0 ). We also conducted sign test to check the significance of the differences. As shown in Table TABREF35 , ARAML performs significantly better than other baselines in all the cases. This result indicates that the samples surrounding true responses provide stable rewards for the generator, and stable RAML training paradigm significantly enhances the performance in both metrics. Further Analysis on Stability To verify the training stability, we conducted experiments on COCO many times and chose the best 5 trials for SeqGAN, LeakGAN, IRL, MaliGAN and ARAML, respectively. Then, we presented the forward/reverse perplexity in the training process in Figure FIGREF38 . We can see that our model with smaller standard deviation is more stable than other GAN baselines in both metrics. Although LeakGAN reaches the best forward perplexity, its standard deviation is extremely large and it performs badly in reverse perplexity, indicating that it generates limited expressions that are grammatical yet divergent from the data distribution. Ablation Study The temperature INLINEFORM0 controls the search space surrounding the real data as we analyze in Section UID13 . To investigate its impact on the performance of our model, we fixed all the other hyper-parameters and test ARAML with different temperatures on COCO. The experimental results are shown in Figure FIGREF41 . We can see that as the temperature becomes larger, forward perplexity increases gradually while Self-BLEU decreases. As mentioned in Section UID13 , large temperatures encourage our generator to explore the samples that are distant from real data distribution, thus the diversity of generated results will be improved. However, these samples distant from the data distribution are more likely to be poor in fluency, leading to worse forward perplexity. Reverse perplexity is influenced by both generation quality and diversity, so the correlation between temperature and reverse perplexity is not intuitive. We can observe that the model with INLINEFORM0 reaches the best reverse perplexity. We have mentioned two common sampling strategies in Section UID13 , i.e. random sampling and constrained sampling. To analyze their impact, we keep all the model structures and hyper-parameters fixed and test ARAML with these two strategies on COCO. Table TABREF45 shows the results. It's obvious that random sampling hurts the model performance except Self-BLEU-1, because it indeed allows low-quality samples available to the generator. Exploring these samples degrades the quality and diversity of generated results. Despite the worse performance on automatic metrics, random sampling doesn't affect the training stability of our framework. The standard deviation of ARAML-R is still smaller than other GAN baselines. Case Study Table TABREF47 presents the examples generated by the models on COCO. We can find that other baselines suffer from grammatical errors (e.g. “in front of flying her kite" from MLE), repetitive expressions (e.g. “A group of people" from IRL) and incoherent statements (e.g. “A group of people sitting on a cell phone” from IRL). By contrast, our model performs well in these sentences and has the ability to generate grammatical and coherent results. Table TABREF48 shows the generated examples on WeiboDial. It's obvious that other baselines don't capture the topic word “late" in the post, thus generate irrelevant responses. ARAML can provide a response that is grammatical and closely relevant to the post. Conclusion We propose a novel adversarial training framework to deal with the instability problem of current GANs for text generation. To address the instability issue caused by policy gradient, we incorporate RAML into the advesarial training paradigm to make our generator acquire stable rewards. Experiments show that our model performs better than several state-of-the-art GAN baselines with lower training variance, yet producing better performance on three text generation tasks. Acknowledgments This work was supported by the National Science Foundation of China (Grant No. 61936010/61876096) and the National Key R&D Program of China (Grant No. 2018YFC0830200). We would like to thank THUNUS NExT Joint-Lab for the support.
What GAN models were used as baselines to compare against?
43eecc576348411b0634611c81589f618cd4fddf
What ĠG AN Ġmodels Ġwere Ġused Ġas Ġbas elines Ġto Ġcompare Ġagainst ?
M LE , ĠSe q GAN , ĠLe ak GAN , ĠMali GAN , ĠI RL , ĠRAM L , ĠDial og GAN , ĠD PG AN
<s> What ĠG AN Ġmodels Ġwere Ġused Ġas Ġbas elines Ġto Ġcompare Ġagainst ? </s> Introduction </s> Natural Ġtext Ġgeneration , Ġas Ġa Ġkey Ġtask Ġin ĠN LP , Ġhas Ġbeen Ġadvanced Ġsubstantially Ġthanks Ġto Ġthe Ġflourish Ġof Ġneural Ġmodels ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ. ĠTypical Ġframeworks Ġsuch Ġas Ġsequence - to - sequence Ġ( seq 2 seq ) Ġhave Ġbeen Ġapplied Ġto Ġvarious Ġgeneration Ġtasks , Ġincluding Ġmachine Ġtranslation ĠB IB REF 2 Ġand Ġdialogue Ġgeneration ĠB IB REF 3 Ġ. ĠThe Ġstandard Ġparadigm Ġto Ġtrain Ġsuch Ġneural Ġmodels Ġis Ġmaximum Ġlikelihood Ġestimation Ġ( M LE ), Ġwhich Ġmaxim izes Ġthe Ġlog - like lihood Ġof Ġobserving Ġeach Ġword Ġin Ġthe Ġtext Ġgiven Ġthe Ġground - truth Ġproceeding Ġcontext ĠB IB REF 4 Ġ. </s> Although Ġwidely Ġused , ĠM LE Ġsuffers Ġfrom Ġthe Ġexposure Ġbias Ġproblem ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ: Ġduring Ġtest , Ġthe Ġmodel Ġsequ entially Ġpredicts Ġthe Ġnext Ġword Ġconditioned Ġon Ġits Ġprevious Ġgenerated Ġwords Ġwhile Ġduring Ġtraining Ġconditioned Ġon Ġground - truth Ġwords . ĠTo Ġtackle Ġthis Ġproblem , Ġgener ative Ġadvers arial Ġnetworks Ġ( GAN ) Ġwith Ġreinforcement Ġlearning Ġ( RL ) Ġtraining Ġapproaches Ġhave Ġbeen Ġintroduced Ġto Ġtext Ġgeneration Ġtasks ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ, ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ, Ġwhere Ġthe Ġdiscrim inator Ġis Ġtrained Ġto Ġdistinguish Ġreal Ġand Ġgenerated Ġtext Ġsamples Ġto Ġprovide Ġreward Ġsignals Ġfor Ġthe Ġgenerator , Ġand Ġthe Ġgenerator Ġis Ġoptimized Ġvia Ġpolicy Ġgradient ĠB IB REF 7 Ġ. </s> However , Ġrecent Ġstudies Ġhave Ġshown Ġthat Ġpotential Ġissues Ġof Ġtraining ĠG AN s Ġon Ġdiscrete Ġdata Ġare Ġmore Ġsevere Ġthan Ġexposure Ġbias ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ. ĠOne Ġof Ġthe Ġfundamental Ġissues Ġwhen Ġgenerating Ġdiscrete Ġtext Ġsamples Ġwith ĠG AN s Ġis Ġtraining Ġinstability . ĠUp dating Ġthe Ġgenerator Ġwith Ġpolicy Ġgradient Ġalways Ġleads Ġto Ġan Ġunstable Ġtraining Ġprocess Ġbecause Ġit 's Ġdifficult Ġfor Ġthe Ġgenerator Ġto Ġderive Ġpositive Ġand Ġstable Ġreward Ġsignals Ġfrom Ġthe Ġdiscrim inator Ġeven Ġwith Ġcareful Ġpre - training ĠB IB REF 8 Ġ. ĠAs Ġa Ġresult , Ġthe Ġgenerator Ġgets Ġlost Ġdue Ġto Ġthe Ġhigh Ġvariance Ġof Ġreward Ġsignals Ġand Ġthe Ġtraining Ġprocess Ġmay Ġfinally Ġcollapse ĠB IB REF 16 Ġ. </s> In Ġthis Ġpaper , Ġwe Ġpropose Ġa Ġnovel Ġadvers arial Ġtraining Ġframework Ġcalled ĠAd vers arial ĠReward ĠAug mented ĠMaximum ĠLike lihood Ġ( AR AM L ) Ġto Ġdeal Ġwith Ġthe Ġinstability Ġissue Ġof Ġtraining ĠG AN s Ġfor Ġtext Ġgeneration . ĠAt Ġeach Ġiteration Ġof Ġadvers arial Ġtraining , Ġwe Ġfirst Ġtrain Ġthe Ġdiscrim inator Ġto Ġassign Ġhigher Ġrewards Ġto Ġreal Ġdata Ġthan Ġto Ġgenerated Ġsamples . ĠThen , Ġinspired Ġby Ġreward Ġaugmented Ġmaximum Ġlikelihood Ġ( RAM L ) ĠB IB REF 17 Ġ, Ġthe Ġgenerator Ġis Ġupdated Ġon Ġthe Ġsamples Ġacquired Ġfrom Ġa Ġstationary Ġdistribution Ġwith Ġmaximum Ġlikelihood Ġestimation Ġ( M LE ), Ġweighted Ġby Ġthe Ġdiscrim inator 's Ġrewards . ĠThis Ġstationary Ġdistribution Ġis Ġdesigned Ġto Ġguarantee Ġthat Ġtraining Ġsamples Ġare Ġsurrounding Ġthe Ġreal Ġdata , Ġthus Ġthe Ġexploration Ġspace Ġof Ġour Ġgenerator Ġis Ġindeed Ġrestricted Ġby Ġthe ĠM LE Ġtraining Ġobjective , Ġresulting Ġin Ġmore Ġstable Ġtraining . ĠCompared Ġto Ġother Ġtext ĠG AN s Ġwith ĠRL Ġtraining Ġtechniques , Ġour Ġframework Ġacqu ires Ġsamples Ġfrom Ġthe Ġstationary Ġdistribution Ġrather Ġthan Ġthe Ġgenerator 's Ġdistribution , Ġand Ġuses ĠRAM L Ġtraining Ġparadigm Ġto Ġoptimize Ġthe Ġgenerator Ġinstead Ġof Ġpolicy Ġgradient . ĠOur Ġcontributions Ġare Ġmainly Ġas Ġfollows : </s> Related ĠWork </s> Recently , Ġtext Ġgeneration Ġhas Ġbeen Ġwidely Ġstudied Ġwith Ġneural Ġmodels Ġtrained Ġwith Ġmaximum Ġlikelihood Ġestimation ĠB IB REF 4 Ġ. ĠHowever , ĠM LE Ġtends Ġto Ġgenerate Ġuniversal Ġtext ĠB IB REF 18 Ġ. ĠVarious Ġmethods Ġhave Ġbeen Ġproposed Ġto Ġenhance Ġthe Ġgeneration Ġquality Ġby Ġrefining Ġthe Ġobjective Ġfunction ĠB IB REF 18 Ġ, ĠB IB REF 19 Ġor Ġmodifying Ġthe Ġgeneration Ġdistribution Ġwith Ġexternal Ġinformation Ġlike Ġtopic ĠB IB REF 20 Ġ, Ġsentence Ġtype ĠB IB REF 21 Ġ, Ġemotion ĠB IB REF 22 Ġand Ġknowledge ĠB IB REF 23 Ġ. </s> As Ġmentioned Ġabove , ĠM LE Ġsuffers Ġfrom Ġthe Ġexposure Ġbias Ġproblem ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ. ĠThus , Ġreinforcement Ġlearning Ġhas Ġbeen Ġintroduced Ġto Ġtext Ġgeneration Ġtasks Ġsuch Ġas Ġpolicy Ġgradient ĠB IB REF 6 Ġand Ġactor - crit ic ĠB IB REF 24 Ġ. ĠB IB REF 17 Ġproposed Ġan Ġefficient Ġand Ġstable Ġapproach Ġcalled ĠReward ĠAug mented ĠMaximum ĠLike lihood Ġ( RAM L ), Ġwhich Ġconnects Ġthe Ġlog - like lihood Ġand Ġexpected Ġrewards Ġto Ġincorporate ĠM LE Ġtraining Ġobjective Ġinto ĠRL Ġframework . </s> Since Ġsome Ġtext Ġgeneration Ġtasks Ġhave Ġno Ġexplicit Ġmetrics Ġto Ġbe Ġdirectly Ġoptimized , Ġadvers arial Ġtraining Ġhas Ġbeen Ġapplied Ġto Ġgenerating Ġdiscrete Ġtext Ġsamples Ġwith Ġa Ġdiscrim inator Ġto Ġlearn Ġa Ġproper Ġreward . ĠFor Ġinstance , ĠSe q GAN ĠB IB REF 7 Ġdevised Ġa Ġdiscrim inator Ġto Ġdistinguish Ġthe Ġreal Ġdata Ġand Ġgenerated Ġsamples , Ġand Ġa Ġgenerator Ġto Ġmaximize Ġthe Ġreward Ġfrom Ġthe Ġdiscrim inator Ġvia Ġpolicy Ġgradient . ĠOther Ġvariants Ġof ĠG AN s Ġhave Ġbeen Ġproposed Ġto Ġimprove Ġthe Ġgenerator Ġor Ġthe Ġdiscrim inator . ĠTo Ġimprove Ġthe Ġgenerator , ĠMali GAN ĠB IB REF 8 Ġdeveloped Ġa Ġnormalized Ġmaximum Ġlikelihood Ġoptimization Ġtarget Ġfor Ġthe Ġgenerator Ġto Ġst ably Ġmodel Ġthe Ġdiscrete Ġsequences . ĠLe ak GAN ĠB IB REF 11 Ġguided Ġthe Ġgenerator Ġwith Ġreward Ġsignals Ġleaked Ġfrom Ġthe Ġdiscrim inator Ġat Ġall Ġgeneration Ġsteps Ġto Ġdeal Ġwith Ġlong Ġtext Ġgeneration Ġtask . ĠMask GAN ĠB IB REF 10 Ġemployed Ġan Ġactor - crit ic Ġarchitecture Ġto Ġmake Ġthe Ġgenerator Ġfill Ġin Ġmissing Ġtext Ġconditioned Ġon Ġthe Ġsurrounding Ġcontext , Ġwhich Ġis Ġexpected Ġto Ġmitigate Ġthe Ġproblem Ġof Ġmode Ġcollapse . ĠAs Ġfor Ġthe Ġdiscrim inator , ĠRank GAN ĠB IB REF 9 Ġreplaced Ġtraditional Ġdiscrim inator Ġwith Ġa Ġrank er Ġto Ġlearn Ġthe Ġrelative Ġranking Ġinformation Ġbetween Ġthe Ġreal Ġtexts Ġand Ġgenerated Ġones . ĠIn verse Ġreinforcement Ġlearning ĠB IB REF 12 Ġused Ġa Ġtrain able Ġreward Ġapprox im ator Ġas Ġthe Ġdiscrim inator Ġto Ġprovide Ġdense Ġreward Ġsignals Ġat Ġeach Ġgeneration Ġstep . ĠD PG AN ĠB IB REF 13 Ġintroduced Ġa Ġlanguage Ġmodel Ġbased Ġdiscrim inator Ġand Ġregarded Ġcross - ent ropy Ġas Ġrewards Ġto Ġpromote Ġthe Ġdiversity Ġof Ġgeneration Ġresults . </s> The Ġmost Ġsimilar Ġworks Ġto Ġour Ġmodel Ġare ĠRAM L ĠB IB REF 17 Ġand ĠMali GAN ĠB IB REF 8 Ġ: Ġ1 ) ĠCompared Ġwith ĠRAM L , Ġour Ġmodel Ġadds Ġa Ġdiscrim inator Ġto Ġlearn Ġthe Ġreward Ġsignals Ġinstead Ġof Ġchoosing Ġexisting Ġmetrics Ġas Ġrewards . ĠWe Ġbelieve Ġthat Ġour Ġmodel Ġcan Ġadapt Ġto Ġvarious Ġtext Ġgeneration Ġtasks , Ġparticularly Ġthose Ġwithout Ġexplicit Ġevaluation Ġmetrics . Ġ2 ) ĠUnlike ĠMali GAN , Ġwe Ġacquire Ġsamples Ġfrom Ġa Ġfixed Ġdistribution Ġnear Ġthe Ġreal Ġdata Ġrather Ġthan Ġthe Ġgenerator 's Ġdistribution , Ġwhich Ġis Ġexpected Ġto Ġmake Ġthe Ġtraining Ġprocess Ġmore Ġstable . </s> Task ĠDefinition Ġand ĠModel ĠOverview </s> Text Ġgeneration Ġcan Ġbe Ġformulated Ġas Ġfollows : Ġgiven Ġthe Ġreal Ġdata Ġdistribution ĠIN LINE FORM 0 Ġ, Ġthe Ġtask Ġis Ġto Ġtrain Ġa Ġgener ative Ġmodel ĠIN LINE FORM 1 Ġwhere ĠIN LINE FORM 2 Ġcan Ġfit ĠIN LINE FORM 3 Ġwell . ĠIn Ġthis Ġformulation , ĠIN LINE FORM 4 Ġand ĠIN LINE FORM 5 Ġdenotes Ġa Ġword Ġin Ġthe Ġvocabulary ĠIN LINE FORM 6 Ġ. </s> Figure ĠFIG REF 3 Ġshows Ġthe Ġoverview Ġof Ġour Ġmodel ĠAR AM L . ĠThis Ġadvers arial Ġtraining Ġframework Ġconsists Ġof Ġtwo Ġphases : Ġ1 ) ĠThe Ġdiscrim inator Ġis Ġtrained Ġto Ġassign Ġhigher Ġrewards Ġto Ġreal Ġdata Ġthan Ġto Ġgenerated Ġdata . Ġ2 ) ĠThe Ġgenerator Ġis Ġtrained Ġon Ġthe Ġsamples Ġacquired Ġfrom Ġa Ġstationary Ġdistribution Ġwith Ġreward Ġaugmented ĠM LE Ġtraining Ġobjective . ĠThis Ġtraining Ġparadigm Ġof Ġthe Ġgenerator Ġindeed Ġconst r ains Ġthe Ġsearch Ġspace Ġwith Ġthe ĠM LE Ġtraining Ġobjective , Ġwhich Ġallev iates Ġthe Ġissue Ġof Ġunstable Ġtraining . </s> Disc rim inator </s> The Ġdiscrim inator ĠIN LINE FORM 0 Ġaims Ġto Ġdistinguish Ġreal Ġdata Ġand Ġgenerated Ġdata Ġlike Ġother ĠG AN s . ĠInspired Ġby ĠLe ast - Square ĠG AN ĠB IB REF 25 Ġ, Ġwe Ġdevise Ġthe Ġloss Ġfunction Ġas Ġfollows : ĠDIS PLAY FORM 0 </s> This Ġloss Ġfunction Ġforces Ġthe Ġdiscrim inator Ġto Ġassign Ġhigher Ġrewards Ġto Ġreal Ġdata Ġthan Ġto Ġgenerated Ġdata , Ġso Ġthe Ġdiscrim inator Ġcan Ġlearn Ġto Ġprovide Ġmore Ġproper Ġrewards Ġas Ġthe Ġtraining Ġproceeds . </s> Gener ator </s> The Ġtraining Ġobjective Ġof Ġour Ġgenerator ĠIN LINE FORM 0 Ġis Ġderived Ġfrom Ġthe Ġobjective Ġof Ġother Ġdiscrete ĠG AN s Ġwith ĠRL Ġtraining Ġmethod : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġdenotes Ġthe Ġrewards Ġfrom Ġthe Ġdiscrim inator ĠIN LINE FORM 1 Ġand Ġthe Ġentropy Ġregular ized Ġterm ĠIN LINE FORM 2 Ġencourages ĠIN LINE FORM 3 Ġto Ġgenerate Ġdiverse Ġtext Ġsamples . ĠIN LINE FORM 4 Ġis Ġa Ġtemperature Ġhyper - param eter Ġto Ġbalance Ġthese Ġtwo Ġterms . </s> As Ġmentioned Ġabove , Ġdiscrete ĠG AN s Ġsuffer Ġfrom Ġthe Ġinstability Ġissue Ġdue Ġto Ġpolicy Ġgradient , Ġthus Ġthey Ġare Ġconsequently Ġdifficult Ġto Ġtrain . ĠInspired Ġby ĠRAM L ĠB IB REF 17 Ġ, Ġwe Ġintroduce Ġan Ġexponential Ġpayoff Ġdistribution ĠIN LINE FORM 0 Ġto Ġconnect ĠRL Ġloss Ġwith ĠRAM L Ġloss : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġ. ĠThus , Ġwe Ġcan Ġrewrite ĠIN LINE FORM 1 Ġwith ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġas Ġfollows : ĠDIS PLAY FORM 0 </s> Following ĠRAM L , Ġwe Ġremove Ġthe Ġconstant Ġterm Ġand Ġoptimize Ġthe ĠKL Ġdivergence Ġin Ġthe Ġopposite Ġdirection : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġa Ġconstant Ġin Ġthe Ġtraining Ġphase Ġof Ġthe Ġgenerator . ĠIt Ġhas Ġbeen Ġproved Ġthat ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġare Ġequivalent Ġup Ġto Ġtheir Ġfirst Ġorder ĠTaylor Ġapprox im ations , Ġand Ġthey Ġhave Ġthe Ġsame Ġglobal Ġoptimum ĠB IB REF 17 Ġ. ĠIN LINE FORM 3 Ġcan Ġbe Ġtrained Ġin Ġa ĠM LE - like Ġfashion Ġbut Ġsampling Ġfrom Ġthe Ġdistribution ĠIN LINE FORM 4 Ġis Ġint ract able Ġin Ġthe Ġadvers arial Ġsetting , Ġbecause ĠIN LINE FORM 5 Ġvaries Ġwith Ġthe Ġdiscrim inator ĠIN LINE FORM 6 Ġ. ĠThus , Ġwe Ġintroduce Ġimportance Ġsampling Ġto Ġseparate Ġsampling Ġprocess Ġfrom ĠIN LINE FORM 7 Ġand Ġobtain Ġthe Ġfinal Ġloss Ġfunction : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġdenotes Ġa Ġstationary Ġdistribution Ġand ĠIN LINE FORM 1 Ġ. ĠTo Ġoptimize Ġthis Ġloss Ġfunction , Ġwe Ġfirst Ġconstruct Ġthe Ġfixed Ġdistribution ĠIN LINE FORM 2 Ġto Ġget Ġsamples , Ġand Ġdevise Ġthe Ġproper Ġreward Ġfunction ĠIN LINE FORM 3 Ġto Ġtrain Ġthe Ġgenerator Ġin Ġa Ġstable Ġand Ġeffective Ġway . </s> We Ġconstruct Ġthe Ġdistribution ĠIN LINE FORM 0 Ġbased Ġon ĠIN LINE FORM 1 Ġ: ĠDIS PLAY FORM 0 </s> In Ġthis Ġway , ĠIN LINE FORM 0 Ġcan Ġbe Ġdesigned Ġto Ġguarantee Ġthat ĠIN LINE FORM 1 Ġis Ġnear ĠIN LINE FORM 2 Ġ, Ġleading Ġto Ġa Ġmore Ġstable Ġtraining Ġprocess . ĠTo Ġobtain Ġa Ġnew Ġsample ĠIN LINE FORM 3 Ġfrom Ġa Ġreal Ġdata Ġsample ĠIN LINE FORM 4 Ġ, Ġwe Ġcan Ġdesign Ġthree Ġsteps Ġwhich Ġcontain Ġsampling Ġan Ġedit Ġdistance ĠIN LINE FORM 5 Ġ, Ġthe Ġpositions ĠIN LINE FORM 6 Ġfor Ġsubstitution Ġand Ġthe Ġnew Ġwords ĠIN LINE FORM 7 Ġfilled Ġinto Ġthe Ġcorresponding Ġpositions . ĠThus , ĠIN LINE FORM 8 Ġcan Ġbe Ġdecomp osed Ġinto Ġthree Ġterms : ĠDIS PLAY FORM 0 </s> The Ġfirst Ġstep Ġis Ġto Ġsample Ġan Ġedit Ġdistance Ġbased Ġon Ġa Ġreal Ġdata Ġsample ĠIN LINE FORM 0 Ġ, Ġwhere ĠIN LINE FORM 1 Ġis Ġa Ġsequence Ġof Ġlength ĠIN LINE FORM 2 Ġ. ĠThe Ġnumber Ġof Ġsentences Ġwhich Ġhave Ġthe Ġedit Ġdistance ĠIN LINE FORM 3 Ġto Ġsome Ġinput Ġsentence Ġcan Ġbe Ġcomputed Ġapproximately Ġas Ġbelow : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġdenotes Ġthe Ġnumber Ġof Ġsentences Ġwhich Ġhave Ġan Ġedit Ġdistance ĠIN LINE FORM 1 Ġto Ġa Ġsentence Ġof Ġlength ĠIN LINE FORM 2 Ġ, Ġand ĠIN LINE FORM 3 Ġindicates Ġthe Ġsize Ġof Ġvocabulary . ĠWe Ġthen Ġfollow ĠB IB REF 17 Ġto Ġre - scale Ġthe Ġcounts Ġby ĠIN LINE FORM 4 Ġand Ġdo Ġnormal ization , Ġso Ġthat Ġwe Ġcan Ġsample Ġan Ġedit Ġdistance ĠIN LINE FORM 5 Ġfrom : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġ, Ġas Ġa Ġtemperature Ġhyper - param eter , Ġrestricts Ġthe Ġsearch Ġspace Ġsurrounding Ġthe Ġoriginal Ġsentence . ĠL arger ĠIN LINE FORM 1 Ġbrings Ġmore Ġsamples Ġwith Ġlong Ġedit Ġdistances . </s> The Ġnext Ġstep Ġis Ġto Ġselect Ġpositions Ġfor Ġsubstitution Ġbased Ġon Ġthe Ġsampled Ġedit Ġdistance ĠIN LINE FORM 0 Ġ. ĠInt uitive ly , Ġwe Ġcan Ġrandomly Ġchoose ĠIN LINE FORM 1 Ġdistinct Ġpositions Ġin ĠIN LINE FORM 2 Ġto Ġbe Ġreplaced Ġby Ġnew Ġwords . ĠThe Ġprobability Ġof Ġchoosing Ġthe Ġposition ĠIN LINE FORM 3 Ġis Ġcalculated Ġas Ġfollows : ĠDIS PLAY FORM 0 </s> Following Ġthis Ġsampling Ġstrategy , Ġwe Ġcan Ġobtain Ġthe Ġposition Ġset ĠIN LINE FORM 0 Ġ. ĠThis Ġstrategy Ġapproximately Ġguarantees Ġthat Ġthe Ġedit Ġdistance Ġbetween Ġa Ġnew Ġsentence Ġand Ġthe Ġoriginal Ġsentence Ġis ĠIN LINE FORM 1 Ġ. </s> At Ġthe Ġfinal Ġstep , Ġour Ġmodel Ġdetermines Ġnew Ġwords Ġfor Ġsubstitution Ġat Ġeach Ġsampled Ġposition ĠIN LINE FORM 0 Ġ. ĠWe Ġcan Ġformulate Ġthis Ġsampling Ġprocess Ġfrom Ġthe Ġoriginal Ġsequence ĠIN LINE FORM 1 Ġto Ġa Ġnew Ġsample ĠIN LINE FORM 2 Ġas Ġa Ġsequential Ġtransition ĠIN LINE FORM 3 Ġ. ĠAt Ġeach Ġstep Ġfrom ĠIN LINE FORM 4 Ġto ĠIN LINE FORM 5 ĠIN LINE FORM 6 Ġ, Ġwe Ġfirst Ġsample Ġa Ġnew Ġword ĠIN LINE FORM 7 Ġfrom Ġthe Ġdistribution ĠIN LINE FORM 8 Ġ, Ġthen Ġreplace Ġthe Ġold Ġword Ġat Ġposition ĠIN LINE FORM 9 Ġof ĠIN LINE FORM 10 Ġto Ġobtain ĠIN LINE FORM 11 Ġ. ĠThe Ġwhole Ġsampling Ġprocess Ġcan Ġbe Ġdecomp osed Ġas Ġfollows : ĠDIS PLAY FORM 0 </s> There Ġare Ġtwo Ġcommon Ġsampling Ġstrategies Ġto Ġmodel ĠIN LINE FORM 0 Ġ, Ġi . e . Ġrandom Ġsampling Ġand Ġconstrained Ġsampling . ĠRandom Ġsampling Ġstrategy Ġsamples Ġa Ġnew Ġword ĠIN LINE FORM 1 Ġaccording Ġto Ġthe Ġuniform Ġdistribution Ġover Ġthe Ġvocabulary ĠIN LINE FORM 2 ĠB IB REF 17 Ġ, Ġwhile Ġconstrained Ġsampling Ġstrategy Ġsamples ĠIN LINE FORM 3 Ġto Ġmaximize Ġthe Ġlanguage Ġmodel Ġscore Ġof Ġthe Ġtarget Ġsentence ĠIN LINE FORM 4 ĠB IB REF 26 Ġ, ĠB IB REF 27 Ġ. ĠHere , Ġwe Ġadopt Ġconstrained Ġsampling Ġin Ġour Ġmodel Ġand Ġcompare Ġthe Ġperformances Ġof Ġtwo Ġstrategies Ġin Ġthe Ġexperiment . </s> We Ġdevise Ġthe Ġreward Ġfunction ĠIN LINE FORM 0 Ġaccording Ġto Ġthe Ġdiscrim inator 's Ġoutput ĠIN LINE FORM 1 Ġand Ġthe Ġstationary Ġdistribution ĠIN LINE FORM 2 Ġ: ĠDIS PLAY FORM 0 </s> Int uitive ly , Ġthis Ġreward Ġfunction Ġencourages Ġthe Ġgenerator Ġto Ġgenerate Ġsentences Ġwith Ġlarge Ġsampling Ġprobability Ġand Ġhigh Ġrewards Ġfrom Ġthe Ġdiscrim inator . ĠThus , Ġthe Ġweight Ġof Ġsamples ĠIN LINE FORM 0 Ġcan Ġbe Ġcalculated Ġas Ġfollows : ĠDIS PLAY FORM 0 </s> So Ġfar , Ġwe Ġcan Ġsuccessfully Ġoptimize Ġthe Ġgenerator 's Ġloss ĠIN LINE FORM 0 Ġvia ĠEqu ation ĠEQ REF 12 Ġ. ĠThis Ġtraining Ġparadigm Ġmakes Ġour Ġgenerator Ġavoid Ġpossible Ġvar iances Ġcaused Ġby Ġpolicy Ġgradient Ġand Ġget Ġmore Ġstable Ġreward Ġsignals Ġfrom Ġthe Ġdiscrim inator , Ġbecause Ġour Ġgenerator Ġis Ġrestricted Ġto Ġexplore Ġthe Ġtraining Ġsamples Ġnear Ġthe Ġreal Ġdata . </s> [ ht b ] ĠAd vers arial ĠReward ĠAug mented ĠMaximum ĠLike lihood Ġ[ 1 ] </s> Total Ġadvers arial Ġtraining Ġiterations : ĠIN LINE FORM 0 </s> Step s Ġof Ġtraining Ġgenerator : ĠIN LINE FORM 0 </s> Step s Ġof Ġtraining Ġdiscrim inator : ĠIN LINE FORM 0 </s> Pre - train Ġthe Ġgenerator ĠIN LINE FORM 0 Ġwith ĠM LE Ġloss ĠGener ate Ġsamples Ġfrom ĠIN LINE FORM 1 ĠPre - train Ġthe Ġdiscrim inator ĠIN LINE FORM 2 Ġvia ĠE q .( ĠEQ REF 6 Ġ) ĠConstruct ĠIN LINE FORM 3 Ġvia ĠE q .( ĠEQ REF 14 Ġ) Ġ- ĠE q .( ĠEQ REF 19 Ġ) Ġeach ĠIN LINE FORM 4 Ġeach ĠIN LINE FORM 5 ĠUpdate ĠIN LINE FORM 6 Ġvia ĠE q .( ĠEQ REF 12 Ġ) Ġeach ĠIN LINE FORM 7 ĠUpdate ĠIN LINE FORM 8 Ġvia ĠE q .( ĠEQ REF 6 Ġ) </s> Ext ension Ġto ĠCond itional ĠText ĠGeneration </s> We Ġhave Ġshown Ġour Ġadvers arial Ġtraining Ġframework Ġfor Ġtext Ġgeneration Ġtasks Ġwithout Ġan Ġinput . ĠActually , Ġit Ġcan Ġalso Ġbe Ġextended Ġto Ġconditional Ġtext Ġgeneration Ġtasks Ġlike Ġdialogue Ġgeneration . ĠGiven Ġthe Ġdata Ġdistribution ĠIN LINE FORM 0 Ġwhere ĠIN LINE FORM 1 Ġdenote Ġcontexts Ġand Ġresponses Ġrespectively , Ġthe Ġobjective Ġfunction Ġof ĠAR AM L 's Ġgenerator Ġcan Ġbe Ġmodified Ġas Ġbelow : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġis Ġtrained Ġto Ġdistinguish Ġwhether ĠIN LINE FORM 2 Ġis Ġthe Ġtrue Ġresponse Ġto ĠIN LINE FORM 3 Ġ. </s> Compar ison Ġwith ĠRAM L Ġand ĠMali GAN </s> The Ġmost Ġsimilar Ġworks Ġto Ġour Ġframework Ġare ĠRAM L ĠB IB REF 17 Ġand ĠMali GAN ĠB IB REF 8 Ġ. ĠThe Ġmain Ġdifference Ġamong Ġthem Ġis Ġthe Ġtraining Ġobjective Ġof Ġtheir Ġgenerators . ĠWe Ġhave Ġshown Ġdifferent Ġobjective Ġfunctions Ġin ĠTable ĠT AB REF 26 Ġ. ĠFor Ġcomparison , Ġwe Ġuse Ġthe Ġform Ġwith Ġno Ġinput Ġfor Ġall Ġthe Ġthree Ġmodels . </s> Our Ġmodel Ġis Ġgreatly Ġinspired Ġby ĠRAM L , Ġwhich Ġgets Ġsamples Ġfrom Ġa Ġnon - param etric Ġdistribution ĠIN LINE FORM 0 Ġconstructed Ġbased Ġon Ġa Ġspecific Ġreward . ĠCompared Ġto ĠRAM L , Ġour Ġreward Ġcomes Ġfrom Ġa Ġlearn able Ġdiscrim inator Ġwhich Ġvaries Ġas Ġthe Ġadvers arial Ġtraining Ġproceeds Ġrather Ġthan Ġa Ġspecific Ġreward Ġfunction . ĠThis Ġdifference Ġequ ips Ġour Ġframework Ġwith Ġthe Ġability Ġto Ġadapt Ġto Ġthe Ġtext Ġgeneration Ġtasks Ġwith Ġno Ġexplicit Ġevaluation Ġmetrics Ġas Ġrewards . </s> Our Ġmodel Ġis Ġalso Ġsimilar Ġto ĠMali GAN , Ġwhich Ġgets Ġsamples Ġfrom Ġthe Ġgenerator 's Ġdistribution . ĠIn ĠMali GAN 's Ġtraining Ġobjective , ĠIN LINE FORM 0 Ġalso Ġindicates Ġthe Ġgenerator 's Ġdistribution Ġbut Ġit 's Ġused Ġin Ġthe Ġsampling Ġphase Ġand Ġfixed Ġat Ġeach Ġoptimization Ġstep . ĠThe Ġweight Ġof Ġsamples ĠIN LINE FORM 1 Ġ. ĠDifferent Ġfrom Ġour Ġmodel , ĠMali GAN Ġacqu ires Ġsamples Ġfrom Ġthe Ġgenerator 's Ġdistribution ĠIN LINE FORM 2 Ġ, Ġwhich Ġusually Ġbrings Ġsamples Ġwith Ġlow Ġrewards Ġeven Ġwith Ġcareful Ġpre - training Ġfor Ġthe Ġgenerator , Ġleading Ġto Ġtraining Ġinstability . ĠInstead , Ġour Ġframework Ġgets Ġsamples Ġfrom Ġa Ġstationary Ġdistribution ĠIN LINE FORM 3 Ġaround Ġreal Ġdata , Ġthus Ġour Ġtraining Ġprocess Ġis Ġmore Ġstable . </s> Dat as ets </s> We Ġevaluated ĠAR AM L Ġon Ġthree Ġdatasets : ĠC OC O Ġimage Ġcaption Ġdataset ĠB IB REF 28 Ġ, ĠEM N LP 2017 ĠW MT Ġdataset Ġand ĠWe ibo Dial Ġsingle - turn Ġdialogue Ġdataset ĠB IB REF 29 Ġ. ĠC OC O Ġand ĠEM N LP 2017 ĠW MT Ġare Ġthe Ġcommon Ġbenchmarks Ġwith Ġno Ġinput Ġto Ġevaluate Ġthe Ġperformance Ġof Ġdiscrete ĠG AN s , Ġand Ġwe Ġfollowed Ġthe Ġexisting Ġworks Ġto Ġpre process Ġthese Ġdatasets ĠB IB REF 12 Ġ, ĠB IB REF 11 Ġ. ĠWe ibo Dial , Ġas Ġa Ġdialogue Ġdataset , Ġwas Ġapplied Ġto Ġtest Ġthe Ġperformance Ġof Ġour Ġmodel Ġwith Ġinput Ġtrigger . ĠWe Ġsimply Ġremoved Ġpost - response Ġpairs Ġcontaining Ġlow - frequency Ġwords Ġand Ġrandomly Ġselected Ġa Ġsubset Ġfor Ġour Ġtraining / test Ġset . ĠThe Ġstatistics Ġof Ġthree Ġdatasets Ġare Ġpresented Ġin ĠTable ĠT AB REF 28 Ġ. </s> Bas elines </s> We Ġcompared Ġour Ġmodel Ġwith ĠM LE , ĠRL Ġand ĠG AN Ġbas elines . ĠSince ĠC OC O Ġand ĠEM N LP 2017 ĠW MT Ġdon 't Ġhave Ġinput Ġwhile ĠWe ibo Dial Ġregards Ġposts Ġas Ġinput , Ġwe Ġchose Ġthe Ġfollowing Ġbas elines Ġrespectively : </s> M LE : Ġa ĠR NN Ġmodel Ġtrained Ġwith ĠM LE Ġobjective ĠB IB REF 4 Ġ. ĠIts Ġextension , ĠSe q 2 Se q , Ġcan Ġwork Ġon Ġthe Ġdialogue Ġdataset ĠB IB REF 2 Ġ. </s> Se q GAN : ĠThe Ġfirst Ġtext ĠG AN Ġmodel Ġthat Ġupdates Ġthe Ġgenerator Ġwith Ġpolicy Ġgradient Ġbased Ġon Ġthe Ġrewards Ġfrom Ġthe Ġdiscrim inator ĠB IB REF 7 Ġ. </s> Le ak GAN : ĠA Ġvariant Ġof ĠSe q GAN Ġthat Ġprovides Ġrewards Ġbased Ġon Ġthe Ġleaked Ġinformation Ġof Ġthe Ġdiscrim inator Ġfor Ġthe Ġgenerator ĠB IB REF 11 Ġ. </s> M ali GAN : ĠA Ġvariant Ġof ĠSe q GAN Ġthat Ġoptim izes Ġthe Ġgenerator Ġwith Ġa Ġnormalized Ġmaximum Ġlikelihood Ġobjective ĠB IB REF 8 Ġ. </s> IR L : ĠThis Ġinverse Ġreinforcement Ġlearning Ġmethod Ġreplaces Ġthe Ġdiscrim inator Ġwith Ġa Ġreward Ġapprox im ator Ġto Ġprovide Ġdense Ġrewards ĠB IB REF 12 Ġ. </s> RAM L : ĠA ĠRL Ġapproach Ġto Ġincorporate ĠM LE Ġobjective Ġinto ĠRL Ġtraining Ġframework , Ġwhich Ġregards ĠB LE U Ġas Ġrewards ĠB IB REF 17 Ġ. </s> Dialog GAN : ĠAn Ġextension Ġof ĠSe q GAN Ġtuned Ġto Ġdialogue Ġgeneration Ġtask Ġwith ĠM LE Ġobjective Ġadded Ġto Ġthe Ġadvers arial Ġobjective ĠB IB REF 16 Ġ. </s> DP GAN : ĠA Ġvariant Ġof ĠDial og GAN Ġwhich Ġuses Ġa Ġlanguage Ġmodel Ġbased Ġdiscrim inator Ġand Ġregards Ġcross - ent ropy Ġas Ġrewards ĠB IB REF 13 Ġ. </s> Note Ġthat ĠM LE , ĠSe q GAN , ĠLe ak GAN , ĠMali GAN Ġand ĠI RL Ġare Ġthe Ġbas elines Ġon ĠC OC O Ġand ĠEM N LP 2017 ĠW MT , Ġwhile ĠM LE , ĠRAM L , ĠDial og GAN , Ġand ĠD PG AN Ġon ĠWe ibo Dial . ĠThe Ġoriginal Ġcodes Ġare Ġused Ġto Ġtest Ġthe Ġbas elines . </s> Im plementation ĠDetails </s> The Ġimplementation Ġdetails Ġof Ġour Ġmodel Ġare Ġshown Ġin ĠTable ĠT AB REF 31 Ġ. ĠFor ĠC OC O Ġ/ ĠEM N LP 2017 , Ġthe Ġgenerator Ġis Ġa ĠL ST M Ġunit ĠB IB REF 30 Ġwith Ġ128 Ġcells , Ġand Ġthe Ġdiscrim inator Ġis Ġimplemented Ġbased Ġon ĠB IB REF 7 Ġ. ĠFor ĠWe ibo Dial , Ġthe Ġgenerator Ġis Ġan Ġenc oder - dec oder Ġstructure Ġwith Ġattention Ġmechanism , Ġwhere Ġboth Ġthe Ġenc oder Ġand Ġthe Ġdec oder Ġconsist Ġof Ġa Ġtwo - layer ĠGR U ĠB IB REF 31 Ġwith Ġ128 Ġcells . ĠThe Ġdiscrim inator Ġis Ġimplemented Ġbased Ġon ĠB IB REF 32 Ġ. ĠThe Ġlanguage Ġmodel Ġused Ġin Ġthe Ġconstrained Ġsampling Ġof ĠAR AM L Ġis Ġimplemented Ġin Ġthe Ġsame Ġsetting Ġas Ġthe Ġgenerators , Ġand Ġis Ġpre - trained Ġon Ġthe Ġtraining Ġset Ġof Ġeach Ġdataset . ĠThe Ġcodes Ġand Ġthe Ġdatasets Ġare Ġavailable Ġat Ġhttps :// github . com / ke pei 110 6 / AR AM L . </s> As Ġfor Ġthe Ġdetails Ġof Ġthe Ġbas elines , Ġthe Ġgenerators Ġof Ġall Ġthe Ġbas elines Ġexcept ĠLe ak GAN Ġare Ġthe Ġsame Ġas Ġours . ĠNote Ġthat Ġthe Ġgenerator Ġof ĠLe ak GAN Ġconsists Ġof Ġa Ġhierarchical ĠL ST M Ġunit , Ġthus Ġwe Ġfollowed Ġthe Ġimplementation Ġin Ġthe Ġoriginal Ġpaper . ĠIn Ġterms Ġof Ġthe Ġdifferences , Ġthe Ġdiscrim inators Ġof ĠG AN Ġbas elines Ġare Ġimplemented Ġbased Ġon Ġthe Ġoriginal Ġpapers . ĠOther Ġhyper - param eters Ġof Ġbas elines Ġincluding Ġbatch Ġsize , Ġlearning Ġrate , Ġand Ġpre - training Ġepoch s , Ġwere Ġset Ġbased Ġon Ġthe Ġoriginal Ġcodes , Ġbecause Ġthe Ġconvergence Ġof Ġbas elines Ġis Ġsensitive Ġto Ġthese Ġhyper - param eters . </s> Language ĠGeneration Ġon ĠC OC O Ġand ĠEM N LP 2017 ĠW MT </s> We Ġadopted Ġforward / reverse Ġperplex ity ĠB IB REF 33 Ġand ĠSelf - BLE U ĠB IB REF 34 Ġto Ġevaluate Ġthe Ġquality Ġof Ġgenerated Ġtexts . ĠForward Ġperplex ity Ġ( P PL - F ) Ġindicates Ġthe Ġperplex ity Ġon Ġthe Ġgenerated Ġdata Ġprovided Ġby Ġa Ġlanguage Ġmodel Ġtrained Ġon Ġreal Ġdata Ġto Ġmeasure Ġthe Ġflu ency Ġof Ġgenerated Ġsamples . ĠReverse Ġperplex ity Ġ( P PL - R ) Ġswitches Ġthe Ġroles Ġof Ġgenerated Ġdata Ġand Ġreal Ġdata Ġto Ġreflect Ġthe Ġdiscrepancy Ġbetween Ġthe Ġgenerated Ġdistribution Ġand Ġthe Ġdata Ġdistribution . ĠSelf - BLE U Ġ( S - BLE U ) Ġregards Ġeach Ġsentence Ġin Ġthe Ġgenerated Ġcollection Ġas Ġhypothesis Ġand Ġthe Ġothers Ġas Ġreference Ġto Ġobtain ĠB LE U Ġscores , Ġwhich Ġevaluates Ġthe Ġdiversity Ġof Ġgenerated Ġresults . </s> Results Ġare Ġshown Ġin ĠTable ĠT AB REF 33 Ġ. ĠLe ak GAN Ġperforms Ġbest Ġon Ġforward Ġperplex ity Ġbecause Ġit Ġcan Ġgenerate Ġmore Ġfluent Ġsamples . ĠAs Ġfor Ġreverse Ġperplex ity , Ġour Ġmodel ĠAR AM L Ġbeats Ġother Ġbas elines , Ġshowing Ġthat Ġour Ġmodel Ġcan Ġfit Ġthe Ġdata Ġdistribution Ġbetter . ĠOther ĠG AN s , Ġparticularly ĠLe ak GAN , Ġobtain Ġhigh Ġreverse Ġperplex ity Ġdue Ġto Ġmode Ġcollapse ĠB IB REF 12 Ġ, Ġthus Ġthey Ġonly Ġcapture Ġlimited Ġfluent Ġexpressions , Ġresulting Ġin Ġlarge Ġdiscrepancy Ġbetween Ġthe Ġgenerated Ġdistribution Ġand Ġdata Ġdistribution . ĠAR AM L Ġalso Ġoutper forms Ġthe Ġbas elines Ġin Ġterms Ġof ĠSelf - BLE U , Ġindicating Ġthat Ġour Ġmodel Ġdoesn 't Ġfall Ġinto Ġmode Ġcollapse Ġwith Ġthe Ġhelp Ġof Ġthe ĠM LE Ġtraining Ġobjective Ġand Ġhas Ġthe Ġability Ġto Ġgenerate Ġmore Ġdiverse Ġsentences . </s> We Ġalso Ġprovide Ġstandard Ġdeviation Ġof Ġeach Ġmetric Ġin ĠTable ĠT AB REF 33 Ġ, Ġreflecting Ġthe Ġstability Ġof Ġeach Ġmodel 's Ġperformance . ĠOur Ġmodel ĠAR AM L Ġnearly Ġachieves Ġthe Ġsmallest Ġstandard Ġdeviation Ġin Ġall Ġthe Ġmetrics , Ġindicating Ġthat Ġour Ġframework Ġoutper forms Ġpolicy Ġgradient Ġin Ġthe Ġstability Ġof Ġadvers arial Ġtraining . </s> Dialogue ĠGeneration Ġon ĠWe ibo Dial </s> Dialogue Ġevaluation Ġis Ġan Ġopen Ġproblem Ġand Ġexisting Ġworks Ġhave Ġfound Ġthat Ġautomatic Ġmetrics Ġhave Ġlow Ġcorrelation Ġto Ġhuman Ġevaluation ĠB IB REF 35 Ġ, ĠB IB REF 36 Ġ, ĠB IB REF 37 Ġ. ĠThus , Ġwe Ġresorted Ġto Ġmanual Ġevaluation Ġto Ġassess Ġthe Ġgeneration Ġquality Ġon ĠWe ibo Dial . ĠWe Ġrandomly Ġsampled Ġ200 Ġposts Ġfrom Ġthe Ġtest Ġset Ġand Ġcollected Ġthe Ġgenerated Ġresults Ġfrom Ġall Ġthe Ġmodels . ĠFor Ġeach Ġpair Ġof Ġresponses Ġ( one Ġfrom ĠAR AM L Ġand Ġthe Ġother Ġfrom Ġa Ġbaseline , Ġgiven Ġthe Ġsame Ġinput Ġpost ), Ġfive Ġannot ators Ġwere Ġhired Ġto Ġlabel Ġwhich Ġresponse Ġis Ġbetter Ġ( i . e . Ġwin , Ġlose Ġor Ġtie ) Ġin Ġterms Ġof Ġgram matical ity Ġ( whether Ġa Ġresponse Ġitself Ġis Ġgram matical Ġand Ġlogical ) Ġand Ġrelevance Ġ( whether Ġa Ġresponse Ġis Ġappropriate Ġand Ġrelevant Ġto Ġthe Ġpost ). ĠThe Ġtwo Ġmetrics Ġwere Ġevaluated Ġindependently . </s> The Ġevaluation Ġresults Ġare Ġshown Ġin ĠTable ĠT AB REF 35 Ġ. ĠTo Ġmeasure Ġthe Ġinter - annot ator Ġagreement , Ġwe Ġcalculated ĠFle iss ' Ġk appa ĠB IB REF 38 Ġfor Ġeach Ġpair - wise Ġcomparison Ġwhere Ġresults Ġshow Ġmoderate Ġagreement Ġ( ĠIN LINE FORM 0 Ġ). ĠWe Ġalso Ġconducted Ġsign Ġtest Ġto Ġcheck Ġthe Ġsignificance Ġof Ġthe Ġdifferences . </s> As Ġshown Ġin ĠTable ĠT AB REF 35 Ġ, ĠAR AM L Ġperforms Ġsignificantly Ġbetter Ġthan Ġother Ġbas elines Ġin Ġall Ġthe Ġcases . ĠThis Ġresult Ġindicates Ġthat Ġthe Ġsamples Ġsurrounding Ġtrue Ġresponses Ġprovide Ġstable Ġrewards Ġfor Ġthe Ġgenerator , Ġand Ġstable ĠRAM L Ġtraining Ġparadigm Ġsignificantly Ġenhances Ġthe Ġperformance Ġin Ġboth Ġmetrics . </s> Further ĠAnalysis Ġon ĠStability </s> To Ġverify Ġthe Ġtraining Ġstability , Ġwe Ġconducted Ġexperiments Ġon ĠC OC O Ġmany Ġtimes Ġand Ġchose Ġthe Ġbest Ġ5 Ġtrials Ġfor ĠSe q GAN , ĠLe ak GAN , ĠI RL , ĠMali GAN Ġand ĠAR AM L , Ġrespectively . ĠThen , Ġwe Ġpresented Ġthe Ġforward / reverse Ġperplex ity Ġin Ġthe Ġtraining Ġprocess Ġin ĠFigure ĠFIG REF 38 Ġ. ĠWe Ġcan Ġsee Ġthat Ġour Ġmodel Ġwith Ġsmaller Ġstandard Ġdeviation Ġis Ġmore Ġstable Ġthan Ġother ĠG AN Ġbas elines Ġin Ġboth Ġmetrics . ĠAlthough ĠLe ak GAN Ġreaches Ġthe Ġbest Ġforward Ġperplex ity , Ġits Ġstandard Ġdeviation Ġis Ġextremely Ġlarge Ġand Ġit Ġperforms Ġbadly Ġin Ġreverse Ġperplex ity , Ġindicating Ġthat Ġit Ġgenerates Ġlimited Ġexpressions Ġthat Ġare Ġgram matical Ġyet Ġdiver gent Ġfrom Ġthe Ġdata Ġdistribution . </s> A bl ation ĠStudy </s> The Ġtemperature ĠIN LINE FORM 0 Ġcontrols Ġthe Ġsearch Ġspace Ġsurrounding Ġthe Ġreal Ġdata Ġas Ġwe Ġanalyze Ġin ĠSection ĠUID 13 Ġ. ĠTo Ġinvestigate Ġits Ġimpact Ġon Ġthe Ġperformance Ġof Ġour Ġmodel , Ġwe Ġfixed Ġall Ġthe Ġother Ġhyper - param eters Ġand Ġtest ĠAR AM L Ġwith Ġdifferent Ġtemperatures Ġon ĠC OC O . </s> The Ġexperimental Ġresults Ġare Ġshown Ġin ĠFigure ĠFIG REF 41 Ġ. ĠWe Ġcan Ġsee Ġthat Ġas Ġthe Ġtemperature Ġbecomes Ġlarger , Ġforward Ġperplex ity Ġincreases Ġgradually Ġwhile ĠSelf - BLE U Ġdecreases . ĠAs Ġmentioned Ġin ĠSection ĠUID 13 Ġ, Ġlarge Ġtemperatures Ġencourage Ġour Ġgenerator Ġto Ġexplore Ġthe Ġsamples Ġthat Ġare Ġdistant Ġfrom Ġreal Ġdata Ġdistribution , Ġthus Ġthe Ġdiversity Ġof Ġgenerated Ġresults Ġwill Ġbe Ġimproved . ĠHowever , Ġthese Ġsamples Ġdistant Ġfrom Ġthe Ġdata Ġdistribution Ġare Ġmore Ġlikely Ġto Ġbe Ġpoor Ġin Ġflu ency , Ġleading Ġto Ġworse Ġforward Ġperplex ity . ĠReverse Ġperplex ity Ġis Ġinfluenced Ġby Ġboth Ġgeneration Ġquality Ġand Ġdiversity , Ġso Ġthe Ġcorrelation Ġbetween Ġtemperature Ġand Ġreverse Ġperplex ity Ġis Ġnot Ġintuitive . ĠWe Ġcan Ġobserve Ġthat Ġthe Ġmodel Ġwith ĠIN LINE FORM 0 Ġreaches Ġthe Ġbest Ġreverse Ġperplex ity . </s> We Ġhave Ġmentioned Ġtwo Ġcommon Ġsampling Ġstrategies Ġin ĠSection ĠUID 13 Ġ, Ġi . e . Ġrandom Ġsampling Ġand Ġconstrained Ġsampling . ĠTo Ġanalyze Ġtheir Ġimpact , Ġwe Ġkeep Ġall Ġthe Ġmodel Ġstructures Ġand Ġhyper - param eters Ġfixed Ġand Ġtest ĠAR AM L Ġwith Ġthese Ġtwo Ġstrategies Ġon ĠC OC O . </s> Table ĠT AB REF 45 Ġshows Ġthe Ġresults . ĠIt 's Ġobvious Ġthat Ġrandom Ġsampling Ġhurts Ġthe Ġmodel Ġperformance Ġexcept ĠSelf - BLE U - 1 , Ġbecause Ġit Ġindeed Ġallows Ġlow - quality Ġsamples Ġavailable Ġto Ġthe Ġgenerator . ĠExpl oring Ġthese Ġsamples Ġde grades Ġthe Ġquality Ġand Ġdiversity Ġof Ġgenerated Ġresults . ĠDespite Ġthe Ġworse Ġperformance Ġon Ġautomatic Ġmetrics , Ġrandom Ġsampling Ġdoesn 't Ġaffect Ġthe Ġtraining Ġstability Ġof Ġour Ġframework . ĠThe Ġstandard Ġdeviation Ġof ĠAR AM L - R Ġis Ġstill Ġsmaller Ġthan Ġother ĠG AN Ġbas elines . </s> Case ĠStudy </s> Table ĠT AB REF 47 Ġpresents Ġthe Ġexamples Ġgenerated Ġby Ġthe Ġmodels Ġon ĠC OC O . ĠWe Ġcan Ġfind Ġthat Ġother Ġbas elines Ġsuffer Ġfrom Ġgram matical Ġerrors Ġ( e . g . ĠâĢ ľ in Ġfront Ġof Ġflying Ġher Ġk ite " Ġfrom ĠM LE ), Ġrepetitive Ġexpressions Ġ( e . g . ĠâĢ ľ A Ġgroup Ġof Ġpeople " Ġfrom ĠI RL ) Ġand Ġinco herent Ġstatements Ġ( e . g . ĠâĢ ľ A Ġgroup Ġof Ġpeople Ġsitting Ġon Ġa Ġcell Ġphone âĢ Ŀ Ġfrom ĠI RL ). ĠBy Ġcontrast , Ġour Ġmodel Ġperforms Ġwell Ġin Ġthese Ġsentences Ġand Ġhas Ġthe Ġability Ġto Ġgenerate Ġgram matical Ġand Ġcoherent Ġresults . </s> Table ĠT AB REF 48 Ġshows Ġthe Ġgenerated Ġexamples Ġon ĠWe ibo Dial . ĠIt 's Ġobvious Ġthat Ġother Ġbas elines Ġdon 't Ġcapture Ġthe Ġtopic Ġword ĠâĢ ľ late " Ġin Ġthe Ġpost , Ġthus Ġgenerate Ġirrelevant Ġresponses . ĠAR AM L Ġcan Ġprovide Ġa Ġresponse Ġthat Ġis Ġgram matical Ġand Ġclosely Ġrelevant Ġto Ġthe Ġpost . </s> Conclusion </s> We Ġpropose Ġa Ġnovel Ġadvers arial Ġtraining Ġframework Ġto Ġdeal Ġwith Ġthe Ġinstability Ġproblem Ġof Ġcurrent ĠG AN s Ġfor Ġtext Ġgeneration . ĠTo Ġaddress Ġthe Ġinstability Ġissue Ġcaused Ġby Ġpolicy Ġgradient , Ġwe Ġincorporate ĠRAM L Ġinto Ġthe Ġad ves arial Ġtraining Ġparadigm Ġto Ġmake Ġour Ġgenerator Ġacquire Ġstable Ġrewards . ĠExper iments Ġshow Ġthat Ġour Ġmodel Ġperforms Ġbetter Ġthan Ġseveral Ġstate - of - the - art ĠG AN Ġbas elines Ġwith Ġlower Ġtraining Ġvariance , Ġyet Ġproducing Ġbetter Ġperformance Ġon Ġthree Ġtext Ġgeneration Ġtasks . </s> Acknowled gments </s> This Ġwork Ġwas Ġsupported Ġby Ġthe ĠNational ĠScience ĠFoundation Ġof ĠChina Ġ( Grant ĠNo . Ġ6 19 36 010 / 6 187 60 96 ) Ġand Ġthe ĠNational ĠKey ĠR & D ĠProgram Ġof ĠChina Ġ( Grant ĠNo . Ġ2018 Y FC 08 30 200 ). ĠWe Ġwould Ġlike Ġto Ġthank ĠTH UN US ĠN Ex T ĠJoint - Lab Ġfor Ġthe Ġsupport .