Introduce a "Refinement Adjustment LSTM-based component" to the decoder
ABSTRACTIVE
While the RNN-based generators with DA gating-vector can prevent the undesirable semantic repetitions, the ARED-based generators show signs of better adapting to a new domain. However, none of the models show significant advantage from out-of-domain data. To better analyze model generalization to an unseen, new domain as well as model leveraging the out-of-domain sources, we propose a new architecture which is an extension of the ARED model. In order to better select, aggregate and control the semantic information, a Refinement Adjustment LSTM-based component (RALSTM) is introduced to the decoder side. The proposed model can learn from unaligned data by jointly training the sentence planning and surface realization to produce natural language sentences. We conducted experiments on four different NLG domains and found that the proposed methods significantly outperformed the state-of-the-art methods regarding BLEU BIBREF15 and slot error rate ERR scores BIBREF4 . The results also showed that our generators could scale to new domains by leveraging the out-of-domain data. To sum up, we make three key contributions in this paper:
1706.00139
Introduction </s> Natural ĠLanguage ĠGeneration Ġ( NL G ) Ġplays Ġa Ġcritical Ġrole Ġin ĠSp oken ĠDialogue ĠSystems Ġ( S DS ) Ġwith Ġtask Ġis Ġto Ġconvert Ġa Ġmeaning Ġrepresentation Ġproduced Ġby Ġthe ĠDialogue ĠManager Ġinto Ġnatural Ġlanguage Ġutter ances . ĠCon ventional Ġapproaches Ġstill Ġrely Ġon Ġcomprehensive Ġhand - tun ing Ġtemplates Ġand Ġrules Ġrequiring Ġexpert Ġknowledge Ġof Ġlinguistic Ġrepresentation , Ġincluding Ġrule - based ĠB IB REF 0 Ġ, Ġcorpus - based Ġn - gram Ġmodels ĠB IB REF 1 Ġ, Ġand Ġa Ġtrain able Ġgenerator ĠB IB REF 2 Ġ. </s> Recently , ĠRec urrent ĠNeural ĠNetworks Ġ( R NN s ) Ġbased Ġapproaches Ġhave Ġshown Ġpromising Ġperformance Ġin Ġtackling Ġthe ĠNL G Ġproblems . ĠThe ĠR NN - based Ġmodels Ġhave Ġbeen Ġapplied Ġfor ĠNL G Ġas Ġa Ġjoint Ġtraining Ġmodel ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġand Ġan Ġend - to - end Ġtraining Ġmodel ĠB IB REF 5 Ġ. ĠA Ġrecurring Ġproblem Ġin Ġsuch Ġsystems Ġis Ġrequiring Ġannot ated Ġdatasets Ġfor Ġparticular Ġdialogue Ġacts Ġ( D As ). ĠTo Ġensure Ġthat Ġthe Ġgenerated Ġutter ance Ġrepresenting Ġthe Ġintended Ġmeaning Ġof Ġthe Ġgiven ĠDA , Ġthe Ġprevious ĠR NN - based Ġmodels Ġwere Ġfurther Ġconditioned Ġon Ġa Ġ1 - hot Ġvector Ġrepresentation Ġof Ġthe ĠDA . ĠB IB REF 3 Ġintroduced Ġa Ġhe uristic Ġgate Ġto Ġensure Ġthat Ġall Ġthe Ġslot - value Ġpair Ġwas Ġaccurately Ġcaptured Ġduring Ġgeneration . ĠB IB REF 4 Ġsubsequently Ġproposed Ġa ĠSem antically ĠCondition ed ĠLong ĠShort - term ĠMemory Ġgenerator Ġ( SC - L ST M ) Ġwhich Ġjointly Ġlearned Ġthe ĠDA Ġg ating Ġsignal Ġand Ġlanguage Ġmodel . </s> More Ġrecently , ĠEnc oder - Dec oder Ġnetworks ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, Ġespecially Ġthe Ġattention al Ġbased Ġmodels ĠB IB REF 8 Ġ, ĠB IB REF 9 Ġhave Ġbeen Ġexplored Ġto Ġsolve Ġthe ĠNL G Ġtasks . ĠThe ĠAttention al ĠR NN ĠEnc oder - Dec oder ĠB IB REF 10 Ġ( AR ED ) Ġbased Ġapproaches Ġhave Ġalso Ġshown Ġimproved Ġperformance Ġon Ġa Ġvariety Ġof Ġtasks , Ġe . g ., Ġimage Ġcaption ing ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ, Ġtext Ġsummar ization ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ. </s> While Ġthe ĠR NN - based Ġgenerators Ġwith ĠDA Ġg ating - vector Ġcan Ġprevent Ġthe Ġundesirable Ġsemantic Ġrepet itions , Ġthe ĠAR ED - based Ġgenerators Ġshow Ġsigns Ġof Ġbetter Ġadapting Ġto Ġa Ġnew Ġdomain . ĠHowever , Ġnone Ġof Ġthe Ġmodels Ġshow Ġsignificant Ġadvantage Ġfrom Ġout - of - domain Ġdata . ĠTo Ġbetter Ġanalyze Ġmodel Ġgeneral ization Ġto Ġan Ġunseen , Ġnew Ġdomain Ġas Ġwell Ġas Ġmodel Ġleveraging Ġthe Ġout - of - domain Ġsources , Ġwe Ġpropose Ġa Ġnew Ġarchitecture Ġwhich Ġis Ġan Ġextension Ġof Ġthe ĠAR ED Ġmodel . ĠIn Ġorder Ġto Ġbetter Ġselect , Ġaggregate Ġand Ġcontrol Ġthe Ġsemantic Ġinformation , Ġa ĠRef inement ĠAdjust ment ĠL ST M - based Ġcomponent Ġ( RAL ST M ) Ġis Ġintroduced Ġto Ġthe Ġdec oder Ġside . ĠThe Ġproposed Ġmodel Ġcan Ġlearn Ġfrom Ġun aligned Ġdata Ġby Ġjointly Ġtraining Ġthe Ġsentence Ġplanning Ġand Ġsurface Ġrealization Ġto Ġproduce Ġnatural Ġlanguage Ġsentences . ĠWe Ġconducted Ġexperiments Ġon Ġfour Ġdifferent ĠNL G Ġdomains Ġand Ġfound Ġthat Ġthe Ġproposed Ġmethods Ġsignificantly Ġoutper formed Ġthe Ġstate - of - the - art Ġmethods Ġregarding ĠB LE U ĠB IB REF 15 Ġand Ġslot Ġerror Ġrate ĠER R Ġscores ĠB IB REF 4 Ġ. ĠThe Ġresults Ġalso Ġshowed Ġthat Ġour Ġgenerators Ġcould Ġscale Ġto Ġnew Ġdomains Ġby Ġleveraging Ġthe Ġout - of - domain Ġdata . ĠTo Ġsum Ġup , Ġwe Ġmake Ġthree Ġkey Ġcontributions Ġin Ġthis Ġpaper : </s> We Ġreview Ġrelated Ġworks Ġin ĠSection Ġ" Related ĠWork " Ġ. ĠFollowing Ġa Ġdetail Ġof Ġproposed Ġmodel Ġin ĠSection Ġ" Rec urrent ĠNeural ĠLanguage ĠGenerator " Ġ, ĠSection Ġ" Exper iments " Ġdescribes Ġdatasets , Ġexperimental Ġsetups , Ġand Ġevaluation Ġmetrics . ĠThe Ġresulting Ġanalysis Ġis Ġpresented Ġin ĠSection Ġ" Results Ġand ĠAnalysis " Ġ. ĠWe Ġconclude Ġwith Ġa Ġbrief Ġsummary Ġand Ġfuture Ġwork Ġin ĠSection Ġ" Conclusion Ġand ĠFuture ĠWork " Ġ. </s> Related ĠWork </s> Recently , ĠR NN s - based Ġmodels Ġhave Ġshown Ġpromising Ġperformance Ġin Ġtackling Ġthe ĠNL G Ġproblems . ĠB IB REF 16 Ġproposed Ġa Ġgenerator Ġusing ĠR NN s Ġto Ġcreate ĠChinese Ġpoetry . ĠB IB REF 11 Ġ, ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġalso Ġused ĠR NN s Ġin Ġa Ġmulti - mod al Ġsetting Ġto Ġsolve Ġimage Ġcaption ing Ġtasks . ĠThe ĠR NN - based ĠSequence Ġto ĠSequence Ġmodels Ġhave Ġapplied Ġto Ġsolve Ġvariety Ġof Ġtasks : Ġconvers ational Ġmodeling ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 19 Ġ, Ġmachine Ġtranslation ĠB IB REF 20 Ġ, ĠB IB REF 21 </s> For Ġtask - oriented Ġdialogue Ġsystems , ĠB IB REF 3 Ġcombined Ġa Ġforward ĠR NN Ġgenerator , Ġa ĠCNN Ġre rank er , Ġand Ġa Ġbackward ĠR NN Ġre rank er Ġto Ġgenerate Ġutter ances . ĠB IB REF 4 Ġproposed ĠSC - L ST M Ġgenerator Ġwhich Ġintroduced Ġa Ġcontrol Ġs igm oid Ġgate Ġto Ġthe ĠL ST M Ġcell Ġto Ġjointly Ġlearn Ġthe Ġg ating Ġmechanism Ġand Ġlanguage Ġmodel . ĠA Ġrecurring Ġproblem Ġin Ġsuch Ġsystems Ġis Ġthe Ġlack Ġof Ġsufficient Ġdomain - specific Ġannot ated Ġdata . ĠB IB REF 22 Ġproposed Ġan Ġout - of - domain Ġmodel Ġwhich Ġwas Ġtrained Ġon Ġcounterfe ited Ġdata Ġby Ġusing Ġsem antically Ġsimilar Ġslots Ġfrom Ġthe Ġtarget Ġdomain Ġinstead Ġof Ġthe Ġslots Ġbelonging Ġto Ġthe Ġout - of - domain Ġdataset . ĠThe Ġresults Ġshowed Ġthat Ġthe Ġmodel Ġcan Ġachieve Ġa Ġsatisfactory Ġperformance Ġwith Ġa Ġsmall Ġamount Ġof Ġin - domain Ġdata Ġby Ġfine Ġtuning Ġthe Ġtarget Ġdomain Ġon Ġthe Ġout - of - domain Ġtrained Ġmodel . </s> More Ġrecently , ĠR NN Ġenc oder - dec oder Ġbased Ġmodels Ġwith Ġattention Ġmechanism ĠB IB REF 10 Ġhave Ġshown Ġimproved Ġperformances Ġin Ġvarious Ġtasks . ĠB IB REF 12 Ġproposed Ġa Ġreview Ġnetwork Ġto Ġthe Ġimage Ġcaption ing , Ġwhich Ġreviews Ġall Ġthe Ġinformation Ġencoded Ġby Ġthe Ġenc oder Ġand Ġproduces Ġa Ġcompact Ġthought Ġvector . ĠB IB REF 9 Ġproposed ĠR NN Ġenc oder - dec oder - based Ġmodel Ġby Ġusing Ġtwo Ġattention Ġlayers Ġto Ġjointly Ġtrain Ġcontent Ġselection Ġand Ġsurface Ġrealization . ĠMore Ġclose Ġto Ġour Ġwork , ĠB IB REF 8 Ġproposed Ġan Ġattentive Ġenc oder - dec oder Ġbased Ġgenerator Ġwhich Ġcomputed Ġthe Ġattention Ġmechanism Ġover Ġthe Ġslot - value Ġpairs . ĠThe Ġmodel Ġshowed Ġa Ġdomain Ġscal ability Ġwhen Ġa Ġvery Ġlimited Ġamount Ġof Ġdata Ġis Ġavailable . </s> Moving Ġfrom Ġa Ġlimited Ġdomain Ġdialogue Ġsystem Ġto Ġan Ġopen Ġdomain Ġdialogue Ġsystem Ġraises Ġsome Ġissues . ĠTherefore , Ġit Ġis Ġimportant Ġto Ġbuild Ġan Ġopen Ġdomain Ġdialogue Ġsystem Ġthat Ġcan Ġmake Ġas Ġmuch Ġuse Ġof Ġexisting Ġabilities Ġof Ġfunctioning Ġfrom Ġother Ġdomains . ĠThere Ġhave Ġbeen Ġseveral Ġworks Ġto Ġtackle Ġthis Ġproblem , Ġsuch Ġas ĠB IB REF 23 Ġusing ĠR NN - based Ġnetworks Ġfor Ġmulti - domain Ġdialogue Ġstate Ġtracking , ĠB IB REF 22 Ġusing Ġa Ġprocedure Ġto Ġtrain Ġmulti - domain Ġvia Ġmultiple Ġadaptation Ġsteps , Ġor ĠB IB REF 24 Ġ, ĠB IB REF 25 Ġadapting Ġof ĠS DS Ġcomponents Ġto Ġnew Ġdomains . </s> Rec urrent ĠNeural ĠLanguage ĠGenerator </s> The Ġrecurrent Ġlanguage Ġgenerator Ġproposed Ġin Ġthis Ġpaper Ġis Ġbased Ġon Ġa Ġneural Ġlanguage Ġgenerator ĠB IB REF 8 Ġ, Ġwhich Ġconsists Ġof Ġthree Ġmain Ġcomponents : Ġ( i ) Ġan ĠEnc oder Ġthat Ġincorporates Ġthe Ġtarget Ġmeaning Ġrepresentation Ġ( MR ) Ġas Ġthe Ġmodel Ġinputs , Ġ( ii ) Ġan ĠAl ign er Ġthat Ġalign s Ġand Ġcontrols Ġthe Ġsemantic Ġelements , Ġand Ġ( iii ) Ġan ĠR NN ĠDecoder Ġthat Ġgenerates Ġoutput Ġsentences . ĠThe Ġgenerator Ġarchitecture Ġis Ġshown Ġin ĠFigure Ġ1 Ġ. ĠThe ĠEnc oder Ġfirst Ġenc odes Ġthe ĠMR Ġinto Ġinput Ġsemantic Ġelements Ġwhich Ġare Ġthen Ġaggreg ated Ġand Ġselected Ġby Ġutilizing Ġan Ġattention - based Ġmechanism Ġby Ġthe ĠAl ign er . ĠThe Ġinput Ġto Ġthe ĠR NN ĠDecoder Ġat Ġeach Ġtime Ġstep Ġis Ġa Ġ1 - hot Ġencoding Ġof Ġa Ġtoken Ġ$\ text bf Ġ{ w } _{ t } $ Ġand Ġan Ġattentive ĠDA Ġrepresentation Ġ$\ text bf Ġ{ d } _{ t } $ Ġ. ĠAt Ġeach Ġtime Ġstep Ġ$ t $ Ġ, ĠR NN ĠDecoder Ġalso Ġcomp utes Ġhow Ġmuch Ġthe Ġfeature Ġvalue Ġvector Ġ$\ text bf Ġ{ s } _{ t - 1 } $ Ġretained Ġfor Ġthe Ġnext Ġcomputational Ġsteps , Ġand Ġadds Ġthis Ġinformation Ġto Ġthe ĠR NN Ġoutput Ġwhich Ġrepresents Ġthe Ġprobability Ġdistribution Ġof Ġthe Ġnext Ġtoken Ġ$\ text bf Ġ{ w } _{ t + 1 } $ Ġ. ĠAt Ġgeneration Ġtime , Ġwe Ġcan Ġsample Ġfrom Ġthis Ġconditional Ġdistribution Ġto Ġobtain Ġthe Ġnext Ġtoken Ġin Ġa Ġgenerated Ġsentence , Ġand Ġfeed Ġit Ġas Ġthe Ġnext Ġinput Ġto Ġthe ĠR NN ĠDecoder . ĠThis Ġprocess Ġfinishes Ġwhen Ġan Ġend Ġsign Ġis Ġgenerated ĠB IB REF 17 Ġ, Ġor Ġsome Ġconstraints Ġare Ġreached ĠB IB REF 16 Ġ. ĠThe Ġmodel Ġcan Ġproduce Ġa Ġsequence Ġof Ġtokens Ġwhich Ġcan Ġfinally Ġbe Ġlex ical ized Ġto Ġform Ġthe Ġrequired Ġutter ance . </s> Enc oder </s> The Ġslots Ġand Ġvalues Ġare Ġseparated Ġparameters Ġused Ġin Ġthe Ġenc oder Ġside . ĠThis Ġembed s Ġthe Ġsource Ġinformation Ġinto Ġa Ġvector Ġrepresentation Ġ$\ text bf Ġ{ z } _{ i } $ Ġwhich Ġis Ġa Ġconc aten ation Ġof Ġembed ding Ġvector Ġrepresentation Ġof Ġeach Ġslot - value Ġpair , Ġand Ġis Ġcomputed Ġby : </s> $$ \ text bf Ġ{ z } _{ i } Ġ= Ġ\ text bf Ġ{ u } _{ i } Ġ\ opl us Ġ\ text bf Ġ{ v } _{ i } $$ Ġ Ġ Ġ( E q . Ġ10 ) </s> where Ġ$\ text bf Ġ{ u } _{ i } $ Ġ, Ġ$\ text bf Ġ{ v } _{ i } $ Ġare Ġthe Ġ$ i $ Ġ- th Ġslot Ġand Ġvalue Ġembed ding Ġvectors , Ġrespectively , Ġand Ġ$\ opl us Ġ$ Ġis Ġvector Ġconc aten ation . ĠThe Ġi Ġindex Ġruns Ġover Ġthe Ġ$ L $ Ġgiven Ġslot - value Ġpairs . ĠIn Ġthis Ġwork , Ġwe Ġuse Ġa Ġ1 - layer , ĠBid irection al ĠL ST M Ġ( Bi - L ST M ) Ġto Ġencode Ġthe Ġsequence Ġof Ġslot - value Ġpairs Ġembed ding . ĠThe ĠBi - L ST M Ġconsists Ġof Ġforward Ġand Ġbackward ĠL ST Ms Ġwhich Ġread Ġthe Ġsequence Ġof Ġslot - value Ġpairs Ġfrom Ġleft - to - right Ġand Ġright - to - left Ġto Ġproduce Ġforward Ġand Ġbackward Ġsequence Ġof Ġhidden Ġstates Ġ( Ġ$\ over right arrow {\ text bf Ġ{ e } _{ 1 } }, Ġ.. , Ġ\ over right arrow {\ text bf Ġ{ e } _{ L }} $ Ġ), Ġand Ġ( Ġ$\ over left arrow {\ text bf Ġ{ e } _{ 1 } }, Ġ.. , Ġ\ over left arrow {\ text bf Ġ{ e } _{ L }} $ Ġ), Ġrespectively . ĠWe Ġthen Ġobtain Ġthe Ġsequence Ġof Ġencoded Ġhidden Ġstates Ġ$\ text bf Ġ{ E } =( \ text bf Ġ{ e } _{ 1 }, Ġ\ text bf Ġ{ e } _{ 2 }, Ġ.. , Ġ\ text bf Ġ{ e } _{ L }) $ Ġwhere Ġ$\ text bf Ġ{\ text bf Ġ{ e }} _{ i } $ Ġis Ġa Ġsum Ġof Ġthe Ġforward Ġhidden Ġstate Ġ$\ over right arrow {\ text bf Ġ{ e } _{ i }} $ Ġand Ġthe Ġbackward Ġone Ġ$\ text bf Ġ{ v } _{ i } $ 0 Ġas Ġfollows : </s> $$ \ text bf Ġ{ e } _{ i } = \ over right arrow {\ text bf Ġ{ e } _{ i }} Ġ+ Ġ\ over left arrow {\ text bf Ġ{ e } _{ i }} $$ Ġ Ġ Ġ( E q . Ġ12 ) </s> Al ign er </s> The ĠAl ign er Ġutilizes Ġattention Ġmechanism Ġto Ġcalculate Ġthe ĠDA Ġrepresentation Ġas Ġfollows : </s> $$ \ beta Ġ_ { t , i } Ġ= Ġ\ frac {\ exp Ġe _{ t , i } Ġ} {\ sum Ġ\ n ol im its Ġ_ { j }\ exp Ġe _{ t , j }} $$ Ġ Ġ Ġ( E q . Ġ14 ) </s> where </s> $$ e _{ t , i } = a (\ text bf Ġ{ e } _{ i }, Ġ\ text bf Ġ{ h } _{ t - 1 }) $$ Ġ Ġ Ġ( E q . Ġ15 ) </s> and Ġ$\ beta Ġ_ { t , i } $ Ġis Ġthe Ġweight Ġof Ġi - th Ġslot - value Ġpair Ġcalculated Ġby Ġthe Ġattention Ġmechanism . ĠThe Ġalignment Ġmodel Ġ$ a $ Ġis Ġcomputed Ġby : </s> $$ a (\ text bf Ġ{ e } _{ i }, Ġ\ text bf Ġ{ h } _{ t - 1 }) Ġ= Ġ\ text bf Ġ{ v } _{ a } ^ {\ top Ġ} \ tan h Ġ( \ text bf Ġ{ W } _{ a }\ text bf Ġ{ e } _{ i } Ġ+ Ġ\ text bf Ġ{ U } _{ a }\ text bf Ġ{ h } _{ t - 1 }) $$ Ġ Ġ Ġ( E q . Ġ16 ) </s> where Ġ$\ text bf Ġ{ v } _{ a }, Ġ\ text bf Ġ{ W } _{ a }, Ġ\ text bf Ġ{ U } _{ a } $ Ġare Ġthe Ġweight Ġmat rices Ġto Ġlearn . ĠFinally , Ġthe ĠAl ign er Ġcalculates Ġdialogue Ġact Ġembed ding Ġ$\ text bf Ġ{ d } _{ t } $ Ġas Ġfollows : </s> $$ \ text bf Ġ{ d } _{ t } Ġ= Ġ\ text bf Ġ{ a } Ġ\ opl us Ġ\ sum Ġ\ n ol im its Ġ_ { i }\ beta Ġ_ { t , i } Ġ\ text bf Ġ{ e } _{ i } $$ Ġ Ġ Ġ( E q . Ġ17 ) </s> where Ġa Ġis Ġvector Ġembed ding Ġof Ġthe Ġaction Ġtype . </s> RAL ST M ĠDecoder </s> The Ġproposed Ġsemantic ĠR AL ST M Ġcell Ġapplied Ġfor ĠDecoder Ġside Ġconsists Ġof Ġthree Ġcomponents : Ġa ĠRef inement Ġcell , Ġa Ġtraditional ĠL ST M Ġcell , Ġand Ġan ĠAdjust ment Ġcell : </s> Firstly , Ġinstead Ġof Ġfeeding Ġthe Ġoriginal Ġinput Ġtoken Ġ$\ text bf Ġ{ w } _{ t } $ Ġinto Ġthe ĠR NN Ġcell , Ġthe Ġinput Ġis Ġrec om puted Ġby Ġusing Ġa Ġsemantic Ġgate Ġas Ġfollows : </s> $$ \ begin { aligned } Ġ\ text bf Ġ{ r } _{ t } & = \ s igma Ġ( \ text bf Ġ{ W } _{ rd }\ text bf Ġ{ d } _{ t } Ġ+ Ġ\ text bf Ġ{ W } _{ rh }\ text bf Ġ{ h } _{ t - 1 }) \\ Ġ\ text bf Ġ{ x } _{ t } & = \ text bf Ġ{ r } _{ t } Ġ\ od ot Ġ\ text bf Ġ{ w } _{ t } Ġ\ end { aligned } $$ Ġ Ġ Ġ( E q . Ġ19 ) </s> where Ġ$\ text bf Ġ{ W } _{ rd } $ Ġand Ġ$\ text bf Ġ{ W } _{ rh } $ Ġare Ġweight Ġmat rices . ĠElement - wise Ġmultiplication Ġ$\ od ot Ġ$ Ġplays Ġa Ġpart Ġin Ġword - level Ġmatching Ġwhich Ġnot Ġonly Ġlearns Ġthe Ġvector Ġsimilarity , Ġbut Ġalso Ġpreserves Ġinformation Ġabout Ġthe Ġtwo Ġvectors . Ġ$\ text bf Ġ{ W } _{ rh } $ Ġacts Ġlike Ġa Ġkey Ġphrase Ġdetector Ġthat Ġlearns Ġto Ġcapture Ġthe Ġpattern Ġof Ġgeneration Ġtokens Ġor Ġthe Ġrelationship Ġbetween Ġmultiple Ġtokens . ĠIn Ġother Ġwords , Ġthe Ġnew Ġinput Ġ$\ text bf Ġ{ x } _{ t } $ Ġconsists Ġof Ġinformation Ġof Ġthe Ġoriginal Ġinput Ġtoken Ġ$\ text bf Ġ{ w } _{ t } $ Ġ, Ġthe ĠDA Ġrepresentation Ġ$\ text bf Ġ{ d } _{ t } $ Ġ, Ġand Ġthe Ġhidden Ġcontext Ġ$\ text bf Ġ{ h } _{ t - 1 } $ Ġ. Ġ$\ text bf Ġ{ r } _{ t } $ Ġis Ġcalled Ġa ĠRef inement Ġgate Ġbecause Ġthe Ġinput Ġtokens Ġare Ġrefined Ġby Ġa Ġcombination Ġg ating Ġinformation Ġof Ġthe Ġattentive ĠDA Ġrepresentation Ġ$\ text bf Ġ{ d } _{ t } $ Ġand Ġthe Ġprevious Ġhidden Ġstate Ġ$\ text bf Ġ{ W } _{ rh } $ 0 Ġ. ĠBy Ġthis Ġway , Ġwe Ġcan Ġrepresent Ġthe Ġwhole Ġsentence Ġbased Ġon Ġthe Ġrefined Ġinputs . </s> Secondly , Ġthe Ġtraditional ĠL ST M Ġnetwork Ġproposed Ġby ĠB IB REF 26 Ġb ah dan au 2014 ne ural Ġin Ġwhich Ġthe Ġinput Ġgate Ġ$\ text bf Ġ{ i } _{ i } $ Ġ, Ġforget Ġgate Ġ$\ text bf Ġ{ f } _{ t } $ Ġand Ġoutput Ġgates Ġ$\ text bf Ġ{ o } _{ t } $ Ġare Ġintroduced Ġto Ġcontrol Ġinformation Ġflow Ġand Ġcomputed Ġas Ġfollows : </s> $$ \ begin { aligned } Ġ\ begin { pm at rix } Ġ\ text bf Ġ{ i } _{ t } Ġ\\ Ġ\ text bf Ġ{ f } _{ t } Ġ\\ Ġ\ text bf Ġ{ o } _{ t } Ġ\\ Ġ\ hat {\ text bf Ġ{ c }} _{ t } Ġ\ end { pm at rix } Ġ& = Ġ\ begin { pm at rix }\ s igma Ġ\\ Ġ\ s igma Ġ\\ Ġ\ s igma Ġ\\ Ġ\ tan h Ġ\ end { pm at rix }\ text bf Ġ{ W } _{ 4 n , 4 n } Ġ\ begin { pm at rix } Ġ\ text bf Ġ{ x } _{ t } Ġ\\ Ġ\ text bf Ġ{ d } _{ t } Ġ\\ Ġ\ text bf Ġ{ h } _{ t - 1 } Ġ\ end { pm at rix } \\ Ġ\ end { aligned } $$ Ġ Ġ Ġ( E q . Ġ20 ) </s> where Ġ$ n $ Ġis Ġhidden Ġlayer Ġsize , Ġ$\ text bf Ġ{ W } _{ 4 n , 4 n } $ Ġis Ġmodel Ġparameters . ĠThe Ġcell Ġmemory Ġvalue Ġ$\ text bf Ġ{ c } _{ t } $ Ġis Ġmodified Ġto Ġdepend Ġon Ġthe ĠDA Ġrepresentation Ġas : </s> $$ \ begin { aligned } Ġ\ text bf Ġ{ c } _{ t } & = \ text bf Ġ{ f } _{ t }\ od ot Ġ\ text bf Ġ{ c } _{ t - 1 } Ġ+ \ text bf Ġ{ i } _{ t }\ od ot Ġ\ hat {\ text bf Ġ{ c }} _{ t } Ġ+ Ġ\ tan h Ġ( \ text bf Ġ{ W } _{ cr }\ text bf Ġ{ r } _{ t }) Ġ\\ Ġ\ t ilde {\ text bf Ġ{ h }} _{ t } & = Ġ\ text bf Ġ{ o } _{ t } Ġ\ od ot Ġ\ tan h Ġ( \ text bf Ġ{ c } _{ t }) Ġ\ end { aligned } $$ Ġ Ġ Ġ( E q . Ġ21 ) </s> where Ġ$\ t ilde {\ text bf Ġ{ h }} _{ t } $ Ġis Ġthe Ġoutput . </s> Third ly , Ġinspired Ġby Ġwork Ġof ĠB IB REF 4 Ġin Ġwhich Ġthe Ġgenerator Ġwas Ġfurther Ġconditioned Ġon Ġa Ġ1 - hot Ġrepresentation Ġvector Ġ$\ text bf Ġ{ s } $ Ġof Ġgiven Ġdialogue Ġact , Ġand Ġwork Ġof ĠB IB REF 27 Ġthat Ġproposed Ġa Ġvisual Ġsent inel Ġgate Ġto Ġmake Ġa Ġdecision Ġon Ġwhether Ġthe Ġmodel Ġshould Ġattend Ġto Ġthe Ġimage Ġor Ġto Ġthe Ġsent inel Ġgate , Ġan Ġadditional Ġg ating Ġcell Ġis Ġintroduced Ġon Ġtop Ġof Ġthe Ġtraditional ĠL ST M Ġto Ġgate Ġanother Ġcontrolling Ġvector Ġ$\ text bf Ġ{ s } $ Ġ. ĠFigure Ġ6 Ġshows Ġhow ĠR AL ST M Ġcontrols Ġthe ĠDA Ġvector Ġ$\ text bf Ġ{ s } $ Ġ. ĠFirst , Ġstarting Ġfrom Ġthe Ġ1 - hot Ġvector Ġof Ġthe ĠDA Ġ$\ text bf Ġ{ s } _{ 0 } $ Ġ, Ġat Ġeach Ġtime Ġstep Ġ$ t $ Ġthe Ġproposed Ġcell Ġcomp utes Ġhow Ġmuch Ġthe ĠL ST M Ġoutput Ġ$\ t ilde {\ text bf Ġ{ h }} _{ t } $ Ġaffects Ġthe ĠDA Ġvector , Ġwhich Ġis Ġcomputed Ġas Ġfollows : </s> $$ \ begin { aligned } Ġ\ text bf Ġ{ a } _{ t } & = \ s igma Ġ( \ text bf Ġ{ W } _{ ax }\ text bf Ġ{ x } _{ t } Ġ+ \ text bf Ġ{ W } _{ ah }\ t ilde {\ text bf Ġ{ h }} _{ t }) \\ Ġ\ text bf Ġ{ s } _{ t } & = \ text bf Ġ{ s } _{ t - 1 } Ġ\ od ot Ġ\ text bf Ġ{ a } _{ t } Ġ\ end { aligned } $$ Ġ Ġ Ġ( E q . Ġ22 ) </s> where Ġ$\ text bf Ġ{ W } _{ ax } $ Ġ, Ġ$\ text bf Ġ{ W } _{ ah } $ Ġare Ġweight Ġmat rices Ġto Ġbe Ġlearned . Ġ$\ text bf Ġ{ a } _{ t } $ Ġis Ġcalled Ġan Ġ$ Adjust ment $ Ġgate Ġsince Ġits Ġtask Ġis Ġto Ġcontrol Ġwhat Ġinformation Ġof Ġthe Ġgiven ĠDA Ġhave Ġbeen Ġgenerated Ġand Ġwhat Ġinformation Ġshould Ġbe Ġretained Ġfor Ġfuture Ġtime Ġsteps . ĠSecond , Ġwe Ġconsider Ġhow Ġmuch Ġthe Ġinformation Ġpreserved Ġin Ġthe ĠDA Ġ$\ text bf Ġ{ s } _{ t } $ Ġcan Ġbe Ġcontributed Ġto Ġthe Ġoutput , Ġin Ġwhich Ġan Ġadditional Ġoutput Ġis Ġcomputed Ġby Ġapplying Ġthe Ġoutput Ġgate Ġ$\ text bf Ġ{ o } _{ t } $ Ġon Ġthe Ġremaining Ġinformation Ġin Ġ$\ text bf Ġ{ s } _{ t } $ Ġas Ġfollows : </s> $$ \ begin { aligned } Ġ\ text bf Ġ{ c } _{ a } & = \ s igma Ġ( \ text bf Ġ{ W } _{ os }\ text bf Ġ{ s } _{ t }) \\ Ġ\ t ilde {\ text bf Ġ{ h }} _{ a } & = Ġ\ text bf Ġ{ o } _{ t } Ġ\ od ot Ġ\ tan h Ġ( \ text bf Ġ{ c } _{ a }) Ġ\ end { aligned } $$ Ġ Ġ Ġ( E q . Ġ23 ) </s> where Ġ$\ text bf Ġ{ W } _{ os } $ Ġis Ġa Ġweight Ġmatrix Ġto Ġproject Ġthe ĠDA Ġpresentation Ġinto Ġthe Ġoutput Ġspace , Ġ$\ t ilde {\ text bf Ġ{ h }} _{ a } $ Ġis Ġthe ĠAdjust ment Ġcell Ġoutput . ĠFinal ĠR AL ST M Ġoutput Ġis Ġa Ġcombination Ġof Ġboth Ġoutputs Ġof Ġthe Ġtraditional ĠL ST M Ġcell Ġand Ġthe ĠAdjust ment Ġcell , Ġand Ġcomputed Ġas Ġfollows : </s> $$ \ text bf Ġ{ h } _{ t } = \ t ilde {\ text bf Ġ{ h }} _{ t } Ġ+ Ġ\ t ilde {\ text bf Ġ{ h }} _{ a } $$ Ġ Ġ Ġ( E q . Ġ24 ) </s> Finally , Ġthe Ġoutput Ġdistribution Ġis Ġcomputed Ġby Ġapplying Ġa Ġsoft max Ġfunction Ġ$ g $ Ġ, Ġand Ġthe Ġdistribution Ġcan Ġbe Ġsampled Ġto Ġobtain Ġthe Ġnext Ġtoken , </s> $$ \ begin { aligned } Ġ& ĠP ( w _{ t + 1 }\ mid Ġw _{ t }, ... w _{ 0 }, \ text bf Ġ{ DA }) = g (\ text bf Ġ{ W } _{ ho }\ text bf Ġ{ h } _{ t }) Ġ\\ Ġ& Ġw _{ t + 1 } Ġ\ sim ĠP ( w _{ t + 1 }\ mid Ġw _{ t }, Ġw _{ t - 1 }, ... w _{ 0 }, \ text bf Ġ{ DA }) Ġ\ end { aligned } $$ Ġ Ġ Ġ( E q . Ġ25 ) </s> where Ġ$\ text bf Ġ{ DA } =( \ text bf Ġ{ s }, Ġ\ text bf Ġ{ z }) $ Ġ. </s> Training </s> The Ġobjective Ġfunction Ġwas Ġthe Ġnegative Ġlog - like lihood Ġand Ġcomputed Ġby : </s> $$ \ text bf Ġ{ F } (\ the ta Ġ) Ġ= Ġ- \ sum Ġ_ { t = 1 } ^{ T }\ text bf Ġ{ y } _{ t } ^ {\ top Ġ} \ log Ġ{\ text bf Ġ{ p } _{ t }} $$ Ġ Ġ Ġ( E q . Ġ27 ) </s> where : Ġ$\ text bf Ġ{ y } _{ t } $ Ġis Ġthe Ġground Ġtruth Ġtoken Ġdistribution , Ġ$\ text bf Ġ{ p } _{ t } $ Ġis Ġthe Ġpredicted Ġtoken Ġdistribution , Ġ$ T $ Ġis Ġlength Ġof Ġthe Ġinput Ġsentence . ĠThe Ġproposed Ġgenerators Ġwere Ġtrained Ġby Ġtreating Ġeach Ġsentence Ġas Ġa Ġmini - batch Ġwith Ġ$ l _{ 2 } $ Ġregular ization Ġadded Ġto Ġthe Ġobjective Ġfunction Ġfor Ġevery Ġ5 Ġtraining Ġexamples . ĠThe Ġmodels Ġwere Ġinitialized Ġwith Ġa Ġpret rained ĠGl ove Ġword Ġembed ding Ġvectors ĠB IB REF 28 Ġand Ġoptimized Ġby Ġusing Ġsto ch astic Ġgradient Ġdescent Ġand Ġback Ġpropagation Ġthrough Ġtime ĠB IB REF 29 Ġ. ĠEarly Ġstopping Ġmechanism Ġwas Ġimplemented Ġto Ġprevent Ġover - fitting Ġby Ġusing Ġa Ġvalidation Ġset Ġas Ġsuggested Ġin ĠB IB REF 30 Ġ. </s> Dec oding </s> The Ġdecoding Ġconsists Ġof Ġtwo Ġphases : Ġ( i ) Ġover - generation , Ġand Ġ( ii ) Ġre ranking . ĠIn Ġthe Ġover - generation , Ġthe Ġgenerator Ġconditioned Ġon Ġboth Ġrepresentations Ġof Ġthe Ġgiven ĠDA Ġuse Ġa Ġbeam Ġsearch Ġto Ġgenerate Ġa Ġset Ġof Ġcandidate Ġresponses . ĠIn Ġthe Ġre ranking Ġphase , Ġcost Ġof Ġthe Ġgenerator Ġis Ġcomputed Ġto Ġform Ġthe Ġre ranking Ġscore Ġ$\ text bf Ġ{ R } $ Ġas Ġfollows : </s> $$ \ text bf Ġ{ R } Ġ= Ġ\ text bf Ġ{ F } (\ the ta Ġ) Ġ+ Ġ\ lambda Ġ\ text bf Ġ{ ER R } $$ Ġ Ġ Ġ( E q . Ġ29 ) </s> where Ġ$\ lambda Ġ$ Ġis Ġa Ġtrade Ġoff Ġconstant Ġand Ġis Ġset Ġto Ġa Ġlarge Ġvalue Ġin Ġorder Ġto Ġseverely Ġpenal ize Ġnonsensical Ġoutputs . ĠThe Ġslot Ġerror Ġrate Ġ$\ text bf Ġ{ ER R } $ Ġ, Ġwhich Ġis Ġthe Ġnumber Ġof Ġslots Ġgenerated Ġthat Ġis Ġeither Ġmissing Ġor Ġredundant , Ġand Ġis Ġcomputed Ġby : </s> $$ \ text bf Ġ{ ER R } Ġ= Ġ\ frac {\ text bf Ġ{ p } Ġ+ Ġ\ text bf Ġ{ q }} {\ text bf Ġ{ N }} $$ Ġ Ġ Ġ( E q . Ġ30 ) </s> where Ġ$\ text bf Ġ{ N } $ Ġis Ġthe Ġtotal Ġnumber Ġof Ġslots Ġin ĠDA , Ġand Ġ$\ text bf Ġ{ p } $ Ġ, Ġ$\ text bf Ġ{ q } $ Ġis Ġthe Ġnumber Ġof Ġmissing Ġand Ġredundant Ġslots , Ġrespectively . </s> Exper iments </s> We Ġextensively Ġconducted Ġa Ġset Ġof Ġexperiments Ġto Ġassess Ġthe Ġeffectiveness Ġof Ġthe Ġproposed Ġmodels Ġby Ġusing Ġseveral Ġmetrics , Ġdatasets , Ġand Ġmodel Ġarchitectures , Ġin Ġorder Ġto Ġcompare Ġto Ġprior Ġmethods . </s> Dat as ets </s> We Ġassessed Ġthe Ġproposed Ġmodels Ġon Ġfour Ġdifferent ĠNL G Ġdomains : Ġfinding Ġa Ġrestaurant , Ġfinding Ġa Ġhotel , Ġbuying Ġa Ġlaptop , Ġand Ġbuying Ġa Ġtelevision . ĠThe ĠRestaurant Ġand ĠHotel Ġwere Ġcollected Ġin ĠB IB REF 4 Ġ, Ġwhile Ġthe ĠL aptop Ġand ĠTV Ġdatasets Ġhave Ġbeen Ġreleased Ġby ĠB IB REF 22 Ġwith Ġa Ġmuch Ġlarger Ġinput Ġspace Ġbut Ġonly Ġone Ġtraining Ġexample Ġfor Ġeach ĠDA Ġso Ġthat Ġthe Ġsystem Ġmust Ġlearn Ġpartial Ġrealization Ġof Ġconcepts Ġand Ġbe Ġable Ġto Ġrecomb ine Ġand Ġapply Ġthem Ġto Ġunseen ĠD As . ĠThis Ġmakes Ġthe ĠNL G Ġtasks Ġfor Ġthe ĠL aptop Ġand ĠTV Ġdomains Ġbecome Ġmuch Ġharder . ĠThe Ġdataset Ġstatistics Ġare Ġshown Ġin ĠTable Ġ1 Ġ. </s> Exper imental ĠSet ups </s> The Ġgenerators Ġwere Ġimplemented Ġusing Ġthe ĠT ensor Flow Ġlibrary ĠB IB REF 31 Ġand Ġtrained Ġwith Ġtraining , Ġvalidation Ġand Ġtesting Ġratio Ġas Ġ3 : 1 : 1 . ĠThe Ġhidden Ġlayer Ġsize , Ġbeam Ġsize Ġwere Ġset Ġto Ġbe Ġ80 Ġand Ġ10 , Ġrespectively , Ġand Ġthe Ġgenerators Ġwere Ġtrained Ġwith Ġa Ġ$ 70 \ % $ Ġof Ġdrop out Ġrate . ĠWe Ġperformed Ġ5 Ġruns Ġwith Ġdifferent Ġrandom Ġinitialization Ġof Ġthe Ġnetwork Ġand Ġthe Ġtraining Ġis Ġterminated Ġby Ġusing Ġearly Ġstopping . ĠWe Ġthen Ġchose Ġa Ġmodel Ġthat Ġyields Ġthe Ġhighest ĠB LE U Ġscore Ġon Ġthe Ġvalidation Ġset Ġas Ġshown Ġin ĠTable Ġ2 Ġ. ĠSince Ġthe Ġtrained Ġmodels Ġcan Ġdiffer Ġdepending Ġon Ġthe Ġinitialization , Ġwe Ġalso Ġreport Ġthe Ġresults Ġwhich Ġwere Ġaveraged Ġover Ġ5 Ġrandomly Ġinitialized Ġnetworks . ĠNote Ġthat , Ġexcept Ġthe Ġresults Ġreported Ġin ĠTable Ġ2 Ġ, Ġall Ġthe Ġresults Ġshown Ġwere Ġaveraged Ġover Ġ5 Ġrandomly Ġinitialized Ġnetworks . ĠWe Ġset Ġ$\ lambda Ġ$ Ġto Ġ1000 Ġto Ġseverely Ġdiscourage Ġthe Ġre rank er Ġfrom Ġselecting Ġutter ances Ġwhich Ġcontain Ġeither Ġredundant Ġor Ġmissing Ġslots . ĠFor Ġeach ĠDA , Ġwe Ġover - generated Ġ20 Ġcandidate Ġsentences Ġand Ġselected Ġthe Ġtop Ġ5 Ġreal izations Ġafter Ġre ranking . ĠMoreover , Ġin Ġorder Ġto Ġbetter Ġunderstand Ġthe Ġeffectiveness Ġof Ġour Ġproposed Ġmethods , Ġwe : Ġ( i ) Ġperformed Ġan Ġab lation Ġexperiments Ġto Ġdemonstrate Ġthe Ġcontribution Ġof Ġeach Ġproposed Ġcells Ġ( T ables Ġ2 Ġ, Ġ3 Ġ), Ġ( ii ) Ġtrained Ġthe Ġmodels Ġon Ġthe ĠL aptop Ġdomain Ġwith Ġvaried Ġproportion Ġof Ġtraining Ġdata , Ġstarting Ġfrom Ġ$ 10 \ % $ Ġto Ġ$ 100 \ % $ Ġ( Figure Ġ3 Ġ), Ġ( iii ) Ġtrained Ġgeneral Ġmodels Ġby Ġmerging Ġall Ġthe Ġdata Ġfrom Ġfour Ġdomains Ġtogether Ġand Ġtested Ġthem Ġin Ġeach Ġindividual Ġdomain Ġ( Figure Ġ4 Ġ), Ġand Ġ( iv ) Ġtrained Ġadaptation Ġmodels Ġon Ġmerging Ġdata Ġfrom Ġrestaurant Ġand Ġhotel Ġdomains , Ġthen Ġfine Ġtuned Ġthe Ġmodel Ġon Ġlaptop Ġdomain Ġwith Ġvaried Ġamount Ġof Ġadaptation Ġdata Ġ( Figure Ġ5 Ġ). </s> E val uation ĠMet rics Ġand ĠBas elines </s> The Ġgenerator Ġperformance Ġwas Ġassessed Ġon Ġthe Ġtwo Ġevaluation Ġmetrics : Ġthe ĠB LE U Ġand Ġthe Ġslot Ġerror Ġrate ĠER R Ġby Ġadopting Ġcode Ġfrom Ġan Ġopen Ġsource Ġbenchmark Ġtool kit Ġfor ĠNatural ĠLanguage ĠGeneration . ĠWe Ġcompared Ġthe Ġproposed Ġmodels Ġagainst Ġthree Ġstrong Ġbas elines Ġwhich Ġhave Ġbeen Ġrecently Ġpublished Ġas Ġstate - of - the - art ĠNL G Ġbenchmarks [ ]. Ġhttps :// github . com / sh awn w un / R NN LG </s> HL ST M Ġproposed Ġby ĠB IB REF 3 Ġth ws j y 15 Ġwhich Ġused Ġa Ġhe uristic Ġgate Ġto Ġensure Ġthat Ġall Ġof Ġthe Ġslot - value Ġinformation Ġwas Ġaccurately Ġcaptured Ġwhen Ġgenerating . </s> S CL ST M Ġproposed Ġby ĠB IB REF 4 Ġw ens cl st m 15 Ġwhich Ġcan Ġjointly Ġlearn Ġthe Ġg ating Ġsignal Ġand Ġlanguage Ġmodel . </s> Enc - Dec Ġproposed Ġby ĠB IB REF 8 Ġwent oward Ġwhich Ġapplied Ġthe Ġattention - based Ġenc oder - dec oder Ġarchitecture . </s> Results </s> We Ġconducted Ġextensive Ġexperiments Ġon Ġour Ġmodels Ġand Ġcompared Ġagainst Ġthe Ġprevious Ġmethods . ĠOverall , Ġthe Ġproposed Ġmodels Ġconsistently Ġachieve Ġthe Ġbetter Ġperformance Ġregarding Ġboth Ġevaluation Ġmetrics Ġacross Ġall Ġdomains Ġin Ġall Ġtest Ġcases . </s> The Ġab lation Ġstudies Ġ( T ables Ġ2 Ġ, Ġ3 Ġ) Ġdemonstrate Ġthe Ġcontribution Ġof Ġdifferent Ġmodel Ġcomponents Ġin Ġwhich Ġthe Ġmodels Ġwere Ġassessed Ġwithout ĠAdjust ment Ġcell Ġ( w / o ĠA ), Ġor Ġwithout ĠRef inement Ġcell Ġ( w / o ĠR ). ĠIt Ġclearly Ġsees Ġthat Ġthe ĠAdjust ment Ġcell Ġcontributes Ġto Ġreducing Ġthe Ġslot Ġerror Ġrate ĠER R Ġscore Ġsince Ġit Ġcan Ġeffectively Ġprevent Ġthe Ġundesirable Ġslot - value Ġpair Ġrepet itions Ġby Ġg ating Ġthe ĠDA Ġvector Ġ$\ text bf Ġ{ s } $ Ġ. ĠA Ġcomparison Ġbetween Ġthe ĠAR ED - based Ġmodels Ġ( den oted Ġby Ġ$ ^ {\ sharp Ġ} $ Ġin ĠTable Ġ2 Ġ) Ġshows Ġthat Ġthe Ġproposed Ġmodels Ġnot Ġonly Ġhave Ġbetter Ġperformance Ġwith Ġhigher Ġthe ĠB LE U Ġscore Ġbut Ġalso Ġsignificantly Ġreduce Ġthe Ġslot Ġerror Ġrate ĠER R Ġscore Ġby Ġa Ġlarge Ġmargin Ġabout Ġ$ 2 \ % $ Ġto Ġ$ 4 \ % $ Ġin Ġevery Ġdatasets . ĠMoreover , Ġa Ġcomparison Ġbetween Ġthe Ġmodels Ġwith Ġg ating Ġthe ĠDA Ġvector Ġalso Ġindicates Ġthat Ġthe Ġproposed Ġmodels Ġ( w / o ĠR , ĠR AL ST M ) Ġhave Ġsignificant Ġimproved Ġperformance Ġon Ġboth Ġthe Ġevaluation Ġmetrics Ġacross Ġthe Ġfour Ġdomains Ġcompared Ġto Ġthe ĠS CL ST M Ġmodel . ĠThe ĠR AL ST M Ġcell Ġwithout Ġthe ĠRef inement Ġcell Ġis Ġsimilar Ġas Ġthe ĠS CL ST M Ġcell . ĠHowever , Ġit Ġobtained Ġthe Ġresults Ġmuch Ġbetter Ġthan Ġthe ĠS CL ST M Ġbas elines . ĠThis Ġstip ulates Ġthe Ġnecessary Ġof Ġthe ĠL ST M Ġenc oder Ġand Ġthe ĠAl ign er Ġin Ġeffectively Ġpartial Ġlearning Ġthe Ġcorrelated Ġorder Ġbetween Ġslot - value Ġrepresentation Ġin Ġthe ĠD As , Ġespecially Ġfor Ġthe Ġunseen Ġdomain Ġwhere Ġthere Ġis Ġonly Ġone Ġtraining Ġexample Ġfor Ġeach ĠDA . ĠTable Ġ3 Ġfurther Ġdemonstrates Ġthe Ġstable Ġstrength Ġof Ġour Ġmodels Ġsince Ġthe Ġresults ' Ġpattern Ġstays Ġunchanged Ġcompared Ġto Ġthose Ġin ĠTable Ġ2 Ġ. </s> Figure Ġ3 Ġshows Ġa Ġcomparison Ġof Ġthree Ġmodels Ġ( Enc - Dec , ĠS CL ST M , Ġand ĠR AL ST M ) Ġwhich Ġwere Ġtrained Ġfrom Ġscratch Ġon Ġthe Ġunseen Ġlaptop Ġdomain Ġin Ġvaried Ġproportion Ġof Ġtraining Ġdata , Ġfrom Ġ$ 1 \ % $ Ġto Ġ$ 100 \ % $ Ġ. ĠIt Ġclearly Ġshows Ġthat Ġthe ĠR AL ST M Ġoutper forms Ġthe Ġprevious Ġmodels Ġin Ġall Ġcases , Ġwhile Ġthe ĠEnc - Dec Ġhas Ġa Ġmuch Ġgreater ĠER R Ġscore Ġcomparing Ġto Ġthe Ġtwo Ġmodels . </s> A Ġcomparison Ġof Ġtop Ġresponses Ġgenerated Ġfor Ġsome Ġinput ĠD As Ġbetween Ġdifferent Ġmodels Ġare Ġshown Ġin ĠTable Ġ4 Ġ. ĠWhile Ġthe Ġprevious Ġmodels Ġstill Ġproduce Ġsome Ġerrors Ġ( missing Ġand Ġmisplaced Ġinformation ), Ġthe Ġproposed Ġmodels Ġ( RAL ST M Ġand Ġthe Ġmodels ĠAll 2 * Ġtrained Ġby Ġpool ing Ġall Ġdatasets Ġtogether ) Ġcan Ġgenerate Ġappropriate Ġsentences . ĠWe Ġalso Ġfound Ġthat Ġthe Ġproposed Ġmodels Ġtend Ġto Ġgenerate Ġmore Ġcomplete Ġand Ġconcise Ġsentences Ġthan Ġthe Ġother Ġmodels . </s> All Ġthese Ġprove Ġthe Ġimportance Ġof Ġthe Ġproposed Ġcomponents : Ġthe ĠRef inement Ġcell Ġin Ġaggreg ating Ġand Ġselecting Ġthe Ġattentive Ġinformation , Ġand Ġthe ĠAdjust ment Ġcell Ġin Ġcontrolling Ġthe Ġfeature Ġvector Ġ( see ĠExamples Ġin ĠFigure Ġ6 Ġ). </s> Figure Ġ4 Ġshows Ġa Ġcomparison Ġperformance Ġof Ġgeneral Ġmodels Ġas Ġdescribed Ġin ĠSection Ġ" Exper imental ĠSet ups " Ġ. ĠThe Ġresults Ġare Ġconsistent Ġwith Ġthe ĠFigure Ġ3 Ġ, Ġin Ġwhich Ġthe ĠR AL ST M Ġhas Ġbetter Ġperformance Ġthan Ġthe ĠEnc - Dec Ġand ĠS CL ST M Ġon Ġall Ġdomains Ġin Ġterms Ġof Ġthe ĠB LE U Ġand Ġthe ĠER R Ġscores , Ġwhile Ġthe ĠEnc - Dec Ġhas Ġdifficulties Ġin Ġreducing Ġthe ĠER R Ġscore . ĠThis Ġindicates Ġthe Ġrelevant Ġcontribution Ġof Ġthe Ġproposed Ġcomponent ĠRef inement Ġand ĠAdjust ment Ġcells Ġto Ġthe Ġoriginal ĠAR ED Ġarchitecture , Ġin Ġwhich Ġthe ĠRef inement Ġwith Ġattention al Ġg ating Ġcan Ġeffectively Ġselect Ġand Ġaggregate Ġthe Ġinformation Ġbefore Ġputting Ġthem Ġinto Ġthe Ġtraditional ĠL ST M Ġcell , Ġwhile Ġthe ĠAdjust ment Ġwith Ġg ating ĠDA Ġvector Ġcan Ġeffectively Ġcontrol Ġthe Ġinformation Ġflow Ġduring Ġgeneration . </s> Figure Ġ5 Ġshows Ġdomain Ġscal ability Ġof Ġthe Ġthree Ġmodels Ġin Ġwhich Ġthe Ġmodels Ġwere Ġfirst Ġtrained Ġon Ġthe Ġmerging Ġout - of - domain ĠRestaurant Ġand ĠHotel Ġdatasets , Ġthen Ġfine Ġtuned Ġthe Ġparameters Ġwith Ġvaried Ġamount Ġof Ġin - domain Ġtraining Ġdata Ġ( l aptop Ġdomain ). ĠThe ĠR AL ST M Ġmodel Ġoutper forms Ġthe Ġprevious Ġmodel Ġin Ġboth Ġcases Ġwhere Ġthe Ġsufficient Ġin - domain Ġdata Ġis Ġused Ġ( as Ġin ĠFigure Ġ5 Ġ- left ) Ġand Ġthe Ġlimited Ġin - domain Ġdata Ġis Ġused Ġ( Figure Ġ5 Ġ- right ). ĠThe ĠFigure Ġ5 Ġ- right Ġalso Ġindicates Ġthat Ġthe ĠR AL ST M Ġmodel Ġcan Ġadapt Ġto Ġa Ġnew , Ġunseen Ġdomain Ġfaster Ġthan Ġthe Ġprevious Ġmodels . </s> Conclusion Ġand ĠFuture ĠWork </s> We Ġpresent Ġan Ġextension Ġof ĠAR ED Ġmodel , Ġin Ġwhich Ġan ĠR AL ST M Ġcomponent Ġis Ġintroduced Ġto Ġselect Ġand Ġaggregate Ġsemantic Ġelements Ġproduced Ġby Ġthe ĠEnc oder , Ġand Ġto Ġgenerate Ġthe Ġrequired Ġsentence . ĠWe Ġassessed Ġthe Ġproposed Ġmodels Ġon Ġfour ĠNL G Ġdomains Ġand Ġcompared Ġto Ġthe Ġstate - of - the - art Ġgenerators . ĠThe Ġproposed Ġmodels Ġempir ically Ġshow Ġconsistent Ġimprovement Ġover Ġthe Ġprevious Ġmethods Ġin Ġboth Ġthe ĠB LE U Ġand ĠER R Ġevaluation Ġmetrics . ĠThe Ġproposed Ġmodels Ġalso Ġshow Ġan Ġability Ġto Ġextend Ġto Ġa Ġnew , Ġunseen Ġdomain Ġno Ġmatter Ġhow Ġmuch Ġthe Ġin - domain Ġtraining Ġdata Ġwas Ġfed . ĠIn Ġthe Ġfuture , Ġit Ġwould Ġbe Ġinteresting Ġto Ġapply Ġthe Ġproposed Ġmodel Ġto Ġother Ġtasks Ġthat Ġcan Ġbe Ġmodeled Ġbased Ġon Ġthe Ġenc oder - dec oder Ġarchitecture , Ġi . e ., Ġimage Ġcaption ing , Ġreading Ġcomprehension , Ġand Ġmachine Ġtranslation .
Introduction Natural Language Generation (NLG) plays a critical role in Spoken Dialogue Systems (SDS) with task is to convert a meaning representation produced by the Dialogue Manager into natural language utterances. Conventional approaches still rely on comprehensive hand-tuning templates and rules requiring expert knowledge of linguistic representation, including rule-based BIBREF0 , corpus-based n-gram models BIBREF1 , and a trainable generator BIBREF2 . Recently, Recurrent Neural Networks (RNNs) based approaches have shown promising performance in tackling the NLG problems. The RNN-based models have been applied for NLG as a joint training model BIBREF3 , BIBREF4 and an end-to-end training model BIBREF5 . A recurring problem in such systems is requiring annotated datasets for particular dialogue acts (DAs). To ensure that the generated utterance representing the intended meaning of the given DA, the previous RNN-based models were further conditioned on a 1-hot vector representation of the DA. BIBREF3 introduced a heuristic gate to ensure that all the slot-value pair was accurately captured during generation. BIBREF4 subsequently proposed a Semantically Conditioned Long Short-term Memory generator (SC-LSTM) which jointly learned the DA gating signal and language model. More recently, Encoder-Decoder networks BIBREF6 , BIBREF7 , especially the attentional based models BIBREF8 , BIBREF9 have been explored to solve the NLG tasks. The Attentional RNN Encoder-Decoder BIBREF10 (ARED) based approaches have also shown improved performance on a variety of tasks, e.g., image captioning BIBREF11 , BIBREF12 , text summarization BIBREF13 , BIBREF14 . While the RNN-based generators with DA gating-vector can prevent the undesirable semantic repetitions, the ARED-based generators show signs of better adapting to a new domain. However, none of the models show significant advantage from out-of-domain data. To better analyze model generalization to an unseen, new domain as well as model leveraging the out-of-domain sources, we propose a new architecture which is an extension of the ARED model. In order to better select, aggregate and control the semantic information, a Refinement Adjustment LSTM-based component (RALSTM) is introduced to the decoder side. The proposed model can learn from unaligned data by jointly training the sentence planning and surface realization to produce natural language sentences. We conducted experiments on four different NLG domains and found that the proposed methods significantly outperformed the state-of-the-art methods regarding BLEU BIBREF15 and slot error rate ERR scores BIBREF4 . The results also showed that our generators could scale to new domains by leveraging the out-of-domain data. To sum up, we make three key contributions in this paper: We review related works in Section "Related Work" . Following a detail of proposed model in Section "Recurrent Neural Language Generator" , Section "Experiments" describes datasets, experimental setups, and evaluation metrics. The resulting analysis is presented in Section "Results and Analysis" . We conclude with a brief summary and future work in Section "Conclusion and Future Work" . Related Work Recently, RNNs-based models have shown promising performance in tackling the NLG problems. BIBREF16 proposed a generator using RNNs to create Chinese poetry. BIBREF11 , BIBREF17 , BIBREF18 also used RNNs in a multi-modal setting to solve image captioning tasks. The RNN-based Sequence to Sequence models have applied to solve variety of tasks: conversational modeling BIBREF6 , BIBREF7 , BIBREF19 , machine translation BIBREF20 , BIBREF21 For task-oriented dialogue systems, BIBREF3 combined a forward RNN generator, a CNN reranker, and a backward RNN reranker to generate utterances. BIBREF4 proposed SC-LSTM generator which introduced a control sigmoid gate to the LSTM cell to jointly learn the gating mechanism and language model. A recurring problem in such systems is the lack of sufficient domain-specific annotated data. BIBREF22 proposed an out-of-domain model which was trained on counterfeited data by using semantically similar slots from the target domain instead of the slots belonging to the out-of-domain dataset. The results showed that the model can achieve a satisfactory performance with a small amount of in-domain data by fine tuning the target domain on the out-of-domain trained model. More recently, RNN encoder-decoder based models with attention mechanism BIBREF10 have shown improved performances in various tasks. BIBREF12 proposed a review network to the image captioning, which reviews all the information encoded by the encoder and produces a compact thought vector. BIBREF9 proposed RNN encoder-decoder-based model by using two attention layers to jointly train content selection and surface realization. More close to our work, BIBREF8 proposed an attentive encoder-decoder based generator which computed the attention mechanism over the slot-value pairs. The model showed a domain scalability when a very limited amount of data is available. Moving from a limited domain dialogue system to an open domain dialogue system raises some issues. Therefore, it is important to build an open domain dialogue system that can make as much use of existing abilities of functioning from other domains. There have been several works to tackle this problem, such as BIBREF23 using RNN-based networks for multi-domain dialogue state tracking, BIBREF22 using a procedure to train multi-domain via multiple adaptation steps, or BIBREF24 , BIBREF25 adapting of SDS components to new domains. Recurrent Neural Language Generator The recurrent language generator proposed in this paper is based on a neural language generator BIBREF8 , which consists of three main components: (i) an Encoder that incorporates the target meaning representation (MR) as the model inputs, (ii) an Aligner that aligns and controls the semantic elements, and (iii) an RNN Decoder that generates output sentences. The generator architecture is shown in Figure 1 . The Encoder first encodes the MR into input semantic elements which are then aggregated and selected by utilizing an attention-based mechanism by the Aligner. The input to the RNN Decoder at each time step is a 1-hot encoding of a token $\textbf {w}_{t}$ and an attentive DA representation $\textbf {d}_{t}$ . At each time step $t$ , RNN Decoder also computes how much the feature value vector $\textbf {s}_{t-1}$ retained for the next computational steps, and adds this information to the RNN output which represents the probability distribution of the next token $\textbf {w}_{t+1}$ . At generation time, we can sample from this conditional distribution to obtain the next token in a generated sentence, and feed it as the next input to the RNN Decoder. This process finishes when an end sign is generated BIBREF17 , or some constraints are reached BIBREF16 . The model can produce a sequence of tokens which can finally be lexicalized to form the required utterance. Encoder The slots and values are separated parameters used in the encoder side. This embeds the source information into a vector representation $\textbf {z}_{i}$ which is a concatenation of embedding vector representation of each slot-value pair, and is computed by: $$\textbf {z}_{i} = \textbf {u}_{i} \oplus \textbf {v}_{i}$$   (Eq. 10) where $\textbf {u}_{i}$ , $\textbf {v}_{i}$ are the $i$ -th slot and value embedding vectors, respectively, and $\oplus $ is vector concatenation. The i index runs over the $L$ given slot-value pairs. In this work, we use a 1-layer, Bidirectional LSTM (Bi-LSTM) to encode the sequence of slot-value pairs embedding. The Bi-LSTM consists of forward and backward LSTMs which read the sequence of slot-value pairs from left-to-right and right-to-left to produce forward and backward sequence of hidden states ( $\overrightarrow{\textbf {e}_{1}}, .., \overrightarrow{\textbf {e}_{L}}$ ), and ( $\overleftarrow{\textbf {e}_{1}}, .., \overleftarrow{\textbf {e}_{L}}$ ), respectively. We then obtain the sequence of encoded hidden states $\textbf {E}=(\textbf {e}_{1}, \textbf {e}_{2}, .., \textbf {e}_{L})$ where $\textbf {\textbf {e}}_{i}$ is a sum of the forward hidden state $\overrightarrow{\textbf {e}_{i}}$ and the backward one $\textbf {v}_{i}$0 as follows: $$\textbf {e}_{i}=\overrightarrow{\textbf {e}_{i}} + \overleftarrow{\textbf {e}_{i}}$$   (Eq. 12) Aligner The Aligner utilizes attention mechanism to calculate the DA representation as follows: $$\beta _{t,i} = \frac{\exp e_{t,i} }{\sum \nolimits _{j}\exp e_{t,j}}$$   (Eq. 14) where $$e_{t,i}=a(\textbf {e}_{i}, \textbf {h}_{t-1})$$   (Eq. 15) and $\beta _{t,i}$ is the weight of i-th slot-value pair calculated by the attention mechanism. The alignment model $a$ is computed by: $$a(\textbf {e}_{i}, \textbf {h}_{t-1}) = \textbf {v}_{a}^{\top }\tanh (\textbf {W}_{a}\textbf {e}_{i} + \textbf {U}_{a}\textbf {h}_{t-1})$$   (Eq. 16) where $\textbf {v}_{a}, \textbf {W}_{a}, \textbf {U}_{a}$ are the weight matrices to learn. Finally, the Aligner calculates dialogue act embedding $\textbf {d}_{t}$ as follows: $$\textbf {d}_{t} = \textbf {a} \oplus \sum \nolimits _{i}\beta _{t,i} \textbf {e}_{i}$$   (Eq. 17) where a is vector embedding of the action type. RALSTM Decoder The proposed semantic RALSTM cell applied for Decoder side consists of three components: a Refinement cell, a traditional LSTM cell, and an Adjustment cell: Firstly, instead of feeding the original input token $\textbf {w}_{t}$ into the RNN cell, the input is recomputed by using a semantic gate as follows: $$\begin{aligned} \textbf {r}_{t}&=\sigma (\textbf {W}_{rd}\textbf {d}_{t} + \textbf {W}_{rh}\textbf {h}_{t-1})\\ \textbf {x}_{t}&=\textbf {r}_{t} \odot \textbf {w}_{t} \end{aligned}$$   (Eq. 19) where $\textbf {W}_{rd}$ and $\textbf {W}_{rh}$ are weight matrices. Element-wise multiplication $\odot $ plays a part in word-level matching which not only learns the vector similarity, but also preserves information about the two vectors. $\textbf {W}_{rh}$ acts like a key phrase detector that learns to capture the pattern of generation tokens or the relationship between multiple tokens. In other words, the new input $\textbf {x}_{t}$ consists of information of the original input token $\textbf {w}_{t}$ , the DA representation $\textbf {d}_{t}$ , and the hidden context $\textbf {h}_{t-1}$ . $\textbf {r}_{t}$ is called a Refinement gate because the input tokens are refined by a combination gating information of the attentive DA representation $\textbf {d}_{t}$ and the previous hidden state $\textbf {W}_{rh}$0 . By this way, we can represent the whole sentence based on the refined inputs. Secondly, the traditional LSTM network proposed by BIBREF26 bahdanau2014neural in which the input gate $\textbf {i}_{i}$ , forget gate $\textbf {f}_{t}$ and output gates $\textbf {o}_{t}$ are introduced to control information flow and computed as follows: $$\begin{aligned} \begin{pmatrix} \textbf {i}_{t} \\ \textbf {f}_{t} \\ \textbf {o}_{t} \\ \hat{\textbf {c}}_{t} \end{pmatrix} &= \begin{pmatrix}\sigma \\ \sigma \\ \sigma \\ \tanh \end{pmatrix}\textbf {W}_{4n,4n} \begin{pmatrix} \textbf {x}_{t} \\ \textbf {d}_{t} \\ \textbf {h}_{t-1} \end{pmatrix}\\ \end{aligned}$$   (Eq. 20) where $n$ is hidden layer size, $\textbf {W}_{4n,4n}$ is model parameters. The cell memory value $\textbf {c}_{t}$ is modified to depend on the DA representation as: $$\begin{aligned} \textbf {c}_{t}&=\textbf {f}_{t}\odot \textbf {c}_{t-1} +\textbf {i}_{t}\odot \hat{\textbf {c}}_{t} + \tanh (\textbf {W}_{cr}\textbf {r}_{t}) \\ \tilde{\textbf {h}}_{t}&= \textbf {o}_{t} \odot \tanh (\textbf {c}_{t}) \end{aligned}$$   (Eq. 21) where $\tilde{\textbf {h}}_{t}$ is the output. Thirdly, inspired by work of BIBREF4 in which the generator was further conditioned on a 1-hot representation vector $\textbf {s}$ of given dialogue act, and work of BIBREF27 that proposed a visual sentinel gate to make a decision on whether the model should attend to the image or to the sentinel gate, an additional gating cell is introduced on top of the traditional LSTM to gate another controlling vector $\textbf {s}$ . Figure 6 shows how RALSTM controls the DA vector $\textbf {s}$ . First, starting from the 1-hot vector of the DA $\textbf {s}_{0}$ , at each time step $t$ the proposed cell computes how much the LSTM output $\tilde{\textbf {h}}_{t}$ affects the DA vector, which is computed as follows: $$\begin{aligned} \textbf {a}_{t}&=\sigma (\textbf {W}_{ax}\textbf {x}_{t} +\textbf {W}_{ah}\tilde{\textbf {h}}_{t})\\ \textbf {s}_{t}&=\textbf {s}_{t-1} \odot \textbf {a}_{t} \end{aligned}$$   (Eq. 22) where $\textbf {W}_{ax}$ , $\textbf {W}_{ah}$ are weight matrices to be learned. $\textbf {a}_{t}$ is called an $Adjustment$ gate since its task is to control what information of the given DA have been generated and what information should be retained for future time steps. Second, we consider how much the information preserved in the DA $\textbf {s}_{t}$ can be contributed to the output, in which an additional output is computed by applying the output gate $\textbf {o}_{t}$ on the remaining information in $\textbf {s}_{t}$ as follows: $$\begin{aligned} \textbf {c}_{a}&=\sigma (\textbf {W}_{os}\textbf {s}_{t})\\ \tilde{\textbf {h}}_{a}&= \textbf {o}_{t} \odot \tanh (\textbf {c}_{a}) \end{aligned}$$   (Eq. 23) where $\textbf {W}_{os}$ is a weight matrix to project the DA presentation into the output space, $\tilde{\textbf {h}}_{a}$ is the Adjustment cell output. Final RALSTM output is a combination of both outputs of the traditional LSTM cell and the Adjustment cell, and computed as follows: $$\textbf {h}_{t}=\tilde{\textbf {h}}_{t} + \tilde{\textbf {h}}_{a}$$   (Eq. 24) Finally, the output distribution is computed by applying a softmax function $g$ , and the distribution can be sampled to obtain the next token, $$\begin{aligned} & P(w_{t+1}\mid w_{t},...w_{0},\textbf {DA})=g(\textbf {W}_{ho}\textbf {h}_{t}) \\ & w_{t+1} \sim P(w_{t+1}\mid w_{t}, w_{t-1},...w_{0},\textbf {DA}) \end{aligned}$$   (Eq. 25) where $\textbf {DA}=(\textbf {s}, \textbf {z})$ . Training The objective function was the negative log-likelihood and computed by: $$\textbf {F}(\theta ) = -\sum _{t=1}^{T}\textbf {y}_{t}^{\top }\log {\textbf {p}_{t}}$$   (Eq. 27) where: $\textbf {y}_{t}$ is the ground truth token distribution, $\textbf {p}_{t}$ is the predicted token distribution, $T$ is length of the input sentence. The proposed generators were trained by treating each sentence as a mini-batch with $l_{2}$ regularization added to the objective function for every 5 training examples. The models were initialized with a pretrained Glove word embedding vectors BIBREF28 and optimized by using stochastic gradient descent and back propagation through time BIBREF29 . Early stopping mechanism was implemented to prevent over-fitting by using a validation set as suggested in BIBREF30 . Decoding The decoding consists of two phases: (i) over-generation, and (ii) reranking. In the over-generation, the generator conditioned on both representations of the given DA use a beam search to generate a set of candidate responses. In the reranking phase, cost of the generator is computed to form the reranking score $\textbf {R}$ as follows: $$\textbf {R} = \textbf {F}(\theta ) + \lambda \textbf {ERR}$$   (Eq. 29) where $\lambda $ is a trade off constant and is set to a large value in order to severely penalize nonsensical outputs. The slot error rate $\textbf {ERR}$ , which is the number of slots generated that is either missing or redundant, and is computed by: $$\textbf {ERR} = \frac{\textbf {p} + \textbf {q}}{\textbf {N}}$$   (Eq. 30) where $\textbf {N}$ is the total number of slots in DA, and $\textbf {p}$ , $\textbf {q}$ is the number of missing and redundant slots, respectively. Experiments We extensively conducted a set of experiments to assess the effectiveness of the proposed models by using several metrics, datasets, and model architectures, in order to compare to prior methods. Datasets We assessed the proposed models on four different NLG domains: finding a restaurant, finding a hotel, buying a laptop, and buying a television. The Restaurant and Hotel were collected in BIBREF4 , while the Laptop and TV datasets have been released by BIBREF22 with a much larger input space but only one training example for each DA so that the system must learn partial realization of concepts and be able to recombine and apply them to unseen DAs. This makes the NLG tasks for the Laptop and TV domains become much harder. The dataset statistics are shown in Table 1 . Experimental Setups The generators were implemented using the TensorFlow library BIBREF31 and trained with training, validation and testing ratio as 3:1:1. The hidden layer size, beam size were set to be 80 and 10, respectively, and the generators were trained with a $70\%$ of dropout rate. We performed 5 runs with different random initialization of the network and the training is terminated by using early stopping. We then chose a model that yields the highest BLEU score on the validation set as shown in Table 2 . Since the trained models can differ depending on the initialization, we also report the results which were averaged over 5 randomly initialized networks. Note that, except the results reported in Table 2 , all the results shown were averaged over 5 randomly initialized networks. We set $\lambda $ to 1000 to severely discourage the reranker from selecting utterances which contain either redundant or missing slots. For each DA, we over-generated 20 candidate sentences and selected the top 5 realizations after reranking. Moreover, in order to better understand the effectiveness of our proposed methods, we: (i) performed an ablation experiments to demonstrate the contribution of each proposed cells (Tables 2 , 3 ), (ii) trained the models on the Laptop domain with varied proportion of training data, starting from $10\%$ to $100\%$ (Figure 3 ), (iii) trained general models by merging all the data from four domains together and tested them in each individual domain (Figure 4 ), and (iv) trained adaptation models on merging data from restaurant and hotel domains, then fine tuned the model on laptop domain with varied amount of adaptation data (Figure 5 ). Evaluation Metrics and Baselines The generator performance was assessed on the two evaluation metrics: the BLEU and the slot error rate ERR by adopting code from an open source benchmark toolkit for Natural Language Generation. We compared the proposed models against three strong baselines which have been recently published as state-of-the-art NLG benchmarks[]. https://github.com/shawnwun/RNNLG HLSTM proposed by BIBREF3 thwsjy15 which used a heuristic gate to ensure that all of the slot-value information was accurately captured when generating. SCLSTM proposed by BIBREF4 wensclstm15 which can jointly learn the gating signal and language model. Enc-Dec proposed by BIBREF8 wentoward which applied the attention-based encoder-decoder architecture. Results We conducted extensive experiments on our models and compared against the previous methods. Overall, the proposed models consistently achieve the better performance regarding both evaluation metrics across all domains in all test cases. The ablation studies (Tables 2 , 3 ) demonstrate the contribution of different model components in which the models were assessed without Adjustment cell (w/o A), or without Refinement cell (w/o R). It clearly sees that the Adjustment cell contributes to reducing the slot error rate ERR score since it can effectively prevent the undesirable slot-value pair repetitions by gating the DA vector $\textbf {s}$ . A comparison between the ARED-based models (denoted by $^{\sharp }$ in Table 2 ) shows that the proposed models not only have better performance with higher the BLEU score but also significantly reduce the slot error rate ERR score by a large margin about $2\%$ to $4\%$ in every datasets. Moreover, a comparison between the models with gating the DA vector also indicates that the proposed models (w/o R, RALSTM) have significant improved performance on both the evaluation metrics across the four domains compared to the SCLSTM model. The RALSTM cell without the Refinement cell is similar as the SCLSTM cell. However, it obtained the results much better than the SCLSTM baselines. This stipulates the necessary of the LSTM encoder and the Aligner in effectively partial learning the correlated order between slot-value representation in the DAs, especially for the unseen domain where there is only one training example for each DA. Table 3 further demonstrates the stable strength of our models since the results' pattern stays unchanged compared to those in Table 2 . Figure 3 shows a comparison of three models (Enc-Dec, SCLSTM, and RALSTM) which were trained from scratch on the unseen laptop domain in varied proportion of training data, from $1\%$ to $100\%$ . It clearly shows that the RALSTM outperforms the previous models in all cases, while the Enc-Dec has a much greater ERR score comparing to the two models. A comparison of top responses generated for some input DAs between different models are shown in Table 4 . While the previous models still produce some errors (missing and misplaced information), the proposed models (RALSTM and the models All2* trained by pooling all datasets together) can generate appropriate sentences. We also found that the proposed models tend to generate more complete and concise sentences than the other models. All these prove the importance of the proposed components: the Refinement cell in aggregating and selecting the attentive information, and the Adjustment cell in controlling the feature vector (see Examples in Figure 6 ). Figure 4 shows a comparison performance of general models as described in Section "Experimental Setups" . The results are consistent with the Figure 3 , in which the RALSTM has better performance than the Enc-Dec and SCLSTM on all domains in terms of the BLEU and the ERR scores, while the Enc-Dec has difficulties in reducing the ERR score. This indicates the relevant contribution of the proposed component Refinement and Adjustment cells to the original ARED architecture, in which the Refinement with attentional gating can effectively select and aggregate the information before putting them into the traditional LSTM cell, while the Adjustment with gating DA vector can effectively control the information flow during generation. Figure 5 shows domain scalability of the three models in which the models were first trained on the merging out-of-domain Restaurant and Hotel datasets, then fine tuned the parameters with varied amount of in-domain training data (laptop domain). The RALSTM model outperforms the previous model in both cases where the sufficient in-domain data is used (as in Figure 5 -left) and the limited in-domain data is used (Figure 5 -right). The Figure 5 -right also indicates that the RALSTM model can adapt to a new, unseen domain faster than the previous models. Conclusion and Future Work We present an extension of ARED model, in which an RALSTM component is introduced to select and aggregate semantic elements produced by the Encoder, and to generate the required sentence. We assessed the proposed models on four NLG domains and compared to the state-of-the-art generators. The proposed models empirically show consistent improvement over the previous methods in both the BLEU and ERR evaluation metrics. The proposed models also show an ability to extend to a new, unseen domain no matter how much the in-domain training data was fed. In the future, it would be interesting to apply the proposed model to other tasks that can be modeled based on the encoder-decoder architecture, i.e., image captioning, reading comprehension, and machine translation.
What is the difference of the proposed model with a standard RNN encoder-decoder?
981fd79dd69581659cb1d4e2b29178e82681eb4d
What Ġis Ġthe Ġdifference Ġof Ġthe Ġproposed Ġmodel Ġwith Ġa Ġstandard ĠR NN Ġenc oder - dec oder ?
Introdu ce Ġa Ġ" Ref inement ĠAdjust ment ĠL ST M - based Ġcomponent " Ġto Ġthe Ġdec oder
<s> What Ġis Ġthe Ġdifference Ġof Ġthe Ġproposed Ġmodel Ġwith Ġa Ġstandard ĠR NN Ġenc oder - dec oder ? </s> Introduction </s> Natural ĠLanguage ĠGeneration Ġ( NL G ) Ġplays Ġa Ġcritical Ġrole Ġin ĠSp oken ĠDialogue ĠSystems Ġ( S DS ) Ġwith Ġtask Ġis Ġto Ġconvert Ġa Ġmeaning Ġrepresentation Ġproduced Ġby Ġthe ĠDialogue ĠManager Ġinto Ġnatural Ġlanguage Ġutter ances . ĠCon ventional Ġapproaches Ġstill Ġrely Ġon Ġcomprehensive Ġhand - tun ing Ġtemplates Ġand Ġrules Ġrequiring Ġexpert Ġknowledge Ġof Ġlinguistic Ġrepresentation , Ġincluding Ġrule - based ĠB IB REF 0 Ġ, Ġcorpus - based Ġn - gram Ġmodels ĠB IB REF 1 Ġ, Ġand Ġa Ġtrain able Ġgenerator ĠB IB REF 2 Ġ. </s> Recently , ĠRec urrent ĠNeural ĠNetworks Ġ( R NN s ) Ġbased Ġapproaches Ġhave Ġshown Ġpromising Ġperformance Ġin Ġtackling Ġthe ĠNL G Ġproblems . ĠThe ĠR NN - based Ġmodels Ġhave Ġbeen Ġapplied Ġfor ĠNL G Ġas Ġa Ġjoint Ġtraining Ġmodel ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġand Ġan Ġend - to - end Ġtraining Ġmodel ĠB IB REF 5 Ġ. ĠA Ġrecurring Ġproblem Ġin Ġsuch Ġsystems Ġis Ġrequiring Ġannot ated Ġdatasets Ġfor Ġparticular Ġdialogue Ġacts Ġ( D As ). ĠTo Ġensure Ġthat Ġthe Ġgenerated Ġutter ance Ġrepresenting Ġthe Ġintended Ġmeaning Ġof Ġthe Ġgiven ĠDA , Ġthe Ġprevious ĠR NN - based Ġmodels Ġwere Ġfurther Ġconditioned Ġon Ġa Ġ1 - hot Ġvector Ġrepresentation Ġof Ġthe ĠDA . ĠB IB REF 3 Ġintroduced Ġa Ġhe uristic Ġgate Ġto Ġensure Ġthat Ġall Ġthe Ġslot - value Ġpair Ġwas Ġaccurately Ġcaptured Ġduring Ġgeneration . ĠB IB REF 4 Ġsubsequently Ġproposed Ġa ĠSem antically ĠCondition ed ĠLong ĠShort - term ĠMemory Ġgenerator Ġ( SC - L ST M ) Ġwhich Ġjointly Ġlearned Ġthe ĠDA Ġg ating Ġsignal Ġand Ġlanguage Ġmodel . </s> More Ġrecently , ĠEnc oder - Dec oder Ġnetworks ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, Ġespecially Ġthe Ġattention al Ġbased Ġmodels ĠB IB REF 8 Ġ, ĠB IB REF 9 Ġhave Ġbeen Ġexplored Ġto Ġsolve Ġthe ĠNL G Ġtasks . ĠThe ĠAttention al ĠR NN ĠEnc oder - Dec oder ĠB IB REF 10 Ġ( AR ED ) Ġbased Ġapproaches Ġhave Ġalso Ġshown Ġimproved Ġperformance Ġon Ġa Ġvariety Ġof Ġtasks , Ġe . g ., Ġimage Ġcaption ing ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ, Ġtext Ġsummar ization ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ. </s> While Ġthe ĠR NN - based Ġgenerators Ġwith ĠDA Ġg ating - vector Ġcan Ġprevent Ġthe Ġundesirable Ġsemantic Ġrepet itions , Ġthe ĠAR ED - based Ġgenerators Ġshow Ġsigns Ġof Ġbetter Ġadapting Ġto Ġa Ġnew Ġdomain . ĠHowever , Ġnone Ġof Ġthe Ġmodels Ġshow Ġsignificant Ġadvantage Ġfrom Ġout - of - domain Ġdata . ĠTo Ġbetter Ġanalyze Ġmodel Ġgeneral ization Ġto Ġan Ġunseen , Ġnew Ġdomain Ġas Ġwell Ġas Ġmodel Ġleveraging Ġthe Ġout - of - domain Ġsources , Ġwe Ġpropose Ġa Ġnew Ġarchitecture Ġwhich Ġis Ġan Ġextension Ġof Ġthe ĠAR ED Ġmodel . ĠIn Ġorder Ġto Ġbetter Ġselect , Ġaggregate Ġand Ġcontrol Ġthe Ġsemantic Ġinformation , Ġa ĠRef inement ĠAdjust ment ĠL ST M - based Ġcomponent Ġ( RAL ST M ) Ġis Ġintroduced Ġto Ġthe Ġdec oder Ġside . ĠThe Ġproposed Ġmodel Ġcan Ġlearn Ġfrom Ġun aligned Ġdata Ġby Ġjointly Ġtraining Ġthe Ġsentence Ġplanning Ġand Ġsurface Ġrealization Ġto Ġproduce Ġnatural Ġlanguage Ġsentences . ĠWe Ġconducted Ġexperiments Ġon Ġfour Ġdifferent ĠNL G Ġdomains Ġand Ġfound Ġthat Ġthe Ġproposed Ġmethods Ġsignificantly Ġoutper formed Ġthe Ġstate - of - the - art Ġmethods Ġregarding ĠB LE U ĠB IB REF 15 Ġand Ġslot Ġerror Ġrate ĠER R Ġscores ĠB IB REF 4 Ġ. ĠThe Ġresults Ġalso Ġshowed Ġthat Ġour Ġgenerators Ġcould Ġscale Ġto Ġnew Ġdomains Ġby Ġleveraging Ġthe Ġout - of - domain Ġdata . ĠTo Ġsum Ġup , Ġwe Ġmake Ġthree Ġkey Ġcontributions Ġin Ġthis Ġpaper : </s> We Ġreview Ġrelated Ġworks Ġin ĠSection Ġ" Related ĠWork " Ġ. ĠFollowing Ġa Ġdetail Ġof Ġproposed Ġmodel Ġin ĠSection Ġ" Rec urrent ĠNeural ĠLanguage ĠGenerator " Ġ, ĠSection Ġ" Exper iments " Ġdescribes Ġdatasets , Ġexperimental Ġsetups , Ġand Ġevaluation Ġmetrics . ĠThe Ġresulting Ġanalysis Ġis Ġpresented Ġin ĠSection Ġ" Results Ġand ĠAnalysis " Ġ. ĠWe Ġconclude Ġwith Ġa Ġbrief Ġsummary Ġand Ġfuture Ġwork Ġin ĠSection Ġ" Conclusion Ġand ĠFuture ĠWork " Ġ. </s> Related ĠWork </s> Recently , ĠR NN s - based Ġmodels Ġhave Ġshown Ġpromising Ġperformance Ġin Ġtackling Ġthe ĠNL G Ġproblems . ĠB IB REF 16 Ġproposed Ġa Ġgenerator Ġusing ĠR NN s Ġto Ġcreate ĠChinese Ġpoetry . ĠB IB REF 11 Ġ, ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġalso Ġused ĠR NN s Ġin Ġa Ġmulti - mod al Ġsetting Ġto Ġsolve Ġimage Ġcaption ing Ġtasks . ĠThe ĠR NN - based ĠSequence Ġto ĠSequence Ġmodels Ġhave Ġapplied Ġto Ġsolve Ġvariety Ġof Ġtasks : Ġconvers ational Ġmodeling ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 19 Ġ, Ġmachine Ġtranslation ĠB IB REF 20 Ġ, ĠB IB REF 21 </s> For Ġtask - oriented Ġdialogue Ġsystems , ĠB IB REF 3 Ġcombined Ġa Ġforward ĠR NN Ġgenerator , Ġa ĠCNN Ġre rank er , Ġand Ġa Ġbackward ĠR NN Ġre rank er Ġto Ġgenerate Ġutter ances . ĠB IB REF 4 Ġproposed ĠSC - L ST M Ġgenerator Ġwhich Ġintroduced Ġa Ġcontrol Ġs igm oid Ġgate Ġto Ġthe ĠL ST M Ġcell Ġto Ġjointly Ġlearn Ġthe Ġg ating Ġmechanism Ġand Ġlanguage Ġmodel . ĠA Ġrecurring Ġproblem Ġin Ġsuch Ġsystems Ġis Ġthe Ġlack Ġof Ġsufficient Ġdomain - specific Ġannot ated Ġdata . ĠB IB REF 22 Ġproposed Ġan Ġout - of - domain Ġmodel Ġwhich Ġwas Ġtrained Ġon Ġcounterfe ited Ġdata Ġby Ġusing Ġsem antically Ġsimilar Ġslots Ġfrom Ġthe Ġtarget Ġdomain Ġinstead Ġof Ġthe Ġslots Ġbelonging Ġto Ġthe Ġout - of - domain Ġdataset . ĠThe Ġresults Ġshowed Ġthat Ġthe Ġmodel Ġcan Ġachieve Ġa Ġsatisfactory Ġperformance Ġwith Ġa Ġsmall Ġamount Ġof Ġin - domain Ġdata Ġby Ġfine Ġtuning Ġthe Ġtarget Ġdomain Ġon Ġthe Ġout - of - domain Ġtrained Ġmodel . </s> More Ġrecently , ĠR NN Ġenc oder - dec oder Ġbased Ġmodels Ġwith Ġattention Ġmechanism ĠB IB REF 10 Ġhave Ġshown Ġimproved Ġperformances Ġin Ġvarious Ġtasks . ĠB IB REF 12 Ġproposed Ġa Ġreview Ġnetwork Ġto Ġthe Ġimage Ġcaption ing , Ġwhich Ġreviews Ġall Ġthe Ġinformation Ġencoded Ġby Ġthe Ġenc oder Ġand Ġproduces Ġa Ġcompact Ġthought Ġvector . ĠB IB REF 9 Ġproposed ĠR NN Ġenc oder - dec oder - based Ġmodel Ġby Ġusing Ġtwo Ġattention Ġlayers Ġto Ġjointly Ġtrain Ġcontent Ġselection Ġand Ġsurface Ġrealization . ĠMore Ġclose Ġto Ġour Ġwork , ĠB IB REF 8 Ġproposed Ġan Ġattentive Ġenc oder - dec oder Ġbased Ġgenerator Ġwhich Ġcomputed Ġthe Ġattention Ġmechanism Ġover Ġthe Ġslot - value Ġpairs . ĠThe Ġmodel Ġshowed Ġa Ġdomain Ġscal ability Ġwhen Ġa Ġvery Ġlimited Ġamount Ġof Ġdata Ġis Ġavailable . </s> Moving Ġfrom Ġa Ġlimited Ġdomain Ġdialogue Ġsystem Ġto Ġan Ġopen Ġdomain Ġdialogue Ġsystem Ġraises Ġsome Ġissues . ĠTherefore , Ġit Ġis Ġimportant Ġto Ġbuild Ġan Ġopen Ġdomain Ġdialogue Ġsystem Ġthat Ġcan Ġmake Ġas Ġmuch Ġuse Ġof Ġexisting Ġabilities Ġof Ġfunctioning Ġfrom Ġother Ġdomains . ĠThere Ġhave Ġbeen Ġseveral Ġworks Ġto Ġtackle Ġthis Ġproblem , Ġsuch Ġas ĠB IB REF 23 Ġusing ĠR NN - based Ġnetworks Ġfor Ġmulti - domain Ġdialogue Ġstate Ġtracking , ĠB IB REF 22 Ġusing Ġa Ġprocedure Ġto Ġtrain Ġmulti - domain Ġvia Ġmultiple Ġadaptation Ġsteps , Ġor ĠB IB REF 24 Ġ, ĠB IB REF 25 Ġadapting Ġof ĠS DS Ġcomponents Ġto Ġnew Ġdomains . </s> Rec urrent ĠNeural ĠLanguage ĠGenerator </s> The Ġrecurrent Ġlanguage Ġgenerator Ġproposed Ġin Ġthis Ġpaper Ġis Ġbased Ġon Ġa Ġneural Ġlanguage Ġgenerator ĠB IB REF 8 Ġ, Ġwhich Ġconsists Ġof Ġthree Ġmain Ġcomponents : Ġ( i ) Ġan ĠEnc oder Ġthat Ġincorporates Ġthe Ġtarget Ġmeaning Ġrepresentation Ġ( MR ) Ġas Ġthe Ġmodel Ġinputs , Ġ( ii ) Ġan ĠAl ign er Ġthat Ġalign s Ġand Ġcontrols Ġthe Ġsemantic Ġelements , Ġand Ġ( iii ) Ġan ĠR NN ĠDecoder Ġthat Ġgenerates Ġoutput Ġsentences . ĠThe Ġgenerator Ġarchitecture Ġis Ġshown Ġin ĠFigure Ġ1 Ġ. ĠThe ĠEnc oder Ġfirst Ġenc odes Ġthe ĠMR Ġinto Ġinput Ġsemantic Ġelements Ġwhich Ġare Ġthen Ġaggreg ated Ġand Ġselected Ġby Ġutilizing Ġan Ġattention - based Ġmechanism Ġby Ġthe ĠAl ign er . ĠThe Ġinput Ġto Ġthe ĠR NN ĠDecoder Ġat Ġeach Ġtime Ġstep Ġis Ġa Ġ1 - hot Ġencoding Ġof Ġa Ġtoken Ġ$\ text bf Ġ{ w } _{ t } $ Ġand Ġan Ġattentive ĠDA Ġrepresentation Ġ$\ text bf Ġ{ d } _{ t } $ Ġ. ĠAt Ġeach Ġtime Ġstep Ġ$ t $ Ġ, ĠR NN ĠDecoder Ġalso Ġcomp utes Ġhow Ġmuch Ġthe Ġfeature Ġvalue Ġvector Ġ$\ text bf Ġ{ s } _{ t - 1 } $ Ġretained Ġfor Ġthe Ġnext Ġcomputational Ġsteps , Ġand Ġadds Ġthis Ġinformation Ġto Ġthe ĠR NN Ġoutput Ġwhich Ġrepresents Ġthe Ġprobability Ġdistribution Ġof Ġthe Ġnext Ġtoken Ġ$\ text bf Ġ{ w } _{ t + 1 } $ Ġ. ĠAt Ġgeneration Ġtime , Ġwe Ġcan Ġsample Ġfrom Ġthis Ġconditional Ġdistribution Ġto Ġobtain Ġthe Ġnext Ġtoken Ġin Ġa Ġgenerated Ġsentence , Ġand Ġfeed Ġit Ġas Ġthe Ġnext Ġinput Ġto Ġthe ĠR NN ĠDecoder . ĠThis Ġprocess Ġfinishes Ġwhen Ġan Ġend Ġsign Ġis Ġgenerated ĠB IB REF 17 Ġ, Ġor Ġsome Ġconstraints Ġare Ġreached ĠB IB REF 16 Ġ. ĠThe Ġmodel Ġcan Ġproduce Ġa Ġsequence Ġof Ġtokens Ġwhich Ġcan Ġfinally Ġbe Ġlex ical ized Ġto Ġform Ġthe Ġrequired Ġutter ance . </s> Enc oder </s> The Ġslots Ġand Ġvalues Ġare Ġseparated Ġparameters Ġused Ġin Ġthe Ġenc oder Ġside . ĠThis Ġembed s Ġthe Ġsource Ġinformation Ġinto Ġa Ġvector Ġrepresentation Ġ$\ text bf Ġ{ z } _{ i } $ Ġwhich Ġis Ġa Ġconc aten ation Ġof Ġembed ding Ġvector Ġrepresentation Ġof Ġeach Ġslot - value Ġpair , Ġand Ġis Ġcomputed Ġby : </s> $$ \ text bf Ġ{ z } _{ i } Ġ= Ġ\ text bf Ġ{ u } _{ i } Ġ\ opl us Ġ\ text bf Ġ{ v } _{ i } $$ Ġ Ġ Ġ( E q . Ġ10 ) </s> where Ġ$\ text bf Ġ{ u } _{ i } $ Ġ, Ġ$\ text bf Ġ{ v } _{ i } $ Ġare Ġthe Ġ$ i $ Ġ- th Ġslot Ġand Ġvalue Ġembed ding Ġvectors , Ġrespectively , Ġand Ġ$\ opl us Ġ$ Ġis Ġvector Ġconc aten ation . ĠThe Ġi Ġindex Ġruns Ġover Ġthe Ġ$ L $ Ġgiven Ġslot - value Ġpairs . ĠIn Ġthis Ġwork , Ġwe Ġuse Ġa Ġ1 - layer , ĠBid irection al ĠL ST M Ġ( Bi - L ST M ) Ġto Ġencode Ġthe Ġsequence Ġof Ġslot - value Ġpairs Ġembed ding . ĠThe ĠBi - L ST M Ġconsists Ġof Ġforward Ġand Ġbackward ĠL ST Ms Ġwhich Ġread Ġthe Ġsequence Ġof Ġslot - value Ġpairs Ġfrom Ġleft - to - right Ġand Ġright - to - left Ġto Ġproduce Ġforward Ġand Ġbackward Ġsequence Ġof Ġhidden Ġstates Ġ( Ġ$\ over right arrow {\ text bf Ġ{ e } _{ 1 } }, Ġ.. , Ġ\ over right arrow {\ text bf Ġ{ e } _{ L }} $ Ġ), Ġand Ġ( Ġ$\ over left arrow {\ text bf Ġ{ e } _{ 1 } }, Ġ.. , Ġ\ over left arrow {\ text bf Ġ{ e } _{ L }} $ Ġ), Ġrespectively . ĠWe Ġthen Ġobtain Ġthe Ġsequence Ġof Ġencoded Ġhidden Ġstates Ġ$\ text bf Ġ{ E } =( \ text bf Ġ{ e } _{ 1 }, Ġ\ text bf Ġ{ e } _{ 2 }, Ġ.. , Ġ\ text bf Ġ{ e } _{ L }) $ Ġwhere Ġ$\ text bf Ġ{\ text bf Ġ{ e }} _{ i } $ Ġis Ġa Ġsum Ġof Ġthe Ġforward Ġhidden Ġstate Ġ$\ over right arrow {\ text bf Ġ{ e } _{ i }} $ Ġand Ġthe Ġbackward Ġone Ġ$\ text bf Ġ{ v } _{ i } $ 0 Ġas Ġfollows : </s> $$ \ text bf Ġ{ e } _{ i } = \ over right arrow {\ text bf Ġ{ e } _{ i }} Ġ+ Ġ\ over left arrow {\ text bf Ġ{ e } _{ i }} $$ Ġ Ġ Ġ( E q . Ġ12 ) </s> Al ign er </s> The ĠAl ign er Ġutilizes Ġattention Ġmechanism Ġto Ġcalculate Ġthe ĠDA Ġrepresentation Ġas Ġfollows : </s> $$ \ beta Ġ_ { t , i } Ġ= Ġ\ frac {\ exp Ġe _{ t , i } Ġ} {\ sum Ġ\ n ol im its Ġ_ { j }\ exp Ġe _{ t , j }} $$ Ġ Ġ Ġ( E q . Ġ14 ) </s> where </s> $$ e _{ t , i } = a (\ text bf Ġ{ e } _{ i }, Ġ\ text bf Ġ{ h } _{ t - 1 }) $$ Ġ Ġ Ġ( E q . Ġ15 ) </s> and Ġ$\ beta Ġ_ { t , i } $ Ġis Ġthe Ġweight Ġof Ġi - th Ġslot - value Ġpair Ġcalculated Ġby Ġthe Ġattention Ġmechanism . ĠThe Ġalignment Ġmodel Ġ$ a $ Ġis Ġcomputed Ġby : </s> $$ a (\ text bf Ġ{ e } _{ i }, Ġ\ text bf Ġ{ h } _{ t - 1 }) Ġ= Ġ\ text bf Ġ{ v } _{ a } ^ {\ top Ġ} \ tan h Ġ( \ text bf Ġ{ W } _{ a }\ text bf Ġ{ e } _{ i } Ġ+ Ġ\ text bf Ġ{ U } _{ a }\ text bf Ġ{ h } _{ t - 1 }) $$ Ġ Ġ Ġ( E q . Ġ16 ) </s> where Ġ$\ text bf Ġ{ v } _{ a }, Ġ\ text bf Ġ{ W } _{ a }, Ġ\ text bf Ġ{ U } _{ a } $ Ġare Ġthe Ġweight Ġmat rices Ġto Ġlearn . ĠFinally , Ġthe ĠAl ign er Ġcalculates Ġdialogue Ġact Ġembed ding Ġ$\ text bf Ġ{ d } _{ t } $ Ġas Ġfollows : </s> $$ \ text bf Ġ{ d } _{ t } Ġ= Ġ\ text bf Ġ{ a } Ġ\ opl us Ġ\ sum Ġ\ n ol im its Ġ_ { i }\ beta Ġ_ { t , i } Ġ\ text bf Ġ{ e } _{ i } $$ Ġ Ġ Ġ( E q . Ġ17 ) </s> where Ġa Ġis Ġvector Ġembed ding Ġof Ġthe Ġaction Ġtype . </s> RAL ST M ĠDecoder </s> The Ġproposed Ġsemantic ĠR AL ST M Ġcell Ġapplied Ġfor ĠDecoder Ġside Ġconsists Ġof Ġthree Ġcomponents : Ġa ĠRef inement Ġcell , Ġa Ġtraditional ĠL ST M Ġcell , Ġand Ġan ĠAdjust ment Ġcell : </s> Firstly , Ġinstead Ġof Ġfeeding Ġthe Ġoriginal Ġinput Ġtoken Ġ$\ text bf Ġ{ w } _{ t } $ Ġinto Ġthe ĠR NN Ġcell , Ġthe Ġinput Ġis Ġrec om puted Ġby Ġusing Ġa Ġsemantic Ġgate Ġas Ġfollows : </s> $$ \ begin { aligned } Ġ\ text bf Ġ{ r } _{ t } & = \ s igma Ġ( \ text bf Ġ{ W } _{ rd }\ text bf Ġ{ d } _{ t } Ġ+ Ġ\ text bf Ġ{ W } _{ rh }\ text bf Ġ{ h } _{ t - 1 }) \\ Ġ\ text bf Ġ{ x } _{ t } & = \ text bf Ġ{ r } _{ t } Ġ\ od ot Ġ\ text bf Ġ{ w } _{ t } Ġ\ end { aligned } $$ Ġ Ġ Ġ( E q . Ġ19 ) </s> where Ġ$\ text bf Ġ{ W } _{ rd } $ Ġand Ġ$\ text bf Ġ{ W } _{ rh } $ Ġare Ġweight Ġmat rices . ĠElement - wise Ġmultiplication Ġ$\ od ot Ġ$ Ġplays Ġa Ġpart Ġin Ġword - level Ġmatching Ġwhich Ġnot Ġonly Ġlearns Ġthe Ġvector Ġsimilarity , Ġbut Ġalso Ġpreserves Ġinformation Ġabout Ġthe Ġtwo Ġvectors . Ġ$\ text bf Ġ{ W } _{ rh } $ Ġacts Ġlike Ġa Ġkey Ġphrase Ġdetector Ġthat Ġlearns Ġto Ġcapture Ġthe Ġpattern Ġof Ġgeneration Ġtokens Ġor Ġthe Ġrelationship Ġbetween Ġmultiple Ġtokens . ĠIn Ġother Ġwords , Ġthe Ġnew Ġinput Ġ$\ text bf Ġ{ x } _{ t } $ Ġconsists Ġof Ġinformation Ġof Ġthe Ġoriginal Ġinput Ġtoken Ġ$\ text bf Ġ{ w } _{ t } $ Ġ, Ġthe ĠDA Ġrepresentation Ġ$\ text bf Ġ{ d } _{ t } $ Ġ, Ġand Ġthe Ġhidden Ġcontext Ġ$\ text bf Ġ{ h } _{ t - 1 } $ Ġ. Ġ$\ text bf Ġ{ r } _{ t } $ Ġis Ġcalled Ġa ĠRef inement Ġgate Ġbecause Ġthe Ġinput Ġtokens Ġare Ġrefined Ġby Ġa Ġcombination Ġg ating Ġinformation Ġof Ġthe Ġattentive ĠDA Ġrepresentation Ġ$\ text bf Ġ{ d } _{ t } $ Ġand Ġthe Ġprevious Ġhidden Ġstate Ġ$\ text bf Ġ{ W } _{ rh } $ 0 Ġ. ĠBy Ġthis Ġway , Ġwe Ġcan Ġrepresent Ġthe Ġwhole Ġsentence Ġbased Ġon Ġthe Ġrefined Ġinputs . </s> Secondly , Ġthe Ġtraditional ĠL ST M Ġnetwork Ġproposed Ġby ĠB IB REF 26 Ġb ah dan au 2014 ne ural Ġin Ġwhich Ġthe Ġinput Ġgate Ġ$\ text bf Ġ{ i } _{ i } $ Ġ, Ġforget Ġgate Ġ$\ text bf Ġ{ f } _{ t } $ Ġand Ġoutput Ġgates Ġ$\ text bf Ġ{ o } _{ t } $ Ġare Ġintroduced Ġto Ġcontrol Ġinformation Ġflow Ġand Ġcomputed Ġas Ġfollows : </s> $$ \ begin { aligned } Ġ\ begin { pm at rix } Ġ\ text bf Ġ{ i } _{ t } Ġ\\ Ġ\ text bf Ġ{ f } _{ t } Ġ\\ Ġ\ text bf Ġ{ o } _{ t } Ġ\\ Ġ\ hat {\ text bf Ġ{ c }} _{ t } Ġ\ end { pm at rix } Ġ& = Ġ\ begin { pm at rix }\ s igma Ġ\\ Ġ\ s igma Ġ\\ Ġ\ s igma Ġ\\ Ġ\ tan h Ġ\ end { pm at rix }\ text bf Ġ{ W } _{ 4 n , 4 n } Ġ\ begin { pm at rix } Ġ\ text bf Ġ{ x } _{ t } Ġ\\ Ġ\ text bf Ġ{ d } _{ t } Ġ\\ Ġ\ text bf Ġ{ h } _{ t - 1 } Ġ\ end { pm at rix } \\ Ġ\ end { aligned } $$ Ġ Ġ Ġ( E q . Ġ20 ) </s> where Ġ$ n $ Ġis Ġhidden Ġlayer Ġsize , Ġ$\ text bf Ġ{ W } _{ 4 n , 4 n } $ Ġis Ġmodel Ġparameters . ĠThe Ġcell Ġmemory Ġvalue Ġ$\ text bf Ġ{ c } _{ t } $ Ġis Ġmodified Ġto Ġdepend Ġon Ġthe ĠDA Ġrepresentation Ġas : </s> $$ \ begin { aligned } Ġ\ text bf Ġ{ c } _{ t } & = \ text bf Ġ{ f } _{ t }\ od ot Ġ\ text bf Ġ{ c } _{ t - 1 } Ġ+ \ text bf Ġ{ i } _{ t }\ od ot Ġ\ hat {\ text bf Ġ{ c }} _{ t } Ġ+ Ġ\ tan h Ġ( \ text bf Ġ{ W } _{ cr }\ text bf Ġ{ r } _{ t }) Ġ\\ Ġ\ t ilde {\ text bf Ġ{ h }} _{ t } & = Ġ\ text bf Ġ{ o } _{ t } Ġ\ od ot Ġ\ tan h Ġ( \ text bf Ġ{ c } _{ t }) Ġ\ end { aligned } $$ Ġ Ġ Ġ( E q . Ġ21 ) </s> where Ġ$\ t ilde {\ text bf Ġ{ h }} _{ t } $ Ġis Ġthe Ġoutput . </s> Third ly , Ġinspired Ġby Ġwork Ġof ĠB IB REF 4 Ġin Ġwhich Ġthe Ġgenerator Ġwas Ġfurther Ġconditioned Ġon Ġa Ġ1 - hot Ġrepresentation Ġvector Ġ$\ text bf Ġ{ s } $ Ġof Ġgiven Ġdialogue Ġact , Ġand Ġwork Ġof ĠB IB REF 27 Ġthat Ġproposed Ġa Ġvisual Ġsent inel Ġgate Ġto Ġmake Ġa Ġdecision Ġon Ġwhether Ġthe Ġmodel Ġshould Ġattend Ġto Ġthe Ġimage Ġor Ġto Ġthe Ġsent inel Ġgate , Ġan Ġadditional Ġg ating Ġcell Ġis Ġintroduced Ġon Ġtop Ġof Ġthe Ġtraditional ĠL ST M Ġto Ġgate Ġanother Ġcontrolling Ġvector Ġ$\ text bf Ġ{ s } $ Ġ. ĠFigure Ġ6 Ġshows Ġhow ĠR AL ST M Ġcontrols Ġthe ĠDA Ġvector Ġ$\ text bf Ġ{ s } $ Ġ. ĠFirst , Ġstarting Ġfrom Ġthe Ġ1 - hot Ġvector Ġof Ġthe ĠDA Ġ$\ text bf Ġ{ s } _{ 0 } $ Ġ, Ġat Ġeach Ġtime Ġstep Ġ$ t $ Ġthe Ġproposed Ġcell Ġcomp utes Ġhow Ġmuch Ġthe ĠL ST M Ġoutput Ġ$\ t ilde {\ text bf Ġ{ h }} _{ t } $ Ġaffects Ġthe ĠDA Ġvector , Ġwhich Ġis Ġcomputed Ġas Ġfollows : </s> $$ \ begin { aligned } Ġ\ text bf Ġ{ a } _{ t } & = \ s igma Ġ( \ text bf Ġ{ W } _{ ax }\ text bf Ġ{ x } _{ t } Ġ+ \ text bf Ġ{ W } _{ ah }\ t ilde {\ text bf Ġ{ h }} _{ t }) \\ Ġ\ text bf Ġ{ s } _{ t } & = \ text bf Ġ{ s } _{ t - 1 } Ġ\ od ot Ġ\ text bf Ġ{ a } _{ t } Ġ\ end { aligned } $$ Ġ Ġ Ġ( E q . Ġ22 ) </s> where Ġ$\ text bf Ġ{ W } _{ ax } $ Ġ, Ġ$\ text bf Ġ{ W } _{ ah } $ Ġare Ġweight Ġmat rices Ġto Ġbe Ġlearned . Ġ$\ text bf Ġ{ a } _{ t } $ Ġis Ġcalled Ġan Ġ$ Adjust ment $ Ġgate Ġsince Ġits Ġtask Ġis Ġto Ġcontrol Ġwhat Ġinformation Ġof Ġthe Ġgiven ĠDA Ġhave Ġbeen Ġgenerated Ġand Ġwhat Ġinformation Ġshould Ġbe Ġretained Ġfor Ġfuture Ġtime Ġsteps . ĠSecond , Ġwe Ġconsider Ġhow Ġmuch Ġthe Ġinformation Ġpreserved Ġin Ġthe ĠDA Ġ$\ text bf Ġ{ s } _{ t } $ Ġcan Ġbe Ġcontributed Ġto Ġthe Ġoutput , Ġin Ġwhich Ġan Ġadditional Ġoutput Ġis Ġcomputed Ġby Ġapplying Ġthe Ġoutput Ġgate Ġ$\ text bf Ġ{ o } _{ t } $ Ġon Ġthe Ġremaining Ġinformation Ġin Ġ$\ text bf Ġ{ s } _{ t } $ Ġas Ġfollows : </s> $$ \ begin { aligned } Ġ\ text bf Ġ{ c } _{ a } & = \ s igma Ġ( \ text bf Ġ{ W } _{ os }\ text bf Ġ{ s } _{ t }) \\ Ġ\ t ilde {\ text bf Ġ{ h }} _{ a } & = Ġ\ text bf Ġ{ o } _{ t } Ġ\ od ot Ġ\ tan h Ġ( \ text bf Ġ{ c } _{ a }) Ġ\ end { aligned } $$ Ġ Ġ Ġ( E q . Ġ23 ) </s> where Ġ$\ text bf Ġ{ W } _{ os } $ Ġis Ġa Ġweight Ġmatrix Ġto Ġproject Ġthe ĠDA Ġpresentation Ġinto Ġthe Ġoutput Ġspace , Ġ$\ t ilde {\ text bf Ġ{ h }} _{ a } $ Ġis Ġthe ĠAdjust ment Ġcell Ġoutput . ĠFinal ĠR AL ST M Ġoutput Ġis Ġa Ġcombination Ġof Ġboth Ġoutputs Ġof Ġthe Ġtraditional ĠL ST M Ġcell Ġand Ġthe ĠAdjust ment Ġcell , Ġand Ġcomputed Ġas Ġfollows : </s> $$ \ text bf Ġ{ h } _{ t } = \ t ilde {\ text bf Ġ{ h }} _{ t } Ġ+ Ġ\ t ilde {\ text bf Ġ{ h }} _{ a } $$ Ġ Ġ Ġ( E q . Ġ24 ) </s> Finally , Ġthe Ġoutput Ġdistribution Ġis Ġcomputed Ġby Ġapplying Ġa Ġsoft max Ġfunction Ġ$ g $ Ġ, Ġand Ġthe Ġdistribution Ġcan Ġbe Ġsampled Ġto Ġobtain Ġthe Ġnext Ġtoken , </s> $$ \ begin { aligned } Ġ& ĠP ( w _{ t + 1 }\ mid Ġw _{ t }, ... w _{ 0 }, \ text bf Ġ{ DA }) = g (\ text bf Ġ{ W } _{ ho }\ text bf Ġ{ h } _{ t }) Ġ\\ Ġ& Ġw _{ t + 1 } Ġ\ sim ĠP ( w _{ t + 1 }\ mid Ġw _{ t }, Ġw _{ t - 1 }, ... w _{ 0 }, \ text bf Ġ{ DA }) Ġ\ end { aligned } $$ Ġ Ġ Ġ( E q . Ġ25 ) </s> where Ġ$\ text bf Ġ{ DA } =( \ text bf Ġ{ s }, Ġ\ text bf Ġ{ z }) $ Ġ. </s> Training </s> The Ġobjective Ġfunction Ġwas Ġthe Ġnegative Ġlog - like lihood Ġand Ġcomputed Ġby : </s> $$ \ text bf Ġ{ F } (\ the ta Ġ) Ġ= Ġ- \ sum Ġ_ { t = 1 } ^{ T }\ text bf Ġ{ y } _{ t } ^ {\ top Ġ} \ log Ġ{\ text bf Ġ{ p } _{ t }} $$ Ġ Ġ Ġ( E q . Ġ27 ) </s> where : Ġ$\ text bf Ġ{ y } _{ t } $ Ġis Ġthe Ġground Ġtruth Ġtoken Ġdistribution , Ġ$\ text bf Ġ{ p } _{ t } $ Ġis Ġthe Ġpredicted Ġtoken Ġdistribution , Ġ$ T $ Ġis Ġlength Ġof Ġthe Ġinput Ġsentence . ĠThe Ġproposed Ġgenerators Ġwere Ġtrained Ġby Ġtreating Ġeach Ġsentence Ġas Ġa Ġmini - batch Ġwith Ġ$ l _{ 2 } $ Ġregular ization Ġadded Ġto Ġthe Ġobjective Ġfunction Ġfor Ġevery Ġ5 Ġtraining Ġexamples . ĠThe Ġmodels Ġwere Ġinitialized Ġwith Ġa Ġpret rained ĠGl ove Ġword Ġembed ding Ġvectors ĠB IB REF 28 Ġand Ġoptimized Ġby Ġusing Ġsto ch astic Ġgradient Ġdescent Ġand Ġback Ġpropagation Ġthrough Ġtime ĠB IB REF 29 Ġ. ĠEarly Ġstopping Ġmechanism Ġwas Ġimplemented Ġto Ġprevent Ġover - fitting Ġby Ġusing Ġa Ġvalidation Ġset Ġas Ġsuggested Ġin ĠB IB REF 30 Ġ. </s> Dec oding </s> The Ġdecoding Ġconsists Ġof Ġtwo Ġphases : Ġ( i ) Ġover - generation , Ġand Ġ( ii ) Ġre ranking . ĠIn Ġthe Ġover - generation , Ġthe Ġgenerator Ġconditioned Ġon Ġboth Ġrepresentations Ġof Ġthe Ġgiven ĠDA Ġuse Ġa Ġbeam Ġsearch Ġto Ġgenerate Ġa Ġset Ġof Ġcandidate Ġresponses . ĠIn Ġthe Ġre ranking Ġphase , Ġcost Ġof Ġthe Ġgenerator Ġis Ġcomputed Ġto Ġform Ġthe Ġre ranking Ġscore Ġ$\ text bf Ġ{ R } $ Ġas Ġfollows : </s> $$ \ text bf Ġ{ R } Ġ= Ġ\ text bf Ġ{ F } (\ the ta Ġ) Ġ+ Ġ\ lambda Ġ\ text bf Ġ{ ER R } $$ Ġ Ġ Ġ( E q . Ġ29 ) </s> where Ġ$\ lambda Ġ$ Ġis Ġa Ġtrade Ġoff Ġconstant Ġand Ġis Ġset Ġto Ġa Ġlarge Ġvalue Ġin Ġorder Ġto Ġseverely Ġpenal ize Ġnonsensical Ġoutputs . ĠThe Ġslot Ġerror Ġrate Ġ$\ text bf Ġ{ ER R } $ Ġ, Ġwhich Ġis Ġthe Ġnumber Ġof Ġslots Ġgenerated Ġthat Ġis Ġeither Ġmissing Ġor Ġredundant , Ġand Ġis Ġcomputed Ġby : </s> $$ \ text bf Ġ{ ER R } Ġ= Ġ\ frac {\ text bf Ġ{ p } Ġ+ Ġ\ text bf Ġ{ q }} {\ text bf Ġ{ N }} $$ Ġ Ġ Ġ( E q . Ġ30 ) </s> where Ġ$\ text bf Ġ{ N } $ Ġis Ġthe Ġtotal Ġnumber Ġof Ġslots Ġin ĠDA , Ġand Ġ$\ text bf Ġ{ p } $ Ġ, Ġ$\ text bf Ġ{ q } $ Ġis Ġthe Ġnumber Ġof Ġmissing Ġand Ġredundant Ġslots , Ġrespectively . </s> Exper iments </s> We Ġextensively Ġconducted Ġa Ġset Ġof Ġexperiments Ġto Ġassess Ġthe Ġeffectiveness Ġof Ġthe Ġproposed Ġmodels Ġby Ġusing Ġseveral Ġmetrics , Ġdatasets , Ġand Ġmodel Ġarchitectures , Ġin Ġorder Ġto Ġcompare Ġto Ġprior Ġmethods . </s> Dat as ets </s> We Ġassessed Ġthe Ġproposed Ġmodels Ġon Ġfour Ġdifferent ĠNL G Ġdomains : Ġfinding Ġa Ġrestaurant , Ġfinding Ġa Ġhotel , Ġbuying Ġa Ġlaptop , Ġand Ġbuying Ġa Ġtelevision . ĠThe ĠRestaurant Ġand ĠHotel Ġwere Ġcollected Ġin ĠB IB REF 4 Ġ, Ġwhile Ġthe ĠL aptop Ġand ĠTV Ġdatasets Ġhave Ġbeen Ġreleased Ġby ĠB IB REF 22 Ġwith Ġa Ġmuch Ġlarger Ġinput Ġspace Ġbut Ġonly Ġone Ġtraining Ġexample Ġfor Ġeach ĠDA Ġso Ġthat Ġthe Ġsystem Ġmust Ġlearn Ġpartial Ġrealization Ġof Ġconcepts Ġand Ġbe Ġable Ġto Ġrecomb ine Ġand Ġapply Ġthem Ġto Ġunseen ĠD As . ĠThis Ġmakes Ġthe ĠNL G Ġtasks Ġfor Ġthe ĠL aptop Ġand ĠTV Ġdomains Ġbecome Ġmuch Ġharder . ĠThe Ġdataset Ġstatistics Ġare Ġshown Ġin ĠTable Ġ1 Ġ. </s> Exper imental ĠSet ups </s> The Ġgenerators Ġwere Ġimplemented Ġusing Ġthe ĠT ensor Flow Ġlibrary ĠB IB REF 31 Ġand Ġtrained Ġwith Ġtraining , Ġvalidation Ġand Ġtesting Ġratio Ġas Ġ3 : 1 : 1 . ĠThe Ġhidden Ġlayer Ġsize , Ġbeam Ġsize Ġwere Ġset Ġto Ġbe Ġ80 Ġand Ġ10 , Ġrespectively , Ġand Ġthe Ġgenerators Ġwere Ġtrained Ġwith Ġa Ġ$ 70 \ % $ Ġof Ġdrop out Ġrate . ĠWe Ġperformed Ġ5 Ġruns Ġwith Ġdifferent Ġrandom Ġinitialization Ġof Ġthe Ġnetwork Ġand Ġthe Ġtraining Ġis Ġterminated Ġby Ġusing Ġearly Ġstopping . ĠWe Ġthen Ġchose Ġa Ġmodel Ġthat Ġyields Ġthe Ġhighest ĠB LE U Ġscore Ġon Ġthe Ġvalidation Ġset Ġas Ġshown Ġin ĠTable Ġ2 Ġ. ĠSince Ġthe Ġtrained Ġmodels Ġcan Ġdiffer Ġdepending Ġon Ġthe Ġinitialization , Ġwe Ġalso Ġreport Ġthe Ġresults Ġwhich Ġwere Ġaveraged Ġover Ġ5 Ġrandomly Ġinitialized Ġnetworks . ĠNote Ġthat , Ġexcept Ġthe Ġresults Ġreported Ġin ĠTable Ġ2 Ġ, Ġall Ġthe Ġresults Ġshown Ġwere Ġaveraged Ġover Ġ5 Ġrandomly Ġinitialized Ġnetworks . ĠWe Ġset Ġ$\ lambda Ġ$ Ġto Ġ1000 Ġto Ġseverely Ġdiscourage Ġthe Ġre rank er Ġfrom Ġselecting Ġutter ances Ġwhich Ġcontain Ġeither Ġredundant Ġor Ġmissing Ġslots . ĠFor Ġeach ĠDA , Ġwe Ġover - generated Ġ20 Ġcandidate Ġsentences Ġand Ġselected Ġthe Ġtop Ġ5 Ġreal izations Ġafter Ġre ranking . ĠMoreover , Ġin Ġorder Ġto Ġbetter Ġunderstand Ġthe Ġeffectiveness Ġof Ġour Ġproposed Ġmethods , Ġwe : Ġ( i ) Ġperformed Ġan Ġab lation Ġexperiments Ġto Ġdemonstrate Ġthe Ġcontribution Ġof Ġeach Ġproposed Ġcells Ġ( T ables Ġ2 Ġ, Ġ3 Ġ), Ġ( ii ) Ġtrained Ġthe Ġmodels Ġon Ġthe ĠL aptop Ġdomain Ġwith Ġvaried Ġproportion Ġof Ġtraining Ġdata , Ġstarting Ġfrom Ġ$ 10 \ % $ Ġto Ġ$ 100 \ % $ Ġ( Figure Ġ3 Ġ), Ġ( iii ) Ġtrained Ġgeneral Ġmodels Ġby Ġmerging Ġall Ġthe Ġdata Ġfrom Ġfour Ġdomains Ġtogether Ġand Ġtested Ġthem Ġin Ġeach Ġindividual Ġdomain Ġ( Figure Ġ4 Ġ), Ġand Ġ( iv ) Ġtrained Ġadaptation Ġmodels Ġon Ġmerging Ġdata Ġfrom Ġrestaurant Ġand Ġhotel Ġdomains , Ġthen Ġfine Ġtuned Ġthe Ġmodel Ġon Ġlaptop Ġdomain Ġwith Ġvaried Ġamount Ġof Ġadaptation Ġdata Ġ( Figure Ġ5 Ġ). </s> E val uation ĠMet rics Ġand ĠBas elines </s> The Ġgenerator Ġperformance Ġwas Ġassessed Ġon Ġthe Ġtwo Ġevaluation Ġmetrics : Ġthe ĠB LE U Ġand Ġthe Ġslot Ġerror Ġrate ĠER R Ġby Ġadopting Ġcode Ġfrom Ġan Ġopen Ġsource Ġbenchmark Ġtool kit Ġfor ĠNatural ĠLanguage ĠGeneration . ĠWe Ġcompared Ġthe Ġproposed Ġmodels Ġagainst Ġthree Ġstrong Ġbas elines Ġwhich Ġhave Ġbeen Ġrecently Ġpublished Ġas Ġstate - of - the - art ĠNL G Ġbenchmarks [ ]. Ġhttps :// github . com / sh awn w un / R NN LG </s> HL ST M Ġproposed Ġby ĠB IB REF 3 Ġth ws j y 15 Ġwhich Ġused Ġa Ġhe uristic Ġgate Ġto Ġensure Ġthat Ġall Ġof Ġthe Ġslot - value Ġinformation Ġwas Ġaccurately Ġcaptured Ġwhen Ġgenerating . </s> S CL ST M Ġproposed Ġby ĠB IB REF 4 Ġw ens cl st m 15 Ġwhich Ġcan Ġjointly Ġlearn Ġthe Ġg ating Ġsignal Ġand Ġlanguage Ġmodel . </s> Enc - Dec Ġproposed Ġby ĠB IB REF 8 Ġwent oward Ġwhich Ġapplied Ġthe Ġattention - based Ġenc oder - dec oder Ġarchitecture . </s> Results </s> We Ġconducted Ġextensive Ġexperiments Ġon Ġour Ġmodels Ġand Ġcompared Ġagainst Ġthe Ġprevious Ġmethods . ĠOverall , Ġthe Ġproposed Ġmodels Ġconsistently Ġachieve Ġthe Ġbetter Ġperformance Ġregarding Ġboth Ġevaluation Ġmetrics Ġacross Ġall Ġdomains Ġin Ġall Ġtest Ġcases . </s> The Ġab lation Ġstudies Ġ( T ables Ġ2 Ġ, Ġ3 Ġ) Ġdemonstrate Ġthe Ġcontribution Ġof Ġdifferent Ġmodel Ġcomponents Ġin Ġwhich Ġthe Ġmodels Ġwere Ġassessed Ġwithout ĠAdjust ment Ġcell Ġ( w / o ĠA ), Ġor Ġwithout ĠRef inement Ġcell Ġ( w / o ĠR ). ĠIt Ġclearly Ġsees Ġthat Ġthe ĠAdjust ment Ġcell Ġcontributes Ġto Ġreducing Ġthe Ġslot Ġerror Ġrate ĠER R Ġscore Ġsince Ġit Ġcan Ġeffectively Ġprevent Ġthe Ġundesirable Ġslot - value Ġpair Ġrepet itions Ġby Ġg ating Ġthe ĠDA Ġvector Ġ$\ text bf Ġ{ s } $ Ġ. ĠA Ġcomparison Ġbetween Ġthe ĠAR ED - based Ġmodels Ġ( den oted Ġby Ġ$ ^ {\ sharp Ġ} $ Ġin ĠTable Ġ2 Ġ) Ġshows Ġthat Ġthe Ġproposed Ġmodels Ġnot Ġonly Ġhave Ġbetter Ġperformance Ġwith Ġhigher Ġthe ĠB LE U Ġscore Ġbut Ġalso Ġsignificantly Ġreduce Ġthe Ġslot Ġerror Ġrate ĠER R Ġscore Ġby Ġa Ġlarge Ġmargin Ġabout Ġ$ 2 \ % $ Ġto Ġ$ 4 \ % $ Ġin Ġevery Ġdatasets . ĠMoreover , Ġa Ġcomparison Ġbetween Ġthe Ġmodels Ġwith Ġg ating Ġthe ĠDA Ġvector Ġalso Ġindicates Ġthat Ġthe Ġproposed Ġmodels Ġ( w / o ĠR , ĠR AL ST M ) Ġhave Ġsignificant Ġimproved Ġperformance Ġon Ġboth Ġthe Ġevaluation Ġmetrics Ġacross Ġthe Ġfour Ġdomains Ġcompared Ġto Ġthe ĠS CL ST M Ġmodel . ĠThe ĠR AL ST M Ġcell Ġwithout Ġthe ĠRef inement Ġcell Ġis Ġsimilar Ġas Ġthe ĠS CL ST M Ġcell . ĠHowever , Ġit Ġobtained Ġthe Ġresults Ġmuch Ġbetter Ġthan Ġthe ĠS CL ST M Ġbas elines . ĠThis Ġstip ulates Ġthe Ġnecessary Ġof Ġthe ĠL ST M Ġenc oder Ġand Ġthe ĠAl ign er Ġin Ġeffectively Ġpartial Ġlearning Ġthe Ġcorrelated Ġorder Ġbetween Ġslot - value Ġrepresentation Ġin Ġthe ĠD As , Ġespecially Ġfor Ġthe Ġunseen Ġdomain Ġwhere Ġthere Ġis Ġonly Ġone Ġtraining Ġexample Ġfor Ġeach ĠDA . ĠTable Ġ3 Ġfurther Ġdemonstrates Ġthe Ġstable Ġstrength Ġof Ġour Ġmodels Ġsince Ġthe Ġresults ' Ġpattern Ġstays Ġunchanged Ġcompared Ġto Ġthose Ġin ĠTable Ġ2 Ġ. </s> Figure Ġ3 Ġshows Ġa Ġcomparison Ġof Ġthree Ġmodels Ġ( Enc - Dec , ĠS CL ST M , Ġand ĠR AL ST M ) Ġwhich Ġwere Ġtrained Ġfrom Ġscratch Ġon Ġthe Ġunseen Ġlaptop Ġdomain Ġin Ġvaried Ġproportion Ġof Ġtraining Ġdata , Ġfrom Ġ$ 1 \ % $ Ġto Ġ$ 100 \ % $ Ġ. ĠIt Ġclearly Ġshows Ġthat Ġthe ĠR AL ST M Ġoutper forms Ġthe Ġprevious Ġmodels Ġin Ġall Ġcases , Ġwhile Ġthe ĠEnc - Dec Ġhas Ġa Ġmuch Ġgreater ĠER R Ġscore Ġcomparing Ġto Ġthe Ġtwo Ġmodels . </s> A Ġcomparison Ġof Ġtop Ġresponses Ġgenerated Ġfor Ġsome Ġinput ĠD As Ġbetween Ġdifferent Ġmodels Ġare Ġshown Ġin ĠTable Ġ4 Ġ. ĠWhile Ġthe Ġprevious Ġmodels Ġstill Ġproduce Ġsome Ġerrors Ġ( missing Ġand Ġmisplaced Ġinformation ), Ġthe Ġproposed Ġmodels Ġ( RAL ST M Ġand Ġthe Ġmodels ĠAll 2 * Ġtrained Ġby Ġpool ing Ġall Ġdatasets Ġtogether ) Ġcan Ġgenerate Ġappropriate Ġsentences . ĠWe Ġalso Ġfound Ġthat Ġthe Ġproposed Ġmodels Ġtend Ġto Ġgenerate Ġmore Ġcomplete Ġand Ġconcise Ġsentences Ġthan Ġthe Ġother Ġmodels . </s> All Ġthese Ġprove Ġthe Ġimportance Ġof Ġthe Ġproposed Ġcomponents : Ġthe ĠRef inement Ġcell Ġin Ġaggreg ating Ġand Ġselecting Ġthe Ġattentive Ġinformation , Ġand Ġthe ĠAdjust ment Ġcell Ġin Ġcontrolling Ġthe Ġfeature Ġvector Ġ( see ĠExamples Ġin ĠFigure Ġ6 Ġ). </s> Figure Ġ4 Ġshows Ġa Ġcomparison Ġperformance Ġof Ġgeneral Ġmodels Ġas Ġdescribed Ġin ĠSection Ġ" Exper imental ĠSet ups " Ġ. ĠThe Ġresults Ġare Ġconsistent Ġwith Ġthe ĠFigure Ġ3 Ġ, Ġin Ġwhich Ġthe ĠR AL ST M Ġhas Ġbetter Ġperformance Ġthan Ġthe ĠEnc - Dec Ġand ĠS CL ST M Ġon Ġall Ġdomains Ġin Ġterms Ġof Ġthe ĠB LE U Ġand Ġthe ĠER R Ġscores , Ġwhile Ġthe ĠEnc - Dec Ġhas Ġdifficulties Ġin Ġreducing Ġthe ĠER R Ġscore . ĠThis Ġindicates Ġthe Ġrelevant Ġcontribution Ġof Ġthe Ġproposed Ġcomponent ĠRef inement Ġand ĠAdjust ment Ġcells Ġto Ġthe Ġoriginal ĠAR ED Ġarchitecture , Ġin Ġwhich Ġthe ĠRef inement Ġwith Ġattention al Ġg ating Ġcan Ġeffectively Ġselect Ġand Ġaggregate Ġthe Ġinformation Ġbefore Ġputting Ġthem Ġinto Ġthe Ġtraditional ĠL ST M Ġcell , Ġwhile Ġthe ĠAdjust ment Ġwith Ġg ating ĠDA Ġvector Ġcan Ġeffectively Ġcontrol Ġthe Ġinformation Ġflow Ġduring Ġgeneration . </s> Figure Ġ5 Ġshows Ġdomain Ġscal ability Ġof Ġthe Ġthree Ġmodels Ġin Ġwhich Ġthe Ġmodels Ġwere Ġfirst Ġtrained Ġon Ġthe Ġmerging Ġout - of - domain ĠRestaurant Ġand ĠHotel Ġdatasets , Ġthen Ġfine Ġtuned Ġthe Ġparameters Ġwith Ġvaried Ġamount Ġof Ġin - domain Ġtraining Ġdata Ġ( l aptop Ġdomain ). ĠThe ĠR AL ST M Ġmodel Ġoutper forms Ġthe Ġprevious Ġmodel Ġin Ġboth Ġcases Ġwhere Ġthe Ġsufficient Ġin - domain Ġdata Ġis Ġused Ġ( as Ġin ĠFigure Ġ5 Ġ- left ) Ġand Ġthe Ġlimited Ġin - domain Ġdata Ġis Ġused Ġ( Figure Ġ5 Ġ- right ). ĠThe ĠFigure Ġ5 Ġ- right Ġalso Ġindicates Ġthat Ġthe ĠR AL ST M Ġmodel Ġcan Ġadapt Ġto Ġa Ġnew , Ġunseen Ġdomain Ġfaster Ġthan Ġthe Ġprevious Ġmodels . </s> Conclusion Ġand ĠFuture ĠWork </s> We Ġpresent Ġan Ġextension Ġof ĠAR ED Ġmodel , Ġin Ġwhich Ġan ĠR AL ST M Ġcomponent Ġis Ġintroduced Ġto Ġselect Ġand Ġaggregate Ġsemantic Ġelements Ġproduced Ġby Ġthe ĠEnc oder , Ġand Ġto Ġgenerate Ġthe Ġrequired Ġsentence . ĠWe Ġassessed Ġthe Ġproposed Ġmodels Ġon Ġfour ĠNL G Ġdomains Ġand Ġcompared Ġto Ġthe Ġstate - of - the - art Ġgenerators . ĠThe Ġproposed Ġmodels Ġempir ically Ġshow Ġconsistent Ġimprovement Ġover Ġthe Ġprevious Ġmethods Ġin Ġboth Ġthe ĠB LE U Ġand ĠER R Ġevaluation Ġmetrics . ĠThe Ġproposed Ġmodels Ġalso Ġshow Ġan Ġability Ġto Ġextend Ġto Ġa Ġnew , Ġunseen Ġdomain Ġno Ġmatter Ġhow Ġmuch Ġthe Ġin - domain Ġtraining Ġdata Ġwas Ġfed . ĠIn Ġthe Ġfuture , Ġit Ġwould Ġbe Ġinteresting Ġto Ġapply Ġthe Ġproposed Ġmodel Ġto Ġother Ġtasks Ġthat Ġcan Ġbe Ġmodeled Ġbased Ġon Ġthe Ġenc oder - dec oder Ġarchitecture , Ġi . e ., Ġimage Ġcaption ing , Ġreading Ġcomprehension , Ġand Ġmachine Ġtranslation .