BLEU-4, NIST-4, ROUGE-4
EXTRACTIVE
BLEU-4, NIST-4, ROUGE-4
EXTRACTIVE
Following lebret2016neural, we used BLEU-4, NIST-4 and ROUGE-4 as the evaluation metrics. We first make a few observations based on the results on the English dataset (Table TABREF15 ). The basic seq2seq model, as well as the model proposed by weather16, perform better than the model proposed by lebret2016neural. Our final model with bifocal attention and gated orthogonalization gives the best performance and does 10% (relative) better than the closest baseline (basic seq2seq) and 21% (relative) better than the current state of the art method BIBREF0 . In Table TABREF16 , we show some qualitative examples of the output generated by different models.
Following lebret2016neural, we used BLEU-4, NIST-4 and ROUGE-4 as the evaluation metrics. We first make a few observations based on the results on the English dataset (Table TABREF15 ). The basic seq2seq model, as well as the model proposed by weather16, perform better than the model proposed by lebret2016neural. Our final model with bifocal attention and gated orthogonalization gives the best performance and does 10% (relative) better than the closest baseline (basic seq2seq) and 21% (relative) better than the current state of the art method BIBREF0 . In Table TABREF16 , we show some qualitative examples of the output generated by different models.
1804.07789
Introduction </s> R end ering Ġnatural Ġlanguage Ġdescriptions Ġfrom Ġstructured Ġdata Ġis Ġrequired Ġin Ġa Ġwide Ġvariety Ġof Ġcommercial Ġapplications Ġsuch Ġas Ġgenerating Ġdescriptions Ġof Ġproducts , Ġhotels , Ġfurniture , Ġetc ., Ġfrom Ġa Ġcorresponding Ġtable Ġof Ġfacts Ġabout Ġthe Ġentity . ĠSuch Ġa Ġtable Ġtypically Ġcontains Ġ{ field , Ġvalue } Ġpairs Ġwhere Ġthe Ġfield Ġis Ġa Ġproperty Ġof Ġthe Ġentity Ġ( e . g ., Ġcolor ) Ġand Ġthe Ġvalue Ġis Ġa Ġset Ġof Ġpossible Ġassignments Ġto Ġthis Ġproperty Ġ( e . g ., Ġcolor Ġ= Ġred ). ĠAnother Ġexample Ġof Ġthis Ġis Ġthe Ġrecently Ġintroduced Ġtask Ġof Ġgenerating Ġone Ġline Ġbiography Ġdescriptions Ġfrom Ġa Ġgiven ĠWikipedia Ġinf ob ox ĠB IB REF 0 Ġ. ĠThe ĠWikipedia Ġinf ob ox Ġserves Ġas Ġa Ġtable Ġof Ġfacts Ġabout Ġa Ġperson Ġand Ġthe Ġfirst Ġsentence Ġfrom Ġthe Ġcorresponding Ġarticle Ġserves Ġas Ġa Ġone Ġline Ġdescription Ġof Ġthe Ġperson . ĠFigure ĠFIG REF 2 Ġillustrates Ġan Ġexample Ġinput Ġinf ob ox Ġwhich Ġcontains Ġfields Ġsuch Ġas ĠBorn , ĠRes idence , ĠNational ity , ĠFields , ĠInstit utions Ġand ĠAlma ĠM ater . ĠEach Ġfield Ġfurther Ġcontains Ġsome Ġwords Ġ( e . g ., Ġparticle Ġphysics , Ġmany - body Ġtheory , Ġetc .). ĠThe Ġcorresponding Ġdescription Ġis Ġcoherent Ġwith Ġthe Ġinformation Ġcontained Ġin Ġthe Ġinf ob ox . </s> Note Ġthat Ġthe Ġnumber Ġof Ġfields Ġin Ġthe Ġinf ob ox Ġand Ġthe Ġordering Ġof Ġthe Ġfields Ġwithin Ġthe Ġinf ob ox Ġvaries Ġfrom Ġperson Ġto Ġperson . ĠGiven Ġthe Ġlarge Ġsize Ġ( 700 K Ġexamples ) Ġand Ġheter ogeneous Ġnature Ġof Ġthe Ġdataset Ġwhich Ġcontains Ġbi ographies Ġof Ġpeople Ġfrom Ġdifferent Ġbackgrounds Ġ( sports , Ġpolitics , Ġarts , Ġetc .), Ġit Ġis Ġhard Ġto Ġcome Ġup Ġwith Ġsimple Ġrule - based Ġtemplates Ġfor Ġgenerating Ġnatural Ġlanguage Ġdescriptions Ġfrom Ġinf ob ox es , Ġthereby Ġmaking Ġa Ġcase Ġfor Ġdata - driven Ġmodels . ĠBased Ġon Ġthe Ġrecent Ġsuccess Ġof Ġdata - driven Ġneural Ġmodels Ġfor Ġvarious Ġother ĠNL G Ġtasks ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, Ġone Ġsimple Ġchoice Ġis Ġto Ġtreat Ġthe Ġinf ob ox Ġas Ġa Ġsequence Ġof Ġ{ field , Ġvalue } Ġpairs Ġand Ġuse Ġa Ġstandard Ġseq 2 seq Ġmodel Ġfor Ġthis Ġtask . ĠHowever , Ġsuch Ġa Ġmodel Ġis Ġtoo Ġgeneric Ġand Ġdoes Ġnot Ġexploit Ġthe Ġspecific Ġcharacteristics Ġof Ġthis Ġtask Ġas Ġexplained Ġbelow . ĠFirst , Ġnote Ġthat Ġwhile Ġgenerating Ġsuch Ġdescriptions Ġfrom Ġstructured Ġdata , Ġa Ġhuman Ġkeeps Ġtrack Ġof Ġinformation Ġat Ġtwo Ġlevels . ĠSpecifically , Ġat Ġa Ġmacro Ġlevel , Ġshe Ġwould Ġfirst Ġdecide Ġwhich Ġfield Ġto Ġmention Ġnext Ġand Ġthen Ġat Ġa Ġmicro Ġlevel Ġdecide Ġwhich Ġof Ġthe Ġvalues Ġin Ġthe Ġfield Ġneeds Ġto Ġbe Ġmentioned Ġnext . ĠFor Ġexample , Ġshe Ġfirst Ġdecides Ġthat Ġat Ġthe Ġcurrent Ġstep , Ġthe Ġfield Ġoccupation Ġneeds Ġattention Ġand Ġthen Ġdecides Ġwhich Ġis Ġthe Ġnext Ġappropriate Ġoccupation Ġto Ġattend Ġto Ġfrom Ġthe Ġset Ġof Ġoccupations Ġ( actor , Ġdirector , Ġproducer , Ġetc .). ĠTo Ġenable Ġthis , Ġwe Ġuse Ġa Ġb if ocal Ġattention Ġmechanism Ġwhich Ġcomp utes Ġan Ġattention Ġover Ġfields Ġat Ġa Ġmacro Ġlevel Ġand Ġover Ġvalues Ġat Ġa Ġmicro Ġlevel . ĠWe Ġthen Ġfuse Ġthese Ġattention Ġweights Ġsuch Ġthat Ġthe Ġattention Ġweight Ġfor Ġa Ġfield Ġalso Ġinfluences Ġthe Ġattention Ġover Ġthe Ġvalues Ġwithin Ġit . ĠFinally , Ġwe Ġfeed Ġa Ġfused Ġcontext Ġvector Ġto Ġthe Ġdec oder Ġwhich Ġcontains Ġboth Ġfield Ġlevel Ġand Ġword Ġlevel Ġinformation . ĠNote Ġthat Ġsuch Ġtwo - level Ġattention Ġmechanisms ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġhave Ġbeen Ġused Ġin Ġthe Ġcontext Ġof Ġun struct ured Ġdata Ġ( as Ġopposed Ġto Ġstructured Ġdata Ġin Ġour Ġcase ), Ġwhere Ġat Ġa Ġmacro Ġlevel Ġone Ġneeds Ġto Ġpay Ġattention Ġto Ġsentences Ġand Ġat Ġa Ġmicro Ġlevel Ġto Ġwords Ġin Ġthe Ġsentences . </s> Next , Ġwe Ġobserve Ġthat Ġwhile Ġrendering Ġthe Ġoutput , Ġonce Ġthe Ġmodel Ġpays Ġattention Ġto Ġa Ġfield Ġ( say , Ġoccupation ) Ġit Ġneeds Ġto Ġstay Ġon Ġthis Ġfield Ġfor Ġa Ġfew Ġtim est eps Ġ( t ill Ġall Ġthe Ġoccupations Ġare Ġproduced Ġin Ġthe Ġoutput ). ĠWe Ġrefer Ġto Ġthis Ġas Ġthe Ġstay Ġon Ġbehavior . ĠFurther , Ġwe Ġnote Ġthat Ġonce Ġthe Ġtokens Ġof Ġa Ġfield Ġare Ġreferred Ġto , Ġthey Ġare Ġusually Ġnot Ġreferred Ġto Ġlater . ĠFor Ġexample , Ġonce Ġall Ġthe Ġoccupations Ġhave Ġbeen Ġlisted Ġin Ġthe Ġoutput Ġwe Ġwill Ġnever Ġvisit Ġthe Ġoccupation Ġfield Ġagain Ġbecause Ġthere Ġis Ġnothing Ġleft Ġto Ġsay Ġabout Ġit . ĠWe Ġrefer Ġto Ġthis Ġas Ġthe Ġnever Ġlook Ġback Ġbehavior . ĠTo Ġmodel Ġthe Ġstay Ġon Ġbehaviour , Ġwe Ġintroduce Ġa Ġforget Ġ( or Ġremember ) Ġgate Ġwhich Ġacts Ġas Ġa Ġsignal Ġto Ġdecide Ġwhen Ġto Ġforget Ġthe Ġcurrent Ġfield Ġ( or Ġequival ently Ġto Ġdecide Ġtill Ġwhen Ġto Ġremember Ġthe Ġcurrent Ġfield ). ĠTo Ġmodel Ġthe Ġnever Ġlook Ġback Ġbehaviour Ġwe Ġintroduce Ġa Ġg ated Ġorth og onal ization Ġmechanism Ġwhich Ġensures Ġthat Ġonce Ġa Ġfield Ġis Ġforgotten , Ġsubsequent Ġfield Ġcontext Ġvectors Ġfed Ġto Ġthe Ġdec oder Ġare Ġorth og onal Ġto Ġ( or Ġdifferent Ġfrom ) Ġthe Ġprevious Ġfield Ġcontext Ġvectors . </s> We Ġexperiment Ġwith Ġthe ĠWiki Bio Ġdataset ĠB IB REF 0 Ġwhich Ġcontains Ġaround Ġ700 K Ġ{ inf ob ox , Ġdescription } Ġpairs Ġand Ġhas Ġa Ġvocabulary Ġof Ġaround Ġ400 K Ġwords . ĠWe Ġshow Ġthat Ġthe Ġproposed Ġmodel Ġgives Ġa Ġrelative Ġimprovement Ġof Ġ21 % Ġand Ġ20 % Ġas Ġcompared Ġto Ġcurrent Ġstate Ġof Ġthe Ġart Ġmodels ĠB IB REF 0 Ġ, ĠB IB REF 9 Ġon Ġthis Ġdataset . ĠThe Ġproposed Ġmodel Ġalso Ġgives Ġa Ġrelative Ġimprovement Ġof Ġ10 % Ġas Ġcompared Ġto Ġthe Ġbasic Ġseq 2 seq Ġmodel . ĠFurther , Ġwe Ġintroduce Ġnew Ġdatasets Ġfor ĠFrench Ġand ĠGerman Ġon Ġthe Ġsame Ġlines Ġas Ġthe ĠEnglish ĠWiki Bio Ġdataset . ĠEven Ġon Ġthese Ġtwo Ġdatasets , Ġour Ġmodel Ġoutper forms Ġthe Ġstate Ġof Ġthe Ġart Ġmethods Ġmentioned Ġabove . </s> Related Ġwork </s> Natural ĠLanguage ĠGeneration Ġhas Ġalways Ġbeen Ġof Ġinterest Ġto Ġthe Ġresearch Ġcommunity Ġand Ġhas Ġreceived Ġa Ġlot Ġof Ġattention Ġin Ġthe Ġpast . ĠThe Ġapproaches Ġfor ĠNL G Ġrange Ġfrom Ġ( i ) Ġrule Ġbased Ġapproaches Ġ( e . g ., ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ) Ġ( ii ) Ġmodular Ġstatistical Ġapproaches Ġwhich Ġdivide Ġthe Ġprocess Ġinto Ġthree Ġphases Ġ( plan ning , Ġselection Ġand Ġsurface Ġrealization ) Ġand Ġuse Ġdata Ġdriven Ġapproaches Ġfor Ġone Ġor Ġmore Ġof Ġthese Ġphases ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ, ĠB IB REF 19 Ġ( iii ) Ġhybrid Ġapproaches Ġwhich Ġrely Ġon Ġa Ġcombination Ġof Ġhand crafted Ġrules Ġand Ġcorpus Ġstatistics ĠB IB REF 20 Ġ, ĠB IB REF 21 Ġ, ĠB IB REF 22 Ġand Ġ( iv ) Ġthe Ġmore Ġrecent Ġneural Ġnetwork Ġbased Ġmodels ĠB IB REF 1 Ġ. </s> Ne ural Ġmodels Ġfor ĠNL G Ġhave Ġbeen Ġproposed Ġin Ġthe Ġcontext Ġof Ġvarious Ġtasks Ġsuch Ġas Ġmachine Ġtranslation ĠB IB REF 1 Ġ, Ġdocument Ġsummar ization ĠB IB REF 2 Ġ, ĠB IB REF 4 Ġ, Ġparaph rase Ġgeneration ĠB IB REF 23 Ġ, Ġimage Ġcaption ing ĠB IB REF 24 Ġ, Ġvideo Ġsummar ization ĠB IB REF 25 Ġ, Ġquery Ġbased Ġdocument Ġsummar ization ĠB IB REF 5 Ġand Ġso Ġon . ĠMost Ġof Ġthese Ġmodels Ġare Ġdata Ġhungry Ġand Ġare Ġtrained Ġon Ġlarge Ġamounts Ġof Ġdata . ĠOn Ġthe Ġother Ġhand , ĠNL G Ġfrom Ġstructured Ġdata Ġhas Ġlargely Ġbeen Ġstudied Ġin Ġthe Ġcontext Ġof Ġsmall Ġdatasets Ġsuch Ġas ĠWeather Gov ĠB IB REF 26 Ġ, ĠRobo C up ĠB IB REF 27 Ġ, ĠNFL ĠRec aps ĠB IB REF 15 Ġ, ĠPro digy - Met e o ĠB IB REF 28 Ġand ĠT U NA ĠChallenge ĠB IB REF 29 Ġ. ĠRecently Ġweather 16 Ġproposed ĠR NN / L ST M Ġbased Ġneural Ġenc oder - dec oder Ġmodels Ġwith Ġattention Ġfor ĠWeather Gov Ġand ĠRobo C up Ġdatasets . </s> Unlike Ġthe Ġdatasets Ġmentioned Ġabove , Ġthe Ġbiography Ġdataset Ġintroduced Ġby Ġle b ret 2016 ne ural Ġis Ġlarger Ġ( 700 K Ġ{ table , Ġdescriptions } Ġpairs ) Ġand Ġhas Ġa Ġmuch Ġlarger Ġvocabulary Ġ( 400 K Ġwords Ġas Ġopposed Ġto Ġaround Ġ350 Ġor Ġfewer Ġwords Ġin Ġthe Ġabove Ġdatasets ). ĠFurther , Ġunlike Ġthe Ġfeed - forward Ġneural Ġnetwork Ġbased Ġmodel Ġproposed Ġby ĠB IB REF 0 Ġwe Ġuse Ġa Ġsequence Ġto Ġsequence Ġmodel Ġand Ġintroduce Ġcomponents Ġto Ġaddress Ġthe Ġpeculiar Ġcharacteristics Ġof Ġthe Ġtask . ĠSpecifically , Ġwe Ġintroduce Ġneural Ġcomponents Ġto Ġaddress Ġthe Ġneed Ġfor Ġattention Ġat Ġtwo Ġlevels Ġand Ġto Ġaddress Ġthe Ġstay Ġon Ġand Ġnever Ġlook Ġback Ġbehaviour Ġrequired Ġby Ġthe Ġdec oder . ĠKidd on Z C 16 Ġhave Ġexplored Ġthe Ġuse Ġof Ġcheck lists Ġto Ġtrack Ġpreviously Ġvisited Ġingredients Ġwhile Ġgenerating Ġrecipes Ġfrom Ġingredients . ĠNote Ġthat Ġtwo - level Ġattention Ġmechanisms Ġhave Ġalso Ġbeen Ġused Ġin Ġthe Ġcontext Ġof Ġsummar ization ĠB IB REF 6 Ġ, Ġdocument Ġclassification ĠB IB REF 7 Ġ, Ġdialog Ġsystems ĠB IB REF 8 Ġ, Ġetc . ĠHowever , Ġthese Ġworks Ġdeal Ġwith Ġun struct ured Ġdata Ġ( sent ences Ġat Ġthe Ġhigher Ġlevel Ġand Ġwords Ġat Ġa Ġlower Ġlevel ) Ġas Ġopposed Ġto Ġstructured Ġdata Ġin Ġour Ġcase . </s> Prop osed Ġmodel </s> As Ġinput Ġwe Ġare Ġgiven Ġan Ġinf ob ox ĠIN LINE FORM 0 Ġ, Ġwhich Ġis Ġa Ġset Ġof Ġpairs ĠIN LINE FORM 1 Ġwhere ĠIN LINE FORM 2 Ġcorresponds Ġto Ġfield Ġnames Ġand ĠIN LINE FORM 3 Ġis Ġthe Ġsequence Ġof Ġcorresponding Ġvalues Ġand ĠIN LINE FORM 4 Ġis Ġthe Ġtotal Ġnumber Ġof Ġfields Ġin ĠIN LINE FORM 5 Ġ. ĠFor Ġexample , ĠIN LINE FORM 6 Ġcould Ġbe Ġone Ġsuch Ġpair Ġin Ġthis Ġset . ĠGiven Ġsuch Ġan Ġinput , Ġthe Ġtask Ġis Ġto Ġgenerate Ġa Ġdescription ĠIN LINE FORM 7 Ġcontaining ĠIN LINE FORM 8 Ġwords . ĠA Ġsimple Ġsolution Ġis Ġto Ġtreat Ġthe Ġinf ob ox Ġas Ġa Ġsequence Ġof Ġfields Ġfollowed Ġby Ġthe Ġvalues Ġcorresponding Ġto Ġthe Ġfield Ġin Ġthe Ġorder Ġof Ġtheir Ġappearance Ġin Ġthe Ġinf ob ox . ĠFor Ġexample , Ġthe Ġinf ob ox Ġcould Ġbe Ġflattened Ġto Ġproduce Ġthe Ġfollowing Ġinput Ġsequence Ġ( the Ġwords Ġin Ġbold Ġare Ġfield Ġnames Ġwhich Ġact Ġas Ġdelim it ers ) </s> [ Name ] ĠJohn ĠDoe Ġ[ Birth _ Date ] Ġ19 ĠMarch Ġ1981 Ġ[ National ity ] ĠIndian Ġ.. ... </s> The Ġproblem Ġcan Ġthen Ġbe Ġcast Ġas Ġa Ġseq 2 seq Ġgeneration Ġproblem Ġand Ġcan Ġbe Ġmodeled Ġusing Ġa Ġstandard Ġneural Ġarchitecture Ġcomprising Ġof Ġthree Ġcomponents Ġ( i ) Ġan Ġinput Ġenc oder Ġ( using ĠGR U / L ST M Ġcells ), Ġ( ii ) Ġan Ġattention Ġmechanism Ġto Ġattend Ġto Ġimportant Ġvalues Ġin Ġthe Ġinput Ġsequence Ġat Ġeach Ġtime Ġstep Ġand Ġ( iii ) Ġa Ġdec oder Ġto Ġdecode Ġthe Ġoutput Ġone Ġword Ġat Ġa Ġtime Ġ( again , Ġusing ĠGR U / L ST M Ġcells ). ĠHowever , Ġthis Ġstandard Ġmodel Ġis Ġtoo Ġgeneric Ġand Ġdoes Ġnot Ġexploit Ġthe Ġspecific Ġcharacteristics Ġof Ġthis Ġtask . ĠWe Ġpropose Ġadditional Ġcomponents , Ġviz ., Ġ( i ) Ġa Ġfused Ġb if ocal Ġattention Ġmechanism Ġwhich Ġoperates Ġon Ġfields Ġ( mac ro ) Ġand Ġvalues Ġ( micro ) Ġand Ġ( ii ) Ġa Ġg ated Ġorth og onal ization Ġmechanism Ġto Ġmodel Ġstay Ġon Ġand Ġnever Ġlook Ġback Ġbehavior . </s> F used ĠB if ocal ĠAttention ĠMechan ism </s> Int uitive ly , Ġwhen Ġa Ġhuman Ġwrites Ġa Ġdescription Ġfrom Ġa Ġtable Ġshe Ġkeeps Ġtrack Ġof Ġinformation Ġat Ġtwo Ġlevels . ĠAt Ġthe Ġmacro Ġlevel , Ġit Ġis Ġimportant Ġto Ġdecide Ġwhich Ġis Ġthe Ġappropriate Ġfield Ġto Ġattend Ġto Ġnext Ġand Ġat Ġa Ġmicro Ġlevel Ġ( i . e ., Ġwithin Ġa Ġfield ) Ġit Ġis Ġimportant Ġto Ġknow Ġwhich Ġvalues Ġto Ġattend Ġto Ġnext . ĠTo Ġcapture Ġthis Ġbehavior , Ġwe Ġuse Ġa Ġb if ocal Ġattention Ġmechanism Ġas Ġdescribed Ġbelow . </s> Mac ro ĠAttention : ĠConsider Ġthe ĠIN LINE FORM 0 Ġ- th Ġfield ĠIN LINE FORM 1 Ġwhich Ġhas Ġvalues ĠIN LINE FORM 2 Ġ. ĠLet ĠIN LINE FORM 3 Ġbe Ġthe Ġrepresentation Ġof Ġthis Ġfield Ġin Ġthe Ġinf ob ox . ĠThis Ġrepresentation Ġcan Ġeither Ġbe Ġ( i ) Ġthe Ġword Ġembed ding Ġof Ġthe Ġfield Ġname Ġor Ġ( ii ) Ġsome Ġfunction ĠIN LINE FORM 4 Ġof Ġthe Ġvalues Ġin Ġthe Ġfield Ġor Ġ( iii ) Ġa Ġconc aten ation Ġof Ġ( i ) Ġand Ġ( ii ). ĠThe Ġfunction ĠIN LINE FORM 5 Ġcould Ġsimply Ġbe Ġthe Ġsum Ġor Ġaverage Ġof Ġthe Ġembed d ings Ġof Ġthe Ġvalues Ġin Ġthe Ġfield . ĠAltern ately , Ġthis Ġfunction Ġcould Ġbe Ġa ĠGR U Ġ( or ĠL ST M ) Ġwhich Ġtreats Ġthese Ġvalues Ġwithin Ġa Ġfield Ġas Ġa Ġsequence Ġand Ġcomp utes Ġthe Ġfield Ġrepresentation Ġas Ġthe Ġfinal Ġrepresentation Ġof Ġthis Ġsequence Ġ( i . e ., Ġthe Ġrepresentation Ġof Ġthe Ġlast Ġtime - step ). ĠWe Ġfound Ġthat Ġbid irection al ĠGR U Ġis Ġa Ġbetter Ġchoice Ġfor ĠIN LINE FORM 6 Ġand Ġconc aten ating Ġthe Ġembed ding Ġof Ġthe Ġfield Ġname Ġwith Ġthis ĠGR U Ġrepresentation Ġworks Ġbest . ĠFurther , Ġusing Ġa Ġbid irection al ĠGR U Ġcell Ġto Ġtake Ġcontextual Ġinformation Ġfrom Ġneighboring Ġfields Ġalso Ġhelps Ġ( these Ġare Ġthe Ġorange Ġcolored Ġcells Ġin Ġthe Ġtop - left Ġblock Ġin ĠFigure ĠFIG REF 3 Ġwith Ġmacro Ġattention ). ĠGiven Ġthese Ġrepresentations ĠIN LINE FORM 7 Ġfor Ġall Ġthe ĠIN LINE FORM 8 Ġfields Ġwe Ġcompute Ġan Ġattention Ġover Ġthe Ġfields Ġ( mac ro Ġlevel ). ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġstate Ġof Ġthe Ġdec oder Ġat Ġtime Ġstep ĠIN LINE FORM 1 Ġ. ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġare Ġparameters , ĠIN LINE FORM 4 Ġis Ġthe Ġtotal Ġnumber Ġof Ġfields Ġin Ġthe Ġinput , ĠIN LINE FORM 5 Ġis Ġthe Ġmacro Ġ( field Ġlevel ) Ġcontext Ġvector Ġat Ġthe ĠIN LINE FORM 6 Ġ- th Ġtime Ġstep Ġof Ġthe Ġdec oder . </s> Micro ĠAttention : ĠLet ĠIN LINE FORM 0 Ġbe Ġthe Ġrepresentation Ġof Ġthe ĠIN LINE FORM 1 Ġ- th Ġvalue Ġin Ġa Ġgiven Ġfield . ĠThis Ġrepresentation Ġcould Ġagain Ġeither Ġbe Ġ( i ) Ġsimply Ġthe Ġembed ding Ġof Ġthis Ġvalue Ġ( ii ) Ġor Ġa Ġcontextual Ġrepresentation Ġcomputed Ġusing Ġa Ġfunction ĠIN LINE FORM 2 Ġwhich Ġalso Ġconsiders Ġthe Ġother Ġvalues Ġin Ġthe Ġfield . ĠFor Ġexample , Ġif ĠIN LINE FORM 3 Ġare Ġthe Ġvalues Ġin Ġa Ġfield Ġthen Ġthese Ġvalues Ġcan Ġbe Ġtreated Ġas Ġa Ġsequence Ġand Ġthe Ġrepresentation Ġof Ġthe ĠIN LINE FORM 4 Ġ- th Ġvalue Ġcan Ġbe Ġcomputed Ġusing Ġa Ġbid irection al ĠGR U Ġover Ġthis Ġsequence . ĠOnce Ġagain , Ġwe Ġfound Ġthat Ġusing Ġa Ġbi - GR U Ġworks Ġbetter Ġthen Ġsimply Ġusing Ġthe Ġembed ding Ġof Ġthe Ġvalue . ĠOnce Ġwe Ġhave Ġsuch Ġa Ġrepresentation Ġcomputed Ġfor Ġall Ġvalues Ġacross Ġall Ġthe Ġfields , Ġwe Ġcompute Ġthe Ġattention Ġover Ġthese Ġvalues Ġ( micro Ġlevel ) Ġas Ġshown Ġbelow Ġ: ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġstate Ġof Ġthe Ġdec oder Ġat Ġtime Ġstep ĠIN LINE FORM 1 Ġ. ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġare Ġparameters , ĠIN LINE FORM 4 Ġis Ġthe Ġtotal Ġnumber Ġof Ġvalues Ġacross Ġall Ġthe Ġfields . </s> F used ĠAttention : ĠInt uitive ly , Ġthe Ġattention Ġweights Ġassigned Ġto Ġa Ġfield Ġshould Ġhave Ġan Ġinfluence Ġon Ġall Ġthe Ġvalues Ġbelonging Ġto Ġthe Ġparticular Ġfield . ĠTo Ġensure Ġthis , Ġwe Ġre we igh Ġthe Ġmicro Ġlevel Ġattention Ġweights Ġbased Ġon Ġthe Ġcorresponding Ġmacro Ġlevel Ġattention Ġweights . ĠIn Ġother Ġwords , Ġwe Ġfuse Ġthe Ġattention Ġweights Ġat Ġthe Ġtwo Ġlevels Ġas : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġfield Ġcorresponding Ġto Ġthe ĠIN LINE FORM 1 Ġ- th Ġvalue , ĠIN LINE FORM 2 Ġis Ġthe Ġmacro Ġlevel Ġcontext Ġvector . </s> G ated ĠOrth og onal ization Ġfor ĠModel ing ĠStay - On Ġand ĠNever ĠLook ĠBack Ġbehaviour </s> We Ġnow Ġdescribe Ġa Ġseries Ġof Ġchoices Ġmade Ġto Ġmodel Ġstay - on Ġand Ġnever Ġlook Ġback Ġbehavior . ĠWe Ġfirst Ġbegin Ġwith Ġthe Ġstay - on Ġproperty Ġwhich Ġessentially Ġimplies Ġthat Ġif Ġwe Ġhave Ġpaid Ġattention Ġto Ġthe Ġfield ĠIN LINE FORM 0 Ġat Ġtim est ep ĠIN LINE FORM 1 Ġthen Ġwe Ġare Ġlikely Ġto Ġpay Ġattention Ġto Ġthe Ġsame Ġfield Ġfor Ġa Ġfew Ġmore Ġtime Ġsteps . ĠFor Ġexample , Ġif Ġwe Ġare Ġfocusing Ġon Ġthe Ġoccupation Ġfield Ġat Ġthis Ġtim est ep Ġthen Ġwe Ġare Ġlikely Ġto Ġfocus Ġon Ġit Ġfor Ġthe Ġnext Ġfew Ġtim est eps Ġtill Ġall Ġrelevant Ġvalues Ġin Ġthis Ġfield Ġhave Ġbeen Ġincluded Ġin Ġthe Ġgenerated Ġdescription . ĠIn Ġother Ġwords , Ġwe Ġwant Ġto Ġremember Ġthe Ġfield Ġcontext Ġvector ĠIN LINE FORM 2 Ġfor Ġa Ġfew Ġtim est eps . ĠOne Ġway Ġof Ġensuring Ġthis Ġis Ġto Ġuse Ġa Ġremember Ġ( or Ġforget ) Ġgate Ġas Ġgiven Ġbelow Ġwhich Ġremembers Ġthe Ġprevious Ġcontext Ġvector Ġwhen Ġrequired Ġand Ġforg ets Ġit Ġwhen Ġit Ġis Ġtime Ġto Ġmove Ġon Ġfrom Ġthat Ġfield . ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġare Ġparameters Ġto Ġbe Ġlearned . ĠThe Ġjob Ġof Ġthe Ġforget Ġgate Ġis Ġto Ġensure Ġthat ĠIN LINE FORM 1 Ġis Ġsimilar Ġto ĠIN LINE FORM 2 Ġwhen Ġrequired Ġ( i . e ., Ġby Ġlearning ĠIN LINE FORM 3 Ġwhen Ġwe Ġwant Ġto Ġcontinue Ġfocusing Ġon Ġthe Ġsame Ġfield ) Ġand Ġdifferent Ġwhen Ġit Ġis Ġtime Ġto Ġmove Ġon Ġ( by Ġlearning Ġthat ĠIN LINE FORM 4 Ġ). </s> Next , Ġthe Ġnever Ġlook Ġback Ġproperty Ġimplies Ġthat Ġonce Ġwe Ġhave Ġmoved Ġaway Ġfrom Ġa Ġfield Ġwe Ġare Ġunlikely Ġto Ġpay Ġattention Ġto Ġit Ġagain . ĠFor Ġexample , Ġonce Ġwe Ġhave Ġrendered Ġall Ġthe Ġoccupations Ġin Ġthe Ġgenerated Ġdescription Ġthere Ġis Ġno Ġneed Ġto Ġreturn Ġback Ġto Ġthe Ġoccupation Ġfield . ĠIn Ġother Ġwords , Ġonce Ġwe Ġhave Ġmoved Ġon Ġ( ĠIN LINE FORM 0 Ġ), Ġwe Ġwant Ġthe Ġsuccessive Ġfield Ġcontext Ġvectors ĠIN LINE FORM 1 Ġto Ġbe Ġvery Ġdifferent Ġfrom Ġthe Ġprevious Ġfield Ġvectors ĠIN LINE FORM 2 Ġ. ĠOne Ġway Ġof Ġensuring Ġthis Ġis Ġto Ġorth og onal ize Ġsuccessive Ġfield Ġvectors Ġusing ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġdot Ġproduct Ġbetween Ġvectors ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġ. ĠThe Ġabove Ġequation Ġessentially Ġsubtract s Ġthe Ġcomponent Ġof ĠIN LINE FORM 3 Ġalong ĠIN LINE FORM 4 Ġ. ĠIN LINE FORM 5 Ġis Ġa Ġlearned Ġparameter Ġwhich Ġcontrols Ġthe Ġdegree Ġof Ġorth og onal ization Ġthereby Ġallowing Ġa Ġsoft Ġorth og onal ization Ġ( i . e ., Ġthe Ġentire Ġcomponent Ġalong ĠIN LINE FORM 6 Ġis Ġnot Ġsubt racted Ġbut Ġonly Ġa Ġfraction Ġof Ġit ). ĠThe Ġabove Ġequation Ġonly Ġensures Ġthat ĠIN LINE FORM 7 Ġis Ġsoft - orth og onal Ġto ĠIN LINE FORM 8 Ġ. ĠAltern ately , Ġwe Ġcould Ġpass Ġthe Ġsequence Ġof Ġcontext Ġvectors , ĠIN LINE FORM 9 Ġgenerated Ġso Ġfar Ġthrough Ġa ĠGR U Ġcell . ĠThe Ġstate Ġof Ġthis ĠGR U Ġcell Ġat Ġeach Ġtime Ġstep Ġwould Ġthus Ġbe Ġaware Ġof Ġthe Ġhistory Ġof Ġthe Ġfield Ġvectors Ġtill Ġthat Ġtim est ep . ĠNow Ġinstead Ġof Ġorth og onal izing ĠIN LINE FORM 10 Ġto ĠIN LINE FORM 11 Ġwe Ġcould Ġorth og onal ize ĠIN LINE FORM 12 Ġto Ġthe Ġhidden Ġstate Ġof Ġthis ĠGR U Ġat Ġtime - step ĠIN LINE FORM 13 Ġ. ĠIn Ġpractice , Ġwe Ġfound Ġthis Ġto Ġwork Ġbetter Ġas Ġit Ġaccounts Ġfor Ġall Ġthe Ġfield Ġvectors Ġin Ġthe Ġhistory Ġinstead Ġof Ġonly Ġthe Ġprevious Ġfield Ġvector . </s> In Ġsummary , ĠEqu ation Ġprovides Ġa Ġmechanism Ġfor Ġremembering Ġthe Ġcurrent Ġfield Ġvector Ġwhen Ġappropriate Ġ( thus Ġcapturing Ġstay - on Ġbehavior ) Ġusing Ġa Ġremember Ġgate . ĠOn Ġthe Ġother Ġhand , ĠEqu ation ĠEQ REF 10 Ġexplicitly Ġensures Ġthat Ġthe Ġfield Ġvector Ġis Ġvery Ġdifferent Ġ( soft - orth og onal ) Ġfrom Ġthe Ġprevious Ġfield Ġvectors Ġonce Ġit Ġis Ġtime Ġto Ġmove Ġon Ġ( thus Ġcapturing Ġnever Ġlook Ġback Ġbehavior ). ĠThe Ġvalue Ġof ĠIN LINE FORM 0 Ġcomputed Ġin ĠEqu ation ĠEQ REF 10 Ġis Ġthen Ġused Ġin ĠEqu ation Ġ. ĠThe ĠIN LINE FORM 1 Ġ( mac ro ) Ġthus Ġobtained Ġis Ġthen Ġconc aten ated Ġwith ĠIN LINE FORM 2 Ġ( micro ) Ġand Ġfed Ġto Ġthe Ġdec oder Ġ( see ĠFig . ĠFIG REF 3 Ġ) </s> Exper imental Ġsetup </s> We Ġnow Ġdescribe Ġour Ġexperimental Ġsetup : </s> Dat as ets </s> We Ġuse Ġthe ĠWiki Bio Ġdataset Ġintroduced Ġby Ġle b ret 2016 ne ural . ĠIt Ġconsists Ġof ĠIN LINE FORM 0 Ġbiography Ġarticles Ġfrom ĠEnglish ĠWikipedia . ĠA Ġbiography Ġarticle Ġcorresponds Ġto Ġa Ġperson Ġ( sports man , Ġpolitician , Ġhistorical Ġfigure , Ġactor , Ġetc .). ĠEach ĠWikipedia Ġarticle Ġhas Ġan Ġaccompanying Ġinf ob ox Ġwhich Ġserves Ġas Ġthe Ġstructured Ġinput Ġand Ġthe Ġtask Ġis Ġto Ġgenerate Ġthe Ġfirst Ġsentence Ġof Ġthe Ġarticle Ġ( which Ġtypically Ġis Ġa Ġone - line Ġdescription Ġof Ġthe Ġperson ). ĠWe Ġused Ġthe Ġsame Ġtrain , Ġvalid Ġand Ġtest Ġsets Ġwhich Ġwere Ġmade Ġpublicly Ġavailable Ġby Ġle b ret 2016 ne ural . </s> We Ġalso Ġintroduce Ġtwo Ġnew Ġbiography Ġdatasets , Ġone Ġin ĠFrench Ġand Ġone Ġin ĠGerman . ĠThese Ġdatasets Ġwere Ġcreated Ġand Ġpre - process ed Ġusing Ġthe Ġsame Ġprocedure Ġas Ġoutlined Ġin Ġle b ret 2016 ne ural . ĠSpecifically , Ġwe Ġextracted Ġthe Ġinf ob ox es Ġand Ġthe Ġfirst Ġsentence Ġfrom Ġthe Ġcorresponding ĠWikipedia Ġarticle . ĠAs Ġwith Ġthe ĠEnglish Ġdataset , Ġwe Ġsplit Ġthe ĠFrench Ġand ĠGerman Ġdatasets Ġrandomly Ġinto Ġtrain Ġ( 80 %), Ġtest Ġ( 10 %) Ġand Ġvalid Ġ( 10 %). ĠThe ĠFrench Ġand ĠGerman Ġdatasets Ġextracted Ġby Ġus Ġhas Ġbeen Ġmade Ġpublicly Ġavailable . ĠThe Ġnumber Ġof Ġexamples Ġwas Ġ170 K Ġand Ġ50 K Ġand Ġthe Ġvocabulary Ġsize Ġwas Ġ297 K Ġand Ġ143 K Ġfor ĠFrench Ġand ĠGerman Ġrespectively . ĠAlthough Ġin Ġthis Ġwork Ġwe Ġfocus Ġonly Ġon Ġgenerating Ġdescriptions Ġin Ġone Ġlanguage , Ġwe Ġhope Ġthat Ġthis Ġdataset Ġwill Ġalso Ġbe Ġuseful Ġfor Ġdeveloping Ġmodels Ġwhich Ġjointly Ġlearn Ġto Ġgenerate Ġdescriptions Ġfrom Ġstructured Ġdata Ġin Ġmultiple Ġlanguages . </s> Mod els Ġcompared </s> We Ġcompare Ġwith Ġthe Ġfollowing Ġmodels : </s> 1 . ĠB IB REF 0 Ġ: ĠThis Ġis Ġa Ġconditional Ġlanguage Ġmodel Ġwhich Ġuses Ġa Ġfeed - forward Ġneural Ġnetwork Ġto Ġpredict Ġthe Ġnext Ġword Ġin Ġthe Ġdescription Ġconditioned Ġon Ġlocal Ġcharacteristics Ġ( i . e ., Ġwords Ġwithin Ġa Ġfield ) Ġand Ġglobal Ġcharacteristics Ġ( i . e ., Ġoverall Ġstructure Ġof Ġthe Ġinf ob ox ). </s> 2 . ĠB IB REF 9 Ġ: ĠThis Ġmodel Ġwas Ġproposed Ġin Ġthe Ġcontext Ġof Ġthe ĠWeather Gov Ġand ĠRobo C up Ġdatasets Ġwhich Ġhave Ġa Ġmuch Ġsmaller Ġvocabulary . ĠThey Ġuse Ġan Ġimproved Ġattention Ġmodel Ġwith Ġadditional Ġregular izer Ġterms Ġwhich Ġinfluence Ġthe Ġweights Ġassigned Ġto Ġthe Ġfields . </s> 3 . ĠBasic ĠSe q 2 Se q : ĠThis Ġis Ġthe Ġvanilla Ġencode - att end - dec ode Ġmodel ĠB IB REF 1 Ġ. ĠFurther , Ġto Ġdeal Ġwith Ġthe Ġlarge Ġvocabulary Ġ( ĠIN LINE FORM 0 Ġ400 K Ġwords ) Ġwe Ġuse Ġa Ġcopying Ġmechanism Ġas Ġa Ġpost - processing Ġstep . ĠSpecifically , Ġwe Ġidentify Ġthe Ġtime Ġsteps Ġat Ġwhich Ġthe Ġdec oder Ġproduces Ġunknown Ġwords Ġ( den oted Ġby Ġthe Ġspecial Ġsymbol ĠUN K ). ĠFor Ġeach Ġsuch Ġtime Ġstep , Ġwe Ġlook Ġat Ġthe Ġattention Ġweights Ġon Ġthe Ġinput Ġwords Ġand Ġreplace Ġthe ĠUN K Ġword Ġby Ġthat Ġinput Ġword Ġwhich Ġhas Ġreceived Ġmaximum Ġattention Ġat Ġthis Ġtim est ep . ĠThis Ġprocess Ġis Ġsimilar Ġto Ġthe Ġone Ġdescribed Ġin ĠB IB REF 30 Ġ. ĠEven Ġle b ret 2016 ne ural Ġhave Ġa Ġcopying Ġmechanism Ġtightly Ġintegrated Ġwith Ġtheir Ġmodel . </s> Hyper param eter Ġtuning </s> We Ġtuned Ġthe Ġhyper param eters Ġof Ġall Ġthe Ġmodels Ġusing Ġa Ġvalidation Ġset . ĠAs Ġmentioned Ġearlier , Ġwe Ġused Ġa Ġbid irection al ĠGR U Ġcell Ġas Ġthe Ġfunction ĠIN LINE FORM 0 Ġfor Ġcomputing Ġthe Ġrepresentation Ġof Ġthe Ġfields Ġand Ġthe Ġvalues Ġ( see ĠSection ĠSEC REF 4 Ġ). ĠFor Ġall Ġthe Ġmodels , Ġwe Ġexperimented Ġwith ĠGR U Ġstate Ġsizes Ġof Ġ128 , Ġ256 Ġand Ġ512 . ĠThe Ġtotal Ġnumber Ġof Ġunique Ġwords Ġin Ġthe Ġcorpus Ġis Ġaround Ġ400 K Ġ( this Ġincludes Ġthe Ġwords Ġin Ġthe Ġinf ob ox Ġand Ġthe Ġdescriptions ). ĠOf Ġthese , Ġwe Ġretained Ġonly Ġthe Ġtop Ġ20 K Ġwords Ġin Ġour Ġvocabulary Ġ( same Ġas ĠB IB REF 0 Ġ). ĠWe Ġinitialized Ġthe Ġembed d ings Ġof Ġthese Ġwords Ġwith Ġ300 Ġdimensional ĠGl ove Ġembed d ings ĠB IB REF 31 Ġ. ĠWe Ġused ĠAdam ĠB IB REF 32 Ġwith Ġa Ġlearning Ġrate Ġof ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġ. ĠWe Ġtrained Ġthe Ġmodel Ġfor Ġa Ġmaximum Ġof Ġ20 Ġepoch s Ġand Ġused Ġearly Ġstopping Ġwith Ġthe Ġpatience Ġset Ġto Ġ5 Ġepoch s . </s> Results Ġand ĠDisc ussions </s> We Ġnow Ġdiscuss Ġthe Ġresults Ġof Ġour Ġexperiments . </s> Compar ison Ġof Ġdifferent Ġmodels </s> Following Ġle b ret 2016 ne ural , Ġwe Ġused ĠB LE U - 4 , ĠN IST - 4 Ġand ĠR OU GE - 4 Ġas Ġthe Ġevaluation Ġmetrics . ĠWe Ġfirst Ġmake Ġa Ġfew Ġobservations Ġbased Ġon Ġthe Ġresults Ġon Ġthe ĠEnglish Ġdataset Ġ( Table ĠT AB REF 15 Ġ). ĠThe Ġbasic Ġseq 2 seq Ġmodel , Ġas Ġwell Ġas Ġthe Ġmodel Ġproposed Ġby Ġweather 16 , Ġperform Ġbetter Ġthan Ġthe Ġmodel Ġproposed Ġby Ġle b ret 2016 ne ural . ĠOur Ġfinal Ġmodel Ġwith Ġb if ocal Ġattention Ġand Ġg ated Ġorth og onal ization Ġgives Ġthe Ġbest Ġperformance Ġand Ġdoes Ġ10 % Ġ( relative ) Ġbetter Ġthan Ġthe Ġclosest Ġbaseline Ġ( basic Ġseq 2 seq ) Ġand Ġ21 % Ġ( relative ) Ġbetter Ġthan Ġthe Ġcurrent Ġstate Ġof Ġthe Ġart Ġmethod ĠB IB REF 0 Ġ. ĠIn ĠTable ĠT AB REF 16 Ġ, Ġwe Ġshow Ġsome Ġqualitative Ġexamples Ġof Ġthe Ġoutput Ġgenerated Ġby Ġdifferent Ġmodels . </s> Human ĠEval uations </s> To Ġmake Ġa Ġqualitative Ġassessment Ġof Ġthe Ġgenerated Ġsentences , Ġwe Ġconducted Ġa Ġhuman Ġstudy Ġon Ġa Ġsample Ġof Ġ500 ĠInf ob ox es Ġwhich Ġwere Ġsampled Ġfrom ĠEnglish Ġdataset . ĠThe Ġannot ators Ġfor Ġthis Ġtask Ġwere Ġundergraduate Ġand Ġgraduate Ġstudents . ĠFor Ġeach Ġof Ġthese Ġinf ob ox es , Ġwe Ġgenerated Ġsumm aries Ġusing Ġthe Ġbasic Ġseq 2 seq Ġmodel Ġand Ġour Ġfinal Ġmodel Ġwith Ġb if ocal Ġattention Ġand Ġg ated Ġorth og onal ization . ĠFor Ġeach Ġdescription Ġand Ġfor Ġeach Ġmodel , Ġwe Ġasked Ġthree Ġannot ators Ġto Ġrank Ġthe Ġoutput Ġof Ġthe Ġsystems Ġbased Ġon Ġi ) Ġadequ acy Ġ( i . e . Ġdoes Ġit Ġcapture Ġrelevant Ġinformation Ġfrom Ġthe Ġinf ob ox ), Ġ( ii ) Ġflu ency Ġ( i . e . Ġgrammar ) Ġand Ġ( iii ) Ġrelative Ġpreference Ġ( i . e ., Ġwhich Ġof Ġthe Ġtwo Ġoutputs Ġwould Ġbe Ġpreferred ). ĠOverall Ġthe Ġaverage Ġflu ency / adequ acy Ġ( on Ġa Ġscale Ġof Ġ5 ) Ġfor Ġbasic Ġseq 2 seq Ġmodel Ġwas ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġfor Ġour Ġmodel Ġrespectively . </s> The Ġresults Ġfrom ĠTable ĠT AB REF 17 Ġsuggest Ġthat Ġin Ġgeneral Ġg ated Ġorth og onal ization Ġmodel Ġperforms Ġbetter Ġthan Ġthe Ġbasic Ġseq 2 seq Ġmodel . ĠAdditionally , Ġannot ators Ġwere Ġasked Ġto Ġverify Ġif Ġthe Ġgenerated Ġsumm aries Ġlook Ġnatural Ġ( i . e , Ġas Ġif Ġthey Ġwere Ġgenerated Ġby Ġhumans ). ĠIn Ġ423 Ġout Ġof Ġ500 Ġcases , Ġthe Ġannot ators Ġsaid ĠâĢ ľ Yes âĢ Ŀ Ġsuggesting Ġthat Ġg ated Ġorth og onal ization Ġmodel Ġindeed Ġproduces Ġgood Ġdescriptions . </s> Performance Ġon Ġdifferent Ġlanguages </s> The Ġresults Ġon Ġthe ĠFrench Ġand ĠGerman Ġdatasets Ġare Ġsummarized Ġin ĠTables ĠT AB REF 20 Ġand ĠT AB REF 20 Ġrespectively . ĠNote Ġthat Ġthe Ġcode Ġof ĠB IB REF 0 Ġis Ġnot Ġpublicly Ġavailable , Ġhence Ġwe Ġcould Ġnot Ġreport Ġnumbers Ġfor ĠFrench Ġand ĠGerman Ġusing Ġtheir Ġmodel . ĠWe Ġobserve Ġthat Ġour Ġfinal Ġmodel Ġgives Ġthe Ġbest Ġperformance Ġ- Ġthough Ġthe Ġb if ocal Ġattention Ġmodel Ġperforms Ġpoorly Ġas Ġcompared Ġto Ġthe Ġbasic Ġseq 2 seq Ġmodel Ġon ĠFrench . ĠHowever , Ġthe Ġoverall Ġperformance Ġfor ĠFrench Ġand ĠGerman Ġare Ġmuch Ġsmaller Ġthan Ġthose Ġfor ĠEnglish . ĠThere Ġcould Ġbe Ġmultiple Ġreasons Ġfor Ġthis . ĠFirst , Ġthe Ġamount Ġof Ġtraining Ġdata Ġin Ġthese Ġtwo Ġlanguages Ġis Ġsmaller Ġthan Ġthat Ġin ĠEnglish . ĠSpecifically , Ġthe Ġamount Ġof Ġtraining Ġdata Ġavailable Ġin ĠFrench Ġ( German ) Ġis Ġonly ĠIN LINE FORM 0 Ġ( ĠIN LINE FORM 1 Ġ) % Ġof Ġthat Ġavailable Ġfor ĠEnglish . ĠSecond , Ġon Ġaverage Ġthe Ġdescriptions Ġin ĠFrench Ġand ĠGerman Ġare Ġlonger Ġthan Ġthat Ġin ĠEnglish Ġ( EN : ĠIN LINE FORM 2 Ġwords , ĠFR : ĠIN LINE FORM 3 Ġwords Ġand ĠDE : ĠIN LINE FORM 4 Ġwords ). ĠFinally , Ġa Ġmanual Ġinspection Ġacross Ġthe Ġthree Ġlanguages Ġsuggests Ġthat Ġthe ĠEnglish Ġdescriptions Ġhave Ġa Ġmore Ġconsistent Ġstructure Ġthan Ġthe ĠFrench Ġdescriptions . ĠFor Ġexample , Ġmost ĠEnglish Ġdescriptions Ġstart Ġwith Ġname Ġfollowed Ġby Ġdate Ġof Ġbirth Ġbut Ġthis Ġis Ġnot Ġthe Ġcase Ġin ĠFrench . ĠHowever , Ġthis Ġis Ġonly Ġa Ġqualitative Ġobservation Ġand Ġit Ġis Ġhard Ġto Ġquantify Ġthis Ġcharacteristic Ġof Ġthe ĠFrench Ġand ĠGerman Ġdatasets . </s> Visual izing ĠAttention ĠWe ights </s> If Ġthe Ġproposed Ġmodel Ġindeed Ġworks Ġwell Ġthen Ġwe Ġshould Ġsee Ġattention Ġweights Ġthat Ġare Ġconsistent Ġwith Ġthe Ġstay Ġon Ġand Ġnever Ġlook Ġback Ġbehavior . ĠTo Ġverify Ġthis , Ġwe Ġplotted Ġthe Ġattention Ġweights Ġin Ġcases Ġwhere Ġthe Ġmodel Ġwith Ġg ated Ġorth og onal ization Ġdoes Ġbetter Ġthan Ġthe Ġmodel Ġwith Ġonly Ġb if ocal Ġattention . ĠFigure ĠFIG REF 21 Ġshows Ġthe Ġattention Ġweights Ġcorresponding Ġto Ġinf ob ox Ġin ĠFigure ĠFIG REF 25 Ġ. ĠNotice Ġthat Ġthe Ġmodel Ġwithout Ġg ated Ġorth og onal ization Ġhas Ġattention Ġon Ġboth Ġname Ġfield Ġand Ġarticle Ġtitle Ġwhile Ġrendering Ġthe Ġname . ĠThe Ġmodel Ġwith Ġg ated Ġorth og onal ization , Ġon Ġthe Ġother Ġhand , Ġstays Ġon Ġthe Ġname Ġfield Ġfor Ġas Ġlong Ġas Ġit Ġis Ġrequired Ġbut Ġthen Ġmoves Ġand Ġnever Ġreturns Ġto Ġit Ġ( as Ġexpected ). </s> Due Ġto Ġlack Ġof Ġspace , Ġwe Ġdo Ġnot Ġshow Ġsimilar Ġplots Ġfor ĠFrench Ġand ĠGerman Ġbut Ġwe Ġwould Ġlike Ġto Ġmention Ġthat , Ġin Ġgeneral , Ġthe Ġdifferences Ġbetween Ġthe Ġattention Ġweights Ġlearned Ġby Ġthe Ġmodel Ġwith Ġand Ġwithout Ġg ated Ġorth og onal ization Ġwere Ġmore Ġpronounced Ġfor Ġthe ĠFrench / German Ġdataset Ġthan Ġthe ĠEnglish Ġdataset . ĠThis Ġis Ġin Ġagreement Ġwith Ġthe Ġresults Ġreported Ġin ĠTable ĠT AB REF 20 Ġand ĠT AB REF 20 Ġwhere Ġthe Ġimprovements Ġgiven Ġby Ġg ated Ġorth og onal ization Ġare Ġmore Ġfor ĠFrench / German Ġthan Ġfor ĠEnglish . </s> Out Ġof Ġdomain Ġresults </s> What Ġif Ġthe Ġmodel Ġsees Ġa Ġdifferent ĠIN LINE FORM 0 Ġof Ġperson Ġat Ġtest Ġtime ? ĠFor Ġexample , Ġwhat Ġif Ġthe Ġtraining Ġdata Ġdoes Ġnot Ġcontain Ġany Ġsports pers ons Ġbut Ġat Ġtest Ġtime Ġwe Ġencounter Ġthe Ġinf ob ox Ġof Ġa Ġsports person . ĠThis Ġis Ġthe Ġsame Ġas Ġseeing Ġout - of - domain Ġdata Ġat Ġtest Ġtime . ĠSuch Ġa Ġsituation Ġis Ġquite Ġexpected Ġin Ġthe Ġproducts Ġdomain Ġwhere Ġnew Ġproducts Ġwith Ġnew Ġfeatures Ġ( fields ) Ġget Ġfrequently Ġadded Ġto Ġthe Ġcatalog . ĠWe Ġwere Ġinterested Ġin Ġthree Ġquestions Ġhere . ĠFirst , Ġwe Ġwanted Ġto Ġsee Ġif Ġtesting Ġthe Ġmodel Ġon Ġout - of - domain Ġdata Ġindeed Ġleads Ġto Ġa Ġdrop Ġin Ġthe Ġperformance . ĠFor Ġthis , Ġwe Ġcompared Ġthe Ġperformance Ġof Ġour Ġbest Ġmodel Ġin Ġtwo Ġscenarios Ġ( i ) Ġtrained Ġon Ġdata Ġfrom Ġall Ġdomains Ġ( including Ġthe Ġtarget Ġdomain ) Ġand Ġtested Ġon Ġthe Ġtarget Ġdomain Ġ( sports , Ġarts ) Ġand Ġ( ii ) Ġtrained Ġon Ġdata Ġfrom Ġall Ġdomains Ġexcept Ġthe Ġtarget Ġdomain Ġand Ġtested Ġon Ġthe Ġtarget Ġdomain . ĠComp aring Ġrows Ġ1 Ġand Ġ2 Ġof ĠTable ĠT AB REF 32 Ġwe Ġobserved Ġa Ġsignificant Ġdrop Ġin Ġthe Ġperformance . ĠNote Ġthat Ġthe Ġnumbers Ġfor Ġsports Ġdomain Ġin Ġrow Ġ1 Ġare Ġmuch Ġbetter Ġthan Ġthe ĠArts Ġdomain Ġbecause Ġroughly Ġ40 % Ġof Ġthe ĠWiki Bio Ġtraining Ġdata Ġcontains Ġsports pers ons . </s> Next , Ġwe Ġwanted Ġto Ġsee Ġif Ġwe Ġcan Ġuse Ġa Ġsmall Ġamount Ġof Ġdata Ġfrom Ġthe Ġtarget Ġdomain Ġto Ġfine Ġtune Ġa Ġmodel Ġtrained Ġon Ġthe Ġout Ġof Ġdomain Ġdata . ĠWe Ġobserve Ġthat Ġeven Ġwith Ġvery Ġsmall Ġamounts Ġof Ġtarget Ġdomain Ġdata Ġthe Ġperformance Ġstarts Ġimproving Ġsignificantly Ġ( see Ġrows Ġ3 Ġand Ġ4 Ġof ĠTable ĠT AB REF 32 Ġ). ĠNote Ġthat Ġif Ġwe Ġtrain Ġa Ġmodel Ġfrom Ġscratch Ġwith Ġonly Ġlimited Ġdata Ġfrom Ġthe Ġtarget Ġdomain Ġinstead Ġof Ġfine - tun ing Ġa Ġmodel Ġtrained Ġon Ġa Ġdifferent Ġsource Ġdomain Ġthen Ġthe Ġperformance Ġis Ġvery Ġpoor . ĠIn Ġparticular , Ġtraining Ġa Ġmodel Ġfrom Ġscratch Ġwith Ġ10 K Ġtraining Ġinstances Ġwe Ġget Ġa ĠB LE U Ġscore Ġof ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġfor Ġarts Ġand Ġsports Ġrespectively . ĠFinally , Ġeven Ġthough Ġthe Ġactual Ġwords Ġused Ġfor Ġdescribing Ġa Ġsports person Ġ( football er , Ġcr ick eter , Ġetc .) Ġwould Ġbe Ġvery Ġdifferent Ġfrom Ġthe Ġwords Ġused Ġto Ġdescribe Ġan Ġartist Ġ( actor , Ġmusician , Ġetc .) Ġthey Ġmight Ġshare Ġmany Ġfields Ġ( for Ġexample , Ġdate Ġof Ġbirth , Ġoccupation , Ġetc .). ĠAs Ġseen Ġin ĠFigure ĠFIG REF 28 Ġ( att ention Ġweights Ġcorresponding Ġto Ġthe Ġinf ob ox Ġin ĠFigure ĠFIG REF 27 Ġ), Ġthe Ġmodel Ġpredicts Ġthe Ġattention Ġweights Ġcorrectly Ġfor Ġcommon Ġfields Ġ( such Ġas Ġoccupation ) Ġbut Ġit Ġis Ġunable Ġto Ġuse Ġthe Ġright Ġvocabulary Ġto Ġdescribe Ġthe Ġoccupation Ġ( since Ġit Ġhas Ġnot Ġseen Ġsuch Ġwords Ġfrequently Ġin Ġthe Ġtraining Ġdata ). ĠHowever , Ġonce Ġwe Ġfine Ġtune Ġthe Ġmodel Ġwith Ġlimited Ġdata Ġfrom Ġthe Ġtarget Ġdomain Ġwe Ġsee Ġthat Ġit Ġpicks Ġup Ġthe Ġnew Ġvocabulary Ġand Ġproduces Ġa Ġcorrect Ġdescription Ġof Ġthe Ġoccupation . </s> Conclusion </s> We Ġpresent Ġa Ġmodel Ġfor Ġgenerating Ġnatural Ġlanguage Ġdescriptions Ġfrom Ġstructured Ġdata . ĠTo Ġaddress Ġspecific Ġcharacteristics Ġof Ġthe Ġproblem Ġwe Ġpropose Ġneural Ġcomponents Ġfor Ġfused Ġb if ocal Ġattention Ġand Ġg ated Ġorth og onal ization Ġto Ġaddress Ġstay Ġon Ġand Ġnever Ġlook Ġback Ġbehavior Ġwhile Ġdecoding . ĠOur Ġfinal Ġmodel Ġoutper forms Ġan Ġexisting Ġstate Ġof Ġthe Ġart Ġmodel Ġon Ġa Ġlarge Ġscale ĠWiki Bio Ġdataset Ġby Ġ21 %. ĠWe Ġalso Ġintroduce Ġdatasets Ġfor ĠFrench Ġand ĠGerman Ġand Ġdemonstrate Ġthat Ġour Ġmodel Ġgives Ġstate Ġof Ġthe Ġart Ġresults Ġon Ġthese Ġdatasets . ĠFinally , Ġwe Ġperform Ġexperiments Ġwith Ġan Ġout - of - domain Ġmodel Ġand Ġshow Ġthat Ġif Ġsuch Ġa Ġmodel Ġis Ġfine - tun ed Ġwith Ġsmall Ġamounts Ġof Ġin Ġdomain Ġdata Ġthen Ġit Ġcan Ġgive Ġan Ġimproved Ġperformance Ġon Ġthe Ġtarget Ġdomain . </s> Given Ġthe Ġmult ilingual Ġnature Ġof Ġthe Ġnew Ġdatasets , Ġas Ġfuture Ġwork , Ġwe Ġwould Ġlike Ġto Ġbuild Ġmodels Ġwhich Ġcan Ġjointly Ġlearn Ġto Ġgenerate Ġnatural Ġlanguage Ġdescriptions Ġfrom Ġstructured Ġdata Ġin Ġmultiple Ġlanguages . ĠOne Ġidea Ġis Ġto Ġreplace Ġthe Ġconcepts Ġin Ġthe Ġinput Ġinf ob ox Ġby ĠWik id ata Ġconcept Ġ ids Ġwhich Ġare Ġlanguage Ġag nostic . ĠA Ġlarge Ġamount Ġof Ġinput Ġvocabulary Ġcould Ġthus Ġbe Ġshared Ġacross Ġlanguages Ġthereby Ġfacilitating Ġjoint Ġlearning . </s> Acknowled gements </s> We Ġthank ĠGoogle Ġfor Ġsupporting ĠPre ks ha ĠN ema Ġthrough Ġtheir ĠGoogle ĠIndia ĠPh . D . ĠFellowship Ġprogram . ĠWe Ġalso Ġthank ĠMicrosoft ĠResearch ĠIndia Ġfor Ġsupporting ĠSh rey as ĠShe tty Ġthrough Ġtheir Ġgenerous Ġtravel Ġgrant Ġfor Ġattending Ġthe Ġconference .
Introduction Rendering natural language descriptions from structured data is required in a wide variety of commercial applications such as generating descriptions of products, hotels, furniture, etc., from a corresponding table of facts about the entity. Such a table typically contains {field, value} pairs where the field is a property of the entity (e.g., color) and the value is a set of possible assignments to this property (e.g., color = red). Another example of this is the recently introduced task of generating one line biography descriptions from a given Wikipedia infobox BIBREF0 . The Wikipedia infobox serves as a table of facts about a person and the first sentence from the corresponding article serves as a one line description of the person. Figure FIGREF2 illustrates an example input infobox which contains fields such as Born, Residence, Nationality, Fields, Institutions and Alma Mater. Each field further contains some words (e.g., particle physics, many-body theory, etc.). The corresponding description is coherent with the information contained in the infobox. Note that the number of fields in the infobox and the ordering of the fields within the infobox varies from person to person. Given the large size (700K examples) and heterogeneous nature of the dataset which contains biographies of people from different backgrounds (sports, politics, arts, etc.), it is hard to come up with simple rule-based templates for generating natural language descriptions from infoboxes, thereby making a case for data-driven models. Based on the recent success of data-driven neural models for various other NLG tasks BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 , one simple choice is to treat the infobox as a sequence of {field, value} pairs and use a standard seq2seq model for this task. However, such a model is too generic and does not exploit the specific characteristics of this task as explained below. First, note that while generating such descriptions from structured data, a human keeps track of information at two levels. Specifically, at a macro level, she would first decide which field to mention next and then at a micro level decide which of the values in the field needs to be mentioned next. For example, she first decides that at the current step, the field occupation needs attention and then decides which is the next appropriate occupation to attend to from the set of occupations (actor, director, producer, etc.). To enable this, we use a bifocal attention mechanism which computes an attention over fields at a macro level and over values at a micro level. We then fuse these attention weights such that the attention weight for a field also influences the attention over the values within it. Finally, we feed a fused context vector to the decoder which contains both field level and word level information. Note that such two-level attention mechanisms BIBREF6 , BIBREF7 , BIBREF8 have been used in the context of unstructured data (as opposed to structured data in our case), where at a macro level one needs to pay attention to sentences and at a micro level to words in the sentences. Next, we observe that while rendering the output, once the model pays attention to a field (say, occupation) it needs to stay on this field for a few timesteps (till all the occupations are produced in the output). We refer to this as the stay on behavior. Further, we note that once the tokens of a field are referred to, they are usually not referred to later. For example, once all the occupations have been listed in the output we will never visit the occupation field again because there is nothing left to say about it. We refer to this as the never look back behavior. To model the stay on behaviour, we introduce a forget (or remember) gate which acts as a signal to decide when to forget the current field (or equivalently to decide till when to remember the current field). To model the never look back behaviour we introduce a gated orthogonalization mechanism which ensures that once a field is forgotten, subsequent field context vectors fed to the decoder are orthogonal to (or different from) the previous field context vectors. We experiment with the WikiBio dataset BIBREF0 which contains around 700K {infobox, description} pairs and has a vocabulary of around 400K words. We show that the proposed model gives a relative improvement of 21% and 20% as compared to current state of the art models BIBREF0 , BIBREF9 on this dataset. The proposed model also gives a relative improvement of 10% as compared to the basic seq2seq model. Further, we introduce new datasets for French and German on the same lines as the English WikiBio dataset. Even on these two datasets, our model outperforms the state of the art methods mentioned above. Related work Natural Language Generation has always been of interest to the research community and has received a lot of attention in the past. The approaches for NLG range from (i) rule based approaches (e.g., BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14 ) (ii) modular statistical approaches which divide the process into three phases (planning, selection and surface realization) and use data driven approaches for one or more of these phases BIBREF15 , BIBREF16 , BIBREF17 , BIBREF18 , BIBREF19 (iii) hybrid approaches which rely on a combination of handcrafted rules and corpus statistics BIBREF20 , BIBREF21 , BIBREF22 and (iv) the more recent neural network based models BIBREF1 . Neural models for NLG have been proposed in the context of various tasks such as machine translation BIBREF1 , document summarization BIBREF2 , BIBREF4 , paraphrase generation BIBREF23 , image captioning BIBREF24 , video summarization BIBREF25 , query based document summarization BIBREF5 and so on. Most of these models are data hungry and are trained on large amounts of data. On the other hand, NLG from structured data has largely been studied in the context of small datasets such as WeatherGov BIBREF26 , RoboCup BIBREF27 , NFL Recaps BIBREF15 , Prodigy-Meteo BIBREF28 and TUNA Challenge BIBREF29 . Recently weather16 proposed RNN/LSTM based neural encoder-decoder models with attention for WeatherGov and RoboCup datasets. Unlike the datasets mentioned above, the biography dataset introduced by lebret2016neural is larger (700K {table, descriptions} pairs) and has a much larger vocabulary (400K words as opposed to around 350 or fewer words in the above datasets). Further, unlike the feed-forward neural network based model proposed by BIBREF0 we use a sequence to sequence model and introduce components to address the peculiar characteristics of the task. Specifically, we introduce neural components to address the need for attention at two levels and to address the stay on and never look back behaviour required by the decoder. KiddonZC16 have explored the use of checklists to track previously visited ingredients while generating recipes from ingredients. Note that two-level attention mechanisms have also been used in the context of summarization BIBREF6 , document classification BIBREF7 , dialog systems BIBREF8 , etc. However, these works deal with unstructured data (sentences at the higher level and words at a lower level) as opposed to structured data in our case. Proposed model As input we are given an infobox INLINEFORM0 , which is a set of pairs INLINEFORM1 where INLINEFORM2 corresponds to field names and INLINEFORM3 is the sequence of corresponding values and INLINEFORM4 is the total number of fields in INLINEFORM5 . For example, INLINEFORM6 could be one such pair in this set. Given such an input, the task is to generate a description INLINEFORM7 containing INLINEFORM8 words. A simple solution is to treat the infobox as a sequence of fields followed by the values corresponding to the field in the order of their appearance in the infobox. For example, the infobox could be flattened to produce the following input sequence (the words in bold are field names which act as delimiters) [Name] John Doe [Birth_Date] 19 March 1981 [Nationality] Indian ..... The problem can then be cast as a seq2seq generation problem and can be modeled using a standard neural architecture comprising of three components (i) an input encoder (using GRU/LSTM cells), (ii) an attention mechanism to attend to important values in the input sequence at each time step and (iii) a decoder to decode the output one word at a time (again, using GRU/LSTM cells). However, this standard model is too generic and does not exploit the specific characteristics of this task. We propose additional components, viz., (i) a fused bifocal attention mechanism which operates on fields (macro) and values (micro) and (ii) a gated orthogonalization mechanism to model stay on and never look back behavior. Fused Bifocal Attention Mechanism Intuitively, when a human writes a description from a table she keeps track of information at two levels. At the macro level, it is important to decide which is the appropriate field to attend to next and at a micro level (i.e., within a field) it is important to know which values to attend to next. To capture this behavior, we use a bifocal attention mechanism as described below. Macro Attention: Consider the INLINEFORM0 -th field INLINEFORM1 which has values INLINEFORM2 . Let INLINEFORM3 be the representation of this field in the infobox. This representation can either be (i) the word embedding of the field name or (ii) some function INLINEFORM4 of the values in the field or (iii) a concatenation of (i) and (ii). The function INLINEFORM5 could simply be the sum or average of the embeddings of the values in the field. Alternately, this function could be a GRU (or LSTM) which treats these values within a field as a sequence and computes the field representation as the final representation of this sequence (i.e., the representation of the last time-step). We found that bidirectional GRU is a better choice for INLINEFORM6 and concatenating the embedding of the field name with this GRU representation works best. Further, using a bidirectional GRU cell to take contextual information from neighboring fields also helps (these are the orange colored cells in the top-left block in Figure FIGREF3 with macro attention). Given these representations INLINEFORM7 for all the INLINEFORM8 fields we compute an attention over the fields (macro level). DISPLAYFORM0 where INLINEFORM0 is the state of the decoder at time step INLINEFORM1 . INLINEFORM2 and INLINEFORM3 are parameters, INLINEFORM4 is the total number of fields in the input, INLINEFORM5 is the macro (field level) context vector at the INLINEFORM6 -th time step of the decoder. Micro Attention: Let INLINEFORM0 be the representation of the INLINEFORM1 -th value in a given field. This representation could again either be (i) simply the embedding of this value (ii) or a contextual representation computed using a function INLINEFORM2 which also considers the other values in the field. For example, if INLINEFORM3 are the values in a field then these values can be treated as a sequence and the representation of the INLINEFORM4 -th value can be computed using a bidirectional GRU over this sequence. Once again, we found that using a bi-GRU works better then simply using the embedding of the value. Once we have such a representation computed for all values across all the fields, we compute the attention over these values (micro level) as shown below : DISPLAYFORM0 where INLINEFORM0 is the state of the decoder at time step INLINEFORM1 . INLINEFORM2 and INLINEFORM3 are parameters, INLINEFORM4 is the total number of values across all the fields. Fused Attention: Intuitively, the attention weights assigned to a field should have an influence on all the values belonging to the particular field. To ensure this, we reweigh the micro level attention weights based on the corresponding macro level attention weights. In other words, we fuse the attention weights at the two levels as: DISPLAYFORM0 where INLINEFORM0 is the field corresponding to the INLINEFORM1 -th value, INLINEFORM2 is the macro level context vector. Gated Orthogonalization for Modeling Stay-On and Never Look Back behaviour We now describe a series of choices made to model stay-on and never look back behavior. We first begin with the stay-on property which essentially implies that if we have paid attention to the field INLINEFORM0 at timestep INLINEFORM1 then we are likely to pay attention to the same field for a few more time steps. For example, if we are focusing on the occupation field at this timestep then we are likely to focus on it for the next few timesteps till all relevant values in this field have been included in the generated description. In other words, we want to remember the field context vector INLINEFORM2 for a few timesteps. One way of ensuring this is to use a remember (or forget) gate as given below which remembers the previous context vector when required and forgets it when it is time to move on from that field. DISPLAYFORM0 where INLINEFORM0 are parameters to be learned. The job of the forget gate is to ensure that INLINEFORM1 is similar to INLINEFORM2 when required (i.e., by learning INLINEFORM3 when we want to continue focusing on the same field) and different when it is time to move on (by learning that INLINEFORM4 ). Next, the never look back property implies that once we have moved away from a field we are unlikely to pay attention to it again. For example, once we have rendered all the occupations in the generated description there is no need to return back to the occupation field. In other words, once we have moved on ( INLINEFORM0 ), we want the successive field context vectors INLINEFORM1 to be very different from the previous field vectors INLINEFORM2 . One way of ensuring this is to orthogonalize successive field vectors using DISPLAYFORM0 where INLINEFORM0 is the dot product between vectors INLINEFORM1 and INLINEFORM2 . The above equation essentially subtracts the component of INLINEFORM3 along INLINEFORM4 . INLINEFORM5 is a learned parameter which controls the degree of orthogonalization thereby allowing a soft orthogonalization (i.e., the entire component along INLINEFORM6 is not subtracted but only a fraction of it). The above equation only ensures that INLINEFORM7 is soft-orthogonal to INLINEFORM8 . Alternately, we could pass the sequence of context vectors, INLINEFORM9 generated so far through a GRU cell. The state of this GRU cell at each time step would thus be aware of the history of the field vectors till that timestep. Now instead of orthogonalizing INLINEFORM10 to INLINEFORM11 we could orthogonalize INLINEFORM12 to the hidden state of this GRU at time-step INLINEFORM13 . In practice, we found this to work better as it accounts for all the field vectors in the history instead of only the previous field vector. In summary, Equation provides a mechanism for remembering the current field vector when appropriate (thus capturing stay-on behavior) using a remember gate. On the other hand, Equation EQREF10 explicitly ensures that the field vector is very different (soft-orthogonal) from the previous field vectors once it is time to move on (thus capturing never look back behavior). The value of INLINEFORM0 computed in Equation EQREF10 is then used in Equation . The INLINEFORM1 (macro) thus obtained is then concatenated with INLINEFORM2 (micro) and fed to the decoder (see Fig. FIGREF3 ) Experimental setup We now describe our experimental setup: Datasets We use the WikiBio dataset introduced by lebret2016neural. It consists of INLINEFORM0 biography articles from English Wikipedia. A biography article corresponds to a person (sportsman, politician, historical figure, actor, etc.). Each Wikipedia article has an accompanying infobox which serves as the structured input and the task is to generate the first sentence of the article (which typically is a one-line description of the person). We used the same train, valid and test sets which were made publicly available by lebret2016neural. We also introduce two new biography datasets, one in French and one in German. These datasets were created and pre-processed using the same procedure as outlined in lebret2016neural. Specifically, we extracted the infoboxes and the first sentence from the corresponding Wikipedia article. As with the English dataset, we split the French and German datasets randomly into train (80%), test (10%) and valid (10%). The French and German datasets extracted by us has been made publicly available. The number of examples was 170K and 50K and the vocabulary size was 297K and 143K for French and German respectively. Although in this work we focus only on generating descriptions in one language, we hope that this dataset will also be useful for developing models which jointly learn to generate descriptions from structured data in multiple languages. Models compared We compare with the following models: 1. BIBREF0 : This is a conditional language model which uses a feed-forward neural network to predict the next word in the description conditioned on local characteristics (i.e., words within a field) and global characteristics (i.e., overall structure of the infobox). 2. BIBREF9 : This model was proposed in the context of the WeatherGov and RoboCup datasets which have a much smaller vocabulary. They use an improved attention model with additional regularizer terms which influence the weights assigned to the fields. 3. Basic Seq2Seq: This is the vanilla encode-attend-decode model BIBREF1 . Further, to deal with the large vocabulary ( INLINEFORM0 400K words) we use a copying mechanism as a post-processing step. Specifically, we identify the time steps at which the decoder produces unknown words (denoted by the special symbol UNK). For each such time step, we look at the attention weights on the input words and replace the UNK word by that input word which has received maximum attention at this timestep. This process is similar to the one described in BIBREF30 . Even lebret2016neural have a copying mechanism tightly integrated with their model. Hyperparameter tuning We tuned the hyperparameters of all the models using a validation set. As mentioned earlier, we used a bidirectional GRU cell as the function INLINEFORM0 for computing the representation of the fields and the values (see Section SECREF4 ). For all the models, we experimented with GRU state sizes of 128, 256 and 512. The total number of unique words in the corpus is around 400K (this includes the words in the infobox and the descriptions). Of these, we retained only the top 20K words in our vocabulary (same as BIBREF0 ). We initialized the embeddings of these words with 300 dimensional Glove embeddings BIBREF31 . We used Adam BIBREF32 with a learning rate of INLINEFORM1 , INLINEFORM2 and INLINEFORM3 . We trained the model for a maximum of 20 epochs and used early stopping with the patience set to 5 epochs. Results and Discussions We now discuss the results of our experiments. Comparison of different models Following lebret2016neural, we used BLEU-4, NIST-4 and ROUGE-4 as the evaluation metrics. We first make a few observations based on the results on the English dataset (Table TABREF15 ). The basic seq2seq model, as well as the model proposed by weather16, perform better than the model proposed by lebret2016neural. Our final model with bifocal attention and gated orthogonalization gives the best performance and does 10% (relative) better than the closest baseline (basic seq2seq) and 21% (relative) better than the current state of the art method BIBREF0 . In Table TABREF16 , we show some qualitative examples of the output generated by different models. Human Evaluations To make a qualitative assessment of the generated sentences, we conducted a human study on a sample of 500 Infoboxes which were sampled from English dataset. The annotators for this task were undergraduate and graduate students. For each of these infoboxes, we generated summaries using the basic seq2seq model and our final model with bifocal attention and gated orthogonalization. For each description and for each model, we asked three annotators to rank the output of the systems based on i) adequacy (i.e. does it capture relevant information from the infobox), (ii) fluency (i.e. grammar) and (iii) relative preference (i.e., which of the two outputs would be preferred). Overall the average fluency/adequacy (on a scale of 5) for basic seq2seq model was INLINEFORM0 and INLINEFORM1 for our model respectively. The results from Table TABREF17 suggest that in general gated orthogonalization model performs better than the basic seq2seq model. Additionally, annotators were asked to verify if the generated summaries look natural (i.e, as if they were generated by humans). In 423 out of 500 cases, the annotators said “Yes” suggesting that gated orthogonalization model indeed produces good descriptions. Performance on different languages The results on the French and German datasets are summarized in Tables TABREF20 and TABREF20 respectively. Note that the code of BIBREF0 is not publicly available, hence we could not report numbers for French and German using their model. We observe that our final model gives the best performance - though the bifocal attention model performs poorly as compared to the basic seq2seq model on French. However, the overall performance for French and German are much smaller than those for English. There could be multiple reasons for this. First, the amount of training data in these two languages is smaller than that in English. Specifically, the amount of training data available in French (German) is only INLINEFORM0 ( INLINEFORM1 )% of that available for English. Second, on average the descriptions in French and German are longer than that in English (EN: INLINEFORM2 words, FR: INLINEFORM3 words and DE: INLINEFORM4 words). Finally, a manual inspection across the three languages suggests that the English descriptions have a more consistent structure than the French descriptions. For example, most English descriptions start with name followed by date of birth but this is not the case in French. However, this is only a qualitative observation and it is hard to quantify this characteristic of the French and German datasets. Visualizing Attention Weights If the proposed model indeed works well then we should see attention weights that are consistent with the stay on and never look back behavior. To verify this, we plotted the attention weights in cases where the model with gated orthogonalization does better than the model with only bifocal attention. Figure FIGREF21 shows the attention weights corresponding to infobox in Figure FIGREF25 . Notice that the model without gated orthogonalization has attention on both name field and article title while rendering the name. The model with gated orthogonalization, on the other hand, stays on the name field for as long as it is required but then moves and never returns to it (as expected). Due to lack of space, we do not show similar plots for French and German but we would like to mention that, in general, the differences between the attention weights learned by the model with and without gated orthogonalization were more pronounced for the French/German dataset than the English dataset. This is in agreement with the results reported in Table TABREF20 and TABREF20 where the improvements given by gated orthogonalization are more for French/German than for English. Out of domain results What if the model sees a different INLINEFORM0 of person at test time? For example, what if the training data does not contain any sportspersons but at test time we encounter the infobox of a sportsperson. This is the same as seeing out-of-domain data at test time. Such a situation is quite expected in the products domain where new products with new features (fields) get frequently added to the catalog. We were interested in three questions here. First, we wanted to see if testing the model on out-of-domain data indeed leads to a drop in the performance. For this, we compared the performance of our best model in two scenarios (i) trained on data from all domains (including the target domain) and tested on the target domain (sports, arts) and (ii) trained on data from all domains except the target domain and tested on the target domain. Comparing rows 1 and 2 of Table TABREF32 we observed a significant drop in the performance. Note that the numbers for sports domain in row 1 are much better than the Arts domain because roughly 40% of the WikiBio training data contains sportspersons. Next, we wanted to see if we can use a small amount of data from the target domain to fine tune a model trained on the out of domain data. We observe that even with very small amounts of target domain data the performance starts improving significantly (see rows 3 and 4 of Table TABREF32 ). Note that if we train a model from scratch with only limited data from the target domain instead of fine-tuning a model trained on a different source domain then the performance is very poor. In particular, training a model from scratch with 10K training instances we get a BLEU score of INLINEFORM0 and INLINEFORM1 for arts and sports respectively. Finally, even though the actual words used for describing a sportsperson (footballer, cricketer, etc.) would be very different from the words used to describe an artist (actor, musician, etc.) they might share many fields (for example, date of birth, occupation, etc.). As seen in Figure FIGREF28 (attention weights corresponding to the infobox in Figure FIGREF27 ), the model predicts the attention weights correctly for common fields (such as occupation) but it is unable to use the right vocabulary to describe the occupation (since it has not seen such words frequently in the training data). However, once we fine tune the model with limited data from the target domain we see that it picks up the new vocabulary and produces a correct description of the occupation. Conclusion We present a model for generating natural language descriptions from structured data. To address specific characteristics of the problem we propose neural components for fused bifocal attention and gated orthogonalization to address stay on and never look back behavior while decoding. Our final model outperforms an existing state of the art model on a large scale WikiBio dataset by 21%. We also introduce datasets for French and German and demonstrate that our model gives state of the art results on these datasets. Finally, we perform experiments with an out-of-domain model and show that if such a model is fine-tuned with small amounts of in domain data then it can give an improved performance on the target domain. Given the multilingual nature of the new datasets, as future work, we would like to build models which can jointly learn to generate natural language descriptions from structured data in multiple languages. One idea is to replace the concepts in the input infobox by Wikidata concept ids which are language agnostic. A large amount of input vocabulary could thus be shared across languages thereby facilitating joint learning. Acknowledgements We thank Google for supporting Preksha Nema through their Google India Ph.D. Fellowship program. We also thank Microsoft Research India for supporting Shreyas Shetty through their generous travel grant for attending the conference.
What metrics are used for evaluation?
c2e475adeddcdc4d637ef0d4f5065b6a9b299827
What Ġmetrics Ġare Ġused Ġfor Ġevaluation ?
BLE U - 4 , ĠN IST - 4 , ĠR OU GE - 4
<s> What Ġmetrics Ġare Ġused Ġfor Ġevaluation ? </s> Introduction </s> R end ering Ġnatural Ġlanguage Ġdescriptions Ġfrom Ġstructured Ġdata Ġis Ġrequired Ġin Ġa Ġwide Ġvariety Ġof Ġcommercial Ġapplications Ġsuch Ġas Ġgenerating Ġdescriptions Ġof Ġproducts , Ġhotels , Ġfurniture , Ġetc ., Ġfrom Ġa Ġcorresponding Ġtable Ġof Ġfacts Ġabout Ġthe Ġentity . ĠSuch Ġa Ġtable Ġtypically Ġcontains Ġ{ field , Ġvalue } Ġpairs Ġwhere Ġthe Ġfield Ġis Ġa Ġproperty Ġof Ġthe Ġentity Ġ( e . g ., Ġcolor ) Ġand Ġthe Ġvalue Ġis Ġa Ġset Ġof Ġpossible Ġassignments Ġto Ġthis Ġproperty Ġ( e . g ., Ġcolor Ġ= Ġred ). ĠAnother Ġexample Ġof Ġthis Ġis Ġthe Ġrecently Ġintroduced Ġtask Ġof Ġgenerating Ġone Ġline Ġbiography Ġdescriptions Ġfrom Ġa Ġgiven ĠWikipedia Ġinf ob ox ĠB IB REF 0 Ġ. ĠThe ĠWikipedia Ġinf ob ox Ġserves Ġas Ġa Ġtable Ġof Ġfacts Ġabout Ġa Ġperson Ġand Ġthe Ġfirst Ġsentence Ġfrom Ġthe Ġcorresponding Ġarticle Ġserves Ġas Ġa Ġone Ġline Ġdescription Ġof Ġthe Ġperson . ĠFigure ĠFIG REF 2 Ġillustrates Ġan Ġexample Ġinput Ġinf ob ox Ġwhich Ġcontains Ġfields Ġsuch Ġas ĠBorn , ĠRes idence , ĠNational ity , ĠFields , ĠInstit utions Ġand ĠAlma ĠM ater . ĠEach Ġfield Ġfurther Ġcontains Ġsome Ġwords Ġ( e . g ., Ġparticle Ġphysics , Ġmany - body Ġtheory , Ġetc .). ĠThe Ġcorresponding Ġdescription Ġis Ġcoherent Ġwith Ġthe Ġinformation Ġcontained Ġin Ġthe Ġinf ob ox . </s> Note Ġthat Ġthe Ġnumber Ġof Ġfields Ġin Ġthe Ġinf ob ox Ġand Ġthe Ġordering Ġof Ġthe Ġfields Ġwithin Ġthe Ġinf ob ox Ġvaries Ġfrom Ġperson Ġto Ġperson . ĠGiven Ġthe Ġlarge Ġsize Ġ( 700 K Ġexamples ) Ġand Ġheter ogeneous Ġnature Ġof Ġthe Ġdataset Ġwhich Ġcontains Ġbi ographies Ġof Ġpeople Ġfrom Ġdifferent Ġbackgrounds Ġ( sports , Ġpolitics , Ġarts , Ġetc .), Ġit Ġis Ġhard Ġto Ġcome Ġup Ġwith Ġsimple Ġrule - based Ġtemplates Ġfor Ġgenerating Ġnatural Ġlanguage Ġdescriptions Ġfrom Ġinf ob ox es , Ġthereby Ġmaking Ġa Ġcase Ġfor Ġdata - driven Ġmodels . ĠBased Ġon Ġthe Ġrecent Ġsuccess Ġof Ġdata - driven Ġneural Ġmodels Ġfor Ġvarious Ġother ĠNL G Ġtasks ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, Ġone Ġsimple Ġchoice Ġis Ġto Ġtreat Ġthe Ġinf ob ox Ġas Ġa Ġsequence Ġof Ġ{ field , Ġvalue } Ġpairs Ġand Ġuse Ġa Ġstandard Ġseq 2 seq Ġmodel Ġfor Ġthis Ġtask . ĠHowever , Ġsuch Ġa Ġmodel Ġis Ġtoo Ġgeneric Ġand Ġdoes Ġnot Ġexploit Ġthe Ġspecific Ġcharacteristics Ġof Ġthis Ġtask Ġas Ġexplained Ġbelow . ĠFirst , Ġnote Ġthat Ġwhile Ġgenerating Ġsuch Ġdescriptions Ġfrom Ġstructured Ġdata , Ġa Ġhuman Ġkeeps Ġtrack Ġof Ġinformation Ġat Ġtwo Ġlevels . ĠSpecifically , Ġat Ġa Ġmacro Ġlevel , Ġshe Ġwould Ġfirst Ġdecide Ġwhich Ġfield Ġto Ġmention Ġnext Ġand Ġthen Ġat Ġa Ġmicro Ġlevel Ġdecide Ġwhich Ġof Ġthe Ġvalues Ġin Ġthe Ġfield Ġneeds Ġto Ġbe Ġmentioned Ġnext . ĠFor Ġexample , Ġshe Ġfirst Ġdecides Ġthat Ġat Ġthe Ġcurrent Ġstep , Ġthe Ġfield Ġoccupation Ġneeds Ġattention Ġand Ġthen Ġdecides Ġwhich Ġis Ġthe Ġnext Ġappropriate Ġoccupation Ġto Ġattend Ġto Ġfrom Ġthe Ġset Ġof Ġoccupations Ġ( actor , Ġdirector , Ġproducer , Ġetc .). ĠTo Ġenable Ġthis , Ġwe Ġuse Ġa Ġb if ocal Ġattention Ġmechanism Ġwhich Ġcomp utes Ġan Ġattention Ġover Ġfields Ġat Ġa Ġmacro Ġlevel Ġand Ġover Ġvalues Ġat Ġa Ġmicro Ġlevel . ĠWe Ġthen Ġfuse Ġthese Ġattention Ġweights Ġsuch Ġthat Ġthe Ġattention Ġweight Ġfor Ġa Ġfield Ġalso Ġinfluences Ġthe Ġattention Ġover Ġthe Ġvalues Ġwithin Ġit . ĠFinally , Ġwe Ġfeed Ġa Ġfused Ġcontext Ġvector Ġto Ġthe Ġdec oder Ġwhich Ġcontains Ġboth Ġfield Ġlevel Ġand Ġword Ġlevel Ġinformation . ĠNote Ġthat Ġsuch Ġtwo - level Ġattention Ġmechanisms ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġhave Ġbeen Ġused Ġin Ġthe Ġcontext Ġof Ġun struct ured Ġdata Ġ( as Ġopposed Ġto Ġstructured Ġdata Ġin Ġour Ġcase ), Ġwhere Ġat Ġa Ġmacro Ġlevel Ġone Ġneeds Ġto Ġpay Ġattention Ġto Ġsentences Ġand Ġat Ġa Ġmicro Ġlevel Ġto Ġwords Ġin Ġthe Ġsentences . </s> Next , Ġwe Ġobserve Ġthat Ġwhile Ġrendering Ġthe Ġoutput , Ġonce Ġthe Ġmodel Ġpays Ġattention Ġto Ġa Ġfield Ġ( say , Ġoccupation ) Ġit Ġneeds Ġto Ġstay Ġon Ġthis Ġfield Ġfor Ġa Ġfew Ġtim est eps Ġ( t ill Ġall Ġthe Ġoccupations Ġare Ġproduced Ġin Ġthe Ġoutput ). ĠWe Ġrefer Ġto Ġthis Ġas Ġthe Ġstay Ġon Ġbehavior . ĠFurther , Ġwe Ġnote Ġthat Ġonce Ġthe Ġtokens Ġof Ġa Ġfield Ġare Ġreferred Ġto , Ġthey Ġare Ġusually Ġnot Ġreferred Ġto Ġlater . ĠFor Ġexample , Ġonce Ġall Ġthe Ġoccupations Ġhave Ġbeen Ġlisted Ġin Ġthe Ġoutput Ġwe Ġwill Ġnever Ġvisit Ġthe Ġoccupation Ġfield Ġagain Ġbecause Ġthere Ġis Ġnothing Ġleft Ġto Ġsay Ġabout Ġit . ĠWe Ġrefer Ġto Ġthis Ġas Ġthe Ġnever Ġlook Ġback Ġbehavior . ĠTo Ġmodel Ġthe Ġstay Ġon Ġbehaviour , Ġwe Ġintroduce Ġa Ġforget Ġ( or Ġremember ) Ġgate Ġwhich Ġacts Ġas Ġa Ġsignal Ġto Ġdecide Ġwhen Ġto Ġforget Ġthe Ġcurrent Ġfield Ġ( or Ġequival ently Ġto Ġdecide Ġtill Ġwhen Ġto Ġremember Ġthe Ġcurrent Ġfield ). ĠTo Ġmodel Ġthe Ġnever Ġlook Ġback Ġbehaviour Ġwe Ġintroduce Ġa Ġg ated Ġorth og onal ization Ġmechanism Ġwhich Ġensures Ġthat Ġonce Ġa Ġfield Ġis Ġforgotten , Ġsubsequent Ġfield Ġcontext Ġvectors Ġfed Ġto Ġthe Ġdec oder Ġare Ġorth og onal Ġto Ġ( or Ġdifferent Ġfrom ) Ġthe Ġprevious Ġfield Ġcontext Ġvectors . </s> We Ġexperiment Ġwith Ġthe ĠWiki Bio Ġdataset ĠB IB REF 0 Ġwhich Ġcontains Ġaround Ġ700 K Ġ{ inf ob ox , Ġdescription } Ġpairs Ġand Ġhas Ġa Ġvocabulary Ġof Ġaround Ġ400 K Ġwords . ĠWe Ġshow Ġthat Ġthe Ġproposed Ġmodel Ġgives Ġa Ġrelative Ġimprovement Ġof Ġ21 % Ġand Ġ20 % Ġas Ġcompared Ġto Ġcurrent Ġstate Ġof Ġthe Ġart Ġmodels ĠB IB REF 0 Ġ, ĠB IB REF 9 Ġon Ġthis Ġdataset . ĠThe Ġproposed Ġmodel Ġalso Ġgives Ġa Ġrelative Ġimprovement Ġof Ġ10 % Ġas Ġcompared Ġto Ġthe Ġbasic Ġseq 2 seq Ġmodel . ĠFurther , Ġwe Ġintroduce Ġnew Ġdatasets Ġfor ĠFrench Ġand ĠGerman Ġon Ġthe Ġsame Ġlines Ġas Ġthe ĠEnglish ĠWiki Bio Ġdataset . ĠEven Ġon Ġthese Ġtwo Ġdatasets , Ġour Ġmodel Ġoutper forms Ġthe Ġstate Ġof Ġthe Ġart Ġmethods Ġmentioned Ġabove . </s> Related Ġwork </s> Natural ĠLanguage ĠGeneration Ġhas Ġalways Ġbeen Ġof Ġinterest Ġto Ġthe Ġresearch Ġcommunity Ġand Ġhas Ġreceived Ġa Ġlot Ġof Ġattention Ġin Ġthe Ġpast . ĠThe Ġapproaches Ġfor ĠNL G Ġrange Ġfrom Ġ( i ) Ġrule Ġbased Ġapproaches Ġ( e . g ., ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ) Ġ( ii ) Ġmodular Ġstatistical Ġapproaches Ġwhich Ġdivide Ġthe Ġprocess Ġinto Ġthree Ġphases Ġ( plan ning , Ġselection Ġand Ġsurface Ġrealization ) Ġand Ġuse Ġdata Ġdriven Ġapproaches Ġfor Ġone Ġor Ġmore Ġof Ġthese Ġphases ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ, ĠB IB REF 19 Ġ( iii ) Ġhybrid Ġapproaches Ġwhich Ġrely Ġon Ġa Ġcombination Ġof Ġhand crafted Ġrules Ġand Ġcorpus Ġstatistics ĠB IB REF 20 Ġ, ĠB IB REF 21 Ġ, ĠB IB REF 22 Ġand Ġ( iv ) Ġthe Ġmore Ġrecent Ġneural Ġnetwork Ġbased Ġmodels ĠB IB REF 1 Ġ. </s> Ne ural Ġmodels Ġfor ĠNL G Ġhave Ġbeen Ġproposed Ġin Ġthe Ġcontext Ġof Ġvarious Ġtasks Ġsuch Ġas Ġmachine Ġtranslation ĠB IB REF 1 Ġ, Ġdocument Ġsummar ization ĠB IB REF 2 Ġ, ĠB IB REF 4 Ġ, Ġparaph rase Ġgeneration ĠB IB REF 23 Ġ, Ġimage Ġcaption ing ĠB IB REF 24 Ġ, Ġvideo Ġsummar ization ĠB IB REF 25 Ġ, Ġquery Ġbased Ġdocument Ġsummar ization ĠB IB REF 5 Ġand Ġso Ġon . ĠMost Ġof Ġthese Ġmodels Ġare Ġdata Ġhungry Ġand Ġare Ġtrained Ġon Ġlarge Ġamounts Ġof Ġdata . ĠOn Ġthe Ġother Ġhand , ĠNL G Ġfrom Ġstructured Ġdata Ġhas Ġlargely Ġbeen Ġstudied Ġin Ġthe Ġcontext Ġof Ġsmall Ġdatasets Ġsuch Ġas ĠWeather Gov ĠB IB REF 26 Ġ, ĠRobo C up ĠB IB REF 27 Ġ, ĠNFL ĠRec aps ĠB IB REF 15 Ġ, ĠPro digy - Met e o ĠB IB REF 28 Ġand ĠT U NA ĠChallenge ĠB IB REF 29 Ġ. ĠRecently Ġweather 16 Ġproposed ĠR NN / L ST M Ġbased Ġneural Ġenc oder - dec oder Ġmodels Ġwith Ġattention Ġfor ĠWeather Gov Ġand ĠRobo C up Ġdatasets . </s> Unlike Ġthe Ġdatasets Ġmentioned Ġabove , Ġthe Ġbiography Ġdataset Ġintroduced Ġby Ġle b ret 2016 ne ural Ġis Ġlarger Ġ( 700 K Ġ{ table , Ġdescriptions } Ġpairs ) Ġand Ġhas Ġa Ġmuch Ġlarger Ġvocabulary Ġ( 400 K Ġwords Ġas Ġopposed Ġto Ġaround Ġ350 Ġor Ġfewer Ġwords Ġin Ġthe Ġabove Ġdatasets ). ĠFurther , Ġunlike Ġthe Ġfeed - forward Ġneural Ġnetwork Ġbased Ġmodel Ġproposed Ġby ĠB IB REF 0 Ġwe Ġuse Ġa Ġsequence Ġto Ġsequence Ġmodel Ġand Ġintroduce Ġcomponents Ġto Ġaddress Ġthe Ġpeculiar Ġcharacteristics Ġof Ġthe Ġtask . ĠSpecifically , Ġwe Ġintroduce Ġneural Ġcomponents Ġto Ġaddress Ġthe Ġneed Ġfor Ġattention Ġat Ġtwo Ġlevels Ġand Ġto Ġaddress Ġthe Ġstay Ġon Ġand Ġnever Ġlook Ġback Ġbehaviour Ġrequired Ġby Ġthe Ġdec oder . ĠKidd on Z C 16 Ġhave Ġexplored Ġthe Ġuse Ġof Ġcheck lists Ġto Ġtrack Ġpreviously Ġvisited Ġingredients Ġwhile Ġgenerating Ġrecipes Ġfrom Ġingredients . ĠNote Ġthat Ġtwo - level Ġattention Ġmechanisms Ġhave Ġalso Ġbeen Ġused Ġin Ġthe Ġcontext Ġof Ġsummar ization ĠB IB REF 6 Ġ, Ġdocument Ġclassification ĠB IB REF 7 Ġ, Ġdialog Ġsystems ĠB IB REF 8 Ġ, Ġetc . ĠHowever , Ġthese Ġworks Ġdeal Ġwith Ġun struct ured Ġdata Ġ( sent ences Ġat Ġthe Ġhigher Ġlevel Ġand Ġwords Ġat Ġa Ġlower Ġlevel ) Ġas Ġopposed Ġto Ġstructured Ġdata Ġin Ġour Ġcase . </s> Prop osed Ġmodel </s> As Ġinput Ġwe Ġare Ġgiven Ġan Ġinf ob ox ĠIN LINE FORM 0 Ġ, Ġwhich Ġis Ġa Ġset Ġof Ġpairs ĠIN LINE FORM 1 Ġwhere ĠIN LINE FORM 2 Ġcorresponds Ġto Ġfield Ġnames Ġand ĠIN LINE FORM 3 Ġis Ġthe Ġsequence Ġof Ġcorresponding Ġvalues Ġand ĠIN LINE FORM 4 Ġis Ġthe Ġtotal Ġnumber Ġof Ġfields Ġin ĠIN LINE FORM 5 Ġ. ĠFor Ġexample , ĠIN LINE FORM 6 Ġcould Ġbe Ġone Ġsuch Ġpair Ġin Ġthis Ġset . ĠGiven Ġsuch Ġan Ġinput , Ġthe Ġtask Ġis Ġto Ġgenerate Ġa Ġdescription ĠIN LINE FORM 7 Ġcontaining ĠIN LINE FORM 8 Ġwords . ĠA Ġsimple Ġsolution Ġis Ġto Ġtreat Ġthe Ġinf ob ox Ġas Ġa Ġsequence Ġof Ġfields Ġfollowed Ġby Ġthe Ġvalues Ġcorresponding Ġto Ġthe Ġfield Ġin Ġthe Ġorder Ġof Ġtheir Ġappearance Ġin Ġthe Ġinf ob ox . ĠFor Ġexample , Ġthe Ġinf ob ox Ġcould Ġbe Ġflattened Ġto Ġproduce Ġthe Ġfollowing Ġinput Ġsequence Ġ( the Ġwords Ġin Ġbold Ġare Ġfield Ġnames Ġwhich Ġact Ġas Ġdelim it ers ) </s> [ Name ] ĠJohn ĠDoe Ġ[ Birth _ Date ] Ġ19 ĠMarch Ġ1981 Ġ[ National ity ] ĠIndian Ġ.. ... </s> The Ġproblem Ġcan Ġthen Ġbe Ġcast Ġas Ġa Ġseq 2 seq Ġgeneration Ġproblem Ġand Ġcan Ġbe Ġmodeled Ġusing Ġa Ġstandard Ġneural Ġarchitecture Ġcomprising Ġof Ġthree Ġcomponents Ġ( i ) Ġan Ġinput Ġenc oder Ġ( using ĠGR U / L ST M Ġcells ), Ġ( ii ) Ġan Ġattention Ġmechanism Ġto Ġattend Ġto Ġimportant Ġvalues Ġin Ġthe Ġinput Ġsequence Ġat Ġeach Ġtime Ġstep Ġand Ġ( iii ) Ġa Ġdec oder Ġto Ġdecode Ġthe Ġoutput Ġone Ġword Ġat Ġa Ġtime Ġ( again , Ġusing ĠGR U / L ST M Ġcells ). ĠHowever , Ġthis Ġstandard Ġmodel Ġis Ġtoo Ġgeneric Ġand Ġdoes Ġnot Ġexploit Ġthe Ġspecific Ġcharacteristics Ġof Ġthis Ġtask . ĠWe Ġpropose Ġadditional Ġcomponents , Ġviz ., Ġ( i ) Ġa Ġfused Ġb if ocal Ġattention Ġmechanism Ġwhich Ġoperates Ġon Ġfields Ġ( mac ro ) Ġand Ġvalues Ġ( micro ) Ġand Ġ( ii ) Ġa Ġg ated Ġorth og onal ization Ġmechanism Ġto Ġmodel Ġstay Ġon Ġand Ġnever Ġlook Ġback Ġbehavior . </s> F used ĠB if ocal ĠAttention ĠMechan ism </s> Int uitive ly , Ġwhen Ġa Ġhuman Ġwrites Ġa Ġdescription Ġfrom Ġa Ġtable Ġshe Ġkeeps Ġtrack Ġof Ġinformation Ġat Ġtwo Ġlevels . ĠAt Ġthe Ġmacro Ġlevel , Ġit Ġis Ġimportant Ġto Ġdecide Ġwhich Ġis Ġthe Ġappropriate Ġfield Ġto Ġattend Ġto Ġnext Ġand Ġat Ġa Ġmicro Ġlevel Ġ( i . e ., Ġwithin Ġa Ġfield ) Ġit Ġis Ġimportant Ġto Ġknow Ġwhich Ġvalues Ġto Ġattend Ġto Ġnext . ĠTo Ġcapture Ġthis Ġbehavior , Ġwe Ġuse Ġa Ġb if ocal Ġattention Ġmechanism Ġas Ġdescribed Ġbelow . </s> Mac ro ĠAttention : ĠConsider Ġthe ĠIN LINE FORM 0 Ġ- th Ġfield ĠIN LINE FORM 1 Ġwhich Ġhas Ġvalues ĠIN LINE FORM 2 Ġ. ĠLet ĠIN LINE FORM 3 Ġbe Ġthe Ġrepresentation Ġof Ġthis Ġfield Ġin Ġthe Ġinf ob ox . ĠThis Ġrepresentation Ġcan Ġeither Ġbe Ġ( i ) Ġthe Ġword Ġembed ding Ġof Ġthe Ġfield Ġname Ġor Ġ( ii ) Ġsome Ġfunction ĠIN LINE FORM 4 Ġof Ġthe Ġvalues Ġin Ġthe Ġfield Ġor Ġ( iii ) Ġa Ġconc aten ation Ġof Ġ( i ) Ġand Ġ( ii ). ĠThe Ġfunction ĠIN LINE FORM 5 Ġcould Ġsimply Ġbe Ġthe Ġsum Ġor Ġaverage Ġof Ġthe Ġembed d ings Ġof Ġthe Ġvalues Ġin Ġthe Ġfield . ĠAltern ately , Ġthis Ġfunction Ġcould Ġbe Ġa ĠGR U Ġ( or ĠL ST M ) Ġwhich Ġtreats Ġthese Ġvalues Ġwithin Ġa Ġfield Ġas Ġa Ġsequence Ġand Ġcomp utes Ġthe Ġfield Ġrepresentation Ġas Ġthe Ġfinal Ġrepresentation Ġof Ġthis Ġsequence Ġ( i . e ., Ġthe Ġrepresentation Ġof Ġthe Ġlast Ġtime - step ). ĠWe Ġfound Ġthat Ġbid irection al ĠGR U Ġis Ġa Ġbetter Ġchoice Ġfor ĠIN LINE FORM 6 Ġand Ġconc aten ating Ġthe Ġembed ding Ġof Ġthe Ġfield Ġname Ġwith Ġthis ĠGR U Ġrepresentation Ġworks Ġbest . ĠFurther , Ġusing Ġa Ġbid irection al ĠGR U Ġcell Ġto Ġtake Ġcontextual Ġinformation Ġfrom Ġneighboring Ġfields Ġalso Ġhelps Ġ( these Ġare Ġthe Ġorange Ġcolored Ġcells Ġin Ġthe Ġtop - left Ġblock Ġin ĠFigure ĠFIG REF 3 Ġwith Ġmacro Ġattention ). ĠGiven Ġthese Ġrepresentations ĠIN LINE FORM 7 Ġfor Ġall Ġthe ĠIN LINE FORM 8 Ġfields Ġwe Ġcompute Ġan Ġattention Ġover Ġthe Ġfields Ġ( mac ro Ġlevel ). ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġstate Ġof Ġthe Ġdec oder Ġat Ġtime Ġstep ĠIN LINE FORM 1 Ġ. ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġare Ġparameters , ĠIN LINE FORM 4 Ġis Ġthe Ġtotal Ġnumber Ġof Ġfields Ġin Ġthe Ġinput , ĠIN LINE FORM 5 Ġis Ġthe Ġmacro Ġ( field Ġlevel ) Ġcontext Ġvector Ġat Ġthe ĠIN LINE FORM 6 Ġ- th Ġtime Ġstep Ġof Ġthe Ġdec oder . </s> Micro ĠAttention : ĠLet ĠIN LINE FORM 0 Ġbe Ġthe Ġrepresentation Ġof Ġthe ĠIN LINE FORM 1 Ġ- th Ġvalue Ġin Ġa Ġgiven Ġfield . ĠThis Ġrepresentation Ġcould Ġagain Ġeither Ġbe Ġ( i ) Ġsimply Ġthe Ġembed ding Ġof Ġthis Ġvalue Ġ( ii ) Ġor Ġa Ġcontextual Ġrepresentation Ġcomputed Ġusing Ġa Ġfunction ĠIN LINE FORM 2 Ġwhich Ġalso Ġconsiders Ġthe Ġother Ġvalues Ġin Ġthe Ġfield . ĠFor Ġexample , Ġif ĠIN LINE FORM 3 Ġare Ġthe Ġvalues Ġin Ġa Ġfield Ġthen Ġthese Ġvalues Ġcan Ġbe Ġtreated Ġas Ġa Ġsequence Ġand Ġthe Ġrepresentation Ġof Ġthe ĠIN LINE FORM 4 Ġ- th Ġvalue Ġcan Ġbe Ġcomputed Ġusing Ġa Ġbid irection al ĠGR U Ġover Ġthis Ġsequence . ĠOnce Ġagain , Ġwe Ġfound Ġthat Ġusing Ġa Ġbi - GR U Ġworks Ġbetter Ġthen Ġsimply Ġusing Ġthe Ġembed ding Ġof Ġthe Ġvalue . ĠOnce Ġwe Ġhave Ġsuch Ġa Ġrepresentation Ġcomputed Ġfor Ġall Ġvalues Ġacross Ġall Ġthe Ġfields , Ġwe Ġcompute Ġthe Ġattention Ġover Ġthese Ġvalues Ġ( micro Ġlevel ) Ġas Ġshown Ġbelow Ġ: ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġstate Ġof Ġthe Ġdec oder Ġat Ġtime Ġstep ĠIN LINE FORM 1 Ġ. ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġare Ġparameters , ĠIN LINE FORM 4 Ġis Ġthe Ġtotal Ġnumber Ġof Ġvalues Ġacross Ġall Ġthe Ġfields . </s> F used ĠAttention : ĠInt uitive ly , Ġthe Ġattention Ġweights Ġassigned Ġto Ġa Ġfield Ġshould Ġhave Ġan Ġinfluence Ġon Ġall Ġthe Ġvalues Ġbelonging Ġto Ġthe Ġparticular Ġfield . ĠTo Ġensure Ġthis , Ġwe Ġre we igh Ġthe Ġmicro Ġlevel Ġattention Ġweights Ġbased Ġon Ġthe Ġcorresponding Ġmacro Ġlevel Ġattention Ġweights . ĠIn Ġother Ġwords , Ġwe Ġfuse Ġthe Ġattention Ġweights Ġat Ġthe Ġtwo Ġlevels Ġas : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġfield Ġcorresponding Ġto Ġthe ĠIN LINE FORM 1 Ġ- th Ġvalue , ĠIN LINE FORM 2 Ġis Ġthe Ġmacro Ġlevel Ġcontext Ġvector . </s> G ated ĠOrth og onal ization Ġfor ĠModel ing ĠStay - On Ġand ĠNever ĠLook ĠBack Ġbehaviour </s> We Ġnow Ġdescribe Ġa Ġseries Ġof Ġchoices Ġmade Ġto Ġmodel Ġstay - on Ġand Ġnever Ġlook Ġback Ġbehavior . ĠWe Ġfirst Ġbegin Ġwith Ġthe Ġstay - on Ġproperty Ġwhich Ġessentially Ġimplies Ġthat Ġif Ġwe Ġhave Ġpaid Ġattention Ġto Ġthe Ġfield ĠIN LINE FORM 0 Ġat Ġtim est ep ĠIN LINE FORM 1 Ġthen Ġwe Ġare Ġlikely Ġto Ġpay Ġattention Ġto Ġthe Ġsame Ġfield Ġfor Ġa Ġfew Ġmore Ġtime Ġsteps . ĠFor Ġexample , Ġif Ġwe Ġare Ġfocusing Ġon Ġthe Ġoccupation Ġfield Ġat Ġthis Ġtim est ep Ġthen Ġwe Ġare Ġlikely Ġto Ġfocus Ġon Ġit Ġfor Ġthe Ġnext Ġfew Ġtim est eps Ġtill Ġall Ġrelevant Ġvalues Ġin Ġthis Ġfield Ġhave Ġbeen Ġincluded Ġin Ġthe Ġgenerated Ġdescription . ĠIn Ġother Ġwords , Ġwe Ġwant Ġto Ġremember Ġthe Ġfield Ġcontext Ġvector ĠIN LINE FORM 2 Ġfor Ġa Ġfew Ġtim est eps . ĠOne Ġway Ġof Ġensuring Ġthis Ġis Ġto Ġuse Ġa Ġremember Ġ( or Ġforget ) Ġgate Ġas Ġgiven Ġbelow Ġwhich Ġremembers Ġthe Ġprevious Ġcontext Ġvector Ġwhen Ġrequired Ġand Ġforg ets Ġit Ġwhen Ġit Ġis Ġtime Ġto Ġmove Ġon Ġfrom Ġthat Ġfield . ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġare Ġparameters Ġto Ġbe Ġlearned . ĠThe Ġjob Ġof Ġthe Ġforget Ġgate Ġis Ġto Ġensure Ġthat ĠIN LINE FORM 1 Ġis Ġsimilar Ġto ĠIN LINE FORM 2 Ġwhen Ġrequired Ġ( i . e ., Ġby Ġlearning ĠIN LINE FORM 3 Ġwhen Ġwe Ġwant Ġto Ġcontinue Ġfocusing Ġon Ġthe Ġsame Ġfield ) Ġand Ġdifferent Ġwhen Ġit Ġis Ġtime Ġto Ġmove Ġon Ġ( by Ġlearning Ġthat ĠIN LINE FORM 4 Ġ). </s> Next , Ġthe Ġnever Ġlook Ġback Ġproperty Ġimplies Ġthat Ġonce Ġwe Ġhave Ġmoved Ġaway Ġfrom Ġa Ġfield Ġwe Ġare Ġunlikely Ġto Ġpay Ġattention Ġto Ġit Ġagain . ĠFor Ġexample , Ġonce Ġwe Ġhave Ġrendered Ġall Ġthe Ġoccupations Ġin Ġthe Ġgenerated Ġdescription Ġthere Ġis Ġno Ġneed Ġto Ġreturn Ġback Ġto Ġthe Ġoccupation Ġfield . ĠIn Ġother Ġwords , Ġonce Ġwe Ġhave Ġmoved Ġon Ġ( ĠIN LINE FORM 0 Ġ), Ġwe Ġwant Ġthe Ġsuccessive Ġfield Ġcontext Ġvectors ĠIN LINE FORM 1 Ġto Ġbe Ġvery Ġdifferent Ġfrom Ġthe Ġprevious Ġfield Ġvectors ĠIN LINE FORM 2 Ġ. ĠOne Ġway Ġof Ġensuring Ġthis Ġis Ġto Ġorth og onal ize Ġsuccessive Ġfield Ġvectors Ġusing ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġdot Ġproduct Ġbetween Ġvectors ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġ. ĠThe Ġabove Ġequation Ġessentially Ġsubtract s Ġthe Ġcomponent Ġof ĠIN LINE FORM 3 Ġalong ĠIN LINE FORM 4 Ġ. ĠIN LINE FORM 5 Ġis Ġa Ġlearned Ġparameter Ġwhich Ġcontrols Ġthe Ġdegree Ġof Ġorth og onal ization Ġthereby Ġallowing Ġa Ġsoft Ġorth og onal ization Ġ( i . e ., Ġthe Ġentire Ġcomponent Ġalong ĠIN LINE FORM 6 Ġis Ġnot Ġsubt racted Ġbut Ġonly Ġa Ġfraction Ġof Ġit ). ĠThe Ġabove Ġequation Ġonly Ġensures Ġthat ĠIN LINE FORM 7 Ġis Ġsoft - orth og onal Ġto ĠIN LINE FORM 8 Ġ. ĠAltern ately , Ġwe Ġcould Ġpass Ġthe Ġsequence Ġof Ġcontext Ġvectors , ĠIN LINE FORM 9 Ġgenerated Ġso Ġfar Ġthrough Ġa ĠGR U Ġcell . ĠThe Ġstate Ġof Ġthis ĠGR U Ġcell Ġat Ġeach Ġtime Ġstep Ġwould Ġthus Ġbe Ġaware Ġof Ġthe Ġhistory Ġof Ġthe Ġfield Ġvectors Ġtill Ġthat Ġtim est ep . ĠNow Ġinstead Ġof Ġorth og onal izing ĠIN LINE FORM 10 Ġto ĠIN LINE FORM 11 Ġwe Ġcould Ġorth og onal ize ĠIN LINE FORM 12 Ġto Ġthe Ġhidden Ġstate Ġof Ġthis ĠGR U Ġat Ġtime - step ĠIN LINE FORM 13 Ġ. ĠIn Ġpractice , Ġwe Ġfound Ġthis Ġto Ġwork Ġbetter Ġas Ġit Ġaccounts Ġfor Ġall Ġthe Ġfield Ġvectors Ġin Ġthe Ġhistory Ġinstead Ġof Ġonly Ġthe Ġprevious Ġfield Ġvector . </s> In Ġsummary , ĠEqu ation Ġprovides Ġa Ġmechanism Ġfor Ġremembering Ġthe Ġcurrent Ġfield Ġvector Ġwhen Ġappropriate Ġ( thus Ġcapturing Ġstay - on Ġbehavior ) Ġusing Ġa Ġremember Ġgate . ĠOn Ġthe Ġother Ġhand , ĠEqu ation ĠEQ REF 10 Ġexplicitly Ġensures Ġthat Ġthe Ġfield Ġvector Ġis Ġvery Ġdifferent Ġ( soft - orth og onal ) Ġfrom Ġthe Ġprevious Ġfield Ġvectors Ġonce Ġit Ġis Ġtime Ġto Ġmove Ġon Ġ( thus Ġcapturing Ġnever Ġlook Ġback Ġbehavior ). ĠThe Ġvalue Ġof ĠIN LINE FORM 0 Ġcomputed Ġin ĠEqu ation ĠEQ REF 10 Ġis Ġthen Ġused Ġin ĠEqu ation Ġ. ĠThe ĠIN LINE FORM 1 Ġ( mac ro ) Ġthus Ġobtained Ġis Ġthen Ġconc aten ated Ġwith ĠIN LINE FORM 2 Ġ( micro ) Ġand Ġfed Ġto Ġthe Ġdec oder Ġ( see ĠFig . ĠFIG REF 3 Ġ) </s> Exper imental Ġsetup </s> We Ġnow Ġdescribe Ġour Ġexperimental Ġsetup : </s> Dat as ets </s> We Ġuse Ġthe ĠWiki Bio Ġdataset Ġintroduced Ġby Ġle b ret 2016 ne ural . ĠIt Ġconsists Ġof ĠIN LINE FORM 0 Ġbiography Ġarticles Ġfrom ĠEnglish ĠWikipedia . ĠA Ġbiography Ġarticle Ġcorresponds Ġto Ġa Ġperson Ġ( sports man , Ġpolitician , Ġhistorical Ġfigure , Ġactor , Ġetc .). ĠEach ĠWikipedia Ġarticle Ġhas Ġan Ġaccompanying Ġinf ob ox Ġwhich Ġserves Ġas Ġthe Ġstructured Ġinput Ġand Ġthe Ġtask Ġis Ġto Ġgenerate Ġthe Ġfirst Ġsentence Ġof Ġthe Ġarticle Ġ( which Ġtypically Ġis Ġa Ġone - line Ġdescription Ġof Ġthe Ġperson ). ĠWe Ġused Ġthe Ġsame Ġtrain , Ġvalid Ġand Ġtest Ġsets Ġwhich Ġwere Ġmade Ġpublicly Ġavailable Ġby Ġle b ret 2016 ne ural . </s> We Ġalso Ġintroduce Ġtwo Ġnew Ġbiography Ġdatasets , Ġone Ġin ĠFrench Ġand Ġone Ġin ĠGerman . ĠThese Ġdatasets Ġwere Ġcreated Ġand Ġpre - process ed Ġusing Ġthe Ġsame Ġprocedure Ġas Ġoutlined Ġin Ġle b ret 2016 ne ural . ĠSpecifically , Ġwe Ġextracted Ġthe Ġinf ob ox es Ġand Ġthe Ġfirst Ġsentence Ġfrom Ġthe Ġcorresponding ĠWikipedia Ġarticle . ĠAs Ġwith Ġthe ĠEnglish Ġdataset , Ġwe Ġsplit Ġthe ĠFrench Ġand ĠGerman Ġdatasets Ġrandomly Ġinto Ġtrain Ġ( 80 %), Ġtest Ġ( 10 %) Ġand Ġvalid Ġ( 10 %). ĠThe ĠFrench Ġand ĠGerman Ġdatasets Ġextracted Ġby Ġus Ġhas Ġbeen Ġmade Ġpublicly Ġavailable . ĠThe Ġnumber Ġof Ġexamples Ġwas Ġ170 K Ġand Ġ50 K Ġand Ġthe Ġvocabulary Ġsize Ġwas Ġ297 K Ġand Ġ143 K Ġfor ĠFrench Ġand ĠGerman Ġrespectively . ĠAlthough Ġin Ġthis Ġwork Ġwe Ġfocus Ġonly Ġon Ġgenerating Ġdescriptions Ġin Ġone Ġlanguage , Ġwe Ġhope Ġthat Ġthis Ġdataset Ġwill Ġalso Ġbe Ġuseful Ġfor Ġdeveloping Ġmodels Ġwhich Ġjointly Ġlearn Ġto Ġgenerate Ġdescriptions Ġfrom Ġstructured Ġdata Ġin Ġmultiple Ġlanguages . </s> Mod els Ġcompared </s> We Ġcompare Ġwith Ġthe Ġfollowing Ġmodels : </s> 1 . ĠB IB REF 0 Ġ: ĠThis Ġis Ġa Ġconditional Ġlanguage Ġmodel Ġwhich Ġuses Ġa Ġfeed - forward Ġneural Ġnetwork Ġto Ġpredict Ġthe Ġnext Ġword Ġin Ġthe Ġdescription Ġconditioned Ġon Ġlocal Ġcharacteristics Ġ( i . e ., Ġwords Ġwithin Ġa Ġfield ) Ġand Ġglobal Ġcharacteristics Ġ( i . e ., Ġoverall Ġstructure Ġof Ġthe Ġinf ob ox ). </s> 2 . ĠB IB REF 9 Ġ: ĠThis Ġmodel Ġwas Ġproposed Ġin Ġthe Ġcontext Ġof Ġthe ĠWeather Gov Ġand ĠRobo C up Ġdatasets Ġwhich Ġhave Ġa Ġmuch Ġsmaller Ġvocabulary . ĠThey Ġuse Ġan Ġimproved Ġattention Ġmodel Ġwith Ġadditional Ġregular izer Ġterms Ġwhich Ġinfluence Ġthe Ġweights Ġassigned Ġto Ġthe Ġfields . </s> 3 . ĠBasic ĠSe q 2 Se q : ĠThis Ġis Ġthe Ġvanilla Ġencode - att end - dec ode Ġmodel ĠB IB REF 1 Ġ. ĠFurther , Ġto Ġdeal Ġwith Ġthe Ġlarge Ġvocabulary Ġ( ĠIN LINE FORM 0 Ġ400 K Ġwords ) Ġwe Ġuse Ġa Ġcopying Ġmechanism Ġas Ġa Ġpost - processing Ġstep . ĠSpecifically , Ġwe Ġidentify Ġthe Ġtime Ġsteps Ġat Ġwhich Ġthe Ġdec oder Ġproduces Ġunknown Ġwords Ġ( den oted Ġby Ġthe Ġspecial Ġsymbol ĠUN K ). ĠFor Ġeach Ġsuch Ġtime Ġstep , Ġwe Ġlook Ġat Ġthe Ġattention Ġweights Ġon Ġthe Ġinput Ġwords Ġand Ġreplace Ġthe ĠUN K Ġword Ġby Ġthat Ġinput Ġword Ġwhich Ġhas Ġreceived Ġmaximum Ġattention Ġat Ġthis Ġtim est ep . ĠThis Ġprocess Ġis Ġsimilar Ġto Ġthe Ġone Ġdescribed Ġin ĠB IB REF 30 Ġ. ĠEven Ġle b ret 2016 ne ural Ġhave Ġa Ġcopying Ġmechanism Ġtightly Ġintegrated Ġwith Ġtheir Ġmodel . </s> Hyper param eter Ġtuning </s> We Ġtuned Ġthe Ġhyper param eters Ġof Ġall Ġthe Ġmodels Ġusing Ġa Ġvalidation Ġset . ĠAs Ġmentioned Ġearlier , Ġwe Ġused Ġa Ġbid irection al ĠGR U Ġcell Ġas Ġthe Ġfunction ĠIN LINE FORM 0 Ġfor Ġcomputing Ġthe Ġrepresentation Ġof Ġthe Ġfields Ġand Ġthe Ġvalues Ġ( see ĠSection ĠSEC REF 4 Ġ). ĠFor Ġall Ġthe Ġmodels , Ġwe Ġexperimented Ġwith ĠGR U Ġstate Ġsizes Ġof Ġ128 , Ġ256 Ġand Ġ512 . ĠThe Ġtotal Ġnumber Ġof Ġunique Ġwords Ġin Ġthe Ġcorpus Ġis Ġaround Ġ400 K Ġ( this Ġincludes Ġthe Ġwords Ġin Ġthe Ġinf ob ox Ġand Ġthe Ġdescriptions ). ĠOf Ġthese , Ġwe Ġretained Ġonly Ġthe Ġtop Ġ20 K Ġwords Ġin Ġour Ġvocabulary Ġ( same Ġas ĠB IB REF 0 Ġ). ĠWe Ġinitialized Ġthe Ġembed d ings Ġof Ġthese Ġwords Ġwith Ġ300 Ġdimensional ĠGl ove Ġembed d ings ĠB IB REF 31 Ġ. ĠWe Ġused ĠAdam ĠB IB REF 32 Ġwith Ġa Ġlearning Ġrate Ġof ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġ. ĠWe Ġtrained Ġthe Ġmodel Ġfor Ġa Ġmaximum Ġof Ġ20 Ġepoch s Ġand Ġused Ġearly Ġstopping Ġwith Ġthe Ġpatience Ġset Ġto Ġ5 Ġepoch s . </s> Results Ġand ĠDisc ussions </s> We Ġnow Ġdiscuss Ġthe Ġresults Ġof Ġour Ġexperiments . </s> Compar ison Ġof Ġdifferent Ġmodels </s> Following Ġle b ret 2016 ne ural , Ġwe Ġused ĠB LE U - 4 , ĠN IST - 4 Ġand ĠR OU GE - 4 Ġas Ġthe Ġevaluation Ġmetrics . ĠWe Ġfirst Ġmake Ġa Ġfew Ġobservations Ġbased Ġon Ġthe Ġresults Ġon Ġthe ĠEnglish Ġdataset Ġ( Table ĠT AB REF 15 Ġ). ĠThe Ġbasic Ġseq 2 seq Ġmodel , Ġas Ġwell Ġas Ġthe Ġmodel Ġproposed Ġby Ġweather 16 , Ġperform Ġbetter Ġthan Ġthe Ġmodel Ġproposed Ġby Ġle b ret 2016 ne ural . ĠOur Ġfinal Ġmodel Ġwith Ġb if ocal Ġattention Ġand Ġg ated Ġorth og onal ization Ġgives Ġthe Ġbest Ġperformance Ġand Ġdoes Ġ10 % Ġ( relative ) Ġbetter Ġthan Ġthe Ġclosest Ġbaseline Ġ( basic Ġseq 2 seq ) Ġand Ġ21 % Ġ( relative ) Ġbetter Ġthan Ġthe Ġcurrent Ġstate Ġof Ġthe Ġart Ġmethod ĠB IB REF 0 Ġ. ĠIn ĠTable ĠT AB REF 16 Ġ, Ġwe Ġshow Ġsome Ġqualitative Ġexamples Ġof Ġthe Ġoutput Ġgenerated Ġby Ġdifferent Ġmodels . </s> Human ĠEval uations </s> To Ġmake Ġa Ġqualitative Ġassessment Ġof Ġthe Ġgenerated Ġsentences , Ġwe Ġconducted Ġa Ġhuman Ġstudy Ġon Ġa Ġsample Ġof Ġ500 ĠInf ob ox es Ġwhich Ġwere Ġsampled Ġfrom ĠEnglish Ġdataset . ĠThe Ġannot ators Ġfor Ġthis Ġtask Ġwere Ġundergraduate Ġand Ġgraduate Ġstudents . ĠFor Ġeach Ġof Ġthese Ġinf ob ox es , Ġwe Ġgenerated Ġsumm aries Ġusing Ġthe Ġbasic Ġseq 2 seq Ġmodel Ġand Ġour Ġfinal Ġmodel Ġwith Ġb if ocal Ġattention Ġand Ġg ated Ġorth og onal ization . ĠFor Ġeach Ġdescription Ġand Ġfor Ġeach Ġmodel , Ġwe Ġasked Ġthree Ġannot ators Ġto Ġrank Ġthe Ġoutput Ġof Ġthe Ġsystems Ġbased Ġon Ġi ) Ġadequ acy Ġ( i . e . Ġdoes Ġit Ġcapture Ġrelevant Ġinformation Ġfrom Ġthe Ġinf ob ox ), Ġ( ii ) Ġflu ency Ġ( i . e . Ġgrammar ) Ġand Ġ( iii ) Ġrelative Ġpreference Ġ( i . e ., Ġwhich Ġof Ġthe Ġtwo Ġoutputs Ġwould Ġbe Ġpreferred ). ĠOverall Ġthe Ġaverage Ġflu ency / adequ acy Ġ( on Ġa Ġscale Ġof Ġ5 ) Ġfor Ġbasic Ġseq 2 seq Ġmodel Ġwas ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġfor Ġour Ġmodel Ġrespectively . </s> The Ġresults Ġfrom ĠTable ĠT AB REF 17 Ġsuggest Ġthat Ġin Ġgeneral Ġg ated Ġorth og onal ization Ġmodel Ġperforms Ġbetter Ġthan Ġthe Ġbasic Ġseq 2 seq Ġmodel . ĠAdditionally , Ġannot ators Ġwere Ġasked Ġto Ġverify Ġif Ġthe Ġgenerated Ġsumm aries Ġlook Ġnatural Ġ( i . e , Ġas Ġif Ġthey Ġwere Ġgenerated Ġby Ġhumans ). ĠIn Ġ423 Ġout Ġof Ġ500 Ġcases , Ġthe Ġannot ators Ġsaid ĠâĢ ľ Yes âĢ Ŀ Ġsuggesting Ġthat Ġg ated Ġorth og onal ization Ġmodel Ġindeed Ġproduces Ġgood Ġdescriptions . </s> Performance Ġon Ġdifferent Ġlanguages </s> The Ġresults Ġon Ġthe ĠFrench Ġand ĠGerman Ġdatasets Ġare Ġsummarized Ġin ĠTables ĠT AB REF 20 Ġand ĠT AB REF 20 Ġrespectively . ĠNote Ġthat Ġthe Ġcode Ġof ĠB IB REF 0 Ġis Ġnot Ġpublicly Ġavailable , Ġhence Ġwe Ġcould Ġnot Ġreport Ġnumbers Ġfor ĠFrench Ġand ĠGerman Ġusing Ġtheir Ġmodel . ĠWe Ġobserve Ġthat Ġour Ġfinal Ġmodel Ġgives Ġthe Ġbest Ġperformance Ġ- Ġthough Ġthe Ġb if ocal Ġattention Ġmodel Ġperforms Ġpoorly Ġas Ġcompared Ġto Ġthe Ġbasic Ġseq 2 seq Ġmodel Ġon ĠFrench . ĠHowever , Ġthe Ġoverall Ġperformance Ġfor ĠFrench Ġand ĠGerman Ġare Ġmuch Ġsmaller Ġthan Ġthose Ġfor ĠEnglish . ĠThere Ġcould Ġbe Ġmultiple Ġreasons Ġfor Ġthis . ĠFirst , Ġthe Ġamount Ġof Ġtraining Ġdata Ġin Ġthese Ġtwo Ġlanguages Ġis Ġsmaller Ġthan Ġthat Ġin ĠEnglish . ĠSpecifically , Ġthe Ġamount Ġof Ġtraining Ġdata Ġavailable Ġin ĠFrench Ġ( German ) Ġis Ġonly ĠIN LINE FORM 0 Ġ( ĠIN LINE FORM 1 Ġ) % Ġof Ġthat Ġavailable Ġfor ĠEnglish . ĠSecond , Ġon Ġaverage Ġthe Ġdescriptions Ġin ĠFrench Ġand ĠGerman Ġare Ġlonger Ġthan Ġthat Ġin ĠEnglish Ġ( EN : ĠIN LINE FORM 2 Ġwords , ĠFR : ĠIN LINE FORM 3 Ġwords Ġand ĠDE : ĠIN LINE FORM 4 Ġwords ). ĠFinally , Ġa Ġmanual Ġinspection Ġacross Ġthe Ġthree Ġlanguages Ġsuggests Ġthat Ġthe ĠEnglish Ġdescriptions Ġhave Ġa Ġmore Ġconsistent Ġstructure Ġthan Ġthe ĠFrench Ġdescriptions . ĠFor Ġexample , Ġmost ĠEnglish Ġdescriptions Ġstart Ġwith Ġname Ġfollowed Ġby Ġdate Ġof Ġbirth Ġbut Ġthis Ġis Ġnot Ġthe Ġcase Ġin ĠFrench . ĠHowever , Ġthis Ġis Ġonly Ġa Ġqualitative Ġobservation Ġand Ġit Ġis Ġhard Ġto Ġquantify Ġthis Ġcharacteristic Ġof Ġthe ĠFrench Ġand ĠGerman Ġdatasets . </s> Visual izing ĠAttention ĠWe ights </s> If Ġthe Ġproposed Ġmodel Ġindeed Ġworks Ġwell Ġthen Ġwe Ġshould Ġsee Ġattention Ġweights Ġthat Ġare Ġconsistent Ġwith Ġthe Ġstay Ġon Ġand Ġnever Ġlook Ġback Ġbehavior . ĠTo Ġverify Ġthis , Ġwe Ġplotted Ġthe Ġattention Ġweights Ġin Ġcases Ġwhere Ġthe Ġmodel Ġwith Ġg ated Ġorth og onal ization Ġdoes Ġbetter Ġthan Ġthe Ġmodel Ġwith Ġonly Ġb if ocal Ġattention . ĠFigure ĠFIG REF 21 Ġshows Ġthe Ġattention Ġweights Ġcorresponding Ġto Ġinf ob ox Ġin ĠFigure ĠFIG REF 25 Ġ. ĠNotice Ġthat Ġthe Ġmodel Ġwithout Ġg ated Ġorth og onal ization Ġhas Ġattention Ġon Ġboth Ġname Ġfield Ġand Ġarticle Ġtitle Ġwhile Ġrendering Ġthe Ġname . ĠThe Ġmodel Ġwith Ġg ated Ġorth og onal ization , Ġon Ġthe Ġother Ġhand , Ġstays Ġon Ġthe Ġname Ġfield Ġfor Ġas Ġlong Ġas Ġit Ġis Ġrequired Ġbut Ġthen Ġmoves Ġand Ġnever Ġreturns Ġto Ġit Ġ( as Ġexpected ). </s> Due Ġto Ġlack Ġof Ġspace , Ġwe Ġdo Ġnot Ġshow Ġsimilar Ġplots Ġfor ĠFrench Ġand ĠGerman Ġbut Ġwe Ġwould Ġlike Ġto Ġmention Ġthat , Ġin Ġgeneral , Ġthe Ġdifferences Ġbetween Ġthe Ġattention Ġweights Ġlearned Ġby Ġthe Ġmodel Ġwith Ġand Ġwithout Ġg ated Ġorth og onal ization Ġwere Ġmore Ġpronounced Ġfor Ġthe ĠFrench / German Ġdataset Ġthan Ġthe ĠEnglish Ġdataset . ĠThis Ġis Ġin Ġagreement Ġwith Ġthe Ġresults Ġreported Ġin ĠTable ĠT AB REF 20 Ġand ĠT AB REF 20 Ġwhere Ġthe Ġimprovements Ġgiven Ġby Ġg ated Ġorth og onal ization Ġare Ġmore Ġfor ĠFrench / German Ġthan Ġfor ĠEnglish . </s> Out Ġof Ġdomain Ġresults </s> What Ġif Ġthe Ġmodel Ġsees Ġa Ġdifferent ĠIN LINE FORM 0 Ġof Ġperson Ġat Ġtest Ġtime ? ĠFor Ġexample , Ġwhat Ġif Ġthe Ġtraining Ġdata Ġdoes Ġnot Ġcontain Ġany Ġsports pers ons Ġbut Ġat Ġtest Ġtime Ġwe Ġencounter Ġthe Ġinf ob ox Ġof Ġa Ġsports person . ĠThis Ġis Ġthe Ġsame Ġas Ġseeing Ġout - of - domain Ġdata Ġat Ġtest Ġtime . ĠSuch Ġa Ġsituation Ġis Ġquite Ġexpected Ġin Ġthe Ġproducts Ġdomain Ġwhere Ġnew Ġproducts Ġwith Ġnew Ġfeatures Ġ( fields ) Ġget Ġfrequently Ġadded Ġto Ġthe Ġcatalog . ĠWe Ġwere Ġinterested Ġin Ġthree Ġquestions Ġhere . ĠFirst , Ġwe Ġwanted Ġto Ġsee Ġif Ġtesting Ġthe Ġmodel Ġon Ġout - of - domain Ġdata Ġindeed Ġleads Ġto Ġa Ġdrop Ġin Ġthe Ġperformance . ĠFor Ġthis , Ġwe Ġcompared Ġthe Ġperformance Ġof Ġour Ġbest Ġmodel Ġin Ġtwo Ġscenarios Ġ( i ) Ġtrained Ġon Ġdata Ġfrom Ġall Ġdomains Ġ( including Ġthe Ġtarget Ġdomain ) Ġand Ġtested Ġon Ġthe Ġtarget Ġdomain Ġ( sports , Ġarts ) Ġand Ġ( ii ) Ġtrained Ġon Ġdata Ġfrom Ġall Ġdomains Ġexcept Ġthe Ġtarget Ġdomain Ġand Ġtested Ġon Ġthe Ġtarget Ġdomain . ĠComp aring Ġrows Ġ1 Ġand Ġ2 Ġof ĠTable ĠT AB REF 32 Ġwe Ġobserved Ġa Ġsignificant Ġdrop Ġin Ġthe Ġperformance . ĠNote Ġthat Ġthe Ġnumbers Ġfor Ġsports Ġdomain Ġin Ġrow Ġ1 Ġare Ġmuch Ġbetter Ġthan Ġthe ĠArts Ġdomain Ġbecause Ġroughly Ġ40 % Ġof Ġthe ĠWiki Bio Ġtraining Ġdata Ġcontains Ġsports pers ons . </s> Next , Ġwe Ġwanted Ġto Ġsee Ġif Ġwe Ġcan Ġuse Ġa Ġsmall Ġamount Ġof Ġdata Ġfrom Ġthe Ġtarget Ġdomain Ġto Ġfine Ġtune Ġa Ġmodel Ġtrained Ġon Ġthe Ġout Ġof Ġdomain Ġdata . ĠWe Ġobserve Ġthat Ġeven Ġwith Ġvery Ġsmall Ġamounts Ġof Ġtarget Ġdomain Ġdata Ġthe Ġperformance Ġstarts Ġimproving Ġsignificantly Ġ( see Ġrows Ġ3 Ġand Ġ4 Ġof ĠTable ĠT AB REF 32 Ġ). ĠNote Ġthat Ġif Ġwe Ġtrain Ġa Ġmodel Ġfrom Ġscratch Ġwith Ġonly Ġlimited Ġdata Ġfrom Ġthe Ġtarget Ġdomain Ġinstead Ġof Ġfine - tun ing Ġa Ġmodel Ġtrained Ġon Ġa Ġdifferent Ġsource Ġdomain Ġthen Ġthe Ġperformance Ġis Ġvery Ġpoor . ĠIn Ġparticular , Ġtraining Ġa Ġmodel Ġfrom Ġscratch Ġwith Ġ10 K Ġtraining Ġinstances Ġwe Ġget Ġa ĠB LE U Ġscore Ġof ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġfor Ġarts Ġand Ġsports Ġrespectively . ĠFinally , Ġeven Ġthough Ġthe Ġactual Ġwords Ġused Ġfor Ġdescribing Ġa Ġsports person Ġ( football er , Ġcr ick eter , Ġetc .) Ġwould Ġbe Ġvery Ġdifferent Ġfrom Ġthe Ġwords Ġused Ġto Ġdescribe Ġan Ġartist Ġ( actor , Ġmusician , Ġetc .) Ġthey Ġmight Ġshare Ġmany Ġfields Ġ( for Ġexample , Ġdate Ġof Ġbirth , Ġoccupation , Ġetc .). ĠAs Ġseen Ġin ĠFigure ĠFIG REF 28 Ġ( att ention Ġweights Ġcorresponding Ġto Ġthe Ġinf ob ox Ġin ĠFigure ĠFIG REF 27 Ġ), Ġthe Ġmodel Ġpredicts Ġthe Ġattention Ġweights Ġcorrectly Ġfor Ġcommon Ġfields Ġ( such Ġas Ġoccupation ) Ġbut Ġit Ġis Ġunable Ġto Ġuse Ġthe Ġright Ġvocabulary Ġto Ġdescribe Ġthe Ġoccupation Ġ( since Ġit Ġhas Ġnot Ġseen Ġsuch Ġwords Ġfrequently Ġin Ġthe Ġtraining Ġdata ). ĠHowever , Ġonce Ġwe Ġfine Ġtune Ġthe Ġmodel Ġwith Ġlimited Ġdata Ġfrom Ġthe Ġtarget Ġdomain Ġwe Ġsee Ġthat Ġit Ġpicks Ġup Ġthe Ġnew Ġvocabulary Ġand Ġproduces Ġa Ġcorrect Ġdescription Ġof Ġthe Ġoccupation . </s> Conclusion </s> We Ġpresent Ġa Ġmodel Ġfor Ġgenerating Ġnatural Ġlanguage Ġdescriptions Ġfrom Ġstructured Ġdata . ĠTo Ġaddress Ġspecific Ġcharacteristics Ġof Ġthe Ġproblem Ġwe Ġpropose Ġneural Ġcomponents Ġfor Ġfused Ġb if ocal Ġattention Ġand Ġg ated Ġorth og onal ization Ġto Ġaddress Ġstay Ġon Ġand Ġnever Ġlook Ġback Ġbehavior Ġwhile Ġdecoding . ĠOur Ġfinal Ġmodel Ġoutper forms Ġan Ġexisting Ġstate Ġof Ġthe Ġart Ġmodel Ġon Ġa Ġlarge Ġscale ĠWiki Bio Ġdataset Ġby Ġ21 %. ĠWe Ġalso Ġintroduce Ġdatasets Ġfor ĠFrench Ġand ĠGerman Ġand Ġdemonstrate Ġthat Ġour Ġmodel Ġgives Ġstate Ġof Ġthe Ġart Ġresults Ġon Ġthese Ġdatasets . ĠFinally , Ġwe Ġperform Ġexperiments Ġwith Ġan Ġout - of - domain Ġmodel Ġand Ġshow Ġthat Ġif Ġsuch Ġa Ġmodel Ġis Ġfine - tun ed Ġwith Ġsmall Ġamounts Ġof Ġin Ġdomain Ġdata Ġthen Ġit Ġcan Ġgive Ġan Ġimproved Ġperformance Ġon Ġthe Ġtarget Ġdomain . </s> Given Ġthe Ġmult ilingual Ġnature Ġof Ġthe Ġnew Ġdatasets , Ġas Ġfuture Ġwork , Ġwe Ġwould Ġlike Ġto Ġbuild Ġmodels Ġwhich Ġcan Ġjointly Ġlearn Ġto Ġgenerate Ġnatural Ġlanguage Ġdescriptions Ġfrom Ġstructured Ġdata Ġin Ġmultiple Ġlanguages . ĠOne Ġidea Ġis Ġto Ġreplace Ġthe Ġconcepts Ġin Ġthe Ġinput Ġinf ob ox Ġby ĠWik id ata Ġconcept Ġ ids Ġwhich Ġare Ġlanguage Ġag nostic . ĠA Ġlarge Ġamount Ġof Ġinput Ġvocabulary Ġcould Ġthus Ġbe Ġshared Ġacross Ġlanguages Ġthereby Ġfacilitating Ġjoint Ġlearning . </s> Acknowled gements </s> We Ġthank ĠGoogle Ġfor Ġsupporting ĠPre ks ha ĠN ema Ġthrough Ġtheir ĠGoogle ĠIndia ĠPh . D . ĠFellowship Ġprogram . ĠWe Ġalso Ġthank ĠMicrosoft ĠResearch ĠIndia Ġfor Ġsupporting ĠSh rey as ĠShe tty Ġthrough Ġtheir Ġgenerous Ġtravel Ġgrant Ġfor Ġattending Ġthe Ġconference .