rther constrain the model on the data structure in order to prevent inaccurate of even contradictory descriptions
EXTRACTIVE
to further constrain the model on the data structure in order to prevent inaccurate of even contradictory descriptions
EXTRACTIVE
In this work we have proposed a hierarchical encoder for structured data, which 1) leverages the structure to form efficient representation of its input; 2) has strong synergy with the hierarchical attention of its associated decoder. This results in an effective and more light-weight model. Experimental evaluation on the RotoWire benchmark shows that our model outperforms competitive baselines in terms of BLEU score and is generally better on qualitative metrics. This way of representing structured databases may lead to automatic inference and enrichment, e.g., by comparing entities. This direction could be driven by very recent operation-guided networks BIBREF39, BIBREF40. In addition, we note that our approach can still lead to erroneous facts or even hallucinations. An interesting perspective might be to further constrain the model on the data structure in order to prevent inaccurate of even contradictory descriptions.
In this work we have proposed a hierarchical encoder for structured data, which 1) leverages the structure to form efficient representation of its input; 2) has strong synergy with the hierarchical attention of its associated decoder. This results in an effective and more light-weight model. Experimental evaluation on the RotoWire benchmark shows that our model outperforms competitive baselines in terms of BLEU score and is generally better on qualitative metrics. This way of representing structured databases may lead to automatic inference and enrichment, e.g., by comparing entities. This direction could be driven by very recent operation-guided networks BIBREF39, BIBREF40. In addition, we note that our approach can still lead to erroneous facts or even hallucinations. An interesting perspective might be to further constrain the model on the data structure in order to prevent inaccurate of even contradictory descriptions.
1912.10011
Introduction </s> Know ledge Ġand / or Ġdata Ġis Ġoften Ġmodeled Ġin Ġa Ġstructure , Ġsuch Ġas Ġindexes , Ġtables , Ġkey - value Ġpairs , Ġor Ġtriple ts . ĠThese Ġdata , Ġby Ġtheir Ġnature Ġ( e . g ., Ġraw Ġdata Ġor Ġlong Ġtime - series Ġdata ), Ġare Ġnot Ġeasily Ġusable Ġby Ġhumans ; Ġoutlining Ġtheir Ġcrucial Ġneed Ġto Ġbe Ġsynthes ized . ĠRecently , Ġnumerous Ġworks Ġhave Ġfocused Ġon Ġleveraging Ġstructured Ġdata Ġin Ġvarious Ġapplications , Ġsuch Ġas Ġquestion Ġanswering ĠB IB REF 0 , ĠB IB REF 1 Ġor Ġtable Ġretrieval ĠB IB REF 2 , ĠB IB REF 3 . ĠOne Ġemerging Ġresearch Ġfield Ġconsists Ġin Ġtransc ribing Ġdata - struct ures Ġinto Ġnatural Ġlanguage Ġin Ġorder Ġto Ġease Ġtheir Ġunderstand abl ity Ġand Ġtheir Ġus abl ity . ĠThis Ġfield Ġis Ġreferred Ġto Ġas ĠâĢ ľ data - to - text " ĠB IB REF 4 Ġand Ġhas Ġits Ġplace Ġin Ġseveral Ġapplication Ġdomains Ġ( such Ġas Ġjournalism ĠB IB REF 5 Ġor Ġmedical Ġdiagnosis ĠB IB REF 6 ) Ġor Ġwide - aud ience Ġapplications Ġ( such Ġas Ġfinancial ĠB IB REF 7 Ġand Ġweather Ġreports ĠB IB REF 8 , Ġor Ġsport Ġbroadcasting ĠB IB REF 9 , ĠB IB REF 10 ). ĠAs Ġan Ġexample , ĠFigure ĠFIG REF 1 Ġshows Ġa Ġdata - st ructure Ġcontaining Ġstatistics Ġon ĠNBA Ġbasketball Ġgames , Ġpaired Ġwith Ġits Ġcorresponding Ġjournalistic Ġdescription . </s> Design ing Ġdata - to - text Ġmodels Ġgives Ġrise Ġto Ġtwo Ġmain Ġchallenges : Ġ1 ) Ġunderstanding Ġstructured Ġdata Ġand Ġ2 ) Ġgenerating Ġassociated Ġdescriptions . ĠRecent Ġdata - to - text Ġmodels ĠB IB REF 11 , ĠB IB REF 12 , ĠB IB REF 13 , ĠB IB REF 10 Ġmostly Ġrely Ġon Ġan Ġenc oder - dec oder Ġarchitecture ĠB IB REF 14 Ġin Ġwhich Ġthe Ġdata - st ructure Ġis Ġfirst Ġencoded Ġsequ entially Ġinto Ġa Ġfixed - size Ġvector ial Ġrepresentation Ġby Ġan Ġenc oder . ĠThen , Ġa Ġdec oder Ġgenerates Ġwords Ġconditioned Ġon Ġthis Ġrepresentation . ĠWith Ġthe Ġintroduction Ġof Ġthe Ġattention Ġmechanism ĠB IB REF 15 Ġon Ġone Ġhand , Ġwhich Ġcomp utes Ġa Ġcontext Ġfocused Ġon Ġimportant Ġelements Ġfrom Ġthe Ġinput Ġat Ġeach Ġdecoding Ġstep Ġand , Ġon Ġthe Ġother Ġhand , Ġthe Ġcopy Ġmechanism ĠB IB REF 16 , ĠB IB REF 17 Ġto Ġdeal Ġwith Ġunknown Ġor Ġrare Ġwords , Ġthese Ġsystems Ġproduce Ġfluent Ġand Ġdomain Ġcomprehensive Ġtexts . ĠFor Ġinstance , ĠRobert i Ġet Ġal . ĠB IB REF 18 Ġtrain Ġa Ġcharacter - wise Ġenc oder - dec oder Ġto Ġgenerate Ġdescriptions Ġof Ġrestaurants Ġbased Ġon Ġtheir Ġattributes , Ġwhile ĠP ud upp ully Ġet Ġal . ĠB IB REF 12 Ġdesign Ġa Ġmore Ġcomplex Ġtwo - step Ġdec oder : Ġthey Ġfirst Ġgenerate Ġa Ġplan Ġof Ġelements Ġto Ġbe Ġmentioned , Ġand Ġthen Ġcondition Ġtext Ġgeneration Ġon Ġthis Ġplan . ĠAlthough Ġprevious Ġwork Ġyield Ġoverall Ġgood Ġresults , Ġwe Ġidentify Ġtwo Ġimportant Ġcaveats , Ġthat Ġhinder Ġprecision Ġ( i . e . Ġfactual Ġmentions ) Ġin Ġthe Ġdescriptions : </s> Lin ear ization Ġof Ġthe Ġdata - st ructure . ĠIn Ġpractice , Ġmost Ġworks Ġfocus Ġon Ġintroducing Ġinnov ating Ġdecoding Ġmodules , Ġand Ġstill Ġrepresent Ġdata Ġas Ġa Ġunique Ġsequence Ġof Ġelements Ġto Ġbe Ġencoded . ĠFor Ġexample , Ġthe Ġtable Ġfrom ĠFigure ĠFIG REF 1 Ġwould Ġbe Ġlinear ized Ġto Ġ[( H awks , ĠH / V , ĠH ), Ġ... , Ġ( Magic , ĠH / V , ĠV ), Ġ... ], Ġeffectively Ġleading Ġto Ġlosing Ġdistinction Ġbetween Ġrows , Ġand Ġtherefore Ġentities . ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge , Ġonly ĠLiu Ġet Ġal . ĠB IB REF 19 , ĠB IB REF 11 Ġpropose Ġenc od ers Ġconstrained Ġby Ġthe Ġstructure Ġbut Ġthese Ġapproaches Ġare Ġdesigned Ġfor Ġsingle - entity Ġstructures . </s> Ar bit rary Ġordering Ġof Ġun ordered Ġcollections Ġin Ġrecurrent Ġnetworks Ġ( R NN ). ĠMost Ġdata - to - text Ġsystems Ġuse ĠR NN s Ġas Ġenc od ers Ġ( such Ġas ĠGR Us Ġor ĠL ST Ms ), Ġthese Ġarchitectures Ġhave Ġhowever Ġsome Ġlimitations . ĠIndeed , Ġthey Ġrequire Ġin Ġpractice Ġtheir Ġinput Ġto Ġbe Ġfed Ġsequ entially . ĠThis Ġway Ġof Ġencoding Ġun ordered Ġsequences Ġ( i . e . Ġcollections Ġof Ġentities ) Ġimplicitly Ġassumes Ġan Ġarbitrary Ġorder Ġwithin Ġthe Ġcollection Ġwhich , Ġas Ġdemonstrated Ġby ĠV iny als Ġet Ġal . ĠB IB REF 20 , Ġsignificantly Ġimpacts Ġthe Ġlearning Ġperformance . </s> To Ġaddress Ġthese Ġshortcomings , Ġwe Ġpropose Ġa Ġnew Ġstructured - data Ġenc oder Ġassuming Ġthat Ġstructures Ġshould Ġbe Ġhierarch ically Ġcaptured . ĠOur Ġcontribution Ġfocuses Ġon Ġthe Ġencoding Ġof Ġthe Ġdata - st ructure , Ġthus Ġthe Ġdec oder Ġis Ġchosen Ġto Ġbe Ġa Ġclassical Ġmodule Ġas Ġused Ġin ĠB IB REF 12 , ĠB IB REF 10 . ĠOur Ġcontribution Ġis Ġthree fold : </s> We Ġmodel Ġthe Ġgeneral Ġstructure Ġof Ġthe Ġdata Ġusing Ġa Ġtwo - level Ġarchitecture , Ġfirst Ġencoding Ġall Ġentities Ġon Ġthe Ġbasis Ġof Ġtheir Ġelements , Ġthen Ġencoding Ġthe Ġdata Ġstructure Ġon Ġthe Ġbasis Ġof Ġits Ġentities ; </s> We Ġintroduce Ġthe ĠTrans former Ġenc oder ĠB IB REF 21 Ġin Ġdata - to - text Ġmodels Ġto Ġensure Ġrobust Ġencoding Ġof Ġeach Ġelement / ent ities Ġin Ġcomparison Ġto Ġall Ġothers , Ġno Ġmatter Ġtheir Ġinitial Ġpositioning ; </s> We Ġintegrate Ġa Ġhierarchical Ġattention Ġmechanism Ġto Ġcompute Ġthe Ġhierarchical Ġcontext Ġfed Ġinto Ġthe Ġdec oder . </s> We Ġreport Ġexperiments Ġon Ġthe ĠR oto Wire Ġbenchmark ĠB IB REF 10 Ġwhich Ġcontains Ġaround Ġ$ 5 K $ Ġstatistical Ġtables Ġof ĠNBA Ġbasketball Ġgames Ġpaired Ġwith Ġhuman - written Ġdescriptions . ĠOur Ġmodel Ġis Ġcompared Ġto Ġseveral Ġstate - of - the - art Ġmodels . ĠResults Ġshow Ġthat Ġthe Ġproposed Ġarchitecture Ġoutper forms Ġprevious Ġmodels Ġon ĠB LE U Ġscore Ġand Ġis Ġgenerally Ġbetter Ġon Ġqualitative Ġmetrics . </s> In Ġthe Ġfollowing , Ġwe Ġfirst Ġpresent Ġa Ġstate - of - the Ġart Ġof Ġdata - to - text Ġliterature Ġ( Section Ġ2 ), Ġand Ġthen Ġdescribe Ġour Ġproposed Ġhierarchical Ġdata Ġenc oder Ġ( Section Ġ3 ). ĠThe Ġevaluation Ġprotocol Ġis Ġpresented Ġin ĠSection Ġ4 , Ġfollowed Ġby Ġthe Ġresults Ġ( Section Ġ5 ). ĠSection Ġ6 Ġconcludes Ġthe Ġpaper Ġand Ġpresents Ġperspectives . </s> Related ĠWork </s> Until Ġrecently , Ġefforts Ġto Ġbring Ġout Ġsemantics Ġfrom Ġstructured - data Ġrelied Ġheavily Ġon Ġexpert Ġknowledge ĠB IB REF 22 , ĠB IB REF 8 . ĠFor Ġexample , Ġin Ġorder Ġto Ġbetter Ġtransc ribe Ġnumerical Ġtime Ġseries Ġof Ġweather Ġdata Ġto Ġa Ġtextual Ġforecast , ĠRe iter Ġet Ġal . ĠB IB REF 8 Ġdevise Ġcomplex Ġtemplate Ġschemes Ġin Ġcollaboration Ġwith Ġweather Ġexperts Ġto Ġbuild Ġa Ġconsistent Ġset Ġof Ġdata - to - word Ġrules . </s> Modern Ġapproaches Ġto Ġthe Ġwide Ġrange Ġof Ġtasks Ġbased Ġon Ġstructured - data Ġ( e . g . Ġtable Ġretrieval ĠB IB REF 2 , ĠB IB REF 23 , Ġtable Ġclassification ĠB IB REF 24 , Ġquestion Ġanswering ĠB IB REF 25 ) Ġnow Ġpropose Ġto Ġleverage Ġprogress Ġin Ġdeep Ġlearning Ġto Ġrepresent Ġthese Ġdata Ġinto Ġa Ġsemantic Ġvector Ġspace Ġ( also Ġcalled Ġembed ding Ġspace ). ĠIn Ġparallel , Ġan Ġemerging Ġtask , Ġcalled ĠâĢ ľ data - to - text ", Ġaims Ġat Ġdescribing Ġstructured Ġdata Ġinto Ġa Ġnatural Ġlanguage Ġdescription . ĠThis Ġtask Ġstems Ġfrom Ġthe Ġneural Ġmachine Ġtranslation Ġ( N MT ) Ġdomain , Ġand Ġearly Ġwork ĠB IB REF 26 , ĠB IB REF 27 , ĠB IB REF 10 Ġrepresent Ġthe Ġdata Ġrecords Ġas Ġa Ġsingle Ġsequence Ġof Ġfacts Ġto Ġbe Ġentirely Ġtranslated Ġinto Ġnatural Ġlanguage . ĠWis eman Ġet Ġal . ĠB IB REF 10 Ġshow Ġthe Ġlimits Ġof Ġtraditional ĠN MT Ġsystems Ġon Ġlarger Ġstructured - data , Ġwhere ĠN MT Ġsystems Ġfail Ġto Ġaccurately Ġextract Ġsalient Ġelements . </s> To Ġimprove Ġthese Ġmodels , Ġa Ġnumber Ġof Ġwork ĠB IB REF 28 , ĠB IB REF 12 , ĠB IB REF 29 Ġproposed Ġinnov ating Ġdecoding Ġmodules Ġbased Ġon Ġplanning Ġand Ġtemplates , Ġto Ġensure Ġfactual Ġand Ġcoherent Ġmentions Ġof Ġrecords Ġin Ġgenerated Ġdescriptions . ĠFor Ġexample , ĠP ud upp ully Ġet Ġal . ĠB IB REF 12 Ġpropose Ġa Ġtwo - step Ġdec oder Ġwhich Ġfirst Ġtargets Ġspecific Ġrecords Ġand Ġthen Ġuse Ġthem Ġas Ġa Ġplan Ġfor Ġthe Ġactual Ġtext Ġgeneration . ĠSimilarly , ĠLi Ġet Ġal . ĠB IB REF 28 Ġproposed Ġa Ġdelayed Ġcopy Ġmechanism Ġwhere Ġtheir Ġdec oder Ġalso Ġacts Ġin Ġtwo Ġsteps : Ġ1 ) Ġusing Ġa Ġclassical ĠL ST M Ġdec oder Ġto Ġgenerate Ġde lex ical ized Ġtext Ġand Ġ2 ) Ġusing Ġa Ġpointer Ġnetwork ĠB IB REF 30 Ġto Ġreplace Ġplace holders Ġby Ġrecords Ġfrom Ġthe Ġinput Ġdata . </s> Cl oser Ġto Ġour Ġwork , Ġvery Ġrecent Ġwork ĠB IB REF 11 , ĠB IB REF 19 , ĠB IB REF 13 Ġhave Ġproposed Ġto Ġtake Ġinto Ġaccount Ġthe Ġdata Ġstructure . ĠMore Ġparticularly , ĠP ud upp ully Ġet Ġal . ĠB IB REF 13 Ġfollow Ġentity - centric Ġtheories ĠB IB REF 31 , ĠB IB REF 32 Ġand Ġpropose Ġa Ġmodel Ġbased Ġon Ġdynamic Ġentity Ġrepresentation Ġat Ġdecoding Ġtime . ĠIt Ġconsists Ġin Ġconditioning Ġthe Ġdec oder Ġon Ġentity Ġrepresentations Ġthat Ġare Ġupdated Ġduring Ġinference Ġat Ġeach Ġdecoding Ġstep . ĠOn Ġthe Ġother Ġhand , ĠLiu Ġet Ġal . ĠB IB REF 11 , ĠB IB REF 19 Ġrather Ġfocus Ġon Ġintroducing Ġstructure Ġinto Ġthe Ġenc oder . ĠFor Ġinstance , Ġthey Ġpropose Ġa Ġdual Ġenc oder ĠB IB REF 19 Ġwhich Ġenc odes Ġseparately Ġthe Ġsequence Ġof Ġelement Ġnames Ġand Ġthe Ġsequence Ġof Ġelement Ġvalues . ĠThese Ġapproaches Ġare Ġhowever Ġdesigned Ġfor Ġsingle - entity Ġdata Ġstructures Ġand Ġdo Ġnot Ġaccount Ġfor Ġdelim itation Ġbetween Ġentities . </s> Our Ġcontribution Ġdiffers Ġfrom Ġprevious Ġwork Ġin Ġseveral Ġaspects . ĠFirst , Ġinstead Ġof Ġflat ly Ġconc aten ating Ġelements Ġfrom Ġthe Ġdata - st ructure Ġand Ġencoding Ġthem Ġas Ġa Ġsequence ĠB IB REF 11 , ĠB IB REF 12 , ĠB IB REF 10 , Ġwe Ġconst rain Ġthe Ġencoding Ġto Ġthe Ġunderlying Ġstructure Ġof Ġthe Ġinput Ġdata , Ġso Ġthat Ġthe Ġdelim itation Ġbetween Ġentities Ġremains Ġclear Ġthroughout Ġthe Ġprocess . ĠSecond , Ġunlike Ġall Ġworks Ġin Ġthe Ġdomain , Ġwe Ġexploit Ġthe ĠTrans former Ġarchitecture ĠB IB REF 21 Ġand Ġleverage Ġits Ġparticular ity Ġto Ġdirectly Ġcompare Ġelements Ġwith Ġeach Ġothers Ġin Ġorder Ġto Ġavoid Ġarbitrary Ġassumptions Ġon Ġtheir Ġordering . ĠFinally , Ġin Ġcontrast Ġto ĠB IB REF 33 , ĠB IB REF 13 Ġthat Ġuse Ġa Ġcomplex Ġupdating Ġmechanism Ġto Ġobtain Ġa Ġdynamic Ġrepresentation Ġof Ġthe Ġinput Ġdata Ġand Ġits Ġentities , Ġwe Ġargue Ġthat Ġexplicit Ġhierarchical Ġencoding Ġnaturally Ġguides Ġthe Ġdecoding Ġprocess Ġvia Ġhierarchical Ġattention . </s> H ier arch ical ĠEnc oder ĠModel Ġfor ĠData - to - Text </s> In Ġthis Ġsection Ġwe Ġintroduce Ġour Ġproposed Ġhierarchical Ġmodel Ġtaking Ġinto Ġaccount Ġthe Ġdata Ġstructure . ĠWe Ġoutline Ġthat Ġthe Ġdecoding Ġcomponent Ġaiming Ġto Ġgenerate Ġdescriptions Ġis Ġconsidered Ġas Ġa Ġblack - box Ġmodule Ġso Ġthat Ġour Ġcontribution Ġis Ġfocused Ġon Ġthe Ġencoding Ġmodule . ĠWe Ġfirst Ġdescribe Ġthe Ġmodel Ġoverview , Ġbefore Ġdetailing Ġthe Ġhierarchical Ġenc oder Ġand Ġthe Ġassociated Ġhierarchical Ġattention . </s> H ier arch ical ĠEnc oder ĠModel Ġfor ĠData - to - Text Ġ: :: ĠNot ation Ġand ĠGeneral ĠOverview </s> Let 's Ġconsider Ġthe Ġfollowing Ġnot ations : </s> $ \ bul let Ġ$ ĠAn Ġentity Ġ$ e _ i $ Ġis Ġa Ġset Ġof Ġ$ J _ i $ Ġun ordered Ġrecords Ġ$\ l brace Ġr _{ i , 1 }, Ġ... , Ġr _{ i , j }, Ġ... , Ġr _{ i , J _ i }\ r brace Ġ$ ; Ġwhere Ġrecord Ġ$ r _{ i , j } $ Ġis Ġdefined Ġas Ġa Ġpair Ġof Ġkey Ġ$ k _{ i , j } $ Ġand Ġvalue Ġ$ v _{ i , j } $. ĠWe Ġoutline Ġthat Ġ$ J _ i $ Ġmight Ġdiffer Ġbetween Ġentities . </s> $ \ bul let Ġ$ ĠA Ġdata - st ructure Ġ$ s $ Ġis Ġan Ġun ordered Ġset Ġof Ġ$ I $ Ġentities Ġ$ e _ i $. ĠWe Ġthus Ġdenote Ġ$ s Ġ\ l brace Ġe _ 1 , Ġ... , Ġe _ i , Ġ... , Ġe _ I \ r brace Ġ$ . </s> $ \ bul let Ġ$ ĠFor Ġeach Ġdata - st ructure , Ġa Ġtextual Ġdescription Ġ$ y $ Ġis Ġassociated . ĠWe Ġrefer Ġto Ġthe Ġfirst Ġ$ t $ Ġwords Ġof Ġa Ġdescription Ġ$ y $ Ġas Ġ$ y _{ 1 : t } $. ĠThus , Ġthe Ġfull Ġsequence Ġof Ġwords Ġcan Ġbe Ġnoted Ġas Ġ$ y Ġ= Ġy _{ 1 : T } $. </s> $ \ bul let Ġ$ ĠThe Ġdataset Ġ$\ math cal Ġ{ D } $ Ġis Ġa Ġcollection Ġof Ġ$ N $ Ġaligned Ġ( data - st ructure , Ġdescription ) Ġpairs Ġ$( s , y ) $. </s> For Ġinstance , ĠFigure ĠFIG REF 1 Ġillustrates Ġa Ġdata - st ructure Ġassociated Ġwith Ġa Ġdescription . ĠThe Ġdata - st ructure Ġincludes Ġa Ġset Ġof Ġentities Ġ( H awks , ĠMagic , ĠAl ĠHor ford , ĠJeff ĠTe ague , Ġ... ). ĠThe Ġentity ĠJeff ĠTe ague Ġis Ġmodeled Ġas Ġa Ġset Ġof Ġrecords Ġ{ ( P TS , Ġ17 ), Ġ( RE B , Ġ0 ), Ġ( AST , Ġ7 ) Ġ... } Ġin Ġwhich , Ġe . g ., Ġthe Ġrecord Ġ( P TS , Ġ17 ) Ġis Ġcharacterized Ġby Ġa Ġkey Ġ( P TS ) Ġand Ġa Ġvalue Ġ( 17 ). </s> For Ġeach Ġdata - st ructure Ġ$ s $ Ġin Ġ$\ math cal Ġ{ D } $, Ġthe Ġobjective Ġfunction Ġaims Ġto Ġgenerate Ġa Ġdescription Ġ$\ hat { y } $ Ġas Ġclose Ġas Ġpossible Ġto Ġthe Ġground Ġtruth Ġ$ y $. ĠThis Ġobjective Ġfunction Ġoptim izes Ġthe Ġfollowing Ġlog - like lihood Ġover Ġthe Ġwhole Ġdataset Ġ$\ math cal Ġ{ D } $ : </s> where Ġ$\ the ta Ġ$ Ġstands Ġfor Ġthe Ġmodel Ġparameters Ġand Ġ$ P (\ hat { y } = y \ Ġ| \ Ġs ; Ġ\ the ta Ġ) $ Ġthe Ġprobability Ġof Ġthe Ġmodel Ġto Ġgenerate Ġthe Ġadequate Ġdescription Ġ$ y $ Ġfor Ġtable Ġ$ s $. </s> During Ġinference , Ġwe Ġgenerate Ġthe Ġsequence Ġ$\ hat { y } ^ * $ Ġwith Ġthe Ġmaximum Ġa Ġposterior i Ġprobability Ġconditioned Ġon Ġtable Ġ$ s $. ĠUsing Ġthe Ġchain Ġrule , Ġwe Ġget : </s> This Ġequation Ġis Ġint ract able Ġin Ġpractice , Ġwe Ġapproximate Ġa Ġsolution Ġusing Ġbeam Ġsearch , Ġas Ġin ĠB IB REF 11 , ĠB IB REF 19 , ĠB IB REF 12 , ĠB IB REF 13 , ĠB IB REF 10 . </s> Our Ġmodel Ġfollows Ġthe Ġenc oder - dec oder Ġarchitecture ĠB IB REF 14 . ĠBecause Ġour Ġcontribution Ġfocuses Ġon Ġthe Ġencoding Ġprocess , Ġwe Ġchose Ġthe Ġdecoding Ġmodule Ġused Ġin ĠB IB REF 12 , ĠB IB REF 10 : Ġa Ġtwo - l ayers ĠL ST M Ġnetwork Ġwith Ġa Ġcopy Ġmechanism . ĠIn Ġorder Ġto Ġsuperv ise Ġthis Ġmechanism , Ġwe Ġassume Ġthat Ġeach Ġrecord Ġvalue Ġthat Ġalso Ġappears Ġin Ġthe Ġtarget Ġis Ġcopied Ġfrom Ġthe Ġdata - st ructure Ġand Ġwe Ġtrain Ġthe Ġmodel Ġto Ġswitch Ġbetween Ġfreely Ġgenerating Ġwords Ġfrom Ġthe Ġvocabulary Ġand Ġcopying Ġwords Ġfrom Ġthe Ġinput . ĠWe Ġnow Ġdescribe Ġthe Ġhierarchical Ġenc oder Ġand Ġthe Ġhierarchical Ġattention . </s> H ier arch ical ĠEnc oder ĠModel Ġfor ĠData - to - Text Ġ: :: ĠHier arch ical ĠEnc oding ĠModel </s> As Ġoutlined Ġin ĠSection ĠSEC REF 2 , Ġmost Ġprevious Ġwork ĠB IB REF 28 , ĠB IB REF 12 , ĠB IB REF 13 , ĠB IB REF 10 , ĠB IB REF 29 Ġmake Ġuse Ġof Ġflat Ġenc od ers Ġthat Ġdo Ġnot Ġexploit Ġthe Ġdata Ġstructure . ĠTo Ġkeep Ġthe Ġsemantics Ġof Ġeach Ġelement Ġfrom Ġthe Ġdata - st ructure , Ġwe Ġpropose Ġa Ġhierarchical Ġenc oder Ġwhich Ġrelies Ġon Ġtwo Ġmodules . ĠThe Ġfirst Ġone Ġ( module ĠA Ġin ĠFigure ĠFIG REF 11 ) Ġis Ġcalled Ġlow - level Ġenc oder Ġand Ġenc odes Ġentities Ġon Ġthe Ġbasis Ġof Ġtheir Ġrecords ; Ġthe Ġsecond Ġone Ġ( module ĠB ), Ġcalled Ġhigh - level Ġenc oder , Ġenc odes Ġthe Ġdata - st ructure Ġon Ġthe Ġbasis Ġof Ġits Ġunderlying Ġentities . ĠIn Ġthe Ġlow - level Ġenc oder , Ġthe Ġtraditional Ġembed ding Ġlayer Ġis Ġreplaced Ġby Ġa Ġrecord Ġembed ding Ġlayer Ġas Ġin ĠB IB REF 11 , ĠB IB REF 12 , ĠB IB REF 10 . ĠWe Ġpresent Ġin Ġwhat Ġfollows Ġthe Ġrecord Ġembed ding Ġlayer Ġand Ġintroduce Ġour Ġtwo Ġhierarchical Ġmodules . </s> H ier arch ical ĠEnc oder ĠModel Ġfor ĠData - to - Text Ġ: :: ĠHier arch ical ĠEnc oding ĠModel Ġ: :: ĠRecord ĠEmb edd ing ĠLayer . </s> The Ġfirst Ġlayer Ġof Ġthe Ġnetwork Ġconsists Ġin Ġlearning Ġtwo Ġembed ding Ġmat rices Ġto Ġembed Ġthe Ġrecord Ġkeys Ġand Ġvalues . ĠKeys Ġ$ k _{ i , j } $ Ġare Ġembedded Ġto Ġ$\ math bf Ġ{ k } _{ i , j } Ġ\ in Ġ\ math bb Ġ{ R } ^{ d } $ Ġand Ġvalues Ġ$ v _{ i , j } $ Ġto Ġ$\ math bf Ġ{ v } _{ i , j } Ġ\ in Ġ\ math bb Ġ{ R } ^{ d } $, Ġwith Ġ$ d $ Ġthe Ġsize Ġof Ġthe Ġembed ding . ĠAs Ġin Ġprevious Ġwork ĠB IB REF 11 , ĠB IB REF 12 , ĠB IB REF 10 , Ġeach Ġrecord Ġembed ding Ġ$\ math bf Ġ{ r } _{ i , j } $ Ġis Ġcomputed Ġby Ġa Ġlinear Ġprojection Ġon Ġthe Ġconc aten ation Ġ$ [ \ math bf Ġ{ k } _{ i , j } $ ; Ġ$\ math bf Ġ{ v } _{ i , j } ] $ Ġfollowed Ġby Ġa Ġnon Ġlinear ity : </s> where Ġ$\ math bf Ġ{ W } _ r Ġ\ in Ġ\ math bb Ġ{ R } ^{ 2 d Ġ\ times Ġd } $ Ġand Ġ$\ math bf Ġ{ b } _ r Ġ\ in Ġ\ math bb Ġ{ R } ^{ d } $ Ġare Ġlearnt Ġparameters . </s> The Ġlow - level Ġenc oder Ġaims Ġat Ġencoding Ġa Ġcollection Ġof Ġrecords Ġbelonging Ġto Ġthe Ġsame Ġentity Ġwhile Ġthe Ġhigh - level Ġenc oder Ġenc odes Ġthe Ġwhole Ġset Ġof Ġentities . ĠBoth Ġthe Ġlow - level Ġand Ġhigh - level Ġenc od ers Ġconsider Ġtheir Ġinput Ġelements Ġas Ġun ordered . ĠWe Ġuse Ġthe ĠTrans former Ġarchitecture Ġfrom ĠB IB REF 21 . ĠFor Ġeach Ġenc oder , Ġwe Ġhave Ġthe Ġfollowing Ġpeculiar ities : </s> the ĠLow - level Ġenc oder Ġenc odes Ġeach Ġentity Ġ$ e _ i $ Ġon Ġthe Ġbasis Ġof Ġits Ġrecord Ġembed d ings Ġ$\ math bf Ġ{ r } _{ i , j } $. ĠEach Ġrecord Ġembed ding Ġ$\ math bf Ġ{ r } _{ i , j } $ Ġis Ġcompared Ġto Ġother Ġrecord Ġembed d ings Ġto Ġlearn Ġits Ġfinal Ġhidden Ġrepresentation Ġ$\ math bf Ġ{ h } _{ i , j } $. ĠFurthermore , Ġwe Ġadd Ġa Ġspecial Ġrecord Ġ[ ENT ] Ġfor Ġeach Ġentity , Ġillustrated Ġin ĠFigure ĠFIG REF 11 Ġas Ġthe Ġlast Ġrecord . ĠSince Ġentities Ġmight Ġhave Ġa Ġvariable Ġnumber Ġof Ġrecords , Ġthis Ġtoken Ġallows Ġto Ġaggregate Ġfinal Ġhidden Ġrecord Ġrepresentations Ġ$\ l brace Ġ\ math bf Ġ{ h } _{ i , j }\ r brace Ġ_ { j = 1 } ^{ J _ i } $ Ġin Ġa Ġfixed - sized Ġrepresentation Ġvector Ġ$\ math bf Ġ{ h } _{ i } $. </s> the ĠHigh - level Ġenc oder Ġenc odes Ġthe Ġdata - st ructure Ġon Ġthe Ġbasis Ġof Ġits Ġentity Ġrepresentation Ġ$\ math bf Ġ{ h } _{ i } $. ĠSimilarly Ġto Ġthe ĠLow - level Ġenc oder , Ġthe Ġfinal Ġhidden Ġstate Ġ$\ math bf Ġ{ e _ i } $ Ġof Ġan Ġentity Ġis Ġcomputed Ġby Ġcomparing Ġentity Ġrepresentation Ġ$\ math bf Ġ{ h } _{ i } $ Ġwith Ġeach Ġothers . ĠThe Ġdata - st ructure Ġrepresentation Ġ$\ math bf Ġ{ z } $ Ġis Ġcomputed Ġas Ġthe Ġmean Ġof Ġthese Ġentity Ġrepresentations , Ġand Ġis Ġused Ġfor Ġthe Ġdec oder Ġinitialization . </s> H ier arch ical ĠEnc oder ĠModel Ġfor ĠData - to - Text Ġ: :: ĠHier arch ical Ġattention </s> To Ġfully Ġleverage Ġthe Ġhierarchical Ġstructure Ġof Ġour Ġenc oder , Ġwe Ġpropose Ġtwo Ġvariants Ġof Ġhierarchical Ġattention Ġmechanism Ġto Ġcompute Ġthe Ġcontext Ġfed Ġto Ġthe Ġdec oder Ġmodule . </s> $ \ bul let Ġ$ ĠTraditional ĠHier arch ical ĠAttention . ĠAs Ġin ĠB IB REF 13 , Ġwe Ġhypothes ize Ġthat Ġa Ġdynamic Ġcontext Ġshould Ġbe Ġcomputed Ġin Ġtwo Ġsteps : Ġfirst Ġattending Ġto Ġentities , Ġthen Ġto Ġrecords Ġcorresponding Ġto Ġthese Ġentities . ĠTo Ġimplement Ġthis Ġhierarchical Ġattention , Ġat Ġeach Ġdecoding Ġstep Ġ$ t $, Ġthe Ġmodel Ġlearns Ġa Ġfirst Ġset Ġof Ġattention Ġscores Ġ$\ alpha Ġ_ { i , t } $ Ġover Ġentities Ġ$ e _ i $ Ġand Ġa Ġsecond Ġset Ġof Ġattention Ġscores Ġ$\ beta Ġ_ { i , j , t } $ Ġover Ġrecords Ġ$ r _{ i , j } $ Ġbelonging Ġto Ġentity Ġ$ e _ i $. ĠThe Ġ$\ alpha Ġ_ { i , t } $ Ġscores Ġare Ġnormalized Ġto Ġform Ġa Ġdistribution Ġover Ġall Ġentities Ġ$ e _ i $, Ġand Ġ$\ beta Ġ_ { i , j , t } $ Ġscores Ġare Ġnormalized Ġto Ġform Ġa Ġdistribution Ġover Ġrecords Ġ$ r _{ i , j } $ Ġof Ġentity Ġ$ e _ i $. ĠEach Ġentity Ġis Ġthen Ġrepresented Ġas Ġa Ġweighted Ġsum Ġof Ġits Ġrecord Ġembed d ings , Ġand Ġthe Ġentire Ġdata Ġstructure Ġis Ġrepresented Ġas Ġa Ġweighted Ġsum Ġof Ġthe Ġentity Ġrepresentations . ĠThe Ġdynamic Ġcontext Ġis Ġcomputed Ġas : </s> ct Ġ= Ġi = 1 I Ġ( i , t Ġ( Ġj Ġi , j , t Ġr i , j Ġ)) </s> where Ġi , t Ġexp ( dt We i ) Ġand Ġi , j , t Ġexp ( dt Wh i , j ) </s> where Ġ$\ math bf Ġ{ d _ t } $ Ġis Ġthe Ġdec oder Ġhidden Ġstate Ġat Ġtime Ġstep Ġ$ t $, Ġ$\ math bf Ġ{ W } _{ \ alpha Ġ} Ġ\ in Ġ\ math bb Ġ{ R } ^{ d \ times Ġd } $ Ġand Ġ$\ math bf Ġ{ W } _{ \ beta Ġ} Ġ\ in Ġ\ math bb Ġ{ R } ^{ d \ times Ġd } $ Ġare Ġlearnt Ġparameters , Ġ$ Ġ\ sum Ġ_ i \ alpha Ġ_ { i , t } Ġ= Ġ1 $, Ġand Ġfor Ġall Ġ$ i Ġ\ in Ġ\ l brace Ġ1 ,... , I \ r brace Ġ$ Ġ$\ sum Ġ_ { j }\ beta Ġ_ { i , j , t } Ġ= Ġ1 $. </s> $ \ bul let Ġ$ ĠKey - guided ĠHier arch ical ĠAttention . ĠThis Ġvariant Ġfollows Ġthe Ġintuition Ġthat Ġonce Ġan Ġentity Ġis Ġchosen Ġfor Ġmention Ġ( thanks Ġto Ġ$\ alpha Ġ_ { i , t } $ ), Ġonly Ġthe Ġtype Ġof Ġrecords Ġis Ġimportant Ġto Ġdetermine Ġthe Ġcontent Ġof Ġthe Ġdescription . ĠFor Ġexample , Ġwhen Ġdeciding Ġto Ġmention Ġa Ġplayer , Ġall Ġexperts Ġautomatically Ġreport Ġhis Ġscore Ġwithout Ġconsideration Ġof Ġits Ġspecific Ġvalue . ĠTo Ġtest Ġthis Ġintuition , Ġwe Ġmodel Ġthe Ġattention Ġscores Ġby Ġcomputing Ġthe Ġ$\ beta Ġ_ { i , j , t } $ Ġscores Ġfrom Ġequation Ġ( SEC REF 16 ) Ġsolely Ġon Ġthe Ġembed ding Ġof Ġthe Ġkey Ġrather Ġthan Ġon Ġthe Ġfull Ġrecord Ġrepresentation Ġ$\ math bf Ġ{ h } _{ i , j } $ : </s> Please Ġnote Ġthat Ġthe Ġdifferent Ġembed d ings Ġand Ġthe Ġmodel Ġparameters Ġpresented Ġin Ġthe Ġmodel Ġcomponents Ġare Ġlearnt Ġusing ĠEqu ation Ġ1 . </s> Exper imental Ġsetup Ġ: :: ĠThe ĠRot ow ire Ġdataset </s> To Ġevaluate Ġthe Ġeffectiveness Ġof Ġour Ġmodel , Ġand Ġdemonstrate Ġits Ġflexibility Ġat Ġhandling Ġheavy Ġdata - st ructure Ġmade Ġof Ġseveral Ġtypes Ġof Ġentities , Ġwe Ġused Ġthe ĠR oto Wire Ġdataset ĠB IB REF 10 . ĠIt Ġincludes Ġbasketball Ġgames Ġstatistical Ġtables Ġpaired Ġwith Ġjournalistic Ġdescriptions Ġof Ġthe Ġgames , Ġas Ġcan Ġbe Ġseen Ġin Ġthe Ġexample Ġof ĠFigure ĠFIG REF 1 . ĠThe Ġdescriptions Ġare Ġprofessionally Ġwritten Ġand Ġaverage Ġ337 Ġwords Ġwith Ġa Ġvocabulary Ġsize Ġof Ġ$ 11 . 3 $ K . ĠThere Ġare Ġ39 Ġdifferent Ġrecord Ġkeys , Ġand Ġthe Ġaverage Ġnumber Ġof Ġrecords Ġ( resp . Ġentities ) Ġin Ġa Ġsingle Ġdata - st ructure Ġis Ġ6 28 Ġ( resp . Ġ28 ). ĠEnt ities Ġare Ġof Ġtwo Ġtypes , Ġeither Ġteam Ġor Ġplayer , Ġand Ġplayer Ġdescriptions Ġdepend Ġon Ġtheir Ġinvolvement Ġin Ġthe Ġgame . ĠWe Ġfollowed Ġthe Ġdata Ġpartitions Ġintroduced Ġwith Ġthe Ġdataset Ġand Ġused Ġa Ġtrain / valid ation / test Ġsets Ġof Ġrespectively Ġ$ 3 , 398 $ / 727 / 728 Ġ( data - st ructure , Ġdescription ) Ġpairs . </s> Exper imental Ġsetup Ġ: :: ĠEvaluation Ġmetrics </s> We Ġevaluate Ġour Ġmodel Ġthrough Ġtwo Ġtypes Ġof Ġmetrics . ĠThe ĠB LE U Ġscore ĠB IB REF 34 Ġaims Ġat Ġmeasuring Ġto Ġwhat Ġextent Ġthe Ġgenerated Ġdescriptions Ġare Ġliterally Ġclosed Ġto Ġthe Ġground Ġtruth . ĠThe Ġsecond Ġcategory Ġdesigned Ġby ĠB IB REF 10 Ġis Ġmore Ġqualitative . </s> Exper imental Ġsetup Ġ: :: ĠEvaluation Ġmetrics Ġ: :: ĠB LE U ĠScore . </s> The ĠB LE U Ġscore ĠB IB REF 34 Ġis Ġcommonly Ġused Ġas Ġan Ġevaluation Ġmetric Ġin Ġtext Ġgeneration Ġtasks . ĠIt Ġestimates Ġthe Ġcorrespondence Ġbetween Ġa Ġmachine Ġoutput Ġand Ġthat Ġof Ġa Ġhuman Ġby Ġcomputing Ġthe Ġnumber Ġof Ġco - occ ur rences Ġfor Ġn gram s Ġ($ n Ġ\ in Ġ{ 1 , Ġ2 , Ġ3 , Ġ4 } $ ) Ġbetween Ġthe Ġgenerated Ġcandidate Ġand Ġthe Ġground Ġtruth . ĠWe Ġuse Ġthe Ġimplementation Ġcode Ġreleased Ġby ĠB IB REF 35 . </s> Exper imental Ġsetup Ġ: :: ĠEvaluation Ġmetrics Ġ: :: ĠInformation Ġextraction - oriented Ġmetrics . </s> These Ġmetrics Ġestimate Ġthe Ġability Ġof Ġour Ġmodel Ġto Ġintegrate Ġelements Ġfrom Ġthe Ġtable Ġin Ġits Ġdescriptions . ĠParticularly , Ġthey Ġcompare Ġthe Ġgold Ġand Ġgenerated Ġdescriptions Ġand Ġmeasure Ġto Ġwhat Ġextent Ġthe Ġextracted Ġrelations Ġare Ġaligned Ġor Ġdiffer . ĠTo Ġdo Ġso , Ġwe Ġfollow Ġthe Ġprotocol Ġpresented Ġin ĠB IB REF 10 . ĠFirst , Ġwe Ġapply Ġan Ġinformation Ġextraction Ġ( IE ) Ġsystem Ġtrained Ġon Ġlabeled Ġrelations Ġfrom Ġthe Ġgold Ġdescriptions Ġof Ġthe ĠR oto Wire Ġtrain Ġdataset . ĠEntity - value Ġpairs Ġare Ġextracted Ġfrom Ġthe Ġdescriptions . ĠFor Ġexample , Ġin Ġthe Ġsentence ĠIsaiah ĠThomas Ġled Ġthe Ġteam Ġin Ġscoring , Ġtotaling Ġ23 Ġpoints Ġ[...] ., Ġan ĠIE Ġtool Ġwill Ġextract Ġthe Ġpair Ġ( Isa iah ĠThomas , Ġ23 , ĠPTS ). ĠSecond , Ġwe Ġcompute Ġthree Ġmetrics Ġon Ġthe Ġextracted Ġinformation : </s> $ \ bul let Ġ$ ĠRel ation ĠGeneration Ġ( RG ) Ġestimates Ġhow Ġwell Ġthe Ġsystem Ġis Ġable Ġto Ġgenerate Ġtext Ġcontaining Ġfactual Ġ( i . e ., Ġcorrect ) Ġrecords . ĠWe Ġmeasure Ġthe Ġprecision Ġand Ġabsolute Ġnumber Ġ( den oted Ġrespectively ĠRG - P % Ġand ĠRG - # ) Ġof Ġunique Ġrelations Ġ$ r $ Ġextracted Ġfrom Ġ$\ hat { y } _{ 1 : T } $ Ġthat Ġalso Ġappear Ġin Ġ$ s $. </s> $ \ bul let Ġ$ ĠContent ĠSelection Ġ( CS ) Ġmeasures Ġhow Ġwell Ġthe Ġgenerated Ġdocument Ġmatches Ġthe Ġgold Ġdocument Ġin Ġterms Ġof Ġmentioned Ġrecords . ĠWe Ġmeasure Ġthe Ġprecision Ġand Ġrecall Ġ( den oted Ġrespectively ĠCS - P % Ġand ĠCS - R %) Ġof Ġunique Ġrelations Ġ$ r $ Ġextracted Ġfrom Ġ$\ hat { y } _{ 1 : T } $ Ġthat Ġare Ġalso Ġextracted Ġfrom Ġ$ y _{ 1 : T } $. </s> $ \ bul let Ġ$ ĠContent ĠOrder ing Ġ( CO ) Ġanaly zes Ġhow Ġwell Ġthe Ġsystem Ġorders Ġthe Ġrecords Ġdiscussed Ġin Ġthe Ġdescription . ĠWe Ġmeasure Ġthe Ġnormalized ĠD amer au - Le ven sh tein Ġdistance ĠB IB REF 36 Ġbetween Ġthe Ġsequences Ġof Ġrecords Ġextracted Ġfrom Ġ$\ hat { y } _{ 1 : T } $ Ġthat Ġare Ġalso Ġextracted Ġfrom Ġ$ y _{ 1 : T } $. </s> CS Ġprimarily Ġtargets Ġthe ĠâĢ ľ what Ġto Ġsay " Ġaspect Ġof Ġevaluation , ĠCO Ġtargets Ġthe ĠâĢ ľ how Ġto Ġsay Ġit " Ġaspect , Ġand ĠRG Ġtargets Ġboth . ĠNote Ġthat Ġfor ĠCS , ĠCO , ĠRG - % Ġand ĠB LE U Ġmetrics , Ġhigher Ġis Ġbetter ; Ġwhich Ġis Ġnot Ġtrue Ġfor ĠRG - # . ĠThe ĠIE Ġsystem Ġused Ġin Ġthe Ġexperiments Ġis Ġable Ġto Ġextract Ġan Ġaverage Ġof Ġ17 Ġfactual Ġrecords Ġfrom Ġgold Ġdescriptions . ĠIn Ġorder Ġto Ġmimic Ġa Ġhuman Ġexpert , Ġa Ġgener ative Ġsystem Ġshould Ġapproach Ġthis Ġnumber Ġand Ġnot Ġoverload Ġgeneration Ġwith Ġbrute Ġfacts . </s> Exper imental Ġsetup Ġ: :: ĠBas elines </s> We Ġcompare Ġour Ġhierarchical Ġmodel Ġagainst Ġthree Ġsystems . ĠFor Ġeach Ġof Ġthem , Ġwe Ġreport Ġthe Ġresults Ġof Ġthe Ġbest Ġperforming Ġmodels Ġpresented Ġin Ġeach Ġpaper . </s> $ \ bul let Ġ$ ĠWis eman ĠB IB REF 10 Ġis Ġa Ġstandard Ġenc oder - dec oder Ġsystem Ġwith Ġcopy Ġmechanism . </s> $ \ bul let Ġ$ ĠLi ĠB IB REF 28 Ġis Ġa Ġstandard Ġenc oder - dec oder Ġwith Ġa Ġdelayed Ġcopy Ġmechanism : Ġtext Ġis Ġfirst Ġgenerated Ġwith Ġplace holders , Ġwhich Ġare Ġreplaced Ġby Ġsalient Ġrecords Ġextracted Ġfrom Ġthe Ġtable Ġby Ġa Ġpointer Ġnetwork . </s> $ \ bul let Ġ$ ĠP ud upp ully - plan ĠB IB REF 12 Ġacts Ġin Ġtwo Ġsteps : Ġa Ġfirst Ġstandard Ġenc oder - dec oder Ġgenerates Ġa Ġplan , Ġi . e . Ġa Ġlist Ġof Ġsalient Ġrecords Ġfrom Ġthe Ġtable ; Ġa Ġsecond Ġstandard Ġenc oder - dec oder Ġgenerates Ġtext Ġfrom Ġthis Ġplan . </s> $ \ bul let Ġ$ ĠP ud upp ully - up dt ĠB IB REF 13 . ĠIt Ġconsists Ġin Ġa Ġstandard Ġenc oder - dec oder , Ġwith Ġan Ġadded Ġmodule Ġaimed Ġat Ġupdating Ġrecord Ġrepresentations Ġduring Ġthe Ġgeneration Ġprocess . ĠAt Ġeach Ġdecoding Ġstep , Ġa Ġg ated Ġrecurrent Ġnetwork Ġcomp utes Ġwhich Ġrecords Ġshould Ġbe Ġupdated Ġand Ġwhat Ġshould Ġbe Ġtheir Ġnew Ġrepresentation . </s> Exper imental Ġsetup Ġ: :: ĠBas elines Ġ: :: ĠModel Ġscenarios </s> We Ġtest Ġthe Ġimportance Ġof Ġthe Ġinput Ġstructure Ġby Ġtraining Ġdifferent Ġvariants Ġof Ġthe Ġproposed Ġarchitecture : </s> $ \ bul let Ġ$ ĠFlat , Ġwhere Ġwe Ġfeed Ġthe Ġinput Ġsequ entially Ġto Ġthe Ġenc oder , Ġlosing Ġall Ġnotion Ġof Ġhierarchy . ĠAs Ġa Ġconsequence , Ġthe Ġmodel Ġuses Ġstandard Ġattention . ĠThis Ġvariant Ġis Ġclosest Ġto ĠWis eman , Ġwith Ġthe Ġexception Ġthat Ġwe Ġuse Ġa ĠTrans former Ġto Ġencode Ġthe Ġinput Ġsequence Ġinstead Ġof Ġan ĠR NN . </s> $ \ bul let Ġ$ ĠHier arch ical - k v Ġis Ġour Ġfull Ġhierarchical Ġmodel , Ġwith Ġtraditional Ġhierarchical Ġattention , Ġi . e . Ġwhere Ġattention Ġover Ġrecords Ġis Ġcomputed Ġon Ġthe Ġfull Ġrecord Ġencoding , Ġas Ġin Ġequation Ġ( SEC REF 16 ). </s> $ \ bul let Ġ$ ĠHier arch ical - k Ġis Ġour Ġfull Ġhierarchical Ġmodel , Ġwith Ġkey - guided Ġhierarchical Ġattention , Ġi . e . Ġwhere Ġattention Ġover Ġrecords Ġis Ġcomputed Ġonly Ġon Ġthe Ġrecord Ġkey Ġrepresentations , Ġas Ġin Ġequation Ġ( DIS PLAY _ FORM 17 ). </s> Exper imental Ġsetup Ġ: :: ĠImplementation Ġdetails </s> The Ġdec oder Ġis Ġthe Ġone Ġused Ġin ĠB IB REF 12 , ĠB IB REF 13 , ĠB IB REF 10 Ġwith Ġthe Ġsame Ġhyper - param eters . ĠFor Ġthe Ġenc oder Ġmodule , Ġboth Ġthe Ġlow - level Ġand Ġhigh - level Ġenc od ers Ġuse Ġa Ġtwo - l ayers Ġmulti - head Ġself - att ention Ġwith Ġtwo Ġheads . ĠTo Ġfit Ġwith Ġthe Ġsmall Ġnumber Ġof Ġrecord Ġkeys Ġin Ġour Ġdataset Ġ( 39 ), Ġtheir Ġembed ding Ġsize Ġis Ġfixed Ġto Ġ20 . ĠThe Ġsize Ġof Ġthe Ġrecord Ġvalue Ġembed d ings Ġand Ġhidden Ġlayers Ġof Ġthe ĠTrans former Ġenc od ers Ġare Ġboth Ġset Ġto Ġ300 . ĠWe Ġuse Ġdrop out Ġat Ġrate Ġ0 . 5 . ĠThe Ġmodels Ġare Ġtrained Ġwith Ġa Ġbatch Ġsize Ġof Ġ64 . ĠWe Ġfollow Ġthe Ġtraining Ġprocedure Ġin ĠB IB REF 21 Ġand Ġtrain Ġthe Ġmodel Ġfor Ġa Ġfixed Ġnumber Ġof Ġ25 K Ġupdates , Ġand Ġaverage Ġthe Ġweights Ġof Ġthe Ġlast Ġ5 Ġcheckpoints Ġ( at Ġevery Ġ1 K Ġupdates ) Ġto Ġensure Ġmore Ġstability Ġacross Ġruns . ĠAll Ġmodels Ġwere Ġtrained Ġwith Ġthe ĠAdam Ġoptim izer ĠB IB REF 37 ; Ġthe Ġinitial Ġlearning Ġrate Ġis Ġ0 . 001 , Ġand Ġis Ġreduced Ġby Ġhalf Ġevery Ġ10 K Ġsteps . ĠWe Ġused Ġbeam Ġsearch Ġwith Ġbeam Ġsize Ġof Ġ5 Ġduring Ġinference . ĠAll Ġthe Ġmodels Ġare Ġimplemented Ġin ĠOpen N MT - py ĠB IB REF 38 . ĠAll Ġcode Ġis Ġavailable Ġat Ġhttps :// github . com / K ai ju ML / data - to - text - h ier arch ical </s> Results </s> Our Ġresults Ġon Ġthe ĠR oto Wire Ġtest set Ġare Ġsummarized Ġin ĠTable ĠT AB REF 25 . ĠFor Ġeach Ġproposed Ġvariant Ġof Ġour Ġarchitecture , Ġwe Ġreport Ġthe Ġmean Ġscore Ġover Ġten Ġruns , Ġas Ġwell Ġas Ġthe Ġstandard Ġdeviation Ġin Ġsubscript . ĠResults Ġare Ġcompared Ġto Ġbas elines ĠB IB REF 12 , ĠB IB REF 13 , ĠB IB REF 10 Ġand Ġvariants Ġof Ġour Ġmodels . ĠWe Ġalso Ġreport Ġthe Ġresult Ġof Ġthe Ġor acle Ġ( met rics Ġon Ġthe Ġgold Ġdescriptions ). ĠPlease Ġnote Ġthat Ġgold Ġdescriptions Ġtriv ially Ġobtain Ġ100 % Ġon Ġall Ġmetrics Ġexpect ĠRG , Ġas Ġthey Ġare Ġall Ġbased Ġon Ġcomparison Ġwith Ġthemselves . ĠRG Ġscores Ġare Ġdifferent , Ġas Ġthe ĠIE Ġsystem Ġis Ġimperfect Ġand Ġfails Ġto Ġextract Ġaccurate Ġentities Ġ4 % Ġof Ġthe Ġtime . ĠRG - # Ġis Ġan Ġabsolute Ġcount . </s> Results Ġ: :: ĠAb lation Ġstudies </s> To Ġevaluate Ġthe Ġimpact Ġof Ġour Ġmodel Ġcomponents , Ġwe Ġfirst Ġcompare Ġscenarios ĠFlat , ĠHier arch ical - k , Ġand ĠHier arch ical - k v . ĠAs Ġshown Ġin ĠTable ĠT AB REF 25 , Ġwe Ġcan Ġsee Ġthe Ġlower Ġresults Ġobtained Ġby Ġthe ĠFlat Ġscenario Ġcompared Ġto Ġthe Ġother Ġscenarios Ġ( e . g . ĠB LE U Ġ$ 16 . 7 $ Ġvs . Ġ$ 17 . 5 $ Ġfor Ġresp . ĠFlat Ġand ĠHier arch ical - k ), Ġsuggesting Ġthe Ġeffectiveness Ġof Ġencoding Ġthe Ġdata - st ructure Ġusing Ġa Ġhierarchy . ĠThis Ġis Ġexpected , Ġas Ġlosing Ġexplicit Ġdelim itation Ġbetween Ġentities Ġmakes Ġit Ġharder Ġa ) Ġfor Ġthe Ġenc oder Ġto Ġencode Ġsemantics Ġof Ġthe Ġobjects Ġcontained Ġin Ġthe Ġtable Ġand Ġb ) Ġfor Ġthe Ġattention Ġmechanism Ġto Ġextract Ġsalient Ġentities / rec ords . </s> Second , Ġthe Ġcomparison Ġbetween Ġscenario ĠHier arch ical - k v Ġand ĠHier arch ical - k Ġshows Ġthat Ġo mitting Ġentirely Ġthe Ġinfluence Ġof Ġthe Ġrecord Ġvalues Ġin Ġthe Ġattention Ġmechanism Ġis Ġmore Ġeffective : Ġthis Ġlast Ġvariant Ġperforms Ġslightly Ġbetter Ġin Ġall Ġmetrics Ġexcept ed ĠCS - R %, Ġreinforcing Ġour Ġintuition Ġthat Ġfocusing Ġon Ġthe Ġstructure Ġmodeling Ġis Ġan Ġimportant Ġpart Ġof Ġdata Ġencoding Ġas Ġwell Ġas Ġconfirming Ġthe Ġintuition Ġexplained Ġin ĠSection ĠSEC REF 16 : Ġonce Ġan Ġentity Ġis Ġselected , Ġfacts Ġabout Ġthis Ġentity Ġare Ġrelevant Ġbased Ġon Ġtheir Ġkey , Ġnot Ġvalue Ġwhich Ġmight Ġadd Ġnoise . ĠTo Ġillustrate Ġthis Ġintuition , Ġwe Ġdepict Ġin ĠFigure ĠFIG REF 27 Ġattention Ġscores Ġ( rec all Ġ$\ alpha Ġ_ { i , t } $ Ġand Ġ$\ beta Ġ_ { i , j , t } $ Ġfrom Ġequations Ġ( SEC REF 16 ) Ġand Ġ( DIS PLAY _ FORM 17 )) Ġfor Ġboth Ġvariants ĠHier arch ical - k v Ġand ĠHier arch ical - k . ĠWe Ġparticularly Ġfocus Ġon Ġthe Ġtimestamp Ġwhere Ġthe Ġmodels Ġshould Ġmention Ġthe Ġnumber Ġof Ġpoints Ġscored Ġduring Ġthe Ġfirst Ġquarter Ġof Ġthe Ġgame . ĠScores Ġof ĠHier arch ical - k Ġare Ġsharp , Ġwith Ġall Ġof Ġthe Ġweight Ġon Ġthe Ġcorrect Ġrecord Ġ( P TS _ Q TR 1 , Ġ26 ) Ġwhereas Ġscores Ġof ĠHier arch ical - k v Ġare Ġmore Ġdistributed Ġover Ġall ĠPTS _ Q TR Ġrecords , Ġultimately Ġfailing Ġto Ġretrieve Ġthe Ġcorrect Ġone . </s> Results Ġ: :: ĠComparison Ġw . r . t . Ġbas elines . </s> From Ġa Ġgeneral Ġpoint Ġof Ġview , Ġwe Ġcan Ġsee Ġfrom ĠTable ĠT AB REF 25 Ġthat Ġour Ġscenarios Ġobtain Ġsignificantly Ġhigher Ġresults Ġin Ġterms Ġof ĠB LE U Ġover Ġall Ġmodels ; Ġour Ġbest Ġmodel ĠHier arch ical - k Ġreaching Ġ$ 17 . 5 $ Ġvs . Ġ$ 16 . 5 $ Ġagainst Ġthe Ġbest Ġbaseline . ĠThis Ġmeans Ġthat Ġour Ġmodels Ġlearns Ġto Ġgenerate Ġfluent Ġsequences Ġof Ġwords , Ġclose Ġto Ġthe Ġgold Ġdescriptions , Ġadequately Ġpicking Ġup Ġon Ġdomain Ġling o . ĠQual itative Ġmetrics Ġare Ġeither Ġbetter Ġor Ġon Ġpar Ġwith Ġbas elines . ĠWe Ġshow Ġin ĠFigure ĠFIG REF 29 Ġa Ġtext Ġgenerated Ġby Ġour Ġbest Ġmodel , Ġwhich Ġcan Ġbe Ġdirectly Ġcompared Ġto Ġthe Ġgold Ġdescription Ġin ĠFigure ĠFIG REF 1 . ĠGeneration Ġis Ġfluent Ġand Ġcontains Ġdomain - specific Ġexpressions . ĠAs Ġreflected Ġin ĠTable ĠT AB REF 25 , Ġthe Ġnumber Ġof Ġcorrect Ġmentions Ġ( in Ġgreen ) Ġoutwe ights Ġthe Ġnumber Ġof Ġincorrect Ġmentions Ġ( in Ġred ). ĠPlease Ġnote Ġthat , Ġas Ġin Ġprevious Ġwork ĠB IB REF 28 , ĠB IB REF 12 , ĠB IB REF 13 , ĠB IB REF 10 , Ġgenerated Ġtexts Ġstill Ġcontain Ġa Ġnumber Ġof Ġincorrect Ġfacts , Ġas Ġwell Ġhallucinations Ġ( in Ġblue ): Ġsentences Ġthat Ġhave Ġno Ġbasis Ġin Ġthe Ġinput Ġdata Ġ( e . g . ĠâĢ ľ [ ...] Ġhe 's Ġnow Ġaveraging Ġ22 Ġpoints Ġ[...] ." ). ĠWhile Ġnot Ġthe Ġdirect Ġfocus Ġof Ġour Ġwork , Ġthis Ġhighlights Ġthat Ġany Ġoperation Ġmeant Ġto Ġenrich Ġthe Ġsemantics Ġof Ġstructured Ġdata Ġcan Ġalso Ġenrich Ġthe Ġdata Ġwith Ġincorrect Ġfacts . </s> Specifically , Ġregarding Ġall Ġbas elines , Ġwe Ġcan Ġoutline Ġthe Ġfollowing Ġstatements . </s> $ \ bul let Ġ$ ĠOur Ġhierarchical Ġmodels Ġachieve Ġsignificantly Ġbetter Ġscores Ġon Ġall Ġmetrics Ġwhen Ġcompared Ġto Ġthe Ġflat Ġarchitecture ĠWis eman , Ġreinforcing Ġthe Ġcrucial Ġrole Ġof Ġstructure Ġin Ġdata Ġsemantics Ġand Ġsal iency . ĠThe Ġanalysis Ġof ĠRG Ġmetrics Ġshows Ġthat ĠWis eman Ġseems Ġto Ġbe Ġthe Ġmore Ġnatural istic Ġin Ġterms Ġof Ġnumber Ġof Ġfactual Ġmentions Ġ( RG # ) Ġsince Ġit Ġis Ġthe Ġclosest Ġscenario Ġto Ġthe Ġgold Ġvalue Ġ( 16 . 83 Ġvs . Ġ17 . 31 Ġfor Ġresp . ĠWis eman Ġand ĠHier arch ical - k ). ĠHowever , ĠWis eman Ġachieves Ġonly Ġ$ 75 . 62 $ % Ġof Ġprecision , Ġeffectively Ġmentioning Ġon Ġaverage Ġa Ġtotal Ġof Ġ$ 22 . 25 $ Ġrecords Ġ( wrong Ġor Ġaccurate ), Ġwhere Ġour Ġmodel ĠHier arch ical - k Ġscores Ġa Ġprecision Ġof Ġ$ 89 . 46 $ %, Ġleading Ġto Ġ$ 23 . 66 $ Ġtotal Ġmentions , Ġjust Ġslightly Ġabove ĠWis eman . </s> $ \ bul let Ġ$ ĠThe Ġcomparison Ġbetween Ġthe ĠFlat Ġscenario Ġand ĠWis eman Ġis Ġparticularly Ġinteresting . ĠIndeed , Ġthese Ġtwo Ġmodels Ġshare Ġthe Ġsame Ġintuition Ġto Ġflatt en Ġthe Ġdata - st ructure . ĠThe Ġonly Ġdifference Ġstands Ġon Ġthe Ġenc oder Ġmechanism : Ġbi - L ST M Ġvs . ĠTrans former , Ġfor ĠWis eman Ġand ĠFlat Ġrespectively . ĠResults Ġshows Ġthat Ġour ĠFlat Ġscenario Ġob tains Ġa Ġsignificant Ġhigher ĠB LE U Ġscore Ġ( 16 . 7 Ġvs . Ġ14 . 5 ) Ġand Ġgenerates Ġfluent Ġdescriptions Ġwith Ġaccurate Ġmentions Ġ( RG - P %) Ġthat Ġare Ġalso Ġincluded Ġin Ġthe Ġgold Ġdescriptions Ġ( CS - R %). ĠThis Ġsuggests Ġthat Ġintroducing Ġthe ĠTrans former Ġarchitecture Ġis Ġpromising Ġway Ġto Ġimplicitly Ġaccount Ġfor Ġdata Ġstructure . </s> $ \ bul let Ġ$ ĠOur Ġhierarchical Ġmodels Ġoutper form Ġthe Ġtwo - step Ġdec od ers Ġof ĠLi Ġand ĠP ud upp ully - plan Ġon Ġboth ĠB LE U Ġand Ġall Ġqualitative Ġmetrics , Ġshowing Ġthat Ġcapturing Ġstructure Ġin Ġthe Ġencoding Ġprocess Ġis Ġmore Ġeffective Ġthat Ġpredicting Ġa Ġstructure Ġin Ġthe Ġdec oder Ġ( i . e ., Ġplanning Ġor Ġtem pl ating ). ĠWhile Ġour Ġmodels Ġsens ibly Ġoutper form Ġin Ġprecision Ġat Ġfactual Ġmentions , Ġthe Ġbaseline ĠP ud upp ully - plan Ġreaches Ġ$ 34 . 28 $ Ġmentions Ġon Ġaverage , Ġshowing Ġthat Ġincorporating Ġmodules Ġdedicated Ġto Ġentity Ġextraction Ġleads Ġto Ġover - f ocusing Ġon Ġentities ; Ġcontrasting Ġwith Ġour Ġmodels Ġthat Ġlearn Ġto Ġgenerate Ġmore Ġbalanced Ġdescriptions . </s> $ \ bul let Ġ$ ĠThe Ġcomparison Ġwith ĠP ud upp ully - up dt Ġshows Ġthat Ġdynamically Ġupdating Ġthe Ġencoding Ġacross Ġthe Ġgeneration Ġprocess Ġcan Ġlead Ġto Ġbetter ĠContent ĠOrder ing Ġ( CO ) Ġand ĠRG - P %. ĠHowever , Ġthis Ġdoes Ġnot Ġhelp Ġwith ĠContent ĠSelection Ġ( CS ) Ġsince Ġour Ġbest Ġmodel ĠHier arch ical - k Ġob tains Ġslightly Ġbetter Ġscores . ĠIndeed , ĠP ud upp ully - up dt Ġupdates Ġrepresentations Ġafter Ġeach Ġmention Ġallowing Ġto Ġkeep Ġtrack Ġof Ġthe Ġmention Ġhistory . ĠThis Ġguides Ġthe Ġordering Ġof Ġmentions Ġ( CO Ġmetric ), Ġeach Ġstep Ġlimiting Ġmore Ġthe Ġnumber Ġof Ġcandidate Ġmentions Ġ( increasing ĠRG - P %). ĠIn Ġcontrast , Ġour Ġmodel Ġenc odes Ġsal iency Ġamong Ġrecords / ent ities Ġmore Ġeffectively Ġ( CS Ġmetric ). ĠWe Ġnote Ġthat Ġwhile Ġour Ġmodel Ġenc odes Ġthe Ġdata - st ructure Ġonce Ġand Ġfor Ġall , ĠP ud upp ully - up dt Ġrecomp utes , Ġvia Ġthe Ġupdates , Ġthe Ġencoding Ġat Ġeach Ġstep Ġand Ġtherefore Ġsignificantly Ġincreases Ġcomputation Ġcomplexity . ĠCombined Ġwith Ġtheir ĠRG - # Ġscore Ġof Ġ$ 30 . 11 $, Ġwe Ġargue Ġthat Ġour Ġmodel Ġis Ġsimpler , Ġand Ġob tains Ġfluent Ġdescription Ġwith Ġaccurate Ġmentions Ġin Ġa Ġmore Ġhuman - like Ġfashion . </s> We Ġwould Ġalso Ġlike Ġto Ġdraw Ġattention Ġto Ġthe Ġnumber Ġof Ġparameters Ġused Ġby Ġthose Ġarchitectures . ĠWe Ġnote Ġthat Ġour Ġscenarios Ġrelies Ġon Ġa Ġlower Ġnumber Ġof Ġparameters Ġ( 14 Ġmillions ) Ġcompared Ġto Ġall Ġbas elines Ġ( ranging Ġfrom Ġ23 Ġto Ġ45 Ġmillions ). ĠThis Ġoutlines Ġthe Ġeffectiveness Ġin Ġthe Ġdesign Ġof Ġour Ġmodel Ġrelying Ġon Ġa Ġstructure Ġencoding , Ġin Ġcontrast Ġto Ġother Ġapproach Ġthat Ġtry Ġto Ġlearn Ġthe Ġstructure Ġof Ġdata / desc ript ions Ġfrom Ġa Ġlinear ized Ġencoding . </s> Conclusion Ġand Ġfuture Ġwork </s> In Ġthis Ġwork Ġwe Ġhave Ġproposed Ġa Ġhierarchical Ġenc oder Ġfor Ġstructured Ġdata , Ġwhich Ġ1 ) Ġlever ages Ġthe Ġstructure Ġto Ġform Ġefficient Ġrepresentation Ġof Ġits Ġinput ; Ġ2 ) Ġhas Ġstrong Ġsynergy Ġwith Ġthe Ġhierarchical Ġattention Ġof Ġits Ġassociated Ġdec oder . ĠThis Ġresults Ġin Ġan Ġeffective Ġand Ġmore Ġlight - weight Ġmodel . ĠExperimental Ġevaluation Ġon Ġthe ĠR oto Wire Ġbenchmark Ġshows Ġthat Ġour Ġmodel Ġoutper forms Ġcompetitive Ġbas elines Ġin Ġterms Ġof ĠB LE U Ġscore Ġand Ġis Ġgenerally Ġbetter Ġon Ġqualitative Ġmetrics . ĠThis Ġway Ġof Ġrepresenting Ġstructured Ġdatabases Ġmay Ġlead Ġto Ġautomatic Ġinference Ġand Ġenrichment , Ġe . g ., Ġby Ġcomparing Ġentities . ĠThis Ġdirection Ġcould Ġbe Ġdriven Ġby Ġvery Ġrecent Ġoperation - guided Ġnetworks ĠB IB REF 39 , ĠB IB REF 40 . ĠIn Ġaddition , Ġwe Ġnote Ġthat Ġour Ġapproach Ġcan Ġstill Ġlead Ġto Ġerroneous Ġfacts Ġor Ġeven Ġhallucinations . ĠAn Ġinteresting Ġperspective Ġmight Ġbe Ġto Ġfurther Ġconst rain Ġthe Ġmodel Ġon Ġthe Ġdata Ġstructure Ġin Ġorder Ġto Ġprevent Ġinaccurate Ġof Ġeven Ġcontradictory Ġdescriptions . </s> Acknowled gements </s> We Ġwould Ġlike Ġto Ġthank Ġthe ĠH 2020 Ġproject ĠAI 4 EU Ġ( 8 256 19 ) Ġwhich Ġpartially Ġsupports ĠLaure ĠSou lier Ġand ĠPatrick ĠGall in ari .
Introduction Knowledge and/or data is often modeled in a structure, such as indexes, tables, key-value pairs, or triplets. These data, by their nature (e.g., raw data or long time-series data), are not easily usable by humans; outlining their crucial need to be synthesized. Recently, numerous works have focused on leveraging structured data in various applications, such as question answering BIBREF0, BIBREF1 or table retrieval BIBREF2, BIBREF3. One emerging research field consists in transcribing data-structures into natural language in order to ease their understandablity and their usablity. This field is referred to as “data-to-text" BIBREF4 and has its place in several application domains (such as journalism BIBREF5 or medical diagnosis BIBREF6) or wide-audience applications (such as financial BIBREF7 and weather reports BIBREF8, or sport broadcasting BIBREF9, BIBREF10). As an example, Figure FIGREF1 shows a data-structure containing statistics on NBA basketball games, paired with its corresponding journalistic description. Designing data-to-text models gives rise to two main challenges: 1) understanding structured data and 2) generating associated descriptions. Recent data-to-text models BIBREF11, BIBREF12, BIBREF13, BIBREF10 mostly rely on an encoder-decoder architecture BIBREF14 in which the data-structure is first encoded sequentially into a fixed-size vectorial representation by an encoder. Then, a decoder generates words conditioned on this representation. With the introduction of the attention mechanism BIBREF15 on one hand, which computes a context focused on important elements from the input at each decoding step and, on the other hand, the copy mechanism BIBREF16, BIBREF17 to deal with unknown or rare words, these systems produce fluent and domain comprehensive texts. For instance, Roberti et al. BIBREF18 train a character-wise encoder-decoder to generate descriptions of restaurants based on their attributes, while Puduppully et al. BIBREF12 design a more complex two-step decoder: they first generate a plan of elements to be mentioned, and then condition text generation on this plan. Although previous work yield overall good results, we identify two important caveats, that hinder precision (i.e. factual mentions) in the descriptions: Linearization of the data-structure. In practice, most works focus on introducing innovating decoding modules, and still represent data as a unique sequence of elements to be encoded. For example, the table from Figure FIGREF1 would be linearized to [(Hawks, H/V, H), ..., (Magic, H/V, V), ...], effectively leading to losing distinction between rows, and therefore entities. To the best of our knowledge, only Liu et al. BIBREF19, BIBREF11 propose encoders constrained by the structure but these approaches are designed for single-entity structures. Arbitrary ordering of unordered collections in recurrent networks (RNN). Most data-to-text systems use RNNs as encoders (such as GRUs or LSTMs), these architectures have however some limitations. Indeed, they require in practice their input to be fed sequentially. This way of encoding unordered sequences (i.e. collections of entities) implicitly assumes an arbitrary order within the collection which, as demonstrated by Vinyals et al. BIBREF20, significantly impacts the learning performance. To address these shortcomings, we propose a new structured-data encoder assuming that structures should be hierarchically captured. Our contribution focuses on the encoding of the data-structure, thus the decoder is chosen to be a classical module as used in BIBREF12, BIBREF10. Our contribution is threefold: We model the general structure of the data using a two-level architecture, first encoding all entities on the basis of their elements, then encoding the data structure on the basis of its entities; We introduce the Transformer encoder BIBREF21 in data-to-text models to ensure robust encoding of each element/entities in comparison to all others, no matter their initial positioning; We integrate a hierarchical attention mechanism to compute the hierarchical context fed into the decoder. We report experiments on the RotoWire benchmark BIBREF10 which contains around $5K$ statistical tables of NBA basketball games paired with human-written descriptions. Our model is compared to several state-of-the-art models. Results show that the proposed architecture outperforms previous models on BLEU score and is generally better on qualitative metrics. In the following, we first present a state-of-the art of data-to-text literature (Section 2), and then describe our proposed hierarchical data encoder (Section 3). The evaluation protocol is presented in Section 4, followed by the results (Section 5). Section 6 concludes the paper and presents perspectives. Related Work Until recently, efforts to bring out semantics from structured-data relied heavily on expert knowledge BIBREF22, BIBREF8. For example, in order to better transcribe numerical time series of weather data to a textual forecast, Reiter et al. BIBREF8 devise complex template schemes in collaboration with weather experts to build a consistent set of data-to-word rules. Modern approaches to the wide range of tasks based on structured-data (e.g. table retrieval BIBREF2, BIBREF23, table classification BIBREF24, question answering BIBREF25) now propose to leverage progress in deep learning to represent these data into a semantic vector space (also called embedding space). In parallel, an emerging task, called “data-to-text", aims at describing structured data into a natural language description. This task stems from the neural machine translation (NMT) domain, and early work BIBREF26, BIBREF27, BIBREF10 represent the data records as a single sequence of facts to be entirely translated into natural language. Wiseman et al. BIBREF10 show the limits of traditional NMT systems on larger structured-data, where NMT systems fail to accurately extract salient elements. To improve these models, a number of work BIBREF28, BIBREF12, BIBREF29 proposed innovating decoding modules based on planning and templates, to ensure factual and coherent mentions of records in generated descriptions. For example, Puduppully et al. BIBREF12 propose a two-step decoder which first targets specific records and then use them as a plan for the actual text generation. Similarly, Li et al. BIBREF28 proposed a delayed copy mechanism where their decoder also acts in two steps: 1) using a classical LSTM decoder to generate delexicalized text and 2) using a pointer network BIBREF30 to replace placeholders by records from the input data. Closer to our work, very recent work BIBREF11, BIBREF19, BIBREF13 have proposed to take into account the data structure. More particularly, Puduppully et al. BIBREF13 follow entity-centric theories BIBREF31, BIBREF32 and propose a model based on dynamic entity representation at decoding time. It consists in conditioning the decoder on entity representations that are updated during inference at each decoding step. On the other hand, Liu et al. BIBREF11, BIBREF19 rather focus on introducing structure into the encoder. For instance, they propose a dual encoder BIBREF19 which encodes separately the sequence of element names and the sequence of element values. These approaches are however designed for single-entity data structures and do not account for delimitation between entities. Our contribution differs from previous work in several aspects. First, instead of flatly concatenating elements from the data-structure and encoding them as a sequence BIBREF11, BIBREF12, BIBREF10, we constrain the encoding to the underlying structure of the input data, so that the delimitation between entities remains clear throughout the process. Second, unlike all works in the domain, we exploit the Transformer architecture BIBREF21 and leverage its particularity to directly compare elements with each others in order to avoid arbitrary assumptions on their ordering. Finally, in contrast to BIBREF33, BIBREF13 that use a complex updating mechanism to obtain a dynamic representation of the input data and its entities, we argue that explicit hierarchical encoding naturally guides the decoding process via hierarchical attention. Hierarchical Encoder Model for Data-to-Text In this section we introduce our proposed hierarchical model taking into account the data structure. We outline that the decoding component aiming to generate descriptions is considered as a black-box module so that our contribution is focused on the encoding module. We first describe the model overview, before detailing the hierarchical encoder and the associated hierarchical attention. Hierarchical Encoder Model for Data-to-Text ::: Notation and General Overview Let's consider the following notations: $\bullet $ An entity $e_i$ is a set of $J_i$ unordered records $\lbrace r_{i,1}, ..., r_{i,j}, ..., r_{i,J_i}\rbrace $; where record $r_{i,j}$ is defined as a pair of key $k_{i,j}$ and value $v_{i,j}$. We outline that $J_i$ might differ between entities. $\bullet $ A data-structure $s$ is an unordered set of $I$ entities $e_i$. We thus denote $s \lbrace e_1, ..., e_i, ..., e_I\rbrace $. $\bullet $ For each data-structure, a textual description $y$ is associated. We refer to the first $t$ words of a description $y$ as $y_{1:t}$. Thus, the full sequence of words can be noted as $y = y_{1:T}$. $\bullet $ The dataset $\mathcal {D}$ is a collection of $N$ aligned (data-structure, description) pairs $(s,y)$. For instance, Figure FIGREF1 illustrates a data-structure associated with a description. The data-structure includes a set of entities (Hawks, Magic, Al Horford, Jeff Teague, ...). The entity Jeff Teague is modeled as a set of records {(PTS, 17), (REB, 0), (AST, 7) ...} in which, e.g., the record (PTS, 17) is characterized by a key (PTS) and a value (17). For each data-structure $s$ in $\mathcal {D}$, the objective function aims to generate a description $\hat{y}$ as close as possible to the ground truth $y$. This objective function optimizes the following log-likelihood over the whole dataset $\mathcal {D}$: where $\theta $ stands for the model parameters and $P(\hat{y}=y\ |\ s; \theta )$ the probability of the model to generate the adequate description $y$ for table $s$. During inference, we generate the sequence $\hat{y}^*$ with the maximum a posteriori probability conditioned on table $s$. Using the chain rule, we get: This equation is intractable in practice, we approximate a solution using beam search, as in BIBREF11, BIBREF19, BIBREF12, BIBREF13, BIBREF10. Our model follows the encoder-decoder architecture BIBREF14. Because our contribution focuses on the encoding process, we chose the decoding module used in BIBREF12, BIBREF10: a two-layers LSTM network with a copy mechanism. In order to supervise this mechanism, we assume that each record value that also appears in the target is copied from the data-structure and we train the model to switch between freely generating words from the vocabulary and copying words from the input. We now describe the hierarchical encoder and the hierarchical attention. Hierarchical Encoder Model for Data-to-Text ::: Hierarchical Encoding Model As outlined in Section SECREF2, most previous work BIBREF28, BIBREF12, BIBREF13, BIBREF10, BIBREF29 make use of flat encoders that do not exploit the data structure. To keep the semantics of each element from the data-structure, we propose a hierarchical encoder which relies on two modules. The first one (module A in Figure FIGREF11) is called low-level encoder and encodes entities on the basis of their records; the second one (module B), called high-level encoder, encodes the data-structure on the basis of its underlying entities. In the low-level encoder, the traditional embedding layer is replaced by a record embedding layer as in BIBREF11, BIBREF12, BIBREF10. We present in what follows the record embedding layer and introduce our two hierarchical modules. Hierarchical Encoder Model for Data-to-Text ::: Hierarchical Encoding Model ::: Record Embedding Layer. The first layer of the network consists in learning two embedding matrices to embed the record keys and values. Keys $k_{i,j}$ are embedded to $\mathbf {k}_{i,j} \in \mathbb {R}^{d}$ and values $v_{i,j}$ to $\mathbf {v}_{i,j} \in \mathbb {R}^{d}$, with $d$ the size of the embedding. As in previous work BIBREF11, BIBREF12, BIBREF10, each record embedding $\mathbf {r}_{i,j}$ is computed by a linear projection on the concatenation $[\mathbf {k}_{i,j}$; $\mathbf {v}_{i,j}]$ followed by a non linearity: where $\mathbf {W}_r \in \mathbb {R}^{2d \times d}$ and $\mathbf {b}_r \in \mathbb {R}^{d}$ are learnt parameters. The low-level encoder aims at encoding a collection of records belonging to the same entity while the high-level encoder encodes the whole set of entities. Both the low-level and high-level encoders consider their input elements as unordered. We use the Transformer architecture from BIBREF21. For each encoder, we have the following peculiarities: the Low-level encoder encodes each entity $e_i$ on the basis of its record embeddings $\mathbf {r}_{i,j}$. Each record embedding $\mathbf {r}_{i,j}$ is compared to other record embeddings to learn its final hidden representation $\mathbf {h}_{i,j}$. Furthermore, we add a special record [ENT] for each entity, illustrated in Figure FIGREF11 as the last record. Since entities might have a variable number of records, this token allows to aggregate final hidden record representations $\lbrace \mathbf {h}_{i,j}\rbrace _{j=1}^{J_i}$ in a fixed-sized representation vector $\mathbf {h}_{i}$. the High-level encoder encodes the data-structure on the basis of its entity representation $\mathbf {h}_{i}$. Similarly to the Low-level encoder, the final hidden state $\mathbf {e_i}$ of an entity is computed by comparing entity representation $\mathbf {h}_{i}$ with each others. The data-structure representation $\mathbf {z}$ is computed as the mean of these entity representations, and is used for the decoder initialization. Hierarchical Encoder Model for Data-to-Text ::: Hierarchical attention To fully leverage the hierarchical structure of our encoder, we propose two variants of hierarchical attention mechanism to compute the context fed to the decoder module. $\bullet $ Traditional Hierarchical Attention. As in BIBREF13, we hypothesize that a dynamic context should be computed in two steps: first attending to entities, then to records corresponding to these entities. To implement this hierarchical attention, at each decoding step $t$, the model learns a first set of attention scores $\alpha _{i,t}$ over entities $e_i$ and a second set of attention scores $\beta _{i,j,t}$ over records $r_{i,j}$ belonging to entity $e_i$. The $\alpha _{i,t}$ scores are normalized to form a distribution over all entities $e_i$, and $\beta _{i,j,t}$ scores are normalized to form a distribution over records $r_{i,j}$ of entity $e_i$. Each entity is then represented as a weighted sum of its record embeddings, and the entire data structure is represented as a weighted sum of the entity representations. The dynamic context is computed as: ct = i=1I (i,t ( j i,j,t ri,j )) where i,t exp(dtWei) and i,j,t exp(dtWhi,j) where $\mathbf {d_t}$ is the decoder hidden state at time step $t$, $\mathbf {W}_{\alpha } \in \mathbb {R}^{d\times d}$ and $\mathbf {W}_{\beta } \in \mathbb {R}^{d\times d}$ are learnt parameters, $ \sum _i\alpha _{i,t} = 1$, and for all $i \in \lbrace 1,...,I\rbrace $ $\sum _{j}\beta _{i,j,t} = 1$. $\bullet $ Key-guided Hierarchical Attention. This variant follows the intuition that once an entity is chosen for mention (thanks to $\alpha _{i,t}$), only the type of records is important to determine the content of the description. For example, when deciding to mention a player, all experts automatically report his score without consideration of its specific value. To test this intuition, we model the attention scores by computing the $\beta _{i,j,t}$ scores from equation (SECREF16) solely on the embedding of the key rather than on the full record representation $\mathbf {h}_{i,j}$: Please note that the different embeddings and the model parameters presented in the model components are learnt using Equation 1. Experimental setup ::: The Rotowire dataset To evaluate the effectiveness of our model, and demonstrate its flexibility at handling heavy data-structure made of several types of entities, we used the RotoWire dataset BIBREF10. It includes basketball games statistical tables paired with journalistic descriptions of the games, as can be seen in the example of Figure FIGREF1. The descriptions are professionally written and average 337 words with a vocabulary size of $11.3$K. There are 39 different record keys, and the average number of records (resp. entities) in a single data-structure is 628 (resp. 28). Entities are of two types, either team or player, and player descriptions depend on their involvement in the game. We followed the data partitions introduced with the dataset and used a train/validation/test sets of respectively $3,398$/727/728 (data-structure, description) pairs. Experimental setup ::: Evaluation metrics We evaluate our model through two types of metrics. The BLEU score BIBREF34 aims at measuring to what extent the generated descriptions are literally closed to the ground truth. The second category designed by BIBREF10 is more qualitative. Experimental setup ::: Evaluation metrics ::: BLEU Score. The BLEU score BIBREF34 is commonly used as an evaluation metric in text generation tasks. It estimates the correspondence between a machine output and that of a human by computing the number of co-occurrences for ngrams ($n \in {1, 2, 3, 4}$) between the generated candidate and the ground truth. We use the implementation code released by BIBREF35. Experimental setup ::: Evaluation metrics ::: Information extraction-oriented metrics. These metrics estimate the ability of our model to integrate elements from the table in its descriptions. Particularly, they compare the gold and generated descriptions and measure to what extent the extracted relations are aligned or differ. To do so, we follow the protocol presented in BIBREF10. First, we apply an information extraction (IE) system trained on labeled relations from the gold descriptions of the RotoWire train dataset. Entity-value pairs are extracted from the descriptions. For example, in the sentence Isaiah Thomas led the team in scoring, totaling 23 points [...]., an IE tool will extract the pair (Isaiah Thomas, 23, PTS). Second, we compute three metrics on the extracted information: $\bullet $ Relation Generation (RG) estimates how well the system is able to generate text containing factual (i.e., correct) records. We measure the precision and absolute number (denoted respectively RG-P% and RG-#) of unique relations $r$ extracted from $\hat{y}_{1:T}$ that also appear in $s$. $\bullet $ Content Selection (CS) measures how well the generated document matches the gold document in terms of mentioned records. We measure the precision and recall (denoted respectively CS-P% and CS-R%) of unique relations $r$ extracted from $\hat{y}_{1:T}$ that are also extracted from $y_{1:T}$. $\bullet $ Content Ordering (CO) analyzes how well the system orders the records discussed in the description. We measure the normalized Damerau-Levenshtein distance BIBREF36 between the sequences of records extracted from $\hat{y}_{1:T}$ that are also extracted from $y_{1:T}$. CS primarily targets the “what to say" aspect of evaluation, CO targets the “how to say it" aspect, and RG targets both. Note that for CS, CO, RG-% and BLEU metrics, higher is better; which is not true for RG-#. The IE system used in the experiments is able to extract an average of 17 factual records from gold descriptions. In order to mimic a human expert, a generative system should approach this number and not overload generation with brute facts. Experimental setup ::: Baselines We compare our hierarchical model against three systems. For each of them, we report the results of the best performing models presented in each paper. $\bullet $ Wiseman BIBREF10 is a standard encoder-decoder system with copy mechanism. $\bullet $ Li BIBREF28 is a standard encoder-decoder with a delayed copy mechanism: text is first generated with placeholders, which are replaced by salient records extracted from the table by a pointer network. $\bullet $ Puduppully-plan BIBREF12 acts in two steps: a first standard encoder-decoder generates a plan, i.e. a list of salient records from the table; a second standard encoder-decoder generates text from this plan. $\bullet $ Puduppully-updt BIBREF13. It consists in a standard encoder-decoder, with an added module aimed at updating record representations during the generation process. At each decoding step, a gated recurrent network computes which records should be updated and what should be their new representation. Experimental setup ::: Baselines ::: Model scenarios We test the importance of the input structure by training different variants of the proposed architecture: $\bullet $ Flat, where we feed the input sequentially to the encoder, losing all notion of hierarchy. As a consequence, the model uses standard attention. This variant is closest to Wiseman, with the exception that we use a Transformer to encode the input sequence instead of an RNN. $\bullet $ Hierarchical-kv is our full hierarchical model, with traditional hierarchical attention, i.e. where attention over records is computed on the full record encoding, as in equation (SECREF16). $\bullet $ Hierarchical-k is our full hierarchical model, with key-guided hierarchical attention, i.e. where attention over records is computed only on the record key representations, as in equation (DISPLAY_FORM17). Experimental setup ::: Implementation details The decoder is the one used in BIBREF12, BIBREF13, BIBREF10 with the same hyper-parameters. For the encoder module, both the low-level and high-level encoders use a two-layers multi-head self-attention with two heads. To fit with the small number of record keys in our dataset (39), their embedding size is fixed to 20. The size of the record value embeddings and hidden layers of the Transformer encoders are both set to 300. We use dropout at rate 0.5. The models are trained with a batch size of 64. We follow the training procedure in BIBREF21 and train the model for a fixed number of 25K updates, and average the weights of the last 5 checkpoints (at every 1K updates) to ensure more stability across runs. All models were trained with the Adam optimizer BIBREF37; the initial learning rate is 0.001, and is reduced by half every 10K steps. We used beam search with beam size of 5 during inference. All the models are implemented in OpenNMT-py BIBREF38. All code is available at https://github.com/KaijuML/data-to-text-hierarchical Results Our results on the RotoWire testset are summarized in Table TABREF25. For each proposed variant of our architecture, we report the mean score over ten runs, as well as the standard deviation in subscript. Results are compared to baselines BIBREF12, BIBREF13, BIBREF10 and variants of our models. We also report the result of the oracle (metrics on the gold descriptions). Please note that gold descriptions trivially obtain 100% on all metrics expect RG, as they are all based on comparison with themselves. RG scores are different, as the IE system is imperfect and fails to extract accurate entities 4% of the time. RG-# is an absolute count. Results ::: Ablation studies To evaluate the impact of our model components, we first compare scenarios Flat, Hierarchical-k, and Hierarchical-kv. As shown in Table TABREF25, we can see the lower results obtained by the Flat scenario compared to the other scenarios (e.g. BLEU $16.7$ vs. $17.5$ for resp. Flat and Hierarchical-k), suggesting the effectiveness of encoding the data-structure using a hierarchy. This is expected, as losing explicit delimitation between entities makes it harder a) for the encoder to encode semantics of the objects contained in the table and b) for the attention mechanism to extract salient entities/records. Second, the comparison between scenario Hierarchical-kv and Hierarchical-k shows that omitting entirely the influence of the record values in the attention mechanism is more effective: this last variant performs slightly better in all metrics excepted CS-R%, reinforcing our intuition that focusing on the structure modeling is an important part of data encoding as well as confirming the intuition explained in Section SECREF16: once an entity is selected, facts about this entity are relevant based on their key, not value which might add noise. To illustrate this intuition, we depict in Figure FIGREF27 attention scores (recall $\alpha _{i,t}$ and $\beta _{i,j,t}$ from equations (SECREF16) and (DISPLAY_FORM17)) for both variants Hierarchical-kv and Hierarchical-k. We particularly focus on the timestamp where the models should mention the number of points scored during the first quarter of the game. Scores of Hierarchical-k are sharp, with all of the weight on the correct record (PTS_QTR1, 26) whereas scores of Hierarchical-kv are more distributed over all PTS_QTR records, ultimately failing to retrieve the correct one. Results ::: Comparison w.r.t. baselines. From a general point of view, we can see from Table TABREF25 that our scenarios obtain significantly higher results in terms of BLEU over all models; our best model Hierarchical-k reaching $17.5$ vs. $16.5$ against the best baseline. This means that our models learns to generate fluent sequences of words, close to the gold descriptions, adequately picking up on domain lingo. Qualitative metrics are either better or on par with baselines. We show in Figure FIGREF29 a text generated by our best model, which can be directly compared to the gold description in Figure FIGREF1. Generation is fluent and contains domain-specific expressions. As reflected in Table TABREF25, the number of correct mentions (in green) outweights the number of incorrect mentions (in red). Please note that, as in previous work BIBREF28, BIBREF12, BIBREF13, BIBREF10, generated texts still contain a number of incorrect facts, as well hallucinations (in blue): sentences that have no basis in the input data (e.g. “[...] he's now averaging 22 points [...]."). While not the direct focus of our work, this highlights that any operation meant to enrich the semantics of structured data can also enrich the data with incorrect facts. Specifically, regarding all baselines, we can outline the following statements. $\bullet $ Our hierarchical models achieve significantly better scores on all metrics when compared to the flat architecture Wiseman, reinforcing the crucial role of structure in data semantics and saliency. The analysis of RG metrics shows that Wiseman seems to be the more naturalistic in terms of number of factual mentions (RG#) since it is the closest scenario to the gold value (16.83 vs. 17.31 for resp. Wiseman and Hierarchical-k). However, Wiseman achieves only $75.62$% of precision, effectively mentioning on average a total of $22.25$ records (wrong or accurate), where our model Hierarchical-k scores a precision of $89.46$%, leading to $23.66$ total mentions, just slightly above Wiseman. $\bullet $ The comparison between the Flat scenario and Wiseman is particularly interesting. Indeed, these two models share the same intuition to flatten the data-structure. The only difference stands on the encoder mechanism: bi-LSTM vs. Transformer, for Wiseman and Flat respectively. Results shows that our Flat scenario obtains a significant higher BLEU score (16.7 vs. 14.5) and generates fluent descriptions with accurate mentions (RG-P%) that are also included in the gold descriptions (CS-R%). This suggests that introducing the Transformer architecture is promising way to implicitly account for data structure. $\bullet $ Our hierarchical models outperform the two-step decoders of Li and Puduppully-plan on both BLEU and all qualitative metrics, showing that capturing structure in the encoding process is more effective that predicting a structure in the decoder (i.e., planning or templating). While our models sensibly outperform in precision at factual mentions, the baseline Puduppully-plan reaches $34.28$ mentions on average, showing that incorporating modules dedicated to entity extraction leads to over-focusing on entities; contrasting with our models that learn to generate more balanced descriptions. $\bullet $ The comparison with Puduppully-updt shows that dynamically updating the encoding across the generation process can lead to better Content Ordering (CO) and RG-P%. However, this does not help with Content Selection (CS) since our best model Hierarchical-k obtains slightly better scores. Indeed, Puduppully-updt updates representations after each mention allowing to keep track of the mention history. This guides the ordering of mentions (CO metric), each step limiting more the number of candidate mentions (increasing RG-P%). In contrast, our model encodes saliency among records/entities more effectively (CS metric). We note that while our model encodes the data-structure once and for all, Puduppully-updt recomputes, via the updates, the encoding at each step and therefore significantly increases computation complexity. Combined with their RG-# score of $30.11$, we argue that our model is simpler, and obtains fluent description with accurate mentions in a more human-like fashion. We would also like to draw attention to the number of parameters used by those architectures. We note that our scenarios relies on a lower number of parameters (14 millions) compared to all baselines (ranging from 23 to 45 millions). This outlines the effectiveness in the design of our model relying on a structure encoding, in contrast to other approach that try to learn the structure of data/descriptions from a linearized encoding. Conclusion and future work In this work we have proposed a hierarchical encoder for structured data, which 1) leverages the structure to form efficient representation of its input; 2) has strong synergy with the hierarchical attention of its associated decoder. This results in an effective and more light-weight model. Experimental evaluation on the RotoWire benchmark shows that our model outperforms competitive baselines in terms of BLEU score and is generally better on qualitative metrics. This way of representing structured databases may lead to automatic inference and enrichment, e.g., by comparing entities. This direction could be driven by very recent operation-guided networks BIBREF39, BIBREF40. In addition, we note that our approach can still lead to erroneous facts or even hallucinations. An interesting perspective might be to further constrain the model on the data structure in order to prevent inaccurate of even contradictory descriptions. Acknowledgements We would like to thank the H2020 project AI4EU (825619) which partially supports Laure Soulier and Patrick Gallinari.
What future possible improvements are listed?
d915b401bb96c9f104a0353bef9254672e6f5a47
What Ġfuture Ġpossible Ġimprovements Ġare Ġlisted ?
r ther Ġconst rain Ġthe Ġmodel Ġon Ġthe Ġdata Ġstructure Ġin Ġorder Ġto Ġprevent Ġinaccurate Ġof Ġeven Ġcontradictory Ġdescriptions
<s> What Ġfuture Ġpossible Ġimprovements Ġare Ġlisted ? </s> Introduction </s> Know ledge Ġand / or Ġdata Ġis Ġoften Ġmodeled Ġin Ġa Ġstructure , Ġsuch Ġas Ġindexes , Ġtables , Ġkey - value Ġpairs , Ġor Ġtriple ts . ĠThese Ġdata , Ġby Ġtheir Ġnature Ġ( e . g ., Ġraw Ġdata Ġor Ġlong Ġtime - series Ġdata ), Ġare Ġnot Ġeasily Ġusable Ġby Ġhumans ; Ġoutlining Ġtheir Ġcrucial Ġneed Ġto Ġbe Ġsynthes ized . ĠRecently , Ġnumerous Ġworks Ġhave Ġfocused Ġon Ġleveraging Ġstructured Ġdata Ġin Ġvarious Ġapplications , Ġsuch Ġas Ġquestion Ġanswering ĠB IB REF 0 , ĠB IB REF 1 Ġor Ġtable Ġretrieval ĠB IB REF 2 , ĠB IB REF 3 . ĠOne Ġemerging Ġresearch Ġfield Ġconsists Ġin Ġtransc ribing Ġdata - struct ures Ġinto Ġnatural Ġlanguage Ġin Ġorder Ġto Ġease Ġtheir Ġunderstand abl ity Ġand Ġtheir Ġus abl ity . ĠThis Ġfield Ġis Ġreferred Ġto Ġas ĠâĢ ľ data - to - text " ĠB IB REF 4 Ġand Ġhas Ġits Ġplace Ġin Ġseveral Ġapplication Ġdomains Ġ( such Ġas Ġjournalism ĠB IB REF 5 Ġor Ġmedical Ġdiagnosis ĠB IB REF 6 ) Ġor Ġwide - aud ience Ġapplications Ġ( such Ġas Ġfinancial ĠB IB REF 7 Ġand Ġweather Ġreports ĠB IB REF 8 , Ġor Ġsport Ġbroadcasting ĠB IB REF 9 , ĠB IB REF 10 ). ĠAs Ġan Ġexample , ĠFigure ĠFIG REF 1 Ġshows Ġa Ġdata - st ructure Ġcontaining Ġstatistics Ġon ĠNBA Ġbasketball Ġgames , Ġpaired Ġwith Ġits Ġcorresponding Ġjournalistic Ġdescription . </s> Design ing Ġdata - to - text Ġmodels Ġgives Ġrise Ġto Ġtwo Ġmain Ġchallenges : Ġ1 ) Ġunderstanding Ġstructured Ġdata Ġand Ġ2 ) Ġgenerating Ġassociated Ġdescriptions . ĠRecent Ġdata - to - text Ġmodels ĠB IB REF 11 , ĠB IB REF 12 , ĠB IB REF 13 , ĠB IB REF 10 Ġmostly Ġrely Ġon Ġan Ġenc oder - dec oder Ġarchitecture ĠB IB REF 14 Ġin Ġwhich Ġthe Ġdata - st ructure Ġis Ġfirst Ġencoded Ġsequ entially Ġinto Ġa Ġfixed - size Ġvector ial Ġrepresentation Ġby Ġan Ġenc oder . ĠThen , Ġa Ġdec oder Ġgenerates Ġwords Ġconditioned Ġon Ġthis Ġrepresentation . ĠWith Ġthe Ġintroduction Ġof Ġthe Ġattention Ġmechanism ĠB IB REF 15 Ġon Ġone Ġhand , Ġwhich Ġcomp utes Ġa Ġcontext Ġfocused Ġon Ġimportant Ġelements Ġfrom Ġthe Ġinput Ġat Ġeach Ġdecoding Ġstep Ġand , Ġon Ġthe Ġother Ġhand , Ġthe Ġcopy Ġmechanism ĠB IB REF 16 , ĠB IB REF 17 Ġto Ġdeal Ġwith Ġunknown Ġor Ġrare Ġwords , Ġthese Ġsystems Ġproduce Ġfluent Ġand Ġdomain Ġcomprehensive Ġtexts . ĠFor Ġinstance , ĠRobert i Ġet Ġal . ĠB IB REF 18 Ġtrain Ġa Ġcharacter - wise Ġenc oder - dec oder Ġto Ġgenerate Ġdescriptions Ġof Ġrestaurants Ġbased Ġon Ġtheir Ġattributes , Ġwhile ĠP ud upp ully Ġet Ġal . ĠB IB REF 12 Ġdesign Ġa Ġmore Ġcomplex Ġtwo - step Ġdec oder : Ġthey Ġfirst Ġgenerate Ġa Ġplan Ġof Ġelements Ġto Ġbe Ġmentioned , Ġand Ġthen Ġcondition Ġtext Ġgeneration Ġon Ġthis Ġplan . ĠAlthough Ġprevious Ġwork Ġyield Ġoverall Ġgood Ġresults , Ġwe Ġidentify Ġtwo Ġimportant Ġcaveats , Ġthat Ġhinder Ġprecision Ġ( i . e . Ġfactual Ġmentions ) Ġin Ġthe Ġdescriptions : </s> Lin ear ization Ġof Ġthe Ġdata - st ructure . ĠIn Ġpractice , Ġmost Ġworks Ġfocus Ġon Ġintroducing Ġinnov ating Ġdecoding Ġmodules , Ġand Ġstill Ġrepresent Ġdata Ġas Ġa Ġunique Ġsequence Ġof Ġelements Ġto Ġbe Ġencoded . ĠFor Ġexample , Ġthe Ġtable Ġfrom ĠFigure ĠFIG REF 1 Ġwould Ġbe Ġlinear ized Ġto Ġ[( H awks , ĠH / V , ĠH ), Ġ... , Ġ( Magic , ĠH / V , ĠV ), Ġ... ], Ġeffectively Ġleading Ġto Ġlosing Ġdistinction Ġbetween Ġrows , Ġand Ġtherefore Ġentities . ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge , Ġonly ĠLiu Ġet Ġal . ĠB IB REF 19 , ĠB IB REF 11 Ġpropose Ġenc od ers Ġconstrained Ġby Ġthe Ġstructure Ġbut Ġthese Ġapproaches Ġare Ġdesigned Ġfor Ġsingle - entity Ġstructures . </s> Ar bit rary Ġordering Ġof Ġun ordered Ġcollections Ġin Ġrecurrent Ġnetworks Ġ( R NN ). ĠMost Ġdata - to - text Ġsystems Ġuse ĠR NN s Ġas Ġenc od ers Ġ( such Ġas ĠGR Us Ġor ĠL ST Ms ), Ġthese Ġarchitectures Ġhave Ġhowever Ġsome Ġlimitations . ĠIndeed , Ġthey Ġrequire Ġin Ġpractice Ġtheir Ġinput Ġto Ġbe Ġfed Ġsequ entially . ĠThis Ġway Ġof Ġencoding Ġun ordered Ġsequences Ġ( i . e . Ġcollections Ġof Ġentities ) Ġimplicitly Ġassumes Ġan Ġarbitrary Ġorder Ġwithin Ġthe Ġcollection Ġwhich , Ġas Ġdemonstrated Ġby ĠV iny als Ġet Ġal . ĠB IB REF 20 , Ġsignificantly Ġimpacts Ġthe Ġlearning Ġperformance . </s> To Ġaddress Ġthese Ġshortcomings , Ġwe Ġpropose Ġa Ġnew Ġstructured - data Ġenc oder Ġassuming Ġthat Ġstructures Ġshould Ġbe Ġhierarch ically Ġcaptured . ĠOur Ġcontribution Ġfocuses Ġon Ġthe Ġencoding Ġof Ġthe Ġdata - st ructure , Ġthus Ġthe Ġdec oder Ġis Ġchosen Ġto Ġbe Ġa Ġclassical Ġmodule Ġas Ġused Ġin ĠB IB REF 12 , ĠB IB REF 10 . ĠOur Ġcontribution Ġis Ġthree fold : </s> We Ġmodel Ġthe Ġgeneral Ġstructure Ġof Ġthe Ġdata Ġusing Ġa Ġtwo - level Ġarchitecture , Ġfirst Ġencoding Ġall Ġentities Ġon Ġthe Ġbasis Ġof Ġtheir Ġelements , Ġthen Ġencoding Ġthe Ġdata Ġstructure Ġon Ġthe Ġbasis Ġof Ġits Ġentities ; </s> We Ġintroduce Ġthe ĠTrans former Ġenc oder ĠB IB REF 21 Ġin Ġdata - to - text Ġmodels Ġto Ġensure Ġrobust Ġencoding Ġof Ġeach Ġelement / ent ities Ġin Ġcomparison Ġto Ġall Ġothers , Ġno Ġmatter Ġtheir Ġinitial Ġpositioning ; </s> We Ġintegrate Ġa Ġhierarchical Ġattention Ġmechanism Ġto Ġcompute Ġthe Ġhierarchical Ġcontext Ġfed Ġinto Ġthe Ġdec oder . </s> We Ġreport Ġexperiments Ġon Ġthe ĠR oto Wire Ġbenchmark ĠB IB REF 10 Ġwhich Ġcontains Ġaround Ġ$ 5 K $ Ġstatistical Ġtables Ġof ĠNBA Ġbasketball Ġgames Ġpaired Ġwith Ġhuman - written Ġdescriptions . ĠOur Ġmodel Ġis Ġcompared Ġto Ġseveral Ġstate - of - the - art Ġmodels . ĠResults Ġshow Ġthat Ġthe Ġproposed Ġarchitecture Ġoutper forms Ġprevious Ġmodels Ġon ĠB LE U Ġscore Ġand Ġis Ġgenerally Ġbetter Ġon Ġqualitative Ġmetrics . </s> In Ġthe Ġfollowing , Ġwe Ġfirst Ġpresent Ġa Ġstate - of - the Ġart Ġof Ġdata - to - text Ġliterature Ġ( Section Ġ2 ), Ġand Ġthen Ġdescribe Ġour Ġproposed Ġhierarchical Ġdata Ġenc oder Ġ( Section Ġ3 ). ĠThe Ġevaluation Ġprotocol Ġis Ġpresented Ġin ĠSection Ġ4 , Ġfollowed Ġby Ġthe Ġresults Ġ( Section Ġ5 ). ĠSection Ġ6 Ġconcludes Ġthe Ġpaper Ġand Ġpresents Ġperspectives . </s> Related ĠWork </s> Until Ġrecently , Ġefforts Ġto Ġbring Ġout Ġsemantics Ġfrom Ġstructured - data Ġrelied Ġheavily Ġon Ġexpert Ġknowledge ĠB IB REF 22 , ĠB IB REF 8 . ĠFor Ġexample , Ġin Ġorder Ġto Ġbetter Ġtransc ribe Ġnumerical Ġtime Ġseries Ġof Ġweather Ġdata Ġto Ġa Ġtextual Ġforecast , ĠRe iter Ġet Ġal . ĠB IB REF 8 Ġdevise Ġcomplex Ġtemplate Ġschemes Ġin Ġcollaboration Ġwith Ġweather Ġexperts Ġto Ġbuild Ġa Ġconsistent Ġset Ġof Ġdata - to - word Ġrules . </s> Modern Ġapproaches Ġto Ġthe Ġwide Ġrange Ġof Ġtasks Ġbased Ġon Ġstructured - data Ġ( e . g . Ġtable Ġretrieval ĠB IB REF 2 , ĠB IB REF 23 , Ġtable Ġclassification ĠB IB REF 24 , Ġquestion Ġanswering ĠB IB REF 25 ) Ġnow Ġpropose Ġto Ġleverage Ġprogress Ġin Ġdeep Ġlearning Ġto Ġrepresent Ġthese Ġdata Ġinto Ġa Ġsemantic Ġvector Ġspace Ġ( also Ġcalled Ġembed ding Ġspace ). ĠIn Ġparallel , Ġan Ġemerging Ġtask , Ġcalled ĠâĢ ľ data - to - text ", Ġaims Ġat Ġdescribing Ġstructured Ġdata Ġinto Ġa Ġnatural Ġlanguage Ġdescription . ĠThis Ġtask Ġstems Ġfrom Ġthe Ġneural Ġmachine Ġtranslation Ġ( N MT ) Ġdomain , Ġand Ġearly Ġwork ĠB IB REF 26 , ĠB IB REF 27 , ĠB IB REF 10 Ġrepresent Ġthe Ġdata Ġrecords Ġas Ġa Ġsingle Ġsequence Ġof Ġfacts Ġto Ġbe Ġentirely Ġtranslated Ġinto Ġnatural Ġlanguage . ĠWis eman Ġet Ġal . ĠB IB REF 10 Ġshow Ġthe Ġlimits Ġof Ġtraditional ĠN MT Ġsystems Ġon Ġlarger Ġstructured - data , Ġwhere ĠN MT Ġsystems Ġfail Ġto Ġaccurately Ġextract Ġsalient Ġelements . </s> To Ġimprove Ġthese Ġmodels , Ġa Ġnumber Ġof Ġwork ĠB IB REF 28 , ĠB IB REF 12 , ĠB IB REF 29 Ġproposed Ġinnov ating Ġdecoding Ġmodules Ġbased Ġon Ġplanning Ġand Ġtemplates , Ġto Ġensure Ġfactual Ġand Ġcoherent Ġmentions Ġof Ġrecords Ġin Ġgenerated Ġdescriptions . ĠFor Ġexample , ĠP ud upp ully Ġet Ġal . ĠB IB REF 12 Ġpropose Ġa Ġtwo - step Ġdec oder Ġwhich Ġfirst Ġtargets Ġspecific Ġrecords Ġand Ġthen Ġuse Ġthem Ġas Ġa Ġplan Ġfor Ġthe Ġactual Ġtext Ġgeneration . ĠSimilarly , ĠLi Ġet Ġal . ĠB IB REF 28 Ġproposed Ġa Ġdelayed Ġcopy Ġmechanism Ġwhere Ġtheir Ġdec oder Ġalso Ġacts Ġin Ġtwo Ġsteps : Ġ1 ) Ġusing Ġa Ġclassical ĠL ST M Ġdec oder Ġto Ġgenerate Ġde lex ical ized Ġtext Ġand Ġ2 ) Ġusing Ġa Ġpointer Ġnetwork ĠB IB REF 30 Ġto Ġreplace Ġplace holders Ġby Ġrecords Ġfrom Ġthe Ġinput Ġdata . </s> Cl oser Ġto Ġour Ġwork , Ġvery Ġrecent Ġwork ĠB IB REF 11 , ĠB IB REF 19 , ĠB IB REF 13 Ġhave Ġproposed Ġto Ġtake Ġinto Ġaccount Ġthe Ġdata Ġstructure . ĠMore Ġparticularly , ĠP ud upp ully Ġet Ġal . ĠB IB REF 13 Ġfollow Ġentity - centric Ġtheories ĠB IB REF 31 , ĠB IB REF 32 Ġand Ġpropose Ġa Ġmodel Ġbased Ġon Ġdynamic Ġentity Ġrepresentation Ġat Ġdecoding Ġtime . ĠIt Ġconsists Ġin Ġconditioning Ġthe Ġdec oder Ġon Ġentity Ġrepresentations Ġthat Ġare Ġupdated Ġduring Ġinference Ġat Ġeach Ġdecoding Ġstep . ĠOn Ġthe Ġother Ġhand , ĠLiu Ġet Ġal . ĠB IB REF 11 , ĠB IB REF 19 Ġrather Ġfocus Ġon Ġintroducing Ġstructure Ġinto Ġthe Ġenc oder . ĠFor Ġinstance , Ġthey Ġpropose Ġa Ġdual Ġenc oder ĠB IB REF 19 Ġwhich Ġenc odes Ġseparately Ġthe Ġsequence Ġof Ġelement Ġnames Ġand Ġthe Ġsequence Ġof Ġelement Ġvalues . ĠThese Ġapproaches Ġare Ġhowever Ġdesigned Ġfor Ġsingle - entity Ġdata Ġstructures Ġand Ġdo Ġnot Ġaccount Ġfor Ġdelim itation Ġbetween Ġentities . </s> Our Ġcontribution Ġdiffers Ġfrom Ġprevious Ġwork Ġin Ġseveral Ġaspects . ĠFirst , Ġinstead Ġof Ġflat ly Ġconc aten ating Ġelements Ġfrom Ġthe Ġdata - st ructure Ġand Ġencoding Ġthem Ġas Ġa Ġsequence ĠB IB REF 11 , ĠB IB REF 12 , ĠB IB REF 10 , Ġwe Ġconst rain Ġthe Ġencoding Ġto Ġthe Ġunderlying Ġstructure Ġof Ġthe Ġinput Ġdata , Ġso Ġthat Ġthe Ġdelim itation Ġbetween Ġentities Ġremains Ġclear Ġthroughout Ġthe Ġprocess . ĠSecond , Ġunlike Ġall Ġworks Ġin Ġthe Ġdomain , Ġwe Ġexploit Ġthe ĠTrans former Ġarchitecture ĠB IB REF 21 Ġand Ġleverage Ġits Ġparticular ity Ġto Ġdirectly Ġcompare Ġelements Ġwith Ġeach Ġothers Ġin Ġorder Ġto Ġavoid Ġarbitrary Ġassumptions Ġon Ġtheir Ġordering . ĠFinally , Ġin Ġcontrast Ġto ĠB IB REF 33 , ĠB IB REF 13 Ġthat Ġuse Ġa Ġcomplex Ġupdating Ġmechanism Ġto Ġobtain Ġa Ġdynamic Ġrepresentation Ġof Ġthe Ġinput Ġdata Ġand Ġits Ġentities , Ġwe Ġargue Ġthat Ġexplicit Ġhierarchical Ġencoding Ġnaturally Ġguides Ġthe Ġdecoding Ġprocess Ġvia Ġhierarchical Ġattention . </s> H ier arch ical ĠEnc oder ĠModel Ġfor ĠData - to - Text </s> In Ġthis Ġsection Ġwe Ġintroduce Ġour Ġproposed Ġhierarchical Ġmodel Ġtaking Ġinto Ġaccount Ġthe Ġdata Ġstructure . ĠWe Ġoutline Ġthat Ġthe Ġdecoding Ġcomponent Ġaiming Ġto Ġgenerate Ġdescriptions Ġis Ġconsidered Ġas Ġa Ġblack - box Ġmodule Ġso Ġthat Ġour Ġcontribution Ġis Ġfocused Ġon Ġthe Ġencoding Ġmodule . ĠWe Ġfirst Ġdescribe Ġthe Ġmodel Ġoverview , Ġbefore Ġdetailing Ġthe Ġhierarchical Ġenc oder Ġand Ġthe Ġassociated Ġhierarchical Ġattention . </s> H ier arch ical ĠEnc oder ĠModel Ġfor ĠData - to - Text Ġ: :: ĠNot ation Ġand ĠGeneral ĠOverview </s> Let 's Ġconsider Ġthe Ġfollowing Ġnot ations : </s> $ \ bul let Ġ$ ĠAn Ġentity Ġ$ e _ i $ Ġis Ġa Ġset Ġof Ġ$ J _ i $ Ġun ordered Ġrecords Ġ$\ l brace Ġr _{ i , 1 }, Ġ... , Ġr _{ i , j }, Ġ... , Ġr _{ i , J _ i }\ r brace Ġ$ ; Ġwhere Ġrecord Ġ$ r _{ i , j } $ Ġis Ġdefined Ġas Ġa Ġpair Ġof Ġkey Ġ$ k _{ i , j } $ Ġand Ġvalue Ġ$ v _{ i , j } $. ĠWe Ġoutline Ġthat Ġ$ J _ i $ Ġmight Ġdiffer Ġbetween Ġentities . </s> $ \ bul let Ġ$ ĠA Ġdata - st ructure Ġ$ s $ Ġis Ġan Ġun ordered Ġset Ġof Ġ$ I $ Ġentities Ġ$ e _ i $. ĠWe Ġthus Ġdenote Ġ$ s Ġ\ l brace Ġe _ 1 , Ġ... , Ġe _ i , Ġ... , Ġe _ I \ r brace Ġ$ . </s> $ \ bul let Ġ$ ĠFor Ġeach Ġdata - st ructure , Ġa Ġtextual Ġdescription Ġ$ y $ Ġis Ġassociated . ĠWe Ġrefer Ġto Ġthe Ġfirst Ġ$ t $ Ġwords Ġof Ġa Ġdescription Ġ$ y $ Ġas Ġ$ y _{ 1 : t } $. ĠThus , Ġthe Ġfull Ġsequence Ġof Ġwords Ġcan Ġbe Ġnoted Ġas Ġ$ y Ġ= Ġy _{ 1 : T } $. </s> $ \ bul let Ġ$ ĠThe Ġdataset Ġ$\ math cal Ġ{ D } $ Ġis Ġa Ġcollection Ġof Ġ$ N $ Ġaligned Ġ( data - st ructure , Ġdescription ) Ġpairs Ġ$( s , y ) $. </s> For Ġinstance , ĠFigure ĠFIG REF 1 Ġillustrates Ġa Ġdata - st ructure Ġassociated Ġwith Ġa Ġdescription . ĠThe Ġdata - st ructure Ġincludes Ġa Ġset Ġof Ġentities Ġ( H awks , ĠMagic , ĠAl ĠHor ford , ĠJeff ĠTe ague , Ġ... ). ĠThe Ġentity ĠJeff ĠTe ague Ġis Ġmodeled Ġas Ġa Ġset Ġof Ġrecords Ġ{ ( P TS , Ġ17 ), Ġ( RE B , Ġ0 ), Ġ( AST , Ġ7 ) Ġ... } Ġin Ġwhich , Ġe . g ., Ġthe Ġrecord Ġ( P TS , Ġ17 ) Ġis Ġcharacterized Ġby Ġa Ġkey Ġ( P TS ) Ġand Ġa Ġvalue Ġ( 17 ). </s> For Ġeach Ġdata - st ructure Ġ$ s $ Ġin Ġ$\ math cal Ġ{ D } $, Ġthe Ġobjective Ġfunction Ġaims Ġto Ġgenerate Ġa Ġdescription Ġ$\ hat { y } $ Ġas Ġclose Ġas Ġpossible Ġto Ġthe Ġground Ġtruth Ġ$ y $. ĠThis Ġobjective Ġfunction Ġoptim izes Ġthe Ġfollowing Ġlog - like lihood Ġover Ġthe Ġwhole Ġdataset Ġ$\ math cal Ġ{ D } $ : </s> where Ġ$\ the ta Ġ$ Ġstands Ġfor Ġthe Ġmodel Ġparameters Ġand Ġ$ P (\ hat { y } = y \ Ġ| \ Ġs ; Ġ\ the ta Ġ) $ Ġthe Ġprobability Ġof Ġthe Ġmodel Ġto Ġgenerate Ġthe Ġadequate Ġdescription Ġ$ y $ Ġfor Ġtable Ġ$ s $. </s> During Ġinference , Ġwe Ġgenerate Ġthe Ġsequence Ġ$\ hat { y } ^ * $ Ġwith Ġthe Ġmaximum Ġa Ġposterior i Ġprobability Ġconditioned Ġon Ġtable Ġ$ s $. ĠUsing Ġthe Ġchain Ġrule , Ġwe Ġget : </s> This Ġequation Ġis Ġint ract able Ġin Ġpractice , Ġwe Ġapproximate Ġa Ġsolution Ġusing Ġbeam Ġsearch , Ġas Ġin ĠB IB REF 11 , ĠB IB REF 19 , ĠB IB REF 12 , ĠB IB REF 13 , ĠB IB REF 10 . </s> Our Ġmodel Ġfollows Ġthe Ġenc oder - dec oder Ġarchitecture ĠB IB REF 14 . ĠBecause Ġour Ġcontribution Ġfocuses Ġon Ġthe Ġencoding Ġprocess , Ġwe Ġchose Ġthe Ġdecoding Ġmodule Ġused Ġin ĠB IB REF 12 , ĠB IB REF 10 : Ġa Ġtwo - l ayers ĠL ST M Ġnetwork Ġwith Ġa Ġcopy Ġmechanism . ĠIn Ġorder Ġto Ġsuperv ise Ġthis Ġmechanism , Ġwe Ġassume Ġthat Ġeach Ġrecord Ġvalue Ġthat Ġalso Ġappears Ġin Ġthe Ġtarget Ġis Ġcopied Ġfrom Ġthe Ġdata - st ructure Ġand Ġwe Ġtrain Ġthe Ġmodel Ġto Ġswitch Ġbetween Ġfreely Ġgenerating Ġwords Ġfrom Ġthe Ġvocabulary Ġand Ġcopying Ġwords Ġfrom Ġthe Ġinput . ĠWe Ġnow Ġdescribe Ġthe Ġhierarchical Ġenc oder Ġand Ġthe Ġhierarchical Ġattention . </s> H ier arch ical ĠEnc oder ĠModel Ġfor ĠData - to - Text Ġ: :: ĠHier arch ical ĠEnc oding ĠModel </s> As Ġoutlined Ġin ĠSection ĠSEC REF 2 , Ġmost Ġprevious Ġwork ĠB IB REF 28 , ĠB IB REF 12 , ĠB IB REF 13 , ĠB IB REF 10 , ĠB IB REF 29 Ġmake Ġuse Ġof Ġflat Ġenc od ers Ġthat Ġdo Ġnot Ġexploit Ġthe Ġdata Ġstructure . ĠTo Ġkeep Ġthe Ġsemantics Ġof Ġeach Ġelement Ġfrom Ġthe Ġdata - st ructure , Ġwe Ġpropose Ġa Ġhierarchical Ġenc oder Ġwhich Ġrelies Ġon Ġtwo Ġmodules . ĠThe Ġfirst Ġone Ġ( module ĠA Ġin ĠFigure ĠFIG REF 11 ) Ġis Ġcalled Ġlow - level Ġenc oder Ġand Ġenc odes Ġentities Ġon Ġthe Ġbasis Ġof Ġtheir Ġrecords ; Ġthe Ġsecond Ġone Ġ( module ĠB ), Ġcalled Ġhigh - level Ġenc oder , Ġenc odes Ġthe Ġdata - st ructure Ġon Ġthe Ġbasis Ġof Ġits Ġunderlying Ġentities . ĠIn Ġthe Ġlow - level Ġenc oder , Ġthe Ġtraditional Ġembed ding Ġlayer Ġis Ġreplaced Ġby Ġa Ġrecord Ġembed ding Ġlayer Ġas Ġin ĠB IB REF 11 , ĠB IB REF 12 , ĠB IB REF 10 . ĠWe Ġpresent Ġin Ġwhat Ġfollows Ġthe Ġrecord Ġembed ding Ġlayer Ġand Ġintroduce Ġour Ġtwo Ġhierarchical Ġmodules . </s> H ier arch ical ĠEnc oder ĠModel Ġfor ĠData - to - Text Ġ: :: ĠHier arch ical ĠEnc oding ĠModel Ġ: :: ĠRecord ĠEmb edd ing ĠLayer . </s> The Ġfirst Ġlayer Ġof Ġthe Ġnetwork Ġconsists Ġin Ġlearning Ġtwo Ġembed ding Ġmat rices Ġto Ġembed Ġthe Ġrecord Ġkeys Ġand Ġvalues . ĠKeys Ġ$ k _{ i , j } $ Ġare Ġembedded Ġto Ġ$\ math bf Ġ{ k } _{ i , j } Ġ\ in Ġ\ math bb Ġ{ R } ^{ d } $ Ġand Ġvalues Ġ$ v _{ i , j } $ Ġto Ġ$\ math bf Ġ{ v } _{ i , j } Ġ\ in Ġ\ math bb Ġ{ R } ^{ d } $, Ġwith Ġ$ d $ Ġthe Ġsize Ġof Ġthe Ġembed ding . ĠAs Ġin Ġprevious Ġwork ĠB IB REF 11 , ĠB IB REF 12 , ĠB IB REF 10 , Ġeach Ġrecord Ġembed ding Ġ$\ math bf Ġ{ r } _{ i , j } $ Ġis Ġcomputed Ġby Ġa Ġlinear Ġprojection Ġon Ġthe Ġconc aten ation Ġ$ [ \ math bf Ġ{ k } _{ i , j } $ ; Ġ$\ math bf Ġ{ v } _{ i , j } ] $ Ġfollowed Ġby Ġa Ġnon Ġlinear ity : </s> where Ġ$\ math bf Ġ{ W } _ r Ġ\ in Ġ\ math bb Ġ{ R } ^{ 2 d Ġ\ times Ġd } $ Ġand Ġ$\ math bf Ġ{ b } _ r Ġ\ in Ġ\ math bb Ġ{ R } ^{ d } $ Ġare Ġlearnt Ġparameters . </s> The Ġlow - level Ġenc oder Ġaims Ġat Ġencoding Ġa Ġcollection Ġof Ġrecords Ġbelonging Ġto Ġthe Ġsame Ġentity Ġwhile Ġthe Ġhigh - level Ġenc oder Ġenc odes Ġthe Ġwhole Ġset Ġof Ġentities . ĠBoth Ġthe Ġlow - level Ġand Ġhigh - level Ġenc od ers Ġconsider Ġtheir Ġinput Ġelements Ġas Ġun ordered . ĠWe Ġuse Ġthe ĠTrans former Ġarchitecture Ġfrom ĠB IB REF 21 . ĠFor Ġeach Ġenc oder , Ġwe Ġhave Ġthe Ġfollowing Ġpeculiar ities : </s> the ĠLow - level Ġenc oder Ġenc odes Ġeach Ġentity Ġ$ e _ i $ Ġon Ġthe Ġbasis Ġof Ġits Ġrecord Ġembed d ings Ġ$\ math bf Ġ{ r } _{ i , j } $. ĠEach Ġrecord Ġembed ding Ġ$\ math bf Ġ{ r } _{ i , j } $ Ġis Ġcompared Ġto Ġother Ġrecord Ġembed d ings Ġto Ġlearn Ġits Ġfinal Ġhidden Ġrepresentation Ġ$\ math bf Ġ{ h } _{ i , j } $. ĠFurthermore , Ġwe Ġadd Ġa Ġspecial Ġrecord Ġ[ ENT ] Ġfor Ġeach Ġentity , Ġillustrated Ġin ĠFigure ĠFIG REF 11 Ġas Ġthe Ġlast Ġrecord . ĠSince Ġentities Ġmight Ġhave Ġa Ġvariable Ġnumber Ġof Ġrecords , Ġthis Ġtoken Ġallows Ġto Ġaggregate Ġfinal Ġhidden Ġrecord Ġrepresentations Ġ$\ l brace Ġ\ math bf Ġ{ h } _{ i , j }\ r brace Ġ_ { j = 1 } ^{ J _ i } $ Ġin Ġa Ġfixed - sized Ġrepresentation Ġvector Ġ$\ math bf Ġ{ h } _{ i } $. </s> the ĠHigh - level Ġenc oder Ġenc odes Ġthe Ġdata - st ructure Ġon Ġthe Ġbasis Ġof Ġits Ġentity Ġrepresentation Ġ$\ math bf Ġ{ h } _{ i } $. ĠSimilarly Ġto Ġthe ĠLow - level Ġenc oder , Ġthe Ġfinal Ġhidden Ġstate Ġ$\ math bf Ġ{ e _ i } $ Ġof Ġan Ġentity Ġis Ġcomputed Ġby Ġcomparing Ġentity Ġrepresentation Ġ$\ math bf Ġ{ h } _{ i } $ Ġwith Ġeach Ġothers . ĠThe Ġdata - st ructure Ġrepresentation Ġ$\ math bf Ġ{ z } $ Ġis Ġcomputed Ġas Ġthe Ġmean Ġof Ġthese Ġentity Ġrepresentations , Ġand Ġis Ġused Ġfor Ġthe Ġdec oder Ġinitialization . </s> H ier arch ical ĠEnc oder ĠModel Ġfor ĠData - to - Text Ġ: :: ĠHier arch ical Ġattention </s> To Ġfully Ġleverage Ġthe Ġhierarchical Ġstructure Ġof Ġour Ġenc oder , Ġwe Ġpropose Ġtwo Ġvariants Ġof Ġhierarchical Ġattention Ġmechanism Ġto Ġcompute Ġthe Ġcontext Ġfed Ġto Ġthe Ġdec oder Ġmodule . </s> $ \ bul let Ġ$ ĠTraditional ĠHier arch ical ĠAttention . ĠAs Ġin ĠB IB REF 13 , Ġwe Ġhypothes ize Ġthat Ġa Ġdynamic Ġcontext Ġshould Ġbe Ġcomputed Ġin Ġtwo Ġsteps : Ġfirst Ġattending Ġto Ġentities , Ġthen Ġto Ġrecords Ġcorresponding Ġto Ġthese Ġentities . ĠTo Ġimplement Ġthis Ġhierarchical Ġattention , Ġat Ġeach Ġdecoding Ġstep Ġ$ t $, Ġthe Ġmodel Ġlearns Ġa Ġfirst Ġset Ġof Ġattention Ġscores Ġ$\ alpha Ġ_ { i , t } $ Ġover Ġentities Ġ$ e _ i $ Ġand Ġa Ġsecond Ġset Ġof Ġattention Ġscores Ġ$\ beta Ġ_ { i , j , t } $ Ġover Ġrecords Ġ$ r _{ i , j } $ Ġbelonging Ġto Ġentity Ġ$ e _ i $. ĠThe Ġ$\ alpha Ġ_ { i , t } $ Ġscores Ġare Ġnormalized Ġto Ġform Ġa Ġdistribution Ġover Ġall Ġentities Ġ$ e _ i $, Ġand Ġ$\ beta Ġ_ { i , j , t } $ Ġscores Ġare Ġnormalized Ġto Ġform Ġa Ġdistribution Ġover Ġrecords Ġ$ r _{ i , j } $ Ġof Ġentity Ġ$ e _ i $. ĠEach Ġentity Ġis Ġthen Ġrepresented Ġas Ġa Ġweighted Ġsum Ġof Ġits Ġrecord Ġembed d ings , Ġand Ġthe Ġentire Ġdata Ġstructure Ġis Ġrepresented Ġas Ġa Ġweighted Ġsum Ġof Ġthe Ġentity Ġrepresentations . ĠThe Ġdynamic Ġcontext Ġis Ġcomputed Ġas : </s> ct Ġ= Ġi = 1 I Ġ( i , t Ġ( Ġj Ġi , j , t Ġr i , j Ġ)) </s> where Ġi , t Ġexp ( dt We i ) Ġand Ġi , j , t Ġexp ( dt Wh i , j ) </s> where Ġ$\ math bf Ġ{ d _ t } $ Ġis Ġthe Ġdec oder Ġhidden Ġstate Ġat Ġtime Ġstep Ġ$ t $, Ġ$\ math bf Ġ{ W } _{ \ alpha Ġ} Ġ\ in Ġ\ math bb Ġ{ R } ^{ d \ times Ġd } $ Ġand Ġ$\ math bf Ġ{ W } _{ \ beta Ġ} Ġ\ in Ġ\ math bb Ġ{ R } ^{ d \ times Ġd } $ Ġare Ġlearnt Ġparameters , Ġ$ Ġ\ sum Ġ_ i \ alpha Ġ_ { i , t } Ġ= Ġ1 $, Ġand Ġfor Ġall Ġ$ i Ġ\ in Ġ\ l brace Ġ1 ,... , I \ r brace Ġ$ Ġ$\ sum Ġ_ { j }\ beta Ġ_ { i , j , t } Ġ= Ġ1 $. </s> $ \ bul let Ġ$ ĠKey - guided ĠHier arch ical ĠAttention . ĠThis Ġvariant Ġfollows Ġthe Ġintuition Ġthat Ġonce Ġan Ġentity Ġis Ġchosen Ġfor Ġmention Ġ( thanks Ġto Ġ$\ alpha Ġ_ { i , t } $ ), Ġonly Ġthe Ġtype Ġof Ġrecords Ġis Ġimportant Ġto Ġdetermine Ġthe Ġcontent Ġof Ġthe Ġdescription . ĠFor Ġexample , Ġwhen Ġdeciding Ġto Ġmention Ġa Ġplayer , Ġall Ġexperts Ġautomatically Ġreport Ġhis Ġscore Ġwithout Ġconsideration Ġof Ġits Ġspecific Ġvalue . ĠTo Ġtest Ġthis Ġintuition , Ġwe Ġmodel Ġthe Ġattention Ġscores Ġby Ġcomputing Ġthe Ġ$\ beta Ġ_ { i , j , t } $ Ġscores Ġfrom Ġequation Ġ( SEC REF 16 ) Ġsolely Ġon Ġthe Ġembed ding Ġof Ġthe Ġkey Ġrather Ġthan Ġon Ġthe Ġfull Ġrecord Ġrepresentation Ġ$\ math bf Ġ{ h } _{ i , j } $ : </s> Please Ġnote Ġthat Ġthe Ġdifferent Ġembed d ings Ġand Ġthe Ġmodel Ġparameters Ġpresented Ġin Ġthe Ġmodel Ġcomponents Ġare Ġlearnt Ġusing ĠEqu ation Ġ1 . </s> Exper imental Ġsetup Ġ: :: ĠThe ĠRot ow ire Ġdataset </s> To Ġevaluate Ġthe Ġeffectiveness Ġof Ġour Ġmodel , Ġand Ġdemonstrate Ġits Ġflexibility Ġat Ġhandling Ġheavy Ġdata - st ructure Ġmade Ġof Ġseveral Ġtypes Ġof Ġentities , Ġwe Ġused Ġthe ĠR oto Wire Ġdataset ĠB IB REF 10 . ĠIt Ġincludes Ġbasketball Ġgames Ġstatistical Ġtables Ġpaired Ġwith Ġjournalistic Ġdescriptions Ġof Ġthe Ġgames , Ġas Ġcan Ġbe Ġseen Ġin Ġthe Ġexample Ġof ĠFigure ĠFIG REF 1 . ĠThe Ġdescriptions Ġare Ġprofessionally Ġwritten Ġand Ġaverage Ġ337 Ġwords Ġwith Ġa Ġvocabulary Ġsize Ġof Ġ$ 11 . 3 $ K . ĠThere Ġare Ġ39 Ġdifferent Ġrecord Ġkeys , Ġand Ġthe Ġaverage Ġnumber Ġof Ġrecords Ġ( resp . Ġentities ) Ġin Ġa Ġsingle Ġdata - st ructure Ġis Ġ6 28 Ġ( resp . Ġ28 ). ĠEnt ities Ġare Ġof Ġtwo Ġtypes , Ġeither Ġteam Ġor Ġplayer , Ġand Ġplayer Ġdescriptions Ġdepend Ġon Ġtheir Ġinvolvement Ġin Ġthe Ġgame . ĠWe Ġfollowed Ġthe Ġdata Ġpartitions Ġintroduced Ġwith Ġthe Ġdataset Ġand Ġused Ġa Ġtrain / valid ation / test Ġsets Ġof Ġrespectively Ġ$ 3 , 398 $ / 727 / 728 Ġ( data - st ructure , Ġdescription ) Ġpairs . </s> Exper imental Ġsetup Ġ: :: ĠEvaluation Ġmetrics </s> We Ġevaluate Ġour Ġmodel Ġthrough Ġtwo Ġtypes Ġof Ġmetrics . ĠThe ĠB LE U Ġscore ĠB IB REF 34 Ġaims Ġat Ġmeasuring Ġto Ġwhat Ġextent Ġthe Ġgenerated Ġdescriptions Ġare Ġliterally Ġclosed Ġto Ġthe Ġground Ġtruth . ĠThe Ġsecond Ġcategory Ġdesigned Ġby ĠB IB REF 10 Ġis Ġmore Ġqualitative . </s> Exper imental Ġsetup Ġ: :: ĠEvaluation Ġmetrics Ġ: :: ĠB LE U ĠScore . </s> The ĠB LE U Ġscore ĠB IB REF 34 Ġis Ġcommonly Ġused Ġas Ġan Ġevaluation Ġmetric Ġin Ġtext Ġgeneration Ġtasks . ĠIt Ġestimates Ġthe Ġcorrespondence Ġbetween Ġa Ġmachine Ġoutput Ġand Ġthat Ġof Ġa Ġhuman Ġby Ġcomputing Ġthe Ġnumber Ġof Ġco - occ ur rences Ġfor Ġn gram s Ġ($ n Ġ\ in Ġ{ 1 , Ġ2 , Ġ3 , Ġ4 } $ ) Ġbetween Ġthe Ġgenerated Ġcandidate Ġand Ġthe Ġground Ġtruth . ĠWe Ġuse Ġthe Ġimplementation Ġcode Ġreleased Ġby ĠB IB REF 35 . </s> Exper imental Ġsetup Ġ: :: ĠEvaluation Ġmetrics Ġ: :: ĠInformation Ġextraction - oriented Ġmetrics . </s> These Ġmetrics Ġestimate Ġthe Ġability Ġof Ġour Ġmodel Ġto Ġintegrate Ġelements Ġfrom Ġthe Ġtable Ġin Ġits Ġdescriptions . ĠParticularly , Ġthey Ġcompare Ġthe Ġgold Ġand Ġgenerated Ġdescriptions Ġand Ġmeasure Ġto Ġwhat Ġextent Ġthe Ġextracted Ġrelations Ġare Ġaligned Ġor Ġdiffer . ĠTo Ġdo Ġso , Ġwe Ġfollow Ġthe Ġprotocol Ġpresented Ġin ĠB IB REF 10 . ĠFirst , Ġwe Ġapply Ġan Ġinformation Ġextraction Ġ( IE ) Ġsystem Ġtrained Ġon Ġlabeled Ġrelations Ġfrom Ġthe Ġgold Ġdescriptions Ġof Ġthe ĠR oto Wire Ġtrain Ġdataset . ĠEntity - value Ġpairs Ġare Ġextracted Ġfrom Ġthe Ġdescriptions . ĠFor Ġexample , Ġin Ġthe Ġsentence ĠIsaiah ĠThomas Ġled Ġthe Ġteam Ġin Ġscoring , Ġtotaling Ġ23 Ġpoints Ġ[...] ., Ġan ĠIE Ġtool Ġwill Ġextract Ġthe Ġpair Ġ( Isa iah ĠThomas , Ġ23 , ĠPTS ). ĠSecond , Ġwe Ġcompute Ġthree Ġmetrics Ġon Ġthe Ġextracted Ġinformation : </s> $ \ bul let Ġ$ ĠRel ation ĠGeneration Ġ( RG ) Ġestimates Ġhow Ġwell Ġthe Ġsystem Ġis Ġable Ġto Ġgenerate Ġtext Ġcontaining Ġfactual Ġ( i . e ., Ġcorrect ) Ġrecords . ĠWe Ġmeasure Ġthe Ġprecision Ġand Ġabsolute Ġnumber Ġ( den oted Ġrespectively ĠRG - P % Ġand ĠRG - # ) Ġof Ġunique Ġrelations Ġ$ r $ Ġextracted Ġfrom Ġ$\ hat { y } _{ 1 : T } $ Ġthat Ġalso Ġappear Ġin Ġ$ s $. </s> $ \ bul let Ġ$ ĠContent ĠSelection Ġ( CS ) Ġmeasures Ġhow Ġwell Ġthe Ġgenerated Ġdocument Ġmatches Ġthe Ġgold Ġdocument Ġin Ġterms Ġof Ġmentioned Ġrecords . ĠWe Ġmeasure Ġthe Ġprecision Ġand Ġrecall Ġ( den oted Ġrespectively ĠCS - P % Ġand ĠCS - R %) Ġof Ġunique Ġrelations Ġ$ r $ Ġextracted Ġfrom Ġ$\ hat { y } _{ 1 : T } $ Ġthat Ġare Ġalso Ġextracted Ġfrom Ġ$ y _{ 1 : T } $. </s> $ \ bul let Ġ$ ĠContent ĠOrder ing Ġ( CO ) Ġanaly zes Ġhow Ġwell Ġthe Ġsystem Ġorders Ġthe Ġrecords Ġdiscussed Ġin Ġthe Ġdescription . ĠWe Ġmeasure Ġthe Ġnormalized ĠD amer au - Le ven sh tein Ġdistance ĠB IB REF 36 Ġbetween Ġthe Ġsequences Ġof Ġrecords Ġextracted Ġfrom Ġ$\ hat { y } _{ 1 : T } $ Ġthat Ġare Ġalso Ġextracted Ġfrom Ġ$ y _{ 1 : T } $. </s> CS Ġprimarily Ġtargets Ġthe ĠâĢ ľ what Ġto Ġsay " Ġaspect Ġof Ġevaluation , ĠCO Ġtargets Ġthe ĠâĢ ľ how Ġto Ġsay Ġit " Ġaspect , Ġand ĠRG Ġtargets Ġboth . ĠNote Ġthat Ġfor ĠCS , ĠCO , ĠRG - % Ġand ĠB LE U Ġmetrics , Ġhigher Ġis Ġbetter ; Ġwhich Ġis Ġnot Ġtrue Ġfor ĠRG - # . ĠThe ĠIE Ġsystem Ġused Ġin Ġthe Ġexperiments Ġis Ġable Ġto Ġextract Ġan Ġaverage Ġof Ġ17 Ġfactual Ġrecords Ġfrom Ġgold Ġdescriptions . ĠIn Ġorder Ġto Ġmimic Ġa Ġhuman Ġexpert , Ġa Ġgener ative Ġsystem Ġshould Ġapproach Ġthis Ġnumber Ġand Ġnot Ġoverload Ġgeneration Ġwith Ġbrute Ġfacts . </s> Exper imental Ġsetup Ġ: :: ĠBas elines </s> We Ġcompare Ġour Ġhierarchical Ġmodel Ġagainst Ġthree Ġsystems . ĠFor Ġeach Ġof Ġthem , Ġwe Ġreport Ġthe Ġresults Ġof Ġthe Ġbest Ġperforming Ġmodels Ġpresented Ġin Ġeach Ġpaper . </s> $ \ bul let Ġ$ ĠWis eman ĠB IB REF 10 Ġis Ġa Ġstandard Ġenc oder - dec oder Ġsystem Ġwith Ġcopy Ġmechanism . </s> $ \ bul let Ġ$ ĠLi ĠB IB REF 28 Ġis Ġa Ġstandard Ġenc oder - dec oder Ġwith Ġa Ġdelayed Ġcopy Ġmechanism : Ġtext Ġis Ġfirst Ġgenerated Ġwith Ġplace holders , Ġwhich Ġare Ġreplaced Ġby Ġsalient Ġrecords Ġextracted Ġfrom Ġthe Ġtable Ġby Ġa Ġpointer Ġnetwork . </s> $ \ bul let Ġ$ ĠP ud upp ully - plan ĠB IB REF 12 Ġacts Ġin Ġtwo Ġsteps : Ġa Ġfirst Ġstandard Ġenc oder - dec oder Ġgenerates Ġa Ġplan , Ġi . e . Ġa Ġlist Ġof Ġsalient Ġrecords Ġfrom Ġthe Ġtable ; Ġa Ġsecond Ġstandard Ġenc oder - dec oder Ġgenerates Ġtext Ġfrom Ġthis Ġplan . </s> $ \ bul let Ġ$ ĠP ud upp ully - up dt ĠB IB REF 13 . ĠIt Ġconsists Ġin Ġa Ġstandard Ġenc oder - dec oder , Ġwith Ġan Ġadded Ġmodule Ġaimed Ġat Ġupdating Ġrecord Ġrepresentations Ġduring Ġthe Ġgeneration Ġprocess . ĠAt Ġeach Ġdecoding Ġstep , Ġa Ġg ated Ġrecurrent Ġnetwork Ġcomp utes Ġwhich Ġrecords Ġshould Ġbe Ġupdated Ġand Ġwhat Ġshould Ġbe Ġtheir Ġnew Ġrepresentation . </s> Exper imental Ġsetup Ġ: :: ĠBas elines Ġ: :: ĠModel Ġscenarios </s> We Ġtest Ġthe Ġimportance Ġof Ġthe Ġinput Ġstructure Ġby Ġtraining Ġdifferent Ġvariants Ġof Ġthe Ġproposed Ġarchitecture : </s> $ \ bul let Ġ$ ĠFlat , Ġwhere Ġwe Ġfeed Ġthe Ġinput Ġsequ entially Ġto Ġthe Ġenc oder , Ġlosing Ġall Ġnotion Ġof Ġhierarchy . ĠAs Ġa Ġconsequence , Ġthe Ġmodel Ġuses Ġstandard Ġattention . ĠThis Ġvariant Ġis Ġclosest Ġto ĠWis eman , Ġwith Ġthe Ġexception Ġthat Ġwe Ġuse Ġa ĠTrans former Ġto Ġencode Ġthe Ġinput Ġsequence Ġinstead Ġof Ġan ĠR NN . </s> $ \ bul let Ġ$ ĠHier arch ical - k v Ġis Ġour Ġfull Ġhierarchical Ġmodel , Ġwith Ġtraditional Ġhierarchical Ġattention , Ġi . e . Ġwhere Ġattention Ġover Ġrecords Ġis Ġcomputed Ġon Ġthe Ġfull Ġrecord Ġencoding , Ġas Ġin Ġequation Ġ( SEC REF 16 ). </s> $ \ bul let Ġ$ ĠHier arch ical - k Ġis Ġour Ġfull Ġhierarchical Ġmodel , Ġwith Ġkey - guided Ġhierarchical Ġattention , Ġi . e . Ġwhere Ġattention Ġover Ġrecords Ġis Ġcomputed Ġonly Ġon Ġthe Ġrecord Ġkey Ġrepresentations , Ġas Ġin Ġequation Ġ( DIS PLAY _ FORM 17 ). </s> Exper imental Ġsetup Ġ: :: ĠImplementation Ġdetails </s> The Ġdec oder Ġis Ġthe Ġone Ġused Ġin ĠB IB REF 12 , ĠB IB REF 13 , ĠB IB REF 10 Ġwith Ġthe Ġsame Ġhyper - param eters . ĠFor Ġthe Ġenc oder Ġmodule , Ġboth Ġthe Ġlow - level Ġand Ġhigh - level Ġenc od ers Ġuse Ġa Ġtwo - l ayers Ġmulti - head Ġself - att ention Ġwith Ġtwo Ġheads . ĠTo Ġfit Ġwith Ġthe Ġsmall Ġnumber Ġof Ġrecord Ġkeys Ġin Ġour Ġdataset Ġ( 39 ), Ġtheir Ġembed ding Ġsize Ġis Ġfixed Ġto Ġ20 . ĠThe Ġsize Ġof Ġthe Ġrecord Ġvalue Ġembed d ings Ġand Ġhidden Ġlayers Ġof Ġthe ĠTrans former Ġenc od ers Ġare Ġboth Ġset Ġto Ġ300 . ĠWe Ġuse Ġdrop out Ġat Ġrate Ġ0 . 5 . ĠThe Ġmodels Ġare Ġtrained Ġwith Ġa Ġbatch Ġsize Ġof Ġ64 . ĠWe Ġfollow Ġthe Ġtraining Ġprocedure Ġin ĠB IB REF 21 Ġand Ġtrain Ġthe Ġmodel Ġfor Ġa Ġfixed Ġnumber Ġof Ġ25 K Ġupdates , Ġand Ġaverage Ġthe Ġweights Ġof Ġthe Ġlast Ġ5 Ġcheckpoints Ġ( at Ġevery Ġ1 K Ġupdates ) Ġto Ġensure Ġmore Ġstability Ġacross Ġruns . ĠAll Ġmodels Ġwere Ġtrained Ġwith Ġthe ĠAdam Ġoptim izer ĠB IB REF 37 ; Ġthe Ġinitial Ġlearning Ġrate Ġis Ġ0 . 001 , Ġand Ġis Ġreduced Ġby Ġhalf Ġevery Ġ10 K Ġsteps . ĠWe Ġused Ġbeam Ġsearch Ġwith Ġbeam Ġsize Ġof Ġ5 Ġduring Ġinference . ĠAll Ġthe Ġmodels Ġare Ġimplemented Ġin ĠOpen N MT - py ĠB IB REF 38 . ĠAll Ġcode Ġis Ġavailable Ġat Ġhttps :// github . com / K ai ju ML / data - to - text - h ier arch ical </s> Results </s> Our Ġresults Ġon Ġthe ĠR oto Wire Ġtest set Ġare Ġsummarized Ġin ĠTable ĠT AB REF 25 . ĠFor Ġeach Ġproposed Ġvariant Ġof Ġour Ġarchitecture , Ġwe Ġreport Ġthe Ġmean Ġscore Ġover Ġten Ġruns , Ġas Ġwell Ġas Ġthe Ġstandard Ġdeviation Ġin Ġsubscript . ĠResults Ġare Ġcompared Ġto Ġbas elines ĠB IB REF 12 , ĠB IB REF 13 , ĠB IB REF 10 Ġand Ġvariants Ġof Ġour Ġmodels . ĠWe Ġalso Ġreport Ġthe Ġresult Ġof Ġthe Ġor acle Ġ( met rics Ġon Ġthe Ġgold Ġdescriptions ). ĠPlease Ġnote Ġthat Ġgold Ġdescriptions Ġtriv ially Ġobtain Ġ100 % Ġon Ġall Ġmetrics Ġexpect ĠRG , Ġas Ġthey Ġare Ġall Ġbased Ġon Ġcomparison Ġwith Ġthemselves . ĠRG Ġscores Ġare Ġdifferent , Ġas Ġthe ĠIE Ġsystem Ġis Ġimperfect Ġand Ġfails Ġto Ġextract Ġaccurate Ġentities Ġ4 % Ġof Ġthe Ġtime . ĠRG - # Ġis Ġan Ġabsolute Ġcount . </s> Results Ġ: :: ĠAb lation Ġstudies </s> To Ġevaluate Ġthe Ġimpact Ġof Ġour Ġmodel Ġcomponents , Ġwe Ġfirst Ġcompare Ġscenarios ĠFlat , ĠHier arch ical - k , Ġand ĠHier arch ical - k v . ĠAs Ġshown Ġin ĠTable ĠT AB REF 25 , Ġwe Ġcan Ġsee Ġthe Ġlower Ġresults Ġobtained Ġby Ġthe ĠFlat Ġscenario Ġcompared Ġto Ġthe Ġother Ġscenarios Ġ( e . g . ĠB LE U Ġ$ 16 . 7 $ Ġvs . Ġ$ 17 . 5 $ Ġfor Ġresp . ĠFlat Ġand ĠHier arch ical - k ), Ġsuggesting Ġthe Ġeffectiveness Ġof Ġencoding Ġthe Ġdata - st ructure Ġusing Ġa Ġhierarchy . ĠThis Ġis Ġexpected , Ġas Ġlosing Ġexplicit Ġdelim itation Ġbetween Ġentities Ġmakes Ġit Ġharder Ġa ) Ġfor Ġthe Ġenc oder Ġto Ġencode Ġsemantics Ġof Ġthe Ġobjects Ġcontained Ġin Ġthe Ġtable Ġand Ġb ) Ġfor Ġthe Ġattention Ġmechanism Ġto Ġextract Ġsalient Ġentities / rec ords . </s> Second , Ġthe Ġcomparison Ġbetween Ġscenario ĠHier arch ical - k v Ġand ĠHier arch ical - k Ġshows Ġthat Ġo mitting Ġentirely Ġthe Ġinfluence Ġof Ġthe Ġrecord Ġvalues Ġin Ġthe Ġattention Ġmechanism Ġis Ġmore Ġeffective : Ġthis Ġlast Ġvariant Ġperforms Ġslightly Ġbetter Ġin Ġall Ġmetrics Ġexcept ed ĠCS - R %, Ġreinforcing Ġour Ġintuition Ġthat Ġfocusing Ġon Ġthe Ġstructure Ġmodeling Ġis Ġan Ġimportant Ġpart Ġof Ġdata Ġencoding Ġas Ġwell Ġas Ġconfirming Ġthe Ġintuition Ġexplained Ġin ĠSection ĠSEC REF 16 : Ġonce Ġan Ġentity Ġis Ġselected , Ġfacts Ġabout Ġthis Ġentity Ġare Ġrelevant Ġbased Ġon Ġtheir Ġkey , Ġnot Ġvalue Ġwhich Ġmight Ġadd Ġnoise . ĠTo Ġillustrate Ġthis Ġintuition , Ġwe Ġdepict Ġin ĠFigure ĠFIG REF 27 Ġattention Ġscores Ġ( rec all Ġ$\ alpha Ġ_ { i , t } $ Ġand Ġ$\ beta Ġ_ { i , j , t } $ Ġfrom Ġequations Ġ( SEC REF 16 ) Ġand Ġ( DIS PLAY _ FORM 17 )) Ġfor Ġboth Ġvariants ĠHier arch ical - k v Ġand ĠHier arch ical - k . ĠWe Ġparticularly Ġfocus Ġon Ġthe Ġtimestamp Ġwhere Ġthe Ġmodels Ġshould Ġmention Ġthe Ġnumber Ġof Ġpoints Ġscored Ġduring Ġthe Ġfirst Ġquarter Ġof Ġthe Ġgame . ĠScores Ġof ĠHier arch ical - k Ġare Ġsharp , Ġwith Ġall Ġof Ġthe Ġweight Ġon Ġthe Ġcorrect Ġrecord Ġ( P TS _ Q TR 1 , Ġ26 ) Ġwhereas Ġscores Ġof ĠHier arch ical - k v Ġare Ġmore Ġdistributed Ġover Ġall ĠPTS _ Q TR Ġrecords , Ġultimately Ġfailing Ġto Ġretrieve Ġthe Ġcorrect Ġone . </s> Results Ġ: :: ĠComparison Ġw . r . t . Ġbas elines . </s> From Ġa Ġgeneral Ġpoint Ġof Ġview , Ġwe Ġcan Ġsee Ġfrom ĠTable ĠT AB REF 25 Ġthat Ġour Ġscenarios Ġobtain Ġsignificantly Ġhigher Ġresults Ġin Ġterms Ġof ĠB LE U Ġover Ġall Ġmodels ; Ġour Ġbest Ġmodel ĠHier arch ical - k Ġreaching Ġ$ 17 . 5 $ Ġvs . Ġ$ 16 . 5 $ Ġagainst Ġthe Ġbest Ġbaseline . ĠThis Ġmeans Ġthat Ġour Ġmodels Ġlearns Ġto Ġgenerate Ġfluent Ġsequences Ġof Ġwords , Ġclose Ġto Ġthe Ġgold Ġdescriptions , Ġadequately Ġpicking Ġup Ġon Ġdomain Ġling o . ĠQual itative Ġmetrics Ġare Ġeither Ġbetter Ġor Ġon Ġpar Ġwith Ġbas elines . ĠWe Ġshow Ġin ĠFigure ĠFIG REF 29 Ġa Ġtext Ġgenerated Ġby Ġour Ġbest Ġmodel , Ġwhich Ġcan Ġbe Ġdirectly Ġcompared Ġto Ġthe Ġgold Ġdescription Ġin ĠFigure ĠFIG REF 1 . ĠGeneration Ġis Ġfluent Ġand Ġcontains Ġdomain - specific Ġexpressions . ĠAs Ġreflected Ġin ĠTable ĠT AB REF 25 , Ġthe Ġnumber Ġof Ġcorrect Ġmentions Ġ( in Ġgreen ) Ġoutwe ights Ġthe Ġnumber Ġof Ġincorrect Ġmentions Ġ( in Ġred ). ĠPlease Ġnote Ġthat , Ġas Ġin Ġprevious Ġwork ĠB IB REF 28 , ĠB IB REF 12 , ĠB IB REF 13 , ĠB IB REF 10 , Ġgenerated Ġtexts Ġstill Ġcontain Ġa Ġnumber Ġof Ġincorrect Ġfacts , Ġas Ġwell Ġhallucinations Ġ( in Ġblue ): Ġsentences Ġthat Ġhave Ġno Ġbasis Ġin Ġthe Ġinput Ġdata Ġ( e . g . ĠâĢ ľ [ ...] Ġhe 's Ġnow Ġaveraging Ġ22 Ġpoints Ġ[...] ." ). ĠWhile Ġnot Ġthe Ġdirect Ġfocus Ġof Ġour Ġwork , Ġthis Ġhighlights Ġthat Ġany Ġoperation Ġmeant Ġto Ġenrich Ġthe Ġsemantics Ġof Ġstructured Ġdata Ġcan Ġalso Ġenrich Ġthe Ġdata Ġwith Ġincorrect Ġfacts . </s> Specifically , Ġregarding Ġall Ġbas elines , Ġwe Ġcan Ġoutline Ġthe Ġfollowing Ġstatements . </s> $ \ bul let Ġ$ ĠOur Ġhierarchical Ġmodels Ġachieve Ġsignificantly Ġbetter Ġscores Ġon Ġall Ġmetrics Ġwhen Ġcompared Ġto Ġthe Ġflat Ġarchitecture ĠWis eman , Ġreinforcing Ġthe Ġcrucial Ġrole Ġof Ġstructure Ġin Ġdata Ġsemantics Ġand Ġsal iency . ĠThe Ġanalysis Ġof ĠRG Ġmetrics Ġshows Ġthat ĠWis eman Ġseems Ġto Ġbe Ġthe Ġmore Ġnatural istic Ġin Ġterms Ġof Ġnumber Ġof Ġfactual Ġmentions Ġ( RG # ) Ġsince Ġit Ġis Ġthe Ġclosest Ġscenario Ġto Ġthe Ġgold Ġvalue Ġ( 16 . 83 Ġvs . Ġ17 . 31 Ġfor Ġresp . ĠWis eman Ġand ĠHier arch ical - k ). ĠHowever , ĠWis eman Ġachieves Ġonly Ġ$ 75 . 62 $ % Ġof Ġprecision , Ġeffectively Ġmentioning Ġon Ġaverage Ġa Ġtotal Ġof Ġ$ 22 . 25 $ Ġrecords Ġ( wrong Ġor Ġaccurate ), Ġwhere Ġour Ġmodel ĠHier arch ical - k Ġscores Ġa Ġprecision Ġof Ġ$ 89 . 46 $ %, Ġleading Ġto Ġ$ 23 . 66 $ Ġtotal Ġmentions , Ġjust Ġslightly Ġabove ĠWis eman . </s> $ \ bul let Ġ$ ĠThe Ġcomparison Ġbetween Ġthe ĠFlat Ġscenario Ġand ĠWis eman Ġis Ġparticularly Ġinteresting . ĠIndeed , Ġthese Ġtwo Ġmodels Ġshare Ġthe Ġsame Ġintuition Ġto Ġflatt en Ġthe Ġdata - st ructure . ĠThe Ġonly Ġdifference Ġstands Ġon Ġthe Ġenc oder Ġmechanism : Ġbi - L ST M Ġvs . ĠTrans former , Ġfor ĠWis eman Ġand ĠFlat Ġrespectively . ĠResults Ġshows Ġthat Ġour ĠFlat Ġscenario Ġob tains Ġa Ġsignificant Ġhigher ĠB LE U Ġscore Ġ( 16 . 7 Ġvs . Ġ14 . 5 ) Ġand Ġgenerates Ġfluent Ġdescriptions Ġwith Ġaccurate Ġmentions Ġ( RG - P %) Ġthat Ġare Ġalso Ġincluded Ġin Ġthe Ġgold Ġdescriptions Ġ( CS - R %). ĠThis Ġsuggests Ġthat Ġintroducing Ġthe ĠTrans former Ġarchitecture Ġis Ġpromising Ġway Ġto Ġimplicitly Ġaccount Ġfor Ġdata Ġstructure . </s> $ \ bul let Ġ$ ĠOur Ġhierarchical Ġmodels Ġoutper form Ġthe Ġtwo - step Ġdec od ers Ġof ĠLi Ġand ĠP ud upp ully - plan Ġon Ġboth ĠB LE U Ġand Ġall Ġqualitative Ġmetrics , Ġshowing Ġthat Ġcapturing Ġstructure Ġin Ġthe Ġencoding Ġprocess Ġis Ġmore Ġeffective Ġthat Ġpredicting Ġa Ġstructure Ġin Ġthe Ġdec oder Ġ( i . e ., Ġplanning Ġor Ġtem pl ating ). ĠWhile Ġour Ġmodels Ġsens ibly Ġoutper form Ġin Ġprecision Ġat Ġfactual Ġmentions , Ġthe Ġbaseline ĠP ud upp ully - plan Ġreaches Ġ$ 34 . 28 $ Ġmentions Ġon Ġaverage , Ġshowing Ġthat Ġincorporating Ġmodules Ġdedicated Ġto Ġentity Ġextraction Ġleads Ġto Ġover - f ocusing Ġon Ġentities ; Ġcontrasting Ġwith Ġour Ġmodels Ġthat Ġlearn Ġto Ġgenerate Ġmore Ġbalanced Ġdescriptions . </s> $ \ bul let Ġ$ ĠThe Ġcomparison Ġwith ĠP ud upp ully - up dt Ġshows Ġthat Ġdynamically Ġupdating Ġthe Ġencoding Ġacross Ġthe Ġgeneration Ġprocess Ġcan Ġlead Ġto Ġbetter ĠContent ĠOrder ing Ġ( CO ) Ġand ĠRG - P %. ĠHowever , Ġthis Ġdoes Ġnot Ġhelp Ġwith ĠContent ĠSelection Ġ( CS ) Ġsince Ġour Ġbest Ġmodel ĠHier arch ical - k Ġob tains Ġslightly Ġbetter Ġscores . ĠIndeed , ĠP ud upp ully - up dt Ġupdates Ġrepresentations Ġafter Ġeach Ġmention Ġallowing Ġto Ġkeep Ġtrack Ġof Ġthe Ġmention Ġhistory . ĠThis Ġguides Ġthe Ġordering Ġof Ġmentions Ġ( CO Ġmetric ), Ġeach Ġstep Ġlimiting Ġmore Ġthe Ġnumber Ġof Ġcandidate Ġmentions Ġ( increasing ĠRG - P %). ĠIn Ġcontrast , Ġour Ġmodel Ġenc odes Ġsal iency Ġamong Ġrecords / ent ities Ġmore Ġeffectively Ġ( CS Ġmetric ). ĠWe Ġnote Ġthat Ġwhile Ġour Ġmodel Ġenc odes Ġthe Ġdata - st ructure Ġonce Ġand Ġfor Ġall , ĠP ud upp ully - up dt Ġrecomp utes , Ġvia Ġthe Ġupdates , Ġthe Ġencoding Ġat Ġeach Ġstep Ġand Ġtherefore Ġsignificantly Ġincreases Ġcomputation Ġcomplexity . ĠCombined Ġwith Ġtheir ĠRG - # Ġscore Ġof Ġ$ 30 . 11 $, Ġwe Ġargue Ġthat Ġour Ġmodel Ġis Ġsimpler , Ġand Ġob tains Ġfluent Ġdescription Ġwith Ġaccurate Ġmentions Ġin Ġa Ġmore Ġhuman - like Ġfashion . </s> We Ġwould Ġalso Ġlike Ġto Ġdraw Ġattention Ġto Ġthe Ġnumber Ġof Ġparameters Ġused Ġby Ġthose Ġarchitectures . ĠWe Ġnote Ġthat Ġour Ġscenarios Ġrelies Ġon Ġa Ġlower Ġnumber Ġof Ġparameters Ġ( 14 Ġmillions ) Ġcompared Ġto Ġall Ġbas elines Ġ( ranging Ġfrom Ġ23 Ġto Ġ45 Ġmillions ). ĠThis Ġoutlines Ġthe Ġeffectiveness Ġin Ġthe Ġdesign Ġof Ġour Ġmodel Ġrelying Ġon Ġa Ġstructure Ġencoding , Ġin Ġcontrast Ġto Ġother Ġapproach Ġthat Ġtry Ġto Ġlearn Ġthe Ġstructure Ġof Ġdata / desc ript ions Ġfrom Ġa Ġlinear ized Ġencoding . </s> Conclusion Ġand Ġfuture Ġwork </s> In Ġthis Ġwork Ġwe Ġhave Ġproposed Ġa Ġhierarchical Ġenc oder Ġfor Ġstructured Ġdata , Ġwhich Ġ1 ) Ġlever ages Ġthe Ġstructure Ġto Ġform Ġefficient Ġrepresentation Ġof Ġits Ġinput ; Ġ2 ) Ġhas Ġstrong Ġsynergy Ġwith Ġthe Ġhierarchical Ġattention Ġof Ġits Ġassociated Ġdec oder . ĠThis Ġresults Ġin Ġan Ġeffective Ġand Ġmore Ġlight - weight Ġmodel . ĠExperimental Ġevaluation Ġon Ġthe ĠR oto Wire Ġbenchmark Ġshows Ġthat Ġour Ġmodel Ġoutper forms Ġcompetitive Ġbas elines Ġin Ġterms Ġof ĠB LE U Ġscore Ġand Ġis Ġgenerally Ġbetter Ġon Ġqualitative Ġmetrics . ĠThis Ġway Ġof Ġrepresenting Ġstructured Ġdatabases Ġmay Ġlead Ġto Ġautomatic Ġinference Ġand Ġenrichment , Ġe . g ., Ġby Ġcomparing Ġentities . ĠThis Ġdirection Ġcould Ġbe Ġdriven Ġby Ġvery Ġrecent Ġoperation - guided Ġnetworks ĠB IB REF 39 , ĠB IB REF 40 . ĠIn Ġaddition , Ġwe Ġnote Ġthat Ġour Ġapproach Ġcan Ġstill Ġlead Ġto Ġerroneous Ġfacts Ġor Ġeven Ġhallucinations . ĠAn Ġinteresting Ġperspective Ġmight Ġbe Ġto Ġfurther Ġconst rain Ġthe Ġmodel Ġon Ġthe Ġdata Ġstructure Ġin Ġorder Ġto Ġprevent Ġinaccurate Ġof Ġeven Ġcontradictory Ġdescriptions . </s> Acknowled gements </s> We Ġwould Ġlike Ġto Ġthank Ġthe ĠH 2020 Ġproject ĠAI 4 EU Ġ( 8 256 19 ) Ġwhich Ġpartially Ġsupports ĠLaure ĠSou lier Ġand ĠPatrick ĠGall in ari .