about 500
ABSTRACTIVE
Unanswerable
NONE
The data collection was performed separately for models in the WikiBio-Systems and WikiBio-Hyperparams categories. 1100 tables were sampled from the development set, and for each table we got 8 different sentence pairs annotated across the two categories, resulting in a total of 8800 pairwise comparisons. Each pair was judged by one worker only which means there may be noise at the instance-level, but the aggregated system-level scores had low variance (cf. Table TABREF32 ). In total around 500 different workers were involved in the annotation. References were also included in the evaluation, and they received a lower score than PG-Net, highlighting the divergence in WikiBio.
1906.01081
Introduction </s> The Ġtask Ġof Ġgenerating Ġnatural Ġlanguage Ġdescriptions Ġof Ġstructured Ġdata Ġ( such Ġas Ġtables ) ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġhas Ġseen Ġa Ġgrowth Ġin Ġinterest Ġwith Ġthe Ġrise Ġof Ġsequence Ġto Ġsequence Ġmodels Ġthat Ġprovide Ġan Ġeasy Ġway Ġof Ġencoding Ġtables Ġand Ġgenerating Ġtext Ġfrom Ġthem ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ. </s> For Ġtext Ġgeneration Ġtasks , Ġthe Ġonly Ġgold Ġstandard Ġmetric Ġis Ġto Ġshow Ġthe Ġoutput Ġto Ġhumans Ġfor Ġjudging Ġits Ġquality , Ġbut Ġthis Ġis Ġtoo Ġexpensive Ġto Ġapply Ġrepeatedly Ġanytime Ġsmall Ġmodifications Ġare Ġmade Ġto Ġa Ġsystem . ĠHence , Ġautomatic Ġmetrics Ġthat Ġcompare Ġthe Ġgenerated Ġtext Ġto Ġone Ġor Ġmore Ġreference Ġtexts Ġare Ġroutinely Ġused Ġto Ġcompare Ġmodels ĠB IB REF 7 Ġ. ĠFor Ġtable - to - text Ġgeneration , Ġautomatic Ġevaluation Ġhas Ġlargely Ġrelied Ġon ĠB LE U ĠB IB REF 8 Ġand ĠR OU GE ĠB IB REF 9 Ġ. ĠThe Ġunderlying Ġassumption Ġbehind Ġthese Ġmetrics Ġis Ġthat Ġthe Ġreference Ġtext Ġis Ġgold - standard , Ġi . e ., Ġit Ġis Ġthe Ġideal Ġtarget Ġtext Ġthat Ġa Ġsystem Ġshould Ġgenerate . ĠIn Ġpractice , Ġhowever , Ġwhen Ġdatasets Ġare Ġcollected Ġautomatically Ġand Ġhe ur istically , Ġthe Ġreference Ġtexts Ġare Ġoften Ġnot Ġideal . ĠFigure ĠFIG REF 2 Ġshows Ġan Ġexample Ġfrom Ġthe ĠWiki Bio Ġdataset ĠB IB REF 0 Ġ. ĠHere Ġthe Ġreference Ġcontains Ġextra Ġinformation Ġwhich Ġno Ġsystem Ġcan Ġbe Ġexpected Ġto Ġproduce Ġgiven Ġonly Ġthe Ġassociated Ġtable . ĠWe Ġcall Ġsuch Ġreference Ġtexts Ġdiver gent Ġfrom Ġthe Ġtable . </s> We Ġshow Ġthat Ġexisting Ġautomatic Ġmetrics , Ġincluding ĠB LE U , Ġcorrelate Ġpoorly Ġwith Ġhuman Ġjudgments Ġwhen Ġthe Ġevaluation Ġsets Ġcontain Ġdiver gent Ġreferences Ġ( Â§ ĠSEC REF 36 Ġ). ĠFor Ġmany Ġtable - to - text Ġgeneration Ġtasks , Ġthe Ġtables Ġthemselves Ġare Ġin Ġa Ġpseudo - natural Ġlanguage Ġformat Ġ( e . g ., ĠWiki Bio , ĠWeb NL G ĠB IB REF 6 Ġ, Ġand ĠE 2 E - NL G ĠB IB REF 10 Ġ). ĠIn Ġsuch Ġcases Ġwe Ġpropose Ġto Ġcompare Ġthe Ġgenerated Ġtext Ġto Ġthe Ġunderlying Ġtable Ġas Ġwell Ġto Ġimprove Ġevaluation . ĠWe Ġdevelop Ġa Ġnew Ġmetric , ĠPAR ENT Ġ( Pre cision ĠAnd ĠRecall Ġof ĠEnt ailed ĠN - gram s Ġfrom Ġthe ĠTable ) Ġ( Â§ ĠSEC REF 3 Ġ). ĠWhen Ġcomputing Ġprecision , ĠPAR ENT Ġeffectively Ġuses Ġa Ġunion Ġof Ġthe Ġreference Ġand Ġthe Ġtable , Ġto Ġreward Ġcorrect Ġinformation Ġmissing Ġfrom Ġthe Ġreference . ĠWhen Ġcomputing Ġrecall , Ġit Ġuses Ġan Ġintersection Ġof Ġthe Ġreference Ġand Ġthe Ġtable , Ġto Ġignore Ġextra Ġincorrect Ġinformation Ġin Ġthe Ġreference . ĠThe Ġunion Ġand Ġintersection Ġare Ġcomputed Ġwith Ġthe Ġhelp Ġof Ġan Ġentail ment Ġmodel Ġto Ġdecide Ġif Ġa Ġtext Ġn - gram Ġis Ġent ailed Ġby Ġthe Ġtable . ĠWe Ġshow Ġthat Ġthis Ġmethod Ġis Ġmore Ġeffective Ġthan Ġusing Ġthe Ġtable Ġas Ġan Ġadditional Ġreference . ĠOur Ġmain Ġcontributions Ġare : </s> Table - to - Text ĠGeneration </s> We Ġbriefly Ġreview Ġthe Ġtask Ġof Ġgenerating Ġnatural Ġlanguage Ġdescriptions Ġof Ġsemi - struct ured Ġdata , Ġwhich Ġwe Ġrefer Ġto Ġas Ġtables Ġhence forth ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ. ĠTables Ġcan Ġbe Ġexpressed Ġas Ġset Ġof Ġrecords ĠIN LINE FORM 0 Ġ, Ġwhere Ġeach Ġrecord Ġis Ġa Ġtuple Ġ( entity , Ġattribute , Ġvalue ). ĠWhen Ġall Ġthe Ġrecords Ġare Ġabout Ġthe Ġsame Ġentity , Ġwe Ġcan Ġtrunc ate Ġthe Ġrecords Ġto Ġ( attribute , Ġvalue ) Ġpairs . ĠFor Ġexample , Ġfor Ġthe Ġtable Ġin ĠFigure ĠFIG REF 2 Ġ, Ġthe Ġrecords Ġare Ġ{ ( Birth ĠName , ĠMichael ĠDahl quist ), Ġ( Born , ĠDecember Ġ22 Ġ1965 ), Ġ... }. ĠThe Ġtask Ġis Ġto Ġgenerate Ġa Ġtext ĠIN LINE FORM 1 Ġwhich Ġsummarizes Ġthe Ġrecords Ġin Ġa Ġfluent Ġand Ġgram matical Ġmanner . ĠFor Ġtraining Ġand Ġevaluation Ġwe Ġfurther Ġassume Ġthat Ġwe Ġhave Ġa Ġreference Ġdescription ĠIN LINE FORM 2 Ġavailable Ġfor Ġeach Ġtable . ĠWe Ġlet ĠIN LINE FORM 3 Ġdenote Ġan Ġevaluation Ġset Ġof Ġtables , Ġreferences Ġand Ġtexts Ġgenerated Ġfrom Ġa Ġmodel ĠIN LINE FORM 4 Ġ, Ġand ĠIN LINE FORM 5 Ġ, ĠIN LINE FORM 6 Ġdenote Ġthe Ġcollection Ġof Ġn - gram s Ġof Ġorder ĠIN LINE FORM 7 Ġin ĠIN LINE FORM 8 Ġand ĠIN LINE FORM 9 Ġ, Ġrespectively . ĠWe Ġuse ĠIN LINE FORM 10 Ġto Ġdenote Ġthe Ġcount Ġof Ġn - gram ĠIN LINE FORM 11 Ġin ĠIN LINE FORM 12 Ġ, Ġand ĠIN LINE FORM 13 Ġto Ġdenote Ġthe Ġminimum Ġof Ġits Ġcounts Ġin ĠIN LINE FORM 14 Ġand ĠIN LINE FORM 15 Ġ. ĠOur Ġgoal Ġis Ġto Ġassign Ġa Ġscore Ġto Ġthe Ġmodel , Ġwhich Ġcorrelates Ġhighly Ġwith Ġhuman Ġjudgments Ġof Ġthe Ġquality Ġof Ġthat Ġmodel . </s> PAR ENT </s> PAR ENT Ġevaluates Ġeach Ġinstance ĠIN LINE FORM 0 Ġseparately , Ġby Ġcomputing Ġthe Ġprecision Ġand Ġrecall Ġof ĠIN LINE FORM 1 Ġagainst Ġboth ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġ. </s> E val uation Ġvia ĠInformation ĠExt raction </s> BI B REF 1 Ġproposed Ġto Ġuse Ġan Ġauxiliary Ġmodel , Ġtrained Ġto Ġextract Ġstructured Ġrecords Ġfrom Ġtext , Ġfor Ġevaluation . ĠHowever , Ġthe Ġextraction Ġmodel Ġpresented Ġin Ġthat Ġwork Ġis Ġlimited Ġto Ġthe Ġclosed - domain Ġsetting Ġof Ġbasketball Ġgame Ġtables Ġand Ġsumm aries . ĠIn Ġparticular , Ġthey Ġassume Ġthat Ġeach Ġtable Ġhas Ġexactly Ġthe Ġsame Ġset Ġof Ġattributes Ġfor Ġeach Ġentity , Ġand Ġthat Ġthe Ġentities Ġcan Ġbe Ġidentified Ġin Ġthe Ġtext Ġvia Ġstring Ġmatching . ĠThese Ġassumptions Ġare Ġnot Ġvalid Ġfor Ġthe Ġopen - domain ĠWiki Bio Ġdataset , Ġand Ġhence Ġwe Ġtrain Ġour Ġown Ġextraction Ġmodel Ġto Ġreplicate Ġtheir Ġevaluation Ġscheme . </s> Our Ġextraction Ġsystem Ġis Ġa Ġpointer - gener ator Ġnetwork ĠB IB REF 19 Ġ, Ġwhich Ġlearns Ġto Ġproduce Ġa Ġlinear ized Ġversion Ġof Ġthe Ġtable Ġfrom Ġthe Ġtext . ĠThe Ġnetwork Ġlearns Ġwhich Ġattributes Ġneed Ġto Ġbe Ġpopulated Ġin Ġthe Ġoutput Ġtable , Ġalong Ġwith Ġtheir Ġvalues . ĠIt Ġis Ġtrained Ġon Ġthe Ġtraining Ġset Ġof ĠWiki Bio . ĠAt Ġtest Ġtime Ġwe Ġparsed Ġthe Ġoutput Ġstrings Ġinto Ġa Ġset Ġof Ġ( attribute , Ġvalue ) Ġtu ples Ġand Ġcompare Ġit Ġto Ġthe Ġground Ġtruth Ġtable . ĠThe ĠF - score Ġof Ġthis Ġtext - to - table Ġsystem Ġwas ĠIN LINE FORM 0 Ġ, Ġwhich Ġis Ġcomparable Ġto Ġother Ġchallenging Ġopen - domain Ġsettings ĠB IB REF 20 Ġ. ĠMore Ġdetails Ġare Ġincluded Ġin Ġthe ĠAppendix ĠSEC REF 52 Ġ. </s> Given Ġthis Ġinformation Ġextraction Ġsystem , Ġwe Ġconsider Ġthe Ġfollowing Ġmetrics Ġfor Ġevaluation , Ġalong Ġthe Ġlines Ġof ĠB IB REF 1 Ġ. ĠContent ĠSelection Ġ( CS ): ĠF - score Ġfor Ġthe Ġ( attribute , Ġvalue ) Ġpairs Ġextracted Ġfrom Ġthe Ġgenerated Ġtext Ġcompared Ġto Ġthose Ġextracted Ġfrom Ġthe Ġreference . ĠRel ation ĠGeneration Ġ( RG ): ĠPrecision Ġfor Ġthe Ġ( attribute , Ġvalue ) Ġpairs Ġextracted Ġfrom Ġthe Ġgenerated Ġtext Ġcompared Ġto Ġthose Ġin Ġthe Ġground Ġtruth Ġtable . ĠRG - F : ĠSince Ġour Ġtask Ġemphasizes Ġthe Ġrecall Ġof Ġinformation Ġfrom Ġthe Ġtable Ġas Ġwell , Ġwe Ġconsider Ġanother Ġvariant Ġwhich Ġcomp utes Ġthe ĠF - score Ġof Ġthe Ġextracted Ġpairs Ġto Ġthose Ġin Ġthe Ġtable . ĠWe Ġomit Ġthe Ġcontent Ġordering Ġmetric , Ġsince Ġour Ġextraction Ġsystem Ġdoes Ġnot Ġalign Ġrecords Ġto Ġthe Ġinput Ġtext . </s> Exper iments Ġ& ĠResults </s> In Ġthis Ġsection Ġwe Ġcompare Ġseveral Ġautomatic Ġevaluation Ġmetrics Ġby Ġchecking Ġtheir Ġcorrelation Ġwith Ġthe Ġscores Ġassigned Ġby Ġhumans Ġto Ġtable - to - text Ġmodels . ĠSpecifically , Ġgiven ĠIN LINE FORM 0 Ġmodels ĠIN LINE FORM 1 Ġ, Ġand Ġtheir Ġoutputs Ġon Ġan Ġevaluation Ġset , Ġwe Ġshow Ġthese Ġgenerated Ġtexts Ġto Ġhumans Ġto Ġjudge Ġtheir Ġquality , Ġand Ġobtain Ġaggreg ated Ġhuman Ġevaluation Ġscores Ġfor Ġall Ġthe Ġmodels , ĠIN LINE FORM 2 Ġ( Â§ ĠSEC REF 33 Ġ). ĠNext , Ġto Ġevaluate Ġan Ġautomatic Ġmetric , Ġwe Ġcompute Ġthe Ġscores Ġit Ġassigns Ġto Ġeach Ġmodel , ĠIN LINE FORM 3 Ġ, Ġand Ġcheck Ġthe ĠPearson Ġcorrelation Ġbetween ĠIN LINE FORM 4 Ġand ĠIN LINE FORM 5 ĠB IB REF 21 Ġ. </s> Data Ġ& ĠModels </s> Our Ġmain Ġexperiments Ġare Ġon Ġthe ĠWiki Bio Ġdataset ĠB IB REF 0 Ġ, Ġwhich Ġis Ġautomatically Ġconstructed Ġand Ġcontains Ġmany Ġdiver gent Ġreferences . ĠIn ĠÂ§ ĠSEC REF 47 Ġwe Ġalso Ġpresent Ġresults Ġon Ġthe Ġdata Ġreleased Ġas Ġpart Ġof Ġthe ĠWeb NL G Ġchallenge . </s> We Ġdeveloped Ġseveral Ġmodels Ġof Ġvarying Ġquality Ġfor Ġgenerating Ġtext Ġfrom Ġthe Ġtables Ġin ĠWiki Bio . ĠThis Ġgives Ġus Ġa Ġdiverse Ġset Ġof Ġoutputs Ġto Ġevaluate Ġthe Ġautomatic Ġmetrics Ġon . ĠTable ĠT AB REF 32 Ġlists Ġthe Ġmodels Ġalong Ġwith Ġtheir Ġhyper param eter Ġsettings Ġand Ġtheir Ġscores Ġfrom Ġthe Ġhuman Ġevaluation Ġ( Â§ ĠSEC REF 33 Ġ). ĠOur Ġfocus Ġis Ġprimarily Ġon Ġneural Ġsequence - to - sequence Ġmethods Ġsince Ġthese Ġare Ġmost Ġwidely Ġused , Ġbut Ġwe Ġalso Ġinclude Ġa Ġtemplate - based Ġbaseline . ĠAll Ġneural Ġmodels Ġwere Ġtrained Ġon Ġthe ĠWiki Bio Ġtraining Ġset . ĠTraining Ġdetails Ġand Ġsample Ġoutputs Ġare Ġincluded Ġin ĠApp end ices ĠSEC REF 56 Ġ& ĠSEC REF 57 Ġ. </s> We Ġdivide Ġthese Ġmodels Ġinto Ġtwo Ġcategories Ġand Ġmeasure Ġcorrelation Ġseparately Ġfor Ġboth Ġthe Ġcategories . ĠThe Ġfirst Ġcategory , ĠWiki Bio - System s , Ġincludes Ġone Ġmodel Ġeach Ġfrom Ġthe Ġfour Ġfamilies Ġlisted Ġin ĠTable ĠT AB REF 32 Ġ. ĠThis Ġcategory Ġtests Ġwhether Ġa Ġmetric Ġcan Ġbe Ġused Ġto Ġcompare Ġdifferent Ġmodel Ġfamilies Ġwith Ġa Ġlarge Ġvariation Ġin Ġthe Ġquality Ġof Ġtheir Ġoutputs . ĠThe Ġsecond Ġcategory , ĠWiki Bio - Hyper params , Ġincludes Ġ13 Ġdifferent Ġhyper param eter Ġsettings Ġof ĠPG - Net ĠB IB REF 19 Ġ, Ġwhich Ġwas Ġthe Ġbest Ġperforming Ġsystem Ġoverall . Ġ9 Ġof Ġthese Ġwere Ġobtained Ġby Ġvarying Ġthe Ġbeam Ġsize Ġand Ġlength Ġnormal ization Ġpenalty Ġof Ġthe Ġdec oder Ġnetwork ĠB IB REF 23 Ġ, Ġand Ġthe Ġremaining Ġ4 Ġwere Ġobtained Ġby Ġre - scoring Ġbeams Ġof Ġsize Ġ8 Ġwith Ġthe Ġinformation Ġextraction Ġmodel Ġdescribed Ġin ĠÂ§ ĠSEC REF 4 Ġ. ĠAll Ġthe Ġmodels Ġin Ġthis Ġcategory Ġproduce Ġhigh Ġquality Ġfluent Ġtexts , Ġand Ġdiffer Ġprimarily Ġon Ġthe Ġquantity Ġand Ġaccuracy Ġof Ġthe Ġinformation Ġthey Ġexpress . ĠHere Ġwe Ġare Ġtesting Ġwhether Ġa Ġmetric Ġcan Ġbe Ġused Ġto Ġcompare Ġsimilar Ġsystems Ġwith Ġa Ġsmall Ġvariation Ġin Ġperformance . ĠThis Ġis Ġan Ġimportant Ġuse - case Ġas Ġmetrics Ġare Ġoften Ġused Ġto Ġtune Ġhyper param eters Ġof Ġa Ġmodel . </s> Human ĠEvaluation </s> We Ġcollected Ġhuman Ġjudgments Ġon Ġthe Ġquality Ġof Ġthe Ġ16 Ġmodels Ġtrained Ġfor ĠWiki Bio , Ġplus Ġthe Ġreference Ġtexts . ĠWorkers Ġon Ġa Ġcrowd - s ourcing Ġplatform , Ġproficient Ġin ĠEnglish , Ġwere Ġshown Ġa Ġtable Ġwith Ġpairs Ġof Ġgenerated Ġtexts , Ġor Ġa Ġgenerated Ġtext Ġand Ġthe Ġreference , Ġand Ġasked Ġto Ġselect Ġthe Ġone Ġthey Ġprefer . ĠFigure ĠFIG REF 34 Ġshows Ġthe Ġinstructions Ġthey Ġwere Ġgiven . ĠP aired Ġcomparisons Ġhave Ġbeen Ġshown Ġto Ġbe Ġsuperior Ġto Ġrating Ġscales Ġfor Ġcomparing Ġgenerated Ġtexts ĠB IB REF 24 Ġ. ĠHowever , Ġfor Ġmeasuring Ġcorrelation Ġthe Ġcomparisons Ġneed Ġto Ġbe Ġaggreg ated Ġinto Ġreal - valued Ġscores , ĠIN LINE FORM 0 Ġ, Ġfor Ġeach Ġof Ġthe ĠIN LINE FORM 1 Ġmodels . ĠFor Ġthis , Ġwe Ġuse ĠThur stone 's Ġmethod ĠB IB REF 22 Ġ, Ġwhich Ġassigns Ġa Ġscore Ġto Ġeach Ġmodel Ġbased Ġon Ġhow Ġmany Ġtimes Ġit Ġwas Ġpreferred Ġover Ġan Ġalternative . </s> The Ġdata Ġcollection Ġwas Ġperformed Ġseparately Ġfor Ġmodels Ġin Ġthe ĠWiki Bio - System s Ġand ĠWiki Bio - Hyper params Ġcategories . Ġ1100 Ġtables Ġwere Ġsampled Ġfrom Ġthe Ġdevelopment Ġset , Ġand Ġfor Ġeach Ġtable Ġwe Ġgot Ġ8 Ġdifferent Ġsentence Ġpairs Ġannot ated Ġacross Ġthe Ġtwo Ġcategories , Ġresulting Ġin Ġa Ġtotal Ġof Ġ8 800 Ġpair wise Ġcomparisons . ĠEach Ġpair Ġwas Ġjudged Ġby Ġone Ġworker Ġonly Ġwhich Ġmeans Ġthere Ġmay Ġbe Ġnoise Ġat Ġthe Ġinstance - level , Ġbut Ġthe Ġaggreg ated Ġsystem - level Ġscores Ġhad Ġlow Ġvariance Ġ( cf . ĠTable ĠT AB REF 32 Ġ). ĠIn Ġtotal Ġaround Ġ500 Ġdifferent Ġworkers Ġwere Ġinvolved Ġin Ġthe Ġannotation . ĠReferences Ġwere Ġalso Ġincluded Ġin Ġthe Ġevaluation , Ġand Ġthey Ġreceived Ġa Ġlower Ġscore Ġthan ĠPG - Net , Ġhighlighting Ġthe Ġdivergence Ġin ĠWiki Bio . </s> Compared ĠMet rics </s> Text Ġonly : ĠWe Ġcompare ĠB LE U ĠB IB REF 8 Ġ, ĠR OU GE ĠB IB REF 9 Ġ, ĠMET E OR ĠB IB REF 18 Ġ, ĠC ID Er Ġand ĠC ID Er - D ĠB IB REF 25 Ġusing Ġtheir Ġpublicly Ġavailable Ġimplementations . </s> Information ĠExt raction Ġbased : ĠWe Ġcompare Ġthe ĠCS , ĠRG Ġand ĠRG - F Ġmetrics Ġdiscussed Ġin ĠÂ§ ĠSEC REF 4 Ġ. </s> Text Ġ& ĠTable : ĠWe Ġcompare Ġa Ġvariant Ġof ĠB LE U , Ġden oted Ġas ĠB LE U - T , Ġwhere Ġthe Ġvalues Ġfrom Ġthe Ġtable Ġare Ġused Ġas Ġadditional Ġreferences . ĠB LE U - T Ġdraws Ġinspiration Ġfrom Ġi BLE U ĠB IB REF 26 Ġbut Ġinstead Ġrewards Ġn - gram s Ġwhich Ġmatch Ġthe Ġtable Ġrather Ġthan Ġpenal izing Ġthem . ĠFor ĠPAR ENT , Ġwe Ġcompare Ġboth Ġthe Ġword - over lap Ġmodel Ġ( PAR ENT - W ) Ġand Ġthe Ġco - occ urrence Ġmodel Ġ( PAR ENT - C ) Ġfor Ġdetermining Ġentail ment . ĠWe Ġalso Ġcompare Ġversions Ġwhere Ġa Ġsingle ĠIN LINE FORM 0 Ġis Ġtuned Ġon Ġthe Ġentire Ġdataset Ġto Ġmaximize Ġcorrelation Ġwith Ġhuman Ġjudgments , Ġden oted Ġas ĠPAR ENT * - W / C . </s> Cor relation ĠComparison </s> We Ġuse Ġboot strap Ġsampling Ġ( 500 Ġiterations ) Ġover Ġthe Ġ1100 Ġtables Ġfor Ġwhich Ġwe Ġcollected Ġhuman Ġannotations Ġto Ġget Ġan Ġidea Ġof Ġhow Ġthe Ġcorrelation Ġof Ġeach Ġmetric Ġvaries Ġwith Ġthe Ġunderlying Ġdata . ĠIn Ġeach Ġiteration , Ġwe Ġsample Ġwith Ġreplacement , Ġtables Ġalong Ġwith Ġtheir Ġreferences Ġand Ġall Ġthe Ġgenerated Ġtexts Ġfor Ġthat Ġtable . ĠThen Ġwe Ġcompute Ġaggreg ated Ġhuman Ġevaluation Ġand Ġmetric Ġscores Ġfor Ġeach Ġof Ġthe Ġmodels Ġand Ġcompute Ġthe Ġcorrelation Ġbetween Ġthe Ġtwo . ĠWe Ġreport Ġthe Ġaverage Ġcorrelation Ġacross Ġall Ġboot strap Ġsamples Ġfor Ġeach Ġmetric Ġin ĠTable ĠT AB REF 37 Ġ. ĠThe Ġdistribution Ġof Ġcorrelations Ġfor Ġthe Ġbest Ġperforming Ġmetrics Ġare Ġshown Ġin ĠFigure ĠFIG REF 38 Ġ. </s> Table ĠT AB REF 37 Ġalso Ġindicates Ġwhether ĠPAR ENT Ġis Ġsignificantly Ġbetter Ġthan Ġa Ġbaseline Ġmetric . ĠB IB REF 21 Ġsuggest Ġusing Ġthe ĠWilliam 's Ġtest Ġfor Ġthis Ġpurpose , Ġbut Ġsince Ġwe Ġare Ġcomputing Ġcorrelations Ġbetween Ġonly Ġ4 / 13 Ġsystems Ġat Ġa Ġtime , Ġthis Ġtest Ġhas Ġvery Ġweak Ġpower Ġin Ġour Ġcase . ĠHence , Ġwe Ġuse Ġthe Ġboot strap Ġsamples Ġto Ġobtain Ġa ĠIN LINE FORM 0 Ġconfidence Ġinterval Ġof Ġthe Ġdifference Ġin Ġcorrelation Ġbetween ĠPAR ENT Ġand Ġany Ġother Ġmetric Ġand Ġcheck Ġwhether Ġthis Ġis Ġabove Ġ0 ĠB IB REF 27 Ġ. </s> Cor relations Ġare Ġhigher Ġfor Ġthe Ġsystems Ġcategory Ġthan Ġthe Ġhyper params Ġcategory . ĠThe Ġlatter Ġis Ġa Ġmore Ġdifficult Ġsetting Ġsince Ġvery Ġsimilar Ġmodels Ġare Ġcompared , Ġand Ġhence Ġthe Ġvariance Ġof Ġthe Ġcorrelations Ġis Ġalso Ġhigh . ĠCommon ly Ġused Ġmetrics Ġwhich Ġonly Ġrely Ġon Ġthe Ġreference Ġ( BLE U , ĠR OU GE , ĠMET E OR , ĠC ID Er ) Ġhave Ġonly Ġweak Ġcorrelations Ġwith Ġhuman Ġjudgments . ĠIn Ġthe Ġhyper params Ġcategory , Ġthese Ġare Ġoften Ġnegative , Ġimplying Ġthat Ġtuning Ġmodels Ġbased Ġon Ġthese Ġmay Ġlead Ġto Ġselecting Ġworse Ġmodels . ĠB LE U Ġperforms Ġthe Ġbest Ġamong Ġthese , Ġand Ġadding Ġn - gram s Ġfrom Ġthe Ġtable Ġas Ġreferences Ġimproves Ġthis Ġfurther Ġ( BLE U - T ). </s> Among Ġthe Ġextract ive Ġevaluation Ġmetrics , ĠCS , Ġwhich Ġalso Ġonly Ġrelies Ġon Ġthe Ġreference , Ġhas Ġpoor Ġcorrelation Ġin Ġthe Ġhyper params Ġcategory . ĠRG - F , Ġand Ġboth Ġvariants Ġof Ġthe ĠPAR ENT Ġmetric Ġachieve Ġthe Ġhighest Ġcorrelation Ġfor Ġboth Ġsettings . ĠThere Ġis Ġno Ġsignificant Ġdifference Ġamong Ġthese Ġfor Ġthe Ġhyper params Ġcategory , Ġbut Ġfor Ġsystems , ĠPAR ENT - W Ġis Ġsignificantly Ġbetter Ġthan Ġthe Ġother Ġtwo . ĠWhile ĠRG - F Ġneeds Ġa Ġfull Ġinformation Ġextraction Ġpipeline Ġin Ġits Ġimplementation , ĠPAR ENT - C Ġonly Ġrelies Ġon Ġco - occ urrence Ġcounts , Ġand ĠPAR ENT - W Ġcan Ġbe Ġused Ġout - of - the - box Ġfor Ġany Ġdataset . ĠTo Ġour Ġknowledge , Ġthis Ġis Ġthe Ġfirst Ġrigorous Ġevaluation Ġof Ġusing Ġinformation Ġextraction Ġfor Ġgeneration Ġevaluation . </s> On Ġthis Ġdataset , Ġthe Ġword - over lap Ġmodel Ġshowed Ġhigher Ġcorrelation Ġthan Ġthe Ġco - occ urrence Ġmodel Ġfor Ġentail ment . ĠIn ĠÂ§ ĠSEC REF 47 Ġwe Ġwill Ġshow Ġthat Ġfor Ġthe ĠWeb NL G Ġdataset , Ġwhere Ġmore Ġparaph r asing Ġis Ġinvolved Ġbetween Ġthe Ġtable Ġand Ġtext , Ġthe Ġopposite Ġis Ġtrue . ĠLastly , Ġwe Ġnote Ġthat Ġthe Ġhe uristic Ġfor Ġselecting ĠIN LINE FORM 0 Ġis Ġsufficient Ġto Ġproduce Ġhigh Ġcorrelations Ġfor ĠPAR ENT , Ġhowever , Ġif Ġhuman Ġannotations Ġare Ġavailable , Ġthis Ġcan Ġbe Ġtuned Ġto Ġproduce Ġsignificantly Ġhigher Ġcorrelations Ġ( PAR ENT * - W / C ). </s> Analysis </s> In Ġthis Ġsection Ġwe Ġfurther Ġanalyze Ġthe Ġperformance Ġof ĠPAR ENT - W Ġunder Ġdifferent Ġconditions , Ġand Ġcompare Ġto Ġthe Ġother Ġbest Ġmetrics Ġfrom ĠTable ĠT AB REF 37 Ġ. </s> To Ġstudy Ġthe Ġcorrelation Ġas Ġwe Ġvary Ġthe Ġnumber Ġof Ġdiver gent Ġreferences , Ġwe Ġalso Ġcollected Ġbinary Ġlabels Ġfrom Ġworkers Ġfor Ġwhether Ġa Ġreference Ġis Ġent ailed Ġby Ġthe Ġcorresponding Ġtable . ĠWe Ġdefine Ġa Ġreference Ġas Ġent ailed Ġwhen Ġit Ġmentions Ġonly Ġinformation Ġwhich Ġcan Ġbe Ġinferred Ġfrom Ġthe Ġtable . ĠEach Ġtable Ġand Ġreference Ġpair Ġwas Ġjudged Ġby Ġ3 Ġindependent Ġworkers , Ġand Ġwe Ġused Ġthe Ġmajority Ġvote Ġas Ġthe Ġlabel Ġfor Ġthat Ġpair . ĠOverall , Ġonly ĠIN LINE FORM 0 Ġof Ġthe Ġreferences Ġwere Ġlabeled Ġas Ġent ailed Ġby Ġthe Ġtable . ĠFle iss ' ĠIN LINE FORM 1 Ġwas ĠIN LINE FORM 2 Ġ, Ġwhich Ġindicates Ġa Ġfair Ġagreement . ĠWe Ġfound Ġthe Ġworkers Ġsometimes Ġdisagreed Ġon Ġwhat Ġinformation Ġcan Ġbe Ġreasonably Ġent ailed Ġby Ġthe Ġtable . </s> Figure ĠFIG REF 40 Ġshows Ġthe Ġcorrelations Ġas Ġwe Ġvary Ġthe Ġpercent Ġof Ġent ailed Ġexamples Ġin Ġthe Ġevaluation Ġset Ġof ĠWiki Bio . ĠEach Ġpoint Ġis Ġobtained Ġby Ġfixing Ġthe Ġdesired Ġproportion Ġof Ġent ailed Ġexamples , Ġand Ġsampling Ġsubs ets Ġfrom Ġthe Ġfull Ġset Ġwhich Ġsatisfy Ġthis Ġproportion . ĠPAR ENT Ġand ĠRG - F Ġremain Ġstable Ġand Ġshow Ġa Ġhigh Ġcorrelation Ġacross Ġthe Ġentire Ġrange , Ġwhereas ĠB LE U Ġand ĠB LE U - T Ġvary Ġa Ġlot . ĠIn Ġthe Ġhyper params Ġcategory , Ġthe Ġlatter Ġtwo Ġhave Ġthe Ġworst Ġcorrelation Ġwhen Ġthe Ġevaluation Ġset Ġcontains Ġonly Ġent ailed Ġexamples , Ġwhich Ġmay Ġseem Ġsurprising . ĠHowever , Ġon Ġcloser Ġexamination Ġwe Ġfound Ġthat Ġthis Ġsubset Ġtends Ġto Ġomit Ġa Ġlot Ġof Ġinformation Ġfrom Ġthe Ġtables . ĠSystems Ġwhich Ġproduce Ġmore Ġinformation Ġthan Ġthese Ġreferences Ġare Ġpenal ized Ġby ĠB LE U , Ġbut Ġnot Ġin Ġthe Ġhuman Ġevaluation . ĠPAR ENT Ġover comes Ġthis Ġissue Ġby Ġmeasuring Ġrecall Ġagainst Ġthe Ġtable Ġin Ġaddition Ġto Ġthe Ġreference . </s> We Ġcheck Ġhow Ġdifferent Ġcomponents Ġin Ġthe Ġcomputation Ġof ĠPAR ENT Ġcontribute Ġto Ġits Ġcorrelation Ġto Ġhuman Ġjudgments . ĠSpecifically , Ġwe Ġremove Ġthe Ġprobability ĠIN LINE FORM 0 Ġof Ġan Ġn - gram ĠIN LINE FORM 1 Ġbeing Ġent ailed Ġby Ġthe Ġtable Ġfrom ĠE qs . ĠEQ REF 19 Ġand ĠEQ REF 23 Ġ. ĠThe Ġaverage Ġcorrelation Ġfor ĠPAR ENT - W Ġdrops Ġto ĠIN LINE FORM 5 Ġin Ġthis Ġcase . ĠWe Ġalso Ġtry Ġa Ġvariant Ġof ĠPAR ENT Ġwith ĠIN LINE FORM 6 Ġ, Ġwhich Ġremoves Ġthe Ġcontribution Ġof ĠTable ĠRecall Ġ( E q . ĠEQ REF 22 Ġ). ĠThe Ġaverage Ġcorrelation Ġis ĠIN LINE FORM 7 Ġin Ġthis Ġcase . ĠWith Ġthese Ġcomponents , Ġthe Ġcorrelation Ġis ĠIN LINE FORM 8 Ġ, Ġshowing Ġthat Ġthey Ġare Ġcrucial Ġto Ġthe Ġperformance Ġof ĠPAR ENT . </s> BI B REF 28 Ġpoint Ġout Ġthat Ġhill - cl im bing Ġon Ġan Ġautomatic Ġmetric Ġis Ġmeaningless Ġif Ġthat Ġmetric Ġhas Ġa Ġlow Ġinstance - level Ġcorrelation Ġto Ġhuman Ġjudgments . ĠIn ĠTable ĠT AB REF 46 Ġwe Ġshow Ġthe Ġaverage Ġaccuracy Ġof Ġthe Ġmetrics Ġin Ġmaking Ġthe Ġsame Ġjudgments Ġas Ġhumans Ġbetween Ġpairs Ġof Ġgenerated Ġtexts . ĠBoth Ġvariants Ġof ĠPAR ENT Ġare Ġsignificantly Ġbetter Ġthan Ġthe Ġother Ġmetrics , Ġhowever Ġthe Ġbest Ġaccuracy Ġis Ġonly ĠIN LINE FORM 0 Ġfor Ġthe Ġbinary Ġtask . ĠThis Ġis Ġa Ġchallenging Ġtask , Ġsince Ġthere Ġare Ġtypically Ġonly Ġsubtle Ġdifferences Ġbetween Ġthe Ġtexts . ĠA chie ving Ġhigher Ġinstance - level Ġaccur acies Ġwill Ġrequire Ġmore Ġsophisticated Ġlanguage Ġunderstanding Ġmodels Ġfor Ġevaluation . </s> Web NL G ĠDat as et </s> To Ġcheck Ġhow ĠPAR ENT Ġcorrelates Ġwith Ġhuman Ġjudgments Ġwhen Ġthe Ġreferences Ġare Ġelic ited Ġfrom Ġhumans Ġ( and Ġless Ġlikely Ġto Ġbe Ġdiver gent ), Ġwe Ġcheck Ġits Ġcorrelation Ġwith Ġthe Ġhuman Ġratings Ġprovided Ġfor Ġthe Ġsystems Ġcompeting Ġin Ġthe ĠWeb NL G Ġchallenge ĠB IB REF 6 Ġ. ĠThe Ġtask Ġis Ġto Ġgenerate Ġtext Ġdescribing Ġ1 - 5 ĠR DF Ġtri ples Ġ( e . g . ĠJohn ĠE ĠBl aha , Ġbirth Place , ĠSan ĠAntonio ), Ġand Ġhuman Ġratings Ġwere Ġcollected Ġfor Ġthe Ġoutputs Ġof Ġ9 Ġparticipating Ġsystems Ġon Ġ223 Ġinstances . ĠThese Ġsystems Ġinclude Ġa Ġmix Ġof Ġpip el ined , Ġstatistical Ġand Ġneural Ġmethods . ĠEach Ġinstance Ġhas Ġupt o Ġ3 Ġreference Ġtexts Ġassociated Ġwith Ġthe ĠR DF Ġtri ples , Ġwhich Ġwe Ġuse Ġfor Ġevaluation . </s> The Ġhuman Ġratings Ġwere Ġcollected Ġon Ġ3 Ġdistinct Ġaspects ĠâĢĵ Ġgram matical ity , Ġflu ency Ġand Ġsemantics , Ġwhere Ġsemantics Ġcorresponds Ġto Ġthe Ġdegree Ġto Ġwhich Ġa Ġgenerated Ġtext Ġagrees Ġwith Ġthe Ġmeaning Ġof Ġthe Ġunderlying ĠR DF Ġtri ples . ĠWe Ġreport Ġthe Ġcorrelation Ġof Ġseveral Ġmetrics Ġwith Ġthese Ġratings Ġin ĠTable ĠT AB REF 48 Ġ. ĠBoth Ġvariants Ġof ĠPAR ENT Ġare Ġeither Ġcompetitive Ġor Ġbetter Ġthan Ġthe Ġother Ġmetrics Ġin Ġterms Ġof Ġthe Ġaverage Ġcorrelation Ġto Ġall Ġthree Ġaspects . ĠThis Ġshows Ġthat ĠPAR ENT Ġis Ġapplicable Ġfor Ġhigh Ġquality Ġreferences Ġas Ġwell . </s> While ĠB LE U Ġhas Ġthe Ġhighest Ġcorrelation Ġfor Ġthe Ġgrammar Ġand Ġflu ency Ġaspects , ĠPAR ENT Ġdoes Ġbest Ġfor Ġsemantics . ĠThis Ġsuggests Ġthat Ġthe Ġinclusion Ġof Ġsource Ġtables Ġinto Ġthe Ġevaluation Ġo rients Ġthe Ġmetric Ġmore Ġtowards Ġmeasuring Ġthe Ġfidelity Ġof Ġthe Ġcontent Ġof Ġthe Ġgeneration . ĠA Ġsimilar Ġtrend Ġis Ġseen Ġcomparing ĠB LE U Ġand ĠB LE U - T . ĠAs Ġmodern Ġneural Ġtext Ġgeneration Ġsystems Ġare Ġtypically Ġvery Ġfluent , Ġmeasuring Ġtheir Ġfidelity Ġis Ġof Ġincreasing Ġimportance . ĠBetween Ġthe Ġtwo Ġentail ment Ġmodels , ĠPAR ENT - C Ġis Ġbetter Ġdue Ġto Ġits Ġhigher Ġcorrelation Ġwith Ġthe Ġgram matical ity Ġand Ġflu ency Ġaspects . </s> The ĠIN LINE FORM 0 Ġparameter Ġin Ġthe Ġcalculation Ġof ĠPAR ENT Ġdecides Ġwhether Ġto Ġcompute Ġrecall Ġagainst Ġthe Ġtable Ġor Ġthe Ġreference Ġ( E q . ĠEQ REF 22 Ġ). ĠFigure ĠFIG REF 50 Ġshows Ġthe Ġdistribution Ġof Ġthe Ġvalues Ġtaken Ġby ĠIN LINE FORM 1 Ġusing Ġthe Ġhe uristic Ġdescribed Ġin ĠÂ§ ĠSEC REF 3 Ġfor Ġinstances Ġin Ġboth ĠWiki Bio Ġand ĠWeb NL G . ĠFor ĠWiki Bio , Ġthe Ġrecall Ġof Ġthe Ġreferences Ġagainst Ġthe Ġtable Ġis Ġgenerally Ġlow , Ġand Ġhence Ġthe Ġrecall Ġof Ġthe Ġgenerated Ġtext Ġrelies Ġmore Ġon Ġthe Ġtable . ĠFor ĠWeb NL G , Ġwhere Ġthe Ġreferences Ġare Ġelic ited Ġfrom Ġhumans , Ġthis Ġrecall Ġis Ġmuch Ġhigher Ġ( often ĠIN LINE FORM 2 Ġ), Ġand Ġhence Ġthe Ġrecall Ġof Ġthe Ġgenerated Ġtext Ġrelies Ġmore Ġon Ġthe Ġreference . </s> Related ĠWork </s> Over Ġthe Ġyears Ġseveral Ġstudies Ġhave Ġevaluated Ġautomatic Ġmetrics Ġfor Ġmeasuring Ġtext Ġgeneration Ġperformance ĠB IB REF 29 Ġ, ĠB IB REF 30 Ġ, ĠB IB REF 31 Ġ, ĠB IB REF 32 Ġ, ĠB IB REF 33 Ġ, ĠB IB REF 34 Ġ, ĠB IB REF 35 Ġ. ĠThe Ġonly Ġconsensus Ġfrom Ġthese Ġstudies Ġseems Ġto Ġbe Ġthat Ġno Ġsingle Ġmetric Ġis Ġsuitable Ġacross Ġall Ġtasks . ĠA Ġrecurring Ġtheme Ġis Ġthat Ġmetrics Ġlike ĠB LE U Ġand ĠN IST ĠB IB REF 36 Ġare Ġnot Ġsuitable Ġfor Ġjudging Ġcontent Ġquality Ġin ĠNL G . ĠRecently , ĠB IB REF 37 Ġdid Ġa Ġcomprehensive Ġstudy Ġof Ġseveral Ġmetrics Ġon Ġthe Ġoutputs Ġof Ġstate - of - the - art ĠNL G Ġsystems , Ġand Ġfound Ġthat Ġwhile Ġthey Ġshowed Ġacceptable Ġcorrelation Ġwith Ġhuman Ġjudgments Ġat Ġthe Ġsystem Ġlevel , Ġthey Ġfailed Ġto Ġshow Ġany Ġcorrelation Ġat Ġthe Ġsentence Ġlevel . ĠO urs Ġis Ġthe Ġfirst Ġstudy Ġwhich Ġchecks Ġthe Ġquality Ġof Ġmetrics Ġwhen Ġtable - to - text Ġreferences Ġare Ġdiver gent . ĠWe Ġshow Ġthat Ġin Ġthis Ġcase Ġeven Ġsystem Ġlevel Ġcorrelations Ġcan Ġbe Ġunreliable . </s> Hall uc ination ĠB IB REF 38 Ġ, ĠB IB REF 39 Ġrefers Ġto Ġwhen Ġan ĠNL G Ġsystem Ġgenerates Ġtext Ġwhich Ġmentions Ġextra Ġinformation Ġthan Ġwhat Ġis Ġpresent Ġin Ġthe Ġsource Ġfrom Ġwhich Ġit Ġis Ġgenerated . ĠDiver gence Ġcan Ġbe Ġviewed Ġas Ġhalluc ination Ġin Ġthe Ġreference Ġtext Ġitself . ĠPAR ENT Ġdeals Ġwith Ġhalluc ination Ġby Ġdiscount ing Ġn - gram s Ġwhich Ġdo Ġnot Ġoverlap Ġwith Ġeither Ġthe Ġreference Ġor Ġthe Ġtable . </s> PAR ENT Ġdraws Ġinspiration Ġfrom Ġi BLE U ĠB IB REF 26 Ġ, Ġa Ġmetric Ġfor Ġevaluating Ġparaph rase Ġgeneration , Ġwhich Ġcompares Ġthe Ġgenerated Ġtext Ġto Ġboth Ġthe Ġsource Ġtext Ġand Ġthe Ġreference . ĠWhile Ġi BLE U Ġpenal izes Ġtexts Ġwhich Ġmatch Ġthe Ġsource , Ġhere Ġwe Ġreward Ġsuch Ġtexts Ġsince Ġour Ġtask Ġvalues Ġaccuracy Ġof Ġgenerated Ġtext Ġmore Ġthan Ġthe Ġneed Ġfor Ġparaph r asing Ġthe Ġtab ular Ġcontent ĠB IB REF 40 Ġ. ĠSimilar Ġto ĠS ARI Ġfor Ġtext Ġsimpl ification ĠB IB REF 41 Ġand ĠQ - BLE U Ġfor Ġquestion Ġgeneration ĠB IB REF 42 Ġ, ĠPAR ENT Ġfalls Ġunder Ġthe Ġcategory Ġof Ġtask - specific Ġmetrics . </s> Con clusions </s> We Ġstudy Ġthe Ġautomatic Ġevaluation Ġof Ġtable - to - text Ġsystems Ġwhen Ġthe Ġreferences Ġdiver ge Ġfrom Ġthe Ġtable . ĠWe Ġpropose Ġa Ġnew Ġmetric , ĠPAR ENT , Ġwhich Ġshows Ġthe Ġhighest Ġcorrelation Ġwith Ġhumans Ġacross Ġa Ġrange Ġof Ġsettings Ġwith Ġdiver gent Ġreferences Ġin ĠWiki Bio . ĠWe Ġalso Ġperform Ġthe Ġfirst Ġempirical Ġevaluation Ġof Ġinformation Ġextraction Ġbased Ġmetrics ĠB IB REF 1 Ġ, Ġand Ġfind ĠRG - F Ġto Ġbe Ġeffective . ĠLastly , Ġwe Ġshow Ġthat ĠPAR ENT Ġis Ġcomparable Ġto Ġthe Ġbest Ġexisting Ġmetrics Ġwhen Ġreferences Ġare Ġelic ited Ġby Ġhumans Ġon Ġthe ĠWeb NL G Ġdata . </s> Acknowled gements </s> B hu wan ĠD hing ra Ġis Ġsupported Ġby Ġa Ġfellowship Ġfrom ĠSiem ens , Ġand Ġby Ġgrants Ġfrom ĠGoogle . ĠWe Ġthank ĠMar uan ĠAl - S hed iv at , ĠIan ĠTen ney , ĠTom ĠKw iat kowski , ĠMichael ĠCollins , ĠSlav ĠPet rov , ĠJason ĠBald ridge , ĠDavid ĠRe itter Ġand Ġother Ġmembers Ġof Ġthe ĠGoogle ĠAI ĠLanguage Ġteam Ġfor Ġhelpful Ġdiscussions Ġand Ġsuggestions . ĠWe Ġthank ĠSam ĠWis eman Ġfor Ġsharing Ġdata Ġfor Ġan Ġearlier Ġversion Ġof Ġthis Ġpaper . ĠWe Ġalso Ġthank Ġthe Ġanonymous Ġreviewers Ġfor Ġtheir Ġfeedback . </s> Information ĠExt raction ĠSystem </s> For Ġevaluation Ġvia Ġinformation Ġextraction ĠB IB REF 1 Ġwe Ġtrain Ġa Ġmodel Ġfor ĠWiki Bio Ġwhich Ġaccepts Ġtext Ġas Ġinput Ġand Ġgenerates Ġa Ġtable Ġas Ġthe Ġoutput . ĠTables Ġin ĠWiki Bio Ġare Ġopen - domain , Ġwithout Ġany Ġfixed Ġschema Ġfor Ġwhich Ġattributes Ġmay Ġbe Ġpresent Ġor Ġabsent Ġin Ġan Ġinstance . ĠHence Ġwe Ġemploy Ġthe ĠPo inter - Gener ator ĠNetwork Ġ( PG - Net ) ĠB IB REF 19 Ġfor Ġthis Ġpurpose . ĠSpecifically , Ġwe Ġuse Ġa Ġsequence - to - sequence Ġmodel , Ġwhose Ġenc oder Ġand Ġdec oder Ġare Ġboth Ġsingle - layer Ġbi - direction al ĠL ST Ms . ĠThe Ġdec oder Ġis Ġaugmented Ġwith Ġan Ġattention Ġmechanism Ġover Ġthe Ġstates Ġof Ġthe Ġenc oder . ĠFurther , Ġit Ġalso Ġuses Ġa Ġcopy Ġmechanism Ġto Ġoptionally Ġcopy Ġtokens Ġdirectly Ġfrom Ġthe Ġsource Ġtext . ĠWe Ġdo Ġnot Ġuse Ġthe Ġcoverage Ġmechanism Ġof ĠB IB REF 19 Ġsince Ġthat Ġis Ġspecific Ġto Ġthe Ġtask Ġof Ġsummar ization Ġthey Ġstudy . ĠThe Ġdec oder Ġis Ġtrained Ġto Ġproduce Ġa Ġlinear ized Ġversion Ġof Ġthe Ġtable Ġwhere Ġthe Ġrows Ġand Ġcolumns Ġare Ġflattened Ġinto Ġa Ġsequence , Ġand Ġseparate Ġby Ġspecial Ġtokens . ĠFigure ĠFIG REF 53 Ġshows Ġan Ġexample . </s> Clearly , Ġsince Ġthe Ġreferences Ġare Ġdiver gent , Ġthe Ġmodel Ġcannot Ġbe Ġexpected Ġto Ġproduce Ġthe Ġentire Ġtable , Ġand Ġwe Ġsee Ġsome Ġfalse Ġinformation Ġbeing Ġhalluc inated Ġafter Ġtraining . ĠNevertheless , Ġas Ġwe Ġshow Ġin ĠÂ§ ĠSEC REF 36 Ġ, Ġthis Ġsystem Ġcan Ġbe Ġused Ġfor Ġevaluating Ġgenerated Ġtexts . ĠAfter Ġtraining , Ġwe Ġcan Ġparse Ġthe Ġoutput Ġsequence Ġalong Ġthe Ġspecial Ġtokens ĠIN LINE FORM 0 ĠR ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 ĠC ĠIN LINE FORM 3 Ġto Ġget Ġa Ġset Ġof Ġ( attribute , Ġvalue ) Ġpairs . ĠTable ĠT AB REF 54 Ġshows Ġthe Ġprecision , Ġrecall Ġand ĠF - score Ġof Ġthese Ġextracted Ġpairs Ġagainst Ġthe Ġground Ġtruth Ġtables , Ġwhere Ġthe Ġattributes Ġand Ġvalues Ġare Ġcompared Ġusing Ġan Ġexact Ġstring Ġmatch . </s> Hyper param eters </s> After Ġtuning Ġwe Ġfound Ġthe Ġsame Ġset Ġof Ġhyper param eters Ġto Ġwork Ġwell Ġfor Ġboth Ġthe Ġtable - to - text ĠPG - Net , Ġand Ġthe Ġinverse Ġinformation Ġextraction ĠPG - Net . ĠThe Ġhidden Ġstate Ġsize Ġof Ġthe Ġbi L ST Ms Ġwas Ġset Ġto Ġ200 . ĠThe Ġinput Ġand Ġoutput Ġvoc ab ular ies Ġwere Ġset Ġto Ġ5 0000 Ġmost Ġcommon Ġwords Ġin Ġthe Ġcorpus , Ġwith Ġadditional Ġspecial Ġsymbols Ġfor Ġtable Ġattribute Ġnames Ġ( such Ġas ĠâĢ ľ birth - date âĢ Ŀ ). ĠThe Ġembed d ings Ġof Ġthe Ġtokens Ġin Ġthe Ġvocabulary Ġwere Ġinitialized Ġwith ĠGl ove ĠB IB REF 43 Ġ. ĠLearning Ġrate Ġof ĠIN LINE FORM 0 Ġwas Ġused Ġduring Ġtraining , Ġwith Ġthe ĠAdam Ġoptim izer , Ġand Ġa Ġdrop out Ġof ĠIN LINE FORM 1 Ġwas Ġalso Ġapplied Ġto Ġthe Ġoutputs Ġof Ġthe Ġbi L ST M . ĠModels Ġwere Ġtrained Ġtill Ġthe Ġloss Ġon Ġthe Ġdev Ġset Ġstopped Ġdropping . ĠMaximum Ġlength Ġof Ġa Ġdec oded Ġtext Ġwas Ġset Ġto Ġ40 Ġtokens , Ġand Ġthat Ġof Ġthe Ġtables Ġwas Ġset Ġto Ġ120 Ġtokens . ĠVarious Ġbeam Ġsizes Ġand Ġlength Ġnormal ization Ġpenalties Ġwere Ġapplied Ġfor Ġthe Ġtable - to - text Ġsystem , Ġwhich Ġare Ġlisted Ġin Ġthe Ġmain Ġpaper . ĠFor Ġthe Ġinformation Ġextraction Ġsystem , Ġwe Ġfound Ġa Ġbeam Ġsize Ġof Ġ8 Ġand Ġno Ġlength Ġpenalty Ġto Ġproduce Ġthe Ġhighest ĠF - score Ġon Ġthe Ġdev Ġset . </s> Sample ĠOutput s </s> Table ĠT AB REF 55 Ġshows Ġsome Ġsample Ġreferences Ġand Ġthe Ġcorresponding Ġpredictions Ġfrom Ġthe Ġbest Ġperforming Ġmodel , ĠPG - Net Ġfor ĠWiki Bio .
Introduction The task of generating natural language descriptions of structured data (such as tables) BIBREF2 , BIBREF3 , BIBREF4 has seen a growth in interest with the rise of sequence to sequence models that provide an easy way of encoding tables and generating text from them BIBREF0 , BIBREF1 , BIBREF5 , BIBREF6 . For text generation tasks, the only gold standard metric is to show the output to humans for judging its quality, but this is too expensive to apply repeatedly anytime small modifications are made to a system. Hence, automatic metrics that compare the generated text to one or more reference texts are routinely used to compare models BIBREF7 . For table-to-text generation, automatic evaluation has largely relied on BLEU BIBREF8 and ROUGE BIBREF9 . The underlying assumption behind these metrics is that the reference text is gold-standard, i.e., it is the ideal target text that a system should generate. In practice, however, when datasets are collected automatically and heuristically, the reference texts are often not ideal. Figure FIGREF2 shows an example from the WikiBio dataset BIBREF0 . Here the reference contains extra information which no system can be expected to produce given only the associated table. We call such reference texts divergent from the table. We show that existing automatic metrics, including BLEU, correlate poorly with human judgments when the evaluation sets contain divergent references (§ SECREF36 ). For many table-to-text generation tasks, the tables themselves are in a pseudo-natural language format (e.g., WikiBio, WebNLG BIBREF6 , and E2E-NLG BIBREF10 ). In such cases we propose to compare the generated text to the underlying table as well to improve evaluation. We develop a new metric, PARENT (Precision And Recall of Entailed N-grams from the Table) (§ SECREF3 ). When computing precision, PARENT effectively uses a union of the reference and the table, to reward correct information missing from the reference. When computing recall, it uses an intersection of the reference and the table, to ignore extra incorrect information in the reference. The union and intersection are computed with the help of an entailment model to decide if a text n-gram is entailed by the table. We show that this method is more effective than using the table as an additional reference. Our main contributions are: Table-to-Text Generation We briefly review the task of generating natural language descriptions of semi-structured data, which we refer to as tables henceforth BIBREF11 , BIBREF12 . Tables can be expressed as set of records INLINEFORM0 , where each record is a tuple (entity, attribute, value). When all the records are about the same entity, we can truncate the records to (attribute, value) pairs. For example, for the table in Figure FIGREF2 , the records are {(Birth Name, Michael Dahlquist), (Born, December 22 1965), ...}. The task is to generate a text INLINEFORM1 which summarizes the records in a fluent and grammatical manner. For training and evaluation we further assume that we have a reference description INLINEFORM2 available for each table. We let INLINEFORM3 denote an evaluation set of tables, references and texts generated from a model INLINEFORM4 , and INLINEFORM5 , INLINEFORM6 denote the collection of n-grams of order INLINEFORM7 in INLINEFORM8 and INLINEFORM9 , respectively. We use INLINEFORM10 to denote the count of n-gram INLINEFORM11 in INLINEFORM12 , and INLINEFORM13 to denote the minimum of its counts in INLINEFORM14 and INLINEFORM15 . Our goal is to assign a score to the model, which correlates highly with human judgments of the quality of that model. PARENT PARENT evaluates each instance INLINEFORM0 separately, by computing the precision and recall of INLINEFORM1 against both INLINEFORM2 and INLINEFORM3 . Evaluation via Information Extraction BIBREF1 proposed to use an auxiliary model, trained to extract structured records from text, for evaluation. However, the extraction model presented in that work is limited to the closed-domain setting of basketball game tables and summaries. In particular, they assume that each table has exactly the same set of attributes for each entity, and that the entities can be identified in the text via string matching. These assumptions are not valid for the open-domain WikiBio dataset, and hence we train our own extraction model to replicate their evaluation scheme. Our extraction system is a pointer-generator network BIBREF19 , which learns to produce a linearized version of the table from the text. The network learns which attributes need to be populated in the output table, along with their values. It is trained on the training set of WikiBio. At test time we parsed the output strings into a set of (attribute, value) tuples and compare it to the ground truth table. The F-score of this text-to-table system was INLINEFORM0 , which is comparable to other challenging open-domain settings BIBREF20 . More details are included in the Appendix SECREF52 . Given this information extraction system, we consider the following metrics for evaluation, along the lines of BIBREF1 . Content Selection (CS): F-score for the (attribute, value) pairs extracted from the generated text compared to those extracted from the reference. Relation Generation (RG): Precision for the (attribute, value) pairs extracted from the generated text compared to those in the ground truth table. RG-F: Since our task emphasizes the recall of information from the table as well, we consider another variant which computes the F-score of the extracted pairs to those in the table. We omit the content ordering metric, since our extraction system does not align records to the input text. Experiments & Results In this section we compare several automatic evaluation metrics by checking their correlation with the scores assigned by humans to table-to-text models. Specifically, given INLINEFORM0 models INLINEFORM1 , and their outputs on an evaluation set, we show these generated texts to humans to judge their quality, and obtain aggregated human evaluation scores for all the models, INLINEFORM2 (§ SECREF33 ). Next, to evaluate an automatic metric, we compute the scores it assigns to each model, INLINEFORM3 , and check the Pearson correlation between INLINEFORM4 and INLINEFORM5 BIBREF21 . Data & Models Our main experiments are on the WikiBio dataset BIBREF0 , which is automatically constructed and contains many divergent references. In § SECREF47 we also present results on the data released as part of the WebNLG challenge. We developed several models of varying quality for generating text from the tables in WikiBio. This gives us a diverse set of outputs to evaluate the automatic metrics on. Table TABREF32 lists the models along with their hyperparameter settings and their scores from the human evaluation (§ SECREF33 ). Our focus is primarily on neural sequence-to-sequence methods since these are most widely used, but we also include a template-based baseline. All neural models were trained on the WikiBio training set. Training details and sample outputs are included in Appendices SECREF56 & SECREF57 . We divide these models into two categories and measure correlation separately for both the categories. The first category, WikiBio-Systems, includes one model each from the four families listed in Table TABREF32 . This category tests whether a metric can be used to compare different model families with a large variation in the quality of their outputs. The second category, WikiBio-Hyperparams, includes 13 different hyperparameter settings of PG-Net BIBREF19 , which was the best performing system overall. 9 of these were obtained by varying the beam size and length normalization penalty of the decoder network BIBREF23 , and the remaining 4 were obtained by re-scoring beams of size 8 with the information extraction model described in § SECREF4 . All the models in this category produce high quality fluent texts, and differ primarily on the quantity and accuracy of the information they express. Here we are testing whether a metric can be used to compare similar systems with a small variation in performance. This is an important use-case as metrics are often used to tune hyperparameters of a model. Human Evaluation We collected human judgments on the quality of the 16 models trained for WikiBio, plus the reference texts. Workers on a crowd-sourcing platform, proficient in English, were shown a table with pairs of generated texts, or a generated text and the reference, and asked to select the one they prefer. Figure FIGREF34 shows the instructions they were given. Paired comparisons have been shown to be superior to rating scales for comparing generated texts BIBREF24 . However, for measuring correlation the comparisons need to be aggregated into real-valued scores, INLINEFORM0 , for each of the INLINEFORM1 models. For this, we use Thurstone's method BIBREF22 , which assigns a score to each model based on how many times it was preferred over an alternative. The data collection was performed separately for models in the WikiBio-Systems and WikiBio-Hyperparams categories. 1100 tables were sampled from the development set, and for each table we got 8 different sentence pairs annotated across the two categories, resulting in a total of 8800 pairwise comparisons. Each pair was judged by one worker only which means there may be noise at the instance-level, but the aggregated system-level scores had low variance (cf. Table TABREF32 ). In total around 500 different workers were involved in the annotation. References were also included in the evaluation, and they received a lower score than PG-Net, highlighting the divergence in WikiBio. Compared Metrics Text only: We compare BLEU BIBREF8 , ROUGE BIBREF9 , METEOR BIBREF18 , CIDEr and CIDEr-D BIBREF25 using their publicly available implementations. Information Extraction based: We compare the CS, RG and RG-F metrics discussed in § SECREF4 . Text & Table: We compare a variant of BLEU, denoted as BLEU-T, where the values from the table are used as additional references. BLEU-T draws inspiration from iBLEU BIBREF26 but instead rewards n-grams which match the table rather than penalizing them. For PARENT, we compare both the word-overlap model (PARENT-W) and the co-occurrence model (PARENT-C) for determining entailment. We also compare versions where a single INLINEFORM0 is tuned on the entire dataset to maximize correlation with human judgments, denoted as PARENT*-W/C. Correlation Comparison We use bootstrap sampling (500 iterations) over the 1100 tables for which we collected human annotations to get an idea of how the correlation of each metric varies with the underlying data. In each iteration, we sample with replacement, tables along with their references and all the generated texts for that table. Then we compute aggregated human evaluation and metric scores for each of the models and compute the correlation between the two. We report the average correlation across all bootstrap samples for each metric in Table TABREF37 . The distribution of correlations for the best performing metrics are shown in Figure FIGREF38 . Table TABREF37 also indicates whether PARENT is significantly better than a baseline metric. BIBREF21 suggest using the William's test for this purpose, but since we are computing correlations between only 4/13 systems at a time, this test has very weak power in our case. Hence, we use the bootstrap samples to obtain a INLINEFORM0 confidence interval of the difference in correlation between PARENT and any other metric and check whether this is above 0 BIBREF27 . Correlations are higher for the systems category than the hyperparams category. The latter is a more difficult setting since very similar models are compared, and hence the variance of the correlations is also high. Commonly used metrics which only rely on the reference (BLEU, ROUGE, METEOR, CIDEr) have only weak correlations with human judgments. In the hyperparams category, these are often negative, implying that tuning models based on these may lead to selecting worse models. BLEU performs the best among these, and adding n-grams from the table as references improves this further (BLEU-T). Among the extractive evaluation metrics, CS, which also only relies on the reference, has poor correlation in the hyperparams category. RG-F, and both variants of the PARENT metric achieve the highest correlation for both settings. There is no significant difference among these for the hyperparams category, but for systems, PARENT-W is significantly better than the other two. While RG-F needs a full information extraction pipeline in its implementation, PARENT-C only relies on co-occurrence counts, and PARENT-W can be used out-of-the-box for any dataset. To our knowledge, this is the first rigorous evaluation of using information extraction for generation evaluation. On this dataset, the word-overlap model showed higher correlation than the co-occurrence model for entailment. In § SECREF47 we will show that for the WebNLG dataset, where more paraphrasing is involved between the table and text, the opposite is true. Lastly, we note that the heuristic for selecting INLINEFORM0 is sufficient to produce high correlations for PARENT, however, if human annotations are available, this can be tuned to produce significantly higher correlations (PARENT*-W/C). Analysis In this section we further analyze the performance of PARENT-W under different conditions, and compare to the other best metrics from Table TABREF37 . To study the correlation as we vary the number of divergent references, we also collected binary labels from workers for whether a reference is entailed by the corresponding table. We define a reference as entailed when it mentions only information which can be inferred from the table. Each table and reference pair was judged by 3 independent workers, and we used the majority vote as the label for that pair. Overall, only INLINEFORM0 of the references were labeled as entailed by the table. Fleiss' INLINEFORM1 was INLINEFORM2 , which indicates a fair agreement. We found the workers sometimes disagreed on what information can be reasonably entailed by the table. Figure FIGREF40 shows the correlations as we vary the percent of entailed examples in the evaluation set of WikiBio. Each point is obtained by fixing the desired proportion of entailed examples, and sampling subsets from the full set which satisfy this proportion. PARENT and RG-F remain stable and show a high correlation across the entire range, whereas BLEU and BLEU-T vary a lot. In the hyperparams category, the latter two have the worst correlation when the evaluation set contains only entailed examples, which may seem surprising. However, on closer examination we found that this subset tends to omit a lot of information from the tables. Systems which produce more information than these references are penalized by BLEU, but not in the human evaluation. PARENT overcomes this issue by measuring recall against the table in addition to the reference. We check how different components in the computation of PARENT contribute to its correlation to human judgments. Specifically, we remove the probability INLINEFORM0 of an n-gram INLINEFORM1 being entailed by the table from Eqs. EQREF19 and EQREF23 . The average correlation for PARENT-W drops to INLINEFORM5 in this case. We also try a variant of PARENT with INLINEFORM6 , which removes the contribution of Table Recall (Eq. EQREF22 ). The average correlation is INLINEFORM7 in this case. With these components, the correlation is INLINEFORM8 , showing that they are crucial to the performance of PARENT. BIBREF28 point out that hill-climbing on an automatic metric is meaningless if that metric has a low instance-level correlation to human judgments. In Table TABREF46 we show the average accuracy of the metrics in making the same judgments as humans between pairs of generated texts. Both variants of PARENT are significantly better than the other metrics, however the best accuracy is only INLINEFORM0 for the binary task. This is a challenging task, since there are typically only subtle differences between the texts. Achieving higher instance-level accuracies will require more sophisticated language understanding models for evaluation. WebNLG Dataset To check how PARENT correlates with human judgments when the references are elicited from humans (and less likely to be divergent), we check its correlation with the human ratings provided for the systems competing in the WebNLG challenge BIBREF6 . The task is to generate text describing 1-5 RDF triples (e.g. John E Blaha, birthPlace, San Antonio), and human ratings were collected for the outputs of 9 participating systems on 223 instances. These systems include a mix of pipelined, statistical and neural methods. Each instance has upto 3 reference texts associated with the RDF triples, which we use for evaluation. The human ratings were collected on 3 distinct aspects – grammaticality, fluency and semantics, where semantics corresponds to the degree to which a generated text agrees with the meaning of the underlying RDF triples. We report the correlation of several metrics with these ratings in Table TABREF48 . Both variants of PARENT are either competitive or better than the other metrics in terms of the average correlation to all three aspects. This shows that PARENT is applicable for high quality references as well. While BLEU has the highest correlation for the grammar and fluency aspects, PARENT does best for semantics. This suggests that the inclusion of source tables into the evaluation orients the metric more towards measuring the fidelity of the content of the generation. A similar trend is seen comparing BLEU and BLEU-T. As modern neural text generation systems are typically very fluent, measuring their fidelity is of increasing importance. Between the two entailment models, PARENT-C is better due to its higher correlation with the grammaticality and fluency aspects. The INLINEFORM0 parameter in the calculation of PARENT decides whether to compute recall against the table or the reference (Eq. EQREF22 ). Figure FIGREF50 shows the distribution of the values taken by INLINEFORM1 using the heuristic described in § SECREF3 for instances in both WikiBio and WebNLG. For WikiBio, the recall of the references against the table is generally low, and hence the recall of the generated text relies more on the table. For WebNLG, where the references are elicited from humans, this recall is much higher (often INLINEFORM2 ), and hence the recall of the generated text relies more on the reference. Related Work Over the years several studies have evaluated automatic metrics for measuring text generation performance BIBREF29 , BIBREF30 , BIBREF31 , BIBREF32 , BIBREF33 , BIBREF34 , BIBREF35 . The only consensus from these studies seems to be that no single metric is suitable across all tasks. A recurring theme is that metrics like BLEU and NIST BIBREF36 are not suitable for judging content quality in NLG. Recently, BIBREF37 did a comprehensive study of several metrics on the outputs of state-of-the-art NLG systems, and found that while they showed acceptable correlation with human judgments at the system level, they failed to show any correlation at the sentence level. Ours is the first study which checks the quality of metrics when table-to-text references are divergent. We show that in this case even system level correlations can be unreliable. Hallucination BIBREF38 , BIBREF39 refers to when an NLG system generates text which mentions extra information than what is present in the source from which it is generated. Divergence can be viewed as hallucination in the reference text itself. PARENT deals with hallucination by discounting n-grams which do not overlap with either the reference or the table. PARENT draws inspiration from iBLEU BIBREF26 , a metric for evaluating paraphrase generation, which compares the generated text to both the source text and the reference. While iBLEU penalizes texts which match the source, here we reward such texts since our task values accuracy of generated text more than the need for paraphrasing the tabular content BIBREF40 . Similar to SARI for text simplification BIBREF41 and Q-BLEU for question generation BIBREF42 , PARENT falls under the category of task-specific metrics. Conclusions We study the automatic evaluation of table-to-text systems when the references diverge from the table. We propose a new metric, PARENT, which shows the highest correlation with humans across a range of settings with divergent references in WikiBio. We also perform the first empirical evaluation of information extraction based metrics BIBREF1 , and find RG-F to be effective. Lastly, we show that PARENT is comparable to the best existing metrics when references are elicited by humans on the WebNLG data. Acknowledgements Bhuwan Dhingra is supported by a fellowship from Siemens, and by grants from Google. We thank Maruan Al-Shedivat, Ian Tenney, Tom Kwiatkowski, Michael Collins, Slav Petrov, Jason Baldridge, David Reitter and other members of the Google AI Language team for helpful discussions and suggestions. We thank Sam Wiseman for sharing data for an earlier version of this paper. We also thank the anonymous reviewers for their feedback. Information Extraction System For evaluation via information extraction BIBREF1 we train a model for WikiBio which accepts text as input and generates a table as the output. Tables in WikiBio are open-domain, without any fixed schema for which attributes may be present or absent in an instance. Hence we employ the Pointer-Generator Network (PG-Net) BIBREF19 for this purpose. Specifically, we use a sequence-to-sequence model, whose encoder and decoder are both single-layer bi-directional LSTMs. The decoder is augmented with an attention mechanism over the states of the encoder. Further, it also uses a copy mechanism to optionally copy tokens directly from the source text. We do not use the coverage mechanism of BIBREF19 since that is specific to the task of summarization they study. The decoder is trained to produce a linearized version of the table where the rows and columns are flattened into a sequence, and separate by special tokens. Figure FIGREF53 shows an example. Clearly, since the references are divergent, the model cannot be expected to produce the entire table, and we see some false information being hallucinated after training. Nevertheless, as we show in § SECREF36 , this system can be used for evaluating generated texts. After training, we can parse the output sequence along the special tokens INLINEFORM0 R INLINEFORM1 and INLINEFORM2 C INLINEFORM3 to get a set of (attribute, value) pairs. Table TABREF54 shows the precision, recall and F-score of these extracted pairs against the ground truth tables, where the attributes and values are compared using an exact string match. Hyperparameters After tuning we found the same set of hyperparameters to work well for both the table-to-text PG-Net, and the inverse information extraction PG-Net. The hidden state size of the biLSTMs was set to 200. The input and output vocabularies were set to 50000 most common words in the corpus, with additional special symbols for table attribute names (such as “birth-date”). The embeddings of the tokens in the vocabulary were initialized with Glove BIBREF43 . Learning rate of INLINEFORM0 was used during training, with the Adam optimizer, and a dropout of INLINEFORM1 was also applied to the outputs of the biLSTM. Models were trained till the loss on the dev set stopped dropping. Maximum length of a decoded text was set to 40 tokens, and that of the tables was set to 120 tokens. Various beam sizes and length normalization penalties were applied for the table-to-text system, which are listed in the main paper. For the information extraction system, we found a beam size of 8 and no length penalty to produce the highest F-score on the dev set. Sample Outputs Table TABREF55 shows some sample references and the corresponding predictions from the best performing model, PG-Net for WikiBio.
How many people participated in their evaluation study of table-to-text models?
bf3b27a4f4be1f9ae31319877fd0c75c03126fd5
How Ġmany Ġpeople Ġparticipated Ġin Ġtheir Ġevaluation Ġstudy Ġof Ġtable - to - text Ġmodels ?
about Ġ500
<s> How Ġmany Ġpeople Ġparticipated Ġin Ġtheir Ġevaluation Ġstudy Ġof Ġtable - to - text Ġmodels ? </s> Introduction </s> The Ġtask Ġof Ġgenerating Ġnatural Ġlanguage Ġdescriptions Ġof Ġstructured Ġdata Ġ( such Ġas Ġtables ) ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġhas Ġseen Ġa Ġgrowth Ġin Ġinterest Ġwith Ġthe Ġrise Ġof Ġsequence Ġto Ġsequence Ġmodels Ġthat Ġprovide Ġan Ġeasy Ġway Ġof Ġencoding Ġtables Ġand Ġgenerating Ġtext Ġfrom Ġthem ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ. </s> For Ġtext Ġgeneration Ġtasks , Ġthe Ġonly Ġgold Ġstandard Ġmetric Ġis Ġto Ġshow Ġthe Ġoutput Ġto Ġhumans Ġfor Ġjudging Ġits Ġquality , Ġbut Ġthis Ġis Ġtoo Ġexpensive Ġto Ġapply Ġrepeatedly Ġanytime Ġsmall Ġmodifications Ġare Ġmade Ġto Ġa Ġsystem . ĠHence , Ġautomatic Ġmetrics Ġthat Ġcompare Ġthe Ġgenerated Ġtext Ġto Ġone Ġor Ġmore Ġreference Ġtexts Ġare Ġroutinely Ġused Ġto Ġcompare Ġmodels ĠB IB REF 7 Ġ. ĠFor Ġtable - to - text Ġgeneration , Ġautomatic Ġevaluation Ġhas Ġlargely Ġrelied Ġon ĠB LE U ĠB IB REF 8 Ġand ĠR OU GE ĠB IB REF 9 Ġ. ĠThe Ġunderlying Ġassumption Ġbehind Ġthese Ġmetrics Ġis Ġthat Ġthe Ġreference Ġtext Ġis Ġgold - standard , Ġi . e ., Ġit Ġis Ġthe Ġideal Ġtarget Ġtext Ġthat Ġa Ġsystem Ġshould Ġgenerate . ĠIn Ġpractice , Ġhowever , Ġwhen Ġdatasets Ġare Ġcollected Ġautomatically Ġand Ġhe ur istically , Ġthe Ġreference Ġtexts Ġare Ġoften Ġnot Ġideal . ĠFigure ĠFIG REF 2 Ġshows Ġan Ġexample Ġfrom Ġthe ĠWiki Bio Ġdataset ĠB IB REF 0 Ġ. ĠHere Ġthe Ġreference Ġcontains Ġextra Ġinformation Ġwhich Ġno Ġsystem Ġcan Ġbe Ġexpected Ġto Ġproduce Ġgiven Ġonly Ġthe Ġassociated Ġtable . ĠWe Ġcall Ġsuch Ġreference Ġtexts Ġdiver gent Ġfrom Ġthe Ġtable . </s> We Ġshow Ġthat Ġexisting Ġautomatic Ġmetrics , Ġincluding ĠB LE U , Ġcorrelate Ġpoorly Ġwith Ġhuman Ġjudgments Ġwhen Ġthe Ġevaluation Ġsets Ġcontain Ġdiver gent Ġreferences Ġ( Â§ ĠSEC REF 36 Ġ). ĠFor Ġmany Ġtable - to - text Ġgeneration Ġtasks , Ġthe Ġtables Ġthemselves Ġare Ġin Ġa Ġpseudo - natural Ġlanguage Ġformat Ġ( e . g ., ĠWiki Bio , ĠWeb NL G ĠB IB REF 6 Ġ, Ġand ĠE 2 E - NL G ĠB IB REF 10 Ġ). ĠIn Ġsuch Ġcases Ġwe Ġpropose Ġto Ġcompare Ġthe Ġgenerated Ġtext Ġto Ġthe Ġunderlying Ġtable Ġas Ġwell Ġto Ġimprove Ġevaluation . ĠWe Ġdevelop Ġa Ġnew Ġmetric , ĠPAR ENT Ġ( Pre cision ĠAnd ĠRecall Ġof ĠEnt ailed ĠN - gram s Ġfrom Ġthe ĠTable ) Ġ( Â§ ĠSEC REF 3 Ġ). ĠWhen Ġcomputing Ġprecision , ĠPAR ENT Ġeffectively Ġuses Ġa Ġunion Ġof Ġthe Ġreference Ġand Ġthe Ġtable , Ġto Ġreward Ġcorrect Ġinformation Ġmissing Ġfrom Ġthe Ġreference . ĠWhen Ġcomputing Ġrecall , Ġit Ġuses Ġan Ġintersection Ġof Ġthe Ġreference Ġand Ġthe Ġtable , Ġto Ġignore Ġextra Ġincorrect Ġinformation Ġin Ġthe Ġreference . ĠThe Ġunion Ġand Ġintersection Ġare Ġcomputed Ġwith Ġthe Ġhelp Ġof Ġan Ġentail ment Ġmodel Ġto Ġdecide Ġif Ġa Ġtext Ġn - gram Ġis Ġent ailed Ġby Ġthe Ġtable . ĠWe Ġshow Ġthat Ġthis Ġmethod Ġis Ġmore Ġeffective Ġthan Ġusing Ġthe Ġtable Ġas Ġan Ġadditional Ġreference . ĠOur Ġmain Ġcontributions Ġare : </s> Table - to - Text ĠGeneration </s> We Ġbriefly Ġreview Ġthe Ġtask Ġof Ġgenerating Ġnatural Ġlanguage Ġdescriptions Ġof Ġsemi - struct ured Ġdata , Ġwhich Ġwe Ġrefer Ġto Ġas Ġtables Ġhence forth ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ. ĠTables Ġcan Ġbe Ġexpressed Ġas Ġset Ġof Ġrecords ĠIN LINE FORM 0 Ġ, Ġwhere Ġeach Ġrecord Ġis Ġa Ġtuple Ġ( entity , Ġattribute , Ġvalue ). ĠWhen Ġall Ġthe Ġrecords Ġare Ġabout Ġthe Ġsame Ġentity , Ġwe Ġcan Ġtrunc ate Ġthe Ġrecords Ġto Ġ( attribute , Ġvalue ) Ġpairs . ĠFor Ġexample , Ġfor Ġthe Ġtable Ġin ĠFigure ĠFIG REF 2 Ġ, Ġthe Ġrecords Ġare Ġ{ ( Birth ĠName , ĠMichael ĠDahl quist ), Ġ( Born , ĠDecember Ġ22 Ġ1965 ), Ġ... }. ĠThe Ġtask Ġis Ġto Ġgenerate Ġa Ġtext ĠIN LINE FORM 1 Ġwhich Ġsummarizes Ġthe Ġrecords Ġin Ġa Ġfluent Ġand Ġgram matical Ġmanner . ĠFor Ġtraining Ġand Ġevaluation Ġwe Ġfurther Ġassume Ġthat Ġwe Ġhave Ġa Ġreference Ġdescription ĠIN LINE FORM 2 Ġavailable Ġfor Ġeach Ġtable . ĠWe Ġlet ĠIN LINE FORM 3 Ġdenote Ġan Ġevaluation Ġset Ġof Ġtables , Ġreferences Ġand Ġtexts Ġgenerated Ġfrom Ġa Ġmodel ĠIN LINE FORM 4 Ġ, Ġand ĠIN LINE FORM 5 Ġ, ĠIN LINE FORM 6 Ġdenote Ġthe Ġcollection Ġof Ġn - gram s Ġof Ġorder ĠIN LINE FORM 7 Ġin ĠIN LINE FORM 8 Ġand ĠIN LINE FORM 9 Ġ, Ġrespectively . ĠWe Ġuse ĠIN LINE FORM 10 Ġto Ġdenote Ġthe Ġcount Ġof Ġn - gram ĠIN LINE FORM 11 Ġin ĠIN LINE FORM 12 Ġ, Ġand ĠIN LINE FORM 13 Ġto Ġdenote Ġthe Ġminimum Ġof Ġits Ġcounts Ġin ĠIN LINE FORM 14 Ġand ĠIN LINE FORM 15 Ġ. ĠOur Ġgoal Ġis Ġto Ġassign Ġa Ġscore Ġto Ġthe Ġmodel , Ġwhich Ġcorrelates Ġhighly Ġwith Ġhuman Ġjudgments Ġof Ġthe Ġquality Ġof Ġthat Ġmodel . </s> PAR ENT </s> PAR ENT Ġevaluates Ġeach Ġinstance ĠIN LINE FORM 0 Ġseparately , Ġby Ġcomputing Ġthe Ġprecision Ġand Ġrecall Ġof ĠIN LINE FORM 1 Ġagainst Ġboth ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġ. </s> E val uation Ġvia ĠInformation ĠExt raction </s> BI B REF 1 Ġproposed Ġto Ġuse Ġan Ġauxiliary Ġmodel , Ġtrained Ġto Ġextract Ġstructured Ġrecords Ġfrom Ġtext , Ġfor Ġevaluation . ĠHowever , Ġthe Ġextraction Ġmodel Ġpresented Ġin Ġthat Ġwork Ġis Ġlimited Ġto Ġthe Ġclosed - domain Ġsetting Ġof Ġbasketball Ġgame Ġtables Ġand Ġsumm aries . ĠIn Ġparticular , Ġthey Ġassume Ġthat Ġeach Ġtable Ġhas Ġexactly Ġthe Ġsame Ġset Ġof Ġattributes Ġfor Ġeach Ġentity , Ġand Ġthat Ġthe Ġentities Ġcan Ġbe Ġidentified Ġin Ġthe Ġtext Ġvia Ġstring Ġmatching . ĠThese Ġassumptions Ġare Ġnot Ġvalid Ġfor Ġthe Ġopen - domain ĠWiki Bio Ġdataset , Ġand Ġhence Ġwe Ġtrain Ġour Ġown Ġextraction Ġmodel Ġto Ġreplicate Ġtheir Ġevaluation Ġscheme . </s> Our Ġextraction Ġsystem Ġis Ġa Ġpointer - gener ator Ġnetwork ĠB IB REF 19 Ġ, Ġwhich Ġlearns Ġto Ġproduce Ġa Ġlinear ized Ġversion Ġof Ġthe Ġtable Ġfrom Ġthe Ġtext . ĠThe Ġnetwork Ġlearns Ġwhich Ġattributes Ġneed Ġto Ġbe Ġpopulated Ġin Ġthe Ġoutput Ġtable , Ġalong Ġwith Ġtheir Ġvalues . ĠIt Ġis Ġtrained Ġon Ġthe Ġtraining Ġset Ġof ĠWiki Bio . ĠAt Ġtest Ġtime Ġwe Ġparsed Ġthe Ġoutput Ġstrings Ġinto Ġa Ġset Ġof Ġ( attribute , Ġvalue ) Ġtu ples Ġand Ġcompare Ġit Ġto Ġthe Ġground Ġtruth Ġtable . ĠThe ĠF - score Ġof Ġthis Ġtext - to - table Ġsystem Ġwas ĠIN LINE FORM 0 Ġ, Ġwhich Ġis Ġcomparable Ġto Ġother Ġchallenging Ġopen - domain Ġsettings ĠB IB REF 20 Ġ. ĠMore Ġdetails Ġare Ġincluded Ġin Ġthe ĠAppendix ĠSEC REF 52 Ġ. </s> Given Ġthis Ġinformation Ġextraction Ġsystem , Ġwe Ġconsider Ġthe Ġfollowing Ġmetrics Ġfor Ġevaluation , Ġalong Ġthe Ġlines Ġof ĠB IB REF 1 Ġ. ĠContent ĠSelection Ġ( CS ): ĠF - score Ġfor Ġthe Ġ( attribute , Ġvalue ) Ġpairs Ġextracted Ġfrom Ġthe Ġgenerated Ġtext Ġcompared Ġto Ġthose Ġextracted Ġfrom Ġthe Ġreference . ĠRel ation ĠGeneration Ġ( RG ): ĠPrecision Ġfor Ġthe Ġ( attribute , Ġvalue ) Ġpairs Ġextracted Ġfrom Ġthe Ġgenerated Ġtext Ġcompared Ġto Ġthose Ġin Ġthe Ġground Ġtruth Ġtable . ĠRG - F : ĠSince Ġour Ġtask Ġemphasizes Ġthe Ġrecall Ġof Ġinformation Ġfrom Ġthe Ġtable Ġas Ġwell , Ġwe Ġconsider Ġanother Ġvariant Ġwhich Ġcomp utes Ġthe ĠF - score Ġof Ġthe Ġextracted Ġpairs Ġto Ġthose Ġin Ġthe Ġtable . ĠWe Ġomit Ġthe Ġcontent Ġordering Ġmetric , Ġsince Ġour Ġextraction Ġsystem Ġdoes Ġnot Ġalign Ġrecords Ġto Ġthe Ġinput Ġtext . </s> Exper iments Ġ& ĠResults </s> In Ġthis Ġsection Ġwe Ġcompare Ġseveral Ġautomatic Ġevaluation Ġmetrics Ġby Ġchecking Ġtheir Ġcorrelation Ġwith Ġthe Ġscores Ġassigned Ġby Ġhumans Ġto Ġtable - to - text Ġmodels . ĠSpecifically , Ġgiven ĠIN LINE FORM 0 Ġmodels ĠIN LINE FORM 1 Ġ, Ġand Ġtheir Ġoutputs Ġon Ġan Ġevaluation Ġset , Ġwe Ġshow Ġthese Ġgenerated Ġtexts Ġto Ġhumans Ġto Ġjudge Ġtheir Ġquality , Ġand Ġobtain Ġaggreg ated Ġhuman Ġevaluation Ġscores Ġfor Ġall Ġthe Ġmodels , ĠIN LINE FORM 2 Ġ( Â§ ĠSEC REF 33 Ġ). ĠNext , Ġto Ġevaluate Ġan Ġautomatic Ġmetric , Ġwe Ġcompute Ġthe Ġscores Ġit Ġassigns Ġto Ġeach Ġmodel , ĠIN LINE FORM 3 Ġ, Ġand Ġcheck Ġthe ĠPearson Ġcorrelation Ġbetween ĠIN LINE FORM 4 Ġand ĠIN LINE FORM 5 ĠB IB REF 21 Ġ. </s> Data Ġ& ĠModels </s> Our Ġmain Ġexperiments Ġare Ġon Ġthe ĠWiki Bio Ġdataset ĠB IB REF 0 Ġ, Ġwhich Ġis Ġautomatically Ġconstructed Ġand Ġcontains Ġmany Ġdiver gent Ġreferences . ĠIn ĠÂ§ ĠSEC REF 47 Ġwe Ġalso Ġpresent Ġresults Ġon Ġthe Ġdata Ġreleased Ġas Ġpart Ġof Ġthe ĠWeb NL G Ġchallenge . </s> We Ġdeveloped Ġseveral Ġmodels Ġof Ġvarying Ġquality Ġfor Ġgenerating Ġtext Ġfrom Ġthe Ġtables Ġin ĠWiki Bio . ĠThis Ġgives Ġus Ġa Ġdiverse Ġset Ġof Ġoutputs Ġto Ġevaluate Ġthe Ġautomatic Ġmetrics Ġon . ĠTable ĠT AB REF 32 Ġlists Ġthe Ġmodels Ġalong Ġwith Ġtheir Ġhyper param eter Ġsettings Ġand Ġtheir Ġscores Ġfrom Ġthe Ġhuman Ġevaluation Ġ( Â§ ĠSEC REF 33 Ġ). ĠOur Ġfocus Ġis Ġprimarily Ġon Ġneural Ġsequence - to - sequence Ġmethods Ġsince Ġthese Ġare Ġmost Ġwidely Ġused , Ġbut Ġwe Ġalso Ġinclude Ġa Ġtemplate - based Ġbaseline . ĠAll Ġneural Ġmodels Ġwere Ġtrained Ġon Ġthe ĠWiki Bio Ġtraining Ġset . ĠTraining Ġdetails Ġand Ġsample Ġoutputs Ġare Ġincluded Ġin ĠApp end ices ĠSEC REF 56 Ġ& ĠSEC REF 57 Ġ. </s> We Ġdivide Ġthese Ġmodels Ġinto Ġtwo Ġcategories Ġand Ġmeasure Ġcorrelation Ġseparately Ġfor Ġboth Ġthe Ġcategories . ĠThe Ġfirst Ġcategory , ĠWiki Bio - System s , Ġincludes Ġone Ġmodel Ġeach Ġfrom Ġthe Ġfour Ġfamilies Ġlisted Ġin ĠTable ĠT AB REF 32 Ġ. ĠThis Ġcategory Ġtests Ġwhether Ġa Ġmetric Ġcan Ġbe Ġused Ġto Ġcompare Ġdifferent Ġmodel Ġfamilies Ġwith Ġa Ġlarge Ġvariation Ġin Ġthe Ġquality Ġof Ġtheir Ġoutputs . ĠThe Ġsecond Ġcategory , ĠWiki Bio - Hyper params , Ġincludes Ġ13 Ġdifferent Ġhyper param eter Ġsettings Ġof ĠPG - Net ĠB IB REF 19 Ġ, Ġwhich Ġwas Ġthe Ġbest Ġperforming Ġsystem Ġoverall . Ġ9 Ġof Ġthese Ġwere Ġobtained Ġby Ġvarying Ġthe Ġbeam Ġsize Ġand Ġlength Ġnormal ization Ġpenalty Ġof Ġthe Ġdec oder Ġnetwork ĠB IB REF 23 Ġ, Ġand Ġthe Ġremaining Ġ4 Ġwere Ġobtained Ġby Ġre - scoring Ġbeams Ġof Ġsize Ġ8 Ġwith Ġthe Ġinformation Ġextraction Ġmodel Ġdescribed Ġin ĠÂ§ ĠSEC REF 4 Ġ. ĠAll Ġthe Ġmodels Ġin Ġthis Ġcategory Ġproduce Ġhigh Ġquality Ġfluent Ġtexts , Ġand Ġdiffer Ġprimarily Ġon Ġthe Ġquantity Ġand Ġaccuracy Ġof Ġthe Ġinformation Ġthey Ġexpress . ĠHere Ġwe Ġare Ġtesting Ġwhether Ġa Ġmetric Ġcan Ġbe Ġused Ġto Ġcompare Ġsimilar Ġsystems Ġwith Ġa Ġsmall Ġvariation Ġin Ġperformance . ĠThis Ġis Ġan Ġimportant Ġuse - case Ġas Ġmetrics Ġare Ġoften Ġused Ġto Ġtune Ġhyper param eters Ġof Ġa Ġmodel . </s> Human ĠEvaluation </s> We Ġcollected Ġhuman Ġjudgments Ġon Ġthe Ġquality Ġof Ġthe Ġ16 Ġmodels Ġtrained Ġfor ĠWiki Bio , Ġplus Ġthe Ġreference Ġtexts . ĠWorkers Ġon Ġa Ġcrowd - s ourcing Ġplatform , Ġproficient Ġin ĠEnglish , Ġwere Ġshown Ġa Ġtable Ġwith Ġpairs Ġof Ġgenerated Ġtexts , Ġor Ġa Ġgenerated Ġtext Ġand Ġthe Ġreference , Ġand Ġasked Ġto Ġselect Ġthe Ġone Ġthey Ġprefer . ĠFigure ĠFIG REF 34 Ġshows Ġthe Ġinstructions Ġthey Ġwere Ġgiven . ĠP aired Ġcomparisons Ġhave Ġbeen Ġshown Ġto Ġbe Ġsuperior Ġto Ġrating Ġscales Ġfor Ġcomparing Ġgenerated Ġtexts ĠB IB REF 24 Ġ. ĠHowever , Ġfor Ġmeasuring Ġcorrelation Ġthe Ġcomparisons Ġneed Ġto Ġbe Ġaggreg ated Ġinto Ġreal - valued Ġscores , ĠIN LINE FORM 0 Ġ, Ġfor Ġeach Ġof Ġthe ĠIN LINE FORM 1 Ġmodels . ĠFor Ġthis , Ġwe Ġuse ĠThur stone 's Ġmethod ĠB IB REF 22 Ġ, Ġwhich Ġassigns Ġa Ġscore Ġto Ġeach Ġmodel Ġbased Ġon Ġhow Ġmany Ġtimes Ġit Ġwas Ġpreferred Ġover Ġan Ġalternative . </s> The Ġdata Ġcollection Ġwas Ġperformed Ġseparately Ġfor Ġmodels Ġin Ġthe ĠWiki Bio - System s Ġand ĠWiki Bio - Hyper params Ġcategories . Ġ1100 Ġtables Ġwere Ġsampled Ġfrom Ġthe Ġdevelopment Ġset , Ġand Ġfor Ġeach Ġtable Ġwe Ġgot Ġ8 Ġdifferent Ġsentence Ġpairs Ġannot ated Ġacross Ġthe Ġtwo Ġcategories , Ġresulting Ġin Ġa Ġtotal Ġof Ġ8 800 Ġpair wise Ġcomparisons . ĠEach Ġpair Ġwas Ġjudged Ġby Ġone Ġworker Ġonly Ġwhich Ġmeans Ġthere Ġmay Ġbe Ġnoise Ġat Ġthe Ġinstance - level , Ġbut Ġthe Ġaggreg ated Ġsystem - level Ġscores Ġhad Ġlow Ġvariance Ġ( cf . ĠTable ĠT AB REF 32 Ġ). ĠIn Ġtotal Ġaround Ġ500 Ġdifferent Ġworkers Ġwere Ġinvolved Ġin Ġthe Ġannotation . ĠReferences Ġwere Ġalso Ġincluded Ġin Ġthe Ġevaluation , Ġand Ġthey Ġreceived Ġa Ġlower Ġscore Ġthan ĠPG - Net , Ġhighlighting Ġthe Ġdivergence Ġin ĠWiki Bio . </s> Compared ĠMet rics </s> Text Ġonly : ĠWe Ġcompare ĠB LE U ĠB IB REF 8 Ġ, ĠR OU GE ĠB IB REF 9 Ġ, ĠMET E OR ĠB IB REF 18 Ġ, ĠC ID Er Ġand ĠC ID Er - D ĠB IB REF 25 Ġusing Ġtheir Ġpublicly Ġavailable Ġimplementations . </s> Information ĠExt raction Ġbased : ĠWe Ġcompare Ġthe ĠCS , ĠRG Ġand ĠRG - F Ġmetrics Ġdiscussed Ġin ĠÂ§ ĠSEC REF 4 Ġ. </s> Text Ġ& ĠTable : ĠWe Ġcompare Ġa Ġvariant Ġof ĠB LE U , Ġden oted Ġas ĠB LE U - T , Ġwhere Ġthe Ġvalues Ġfrom Ġthe Ġtable Ġare Ġused Ġas Ġadditional Ġreferences . ĠB LE U - T Ġdraws Ġinspiration Ġfrom Ġi BLE U ĠB IB REF 26 Ġbut Ġinstead Ġrewards Ġn - gram s Ġwhich Ġmatch Ġthe Ġtable Ġrather Ġthan Ġpenal izing Ġthem . ĠFor ĠPAR ENT , Ġwe Ġcompare Ġboth Ġthe Ġword - over lap Ġmodel Ġ( PAR ENT - W ) Ġand Ġthe Ġco - occ urrence Ġmodel Ġ( PAR ENT - C ) Ġfor Ġdetermining Ġentail ment . ĠWe Ġalso Ġcompare Ġversions Ġwhere Ġa Ġsingle ĠIN LINE FORM 0 Ġis Ġtuned Ġon Ġthe Ġentire Ġdataset Ġto Ġmaximize Ġcorrelation Ġwith Ġhuman Ġjudgments , Ġden oted Ġas ĠPAR ENT * - W / C . </s> Cor relation ĠComparison </s> We Ġuse Ġboot strap Ġsampling Ġ( 500 Ġiterations ) Ġover Ġthe Ġ1100 Ġtables Ġfor Ġwhich Ġwe Ġcollected Ġhuman Ġannotations Ġto Ġget Ġan Ġidea Ġof Ġhow Ġthe Ġcorrelation Ġof Ġeach Ġmetric Ġvaries Ġwith Ġthe Ġunderlying Ġdata . ĠIn Ġeach Ġiteration , Ġwe Ġsample Ġwith Ġreplacement , Ġtables Ġalong Ġwith Ġtheir Ġreferences Ġand Ġall Ġthe Ġgenerated Ġtexts Ġfor Ġthat Ġtable . ĠThen Ġwe Ġcompute Ġaggreg ated Ġhuman Ġevaluation Ġand Ġmetric Ġscores Ġfor Ġeach Ġof Ġthe Ġmodels Ġand Ġcompute Ġthe Ġcorrelation Ġbetween Ġthe Ġtwo . ĠWe Ġreport Ġthe Ġaverage Ġcorrelation Ġacross Ġall Ġboot strap Ġsamples Ġfor Ġeach Ġmetric Ġin ĠTable ĠT AB REF 37 Ġ. ĠThe Ġdistribution Ġof Ġcorrelations Ġfor Ġthe Ġbest Ġperforming Ġmetrics Ġare Ġshown Ġin ĠFigure ĠFIG REF 38 Ġ. </s> Table ĠT AB REF 37 Ġalso Ġindicates Ġwhether ĠPAR ENT Ġis Ġsignificantly Ġbetter Ġthan Ġa Ġbaseline Ġmetric . ĠB IB REF 21 Ġsuggest Ġusing Ġthe ĠWilliam 's Ġtest Ġfor Ġthis Ġpurpose , Ġbut Ġsince Ġwe Ġare Ġcomputing Ġcorrelations Ġbetween Ġonly Ġ4 / 13 Ġsystems Ġat Ġa Ġtime , Ġthis Ġtest Ġhas Ġvery Ġweak Ġpower Ġin Ġour Ġcase . ĠHence , Ġwe Ġuse Ġthe Ġboot strap Ġsamples Ġto Ġobtain Ġa ĠIN LINE FORM 0 Ġconfidence Ġinterval Ġof Ġthe Ġdifference Ġin Ġcorrelation Ġbetween ĠPAR ENT Ġand Ġany Ġother Ġmetric Ġand Ġcheck Ġwhether Ġthis Ġis Ġabove Ġ0 ĠB IB REF 27 Ġ. </s> Cor relations Ġare Ġhigher Ġfor Ġthe Ġsystems Ġcategory Ġthan Ġthe Ġhyper params Ġcategory . ĠThe Ġlatter Ġis Ġa Ġmore Ġdifficult Ġsetting Ġsince Ġvery Ġsimilar Ġmodels Ġare Ġcompared , Ġand Ġhence Ġthe Ġvariance Ġof Ġthe Ġcorrelations Ġis Ġalso Ġhigh . ĠCommon ly Ġused Ġmetrics Ġwhich Ġonly Ġrely Ġon Ġthe Ġreference Ġ( BLE U , ĠR OU GE , ĠMET E OR , ĠC ID Er ) Ġhave Ġonly Ġweak Ġcorrelations Ġwith Ġhuman Ġjudgments . ĠIn Ġthe Ġhyper params Ġcategory , Ġthese Ġare Ġoften Ġnegative , Ġimplying Ġthat Ġtuning Ġmodels Ġbased Ġon Ġthese Ġmay Ġlead Ġto Ġselecting Ġworse Ġmodels . ĠB LE U Ġperforms Ġthe Ġbest Ġamong Ġthese , Ġand Ġadding Ġn - gram s Ġfrom Ġthe Ġtable Ġas Ġreferences Ġimproves Ġthis Ġfurther Ġ( BLE U - T ). </s> Among Ġthe Ġextract ive Ġevaluation Ġmetrics , ĠCS , Ġwhich Ġalso Ġonly Ġrelies Ġon Ġthe Ġreference , Ġhas Ġpoor Ġcorrelation Ġin Ġthe Ġhyper params Ġcategory . ĠRG - F , Ġand Ġboth Ġvariants Ġof Ġthe ĠPAR ENT Ġmetric Ġachieve Ġthe Ġhighest Ġcorrelation Ġfor Ġboth Ġsettings . ĠThere Ġis Ġno Ġsignificant Ġdifference Ġamong Ġthese Ġfor Ġthe Ġhyper params Ġcategory , Ġbut Ġfor Ġsystems , ĠPAR ENT - W Ġis Ġsignificantly Ġbetter Ġthan Ġthe Ġother Ġtwo . ĠWhile ĠRG - F Ġneeds Ġa Ġfull Ġinformation Ġextraction Ġpipeline Ġin Ġits Ġimplementation , ĠPAR ENT - C Ġonly Ġrelies Ġon Ġco - occ urrence Ġcounts , Ġand ĠPAR ENT - W Ġcan Ġbe Ġused Ġout - of - the - box Ġfor Ġany Ġdataset . ĠTo Ġour Ġknowledge , Ġthis Ġis Ġthe Ġfirst Ġrigorous Ġevaluation Ġof Ġusing Ġinformation Ġextraction Ġfor Ġgeneration Ġevaluation . </s> On Ġthis Ġdataset , Ġthe Ġword - over lap Ġmodel Ġshowed Ġhigher Ġcorrelation Ġthan Ġthe Ġco - occ urrence Ġmodel Ġfor Ġentail ment . ĠIn ĠÂ§ ĠSEC REF 47 Ġwe Ġwill Ġshow Ġthat Ġfor Ġthe ĠWeb NL G Ġdataset , Ġwhere Ġmore Ġparaph r asing Ġis Ġinvolved Ġbetween Ġthe Ġtable Ġand Ġtext , Ġthe Ġopposite Ġis Ġtrue . ĠLastly , Ġwe Ġnote Ġthat Ġthe Ġhe uristic Ġfor Ġselecting ĠIN LINE FORM 0 Ġis Ġsufficient Ġto Ġproduce Ġhigh Ġcorrelations Ġfor ĠPAR ENT , Ġhowever , Ġif Ġhuman Ġannotations Ġare Ġavailable , Ġthis Ġcan Ġbe Ġtuned Ġto Ġproduce Ġsignificantly Ġhigher Ġcorrelations Ġ( PAR ENT * - W / C ). </s> Analysis </s> In Ġthis Ġsection Ġwe Ġfurther Ġanalyze Ġthe Ġperformance Ġof ĠPAR ENT - W Ġunder Ġdifferent Ġconditions , Ġand Ġcompare Ġto Ġthe Ġother Ġbest Ġmetrics Ġfrom ĠTable ĠT AB REF 37 Ġ. </s> To Ġstudy Ġthe Ġcorrelation Ġas Ġwe Ġvary Ġthe Ġnumber Ġof Ġdiver gent Ġreferences , Ġwe Ġalso Ġcollected Ġbinary Ġlabels Ġfrom Ġworkers Ġfor Ġwhether Ġa Ġreference Ġis Ġent ailed Ġby Ġthe Ġcorresponding Ġtable . ĠWe Ġdefine Ġa Ġreference Ġas Ġent ailed Ġwhen Ġit Ġmentions Ġonly Ġinformation Ġwhich Ġcan Ġbe Ġinferred Ġfrom Ġthe Ġtable . ĠEach Ġtable Ġand Ġreference Ġpair Ġwas Ġjudged Ġby Ġ3 Ġindependent Ġworkers , Ġand Ġwe Ġused Ġthe Ġmajority Ġvote Ġas Ġthe Ġlabel Ġfor Ġthat Ġpair . ĠOverall , Ġonly ĠIN LINE FORM 0 Ġof Ġthe Ġreferences Ġwere Ġlabeled Ġas Ġent ailed Ġby Ġthe Ġtable . ĠFle iss ' ĠIN LINE FORM 1 Ġwas ĠIN LINE FORM 2 Ġ, Ġwhich Ġindicates Ġa Ġfair Ġagreement . ĠWe Ġfound Ġthe Ġworkers Ġsometimes Ġdisagreed Ġon Ġwhat Ġinformation Ġcan Ġbe Ġreasonably Ġent ailed Ġby Ġthe Ġtable . </s> Figure ĠFIG REF 40 Ġshows Ġthe Ġcorrelations Ġas Ġwe Ġvary Ġthe Ġpercent Ġof Ġent ailed Ġexamples Ġin Ġthe Ġevaluation Ġset Ġof ĠWiki Bio . ĠEach Ġpoint Ġis Ġobtained Ġby Ġfixing Ġthe Ġdesired Ġproportion Ġof Ġent ailed Ġexamples , Ġand Ġsampling Ġsubs ets Ġfrom Ġthe Ġfull Ġset Ġwhich Ġsatisfy Ġthis Ġproportion . ĠPAR ENT Ġand ĠRG - F Ġremain Ġstable Ġand Ġshow Ġa Ġhigh Ġcorrelation Ġacross Ġthe Ġentire Ġrange , Ġwhereas ĠB LE U Ġand ĠB LE U - T Ġvary Ġa Ġlot . ĠIn Ġthe Ġhyper params Ġcategory , Ġthe Ġlatter Ġtwo Ġhave Ġthe Ġworst Ġcorrelation Ġwhen Ġthe Ġevaluation Ġset Ġcontains Ġonly Ġent ailed Ġexamples , Ġwhich Ġmay Ġseem Ġsurprising . ĠHowever , Ġon Ġcloser Ġexamination Ġwe Ġfound Ġthat Ġthis Ġsubset Ġtends Ġto Ġomit Ġa Ġlot Ġof Ġinformation Ġfrom Ġthe Ġtables . ĠSystems Ġwhich Ġproduce Ġmore Ġinformation Ġthan Ġthese Ġreferences Ġare Ġpenal ized Ġby ĠB LE U , Ġbut Ġnot Ġin Ġthe Ġhuman Ġevaluation . ĠPAR ENT Ġover comes Ġthis Ġissue Ġby Ġmeasuring Ġrecall Ġagainst Ġthe Ġtable Ġin Ġaddition Ġto Ġthe Ġreference . </s> We Ġcheck Ġhow Ġdifferent Ġcomponents Ġin Ġthe Ġcomputation Ġof ĠPAR ENT Ġcontribute Ġto Ġits Ġcorrelation Ġto Ġhuman Ġjudgments . ĠSpecifically , Ġwe Ġremove Ġthe Ġprobability ĠIN LINE FORM 0 Ġof Ġan Ġn - gram ĠIN LINE FORM 1 Ġbeing Ġent ailed Ġby Ġthe Ġtable Ġfrom ĠE qs . ĠEQ REF 19 Ġand ĠEQ REF 23 Ġ. ĠThe Ġaverage Ġcorrelation Ġfor ĠPAR ENT - W Ġdrops Ġto ĠIN LINE FORM 5 Ġin Ġthis Ġcase . ĠWe Ġalso Ġtry Ġa Ġvariant Ġof ĠPAR ENT Ġwith ĠIN LINE FORM 6 Ġ, Ġwhich Ġremoves Ġthe Ġcontribution Ġof ĠTable ĠRecall Ġ( E q . ĠEQ REF 22 Ġ). ĠThe Ġaverage Ġcorrelation Ġis ĠIN LINE FORM 7 Ġin Ġthis Ġcase . ĠWith Ġthese Ġcomponents , Ġthe Ġcorrelation Ġis ĠIN LINE FORM 8 Ġ, Ġshowing Ġthat Ġthey Ġare Ġcrucial Ġto Ġthe Ġperformance Ġof ĠPAR ENT . </s> BI B REF 28 Ġpoint Ġout Ġthat Ġhill - cl im bing Ġon Ġan Ġautomatic Ġmetric Ġis Ġmeaningless Ġif Ġthat Ġmetric Ġhas Ġa Ġlow Ġinstance - level Ġcorrelation Ġto Ġhuman Ġjudgments . ĠIn ĠTable ĠT AB REF 46 Ġwe Ġshow Ġthe Ġaverage Ġaccuracy Ġof Ġthe Ġmetrics Ġin Ġmaking Ġthe Ġsame Ġjudgments Ġas Ġhumans Ġbetween Ġpairs Ġof Ġgenerated Ġtexts . ĠBoth Ġvariants Ġof ĠPAR ENT Ġare Ġsignificantly Ġbetter Ġthan Ġthe Ġother Ġmetrics , Ġhowever Ġthe Ġbest Ġaccuracy Ġis Ġonly ĠIN LINE FORM 0 Ġfor Ġthe Ġbinary Ġtask . ĠThis Ġis Ġa Ġchallenging Ġtask , Ġsince Ġthere Ġare Ġtypically Ġonly Ġsubtle Ġdifferences Ġbetween Ġthe Ġtexts . ĠA chie ving Ġhigher Ġinstance - level Ġaccur acies Ġwill Ġrequire Ġmore Ġsophisticated Ġlanguage Ġunderstanding Ġmodels Ġfor Ġevaluation . </s> Web NL G ĠDat as et </s> To Ġcheck Ġhow ĠPAR ENT Ġcorrelates Ġwith Ġhuman Ġjudgments Ġwhen Ġthe Ġreferences Ġare Ġelic ited Ġfrom Ġhumans Ġ( and Ġless Ġlikely Ġto Ġbe Ġdiver gent ), Ġwe Ġcheck Ġits Ġcorrelation Ġwith Ġthe Ġhuman Ġratings Ġprovided Ġfor Ġthe Ġsystems Ġcompeting Ġin Ġthe ĠWeb NL G Ġchallenge ĠB IB REF 6 Ġ. ĠThe Ġtask Ġis Ġto Ġgenerate Ġtext Ġdescribing Ġ1 - 5 ĠR DF Ġtri ples Ġ( e . g . ĠJohn ĠE ĠBl aha , Ġbirth Place , ĠSan ĠAntonio ), Ġand Ġhuman Ġratings Ġwere Ġcollected Ġfor Ġthe Ġoutputs Ġof Ġ9 Ġparticipating Ġsystems Ġon Ġ223 Ġinstances . ĠThese Ġsystems Ġinclude Ġa Ġmix Ġof Ġpip el ined , Ġstatistical Ġand Ġneural Ġmethods . ĠEach Ġinstance Ġhas Ġupt o Ġ3 Ġreference Ġtexts Ġassociated Ġwith Ġthe ĠR DF Ġtri ples , Ġwhich Ġwe Ġuse Ġfor Ġevaluation . </s> The Ġhuman Ġratings Ġwere Ġcollected Ġon Ġ3 Ġdistinct Ġaspects ĠâĢĵ Ġgram matical ity , Ġflu ency Ġand Ġsemantics , Ġwhere Ġsemantics Ġcorresponds Ġto Ġthe Ġdegree Ġto Ġwhich Ġa Ġgenerated Ġtext Ġagrees Ġwith Ġthe Ġmeaning Ġof Ġthe Ġunderlying ĠR DF Ġtri ples . ĠWe Ġreport Ġthe Ġcorrelation Ġof Ġseveral Ġmetrics Ġwith Ġthese Ġratings Ġin ĠTable ĠT AB REF 48 Ġ. ĠBoth Ġvariants Ġof ĠPAR ENT Ġare Ġeither Ġcompetitive Ġor Ġbetter Ġthan Ġthe Ġother Ġmetrics Ġin Ġterms Ġof Ġthe Ġaverage Ġcorrelation Ġto Ġall Ġthree Ġaspects . ĠThis Ġshows Ġthat ĠPAR ENT Ġis Ġapplicable Ġfor Ġhigh Ġquality Ġreferences Ġas Ġwell . </s> While ĠB LE U Ġhas Ġthe Ġhighest Ġcorrelation Ġfor Ġthe Ġgrammar Ġand Ġflu ency Ġaspects , ĠPAR ENT Ġdoes Ġbest Ġfor Ġsemantics . ĠThis Ġsuggests Ġthat Ġthe Ġinclusion Ġof Ġsource Ġtables Ġinto Ġthe Ġevaluation Ġo rients Ġthe Ġmetric Ġmore Ġtowards Ġmeasuring Ġthe Ġfidelity Ġof Ġthe Ġcontent Ġof Ġthe Ġgeneration . ĠA Ġsimilar Ġtrend Ġis Ġseen Ġcomparing ĠB LE U Ġand ĠB LE U - T . ĠAs Ġmodern Ġneural Ġtext Ġgeneration Ġsystems Ġare Ġtypically Ġvery Ġfluent , Ġmeasuring Ġtheir Ġfidelity Ġis Ġof Ġincreasing Ġimportance . ĠBetween Ġthe Ġtwo Ġentail ment Ġmodels , ĠPAR ENT - C Ġis Ġbetter Ġdue Ġto Ġits Ġhigher Ġcorrelation Ġwith Ġthe Ġgram matical ity Ġand Ġflu ency Ġaspects . </s> The ĠIN LINE FORM 0 Ġparameter Ġin Ġthe Ġcalculation Ġof ĠPAR ENT Ġdecides Ġwhether Ġto Ġcompute Ġrecall Ġagainst Ġthe Ġtable Ġor Ġthe Ġreference Ġ( E q . ĠEQ REF 22 Ġ). ĠFigure ĠFIG REF 50 Ġshows Ġthe Ġdistribution Ġof Ġthe Ġvalues Ġtaken Ġby ĠIN LINE FORM 1 Ġusing Ġthe Ġhe uristic Ġdescribed Ġin ĠÂ§ ĠSEC REF 3 Ġfor Ġinstances Ġin Ġboth ĠWiki Bio Ġand ĠWeb NL G . ĠFor ĠWiki Bio , Ġthe Ġrecall Ġof Ġthe Ġreferences Ġagainst Ġthe Ġtable Ġis Ġgenerally Ġlow , Ġand Ġhence Ġthe Ġrecall Ġof Ġthe Ġgenerated Ġtext Ġrelies Ġmore Ġon Ġthe Ġtable . ĠFor ĠWeb NL G , Ġwhere Ġthe Ġreferences Ġare Ġelic ited Ġfrom Ġhumans , Ġthis Ġrecall Ġis Ġmuch Ġhigher Ġ( often ĠIN LINE FORM 2 Ġ), Ġand Ġhence Ġthe Ġrecall Ġof Ġthe Ġgenerated Ġtext Ġrelies Ġmore Ġon Ġthe Ġreference . </s> Related ĠWork </s> Over Ġthe Ġyears Ġseveral Ġstudies Ġhave Ġevaluated Ġautomatic Ġmetrics Ġfor Ġmeasuring Ġtext Ġgeneration Ġperformance ĠB IB REF 29 Ġ, ĠB IB REF 30 Ġ, ĠB IB REF 31 Ġ, ĠB IB REF 32 Ġ, ĠB IB REF 33 Ġ, ĠB IB REF 34 Ġ, ĠB IB REF 35 Ġ. ĠThe Ġonly Ġconsensus Ġfrom Ġthese Ġstudies Ġseems Ġto Ġbe Ġthat Ġno Ġsingle Ġmetric Ġis Ġsuitable Ġacross Ġall Ġtasks . ĠA Ġrecurring Ġtheme Ġis Ġthat Ġmetrics Ġlike ĠB LE U Ġand ĠN IST ĠB IB REF 36 Ġare Ġnot Ġsuitable Ġfor Ġjudging Ġcontent Ġquality Ġin ĠNL G . ĠRecently , ĠB IB REF 37 Ġdid Ġa Ġcomprehensive Ġstudy Ġof Ġseveral Ġmetrics Ġon Ġthe Ġoutputs Ġof Ġstate - of - the - art ĠNL G Ġsystems , Ġand Ġfound Ġthat Ġwhile Ġthey Ġshowed Ġacceptable Ġcorrelation Ġwith Ġhuman Ġjudgments Ġat Ġthe Ġsystem Ġlevel , Ġthey Ġfailed Ġto Ġshow Ġany Ġcorrelation Ġat Ġthe Ġsentence Ġlevel . ĠO urs Ġis Ġthe Ġfirst Ġstudy Ġwhich Ġchecks Ġthe Ġquality Ġof Ġmetrics Ġwhen Ġtable - to - text Ġreferences Ġare Ġdiver gent . ĠWe Ġshow Ġthat Ġin Ġthis Ġcase Ġeven Ġsystem Ġlevel Ġcorrelations Ġcan Ġbe Ġunreliable . </s> Hall uc ination ĠB IB REF 38 Ġ, ĠB IB REF 39 Ġrefers Ġto Ġwhen Ġan ĠNL G Ġsystem Ġgenerates Ġtext Ġwhich Ġmentions Ġextra Ġinformation Ġthan Ġwhat Ġis Ġpresent Ġin Ġthe Ġsource Ġfrom Ġwhich Ġit Ġis Ġgenerated . ĠDiver gence Ġcan Ġbe Ġviewed Ġas Ġhalluc ination Ġin Ġthe Ġreference Ġtext Ġitself . ĠPAR ENT Ġdeals Ġwith Ġhalluc ination Ġby Ġdiscount ing Ġn - gram s Ġwhich Ġdo Ġnot Ġoverlap Ġwith Ġeither Ġthe Ġreference Ġor Ġthe Ġtable . </s> PAR ENT Ġdraws Ġinspiration Ġfrom Ġi BLE U ĠB IB REF 26 Ġ, Ġa Ġmetric Ġfor Ġevaluating Ġparaph rase Ġgeneration , Ġwhich Ġcompares Ġthe Ġgenerated Ġtext Ġto Ġboth Ġthe Ġsource Ġtext Ġand Ġthe Ġreference . ĠWhile Ġi BLE U Ġpenal izes Ġtexts Ġwhich Ġmatch Ġthe Ġsource , Ġhere Ġwe Ġreward Ġsuch Ġtexts Ġsince Ġour Ġtask Ġvalues Ġaccuracy Ġof Ġgenerated Ġtext Ġmore Ġthan Ġthe Ġneed Ġfor Ġparaph r asing Ġthe Ġtab ular Ġcontent ĠB IB REF 40 Ġ. ĠSimilar Ġto ĠS ARI Ġfor Ġtext Ġsimpl ification ĠB IB REF 41 Ġand ĠQ - BLE U Ġfor Ġquestion Ġgeneration ĠB IB REF 42 Ġ, ĠPAR ENT Ġfalls Ġunder Ġthe Ġcategory Ġof Ġtask - specific Ġmetrics . </s> Con clusions </s> We Ġstudy Ġthe Ġautomatic Ġevaluation Ġof Ġtable - to - text Ġsystems Ġwhen Ġthe Ġreferences Ġdiver ge Ġfrom Ġthe Ġtable . ĠWe Ġpropose Ġa Ġnew Ġmetric , ĠPAR ENT , Ġwhich Ġshows Ġthe Ġhighest Ġcorrelation Ġwith Ġhumans Ġacross Ġa Ġrange Ġof Ġsettings Ġwith Ġdiver gent Ġreferences Ġin ĠWiki Bio . ĠWe Ġalso Ġperform Ġthe Ġfirst Ġempirical Ġevaluation Ġof Ġinformation Ġextraction Ġbased Ġmetrics ĠB IB REF 1 Ġ, Ġand Ġfind ĠRG - F Ġto Ġbe Ġeffective . ĠLastly , Ġwe Ġshow Ġthat ĠPAR ENT Ġis Ġcomparable Ġto Ġthe Ġbest Ġexisting Ġmetrics Ġwhen Ġreferences Ġare Ġelic ited Ġby Ġhumans Ġon Ġthe ĠWeb NL G Ġdata . </s> Acknowled gements </s> B hu wan ĠD hing ra Ġis Ġsupported Ġby Ġa Ġfellowship Ġfrom ĠSiem ens , Ġand Ġby Ġgrants Ġfrom ĠGoogle . ĠWe Ġthank ĠMar uan ĠAl - S hed iv at , ĠIan ĠTen ney , ĠTom ĠKw iat kowski , ĠMichael ĠCollins , ĠSlav ĠPet rov , ĠJason ĠBald ridge , ĠDavid ĠRe itter Ġand Ġother Ġmembers Ġof Ġthe ĠGoogle ĠAI ĠLanguage Ġteam Ġfor Ġhelpful Ġdiscussions Ġand Ġsuggestions . ĠWe Ġthank ĠSam ĠWis eman Ġfor Ġsharing Ġdata Ġfor Ġan Ġearlier Ġversion Ġof Ġthis Ġpaper . ĠWe Ġalso Ġthank Ġthe Ġanonymous Ġreviewers Ġfor Ġtheir Ġfeedback . </s> Information ĠExt raction ĠSystem </s> For Ġevaluation Ġvia Ġinformation Ġextraction ĠB IB REF 1 Ġwe Ġtrain Ġa Ġmodel Ġfor ĠWiki Bio Ġwhich Ġaccepts Ġtext Ġas Ġinput Ġand Ġgenerates Ġa Ġtable Ġas Ġthe Ġoutput . ĠTables Ġin ĠWiki Bio Ġare Ġopen - domain , Ġwithout Ġany Ġfixed Ġschema Ġfor Ġwhich Ġattributes Ġmay Ġbe Ġpresent Ġor Ġabsent Ġin Ġan Ġinstance . ĠHence Ġwe Ġemploy Ġthe ĠPo inter - Gener ator ĠNetwork Ġ( PG - Net ) ĠB IB REF 19 Ġfor Ġthis Ġpurpose . ĠSpecifically , Ġwe Ġuse Ġa Ġsequence - to - sequence Ġmodel , Ġwhose Ġenc oder Ġand Ġdec oder Ġare Ġboth Ġsingle - layer Ġbi - direction al ĠL ST Ms . ĠThe Ġdec oder Ġis Ġaugmented Ġwith Ġan Ġattention Ġmechanism Ġover Ġthe Ġstates Ġof Ġthe Ġenc oder . ĠFurther , Ġit Ġalso Ġuses Ġa Ġcopy Ġmechanism Ġto Ġoptionally Ġcopy Ġtokens Ġdirectly Ġfrom Ġthe Ġsource Ġtext . ĠWe Ġdo Ġnot Ġuse Ġthe Ġcoverage Ġmechanism Ġof ĠB IB REF 19 Ġsince Ġthat Ġis Ġspecific Ġto Ġthe Ġtask Ġof Ġsummar ization Ġthey Ġstudy . ĠThe Ġdec oder Ġis Ġtrained Ġto Ġproduce Ġa Ġlinear ized Ġversion Ġof Ġthe Ġtable Ġwhere Ġthe Ġrows Ġand Ġcolumns Ġare Ġflattened Ġinto Ġa Ġsequence , Ġand Ġseparate Ġby Ġspecial Ġtokens . ĠFigure ĠFIG REF 53 Ġshows Ġan Ġexample . </s> Clearly , Ġsince Ġthe Ġreferences Ġare Ġdiver gent , Ġthe Ġmodel Ġcannot Ġbe Ġexpected Ġto Ġproduce Ġthe Ġentire Ġtable , Ġand Ġwe Ġsee Ġsome Ġfalse Ġinformation Ġbeing Ġhalluc inated Ġafter Ġtraining . ĠNevertheless , Ġas Ġwe Ġshow Ġin ĠÂ§ ĠSEC REF 36 Ġ, Ġthis Ġsystem Ġcan Ġbe Ġused Ġfor Ġevaluating Ġgenerated Ġtexts . ĠAfter Ġtraining , Ġwe Ġcan Ġparse Ġthe Ġoutput Ġsequence Ġalong Ġthe Ġspecial Ġtokens ĠIN LINE FORM 0 ĠR ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 ĠC ĠIN LINE FORM 3 Ġto Ġget Ġa Ġset Ġof Ġ( attribute , Ġvalue ) Ġpairs . ĠTable ĠT AB REF 54 Ġshows Ġthe Ġprecision , Ġrecall Ġand ĠF - score Ġof Ġthese Ġextracted Ġpairs Ġagainst Ġthe Ġground Ġtruth Ġtables , Ġwhere Ġthe Ġattributes Ġand Ġvalues Ġare Ġcompared Ġusing Ġan Ġexact Ġstring Ġmatch . </s> Hyper param eters </s> After Ġtuning Ġwe Ġfound Ġthe Ġsame Ġset Ġof Ġhyper param eters Ġto Ġwork Ġwell Ġfor Ġboth Ġthe Ġtable - to - text ĠPG - Net , Ġand Ġthe Ġinverse Ġinformation Ġextraction ĠPG - Net . ĠThe Ġhidden Ġstate Ġsize Ġof Ġthe Ġbi L ST Ms Ġwas Ġset Ġto Ġ200 . ĠThe Ġinput Ġand Ġoutput Ġvoc ab ular ies Ġwere Ġset Ġto Ġ5 0000 Ġmost Ġcommon Ġwords Ġin Ġthe Ġcorpus , Ġwith Ġadditional Ġspecial Ġsymbols Ġfor Ġtable Ġattribute Ġnames Ġ( such Ġas ĠâĢ ľ birth - date âĢ Ŀ ). ĠThe Ġembed d ings Ġof Ġthe Ġtokens Ġin Ġthe Ġvocabulary Ġwere Ġinitialized Ġwith ĠGl ove ĠB IB REF 43 Ġ. ĠLearning Ġrate Ġof ĠIN LINE FORM 0 Ġwas Ġused Ġduring Ġtraining , Ġwith Ġthe ĠAdam Ġoptim izer , Ġand Ġa Ġdrop out Ġof ĠIN LINE FORM 1 Ġwas Ġalso Ġapplied Ġto Ġthe Ġoutputs Ġof Ġthe Ġbi L ST M . ĠModels Ġwere Ġtrained Ġtill Ġthe Ġloss Ġon Ġthe Ġdev Ġset Ġstopped Ġdropping . ĠMaximum Ġlength Ġof Ġa Ġdec oded Ġtext Ġwas Ġset Ġto Ġ40 Ġtokens , Ġand Ġthat Ġof Ġthe Ġtables Ġwas Ġset Ġto Ġ120 Ġtokens . ĠVarious Ġbeam Ġsizes Ġand Ġlength Ġnormal ization Ġpenalties Ġwere Ġapplied Ġfor Ġthe Ġtable - to - text Ġsystem , Ġwhich Ġare Ġlisted Ġin Ġthe Ġmain Ġpaper . ĠFor Ġthe Ġinformation Ġextraction Ġsystem , Ġwe Ġfound Ġa Ġbeam Ġsize Ġof Ġ8 Ġand Ġno Ġlength Ġpenalty Ġto Ġproduce Ġthe Ġhighest ĠF - score Ġon Ġthe Ġdev Ġset . </s> Sample ĠOutput s </s> Table ĠT AB REF 55 Ġshows Ġsome Ġsample Ġreferences Ġand Ġthe Ġcorresponding Ġpredictions Ġfrom Ġthe Ġbest Ġperforming Ġmodel , ĠPG - Net Ġfor ĠWiki Bio .