BLEU , NIST , METEOR , ROUGE-L, CIDEr , evaluation script, automatic evaluation, human evaluation, minimum edit evaluation, word error rate (WER), factual errors and their types, fluency issues, acceptability of the output for production use in a news agency
EXTRACTIVE
BLEU, NIST, METEOR, ROUGE-L, CIDEr
EXTRACTIVE
In Table TABREF15 we measure BLEU BIBREF19, NIST BIBREF20, METEOR BIBREF21, ROUGE-L BIBREF22 and CIDEr BIBREF23 metrics on the 2018 E2E NLG Challenge test data using the evaluation script provided by the organizers. Our generation system is compared to the official shared task baseline system, TGen BIBREF24, as well as to the top performing participant system on each score (ST top). Our system outperforms the TGen baseline on 3 out of 5 metrics (BLEU, METEOR and ROUGE-L), which is on par with the official shared task results, where not a single one participant system was able to surpass the baseline on all five metrics. On two metrics, BLEU and METEOR, our system outperforms the best shared task participants. Our alignment serves as a gold standard reflecting which events the journalists have chosen to mention for each game. In our generation task, we are presented with the problem of selecting appropriate events from the full game statistics. We use the gold standard selection during training and validation of the text generation model, as well as the automatic evaluation. As we deploy our text generation model for manual evaluation, we use a Conditional Random Field (CRF) model to predict which events to mention. The second human evaluation aimed at judging the acceptability of the output for production use in a news agency. The output is evaluated in terms of its usability for a news channel labelled as being machine-generated, i.e. not aiming at the level of a human journalist equipped with substantial background information. The evaluation was carried out by two journalists from the STT agency, who split the 59 games among themselves approximately evenly. The first journalist edited the games to a form corresponding to a draft for subsequent minor post-editing by a human, simulating the use of the generated output as a product where the final customer is expected to do own post-editing before publication. The second journalist directly edited the news to a state ready for direct publication in a news stream labeled as machine-generated news. In addition to correcting factual errors, the journalists removed excessive repetition, improved text fluency, as well as occasionally included important facts which the system left ungenerated. The WER measured against the output considered ready for post-editing, is 9.9% (11.2% disregarding punctuation), only slightly worse than the evaluation with only the factual and grammatical errors corrected. The WER measured against the output considered ready for direct release, was 22.0% (24.4% disregarding punctuation). In other words, 75–90% of the generated text can be directly used, depending on the expected post-editing effort. In the minimum edit evaluation, carried out by the annotator who created the news corpus, only factual mistakes and grammatical errors are corrected, resulting in text which may remain awkward or unfluent. The word error rate (WER) of the generated text compared to its corrected variant as a reference is 5.6% (6.2% disregarding punctuation). The WER measure is defined as the number of insertions, substitutions, and deletions divided by the total length of the reference, in terms of tokens. The measure is the edit distance of the generated text and its corrected variant, directly reflecting the amount of effort needed to correct the generated output. The factual errors and their types are summarized in Table TABREF23. From the total of 510 game events generated by the system, 78 of these contained a factual error, i.e. 84.7% were generated without factual errors. Most fluency issues relate to the overall flow and structure of the report. Addressing these issues would require the model to take into account multiple events in a game, and combine the information more flexibly to avoid repetition. For instance, the output may repeatedly mention the period number for all goals in the same period. Likewise, this setup sometimes results in unnatural, yet grammatical, repetition of words across consecutive sentences. Even though the model has learned a selection of verbs meaning to score a goal, it is unable to ensure their varied use. While not successful in our initial experiments, generating text based on the multi-event alignments or at document level may eventually overcome these issues.
In Table TABREF15 we measure BLEU BIBREF19, NIST BIBREF20, METEOR BIBREF21, ROUGE-L BIBREF22 and CIDEr BIBREF23 metrics on the 2018 E2E NLG Challenge test data using the evaluation script provided by the organizers. Our generation system is compared to the official shared task baseline system, TGen BIBREF24, as well as to the top performing participant system on each score (ST top). Our system outperforms the TGen baseline on 3 out of 5 metrics (BLEU, METEOR and ROUGE-L), which is on par with the official shared task results, where not a single one participant system was able to surpass the baseline on all five metrics. On two metrics, BLEU and METEOR, our system outperforms the best shared task participants.
1910.01863
Introduction </s> Autom ated , Ġor Ġrobotic , Ġjournalism Ġaims Ġat Ġnews Ġgeneration Ġfrom Ġstructured Ġdata Ġsources , Ġeither Ġas Ġthe Ġfinal Ġproduct Ġor Ġas Ġa Ġdraft Ġfor Ġsubsequent Ġpost - ed iting . ĠAt Ġpresent , Ġautomated Ġjournalism Ġtypically Ġfocuses Ġon Ġdomains Ġsuch Ġas Ġsports , Ġfinance Ġand Ġsimilar Ġstatistics - based Ġreporting , Ġwhere Ġthere Ġis Ġa Ġcommercial Ġproduct Ġpotential Ġdue Ġto Ġthe Ġhigh Ġvolume Ġof Ġnews , Ġcombined Ġwith Ġthe Ġexpectation Ġof Ġa Ġrelatively Ġstraightforward Ġtask . </s> News Ġgeneration Ġsystems âĢĶ especially Ġthose Ġdeployed Ġin Ġpractice âĢĶ t end Ġto Ġbe Ġbased Ġon Ġintricate Ġtemplate Ġfilling , Ġaiming Ġto Ġgive Ġthe Ġusers Ġthe Ġfull Ġcontrol Ġof Ġthe Ġgenerated Ġfacts , Ġwhile Ġmaintaining Ġa Ġreasonable Ġvariability Ġof Ġthe Ġresulting Ġtext . ĠThis Ġcomes Ġat Ġthe Ġprice Ġof Ġhaving Ġto Ġdevelop Ġthe Ġtemplates Ġand Ġspecify Ġtheir Ġcontrol Ġlogic , Ġneither Ġof Ġwhich Ġare Ġtasks Ġnaturally Ġfitting Ġjournalists ' Ġwork . ĠFurther , Ġthis Ġdevelopment Ġneeds Ġto Ġbe Ġrepeated Ġfor Ġevery Ġdomain , Ġas Ġthe Ġtemplates Ġare Ġnot Ġeasily Ġtransferred Ġacross Ġdomains . ĠExamples Ġof Ġthe Ġtemplate - based Ġnews Ġgeneration Ġsystems Ġfor ĠFinnish Ġare ĠVo itto Ġby Ġthe ĠFinnish ĠPublic ĠService ĠBroadcasting ĠCompany Ġ( Y LE ) Ġused Ġfor Ġsports Ġnews Ġgeneration , Ġas Ġwell Ġas ĠVa al ib otti ĠB IB REF 0 , Ġa Ġhybrid Ġmachine Ġlearning Ġand Ġtemplate - based Ġsystem Ġused Ġfor Ġelection Ġnews . </s> BI B REF 1 Ġsuggested Ġa Ġneural Ġtemplate Ġgeneration , Ġwhich Ġjointly Ġmodels Ġlatent Ġtemplates Ġand Ġtext Ġgeneration . ĠSuch Ġa Ġsystem Ġincreases Ġinterpret ability Ġand Ġcontro ll ability Ġof Ġthe Ġgeneration , Ġhowever , Ġrecent Ġsequence - to - sequence Ġsystems Ġrepresent Ġthe Ġstate - of - the - art Ġin Ġdata - to - text Ġgeneration . ĠB IB REF 2 </s> In Ġthis Ġpaper , Ġwe Ġreport Ġon Ġthe Ġdevelopment Ġof Ġa Ġnews Ġgeneration Ġsystem Ġfor Ġthe ĠFinnish Ġice Ġhockey Ġnews Ġdomain , Ġbased Ġon Ġsequence - to - sequence Ġmethods . ĠIn Ġorder Ġto Ġtrain Ġsuch Ġa Ġsystem , Ġwe Ġcompile Ġa Ġcorpus Ġof Ġnews Ġbased Ġon Ġover Ġ2000 Ġgame Ġreports Ġfrom Ġthe ĠFinnish ĠNews ĠAgency ĠST T . ĠWhile Ġdeveloping Ġthis Ġcorpus Ġinto Ġa Ġform Ġsuitable Ġfor Ġtraining Ġof Ġend - to - end Ġsystems Ġnaturally Ġrequires Ġmanual Ġeffort , Ġwe Ġargue Ġthat Ġcompiling Ġand Ġrefining Ġa Ġset Ġof Ġtext Ġexamples Ġis Ġa Ġmore Ġnatural Ġway Ġfor Ġjournalists Ġto Ġinteract Ġwith Ġthe Ġsystem , Ġin Ġorder Ġfor Ġthem Ġto Ġcod ify Ġtheir Ġknowledge Ġand Ġto Ġadapt Ġit Ġfor Ġnew Ġdomains . </s> Our Ġaim Ġis Ġto Ġgenerate Ġreports Ġthat Ġgive Ġan Ġoverview Ġof Ġa Ġgame Ġbased Ġon Ġinformation Ġinf err able Ġfrom Ġthe Ġstatistics . ĠSuch Ġreports Ġcan Ġbe Ġused Ġeither Ġas Ġa Ġbasis Ġfor Ġfurther Ġpost - ed iting Ġby Ġa Ġjournalist Ġimprint ing Ġown Ġinsights Ġand Ġbackground Ġinformation , Ġor Ġeven Ġused Ġdirectly Ġas Ġa Ġnews Ġstream Ġlabelled Ġas Ġmachine - generated . </s> In Ġthe Ġfollowing , Ġwe Ġwill Ġintroduce Ġthe Ġnews Ġdataset Ġand Ġthe Ġprocess Ġof Ġits Ġcreation , Ġintroduce Ġan Ġend - to - end Ġmodel Ġfor Ġnews Ġgeneration , Ġand Ġevaluate Ġits Ġoutput Ġrespective Ġto Ġthe Ġabove mentioned Ġobjectives . </s> Ice ĠHockey ĠNews ĠDat as et </s> An Ġice Ġhockey Ġgame Ġis Ġrecorded Ġinto Ġstatistics Ġin Ġterms Ġof Ġdifferent Ġevents Ġoccurring Ġduring Ġplay , Ġsuch Ġas Ġgoals Ġand Ġpenalties . ĠIn Ġorder Ġto Ġtrain Ġa Ġmodel Ġto Ġgenerate Ġgame Ġreports , Ġwe Ġneed Ġaccess Ġto Ġthese Ġevents , Ġas Ġwell Ġas Ġexample Ġnews Ġarticles Ġabout Ġthe Ġgame . ĠOnly Ġrecently Ġhave Ġgame Ġstatistics Ġbecome Ġavailable Ġto Ġthe Ġpublic Ġthrough Ġa Ġweb Ġinterface Ġor ĠAPI , Ġwhereas Ġthe Ġinformation Ġhas Ġtraditionally Ġbeen Ġrecorded Ġas Ġstructured Ġtext Ġfiles . </s> The Ġnews Ġcorpus Ġfrom Ġthe ĠFinnish ĠNews ĠAgency ĠST T Ġincludes , Ġamong Ġall Ġother Ġnews , Ġarticles Ġcovering Ġice Ġhockey Ġgames Ġin Ġthe ĠFinnish Ġleagues Ġduring Ġthe Ġyears Ġ1994 âĢĵ 2018 . ĠIn Ġaddition Ġto Ġnews Ġarticles , Ġthe Ġcorpus Ġalso Ġincludes Ġthe Ġoriginal Ġgame Ġstatistics Ġtext Ġfiles . ĠThis Ġcreates Ġan Ġopportunity Ġto Ġalign Ġthe Ġgame Ġstatistics Ġwith Ġthe Ġcorresponding Ġnews Ġarticles , Ġproducing Ġa Ġdataset Ġof Ġover Ġ20 Ġyears Ġof Ġice Ġhockey Ġdata Ġwith Ġreference Ġnews Ġarticles Ġfor Ġthe Ġgames . ĠWhen Ġautomatically Ġpairing Ġthe Ġgame Ġstatistics Ġwith Ġnews Ġarticles Ġusing Ġdate Ġand Ġteam Ġnames Ġas Ġa Ġhe uristic , Ġwe Ġobtain Ġa Ġtotal Ġof Ġ3 , 454 Ġgames Ġwith Ġexisting Ġstatistics Ġand Ġat Ġleast Ġone Ġcorresponding Ġnews Ġarticle . </s> Ut il izing Ġreal Ġjournalistic Ġmaterial Ġposes Ġa Ġchallenge Ġin Ġthat Ġthe Ġarticles Ġmix Ġinformation Ġthat Ġcan Ġbe Ġfound Ġdirectly Ġin Ġthe Ġgame Ġstatistics Ġ( e . g ., Ġscores Ġand Ġnames ) Ġwith Ġinformation Ġinfer able Ġfrom Ġthe Ġstatistics Ġ( e . g ., Ġstatements Ġsuch Ġas Ġshortly Ġafter ), Ġinformation Ġbased Ġon Ġbackground Ġknowledge Ġ( e . g ., Ġa Ġteam 's Ġhome Ġcity Ġor Ġplayer 's Ġposition ), Ġgame Ġinsight Ġand Ġjudgement Ġbased Ġon Ġviewing Ġthe Ġgame Ġ( e . g ., Ġexpressions Ġsuch Ġas Ġslap shot Ġor Ġtipping Ġthe Ġpuck Ġdescribing Ġthe Ġcharacter Ġof Ġa Ġshot ), Ġand Ġeven Ġplayer Ġinterviews . </s> Therefore , Ġdirectly Ġusing Ġthe Ġlimited Ġamount Ġof Ġactual Ġnews Ġarticles Ġfor Ġend - to - end Ġsystem Ġtraining Ġbecomes Ġproblematic . ĠIn Ġour Ġinitial Ġexperiments Ġthe Ġgeneration Ġmodel Ġlearns Ġto ĠâĢ ľ hall uc inate âĢ Ŀ Ġfacts , Ġas Ġeasily Ġoccurs Ġwhen Ġthe Ġtarget Ġtext Ġis Ġtoo Ġloosely Ġrelated Ġto Ġthe Ġconditioning Ġinput . ĠIn Ġorder Ġto Ġensure Ġthat Ġthe Ġgeneration Ġmodel Ġis Ġable Ġto Ġlearn Ġto Ġgenerate Ġaccurate Ġdescriptions Ġfrom Ġgame Ġstatistics , Ġwe Ġclean Ġthe Ġnews Ġcorpus Ġby Ġmanually Ġalign ing Ġcorresponding Ġtext Ġspans Ġwith Ġgame Ġevents Ġdetailed Ġin Ġthe Ġstatistics . </s> For Ġthe Ġsake Ġof Ġcomparison , Ġlet Ġus Ġconsider Ġthe ĠRot ow ire Ġcorpus ĠB IB REF 3 Ġcontaining Ġbasketball Ġgame Ġsumm aries Ġand Ġstatistics , Ġwhich Ġwas Ġrecently Ġreleased Ġand Ġhas Ġbecome Ġa Ġpopular Ġdata Ġset Ġfor Ġtraining Ġdata - to - text Ġgeneration Ġsystems Ġ( cf ., Ġe . g ., ĠB IB REF 4 , ĠB IB REF 1 , ĠB IB REF 5 ). ĠThe ĠRot ow ire Ġgame Ġsumm aries Ġare Ġstraightforward Ġin Ġtheir Ġstyle Ġof Ġreporting , Ġfocusing Ġon Ġthe Ġgame Ġat Ġhand Ġand Ġtend Ġfor Ġthe Ġmost Ġpart Ġto Ġreference Ġfacts Ġin Ġthe Ġstatistics . ĠBy Ġcontrast , Ġour Ġnews Ġcorpus Ġis Ġmore Ġheter ogeneous , Ġincluding Ġboth Ġarticles Ġfocusing Ġon Ġthe Ġparticular Ġgame Ġand Ġarticles Ġthat Ġtake Ġa Ġbroader Ġperspective Ġ( e . g ., Ġdescribing Ġa Ġplayer 's Ġcareer ). ĠThe ĠST T Ġnews Ġarticles Ġtend Ġto Ġread Ġin Ġthe Ġjournalist 's Ġvoice , Ġputting Ġsubstantial Ġemphasis Ġon Ġthe Ġcharacter Ġof Ġthe Ġgame , Ġoften Ġin Ġcolorful Ġlanguage , Ġas Ġwell Ġas Ġquoting Ġplayers Ġand Ġcoaches . </s> An Ġexample Ġof Ġthe Ġevents Ġavailable Ġin Ġthe Ġgame Ġstatistics , Ġthe Ġactual Ġnews Ġarticle Ġon Ġthe Ġgame , Ġand Ġhow Ġthese Ġalign , Ġis Ġshown Ġin ĠFigure ĠFIG REF 5 . ĠText Ġspans Ġhighlighted Ġwith Ġblue Ġcolor Ġare Ġbased Ġon Ġinformation Ġavailable Ġin Ġthe Ġstatistics , Ġall Ġother Ġbeing Ġexternal Ġinformation . ĠIt Ġillustrates Ġthe Ġtypical Ġportion Ġof Ġa Ġraw Ġarticle Ġthat Ġis Ġnot Ġinf err able Ġfrom Ġthe Ġdata . ĠEnglish Ġtranslations Ġare Ġavailable Ġfor Ġa Ġcomparable Ġexample Ġin ĠFigure ĠFIG REF 27 . </s> Ice ĠHockey ĠNews ĠDat as et Ġ: :: ĠExt raction Ġof ĠGame ĠEvents </s> For Ġeach Ġevent Ġoccurring Ġin Ġa Ġgame Ġand Ġrecorded Ġin Ġstatistics , Ġwe Ġidentify Ġits Ġtype Ġand Ġassociated Ġfeatures . ĠThere Ġare Ġfour Ġevent Ġtypes : Ġend Ġresult , Ġgoal , Ġpenalty Ġand Ġsave . ĠAs Ġa Ġgeneral Ġrule , Ġfor Ġeach Ġgame Ġthe Ġend Ġresult Ġentry Ġspecifies Ġtotal Ġscores , Ġparticipating Ġteams Ġand Ġadditional Ġcircumstances Ġof Ġthe Ġgame Ġsuch Ġas Ġovertime Ġor Ġshootout . ĠThe Ġgoal Ġevent Ġis Ġthe Ġmost Ġfrequent Ġand Ġincludes Ġfeatures Ġsuch Ġas Ġgoal Ġscorer , Ġassists , Ġteam , Ġresulting Ġscore , Ġtime Ġand Ġthe Ġcurrent Ġstate Ġof Ġthe Ġgame Ġ( e . g ., Ġpower Ġplay , Ġpenalty Ġshot ). ĠWe Ġalso Ġderive Ġspecial Ġfeatures Ġthat Ġin Ġmany Ġcases Ġrequire Ġconsideration Ġof Ġother Ġevents Ġin Ġthe Ġcontext , Ġbut Ġper tain Ġto Ġone Ġparticular Ġevent , Ġe . g ., Ġis Ġthe Ġgoal Ġdeciding Ġor Ġfinal . ĠThe Ġpenalty Ġevent Ġspecifies Ġplayer , Ġteam , Ġtime Ġin Ġthe Ġgame Ġand Ġpenalty Ġtime . ĠThe Ġsave Ġevent Ġsummar ises Ġthe Ġnumber Ġof Ġsaves Ġof Ġa Ġgoaltender / team . </s> We Ġperform Ġinformation Ġextraction Ġwith Ġregular Ġexpressions Ġon Ġthe Ġstatistics Ġin Ġorder Ġto Ġstructure Ġthe Ġgame Ġinto Ġa Ġchronological Ġsequence Ġof Ġevents , Ġwhich Ġcan Ġthen Ġbe Ġaligned Ġwith Ġthe Ġcorresponding Ġtext Ġspans Ġfrom Ġthe Ġnews Ġarticle , Ġas Ġwell Ġas Ġused Ġas Ġinput Ġin Ġthe Ġtext Ġgeneration . ĠAn Ġinitial Ġsequence Ġof Ġevents Ġafter Ġinformation Ġextraction Ġis Ġshown Ġon Ġthe Ġleft Ġside Ġin ĠFigure ĠFIG REF 5 . ĠBefore Ġtext Ġgeneration , Ġthese Ġevents Ġare Ġyet Ġenhanced Ġwith Ġadditional Ġinformation Ġderived Ġfrom Ġthe Ġevent Ġitself , Ġor Ġthe Ġgame . </s> Ice ĠHockey ĠNews ĠDat as et Ġ: :: ĠManual ĠAl ignment </s> The Ġmanual Ġalignment Ġof Ġgame Ġevents Ġand Ġthe Ġarticle Ġtext Ġis Ġcarried Ġout Ġby Ġa Ġsingle Ġannot ator Ġin Ġapproximately Ġ6 Ġweeks Ġas Ġan Ġintegral Ġpart Ġof Ġthe Ġsystem Ġdevelopment Ġwork . ĠThe Ġannot ator Ġreceives Ġa Ġsequence Ġof Ġextracted Ġevents Ġand Ġalign s Ġthese Ġwith Ġthe Ġcorresponding Ġnews Ġarticle , Ġtypically Ġbut Ġnot Ġnecessarily Ġa Ġsingle Ġsentence Ġfor Ġevery Ġexpressed Ġevent . ĠAll Ġun ground ed Ġparts Ġwithin Ġthe Ġtext Ġare Ġremoved Ġand Ġif Ġnecessary , Ġthe Ġtext Ġis Ġmodified Ġfor Ġflu ency . ĠWe Ġcannot Ġalign Ġfull Ġsentences Ġwith Ġevents Ġas , Ġfor Ġinstance , ĠB IB REF 6 Ġdo , Ġas Ġoften Ġinformation Ġnot Ġgrounded Ġin Ġthe Ġstatistics Ġis Ġexpressed Ġtogether Ġwith Ġstatistics - based Ġfacts Ġwithin Ġthe Ġsame Ġsentences Ġand Ġclauses . </s> The Ġalignment Ġprocess Ġtherefore Ġfrequently Ġrequires Ġre ph r asing Ġthe Ġtext , Ġfor Ġinstance , Ġin Ġorder Ġto Ġmake Ġit Ġmore Ġneutral Ġand Ġavoid Ġgenerating Ġarbitrary Ġsubjective Ġjud gements . ĠWe Ġfind Ġthat Ġthe Ġnews Ġarticle Ġcommonly Ġincludes Ġdescription Ġthat Ġis Ġnot Ġevident Ġfrom Ġthe Ġdata Ġ( e . g ., Ġsubjective Ġcharacteristics Ġof Ġthe Ġplayer Ġor Ġthe Ġshot ), Ġand Ġoften Ġmay Ġreflect Ġthe Ġreporter 's Ġviewpoint . ĠFor Ġinstance , Ġthe Ġreporter Ġmay Ġevaluate Ġan Ġevent Ġnegatively Ġby Ġwriting ĠPlayer ĠA Ġdid Ġnot Ġmanage Ġto Ġscore Ġmore Ġthan Ġone Ġgoal , Ġreflecting Ġexpectation Ġof Ġthe Ġplayer 's Ġperformance , Ġwhich Ġwe Ġwould Ġchange Ġinto Ġa Ġmore Ġobjective Ġform Ġsuch Ġas ĠPlayer ĠA Ġscored Ġone Ġgoal . ĠSimilarly , Ġword Ġchoices Ġbased Ġon Ġviewing Ġthe Ġgame Ġ( e . g ., Ġslap shot ) Ġare Ġchanged Ġto Ġmore Ġneutral Ġcounterparts . </s> After Ġremoving Ġthe Ġunin fer able Ġparts Ġof Ġa Ġsentence , Ġthe Ġremainder Ġis Ġoften Ġrelatively Ġshort , Ġin Ġwhich Ġcase Ġwe Ġsometimes Ġopt Ġto Ġreplace Ġthe Ġinformation Ġthat Ġwas Ġremoved Ġwith Ġreferences Ġto Ġother Ġappropriate Ġinformation Ġfrom Ġthe Ġgame Ġstatistics , Ġsuch Ġas Ġthe Ġtime Ġof Ġthe Ġevent Ġor Ġthe Ġstate Ġof Ġthe Ġgame . ĠThis Ġserves Ġto Ġmaintain Ġa Ġsimilar Ġsentence Ġand Ġclause Ġstructure Ġas Ġthat Ġof Ġthe Ġoriginal Ġcorpus . </s> In Ġthe Ġmajority Ġof Ġour Ġaligned Ġdata Ġone Ġtext Ġspan Ġrefers Ġto Ġone Ġgame Ġevent . ĠHowever , Ġin Ġsome Ġcases Ġthe Ġsame Ġind iv isible Ġspan Ġcan Ġdescribe Ġmultiple Ġevents , Ġmost Ġcommonly Ġof Ġthe Ġsame Ġtype Ġ( e . g ., ĠPlayer ĠA Ġscored Ġtwo Ġgoals , Ġor ĠTeam ĠA Ġreceived Ġa Ġtotal Ġof Ġeight Ġpenalties ). ĠIn Ġsuch Ġcases , Ġwe Ġproduce Ġmulti - event Ġalign ments , Ġwhere Ġall Ġrelated Ġevents Ġare Ġaligned Ġwith Ġa Ġsingle Ġtext Ġspan . ĠMulti - event Ġalign ments Ġsupport Ġthe Ġdevelopment Ġof Ġmore Ġadvanced Ġgeneration , Ġwhich Ġstands Ġto Ġproduce Ġmore Ġnatural - sounding Ġreports Ġin Ġterms Ġof Ġless Ġrepetition Ġand Ġmore Ġflexible Ġstruct uring . </s> Ice ĠHockey ĠNews ĠDat as et Ġ: :: ĠCorpus ĠStatistics </s> In ĠTable ĠT AB REF 10 , Ġwe Ġsummarize Ġthe Ġoverall Ġsize Ġstatistics Ġof Ġthe Ġfinal Ġice Ġhockey Ġcorpus Ġafter Ġthe Ġstatistics Ġand Ġnews Ġarticles Ġhave Ġbeen Ġautomatically Ġpaired , Ġand Ġevents Ġhave Ġbeen Ġmanually Ġaligned Ġwith Ġthe Ġtext . ĠIn Ġtotal , Ġ2 , 307 Ġgames Ġwere Ġmanually Ġchecked Ġ( 66 . 8 % Ġof Ġthe Ġpaired Ġcorpus ), Ġof Ġwhich Ġ2 , 134 Ġgames Ġwere Ġcorrectly Ġpaired Ġwith Ġthe Ġarticle Ġdescribing Ġthe Ġgame . ĠIn Ġthese Ġgames , Ġ12 , 251 Ġout Ġof Ġ36 , 09 7 Ġevents Ġ( 33 . 9 %) Ġwere Ġreferenced Ġin Ġthe Ġtext Ġand Ġsuccessfully Ġaligned . ĠEnd Ġresult Ġoccurs Ġin Ġnearly Ġall Ġnews Ġarticles Ġas Ġthe Ġfirst Ġevent , Ġwhereas Ġthe Ġgoal Ġevent Ġis Ġby Ġfar Ġthe Ġmost Ġfrequent Ġone , Ġeach Ġgame Ġmentioning Ġon Ġaverage Ġ3 . 1 Ġgoals . ĠThe Ġnumber Ġof Ġaligned Ġevents Ġis Ġgreater Ġthan Ġthe Ġnumber Ġof Ġaligned Ġtext Ġspans Ġdue Ġto Ġmulti - event Ġalign ments . ĠWhile Ġ82 . 1 % Ġof Ġtext Ġspans Ġalign Ġwith Ġa Ġsingle Ġevent , Ġthere Ġis Ġa Ġlong Ġtail Ġof Ġmulti - event Ġalign ments , Ġwith Ġ11 . 2 % Ġalign ing Ġto Ġtwo Ġevents , Ġ3 . 4 % Ġto Ġthree , Ġ1 . 4 % Ġto Ġfour , Ġetc . </s> In ĠTable ĠT AB REF 11 Ġwe Ġmeasure Ġthe Ġlex ical Ġdiversity Ġof Ġoriginal Ġice Ġhockey Ġnews Ġarticles , Ġas Ġwell Ġas Ġthe Ġresulting Ġdataset Ġafter Ġmanual Ġalignment , Ġby Ġcomputing Ġthe ĠStandard ized ĠType âĢĵ Token ĠRatio Ġ( ST TR ). ĠThe Ġmeasure Ġis Ġdefined Ġas Ġthe Ġnumber Ġof Ġunique Ġtokens Ġor Ġle mm as Ġdivided Ġby Ġthe Ġtotal Ġnumber Ġof Ġtokens , Ġcalculated Ġon Ġevery Ġsegment Ġof Ġ1 , 000 Ġtokens Ġseparately Ġand Ġaveraged Ġacross Ġthe Ġcorpus . ĠST TR Ġis Ġmore Ġmeaningful Ġthan Ġthe Ġstandard Ġtype âĢĵ token Ġratio Ġwhen Ġcomparing Ġcorpor a Ġof Ġsubstantially Ġdifferent Ġsizes . ĠBoth Ġcorpor a Ġare Ġtoken ized Ġand Ġle mm at ized Ġusing Ġthe ĠTurk u ĠNeural ĠPars er Ġpipeline ĠB IB REF 7 , ĠB IB REF 8 . ĠST TR Ġof Ġthe Ġaligned Ġcorpus Ġis Ġlower Ġthan Ġin Ġthe Ġoriginal Ġhockey Ġnews Ġon Ġboth Ġword Ġand Ġle mma Ġlevel , Ġindicating Ġa Ġsomewhat âĢĶ but Ġnot Ġsubstantially Ġso âĢĶ more Ġrestricted Ġvocabulary Ġuse Ġin Ġour Ġaligned Ġsubset . </s> Event ĠSelection ĠModel </s> As Ġillustrated Ġpreviously , Ġany Ġgiven Ġnews Ġarticle Ġdescribes Ġonly Ġa Ġsubset Ġof Ġmost Ġnoteworthy Ġevents Ġof Ġa Ġgame . ĠWe Ġobserve Ġthat Ġmost Ġreports Ġare Ġconcise , Ġreferencing Ġon Ġaverage Ġ5 . 7 Ġevents . ĠThe Ġdistribution Ġof Ġevents / report Ġis : Ġ1 st Ġquant ile Ġat Ġ3 Ġevents Ġ( 20 . 9 % Ġof Ġevents Ġin Ġgame ), Ġ2 nd Ġat Ġ5 Ġ( 22 . 2 %), Ġ3 rd Ġat Ġ7 Ġ( 38 . 5 %), Ġ4 th Ġat Ġ36 Ġ( 100 %). </s> Our Ġalignment Ġserves Ġas Ġa Ġgold Ġstandard Ġreflecting Ġwhich Ġevents Ġthe Ġjournalists Ġhave Ġchosen Ġto Ġmention Ġfor Ġeach Ġgame . ĠIn Ġour Ġgeneration Ġtask , Ġwe Ġare Ġpresented Ġwith Ġthe Ġproblem Ġof Ġselecting Ġappropriate Ġevents Ġfrom Ġthe Ġfull Ġgame Ġstatistics . ĠWe Ġuse Ġthe Ġgold Ġstandard Ġselection Ġduring Ġtraining Ġand Ġvalidation Ġof Ġthe Ġtext Ġgeneration Ġmodel , Ġas Ġwell Ġas Ġthe Ġautomatic Ġevaluation . ĠAs Ġwe Ġdeploy Ġour Ġtext Ġgeneration Ġmodel Ġfor Ġmanual Ġevaluation , Ġwe Ġuse Ġa ĠCond itional ĠRandom ĠField Ġ( CR F ) Ġmodel Ġto Ġpredict Ġwhich Ġevents Ġto Ġmention . </s> C asting Ġthe Ġevent Ġselection Ġproblem Ġas Ġa Ġsequence Ġlabeling Ġtask , Ġthe ĠCR F Ġmodel Ġtakes Ġas Ġinput Ġthe Ġfull Ġsequence Ġof Ġevents Ġin Ġone Ġgame Ġtogether Ġwith Ġassociated Ġfeatures Ġfor Ġeach Ġevent , Ġand Ġpredicts Ġa Ġbinary Ġlabel Ġfor Ġeach Ġevent . ĠWe Ġachieve Ġan Ġoverall ĠF - score Ġof Ġ67 . 1 % Ġon Ġthe Ġtest Ġset , Ġwhich Ġbroken Ġdown Ġby Ġevent Ġtype Ġis : Ġend Ġresult Ġ( 98 . 0 %), Ġgoal Ġ( 70 . 2 %), Ġpenalty Ġ( 20 . 1 %), Ġsave Ġ( 47 . 7 %). ĠPen alties Ġare Ġthe Ġmost Ġdifficult Ġto Ġpredict , Ġbeing Ġreported Ġonly Ġ7 . 8 % Ġof Ġthe Ġtime Ġin Ġreality , Ġe . g ., Ġcompared Ġto Ġ54 . 1 % Ġfor Ġgoals . </s> Text ĠGeneration </s> Next , Ġwe Ġpresent Ġthe Ġmodel Ġarchitecture Ġused Ġin Ġtext Ġgeneration , Ġand Ġevaluate Ġthe Ġmodel Ġon Ġa Ġpopular Ġbaseline Ġdataset . ĠAfter Ġthat , Ġwe Ġdescribe Ġthe Ġtraining Ġof Ġthe Ġgeneration Ġmodel Ġon Ġour Ġice Ġhockey Ġcorpus Ġand Ġuse Ġautomatic Ġevaluation Ġmetrics Ġto Ġcompare Ġagainst Ġexisting Ġreferences . </s> Text ĠGeneration Ġ: :: ĠModel ĠArchitecture </s> We Ġuse Ġa Ġpointer - generation Ġnetwork ĠB IB REF 10 , ĠB IB REF 11 , ĠB IB REF 12 , Ġwhere Ġthe Ġneural Ġattention Ġmechanism Ġin Ġthe Ġenc oder - dec oder Ġmodel Ġis Ġadapted Ġto Ġjointly Ġmodel Ġa Ġprobability Ġdistribution Ġover Ġwords Ġfrom Ġthe Ġknown Ġvocabulary , Ġa Ġdistribution Ġover Ġwords Ġfrom Ġthe Ġinput Ġsequence Ġto Ġcopy Ġand Ġa Ġprobability Ġthat Ġcontrols Ġthe Ġcopying Ġmechanism . ĠA Ġseparate Ġcoverage Ġattention Ġvector , Ġa Ġsum Ġof Ġpast Ġattention Ġdistributions , Ġis Ġmaintained Ġto Ġinform Ġthe Ġmodel Ġof Ġits Ġpast Ġattention Ġdecisions . ĠSuch Ġa Ġcoverage Ġmodel Ġis Ġshown Ġto Ġprevent Ġtext Ġrepetition Ġin Ġgenerated Ġoutput ĠB IB REF 13 , ĠB IB REF 12 . </s> The Ġmodel Ġis Ġimplemented Ġusing Ġthe ĠOpen N MT - py Ġlibrary ĠB IB REF 14 . ĠThe Ġenc oder Ġhas Ġtwo Ġbid irection al ĠL ST M Ġlayers Ġwith Ġ500 Ġhidden Ġunits , Ġtogether Ġwith Ġ500 - dimensional Ġword Ġembed d ings . ĠThe Ġdec oder Ġhas Ġtwo Ġun id irection al ĠL ST M Ġlayers Ġwith Ġ500 Ġhidden Ġunits . ĠBoth Ġenc oder Ġand Ġdec oder Ġapply Ġa Ġdrop out Ġof Ġ0 . 3 Ġbetween ĠL ST M Ġlayers . </s> Text ĠGeneration Ġ: :: ĠBas eline ĠExper iments Ġon Ġthe ĠE 2 E ĠDat as et </s> To Ġdemonstrate Ġthe Ġperformance Ġof Ġour Ġgeneration Ġmodel Ġarchitecture , Ġwe Ġreport Ġresults Ġon Ġa Ġknown Ġdataset Ġwith Ġpublished Ġbas elines , Ġnamely Ġthe ĠE 2 E ĠNL G ĠChallenge ĠB IB REF 2 Ġon Ġend - to - end Ġnatural Ġlanguage Ġgeneration Ġin Ġspoken Ġdialogue Ġsystems . ĠThe Ġtask Ġis Ġto Ġproduce Ġa Ġnatural Ġlanguage Ġdescription Ġof Ġa Ġrestaurant Ġbased Ġon Ġa Ġgiven Ġmeaning Ġrepresentation Ġ( MR )âĢĶ an Ġun ordered Ġset Ġof Ġattributes Ġand Ġtheir Ġvalues . ĠThe Ġattributes Ġincluded , Ġamong Ġothers , Ġthe Ġrestaurant Ġname , Ġarea , Ġfood Ġtype Ġand Ġrating . ĠWe Ġrepresent Ġthe Ġgiven ĠMR Ġas Ġa Ġsequence Ġof Ġtokens Ġwhere Ġeach Ġattribute Ġvalue Ġis Ġembedded Ġinto ĠXML - style Ġbeginning Ġand Ġend Ġattribute Ġmarkers , Ġand Ġthe Ġorder Ġof Ġattributes Ġis Ġkept Ġfixed Ġacross Ġthe Ġwhole Ġdataset . ĠThe Ġtarget Ġoutput Ġis Ġa Ġsequence Ġof Ġtokens . ĠWe Ġdo Ġnot Ġapply Ġany Ġexplicit Ġde lex ical ization Ġsteps . </s> In ĠTable ĠT AB REF 15 Ġwe Ġmeasure ĠB LE U ĠB IB REF 19 , ĠN IST ĠB IB REF 20 , ĠMET E OR ĠB IB REF 21 , ĠR OU GE - L ĠB IB REF 22 Ġand ĠC ID Er ĠB IB REF 23 Ġmetrics Ġon Ġthe Ġ2018 ĠE 2 E ĠNL G ĠChallenge Ġtest Ġdata Ġusing Ġthe Ġevaluation Ġscript Ġprovided Ġby Ġthe Ġorganizers . ĠOur Ġgeneration Ġsystem Ġis Ġcompared Ġto Ġthe Ġofficial Ġshared Ġtask Ġbaseline Ġsystem , ĠT Gen ĠB IB REF 24 , Ġas Ġwell Ġas Ġto Ġthe Ġtop Ġperforming Ġparticipant Ġsystem Ġon Ġeach Ġscore Ġ( ST Ġtop ). ĠOur Ġsystem Ġoutper forms Ġthe ĠT Gen Ġbaseline Ġon Ġ3 Ġout Ġof Ġ5 Ġmetrics Ġ( BLE U , ĠMET E OR Ġand ĠR OU GE - L ), Ġwhich Ġis Ġon Ġpar Ġwith Ġthe Ġofficial Ġshared Ġtask Ġresults , Ġwhere Ġnot Ġa Ġsingle Ġone Ġparticipant Ġsystem Ġwas Ġable Ġto Ġsurpass Ġthe Ġbaseline Ġon Ġall Ġfive Ġmetrics . ĠOn Ġtwo Ġmetrics , ĠB LE U Ġand ĠMET E OR , Ġour Ġsystem Ġoutper forms Ġthe Ġbest Ġshared Ġtask Ġparticipants . </s> E 2 E ĠNL G ĠChallenge Ġevaluation Ġis Ġbased Ġon Ġhaving Ġmultiple Ġreferences Ġfor Ġeach ĠMR , Ġon Ġaverage Ġeach Ġunique ĠMR Ġin Ġthe Ġcorpus Ġhaving Ġ8 Ġreference Ġdescriptions . ĠIn Ġthe Ġevaluation , Ġthe Ġoutput Ġfor Ġeach Ġunique ĠMR Ġis Ġcompared Ġagainst Ġall Ġreferences Ġand Ġthe Ġmaximum Ġscore Ġis Ġused , Ġnaturally Ġleading Ġto Ġhigher Ġscores . ĠTo Ġhave Ġmore Ġcomparable Ġnumbers Ġto Ġour Ġice Ġhockey Ġcorpus , Ġwhere Ġwe Ġhave Ġonly Ġone Ġreference Ġfor Ġeach Ġinput Ġevent , Ġwe Ġalso Ġinclude Ġscores Ġobtained Ġby Ġcomparing Ġeach ĠMR Ġto Ġeach Ġof Ġits Ġreference Ġdescriptions Ġseparately Ġas Ġif Ġthey Ġwere Ġindividual Ġdata Ġpoints Ġ( O urs Ġsingle Ġref .). </s> Text ĠGeneration Ġ: :: ĠHockey ĠData ĠRepresent ation </s> Currently , Ġwe Ġconcentrate Ġon Ġtraining Ġthe Ġmodel Ġonly Ġwith Ġtext Ġspans Ġalign ing Ġwith Ġsingle Ġevents , Ġexcluding Ġthe Ġless Ġfrequent Ġmulti - event Ġalign ments . ĠFurthermore , Ġwe Ġare Ġconsidering Ġeach Ġevent Ġas Ġa Ġseparate Ġtraining Ġexample , Ġindependent Ġof Ġother Ġevents Ġin Ġthe Ġgame . </s> Given Ġa Ġsingle Ġevent Ġdescribed Ġas Ġa Ġsequence Ġof Ġfeatures Ġand Ġtheir Ġvalues , Ġour Ġtext Ġgeneration Ġmodel Ġis Ġtrained Ġto Ġproduce Ġthe Ġtext Ġspan Ġaligned Ġwith Ġit . ĠFollowing Ġthe Ġdata Ġrepresentation Ġused Ġin ĠE 2 E ĠNL G ĠChallenge Ġexperiments , Ġthe Ġinput Ġevents Ġare Ġrepresented Ġas Ġa Ġlinear ized Ġsequence Ġof Ġtokens , Ġwhere ĠXML - style Ġbeginning Ġand Ġend Ġtags Ġare Ġused Ġto Ġseparate Ġthe Ġdifferent Ġfeatures Ġ( see ĠFigure ĠFIG REF 18 ). ĠThis Ġallows Ġthe Ġmodel Ġto Ġdirectly Ġcopy Ġsome Ġof Ġthe Ġinput Ġtokens Ġto Ġthe Ġoutput Ġwhen Ġnecessary . ĠThe Ġability Ġto Ġcopy Ġtokens Ġis Ġespecially Ġimportant Ġwith Ġplayer Ġnames Ġand Ġexact Ġtimes , Ġwhere Ġthe Ġvocabulary Ġis Ġsparse , Ġand Ġmany Ġof Ġthese Ġcan Ġeven Ġbe Ġpreviously Ġunseen , Ġunknown Ġtokens . ĠIn Ġaddition , Ġwe Ġalso Ġinclude Ġfeatures Ġthat Ġare Ġmeant Ġto Ġinform Ġgeneration Ġwithout Ġbeing Ġcopied Ġthemselves , Ġfor Ġexample , Ġthe Ġtype Ġof Ġthe Ġevent . ĠThe Ġtarget Ġof Ġgeneration Ġis Ġa Ġtoken ized Ġsequence Ġof Ġwords , Ġwhere Ġalso Ġd ashes Ġinside Ġscores Ġare Ġseparated , Ġallowing Ġthe Ġmodel Ġto Ġswap Ġscores Ġwhen Ġnecessary . ĠThe Ġreference Ġtext Ġsometimes Ġflips Ġthe Ġorder Ġof Ġthe Ġteams , Ġrequiring Ġthe Ġscore Ġto Ġbe Ġinverted Ġas Ġwell Ġ( Team ĠA âĢĵ Team ĠB Ġ0 âĢĵ 1 Ġinto ĠTeam ĠB Ġwon ĠTeam ĠA Ġ1 âĢĵ 0 ). </s> One Ġparticular Ġchallenge Ġin Ġour Ġcorpus Ġis Ġthat Ġthe Ġdecision Ġof Ġwhich Ġaspects Ġto Ġfocus Ġon , Ġi . e ., Ġwhich Ġparticular Ġfeatures Ġfrom Ġthe Ġsource Ġevent Ġto Ġverbal ize , Ġis Ġrelatively Ġarbitrary . ĠFor Ġexample , Ġsometimes Ġa Ġjournalist Ġmentions Ġthe Ġplayer Ġor Ġplayers Ġassisting Ġa Ġgoal , Ġbut Ġin Ġmany Ġcases Ġit Ġis Ġleft Ġout . ĠBoth Ġoptions Ġare Ġequally Ġcorrect , Ġbut Ġoverlap - based Ġmetrics Ġsuch Ġas ĠB LE U Ġpenal ize Ġsuch Ġcreative Ġvariation . ĠBy Ġcontrast , Ġin Ġother Ġtext Ġgeneration Ġdatasets Ġsuch Ġas Ġthe ĠE 2 E ĠNL G ĠChallenge , Ġthe Ġoutput Ġtext Ġin Ġgeneral Ġdescribes Ġall Ġinput Ġfeatures . ĠTo Ġaccount Ġfor Ġthis Ġvariation , Ġwe Ġinclude Ġa Ġlength Ġfeature Ġto Ġact Ġas Ġa Ġminimal Ġsupervision Ġsignal Ġfor Ġthe Ġmodel Ġto Ġrely Ġon . ĠWe Ġdivide Ġoutput Ġtext Ġlengths Ġinto Ġthree Ġevenly Ġsized Ġcategories Ġ( short , Ġmedium Ġand Ġlong ) Ġto Ġprovide Ġa Ġhint Ġto Ġthe Ġmodel Ġduring Ġtraining Ġof Ġhow Ġlong Ġand Ġdetailed Ġoutput Ġit Ġis Ġexpected Ġto Ġgenerate Ġfor Ġeach Ġtraining Ġexample . ĠAt Ġtest Ġtime , Ġwe Ġthen Ġhave Ġthe Ġpossibility Ġto Ġcontrol Ġthe Ġdesired Ġapproximate Ġlength Ġof Ġthe Ġgenerated Ġtext . ĠIn Ġthe Ġexperiments Ġthroughout Ġthis Ġpaper , Ġwe Ġgenerate Ġall Ġthree Ġlength Ġvariants Ġfor Ġeach Ġevent Ġand Ġpick Ġthe Ġone Ġwith Ġthe Ġhighest Ġaverage Ġconfidence Ġof Ġthe Ġgeneration Ġmodel . </s> Text ĠGeneration Ġ: :: ĠTraining Ġand ĠOptim ization </s> The Ġmodel Ġis Ġtrained Ġusing Ġthe ĠAdam Ġoptim izer Ġwith Ġlearning Ġrate Ġof Ġ0 . 000 5 Ġand Ġbatch Ġsize Ġof Ġ32 . ĠThe Ġmodel Ġis Ġtrained Ġfor Ġa Ġmaximum Ġof Ġ8000 Ġsteps Ġ( ca . Ġ40 Ġepoch s ), Ġand Ġthe Ġfinal Ġmodel Ġis Ġchosen Ġbased Ġon Ġvalidation Ġset Ġperformance . ĠWe Ġuse Ġ80 % Ġof Ġthe Ġaligned Ġgames Ġfor Ġtraining , Ġ10 % Ġfor Ġvalidation Ġand Ġ10 % Ġfor Ġtesting . </s> In Ġour Ġinitial Ġexperiments , Ġwe Ġused Ġthe ĠRB FO pt Ġlibrary ĠB IB REF 25 Ġfor Ġhyper param eter Ġtuning , Ġmaximizing Ġvalidation Ġset ĠB LE U Ġscore . ĠHowever , Ġwhen Ġmanually Ġinspecting Ġgeneration Ġresults Ġon Ġthe Ġvalidation Ġset , Ġwe Ġnoticed Ġthat Ġmodels Ġwith Ġa Ġhigher ĠB LE U Ġscore Ġresult Ġin Ġmore Ġfluent Ġtext Ġbut Ġgenerate Ġmore Ġfactual Ġmistakes . ĠThis Ġobservation Ġis Ġsupported Ġby ĠB IB REF 3 , Ġwho Ġnote Ġthat ĠB LE U Ġscore Ġtends Ġto Ġreward Ġfluent Ġtext Ġrather Ġthan Ġother Ġaspects Ġdesirable Ġin Ġgeneration Ġof Ġsports Ġsumm aries Ġfrom Ġdata . ĠFor Ġthis Ġreason , Ġwe Ġultimately Ġdecided Ġto Ġuse Ġhyper param eters Ġmanually Ġtuned Ġon Ġthe Ġvalidation Ġset Ġto Ġgive Ġa Ġgood Ġperceived Ġbalance Ġbetween Ġfluent Ġand Ġfact ually Ġcorrect Ġtext . </s> Text ĠGeneration Ġ: :: ĠAutomatic ĠEvaluation Ġof ĠHockey ĠGeneration </s> In ĠTable ĠT AB REF 21 , Ġwe Ġprovide Ġevaluation Ġresults Ġusing Ġthe Ġfive Ġaforementioned Ġmetrics . ĠWe Ġevaluate Ġon Ġevent Ġlevel Ġusing Ġgold Ġstandard Ġevent Ġselection , Ġwhere Ġeach Ġgenerated Ġevent Ġdescription Ġis Ġcompared Ġto Ġits Ġexisting Ġreference Ġtext . ĠAs Ġthe Ġmodel Ġis Ġtrained Ġto Ġproduce Ġa Ġtoken ized Ġsequence , Ġwe Ġapply Ġa Ġdet oken izer Ġto Ġbe Ġable Ġto Ġcompare Ġagainst Ġthe Ġoriginal Ġunt oken ized Ġreference Ġtext . ĠOn Ġthe Ġtest Ġset , Ġthe Ġmodel Ġachieves Ġa ĠB LE U Ġscore Ġof Ġ19 . 67 . ĠTo Ġthe Ġextent Ġthat Ġdifferent Ġdatasets Ġallow Ġcomparison , Ġthe Ġbest Ġreported Ġscore Ġon Ġthe ĠRot ow ire Ġbasketball Ġnews Ġcorpus Ġis Ġ16 . 50 ĠB IB REF 5 . ĠCompared Ġto Ġour Ġearlier ĠE 2 E Ġbaseline Ġexperiment , Ġwe Ġscore Ġlower Ġthan Ġour Ġclosest Ġcomparable Ġreference Ġof Ġ31 . 90 Ġ( with Ġsingle Ġreferences ), Ġwhich Ġis Ġunderstandable Ġdue Ġto Ġthe Ġmuch Ġsmaller Ġtrain Ġset Ġsize Ġfor Ġthe Ġhockey Ġcorpus Ġ( about Ġ13 % Ġin Ġsize ). </s> In ĠFigure ĠFIG REF 22 , Ġwe Ġplot Ġthe Ġlearning Ġcurve Ġwith Ġincreasing Ġsizes Ġof Ġtraining Ġdata Ġin Ġorder Ġto Ġillustrate Ġhow Ġgeneration Ġperformance Ġbenefits Ġfrom Ġmore Ġdata . ĠThe Ġlearning Ġcurve Ġis Ġstill Ġsteadily Ġincreasing Ġwhen Ġusing Ġ100 % Ġof Ġthe Ġtraining Ġdata Ġcurrently Ġavailable , Ġwhich Ġindicates Ġthat Ġmore Ġdata Ġwould Ġmost Ġlikely Ġfurther Ġimprove Ġthe Ġperformance . </s> Human Ġevaluation </s> As Ġour Ġobjective Ġis Ġpractically Ġusable Ġnews Ġgeneration , Ġwe Ġcarry Ġout Ġa Ġmanual Ġevaluation Ġof Ġits Ġoutput Ġon Ġ59 Ġrandomly Ġselected Ġgames Ġfrom Ġthe Ġtest Ġset , Ġfocusing Ġin Ġparticular Ġon Ġthe Ġcorrections Ġthat Ġwould Ġbe Ġnecessary Ġto Ġobtain Ġacceptable Ġoutput . ĠExample Ġcorrections Ġare Ġshown Ġin ĠFigure ĠFIG REF 27 . ĠThese Ġfull Ġgame Ġreports Ġare Ġgenerated Ġby Ġfirst Ġapplying Ġthe Ġselection Ġmodel Ġdescribed Ġin ĠSection ĠSEC REF 3 Ġto Ġselect Ġthe Ġevents Ġto Ġbe Ġincluded Ġin Ġthe Ġreport , Ġand Ġthen Ġusing Ġthe Ġtext Ġgeneration Ġmodel Ġto Ġverbal ize Ġeach Ġselected Ġevent . ĠThe Ġgenerated Ġtexts Ġof Ġthe Ġevents Ġare Ġthen Ġdet oken ized Ġand Ġconc aten ated Ġin Ġchronological Ġorder Ġinto Ġa Ġfull Ġgame Ġreport . </s> Human Ġevaluation Ġ: :: ĠMinimum ĠEdit ĠEvaluation </s> In Ġthe Ġminimum Ġedit Ġevaluation , Ġcarried Ġout Ġby Ġthe Ġannot ator Ġwho Ġcreated Ġthe Ġnews Ġcorpus , Ġonly Ġfactual Ġmistakes Ġand Ġgram matical Ġerrors Ġare Ġcorrected , Ġresulting Ġin Ġtext Ġwhich Ġmay Ġremain Ġawkward Ġor Ġunf luent . ĠThe Ġword Ġerror Ġrate Ġ( WER ) Ġof Ġthe Ġgenerated Ġtext Ġcompared Ġto Ġits Ġcorrected Ġvariant Ġas Ġa Ġreference Ġis Ġ5 . 6 % Ġ( 6 . 2 % Ġdisreg arding Ġpunct uation ). ĠThe ĠW ER Ġmeasure Ġis Ġdefined Ġas Ġthe Ġnumber Ġof Ġinsert ions , Ġsubstit utions , Ġand Ġdelet ions Ġdivided Ġby Ġthe Ġtotal Ġlength Ġof Ġthe Ġreference , Ġin Ġterms Ġof Ġtokens . ĠThe Ġmeasure Ġis Ġthe Ġedit Ġdistance Ġof Ġthe Ġgenerated Ġtext Ġand Ġits Ġcorrected Ġvariant , Ġdirectly Ġreflecting Ġthe Ġamount Ġof Ġeffort Ġneeded Ġto Ġcorrect Ġthe Ġgenerated Ġoutput . </s> Human Ġevaluation Ġ: :: ĠMinimum ĠEdit ĠEvaluation Ġ: :: ĠFact ual ĠCorrect ness </s> The Ġfactual Ġerrors Ġand Ġtheir Ġtypes Ġare Ġsummarized Ġin ĠTable ĠT AB REF 23 . ĠFrom Ġthe Ġtotal Ġof Ġ510 Ġgame Ġevents Ġgenerated Ġby Ġthe Ġsystem , Ġ78 Ġof Ġthese Ġcontained Ġa Ġfactual Ġerror , Ġi . e . Ġ84 . 7 % Ġwere Ġgenerated Ġwithout Ġfactual Ġerrors . </s> The Ġmost Ġnotable Ġerrors Ġinvolved Ġplayer Ġor Ġteam Ġnames . ĠThe Ġinput Ġfor Ġa Ġsingle Ġgame Ġevent Ġmay Ġcontain Ġmore Ġthan Ġone Ġname Ġ( e . g . Ġthe Ġgoal Ġscorer Ġand Ġup Ġto Ġtwo Ġassisting Ġplayers ). ĠIn Ġthese Ġcases , Ġthe Ġmodel Ġoccasionally Ġpaired Ġfirst Ġname Ġand Ġsurname Ġincorrectly . ĠLess Ġfrequent Ġerrors Ġinclude Ġwrong Ġteam Ġpair ings Ġwhen Ġreferring Ġto Ġthe Ġend Ġresult Ġof Ġthe Ġgame , Ġe . g ., ĠTeam ĠA Ġlost Ġto ĠTeam ĠA . </s> In Ġsentences Ġwhere Ġthe Ġexact Ġtime Ġof Ġa Ġgame Ġevent Ġis Ġgenerated Ġas Ġa Ġnumber Ġ( at Ġtime Ġ39 . 54 ) Ġthe Ġmodel Ġcopied Ġthe Ġtime Ġreliably , Ġbut Ġwhen Ġthe Ġapproximate Ġtime Ġof Ġan Ġevent Ġis Ġverbal ized Ġ( at Ġthe Ġend Ġof Ġthe Ġsecond Ġperiod ) Ġthere Ġwere Ġoccasional Ġerrors . ĠAnother Ġnotable Ġerror Ġcategory Ġis Ġtypes Ġof Ġgoals Ġor Ġtheir Ġscores Ġ( e . g . Ġ3 âĢĵ 0 , Ġdeciding Ġgoal , Ġtying Ġgoal ). ĠIn Ġthis Ġcategory , Ġthe Ġerror Ġsimilarly Ġoccurred Ġin Ġcases Ġwhen Ġthe Ġreference Ġis Ġverbal Ġ( third Ġgoal ), Ġbut Ġoccasionally Ġalso Ġin Ġnumerical Ġrepresentation Ġ( 3 âĢĵ 0 Ġinstead Ġof Ġ0 âĢĵ 3 ). ĠThe Ġother , Ġless Ġcommon Ġcategories Ġrelate Ġto Ġgame Ġtotal Ġscores , Ġpower Ġplay , Ġassists Ġand Ġpenalties . </s> Human Ġevaluation Ġ: :: ĠMinimum ĠEdit ĠEvaluation Ġ: :: ĠFlu ency </s> Overall , Ġthe Ġgenerated Ġtext Ġis Ġhighly Ġgram matical . ĠThe Ġmost Ġfrequent Ġgram matical Ġerror Ġin Ġthe Ġoutput Ġis Ġuncon jug ated Ġplayer Ġnames ; Ġcommonly Ġnames Ġin Ġtheir ĠFinnish Ġnom in ative Ġcase Ġthat Ġshould Ġbe Ġin Ġgen itive . ĠIn Ġa Ġfew Ġcases , Ġthe Ġmodel Ġmakes Ġgram matical Ġerrors Ġwhen Ġit Ġcopies Ġan Ġincompatible Ġword Ġfrom Ġthe Ġinput , Ġe . g ., Ġa Ġname Ġinstead Ġof Ġa Ġtime Ġreference . </s> Both Ġerror Ġtypes Ġcommonly Ġoccur Ġwhen Ġthe Ġmodel Ġcopies Ġfrom Ġthe Ġinput . ĠAs Ġit Ġoperates Ġon Ġthe Ġtoken Ġrather Ġthan Ġsub - word Ġlevel , Ġit Ġis Ġchallenging Ġto Ġmap Ġbetween Ġword Ġforms Ġand Ġinf lect Ġinf requent Ġwords Ġsuch Ġas Ġplayer Ġnames . ĠThe Ġlatter Ġerror Ġlikely Ġoccurs Ġwhen Ġthe Ġcopy Ġattention Ġis Ġindec isive Ġand Ġfails Ġto Ġrecognize Ġthe Ġtype Ġfor Ġinf requent Ġtokens . </s> Most Ġflu ency Ġissues Ġrelate Ġto Ġthe Ġoverall Ġflow Ġand Ġstructure Ġof Ġthe Ġreport . ĠAdd ressing Ġthese Ġissues Ġwould Ġrequire Ġthe Ġmodel Ġto Ġtake Ġinto Ġaccount Ġmultiple Ġevents Ġin Ġa Ġgame , Ġand Ġcombine Ġthe Ġinformation Ġmore Ġflex ibly Ġto Ġavoid Ġrepetition . ĠFor Ġinstance , Ġthe Ġoutput Ġmay Ġrepeatedly Ġmention Ġthe Ġperiod Ġnumber Ġfor Ġall Ġgoals Ġin Ġthe Ġsame Ġperiod . ĠLikewise , Ġthis Ġsetup Ġsometimes Ġresults Ġin Ġunnatural , Ġyet Ġgram matical , Ġrepetition Ġof Ġwords Ġacross Ġconsecutive Ġsentences . ĠEven Ġthough Ġthe Ġmodel Ġhas Ġlearned Ġa Ġselection Ġof Ġverbs Ġmeaning Ġto Ġscore Ġa Ġgoal , Ġit Ġis Ġunable Ġto Ġensure Ġtheir Ġvaried Ġuse . ĠWhile Ġnot Ġsuccessful Ġin Ġour Ġinitial Ġexperiments , Ġgenerating Ġtext Ġbased Ġon Ġthe Ġmulti - event Ġalign ments Ġor Ġat Ġdocument Ġlevel Ġmay Ġeventually Ġovercome Ġthese Ġissues . </s> Human Ġevaluation Ġ: :: ĠProduct - Read iness ĠEvaluation </s> The Ġsecond Ġhuman Ġevaluation Ġaimed Ġat Ġjudging Ġthe Ġaccept ability Ġof Ġthe Ġoutput Ġfor Ġproduction Ġuse Ġin Ġa Ġnews Ġagency . ĠThe Ġoutput Ġis Ġevaluated Ġin Ġterms Ġof Ġits Ġusability Ġfor Ġa Ġnews Ġchannel Ġlabelled Ġas Ġbeing Ġmachine - generated , Ġi . e . Ġnot Ġaiming Ġat Ġthe Ġlevel Ġof Ġa Ġhuman Ġjournalist Ġequipped Ġwith Ġsubstantial Ġbackground Ġinformation . ĠThe Ġevaluation Ġwas Ġcarried Ġout Ġby Ġtwo Ġjournalists Ġfrom Ġthe ĠST T Ġagency , Ġwho Ġsplit Ġthe Ġ59 Ġgames Ġamong Ġthemselves Ġapproximately Ġevenly . ĠThe Ġfirst Ġjournalist Ġedited Ġthe Ġgames Ġto Ġa Ġform Ġcorresponding Ġto Ġa Ġdraft Ġfor Ġsubsequent Ġminor Ġpost - ed iting Ġby Ġa Ġhuman , Ġsim ulating Ġthe Ġuse Ġof Ġthe Ġgenerated Ġoutput Ġas Ġa Ġproduct Ġwhere Ġthe Ġfinal Ġcustomer Ġis Ġexpected Ġto Ġdo Ġown Ġpost - ed iting Ġbefore Ġpublication . ĠThe Ġsecond Ġjournalist Ġdirectly Ġedited Ġthe Ġnews Ġto Ġa Ġstate Ġready Ġfor Ġdirect Ġpublication Ġin Ġa Ġnews Ġstream Ġlabeled Ġas Ġmachine - generated Ġnews . ĠIn Ġaddition Ġto Ġcorrecting Ġfactual Ġerrors , Ġthe Ġjournalists Ġremoved Ġexcessive Ġrepetition , Ġimproved Ġtext Ġflu ency , Ġas Ġwell Ġas Ġoccasionally Ġincluded Ġimportant Ġfacts Ġwhich Ġthe Ġsystem Ġleft Ġun generated . ĠThe ĠW ER Ġmeasured Ġagainst Ġthe Ġoutput Ġconsidered Ġready Ġfor Ġpost - ed iting , Ġis Ġ9 . 9 % Ġ( 11 . 2 % Ġdisreg arding Ġpunct uation ), Ġonly Ġslightly Ġworse Ġthan Ġthe Ġevaluation Ġwith Ġonly Ġthe Ġfactual Ġand Ġgram matical Ġerrors Ġcorrected . ĠThe ĠW ER Ġmeasured Ġagainst Ġthe Ġoutput Ġconsidered Ġready Ġfor Ġdirect Ġrelease , Ġwas Ġ22 . 0 % Ġ( 24 . 4 % Ġdisreg arding Ġpunct uation ). ĠIn Ġother Ġwords , Ġ75 âĢĵ 90 % Ġof Ġthe Ġgenerated Ġtext Ġcan Ġbe Ġdirectly Ġused , Ġdepending Ġon Ġthe Ġexpected Ġpost - ed iting Ġeffort . </s> Figure ĠFIG REF 27 Ġshows Ġtwo Ġexample Ġgames Ġalong Ġwith Ġthe Ġgenerated Ġreports Ġand Ġmanual Ġcorrections Ġmade Ġby Ġthe Ġjournalist Ġin Ġorder Ġto Ġprepare Ġit Ġfor Ġpublication . ĠLit eral Ġtranslations Ġfrom Ġthe Ġgenerated , Ġunc orrect ed ĠFinnish Ġinto ĠEnglish Ġare Ġprovided Ġfor Ġreference . </s> Con clusions Ġand ĠFuture ĠWork </s> We Ġdeveloped Ġand Ġevaluated Ġan Ġend - to - end Ġsystem Ġfor Ġnews Ġgeneration Ġfrom Ġstructured Ġdata , Ġusing Ġa Ġcorpus Ġof Ġnews Ġand Ġgame Ġstatistics Ġin Ġthe Ġice Ġhockey Ġdomain . ĠIn Ġterms Ġof Ġthe Ġdata , Ġour Ġprimary Ġfinding Ġwas Ġthe Ġlevel Ġto Ġwhich Ġprofessionally Ġproduced Ġnews Ġcontain Ġinformation Ġthat Ġcannot Ġbe Ġinferred Ġfrom Ġthe Ġgame Ġstatistics . ĠThis Ġleads Ġto Ġthe Ġmodel Ġlearning Ġto Ġ" hall uc inate " Ġfacts Ġand Ġnecess itates Ġa Ġmanual Ġalignment Ġand Ġediting Ġof Ġthe Ġtraining Ġdata . ĠOnce Ġwe Ġcreated Ġa Ġsuitable Ġtraining Ġdataset , Ġwe Ġwere Ġable Ġto Ġgenerate Ġhighly Ġgram matical Ġtext Ġwhich , Ġin Ġterms Ġof Ġword Ġerror Ġrate Ġ( edit Ġdistance ), Ġwas Ġrelatively Ġclose Ġto Ġwhat Ġwas Ġjudged Ġas Ġa Ġviable Ġproduct Ġby Ġdomain Ġjournalists . ĠWe Ġfound Ġthat Ġmost Ġfactual Ġerrors Ġin Ġthe Ġgenerated Ġoutput Ġfall Ġinto Ġa Ġsmall Ġnumber Ġof Ġcategories , Ġmostly Ġrelated Ġto Ġcopying Ġnames Ġfrom Ġthe Ġinput , Ġtypes Ġof Ġevents , Ġand Ġtime Ġreferences . ĠAdd ressing Ġthese Ġerrors Ġis Ġa Ġmatter Ġof Ġfuture Ġwork Ġand Ġcan Ġbe Ġapproached Ġusing Ġdata Ġaug mentation Ġtechniques Ġas Ġwell Ġas Ġintroducing Ġsub - word Ġunits Ġwhich Ġwould Ġallow Ġthe Ġmodel Ġto Ġdeal Ġwith Ġinf lections . </s> Currently , Ġwe Ġonly Ġgenerate Ġthe Ġnews Ġas Ġindependent Ġevents , Ġwith Ġroughly Ġone Ġsentence Ġcorresponding Ġto Ġone Ġevent . ĠAs Ġthis Ġleads Ġto Ġa Ġsomewhat Ġunnatural Ġtext , Ġwe Ġhave Ġattempted Ġin Ġpreliminary Ġexperiments Ġto Ġgenerate Ġwhole Ġnews Ġtexts Ġat Ġonce , Ġas Ġwell Ġas Ġsentences Ġcombining Ġseveral Ġevents , Ġnevertheless Ġwith Ġresults Ġfar Ġfrom Ġuseful . ĠThis Ġis Ġlikely Ġdue Ġto Ġthe Ġrelatively Ġsmall Ġnumber Ġof Ġtraining Ġexamples Ġwhere Ġa Ġsingle Ġsentence Ġaccounts Ġfor Ġseveral Ġdistinct Ġevents . ĠWe Ġwill Ġfocus Ġon Ġthis Ġproblem Ġin Ġour Ġfuture Ġwork , Ġinvestigating Ġmethods Ġwhich Ġwould Ġallow Ġpre - training Ġthe Ġgeneration Ġmodel Ġso Ġas Ġto Ġbe Ġable Ġto Ġsuccessfully Ġaccept Ġseveral Ġevents Ġon Ġits Ġinput . </s> The Ġnew Ġdataset , Ġthe Ġoriginal Ġnews Ġcorpus Ġand Ġthe Ġsource Ġcode Ġof Ġthe Ġmodel Ġare Ġavailable Ġfor Ġresearch Ġuse . </s> Acknowled gments </s> We Ġgrate fully Ġacknowledge Ġthe Ġcollaboration Ġof ĠMai ja ĠPa ikk ala , ĠS alla ĠSal m ela Ġand ĠP ih la ĠLeh mus j oki Ġof Ġthe ĠFinnish ĠNews ĠAgency ĠST T , Ġas Ġwell Ġas Ġthe Ġsupport Ġof Ġthe ĠGoogle ĠDigital ĠNews ĠInnovation ĠFund , ĠAcademy Ġof ĠFinland , ĠC SC ĠâĢĵ ĠIT ĠCenter Ġfor ĠScience , Ġand Ġthe ĠNVIDIA ĠCorporation ĠGPU ĠGrant ĠProgram .
Introduction Automated, or robotic, journalism aims at news generation from structured data sources, either as the final product or as a draft for subsequent post-editing. At present, automated journalism typically focuses on domains such as sports, finance and similar statistics-based reporting, where there is a commercial product potential due to the high volume of news, combined with the expectation of a relatively straightforward task. News generation systems—especially those deployed in practice—tend to be based on intricate template filling, aiming to give the users the full control of the generated facts, while maintaining a reasonable variability of the resulting text. This comes at the price of having to develop the templates and specify their control logic, neither of which are tasks naturally fitting journalists' work. Further, this development needs to be repeated for every domain, as the templates are not easily transferred across domains. Examples of the template-based news generation systems for Finnish are Voitto by the Finnish Public Service Broadcasting Company (YLE) used for sports news generation, as well as Vaalibotti BIBREF0, a hybrid machine learning and template-based system used for election news. BIBREF1 suggested a neural template generation, which jointly models latent templates and text generation. Such a system increases interpretability and controllability of the generation, however, recent sequence-to-sequence systems represent the state-of-the-art in data-to-text generation. BIBREF2 In this paper, we report on the development of a news generation system for the Finnish ice hockey news domain, based on sequence-to-sequence methods. In order to train such a system, we compile a corpus of news based on over 2000 game reports from the Finnish News Agency STT. While developing this corpus into a form suitable for training of end-to-end systems naturally requires manual effort, we argue that compiling and refining a set of text examples is a more natural way for journalists to interact with the system, in order for them to codify their knowledge and to adapt it for new domains. Our aim is to generate reports that give an overview of a game based on information inferrable from the statistics. Such reports can be used either as a basis for further post-editing by a journalist imprinting own insights and background information, or even used directly as a news stream labelled as machine-generated. In the following, we will introduce the news dataset and the process of its creation, introduce an end-to-end model for news generation, and evaluate its output respective to the abovementioned objectives. Ice Hockey News Dataset An ice hockey game is recorded into statistics in terms of different events occurring during play, such as goals and penalties. In order to train a model to generate game reports, we need access to these events, as well as example news articles about the game. Only recently have game statistics become available to the public through a web interface or API, whereas the information has traditionally been recorded as structured text files. The news corpus from the Finnish News Agency STT includes, among all other news, articles covering ice hockey games in the Finnish leagues during the years 1994–2018. In addition to news articles, the corpus also includes the original game statistics text files. This creates an opportunity to align the game statistics with the corresponding news articles, producing a dataset of over 20 years of ice hockey data with reference news articles for the games. When automatically pairing the game statistics with news articles using date and team names as a heuristic, we obtain a total of 3,454 games with existing statistics and at least one corresponding news article. Utilizing real journalistic material poses a challenge in that the articles mix information that can be found directly in the game statistics (e.g., scores and names) with information inferable from the statistics (e.g., statements such as shortly after), information based on background knowledge (e.g., a team's home city or player's position), game insight and judgement based on viewing the game (e.g., expressions such as slapshot or tipping the puck describing the character of a shot), and even player interviews. Therefore, directly using the limited amount of actual news articles for end-to-end system training becomes problematic. In our initial experiments the generation model learns to “hallucinate” facts, as easily occurs when the target text is too loosely related to the conditioning input. In order to ensure that the generation model is able to learn to generate accurate descriptions from game statistics, we clean the news corpus by manually aligning corresponding text spans with game events detailed in the statistics. For the sake of comparison, let us consider the Rotowire corpus BIBREF3 containing basketball game summaries and statistics, which was recently released and has become a popular data set for training data-to-text generation systems (cf., e.g., BIBREF4, BIBREF1, BIBREF5). The Rotowire game summaries are straightforward in their style of reporting, focusing on the game at hand and tend for the most part to reference facts in the statistics. By contrast, our news corpus is more heterogeneous, including both articles focusing on the particular game and articles that take a broader perspective (e.g., describing a player's career). The STT news articles tend to read in the journalist's voice, putting substantial emphasis on the character of the game, often in colorful language, as well as quoting players and coaches. An example of the events available in the game statistics, the actual news article on the game, and how these align, is shown in Figure FIGREF5. Text spans highlighted with blue color are based on information available in the statistics, all other being external information. It illustrates the typical portion of a raw article that is not inferrable from the data. English translations are available for a comparable example in Figure FIGREF27. Ice Hockey News Dataset ::: Extraction of Game Events For each event occurring in a game and recorded in statistics, we identify its type and associated features. There are four event types: end result, goal, penalty and save. As a general rule, for each game the end result entry specifies total scores, participating teams and additional circumstances of the game such as overtime or shootout. The goal event is the most frequent and includes features such as goal scorer, assists, team, resulting score, time and the current state of the game (e.g., power play, penalty shot). We also derive special features that in many cases require consideration of other events in the context, but pertain to one particular event, e.g., is the goal deciding or final. The penalty event specifies player, team, time in the game and penalty time. The save event summarises the number of saves of a goaltender/team. We perform information extraction with regular expressions on the statistics in order to structure the game into a chronological sequence of events, which can then be aligned with the corresponding text spans from the news article, as well as used as input in the text generation. An initial sequence of events after information extraction is shown on the left side in Figure FIGREF5. Before text generation, these events are yet enhanced with additional information derived from the event itself, or the game. Ice Hockey News Dataset ::: Manual Alignment The manual alignment of game events and the article text is carried out by a single annotator in approximately 6 weeks as an integral part of the system development work. The annotator receives a sequence of extracted events and aligns these with the corresponding news article, typically but not necessarily a single sentence for every expressed event. All ungrounded parts within the text are removed and if necessary, the text is modified for fluency. We cannot align full sentences with events as, for instance, BIBREF6 do, as often information not grounded in the statistics is expressed together with statistics-based facts within the same sentences and clauses. The alignment process therefore frequently requires rephrasing the text, for instance, in order to make it more neutral and avoid generating arbitrary subjective judgements. We find that the news article commonly includes description that is not evident from the data (e.g., subjective characteristics of the player or the shot), and often may reflect the reporter's viewpoint. For instance, the reporter may evaluate an event negatively by writing Player A did not manage to score more than one goal, reflecting expectation of the player's performance, which we would change into a more objective form such as Player A scored one goal. Similarly, word choices based on viewing the game (e.g., slapshot) are changed to more neutral counterparts. After removing the uninferable parts of a sentence, the remainder is often relatively short, in which case we sometimes opt to replace the information that was removed with references to other appropriate information from the game statistics, such as the time of the event or the state of the game. This serves to maintain a similar sentence and clause structure as that of the original corpus. In the majority of our aligned data one text span refers to one game event. However, in some cases the same indivisible span can describe multiple events, most commonly of the same type (e.g., Player A scored two goals, or Team A received a total of eight penalties). In such cases, we produce multi-event alignments, where all related events are aligned with a single text span. Multi-event alignments support the development of more advanced generation, which stands to produce more natural-sounding reports in terms of less repetition and more flexible structuring. Ice Hockey News Dataset ::: Corpus Statistics In Table TABREF10, we summarize the overall size statistics of the final ice hockey corpus after the statistics and news articles have been automatically paired, and events have been manually aligned with the text. In total, 2,307 games were manually checked (66.8% of the paired corpus), of which 2,134 games were correctly paired with the article describing the game. In these games, 12,251 out of 36,097 events (33.9%) were referenced in the text and successfully aligned. End result occurs in nearly all news articles as the first event, whereas the goal event is by far the most frequent one, each game mentioning on average 3.1 goals. The number of aligned events is greater than the number of aligned text spans due to multi-event alignments. While 82.1% of text spans align with a single event, there is a long tail of multi-event alignments, with 11.2% aligning to two events, 3.4% to three, 1.4% to four, etc. In Table TABREF11 we measure the lexical diversity of original ice hockey news articles, as well as the resulting dataset after manual alignment, by computing the Standardized Type–Token Ratio (STTR). The measure is defined as the number of unique tokens or lemmas divided by the total number of tokens, calculated on every segment of 1,000 tokens separately and averaged across the corpus. STTR is more meaningful than the standard type–token ratio when comparing corpora of substantially different sizes. Both corpora are tokenized and lemmatized using the Turku Neural Parser pipeline BIBREF7, BIBREF8. STTR of the aligned corpus is lower than in the original hockey news on both word and lemma level, indicating a somewhat—but not substantially so—more restricted vocabulary use in our aligned subset. Event Selection Model As illustrated previously, any given news article describes only a subset of most noteworthy events of a game. We observe that most reports are concise, referencing on average 5.7 events. The distribution of events/report is: 1st quantile at 3 events (20.9% of events in game), 2nd at 5 (22.2%), 3rd at 7 (38.5%), 4th at 36 (100%). Our alignment serves as a gold standard reflecting which events the journalists have chosen to mention for each game. In our generation task, we are presented with the problem of selecting appropriate events from the full game statistics. We use the gold standard selection during training and validation of the text generation model, as well as the automatic evaluation. As we deploy our text generation model for manual evaluation, we use a Conditional Random Field (CRF) model to predict which events to mention. Casting the event selection problem as a sequence labeling task, the CRF model takes as input the full sequence of events in one game together with associated features for each event, and predicts a binary label for each event. We achieve an overall F-score of 67.1% on the test set, which broken down by event type is: end result (98.0%), goal (70.2%), penalty (20.1%), save (47.7%). Penalties are the most difficult to predict, being reported only 7.8% of the time in reality, e.g., compared to 54.1% for goals. Text Generation Next, we present the model architecture used in text generation, and evaluate the model on a popular baseline dataset. After that, we describe the training of the generation model on our ice hockey corpus and use automatic evaluation metrics to compare against existing references. Text Generation ::: Model Architecture We use a pointer-generation network BIBREF10, BIBREF11, BIBREF12, where the neural attention mechanism in the encoder-decoder model is adapted to jointly model a probability distribution over words from the known vocabulary, a distribution over words from the input sequence to copy and a probability that controls the copying mechanism. A separate coverage attention vector, a sum of past attention distributions, is maintained to inform the model of its past attention decisions. Such a coverage model is shown to prevent text repetition in generated output BIBREF13, BIBREF12. The model is implemented using the OpenNMT-py library BIBREF14. The encoder has two bidirectional LSTM layers with 500 hidden units, together with 500-dimensional word embeddings. The decoder has two unidirectional LSTM layers with 500 hidden units. Both encoder and decoder apply a dropout of 0.3 between LSTM layers. Text Generation ::: Baseline Experiments on the E2E Dataset To demonstrate the performance of our generation model architecture, we report results on a known dataset with published baselines, namely the E2E NLG Challenge BIBREF2 on end-to-end natural language generation in spoken dialogue systems. The task is to produce a natural language description of a restaurant based on a given meaning representation (MR)—an unordered set of attributes and their values. The attributes included, among others, the restaurant name, area, food type and rating. We represent the given MR as a sequence of tokens where each attribute value is embedded into XML-style beginning and end attribute markers, and the order of attributes is kept fixed across the whole dataset. The target output is a sequence of tokens. We do not apply any explicit delexicalization steps. In Table TABREF15 we measure BLEU BIBREF19, NIST BIBREF20, METEOR BIBREF21, ROUGE-L BIBREF22 and CIDEr BIBREF23 metrics on the 2018 E2E NLG Challenge test data using the evaluation script provided by the organizers. Our generation system is compared to the official shared task baseline system, TGen BIBREF24, as well as to the top performing participant system on each score (ST top). Our system outperforms the TGen baseline on 3 out of 5 metrics (BLEU, METEOR and ROUGE-L), which is on par with the official shared task results, where not a single one participant system was able to surpass the baseline on all five metrics. On two metrics, BLEU and METEOR, our system outperforms the best shared task participants. E2E NLG Challenge evaluation is based on having multiple references for each MR, on average each unique MR in the corpus having 8 reference descriptions. In the evaluation, the output for each unique MR is compared against all references and the maximum score is used, naturally leading to higher scores. To have more comparable numbers to our ice hockey corpus, where we have only one reference for each input event, we also include scores obtained by comparing each MR to each of its reference descriptions separately as if they were individual data points (Ours single ref.). Text Generation ::: Hockey Data Representation Currently, we concentrate on training the model only with text spans aligning with single events, excluding the less frequent multi-event alignments. Furthermore, we are considering each event as a separate training example, independent of other events in the game. Given a single event described as a sequence of features and their values, our text generation model is trained to produce the text span aligned with it. Following the data representation used in E2E NLG Challenge experiments, the input events are represented as a linearized sequence of tokens, where XML-style beginning and end tags are used to separate the different features (see Figure FIGREF18). This allows the model to directly copy some of the input tokens to the output when necessary. The ability to copy tokens is especially important with player names and exact times, where the vocabulary is sparse, and many of these can even be previously unseen, unknown tokens. In addition, we also include features that are meant to inform generation without being copied themselves, for example, the type of the event. The target of generation is a tokenized sequence of words, where also dashes inside scores are separated, allowing the model to swap scores when necessary. The reference text sometimes flips the order of the teams, requiring the score to be inverted as well (Team A–Team B 0–1 into Team B won Team A 1–0). One particular challenge in our corpus is that the decision of which aspects to focus on, i.e., which particular features from the source event to verbalize, is relatively arbitrary. For example, sometimes a journalist mentions the player or players assisting a goal, but in many cases it is left out. Both options are equally correct, but overlap-based metrics such as BLEU penalize such creative variation. By contrast, in other text generation datasets such as the E2E NLG Challenge, the output text in general describes all input features. To account for this variation, we include a length feature to act as a minimal supervision signal for the model to rely on. We divide output text lengths into three evenly sized categories (short, medium and long) to provide a hint to the model during training of how long and detailed output it is expected to generate for each training example. At test time, we then have the possibility to control the desired approximate length of the generated text. In the experiments throughout this paper, we generate all three length variants for each event and pick the one with the highest average confidence of the generation model. Text Generation ::: Training and Optimization The model is trained using the Adam optimizer with learning rate of 0.0005 and batch size of 32. The model is trained for a maximum of 8000 steps (ca. 40 epochs), and the final model is chosen based on validation set performance. We use 80% of the aligned games for training, 10% for validation and 10% for testing. In our initial experiments, we used the RBFOpt library BIBREF25 for hyperparameter tuning, maximizing validation set BLEU score. However, when manually inspecting generation results on the validation set, we noticed that models with a higher BLEU score result in more fluent text but generate more factual mistakes. This observation is supported by BIBREF3, who note that BLEU score tends to reward fluent text rather than other aspects desirable in generation of sports summaries from data. For this reason, we ultimately decided to use hyperparameters manually tuned on the validation set to give a good perceived balance between fluent and factually correct text. Text Generation ::: Automatic Evaluation of Hockey Generation In Table TABREF21, we provide evaluation results using the five aforementioned metrics. We evaluate on event level using gold standard event selection, where each generated event description is compared to its existing reference text. As the model is trained to produce a tokenized sequence, we apply a detokenizer to be able to compare against the original untokenized reference text. On the test set, the model achieves a BLEU score of 19.67. To the extent that different datasets allow comparison, the best reported score on the Rotowire basketball news corpus is 16.50 BIBREF5. Compared to our earlier E2E baseline experiment, we score lower than our closest comparable reference of 31.90 (with single references), which is understandable due to the much smaller train set size for the hockey corpus (about 13% in size). In Figure FIGREF22, we plot the learning curve with increasing sizes of training data in order to illustrate how generation performance benefits from more data. The learning curve is still steadily increasing when using 100% of the training data currently available, which indicates that more data would most likely further improve the performance. Human evaluation As our objective is practically usable news generation, we carry out a manual evaluation of its output on 59 randomly selected games from the test set, focusing in particular on the corrections that would be necessary to obtain acceptable output. Example corrections are shown in Figure FIGREF27. These full game reports are generated by first applying the selection model described in Section SECREF3 to select the events to be included in the report, and then using the text generation model to verbalize each selected event. The generated texts of the events are then detokenized and concatenated in chronological order into a full game report. Human evaluation ::: Minimum Edit Evaluation In the minimum edit evaluation, carried out by the annotator who created the news corpus, only factual mistakes and grammatical errors are corrected, resulting in text which may remain awkward or unfluent. The word error rate (WER) of the generated text compared to its corrected variant as a reference is 5.6% (6.2% disregarding punctuation). The WER measure is defined as the number of insertions, substitutions, and deletions divided by the total length of the reference, in terms of tokens. The measure is the edit distance of the generated text and its corrected variant, directly reflecting the amount of effort needed to correct the generated output. Human evaluation ::: Minimum Edit Evaluation ::: Factual Correctness The factual errors and their types are summarized in Table TABREF23. From the total of 510 game events generated by the system, 78 of these contained a factual error, i.e. 84.7% were generated without factual errors. The most notable errors involved player or team names. The input for a single game event may contain more than one name (e.g. the goal scorer and up to two assisting players). In these cases, the model occasionally paired first name and surname incorrectly. Less frequent errors include wrong team pairings when referring to the end result of the game, e.g., Team A lost to Team A. In sentences where the exact time of a game event is generated as a number (at time 39.54) the model copied the time reliably, but when the approximate time of an event is verbalized (at the end of the second period) there were occasional errors. Another notable error category is types of goals or their scores (e.g. 3–0, deciding goal, tying goal). In this category, the error similarly occurred in cases when the reference is verbal (third goal), but occasionally also in numerical representation (3–0 instead of 0–3). The other, less common categories relate to game total scores, power play, assists and penalties. Human evaluation ::: Minimum Edit Evaluation ::: Fluency Overall, the generated text is highly grammatical. The most frequent grammatical error in the output is unconjugated player names; commonly names in their Finnish nominative case that should be in genitive. In a few cases, the model makes grammatical errors when it copies an incompatible word from the input, e.g., a name instead of a time reference. Both error types commonly occur when the model copies from the input. As it operates on the token rather than sub-word level, it is challenging to map between word forms and inflect infrequent words such as player names. The latter error likely occurs when the copy attention is indecisive and fails to recognize the type for infrequent tokens. Most fluency issues relate to the overall flow and structure of the report. Addressing these issues would require the model to take into account multiple events in a game, and combine the information more flexibly to avoid repetition. For instance, the output may repeatedly mention the period number for all goals in the same period. Likewise, this setup sometimes results in unnatural, yet grammatical, repetition of words across consecutive sentences. Even though the model has learned a selection of verbs meaning to score a goal, it is unable to ensure their varied use. While not successful in our initial experiments, generating text based on the multi-event alignments or at document level may eventually overcome these issues. Human evaluation ::: Product-Readiness Evaluation The second human evaluation aimed at judging the acceptability of the output for production use in a news agency. The output is evaluated in terms of its usability for a news channel labelled as being machine-generated, i.e. not aiming at the level of a human journalist equipped with substantial background information. The evaluation was carried out by two journalists from the STT agency, who split the 59 games among themselves approximately evenly. The first journalist edited the games to a form corresponding to a draft for subsequent minor post-editing by a human, simulating the use of the generated output as a product where the final customer is expected to do own post-editing before publication. The second journalist directly edited the news to a state ready for direct publication in a news stream labeled as machine-generated news. In addition to correcting factual errors, the journalists removed excessive repetition, improved text fluency, as well as occasionally included important facts which the system left ungenerated. The WER measured against the output considered ready for post-editing, is 9.9% (11.2% disregarding punctuation), only slightly worse than the evaluation with only the factual and grammatical errors corrected. The WER measured against the output considered ready for direct release, was 22.0% (24.4% disregarding punctuation). In other words, 75–90% of the generated text can be directly used, depending on the expected post-editing effort. Figure FIGREF27 shows two example games along with the generated reports and manual corrections made by the journalist in order to prepare it for publication. Literal translations from the generated, uncorrected Finnish into English are provided for reference. Conclusions and Future Work We developed and evaluated an end-to-end system for news generation from structured data, using a corpus of news and game statistics in the ice hockey domain. In terms of the data, our primary finding was the level to which professionally produced news contain information that cannot be inferred from the game statistics. This leads to the model learning to "hallucinate" facts and necessitates a manual alignment and editing of the training data. Once we created a suitable training dataset, we were able to generate highly grammatical text which, in terms of word error rate (edit distance), was relatively close to what was judged as a viable product by domain journalists. We found that most factual errors in the generated output fall into a small number of categories, mostly related to copying names from the input, types of events, and time references. Addressing these errors is a matter of future work and can be approached using data augmentation techniques as well as introducing sub-word units which would allow the model to deal with inflections. Currently, we only generate the news as independent events, with roughly one sentence corresponding to one event. As this leads to a somewhat unnatural text, we have attempted in preliminary experiments to generate whole news texts at once, as well as sentences combining several events, nevertheless with results far from useful. This is likely due to the relatively small number of training examples where a single sentence accounts for several distinct events. We will focus on this problem in our future work, investigating methods which would allow pre-training the generation model so as to be able to successfully accept several events on its input. The new dataset, the original news corpus and the source code of the model are available for research use. Acknowledgments We gratefully acknowledge the collaboration of Maija Paikkala, Salla Salmela and Pihla Lehmusjoki of the Finnish News Agency STT, as well as the support of the Google Digital News Innovation Fund, Academy of Finland, CSC – IT Center for Science, and the NVIDIA Corporation GPU Grant Program.
What evaluation criteria and metrics were used to evaluate the generated text?
abe2393415e533cb06311e74ed1c5674cff8571f
What Ġevaluation Ġcriteria Ġand Ġmetrics Ġwere Ġused Ġto Ġevaluate Ġthe Ġgenerated Ġtext ?
BLE U Ġ, ĠN IST Ġ, ĠMET E OR Ġ, ĠR OU GE - L , ĠC ID Er Ġ, Ġevaluation Ġscript , Ġautomatic Ġevaluation , Ġhuman Ġevaluation , Ġminimum Ġedit Ġevaluation , Ġword Ġerror Ġrate Ġ( WER ), Ġfactual Ġerrors Ġand Ġtheir Ġtypes , Ġflu ency Ġissues , Ġaccept ability Ġof Ġthe Ġoutput Ġfor Ġproduction Ġuse Ġin Ġa Ġnews Ġagency
<s> What Ġevaluation Ġcriteria Ġand Ġmetrics Ġwere Ġused Ġto Ġevaluate Ġthe Ġgenerated Ġtext ? </s> Introduction </s> Autom ated , Ġor Ġrobotic , Ġjournalism Ġaims Ġat Ġnews Ġgeneration Ġfrom Ġstructured Ġdata Ġsources , Ġeither Ġas Ġthe Ġfinal Ġproduct Ġor Ġas Ġa Ġdraft Ġfor Ġsubsequent Ġpost - ed iting . ĠAt Ġpresent , Ġautomated Ġjournalism Ġtypically Ġfocuses Ġon Ġdomains Ġsuch Ġas Ġsports , Ġfinance Ġand Ġsimilar Ġstatistics - based Ġreporting , Ġwhere Ġthere Ġis Ġa Ġcommercial Ġproduct Ġpotential Ġdue Ġto Ġthe Ġhigh Ġvolume Ġof Ġnews , Ġcombined Ġwith Ġthe Ġexpectation Ġof Ġa Ġrelatively Ġstraightforward Ġtask . </s> News Ġgeneration Ġsystems âĢĶ especially Ġthose Ġdeployed Ġin Ġpractice âĢĶ t end Ġto Ġbe Ġbased Ġon Ġintricate Ġtemplate Ġfilling , Ġaiming Ġto Ġgive Ġthe Ġusers Ġthe Ġfull Ġcontrol Ġof Ġthe Ġgenerated Ġfacts , Ġwhile Ġmaintaining Ġa Ġreasonable Ġvariability Ġof Ġthe Ġresulting Ġtext . ĠThis Ġcomes Ġat Ġthe Ġprice Ġof Ġhaving Ġto Ġdevelop Ġthe Ġtemplates Ġand Ġspecify Ġtheir Ġcontrol Ġlogic , Ġneither Ġof Ġwhich Ġare Ġtasks Ġnaturally Ġfitting Ġjournalists ' Ġwork . ĠFurther , Ġthis Ġdevelopment Ġneeds Ġto Ġbe Ġrepeated Ġfor Ġevery Ġdomain , Ġas Ġthe Ġtemplates Ġare Ġnot Ġeasily Ġtransferred Ġacross Ġdomains . ĠExamples Ġof Ġthe Ġtemplate - based Ġnews Ġgeneration Ġsystems Ġfor ĠFinnish Ġare ĠVo itto Ġby Ġthe ĠFinnish ĠPublic ĠService ĠBroadcasting ĠCompany Ġ( Y LE ) Ġused Ġfor Ġsports Ġnews Ġgeneration , Ġas Ġwell Ġas ĠVa al ib otti ĠB IB REF 0 , Ġa Ġhybrid Ġmachine Ġlearning Ġand Ġtemplate - based Ġsystem Ġused Ġfor Ġelection Ġnews . </s> BI B REF 1 Ġsuggested Ġa Ġneural Ġtemplate Ġgeneration , Ġwhich Ġjointly Ġmodels Ġlatent Ġtemplates Ġand Ġtext Ġgeneration . ĠSuch Ġa Ġsystem Ġincreases Ġinterpret ability Ġand Ġcontro ll ability Ġof Ġthe Ġgeneration , Ġhowever , Ġrecent Ġsequence - to - sequence Ġsystems Ġrepresent Ġthe Ġstate - of - the - art Ġin Ġdata - to - text Ġgeneration . ĠB IB REF 2 </s> In Ġthis Ġpaper , Ġwe Ġreport Ġon Ġthe Ġdevelopment Ġof Ġa Ġnews Ġgeneration Ġsystem Ġfor Ġthe ĠFinnish Ġice Ġhockey Ġnews Ġdomain , Ġbased Ġon Ġsequence - to - sequence Ġmethods . ĠIn Ġorder Ġto Ġtrain Ġsuch Ġa Ġsystem , Ġwe Ġcompile Ġa Ġcorpus Ġof Ġnews Ġbased Ġon Ġover Ġ2000 Ġgame Ġreports Ġfrom Ġthe ĠFinnish ĠNews ĠAgency ĠST T . ĠWhile Ġdeveloping Ġthis Ġcorpus Ġinto Ġa Ġform Ġsuitable Ġfor Ġtraining Ġof Ġend - to - end Ġsystems Ġnaturally Ġrequires Ġmanual Ġeffort , Ġwe Ġargue Ġthat Ġcompiling Ġand Ġrefining Ġa Ġset Ġof Ġtext Ġexamples Ġis Ġa Ġmore Ġnatural Ġway Ġfor Ġjournalists Ġto Ġinteract Ġwith Ġthe Ġsystem , Ġin Ġorder Ġfor Ġthem Ġto Ġcod ify Ġtheir Ġknowledge Ġand Ġto Ġadapt Ġit Ġfor Ġnew Ġdomains . </s> Our Ġaim Ġis Ġto Ġgenerate Ġreports Ġthat Ġgive Ġan Ġoverview Ġof Ġa Ġgame Ġbased Ġon Ġinformation Ġinf err able Ġfrom Ġthe Ġstatistics . ĠSuch Ġreports Ġcan Ġbe Ġused Ġeither Ġas Ġa Ġbasis Ġfor Ġfurther Ġpost - ed iting Ġby Ġa Ġjournalist Ġimprint ing Ġown Ġinsights Ġand Ġbackground Ġinformation , Ġor Ġeven Ġused Ġdirectly Ġas Ġa Ġnews Ġstream Ġlabelled Ġas Ġmachine - generated . </s> In Ġthe Ġfollowing , Ġwe Ġwill Ġintroduce Ġthe Ġnews Ġdataset Ġand Ġthe Ġprocess Ġof Ġits Ġcreation , Ġintroduce Ġan Ġend - to - end Ġmodel Ġfor Ġnews Ġgeneration , Ġand Ġevaluate Ġits Ġoutput Ġrespective Ġto Ġthe Ġabove mentioned Ġobjectives . </s> Ice ĠHockey ĠNews ĠDat as et </s> An Ġice Ġhockey Ġgame Ġis Ġrecorded Ġinto Ġstatistics Ġin Ġterms Ġof Ġdifferent Ġevents Ġoccurring Ġduring Ġplay , Ġsuch Ġas Ġgoals Ġand Ġpenalties . ĠIn Ġorder Ġto Ġtrain Ġa Ġmodel Ġto Ġgenerate Ġgame Ġreports , Ġwe Ġneed Ġaccess Ġto Ġthese Ġevents , Ġas Ġwell Ġas Ġexample Ġnews Ġarticles Ġabout Ġthe Ġgame . ĠOnly Ġrecently Ġhave Ġgame Ġstatistics Ġbecome Ġavailable Ġto Ġthe Ġpublic Ġthrough Ġa Ġweb Ġinterface Ġor ĠAPI , Ġwhereas Ġthe Ġinformation Ġhas Ġtraditionally Ġbeen Ġrecorded Ġas Ġstructured Ġtext Ġfiles . </s> The Ġnews Ġcorpus Ġfrom Ġthe ĠFinnish ĠNews ĠAgency ĠST T Ġincludes , Ġamong Ġall Ġother Ġnews , Ġarticles Ġcovering Ġice Ġhockey Ġgames Ġin Ġthe ĠFinnish Ġleagues Ġduring Ġthe Ġyears Ġ1994 âĢĵ 2018 . ĠIn Ġaddition Ġto Ġnews Ġarticles , Ġthe Ġcorpus Ġalso Ġincludes Ġthe Ġoriginal Ġgame Ġstatistics Ġtext Ġfiles . ĠThis Ġcreates Ġan Ġopportunity Ġto Ġalign Ġthe Ġgame Ġstatistics Ġwith Ġthe Ġcorresponding Ġnews Ġarticles , Ġproducing Ġa Ġdataset Ġof Ġover Ġ20 Ġyears Ġof Ġice Ġhockey Ġdata Ġwith Ġreference Ġnews Ġarticles Ġfor Ġthe Ġgames . ĠWhen Ġautomatically Ġpairing Ġthe Ġgame Ġstatistics Ġwith Ġnews Ġarticles Ġusing Ġdate Ġand Ġteam Ġnames Ġas Ġa Ġhe uristic , Ġwe Ġobtain Ġa Ġtotal Ġof Ġ3 , 454 Ġgames Ġwith Ġexisting Ġstatistics Ġand Ġat Ġleast Ġone Ġcorresponding Ġnews Ġarticle . </s> Ut il izing Ġreal Ġjournalistic Ġmaterial Ġposes Ġa Ġchallenge Ġin Ġthat Ġthe Ġarticles Ġmix Ġinformation Ġthat Ġcan Ġbe Ġfound Ġdirectly Ġin Ġthe Ġgame Ġstatistics Ġ( e . g ., Ġscores Ġand Ġnames ) Ġwith Ġinformation Ġinfer able Ġfrom Ġthe Ġstatistics Ġ( e . g ., Ġstatements Ġsuch Ġas Ġshortly Ġafter ), Ġinformation Ġbased Ġon Ġbackground Ġknowledge Ġ( e . g ., Ġa Ġteam 's Ġhome Ġcity Ġor Ġplayer 's Ġposition ), Ġgame Ġinsight Ġand Ġjudgement Ġbased Ġon Ġviewing Ġthe Ġgame Ġ( e . g ., Ġexpressions Ġsuch Ġas Ġslap shot Ġor Ġtipping Ġthe Ġpuck Ġdescribing Ġthe Ġcharacter Ġof Ġa Ġshot ), Ġand Ġeven Ġplayer Ġinterviews . </s> Therefore , Ġdirectly Ġusing Ġthe Ġlimited Ġamount Ġof Ġactual Ġnews Ġarticles Ġfor Ġend - to - end Ġsystem Ġtraining Ġbecomes Ġproblematic . ĠIn Ġour Ġinitial Ġexperiments Ġthe Ġgeneration Ġmodel Ġlearns Ġto ĠâĢ ľ hall uc inate âĢ Ŀ Ġfacts , Ġas Ġeasily Ġoccurs Ġwhen Ġthe Ġtarget Ġtext Ġis Ġtoo Ġloosely Ġrelated Ġto Ġthe Ġconditioning Ġinput . ĠIn Ġorder Ġto Ġensure Ġthat Ġthe Ġgeneration Ġmodel Ġis Ġable Ġto Ġlearn Ġto Ġgenerate Ġaccurate Ġdescriptions Ġfrom Ġgame Ġstatistics , Ġwe Ġclean Ġthe Ġnews Ġcorpus Ġby Ġmanually Ġalign ing Ġcorresponding Ġtext Ġspans Ġwith Ġgame Ġevents Ġdetailed Ġin Ġthe Ġstatistics . </s> For Ġthe Ġsake Ġof Ġcomparison , Ġlet Ġus Ġconsider Ġthe ĠRot ow ire Ġcorpus ĠB IB REF 3 Ġcontaining Ġbasketball Ġgame Ġsumm aries Ġand Ġstatistics , Ġwhich Ġwas Ġrecently Ġreleased Ġand Ġhas Ġbecome Ġa Ġpopular Ġdata Ġset Ġfor Ġtraining Ġdata - to - text Ġgeneration Ġsystems Ġ( cf ., Ġe . g ., ĠB IB REF 4 , ĠB IB REF 1 , ĠB IB REF 5 ). ĠThe ĠRot ow ire Ġgame Ġsumm aries Ġare Ġstraightforward Ġin Ġtheir Ġstyle Ġof Ġreporting , Ġfocusing Ġon Ġthe Ġgame Ġat Ġhand Ġand Ġtend Ġfor Ġthe Ġmost Ġpart Ġto Ġreference Ġfacts Ġin Ġthe Ġstatistics . ĠBy Ġcontrast , Ġour Ġnews Ġcorpus Ġis Ġmore Ġheter ogeneous , Ġincluding Ġboth Ġarticles Ġfocusing Ġon Ġthe Ġparticular Ġgame Ġand Ġarticles Ġthat Ġtake Ġa Ġbroader Ġperspective Ġ( e . g ., Ġdescribing Ġa Ġplayer 's Ġcareer ). ĠThe ĠST T Ġnews Ġarticles Ġtend Ġto Ġread Ġin Ġthe Ġjournalist 's Ġvoice , Ġputting Ġsubstantial Ġemphasis Ġon Ġthe Ġcharacter Ġof Ġthe Ġgame , Ġoften Ġin Ġcolorful Ġlanguage , Ġas Ġwell Ġas Ġquoting Ġplayers Ġand Ġcoaches . </s> An Ġexample Ġof Ġthe Ġevents Ġavailable Ġin Ġthe Ġgame Ġstatistics , Ġthe Ġactual Ġnews Ġarticle Ġon Ġthe Ġgame , Ġand Ġhow Ġthese Ġalign , Ġis Ġshown Ġin ĠFigure ĠFIG REF 5 . ĠText Ġspans Ġhighlighted Ġwith Ġblue Ġcolor Ġare Ġbased Ġon Ġinformation Ġavailable Ġin Ġthe Ġstatistics , Ġall Ġother Ġbeing Ġexternal Ġinformation . ĠIt Ġillustrates Ġthe Ġtypical Ġportion Ġof Ġa Ġraw Ġarticle Ġthat Ġis Ġnot Ġinf err able Ġfrom Ġthe Ġdata . ĠEnglish Ġtranslations Ġare Ġavailable Ġfor Ġa Ġcomparable Ġexample Ġin ĠFigure ĠFIG REF 27 . </s> Ice ĠHockey ĠNews ĠDat as et Ġ: :: ĠExt raction Ġof ĠGame ĠEvents </s> For Ġeach Ġevent Ġoccurring Ġin Ġa Ġgame Ġand Ġrecorded Ġin Ġstatistics , Ġwe Ġidentify Ġits Ġtype Ġand Ġassociated Ġfeatures . ĠThere Ġare Ġfour Ġevent Ġtypes : Ġend Ġresult , Ġgoal , Ġpenalty Ġand Ġsave . ĠAs Ġa Ġgeneral Ġrule , Ġfor Ġeach Ġgame Ġthe Ġend Ġresult Ġentry Ġspecifies Ġtotal Ġscores , Ġparticipating Ġteams Ġand Ġadditional Ġcircumstances Ġof Ġthe Ġgame Ġsuch Ġas Ġovertime Ġor Ġshootout . ĠThe Ġgoal Ġevent Ġis Ġthe Ġmost Ġfrequent Ġand Ġincludes Ġfeatures Ġsuch Ġas Ġgoal Ġscorer , Ġassists , Ġteam , Ġresulting Ġscore , Ġtime Ġand Ġthe Ġcurrent Ġstate Ġof Ġthe Ġgame Ġ( e . g ., Ġpower Ġplay , Ġpenalty Ġshot ). ĠWe Ġalso Ġderive Ġspecial Ġfeatures Ġthat Ġin Ġmany Ġcases Ġrequire Ġconsideration Ġof Ġother Ġevents Ġin Ġthe Ġcontext , Ġbut Ġper tain Ġto Ġone Ġparticular Ġevent , Ġe . g ., Ġis Ġthe Ġgoal Ġdeciding Ġor Ġfinal . ĠThe Ġpenalty Ġevent Ġspecifies Ġplayer , Ġteam , Ġtime Ġin Ġthe Ġgame Ġand Ġpenalty Ġtime . ĠThe Ġsave Ġevent Ġsummar ises Ġthe Ġnumber Ġof Ġsaves Ġof Ġa Ġgoaltender / team . </s> We Ġperform Ġinformation Ġextraction Ġwith Ġregular Ġexpressions Ġon Ġthe Ġstatistics Ġin Ġorder Ġto Ġstructure Ġthe Ġgame Ġinto Ġa Ġchronological Ġsequence Ġof Ġevents , Ġwhich Ġcan Ġthen Ġbe Ġaligned Ġwith Ġthe Ġcorresponding Ġtext Ġspans Ġfrom Ġthe Ġnews Ġarticle , Ġas Ġwell Ġas Ġused Ġas Ġinput Ġin Ġthe Ġtext Ġgeneration . ĠAn Ġinitial Ġsequence Ġof Ġevents Ġafter Ġinformation Ġextraction Ġis Ġshown Ġon Ġthe Ġleft Ġside Ġin ĠFigure ĠFIG REF 5 . ĠBefore Ġtext Ġgeneration , Ġthese Ġevents Ġare Ġyet Ġenhanced Ġwith Ġadditional Ġinformation Ġderived Ġfrom Ġthe Ġevent Ġitself , Ġor Ġthe Ġgame . </s> Ice ĠHockey ĠNews ĠDat as et Ġ: :: ĠManual ĠAl ignment </s> The Ġmanual Ġalignment Ġof Ġgame Ġevents Ġand Ġthe Ġarticle Ġtext Ġis Ġcarried Ġout Ġby Ġa Ġsingle Ġannot ator Ġin Ġapproximately Ġ6 Ġweeks Ġas Ġan Ġintegral Ġpart Ġof Ġthe Ġsystem Ġdevelopment Ġwork . ĠThe Ġannot ator Ġreceives Ġa Ġsequence Ġof Ġextracted Ġevents Ġand Ġalign s Ġthese Ġwith Ġthe Ġcorresponding Ġnews Ġarticle , Ġtypically Ġbut Ġnot Ġnecessarily Ġa Ġsingle Ġsentence Ġfor Ġevery Ġexpressed Ġevent . ĠAll Ġun ground ed Ġparts Ġwithin Ġthe Ġtext Ġare Ġremoved Ġand Ġif Ġnecessary , Ġthe Ġtext Ġis Ġmodified Ġfor Ġflu ency . ĠWe Ġcannot Ġalign Ġfull Ġsentences Ġwith Ġevents Ġas , Ġfor Ġinstance , ĠB IB REF 6 Ġdo , Ġas Ġoften Ġinformation Ġnot Ġgrounded Ġin Ġthe Ġstatistics Ġis Ġexpressed Ġtogether Ġwith Ġstatistics - based Ġfacts Ġwithin Ġthe Ġsame Ġsentences Ġand Ġclauses . </s> The Ġalignment Ġprocess Ġtherefore Ġfrequently Ġrequires Ġre ph r asing Ġthe Ġtext , Ġfor Ġinstance , Ġin Ġorder Ġto Ġmake Ġit Ġmore Ġneutral Ġand Ġavoid Ġgenerating Ġarbitrary Ġsubjective Ġjud gements . ĠWe Ġfind Ġthat Ġthe Ġnews Ġarticle Ġcommonly Ġincludes Ġdescription Ġthat Ġis Ġnot Ġevident Ġfrom Ġthe Ġdata Ġ( e . g ., Ġsubjective Ġcharacteristics Ġof Ġthe Ġplayer Ġor Ġthe Ġshot ), Ġand Ġoften Ġmay Ġreflect Ġthe Ġreporter 's Ġviewpoint . ĠFor Ġinstance , Ġthe Ġreporter Ġmay Ġevaluate Ġan Ġevent Ġnegatively Ġby Ġwriting ĠPlayer ĠA Ġdid Ġnot Ġmanage Ġto Ġscore Ġmore Ġthan Ġone Ġgoal , Ġreflecting Ġexpectation Ġof Ġthe Ġplayer 's Ġperformance , Ġwhich Ġwe Ġwould Ġchange Ġinto Ġa Ġmore Ġobjective Ġform Ġsuch Ġas ĠPlayer ĠA Ġscored Ġone Ġgoal . ĠSimilarly , Ġword Ġchoices Ġbased Ġon Ġviewing Ġthe Ġgame Ġ( e . g ., Ġslap shot ) Ġare Ġchanged Ġto Ġmore Ġneutral Ġcounterparts . </s> After Ġremoving Ġthe Ġunin fer able Ġparts Ġof Ġa Ġsentence , Ġthe Ġremainder Ġis Ġoften Ġrelatively Ġshort , Ġin Ġwhich Ġcase Ġwe Ġsometimes Ġopt Ġto Ġreplace Ġthe Ġinformation Ġthat Ġwas Ġremoved Ġwith Ġreferences Ġto Ġother Ġappropriate Ġinformation Ġfrom Ġthe Ġgame Ġstatistics , Ġsuch Ġas Ġthe Ġtime Ġof Ġthe Ġevent Ġor Ġthe Ġstate Ġof Ġthe Ġgame . ĠThis Ġserves Ġto Ġmaintain Ġa Ġsimilar Ġsentence Ġand Ġclause Ġstructure Ġas Ġthat Ġof Ġthe Ġoriginal Ġcorpus . </s> In Ġthe Ġmajority Ġof Ġour Ġaligned Ġdata Ġone Ġtext Ġspan Ġrefers Ġto Ġone Ġgame Ġevent . ĠHowever , Ġin Ġsome Ġcases Ġthe Ġsame Ġind iv isible Ġspan Ġcan Ġdescribe Ġmultiple Ġevents , Ġmost Ġcommonly Ġof Ġthe Ġsame Ġtype Ġ( e . g ., ĠPlayer ĠA Ġscored Ġtwo Ġgoals , Ġor ĠTeam ĠA Ġreceived Ġa Ġtotal Ġof Ġeight Ġpenalties ). ĠIn Ġsuch Ġcases , Ġwe Ġproduce Ġmulti - event Ġalign ments , Ġwhere Ġall Ġrelated Ġevents Ġare Ġaligned Ġwith Ġa Ġsingle Ġtext Ġspan . ĠMulti - event Ġalign ments Ġsupport Ġthe Ġdevelopment Ġof Ġmore Ġadvanced Ġgeneration , Ġwhich Ġstands Ġto Ġproduce Ġmore Ġnatural - sounding Ġreports Ġin Ġterms Ġof Ġless Ġrepetition Ġand Ġmore Ġflexible Ġstruct uring . </s> Ice ĠHockey ĠNews ĠDat as et Ġ: :: ĠCorpus ĠStatistics </s> In ĠTable ĠT AB REF 10 , Ġwe Ġsummarize Ġthe Ġoverall Ġsize Ġstatistics Ġof Ġthe Ġfinal Ġice Ġhockey Ġcorpus Ġafter Ġthe Ġstatistics Ġand Ġnews Ġarticles Ġhave Ġbeen Ġautomatically Ġpaired , Ġand Ġevents Ġhave Ġbeen Ġmanually Ġaligned Ġwith Ġthe Ġtext . ĠIn Ġtotal , Ġ2 , 307 Ġgames Ġwere Ġmanually Ġchecked Ġ( 66 . 8 % Ġof Ġthe Ġpaired Ġcorpus ), Ġof Ġwhich Ġ2 , 134 Ġgames Ġwere Ġcorrectly Ġpaired Ġwith Ġthe Ġarticle Ġdescribing Ġthe Ġgame . ĠIn Ġthese Ġgames , Ġ12 , 251 Ġout Ġof Ġ36 , 09 7 Ġevents Ġ( 33 . 9 %) Ġwere Ġreferenced Ġin Ġthe Ġtext Ġand Ġsuccessfully Ġaligned . ĠEnd Ġresult Ġoccurs Ġin Ġnearly Ġall Ġnews Ġarticles Ġas Ġthe Ġfirst Ġevent , Ġwhereas Ġthe Ġgoal Ġevent Ġis Ġby Ġfar Ġthe Ġmost Ġfrequent Ġone , Ġeach Ġgame Ġmentioning Ġon Ġaverage Ġ3 . 1 Ġgoals . ĠThe Ġnumber Ġof Ġaligned Ġevents Ġis Ġgreater Ġthan Ġthe Ġnumber Ġof Ġaligned Ġtext Ġspans Ġdue Ġto Ġmulti - event Ġalign ments . ĠWhile Ġ82 . 1 % Ġof Ġtext Ġspans Ġalign Ġwith Ġa Ġsingle Ġevent , Ġthere Ġis Ġa Ġlong Ġtail Ġof Ġmulti - event Ġalign ments , Ġwith Ġ11 . 2 % Ġalign ing Ġto Ġtwo Ġevents , Ġ3 . 4 % Ġto Ġthree , Ġ1 . 4 % Ġto Ġfour , Ġetc . </s> In ĠTable ĠT AB REF 11 Ġwe Ġmeasure Ġthe Ġlex ical Ġdiversity Ġof Ġoriginal Ġice Ġhockey Ġnews Ġarticles , Ġas Ġwell Ġas Ġthe Ġresulting Ġdataset Ġafter Ġmanual Ġalignment , Ġby Ġcomputing Ġthe ĠStandard ized ĠType âĢĵ Token ĠRatio Ġ( ST TR ). ĠThe Ġmeasure Ġis Ġdefined Ġas Ġthe Ġnumber Ġof Ġunique Ġtokens Ġor Ġle mm as Ġdivided Ġby Ġthe Ġtotal Ġnumber Ġof Ġtokens , Ġcalculated Ġon Ġevery Ġsegment Ġof Ġ1 , 000 Ġtokens Ġseparately Ġand Ġaveraged Ġacross Ġthe Ġcorpus . ĠST TR Ġis Ġmore Ġmeaningful Ġthan Ġthe Ġstandard Ġtype âĢĵ token Ġratio Ġwhen Ġcomparing Ġcorpor a Ġof Ġsubstantially Ġdifferent Ġsizes . ĠBoth Ġcorpor a Ġare Ġtoken ized Ġand Ġle mm at ized Ġusing Ġthe ĠTurk u ĠNeural ĠPars er Ġpipeline ĠB IB REF 7 , ĠB IB REF 8 . ĠST TR Ġof Ġthe Ġaligned Ġcorpus Ġis Ġlower Ġthan Ġin Ġthe Ġoriginal Ġhockey Ġnews Ġon Ġboth Ġword Ġand Ġle mma Ġlevel , Ġindicating Ġa Ġsomewhat âĢĶ but Ġnot Ġsubstantially Ġso âĢĶ more Ġrestricted Ġvocabulary Ġuse Ġin Ġour Ġaligned Ġsubset . </s> Event ĠSelection ĠModel </s> As Ġillustrated Ġpreviously , Ġany Ġgiven Ġnews Ġarticle Ġdescribes Ġonly Ġa Ġsubset Ġof Ġmost Ġnoteworthy Ġevents Ġof Ġa Ġgame . ĠWe Ġobserve Ġthat Ġmost Ġreports Ġare Ġconcise , Ġreferencing Ġon Ġaverage Ġ5 . 7 Ġevents . ĠThe Ġdistribution Ġof Ġevents / report Ġis : Ġ1 st Ġquant ile Ġat Ġ3 Ġevents Ġ( 20 . 9 % Ġof Ġevents Ġin Ġgame ), Ġ2 nd Ġat Ġ5 Ġ( 22 . 2 %), Ġ3 rd Ġat Ġ7 Ġ( 38 . 5 %), Ġ4 th Ġat Ġ36 Ġ( 100 %). </s> Our Ġalignment Ġserves Ġas Ġa Ġgold Ġstandard Ġreflecting Ġwhich Ġevents Ġthe Ġjournalists Ġhave Ġchosen Ġto Ġmention Ġfor Ġeach Ġgame . ĠIn Ġour Ġgeneration Ġtask , Ġwe Ġare Ġpresented Ġwith Ġthe Ġproblem Ġof Ġselecting Ġappropriate Ġevents Ġfrom Ġthe Ġfull Ġgame Ġstatistics . ĠWe Ġuse Ġthe Ġgold Ġstandard Ġselection Ġduring Ġtraining Ġand Ġvalidation Ġof Ġthe Ġtext Ġgeneration Ġmodel , Ġas Ġwell Ġas Ġthe Ġautomatic Ġevaluation . ĠAs Ġwe Ġdeploy Ġour Ġtext Ġgeneration Ġmodel Ġfor Ġmanual Ġevaluation , Ġwe Ġuse Ġa ĠCond itional ĠRandom ĠField Ġ( CR F ) Ġmodel Ġto Ġpredict Ġwhich Ġevents Ġto Ġmention . </s> C asting Ġthe Ġevent Ġselection Ġproblem Ġas Ġa Ġsequence Ġlabeling Ġtask , Ġthe ĠCR F Ġmodel Ġtakes Ġas Ġinput Ġthe Ġfull Ġsequence Ġof Ġevents Ġin Ġone Ġgame Ġtogether Ġwith Ġassociated Ġfeatures Ġfor Ġeach Ġevent , Ġand Ġpredicts Ġa Ġbinary Ġlabel Ġfor Ġeach Ġevent . ĠWe Ġachieve Ġan Ġoverall ĠF - score Ġof Ġ67 . 1 % Ġon Ġthe Ġtest Ġset , Ġwhich Ġbroken Ġdown Ġby Ġevent Ġtype Ġis : Ġend Ġresult Ġ( 98 . 0 %), Ġgoal Ġ( 70 . 2 %), Ġpenalty Ġ( 20 . 1 %), Ġsave Ġ( 47 . 7 %). ĠPen alties Ġare Ġthe Ġmost Ġdifficult Ġto Ġpredict , Ġbeing Ġreported Ġonly Ġ7 . 8 % Ġof Ġthe Ġtime Ġin Ġreality , Ġe . g ., Ġcompared Ġto Ġ54 . 1 % Ġfor Ġgoals . </s> Text ĠGeneration </s> Next , Ġwe Ġpresent Ġthe Ġmodel Ġarchitecture Ġused Ġin Ġtext Ġgeneration , Ġand Ġevaluate Ġthe Ġmodel Ġon Ġa Ġpopular Ġbaseline Ġdataset . ĠAfter Ġthat , Ġwe Ġdescribe Ġthe Ġtraining Ġof Ġthe Ġgeneration Ġmodel Ġon Ġour Ġice Ġhockey Ġcorpus Ġand Ġuse Ġautomatic Ġevaluation Ġmetrics Ġto Ġcompare Ġagainst Ġexisting Ġreferences . </s> Text ĠGeneration Ġ: :: ĠModel ĠArchitecture </s> We Ġuse Ġa Ġpointer - generation Ġnetwork ĠB IB REF 10 , ĠB IB REF 11 , ĠB IB REF 12 , Ġwhere Ġthe Ġneural Ġattention Ġmechanism Ġin Ġthe Ġenc oder - dec oder Ġmodel Ġis Ġadapted Ġto Ġjointly Ġmodel Ġa Ġprobability Ġdistribution Ġover Ġwords Ġfrom Ġthe Ġknown Ġvocabulary , Ġa Ġdistribution Ġover Ġwords Ġfrom Ġthe Ġinput Ġsequence Ġto Ġcopy Ġand Ġa Ġprobability Ġthat Ġcontrols Ġthe Ġcopying Ġmechanism . ĠA Ġseparate Ġcoverage Ġattention Ġvector , Ġa Ġsum Ġof Ġpast Ġattention Ġdistributions , Ġis Ġmaintained Ġto Ġinform Ġthe Ġmodel Ġof Ġits Ġpast Ġattention Ġdecisions . ĠSuch Ġa Ġcoverage Ġmodel Ġis Ġshown Ġto Ġprevent Ġtext Ġrepetition Ġin Ġgenerated Ġoutput ĠB IB REF 13 , ĠB IB REF 12 . </s> The Ġmodel Ġis Ġimplemented Ġusing Ġthe ĠOpen N MT - py Ġlibrary ĠB IB REF 14 . ĠThe Ġenc oder Ġhas Ġtwo Ġbid irection al ĠL ST M Ġlayers Ġwith Ġ500 Ġhidden Ġunits , Ġtogether Ġwith Ġ500 - dimensional Ġword Ġembed d ings . ĠThe Ġdec oder Ġhas Ġtwo Ġun id irection al ĠL ST M Ġlayers Ġwith Ġ500 Ġhidden Ġunits . ĠBoth Ġenc oder Ġand Ġdec oder Ġapply Ġa Ġdrop out Ġof Ġ0 . 3 Ġbetween ĠL ST M Ġlayers . </s> Text ĠGeneration Ġ: :: ĠBas eline ĠExper iments Ġon Ġthe ĠE 2 E ĠDat as et </s> To Ġdemonstrate Ġthe Ġperformance Ġof Ġour Ġgeneration Ġmodel Ġarchitecture , Ġwe Ġreport Ġresults Ġon Ġa Ġknown Ġdataset Ġwith Ġpublished Ġbas elines , Ġnamely Ġthe ĠE 2 E ĠNL G ĠChallenge ĠB IB REF 2 Ġon Ġend - to - end Ġnatural Ġlanguage Ġgeneration Ġin Ġspoken Ġdialogue Ġsystems . ĠThe Ġtask Ġis Ġto Ġproduce Ġa Ġnatural Ġlanguage Ġdescription Ġof Ġa Ġrestaurant Ġbased Ġon Ġa Ġgiven Ġmeaning Ġrepresentation Ġ( MR )âĢĶ an Ġun ordered Ġset Ġof Ġattributes Ġand Ġtheir Ġvalues . ĠThe Ġattributes Ġincluded , Ġamong Ġothers , Ġthe Ġrestaurant Ġname , Ġarea , Ġfood Ġtype Ġand Ġrating . ĠWe Ġrepresent Ġthe Ġgiven ĠMR Ġas Ġa Ġsequence Ġof Ġtokens Ġwhere Ġeach Ġattribute Ġvalue Ġis Ġembedded Ġinto ĠXML - style Ġbeginning Ġand Ġend Ġattribute Ġmarkers , Ġand Ġthe Ġorder Ġof Ġattributes Ġis Ġkept Ġfixed Ġacross Ġthe Ġwhole Ġdataset . ĠThe Ġtarget Ġoutput Ġis Ġa Ġsequence Ġof Ġtokens . ĠWe Ġdo Ġnot Ġapply Ġany Ġexplicit Ġde lex ical ization Ġsteps . </s> In ĠTable ĠT AB REF 15 Ġwe Ġmeasure ĠB LE U ĠB IB REF 19 , ĠN IST ĠB IB REF 20 , ĠMET E OR ĠB IB REF 21 , ĠR OU GE - L ĠB IB REF 22 Ġand ĠC ID Er ĠB IB REF 23 Ġmetrics Ġon Ġthe Ġ2018 ĠE 2 E ĠNL G ĠChallenge Ġtest Ġdata Ġusing Ġthe Ġevaluation Ġscript Ġprovided Ġby Ġthe Ġorganizers . ĠOur Ġgeneration Ġsystem Ġis Ġcompared Ġto Ġthe Ġofficial Ġshared Ġtask Ġbaseline Ġsystem , ĠT Gen ĠB IB REF 24 , Ġas Ġwell Ġas Ġto Ġthe Ġtop Ġperforming Ġparticipant Ġsystem Ġon Ġeach Ġscore Ġ( ST Ġtop ). ĠOur Ġsystem Ġoutper forms Ġthe ĠT Gen Ġbaseline Ġon Ġ3 Ġout Ġof Ġ5 Ġmetrics Ġ( BLE U , ĠMET E OR Ġand ĠR OU GE - L ), Ġwhich Ġis Ġon Ġpar Ġwith Ġthe Ġofficial Ġshared Ġtask Ġresults , Ġwhere Ġnot Ġa Ġsingle Ġone Ġparticipant Ġsystem Ġwas Ġable Ġto Ġsurpass Ġthe Ġbaseline Ġon Ġall Ġfive Ġmetrics . ĠOn Ġtwo Ġmetrics , ĠB LE U Ġand ĠMET E OR , Ġour Ġsystem Ġoutper forms Ġthe Ġbest Ġshared Ġtask Ġparticipants . </s> E 2 E ĠNL G ĠChallenge Ġevaluation Ġis Ġbased Ġon Ġhaving Ġmultiple Ġreferences Ġfor Ġeach ĠMR , Ġon Ġaverage Ġeach Ġunique ĠMR Ġin Ġthe Ġcorpus Ġhaving Ġ8 Ġreference Ġdescriptions . ĠIn Ġthe Ġevaluation , Ġthe Ġoutput Ġfor Ġeach Ġunique ĠMR Ġis Ġcompared Ġagainst Ġall Ġreferences Ġand Ġthe Ġmaximum Ġscore Ġis Ġused , Ġnaturally Ġleading Ġto Ġhigher Ġscores . ĠTo Ġhave Ġmore Ġcomparable Ġnumbers Ġto Ġour Ġice Ġhockey Ġcorpus , Ġwhere Ġwe Ġhave Ġonly Ġone Ġreference Ġfor Ġeach Ġinput Ġevent , Ġwe Ġalso Ġinclude Ġscores Ġobtained Ġby Ġcomparing Ġeach ĠMR Ġto Ġeach Ġof Ġits Ġreference Ġdescriptions Ġseparately Ġas Ġif Ġthey Ġwere Ġindividual Ġdata Ġpoints Ġ( O urs Ġsingle Ġref .). </s> Text ĠGeneration Ġ: :: ĠHockey ĠData ĠRepresent ation </s> Currently , Ġwe Ġconcentrate Ġon Ġtraining Ġthe Ġmodel Ġonly Ġwith Ġtext Ġspans Ġalign ing Ġwith Ġsingle Ġevents , Ġexcluding Ġthe Ġless Ġfrequent Ġmulti - event Ġalign ments . ĠFurthermore , Ġwe Ġare Ġconsidering Ġeach Ġevent Ġas Ġa Ġseparate Ġtraining Ġexample , Ġindependent Ġof Ġother Ġevents Ġin Ġthe Ġgame . </s> Given Ġa Ġsingle Ġevent Ġdescribed Ġas Ġa Ġsequence Ġof Ġfeatures Ġand Ġtheir Ġvalues , Ġour Ġtext Ġgeneration Ġmodel Ġis Ġtrained Ġto Ġproduce Ġthe Ġtext Ġspan Ġaligned Ġwith Ġit . ĠFollowing Ġthe Ġdata Ġrepresentation Ġused Ġin ĠE 2 E ĠNL G ĠChallenge Ġexperiments , Ġthe Ġinput Ġevents Ġare Ġrepresented Ġas Ġa Ġlinear ized Ġsequence Ġof Ġtokens , Ġwhere ĠXML - style Ġbeginning Ġand Ġend Ġtags Ġare Ġused Ġto Ġseparate Ġthe Ġdifferent Ġfeatures Ġ( see ĠFigure ĠFIG REF 18 ). ĠThis Ġallows Ġthe Ġmodel Ġto Ġdirectly Ġcopy Ġsome Ġof Ġthe Ġinput Ġtokens Ġto Ġthe Ġoutput Ġwhen Ġnecessary . ĠThe Ġability Ġto Ġcopy Ġtokens Ġis Ġespecially Ġimportant Ġwith Ġplayer Ġnames Ġand Ġexact Ġtimes , Ġwhere Ġthe Ġvocabulary Ġis Ġsparse , Ġand Ġmany Ġof Ġthese Ġcan Ġeven Ġbe Ġpreviously Ġunseen , Ġunknown Ġtokens . ĠIn Ġaddition , Ġwe Ġalso Ġinclude Ġfeatures Ġthat Ġare Ġmeant Ġto Ġinform Ġgeneration Ġwithout Ġbeing Ġcopied Ġthemselves , Ġfor Ġexample , Ġthe Ġtype Ġof Ġthe Ġevent . ĠThe Ġtarget Ġof Ġgeneration Ġis Ġa Ġtoken ized Ġsequence Ġof Ġwords , Ġwhere Ġalso Ġd ashes Ġinside Ġscores Ġare Ġseparated , Ġallowing Ġthe Ġmodel Ġto Ġswap Ġscores Ġwhen Ġnecessary . ĠThe Ġreference Ġtext Ġsometimes Ġflips Ġthe Ġorder Ġof Ġthe Ġteams , Ġrequiring Ġthe Ġscore Ġto Ġbe Ġinverted Ġas Ġwell Ġ( Team ĠA âĢĵ Team ĠB Ġ0 âĢĵ 1 Ġinto ĠTeam ĠB Ġwon ĠTeam ĠA Ġ1 âĢĵ 0 ). </s> One Ġparticular Ġchallenge Ġin Ġour Ġcorpus Ġis Ġthat Ġthe Ġdecision Ġof Ġwhich Ġaspects Ġto Ġfocus Ġon , Ġi . e ., Ġwhich Ġparticular Ġfeatures Ġfrom Ġthe Ġsource Ġevent Ġto Ġverbal ize , Ġis Ġrelatively Ġarbitrary . ĠFor Ġexample , Ġsometimes Ġa Ġjournalist Ġmentions Ġthe Ġplayer Ġor Ġplayers Ġassisting Ġa Ġgoal , Ġbut Ġin Ġmany Ġcases Ġit Ġis Ġleft Ġout . ĠBoth Ġoptions Ġare Ġequally Ġcorrect , Ġbut Ġoverlap - based Ġmetrics Ġsuch Ġas ĠB LE U Ġpenal ize Ġsuch Ġcreative Ġvariation . ĠBy Ġcontrast , Ġin Ġother Ġtext Ġgeneration Ġdatasets Ġsuch Ġas Ġthe ĠE 2 E ĠNL G ĠChallenge , Ġthe Ġoutput Ġtext Ġin Ġgeneral Ġdescribes Ġall Ġinput Ġfeatures . ĠTo Ġaccount Ġfor Ġthis Ġvariation , Ġwe Ġinclude Ġa Ġlength Ġfeature Ġto Ġact Ġas Ġa Ġminimal Ġsupervision Ġsignal Ġfor Ġthe Ġmodel Ġto Ġrely Ġon . ĠWe Ġdivide Ġoutput Ġtext Ġlengths Ġinto Ġthree Ġevenly Ġsized Ġcategories Ġ( short , Ġmedium Ġand Ġlong ) Ġto Ġprovide Ġa Ġhint Ġto Ġthe Ġmodel Ġduring Ġtraining Ġof Ġhow Ġlong Ġand Ġdetailed Ġoutput Ġit Ġis Ġexpected Ġto Ġgenerate Ġfor Ġeach Ġtraining Ġexample . ĠAt Ġtest Ġtime , Ġwe Ġthen Ġhave Ġthe Ġpossibility Ġto Ġcontrol Ġthe Ġdesired Ġapproximate Ġlength Ġof Ġthe Ġgenerated Ġtext . ĠIn Ġthe Ġexperiments Ġthroughout Ġthis Ġpaper , Ġwe Ġgenerate Ġall Ġthree Ġlength Ġvariants Ġfor Ġeach Ġevent Ġand Ġpick Ġthe Ġone Ġwith Ġthe Ġhighest Ġaverage Ġconfidence Ġof Ġthe Ġgeneration Ġmodel . </s> Text ĠGeneration Ġ: :: ĠTraining Ġand ĠOptim ization </s> The Ġmodel Ġis Ġtrained Ġusing Ġthe ĠAdam Ġoptim izer Ġwith Ġlearning Ġrate Ġof Ġ0 . 000 5 Ġand Ġbatch Ġsize Ġof Ġ32 . ĠThe Ġmodel Ġis Ġtrained Ġfor Ġa Ġmaximum Ġof Ġ8000 Ġsteps Ġ( ca . Ġ40 Ġepoch s ), Ġand Ġthe Ġfinal Ġmodel Ġis Ġchosen Ġbased Ġon Ġvalidation Ġset Ġperformance . ĠWe Ġuse Ġ80 % Ġof Ġthe Ġaligned Ġgames Ġfor Ġtraining , Ġ10 % Ġfor Ġvalidation Ġand Ġ10 % Ġfor Ġtesting . </s> In Ġour Ġinitial Ġexperiments , Ġwe Ġused Ġthe ĠRB FO pt Ġlibrary ĠB IB REF 25 Ġfor Ġhyper param eter Ġtuning , Ġmaximizing Ġvalidation Ġset ĠB LE U Ġscore . ĠHowever , Ġwhen Ġmanually Ġinspecting Ġgeneration Ġresults Ġon Ġthe Ġvalidation Ġset , Ġwe Ġnoticed Ġthat Ġmodels Ġwith Ġa Ġhigher ĠB LE U Ġscore Ġresult Ġin Ġmore Ġfluent Ġtext Ġbut Ġgenerate Ġmore Ġfactual Ġmistakes . ĠThis Ġobservation Ġis Ġsupported Ġby ĠB IB REF 3 , Ġwho Ġnote Ġthat ĠB LE U Ġscore Ġtends Ġto Ġreward Ġfluent Ġtext Ġrather Ġthan Ġother Ġaspects Ġdesirable Ġin Ġgeneration Ġof Ġsports Ġsumm aries Ġfrom Ġdata . ĠFor Ġthis Ġreason , Ġwe Ġultimately Ġdecided Ġto Ġuse Ġhyper param eters Ġmanually Ġtuned Ġon Ġthe Ġvalidation Ġset Ġto Ġgive Ġa Ġgood Ġperceived Ġbalance Ġbetween Ġfluent Ġand Ġfact ually Ġcorrect Ġtext . </s> Text ĠGeneration Ġ: :: ĠAutomatic ĠEvaluation Ġof ĠHockey ĠGeneration </s> In ĠTable ĠT AB REF 21 , Ġwe Ġprovide Ġevaluation Ġresults Ġusing Ġthe Ġfive Ġaforementioned Ġmetrics . ĠWe Ġevaluate Ġon Ġevent Ġlevel Ġusing Ġgold Ġstandard Ġevent Ġselection , Ġwhere Ġeach Ġgenerated Ġevent Ġdescription Ġis Ġcompared Ġto Ġits Ġexisting Ġreference Ġtext . ĠAs Ġthe Ġmodel Ġis Ġtrained Ġto Ġproduce Ġa Ġtoken ized Ġsequence , Ġwe Ġapply Ġa Ġdet oken izer Ġto Ġbe Ġable Ġto Ġcompare Ġagainst Ġthe Ġoriginal Ġunt oken ized Ġreference Ġtext . ĠOn Ġthe Ġtest Ġset , Ġthe Ġmodel Ġachieves Ġa ĠB LE U Ġscore Ġof Ġ19 . 67 . ĠTo Ġthe Ġextent Ġthat Ġdifferent Ġdatasets Ġallow Ġcomparison , Ġthe Ġbest Ġreported Ġscore Ġon Ġthe ĠRot ow ire Ġbasketball Ġnews Ġcorpus Ġis Ġ16 . 50 ĠB IB REF 5 . ĠCompared Ġto Ġour Ġearlier ĠE 2 E Ġbaseline Ġexperiment , Ġwe Ġscore Ġlower Ġthan Ġour Ġclosest Ġcomparable Ġreference Ġof Ġ31 . 90 Ġ( with Ġsingle Ġreferences ), Ġwhich Ġis Ġunderstandable Ġdue Ġto Ġthe Ġmuch Ġsmaller Ġtrain Ġset Ġsize Ġfor Ġthe Ġhockey Ġcorpus Ġ( about Ġ13 % Ġin Ġsize ). </s> In ĠFigure ĠFIG REF 22 , Ġwe Ġplot Ġthe Ġlearning Ġcurve Ġwith Ġincreasing Ġsizes Ġof Ġtraining Ġdata Ġin Ġorder Ġto Ġillustrate Ġhow Ġgeneration Ġperformance Ġbenefits Ġfrom Ġmore Ġdata . ĠThe Ġlearning Ġcurve Ġis Ġstill Ġsteadily Ġincreasing Ġwhen Ġusing Ġ100 % Ġof Ġthe Ġtraining Ġdata Ġcurrently Ġavailable , Ġwhich Ġindicates Ġthat Ġmore Ġdata Ġwould Ġmost Ġlikely Ġfurther Ġimprove Ġthe Ġperformance . </s> Human Ġevaluation </s> As Ġour Ġobjective Ġis Ġpractically Ġusable Ġnews Ġgeneration , Ġwe Ġcarry Ġout Ġa Ġmanual Ġevaluation Ġof Ġits Ġoutput Ġon Ġ59 Ġrandomly Ġselected Ġgames Ġfrom Ġthe Ġtest Ġset , Ġfocusing Ġin Ġparticular Ġon Ġthe Ġcorrections Ġthat Ġwould Ġbe Ġnecessary Ġto Ġobtain Ġacceptable Ġoutput . ĠExample Ġcorrections Ġare Ġshown Ġin ĠFigure ĠFIG REF 27 . ĠThese Ġfull Ġgame Ġreports Ġare Ġgenerated Ġby Ġfirst Ġapplying Ġthe Ġselection Ġmodel Ġdescribed Ġin ĠSection ĠSEC REF 3 Ġto Ġselect Ġthe Ġevents Ġto Ġbe Ġincluded Ġin Ġthe Ġreport , Ġand Ġthen Ġusing Ġthe Ġtext Ġgeneration Ġmodel Ġto Ġverbal ize Ġeach Ġselected Ġevent . ĠThe Ġgenerated Ġtexts Ġof Ġthe Ġevents Ġare Ġthen Ġdet oken ized Ġand Ġconc aten ated Ġin Ġchronological Ġorder Ġinto Ġa Ġfull Ġgame Ġreport . </s> Human Ġevaluation Ġ: :: ĠMinimum ĠEdit ĠEvaluation </s> In Ġthe Ġminimum Ġedit Ġevaluation , Ġcarried Ġout Ġby Ġthe Ġannot ator Ġwho Ġcreated Ġthe Ġnews Ġcorpus , Ġonly Ġfactual Ġmistakes Ġand Ġgram matical Ġerrors Ġare Ġcorrected , Ġresulting Ġin Ġtext Ġwhich Ġmay Ġremain Ġawkward Ġor Ġunf luent . ĠThe Ġword Ġerror Ġrate Ġ( WER ) Ġof Ġthe Ġgenerated Ġtext Ġcompared Ġto Ġits Ġcorrected Ġvariant Ġas Ġa Ġreference Ġis Ġ5 . 6 % Ġ( 6 . 2 % Ġdisreg arding Ġpunct uation ). ĠThe ĠW ER Ġmeasure Ġis Ġdefined Ġas Ġthe Ġnumber Ġof Ġinsert ions , Ġsubstit utions , Ġand Ġdelet ions Ġdivided Ġby Ġthe Ġtotal Ġlength Ġof Ġthe Ġreference , Ġin Ġterms Ġof Ġtokens . ĠThe Ġmeasure Ġis Ġthe Ġedit Ġdistance Ġof Ġthe Ġgenerated Ġtext Ġand Ġits Ġcorrected Ġvariant , Ġdirectly Ġreflecting Ġthe Ġamount Ġof Ġeffort Ġneeded Ġto Ġcorrect Ġthe Ġgenerated Ġoutput . </s> Human Ġevaluation Ġ: :: ĠMinimum ĠEdit ĠEvaluation Ġ: :: ĠFact ual ĠCorrect ness </s> The Ġfactual Ġerrors Ġand Ġtheir Ġtypes Ġare Ġsummarized Ġin ĠTable ĠT AB REF 23 . ĠFrom Ġthe Ġtotal Ġof Ġ510 Ġgame Ġevents Ġgenerated Ġby Ġthe Ġsystem , Ġ78 Ġof Ġthese Ġcontained Ġa Ġfactual Ġerror , Ġi . e . Ġ84 . 7 % Ġwere Ġgenerated Ġwithout Ġfactual Ġerrors . </s> The Ġmost Ġnotable Ġerrors Ġinvolved Ġplayer Ġor Ġteam Ġnames . ĠThe Ġinput Ġfor Ġa Ġsingle Ġgame Ġevent Ġmay Ġcontain Ġmore Ġthan Ġone Ġname Ġ( e . g . Ġthe Ġgoal Ġscorer Ġand Ġup Ġto Ġtwo Ġassisting Ġplayers ). ĠIn Ġthese Ġcases , Ġthe Ġmodel Ġoccasionally Ġpaired Ġfirst Ġname Ġand Ġsurname Ġincorrectly . ĠLess Ġfrequent Ġerrors Ġinclude Ġwrong Ġteam Ġpair ings Ġwhen Ġreferring Ġto Ġthe Ġend Ġresult Ġof Ġthe Ġgame , Ġe . g ., ĠTeam ĠA Ġlost Ġto ĠTeam ĠA . </s> In Ġsentences Ġwhere Ġthe Ġexact Ġtime Ġof Ġa Ġgame Ġevent Ġis Ġgenerated Ġas Ġa Ġnumber Ġ( at Ġtime Ġ39 . 54 ) Ġthe Ġmodel Ġcopied Ġthe Ġtime Ġreliably , Ġbut Ġwhen Ġthe Ġapproximate Ġtime Ġof Ġan Ġevent Ġis Ġverbal ized Ġ( at Ġthe Ġend Ġof Ġthe Ġsecond Ġperiod ) Ġthere Ġwere Ġoccasional Ġerrors . ĠAnother Ġnotable Ġerror Ġcategory Ġis Ġtypes Ġof Ġgoals Ġor Ġtheir Ġscores Ġ( e . g . Ġ3 âĢĵ 0 , Ġdeciding Ġgoal , Ġtying Ġgoal ). ĠIn Ġthis Ġcategory , Ġthe Ġerror Ġsimilarly Ġoccurred Ġin Ġcases Ġwhen Ġthe Ġreference Ġis Ġverbal Ġ( third Ġgoal ), Ġbut Ġoccasionally Ġalso Ġin Ġnumerical Ġrepresentation Ġ( 3 âĢĵ 0 Ġinstead Ġof Ġ0 âĢĵ 3 ). ĠThe Ġother , Ġless Ġcommon Ġcategories Ġrelate Ġto Ġgame Ġtotal Ġscores , Ġpower Ġplay , Ġassists Ġand Ġpenalties . </s> Human Ġevaluation Ġ: :: ĠMinimum ĠEdit ĠEvaluation Ġ: :: ĠFlu ency </s> Overall , Ġthe Ġgenerated Ġtext Ġis Ġhighly Ġgram matical . ĠThe Ġmost Ġfrequent Ġgram matical Ġerror Ġin Ġthe Ġoutput Ġis Ġuncon jug ated Ġplayer Ġnames ; Ġcommonly Ġnames Ġin Ġtheir ĠFinnish Ġnom in ative Ġcase Ġthat Ġshould Ġbe Ġin Ġgen itive . ĠIn Ġa Ġfew Ġcases , Ġthe Ġmodel Ġmakes Ġgram matical Ġerrors Ġwhen Ġit Ġcopies Ġan Ġincompatible Ġword Ġfrom Ġthe Ġinput , Ġe . g ., Ġa Ġname Ġinstead Ġof Ġa Ġtime Ġreference . </s> Both Ġerror Ġtypes Ġcommonly Ġoccur Ġwhen Ġthe Ġmodel Ġcopies Ġfrom Ġthe Ġinput . ĠAs Ġit Ġoperates Ġon Ġthe Ġtoken Ġrather Ġthan Ġsub - word Ġlevel , Ġit Ġis Ġchallenging Ġto Ġmap Ġbetween Ġword Ġforms Ġand Ġinf lect Ġinf requent Ġwords Ġsuch Ġas Ġplayer Ġnames . ĠThe Ġlatter Ġerror Ġlikely Ġoccurs Ġwhen Ġthe Ġcopy Ġattention Ġis Ġindec isive Ġand Ġfails Ġto Ġrecognize Ġthe Ġtype Ġfor Ġinf requent Ġtokens . </s> Most Ġflu ency Ġissues Ġrelate Ġto Ġthe Ġoverall Ġflow Ġand Ġstructure Ġof Ġthe Ġreport . ĠAdd ressing Ġthese Ġissues Ġwould Ġrequire Ġthe Ġmodel Ġto Ġtake Ġinto Ġaccount Ġmultiple Ġevents Ġin Ġa Ġgame , Ġand Ġcombine Ġthe Ġinformation Ġmore Ġflex ibly Ġto Ġavoid Ġrepetition . ĠFor Ġinstance , Ġthe Ġoutput Ġmay Ġrepeatedly Ġmention Ġthe Ġperiod Ġnumber Ġfor Ġall Ġgoals Ġin Ġthe Ġsame Ġperiod . ĠLikewise , Ġthis Ġsetup Ġsometimes Ġresults Ġin Ġunnatural , Ġyet Ġgram matical , Ġrepetition Ġof Ġwords Ġacross Ġconsecutive Ġsentences . ĠEven Ġthough Ġthe Ġmodel Ġhas Ġlearned Ġa Ġselection Ġof Ġverbs Ġmeaning Ġto Ġscore Ġa Ġgoal , Ġit Ġis Ġunable Ġto Ġensure Ġtheir Ġvaried Ġuse . ĠWhile Ġnot Ġsuccessful Ġin Ġour Ġinitial Ġexperiments , Ġgenerating Ġtext Ġbased Ġon Ġthe Ġmulti - event Ġalign ments Ġor Ġat Ġdocument Ġlevel Ġmay Ġeventually Ġovercome Ġthese Ġissues . </s> Human Ġevaluation Ġ: :: ĠProduct - Read iness ĠEvaluation </s> The Ġsecond Ġhuman Ġevaluation Ġaimed Ġat Ġjudging Ġthe Ġaccept ability Ġof Ġthe Ġoutput Ġfor Ġproduction Ġuse Ġin Ġa Ġnews Ġagency . ĠThe Ġoutput Ġis Ġevaluated Ġin Ġterms Ġof Ġits Ġusability Ġfor Ġa Ġnews Ġchannel Ġlabelled Ġas Ġbeing Ġmachine - generated , Ġi . e . Ġnot Ġaiming Ġat Ġthe Ġlevel Ġof Ġa Ġhuman Ġjournalist Ġequipped Ġwith Ġsubstantial Ġbackground Ġinformation . ĠThe Ġevaluation Ġwas Ġcarried Ġout Ġby Ġtwo Ġjournalists Ġfrom Ġthe ĠST T Ġagency , Ġwho Ġsplit Ġthe Ġ59 Ġgames Ġamong Ġthemselves Ġapproximately Ġevenly . ĠThe Ġfirst Ġjournalist Ġedited Ġthe Ġgames Ġto Ġa Ġform Ġcorresponding Ġto Ġa Ġdraft Ġfor Ġsubsequent Ġminor Ġpost - ed iting Ġby Ġa Ġhuman , Ġsim ulating Ġthe Ġuse Ġof Ġthe Ġgenerated Ġoutput Ġas Ġa Ġproduct Ġwhere Ġthe Ġfinal Ġcustomer Ġis Ġexpected Ġto Ġdo Ġown Ġpost - ed iting Ġbefore Ġpublication . ĠThe Ġsecond Ġjournalist Ġdirectly Ġedited Ġthe Ġnews Ġto Ġa Ġstate Ġready Ġfor Ġdirect Ġpublication Ġin Ġa Ġnews Ġstream Ġlabeled Ġas Ġmachine - generated Ġnews . ĠIn Ġaddition Ġto Ġcorrecting Ġfactual Ġerrors , Ġthe Ġjournalists Ġremoved Ġexcessive Ġrepetition , Ġimproved Ġtext Ġflu ency , Ġas Ġwell Ġas Ġoccasionally Ġincluded Ġimportant Ġfacts Ġwhich Ġthe Ġsystem Ġleft Ġun generated . ĠThe ĠW ER Ġmeasured Ġagainst Ġthe Ġoutput Ġconsidered Ġready Ġfor Ġpost - ed iting , Ġis Ġ9 . 9 % Ġ( 11 . 2 % Ġdisreg arding Ġpunct uation ), Ġonly Ġslightly Ġworse Ġthan Ġthe Ġevaluation Ġwith Ġonly Ġthe Ġfactual Ġand Ġgram matical Ġerrors Ġcorrected . ĠThe ĠW ER Ġmeasured Ġagainst Ġthe Ġoutput Ġconsidered Ġready Ġfor Ġdirect Ġrelease , Ġwas Ġ22 . 0 % Ġ( 24 . 4 % Ġdisreg arding Ġpunct uation ). ĠIn Ġother Ġwords , Ġ75 âĢĵ 90 % Ġof Ġthe Ġgenerated Ġtext Ġcan Ġbe Ġdirectly Ġused , Ġdepending Ġon Ġthe Ġexpected Ġpost - ed iting Ġeffort . </s> Figure ĠFIG REF 27 Ġshows Ġtwo Ġexample Ġgames Ġalong Ġwith Ġthe Ġgenerated Ġreports Ġand Ġmanual Ġcorrections Ġmade Ġby Ġthe Ġjournalist Ġin Ġorder Ġto Ġprepare Ġit Ġfor Ġpublication . ĠLit eral Ġtranslations Ġfrom Ġthe Ġgenerated , Ġunc orrect ed ĠFinnish Ġinto ĠEnglish Ġare Ġprovided Ġfor Ġreference . </s> Con clusions Ġand ĠFuture ĠWork </s> We Ġdeveloped Ġand Ġevaluated Ġan Ġend - to - end Ġsystem Ġfor Ġnews Ġgeneration Ġfrom Ġstructured Ġdata , Ġusing Ġa Ġcorpus Ġof Ġnews Ġand Ġgame Ġstatistics Ġin Ġthe Ġice Ġhockey Ġdomain . ĠIn Ġterms Ġof Ġthe Ġdata , Ġour Ġprimary Ġfinding Ġwas Ġthe Ġlevel Ġto Ġwhich Ġprofessionally Ġproduced Ġnews Ġcontain Ġinformation Ġthat Ġcannot Ġbe Ġinferred Ġfrom Ġthe Ġgame Ġstatistics . ĠThis Ġleads Ġto Ġthe Ġmodel Ġlearning Ġto Ġ" hall uc inate " Ġfacts Ġand Ġnecess itates Ġa Ġmanual Ġalignment Ġand Ġediting Ġof Ġthe Ġtraining Ġdata . ĠOnce Ġwe Ġcreated Ġa Ġsuitable Ġtraining Ġdataset , Ġwe Ġwere Ġable Ġto Ġgenerate Ġhighly Ġgram matical Ġtext Ġwhich , Ġin Ġterms Ġof Ġword Ġerror Ġrate Ġ( edit Ġdistance ), Ġwas Ġrelatively Ġclose Ġto Ġwhat Ġwas Ġjudged Ġas Ġa Ġviable Ġproduct Ġby Ġdomain Ġjournalists . ĠWe Ġfound Ġthat Ġmost Ġfactual Ġerrors Ġin Ġthe Ġgenerated Ġoutput Ġfall Ġinto Ġa Ġsmall Ġnumber Ġof Ġcategories , Ġmostly Ġrelated Ġto Ġcopying Ġnames Ġfrom Ġthe Ġinput , Ġtypes Ġof Ġevents , Ġand Ġtime Ġreferences . ĠAdd ressing Ġthese Ġerrors Ġis Ġa Ġmatter Ġof Ġfuture Ġwork Ġand Ġcan Ġbe Ġapproached Ġusing Ġdata Ġaug mentation Ġtechniques Ġas Ġwell Ġas Ġintroducing Ġsub - word Ġunits Ġwhich Ġwould Ġallow Ġthe Ġmodel Ġto Ġdeal Ġwith Ġinf lections . </s> Currently , Ġwe Ġonly Ġgenerate Ġthe Ġnews Ġas Ġindependent Ġevents , Ġwith Ġroughly Ġone Ġsentence Ġcorresponding Ġto Ġone Ġevent . ĠAs Ġthis Ġleads Ġto Ġa Ġsomewhat Ġunnatural Ġtext , Ġwe Ġhave Ġattempted Ġin Ġpreliminary Ġexperiments Ġto Ġgenerate Ġwhole Ġnews Ġtexts Ġat Ġonce , Ġas Ġwell Ġas Ġsentences Ġcombining Ġseveral Ġevents , Ġnevertheless Ġwith Ġresults Ġfar Ġfrom Ġuseful . ĠThis Ġis Ġlikely Ġdue Ġto Ġthe Ġrelatively Ġsmall Ġnumber Ġof Ġtraining Ġexamples Ġwhere Ġa Ġsingle Ġsentence Ġaccounts Ġfor Ġseveral Ġdistinct Ġevents . ĠWe Ġwill Ġfocus Ġon Ġthis Ġproblem Ġin Ġour Ġfuture Ġwork , Ġinvestigating Ġmethods Ġwhich Ġwould Ġallow Ġpre - training Ġthe Ġgeneration Ġmodel Ġso Ġas Ġto Ġbe Ġable Ġto Ġsuccessfully Ġaccept Ġseveral Ġevents Ġon Ġits Ġinput . </s> The Ġnew Ġdataset , Ġthe Ġoriginal Ġnews Ġcorpus Ġand Ġthe Ġsource Ġcode Ġof Ġthe Ġmodel Ġare Ġavailable Ġfor Ġresearch Ġuse . </s> Acknowled gments </s> We Ġgrate fully Ġacknowledge Ġthe Ġcollaboration Ġof ĠMai ja ĠPa ikk ala , ĠS alla ĠSal m ela Ġand ĠP ih la ĠLeh mus j oki Ġof Ġthe ĠFinnish ĠNews ĠAgency ĠST T , Ġas Ġwell Ġas Ġthe Ġsupport Ġof Ġthe ĠGoogle ĠDigital ĠNews ĠInnovation ĠFund , ĠAcademy Ġof ĠFinland , ĠC SC ĠâĢĵ ĠIT ĠCenter Ġfor ĠScience , Ġand Ġthe ĠNVIDIA ĠCorporation ĠGPU ĠGrant ĠProgram .