the objective of our model is sum of the two processes, jointly trained using "teacher-forcing" algorithm, we feed the ground-truth summary to each decoder and minimize the objective, At test time, each time step we choose the predicted word by $\hat{y} = argmax_{y^{\prime }} P(y^{\prime }|x)$ , use beam search to generate the draft summaries, and use greedy search to generate the refined summaries., the model can be trained end-to-end
EXTRACTIVE
the model can be trained end-to-end
EXTRACTIVE
1. We propose a natural language generation model based on BERT, making good use of the pre-trained language model in the encoder and decoder process, and the model can be trained end-to-end without handcrafted features. During model training, the objective of our model is sum of the two processes, jointly trained using "teacher-forcing" algorithm. During training we feed the ground-truth summary to each decoder and minimize the objective. $$L_{model} = \hat{L}_{dec} + \hat{L}_{refine}$$ (Eq. 23) At test time, each time step we choose the predicted word by $\hat{y} = argmax_{y^{\prime }} P(y^{\prime }|x)$ , use beam search to generate the draft summaries, and use greedy search to generate the refined summaries.
1. We propose a natural language generation model based on BERT, making good use of the pre-trained language model in the encoder and decoder process, and the model can be trained end-to-end without handcrafted features.
1902.09243
Introduction </s> Text Ġsummar ization Ġgenerates Ġsumm aries Ġfrom Ġinput Ġdocuments Ġwhile Ġkeeping Ġsalient Ġinformation . ĠIt Ġis Ġan Ġimportant Ġtask Ġand Ġcan Ġbe Ġapplied Ġto Ġseveral Ġreal - world Ġapplications . ĠMany Ġmethods Ġhave Ġbeen Ġproposed Ġto Ġsolve Ġthe Ġtext Ġsummar ization Ġproblem ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ. ĠThere Ġare Ġtwo Ġmain Ġtext Ġsummar ization Ġtechniques : Ġextract ive Ġand Ġabstract ive . ĠExtract ive Ġsummar ization Ġgenerates Ġsummary Ġby Ġselecting Ġsalient Ġsentences Ġor Ġphrases Ġfrom Ġthe Ġsource Ġtext , Ġwhile Ġabstract ive Ġmethods Ġparaph rase Ġand Ġrest ructure Ġsentences Ġto Ġcompose Ġthe Ġsummary . ĠWe Ġfocus Ġon Ġabstract ive Ġsummar ization Ġin Ġthis Ġwork Ġas Ġit Ġis Ġmore Ġflexible Ġand Ġthus Ġcan Ġgenerate Ġmore Ġdiverse Ġsumm aries . </s> Recently , Ġmany Ġabstract ive Ġapproaches Ġare Ġintroduced Ġbased Ġon Ġneural Ġsequence - to - sequence Ġframework ĠB IB REF 4 Ġ, ĠB IB REF 0 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 5 Ġ. ĠBased Ġon Ġthe Ġsequence - to - sequence Ġmodel Ġwith Ġcopy Ġmechanism ĠB IB REF 6 Ġ, ĠB IB REF 0 Ġincorporates Ġa Ġcoverage Ġvector Ġto Ġtrack Ġand Ġcontrol Ġattention Ġscores Ġon Ġsource Ġtext . ĠB IB REF 4 Ġintroduce Ġintra - tem poral Ġattention Ġprocesses Ġin Ġthe Ġenc oder Ġand Ġdec oder Ġto Ġaddress Ġthe Ġrepetition Ġand Ġinco herent Ġproblem . </s> There Ġare Ġtwo Ġissues Ġin Ġprevious Ġabstract ive Ġmethods : Ġ1 ) Ġthese Ġmethods Ġuse Ġleft - context - only Ġdec oder , Ġthus Ġdo Ġnot Ġhave Ġcomplete Ġcontext Ġwhen Ġpredicting Ġeach Ġword . Ġ2 ) Ġthey Ġdo Ġnot Ġutilize Ġthe Ġpre - trained Ġcontextual ized Ġlanguage Ġmodels Ġon Ġthe Ġdec oder Ġside , Ġso Ġit Ġis Ġmore Ġdifficult Ġfor Ġthe Ġdec oder Ġto Ġlearn Ġsummary Ġrepresentations , Ġcontext Ġinteractions Ġand Ġlanguage Ġmodeling Ġtogether . </s> Recently , ĠB ERT Ġhas Ġbeen Ġsuccessfully Ġused Ġin Ġvarious Ġnatural Ġlanguage Ġprocessing Ġtasks , Ġsuch Ġas Ġtextual Ġentail ment , Ġname Ġentity Ġrecognition Ġand Ġmachine Ġreading Ġcomprehens ions . ĠIn Ġthis Ġpaper , Ġwe Ġpresent Ġa Ġnovel Ġnatural Ġlanguage Ġgeneration Ġmodel Ġbased Ġon Ġpre - trained Ġlanguage Ġmodels Ġ( we Ġuse ĠB ERT Ġin Ġthis Ġwork ). ĠAs Ġfar Ġas Ġwe Ġknow , Ġthis Ġis Ġthe Ġfirst Ġwork Ġto Ġextend ĠB ERT Ġto Ġthe Ġsequence Ġgeneration Ġtask . ĠTo Ġaddress Ġthe Ġabove Ġissues Ġof Ġprevious Ġabstract ive Ġmethods , Ġin Ġour Ġmodel , Ġwe Ġdesign Ġa Ġtwo - stage Ġdecoding Ġprocess Ġto Ġmake Ġgood Ġuse Ġof ĠB ERT 's Ġcontext Ġmodeling Ġability . ĠOn Ġthe Ġfirst Ġstage , Ġwe Ġgenerate Ġthe Ġsummary Ġusing Ġa Ġleft - context - only - dec oder . ĠOn Ġthe Ġsecond Ġstage , Ġwe Ġmask Ġeach Ġword Ġof Ġthe Ġsummary Ġand Ġpredict Ġthe Ġrefined Ġword Ġone - by - one Ġusing Ġa Ġrefine Ġdec oder . ĠTo Ġfurther Ġimprove Ġthe Ġnatural ness Ġof Ġthe Ġgenerated Ġsequence , Ġwe Ġcooperate Ġreinforcement Ġobjective Ġwith Ġthe Ġrefine Ġdec oder . </s> The Ġmain Ġcontributions Ġof Ġthis Ġwork Ġare : </s> 1 . ĠWe Ġpropose Ġa Ġnatural Ġlanguage Ġgeneration Ġmodel Ġbased Ġon ĠB ERT , Ġmaking Ġgood Ġuse Ġof Ġthe Ġpre - trained Ġlanguage Ġmodel Ġin Ġthe Ġenc oder Ġand Ġdec oder Ġprocess , Ġand Ġthe Ġmodel Ġcan Ġbe Ġtrained Ġend - to - end Ġwithout Ġhand crafted Ġfeatures . </s> 2 . ĠWe Ġdesign Ġa Ġtwo - stage Ġdec oder Ġprocess . ĠIn Ġthis Ġarchitecture , Ġour Ġmodel Ġcan Ġgenerate Ġeach Ġword Ġof Ġthe Ġsummary Ġconsidering Ġboth Ġsides ' Ġcontext Ġinformation . </s> 3 . ĠWe Ġconduct Ġexperiments Ġon Ġthe Ġbenchmark Ġdatasets ĠCNN / Daily ĠMail Ġand ĠNew ĠYork ĠTimes . ĠOur Ġmodel Ġachieves Ġa Ġ33 . 33 Ġaverage Ġof ĠR OU GE - 1 , ĠR OU GE - 2 Ġand ĠR OU GE - L Ġon Ġthe ĠCNN / Daily ĠMail , Ġwhich Ġis Ġstate - of - the - art . ĠOn Ġthe ĠNew ĠYork ĠTimes Ġdataset , Ġour Ġmodel Ġachieves Ġabout Ġ5 . 6 % Ġrelative Ġimprovement Ġover ĠR OU GE - 1 . </s> Text ĠSum mar ization </s> In Ġthis Ġpaper , Ġwe Ġfocus Ġon Ġsingle - document Ġmulti - sent ence Ġsummar ization Ġand Ġpropose Ġa Ġsupervised Ġabstract ive Ġmodel Ġbased Ġon Ġthe Ġneural Ġattentive Ġsequence - to - sequence Ġframework Ġwhich Ġconsists Ġof Ġtwo Ġparts : Ġa Ġneural Ġnetwork Ġfor Ġthe Ġenc oder Ġand Ġanother Ġnetwork Ġfor Ġthe Ġdec oder . ĠThe Ġenc oder Ġenc odes Ġthe Ġinput Ġsequence Ġto Ġintermediate Ġrepresentation Ġand Ġthe Ġdec oder Ġpredicts Ġone Ġword Ġat Ġa Ġtime Ġstep Ġgiven Ġthe Ġinput Ġsequence Ġrepresentation Ġvector Ġand Ġprevious Ġdec oded Ġoutput . ĠThe Ġgoal Ġof Ġthe Ġmodel Ġis Ġto Ġmaximize Ġthe Ġprobability Ġof Ġgenerating Ġthe Ġcorrect Ġtarget Ġsequences . ĠIn Ġthe Ġencoding Ġand Ġgeneration Ġprocess , Ġthe Ġattention Ġmechanism Ġis Ġused Ġto Ġconcentrate Ġon Ġthe Ġmost Ġimportant Ġpositions Ġof Ġtext . ĠThe Ġlearning Ġobjective Ġof Ġmost Ġsequence - to - sequence Ġmodels Ġis Ġto Ġminimize Ġthe Ġnegative Ġlog Ġlikelihood Ġof Ġthe Ġgenerated Ġsequence Ġas Ġfollowing Ġequation Ġshows , Ġwhere Ġ$ y ^ * _ i $ Ġis Ġthe Ġi - th Ġground - truth Ġsummary Ġtoken . </s> $$ L oss Ġ= Ġ- Ġ\ log Ġ\ sum Ġ_ { t = 1 } ^ N ĠP ( y _ t ^ * | y _{ < t } ^ *, ĠX ) $$ Ġ Ġ Ġ( E q . Ġ3 ) </s> However , Ġwith Ġthis Ġobjective , Ġtraditional Ġsequence Ġgeneration Ġmodels Ġconsider Ġonly Ġone Ġdirection Ġcontext Ġin Ġthe Ġdecoding Ġprocess , Ġwhich Ġcould Ġcause Ġperformance Ġdegradation Ġsince Ġcomplete Ġcontext Ġof Ġone Ġtoken Ġcontains Ġpreceding Ġand Ġfollowing Ġtokens , Ġthus Ġfeeding Ġonly Ġpreceded Ġdec oded Ġwords Ġto Ġthe Ġdec oder Ġso Ġthat Ġthe Ġmodel Ġmay Ġgenerate Ġunnatural Ġsequences . ĠFor Ġexample , Ġattentive Ġsequence - to - sequence Ġmodels Ġoften Ġgenerate Ġsequences Ġwith Ġrepeated Ġphrases Ġwhich Ġharm Ġthe Ġnatural ness . ĠSome Ġprevious Ġworks Ġmitigate Ġthis Ġproblem Ġby Ġimproving Ġthe Ġattention Ġcalculation Ġprocess , Ġbut Ġin Ġthis Ġpaper Ġwe Ġshow Ġthat Ġfeeding Ġbi - direction al Ġcontext Ġinstead Ġof Ġleft - only - context Ġcan Ġbetter Ġalleviate Ġthis Ġproblem . </s> Text Ġsummar ization Ġmodels Ġare Ġusually Ġclassified Ġto Ġabstract ive Ġand Ġextract ive Ġones . ĠRecently , Ġextract ive Ġmodels Ġlike ĠDeep Channel ĠB IB REF 8 Ġ, Ġr nn - ext + RL ĠB IB REF 9 Ġand ĠNe u S UM ĠB IB REF 2 Ġachieve Ġhigher Ġperformances Ġusing Ġwell - designed Ġstructures . ĠFor Ġexample , ĠDeep Channel Ġpropose Ġa Ġsal ience Ġestimation Ġnetwork Ġand Ġiter atively Ġextract Ġsalient Ġsentences . ĠB IB REF 16 Ġtrain Ġa Ġsentence Ġcompression Ġmodel Ġto Ġteach Ġanother Ġlatent Ġvariable Ġextract ive Ġmodel . </s> Also , Ġseveral Ġrecent Ġworks Ġfocus Ġon Ġimproving Ġabstract ive Ġmethods . ĠB IB REF 3 Ġdesign Ġa Ġcontent Ġselector Ġto Ġover - d eter mine Ġphrases Ġin Ġa Ġsource Ġdocument Ġthat Ġshould Ġbe Ġpart Ġof Ġthe Ġsummary . ĠB IB REF 11 Ġintroduce Ġinconsistency Ġloss Ġto Ġforce Ġwords Ġin Ġless Ġattended Ġsentences ( which Ġdetermined Ġby Ġextract ive Ġmodel ) Ġto Ġhave Ġlower Ġgeneration Ġprobabilities . ĠB IB REF 5 Ġextend Ġseq 2 seq Ġmodel Ġwith Ġan Ġinformation Ġselection Ġnetwork Ġto Ġgenerate Ġmore Ġinformative Ġsumm aries . </s> Bi - D irection al ĠPre - Tr ained ĠContext ĠEnc od ers </s> Recently , Ġcontext Ġenc od ers Ġsuch Ġas ĠEL Mo , ĠG PT , Ġand ĠB ERT Ġhave Ġbeen Ġwidely Ġused Ġin Ġmany ĠN LP Ġtasks . ĠThese Ġmodels Ġare Ġpre - trained Ġon Ġa Ġhuge Ġunl abel ed Ġcorpus Ġand Ġcan Ġgenerate Ġbetter Ġcontextual ized Ġtoken Ġembed d ings , Ġthus Ġthe Ġapproaches Ġbuilt Ġon Ġtop Ġof Ġthem Ġcan Ġachieve Ġbetter Ġperformance . </s> Since Ġour Ġmethod Ġis Ġbased Ġon ĠB ERT , Ġwe Ġillustrate Ġthe Ġprocess Ġbriefly Ġhere . ĠB ERT Ġconsists Ġof Ġseveral Ġlayers . ĠIn Ġeach Ġlayer Ġthere Ġis Ġfirst Ġa Ġmulti - head Ġself - att ention Ġsub - layer Ġand Ġthen Ġa Ġlinear Ġaff ine Ġsub - layer Ġwith Ġthe Ġresidual Ġconnection . ĠIn Ġeach Ġself - att ention Ġsub - layer Ġthe Ġattention Ġscores Ġ$ e _{ ij } $ Ġare Ġfirst Ġcalculated Ġas ĠE q . Ġ( Ġ5 Ġ) Ġ() Ġshows , Ġin Ġwhich Ġ$ d _ e $ Ġis Ġoutput Ġdimension , Ġand Ġ$ W ^ Q , ĠW ^ K , ĠW ^ V $ Ġare Ġparameter Ġmat rices . </s> $$ & a _{ ij } Ġ= Ġ\ c frac { ( h _ i W _ Q )( h _ j W _ K ) ^ T }{ \ sq rt { d _ e }} Ġ Ġ\\ Ġ& e _{ ij } Ġ= Ġ\ c frac {\ exp Ġ{ e _{ ij }} }{ \ sum Ġ_ { k = 1 } ^ N \ exp Ġ{ e _{ ik }}} Ġ$$ Ġ Ġ Ġ( E q . Ġ5 ) </s> Then Ġthe Ġoutput Ġis Ġcalculated Ġas ĠE q . Ġ( Ġ6 Ġ) Ġshows , Ġwhich Ġis Ġthe Ġweighted Ġsum Ġof Ġprevious Ġoutputs Ġ$ h $ Ġadded Ġby Ġprevious Ġoutput Ġ$ h _ i $ Ġ. ĠThe Ġlast Ġlayer Ġoutputs Ġis Ġcontext Ġencoding Ġof Ġinput Ġsequence . </s> $$ o _ i Ġ= Ġh _ i Ġ+ Ġ\ sum Ġ_ { j = 1 } ^{ N } Ġe _{ ij } ( h _ j ĠW _ V ) Ġ$$ Ġ Ġ Ġ( E q . Ġ6 ) </s> Despite Ġthe Ġwide Ġusage Ġand Ġhuge Ġsuccess , Ġthere Ġis Ġalso Ġa Ġmismatch Ġproblem Ġbetween Ġthese Ġpre - trained Ġcontext Ġenc od ers Ġand Ġsequence - to - sequence Ġmodels . ĠThe Ġissue Ġis Ġthat Ġwhile Ġusing Ġa Ġpre - trained Ġcontext Ġenc oder Ġlike ĠG PT Ġor ĠB ERT , Ġthey Ġmodel Ġtoken - level Ġrepresentations Ġby Ġconditioning Ġon Ġboth Ġdirection Ġcontext . ĠDuring Ġpre - training , Ġthey Ġare Ġfed Ġwith Ġcomplete Ġsequences . ĠHowever , Ġwith Ġa Ġleft - context - only Ġdec oder , Ġthese Ġpre - trained Ġlanguage Ġmodels Ġwill Ġsuffer Ġfrom Ġincomplete Ġand Ġinconsistent Ġcontext Ġand Ġthus Ġcannot Ġgenerate Ġgood Ġenough Ġcontext - aware Ġword Ġrepresentations , Ġespecially Ġduring Ġthe Ġinference Ġprocess . </s> Model </s> In Ġthis Ġsection , Ġwe Ġdescribe Ġthe Ġstructure Ġof Ġour Ġmodel , Ġwhich Ġlearns Ġto Ġgenerate Ġan Ġabstract ive Ġmulti - sent ence Ġsummary Ġfrom Ġa Ġgiven Ġsource Ġdocument . </s> Based Ġon Ġthe Ġsequence - to - sequence Ġframework Ġbuilt Ġon Ġtop Ġof ĠB ERT , Ġwe Ġfirst Ġdesign Ġa Ġrefine Ġdec oder Ġat Ġword - level Ġto Ġtackle Ġthe Ġtwo Ġproblems Ġdescribed Ġin Ġthe Ġabove Ġsection . ĠWe Ġalso Ġintroduce Ġa Ġdiscrete Ġobjective Ġfor Ġthe Ġrefine Ġdec od ers Ġto Ġreduce Ġthe Ġexposure Ġbias Ġproblem . ĠThe Ġoverall Ġstructure Ġof Ġour Ġmodel Ġis Ġillustrated Ġin ĠFigure Ġ1 Ġ. </s> Problem ĠForm ulation </s> We Ġdenote Ġthe Ġinput Ġdocument Ġas Ġ$ X Ġ= Ġ\ l brace Ġx _ 1 , Ġ\ ld ots Ġ, Ġx _ m \ r brace Ġ$ Ġwhere Ġ$ x _ i Ġ\ in Ġ\ math cal Ġ{ X } $ Ġrepresents Ġone Ġsource Ġtoken . ĠThe Ġcorresponding Ġsummary Ġis Ġden oted Ġas Ġ$ Y Ġ= Ġ\ l brace Ġy _ 1 , Ġ\ ld ots Ġ, Ġy _ L \ r brace Ġ$ Ġ, Ġ$ L $ Ġrepresents Ġthe Ġsummary Ġlength . </s> Given Ġinput Ġdocument Ġ$ X $ Ġ, Ġwe Ġfirst Ġpredict Ġthe Ġsummary Ġdraft Ġby Ġa Ġleft - context - only Ġdec oder , Ġand Ġthen Ġusing Ġthe Ġgenerated Ġsummary Ġdraft Ġwe Ġcan Ġcondition Ġon Ġboth Ġcontext Ġsides Ġand Ġrefine Ġthe Ġcontent Ġof Ġthe Ġsummary . ĠThe Ġdraft Ġwill Ġguide Ġand Ġconst rain Ġthe Ġrefine Ġprocess Ġof Ġsummary . </s> Summary ĠDraft ĠGeneration </s> The Ġsummary Ġdraft Ġis Ġbased Ġon Ġthe Ġsequence - to - sequence Ġmodel . ĠOn Ġthe Ġenc oder Ġside Ġthe Ġinput Ġdocument Ġ$ X $ Ġis Ġencoded Ġinto Ġrepresentation Ġvectors Ġ$ H Ġ= Ġ\ l brace Ġh _ 1 , Ġ\ ld ots Ġ, Ġh _ m \ r brace Ġ$ Ġ, Ġand Ġthen Ġfed Ġto Ġthe Ġdec oder Ġto Ġgenerate Ġthe Ġsummary Ġdraft Ġ$ A Ġ= Ġ\ l brace Ġa _ 1 , Ġ\ ld ots Ġ, Ġa _{ | a | }\ r brace Ġ$ Ġ. </s> We Ġsimply Ġuse ĠB ERT Ġas Ġthe Ġenc oder . ĠIt Ġfirst Ġmaps Ġthe Ġinput Ġsequence Ġto Ġword Ġembed d ings Ġand Ġthen Ġcomp utes Ġdocument Ġembed d ings Ġas Ġthe Ġenc oder 's Ġoutput , Ġden oted Ġby Ġfollowing Ġequation . </s> $$ H Ġ= ĠB ERT ( x _ 1 , Ġ\ ld ots Ġ, Ġx _ m ) $$ Ġ Ġ Ġ( E q . Ġ10 ) </s> In Ġthe Ġdraft Ġdec oder , Ġwe Ġfirst Ġintroduce ĠB ERT 's Ġword Ġembed ding Ġmatrix Ġto Ġmap Ġthe Ġprevious Ġsummary Ġdraft Ġoutputs Ġ$\ l brace Ġy _ 1 , Ġ\ ld ots Ġ, Ġy _{ t - 1 }\ r brace Ġ$ Ġinto Ġembed d ings Ġvectors Ġ$\ l brace Ġq _ 1 , Ġ\ ld ots Ġ, Ġq _{ t - 1 }\ r brace Ġ$ Ġat Ġt - th Ġtime Ġstep . ĠNote Ġthat Ġas Ġthe Ġinput Ġsequence Ġof Ġthe Ġdec oder Ġis Ġnot Ġcomplete , Ġwe Ġdo Ġnot Ġuse Ġthe ĠB ERT Ġnetwork Ġto Ġpredict Ġthe Ġcontext Ġvectors Ġhere . </s> Then Ġwe Ġintroduce Ġan Ġ$ N $ Ġlayer ĠTrans former Ġdec oder Ġto Ġlearn Ġthe Ġconditional Ġprobability Ġ$ P ( A | H ) $ Ġ. ĠTrans former 's Ġdec oder - enc oder Ġmulti - head Ġattention Ġhelps Ġthe Ġdec oder Ġlearn Ġsoft Ġalign ments Ġbetween Ġsummary Ġand Ġsource Ġdocument . ĠAt Ġthe Ġt - th Ġtime Ġstep , Ġthe Ġdraft Ġdec oder Ġpredicts Ġoutput Ġprobability Ġconditioned Ġon Ġprevious Ġoutputs Ġand Ġenc oder Ġhidden Ġrepresentations Ġas ĠE q . Ġ( Ġ13 Ġ) Ġshows , Ġin Ġwhich Ġ$ q _{ < t } Ġ= Ġ\ l brace Ġq _ 1 , Ġ\ ld ots Ġ, Ġq _{ t - 1 }\ r brace Ġ$ Ġ. ĠEach Ġgenerated Ġsequence Ġwill Ġbe Ġtrunc ated Ġin Ġthe Ġfirst Ġposition Ġof Ġa Ġspecial Ġtoken Ġ'[ P AD ] '. </s> $$ & P ^{ voc ab } _ t ( w ) Ġ= Ġf _{ dec } ( q _{ < t }, ĠH ) Ġ Ġ\\ Ġ& L _{ dec } Ġ= Ġ\ sum Ġ_ { i = 1 } ^{ | a | } Ġ- \ log ĠP ( a _ i Ġ= Ġy _ i ^ * | a _{ < Ġi }, ĠH ) Ġ$$ Ġ Ġ Ġ( E q . Ġ13 ) </s> As ĠE q . Ġ() Ġshows , Ġthe Ġdec oder 's Ġlearning Ġobjective Ġis Ġto Ġminimize Ġnegative Ġlikelihood Ġof Ġconditional Ġprobability , Ġin Ġwhich Ġ$ y _ i ^ * $ Ġis Ġthe Ġi - th Ġground Ġtruth Ġword Ġof Ġsummary . </s> However Ġa Ġdec oder Ġwith Ġthis Ġstructure Ġis Ġnot Ġsufficient Ġenough : Ġif Ġwe Ġuse Ġthe ĠB ERT Ġnetwork Ġin Ġthis Ġdec oder , Ġthen Ġduring Ġtraining Ġand Ġinference , Ġin - complete Ġcontext ( part Ġof Ġsentence ) Ġis Ġfed Ġinto Ġthe ĠB ERT Ġmodule , Ġand Ġalthough Ġwe Ġcan Ġfine - t une ĠB ERT 's Ġparameters , Ġthe Ġinput Ġdistribution Ġis Ġquite Ġdifferent Ġfrom Ġthe Ġpre - train Ġprocess , Ġand Ġthus Ġharms Ġthe Ġquality Ġof Ġgenerated Ġcontext Ġrepresentations . </s> If Ġwe Ġjust Ġuse Ġthe Ġembed ding Ġmatrix Ġhere , Ġit Ġwill Ġbe Ġmore Ġdifficult Ġfor Ġthe Ġdec oder Ġwith Ġfresh Ġparameters Ġto Ġlearn Ġto Ġmodel Ġrepresentations Ġas Ġwell Ġas Ġvocabulary Ġprobabilities , Ġfrom Ġa Ġrelative Ġsmall Ġcorpus Ġcompared Ġto ĠB ERT 's Ġhuge Ġpre - training Ġcorpus . ĠIn Ġa Ġword , Ġthe Ġdec oder Ġcannot Ġutilize ĠB ERT 's Ġability Ġto Ġgenerate Ġhigh Ġquality Ġcontext Ġvectors , Ġwhich Ġwill Ġalso Ġharm Ġperformance . </s> This Ġissue Ġexists Ġwhen Ġusing Ġany Ġother Ġcontextual ized Ġword Ġrepresentations , Ġso Ġwe Ġdesign Ġa Ġrefine Ġprocess Ġto Ġmitigate Ġit Ġin Ġour Ġapproach Ġwhich Ġwill Ġbe Ġdescribed Ġin Ġthe Ġnext Ġsub - section . </s> As Ġsome Ġsummary Ġtokens Ġare Ġout - of - voc abulary Ġwords Ġand Ġoccurs Ġin Ġinput Ġdocument , Ġwe Ġincorporate Ġcopy Ġmechanism ĠB IB REF 6 Ġbased Ġon Ġthe ĠTrans former Ġdec oder , Ġwe Ġwill Ġdescribe Ġit Ġbriefly . </s> At Ġdec oder Ġtime Ġstep Ġ$ t $ Ġ, Ġwe Ġfirst Ġcalculate Ġthe Ġattention Ġprobability Ġdistribution Ġover Ġsource Ġdocument Ġ$ X $ Ġusing Ġthe Ġbi - linear Ġdot Ġproduct Ġof Ġthe Ġlast Ġlayer Ġdec oder Ġoutput Ġof ĠTrans former Ġ$ o _ t $ Ġand Ġthe Ġenc oder Ġoutput Ġ$ h _ j $ Ġ, Ġas ĠE q . Ġ( Ġ15 Ġ) Ġ() Ġshows . </s> $$ u _ t ^ j Ġ= & Ġo _ t ĠW _ c Ġh _ j Ġ\\ Ġ\ alpha Ġ_ t ^ j Ġ= & Ġ\ c frac {\ exp Ġ{ u _ t ^ j }} {\ sum Ġ_ { k = 1 } ^ N \ exp Ġ{ u _ t ^ k }} Ġ$$ Ġ Ġ Ġ( E q . Ġ15 ) </s> We Ġthen Ġcalculate Ġcopying Ġgate Ġ$ g _ t \ in Ġ[ 0 , Ġ1 ] $ Ġ, Ġwhich Ġmakes Ġa Ġsoft Ġchoice Ġbetween Ġselecting Ġfrom Ġsource Ġand Ġgenerating Ġfrom Ġvocabulary , Ġ$ W _ c , ĠW _ g , Ġb _ g $ Ġare Ġparameters : </s> $$ g _ t Ġ= Ġs igm oid ( W _ g Ġ\ cd ot Ġ[ o _ t , Ġh ] Ġ+ Ġb _ g ) Ġ$$ Ġ Ġ Ġ( E q . Ġ16 ) </s> Using Ġ$ g _ t $ Ġwe Ġcalculate Ġthe Ġweighted Ġsum Ġof Ġcopy Ġprobability Ġand Ġgeneration Ġprobability Ġto Ġget Ġthe Ġfinal Ġpredicted Ġprobability Ġof Ġextended Ġvocabulary Ġ$\ math cal Ġ{ V } Ġ+ Ġ\ math cal Ġ{ X } $ Ġ, Ġwhere Ġ$\ math cal Ġ{ X } $ Ġis Ġthe Ġset Ġof Ġout Ġof Ġvocabulary Ġwords Ġfrom Ġthe Ġsource Ġdocument . ĠThe Ġfinal Ġprobability Ġis Ġcalculated Ġas Ġfollow : </s> $$ P _ t ( w ) Ġ= Ġ( 1 - g _ t ) P _ t ^{ voc ab } ( w ) Ġ+ Ġg _ t \ sum Ġ_ { i : w _ i = w } Ġ\ alpha Ġ_ t ^ i $$ Ġ Ġ Ġ( E q . Ġ17 ) </s> Summary ĠRef ine ĠProcess </s> The Ġmain Ġreason Ġto Ġintroduce Ġthe Ġrefine Ġprocess Ġis Ġto Ġenhance Ġthe Ġdec oder Ġusing ĠB ERT 's Ġcontextual ized Ġrepresentations , Ġso Ġwe Ġdo Ġnot Ġmodify Ġthe Ġenc oder Ġand Ġreuse Ġit Ġduring Ġthis Ġprocess . </s> On Ġthe Ġdec oder Ġside , Ġwe Ġpropose Ġa Ġnew Ġword - level Ġrefine Ġdec oder . ĠThe Ġrefine Ġdec oder Ġreceives Ġa Ġgenerated Ġsummary Ġdraft Ġas Ġinput , Ġand Ġoutputs Ġa Ġrefined Ġsummary . ĠIt Ġfirst Ġmasks Ġeach Ġword Ġin Ġthe Ġsummary Ġdraft Ġone Ġby Ġone , Ġthen Ġfeeds Ġthe Ġdraft Ġto ĠB ERT Ġto Ġgenerate Ġcontext Ġvectors . ĠFinally Ġit Ġpredicts Ġa Ġrefined Ġsummary Ġword Ġusing Ġan Ġ$ N $ Ġlayer ĠTrans former Ġdec oder Ġwhich Ġis Ġthe Ġsame Ġas Ġthe Ġdraft Ġdec oder . ĠAt Ġt - th Ġtime Ġstep Ġthe Ġn - th Ġword Ġof Ġinput Ġsummary Ġis Ġmasked , Ġand Ġthe Ġdec oder Ġpredicts Ġthe Ġn - th Ġrefined Ġword Ġgiven Ġother Ġwords Ġof Ġthe Ġsummary . </s> The Ġlearning Ġobjective Ġof Ġthis Ġprocess Ġis Ġshown Ġin ĠE q . Ġ( Ġ19 Ġ), Ġ$ y _ i $ Ġis Ġthe Ġi - th Ġsummary Ġword Ġand Ġ$ y _{ i } ^ * $ Ġfor Ġthe Ġground - truth Ġsummary Ġword , Ġand Ġ$ a _{ \ ne Ġi } Ġ= Ġ\ l brace Ġa _ 1 , Ġ\ ld ots Ġ, Ġa _{ i - 1 }, Ġa _{ i + 1 }, Ġ\ ld ots Ġ, Ġa _{ | y | }\ r brace Ġ$ Ġ. </s> $$ L _{ ref ine } Ġ= Ġ\ sum Ġ_ { i = 1 } ^{ | y | } Ġ- \ log ĠP ( y _ i Ġ= Ġy _ i ^ * | a _{ \ ne Ġi }, ĠH ) Ġ$$ Ġ Ġ Ġ( E q . Ġ19 ) </s> From Ġthe Ġview Ġof ĠB ERT Ġor Ġother Ġcontextual ized Ġembed d ings , Ġthe Ġrefine Ġdecoding Ġprocess Ġprovides Ġa Ġmore Ġcomplete Ġinput Ġsequence Ġwhich Ġis Ġconsistent Ġwith Ġtheir Ġpre - training Ġprocesses . ĠInt uitive ly , Ġthis Ġprocess Ġworks Ġas Ġfollows : Ġfirst Ġthe Ġdraft Ġdec oder Ġwrites Ġa Ġsummary Ġdraft Ġbased Ġon Ġa Ġdocument , Ġand Ġthen Ġthe Ġrefine Ġdec oder Ġedits Ġthe Ġdraft . ĠIt Ġconcent rates Ġon Ġone Ġword Ġat Ġa Ġtime , Ġbased Ġon Ġthe Ġsource Ġdocument Ġas Ġwell Ġas Ġother Ġwords . </s> We Ġdesign Ġthe Ġword - level Ġrefine Ġdec oder Ġbecause Ġthis Ġprocess Ġis Ġsimilar Ġto Ġthe Ġclo ze Ġtask Ġin ĠB ERT 's Ġpre - train Ġprocess , Ġtherefore Ġby Ġusing Ġthe Ġability Ġof Ġthe Ġcontextual Ġlanguage Ġmodel Ġthe Ġdec oder Ġcan Ġgenerate Ġmore Ġfluent Ġand Ġnatural Ġsequences . </s> The Ġparameters Ġare Ġshared Ġbetween Ġthe Ġdraft Ġdec oder Ġand Ġrefine Ġdec oder , Ġas Ġwe Ġfind Ġthat Ġusing Ġindividual Ġparameters Ġthe Ġmodel 's Ġperformance Ġde grades Ġa Ġlot . ĠThe Ġreason Ġmay Ġbe Ġthat Ġwe Ġuse Ġteach - forcing Ġduring Ġtraining , Ġand Ġthus Ġthe Ġword - level Ġrefine Ġdec oder Ġlearns Ġto Ġpredict Ġwords Ġgiven Ġall Ġthe Ġother Ġground - truth Ġwords Ġof Ġsummary . ĠThis Ġobjective Ġis Ġsimilar Ġto Ġthe Ġlanguage Ġmodel 's Ġpre - train Ġobjective , Ġand Ġis Ġprobably Ġnot Ġenough Ġfor Ġthe Ġdec oder Ġto Ġlearn Ġto Ġgenerate Ġrefined Ġsumm aries . ĠSo Ġin Ġour Ġmodel Ġall Ġdec od ers Ġshare Ġthe Ġsame Ġparameters . </s> Researchers Ġusually Ġuse ĠR OU GE Ġas Ġthe Ġevaluation Ġmetric Ġfor Ġsummar ization , Ġhowever Ġduring Ġsequence - to - sequence Ġmodel Ġtraining , Ġthe Ġobjective Ġis Ġto Ġmaximize Ġthe Ġlog Ġlikelihood Ġof Ġgenerated Ġsequences . ĠThis Ġmis - match Ġharms Ġthe Ġmodel 's Ġperformance , Ġso Ġwe Ġadd Ġa Ġdiscrete Ġobjective Ġto Ġthe Ġmodel , Ġand Ġoptimize Ġit Ġby Ġintroducing Ġthe Ġpolicy Ġgradient Ġmethod . ĠFor Ġexample , Ġthe Ġdiscrete Ġobjective Ġfor Ġthe Ġsummary Ġdraft Ġprocess Ġis Ġas ĠE q . Ġ( Ġ21 Ġ) Ġshows , Ġwhere Ġ$ a ^ s $ Ġis Ġthe Ġdraft Ġsummary Ġsampled Ġfrom Ġpredicted Ġdistribution , Ġand Ġ$ R ( a ^ s ) $ Ġis Ġthe Ġreward Ġscore Ġcompared Ġwith Ġthe Ġground - truth Ġsummary , Ġwe Ġuse ĠR OU GE - L Ġin Ġour Ġexperiment . ĠTo Ġbalance Ġbetween Ġoptimizing Ġthe Ġdiscrete Ġobjective Ġand Ġgenerating Ġreadable Ġsequences , Ġwe Ġmix Ġthe Ġdiscrete Ġobjective Ġwith Ġmaximum - like lihood Ġobjective . ĠAs ĠE q . Ġ() Ġshows , Ġminimizing Ġ$\ hat { L } _{ dec } $ Ġis Ġthe Ġfinal Ġobjective Ġfor Ġthe Ġdraft Ġprocess , Ġnote Ġhere Ġ$ L _{ dec } $ Ġis Ġ$ - log P ( a | x ) $ Ġ. ĠIn Ġthe Ġrefine Ġprocess Ġwe Ġintroduce Ġsimilar Ġobjectives . </s> $$ L ^{ rl } _{ dec } Ġ= ĠR ( a ^ s )\ cd ot Ġ[- \ log Ġ( P ( a ^ s | x )) ] Ġ Ġ\\ Ġ\ hat { L } _{ dec } Ġ= Ġ\ gam ma Ġ* ĠL ^{ rl } _{ dec } Ġ+ Ġ( 1 Ġ- Ġ\ gam ma Ġ) Ġ* ĠL _{ dec } Ġ$$ Ġ Ġ Ġ( E q . Ġ21 ) </s> Learning Ġand ĠIn ference </s> During Ġmodel Ġtraining , Ġthe Ġobjective Ġof Ġour Ġmodel Ġis Ġsum Ġof Ġthe Ġtwo Ġprocesses , Ġjointly Ġtrained Ġusing Ġ" te acher - forcing " Ġalgorithm . ĠDuring Ġtraining Ġwe Ġfeed Ġthe Ġground - truth Ġsummary Ġto Ġeach Ġdec oder Ġand Ġminimize Ġthe Ġobjective . </s> $$ L _{ model } Ġ= Ġ\ hat { L } _{ dec } Ġ+ Ġ\ hat { L } _{ ref ine } $$ Ġ Ġ Ġ( E q . Ġ23 ) </s> At Ġtest Ġtime , Ġeach Ġtime Ġstep Ġwe Ġchoose Ġthe Ġpredicted Ġword Ġby Ġ$\ hat { y } Ġ= Ġarg max _{ y ^ {\ prime Ġ}} ĠP ( y ^ {\ prime Ġ} | x ) $ Ġ, Ġuse Ġbeam Ġsearch Ġto Ġgenerate Ġthe Ġdraft Ġsumm aries , Ġand Ġuse Ġgreedy Ġsearch Ġto Ġgenerate Ġthe Ġrefined Ġsumm aries . </s> Settings </s> In Ġthis Ġwork , Ġall Ġof Ġour Ġmodels Ġare Ġbuilt Ġon Ġ$ BER T _{ B ASE } $ Ġ, Ġalthough Ġanother Ġlarger Ġpre - trained Ġmodel Ġwith Ġbetter Ġperformance Ġ( Ġ$ BER T _{ L AR GE } $ Ġ) Ġhas Ġpublished Ġbut Ġit Ġcosts Ġtoo Ġmuch Ġtime Ġand ĠGPU Ġmemory . ĠWe Ġuse ĠWord P iece Ġembed d ings Ġwith Ġa Ġ30 , 000 Ġvocabulary Ġwhich Ġis Ġthe Ġsame Ġas ĠB ERT . ĠWe Ġset Ġthe Ġlayer Ġof Ġtransformer Ġdec od ers Ġto Ġ12 ( 8 Ġon ĠNYT 50 ), Ġand Ġset Ġthe Ġattention Ġheads Ġnumber Ġto Ġ12 ( 8 Ġon ĠNYT 50 ), Ġset Ġfully - connected Ġsub - layer Ġhidden Ġsize Ġto Ġ30 72 . ĠWe Ġtrain Ġthe Ġmodel Ġusing Ġan ĠAdam Ġoptim izer Ġwith Ġlearning Ġrate Ġof Ġ$ 3 e - 4 $ Ġ, Ġ$\ beta Ġ_ 1 = 0 . 9 $ Ġ, Ġ$\ beta Ġ_ 2 = 0 . 999 $ Ġand Ġ$\ ep sil on Ġ= 10 ^{ - 9 } $ Ġand Ġuse Ġa Ġdynamic Ġlearning Ġrate Ġduring Ġthe Ġtraining Ġprocess . ĠFor Ġregular ization , Ġwe Ġuse Ġdrop out ĠB IB REF 13 Ġand Ġlabel Ġsmoot hing ĠB IB REF 14 Ġin Ġour Ġmodels Ġand Ġset Ġthe Ġdrop out Ġrate Ġto Ġ0 . 15 , Ġand Ġthe Ġlabel Ġsmoot hing Ġvalue Ġto Ġ0 . 1 . ĠWe Ġset Ġthe ĠRL Ġobjective Ġfactor Ġ$\ gam ma Ġ$ Ġto Ġ0 . 99 . </s> During Ġtraining , Ġwe Ġset Ġthe Ġbatch Ġsize Ġto Ġ36 , Ġand Ġtrain Ġfor Ġ4 Ġepoch s ( 8 Ġepoch s Ġfor ĠNYT 50 Ġsince Ġit Ġhas Ġmany Ġfewer Ġtraining Ġsamples ), Ġafter Ġtraining Ġthe Ġbest Ġmodel Ġare Ġselected Ġfrom Ġlast Ġ10 Ġmodels Ġbased Ġon Ġdevelopment Ġset Ġperformance . ĠDue Ġto ĠGPU Ġmemory Ġlimit , Ġwe Ġuse Ġgradient Ġaccumulation , Ġset Ġaccumulate Ġstep Ġto Ġ12 Ġand Ġfeed Ġ3 Ġsamples Ġat Ġeach Ġstep . ĠWe Ġuse Ġbeam Ġsize Ġ4 Ġand Ġlength Ġpenalty Ġof Ġ1 . 0 Ġto Ġgenerate Ġlogical Ġform Ġsequences . </s> We Ġfilter Ġrepeated Ġtri - gram s Ġin Ġbeam - search Ġprocess Ġby Ġsetting Ġword Ġprobability Ġto Ġzero Ġif Ġit Ġwill Ġgenerate Ġan Ġtri - gram Ġwhich Ġexists Ġin Ġthe Ġexisting Ġsummary . ĠIt Ġis Ġa Ġnice Ġmethod Ġto Ġavoid Ġphrase Ġrepetition Ġsince Ġthe Ġtwo Ġdatasets Ġseldom Ġcontains Ġrepeated Ġtri - gram s Ġin Ġone Ġsummary . ĠWe Ġalso Ġfine Ġtune Ġthe Ġgenerated Ġsequences Ġusing Ġanother Ġtwo Ġsimple Ġrules . ĠWhen Ġthere Ġare Ġmulti Ġsummary Ġsentences Ġwith Ġexactly Ġthe Ġsame Ġcontent , Ġwe Ġkeep Ġthe Ġfirst Ġone Ġand Ġremove Ġthe Ġother Ġsentences ; Ġwe Ġalso Ġremove Ġsentences Ġwith Ġless Ġthan Ġ3 Ġwords Ġfrom Ġthe Ġresult . </s> To Ġevaluate Ġthe Ġperformance Ġof Ġour Ġmodel , Ġwe Ġconduct Ġexperiments Ġon ĠCNN / Daily ĠMail Ġdataset , Ġwhich Ġis Ġa Ġlarge Ġcollection Ġof Ġnews Ġarticles Ġand Ġmodified Ġfor Ġsummar ization . ĠFollowing ĠB IB REF 0 Ġwe Ġchoose Ġthe Ġnon - an onym ized Ġversion Ġof Ġthe Ġdataset , Ġwhich Ġconsists Ġof Ġmore Ġthan Ġ280 , 000 Ġtraining Ġsamples Ġand Ġ11 490 Ġtest Ġset Ġsamples . </s> We Ġalso Ġconduct Ġexperiments Ġon Ġthe ĠNew ĠYork ĠTimes ( NY T ) Ġdataset Ġwhich Ġalso Ġconsists Ġof Ġmany Ġnews Ġarticles . ĠThe Ġoriginal Ġdataset Ġcan Ġbe Ġapplied Ġhere . ĠIn Ġour Ġexperiment , Ġwe Ġfollow Ġthe Ġdataset Ġsplits Ġand Ġother Ġpre - process Ġsettings Ġof ĠB IB REF 15 Ġ. ĠWe Ġfirst Ġfilter Ġall Ġsamples Ġwithout Ġa Ġfull Ġarticle Ġtext Ġor Ġabstract Ġand Ġthen Ġremove Ġall Ġsamples Ġwith Ġsumm aries Ġshorter Ġthan Ġ50 Ġwords . ĠThen Ġwe Ġchoose Ġthe Ġtest Ġset Ġbased Ġon Ġthe Ġdate Ġof Ġpublication ( all Ġexamples Ġpublished Ġafter ĠJanuary Ġ1 , Ġ2007 ). ĠThe Ġfinal Ġdataset Ġcontains Ġ22 , 000 Ġtraining Ġsamples Ġand Ġ3 , 452 Ġtest Ġsamples Ġand Ġis Ġcalled ĠNYT 50 Ġsince Ġall Ġsumm aries Ġare Ġlonger Ġthan Ġ50 Ġwords . </s> We Ġtoken ize Ġall Ġsequences Ġof Ġthe Ġtwo Ġdatasets Ġusing Ġthe ĠWord P iece Ġtoken izer . ĠAfter Ġtoken izing , Ġthe Ġaverage Ġarticle Ġlength Ġand Ġsummary Ġlength Ġof ĠCNN / Daily ĠMail Ġare Ġ6 91 Ġand Ġ51 , Ġand ĠNYT 50 's Ġaverage Ġarticle Ġlength Ġand Ġsummary Ġlength Ġare Ġ115 2 Ġand Ġ75 . ĠWe Ġtrunc ate Ġthe Ġarticle Ġlength Ġto Ġ512 , Ġand Ġthe Ġsummary Ġlength Ġto Ġ100 Ġin Ġour Ġexperiment ( max Ġsummary Ġlength Ġis Ġset Ġto Ġ150 Ġon ĠNYT 50 Ġas Ġits Ġaverage Ġgolden Ġsummary Ġlength Ġis Ġlonger ). </s> On ĠCNN / Daily ĠMail Ġdataset , Ġwe Ġreport Ġthe Ġfull - length ĠF - 1 Ġscore Ġof Ġthe ĠR OU GE - 1 , ĠR OU GE - 2 Ġand ĠR OU GE - L Ġmetrics , Ġcalculated Ġusing ĠPy R ou ge Ġpackage Ġand Ġthe ĠPorter Ġstem mer Ġoption . ĠOn ĠNYT 50 , Ġfollowing ĠB IB REF 4 Ġwe Ġevaluate Ġlimited Ġlength ĠR OU GE Ġrecall Ġscore ( limit Ġthe Ġgenerated Ġsummary Ġlength Ġto Ġthe Ġground Ġtruth Ġlength ). ĠWe Ġsplit ĠNYT 50 Ġsumm aries Ġinto Ġsentences Ġby Ġsem icol ons Ġto Ġcalculate Ġthe ĠR OU GE Ġscores . </s> Results Ġand ĠAnalysis </s> Table Ġ1 Ġshows Ġthe Ġresults Ġon ĠCNN / Daily ĠMail Ġdataset , Ġwe Ġcompare Ġthe Ġperformance Ġof Ġmany Ġrecent Ġapproaches Ġwith Ġour Ġmodel . ĠWe Ġclassify Ġthem Ġto Ġtwo Ġgroups Ġbased Ġon Ġwhether Ġthey Ġare Ġextract ive Ġor Ġabstract ive Ġmodels . ĠAs Ġthe Ġlast Ġline Ġof Ġthe Ġtable Ġshows , Ġthe ĠR OU GE - 1 Ġand ĠR OU GE - 2 Ġscore Ġof Ġour Ġfull Ġmodel Ġis Ġcomparable Ġwith ĠDC A , Ġand Ġoutper forms Ġon ĠR OU GE - L . ĠAlso , Ġcompared Ġto Ġextract ive Ġmodels ĠNe u S UM Ġand ĠMAS K - Ġ$ LM ^{ global } $ Ġ, Ġwe Ġachieve Ġslight Ġhigher ĠR OU GE - 1 . ĠExcept Ġthe Ġfour Ġscores , Ġour Ġmodel Ġoutper forms Ġthese Ġmodels Ġon Ġall Ġthe Ġother Ġscores , Ġand Ġsince Ġwe Ġhave Ġ95 % Ġconfidence Ġinterval Ġof Ġat Ġmost Ġ$\ pm Ġ$ Ġ0 . 20 , Ġthese Ġimprovements Ġare Ġstatistically Ġsignificant . </s> As Ġthe Ġlast Ġfour Ġlines Ġof ĠTable Ġ1 Ġshow , Ġwe Ġconduct Ġan Ġab lation Ġstudy Ġon Ġour Ġmodel Ġvariants Ġto Ġanalyze Ġthe Ġimportance Ġof Ġeach Ġcomponent . ĠWe Ġuse Ġthree Ġab lation Ġmodels Ġfor Ġthe Ġexperiments . ĠOne - Stage : ĠA Ġsequence - to - sequence Ġmodel Ġwith Ġcopy Ġmechanism Ġbased Ġon ĠB ERT ; ĠTwo - Stage : ĠAdding Ġthe Ġword - ref ine Ġdec oder Ġto Ġthe ĠOne - Stage Ġmodel ; ĠTwo - Stage Ġ+ ĠRL : ĠFull Ġmodel Ġwith Ġrefine Ġprocess Ġcooper ated Ġwith ĠRL Ġobjective . </s> First , Ġwe Ġcompare Ġthe ĠTwo - Stage + RL Ġmodel Ġwith ĠTwo - Stage Ġab lation , Ġwe Ġobserve Ġthat Ġthe Ġfull Ġmodel Ġoutper forms Ġby Ġ0 . 30 Ġon Ġaverage ĠR OU GE , Ġsuggesting Ġthat Ġthe Ġreinforcement Ġobjective Ġhelps Ġthe Ġmodel Ġeffectively . ĠThen Ġwe Ġanalyze Ġthe Ġeffect Ġof Ġrefine Ġprocess Ġby Ġremoving Ġword - level Ġrefine Ġfrom Ġthe ĠTwo - Stage Ġmodel , Ġwe Ġobserve Ġthat Ġwithout Ġthe Ġrefine Ġprocess Ġthe Ġaverage ĠR OU GE Ġscore Ġdrops Ġby Ġ1 . 69 . ĠThe Ġab lation Ġstudy Ġshows Ġthat Ġeach Ġmodule Ġis Ġnecessary Ġfor Ġour Ġfull Ġmodel , Ġand Ġthe Ġimprovements Ġare Ġstatistically Ġsignificant Ġon Ġall Ġmetrics . </s> To Ġevaluate Ġthe Ġimpact Ġof Ġsummary Ġlength Ġon Ġmodel Ġperformance , Ġwe Ġcompare Ġthe Ġaverage Ġrou ge Ġscore Ġimprovements Ġof Ġour Ġmodel Ġwith Ġdifferent Ġlength Ġof Ġground - truth Ġsumm aries . ĠAs Ġthe Ġabove Ġsub - figure Ġof ĠFigure Ġ2 Ġshows , Ġcompared Ġto ĠPo inter - Gener ator Ġwith ĠCoverage , Ġon Ġlength Ġinterval Ġ40 - 80 ( occup ies Ġabout Ġ70 % Ġof Ġtest Ġset ) Ġthe Ġimprovements Ġof Ġour Ġmodel Ġare Ġhigher Ġthan Ġshorter Ġsamples , Ġconfirms Ġthat Ġwith Ġbetter Ġcontext Ġrepresentations , Ġin Ġlonger Ġdocuments Ġour Ġmodel Ġcan Ġachieve Ġhigher Ġperformance . </s> As Ġthe Ġbelow Ġsub - figure Ġof ĠFigure Ġ2 Ġshows , Ġcompared Ġto Ġextract ive Ġbaseline : ĠLead - 3 ĠB IB REF 0 Ġ, Ġthe Ġadvantage Ġof Ġour Ġmodel Ġwill Ġfall Ġwhen Ġgolden Ġsummary Ġlength Ġis Ġgreater Ġthan Ġ80 . ĠThis Ġprobably Ġbecause Ġthat Ġwe Ġtrunc ate Ġthe Ġlong Ġdocuments Ġand Ġgolden Ġsumm aries Ġand Ġcannot Ġget Ġfull Ġinformation , Ġit Ġcould Ġalso Ġbecause Ġthat Ġthe Ġtraining Ġdata Ġin Ġthese Ġintervals Ġis Ġtoo Ġfew Ġto Ġtrain Ġan Ġabstract ive Ġmodel , Ġso Ġsimple Ġextract ive Ġmethod Ġwill Ġnot Ġfall Ġtoo Ġfar Ġbehind . </s> Additional ĠResults Ġon ĠNYT 50 </s> Table Ġ2 Ġshows Ġexperiments Ġon Ġthe ĠNYT 50 Ġcorpus . ĠSince Ġthe Ġshort Ġsummary Ġsamples Ġare Ġfiltered , ĠNYT 50 Ġhas Ġaverage Ġlonger Ġsumm aries Ġthan ĠCNN / Daily ĠMail . ĠSo Ġthe Ġmodel Ġneeds Ġto Ġcatch Ġlong - term Ġdependency Ġof Ġthe Ġsequences Ġto Ġgenerate Ġgood Ġsumm aries . </s> The Ġfirst Ġtwo Ġlines Ġof ĠTable Ġ2 Ġshow Ġresults Ġof Ġthe Ġtwo Ġbas elines Ġintroduced Ġby ĠB IB REF 15 Ġ: Ġthese Ġbas elines Ġselect Ġfirst Ġn Ġsentences , Ġor Ġselect Ġthe Ġfirst Ġk Ġwords Ġfrom Ġthe Ġoriginal Ġdocument . ĠAlso Ġwe Ġcompare Ġperformance Ġof Ġour Ġmodel Ġwith Ġtwo Ġrecent Ġmodels , Ġwe Ġsee Ġ2 . 39 ĠR OU GE - 1 Ġimprovements Ġcompared Ġto Ġthe ĠML + RL Ġwith Ġintra - att n Ġapproach ( pre vious ĠS OTA ) Ġcarries Ġover Ġto Ġthis Ġdataset , Ġwhich Ġis Ġa Ġlarge Ġmargin . ĠOn ĠR OU GE - 2 , Ġour Ġmodel Ġalso Ġget Ġan Ġimprovement Ġof Ġ0 . 51 . ĠThe Ġexperiment Ġproves Ġthat Ġour Ġapproach Ġcan Ġoutper form Ġcompetitive Ġmethods Ġon Ġdifferent Ġdata Ġdistributions . </s> Pre - trained Ġlanguage Ġmodels </s> Pre - trained Ġword Ġvectors ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ, ĠB IB REF 19 Ġhave Ġbeen Ġwidely Ġused Ġin Ġmany ĠN LP Ġtasks . ĠMore Ġrecently , Ġpre - trained Ġlanguage Ġmodels Ġ( EL Mo , ĠG PT Ġand ĠB ERT ), Ġhave Ġalso Ġachieved Ġgreat Ġsuccess Ġon Ġseveral ĠN LP Ġproblems Ġsuch Ġas Ġtextual Ġentail ment , Ġsemantic Ġsimilarity , Ġreading Ġcomprehension , Ġand Ġquestion Ġanswering ĠB IB REF 20 Ġ, ĠB IB REF 21 Ġ, ĠB IB REF 22 Ġ. </s> Some Ġrecent Ġworks Ġalso Ġfocus Ġon Ġleveraging Ġpre - trained Ġlanguage Ġmodels Ġin Ġsummar ization . ĠB IB REF 23 Ġpret rain Ġa Ġlanguage Ġmodel Ġand Ġuse Ġit Ġas Ġthe Ġsentiment Ġanalys er Ġwhen Ġgenerating Ġreviews Ġof Ġgoods . ĠB IB REF 24 Ġtrain Ġa Ġlanguage Ġmodel Ġon Ġgolden Ġsumm aries , Ġand Ġthen Ġuse Ġit Ġon Ġthe Ġdec oder Ġside Ġto Ġincorporate Ġprior Ġknowledge . </s> In Ġthis Ġwork , Ġwe Ġuse ĠB ERT ( which Ġis Ġa Ġpre - trained Ġlanguage Ġmodel Ġusing Ġlarge Ġscale Ġunl abel ed Ġdata ) Ġon Ġthe Ġenc oder Ġand Ġdec oder Ġof Ġa Ġseq 2 seq Ġmodel , Ġand Ġby Ġdesigning Ġa Ġtwo Ġstage Ġdecoding Ġstructure Ġwe Ġbuild Ġa Ġcompetitive Ġmodel Ġfor Ġabstract ive Ġtext Ġsummar ization . </s> Conclusion Ġand ĠFuture ĠWork </s> In Ġthis Ġwork , Ġwe Ġpropose Ġa Ġtwo - stage Ġmodel Ġbased Ġon Ġsequence - to - sequence Ġparadigm . ĠOur Ġmodel Ġutilize ĠB ERT Ġon Ġboth Ġenc oder Ġand Ġdec oder Ġsides , Ġand Ġintroduce Ġreinforce Ġobjective Ġin Ġlearning Ġprocess . ĠWe Ġevaluate Ġour Ġmodel Ġon Ġtwo Ġbenchmark Ġdatasets ĠCNN / Daily ĠMail Ġand ĠNew ĠYork ĠTimes , Ġthe Ġexperimental Ġresults Ġshow Ġthat Ġcompared Ġto Ġprevious Ġsystems Ġour Ġapproach Ġeffectively Ġimproves Ġperformance . </s> Although Ġour Ġexperiments Ġare Ġconducted Ġon Ġsummar ization Ġtask , Ġour Ġmodel Ġcan Ġbe Ġused Ġin Ġmost Ġnatural Ġlanguage Ġgeneration Ġtasks , Ġsuch Ġas Ġmachine Ġtranslation , Ġquestion Ġgeneration Ġand Ġparaph r asing . ĠThe Ġrefine Ġdec oder Ġand Ġmixed Ġobjective Ġcan Ġalso Ġbe Ġapplied Ġon Ġother Ġsequence Ġgeneration Ġtasks , Ġand Ġwe Ġwill Ġinvestigate Ġon Ġthem Ġin Ġfuture Ġwork .
Introduction Text summarization generates summaries from input documents while keeping salient information. It is an important task and can be applied to several real-world applications. Many methods have been proposed to solve the text summarization problem BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 . There are two main text summarization techniques: extractive and abstractive. Extractive summarization generates summary by selecting salient sentences or phrases from the source text, while abstractive methods paraphrase and restructure sentences to compose the summary. We focus on abstractive summarization in this work as it is more flexible and thus can generate more diverse summaries. Recently, many abstractive approaches are introduced based on neural sequence-to-sequence framework BIBREF4 , BIBREF0 , BIBREF3 , BIBREF5 . Based on the sequence-to-sequence model with copy mechanism BIBREF6 , BIBREF0 incorporates a coverage vector to track and control attention scores on source text. BIBREF4 introduce intra-temporal attention processes in the encoder and decoder to address the repetition and incoherent problem. There are two issues in previous abstractive methods: 1) these methods use left-context-only decoder, thus do not have complete context when predicting each word. 2) they do not utilize the pre-trained contextualized language models on the decoder side, so it is more difficult for the decoder to learn summary representations, context interactions and language modeling together. Recently, BERT has been successfully used in various natural language processing tasks, such as textual entailment, name entity recognition and machine reading comprehensions. In this paper, we present a novel natural language generation model based on pre-trained language models (we use BERT in this work). As far as we know, this is the first work to extend BERT to the sequence generation task. To address the above issues of previous abstractive methods, in our model, we design a two-stage decoding process to make good use of BERT's context modeling ability. On the first stage, we generate the summary using a left-context-only-decoder. On the second stage, we mask each word of the summary and predict the refined word one-by-one using a refine decoder. To further improve the naturalness of the generated sequence, we cooperate reinforcement objective with the refine decoder. The main contributions of this work are: 1. We propose a natural language generation model based on BERT, making good use of the pre-trained language model in the encoder and decoder process, and the model can be trained end-to-end without handcrafted features. 2. We design a two-stage decoder process. In this architecture, our model can generate each word of the summary considering both sides' context information. 3. We conduct experiments on the benchmark datasets CNN/Daily Mail and New York Times. Our model achieves a 33.33 average of ROUGE-1, ROUGE-2 and ROUGE-L on the CNN/Daily Mail, which is state-of-the-art. On the New York Times dataset, our model achieves about 5.6% relative improvement over ROUGE-1. Text Summarization In this paper, we focus on single-document multi-sentence summarization and propose a supervised abstractive model based on the neural attentive sequence-to-sequence framework which consists of two parts: a neural network for the encoder and another network for the decoder. The encoder encodes the input sequence to intermediate representation and the decoder predicts one word at a time step given the input sequence representation vector and previous decoded output. The goal of the model is to maximize the probability of generating the correct target sequences. In the encoding and generation process, the attention mechanism is used to concentrate on the most important positions of text. The learning objective of most sequence-to-sequence models is to minimize the negative log likelihood of the generated sequence as following equation shows, where $y^*_i$ is the i-th ground-truth summary token. $$Loss = - \log \sum _{t=1}^N P(y_t^*|y_{<t}^*, X)$$   (Eq. 3) However, with this objective, traditional sequence generation models consider only one direction context in the decoding process, which could cause performance degradation since complete context of one token contains preceding and following tokens, thus feeding only preceded decoded words to the decoder so that the model may generate unnatural sequences. For example, attentive sequence-to-sequence models often generate sequences with repeated phrases which harm the naturalness. Some previous works mitigate this problem by improving the attention calculation process, but in this paper we show that feeding bi-directional context instead of left-only-context can better alleviate this problem. Text summarization models are usually classified to abstractive and extractive ones. Recently, extractive models like DeepChannel BIBREF8 , rnn-ext+RL BIBREF9 and NeuSUM BIBREF2 achieve higher performances using well-designed structures. For example, DeepChannel propose a salience estimation network and iteratively extract salient sentences. BIBREF16 train a sentence compression model to teach another latent variable extractive model. Also, several recent works focus on improving abstractive methods. BIBREF3 design a content selector to over-determine phrases in a source document that should be part of the summary. BIBREF11 introduce inconsistency loss to force words in less attended sentences(which determined by extractive model) to have lower generation probabilities. BIBREF5 extend seq2seq model with an information selection network to generate more informative summaries. Bi-Directional Pre-Trained Context Encoders Recently, context encoders such as ELMo, GPT, and BERT have been widely used in many NLP tasks. These models are pre-trained on a huge unlabeled corpus and can generate better contextualized token embeddings, thus the approaches built on top of them can achieve better performance. Since our method is based on BERT, we illustrate the process briefly here. BERT consists of several layers. In each layer there is first a multi-head self-attention sub-layer and then a linear affine sub-layer with the residual connection. In each self-attention sub-layer the attention scores $e_{ij}$ are first calculated as Eq. ( 5 ) () shows, in which $d_e$ is output dimension, and $W^Q, W^K, W^V$ are parameter matrices. $$&a_{ij} = \cfrac{(h_iW_Q)(h_jW_K)^T}{\sqrt{d_e}}  \\ &e_{ij} = \cfrac{\exp {e_{ij}}}{\sum _{k=1}^N\exp {e_{ik}}} $$   (Eq. 5) Then the output is calculated as Eq. ( 6 ) shows, which is the weighted sum of previous outputs $h$ added by previous output $h_i$ . The last layer outputs is context encoding of input sequence. $$o_i = h_i + \sum _{j=1}^{N} e_{ij}(h_j W_V) $$   (Eq. 6) Despite the wide usage and huge success, there is also a mismatch problem between these pre-trained context encoders and sequence-to-sequence models. The issue is that while using a pre-trained context encoder like GPT or BERT, they model token-level representations by conditioning on both direction context. During pre-training, they are fed with complete sequences. However, with a left-context-only decoder, these pre-trained language models will suffer from incomplete and inconsistent context and thus cannot generate good enough context-aware word representations, especially during the inference process. Model In this section, we describe the structure of our model, which learns to generate an abstractive multi-sentence summary from a given source document. Based on the sequence-to-sequence framework built on top of BERT, we first design a refine decoder at word-level to tackle the two problems described in the above section. We also introduce a discrete objective for the refine decoders to reduce the exposure bias problem. The overall structure of our model is illustrated in Figure 1 . Problem Formulation We denote the input document as $X = \lbrace x_1, \ldots , x_m\rbrace $ where $x_i \in \mathcal {X}$ represents one source token. The corresponding summary is denoted as $Y = \lbrace y_1, \ldots , y_L\rbrace $ , $L$ represents the summary length. Given input document $X$ , we first predict the summary draft by a left-context-only decoder, and then using the generated summary draft we can condition on both context sides and refine the content of the summary. The draft will guide and constrain the refine process of summary. Summary Draft Generation The summary draft is based on the sequence-to-sequence model. On the encoder side the input document $X$ is encoded into representation vectors $H = \lbrace h_1, \ldots , h_m\rbrace $ , and then fed to the decoder to generate the summary draft $A = \lbrace a_1, \ldots , a_{|a|}\rbrace $ . We simply use BERT as the encoder. It first maps the input sequence to word embeddings and then computes document embeddings as the encoder's output, denoted by following equation. $$H = BERT(x_1, \ldots , x_m)$$   (Eq. 10) In the draft decoder, we first introduce BERT's word embedding matrix to map the previous summary draft outputs $\lbrace y_1, \ldots , y_{t-1}\rbrace $ into embeddings vectors $\lbrace q_1, \ldots , q_{t-1}\rbrace $ at t-th time step. Note that as the input sequence of the decoder is not complete, we do not use the BERT network to predict the context vectors here. Then we introduce an $N$ layer Transformer decoder to learn the conditional probability $P(A|H)$ . Transformer's decoder-encoder multi-head attention helps the decoder learn soft alignments between summary and source document. At the t-th time step, the draft decoder predicts output probability conditioned on previous outputs and encoder hidden representations as Eq. ( 13 ) shows, in which $q_{<t} = \lbrace q_1, \ldots , q_{t-1}\rbrace $ . Each generated sequence will be truncated in the first position of a special token '[PAD]'. $$&P^{vocab}_t(w) = f_{dec}(q_{<t}, H)  \\ &L_{dec} = \sum _{i=1}^{|a|} -\log P(a_i = y_i^*|a_{< i}, H) $$   (Eq. 13) As Eq. () shows, the decoder's learning objective is to minimize negative likelihood of conditional probability, in which $y_i^*$ is the i-th ground truth word of summary. However a decoder with this structure is not sufficient enough: if we use the BERT network in this decoder, then during training and inference, in-complete context(part of sentence) is fed into the BERT module, and although we can fine-tune BERT's parameters, the input distribution is quite different from the pre-train process, and thus harms the quality of generated context representations. If we just use the embedding matrix here, it will be more difficult for the decoder with fresh parameters to learn to model representations as well as vocabulary probabilities, from a relative small corpus compared to BERT's huge pre-training corpus. In a word, the decoder cannot utilize BERT's ability to generate high quality context vectors, which will also harm performance. This issue exists when using any other contextualized word representations, so we design a refine process to mitigate it in our approach which will be described in the next sub-section. As some summary tokens are out-of-vocabulary words and occurs in input document, we incorporate copy mechanism BIBREF6 based on the Transformer decoder, we will describe it briefly. At decoder time step $t$ , we first calculate the attention probability distribution over source document $X$ using the bi-linear dot product of the last layer decoder output of Transformer $o_t$ and the encoder output $h_j$ , as Eq. ( 15 ) () shows. $$u_t^j =& o_t W_c h_j \\ \alpha _t^j =& \cfrac{\exp {u_t^j}}{\sum _{k=1}^N\exp {u_t^k}} $$   (Eq. 15) We then calculate copying gate $g_t\in [0, 1]$ , which makes a soft choice between selecting from source and generating from vocabulary, $W_c, W_g, b_g$ are parameters: $$g_t = sigmoid(W_g \cdot [o_t, h] + b_g) $$   (Eq. 16) Using $g_t$ we calculate the weighted sum of copy probability and generation probability to get the final predicted probability of extended vocabulary $\mathcal {V} + \mathcal {X}$ , where $\mathcal {X}$ is the set of out of vocabulary words from the source document. The final probability is calculated as follow: $$P_t(w) = (1-g_t)P_t^{vocab}(w) + g_t\sum _{i:w_i=w} \alpha _t^i$$   (Eq. 17) Summary Refine Process The main reason to introduce the refine process is to enhance the decoder using BERT's contextualized representations, so we do not modify the encoder and reuse it during this process. On the decoder side, we propose a new word-level refine decoder. The refine decoder receives a generated summary draft as input, and outputs a refined summary. It first masks each word in the summary draft one by one, then feeds the draft to BERT to generate context vectors. Finally it predicts a refined summary word using an $N$ layer Transformer decoder which is the same as the draft decoder. At t-th time step the n-th word of input summary is masked, and the decoder predicts the n-th refined word given other words of the summary. The learning objective of this process is shown in Eq. ( 19 ), $y_i$ is the i-th summary word and $y_{i}^*$ for the ground-truth summary word, and $a_{\ne i} = \lbrace a_1, \ldots , a_{i-1}, a_{i+1}, \ldots , a_{|y|}\rbrace $ . $$L_{refine} = \sum _{i=1}^{|y|} -\log P(y_i = y_i^*|a_{\ne i}, H) $$   (Eq. 19) From the view of BERT or other contextualized embeddings, the refine decoding process provides a more complete input sequence which is consistent with their pre-training processes. Intuitively, this process works as follows: first the draft decoder writes a summary draft based on a document, and then the refine decoder edits the draft. It concentrates on one word at a time, based on the source document as well as other words. We design the word-level refine decoder because this process is similar to the cloze task in BERT's pre-train process, therefore by using the ability of the contextual language model the decoder can generate more fluent and natural sequences. The parameters are shared between the draft decoder and refine decoder, as we find that using individual parameters the model's performance degrades a lot. The reason may be that we use teach-forcing during training, and thus the word-level refine decoder learns to predict words given all the other ground-truth words of summary. This objective is similar to the language model's pre-train objective, and is probably not enough for the decoder to learn to generate refined summaries. So in our model all decoders share the same parameters. Researchers usually use ROUGE as the evaluation metric for summarization, however during sequence-to-sequence model training, the objective is to maximize the log likelihood of generated sequences. This mis-match harms the model's performance, so we add a discrete objective to the model, and optimize it by introducing the policy gradient method. For example, the discrete objective for the summary draft process is as Eq. ( 21 ) shows, where $a^s$ is the draft summary sampled from predicted distribution, and $R(a^s)$ is the reward score compared with the ground-truth summary, we use ROUGE-L in our experiment. To balance between optimizing the discrete objective and generating readable sequences, we mix the discrete objective with maximum-likelihood objective. As Eq. () shows, minimizing $\hat{L}_{dec}$ is the final objective for the draft process, note here $L_{dec}$ is $-logP(a|x)$ . In the refine process we introduce similar objectives. $$L^{rl}_{dec} = R(a^s)\cdot [-\log (P(a^s|x))]  \\ \hat{L}_{dec} = \gamma * L^{rl}_{dec} + (1 - \gamma ) * L_{dec} $$   (Eq. 21) Learning and Inference During model training, the objective of our model is sum of the two processes, jointly trained using "teacher-forcing" algorithm. During training we feed the ground-truth summary to each decoder and minimize the objective. $$L_{model} = \hat{L}_{dec} + \hat{L}_{refine}$$   (Eq. 23) At test time, each time step we choose the predicted word by $\hat{y} = argmax_{y^{\prime }} P(y^{\prime }|x)$ , use beam search to generate the draft summaries, and use greedy search to generate the refined summaries. Settings In this work, all of our models are built on $BERT_{BASE}$ , although another larger pre-trained model with better performance ( $BERT_{LARGE}$ ) has published but it costs too much time and GPU memory. We use WordPiece embeddings with a 30,000 vocabulary which is the same as BERT. We set the layer of transformer decoders to 12(8 on NYT50), and set the attention heads number to 12(8 on NYT50), set fully-connected sub-layer hidden size to 3072. We train the model using an Adam optimizer with learning rate of $3e-4$ , $\beta _1=0.9$ , $\beta _2=0.999$ and $\epsilon =10^{-9}$ and use a dynamic learning rate during the training process. For regularization, we use dropout BIBREF13 and label smoothing BIBREF14 in our models and set the dropout rate to 0.15, and the label smoothing value to 0.1. We set the RL objective factor $\gamma $ to 0.99. During training, we set the batch size to 36, and train for 4 epochs(8 epochs for NYT50 since it has many fewer training samples), after training the best model are selected from last 10 models based on development set performance. Due to GPU memory limit, we use gradient accumulation, set accumulate step to 12 and feed 3 samples at each step. We use beam size 4 and length penalty of 1.0 to generate logical form sequences. We filter repeated tri-grams in beam-search process by setting word probability to zero if it will generate an tri-gram which exists in the existing summary. It is a nice method to avoid phrase repetition since the two datasets seldom contains repeated tri-grams in one summary. We also fine tune the generated sequences using another two simple rules. When there are multi summary sentences with exactly the same content, we keep the first one and remove the other sentences; we also remove sentences with less than 3 words from the result. To evaluate the performance of our model, we conduct experiments on CNN/Daily Mail dataset, which is a large collection of news articles and modified for summarization. Following BIBREF0 we choose the non-anonymized version of the dataset, which consists of more than 280,000 training samples and 11490 test set samples. We also conduct experiments on the New York Times(NYT) dataset which also consists of many news articles. The original dataset can be applied here. In our experiment, we follow the dataset splits and other pre-process settings of BIBREF15 . We first filter all samples without a full article text or abstract and then remove all samples with summaries shorter than 50 words. Then we choose the test set based on the date of publication(all examples published after January 1, 2007). The final dataset contains 22,000 training samples and 3,452 test samples and is called NYT50 since all summaries are longer than 50 words. We tokenize all sequences of the two datasets using the WordPiece tokenizer. After tokenizing, the average article length and summary length of CNN/Daily Mail are 691 and 51, and NYT50's average article length and summary length are 1152 and 75. We truncate the article length to 512, and the summary length to 100 in our experiment(max summary length is set to 150 on NYT50 as its average golden summary length is longer). On CNN/Daily Mail dataset, we report the full-length F-1 score of the ROUGE-1, ROUGE-2 and ROUGE-L metrics, calculated using PyRouge package and the Porter stemmer option. On NYT50, following BIBREF4 we evaluate limited length ROUGE recall score(limit the generated summary length to the ground truth length). We split NYT50 summaries into sentences by semicolons to calculate the ROUGE scores. Results and Analysis Table 1 shows the results on CNN/Daily Mail dataset, we compare the performance of many recent approaches with our model. We classify them to two groups based on whether they are extractive or abstractive models. As the last line of the table shows, the ROUGE-1 and ROUGE-2 score of our full model is comparable with DCA, and outperforms on ROUGE-L. Also, compared to extractive models NeuSUM and MASK- $LM^{global}$ , we achieve slight higher ROUGE-1. Except the four scores, our model outperforms these models on all the other scores, and since we have 95% confidence interval of at most $\pm $ 0.20, these improvements are statistically significant. As the last four lines of Table 1 show, we conduct an ablation study on our model variants to analyze the importance of each component. We use three ablation models for the experiments. One-Stage: A sequence-to-sequence model with copy mechanism based on BERT; Two-Stage: Adding the word-refine decoder to the One-Stage model; Two-Stage + RL: Full model with refine process cooperated with RL objective. First, we compare the Two-Stage+RL model with Two-Stage ablation, we observe that the full model outperforms by 0.30 on average ROUGE, suggesting that the reinforcement objective helps the model effectively. Then we analyze the effect of refine process by removing word-level refine from the Two-Stage model, we observe that without the refine process the average ROUGE score drops by 1.69. The ablation study shows that each module is necessary for our full model, and the improvements are statistically significant on all metrics. To evaluate the impact of summary length on model performance, we compare the average rouge score improvements of our model with different length of ground-truth summaries. As the above sub-figure of Figure 2 shows, compared to Pointer-Generator with Coverage, on length interval 40-80(occupies about 70% of test set) the improvements of our model are higher than shorter samples, confirms that with better context representations, in longer documents our model can achieve higher performance. As the below sub-figure of Figure 2 shows, compared to extractive baseline: Lead-3 BIBREF0 , the advantage of our model will fall when golden summary length is greater than 80. This probably because that we truncate the long documents and golden summaries and cannot get full information, it could also because that the training data in these intervals is too few to train an abstractive model, so simple extractive method will not fall too far behind. Additional Results on NYT50 Table 2 shows experiments on the NYT50 corpus. Since the short summary samples are filtered, NYT50 has average longer summaries than CNN/Daily Mail. So the model needs to catch long-term dependency of the sequences to generate good summaries. The first two lines of Table 2 show results of the two baselines introduced by BIBREF15 : these baselines select first n sentences, or select the first k words from the original document. Also we compare performance of our model with two recent models, we see 2.39 ROUGE-1 improvements compared to the ML+RL with intra-attn approach(previous SOTA) carries over to this dataset, which is a large margin. On ROUGE-2, our model also get an improvement of 0.51. The experiment proves that our approach can outperform competitive methods on different data distributions. Pre-trained language models Pre-trained word vectors BIBREF17 , BIBREF18 , BIBREF19 have been widely used in many NLP tasks. More recently, pre-trained language models (ELMo, GPT and BERT), have also achieved great success on several NLP problems such as textual entailment, semantic similarity, reading comprehension, and question answering BIBREF20 , BIBREF21 , BIBREF22 . Some recent works also focus on leveraging pre-trained language models in summarization. BIBREF23 pretrain a language model and use it as the sentiment analyser when generating reviews of goods. BIBREF24 train a language model on golden summaries, and then use it on the decoder side to incorporate prior knowledge. In this work, we use BERT(which is a pre-trained language model using large scale unlabeled data) on the encoder and decoder of a seq2seq model, and by designing a two stage decoding structure we build a competitive model for abstractive text summarization. Conclusion and Future Work In this work, we propose a two-stage model based on sequence-to-sequence paradigm. Our model utilize BERT on both encoder and decoder sides, and introduce reinforce objective in learning process. We evaluate our model on two benchmark datasets CNN/Daily Mail and New York Times, the experimental results show that compared to previous systems our approach effectively improves performance. Although our experiments are conducted on summarization task, our model can be used in most natural language generation tasks, such as machine translation, question generation and paraphrasing. The refine decoder and mixed objective can also be applied on other sequence generation tasks, and we will investigate on them in future work.
How are the different components of the model trained? Is it trained end-to-end?
feafcc1c4026d7f55a2c8ce7850d7e12030b5c22
How Ġare Ġthe Ġdifferent Ġcomponents Ġof Ġthe Ġmodel Ġtrained ? ĠIs Ġit Ġtrained Ġend - to - end ?
the Ġobjective Ġof Ġour Ġmodel Ġis Ġsum Ġof Ġthe Ġtwo Ġprocesses , Ġjointly Ġtrained Ġusing Ġ" te acher - forcing " Ġalgorithm , Ġwe Ġfeed Ġthe Ġground - truth Ġsummary Ġto Ġeach Ġdec oder Ġand Ġminimize Ġthe Ġobjective , ĠAt Ġtest Ġtime , Ġeach Ġtime Ġstep Ġwe Ġchoose Ġthe Ġpredicted Ġword Ġby Ġ$\ hat { y } Ġ= Ġarg max _{ y ^ {\ prime Ġ}} ĠP ( y ^ {\ prime Ġ} | x ) $ Ġ, Ġuse Ġbeam Ġsearch Ġto Ġgenerate Ġthe Ġdraft Ġsumm aries , Ġand Ġuse Ġgreedy Ġsearch Ġto Ġgenerate Ġthe Ġrefined Ġsumm aries ., Ġthe Ġmodel Ġcan Ġbe Ġtrained Ġend - to - end
<s> How Ġare Ġthe Ġdifferent Ġcomponents Ġof Ġthe Ġmodel Ġtrained ? ĠIs Ġit Ġtrained Ġend - to - end ? </s> Introduction </s> Text Ġsummar ization Ġgenerates Ġsumm aries Ġfrom Ġinput Ġdocuments Ġwhile Ġkeeping Ġsalient Ġinformation . ĠIt Ġis Ġan Ġimportant Ġtask Ġand Ġcan Ġbe Ġapplied Ġto Ġseveral Ġreal - world Ġapplications . ĠMany Ġmethods Ġhave Ġbeen Ġproposed Ġto Ġsolve Ġthe Ġtext Ġsummar ization Ġproblem ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ. ĠThere Ġare Ġtwo Ġmain Ġtext Ġsummar ization Ġtechniques : Ġextract ive Ġand Ġabstract ive . ĠExtract ive Ġsummar ization Ġgenerates Ġsummary Ġby Ġselecting Ġsalient Ġsentences Ġor Ġphrases Ġfrom Ġthe Ġsource Ġtext , Ġwhile Ġabstract ive Ġmethods Ġparaph rase Ġand Ġrest ructure Ġsentences Ġto Ġcompose Ġthe Ġsummary . ĠWe Ġfocus Ġon Ġabstract ive Ġsummar ization Ġin Ġthis Ġwork Ġas Ġit Ġis Ġmore Ġflexible Ġand Ġthus Ġcan Ġgenerate Ġmore Ġdiverse Ġsumm aries . </s> Recently , Ġmany Ġabstract ive Ġapproaches Ġare Ġintroduced Ġbased Ġon Ġneural Ġsequence - to - sequence Ġframework ĠB IB REF 4 Ġ, ĠB IB REF 0 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 5 Ġ. ĠBased Ġon Ġthe Ġsequence - to - sequence Ġmodel Ġwith Ġcopy Ġmechanism ĠB IB REF 6 Ġ, ĠB IB REF 0 Ġincorporates Ġa Ġcoverage Ġvector Ġto Ġtrack Ġand Ġcontrol Ġattention Ġscores Ġon Ġsource Ġtext . ĠB IB REF 4 Ġintroduce Ġintra - tem poral Ġattention Ġprocesses Ġin Ġthe Ġenc oder Ġand Ġdec oder Ġto Ġaddress Ġthe Ġrepetition Ġand Ġinco herent Ġproblem . </s> There Ġare Ġtwo Ġissues Ġin Ġprevious Ġabstract ive Ġmethods : Ġ1 ) Ġthese Ġmethods Ġuse Ġleft - context - only Ġdec oder , Ġthus Ġdo Ġnot Ġhave Ġcomplete Ġcontext Ġwhen Ġpredicting Ġeach Ġword . Ġ2 ) Ġthey Ġdo Ġnot Ġutilize Ġthe Ġpre - trained Ġcontextual ized Ġlanguage Ġmodels Ġon Ġthe Ġdec oder Ġside , Ġso Ġit Ġis Ġmore Ġdifficult Ġfor Ġthe Ġdec oder Ġto Ġlearn Ġsummary Ġrepresentations , Ġcontext Ġinteractions Ġand Ġlanguage Ġmodeling Ġtogether . </s> Recently , ĠB ERT Ġhas Ġbeen Ġsuccessfully Ġused Ġin Ġvarious Ġnatural Ġlanguage Ġprocessing Ġtasks , Ġsuch Ġas Ġtextual Ġentail ment , Ġname Ġentity Ġrecognition Ġand Ġmachine Ġreading Ġcomprehens ions . ĠIn Ġthis Ġpaper , Ġwe Ġpresent Ġa Ġnovel Ġnatural Ġlanguage Ġgeneration Ġmodel Ġbased Ġon Ġpre - trained Ġlanguage Ġmodels Ġ( we Ġuse ĠB ERT Ġin Ġthis Ġwork ). ĠAs Ġfar Ġas Ġwe Ġknow , Ġthis Ġis Ġthe Ġfirst Ġwork Ġto Ġextend ĠB ERT Ġto Ġthe Ġsequence Ġgeneration Ġtask . ĠTo Ġaddress Ġthe Ġabove Ġissues Ġof Ġprevious Ġabstract ive Ġmethods , Ġin Ġour Ġmodel , Ġwe Ġdesign Ġa Ġtwo - stage Ġdecoding Ġprocess Ġto Ġmake Ġgood Ġuse Ġof ĠB ERT 's Ġcontext Ġmodeling Ġability . ĠOn Ġthe Ġfirst Ġstage , Ġwe Ġgenerate Ġthe Ġsummary Ġusing Ġa Ġleft - context - only - dec oder . ĠOn Ġthe Ġsecond Ġstage , Ġwe Ġmask Ġeach Ġword Ġof Ġthe Ġsummary Ġand Ġpredict Ġthe Ġrefined Ġword Ġone - by - one Ġusing Ġa Ġrefine Ġdec oder . ĠTo Ġfurther Ġimprove Ġthe Ġnatural ness Ġof Ġthe Ġgenerated Ġsequence , Ġwe Ġcooperate Ġreinforcement Ġobjective Ġwith Ġthe Ġrefine Ġdec oder . </s> The Ġmain Ġcontributions Ġof Ġthis Ġwork Ġare : </s> 1 . ĠWe Ġpropose Ġa Ġnatural Ġlanguage Ġgeneration Ġmodel Ġbased Ġon ĠB ERT , Ġmaking Ġgood Ġuse Ġof Ġthe Ġpre - trained Ġlanguage Ġmodel Ġin Ġthe Ġenc oder Ġand Ġdec oder Ġprocess , Ġand Ġthe Ġmodel Ġcan Ġbe Ġtrained Ġend - to - end Ġwithout Ġhand crafted Ġfeatures . </s> 2 . ĠWe Ġdesign Ġa Ġtwo - stage Ġdec oder Ġprocess . ĠIn Ġthis Ġarchitecture , Ġour Ġmodel Ġcan Ġgenerate Ġeach Ġword Ġof Ġthe Ġsummary Ġconsidering Ġboth Ġsides ' Ġcontext Ġinformation . </s> 3 . ĠWe Ġconduct Ġexperiments Ġon Ġthe Ġbenchmark Ġdatasets ĠCNN / Daily ĠMail Ġand ĠNew ĠYork ĠTimes . ĠOur Ġmodel Ġachieves Ġa Ġ33 . 33 Ġaverage Ġof ĠR OU GE - 1 , ĠR OU GE - 2 Ġand ĠR OU GE - L Ġon Ġthe ĠCNN / Daily ĠMail , Ġwhich Ġis Ġstate - of - the - art . ĠOn Ġthe ĠNew ĠYork ĠTimes Ġdataset , Ġour Ġmodel Ġachieves Ġabout Ġ5 . 6 % Ġrelative Ġimprovement Ġover ĠR OU GE - 1 . </s> Text ĠSum mar ization </s> In Ġthis Ġpaper , Ġwe Ġfocus Ġon Ġsingle - document Ġmulti - sent ence Ġsummar ization Ġand Ġpropose Ġa Ġsupervised Ġabstract ive Ġmodel Ġbased Ġon Ġthe Ġneural Ġattentive Ġsequence - to - sequence Ġframework Ġwhich Ġconsists Ġof Ġtwo Ġparts : Ġa Ġneural Ġnetwork Ġfor Ġthe Ġenc oder Ġand Ġanother Ġnetwork Ġfor Ġthe Ġdec oder . ĠThe Ġenc oder Ġenc odes Ġthe Ġinput Ġsequence Ġto Ġintermediate Ġrepresentation Ġand Ġthe Ġdec oder Ġpredicts Ġone Ġword Ġat Ġa Ġtime Ġstep Ġgiven Ġthe Ġinput Ġsequence Ġrepresentation Ġvector Ġand Ġprevious Ġdec oded Ġoutput . ĠThe Ġgoal Ġof Ġthe Ġmodel Ġis Ġto Ġmaximize Ġthe Ġprobability Ġof Ġgenerating Ġthe Ġcorrect Ġtarget Ġsequences . ĠIn Ġthe Ġencoding Ġand Ġgeneration Ġprocess , Ġthe Ġattention Ġmechanism Ġis Ġused Ġto Ġconcentrate Ġon Ġthe Ġmost Ġimportant Ġpositions Ġof Ġtext . ĠThe Ġlearning Ġobjective Ġof Ġmost Ġsequence - to - sequence Ġmodels Ġis Ġto Ġminimize Ġthe Ġnegative Ġlog Ġlikelihood Ġof Ġthe Ġgenerated Ġsequence Ġas Ġfollowing Ġequation Ġshows , Ġwhere Ġ$ y ^ * _ i $ Ġis Ġthe Ġi - th Ġground - truth Ġsummary Ġtoken . </s> $$ L oss Ġ= Ġ- Ġ\ log Ġ\ sum Ġ_ { t = 1 } ^ N ĠP ( y _ t ^ * | y _{ < t } ^ *, ĠX ) $$ Ġ Ġ Ġ( E q . Ġ3 ) </s> However , Ġwith Ġthis Ġobjective , Ġtraditional Ġsequence Ġgeneration Ġmodels Ġconsider Ġonly Ġone Ġdirection Ġcontext Ġin Ġthe Ġdecoding Ġprocess , Ġwhich Ġcould Ġcause Ġperformance Ġdegradation Ġsince Ġcomplete Ġcontext Ġof Ġone Ġtoken Ġcontains Ġpreceding Ġand Ġfollowing Ġtokens , Ġthus Ġfeeding Ġonly Ġpreceded Ġdec oded Ġwords Ġto Ġthe Ġdec oder Ġso Ġthat Ġthe Ġmodel Ġmay Ġgenerate Ġunnatural Ġsequences . ĠFor Ġexample , Ġattentive Ġsequence - to - sequence Ġmodels Ġoften Ġgenerate Ġsequences Ġwith Ġrepeated Ġphrases Ġwhich Ġharm Ġthe Ġnatural ness . ĠSome Ġprevious Ġworks Ġmitigate Ġthis Ġproblem Ġby Ġimproving Ġthe Ġattention Ġcalculation Ġprocess , Ġbut Ġin Ġthis Ġpaper Ġwe Ġshow Ġthat Ġfeeding Ġbi - direction al Ġcontext Ġinstead Ġof Ġleft - only - context Ġcan Ġbetter Ġalleviate Ġthis Ġproblem . </s> Text Ġsummar ization Ġmodels Ġare Ġusually Ġclassified Ġto Ġabstract ive Ġand Ġextract ive Ġones . ĠRecently , Ġextract ive Ġmodels Ġlike ĠDeep Channel ĠB IB REF 8 Ġ, Ġr nn - ext + RL ĠB IB REF 9 Ġand ĠNe u S UM ĠB IB REF 2 Ġachieve Ġhigher Ġperformances Ġusing Ġwell - designed Ġstructures . ĠFor Ġexample , ĠDeep Channel Ġpropose Ġa Ġsal ience Ġestimation Ġnetwork Ġand Ġiter atively Ġextract Ġsalient Ġsentences . ĠB IB REF 16 Ġtrain Ġa Ġsentence Ġcompression Ġmodel Ġto Ġteach Ġanother Ġlatent Ġvariable Ġextract ive Ġmodel . </s> Also , Ġseveral Ġrecent Ġworks Ġfocus Ġon Ġimproving Ġabstract ive Ġmethods . ĠB IB REF 3 Ġdesign Ġa Ġcontent Ġselector Ġto Ġover - d eter mine Ġphrases Ġin Ġa Ġsource Ġdocument Ġthat Ġshould Ġbe Ġpart Ġof Ġthe Ġsummary . ĠB IB REF 11 Ġintroduce Ġinconsistency Ġloss Ġto Ġforce Ġwords Ġin Ġless Ġattended Ġsentences ( which Ġdetermined Ġby Ġextract ive Ġmodel ) Ġto Ġhave Ġlower Ġgeneration Ġprobabilities . ĠB IB REF 5 Ġextend Ġseq 2 seq Ġmodel Ġwith Ġan Ġinformation Ġselection Ġnetwork Ġto Ġgenerate Ġmore Ġinformative Ġsumm aries . </s> Bi - D irection al ĠPre - Tr ained ĠContext ĠEnc od ers </s> Recently , Ġcontext Ġenc od ers Ġsuch Ġas ĠEL Mo , ĠG PT , Ġand ĠB ERT Ġhave Ġbeen Ġwidely Ġused Ġin Ġmany ĠN LP Ġtasks . ĠThese Ġmodels Ġare Ġpre - trained Ġon Ġa Ġhuge Ġunl abel ed Ġcorpus Ġand Ġcan Ġgenerate Ġbetter Ġcontextual ized Ġtoken Ġembed d ings , Ġthus Ġthe Ġapproaches Ġbuilt Ġon Ġtop Ġof Ġthem Ġcan Ġachieve Ġbetter Ġperformance . </s> Since Ġour Ġmethod Ġis Ġbased Ġon ĠB ERT , Ġwe Ġillustrate Ġthe Ġprocess Ġbriefly Ġhere . ĠB ERT Ġconsists Ġof Ġseveral Ġlayers . ĠIn Ġeach Ġlayer Ġthere Ġis Ġfirst Ġa Ġmulti - head Ġself - att ention Ġsub - layer Ġand Ġthen Ġa Ġlinear Ġaff ine Ġsub - layer Ġwith Ġthe Ġresidual Ġconnection . ĠIn Ġeach Ġself - att ention Ġsub - layer Ġthe Ġattention Ġscores Ġ$ e _{ ij } $ Ġare Ġfirst Ġcalculated Ġas ĠE q . Ġ( Ġ5 Ġ) Ġ() Ġshows , Ġin Ġwhich Ġ$ d _ e $ Ġis Ġoutput Ġdimension , Ġand Ġ$ W ^ Q , ĠW ^ K , ĠW ^ V $ Ġare Ġparameter Ġmat rices . </s> $$ & a _{ ij } Ġ= Ġ\ c frac { ( h _ i W _ Q )( h _ j W _ K ) ^ T }{ \ sq rt { d _ e }} Ġ Ġ\\ Ġ& e _{ ij } Ġ= Ġ\ c frac {\ exp Ġ{ e _{ ij }} }{ \ sum Ġ_ { k = 1 } ^ N \ exp Ġ{ e _{ ik }}} Ġ$$ Ġ Ġ Ġ( E q . Ġ5 ) </s> Then Ġthe Ġoutput Ġis Ġcalculated Ġas ĠE q . Ġ( Ġ6 Ġ) Ġshows , Ġwhich Ġis Ġthe Ġweighted Ġsum Ġof Ġprevious Ġoutputs Ġ$ h $ Ġadded Ġby Ġprevious Ġoutput Ġ$ h _ i $ Ġ. ĠThe Ġlast Ġlayer Ġoutputs Ġis Ġcontext Ġencoding Ġof Ġinput Ġsequence . </s> $$ o _ i Ġ= Ġh _ i Ġ+ Ġ\ sum Ġ_ { j = 1 } ^{ N } Ġe _{ ij } ( h _ j ĠW _ V ) Ġ$$ Ġ Ġ Ġ( E q . Ġ6 ) </s> Despite Ġthe Ġwide Ġusage Ġand Ġhuge Ġsuccess , Ġthere Ġis Ġalso Ġa Ġmismatch Ġproblem Ġbetween Ġthese Ġpre - trained Ġcontext Ġenc od ers Ġand Ġsequence - to - sequence Ġmodels . ĠThe Ġissue Ġis Ġthat Ġwhile Ġusing Ġa Ġpre - trained Ġcontext Ġenc oder Ġlike ĠG PT Ġor ĠB ERT , Ġthey Ġmodel Ġtoken - level Ġrepresentations Ġby Ġconditioning Ġon Ġboth Ġdirection Ġcontext . ĠDuring Ġpre - training , Ġthey Ġare Ġfed Ġwith Ġcomplete Ġsequences . ĠHowever , Ġwith Ġa Ġleft - context - only Ġdec oder , Ġthese Ġpre - trained Ġlanguage Ġmodels Ġwill Ġsuffer Ġfrom Ġincomplete Ġand Ġinconsistent Ġcontext Ġand Ġthus Ġcannot Ġgenerate Ġgood Ġenough Ġcontext - aware Ġword Ġrepresentations , Ġespecially Ġduring Ġthe Ġinference Ġprocess . </s> Model </s> In Ġthis Ġsection , Ġwe Ġdescribe Ġthe Ġstructure Ġof Ġour Ġmodel , Ġwhich Ġlearns Ġto Ġgenerate Ġan Ġabstract ive Ġmulti - sent ence Ġsummary Ġfrom Ġa Ġgiven Ġsource Ġdocument . </s> Based Ġon Ġthe Ġsequence - to - sequence Ġframework Ġbuilt Ġon Ġtop Ġof ĠB ERT , Ġwe Ġfirst Ġdesign Ġa Ġrefine Ġdec oder Ġat Ġword - level Ġto Ġtackle Ġthe Ġtwo Ġproblems Ġdescribed Ġin Ġthe Ġabove Ġsection . ĠWe Ġalso Ġintroduce Ġa Ġdiscrete Ġobjective Ġfor Ġthe Ġrefine Ġdec od ers Ġto Ġreduce Ġthe Ġexposure Ġbias Ġproblem . ĠThe Ġoverall Ġstructure Ġof Ġour Ġmodel Ġis Ġillustrated Ġin ĠFigure Ġ1 Ġ. </s> Problem ĠForm ulation </s> We Ġdenote Ġthe Ġinput Ġdocument Ġas Ġ$ X Ġ= Ġ\ l brace Ġx _ 1 , Ġ\ ld ots Ġ, Ġx _ m \ r brace Ġ$ Ġwhere Ġ$ x _ i Ġ\ in Ġ\ math cal Ġ{ X } $ Ġrepresents Ġone Ġsource Ġtoken . ĠThe Ġcorresponding Ġsummary Ġis Ġden oted Ġas Ġ$ Y Ġ= Ġ\ l brace Ġy _ 1 , Ġ\ ld ots Ġ, Ġy _ L \ r brace Ġ$ Ġ, Ġ$ L $ Ġrepresents Ġthe Ġsummary Ġlength . </s> Given Ġinput Ġdocument Ġ$ X $ Ġ, Ġwe Ġfirst Ġpredict Ġthe Ġsummary Ġdraft Ġby Ġa Ġleft - context - only Ġdec oder , Ġand Ġthen Ġusing Ġthe Ġgenerated Ġsummary Ġdraft Ġwe Ġcan Ġcondition Ġon Ġboth Ġcontext Ġsides Ġand Ġrefine Ġthe Ġcontent Ġof Ġthe Ġsummary . ĠThe Ġdraft Ġwill Ġguide Ġand Ġconst rain Ġthe Ġrefine Ġprocess Ġof Ġsummary . </s> Summary ĠDraft ĠGeneration </s> The Ġsummary Ġdraft Ġis Ġbased Ġon Ġthe Ġsequence - to - sequence Ġmodel . ĠOn Ġthe Ġenc oder Ġside Ġthe Ġinput Ġdocument Ġ$ X $ Ġis Ġencoded Ġinto Ġrepresentation Ġvectors Ġ$ H Ġ= Ġ\ l brace Ġh _ 1 , Ġ\ ld ots Ġ, Ġh _ m \ r brace Ġ$ Ġ, Ġand Ġthen Ġfed Ġto Ġthe Ġdec oder Ġto Ġgenerate Ġthe Ġsummary Ġdraft Ġ$ A Ġ= Ġ\ l brace Ġa _ 1 , Ġ\ ld ots Ġ, Ġa _{ | a | }\ r brace Ġ$ Ġ. </s> We Ġsimply Ġuse ĠB ERT Ġas Ġthe Ġenc oder . ĠIt Ġfirst Ġmaps Ġthe Ġinput Ġsequence Ġto Ġword Ġembed d ings Ġand Ġthen Ġcomp utes Ġdocument Ġembed d ings Ġas Ġthe Ġenc oder 's Ġoutput , Ġden oted Ġby Ġfollowing Ġequation . </s> $$ H Ġ= ĠB ERT ( x _ 1 , Ġ\ ld ots Ġ, Ġx _ m ) $$ Ġ Ġ Ġ( E q . Ġ10 ) </s> In Ġthe Ġdraft Ġdec oder , Ġwe Ġfirst Ġintroduce ĠB ERT 's Ġword Ġembed ding Ġmatrix Ġto Ġmap Ġthe Ġprevious Ġsummary Ġdraft Ġoutputs Ġ$\ l brace Ġy _ 1 , Ġ\ ld ots Ġ, Ġy _{ t - 1 }\ r brace Ġ$ Ġinto Ġembed d ings Ġvectors Ġ$\ l brace Ġq _ 1 , Ġ\ ld ots Ġ, Ġq _{ t - 1 }\ r brace Ġ$ Ġat Ġt - th Ġtime Ġstep . ĠNote Ġthat Ġas Ġthe Ġinput Ġsequence Ġof Ġthe Ġdec oder Ġis Ġnot Ġcomplete , Ġwe Ġdo Ġnot Ġuse Ġthe ĠB ERT Ġnetwork Ġto Ġpredict Ġthe Ġcontext Ġvectors Ġhere . </s> Then Ġwe Ġintroduce Ġan Ġ$ N $ Ġlayer ĠTrans former Ġdec oder Ġto Ġlearn Ġthe Ġconditional Ġprobability Ġ$ P ( A | H ) $ Ġ. ĠTrans former 's Ġdec oder - enc oder Ġmulti - head Ġattention Ġhelps Ġthe Ġdec oder Ġlearn Ġsoft Ġalign ments Ġbetween Ġsummary Ġand Ġsource Ġdocument . ĠAt Ġthe Ġt - th Ġtime Ġstep , Ġthe Ġdraft Ġdec oder Ġpredicts Ġoutput Ġprobability Ġconditioned Ġon Ġprevious Ġoutputs Ġand Ġenc oder Ġhidden Ġrepresentations Ġas ĠE q . Ġ( Ġ13 Ġ) Ġshows , Ġin Ġwhich Ġ$ q _{ < t } Ġ= Ġ\ l brace Ġq _ 1 , Ġ\ ld ots Ġ, Ġq _{ t - 1 }\ r brace Ġ$ Ġ. ĠEach Ġgenerated Ġsequence Ġwill Ġbe Ġtrunc ated Ġin Ġthe Ġfirst Ġposition Ġof Ġa Ġspecial Ġtoken Ġ'[ P AD ] '. </s> $$ & P ^{ voc ab } _ t ( w ) Ġ= Ġf _{ dec } ( q _{ < t }, ĠH ) Ġ Ġ\\ Ġ& L _{ dec } Ġ= Ġ\ sum Ġ_ { i = 1 } ^{ | a | } Ġ- \ log ĠP ( a _ i Ġ= Ġy _ i ^ * | a _{ < Ġi }, ĠH ) Ġ$$ Ġ Ġ Ġ( E q . Ġ13 ) </s> As ĠE q . Ġ() Ġshows , Ġthe Ġdec oder 's Ġlearning Ġobjective Ġis Ġto Ġminimize Ġnegative Ġlikelihood Ġof Ġconditional Ġprobability , Ġin Ġwhich Ġ$ y _ i ^ * $ Ġis Ġthe Ġi - th Ġground Ġtruth Ġword Ġof Ġsummary . </s> However Ġa Ġdec oder Ġwith Ġthis Ġstructure Ġis Ġnot Ġsufficient Ġenough : Ġif Ġwe Ġuse Ġthe ĠB ERT Ġnetwork Ġin Ġthis Ġdec oder , Ġthen Ġduring Ġtraining Ġand Ġinference , Ġin - complete Ġcontext ( part Ġof Ġsentence ) Ġis Ġfed Ġinto Ġthe ĠB ERT Ġmodule , Ġand Ġalthough Ġwe Ġcan Ġfine - t une ĠB ERT 's Ġparameters , Ġthe Ġinput Ġdistribution Ġis Ġquite Ġdifferent Ġfrom Ġthe Ġpre - train Ġprocess , Ġand Ġthus Ġharms Ġthe Ġquality Ġof Ġgenerated Ġcontext Ġrepresentations . </s> If Ġwe Ġjust Ġuse Ġthe Ġembed ding Ġmatrix Ġhere , Ġit Ġwill Ġbe Ġmore Ġdifficult Ġfor Ġthe Ġdec oder Ġwith Ġfresh Ġparameters Ġto Ġlearn Ġto Ġmodel Ġrepresentations Ġas Ġwell Ġas Ġvocabulary Ġprobabilities , Ġfrom Ġa Ġrelative Ġsmall Ġcorpus Ġcompared Ġto ĠB ERT 's Ġhuge Ġpre - training Ġcorpus . ĠIn Ġa Ġword , Ġthe Ġdec oder Ġcannot Ġutilize ĠB ERT 's Ġability Ġto Ġgenerate Ġhigh Ġquality Ġcontext Ġvectors , Ġwhich Ġwill Ġalso Ġharm Ġperformance . </s> This Ġissue Ġexists Ġwhen Ġusing Ġany Ġother Ġcontextual ized Ġword Ġrepresentations , Ġso Ġwe Ġdesign Ġa Ġrefine Ġprocess Ġto Ġmitigate Ġit Ġin Ġour Ġapproach Ġwhich Ġwill Ġbe Ġdescribed Ġin Ġthe Ġnext Ġsub - section . </s> As Ġsome Ġsummary Ġtokens Ġare Ġout - of - voc abulary Ġwords Ġand Ġoccurs Ġin Ġinput Ġdocument , Ġwe Ġincorporate Ġcopy Ġmechanism ĠB IB REF 6 Ġbased Ġon Ġthe ĠTrans former Ġdec oder , Ġwe Ġwill Ġdescribe Ġit Ġbriefly . </s> At Ġdec oder Ġtime Ġstep Ġ$ t $ Ġ, Ġwe Ġfirst Ġcalculate Ġthe Ġattention Ġprobability Ġdistribution Ġover Ġsource Ġdocument Ġ$ X $ Ġusing Ġthe Ġbi - linear Ġdot Ġproduct Ġof Ġthe Ġlast Ġlayer Ġdec oder Ġoutput Ġof ĠTrans former Ġ$ o _ t $ Ġand Ġthe Ġenc oder Ġoutput Ġ$ h _ j $ Ġ, Ġas ĠE q . Ġ( Ġ15 Ġ) Ġ() Ġshows . </s> $$ u _ t ^ j Ġ= & Ġo _ t ĠW _ c Ġh _ j Ġ\\ Ġ\ alpha Ġ_ t ^ j Ġ= & Ġ\ c frac {\ exp Ġ{ u _ t ^ j }} {\ sum Ġ_ { k = 1 } ^ N \ exp Ġ{ u _ t ^ k }} Ġ$$ Ġ Ġ Ġ( E q . Ġ15 ) </s> We Ġthen Ġcalculate Ġcopying Ġgate Ġ$ g _ t \ in Ġ[ 0 , Ġ1 ] $ Ġ, Ġwhich Ġmakes Ġa Ġsoft Ġchoice Ġbetween Ġselecting Ġfrom Ġsource Ġand Ġgenerating Ġfrom Ġvocabulary , Ġ$ W _ c , ĠW _ g , Ġb _ g $ Ġare Ġparameters : </s> $$ g _ t Ġ= Ġs igm oid ( W _ g Ġ\ cd ot Ġ[ o _ t , Ġh ] Ġ+ Ġb _ g ) Ġ$$ Ġ Ġ Ġ( E q . Ġ16 ) </s> Using Ġ$ g _ t $ Ġwe Ġcalculate Ġthe Ġweighted Ġsum Ġof Ġcopy Ġprobability Ġand Ġgeneration Ġprobability Ġto Ġget Ġthe Ġfinal Ġpredicted Ġprobability Ġof Ġextended Ġvocabulary Ġ$\ math cal Ġ{ V } Ġ+ Ġ\ math cal Ġ{ X } $ Ġ, Ġwhere Ġ$\ math cal Ġ{ X } $ Ġis Ġthe Ġset Ġof Ġout Ġof Ġvocabulary Ġwords Ġfrom Ġthe Ġsource Ġdocument . ĠThe Ġfinal Ġprobability Ġis Ġcalculated Ġas Ġfollow : </s> $$ P _ t ( w ) Ġ= Ġ( 1 - g _ t ) P _ t ^{ voc ab } ( w ) Ġ+ Ġg _ t \ sum Ġ_ { i : w _ i = w } Ġ\ alpha Ġ_ t ^ i $$ Ġ Ġ Ġ( E q . Ġ17 ) </s> Summary ĠRef ine ĠProcess </s> The Ġmain Ġreason Ġto Ġintroduce Ġthe Ġrefine Ġprocess Ġis Ġto Ġenhance Ġthe Ġdec oder Ġusing ĠB ERT 's Ġcontextual ized Ġrepresentations , Ġso Ġwe Ġdo Ġnot Ġmodify Ġthe Ġenc oder Ġand Ġreuse Ġit Ġduring Ġthis Ġprocess . </s> On Ġthe Ġdec oder Ġside , Ġwe Ġpropose Ġa Ġnew Ġword - level Ġrefine Ġdec oder . ĠThe Ġrefine Ġdec oder Ġreceives Ġa Ġgenerated Ġsummary Ġdraft Ġas Ġinput , Ġand Ġoutputs Ġa Ġrefined Ġsummary . ĠIt Ġfirst Ġmasks Ġeach Ġword Ġin Ġthe Ġsummary Ġdraft Ġone Ġby Ġone , Ġthen Ġfeeds Ġthe Ġdraft Ġto ĠB ERT Ġto Ġgenerate Ġcontext Ġvectors . ĠFinally Ġit Ġpredicts Ġa Ġrefined Ġsummary Ġword Ġusing Ġan Ġ$ N $ Ġlayer ĠTrans former Ġdec oder Ġwhich Ġis Ġthe Ġsame Ġas Ġthe Ġdraft Ġdec oder . ĠAt Ġt - th Ġtime Ġstep Ġthe Ġn - th Ġword Ġof Ġinput Ġsummary Ġis Ġmasked , Ġand Ġthe Ġdec oder Ġpredicts Ġthe Ġn - th Ġrefined Ġword Ġgiven Ġother Ġwords Ġof Ġthe Ġsummary . </s> The Ġlearning Ġobjective Ġof Ġthis Ġprocess Ġis Ġshown Ġin ĠE q . Ġ( Ġ19 Ġ), Ġ$ y _ i $ Ġis Ġthe Ġi - th Ġsummary Ġword Ġand Ġ$ y _{ i } ^ * $ Ġfor Ġthe Ġground - truth Ġsummary Ġword , Ġand Ġ$ a _{ \ ne Ġi } Ġ= Ġ\ l brace Ġa _ 1 , Ġ\ ld ots Ġ, Ġa _{ i - 1 }, Ġa _{ i + 1 }, Ġ\ ld ots Ġ, Ġa _{ | y | }\ r brace Ġ$ Ġ. </s> $$ L _{ ref ine } Ġ= Ġ\ sum Ġ_ { i = 1 } ^{ | y | } Ġ- \ log ĠP ( y _ i Ġ= Ġy _ i ^ * | a _{ \ ne Ġi }, ĠH ) Ġ$$ Ġ Ġ Ġ( E q . Ġ19 ) </s> From Ġthe Ġview Ġof ĠB ERT Ġor Ġother Ġcontextual ized Ġembed d ings , Ġthe Ġrefine Ġdecoding Ġprocess Ġprovides Ġa Ġmore Ġcomplete Ġinput Ġsequence Ġwhich Ġis Ġconsistent Ġwith Ġtheir Ġpre - training Ġprocesses . ĠInt uitive ly , Ġthis Ġprocess Ġworks Ġas Ġfollows : Ġfirst Ġthe Ġdraft Ġdec oder Ġwrites Ġa Ġsummary Ġdraft Ġbased Ġon Ġa Ġdocument , Ġand Ġthen Ġthe Ġrefine Ġdec oder Ġedits Ġthe Ġdraft . ĠIt Ġconcent rates Ġon Ġone Ġword Ġat Ġa Ġtime , Ġbased Ġon Ġthe Ġsource Ġdocument Ġas Ġwell Ġas Ġother Ġwords . </s> We Ġdesign Ġthe Ġword - level Ġrefine Ġdec oder Ġbecause Ġthis Ġprocess Ġis Ġsimilar Ġto Ġthe Ġclo ze Ġtask Ġin ĠB ERT 's Ġpre - train Ġprocess , Ġtherefore Ġby Ġusing Ġthe Ġability Ġof Ġthe Ġcontextual Ġlanguage Ġmodel Ġthe Ġdec oder Ġcan Ġgenerate Ġmore Ġfluent Ġand Ġnatural Ġsequences . </s> The Ġparameters Ġare Ġshared Ġbetween Ġthe Ġdraft Ġdec oder Ġand Ġrefine Ġdec oder , Ġas Ġwe Ġfind Ġthat Ġusing Ġindividual Ġparameters Ġthe Ġmodel 's Ġperformance Ġde grades Ġa Ġlot . ĠThe Ġreason Ġmay Ġbe Ġthat Ġwe Ġuse Ġteach - forcing Ġduring Ġtraining , Ġand Ġthus Ġthe Ġword - level Ġrefine Ġdec oder Ġlearns Ġto Ġpredict Ġwords Ġgiven Ġall Ġthe Ġother Ġground - truth Ġwords Ġof Ġsummary . ĠThis Ġobjective Ġis Ġsimilar Ġto Ġthe Ġlanguage Ġmodel 's Ġpre - train Ġobjective , Ġand Ġis Ġprobably Ġnot Ġenough Ġfor Ġthe Ġdec oder Ġto Ġlearn Ġto Ġgenerate Ġrefined Ġsumm aries . ĠSo Ġin Ġour Ġmodel Ġall Ġdec od ers Ġshare Ġthe Ġsame Ġparameters . </s> Researchers Ġusually Ġuse ĠR OU GE Ġas Ġthe Ġevaluation Ġmetric Ġfor Ġsummar ization , Ġhowever Ġduring Ġsequence - to - sequence Ġmodel Ġtraining , Ġthe Ġobjective Ġis Ġto Ġmaximize Ġthe Ġlog Ġlikelihood Ġof Ġgenerated Ġsequences . ĠThis Ġmis - match Ġharms Ġthe Ġmodel 's Ġperformance , Ġso Ġwe Ġadd Ġa Ġdiscrete Ġobjective Ġto Ġthe Ġmodel , Ġand Ġoptimize Ġit Ġby Ġintroducing Ġthe Ġpolicy Ġgradient Ġmethod . ĠFor Ġexample , Ġthe Ġdiscrete Ġobjective Ġfor Ġthe Ġsummary Ġdraft Ġprocess Ġis Ġas ĠE q . Ġ( Ġ21 Ġ) Ġshows , Ġwhere Ġ$ a ^ s $ Ġis Ġthe Ġdraft Ġsummary Ġsampled Ġfrom Ġpredicted Ġdistribution , Ġand Ġ$ R ( a ^ s ) $ Ġis Ġthe Ġreward Ġscore Ġcompared Ġwith Ġthe Ġground - truth Ġsummary , Ġwe Ġuse ĠR OU GE - L Ġin Ġour Ġexperiment . ĠTo Ġbalance Ġbetween Ġoptimizing Ġthe Ġdiscrete Ġobjective Ġand Ġgenerating Ġreadable Ġsequences , Ġwe Ġmix Ġthe Ġdiscrete Ġobjective Ġwith Ġmaximum - like lihood Ġobjective . ĠAs ĠE q . Ġ() Ġshows , Ġminimizing Ġ$\ hat { L } _{ dec } $ Ġis Ġthe Ġfinal Ġobjective Ġfor Ġthe Ġdraft Ġprocess , Ġnote Ġhere Ġ$ L _{ dec } $ Ġis Ġ$ - log P ( a | x ) $ Ġ. ĠIn Ġthe Ġrefine Ġprocess Ġwe Ġintroduce Ġsimilar Ġobjectives . </s> $$ L ^{ rl } _{ dec } Ġ= ĠR ( a ^ s )\ cd ot Ġ[- \ log Ġ( P ( a ^ s | x )) ] Ġ Ġ\\ Ġ\ hat { L } _{ dec } Ġ= Ġ\ gam ma Ġ* ĠL ^{ rl } _{ dec } Ġ+ Ġ( 1 Ġ- Ġ\ gam ma Ġ) Ġ* ĠL _{ dec } Ġ$$ Ġ Ġ Ġ( E q . Ġ21 ) </s> Learning Ġand ĠIn ference </s> During Ġmodel Ġtraining , Ġthe Ġobjective Ġof Ġour Ġmodel Ġis Ġsum Ġof Ġthe Ġtwo Ġprocesses , Ġjointly Ġtrained Ġusing Ġ" te acher - forcing " Ġalgorithm . ĠDuring Ġtraining Ġwe Ġfeed Ġthe Ġground - truth Ġsummary Ġto Ġeach Ġdec oder Ġand Ġminimize Ġthe Ġobjective . </s> $$ L _{ model } Ġ= Ġ\ hat { L } _{ dec } Ġ+ Ġ\ hat { L } _{ ref ine } $$ Ġ Ġ Ġ( E q . Ġ23 ) </s> At Ġtest Ġtime , Ġeach Ġtime Ġstep Ġwe Ġchoose Ġthe Ġpredicted Ġword Ġby Ġ$\ hat { y } Ġ= Ġarg max _{ y ^ {\ prime Ġ}} ĠP ( y ^ {\ prime Ġ} | x ) $ Ġ, Ġuse Ġbeam Ġsearch Ġto Ġgenerate Ġthe Ġdraft Ġsumm aries , Ġand Ġuse Ġgreedy Ġsearch Ġto Ġgenerate Ġthe Ġrefined Ġsumm aries . </s> Settings </s> In Ġthis Ġwork , Ġall Ġof Ġour Ġmodels Ġare Ġbuilt Ġon Ġ$ BER T _{ B ASE } $ Ġ, Ġalthough Ġanother Ġlarger Ġpre - trained Ġmodel Ġwith Ġbetter Ġperformance Ġ( Ġ$ BER T _{ L AR GE } $ Ġ) Ġhas Ġpublished Ġbut Ġit Ġcosts Ġtoo Ġmuch Ġtime Ġand ĠGPU Ġmemory . ĠWe Ġuse ĠWord P iece Ġembed d ings Ġwith Ġa Ġ30 , 000 Ġvocabulary Ġwhich Ġis Ġthe Ġsame Ġas ĠB ERT . ĠWe Ġset Ġthe Ġlayer Ġof Ġtransformer Ġdec od ers Ġto Ġ12 ( 8 Ġon ĠNYT 50 ), Ġand Ġset Ġthe Ġattention Ġheads Ġnumber Ġto Ġ12 ( 8 Ġon ĠNYT 50 ), Ġset Ġfully - connected Ġsub - layer Ġhidden Ġsize Ġto Ġ30 72 . ĠWe Ġtrain Ġthe Ġmodel Ġusing Ġan ĠAdam Ġoptim izer Ġwith Ġlearning Ġrate Ġof Ġ$ 3 e - 4 $ Ġ, Ġ$\ beta Ġ_ 1 = 0 . 9 $ Ġ, Ġ$\ beta Ġ_ 2 = 0 . 999 $ Ġand Ġ$\ ep sil on Ġ= 10 ^{ - 9 } $ Ġand Ġuse Ġa Ġdynamic Ġlearning Ġrate Ġduring Ġthe Ġtraining Ġprocess . ĠFor Ġregular ization , Ġwe Ġuse Ġdrop out ĠB IB REF 13 Ġand Ġlabel Ġsmoot hing ĠB IB REF 14 Ġin Ġour Ġmodels Ġand Ġset Ġthe Ġdrop out Ġrate Ġto Ġ0 . 15 , Ġand Ġthe Ġlabel Ġsmoot hing Ġvalue Ġto Ġ0 . 1 . ĠWe Ġset Ġthe ĠRL Ġobjective Ġfactor Ġ$\ gam ma Ġ$ Ġto Ġ0 . 99 . </s> During Ġtraining , Ġwe Ġset Ġthe Ġbatch Ġsize Ġto Ġ36 , Ġand Ġtrain Ġfor Ġ4 Ġepoch s ( 8 Ġepoch s Ġfor ĠNYT 50 Ġsince Ġit Ġhas Ġmany Ġfewer Ġtraining Ġsamples ), Ġafter Ġtraining Ġthe Ġbest Ġmodel Ġare Ġselected Ġfrom Ġlast Ġ10 Ġmodels Ġbased Ġon Ġdevelopment Ġset Ġperformance . ĠDue Ġto ĠGPU Ġmemory Ġlimit , Ġwe Ġuse Ġgradient Ġaccumulation , Ġset Ġaccumulate Ġstep Ġto Ġ12 Ġand Ġfeed Ġ3 Ġsamples Ġat Ġeach Ġstep . ĠWe Ġuse Ġbeam Ġsize Ġ4 Ġand Ġlength Ġpenalty Ġof Ġ1 . 0 Ġto Ġgenerate Ġlogical Ġform Ġsequences . </s> We Ġfilter Ġrepeated Ġtri - gram s Ġin Ġbeam - search Ġprocess Ġby Ġsetting Ġword Ġprobability Ġto Ġzero Ġif Ġit Ġwill Ġgenerate Ġan Ġtri - gram Ġwhich Ġexists Ġin Ġthe Ġexisting Ġsummary . ĠIt Ġis Ġa Ġnice Ġmethod Ġto Ġavoid Ġphrase Ġrepetition Ġsince Ġthe Ġtwo Ġdatasets Ġseldom Ġcontains Ġrepeated Ġtri - gram s Ġin Ġone Ġsummary . ĠWe Ġalso Ġfine Ġtune Ġthe Ġgenerated Ġsequences Ġusing Ġanother Ġtwo Ġsimple Ġrules . ĠWhen Ġthere Ġare Ġmulti Ġsummary Ġsentences Ġwith Ġexactly Ġthe Ġsame Ġcontent , Ġwe Ġkeep Ġthe Ġfirst Ġone Ġand Ġremove Ġthe Ġother Ġsentences ; Ġwe Ġalso Ġremove Ġsentences Ġwith Ġless Ġthan Ġ3 Ġwords Ġfrom Ġthe Ġresult . </s> To Ġevaluate Ġthe Ġperformance Ġof Ġour Ġmodel , Ġwe Ġconduct Ġexperiments Ġon ĠCNN / Daily ĠMail Ġdataset , Ġwhich Ġis Ġa Ġlarge Ġcollection Ġof Ġnews Ġarticles Ġand Ġmodified Ġfor Ġsummar ization . ĠFollowing ĠB IB REF 0 Ġwe Ġchoose Ġthe Ġnon - an onym ized Ġversion Ġof Ġthe Ġdataset , Ġwhich Ġconsists Ġof Ġmore Ġthan Ġ280 , 000 Ġtraining Ġsamples Ġand Ġ11 490 Ġtest Ġset Ġsamples . </s> We Ġalso Ġconduct Ġexperiments Ġon Ġthe ĠNew ĠYork ĠTimes ( NY T ) Ġdataset Ġwhich Ġalso Ġconsists Ġof Ġmany Ġnews Ġarticles . ĠThe Ġoriginal Ġdataset Ġcan Ġbe Ġapplied Ġhere . ĠIn Ġour Ġexperiment , Ġwe Ġfollow Ġthe Ġdataset Ġsplits Ġand Ġother Ġpre - process Ġsettings Ġof ĠB IB REF 15 Ġ. ĠWe Ġfirst Ġfilter Ġall Ġsamples Ġwithout Ġa Ġfull Ġarticle Ġtext Ġor Ġabstract Ġand Ġthen Ġremove Ġall Ġsamples Ġwith Ġsumm aries Ġshorter Ġthan Ġ50 Ġwords . ĠThen Ġwe Ġchoose Ġthe Ġtest Ġset Ġbased Ġon Ġthe Ġdate Ġof Ġpublication ( all Ġexamples Ġpublished Ġafter ĠJanuary Ġ1 , Ġ2007 ). ĠThe Ġfinal Ġdataset Ġcontains Ġ22 , 000 Ġtraining Ġsamples Ġand Ġ3 , 452 Ġtest Ġsamples Ġand Ġis Ġcalled ĠNYT 50 Ġsince Ġall Ġsumm aries Ġare Ġlonger Ġthan Ġ50 Ġwords . </s> We Ġtoken ize Ġall Ġsequences Ġof Ġthe Ġtwo Ġdatasets Ġusing Ġthe ĠWord P iece Ġtoken izer . ĠAfter Ġtoken izing , Ġthe Ġaverage Ġarticle Ġlength Ġand Ġsummary Ġlength Ġof ĠCNN / Daily ĠMail Ġare Ġ6 91 Ġand Ġ51 , Ġand ĠNYT 50 's Ġaverage Ġarticle Ġlength Ġand Ġsummary Ġlength Ġare Ġ115 2 Ġand Ġ75 . ĠWe Ġtrunc ate Ġthe Ġarticle Ġlength Ġto Ġ512 , Ġand Ġthe Ġsummary Ġlength Ġto Ġ100 Ġin Ġour Ġexperiment ( max Ġsummary Ġlength Ġis Ġset Ġto Ġ150 Ġon ĠNYT 50 Ġas Ġits Ġaverage Ġgolden Ġsummary Ġlength Ġis Ġlonger ). </s> On ĠCNN / Daily ĠMail Ġdataset , Ġwe Ġreport Ġthe Ġfull - length ĠF - 1 Ġscore Ġof Ġthe ĠR OU GE - 1 , ĠR OU GE - 2 Ġand ĠR OU GE - L Ġmetrics , Ġcalculated Ġusing ĠPy R ou ge Ġpackage Ġand Ġthe ĠPorter Ġstem mer Ġoption . ĠOn ĠNYT 50 , Ġfollowing ĠB IB REF 4 Ġwe Ġevaluate Ġlimited Ġlength ĠR OU GE Ġrecall Ġscore ( limit Ġthe Ġgenerated Ġsummary Ġlength Ġto Ġthe Ġground Ġtruth Ġlength ). ĠWe Ġsplit ĠNYT 50 Ġsumm aries Ġinto Ġsentences Ġby Ġsem icol ons Ġto Ġcalculate Ġthe ĠR OU GE Ġscores . </s> Results Ġand ĠAnalysis </s> Table Ġ1 Ġshows Ġthe Ġresults Ġon ĠCNN / Daily ĠMail Ġdataset , Ġwe Ġcompare Ġthe Ġperformance Ġof Ġmany Ġrecent Ġapproaches Ġwith Ġour Ġmodel . ĠWe Ġclassify Ġthem Ġto Ġtwo Ġgroups Ġbased Ġon Ġwhether Ġthey Ġare Ġextract ive Ġor Ġabstract ive Ġmodels . ĠAs Ġthe Ġlast Ġline Ġof Ġthe Ġtable Ġshows , Ġthe ĠR OU GE - 1 Ġand ĠR OU GE - 2 Ġscore Ġof Ġour Ġfull Ġmodel Ġis Ġcomparable Ġwith ĠDC A , Ġand Ġoutper forms Ġon ĠR OU GE - L . ĠAlso , Ġcompared Ġto Ġextract ive Ġmodels ĠNe u S UM Ġand ĠMAS K - Ġ$ LM ^{ global } $ Ġ, Ġwe Ġachieve Ġslight Ġhigher ĠR OU GE - 1 . ĠExcept Ġthe Ġfour Ġscores , Ġour Ġmodel Ġoutper forms Ġthese Ġmodels Ġon Ġall Ġthe Ġother Ġscores , Ġand Ġsince Ġwe Ġhave Ġ95 % Ġconfidence Ġinterval Ġof Ġat Ġmost Ġ$\ pm Ġ$ Ġ0 . 20 , Ġthese Ġimprovements Ġare Ġstatistically Ġsignificant . </s> As Ġthe Ġlast Ġfour Ġlines Ġof ĠTable Ġ1 Ġshow , Ġwe Ġconduct Ġan Ġab lation Ġstudy Ġon Ġour Ġmodel Ġvariants Ġto Ġanalyze Ġthe Ġimportance Ġof Ġeach Ġcomponent . ĠWe Ġuse Ġthree Ġab lation Ġmodels Ġfor Ġthe Ġexperiments . ĠOne - Stage : ĠA Ġsequence - to - sequence Ġmodel Ġwith Ġcopy Ġmechanism Ġbased Ġon ĠB ERT ; ĠTwo - Stage : ĠAdding Ġthe Ġword - ref ine Ġdec oder Ġto Ġthe ĠOne - Stage Ġmodel ; ĠTwo - Stage Ġ+ ĠRL : ĠFull Ġmodel Ġwith Ġrefine Ġprocess Ġcooper ated Ġwith ĠRL Ġobjective . </s> First , Ġwe Ġcompare Ġthe ĠTwo - Stage + RL Ġmodel Ġwith ĠTwo - Stage Ġab lation , Ġwe Ġobserve Ġthat Ġthe Ġfull Ġmodel Ġoutper forms Ġby Ġ0 . 30 Ġon Ġaverage ĠR OU GE , Ġsuggesting Ġthat Ġthe Ġreinforcement Ġobjective Ġhelps Ġthe Ġmodel Ġeffectively . ĠThen Ġwe Ġanalyze Ġthe Ġeffect Ġof Ġrefine Ġprocess Ġby Ġremoving Ġword - level Ġrefine Ġfrom Ġthe ĠTwo - Stage Ġmodel , Ġwe Ġobserve Ġthat Ġwithout Ġthe Ġrefine Ġprocess Ġthe Ġaverage ĠR OU GE Ġscore Ġdrops Ġby Ġ1 . 69 . ĠThe Ġab lation Ġstudy Ġshows Ġthat Ġeach Ġmodule Ġis Ġnecessary Ġfor Ġour Ġfull Ġmodel , Ġand Ġthe Ġimprovements Ġare Ġstatistically Ġsignificant Ġon Ġall Ġmetrics . </s> To Ġevaluate Ġthe Ġimpact Ġof Ġsummary Ġlength Ġon Ġmodel Ġperformance , Ġwe Ġcompare Ġthe Ġaverage Ġrou ge Ġscore Ġimprovements Ġof Ġour Ġmodel Ġwith Ġdifferent Ġlength Ġof Ġground - truth Ġsumm aries . ĠAs Ġthe Ġabove Ġsub - figure Ġof ĠFigure Ġ2 Ġshows , Ġcompared Ġto ĠPo inter - Gener ator Ġwith ĠCoverage , Ġon Ġlength Ġinterval Ġ40 - 80 ( occup ies Ġabout Ġ70 % Ġof Ġtest Ġset ) Ġthe Ġimprovements Ġof Ġour Ġmodel Ġare Ġhigher Ġthan Ġshorter Ġsamples , Ġconfirms Ġthat Ġwith Ġbetter Ġcontext Ġrepresentations , Ġin Ġlonger Ġdocuments Ġour Ġmodel Ġcan Ġachieve Ġhigher Ġperformance . </s> As Ġthe Ġbelow Ġsub - figure Ġof ĠFigure Ġ2 Ġshows , Ġcompared Ġto Ġextract ive Ġbaseline : ĠLead - 3 ĠB IB REF 0 Ġ, Ġthe Ġadvantage Ġof Ġour Ġmodel Ġwill Ġfall Ġwhen Ġgolden Ġsummary Ġlength Ġis Ġgreater Ġthan Ġ80 . ĠThis Ġprobably Ġbecause Ġthat Ġwe Ġtrunc ate Ġthe Ġlong Ġdocuments Ġand Ġgolden Ġsumm aries Ġand Ġcannot Ġget Ġfull Ġinformation , Ġit Ġcould Ġalso Ġbecause Ġthat Ġthe Ġtraining Ġdata Ġin Ġthese Ġintervals Ġis Ġtoo Ġfew Ġto Ġtrain Ġan Ġabstract ive Ġmodel , Ġso Ġsimple Ġextract ive Ġmethod Ġwill Ġnot Ġfall Ġtoo Ġfar Ġbehind . </s> Additional ĠResults Ġon ĠNYT 50 </s> Table Ġ2 Ġshows Ġexperiments Ġon Ġthe ĠNYT 50 Ġcorpus . ĠSince Ġthe Ġshort Ġsummary Ġsamples Ġare Ġfiltered , ĠNYT 50 Ġhas Ġaverage Ġlonger Ġsumm aries Ġthan ĠCNN / Daily ĠMail . ĠSo Ġthe Ġmodel Ġneeds Ġto Ġcatch Ġlong - term Ġdependency Ġof Ġthe Ġsequences Ġto Ġgenerate Ġgood Ġsumm aries . </s> The Ġfirst Ġtwo Ġlines Ġof ĠTable Ġ2 Ġshow Ġresults Ġof Ġthe Ġtwo Ġbas elines Ġintroduced Ġby ĠB IB REF 15 Ġ: Ġthese Ġbas elines Ġselect Ġfirst Ġn Ġsentences , Ġor Ġselect Ġthe Ġfirst Ġk Ġwords Ġfrom Ġthe Ġoriginal Ġdocument . ĠAlso Ġwe Ġcompare Ġperformance Ġof Ġour Ġmodel Ġwith Ġtwo Ġrecent Ġmodels , Ġwe Ġsee Ġ2 . 39 ĠR OU GE - 1 Ġimprovements Ġcompared Ġto Ġthe ĠML + RL Ġwith Ġintra - att n Ġapproach ( pre vious ĠS OTA ) Ġcarries Ġover Ġto Ġthis Ġdataset , Ġwhich Ġis Ġa Ġlarge Ġmargin . ĠOn ĠR OU GE - 2 , Ġour Ġmodel Ġalso Ġget Ġan Ġimprovement Ġof Ġ0 . 51 . ĠThe Ġexperiment Ġproves Ġthat Ġour Ġapproach Ġcan Ġoutper form Ġcompetitive Ġmethods Ġon Ġdifferent Ġdata Ġdistributions . </s> Pre - trained Ġlanguage Ġmodels </s> Pre - trained Ġword Ġvectors ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ, ĠB IB REF 19 Ġhave Ġbeen Ġwidely Ġused Ġin Ġmany ĠN LP Ġtasks . ĠMore Ġrecently , Ġpre - trained Ġlanguage Ġmodels Ġ( EL Mo , ĠG PT Ġand ĠB ERT ), Ġhave Ġalso Ġachieved Ġgreat Ġsuccess Ġon Ġseveral ĠN LP Ġproblems Ġsuch Ġas Ġtextual Ġentail ment , Ġsemantic Ġsimilarity , Ġreading Ġcomprehension , Ġand Ġquestion Ġanswering ĠB IB REF 20 Ġ, ĠB IB REF 21 Ġ, ĠB IB REF 22 Ġ. </s> Some Ġrecent Ġworks Ġalso Ġfocus Ġon Ġleveraging Ġpre - trained Ġlanguage Ġmodels Ġin Ġsummar ization . ĠB IB REF 23 Ġpret rain Ġa Ġlanguage Ġmodel Ġand Ġuse Ġit Ġas Ġthe Ġsentiment Ġanalys er Ġwhen Ġgenerating Ġreviews Ġof Ġgoods . ĠB IB REF 24 Ġtrain Ġa Ġlanguage Ġmodel Ġon Ġgolden Ġsumm aries , Ġand Ġthen Ġuse Ġit Ġon Ġthe Ġdec oder Ġside Ġto Ġincorporate Ġprior Ġknowledge . </s> In Ġthis Ġwork , Ġwe Ġuse ĠB ERT ( which Ġis Ġa Ġpre - trained Ġlanguage Ġmodel Ġusing Ġlarge Ġscale Ġunl abel ed Ġdata ) Ġon Ġthe Ġenc oder Ġand Ġdec oder Ġof Ġa Ġseq 2 seq Ġmodel , Ġand Ġby Ġdesigning Ġa Ġtwo Ġstage Ġdecoding Ġstructure Ġwe Ġbuild Ġa Ġcompetitive Ġmodel Ġfor Ġabstract ive Ġtext Ġsummar ization . </s> Conclusion Ġand ĠFuture ĠWork </s> In Ġthis Ġwork , Ġwe Ġpropose Ġa Ġtwo - stage Ġmodel Ġbased Ġon Ġsequence - to - sequence Ġparadigm . ĠOur Ġmodel Ġutilize ĠB ERT Ġon Ġboth Ġenc oder Ġand Ġdec oder Ġsides , Ġand Ġintroduce Ġreinforce Ġobjective Ġin Ġlearning Ġprocess . ĠWe Ġevaluate Ġour Ġmodel Ġon Ġtwo Ġbenchmark Ġdatasets ĠCNN / Daily ĠMail Ġand ĠNew ĠYork ĠTimes , Ġthe Ġexperimental Ġresults Ġshow Ġthat Ġcompared Ġto Ġprevious Ġsystems Ġour Ġapproach Ġeffectively Ġimproves Ġperformance . </s> Although Ġour Ġexperiments Ġare Ġconducted Ġon Ġsummar ization Ġtask , Ġour Ġmodel Ġcan Ġbe Ġused Ġin Ġmost Ġnatural Ġlanguage Ġgeneration Ġtasks , Ġsuch Ġas Ġmachine Ġtranslation , Ġquestion Ġgeneration Ġand Ġparaph r asing . ĠThe Ġrefine Ġdec oder Ġand Ġmixed Ġobjective Ġcan Ġalso Ġbe Ġapplied Ġon Ġother Ġsequence Ġgeneration Ġtasks , Ġand Ġwe Ġwill Ġinvestigate Ġon Ġthem Ġin Ġfuture Ġwork .