SR
EXTRACTIVE
SR
EXTRACTIVE
Among all three pre-training tasks, SR works slightly better than the other two tasks (i.e., NSG and MDG). We also tried to randomly use all the three tasks during training with 1/3 probability each (indicated as ALL). Interesting, we observed that, in general, All outperforms all three tasks when employing unlabeled documents of training splits of CNNDM or NYT, which might be due to limited number of unlabeled documents of the training splits. After adding more data (i.e., GIAG-CM) to pre-training, SR consistently achieves highest ROUGE-2 on both CNNDM and NYT. We conclude that SR is the most effective task for pre-training since sentence reordering task requires comprehensively understanding a document in a wide coverage, going beyond individual words and sentences, which is highly close to the essense of abstractive document summarization.
Among all three pre-training tasks, SR works slightly better than the other two tasks (i.e., NSG and MDG). We also tried to randomly use all the three tasks during training with 1/3 probability each (indicated as ALL). Interesting, we observed that, in general, All outperforms all three tasks when employing unlabeled documents of training splits of CNNDM or NYT, which might be due to limited number of unlabeled documents of the training splits. After adding more data (i.e., GIAG-CM) to pre-training, SR consistently achieves highest ROUGE-2 on both CNNDM and NYT. We conclude that SR is the most effective task for pre-training since sentence reordering task requires comprehensively understanding a document in a wide coverage, going beyond individual words and sentences, which is highly close to the essense of abstractive document summarization.
2004.01853
Introduction </s> Large Ġpre - trained Ġlanguage Ġmodels ĠB IB REF 0 , ĠB IB REF 1 , ĠB IB REF 2 , ĠB IB REF 3 , ĠB IB REF 4 Ġimproved Ġthe Ġstate - of - the - art Ġof Ġvarious Ġnatural Ġlanguage Ġunderstanding Ġ( NL U ) Ġtasks Ġsuch Ġas Ġquestion Ġanswering Ġ( e . g ., ĠS Qu AD ; ĠB IB REF 5 ), Ġnatural Ġlanguage Ġinference Ġ( e . g ., ĠMN LI ; ĠB IB REF 6 ) Ġas Ġwell Ġas Ġtext Ġclassification ĠB IB REF 7 . ĠThese Ġmodels Ġ( i . e ., Ġlarge ĠL ST Ms ; ĠB IB REF 8 Ġor ĠTransformers ; ĠB IB REF 9 ) Ġare Ġpre - trained Ġon Ġlarge Ġscale Ġunl abel ed Ġtext Ġwith Ġlanguage Ġmodeling ĠB IB REF 0 , ĠB IB REF 1 , Ġmasked Ġlanguage Ġmodeling ĠB IB REF 2 , ĠB IB REF 4 Ġand Ġperm utation Ġlanguage Ġmodeling ĠB IB REF 3 Ġobjectives . ĠIn ĠNL U Ġtasks , Ġpre - trained Ġlanguage Ġmodels Ġare Ġmostly Ġused Ġas Ġtext Ġenc od ers . </s> Abstract ive Ġdocument Ġsummar ization Ġaims Ġto Ġrewrite Ġa Ġlong Ġdocument Ġto Ġits Ġshorter Ġform Ġwhile Ġstill Ġretaining Ġits Ġimportant Ġinformation . ĠDifferent Ġfrom Ġextract ive Ġdocument Ġsummar ization Ġthat Ġext acts Ġimportant Ġsentences , Ġabstract ive Ġdocument Ġsummar ization Ġmay Ġparaph rase Ġoriginal Ġsentences Ġor Ġdelete Ġcontents Ġfrom Ġthem . ĠFor Ġmore Ġdetails Ġon Ġdifferences Ġbetween Ġabstract ive Ġand Ġextract ive Ġdocument Ġsummary , Ġwe Ġrefer Ġthe Ġinterested Ġreaders Ġto ĠN en k ova : Mc Ke own : 2011 Ġand ĠSection ĠSEC REF 2 . ĠThis Ġtask Ġis Ġusually Ġframed Ġas Ġa Ġsequence - to - sequence Ġlearning Ġproblem ĠB IB REF 10 , ĠB IB REF 11 . ĠIn Ġthis Ġpaper , Ġwe Ġadopt Ġthe Ġsequence - to - sequence Ġ( seq 2 seq ) ĠTrans former ĠB IB REF 9 , Ġwhich Ġhas Ġbeen Ġdemonstrated Ġto Ġbe Ġthe Ġstate - of - the - art Ġfor Ġseq 2 seq Ġmodeling ĠB IB REF 9 , ĠB IB REF 12 . ĠUnfortunately , Ġtraining Ġlarge Ġseq 2 seq ĠTransformers Ġon Ġlimited Ġsupervised Ġsummar ization Ġdata Ġis Ġchallenging ĠB IB REF 12 Ġ( re fer Ġto ĠSection ĠSEC REF 5 ). ĠThe Ġseq 2 seq ĠTrans former Ġhas Ġan Ġenc oder Ġand Ġa Ġdec oder ĠTrans former . ĠAbstract ive Ġsummar ization Ġrequires Ġboth Ġencoding Ġof Ġan Ġinput Ġdocument Ġand Ġgeneration Ġof Ġa Ġsummary Ġusually Ġcontaining Ġmultiple Ġsentences . ĠAs Ġmentioned Ġearlier , Ġwe Ġcan Ġtake Ġadvantage Ġof Ġrecent Ġpre - trained ĠTrans former Ġenc od ers Ġfor Ġthe Ġdocument Ġencoding Ġpart Ġas Ġin Ġli u 2019 text . ĠHowever , Ġli u 2019 text Ġleave Ġthe Ġdec oder Ġrandomly Ġinitialized . ĠIn Ġthis Ġpaper , Ġwe Ġaim Ġto Ġpre - train Ġboth Ġthe Ġenc oder Ġ( i . e ., Ġthe Ġencoding Ġpart ) Ġand Ġdec oder Ġ( i . e ., Ġthe Ġgeneration Ġpart ) Ġof Ġa Ġseq 2 seq ĠTrans former Ġ, Ġwhich Ġis Ġable Ġto Ġimprove Ġabstract ive Ġsummar ization Ġperformance . </s> Based Ġon Ġthe Ġabove Ġobservations , Ġwe Ġpropose ĠStep Ġ( as Ġshorthand Ġfor ĠSequence - to - Sequ ence ĠTransform Er ĠPre - training ), Ġwhich Ġcan Ġbe Ġpre - trained Ġon Ġlarge Ġscale Ġunl abel ed Ġdocuments . ĠSpecifically , Ġwe Ġdesign Ġthree Ġtasks Ġfor Ġseq 2 seq Ġmodel Ġpre - training , Ġnamely ĠSent ence ĠRe ordering Ġ( SR ), ĠNext ĠSent ence ĠGeneration Ġ( NS G ), Ġand ĠMask ed ĠDocument ĠGeneration Ġ( MD G ). ĠSR Ġlearns Ġto Ġrecover Ġa Ġdocument Ġwith Ġrandomly Ġshuff led Ġsentences . ĠNS G Ġgenerates Ġthe Ġnext Ġsegment Ġof Ġa Ġdocument Ġbased Ġon Ġits Ġpreceding Ġsegment . ĠMD G Ġrecovers Ġa Ġmasked Ġdocument Ġto Ġits Ġoriginal Ġform . ĠAfter Ġpre - t rian ing ĠStep Ġusing Ġthe Ġthree Ġtasks Ġon Ġunl abel ed Ġdocuments , Ġwe Ġfine - t une Ġit Ġon Ġsupervised Ġsummar ization Ġdatasets . </s> We Ġevaluate Ġour Ġmethods Ġon Ġtwo Ġsummar ization Ġdatasets Ġ( i . e ., Ġthe ĠCNN / Daily Mail Ġand Ġthe ĠNew ĠYork ĠTimes Ġdatasets ). ĠExper iments Ġshow Ġthat Ġall Ġthree Ġtasks Ġwe Ġpropose Ġcan Ġimprove Ġupon Ġa Ġheavily Ġtuned Ġlarge Ġseq 2 seq ĠTrans former Ġwhich Ġalready Ġincludes Ġa Ġstrong Ġpre - trained Ġenc oder Ġby Ġa Ġlarge Ġmargin . ĠCompared Ġto Ġthe Ġbest Ġpublished Ġabstract ive Ġmodels , ĠStep Ġimproves Ġthe ĠR OU GE - 2 Ġby Ġ0 . 8 Ġon Ġthe ĠCNN / Daily Mail Ġdataset Ġand Ġby Ġ2 . 4 Ġon Ġthe ĠNew ĠYork ĠTimes Ġdataset Ġusing Ġour Ġbest Ġperforming Ġtask Ġfor Ġpre - training . ĠHuman Ġexperiments Ġalso Ġshow Ġthat ĠStep Ġcan Ġproduce Ġsignificantly Ġbetter Ġsumm aries Ġin Ġcomparison Ġwith Ġrecent Ġstrong Ġabstract ive Ġmodels . </s> Related ĠWork </s> This Ġsection Ġintroduces Ġextract ive Ġand Ġabstract ive Ġdocument Ġsummar ization Ġas Ġwell Ġas Ġpre - training Ġmethods Ġfor Ġnatural Ġlanguage Ġprocessing Ġtasks . </s> Related ĠWork Ġ: :: ĠExtract ive ĠSum mar ization </s> Ext ractive Ġsummar ization Ġsystems Ġlearn Ġto Ġfind Ġthe Ġinformative Ġsentences Ġin Ġa Ġdocument Ġas Ġits Ġsummary . ĠThis Ġtask Ġis Ġusually Ġviewed Ġas Ġa Ġsentence Ġranking Ġproblem ĠB IB REF 13 , ĠB IB REF 14 Ġusing Ġscores Ġfrom Ġa Ġbinary Ġ( sequence ) Ġclassification Ġmodel , Ġwhich Ġpredicts Ġwhether Ġa Ġsentence Ġis Ġin Ġthe Ġsummary Ġor Ġnot . ĠExtract ive Ġneural Ġmodels Ġemploy Ġhierarchical ĠL ST Ms / CNN s Ġas Ġthe Ġfeature Ġlearning Ġpart Ġof Ġthe Ġbinary Ġ( sequence ) Ġclass ifier ĠB IB REF 15 , ĠB IB REF 16 , ĠB IB REF 17 , ĠB IB REF 18 , Ġwhich Ġlargely Ġoutper forms Ġdiscrete Ġfeature Ġbased Ġmodels ĠB IB REF 19 , ĠB IB REF 20 , ĠB IB REF 21 . ĠVery Ġrecently , Ġthe Ġfeature Ġlearning Ġpart Ġwas Ġreplaced Ġagain Ġwith Ġpre - trained Ġtransform ers ĠB IB REF 22 , ĠB IB REF 23 Ġthat Ġlead Ġto Ġanother Ġhuge Ġimprovement Ġof Ġsummar ization Ġperformance . ĠHowever , Ġextract ive Ġmodels Ġhave Ġtheir Ġown Ġlimitations . ĠFor Ġexample , Ġthe Ġextracted Ġsentences Ġmight Ġbe Ġtoo Ġlong Ġand Ġredundant . ĠBesides , Ġhuman Ġwritten Ġsumm aries Ġin Ġtheir Ġnature Ġare Ġabstract ive . ĠTherefore , Ġwe Ġfocus Ġon Ġabstract ive Ġsummar ization Ġin Ġthis Ġpaper . </s> Related ĠWork Ġ: :: ĠAbstract ive ĠSum mar ization </s> The Ġgoal Ġof Ġabstract ive Ġsummar ization Ġis Ġto Ġgenerate Ġsumm aries Ġby Ġrewriting Ġa Ġdocument , Ġwhich Ġis Ġa Ġsequence - to - sequence Ġlearning Ġproblem . Ġseq 2 seq Ġattentive ĠL ST Ms ĠB IB REF 8 , ĠB IB REF 24 Ġare Ġemployed Ġin Ġn all ap ati 2016 ab stract ive . ĠEven Ġthese Ġmodels Ġare Ġextended Ġwith Ġcopy Ġmechanism ĠB IB REF 25 , Ġcoverage Ġmodel ĠB IB REF 11 Ġand Ġreinforcement Ġlearning ĠB IB REF 26 , Ġtheir Ġresults Ġare Ġstill Ġvery Ġclose Ġto Ġthat Ġof ĠLead 3 Ġwhich Ġselects Ġthe Ġleading Ġthree Ġsentences Ġof Ġa Ġdocument Ġas Ġits Ġsummary . ĠOne Ġpossible Ġreason Ġis Ġthat ĠL ST Ms Ġwithout Ġpre - training Ġare Ġnot Ġpowerful Ġenough . Ġli u 2019 text Ġused Ġa Ġseq 2 seq ĠTrans former Ġmodel Ġwith Ġits Ġenc oder Ġinitialized Ġwith Ġa Ġpre - trained ĠTrans former Ġ( i . e ., ĠB ERT ; ĠB IB REF 2 ) Ġand Ġachieved Ġthe Ġstate - of - the - art Ġperformance . ĠOur Ġwork Ġgoes Ġone Ġstep Ġfurther , Ġwe Ġpropose Ġa Ġmethod Ġto Ġpre - train Ġthe Ġdec oder Ġtogether Ġwith Ġthe Ġenc oder Ġand Ġthen Ġinitialize Ġboth Ġthe Ġenc oder Ġand Ġdec oder Ġof Ġa Ġsummar ization Ġmodel Ġwith Ġthe Ġpre - trained ĠTransformers . </s> There Ġis Ġalso Ġa Ġline Ġof Ġwork Ġthat Ġbridges Ġextract ive Ġand Ġabstract ive Ġmodels Ġwith Ġreinforcement Ġlearning ĠB IB REF 27 , Ġattention Ġfusion ĠB IB REF 28 Ġand Ġbottom - up Ġattention ĠB IB REF 29 , Ġwhile Ġour Ġmodel Ġis Ġconcept ually Ġsimpler . </s> Related ĠWork Ġ: :: ĠPre - training </s> Pre - training Ġmethods Ġdraw Ġa Ġlot Ġof Ġattention Ġrecently . Ġpet ers 2018 deep Ġand Ġrad ford : 2019 : ar x iv Ġpre - trained ĠL ST M Ġand ĠTrans former Ġenc od ers Ġusing Ġlanguage Ġmodeling Ġobjectives . ĠTo Ġleverage Ġthe Ġcontext Ġin Ġboth Ġdirections , ĠB IB REF 2 Ġproposed ĠB ERT , Ġwhich Ġis Ġtrained Ġwith Ġthe Ġmask Ġlanguage Ġmodeling Ġobjective . ĠXL Net ĠB IB REF 3 Ġis Ġtrained Ġwith Ġperm utation Ġlanguage Ġmodeling Ġobjective , Ġwhich Ġremoves Ġthe Ġindependence Ġassumption Ġof Ġmasked Ġtokens Ġin ĠB ERT . ĠRo BER Ta ĠB IB REF 4 Ġextends ĠB ERT Ġwith Ġmore Ġtraining Ġdata Ġand Ġbetter Ġtraining Ġstrategies . ĠAll Ġthe Ġmethods Ġabove Ġfocus Ġon Ġpre - training Ġan Ġenc oder , Ġwhile Ġwe Ġpropose Ġmethods Ġto Ġpre - train Ġboth Ġthe Ġenc oder Ġand Ġdec oder Ġof Ġa Ġseq 2 seq Ġmodel . </s> d ong 2019 un ified Ġproposed Ġa ĠTrans former Ġlanguage Ġmodel Ġthat Ġcan Ġbe Ġused Ġfor Ġboth Ġnatural Ġlanguage Ġunderstanding Ġand Ġgeneration Ġtasks , Ġwhich Ġis Ġpre - trained Ġusing Ġmasked , Ġun id irection al Ġand Ġseq 2 seq Ġlanguage Ġmodeling Ġobjectives . ĠTheir Ġmethod Ġtries Ġto Ġpre - train Ġa Ġseq 2 seq ĠTrans former Ġwith Ġits Ġenc oder Ġand Ġdec oder Ġparameters Ġshared . ĠDifferent ly , Ġwe Ġpre - train Ġa Ġseq 2 seq ĠTrans former Ġwith Ġseparate Ġparameters Ġfor Ġthe Ġenc oder Ġand Ġdec oder . Ġsong 2019 mass Ġproposed Ġa Ġmethod Ġto Ġpre - train Ġa Ġseq 2 seq ĠTrans former Ġby Ġmask ing Ġa Ġspan Ġof Ġtext Ġand Ġthen Ġpredicting Ġthe Ġoriginal Ġtext Ġwith Ġmasked Ġtokens Ġat Ġother Ġpositions . ĠTheir Ġpre - training Ġtask Ġis Ġsimilar Ġto Ġour ĠMask ed ĠDocument ĠGeneration Ġtask , Ġbut Ġwe Ġapply Ġa Ġdifferent Ġmask ing Ġstrategy Ġand Ġpredict Ġthe Ġoriginal Ġtext Ġwithout Ġmasked Ġtokens . ĠBesides , Ġwe Ġpropose Ġanother Ġtwo Ġtasks Ġfor Ġseq 2 seq Ġmodel Ġpre - training . ĠB IB REF 30 Ġtested Ġtheir Ġmodel Ġon Ġsentence - level Ġtasks Ġ( e . g ., Ġmachine Ġtranslation Ġand Ġsentence Ġcompression ), Ġwhile Ġwe Ġaim Ġto Ġsolve Ġdocument - level Ġtasks Ġ( e . g ., Ġabstract ive Ġdocument Ġsummar ization ). </s> Sequ ence - to - Sequ ence ĠTrans former ĠPre - training </s> This Ġsection Ġfirst Ġintroduces Ġthe Ġbackbone Ġarchitecture Ġof Ġour Ġabstract ive Ġsummar ization Ġmodel ĠStep . ĠWe Ġthen Ġdescribe Ġmethods Ġto Ġpre - train ĠStep Ġand Ġfinally Ġmove Ġon Ġto Ġthe Ġfine - tun ing Ġon Ġsummar ization Ġdatasets . </s> Sequ ence - to - Sequ ence ĠTrans former ĠPre - training Ġ: :: ĠArchitecture </s> In Ġthis Ġwork , Ġthe Ġtask Ġof Ġabstract ive Ġdocument Ġsummar ization Ġis Ġmodeled Ġas Ġa Ġsequence - to - sequence Ġlearning Ġproblem , Ġwhere Ġa Ġdocument Ġis Ġviewed Ġas Ġa Ġsequence Ġof Ġtokens Ġand Ġits Ġcorresponding Ġsummary Ġas Ġanother Ġsequence Ġof Ġtokens . ĠWe Ġadopt Ġthe Ġseq 2 seq ĠTrans former Ġarchitecture ĠB IB REF 9 , Ġwhich Ġincludes Ġan Ġenc oder ĠTrans former Ġand Ġa Ġdec oder ĠTrans former . ĠBoth Ġthe Ġenc oder Ġand Ġdec oder ĠTransformers Ġhave Ġmultiple Ġlayers Ġand Ġeach Ġlayer Ġcontains Ġa Ġmulti - head Ġattentive Ġsub - layer Ġfollowed Ġby Ġa Ġfully Ġconnected Ġsub - layer Ġwith Ġresidual Ġconnections ĠB IB REF 31 Ġand Ġlayer Ġnormal ization ĠB IB REF 32 . </s> Let Ġus Ġuse Ġ$ X Ġ= Ġ( x _ 1 , Ġx _ 2 , Ġ\ d ots Ġ, Ġx _{ | X | }) $ Ġto Ġdenote Ġa Ġdocument Ġand Ġuse Ġ$ Y Ġ= Ġ( y _ 1 , Ġy _ 2 , Ġ\ d ots Ġ, Ġy _{ | Y | }) $ Ġto Ġdenote Ġits Ġsummary . ĠThe Ġenc oder Ġtakes Ġthe Ġdocument Ġ$ X $ Ġas Ġinput Ġand Ġtransforms Ġit Ġto Ġits Ġcontextual Ġrepresentations . ĠThe Ġdec oder Ġlearns Ġto Ġgenerate Ġthe Ġsummary Ġ$ Y $ Ġone Ġtoken Ġat Ġa Ġtime Ġbased Ġon Ġthe Ġcontextual Ġrepresentations Ġand Ġall Ġpreceding Ġtokens Ġthat Ġhave Ġbeen Ġgenerated Ġso Ġfar : </s> where Ġ$ y _{ < t } $ Ġstands Ġfor Ġall Ġtokens Ġbefore Ġposition Ġ$ t $ Ġ( i . e ., Ġ$ y _{ < t } =( y _ 1 , Ġy _ 2 , Ġ\ d ots Ġ, Ġy _{ t - 1 }) $ ). ĠThis Ġmodel Ġcan Ġbe Ġtrained Ġby Ġminimizing Ġthe Ġnegative Ġlog - like lihood Ġof Ġthe Ġtraining Ġdocument - summary Ġpairs . </s> Sequ ence - to - Sequ ence ĠTrans former ĠPre - training Ġ: :: ĠPre - training ĠT asks </s> Training Ġa Ġseq 2 seq ĠTrans former Ġmodel Ġon Ġa Ġsummar ization Ġdataset Ġfrom Ġscratch Ġis Ġdifficult Ġdue Ġto Ġthe Ġlimited Ġnumber Ġof Ġdocument - summary Ġpairs . ĠPre - trained ĠTrans former Ġenc od ers Ġsuch Ġas ĠB ERT ĠB IB REF 2 Ġand ĠRo BER Ta ĠB IB REF 4 Ġhave Ġachieved Ġgreat Ġsuccess Ġin Ġmany Ġnatural Ġlanguage Ġunderstanding Ġtasks . ĠTherefore , Ġwe Ġfirst Ġinitialize Ġthe Ġenc oder Ġof Ġour Ġseq 2 seq ĠTrans former Ġsummar ization Ġmodel ĠStep Ġwith Ġan Ġexisting Ġpre - trained ĠTrans former Ġenc oder Ġ( i . e ., ĠRo BER Ta ) Ġto Ġenhance Ġits Ġlanguage Ġunderstanding Ġcapabilities . ĠTo Ġhelp ĠStep Ġgain Ġlanguage Ġgeneration Ġcapabilities Ġand Ġthe Ġabilities Ġof Ġassoci ating Ġgenerated Ġtext Ġwith Ġenc oder Ġoutputs , Ġwe Ġcontinue Ġto Ġpre - train Ġit Ġon Ġunl abel ed Ġtext . ĠIn Ġthe Ġfollowing , Ġwe Ġdescribe Ġour Ġpre - training Ġtasks . </s> Sequ ence - to - Sequ ence ĠTrans former ĠPre - training Ġ: :: ĠPre - training ĠT asks Ġ: :: ĠSent ence ĠRe ordering </s> A Ġdocument Ġis Ġtypically Ġcomposed Ġof Ġmultiple Ġsentences Ġseparated Ġby Ġfull Ġstops . ĠIn Ġthis Ġtask , Ġwe Ġfirst Ġshuffle Ġthe Ġdocument Ġby Ġsentences Ġand Ġthen Ġrecover Ġthe Ġoriginal Ġdocument . ĠThere Ġare Ġseveral Ġreasons Ġwhy Ġwe Ġdesign Ġthis Ġtask . ĠFirst , Ġa Ġsummary Ġof Ġa Ġdocument Ġusually Ġconsists Ġof Ġmultiple Ġsentences . ĠWe Ġexpect Ġthat ĠStep Ġlearns Ġto Ġgenerate Ġlong Ġand Ġcoherent Ġsumm aries Ġ( ac ross Ġsentences ). ĠThe Ġoutput Ġof Ġthe Ġtask Ġ( i . e ., Ġthe Ġoriginal Ġdocument ) Ġalso Ġcontains Ġmultiple Ġsentences . ĠSecond , Ġsentence Ġre ordering Ġ( or Ġcontent Ġre ordering ) Ġis Ġnecessary Ġfor Ġsummar ization . ĠAccording Ġto Ġthe Ġstatistics Ġon Ġtraining Ġsets Ġof Ġour Ġsummar ization Ġdatasets , Ġcontents Ġof Ġthe Ġoriginal Ġdocuments Ġare Ġre ordered Ġin Ġtheir Ġsumm aries Ġfor Ġ40 % Ġof Ġcases . ĠWe Ġdefine Ġcontent Ġre ordering Ġas Ġfollows . ĠFor Ġeach Ġdocument - summary Ġpair , Ġwe Ġfirst Ġmap Ġeach Ġsentence Ġin Ġthe Ġsummary Ġto Ġone Ġsentence Ġin Ġits Ġpaired Ġdocument Ġby Ġmaximizing Ġthe ĠR OU GE Ġscore . ĠIf Ġthe Ġsequence Ġof Ġsentences Ġin Ġthe Ġsummary Ġis Ġdifferent Ġfrom Ġthe Ġsequence Ġof Ġtheir Ġmapped Ġsentences Ġin Ġthe Ġoriginal Ġdocument , Ġwe Ġcount Ġthis Ġas Ġone Ġcontent Ġre ordering . ĠThird ly , Ġabstract ive Ġsummary Ġrequires Ġreprodu cing Ġfactual Ġdetails Ġ( e . g ., Ġnamed Ġentities , Ġfigures ) Ġfrom Ġsource Ġtext . ĠWe Ġalso Ġexpect ĠStep Ġto Ġlearn Ġto Ġcopy Ġtokens . ĠHere Ġis Ġa Ġformal Ġdefinition Ġof Ġthis Ġtask . ĠLet Ġus Ġchange Ġthe Ġnotation Ġof Ġa Ġdocument Ġslightly Ġin Ġthis Ġparagraph . ĠLet Ġ$ X =( S _ 1 , ĠS _ 2 , Ġ\ d ots Ġ, ĠS _ m ) $ Ġdenote Ġa Ġdocument , Ġwhere Ġ$ S _ i Ġ= Ġ( w ^ i _ 1 , Ġw ^ i _ 2 , Ġ\ d ots Ġ, Ġw ^ i _{ | S _ i | }) $ Ġis Ġa Ġsentence Ġin Ġit , Ġ$ w ^ i _ j $ Ġis Ġa Ġword Ġin Ġ$ S _ i $ Ġand Ġ$ m $ Ġis Ġthe Ġnumber Ġof Ġsentences . Ġ$ X $ Ġis Ġstill Ġa Ġsequence Ġof Ġtokens Ġ( by Ġconc aten ating Ġtokens Ġin Ġall Ġsentences ). ĠLet Ġ$ A = \ text {\ tt Ġperm utation } ( m )=( a _ 1 , a _ 2 , \ d ots Ġ, Ġa _ m ) $ Ġdenote Ġa Ġperm uted Ġrange Ġof Ġ$( 1 , Ġ2 , Ġ\ d ots Ġ, Ġm ) $ Ġand Ġtherefore Ġ$\ hat { X } _ S =( S _{ a _ 1 }, ĠS _{ a _ 2 }, Ġ\ d ots Ġ, ĠS _{ a _ m }) $ Ġis Ġthe Ġshuff led Ġdocument . ĠNote Ġthat Ġ$\ hat { X } _ S $ Ġis Ġa Ġsequence Ġof Ġtokens Ġby Ġconc aten ating Ġall Ġshuff led Ġsentences . ĠStep Ġcan Ġbe Ġtrained Ġon Ġ$\ l angle Ġ\ hat { X } _ S , ĠX Ġ\ r angle Ġ$ Ġpairs Ġconstructed Ġfrom Ġunl abel ed Ġdocuments , Ġas Ġdemonstrated Ġin ĠFigure ĠFIG REF 5 . </s> Note Ġthat Ġdocument Ġrotation Ġis Ġa Ġspecial Ġcase Ġof Ġsentence Ġre ordering Ġwith Ġsignificant Ġamount Ġof Ġpartially Ġordered Ġsentences , Ġwhich Ġwe Ġbelieve Ġis Ġa Ġsimpler Ġtask . ĠIn Ġthis Ġwork , Ġwe Ġthus Ġonly Ġconsider Ġthe Ġgeneral Ġcase Ġof Ġsentence Ġre ordering . </s> Sequ ence - to - Sequ ence ĠTrans former ĠPre - training Ġ: :: ĠPre - training ĠT asks Ġ: :: ĠNext ĠSent ence ĠGeneration </s> The Ġsecond Ġpre - training Ġtask Ġlever ages Ġthe Ġnatural Ġorder Ġof Ġtext . ĠNext ĠSent ence ĠGeneration Ġ( NS G ) Ġuses Ġone Ġspan Ġof Ġtext Ġin Ġa Ġdocument Ġto Ġpredict Ġits Ġnext Ġspan Ġof Ġtext , Ġas Ġshown Ġin ĠFigure ĠFIG REF 5 . ĠSpecifically , Ġwe Ġsplit Ġa Ġdocument Ġinto Ġtwo Ġsegments Ġ( i . e ., Ġ$ G _ 1 $ Ġand Ġ$ G _ 2 $ ). ĠNote Ġthat Ġeach Ġsegment Ġmight Ġcontain Ġmultiple Ġsentences , Ġwhich Ġfits Ġthe Ġdocument Ġsummar ization Ġtask Ġvery Ġwell , Ġsince Ġeither Ġa Ġdocument Ġor Ġits Ġsummary Ġusually Ġincludes Ġmultiple Ġsentences . ĠInt uitive ly , Ġin Ġa Ġdocument , Ġsentences Ġare Ġhighly Ġcorrelated Ġwith Ġtheir Ġpreceding Ġsentences Ġdue Ġto Ġthe Ġcontext Ġdependent Ġnature Ġof Ġdocuments Ġor Ġlanguage . ĠWe Ġintend Ġour Ġmodel Ġto Ġlearn Ġto Ġgenerate Ġmultiple Ġsentences Ġand Ġalso Ġlearn Ġto Ġfocus Ġon Ġpreceding Ġcontext . </s> We Ġhave Ġat Ġleast Ġtwo Ġoptions Ġfor Ġthe Ġsplitting Ġposition Ġof Ġthe Ġtwo Ġsegments . ĠOption Ġone : Ġthe Ġposition Ġright Ġafter Ġa Ġfull - stop Ġsymbol Ġ( such Ġas Ġperiod , Ġquestion Ġmark , Ġetc .) Ġis Ġselected Ġas Ġthe Ġsplitting Ġpoint , Ġwhich Ġensures Ġfull Ġsentences Ġfor Ġeach Ġsegment . ĠOption Ġtwo : Ġthe Ġsplitting Ġpoint Ġcan Ġbe Ġat Ġany Ġposition Ġwithin Ġthe Ġdocument . ĠWe Ġchoose Ġthe Ġsecond Ġoption , Ġwhich Ġmay Ġlead Ġto Ġincomplete Ġsentences Ġin Ġsegments . ĠWe Ġintend Ġto Ġforce Ġthe Ġenc oder Ġand Ġdec oder Ġto Ġunderstand Ġinput Ġtext Ġwithout Ġcomplete Ġinformation , Ġwhich Ġwe Ġbelieve Ġis Ġmore Ġchallenging Ġcompared Ġto Ġoption Ġone . ĠBesides , Ġas Ġa Ġcommon Ġwisdom Ġin Ġabstract ive Ġsummar ization , Ġdocuments Ġare Ġtrunc ated Ġto Ġa Ġfixed Ġnumber Ġof Ġtokens , Ġwhich Ġmay Ġalso Ġcontain Ġincomplete Ġsentences . ĠWe Ġuse Ġoption Ġtwo Ġto Ġreduce Ġthe Ġpre - training Ġand Ġfine - tun ing Ġinput Ġmismatch . ĠIn Ġthis Ġtask , Ġwe Ġtrain Ġthe Ġmodel ĠStep Ġon Ġlarge Ġamount Ġof Ġ$\ l angle ĠG _ 1 , ĠG _ 2 \ r angle Ġ$ Ġpairs Ġconstructed Ġfollowing Ġthe Ġoption Ġtwo Ġsplitting Ġstrategy . </s> Next Ġsentence Ġprediction Ġhas Ġbeen Ġused Ġin Ġskip - thought Ġvectors ĠB IB REF 33 . ĠThere Ġare Ġtwo Ġdifferences . ĠFirst , Ġeach Ġsegment Ġin Ġtheir Ġmodel Ġonly Ġhas Ġone Ġsentence ; Ġsecond , Ġthey Ġuse Ġthis Ġtask Ġto Ġpre - train Ġan Ġenc oder Ġrather Ġthan Ġan Ġentire Ġseq 2 seq Ġmodel . ĠB IB REF 2 Ġintroduced Ġa Ġtask Ġnamed Ġnext Ġsentence Ġprediction Ġ( N SP ), Ġwhich Ġis Ġdifferent Ġfrom Ġthis Ġtask . ĠN SP Ġis Ġa Ġclassification Ġtask , Ġbut ĠNS G Ġis Ġa Ġgeneration Ġtask , Ġwhich Ġintends Ġto Ġpre - train Ġa Ġgeneration Ġmodel . </s> Sequ ence - to - Sequ ence ĠTrans former ĠPre - training Ġ: :: ĠPre - training ĠT asks Ġ: :: ĠMask ed ĠDocument ĠGeneration </s> The Ġthird Ġtask Ġwe Ġconsider Ġis ĠMask ed ĠDocument ĠGeneration Ġ( MD G ) Ġthat Ġlearns Ġto Ġrecover Ġa Ġdocument Ġwith Ġa Ġmasked Ġspan Ġof Ġtokens Ġ( see ĠFigure ĠFIG REF 5 ). ĠFor Ġsimplicity , Ġa Ġdocument Ġconsisting Ġof Ġa Ġsequence Ġof Ġtokens Ġis Ġden oted Ġas Ġ$ X =( x _ 1 , Ġx _ 2 , Ġ\ cd ots Ġ, Ġx _{ | X | }) $. ĠWe Ġrandomly Ġsample Ġthe Ġlength Ġof Ġthe Ġspan Ġ$ l $ Ġfrom Ġa Ġdiscrete Ġuniform Ġdistribution Ġ$\ math cal Ġ{ U } ( a , Ġb ) $ Ġand Ġthe Ġspan Ġstart Ġposition Ġ$ k $ Ġfrom Ġanother Ġdiscrete Ġuniform Ġdistribution Ġ$\ math cal Ġ{ U } ( 1 , Ġ| X | - l + 1 ) $ Ġ( see ĠSection ĠSEC REF 4 Ġfor Ġmore Ġdetails ). ĠThus , Ġ$\ math cal Ġ{ M } =( x _ k , Ġx _{ k + 1 }, Ġ\ cd ots Ġ, Ġx _{ k + l - 1 }) $ Ġis Ġthe Ġtext Ġspan Ġto Ġbe Ġmasked . </s> One Ġstraightforward Ġmask ing Ġstrategy Ġis Ġto Ġreplace Ġeach Ġtoken Ġresiding Ġin Ġ$\ math cal Ġ{ M } $ Ġwith Ġa Ġspecial Ġ[ MAS K ] Ġtoken . ĠHowever , Ġwe Ġrefrain Ġfrom Ġdoing Ġso Ġbecause Ġof Ġthe Ġfollowing Ġthree Ġreasons . ĠUsually , Ġ[ MAS K ] Ġtokens Ġwill Ġnot Ġappear Ġin Ġdownstream Ġtasks . ĠSecond , Ġentirely Ġmask ing Ġa Ġcontinuous Ġsub - sequence Ġof Ġ$ X $ Ġmay Ġmake Ġthe Ġwhole Ġdocument Ġincomprehensible , Ġwhich Ġmight Ġbe Ġtoo Ġchallenging Ġfor Ġour Ġmodel Ġto Ġlearn . ĠThird , Ġsimilar Ġto ĠSR , Ġavoiding Ġreplacing Ġevery Ġtoken Ġwith Ġ[ MAS K ] Ġalso Ġhelps Ġour Ġmodel Ġlearn Ġthe Ġability Ġof Ġcopying Ġtokens Ġfrom Ġthe Ġinput Ġwhile Ġpreserving Ġthe Ġability Ġof Ġgenerating Ġnovel Ġtokens . </s> In Ġthe Ġsub - sequence Ġ$\ math cal Ġ{ M } $, Ġeach Ġtoken Ġis Ġprocessed Ġwith Ġone Ġof Ġthe Ġthree Ġstrategies : Ġ1 ) Ġreplaced Ġwith Ġthe Ġ[ MAS K ] Ġtoken ; Ġ2 ) Ġreplaced Ġwith Ġa Ġrandom Ġtoken ; Ġ3 ) Ġremain Ġunchanged . ĠInspired Ġby ĠB ERT ĠB IB REF 2 , Ġfor Ġ80 % Ġtokens , Ġwe Ġfollow Ġstrategy Ġ1 ). ĠIn Ġ10 % Ġof Ġcases , Ġwe Ġemploy Ġstrategy Ġ2 ) Ġand Ġwe Ġuse Ġstrategy Ġ3 ) Ġfor Ġthe Ġremaining Ġ10 % Ġof Ġcases . ĠLet Ġ$\ hat { X } _ M $ Ġdenote Ġthe Ġdocument Ġafter Ġthe Ġapplication Ġof Ġour Ġmask ing Ġstrategy . ĠWe Ġcould Ġcreate Ġinfinite Ġamount Ġof Ġ$\ l angle Ġ\ hat { X } _ M , X \ r angle Ġ$ Ġpairs Ġto Ġtrain ĠStep . </s> During Ġpre - training , Ġwe Ġcould Ġalso Ġemploy Ġall Ġthe Ġthree Ġtasks Ġ( i . e ., ĠSR , ĠNS G , ĠMD G ) Ġtogether . ĠFor Ġeach Ġtraining Ġbatch , Ġwe Ġrandomly Ġchoose Ġone Ġtask Ġand Ġeach Ġtask Ġis Ġused Ġfor Ġ$ 1 / 3 $ Ġof Ġthe Ġtime . </s> Sequ ence - to - Sequ ence ĠTrans former ĠPre - training Ġ: :: ĠFine - tun ing </s> After Ġpre - training ĠStep Ġwith Ġthe Ġthree Ġtasks Ġintroduced Ġin ĠSection ĠSEC REF 9 , Ġwe Ġfine - t une Ġthe Ġmodel Ġon Ġabstract ive Ġdocument Ġsummar ization Ġdatasets . ĠThe Ġfine - tun ing Ġprocess Ġis Ġstraightforward . ĠWe Ġsimply Ġcontinue Ġto Ġtrain ĠStep Ġon Ġthe Ġsupervised Ġdocument - summary Ġpairs . ĠSimilar Ġto Ġother Ġseq 2 seq Ġsummar ization Ġmodels , Ġwe Ġdo Ġbeam Ġsearch Ġduring Ġthe Ġgeneration Ġof Ġsumm aries . </s> Exper imental ĠSetup </s> In Ġthis Ġsection , Ġwe Ġpresent Ġthe Ġexperimental Ġsetup Ġfor Ġevaluating Ġour Ġsummar ization Ġmodels . ĠWe Ġfirst Ġintroduce Ġthe Ġdatasets Ġused Ġfor Ġour Ġexperiments . ĠThen Ġwe Ġdescribe Ġtraining Ġdetails Ġof Ġour Ġmodels Ġas Ġwell Ġas Ġour Ġevaluation Ġprotocols . </s> Exper imental ĠSetup Ġ: :: ĠDat as ets </s> We Ġassess Ġthe Ġsummar ization Ġperformance Ġof Ġour Ġmodels Ġon Ġtwo Ġbenchmark Ġdatasets : Ġthe ĠCNN / Daily Mail Ġ( CNN DM ) Ġdataset ĠB IB REF 34 , ĠB IB REF 11 Ġand Ġthe ĠNew ĠYork ĠTimes Ġ( NY T ) Ġdataset ĠB IB REF 35 . ĠWe Ġpre - train Ġour Ġmodels Ġon Ġthe ĠG IG A - CM Ġdataset Ġintroduced Ġin Ġz hang - et al - 2019 - h iber t . </s> Exper imental ĠSetup Ġ: :: ĠDat as ets Ġ: :: ĠCNN DM </s> CNN DM Ġcontains Ġnews Ġarticles Ġand Ġtheir Ġassociated Ġhighlights Ġ( i . e ., Ġsumm aries ) Ġcollected Ġfrom Ġthe ĠCNN Ġand ĠDaily ĠMail ĠOnline Ġwebsites . ĠFollowing Ġprevious Ġwork ĠB IB REF 11 , ĠB IB REF 22 , ĠB IB REF 23 , Ġwe Ġuse Ġthe Ġnon - an onym ized Ġversion Ġof ĠCNN DM . ĠSpecifically , Ġwe Ġpre process Ġthe Ġdataset Ġwith Ġthe Ġpublicly Ġavailable Ġscripts Ġprovided Ġby Ġsee 2017 get Ġand Ġobtain Ġ287 , 226 Ġdocument - summary Ġpairs Ġfor Ġtraining , Ġ13 , 368 Ġfor Ġvalidation Ġand Ġ11 , 490 Ġfor Ġtest . </s> Exper imental ĠSetup Ġ: :: ĠDat as ets Ġ: :: ĠNYT </s> The ĠNYT Ġdataset Ġis Ġa Ġcollection Ġof Ġarticles Ġalong Ġwith Ġmulti - sent ence Ġsumm aries Ġwritten Ġby Ġlibrary Ġscientists . ĠWe Ġclosely Ġfollow Ġthe Ġpre processing Ġprocedures Ġdescribed Ġin Ġdur rett 2016 learning Ġand Ġli u 2019 text . ĠThe Ġtest Ġset Ġis Ġconstructed Ġby Ġincluding Ġall Ġarticles Ġpublished Ġon ĠJanuary Ġ1 , Ġ2017 Ġor Ġlater , Ġwhich Ġcontains Ġ9 , 07 6 Ġarticles . ĠThe Ġremaining Ġ100 , 8 34 Ġarticles Ġare Ġsplit Ġinto Ġa Ġtraining Ġset Ġof Ġ96 , 8 34 Ġexamples Ġand Ġa Ġvalidation Ġset Ġof Ġ4 , 000 Ġexamples . ĠAs Ġin ĠB IB REF 36 , Ġwe Ġalso Ġremove Ġarticles Ġwhose Ġsumm aries Ġcontain Ġless Ġthan Ġ50 Ġwords Ġfrom Ġthe Ġtest Ġset , Ġand Ġthe Ġresulting Ġtest Ġset Ġcontains Ġ3 , 452 Ġexamples . </s> Exper imental ĠSetup Ġ: :: ĠDat as ets Ġ: :: ĠG IG A - CM </s> To Ġpre - train Ġour Ġmodel Ġwith Ġthe Ġtasks Ġintroduced Ġin ĠSection ĠSEC REF 9 , Ġfollowing Ġthe Ġprocedures Ġin ĠB IB REF 22 , Ġwe Ġcreated Ġthe ĠG IG A - CM Ġdataset , Ġwhich Ġcontains Ġonly Ġunl abel ed Ġdocuments . ĠThe Ġtraining Ġset Ġof ĠG IG A - CM Ġis Ġcomposed Ġof Ġ6 , 5 21 , 658 Ġdocuments Ġsampled Ġfrom Ġthe ĠEnglish ĠGig aw ord Ġdataset Ġand Ġthe Ġtraining Ġdocuments Ġin ĠCNN DM . ĠWe Ġused Ġthe Ġ13 , 368 Ġdocuments Ġin Ġthe Ġvalidation Ġsplit Ġof ĠCNN DM Ġas Ġits Ġvalidation Ġset . ĠNote Ġthat Ġthe ĠGig aw ord Ġdataset Ġoverl aps Ġwith Ġthe ĠNYT Ġdataset Ġand Ġwe Ġtherefore Ġexclude Ġthe Ġtest Ġset Ġof ĠNYT Ġfrom Ġthe Ġtraining Ġset Ġof ĠG IG A - CM . </s> For ĠCNN DM , ĠNYT Ġand ĠG IG A - CM Ġdatasets , Ġwe Ġsegment Ġand Ġtoken ize Ġdocuments Ġand / or Ġsumm aries Ġ( G IG A - CM Ġonly Ġcontains Ġdocuments ) Ġusing Ġthe ĠStanford ĠCore N LP Ġtool kit ĠB IB REF 37 . ĠTo Ġreduce Ġthe Ġvocabulary Ġsize , Ġwe Ġfurther Ġapply Ġthe ĠUTF 8 Ġbased ĠB PE ĠB IB REF 38 Ġintroduced Ġin ĠG PT - 2 ĠB IB REF 39 Ġto Ġall Ġdatasets . ĠAs Ġa Ġcommon Ġwisdom Ġin Ġabstract ive Ġsummar ization , Ġdocuments Ġand Ġsumm aries Ġin ĠCNN DM Ġand ĠNYT Ġare Ġusually Ġtrunc ated Ġto Ġ512 Ġand Ġ256 Ġtokens , Ġrespectively . </s> We Ġleverage Ġunl abel ed Ġdocuments Ġdifferently Ġfor Ġdifferent Ġpre - training Ġtasks Ġ( see ĠSection ĠSEC REF 9 ). ĠWe Ġfirst Ġsplit Ġeach Ġdocument Ġinto Ġ512 Ġtoken Ġsegments Ġif Ġit Ġcontains Ġmore Ġthan Ġ512 Ġtokens Ġ( se gments Ġor Ġdocuments Ġwith Ġless Ġthan Ġ512 Ġtokens Ġare Ġremoved ). ĠIn ĠSent ence ĠRe ordering Ġ( SR ) Ġand ĠMask ed ĠDocument ĠGeneration Ġ( MD G ), Ġwe Ġuse Ġthe Ġsegment Ġafter Ġtransformation Ġto Ġpredict Ġthe Ġoriginal Ġsegment . ĠWe Ġset Ġthe Ġminimum Ġmasked Ġlength Ġ$ a = 100 $ Ġand Ġthe Ġmaximum Ġmasked Ġlength Ġ$ b = 256 $ Ġin ĠMD G . ĠIn ĠNext ĠSent ence ĠGeneration Ġ( NS G ), Ġeach Ġsegment Ġis Ġused Ġto Ġpredict Ġits Ġnext Ġ256 Ġtokens . </s> Exper imental ĠSetup Ġ: :: ĠImplementation ĠDetails </s> As Ġmentioned Ġin ĠSection ĠSEC REF 3 , Ġour Ġmodel Ġis Ġa ĠSe q 2 Se q ĠTrans former Ġmodel ĠB IB REF 9 . ĠThe Ġenc oder Ġis Ġinitialized Ġwith Ġthe Ġ$\ text { Ro BER Ta } _{ \ text { L AR GE }} $ Ġmodel ĠB IB REF 4 , Ġand Ġtherefore Ġthey Ġshare Ġthe Ġsame Ġarchitecture . ĠSpecifically , Ġthe Ġenc oder Ġis Ġa Ġ24 - layer ĠTrans former . ĠEach Ġlayer Ġhas Ġ16 Ġattention Ġheads Ġand Ġits Ġhidden Ġsize Ġand Ġfeed - forward Ġfilter Ġsize Ġare Ġ1 , 024 Ġand Ġ4 , 09 6 , Ġrespectively . ĠThe Ġdec oder Ġis Ġshall ower Ġwith Ġ6 Ġlayers . ĠThe Ġhidden Ġsize Ġand Ġnumber Ġof Ġattention Ġhead Ġof Ġthe Ġdec oder Ġare Ġidentical Ġto Ġthese Ġof Ġthe Ġenc oder , Ġbut Ġthe Ġfeed - forward Ġfilter Ġsize Ġis Ġ2 , 048 . ĠWe Ġuse Ġa Ġsmaller Ġfilter Ġsize Ġin Ġthe Ġdec oder Ġto Ġreduce Ġthe Ġcomputational Ġand Ġmemory Ġcost . ĠThe Ġdrop out Ġrates Ġof Ġall Ġlayers Ġin Ġthe Ġenc oder Ġare Ġset Ġto Ġ0 . 1 Ġand Ġall Ġdrop out Ġrates Ġin Ġthe Ġdec oder Ġare Ġset Ġto Ġ0 . 3 . ĠOur Ġmodels Ġare Ġoptimized Ġusing ĠAdam ĠB IB REF 40 Ġwith Ġ$\ beta Ġ_ 1 = 0 . 9 $, Ġ$\ beta Ġ_ 2 = 0 . 98 $. ĠThe Ġother Ġoptimization Ġhyper - param eters Ġfor Ġpre - training Ġand Ġfine - tun ing Ġare Ġdifferent . ĠIn Ġthe Ġpre - training Ġstage , Ġthe Ġenc oder Ġis Ġinitialized Ġwith Ġa Ġpre - trained Ġmodel Ġwhile Ġthe Ġdec oder Ġis Ġrandomly Ġinitialized . ĠTherefore , Ġwe Ġused Ġtwo Ġseparate Ġoptim izers Ġfor Ġthe Ġenc oder Ġand Ġdec oder Ġwith Ġa Ġsmaller Ġlearning Ġrate Ġfor Ġthe Ġenc oder Ġoptim izer . ĠLearning Ġrates Ġof Ġthe Ġenc oder Ġand Ġdec oder Ġare Ġset Ġto Ġ$ 2 e - 5 $ Ġand Ġ$ 1 e - 4 $ Ġwith Ġ10 , 000 Ġwarm up Ġsteps , Ġrespectively . ĠWe Ġalso Ġadopted Ġthe Ġsame Ġlearning Ġrate Ġschedule Ġstrategies Ġas ĠB IB REF 9 . ĠWe Ġused Ġsmaller Ġbatch Ġsizes Ġfor Ġdatasets Ġwith Ġless Ġexamples Ġ( i . e ., Ġ1 , 024 Ġfor ĠG IG A - CM , Ġ256 Ġfor ĠCNN DM Ġand Ġ128 Ġfor ĠNYT ) Ġto Ġensure Ġeach Ġepoch Ġhas Ġsufficient Ġnumber Ġof Ġmodel Ġupdates . ĠWe Ġtrained Ġour Ġmodels Ġuntil Ġtheir Ġconvergence Ġof Ġvalidation Ġperplex ities Ġ( around Ġ30 Ġepoch s Ġon ĠG IG A - CM , Ġ60 Ġepoch s Ġon ĠCNN DM Ġand Ġ40 Ġepoch s Ġon ĠNYT ). ĠOne Ġepoch Ġon ĠG IG A - CM Ġtakes Ġaround Ġ24 Ġhours Ġwith Ġ8 ĠNvidia ĠTesla ĠV 100 ĠGPUs . ĠThe Ġtime Ġcosts Ġfor Ġdifferent Ġpre - training Ġtasks Ġare Ġclose . </s> Most Ġof Ġthe Ġhyper - param eters Ġin Ġthe Ġfine - tun ing Ġstage Ġare Ġthe Ġsame Ġas Ġthese Ġin Ġthe Ġpre - training Ġstage . ĠThe Ġdifferences Ġare Ġas Ġfollows . ĠThe Ġlearning Ġrates Ġfor Ġboth Ġthe Ġenc oder Ġand Ġdec oder Ġare Ġset Ġto Ġ$ 2 e - 5 $ Ġwith Ġ4 , 000 Ġwarm up Ġsteps , Ġsince Ġboth Ġthe Ġenc oder Ġand Ġdec oder Ġare Ġalready Ġpre - trained . ĠWe Ġtrained Ġour Ġmodels Ġfor Ġ50 Ġepoch s Ġ( s aved Ġper Ġepoch ) Ġand Ġselected Ġthe Ġbest Ġmodel Ġw . r . t . ĠR OU GE Ġscore Ġon Ġthe Ġvalidation Ġset Ġ. ĠDuring Ġdecoding , Ġwe Ġapplied Ġbeam Ġsearch Ġwith Ġbeam Ġsize Ġof Ġ5 . ĠFollowing ĠB IB REF 26 , Ġwe Ġalso Ġblocked Ġrepeating Ġtrig rams Ġduring Ġbeam Ġsearch Ġand Ġtuned Ġthe Ġminimum Ġsummary Ġlength Ġon Ġthe Ġvalidation Ġset . ĠSimilar Ġto Ġthe Ġpre - training Ġprocess , Ġthe Ġdatasets Ġwith Ġless Ġinstances Ġwere Ġfine - tun ed Ġwith Ġsmaller Ġbatch Ġsize Ġ( i . e ., Ġ768 Ġfor ĠCNN DM Ġand Ġ64 Ġfor ĠNYT ). </s> Exper imental ĠSetup Ġ: :: ĠEval uations </s> We Ġused ĠR OU GE ĠB IB REF 41 Ġto Ġmeasure Ġthe Ġquality Ġof Ġdifferent Ġsummar ization Ġmodel Ġoutputs . ĠWe Ġreported Ġfull - length ĠF 1 Ġbased ĠR OU GE - 1 , ĠR OU GE - 2 Ġand ĠR OU GE - L Ġscores Ġon ĠCNN DM , Ġwhile Ġwe Ġused Ġthe Ġlimited - length Ġrecall Ġbased ĠR OU GE - 1 , ĠR OU GE - 2 Ġand ĠR OU GE - L Ġon ĠNYT Ġfollowing ĠB IB REF 36 . ĠThe ĠR OU GE Ġscores Ġare Ġcomputed Ġusing Ġthe ĠR OU GE - 1 . 5 . 5 . pl Ġscript . </s> Since Ġsumm aries Ġgenerated Ġby Ġabstract ive Ġmodels Ġmay Ġproduce Ġdis f luent Ġor Ġun gram matical Ġoutputs , Ġwe Ġalso Ġevaluated Ġabstract ive Ġsystems Ġby Ġelic iting Ġhuman Ġjud gements . ĠFollowing Ġprevious Ġwork ĠB IB REF 15 , ĠB IB REF 17 , Ġ20 Ġdocuments Ġare Ġrandomly Ġsampled Ġfrom Ġthe Ġtest Ġsplit Ġof ĠCNN DM . ĠParticipants Ġare Ġpresented Ġwith Ġa Ġdocument Ġand Ġa Ġlist Ġof Ġoutputs Ġgenerated Ġby Ġdifferent Ġabstract ive Ġsummar ization Ġsystems . ĠThen Ġthey Ġare Ġasked Ġto Ġrank Ġthe Ġoutputs Ġaccording Ġto Ġinform at iveness Ġ( does Ġthe Ġsummary Ġcapture Ġthe Ġinformative Ġpart Ġof Ġthe Ġdocument ?), Ġflu ency Ġ( is Ġthe Ġsummary Ġgram matical ?), Ġand Ġsuccinct ness Ġ( does Ġthe Ġsummary Ġexpress Ġthe Ġdocument Ġclearly Ġin Ġa Ġfew Ġwords ?) </s> Results Ġ: :: ĠAutomatic ĠEvaluation </s> The Ġresults Ġon Ġthe ĠCNN DM Ġare Ġsummarized Ġin ĠTable ĠT AB REF 25 . ĠThe Ġfirst Ġand Ġsecond Ġblocks Ġshow Ġresults Ġof Ġprevious Ġextract ive Ġand Ġabstract ive Ġmodels , Ġrespectively . ĠResults Ġof ĠStep Ġare Ġall Ġlisted Ġin Ġthe Ġthird Ġblock . ĠLead 3 Ġis Ġa Ġbaseline Ġwhich Ġsimply Ġtakes Ġthe Ġfirst Ġthree Ġsentences Ġof Ġa Ġdocument Ġas Ġits Ġsummary . ĠB ER TE xt ĠB IB REF 23 Ġis Ġan Ġextract ive Ġmodel Ġfine - tun ing Ġon ĠB ERT ĠB IB REF 2 Ġthat Ġoutper forms Ġother Ġextract ive Ġsystems . ĠPT Gen ĠB IB REF 11 , ĠDRM ĠB IB REF 26 , Ġand ĠDC A ĠB IB REF 42 Ġare Ġsequence - to - sequence Ġlearning Ġbased Ġmodels Ġextended Ġwith Ġcopy Ġand Ġcoverage Ġmechanism , Ġreinforcement Ġlearning , Ġand Ġdeep Ġcommunicating Ġagents Ġindividually . ĠBottom Up ĠB IB REF 29 Ġassisted Ġsummary Ġgeneration Ġwith Ġa Ġword Ġprediction Ġmodel . ĠB ER TA bs ĠB IB REF 23 Ġand ĠUni LM ĠB IB REF 43 Ġare Ġboth Ġpre - training Ġbased Ġseq 2 seq Ġsummar ization Ġmodels . ĠWe Ġalso Ġimplemented Ġthree Ġabstract ive Ġmodels Ġas Ġour Ġbas elines . ĠTrans former - S 2 S Ġis Ġ6 - layer Ġseq 2 seq ĠTrans former ĠB IB REF 9 Ġwith Ġrandom Ġinitialization . ĠWhen Ġwe Ġreplaced Ġthe Ġenc oder Ġof ĠTrans former - S 2 S Ġwith Ġ$\ text { Ro BER Ta } _ \ text { B ASE } $ ĠB IB REF 4 , Ġ$\ text { Ro BER Ta } _ \ text { B ASE } $ - S 2 S Ġoutper forms ĠTrans former - S 2 S Ġby Ġnearly Ġ2 ĠR OU GE , Ġwhich Ġdemonstrates Ġthe Ġeffectiveness Ġof Ġpre - trained Ġmodels . ĠWith Ġeven Ġlarger Ġpre - trained Ġmodel Ġ$\ text { Ro BER Ta } _ \ text { L AR GE } $, Ġ$\ text { Ro BER Ta } $ - S 2 S Ġis Ġcomparable Ġwith Ġthe Ġbest Ġpublished Ġabstract ive Ġmodel ĠUni LM ĠB IB REF 43 . </s> Based Ġon Ġ$\ text { Ro BER Ta } $ - S 2 S Ġ( the Ġsizes Ġof ĠStep Ġand Ġ$\ text { Ro BER Ta } $ - S 2 S Ġare Ġidentical ), Ġwe Ġstudy Ġthe Ġeffect Ġof Ġdifferent Ġpre - training Ġtasks Ġ( see ĠSection ĠSEC REF 9 ). ĠWe Ġfirst Ġpre - train ĠStep Ġon Ġunl abel ed Ġdocuments Ġof ĠCNN DM Ġtraining Ġsplit Ġto Ġget Ġquick Ġfeedback , Ġden oted Ġas ĠStep Ġ( in - domain ). ĠFrom Ġthe Ġtop Ġpart Ġof Ġthe Ġthird Ġblock Ġin ĠTable ĠT AB REF 25 , Ġwe Ġcan Ġsee Ġthat ĠSent ence ĠRe ordering Ġ( SR ), ĠNext ĠSent ence ĠGeneration Ġ( NS G ) Ġand ĠMask ed ĠDocument ĠGeneration Ġ( MD G ) Ġcan Ġall Ġimprove Ġ$\ text { Ro BER Ta } $ - S 2 S Ġsignificantly Ġmeasured Ġby Ġthe ĠR OU GE Ġscript . ĠNote Ġthat Ġaccording Ġto Ġthe ĠR OU GE Ġscript , Ġ$\ pm Ġ0 . 22 $ ĠR OU GE Ġalmost Ġalways Ġmeans Ġa Ġsignificant Ġdifference Ġwith Ġ$ p Ġ< Ġ0 . 05 $. ĠInteresting , Ġeven ĠStep Ġis Ġpre - trained Ġon Ġ230 Ġmillion Ġwords , Ġit Ġoutper forms ĠUni LM Ġthat Ġis Ġpre - trained Ġon Ġ3 , 000 Ġmillion Ġwords ĠB IB REF 43 . ĠWhen Ġwe Ġpre - train ĠStep Ġon Ġeven Ġlarger Ġdataset Ġ( i . e ., ĠG IG A - CM ), Ġthe Ġresults Ġare Ġfurther Ġimproved Ġand ĠStep Ġoutper forms Ġall Ġmodels Ġin Ġcomparison , Ġas Ġlisted Ġin Ġthe Ġbottom Ġpart Ġof ĠTable ĠT AB REF 25 . </s> Table ĠT AB REF 26 Ġpresents Ġresults Ġon ĠNYT Ġdataset . ĠFollowing Ġthe Ġsame Ġevaluation Ġprotocol Ġas ĠB IB REF 36 , Ġwe Ġadopted Ġthe Ġlimited - length Ġrecall Ġbased ĠR OU GE , Ġwhere Ġwe Ġtrunc ated Ġthe Ġpredicted Ġsumm aries Ġto Ġthe Ġlength Ġof Ġthe Ġgold Ġones . ĠAgain , Ġthe Ġfirst Ġand Ġsecond Ġblocks Ġshow Ġresults Ġof Ġprevious Ġextract ive Ġand Ġabstract ive Ġmodels , Ġrespectively . ĠResults Ġof ĠStep Ġare Ġlisted Ġin Ġthe Ġthird Ġblock . ĠSimilar Ġto Ġthe Ġtrends Ġin ĠCNN DM , ĠStep Ġleads Ġsignificant Ġperformance Ġgains Ġ( with Ġ$ p < 0 . 05 $ ) Ġcompared Ġto Ġall Ġother Ġmodels Ġin ĠTable ĠT AB REF 26 . </s> Among Ġall Ġthree Ġpre - training Ġtasks , ĠSR Ġworks Ġslightly Ġbetter Ġthan Ġthe Ġother Ġtwo Ġtasks Ġ( i . e ., ĠNS G Ġand ĠMD G ). ĠWe Ġalso Ġtried Ġto Ġrandomly Ġuse Ġall Ġthe Ġthree Ġtasks Ġduring Ġtraining Ġwith Ġ1 / 3 Ġprobability Ġeach Ġ( ind icated Ġas ĠALL ). ĠInteresting , Ġwe Ġobserved Ġthat , Ġin Ġgeneral , ĠAll Ġoutper forms Ġall Ġthree Ġtasks Ġwhen Ġemploying Ġunl abel ed Ġdocuments Ġof Ġtraining Ġsplits Ġof ĠCNN DM Ġor ĠNYT , Ġwhich Ġmight Ġbe Ġdue Ġto Ġlimited Ġnumber Ġof Ġunl abel ed Ġdocuments Ġof Ġthe Ġtraining Ġsplits . ĠAfter Ġadding Ġmore Ġdata Ġ( i . e ., ĠG IA G - CM ) Ġto Ġpre - training , ĠSR Ġconsistently Ġachieves Ġhighest ĠR OU GE - 2 Ġon Ġboth ĠCNN DM Ġand ĠNYT . ĠWe Ġconclude Ġthat ĠSR Ġis Ġthe Ġmost Ġeffective Ġtask Ġfor Ġpre - training Ġsince Ġsentence Ġre ordering Ġtask Ġrequires Ġcomprehens ively Ġunderstanding Ġa Ġdocument Ġin Ġa Ġwide Ġcoverage , Ġgoing Ġbeyond Ġindividual Ġwords Ġand Ġsentences , Ġwhich Ġis Ġhighly Ġclose Ġto Ġthe Ġess ense Ġof Ġabstract ive Ġdocument Ġsummar ization . </s> Results Ġ: :: ĠHuman ĠEvaluation </s> We Ġalso Ġconducted Ġhuman Ġevaluation Ġwith Ġ20 Ġdocuments Ġrandomly Ġsampled Ġfrom Ġthe Ġtest Ġsplit Ġof ĠCNN DM . ĠWe Ġcompared Ġthe Ġbest Ġpre forming ĠStep Ġmodel Ġ( i . e ., Ġpre - training Ġon Ġthe ĠG IG A - CM Ġdataset Ġusing ĠSR Ġtask ) Ġwith Ġhuman Ġreferences Ġ( den oted Ġas ĠGold ), Ġ$\ text { Ro BER Ta } $ - S 2 S , Ġand Ġtwo Ġpre - training Ġbased Ġmodels , ĠB ER TA bs ĠB IB REF 23 Ġand ĠUni LM ĠB IB REF 43 . ĠParticipants Ġwere Ġasked Ġto Ġrank Ġthe Ġoutputs Ġof Ġthese Ġsystems Ġfrom Ġbest Ġto Ġworst . ĠWe Ġreport Ġthe Ġproportions Ġof Ġsystem Ġrankings Ġand Ġmean Ġrank Ġ( lower Ġis Ġbetter ) Ġin ĠTable ĠT AB REF 29 . ĠThe Ġoutput Ġof ĠStep Ġis Ġselected Ġas Ġthe Ġbest Ġfor Ġthe Ġ25 % Ġof Ġcases Ġand Ġwe Ġobtained Ġlower Ġmean Ġrank Ġthan Ġall Ġsystems Ġexcept Ġfor ĠGold , Ġwhich Ġshows Ġthe Ġparticipants ' Ġpreference Ġfor Ġour Ġmodel . ĠThen Ġwe Ġconverted Ġranking Ġnumbers Ġinto Ġratings Ġ( i . e ., Ġrank Ġ$ i $ Ġis Ġconverted Ġinto Ġ$ 6 - i $ ) Ġand Ġapplied Ġthe Ġstudent Ġ$ t $ - test Ġon Ġthe Ġratings . ĠStep Ġis Ġsignificantly Ġbetter Ġthan Ġall Ġother Ġsystems Ġin Ġcomparison Ġwith Ġ$ p < 0 . 05 $. ĠBut Ġit Ġstill Ġl ags Ġbehind Ġhuman . ĠOne Ġpossible Ġreason Ġis Ġthat ĠStep Ġ( as Ġwell Ġas Ġother Ġsystems ) Ġonly Ġtakes Ġthe Ġfirst Ġ512 Ġtokens Ġof Ġa Ġlong Ġdocument Ġas Ġinput Ġand Ġthus Ġmay Ġlose Ġinformation Ġresiding Ġin Ġthe Ġfollowing Ġtokens . </s> Conclusion </s> We Ġproposed ĠStep , Ġa Ġseq 2 seq Ġtransformer Ġpre - training Ġapproach , Ġfor Ġabstract ive Ġdocument Ġsummar ization . ĠSpecifically , Ġthree Ġpre - training Ġtasks Ġare Ġdesigned , Ġsentence Ġre ordering , Ġnext Ġsentence Ġgeneration , Ġand Ġmasked Ġdocument Ġgeneration . ĠWhen Ġwe Ġonly Ġemploy Ġthe Ġunl abel ed Ġdocuments Ġin Ġthe Ġtraining Ġsplits Ġof Ġsummar ization Ġdatasets Ġto Ġpre - training ĠStep Ġwith Ġour Ġproposed Ġtasks , Ġthe Ġsummar ization Ġmodel Ġbased Ġon Ġthe Ġpre - trained ĠStep Ġoutper forms Ġthe Ġbest Ġpublished Ġabstract ive Ġsystem . ĠInv olving Ġlarge Ġscale Ġdata Ġto Ġpre - training Ġleads Ġto Ġlarger Ġperformance Ġgains . ĠBy Ġusing Ġthe Ġbest Ġperforming Ġpre - training Ġtask , ĠStep Ġachieves Ġ0 . 8 Ġabsolute ĠR OU GE - 2 Ġimprovements Ġon ĠCNN / Daily Mail Ġand Ġ2 . 4 Ġabsolute ĠR OU GE - 2 Ġimprovements Ġon ĠNew ĠYork ĠTimes . ĠIn Ġthe Ġfuture , Ġwe Ġwould Ġlike Ġto Ġinvestigate Ġother Ġtasks Ġto Ġpre - train Ġthe Ġseq 2 seq Ġtransformer Ġmodel . ĠPre - training Ġfor Ġun super vised Ġabstract ive Ġsummar ization Ġis Ġalso Ġan Ġinteresting Ġdirection Ġand Ġworth Ġexploration .
Introduction Large pre-trained language models BIBREF0, BIBREF1, BIBREF2, BIBREF3, BIBREF4 improved the state-of-the-art of various natural language understanding (NLU) tasks such as question answering (e.g., SQuAD; BIBREF5), natural language inference (e.g., MNLI; BIBREF6) as well as text classification BIBREF7. These models (i.e., large LSTMs; BIBREF8 or Transformers; BIBREF9) are pre-trained on large scale unlabeled text with language modeling BIBREF0, BIBREF1, masked language modeling BIBREF2, BIBREF4 and permutation language modeling BIBREF3 objectives. In NLU tasks, pre-trained language models are mostly used as text encoders. Abstractive document summarization aims to rewrite a long document to its shorter form while still retaining its important information. Different from extractive document summarization that extacts important sentences, abstractive document summarization may paraphrase original sentences or delete contents from them. For more details on differences between abstractive and extractive document summary, we refer the interested readers to Nenkova:McKeown:2011 and Section SECREF2. This task is usually framed as a sequence-to-sequence learning problem BIBREF10, BIBREF11. In this paper, we adopt the sequence-to-sequence (seq2seq) Transformer BIBREF9, which has been demonstrated to be the state-of-the-art for seq2seq modeling BIBREF9, BIBREF12. Unfortunately, training large seq2seq Transformers on limited supervised summarization data is challenging BIBREF12 (refer to Section SECREF5). The seq2seq Transformer has an encoder and a decoder Transformer. Abstractive summarization requires both encoding of an input document and generation of a summary usually containing multiple sentences. As mentioned earlier, we can take advantage of recent pre-trained Transformer encoders for the document encoding part as in liu2019text. However, liu2019text leave the decoder randomly initialized. In this paper, we aim to pre-train both the encoder (i.e., the encoding part) and decoder (i.e., the generation part) of a seq2seq Transformer , which is able to improve abstractive summarization performance. Based on the above observations, we propose Step (as shorthand for Sequence-to-Sequence TransformEr Pre-training), which can be pre-trained on large scale unlabeled documents. Specifically, we design three tasks for seq2seq model pre-training, namely Sentence Reordering (SR), Next Sentence Generation (NSG), and Masked Document Generation (MDG). SR learns to recover a document with randomly shuffled sentences. NSG generates the next segment of a document based on its preceding segment. MDG recovers a masked document to its original form. After pre-trianing Step using the three tasks on unlabeled documents, we fine-tune it on supervised summarization datasets. We evaluate our methods on two summarization datasets (i.e., the CNN/DailyMail and the New York Times datasets). Experiments show that all three tasks we propose can improve upon a heavily tuned large seq2seq Transformer which already includes a strong pre-trained encoder by a large margin. Compared to the best published abstractive models, Step improves the ROUGE-2 by 0.8 on the CNN/DailyMail dataset and by 2.4 on the New York Times dataset using our best performing task for pre-training. Human experiments also show that Step can produce significantly better summaries in comparison with recent strong abstractive models. Related Work This section introduces extractive and abstractive document summarization as well as pre-training methods for natural language processing tasks. Related Work ::: Extractive Summarization Extractive summarization systems learn to find the informative sentences in a document as its summary. This task is usually viewed as a sentence ranking problem BIBREF13, BIBREF14 using scores from a binary (sequence) classification model, which predicts whether a sentence is in the summary or not. Extractive neural models employ hierarchical LSTMs/CNNs as the feature learning part of the binary (sequence) classifier BIBREF15, BIBREF16, BIBREF17, BIBREF18, which largely outperforms discrete feature based models BIBREF19, BIBREF20, BIBREF21. Very recently, the feature learning part was replaced again with pre-trained transformers BIBREF22, BIBREF23 that lead to another huge improvement of summarization performance. However, extractive models have their own limitations. For example, the extracted sentences might be too long and redundant. Besides, human written summaries in their nature are abstractive. Therefore, we focus on abstractive summarization in this paper. Related Work ::: Abstractive Summarization The goal of abstractive summarization is to generate summaries by rewriting a document, which is a sequence-to-sequence learning problem. seq2seq attentive LSTMs BIBREF8, BIBREF24 are employed in nallapati2016abstractive. Even these models are extended with copy mechanism BIBREF25, coverage model BIBREF11 and reinforcement learning BIBREF26, their results are still very close to that of Lead3 which selects the leading three sentences of a document as its summary. One possible reason is that LSTMs without pre-training are not powerful enough. liu2019text used a seq2seq Transformer model with its encoder initialized with a pre-trained Transformer (i.e., BERT; BIBREF2) and achieved the state-of-the-art performance. Our work goes one step further, we propose a method to pre-train the decoder together with the encoder and then initialize both the encoder and decoder of a summarization model with the pre-trained Transformers. There is also a line of work that bridges extractive and abstractive models with reinforcement learning BIBREF27, attention fusion BIBREF28 and bottom-up attention BIBREF29, while our model is conceptually simpler. Related Work ::: Pre-training Pre-training methods draw a lot of attention recently. peters2018deep and radford:2019:arxiv pre-trained LSTM and Transformer encoders using language modeling objectives. To leverage the context in both directions, BIBREF2 proposed BERT, which is trained with the mask language modeling objective. XLNet BIBREF3 is trained with permutation language modeling objective, which removes the independence assumption of masked tokens in BERT. RoBERTa BIBREF4 extends BERT with more training data and better training strategies. All the methods above focus on pre-training an encoder, while we propose methods to pre-train both the encoder and decoder of a seq2seq model. dong2019unified proposed a Transformer language model that can be used for both natural language understanding and generation tasks, which is pre-trained using masked, unidirectional and seq2seq language modeling objectives. Their method tries to pre-train a seq2seq Transformer with its encoder and decoder parameters shared. Differently, we pre-train a seq2seq Transformer with separate parameters for the encoder and decoder. song2019mass proposed a method to pre-train a seq2seq Transformer by masking a span of text and then predicting the original text with masked tokens at other positions. Their pre-training task is similar to our Masked Document Generation task, but we apply a different masking strategy and predict the original text without masked tokens. Besides, we propose another two tasks for seq2seq model pre-training. BIBREF30 tested their model on sentence-level tasks (e.g., machine translation and sentence compression), while we aim to solve document-level tasks (e.g., abstractive document summarization). Sequence-to-Sequence Transformer Pre-training This section first introduces the backbone architecture of our abstractive summarization model Step. We then describe methods to pre-train Step and finally move on to the fine-tuning on summarization datasets. Sequence-to-Sequence Transformer Pre-training ::: Architecture In this work, the task of abstractive document summarization is modeled as a sequence-to-sequence learning problem, where a document is viewed as a sequence of tokens and its corresponding summary as another sequence of tokens. We adopt the seq2seq Transformer architecture BIBREF9, which includes an encoder Transformer and a decoder Transformer. Both the encoder and decoder Transformers have multiple layers and each layer contains a multi-head attentive sub-layer followed by a fully connected sub-layer with residual connections BIBREF31 and layer normalization BIBREF32. Let us use $X = (x_1, x_2, \dots , x_{|X|})$ to denote a document and use $Y = (y_1, y_2, \dots , y_{|Y|})$ to denote its summary. The encoder takes the document $X$ as input and transforms it to its contextual representations. The decoder learns to generate the summary $Y$ one token at a time based on the contextual representations and all preceding tokens that have been generated so far: where $y_{<t}$ stands for all tokens before position $t$ (i.e., $y_{<t}=(y_1, y_2, \dots , y_{t-1})$). This model can be trained by minimizing the negative log-likelihood of the training document-summary pairs. Sequence-to-Sequence Transformer Pre-training ::: Pre-training Tasks Training a seq2seq Transformer model on a summarization dataset from scratch is difficult due to the limited number of document-summary pairs. Pre-trained Transformer encoders such as BERT BIBREF2 and RoBERTa BIBREF4 have achieved great success in many natural language understanding tasks. Therefore, we first initialize the encoder of our seq2seq Transformer summarization model Step with an existing pre-trained Transformer encoder (i.e., RoBERTa) to enhance its language understanding capabilities. To help Step gain language generation capabilities and the abilities of associating generated text with encoder outputs, we continue to pre-train it on unlabeled text. In the following, we describe our pre-training tasks. Sequence-to-Sequence Transformer Pre-training ::: Pre-training Tasks ::: Sentence Reordering A document is typically composed of multiple sentences separated by full stops. In this task, we first shuffle the document by sentences and then recover the original document. There are several reasons why we design this task. First, a summary of a document usually consists of multiple sentences. We expect that Step learns to generate long and coherent summaries (across sentences). The output of the task (i.e., the original document) also contains multiple sentences. Second, sentence reordering (or content reordering) is necessary for summarization. According to the statistics on training sets of our summarization datasets, contents of the original documents are reordered in their summaries for 40% of cases. We define content reordering as follows. For each document-summary pair, we first map each sentence in the summary to one sentence in its paired document by maximizing the ROUGE score. If the sequence of sentences in the summary is different from the sequence of their mapped sentences in the original document, we count this as one content reordering. Thirdly, abstractive summary requires reproducing factual details (e.g., named entities, figures) from source text. We also expect Step to learn to copy tokens. Here is a formal definition of this task. Let us change the notation of a document slightly in this paragraph. Let $X=(S_1, S_2, \dots , S_m)$ denote a document, where $S_i = (w^i_1, w^i_2, \dots , w^i_{|S_i|})$ is a sentence in it, $w^i_j$ is a word in $S_i$ and $m$ is the number of sentences. $X$ is still a sequence of tokens (by concatenating tokens in all sentences). Let $A=\text{\tt permutation}(m)=(a_1,a_2,\dots , a_m)$ denote a permuted range of $(1, 2, \dots , m)$ and therefore $\hat{X}_S=(S_{a_1}, S_{a_2}, \dots , S_{a_m})$ is the shuffled document. Note that $\hat{X}_S$ is a sequence of tokens by concatenating all shuffled sentences. Step can be trained on $\langle \hat{X}_S, X \rangle $ pairs constructed from unlabeled documents, as demonstrated in Figure FIGREF5. Note that document rotation is a special case of sentence reordering with significant amount of partially ordered sentences, which we believe is a simpler task. In this work, we thus only consider the general case of sentence reordering. Sequence-to-Sequence Transformer Pre-training ::: Pre-training Tasks ::: Next Sentence Generation The second pre-training task leverages the natural order of text. Next Sentence Generation (NSG) uses one span of text in a document to predict its next span of text, as shown in Figure FIGREF5. Specifically, we split a document into two segments (i.e., $G_1$ and $G_2$). Note that each segment might contain multiple sentences, which fits the document summarization task very well, since either a document or its summary usually includes multiple sentences. Intuitively, in a document, sentences are highly correlated with their preceding sentences due to the context dependent nature of documents or language. We intend our model to learn to generate multiple sentences and also learn to focus on preceding context. We have at least two options for the splitting position of the two segments. Option one: the position right after a full-stop symbol (such as period, question mark, etc.) is selected as the splitting point, which ensures full sentences for each segment. Option two: the splitting point can be at any position within the document. We choose the second option, which may lead to incomplete sentences in segments. We intend to force the encoder and decoder to understand input text without complete information, which we believe is more challenging compared to option one. Besides, as a common wisdom in abstractive summarization, documents are truncated to a fixed number of tokens, which may also contain incomplete sentences. We use option two to reduce the pre-training and fine-tuning input mismatch. In this task, we train the model Step on large amount of $\langle G_1, G_2\rangle $ pairs constructed following the option two splitting strategy. Next sentence prediction has been used in skip-thought vectors BIBREF33. There are two differences. First, each segment in their model only has one sentence; second, they use this task to pre-train an encoder rather than an entire seq2seq model. BIBREF2 introduced a task named next sentence prediction (NSP), which is different from this task. NSP is a classification task, but NSG is a generation task, which intends to pre-train a generation model. Sequence-to-Sequence Transformer Pre-training ::: Pre-training Tasks ::: Masked Document Generation The third task we consider is Masked Document Generation (MDG) that learns to recover a document with a masked span of tokens (see Figure FIGREF5). For simplicity, a document consisting of a sequence of tokens is denoted as $X=(x_1, x_2, \cdots , x_{|X|})$. We randomly sample the length of the span $l$ from a discrete uniform distribution $\mathcal {U}(a, b)$ and the span start position $k$ from another discrete uniform distribution $\mathcal {U}(1, |X|-l+1)$ (see Section SECREF4 for more details). Thus, $\mathcal {M}=(x_k, x_{k+1}, \cdots , x_{k+l-1})$ is the text span to be masked. One straightforward masking strategy is to replace each token residing in $\mathcal {M}$ with a special [MASK] token. However, we refrain from doing so because of the following three reasons. Usually, [MASK] tokens will not appear in downstream tasks. Second, entirely masking a continuous sub-sequence of $X$ may make the whole document incomprehensible, which might be too challenging for our model to learn. Third, similar to SR, avoiding replacing every token with [MASK] also helps our model learn the ability of copying tokens from the input while preserving the ability of generating novel tokens. In the sub-sequence $\mathcal {M}$, each token is processed with one of the three strategies: 1) replaced with the [MASK] token; 2) replaced with a random token; 3) remain unchanged. Inspired by BERT BIBREF2, for 80% tokens, we follow strategy 1). In 10% of cases, we employ strategy 2) and we use strategy 3) for the remaining 10% of cases. Let $\hat{X}_M$ denote the document after the application of our masking strategy. We could create infinite amount of $\langle \hat{X}_M,X\rangle $ pairs to train Step. During pre-training, we could also employ all the three tasks (i.e., SR, NSG, MDG) together. For each training batch, we randomly choose one task and each task is used for $1/3$ of the time. Sequence-to-Sequence Transformer Pre-training ::: Fine-tuning After pre-training Step with the three tasks introduced in Section SECREF9, we fine-tune the model on abstractive document summarization datasets. The fine-tuning process is straightforward. We simply continue to train Step on the supervised document-summary pairs. Similar to other seq2seq summarization models, we do beam search during the generation of summaries. Experimental Setup In this section, we present the experimental setup for evaluating our summarization models. We first introduce the datasets used for our experiments. Then we describe training details of our models as well as our evaluation protocols. Experimental Setup ::: Datasets We assess the summarization performance of our models on two benchmark datasets: the CNN/DailyMail (CNNDM) dataset BIBREF34, BIBREF11 and the New York Times (NYT) dataset BIBREF35. We pre-train our models on the GIGA-CM dataset introduced in zhang-etal-2019-hibert. Experimental Setup ::: Datasets ::: CNNDM CNNDM contains news articles and their associated highlights (i.e., summaries) collected from the CNN and Daily Mail Online websites. Following previous work BIBREF11, BIBREF22, BIBREF23, we use the non-anonymized version of CNNDM. Specifically, we preprocess the dataset with the publicly available scripts provided by see2017get and obtain 287,226 document-summary pairs for training, 13,368 for validation and 11,490 for test. Experimental Setup ::: Datasets ::: NYT The NYT dataset is a collection of articles along with multi-sentence summaries written by library scientists. We closely follow the preprocessing procedures described in durrett2016learning and liu2019text. The test set is constructed by including all articles published on January 1, 2017 or later, which contains 9,076 articles. The remaining 100,834 articles are split into a training set of 96,834 examples and a validation set of 4,000 examples. As in BIBREF36, we also remove articles whose summaries contain less than 50 words from the test set, and the resulting test set contains 3,452 examples. Experimental Setup ::: Datasets ::: GIGA-CM To pre-train our model with the tasks introduced in Section SECREF9, following the procedures in BIBREF22, we created the GIGA-CM dataset, which contains only unlabeled documents. The training set of GIGA-CM is composed of 6,521,658 documents sampled from the English Gigaword dataset and the training documents in CNNDM. We used the 13,368 documents in the validation split of CNNDM as its validation set. Note that the Gigaword dataset overlaps with the NYT dataset and we therefore exclude the test set of NYT from the training set of GIGA-CM. For CNNDM, NYT and GIGA-CM datasets, we segment and tokenize documents and/or summaries (GIGA-CM only contains documents) using the Stanford CoreNLP toolkit BIBREF37. To reduce the vocabulary size, we further apply the UTF8 based BPE BIBREF38 introduced in GPT-2 BIBREF39 to all datasets. As a common wisdom in abstractive summarization, documents and summaries in CNNDM and NYT are usually truncated to 512 and 256 tokens, respectively. We leverage unlabeled documents differently for different pre-training tasks (see Section SECREF9). We first split each document into 512 token segments if it contains more than 512 tokens (segments or documents with less than 512 tokens are removed). In Sentence Reordering (SR) and Masked Document Generation (MDG), we use the segment after transformation to predict the original segment. We set the minimum masked length $a=100$ and the maximum masked length $b=256$ in MDG. In Next Sentence Generation (NSG), each segment is used to predict its next 256 tokens. Experimental Setup ::: Implementation Details As mentioned in Section SECREF3, our model is a Seq2Seq Transformer model BIBREF9. The encoder is initialized with the $\text{RoBERTa}_{\text{LARGE}}$ model BIBREF4, and therefore they share the same architecture. Specifically, the encoder is a 24-layer Transformer. Each layer has 16 attention heads and its hidden size and feed-forward filter size are 1,024 and 4,096, respectively. The decoder is shallower with 6 layers. The hidden size and number of attention head of the decoder are identical to these of the encoder, but the feed-forward filter size is 2,048. We use a smaller filter size in the decoder to reduce the computational and memory cost. The dropout rates of all layers in the encoder are set to 0.1 and all dropout rates in the decoder are set to 0.3. Our models are optimized using Adam BIBREF40 with $\beta _1=0.9$, $\beta _2=0.98$. The other optimization hyper-parameters for pre-training and fine-tuning are different. In the pre-training stage, the encoder is initialized with a pre-trained model while the decoder is randomly initialized. Therefore, we used two separate optimizers for the encoder and decoder with a smaller learning rate for the encoder optimizer. Learning rates of the encoder and decoder are set to $2e-5$ and $1e-4$ with 10,000 warmup steps, respectively. We also adopted the same learning rate schedule strategies as BIBREF9. We used smaller batch sizes for datasets with less examples (i.e., 1,024 for GIGA-CM, 256 for CNNDM and 128 for NYT) to ensure each epoch has sufficient number of model updates. We trained our models until their convergence of validation perplexities (around 30 epochs on GIGA-CM, 60 epochs on CNNDM and 40 epochs on NYT). One epoch on GIGA-CM takes around 24 hours with 8 Nvidia Tesla V100 GPUs. The time costs for different pre-training tasks are close. Most of the hyper-parameters in the fine-tuning stage are the same as these in the pre-training stage. The differences are as follows. The learning rates for both the encoder and decoder are set to $2e-5$ with 4,000 warmup steps, since both the encoder and decoder are already pre-trained. We trained our models for 50 epochs (saved per epoch) and selected the best model w.r.t. ROUGE score on the validation set . During decoding, we applied beam search with beam size of 5. Following BIBREF26, we also blocked repeating trigrams during beam search and tuned the minimum summary length on the validation set. Similar to the pre-training process, the datasets with less instances were fine-tuned with smaller batch size (i.e., 768 for CNNDM and 64 for NYT). Experimental Setup ::: Evaluations We used ROUGE BIBREF41 to measure the quality of different summarization model outputs. We reported full-length F1 based ROUGE-1, ROUGE-2 and ROUGE-L scores on CNNDM, while we used the limited-length recall based ROUGE-1, ROUGE-2 and ROUGE-L on NYT following BIBREF36. The ROUGE scores are computed using the ROUGE-1.5.5.pl script. Since summaries generated by abstractive models may produce disfluent or ungrammatical outputs, we also evaluated abstractive systems by eliciting human judgements. Following previous work BIBREF15, BIBREF17, 20 documents are randomly sampled from the test split of CNNDM. Participants are presented with a document and a list of outputs generated by different abstractive summarization systems. Then they are asked to rank the outputs according to informativeness (does the summary capture the informative part of the document?), fluency (is the summary grammatical?), and succinctness (does the summary express the document clearly in a few words?) Results ::: Automatic Evaluation The results on the CNNDM are summarized in Table TABREF25. The first and second blocks show results of previous extractive and abstractive models, respectively. Results of Step are all listed in the third block. Lead3 is a baseline which simply takes the first three sentences of a document as its summary. BERTExt BIBREF23 is an extractive model fine-tuning on BERT BIBREF2 that outperforms other extractive systems. PTGen BIBREF11, DRM BIBREF26, and DCA BIBREF42 are sequence-to-sequence learning based models extended with copy and coverage mechanism, reinforcement learning, and deep communicating agents individually. BottomUp BIBREF29 assisted summary generation with a word prediction model. BERTAbs BIBREF23 and UniLM BIBREF43 are both pre-training based seq2seq summarization models. We also implemented three abstractive models as our baselines. Transformer-S2S is 6-layer seq2seq Transformer BIBREF9 with random initialization. When we replaced the encoder of Transformer-S2S with $\text{RoBERTa}_\text{BASE}$ BIBREF4, $\text{RoBERTa}_\text{BASE}$-S2S outperforms Transformer-S2S by nearly 2 ROUGE, which demonstrates the effectiveness of pre-trained models. With even larger pre-trained model $\text{RoBERTa}_\text{LARGE}$, $\text{RoBERTa}$-S2S is comparable with the best published abstractive model UniLM BIBREF43. Based on $\text{RoBERTa}$-S2S (the sizes of Step and $\text{RoBERTa}$-S2S are identical), we study the effect of different pre-training tasks (see Section SECREF9). We first pre-train Step on unlabeled documents of CNNDM training split to get quick feedback, denoted as Step (in-domain). From the top part of the third block in Table TABREF25, we can see that Sentence Reordering (SR), Next Sentence Generation (NSG) and Masked Document Generation (MDG) can all improve $\text{RoBERTa}$-S2S significantly measured by the ROUGE script. Note that according to the ROUGE script, $\pm 0.22$ ROUGE almost always means a significant difference with $p < 0.05$. Interesting, even Step is pre-trained on 230 million words, it outperforms UniLM that is pre-trained on 3,000 million words BIBREF43. When we pre-train Step on even larger dataset (i.e., GIGA-CM), the results are further improved and Step outperforms all models in comparison, as listed in the bottom part of Table TABREF25. Table TABREF26 presents results on NYT dataset. Following the same evaluation protocol as BIBREF36, we adopted the limited-length recall based ROUGE, where we truncated the predicted summaries to the length of the gold ones. Again, the first and second blocks show results of previous extractive and abstractive models, respectively. Results of Step are listed in the third block. Similar to the trends in CNNDM, Step leads significant performance gains (with $p<0.05$) compared to all other models in Table TABREF26. Among all three pre-training tasks, SR works slightly better than the other two tasks (i.e., NSG and MDG). We also tried to randomly use all the three tasks during training with 1/3 probability each (indicated as ALL). Interesting, we observed that, in general, All outperforms all three tasks when employing unlabeled documents of training splits of CNNDM or NYT, which might be due to limited number of unlabeled documents of the training splits. After adding more data (i.e., GIAG-CM) to pre-training, SR consistently achieves highest ROUGE-2 on both CNNDM and NYT. We conclude that SR is the most effective task for pre-training since sentence reordering task requires comprehensively understanding a document in a wide coverage, going beyond individual words and sentences, which is highly close to the essense of abstractive document summarization. Results ::: Human Evaluation We also conducted human evaluation with 20 documents randomly sampled from the test split of CNNDM. We compared the best preforming Step model (i.e., pre-training on the GIGA-CM dataset using SR task) with human references (denoted as Gold), $\text{RoBERTa}$-S2S, and two pre-training based models, BERTAbs BIBREF23 and UniLM BIBREF43. Participants were asked to rank the outputs of these systems from best to worst. We report the proportions of system rankings and mean rank (lower is better) in Table TABREF29. The output of Step is selected as the best for the 25% of cases and we obtained lower mean rank than all systems except for Gold, which shows the participants' preference for our model. Then we converted ranking numbers into ratings (i.e., rank $i$ is converted into $6-i$) and applied the student $t$-test on the ratings. Step is significantly better than all other systems in comparison with $p<0.05$. But it still lags behind human. One possible reason is that Step (as well as other systems) only takes the first 512 tokens of a long document as input and thus may lose information residing in the following tokens. Conclusion We proposed Step, a seq2seq transformer pre-training approach, for abstractive document summarization. Specifically, three pre-training tasks are designed, sentence reordering, next sentence generation, and masked document generation. When we only employ the unlabeled documents in the training splits of summarization datasets to pre-training Step with our proposed tasks, the summarization model based on the pre-trained Step outperforms the best published abstractive system. Involving large scale data to pre-training leads to larger performance gains. By using the best performing pre-training task, Step achieves 0.8 absolute ROUGE-2 improvements on CNN/DailyMail and 2.4 absolute ROUGE-2 improvements on New York Times. In the future, we would like to investigate other tasks to pre-train the seq2seq transformer model. Pre-training for unsupervised abstractive summarization is also an interesting direction and worth exploration.
Which of the three pretraining tasks is the most helpful?
ed2eb4e54b641b7670ab5a7060c7b16c628699ab
Which Ġof Ġthe Ġthree Ġpret raining Ġtasks Ġis Ġthe Ġmost Ġhelpful ?
SR
<s> Which Ġof Ġthe Ġthree Ġpret raining Ġtasks Ġis Ġthe Ġmost Ġhelpful ? </s> Introduction </s> Large Ġpre - trained Ġlanguage Ġmodels ĠB IB REF 0 , ĠB IB REF 1 , ĠB IB REF 2 , ĠB IB REF 3 , ĠB IB REF 4 Ġimproved Ġthe Ġstate - of - the - art Ġof Ġvarious Ġnatural Ġlanguage Ġunderstanding Ġ( NL U ) Ġtasks Ġsuch Ġas Ġquestion Ġanswering Ġ( e . g ., ĠS Qu AD ; ĠB IB REF 5 ), Ġnatural Ġlanguage Ġinference Ġ( e . g ., ĠMN LI ; ĠB IB REF 6 ) Ġas Ġwell Ġas Ġtext Ġclassification ĠB IB REF 7 . ĠThese Ġmodels Ġ( i . e ., Ġlarge ĠL ST Ms ; ĠB IB REF 8 Ġor ĠTransformers ; ĠB IB REF 9 ) Ġare Ġpre - trained Ġon Ġlarge Ġscale Ġunl abel ed Ġtext Ġwith Ġlanguage Ġmodeling ĠB IB REF 0 , ĠB IB REF 1 , Ġmasked Ġlanguage Ġmodeling ĠB IB REF 2 , ĠB IB REF 4 Ġand Ġperm utation Ġlanguage Ġmodeling ĠB IB REF 3 Ġobjectives . ĠIn ĠNL U Ġtasks , Ġpre - trained Ġlanguage Ġmodels Ġare Ġmostly Ġused Ġas Ġtext Ġenc od ers . </s> Abstract ive Ġdocument Ġsummar ization Ġaims Ġto Ġrewrite Ġa Ġlong Ġdocument Ġto Ġits Ġshorter Ġform Ġwhile Ġstill Ġretaining Ġits Ġimportant Ġinformation . ĠDifferent Ġfrom Ġextract ive Ġdocument Ġsummar ization Ġthat Ġext acts Ġimportant Ġsentences , Ġabstract ive Ġdocument Ġsummar ization Ġmay Ġparaph rase Ġoriginal Ġsentences Ġor Ġdelete Ġcontents Ġfrom Ġthem . ĠFor Ġmore Ġdetails Ġon Ġdifferences Ġbetween Ġabstract ive Ġand Ġextract ive Ġdocument Ġsummary , Ġwe Ġrefer Ġthe Ġinterested Ġreaders Ġto ĠN en k ova : Mc Ke own : 2011 Ġand ĠSection ĠSEC REF 2 . ĠThis Ġtask Ġis Ġusually Ġframed Ġas Ġa Ġsequence - to - sequence Ġlearning Ġproblem ĠB IB REF 10 , ĠB IB REF 11 . ĠIn Ġthis Ġpaper , Ġwe Ġadopt Ġthe Ġsequence - to - sequence Ġ( seq 2 seq ) ĠTrans former ĠB IB REF 9 , Ġwhich Ġhas Ġbeen Ġdemonstrated Ġto Ġbe Ġthe Ġstate - of - the - art Ġfor Ġseq 2 seq Ġmodeling ĠB IB REF 9 , ĠB IB REF 12 . ĠUnfortunately , Ġtraining Ġlarge Ġseq 2 seq ĠTransformers Ġon Ġlimited Ġsupervised Ġsummar ization Ġdata Ġis Ġchallenging ĠB IB REF 12 Ġ( re fer Ġto ĠSection ĠSEC REF 5 ). ĠThe Ġseq 2 seq ĠTrans former Ġhas Ġan Ġenc oder Ġand Ġa Ġdec oder ĠTrans former . ĠAbstract ive Ġsummar ization Ġrequires Ġboth Ġencoding Ġof Ġan Ġinput Ġdocument Ġand Ġgeneration Ġof Ġa Ġsummary Ġusually Ġcontaining Ġmultiple Ġsentences . ĠAs Ġmentioned Ġearlier , Ġwe Ġcan Ġtake Ġadvantage Ġof Ġrecent Ġpre - trained ĠTrans former Ġenc od ers Ġfor Ġthe Ġdocument Ġencoding Ġpart Ġas Ġin Ġli u 2019 text . ĠHowever , Ġli u 2019 text Ġleave Ġthe Ġdec oder Ġrandomly Ġinitialized . ĠIn Ġthis Ġpaper , Ġwe Ġaim Ġto Ġpre - train Ġboth Ġthe Ġenc oder Ġ( i . e ., Ġthe Ġencoding Ġpart ) Ġand Ġdec oder Ġ( i . e ., Ġthe Ġgeneration Ġpart ) Ġof Ġa Ġseq 2 seq ĠTrans former Ġ, Ġwhich Ġis Ġable Ġto Ġimprove Ġabstract ive Ġsummar ization Ġperformance . </s> Based Ġon Ġthe Ġabove Ġobservations , Ġwe Ġpropose ĠStep Ġ( as Ġshorthand Ġfor ĠSequence - to - Sequ ence ĠTransform Er ĠPre - training ), Ġwhich Ġcan Ġbe Ġpre - trained Ġon Ġlarge Ġscale Ġunl abel ed Ġdocuments . ĠSpecifically , Ġwe Ġdesign Ġthree Ġtasks Ġfor Ġseq 2 seq Ġmodel Ġpre - training , Ġnamely ĠSent ence ĠRe ordering Ġ( SR ), ĠNext ĠSent ence ĠGeneration Ġ( NS G ), Ġand ĠMask ed ĠDocument ĠGeneration Ġ( MD G ). ĠSR Ġlearns Ġto Ġrecover Ġa Ġdocument Ġwith Ġrandomly Ġshuff led Ġsentences . ĠNS G Ġgenerates Ġthe Ġnext Ġsegment Ġof Ġa Ġdocument Ġbased Ġon Ġits Ġpreceding Ġsegment . ĠMD G Ġrecovers Ġa Ġmasked Ġdocument Ġto Ġits Ġoriginal Ġform . ĠAfter Ġpre - t rian ing ĠStep Ġusing Ġthe Ġthree Ġtasks Ġon Ġunl abel ed Ġdocuments , Ġwe Ġfine - t une Ġit Ġon Ġsupervised Ġsummar ization Ġdatasets . </s> We Ġevaluate Ġour Ġmethods Ġon Ġtwo Ġsummar ization Ġdatasets Ġ( i . e ., Ġthe ĠCNN / Daily Mail Ġand Ġthe ĠNew ĠYork ĠTimes Ġdatasets ). ĠExper iments Ġshow Ġthat Ġall Ġthree Ġtasks Ġwe Ġpropose Ġcan Ġimprove Ġupon Ġa Ġheavily Ġtuned Ġlarge Ġseq 2 seq ĠTrans former Ġwhich Ġalready Ġincludes Ġa Ġstrong Ġpre - trained Ġenc oder Ġby Ġa Ġlarge Ġmargin . ĠCompared Ġto Ġthe Ġbest Ġpublished Ġabstract ive Ġmodels , ĠStep Ġimproves Ġthe ĠR OU GE - 2 Ġby Ġ0 . 8 Ġon Ġthe ĠCNN / Daily Mail Ġdataset Ġand Ġby Ġ2 . 4 Ġon Ġthe ĠNew ĠYork ĠTimes Ġdataset Ġusing Ġour Ġbest Ġperforming Ġtask Ġfor Ġpre - training . ĠHuman Ġexperiments Ġalso Ġshow Ġthat ĠStep Ġcan Ġproduce Ġsignificantly Ġbetter Ġsumm aries Ġin Ġcomparison Ġwith Ġrecent Ġstrong Ġabstract ive Ġmodels . </s> Related ĠWork </s> This Ġsection Ġintroduces Ġextract ive Ġand Ġabstract ive Ġdocument Ġsummar ization Ġas Ġwell Ġas Ġpre - training Ġmethods Ġfor Ġnatural Ġlanguage Ġprocessing Ġtasks . </s> Related ĠWork Ġ: :: ĠExtract ive ĠSum mar ization </s> Ext ractive Ġsummar ization Ġsystems Ġlearn Ġto Ġfind Ġthe Ġinformative Ġsentences Ġin Ġa Ġdocument Ġas Ġits Ġsummary . ĠThis Ġtask Ġis Ġusually Ġviewed Ġas Ġa Ġsentence Ġranking Ġproblem ĠB IB REF 13 , ĠB IB REF 14 Ġusing Ġscores Ġfrom Ġa Ġbinary Ġ( sequence ) Ġclassification Ġmodel , Ġwhich Ġpredicts Ġwhether Ġa Ġsentence Ġis Ġin Ġthe Ġsummary Ġor Ġnot . ĠExtract ive Ġneural Ġmodels Ġemploy Ġhierarchical ĠL ST Ms / CNN s Ġas Ġthe Ġfeature Ġlearning Ġpart Ġof Ġthe Ġbinary Ġ( sequence ) Ġclass ifier ĠB IB REF 15 , ĠB IB REF 16 , ĠB IB REF 17 , ĠB IB REF 18 , Ġwhich Ġlargely Ġoutper forms Ġdiscrete Ġfeature Ġbased Ġmodels ĠB IB REF 19 , ĠB IB REF 20 , ĠB IB REF 21 . ĠVery Ġrecently , Ġthe Ġfeature Ġlearning Ġpart Ġwas Ġreplaced Ġagain Ġwith Ġpre - trained Ġtransform ers ĠB IB REF 22 , ĠB IB REF 23 Ġthat Ġlead Ġto Ġanother Ġhuge Ġimprovement Ġof Ġsummar ization Ġperformance . ĠHowever , Ġextract ive Ġmodels Ġhave Ġtheir Ġown Ġlimitations . ĠFor Ġexample , Ġthe Ġextracted Ġsentences Ġmight Ġbe Ġtoo Ġlong Ġand Ġredundant . ĠBesides , Ġhuman Ġwritten Ġsumm aries Ġin Ġtheir Ġnature Ġare Ġabstract ive . ĠTherefore , Ġwe Ġfocus Ġon Ġabstract ive Ġsummar ization Ġin Ġthis Ġpaper . </s> Related ĠWork Ġ: :: ĠAbstract ive ĠSum mar ization </s> The Ġgoal Ġof Ġabstract ive Ġsummar ization Ġis Ġto Ġgenerate Ġsumm aries Ġby Ġrewriting Ġa Ġdocument , Ġwhich Ġis Ġa Ġsequence - to - sequence Ġlearning Ġproblem . Ġseq 2 seq Ġattentive ĠL ST Ms ĠB IB REF 8 , ĠB IB REF 24 Ġare Ġemployed Ġin Ġn all ap ati 2016 ab stract ive . ĠEven Ġthese Ġmodels Ġare Ġextended Ġwith Ġcopy Ġmechanism ĠB IB REF 25 , Ġcoverage Ġmodel ĠB IB REF 11 Ġand Ġreinforcement Ġlearning ĠB IB REF 26 , Ġtheir Ġresults Ġare Ġstill Ġvery Ġclose Ġto Ġthat Ġof ĠLead 3 Ġwhich Ġselects Ġthe Ġleading Ġthree Ġsentences Ġof Ġa Ġdocument Ġas Ġits Ġsummary . ĠOne Ġpossible Ġreason Ġis Ġthat ĠL ST Ms Ġwithout Ġpre - training Ġare Ġnot Ġpowerful Ġenough . Ġli u 2019 text Ġused Ġa Ġseq 2 seq ĠTrans former Ġmodel Ġwith Ġits Ġenc oder Ġinitialized Ġwith Ġa Ġpre - trained ĠTrans former Ġ( i . e ., ĠB ERT ; ĠB IB REF 2 ) Ġand Ġachieved Ġthe Ġstate - of - the - art Ġperformance . ĠOur Ġwork Ġgoes Ġone Ġstep Ġfurther , Ġwe Ġpropose Ġa Ġmethod Ġto Ġpre - train Ġthe Ġdec oder Ġtogether Ġwith Ġthe Ġenc oder Ġand Ġthen Ġinitialize Ġboth Ġthe Ġenc oder Ġand Ġdec oder Ġof Ġa Ġsummar ization Ġmodel Ġwith Ġthe Ġpre - trained ĠTransformers . </s> There Ġis Ġalso Ġa Ġline Ġof Ġwork Ġthat Ġbridges Ġextract ive Ġand Ġabstract ive Ġmodels Ġwith Ġreinforcement Ġlearning ĠB IB REF 27 , Ġattention Ġfusion ĠB IB REF 28 Ġand Ġbottom - up Ġattention ĠB IB REF 29 , Ġwhile Ġour Ġmodel Ġis Ġconcept ually Ġsimpler . </s> Related ĠWork Ġ: :: ĠPre - training </s> Pre - training Ġmethods Ġdraw Ġa Ġlot Ġof Ġattention Ġrecently . Ġpet ers 2018 deep Ġand Ġrad ford : 2019 : ar x iv Ġpre - trained ĠL ST M Ġand ĠTrans former Ġenc od ers Ġusing Ġlanguage Ġmodeling Ġobjectives . ĠTo Ġleverage Ġthe Ġcontext Ġin Ġboth Ġdirections , ĠB IB REF 2 Ġproposed ĠB ERT , Ġwhich Ġis Ġtrained Ġwith Ġthe Ġmask Ġlanguage Ġmodeling Ġobjective . ĠXL Net ĠB IB REF 3 Ġis Ġtrained Ġwith Ġperm utation Ġlanguage Ġmodeling Ġobjective , Ġwhich Ġremoves Ġthe Ġindependence Ġassumption Ġof Ġmasked Ġtokens Ġin ĠB ERT . ĠRo BER Ta ĠB IB REF 4 Ġextends ĠB ERT Ġwith Ġmore Ġtraining Ġdata Ġand Ġbetter Ġtraining Ġstrategies . ĠAll Ġthe Ġmethods Ġabove Ġfocus Ġon Ġpre - training Ġan Ġenc oder , Ġwhile Ġwe Ġpropose Ġmethods Ġto Ġpre - train Ġboth Ġthe Ġenc oder Ġand Ġdec oder Ġof Ġa Ġseq 2 seq Ġmodel . </s> d ong 2019 un ified Ġproposed Ġa ĠTrans former Ġlanguage Ġmodel Ġthat Ġcan Ġbe Ġused Ġfor Ġboth Ġnatural Ġlanguage Ġunderstanding Ġand Ġgeneration Ġtasks , Ġwhich Ġis Ġpre - trained Ġusing Ġmasked , Ġun id irection al Ġand Ġseq 2 seq Ġlanguage Ġmodeling Ġobjectives . ĠTheir Ġmethod Ġtries Ġto Ġpre - train Ġa Ġseq 2 seq ĠTrans former Ġwith Ġits Ġenc oder Ġand Ġdec oder Ġparameters Ġshared . ĠDifferent ly , Ġwe Ġpre - train Ġa Ġseq 2 seq ĠTrans former Ġwith Ġseparate Ġparameters Ġfor Ġthe Ġenc oder Ġand Ġdec oder . Ġsong 2019 mass Ġproposed Ġa Ġmethod Ġto Ġpre - train Ġa Ġseq 2 seq ĠTrans former Ġby Ġmask ing Ġa Ġspan Ġof Ġtext Ġand Ġthen Ġpredicting Ġthe Ġoriginal Ġtext Ġwith Ġmasked Ġtokens Ġat Ġother Ġpositions . ĠTheir Ġpre - training Ġtask Ġis Ġsimilar Ġto Ġour ĠMask ed ĠDocument ĠGeneration Ġtask , Ġbut Ġwe Ġapply Ġa Ġdifferent Ġmask ing Ġstrategy Ġand Ġpredict Ġthe Ġoriginal Ġtext Ġwithout Ġmasked Ġtokens . ĠBesides , Ġwe Ġpropose Ġanother Ġtwo Ġtasks Ġfor Ġseq 2 seq Ġmodel Ġpre - training . ĠB IB REF 30 Ġtested Ġtheir Ġmodel Ġon Ġsentence - level Ġtasks Ġ( e . g ., Ġmachine Ġtranslation Ġand Ġsentence Ġcompression ), Ġwhile Ġwe Ġaim Ġto Ġsolve Ġdocument - level Ġtasks Ġ( e . g ., Ġabstract ive Ġdocument Ġsummar ization ). </s> Sequ ence - to - Sequ ence ĠTrans former ĠPre - training </s> This Ġsection Ġfirst Ġintroduces Ġthe Ġbackbone Ġarchitecture Ġof Ġour Ġabstract ive Ġsummar ization Ġmodel ĠStep . ĠWe Ġthen Ġdescribe Ġmethods Ġto Ġpre - train ĠStep Ġand Ġfinally Ġmove Ġon Ġto Ġthe Ġfine - tun ing Ġon Ġsummar ization Ġdatasets . </s> Sequ ence - to - Sequ ence ĠTrans former ĠPre - training Ġ: :: ĠArchitecture </s> In Ġthis Ġwork , Ġthe Ġtask Ġof Ġabstract ive Ġdocument Ġsummar ization Ġis Ġmodeled Ġas Ġa Ġsequence - to - sequence Ġlearning Ġproblem , Ġwhere Ġa Ġdocument Ġis Ġviewed Ġas Ġa Ġsequence Ġof Ġtokens Ġand Ġits Ġcorresponding Ġsummary Ġas Ġanother Ġsequence Ġof Ġtokens . ĠWe Ġadopt Ġthe Ġseq 2 seq ĠTrans former Ġarchitecture ĠB IB REF 9 , Ġwhich Ġincludes Ġan Ġenc oder ĠTrans former Ġand Ġa Ġdec oder ĠTrans former . ĠBoth Ġthe Ġenc oder Ġand Ġdec oder ĠTransformers Ġhave Ġmultiple Ġlayers Ġand Ġeach Ġlayer Ġcontains Ġa Ġmulti - head Ġattentive Ġsub - layer Ġfollowed Ġby Ġa Ġfully Ġconnected Ġsub - layer Ġwith Ġresidual Ġconnections ĠB IB REF 31 Ġand Ġlayer Ġnormal ization ĠB IB REF 32 . </s> Let Ġus Ġuse Ġ$ X Ġ= Ġ( x _ 1 , Ġx _ 2 , Ġ\ d ots Ġ, Ġx _{ | X | }) $ Ġto Ġdenote Ġa Ġdocument Ġand Ġuse Ġ$ Y Ġ= Ġ( y _ 1 , Ġy _ 2 , Ġ\ d ots Ġ, Ġy _{ | Y | }) $ Ġto Ġdenote Ġits Ġsummary . ĠThe Ġenc oder Ġtakes Ġthe Ġdocument Ġ$ X $ Ġas Ġinput Ġand Ġtransforms Ġit Ġto Ġits Ġcontextual Ġrepresentations . ĠThe Ġdec oder Ġlearns Ġto Ġgenerate Ġthe Ġsummary Ġ$ Y $ Ġone Ġtoken Ġat Ġa Ġtime Ġbased Ġon Ġthe Ġcontextual Ġrepresentations Ġand Ġall Ġpreceding Ġtokens Ġthat Ġhave Ġbeen Ġgenerated Ġso Ġfar : </s> where Ġ$ y _{ < t } $ Ġstands Ġfor Ġall Ġtokens Ġbefore Ġposition Ġ$ t $ Ġ( i . e ., Ġ$ y _{ < t } =( y _ 1 , Ġy _ 2 , Ġ\ d ots Ġ, Ġy _{ t - 1 }) $ ). ĠThis Ġmodel Ġcan Ġbe Ġtrained Ġby Ġminimizing Ġthe Ġnegative Ġlog - like lihood Ġof Ġthe Ġtraining Ġdocument - summary Ġpairs . </s> Sequ ence - to - Sequ ence ĠTrans former ĠPre - training Ġ: :: ĠPre - training ĠT asks </s> Training Ġa Ġseq 2 seq ĠTrans former Ġmodel Ġon Ġa Ġsummar ization Ġdataset Ġfrom Ġscratch Ġis Ġdifficult Ġdue Ġto Ġthe Ġlimited Ġnumber Ġof Ġdocument - summary Ġpairs . ĠPre - trained ĠTrans former Ġenc od ers Ġsuch Ġas ĠB ERT ĠB IB REF 2 Ġand ĠRo BER Ta ĠB IB REF 4 Ġhave Ġachieved Ġgreat Ġsuccess Ġin Ġmany Ġnatural Ġlanguage Ġunderstanding Ġtasks . ĠTherefore , Ġwe Ġfirst Ġinitialize Ġthe Ġenc oder Ġof Ġour Ġseq 2 seq ĠTrans former Ġsummar ization Ġmodel ĠStep Ġwith Ġan Ġexisting Ġpre - trained ĠTrans former Ġenc oder Ġ( i . e ., ĠRo BER Ta ) Ġto Ġenhance Ġits Ġlanguage Ġunderstanding Ġcapabilities . ĠTo Ġhelp ĠStep Ġgain Ġlanguage Ġgeneration Ġcapabilities Ġand Ġthe Ġabilities Ġof Ġassoci ating Ġgenerated Ġtext Ġwith Ġenc oder Ġoutputs , Ġwe Ġcontinue Ġto Ġpre - train Ġit Ġon Ġunl abel ed Ġtext . ĠIn Ġthe Ġfollowing , Ġwe Ġdescribe Ġour Ġpre - training Ġtasks . </s> Sequ ence - to - Sequ ence ĠTrans former ĠPre - training Ġ: :: ĠPre - training ĠT asks Ġ: :: ĠSent ence ĠRe ordering </s> A Ġdocument Ġis Ġtypically Ġcomposed Ġof Ġmultiple Ġsentences Ġseparated Ġby Ġfull Ġstops . ĠIn Ġthis Ġtask , Ġwe Ġfirst Ġshuffle Ġthe Ġdocument Ġby Ġsentences Ġand Ġthen Ġrecover Ġthe Ġoriginal Ġdocument . ĠThere Ġare Ġseveral Ġreasons Ġwhy Ġwe Ġdesign Ġthis Ġtask . ĠFirst , Ġa Ġsummary Ġof Ġa Ġdocument Ġusually Ġconsists Ġof Ġmultiple Ġsentences . ĠWe Ġexpect Ġthat ĠStep Ġlearns Ġto Ġgenerate Ġlong Ġand Ġcoherent Ġsumm aries Ġ( ac ross Ġsentences ). ĠThe Ġoutput Ġof Ġthe Ġtask Ġ( i . e ., Ġthe Ġoriginal Ġdocument ) Ġalso Ġcontains Ġmultiple Ġsentences . ĠSecond , Ġsentence Ġre ordering Ġ( or Ġcontent Ġre ordering ) Ġis Ġnecessary Ġfor Ġsummar ization . ĠAccording Ġto Ġthe Ġstatistics Ġon Ġtraining Ġsets Ġof Ġour Ġsummar ization Ġdatasets , Ġcontents Ġof Ġthe Ġoriginal Ġdocuments Ġare Ġre ordered Ġin Ġtheir Ġsumm aries Ġfor Ġ40 % Ġof Ġcases . ĠWe Ġdefine Ġcontent Ġre ordering Ġas Ġfollows . ĠFor Ġeach Ġdocument - summary Ġpair , Ġwe Ġfirst Ġmap Ġeach Ġsentence Ġin Ġthe Ġsummary Ġto Ġone Ġsentence Ġin Ġits Ġpaired Ġdocument Ġby Ġmaximizing Ġthe ĠR OU GE Ġscore . ĠIf Ġthe Ġsequence Ġof Ġsentences Ġin Ġthe Ġsummary Ġis Ġdifferent Ġfrom Ġthe Ġsequence Ġof Ġtheir Ġmapped Ġsentences Ġin Ġthe Ġoriginal Ġdocument , Ġwe Ġcount Ġthis Ġas Ġone Ġcontent Ġre ordering . ĠThird ly , Ġabstract ive Ġsummary Ġrequires Ġreprodu cing Ġfactual Ġdetails Ġ( e . g ., Ġnamed Ġentities , Ġfigures ) Ġfrom Ġsource Ġtext . ĠWe Ġalso Ġexpect ĠStep Ġto Ġlearn Ġto Ġcopy Ġtokens . ĠHere Ġis Ġa Ġformal Ġdefinition Ġof Ġthis Ġtask . ĠLet Ġus Ġchange Ġthe Ġnotation Ġof Ġa Ġdocument Ġslightly Ġin Ġthis Ġparagraph . ĠLet Ġ$ X =( S _ 1 , ĠS _ 2 , Ġ\ d ots Ġ, ĠS _ m ) $ Ġdenote Ġa Ġdocument , Ġwhere Ġ$ S _ i Ġ= Ġ( w ^ i _ 1 , Ġw ^ i _ 2 , Ġ\ d ots Ġ, Ġw ^ i _{ | S _ i | }) $ Ġis Ġa Ġsentence Ġin Ġit , Ġ$ w ^ i _ j $ Ġis Ġa Ġword Ġin Ġ$ S _ i $ Ġand Ġ$ m $ Ġis Ġthe Ġnumber Ġof Ġsentences . Ġ$ X $ Ġis Ġstill Ġa Ġsequence Ġof Ġtokens Ġ( by Ġconc aten ating Ġtokens Ġin Ġall Ġsentences ). ĠLet Ġ$ A = \ text {\ tt Ġperm utation } ( m )=( a _ 1 , a _ 2 , \ d ots Ġ, Ġa _ m ) $ Ġdenote Ġa Ġperm uted Ġrange Ġof Ġ$( 1 , Ġ2 , Ġ\ d ots Ġ, Ġm ) $ Ġand Ġtherefore Ġ$\ hat { X } _ S =( S _{ a _ 1 }, ĠS _{ a _ 2 }, Ġ\ d ots Ġ, ĠS _{ a _ m }) $ Ġis Ġthe Ġshuff led Ġdocument . ĠNote Ġthat Ġ$\ hat { X } _ S $ Ġis Ġa Ġsequence Ġof Ġtokens Ġby Ġconc aten ating Ġall Ġshuff led Ġsentences . ĠStep Ġcan Ġbe Ġtrained Ġon Ġ$\ l angle Ġ\ hat { X } _ S , ĠX Ġ\ r angle Ġ$ Ġpairs Ġconstructed Ġfrom Ġunl abel ed Ġdocuments , Ġas Ġdemonstrated Ġin ĠFigure ĠFIG REF 5 . </s> Note Ġthat Ġdocument Ġrotation Ġis Ġa Ġspecial Ġcase Ġof Ġsentence Ġre ordering Ġwith Ġsignificant Ġamount Ġof Ġpartially Ġordered Ġsentences , Ġwhich Ġwe Ġbelieve Ġis Ġa Ġsimpler Ġtask . ĠIn Ġthis Ġwork , Ġwe Ġthus Ġonly Ġconsider Ġthe Ġgeneral Ġcase Ġof Ġsentence Ġre ordering . </s> Sequ ence - to - Sequ ence ĠTrans former ĠPre - training Ġ: :: ĠPre - training ĠT asks Ġ: :: ĠNext ĠSent ence ĠGeneration </s> The Ġsecond Ġpre - training Ġtask Ġlever ages Ġthe Ġnatural Ġorder Ġof Ġtext . ĠNext ĠSent ence ĠGeneration Ġ( NS G ) Ġuses Ġone Ġspan Ġof Ġtext Ġin Ġa Ġdocument Ġto Ġpredict Ġits Ġnext Ġspan Ġof Ġtext , Ġas Ġshown Ġin ĠFigure ĠFIG REF 5 . ĠSpecifically , Ġwe Ġsplit Ġa Ġdocument Ġinto Ġtwo Ġsegments Ġ( i . e ., Ġ$ G _ 1 $ Ġand Ġ$ G _ 2 $ ). ĠNote Ġthat Ġeach Ġsegment Ġmight Ġcontain Ġmultiple Ġsentences , Ġwhich Ġfits Ġthe Ġdocument Ġsummar ization Ġtask Ġvery Ġwell , Ġsince Ġeither Ġa Ġdocument Ġor Ġits Ġsummary Ġusually Ġincludes Ġmultiple Ġsentences . ĠInt uitive ly , Ġin Ġa Ġdocument , Ġsentences Ġare Ġhighly Ġcorrelated Ġwith Ġtheir Ġpreceding Ġsentences Ġdue Ġto Ġthe Ġcontext Ġdependent Ġnature Ġof Ġdocuments Ġor Ġlanguage . ĠWe Ġintend Ġour Ġmodel Ġto Ġlearn Ġto Ġgenerate Ġmultiple Ġsentences Ġand Ġalso Ġlearn Ġto Ġfocus Ġon Ġpreceding Ġcontext . </s> We Ġhave Ġat Ġleast Ġtwo Ġoptions Ġfor Ġthe Ġsplitting Ġposition Ġof Ġthe Ġtwo Ġsegments . ĠOption Ġone : Ġthe Ġposition Ġright Ġafter Ġa Ġfull - stop Ġsymbol Ġ( such Ġas Ġperiod , Ġquestion Ġmark , Ġetc .) Ġis Ġselected Ġas Ġthe Ġsplitting Ġpoint , Ġwhich Ġensures Ġfull Ġsentences Ġfor Ġeach Ġsegment . ĠOption Ġtwo : Ġthe Ġsplitting Ġpoint Ġcan Ġbe Ġat Ġany Ġposition Ġwithin Ġthe Ġdocument . ĠWe Ġchoose Ġthe Ġsecond Ġoption , Ġwhich Ġmay Ġlead Ġto Ġincomplete Ġsentences Ġin Ġsegments . ĠWe Ġintend Ġto Ġforce Ġthe Ġenc oder Ġand Ġdec oder Ġto Ġunderstand Ġinput Ġtext Ġwithout Ġcomplete Ġinformation , Ġwhich Ġwe Ġbelieve Ġis Ġmore Ġchallenging Ġcompared Ġto Ġoption Ġone . ĠBesides , Ġas Ġa Ġcommon Ġwisdom Ġin Ġabstract ive Ġsummar ization , Ġdocuments Ġare Ġtrunc ated Ġto Ġa Ġfixed Ġnumber Ġof Ġtokens , Ġwhich Ġmay Ġalso Ġcontain Ġincomplete Ġsentences . ĠWe Ġuse Ġoption Ġtwo Ġto Ġreduce Ġthe Ġpre - training Ġand Ġfine - tun ing Ġinput Ġmismatch . ĠIn Ġthis Ġtask , Ġwe Ġtrain Ġthe Ġmodel ĠStep Ġon Ġlarge Ġamount Ġof Ġ$\ l angle ĠG _ 1 , ĠG _ 2 \ r angle Ġ$ Ġpairs Ġconstructed Ġfollowing Ġthe Ġoption Ġtwo Ġsplitting Ġstrategy . </s> Next Ġsentence Ġprediction Ġhas Ġbeen Ġused Ġin Ġskip - thought Ġvectors ĠB IB REF 33 . ĠThere Ġare Ġtwo Ġdifferences . ĠFirst , Ġeach Ġsegment Ġin Ġtheir Ġmodel Ġonly Ġhas Ġone Ġsentence ; Ġsecond , Ġthey Ġuse Ġthis Ġtask Ġto Ġpre - train Ġan Ġenc oder Ġrather Ġthan Ġan Ġentire Ġseq 2 seq Ġmodel . ĠB IB REF 2 Ġintroduced Ġa Ġtask Ġnamed Ġnext Ġsentence Ġprediction Ġ( N SP ), Ġwhich Ġis Ġdifferent Ġfrom Ġthis Ġtask . ĠN SP Ġis Ġa Ġclassification Ġtask , Ġbut ĠNS G Ġis Ġa Ġgeneration Ġtask , Ġwhich Ġintends Ġto Ġpre - train Ġa Ġgeneration Ġmodel . </s> Sequ ence - to - Sequ ence ĠTrans former ĠPre - training Ġ: :: ĠPre - training ĠT asks Ġ: :: ĠMask ed ĠDocument ĠGeneration </s> The Ġthird Ġtask Ġwe Ġconsider Ġis ĠMask ed ĠDocument ĠGeneration Ġ( MD G ) Ġthat Ġlearns Ġto Ġrecover Ġa Ġdocument Ġwith Ġa Ġmasked Ġspan Ġof Ġtokens Ġ( see ĠFigure ĠFIG REF 5 ). ĠFor Ġsimplicity , Ġa Ġdocument Ġconsisting Ġof Ġa Ġsequence Ġof Ġtokens Ġis Ġden oted Ġas Ġ$ X =( x _ 1 , Ġx _ 2 , Ġ\ cd ots Ġ, Ġx _{ | X | }) $. ĠWe Ġrandomly Ġsample Ġthe Ġlength Ġof Ġthe Ġspan Ġ$ l $ Ġfrom Ġa Ġdiscrete Ġuniform Ġdistribution Ġ$\ math cal Ġ{ U } ( a , Ġb ) $ Ġand Ġthe Ġspan Ġstart Ġposition Ġ$ k $ Ġfrom Ġanother Ġdiscrete Ġuniform Ġdistribution Ġ$\ math cal Ġ{ U } ( 1 , Ġ| X | - l + 1 ) $ Ġ( see ĠSection ĠSEC REF 4 Ġfor Ġmore Ġdetails ). ĠThus , Ġ$\ math cal Ġ{ M } =( x _ k , Ġx _{ k + 1 }, Ġ\ cd ots Ġ, Ġx _{ k + l - 1 }) $ Ġis Ġthe Ġtext Ġspan Ġto Ġbe Ġmasked . </s> One Ġstraightforward Ġmask ing Ġstrategy Ġis Ġto Ġreplace Ġeach Ġtoken Ġresiding Ġin Ġ$\ math cal Ġ{ M } $ Ġwith Ġa Ġspecial Ġ[ MAS K ] Ġtoken . ĠHowever , Ġwe Ġrefrain Ġfrom Ġdoing Ġso Ġbecause Ġof Ġthe Ġfollowing Ġthree Ġreasons . ĠUsually , Ġ[ MAS K ] Ġtokens Ġwill Ġnot Ġappear Ġin Ġdownstream Ġtasks . ĠSecond , Ġentirely Ġmask ing Ġa Ġcontinuous Ġsub - sequence Ġof Ġ$ X $ Ġmay Ġmake Ġthe Ġwhole Ġdocument Ġincomprehensible , Ġwhich Ġmight Ġbe Ġtoo Ġchallenging Ġfor Ġour Ġmodel Ġto Ġlearn . ĠThird , Ġsimilar Ġto ĠSR , Ġavoiding Ġreplacing Ġevery Ġtoken Ġwith Ġ[ MAS K ] Ġalso Ġhelps Ġour Ġmodel Ġlearn Ġthe Ġability Ġof Ġcopying Ġtokens Ġfrom Ġthe Ġinput Ġwhile Ġpreserving Ġthe Ġability Ġof Ġgenerating Ġnovel Ġtokens . </s> In Ġthe Ġsub - sequence Ġ$\ math cal Ġ{ M } $, Ġeach Ġtoken Ġis Ġprocessed Ġwith Ġone Ġof Ġthe Ġthree Ġstrategies : Ġ1 ) Ġreplaced Ġwith Ġthe Ġ[ MAS K ] Ġtoken ; Ġ2 ) Ġreplaced Ġwith Ġa Ġrandom Ġtoken ; Ġ3 ) Ġremain Ġunchanged . ĠInspired Ġby ĠB ERT ĠB IB REF 2 , Ġfor Ġ80 % Ġtokens , Ġwe Ġfollow Ġstrategy Ġ1 ). ĠIn Ġ10 % Ġof Ġcases , Ġwe Ġemploy Ġstrategy Ġ2 ) Ġand Ġwe Ġuse Ġstrategy Ġ3 ) Ġfor Ġthe Ġremaining Ġ10 % Ġof Ġcases . ĠLet Ġ$\ hat { X } _ M $ Ġdenote Ġthe Ġdocument Ġafter Ġthe Ġapplication Ġof Ġour Ġmask ing Ġstrategy . ĠWe Ġcould Ġcreate Ġinfinite Ġamount Ġof Ġ$\ l angle Ġ\ hat { X } _ M , X \ r angle Ġ$ Ġpairs Ġto Ġtrain ĠStep . </s> During Ġpre - training , Ġwe Ġcould Ġalso Ġemploy Ġall Ġthe Ġthree Ġtasks Ġ( i . e ., ĠSR , ĠNS G , ĠMD G ) Ġtogether . ĠFor Ġeach Ġtraining Ġbatch , Ġwe Ġrandomly Ġchoose Ġone Ġtask Ġand Ġeach Ġtask Ġis Ġused Ġfor Ġ$ 1 / 3 $ Ġof Ġthe Ġtime . </s> Sequ ence - to - Sequ ence ĠTrans former ĠPre - training Ġ: :: ĠFine - tun ing </s> After Ġpre - training ĠStep Ġwith Ġthe Ġthree Ġtasks Ġintroduced Ġin ĠSection ĠSEC REF 9 , Ġwe Ġfine - t une Ġthe Ġmodel Ġon Ġabstract ive Ġdocument Ġsummar ization Ġdatasets . ĠThe Ġfine - tun ing Ġprocess Ġis Ġstraightforward . ĠWe Ġsimply Ġcontinue Ġto Ġtrain ĠStep Ġon Ġthe Ġsupervised Ġdocument - summary Ġpairs . ĠSimilar Ġto Ġother Ġseq 2 seq Ġsummar ization Ġmodels , Ġwe Ġdo Ġbeam Ġsearch Ġduring Ġthe Ġgeneration Ġof Ġsumm aries . </s> Exper imental ĠSetup </s> In Ġthis Ġsection , Ġwe Ġpresent Ġthe Ġexperimental Ġsetup Ġfor Ġevaluating Ġour Ġsummar ization Ġmodels . ĠWe Ġfirst Ġintroduce Ġthe Ġdatasets Ġused Ġfor Ġour Ġexperiments . ĠThen Ġwe Ġdescribe Ġtraining Ġdetails Ġof Ġour Ġmodels Ġas Ġwell Ġas Ġour Ġevaluation Ġprotocols . </s> Exper imental ĠSetup Ġ: :: ĠDat as ets </s> We Ġassess Ġthe Ġsummar ization Ġperformance Ġof Ġour Ġmodels Ġon Ġtwo Ġbenchmark Ġdatasets : Ġthe ĠCNN / Daily Mail Ġ( CNN DM ) Ġdataset ĠB IB REF 34 , ĠB IB REF 11 Ġand Ġthe ĠNew ĠYork ĠTimes Ġ( NY T ) Ġdataset ĠB IB REF 35 . ĠWe Ġpre - train Ġour Ġmodels Ġon Ġthe ĠG IG A - CM Ġdataset Ġintroduced Ġin Ġz hang - et al - 2019 - h iber t . </s> Exper imental ĠSetup Ġ: :: ĠDat as ets Ġ: :: ĠCNN DM </s> CNN DM Ġcontains Ġnews Ġarticles Ġand Ġtheir Ġassociated Ġhighlights Ġ( i . e ., Ġsumm aries ) Ġcollected Ġfrom Ġthe ĠCNN Ġand ĠDaily ĠMail ĠOnline Ġwebsites . ĠFollowing Ġprevious Ġwork ĠB IB REF 11 , ĠB IB REF 22 , ĠB IB REF 23 , Ġwe Ġuse Ġthe Ġnon - an onym ized Ġversion Ġof ĠCNN DM . ĠSpecifically , Ġwe Ġpre process Ġthe Ġdataset Ġwith Ġthe Ġpublicly Ġavailable Ġscripts Ġprovided Ġby Ġsee 2017 get Ġand Ġobtain Ġ287 , 226 Ġdocument - summary Ġpairs Ġfor Ġtraining , Ġ13 , 368 Ġfor Ġvalidation Ġand Ġ11 , 490 Ġfor Ġtest . </s> Exper imental ĠSetup Ġ: :: ĠDat as ets Ġ: :: ĠNYT </s> The ĠNYT Ġdataset Ġis Ġa Ġcollection Ġof Ġarticles Ġalong Ġwith Ġmulti - sent ence Ġsumm aries Ġwritten Ġby Ġlibrary Ġscientists . ĠWe Ġclosely Ġfollow Ġthe Ġpre processing Ġprocedures Ġdescribed Ġin Ġdur rett 2016 learning Ġand Ġli u 2019 text . ĠThe Ġtest Ġset Ġis Ġconstructed Ġby Ġincluding Ġall Ġarticles Ġpublished Ġon ĠJanuary Ġ1 , Ġ2017 Ġor Ġlater , Ġwhich Ġcontains Ġ9 , 07 6 Ġarticles . ĠThe Ġremaining Ġ100 , 8 34 Ġarticles Ġare Ġsplit Ġinto Ġa Ġtraining Ġset Ġof Ġ96 , 8 34 Ġexamples Ġand Ġa Ġvalidation Ġset Ġof Ġ4 , 000 Ġexamples . ĠAs Ġin ĠB IB REF 36 , Ġwe Ġalso Ġremove Ġarticles Ġwhose Ġsumm aries Ġcontain Ġless Ġthan Ġ50 Ġwords Ġfrom Ġthe Ġtest Ġset , Ġand Ġthe Ġresulting Ġtest Ġset Ġcontains Ġ3 , 452 Ġexamples . </s> Exper imental ĠSetup Ġ: :: ĠDat as ets Ġ: :: ĠG IG A - CM </s> To Ġpre - train Ġour Ġmodel Ġwith Ġthe Ġtasks Ġintroduced Ġin ĠSection ĠSEC REF 9 , Ġfollowing Ġthe Ġprocedures Ġin ĠB IB REF 22 , Ġwe Ġcreated Ġthe ĠG IG A - CM Ġdataset , Ġwhich Ġcontains Ġonly Ġunl abel ed Ġdocuments . ĠThe Ġtraining Ġset Ġof ĠG IG A - CM Ġis Ġcomposed Ġof Ġ6 , 5 21 , 658 Ġdocuments Ġsampled Ġfrom Ġthe ĠEnglish ĠGig aw ord Ġdataset Ġand Ġthe Ġtraining Ġdocuments Ġin ĠCNN DM . ĠWe Ġused Ġthe Ġ13 , 368 Ġdocuments Ġin Ġthe Ġvalidation Ġsplit Ġof ĠCNN DM Ġas Ġits Ġvalidation Ġset . ĠNote Ġthat Ġthe ĠGig aw ord Ġdataset Ġoverl aps Ġwith Ġthe ĠNYT Ġdataset Ġand Ġwe Ġtherefore Ġexclude Ġthe Ġtest Ġset Ġof ĠNYT Ġfrom Ġthe Ġtraining Ġset Ġof ĠG IG A - CM . </s> For ĠCNN DM , ĠNYT Ġand ĠG IG A - CM Ġdatasets , Ġwe Ġsegment Ġand Ġtoken ize Ġdocuments Ġand / or Ġsumm aries Ġ( G IG A - CM Ġonly Ġcontains Ġdocuments ) Ġusing Ġthe ĠStanford ĠCore N LP Ġtool kit ĠB IB REF 37 . ĠTo Ġreduce Ġthe Ġvocabulary Ġsize , Ġwe Ġfurther Ġapply Ġthe ĠUTF 8 Ġbased ĠB PE ĠB IB REF 38 Ġintroduced Ġin ĠG PT - 2 ĠB IB REF 39 Ġto Ġall Ġdatasets . ĠAs Ġa Ġcommon Ġwisdom Ġin Ġabstract ive Ġsummar ization , Ġdocuments Ġand Ġsumm aries Ġin ĠCNN DM Ġand ĠNYT Ġare Ġusually Ġtrunc ated Ġto Ġ512 Ġand Ġ256 Ġtokens , Ġrespectively . </s> We Ġleverage Ġunl abel ed Ġdocuments Ġdifferently Ġfor Ġdifferent Ġpre - training Ġtasks Ġ( see ĠSection ĠSEC REF 9 ). ĠWe Ġfirst Ġsplit Ġeach Ġdocument Ġinto Ġ512 Ġtoken Ġsegments Ġif Ġit Ġcontains Ġmore Ġthan Ġ512 Ġtokens Ġ( se gments Ġor Ġdocuments Ġwith Ġless Ġthan Ġ512 Ġtokens Ġare Ġremoved ). ĠIn ĠSent ence ĠRe ordering Ġ( SR ) Ġand ĠMask ed ĠDocument ĠGeneration Ġ( MD G ), Ġwe Ġuse Ġthe Ġsegment Ġafter Ġtransformation Ġto Ġpredict Ġthe Ġoriginal Ġsegment . ĠWe Ġset Ġthe Ġminimum Ġmasked Ġlength Ġ$ a = 100 $ Ġand Ġthe Ġmaximum Ġmasked Ġlength Ġ$ b = 256 $ Ġin ĠMD G . ĠIn ĠNext ĠSent ence ĠGeneration Ġ( NS G ), Ġeach Ġsegment Ġis Ġused Ġto Ġpredict Ġits Ġnext Ġ256 Ġtokens . </s> Exper imental ĠSetup Ġ: :: ĠImplementation ĠDetails </s> As Ġmentioned Ġin ĠSection ĠSEC REF 3 , Ġour Ġmodel Ġis Ġa ĠSe q 2 Se q ĠTrans former Ġmodel ĠB IB REF 9 . ĠThe Ġenc oder Ġis Ġinitialized Ġwith Ġthe Ġ$\ text { Ro BER Ta } _{ \ text { L AR GE }} $ Ġmodel ĠB IB REF 4 , Ġand Ġtherefore Ġthey Ġshare Ġthe Ġsame Ġarchitecture . ĠSpecifically , Ġthe Ġenc oder Ġis Ġa Ġ24 - layer ĠTrans former . ĠEach Ġlayer Ġhas Ġ16 Ġattention Ġheads Ġand Ġits Ġhidden Ġsize Ġand Ġfeed - forward Ġfilter Ġsize Ġare Ġ1 , 024 Ġand Ġ4 , 09 6 , Ġrespectively . ĠThe Ġdec oder Ġis Ġshall ower Ġwith Ġ6 Ġlayers . ĠThe Ġhidden Ġsize Ġand Ġnumber Ġof Ġattention Ġhead Ġof Ġthe Ġdec oder Ġare Ġidentical Ġto Ġthese Ġof Ġthe Ġenc oder , Ġbut Ġthe Ġfeed - forward Ġfilter Ġsize Ġis Ġ2 , 048 . ĠWe Ġuse Ġa Ġsmaller Ġfilter Ġsize Ġin Ġthe Ġdec oder Ġto Ġreduce Ġthe Ġcomputational Ġand Ġmemory Ġcost . ĠThe Ġdrop out Ġrates Ġof Ġall Ġlayers Ġin Ġthe Ġenc oder Ġare Ġset Ġto Ġ0 . 1 Ġand Ġall Ġdrop out Ġrates Ġin Ġthe Ġdec oder Ġare Ġset Ġto Ġ0 . 3 . ĠOur Ġmodels Ġare Ġoptimized Ġusing ĠAdam ĠB IB REF 40 Ġwith Ġ$\ beta Ġ_ 1 = 0 . 9 $, Ġ$\ beta Ġ_ 2 = 0 . 98 $. ĠThe Ġother Ġoptimization Ġhyper - param eters Ġfor Ġpre - training Ġand Ġfine - tun ing Ġare Ġdifferent . ĠIn Ġthe Ġpre - training Ġstage , Ġthe Ġenc oder Ġis Ġinitialized Ġwith Ġa Ġpre - trained Ġmodel Ġwhile Ġthe Ġdec oder Ġis Ġrandomly Ġinitialized . ĠTherefore , Ġwe Ġused Ġtwo Ġseparate Ġoptim izers Ġfor Ġthe Ġenc oder Ġand Ġdec oder Ġwith Ġa Ġsmaller Ġlearning Ġrate Ġfor Ġthe Ġenc oder Ġoptim izer . ĠLearning Ġrates Ġof Ġthe Ġenc oder Ġand Ġdec oder Ġare Ġset Ġto Ġ$ 2 e - 5 $ Ġand Ġ$ 1 e - 4 $ Ġwith Ġ10 , 000 Ġwarm up Ġsteps , Ġrespectively . ĠWe Ġalso Ġadopted Ġthe Ġsame Ġlearning Ġrate Ġschedule Ġstrategies Ġas ĠB IB REF 9 . ĠWe Ġused Ġsmaller Ġbatch Ġsizes Ġfor Ġdatasets Ġwith Ġless Ġexamples Ġ( i . e ., Ġ1 , 024 Ġfor ĠG IG A - CM , Ġ256 Ġfor ĠCNN DM Ġand Ġ128 Ġfor ĠNYT ) Ġto Ġensure Ġeach Ġepoch Ġhas Ġsufficient Ġnumber Ġof Ġmodel Ġupdates . ĠWe Ġtrained Ġour Ġmodels Ġuntil Ġtheir Ġconvergence Ġof Ġvalidation Ġperplex ities Ġ( around Ġ30 Ġepoch s Ġon ĠG IG A - CM , Ġ60 Ġepoch s Ġon ĠCNN DM Ġand Ġ40 Ġepoch s Ġon ĠNYT ). ĠOne Ġepoch Ġon ĠG IG A - CM Ġtakes Ġaround Ġ24 Ġhours Ġwith Ġ8 ĠNvidia ĠTesla ĠV 100 ĠGPUs . ĠThe Ġtime Ġcosts Ġfor Ġdifferent Ġpre - training Ġtasks Ġare Ġclose . </s> Most Ġof Ġthe Ġhyper - param eters Ġin Ġthe Ġfine - tun ing Ġstage Ġare Ġthe Ġsame Ġas Ġthese Ġin Ġthe Ġpre - training Ġstage . ĠThe Ġdifferences Ġare Ġas Ġfollows . ĠThe Ġlearning Ġrates Ġfor Ġboth Ġthe Ġenc oder Ġand Ġdec oder Ġare Ġset Ġto Ġ$ 2 e - 5 $ Ġwith Ġ4 , 000 Ġwarm up Ġsteps , Ġsince Ġboth Ġthe Ġenc oder Ġand Ġdec oder Ġare Ġalready Ġpre - trained . ĠWe Ġtrained Ġour Ġmodels Ġfor Ġ50 Ġepoch s Ġ( s aved Ġper Ġepoch ) Ġand Ġselected Ġthe Ġbest Ġmodel Ġw . r . t . ĠR OU GE Ġscore Ġon Ġthe Ġvalidation Ġset Ġ. ĠDuring Ġdecoding , Ġwe Ġapplied Ġbeam Ġsearch Ġwith Ġbeam Ġsize Ġof Ġ5 . ĠFollowing ĠB IB REF 26 , Ġwe Ġalso Ġblocked Ġrepeating Ġtrig rams Ġduring Ġbeam Ġsearch Ġand Ġtuned Ġthe Ġminimum Ġsummary Ġlength Ġon Ġthe Ġvalidation Ġset . ĠSimilar Ġto Ġthe Ġpre - training Ġprocess , Ġthe Ġdatasets Ġwith Ġless Ġinstances Ġwere Ġfine - tun ed Ġwith Ġsmaller Ġbatch Ġsize Ġ( i . e ., Ġ768 Ġfor ĠCNN DM Ġand Ġ64 Ġfor ĠNYT ). </s> Exper imental ĠSetup Ġ: :: ĠEval uations </s> We Ġused ĠR OU GE ĠB IB REF 41 Ġto Ġmeasure Ġthe Ġquality Ġof Ġdifferent Ġsummar ization Ġmodel Ġoutputs . ĠWe Ġreported Ġfull - length ĠF 1 Ġbased ĠR OU GE - 1 , ĠR OU GE - 2 Ġand ĠR OU GE - L Ġscores Ġon ĠCNN DM , Ġwhile Ġwe Ġused Ġthe Ġlimited - length Ġrecall Ġbased ĠR OU GE - 1 , ĠR OU GE - 2 Ġand ĠR OU GE - L Ġon ĠNYT Ġfollowing ĠB IB REF 36 . ĠThe ĠR OU GE Ġscores Ġare Ġcomputed Ġusing Ġthe ĠR OU GE - 1 . 5 . 5 . pl Ġscript . </s> Since Ġsumm aries Ġgenerated Ġby Ġabstract ive Ġmodels Ġmay Ġproduce Ġdis f luent Ġor Ġun gram matical Ġoutputs , Ġwe Ġalso Ġevaluated Ġabstract ive Ġsystems Ġby Ġelic iting Ġhuman Ġjud gements . ĠFollowing Ġprevious Ġwork ĠB IB REF 15 , ĠB IB REF 17 , Ġ20 Ġdocuments Ġare Ġrandomly Ġsampled Ġfrom Ġthe Ġtest Ġsplit Ġof ĠCNN DM . ĠParticipants Ġare Ġpresented Ġwith Ġa Ġdocument Ġand Ġa Ġlist Ġof Ġoutputs Ġgenerated Ġby Ġdifferent Ġabstract ive Ġsummar ization Ġsystems . ĠThen Ġthey Ġare Ġasked Ġto Ġrank Ġthe Ġoutputs Ġaccording Ġto Ġinform at iveness Ġ( does Ġthe Ġsummary Ġcapture Ġthe Ġinformative Ġpart Ġof Ġthe Ġdocument ?), Ġflu ency Ġ( is Ġthe Ġsummary Ġgram matical ?), Ġand Ġsuccinct ness Ġ( does Ġthe Ġsummary Ġexpress Ġthe Ġdocument Ġclearly Ġin Ġa Ġfew Ġwords ?) </s> Results Ġ: :: ĠAutomatic ĠEvaluation </s> The Ġresults Ġon Ġthe ĠCNN DM Ġare Ġsummarized Ġin ĠTable ĠT AB REF 25 . ĠThe Ġfirst Ġand Ġsecond Ġblocks Ġshow Ġresults Ġof Ġprevious Ġextract ive Ġand Ġabstract ive Ġmodels , Ġrespectively . ĠResults Ġof ĠStep Ġare Ġall Ġlisted Ġin Ġthe Ġthird Ġblock . ĠLead 3 Ġis Ġa Ġbaseline Ġwhich Ġsimply Ġtakes Ġthe Ġfirst Ġthree Ġsentences Ġof Ġa Ġdocument Ġas Ġits Ġsummary . ĠB ER TE xt ĠB IB REF 23 Ġis Ġan Ġextract ive Ġmodel Ġfine - tun ing Ġon ĠB ERT ĠB IB REF 2 Ġthat Ġoutper forms Ġother Ġextract ive Ġsystems . ĠPT Gen ĠB IB REF 11 , ĠDRM ĠB IB REF 26 , Ġand ĠDC A ĠB IB REF 42 Ġare Ġsequence - to - sequence Ġlearning Ġbased Ġmodels Ġextended Ġwith Ġcopy Ġand Ġcoverage Ġmechanism , Ġreinforcement Ġlearning , Ġand Ġdeep Ġcommunicating Ġagents Ġindividually . ĠBottom Up ĠB IB REF 29 Ġassisted Ġsummary Ġgeneration Ġwith Ġa Ġword Ġprediction Ġmodel . ĠB ER TA bs ĠB IB REF 23 Ġand ĠUni LM ĠB IB REF 43 Ġare Ġboth Ġpre - training Ġbased Ġseq 2 seq Ġsummar ization Ġmodels . ĠWe Ġalso Ġimplemented Ġthree Ġabstract ive Ġmodels Ġas Ġour Ġbas elines . ĠTrans former - S 2 S Ġis Ġ6 - layer Ġseq 2 seq ĠTrans former ĠB IB REF 9 Ġwith Ġrandom Ġinitialization . ĠWhen Ġwe Ġreplaced Ġthe Ġenc oder Ġof ĠTrans former - S 2 S Ġwith Ġ$\ text { Ro BER Ta } _ \ text { B ASE } $ ĠB IB REF 4 , Ġ$\ text { Ro BER Ta } _ \ text { B ASE } $ - S 2 S Ġoutper forms ĠTrans former - S 2 S Ġby Ġnearly Ġ2 ĠR OU GE , Ġwhich Ġdemonstrates Ġthe Ġeffectiveness Ġof Ġpre - trained Ġmodels . ĠWith Ġeven Ġlarger Ġpre - trained Ġmodel Ġ$\ text { Ro BER Ta } _ \ text { L AR GE } $, Ġ$\ text { Ro BER Ta } $ - S 2 S Ġis Ġcomparable Ġwith Ġthe Ġbest Ġpublished Ġabstract ive Ġmodel ĠUni LM ĠB IB REF 43 . </s> Based Ġon Ġ$\ text { Ro BER Ta } $ - S 2 S Ġ( the Ġsizes Ġof ĠStep Ġand Ġ$\ text { Ro BER Ta } $ - S 2 S Ġare Ġidentical ), Ġwe Ġstudy Ġthe Ġeffect Ġof Ġdifferent Ġpre - training Ġtasks Ġ( see ĠSection ĠSEC REF 9 ). ĠWe Ġfirst Ġpre - train ĠStep Ġon Ġunl abel ed Ġdocuments Ġof ĠCNN DM Ġtraining Ġsplit Ġto Ġget Ġquick Ġfeedback , Ġden oted Ġas ĠStep Ġ( in - domain ). ĠFrom Ġthe Ġtop Ġpart Ġof Ġthe Ġthird Ġblock Ġin ĠTable ĠT AB REF 25 , Ġwe Ġcan Ġsee Ġthat ĠSent ence ĠRe ordering Ġ( SR ), ĠNext ĠSent ence ĠGeneration Ġ( NS G ) Ġand ĠMask ed ĠDocument ĠGeneration Ġ( MD G ) Ġcan Ġall Ġimprove Ġ$\ text { Ro BER Ta } $ - S 2 S Ġsignificantly Ġmeasured Ġby Ġthe ĠR OU GE Ġscript . ĠNote Ġthat Ġaccording Ġto Ġthe ĠR OU GE Ġscript , Ġ$\ pm Ġ0 . 22 $ ĠR OU GE Ġalmost Ġalways Ġmeans Ġa Ġsignificant Ġdifference Ġwith Ġ$ p Ġ< Ġ0 . 05 $. ĠInteresting , Ġeven ĠStep Ġis Ġpre - trained Ġon Ġ230 Ġmillion Ġwords , Ġit Ġoutper forms ĠUni LM Ġthat Ġis Ġpre - trained Ġon Ġ3 , 000 Ġmillion Ġwords ĠB IB REF 43 . ĠWhen Ġwe Ġpre - train ĠStep Ġon Ġeven Ġlarger Ġdataset Ġ( i . e ., ĠG IG A - CM ), Ġthe Ġresults Ġare Ġfurther Ġimproved Ġand ĠStep Ġoutper forms Ġall Ġmodels Ġin Ġcomparison , Ġas Ġlisted Ġin Ġthe Ġbottom Ġpart Ġof ĠTable ĠT AB REF 25 . </s> Table ĠT AB REF 26 Ġpresents Ġresults Ġon ĠNYT Ġdataset . ĠFollowing Ġthe Ġsame Ġevaluation Ġprotocol Ġas ĠB IB REF 36 , Ġwe Ġadopted Ġthe Ġlimited - length Ġrecall Ġbased ĠR OU GE , Ġwhere Ġwe Ġtrunc ated Ġthe Ġpredicted Ġsumm aries Ġto Ġthe Ġlength Ġof Ġthe Ġgold Ġones . ĠAgain , Ġthe Ġfirst Ġand Ġsecond Ġblocks Ġshow Ġresults Ġof Ġprevious Ġextract ive Ġand Ġabstract ive Ġmodels , Ġrespectively . ĠResults Ġof ĠStep Ġare Ġlisted Ġin Ġthe Ġthird Ġblock . ĠSimilar Ġto Ġthe Ġtrends Ġin ĠCNN DM , ĠStep Ġleads Ġsignificant Ġperformance Ġgains Ġ( with Ġ$ p < 0 . 05 $ ) Ġcompared Ġto Ġall Ġother Ġmodels Ġin ĠTable ĠT AB REF 26 . </s> Among Ġall Ġthree Ġpre - training Ġtasks , ĠSR Ġworks Ġslightly Ġbetter Ġthan Ġthe Ġother Ġtwo Ġtasks Ġ( i . e ., ĠNS G Ġand ĠMD G ). ĠWe Ġalso Ġtried Ġto Ġrandomly Ġuse Ġall Ġthe Ġthree Ġtasks Ġduring Ġtraining Ġwith Ġ1 / 3 Ġprobability Ġeach Ġ( ind icated Ġas ĠALL ). ĠInteresting , Ġwe Ġobserved Ġthat , Ġin Ġgeneral , ĠAll Ġoutper forms Ġall Ġthree Ġtasks Ġwhen Ġemploying Ġunl abel ed Ġdocuments Ġof Ġtraining Ġsplits Ġof ĠCNN DM Ġor ĠNYT , Ġwhich Ġmight Ġbe Ġdue Ġto Ġlimited Ġnumber Ġof Ġunl abel ed Ġdocuments Ġof Ġthe Ġtraining Ġsplits . ĠAfter Ġadding Ġmore Ġdata Ġ( i . e ., ĠG IA G - CM ) Ġto Ġpre - training , ĠSR Ġconsistently Ġachieves Ġhighest ĠR OU GE - 2 Ġon Ġboth ĠCNN DM Ġand ĠNYT . ĠWe Ġconclude Ġthat ĠSR Ġis Ġthe Ġmost Ġeffective Ġtask Ġfor Ġpre - training Ġsince Ġsentence Ġre ordering Ġtask Ġrequires Ġcomprehens ively Ġunderstanding Ġa Ġdocument Ġin Ġa Ġwide Ġcoverage , Ġgoing Ġbeyond Ġindividual Ġwords Ġand Ġsentences , Ġwhich Ġis Ġhighly Ġclose Ġto Ġthe Ġess ense Ġof Ġabstract ive Ġdocument Ġsummar ization . </s> Results Ġ: :: ĠHuman ĠEvaluation </s> We Ġalso Ġconducted Ġhuman Ġevaluation Ġwith Ġ20 Ġdocuments Ġrandomly Ġsampled Ġfrom Ġthe Ġtest Ġsplit Ġof ĠCNN DM . ĠWe Ġcompared Ġthe Ġbest Ġpre forming ĠStep Ġmodel Ġ( i . e ., Ġpre - training Ġon Ġthe ĠG IG A - CM Ġdataset Ġusing ĠSR Ġtask ) Ġwith Ġhuman Ġreferences Ġ( den oted Ġas ĠGold ), Ġ$\ text { Ro BER Ta } $ - S 2 S , Ġand Ġtwo Ġpre - training Ġbased Ġmodels , ĠB ER TA bs ĠB IB REF 23 Ġand ĠUni LM ĠB IB REF 43 . ĠParticipants Ġwere Ġasked Ġto Ġrank Ġthe Ġoutputs Ġof Ġthese Ġsystems Ġfrom Ġbest Ġto Ġworst . ĠWe Ġreport Ġthe Ġproportions Ġof Ġsystem Ġrankings Ġand Ġmean Ġrank Ġ( lower Ġis Ġbetter ) Ġin ĠTable ĠT AB REF 29 . ĠThe Ġoutput Ġof ĠStep Ġis Ġselected Ġas Ġthe Ġbest Ġfor Ġthe Ġ25 % Ġof Ġcases Ġand Ġwe Ġobtained Ġlower Ġmean Ġrank Ġthan Ġall Ġsystems Ġexcept Ġfor ĠGold , Ġwhich Ġshows Ġthe Ġparticipants ' Ġpreference Ġfor Ġour Ġmodel . ĠThen Ġwe Ġconverted Ġranking Ġnumbers Ġinto Ġratings Ġ( i . e ., Ġrank Ġ$ i $ Ġis Ġconverted Ġinto Ġ$ 6 - i $ ) Ġand Ġapplied Ġthe Ġstudent Ġ$ t $ - test Ġon Ġthe Ġratings . ĠStep Ġis Ġsignificantly Ġbetter Ġthan Ġall Ġother Ġsystems Ġin Ġcomparison Ġwith Ġ$ p < 0 . 05 $. ĠBut Ġit Ġstill Ġl ags Ġbehind Ġhuman . ĠOne Ġpossible Ġreason Ġis Ġthat ĠStep Ġ( as Ġwell Ġas Ġother Ġsystems ) Ġonly Ġtakes Ġthe Ġfirst Ġ512 Ġtokens Ġof Ġa Ġlong Ġdocument Ġas Ġinput Ġand Ġthus Ġmay Ġlose Ġinformation Ġresiding Ġin Ġthe Ġfollowing Ġtokens . </s> Conclusion </s> We Ġproposed ĠStep , Ġa Ġseq 2 seq Ġtransformer Ġpre - training Ġapproach , Ġfor Ġabstract ive Ġdocument Ġsummar ization . ĠSpecifically , Ġthree Ġpre - training Ġtasks Ġare Ġdesigned , Ġsentence Ġre ordering , Ġnext Ġsentence Ġgeneration , Ġand Ġmasked Ġdocument Ġgeneration . ĠWhen Ġwe Ġonly Ġemploy Ġthe Ġunl abel ed Ġdocuments Ġin Ġthe Ġtraining Ġsplits Ġof Ġsummar ization Ġdatasets Ġto Ġpre - training ĠStep Ġwith Ġour Ġproposed Ġtasks , Ġthe Ġsummar ization Ġmodel Ġbased Ġon Ġthe Ġpre - trained ĠStep Ġoutper forms Ġthe Ġbest Ġpublished Ġabstract ive Ġsystem . ĠInv olving Ġlarge Ġscale Ġdata Ġto Ġpre - training Ġleads Ġto Ġlarger Ġperformance Ġgains . ĠBy Ġusing Ġthe Ġbest Ġperforming Ġpre - training Ġtask , ĠStep Ġachieves Ġ0 . 8 Ġabsolute ĠR OU GE - 2 Ġimprovements Ġon ĠCNN / Daily Mail Ġand Ġ2 . 4 Ġabsolute ĠR OU GE - 2 Ġimprovements Ġon ĠNew ĠYork ĠTimes . ĠIn Ġthe Ġfuture , Ġwe Ġwould Ġlike Ġto Ġinvestigate Ġother Ġtasks Ġto Ġpre - train Ġthe Ġseq 2 seq Ġtransformer Ġmodel . ĠPre - training Ġfor Ġun super vised Ġabstract ive Ġsummar ization Ġis Ġalso Ġan Ġinteresting Ġdirection Ġand Ġworth Ġexploration .