Best results on unigram:
CNN/Daily Mail: Rogue F1 43.85
NYT: Rogue Recall 49.02
XSum: Rogue F1 38.81
ABSTRACTIVE
Highest scores for ROUGE-1, ROUGE-2 and ROUGE-L on CNN/DailyMail test set are 43.85, 20.34 and 39.90 respectively; on the XSum test set 38.81, 16.50 and 31.27 and on the NYT test set 49.02, 31.02 and 45.55
ABSTRACTIVE
We evaluated summarization quality automatically using ROUGE BIBREF32. We report unigram and bigram overlap (ROUGE-1 and ROUGE-2) as a means of assessing informativeness and the longest common subsequence (ROUGE-L) as a means of assessing fluency. Table TABREF23 summarizes our results on the CNN/DailyMail dataset. The first block in the table includes the results of an extractive Oracle system as an upper bound. We also present the Lead-3 baseline (which simply selects the first three sentences in a document). The second block in the table includes various extractive models trained on the CNN/DailyMail dataset (see Section SECREF5 for an overview). For comparison to our own model, we also implemented a non-pretrained Transformer baseline (TransformerExt) which uses the same architecture as BertSumExt, but with fewer parameters. It is randomly initialized and only trained on the summarization task. TransformerExt has 6 layers, the hidden size is 512, and the feed-forward filter size is 2,048. The model was trained with same settings as in BIBREF3. The third block in Table TABREF23 highlights the performance of several abstractive models on the CNN/DailyMail dataset (see Section SECREF6 for an overview). We also include an abstractive Transformer baseline (TransformerAbs) which has the same decoder as our abstractive BertSum models; the encoder is a 6-layer Transformer with 768 hidden size and 2,048 feed-forward filter size. The fourth block reports results with fine-tuned Bert models: BertSumExt and its two variants (one without interval embeddings, and one with the large version of Bert), BertSumAbs, and BertSumExtAbs. Bert-based models outperform the Lead-3 baseline which is not a strawman; on the CNN/DailyMail corpus it is indeed superior to several extractive BIBREF7, BIBREF8, BIBREF19 and abstractive models BIBREF6. Bert models collectively outperform all previously proposed extractive and abstractive systems, only falling behind the Oracle upper bound. Among Bert variants, BertSumExt performs best which is not entirely surprising; CNN/DailyMail summaries are somewhat extractive and even abstractive models are prone to copying sentences from the source document when trained on this dataset BIBREF6. Perhaps unsurprisingly we observe that larger versions of Bert lead to performance improvements and that interval embeddings bring only slight gains. Table TABREF24 presents results on the NYT dataset. Following the evaluation protocol in BIBREF27, we use limited-length ROUGE Recall, where predicted summaries are truncated to the length of the gold summaries. Again, we report the performance of the Oracle upper bound and Lead-3 baseline. The second block in the table contains previously proposed extractive models as well as our own Transformer baseline. Compress BIBREF27 is an ILP-based model which combines compression and anaphoricity constraints. The third block includes abstractive models from the literature, and our Transformer baseline. Bert-based models are shown in the fourth block. Again, we observe that they outperform previously proposed approaches. On this dataset, abstractive Bert models generally perform better compared to BertSumExt, almost approaching Oracle performance. Table TABREF26 summarizes our results on the XSum dataset. Recall that summaries in this dataset are highly abstractive (see Table TABREF12) consisting of a single sentence conveying the gist of the document. Extractive models here perform poorly as corroborated by the low performance of the Lead baseline (which simply selects the leading sentence from the document), and the Oracle (which selects a single-best sentence in each document) in Table TABREF26. As a result, we do not report results for extractive models on this dataset. The second block in Table TABREF26 presents the results of various abstractive models taken from BIBREF22 and also includes our own abstractive Transformer baseline. In the third block we show the results of our Bert summarizers which again are superior to all previously reported models (by a wide margin). FLOAT SELECTED: Table 2: ROUGE F1 results on CNN/DailyMail test set (R1 and R2 are shorthands for unigram and bigram overlap; RL is the longest common subsequence). Results for comparison systems are taken from the authors’ respective papers or obtained on our data by running publicly released software. FLOAT SELECTED: Table 4: ROUGE F1 results on the XSum test set. Results for comparison systems are taken from the authors’ respective papers or obtained on our data by running publicly released software. FLOAT SELECTED: Table 3: ROUGE Recall results on NYT test set. Results for comparison systems are taken from the authors’ respective papers or obtained on our data by running publicly released software. Table cells are filled with — whenever results are not available.
FLOAT SELECTED: Table 2: ROUGE F1 results on CNN/DailyMail test set (R1 and R2 are shorthands for unigram and bigram overlap; RL is the longest common subsequence). Results for comparison systems are taken from the authors’ respective papers or obtained on our data by running publicly released software. FLOAT SELECTED: Table 4: ROUGE F1 results on the XSum test set. Results for comparison systems are taken from the authors’ respective papers or obtained on our data by running publicly released software.
1908.08345
Introduction </s> Language Ġmodel Ġpret raining Ġhas Ġadvanced Ġthe Ġstate Ġof Ġthe Ġart Ġin Ġmany ĠN LP Ġtasks Ġranging Ġfrom Ġsentiment Ġanalysis , Ġto Ġquestion Ġanswering , Ġnatural Ġlanguage Ġinference , Ġnamed Ġentity Ġrecognition , Ġand Ġtextual Ġsimilarity . ĠState - of - the - art Ġpret rained Ġmodels Ġinclude ĠEL Mo ĠB IB REF 1 , ĠG PT ĠB IB REF 2 , Ġand Ġmore Ġrecently ĠBid irection al ĠEnc oder ĠRepresent ations Ġfrom ĠTransformers Ġ( B ert ; ĠB IB REF 0 ). ĠBert Ġcombines Ġboth Ġword Ġand Ġsentence Ġrepresentations Ġin Ġa Ġsingle Ġvery Ġlarge ĠTrans former ĠB IB REF 3 ; Ġit Ġis Ġpret rained Ġon Ġvast Ġamounts Ġof Ġtext , Ġwith Ġan Ġun super vised Ġobjective Ġof Ġmasked Ġlanguage Ġmodeling Ġand Ġnext - sent ence Ġprediction Ġand Ġcan Ġbe Ġfine - tun ed Ġwith Ġvarious Ġtask - specific Ġobjectives . </s> In Ġmost Ġcases , Ġpret rained Ġlanguage Ġmodels Ġhave Ġbeen Ġemployed Ġas Ġenc od ers Ġfor Ġsentence - Ġand Ġparagraph - level Ġnatural Ġlanguage Ġunderstanding Ġproblems ĠB IB REF 0 Ġinvolving Ġvarious Ġclassification Ġtasks Ġ( e . g ., Ġpredicting Ġwhether Ġany Ġtwo Ġsentences Ġare Ġin Ġan Ġentail ment Ġrelationship ; Ġor Ġdetermining Ġthe Ġcompletion Ġof Ġa Ġsentence Ġamong Ġfour Ġalternative Ġsentences ). ĠIn Ġthis Ġpaper , Ġwe Ġexamine Ġthe Ġinfluence Ġof Ġlanguage Ġmodel Ġpret raining Ġon Ġtext Ġsummar ization . ĠDifferent Ġfrom Ġprevious Ġtasks , Ġsummar ization Ġrequires Ġwide - co verage Ġnatural Ġlanguage Ġunderstanding Ġgoing Ġbeyond Ġthe Ġmeaning Ġof Ġindividual Ġwords Ġand Ġsentences . ĠThe Ġaim Ġis Ġto Ġcond ense Ġa Ġdocument Ġinto Ġa Ġshorter Ġversion Ġwhile Ġpreserving Ġmost Ġof Ġits Ġmeaning . ĠFurthermore , Ġunder Ġabstract ive Ġmodeling Ġformulations , Ġthe Ġtask Ġrequires Ġlanguage Ġgeneration Ġcapabilities Ġin Ġorder Ġto Ġcreate Ġsumm aries Ġcontaining Ġnovel Ġwords Ġand Ġphrases Ġnot Ġfeatured Ġin Ġthe Ġsource Ġtext , Ġwhile Ġextract ive Ġsummar ization Ġis Ġoften Ġdefined Ġas Ġa Ġbinary Ġclassification Ġtask Ġwith Ġlabels Ġindicating Ġwhether Ġa Ġtext Ġspan Ġ( typically Ġa Ġsentence ) Ġshould Ġbe Ġincluded Ġin Ġthe Ġsummary . </s> We Ġexplore Ġthe Ġpotential Ġof ĠBert Ġfor Ġtext Ġsummar ization Ġunder Ġa Ġgeneral Ġframework Ġencomp assing Ġboth Ġextract ive Ġand Ġabstract ive Ġmodeling Ġparad ig ms . ĠWe Ġpropose Ġa Ġnovel Ġdocument - level Ġenc oder Ġbased Ġon ĠBert Ġwhich Ġis Ġable Ġto Ġencode Ġa Ġdocument Ġand Ġobtain Ġrepresentations Ġfor Ġits Ġsentences . ĠOur Ġextract ive Ġmodel Ġis Ġbuilt Ġon Ġtop Ġof Ġthis Ġenc oder Ġby Ġstacking Ġseveral Ġinter - sent ence ĠTrans former Ġlayers Ġto Ġcapture Ġdocument - level Ġfeatures Ġfor Ġextracting Ġsentences . ĠOur Ġabstract ive Ġmodel Ġadop ts Ġan Ġenc oder - dec oder Ġarchitecture , Ġcombining Ġthe Ġsame Ġpret rained ĠBert Ġenc oder Ġwith Ġa Ġrandomly - initialized ĠTrans former Ġdec oder ĠB IB REF 3 . ĠWe Ġdesign Ġa Ġnew Ġtraining Ġschedule Ġwhich Ġseparates Ġthe Ġoptim izers Ġof Ġthe Ġenc oder Ġand Ġthe Ġdec oder Ġin Ġorder Ġto Ġaccommodate Ġthe Ġfact Ġthat Ġthe Ġformer Ġis Ġpret rained Ġwhile Ġthe Ġlatter Ġmust Ġbe Ġtrained Ġfrom Ġscratch . ĠFinally , Ġmotivated Ġby Ġprevious Ġwork Ġshowing Ġthat Ġthe Ġcombination Ġof Ġextract ive Ġand Ġabstract ive Ġobjectives Ġcan Ġhelp Ġgenerate Ġbetter Ġsumm aries ĠB IB REF 4 , Ġwe Ġpresent Ġa Ġtwo - stage Ġapproach Ġwhere Ġthe Ġenc oder Ġis Ġfine - tun ed Ġtwice , Ġfirst Ġwith Ġan Ġextract ive Ġobjective Ġand Ġsubsequently Ġon Ġthe Ġabstract ive Ġsummar ization Ġtask . </s> We Ġevaluate Ġthe Ġproposed Ġapproach Ġon Ġthree Ġsingle - document Ġnews Ġsummar ization Ġdatasets Ġrepresentative Ġof Ġdifferent Ġwriting Ġconventions Ġ( e . g ., Ġimportant Ġinformation Ġis Ġconcentrated Ġat Ġthe Ġbeginning Ġof Ġthe Ġdocument Ġor Ġdistributed Ġmore Ġevenly Ġthroughout ) Ġand Ġsummary Ġstyles Ġ( e . g ., Ġverb ose Ġvs . Ġmore Ġte legraph ic ; Ġextract ive Ġvs . Ġabstract ive ). ĠAcross Ġdatasets , Ġwe Ġexperiment ally Ġshow Ġthat Ġthe Ġproposed Ġmodels Ġachieve Ġstate - of - the - art Ġresults Ġunder Ġboth Ġextract ive Ġand Ġabstract ive Ġsettings . ĠOur Ġcontributions Ġin Ġthis Ġwork Ġare Ġthree - fold : Ġa ) Ġwe Ġhighlight Ġthe Ġimportance Ġof Ġdocument Ġencoding Ġfor Ġthe Ġsummar ization Ġtask ; Ġa Ġvariety Ġof Ġrecently Ġproposed Ġtechniques Ġaim Ġto Ġenhance Ġsummar ization Ġperformance Ġvia Ġcopying Ġmechanisms ĠB IB REF 5 , ĠB IB REF 6 , ĠB IB REF 7 , Ġreinforcement Ġlearning ĠB IB REF 8 , ĠB IB REF 9 , ĠB IB REF 10 , Ġand Ġmultiple Ġcommunicating Ġenc od ers ĠB IB REF 11 . ĠWe Ġachieve Ġbetter Ġresults Ġwith Ġa Ġminimum - requ irement Ġmodel Ġwithout Ġusing Ġany Ġof Ġthese Ġmechanisms ; Ġb ) Ġwe Ġshowcase Ġways Ġto Ġeffectively Ġemploy Ġpret rained Ġlanguage Ġmodels Ġin Ġsummar ization Ġunder Ġboth Ġextract ive Ġand Ġabstract ive Ġsettings ; Ġwe Ġwould Ġexpect Ġany Ġimprovements Ġin Ġmodel Ġpret raining Ġto Ġtranslate Ġin Ġbetter Ġsummar ization Ġin Ġthe Ġfuture ; Ġand Ġc ) Ġthe Ġproposed Ġmodels Ġcan Ġbe Ġused Ġas Ġa Ġstepping Ġstone Ġto Ġfurther Ġimprove Ġsummar ization Ġperformance Ġas Ġwell Ġas Ġbas elines Ġagainst Ġwhich Ġnew Ġproposals Ġare Ġtested . </s> Background Ġ: :: ĠPret rained ĠLanguage ĠModels </s> P ret rained Ġlanguage Ġmodels ĠB IB REF 1 , ĠB IB REF 2 , ĠB IB REF 0 , ĠB IB REF 12 , ĠB IB REF 13 Ġhave Ġrecently Ġemerged Ġas Ġa Ġkey Ġtechnology Ġfor Ġachieving Ġimpressive Ġgains Ġin Ġa Ġwide Ġvariety Ġof Ġnatural Ġlanguage Ġtasks . ĠThese Ġmodels Ġextend Ġthe Ġidea Ġof Ġword Ġembed d ings Ġby Ġlearning Ġcontextual Ġrepresentations Ġfrom Ġlarge - scale Ġcorpor a Ġusing Ġa Ġlanguage Ġmodeling Ġobjective . ĠBid irection al ĠEnc oder ĠRepresent ations Ġfrom ĠTransformers Ġ( B ert ; ĠB IB REF 0 ) Ġis Ġa Ġnew Ġlanguage Ġrepresentation Ġmodel Ġwhich Ġis Ġtrained Ġwith Ġa Ġmasked Ġlanguage Ġmodeling Ġand Ġa ĠâĢ ľ next Ġsentence Ġprediction âĢ Ŀ Ġtask Ġon Ġa Ġcorpus Ġof Ġ3 , 300 M Ġwords . </s> The Ġgeneral Ġarchitecture Ġof ĠBert Ġis Ġshown Ġin Ġthe Ġleft Ġpart Ġof ĠFigure ĠFIG REF 2 . ĠInput Ġtext Ġis Ġfirst Ġpre process ed Ġby Ġinserting Ġtwo Ġspecial Ġtokens . Ġ[ cl s ] Ġis Ġapp ended Ġto Ġthe Ġbeginning Ġof Ġthe Ġtext ; Ġthe Ġoutput Ġrepresentation Ġof Ġthis Ġtoken Ġis Ġused Ġto Ġaggregate Ġinformation Ġfrom Ġthe Ġwhole Ġsequence Ġ( e . g ., Ġfor Ġclassification Ġtasks ). ĠAnd Ġtoken Ġ[ se p ] Ġis Ġinserted Ġafter Ġeach Ġsentence Ġas Ġan Ġindicator Ġof Ġsentence Ġboundaries . ĠThe Ġmodified Ġtext Ġis Ġthen Ġrepresented Ġas Ġa Ġsequence Ġof Ġtokens Ġ$ X =[ w _ 1 , w _ 2 , \ cd ots Ġ, w _ n ] $. ĠEach Ġtoken Ġ$ w _ i $ Ġis Ġassigned Ġthree Ġkinds Ġof Ġembed d ings : Ġtoken Ġembed d ings Ġindicate Ġthe Ġmeaning Ġof Ġeach Ġtoken , Ġsegment ation Ġembed d ings Ġare Ġused Ġto Ġdiscriminate Ġbetween Ġtwo Ġsentences Ġ( e . g ., Ġduring Ġa Ġsentence - pair Ġclassification Ġtask ) Ġand Ġposition Ġembed d ings Ġindicate Ġthe Ġposition Ġof Ġeach Ġtoken Ġwithin Ġthe Ġtext Ġsequence . ĠThese Ġthree Ġembed d ings Ġare Ġsummed Ġto Ġa Ġsingle Ġinput Ġvector Ġ$ x _ i $ Ġand Ġfed Ġto Ġa Ġbid irection al ĠTrans former Ġwith Ġmultiple Ġlayers : </s> where Ġ$ h ^ 0 = x $ Ġare Ġthe Ġinput Ġvectors ; Ġ$\ math rm Ġ{ L N } $ Ġis Ġthe Ġlayer Ġnormal ization Ġoperation ĠB IB REF 14 ; Ġ$\ math rm Ġ{ M HA tt } $ Ġis Ġthe Ġmulti - head Ġattention Ġoperation ĠB IB REF 3 ; Ġsupers cript Ġ$ l $ Ġindicates Ġthe Ġdepth Ġof Ġthe Ġstacked Ġlayer . ĠOn Ġthe Ġtop Ġlayer , ĠBert Ġwill Ġgenerate Ġan Ġoutput Ġvector Ġ$ t _ i $ Ġfor Ġeach Ġtoken Ġwith Ġrich Ġcontextual Ġinformation . </s> P ret rained Ġlanguage Ġmodels Ġare Ġusually Ġused Ġto Ġenhance Ġperformance Ġin Ġlanguage Ġunderstanding Ġtasks . ĠVery Ġrecently , Ġthere Ġhave Ġbeen Ġattempts Ġto Ġapply Ġpret rained Ġmodels Ġto Ġvarious Ġgeneration Ġproblems ĠB IB REF 15 , ĠB IB REF 16 . ĠWhen Ġfine - tun ing Ġfor Ġa Ġspecific Ġtask , Ġunlike ĠEL Mo Ġwhose Ġparameters Ġare Ġusually Ġfixed , Ġparameters Ġin ĠBert Ġare Ġjointly Ġfine - tun ed Ġwith Ġadditional Ġtask - specific Ġparameters . </s> Background Ġ: :: ĠExtract ive ĠSum mar ization </s> Ext ractive Ġsummar ization Ġsystems Ġcreate Ġa Ġsummary Ġby Ġidentifying Ġ( and Ġsubsequently Ġconc aten ating ) Ġthe Ġmost Ġimportant Ġsentences Ġin Ġa Ġdocument . ĠNeural Ġmodels Ġconsider Ġextract ive Ġsummar ization Ġas Ġa Ġsentence Ġclassification Ġproblem : Ġa Ġneural Ġenc oder Ġcreates Ġsentence Ġrepresentations Ġand Ġa Ġclass ifier Ġpredicts Ġwhich Ġsentences Ġshould Ġbe Ġselected Ġas Ġsumm aries . ĠSum ma Ru NN er ĠB IB REF 7 Ġis Ġone Ġof Ġthe Ġearliest Ġneural Ġapproaches Ġadopting Ġan Ġenc oder Ġbased Ġon ĠRec urrent ĠNeural ĠNetworks . ĠRefresh ĠB IB REF 8 Ġis Ġa Ġreinforcement Ġlearning - based Ġsystem Ġtrained Ġby Ġglobally Ġoptimizing Ġthe ĠR OU GE Ġmetric . ĠMore Ġrecent Ġwork Ġachieves Ġhigher Ġperformance Ġwith Ġmore Ġsophisticated Ġmodel Ġstructures . ĠLat ent ĠB IB REF 17 Ġframes Ġextract ive Ġsummar ization Ġas Ġa Ġlatent Ġvariable Ġinference Ġproblem ; Ġinstead Ġof Ġmaximizing Ġthe Ġlikelihood Ġof ĠâĢ ľ gold âĢ Ŀ Ġstandard Ġlabels , Ġtheir Ġlatent Ġmodel Ġdirectly Ġmaxim izes Ġthe Ġlikelihood Ġof Ġhuman Ġsumm aries Ġgiven Ġselected Ġsentences . ĠSum o ĠB IB REF 18 Ġcapital izes Ġon Ġthe Ġnotion Ġof Ġstructured Ġattention Ġto Ġinduce Ġa Ġmulti - root Ġdependency Ġtree Ġrepresentation Ġof Ġthe Ġdocument Ġwhile Ġpredicting Ġthe Ġoutput Ġsummary . ĠNe u Sum ĠB IB REF 19 Ġscores Ġand Ġselects Ġsentences Ġjointly Ġand Ġrepresents Ġthe Ġstate Ġof Ġthe Ġart Ġin Ġextract ive Ġsummar ization . </s> Background Ġ: :: ĠAbstract ive ĠSum mar ization </s> Ne ural Ġapproaches Ġto Ġabstract ive Ġsummar ization Ġconceptual ize Ġthe Ġtask Ġas Ġa Ġsequence - to - sequence Ġproblem , Ġwhere Ġan Ġenc oder Ġmaps Ġa Ġsequence Ġof Ġtokens Ġin Ġthe Ġsource Ġdocument Ġ$\ math bf Ġ{ x } Ġ= Ġ[ x _ 1 , Ġ... , Ġx _ n ] $ Ġto Ġa Ġsequence Ġof Ġcontinuous Ġrepresentations Ġ$\ math bf Ġ{ z } Ġ= Ġ[ z _ 1 , Ġ... , Ġz _ n ] $, Ġand Ġa Ġdec oder Ġthen Ġgenerates Ġthe Ġtarget Ġsummary Ġ$\ math bf Ġ{ y } Ġ= Ġ[ y _ 1 , Ġ... , Ġy _ m ] $ Ġtoken - by - token , Ġin Ġan Ġauto - reg ressive Ġmanner , Ġhence Ġmodeling Ġthe Ġconditional Ġprobability : Ġ$ p ( y _ 1 , Ġ... , Ġy _ m | x _ 1 , Ġ... , Ġx _ n ) $. </s> BI B REF 20 Ġand ĠB IB REF 21 Ġwere Ġamong Ġthe Ġfirst Ġto Ġapply Ġthe Ġneural Ġenc oder - dec oder Ġarchitecture Ġto Ġtext Ġsummar ization . ĠB IB REF 6 Ġenhance Ġthis Ġmodel Ġwith Ġa Ġpointer - gener ator Ġnetwork Ġ( PT gen ) Ġwhich Ġallows Ġit Ġto Ġcopy Ġwords Ġfrom Ġthe Ġsource Ġtext , Ġand Ġa Ġcoverage Ġmechanism Ġ( C ov ) Ġwhich Ġkeeps Ġtrack Ġof Ġwords Ġthat Ġhave Ġbeen Ġsummarized . ĠB IB REF 11 Ġpropose Ġan Ġabstract ive Ġsystem Ġwhere Ġmultiple Ġagents Ġ( enc od ers ) Ġrepresent Ġthe Ġdocument Ġtogether Ġwith Ġa Ġhierarchical Ġattention Ġmechanism Ġ( over Ġthe Ġagents ) Ġfor Ġdecoding . ĠTheir ĠDeep ĠCommun icating ĠAgents Ġ( D CA ) Ġmodel Ġis Ġtrained Ġend - to - end Ġwith Ġreinforcement Ġlearning . ĠB IB REF 9 Ġalso Ġpresent Ġa Ġdeep Ġreinforced Ġmodel Ġ( DR M ) Ġfor Ġabstract ive Ġsummar ization Ġwhich Ġhandles Ġthe Ġcoverage Ġproblem Ġwith Ġan Ġintra - att ention Ġmechanism Ġwhere Ġthe Ġdec oder Ġattends Ġover Ġpreviously Ġgenerated Ġwords . ĠB IB REF 4 Ġfollow Ġa Ġbottom - up Ġapproach Ġ( Bottom Up ); Ġa Ġcontent Ġselector Ġfirst Ġdetermines Ġwhich Ġphrases Ġin Ġthe Ġsource Ġdocument Ġshould Ġbe Ġpart Ġof Ġthe Ġsummary , Ġand Ġa Ġcopy Ġmechanism Ġis Ġapplied Ġonly Ġto Ġpre selected Ġphrases Ġduring Ġdecoding . ĠB IB REF 22 Ġpropose Ġan Ġabstract ive Ġmodel Ġwhich Ġis Ġparticularly Ġsuited Ġto Ġextreme Ġsummar ization Ġ( i . e ., Ġsingle Ġsentence Ġsumm aries ), Ġbased Ġon Ġconv olution al Ġneural Ġnetworks Ġand Ġadditionally Ġconditioned Ġon Ġtopic Ġdistributions Ġ( T Con v S 2 S ). </s> Fine - tun ing ĠBert Ġfor ĠSum mar ization Ġ: :: ĠSum mar ization ĠEnc oder </s> Although ĠBert Ġhas Ġbeen Ġused Ġto Ġfine - t une Ġvarious ĠN LP Ġtasks , Ġits Ġapplication Ġto Ġsummar ization Ġis Ġnot Ġas Ġstraightforward . ĠSince ĠBert Ġis Ġtrained Ġas Ġa Ġmasked - language Ġmodel , Ġthe Ġoutput Ġvectors Ġare Ġgrounded Ġto Ġtokens Ġinstead Ġof Ġsentences , Ġwhile Ġin Ġextract ive Ġsummar ization , Ġmost Ġmodels Ġmanipulate Ġsentence - level Ġrepresentations . ĠAlthough Ġsegment ation Ġembed d ings Ġrepresent Ġdifferent Ġsentences Ġin ĠBert , Ġthey Ġonly Ġapply Ġto Ġsentence - pair Ġinputs , Ġwhile Ġin Ġsummar ization Ġwe Ġmust Ġencode Ġand Ġmanipulate Ġmulti - sent ential Ġinputs . ĠFigure ĠFIG REF 2 Ġillustrates Ġour Ġproposed ĠBert Ġarchitecture Ġfor ĠSum mar ization Ġ( which Ġwe Ġcall ĠBert Sum ). </s> In Ġorder Ġto Ġrepresent Ġindividual Ġsentences , Ġwe Ġinsert Ġexternal Ġ[ cl s ] Ġtokens Ġat Ġthe Ġstart Ġof Ġeach Ġsentence , Ġand Ġeach Ġ[ cl s ] Ġsymbol Ġcollects Ġfeatures Ġfor Ġthe Ġsentence Ġpreceding Ġit . ĠWe Ġalso Ġuse Ġinterval Ġsegment Ġembed d ings Ġto Ġdistinguish Ġmultiple Ġsentences Ġwithin Ġa Ġdocument . ĠFor Ġ$ sent _ i $ Ġwe Ġassign Ġsegment Ġembed ding Ġ$ E _ A $ Ġor Ġ$ E _ B $ Ġdepending Ġon Ġwhether Ġ$ i $ Ġis Ġodd Ġor Ġeven . ĠFor Ġexample , Ġfor Ġdocument Ġ$ [ sent _ 1 , Ġsent _ 2 , Ġsent _ 3 , Ġsent _ 4 , Ġsent _ 5 ] $, Ġwe Ġwould Ġassign Ġembed d ings Ġ$ [ E _ A , ĠE _ B , ĠE _ A , E _ B , ĠE _ A ] $. ĠThis Ġway , Ġdocument Ġrepresentations Ġare Ġlearned Ġhierarch ically Ġwhere Ġlower ĠTrans former Ġlayers Ġrepresent Ġadjacent Ġsentences , Ġwhile Ġhigher Ġlayers , Ġin Ġcombination Ġwith Ġself - att ention , Ġrepresent Ġmulti - sent ence Ġdiscourse . </s> Position Ġembed d ings Ġin Ġthe Ġoriginal ĠBert Ġmodel Ġhave Ġa Ġmaximum Ġlength Ġof Ġ512 ; Ġwe Ġovercome Ġthis Ġlimitation Ġby Ġadding Ġmore Ġposition Ġembed d ings Ġthat Ġare Ġinitialized Ġrandomly Ġand Ġfine - tun ed Ġwith Ġother Ġparameters Ġin Ġthe Ġenc oder . </s> Fine - tun ing ĠBert Ġfor ĠSum mar ization Ġ: :: ĠExtract ive ĠSum mar ization </s> Let Ġ$ d $ Ġdenote Ġa Ġdocument Ġcontaining Ġsentences Ġ$ [ sent _ 1 , Ġsent _ 2 , Ġ\ cd ots Ġ, Ġsent _ m ] $, Ġwhere Ġ$ sent _ i $ Ġis Ġthe Ġ$ i $ - th Ġsentence Ġin Ġthe Ġdocument . ĠExtract ive Ġsummar ization Ġcan Ġbe Ġdefined Ġas Ġthe Ġtask Ġof Ġassigning Ġa Ġlabel Ġ$ y _ i Ġ\ in Ġ\ l brace Ġ0 , Ġ1 \ r brace Ġ$ Ġto Ġeach Ġ$ sent _ i $, Ġindicating Ġwhether Ġthe Ġsentence Ġshould Ġbe Ġincluded Ġin Ġthe Ġsummary . ĠIt Ġis Ġassumed Ġthat Ġsummary Ġsentences Ġrepresent Ġthe Ġmost Ġimportant Ġcontent Ġof Ġthe Ġdocument . </s> With ĠBert Sum , Ġvector Ġ$ t _ i $ Ġwhich Ġis Ġthe Ġvector Ġof Ġthe Ġ$ i $ - th Ġ[ cl s ] Ġsymbol Ġfrom Ġthe Ġtop Ġlayer Ġcan Ġbe Ġused Ġas Ġthe Ġrepresentation Ġfor Ġ$ sent _ i $. ĠSeveral Ġinter - sent ence ĠTrans former Ġlayers Ġare Ġthen Ġstacked Ġon Ġtop Ġof ĠBert Ġoutputs , Ġto Ġcapture Ġdocument - level Ġfeatures Ġfor Ġextracting Ġsumm aries : </s> where Ġ$ h ^ 0 = \ math rm Ġ{ Pos Emb } ( T ) $ ; Ġ$ T $ Ġdenotes Ġthe Ġsentence Ġvectors Ġoutput Ġby ĠBert Sum , Ġand Ġfunction Ġ$\ math rm Ġ{ Pos Emb } $ Ġadds Ġsin us oid Ġpositional Ġembed d ings ĠB IB REF 3 Ġto Ġ$ T $, Ġindicating Ġthe Ġposition Ġof Ġeach Ġsentence . </s> The Ġfinal Ġoutput Ġlayer Ġis Ġa Ġs igm oid Ġclass ifier : </s> where Ġ$ h ^ L _ i $ Ġis Ġthe Ġvector Ġfor Ġ$ sent _ i $ Ġfrom Ġthe Ġtop Ġlayer Ġ( the Ġ$ L $ - th Ġlayer Ġ) Ġof Ġthe ĠTrans former . ĠIn Ġexperiments , Ġwe Ġimplemented ĠTransformers Ġwith Ġ$ L = 1 , Ġ2 , Ġ3 $ Ġand Ġfound Ġthat Ġa ĠTrans former Ġwith Ġ$ L = 2 $ Ġperformed Ġbest . ĠWe Ġname Ġthis Ġmodel ĠBert Sum Ext . </s> The Ġloss Ġof Ġthe Ġmodel Ġis Ġthe Ġbinary Ġclassification Ġentropy Ġof Ġprediction Ġ$\ hat { y } _ i $ Ġagainst Ġgold Ġlabel Ġ$ y _ i $. ĠInter - sent ence ĠTrans former Ġlayers Ġare Ġjointly Ġfine - tun ed Ġwith ĠBert Sum . ĠWe Ġuse Ġthe ĠAdam Ġoptim izer Ġwith Ġ$\ beta Ġ_ 1 = 0 . 9 $, Ġand Ġ$\ beta Ġ_ 2 = 0 . 999 $ ). ĠOur Ġlearning Ġrate Ġschedule Ġfollows ĠB IB REF 3 Ġwith Ġwarming - up Ġ($ Ġ\ oper at orn ame {\ oper at orn ame { warm up }} = 10 , 000 $ ): </s> Fine - tun ing ĠBert Ġfor ĠSum mar ization Ġ: :: ĠAbstract ive ĠSum mar ization </s> We Ġuse Ġa Ġstandard Ġenc oder - dec oder Ġframework Ġfor Ġabstract ive Ġsummar ization ĠB IB REF 6 . ĠThe Ġenc oder Ġis Ġthe Ġpret rained ĠBert Sum Ġand Ġthe Ġdec oder Ġis Ġa Ġ6 - lay ered ĠTrans former Ġinitialized Ġrandomly . ĠIt Ġis Ġconceivable Ġthat Ġthere Ġis Ġa Ġmismatch Ġbetween Ġthe Ġenc oder Ġand Ġthe Ġdec oder , Ġsince Ġthe Ġformer Ġis Ġpret rained Ġwhile Ġthe Ġlatter Ġmust Ġbe Ġtrained Ġfrom Ġscratch . ĠThis Ġcan Ġmake Ġfine - tun ing Ġunstable ; Ġfor Ġexample , Ġthe Ġenc oder Ġmight Ġover fit Ġthe Ġdata Ġwhile Ġthe Ġdec oder Ġunder fits , Ġor Ġvice Ġversa . ĠTo Ġcircumvent Ġthis , Ġwe Ġdesign Ġa Ġnew Ġfine - tun ing Ġschedule Ġwhich Ġseparates Ġthe Ġoptim izers Ġof Ġthe Ġenc oder Ġand Ġthe Ġdec oder . </s> We Ġuse Ġtwo ĠAdam Ġoptim izers Ġwith Ġ$\ beta Ġ_ 1 = 0 . 9 $ Ġand Ġ$\ beta Ġ_ 2 = 0 . 999 $ Ġfor Ġthe Ġenc oder Ġand Ġthe Ġdec oder , Ġrespectively , Ġeach Ġwith Ġdifferent Ġwarm up - steps Ġand Ġlearning Ġrates : </s> where Ġ$\ t ilde { lr } _{ \ math cal Ġ{ E }} = 2 e ^{ - 3 } $, Ġand Ġ$\ oper at orn ame {\ oper at orn ame { warm up }} _{ \ math cal Ġ{ E }} = 20 , 000 $ Ġfor Ġthe Ġenc oder Ġand Ġ$\ t ilde { lr } _{ \ math cal Ġ{ D }} = 0 . 1 $, Ġand Ġ$\ oper at orn ame {\ oper at orn ame { warm up }} _{ \ math cal Ġ{ D }} = 10 , 000 $ Ġfor Ġthe Ġdec oder . ĠThis Ġis Ġbased Ġon Ġthe Ġassumption Ġthat Ġthe Ġpret rained Ġenc oder Ġshould Ġbe Ġfine - tun ed Ġwith Ġa Ġsmaller Ġlearning Ġrate Ġand Ġsmoother Ġdecay Ġ( so Ġthat Ġthe Ġenc oder Ġcan Ġbe Ġtrained Ġwith Ġmore Ġaccurate Ġgrad ients Ġwhen Ġthe Ġdec oder Ġis Ġbecoming Ġstable ). </s> In Ġaddition , Ġwe Ġpropose Ġa Ġtwo - stage Ġfine - tun ing Ġapproach , Ġwhere Ġwe Ġfirst Ġfine - t une Ġthe Ġenc oder Ġon Ġthe Ġextract ive Ġsummar ization Ġtask Ġ( Section ĠSEC REF 8 ) Ġand Ġthen Ġfine - t une Ġit Ġon Ġthe Ġabstract ive Ġsummar ization Ġtask Ġ( Section ĠSEC REF 13 ). ĠPrevious Ġwork ĠB IB REF 4 , ĠB IB REF 23 Ġsuggests Ġthat Ġusing Ġextract ive Ġobjectives Ġcan Ġboost Ġthe Ġperformance Ġof Ġabstract ive Ġsummar ization . ĠAlso Ġnotice Ġthat Ġthis Ġtwo - stage Ġapproach Ġis Ġconcept ually Ġvery Ġsimple , Ġthe Ġmodel Ġcan Ġtake Ġadvantage Ġof Ġinformation Ġshared Ġbetween Ġthese Ġtwo Ġtasks , Ġwithout Ġfundamentally Ġchanging Ġits Ġarchitecture . ĠWe Ġname Ġthe Ġdefault Ġabstract ive Ġmodel ĠBert Sum Abs Ġand Ġthe Ġtwo - stage Ġfine - tun ed Ġmodel ĠBert Sum Ext Abs . </s> Exper imental ĠSetup </s> In Ġthis Ġsection , Ġwe Ġdescribe Ġthe Ġsummar ization Ġdatasets Ġused Ġin Ġour Ġexperiments Ġand Ġdiscuss Ġvarious Ġimplementation Ġdetails . </s> Exper imental ĠSetup Ġ: :: ĠSum mar ization ĠDat as ets </s> We Ġevaluated Ġour Ġmodel Ġon Ġthree Ġbenchmark Ġdatasets , Ġnamely Ġthe ĠCNN / Daily Mail Ġnews Ġhighlights Ġdataset ĠB IB REF 24 , Ġthe ĠNew ĠYork ĠTimes ĠAn not ated ĠCorpus Ġ( NY T ; ĠB IB REF 25 ), Ġand ĠX Sum ĠB IB REF 22 . ĠThese Ġdatasets Ġrepresent Ġdifferent Ġsummary Ġstyles Ġranging Ġfrom Ġhighlights Ġto Ġvery Ġbrief Ġone Ġsentence Ġsumm aries . ĠThe Ġsumm aries Ġalso Ġvary Ġwith Ġrespect Ġto Ġthe Ġtype Ġof Ġrewriting Ġoperations Ġthey Ġexempl ify Ġ( e . g ., Ġsome Ġshowcase Ġmore Ġcut Ġand Ġpaste Ġoperations Ġwhile Ġothers Ġare Ġgenuinely Ġabstract ive ). ĠTable ĠT AB REF 12 Ġpresents Ġstatistics Ġon Ġthese Ġdatasets Ġ( test Ġset ); Ġexample Ġ( gold - standard ) Ġsumm aries Ġare Ġprovided Ġin Ġthe Ġsupplementary Ġmaterial . </s> Exper imental ĠSetup Ġ: :: ĠSum mar ization ĠDat as ets Ġ: :: ĠCNN / Daily Mail </s> cont ains Ġnews Ġarticles Ġand Ġassociated Ġhighlights , Ġi . e ., Ġa Ġfew Ġbullet Ġpoints Ġgiving Ġa Ġbrief Ġoverview Ġof Ġthe Ġarticle . ĠWe Ġused Ġthe Ġstandard Ġsplits Ġof ĠB IB REF 24 Ġfor Ġtraining , Ġvalidation , Ġand Ġtesting Ġ( 90 , 266 / 1 , 220 / 1 , 09 3 ĠCNN Ġdocuments Ġand Ġ196 , 96 1 / 12 , 148 / 10 , 397 ĠDaily Mail Ġdocuments ). ĠWe Ġdid Ġnot Ġanonym ize Ġentities . ĠWe Ġfirst Ġsplit Ġsentences Ġwith Ġthe ĠStanford ĠCore N LP Ġtool kit ĠB IB REF 26 Ġand Ġpre - process ed Ġthe Ġdataset Ġfollowing ĠB IB REF 6 . ĠInput Ġdocuments Ġwere Ġtrunc ated Ġto Ġ512 Ġtokens . </s> Exper imental ĠSetup Ġ: :: ĠSum mar ization ĠDat as ets Ġ: :: ĠNYT </s> cont ains Ġ110 , 540 Ġarticles Ġwith Ġabstract ive Ġsumm aries . ĠFollowing ĠB IB REF 27 , Ġwe Ġsplit Ġthese Ġinto Ġ100 , 8 34 / 9 , 706 Ġtraining / test Ġexamples , Ġbased Ġon Ġthe Ġdate Ġof Ġpublication Ġ( the Ġtest Ġset Ġcontains Ġall Ġarticles Ġpublished Ġfrom ĠJanuary Ġ1 , Ġ2007 Ġonward ). ĠWe Ġused Ġ4 , 000 Ġexamples Ġfrom Ġthe Ġtraining Ġas Ġvalidation Ġset . ĠWe Ġalso Ġfollowed Ġtheir Ġfiltering Ġprocedure , Ġdocuments Ġwith Ġsumm aries Ġless Ġthan Ġ50 Ġwords Ġwere Ġremoved Ġfrom Ġthe Ġdataset . ĠThe Ġfiltered Ġtest Ġset Ġ( NY T 50 ) Ġincludes Ġ3 , 452 Ġexamples . ĠSent ences Ġwere Ġsplit Ġwith Ġthe ĠStanford ĠCore N LP Ġtool kit ĠB IB REF 26 Ġand Ġpre - process ed Ġfollowing ĠB IB REF 27 . ĠInput Ġdocuments Ġwere Ġtrunc ated Ġto Ġ800 Ġtokens . </s> Exper imental ĠSetup Ġ: :: ĠSum mar ization ĠDat as ets Ġ: :: ĠX Sum </s> cont ains Ġ226 , 7 11 Ġnews Ġarticles Ġaccompanied Ġwith Ġa Ġone - sent ence Ġsummary , Ġanswering Ġthe Ġquestion ĠâĢ ľ What Ġis Ġthis Ġarticle Ġabout ? âĢ Ŀ . ĠWe Ġused Ġthe Ġsplits Ġof ĠB IB REF 22 Ġfor Ġtraining , Ġvalidation , Ġand Ġtesting Ġ( 204 , 045 / 11 , 332 / 11 , 334 ) Ġand Ġfollowed Ġthe Ġpre - processing Ġintroduced Ġin Ġtheir Ġwork . ĠInput Ġdocuments Ġwere Ġtrunc ated Ġto Ġ512 Ġtokens . </s> Aside Ġfrom Ġvarious Ġstatistics Ġon Ġthe Ġthree Ġdatasets , ĠTable ĠT AB REF 12 Ġalso Ġreports Ġthe Ġproportion Ġof Ġnovel Ġbi - gram s Ġin Ġgold Ġsumm aries Ġas Ġa Ġmeasure Ġof Ġtheir Ġabstract iveness . ĠWe Ġwould Ġexpect Ġmodels Ġwith Ġextract ive Ġbiases Ġto Ġperform Ġbetter Ġon Ġdatasets Ġwith Ġ( mostly ) Ġextract ive Ġsumm aries , Ġand Ġabstract ive Ġmodels Ġto Ġperform Ġmore Ġrewrite Ġoperations Ġon Ġdatasets Ġwith Ġabstract ive Ġsumm aries . ĠCNN / Daily Mail Ġand ĠNYT Ġare Ġsomewhat Ġabstract ive , Ġwhile ĠX Sum Ġis Ġhighly Ġabstract ive . </s> Exper imental ĠSetup Ġ: :: ĠImplementation ĠDetails </s> For Ġboth Ġextract ive Ġand Ġabstract ive Ġsettings , Ġwe Ġused ĠPy Tor ch , ĠOpen N MT ĠB IB REF 28 Ġand Ġthe Ġ` bert - base - unc ased ' Ġversion Ġof ĠBert Ġto Ġimplement ĠBert Sum . ĠBoth Ġsource Ġand Ġtarget Ġtexts Ġwere Ġtoken ized Ġwith ĠBert 's Ġsub words Ġtoken izer . </s> Exper imental ĠSetup Ġ: :: ĠImplementation ĠDetails Ġ: :: ĠExtract ive ĠSum mar ization </s> All Ġextract ive Ġmodels Ġwere Ġtrained Ġfor Ġ50 , 000 Ġsteps Ġon Ġ3 ĠGPUs Ġ( GT X Ġ1080 ĠTi ) Ġwith Ġgradient Ġaccumulation Ġevery Ġtwo Ġsteps . ĠModel Ġcheckpoints Ġwere Ġsaved Ġand Ġevaluated Ġon Ġthe Ġvalidation Ġset Ġevery Ġ1 , 000 Ġsteps . ĠWe Ġselected Ġthe Ġtop - 3 Ġcheckpoints Ġbased Ġon Ġthe Ġevaluation Ġloss Ġon Ġthe Ġvalidation Ġset , Ġand Ġreport Ġthe Ġaveraged Ġresults Ġon Ġthe Ġtest Ġset . ĠWe Ġused Ġa Ġgreedy Ġalgorithm Ġsimilar Ġto ĠB IB REF 7 Ġto Ġobtain Ġan Ġor acle Ġsummary Ġfor Ġeach Ġdocument Ġto Ġtrain Ġextract ive Ġmodels . ĠThe Ġalgorithm Ġgenerates Ġan Ġor acle Ġconsisting Ġof Ġmultiple Ġsentences Ġwhich Ġmaximize Ġthe ĠR OU GE - 2 Ġscore Ġagainst Ġthe Ġgold Ġsummary . </s> When Ġpredicting Ġsumm aries Ġfor Ġa Ġnew Ġdocument , Ġwe Ġfirst Ġuse Ġthe Ġmodel Ġto Ġobtain Ġthe Ġscore Ġfor Ġeach Ġsentence . ĠWe Ġthen Ġrank Ġthese Ġsentences Ġby Ġtheir Ġscores Ġfrom Ġhighest Ġto Ġlowest , Ġand Ġselect Ġthe Ġtop - 3 Ġsentences Ġas Ġthe Ġsummary . </s> During Ġsentence Ġselection Ġwe Ġuse ĠTr ig ram ĠBl ocking Ġto Ġreduce Ġredundancy ĠB IB REF 9 . ĠGiven Ġsummary Ġ$ S $ Ġand Ġcandidate Ġsentence Ġ$ c $, Ġwe Ġskip Ġ$ c $ Ġif Ġthere Ġexists Ġa Ġtrig ram Ġoverlapping Ġbetween Ġ$ c $ Ġand Ġ$ S $. ĠThe Ġintuition Ġis Ġsimilar Ġto ĠMax imal ĠMarg inal ĠRe lev ance Ġ( MM R ; ĠB IB REF 29 ); Ġwe Ġwish Ġto Ġminimize Ġthe Ġsimilarity Ġbetween Ġthe Ġsentence Ġbeing Ġconsidered Ġand Ġsentences Ġwhich Ġhave Ġbeen Ġalready Ġselected Ġas Ġpart Ġof Ġthe Ġsummary . </s> Exper imental ĠSetup Ġ: :: ĠImplementation ĠDetails Ġ: :: ĠAbstract ive ĠSum mar ization </s> In Ġall Ġabstract ive Ġmodels , Ġwe Ġapplied Ġdrop out Ġ( with Ġprobability Ġ$ 0 . 1 $ ) Ġbefore Ġall Ġlinear Ġlayers ; Ġlabel Ġsmoot hing ĠB IB REF 30 Ġwith Ġsmoot hing Ġfactor Ġ$ 0 . 1 $ Ġwas Ġalso Ġused . ĠOur ĠTrans former Ġdec oder Ġhas Ġ768 Ġhidden Ġunits Ġand Ġthe Ġhidden Ġsize Ġfor Ġall Ġfeed - forward Ġlayers Ġis Ġ2 , 048 . ĠAll Ġmodels Ġwere Ġtrained Ġfor Ġ200 , 000 Ġsteps Ġon Ġ4 ĠGPUs Ġ( GT X Ġ1080 ĠTi ) Ġwith Ġgradient Ġaccumulation Ġevery Ġfive Ġsteps . ĠModel Ġcheckpoints Ġwere Ġsaved Ġand Ġevaluated Ġon Ġthe Ġvalidation Ġset Ġevery Ġ2 , 500 Ġsteps . ĠWe Ġselected Ġthe Ġtop - 3 Ġcheckpoints Ġbased Ġon Ġtheir Ġevaluation Ġloss Ġon Ġthe Ġvalidation Ġset , Ġand Ġreport Ġthe Ġaveraged Ġresults Ġon Ġthe Ġtest Ġset . </s> During Ġdecoding Ġwe Ġused Ġbeam Ġsearch Ġ( size Ġ5 ), Ġand Ġtuned Ġthe Ġ$\ alpha Ġ$ Ġfor Ġthe Ġlength Ġpenalty ĠB IB REF 31 Ġbetween Ġ$ 0 . 6 $ Ġand Ġ1 Ġon Ġthe Ġvalidation Ġset ; Ġwe Ġdecode Ġuntil Ġan Ġend - of - sequence Ġtoken Ġis Ġemitted Ġand Ġrepeated Ġtrig rams Ġare Ġblocked ĠB IB REF 9 . ĠIt Ġis Ġworth Ġnoting Ġthat Ġour Ġdec oder Ġapplies Ġneither Ġa Ġcopy Ġnor Ġa Ġcoverage Ġmechanism ĠB IB REF 6 , Ġdespite Ġtheir Ġpopularity Ġin Ġabstract ive Ġsummar ization . ĠThis Ġis Ġmainly Ġbecause Ġwe Ġfocus Ġon Ġbuilding Ġa Ġminimum - requ irements Ġmodel Ġand Ġthese Ġmechanisms Ġmay Ġintroduce Ġadditional Ġhyper - param eters Ġto Ġtune . ĠThanks Ġto Ġthe Ġsub words Ġtoken izer , Ġwe Ġalso Ġrarely Ġobserve Ġissues Ġwith Ġout - of - voc abulary Ġwords Ġin Ġthe Ġoutput ; Ġmoreover , Ġtrig ram - blocking Ġproduces Ġdiverse Ġsumm aries Ġmanaging Ġto Ġreduce Ġrepet itions . </s> Results Ġ: :: ĠAutomatic ĠEvaluation </s> We Ġevaluated Ġsummar ization Ġquality Ġautomatically Ġusing ĠR OU GE ĠB IB REF 32 . ĠWe Ġreport Ġun ig ram Ġand Ġbig ram Ġoverlap Ġ( R OU GE - 1 Ġand ĠR OU GE - 2 ) Ġas Ġa Ġmeans Ġof Ġassessing Ġinform at iveness Ġand Ġthe Ġlongest Ġcommon Ġsubsequ ence Ġ( R OU GE - L ) Ġas Ġa Ġmeans Ġof Ġassessing Ġflu ency . ĠTable ĠT AB REF 23 Ġsummarizes Ġour Ġresults Ġon Ġthe ĠCNN / Daily Mail Ġdataset . ĠThe Ġfirst Ġblock Ġin Ġthe Ġtable Ġincludes Ġthe Ġresults Ġof Ġan Ġextract ive ĠOracle Ġsystem Ġas Ġan Ġupper Ġbound . ĠWe Ġalso Ġpresent Ġthe ĠLead - 3 Ġbaseline Ġ( which Ġsimply Ġselects Ġthe Ġfirst Ġthree Ġsentences Ġin Ġa Ġdocument ). ĠThe Ġsecond Ġblock Ġin Ġthe Ġtable Ġincludes Ġvarious Ġextract ive Ġmodels Ġtrained Ġon Ġthe ĠCNN / Daily Mail Ġdataset Ġ( see ĠSection ĠSEC REF 5 Ġfor Ġan Ġoverview ). ĠFor Ġcomparison Ġto Ġour Ġown Ġmodel , Ġwe Ġalso Ġimplemented Ġa Ġnon - pret rained ĠTrans former Ġbaseline Ġ( Trans former Ext ) Ġwhich Ġuses Ġthe Ġsame Ġarchitecture Ġas ĠBert Sum Ext , Ġbut Ġwith Ġfewer Ġparameters . ĠIt Ġis Ġrandomly Ġinitialized Ġand Ġonly Ġtrained Ġon Ġthe Ġsummar ization Ġtask . ĠTrans former Ext Ġhas Ġ6 Ġlayers , Ġthe Ġhidden Ġsize Ġis Ġ512 , Ġand Ġthe Ġfeed - forward Ġfilter Ġsize Ġis Ġ2 , 048 . ĠThe Ġmodel Ġwas Ġtrained Ġwith Ġsame Ġsettings Ġas Ġin ĠB IB REF 3 . ĠThe Ġthird Ġblock Ġin ĠTable ĠT AB REF 23 Ġhighlights Ġthe Ġperformance Ġof Ġseveral Ġabstract ive Ġmodels Ġon Ġthe ĠCNN / Daily Mail Ġdataset Ġ( see ĠSection ĠSEC REF 6 Ġfor Ġan Ġoverview ). ĠWe Ġalso Ġinclude Ġan Ġabstract ive ĠTrans former Ġbaseline Ġ( Trans former Abs ) Ġwhich Ġhas Ġthe Ġsame Ġdec oder Ġas Ġour Ġabstract ive ĠBert Sum Ġmodels ; Ġthe Ġenc oder Ġis Ġa Ġ6 - layer ĠTrans former Ġwith Ġ768 Ġhidden Ġsize Ġand Ġ2 , 048 Ġfeed - forward Ġfilter Ġsize . ĠThe Ġfourth Ġblock Ġreports Ġresults Ġwith Ġfine - tun ed ĠBert Ġmodels : ĠBert Sum Ext Ġand Ġits Ġtwo Ġvariants Ġ( one Ġwithout Ġinterval Ġembed d ings , Ġand Ġone Ġwith Ġthe Ġlarge Ġversion Ġof ĠBert ), ĠBert Sum Abs , Ġand ĠBert Sum Ext Abs . ĠBert - based Ġmodels Ġoutper form Ġthe ĠLead - 3 Ġbaseline Ġwhich Ġis Ġnot Ġa Ġstraw man ; Ġon Ġthe ĠCNN / Daily Mail Ġcorpus Ġit Ġis Ġindeed Ġsuperior Ġto Ġseveral Ġextract ive ĠB IB REF 7 , ĠB IB REF 8 , ĠB IB REF 19 Ġand Ġabstract ive Ġmodels ĠB IB REF 6 . ĠBert Ġmodels Ġcollectively Ġoutper form Ġall Ġpreviously Ġproposed Ġextract ive Ġand Ġabstract ive Ġsystems , Ġonly Ġfalling Ġbehind Ġthe ĠOracle Ġupper Ġbound . ĠAmong ĠBert Ġvariants , ĠBert Sum Ext Ġperforms Ġbest Ġwhich Ġis Ġnot Ġentirely Ġsurprising ; ĠCNN / Daily Mail Ġsumm aries Ġare Ġsomewhat Ġextract ive Ġand Ġeven Ġabstract ive Ġmodels Ġare Ġprone Ġto Ġcopying Ġsentences Ġfrom Ġthe Ġsource Ġdocument Ġwhen Ġtrained Ġon Ġthis Ġdataset ĠB IB REF 6 . ĠPerhaps Ġunsurprisingly Ġwe Ġobserve Ġthat Ġlarger Ġversions Ġof ĠBert Ġlead Ġto Ġperformance Ġimprovements Ġand Ġthat Ġinterval Ġembed d ings Ġbring Ġonly Ġslight Ġgains . ĠTable ĠT AB REF 24 Ġpresents Ġresults Ġon Ġthe ĠNYT Ġdataset . ĠFollowing Ġthe Ġevaluation Ġprotocol Ġin ĠB IB REF 27 , Ġwe Ġuse Ġlimited - length ĠR OU GE ĠRecall , Ġwhere Ġpredicted Ġsumm aries Ġare Ġtrunc ated Ġto Ġthe Ġlength Ġof Ġthe Ġgold Ġsumm aries . ĠAgain , Ġwe Ġreport Ġthe Ġperformance Ġof Ġthe ĠOracle Ġupper Ġbound Ġand ĠLead - 3 Ġbaseline . ĠThe Ġsecond Ġblock Ġin Ġthe Ġtable Ġcontains Ġpreviously Ġproposed Ġextract ive Ġmodels Ġas Ġwell Ġas Ġour Ġown ĠTrans former Ġbaseline . ĠComp ress ĠB IB REF 27 Ġis Ġan ĠIL P - based Ġmodel Ġwhich Ġcombines Ġcompression Ġand Ġan aph oric ity Ġconstraints . ĠThe Ġthird Ġblock Ġincludes Ġabstract ive Ġmodels Ġfrom Ġthe Ġliterature , Ġand Ġour ĠTrans former Ġbaseline . ĠBert - based Ġmodels Ġare Ġshown Ġin Ġthe Ġfourth Ġblock . ĠAgain , Ġwe Ġobserve Ġthat Ġthey Ġoutper form Ġpreviously Ġproposed Ġapproaches . ĠOn Ġthis Ġdataset , Ġabstract ive ĠBert Ġmodels Ġgenerally Ġperform Ġbetter Ġcompared Ġto ĠBert Sum Ext , Ġalmost Ġapproaching ĠOracle Ġperformance . </s> Table ĠT AB REF 26 Ġsummarizes Ġour Ġresults Ġon Ġthe ĠX Sum Ġdataset . ĠRecall Ġthat Ġsumm aries Ġin Ġthis Ġdataset Ġare Ġhighly Ġabstract ive Ġ( see ĠTable ĠT AB REF 12 ) Ġconsisting Ġof Ġa Ġsingle Ġsentence Ġconve ying Ġthe Ġgist Ġof Ġthe Ġdocument . ĠExtract ive Ġmodels Ġhere Ġperform Ġpoorly Ġas Ġcorrobor ated Ġby Ġthe Ġlow Ġperformance Ġof Ġthe ĠLead Ġbaseline Ġ( which Ġsimply Ġselects Ġthe Ġleading Ġsentence Ġfrom Ġthe Ġdocument ), Ġand Ġthe ĠOracle Ġ( which Ġselects Ġa Ġsingle - best Ġsentence Ġin Ġeach Ġdocument ) Ġin ĠTable ĠT AB REF 26 . ĠAs Ġa Ġresult , Ġwe Ġdo Ġnot Ġreport Ġresults Ġfor Ġextract ive Ġmodels Ġon Ġthis Ġdataset . ĠThe Ġsecond Ġblock Ġin ĠTable ĠT AB REF 26 Ġpresents Ġthe Ġresults Ġof Ġvarious Ġabstract ive Ġmodels Ġtaken Ġfrom ĠB IB REF 22 Ġand Ġalso Ġincludes Ġour Ġown Ġabstract ive ĠTrans former Ġbaseline . ĠIn Ġthe Ġthird Ġblock Ġwe Ġshow Ġthe Ġresults Ġof Ġour ĠBert Ġsummar izers Ġwhich Ġagain Ġare Ġsuperior Ġto Ġall Ġpreviously Ġreported Ġmodels Ġ( by Ġa Ġwide Ġmargin ). </s> Results Ġ: :: ĠModel ĠAnalysis Ġ: :: ĠLearning ĠRates </s> Rec all Ġthat Ġour Ġabstract ive Ġmodel Ġuses Ġseparate Ġoptim izers Ġfor Ġthe Ġenc oder Ġand Ġdec oder . ĠIn ĠTable ĠT AB REF 27 Ġwe Ġexamine Ġwhether Ġthe Ġcombination Ġof Ġdifferent Ġlearning Ġrates Ġ($ \ t ilde { lr } _{ \ math cal Ġ{ E }} $ Ġand Ġ$\ t ilde { lr } _{ \ math cal Ġ{ D }} $ ) Ġis Ġindeed Ġbeneficial . ĠSpecifically , Ġwe Ġreport Ġmodel Ġperplex ity Ġon Ġthe ĠCNN / Daily Mail Ġvalidation Ġset Ġfor Ġvarying Ġenc oder / dec oder Ġlearning Ġrates . ĠWe Ġcan Ġsee Ġthat Ġthe Ġmodel Ġperforms Ġbest Ġwith Ġ$\ t ilde { lr } _{ \ math cal Ġ{ E }} = 2 e - 3 $ Ġand Ġ$\ t ilde { lr } _{ \ math cal Ġ{ D }} = 0 . 1 $. </s> Results Ġ: :: ĠModel ĠAnalysis Ġ: :: ĠPosition Ġof ĠExt racted ĠSent ences </s> In Ġaddition Ġto Ġthe Ġevaluation Ġbased Ġon ĠR OU GE , Ġwe Ġalso Ġanalyzed Ġin Ġmore Ġdetail Ġthe Ġsumm aries Ġproduced Ġby Ġour Ġmodel . ĠFor Ġthe Ġextract ive Ġsetting , Ġwe Ġlooked Ġat Ġthe Ġposition Ġ( in Ġthe Ġsource Ġdocument ) Ġof Ġthe Ġsentences Ġwhich Ġwere Ġselected Ġto Ġappear Ġin Ġthe Ġsummary . ĠFigure ĠFIG REF 31 Ġshows Ġthe Ġproportion Ġof Ġselected Ġsummary Ġsentences Ġwhich Ġappear Ġin Ġthe Ġsource Ġdocument Ġat Ġpositions Ġ1 , Ġ2 , Ġand Ġso Ġon . ĠThe Ġanalysis Ġwas Ġconducted Ġon Ġthe ĠCNN / Daily Mail Ġdataset Ġfor ĠOracle Ġsumm aries , Ġand Ġthose Ġproduced Ġby ĠBert Sum Ext Ġand Ġthe ĠTrans former Ext . ĠWe Ġcan Ġsee Ġthat ĠOracle Ġsummary Ġsentences Ġare Ġfairly Ġsmoothly Ġdistributed Ġacross Ġdocuments , Ġwhile Ġsumm aries Ġcreated Ġby ĠTrans former Ext Ġmostly Ġconcentrate Ġon Ġthe Ġfirst Ġdocument Ġsentences . ĠBert Sum Ext Ġoutputs Ġare Ġmore Ġsimilar Ġto ĠOracle Ġsumm aries , Ġindicating Ġthat Ġwith Ġthe Ġpret rained Ġenc oder , Ġthe Ġmodel Ġrelies Ġless Ġon Ġshallow Ġposition Ġfeatures , Ġand Ġlearns Ġdeeper Ġdocument Ġrepresentations . </s> Results Ġ: :: ĠModel ĠAnalysis Ġ: :: ĠNovel ĠN - gram s </s> We Ġalso Ġanalyzed Ġthe Ġoutput Ġof Ġabstract ive Ġsystems Ġby Ġcalculating Ġthe Ġproportion Ġof Ġnovel Ġn - gram s Ġthat Ġappear Ġin Ġthe Ġsumm aries Ġbut Ġnot Ġin Ġthe Ġsource Ġtexts . ĠThe Ġresults Ġare Ġshown Ġin ĠFigure ĠFIG REF 33 . ĠIn Ġthe ĠCNN / Daily Mail Ġdataset , Ġthe Ġproportion Ġof Ġnovel Ġn - gram s Ġin Ġautomatically Ġgenerated Ġsumm aries Ġis Ġmuch Ġlower Ġcompared Ġto Ġreference Ġsumm aries , Ġbut Ġin ĠX Sum , Ġthis Ġgap Ġis Ġmuch Ġsmaller . ĠWe Ġalso Ġobserve Ġthat Ġon ĠCNN / Daily Mail , ĠBert Ext Abs Ġproduces Ġless Ġnovel Ġn - ng rams Ġthan ĠBert Abs , Ġwhich Ġis Ġnot Ġsurprising . ĠBert Ext Abs Ġis Ġmore Ġbiased Ġtowards Ġselecting Ġsentences Ġfrom Ġthe Ġsource Ġdocument Ġsince Ġit Ġis Ġinitially Ġtrained Ġas Ġan Ġextract ive Ġmodel . ĠThe Ġsupplementary Ġmaterial Ġincludes Ġexamples Ġof Ġsystem Ġoutput Ġand Ġadditional Ġab lation Ġstudies . </s> Results Ġ: :: ĠHuman ĠEvaluation </s> In Ġaddition Ġto Ġautomatic Ġevaluation , Ġwe Ġalso Ġevaluated Ġsystem Ġoutput Ġby Ġelic iting Ġhuman Ġjudgments . ĠWe Ġreport Ġexperiments Ġfollowing Ġa Ġquestion - ans w ering Ġ( Q A ) Ġparadigm ĠB IB REF 33 , ĠB IB REF 8 Ġwhich Ġquant ifies Ġthe Ġdegree Ġto Ġwhich Ġsummar ization Ġmodels Ġretain Ġkey Ġinformation Ġfrom Ġthe Ġdocument . ĠUnder Ġthis Ġparadigm , Ġa Ġset Ġof Ġquestions Ġis Ġcreated Ġbased Ġon Ġthe Ġgold Ġsummary Ġunder Ġthe Ġassumption Ġthat Ġit Ġhighlights Ġthe Ġmost Ġimportant Ġdocument Ġcontent . ĠParticipants Ġare Ġthen Ġasked Ġto Ġanswer Ġthese Ġquestions Ġby Ġreading Ġsystem Ġsumm aries Ġalone Ġwithout Ġaccess Ġto Ġthe Ġarticle . ĠThe Ġmore Ġquestions Ġa Ġsystem Ġcan Ġanswer , Ġthe Ġbetter Ġit Ġis Ġat Ġsummar izing Ġthe Ġdocument Ġas Ġa Ġwhole . ĠMoreover , Ġwe Ġalso Ġassessed Ġthe Ġoverall Ġquality Ġof Ġthe Ġsumm aries Ġproduced Ġby Ġabstract ive Ġsystems Ġwhich Ġdue Ġto Ġtheir Ġability Ġto Ġrewrite Ġcontent Ġmay Ġproduce Ġdis f luent Ġor Ġun gram matical Ġoutput . ĠSpecifically , Ġwe Ġfollowed Ġthe ĠBest - W orst ĠSc aling ĠB IB REF 34 Ġmethod Ġwhere Ġparticipants Ġwere Ġpresented Ġwith Ġthe Ġoutput Ġof Ġtwo Ġsystems Ġ( and Ġthe Ġoriginal Ġdocument ) Ġand Ġasked Ġto Ġdecide Ġwhich Ġone Ġwas Ġbetter Ġaccording Ġto Ġthe Ġcriteria Ġof ĠIn format iveness , ĠFlu ency , Ġand ĠSu cc inct ness . </s> Both Ġtypes Ġof Ġevaluation Ġwere Ġconducted Ġon Ġthe ĠAmazon ĠMechanical ĠTurk Ġplatform . ĠFor Ġthe ĠCNN / Daily Mail Ġand ĠNYT Ġdatasets Ġwe Ġused Ġthe Ġsame Ġdocuments Ġ( 20 Ġin Ġtotal ) Ġand Ġquestions Ġfrom Ġprevious Ġwork ĠB IB REF 8 , ĠB IB REF 18 . ĠFor ĠX Sum , Ġwe Ġrandomly Ġselected Ġ20 Ġdocuments Ġ( and Ġtheir Ġquestions ) Ġfrom Ġthe Ġrelease Ġof ĠB IB REF 22 . ĠWe Ġelic ited Ġ3 Ġresponses Ġper ĠHIT . ĠWith Ġregard Ġto ĠQ A Ġevaluation , Ġwe Ġadopted Ġthe Ġscoring Ġmechanism Ġfrom ĠB IB REF 33 ; Ġcorrect Ġanswers Ġwere Ġmarked Ġwith Ġa Ġscore Ġof Ġone , Ġpartially Ġcorrect Ġanswers Ġwith Ġ0 . 5 , Ġand Ġzero Ġotherwise . ĠFor Ġquality - based Ġevaluation , Ġthe Ġrating Ġof Ġeach Ġsystem Ġwas Ġcomputed Ġas Ġthe Ġpercentage Ġof Ġtimes Ġit Ġwas Ġchosen Ġas Ġbetter Ġminus Ġthe Ġtimes Ġit Ġwas Ġselected Ġas Ġworse . ĠRatings Ġthus Ġrange Ġfrom Ġ- 1 Ġ( worst ) Ġto Ġ1 Ġ( best ). </s> Results Ġfor Ġextract ive Ġand Ġabstract ive Ġsystems Ġare Ġshown Ġin ĠTables ĠT AB REF 37 Ġand ĠT AB REF 38 , Ġrespectively . ĠWe Ġcompared Ġthe Ġbest Ġperforming ĠBert Sum Ġmodel Ġin Ġeach Ġsetting Ġ( ext ractive Ġor Ġabstract ive ) Ġagainst Ġvarious Ġstate - of - the - art Ġsystems Ġ( whose Ġoutput Ġis Ġpublicly Ġavailable ), Ġthe ĠLead Ġbaseline , Ġand Ġthe ĠGold Ġstandard Ġas Ġan Ġupper Ġbound . ĠAs Ġshown Ġin Ġboth Ġtables Ġparticipants Ġoverwhelmingly Ġprefer Ġthe Ġoutput Ġof Ġour Ġmodel Ġagainst Ġcomparison Ġsystems Ġacross Ġdatasets Ġand Ġevaluation Ġparad ig ms . ĠAll Ġdifferences Ġbetween ĠBert Sum Ġand Ġcomparison Ġmodels Ġare Ġstatistically Ġsignificant Ġ($ p < 0 . 05 $ ), Ġwith Ġthe Ġexception Ġof ĠT Con v S 2 S Ġ( see ĠTable ĠT AB REF 38 ; ĠX Sum ) Ġin Ġthe ĠQ A Ġevaluation Ġsetting . </s> Con clusions </s> In Ġthis Ġpaper , Ġwe Ġshowcased Ġhow Ġpret rained ĠBert Ġcan Ġbe Ġuse fully Ġapplied Ġin Ġtext Ġsummar ization . ĠWe Ġintroduced Ġa Ġnovel Ġdocument - level Ġenc oder Ġand Ġproposed Ġa Ġgeneral Ġframework Ġfor Ġboth Ġabstract ive Ġand Ġextract ive Ġsummar ization . ĠExperimental Ġresults Ġacross Ġthree Ġdatasets Ġshow Ġthat Ġour Ġmodel Ġachieves Ġstate - of - the - art Ġresults Ġacross Ġthe Ġboard Ġunder Ġautomatic Ġand Ġhuman - based Ġevaluation Ġprotocols . ĠAlthough Ġwe Ġmainly Ġfocused Ġon Ġdocument Ġencoding Ġfor Ġsummar ization , Ġin Ġthe Ġfuture , Ġwe Ġwould Ġlike Ġto Ġtake Ġadvantage Ġthe Ġcapabilities Ġof ĠBert Ġfor Ġlanguage Ġgeneration . </s> Acknowled gments </s> This Ġresearch Ġis Ġsupported Ġby Ġa ĠGoogle ĠPhD ĠFellowship Ġto Ġthe Ġfirst Ġauthor . ĠWe Ġgrate fully Ġacknowledge Ġthe Ġsupport Ġof Ġthe ĠEuropean ĠResearch ĠCouncil Ġ( L ap ata , Ġaward Ġnumber Ġ68 17 60 , ĠâĢ ľ Trans l ating ĠMultiple ĠMod alities Ġinto ĠText âĢ Ŀ ). ĠWe Ġwould Ġalso Ġlike Ġto Ġthank ĠSh ashi ĠNar ayan Ġfor Ġproviding Ġus Ġwith Ġthe ĠX Sum Ġdataset .
Introduction Language model pretraining has advanced the state of the art in many NLP tasks ranging from sentiment analysis, to question answering, natural language inference, named entity recognition, and textual similarity. State-of-the-art pretrained models include ELMo BIBREF1, GPT BIBREF2, and more recently Bidirectional Encoder Representations from Transformers (Bert; BIBREF0). Bert combines both word and sentence representations in a single very large Transformer BIBREF3; it is pretrained on vast amounts of text, with an unsupervised objective of masked language modeling and next-sentence prediction and can be fine-tuned with various task-specific objectives. In most cases, pretrained language models have been employed as encoders for sentence- and paragraph-level natural language understanding problems BIBREF0 involving various classification tasks (e.g., predicting whether any two sentences are in an entailment relationship; or determining the completion of a sentence among four alternative sentences). In this paper, we examine the influence of language model pretraining on text summarization. Different from previous tasks, summarization requires wide-coverage natural language understanding going beyond the meaning of individual words and sentences. The aim is to condense a document into a shorter version while preserving most of its meaning. Furthermore, under abstractive modeling formulations, the task requires language generation capabilities in order to create summaries containing novel words and phrases not featured in the source text, while extractive summarization is often defined as a binary classification task with labels indicating whether a text span (typically a sentence) should be included in the summary. We explore the potential of Bert for text summarization under a general framework encompassing both extractive and abstractive modeling paradigms. We propose a novel document-level encoder based on Bert which is able to encode a document and obtain representations for its sentences. Our extractive model is built on top of this encoder by stacking several inter-sentence Transformer layers to capture document-level features for extracting sentences. Our abstractive model adopts an encoder-decoder architecture, combining the same pretrained Bert encoder with a randomly-initialized Transformer decoder BIBREF3. We design a new training schedule which separates the optimizers of the encoder and the decoder in order to accommodate the fact that the former is pretrained while the latter must be trained from scratch. Finally, motivated by previous work showing that the combination of extractive and abstractive objectives can help generate better summaries BIBREF4, we present a two-stage approach where the encoder is fine-tuned twice, first with an extractive objective and subsequently on the abstractive summarization task. We evaluate the proposed approach on three single-document news summarization datasets representative of different writing conventions (e.g., important information is concentrated at the beginning of the document or distributed more evenly throughout) and summary styles (e.g., verbose vs. more telegraphic; extractive vs. abstractive). Across datasets, we experimentally show that the proposed models achieve state-of-the-art results under both extractive and abstractive settings. Our contributions in this work are three-fold: a) we highlight the importance of document encoding for the summarization task; a variety of recently proposed techniques aim to enhance summarization performance via copying mechanisms BIBREF5, BIBREF6, BIBREF7, reinforcement learning BIBREF8, BIBREF9, BIBREF10, and multiple communicating encoders BIBREF11. We achieve better results with a minimum-requirement model without using any of these mechanisms; b) we showcase ways to effectively employ pretrained language models in summarization under both extractive and abstractive settings; we would expect any improvements in model pretraining to translate in better summarization in the future; and c) the proposed models can be used as a stepping stone to further improve summarization performance as well as baselines against which new proposals are tested. Background ::: Pretrained Language Models Pretrained language models BIBREF1, BIBREF2, BIBREF0, BIBREF12, BIBREF13 have recently emerged as a key technology for achieving impressive gains in a wide variety of natural language tasks. These models extend the idea of word embeddings by learning contextual representations from large-scale corpora using a language modeling objective. Bidirectional Encoder Representations from Transformers (Bert; BIBREF0) is a new language representation model which is trained with a masked language modeling and a “next sentence prediction” task on a corpus of 3,300M words. The general architecture of Bert is shown in the left part of Figure FIGREF2. Input text is first preprocessed by inserting two special tokens. [cls] is appended to the beginning of the text; the output representation of this token is used to aggregate information from the whole sequence (e.g., for classification tasks). And token [sep] is inserted after each sentence as an indicator of sentence boundaries. The modified text is then represented as a sequence of tokens $X=[w_1,w_2,\cdots ,w_n]$. Each token $w_i$ is assigned three kinds of embeddings: token embeddings indicate the meaning of each token, segmentation embeddings are used to discriminate between two sentences (e.g., during a sentence-pair classification task) and position embeddings indicate the position of each token within the text sequence. These three embeddings are summed to a single input vector $x_i$ and fed to a bidirectional Transformer with multiple layers: where $h^0=x$ are the input vectors; $\mathrm {LN}$ is the layer normalization operation BIBREF14; $\mathrm {MHAtt}$ is the multi-head attention operation BIBREF3; superscript $l$ indicates the depth of the stacked layer. On the top layer, Bert will generate an output vector $t_i$ for each token with rich contextual information. Pretrained language models are usually used to enhance performance in language understanding tasks. Very recently, there have been attempts to apply pretrained models to various generation problems BIBREF15, BIBREF16. When fine-tuning for a specific task, unlike ELMo whose parameters are usually fixed, parameters in Bert are jointly fine-tuned with additional task-specific parameters. Background ::: Extractive Summarization Extractive summarization systems create a summary by identifying (and subsequently concatenating) the most important sentences in a document. Neural models consider extractive summarization as a sentence classification problem: a neural encoder creates sentence representations and a classifier predicts which sentences should be selected as summaries. SummaRuNNer BIBREF7 is one of the earliest neural approaches adopting an encoder based on Recurrent Neural Networks. Refresh BIBREF8 is a reinforcement learning-based system trained by globally optimizing the ROUGE metric. More recent work achieves higher performance with more sophisticated model structures. Latent BIBREF17 frames extractive summarization as a latent variable inference problem; instead of maximizing the likelihood of “gold” standard labels, their latent model directly maximizes the likelihood of human summaries given selected sentences. Sumo BIBREF18 capitalizes on the notion of structured attention to induce a multi-root dependency tree representation of the document while predicting the output summary. NeuSum BIBREF19 scores and selects sentences jointly and represents the state of the art in extractive summarization. Background ::: Abstractive Summarization Neural approaches to abstractive summarization conceptualize the task as a sequence-to-sequence problem, where an encoder maps a sequence of tokens in the source document $\mathbf {x} = [x_1, ..., x_n]$ to a sequence of continuous representations $\mathbf {z} = [z_1, ..., z_n]$, and a decoder then generates the target summary $\mathbf {y} = [y_1, ..., y_m]$ token-by-token, in an auto-regressive manner, hence modeling the conditional probability: $p(y_1, ..., y_m|x_1, ..., x_n)$. BIBREF20 and BIBREF21 were among the first to apply the neural encoder-decoder architecture to text summarization. BIBREF6 enhance this model with a pointer-generator network (PTgen) which allows it to copy words from the source text, and a coverage mechanism (Cov) which keeps track of words that have been summarized. BIBREF11 propose an abstractive system where multiple agents (encoders) represent the document together with a hierarchical attention mechanism (over the agents) for decoding. Their Deep Communicating Agents (DCA) model is trained end-to-end with reinforcement learning. BIBREF9 also present a deep reinforced model (DRM) for abstractive summarization which handles the coverage problem with an intra-attention mechanism where the decoder attends over previously generated words. BIBREF4 follow a bottom-up approach (BottomUp); a content selector first determines which phrases in the source document should be part of the summary, and a copy mechanism is applied only to preselected phrases during decoding. BIBREF22 propose an abstractive model which is particularly suited to extreme summarization (i.e., single sentence summaries), based on convolutional neural networks and additionally conditioned on topic distributions (TConvS2S). Fine-tuning Bert for Summarization ::: Summarization Encoder Although Bert has been used to fine-tune various NLP tasks, its application to summarization is not as straightforward. Since Bert is trained as a masked-language model, the output vectors are grounded to tokens instead of sentences, while in extractive summarization, most models manipulate sentence-level representations. Although segmentation embeddings represent different sentences in Bert, they only apply to sentence-pair inputs, while in summarization we must encode and manipulate multi-sentential inputs. Figure FIGREF2 illustrates our proposed Bert architecture for Summarization (which we call BertSum). In order to represent individual sentences, we insert external [cls] tokens at the start of each sentence, and each [cls] symbol collects features for the sentence preceding it. We also use interval segment embeddings to distinguish multiple sentences within a document. For $sent_i$ we assign segment embedding $E_A$ or $E_B$ depending on whether $i$ is odd or even. For example, for document $[sent_1, sent_2, sent_3, sent_4, sent_5]$, we would assign embeddings $[E_A, E_B, E_A,E_B, E_A]$. This way, document representations are learned hierarchically where lower Transformer layers represent adjacent sentences, while higher layers, in combination with self-attention, represent multi-sentence discourse. Position embeddings in the original Bert model have a maximum length of 512; we overcome this limitation by adding more position embeddings that are initialized randomly and fine-tuned with other parameters in the encoder. Fine-tuning Bert for Summarization ::: Extractive Summarization Let $d$ denote a document containing sentences $[sent_1, sent_2, \cdots , sent_m]$, where $sent_i$ is the $i$-th sentence in the document. Extractive summarization can be defined as the task of assigning a label $y_i \in \lbrace 0, 1\rbrace $ to each $sent_i$, indicating whether the sentence should be included in the summary. It is assumed that summary sentences represent the most important content of the document. With BertSum, vector $t_i$ which is the vector of the $i$-th [cls] symbol from the top layer can be used as the representation for $sent_i$. Several inter-sentence Transformer layers are then stacked on top of Bert outputs, to capture document-level features for extracting summaries: where $h^0=\mathrm {PosEmb}(T)$; $T$ denotes the sentence vectors output by BertSum, and function $\mathrm {PosEmb}$ adds sinusoid positional embeddings BIBREF3 to $T$, indicating the position of each sentence. The final output layer is a sigmoid classifier: where $h^L_i$ is the vector for $sent_i$ from the top layer (the $L$-th layer ) of the Transformer. In experiments, we implemented Transformers with $L=1, 2, 3$ and found that a Transformer with $L=2$ performed best. We name this model BertSumExt. The loss of the model is the binary classification entropy of prediction $\hat{y}_i$ against gold label $y_i$. Inter-sentence Transformer layers are jointly fine-tuned with BertSum. We use the Adam optimizer with $\beta _1=0.9$, and $\beta _2=0.999$). Our learning rate schedule follows BIBREF3 with warming-up ($ \operatorname{\operatorname{warmup}}=10,000$): Fine-tuning Bert for Summarization ::: Abstractive Summarization We use a standard encoder-decoder framework for abstractive summarization BIBREF6. The encoder is the pretrained BertSum and the decoder is a 6-layered Transformer initialized randomly. It is conceivable that there is a mismatch between the encoder and the decoder, since the former is pretrained while the latter must be trained from scratch. This can make fine-tuning unstable; for example, the encoder might overfit the data while the decoder underfits, or vice versa. To circumvent this, we design a new fine-tuning schedule which separates the optimizers of the encoder and the decoder. We use two Adam optimizers with $\beta _1=0.9$ and $\beta _2=0.999$ for the encoder and the decoder, respectively, each with different warmup-steps and learning rates: where $\tilde{lr}_{\mathcal {E}}=2e^{-3}$, and $\operatorname{\operatorname{warmup}}_{\mathcal {E}}=20,000$ for the encoder and $\tilde{lr}_{\mathcal {D}}=0.1$, and $\operatorname{\operatorname{warmup}}_{\mathcal {D}}=10,000$ for the decoder. This is based on the assumption that the pretrained encoder should be fine-tuned with a smaller learning rate and smoother decay (so that the encoder can be trained with more accurate gradients when the decoder is becoming stable). In addition, we propose a two-stage fine-tuning approach, where we first fine-tune the encoder on the extractive summarization task (Section SECREF8) and then fine-tune it on the abstractive summarization task (Section SECREF13). Previous work BIBREF4, BIBREF23 suggests that using extractive objectives can boost the performance of abstractive summarization. Also notice that this two-stage approach is conceptually very simple, the model can take advantage of information shared between these two tasks, without fundamentally changing its architecture. We name the default abstractive model BertSumAbs and the two-stage fine-tuned model BertSumExtAbs. Experimental Setup In this section, we describe the summarization datasets used in our experiments and discuss various implementation details. Experimental Setup ::: Summarization Datasets We evaluated our model on three benchmark datasets, namely the CNN/DailyMail news highlights dataset BIBREF24, the New York Times Annotated Corpus (NYT; BIBREF25), and XSum BIBREF22. These datasets represent different summary styles ranging from highlights to very brief one sentence summaries. The summaries also vary with respect to the type of rewriting operations they exemplify (e.g., some showcase more cut and paste operations while others are genuinely abstractive). Table TABREF12 presents statistics on these datasets (test set); example (gold-standard) summaries are provided in the supplementary material. Experimental Setup ::: Summarization Datasets ::: CNN/DailyMail contains news articles and associated highlights, i.e., a few bullet points giving a brief overview of the article. We used the standard splits of BIBREF24 for training, validation, and testing (90,266/1,220/1,093 CNN documents and 196,961/12,148/10,397 DailyMail documents). We did not anonymize entities. We first split sentences with the Stanford CoreNLP toolkit BIBREF26 and pre-processed the dataset following BIBREF6. Input documents were truncated to 512 tokens. Experimental Setup ::: Summarization Datasets ::: NYT contains 110,540 articles with abstractive summaries. Following BIBREF27, we split these into 100,834/9,706 training/test examples, based on the date of publication (the test set contains all articles published from January 1, 2007 onward). We used 4,000 examples from the training as validation set. We also followed their filtering procedure, documents with summaries less than 50 words were removed from the dataset. The filtered test set (NYT50) includes 3,452 examples. Sentences were split with the Stanford CoreNLP toolkit BIBREF26 and pre-processed following BIBREF27. Input documents were truncated to 800 tokens. Experimental Setup ::: Summarization Datasets ::: XSum contains 226,711 news articles accompanied with a one-sentence summary, answering the question “What is this article about?”. We used the splits of BIBREF22 for training, validation, and testing (204,045/11,332/11,334) and followed the pre-processing introduced in their work. Input documents were truncated to 512 tokens. Aside from various statistics on the three datasets, Table TABREF12 also reports the proportion of novel bi-grams in gold summaries as a measure of their abstractiveness. We would expect models with extractive biases to perform better on datasets with (mostly) extractive summaries, and abstractive models to perform more rewrite operations on datasets with abstractive summaries. CNN/DailyMail and NYT are somewhat abstractive, while XSum is highly abstractive. Experimental Setup ::: Implementation Details For both extractive and abstractive settings, we used PyTorch, OpenNMT BIBREF28 and the `bert-base-uncased' version of Bert to implement BertSum. Both source and target texts were tokenized with Bert's subwords tokenizer. Experimental Setup ::: Implementation Details ::: Extractive Summarization All extractive models were trained for 50,000 steps on 3 GPUs (GTX 1080 Ti) with gradient accumulation every two steps. Model checkpoints were saved and evaluated on the validation set every 1,000 steps. We selected the top-3 checkpoints based on the evaluation loss on the validation set, and report the averaged results on the test set. We used a greedy algorithm similar to BIBREF7 to obtain an oracle summary for each document to train extractive models. The algorithm generates an oracle consisting of multiple sentences which maximize the ROUGE-2 score against the gold summary. When predicting summaries for a new document, we first use the model to obtain the score for each sentence. We then rank these sentences by their scores from highest to lowest, and select the top-3 sentences as the summary. During sentence selection we use Trigram Blocking to reduce redundancy BIBREF9. Given summary $S$ and candidate sentence $c$, we skip $c$ if there exists a trigram overlapping between $c$ and $S$. The intuition is similar to Maximal Marginal Relevance (MMR; BIBREF29); we wish to minimize the similarity between the sentence being considered and sentences which have been already selected as part of the summary. Experimental Setup ::: Implementation Details ::: Abstractive Summarization In all abstractive models, we applied dropout (with probability $0.1$) before all linear layers; label smoothing BIBREF30 with smoothing factor $0.1$ was also used. Our Transformer decoder has 768 hidden units and the hidden size for all feed-forward layers is 2,048. All models were trained for 200,000 steps on 4 GPUs (GTX 1080 Ti) with gradient accumulation every five steps. Model checkpoints were saved and evaluated on the validation set every 2,500 steps. We selected the top-3 checkpoints based on their evaluation loss on the validation set, and report the averaged results on the test set. During decoding we used beam search (size 5), and tuned the $\alpha $ for the length penalty BIBREF31 between $0.6$ and 1 on the validation set; we decode until an end-of-sequence token is emitted and repeated trigrams are blocked BIBREF9. It is worth noting that our decoder applies neither a copy nor a coverage mechanism BIBREF6, despite their popularity in abstractive summarization. This is mainly because we focus on building a minimum-requirements model and these mechanisms may introduce additional hyper-parameters to tune. Thanks to the subwords tokenizer, we also rarely observe issues with out-of-vocabulary words in the output; moreover, trigram-blocking produces diverse summaries managing to reduce repetitions. Results ::: Automatic Evaluation We evaluated summarization quality automatically using ROUGE BIBREF32. We report unigram and bigram overlap (ROUGE-1 and ROUGE-2) as a means of assessing informativeness and the longest common subsequence (ROUGE-L) as a means of assessing fluency. Table TABREF23 summarizes our results on the CNN/DailyMail dataset. The first block in the table includes the results of an extractive Oracle system as an upper bound. We also present the Lead-3 baseline (which simply selects the first three sentences in a document). The second block in the table includes various extractive models trained on the CNN/DailyMail dataset (see Section SECREF5 for an overview). For comparison to our own model, we also implemented a non-pretrained Transformer baseline (TransformerExt) which uses the same architecture as BertSumExt, but with fewer parameters. It is randomly initialized and only trained on the summarization task. TransformerExt has 6 layers, the hidden size is 512, and the feed-forward filter size is 2,048. The model was trained with same settings as in BIBREF3. The third block in Table TABREF23 highlights the performance of several abstractive models on the CNN/DailyMail dataset (see Section SECREF6 for an overview). We also include an abstractive Transformer baseline (TransformerAbs) which has the same decoder as our abstractive BertSum models; the encoder is a 6-layer Transformer with 768 hidden size and 2,048 feed-forward filter size. The fourth block reports results with fine-tuned Bert models: BertSumExt and its two variants (one without interval embeddings, and one with the large version of Bert), BertSumAbs, and BertSumExtAbs. Bert-based models outperform the Lead-3 baseline which is not a strawman; on the CNN/DailyMail corpus it is indeed superior to several extractive BIBREF7, BIBREF8, BIBREF19 and abstractive models BIBREF6. Bert models collectively outperform all previously proposed extractive and abstractive systems, only falling behind the Oracle upper bound. Among Bert variants, BertSumExt performs best which is not entirely surprising; CNN/DailyMail summaries are somewhat extractive and even abstractive models are prone to copying sentences from the source document when trained on this dataset BIBREF6. Perhaps unsurprisingly we observe that larger versions of Bert lead to performance improvements and that interval embeddings bring only slight gains. Table TABREF24 presents results on the NYT dataset. Following the evaluation protocol in BIBREF27, we use limited-length ROUGE Recall, where predicted summaries are truncated to the length of the gold summaries. Again, we report the performance of the Oracle upper bound and Lead-3 baseline. The second block in the table contains previously proposed extractive models as well as our own Transformer baseline. Compress BIBREF27 is an ILP-based model which combines compression and anaphoricity constraints. The third block includes abstractive models from the literature, and our Transformer baseline. Bert-based models are shown in the fourth block. Again, we observe that they outperform previously proposed approaches. On this dataset, abstractive Bert models generally perform better compared to BertSumExt, almost approaching Oracle performance. Table TABREF26 summarizes our results on the XSum dataset. Recall that summaries in this dataset are highly abstractive (see Table TABREF12) consisting of a single sentence conveying the gist of the document. Extractive models here perform poorly as corroborated by the low performance of the Lead baseline (which simply selects the leading sentence from the document), and the Oracle (which selects a single-best sentence in each document) in Table TABREF26. As a result, we do not report results for extractive models on this dataset. The second block in Table TABREF26 presents the results of various abstractive models taken from BIBREF22 and also includes our own abstractive Transformer baseline. In the third block we show the results of our Bert summarizers which again are superior to all previously reported models (by a wide margin). Results ::: Model Analysis ::: Learning Rates Recall that our abstractive model uses separate optimizers for the encoder and decoder. In Table TABREF27 we examine whether the combination of different learning rates ($\tilde{lr}_{\mathcal {E}}$ and $\tilde{lr}_{\mathcal {D}}$) is indeed beneficial. Specifically, we report model perplexity on the CNN/DailyMail validation set for varying encoder/decoder learning rates. We can see that the model performs best with $\tilde{lr}_{\mathcal {E}}=2e-3$ and $\tilde{lr}_{\mathcal {D}}=0.1$. Results ::: Model Analysis ::: Position of Extracted Sentences In addition to the evaluation based on ROUGE, we also analyzed in more detail the summaries produced by our model. For the extractive setting, we looked at the position (in the source document) of the sentences which were selected to appear in the summary. Figure FIGREF31 shows the proportion of selected summary sentences which appear in the source document at positions 1, 2, and so on. The analysis was conducted on the CNN/DailyMail dataset for Oracle summaries, and those produced by BertSumExt and the TransformerExt. We can see that Oracle summary sentences are fairly smoothly distributed across documents, while summaries created by TransformerExt mostly concentrate on the first document sentences. BertSumExt outputs are more similar to Oracle summaries, indicating that with the pretrained encoder, the model relies less on shallow position features, and learns deeper document representations. Results ::: Model Analysis ::: Novel N-grams We also analyzed the output of abstractive systems by calculating the proportion of novel n-grams that appear in the summaries but not in the source texts. The results are shown in Figure FIGREF33. In the CNN/DailyMail dataset, the proportion of novel n-grams in automatically generated summaries is much lower compared to reference summaries, but in XSum, this gap is much smaller. We also observe that on CNN/DailyMail, BertExtAbs produces less novel n-ngrams than BertAbs, which is not surprising. BertExtAbs is more biased towards selecting sentences from the source document since it is initially trained as an extractive model. The supplementary material includes examples of system output and additional ablation studies. Results ::: Human Evaluation In addition to automatic evaluation, we also evaluated system output by eliciting human judgments. We report experiments following a question-answering (QA) paradigm BIBREF33, BIBREF8 which quantifies the degree to which summarization models retain key information from the document. Under this paradigm, a set of questions is created based on the gold summary under the assumption that it highlights the most important document content. Participants are then asked to answer these questions by reading system summaries alone without access to the article. The more questions a system can answer, the better it is at summarizing the document as a whole. Moreover, we also assessed the overall quality of the summaries produced by abstractive systems which due to their ability to rewrite content may produce disfluent or ungrammatical output. Specifically, we followed the Best-Worst Scaling BIBREF34 method where participants were presented with the output of two systems (and the original document) and asked to decide which one was better according to the criteria of Informativeness, Fluency, and Succinctness. Both types of evaluation were conducted on the Amazon Mechanical Turk platform. For the CNN/DailyMail and NYT datasets we used the same documents (20 in total) and questions from previous work BIBREF8, BIBREF18. For XSum, we randomly selected 20 documents (and their questions) from the release of BIBREF22. We elicited 3 responses per HIT. With regard to QA evaluation, we adopted the scoring mechanism from BIBREF33; correct answers were marked with a score of one, partially correct answers with 0.5, and zero otherwise. For quality-based evaluation, the rating of each system was computed as the percentage of times it was chosen as better minus the times it was selected as worse. Ratings thus range from -1 (worst) to 1 (best). Results for extractive and abstractive systems are shown in Tables TABREF37 and TABREF38, respectively. We compared the best performing BertSum model in each setting (extractive or abstractive) against various state-of-the-art systems (whose output is publicly available), the Lead baseline, and the Gold standard as an upper bound. As shown in both tables participants overwhelmingly prefer the output of our model against comparison systems across datasets and evaluation paradigms. All differences between BertSum and comparison models are statistically significant ($p<0.05$), with the exception of TConvS2S (see Table TABREF38; XSum) in the QA evaluation setting. Conclusions In this paper, we showcased how pretrained Bert can be usefully applied in text summarization. We introduced a novel document-level encoder and proposed a general framework for both abstractive and extractive summarization. Experimental results across three datasets show that our model achieves state-of-the-art results across the board under automatic and human-based evaluation protocols. Although we mainly focused on document encoding for summarization, in the future, we would like to take advantage the capabilities of Bert for language generation. Acknowledgments This research is supported by a Google PhD Fellowship to the first author. We gratefully acknowledge the support of the European Research Council (Lapata, award number 681760, “Translating Multiple Modalities into Text”). We would also like to thank Shashi Narayan for providing us with the XSum dataset.
What rouge score do they achieve?
c17b609b0b090d7e8f99de1445be04f8f66367d4
What Ġrou ge Ġscore Ġdo Ġthey Ġachieve ?
Best Ġresults Ġon Ġun ig ram : Ċ CNN / Daily ĠMail : ĠRogue ĠF 1 Ġ43 . 85 Ċ NY T : ĠRogue ĠRecall Ġ49 . 02 Ċ X Sum : ĠRogue ĠF 1 Ġ38 . 81
<s> What Ġrou ge Ġscore Ġdo Ġthey Ġachieve ? </s> Introduction </s> Language Ġmodel Ġpret raining Ġhas Ġadvanced Ġthe Ġstate Ġof Ġthe Ġart Ġin Ġmany ĠN LP Ġtasks Ġranging Ġfrom Ġsentiment Ġanalysis , Ġto Ġquestion Ġanswering , Ġnatural Ġlanguage Ġinference , Ġnamed Ġentity Ġrecognition , Ġand Ġtextual Ġsimilarity . ĠState - of - the - art Ġpret rained Ġmodels Ġinclude ĠEL Mo ĠB IB REF 1 , ĠG PT ĠB IB REF 2 , Ġand Ġmore Ġrecently ĠBid irection al ĠEnc oder ĠRepresent ations Ġfrom ĠTransformers Ġ( B ert ; ĠB IB REF 0 ). ĠBert Ġcombines Ġboth Ġword Ġand Ġsentence Ġrepresentations Ġin Ġa Ġsingle Ġvery Ġlarge ĠTrans former ĠB IB REF 3 ; Ġit Ġis Ġpret rained Ġon Ġvast Ġamounts Ġof Ġtext , Ġwith Ġan Ġun super vised Ġobjective Ġof Ġmasked Ġlanguage Ġmodeling Ġand Ġnext - sent ence Ġprediction Ġand Ġcan Ġbe Ġfine - tun ed Ġwith Ġvarious Ġtask - specific Ġobjectives . </s> In Ġmost Ġcases , Ġpret rained Ġlanguage Ġmodels Ġhave Ġbeen Ġemployed Ġas Ġenc od ers Ġfor Ġsentence - Ġand Ġparagraph - level Ġnatural Ġlanguage Ġunderstanding Ġproblems ĠB IB REF 0 Ġinvolving Ġvarious Ġclassification Ġtasks Ġ( e . g ., Ġpredicting Ġwhether Ġany Ġtwo Ġsentences Ġare Ġin Ġan Ġentail ment Ġrelationship ; Ġor Ġdetermining Ġthe Ġcompletion Ġof Ġa Ġsentence Ġamong Ġfour Ġalternative Ġsentences ). ĠIn Ġthis Ġpaper , Ġwe Ġexamine Ġthe Ġinfluence Ġof Ġlanguage Ġmodel Ġpret raining Ġon Ġtext Ġsummar ization . ĠDifferent Ġfrom Ġprevious Ġtasks , Ġsummar ization Ġrequires Ġwide - co verage Ġnatural Ġlanguage Ġunderstanding Ġgoing Ġbeyond Ġthe Ġmeaning Ġof Ġindividual Ġwords Ġand Ġsentences . ĠThe Ġaim Ġis Ġto Ġcond ense Ġa Ġdocument Ġinto Ġa Ġshorter Ġversion Ġwhile Ġpreserving Ġmost Ġof Ġits Ġmeaning . ĠFurthermore , Ġunder Ġabstract ive Ġmodeling Ġformulations , Ġthe Ġtask Ġrequires Ġlanguage Ġgeneration Ġcapabilities Ġin Ġorder Ġto Ġcreate Ġsumm aries Ġcontaining Ġnovel Ġwords Ġand Ġphrases Ġnot Ġfeatured Ġin Ġthe Ġsource Ġtext , Ġwhile Ġextract ive Ġsummar ization Ġis Ġoften Ġdefined Ġas Ġa Ġbinary Ġclassification Ġtask Ġwith Ġlabels Ġindicating Ġwhether Ġa Ġtext Ġspan Ġ( typically Ġa Ġsentence ) Ġshould Ġbe Ġincluded Ġin Ġthe Ġsummary . </s> We Ġexplore Ġthe Ġpotential Ġof ĠBert Ġfor Ġtext Ġsummar ization Ġunder Ġa Ġgeneral Ġframework Ġencomp assing Ġboth Ġextract ive Ġand Ġabstract ive Ġmodeling Ġparad ig ms . ĠWe Ġpropose Ġa Ġnovel Ġdocument - level Ġenc oder Ġbased Ġon ĠBert Ġwhich Ġis Ġable Ġto Ġencode Ġa Ġdocument Ġand Ġobtain Ġrepresentations Ġfor Ġits Ġsentences . ĠOur Ġextract ive Ġmodel Ġis Ġbuilt Ġon Ġtop Ġof Ġthis Ġenc oder Ġby Ġstacking Ġseveral Ġinter - sent ence ĠTrans former Ġlayers Ġto Ġcapture Ġdocument - level Ġfeatures Ġfor Ġextracting Ġsentences . ĠOur Ġabstract ive Ġmodel Ġadop ts Ġan Ġenc oder - dec oder Ġarchitecture , Ġcombining Ġthe Ġsame Ġpret rained ĠBert Ġenc oder Ġwith Ġa Ġrandomly - initialized ĠTrans former Ġdec oder ĠB IB REF 3 . ĠWe Ġdesign Ġa Ġnew Ġtraining Ġschedule Ġwhich Ġseparates Ġthe Ġoptim izers Ġof Ġthe Ġenc oder Ġand Ġthe Ġdec oder Ġin Ġorder Ġto Ġaccommodate Ġthe Ġfact Ġthat Ġthe Ġformer Ġis Ġpret rained Ġwhile Ġthe Ġlatter Ġmust Ġbe Ġtrained Ġfrom Ġscratch . ĠFinally , Ġmotivated Ġby Ġprevious Ġwork Ġshowing Ġthat Ġthe Ġcombination Ġof Ġextract ive Ġand Ġabstract ive Ġobjectives Ġcan Ġhelp Ġgenerate Ġbetter Ġsumm aries ĠB IB REF 4 , Ġwe Ġpresent Ġa Ġtwo - stage Ġapproach Ġwhere Ġthe Ġenc oder Ġis Ġfine - tun ed Ġtwice , Ġfirst Ġwith Ġan Ġextract ive Ġobjective Ġand Ġsubsequently Ġon Ġthe Ġabstract ive Ġsummar ization Ġtask . </s> We Ġevaluate Ġthe Ġproposed Ġapproach Ġon Ġthree Ġsingle - document Ġnews Ġsummar ization Ġdatasets Ġrepresentative Ġof Ġdifferent Ġwriting Ġconventions Ġ( e . g ., Ġimportant Ġinformation Ġis Ġconcentrated Ġat Ġthe Ġbeginning Ġof Ġthe Ġdocument Ġor Ġdistributed Ġmore Ġevenly Ġthroughout ) Ġand Ġsummary Ġstyles Ġ( e . g ., Ġverb ose Ġvs . Ġmore Ġte legraph ic ; Ġextract ive Ġvs . Ġabstract ive ). ĠAcross Ġdatasets , Ġwe Ġexperiment ally Ġshow Ġthat Ġthe Ġproposed Ġmodels Ġachieve Ġstate - of - the - art Ġresults Ġunder Ġboth Ġextract ive Ġand Ġabstract ive Ġsettings . ĠOur Ġcontributions Ġin Ġthis Ġwork Ġare Ġthree - fold : Ġa ) Ġwe Ġhighlight Ġthe Ġimportance Ġof Ġdocument Ġencoding Ġfor Ġthe Ġsummar ization Ġtask ; Ġa Ġvariety Ġof Ġrecently Ġproposed Ġtechniques Ġaim Ġto Ġenhance Ġsummar ization Ġperformance Ġvia Ġcopying Ġmechanisms ĠB IB REF 5 , ĠB IB REF 6 , ĠB IB REF 7 , Ġreinforcement Ġlearning ĠB IB REF 8 , ĠB IB REF 9 , ĠB IB REF 10 , Ġand Ġmultiple Ġcommunicating Ġenc od ers ĠB IB REF 11 . ĠWe Ġachieve Ġbetter Ġresults Ġwith Ġa Ġminimum - requ irement Ġmodel Ġwithout Ġusing Ġany Ġof Ġthese Ġmechanisms ; Ġb ) Ġwe Ġshowcase Ġways Ġto Ġeffectively Ġemploy Ġpret rained Ġlanguage Ġmodels Ġin Ġsummar ization Ġunder Ġboth Ġextract ive Ġand Ġabstract ive Ġsettings ; Ġwe Ġwould Ġexpect Ġany Ġimprovements Ġin Ġmodel Ġpret raining Ġto Ġtranslate Ġin Ġbetter Ġsummar ization Ġin Ġthe Ġfuture ; Ġand Ġc ) Ġthe Ġproposed Ġmodels Ġcan Ġbe Ġused Ġas Ġa Ġstepping Ġstone Ġto Ġfurther Ġimprove Ġsummar ization Ġperformance Ġas Ġwell Ġas Ġbas elines Ġagainst Ġwhich Ġnew Ġproposals Ġare Ġtested . </s> Background Ġ: :: ĠPret rained ĠLanguage ĠModels </s> P ret rained Ġlanguage Ġmodels ĠB IB REF 1 , ĠB IB REF 2 , ĠB IB REF 0 , ĠB IB REF 12 , ĠB IB REF 13 Ġhave Ġrecently Ġemerged Ġas Ġa Ġkey Ġtechnology Ġfor Ġachieving Ġimpressive Ġgains Ġin Ġa Ġwide Ġvariety Ġof Ġnatural Ġlanguage Ġtasks . ĠThese Ġmodels Ġextend Ġthe Ġidea Ġof Ġword Ġembed d ings Ġby Ġlearning Ġcontextual Ġrepresentations Ġfrom Ġlarge - scale Ġcorpor a Ġusing Ġa Ġlanguage Ġmodeling Ġobjective . ĠBid irection al ĠEnc oder ĠRepresent ations Ġfrom ĠTransformers Ġ( B ert ; ĠB IB REF 0 ) Ġis Ġa Ġnew Ġlanguage Ġrepresentation Ġmodel Ġwhich Ġis Ġtrained Ġwith Ġa Ġmasked Ġlanguage Ġmodeling Ġand Ġa ĠâĢ ľ next Ġsentence Ġprediction âĢ Ŀ Ġtask Ġon Ġa Ġcorpus Ġof Ġ3 , 300 M Ġwords . </s> The Ġgeneral Ġarchitecture Ġof ĠBert Ġis Ġshown Ġin Ġthe Ġleft Ġpart Ġof ĠFigure ĠFIG REF 2 . ĠInput Ġtext Ġis Ġfirst Ġpre process ed Ġby Ġinserting Ġtwo Ġspecial Ġtokens . Ġ[ cl s ] Ġis Ġapp ended Ġto Ġthe Ġbeginning Ġof Ġthe Ġtext ; Ġthe Ġoutput Ġrepresentation Ġof Ġthis Ġtoken Ġis Ġused Ġto Ġaggregate Ġinformation Ġfrom Ġthe Ġwhole Ġsequence Ġ( e . g ., Ġfor Ġclassification Ġtasks ). ĠAnd Ġtoken Ġ[ se p ] Ġis Ġinserted Ġafter Ġeach Ġsentence Ġas Ġan Ġindicator Ġof Ġsentence Ġboundaries . ĠThe Ġmodified Ġtext Ġis Ġthen Ġrepresented Ġas Ġa Ġsequence Ġof Ġtokens Ġ$ X =[ w _ 1 , w _ 2 , \ cd ots Ġ, w _ n ] $. ĠEach Ġtoken Ġ$ w _ i $ Ġis Ġassigned Ġthree Ġkinds Ġof Ġembed d ings : Ġtoken Ġembed d ings Ġindicate Ġthe Ġmeaning Ġof Ġeach Ġtoken , Ġsegment ation Ġembed d ings Ġare Ġused Ġto Ġdiscriminate Ġbetween Ġtwo Ġsentences Ġ( e . g ., Ġduring Ġa Ġsentence - pair Ġclassification Ġtask ) Ġand Ġposition Ġembed d ings Ġindicate Ġthe Ġposition Ġof Ġeach Ġtoken Ġwithin Ġthe Ġtext Ġsequence . ĠThese Ġthree Ġembed d ings Ġare Ġsummed Ġto Ġa Ġsingle Ġinput Ġvector Ġ$ x _ i $ Ġand Ġfed Ġto Ġa Ġbid irection al ĠTrans former Ġwith Ġmultiple Ġlayers : </s> where Ġ$ h ^ 0 = x $ Ġare Ġthe Ġinput Ġvectors ; Ġ$\ math rm Ġ{ L N } $ Ġis Ġthe Ġlayer Ġnormal ization Ġoperation ĠB IB REF 14 ; Ġ$\ math rm Ġ{ M HA tt } $ Ġis Ġthe Ġmulti - head Ġattention Ġoperation ĠB IB REF 3 ; Ġsupers cript Ġ$ l $ Ġindicates Ġthe Ġdepth Ġof Ġthe Ġstacked Ġlayer . ĠOn Ġthe Ġtop Ġlayer , ĠBert Ġwill Ġgenerate Ġan Ġoutput Ġvector Ġ$ t _ i $ Ġfor Ġeach Ġtoken Ġwith Ġrich Ġcontextual Ġinformation . </s> P ret rained Ġlanguage Ġmodels Ġare Ġusually Ġused Ġto Ġenhance Ġperformance Ġin Ġlanguage Ġunderstanding Ġtasks . ĠVery Ġrecently , Ġthere Ġhave Ġbeen Ġattempts Ġto Ġapply Ġpret rained Ġmodels Ġto Ġvarious Ġgeneration Ġproblems ĠB IB REF 15 , ĠB IB REF 16 . ĠWhen Ġfine - tun ing Ġfor Ġa Ġspecific Ġtask , Ġunlike ĠEL Mo Ġwhose Ġparameters Ġare Ġusually Ġfixed , Ġparameters Ġin ĠBert Ġare Ġjointly Ġfine - tun ed Ġwith Ġadditional Ġtask - specific Ġparameters . </s> Background Ġ: :: ĠExtract ive ĠSum mar ization </s> Ext ractive Ġsummar ization Ġsystems Ġcreate Ġa Ġsummary Ġby Ġidentifying Ġ( and Ġsubsequently Ġconc aten ating ) Ġthe Ġmost Ġimportant Ġsentences Ġin Ġa Ġdocument . ĠNeural Ġmodels Ġconsider Ġextract ive Ġsummar ization Ġas Ġa Ġsentence Ġclassification Ġproblem : Ġa Ġneural Ġenc oder Ġcreates Ġsentence Ġrepresentations Ġand Ġa Ġclass ifier Ġpredicts Ġwhich Ġsentences Ġshould Ġbe Ġselected Ġas Ġsumm aries . ĠSum ma Ru NN er ĠB IB REF 7 Ġis Ġone Ġof Ġthe Ġearliest Ġneural Ġapproaches Ġadopting Ġan Ġenc oder Ġbased Ġon ĠRec urrent ĠNeural ĠNetworks . ĠRefresh ĠB IB REF 8 Ġis Ġa Ġreinforcement Ġlearning - based Ġsystem Ġtrained Ġby Ġglobally Ġoptimizing Ġthe ĠR OU GE Ġmetric . ĠMore Ġrecent Ġwork Ġachieves Ġhigher Ġperformance Ġwith Ġmore Ġsophisticated Ġmodel Ġstructures . ĠLat ent ĠB IB REF 17 Ġframes Ġextract ive Ġsummar ization Ġas Ġa Ġlatent Ġvariable Ġinference Ġproblem ; Ġinstead Ġof Ġmaximizing Ġthe Ġlikelihood Ġof ĠâĢ ľ gold âĢ Ŀ Ġstandard Ġlabels , Ġtheir Ġlatent Ġmodel Ġdirectly Ġmaxim izes Ġthe Ġlikelihood Ġof Ġhuman Ġsumm aries Ġgiven Ġselected Ġsentences . ĠSum o ĠB IB REF 18 Ġcapital izes Ġon Ġthe Ġnotion Ġof Ġstructured Ġattention Ġto Ġinduce Ġa Ġmulti - root Ġdependency Ġtree Ġrepresentation Ġof Ġthe Ġdocument Ġwhile Ġpredicting Ġthe Ġoutput Ġsummary . ĠNe u Sum ĠB IB REF 19 Ġscores Ġand Ġselects Ġsentences Ġjointly Ġand Ġrepresents Ġthe Ġstate Ġof Ġthe Ġart Ġin Ġextract ive Ġsummar ization . </s> Background Ġ: :: ĠAbstract ive ĠSum mar ization </s> Ne ural Ġapproaches Ġto Ġabstract ive Ġsummar ization Ġconceptual ize Ġthe Ġtask Ġas Ġa Ġsequence - to - sequence Ġproblem , Ġwhere Ġan Ġenc oder Ġmaps Ġa Ġsequence Ġof Ġtokens Ġin Ġthe Ġsource Ġdocument Ġ$\ math bf Ġ{ x } Ġ= Ġ[ x _ 1 , Ġ... , Ġx _ n ] $ Ġto Ġa Ġsequence Ġof Ġcontinuous Ġrepresentations Ġ$\ math bf Ġ{ z } Ġ= Ġ[ z _ 1 , Ġ... , Ġz _ n ] $, Ġand Ġa Ġdec oder Ġthen Ġgenerates Ġthe Ġtarget Ġsummary Ġ$\ math bf Ġ{ y } Ġ= Ġ[ y _ 1 , Ġ... , Ġy _ m ] $ Ġtoken - by - token , Ġin Ġan Ġauto - reg ressive Ġmanner , Ġhence Ġmodeling Ġthe Ġconditional Ġprobability : Ġ$ p ( y _ 1 , Ġ... , Ġy _ m | x _ 1 , Ġ... , Ġx _ n ) $. </s> BI B REF 20 Ġand ĠB IB REF 21 Ġwere Ġamong Ġthe Ġfirst Ġto Ġapply Ġthe Ġneural Ġenc oder - dec oder Ġarchitecture Ġto Ġtext Ġsummar ization . ĠB IB REF 6 Ġenhance Ġthis Ġmodel Ġwith Ġa Ġpointer - gener ator Ġnetwork Ġ( PT gen ) Ġwhich Ġallows Ġit Ġto Ġcopy Ġwords Ġfrom Ġthe Ġsource Ġtext , Ġand Ġa Ġcoverage Ġmechanism Ġ( C ov ) Ġwhich Ġkeeps Ġtrack Ġof Ġwords Ġthat Ġhave Ġbeen Ġsummarized . ĠB IB REF 11 Ġpropose Ġan Ġabstract ive Ġsystem Ġwhere Ġmultiple Ġagents Ġ( enc od ers ) Ġrepresent Ġthe Ġdocument Ġtogether Ġwith Ġa Ġhierarchical Ġattention Ġmechanism Ġ( over Ġthe Ġagents ) Ġfor Ġdecoding . ĠTheir ĠDeep ĠCommun icating ĠAgents Ġ( D CA ) Ġmodel Ġis Ġtrained Ġend - to - end Ġwith Ġreinforcement Ġlearning . ĠB IB REF 9 Ġalso Ġpresent Ġa Ġdeep Ġreinforced Ġmodel Ġ( DR M ) Ġfor Ġabstract ive Ġsummar ization Ġwhich Ġhandles Ġthe Ġcoverage Ġproblem Ġwith Ġan Ġintra - att ention Ġmechanism Ġwhere Ġthe Ġdec oder Ġattends Ġover Ġpreviously Ġgenerated Ġwords . ĠB IB REF 4 Ġfollow Ġa Ġbottom - up Ġapproach Ġ( Bottom Up ); Ġa Ġcontent Ġselector Ġfirst Ġdetermines Ġwhich Ġphrases Ġin Ġthe Ġsource Ġdocument Ġshould Ġbe Ġpart Ġof Ġthe Ġsummary , Ġand Ġa Ġcopy Ġmechanism Ġis Ġapplied Ġonly Ġto Ġpre selected Ġphrases Ġduring Ġdecoding . ĠB IB REF 22 Ġpropose Ġan Ġabstract ive Ġmodel Ġwhich Ġis Ġparticularly Ġsuited Ġto Ġextreme Ġsummar ization Ġ( i . e ., Ġsingle Ġsentence Ġsumm aries ), Ġbased Ġon Ġconv olution al Ġneural Ġnetworks Ġand Ġadditionally Ġconditioned Ġon Ġtopic Ġdistributions Ġ( T Con v S 2 S ). </s> Fine - tun ing ĠBert Ġfor ĠSum mar ization Ġ: :: ĠSum mar ization ĠEnc oder </s> Although ĠBert Ġhas Ġbeen Ġused Ġto Ġfine - t une Ġvarious ĠN LP Ġtasks , Ġits Ġapplication Ġto Ġsummar ization Ġis Ġnot Ġas Ġstraightforward . ĠSince ĠBert Ġis Ġtrained Ġas Ġa Ġmasked - language Ġmodel , Ġthe Ġoutput Ġvectors Ġare Ġgrounded Ġto Ġtokens Ġinstead Ġof Ġsentences , Ġwhile Ġin Ġextract ive Ġsummar ization , Ġmost Ġmodels Ġmanipulate Ġsentence - level Ġrepresentations . ĠAlthough Ġsegment ation Ġembed d ings Ġrepresent Ġdifferent Ġsentences Ġin ĠBert , Ġthey Ġonly Ġapply Ġto Ġsentence - pair Ġinputs , Ġwhile Ġin Ġsummar ization Ġwe Ġmust Ġencode Ġand Ġmanipulate Ġmulti - sent ential Ġinputs . ĠFigure ĠFIG REF 2 Ġillustrates Ġour Ġproposed ĠBert Ġarchitecture Ġfor ĠSum mar ization Ġ( which Ġwe Ġcall ĠBert Sum ). </s> In Ġorder Ġto Ġrepresent Ġindividual Ġsentences , Ġwe Ġinsert Ġexternal Ġ[ cl s ] Ġtokens Ġat Ġthe Ġstart Ġof Ġeach Ġsentence , Ġand Ġeach Ġ[ cl s ] Ġsymbol Ġcollects Ġfeatures Ġfor Ġthe Ġsentence Ġpreceding Ġit . ĠWe Ġalso Ġuse Ġinterval Ġsegment Ġembed d ings Ġto Ġdistinguish Ġmultiple Ġsentences Ġwithin Ġa Ġdocument . ĠFor Ġ$ sent _ i $ Ġwe Ġassign Ġsegment Ġembed ding Ġ$ E _ A $ Ġor Ġ$ E _ B $ Ġdepending Ġon Ġwhether Ġ$ i $ Ġis Ġodd Ġor Ġeven . ĠFor Ġexample , Ġfor Ġdocument Ġ$ [ sent _ 1 , Ġsent _ 2 , Ġsent _ 3 , Ġsent _ 4 , Ġsent _ 5 ] $, Ġwe Ġwould Ġassign Ġembed d ings Ġ$ [ E _ A , ĠE _ B , ĠE _ A , E _ B , ĠE _ A ] $. ĠThis Ġway , Ġdocument Ġrepresentations Ġare Ġlearned Ġhierarch ically Ġwhere Ġlower ĠTrans former Ġlayers Ġrepresent Ġadjacent Ġsentences , Ġwhile Ġhigher Ġlayers , Ġin Ġcombination Ġwith Ġself - att ention , Ġrepresent Ġmulti - sent ence Ġdiscourse . </s> Position Ġembed d ings Ġin Ġthe Ġoriginal ĠBert Ġmodel Ġhave Ġa Ġmaximum Ġlength Ġof Ġ512 ; Ġwe Ġovercome Ġthis Ġlimitation Ġby Ġadding Ġmore Ġposition Ġembed d ings Ġthat Ġare Ġinitialized Ġrandomly Ġand Ġfine - tun ed Ġwith Ġother Ġparameters Ġin Ġthe Ġenc oder . </s> Fine - tun ing ĠBert Ġfor ĠSum mar ization Ġ: :: ĠExtract ive ĠSum mar ization </s> Let Ġ$ d $ Ġdenote Ġa Ġdocument Ġcontaining Ġsentences Ġ$ [ sent _ 1 , Ġsent _ 2 , Ġ\ cd ots Ġ, Ġsent _ m ] $, Ġwhere Ġ$ sent _ i $ Ġis Ġthe Ġ$ i $ - th Ġsentence Ġin Ġthe Ġdocument . ĠExtract ive Ġsummar ization Ġcan Ġbe Ġdefined Ġas Ġthe Ġtask Ġof Ġassigning Ġa Ġlabel Ġ$ y _ i Ġ\ in Ġ\ l brace Ġ0 , Ġ1 \ r brace Ġ$ Ġto Ġeach Ġ$ sent _ i $, Ġindicating Ġwhether Ġthe Ġsentence Ġshould Ġbe Ġincluded Ġin Ġthe Ġsummary . ĠIt Ġis Ġassumed Ġthat Ġsummary Ġsentences Ġrepresent Ġthe Ġmost Ġimportant Ġcontent Ġof Ġthe Ġdocument . </s> With ĠBert Sum , Ġvector Ġ$ t _ i $ Ġwhich Ġis Ġthe Ġvector Ġof Ġthe Ġ$ i $ - th Ġ[ cl s ] Ġsymbol Ġfrom Ġthe Ġtop Ġlayer Ġcan Ġbe Ġused Ġas Ġthe Ġrepresentation Ġfor Ġ$ sent _ i $. ĠSeveral Ġinter - sent ence ĠTrans former Ġlayers Ġare Ġthen Ġstacked Ġon Ġtop Ġof ĠBert Ġoutputs , Ġto Ġcapture Ġdocument - level Ġfeatures Ġfor Ġextracting Ġsumm aries : </s> where Ġ$ h ^ 0 = \ math rm Ġ{ Pos Emb } ( T ) $ ; Ġ$ T $ Ġdenotes Ġthe Ġsentence Ġvectors Ġoutput Ġby ĠBert Sum , Ġand Ġfunction Ġ$\ math rm Ġ{ Pos Emb } $ Ġadds Ġsin us oid Ġpositional Ġembed d ings ĠB IB REF 3 Ġto Ġ$ T $, Ġindicating Ġthe Ġposition Ġof Ġeach Ġsentence . </s> The Ġfinal Ġoutput Ġlayer Ġis Ġa Ġs igm oid Ġclass ifier : </s> where Ġ$ h ^ L _ i $ Ġis Ġthe Ġvector Ġfor Ġ$ sent _ i $ Ġfrom Ġthe Ġtop Ġlayer Ġ( the Ġ$ L $ - th Ġlayer Ġ) Ġof Ġthe ĠTrans former . ĠIn Ġexperiments , Ġwe Ġimplemented ĠTransformers Ġwith Ġ$ L = 1 , Ġ2 , Ġ3 $ Ġand Ġfound Ġthat Ġa ĠTrans former Ġwith Ġ$ L = 2 $ Ġperformed Ġbest . ĠWe Ġname Ġthis Ġmodel ĠBert Sum Ext . </s> The Ġloss Ġof Ġthe Ġmodel Ġis Ġthe Ġbinary Ġclassification Ġentropy Ġof Ġprediction Ġ$\ hat { y } _ i $ Ġagainst Ġgold Ġlabel Ġ$ y _ i $. ĠInter - sent ence ĠTrans former Ġlayers Ġare Ġjointly Ġfine - tun ed Ġwith ĠBert Sum . ĠWe Ġuse Ġthe ĠAdam Ġoptim izer Ġwith Ġ$\ beta Ġ_ 1 = 0 . 9 $, Ġand Ġ$\ beta Ġ_ 2 = 0 . 999 $ ). ĠOur Ġlearning Ġrate Ġschedule Ġfollows ĠB IB REF 3 Ġwith Ġwarming - up Ġ($ Ġ\ oper at orn ame {\ oper at orn ame { warm up }} = 10 , 000 $ ): </s> Fine - tun ing ĠBert Ġfor ĠSum mar ization Ġ: :: ĠAbstract ive ĠSum mar ization </s> We Ġuse Ġa Ġstandard Ġenc oder - dec oder Ġframework Ġfor Ġabstract ive Ġsummar ization ĠB IB REF 6 . ĠThe Ġenc oder Ġis Ġthe Ġpret rained ĠBert Sum Ġand Ġthe Ġdec oder Ġis Ġa Ġ6 - lay ered ĠTrans former Ġinitialized Ġrandomly . ĠIt Ġis Ġconceivable Ġthat Ġthere Ġis Ġa Ġmismatch Ġbetween Ġthe Ġenc oder Ġand Ġthe Ġdec oder , Ġsince Ġthe Ġformer Ġis Ġpret rained Ġwhile Ġthe Ġlatter Ġmust Ġbe Ġtrained Ġfrom Ġscratch . ĠThis Ġcan Ġmake Ġfine - tun ing Ġunstable ; Ġfor Ġexample , Ġthe Ġenc oder Ġmight Ġover fit Ġthe Ġdata Ġwhile Ġthe Ġdec oder Ġunder fits , Ġor Ġvice Ġversa . ĠTo Ġcircumvent Ġthis , Ġwe Ġdesign Ġa Ġnew Ġfine - tun ing Ġschedule Ġwhich Ġseparates Ġthe Ġoptim izers Ġof Ġthe Ġenc oder Ġand Ġthe Ġdec oder . </s> We Ġuse Ġtwo ĠAdam Ġoptim izers Ġwith Ġ$\ beta Ġ_ 1 = 0 . 9 $ Ġand Ġ$\ beta Ġ_ 2 = 0 . 999 $ Ġfor Ġthe Ġenc oder Ġand Ġthe Ġdec oder , Ġrespectively , Ġeach Ġwith Ġdifferent Ġwarm up - steps Ġand Ġlearning Ġrates : </s> where Ġ$\ t ilde { lr } _{ \ math cal Ġ{ E }} = 2 e ^{ - 3 } $, Ġand Ġ$\ oper at orn ame {\ oper at orn ame { warm up }} _{ \ math cal Ġ{ E }} = 20 , 000 $ Ġfor Ġthe Ġenc oder Ġand Ġ$\ t ilde { lr } _{ \ math cal Ġ{ D }} = 0 . 1 $, Ġand Ġ$\ oper at orn ame {\ oper at orn ame { warm up }} _{ \ math cal Ġ{ D }} = 10 , 000 $ Ġfor Ġthe Ġdec oder . ĠThis Ġis Ġbased Ġon Ġthe Ġassumption Ġthat Ġthe Ġpret rained Ġenc oder Ġshould Ġbe Ġfine - tun ed Ġwith Ġa Ġsmaller Ġlearning Ġrate Ġand Ġsmoother Ġdecay Ġ( so Ġthat Ġthe Ġenc oder Ġcan Ġbe Ġtrained Ġwith Ġmore Ġaccurate Ġgrad ients Ġwhen Ġthe Ġdec oder Ġis Ġbecoming Ġstable ). </s> In Ġaddition , Ġwe Ġpropose Ġa Ġtwo - stage Ġfine - tun ing Ġapproach , Ġwhere Ġwe Ġfirst Ġfine - t une Ġthe Ġenc oder Ġon Ġthe Ġextract ive Ġsummar ization Ġtask Ġ( Section ĠSEC REF 8 ) Ġand Ġthen Ġfine - t une Ġit Ġon Ġthe Ġabstract ive Ġsummar ization Ġtask Ġ( Section ĠSEC REF 13 ). ĠPrevious Ġwork ĠB IB REF 4 , ĠB IB REF 23 Ġsuggests Ġthat Ġusing Ġextract ive Ġobjectives Ġcan Ġboost Ġthe Ġperformance Ġof Ġabstract ive Ġsummar ization . ĠAlso Ġnotice Ġthat Ġthis Ġtwo - stage Ġapproach Ġis Ġconcept ually Ġvery Ġsimple , Ġthe Ġmodel Ġcan Ġtake Ġadvantage Ġof Ġinformation Ġshared Ġbetween Ġthese Ġtwo Ġtasks , Ġwithout Ġfundamentally Ġchanging Ġits Ġarchitecture . ĠWe Ġname Ġthe Ġdefault Ġabstract ive Ġmodel ĠBert Sum Abs Ġand Ġthe Ġtwo - stage Ġfine - tun ed Ġmodel ĠBert Sum Ext Abs . </s> Exper imental ĠSetup </s> In Ġthis Ġsection , Ġwe Ġdescribe Ġthe Ġsummar ization Ġdatasets Ġused Ġin Ġour Ġexperiments Ġand Ġdiscuss Ġvarious Ġimplementation Ġdetails . </s> Exper imental ĠSetup Ġ: :: ĠSum mar ization ĠDat as ets </s> We Ġevaluated Ġour Ġmodel Ġon Ġthree Ġbenchmark Ġdatasets , Ġnamely Ġthe ĠCNN / Daily Mail Ġnews Ġhighlights Ġdataset ĠB IB REF 24 , Ġthe ĠNew ĠYork ĠTimes ĠAn not ated ĠCorpus Ġ( NY T ; ĠB IB REF 25 ), Ġand ĠX Sum ĠB IB REF 22 . ĠThese Ġdatasets Ġrepresent Ġdifferent Ġsummary Ġstyles Ġranging Ġfrom Ġhighlights Ġto Ġvery Ġbrief Ġone Ġsentence Ġsumm aries . ĠThe Ġsumm aries Ġalso Ġvary Ġwith Ġrespect Ġto Ġthe Ġtype Ġof Ġrewriting Ġoperations Ġthey Ġexempl ify Ġ( e . g ., Ġsome Ġshowcase Ġmore Ġcut Ġand Ġpaste Ġoperations Ġwhile Ġothers Ġare Ġgenuinely Ġabstract ive ). ĠTable ĠT AB REF 12 Ġpresents Ġstatistics Ġon Ġthese Ġdatasets Ġ( test Ġset ); Ġexample Ġ( gold - standard ) Ġsumm aries Ġare Ġprovided Ġin Ġthe Ġsupplementary Ġmaterial . </s> Exper imental ĠSetup Ġ: :: ĠSum mar ization ĠDat as ets Ġ: :: ĠCNN / Daily Mail </s> cont ains Ġnews Ġarticles Ġand Ġassociated Ġhighlights , Ġi . e ., Ġa Ġfew Ġbullet Ġpoints Ġgiving Ġa Ġbrief Ġoverview Ġof Ġthe Ġarticle . ĠWe Ġused Ġthe Ġstandard Ġsplits Ġof ĠB IB REF 24 Ġfor Ġtraining , Ġvalidation , Ġand Ġtesting Ġ( 90 , 266 / 1 , 220 / 1 , 09 3 ĠCNN Ġdocuments Ġand Ġ196 , 96 1 / 12 , 148 / 10 , 397 ĠDaily Mail Ġdocuments ). ĠWe Ġdid Ġnot Ġanonym ize Ġentities . ĠWe Ġfirst Ġsplit Ġsentences Ġwith Ġthe ĠStanford ĠCore N LP Ġtool kit ĠB IB REF 26 Ġand Ġpre - process ed Ġthe Ġdataset Ġfollowing ĠB IB REF 6 . ĠInput Ġdocuments Ġwere Ġtrunc ated Ġto Ġ512 Ġtokens . </s> Exper imental ĠSetup Ġ: :: ĠSum mar ization ĠDat as ets Ġ: :: ĠNYT </s> cont ains Ġ110 , 540 Ġarticles Ġwith Ġabstract ive Ġsumm aries . ĠFollowing ĠB IB REF 27 , Ġwe Ġsplit Ġthese Ġinto Ġ100 , 8 34 / 9 , 706 Ġtraining / test Ġexamples , Ġbased Ġon Ġthe Ġdate Ġof Ġpublication Ġ( the Ġtest Ġset Ġcontains Ġall Ġarticles Ġpublished Ġfrom ĠJanuary Ġ1 , Ġ2007 Ġonward ). ĠWe Ġused Ġ4 , 000 Ġexamples Ġfrom Ġthe Ġtraining Ġas Ġvalidation Ġset . ĠWe Ġalso Ġfollowed Ġtheir Ġfiltering Ġprocedure , Ġdocuments Ġwith Ġsumm aries Ġless Ġthan Ġ50 Ġwords Ġwere Ġremoved Ġfrom Ġthe Ġdataset . ĠThe Ġfiltered Ġtest Ġset Ġ( NY T 50 ) Ġincludes Ġ3 , 452 Ġexamples . ĠSent ences Ġwere Ġsplit Ġwith Ġthe ĠStanford ĠCore N LP Ġtool kit ĠB IB REF 26 Ġand Ġpre - process ed Ġfollowing ĠB IB REF 27 . ĠInput Ġdocuments Ġwere Ġtrunc ated Ġto Ġ800 Ġtokens . </s> Exper imental ĠSetup Ġ: :: ĠSum mar ization ĠDat as ets Ġ: :: ĠX Sum </s> cont ains Ġ226 , 7 11 Ġnews Ġarticles Ġaccompanied Ġwith Ġa Ġone - sent ence Ġsummary , Ġanswering Ġthe Ġquestion ĠâĢ ľ What Ġis Ġthis Ġarticle Ġabout ? âĢ Ŀ . ĠWe Ġused Ġthe Ġsplits Ġof ĠB IB REF 22 Ġfor Ġtraining , Ġvalidation , Ġand Ġtesting Ġ( 204 , 045 / 11 , 332 / 11 , 334 ) Ġand Ġfollowed Ġthe Ġpre - processing Ġintroduced Ġin Ġtheir Ġwork . ĠInput Ġdocuments Ġwere Ġtrunc ated Ġto Ġ512 Ġtokens . </s> Aside Ġfrom Ġvarious Ġstatistics Ġon Ġthe Ġthree Ġdatasets , ĠTable ĠT AB REF 12 Ġalso Ġreports Ġthe Ġproportion Ġof Ġnovel Ġbi - gram s Ġin Ġgold Ġsumm aries Ġas Ġa Ġmeasure Ġof Ġtheir Ġabstract iveness . ĠWe Ġwould Ġexpect Ġmodels Ġwith Ġextract ive Ġbiases Ġto Ġperform Ġbetter Ġon Ġdatasets Ġwith Ġ( mostly ) Ġextract ive Ġsumm aries , Ġand Ġabstract ive Ġmodels Ġto Ġperform Ġmore Ġrewrite Ġoperations Ġon Ġdatasets Ġwith Ġabstract ive Ġsumm aries . ĠCNN / Daily Mail Ġand ĠNYT Ġare Ġsomewhat Ġabstract ive , Ġwhile ĠX Sum Ġis Ġhighly Ġabstract ive . </s> Exper imental ĠSetup Ġ: :: ĠImplementation ĠDetails </s> For Ġboth Ġextract ive Ġand Ġabstract ive Ġsettings , Ġwe Ġused ĠPy Tor ch , ĠOpen N MT ĠB IB REF 28 Ġand Ġthe Ġ` bert - base - unc ased ' Ġversion Ġof ĠBert Ġto Ġimplement ĠBert Sum . ĠBoth Ġsource Ġand Ġtarget Ġtexts Ġwere Ġtoken ized Ġwith ĠBert 's Ġsub words Ġtoken izer . </s> Exper imental ĠSetup Ġ: :: ĠImplementation ĠDetails Ġ: :: ĠExtract ive ĠSum mar ization </s> All Ġextract ive Ġmodels Ġwere Ġtrained Ġfor Ġ50 , 000 Ġsteps Ġon Ġ3 ĠGPUs Ġ( GT X Ġ1080 ĠTi ) Ġwith Ġgradient Ġaccumulation Ġevery Ġtwo Ġsteps . ĠModel Ġcheckpoints Ġwere Ġsaved Ġand Ġevaluated Ġon Ġthe Ġvalidation Ġset Ġevery Ġ1 , 000 Ġsteps . ĠWe Ġselected Ġthe Ġtop - 3 Ġcheckpoints Ġbased Ġon Ġthe Ġevaluation Ġloss Ġon Ġthe Ġvalidation Ġset , Ġand Ġreport Ġthe Ġaveraged Ġresults Ġon Ġthe Ġtest Ġset . ĠWe Ġused Ġa Ġgreedy Ġalgorithm Ġsimilar Ġto ĠB IB REF 7 Ġto Ġobtain Ġan Ġor acle Ġsummary Ġfor Ġeach Ġdocument Ġto Ġtrain Ġextract ive Ġmodels . ĠThe Ġalgorithm Ġgenerates Ġan Ġor acle Ġconsisting Ġof Ġmultiple Ġsentences Ġwhich Ġmaximize Ġthe ĠR OU GE - 2 Ġscore Ġagainst Ġthe Ġgold Ġsummary . </s> When Ġpredicting Ġsumm aries Ġfor Ġa Ġnew Ġdocument , Ġwe Ġfirst Ġuse Ġthe Ġmodel Ġto Ġobtain Ġthe Ġscore Ġfor Ġeach Ġsentence . ĠWe Ġthen Ġrank Ġthese Ġsentences Ġby Ġtheir Ġscores Ġfrom Ġhighest Ġto Ġlowest , Ġand Ġselect Ġthe Ġtop - 3 Ġsentences Ġas Ġthe Ġsummary . </s> During Ġsentence Ġselection Ġwe Ġuse ĠTr ig ram ĠBl ocking Ġto Ġreduce Ġredundancy ĠB IB REF 9 . ĠGiven Ġsummary Ġ$ S $ Ġand Ġcandidate Ġsentence Ġ$ c $, Ġwe Ġskip Ġ$ c $ Ġif Ġthere Ġexists Ġa Ġtrig ram Ġoverlapping Ġbetween Ġ$ c $ Ġand Ġ$ S $. ĠThe Ġintuition Ġis Ġsimilar Ġto ĠMax imal ĠMarg inal ĠRe lev ance Ġ( MM R ; ĠB IB REF 29 ); Ġwe Ġwish Ġto Ġminimize Ġthe Ġsimilarity Ġbetween Ġthe Ġsentence Ġbeing Ġconsidered Ġand Ġsentences Ġwhich Ġhave Ġbeen Ġalready Ġselected Ġas Ġpart Ġof Ġthe Ġsummary . </s> Exper imental ĠSetup Ġ: :: ĠImplementation ĠDetails Ġ: :: ĠAbstract ive ĠSum mar ization </s> In Ġall Ġabstract ive Ġmodels , Ġwe Ġapplied Ġdrop out Ġ( with Ġprobability Ġ$ 0 . 1 $ ) Ġbefore Ġall Ġlinear Ġlayers ; Ġlabel Ġsmoot hing ĠB IB REF 30 Ġwith Ġsmoot hing Ġfactor Ġ$ 0 . 1 $ Ġwas Ġalso Ġused . ĠOur ĠTrans former Ġdec oder Ġhas Ġ768 Ġhidden Ġunits Ġand Ġthe Ġhidden Ġsize Ġfor Ġall Ġfeed - forward Ġlayers Ġis Ġ2 , 048 . ĠAll Ġmodels Ġwere Ġtrained Ġfor Ġ200 , 000 Ġsteps Ġon Ġ4 ĠGPUs Ġ( GT X Ġ1080 ĠTi ) Ġwith Ġgradient Ġaccumulation Ġevery Ġfive Ġsteps . ĠModel Ġcheckpoints Ġwere Ġsaved Ġand Ġevaluated Ġon Ġthe Ġvalidation Ġset Ġevery Ġ2 , 500 Ġsteps . ĠWe Ġselected Ġthe Ġtop - 3 Ġcheckpoints Ġbased Ġon Ġtheir Ġevaluation Ġloss Ġon Ġthe Ġvalidation Ġset , Ġand Ġreport Ġthe Ġaveraged Ġresults Ġon Ġthe Ġtest Ġset . </s> During Ġdecoding Ġwe Ġused Ġbeam Ġsearch Ġ( size Ġ5 ), Ġand Ġtuned Ġthe Ġ$\ alpha Ġ$ Ġfor Ġthe Ġlength Ġpenalty ĠB IB REF 31 Ġbetween Ġ$ 0 . 6 $ Ġand Ġ1 Ġon Ġthe Ġvalidation Ġset ; Ġwe Ġdecode Ġuntil Ġan Ġend - of - sequence Ġtoken Ġis Ġemitted Ġand Ġrepeated Ġtrig rams Ġare Ġblocked ĠB IB REF 9 . ĠIt Ġis Ġworth Ġnoting Ġthat Ġour Ġdec oder Ġapplies Ġneither Ġa Ġcopy Ġnor Ġa Ġcoverage Ġmechanism ĠB IB REF 6 , Ġdespite Ġtheir Ġpopularity Ġin Ġabstract ive Ġsummar ization . ĠThis Ġis Ġmainly Ġbecause Ġwe Ġfocus Ġon Ġbuilding Ġa Ġminimum - requ irements Ġmodel Ġand Ġthese Ġmechanisms Ġmay Ġintroduce Ġadditional Ġhyper - param eters Ġto Ġtune . ĠThanks Ġto Ġthe Ġsub words Ġtoken izer , Ġwe Ġalso Ġrarely Ġobserve Ġissues Ġwith Ġout - of - voc abulary Ġwords Ġin Ġthe Ġoutput ; Ġmoreover , Ġtrig ram - blocking Ġproduces Ġdiverse Ġsumm aries Ġmanaging Ġto Ġreduce Ġrepet itions . </s> Results Ġ: :: ĠAutomatic ĠEvaluation </s> We Ġevaluated Ġsummar ization Ġquality Ġautomatically Ġusing ĠR OU GE ĠB IB REF 32 . ĠWe Ġreport Ġun ig ram Ġand Ġbig ram Ġoverlap Ġ( R OU GE - 1 Ġand ĠR OU GE - 2 ) Ġas Ġa Ġmeans Ġof Ġassessing Ġinform at iveness Ġand Ġthe Ġlongest Ġcommon Ġsubsequ ence Ġ( R OU GE - L ) Ġas Ġa Ġmeans Ġof Ġassessing Ġflu ency . ĠTable ĠT AB REF 23 Ġsummarizes Ġour Ġresults Ġon Ġthe ĠCNN / Daily Mail Ġdataset . ĠThe Ġfirst Ġblock Ġin Ġthe Ġtable Ġincludes Ġthe Ġresults Ġof Ġan Ġextract ive ĠOracle Ġsystem Ġas Ġan Ġupper Ġbound . ĠWe Ġalso Ġpresent Ġthe ĠLead - 3 Ġbaseline Ġ( which Ġsimply Ġselects Ġthe Ġfirst Ġthree Ġsentences Ġin Ġa Ġdocument ). ĠThe Ġsecond Ġblock Ġin Ġthe Ġtable Ġincludes Ġvarious Ġextract ive Ġmodels Ġtrained Ġon Ġthe ĠCNN / Daily Mail Ġdataset Ġ( see ĠSection ĠSEC REF 5 Ġfor Ġan Ġoverview ). ĠFor Ġcomparison Ġto Ġour Ġown Ġmodel , Ġwe Ġalso Ġimplemented Ġa Ġnon - pret rained ĠTrans former Ġbaseline Ġ( Trans former Ext ) Ġwhich Ġuses Ġthe Ġsame Ġarchitecture Ġas ĠBert Sum Ext , Ġbut Ġwith Ġfewer Ġparameters . ĠIt Ġis Ġrandomly Ġinitialized Ġand Ġonly Ġtrained Ġon Ġthe Ġsummar ization Ġtask . ĠTrans former Ext Ġhas Ġ6 Ġlayers , Ġthe Ġhidden Ġsize Ġis Ġ512 , Ġand Ġthe Ġfeed - forward Ġfilter Ġsize Ġis Ġ2 , 048 . ĠThe Ġmodel Ġwas Ġtrained Ġwith Ġsame Ġsettings Ġas Ġin ĠB IB REF 3 . ĠThe Ġthird Ġblock Ġin ĠTable ĠT AB REF 23 Ġhighlights Ġthe Ġperformance Ġof Ġseveral Ġabstract ive Ġmodels Ġon Ġthe ĠCNN / Daily Mail Ġdataset Ġ( see ĠSection ĠSEC REF 6 Ġfor Ġan Ġoverview ). ĠWe Ġalso Ġinclude Ġan Ġabstract ive ĠTrans former Ġbaseline Ġ( Trans former Abs ) Ġwhich Ġhas Ġthe Ġsame Ġdec oder Ġas Ġour Ġabstract ive ĠBert Sum Ġmodels ; Ġthe Ġenc oder Ġis Ġa Ġ6 - layer ĠTrans former Ġwith Ġ768 Ġhidden Ġsize Ġand Ġ2 , 048 Ġfeed - forward Ġfilter Ġsize . ĠThe Ġfourth Ġblock Ġreports Ġresults Ġwith Ġfine - tun ed ĠBert Ġmodels : ĠBert Sum Ext Ġand Ġits Ġtwo Ġvariants Ġ( one Ġwithout Ġinterval Ġembed d ings , Ġand Ġone Ġwith Ġthe Ġlarge Ġversion Ġof ĠBert ), ĠBert Sum Abs , Ġand ĠBert Sum Ext Abs . ĠBert - based Ġmodels Ġoutper form Ġthe ĠLead - 3 Ġbaseline Ġwhich Ġis Ġnot Ġa Ġstraw man ; Ġon Ġthe ĠCNN / Daily Mail Ġcorpus Ġit Ġis Ġindeed Ġsuperior Ġto Ġseveral Ġextract ive ĠB IB REF 7 , ĠB IB REF 8 , ĠB IB REF 19 Ġand Ġabstract ive Ġmodels ĠB IB REF 6 . ĠBert Ġmodels Ġcollectively Ġoutper form Ġall Ġpreviously Ġproposed Ġextract ive Ġand Ġabstract ive Ġsystems , Ġonly Ġfalling Ġbehind Ġthe ĠOracle Ġupper Ġbound . ĠAmong ĠBert Ġvariants , ĠBert Sum Ext Ġperforms Ġbest Ġwhich Ġis Ġnot Ġentirely Ġsurprising ; ĠCNN / Daily Mail Ġsumm aries Ġare Ġsomewhat Ġextract ive Ġand Ġeven Ġabstract ive Ġmodels Ġare Ġprone Ġto Ġcopying Ġsentences Ġfrom Ġthe Ġsource Ġdocument Ġwhen Ġtrained Ġon Ġthis Ġdataset ĠB IB REF 6 . ĠPerhaps Ġunsurprisingly Ġwe Ġobserve Ġthat Ġlarger Ġversions Ġof ĠBert Ġlead Ġto Ġperformance Ġimprovements Ġand Ġthat Ġinterval Ġembed d ings Ġbring Ġonly Ġslight Ġgains . ĠTable ĠT AB REF 24 Ġpresents Ġresults Ġon Ġthe ĠNYT Ġdataset . ĠFollowing Ġthe Ġevaluation Ġprotocol Ġin ĠB IB REF 27 , Ġwe Ġuse Ġlimited - length ĠR OU GE ĠRecall , Ġwhere Ġpredicted Ġsumm aries Ġare Ġtrunc ated Ġto Ġthe Ġlength Ġof Ġthe Ġgold Ġsumm aries . ĠAgain , Ġwe Ġreport Ġthe Ġperformance Ġof Ġthe ĠOracle Ġupper Ġbound Ġand ĠLead - 3 Ġbaseline . ĠThe Ġsecond Ġblock Ġin Ġthe Ġtable Ġcontains Ġpreviously Ġproposed Ġextract ive Ġmodels Ġas Ġwell Ġas Ġour Ġown ĠTrans former Ġbaseline . ĠComp ress ĠB IB REF 27 Ġis Ġan ĠIL P - based Ġmodel Ġwhich Ġcombines Ġcompression Ġand Ġan aph oric ity Ġconstraints . ĠThe Ġthird Ġblock Ġincludes Ġabstract ive Ġmodels Ġfrom Ġthe Ġliterature , Ġand Ġour ĠTrans former Ġbaseline . ĠBert - based Ġmodels Ġare Ġshown Ġin Ġthe Ġfourth Ġblock . ĠAgain , Ġwe Ġobserve Ġthat Ġthey Ġoutper form Ġpreviously Ġproposed Ġapproaches . ĠOn Ġthis Ġdataset , Ġabstract ive ĠBert Ġmodels Ġgenerally Ġperform Ġbetter Ġcompared Ġto ĠBert Sum Ext , Ġalmost Ġapproaching ĠOracle Ġperformance . </s> Table ĠT AB REF 26 Ġsummarizes Ġour Ġresults Ġon Ġthe ĠX Sum Ġdataset . ĠRecall Ġthat Ġsumm aries Ġin Ġthis Ġdataset Ġare Ġhighly Ġabstract ive Ġ( see ĠTable ĠT AB REF 12 ) Ġconsisting Ġof Ġa Ġsingle Ġsentence Ġconve ying Ġthe Ġgist Ġof Ġthe Ġdocument . ĠExtract ive Ġmodels Ġhere Ġperform Ġpoorly Ġas Ġcorrobor ated Ġby Ġthe Ġlow Ġperformance Ġof Ġthe ĠLead Ġbaseline Ġ( which Ġsimply Ġselects Ġthe Ġleading Ġsentence Ġfrom Ġthe Ġdocument ), Ġand Ġthe ĠOracle Ġ( which Ġselects Ġa Ġsingle - best Ġsentence Ġin Ġeach Ġdocument ) Ġin ĠTable ĠT AB REF 26 . ĠAs Ġa Ġresult , Ġwe Ġdo Ġnot Ġreport Ġresults Ġfor Ġextract ive Ġmodels Ġon Ġthis Ġdataset . ĠThe Ġsecond Ġblock Ġin ĠTable ĠT AB REF 26 Ġpresents Ġthe Ġresults Ġof Ġvarious Ġabstract ive Ġmodels Ġtaken Ġfrom ĠB IB REF 22 Ġand Ġalso Ġincludes Ġour Ġown Ġabstract ive ĠTrans former Ġbaseline . ĠIn Ġthe Ġthird Ġblock Ġwe Ġshow Ġthe Ġresults Ġof Ġour ĠBert Ġsummar izers Ġwhich Ġagain Ġare Ġsuperior Ġto Ġall Ġpreviously Ġreported Ġmodels Ġ( by Ġa Ġwide Ġmargin ). </s> Results Ġ: :: ĠModel ĠAnalysis Ġ: :: ĠLearning ĠRates </s> Rec all Ġthat Ġour Ġabstract ive Ġmodel Ġuses Ġseparate Ġoptim izers Ġfor Ġthe Ġenc oder Ġand Ġdec oder . ĠIn ĠTable ĠT AB REF 27 Ġwe Ġexamine Ġwhether Ġthe Ġcombination Ġof Ġdifferent Ġlearning Ġrates Ġ($ \ t ilde { lr } _{ \ math cal Ġ{ E }} $ Ġand Ġ$\ t ilde { lr } _{ \ math cal Ġ{ D }} $ ) Ġis Ġindeed Ġbeneficial . ĠSpecifically , Ġwe Ġreport Ġmodel Ġperplex ity Ġon Ġthe ĠCNN / Daily Mail Ġvalidation Ġset Ġfor Ġvarying Ġenc oder / dec oder Ġlearning Ġrates . ĠWe Ġcan Ġsee Ġthat Ġthe Ġmodel Ġperforms Ġbest Ġwith Ġ$\ t ilde { lr } _{ \ math cal Ġ{ E }} = 2 e - 3 $ Ġand Ġ$\ t ilde { lr } _{ \ math cal Ġ{ D }} = 0 . 1 $. </s> Results Ġ: :: ĠModel ĠAnalysis Ġ: :: ĠPosition Ġof ĠExt racted ĠSent ences </s> In Ġaddition Ġto Ġthe Ġevaluation Ġbased Ġon ĠR OU GE , Ġwe Ġalso Ġanalyzed Ġin Ġmore Ġdetail Ġthe Ġsumm aries Ġproduced Ġby Ġour Ġmodel . ĠFor Ġthe Ġextract ive Ġsetting , Ġwe Ġlooked Ġat Ġthe Ġposition Ġ( in Ġthe Ġsource Ġdocument ) Ġof Ġthe Ġsentences Ġwhich Ġwere Ġselected Ġto Ġappear Ġin Ġthe Ġsummary . ĠFigure ĠFIG REF 31 Ġshows Ġthe Ġproportion Ġof Ġselected Ġsummary Ġsentences Ġwhich Ġappear Ġin Ġthe Ġsource Ġdocument Ġat Ġpositions Ġ1 , Ġ2 , Ġand Ġso Ġon . ĠThe Ġanalysis Ġwas Ġconducted Ġon Ġthe ĠCNN / Daily Mail Ġdataset Ġfor ĠOracle Ġsumm aries , Ġand Ġthose Ġproduced Ġby ĠBert Sum Ext Ġand Ġthe ĠTrans former Ext . ĠWe Ġcan Ġsee Ġthat ĠOracle Ġsummary Ġsentences Ġare Ġfairly Ġsmoothly Ġdistributed Ġacross Ġdocuments , Ġwhile Ġsumm aries Ġcreated Ġby ĠTrans former Ext Ġmostly Ġconcentrate Ġon Ġthe Ġfirst Ġdocument Ġsentences . ĠBert Sum Ext Ġoutputs Ġare Ġmore Ġsimilar Ġto ĠOracle Ġsumm aries , Ġindicating Ġthat Ġwith Ġthe Ġpret rained Ġenc oder , Ġthe Ġmodel Ġrelies Ġless Ġon Ġshallow Ġposition Ġfeatures , Ġand Ġlearns Ġdeeper Ġdocument Ġrepresentations . </s> Results Ġ: :: ĠModel ĠAnalysis Ġ: :: ĠNovel ĠN - gram s </s> We Ġalso Ġanalyzed Ġthe Ġoutput Ġof Ġabstract ive Ġsystems Ġby Ġcalculating Ġthe Ġproportion Ġof Ġnovel Ġn - gram s Ġthat Ġappear Ġin Ġthe Ġsumm aries Ġbut Ġnot Ġin Ġthe Ġsource Ġtexts . ĠThe Ġresults Ġare Ġshown Ġin ĠFigure ĠFIG REF 33 . ĠIn Ġthe ĠCNN / Daily Mail Ġdataset , Ġthe Ġproportion Ġof Ġnovel Ġn - gram s Ġin Ġautomatically Ġgenerated Ġsumm aries Ġis Ġmuch Ġlower Ġcompared Ġto Ġreference Ġsumm aries , Ġbut Ġin ĠX Sum , Ġthis Ġgap Ġis Ġmuch Ġsmaller . ĠWe Ġalso Ġobserve Ġthat Ġon ĠCNN / Daily Mail , ĠBert Ext Abs Ġproduces Ġless Ġnovel Ġn - ng rams Ġthan ĠBert Abs , Ġwhich Ġis Ġnot Ġsurprising . ĠBert Ext Abs Ġis Ġmore Ġbiased Ġtowards Ġselecting Ġsentences Ġfrom Ġthe Ġsource Ġdocument Ġsince Ġit Ġis Ġinitially Ġtrained Ġas Ġan Ġextract ive Ġmodel . ĠThe Ġsupplementary Ġmaterial Ġincludes Ġexamples Ġof Ġsystem Ġoutput Ġand Ġadditional Ġab lation Ġstudies . </s> Results Ġ: :: ĠHuman ĠEvaluation </s> In Ġaddition Ġto Ġautomatic Ġevaluation , Ġwe Ġalso Ġevaluated Ġsystem Ġoutput Ġby Ġelic iting Ġhuman Ġjudgments . ĠWe Ġreport Ġexperiments Ġfollowing Ġa Ġquestion - ans w ering Ġ( Q A ) Ġparadigm ĠB IB REF 33 , ĠB IB REF 8 Ġwhich Ġquant ifies Ġthe Ġdegree Ġto Ġwhich Ġsummar ization Ġmodels Ġretain Ġkey Ġinformation Ġfrom Ġthe Ġdocument . ĠUnder Ġthis Ġparadigm , Ġa Ġset Ġof Ġquestions Ġis Ġcreated Ġbased Ġon Ġthe Ġgold Ġsummary Ġunder Ġthe Ġassumption Ġthat Ġit Ġhighlights Ġthe Ġmost Ġimportant Ġdocument Ġcontent . ĠParticipants Ġare Ġthen Ġasked Ġto Ġanswer Ġthese Ġquestions Ġby Ġreading Ġsystem Ġsumm aries Ġalone Ġwithout Ġaccess Ġto Ġthe Ġarticle . ĠThe Ġmore Ġquestions Ġa Ġsystem Ġcan Ġanswer , Ġthe Ġbetter Ġit Ġis Ġat Ġsummar izing Ġthe Ġdocument Ġas Ġa Ġwhole . ĠMoreover , Ġwe Ġalso Ġassessed Ġthe Ġoverall Ġquality Ġof Ġthe Ġsumm aries Ġproduced Ġby Ġabstract ive Ġsystems Ġwhich Ġdue Ġto Ġtheir Ġability Ġto Ġrewrite Ġcontent Ġmay Ġproduce Ġdis f luent Ġor Ġun gram matical Ġoutput . ĠSpecifically , Ġwe Ġfollowed Ġthe ĠBest - W orst ĠSc aling ĠB IB REF 34 Ġmethod Ġwhere Ġparticipants Ġwere Ġpresented Ġwith Ġthe Ġoutput Ġof Ġtwo Ġsystems Ġ( and Ġthe Ġoriginal Ġdocument ) Ġand Ġasked Ġto Ġdecide Ġwhich Ġone Ġwas Ġbetter Ġaccording Ġto Ġthe Ġcriteria Ġof ĠIn format iveness , ĠFlu ency , Ġand ĠSu cc inct ness . </s> Both Ġtypes Ġof Ġevaluation Ġwere Ġconducted Ġon Ġthe ĠAmazon ĠMechanical ĠTurk Ġplatform . ĠFor Ġthe ĠCNN / Daily Mail Ġand ĠNYT Ġdatasets Ġwe Ġused Ġthe Ġsame Ġdocuments Ġ( 20 Ġin Ġtotal ) Ġand Ġquestions Ġfrom Ġprevious Ġwork ĠB IB REF 8 , ĠB IB REF 18 . ĠFor ĠX Sum , Ġwe Ġrandomly Ġselected Ġ20 Ġdocuments Ġ( and Ġtheir Ġquestions ) Ġfrom Ġthe Ġrelease Ġof ĠB IB REF 22 . ĠWe Ġelic ited Ġ3 Ġresponses Ġper ĠHIT . ĠWith Ġregard Ġto ĠQ A Ġevaluation , Ġwe Ġadopted Ġthe Ġscoring Ġmechanism Ġfrom ĠB IB REF 33 ; Ġcorrect Ġanswers Ġwere Ġmarked Ġwith Ġa Ġscore Ġof Ġone , Ġpartially Ġcorrect Ġanswers Ġwith Ġ0 . 5 , Ġand Ġzero Ġotherwise . ĠFor Ġquality - based Ġevaluation , Ġthe Ġrating Ġof Ġeach Ġsystem Ġwas Ġcomputed Ġas Ġthe Ġpercentage Ġof Ġtimes Ġit Ġwas Ġchosen Ġas Ġbetter Ġminus Ġthe Ġtimes Ġit Ġwas Ġselected Ġas Ġworse . ĠRatings Ġthus Ġrange Ġfrom Ġ- 1 Ġ( worst ) Ġto Ġ1 Ġ( best ). </s> Results Ġfor Ġextract ive Ġand Ġabstract ive Ġsystems Ġare Ġshown Ġin ĠTables ĠT AB REF 37 Ġand ĠT AB REF 38 , Ġrespectively . ĠWe Ġcompared Ġthe Ġbest Ġperforming ĠBert Sum Ġmodel Ġin Ġeach Ġsetting Ġ( ext ractive Ġor Ġabstract ive ) Ġagainst Ġvarious Ġstate - of - the - art Ġsystems Ġ( whose Ġoutput Ġis Ġpublicly Ġavailable ), Ġthe ĠLead Ġbaseline , Ġand Ġthe ĠGold Ġstandard Ġas Ġan Ġupper Ġbound . ĠAs Ġshown Ġin Ġboth Ġtables Ġparticipants Ġoverwhelmingly Ġprefer Ġthe Ġoutput Ġof Ġour Ġmodel Ġagainst Ġcomparison Ġsystems Ġacross Ġdatasets Ġand Ġevaluation Ġparad ig ms . ĠAll Ġdifferences Ġbetween ĠBert Sum Ġand Ġcomparison Ġmodels Ġare Ġstatistically Ġsignificant Ġ($ p < 0 . 05 $ ), Ġwith Ġthe Ġexception Ġof ĠT Con v S 2 S Ġ( see ĠTable ĠT AB REF 38 ; ĠX Sum ) Ġin Ġthe ĠQ A Ġevaluation Ġsetting . </s> Con clusions </s> In Ġthis Ġpaper , Ġwe Ġshowcased Ġhow Ġpret rained ĠBert Ġcan Ġbe Ġuse fully Ġapplied Ġin Ġtext Ġsummar ization . ĠWe Ġintroduced Ġa Ġnovel Ġdocument - level Ġenc oder Ġand Ġproposed Ġa Ġgeneral Ġframework Ġfor Ġboth Ġabstract ive Ġand Ġextract ive Ġsummar ization . ĠExperimental Ġresults Ġacross Ġthree Ġdatasets Ġshow Ġthat Ġour Ġmodel Ġachieves Ġstate - of - the - art Ġresults Ġacross Ġthe Ġboard Ġunder Ġautomatic Ġand Ġhuman - based Ġevaluation Ġprotocols . ĠAlthough Ġwe Ġmainly Ġfocused Ġon Ġdocument Ġencoding Ġfor Ġsummar ization , Ġin Ġthe Ġfuture , Ġwe Ġwould Ġlike Ġto Ġtake Ġadvantage Ġthe Ġcapabilities Ġof ĠBert Ġfor Ġlanguage Ġgeneration . </s> Acknowled gments </s> This Ġresearch Ġis Ġsupported Ġby Ġa ĠGoogle ĠPhD ĠFellowship Ġto Ġthe Ġfirst Ġauthor . ĠWe Ġgrate fully Ġacknowledge Ġthe Ġsupport Ġof Ġthe ĠEuropean ĠResearch ĠCouncil Ġ( L ap ata , Ġaward Ġnumber Ġ68 17 60 , ĠâĢ ľ Trans l ating ĠMultiple ĠMod alities Ġinto ĠText âĢ Ŀ ). ĠWe Ġwould Ġalso Ġlike Ġto Ġthank ĠSh ashi ĠNar ayan Ġfor Ġproviding Ġus Ġwith Ġthe ĠX Sum Ġdataset .