BIBREF26 
EXTRACTIVE
BIBREF26
EXTRACTIVE
Following BIBREF11 , we experiment on the non-anonymized version of . When generating summaries, we follow standard practice of tuning the maximum output length and disallow repeating the same trigram BIBREF27 , BIBREF14 . For this task we train language model representations on the combination of newscrawl and the training data. Table TABREF16 shows that pre-trained embeddings can significantly improve on top of a strong baseline transformer. We also compare to BIBREF26 who use a task-specific architecture compared to our generic sequence to sequence baseline. Pre-trained representations are complementary to their method. FLOAT SELECTED: Table 3: Abstractive summarization results on CNNDailyMail. ELMo inputs achieve a new state of the art.
Following BIBREF11 , we experiment on the non-anonymized version of . When generating summaries, we follow standard practice of tuning the maximum output length and disallow repeating the same trigram BIBREF27 , BIBREF14 . For this task we train language model representations on the combination of newscrawl and the training data. Table TABREF16 shows that pre-trained embeddings can significantly improve on top of a strong baseline transformer. We also compare to BIBREF26 who use a task-specific architecture compared to our generic sequence to sequence baseline. Pre-trained representations are complementary to their method. FLOAT SELECTED: Table 3: Abstractive summarization results on CNNDailyMail. ELMo inputs achieve a new state of the art.
1903.09722
Introduction </s> Pre - training Ġof Ġlanguage Ġmodels Ġhas Ġbeen Ġshown Ġto Ġprovide Ġlarge Ġimprovements Ġfor Ġa Ġrange Ġof Ġlanguage Ġunderstanding Ġtasks ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ. ĠThe Ġkey Ġidea Ġis Ġto Ġtrain Ġa Ġlarge Ġgener ative Ġmodel Ġon Ġvast Ġcorpor a Ġand Ġuse Ġthe Ġresulting Ġrepresentations Ġon Ġtasks Ġfor Ġwhich Ġonly Ġlimited Ġamounts Ġof Ġlabeled Ġdata Ġis Ġavailable . ĠPre - training Ġof Ġsequence Ġto Ġsequence Ġmodels Ġhas Ġbeen Ġpreviously Ġinvestigated Ġfor Ġtext Ġclassification ĠB IB REF 4 Ġbut Ġnot Ġfor Ġtext Ġgeneration . ĠIn Ġneural Ġmachine Ġtranslation , Ġthere Ġhas Ġbeen Ġwork Ġon Ġtransferring Ġrepresentations Ġfrom Ġhigh - resource Ġlanguage Ġpairs Ġto Ġlow - resource Ġsettings ĠB IB REF 5 Ġ. </s> In Ġthis Ġpaper , Ġwe Ġapply Ġpre - trained Ġrepresentations Ġfrom Ġlanguage Ġmodels Ġto Ġlanguage Ġgeneration Ġtasks Ġthat Ġcan Ġbe Ġmodeled Ġby Ġsequence Ġto Ġsequence Ġarchitectures . ĠPrevious Ġwork Ġon Ġintegrating Ġlanguage Ġmodels Ġwith Ġsequence Ġto Ġsequence Ġmodels Ġfocused Ġon Ġthe Ġdec oder Ġnetwork Ġand Ġadded Ġlanguage Ġmodel Ġrepresentations Ġright Ġbefore Ġthe Ġoutput Ġof Ġthe Ġdec oder ĠB IB REF 6 Ġ. ĠWe Ġextend Ġtheir Ġstudy Ġby Ġinvestigating Ġseveral Ġother Ġstrategies Ġsuch Ġas Ġinput ting ĠEL Mo - style Ġrepresentations ĠB IB REF 0 Ġor Ġfine - tun ing Ġthe Ġlanguage Ġmodel Ġ( Â§ ĠSEC REF 2 Ġ). </s> Our Ġexperiments Ġrely Ġon Ġstrong Ġtransformer - based Ġlanguage Ġmodels Ġtrained Ġon Ġup Ġto Ġsix Ġbillion Ġtokens Ġ( Â§ ĠSEC REF 3 Ġ). ĠWe Ġpresent Ġa Ġdetailed Ġstudy Ġof Ġvarious Ġstrategies Ġin Ġdifferent Ġsimulated Ġlabeled Ġtraining Ġdata Ġscenarios Ġand Ġobserve Ġthe Ġlargest Ġimprovements Ġin Ġlow - resource Ġsettings Ġbut Ġgains Ġof Ġover Ġ1 ĠB LE U Ġare Ġstill Ġpossible Ġwhen Ġfive Ġmillion Ġsentence - p airs Ġare Ġavailable . ĠThe Ġmost Ġsuccessful Ġstrategy Ġto Ġintegrate Ġpre - trained Ġrepresentations Ġis Ġas Ġinput Ġto Ġthe Ġenc oder Ġnetwork Ġ( Â§ ĠSEC REF 4 Ġ). </s> Str ateg ies Ġto Ġadd Ġrepresentations </s> We Ġconsider Ġaugment ing Ġa Ġstandard Ġsequence Ġto Ġsequence Ġmodel Ġwith Ġpre - trained Ġrepresentations Ġfollowing Ġan ĠEL Mo - style Ġregime Ġ( Â§ ĠSEC REF 2 Ġ) Ġas Ġwell Ġas Ġby Ġfine - tun ing Ġthe Ġlanguage Ġmodel Ġ( Â§ ĠSEC REF 3 Ġ). </s> EL Mo Ġaug mentation </s> The ĠEL Mo Ġapproach Ġof ĠB IB REF 0 Ġforms Ġcontextual ized Ġword Ġembed d ings Ġbased Ġon Ġlanguage Ġmodel Ġrepresentations Ġwithout Ġadjusting Ġthe Ġactual Ġlanguage Ġmodel Ġparameters . ĠSpecifically , Ġthe ĠEL Mo Ġmodule Ġcontains Ġa Ġset Ġof Ġparameters ĠIN LINE FORM 0 Ġto Ġform Ġa Ġlinear Ġcombination Ġof Ġthe ĠIN LINE FORM 1 Ġlayers Ġof Ġthe Ġlanguage Ġmodel : ĠEL Mo Ġ= ĠIN LINE FORM 2 Ġwhere ĠIN LINE FORM 3 Ġis Ġa Ġlearned Ġscal ar , ĠIN LINE FORM 4 Ġis Ġa Ġconstant Ġto Ġnormal ize Ġthe ĠIN LINE FORM 5 Ġto Ġsum Ġto Ġone Ġand ĠIN LINE FORM 6 Ġis Ġthe Ġoutput Ġof Ġthe ĠIN LINE FORM 7 Ġ- th Ġlanguage Ġmodel Ġlayer ; Ġthe Ġmodule Ġalso Ġconsiders Ġthe Ġinput Ġword Ġembed d ings Ġof Ġthe Ġlanguage Ġmodel . ĠWe Ġalso Ġapply Ġlayer Ġnormal ization ĠB IB REF 7 Ġto Ġeach ĠIN LINE FORM 8 Ġbefore Ġcomputing ĠEL Mo Ġvectors . </s> We Ġexperiment Ġwith Ġan ĠEL Mo Ġmodule Ġto Ġinput Ġcontextual ized Ġembed d ings Ġeither Ġto Ġthe Ġenc oder Ġ() Ġor Ġthe Ġdec oder Ġ(). ĠThis Ġprovides Ġword Ġrepresentations Ġspecific Ġto Ġthe Ġcurrent Ġinput Ġsentence Ġand Ġthese Ġrepresentations Ġhave Ġbeen Ġtrained Ġon Ġmuch Ġmore Ġdata Ġthan Ġis Ġavailable Ġfor Ġthe Ġtext Ġgeneration Ġtask . </s> Fine - tun ing Ġapproach </s> Fine - tun ing Ġthe Ġpre - trained Ġrepresentations Ġadjusts Ġthe Ġlanguage Ġmodel Ġparameters Ġby Ġthe Ġlearning Ġsignal Ġof Ġthe Ġend - task ĠB IB REF 1 Ġ, ĠB IB REF 3 Ġ. ĠWe Ġreplace Ġlearned Ġinput Ġword Ġembed d ings Ġin Ġthe Ġenc oder Ġnetwork Ġwith Ġthe Ġoutput Ġof Ġthe Ġlanguage Ġmodel Ġ(). ĠSpecifically , Ġwe Ġuse Ġthe Ġlanguage Ġmodel Ġrepresentation Ġof Ġthe Ġlayer Ġbefore Ġthe Ġsoft max Ġand Ġfeed Ġit Ġto Ġthe Ġenc oder . ĠWe Ġalso Ġadd Ġdrop out Ġto Ġthe Ġlanguage Ġmodel Ġoutput . ĠTun ing Ġseparate Ġlearning Ġrates Ġfor Ġthe Ġlanguage Ġmodel Ġand Ġthe Ġsequence Ġto Ġsequence Ġmodel Ġmay Ġlead Ġto Ġbetter Ġperformance Ġbut Ġwe Ġleave Ġthis Ġto Ġfuture Ġwork . ĠHowever , Ġwe Ġdo Ġtune Ġthe Ġnumber Ġof Ġenc oder Ġblocks ĠIN LINE FORM 0 Ġas Ġwe Ġfound Ġthis Ġimportant Ġto Ġobtain Ġgood Ġaccuracy Ġfor Ġthis Ġsetting . ĠWe Ġapply Ġthe Ġsame Ġstrategy Ġto Ġthe Ġdec oder : Ġwe Ġinput Ġlanguage Ġmodel Ġrepresentations Ġto Ġthe Ġdec oder Ġnetwork Ġand Ġfine - t une Ġthe Ġlanguage Ġmodel Ġwhen Ġtraining Ġthe Ġsequence Ġto Ġsequence Ġmodel Ġ(). </s> Dat as ets </s> We Ġtrain Ġlanguage Ġmodels Ġon Ġtwo Ġlanguages : ĠOne Ġmodel Ġis Ġestimated Ġon Ġthe ĠGerman Ġnew sc rawl Ġdistributed Ġby ĠW MT ' 18 Ġcomprising Ġ260 M Ġsentences Ġor Ġ6 B Ġtokens . ĠAnother Ġmodel Ġis Ġtrained Ġon Ġthe ĠEnglish Ġnew sc rawl Ġdata Ġcomprising Ġ193 M Ġsentences Ġor Ġ5 B Ġtokens . ĠWe Ġlearn Ġa Ġjoint ĠByte - P air - Enc oding Ġ( B PE ; ĠS enn rich Ġet Ġal ., Ġ2016 ) Ġvocabulary Ġof Ġ37 K Ġtypes Ġon Ġthe ĠGerman Ġand ĠEnglish Ġnew sc rawl Ġand Ġtrain Ġthe Ġlanguage Ġmodels Ġwith Ġthis Ġvocabulary . </s> We Ġconsider Ġtwo Ġbenchmarks : ĠMost Ġexperiments Ġare Ġrun Ġon Ġthe ĠW MT ' 18 ĠEnglish - German Ġ( en - de ) Ġnews Ġtranslation Ġtask Ġand Ġwe Ġvalidate Ġour Ġfindings Ġon Ġthe ĠW MT ' 18 ĠEnglish - Turkish Ġ( en - tr ) Ġnews Ġtask . ĠFor ĠW MT ' 18 ĠEnglish - German , Ġthe Ġtraining Ġcorpus Ġconsists Ġof Ġall Ġavailable Ġbite xt Ġexcluding Ġthe ĠPar a C rawl Ġcorpus Ġand Ġwe Ġremove Ġsentences Ġlonger Ġthan Ġ250 Ġtokens Ġas Ġwell Ġas Ġsentence - p airs Ġwith Ġa Ġsource / target Ġlength Ġratio Ġexceeding Ġ1 . 5 . ĠThis Ġresults Ġin Ġ5 . 18 M Ġsentence Ġpairs . ĠWe Ġtoken ize Ġall Ġdata Ġwith Ġthe ĠMoses Ġtoken izer ĠB IB REF 8 Ġand Ġapply Ġthe ĠB PE Ġvocabulary Ġlearned Ġon Ġthe Ġmon oling ual Ġcorpor a . </s> For ĠW MT ' 18 ĠEnglish - Turkish , Ġwe Ġuse Ġall Ġof Ġthe Ġavailable Ġbite xt Ġcomprising Ġ208 K Ġsentence - p airs Ġwithout Ġany Ġfiltering . ĠWe Ġdevelop Ġon Ġnew st est 2017 Ġand Ġtest Ġon Ġnew st est 2018 . ĠFor Ġen - tr Ġwe Ġonly Ġexperiment Ġwith Ġadding Ġrepresentations Ġto Ġthe Ġenc oder Ġand Ġtherefore Ġapply Ġthe Ġlanguage Ġmodel Ġvocabulary Ġto Ġthe Ġsource Ġside . ĠFor Ġthe Ġtarget Ġvocabulary Ġwe Ġlearn Ġa ĠB PE Ġcode Ġwith Ġ32 K Ġmerge Ġoperations Ġon Ġthe ĠTurkish Ġside Ġof Ġthe Ġbite xt . ĠBoth Ġdatasets Ġare Ġevaluated Ġin Ġterms Ġof Ġcase - sensitive Ġde - token ized ĠB LE U ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ. </s> We Ġconsider Ġthe Ġabstract ive Ġdocument Ġsummar ization Ġtask Ġcomprising Ġover Ġ280 K Ġnews Ġarticles Ġpaired Ġwith Ġmulti - sent ence Ġsumm aries . Ġis Ġa Ġwidely Ġused Ġdataset Ġfor Ġabstract ive Ġtext Ġsummar ization . ĠFollowing ĠB IB REF 11 Ġ, Ġwe Ġreport Ġresults Ġon Ġthe Ġnon - an onym ized Ġversion Ġof Ġrather Ġthan Ġthe Ġentity - an onym ized Ġversion ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġbecause Ġthe Ġlanguage Ġmodel Ġwas Ġtrained Ġon Ġfull Ġtext . ĠArticles Ġare Ġtrunc ated Ġto Ġ400 Ġtokens ĠB IB REF 11 Ġand Ġwe Ġuse Ġa ĠB PE Ġvocabulary Ġof Ġ32 K Ġtypes ĠB IB REF 14 Ġ. ĠWe Ġevaluate Ġin Ġterms Ġof ĠF 1 - R ou ge , Ġthat Ġis ĠRouge - 1 , ĠRouge - 2 Ġand ĠRouge - L ĠB IB REF 15 Ġ. </s> Language Ġmodel Ġpre - training </s> We Ġconsider Ġtwo Ġtypes Ġof Ġarchitectures : Ġa Ġbi - direction al Ġlanguage Ġmodel Ġto Ġaugment Ġthe Ġsequence Ġto Ġsequence Ġenc oder Ġand Ġa Ġun i - direction al Ġmodel Ġto Ġaugment Ġthe Ġdec oder . ĠBoth Ġuse Ġself - att ention ĠB IB REF 16 Ġand Ġthe Ġun i - direction al Ġmodel Ġcontains ĠIN LINE FORM 0 Ġtransformer Ġblocks , Ġfollowed Ġby Ġa Ġword Ġclass ifier Ġto Ġpredict Ġthe Ġnext Ġword Ġon Ġthe Ġright . ĠThe Ġbi - direction al Ġmodel Ġsolves Ġa Ġclo ze - style Ġtoken Ġprediction Ġtask Ġat Ġtraining Ġtime ĠB IB REF 17 Ġ. ĠThe Ġmodel Ġconsists Ġof Ġtwo Ġtowers , Ġthe Ġforward Ġtower Ġoperates Ġleft - to - right Ġand Ġthe Ġtower Ġoperating Ġright - to - left Ġas Ġbackward Ġtower ; Ġeach Ġtower Ġcontains ĠIN LINE FORM 1 Ġtransformer Ġblocks . ĠThe Ġforward Ġand Ġbackward Ġrepresentations Ġare Ġcombined Ġvia Ġa Ġself - att ention Ġmodule Ġand Ġthe Ġoutput Ġof Ġthis Ġmodule Ġis Ġused Ġto Ġpredict Ġthe Ġtoken Ġat Ġposition ĠIN LINE FORM 2 Ġ. ĠThe Ġmodel Ġhas Ġaccess Ġto Ġthe Ġentire Ġinput Ġsurrounding Ġthe Ġcurrent Ġtarget Ġtoken . ĠModels Ġuse Ġthe Ġstandard Ġsettings Ġfor Ġthe ĠBig ĠTrans former ĠB IB REF 16 Ġ. ĠThe Ġbi - direction al Ġmodel Ġcontains Ġ353 M Ġparameters Ġand Ġthe Ġun i - direction al Ġmodel Ġ190 M Ġparameters . ĠBoth Ġmodels Ġwere Ġtrained Ġfor Ġ1 M Ġsteps Ġusing ĠN ester ov 's Ġaccelerated Ġgradient ĠB IB REF 18 Ġwith Ġmomentum ĠIN LINE FORM 3 Ġfollowing ĠB IB REF 19 Ġ. ĠThe Ġlearning Ġrate Ġis Ġlin early Ġwarmed Ġup Ġfrom ĠIN LINE FORM 4 Ġto Ġ1 Ġfor Ġ16 K Ġsteps Ġand Ġthen Ġan ne aled Ġusing Ġa Ġcos ine Ġlearning Ġrate Ġschedule Ġwith Ġa Ġsingle Ġphase Ġto Ġ0 . 0001 ĠB IB REF 20 Ġ. ĠWe Ġtrain Ġon Ġ32 ĠNvidia ĠV 100 ĠS XM 2 ĠGPUs Ġand Ġuse Ġthe ĠN CC L 2 Ġlibrary Ġas Ġwell Ġas Ġthe Ġtorch Ġdistributed Ġpackage Ġfor Ġinter - GPU Ġcommunication . ĠTraining Ġrelies Ġon Ġ16 - bit Ġfloating Ġpoint Ġoperations ĠB IB REF 21 Ġand Ġit Ġtook Ġsix Ġdays Ġfor Ġthe Ġbi - direction al Ġmodel Ġand Ġfour Ġdays Ġfor Ġthe Ġun i - direction al Ġmodel . </s> Sequ ence Ġto Ġsequence Ġmodel </s> We Ġuse Ġthe Ġtransformer Ġimplementation Ġof Ġthe Ġfair seq Ġtool kit ĠB IB REF 22 Ġ. ĠThe ĠW MT Ġen - de Ġand Ġen - tr Ġexperiments Ġare Ġbased Ġon Ġthe ĠBig ĠTrans former Ġsequence Ġto Ġsequence Ġarchitecture Ġwith Ġ6 Ġblocks Ġin Ġthe Ġenc oder Ġand Ġdec oder . ĠFor Ġabstract ive Ġsummar ization Ġwe Ġuse Ġa Ġbase Ġtransformer Ġmodel ĠB IB REF 16 Ġ. ĠWe Ġtune Ġdrop out Ġvalues Ġof Ġbetween Ġ0 . 1 Ġand Ġ0 . 4 Ġon Ġthe Ġvalidation Ġset . ĠModels Ġare Ġoptimized Ġwith ĠAdam ĠB IB REF 23 Ġusing ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, Ġand ĠIN LINE FORM 2 Ġand Ġwe Ġuse Ġthe Ġsame Ġlearning Ġrate Ġschedule Ġas ĠB IB REF 16 Ġ; Ġwe Ġperform Ġ10 K - 200 K Ġdepending Ġon Ġbite xt Ġsize . ĠAll Ġmodels Ġuse Ġlabel Ġsmoot hing Ġwith Ġa Ġuniform Ġprior Ġdistribution Ġover Ġthe Ġvocabulary ĠIN LINE FORM 3 ĠB IB REF 24 Ġ, ĠB IB REF 25 Ġ. ĠWe Ġrun Ġexperiments Ġon Ġ8 ĠGPUs Ġand Ġgenerate Ġtranslations Ġwith Ġa Ġbeam Ġof Ġsize Ġ5 . </s> Machine Ġtranslation </s> We Ġfirst Ġpresent Ġa Ġcomparison Ġof Ġthe Ġvarious Ġstrategies Ġin Ġdifferent Ġsimulated Ġparallel Ġcorpus Ġsize Ġsettings . ĠFor Ġeach Ġexperiment , Ġwe Ġtune Ġthe Ġdrop out Ġapplied Ġto Ġthe Ġlanguage Ġmodel Ġrepresentations , Ġand Ġwe Ġreduce Ġthe Ġnumber Ġof Ġoptim izer Ġsteps Ġfor Ġsmaller Ġbite xt Ġsetups Ġas Ġmodels Ġconverge Ġfaster ; Ġall Ġother Ġhyper - param eters Ġare Ġequal Ġbetween Ġsetups . ĠOur Ġbaseline Ġis Ġa ĠBig ĠTrans former Ġmodel Ġand Ġwe Ġalso Ġconsider Ġa Ġvariant Ġwhere Ġwe Ġshare Ġtoken Ġembed d ings Ġbetween Ġthe Ġenc oder Ġand Ġdec oder Ġ( ; ĠIn an Ġet Ġal ., Ġ2016 ; ĠPress Ġ& ĠWolf , Ġ2016 ). </s> Figure ĠFIG REF 10 Ġshows Ġresults Ġaveraged Ġover Ġsix Ġtest Ġsets Ġrelative Ġto Ġthe Ġbaseline Ġwhich Ġdoes Ġnot Ġshare Ġsource Ġand Ġtarget Ġembed d ings Ġ( App endix ĠSEC REF 6 Ġshows Ġa Ġdetailed Ġbreakdown ). Ġperforms Ġvery Ġwell Ġwith Ġlittle Ġlabeled Ġdata Ġbut Ġthe Ġgains Ġer ode Ġto Ġpractically Ġzero Ġin Ġlarge Ġbite xt Ġsettings . ĠPre - trained Ġlanguage Ġmodel Ġrepresentations Ġare Ġmost Ġeffective Ġin Ġlow Ġbite xt Ġsetups . ĠThe Ġbest Ġperforming Ġstrategy Ġis ĠEL Mo Ġembed d ings Ġinput Ġto Ġthe Ġenc oder Ġ(). ĠThis Ġimproves Ġthe Ġbaseline Ġby Ġ3 . 8 ĠB LE U Ġin Ġthe Ġ160 K Ġbite xt Ġsetting Ġand Ġit Ġstill Ġimproves Ġthe Ġ5 . 2 M Ġsetting Ġby Ġover Ġ1 ĠB LE U . </s> We Ġfurther Ġimprove Ġby Ġsharing Ġlearned Ġword Ġrepresentations Ġin Ġthe Ġdec oder Ġby Ġtying Ġinput Ġand Ġoutput Ġembed d ings Ġ(). ĠThis Ġconfiguration Ġperforms Ġeven Ġbetter Ġthan Ġwith Ġa Ġgain Ġof Ġ5 . 3 ĠB LE U Ġin Ġthe Ġ160 K Ġsetup . ĠSharing Ġdec oder Ġembed d ings Ġis Ġequally Ġapplicable Ġto Ġ. ĠLanguage Ġmodel Ġrepresentations Ġare Ġmuch Ġless Ġeffective Ġin Ġthe Ġdec oder : Ġimproves Ġthe Ġ160 K Ġbite xt Ġsetup Ġbut Ġyields Ġno Ġimprovements Ġthereafter Ġand Ġperforms Ġeven Ġworse . ĠWe Ġconjecture Ġthat Ġpre - trained Ġrepresentations Ġgive Ġmuch Ġeasier Ġwins Ġin Ġthe Ġenc oder . ĠTable ĠT AB REF 14 Ġshows Ġadditional Ġresults Ġon Ġnew st est 2018 . </s> Pre - trained Ġrepresentations Ġmostly Ġimpacts Ġthe Ġtraining Ġtime Ġof Ġthe Ġsequence Ġto Ġsequence Ġmodel Ġ( see ĠAppendix ĠSEC REF 7 Ġ): Ġslows Ġthroughput Ġduring Ġtraining Ġby Ġabout Ġ5 . 3 x Ġand Ġis Ġeven Ġslower Ġbecause Ġof Ġthe Ġneed Ġto Ġback prop agate Ġthrough Ġthe ĠLM Ġfor Ġfine - tun ing Ġ( 9 . 2 x ). ĠHowever , Ġinference Ġis Ġonly Ġ12 - 14 % Ġslower Ġthan Ġthe Ġbaseline Ġwhen Ġadding Ġpre - trained Ġembed d ings Ġto Ġthe Ġenc oder Ġ( , Ġ). ĠThis Ġis Ġbecause Ġthe ĠLM Ġcomputation Ġcan Ġbe Ġpar al el ized Ġfor Ġall Ġinput Ġtokens . ĠIn ference Ġis Ġmuch Ġslower Ġwhen Ġadding Ġrepresentations Ġto Ġthe Ġdec oder Ġbecause Ġthe ĠLM Ġneeds Ġto Ġbe Ġinvoked Ġrepeatedly . ĠOur Ġcurrent Ġimplementation Ġdoes Ġnot Ġcache ĠLM Ġoperations Ġfor Ġthe Ġprevious Ġstate Ġand Ġcan Ġbe Ġmade Ġmuch Ġfaster . </s> The Ġbaseline Ġuses Ġa ĠB PE Ġvocabulary Ġestimated Ġon Ġthe Ġlanguage Ġmodel Ġcorpor a Ġ( Â§ ĠSEC REF 3 Ġ). ĠAppendix ĠSEC REF 6 Ġshows Ġthat Ġthis Ġvocabulary Ġactually Ġleads Ġto Ġsl ig t ly Ġbetter Ġperformance Ġthan Ġa Ġjoint ĠB PE Ġcode Ġlearned Ġon Ġthe Ġbite xt Ġas Ġis Ġusual . </s> Next , Ġwe Ġvalidate Ġour Ġfindings Ġon Ġthe ĠW MT ' 18 ĠEnglish - Turkish Ġtask Ġfor Ġwhich Ġthe Ġbite xt Ġis Ġtruly Ġlimited Ġ( 208 K Ġsentence - p airs ). ĠWe Ġuse Ġthe Ġlanguage Ġmodel Ġvoc ab Ġfor Ġthe Ġthe ĠEnglish Ġside Ġof Ġthe Ġbite xt Ġand Ġa ĠB PE Ġvocabulary Ġlearned Ġon Ġthe ĠTurkish Ġside . ĠTable ĠT AB REF 15 Ġshows Ġthat ĠEL Mo Ġembed d ings Ġfor Ġthe Ġenc oder Ġimprove ĠEnglish - Turkish Ġtranslation . </s> Abstract ive Ġsummar ization </s> Following ĠB IB REF 11 Ġ, Ġwe Ġexperiment Ġon Ġthe Ġnon - an onym ized Ġversion Ġof Ġ. ĠWhen Ġgenerating Ġsumm aries , Ġwe Ġfollow Ġstandard Ġpractice Ġof Ġtuning Ġthe Ġmaximum Ġoutput Ġlength Ġand Ġdis allow Ġrepeating Ġthe Ġsame Ġtrig ram ĠB IB REF 27 Ġ, ĠB IB REF 14 Ġ. ĠFor Ġthis Ġtask Ġwe Ġtrain Ġlanguage Ġmodel Ġrepresentations Ġon Ġthe Ġcombination Ġof Ġnew sc rawl Ġand Ġthe Ġtraining Ġdata . ĠTable ĠT AB REF 16 Ġshows Ġthat Ġpre - trained Ġembed d ings Ġcan Ġsignificantly Ġimprove Ġon Ġtop Ġof Ġa Ġstrong Ġbaseline Ġtransformer . ĠWe Ġalso Ġcompare Ġto ĠB IB REF 26 Ġwho Ġuse Ġa Ġtask - specific Ġarchitecture Ġcompared Ġto Ġour Ġgeneric Ġsequence Ġto Ġsequence Ġbaseline . ĠPre - trained Ġrepresentations Ġare Ġcomplementary Ġto Ġtheir Ġmethod . </s> Conclusion </s> We Ġpresented Ġan Ġanalysis Ġof Ġdifferent Ġstrategies Ġto Ġadd Ġpre - trained Ġlanguage Ġmodel Ġrepresentations Ġto Ġsequence Ġto Ġsequence Ġmodels Ġfor Ġneural Ġmachine Ġtranslation Ġand Ġabstract ive Ġdocument Ġsummar ization . ĠAdding Ġpre - trained Ġrepresentations Ġis Ġvery Ġeffective Ġfor Ġthe Ġenc oder Ġnetwork Ġand Ġwhile Ġreturns Ġdiminish Ġwhen Ġmore Ġlabeled Ġdata Ġbecomes Ġavailable , Ġwe Ġstill Ġobserve Ġimprovements Ġwhen Ġmillions Ġof Ġexamples Ġare Ġavailable . ĠIn Ġfuture Ġresearch Ġwe Ġwill Ġinvestigate Ġways Ġto Ġimprove Ġthe Ġdec oder Ġwith Ġpre - trained Ġrepresentations .
Introduction Pre-training of language models has been shown to provide large improvements for a range of language understanding tasks BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 . The key idea is to train a large generative model on vast corpora and use the resulting representations on tasks for which only limited amounts of labeled data is available. Pre-training of sequence to sequence models has been previously investigated for text classification BIBREF4 but not for text generation. In neural machine translation, there has been work on transferring representations from high-resource language pairs to low-resource settings BIBREF5 . In this paper, we apply pre-trained representations from language models to language generation tasks that can be modeled by sequence to sequence architectures. Previous work on integrating language models with sequence to sequence models focused on the decoder network and added language model representations right before the output of the decoder BIBREF6 . We extend their study by investigating several other strategies such as inputting ELMo-style representations BIBREF0 or fine-tuning the language model (§ SECREF2 ). Our experiments rely on strong transformer-based language models trained on up to six billion tokens (§ SECREF3 ). We present a detailed study of various strategies in different simulated labeled training data scenarios and observe the largest improvements in low-resource settings but gains of over 1 BLEU are still possible when five million sentence-pairs are available. The most successful strategy to integrate pre-trained representations is as input to the encoder network (§ SECREF4 ). Strategies to add representations We consider augmenting a standard sequence to sequence model with pre-trained representations following an ELMo-style regime (§ SECREF2 ) as well as by fine-tuning the language model (§ SECREF3 ). ELMo augmentation The ELMo approach of BIBREF0 forms contextualized word embeddings based on language model representations without adjusting the actual language model parameters. Specifically, the ELMo module contains a set of parameters INLINEFORM0 to form a linear combination of the INLINEFORM1 layers of the language model: ELMo = INLINEFORM2 where INLINEFORM3 is a learned scalar, INLINEFORM4 is a constant to normalize the INLINEFORM5 to sum to one and INLINEFORM6 is the output of the INLINEFORM7 -th language model layer; the module also considers the input word embeddings of the language model. We also apply layer normalization BIBREF7 to each INLINEFORM8 before computing ELMo vectors. We experiment with an ELMo module to input contextualized embeddings either to the encoder () or the decoder (). This provides word representations specific to the current input sentence and these representations have been trained on much more data than is available for the text generation task. Fine-tuning approach Fine-tuning the pre-trained representations adjusts the language model parameters by the learning signal of the end-task BIBREF1 , BIBREF3 . We replace learned input word embeddings in the encoder network with the output of the language model (). Specifically, we use the language model representation of the layer before the softmax and feed it to the encoder. We also add dropout to the language model output. Tuning separate learning rates for the language model and the sequence to sequence model may lead to better performance but we leave this to future work. However, we do tune the number of encoder blocks INLINEFORM0 as we found this important to obtain good accuracy for this setting. We apply the same strategy to the decoder: we input language model representations to the decoder network and fine-tune the language model when training the sequence to sequence model (). Datasets We train language models on two languages: One model is estimated on the German newscrawl distributed by WMT'18 comprising 260M sentences or 6B tokens. Another model is trained on the English newscrawl data comprising 193M sentences or 5B tokens. We learn a joint Byte-Pair-Encoding (BPE; Sennrich et al., 2016) vocabulary of 37K types on the German and English newscrawl and train the language models with this vocabulary. We consider two benchmarks: Most experiments are run on the WMT'18 English-German (en-de) news translation task and we validate our findings on the WMT'18 English-Turkish (en-tr) news task. For WMT'18 English-German, the training corpus consists of all available bitext excluding the ParaCrawl corpus and we remove sentences longer than 250 tokens as well as sentence-pairs with a source/target length ratio exceeding 1.5. This results in 5.18M sentence pairs. We tokenize all data with the Moses tokenizer BIBREF8 and apply the BPE vocabulary learned on the monolingual corpora. For WMT'18 English-Turkish, we use all of the available bitext comprising 208K sentence-pairs without any filtering. We develop on newstest2017 and test on newstest2018. For en-tr we only experiment with adding representations to the encoder and therefore apply the language model vocabulary to the source side. For the target vocabulary we learn a BPE code with 32K merge operations on the Turkish side of the bitext. Both datasets are evaluated in terms of case-sensitive de-tokenized BLEU BIBREF9 , BIBREF10 . We consider the abstractive document summarization task comprising over 280K news articles paired with multi-sentence summaries. is a widely used dataset for abstractive text summarization. Following BIBREF11 , we report results on the non-anonymized version of rather than the entity-anonymized version BIBREF12 , BIBREF13 because the language model was trained on full text. Articles are truncated to 400 tokens BIBREF11 and we use a BPE vocabulary of 32K types BIBREF14 . We evaluate in terms of F1-Rouge, that is Rouge-1, Rouge-2 and Rouge-L BIBREF15 . Language model pre-training We consider two types of architectures: a bi-directional language model to augment the sequence to sequence encoder and a uni-directional model to augment the decoder. Both use self-attention BIBREF16 and the uni-directional model contains INLINEFORM0 transformer blocks, followed by a word classifier to predict the next word on the right. The bi-directional model solves a cloze-style token prediction task at training time BIBREF17 . The model consists of two towers, the forward tower operates left-to-right and the tower operating right-to-left as backward tower; each tower contains INLINEFORM1 transformer blocks. The forward and backward representations are combined via a self-attention module and the output of this module is used to predict the token at position INLINEFORM2 . The model has access to the entire input surrounding the current target token. Models use the standard settings for the Big Transformer BIBREF16 . The bi-directional model contains 353M parameters and the uni-directional model 190M parameters. Both models were trained for 1M steps using Nesterov's accelerated gradient BIBREF18 with momentum INLINEFORM3 following BIBREF19 . The learning rate is linearly warmed up from INLINEFORM4 to 1 for 16K steps and then annealed using a cosine learning rate schedule with a single phase to 0.0001 BIBREF20 . We train on 32 Nvidia V100 SXM2 GPUs and use the NCCL2 library as well as the torch distributed package for inter-GPU communication. Training relies on 16-bit floating point operations BIBREF21 and it took six days for the bi-directional model and four days for the uni-directional model. Sequence to sequence model We use the transformer implementation of the fairseq toolkit BIBREF22 . The WMT en-de and en-tr experiments are based on the Big Transformer sequence to sequence architecture with 6 blocks in the encoder and decoder. For abstractive summarization we use a base transformer model BIBREF16 . We tune dropout values of between 0.1 and 0.4 on the validation set. Models are optimized with Adam BIBREF23 using INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 and we use the same learning rate schedule as BIBREF16 ; we perform 10K-200K depending on bitext size. All models use label smoothing with a uniform prior distribution over the vocabulary INLINEFORM3 BIBREF24 , BIBREF25 . We run experiments on 8 GPUs and generate translations with a beam of size 5. Machine translation We first present a comparison of the various strategies in different simulated parallel corpus size settings. For each experiment, we tune the dropout applied to the language model representations, and we reduce the number of optimizer steps for smaller bitext setups as models converge faster; all other hyper-parameters are equal between setups. Our baseline is a Big Transformer model and we also consider a variant where we share token embeddings between the encoder and decoder (; Inan et al., 2016; Press & Wolf, 2016). Figure FIGREF10 shows results averaged over six test sets relative to the baseline which does not share source and target embeddings (Appendix SECREF6 shows a detailed breakdown). performs very well with little labeled data but the gains erode to practically zero in large bitext settings. Pre-trained language model representations are most effective in low bitext setups. The best performing strategy is ELMo embeddings input to the encoder (). This improves the baseline by 3.8 BLEU in the 160K bitext setting and it still improves the 5.2M setting by over 1 BLEU. We further improve by sharing learned word representations in the decoder by tying input and output embeddings (). This configuration performs even better than with a gain of 5.3 BLEU in the 160K setup. Sharing decoder embeddings is equally applicable to . Language model representations are much less effective in the decoder: improves the 160K bitext setup but yields no improvements thereafter and performs even worse. We conjecture that pre-trained representations give much easier wins in the encoder. Table TABREF14 shows additional results on newstest2018. Pre-trained representations mostly impacts the training time of the sequence to sequence model (see Appendix SECREF7 ): slows throughput during training by about 5.3x and is even slower because of the need to backpropagate through the LM for fine-tuning (9.2x). However, inference is only 12-14% slower than the baseline when adding pre-trained embeddings to the encoder (, ). This is because the LM computation can be paralelized for all input tokens. Inference is much slower when adding representations to the decoder because the LM needs to be invoked repeatedly. Our current implementation does not cache LM operations for the previous state and can be made much faster. The baseline uses a BPE vocabulary estimated on the language model corpora (§ SECREF3 ). Appendix SECREF6 shows that this vocabulary actually leads to sligtly better performance than a joint BPE code learned on the bitext as is usual. Next, we validate our findings on the WMT'18 English-Turkish task for which the bitext is truly limited (208K sentence-pairs). We use the language model vocab for the the English side of the bitext and a BPE vocabulary learned on the Turkish side. Table TABREF15 shows that ELMo embeddings for the encoder improve English-Turkish translation. Abstractive summarization Following BIBREF11 , we experiment on the non-anonymized version of . When generating summaries, we follow standard practice of tuning the maximum output length and disallow repeating the same trigram BIBREF27 , BIBREF14 . For this task we train language model representations on the combination of newscrawl and the training data. Table TABREF16 shows that pre-trained embeddings can significantly improve on top of a strong baseline transformer. We also compare to BIBREF26 who use a task-specific architecture compared to our generic sequence to sequence baseline. Pre-trained representations are complementary to their method. Conclusion We presented an analysis of different strategies to add pre-trained language model representations to sequence to sequence models for neural machine translation and abstractive document summarization. Adding pre-trained representations is very effective for the encoder network and while returns diminish when more labeled data becomes available, we still observe improvements when millions of examples are available. In future research we will investigate ways to improve the decoder with pre-trained representations.
What is the previous state-of-the-art in summarization?
ab0fd94dfc291cf3e54e9b7a7f78b852ddc1a797
What Ġis Ġthe Ġprevious Ġstate - of - the - art Ġin Ġsummar ization ?
BI B REF 26 Ġ
<s> What Ġis Ġthe Ġprevious Ġstate - of - the - art Ġin Ġsummar ization ? </s> Introduction </s> Pre - training Ġof Ġlanguage Ġmodels Ġhas Ġbeen Ġshown Ġto Ġprovide Ġlarge Ġimprovements Ġfor Ġa Ġrange Ġof Ġlanguage Ġunderstanding Ġtasks ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ. ĠThe Ġkey Ġidea Ġis Ġto Ġtrain Ġa Ġlarge Ġgener ative Ġmodel Ġon Ġvast Ġcorpor a Ġand Ġuse Ġthe Ġresulting Ġrepresentations Ġon Ġtasks Ġfor Ġwhich Ġonly Ġlimited Ġamounts Ġof Ġlabeled Ġdata Ġis Ġavailable . ĠPre - training Ġof Ġsequence Ġto Ġsequence Ġmodels Ġhas Ġbeen Ġpreviously Ġinvestigated Ġfor Ġtext Ġclassification ĠB IB REF 4 Ġbut Ġnot Ġfor Ġtext Ġgeneration . ĠIn Ġneural Ġmachine Ġtranslation , Ġthere Ġhas Ġbeen Ġwork Ġon Ġtransferring Ġrepresentations Ġfrom Ġhigh - resource Ġlanguage Ġpairs Ġto Ġlow - resource Ġsettings ĠB IB REF 5 Ġ. </s> In Ġthis Ġpaper , Ġwe Ġapply Ġpre - trained Ġrepresentations Ġfrom Ġlanguage Ġmodels Ġto Ġlanguage Ġgeneration Ġtasks Ġthat Ġcan Ġbe Ġmodeled Ġby Ġsequence Ġto Ġsequence Ġarchitectures . ĠPrevious Ġwork Ġon Ġintegrating Ġlanguage Ġmodels Ġwith Ġsequence Ġto Ġsequence Ġmodels Ġfocused Ġon Ġthe Ġdec oder Ġnetwork Ġand Ġadded Ġlanguage Ġmodel Ġrepresentations Ġright Ġbefore Ġthe Ġoutput Ġof Ġthe Ġdec oder ĠB IB REF 6 Ġ. ĠWe Ġextend Ġtheir Ġstudy Ġby Ġinvestigating Ġseveral Ġother Ġstrategies Ġsuch Ġas Ġinput ting ĠEL Mo - style Ġrepresentations ĠB IB REF 0 Ġor Ġfine - tun ing Ġthe Ġlanguage Ġmodel Ġ( Â§ ĠSEC REF 2 Ġ). </s> Our Ġexperiments Ġrely Ġon Ġstrong Ġtransformer - based Ġlanguage Ġmodels Ġtrained Ġon Ġup Ġto Ġsix Ġbillion Ġtokens Ġ( Â§ ĠSEC REF 3 Ġ). ĠWe Ġpresent Ġa Ġdetailed Ġstudy Ġof Ġvarious Ġstrategies Ġin Ġdifferent Ġsimulated Ġlabeled Ġtraining Ġdata Ġscenarios Ġand Ġobserve Ġthe Ġlargest Ġimprovements Ġin Ġlow - resource Ġsettings Ġbut Ġgains Ġof Ġover Ġ1 ĠB LE U Ġare Ġstill Ġpossible Ġwhen Ġfive Ġmillion Ġsentence - p airs Ġare Ġavailable . ĠThe Ġmost Ġsuccessful Ġstrategy Ġto Ġintegrate Ġpre - trained Ġrepresentations Ġis Ġas Ġinput Ġto Ġthe Ġenc oder Ġnetwork Ġ( Â§ ĠSEC REF 4 Ġ). </s> Str ateg ies Ġto Ġadd Ġrepresentations </s> We Ġconsider Ġaugment ing Ġa Ġstandard Ġsequence Ġto Ġsequence Ġmodel Ġwith Ġpre - trained Ġrepresentations Ġfollowing Ġan ĠEL Mo - style Ġregime Ġ( Â§ ĠSEC REF 2 Ġ) Ġas Ġwell Ġas Ġby Ġfine - tun ing Ġthe Ġlanguage Ġmodel Ġ( Â§ ĠSEC REF 3 Ġ). </s> EL Mo Ġaug mentation </s> The ĠEL Mo Ġapproach Ġof ĠB IB REF 0 Ġforms Ġcontextual ized Ġword Ġembed d ings Ġbased Ġon Ġlanguage Ġmodel Ġrepresentations Ġwithout Ġadjusting Ġthe Ġactual Ġlanguage Ġmodel Ġparameters . ĠSpecifically , Ġthe ĠEL Mo Ġmodule Ġcontains Ġa Ġset Ġof Ġparameters ĠIN LINE FORM 0 Ġto Ġform Ġa Ġlinear Ġcombination Ġof Ġthe ĠIN LINE FORM 1 Ġlayers Ġof Ġthe Ġlanguage Ġmodel : ĠEL Mo Ġ= ĠIN LINE FORM 2 Ġwhere ĠIN LINE FORM 3 Ġis Ġa Ġlearned Ġscal ar , ĠIN LINE FORM 4 Ġis Ġa Ġconstant Ġto Ġnormal ize Ġthe ĠIN LINE FORM 5 Ġto Ġsum Ġto Ġone Ġand ĠIN LINE FORM 6 Ġis Ġthe Ġoutput Ġof Ġthe ĠIN LINE FORM 7 Ġ- th Ġlanguage Ġmodel Ġlayer ; Ġthe Ġmodule Ġalso Ġconsiders Ġthe Ġinput Ġword Ġembed d ings Ġof Ġthe Ġlanguage Ġmodel . ĠWe Ġalso Ġapply Ġlayer Ġnormal ization ĠB IB REF 7 Ġto Ġeach ĠIN LINE FORM 8 Ġbefore Ġcomputing ĠEL Mo Ġvectors . </s> We Ġexperiment Ġwith Ġan ĠEL Mo Ġmodule Ġto Ġinput Ġcontextual ized Ġembed d ings Ġeither Ġto Ġthe Ġenc oder Ġ() Ġor Ġthe Ġdec oder Ġ(). ĠThis Ġprovides Ġword Ġrepresentations Ġspecific Ġto Ġthe Ġcurrent Ġinput Ġsentence Ġand Ġthese Ġrepresentations Ġhave Ġbeen Ġtrained Ġon Ġmuch Ġmore Ġdata Ġthan Ġis Ġavailable Ġfor Ġthe Ġtext Ġgeneration Ġtask . </s> Fine - tun ing Ġapproach </s> Fine - tun ing Ġthe Ġpre - trained Ġrepresentations Ġadjusts Ġthe Ġlanguage Ġmodel Ġparameters Ġby Ġthe Ġlearning Ġsignal Ġof Ġthe Ġend - task ĠB IB REF 1 Ġ, ĠB IB REF 3 Ġ. ĠWe Ġreplace Ġlearned Ġinput Ġword Ġembed d ings Ġin Ġthe Ġenc oder Ġnetwork Ġwith Ġthe Ġoutput Ġof Ġthe Ġlanguage Ġmodel Ġ(). ĠSpecifically , Ġwe Ġuse Ġthe Ġlanguage Ġmodel Ġrepresentation Ġof Ġthe Ġlayer Ġbefore Ġthe Ġsoft max Ġand Ġfeed Ġit Ġto Ġthe Ġenc oder . ĠWe Ġalso Ġadd Ġdrop out Ġto Ġthe Ġlanguage Ġmodel Ġoutput . ĠTun ing Ġseparate Ġlearning Ġrates Ġfor Ġthe Ġlanguage Ġmodel Ġand Ġthe Ġsequence Ġto Ġsequence Ġmodel Ġmay Ġlead Ġto Ġbetter Ġperformance Ġbut Ġwe Ġleave Ġthis Ġto Ġfuture Ġwork . ĠHowever , Ġwe Ġdo Ġtune Ġthe Ġnumber Ġof Ġenc oder Ġblocks ĠIN LINE FORM 0 Ġas Ġwe Ġfound Ġthis Ġimportant Ġto Ġobtain Ġgood Ġaccuracy Ġfor Ġthis Ġsetting . ĠWe Ġapply Ġthe Ġsame Ġstrategy Ġto Ġthe Ġdec oder : Ġwe Ġinput Ġlanguage Ġmodel Ġrepresentations Ġto Ġthe Ġdec oder Ġnetwork Ġand Ġfine - t une Ġthe Ġlanguage Ġmodel Ġwhen Ġtraining Ġthe Ġsequence Ġto Ġsequence Ġmodel Ġ(). </s> Dat as ets </s> We Ġtrain Ġlanguage Ġmodels Ġon Ġtwo Ġlanguages : ĠOne Ġmodel Ġis Ġestimated Ġon Ġthe ĠGerman Ġnew sc rawl Ġdistributed Ġby ĠW MT ' 18 Ġcomprising Ġ260 M Ġsentences Ġor Ġ6 B Ġtokens . ĠAnother Ġmodel Ġis Ġtrained Ġon Ġthe ĠEnglish Ġnew sc rawl Ġdata Ġcomprising Ġ193 M Ġsentences Ġor Ġ5 B Ġtokens . ĠWe Ġlearn Ġa Ġjoint ĠByte - P air - Enc oding Ġ( B PE ; ĠS enn rich Ġet Ġal ., Ġ2016 ) Ġvocabulary Ġof Ġ37 K Ġtypes Ġon Ġthe ĠGerman Ġand ĠEnglish Ġnew sc rawl Ġand Ġtrain Ġthe Ġlanguage Ġmodels Ġwith Ġthis Ġvocabulary . </s> We Ġconsider Ġtwo Ġbenchmarks : ĠMost Ġexperiments Ġare Ġrun Ġon Ġthe ĠW MT ' 18 ĠEnglish - German Ġ( en - de ) Ġnews Ġtranslation Ġtask Ġand Ġwe Ġvalidate Ġour Ġfindings Ġon Ġthe ĠW MT ' 18 ĠEnglish - Turkish Ġ( en - tr ) Ġnews Ġtask . ĠFor ĠW MT ' 18 ĠEnglish - German , Ġthe Ġtraining Ġcorpus Ġconsists Ġof Ġall Ġavailable Ġbite xt Ġexcluding Ġthe ĠPar a C rawl Ġcorpus Ġand Ġwe Ġremove Ġsentences Ġlonger Ġthan Ġ250 Ġtokens Ġas Ġwell Ġas Ġsentence - p airs Ġwith Ġa Ġsource / target Ġlength Ġratio Ġexceeding Ġ1 . 5 . ĠThis Ġresults Ġin Ġ5 . 18 M Ġsentence Ġpairs . ĠWe Ġtoken ize Ġall Ġdata Ġwith Ġthe ĠMoses Ġtoken izer ĠB IB REF 8 Ġand Ġapply Ġthe ĠB PE Ġvocabulary Ġlearned Ġon Ġthe Ġmon oling ual Ġcorpor a . </s> For ĠW MT ' 18 ĠEnglish - Turkish , Ġwe Ġuse Ġall Ġof Ġthe Ġavailable Ġbite xt Ġcomprising Ġ208 K Ġsentence - p airs Ġwithout Ġany Ġfiltering . ĠWe Ġdevelop Ġon Ġnew st est 2017 Ġand Ġtest Ġon Ġnew st est 2018 . ĠFor Ġen - tr Ġwe Ġonly Ġexperiment Ġwith Ġadding Ġrepresentations Ġto Ġthe Ġenc oder Ġand Ġtherefore Ġapply Ġthe Ġlanguage Ġmodel Ġvocabulary Ġto Ġthe Ġsource Ġside . ĠFor Ġthe Ġtarget Ġvocabulary Ġwe Ġlearn Ġa ĠB PE Ġcode Ġwith Ġ32 K Ġmerge Ġoperations Ġon Ġthe ĠTurkish Ġside Ġof Ġthe Ġbite xt . ĠBoth Ġdatasets Ġare Ġevaluated Ġin Ġterms Ġof Ġcase - sensitive Ġde - token ized ĠB LE U ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ. </s> We Ġconsider Ġthe Ġabstract ive Ġdocument Ġsummar ization Ġtask Ġcomprising Ġover Ġ280 K Ġnews Ġarticles Ġpaired Ġwith Ġmulti - sent ence Ġsumm aries . Ġis Ġa Ġwidely Ġused Ġdataset Ġfor Ġabstract ive Ġtext Ġsummar ization . ĠFollowing ĠB IB REF 11 Ġ, Ġwe Ġreport Ġresults Ġon Ġthe Ġnon - an onym ized Ġversion Ġof Ġrather Ġthan Ġthe Ġentity - an onym ized Ġversion ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġbecause Ġthe Ġlanguage Ġmodel Ġwas Ġtrained Ġon Ġfull Ġtext . ĠArticles Ġare Ġtrunc ated Ġto Ġ400 Ġtokens ĠB IB REF 11 Ġand Ġwe Ġuse Ġa ĠB PE Ġvocabulary Ġof Ġ32 K Ġtypes ĠB IB REF 14 Ġ. ĠWe Ġevaluate Ġin Ġterms Ġof ĠF 1 - R ou ge , Ġthat Ġis ĠRouge - 1 , ĠRouge - 2 Ġand ĠRouge - L ĠB IB REF 15 Ġ. </s> Language Ġmodel Ġpre - training </s> We Ġconsider Ġtwo Ġtypes Ġof Ġarchitectures : Ġa Ġbi - direction al Ġlanguage Ġmodel Ġto Ġaugment Ġthe Ġsequence Ġto Ġsequence Ġenc oder Ġand Ġa Ġun i - direction al Ġmodel Ġto Ġaugment Ġthe Ġdec oder . ĠBoth Ġuse Ġself - att ention ĠB IB REF 16 Ġand Ġthe Ġun i - direction al Ġmodel Ġcontains ĠIN LINE FORM 0 Ġtransformer Ġblocks , Ġfollowed Ġby Ġa Ġword Ġclass ifier Ġto Ġpredict Ġthe Ġnext Ġword Ġon Ġthe Ġright . ĠThe Ġbi - direction al Ġmodel Ġsolves Ġa Ġclo ze - style Ġtoken Ġprediction Ġtask Ġat Ġtraining Ġtime ĠB IB REF 17 Ġ. ĠThe Ġmodel Ġconsists Ġof Ġtwo Ġtowers , Ġthe Ġforward Ġtower Ġoperates Ġleft - to - right Ġand Ġthe Ġtower Ġoperating Ġright - to - left Ġas Ġbackward Ġtower ; Ġeach Ġtower Ġcontains ĠIN LINE FORM 1 Ġtransformer Ġblocks . ĠThe Ġforward Ġand Ġbackward Ġrepresentations Ġare Ġcombined Ġvia Ġa Ġself - att ention Ġmodule Ġand Ġthe Ġoutput Ġof Ġthis Ġmodule Ġis Ġused Ġto Ġpredict Ġthe Ġtoken Ġat Ġposition ĠIN LINE FORM 2 Ġ. ĠThe Ġmodel Ġhas Ġaccess Ġto Ġthe Ġentire Ġinput Ġsurrounding Ġthe Ġcurrent Ġtarget Ġtoken . ĠModels Ġuse Ġthe Ġstandard Ġsettings Ġfor Ġthe ĠBig ĠTrans former ĠB IB REF 16 Ġ. ĠThe Ġbi - direction al Ġmodel Ġcontains Ġ353 M Ġparameters Ġand Ġthe Ġun i - direction al Ġmodel Ġ190 M Ġparameters . ĠBoth Ġmodels Ġwere Ġtrained Ġfor Ġ1 M Ġsteps Ġusing ĠN ester ov 's Ġaccelerated Ġgradient ĠB IB REF 18 Ġwith Ġmomentum ĠIN LINE FORM 3 Ġfollowing ĠB IB REF 19 Ġ. ĠThe Ġlearning Ġrate Ġis Ġlin early Ġwarmed Ġup Ġfrom ĠIN LINE FORM 4 Ġto Ġ1 Ġfor Ġ16 K Ġsteps Ġand Ġthen Ġan ne aled Ġusing Ġa Ġcos ine Ġlearning Ġrate Ġschedule Ġwith Ġa Ġsingle Ġphase Ġto Ġ0 . 0001 ĠB IB REF 20 Ġ. ĠWe Ġtrain Ġon Ġ32 ĠNvidia ĠV 100 ĠS XM 2 ĠGPUs Ġand Ġuse Ġthe ĠN CC L 2 Ġlibrary Ġas Ġwell Ġas Ġthe Ġtorch Ġdistributed Ġpackage Ġfor Ġinter - GPU Ġcommunication . ĠTraining Ġrelies Ġon Ġ16 - bit Ġfloating Ġpoint Ġoperations ĠB IB REF 21 Ġand Ġit Ġtook Ġsix Ġdays Ġfor Ġthe Ġbi - direction al Ġmodel Ġand Ġfour Ġdays Ġfor Ġthe Ġun i - direction al Ġmodel . </s> Sequ ence Ġto Ġsequence Ġmodel </s> We Ġuse Ġthe Ġtransformer Ġimplementation Ġof Ġthe Ġfair seq Ġtool kit ĠB IB REF 22 Ġ. ĠThe ĠW MT Ġen - de Ġand Ġen - tr Ġexperiments Ġare Ġbased Ġon Ġthe ĠBig ĠTrans former Ġsequence Ġto Ġsequence Ġarchitecture Ġwith Ġ6 Ġblocks Ġin Ġthe Ġenc oder Ġand Ġdec oder . ĠFor Ġabstract ive Ġsummar ization Ġwe Ġuse Ġa Ġbase Ġtransformer Ġmodel ĠB IB REF 16 Ġ. ĠWe Ġtune Ġdrop out Ġvalues Ġof Ġbetween Ġ0 . 1 Ġand Ġ0 . 4 Ġon Ġthe Ġvalidation Ġset . ĠModels Ġare Ġoptimized Ġwith ĠAdam ĠB IB REF 23 Ġusing ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, Ġand ĠIN LINE FORM 2 Ġand Ġwe Ġuse Ġthe Ġsame Ġlearning Ġrate Ġschedule Ġas ĠB IB REF 16 Ġ; Ġwe Ġperform Ġ10 K - 200 K Ġdepending Ġon Ġbite xt Ġsize . ĠAll Ġmodels Ġuse Ġlabel Ġsmoot hing Ġwith Ġa Ġuniform Ġprior Ġdistribution Ġover Ġthe Ġvocabulary ĠIN LINE FORM 3 ĠB IB REF 24 Ġ, ĠB IB REF 25 Ġ. ĠWe Ġrun Ġexperiments Ġon Ġ8 ĠGPUs Ġand Ġgenerate Ġtranslations Ġwith Ġa Ġbeam Ġof Ġsize Ġ5 . </s> Machine Ġtranslation </s> We Ġfirst Ġpresent Ġa Ġcomparison Ġof Ġthe Ġvarious Ġstrategies Ġin Ġdifferent Ġsimulated Ġparallel Ġcorpus Ġsize Ġsettings . ĠFor Ġeach Ġexperiment , Ġwe Ġtune Ġthe Ġdrop out Ġapplied Ġto Ġthe Ġlanguage Ġmodel Ġrepresentations , Ġand Ġwe Ġreduce Ġthe Ġnumber Ġof Ġoptim izer Ġsteps Ġfor Ġsmaller Ġbite xt Ġsetups Ġas Ġmodels Ġconverge Ġfaster ; Ġall Ġother Ġhyper - param eters Ġare Ġequal Ġbetween Ġsetups . ĠOur Ġbaseline Ġis Ġa ĠBig ĠTrans former Ġmodel Ġand Ġwe Ġalso Ġconsider Ġa Ġvariant Ġwhere Ġwe Ġshare Ġtoken Ġembed d ings Ġbetween Ġthe Ġenc oder Ġand Ġdec oder Ġ( ; ĠIn an Ġet Ġal ., Ġ2016 ; ĠPress Ġ& ĠWolf , Ġ2016 ). </s> Figure ĠFIG REF 10 Ġshows Ġresults Ġaveraged Ġover Ġsix Ġtest Ġsets Ġrelative Ġto Ġthe Ġbaseline Ġwhich Ġdoes Ġnot Ġshare Ġsource Ġand Ġtarget Ġembed d ings Ġ( App endix ĠSEC REF 6 Ġshows Ġa Ġdetailed Ġbreakdown ). Ġperforms Ġvery Ġwell Ġwith Ġlittle Ġlabeled Ġdata Ġbut Ġthe Ġgains Ġer ode Ġto Ġpractically Ġzero Ġin Ġlarge Ġbite xt Ġsettings . ĠPre - trained Ġlanguage Ġmodel Ġrepresentations Ġare Ġmost Ġeffective Ġin Ġlow Ġbite xt Ġsetups . ĠThe Ġbest Ġperforming Ġstrategy Ġis ĠEL Mo Ġembed d ings Ġinput Ġto Ġthe Ġenc oder Ġ(). ĠThis Ġimproves Ġthe Ġbaseline Ġby Ġ3 . 8 ĠB LE U Ġin Ġthe Ġ160 K Ġbite xt Ġsetting Ġand Ġit Ġstill Ġimproves Ġthe Ġ5 . 2 M Ġsetting Ġby Ġover Ġ1 ĠB LE U . </s> We Ġfurther Ġimprove Ġby Ġsharing Ġlearned Ġword Ġrepresentations Ġin Ġthe Ġdec oder Ġby Ġtying Ġinput Ġand Ġoutput Ġembed d ings Ġ(). ĠThis Ġconfiguration Ġperforms Ġeven Ġbetter Ġthan Ġwith Ġa Ġgain Ġof Ġ5 . 3 ĠB LE U Ġin Ġthe Ġ160 K Ġsetup . ĠSharing Ġdec oder Ġembed d ings Ġis Ġequally Ġapplicable Ġto Ġ. ĠLanguage Ġmodel Ġrepresentations Ġare Ġmuch Ġless Ġeffective Ġin Ġthe Ġdec oder : Ġimproves Ġthe Ġ160 K Ġbite xt Ġsetup Ġbut Ġyields Ġno Ġimprovements Ġthereafter Ġand Ġperforms Ġeven Ġworse . ĠWe Ġconjecture Ġthat Ġpre - trained Ġrepresentations Ġgive Ġmuch Ġeasier Ġwins Ġin Ġthe Ġenc oder . ĠTable ĠT AB REF 14 Ġshows Ġadditional Ġresults Ġon Ġnew st est 2018 . </s> Pre - trained Ġrepresentations Ġmostly Ġimpacts Ġthe Ġtraining Ġtime Ġof Ġthe Ġsequence Ġto Ġsequence Ġmodel Ġ( see ĠAppendix ĠSEC REF 7 Ġ): Ġslows Ġthroughput Ġduring Ġtraining Ġby Ġabout Ġ5 . 3 x Ġand Ġis Ġeven Ġslower Ġbecause Ġof Ġthe Ġneed Ġto Ġback prop agate Ġthrough Ġthe ĠLM Ġfor Ġfine - tun ing Ġ( 9 . 2 x ). ĠHowever , Ġinference Ġis Ġonly Ġ12 - 14 % Ġslower Ġthan Ġthe Ġbaseline Ġwhen Ġadding Ġpre - trained Ġembed d ings Ġto Ġthe Ġenc oder Ġ( , Ġ). ĠThis Ġis Ġbecause Ġthe ĠLM Ġcomputation Ġcan Ġbe Ġpar al el ized Ġfor Ġall Ġinput Ġtokens . ĠIn ference Ġis Ġmuch Ġslower Ġwhen Ġadding Ġrepresentations Ġto Ġthe Ġdec oder Ġbecause Ġthe ĠLM Ġneeds Ġto Ġbe Ġinvoked Ġrepeatedly . ĠOur Ġcurrent Ġimplementation Ġdoes Ġnot Ġcache ĠLM Ġoperations Ġfor Ġthe Ġprevious Ġstate Ġand Ġcan Ġbe Ġmade Ġmuch Ġfaster . </s> The Ġbaseline Ġuses Ġa ĠB PE Ġvocabulary Ġestimated Ġon Ġthe Ġlanguage Ġmodel Ġcorpor a Ġ( Â§ ĠSEC REF 3 Ġ). ĠAppendix ĠSEC REF 6 Ġshows Ġthat Ġthis Ġvocabulary Ġactually Ġleads Ġto Ġsl ig t ly Ġbetter Ġperformance Ġthan Ġa Ġjoint ĠB PE Ġcode Ġlearned Ġon Ġthe Ġbite xt Ġas Ġis Ġusual . </s> Next , Ġwe Ġvalidate Ġour Ġfindings Ġon Ġthe ĠW MT ' 18 ĠEnglish - Turkish Ġtask Ġfor Ġwhich Ġthe Ġbite xt Ġis Ġtruly Ġlimited Ġ( 208 K Ġsentence - p airs ). ĠWe Ġuse Ġthe Ġlanguage Ġmodel Ġvoc ab Ġfor Ġthe Ġthe ĠEnglish Ġside Ġof Ġthe Ġbite xt Ġand Ġa ĠB PE Ġvocabulary Ġlearned Ġon Ġthe ĠTurkish Ġside . ĠTable ĠT AB REF 15 Ġshows Ġthat ĠEL Mo Ġembed d ings Ġfor Ġthe Ġenc oder Ġimprove ĠEnglish - Turkish Ġtranslation . </s> Abstract ive Ġsummar ization </s> Following ĠB IB REF 11 Ġ, Ġwe Ġexperiment Ġon Ġthe Ġnon - an onym ized Ġversion Ġof Ġ. ĠWhen Ġgenerating Ġsumm aries , Ġwe Ġfollow Ġstandard Ġpractice Ġof Ġtuning Ġthe Ġmaximum Ġoutput Ġlength Ġand Ġdis allow Ġrepeating Ġthe Ġsame Ġtrig ram ĠB IB REF 27 Ġ, ĠB IB REF 14 Ġ. ĠFor Ġthis Ġtask Ġwe Ġtrain Ġlanguage Ġmodel Ġrepresentations Ġon Ġthe Ġcombination Ġof Ġnew sc rawl Ġand Ġthe Ġtraining Ġdata . ĠTable ĠT AB REF 16 Ġshows Ġthat Ġpre - trained Ġembed d ings Ġcan Ġsignificantly Ġimprove Ġon Ġtop Ġof Ġa Ġstrong Ġbaseline Ġtransformer . ĠWe Ġalso Ġcompare Ġto ĠB IB REF 26 Ġwho Ġuse Ġa Ġtask - specific Ġarchitecture Ġcompared Ġto Ġour Ġgeneric Ġsequence Ġto Ġsequence Ġbaseline . ĠPre - trained Ġrepresentations Ġare Ġcomplementary Ġto Ġtheir Ġmethod . </s> Conclusion </s> We Ġpresented Ġan Ġanalysis Ġof Ġdifferent Ġstrategies Ġto Ġadd Ġpre - trained Ġlanguage Ġmodel Ġrepresentations Ġto Ġsequence Ġto Ġsequence Ġmodels Ġfor Ġneural Ġmachine Ġtranslation Ġand Ġabstract ive Ġdocument Ġsummar ization . ĠAdding Ġpre - trained Ġrepresentations Ġis Ġvery Ġeffective Ġfor Ġthe Ġenc oder Ġnetwork Ġand Ġwhile Ġreturns Ġdiminish Ġwhen Ġmore Ġlabeled Ġdata Ġbecomes Ġavailable , Ġwe Ġstill Ġobserve Ġimprovements Ġwhen Ġmillions Ġof Ġexamples Ġare Ġavailable . ĠIn Ġfuture Ġresearch Ġwe Ġwill Ġinvestigate Ġways Ġto Ġimprove Ġthe Ġdec oder Ġwith Ġpre - trained Ġrepresentations .