absolute improvement of 18.2% over the Pointer-Gen baseline
EXTRACTIVE
We then compare different models using the sensationalism score in Table TABREF30. The Pointer-Gen baseline model achieves a 42.6% sensationalism score, which is the minimum that a typical summarization model achieves. By filtering out low-sensational headlines, Pointer-Gen+Same-FT and Pointer-Gen+Pos-FT achieves higher sensationalism scores, which implies the effectiveness of our sensationalism scorer. Our Pointer-Gen+ARL-SEN model achieves the best performance of 60.8%. This is an absolute improvement of 18.2% over the Pointer-Gen baseline. The Chi-square test on the results confirms that Pointer-Gen+ARL-SEN is statistically significantly more sensational than all the other baseline models, with the largest p-value less than 0.01. Also, we find that the test set headlines achieves 57.8% sensationalism score, much larger than Pointer-Gen baseline, which also supports our intuition that generated headlines will be less sensational than the original one. On the other hand, we found that Pointer-Gen+Pos is much worse than other baselines. The reason is that training on sensational samples alone discards around 80% of the whole training set that is also helpful for maintaining relevance and a good language model. It shows the necessity of using RL.
1909.03582
Introduction </s> Head line Ġgeneration Ġis Ġthe Ġprocess Ġof Ġcreating Ġa Ġheadline - style Ġsentence Ġgiven Ġan Ġinput Ġarticle . ĠThe Ġresearch Ġcommunity Ġhas Ġbeen Ġregarding Ġthe Ġtask Ġof Ġheadline Ġgeneration Ġas Ġa Ġsummar ization Ġtask ĠB IB REF 1 , Ġignoring Ġthe Ġfundamental Ġdifferences Ġbetween Ġheadlines Ġand Ġsumm aries . ĠWhile Ġsumm aries Ġaim Ġto Ġcontain Ġmost Ġof Ġthe Ġimportant Ġinformation Ġfrom Ġthe Ġarticles , Ġheadlines Ġdo Ġnot Ġnecessarily Ġneed Ġto . ĠInstead , Ġa Ġgood Ġheadline Ġneeds Ġto Ġcapture Ġpeople 's Ġattention Ġand Ġserve Ġas Ġan Ġirresistible Ġinvitation Ġfor Ġusers Ġto Ġread Ġthrough Ġthe Ġarticle . ĠFor Ġexample , Ġthe Ġheadline ĠâĢ ľ $ 2 ĠBillion ĠWorth Ġof ĠFree ĠMedia Ġfor ĠTrump âĢ Ŀ , Ġwhich Ġgives Ġonly Ġan Ġintriguing Ġhint , Ġis Ġconsidered Ġbetter Ġthan Ġthe Ġsummar ization Ġstyle Ġheadline ĠâĢ ľ Me asuring ĠTrump âĢ Ļ s ĠMedia ĠDomin ance âĢ Ŀ Ġ, Ġas Ġthe Ġformer Ġgets Ġalmost Ġthree Ġtimes Ġthe Ġreaders Ġas Ġthe Ġlatter . ĠGener ating Ġheadlines Ġwith Ġmany Ġclicks Ġis Ġespecially Ġimportant Ġin Ġthis Ġdigital Ġage , Ġbecause Ġmany Ġof Ġthe Ġrevenues Ġof Ġjournalism Ġcome Ġfrom Ġonline Ġadvertisements Ġand Ġgetting Ġmore Ġuser Ġclicks Ġmeans Ġbeing Ġmore Ġcompetitive Ġin Ġthe Ġmarket . ĠHowever , Ġmost Ġexisting Ġwebsites Ġna ively Ġgenerate Ġsensational Ġheadlines Ġusing Ġonly Ġkeywords Ġor Ġtemplates . ĠInstead , Ġthis Ġpaper Ġaims Ġto Ġlearn Ġa Ġmodel Ġthat Ġgenerates Ġsensational Ġheadlines Ġbased Ġon Ġan Ġinput Ġarticle Ġwithout Ġlabeled Ġdata . </s> To Ġgenerate Ġsensational Ġheadlines , Ġthere Ġare Ġtwo Ġmain Ġchallenges . ĠFirstly , Ġthere Ġis Ġa Ġlack Ġof Ġsensational ism Ġscorer Ġto Ġmeasure Ġhow Ġsensational Ġa Ġheadline Ġis . ĠSome Ġresearchers Ġhave Ġtried Ġto Ġmanually Ġlabel Ġheadlines Ġas Ġclick b ait Ġor Ġnon - click b ait ĠB IB REF 2 , ĠB IB REF 3 . ĠHowever , Ġthese Ġhuman - annot ated Ġdatasets Ġare Ġusually Ġsmall Ġand Ġexpensive Ġto Ġcollect . ĠTo Ġcapture Ġa Ġlarge Ġvariety Ġof Ġsensational ization Ġpatterns , Ġwe Ġneed Ġa Ġcheap Ġand Ġeasy Ġway Ġto Ġcollect Ġa Ġlarge Ġnumber Ġof Ġsensational Ġheadlines . ĠThus , Ġwe Ġpropose Ġa Ġdistant Ġsupervision Ġstrategy Ġto Ġcollect Ġa Ġsensational ism Ġdataset . ĠWe Ġregard Ġheadlines Ġreceiving Ġlots Ġof Ġcomments Ġas Ġsensational Ġsamples Ġand Ġthe Ġheadlines Ġgenerated Ġby Ġa Ġsummar ization Ġmodel Ġas Ġnon - s ens ational Ġsamples . ĠExperimental Ġresults Ġshow Ġthat Ġby Ġdistinguishing Ġthese Ġtwo Ġtypes Ġof Ġheadlines , Ġwe Ġcan Ġpartially Ġteach Ġthe Ġmodel Ġa Ġsense Ġof Ġbeing Ġsensational . </s> Secondly , Ġafter Ġtraining Ġa Ġsensational ism Ġscorer Ġon Ġour Ġsensational ism Ġdataset , Ġa Ġnatural Ġway Ġto Ġgenerate Ġsensational Ġheadlines Ġis Ġto Ġmaximize Ġthe Ġsensational ism Ġscore Ġusing Ġreinforcement Ġlearning Ġ( RL ). ĠHowever , Ġthe Ġfollowing Ġshows Ġan Ġexample Ġof Ġa ĠRL Ġmodel Ġmaximizing Ġthe Ġsensational ism Ġscore Ġby Ġgenerating Ġa Ġvery Ġunnatural Ġsentence , Ġwhile Ġits Ġsensational ism Ġscorer Ġgave Ġa Ġvery Ġhigh Ġscore Ġof Ġ0 . 9999 6 : ĠUTF 8 g bs n Ġå į ģ ä¸ ª åı ¯ ç © ¿ æĪ ´ äº § å ĵ ģ çļĦ è ® ¾ è ® ¡ åİ Ł åĪ Ļ è¿ Ļ æ ¶ Ī æ ģ ¯ æ ¶ Ī æ ģ ¯ åı ¯ æĥ ľ è¯ ´ æĺ İ ĠTen Ġdesign Ġprinciples Ġfor Ġwearable Ġdevices , Ġthis Ġmessage Ġmessage Ġpity Ġintroduction . ĠThis Ġhappens Ġbecause Ġthe Ġsensational ism Ġscorer Ġcan Ġmake Ġmistakes Ġand ĠRL Ġcan Ġgenerate Ġunnatural Ġphrases Ġwhich Ġfools Ġour Ġsensational ism Ġscorer . ĠThus , Ġhow Ġto Ġeffectively Ġleverage ĠRL Ġwith Ġnoisy Ġrewards Ġremains Ġan Ġopen Ġproblem . ĠTo Ġdeal Ġwith Ġthe Ġnoisy Ġreward , Ġwe Ġintroduce ĠAuto - tun ed ĠRein forcement ĠLearning Ġ( AR L ). ĠOur Ġmodel Ġautomatically Ġtunes Ġthe Ġratio Ġbetween ĠM LE Ġand ĠRL Ġbased Ġon Ġhow Ġsensational Ġthe Ġtraining Ġheadline Ġis . ĠIn Ġthis Ġway , Ġwe Ġeffectively Ġtake Ġadvantage Ġof ĠRL Ġwith Ġa Ġnoisy Ġreward Ġto Ġgenerate Ġheadlines Ġthat Ġare Ġboth Ġsensational Ġand Ġfluent . </s> The Ġmajor Ġcontributions Ġof Ġthis Ġpaper Ġare Ġas Ġfollows : Ġ1 ) ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge , Ġwe Ġpropose Ġthe Ġfirst - ever Ġmodel Ġthat Ġtackles Ġthe Ġsensational Ġheadline Ġgeneration Ġtask Ġwith Ġreinforcement Ġlearning Ġtechniques . Ġ2 ) ĠWithout Ġhuman - annot ated Ġdata , Ġwe Ġpropose Ġa Ġdistant Ġsupervision Ġstrategy Ġto Ġtrain Ġa Ġsensational ism Ġscorer Ġas Ġa Ġreward Ġfunction . 3 ) ĠWe Ġpropose Ġa Ġnovel Ġloss Ġfunction , ĠAuto - tun ed ĠRein forcement ĠLearning , Ġto Ġgive Ġdynamic Ġweights Ġto Ġbalance Ġbetween ĠM LE Ġand ĠRL . ĠOur Ġcode Ġwill Ġbe Ġreleased Ġ. </s> S ens ational ism ĠSc orer </s> To Ġevaluate Ġthe Ġsensational ism Ġintensity Ġscore Ġ$\ alpha Ġ_ {\ text { sen }} $ Ġof Ġa Ġheadline , Ġwe Ġcollect Ġa Ġsensational ism Ġdataset Ġand Ġthen Ġtrain Ġa Ġsensational ism Ġscorer . ĠFor Ġthe Ġsensational ism Ġdataset Ġcollection , Ġwe Ġchoose Ġheadlines Ġwith Ġmany Ġcomments Ġfrom Ġpopular Ġonline Ġwebsites Ġas Ġpositive Ġsamples . ĠFor Ġthe Ġnegative Ġsamples , Ġwe Ġpropose Ġto Ġuse Ġthe Ġgenerated Ġheadlines Ġfrom Ġa Ġsentence Ġsummar ization Ġmodel . ĠInt uitive ly , Ġthe Ġsummar ization Ġmodel , Ġwhich Ġis Ġtrained Ġto Ġpreserve Ġthe Ġsemantic Ġmeaning , Ġwill Ġlose Ġthe Ġsensational ization Ġability Ġand Ġthus Ġthe Ġgenerated Ġnegative Ġsamples Ġwill Ġbe Ġless Ġsensational Ġthan Ġthe Ġoriginal Ġone , Ġsimilar Ġto Ġthe Ġobfusc ation Ġof Ġstyle Ġafter Ġback - translation ĠB IB REF 4 . ĠFor Ġexample , Ġan Ġoriginal Ġheadline Ġlike ĠUTF 8 g bs n âĢ ľ ä¸Ģ è ¶ Ł æ Į £ 10 ä¸ ĩ ï ¼ Ł é ĵ ģ æĢ » å ¢ ŀ å¼ Ģ çĶ ³ éĢ ļ ãĢģ é ¡ º ä¸ ° ä¸ ĵ åĪ Ĺ " Ġ( One Ġtrip Ġto Ġearn Ġ100 Ġthousand ? ĠChina ĠRailway Ġopens Ġnew ĠShe nt ong Ġand ĠSh un f eng Ġspecial Ġlines ) Ġwill Ġbecome ĠUTF 8 g bs n âĢ ľ ä¸Ń é ĵ ģ æĢ » å°Ĩ å ¢ ŀ å¼ Ģ äº ¬ å¹ ¿ ä¸ ¤ åĪ Ĺ å¿ « éĢ Ĵ ä¸ ĵ åĪ Ĺ " Ġ( China ĠRailway Ġopens Ġtwo Ġspecial Ġlines Ġfor Ġexpress ) Ġfrom Ġthe Ġbaseline Ġmodel , Ġwhich Ġloses Ġthe Ġsensational Ġphrases Ġof ĠUTF 8 g bs n âĢ ľ ä¸Ģ è ¶ Ł æ Į £ 10 ä¸ ĩ ï ¼ Ł " Ġ( One Ġtrip Ġto Ġearn Ġ100 Ġthousand ?) Ġ. ĠWe Ġthen Ġtrain Ġthe Ġsensational ism Ġscorer Ġby Ġclass ifying Ġsensational Ġand Ġnon - s ens ational Ġheadlines Ġusing Ġa Ġone - layer ĠCNN Ġwith Ġa Ġbinary Ġcross Ġentropy Ġloss Ġ$ L _{ \ text { sen }} $. ĠFirstly , Ġ1 - D Ġconv olution Ġis Ġused Ġto Ġextract Ġword Ġfeatures Ġfrom Ġthe Ġinput Ġembed d ings Ġof Ġa Ġheadline . ĠThis Ġis Ġfollowed Ġby Ġa ĠRe LU Ġactivation Ġlayer Ġand Ġa Ġmax - pool ing Ġlayer Ġalong Ġthe Ġtime Ġdimension . ĠAll Ġfeatures Ġfrom Ġdifferent Ġchannels Ġare Ġconc aten ated Ġtogether Ġand Ġprojected Ġto Ġthe Ġsensational ism Ġscore Ġby Ġadding Ġanother Ġfully Ġconnected Ġlayer Ġwith Ġs igm oid Ġactivation . ĠBinary Ġcross Ġentropy Ġis Ġused Ġto Ġcompute Ġthe Ġloss Ġ$ L _{ \ text { sen }} $. </s> S ens ational ism ĠSc orer Ġ: :: ĠTraining ĠDetails Ġand ĠDat as et </s> For Ġthe ĠCNN Ġmodel , Ġwe Ġchoose Ġfilter Ġsizes Ġof Ġ1 , Ġ3 , Ġand Ġ5 Ġrespectively . ĠAdam Ġis Ġused Ġto Ġoptimize Ġ$ L _{ sen } $ Ġwith Ġa Ġlearning Ġrate Ġof Ġ0 . 0001 . ĠWe Ġset Ġthe Ġembed ding Ġsize Ġas Ġ300 Ġand Ġinitialize Ġit Ġfrom Ġq iu 2018 re vis iting Ġtrained Ġon Ġthe ĠWe ibo Ġcorpus Ġwith Ġword Ġand Ġcharacter Ġfeatures . ĠWe Ġfix Ġthe Ġembed d ings Ġduring Ġtraining . ĠFor Ġdataset Ġcollection , Ġwe Ġutilize Ġthe Ġheadlines Ġcollected Ġin Ġq in 2018 automatic , Ġlin 2019 learning Ġfrom ĠTen cent ĠNews , Ġone Ġof Ġthe Ġmost Ġpopular ĠChinese Ġnews Ġwebsites , Ġas Ġthe Ġpositive Ġsamples . ĠWe Ġfollow Ġthe Ġsame Ġdata Ġsplit Ġas Ġthe Ġoriginal Ġpaper . ĠAs Ġsome Ġof Ġthe Ġlinks Ġare Ġnot Ġavailable Ġany Ġmore , Ġwe Ġget Ġ170 , 754 Ġtraining Ġsamples Ġand Ġ4 , 511 Ġvalidation Ġsamples . ĠFor Ġthe Ġnegative Ġtraining Ġsamples Ġcollection , Ġwe Ġrandomly Ġselect Ġgenerated Ġheadlines Ġfrom Ġa Ġpointer Ġgenerator ĠB IB REF 0 Ġmodel Ġtrained Ġon ĠLC ST S Ġdataset ĠB IB REF 5 Ġand Ġcreate Ġa Ġbalanced Ġtraining Ġcorpus Ġwhich Ġincludes Ġ351 , 508 Ġtraining Ġsamples Ġand Ġ9 , 022 Ġvalidation Ġsamples . ĠTo Ġevaluate Ġour Ġtrained Ġclass ifier , Ġwe Ġconstruct Ġa Ġtest Ġset Ġby Ġrandomly Ġsampling Ġ100 Ġheadlines Ġfrom Ġthe Ġtest Ġsplit Ġof ĠLC ST S Ġdataset Ġand Ġthe Ġlabels Ġare Ġobtained Ġby Ġ11 Ġhuman Ġannot ators . ĠAnnotations Ġshow Ġthat Ġ52 % Ġheadlines Ġare Ġlabeled Ġas Ġpositive Ġand Ġ48 % Ġheadlines Ġas Ġnegative Ġby Ġmajority Ġvoting Ġ( The Ġdetail Ġon Ġthe Ġannotation Ġcan Ġbe Ġfound Ġin ĠSection ĠSEC REF 26 ). </s> S ens ational ism ĠSc orer Ġ: :: ĠResults Ġand ĠDiscussion </s> Our Ġclass ifier Ġachieves Ġ0 . 65 Ġaccuracy Ġand Ġ0 . 65 Ġaveraged ĠF 1 Ġscore Ġon Ġthe Ġtest Ġset Ġwhile Ġa Ġrandom Ġclass ifier Ġwould Ġonly Ġachieve Ġ0 . 50 Ġaccuracy Ġand Ġ0 . 50 Ġaveraged ĠF 1 Ġscore . ĠThis Ġconfirms Ġthat Ġthe Ġpredicted Ġsensational ism Ġscore Ġcan Ġpartially Ġcapture Ġthe Ġsensational ism Ġof Ġheadlines . ĠOn Ġthe Ġother Ġhand , Ġa Ġmore Ġnatural Ġchoice Ġis Ġto Ġtake Ġheadlines Ġwith Ġfew Ġcomments Ġas Ġnegative Ġexamples . ĠThus , Ġwe Ġtrain Ġanother Ġbaseline Ġclass ifier Ġon Ġa Ġcrawled Ġbalanced Ġsensational ism Ġcorpus Ġof Ġ84 k Ġheadlines Ġwhere Ġthe Ġpositive Ġheadlines Ġhave Ġat Ġleast Ġ28 Ġcomments Ġand Ġthe Ġnegative Ġheadlines Ġhave Ġless Ġthan Ġ5 Ġcomments . ĠHowever , Ġthe Ġresults Ġon Ġthe Ġtest Ġset Ġshow Ġthat Ġthe Ġbaseline Ġclass ifier Ġgets Ġ60 % Ġaccuracy , Ġwhich Ġis Ġworse Ġthan Ġthe Ġproposed Ġclass ifier Ġ( which Ġachieves Ġ65 %). ĠThe Ġreason Ġcould Ġbe Ġthat Ġthe Ġbalanced Ġsensational ism Ġcorpus Ġare Ġsampled Ġfrom Ġdifferent Ġdistributions Ġfrom Ġthe Ġtest Ġset Ġand Ġit Ġis Ġhard Ġfor Ġthe Ġtrained Ġmodel Ġto Ġgeneral ize . ĠTherefore , Ġwe Ġchoose Ġthe Ġproposed Ġone Ġas Ġour Ġsensational ism Ġscorer . ĠTherefore , Ġour Ġnext Ġchallenge Ġis Ġto Ġshow Ġthat Ġhow Ġto Ġleverage Ġthis Ġnoisy Ġsensational ism Ġreward Ġto Ġgenerate Ġsensational Ġheadlines . </s> S ens ational ĠHead line ĠGeneration </s> Our Ġsensational Ġheadline Ġgeneration Ġmodel Ġtakes Ġan Ġarticle Ġas Ġinput Ġand Ġoutput Ġa Ġsensational Ġheadline . ĠThe Ġmodel Ġconsists Ġof Ġa ĠPo inter - Gen Ġheadline Ġgenerator Ġand Ġis Ġtrained Ġby ĠAR L . ĠThe Ġdiagram Ġof ĠAR L Ġcan Ġbe Ġfound Ġin ĠFigure ĠFIG REF 6 . </s> We Ġdenote Ġthe Ġinput Ġarticle Ġas Ġ$ x = \ l brace Ġx _ 1 , x _ 2 , x _ 3 , \ cd ots Ġ, x _ M \ r brace Ġ$ , Ġand Ġthe Ġcorresponding Ġheadline Ġas Ġ$ y ^ * = \ l brace Ġy _ 1 ^ *, y _ 2 ^ *, y _ 3 ^ *, \ cd ots Ġ, y _ T ^ * \ r brace Ġ$ , Ġwhere Ġ$ M $ Ġis Ġthe Ġnumber Ġof Ġtokens Ġin Ġan Ġarticle Ġand Ġ$ T $ Ġis Ġthe Ġnumber Ġof Ġtokens Ġin Ġa Ġheadline . </s> S ens ational ĠHead line ĠGeneration Ġ: :: ĠPo inter - Gen ĠHead line ĠGenerator </s> We Ġchoose ĠPo inter ĠGenerator Ġ( Po inter - Gen ) ĠB IB REF 0 , Ġa Ġwidely Ġused Ġsummar ization Ġmodel , Ġas Ġour Ġheadline Ġgenerator Ġfor Ġits Ġability Ġto Ġcopy Ġwords Ġfrom Ġthe Ġinput Ġarticle . ĠIt Ġtakes Ġa Ġnews Ġarticle Ġas Ġinput Ġand Ġgenerates Ġa Ġheadline . ĠFirstly , Ġthe Ġtokens Ġof Ġeach Ġarticle , Ġ$\ l brace Ġx _ 1 , x _ 2 , x _ 3 , \ cd ots Ġ, x _ M \ r brace Ġ$ , Ġare Ġfed Ġinto Ġthe Ġenc oder Ġone - by - one Ġand Ġthe Ġenc oder Ġgenerates Ġa Ġsequence Ġof Ġhidden Ġstates Ġ$ h _ i $. ĠFor Ġeach Ġdecoding Ġstep Ġ$ t $, Ġthe Ġdec oder Ġreceives Ġthe Ġembed ding Ġfor Ġeach Ġtoken Ġof Ġa Ġheadline Ġ$ y _ t $ Ġas Ġinput Ġand Ġupdates Ġits Ġhidden Ġstates Ġ$ s _ t $. ĠAn Ġattention Ġmechanism Ġfollowing Ġl u ong 2015 effective Ġis Ġused : </s> where Ġ$ v $, Ġ$ W _ h $, Ġ$ W _ s $, Ġand Ġ$ b _{ att n } $ Ġare Ġthe Ġtrain able Ġparameters Ġand Ġ$ h _ t ^ * $ Ġis Ġthe Ġcontext Ġvector . Ġ$ s _ t $ Ġand Ġ$ h _ t ^ * $ Ġare Ġthen Ġcombined Ġto Ġgive Ġa Ġprobability Ġdistribution Ġover Ġthe Ġvocabulary Ġthrough Ġtwo Ġlinear Ġlayers : </s> where Ġ$ V $, Ġ$ b $, Ġ$ V ^{ ^ {\ prime Ġ}} $, Ġand Ġ$ b ^{ ^ {\ prime Ġ}} $ Ġare Ġtrain able Ġparameters . ĠWe Ġuse Ġa Ġpointer Ġgenerator Ġnetwork Ġto Ġenable Ġour Ġmodel Ġto Ġcopy Ġrare / unknown Ġwords Ġfrom Ġthe Ġinput Ġarticle , Ġgiving Ġthe Ġfollowing Ġfinal Ġword Ġprobability : </s> where Ġ$ x ^ t $ Ġis Ġthe Ġembed ding Ġof Ġthe Ġinput Ġword Ġof Ġthe Ġdec oder , Ġ$ w _{ h ^ * } ^ T $, Ġ$ w _ s ^ T $, Ġ$ w _ x ^ T $, Ġand Ġ$ b _{ ptr } $ Ġare Ġtrain able Ġparameters , Ġand Ġ$\ s igma Ġ$ Ġis Ġthe Ġs igm oid Ġfunction . </s> S ens ational ĠHead line ĠGeneration Ġ: :: ĠTraining ĠMethods </s> We Ġfirst Ġbriefly Ġintroduce ĠM LE Ġand ĠRL Ġobjective Ġfunctions , Ġand Ġa Ġnaive Ġway Ġto Ġmix Ġthese Ġtwo Ġby Ġa Ġhyper - param eter Ġ$\ lambda Ġ$ . ĠThen Ġwe Ġpoint Ġout Ġthe Ġchallenge Ġof Ġtraining Ġwith Ġnoisy Ġreward , Ġand Ġpropose ĠAR L Ġto Ġaddress Ġthis Ġissue . </s> S ens ational ĠHead line ĠGeneration Ġ: :: ĠTraining ĠMethods Ġ: :: ĠM LE Ġand ĠRL </s> A Ġheadline Ġgeneration Ġmodel Ġcan Ġbe Ġtrained Ġwith ĠM LE , ĠRL Ġor Ġa Ġcombination Ġof ĠM LE Ġand ĠRL . ĠM LE Ġtraining Ġis Ġto Ġminimize Ġthe Ġnegative Ġlog Ġlikelihood Ġof Ġthe Ġtraining Ġheadlines . ĠWe Ġfeed Ġ$ y ^ * $ Ġinto Ġthe Ġdec oder Ġword Ġby Ġword Ġand Ġmaximize Ġthe Ġlikelihood Ġof Ġ$ y ^ * $. ĠThe Ġloss Ġfunction Ġfor ĠM LE Ġbecomes </s> For ĠRL Ġtraining , Ġwe Ġchoose Ġthe ĠRE IN FOR CE Ġalgorithm ĠB IB REF 6 . ĠIn Ġthe Ġtraining Ġphase , Ġafter Ġencoding Ġan Ġarticle , Ġa Ġheadline Ġ$ y ^ s Ġ= Ġ\ l brace Ġy _ 1 ^ s , Ġy _ 2 ^ s , Ġy _ 3 ^ s , Ġ\ cd ots Ġ, Ġy _ T ^ s \ r brace Ġ$ Ġis Ġobtained Ġby Ġsampling Ġfrom Ġ$ P ( w ) $ Ġfrom Ġour Ġgenerator , Ġand Ġthen Ġa Ġreward Ġof Ġsensational ism Ġor ĠR OU GE ( RG ) Ġis Ġcalculated . </s> We Ġuse Ġthe Ġbaseline Ġreward Ġ$\ hat { R _ t } $ Ġto Ġreduce Ġthe Ġvariance Ġof Ġthe Ġreward , Ġsimilar Ġto Ġran z ato 2015 sequence . ĠTo Ġelaborate , Ġa Ġlinear Ġmodel Ġis Ġdeployed Ġto Ġestimate Ġthe Ġbaseline Ġreward Ġ$\ hat { R _ t } $ Ġbased Ġon Ġ$ t $ - th Ġstate Ġ$ o _ t $ Ġfor Ġeach Ġtim est ep Ġ$ t $. ĠThe Ġparameters Ġof Ġthe Ġlinear Ġmodel Ġare Ġtrained Ġby Ġminimizing Ġthe Ġmean Ġsquare Ġloss Ġbetween Ġ$ R $ Ġand Ġ$\ hat { R _ t } $ : </s> where Ġ$ W _ r $ Ġand Ġ$ b _ r $ Ġare Ġtrain able Ġparameters . ĠTo Ġmaximize Ġthe Ġexpected Ġreward , Ġour Ġloss Ġfunction Ġfor ĠRL Ġbecomes </s> A Ġnaive Ġway Ġto Ġmix Ġthese Ġtwo Ġobjective Ġfunctions Ġusing Ġa Ġhyper - param eter Ġ$\ lambda Ġ$ Ġhas Ġbeen Ġsuccessfully Ġincorporated Ġin Ġthe Ġsummar ization Ġtask ĠB IB REF 7 . ĠIt Ġincludes Ġthe ĠM LE Ġtraining Ġas Ġa Ġlanguage Ġmodel Ġto Ġmitigate Ġthe Ġread ability Ġand Ġquality Ġissues Ġin ĠRL . ĠThe Ġmixed Ġloss Ġfunction Ġis Ġshown Ġas Ġfollows : </s> where Ġ$ * $ Ġis Ġthe Ġreward Ġtype . ĠUsually Ġ$\ lambda Ġ$ Ġis Ġlarge , Ġand Ġp aul us 2017 deep Ġused Ġ0 . 99 84 . </s> S ens ational ĠHead line ĠGeneration Ġ: :: ĠTraining ĠMethods Ġ: :: ĠAuto - tun ed ĠRein forcement ĠLearning </s> App lying Ġthe Ġnaive Ġmixed Ġtraining Ġmethod Ġusing Ġsensational ism Ġscore Ġas Ġthe Ġreward Ġis Ġnot Ġobvious / t riv ial Ġin Ġour Ġtask . ĠThe Ġmain Ġreason Ġis Ġthat Ġour Ġsensational ism Ġreward Ġis Ġnotably Ġmore Ġnoisy Ġand Ġmore Ġfragile Ġthan Ġthe ĠR OU GE - L Ġreward Ġor Ġabstract ive Ġreward Ġused Ġin Ġthe Ġsummar ization Ġtask ĠB IB REF 7 , ĠB IB REF 8 . ĠA Ġhigher ĠR OU GE - L ĠF 1 Ġreward Ġin Ġsummar ization Ġindicates Ġhigher Ġoverlapping Ġratio Ġbetween Ġgeneration Ġand Ġtrue Ġsummary Ġstatistically , Ġbut Ġour Ġsensational ism Ġreward Ġis Ġa Ġlearned Ġscore Ġwhich Ġis Ġfragile Ġto Ġbe Ġfooled Ġwith Ġunnatural Ġsamples . </s> To Ġeffectively Ġtrain Ġthe Ġmodel Ġwith ĠRL Ġunder Ġnoisy Ġsensational ism Ġreward , Ġour Ġidea Ġis Ġto Ġbalance ĠRL Ġwith ĠM LE . ĠHowever , Ġwe Ġargue Ġthat Ġthe Ġweighted Ġratio Ġbetween ĠM LE Ġand ĠRL Ġshould Ġbe Ġsample - dependent , Ġinstead Ġof Ġbeing Ġfixed Ġfor Ġall Ġtraining Ġsamples Ġas Ġin Ġp aul us 2017 deep , Ġk ry sc inski 2018 impro ving . ĠThe Ġreason Ġis Ġthat , ĠRL Ġand ĠM LE Ġhave Ġinconsistent Ġoptimization Ġobjectives . ĠWhen Ġthe Ġtraining Ġheadline Ġis Ġnon - s ens ational , ĠM LE Ġtraining Ġwill Ġencourage Ġour Ġmodel Ġto Ġimitate Ġthe Ġtraining Ġheadline Ġ( thus Ġgenerating Ġnon - s ens ational Ġheadlines ), Ġwhich Ġcounter acts Ġthe Ġeffects Ġof ĠRL Ġtraining Ġto Ġgenerate Ġsensational Ġheadlines . </s> The Ġsensational ism Ġscore Ġis , Ġtherefore , Ġused Ġto Ġgive Ġdynamic Ġweight Ġto ĠM LE Ġand ĠRL . ĠOur ĠAR L Ġloss Ġfunction Ġbecomes : </s> If Ġ$\ alpha Ġ_ {\ text { sen }} ( y ^ *) $ Ġis Ġhigh , Ġmeaning Ġthe Ġtraining Ġheadline Ġis Ġsensational , Ġour Ġloss Ġfunction Ġencourages Ġour Ġmodel Ġto Ġimitate Ġthe Ġsample Ġmore Ġusing Ġthe ĠM LE Ġtraining . ĠIf Ġ$\ alpha Ġ_ {\ text { sen }} ( y ^ *) $ Ġis Ġlow , Ġour Ġloss Ġfunction Ġreplies Ġon ĠRL Ġtraining Ġto Ġimprove Ġthe Ġsensational ism . ĠNote Ġthat Ġthe Ġweight Ġ$\ alpha Ġ_ {\ text { sen }} ( y ^ *) $ Ġis Ġdifferent Ġfrom Ġour Ġsensational ism Ġreward Ġ$\ alpha Ġ_ {\ text { sen }} ( y ^ s ) $ Ġand Ġwe Ġcall Ġthe Ġloss Ġfunction ĠAuto - tun ed ĠRein forcement ĠLearning , Ġbecause Ġthe Ġratio Ġbetween ĠM LE Ġand ĠRL Ġare Ġwell ĠâĢ ľ tun ed âĢ Ŀ Ġtowards Ġdifferent Ġsamples . </s> S ens ational ĠHead line ĠGeneration Ġ: :: ĠDat as et </s> We Ġuse ĠLC ST S ĠB IB REF 5 Ġas Ġour Ġdataset Ġto Ġtrain Ġthe Ġsummar ization Ġmodel . ĠThe Ġdataset Ġis Ġcollected Ġfrom Ġthe ĠChinese Ġmicro blog ging Ġwebsite ĠSina ĠWe ibo . ĠIt Ġcontains Ġover Ġ2 Ġmillion ĠChinese Ġshort Ġtexts Ġwith Ġcorresponding Ġheadlines Ġgiven Ġby Ġthe Ġauthor Ġof Ġeach Ġtext . ĠThe Ġdataset Ġis Ġsplit Ġinto Ġ2 , 400 , 591 Ġsamples Ġfor Ġtraining , Ġ10 , 666 Ġsamples Ġfor Ġvalidation Ġand Ġ7 25 Ġsamples Ġfor Ġtesting . ĠWe Ġtoken ize Ġeach Ġsentence Ġwith ĠJ ie ba Ġand Ġa Ġvocabulary Ġsize Ġof Ġ5 0000 Ġis Ġsaved . </s> S ens ational ĠHead line ĠGeneration Ġ: :: ĠBas elines Ġand ĠOur ĠModels </s> We Ġexperiment Ġand Ġcompare Ġwith Ġthe Ġfollowing Ġmodels . </s> Po inter - Gen Ġis Ġthe Ġbaseline Ġmodel Ġtrained Ġby Ġoptimizing Ġ$ L _ \ text { M LE } $ Ġin ĠEqu ation ĠDIS PLAY _ FORM 13 . </s> Po inter - Gen + Pos Ġis Ġthe Ġbaseline Ġmodel Ġby Ġtraining ĠPo inter - Gen Ġonly Ġon Ġpositive Ġexamples Ġwhose Ġsensational ism Ġscore Ġis Ġlarger Ġthan Ġ0 . 5 </s> Po inter - Gen + Same - FT Ġis Ġthe Ġmodel Ġwhich Ġfine - t unes ĠPo inter - Gen Ġon Ġthe Ġtraining Ġsamples Ġwhose Ġsensational ism Ġscore Ġis Ġlarger Ġthan Ġ0 . 1 </s> Po inter - Gen + Pos - FT Ġis Ġthe Ġmodel Ġwhich Ġfine - t unes ĠPo inter - Gen Ġon Ġthe Ġtraining Ġsamples Ġwhose Ġsensational ism Ġscore Ġis Ġlarger Ġthan Ġ0 . 5 </s> Po inter - Gen + RL - R OU GE Ġis Ġthe Ġbaseline Ġmodel Ġtrained Ġby Ġoptimizing Ġ$ L _ \ text { RL - R OU GE } $ Ġin ĠEqu ation ĠDIS PLAY _ FORM 17 , Ġwith ĠR OU GE - L ĠB IB REF 9 Ġas Ġthe Ġreward . </s> Po inter - Gen + RL - S EN Ġis Ġthe Ġbaseline Ġmodel Ġtrained Ġby Ġoptimizing Ġ$ L _ \ text { RL - S EN } $ Ġin ĠEqu ation ĠDIS PLAY _ FORM 17 , Ġwith Ġ$\ alpha Ġ_ \ text { sen } $ Ġas Ġthe Ġreward . </s> Po inter - Gen + AR L - S EN Ġis Ġour Ġmodel Ġtrained Ġby Ġoptimizing Ġ$ L _ \ text { AR L - S EN } $ Ġin ĠEqu ation ĠDIS PLAY _ FORM 19 , Ġwith Ġ$\ alpha Ġ_ \ text { sen } $ Ġas Ġthe Ġreward . </s> Test Ġset Ġis Ġthe Ġheadlines Ġfrom Ġthe Ġtest Ġset . </s> Note Ġthat Ġwe Ġdidn 't Ġcompare Ġto ĠPo inter - Gen + AR L - R OU GE Ġas Ġit Ġis Ġactually ĠPo inter - GEN . ĠRecall Ġthat Ġ$\ alpha Ġ_ {\ text { sen }} ( y ^ *) $ Ġin ĠEqu ation ĠDIS PLAY _ FORM 19 Ġmeasures Ġhow Ġgood Ġ( based Ġon Ġreward Ġfunction ) Ġis Ġ$ y ^ * $. ĠThen Ġthe Ġloss Ġfunction Ġfor ĠPo inter - Gen + AR L - R OU GE Ġwill Ġbe </s> We Ġalso Ġtried Ġtext Ġstyle Ġtransfer Ġbaseline ĠB IB REF 10 , Ġbut Ġthe Ġgenerated Ġheadlines Ġwere Ġvery Ġpoor Ġ( many Ġunknown Ġwords Ġand Ġirrelevant ). </s> S ens ational ĠHead line ĠGeneration Ġ: :: ĠTraining ĠDetails </s> M LE Ġtraining : ĠAn ĠAdam Ġoptim izer Ġis Ġused Ġwith Ġthe Ġlearning Ġrate Ġof Ġ0 . 0001 Ġto Ġoptimize Ġ$ L _{ \ text { M LE }} $. ĠThe Ġbatch Ġsize Ġis Ġset Ġas Ġ128 Ġand Ġa Ġone - layer , Ġbi - direction al ĠLong ĠShort - Term ĠMemory Ġ( bi - L ST M ) Ġmodel Ġwith Ġ512 Ġhidden Ġsizes Ġand Ġa Ġ350 Ġembed ding Ġsize Ġis Ġutilized . ĠGrad ients Ġwith Ġthe Ġl 2 Ġnorm Ġlarger Ġthan Ġ2 . 0 Ġare Ġclipped . ĠWe Ġstop Ġtraining Ġwhen Ġthe ĠR OU GE - L Ġf - score Ġstops Ġincreasing . </s> Hy brid Ġtraining : ĠAn ĠAdam Ġoptim izer Ġwith Ġa Ġlearning Ġrate Ġof Ġ0 . 0001 Ġis Ġused Ġto Ġoptimize Ġ$ L _{ \ text { RL - * }} $ Ġ( Equ ation ĠDIS PLAY _ FORM 17 ) Ġand Ġ$ L _ \ text {{ AR L - S EN }} $ Ġ( Equ ation ĠDIS PLAY _ FORM 19 ). ĠWhen Ġtraining ĠPo inter - Gen + RL - R OU GE , Ġthe Ġbest Ġ$\ lambda Ġ$ Ġis Ġchosen Ġbased Ġon Ġthe ĠR OU GE - L Ġscore Ġon Ġthe Ġvalidation Ġset . ĠIn Ġour Ġexperiment , Ġ$\ lambda Ġ$ Ġis Ġset Ġas Ġ0 . 95 . ĠAn ĠAdam Ġoptim izer Ġwith Ġa Ġlearning Ġrate Ġof Ġ0 . 001 Ġis Ġused Ġto Ġoptimize Ġ$ L _ b $. ĠWhen Ġtraining ĠPo inter - Gen + AR L - S EN , Ġwe Ġdon 't Ġuse Ġthe Ġfull ĠLC ST S Ġdataset , Ġbut Ġonly Ġheadlines Ġwith Ġa Ġsensational ism Ġscore Ġlarger Ġthan Ġ0 . 1 Ġas Ġwe Ġobserve Ġthat ĠPo inter - Gen + AR L - S EN Ġwill Ġgenerate Ġa Ġfew Ġunnatural Ġphrases Ġwhen Ġusing Ġfull Ġdataset . ĠWe Ġbelieve Ġthe Ġreason Ġis Ġthe Ġhigh Ġratio Ġof ĠRL Ġduring Ġtraining . ĠFigure ĠFIG REF 23 Ġshows Ġthat Ġthe Ġprobability Ġdensity Ġnear Ġ0 Ġis Ġvery Ġhigh , Ġmeaning Ġthat Ġin Ġeach Ġbatch , Ġmany Ġof Ġthe Ġsamples Ġwill Ġhave Ġa Ġvery Ġlow Ġsensational ism Ġscore . ĠOn Ġexpectation , Ġeach Ġsample Ġwill Ġreceive Ġ0 . 239 ĠM LE Ġtraining Ġand Ġ0 . 76 1 ĠRL Ġtraining . ĠThis Ġleads Ġto ĠRL Ġdominant ing Ġthe Ġloss . ĠThus , Ġwe Ġpropose Ġto Ġfilter Ġsamples Ġwith Ġa Ġminimum Ġsensational ism Ġscore Ġwith Ġ0 . 1 Ġand Ġit Ġworks Ġvery Ġwell . ĠFor ĠPo inter - Gen + RL - S EN , Ġwe Ġalso Ġset Ġthe Ġminimum Ġsensational ism Ġscore Ġas Ġ0 . 1 , Ġand Ġ$\ lambda Ġ$ Ġis Ġset Ġas Ġ0 . 5 Ġto Ġremove Ġunnatural Ġphrases , Ġmaking Ġa Ġfair Ġcomparison Ġto ĠPo inter - Gen + AR L - S EN . </s> We Ġstop Ġtraining ĠPo inter - Gen + Same - FT , ĠPo inter - Gen + Pos - FT , ĠPo inter - Gen + RL - S EN Ġand ĠPo inter - Gen + AR L - S EN , Ġwhen Ġ$\ alpha Ġ_ \ text { sen } $ Ġstops Ġincreasing Ġon Ġthe Ġvalidation Ġset . ĠBeam - search Ġwith Ġa Ġbeam Ġsize Ġof Ġ5 Ġis Ġadopted Ġfor Ġdecoding Ġin Ġall Ġmodels . </s> S ens ational ĠHead line ĠGeneration Ġ: :: ĠEvaluation ĠMet rics </s> We Ġbriefly Ġdescribe Ġthe Ġevaluation Ġmetrics Ġbelow . </s> R OU GE : ĠR OU GE Ġis Ġa Ġcommonly Ġused Ġevaluation Ġmetric Ġfor Ġsummar ization . ĠIt Ġmeasures Ġthe ĠN - gram Ġoverlap Ġbetween Ġgenerated Ġand Ġtraining Ġheadlines . ĠWe Ġuse Ġit Ġto Ġevaluate Ġthe Ġrelevance Ġof Ġgenerated Ġheadlines . ĠThe Ġwidely Ġused Ġpy rou ge Ġtool kit Ġis Ġused Ġto Ġcalculate ĠR OU GE - 1 Ġ( RG - 1 ), ĠR OU GE - 2 Ġ( RG - 2 ), Ġand ĠR OU GE - L Ġ( RG - L ). </s> Human Ġevaluation : ĠWe Ġrandomly Ġsample Ġ50 Ġarticles Ġfrom Ġthe Ġtest Ġset Ġand Ġsend Ġthe Ġgenerated Ġheadlines Ġfrom Ġall Ġmodels Ġand Ġcorresponding Ġheadlines Ġin Ġthe Ġtest Ġset Ġto Ġhuman Ġannot ators . ĠWe Ġevaluate Ġthe Ġsensational ism Ġand Ġflu ency Ġof Ġthe Ġheadlines Ġby Ġsetting Ġup Ġtwo Ġindependent Ġhuman Ġannotation Ġtasks . ĠWe Ġask Ġ10 Ġannot ators Ġto Ġlabel Ġeach Ġheadline Ġfor Ġeach Ġtask . ĠFor Ġthe Ġsensational ism Ġannotation , Ġeach Ġannot ator Ġis Ġasked Ġone Ġquestion , ĠâĢ ľ Is Ġthe Ġheadline Ġsensational ? âĢ Ŀ , Ġand Ġhe / she Ġhas Ġto Ġchoose Ġeither Ġ` yes ' Ġor Ġ` no '. ĠThe Ġannot ators Ġwere Ġnot Ġtold Ġwhich Ġsystem Ġthe Ġheadline Ġis Ġfrom . ĠThe Ġprocess Ġof Ġdistributing Ġsamples Ġand Ġrecruiting Ġannot ators Ġis Ġmanaged Ġby ĠCrowd flower . ĠAfter Ġannotation , Ġwe Ġdefine Ġthe Ġsensational ism Ġscore Ġas Ġthe Ġproportion Ġof Ġannotations Ġon Ġall Ġgenerated Ġheadlines Ġfrom Ġone Ġmodel Ġlabeled Ġas Ġ` yes '. ĠFor Ġthe Ġflu ency Ġannotation , Ġwe Ġrepeat Ġthe Ġsame Ġprocedure Ġas Ġfor Ġthe Ġsensational ism Ġannotation , Ġexcept Ġthat Ġwe Ġask Ġeach Ġannot ator Ġthe Ġquestion ĠâĢ ľ Is Ġthe Ġheadline Ġfluent ? âĢ Ŀ ĠWe Ġdefine Ġthe Ġflu ency Ġscore Ġas Ġthe Ġproportion Ġof Ġannotations Ġon Ġall Ġheadlines Ġfrom Ġone Ġspecific Ġmodel Ġlabeled Ġas Ġ` yes '. ĠWe Ġput Ġhuman Ġannotation Ġinstructions Ġin Ġthe Ġsupplemental Ġmaterial . </s> UTF 8 g bs n </s> Results </s> We Ġfirst Ġcompare Ġall Ġfour Ġmodels , ĠPo inter - Gen , ĠPo inter - Gen - RL + R OU GE , ĠPo inter - Gen - RL - S EN , Ġand ĠPo inter - Gen - AR L - S EN , Ġto Ġexisting Ġmodels Ġwith ĠR OU GE Ġin ĠTable ĠT AB REF 25 Ġto Ġestablish Ġthat Ġour Ġmodel Ġproduces Ġrelevant Ġheadlines Ġand Ġwe Ġleave Ġthe Ġsensational ism Ġfor Ġhuman Ġevaluation . ĠNote Ġthat Ġwe Ġonly Ġcompare Ġour Ġmodels Ġto Ġcommonly Ġused Ġstrong Ġsummar ization Ġbas elines , Ġto Ġvalidate Ġthat Ġour Ġimplementation Ġachieves Ġcomparable Ġperformance Ġto Ġexisting Ġwork . ĠIn Ġour Ġimplementation , ĠPo inter - Gen Ġachieves Ġa Ġ34 . 51 ĠRG - 1 Ġscore , Ġ22 . 21 ĠRG - 2 Ġscore , Ġand Ġ31 . 68 ĠRG - L Ġscore , Ġwhich Ġis Ġsimilar Ġto Ġthe Ġresults Ġof Ġgu 2016 inc orpor ating . ĠPo inter - Gen + AR L - S EN , Ġalthough Ġoptimized Ġfor Ġthe Ġsensational ism Ġreward , Ġachieves Ġsimilar Ġperformance Ġto Ġour ĠPo inter - Gen Ġbaseline , Ġwhich Ġmeans Ġthat ĠPo inter - Gen + AR L - S EN Ġstill Ġkeeps Ġits Ġsummar ization Ġability . ĠAn Ġexample Ġof Ġheadlines Ġgenerated Ġfrom Ġdifferent Ġmodels Ġin ĠTable ĠT AB REF 29 Ġshows Ġthat ĠPo inter - Gen Ġand ĠPo inter - Gen + RL - R OU GE Ġlearns Ġto Ġsummarize Ġthe Ġmain Ġpoint Ġof Ġthe Ġarticle : ĠâĢ ľ The ĠNikon ĠD 600 Ġcamera Ġis Ġreported Ġto Ġhave Ġblack Ġspots Ġwhen Ġtaking Ġphotos âĢ Ŀ . ĠPo inter - Gen + RL - S EN Ġmakes Ġthe Ġheadline Ġmore Ġsensational Ġby Ġblaming ĠNikon Ġfor Ġattribut ing Ġthe Ġdamage Ġto Ġthe Ġsm og . ĠPo inter - Gen + AR L - S EN Ġgenerates Ġthe Ġmost Ġsensational Ġheadline Ġby Ġexagger ating Ġthe Ġresult ĠâĢ ľ Getting Ġa Ġserious Ġtrouble ! âĢ Ŀ Ġto Ġmaximize Ġuser 's Ġattention . </s> We Ġthen Ġcompare Ġdifferent Ġmodels Ġusing Ġthe Ġsensational ism Ġscore Ġin ĠTable ĠT AB REF 30 . ĠThe ĠPo inter - Gen Ġbaseline Ġmodel Ġachieves Ġa Ġ42 . 6 % Ġsensational ism Ġscore , Ġwhich Ġis Ġthe Ġminimum Ġthat Ġa Ġtypical Ġsummar ization Ġmodel Ġachieves . ĠBy Ġfiltering Ġout Ġlow - s ens ational Ġheadlines , ĠPo inter - Gen + Same - FT Ġand ĠPo inter - Gen + Pos - FT Ġachieves Ġhigher Ġsensational ism Ġscores , Ġwhich Ġimplies Ġthe Ġeffectiveness Ġof Ġour Ġsensational ism Ġscorer . ĠOur ĠPo inter - Gen + AR L - S EN Ġmodel Ġachieves Ġthe Ġbest Ġperformance Ġof Ġ60 . 8 %. ĠThis Ġis Ġan Ġabsolute Ġimprovement Ġof Ġ18 . 2 % Ġover Ġthe ĠPo inter - Gen Ġbaseline . ĠThe ĠChi - square Ġtest Ġon Ġthe Ġresults Ġconfirms Ġthat ĠPo inter - Gen + AR L - S EN Ġis Ġstatistically Ġsignificantly Ġmore Ġsensational Ġthan Ġall Ġthe Ġother Ġbaseline Ġmodels , Ġwith Ġthe Ġlargest Ġp - value Ġless Ġthan Ġ0 . 01 . ĠAlso , Ġwe Ġfind Ġthat Ġthe Ġtest Ġset Ġheadlines Ġachieves Ġ57 . 8 % Ġsensational ism Ġscore , Ġmuch Ġlarger Ġthan ĠPo inter - Gen Ġbaseline , Ġwhich Ġalso Ġsupports Ġour Ġintuition Ġthat Ġgenerated Ġheadlines Ġwill Ġbe Ġless Ġsensational Ġthan Ġthe Ġoriginal Ġone . ĠOn Ġthe Ġother Ġhand , Ġwe Ġfound Ġthat ĠPo inter - Gen + Pos Ġis Ġmuch Ġworse Ġthan Ġother Ġbas elines . ĠThe Ġreason Ġis Ġthat Ġtraining Ġon Ġsensational Ġsamples Ġalone Ġdisc ards Ġaround Ġ80 % Ġof Ġthe Ġwhole Ġtraining Ġset Ġthat Ġis Ġalso Ġhelpful Ġfor Ġmaintaining Ġrelevance Ġand Ġa Ġgood Ġlanguage Ġmodel . ĠIt Ġshows Ġthe Ġnecessity Ġof Ġusing ĠRL . </s> UTF 8 g bs n </s> In Ġaddition , Ġboth ĠPo inter - Gen + RL - S EN Ġand ĠPo inter - Gen + AR L - S EN , Ġwhich Ġuse Ġthe Ġsensational ism Ġscore Ġas Ġthe Ġreward , Ġobtain Ġstatistically Ġbetter Ġresults Ġthan ĠPo inter - Gen + RL - R OU GE Ġand ĠPo inter - Gen , Ġwith Ġa Ġp - value Ġless Ġthan Ġ0 . 05 Ġby Ġa ĠChi - square Ġtest . ĠThis Ġresult Ġshows Ġthe Ġeffectiveness Ġof ĠRL Ġto Ġgenerate Ġmore Ġsensational Ġheadlines . ĠThe Ġreason Ġis Ġthat Ġeven Ġthough Ġour Ġnoisy Ġclass ifier Ġcould Ġalso Ġlearn Ġto Ġclassify Ġdomains , Ġthe Ġgenerator Ġduring ĠRL Ġtraining Ġis Ġnot Ġallowed Ġto Ġincrease Ġthe Ġreward Ġby Ġshifting Ġdomains Ġbut Ġencouraged Ġto Ġgenerate Ġmore Ġsensational Ġheadlines , Ġdue Ġto Ġthe Ġconsistency Ġconstraint Ġon Ġthe Ġdomains Ġof Ġthe Ġheadline Ġand Ġthe Ġarticle . ĠFurthermore , ĠPo iner - Gen + AR L - S EN Ġgets Ġbetter Ġperformance Ġthan ĠPo inter - Gen + RL - S EN , Ġwhich Ġconfirms Ġthe Ġsuperiority Ġof Ġthe ĠAR L Ġloss Ġfunction . ĠWe Ġalso Ġvisualize Ġin ĠFigure ĠFIG REF 31 Ġa Ġcomparison Ġbetween ĠPo inter - Gen + AR L - S EN Ġand ĠPo inter - Gen + RL - S EN Ġaccording Ġto Ġhow Ġsensational Ġthe Ġtest Ġset Ġheadlines Ġare . ĠThe Ġblue Ġbars Ġdenote Ġthe Ġsmaller Ġscores Ġbetween Ġthe Ġtwo Ġmodels . ĠFor Ġexample , Ġif Ġthe Ġblue Ġbar Ġis Ġ0 . 6 , Ġit Ġmeans Ġthat Ġthe Ġworse Ġmodel Ġbetween ĠPo inter - Gen + RL - S EN Ġand ĠPo inter - Gen + AR L - S EN Ġachieves Ġ0 . 6 . ĠAnd Ġthe Ġcolor Ġof Ġorange / black Ġfurther Ġindicates Ġthe Ġbetter Ġmodel Ġand Ġits Ġscore . ĠWe Ġfind Ġthat ĠPo inter - Gen + AR L - S EN Ġoutper forms ĠPo inter - Gen + RL - S EN Ġfor Ġmost Ġcases . ĠThe Ġimprovement Ġis Ġhigher Ġwhen Ġthe Ġtest Ġset Ġheadlines Ġare Ġnot Ġsensational Ġ( the Ġsensational ism Ġscore Ġis Ġless Ġthan Ġ0 . 5 ), Ġwhich Ġmay Ġbe Ġattributed Ġto Ġthe Ġhigher Ġratio Ġof ĠRL Ġtraining Ġon Ġnon - s ens ational Ġheadlines . </s> Apart Ġfrom Ġthe Ġsensational ism Ġevaluation , Ġwe Ġmeasure Ġthe Ġflu ency Ġof Ġthe Ġheadlines Ġgenerated Ġfrom Ġdifferent Ġmodels . ĠFlu ency Ġscores Ġin ĠTable ĠT AB REF 30 Ġshow Ġthat ĠPo inter - Gen + RL - S EN Ġand ĠPo inter - Gen + AR L - S EN Ġachieve Ġcomparable Ġflu ency Ġperformance Ġto ĠPo inter - Gen Ġand ĠPo inter - Gen + RL - R OU GE . ĠTest Ġset Ġheadlines Ġachieve Ġthe Ġbest Ġperformance Ġamong Ġall Ġmodels , Ġbut Ġthe Ġdifference Ġis Ġnot Ġstatistically Ġsignificant . ĠAlso , Ġwe Ġobserve Ġthat Ġfine - tun ing Ġon Ġsensational Ġheadlines Ġwill Ġhurt Ġthe Ġperformance , Ġboth Ġin Ġsensational ism Ġand Ġflu ency . </s> After Ġmanually Ġchecking Ġthe Ġoutputs , Ġwe Ġobserve Ġthat Ġour Ġmodel Ġis Ġable Ġto Ġgenerate Ġsensational Ġheadlines Ġusing Ġdiverse Ġsensational ization Ġstrategies . ĠThese Ġstrategies Ġinclude , Ġbut Ġare Ġnot Ġlimited Ġto , Ġcreating Ġa Ġcuriosity Ġgap , Ġasking Ġquestions , Ġhighlighting Ġnumbers , Ġbeing Ġemotional Ġand Ġemphasizing Ġthe Ġuser . ĠExamples Ġcan Ġbe Ġfound Ġin ĠTable ĠT AB REF 32 . </s> Related ĠWork </s> Our Ġwork Ġis Ġrelated Ġto Ġsummar ization Ġtasks . ĠAn Ġenc oder - dec oder Ġmodel Ġwas Ġfirst Ġapplied Ġto Ġtwo Ġsentence - level Ġabstract ive Ġsummar ization Ġtasks Ġon Ġthe ĠD UC - 2004 Ġand ĠGig aw ord Ġdatasets ĠB IB REF 12 . ĠThis Ġmodel Ġwas Ġlater Ġextended Ġby Ġselective Ġencoding ĠB IB REF 13 , Ġa Ġcoarse Ġto Ġfine Ġapproach ĠB IB REF 14 , Ġminimum Ġrisk Ġtraining ĠB IB REF 1 , Ġand Ġtopic - aware Ġmodels ĠB IB REF 15 . ĠAs Ġlong Ġsumm aries Ġwere Ġrecognized Ġas Ġimportant , Ġthe ĠCNN / Daily ĠMail Ġdataset Ġwas Ġused Ġin Ġn all ap ati 2016 ab stract ive . ĠGraph - based Ġattention ĠB IB REF 16 , Ġpointer - gener ator Ġwith Ġcoverage Ġloss ĠB IB REF 0 Ġare Ġfurther Ġdeveloped Ġto Ġimprove Ġthe Ġgenerated Ġsumm aries . Ġcel iky il m az 2018 deep Ġproposed Ġdeep Ġcommunicating Ġagents Ġfor Ġrepresenting Ġa Ġlong Ġdocument Ġfor Ġabstract ive Ġsummar ization . ĠIn Ġaddition , Ġmany Ġpapers ĠB IB REF 17 , ĠB IB REF 18 , ĠB IB REF 19 Ġuse Ġextract ive Ġmethods Ġto Ġdirectly Ġselect Ġsentences Ġfrom Ġarticles . ĠHowever , Ġnone Ġof Ġthese Ġwork Ġconsidered Ġthe Ġsensational ism Ġof Ġgenerated Ġoutputs . </s> RL Ġis Ġalso Ġgaining Ġpopularity Ġas Ġit Ġcan Ġdirectly Ġoptimize Ġnon - different iable Ġmetrics ĠB IB REF 20 , ĠB IB REF 21 , ĠB IB REF 22 . Ġp aul us 2017 deep Ġproposed Ġan Ġintra - dec oder Ġmodel Ġand Ġcombined ĠRL Ġand ĠM LE Ġto Ġdeal Ġwith Ġsumm aries Ġwith Ġbad Ġqualities . ĠRL Ġhas Ġalso Ġbeen Ġexplored Ġwith Ġgener ative Ġadvers arial Ġnetworks Ġ( GAN s ) ĠB IB REF 23 . Ġli u 2017 gener ative Ġapplied ĠG AN s Ġon Ġsummar ization Ġtask Ġand Ġachieved Ġbetter Ġperformance . Ġn iu 2018 pol ite Ġtackles Ġthe Ġproblem Ġof Ġpolite Ġgeneration Ġwith Ġpolit eness Ġreward . ĠOur Ġwork Ġis Ġdifferent Ġin Ġthat Ġwe Ġpropose Ġa Ġnovel Ġfunction Ġto Ġbalance ĠRL Ġand ĠM LE . </s> Our Ġtask Ġis Ġalso Ġrelated Ġto Ġtext Ġstyle Ġtransfer . ĠImpl icit Ġmethods ĠB IB REF 10 , ĠB IB REF 24 , ĠB IB REF 4 Ġtransfer Ġthe Ġstyles Ġby Ġseparating Ġsentence Ġrepresentations Ġinto Ġcontent Ġand Ġstyle , Ġfor Ġexample Ġusing Ġback - translation BI B REF 4 . ĠHowever , Ġthese Ġmethods Ġcannot Ġguarantee Ġthe Ġcontent Ġconsistency Ġbetween Ġthe Ġoriginal Ġsentence Ġand Ġtransferred Ġoutput ĠB IB REF 25 . ĠExplicit Ġmethods ĠB IB REF 26 , ĠB IB REF 25 Ġtransfer Ġthe Ġstyle Ġby Ġdirectly Ġidentifying Ġstyle Ġrelated Ġkeywords Ġand Ġmodifying Ġthem . ĠHowever , Ġsensational ism Ġis Ġnot Ġalways Ġrestricted Ġto Ġkeywords , Ġbut Ġthe Ġfull Ġsentence . ĠBy Ġleveraging Ġsmall Ġhuman Ġlabeled ĠEnglish Ġdataset , Ġclick b ait Ġdetection Ġhas Ġbeen Ġwell Ġinvestigated ĠB IB REF 2 , ĠB IB REF 27 , ĠB IB REF 3 . ĠHowever , Ġthese Ġhuman Ġlabeled Ġdataset Ġare Ġnot Ġavailable Ġfor Ġother Ġlanguages , Ġsuch Ġas ĠChinese . </s> Mod eling Ġsensational ism Ġis Ġalso Ġrelated Ġto Ġmodeling Ġemotion . ĠEm otion Ġhas Ġbeen Ġwell Ġinvestigated Ġin Ġboth Ġword Ġlevel BI B REF 28 , ĠB IB REF 29 Ġand Ġsentence Ġlevel BI B REF 30 , ĠB IB REF 31 , ĠB IB REF 32 , ĠB IB REF 33 , ĠB IB REF 34 . ĠIt Ġhas Ġalso Ġbeen Ġconsidered Ġan Ġimportant Ġfactor Ġin Ġengaging Ġinteractive Ġsystems BI B REF 35 , ĠB IB REF 36 , ĠB IB REF 37 . ĠAlthough Ġwe Ġobserve Ġthat Ġsensational Ġheadlines Ġcontain Ġemotion , Ġit Ġis Ġstill Ġnot Ġclear Ġwhich Ġemotion Ġand Ġhow Ġemotions Ġwill Ġinfluence Ġthe Ġsensational ism . </s> Conclusion Ġand ĠFuture ĠWork </s> In Ġthis Ġpaper , Ġwe Ġpropose Ġa Ġmodel Ġthat Ġgenerates Ġsensational Ġheadlines Ġwithout Ġlabeled Ġdata Ġusing ĠRein forcement ĠLearning . ĠFirstly , Ġwe Ġpropose Ġa Ġdistant Ġsupervision Ġstrategy Ġto Ġtrain Ġthe Ġsensational ism Ġscorer . ĠAs Ġa Ġresult , Ġwe Ġachieve Ġ65 % Ġaccuracy Ġbetween Ġthe Ġpredicted Ġsensational ism Ġscore Ġand Ġhuman Ġevaluation . ĠTo Ġeffectively Ġleverage Ġthis Ġnoisy Ġsensational ism Ġscore Ġas Ġthe Ġreward Ġfor ĠRL , Ġwe Ġpropose Ġa Ġnovel Ġloss Ġfunction , ĠAR L , Ġto Ġautomatically Ġbalance ĠRL Ġwith ĠM LE . ĠHuman Ġevaluation Ġconfirms Ġthe Ġeffectiveness Ġof Ġboth Ġour Ġsensational ism Ġscorer Ġand ĠAR L Ġto Ġgenerate Ġmore Ġsensational Ġheadlines . ĠFuture Ġwork Ġcan Ġbe Ġimproving Ġthe Ġsensational ism Ġscorer Ġand Ġinvestigating Ġthe Ġapplications Ġof Ġdynamic Ġbalancing Ġmethods Ġbetween ĠRL Ġand ĠM LE Ġin Ġtext GAN BI B REF 23 . ĠOur Ġwork Ġalso Ġraises Ġthe Ġethical Ġquestions Ġabout Ġgenerating Ġsensational Ġheadlines , Ġwhich Ġcan Ġbe Ġfurther Ġexplored . </s> Acknowled gments </s> Thanks Ġto ĠITS / 319 / 16 FP Ġof ĠInnovation ĠTechnology ĠCommission , ĠHK UST Ġ16 248 016 Ġof ĠHong ĠKong ĠResearch ĠGrants ĠCouncil Ġfor Ġfunding . ĠIn Ġaddition , Ġwe Ġthank ĠZhao jiang ĠLin Ġfor Ġhelpful Ġdiscussion Ġand ĠYan ĠXu , ĠZ ih an ĠLiu Ġfor Ġthe Ġdata Ġcollection .
Introduction Headline generation is the process of creating a headline-style sentence given an input article. The research community has been regarding the task of headline generation as a summarization task BIBREF1, ignoring the fundamental differences between headlines and summaries. While summaries aim to contain most of the important information from the articles, headlines do not necessarily need to. Instead, a good headline needs to capture people's attention and serve as an irresistible invitation for users to read through the article. For example, the headline “$2 Billion Worth of Free Media for Trump”, which gives only an intriguing hint, is considered better than the summarization style headline “Measuring Trump’s Media Dominance” , as the former gets almost three times the readers as the latter. Generating headlines with many clicks is especially important in this digital age, because many of the revenues of journalism come from online advertisements and getting more user clicks means being more competitive in the market. However, most existing websites naively generate sensational headlines using only keywords or templates. Instead, this paper aims to learn a model that generates sensational headlines based on an input article without labeled data. To generate sensational headlines, there are two main challenges. Firstly, there is a lack of sensationalism scorer to measure how sensational a headline is. Some researchers have tried to manually label headlines as clickbait or non-clickbait BIBREF2, BIBREF3. However, these human-annotated datasets are usually small and expensive to collect. To capture a large variety of sensationalization patterns, we need a cheap and easy way to collect a large number of sensational headlines. Thus, we propose a distant supervision strategy to collect a sensationalism dataset. We regard headlines receiving lots of comments as sensational samples and the headlines generated by a summarization model as non-sensational samples. Experimental results show that by distinguishing these two types of headlines, we can partially teach the model a sense of being sensational. Secondly, after training a sensationalism scorer on our sensationalism dataset, a natural way to generate sensational headlines is to maximize the sensationalism score using reinforcement learning (RL). However, the following shows an example of a RL model maximizing the sensationalism score by generating a very unnatural sentence, while its sensationalism scorer gave a very high score of 0.99996: UTF8gbsn 十个可穿戴产品的设计原则这消息消息可惜说明 Ten design principles for wearable devices, this message message pity introduction. This happens because the sensationalism scorer can make mistakes and RL can generate unnatural phrases which fools our sensationalism scorer. Thus, how to effectively leverage RL with noisy rewards remains an open problem. To deal with the noisy reward, we introduce Auto-tuned Reinforcement Learning (ARL). Our model automatically tunes the ratio between MLE and RL based on how sensational the training headline is. In this way, we effectively take advantage of RL with a noisy reward to generate headlines that are both sensational and fluent. The major contributions of this paper are as follows: 1) To the best of our knowledge, we propose the first-ever model that tackles the sensational headline generation task with reinforcement learning techniques. 2) Without human-annotated data, we propose a distant supervision strategy to train a sensationalism scorer as a reward function.3) We propose a novel loss function, Auto-tuned Reinforcement Learning, to give dynamic weights to balance between MLE and RL. Our code will be released . Sensationalism Scorer To evaluate the sensationalism intensity score $\alpha _{\text{sen}}$ of a headline, we collect a sensationalism dataset and then train a sensationalism scorer. For the sensationalism dataset collection, we choose headlines with many comments from popular online websites as positive samples. For the negative samples, we propose to use the generated headlines from a sentence summarization model. Intuitively, the summarization model, which is trained to preserve the semantic meaning, will lose the sensationalization ability and thus the generated negative samples will be less sensational than the original one, similar to the obfuscation of style after back-translation BIBREF4. For example, an original headline like UTF8gbsn“一趟挣10万？铁总增开申通、顺丰专列" (One trip to earn 100 thousand? China Railway opens new Shentong and Shunfeng special lines) will become UTF8gbsn“中铁总将增开京广两列快递专列" (China Railway opens two special lines for express) from the baseline model, which loses the sensational phrases of UTF8gbsn“一趟挣10万？" (One trip to earn 100 thousand?) . We then train the sensationalism scorer by classifying sensational and non-sensational headlines using a one-layer CNN with a binary cross entropy loss $L_{\text{sen}}$. Firstly, 1-D convolution is used to extract word features from the input embeddings of a headline. This is followed by a ReLU activation layer and a max-pooling layer along the time dimension. All features from different channels are concatenated together and projected to the sensationalism score by adding another fully connected layer with sigmoid activation. Binary cross entropy is used to compute the loss $L_{\text{sen}}$. Sensationalism Scorer ::: Training Details and Dataset For the CNN model, we choose filter sizes of 1, 3, and 5 respectively. Adam is used to optimize $L_{sen}$ with a learning rate of 0.0001. We set the embedding size as 300 and initialize it from qiu2018revisiting trained on the Weibo corpus with word and character features. We fix the embeddings during training. For dataset collection, we utilize the headlines collected in qin2018automatic, lin2019learning from Tencent News, one of the most popular Chinese news websites, as the positive samples. We follow the same data split as the original paper. As some of the links are not available any more, we get 170,754 training samples and 4,511 validation samples. For the negative training samples collection, we randomly select generated headlines from a pointer generator BIBREF0 model trained on LCSTS dataset BIBREF5 and create a balanced training corpus which includes 351,508 training samples and 9,022 validation samples. To evaluate our trained classifier, we construct a test set by randomly sampling 100 headlines from the test split of LCSTS dataset and the labels are obtained by 11 human annotators. Annotations show that 52% headlines are labeled as positive and 48% headlines as negative by majority voting (The detail on the annotation can be found in Section SECREF26). Sensationalism Scorer ::: Results and Discussion Our classifier achieves 0.65 accuracy and 0.65 averaged F1 score on the test set while a random classifier would only achieve 0.50 accuracy and 0.50 averaged F1 score. This confirms that the predicted sensationalism score can partially capture the sensationalism of headlines. On the other hand, a more natural choice is to take headlines with few comments as negative examples. Thus, we train another baseline classifier on a crawled balanced sensationalism corpus of 84k headlines where the positive headlines have at least 28 comments and the negative headlines have less than 5 comments. However, the results on the test set show that the baseline classifier gets 60% accuracy, which is worse than the proposed classifier (which achieves 65%). The reason could be that the balanced sensationalism corpus are sampled from different distributions from the test set and it is hard for the trained model to generalize. Therefore, we choose the proposed one as our sensationalism scorer. Therefore, our next challenge is to show that how to leverage this noisy sensationalism reward to generate sensational headlines. Sensational Headline Generation Our sensational headline generation model takes an article as input and output a sensational headline. The model consists of a Pointer-Gen headline generator and is trained by ARL. The diagram of ARL can be found in Figure FIGREF6. We denote the input article as $x=\lbrace x_1,x_2,x_3,\cdots ,x_M\rbrace $, and the corresponding headline as $y^*=\lbrace y_1^*,y_2^*,y_3^*,\cdots ,y_T^*\rbrace $, where $M$ is the number of tokens in an article and $T$ is the number of tokens in a headline. Sensational Headline Generation ::: Pointer-Gen Headline Generator We choose Pointer Generator (Pointer-Gen) BIBREF0, a widely used summarization model, as our headline generator for its ability to copy words from the input article. It takes a news article as input and generates a headline. Firstly, the tokens of each article, $\lbrace x_1,x_2,x_3,\cdots ,x_M\rbrace $, are fed into the encoder one-by-one and the encoder generates a sequence of hidden states $h_i$. For each decoding step $t$, the decoder receives the embedding for each token of a headline $y_t$ as input and updates its hidden states $s_t$. An attention mechanism following luong2015effective is used: where $v$, $W_h$, $W_s$, and $b_{attn}$ are the trainable parameters and $h_t^*$ is the context vector. $s_t$ and $h_t^*$ are then combined to give a probability distribution over the vocabulary through two linear layers: where $V$, $b$, $V^{^{\prime }}$, and $b^{^{\prime }}$ are trainable parameters. We use a pointer generator network to enable our model to copy rare/unknown words from the input article, giving the following final word probability: where $x^t$ is the embedding of the input word of the decoder, $w_{h^*}^T$, $w_s^T$, $w_x^T$, and $b_{ptr}$ are trainable parameters, and $\sigma $ is the sigmoid function. Sensational Headline Generation ::: Training Methods We first briefly introduce MLE and RL objective functions, and a naive way to mix these two by a hyper-parameter $\lambda $. Then we point out the challenge of training with noisy reward, and propose ARL to address this issue. Sensational Headline Generation ::: Training Methods ::: MLE and RL A headline generation model can be trained with MLE, RL or a combination of MLE and RL. MLE training is to minimize the negative log likelihood of the training headlines. We feed $y^*$ into the decoder word by word and maximize the likelihood of $y^*$. The loss function for MLE becomes For RL training, we choose the REINFORCE algorithm BIBREF6. In the training phase, after encoding an article, a headline $y^s = \lbrace y_1^s, y_2^s, y_3^s, \cdots , y_T^s\rbrace $ is obtained by sampling from $P(w)$ from our generator, and then a reward of sensationalism or ROUGE(RG) is calculated. We use the baseline reward $\hat{R_t}$ to reduce the variance of the reward, similar to ranzato2015sequence. To elaborate, a linear model is deployed to estimate the baseline reward $\hat{R_t}$ based on $t$-th state $o_t$ for each timestep $t$. The parameters of the linear model are trained by minimizing the mean square loss between $R$ and $\hat{R_t}$: where $W_r$ and $b_r$ are trainable parameters. To maximize the expected reward, our loss function for RL becomes A naive way to mix these two objective functions using a hyper-parameter $\lambda $ has been successfully incorporated in the summarization task BIBREF7. It includes the MLE training as a language model to mitigate the readability and quality issues in RL. The mixed loss function is shown as follows: where $*$ is the reward type. Usually $\lambda $ is large, and paulus2017deep used 0.9984. Sensational Headline Generation ::: Training Methods ::: Auto-tuned Reinforcement Learning Applying the naive mixed training method using sensationalism score as the reward is not obvious/trivial in our task. The main reason is that our sensationalism reward is notably more noisy and more fragile than the ROUGE-L reward or abstractive reward used in the summarization task BIBREF7, BIBREF8. A higher ROUGE-L F1 reward in summarization indicates higher overlapping ratio between generation and true summary statistically, but our sensationalism reward is a learned score which is fragile to be fooled with unnatural samples. To effectively train the model with RL under noisy sensationalism reward, our idea is to balance RL with MLE. However, we argue that the weighted ratio between MLE and RL should be sample-dependent, instead of being fixed for all training samples as in paulus2017deep, kryscinski2018improving. The reason is that, RL and MLE have inconsistent optimization objectives. When the training headline is non-sensational, MLE training will encourage our model to imitate the training headline (thus generating non-sensational headlines), which counteracts the effects of RL training to generate sensational headlines. The sensationalism score is, therefore, used to give dynamic weight to MLE and RL. Our ARL loss function becomes: If $\alpha _{\text{sen}}(y^*)$ is high, meaning the training headline is sensational, our loss function encourages our model to imitate the sample more using the MLE training. If $\alpha _{\text{sen}}(y^*)$ is low, our loss function replies on RL training to improve the sensationalism. Note that the weight $\alpha _{\text{sen}}(y^*)$ is different from our sensationalism reward $\alpha _{\text{sen}}(y^s)$ and we call the loss function Auto-tuned Reinforcement Learning, because the ratio between MLE and RL are well “tuned” towards different samples. Sensational Headline Generation ::: Dataset We use LCSTS BIBREF5 as our dataset to train the summarization model. The dataset is collected from the Chinese microblogging website Sina Weibo. It contains over 2 million Chinese short texts with corresponding headlines given by the author of each text. The dataset is split into 2,400,591 samples for training, 10,666 samples for validation and 725 samples for testing. We tokenize each sentence with Jieba and a vocabulary size of 50000 is saved. Sensational Headline Generation ::: Baselines and Our Models We experiment and compare with the following models. Pointer-Gen is the baseline model trained by optimizing $L_\text{MLE}$ in Equation DISPLAY_FORM13. Pointer-Gen+Pos is the baseline model by training Pointer-Gen only on positive examples whose sensationalism score is larger than 0.5 Pointer-Gen+Same-FT is the model which fine-tunes Pointer-Gen on the training samples whose sensationalism score is larger than 0.1 Pointer-Gen+Pos-FT is the model which fine-tunes Pointer-Gen on the training samples whose sensationalism score is larger than 0.5 Pointer-Gen+RL-ROUGE is the baseline model trained by optimizing $L_\text{RL-ROUGE}$ in Equation DISPLAY_FORM17, with ROUGE-L BIBREF9 as the reward. Pointer-Gen+RL-SEN is the baseline model trained by optimizing $L_\text{RL-SEN}$ in Equation DISPLAY_FORM17, with $\alpha _\text{sen}$ as the reward. Pointer-Gen+ARL-SEN is our model trained by optimizing $L_\text{ARL-SEN}$ in Equation DISPLAY_FORM19, with $\alpha _\text{sen}$ as the reward. Test set is the headlines from the test set. Note that we didn't compare to Pointer-Gen+ARL-ROUGE as it is actually Pointer-GEN. Recall that $\alpha _{\text{sen}}(y^*)$ in Equation DISPLAY_FORM19 measures how good (based on reward function) is $y^*$. Then the loss function for Pointer-Gen+ARL-ROUGE will be We also tried text style transfer baseline BIBREF10, but the generated headlines were very poor (many unknown words and irrelevant). Sensational Headline Generation ::: Training Details MLE training: An Adam optimizer is used with the learning rate of 0.0001 to optimize $L_{\text{MLE}}$. The batch size is set as 128 and a one-layer, bi-directional Long Short-Term Memory (bi-LSTM) model with 512 hidden sizes and a 350 embedding size is utilized. Gradients with the l2 norm larger than 2.0 are clipped. We stop training when the ROUGE-L f-score stops increasing. Hybrid training: An Adam optimizer with a learning rate of 0.0001 is used to optimize $L_{\text{RL-*}}$ (Equation DISPLAY_FORM17) and $L_\text{{ARL-SEN}}$ (Equation DISPLAY_FORM19). When training Pointer-Gen+RL-ROUGE, the best $\lambda $ is chosen based on the ROUGE-L score on the validation set. In our experiment, $\lambda $ is set as 0.95. An Adam optimizer with a learning rate of 0.001 is used to optimize $L_b$. When training Pointer-Gen+ARL-SEN, we don't use the full LCSTS dataset, but only headlines with a sensationalism score larger than 0.1 as we observe that Pointer-Gen+ARL-SEN will generate a few unnatural phrases when using full dataset. We believe the reason is the high ratio of RL during training. Figure FIGREF23 shows that the probability density near 0 is very high, meaning that in each batch, many of the samples will have a very low sensationalism score. On expectation, each sample will receive 0.239 MLE training and 0.761 RL training. This leads to RL dominanting the loss. Thus, we propose to filter samples with a minimum sensationalism score with 0.1 and it works very well. For Pointer-Gen+RL-SEN, we also set the minimum sensationalism score as 0.1, and $\lambda $ is set as 0.5 to remove unnatural phrases, making a fair comparison to Pointer-Gen+ARL-SEN. We stop training Pointer-Gen+Same-FT, Pointer-Gen+Pos-FT, Pointer-Gen+RL-SEN and Pointer-Gen+ARL-SEN, when $\alpha _\text{sen}$ stops increasing on the validation set. Beam-search with a beam size of 5 is adopted for decoding in all models. Sensational Headline Generation ::: Evaluation Metrics We briefly describe the evaluation metrics below. ROUGE: ROUGE is a commonly used evaluation metric for summarization. It measures the N-gram overlap between generated and training headlines. We use it to evaluate the relevance of generated headlines. The widely used pyrouge toolkit is used to calculate ROUGE-1 (RG-1), ROUGE-2 (RG-2), and ROUGE-L (RG-L). Human evaluation: We randomly sample 50 articles from the test set and send the generated headlines from all models and corresponding headlines in the test set to human annotators. We evaluate the sensationalism and fluency of the headlines by setting up two independent human annotation tasks. We ask 10 annotators to label each headline for each task. For the sensationalism annotation, each annotator is asked one question, “Is the headline sensational?”, and he/she has to choose either `yes' or `no'. The annotators were not told which system the headline is from. The process of distributing samples and recruiting annotators is managed by Crowdflower. After annotation, we define the sensationalism score as the proportion of annotations on all generated headlines from one model labeled as `yes'. For the fluency annotation, we repeat the same procedure as for the sensationalism annotation, except that we ask each annotator the question “Is the headline fluent?” We define the fluency score as the proportion of annotations on all headlines from one specific model labeled as `yes'. We put human annotation instructions in the supplemental material. UTF8gbsn Results We first compare all four models, Pointer-Gen, Pointer-Gen-RL+ROUGE, Pointer-Gen-RL-SEN, and Pointer-Gen-ARL-SEN, to existing models with ROUGE in Table TABREF25 to establish that our model produces relevant headlines and we leave the sensationalism for human evaluation. Note that we only compare our models to commonly used strong summarization baselines, to validate that our implementation achieves comparable performance to existing work. In our implementation, Pointer-Gen achieves a 34.51 RG-1 score, 22.21 RG-2 score, and 31.68 RG-L score, which is similar to the results of gu2016incorporating. Pointer-Gen+ARL-SEN, although optimized for the sensationalism reward, achieves similar performance to our Pointer-Gen baseline, which means that Pointer-Gen+ARL-SEN still keeps its summarization ability. An example of headlines generated from different models in Table TABREF29 shows that Pointer-Gen and Pointer-Gen+RL-ROUGE learns to summarize the main point of the article: “The Nikon D600 camera is reported to have black spots when taking photos”. Pointer-Gen+RL-SEN makes the headline more sensational by blaming Nikon for attributing the damage to the smog. Pointer-Gen+ARL-SEN generates the most sensational headline by exaggerating the result “Getting a serious trouble!” to maximize user's attention. We then compare different models using the sensationalism score in Table TABREF30. The Pointer-Gen baseline model achieves a 42.6% sensationalism score, which is the minimum that a typical summarization model achieves. By filtering out low-sensational headlines, Pointer-Gen+Same-FT and Pointer-Gen+Pos-FT achieves higher sensationalism scores, which implies the effectiveness of our sensationalism scorer. Our Pointer-Gen+ARL-SEN model achieves the best performance of 60.8%. This is an absolute improvement of 18.2% over the Pointer-Gen baseline. The Chi-square test on the results confirms that Pointer-Gen+ARL-SEN is statistically significantly more sensational than all the other baseline models, with the largest p-value less than 0.01. Also, we find that the test set headlines achieves 57.8% sensationalism score, much larger than Pointer-Gen baseline, which also supports our intuition that generated headlines will be less sensational than the original one. On the other hand, we found that Pointer-Gen+Pos is much worse than other baselines. The reason is that training on sensational samples alone discards around 80% of the whole training set that is also helpful for maintaining relevance and a good language model. It shows the necessity of using RL. UTF8gbsn In addition, both Pointer-Gen+RL-SEN and Pointer-Gen+ARL-SEN, which use the sensationalism score as the reward, obtain statistically better results than Pointer-Gen+RL-ROUGE and Pointer-Gen, with a p-value less than 0.05 by a Chi-square test. This result shows the effectiveness of RL to generate more sensational headlines. The reason is that even though our noisy classifier could also learn to classify domains, the generator during RL training is not allowed to increase the reward by shifting domains but encouraged to generate more sensational headlines, due to the consistency constraint on the domains of the headline and the article. Furthermore, Poiner-Gen+ARL-SEN gets better performance than Pointer-Gen+RL-SEN, which confirms the superiority of the ARL loss function. We also visualize in Figure FIGREF31 a comparison between Pointer-Gen+ARL-SEN and Pointer-Gen+RL-SEN according to how sensational the test set headlines are. The blue bars denote the smaller scores between the two models. For example, if the blue bar is 0.6, it means that the worse model between Pointer-Gen+RL-SEN and Pointer-Gen+ARL-SEN achieves 0.6. And the color of orange/black further indicates the better model and its score. We find that Pointer-Gen+ARL-SEN outperforms Pointer-Gen+RL-SEN for most cases. The improvement is higher when the test set headlines are not sensational (the sensationalism score is less than 0.5), which may be attributed to the higher ratio of RL training on non-sensational headlines. Apart from the sensationalism evaluation, we measure the fluency of the headlines generated from different models. Fluency scores in Table TABREF30 show that Pointer-Gen+RL-SEN and Pointer-Gen+ARL-SEN achieve comparable fluency performance to Pointer-Gen and Pointer-Gen+RL-ROUGE. Test set headlines achieve the best performance among all models, but the difference is not statistically significant. Also, we observe that fine-tuning on sensational headlines will hurt the performance, both in sensationalism and fluency. After manually checking the outputs, we observe that our model is able to generate sensational headlines using diverse sensationalization strategies. These strategies include, but are not limited to, creating a curiosity gap, asking questions, highlighting numbers, being emotional and emphasizing the user. Examples can be found in Table TABREF32. Related Work Our work is related to summarization tasks. An encoder-decoder model was first applied to two sentence-level abstractive summarization tasks on the DUC-2004 and Gigaword datasets BIBREF12. This model was later extended by selective encoding BIBREF13, a coarse to fine approach BIBREF14, minimum risk training BIBREF1, and topic-aware models BIBREF15. As long summaries were recognized as important, the CNN/Daily Mail dataset was used in nallapati2016abstractive. Graph-based attention BIBREF16, pointer-generator with coverage loss BIBREF0 are further developed to improve the generated summaries. celikyilmaz2018deep proposed deep communicating agents for representing a long document for abstractive summarization. In addition, many papers BIBREF17, BIBREF18, BIBREF19 use extractive methods to directly select sentences from articles. However, none of these work considered the sensationalism of generated outputs. RL is also gaining popularity as it can directly optimize non-differentiable metrics BIBREF20, BIBREF21, BIBREF22. paulus2017deep proposed an intra-decoder model and combined RL and MLE to deal with summaries with bad qualities. RL has also been explored with generative adversarial networks (GANs) BIBREF23. liu2017generative applied GANs on summarization task and achieved better performance. niu2018polite tackles the problem of polite generation with politeness reward. Our work is different in that we propose a novel function to balance RL and MLE. Our task is also related to text style transfer. Implicit methods BIBREF10, BIBREF24, BIBREF4 transfer the styles by separating sentence representations into content and style, for example using back-translationBIBREF4. However, these methods cannot guarantee the content consistency between the original sentence and transferred output BIBREF25. Explicit methods BIBREF26, BIBREF25 transfer the style by directly identifying style related keywords and modifying them. However, sensationalism is not always restricted to keywords, but the full sentence. By leveraging small human labeled English dataset, clickbait detection has been well investigated BIBREF2, BIBREF27, BIBREF3. However, these human labeled dataset are not available for other languages, such as Chinese. Modeling sensationalism is also related to modeling emotion. Emotion has been well investigated in both word levelBIBREF28, BIBREF29 and sentence levelBIBREF30, BIBREF31, BIBREF32, BIBREF33, BIBREF34. It has also been considered an important factor in engaging interactive systemsBIBREF35, BIBREF36, BIBREF37. Although we observe that sensational headlines contain emotion, it is still not clear which emotion and how emotions will influence the sensationalism. Conclusion and Future Work In this paper, we propose a model that generates sensational headlines without labeled data using Reinforcement Learning. Firstly, we propose a distant supervision strategy to train the sensationalism scorer. As a result, we achieve 65% accuracy between the predicted sensationalism score and human evaluation. To effectively leverage this noisy sensationalism score as the reward for RL, we propose a novel loss function, ARL, to automatically balance RL with MLE. Human evaluation confirms the effectiveness of both our sensationalism scorer and ARL to generate more sensational headlines. Future work can be improving the sensationalism scorer and investigating the applications of dynamic balancing methods between RL and MLE in textGANBIBREF23. Our work also raises the ethical questions about generating sensational headlines, which can be further explored. Acknowledgments Thanks to ITS/319/16FP of Innovation Technology Commission, HKUST 16248016 of Hong Kong Research Grants Council for funding. In addition, we thank Zhaojiang Lin for helpful discussion and Yan Xu, Zihan Liu for the data collection.
What is this method improvement over the best performing state-of-the-art?
ae7c93646aa5f3206cd759904965b4d484d12f83
What Ġis Ġthis Ġmethod Ġimprovement Ġover Ġthe Ġbest Ġperforming Ġstate - of - the - art ?
absolute Ġimprovement Ġof Ġ18 . 2 % Ġover Ġthe ĠPo inter - Gen Ġbaseline
<s> What Ġis Ġthis Ġmethod Ġimprovement Ġover Ġthe Ġbest Ġperforming Ġstate - of - the - art ? </s> Introduction </s> Head line Ġgeneration Ġis Ġthe Ġprocess Ġof Ġcreating Ġa Ġheadline - style Ġsentence Ġgiven Ġan Ġinput Ġarticle . ĠThe Ġresearch Ġcommunity Ġhas Ġbeen Ġregarding Ġthe Ġtask Ġof Ġheadline Ġgeneration Ġas Ġa Ġsummar ization Ġtask ĠB IB REF 1 , Ġignoring Ġthe Ġfundamental Ġdifferences Ġbetween Ġheadlines Ġand Ġsumm aries . ĠWhile Ġsumm aries Ġaim Ġto Ġcontain Ġmost Ġof Ġthe Ġimportant Ġinformation Ġfrom Ġthe Ġarticles , Ġheadlines Ġdo Ġnot Ġnecessarily Ġneed Ġto . ĠInstead , Ġa Ġgood Ġheadline Ġneeds Ġto Ġcapture Ġpeople 's Ġattention Ġand Ġserve Ġas Ġan Ġirresistible Ġinvitation Ġfor Ġusers Ġto Ġread Ġthrough Ġthe Ġarticle . ĠFor Ġexample , Ġthe Ġheadline ĠâĢ ľ $ 2 ĠBillion ĠWorth Ġof ĠFree ĠMedia Ġfor ĠTrump âĢ Ŀ , Ġwhich Ġgives Ġonly Ġan Ġintriguing Ġhint , Ġis Ġconsidered Ġbetter Ġthan Ġthe Ġsummar ization Ġstyle Ġheadline ĠâĢ ľ Me asuring ĠTrump âĢ Ļ s ĠMedia ĠDomin ance âĢ Ŀ Ġ, Ġas Ġthe Ġformer Ġgets Ġalmost Ġthree Ġtimes Ġthe Ġreaders Ġas Ġthe Ġlatter . ĠGener ating Ġheadlines Ġwith Ġmany Ġclicks Ġis Ġespecially Ġimportant Ġin Ġthis Ġdigital Ġage , Ġbecause Ġmany Ġof Ġthe Ġrevenues Ġof Ġjournalism Ġcome Ġfrom Ġonline Ġadvertisements Ġand Ġgetting Ġmore Ġuser Ġclicks Ġmeans Ġbeing Ġmore Ġcompetitive Ġin Ġthe Ġmarket . ĠHowever , Ġmost Ġexisting Ġwebsites Ġna ively Ġgenerate Ġsensational Ġheadlines Ġusing Ġonly Ġkeywords Ġor Ġtemplates . ĠInstead , Ġthis Ġpaper Ġaims Ġto Ġlearn Ġa Ġmodel Ġthat Ġgenerates Ġsensational Ġheadlines Ġbased Ġon Ġan Ġinput Ġarticle Ġwithout Ġlabeled Ġdata . </s> To Ġgenerate Ġsensational Ġheadlines , Ġthere Ġare Ġtwo Ġmain Ġchallenges . ĠFirstly , Ġthere Ġis Ġa Ġlack Ġof Ġsensational ism Ġscorer Ġto Ġmeasure Ġhow Ġsensational Ġa Ġheadline Ġis . ĠSome Ġresearchers Ġhave Ġtried Ġto Ġmanually Ġlabel Ġheadlines Ġas Ġclick b ait Ġor Ġnon - click b ait ĠB IB REF 2 , ĠB IB REF 3 . ĠHowever , Ġthese Ġhuman - annot ated Ġdatasets Ġare Ġusually Ġsmall Ġand Ġexpensive Ġto Ġcollect . ĠTo Ġcapture Ġa Ġlarge Ġvariety Ġof Ġsensational ization Ġpatterns , Ġwe Ġneed Ġa Ġcheap Ġand Ġeasy Ġway Ġto Ġcollect Ġa Ġlarge Ġnumber Ġof Ġsensational Ġheadlines . ĠThus , Ġwe Ġpropose Ġa Ġdistant Ġsupervision Ġstrategy Ġto Ġcollect Ġa Ġsensational ism Ġdataset . ĠWe Ġregard Ġheadlines Ġreceiving Ġlots Ġof Ġcomments Ġas Ġsensational Ġsamples Ġand Ġthe Ġheadlines Ġgenerated Ġby Ġa Ġsummar ization Ġmodel Ġas Ġnon - s ens ational Ġsamples . ĠExperimental Ġresults Ġshow Ġthat Ġby Ġdistinguishing Ġthese Ġtwo Ġtypes Ġof Ġheadlines , Ġwe Ġcan Ġpartially Ġteach Ġthe Ġmodel Ġa Ġsense Ġof Ġbeing Ġsensational . </s> Secondly , Ġafter Ġtraining Ġa Ġsensational ism Ġscorer Ġon Ġour Ġsensational ism Ġdataset , Ġa Ġnatural Ġway Ġto Ġgenerate Ġsensational Ġheadlines Ġis Ġto Ġmaximize Ġthe Ġsensational ism Ġscore Ġusing Ġreinforcement Ġlearning Ġ( RL ). ĠHowever , Ġthe Ġfollowing Ġshows Ġan Ġexample Ġof Ġa ĠRL Ġmodel Ġmaximizing Ġthe Ġsensational ism Ġscore Ġby Ġgenerating Ġa Ġvery Ġunnatural Ġsentence , Ġwhile Ġits Ġsensational ism Ġscorer Ġgave Ġa Ġvery Ġhigh Ġscore Ġof Ġ0 . 9999 6 : ĠUTF 8 g bs n Ġå į ģ ä¸ ª åı ¯ ç © ¿ æĪ ´ äº § å ĵ ģ çļĦ è ® ¾ è ® ¡ åİ Ł åĪ Ļ è¿ Ļ æ ¶ Ī æ ģ ¯ æ ¶ Ī æ ģ ¯ åı ¯ æĥ ľ è¯ ´ æĺ İ ĠTen Ġdesign Ġprinciples Ġfor Ġwearable Ġdevices , Ġthis Ġmessage Ġmessage Ġpity Ġintroduction . ĠThis Ġhappens Ġbecause Ġthe Ġsensational ism Ġscorer Ġcan Ġmake Ġmistakes Ġand ĠRL Ġcan Ġgenerate Ġunnatural Ġphrases Ġwhich Ġfools Ġour Ġsensational ism Ġscorer . ĠThus , Ġhow Ġto Ġeffectively Ġleverage ĠRL Ġwith Ġnoisy Ġrewards Ġremains Ġan Ġopen Ġproblem . ĠTo Ġdeal Ġwith Ġthe Ġnoisy Ġreward , Ġwe Ġintroduce ĠAuto - tun ed ĠRein forcement ĠLearning Ġ( AR L ). ĠOur Ġmodel Ġautomatically Ġtunes Ġthe Ġratio Ġbetween ĠM LE Ġand ĠRL Ġbased Ġon Ġhow Ġsensational Ġthe Ġtraining Ġheadline Ġis . ĠIn Ġthis Ġway , Ġwe Ġeffectively Ġtake Ġadvantage Ġof ĠRL Ġwith Ġa Ġnoisy Ġreward Ġto Ġgenerate Ġheadlines Ġthat Ġare Ġboth Ġsensational Ġand Ġfluent . </s> The Ġmajor Ġcontributions Ġof Ġthis Ġpaper Ġare Ġas Ġfollows : Ġ1 ) ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge , Ġwe Ġpropose Ġthe Ġfirst - ever Ġmodel Ġthat Ġtackles Ġthe Ġsensational Ġheadline Ġgeneration Ġtask Ġwith Ġreinforcement Ġlearning Ġtechniques . Ġ2 ) ĠWithout Ġhuman - annot ated Ġdata , Ġwe Ġpropose Ġa Ġdistant Ġsupervision Ġstrategy Ġto Ġtrain Ġa Ġsensational ism Ġscorer Ġas Ġa Ġreward Ġfunction . 3 ) ĠWe Ġpropose Ġa Ġnovel Ġloss Ġfunction , ĠAuto - tun ed ĠRein forcement ĠLearning , Ġto Ġgive Ġdynamic Ġweights Ġto Ġbalance Ġbetween ĠM LE Ġand ĠRL . ĠOur Ġcode Ġwill Ġbe Ġreleased Ġ. </s> S ens ational ism ĠSc orer </s> To Ġevaluate Ġthe Ġsensational ism Ġintensity Ġscore Ġ$\ alpha Ġ_ {\ text { sen }} $ Ġof Ġa Ġheadline , Ġwe Ġcollect Ġa Ġsensational ism Ġdataset Ġand Ġthen Ġtrain Ġa Ġsensational ism Ġscorer . ĠFor Ġthe Ġsensational ism Ġdataset Ġcollection , Ġwe Ġchoose Ġheadlines Ġwith Ġmany Ġcomments Ġfrom Ġpopular Ġonline Ġwebsites Ġas Ġpositive Ġsamples . ĠFor Ġthe Ġnegative Ġsamples , Ġwe Ġpropose Ġto Ġuse Ġthe Ġgenerated Ġheadlines Ġfrom Ġa Ġsentence Ġsummar ization Ġmodel . ĠInt uitive ly , Ġthe Ġsummar ization Ġmodel , Ġwhich Ġis Ġtrained Ġto Ġpreserve Ġthe Ġsemantic Ġmeaning , Ġwill Ġlose Ġthe Ġsensational ization Ġability Ġand Ġthus Ġthe Ġgenerated Ġnegative Ġsamples Ġwill Ġbe Ġless Ġsensational Ġthan Ġthe Ġoriginal Ġone , Ġsimilar Ġto Ġthe Ġobfusc ation Ġof Ġstyle Ġafter Ġback - translation ĠB IB REF 4 . ĠFor Ġexample , Ġan Ġoriginal Ġheadline Ġlike ĠUTF 8 g bs n âĢ ľ ä¸Ģ è ¶ Ł æ Į £ 10 ä¸ ĩ ï ¼ Ł é ĵ ģ æĢ » å ¢ ŀ å¼ Ģ çĶ ³ éĢ ļ ãĢģ é ¡ º ä¸ ° ä¸ ĵ åĪ Ĺ " Ġ( One Ġtrip Ġto Ġearn Ġ100 Ġthousand ? ĠChina ĠRailway Ġopens Ġnew ĠShe nt ong Ġand ĠSh un f eng Ġspecial Ġlines ) Ġwill Ġbecome ĠUTF 8 g bs n âĢ ľ ä¸Ń é ĵ ģ æĢ » å°Ĩ å ¢ ŀ å¼ Ģ äº ¬ å¹ ¿ ä¸ ¤ åĪ Ĺ å¿ « éĢ Ĵ ä¸ ĵ åĪ Ĺ " Ġ( China ĠRailway Ġopens Ġtwo Ġspecial Ġlines Ġfor Ġexpress ) Ġfrom Ġthe Ġbaseline Ġmodel , Ġwhich Ġloses Ġthe Ġsensational Ġphrases Ġof ĠUTF 8 g bs n âĢ ľ ä¸Ģ è ¶ Ł æ Į £ 10 ä¸ ĩ ï ¼ Ł " Ġ( One Ġtrip Ġto Ġearn Ġ100 Ġthousand ?) Ġ. ĠWe Ġthen Ġtrain Ġthe Ġsensational ism Ġscorer Ġby Ġclass ifying Ġsensational Ġand Ġnon - s ens ational Ġheadlines Ġusing Ġa Ġone - layer ĠCNN Ġwith Ġa Ġbinary Ġcross Ġentropy Ġloss Ġ$ L _{ \ text { sen }} $. ĠFirstly , Ġ1 - D Ġconv olution Ġis Ġused Ġto Ġextract Ġword Ġfeatures Ġfrom Ġthe Ġinput Ġembed d ings Ġof Ġa Ġheadline . ĠThis Ġis Ġfollowed Ġby Ġa ĠRe LU Ġactivation Ġlayer Ġand Ġa Ġmax - pool ing Ġlayer Ġalong Ġthe Ġtime Ġdimension . ĠAll Ġfeatures Ġfrom Ġdifferent Ġchannels Ġare Ġconc aten ated Ġtogether Ġand Ġprojected Ġto Ġthe Ġsensational ism Ġscore Ġby Ġadding Ġanother Ġfully Ġconnected Ġlayer Ġwith Ġs igm oid Ġactivation . ĠBinary Ġcross Ġentropy Ġis Ġused Ġto Ġcompute Ġthe Ġloss Ġ$ L _{ \ text { sen }} $. </s> S ens ational ism ĠSc orer Ġ: :: ĠTraining ĠDetails Ġand ĠDat as et </s> For Ġthe ĠCNN Ġmodel , Ġwe Ġchoose Ġfilter Ġsizes Ġof Ġ1 , Ġ3 , Ġand Ġ5 Ġrespectively . ĠAdam Ġis Ġused Ġto Ġoptimize Ġ$ L _{ sen } $ Ġwith Ġa Ġlearning Ġrate Ġof Ġ0 . 0001 . ĠWe Ġset Ġthe Ġembed ding Ġsize Ġas Ġ300 Ġand Ġinitialize Ġit Ġfrom Ġq iu 2018 re vis iting Ġtrained Ġon Ġthe ĠWe ibo Ġcorpus Ġwith Ġword Ġand Ġcharacter Ġfeatures . ĠWe Ġfix Ġthe Ġembed d ings Ġduring Ġtraining . ĠFor Ġdataset Ġcollection , Ġwe Ġutilize Ġthe Ġheadlines Ġcollected Ġin Ġq in 2018 automatic , Ġlin 2019 learning Ġfrom ĠTen cent ĠNews , Ġone Ġof Ġthe Ġmost Ġpopular ĠChinese Ġnews Ġwebsites , Ġas Ġthe Ġpositive Ġsamples . ĠWe Ġfollow Ġthe Ġsame Ġdata Ġsplit Ġas Ġthe Ġoriginal Ġpaper . ĠAs Ġsome Ġof Ġthe Ġlinks Ġare Ġnot Ġavailable Ġany Ġmore , Ġwe Ġget Ġ170 , 754 Ġtraining Ġsamples Ġand Ġ4 , 511 Ġvalidation Ġsamples . ĠFor Ġthe Ġnegative Ġtraining Ġsamples Ġcollection , Ġwe Ġrandomly Ġselect Ġgenerated Ġheadlines Ġfrom Ġa Ġpointer Ġgenerator ĠB IB REF 0 Ġmodel Ġtrained Ġon ĠLC ST S Ġdataset ĠB IB REF 5 Ġand Ġcreate Ġa Ġbalanced Ġtraining Ġcorpus Ġwhich Ġincludes Ġ351 , 508 Ġtraining Ġsamples Ġand Ġ9 , 022 Ġvalidation Ġsamples . ĠTo Ġevaluate Ġour Ġtrained Ġclass ifier , Ġwe Ġconstruct Ġa Ġtest Ġset Ġby Ġrandomly Ġsampling Ġ100 Ġheadlines Ġfrom Ġthe Ġtest Ġsplit Ġof ĠLC ST S Ġdataset Ġand Ġthe Ġlabels Ġare Ġobtained Ġby Ġ11 Ġhuman Ġannot ators . ĠAnnotations Ġshow Ġthat Ġ52 % Ġheadlines Ġare Ġlabeled Ġas Ġpositive Ġand Ġ48 % Ġheadlines Ġas Ġnegative Ġby Ġmajority Ġvoting Ġ( The Ġdetail Ġon Ġthe Ġannotation Ġcan Ġbe Ġfound Ġin ĠSection ĠSEC REF 26 ). </s> S ens ational ism ĠSc orer Ġ: :: ĠResults Ġand ĠDiscussion </s> Our Ġclass ifier Ġachieves Ġ0 . 65 Ġaccuracy Ġand Ġ0 . 65 Ġaveraged ĠF 1 Ġscore Ġon Ġthe Ġtest Ġset Ġwhile Ġa Ġrandom Ġclass ifier Ġwould Ġonly Ġachieve Ġ0 . 50 Ġaccuracy Ġand Ġ0 . 50 Ġaveraged ĠF 1 Ġscore . ĠThis Ġconfirms Ġthat Ġthe Ġpredicted Ġsensational ism Ġscore Ġcan Ġpartially Ġcapture Ġthe Ġsensational ism Ġof Ġheadlines . ĠOn Ġthe Ġother Ġhand , Ġa Ġmore Ġnatural Ġchoice Ġis Ġto Ġtake Ġheadlines Ġwith Ġfew Ġcomments Ġas Ġnegative Ġexamples . ĠThus , Ġwe Ġtrain Ġanother Ġbaseline Ġclass ifier Ġon Ġa Ġcrawled Ġbalanced Ġsensational ism Ġcorpus Ġof Ġ84 k Ġheadlines Ġwhere Ġthe Ġpositive Ġheadlines Ġhave Ġat Ġleast Ġ28 Ġcomments Ġand Ġthe Ġnegative Ġheadlines Ġhave Ġless Ġthan Ġ5 Ġcomments . ĠHowever , Ġthe Ġresults Ġon Ġthe Ġtest Ġset Ġshow Ġthat Ġthe Ġbaseline Ġclass ifier Ġgets Ġ60 % Ġaccuracy , Ġwhich Ġis Ġworse Ġthan Ġthe Ġproposed Ġclass ifier Ġ( which Ġachieves Ġ65 %). ĠThe Ġreason Ġcould Ġbe Ġthat Ġthe Ġbalanced Ġsensational ism Ġcorpus Ġare Ġsampled Ġfrom Ġdifferent Ġdistributions Ġfrom Ġthe Ġtest Ġset Ġand Ġit Ġis Ġhard Ġfor Ġthe Ġtrained Ġmodel Ġto Ġgeneral ize . ĠTherefore , Ġwe Ġchoose Ġthe Ġproposed Ġone Ġas Ġour Ġsensational ism Ġscorer . ĠTherefore , Ġour Ġnext Ġchallenge Ġis Ġto Ġshow Ġthat Ġhow Ġto Ġleverage Ġthis Ġnoisy Ġsensational ism Ġreward Ġto Ġgenerate Ġsensational Ġheadlines . </s> S ens ational ĠHead line ĠGeneration </s> Our Ġsensational Ġheadline Ġgeneration Ġmodel Ġtakes Ġan Ġarticle Ġas Ġinput Ġand Ġoutput Ġa Ġsensational Ġheadline . ĠThe Ġmodel Ġconsists Ġof Ġa ĠPo inter - Gen Ġheadline Ġgenerator Ġand Ġis Ġtrained Ġby ĠAR L . ĠThe Ġdiagram Ġof ĠAR L Ġcan Ġbe Ġfound Ġin ĠFigure ĠFIG REF 6 . </s> We Ġdenote Ġthe Ġinput Ġarticle Ġas Ġ$ x = \ l brace Ġx _ 1 , x _ 2 , x _ 3 , \ cd ots Ġ, x _ M \ r brace Ġ$ , Ġand Ġthe Ġcorresponding Ġheadline Ġas Ġ$ y ^ * = \ l brace Ġy _ 1 ^ *, y _ 2 ^ *, y _ 3 ^ *, \ cd ots Ġ, y _ T ^ * \ r brace Ġ$ , Ġwhere Ġ$ M $ Ġis Ġthe Ġnumber Ġof Ġtokens Ġin Ġan Ġarticle Ġand Ġ$ T $ Ġis Ġthe Ġnumber Ġof Ġtokens Ġin Ġa Ġheadline . </s> S ens ational ĠHead line ĠGeneration Ġ: :: ĠPo inter - Gen ĠHead line ĠGenerator </s> We Ġchoose ĠPo inter ĠGenerator Ġ( Po inter - Gen ) ĠB IB REF 0 , Ġa Ġwidely Ġused Ġsummar ization Ġmodel , Ġas Ġour Ġheadline Ġgenerator Ġfor Ġits Ġability Ġto Ġcopy Ġwords Ġfrom Ġthe Ġinput Ġarticle . ĠIt Ġtakes Ġa Ġnews Ġarticle Ġas Ġinput Ġand Ġgenerates Ġa Ġheadline . ĠFirstly , Ġthe Ġtokens Ġof Ġeach Ġarticle , Ġ$\ l brace Ġx _ 1 , x _ 2 , x _ 3 , \ cd ots Ġ, x _ M \ r brace Ġ$ , Ġare Ġfed Ġinto Ġthe Ġenc oder Ġone - by - one Ġand Ġthe Ġenc oder Ġgenerates Ġa Ġsequence Ġof Ġhidden Ġstates Ġ$ h _ i $. ĠFor Ġeach Ġdecoding Ġstep Ġ$ t $, Ġthe Ġdec oder Ġreceives Ġthe Ġembed ding Ġfor Ġeach Ġtoken Ġof Ġa Ġheadline Ġ$ y _ t $ Ġas Ġinput Ġand Ġupdates Ġits Ġhidden Ġstates Ġ$ s _ t $. ĠAn Ġattention Ġmechanism Ġfollowing Ġl u ong 2015 effective Ġis Ġused : </s> where Ġ$ v $, Ġ$ W _ h $, Ġ$ W _ s $, Ġand Ġ$ b _{ att n } $ Ġare Ġthe Ġtrain able Ġparameters Ġand Ġ$ h _ t ^ * $ Ġis Ġthe Ġcontext Ġvector . Ġ$ s _ t $ Ġand Ġ$ h _ t ^ * $ Ġare Ġthen Ġcombined Ġto Ġgive Ġa Ġprobability Ġdistribution Ġover Ġthe Ġvocabulary Ġthrough Ġtwo Ġlinear Ġlayers : </s> where Ġ$ V $, Ġ$ b $, Ġ$ V ^{ ^ {\ prime Ġ}} $, Ġand Ġ$ b ^{ ^ {\ prime Ġ}} $ Ġare Ġtrain able Ġparameters . ĠWe Ġuse Ġa Ġpointer Ġgenerator Ġnetwork Ġto Ġenable Ġour Ġmodel Ġto Ġcopy Ġrare / unknown Ġwords Ġfrom Ġthe Ġinput Ġarticle , Ġgiving Ġthe Ġfollowing Ġfinal Ġword Ġprobability : </s> where Ġ$ x ^ t $ Ġis Ġthe Ġembed ding Ġof Ġthe Ġinput Ġword Ġof Ġthe Ġdec oder , Ġ$ w _{ h ^ * } ^ T $, Ġ$ w _ s ^ T $, Ġ$ w _ x ^ T $, Ġand Ġ$ b _{ ptr } $ Ġare Ġtrain able Ġparameters , Ġand Ġ$\ s igma Ġ$ Ġis Ġthe Ġs igm oid Ġfunction . </s> S ens ational ĠHead line ĠGeneration Ġ: :: ĠTraining ĠMethods </s> We Ġfirst Ġbriefly Ġintroduce ĠM LE Ġand ĠRL Ġobjective Ġfunctions , Ġand Ġa Ġnaive Ġway Ġto Ġmix Ġthese Ġtwo Ġby Ġa Ġhyper - param eter Ġ$\ lambda Ġ$ . ĠThen Ġwe Ġpoint Ġout Ġthe Ġchallenge Ġof Ġtraining Ġwith Ġnoisy Ġreward , Ġand Ġpropose ĠAR L Ġto Ġaddress Ġthis Ġissue . </s> S ens ational ĠHead line ĠGeneration Ġ: :: ĠTraining ĠMethods Ġ: :: ĠM LE Ġand ĠRL </s> A Ġheadline Ġgeneration Ġmodel Ġcan Ġbe Ġtrained Ġwith ĠM LE , ĠRL Ġor Ġa Ġcombination Ġof ĠM LE Ġand ĠRL . ĠM LE Ġtraining Ġis Ġto Ġminimize Ġthe Ġnegative Ġlog Ġlikelihood Ġof Ġthe Ġtraining Ġheadlines . ĠWe Ġfeed Ġ$ y ^ * $ Ġinto Ġthe Ġdec oder Ġword Ġby Ġword Ġand Ġmaximize Ġthe Ġlikelihood Ġof Ġ$ y ^ * $. ĠThe Ġloss Ġfunction Ġfor ĠM LE Ġbecomes </s> For ĠRL Ġtraining , Ġwe Ġchoose Ġthe ĠRE IN FOR CE Ġalgorithm ĠB IB REF 6 . ĠIn Ġthe Ġtraining Ġphase , Ġafter Ġencoding Ġan Ġarticle , Ġa Ġheadline Ġ$ y ^ s Ġ= Ġ\ l brace Ġy _ 1 ^ s , Ġy _ 2 ^ s , Ġy _ 3 ^ s , Ġ\ cd ots Ġ, Ġy _ T ^ s \ r brace Ġ$ Ġis Ġobtained Ġby Ġsampling Ġfrom Ġ$ P ( w ) $ Ġfrom Ġour Ġgenerator , Ġand Ġthen Ġa Ġreward Ġof Ġsensational ism Ġor ĠR OU GE ( RG ) Ġis Ġcalculated . </s> We Ġuse Ġthe Ġbaseline Ġreward Ġ$\ hat { R _ t } $ Ġto Ġreduce Ġthe Ġvariance Ġof Ġthe Ġreward , Ġsimilar Ġto Ġran z ato 2015 sequence . ĠTo Ġelaborate , Ġa Ġlinear Ġmodel Ġis Ġdeployed Ġto Ġestimate Ġthe Ġbaseline Ġreward Ġ$\ hat { R _ t } $ Ġbased Ġon Ġ$ t $ - th Ġstate Ġ$ o _ t $ Ġfor Ġeach Ġtim est ep Ġ$ t $. ĠThe Ġparameters Ġof Ġthe Ġlinear Ġmodel Ġare Ġtrained Ġby Ġminimizing Ġthe Ġmean Ġsquare Ġloss Ġbetween Ġ$ R $ Ġand Ġ$\ hat { R _ t } $ : </s> where Ġ$ W _ r $ Ġand Ġ$ b _ r $ Ġare Ġtrain able Ġparameters . ĠTo Ġmaximize Ġthe Ġexpected Ġreward , Ġour Ġloss Ġfunction Ġfor ĠRL Ġbecomes </s> A Ġnaive Ġway Ġto Ġmix Ġthese Ġtwo Ġobjective Ġfunctions Ġusing Ġa Ġhyper - param eter Ġ$\ lambda Ġ$ Ġhas Ġbeen Ġsuccessfully Ġincorporated Ġin Ġthe Ġsummar ization Ġtask ĠB IB REF 7 . ĠIt Ġincludes Ġthe ĠM LE Ġtraining Ġas Ġa Ġlanguage Ġmodel Ġto Ġmitigate Ġthe Ġread ability Ġand Ġquality Ġissues Ġin ĠRL . ĠThe Ġmixed Ġloss Ġfunction Ġis Ġshown Ġas Ġfollows : </s> where Ġ$ * $ Ġis Ġthe Ġreward Ġtype . ĠUsually Ġ$\ lambda Ġ$ Ġis Ġlarge , Ġand Ġp aul us 2017 deep Ġused Ġ0 . 99 84 . </s> S ens ational ĠHead line ĠGeneration Ġ: :: ĠTraining ĠMethods Ġ: :: ĠAuto - tun ed ĠRein forcement ĠLearning </s> App lying Ġthe Ġnaive Ġmixed Ġtraining Ġmethod Ġusing Ġsensational ism Ġscore Ġas Ġthe Ġreward Ġis Ġnot Ġobvious / t riv ial Ġin Ġour Ġtask . ĠThe Ġmain Ġreason Ġis Ġthat Ġour Ġsensational ism Ġreward Ġis Ġnotably Ġmore Ġnoisy Ġand Ġmore Ġfragile Ġthan Ġthe ĠR OU GE - L Ġreward Ġor Ġabstract ive Ġreward Ġused Ġin Ġthe Ġsummar ization Ġtask ĠB IB REF 7 , ĠB IB REF 8 . ĠA Ġhigher ĠR OU GE - L ĠF 1 Ġreward Ġin Ġsummar ization Ġindicates Ġhigher Ġoverlapping Ġratio Ġbetween Ġgeneration Ġand Ġtrue Ġsummary Ġstatistically , Ġbut Ġour Ġsensational ism Ġreward Ġis Ġa Ġlearned Ġscore Ġwhich Ġis Ġfragile Ġto Ġbe Ġfooled Ġwith Ġunnatural Ġsamples . </s> To Ġeffectively Ġtrain Ġthe Ġmodel Ġwith ĠRL Ġunder Ġnoisy Ġsensational ism Ġreward , Ġour Ġidea Ġis Ġto Ġbalance ĠRL Ġwith ĠM LE . ĠHowever , Ġwe Ġargue Ġthat Ġthe Ġweighted Ġratio Ġbetween ĠM LE Ġand ĠRL Ġshould Ġbe Ġsample - dependent , Ġinstead Ġof Ġbeing Ġfixed Ġfor Ġall Ġtraining Ġsamples Ġas Ġin Ġp aul us 2017 deep , Ġk ry sc inski 2018 impro ving . ĠThe Ġreason Ġis Ġthat , ĠRL Ġand ĠM LE Ġhave Ġinconsistent Ġoptimization Ġobjectives . ĠWhen Ġthe Ġtraining Ġheadline Ġis Ġnon - s ens ational , ĠM LE Ġtraining Ġwill Ġencourage Ġour Ġmodel Ġto Ġimitate Ġthe Ġtraining Ġheadline Ġ( thus Ġgenerating Ġnon - s ens ational Ġheadlines ), Ġwhich Ġcounter acts Ġthe Ġeffects Ġof ĠRL Ġtraining Ġto Ġgenerate Ġsensational Ġheadlines . </s> The Ġsensational ism Ġscore Ġis , Ġtherefore , Ġused Ġto Ġgive Ġdynamic Ġweight Ġto ĠM LE Ġand ĠRL . ĠOur ĠAR L Ġloss Ġfunction Ġbecomes : </s> If Ġ$\ alpha Ġ_ {\ text { sen }} ( y ^ *) $ Ġis Ġhigh , Ġmeaning Ġthe Ġtraining Ġheadline Ġis Ġsensational , Ġour Ġloss Ġfunction Ġencourages Ġour Ġmodel Ġto Ġimitate Ġthe Ġsample Ġmore Ġusing Ġthe ĠM LE Ġtraining . ĠIf Ġ$\ alpha Ġ_ {\ text { sen }} ( y ^ *) $ Ġis Ġlow , Ġour Ġloss Ġfunction Ġreplies Ġon ĠRL Ġtraining Ġto Ġimprove Ġthe Ġsensational ism . ĠNote Ġthat Ġthe Ġweight Ġ$\ alpha Ġ_ {\ text { sen }} ( y ^ *) $ Ġis Ġdifferent Ġfrom Ġour Ġsensational ism Ġreward Ġ$\ alpha Ġ_ {\ text { sen }} ( y ^ s ) $ Ġand Ġwe Ġcall Ġthe Ġloss Ġfunction ĠAuto - tun ed ĠRein forcement ĠLearning , Ġbecause Ġthe Ġratio Ġbetween ĠM LE Ġand ĠRL Ġare Ġwell ĠâĢ ľ tun ed âĢ Ŀ Ġtowards Ġdifferent Ġsamples . </s> S ens ational ĠHead line ĠGeneration Ġ: :: ĠDat as et </s> We Ġuse ĠLC ST S ĠB IB REF 5 Ġas Ġour Ġdataset Ġto Ġtrain Ġthe Ġsummar ization Ġmodel . ĠThe Ġdataset Ġis Ġcollected Ġfrom Ġthe ĠChinese Ġmicro blog ging Ġwebsite ĠSina ĠWe ibo . ĠIt Ġcontains Ġover Ġ2 Ġmillion ĠChinese Ġshort Ġtexts Ġwith Ġcorresponding Ġheadlines Ġgiven Ġby Ġthe Ġauthor Ġof Ġeach Ġtext . ĠThe Ġdataset Ġis Ġsplit Ġinto Ġ2 , 400 , 591 Ġsamples Ġfor Ġtraining , Ġ10 , 666 Ġsamples Ġfor Ġvalidation Ġand Ġ7 25 Ġsamples Ġfor Ġtesting . ĠWe Ġtoken ize Ġeach Ġsentence Ġwith ĠJ ie ba Ġand Ġa Ġvocabulary Ġsize Ġof Ġ5 0000 Ġis Ġsaved . </s> S ens ational ĠHead line ĠGeneration Ġ: :: ĠBas elines Ġand ĠOur ĠModels </s> We Ġexperiment Ġand Ġcompare Ġwith Ġthe Ġfollowing Ġmodels . </s> Po inter - Gen Ġis Ġthe Ġbaseline Ġmodel Ġtrained Ġby Ġoptimizing Ġ$ L _ \ text { M LE } $ Ġin ĠEqu ation ĠDIS PLAY _ FORM 13 . </s> Po inter - Gen + Pos Ġis Ġthe Ġbaseline Ġmodel Ġby Ġtraining ĠPo inter - Gen Ġonly Ġon Ġpositive Ġexamples Ġwhose Ġsensational ism Ġscore Ġis Ġlarger Ġthan Ġ0 . 5 </s> Po inter - Gen + Same - FT Ġis Ġthe Ġmodel Ġwhich Ġfine - t unes ĠPo inter - Gen Ġon Ġthe Ġtraining Ġsamples Ġwhose Ġsensational ism Ġscore Ġis Ġlarger Ġthan Ġ0 . 1 </s> Po inter - Gen + Pos - FT Ġis Ġthe Ġmodel Ġwhich Ġfine - t unes ĠPo inter - Gen Ġon Ġthe Ġtraining Ġsamples Ġwhose Ġsensational ism Ġscore Ġis Ġlarger Ġthan Ġ0 . 5 </s> Po inter - Gen + RL - R OU GE Ġis Ġthe Ġbaseline Ġmodel Ġtrained Ġby Ġoptimizing Ġ$ L _ \ text { RL - R OU GE } $ Ġin ĠEqu ation ĠDIS PLAY _ FORM 17 , Ġwith ĠR OU GE - L ĠB IB REF 9 Ġas Ġthe Ġreward . </s> Po inter - Gen + RL - S EN Ġis Ġthe Ġbaseline Ġmodel Ġtrained Ġby Ġoptimizing Ġ$ L _ \ text { RL - S EN } $ Ġin ĠEqu ation ĠDIS PLAY _ FORM 17 , Ġwith Ġ$\ alpha Ġ_ \ text { sen } $ Ġas Ġthe Ġreward . </s> Po inter - Gen + AR L - S EN Ġis Ġour Ġmodel Ġtrained Ġby Ġoptimizing Ġ$ L _ \ text { AR L - S EN } $ Ġin ĠEqu ation ĠDIS PLAY _ FORM 19 , Ġwith Ġ$\ alpha Ġ_ \ text { sen } $ Ġas Ġthe Ġreward . </s> Test Ġset Ġis Ġthe Ġheadlines Ġfrom Ġthe Ġtest Ġset . </s> Note Ġthat Ġwe Ġdidn 't Ġcompare Ġto ĠPo inter - Gen + AR L - R OU GE Ġas Ġit Ġis Ġactually ĠPo inter - GEN . ĠRecall Ġthat Ġ$\ alpha Ġ_ {\ text { sen }} ( y ^ *) $ Ġin ĠEqu ation ĠDIS PLAY _ FORM 19 Ġmeasures Ġhow Ġgood Ġ( based Ġon Ġreward Ġfunction ) Ġis Ġ$ y ^ * $. ĠThen Ġthe Ġloss Ġfunction Ġfor ĠPo inter - Gen + AR L - R OU GE Ġwill Ġbe </s> We Ġalso Ġtried Ġtext Ġstyle Ġtransfer Ġbaseline ĠB IB REF 10 , Ġbut Ġthe Ġgenerated Ġheadlines Ġwere Ġvery Ġpoor Ġ( many Ġunknown Ġwords Ġand Ġirrelevant ). </s> S ens ational ĠHead line ĠGeneration Ġ: :: ĠTraining ĠDetails </s> M LE Ġtraining : ĠAn ĠAdam Ġoptim izer Ġis Ġused Ġwith Ġthe Ġlearning Ġrate Ġof Ġ0 . 0001 Ġto Ġoptimize Ġ$ L _{ \ text { M LE }} $. ĠThe Ġbatch Ġsize Ġis Ġset Ġas Ġ128 Ġand Ġa Ġone - layer , Ġbi - direction al ĠLong ĠShort - Term ĠMemory Ġ( bi - L ST M ) Ġmodel Ġwith Ġ512 Ġhidden Ġsizes Ġand Ġa Ġ350 Ġembed ding Ġsize Ġis Ġutilized . ĠGrad ients Ġwith Ġthe Ġl 2 Ġnorm Ġlarger Ġthan Ġ2 . 0 Ġare Ġclipped . ĠWe Ġstop Ġtraining Ġwhen Ġthe ĠR OU GE - L Ġf - score Ġstops Ġincreasing . </s> Hy brid Ġtraining : ĠAn ĠAdam Ġoptim izer Ġwith Ġa Ġlearning Ġrate Ġof Ġ0 . 0001 Ġis Ġused Ġto Ġoptimize Ġ$ L _{ \ text { RL - * }} $ Ġ( Equ ation ĠDIS PLAY _ FORM 17 ) Ġand Ġ$ L _ \ text {{ AR L - S EN }} $ Ġ( Equ ation ĠDIS PLAY _ FORM 19 ). ĠWhen Ġtraining ĠPo inter - Gen + RL - R OU GE , Ġthe Ġbest Ġ$\ lambda Ġ$ Ġis Ġchosen Ġbased Ġon Ġthe ĠR OU GE - L Ġscore Ġon Ġthe Ġvalidation Ġset . ĠIn Ġour Ġexperiment , Ġ$\ lambda Ġ$ Ġis Ġset Ġas Ġ0 . 95 . ĠAn ĠAdam Ġoptim izer Ġwith Ġa Ġlearning Ġrate Ġof Ġ0 . 001 Ġis Ġused Ġto Ġoptimize Ġ$ L _ b $. ĠWhen Ġtraining ĠPo inter - Gen + AR L - S EN , Ġwe Ġdon 't Ġuse Ġthe Ġfull ĠLC ST S Ġdataset , Ġbut Ġonly Ġheadlines Ġwith Ġa Ġsensational ism Ġscore Ġlarger Ġthan Ġ0 . 1 Ġas Ġwe Ġobserve Ġthat ĠPo inter - Gen + AR L - S EN Ġwill Ġgenerate Ġa Ġfew Ġunnatural Ġphrases Ġwhen Ġusing Ġfull Ġdataset . ĠWe Ġbelieve Ġthe Ġreason Ġis Ġthe Ġhigh Ġratio Ġof ĠRL Ġduring Ġtraining . ĠFigure ĠFIG REF 23 Ġshows Ġthat Ġthe Ġprobability Ġdensity Ġnear Ġ0 Ġis Ġvery Ġhigh , Ġmeaning Ġthat Ġin Ġeach Ġbatch , Ġmany Ġof Ġthe Ġsamples Ġwill Ġhave Ġa Ġvery Ġlow Ġsensational ism Ġscore . ĠOn Ġexpectation , Ġeach Ġsample Ġwill Ġreceive Ġ0 . 239 ĠM LE Ġtraining Ġand Ġ0 . 76 1 ĠRL Ġtraining . ĠThis Ġleads Ġto ĠRL Ġdominant ing Ġthe Ġloss . ĠThus , Ġwe Ġpropose Ġto Ġfilter Ġsamples Ġwith Ġa Ġminimum Ġsensational ism Ġscore Ġwith Ġ0 . 1 Ġand Ġit Ġworks Ġvery Ġwell . ĠFor ĠPo inter - Gen + RL - S EN , Ġwe Ġalso Ġset Ġthe Ġminimum Ġsensational ism Ġscore Ġas Ġ0 . 1 , Ġand Ġ$\ lambda Ġ$ Ġis Ġset Ġas Ġ0 . 5 Ġto Ġremove Ġunnatural Ġphrases , Ġmaking Ġa Ġfair Ġcomparison Ġto ĠPo inter - Gen + AR L - S EN . </s> We Ġstop Ġtraining ĠPo inter - Gen + Same - FT , ĠPo inter - Gen + Pos - FT , ĠPo inter - Gen + RL - S EN Ġand ĠPo inter - Gen + AR L - S EN , Ġwhen Ġ$\ alpha Ġ_ \ text { sen } $ Ġstops Ġincreasing Ġon Ġthe Ġvalidation Ġset . ĠBeam - search Ġwith Ġa Ġbeam Ġsize Ġof Ġ5 Ġis Ġadopted Ġfor Ġdecoding Ġin Ġall Ġmodels . </s> S ens ational ĠHead line ĠGeneration Ġ: :: ĠEvaluation ĠMet rics </s> We Ġbriefly Ġdescribe Ġthe Ġevaluation Ġmetrics Ġbelow . </s> R OU GE : ĠR OU GE Ġis Ġa Ġcommonly Ġused Ġevaluation Ġmetric Ġfor Ġsummar ization . ĠIt Ġmeasures Ġthe ĠN - gram Ġoverlap Ġbetween Ġgenerated Ġand Ġtraining Ġheadlines . ĠWe Ġuse Ġit Ġto Ġevaluate Ġthe Ġrelevance Ġof Ġgenerated Ġheadlines . ĠThe Ġwidely Ġused Ġpy rou ge Ġtool kit Ġis Ġused Ġto Ġcalculate ĠR OU GE - 1 Ġ( RG - 1 ), ĠR OU GE - 2 Ġ( RG - 2 ), Ġand ĠR OU GE - L Ġ( RG - L ). </s> Human Ġevaluation : ĠWe Ġrandomly Ġsample Ġ50 Ġarticles Ġfrom Ġthe Ġtest Ġset Ġand Ġsend Ġthe Ġgenerated Ġheadlines Ġfrom Ġall Ġmodels Ġand Ġcorresponding Ġheadlines Ġin Ġthe Ġtest Ġset Ġto Ġhuman Ġannot ators . ĠWe Ġevaluate Ġthe Ġsensational ism Ġand Ġflu ency Ġof Ġthe Ġheadlines Ġby Ġsetting Ġup Ġtwo Ġindependent Ġhuman Ġannotation Ġtasks . ĠWe Ġask Ġ10 Ġannot ators Ġto Ġlabel Ġeach Ġheadline Ġfor Ġeach Ġtask . ĠFor Ġthe Ġsensational ism Ġannotation , Ġeach Ġannot ator Ġis Ġasked Ġone Ġquestion , ĠâĢ ľ Is Ġthe Ġheadline Ġsensational ? âĢ Ŀ , Ġand Ġhe / she Ġhas Ġto Ġchoose Ġeither Ġ` yes ' Ġor Ġ` no '. ĠThe Ġannot ators Ġwere Ġnot Ġtold Ġwhich Ġsystem Ġthe Ġheadline Ġis Ġfrom . ĠThe Ġprocess Ġof Ġdistributing Ġsamples Ġand Ġrecruiting Ġannot ators Ġis Ġmanaged Ġby ĠCrowd flower . ĠAfter Ġannotation , Ġwe Ġdefine Ġthe Ġsensational ism Ġscore Ġas Ġthe Ġproportion Ġof Ġannotations Ġon Ġall Ġgenerated Ġheadlines Ġfrom Ġone Ġmodel Ġlabeled Ġas Ġ` yes '. ĠFor Ġthe Ġflu ency Ġannotation , Ġwe Ġrepeat Ġthe Ġsame Ġprocedure Ġas Ġfor Ġthe Ġsensational ism Ġannotation , Ġexcept Ġthat Ġwe Ġask Ġeach Ġannot ator Ġthe Ġquestion ĠâĢ ľ Is Ġthe Ġheadline Ġfluent ? âĢ Ŀ ĠWe Ġdefine Ġthe Ġflu ency Ġscore Ġas Ġthe Ġproportion Ġof Ġannotations Ġon Ġall Ġheadlines Ġfrom Ġone Ġspecific Ġmodel Ġlabeled Ġas Ġ` yes '. ĠWe Ġput Ġhuman Ġannotation Ġinstructions Ġin Ġthe Ġsupplemental Ġmaterial . </s> UTF 8 g bs n </s> Results </s> We Ġfirst Ġcompare Ġall Ġfour Ġmodels , ĠPo inter - Gen , ĠPo inter - Gen - RL + R OU GE , ĠPo inter - Gen - RL - S EN , Ġand ĠPo inter - Gen - AR L - S EN , Ġto Ġexisting Ġmodels Ġwith ĠR OU GE Ġin ĠTable ĠT AB REF 25 Ġto Ġestablish Ġthat Ġour Ġmodel Ġproduces Ġrelevant Ġheadlines Ġand Ġwe Ġleave Ġthe Ġsensational ism Ġfor Ġhuman Ġevaluation . ĠNote Ġthat Ġwe Ġonly Ġcompare Ġour Ġmodels Ġto Ġcommonly Ġused Ġstrong Ġsummar ization Ġbas elines , Ġto Ġvalidate Ġthat Ġour Ġimplementation Ġachieves Ġcomparable Ġperformance Ġto Ġexisting Ġwork . ĠIn Ġour Ġimplementation , ĠPo inter - Gen Ġachieves Ġa Ġ34 . 51 ĠRG - 1 Ġscore , Ġ22 . 21 ĠRG - 2 Ġscore , Ġand Ġ31 . 68 ĠRG - L Ġscore , Ġwhich Ġis Ġsimilar Ġto Ġthe Ġresults Ġof Ġgu 2016 inc orpor ating . ĠPo inter - Gen + AR L - S EN , Ġalthough Ġoptimized Ġfor Ġthe Ġsensational ism Ġreward , Ġachieves Ġsimilar Ġperformance Ġto Ġour ĠPo inter - Gen Ġbaseline , Ġwhich Ġmeans Ġthat ĠPo inter - Gen + AR L - S EN Ġstill Ġkeeps Ġits Ġsummar ization Ġability . ĠAn Ġexample Ġof Ġheadlines Ġgenerated Ġfrom Ġdifferent Ġmodels Ġin ĠTable ĠT AB REF 29 Ġshows Ġthat ĠPo inter - Gen Ġand ĠPo inter - Gen + RL - R OU GE Ġlearns Ġto Ġsummarize Ġthe Ġmain Ġpoint Ġof Ġthe Ġarticle : ĠâĢ ľ The ĠNikon ĠD 600 Ġcamera Ġis Ġreported Ġto Ġhave Ġblack Ġspots Ġwhen Ġtaking Ġphotos âĢ Ŀ . ĠPo inter - Gen + RL - S EN Ġmakes Ġthe Ġheadline Ġmore Ġsensational Ġby Ġblaming ĠNikon Ġfor Ġattribut ing Ġthe Ġdamage Ġto Ġthe Ġsm og . ĠPo inter - Gen + AR L - S EN Ġgenerates Ġthe Ġmost Ġsensational Ġheadline Ġby Ġexagger ating Ġthe Ġresult ĠâĢ ľ Getting Ġa Ġserious Ġtrouble ! âĢ Ŀ Ġto Ġmaximize Ġuser 's Ġattention . </s> We Ġthen Ġcompare Ġdifferent Ġmodels Ġusing Ġthe Ġsensational ism Ġscore Ġin ĠTable ĠT AB REF 30 . ĠThe ĠPo inter - Gen Ġbaseline Ġmodel Ġachieves Ġa Ġ42 . 6 % Ġsensational ism Ġscore , Ġwhich Ġis Ġthe Ġminimum Ġthat Ġa Ġtypical Ġsummar ization Ġmodel Ġachieves . ĠBy Ġfiltering Ġout Ġlow - s ens ational Ġheadlines , ĠPo inter - Gen + Same - FT Ġand ĠPo inter - Gen + Pos - FT Ġachieves Ġhigher Ġsensational ism Ġscores , Ġwhich Ġimplies Ġthe Ġeffectiveness Ġof Ġour Ġsensational ism Ġscorer . ĠOur ĠPo inter - Gen + AR L - S EN Ġmodel Ġachieves Ġthe Ġbest Ġperformance Ġof Ġ60 . 8 %. ĠThis Ġis Ġan Ġabsolute Ġimprovement Ġof Ġ18 . 2 % Ġover Ġthe ĠPo inter - Gen Ġbaseline . ĠThe ĠChi - square Ġtest Ġon Ġthe Ġresults Ġconfirms Ġthat ĠPo inter - Gen + AR L - S EN Ġis Ġstatistically Ġsignificantly Ġmore Ġsensational Ġthan Ġall Ġthe Ġother Ġbaseline Ġmodels , Ġwith Ġthe Ġlargest Ġp - value Ġless Ġthan Ġ0 . 01 . ĠAlso , Ġwe Ġfind Ġthat Ġthe Ġtest Ġset Ġheadlines Ġachieves Ġ57 . 8 % Ġsensational ism Ġscore , Ġmuch Ġlarger Ġthan ĠPo inter - Gen Ġbaseline , Ġwhich Ġalso Ġsupports Ġour Ġintuition Ġthat Ġgenerated Ġheadlines Ġwill Ġbe Ġless Ġsensational Ġthan Ġthe Ġoriginal Ġone . ĠOn Ġthe Ġother Ġhand , Ġwe Ġfound Ġthat ĠPo inter - Gen + Pos Ġis Ġmuch Ġworse Ġthan Ġother Ġbas elines . ĠThe Ġreason Ġis Ġthat Ġtraining Ġon Ġsensational Ġsamples Ġalone Ġdisc ards Ġaround Ġ80 % Ġof Ġthe Ġwhole Ġtraining Ġset Ġthat Ġis Ġalso Ġhelpful Ġfor Ġmaintaining Ġrelevance Ġand Ġa Ġgood Ġlanguage Ġmodel . ĠIt Ġshows Ġthe Ġnecessity Ġof Ġusing ĠRL . </s> UTF 8 g bs n </s> In Ġaddition , Ġboth ĠPo inter - Gen + RL - S EN Ġand ĠPo inter - Gen + AR L - S EN , Ġwhich Ġuse Ġthe Ġsensational ism Ġscore Ġas Ġthe Ġreward , Ġobtain Ġstatistically Ġbetter Ġresults Ġthan ĠPo inter - Gen + RL - R OU GE Ġand ĠPo inter - Gen , Ġwith Ġa Ġp - value Ġless Ġthan Ġ0 . 05 Ġby Ġa ĠChi - square Ġtest . ĠThis Ġresult Ġshows Ġthe Ġeffectiveness Ġof ĠRL Ġto Ġgenerate Ġmore Ġsensational Ġheadlines . ĠThe Ġreason Ġis Ġthat Ġeven Ġthough Ġour Ġnoisy Ġclass ifier Ġcould Ġalso Ġlearn Ġto Ġclassify Ġdomains , Ġthe Ġgenerator Ġduring ĠRL Ġtraining Ġis Ġnot Ġallowed Ġto Ġincrease Ġthe Ġreward Ġby Ġshifting Ġdomains Ġbut Ġencouraged Ġto Ġgenerate Ġmore Ġsensational Ġheadlines , Ġdue Ġto Ġthe Ġconsistency Ġconstraint Ġon Ġthe Ġdomains Ġof Ġthe Ġheadline Ġand Ġthe Ġarticle . ĠFurthermore , ĠPo iner - Gen + AR L - S EN Ġgets Ġbetter Ġperformance Ġthan ĠPo inter - Gen + RL - S EN , Ġwhich Ġconfirms Ġthe Ġsuperiority Ġof Ġthe ĠAR L Ġloss Ġfunction . ĠWe Ġalso Ġvisualize Ġin ĠFigure ĠFIG REF 31 Ġa Ġcomparison Ġbetween ĠPo inter - Gen + AR L - S EN Ġand ĠPo inter - Gen + RL - S EN Ġaccording Ġto Ġhow Ġsensational Ġthe Ġtest Ġset Ġheadlines Ġare . ĠThe Ġblue Ġbars Ġdenote Ġthe Ġsmaller Ġscores Ġbetween Ġthe Ġtwo Ġmodels . ĠFor Ġexample , Ġif Ġthe Ġblue Ġbar Ġis Ġ0 . 6 , Ġit Ġmeans Ġthat Ġthe Ġworse Ġmodel Ġbetween ĠPo inter - Gen + RL - S EN Ġand ĠPo inter - Gen + AR L - S EN Ġachieves Ġ0 . 6 . ĠAnd Ġthe Ġcolor Ġof Ġorange / black Ġfurther Ġindicates Ġthe Ġbetter Ġmodel Ġand Ġits Ġscore . ĠWe Ġfind Ġthat ĠPo inter - Gen + AR L - S EN Ġoutper forms ĠPo inter - Gen + RL - S EN Ġfor Ġmost Ġcases . ĠThe Ġimprovement Ġis Ġhigher Ġwhen Ġthe Ġtest Ġset Ġheadlines Ġare Ġnot Ġsensational Ġ( the Ġsensational ism Ġscore Ġis Ġless Ġthan Ġ0 . 5 ), Ġwhich Ġmay Ġbe Ġattributed Ġto Ġthe Ġhigher Ġratio Ġof ĠRL Ġtraining Ġon Ġnon - s ens ational Ġheadlines . </s> Apart Ġfrom Ġthe Ġsensational ism Ġevaluation , Ġwe Ġmeasure Ġthe Ġflu ency Ġof Ġthe Ġheadlines Ġgenerated Ġfrom Ġdifferent Ġmodels . ĠFlu ency Ġscores Ġin ĠTable ĠT AB REF 30 Ġshow Ġthat ĠPo inter - Gen + RL - S EN Ġand ĠPo inter - Gen + AR L - S EN Ġachieve Ġcomparable Ġflu ency Ġperformance Ġto ĠPo inter - Gen Ġand ĠPo inter - Gen + RL - R OU GE . ĠTest Ġset Ġheadlines Ġachieve Ġthe Ġbest Ġperformance Ġamong Ġall Ġmodels , Ġbut Ġthe Ġdifference Ġis Ġnot Ġstatistically Ġsignificant . ĠAlso , Ġwe Ġobserve Ġthat Ġfine - tun ing Ġon Ġsensational Ġheadlines Ġwill Ġhurt Ġthe Ġperformance , Ġboth Ġin Ġsensational ism Ġand Ġflu ency . </s> After Ġmanually Ġchecking Ġthe Ġoutputs , Ġwe Ġobserve Ġthat Ġour Ġmodel Ġis Ġable Ġto Ġgenerate Ġsensational Ġheadlines Ġusing Ġdiverse Ġsensational ization Ġstrategies . ĠThese Ġstrategies Ġinclude , Ġbut Ġare Ġnot Ġlimited Ġto , Ġcreating Ġa Ġcuriosity Ġgap , Ġasking Ġquestions , Ġhighlighting Ġnumbers , Ġbeing Ġemotional Ġand Ġemphasizing Ġthe Ġuser . ĠExamples Ġcan Ġbe Ġfound Ġin ĠTable ĠT AB REF 32 . </s> Related ĠWork </s> Our Ġwork Ġis Ġrelated Ġto Ġsummar ization Ġtasks . ĠAn Ġenc oder - dec oder Ġmodel Ġwas Ġfirst Ġapplied Ġto Ġtwo Ġsentence - level Ġabstract ive Ġsummar ization Ġtasks Ġon Ġthe ĠD UC - 2004 Ġand ĠGig aw ord Ġdatasets ĠB IB REF 12 . ĠThis Ġmodel Ġwas Ġlater Ġextended Ġby Ġselective Ġencoding ĠB IB REF 13 , Ġa Ġcoarse Ġto Ġfine Ġapproach ĠB IB REF 14 , Ġminimum Ġrisk Ġtraining ĠB IB REF 1 , Ġand Ġtopic - aware Ġmodels ĠB IB REF 15 . ĠAs Ġlong Ġsumm aries Ġwere Ġrecognized Ġas Ġimportant , Ġthe ĠCNN / Daily ĠMail Ġdataset Ġwas Ġused Ġin Ġn all ap ati 2016 ab stract ive . ĠGraph - based Ġattention ĠB IB REF 16 , Ġpointer - gener ator Ġwith Ġcoverage Ġloss ĠB IB REF 0 Ġare Ġfurther Ġdeveloped Ġto Ġimprove Ġthe Ġgenerated Ġsumm aries . Ġcel iky il m az 2018 deep Ġproposed Ġdeep Ġcommunicating Ġagents Ġfor Ġrepresenting Ġa Ġlong Ġdocument Ġfor Ġabstract ive Ġsummar ization . ĠIn Ġaddition , Ġmany Ġpapers ĠB IB REF 17 , ĠB IB REF 18 , ĠB IB REF 19 Ġuse Ġextract ive Ġmethods Ġto Ġdirectly Ġselect Ġsentences Ġfrom Ġarticles . ĠHowever , Ġnone Ġof Ġthese Ġwork Ġconsidered Ġthe Ġsensational ism Ġof Ġgenerated Ġoutputs . </s> RL Ġis Ġalso Ġgaining Ġpopularity Ġas Ġit Ġcan Ġdirectly Ġoptimize Ġnon - different iable Ġmetrics ĠB IB REF 20 , ĠB IB REF 21 , ĠB IB REF 22 . Ġp aul us 2017 deep Ġproposed Ġan Ġintra - dec oder Ġmodel Ġand Ġcombined ĠRL Ġand ĠM LE Ġto Ġdeal Ġwith Ġsumm aries Ġwith Ġbad Ġqualities . ĠRL Ġhas Ġalso Ġbeen Ġexplored Ġwith Ġgener ative Ġadvers arial Ġnetworks Ġ( GAN s ) ĠB IB REF 23 . Ġli u 2017 gener ative Ġapplied ĠG AN s Ġon Ġsummar ization Ġtask Ġand Ġachieved Ġbetter Ġperformance . Ġn iu 2018 pol ite Ġtackles Ġthe Ġproblem Ġof Ġpolite Ġgeneration Ġwith Ġpolit eness Ġreward . ĠOur Ġwork Ġis Ġdifferent Ġin Ġthat Ġwe Ġpropose Ġa Ġnovel Ġfunction Ġto Ġbalance ĠRL Ġand ĠM LE . </s> Our Ġtask Ġis Ġalso Ġrelated Ġto Ġtext Ġstyle Ġtransfer . ĠImpl icit Ġmethods ĠB IB REF 10 , ĠB IB REF 24 , ĠB IB REF 4 Ġtransfer Ġthe Ġstyles Ġby Ġseparating Ġsentence Ġrepresentations Ġinto Ġcontent Ġand Ġstyle , Ġfor Ġexample Ġusing Ġback - translation BI B REF 4 . ĠHowever , Ġthese Ġmethods Ġcannot Ġguarantee Ġthe Ġcontent Ġconsistency Ġbetween Ġthe Ġoriginal Ġsentence Ġand Ġtransferred Ġoutput ĠB IB REF 25 . ĠExplicit Ġmethods ĠB IB REF 26 , ĠB IB REF 25 Ġtransfer Ġthe Ġstyle Ġby Ġdirectly Ġidentifying Ġstyle Ġrelated Ġkeywords Ġand Ġmodifying Ġthem . ĠHowever , Ġsensational ism Ġis Ġnot Ġalways Ġrestricted Ġto Ġkeywords , Ġbut Ġthe Ġfull Ġsentence . ĠBy Ġleveraging Ġsmall Ġhuman Ġlabeled ĠEnglish Ġdataset , Ġclick b ait Ġdetection Ġhas Ġbeen Ġwell Ġinvestigated ĠB IB REF 2 , ĠB IB REF 27 , ĠB IB REF 3 . ĠHowever , Ġthese Ġhuman Ġlabeled Ġdataset Ġare Ġnot Ġavailable Ġfor Ġother Ġlanguages , Ġsuch Ġas ĠChinese . </s> Mod eling Ġsensational ism Ġis Ġalso Ġrelated Ġto Ġmodeling Ġemotion . ĠEm otion Ġhas Ġbeen Ġwell Ġinvestigated Ġin Ġboth Ġword Ġlevel BI B REF 28 , ĠB IB REF 29 Ġand Ġsentence Ġlevel BI B REF 30 , ĠB IB REF 31 , ĠB IB REF 32 , ĠB IB REF 33 , ĠB IB REF 34 . ĠIt Ġhas Ġalso Ġbeen Ġconsidered Ġan Ġimportant Ġfactor Ġin Ġengaging Ġinteractive Ġsystems BI B REF 35 , ĠB IB REF 36 , ĠB IB REF 37 . ĠAlthough Ġwe Ġobserve Ġthat Ġsensational Ġheadlines Ġcontain Ġemotion , Ġit Ġis Ġstill Ġnot Ġclear Ġwhich Ġemotion Ġand Ġhow Ġemotions Ġwill Ġinfluence Ġthe Ġsensational ism . </s> Conclusion Ġand ĠFuture ĠWork </s> In Ġthis Ġpaper , Ġwe Ġpropose Ġa Ġmodel Ġthat Ġgenerates Ġsensational Ġheadlines Ġwithout Ġlabeled Ġdata Ġusing ĠRein forcement ĠLearning . ĠFirstly , Ġwe Ġpropose Ġa Ġdistant Ġsupervision Ġstrategy Ġto Ġtrain Ġthe Ġsensational ism Ġscorer . ĠAs Ġa Ġresult , Ġwe Ġachieve Ġ65 % Ġaccuracy Ġbetween Ġthe Ġpredicted Ġsensational ism Ġscore Ġand Ġhuman Ġevaluation . ĠTo Ġeffectively Ġleverage Ġthis Ġnoisy Ġsensational ism Ġscore Ġas Ġthe Ġreward Ġfor ĠRL , Ġwe Ġpropose Ġa Ġnovel Ġloss Ġfunction , ĠAR L , Ġto Ġautomatically Ġbalance ĠRL Ġwith ĠM LE . ĠHuman Ġevaluation Ġconfirms Ġthe Ġeffectiveness Ġof Ġboth Ġour Ġsensational ism Ġscorer Ġand ĠAR L Ġto Ġgenerate Ġmore Ġsensational Ġheadlines . ĠFuture Ġwork Ġcan Ġbe Ġimproving Ġthe Ġsensational ism Ġscorer Ġand Ġinvestigating Ġthe Ġapplications Ġof Ġdynamic Ġbalancing Ġmethods Ġbetween ĠRL Ġand ĠM LE Ġin Ġtext GAN BI B REF 23 . ĠOur Ġwork Ġalso Ġraises Ġthe Ġethical Ġquestions Ġabout Ġgenerating Ġsensational Ġheadlines , Ġwhich Ġcan Ġbe Ġfurther Ġexplored . </s> Acknowled gments </s> Thanks Ġto ĠITS / 319 / 16 FP Ġof ĠInnovation ĠTechnology ĠCommission , ĠHK UST Ġ16 248 016 Ġof ĠHong ĠKong ĠResearch ĠGrants ĠCouncil Ġfor Ġfunding . ĠIn Ġaddition , Ġwe Ġthank ĠZhao jiang ĠLin Ġfor Ġhelpful Ġdiscussion Ġand ĠYan ĠXu , ĠZ ih an ĠLiu Ġfor Ġthe Ġdata Ġcollection .