average unique predictions, illustrate the difference of predictions between our proposed models, we show an example chosen from the KP20k validation set
EXTRACTIVE
To verify our assumption that target encoding and orthogonal regularization help to boost the diversity of generated sequences, we use two metrics, one quantitative and one qualitative, to measure diversity of generation. First, we simply calculate the average unique predictions produced by both INLINEFORM0 and INLINEFORM1 in experiments shown in Section SECREF36 . The resulting numbers are 20.38 and 89.70 for INLINEFORM2 and INLINEFORM3 respectively. Second, from the model running on the KP20k validation set, we randomly sample 2000 decoder hidden states at INLINEFORM4 steps following a delimiter ( INLINEFORM5 ) and apply an unsupervised clustering method (t-SNE BIBREF35 ) on them. From the Figure FIGREF46 we can see that hidden states sampled from INLINEFORM6 are easier to cluster while hidden states sampled from INLINEFORM7 yield one mass of vectors with no obvious distinct clusters. Results on both metrics suggest target encoding and orthogonal regularization indeed help diversifying generation of our model. To illustrate the difference of predictions between our proposed models, we show an example chosen from the KP20k validation set in Appendix SECREF10 . In this example there are 29 ground truth phrases. Neither of the models is able to generate all of the keyphrases, but it is obvious that the predictions from INLINEFORM0 all start with “test”, while predictions from INLINEFORM1 are diverse. This to some extent verifies our assumption that without the target encoder and orthogonal regularization, decoder states following delimiters are less diverse.
1810.05241
Introduction </s> Key phrase Ġgeneration Ġis Ġthe Ġtask Ġof Ġautomatically Ġpredicting Ġkey ph r ases Ġgiven Ġa Ġsource Ġtext . ĠDes ired Ġkey ph r ases Ġare Ġoften Ġmulti - word Ġunits Ġthat Ġsummarize Ġthe Ġhigh - level Ġmeaning Ġand Ġhighlight Ġcertain Ġimportant Ġtopics Ġor Ġinformation Ġof Ġthe Ġsource Ġtext . ĠConsequently , Ġmodels Ġthat Ġcan Ġsuccessfully Ġperform Ġthis Ġtask Ġshould Ġbe Ġcapable Ġof Ġnot Ġonly Ġdist illing Ġhigh - level Ġinformation Ġfrom Ġa Ġdocument , Ġbut Ġalso Ġlocating Ġspecific , Ġimportant Ġsnippets Ġtherein . </s> To Ġmake Ġthe Ġproblem Ġeven Ġmore Ġchallenging , Ġa Ġkey phrase Ġmay Ġor Ġmay Ġnot Ġbe Ġa Ġsubst ring Ġof Ġthe Ġsource Ġtext Ġ( i . e ., Ġit Ġmay Ġbe Ġpresent Ġor Ġabsent ). ĠMoreover , Ġa Ġgiven Ġsource Ġtext Ġis Ġusually Ġassociated Ġwith Ġa Ġset Ġof Ġmultiple Ġkey ph r ases . ĠThus , Ġkey phrase Ġgeneration Ġis Ġan Ġinstance Ġof Ġthe Ġset Ġgeneration Ġproblem , Ġwhere Ġboth Ġthe Ġsize Ġof Ġthe Ġset Ġand Ġthe Ġsize Ġ( i . e ., Ġthe Ġnumber Ġof Ġtokens Ġin Ġa Ġphrase ) Ġof Ġeach Ġelement Ġcan Ġvary Ġdepending Ġon Ġthe Ġsource . </s> Similar Ġto Ġsummar ization , Ġkey phrase Ġgeneration Ġis Ġoften Ġformulated Ġas Ġa Ġsequence - to - sequence Ġ( Se q 2 Se q ) Ġgeneration Ġtask Ġin Ġmost Ġprior Ġstudies ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ. ĠCondition ed Ġon Ġa Ġsource Ġtext , ĠSe q 2 Se q Ġmodels Ġgenerate Ġphrases Ġindividually Ġor Ġas Ġa Ġlonger Ġsequence Ġjoint ed Ġby Ġdelim iting Ġtokens . ĠSince Ġstandard ĠSe q 2 Se q Ġmodels Ġgenerate Ġonly Ġone Ġsequence Ġat Ġa Ġtime , Ġthus Ġto Ġgenerate Ġmultiple Ġphrases , Ġa Ġcommon Ġapproach Ġis Ġto Ġover - gener ate Ġusing Ġbeam Ġsearch Ġwith Ġa Ġlarge Ġbeam Ġwidth . ĠModels Ġare Ġthen Ġevaluated Ġby Ġtaking Ġa Ġfixed Ġnumber Ġof Ġtop Ġpredicted Ġphrases Ġ( typically Ġ5 Ġor Ġ10 ) Ġand Ġcomparing Ġthem Ġagainst Ġthe Ġground Ġtruth Ġkey ph r ases . </s> Though Ġthis Ġapproach Ġhas Ġachieved Ġgood Ġempirical Ġresults , Ġwe Ġargue Ġthat Ġit Ġsuffers Ġfrom Ġtwo Ġmajor Ġlimitations . ĠFirstly , Ġmodels Ġthat Ġuse Ġbeam Ġsearch Ġto Ġgenerate Ġmultiple Ġkey ph r ases Ġgenerally Ġlack Ġthe Ġability Ġto Ġdetermine Ġthe Ġdynamic Ġnumber Ġof Ġkey ph r ases Ġneeded Ġfor Ġdifferent Ġsource Ġtexts . ĠMeanwhile , Ġthe Ġparallel ism Ġin Ġbeam Ġsearch Ġalso Ġfails Ġto Ġmodel Ġthe Ġinter - relation Ġamong Ġthe Ġgenerated Ġphrases , Ġwhich Ġcan Ġoften Ġresult Ġin Ġdiminished Ġdiversity Ġin Ġthe Ġoutput . ĠAlthough Ġcertain Ġexisting Ġmodels Ġtake Ġoutput Ġdiversity Ġinto Ġconsideration Ġduring Ġtraining ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, Ġthe Ġeffort Ġis Ġsignificantly Ġundermined Ġduring Ġdecoding Ġdue Ġto Ġthe Ġreliance Ġon Ġover - generation Ġand Ġphrase Ġranking Ġwith Ġbeam Ġsearch . </s> Secondly , Ġthe Ġcurrent Ġevaluation Ġsetup Ġis Ġrather Ġproblematic , Ġsince Ġexisting Ġstudies Ġattempt Ġto Ġmatch Ġa Ġfixed Ġnumber Ġof Ġoutputs Ġagainst Ġa Ġvariable Ġnumber Ġof Ġground Ġtruth Ġkey ph r ases . ĠEm pir ically , Ġthe Ġnumber Ġof Ġkey ph r ases Ġcan Ġvary Ġdrastically Ġfor Ġdifferent Ġsource Ġtexts , Ġdepending Ġon Ġa Ġplethora Ġof Ġfactors Ġincluding Ġthe Ġlength Ġor Ġgenre Ġof Ġthe Ġtext , Ġthe Ġgran ularity Ġof Ġkey phrase Ġannotation , Ġetc . ĠFor Ġthe Ġseveral Ġcommonly Ġused Ġkey phrase Ġgeneration Ġdatasets , Ġfor Ġexample , Ġthe Ġaverage Ġnumber Ġof Ġkey ph r ases Ġper Ġdata Ġpoint Ġcan Ġrange Ġfrom Ġ5 . 3 Ġto Ġ15 . 7 , Ġwith Ġvar iances Ġsometimes Ġas Ġlarge Ġas Ġ64 . 6 Ġ( Table ĠT AB REF 1 Ġ). ĠTherefore , Ġusing Ġan Ġarbitrary , Ġfixed Ġnumber ĠIN LINE FORM 0 Ġto Ġevaluate Ġentire Ġdatasets Ġis Ġnot Ġappropriate . ĠIn Ġfact , Ġunder Ġthis Ġevaluation Ġsetup , Ġthe ĠF 1 Ġscore Ġfor Ġthe Ġor acle Ġmodel Ġon Ġthe ĠKP 20 k Ġdataset Ġis Ġ0 . 8 58 Ġfor ĠIN LINE FORM 1 Ġand Ġ0 . 626 Ġfor ĠIN LINE FORM 2 Ġ, Ġwhich Ġapparently Ġposes Ġserious Ġnormal ization Ġissues Ġas Ġevaluation Ġmetrics . </s> To Ġovercome Ġthese Ġproblems , Ġwe Ġpropose Ġnovel Ġdecoding Ġstrategies Ġand Ġevaluation Ġmetrics Ġfor Ġthe Ġkey phrase Ġgeneration Ġtask . ĠThe Ġmain Ġcontributions Ġof Ġthis Ġwork Ġare Ġas Ġfollows : </s> Key phrase ĠExt raction Ġand ĠGeneration </s> Traditional Ġkey phrase Ġextraction Ġhas Ġbeen Ġstudied Ġextensively Ġin Ġpast Ġdecades . ĠIn Ġmost Ġexisting Ġliterature , Ġkey phrase Ġextraction Ġhas Ġbeen Ġformulated Ġas Ġa Ġtwo - step Ġprocess . ĠFirst , Ġlex ical Ġfeatures Ġsuch Ġas Ġpart - of - speech Ġtags Ġare Ġused Ġto Ġdetermine Ġa Ġlist Ġof Ġphrase Ġcandidates Ġby Ġhe uristic Ġmethods ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ. ĠSecond , Ġa Ġranking Ġalgorithm Ġis Ġadopted Ġto Ġrank Ġthe Ġcandidate Ġlist Ġand Ġthe Ġtop Ġranked Ġcandidates Ġare Ġselected Ġas Ġkey ph r ases . ĠA Ġwide Ġvariety Ġof Ġmethods Ġwere Ġapplied Ġfor Ġranking , Ġsuch Ġas Ġbag ged Ġdecision Ġtrees ĠB IB REF 8 Ġ, ĠB IB REF 9 Ġ, ĠMulti - Layer ĠPer cept ron , ĠSupport ĠVector ĠMachine ĠB IB REF 9 Ġand ĠPage Rank ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ. ĠRecently , ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġused Ġsequence Ġlabeling Ġmodels Ġto Ġextract Ġkey ph r ases Ġfrom Ġtext . ĠSimilarly , ĠB IB REF 16 Ġused ĠPo inter ĠNetworks Ġto Ġpoint Ġto Ġthe Ġstart Ġand Ġend Ġpositions Ġof Ġkey ph r ases Ġin Ġa Ġsource Ġtext . </s> The Ġmain Ġdrawback Ġof Ġkey phrase Ġextraction Ġis Ġthat Ġsometimes Ġkey ph r ases Ġare Ġabsent Ġfrom Ġthe Ġsource Ġtext , Ġthus Ġan Ġextract ive Ġmodel Ġwill Ġfail Ġpredicting Ġthose Ġkey ph r ases . ĠB IB REF 0 Ġfirst Ġproposed Ġthe ĠCopy R NN , Ġa Ġneural Ġgener ative Ġmodel Ġthat Ġboth Ġgenerates Ġwords Ġfrom Ġvocabulary Ġand Ġpoints Ġto Ġwords Ġfrom Ġthe Ġsource Ġtext . ĠRecently , Ġbased Ġon Ġthe ĠCopy R NN Ġarchitecture , ĠB IB REF 1 Ġproposed Ġthe ĠCor r R NN , Ġwhich Ġtakes Ġstates Ġand Ġattention Ġvectors Ġfrom Ġprevious Ġsteps Ġinto Ġaccount Ġin Ġboth Ġenc oder Ġand Ġdec oder Ġto Ġreduce Ġduplication Ġand Ġimprove Ġcoverage . ĠB IB REF 2 Ġproposed Ġsemi - super vised Ġmethods Ġby Ġleveraging Ġboth Ġlabeled Ġand Ġunl abel ed Ġdata Ġfor Ġtraining . ĠB IB REF 3 Ġ, ĠB IB REF 2 Ġproposed Ġto Ġuse Ġstructure Ġinformation Ġ( e . g ., Ġtitle Ġof Ġsource Ġtext ) Ġto Ġimprove Ġkey phrase Ġgeneration Ġperformance . ĠNote Ġthat Ġnone Ġof Ġthe Ġabove Ġworks Ġare Ġable Ġto Ġgenerate Ġvariable Ġnumber Ġof Ġphrases , Ġwhich Ġis Ġone Ġof Ġour Ġcontributions . </s> Sequ ence Ġto ĠSequence ĠGeneration </s> Sequ ence Ġto ĠSequence Ġ( Se q 2 Se q ) Ġlearning Ġwas Ġfirst Ġintroduced Ġby ĠB IB REF 17 Ġ; Ġtogether Ġwith Ġthe Ġsoft Ġattention Ġmechanism Ġof ĠB IB REF 18 Ġ, Ġit Ġhas Ġbeen Ġwidely Ġused Ġin Ġnatural Ġlanguage Ġgeneration Ġtasks . ĠB IB REF 19 Ġ, ĠB IB REF 20 Ġused Ġa Ġmixture Ġof Ġgeneration Ġand Ġpointing Ġto Ġovercome Ġthe Ġproblem Ġof Ġlarge Ġvocabulary Ġsize . ĠB IB REF 21 Ġ, ĠB IB REF 22 Ġapplied ĠSe q 2 Se q Ġmodels Ġon Ġsummary Ġgeneration Ġtasks , Ġwhile ĠB IB REF 23 Ġ, ĠB IB REF 24 Ġgenerated Ġquestions Ġconditioned Ġon Ġdocuments Ġand Ġanswers Ġfrom Ġmachine Ġcomprehension Ġdatasets . ĠSe q 2 Se q Ġwas Ġalso Ġapplied Ġon Ġneural Ġsentence Ġsimpl ification ĠB IB REF 25 Ġand Ġparaph rase Ġgeneration Ġtasks ĠB IB REF 26 Ġ. </s> Given Ġa Ġsource Ġtext Ġconsisting Ġof ĠIN LINE FORM 0 Ġwords ĠIN LINE FORM 1 Ġ, Ġthe Ġenc oder Ġconverts Ġtheir Ġcorresponding Ġembed d ings ĠIN LINE FORM 2 Ġinto Ġa Ġset Ġof ĠIN LINE FORM 3 Ġreal - valued Ġvectors ĠIN LINE FORM 4 Ġwith Ġa Ġbid irection al ĠGR U ĠB IB REF 27 Ġ: ĠDIS PLAY FORM 0 </s> Drop out ĠB IB REF 28 Ġis Ġapplied Ġto Ġboth ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġfor Ġregular ization . </s> The Ġdec oder Ġis Ġa Ġun i - direction al ĠGR U , Ġwhich Ġgenerates Ġa Ġnew Ġstate ĠIN LINE FORM 0 Ġat Ġeach Ġtime - step ĠIN LINE FORM 1 Ġfrom Ġthe Ġword Ġembed ding ĠIN LINE FORM 2 Ġand Ġthe Ġrecurrent Ġstate ĠIN LINE FORM 3 Ġ: ĠDIS PLAY FORM 0 </s> The Ġinitial Ġstate ĠIN LINE FORM 0 Ġis Ġderived Ġfrom Ġthe Ġfinal Ġenc oder Ġstate ĠIN LINE FORM 1 Ġby Ġapplying Ġa Ġsingle - layer Ġfeed - forward Ġneural Ġnet Ġ( F NN ): ĠIN LINE FORM 2 Ġ. ĠDrop out Ġis Ġapplied Ġto Ġboth Ġthe Ġembed d ings ĠIN LINE FORM 3 Ġand Ġthe ĠGR U Ġstates ĠIN LINE FORM 4 Ġ. </s> When Ġgenerating Ġtoken ĠIN LINE FORM 0 Ġ, Ġin Ġorder Ġto Ġbetter Ġincorporate Ġinformation Ġfrom Ġthe Ġsource Ġtext , Ġan Ġattention Ġmechanism ĠB IB REF 18 Ġis Ġemployed Ġto Ġinfer Ġthe Ġimportance ĠIN LINE FORM 1 Ġof Ġeach Ġsource Ġword ĠIN LINE FORM 2 Ġgiven Ġthe Ġcurrent Ġdec oder Ġstate ĠIN LINE FORM 3 Ġ. ĠThis Ġimportance Ġis Ġmeasured Ġby Ġan Ġenergy Ġfunction Ġwith Ġa Ġ2 - layer ĠF NN : ĠDIS PLAY FORM 0 </s> The Ġoutput Ġover Ġall Ġdecoding Ġsteps ĠIN LINE FORM 0 Ġthus Ġdefine Ġa Ġdistribution Ġover Ġthe Ġsource Ġsequence : ĠDIS PLAY FORM 0 </s> These Ġattention Ġscores Ġare Ġthen Ġused Ġas Ġweights Ġfor Ġa Ġrefined Ġrepresentation Ġof Ġthe Ġsource Ġenc od ings , Ġwhich Ġis Ġthen Ġconc aten ated Ġto Ġthe Ġdec oder Ġstate ĠIN LINE FORM 0 Ġto Ġderive Ġa Ġgener ative Ġdistribution ĠIN LINE FORM 1 Ġ: ĠDIS PLAY FORM 0 </s> where Ġthe Ġoutput Ġsize Ġof ĠIN LINE FORM 0 Ġequals Ġto Ġthe Ġtarget Ġvocabulary Ġsize . ĠSub script ĠIN LINE FORM 1 Ġindicates Ġthe Ġabstract ive Ġnature Ġof ĠIN LINE FORM 2 Ġsince Ġit Ġis Ġa Ġdistribution Ġover Ġa Ġprescribed Ġvocabulary . </s> We Ġemploy Ġthe Ġpointer Ġsoft max ĠB IB REF 19 Ġmechanism Ġto Ġswitch Ġbetween Ġgenerating Ġa Ġtoken ĠIN LINE FORM 0 Ġ( from Ġa Ġvocabulary ) Ġand Ġpointing Ġ( to Ġa Ġtoken Ġin Ġthe Ġsource Ġtext ). ĠSpecifically , Ġthe Ġpointer Ġsoft max Ġmodule Ġcomp utes Ġa Ġscal ar Ġswitch ĠIN LINE FORM 1 Ġat Ġeach Ġgeneration Ġtime - step Ġand Ġuses Ġit Ġto Ġinterpol ate Ġthe Ġabstract ive Ġdistribution ĠIN LINE FORM 2 Ġover Ġthe Ġvocabulary Ġ( see ĠEqu ation ĠEQ REF 16 Ġ) Ġand Ġthe Ġextract ive Ġdistribution ĠIN LINE FORM 3 Ġover Ġthe Ġsource Ġtext Ġtokens : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġconditioned Ġon Ġboth Ġthe Ġattention - weight ed Ġsource Ġrepresentation ĠIN LINE FORM 1 Ġand Ġthe Ġdec oder Ġstate ĠIN LINE FORM 2 Ġ: ĠDIS PLAY FORM 0 </s> Model ĠArchitecture </s> Given Ġa Ġpiece Ġof Ġsource Ġtext , Ġour Ġobjective Ġis Ġto Ġgenerate Ġa Ġvariable Ġnumber Ġof Ġmulti - word Ġphrases . ĠTo Ġthis Ġend , Ġwe Ġopt Ġfor Ġthe Ġsequence - to - sequence Ġframework Ġ( Se q 2 Se q ) Ġas Ġthe Ġbasis Ġof Ġour Ġmodel , Ġcombined Ġwith Ġattention Ġand Ġpointer Ġsoft max Ġmechanisms Ġin Ġthe Ġdec oder . </s> Since Ġeach Ġdata Ġexample Ġcontains Ġone Ġsource Ġtext Ġsequence Ġand Ġmultiple Ġtarget Ġphrase Ġsequences Ġ( d ub bed ĠOne 2 Many , Ġand Ġeach Ġsequence Ġcan Ġbe Ġof Ġmulti - word ), Ġtwo Ġparad ig ms Ġcan Ġbe Ġadopted Ġfor Ġtraining ĠSe q 2 Se q Ġmodels . ĠThe Ġfirst Ġone ĠB IB REF 0 Ġis Ġto Ġdivide Ġeach ĠOne 2 Many Ġdata Ġexample Ġinto Ġmultiple ĠOne 2 One Ġexamples , Ġand Ġthe Ġresulting Ġmodels Ġ( e . g . ĠCopy R NN ) Ġcan Ġgenerate Ġone Ġphrase Ġat Ġonce Ġand Ġmust Ġrely Ġon Ġbeam Ġsearch Ġtechnique Ġto Ġproduce Ġmore Ġunique Ġphrases . </s> To Ġenable Ġmodels Ġto Ġgenerate Ġmultiple Ġphrases Ġand Ġcontrol Ġthe Ġnumber Ġto Ġoutput , Ġwe Ġpropose Ġthe Ġsecond Ġtraining Ġparadigm ĠOne 2 Se q , Ġin Ġwhich Ġwe Ġconc aten ate Ġmultiple Ġphrases Ġinto Ġa Ġsingle Ġsequence Ġwith Ġa Ġdelim iter ĠIN LINE FORM 0 ĠSE P ĠIN LINE FORM 1 Ġ, Ġand Ġthis Ġconc aten ated Ġsequence Ġis Ġthen Ġused Ġas Ġthe Ġtarget Ġfor Ġsequence Ġgeneration Ġduring Ġtraining . ĠAn Ġoverview Ġof Ġthe Ġmodel 's Ġstructure Ġis Ġshown Ġin ĠFigure ĠFIG REF 8 Ġ. </s> Not ations </s> In Ġthe Ġfollowing Ġsubsections , Ġwe Ġuse ĠIN LINE FORM 0 Ġto Ġdenote Ġinput Ġtext Ġtokens , ĠIN LINE FORM 1 Ġto Ġdenote Ġtoken Ġembed d ings , ĠIN LINE FORM 2 Ġto Ġdenote Ġhidden Ġstates , Ġand ĠIN LINE FORM 3 Ġto Ġdenote Ġoutput Ġtext Ġtokens . ĠSup ers cript s Ġdenote Ġtime - steps Ġin Ġa Ġsequence , Ġand Ġsubscript s ĠIN LINE FORM 4 Ġand ĠIN LINE FORM 5 Ġindicate Ġwhether Ġa Ġvariable Ġresides Ġin Ġthe Ġenc oder Ġor Ġthe Ġdec oder Ġof Ġthe Ġmodel , Ġrespectively . ĠThe Ġabsence Ġof Ġa Ġsupers cript Ġindicates Ġmultipl icity Ġin Ġthe Ġtime Ġdimension . ĠIN LINE FORM 6 Ġrefers Ġto Ġa Ġlinear Ġtransformation Ġand ĠIN LINE FORM 7 Ġrefers Ġto Ġit Ġfollowed Ġby Ġa Ġnon - linear Ġactivation Ġfunction ĠIN LINE FORM 8 Ġ. ĠAng led Ġbrackets , ĠIN LINE FORM 9 Ġ, Ġdenote Ġconc aten ation . </s> Mech an isms Ġfor ĠD iverse ĠGeneration </s> There Ġare Ġusually Ġmultiple Ġkey ph r ases Ġfor Ġa Ġgiven Ġsource Ġtext Ġbecause Ġeach Ġkey phrase Ġrepresents Ġcertain Ġaspects Ġof Ġthe Ġtext . ĠTherefore Ġkey phrase Ġdiversity Ġis Ġdesired Ġfor Ġthe Ġkey phrase Ġgeneration . ĠMost Ġprevious Ġkey phrase Ġgeneration Ġmodels Ġgenerate Ġmultiple Ġphrases Ġby Ġover - generation , Ġwhich Ġis Ġhighly Ġprone Ġto Ġgenerate Ġsimilar Ġphrases Ġdue Ġto Ġthe Ġnature Ġof Ġbeam Ġsearch . ĠGiven Ġour Ġobjective Ġto Ġgenerate Ġvariable Ġnumbers Ġof Ġkey ph r ases , Ġwe Ġneed Ġto Ġadopt Ġnew Ġstrategies Ġfor Ġachieving Ġbetter Ġdiversity Ġin Ġthe Ġoutput . </s> Rec all Ġthat Ġwe Ġrepresent Ġvariable Ġnumbers Ġof Ġkey ph r ases Ġas Ġdelim iter - separ ated Ġsequences . ĠOne Ġparticular Ġissue Ġwe Ġobserved Ġduring Ġerror Ġanalysis Ġis Ġthat Ġthe Ġmodel Ġtends Ġto Ġproduce Ġidentical Ġtokens Ġfollowing Ġthe Ġdelim iter Ġtoken . ĠFor Ġexample , Ġsuppose Ġa Ġtarget Ġsequence Ġcontains ĠIN LINE FORM 0 Ġdelim iter Ġtokens Ġat Ġtime - steps ĠIN LINE FORM 1 Ġ. ĠDuring Ġtraining , Ġthe Ġmodel Ġis Ġrewarded Ġfor Ġgenerating Ġthe Ġsame Ġdelim iter Ġtoken Ġat Ġthese Ġtime - steps , Ġwhich Ġpresumably Ġintroduces Ġmuch Ġhom ogeneity Ġin Ġthe Ġcorresponding Ġdec oder Ġstates ĠIN LINE FORM 2 Ġ. ĠWhen Ġthese Ġstates Ġare Ġsubsequently Ġused Ġas Ġinputs Ġat Ġthe Ġtime - steps Ġimmediately Ġfollowing Ġthe Ġdelim iter , Ġthe Ġdec oder Ġnaturally Ġproduces Ġhighly Ġsimilar Ġdistributions Ġover Ġthe Ġfollowing Ġtokens , Ġresulting Ġin Ġidentical Ġtokens Ġbeing Ġdec oded . ĠTo Ġalleviate Ġthis Ġproblem , Ġwe Ġpropose Ġtwo Ġplug - in Ġcomponents Ġfor Ġthe Ġsequential Ġgeneration Ġmodel . </s> We Ġpropose Ġa Ġmechanism Ġcalled Ġsemantic Ġcoverage Ġthat Ġfocuses Ġon Ġthe Ġsemantic Ġrepresentations Ġof Ġgenerated Ġphrases . ĠSpecifically , Ġwe Ġintroduce Ġanother Ġun i - direction al Ġrecurrent Ġmodel ĠIN LINE FORM 0 Ġ( d ub bed Ġtarget Ġenc oder ) Ġwhich Ġenc odes Ġdec oder - generated Ġtokens ĠIN LINE FORM 1 Ġ, Ġwhere ĠIN LINE FORM 2 Ġ, Ġinto Ġhidden Ġstates ĠIN LINE FORM 3 Ġ. ĠThis Ġstate Ġis Ġthen Ġtaken Ġas Ġan Ġextra Ġinput Ġto Ġthe Ġdec oder ĠGR U , Ġmodifying ĠEqu ation ĠEQ REF 12 Ġto : ĠDIS PLAY FORM 0 </s> If Ġthe Ġtarget Ġenc oder Ġwere Ġto Ġbe Ġupdated Ġwith Ġthe Ġtraining Ġsignal Ġfrom Ġgeneration Ġ( i . e ., Ġback prop ag ating Ġerror Ġfrom Ġthe Ġdec oder ĠGR U Ġto Ġthe Ġtarget Ġenc oder ), Ġthe Ġresulting Ġdec oder Ġis Ġessentially Ġa Ġ2 - layer ĠGR U Ġwith Ġresidual Ġconnections . ĠInstead , Ġinspired Ġby Ġprevious Ġrepresentation Ġlearning Ġworks ĠB IB REF 29 Ġ, ĠB IB REF 30 Ġ, ĠB IB REF 31 Ġ, Ġwe Ġtrain Ġthe Ġtarget Ġenc oder Ġin Ġan Ġself - super vised Ġfashion Ġ( Figure ĠFIG REF 8 Ġ). ĠThat Ġis , Ġwe Ġextract Ġtarget Ġenc oder 's Ġfinal Ġhidden Ġstate Ġvector ĠIN LINE FORM 0 Ġ, Ġwhere ĠIN LINE FORM 1 Ġis Ġthe Ġlength Ġof Ġtarget Ġsequence , Ġand Ġuse Ġit Ġas Ġa Ġgeneral Ġrepresentation Ġof Ġthe Ġtarget Ġphrases . ĠWe Ġtrain Ġby Ġmaximizing Ġthe Ġmutual Ġinformation Ġbetween Ġthese Ġphrase Ġrepresentations Ġand Ġthe Ġfinal Ġstate Ġof Ġthe Ġsource Ġenc oder ĠIN LINE FORM 2 Ġas Ġfollows . ĠFor Ġeach Ġphrase Ġrepresentation Ġvector ĠIN LINE FORM 3 Ġ, Ġwe Ġtake Ġthe Ġen oc d ings ĠIN LINE FORM 4 Ġof ĠIN LINE FORM 5 Ġdifferent Ġsource Ġtexts , Ġwhere ĠIN LINE FORM 6 Ġis Ġthe Ġenc oder Ġrepresentation Ġfor Ġthe Ġcurrent Ġsource Ġtext , Ġand Ġthe Ġremaining ĠIN LINE FORM 7 Ġare Ġnegative Ġsamples Ġ( sam pled Ġat Ġrandom ) Ġfrom Ġthe Ġtraining Ġdata . ĠThe Ġtarget Ġenc oder Ġis Ġtrained Ġto Ġminimize Ġthe Ġclassification Ġloss : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġbi - linear Ġtransformation . </s> The Ġmotivation Ġhere Ġis Ġto Ġconst rain Ġthe Ġoverall Ġrepresentation Ġof Ġgenerated Ġkey phrase Ġto Ġbe Ġsem antically Ġclose Ġto Ġthe Ġoverall Ġmeaning Ġof Ġthe Ġsource Ġtext . ĠWith Ġsuch Ġrepresentations Ġas Ġinput Ġto Ġthe Ġdec oder , Ġthe Ġsemantic Ġcoverage Ġmechanism Ġcan Ġpotentially Ġhelp Ġto Ġprovide Ġuseful Ġkey phrase Ġinformation Ġand Ġguide Ġgeneration . </s> We Ġalso Ġpropose Ġorth og onal Ġregular ization , Ġwhich Ġexplicitly Ġencourages Ġthe Ġdelim iter - gener ating Ġdec oder Ġstates Ġto Ġbe Ġdifferent Ġfrom Ġeach Ġother . ĠThis Ġis Ġinspired Ġby ĠB IB REF 32 Ġ, Ġwho Ġuse Ġorth og onal Ġregular ization Ġto Ġencourage Ġrepresentations Ġacross Ġdomains Ġto Ġbe Ġas Ġdistinct Ġas Ġpossible . ĠSpecifically , Ġwe Ġstack Ġthe Ġdec oder Ġhidden Ġstates Ġcorresponding Ġto Ġdelim it ers Ġtogether Ġto Ġform Ġmatrix ĠIN LINE FORM 0 Ġand Ġuse Ġthe Ġfollowing Ġequation Ġas Ġthe Ġorth og onal Ġregular ization Ġloss : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġmatrix Ġtrans pose Ġof ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġis Ġthe Ġidentity Ġmatrix Ġof Ġrank ĠIN LINE FORM 3 Ġ, ĠIN LINE FORM 4 Ġindicates Ġelement Ġwise Ġmultiplication , ĠIN LINE FORM 5 Ġindicates ĠIN LINE FORM 6 Ġnorm Ġof Ġeach Ġelement Ġin Ġa Ġmatrix ĠIN LINE FORM 7 Ġ. ĠThis Ġloss Ġfunction Ġprefers Ġorth og on ality Ġamong Ġthe Ġhidden Ġstates ĠIN LINE FORM 8 Ġand Ġthus Ġimproves Ġdiversity Ġin Ġthe Ġtokens Ġfollowing Ġthe Ġdelim it ers . </s> We Ġadopt Ġthe Ġwidely Ġused Ġnegative Ġlog - like lihood Ġloss Ġin Ġour Ġsequence Ġgeneration Ġmodel , Ġden oted Ġas ĠIN LINE FORM 0 Ġ. ĠThe Ġoverall Ġloss Ġwe Ġuse Ġin Ġour Ġmodel Ġis ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġare Ġhyper - param eters . </s> Dec oding ĠStrategies </s> According Ġto Ġdifferent Ġtask Ġrequirements , Ġvarious Ġdecoding Ġmethods Ġcan Ġbe Ġapplied Ġto Ġgenerate Ġthe Ġtarget Ġsequence ĠIN LINE FORM 0 Ġ. ĠPrior Ġstudies ĠB IB REF 0 Ġ, ĠB IB REF 7 Ġfocus Ġmore Ġon Ġgenerating Ġexcessive Ġnumber Ġof Ġphrases Ġby Ġleveraging Ġbeam Ġsearch Ġto Ġprolifer ate Ġthe Ġoutput Ġphrases . ĠIn Ġcontrast , Ġmodels Ġtrained Ġunder ĠOne 2 Se q Ġparadigm Ġare Ġcapable Ġof Ġdetermining Ġthe Ġproper Ġnumber Ġof Ġphrases Ġto Ġoutput . ĠIn Ġlight Ġof Ġprevious Ġresearch Ġin Ġpsychology ĠB IB REF 33 Ġ, ĠB IB REF 34 Ġ, Ġwe Ġname Ġthese Ġtwo Ġdecoding / search Ġstrategies Ġas ĠEx haust ive ĠDec oding Ġand ĠSelf - termin ating ĠDec oding , Ġrespectively , Ġdue Ġto Ġtheir Ġresemblance Ġto Ġthe Ġway Ġhumans Ġbehave Ġin Ġserial Ġmemory Ġtasks . ĠSimply Ġspeaking , Ġthe Ġmajor Ġdifference Ġlies Ġin Ġwhether Ġa Ġmodel Ġis Ġcapable Ġof Ġcontrolling Ġthe Ġnumber Ġof Ġphrases Ġto Ġoutput . ĠWe Ġdescribe Ġthe Ġdetailed Ġdecoding Ġstrategies Ġused Ġin Ġthis Ġstudy Ġas Ġfollows : </s> As Ġtraditional Ġkey phrase Ġtasks Ġevaluate Ġmodels Ġwith Ġa Ġfixed Ġnumber Ġof Ġtop - ranked Ġpredictions Ġ( say ĠF - score Ġ@ 5 Ġand Ġ@ 10 ), Ġexisting Ġkey phrase Ġgeneration Ġstudies Ġhave Ġto Ġover - gener ate Ġphrases Ġby Ġmeans Ġof Ġbeam Ġsearch Ġ( common ly Ġwith Ġa Ġlarge Ġbeam Ġsize , Ġe . g ., Ġ150 Ġand Ġ200 Ġin ĠB IB REF 3 Ġ, ĠB IB REF 0 Ġ, Ġrespectively ), Ġa Ġhe uristic Ġsearch Ġalgorithm Ġthat Ġreturns ĠIN LINE FORM 0 Ġapproximate Ġoptimal Ġsequences . ĠFor Ġthe ĠOne 2 One Ġsetting , Ġeach Ġreturned Ġsequence Ġis Ġa Ġunique Ġphrase Ġitself . ĠBut Ġfor ĠOne 2 Se q , Ġeach Ġproduced Ġsequence Ġcontains Ġseveral Ġphrases Ġand Ġadditional Ġprocesses ĠB IB REF 2 Ġare Ġneeded Ġto Ġobtain Ġthe Ġfinal Ġunique Ġ( ordered ) Ġphrase Ġlist . </s> It Ġis Ġworth Ġnoting Ġthat Ġthe Ġtime Ġcomplexity Ġof Ġbeam Ġsearch Ġis ĠIN LINE FORM 0 Ġ, Ġwhere ĠIN LINE FORM 1 Ġis Ġthe Ġbeam Ġwidth , Ġand ĠIN LINE FORM 2 Ġis Ġthe Ġmaximum Ġlength Ġof Ġgenerated Ġsequences . ĠTherefore Ġthe Ġexhaustive Ġdecoding Ġis Ġgenerally Ġvery Ġcomput ationally Ġexpensive , Ġespecially Ġfor ĠOne 2 Se q Ġsetting Ġwhere ĠIN LINE FORM 3 Ġis Ġmuch Ġlarger Ġthan Ġin ĠOne 2 One . ĠIt Ġis Ġalso Ġwasteful Ġas Ġwe Ġobserve Ġthat Ġless Ġthan Ġ5 % Ġof Ġphrases Ġgenerated Ġby ĠOne 2 Se q Ġmodels Ġare Ġunique . </s> An Ġinnate Ġcharacteristic Ġof Ġkey phrase Ġtasks Ġis Ġthat Ġthe Ġnumber Ġof Ġkey ph r ases Ġvaries Ġdepending Ġon Ġthe Ġdocument Ġand Ġdataset Ġgenre , Ġtherefore Ġdynamically Ġoutput ting Ġa Ġvariable Ġnumber Ġof Ġphrases Ġis Ġa Ġdesirable Ġproperty Ġfor Ġkey phrase Ġgeneration Ġmodels . ĠSince Ġour Ġproposed Ġmodel Ġis Ġtrained Ġto Ġgenerate Ġa Ġvariable Ġnumber Ġof Ġphrases Ġas Ġa Ġsingle Ġsequence Ġjoined Ġby Ġdelim it ers , Ġwe Ġcan Ġobtain Ġmultiple Ġphrases Ġby Ġsimply Ġdecoding Ġa Ġsingle Ġsequence Ġfor Ġeach Ġgiven Ġsource Ġtext . ĠThe Ġresulting Ġmodel Ġthus Ġimplicitly Ġperforms Ġthe Ġadditional Ġtask Ġof Ġdynamically Ġestimating Ġthe Ġproper Ġsize Ġof Ġthe Ġtarget Ġphrase Ġset : Ġonce Ġthe Ġmodel Ġbelieves Ġthat Ġan Ġadequate Ġnumber Ġof Ġphrases Ġhave Ġbeen Ġgenerated , Ġit Ġoutputs Ġa Ġspecial Ġtoken ĠIN LINE FORM 0 ĠE OS ĠIN LINE FORM 1 Ġto Ġterminate Ġthe Ġdecoding Ġprocess . </s> One Ġnotable Ġattribute Ġof Ġthe Ġself - termin ating Ġdecoding Ġstrategy Ġis Ġthat , Ġby Ġgenerating Ġa Ġset Ġof Ġphrases Ġin Ġa Ġsingle Ġsequence , Ġthe Ġmodel Ġconditions Ġits Ġcurrent Ġgeneration Ġon Ġall Ġpreviously Ġgenerated Ġphrases . ĠCompared Ġto Ġthe Ġexhaustive Ġstrategy Ġ( i . e ., Ġphrases Ġbeing Ġgenerated Ġindependently Ġby Ġbeam Ġsearch Ġin Ġparallel ), Ġour Ġmodel Ġcan Ġmodel Ġthe Ġdependency Ġamong Ġits Ġoutput Ġin Ġa Ġmore Ġexplicit Ġfashion . ĠAdditionally , Ġsince Ġmultiple Ġphrases Ġare Ġdec oded Ġas Ġa Ġsingle Ġsequence , Ġdecoding Ġcan Ġbe Ġperformed Ġmore Ġefficiently Ġthan Ġexhaustive Ġdecoding Ġby Ġconducting Ġgreedy Ġsearch Ġor Ġbeam Ġsearch Ġon Ġonly Ġthe Ġtop - sc ored Ġsequence . </s> E val uating ĠKey phrase ĠGeneration </s> Form ally , Ġgiven Ġa Ġsource Ġtext , Ġsuppose Ġthat Ġa Ġmodel Ġpredicts Ġa Ġlist Ġof Ġunique Ġkey ph r ases ĠIN LINE FORM 0 Ġordered Ġby Ġthe Ġquality Ġof Ġthe Ġpredictions ĠIN LINE FORM 1 Ġ, Ġand Ġthat Ġthe Ġground Ġtruth Ġkey ph r ases Ġfor Ġthe Ġgiven Ġsource Ġtext Ġis Ġthe Ġor acle Ġset ĠIN LINE FORM 2 Ġ. ĠWhen Ġonly Ġthe Ġtop ĠIN LINE FORM 3 Ġpredictions ĠIN LINE FORM 4 Ġare Ġused Ġfor Ġevaluation , Ġprecision , Ġrecall , Ġand ĠF ĠIN LINE FORM 5 Ġscore Ġare Ġconsequently Ġconditioned Ġon ĠIN LINE FORM 6 Ġand Ġdefined Ġas : ĠDIS PLAY FORM 0 </s> As Ġdiscussed Ġin ĠSection ĠSEC REF 1 Ġ, Ġthe Ġnumber Ġof Ġgenerated Ġkey ph r ases Ġused Ġfor Ġevaluation Ġcan Ġhave Ġa Ġcritical Ġimpact Ġon Ġthe Ġquality Ġof Ġthe Ġresulting Ġevaluation Ġmetrics . ĠHere Ġwe Ġcompare Ġthree Ġchoices Ġof ĠIN LINE FORM 0 Ġand Ġthe Ġimplications Ġon Ġkey phrase Ġevaluation Ġfor Ġeach Ġchoice : </s> A Ġsimple Ġremedy Ġis Ġto Ġset ĠIN LINE FORM 0 Ġas Ġa Ġvariable Ġnumber Ġwhich Ġis Ġspecific Ġto Ġeach Ġdata Ġexample . ĠHere Ġwe Ġdefine Ġtwo Ġnew Ġmetrics : </s> By Ġsimply Ġextending Ġthe Ġconstant Ġnumber ĠIN LINE FORM 0 Ġto Ġdifferent Ġvariables Ġaccordingly , Ġboth ĠF ĠIN LINE FORM 1 Ġ@ ĠIN LINE FORM 2 Ġand ĠF ĠIN LINE FORM 3 Ġ@ ĠIN LINE FORM 4 Ġare Ġcapable Ġof Ġreflecting Ġthe Ġnature Ġof Ġvariable Ġnumber Ġof Ġphrases Ġfor Ġeach Ġdocument , Ġand Ġa Ġmodel Ġcan Ġachieve Ġthe Ġmaximum ĠIN LINE FORM 5 Ġscore Ġof ĠIN LINE FORM 6 Ġif Ġand Ġonly Ġif Ġit Ġpredicts Ġthe Ġexact Ġsame Ġphrases Ġas Ġthe Ġground Ġtruth . ĠAnother Ġmerit Ġof ĠF ĠIN LINE FORM 7 Ġ@ ĠIN LINE FORM 8 Ġis Ġthat Ġit Ġis Ġindependent Ġfrom Ġmodel Ġoutputs , Ġtherefore Ġwe Ġcan Ġuse Ġit Ġto Ġcompare Ġexisting Ġmodels . </s> Dat as ets Ġand ĠExper iments </s> In Ġthis Ġsection , Ġwe Ġreport Ġour Ġexperiment Ġresults Ġon Ġmultiple Ġdatasets Ġand Ġcompare Ġwith Ġexisting Ġmodels . ĠWe Ġuse ĠIN LINE FORM 0 Ġto Ġrefer Ġto Ġthe Ġdelim iter - con cat en ated Ġsequence - to - sequ ences Ġmodel Ġdescribed Ġin ĠSection ĠSEC REF 3 Ġ; ĠIN LINE FORM 1 Ġrefers Ġto Ġthe Ġmodel Ġaugmented Ġwith Ġorth og onal Ġregular ization Ġand Ġsemantic Ġcoverage Ġmechanism . </s> To Ġconstruct Ġtarget Ġsequences Ġfor Ġtraining ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġ, Ġground Ġtruth Ġkey ph r ases Ġare Ġsorted Ġby Ġtheir Ġorder Ġof Ġfirst Ġoccurrence Ġin Ġthe Ġsource Ġtext . ĠKey ph r ases Ġthat Ġdo Ġnot Ġappear Ġin Ġthe Ġsource Ġtext Ġare Ġapp ended Ġto Ġthe Ġend . ĠThis Ġorder Ġmay Ġguide Ġthe Ġattention Ġmechanism Ġto Ġattend Ġto Ġsource Ġpositions Ġin Ġa Ġsmoother Ġway . ĠImplementation Ġdetails Ġcan Ġbe Ġfound Ġin ĠAppendix ĠSEC REF 9 Ġ. </s> We Ġinclude Ġfour Ġnon - ne ural Ġextract ive Ġmodels Ġand ĠCopy R NN ĠB IB REF 0 Ġas Ġbas elines . ĠWe Ġuse ĠCopy R NN Ġto Ġdenote Ġthe Ġmodel Ġreported Ġby ĠB IB REF 0 Ġ, ĠCopy R NN * Ġto Ġdenote Ġour Ġimplementation Ġof ĠCopy R NN Ġbased Ġon Ġtheir Ġopen Ġsourced Ġcode . ĠTo Ġdraw Ġfair Ġcomparison Ġwith Ġexisting Ġstudy , Ġwe Ġuse Ġthe Ġsame Ġmodel Ġhyper param eter Ġsetting Ġas Ġused Ġin ĠB IB REF 0 Ġand Ġuse Ġexhaustive Ġdecoding Ġstrategy Ġfor Ġmost Ġexperiments . ĠK EA ĠB IB REF 4 Ġand ĠMau i ĠB IB REF 8 Ġare Ġtrained Ġon Ġa Ġsubset Ġof Ġ50 , 000 Ġdocuments Ġfrom Ġeither ĠKP 20 k Ġ( Table ĠT AB REF 35 Ġ) Ġor ĠStack Ex Ġ( Table ĠT AB REF 37 Ġ) Ġinstead Ġof Ġall Ġdocuments Ġdue Ġto Ġimplementation Ġlimits Ġ( without Ġfine - tun ing Ġon Ġtarget Ġdataset ). </s> In ĠSection ĠSEC REF 42 Ġ, Ġwe Ġapply Ġthe Ġself - termin ating Ġdecoding Ġstrategy . ĠSince Ġno Ġexisting Ġmodel Ġsupports Ġsuch Ġdecoding Ġstrategy , Ġwe Ġonly Ġreport Ġresults Ġfrom Ġour Ġproposed Ġmodels . ĠThey Ġcan Ġbe Ġused Ġfor Ġcomparison Ġin Ġfuture Ġstudies . </s> Exper iments Ġon ĠScientific ĠPublications </s> Our Ġfirst Ġdataset Ġconsists Ġof Ġa Ġcollection Ġof Ġscientific Ġpublication Ġdatasets , Ġnamely ĠKP 20 k , ĠIn spec , ĠK rap iv in , ĠN US , Ġand ĠSem E val , Ġthat Ġhave Ġbeen Ġwidely Ġused Ġin Ġexisting Ġliterature ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ. ĠKP 20 k , Ġfor Ġexample , Ġwas Ġintroduced Ġby ĠB IB REF 0 Ġand Ġcomprises Ġmore Ġthan Ġhalf Ġa Ġmillion Ġscientific Ġpublications . ĠFor Ġeach Ġarticle , Ġthe Ġabstract Ġand Ġtitle Ġare Ġused Ġas Ġthe Ġsource Ġtext Ġwhile Ġthe Ġauthor Ġkeywords Ġare Ġused Ġas Ġtarget . ĠThe Ġother Ġfour Ġdatasets Ġcontain Ġmuch Ġfewer Ġarticles , Ġand Ġthus Ġused Ġto Ġtest Ġtransfer ability Ġof Ġour Ġmodel Ġ( without Ġfine - tun ing ). </s> We Ġreport Ġour Ġmodel 's Ġperformance Ġon Ġthe Ġpresent - key phrase Ġportion Ġof Ġthe ĠKP 20 k Ġdataset Ġin ĠTable ĠT AB REF 35 Ġ. ĠTo Ġcompare Ġwith Ġprevious Ġworks , Ġwe Ġprovide Ġcompute ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġscores . ĠThe Ġnew Ġproposed ĠF ĠIN LINE FORM 2 Ġ@ ĠIN LINE FORM 3 Ġmetric Ġindicates Ġconsistent Ġranking Ġwith ĠIN LINE FORM 4 Ġfor Ġmost Ġcases . ĠDue Ġto Ġits Ġtarget Ġnumber Ġsensitivity , Ġwe Ġfind Ġthat Ġits Ġvalue Ġis Ġcloser Ġto ĠIN LINE FORM 5 Ġfor ĠKP 20 k Ġand ĠK rap iv in Ġwhere Ġaverage Ġtarget Ġkey ph r ases Ġis Ġless Ġand Ġcloser Ġto ĠIN LINE FORM 6 Ġfor Ġthe Ġother Ġthree Ġdatasets . </s> From Ġthe Ġresult Ġwe Ġcan Ġsee Ġthat Ġthe Ġneural - based Ġmodels Ġoutper form Ġnon - ne ural Ġmodels Ġby Ġlarge Ġmargins . ĠOur Ġimplemented ĠCopy R NN Ġachieves Ġbetter Ġor Ġcomparable Ġperformance Ġagainst Ġthe Ġoriginal Ġmodel , Ġand Ġon ĠN US Ġand ĠSem E val Ġthe Ġadvantage Ġis Ġmore Ġsalient . </s> As Ġfor Ġthe Ġproposed Ġmodels , Ġboth ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġyield Ġcomparable Ġresults Ġto ĠCopy R NN , Ġindicating Ġthat ĠOne 2 Se q Ġparadigm Ġcan Ġwork Ġwell Ġas Ġan Ġalternative Ġoption Ġfor Ġthe Ġkey phrase Ġgeneration . ĠIN LINE FORM 2 Ġoutper forms ĠIN LINE FORM 3 Ġon Ġall Ġmetrics , Ġsuggesting Ġthe Ġsemantic Ġcoverage Ġand Ġorth og onal Ġregular ization Ġhelp Ġthe Ġmodel Ġto Ġgenerate Ġhigher Ġquality Ġkey ph r ases Ġand Ġachieve Ġbetter Ġgeneral iz ability . ĠTo Ġour Ġsurprise , Ġon Ġthe Ġmetric ĠF ĠIN LINE FORM 4 Ġ@ 10 Ġfor ĠKP 20 k Ġand ĠK rap iv in Ġ( average Ġnumber Ġof Ġkey ph r ases Ġis Ġonly Ġ5 ), Ġwhere Ġhigh - rec all Ġmodels Ġlike ĠCopy R NN Ġare Ġmore Ġfavored , ĠIN LINE FORM 5 Ġis Ġstill Ġable Ġto Ġoutper form ĠOne 2 One Ġbas elines , Ġindicating Ġthat Ġthe Ġproposed Ġmechanisms Ġfor Ġdiverse Ġgeneration Ġare Ġeffective . </s> Exper iments Ġon ĠThe ĠStack Ex ĠDat as et </s> Insp ired Ġby Ġthe ĠStack L ite Ġtag Ġrecommendation Ġtask Ġon ĠK agg le , Ġwe Ġbuild Ġa Ġnew Ġbenchmark Ġbased Ġon Ġthe Ġpublic ĠStack Ex change Ġdata . ĠWe Ġuse Ġquestions Ġwith Ġtitles Ġas Ġsource , Ġand Ġuser - ass igned Ġtags Ġas Ġtarget Ġkey ph r ases . </s> Since Ġoft entimes Ġthe Ġquestions Ġon ĠStack Ex change Ġcontain Ġless Ġinformation Ġthan Ġin Ġscientific Ġpublications , Ġthere Ġare Ġfewer Ġkey ph r ases Ġper Ġdata Ġpoint Ġin ĠStack Ex . ĠFurthermore , ĠStack Ex change Ġuses Ġa Ġtag Ġrecommendation Ġsystem Ġthat Ġsuggests Ġtopic - relevant Ġtags Ġto Ġusers Ġwhile Ġsubmitting Ġquestions ; Ġtherefore , Ġwe Ġare Ġmore Ġlikely Ġto Ġsee Ġgeneral Ġterminology Ġsuch Ġas ĠLinux Ġand ĠJava . ĠThis Ġcharacteristic Ġchallenges Ġmodels Ġwith Ġrespect Ġto Ġtheir Ġability Ġto Ġdist ill Ġmajor Ġtopics Ġof Ġa Ġquestion Ġrather Ġthan Ġselecting Ġspecific Ġsnippets Ġfrom Ġthe Ġtext . </s> We Ġreport Ġour Ġmodels ' Ġperformance Ġon ĠStack Ex Ġin ĠTable ĠT AB REF 37 Ġ. ĠResults Ġshow ĠIN LINE FORM 0 Ġperforms Ġthe Ġbest ; Ġon Ġthe Ġabsent - key phrase Ġgeneration Ġtasks , Ġit Ġoutper forms ĠIN LINE FORM 1 Ġby Ġa Ġlarge Ġmargin . </s> Gener ating ĠVariable ĠNumber ĠKey ph r ases </s> One Ġkey Ġadvantage Ġof Ġour Ġproposed Ġmodel Ġis Ġthe Ġcapability Ġof Ġpredicting Ġthe Ġnumber Ġof Ġkey ph r ases Ġconditioned Ġon Ġthe Ġgiven Ġsource Ġtext . ĠWe Ġthus Ġconduct Ġa Ġset Ġof Ġexperiments Ġon ĠKP 20 k Ġand ĠStack Ex Ġpresent Ġkey phrase Ġgeneration Ġtasks , Ġas Ġshown Ġin ĠTable ĠT AB REF 39 Ġ, Ġto Ġstudy Ġsuch Ġbehavior . ĠWe Ġadopt Ġthe Ġself - termin ating Ġdecoding Ġstrategy Ġ( Section ĠSEC REF 28 Ġ), Ġand Ġuse Ġboth ĠF ĠIN LINE FORM 0 Ġ@ ĠIN LINE FORM 1 Ġand ĠF ĠIN LINE FORM 2 Ġ@ ĠIN LINE FORM 3 Ġ( Section ĠSEC REF 4 Ġ) Ġto Ġevaluate . </s> In Ġthese Ġexperiments , Ġwe Ġuse Ġbeam Ġsearch Ġas Ġin Ġmost ĠNatural ĠLanguage ĠGeneration Ġ( NL G ) Ġtasks , Ġi . e ., Ġonly Ġuse Ġthe Ġtop Ġranked Ġprediction Ġsequence Ġas Ġoutput . ĠWe Ġcompare Ġthe Ġresults Ġwith Ġgreedy Ġsearch . ĠSince Ġno Ġexisting Ġmodel Ġis Ġcapable Ġof Ġgenerating Ġvariable Ġnumber Ġof Ġkey ph r ases , Ġin Ġthis Ġsubsection Ġwe Ġonly Ġreport Ġperformance Ġon Ġsuch Ġsetting Ġfrom ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġ. </s> From ĠTable ĠT AB REF 39 Ġwe Ġobserve Ġthat Ġin Ġthe Ġvariable Ġnumber Ġgeneration Ġsetting , Ġgreedy Ġsearch Ġoutper forms Ġbeam Ġsearch Ġconsistently . ĠThis Ġmay Ġbecause Ġbeam Ġsearch Ġtends Ġto Ġgenerate Ġshort Ġand Ġsimilar Ġsequences . ĠWe Ġcan Ġalso Ġsee Ġthe Ġresulting ĠF ĠIN LINE FORM 0 Ġ@ ĠIN LINE FORM 1 Ġscores Ġare Ġgenerally Ġlower Ġthan Ġresults Ġreported Ġin Ġprevious Ġsubsections , Ġthis Ġsuggests Ġan Ġover - generation Ġdecoding Ġstrategy Ġmay Ġstill Ġbenefit Ġfrom Ġachieving Ġhigher Ġrecall . </s> A bl ation ĠStudy </s> We Ġconduct Ġan Ġab lation Ġexperiment Ġto Ġstudy Ġthe Ġeffects Ġof Ġorth og onal Ġregular ization Ġand Ġsemantic Ġcoverage Ġmechanism Ġon ĠIN LINE FORM 0 Ġ. ĠAs Ġshown Ġin ĠTable ĠT AB REF 44 Ġ, Ġsemantic Ġcoverage Ġprovides Ġsignificant Ġboost Ġto ĠIN LINE FORM 1 Ġ' s Ġperformance Ġon Ġall Ġdatasets . ĠOrth og onal Ġregular ization Ġhurts Ġperformance Ġwhen Ġis Ġsolely Ġapplied Ġto ĠIN LINE FORM 2 Ġmodel . ĠInterestingly , Ġwhen Ġboth Ġcomponents Ġare Ġenabled Ġ( ĠIN LINE FORM 3 Ġ), Ġthe Ġmodel Ġoutper forms ĠIN LINE FORM 4 Ġby Ġa Ġnoticeable Ġmargin Ġon Ġall Ġdatasets , Ġthis Ġsuggests Ġthe Ġtwo Ġcomponents Ġhelp Ġkey phrase Ġgeneration Ġin Ġa Ġsy ner getic Ġway . ĠOne Ġfuture Ġdirection Ġis Ġto Ġapply Ġorth og onal Ġregular ization Ġdirectly Ġon Ġtarget Ġenc oder , Ġsince Ġthe Ġregular izer Ġcan Ġpotentially Ġdivers ify Ġtarget Ġrepresentations Ġat Ġphrase Ġlevel , Ġwhich Ġmay Ġfurther Ġencourage Ġdiverse Ġkey phrase Ġgeneration Ġin Ġdec oder . </s> Visual izing ĠD ivers ified ĠGeneration </s> To Ġverify Ġour Ġassumption Ġthat Ġtarget Ġencoding Ġand Ġorth og onal Ġregular ization Ġhelp Ġto Ġboost Ġthe Ġdiversity Ġof Ġgenerated Ġsequences , Ġwe Ġuse Ġtwo Ġmetrics , Ġone Ġquantitative Ġand Ġone Ġqualitative , Ġto Ġmeasure Ġdiversity Ġof Ġgeneration . </s> First , Ġwe Ġsimply Ġcalculate Ġthe Ġaverage Ġunique Ġpredictions Ġproduced Ġby Ġboth ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġin Ġexperiments Ġshown Ġin ĠSection ĠSEC REF 36 Ġ. ĠThe Ġresulting Ġnumbers Ġare Ġ20 . 38 Ġand Ġ89 . 70 Ġfor ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġrespectively . ĠSecond , Ġfrom Ġthe Ġmodel Ġrunning Ġon Ġthe ĠKP 20 k Ġvalidation Ġset , Ġwe Ġrandomly Ġsample Ġ2000 Ġdec oder Ġhidden Ġstates Ġat ĠIN LINE FORM 4 Ġsteps Ġfollowing Ġa Ġdelim iter Ġ( ĠIN LINE FORM 5 Ġ) Ġand Ġapply Ġan Ġun super vised Ġclust ering Ġmethod Ġ( t - S NE ĠB IB REF 35 Ġ) Ġon Ġthem . ĠFrom Ġthe ĠFigure ĠFIG REF 46 Ġwe Ġcan Ġsee Ġthat Ġhidden Ġstates Ġsampled Ġfrom ĠIN LINE FORM 6 Ġare Ġeasier Ġto Ġcluster Ġwhile Ġhidden Ġstates Ġsampled Ġfrom ĠIN LINE FORM 7 Ġyield Ġone Ġmass Ġof Ġvectors Ġwith Ġno Ġobvious Ġdistinct Ġclusters . ĠResults Ġon Ġboth Ġmetrics Ġsuggest Ġtarget Ġencoding Ġand Ġorth og onal Ġregular ization Ġindeed Ġhelp Ġdivers ifying Ġgeneration Ġof Ġour Ġmodel . </s> Qual itative ĠAnalysis </s> To Ġillustrate Ġthe Ġdifference Ġof Ġpredictions Ġbetween Ġour Ġproposed Ġmodels , Ġwe Ġshow Ġan Ġexample Ġchosen Ġfrom Ġthe ĠKP 20 k Ġvalidation Ġset Ġin ĠAppendix ĠSEC REF 10 Ġ. ĠIn Ġthis Ġexample Ġthere Ġare Ġ29 Ġground Ġtruth Ġphrases . ĠNeither Ġof Ġthe Ġmodels Ġis Ġable Ġto Ġgenerate Ġall Ġof Ġthe Ġkey ph r ases , Ġbut Ġit Ġis Ġobvious Ġthat Ġthe Ġpredictions Ġfrom ĠIN LINE FORM 0 Ġall Ġstart Ġwith ĠâĢ ľ test âĢ Ŀ , Ġwhile Ġpredictions Ġfrom ĠIN LINE FORM 1 Ġare Ġdiverse . ĠThis Ġto Ġsome Ġextent Ġver ifies Ġour Ġassumption Ġthat Ġwithout Ġthe Ġtarget Ġenc oder Ġand Ġorth og onal Ġregular ization , Ġdec oder Ġstates Ġfollowing Ġdelim it ers Ġare Ġless Ġdiverse . </s> Conclusion Ġand ĠFuture ĠWork </s> We Ġpropose Ġa Ġrecurrent Ġgener ative Ġmodel Ġthat Ġsequ entially Ġgenerates Ġmultiple Ġkey ph r ases , Ġwith Ġtwo Ġextra Ġmodules Ġthat Ġenhance Ġgeneration Ġdiversity . ĠWe Ġpropose Ġnew Ġmetrics Ġto Ġevaluate Ġkey phrase Ġgeneration . ĠOur Ġmodel Ġshows Ġcompetitive Ġperformance Ġon Ġa Ġset Ġof Ġkey phrase Ġgeneration Ġdatasets , Ġincluding Ġone Ġintroduced Ġin Ġthis Ġwork . ĠIn Ġfuture Ġwork , Ġwe Ġplan Ġto Ġinvestigate Ġhow Ġtarget Ġphrase Ġorder Ġaffects Ġthe Ġgeneration Ġbehavior , Ġand Ġfurther Ġexplore Ġset Ġgeneration Ġin Ġan Ġorder Ġinvari ant Ġfashion . </s> Exper iment ĠResults Ġon ĠKP 20 k ĠAbsent ĠSub set </s> Gener ating Ġabsent Ġkey ph r ases Ġon Ġscientific Ġpublication Ġdatasets Ġis Ġa Ġrather Ġchallenging Ġproblem . ĠEx isting Ġstudies Ġoften Ġachieve Ġseemingly Ġgood Ġperformance Ġby Ġmeasuring Ġrecall Ġon Ġtens Ġand Ġsometimes Ġhundreds Ġof Ġkey ph r ases Ġproduced Ġby Ġexhaustive Ġdecoding Ġwith Ġa Ġlarge Ġbeam Ġsize ĠâĢĶ Ġthus Ġcompletely Ġignoring Ġprecision . </s> We Ġreport Ġthe Ġmodels ' ĠR @ 10 / 50 Ġscores Ġon Ġthe Ġabsent Ġportion Ġof Ġfive Ġscientific Ġpaper Ġdatasets Ġin ĠTable ĠT AB REF 48 Ġto Ġbe Ġin Ġline Ġwith Ġprevious Ġstudies . </s> The Ġabsent Ġkey phrase Ġprediction Ġhighly Ġprefers Ġrecall - oriented Ġmodels , Ġtherefore ĠCopy R NN Ġwith Ġbeam Ġsize Ġof Ġ200 Ġis Ġinn ately Ġproper Ġfor Ġthis Ġtask Ġsetting . ĠHow erer , Ġfrom Ġthe Ġresults Ġwe Ġobserve Ġthat Ġwith Ġthe Ġhelp Ġof Ġexhaustive Ġdecoding Ġand Ġdiverse Ġmechanisms , ĠIN LINE FORM 0 Ġis Ġable Ġto Ġperform Ġcompar ably Ġto ĠCopy R NN Ġmodel , Ġand Ġit Ġgenerally Ġworks Ġbetter Ġfor Ġtop Ġpredictions . ĠEven Ġthough Ġthe Ġtrend Ġof Ġmodels ' Ġperformance Ġsomewhat Ġmatches Ġwhat Ġwe Ġobserve Ġon Ġthe Ġpresent Ġdata , Ġwe Ġargue Ġthat Ġit Ġis Ġhard Ġto Ġcompare Ġdifferent Ġmodels ' Ġperformance Ġon Ġsuch Ġscale . ĠWe Ġargue Ġthat ĠStack Ex Ġis Ġbetter Ġtest bed s Ġfor Ġabsent Ġkey phrase Ġgeneration . </s> Im plementation ĠDetails </s> Im ple m nt ation Ġdetails Ġof Ġour Ġproposed Ġmodels Ġare Ġas Ġfollows . ĠIn Ġall Ġexperiments , Ġthe Ġword Ġembed d ings Ġare Ġinitialized Ġwith Ġ100 - dimensional Ġrandom Ġmat rices . ĠThe Ġnumber Ġof Ġhidden Ġunits Ġin Ġboth Ġthe Ġenc oder Ġand Ġdec oder ĠGR U Ġare Ġ150 . ĠThe Ġnumber Ġof Ġhidden Ġunits Ġin Ġtarget Ġenc oder ĠGR U Ġis Ġ150 . ĠThe Ġsize Ġof Ġvocabulary Ġis Ġ50 , 000 . </s> The Ġnumbers Ġof Ġhidden Ġunits Ġin ĠML Ps Ġdescribed Ġin ĠSection ĠSEC REF 3 Ġare Ġas Ġfollows . ĠDuring Ġnegative Ġsampling , Ġwe Ġrandomly Ġsample Ġ16 Ġsamples Ġfrom Ġthe Ġsame Ġbatch , Ġthus Ġtarget Ġencoding Ġloss Ġin ĠEqu ation ĠEQ REF 23 Ġis Ġa Ġ17 - way Ġclassification Ġloss . ĠIn ĠIN LINE FORM 0 Ġ, Ġwe Ġset Ġboth Ġthe ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġin ĠEqu ation ĠEQ REF 27 Ġto Ġbe Ġ0 . 3 . ĠIn Ġall Ġexperiments , Ġwe Ġuse Ġa Ġdrop out Ġrate Ġof Ġ0 . 1 . </s> We Ġuse ĠAdam ĠB IB REF 36 Ġas Ġthe Ġstep Ġrule Ġfor Ġoptimization . ĠThe Ġlearning Ġrate Ġis ĠIN LINE FORM 0 Ġ. ĠThe Ġmodel Ġis Ġimplemented Ġusing ĠPy Tor ch ĠB IB REF 38 Ġand ĠOpen N MT ĠB IB REF 37 Ġ. </s> For Ġexhaustive Ġdecoding , Ġwe Ġuse Ġa Ġbeam Ġsize Ġof Ġ50 Ġand Ġa Ġmaximum Ġsequence Ġlength Ġof Ġ40 . </s> Following ĠB IB REF 0 Ġ, Ġlower case Ġand Ġstemming Ġare Ġperformed Ġon Ġboth Ġthe Ġground Ġtruth Ġand Ġgenerated Ġkey ph r ases Ġduring Ġevaluation . </s> We Ġleave Ġout Ġ2 , 000 Ġdata Ġexamples Ġas Ġvalidation Ġset Ġfor Ġboth ĠKP 20 k Ġand ĠStack Ex Ġand Ġuse Ġthem Ġto Ġidentify Ġoptimal Ġcheckpoints Ġfor Ġtesting . ĠAnd Ġall Ġthe Ġscores Ġreported Ġin Ġthis Ġpaper Ġare Ġfrom Ġcheckpoints Ġwith Ġbest Ġperformances Ġ( F ĠIN LINE FORM 0 Ġ@ ĠIN LINE FORM 1 Ġ) Ġon Ġvalidation Ġset . </s> Example ĠOutput </s> See ĠTable ĠT AB REF 49 Ġ.
Introduction Keyphrase generation is the task of automatically predicting keyphrases given a source text. Desired keyphrases are often multi-word units that summarize the high-level meaning and highlight certain important topics or information of the source text. Consequently, models that can successfully perform this task should be capable of not only distilling high-level information from a document, but also locating specific, important snippets therein. To make the problem even more challenging, a keyphrase may or may not be a substring of the source text (i.e., it may be present or absent). Moreover, a given source text is usually associated with a set of multiple keyphrases. Thus, keyphrase generation is an instance of the set generation problem, where both the size of the set and the size (i.e., the number of tokens in a phrase) of each element can vary depending on the source. Similar to summarization, keyphrase generation is often formulated as a sequence-to-sequence (Seq2Seq) generation task in most prior studies BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 . Conditioned on a source text, Seq2Seq models generate phrases individually or as a longer sequence jointed by delimiting tokens. Since standard Seq2Seq models generate only one sequence at a time, thus to generate multiple phrases, a common approach is to over-generate using beam search with a large beam width. Models are then evaluated by taking a fixed number of top predicted phrases (typically 5 or 10) and comparing them against the ground truth keyphrases. Though this approach has achieved good empirical results, we argue that it suffers from two major limitations. Firstly, models that use beam search to generate multiple keyphrases generally lack the ability to determine the dynamic number of keyphrases needed for different source texts. Meanwhile, the parallelism in beam search also fails to model the inter-relation among the generated phrases, which can often result in diminished diversity in the output. Although certain existing models take output diversity into consideration during training BIBREF1 , BIBREF2 , the effort is significantly undermined during decoding due to the reliance on over-generation and phrase ranking with beam search. Secondly, the current evaluation setup is rather problematic, since existing studies attempt to match a fixed number of outputs against a variable number of ground truth keyphrases. Empirically, the number of keyphrases can vary drastically for different source texts, depending on a plethora of factors including the length or genre of the text, the granularity of keyphrase annotation, etc. For the several commonly used keyphrase generation datasets, for example, the average number of keyphrases per data point can range from 5.3 to 15.7, with variances sometimes as large as 64.6 (Table TABREF1 ). Therefore, using an arbitrary, fixed number INLINEFORM0 to evaluate entire datasets is not appropriate. In fact, under this evaluation setup, the F1 score for the oracle model on the KP20k dataset is 0.858 for INLINEFORM1 and 0.626 for INLINEFORM2 , which apparently poses serious normalization issues as evaluation metrics. To overcome these problems, we propose novel decoding strategies and evaluation metrics for the keyphrase generation task. The main contributions of this work are as follows: Keyphrase Extraction and Generation Traditional keyphrase extraction has been studied extensively in past decades. In most existing literature, keyphrase extraction has been formulated as a two-step process. First, lexical features such as part-of-speech tags are used to determine a list of phrase candidates by heuristic methods BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 . Second, a ranking algorithm is adopted to rank the candidate list and the top ranked candidates are selected as keyphrases. A wide variety of methods were applied for ranking, such as bagged decision trees BIBREF8 , BIBREF9 , Multi-Layer Perceptron, Support Vector Machine BIBREF9 and PageRank BIBREF10 , BIBREF11 , BIBREF12 . Recently, BIBREF13 , BIBREF14 , BIBREF15 used sequence labeling models to extract keyphrases from text. Similarly, BIBREF16 used Pointer Networks to point to the start and end positions of keyphrases in a source text. The main drawback of keyphrase extraction is that sometimes keyphrases are absent from the source text, thus an extractive model will fail predicting those keyphrases. BIBREF0 first proposed the CopyRNN, a neural generative model that both generates words from vocabulary and points to words from the source text. Recently, based on the CopyRNN architecture, BIBREF1 proposed the CorrRNN, which takes states and attention vectors from previous steps into account in both encoder and decoder to reduce duplication and improve coverage. BIBREF2 proposed semi-supervised methods by leveraging both labeled and unlabeled data for training. BIBREF3 , BIBREF2 proposed to use structure information (e.g., title of source text) to improve keyphrase generation performance. Note that none of the above works are able to generate variable number of phrases, which is one of our contributions. Sequence to Sequence Generation Sequence to Sequence (Seq2Seq) learning was first introduced by BIBREF17 ; together with the soft attention mechanism of BIBREF18 , it has been widely used in natural language generation tasks. BIBREF19 , BIBREF20 used a mixture of generation and pointing to overcome the problem of large vocabulary size. BIBREF21 , BIBREF22 applied Seq2Seq models on summary generation tasks, while BIBREF23 , BIBREF24 generated questions conditioned on documents and answers from machine comprehension datasets. Seq2Seq was also applied on neural sentence simplification BIBREF25 and paraphrase generation tasks BIBREF26 . Given a source text consisting of INLINEFORM0 words INLINEFORM1 , the encoder converts their corresponding embeddings INLINEFORM2 into a set of INLINEFORM3 real-valued vectors INLINEFORM4 with a bidirectional GRU BIBREF27 : DISPLAYFORM0 Dropout BIBREF28 is applied to both INLINEFORM0 and INLINEFORM1 for regularization. The decoder is a uni-directional GRU, which generates a new state INLINEFORM0 at each time-step INLINEFORM1 from the word embedding INLINEFORM2 and the recurrent state INLINEFORM3 : DISPLAYFORM0 The initial state INLINEFORM0 is derived from the final encoder state INLINEFORM1 by applying a single-layer feed-forward neural net (FNN): INLINEFORM2 . Dropout is applied to both the embeddings INLINEFORM3 and the GRU states INLINEFORM4 . When generating token INLINEFORM0 , in order to better incorporate information from the source text, an attention mechanism BIBREF18 is employed to infer the importance INLINEFORM1 of each source word INLINEFORM2 given the current decoder state INLINEFORM3 . This importance is measured by an energy function with a 2-layer FNN: DISPLAYFORM0 The output over all decoding steps INLINEFORM0 thus define a distribution over the source sequence: DISPLAYFORM0 These attention scores are then used as weights for a refined representation of the source encodings, which is then concatenated to the decoder state INLINEFORM0 to derive a generative distribution INLINEFORM1 : DISPLAYFORM0 where the output size of INLINEFORM0 equals to the target vocabulary size. Subscript INLINEFORM1 indicates the abstractive nature of INLINEFORM2 since it is a distribution over a prescribed vocabulary. We employ the pointer softmax BIBREF19 mechanism to switch between generating a token INLINEFORM0 (from a vocabulary) and pointing (to a token in the source text). Specifically, the pointer softmax module computes a scalar switch INLINEFORM1 at each generation time-step and uses it to interpolate the abstractive distribution INLINEFORM2 over the vocabulary (see Equation EQREF16 ) and the extractive distribution INLINEFORM3 over the source text tokens: DISPLAYFORM0 where INLINEFORM0 is conditioned on both the attention-weighted source representation INLINEFORM1 and the decoder state INLINEFORM2 : DISPLAYFORM0 Model Architecture Given a piece of source text, our objective is to generate a variable number of multi-word phrases. To this end, we opt for the sequence-to-sequence framework (Seq2Seq) as the basis of our model, combined with attention and pointer softmax mechanisms in the decoder. Since each data example contains one source text sequence and multiple target phrase sequences (dubbed One2Many, and each sequence can be of multi-word), two paradigms can be adopted for training Seq2Seq models. The first one BIBREF0 is to divide each One2Many data example into multiple One2One examples, and the resulting models (e.g. CopyRNN) can generate one phrase at once and must rely on beam search technique to produce more unique phrases. To enable models to generate multiple phrases and control the number to output, we propose the second training paradigm One2Seq, in which we concatenate multiple phrases into a single sequence with a delimiter INLINEFORM0 SEP INLINEFORM1 , and this concatenated sequence is then used as the target for sequence generation during training. An overview of the model's structure is shown in Figure FIGREF8 . Notations In the following subsections, we use INLINEFORM0 to denote input text tokens, INLINEFORM1 to denote token embeddings, INLINEFORM2 to denote hidden states, and INLINEFORM3 to denote output text tokens. Superscripts denote time-steps in a sequence, and subscripts INLINEFORM4 and INLINEFORM5 indicate whether a variable resides in the encoder or the decoder of the model, respectively. The absence of a superscript indicates multiplicity in the time dimension. INLINEFORM6 refers to a linear transformation and INLINEFORM7 refers to it followed by a non-linear activation function INLINEFORM8 . Angled brackets, INLINEFORM9 , denote concatenation. Mechanisms for Diverse Generation There are usually multiple keyphrases for a given source text because each keyphrase represents certain aspects of the text. Therefore keyphrase diversity is desired for the keyphrase generation. Most previous keyphrase generation models generate multiple phrases by over-generation, which is highly prone to generate similar phrases due to the nature of beam search. Given our objective to generate variable numbers of keyphrases, we need to adopt new strategies for achieving better diversity in the output. Recall that we represent variable numbers of keyphrases as delimiter-separated sequences. One particular issue we observed during error analysis is that the model tends to produce identical tokens following the delimiter token. For example, suppose a target sequence contains INLINEFORM0 delimiter tokens at time-steps INLINEFORM1 . During training, the model is rewarded for generating the same delimiter token at these time-steps, which presumably introduces much homogeneity in the corresponding decoder states INLINEFORM2 . When these states are subsequently used as inputs at the time-steps immediately following the delimiter, the decoder naturally produces highly similar distributions over the following tokens, resulting in identical tokens being decoded. To alleviate this problem, we propose two plug-in components for the sequential generation model. We propose a mechanism called semantic coverage that focuses on the semantic representations of generated phrases. Specifically, we introduce another uni-directional recurrent model INLINEFORM0 (dubbed target encoder) which encodes decoder-generated tokens INLINEFORM1 , where INLINEFORM2 , into hidden states INLINEFORM3 . This state is then taken as an extra input to the decoder GRU, modifying Equation EQREF12 to: DISPLAYFORM0 If the target encoder were to be updated with the training signal from generation (i.e., backpropagating error from the decoder GRU to the target encoder), the resulting decoder is essentially a 2-layer GRU with residual connections. Instead, inspired by previous representation learning works BIBREF29 , BIBREF30 , BIBREF31 , we train the target encoder in an self-supervised fashion (Figure FIGREF8 ). That is, we extract target encoder's final hidden state vector INLINEFORM0 , where INLINEFORM1 is the length of target sequence, and use it as a general representation of the target phrases. We train by maximizing the mutual information between these phrase representations and the final state of the source encoder INLINEFORM2 as follows. For each phrase representation vector INLINEFORM3 , we take the enocdings INLINEFORM4 of INLINEFORM5 different source texts, where INLINEFORM6 is the encoder representation for the current source text, and the remaining INLINEFORM7 are negative samples (sampled at random) from the training data. The target encoder is trained to minimize the classification loss: DISPLAYFORM0 where INLINEFORM0 is bi-linear transformation. The motivation here is to constrain the overall representation of generated keyphrase to be semantically close to the overall meaning of the source text. With such representations as input to the decoder, the semantic coverage mechanism can potentially help to provide useful keyphrase information and guide generation. We also propose orthogonal regularization, which explicitly encourages the delimiter-generating decoder states to be different from each other. This is inspired by BIBREF32 , who use orthogonal regularization to encourage representations across domains to be as distinct as possible. Specifically, we stack the decoder hidden states corresponding to delimiters together to form matrix INLINEFORM0 and use the following equation as the orthogonal regularization loss: DISPLAYFORM0 where INLINEFORM0 is the matrix transpose of INLINEFORM1 , INLINEFORM2 is the identity matrix of rank INLINEFORM3 , INLINEFORM4 indicates element wise multiplication, INLINEFORM5 indicates INLINEFORM6 norm of each element in a matrix INLINEFORM7 . This loss function prefers orthogonality among the hidden states INLINEFORM8 and thus improves diversity in the tokens following the delimiters. We adopt the widely used negative log-likelihood loss in our sequence generation model, denoted as INLINEFORM0 . The overall loss we use in our model is DISPLAYFORM0 where INLINEFORM0 and INLINEFORM1 are hyper-parameters. Decoding Strategies According to different task requirements, various decoding methods can be applied to generate the target sequence INLINEFORM0 . Prior studies BIBREF0 , BIBREF7 focus more on generating excessive number of phrases by leveraging beam search to proliferate the output phrases. In contrast, models trained under One2Seq paradigm are capable of determining the proper number of phrases to output. In light of previous research in psychology BIBREF33 , BIBREF34 , we name these two decoding/search strategies as Exhaustive Decoding and Self-terminating Decoding, respectively, due to their resemblance to the way humans behave in serial memory tasks. Simply speaking, the major difference lies in whether a model is capable of controlling the number of phrases to output. We describe the detailed decoding strategies used in this study as follows: As traditional keyphrase tasks evaluate models with a fixed number of top-ranked predictions (say F-score @5 and @10), existing keyphrase generation studies have to over-generate phrases by means of beam search (commonly with a large beam size, e.g., 150 and 200 in BIBREF3 , BIBREF0 , respectively), a heuristic search algorithm that returns INLINEFORM0 approximate optimal sequences. For the One2One setting, each returned sequence is a unique phrase itself. But for One2Seq, each produced sequence contains several phrases and additional processes BIBREF2 are needed to obtain the final unique (ordered) phrase list. It is worth noting that the time complexity of beam search is INLINEFORM0 , where INLINEFORM1 is the beam width, and INLINEFORM2 is the maximum length of generated sequences. Therefore the exhaustive decoding is generally very computationally expensive, especially for One2Seq setting where INLINEFORM3 is much larger than in One2One. It is also wasteful as we observe that less than 5% of phrases generated by One2Seq models are unique. An innate characteristic of keyphrase tasks is that the number of keyphrases varies depending on the document and dataset genre, therefore dynamically outputting a variable number of phrases is a desirable property for keyphrase generation models. Since our proposed model is trained to generate a variable number of phrases as a single sequence joined by delimiters, we can obtain multiple phrases by simply decoding a single sequence for each given source text. The resulting model thus implicitly performs the additional task of dynamically estimating the proper size of the target phrase set: once the model believes that an adequate number of phrases have been generated, it outputs a special token INLINEFORM0 EOS INLINEFORM1 to terminate the decoding process. One notable attribute of the self-terminating decoding strategy is that, by generating a set of phrases in a single sequence, the model conditions its current generation on all previously generated phrases. Compared to the exhaustive strategy (i.e., phrases being generated independently by beam search in parallel), our model can model the dependency among its output in a more explicit fashion. Additionally, since multiple phrases are decoded as a single sequence, decoding can be performed more efficiently than exhaustive decoding by conducting greedy search or beam search on only the top-scored sequence. Evaluating Keyphrase Generation Formally, given a source text, suppose that a model predicts a list of unique keyphrases INLINEFORM0 ordered by the quality of the predictions INLINEFORM1 , and that the ground truth keyphrases for the given source text is the oracle set INLINEFORM2 . When only the top INLINEFORM3 predictions INLINEFORM4 are used for evaluation, precision, recall, and F INLINEFORM5 score are consequently conditioned on INLINEFORM6 and defined as: DISPLAYFORM0 As discussed in Section SECREF1 , the number of generated keyphrases used for evaluation can have a critical impact on the quality of the resulting evaluation metrics. Here we compare three choices of INLINEFORM0 and the implications on keyphrase evaluation for each choice: A simple remedy is to set INLINEFORM0 as a variable number which is specific to each data example. Here we define two new metrics: By simply extending the constant number INLINEFORM0 to different variables accordingly, both F INLINEFORM1 @ INLINEFORM2 and F INLINEFORM3 @ INLINEFORM4 are capable of reflecting the nature of variable number of phrases for each document, and a model can achieve the maximum INLINEFORM5 score of INLINEFORM6 if and only if it predicts the exact same phrases as the ground truth. Another merit of F INLINEFORM7 @ INLINEFORM8 is that it is independent from model outputs, therefore we can use it to compare existing models. Datasets and Experiments In this section, we report our experiment results on multiple datasets and compare with existing models. We use INLINEFORM0 to refer to the delimiter-concatenated sequence-to-sequences model described in Section SECREF3 ; INLINEFORM1 refers to the model augmented with orthogonal regularization and semantic coverage mechanism. To construct target sequences for training INLINEFORM0 and INLINEFORM1 , ground truth keyphrases are sorted by their order of first occurrence in the source text. Keyphrases that do not appear in the source text are appended to the end. This order may guide the attention mechanism to attend to source positions in a smoother way. Implementation details can be found in Appendix SECREF9 . We include four non-neural extractive models and CopyRNN BIBREF0 as baselines. We use CopyRNN to denote the model reported by BIBREF0 , CopyRNN* to denote our implementation of CopyRNN based on their open sourced code. To draw fair comparison with existing study, we use the same model hyperparameter setting as used in BIBREF0 and use exhaustive decoding strategy for most experiments. KEA BIBREF4 and Maui BIBREF8 are trained on a subset of 50,000 documents from either KP20k (Table TABREF35 ) or StackEx (Table TABREF37 ) instead of all documents due to implementation limits (without fine-tuning on target dataset). In Section SECREF42 , we apply the self-terminating decoding strategy. Since no existing model supports such decoding strategy, we only report results from our proposed models. They can be used for comparison in future studies. Experiments on Scientific Publications Our first dataset consists of a collection of scientific publication datasets, namely KP20k, Inspec, Krapivin, NUS, and SemEval, that have been widely used in existing literature BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 . KP20k, for example, was introduced by BIBREF0 and comprises more than half a million scientific publications. For each article, the abstract and title are used as the source text while the author keywords are used as target. The other four datasets contain much fewer articles, and thus used to test transferability of our model (without fine-tuning). We report our model's performance on the present-keyphrase portion of the KP20k dataset in Table TABREF35 . To compare with previous works, we provide compute INLINEFORM0 and INLINEFORM1 scores. The new proposed F INLINEFORM2 @ INLINEFORM3 metric indicates consistent ranking with INLINEFORM4 for most cases. Due to its target number sensitivity, we find that its value is closer to INLINEFORM5 for KP20k and Krapivin where average target keyphrases is less and closer to INLINEFORM6 for the other three datasets. From the result we can see that the neural-based models outperform non-neural models by large margins. Our implemented CopyRNN achieves better or comparable performance against the original model, and on NUS and SemEval the advantage is more salient. As for the proposed models, both INLINEFORM0 and INLINEFORM1 yield comparable results to CopyRNN, indicating that One2Seq paradigm can work well as an alternative option for the keyphrase generation. INLINEFORM2 outperforms INLINEFORM3 on all metrics, suggesting the semantic coverage and orthogonal regularization help the model to generate higher quality keyphrases and achieve better generalizability. To our surprise, on the metric F INLINEFORM4 @10 for KP20k and Krapivin (average number of keyphrases is only 5), where high-recall models like CopyRNN are more favored, INLINEFORM5 is still able to outperform One2One baselines, indicating that the proposed mechanisms for diverse generation are effective. Experiments on The StackEx Dataset Inspired by the StackLite tag recommendation task on Kaggle, we build a new benchmark based on the public StackExchange data. We use questions with titles as source, and user-assigned tags as target keyphrases. Since oftentimes the questions on StackExchange contain less information than in scientific publications, there are fewer keyphrases per data point in StackEx. Furthermore, StackExchange uses a tag recommendation system that suggests topic-relevant tags to users while submitting questions; therefore, we are more likely to see general terminology such as Linux and Java. This characteristic challenges models with respect to their ability to distill major topics of a question rather than selecting specific snippets from the text. We report our models' performance on StackEx in Table TABREF37 . Results show INLINEFORM0 performs the best; on the absent-keyphrase generation tasks, it outperforms INLINEFORM1 by a large margin. Generating Variable Number Keyphrases One key advantage of our proposed model is the capability of predicting the number of keyphrases conditioned on the given source text. We thus conduct a set of experiments on KP20k and StackEx present keyphrase generation tasks, as shown in Table TABREF39 , to study such behavior. We adopt the self-terminating decoding strategy (Section SECREF28 ), and use both F INLINEFORM0 @ INLINEFORM1 and F INLINEFORM2 @ INLINEFORM3 (Section SECREF4 ) to evaluate. In these experiments, we use beam search as in most Natural Language Generation (NLG) tasks, i.e., only use the top ranked prediction sequence as output. We compare the results with greedy search. Since no existing model is capable of generating variable number of keyphrases, in this subsection we only report performance on such setting from INLINEFORM0 and INLINEFORM1 . From Table TABREF39 we observe that in the variable number generation setting, greedy search outperforms beam search consistently. This may because beam search tends to generate short and similar sequences. We can also see the resulting F INLINEFORM0 @ INLINEFORM1 scores are generally lower than results reported in previous subsections, this suggests an over-generation decoding strategy may still benefit from achieving higher recall. Ablation Study We conduct an ablation experiment to study the effects of orthogonal regularization and semantic coverage mechanism on INLINEFORM0 . As shown in Table TABREF44 , semantic coverage provides significant boost to INLINEFORM1 's performance on all datasets. Orthogonal regularization hurts performance when is solely applied to INLINEFORM2 model. Interestingly, when both components are enabled ( INLINEFORM3 ), the model outperforms INLINEFORM4 by a noticeable margin on all datasets, this suggests the two components help keyphrase generation in a synergetic way. One future direction is to apply orthogonal regularization directly on target encoder, since the regularizer can potentially diversify target representations at phrase level, which may further encourage diverse keyphrase generation in decoder. Visualizing Diversified Generation To verify our assumption that target encoding and orthogonal regularization help to boost the diversity of generated sequences, we use two metrics, one quantitative and one qualitative, to measure diversity of generation. First, we simply calculate the average unique predictions produced by both INLINEFORM0 and INLINEFORM1 in experiments shown in Section SECREF36 . The resulting numbers are 20.38 and 89.70 for INLINEFORM2 and INLINEFORM3 respectively. Second, from the model running on the KP20k validation set, we randomly sample 2000 decoder hidden states at INLINEFORM4 steps following a delimiter ( INLINEFORM5 ) and apply an unsupervised clustering method (t-SNE BIBREF35 ) on them. From the Figure FIGREF46 we can see that hidden states sampled from INLINEFORM6 are easier to cluster while hidden states sampled from INLINEFORM7 yield one mass of vectors with no obvious distinct clusters. Results on both metrics suggest target encoding and orthogonal regularization indeed help diversifying generation of our model. Qualitative Analysis To illustrate the difference of predictions between our proposed models, we show an example chosen from the KP20k validation set in Appendix SECREF10 . In this example there are 29 ground truth phrases. Neither of the models is able to generate all of the keyphrases, but it is obvious that the predictions from INLINEFORM0 all start with “test”, while predictions from INLINEFORM1 are diverse. This to some extent verifies our assumption that without the target encoder and orthogonal regularization, decoder states following delimiters are less diverse. Conclusion and Future Work We propose a recurrent generative model that sequentially generates multiple keyphrases, with two extra modules that enhance generation diversity. We propose new metrics to evaluate keyphrase generation. Our model shows competitive performance on a set of keyphrase generation datasets, including one introduced in this work. In future work, we plan to investigate how target phrase order affects the generation behavior, and further explore set generation in an order invariant fashion. Experiment Results on KP20k Absent Subset Generating absent keyphrases on scientific publication datasets is a rather challenging problem. Existing studies often achieve seemingly good performance by measuring recall on tens and sometimes hundreds of keyphrases produced by exhaustive decoding with a large beam size — thus completely ignoring precision. We report the models' R@10/50 scores on the absent portion of five scientific paper datasets in Table TABREF48 to be in line with previous studies. The absent keyphrase prediction highly prefers recall-oriented models, therefore CopyRNN with beam size of 200 is innately proper for this task setting. Howerer, from the results we observe that with the help of exhaustive decoding and diverse mechanisms, INLINEFORM0 is able to perform comparably to CopyRNN model, and it generally works better for top predictions. Even though the trend of models' performance somewhat matches what we observe on the present data, we argue that it is hard to compare different models' performance on such scale. We argue that StackEx is better testbeds for absent keyphrase generation. Implementation Details Implemntation details of our proposed models are as follows. In all experiments, the word embeddings are initialized with 100-dimensional random matrices. The number of hidden units in both the encoder and decoder GRU are 150. The number of hidden units in target encoder GRU is 150. The size of vocabulary is 50,000. The numbers of hidden units in MLPs described in Section SECREF3 are as follows. During negative sampling, we randomly sample 16 samples from the same batch, thus target encoding loss in Equation EQREF23 is a 17-way classification loss. In INLINEFORM0 , we set both the INLINEFORM1 and INLINEFORM2 in Equation EQREF27 to be 0.3. In all experiments, we use a dropout rate of 0.1. We use Adam BIBREF36 as the step rule for optimization. The learning rate is INLINEFORM0 . The model is implemented using PyTorch BIBREF38 and OpenNMT BIBREF37 . For exhaustive decoding, we use a beam size of 50 and a maximum sequence length of 40. Following BIBREF0 , lowercase and stemming are performed on both the ground truth and generated keyphrases during evaluation. We leave out 2,000 data examples as validation set for both KP20k and StackEx and use them to identify optimal checkpoints for testing. And all the scores reported in this paper are from checkpoints with best performances (F INLINEFORM0 @ INLINEFORM1 ) on validation set. Example Output See Table TABREF49 .
How is keyphrase diversity measured?
1e4dbfc556cf237accb8b370de2f164fa723687b
How Ġis Ġkey phrase Ġdiversity Ġmeasured ?
average Ġunique Ġpredictions , Ġillustrate Ġthe Ġdifference Ġof Ġpredictions Ġbetween Ġour Ġproposed Ġmodels , Ġwe Ġshow Ġan Ġexample Ġchosen Ġfrom Ġthe ĠKP 20 k Ġvalidation Ġset
<s> How Ġis Ġkey phrase Ġdiversity Ġmeasured ? </s> Introduction </s> Key phrase Ġgeneration Ġis Ġthe Ġtask Ġof Ġautomatically Ġpredicting Ġkey ph r ases Ġgiven Ġa Ġsource Ġtext . ĠDes ired Ġkey ph r ases Ġare Ġoften Ġmulti - word Ġunits Ġthat Ġsummarize Ġthe Ġhigh - level Ġmeaning Ġand Ġhighlight Ġcertain Ġimportant Ġtopics Ġor Ġinformation Ġof Ġthe Ġsource Ġtext . ĠConsequently , Ġmodels Ġthat Ġcan Ġsuccessfully Ġperform Ġthis Ġtask Ġshould Ġbe Ġcapable Ġof Ġnot Ġonly Ġdist illing Ġhigh - level Ġinformation Ġfrom Ġa Ġdocument , Ġbut Ġalso Ġlocating Ġspecific , Ġimportant Ġsnippets Ġtherein . </s> To Ġmake Ġthe Ġproblem Ġeven Ġmore Ġchallenging , Ġa Ġkey phrase Ġmay Ġor Ġmay Ġnot Ġbe Ġa Ġsubst ring Ġof Ġthe Ġsource Ġtext Ġ( i . e ., Ġit Ġmay Ġbe Ġpresent Ġor Ġabsent ). ĠMoreover , Ġa Ġgiven Ġsource Ġtext Ġis Ġusually Ġassociated Ġwith Ġa Ġset Ġof Ġmultiple Ġkey ph r ases . ĠThus , Ġkey phrase Ġgeneration Ġis Ġan Ġinstance Ġof Ġthe Ġset Ġgeneration Ġproblem , Ġwhere Ġboth Ġthe Ġsize Ġof Ġthe Ġset Ġand Ġthe Ġsize Ġ( i . e ., Ġthe Ġnumber Ġof Ġtokens Ġin Ġa Ġphrase ) Ġof Ġeach Ġelement Ġcan Ġvary Ġdepending Ġon Ġthe Ġsource . </s> Similar Ġto Ġsummar ization , Ġkey phrase Ġgeneration Ġis Ġoften Ġformulated Ġas Ġa Ġsequence - to - sequence Ġ( Se q 2 Se q ) Ġgeneration Ġtask Ġin Ġmost Ġprior Ġstudies ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ. ĠCondition ed Ġon Ġa Ġsource Ġtext , ĠSe q 2 Se q Ġmodels Ġgenerate Ġphrases Ġindividually Ġor Ġas Ġa Ġlonger Ġsequence Ġjoint ed Ġby Ġdelim iting Ġtokens . ĠSince Ġstandard ĠSe q 2 Se q Ġmodels Ġgenerate Ġonly Ġone Ġsequence Ġat Ġa Ġtime , Ġthus Ġto Ġgenerate Ġmultiple Ġphrases , Ġa Ġcommon Ġapproach Ġis Ġto Ġover - gener ate Ġusing Ġbeam Ġsearch Ġwith Ġa Ġlarge Ġbeam Ġwidth . ĠModels Ġare Ġthen Ġevaluated Ġby Ġtaking Ġa Ġfixed Ġnumber Ġof Ġtop Ġpredicted Ġphrases Ġ( typically Ġ5 Ġor Ġ10 ) Ġand Ġcomparing Ġthem Ġagainst Ġthe Ġground Ġtruth Ġkey ph r ases . </s> Though Ġthis Ġapproach Ġhas Ġachieved Ġgood Ġempirical Ġresults , Ġwe Ġargue Ġthat Ġit Ġsuffers Ġfrom Ġtwo Ġmajor Ġlimitations . ĠFirstly , Ġmodels Ġthat Ġuse Ġbeam Ġsearch Ġto Ġgenerate Ġmultiple Ġkey ph r ases Ġgenerally Ġlack Ġthe Ġability Ġto Ġdetermine Ġthe Ġdynamic Ġnumber Ġof Ġkey ph r ases Ġneeded Ġfor Ġdifferent Ġsource Ġtexts . ĠMeanwhile , Ġthe Ġparallel ism Ġin Ġbeam Ġsearch Ġalso Ġfails Ġto Ġmodel Ġthe Ġinter - relation Ġamong Ġthe Ġgenerated Ġphrases , Ġwhich Ġcan Ġoften Ġresult Ġin Ġdiminished Ġdiversity Ġin Ġthe Ġoutput . ĠAlthough Ġcertain Ġexisting Ġmodels Ġtake Ġoutput Ġdiversity Ġinto Ġconsideration Ġduring Ġtraining ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, Ġthe Ġeffort Ġis Ġsignificantly Ġundermined Ġduring Ġdecoding Ġdue Ġto Ġthe Ġreliance Ġon Ġover - generation Ġand Ġphrase Ġranking Ġwith Ġbeam Ġsearch . </s> Secondly , Ġthe Ġcurrent Ġevaluation Ġsetup Ġis Ġrather Ġproblematic , Ġsince Ġexisting Ġstudies Ġattempt Ġto Ġmatch Ġa Ġfixed Ġnumber Ġof Ġoutputs Ġagainst Ġa Ġvariable Ġnumber Ġof Ġground Ġtruth Ġkey ph r ases . ĠEm pir ically , Ġthe Ġnumber Ġof Ġkey ph r ases Ġcan Ġvary Ġdrastically Ġfor Ġdifferent Ġsource Ġtexts , Ġdepending Ġon Ġa Ġplethora Ġof Ġfactors Ġincluding Ġthe Ġlength Ġor Ġgenre Ġof Ġthe Ġtext , Ġthe Ġgran ularity Ġof Ġkey phrase Ġannotation , Ġetc . ĠFor Ġthe Ġseveral Ġcommonly Ġused Ġkey phrase Ġgeneration Ġdatasets , Ġfor Ġexample , Ġthe Ġaverage Ġnumber Ġof Ġkey ph r ases Ġper Ġdata Ġpoint Ġcan Ġrange Ġfrom Ġ5 . 3 Ġto Ġ15 . 7 , Ġwith Ġvar iances Ġsometimes Ġas Ġlarge Ġas Ġ64 . 6 Ġ( Table ĠT AB REF 1 Ġ). ĠTherefore , Ġusing Ġan Ġarbitrary , Ġfixed Ġnumber ĠIN LINE FORM 0 Ġto Ġevaluate Ġentire Ġdatasets Ġis Ġnot Ġappropriate . ĠIn Ġfact , Ġunder Ġthis Ġevaluation Ġsetup , Ġthe ĠF 1 Ġscore Ġfor Ġthe Ġor acle Ġmodel Ġon Ġthe ĠKP 20 k Ġdataset Ġis Ġ0 . 8 58 Ġfor ĠIN LINE FORM 1 Ġand Ġ0 . 626 Ġfor ĠIN LINE FORM 2 Ġ, Ġwhich Ġapparently Ġposes Ġserious Ġnormal ization Ġissues Ġas Ġevaluation Ġmetrics . </s> To Ġovercome Ġthese Ġproblems , Ġwe Ġpropose Ġnovel Ġdecoding Ġstrategies Ġand Ġevaluation Ġmetrics Ġfor Ġthe Ġkey phrase Ġgeneration Ġtask . ĠThe Ġmain Ġcontributions Ġof Ġthis Ġwork Ġare Ġas Ġfollows : </s> Key phrase ĠExt raction Ġand ĠGeneration </s> Traditional Ġkey phrase Ġextraction Ġhas Ġbeen Ġstudied Ġextensively Ġin Ġpast Ġdecades . ĠIn Ġmost Ġexisting Ġliterature , Ġkey phrase Ġextraction Ġhas Ġbeen Ġformulated Ġas Ġa Ġtwo - step Ġprocess . ĠFirst , Ġlex ical Ġfeatures Ġsuch Ġas Ġpart - of - speech Ġtags Ġare Ġused Ġto Ġdetermine Ġa Ġlist Ġof Ġphrase Ġcandidates Ġby Ġhe uristic Ġmethods ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ. ĠSecond , Ġa Ġranking Ġalgorithm Ġis Ġadopted Ġto Ġrank Ġthe Ġcandidate Ġlist Ġand Ġthe Ġtop Ġranked Ġcandidates Ġare Ġselected Ġas Ġkey ph r ases . ĠA Ġwide Ġvariety Ġof Ġmethods Ġwere Ġapplied Ġfor Ġranking , Ġsuch Ġas Ġbag ged Ġdecision Ġtrees ĠB IB REF 8 Ġ, ĠB IB REF 9 Ġ, ĠMulti - Layer ĠPer cept ron , ĠSupport ĠVector ĠMachine ĠB IB REF 9 Ġand ĠPage Rank ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ. ĠRecently , ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġused Ġsequence Ġlabeling Ġmodels Ġto Ġextract Ġkey ph r ases Ġfrom Ġtext . ĠSimilarly , ĠB IB REF 16 Ġused ĠPo inter ĠNetworks Ġto Ġpoint Ġto Ġthe Ġstart Ġand Ġend Ġpositions Ġof Ġkey ph r ases Ġin Ġa Ġsource Ġtext . </s> The Ġmain Ġdrawback Ġof Ġkey phrase Ġextraction Ġis Ġthat Ġsometimes Ġkey ph r ases Ġare Ġabsent Ġfrom Ġthe Ġsource Ġtext , Ġthus Ġan Ġextract ive Ġmodel Ġwill Ġfail Ġpredicting Ġthose Ġkey ph r ases . ĠB IB REF 0 Ġfirst Ġproposed Ġthe ĠCopy R NN , Ġa Ġneural Ġgener ative Ġmodel Ġthat Ġboth Ġgenerates Ġwords Ġfrom Ġvocabulary Ġand Ġpoints Ġto Ġwords Ġfrom Ġthe Ġsource Ġtext . ĠRecently , Ġbased Ġon Ġthe ĠCopy R NN Ġarchitecture , ĠB IB REF 1 Ġproposed Ġthe ĠCor r R NN , Ġwhich Ġtakes Ġstates Ġand Ġattention Ġvectors Ġfrom Ġprevious Ġsteps Ġinto Ġaccount Ġin Ġboth Ġenc oder Ġand Ġdec oder Ġto Ġreduce Ġduplication Ġand Ġimprove Ġcoverage . ĠB IB REF 2 Ġproposed Ġsemi - super vised Ġmethods Ġby Ġleveraging Ġboth Ġlabeled Ġand Ġunl abel ed Ġdata Ġfor Ġtraining . ĠB IB REF 3 Ġ, ĠB IB REF 2 Ġproposed Ġto Ġuse Ġstructure Ġinformation Ġ( e . g ., Ġtitle Ġof Ġsource Ġtext ) Ġto Ġimprove Ġkey phrase Ġgeneration Ġperformance . ĠNote Ġthat Ġnone Ġof Ġthe Ġabove Ġworks Ġare Ġable Ġto Ġgenerate Ġvariable Ġnumber Ġof Ġphrases , Ġwhich Ġis Ġone Ġof Ġour Ġcontributions . </s> Sequ ence Ġto ĠSequence ĠGeneration </s> Sequ ence Ġto ĠSequence Ġ( Se q 2 Se q ) Ġlearning Ġwas Ġfirst Ġintroduced Ġby ĠB IB REF 17 Ġ; Ġtogether Ġwith Ġthe Ġsoft Ġattention Ġmechanism Ġof ĠB IB REF 18 Ġ, Ġit Ġhas Ġbeen Ġwidely Ġused Ġin Ġnatural Ġlanguage Ġgeneration Ġtasks . ĠB IB REF 19 Ġ, ĠB IB REF 20 Ġused Ġa Ġmixture Ġof Ġgeneration Ġand Ġpointing Ġto Ġovercome Ġthe Ġproblem Ġof Ġlarge Ġvocabulary Ġsize . ĠB IB REF 21 Ġ, ĠB IB REF 22 Ġapplied ĠSe q 2 Se q Ġmodels Ġon Ġsummary Ġgeneration Ġtasks , Ġwhile ĠB IB REF 23 Ġ, ĠB IB REF 24 Ġgenerated Ġquestions Ġconditioned Ġon Ġdocuments Ġand Ġanswers Ġfrom Ġmachine Ġcomprehension Ġdatasets . ĠSe q 2 Se q Ġwas Ġalso Ġapplied Ġon Ġneural Ġsentence Ġsimpl ification ĠB IB REF 25 Ġand Ġparaph rase Ġgeneration Ġtasks ĠB IB REF 26 Ġ. </s> Given Ġa Ġsource Ġtext Ġconsisting Ġof ĠIN LINE FORM 0 Ġwords ĠIN LINE FORM 1 Ġ, Ġthe Ġenc oder Ġconverts Ġtheir Ġcorresponding Ġembed d ings ĠIN LINE FORM 2 Ġinto Ġa Ġset Ġof ĠIN LINE FORM 3 Ġreal - valued Ġvectors ĠIN LINE FORM 4 Ġwith Ġa Ġbid irection al ĠGR U ĠB IB REF 27 Ġ: ĠDIS PLAY FORM 0 </s> Drop out ĠB IB REF 28 Ġis Ġapplied Ġto Ġboth ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġfor Ġregular ization . </s> The Ġdec oder Ġis Ġa Ġun i - direction al ĠGR U , Ġwhich Ġgenerates Ġa Ġnew Ġstate ĠIN LINE FORM 0 Ġat Ġeach Ġtime - step ĠIN LINE FORM 1 Ġfrom Ġthe Ġword Ġembed ding ĠIN LINE FORM 2 Ġand Ġthe Ġrecurrent Ġstate ĠIN LINE FORM 3 Ġ: ĠDIS PLAY FORM 0 </s> The Ġinitial Ġstate ĠIN LINE FORM 0 Ġis Ġderived Ġfrom Ġthe Ġfinal Ġenc oder Ġstate ĠIN LINE FORM 1 Ġby Ġapplying Ġa Ġsingle - layer Ġfeed - forward Ġneural Ġnet Ġ( F NN ): ĠIN LINE FORM 2 Ġ. ĠDrop out Ġis Ġapplied Ġto Ġboth Ġthe Ġembed d ings ĠIN LINE FORM 3 Ġand Ġthe ĠGR U Ġstates ĠIN LINE FORM 4 Ġ. </s> When Ġgenerating Ġtoken ĠIN LINE FORM 0 Ġ, Ġin Ġorder Ġto Ġbetter Ġincorporate Ġinformation Ġfrom Ġthe Ġsource Ġtext , Ġan Ġattention Ġmechanism ĠB IB REF 18 Ġis Ġemployed Ġto Ġinfer Ġthe Ġimportance ĠIN LINE FORM 1 Ġof Ġeach Ġsource Ġword ĠIN LINE FORM 2 Ġgiven Ġthe Ġcurrent Ġdec oder Ġstate ĠIN LINE FORM 3 Ġ. ĠThis Ġimportance Ġis Ġmeasured Ġby Ġan Ġenergy Ġfunction Ġwith Ġa Ġ2 - layer ĠF NN : ĠDIS PLAY FORM 0 </s> The Ġoutput Ġover Ġall Ġdecoding Ġsteps ĠIN LINE FORM 0 Ġthus Ġdefine Ġa Ġdistribution Ġover Ġthe Ġsource Ġsequence : ĠDIS PLAY FORM 0 </s> These Ġattention Ġscores Ġare Ġthen Ġused Ġas Ġweights Ġfor Ġa Ġrefined Ġrepresentation Ġof Ġthe Ġsource Ġenc od ings , Ġwhich Ġis Ġthen Ġconc aten ated Ġto Ġthe Ġdec oder Ġstate ĠIN LINE FORM 0 Ġto Ġderive Ġa Ġgener ative Ġdistribution ĠIN LINE FORM 1 Ġ: ĠDIS PLAY FORM 0 </s> where Ġthe Ġoutput Ġsize Ġof ĠIN LINE FORM 0 Ġequals Ġto Ġthe Ġtarget Ġvocabulary Ġsize . ĠSub script ĠIN LINE FORM 1 Ġindicates Ġthe Ġabstract ive Ġnature Ġof ĠIN LINE FORM 2 Ġsince Ġit Ġis Ġa Ġdistribution Ġover Ġa Ġprescribed Ġvocabulary . </s> We Ġemploy Ġthe Ġpointer Ġsoft max ĠB IB REF 19 Ġmechanism Ġto Ġswitch Ġbetween Ġgenerating Ġa Ġtoken ĠIN LINE FORM 0 Ġ( from Ġa Ġvocabulary ) Ġand Ġpointing Ġ( to Ġa Ġtoken Ġin Ġthe Ġsource Ġtext ). ĠSpecifically , Ġthe Ġpointer Ġsoft max Ġmodule Ġcomp utes Ġa Ġscal ar Ġswitch ĠIN LINE FORM 1 Ġat Ġeach Ġgeneration Ġtime - step Ġand Ġuses Ġit Ġto Ġinterpol ate Ġthe Ġabstract ive Ġdistribution ĠIN LINE FORM 2 Ġover Ġthe Ġvocabulary Ġ( see ĠEqu ation ĠEQ REF 16 Ġ) Ġand Ġthe Ġextract ive Ġdistribution ĠIN LINE FORM 3 Ġover Ġthe Ġsource Ġtext Ġtokens : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġconditioned Ġon Ġboth Ġthe Ġattention - weight ed Ġsource Ġrepresentation ĠIN LINE FORM 1 Ġand Ġthe Ġdec oder Ġstate ĠIN LINE FORM 2 Ġ: ĠDIS PLAY FORM 0 </s> Model ĠArchitecture </s> Given Ġa Ġpiece Ġof Ġsource Ġtext , Ġour Ġobjective Ġis Ġto Ġgenerate Ġa Ġvariable Ġnumber Ġof Ġmulti - word Ġphrases . ĠTo Ġthis Ġend , Ġwe Ġopt Ġfor Ġthe Ġsequence - to - sequence Ġframework Ġ( Se q 2 Se q ) Ġas Ġthe Ġbasis Ġof Ġour Ġmodel , Ġcombined Ġwith Ġattention Ġand Ġpointer Ġsoft max Ġmechanisms Ġin Ġthe Ġdec oder . </s> Since Ġeach Ġdata Ġexample Ġcontains Ġone Ġsource Ġtext Ġsequence Ġand Ġmultiple Ġtarget Ġphrase Ġsequences Ġ( d ub bed ĠOne 2 Many , Ġand Ġeach Ġsequence Ġcan Ġbe Ġof Ġmulti - word ), Ġtwo Ġparad ig ms Ġcan Ġbe Ġadopted Ġfor Ġtraining ĠSe q 2 Se q Ġmodels . ĠThe Ġfirst Ġone ĠB IB REF 0 Ġis Ġto Ġdivide Ġeach ĠOne 2 Many Ġdata Ġexample Ġinto Ġmultiple ĠOne 2 One Ġexamples , Ġand Ġthe Ġresulting Ġmodels Ġ( e . g . ĠCopy R NN ) Ġcan Ġgenerate Ġone Ġphrase Ġat Ġonce Ġand Ġmust Ġrely Ġon Ġbeam Ġsearch Ġtechnique Ġto Ġproduce Ġmore Ġunique Ġphrases . </s> To Ġenable Ġmodels Ġto Ġgenerate Ġmultiple Ġphrases Ġand Ġcontrol Ġthe Ġnumber Ġto Ġoutput , Ġwe Ġpropose Ġthe Ġsecond Ġtraining Ġparadigm ĠOne 2 Se q , Ġin Ġwhich Ġwe Ġconc aten ate Ġmultiple Ġphrases Ġinto Ġa Ġsingle Ġsequence Ġwith Ġa Ġdelim iter ĠIN LINE FORM 0 ĠSE P ĠIN LINE FORM 1 Ġ, Ġand Ġthis Ġconc aten ated Ġsequence Ġis Ġthen Ġused Ġas Ġthe Ġtarget Ġfor Ġsequence Ġgeneration Ġduring Ġtraining . ĠAn Ġoverview Ġof Ġthe Ġmodel 's Ġstructure Ġis Ġshown Ġin ĠFigure ĠFIG REF 8 Ġ. </s> Not ations </s> In Ġthe Ġfollowing Ġsubsections , Ġwe Ġuse ĠIN LINE FORM 0 Ġto Ġdenote Ġinput Ġtext Ġtokens , ĠIN LINE FORM 1 Ġto Ġdenote Ġtoken Ġembed d ings , ĠIN LINE FORM 2 Ġto Ġdenote Ġhidden Ġstates , Ġand ĠIN LINE FORM 3 Ġto Ġdenote Ġoutput Ġtext Ġtokens . ĠSup ers cript s Ġdenote Ġtime - steps Ġin Ġa Ġsequence , Ġand Ġsubscript s ĠIN LINE FORM 4 Ġand ĠIN LINE FORM 5 Ġindicate Ġwhether Ġa Ġvariable Ġresides Ġin Ġthe Ġenc oder Ġor Ġthe Ġdec oder Ġof Ġthe Ġmodel , Ġrespectively . ĠThe Ġabsence Ġof Ġa Ġsupers cript Ġindicates Ġmultipl icity Ġin Ġthe Ġtime Ġdimension . ĠIN LINE FORM 6 Ġrefers Ġto Ġa Ġlinear Ġtransformation Ġand ĠIN LINE FORM 7 Ġrefers Ġto Ġit Ġfollowed Ġby Ġa Ġnon - linear Ġactivation Ġfunction ĠIN LINE FORM 8 Ġ. ĠAng led Ġbrackets , ĠIN LINE FORM 9 Ġ, Ġdenote Ġconc aten ation . </s> Mech an isms Ġfor ĠD iverse ĠGeneration </s> There Ġare Ġusually Ġmultiple Ġkey ph r ases Ġfor Ġa Ġgiven Ġsource Ġtext Ġbecause Ġeach Ġkey phrase Ġrepresents Ġcertain Ġaspects Ġof Ġthe Ġtext . ĠTherefore Ġkey phrase Ġdiversity Ġis Ġdesired Ġfor Ġthe Ġkey phrase Ġgeneration . ĠMost Ġprevious Ġkey phrase Ġgeneration Ġmodels Ġgenerate Ġmultiple Ġphrases Ġby Ġover - generation , Ġwhich Ġis Ġhighly Ġprone Ġto Ġgenerate Ġsimilar Ġphrases Ġdue Ġto Ġthe Ġnature Ġof Ġbeam Ġsearch . ĠGiven Ġour Ġobjective Ġto Ġgenerate Ġvariable Ġnumbers Ġof Ġkey ph r ases , Ġwe Ġneed Ġto Ġadopt Ġnew Ġstrategies Ġfor Ġachieving Ġbetter Ġdiversity Ġin Ġthe Ġoutput . </s> Rec all Ġthat Ġwe Ġrepresent Ġvariable Ġnumbers Ġof Ġkey ph r ases Ġas Ġdelim iter - separ ated Ġsequences . ĠOne Ġparticular Ġissue Ġwe Ġobserved Ġduring Ġerror Ġanalysis Ġis Ġthat Ġthe Ġmodel Ġtends Ġto Ġproduce Ġidentical Ġtokens Ġfollowing Ġthe Ġdelim iter Ġtoken . ĠFor Ġexample , Ġsuppose Ġa Ġtarget Ġsequence Ġcontains ĠIN LINE FORM 0 Ġdelim iter Ġtokens Ġat Ġtime - steps ĠIN LINE FORM 1 Ġ. ĠDuring Ġtraining , Ġthe Ġmodel Ġis Ġrewarded Ġfor Ġgenerating Ġthe Ġsame Ġdelim iter Ġtoken Ġat Ġthese Ġtime - steps , Ġwhich Ġpresumably Ġintroduces Ġmuch Ġhom ogeneity Ġin Ġthe Ġcorresponding Ġdec oder Ġstates ĠIN LINE FORM 2 Ġ. ĠWhen Ġthese Ġstates Ġare Ġsubsequently Ġused Ġas Ġinputs Ġat Ġthe Ġtime - steps Ġimmediately Ġfollowing Ġthe Ġdelim iter , Ġthe Ġdec oder Ġnaturally Ġproduces Ġhighly Ġsimilar Ġdistributions Ġover Ġthe Ġfollowing Ġtokens , Ġresulting Ġin Ġidentical Ġtokens Ġbeing Ġdec oded . ĠTo Ġalleviate Ġthis Ġproblem , Ġwe Ġpropose Ġtwo Ġplug - in Ġcomponents Ġfor Ġthe Ġsequential Ġgeneration Ġmodel . </s> We Ġpropose Ġa Ġmechanism Ġcalled Ġsemantic Ġcoverage Ġthat Ġfocuses Ġon Ġthe Ġsemantic Ġrepresentations Ġof Ġgenerated Ġphrases . ĠSpecifically , Ġwe Ġintroduce Ġanother Ġun i - direction al Ġrecurrent Ġmodel ĠIN LINE FORM 0 Ġ( d ub bed Ġtarget Ġenc oder ) Ġwhich Ġenc odes Ġdec oder - generated Ġtokens ĠIN LINE FORM 1 Ġ, Ġwhere ĠIN LINE FORM 2 Ġ, Ġinto Ġhidden Ġstates ĠIN LINE FORM 3 Ġ. ĠThis Ġstate Ġis Ġthen Ġtaken Ġas Ġan Ġextra Ġinput Ġto Ġthe Ġdec oder ĠGR U , Ġmodifying ĠEqu ation ĠEQ REF 12 Ġto : ĠDIS PLAY FORM 0 </s> If Ġthe Ġtarget Ġenc oder Ġwere Ġto Ġbe Ġupdated Ġwith Ġthe Ġtraining Ġsignal Ġfrom Ġgeneration Ġ( i . e ., Ġback prop ag ating Ġerror Ġfrom Ġthe Ġdec oder ĠGR U Ġto Ġthe Ġtarget Ġenc oder ), Ġthe Ġresulting Ġdec oder Ġis Ġessentially Ġa Ġ2 - layer ĠGR U Ġwith Ġresidual Ġconnections . ĠInstead , Ġinspired Ġby Ġprevious Ġrepresentation Ġlearning Ġworks ĠB IB REF 29 Ġ, ĠB IB REF 30 Ġ, ĠB IB REF 31 Ġ, Ġwe Ġtrain Ġthe Ġtarget Ġenc oder Ġin Ġan Ġself - super vised Ġfashion Ġ( Figure ĠFIG REF 8 Ġ). ĠThat Ġis , Ġwe Ġextract Ġtarget Ġenc oder 's Ġfinal Ġhidden Ġstate Ġvector ĠIN LINE FORM 0 Ġ, Ġwhere ĠIN LINE FORM 1 Ġis Ġthe Ġlength Ġof Ġtarget Ġsequence , Ġand Ġuse Ġit Ġas Ġa Ġgeneral Ġrepresentation Ġof Ġthe Ġtarget Ġphrases . ĠWe Ġtrain Ġby Ġmaximizing Ġthe Ġmutual Ġinformation Ġbetween Ġthese Ġphrase Ġrepresentations Ġand Ġthe Ġfinal Ġstate Ġof Ġthe Ġsource Ġenc oder ĠIN LINE FORM 2 Ġas Ġfollows . ĠFor Ġeach Ġphrase Ġrepresentation Ġvector ĠIN LINE FORM 3 Ġ, Ġwe Ġtake Ġthe Ġen oc d ings ĠIN LINE FORM 4 Ġof ĠIN LINE FORM 5 Ġdifferent Ġsource Ġtexts , Ġwhere ĠIN LINE FORM 6 Ġis Ġthe Ġenc oder Ġrepresentation Ġfor Ġthe Ġcurrent Ġsource Ġtext , Ġand Ġthe Ġremaining ĠIN LINE FORM 7 Ġare Ġnegative Ġsamples Ġ( sam pled Ġat Ġrandom ) Ġfrom Ġthe Ġtraining Ġdata . ĠThe Ġtarget Ġenc oder Ġis Ġtrained Ġto Ġminimize Ġthe Ġclassification Ġloss : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġbi - linear Ġtransformation . </s> The Ġmotivation Ġhere Ġis Ġto Ġconst rain Ġthe Ġoverall Ġrepresentation Ġof Ġgenerated Ġkey phrase Ġto Ġbe Ġsem antically Ġclose Ġto Ġthe Ġoverall Ġmeaning Ġof Ġthe Ġsource Ġtext . ĠWith Ġsuch Ġrepresentations Ġas Ġinput Ġto Ġthe Ġdec oder , Ġthe Ġsemantic Ġcoverage Ġmechanism Ġcan Ġpotentially Ġhelp Ġto Ġprovide Ġuseful Ġkey phrase Ġinformation Ġand Ġguide Ġgeneration . </s> We Ġalso Ġpropose Ġorth og onal Ġregular ization , Ġwhich Ġexplicitly Ġencourages Ġthe Ġdelim iter - gener ating Ġdec oder Ġstates Ġto Ġbe Ġdifferent Ġfrom Ġeach Ġother . ĠThis Ġis Ġinspired Ġby ĠB IB REF 32 Ġ, Ġwho Ġuse Ġorth og onal Ġregular ization Ġto Ġencourage Ġrepresentations Ġacross Ġdomains Ġto Ġbe Ġas Ġdistinct Ġas Ġpossible . ĠSpecifically , Ġwe Ġstack Ġthe Ġdec oder Ġhidden Ġstates Ġcorresponding Ġto Ġdelim it ers Ġtogether Ġto Ġform Ġmatrix ĠIN LINE FORM 0 Ġand Ġuse Ġthe Ġfollowing Ġequation Ġas Ġthe Ġorth og onal Ġregular ization Ġloss : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġmatrix Ġtrans pose Ġof ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġis Ġthe Ġidentity Ġmatrix Ġof Ġrank ĠIN LINE FORM 3 Ġ, ĠIN LINE FORM 4 Ġindicates Ġelement Ġwise Ġmultiplication , ĠIN LINE FORM 5 Ġindicates ĠIN LINE FORM 6 Ġnorm Ġof Ġeach Ġelement Ġin Ġa Ġmatrix ĠIN LINE FORM 7 Ġ. ĠThis Ġloss Ġfunction Ġprefers Ġorth og on ality Ġamong Ġthe Ġhidden Ġstates ĠIN LINE FORM 8 Ġand Ġthus Ġimproves Ġdiversity Ġin Ġthe Ġtokens Ġfollowing Ġthe Ġdelim it ers . </s> We Ġadopt Ġthe Ġwidely Ġused Ġnegative Ġlog - like lihood Ġloss Ġin Ġour Ġsequence Ġgeneration Ġmodel , Ġden oted Ġas ĠIN LINE FORM 0 Ġ. ĠThe Ġoverall Ġloss Ġwe Ġuse Ġin Ġour Ġmodel Ġis ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġare Ġhyper - param eters . </s> Dec oding ĠStrategies </s> According Ġto Ġdifferent Ġtask Ġrequirements , Ġvarious Ġdecoding Ġmethods Ġcan Ġbe Ġapplied Ġto Ġgenerate Ġthe Ġtarget Ġsequence ĠIN LINE FORM 0 Ġ. ĠPrior Ġstudies ĠB IB REF 0 Ġ, ĠB IB REF 7 Ġfocus Ġmore Ġon Ġgenerating Ġexcessive Ġnumber Ġof Ġphrases Ġby Ġleveraging Ġbeam Ġsearch Ġto Ġprolifer ate Ġthe Ġoutput Ġphrases . ĠIn Ġcontrast , Ġmodels Ġtrained Ġunder ĠOne 2 Se q Ġparadigm Ġare Ġcapable Ġof Ġdetermining Ġthe Ġproper Ġnumber Ġof Ġphrases Ġto Ġoutput . ĠIn Ġlight Ġof Ġprevious Ġresearch Ġin Ġpsychology ĠB IB REF 33 Ġ, ĠB IB REF 34 Ġ, Ġwe Ġname Ġthese Ġtwo Ġdecoding / search Ġstrategies Ġas ĠEx haust ive ĠDec oding Ġand ĠSelf - termin ating ĠDec oding , Ġrespectively , Ġdue Ġto Ġtheir Ġresemblance Ġto Ġthe Ġway Ġhumans Ġbehave Ġin Ġserial Ġmemory Ġtasks . ĠSimply Ġspeaking , Ġthe Ġmajor Ġdifference Ġlies Ġin Ġwhether Ġa Ġmodel Ġis Ġcapable Ġof Ġcontrolling Ġthe Ġnumber Ġof Ġphrases Ġto Ġoutput . ĠWe Ġdescribe Ġthe Ġdetailed Ġdecoding Ġstrategies Ġused Ġin Ġthis Ġstudy Ġas Ġfollows : </s> As Ġtraditional Ġkey phrase Ġtasks Ġevaluate Ġmodels Ġwith Ġa Ġfixed Ġnumber Ġof Ġtop - ranked Ġpredictions Ġ( say ĠF - score Ġ@ 5 Ġand Ġ@ 10 ), Ġexisting Ġkey phrase Ġgeneration Ġstudies Ġhave Ġto Ġover - gener ate Ġphrases Ġby Ġmeans Ġof Ġbeam Ġsearch Ġ( common ly Ġwith Ġa Ġlarge Ġbeam Ġsize , Ġe . g ., Ġ150 Ġand Ġ200 Ġin ĠB IB REF 3 Ġ, ĠB IB REF 0 Ġ, Ġrespectively ), Ġa Ġhe uristic Ġsearch Ġalgorithm Ġthat Ġreturns ĠIN LINE FORM 0 Ġapproximate Ġoptimal Ġsequences . ĠFor Ġthe ĠOne 2 One Ġsetting , Ġeach Ġreturned Ġsequence Ġis Ġa Ġunique Ġphrase Ġitself . ĠBut Ġfor ĠOne 2 Se q , Ġeach Ġproduced Ġsequence Ġcontains Ġseveral Ġphrases Ġand Ġadditional Ġprocesses ĠB IB REF 2 Ġare Ġneeded Ġto Ġobtain Ġthe Ġfinal Ġunique Ġ( ordered ) Ġphrase Ġlist . </s> It Ġis Ġworth Ġnoting Ġthat Ġthe Ġtime Ġcomplexity Ġof Ġbeam Ġsearch Ġis ĠIN LINE FORM 0 Ġ, Ġwhere ĠIN LINE FORM 1 Ġis Ġthe Ġbeam Ġwidth , Ġand ĠIN LINE FORM 2 Ġis Ġthe Ġmaximum Ġlength Ġof Ġgenerated Ġsequences . ĠTherefore Ġthe Ġexhaustive Ġdecoding Ġis Ġgenerally Ġvery Ġcomput ationally Ġexpensive , Ġespecially Ġfor ĠOne 2 Se q Ġsetting Ġwhere ĠIN LINE FORM 3 Ġis Ġmuch Ġlarger Ġthan Ġin ĠOne 2 One . ĠIt Ġis Ġalso Ġwasteful Ġas Ġwe Ġobserve Ġthat Ġless Ġthan Ġ5 % Ġof Ġphrases Ġgenerated Ġby ĠOne 2 Se q Ġmodels Ġare Ġunique . </s> An Ġinnate Ġcharacteristic Ġof Ġkey phrase Ġtasks Ġis Ġthat Ġthe Ġnumber Ġof Ġkey ph r ases Ġvaries Ġdepending Ġon Ġthe Ġdocument Ġand Ġdataset Ġgenre , Ġtherefore Ġdynamically Ġoutput ting Ġa Ġvariable Ġnumber Ġof Ġphrases Ġis Ġa Ġdesirable Ġproperty Ġfor Ġkey phrase Ġgeneration Ġmodels . ĠSince Ġour Ġproposed Ġmodel Ġis Ġtrained Ġto Ġgenerate Ġa Ġvariable Ġnumber Ġof Ġphrases Ġas Ġa Ġsingle Ġsequence Ġjoined Ġby Ġdelim it ers , Ġwe Ġcan Ġobtain Ġmultiple Ġphrases Ġby Ġsimply Ġdecoding Ġa Ġsingle Ġsequence Ġfor Ġeach Ġgiven Ġsource Ġtext . ĠThe Ġresulting Ġmodel Ġthus Ġimplicitly Ġperforms Ġthe Ġadditional Ġtask Ġof Ġdynamically Ġestimating Ġthe Ġproper Ġsize Ġof Ġthe Ġtarget Ġphrase Ġset : Ġonce Ġthe Ġmodel Ġbelieves Ġthat Ġan Ġadequate Ġnumber Ġof Ġphrases Ġhave Ġbeen Ġgenerated , Ġit Ġoutputs Ġa Ġspecial Ġtoken ĠIN LINE FORM 0 ĠE OS ĠIN LINE FORM 1 Ġto Ġterminate Ġthe Ġdecoding Ġprocess . </s> One Ġnotable Ġattribute Ġof Ġthe Ġself - termin ating Ġdecoding Ġstrategy Ġis Ġthat , Ġby Ġgenerating Ġa Ġset Ġof Ġphrases Ġin Ġa Ġsingle Ġsequence , Ġthe Ġmodel Ġconditions Ġits Ġcurrent Ġgeneration Ġon Ġall Ġpreviously Ġgenerated Ġphrases . ĠCompared Ġto Ġthe Ġexhaustive Ġstrategy Ġ( i . e ., Ġphrases Ġbeing Ġgenerated Ġindependently Ġby Ġbeam Ġsearch Ġin Ġparallel ), Ġour Ġmodel Ġcan Ġmodel Ġthe Ġdependency Ġamong Ġits Ġoutput Ġin Ġa Ġmore Ġexplicit Ġfashion . ĠAdditionally , Ġsince Ġmultiple Ġphrases Ġare Ġdec oded Ġas Ġa Ġsingle Ġsequence , Ġdecoding Ġcan Ġbe Ġperformed Ġmore Ġefficiently Ġthan Ġexhaustive Ġdecoding Ġby Ġconducting Ġgreedy Ġsearch Ġor Ġbeam Ġsearch Ġon Ġonly Ġthe Ġtop - sc ored Ġsequence . </s> E val uating ĠKey phrase ĠGeneration </s> Form ally , Ġgiven Ġa Ġsource Ġtext , Ġsuppose Ġthat Ġa Ġmodel Ġpredicts Ġa Ġlist Ġof Ġunique Ġkey ph r ases ĠIN LINE FORM 0 Ġordered Ġby Ġthe Ġquality Ġof Ġthe Ġpredictions ĠIN LINE FORM 1 Ġ, Ġand Ġthat Ġthe Ġground Ġtruth Ġkey ph r ases Ġfor Ġthe Ġgiven Ġsource Ġtext Ġis Ġthe Ġor acle Ġset ĠIN LINE FORM 2 Ġ. ĠWhen Ġonly Ġthe Ġtop ĠIN LINE FORM 3 Ġpredictions ĠIN LINE FORM 4 Ġare Ġused Ġfor Ġevaluation , Ġprecision , Ġrecall , Ġand ĠF ĠIN LINE FORM 5 Ġscore Ġare Ġconsequently Ġconditioned Ġon ĠIN LINE FORM 6 Ġand Ġdefined Ġas : ĠDIS PLAY FORM 0 </s> As Ġdiscussed Ġin ĠSection ĠSEC REF 1 Ġ, Ġthe Ġnumber Ġof Ġgenerated Ġkey ph r ases Ġused Ġfor Ġevaluation Ġcan Ġhave Ġa Ġcritical Ġimpact Ġon Ġthe Ġquality Ġof Ġthe Ġresulting Ġevaluation Ġmetrics . ĠHere Ġwe Ġcompare Ġthree Ġchoices Ġof ĠIN LINE FORM 0 Ġand Ġthe Ġimplications Ġon Ġkey phrase Ġevaluation Ġfor Ġeach Ġchoice : </s> A Ġsimple Ġremedy Ġis Ġto Ġset ĠIN LINE FORM 0 Ġas Ġa Ġvariable Ġnumber Ġwhich Ġis Ġspecific Ġto Ġeach Ġdata Ġexample . ĠHere Ġwe Ġdefine Ġtwo Ġnew Ġmetrics : </s> By Ġsimply Ġextending Ġthe Ġconstant Ġnumber ĠIN LINE FORM 0 Ġto Ġdifferent Ġvariables Ġaccordingly , Ġboth ĠF ĠIN LINE FORM 1 Ġ@ ĠIN LINE FORM 2 Ġand ĠF ĠIN LINE FORM 3 Ġ@ ĠIN LINE FORM 4 Ġare Ġcapable Ġof Ġreflecting Ġthe Ġnature Ġof Ġvariable Ġnumber Ġof Ġphrases Ġfor Ġeach Ġdocument , Ġand Ġa Ġmodel Ġcan Ġachieve Ġthe Ġmaximum ĠIN LINE FORM 5 Ġscore Ġof ĠIN LINE FORM 6 Ġif Ġand Ġonly Ġif Ġit Ġpredicts Ġthe Ġexact Ġsame Ġphrases Ġas Ġthe Ġground Ġtruth . ĠAnother Ġmerit Ġof ĠF ĠIN LINE FORM 7 Ġ@ ĠIN LINE FORM 8 Ġis Ġthat Ġit Ġis Ġindependent Ġfrom Ġmodel Ġoutputs , Ġtherefore Ġwe Ġcan Ġuse Ġit Ġto Ġcompare Ġexisting Ġmodels . </s> Dat as ets Ġand ĠExper iments </s> In Ġthis Ġsection , Ġwe Ġreport Ġour Ġexperiment Ġresults Ġon Ġmultiple Ġdatasets Ġand Ġcompare Ġwith Ġexisting Ġmodels . ĠWe Ġuse ĠIN LINE FORM 0 Ġto Ġrefer Ġto Ġthe Ġdelim iter - con cat en ated Ġsequence - to - sequ ences Ġmodel Ġdescribed Ġin ĠSection ĠSEC REF 3 Ġ; ĠIN LINE FORM 1 Ġrefers Ġto Ġthe Ġmodel Ġaugmented Ġwith Ġorth og onal Ġregular ization Ġand Ġsemantic Ġcoverage Ġmechanism . </s> To Ġconstruct Ġtarget Ġsequences Ġfor Ġtraining ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġ, Ġground Ġtruth Ġkey ph r ases Ġare Ġsorted Ġby Ġtheir Ġorder Ġof Ġfirst Ġoccurrence Ġin Ġthe Ġsource Ġtext . ĠKey ph r ases Ġthat Ġdo Ġnot Ġappear Ġin Ġthe Ġsource Ġtext Ġare Ġapp ended Ġto Ġthe Ġend . ĠThis Ġorder Ġmay Ġguide Ġthe Ġattention Ġmechanism Ġto Ġattend Ġto Ġsource Ġpositions Ġin Ġa Ġsmoother Ġway . ĠImplementation Ġdetails Ġcan Ġbe Ġfound Ġin ĠAppendix ĠSEC REF 9 Ġ. </s> We Ġinclude Ġfour Ġnon - ne ural Ġextract ive Ġmodels Ġand ĠCopy R NN ĠB IB REF 0 Ġas Ġbas elines . ĠWe Ġuse ĠCopy R NN Ġto Ġdenote Ġthe Ġmodel Ġreported Ġby ĠB IB REF 0 Ġ, ĠCopy R NN * Ġto Ġdenote Ġour Ġimplementation Ġof ĠCopy R NN Ġbased Ġon Ġtheir Ġopen Ġsourced Ġcode . ĠTo Ġdraw Ġfair Ġcomparison Ġwith Ġexisting Ġstudy , Ġwe Ġuse Ġthe Ġsame Ġmodel Ġhyper param eter Ġsetting Ġas Ġused Ġin ĠB IB REF 0 Ġand Ġuse Ġexhaustive Ġdecoding Ġstrategy Ġfor Ġmost Ġexperiments . ĠK EA ĠB IB REF 4 Ġand ĠMau i ĠB IB REF 8 Ġare Ġtrained Ġon Ġa Ġsubset Ġof Ġ50 , 000 Ġdocuments Ġfrom Ġeither ĠKP 20 k Ġ( Table ĠT AB REF 35 Ġ) Ġor ĠStack Ex Ġ( Table ĠT AB REF 37 Ġ) Ġinstead Ġof Ġall Ġdocuments Ġdue Ġto Ġimplementation Ġlimits Ġ( without Ġfine - tun ing Ġon Ġtarget Ġdataset ). </s> In ĠSection ĠSEC REF 42 Ġ, Ġwe Ġapply Ġthe Ġself - termin ating Ġdecoding Ġstrategy . ĠSince Ġno Ġexisting Ġmodel Ġsupports Ġsuch Ġdecoding Ġstrategy , Ġwe Ġonly Ġreport Ġresults Ġfrom Ġour Ġproposed Ġmodels . ĠThey Ġcan Ġbe Ġused Ġfor Ġcomparison Ġin Ġfuture Ġstudies . </s> Exper iments Ġon ĠScientific ĠPublications </s> Our Ġfirst Ġdataset Ġconsists Ġof Ġa Ġcollection Ġof Ġscientific Ġpublication Ġdatasets , Ġnamely ĠKP 20 k , ĠIn spec , ĠK rap iv in , ĠN US , Ġand ĠSem E val , Ġthat Ġhave Ġbeen Ġwidely Ġused Ġin Ġexisting Ġliterature ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ. ĠKP 20 k , Ġfor Ġexample , Ġwas Ġintroduced Ġby ĠB IB REF 0 Ġand Ġcomprises Ġmore Ġthan Ġhalf Ġa Ġmillion Ġscientific Ġpublications . ĠFor Ġeach Ġarticle , Ġthe Ġabstract Ġand Ġtitle Ġare Ġused Ġas Ġthe Ġsource Ġtext Ġwhile Ġthe Ġauthor Ġkeywords Ġare Ġused Ġas Ġtarget . ĠThe Ġother Ġfour Ġdatasets Ġcontain Ġmuch Ġfewer Ġarticles , Ġand Ġthus Ġused Ġto Ġtest Ġtransfer ability Ġof Ġour Ġmodel Ġ( without Ġfine - tun ing ). </s> We Ġreport Ġour Ġmodel 's Ġperformance Ġon Ġthe Ġpresent - key phrase Ġportion Ġof Ġthe ĠKP 20 k Ġdataset Ġin ĠTable ĠT AB REF 35 Ġ. ĠTo Ġcompare Ġwith Ġprevious Ġworks , Ġwe Ġprovide Ġcompute ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġscores . ĠThe Ġnew Ġproposed ĠF ĠIN LINE FORM 2 Ġ@ ĠIN LINE FORM 3 Ġmetric Ġindicates Ġconsistent Ġranking Ġwith ĠIN LINE FORM 4 Ġfor Ġmost Ġcases . ĠDue Ġto Ġits Ġtarget Ġnumber Ġsensitivity , Ġwe Ġfind Ġthat Ġits Ġvalue Ġis Ġcloser Ġto ĠIN LINE FORM 5 Ġfor ĠKP 20 k Ġand ĠK rap iv in Ġwhere Ġaverage Ġtarget Ġkey ph r ases Ġis Ġless Ġand Ġcloser Ġto ĠIN LINE FORM 6 Ġfor Ġthe Ġother Ġthree Ġdatasets . </s> From Ġthe Ġresult Ġwe Ġcan Ġsee Ġthat Ġthe Ġneural - based Ġmodels Ġoutper form Ġnon - ne ural Ġmodels Ġby Ġlarge Ġmargins . ĠOur Ġimplemented ĠCopy R NN Ġachieves Ġbetter Ġor Ġcomparable Ġperformance Ġagainst Ġthe Ġoriginal Ġmodel , Ġand Ġon ĠN US Ġand ĠSem E val Ġthe Ġadvantage Ġis Ġmore Ġsalient . </s> As Ġfor Ġthe Ġproposed Ġmodels , Ġboth ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġyield Ġcomparable Ġresults Ġto ĠCopy R NN , Ġindicating Ġthat ĠOne 2 Se q Ġparadigm Ġcan Ġwork Ġwell Ġas Ġan Ġalternative Ġoption Ġfor Ġthe Ġkey phrase Ġgeneration . ĠIN LINE FORM 2 Ġoutper forms ĠIN LINE FORM 3 Ġon Ġall Ġmetrics , Ġsuggesting Ġthe Ġsemantic Ġcoverage Ġand Ġorth og onal Ġregular ization Ġhelp Ġthe Ġmodel Ġto Ġgenerate Ġhigher Ġquality Ġkey ph r ases Ġand Ġachieve Ġbetter Ġgeneral iz ability . ĠTo Ġour Ġsurprise , Ġon Ġthe Ġmetric ĠF ĠIN LINE FORM 4 Ġ@ 10 Ġfor ĠKP 20 k Ġand ĠK rap iv in Ġ( average Ġnumber Ġof Ġkey ph r ases Ġis Ġonly Ġ5 ), Ġwhere Ġhigh - rec all Ġmodels Ġlike ĠCopy R NN Ġare Ġmore Ġfavored , ĠIN LINE FORM 5 Ġis Ġstill Ġable Ġto Ġoutper form ĠOne 2 One Ġbas elines , Ġindicating Ġthat Ġthe Ġproposed Ġmechanisms Ġfor Ġdiverse Ġgeneration Ġare Ġeffective . </s> Exper iments Ġon ĠThe ĠStack Ex ĠDat as et </s> Insp ired Ġby Ġthe ĠStack L ite Ġtag Ġrecommendation Ġtask Ġon ĠK agg le , Ġwe Ġbuild Ġa Ġnew Ġbenchmark Ġbased Ġon Ġthe Ġpublic ĠStack Ex change Ġdata . ĠWe Ġuse Ġquestions Ġwith Ġtitles Ġas Ġsource , Ġand Ġuser - ass igned Ġtags Ġas Ġtarget Ġkey ph r ases . </s> Since Ġoft entimes Ġthe Ġquestions Ġon ĠStack Ex change Ġcontain Ġless Ġinformation Ġthan Ġin Ġscientific Ġpublications , Ġthere Ġare Ġfewer Ġkey ph r ases Ġper Ġdata Ġpoint Ġin ĠStack Ex . ĠFurthermore , ĠStack Ex change Ġuses Ġa Ġtag Ġrecommendation Ġsystem Ġthat Ġsuggests Ġtopic - relevant Ġtags Ġto Ġusers Ġwhile Ġsubmitting Ġquestions ; Ġtherefore , Ġwe Ġare Ġmore Ġlikely Ġto Ġsee Ġgeneral Ġterminology Ġsuch Ġas ĠLinux Ġand ĠJava . ĠThis Ġcharacteristic Ġchallenges Ġmodels Ġwith Ġrespect Ġto Ġtheir Ġability Ġto Ġdist ill Ġmajor Ġtopics Ġof Ġa Ġquestion Ġrather Ġthan Ġselecting Ġspecific Ġsnippets Ġfrom Ġthe Ġtext . </s> We Ġreport Ġour Ġmodels ' Ġperformance Ġon ĠStack Ex Ġin ĠTable ĠT AB REF 37 Ġ. ĠResults Ġshow ĠIN LINE FORM 0 Ġperforms Ġthe Ġbest ; Ġon Ġthe Ġabsent - key phrase Ġgeneration Ġtasks , Ġit Ġoutper forms ĠIN LINE FORM 1 Ġby Ġa Ġlarge Ġmargin . </s> Gener ating ĠVariable ĠNumber ĠKey ph r ases </s> One Ġkey Ġadvantage Ġof Ġour Ġproposed Ġmodel Ġis Ġthe Ġcapability Ġof Ġpredicting Ġthe Ġnumber Ġof Ġkey ph r ases Ġconditioned Ġon Ġthe Ġgiven Ġsource Ġtext . ĠWe Ġthus Ġconduct Ġa Ġset Ġof Ġexperiments Ġon ĠKP 20 k Ġand ĠStack Ex Ġpresent Ġkey phrase Ġgeneration Ġtasks , Ġas Ġshown Ġin ĠTable ĠT AB REF 39 Ġ, Ġto Ġstudy Ġsuch Ġbehavior . ĠWe Ġadopt Ġthe Ġself - termin ating Ġdecoding Ġstrategy Ġ( Section ĠSEC REF 28 Ġ), Ġand Ġuse Ġboth ĠF ĠIN LINE FORM 0 Ġ@ ĠIN LINE FORM 1 Ġand ĠF ĠIN LINE FORM 2 Ġ@ ĠIN LINE FORM 3 Ġ( Section ĠSEC REF 4 Ġ) Ġto Ġevaluate . </s> In Ġthese Ġexperiments , Ġwe Ġuse Ġbeam Ġsearch Ġas Ġin Ġmost ĠNatural ĠLanguage ĠGeneration Ġ( NL G ) Ġtasks , Ġi . e ., Ġonly Ġuse Ġthe Ġtop Ġranked Ġprediction Ġsequence Ġas Ġoutput . ĠWe Ġcompare Ġthe Ġresults Ġwith Ġgreedy Ġsearch . ĠSince Ġno Ġexisting Ġmodel Ġis Ġcapable Ġof Ġgenerating Ġvariable Ġnumber Ġof Ġkey ph r ases , Ġin Ġthis Ġsubsection Ġwe Ġonly Ġreport Ġperformance Ġon Ġsuch Ġsetting Ġfrom ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġ. </s> From ĠTable ĠT AB REF 39 Ġwe Ġobserve Ġthat Ġin Ġthe Ġvariable Ġnumber Ġgeneration Ġsetting , Ġgreedy Ġsearch Ġoutper forms Ġbeam Ġsearch Ġconsistently . ĠThis Ġmay Ġbecause Ġbeam Ġsearch Ġtends Ġto Ġgenerate Ġshort Ġand Ġsimilar Ġsequences . ĠWe Ġcan Ġalso Ġsee Ġthe Ġresulting ĠF ĠIN LINE FORM 0 Ġ@ ĠIN LINE FORM 1 Ġscores Ġare Ġgenerally Ġlower Ġthan Ġresults Ġreported Ġin Ġprevious Ġsubsections , Ġthis Ġsuggests Ġan Ġover - generation Ġdecoding Ġstrategy Ġmay Ġstill Ġbenefit Ġfrom Ġachieving Ġhigher Ġrecall . </s> A bl ation ĠStudy </s> We Ġconduct Ġan Ġab lation Ġexperiment Ġto Ġstudy Ġthe Ġeffects Ġof Ġorth og onal Ġregular ization Ġand Ġsemantic Ġcoverage Ġmechanism Ġon ĠIN LINE FORM 0 Ġ. ĠAs Ġshown Ġin ĠTable ĠT AB REF 44 Ġ, Ġsemantic Ġcoverage Ġprovides Ġsignificant Ġboost Ġto ĠIN LINE FORM 1 Ġ' s Ġperformance Ġon Ġall Ġdatasets . ĠOrth og onal Ġregular ization Ġhurts Ġperformance Ġwhen Ġis Ġsolely Ġapplied Ġto ĠIN LINE FORM 2 Ġmodel . ĠInterestingly , Ġwhen Ġboth Ġcomponents Ġare Ġenabled Ġ( ĠIN LINE FORM 3 Ġ), Ġthe Ġmodel Ġoutper forms ĠIN LINE FORM 4 Ġby Ġa Ġnoticeable Ġmargin Ġon Ġall Ġdatasets , Ġthis Ġsuggests Ġthe Ġtwo Ġcomponents Ġhelp Ġkey phrase Ġgeneration Ġin Ġa Ġsy ner getic Ġway . ĠOne Ġfuture Ġdirection Ġis Ġto Ġapply Ġorth og onal Ġregular ization Ġdirectly Ġon Ġtarget Ġenc oder , Ġsince Ġthe Ġregular izer Ġcan Ġpotentially Ġdivers ify Ġtarget Ġrepresentations Ġat Ġphrase Ġlevel , Ġwhich Ġmay Ġfurther Ġencourage Ġdiverse Ġkey phrase Ġgeneration Ġin Ġdec oder . </s> Visual izing ĠD ivers ified ĠGeneration </s> To Ġverify Ġour Ġassumption Ġthat Ġtarget Ġencoding Ġand Ġorth og onal Ġregular ization Ġhelp Ġto Ġboost Ġthe Ġdiversity Ġof Ġgenerated Ġsequences , Ġwe Ġuse Ġtwo Ġmetrics , Ġone Ġquantitative Ġand Ġone Ġqualitative , Ġto Ġmeasure Ġdiversity Ġof Ġgeneration . </s> First , Ġwe Ġsimply Ġcalculate Ġthe Ġaverage Ġunique Ġpredictions Ġproduced Ġby Ġboth ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġin Ġexperiments Ġshown Ġin ĠSection ĠSEC REF 36 Ġ. ĠThe Ġresulting Ġnumbers Ġare Ġ20 . 38 Ġand Ġ89 . 70 Ġfor ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġrespectively . ĠSecond , Ġfrom Ġthe Ġmodel Ġrunning Ġon Ġthe ĠKP 20 k Ġvalidation Ġset , Ġwe Ġrandomly Ġsample Ġ2000 Ġdec oder Ġhidden Ġstates Ġat ĠIN LINE FORM 4 Ġsteps Ġfollowing Ġa Ġdelim iter Ġ( ĠIN LINE FORM 5 Ġ) Ġand Ġapply Ġan Ġun super vised Ġclust ering Ġmethod Ġ( t - S NE ĠB IB REF 35 Ġ) Ġon Ġthem . ĠFrom Ġthe ĠFigure ĠFIG REF 46 Ġwe Ġcan Ġsee Ġthat Ġhidden Ġstates Ġsampled Ġfrom ĠIN LINE FORM 6 Ġare Ġeasier Ġto Ġcluster Ġwhile Ġhidden Ġstates Ġsampled Ġfrom ĠIN LINE FORM 7 Ġyield Ġone Ġmass Ġof Ġvectors Ġwith Ġno Ġobvious Ġdistinct Ġclusters . ĠResults Ġon Ġboth Ġmetrics Ġsuggest Ġtarget Ġencoding Ġand Ġorth og onal Ġregular ization Ġindeed Ġhelp Ġdivers ifying Ġgeneration Ġof Ġour Ġmodel . </s> Qual itative ĠAnalysis </s> To Ġillustrate Ġthe Ġdifference Ġof Ġpredictions Ġbetween Ġour Ġproposed Ġmodels , Ġwe Ġshow Ġan Ġexample Ġchosen Ġfrom Ġthe ĠKP 20 k Ġvalidation Ġset Ġin ĠAppendix ĠSEC REF 10 Ġ. ĠIn Ġthis Ġexample Ġthere Ġare Ġ29 Ġground Ġtruth Ġphrases . ĠNeither Ġof Ġthe Ġmodels Ġis Ġable Ġto Ġgenerate Ġall Ġof Ġthe Ġkey ph r ases , Ġbut Ġit Ġis Ġobvious Ġthat Ġthe Ġpredictions Ġfrom ĠIN LINE FORM 0 Ġall Ġstart Ġwith ĠâĢ ľ test âĢ Ŀ , Ġwhile Ġpredictions Ġfrom ĠIN LINE FORM 1 Ġare Ġdiverse . ĠThis Ġto Ġsome Ġextent Ġver ifies Ġour Ġassumption Ġthat Ġwithout Ġthe Ġtarget Ġenc oder Ġand Ġorth og onal Ġregular ization , Ġdec oder Ġstates Ġfollowing Ġdelim it ers Ġare Ġless Ġdiverse . </s> Conclusion Ġand ĠFuture ĠWork </s> We Ġpropose Ġa Ġrecurrent Ġgener ative Ġmodel Ġthat Ġsequ entially Ġgenerates Ġmultiple Ġkey ph r ases , Ġwith Ġtwo Ġextra Ġmodules Ġthat Ġenhance Ġgeneration Ġdiversity . ĠWe Ġpropose Ġnew Ġmetrics Ġto Ġevaluate Ġkey phrase Ġgeneration . ĠOur Ġmodel Ġshows Ġcompetitive Ġperformance Ġon Ġa Ġset Ġof Ġkey phrase Ġgeneration Ġdatasets , Ġincluding Ġone Ġintroduced Ġin Ġthis Ġwork . ĠIn Ġfuture Ġwork , Ġwe Ġplan Ġto Ġinvestigate Ġhow Ġtarget Ġphrase Ġorder Ġaffects Ġthe Ġgeneration Ġbehavior , Ġand Ġfurther Ġexplore Ġset Ġgeneration Ġin Ġan Ġorder Ġinvari ant Ġfashion . </s> Exper iment ĠResults Ġon ĠKP 20 k ĠAbsent ĠSub set </s> Gener ating Ġabsent Ġkey ph r ases Ġon Ġscientific Ġpublication Ġdatasets Ġis Ġa Ġrather Ġchallenging Ġproblem . ĠEx isting Ġstudies Ġoften Ġachieve Ġseemingly Ġgood Ġperformance Ġby Ġmeasuring Ġrecall Ġon Ġtens Ġand Ġsometimes Ġhundreds Ġof Ġkey ph r ases Ġproduced Ġby Ġexhaustive Ġdecoding Ġwith Ġa Ġlarge Ġbeam Ġsize ĠâĢĶ Ġthus Ġcompletely Ġignoring Ġprecision . </s> We Ġreport Ġthe Ġmodels ' ĠR @ 10 / 50 Ġscores Ġon Ġthe Ġabsent Ġportion Ġof Ġfive Ġscientific Ġpaper Ġdatasets Ġin ĠTable ĠT AB REF 48 Ġto Ġbe Ġin Ġline Ġwith Ġprevious Ġstudies . </s> The Ġabsent Ġkey phrase Ġprediction Ġhighly Ġprefers Ġrecall - oriented Ġmodels , Ġtherefore ĠCopy R NN Ġwith Ġbeam Ġsize Ġof Ġ200 Ġis Ġinn ately Ġproper Ġfor Ġthis Ġtask Ġsetting . ĠHow erer , Ġfrom Ġthe Ġresults Ġwe Ġobserve Ġthat Ġwith Ġthe Ġhelp Ġof Ġexhaustive Ġdecoding Ġand Ġdiverse Ġmechanisms , ĠIN LINE FORM 0 Ġis Ġable Ġto Ġperform Ġcompar ably Ġto ĠCopy R NN Ġmodel , Ġand Ġit Ġgenerally Ġworks Ġbetter Ġfor Ġtop Ġpredictions . ĠEven Ġthough Ġthe Ġtrend Ġof Ġmodels ' Ġperformance Ġsomewhat Ġmatches Ġwhat Ġwe Ġobserve Ġon Ġthe Ġpresent Ġdata , Ġwe Ġargue Ġthat Ġit Ġis Ġhard Ġto Ġcompare Ġdifferent Ġmodels ' Ġperformance Ġon Ġsuch Ġscale . ĠWe Ġargue Ġthat ĠStack Ex Ġis Ġbetter Ġtest bed s Ġfor Ġabsent Ġkey phrase Ġgeneration . </s> Im plementation ĠDetails </s> Im ple m nt ation Ġdetails Ġof Ġour Ġproposed Ġmodels Ġare Ġas Ġfollows . ĠIn Ġall Ġexperiments , Ġthe Ġword Ġembed d ings Ġare Ġinitialized Ġwith Ġ100 - dimensional Ġrandom Ġmat rices . ĠThe Ġnumber Ġof Ġhidden Ġunits Ġin Ġboth Ġthe Ġenc oder Ġand Ġdec oder ĠGR U Ġare Ġ150 . ĠThe Ġnumber Ġof Ġhidden Ġunits Ġin Ġtarget Ġenc oder ĠGR U Ġis Ġ150 . ĠThe Ġsize Ġof Ġvocabulary Ġis Ġ50 , 000 . </s> The Ġnumbers Ġof Ġhidden Ġunits Ġin ĠML Ps Ġdescribed Ġin ĠSection ĠSEC REF 3 Ġare Ġas Ġfollows . ĠDuring Ġnegative Ġsampling , Ġwe Ġrandomly Ġsample Ġ16 Ġsamples Ġfrom Ġthe Ġsame Ġbatch , Ġthus Ġtarget Ġencoding Ġloss Ġin ĠEqu ation ĠEQ REF 23 Ġis Ġa Ġ17 - way Ġclassification Ġloss . ĠIn ĠIN LINE FORM 0 Ġ, Ġwe Ġset Ġboth Ġthe ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġin ĠEqu ation ĠEQ REF 27 Ġto Ġbe Ġ0 . 3 . ĠIn Ġall Ġexperiments , Ġwe Ġuse Ġa Ġdrop out Ġrate Ġof Ġ0 . 1 . </s> We Ġuse ĠAdam ĠB IB REF 36 Ġas Ġthe Ġstep Ġrule Ġfor Ġoptimization . ĠThe Ġlearning Ġrate Ġis ĠIN LINE FORM 0 Ġ. ĠThe Ġmodel Ġis Ġimplemented Ġusing ĠPy Tor ch ĠB IB REF 38 Ġand ĠOpen N MT ĠB IB REF 37 Ġ. </s> For Ġexhaustive Ġdecoding , Ġwe Ġuse Ġa Ġbeam Ġsize Ġof Ġ50 Ġand Ġa Ġmaximum Ġsequence Ġlength Ġof Ġ40 . </s> Following ĠB IB REF 0 Ġ, Ġlower case Ġand Ġstemming Ġare Ġperformed Ġon Ġboth Ġthe Ġground Ġtruth Ġand Ġgenerated Ġkey ph r ases Ġduring Ġevaluation . </s> We Ġleave Ġout Ġ2 , 000 Ġdata Ġexamples Ġas Ġvalidation Ġset Ġfor Ġboth ĠKP 20 k Ġand ĠStack Ex Ġand Ġuse Ġthem Ġto Ġidentify Ġoptimal Ġcheckpoints Ġfor Ġtesting . ĠAnd Ġall Ġthe Ġscores Ġreported Ġin Ġthis Ġpaper Ġare Ġfrom Ġcheckpoints Ġwith Ġbest Ġperformances Ġ( F ĠIN LINE FORM 0 Ġ@ ĠIN LINE FORM 1 Ġ) Ġon Ġvalidation Ġset . </s> Example ĠOutput </s> See ĠTable ĠT AB REF 49 Ġ.