Using the OpenIE toolbox and applying heuristic rules to select the most relevant relation.
ABSTRACTIVE
off-the-shelf toolbox of OpenIE
EXTRACTIVE
To address this issue, we extract the structured answer-relevant relations from sentences and propose a method to jointly model such structured relation and the unstructured sentence for question generation. The structured answer-relevant relation is likely to be to the point context and thus can help keep the generated question to the point. For example, Figure FIGREF1 shows our framework can extract the right answer-relevant relation (“The daily mean temperature in January”, “is”, “32.6$^\circ $F (0.3$^\circ $C)”) among multiple facts. With the help of such structured information, our model is less likely to be confused by sentences with a complex structure. Specifically, we firstly extract multiple relations with an off-the-shelf Open Information Extraction (OpenIE) toolbox BIBREF7, then we select the relation that is most relevant to the answer with carefully designed heuristic rules.
We utilize an off-the-shelf toolbox of OpenIE to the derive structured answer-relevant relations from sentences as to the point contexts. Relations extracted by OpenIE can be represented either in a triple format or in an n-ary format with several secondary arguments, and we employ the latter to keep the extractions as informative as possible and avoid extracting too many similar relations in different granularities from one sentence. We join all arguments in the extracted n-ary relation into a sequence as our to the point context. Figure FIGREF5 shows n-ary relations extracted from OpenIE. As we can see, OpenIE extracts multiple relations for complex sentences. Here we select the most informative relation according to three criteria in the order of descending importance: (1) having the maximal number of overlapped tokens between the answer and the relation; (2) being assigned the highest confidence score by OpenIE; (3) containing maximum non-stop words. As shown in Figure FIGREF5, our criteria can select answer-relevant relations (waved in Figure FIGREF5), which is especially useful for sentences with extraneous information. In rare cases, OpenIE cannot extract any relation, we treat the sentence itself as the to the point context.
1910.06036
Introduction </s> Question ĠGeneration Ġ( Q G ) Ġis Ġthe Ġtask Ġof Ġautomatically Ġcreating Ġquestions Ġfrom Ġa Ġrange Ġof Ġinputs , Ġsuch Ġas Ġnatural Ġlanguage Ġtext ĠB IB REF 0 , Ġknowledge Ġbase ĠB IB REF 1 Ġand Ġimage ĠB IB REF 2 . ĠQ G Ġis Ġan Ġincreasingly Ġimportant Ġarea Ġin ĠN LP Ġwith Ġvarious Ġapplication Ġscenarios Ġsuch Ġas Ġintelligence Ġtutor Ġsystems , Ġopen - domain Ġchat bots Ġand Ġquestion Ġanswering Ġdataset Ġconstruction . ĠIn Ġthis Ġpaper , Ġwe Ġfocus Ġon Ġquestion Ġgeneration Ġfrom Ġreading Ġcomprehension Ġmaterials Ġlike ĠS Qu AD ĠB IB REF 3 . ĠAs Ġshown Ġin ĠFigure ĠFIG REF 1 , Ġgiven Ġa Ġsentence Ġin Ġthe Ġreading Ġcomprehension Ġparagraph Ġand Ġthe Ġtext Ġfragment Ġ( i . e ., Ġthe Ġanswer ) Ġthat Ġwe Ġwant Ġto Ġask Ġabout , Ġwe Ġaim Ġto Ġgenerate Ġa Ġquestion Ġthat Ġis Ġasked Ġabout Ġthe Ġspecified Ġanswer . </s> Question Ġgeneration Ġfor Ġreading Ġcomprehension Ġis Ġfirst ly Ġformal ized Ġas Ġa Ġdecl ar ative - to - inter rog ative Ġsentence Ġtransformation Ġproblem Ġwith Ġpred efined Ġrules Ġor Ġtemplates ĠB IB REF 4 , ĠB IB REF 0 . ĠWith Ġthe Ġrise Ġof Ġneural Ġmodels , ĠDu 2017 Learning TA Ġpropose Ġto Ġmodel Ġthis Ġtask Ġunder Ġthe Ġsequence - to - sequence Ġ( Se q 2 Se q ) Ġlearning Ġframework ĠB IB REF 5 Ġwith Ġattention Ġmechanism ĠB IB REF 6 . ĠHowever , Ġquestion Ġgeneration Ġis Ġa Ġone - to - many Ġsequence Ġgeneration Ġproblem , Ġi . e ., Ġseveral Ġaspects Ġcan Ġbe Ġasked Ġgiven Ġa Ġsentence . ĠZhou 2017 Ne ural Q G Ġpropose Ġthe Ġanswer - aware Ġquestion Ġgeneration Ġsetting Ġwhich Ġassumes Ġthe Ġanswer , Ġa Ġcontiguous Ġspan Ġinside Ġthe Ġinput Ġsentence , Ġis Ġalready Ġknown Ġbefore Ġquestion Ġgeneration . ĠTo Ġcapture Ġanswer - relevant Ġwords Ġin Ġthe Ġsentence , Ġthey Ġadopt Ġa ĠB IO Ġtagging Ġscheme Ġto Ġincorporate Ġthe Ġanswer Ġposition Ġembed ding Ġin ĠSe q 2 Se q Ġlearning . ĠFurthermore , ĠSun 2018 Answer focused AP Ġpropose Ġthat Ġtokens Ġclose Ġto Ġthe Ġanswer Ġfragments Ġare Ġmore Ġlikely Ġto Ġbe Ġanswer - relevant . ĠTherefore , Ġthey Ġexplicitly Ġencode Ġthe Ġrelative Ġdistance Ġbetween Ġsentence Ġwords Ġand Ġthe Ġanswer Ġvia Ġposition Ġembed ding Ġand Ġposition - aware Ġattention . </s> Although Ġexisting Ġproximity - based Ġanswer - aware Ġapproaches Ġachieve Ġreasonable Ġperformance , Ġwe Ġargue Ġthat Ġsuch Ġintuition Ġmay Ġnot Ġapply Ġto Ġall Ġcases Ġespecially Ġfor Ġsentences Ġwith Ġcomplex Ġstructure . ĠFor Ġexample , ĠFigure ĠFIG REF 1 Ġshows Ġsuch Ġan Ġexample Ġwhere Ġthose Ġapproaches Ġfail . ĠThis Ġsentence Ġcontains Ġa Ġfew Ġfacts Ġand Ġdue Ġto Ġthe Ġparent hesis Ġ( i . e . ĠâĢ ľ the Ġarea 's Ġcold est Ġmonth âĢ Ŀ ), Ġsome Ġfacts Ġintertw ine : ĠâĢ ľ The Ġdaily Ġmean Ġtemperature Ġin ĠJanuary Ġis Ġ0 . 3 $ ^ \ circ Ġ$ C âĢ Ŀ Ġand ĠâĢ ľ January Ġis Ġthe Ġarea 's Ġcold est Ġmonth âĢ Ŀ . ĠFrom Ġthe Ġquestion Ġgenerated Ġby Ġa Ġproximity - based Ġanswer - aware Ġbaseline , Ġwe Ġfind Ġthat Ġit Ġwrongly Ġuses Ġthe Ġword ĠâĢ ľ cold est âĢ Ŀ Ġbut Ġmisses Ġthe Ġcorrect Ġword ĠâĢ ľ mean âĢ Ŀ Ġbecause ĠâĢ ľ cold est âĢ Ŀ Ġhas Ġa Ġshorter Ġdistance Ġto Ġthe Ġanswer ĠâĢ ľ 0 . 3 $ ^ \ circ Ġ$ C âĢ Ŀ . </s> In Ġsummary , Ġtheir Ġintuition Ġthat ĠâĢ ľ the Ġneighboring Ġwords Ġof Ġthe Ġanswer Ġare Ġmore Ġlikely Ġto Ġbe Ġanswer - relevant Ġand Ġhave Ġa Ġhigher Ġchance Ġto Ġbe Ġused Ġin Ġthe Ġquestion âĢ Ŀ Ġis Ġnot Ġreliable . ĠTo Ġquant itatively Ġshow Ġthis Ġdrawback Ġof Ġthese Ġmodels , Ġwe Ġimplement Ġthe Ġapproach Ġproposed Ġby ĠSun 2018 Answer focused AP Ġand Ġanalyze Ġits Ġperformance Ġunder Ġdifferent Ġrelative Ġdistances Ġbetween Ġthe Ġanswer Ġand Ġother Ġnon - stop Ġsentence Ġwords Ġthat Ġalso Ġappear Ġin Ġthe Ġground Ġtruth Ġquestion . ĠThe Ġresults Ġare Ġshown Ġin ĠTable ĠT AB REF 2 . ĠWe Ġfind Ġthat Ġthe Ġperformance Ġdrops Ġat Ġmost Ġ36 % Ġwhen Ġthe Ġrelative Ġdistance Ġincreases Ġfrom ĠâĢ ľ $ 0 \ sim Ġ10 $ âĢ Ŀ Ġto ĠâĢ ľ $ > 10 $ âĢ Ŀ . ĠIn Ġother Ġwords , Ġwhen Ġthe Ġuseful Ġcontext Ġis Ġlocated Ġfar Ġaway Ġfrom Ġthe Ġanswer , Ġcurrent Ġproximity - based Ġanswer - aware Ġapproaches Ġwill Ġbecome Ġless Ġeffective , Ġsince Ġthey Ġoverly Ġemphasize Ġneighboring Ġwords Ġof Ġthe Ġanswer . </s> To Ġaddress Ġthis Ġissue , Ġwe Ġextract Ġthe Ġstructured Ġanswer - relevant Ġrelations Ġfrom Ġsentences Ġand Ġpropose Ġa Ġmethod Ġto Ġjointly Ġmodel Ġsuch Ġstructured Ġrelation Ġand Ġthe Ġun struct ured Ġsentence Ġfor Ġquestion Ġgeneration . ĠThe Ġstructured Ġanswer - relevant Ġrelation Ġis Ġlikely Ġto Ġbe Ġto Ġthe Ġpoint Ġcontext Ġand Ġthus Ġcan Ġhelp Ġkeep Ġthe Ġgenerated Ġquestion Ġto Ġthe Ġpoint . ĠFor Ġexample , ĠFigure ĠFIG REF 1 Ġshows Ġour Ġframework Ġcan Ġextract Ġthe Ġright Ġanswer - relevant Ġrelation Ġ( âĢ ľ The Ġdaily Ġmean Ġtemperature Ġin ĠJanuary âĢ Ŀ , ĠâĢ ľ is âĢ Ŀ , ĠâĢ ľ 32 . 6 $ ^ \ circ Ġ$ F Ġ( 0 . 3 $ ^ \ circ Ġ$ C ) âĢ Ŀ ) Ġamong Ġmultiple Ġfacts . ĠWith Ġthe Ġhelp Ġof Ġsuch Ġstructured Ġinformation , Ġour Ġmodel Ġis Ġless Ġlikely Ġto Ġbe Ġconfused Ġby Ġsentences Ġwith Ġa Ġcomplex Ġstructure . ĠSpecifically , Ġwe Ġfirst ly Ġextract Ġmultiple Ġrelations Ġwith Ġan Ġoff - the - she lf ĠOpen ĠInformation ĠExt raction Ġ( Open IE ) Ġtool box ĠB IB REF 7 , Ġthen Ġwe Ġselect Ġthe Ġrelation Ġthat Ġis Ġmost Ġrelevant Ġto Ġthe Ġanswer Ġwith Ġcarefully Ġdesigned Ġhe uristic Ġrules . </s> Nevertheless , Ġit Ġis Ġchallenging Ġto Ġtrain Ġa Ġmodel Ġto Ġeffectively Ġutilize Ġboth Ġthe Ġun struct ured Ġsentence Ġand Ġthe Ġstructured Ġanswer - relevant Ġrelation Ġbecause Ġboth Ġof Ġthem Ġcould Ġbe Ġnoisy : Ġthe Ġun struct ured Ġsentence Ġmay Ġcontain Ġmultiple Ġfacts Ġwhich Ġare Ġirrelevant Ġto Ġthe Ġtarget Ġquestion , Ġwhile Ġthe Ġlimitation Ġof Ġthe ĠOpen IE Ġtool Ġmay Ġproduce Ġless Ġaccurate Ġextracted Ġrelations . ĠTo Ġexplore Ġtheir Ġadvantages Ġsimultaneously Ġand Ġavoid Ġthe Ġdrawbacks , Ġwe Ġdesign Ġa Ġg ated Ġattention Ġmechanism Ġand Ġa Ġdual Ġcopy Ġmechanism Ġbased Ġon Ġthe Ġenc oder - dec oder Ġframework , Ġwhere Ġthe Ġformer Ġlearns Ġto Ġcontrol Ġthe Ġinformation Ġflow Ġbetween Ġthe Ġun struct ured Ġand Ġstructured Ġinputs , Ġwhile Ġthe Ġlatter Ġlearns Ġto Ġcopy Ġwords Ġfrom Ġtwo Ġsources Ġto Ġmaintain Ġthe Ġinform at iveness Ġand Ġfaith fulness Ġof Ġgenerated Ġquestions . </s> In Ġthe Ġevaluations Ġon Ġthe ĠS Qu AD Ġdataset , Ġour Ġsystem Ġachieves Ġsignificant Ġand Ġconsistent Ġimprovement Ġas Ġcompared Ġto Ġall Ġbaseline Ġmethods . ĠIn Ġparticular , Ġwe Ġdemonstrate Ġthat Ġthe Ġimprovement Ġis Ġmore Ġsignificant Ġwith Ġa Ġlarger Ġrelative Ġdistance Ġbetween Ġthe Ġanswer Ġand Ġother Ġnon - stop Ġsentence Ġwords Ġthat Ġalso Ġappear Ġin Ġthe Ġground Ġtruth Ġquestion . ĠFurthermore , Ġour Ġmodel Ġis Ġcapable Ġof Ġgenerating Ġdiverse Ġquestions Ġfor Ġa Ġsingle Ġsentence - answer Ġpair Ġwhere Ġthe Ġsentence Ġconve ys Ġmultiple Ġrelations Ġof Ġits Ġanswer Ġfragment . </s> Fram ework ĠDescription </s> In Ġthis Ġsection , Ġwe Ġfirst Ġintroduce Ġthe Ġtask Ġdefinition Ġand Ġour Ġprotocol Ġto Ġextract Ġstructured Ġanswer - relevant Ġrelations . ĠThen Ġwe Ġformal ize Ġthe Ġtask Ġunder Ġthe Ġenc oder - dec oder Ġframework Ġwith Ġg ated Ġattention Ġand Ġdual Ġcopy Ġmechanism . </s> Fram ework ĠDescription Ġ: :: ĠProblem ĠDefinition </s> We Ġformal ize Ġour Ġtask Ġas Ġan Ġanswer - aware ĠQuestion ĠGeneration Ġ( Q G ) Ġproblem ĠB IB REF 8 , Ġwhich Ġassumes Ġanswer Ġphrases Ġare Ġgiven Ġbefore Ġgenerating Ġquestions . ĠMoreover , Ġanswer Ġphrases Ġare Ġshown Ġas Ġtext Ġfragments Ġin Ġpassages . ĠForm ally , Ġgiven Ġthe Ġsentence Ġ$ S $, Ġthe Ġanswer Ġ$ A $, Ġand Ġthe Ġanswer - relevant Ġrelation Ġ$ M $, Ġthe Ġtask Ġof ĠQ G Ġaims Ġto Ġfind Ġthe Ġbest Ġquestion Ġ$\ over line { Q } $ Ġsuch Ġthat , </s> where Ġ$ A $ Ġis Ġa Ġcontiguous Ġspan Ġinside Ġ$ S $. </s> Fram ework ĠDescription Ġ: :: ĠAnswer - relevant ĠRel ation ĠExt raction </s> We Ġutilize Ġan Ġoff - the - she lf Ġtool box Ġof ĠOpen IE Ġto Ġthe Ġderive Ġstructured Ġanswer - relevant Ġrelations Ġfrom Ġsentences Ġas Ġto Ġthe Ġpoint Ġcontexts . ĠRelations Ġextracted Ġby ĠOpen IE Ġcan Ġbe Ġrepresented Ġeither Ġin Ġa Ġtriple Ġformat Ġor Ġin Ġan Ġn - ary Ġformat Ġwith Ġseveral Ġsecondary Ġarguments , Ġand Ġwe Ġemploy Ġthe Ġlatter Ġto Ġkeep Ġthe Ġextract ions Ġas Ġinformative Ġas Ġpossible Ġand Ġavoid Ġextracting Ġtoo Ġmany Ġsimilar Ġrelations Ġin Ġdifferent Ġgran ular ities Ġfrom Ġone Ġsentence . ĠWe Ġjoin Ġall Ġarguments Ġin Ġthe Ġextracted Ġn - ary Ġrelation Ġinto Ġa Ġsequence Ġas Ġour Ġto Ġthe Ġpoint Ġcontext . ĠFigure ĠFIG REF 5 Ġshows Ġn - ary Ġrelations Ġextracted Ġfrom ĠOpen IE . ĠAs Ġwe Ġcan Ġsee , ĠOpen IE Ġextracts Ġmultiple Ġrelations Ġfor Ġcomplex Ġsentences . ĠHere Ġwe Ġselect Ġthe Ġmost Ġinformative Ġrelation Ġaccording Ġto Ġthree Ġcriteria Ġin Ġthe Ġorder Ġof Ġdescending Ġimportance : Ġ( 1 ) Ġhaving Ġthe Ġmaximal Ġnumber Ġof Ġoverl apped Ġtokens Ġbetween Ġthe Ġanswer Ġand Ġthe Ġrelation ; Ġ( 2 ) Ġbeing Ġassigned Ġthe Ġhighest Ġconfidence Ġscore Ġby ĠOpen IE ; Ġ( 3 ) Ġcontaining Ġmaximum Ġnon - stop Ġwords . ĠAs Ġshown Ġin ĠFigure ĠFIG REF 5 , Ġour Ġcriteria Ġcan Ġselect Ġanswer - relevant Ġrelations Ġ( w aved Ġin ĠFigure ĠFIG REF 5 ), Ġwhich Ġis Ġespecially Ġuseful Ġfor Ġsentences Ġwith Ġextr aneous Ġinformation . ĠIn Ġrare Ġcases , ĠOpen IE Ġcannot Ġextract Ġany Ġrelation , Ġwe Ġtreat Ġthe Ġsentence Ġitself Ġas Ġthe Ġto Ġthe Ġpoint Ġcontext . </s> Table ĠT AB REF 8 Ġshows Ġsome Ġstatistics Ġto Ġverify Ġthe Ġintuition Ġthat Ġthe Ġextracted Ġrelations Ġcan Ġserve Ġas Ġmore Ġto Ġthe Ġpoint Ġcontext . ĠWe Ġfind Ġthat Ġthe Ġtokens Ġin Ġrelations Ġare Ġ61 % Ġmore Ġlikely Ġto Ġbe Ġused Ġin Ġthe Ġtarget Ġquestion Ġthan Ġthe Ġtokens Ġin Ġsentences , Ġand Ġthus Ġthey Ġare Ġmore Ġto Ġthe Ġpoint . ĠOn Ġthe Ġother Ġhand , Ġon Ġaverage Ġthe Ġsentences Ġcontain Ġone Ġmore Ġquestion Ġtoken Ġthan Ġthe Ġrelations Ġ( 1 . 86 Ġv . s . Ġ2 . 87 ). ĠTherefore , Ġit Ġis Ġstill Ġnecessary Ġto Ġtake Ġthe Ġoriginal Ġsentence Ġinto Ġaccount Ġto Ġgenerate Ġa Ġmore Ġaccurate Ġquestion . </s> Fram ework ĠDescription Ġ: :: ĠOur ĠProp osed ĠModel Ġ: :: ĠOverview . </s> As Ġshown Ġin ĠFigure ĠFIG REF 10 , Ġour Ġframework Ġconsists Ġoff our Ġcomponents Ġ( 1 ) ĠSent ence ĠEnc oder Ġand ĠRel ation ĠEnc oder , Ġ( 2 ) ĠDecoder , Ġ( 3 ) ĠG ated ĠAttention ĠMechan ism Ġand Ġ( 4 ) ĠDual ĠCopy ĠMechan ism . ĠThe Ġsentence Ġenc oder Ġand Ġrelation Ġenc oder Ġencode Ġthe Ġun struct ured Ġsentence Ġand Ġthe Ġstructured Ġanswer - relevant Ġrelation , Ġrespectively . ĠTo Ġselect Ġand Ġcombine Ġthe Ġsource Ġinformation Ġfrom Ġthe Ġtwo Ġenc od ers , Ġa Ġg ated Ġattention Ġmechanism Ġis Ġemployed Ġto Ġjointly Ġattend Ġboth Ġcontextual ized Ġinformation Ġsources , Ġand Ġa Ġdual Ġcopy Ġmechanism Ġcopies Ġwords Ġfrom Ġeither Ġthe Ġsentence Ġor Ġthe Ġrelation . </s> Fram ework ĠDescription Ġ: :: ĠOur ĠProp osed ĠModel Ġ: :: ĠAnswer - aware ĠEnc oder . </s> We Ġemploy Ġtwo Ġenc od ers Ġto Ġintegrate Ġinformation Ġfrom Ġthe Ġun struct ured Ġsentence Ġ$ S $ Ġand Ġthe Ġanswer - relevant Ġrelation Ġ$ M $ Ġseparately . ĠSent ence Ġenc oder Ġtakes Ġin Ġfeature - en riched Ġembed d ings Ġincluding Ġword Ġembed d ings Ġ$\ math bf Ġ{ w } $, Ġlinguistic Ġembed d ings Ġ$\ math bf Ġ{ l } $ Ġand Ġanswer Ġposition Ġembed d ings Ġ$\ math bf Ġ{ a } $. ĠWe Ġfollow ĠB IB REF 9 Ġto Ġtransform ĠPOS Ġand ĠN ER Ġtags Ġinto Ġcontinuous Ġrepresentation Ġ($ \ math bf Ġ{ l } ^ p $ Ġand Ġ$\ math bf Ġ{ l } ^ n $ ) Ġand Ġadopt Ġa ĠB IO Ġlab elling Ġscheme Ġto Ġderive Ġthe Ġanswer Ġposition Ġembed ding Ġ( B : Ġthe Ġfirst Ġtoken Ġof Ġthe Ġanswer , ĠI : Ġtokens Ġwithin Ġthe Ġanswer Ġfragment Ġexcept Ġthe Ġfirst Ġone , ĠO : Ġtokens Ġoutside Ġof Ġthe Ġanswer Ġfragment ). ĠFor Ġeach Ġword Ġ$ w _ i $ Ġin Ġthe Ġsentence Ġ$ S $, Ġwe Ġsimply Ġconc aten ate Ġall Ġfeatures Ġas Ġinput : Ġ$\ math bf Ġ{ x } _ i ^ s = Ġ[ \ math bf Ġ{ w } _ i ; Ġ\ math bf Ġ{ l } ^ p _ i ; Ġ\ math bf Ġ{ l } ^ n _ i ; Ġ\ math bf Ġ{ a } _ i ] $. ĠHere Ġ$ [ \ math bf Ġ{ a }; \ math bf Ġ{ b } ] $ Ġdenotes Ġthe Ġconc aten ation Ġof Ġvectors Ġ$\ math bf Ġ{ a } $ Ġand Ġ$\ math bf Ġ{ b } $. </s> We Ġuse Ġbid irection al ĠL ST Ms Ġto Ġencode Ġthe Ġsentence Ġ$( \ math bf Ġ{ x } _ 1 ^ s , Ġ\ math bf Ġ{ x } _ 2 ^ s , Ġ... , Ġ\ math bf Ġ{ x } _ n ^ s ) $ Ġto Ġget Ġa Ġcontextual ized Ġrepresentation Ġfor Ġeach Ġtoken : </s> where Ġ$\ over right arrow {\ math bf Ġ{ h }} ^{ s } _ i $ Ġand Ġ$\ over left arrow {\ math bf Ġ{ h }} ^{ s } _ i $ Ġare Ġthe Ġhidden Ġstates Ġat Ġthe Ġ$ i $ - th Ġtime Ġstep Ġof Ġthe Ġforward Ġand Ġthe Ġbackward ĠL ST Ms . ĠThe Ġoutput Ġstate Ġof Ġthe Ġsentence Ġenc oder Ġis Ġthe Ġconc aten ation Ġof Ġforward Ġand Ġbackward Ġhidden Ġstates : Ġ$\ math bf Ġ{ h } ^{ s } _ i =[ \ over right arrow {\ math bf Ġ{ h }} ^{ s } _ i ; \ over left arrow {\ math bf Ġ{ h }} ^{ s } _ i ] $. ĠThe Ġcontextual ized Ġrepresentation Ġof Ġthe Ġsentence Ġis Ġ$( \ math bf Ġ{ h } ^{ s } _ 1 , Ġ\ math bf Ġ{ h } ^{ s } _ 2 , Ġ... , Ġ\ math bf Ġ{ h } ^{ s } _ n ) $. </s> For Ġthe Ġrelation Ġenc oder , Ġwe Ġfirst ly Ġjoin Ġall Ġitems Ġin Ġthe Ġn - ary Ġrelation Ġ$ M $ Ġinto Ġa Ġsequence . ĠThen Ġwe Ġonly Ġtake Ġanswer Ġposition Ġembed ding Ġas Ġan Ġextra Ġfeature Ġfor Ġthe Ġsequence : Ġ$\ math bf Ġ{ x } _ i ^ m = Ġ[ \ math bf Ġ{ w } _ i ; Ġ\ math bf Ġ{ a } _ i ] $. ĠSimilarly , Ġwe Ġtake Ġanother Ġbid irection al ĠL ST Ms Ġto Ġencode Ġthe Ġrelation Ġsequence Ġand Ġderive Ġthe Ġcorresponding Ġcontextual ized Ġrepresentation Ġ$( \ math bf Ġ{ h } ^{ m } _ 1 , Ġ\ math bf Ġ{ h } ^{ m } _ 2 , Ġ... , Ġ\ math bf Ġ{ h } ^{ m } _ n ) $. </s> Fram ework ĠDescription Ġ: :: ĠOur ĠProp osed ĠModel Ġ: :: ĠDecoder . </s> We Ġuse Ġan ĠL ST M Ġas Ġthe Ġdec oder Ġto Ġgenerate Ġthe Ġquestion . ĠThe Ġdec oder Ġpredicts Ġthe Ġword Ġprobability Ġdistribution Ġat Ġeach Ġdecoding Ġtim est ep Ġto Ġgenerate Ġthe Ġquestion . ĠAt Ġthe Ġt - th Ġtim est ep , Ġit Ġreads Ġthe Ġword Ġembed ding Ġ$\ math bf Ġ{ w } _{ t } $ Ġand Ġthe Ġhidden Ġstate Ġ$\ math bf Ġ{ u } _{ t - 1 } $ Ġof Ġthe Ġprevious Ġtim est ep Ġto Ġgenerate Ġthe Ġcurrent Ġhidden Ġstate : </s> Fram ework ĠDescription Ġ: :: ĠOur ĠProp osed ĠModel Ġ: :: ĠG ated ĠAttention ĠMechan ism . </s> We Ġdesign Ġa Ġg ated Ġattention Ġmechanism Ġto Ġjointly Ġattend Ġthe Ġsentence Ġrepresentation Ġand Ġthe Ġrelation Ġrepresentation . ĠFor Ġsentence Ġrepresentation Ġ$( \ math bf Ġ{ h } ^{ s } _ 1 , Ġ\ math bf Ġ{ h } ^{ s } _ 2 , Ġ... , Ġ\ math bf Ġ{ h } ^{ s } _ n ) $, Ġwe Ġemploy Ġthe ĠLu ong 2015 Effective AT 's Ġattention Ġmechanism Ġto Ġobtain Ġthe Ġsentence Ġcontext Ġvector Ġ$\ math bf Ġ{ c } ^ s _ t $, </s> where Ġ$\ math bf Ġ{ W } _ a $ Ġis Ġa Ġtrain able Ġweight . ĠSimilarly , Ġwe Ġobtain Ġthe Ġvector Ġ$\ math bf Ġ{ c } ^ m _ t $ Ġfrom Ġthe Ġrelation Ġrepresentation Ġ$( \ math bf Ġ{ h } ^{ m } _ 1 , Ġ\ math bf Ġ{ h } ^{ m } _ 2 , Ġ... , Ġ\ math bf Ġ{ h } ^{ m } _ n ) $. ĠTo Ġjointly Ġmodel Ġthe Ġsentence Ġand Ġthe Ġrelation , Ġa Ġg ating Ġmechanism Ġis Ġdesigned Ġto Ġcontrol Ġthe Ġinformation Ġflow Ġfrom Ġtwo Ġsources : </s> where Ġ$\ od ot Ġ$ Ġrepresents Ġelement - wise Ġdot Ġproduction Ġand Ġ$\ math bf Ġ{ W } _ g , Ġ\ math bf Ġ{ W } _ h $ Ġare Ġtrain able Ġweights . ĠFinally , Ġthe Ġpredicted Ġprobability Ġdistribution Ġover Ġthe Ġvocabulary Ġ$ V $ Ġis Ġcomputed Ġas : </s> where Ġ$\ math bf Ġ{ W } _ V $ Ġand Ġ$\ math bf Ġ{ b } _ V $ Ġare Ġparameters . </s> Fram ework ĠDescription Ġ: :: ĠOur ĠProp osed ĠModel Ġ: :: ĠDual ĠCopy ĠMechan ism . </s> To Ġdeal Ġwith Ġthe Ġrare Ġand Ġunknown Ġwords , Ġthe Ġdec oder Ġapplies Ġthe Ġpointing Ġmethod ĠB IB REF 10 , ĠB IB REF 11 , ĠB IB REF 12 Ġto Ġallow Ġcopying Ġa Ġtoken Ġfrom Ġthe Ġinput Ġsentence Ġat Ġthe Ġ$ t $ - th Ġdecoding Ġstep . ĠWe Ġreuse Ġthe Ġattention Ġscore Ġ$\ math bf Ġ{\ alpha Ġ} _{ t } ^ s $ Ġand Ġ$\ math bf Ġ{\ alpha Ġ} _{ t } ^ m $ Ġto Ġderive Ġthe Ġcopy Ġprobability Ġover Ġtwo Ġsource Ġinputs : </s> Different Ġfrom Ġthe Ġstandard Ġpointing Ġmethod , Ġwe Ġdesign Ġa Ġdual Ġcopy Ġmechanism Ġto Ġcopy Ġfrom Ġtwo Ġsources Ġwith Ġtwo Ġgates . ĠThe Ġfirst Ġgate Ġis Ġdesigned Ġfor Ġdetermining Ġcopy Ġtokens Ġfrom Ġtwo Ġsources Ġof Ġinputs Ġor Ġgenerate Ġnext Ġword Ġfrom Ġ$ P _ V $, Ġwhich Ġis Ġcomputed Ġas Ġ$ g ^ v _ t Ġ= Ġ\ text { s igm oid } (\ math bf Ġ{ w } ^ v _ g Ġ\ t ilde {\ math bf Ġ{ h }} _ t Ġ+ Ġb ^ v _ g ) $. ĠThe Ġsecond Ġgate Ġtakes Ġcharge Ġof Ġselecting Ġthe Ġsource Ġ( sent ence Ġor Ġrelation ) Ġto Ġcopy Ġfrom , Ġwhich Ġis Ġcomputed Ġas Ġ$ g ^ c _ t Ġ= Ġ\ text { s igm oid } (\ math bf Ġ{ w } ^ c _ g Ġ[ \ math bf Ġ{ c } _ t ^ s ; \ math bf Ġ{ c } _ t ^ m ] Ġ+ Ġb ^ c _ g ) $. ĠFinally , Ġwe Ġcombine Ġall Ġprobabilities Ġ$ P _ V $, Ġ$ P _ S $ Ġand Ġ$ P _ M $ Ġthrough Ġtwo Ġsoft Ġgates Ġ$ g ^ v _ t $ Ġand Ġ$ g ^ c _ t $. ĠThe Ġprobability Ġof Ġpredicting Ġ$ w $ Ġas Ġthe Ġ$ t $ - th Ġtoken Ġof Ġthe Ġquestion Ġis : </s> Fram ework ĠDescription Ġ: :: ĠOur ĠProp osed ĠModel Ġ: :: ĠTraining Ġand ĠIn ference . </s> Given Ġthe Ġanswer Ġ$ A $, Ġsentence Ġ$ S $ Ġand Ġrelation Ġ$ M $, Ġthe Ġtraining Ġobjective Ġis Ġto Ġminimize Ġthe Ġnegative Ġlog - like lihood Ġwith Ġregard Ġto Ġall Ġparameters : </s> where Ġ$\ math cal Ġ{\ l brace Ġ} Q \ r brace Ġ$ Ġis Ġthe Ġset Ġof Ġall Ġtraining Ġinstances , Ġ$\ the ta Ġ$ Ġdenotes Ġmodel Ġparameters Ġand Ġ$\ text { log } ĠP ( Q | A , S , M ; \ the ta Ġ) $ Ġis Ġthe Ġconditional Ġlog - like lihood Ġof Ġ$ Q $. </s> In Ġtesting , Ġour Ġmodel Ġtargets Ġto Ġgenerate Ġa Ġquestion Ġ$ Q $ Ġby Ġmaximizing : </s> Exper imental ĠSetting Ġ: :: ĠDat as et Ġ& ĠMet rics </s> We Ġconduct Ġexperiments Ġon Ġthe ĠS Qu AD Ġdataset ĠB IB REF 3 . ĠIt Ġcontains Ġ5 36 ĠWikipedia Ġarticles Ġand Ġ100 k Ġcrowd - s ourced Ġquestion - answer Ġpairs . ĠThe Ġquestions Ġare Ġwritten Ġby Ġcrowd - workers Ġand Ġthe Ġanswers Ġare Ġspans Ġof Ġtokens Ġin Ġthe Ġarticles . ĠWe Ġemploy Ġtwo Ġdifferent Ġdata Ġsplits Ġby Ġfollowing ĠZhou 2017 Ne ural Q G Ġand ĠDu 2017 Learning TA Ġ. ĠIn ĠZhou 2017 Ne ural Q G , Ġthe Ġoriginal ĠS Qu AD Ġdevelopment Ġset Ġis Ġevenly Ġdivided Ġinto Ġdev Ġand Ġtest Ġsets , Ġwhile ĠDu 2017 Learning TA Ġtreats ĠS Qu AD Ġdevelopment Ġset Ġas Ġits Ġdevelopment Ġset Ġand Ġsplits Ġoriginal ĠS Qu AD Ġtraining Ġset Ġinto Ġa Ġtraining Ġset Ġand Ġa Ġtest Ġset . ĠWe Ġalso Ġfilter Ġout Ġquestions Ġwhich Ġdo Ġnot Ġhave Ġany Ġoverl apped Ġnon - stop Ġwords Ġwith Ġthe Ġcorresponding Ġsentences Ġand Ġperform Ġsome Ġpre processing Ġsteps , Ġsuch Ġas Ġtoken ization Ġand Ġsentence Ġsplitting . ĠThe Ġdata Ġstatistics Ġare Ġgiven Ġin ĠTable ĠT AB REF 27 . </s> We Ġevaluate Ġwith Ġall Ġcommonly - used Ġmetrics Ġin Ġquestion Ġgeneration ĠB IB REF 13 : ĠB LE U - 1 Ġ( B 1 ), ĠB LE U - 2 Ġ( B 2 ), ĠB LE U - 3 Ġ( B 3 ), ĠB LE U - 4 Ġ( B 4 ) ĠB IB REF 17 , ĠMET E OR Ġ( MET ) ĠB IB REF 18 Ġand ĠR OU GE - L Ġ( R - L ) ĠB IB REF 19 . ĠWe Ġuse Ġthe Ġevaluation Ġscript Ġreleased Ġby ĠChen 2015 Microsoft CC . </s> Exper imental ĠSetting Ġ: :: ĠBas eline ĠModels </s> We Ġcompare Ġwith Ġthe Ġfollowing Ġmodels . </s> [ left margin = * ] </s> s 2 s ĠB IB REF 13 Ġproposes Ġan Ġattention - based Ġsequence - to - sequence Ġneural Ġnetwork Ġfor Ġquestion Ġgeneration . </s> N Q G ++ ĠB IB REF 9 Ġtakes Ġthe Ġanswer Ġposition Ġfeature Ġand Ġlinguistic Ġfeatures Ġinto Ġconsideration Ġand Ġequ ips Ġthe ĠSe q 2 Se q Ġmodel Ġwith Ġcopy Ġmechanism . </s> M 2 S + cp ĠB IB REF 14 Ġconducts Ġmulti - pers pect ive Ġmatching Ġbetween Ġthe Ġanswer Ġand Ġthe Ġsentence Ġto Ġderive Ġan Ġanswer - aware Ġsentence Ġrepresentation Ġfor Ġquestion Ġgeneration . </s> s 2 s + MP + G SA ĠB IB REF 8 Ġintroduces Ġa Ġg ated Ġself - att ention Ġinto Ġthe Ġenc oder Ġand Ġa Ġmax out Ġpointer Ġmechanism Ġinto Ġthe Ġdec oder . ĠWe Ġreport Ġtheir Ġsentence - level Ġresults Ġfor Ġa Ġfair Ġcomparison . </s> Hy brid ĠB IB REF 15 Ġis Ġa Ġhybrid Ġmodel Ġwhich Ġconsiders Ġthe Ġanswer Ġembed ding Ġfor Ġthe Ġquestion Ġword Ġgeneration Ġand Ġthe Ġposition Ġof Ġcontext Ġwords Ġfor Ġmodeling Ġthe Ġrelative Ġdistance Ġbetween Ġthe Ġcontext Ġwords Ġand Ġthe Ġanswer . </s> AS s 2 s ĠB IB REF 16 Ġreplaces Ġthe Ġanswer Ġin Ġthe Ġsentence Ġwith Ġa Ġspecial Ġtoken Ġto Ġavoid Ġits Ġappearance Ġin Ġthe Ġgenerated Ġquestions . </s> Exper imental ĠSetting Ġ: :: ĠImplementation ĠDetails </s> We Ġtake Ġthe Ġmost Ġfrequent Ġ20 k Ġwords Ġas Ġour Ġvocabulary Ġand Ġuse Ġthe ĠGl o Ve Ġword Ġembed d ings ĠB IB REF 20 Ġfor Ġinitialization . ĠThe Ġembed ding Ġdimensions Ġfor ĠPOS , ĠN ER , Ġanswer Ġposition Ġare Ġset Ġto Ġ20 . ĠWe Ġuse Ġtwo - layer ĠL ST Ms Ġin Ġboth Ġenc oder Ġand Ġdec oder , Ġand Ġthe ĠL ST Ms Ġhidden Ġunit Ġsize Ġis Ġset Ġto Ġ600 . </s> We Ġuse Ġdrop out ĠB IB REF 21 Ġwith Ġthe Ġprobability Ġ$ p = 0 . 3 $. ĠAll Ġtrain able Ġparameters , Ġexcept Ġword Ġembed d ings , Ġare Ġrandomly Ġinitialized Ġwith Ġthe ĠXavier Ġuniform Ġin Ġ$( - 0 . 1 , Ġ0 . 1 ) $ ĠB IB REF 22 . ĠFor Ġoptimization Ġin Ġthe Ġtraining , Ġwe Ġuse ĠSG D Ġas Ġthe Ġoptim izer Ġwith Ġa Ġmin ib atch Ġsize Ġof Ġ64 Ġand Ġan Ġinitial Ġlearning Ġrate Ġof Ġ1 . 0 . ĠWe Ġtrain Ġthe Ġmodel Ġfor Ġ15 Ġepoch s Ġand Ġstart Ġhal ving Ġthe Ġlearning Ġrate Ġafter Ġthe Ġ8 th Ġepoch . ĠWe Ġset Ġthe Ġgradient Ġnorm Ġupper Ġbound Ġto Ġ3 Ġduring Ġthe Ġtraining . </s> We Ġadopt Ġthe Ġteacher - forcing Ġfor Ġthe Ġtraining . ĠIn Ġthe Ġtesting , Ġwe Ġselect Ġthe Ġmodel Ġwith Ġthe Ġlowest Ġperplex ity Ġand Ġbeam Ġsearch Ġwith Ġsize Ġ3 Ġis Ġemployed Ġfor Ġgenerating Ġquestions . ĠAll Ġhyper - param eters Ġand Ġmodels Ġare Ġselected Ġon Ġthe Ġvalidation Ġdataset . </s> Results Ġand ĠAnalysis Ġ: :: ĠMain ĠResults </s> Table ĠT AB REF 30 Ġshows Ġautomatic Ġevaluation Ġresults Ġfor Ġour Ġmodel Ġand Ġbas elines Ġ( cop ied Ġfrom Ġtheir Ġpapers ). ĠOur Ġproposed Ġmodel Ġwhich Ġcombines Ġstructured Ġanswer - relevant Ġrelations Ġand Ġun struct ured Ġsentences Ġachieves Ġsignificant Ġimprovements Ġover Ġproximity - based Ġanswer - aware Ġmodels ĠB IB REF 9 , ĠB IB REF 15 Ġon Ġboth Ġdataset Ġsplits . ĠPresumably , Ġour Ġstructured Ġanswer - relevant Ġrelation Ġis Ġa Ġgeneral ization Ġof Ġthe Ġcontext Ġexplored Ġby Ġthe Ġproximity - based Ġmethods Ġbecause Ġthey Ġcan Ġonly Ġcapture Ġshort Ġdependencies Ġaround Ġanswer Ġfragments Ġwhile Ġour Ġextract ions Ġcan Ġcapture Ġboth Ġshort Ġand Ġlong Ġdependencies Ġgiven Ġthe Ġanswer Ġfragments . ĠMoreover , Ġour Ġproposed Ġframework Ġis Ġa Ġgeneral Ġone Ġto Ġjointly Ġleverage Ġstructured Ġrelations Ġand Ġun struct ured Ġsentences . ĠAll Ġcompared Ġbaseline Ġmodels Ġwhich Ġonly Ġconsider Ġun struct ured Ġsentences Ġcan Ġbe Ġfurther Ġenhanced Ġunder Ġour Ġframework . </s> Rec all Ġthat Ġexisting Ġproximity - based Ġanswer - aware Ġmodels Ġperform Ġpoorly Ġwhen Ġthe Ġdistance Ġbetween Ġthe Ġanswer Ġfragment Ġand Ġother Ġnon - stop Ġsentence Ġwords Ġthat Ġalso Ġappear Ġin Ġthe Ġground Ġtruth Ġquestion Ġis Ġlarge Ġ( Table ĠT AB REF 2 ). ĠHere Ġwe Ġinvestigate Ġwhether Ġour Ġproposed Ġmodel Ġusing Ġthe Ġstructured Ġanswer - relevant Ġrelations Ġcan Ġalleviate Ġthis Ġissue Ġor Ġnot , Ġby Ġconducting Ġexperiments Ġfor Ġour Ġmodel Ġunder Ġthe Ġsame Ġsetting Ġas Ġin ĠTable ĠT AB REF 2 . ĠThe Ġbroken - down Ġperformances Ġby Ġdifferent Ġrelative Ġdistances Ġare Ġshown Ġin ĠTable ĠT AB REF 40 . ĠWe Ġfind Ġthat Ġour Ġproposed Ġmodel Ġoutper forms ĠHybrid Ġ( our Ġre - im ple mented Ġversion Ġfor Ġthis Ġexperiment ) Ġon Ġall Ġranges Ġof Ġrelative Ġdistances , Ġwhich Ġshows Ġthat Ġthe Ġstructured Ġanswer - relevant Ġrelations Ġcan Ġcapture Ġboth Ġshort Ġand Ġlong Ġterm Ġanswer - relevant Ġdependencies Ġof Ġthe Ġanswer Ġin Ġsentences . ĠFurthermore , Ġcomparing Ġthe Ġperformance Ġdifference Ġbetween ĠHybrid Ġand Ġour Ġmodel , Ġwe Ġfind Ġthe Ġimprovements Ġbecome Ġmore Ġsignificant Ġwhen Ġthe Ġdistance Ġincreases Ġfrom ĠâĢ ľ $ 0 \ sim Ġ10 $ âĢ Ŀ Ġto ĠâĢ ľ $ > 10 $ âĢ Ŀ . ĠOne Ġreason Ġis Ġthat Ġour Ġmodel Ġcan Ġextract Ġrelations Ġwith Ġdistant Ġdependencies Ġto Ġthe Ġanswer , Ġwhich Ġgreatly Ġhelps Ġour Ġmodel Ġignore Ġthe Ġextr aneous Ġinformation . ĠPro x imity - based Ġanswer - aware Ġmodels Ġmay Ġoverly Ġemphasize Ġthe Ġneighboring Ġwords Ġof Ġanswers Ġand Ġbecome Ġless Ġeffective Ġas Ġthe Ġuseful Ġcontext Ġbecomes Ġfurther Ġaway Ġfrom Ġthe Ġanswer Ġin Ġthe Ġcomplex Ġsentences . ĠIn Ġfact , Ġthe Ġbreakdown Ġintervals Ġin ĠTable ĠT AB REF 40 Ġnaturally Ġbound Ġits Ġsentence Ġlength , Ġsay Ġfor ĠâĢ ľ $ > 10 $ âĢ Ŀ , Ġthe Ġsentences Ġin Ġthis Ġgroup Ġmust Ġbe Ġlonger Ġthan Ġ10 . ĠThus , Ġthe Ġlength Ġvar iances Ġin Ġthese Ġtwo Ġintervals Ġcould Ġbe Ġsignificant . ĠTo Ġfurther Ġvalidate Ġwhether Ġour Ġmodel Ġcan Ġextract Ġlong Ġterm Ġdependency Ġwords . ĠWe Ġre run Ġthe Ġanalysis Ġof ĠTable ĠT AB REF 40 Ġonly Ġfor Ġlong Ġsentences Ġ( length Ġ$ > $ Ġ20 ) Ġof Ġeach Ġinterval . ĠThe Ġimprovement Ġpercentages Ġover ĠHybrid Ġare Ġshown Ġin ĠTable ĠT AB REF 40 , Ġwhich Ġbecome Ġmore Ġsignificant Ġwhen Ġthe Ġdistance Ġincreases Ġfrom ĠâĢ ľ $ 0 \ sim Ġ10 $ âĢ Ŀ Ġto ĠâĢ ľ $ > 10 $ âĢ Ŀ . </s> Results Ġand ĠAnalysis Ġ: :: ĠCase ĠStudy </s> Figure ĠFIG REF 42 Ġprovides Ġexample Ġquestions Ġgenerated Ġby Ġcrowd - workers Ġ( ground Ġtruth Ġquestions ), Ġthe Ġbaseline ĠHybrid ĠB IB REF 15 , Ġand Ġour Ġmodel . ĠIn Ġthe Ġfirst Ġcase , Ġthere Ġare Ġtwo Ġsubsequ ences Ġin Ġthe Ġinput Ġand Ġthe Ġanswer Ġhas Ġno Ġrelation Ġwith Ġthe Ġsecond Ġsubsequ ence . ĠHowever , Ġwe Ġsee Ġthat Ġthe Ġbaseline Ġmodel Ġprediction Ġcopies Ġirrelevant Ġwords ĠâĢ ľ The ĠNew ĠYork ĠTimes âĢ Ŀ Ġwhile Ġour Ġmodel Ġcan Ġavoid Ġusing Ġthe Ġextr aneous Ġsubsequ ence ĠâĢ ľ The ĠNew ĠYork ĠTimes Ġnoted Ġ... âĢ Ŀ Ġwith Ġthe Ġhelp Ġof Ġthe Ġstructured Ġanswer - relevant Ġrelation . ĠCompared Ġwith Ġthe Ġground Ġtruth Ġquestion , Ġour Ġmodel Ġcannot Ġcapture Ġthe Ġcross - sent ence Ġinformation Ġlike ĠâĢ ľ her Ġfifth Ġalbum âĢ Ŀ , Ġwhere Ġthe Ġtechniques Ġin Ġparagraph - level ĠQ G Ġmodels ĠB IB REF 8 Ġmay Ġhelp . ĠIn Ġthe Ġsecond Ġcase , Ġas Ġdiscussed Ġin ĠSection ĠSEC REF 1 , Ġthis Ġsentence Ġcontains Ġa Ġfew Ġfacts Ġand Ġsome Ġfacts Ġintertw ine . ĠWe Ġfind Ġthat Ġour Ġmodel Ġcan Ġcapture Ġdistant Ġanswer - relevant Ġdependencies Ġsuch Ġas ĠâĢ ľ mean Ġtemperature âĢ Ŀ Ġwhile Ġthe Ġproximity - based Ġbaseline Ġmodel Ġwrongly Ġtakes Ġneighboring Ġwords Ġof Ġthe Ġanswer Ġlike ĠâĢ ľ cold est âĢ Ŀ Ġin Ġthe Ġgenerated Ġquestion . </s> Results Ġand ĠAnalysis Ġ: :: ĠD iverse ĠQuestion ĠGeneration </s> Another Ġinteresting Ġobservation Ġis Ġthat Ġfor Ġthe Ġsame Ġanswer - sent ence Ġpair , Ġour Ġmodel Ġcan Ġgenerate Ġdiverse Ġquestions Ġby Ġtaking Ġdifferent Ġanswer - relevant Ġrelations Ġas Ġinput . ĠSuch Ġcapability Ġimproves Ġthe Ġinterpret ability Ġof Ġour Ġmodel Ġbecause Ġthe Ġmodel Ġis Ġgiven Ġnot Ġonly Ġwhat Ġto Ġbe Ġasked Ġ( i . e ., Ġthe Ġanswer ) Ġbut Ġalso Ġthe Ġrelated Ġfact Ġ( i . e ., Ġthe Ġanswer - relevant Ġrelation ) Ġto Ġbe Ġcovered Ġin Ġthe Ġquestion . ĠIn Ġcontrast , Ġproximity - based Ġanswer - aware Ġmodels Ġcan Ġonly Ġgenerate Ġone Ġquestion Ġgiven Ġthe Ġsentence - answer Ġpair Ġregardless Ġof Ġhow Ġmany Ġanswer - relevant Ġrelations Ġin Ġthe Ġsentence . ĠWe Ġthink Ġsuch Ġcapability Ġcan Ġalso Ġvalidate Ġour Ġmotivation : Ġquestions Ġshould Ġbe Ġgenerated Ġaccording Ġto Ġthe Ġanswer - aware Ġrelations Ġinstead Ġof Ġneighboring Ġwords Ġof Ġanswer Ġfragments . ĠFigure ĠFIG REF 45 Ġshow Ġtwo Ġexamples Ġof Ġdiverse Ġquestion Ġgeneration . ĠIn Ġthe Ġfirst Ġcase , Ġthe Ġanswer Ġfragment Ġ` H ugh ĠL . ĠDry den ' Ġis Ġthe Ġapp os itive Ġto Ġ` NASA ĠDeputy ĠAdministrator ' Ġbut Ġthe Ġsubject Ġto Ġthe Ġfollowing Ġtokens Ġ` announced Ġthe ĠApollo Ġprogram Ġ... '. ĠOur Ġframework Ġcan Ġextract Ġthese Ġtwo Ġanswer - relevant Ġrelations , Ġand Ġby Ġfeeding Ġthem Ġto Ġour Ġmodel Ġseparately , Ġwe Ġcan Ġreceive Ġtwo Ġquestions Ġasking Ġdifferent Ġrelations Ġwith Ġregard Ġto Ġthe Ġanswer . </s> Related ĠWork </s> The Ġtopic Ġof Ġquestion Ġgeneration , Ġinitially Ġmotivated Ġfor Ġeducational Ġpurposes , Ġis Ġtackled Ġby Ġdesigning Ġmany Ġcomplex Ġrules Ġfor Ġspecific Ġquestion Ġtypes ĠB IB REF 4 , ĠB IB REF 23 . ĠHe il man 2010 Good Q S Ġimprove Ġrule - based Ġquestion Ġgeneration Ġby Ġintroducing Ġa Ġstatistical Ġranking Ġmodel . ĠFirst , Ġthey Ġremove Ġextr aneous Ġinformation Ġin Ġthe Ġsentence Ġto Ġtransform Ġit Ġinto Ġa Ġsimpler Ġone , Ġwhich Ġcan Ġbe Ġtransformed Ġeasily Ġinto Ġa Ġsuccinct Ġquestion Ġwith Ġpred efined Ġsets Ġof Ġgeneral Ġrules . ĠThen Ġthey Ġadopt Ġan Ġover gener ate - and - rank Ġapproach Ġto Ġselect Ġthe Ġbest Ġcandidate Ġconsidering Ġseveral Ġfeatures . </s> With Ġthe Ġrise Ġof Ġdominant Ġneural Ġsequence - to - sequence Ġlearning Ġmodels ĠB IB REF 5 , ĠDu 2017 Learning TA Ġframe Ġquestion Ġgeneration Ġas Ġa Ġsequence - to - sequence Ġlearning Ġproblem . ĠCompared Ġwith Ġrule - based Ġapproaches , Ġneural Ġmodels ĠB IB REF 24 Ġcan Ġgenerate Ġmore Ġfluent Ġand Ġgram matical Ġquestions . ĠHowever , Ġquestion Ġgeneration Ġis Ġa Ġone - to - many Ġsequence Ġgeneration Ġproblem , Ġi . e ., Ġseveral Ġaspects Ġcan Ġbe Ġasked Ġgiven Ġa Ġsentence , Ġwhich Ġconf uses Ġthe Ġmodel Ġduring Ġtrain Ġand Ġprevents Ġconcrete Ġautomatic Ġevaluation . ĠTo Ġtackle Ġthis Ġissue , ĠZhou 2017 Ne ural Q G Ġpropose Ġthe Ġanswer - aware Ġquestion Ġgeneration Ġsetting Ġwhich Ġassumes Ġthe Ġanswer Ġis Ġalready Ġknown Ġand Ġacts Ġas Ġa Ġcontiguous Ġspan Ġinside Ġthe Ġinput Ġsentence . ĠThey Ġadopt Ġa ĠB IO Ġtagging Ġscheme Ġto Ġincorporate Ġthe Ġanswer Ġposition Ġinformation Ġas Ġlearned Ġembed ding Ġfeatures Ġin ĠSe q 2 Se q Ġlearning . ĠSong 2018 L ever aging CI Ġexplicitly Ġmodel Ġthe Ġinformation Ġbetween Ġanswer Ġand Ġsentence Ġwith Ġa Ġmulti - pers pect ive Ġmatching Ġmodel . ĠKim 2019 Impro ving N Q Ġalso Ġfocus Ġon Ġthe Ġanswer Ġinformation Ġand Ġproposed Ġan Ġanswer - separ ated ĠSe q 2 Se q Ġmodel Ġby Ġmask ing Ġthe Ġanswer Ġwith Ġspecial Ġtokens . ĠAll Ġanswer - aware Ġneural Ġmodels Ġtreat Ġquestion Ġgeneration Ġas Ġa Ġone - to - one Ġmapping Ġproblem , Ġbut Ġexisting Ġmodels Ġperform Ġpoorly Ġfor Ġsentences Ġwith Ġa Ġcomplex Ġstructure Ġ( as Ġshown Ġin ĠTable ĠT AB REF 2 ). </s> Our Ġwork Ġis Ġinspired Ġby Ġthe Ġprocess Ġof Ġextr aneous Ġinformation Ġremoving Ġin ĠB IB REF 0 , ĠB IB REF 25 . ĠDifferent Ġfrom ĠHe il man 2010 Good Q S Ġwhich Ġdirectly Ġuse Ġthe Ġsimplified Ġsentence Ġfor Ġgeneration Ġand Ġca o 2018 faith ful Ġwhich Ġonly Ġconsider Ġaggregate Ġtwo Ġsources Ġof Ġinformation Ġvia Ġg ated Ġattention Ġin Ġsummar ization , Ġwe Ġpropose Ġto Ġcombine Ġthe Ġstructured Ġanswer - relevant Ġrelation Ġand Ġthe Ġoriginal Ġsentence . ĠFact oid Ġquestion Ġgeneration Ġfrom Ġstructured Ġtext Ġis Ġinitially Ġinvestigated Ġby ĠSer ban 2016 Gener ating F Q , Ġbut Ġour Ġfocus Ġhere Ġis Ġleveraging Ġstructured Ġinputs Ġto Ġhelp Ġquestion Ġgeneration Ġover Ġun struct ured Ġsentences . ĠOur Ġproposed Ġmodel Ġcan Ġtake Ġadvantage Ġof Ġun struct ured Ġsentences Ġand Ġstructured Ġanswer - relevant Ġrelations Ġto Ġmaintain Ġinform at iveness Ġand Ġfaith fulness Ġof Ġgenerated Ġquestions . ĠThe Ġproposed Ġmodel Ġcan Ġalso Ġbe Ġgeneralized Ġin Ġother Ġconditional Ġsequence Ġgeneration Ġtasks Ġwhich Ġrequire Ġmultiple Ġsources Ġof Ġinputs , Ġe . g ., Ġdistract or Ġgeneration Ġfor Ġmultiple Ġchoice Ġquestions ĠB IB REF 26 . </s> Con clusions Ġand ĠFuture ĠWork </s> In Ġthis Ġpaper , Ġwe Ġpropose Ġa Ġquestion Ġgeneration Ġsystem Ġwhich Ġcombines Ġun struct ured Ġsentences Ġand Ġstructured Ġanswer - relevant Ġrelations Ġfor Ġgeneration . ĠThe Ġun struct ured Ġsentences Ġmaintain Ġthe Ġinform at iveness Ġof Ġgenerated Ġquestions Ġwhile Ġstructured Ġanswer - relevant Ġrelations Ġkeep Ġthe Ġfaith fulness Ġof Ġquestions . ĠExt ensive Ġexperiments Ġdemonstrate Ġthat Ġour Ġproposed Ġmodel Ġachieves Ġstate - of - the - art Ġperformance Ġacross Ġseveral Ġmetrics . ĠFurthermore , Ġour Ġmodel Ġcan Ġgenerate Ġdiverse Ġquestions Ġwith Ġdifferent Ġstructured Ġanswer - relevant Ġrelations . ĠFor Ġfuture Ġwork , Ġthere Ġare Ġsome Ġinteresting Ġdimensions Ġto Ġexplore , Ġsuch Ġas Ġdifficulty Ġlevels ĠB IB REF 27 , Ġparagraph - level Ġinformation ĠB IB REF 8 Ġand Ġconvers ational Ġquestion Ġgeneration ĠB IB REF 28 . </s> Acknowled gments </s> This Ġwork Ġis Ġsupported Ġby Ġthe ĠResearch ĠGrants ĠCouncil Ġof Ġthe ĠHong ĠKong ĠSpecial ĠAdministrative ĠRegion , ĠChina Ġ( No . ĠCU HK Ġ14 20 88 15 Ġand ĠNo . ĠCU HK Ġ142 107 17 Ġof Ġthe ĠGeneral ĠResearch ĠFund ). ĠWe Ġwould Ġlike Ġto Ġthank Ġthe Ġanonymous Ġreviewers Ġfor Ġtheir Ġcomments . ĠWe Ġwould Ġalso Ġlike Ġto Ġthank ĠDepartment Ġof ĠComputer ĠScience Ġand ĠEngineering , ĠThe ĠChinese ĠUniversity Ġof ĠHong ĠKong Ġfor Ġthe Ġconference Ġgrant Ġsupport .
Introduction Question Generation (QG) is the task of automatically creating questions from a range of inputs, such as natural language text BIBREF0, knowledge base BIBREF1 and image BIBREF2. QG is an increasingly important area in NLP with various application scenarios such as intelligence tutor systems, open-domain chatbots and question answering dataset construction. In this paper, we focus on question generation from reading comprehension materials like SQuAD BIBREF3. As shown in Figure FIGREF1, given a sentence in the reading comprehension paragraph and the text fragment (i.e., the answer) that we want to ask about, we aim to generate a question that is asked about the specified answer. Question generation for reading comprehension is firstly formalized as a declarative-to-interrogative sentence transformation problem with predefined rules or templates BIBREF4, BIBREF0. With the rise of neural models, Du2017LearningTA propose to model this task under the sequence-to-sequence (Seq2Seq) learning framework BIBREF5 with attention mechanism BIBREF6. However, question generation is a one-to-many sequence generation problem, i.e., several aspects can be asked given a sentence. Zhou2017NeuralQG propose the answer-aware question generation setting which assumes the answer, a contiguous span inside the input sentence, is already known before question generation. To capture answer-relevant words in the sentence, they adopt a BIO tagging scheme to incorporate the answer position embedding in Seq2Seq learning. Furthermore, Sun2018AnswerfocusedAP propose that tokens close to the answer fragments are more likely to be answer-relevant. Therefore, they explicitly encode the relative distance between sentence words and the answer via position embedding and position-aware attention. Although existing proximity-based answer-aware approaches achieve reasonable performance, we argue that such intuition may not apply to all cases especially for sentences with complex structure. For example, Figure FIGREF1 shows such an example where those approaches fail. This sentence contains a few facts and due to the parenthesis (i.e. “the area's coldest month”), some facts intertwine: “The daily mean temperature in January is 0.3$^\circ $C” and “January is the area's coldest month”. From the question generated by a proximity-based answer-aware baseline, we find that it wrongly uses the word “coldest” but misses the correct word “mean” because “coldest” has a shorter distance to the answer “0.3$^\circ $C”. In summary, their intuition that “the neighboring words of the answer are more likely to be answer-relevant and have a higher chance to be used in the question” is not reliable. To quantitatively show this drawback of these models, we implement the approach proposed by Sun2018AnswerfocusedAP and analyze its performance under different relative distances between the answer and other non-stop sentence words that also appear in the ground truth question. The results are shown in Table TABREF2. We find that the performance drops at most 36% when the relative distance increases from “$0\sim 10$” to “$>10$”. In other words, when the useful context is located far away from the answer, current proximity-based answer-aware approaches will become less effective, since they overly emphasize neighboring words of the answer. To address this issue, we extract the structured answer-relevant relations from sentences and propose a method to jointly model such structured relation and the unstructured sentence for question generation. The structured answer-relevant relation is likely to be to the point context and thus can help keep the generated question to the point. For example, Figure FIGREF1 shows our framework can extract the right answer-relevant relation (“The daily mean temperature in January”, “is”, “32.6$^\circ $F (0.3$^\circ $C)”) among multiple facts. With the help of such structured information, our model is less likely to be confused by sentences with a complex structure. Specifically, we firstly extract multiple relations with an off-the-shelf Open Information Extraction (OpenIE) toolbox BIBREF7, then we select the relation that is most relevant to the answer with carefully designed heuristic rules. Nevertheless, it is challenging to train a model to effectively utilize both the unstructured sentence and the structured answer-relevant relation because both of them could be noisy: the unstructured sentence may contain multiple facts which are irrelevant to the target question, while the limitation of the OpenIE tool may produce less accurate extracted relations. To explore their advantages simultaneously and avoid the drawbacks, we design a gated attention mechanism and a dual copy mechanism based on the encoder-decoder framework, where the former learns to control the information flow between the unstructured and structured inputs, while the latter learns to copy words from two sources to maintain the informativeness and faithfulness of generated questions. In the evaluations on the SQuAD dataset, our system achieves significant and consistent improvement as compared to all baseline methods. In particular, we demonstrate that the improvement is more significant with a larger relative distance between the answer and other non-stop sentence words that also appear in the ground truth question. Furthermore, our model is capable of generating diverse questions for a single sentence-answer pair where the sentence conveys multiple relations of its answer fragment. Framework Description In this section, we first introduce the task definition and our protocol to extract structured answer-relevant relations. Then we formalize the task under the encoder-decoder framework with gated attention and dual copy mechanism. Framework Description ::: Problem Definition We formalize our task as an answer-aware Question Generation (QG) problem BIBREF8, which assumes answer phrases are given before generating questions. Moreover, answer phrases are shown as text fragments in passages. Formally, given the sentence $S$, the answer $A$, and the answer-relevant relation $M$, the task of QG aims to find the best question $\overline{Q}$ such that, where $A$ is a contiguous span inside $S$. Framework Description ::: Answer-relevant Relation Extraction We utilize an off-the-shelf toolbox of OpenIE to the derive structured answer-relevant relations from sentences as to the point contexts. Relations extracted by OpenIE can be represented either in a triple format or in an n-ary format with several secondary arguments, and we employ the latter to keep the extractions as informative as possible and avoid extracting too many similar relations in different granularities from one sentence. We join all arguments in the extracted n-ary relation into a sequence as our to the point context. Figure FIGREF5 shows n-ary relations extracted from OpenIE. As we can see, OpenIE extracts multiple relations for complex sentences. Here we select the most informative relation according to three criteria in the order of descending importance: (1) having the maximal number of overlapped tokens between the answer and the relation; (2) being assigned the highest confidence score by OpenIE; (3) containing maximum non-stop words. As shown in Figure FIGREF5, our criteria can select answer-relevant relations (waved in Figure FIGREF5), which is especially useful for sentences with extraneous information. In rare cases, OpenIE cannot extract any relation, we treat the sentence itself as the to the point context. Table TABREF8 shows some statistics to verify the intuition that the extracted relations can serve as more to the point context. We find that the tokens in relations are 61% more likely to be used in the target question than the tokens in sentences, and thus they are more to the point. On the other hand, on average the sentences contain one more question token than the relations (1.86 v.s. 2.87). Therefore, it is still necessary to take the original sentence into account to generate a more accurate question. Framework Description ::: Our Proposed Model ::: Overview. As shown in Figure FIGREF10, our framework consists offour components (1) Sentence Encoder and Relation Encoder, (2) Decoder, (3) Gated Attention Mechanism and (4) Dual Copy Mechanism. The sentence encoder and relation encoder encode the unstructured sentence and the structured answer-relevant relation, respectively. To select and combine the source information from the two encoders, a gated attention mechanism is employed to jointly attend both contextualized information sources, and a dual copy mechanism copies words from either the sentence or the relation. Framework Description ::: Our Proposed Model ::: Answer-aware Encoder. We employ two encoders to integrate information from the unstructured sentence $S$ and the answer-relevant relation $M$ separately. Sentence encoder takes in feature-enriched embeddings including word embeddings $\mathbf {w}$, linguistic embeddings $\mathbf {l}$ and answer position embeddings $\mathbf {a}$. We follow BIBREF9 to transform POS and NER tags into continuous representation ($\mathbf {l}^p$ and $\mathbf {l}^n$) and adopt a BIO labelling scheme to derive the answer position embedding (B: the first token of the answer, I: tokens within the answer fragment except the first one, O: tokens outside of the answer fragment). For each word $w_i$ in the sentence $S$, we simply concatenate all features as input: $\mathbf {x}_i^s= [\mathbf {w}_i; \mathbf {l}^p_i; \mathbf {l}^n_i; \mathbf {a}_i]$. Here $[\mathbf {a};\mathbf {b}]$ denotes the concatenation of vectors $\mathbf {a}$ and $\mathbf {b}$. We use bidirectional LSTMs to encode the sentence $(\mathbf {x}_1^s, \mathbf {x}_2^s, ..., \mathbf {x}_n^s)$ to get a contextualized representation for each token: where $\overrightarrow{\mathbf {h}}^{s}_i$ and $\overleftarrow{\mathbf {h}}^{s}_i$ are the hidden states at the $i$-th time step of the forward and the backward LSTMs. The output state of the sentence encoder is the concatenation of forward and backward hidden states: $\mathbf {h}^{s}_i=[\overrightarrow{\mathbf {h}}^{s}_i;\overleftarrow{\mathbf {h}}^{s}_i]$. The contextualized representation of the sentence is $(\mathbf {h}^{s}_1, \mathbf {h}^{s}_2, ..., \mathbf {h}^{s}_n)$. For the relation encoder, we firstly join all items in the n-ary relation $M$ into a sequence. Then we only take answer position embedding as an extra feature for the sequence: $\mathbf {x}_i^m= [\mathbf {w}_i; \mathbf {a}_i]$. Similarly, we take another bidirectional LSTMs to encode the relation sequence and derive the corresponding contextualized representation $(\mathbf {h}^{m}_1, \mathbf {h}^{m}_2, ..., \mathbf {h}^{m}_n)$. Framework Description ::: Our Proposed Model ::: Decoder. We use an LSTM as the decoder to generate the question. The decoder predicts the word probability distribution at each decoding timestep to generate the question. At the t-th timestep, it reads the word embedding $\mathbf {w}_{t}$ and the hidden state $\mathbf {u}_{t-1}$ of the previous timestep to generate the current hidden state: Framework Description ::: Our Proposed Model ::: Gated Attention Mechanism. We design a gated attention mechanism to jointly attend the sentence representation and the relation representation. For sentence representation $(\mathbf {h}^{s}_1, \mathbf {h}^{s}_2, ..., \mathbf {h}^{s}_n)$, we employ the Luong2015EffectiveAT's attention mechanism to obtain the sentence context vector $\mathbf {c}^s_t$, where $\mathbf {W}_a$ is a trainable weight. Similarly, we obtain the vector $\mathbf {c}^m_t$ from the relation representation $(\mathbf {h}^{m}_1, \mathbf {h}^{m}_2, ..., \mathbf {h}^{m}_n)$. To jointly model the sentence and the relation, a gating mechanism is designed to control the information flow from two sources: where $\odot $ represents element-wise dot production and $\mathbf {W}_g, \mathbf {W}_h$ are trainable weights. Finally, the predicted probability distribution over the vocabulary $V$ is computed as: where $\mathbf {W}_V$ and $\mathbf {b}_V$ are parameters. Framework Description ::: Our Proposed Model ::: Dual Copy Mechanism. To deal with the rare and unknown words, the decoder applies the pointing method BIBREF10, BIBREF11, BIBREF12 to allow copying a token from the input sentence at the $t$-th decoding step. We reuse the attention score $\mathbf {\alpha }_{t}^s$ and $\mathbf {\alpha }_{t}^m$ to derive the copy probability over two source inputs: Different from the standard pointing method, we design a dual copy mechanism to copy from two sources with two gates. The first gate is designed for determining copy tokens from two sources of inputs or generate next word from $P_V$, which is computed as $g^v_t = \text{sigmoid}(\mathbf {w}^v_g \tilde{\mathbf {h}}_t + b^v_g)$. The second gate takes charge of selecting the source (sentence or relation) to copy from, which is computed as $g^c_t = \text{sigmoid}(\mathbf {w}^c_g [\mathbf {c}_t^s;\mathbf {c}_t^m] + b^c_g)$. Finally, we combine all probabilities $P_V$, $P_S$ and $P_M$ through two soft gates $g^v_t$ and $g^c_t$. The probability of predicting $w$ as the $t$-th token of the question is: Framework Description ::: Our Proposed Model ::: Training and Inference. Given the answer $A$, sentence $S$ and relation $M$, the training objective is to minimize the negative log-likelihood with regard to all parameters: where $\mathcal {\lbrace }Q\rbrace $ is the set of all training instances, $\theta $ denotes model parameters and $\text{log} P(Q|A,S,M;\theta )$ is the conditional log-likelihood of $Q$. In testing, our model targets to generate a question $Q$ by maximizing: Experimental Setting ::: Dataset & Metrics We conduct experiments on the SQuAD dataset BIBREF3. It contains 536 Wikipedia articles and 100k crowd-sourced question-answer pairs. The questions are written by crowd-workers and the answers are spans of tokens in the articles. We employ two different data splits by following Zhou2017NeuralQG and Du2017LearningTA . In Zhou2017NeuralQG, the original SQuAD development set is evenly divided into dev and test sets, while Du2017LearningTA treats SQuAD development set as its development set and splits original SQuAD training set into a training set and a test set. We also filter out questions which do not have any overlapped non-stop words with the corresponding sentences and perform some preprocessing steps, such as tokenization and sentence splitting. The data statistics are given in Table TABREF27. We evaluate with all commonly-used metrics in question generation BIBREF13: BLEU-1 (B1), BLEU-2 (B2), BLEU-3 (B3), BLEU-4 (B4) BIBREF17, METEOR (MET) BIBREF18 and ROUGE-L (R-L) BIBREF19. We use the evaluation script released by Chen2015MicrosoftCC. Experimental Setting ::: Baseline Models We compare with the following models. [leftmargin=*] s2s BIBREF13 proposes an attention-based sequence-to-sequence neural network for question generation. NQG++ BIBREF9 takes the answer position feature and linguistic features into consideration and equips the Seq2Seq model with copy mechanism. M2S+cp BIBREF14 conducts multi-perspective matching between the answer and the sentence to derive an answer-aware sentence representation for question generation. s2s+MP+GSA BIBREF8 introduces a gated self-attention into the encoder and a maxout pointer mechanism into the decoder. We report their sentence-level results for a fair comparison. Hybrid BIBREF15 is a hybrid model which considers the answer embedding for the question word generation and the position of context words for modeling the relative distance between the context words and the answer. ASs2s BIBREF16 replaces the answer in the sentence with a special token to avoid its appearance in the generated questions. Experimental Setting ::: Implementation Details We take the most frequent 20k words as our vocabulary and use the GloVe word embeddings BIBREF20 for initialization. The embedding dimensions for POS, NER, answer position are set to 20. We use two-layer LSTMs in both encoder and decoder, and the LSTMs hidden unit size is set to 600. We use dropout BIBREF21 with the probability $p=0.3$. All trainable parameters, except word embeddings, are randomly initialized with the Xavier uniform in $(-0.1, 0.1)$ BIBREF22. For optimization in the training, we use SGD as the optimizer with a minibatch size of 64 and an initial learning rate of 1.0. We train the model for 15 epochs and start halving the learning rate after the 8th epoch. We set the gradient norm upper bound to 3 during the training. We adopt the teacher-forcing for the training. In the testing, we select the model with the lowest perplexity and beam search with size 3 is employed for generating questions. All hyper-parameters and models are selected on the validation dataset. Results and Analysis ::: Main Results Table TABREF30 shows automatic evaluation results for our model and baselines (copied from their papers). Our proposed model which combines structured answer-relevant relations and unstructured sentences achieves significant improvements over proximity-based answer-aware models BIBREF9, BIBREF15 on both dataset splits. Presumably, our structured answer-relevant relation is a generalization of the context explored by the proximity-based methods because they can only capture short dependencies around answer fragments while our extractions can capture both short and long dependencies given the answer fragments. Moreover, our proposed framework is a general one to jointly leverage structured relations and unstructured sentences. All compared baseline models which only consider unstructured sentences can be further enhanced under our framework. Recall that existing proximity-based answer-aware models perform poorly when the distance between the answer fragment and other non-stop sentence words that also appear in the ground truth question is large (Table TABREF2). Here we investigate whether our proposed model using the structured answer-relevant relations can alleviate this issue or not, by conducting experiments for our model under the same setting as in Table TABREF2. The broken-down performances by different relative distances are shown in Table TABREF40. We find that our proposed model outperforms Hybrid (our re-implemented version for this experiment) on all ranges of relative distances, which shows that the structured answer-relevant relations can capture both short and long term answer-relevant dependencies of the answer in sentences. Furthermore, comparing the performance difference between Hybrid and our model, we find the improvements become more significant when the distance increases from “$0\sim 10$” to “$>10$”. One reason is that our model can extract relations with distant dependencies to the answer, which greatly helps our model ignore the extraneous information. Proximity-based answer-aware models may overly emphasize the neighboring words of answers and become less effective as the useful context becomes further away from the answer in the complex sentences. In fact, the breakdown intervals in Table TABREF40 naturally bound its sentence length, say for “$>10$”, the sentences in this group must be longer than 10. Thus, the length variances in these two intervals could be significant. To further validate whether our model can extract long term dependency words. We rerun the analysis of Table TABREF40 only for long sentences (length $>$ 20) of each interval. The improvement percentages over Hybrid are shown in Table TABREF40, which become more significant when the distance increases from “$0\sim 10$” to “$>10$”. Results and Analysis ::: Case Study Figure FIGREF42 provides example questions generated by crowd-workers (ground truth questions), the baseline Hybrid BIBREF15, and our model. In the first case, there are two subsequences in the input and the answer has no relation with the second subsequence. However, we see that the baseline model prediction copies irrelevant words “The New York Times” while our model can avoid using the extraneous subsequence “The New York Times noted ...” with the help of the structured answer-relevant relation. Compared with the ground truth question, our model cannot capture the cross-sentence information like “her fifth album”, where the techniques in paragraph-level QG models BIBREF8 may help. In the second case, as discussed in Section SECREF1, this sentence contains a few facts and some facts intertwine. We find that our model can capture distant answer-relevant dependencies such as “mean temperature” while the proximity-based baseline model wrongly takes neighboring words of the answer like “coldest” in the generated question. Results and Analysis ::: Diverse Question Generation Another interesting observation is that for the same answer-sentence pair, our model can generate diverse questions by taking different answer-relevant relations as input. Such capability improves the interpretability of our model because the model is given not only what to be asked (i.e., the answer) but also the related fact (i.e., the answer-relevant relation) to be covered in the question. In contrast, proximity-based answer-aware models can only generate one question given the sentence-answer pair regardless of how many answer-relevant relations in the sentence. We think such capability can also validate our motivation: questions should be generated according to the answer-aware relations instead of neighboring words of answer fragments. Figure FIGREF45 show two examples of diverse question generation. In the first case, the answer fragment `Hugh L. Dryden' is the appositive to `NASA Deputy Administrator' but the subject to the following tokens `announced the Apollo program ...'. Our framework can extract these two answer-relevant relations, and by feeding them to our model separately, we can receive two questions asking different relations with regard to the answer. Related Work The topic of question generation, initially motivated for educational purposes, is tackled by designing many complex rules for specific question types BIBREF4, BIBREF23. Heilman2010GoodQS improve rule-based question generation by introducing a statistical ranking model. First, they remove extraneous information in the sentence to transform it into a simpler one, which can be transformed easily into a succinct question with predefined sets of general rules. Then they adopt an overgenerate-and-rank approach to select the best candidate considering several features. With the rise of dominant neural sequence-to-sequence learning models BIBREF5, Du2017LearningTA frame question generation as a sequence-to-sequence learning problem. Compared with rule-based approaches, neural models BIBREF24 can generate more fluent and grammatical questions. However, question generation is a one-to-many sequence generation problem, i.e., several aspects can be asked given a sentence, which confuses the model during train and prevents concrete automatic evaluation. To tackle this issue, Zhou2017NeuralQG propose the answer-aware question generation setting which assumes the answer is already known and acts as a contiguous span inside the input sentence. They adopt a BIO tagging scheme to incorporate the answer position information as learned embedding features in Seq2Seq learning. Song2018LeveragingCI explicitly model the information between answer and sentence with a multi-perspective matching model. Kim2019ImprovingNQ also focus on the answer information and proposed an answer-separated Seq2Seq model by masking the answer with special tokens. All answer-aware neural models treat question generation as a one-to-one mapping problem, but existing models perform poorly for sentences with a complex structure (as shown in Table TABREF2). Our work is inspired by the process of extraneous information removing in BIBREF0, BIBREF25. Different from Heilman2010GoodQS which directly use the simplified sentence for generation and cao2018faithful which only consider aggregate two sources of information via gated attention in summarization, we propose to combine the structured answer-relevant relation and the original sentence. Factoid question generation from structured text is initially investigated by Serban2016GeneratingFQ, but our focus here is leveraging structured inputs to help question generation over unstructured sentences. Our proposed model can take advantage of unstructured sentences and structured answer-relevant relations to maintain informativeness and faithfulness of generated questions. The proposed model can also be generalized in other conditional sequence generation tasks which require multiple sources of inputs, e.g., distractor generation for multiple choice questions BIBREF26. Conclusions and Future Work In this paper, we propose a question generation system which combines unstructured sentences and structured answer-relevant relations for generation. The unstructured sentences maintain the informativeness of generated questions while structured answer-relevant relations keep the faithfulness of questions. Extensive experiments demonstrate that our proposed model achieves state-of-the-art performance across several metrics. Furthermore, our model can generate diverse questions with different structured answer-relevant relations. For future work, there are some interesting dimensions to explore, such as difficulty levels BIBREF27, paragraph-level information BIBREF8 and conversational question generation BIBREF28. Acknowledgments This work is supported by the Research Grants Council of the Hong Kong Special Administrative Region, China (No. CUHK 14208815 and No. CUHK 14210717 of the General Research Fund). We would like to thank the anonymous reviewers for their comments. We would also like to thank Department of Computer Science and Engineering, The Chinese University of Hong Kong for the conference grant support.
How they extract "structured answer-relevant relation"?
1d197cbcac7b3f4015416f0152a6692e881ada6c
How Ġthey Ġextract Ġ" struct ured Ġanswer - relevant Ġrelation "?
Using Ġthe ĠOpen IE Ġtool box Ġand Ġapplying Ġhe uristic Ġrules Ġto Ġselect Ġthe Ġmost Ġrelevant Ġrelation .
<s> How Ġthey Ġextract Ġ" struct ured Ġanswer - relevant Ġrelation "? </s> Introduction </s> Question ĠGeneration Ġ( Q G ) Ġis Ġthe Ġtask Ġof Ġautomatically Ġcreating Ġquestions Ġfrom Ġa Ġrange Ġof Ġinputs , Ġsuch Ġas Ġnatural Ġlanguage Ġtext ĠB IB REF 0 , Ġknowledge Ġbase ĠB IB REF 1 Ġand Ġimage ĠB IB REF 2 . ĠQ G Ġis Ġan Ġincreasingly Ġimportant Ġarea Ġin ĠN LP Ġwith Ġvarious Ġapplication Ġscenarios Ġsuch Ġas Ġintelligence Ġtutor Ġsystems , Ġopen - domain Ġchat bots Ġand Ġquestion Ġanswering Ġdataset Ġconstruction . ĠIn Ġthis Ġpaper , Ġwe Ġfocus Ġon Ġquestion Ġgeneration Ġfrom Ġreading Ġcomprehension Ġmaterials Ġlike ĠS Qu AD ĠB IB REF 3 . ĠAs Ġshown Ġin ĠFigure ĠFIG REF 1 , Ġgiven Ġa Ġsentence Ġin Ġthe Ġreading Ġcomprehension Ġparagraph Ġand Ġthe Ġtext Ġfragment Ġ( i . e ., Ġthe Ġanswer ) Ġthat Ġwe Ġwant Ġto Ġask Ġabout , Ġwe Ġaim Ġto Ġgenerate Ġa Ġquestion Ġthat Ġis Ġasked Ġabout Ġthe Ġspecified Ġanswer . </s> Question Ġgeneration Ġfor Ġreading Ġcomprehension Ġis Ġfirst ly Ġformal ized Ġas Ġa Ġdecl ar ative - to - inter rog ative Ġsentence Ġtransformation Ġproblem Ġwith Ġpred efined Ġrules Ġor Ġtemplates ĠB IB REF 4 , ĠB IB REF 0 . ĠWith Ġthe Ġrise Ġof Ġneural Ġmodels , ĠDu 2017 Learning TA Ġpropose Ġto Ġmodel Ġthis Ġtask Ġunder Ġthe Ġsequence - to - sequence Ġ( Se q 2 Se q ) Ġlearning Ġframework ĠB IB REF 5 Ġwith Ġattention Ġmechanism ĠB IB REF 6 . ĠHowever , Ġquestion Ġgeneration Ġis Ġa Ġone - to - many Ġsequence Ġgeneration Ġproblem , Ġi . e ., Ġseveral Ġaspects Ġcan Ġbe Ġasked Ġgiven Ġa Ġsentence . ĠZhou 2017 Ne ural Q G Ġpropose Ġthe Ġanswer - aware Ġquestion Ġgeneration Ġsetting Ġwhich Ġassumes Ġthe Ġanswer , Ġa Ġcontiguous Ġspan Ġinside Ġthe Ġinput Ġsentence , Ġis Ġalready Ġknown Ġbefore Ġquestion Ġgeneration . ĠTo Ġcapture Ġanswer - relevant Ġwords Ġin Ġthe Ġsentence , Ġthey Ġadopt Ġa ĠB IO Ġtagging Ġscheme Ġto Ġincorporate Ġthe Ġanswer Ġposition Ġembed ding Ġin ĠSe q 2 Se q Ġlearning . ĠFurthermore , ĠSun 2018 Answer focused AP Ġpropose Ġthat Ġtokens Ġclose Ġto Ġthe Ġanswer Ġfragments Ġare Ġmore Ġlikely Ġto Ġbe Ġanswer - relevant . ĠTherefore , Ġthey Ġexplicitly Ġencode Ġthe Ġrelative Ġdistance Ġbetween Ġsentence Ġwords Ġand Ġthe Ġanswer Ġvia Ġposition Ġembed ding Ġand Ġposition - aware Ġattention . </s> Although Ġexisting Ġproximity - based Ġanswer - aware Ġapproaches Ġachieve Ġreasonable Ġperformance , Ġwe Ġargue Ġthat Ġsuch Ġintuition Ġmay Ġnot Ġapply Ġto Ġall Ġcases Ġespecially Ġfor Ġsentences Ġwith Ġcomplex Ġstructure . ĠFor Ġexample , ĠFigure ĠFIG REF 1 Ġshows Ġsuch Ġan Ġexample Ġwhere Ġthose Ġapproaches Ġfail . ĠThis Ġsentence Ġcontains Ġa Ġfew Ġfacts Ġand Ġdue Ġto Ġthe Ġparent hesis Ġ( i . e . ĠâĢ ľ the Ġarea 's Ġcold est Ġmonth âĢ Ŀ ), Ġsome Ġfacts Ġintertw ine : ĠâĢ ľ The Ġdaily Ġmean Ġtemperature Ġin ĠJanuary Ġis Ġ0 . 3 $ ^ \ circ Ġ$ C âĢ Ŀ Ġand ĠâĢ ľ January Ġis Ġthe Ġarea 's Ġcold est Ġmonth âĢ Ŀ . ĠFrom Ġthe Ġquestion Ġgenerated Ġby Ġa Ġproximity - based Ġanswer - aware Ġbaseline , Ġwe Ġfind Ġthat Ġit Ġwrongly Ġuses Ġthe Ġword ĠâĢ ľ cold est âĢ Ŀ Ġbut Ġmisses Ġthe Ġcorrect Ġword ĠâĢ ľ mean âĢ Ŀ Ġbecause ĠâĢ ľ cold est âĢ Ŀ Ġhas Ġa Ġshorter Ġdistance Ġto Ġthe Ġanswer ĠâĢ ľ 0 . 3 $ ^ \ circ Ġ$ C âĢ Ŀ . </s> In Ġsummary , Ġtheir Ġintuition Ġthat ĠâĢ ľ the Ġneighboring Ġwords Ġof Ġthe Ġanswer Ġare Ġmore Ġlikely Ġto Ġbe Ġanswer - relevant Ġand Ġhave Ġa Ġhigher Ġchance Ġto Ġbe Ġused Ġin Ġthe Ġquestion âĢ Ŀ Ġis Ġnot Ġreliable . ĠTo Ġquant itatively Ġshow Ġthis Ġdrawback Ġof Ġthese Ġmodels , Ġwe Ġimplement Ġthe Ġapproach Ġproposed Ġby ĠSun 2018 Answer focused AP Ġand Ġanalyze Ġits Ġperformance Ġunder Ġdifferent Ġrelative Ġdistances Ġbetween Ġthe Ġanswer Ġand Ġother Ġnon - stop Ġsentence Ġwords Ġthat Ġalso Ġappear Ġin Ġthe Ġground Ġtruth Ġquestion . ĠThe Ġresults Ġare Ġshown Ġin ĠTable ĠT AB REF 2 . ĠWe Ġfind Ġthat Ġthe Ġperformance Ġdrops Ġat Ġmost Ġ36 % Ġwhen Ġthe Ġrelative Ġdistance Ġincreases Ġfrom ĠâĢ ľ $ 0 \ sim Ġ10 $ âĢ Ŀ Ġto ĠâĢ ľ $ > 10 $ âĢ Ŀ . ĠIn Ġother Ġwords , Ġwhen Ġthe Ġuseful Ġcontext Ġis Ġlocated Ġfar Ġaway Ġfrom Ġthe Ġanswer , Ġcurrent Ġproximity - based Ġanswer - aware Ġapproaches Ġwill Ġbecome Ġless Ġeffective , Ġsince Ġthey Ġoverly Ġemphasize Ġneighboring Ġwords Ġof Ġthe Ġanswer . </s> To Ġaddress Ġthis Ġissue , Ġwe Ġextract Ġthe Ġstructured Ġanswer - relevant Ġrelations Ġfrom Ġsentences Ġand Ġpropose Ġa Ġmethod Ġto Ġjointly Ġmodel Ġsuch Ġstructured Ġrelation Ġand Ġthe Ġun struct ured Ġsentence Ġfor Ġquestion Ġgeneration . ĠThe Ġstructured Ġanswer - relevant Ġrelation Ġis Ġlikely Ġto Ġbe Ġto Ġthe Ġpoint Ġcontext Ġand Ġthus Ġcan Ġhelp Ġkeep Ġthe Ġgenerated Ġquestion Ġto Ġthe Ġpoint . ĠFor Ġexample , ĠFigure ĠFIG REF 1 Ġshows Ġour Ġframework Ġcan Ġextract Ġthe Ġright Ġanswer - relevant Ġrelation Ġ( âĢ ľ The Ġdaily Ġmean Ġtemperature Ġin ĠJanuary âĢ Ŀ , ĠâĢ ľ is âĢ Ŀ , ĠâĢ ľ 32 . 6 $ ^ \ circ Ġ$ F Ġ( 0 . 3 $ ^ \ circ Ġ$ C ) âĢ Ŀ ) Ġamong Ġmultiple Ġfacts . ĠWith Ġthe Ġhelp Ġof Ġsuch Ġstructured Ġinformation , Ġour Ġmodel Ġis Ġless Ġlikely Ġto Ġbe Ġconfused Ġby Ġsentences Ġwith Ġa Ġcomplex Ġstructure . ĠSpecifically , Ġwe Ġfirst ly Ġextract Ġmultiple Ġrelations Ġwith Ġan Ġoff - the - she lf ĠOpen ĠInformation ĠExt raction Ġ( Open IE ) Ġtool box ĠB IB REF 7 , Ġthen Ġwe Ġselect Ġthe Ġrelation Ġthat Ġis Ġmost Ġrelevant Ġto Ġthe Ġanswer Ġwith Ġcarefully Ġdesigned Ġhe uristic Ġrules . </s> Nevertheless , Ġit Ġis Ġchallenging Ġto Ġtrain Ġa Ġmodel Ġto Ġeffectively Ġutilize Ġboth Ġthe Ġun struct ured Ġsentence Ġand Ġthe Ġstructured Ġanswer - relevant Ġrelation Ġbecause Ġboth Ġof Ġthem Ġcould Ġbe Ġnoisy : Ġthe Ġun struct ured Ġsentence Ġmay Ġcontain Ġmultiple Ġfacts Ġwhich Ġare Ġirrelevant Ġto Ġthe Ġtarget Ġquestion , Ġwhile Ġthe Ġlimitation Ġof Ġthe ĠOpen IE Ġtool Ġmay Ġproduce Ġless Ġaccurate Ġextracted Ġrelations . ĠTo Ġexplore Ġtheir Ġadvantages Ġsimultaneously Ġand Ġavoid Ġthe Ġdrawbacks , Ġwe Ġdesign Ġa Ġg ated Ġattention Ġmechanism Ġand Ġa Ġdual Ġcopy Ġmechanism Ġbased Ġon Ġthe Ġenc oder - dec oder Ġframework , Ġwhere Ġthe Ġformer Ġlearns Ġto Ġcontrol Ġthe Ġinformation Ġflow Ġbetween Ġthe Ġun struct ured Ġand Ġstructured Ġinputs , Ġwhile Ġthe Ġlatter Ġlearns Ġto Ġcopy Ġwords Ġfrom Ġtwo Ġsources Ġto Ġmaintain Ġthe Ġinform at iveness Ġand Ġfaith fulness Ġof Ġgenerated Ġquestions . </s> In Ġthe Ġevaluations Ġon Ġthe ĠS Qu AD Ġdataset , Ġour Ġsystem Ġachieves Ġsignificant Ġand Ġconsistent Ġimprovement Ġas Ġcompared Ġto Ġall Ġbaseline Ġmethods . ĠIn Ġparticular , Ġwe Ġdemonstrate Ġthat Ġthe Ġimprovement Ġis Ġmore Ġsignificant Ġwith Ġa Ġlarger Ġrelative Ġdistance Ġbetween Ġthe Ġanswer Ġand Ġother Ġnon - stop Ġsentence Ġwords Ġthat Ġalso Ġappear Ġin Ġthe Ġground Ġtruth Ġquestion . ĠFurthermore , Ġour Ġmodel Ġis Ġcapable Ġof Ġgenerating Ġdiverse Ġquestions Ġfor Ġa Ġsingle Ġsentence - answer Ġpair Ġwhere Ġthe Ġsentence Ġconve ys Ġmultiple Ġrelations Ġof Ġits Ġanswer Ġfragment . </s> Fram ework ĠDescription </s> In Ġthis Ġsection , Ġwe Ġfirst Ġintroduce Ġthe Ġtask Ġdefinition Ġand Ġour Ġprotocol Ġto Ġextract Ġstructured Ġanswer - relevant Ġrelations . ĠThen Ġwe Ġformal ize Ġthe Ġtask Ġunder Ġthe Ġenc oder - dec oder Ġframework Ġwith Ġg ated Ġattention Ġand Ġdual Ġcopy Ġmechanism . </s> Fram ework ĠDescription Ġ: :: ĠProblem ĠDefinition </s> We Ġformal ize Ġour Ġtask Ġas Ġan Ġanswer - aware ĠQuestion ĠGeneration Ġ( Q G ) Ġproblem ĠB IB REF 8 , Ġwhich Ġassumes Ġanswer Ġphrases Ġare Ġgiven Ġbefore Ġgenerating Ġquestions . ĠMoreover , Ġanswer Ġphrases Ġare Ġshown Ġas Ġtext Ġfragments Ġin Ġpassages . ĠForm ally , Ġgiven Ġthe Ġsentence Ġ$ S $, Ġthe Ġanswer Ġ$ A $, Ġand Ġthe Ġanswer - relevant Ġrelation Ġ$ M $, Ġthe Ġtask Ġof ĠQ G Ġaims Ġto Ġfind Ġthe Ġbest Ġquestion Ġ$\ over line { Q } $ Ġsuch Ġthat , </s> where Ġ$ A $ Ġis Ġa Ġcontiguous Ġspan Ġinside Ġ$ S $. </s> Fram ework ĠDescription Ġ: :: ĠAnswer - relevant ĠRel ation ĠExt raction </s> We Ġutilize Ġan Ġoff - the - she lf Ġtool box Ġof ĠOpen IE Ġto Ġthe Ġderive Ġstructured Ġanswer - relevant Ġrelations Ġfrom Ġsentences Ġas Ġto Ġthe Ġpoint Ġcontexts . ĠRelations Ġextracted Ġby ĠOpen IE Ġcan Ġbe Ġrepresented Ġeither Ġin Ġa Ġtriple Ġformat Ġor Ġin Ġan Ġn - ary Ġformat Ġwith Ġseveral Ġsecondary Ġarguments , Ġand Ġwe Ġemploy Ġthe Ġlatter Ġto Ġkeep Ġthe Ġextract ions Ġas Ġinformative Ġas Ġpossible Ġand Ġavoid Ġextracting Ġtoo Ġmany Ġsimilar Ġrelations Ġin Ġdifferent Ġgran ular ities Ġfrom Ġone Ġsentence . ĠWe Ġjoin Ġall Ġarguments Ġin Ġthe Ġextracted Ġn - ary Ġrelation Ġinto Ġa Ġsequence Ġas Ġour Ġto Ġthe Ġpoint Ġcontext . ĠFigure ĠFIG REF 5 Ġshows Ġn - ary Ġrelations Ġextracted Ġfrom ĠOpen IE . ĠAs Ġwe Ġcan Ġsee , ĠOpen IE Ġextracts Ġmultiple Ġrelations Ġfor Ġcomplex Ġsentences . ĠHere Ġwe Ġselect Ġthe Ġmost Ġinformative Ġrelation Ġaccording Ġto Ġthree Ġcriteria Ġin Ġthe Ġorder Ġof Ġdescending Ġimportance : Ġ( 1 ) Ġhaving Ġthe Ġmaximal Ġnumber Ġof Ġoverl apped Ġtokens Ġbetween Ġthe Ġanswer Ġand Ġthe Ġrelation ; Ġ( 2 ) Ġbeing Ġassigned Ġthe Ġhighest Ġconfidence Ġscore Ġby ĠOpen IE ; Ġ( 3 ) Ġcontaining Ġmaximum Ġnon - stop Ġwords . ĠAs Ġshown Ġin ĠFigure ĠFIG REF 5 , Ġour Ġcriteria Ġcan Ġselect Ġanswer - relevant Ġrelations Ġ( w aved Ġin ĠFigure ĠFIG REF 5 ), Ġwhich Ġis Ġespecially Ġuseful Ġfor Ġsentences Ġwith Ġextr aneous Ġinformation . ĠIn Ġrare Ġcases , ĠOpen IE Ġcannot Ġextract Ġany Ġrelation , Ġwe Ġtreat Ġthe Ġsentence Ġitself Ġas Ġthe Ġto Ġthe Ġpoint Ġcontext . </s> Table ĠT AB REF 8 Ġshows Ġsome Ġstatistics Ġto Ġverify Ġthe Ġintuition Ġthat Ġthe Ġextracted Ġrelations Ġcan Ġserve Ġas Ġmore Ġto Ġthe Ġpoint Ġcontext . ĠWe Ġfind Ġthat Ġthe Ġtokens Ġin Ġrelations Ġare Ġ61 % Ġmore Ġlikely Ġto Ġbe Ġused Ġin Ġthe Ġtarget Ġquestion Ġthan Ġthe Ġtokens Ġin Ġsentences , Ġand Ġthus Ġthey Ġare Ġmore Ġto Ġthe Ġpoint . ĠOn Ġthe Ġother Ġhand , Ġon Ġaverage Ġthe Ġsentences Ġcontain Ġone Ġmore Ġquestion Ġtoken Ġthan Ġthe Ġrelations Ġ( 1 . 86 Ġv . s . Ġ2 . 87 ). ĠTherefore , Ġit Ġis Ġstill Ġnecessary Ġto Ġtake Ġthe Ġoriginal Ġsentence Ġinto Ġaccount Ġto Ġgenerate Ġa Ġmore Ġaccurate Ġquestion . </s> Fram ework ĠDescription Ġ: :: ĠOur ĠProp osed ĠModel Ġ: :: ĠOverview . </s> As Ġshown Ġin ĠFigure ĠFIG REF 10 , Ġour Ġframework Ġconsists Ġoff our Ġcomponents Ġ( 1 ) ĠSent ence ĠEnc oder Ġand ĠRel ation ĠEnc oder , Ġ( 2 ) ĠDecoder , Ġ( 3 ) ĠG ated ĠAttention ĠMechan ism Ġand Ġ( 4 ) ĠDual ĠCopy ĠMechan ism . ĠThe Ġsentence Ġenc oder Ġand Ġrelation Ġenc oder Ġencode Ġthe Ġun struct ured Ġsentence Ġand Ġthe Ġstructured Ġanswer - relevant Ġrelation , Ġrespectively . ĠTo Ġselect Ġand Ġcombine Ġthe Ġsource Ġinformation Ġfrom Ġthe Ġtwo Ġenc od ers , Ġa Ġg ated Ġattention Ġmechanism Ġis Ġemployed Ġto Ġjointly Ġattend Ġboth Ġcontextual ized Ġinformation Ġsources , Ġand Ġa Ġdual Ġcopy Ġmechanism Ġcopies Ġwords Ġfrom Ġeither Ġthe Ġsentence Ġor Ġthe Ġrelation . </s> Fram ework ĠDescription Ġ: :: ĠOur ĠProp osed ĠModel Ġ: :: ĠAnswer - aware ĠEnc oder . </s> We Ġemploy Ġtwo Ġenc od ers Ġto Ġintegrate Ġinformation Ġfrom Ġthe Ġun struct ured Ġsentence Ġ$ S $ Ġand Ġthe Ġanswer - relevant Ġrelation Ġ$ M $ Ġseparately . ĠSent ence Ġenc oder Ġtakes Ġin Ġfeature - en riched Ġembed d ings Ġincluding Ġword Ġembed d ings Ġ$\ math bf Ġ{ w } $, Ġlinguistic Ġembed d ings Ġ$\ math bf Ġ{ l } $ Ġand Ġanswer Ġposition Ġembed d ings Ġ$\ math bf Ġ{ a } $. ĠWe Ġfollow ĠB IB REF 9 Ġto Ġtransform ĠPOS Ġand ĠN ER Ġtags Ġinto Ġcontinuous Ġrepresentation Ġ($ \ math bf Ġ{ l } ^ p $ Ġand Ġ$\ math bf Ġ{ l } ^ n $ ) Ġand Ġadopt Ġa ĠB IO Ġlab elling Ġscheme Ġto Ġderive Ġthe Ġanswer Ġposition Ġembed ding Ġ( B : Ġthe Ġfirst Ġtoken Ġof Ġthe Ġanswer , ĠI : Ġtokens Ġwithin Ġthe Ġanswer Ġfragment Ġexcept Ġthe Ġfirst Ġone , ĠO : Ġtokens Ġoutside Ġof Ġthe Ġanswer Ġfragment ). ĠFor Ġeach Ġword Ġ$ w _ i $ Ġin Ġthe Ġsentence Ġ$ S $, Ġwe Ġsimply Ġconc aten ate Ġall Ġfeatures Ġas Ġinput : Ġ$\ math bf Ġ{ x } _ i ^ s = Ġ[ \ math bf Ġ{ w } _ i ; Ġ\ math bf Ġ{ l } ^ p _ i ; Ġ\ math bf Ġ{ l } ^ n _ i ; Ġ\ math bf Ġ{ a } _ i ] $. ĠHere Ġ$ [ \ math bf Ġ{ a }; \ math bf Ġ{ b } ] $ Ġdenotes Ġthe Ġconc aten ation Ġof Ġvectors Ġ$\ math bf Ġ{ a } $ Ġand Ġ$\ math bf Ġ{ b } $. </s> We Ġuse Ġbid irection al ĠL ST Ms Ġto Ġencode Ġthe Ġsentence Ġ$( \ math bf Ġ{ x } _ 1 ^ s , Ġ\ math bf Ġ{ x } _ 2 ^ s , Ġ... , Ġ\ math bf Ġ{ x } _ n ^ s ) $ Ġto Ġget Ġa Ġcontextual ized Ġrepresentation Ġfor Ġeach Ġtoken : </s> where Ġ$\ over right arrow {\ math bf Ġ{ h }} ^{ s } _ i $ Ġand Ġ$\ over left arrow {\ math bf Ġ{ h }} ^{ s } _ i $ Ġare Ġthe Ġhidden Ġstates Ġat Ġthe Ġ$ i $ - th Ġtime Ġstep Ġof Ġthe Ġforward Ġand Ġthe Ġbackward ĠL ST Ms . ĠThe Ġoutput Ġstate Ġof Ġthe Ġsentence Ġenc oder Ġis Ġthe Ġconc aten ation Ġof Ġforward Ġand Ġbackward Ġhidden Ġstates : Ġ$\ math bf Ġ{ h } ^{ s } _ i =[ \ over right arrow {\ math bf Ġ{ h }} ^{ s } _ i ; \ over left arrow {\ math bf Ġ{ h }} ^{ s } _ i ] $. ĠThe Ġcontextual ized Ġrepresentation Ġof Ġthe Ġsentence Ġis Ġ$( \ math bf Ġ{ h } ^{ s } _ 1 , Ġ\ math bf Ġ{ h } ^{ s } _ 2 , Ġ... , Ġ\ math bf Ġ{ h } ^{ s } _ n ) $. </s> For Ġthe Ġrelation Ġenc oder , Ġwe Ġfirst ly Ġjoin Ġall Ġitems Ġin Ġthe Ġn - ary Ġrelation Ġ$ M $ Ġinto Ġa Ġsequence . ĠThen Ġwe Ġonly Ġtake Ġanswer Ġposition Ġembed ding Ġas Ġan Ġextra Ġfeature Ġfor Ġthe Ġsequence : Ġ$\ math bf Ġ{ x } _ i ^ m = Ġ[ \ math bf Ġ{ w } _ i ; Ġ\ math bf Ġ{ a } _ i ] $. ĠSimilarly , Ġwe Ġtake Ġanother Ġbid irection al ĠL ST Ms Ġto Ġencode Ġthe Ġrelation Ġsequence Ġand Ġderive Ġthe Ġcorresponding Ġcontextual ized Ġrepresentation Ġ$( \ math bf Ġ{ h } ^{ m } _ 1 , Ġ\ math bf Ġ{ h } ^{ m } _ 2 , Ġ... , Ġ\ math bf Ġ{ h } ^{ m } _ n ) $. </s> Fram ework ĠDescription Ġ: :: ĠOur ĠProp osed ĠModel Ġ: :: ĠDecoder . </s> We Ġuse Ġan ĠL ST M Ġas Ġthe Ġdec oder Ġto Ġgenerate Ġthe Ġquestion . ĠThe Ġdec oder Ġpredicts Ġthe Ġword Ġprobability Ġdistribution Ġat Ġeach Ġdecoding Ġtim est ep Ġto Ġgenerate Ġthe Ġquestion . ĠAt Ġthe Ġt - th Ġtim est ep , Ġit Ġreads Ġthe Ġword Ġembed ding Ġ$\ math bf Ġ{ w } _{ t } $ Ġand Ġthe Ġhidden Ġstate Ġ$\ math bf Ġ{ u } _{ t - 1 } $ Ġof Ġthe Ġprevious Ġtim est ep Ġto Ġgenerate Ġthe Ġcurrent Ġhidden Ġstate : </s> Fram ework ĠDescription Ġ: :: ĠOur ĠProp osed ĠModel Ġ: :: ĠG ated ĠAttention ĠMechan ism . </s> We Ġdesign Ġa Ġg ated Ġattention Ġmechanism Ġto Ġjointly Ġattend Ġthe Ġsentence Ġrepresentation Ġand Ġthe Ġrelation Ġrepresentation . ĠFor Ġsentence Ġrepresentation Ġ$( \ math bf Ġ{ h } ^{ s } _ 1 , Ġ\ math bf Ġ{ h } ^{ s } _ 2 , Ġ... , Ġ\ math bf Ġ{ h } ^{ s } _ n ) $, Ġwe Ġemploy Ġthe ĠLu ong 2015 Effective AT 's Ġattention Ġmechanism Ġto Ġobtain Ġthe Ġsentence Ġcontext Ġvector Ġ$\ math bf Ġ{ c } ^ s _ t $, </s> where Ġ$\ math bf Ġ{ W } _ a $ Ġis Ġa Ġtrain able Ġweight . ĠSimilarly , Ġwe Ġobtain Ġthe Ġvector Ġ$\ math bf Ġ{ c } ^ m _ t $ Ġfrom Ġthe Ġrelation Ġrepresentation Ġ$( \ math bf Ġ{ h } ^{ m } _ 1 , Ġ\ math bf Ġ{ h } ^{ m } _ 2 , Ġ... , Ġ\ math bf Ġ{ h } ^{ m } _ n ) $. ĠTo Ġjointly Ġmodel Ġthe Ġsentence Ġand Ġthe Ġrelation , Ġa Ġg ating Ġmechanism Ġis Ġdesigned Ġto Ġcontrol Ġthe Ġinformation Ġflow Ġfrom Ġtwo Ġsources : </s> where Ġ$\ od ot Ġ$ Ġrepresents Ġelement - wise Ġdot Ġproduction Ġand Ġ$\ math bf Ġ{ W } _ g , Ġ\ math bf Ġ{ W } _ h $ Ġare Ġtrain able Ġweights . ĠFinally , Ġthe Ġpredicted Ġprobability Ġdistribution Ġover Ġthe Ġvocabulary Ġ$ V $ Ġis Ġcomputed Ġas : </s> where Ġ$\ math bf Ġ{ W } _ V $ Ġand Ġ$\ math bf Ġ{ b } _ V $ Ġare Ġparameters . </s> Fram ework ĠDescription Ġ: :: ĠOur ĠProp osed ĠModel Ġ: :: ĠDual ĠCopy ĠMechan ism . </s> To Ġdeal Ġwith Ġthe Ġrare Ġand Ġunknown Ġwords , Ġthe Ġdec oder Ġapplies Ġthe Ġpointing Ġmethod ĠB IB REF 10 , ĠB IB REF 11 , ĠB IB REF 12 Ġto Ġallow Ġcopying Ġa Ġtoken Ġfrom Ġthe Ġinput Ġsentence Ġat Ġthe Ġ$ t $ - th Ġdecoding Ġstep . ĠWe Ġreuse Ġthe Ġattention Ġscore Ġ$\ math bf Ġ{\ alpha Ġ} _{ t } ^ s $ Ġand Ġ$\ math bf Ġ{\ alpha Ġ} _{ t } ^ m $ Ġto Ġderive Ġthe Ġcopy Ġprobability Ġover Ġtwo Ġsource Ġinputs : </s> Different Ġfrom Ġthe Ġstandard Ġpointing Ġmethod , Ġwe Ġdesign Ġa Ġdual Ġcopy Ġmechanism Ġto Ġcopy Ġfrom Ġtwo Ġsources Ġwith Ġtwo Ġgates . ĠThe Ġfirst Ġgate Ġis Ġdesigned Ġfor Ġdetermining Ġcopy Ġtokens Ġfrom Ġtwo Ġsources Ġof Ġinputs Ġor Ġgenerate Ġnext Ġword Ġfrom Ġ$ P _ V $, Ġwhich Ġis Ġcomputed Ġas Ġ$ g ^ v _ t Ġ= Ġ\ text { s igm oid } (\ math bf Ġ{ w } ^ v _ g Ġ\ t ilde {\ math bf Ġ{ h }} _ t Ġ+ Ġb ^ v _ g ) $. ĠThe Ġsecond Ġgate Ġtakes Ġcharge Ġof Ġselecting Ġthe Ġsource Ġ( sent ence Ġor Ġrelation ) Ġto Ġcopy Ġfrom , Ġwhich Ġis Ġcomputed Ġas Ġ$ g ^ c _ t Ġ= Ġ\ text { s igm oid } (\ math bf Ġ{ w } ^ c _ g Ġ[ \ math bf Ġ{ c } _ t ^ s ; \ math bf Ġ{ c } _ t ^ m ] Ġ+ Ġb ^ c _ g ) $. ĠFinally , Ġwe Ġcombine Ġall Ġprobabilities Ġ$ P _ V $, Ġ$ P _ S $ Ġand Ġ$ P _ M $ Ġthrough Ġtwo Ġsoft Ġgates Ġ$ g ^ v _ t $ Ġand Ġ$ g ^ c _ t $. ĠThe Ġprobability Ġof Ġpredicting Ġ$ w $ Ġas Ġthe Ġ$ t $ - th Ġtoken Ġof Ġthe Ġquestion Ġis : </s> Fram ework ĠDescription Ġ: :: ĠOur ĠProp osed ĠModel Ġ: :: ĠTraining Ġand ĠIn ference . </s> Given Ġthe Ġanswer Ġ$ A $, Ġsentence Ġ$ S $ Ġand Ġrelation Ġ$ M $, Ġthe Ġtraining Ġobjective Ġis Ġto Ġminimize Ġthe Ġnegative Ġlog - like lihood Ġwith Ġregard Ġto Ġall Ġparameters : </s> where Ġ$\ math cal Ġ{\ l brace Ġ} Q \ r brace Ġ$ Ġis Ġthe Ġset Ġof Ġall Ġtraining Ġinstances , Ġ$\ the ta Ġ$ Ġdenotes Ġmodel Ġparameters Ġand Ġ$\ text { log } ĠP ( Q | A , S , M ; \ the ta Ġ) $ Ġis Ġthe Ġconditional Ġlog - like lihood Ġof Ġ$ Q $. </s> In Ġtesting , Ġour Ġmodel Ġtargets Ġto Ġgenerate Ġa Ġquestion Ġ$ Q $ Ġby Ġmaximizing : </s> Exper imental ĠSetting Ġ: :: ĠDat as et Ġ& ĠMet rics </s> We Ġconduct Ġexperiments Ġon Ġthe ĠS Qu AD Ġdataset ĠB IB REF 3 . ĠIt Ġcontains Ġ5 36 ĠWikipedia Ġarticles Ġand Ġ100 k Ġcrowd - s ourced Ġquestion - answer Ġpairs . ĠThe Ġquestions Ġare Ġwritten Ġby Ġcrowd - workers Ġand Ġthe Ġanswers Ġare Ġspans Ġof Ġtokens Ġin Ġthe Ġarticles . ĠWe Ġemploy Ġtwo Ġdifferent Ġdata Ġsplits Ġby Ġfollowing ĠZhou 2017 Ne ural Q G Ġand ĠDu 2017 Learning TA Ġ. ĠIn ĠZhou 2017 Ne ural Q G , Ġthe Ġoriginal ĠS Qu AD Ġdevelopment Ġset Ġis Ġevenly Ġdivided Ġinto Ġdev Ġand Ġtest Ġsets , Ġwhile ĠDu 2017 Learning TA Ġtreats ĠS Qu AD Ġdevelopment Ġset Ġas Ġits Ġdevelopment Ġset Ġand Ġsplits Ġoriginal ĠS Qu AD Ġtraining Ġset Ġinto Ġa Ġtraining Ġset Ġand Ġa Ġtest Ġset . ĠWe Ġalso Ġfilter Ġout Ġquestions Ġwhich Ġdo Ġnot Ġhave Ġany Ġoverl apped Ġnon - stop Ġwords Ġwith Ġthe Ġcorresponding Ġsentences Ġand Ġperform Ġsome Ġpre processing Ġsteps , Ġsuch Ġas Ġtoken ization Ġand Ġsentence Ġsplitting . ĠThe Ġdata Ġstatistics Ġare Ġgiven Ġin ĠTable ĠT AB REF 27 . </s> We Ġevaluate Ġwith Ġall Ġcommonly - used Ġmetrics Ġin Ġquestion Ġgeneration ĠB IB REF 13 : ĠB LE U - 1 Ġ( B 1 ), ĠB LE U - 2 Ġ( B 2 ), ĠB LE U - 3 Ġ( B 3 ), ĠB LE U - 4 Ġ( B 4 ) ĠB IB REF 17 , ĠMET E OR Ġ( MET ) ĠB IB REF 18 Ġand ĠR OU GE - L Ġ( R - L ) ĠB IB REF 19 . ĠWe Ġuse Ġthe Ġevaluation Ġscript Ġreleased Ġby ĠChen 2015 Microsoft CC . </s> Exper imental ĠSetting Ġ: :: ĠBas eline ĠModels </s> We Ġcompare Ġwith Ġthe Ġfollowing Ġmodels . </s> [ left margin = * ] </s> s 2 s ĠB IB REF 13 Ġproposes Ġan Ġattention - based Ġsequence - to - sequence Ġneural Ġnetwork Ġfor Ġquestion Ġgeneration . </s> N Q G ++ ĠB IB REF 9 Ġtakes Ġthe Ġanswer Ġposition Ġfeature Ġand Ġlinguistic Ġfeatures Ġinto Ġconsideration Ġand Ġequ ips Ġthe ĠSe q 2 Se q Ġmodel Ġwith Ġcopy Ġmechanism . </s> M 2 S + cp ĠB IB REF 14 Ġconducts Ġmulti - pers pect ive Ġmatching Ġbetween Ġthe Ġanswer Ġand Ġthe Ġsentence Ġto Ġderive Ġan Ġanswer - aware Ġsentence Ġrepresentation Ġfor Ġquestion Ġgeneration . </s> s 2 s + MP + G SA ĠB IB REF 8 Ġintroduces Ġa Ġg ated Ġself - att ention Ġinto Ġthe Ġenc oder Ġand Ġa Ġmax out Ġpointer Ġmechanism Ġinto Ġthe Ġdec oder . ĠWe Ġreport Ġtheir Ġsentence - level Ġresults Ġfor Ġa Ġfair Ġcomparison . </s> Hy brid ĠB IB REF 15 Ġis Ġa Ġhybrid Ġmodel Ġwhich Ġconsiders Ġthe Ġanswer Ġembed ding Ġfor Ġthe Ġquestion Ġword Ġgeneration Ġand Ġthe Ġposition Ġof Ġcontext Ġwords Ġfor Ġmodeling Ġthe Ġrelative Ġdistance Ġbetween Ġthe Ġcontext Ġwords Ġand Ġthe Ġanswer . </s> AS s 2 s ĠB IB REF 16 Ġreplaces Ġthe Ġanswer Ġin Ġthe Ġsentence Ġwith Ġa Ġspecial Ġtoken Ġto Ġavoid Ġits Ġappearance Ġin Ġthe Ġgenerated Ġquestions . </s> Exper imental ĠSetting Ġ: :: ĠImplementation ĠDetails </s> We Ġtake Ġthe Ġmost Ġfrequent Ġ20 k Ġwords Ġas Ġour Ġvocabulary Ġand Ġuse Ġthe ĠGl o Ve Ġword Ġembed d ings ĠB IB REF 20 Ġfor Ġinitialization . ĠThe Ġembed ding Ġdimensions Ġfor ĠPOS , ĠN ER , Ġanswer Ġposition Ġare Ġset Ġto Ġ20 . ĠWe Ġuse Ġtwo - layer ĠL ST Ms Ġin Ġboth Ġenc oder Ġand Ġdec oder , Ġand Ġthe ĠL ST Ms Ġhidden Ġunit Ġsize Ġis Ġset Ġto Ġ600 . </s> We Ġuse Ġdrop out ĠB IB REF 21 Ġwith Ġthe Ġprobability Ġ$ p = 0 . 3 $. ĠAll Ġtrain able Ġparameters , Ġexcept Ġword Ġembed d ings , Ġare Ġrandomly Ġinitialized Ġwith Ġthe ĠXavier Ġuniform Ġin Ġ$( - 0 . 1 , Ġ0 . 1 ) $ ĠB IB REF 22 . ĠFor Ġoptimization Ġin Ġthe Ġtraining , Ġwe Ġuse ĠSG D Ġas Ġthe Ġoptim izer Ġwith Ġa Ġmin ib atch Ġsize Ġof Ġ64 Ġand Ġan Ġinitial Ġlearning Ġrate Ġof Ġ1 . 0 . ĠWe Ġtrain Ġthe Ġmodel Ġfor Ġ15 Ġepoch s Ġand Ġstart Ġhal ving Ġthe Ġlearning Ġrate Ġafter Ġthe Ġ8 th Ġepoch . ĠWe Ġset Ġthe Ġgradient Ġnorm Ġupper Ġbound Ġto Ġ3 Ġduring Ġthe Ġtraining . </s> We Ġadopt Ġthe Ġteacher - forcing Ġfor Ġthe Ġtraining . ĠIn Ġthe Ġtesting , Ġwe Ġselect Ġthe Ġmodel Ġwith Ġthe Ġlowest Ġperplex ity Ġand Ġbeam Ġsearch Ġwith Ġsize Ġ3 Ġis Ġemployed Ġfor Ġgenerating Ġquestions . ĠAll Ġhyper - param eters Ġand Ġmodels Ġare Ġselected Ġon Ġthe Ġvalidation Ġdataset . </s> Results Ġand ĠAnalysis Ġ: :: ĠMain ĠResults </s> Table ĠT AB REF 30 Ġshows Ġautomatic Ġevaluation Ġresults Ġfor Ġour Ġmodel Ġand Ġbas elines Ġ( cop ied Ġfrom Ġtheir Ġpapers ). ĠOur Ġproposed Ġmodel Ġwhich Ġcombines Ġstructured Ġanswer - relevant Ġrelations Ġand Ġun struct ured Ġsentences Ġachieves Ġsignificant Ġimprovements Ġover Ġproximity - based Ġanswer - aware Ġmodels ĠB IB REF 9 , ĠB IB REF 15 Ġon Ġboth Ġdataset Ġsplits . ĠPresumably , Ġour Ġstructured Ġanswer - relevant Ġrelation Ġis Ġa Ġgeneral ization Ġof Ġthe Ġcontext Ġexplored Ġby Ġthe Ġproximity - based Ġmethods Ġbecause Ġthey Ġcan Ġonly Ġcapture Ġshort Ġdependencies Ġaround Ġanswer Ġfragments Ġwhile Ġour Ġextract ions Ġcan Ġcapture Ġboth Ġshort Ġand Ġlong Ġdependencies Ġgiven Ġthe Ġanswer Ġfragments . ĠMoreover , Ġour Ġproposed Ġframework Ġis Ġa Ġgeneral Ġone Ġto Ġjointly Ġleverage Ġstructured Ġrelations Ġand Ġun struct ured Ġsentences . ĠAll Ġcompared Ġbaseline Ġmodels Ġwhich Ġonly Ġconsider Ġun struct ured Ġsentences Ġcan Ġbe Ġfurther Ġenhanced Ġunder Ġour Ġframework . </s> Rec all Ġthat Ġexisting Ġproximity - based Ġanswer - aware Ġmodels Ġperform Ġpoorly Ġwhen Ġthe Ġdistance Ġbetween Ġthe Ġanswer Ġfragment Ġand Ġother Ġnon - stop Ġsentence Ġwords Ġthat Ġalso Ġappear Ġin Ġthe Ġground Ġtruth Ġquestion Ġis Ġlarge Ġ( Table ĠT AB REF 2 ). ĠHere Ġwe Ġinvestigate Ġwhether Ġour Ġproposed Ġmodel Ġusing Ġthe Ġstructured Ġanswer - relevant Ġrelations Ġcan Ġalleviate Ġthis Ġissue Ġor Ġnot , Ġby Ġconducting Ġexperiments Ġfor Ġour Ġmodel Ġunder Ġthe Ġsame Ġsetting Ġas Ġin ĠTable ĠT AB REF 2 . ĠThe Ġbroken - down Ġperformances Ġby Ġdifferent Ġrelative Ġdistances Ġare Ġshown Ġin ĠTable ĠT AB REF 40 . ĠWe Ġfind Ġthat Ġour Ġproposed Ġmodel Ġoutper forms ĠHybrid Ġ( our Ġre - im ple mented Ġversion Ġfor Ġthis Ġexperiment ) Ġon Ġall Ġranges Ġof Ġrelative Ġdistances , Ġwhich Ġshows Ġthat Ġthe Ġstructured Ġanswer - relevant Ġrelations Ġcan Ġcapture Ġboth Ġshort Ġand Ġlong Ġterm Ġanswer - relevant Ġdependencies Ġof Ġthe Ġanswer Ġin Ġsentences . ĠFurthermore , Ġcomparing Ġthe Ġperformance Ġdifference Ġbetween ĠHybrid Ġand Ġour Ġmodel , Ġwe Ġfind Ġthe Ġimprovements Ġbecome Ġmore Ġsignificant Ġwhen Ġthe Ġdistance Ġincreases Ġfrom ĠâĢ ľ $ 0 \ sim Ġ10 $ âĢ Ŀ Ġto ĠâĢ ľ $ > 10 $ âĢ Ŀ . ĠOne Ġreason Ġis Ġthat Ġour Ġmodel Ġcan Ġextract Ġrelations Ġwith Ġdistant Ġdependencies Ġto Ġthe Ġanswer , Ġwhich Ġgreatly Ġhelps Ġour Ġmodel Ġignore Ġthe Ġextr aneous Ġinformation . ĠPro x imity - based Ġanswer - aware Ġmodels Ġmay Ġoverly Ġemphasize Ġthe Ġneighboring Ġwords Ġof Ġanswers Ġand Ġbecome Ġless Ġeffective Ġas Ġthe Ġuseful Ġcontext Ġbecomes Ġfurther Ġaway Ġfrom Ġthe Ġanswer Ġin Ġthe Ġcomplex Ġsentences . ĠIn Ġfact , Ġthe Ġbreakdown Ġintervals Ġin ĠTable ĠT AB REF 40 Ġnaturally Ġbound Ġits Ġsentence Ġlength , Ġsay Ġfor ĠâĢ ľ $ > 10 $ âĢ Ŀ , Ġthe Ġsentences Ġin Ġthis Ġgroup Ġmust Ġbe Ġlonger Ġthan Ġ10 . ĠThus , Ġthe Ġlength Ġvar iances Ġin Ġthese Ġtwo Ġintervals Ġcould Ġbe Ġsignificant . ĠTo Ġfurther Ġvalidate Ġwhether Ġour Ġmodel Ġcan Ġextract Ġlong Ġterm Ġdependency Ġwords . ĠWe Ġre run Ġthe Ġanalysis Ġof ĠTable ĠT AB REF 40 Ġonly Ġfor Ġlong Ġsentences Ġ( length Ġ$ > $ Ġ20 ) Ġof Ġeach Ġinterval . ĠThe Ġimprovement Ġpercentages Ġover ĠHybrid Ġare Ġshown Ġin ĠTable ĠT AB REF 40 , Ġwhich Ġbecome Ġmore Ġsignificant Ġwhen Ġthe Ġdistance Ġincreases Ġfrom ĠâĢ ľ $ 0 \ sim Ġ10 $ âĢ Ŀ Ġto ĠâĢ ľ $ > 10 $ âĢ Ŀ . </s> Results Ġand ĠAnalysis Ġ: :: ĠCase ĠStudy </s> Figure ĠFIG REF 42 Ġprovides Ġexample Ġquestions Ġgenerated Ġby Ġcrowd - workers Ġ( ground Ġtruth Ġquestions ), Ġthe Ġbaseline ĠHybrid ĠB IB REF 15 , Ġand Ġour Ġmodel . ĠIn Ġthe Ġfirst Ġcase , Ġthere Ġare Ġtwo Ġsubsequ ences Ġin Ġthe Ġinput Ġand Ġthe Ġanswer Ġhas Ġno Ġrelation Ġwith Ġthe Ġsecond Ġsubsequ ence . ĠHowever , Ġwe Ġsee Ġthat Ġthe Ġbaseline Ġmodel Ġprediction Ġcopies Ġirrelevant Ġwords ĠâĢ ľ The ĠNew ĠYork ĠTimes âĢ Ŀ Ġwhile Ġour Ġmodel Ġcan Ġavoid Ġusing Ġthe Ġextr aneous Ġsubsequ ence ĠâĢ ľ The ĠNew ĠYork ĠTimes Ġnoted Ġ... âĢ Ŀ Ġwith Ġthe Ġhelp Ġof Ġthe Ġstructured Ġanswer - relevant Ġrelation . ĠCompared Ġwith Ġthe Ġground Ġtruth Ġquestion , Ġour Ġmodel Ġcannot Ġcapture Ġthe Ġcross - sent ence Ġinformation Ġlike ĠâĢ ľ her Ġfifth Ġalbum âĢ Ŀ , Ġwhere Ġthe Ġtechniques Ġin Ġparagraph - level ĠQ G Ġmodels ĠB IB REF 8 Ġmay Ġhelp . ĠIn Ġthe Ġsecond Ġcase , Ġas Ġdiscussed Ġin ĠSection ĠSEC REF 1 , Ġthis Ġsentence Ġcontains Ġa Ġfew Ġfacts Ġand Ġsome Ġfacts Ġintertw ine . ĠWe Ġfind Ġthat Ġour Ġmodel Ġcan Ġcapture Ġdistant Ġanswer - relevant Ġdependencies Ġsuch Ġas ĠâĢ ľ mean Ġtemperature âĢ Ŀ Ġwhile Ġthe Ġproximity - based Ġbaseline Ġmodel Ġwrongly Ġtakes Ġneighboring Ġwords Ġof Ġthe Ġanswer Ġlike ĠâĢ ľ cold est âĢ Ŀ Ġin Ġthe Ġgenerated Ġquestion . </s> Results Ġand ĠAnalysis Ġ: :: ĠD iverse ĠQuestion ĠGeneration </s> Another Ġinteresting Ġobservation Ġis Ġthat Ġfor Ġthe Ġsame Ġanswer - sent ence Ġpair , Ġour Ġmodel Ġcan Ġgenerate Ġdiverse Ġquestions Ġby Ġtaking Ġdifferent Ġanswer - relevant Ġrelations Ġas Ġinput . ĠSuch Ġcapability Ġimproves Ġthe Ġinterpret ability Ġof Ġour Ġmodel Ġbecause Ġthe Ġmodel Ġis Ġgiven Ġnot Ġonly Ġwhat Ġto Ġbe Ġasked Ġ( i . e ., Ġthe Ġanswer ) Ġbut Ġalso Ġthe Ġrelated Ġfact Ġ( i . e ., Ġthe Ġanswer - relevant Ġrelation ) Ġto Ġbe Ġcovered Ġin Ġthe Ġquestion . ĠIn Ġcontrast , Ġproximity - based Ġanswer - aware Ġmodels Ġcan Ġonly Ġgenerate Ġone Ġquestion Ġgiven Ġthe Ġsentence - answer Ġpair Ġregardless Ġof Ġhow Ġmany Ġanswer - relevant Ġrelations Ġin Ġthe Ġsentence . ĠWe Ġthink Ġsuch Ġcapability Ġcan Ġalso Ġvalidate Ġour Ġmotivation : Ġquestions Ġshould Ġbe Ġgenerated Ġaccording Ġto Ġthe Ġanswer - aware Ġrelations Ġinstead Ġof Ġneighboring Ġwords Ġof Ġanswer Ġfragments . ĠFigure ĠFIG REF 45 Ġshow Ġtwo Ġexamples Ġof Ġdiverse Ġquestion Ġgeneration . ĠIn Ġthe Ġfirst Ġcase , Ġthe Ġanswer Ġfragment Ġ` H ugh ĠL . ĠDry den ' Ġis Ġthe Ġapp os itive Ġto Ġ` NASA ĠDeputy ĠAdministrator ' Ġbut Ġthe Ġsubject Ġto Ġthe Ġfollowing Ġtokens Ġ` announced Ġthe ĠApollo Ġprogram Ġ... '. ĠOur Ġframework Ġcan Ġextract Ġthese Ġtwo Ġanswer - relevant Ġrelations , Ġand Ġby Ġfeeding Ġthem Ġto Ġour Ġmodel Ġseparately , Ġwe Ġcan Ġreceive Ġtwo Ġquestions Ġasking Ġdifferent Ġrelations Ġwith Ġregard Ġto Ġthe Ġanswer . </s> Related ĠWork </s> The Ġtopic Ġof Ġquestion Ġgeneration , Ġinitially Ġmotivated Ġfor Ġeducational Ġpurposes , Ġis Ġtackled Ġby Ġdesigning Ġmany Ġcomplex Ġrules Ġfor Ġspecific Ġquestion Ġtypes ĠB IB REF 4 , ĠB IB REF 23 . ĠHe il man 2010 Good Q S Ġimprove Ġrule - based Ġquestion Ġgeneration Ġby Ġintroducing Ġa Ġstatistical Ġranking Ġmodel . ĠFirst , Ġthey Ġremove Ġextr aneous Ġinformation Ġin Ġthe Ġsentence Ġto Ġtransform Ġit Ġinto Ġa Ġsimpler Ġone , Ġwhich Ġcan Ġbe Ġtransformed Ġeasily Ġinto Ġa Ġsuccinct Ġquestion Ġwith Ġpred efined Ġsets Ġof Ġgeneral Ġrules . ĠThen Ġthey Ġadopt Ġan Ġover gener ate - and - rank Ġapproach Ġto Ġselect Ġthe Ġbest Ġcandidate Ġconsidering Ġseveral Ġfeatures . </s> With Ġthe Ġrise Ġof Ġdominant Ġneural Ġsequence - to - sequence Ġlearning Ġmodels ĠB IB REF 5 , ĠDu 2017 Learning TA Ġframe Ġquestion Ġgeneration Ġas Ġa Ġsequence - to - sequence Ġlearning Ġproblem . ĠCompared Ġwith Ġrule - based Ġapproaches , Ġneural Ġmodels ĠB IB REF 24 Ġcan Ġgenerate Ġmore Ġfluent Ġand Ġgram matical Ġquestions . ĠHowever , Ġquestion Ġgeneration Ġis Ġa Ġone - to - many Ġsequence Ġgeneration Ġproblem , Ġi . e ., Ġseveral Ġaspects Ġcan Ġbe Ġasked Ġgiven Ġa Ġsentence , Ġwhich Ġconf uses Ġthe Ġmodel Ġduring Ġtrain Ġand Ġprevents Ġconcrete Ġautomatic Ġevaluation . ĠTo Ġtackle Ġthis Ġissue , ĠZhou 2017 Ne ural Q G Ġpropose Ġthe Ġanswer - aware Ġquestion Ġgeneration Ġsetting Ġwhich Ġassumes Ġthe Ġanswer Ġis Ġalready Ġknown Ġand Ġacts Ġas Ġa Ġcontiguous Ġspan Ġinside Ġthe Ġinput Ġsentence . ĠThey Ġadopt Ġa ĠB IO Ġtagging Ġscheme Ġto Ġincorporate Ġthe Ġanswer Ġposition Ġinformation Ġas Ġlearned Ġembed ding Ġfeatures Ġin ĠSe q 2 Se q Ġlearning . ĠSong 2018 L ever aging CI Ġexplicitly Ġmodel Ġthe Ġinformation Ġbetween Ġanswer Ġand Ġsentence Ġwith Ġa Ġmulti - pers pect ive Ġmatching Ġmodel . ĠKim 2019 Impro ving N Q Ġalso Ġfocus Ġon Ġthe Ġanswer Ġinformation Ġand Ġproposed Ġan Ġanswer - separ ated ĠSe q 2 Se q Ġmodel Ġby Ġmask ing Ġthe Ġanswer Ġwith Ġspecial Ġtokens . ĠAll Ġanswer - aware Ġneural Ġmodels Ġtreat Ġquestion Ġgeneration Ġas Ġa Ġone - to - one Ġmapping Ġproblem , Ġbut Ġexisting Ġmodels Ġperform Ġpoorly Ġfor Ġsentences Ġwith Ġa Ġcomplex Ġstructure Ġ( as Ġshown Ġin ĠTable ĠT AB REF 2 ). </s> Our Ġwork Ġis Ġinspired Ġby Ġthe Ġprocess Ġof Ġextr aneous Ġinformation Ġremoving Ġin ĠB IB REF 0 , ĠB IB REF 25 . ĠDifferent Ġfrom ĠHe il man 2010 Good Q S Ġwhich Ġdirectly Ġuse Ġthe Ġsimplified Ġsentence Ġfor Ġgeneration Ġand Ġca o 2018 faith ful Ġwhich Ġonly Ġconsider Ġaggregate Ġtwo Ġsources Ġof Ġinformation Ġvia Ġg ated Ġattention Ġin Ġsummar ization , Ġwe Ġpropose Ġto Ġcombine Ġthe Ġstructured Ġanswer - relevant Ġrelation Ġand Ġthe Ġoriginal Ġsentence . ĠFact oid Ġquestion Ġgeneration Ġfrom Ġstructured Ġtext Ġis Ġinitially Ġinvestigated Ġby ĠSer ban 2016 Gener ating F Q , Ġbut Ġour Ġfocus Ġhere Ġis Ġleveraging Ġstructured Ġinputs Ġto Ġhelp Ġquestion Ġgeneration Ġover Ġun struct ured Ġsentences . ĠOur Ġproposed Ġmodel Ġcan Ġtake Ġadvantage Ġof Ġun struct ured Ġsentences Ġand Ġstructured Ġanswer - relevant Ġrelations Ġto Ġmaintain Ġinform at iveness Ġand Ġfaith fulness Ġof Ġgenerated Ġquestions . ĠThe Ġproposed Ġmodel Ġcan Ġalso Ġbe Ġgeneralized Ġin Ġother Ġconditional Ġsequence Ġgeneration Ġtasks Ġwhich Ġrequire Ġmultiple Ġsources Ġof Ġinputs , Ġe . g ., Ġdistract or Ġgeneration Ġfor Ġmultiple Ġchoice Ġquestions ĠB IB REF 26 . </s> Con clusions Ġand ĠFuture ĠWork </s> In Ġthis Ġpaper , Ġwe Ġpropose Ġa Ġquestion Ġgeneration Ġsystem Ġwhich Ġcombines Ġun struct ured Ġsentences Ġand Ġstructured Ġanswer - relevant Ġrelations Ġfor Ġgeneration . ĠThe Ġun struct ured Ġsentences Ġmaintain Ġthe Ġinform at iveness Ġof Ġgenerated Ġquestions Ġwhile Ġstructured Ġanswer - relevant Ġrelations Ġkeep Ġthe Ġfaith fulness Ġof Ġquestions . ĠExt ensive Ġexperiments Ġdemonstrate Ġthat Ġour Ġproposed Ġmodel Ġachieves Ġstate - of - the - art Ġperformance Ġacross Ġseveral Ġmetrics . ĠFurthermore , Ġour Ġmodel Ġcan Ġgenerate Ġdiverse Ġquestions Ġwith Ġdifferent Ġstructured Ġanswer - relevant Ġrelations . ĠFor Ġfuture Ġwork , Ġthere Ġare Ġsome Ġinteresting Ġdimensions Ġto Ġexplore , Ġsuch Ġas Ġdifficulty Ġlevels ĠB IB REF 27 , Ġparagraph - level Ġinformation ĠB IB REF 8 Ġand Ġconvers ational Ġquestion Ġgeneration ĠB IB REF 28 . </s> Acknowled gments </s> This Ġwork Ġis Ġsupported Ġby Ġthe ĠResearch ĠGrants ĠCouncil Ġof Ġthe ĠHong ĠKong ĠSpecial ĠAdministrative ĠRegion , ĠChina Ġ( No . ĠCU HK Ġ14 20 88 15 Ġand ĠNo . ĠCU HK Ġ142 107 17 Ġof Ġthe ĠGeneral ĠResearch ĠFund ). ĠWe Ġwould Ġlike Ġto Ġthank Ġthe Ġanonymous Ġreviewers Ġfor Ġtheir Ġcomments . ĠWe Ġwould Ġalso Ġlike Ġto Ġthank ĠDepartment Ġof ĠComputer ĠScience Ġand ĠEngineering , ĠThe ĠChinese ĠUniversity Ġof ĠHong ĠKong Ġfor Ġthe Ġconference Ġgrant Ġsupport .