The question generation model provides each candidate answer with a score by measuring semantic relevance between the question and the generated question based on the semantics of the candidate answer. 
EXTRACTIVE
framework consisting of both a question answering model and a question generation model
EXTRACTIVE
We implement a framework consisting of both a question answering model and a question generation model, both of which take the knowledge extracted from the document as well as relevant facts from an external knowledge base such as Freebase/ProBase/Reverb/NELL. The question answering model gives each candidate answer a score by measuring the semantic relevance between representation and the candidate answer representation in vector space. The question generation model provides each candidate answer with a score by measuring semantic relevance between the question and the generated question based on the semantics of the candidate answer. We implement an MRC model BiDAF BIBREF10 as a baseline for the proposed dataset. To test the scalability of our approach in leveraging external KBs, we use both manually created and automatically extracted KBs, including Freebase BIBREF11 , ProBase BIBREF12 , NELL BIBREF13 and Reverb BIBREF14 . Experiments show that incorporating evidence from external KBs improves both the matching-based and question generation-based approaches. Qualitative analysis shows the advantages and limitations of our approaches, as well as the remaining challenges.
We implement a framework consisting of both a question answering model and a question generation model, both of which take the knowledge extracted from the document as well as relevant facts from an external knowledge base such as Freebase/ProBase/Reverb/NELL. The question answering model gives each candidate answer a score by measuring the semantic relevance between representation and the candidate answer representation in vector space. The question generation model provides each candidate answer with a score by measuring semantic relevance between the question and the generated question based on the semantics of the candidate answer. We implement an MRC model BiDAF BIBREF10 as a baseline for the proposed dataset. To test the scalability of our approach in leveraging external KBs, we use both manually created and automatically extracted KBs, including Freebase BIBREF11 , ProBase BIBREF12 , NELL BIBREF13 and Reverb BIBREF14 . Experiments show that incorporating evidence from external KBs improves both the matching-based and question generation-based approaches. Qualitative analysis shows the advantages and limitations of our approaches, as well as the remaining challenges.
1809.04267
Introduction </s> In Ġrecent Ġyears , Ġthere Ġhas Ġbeen Ġan Ġincreasing Ġinterest Ġin ĠMachine Ġreading Ġcomprehension Ġ( M RC ), Ġwhich Ġplays Ġa Ġvital Ġrole Ġin Ġthe Ġassessment Ġof Ġhow Ġwell Ġa Ġmachine Ġcould Ġunderstand Ġnatural Ġlanguage . ĠSeveral Ġdatasets ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġfor Ġmachine Ġreading Ġcomprehension Ġhave Ġbeen Ġreleased Ġin Ġrecent Ġyears Ġand Ġhave Ġdriven Ġthe Ġevolution Ġof Ġpowerful Ġneural Ġmodels . ĠHowever , Ġmuch Ġof Ġthe Ġresearch Ġup Ġto Ġnow Ġhas Ġbeen Ġdominated Ġby Ġanswering Ġquestions Ġthat Ġcan Ġbe Ġwell Ġsolved Ġsolved Ġusing Ġsuperficial Ġinformation , Ġyet Ġstruggles Ġto Ġdo Ġaccurate Ġnatural Ġlanguage Ġunderstanding Ġand Ġreasoning . ĠFor Ġexample , ĠB IB REF 3 Ġj ia 2017 Ad vers arial Ġshow Ġthat Ġexisting Ġmachine Ġlearning Ġsystems Ġfor ĠM RC Ġperform Ġpoorly Ġunder Ġadvers arial Ġevaluation . ĠRecent Ġdevelopments Ġin ĠM RC Ġdatasets ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġhave Ġheightened Ġthe Ġneed Ġfor Ġdeep Ġunderstanding . </s> Know ledge Ġhas Ġa Ġpivotal Ġrole Ġin Ġaccurately Ġunderstanding Ġand Ġreasoning Ġnatural Ġlanguage Ġin ĠM RC . ĠPrevious Ġresearch ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġhas Ġestablished Ġthat Ġhuman Ġreading Ġcomprehension Ġrequires Ġboth Ġwords Ġand Ġworld Ġknowledge . ĠIn Ġthis Ġpaper , Ġwe Ġconsider Ġwords Ġand Ġworld Ġknowledge Ġin Ġthe Ġformat Ġof Ġtriple ts Ġ( subject , Ġpredicate , Ġobject ). ĠSpecifically , Ġwe Ġbelieve Ġthe Ġadvantages Ġof Ġusing Ġknowledge Ġin ĠM RC Ġare Ġthree - fold . ĠFirst , Ġutilizing Ġknowledge Ġin ĠM RC Ġsupports Ġreasoning Ġover Ġmultiple Ġtriple ts Ġbecause Ġa Ġsingle Ġtriple t Ġmay Ġnot Ġcover Ġthe Ġentire Ġquestion . ĠMulti - hop Ġreasoning Ġis Ġalso Ġa Ġlong - standing Ġgoal Ġin Ġquestion Ġanswering . ĠSecond , Ġbuilding Ġa Ġquestion Ġanswering Ġsystem Ġbased Ġon Ġtriple t - style Ġknowledge Ġfacilitates Ġthe Ġinterpret ability Ġof Ġthe Ġdecision Ġmaking Ġprocess . ĠTriple ts Ġorganize Ġthe Ġdocument Ġtogether Ġwith ĠKB s Ġas Ġa Ġgraph , Ġwhere Ġa Ġwell - designed Ġmodel Ġsuch Ġas ĠPC Net , Ġwhich Ġwe Ġwill Ġdescribe Ġin Ġa Ġlater Ġsection , Ġexpressly Ġreveal Ġrational es Ġfor Ġtheir Ġpredictions . ĠThird , Ġrepresenting Ġthe Ġdocuments Ġas Ġknowledge Ġallows Ġfor Ġease Ġof Ġaccessing Ġand Ġleveraging Ġthe Ġknowledge Ġfrom Ġexternal / background Ġknowledge Ġbecause Ġthe Ġknowledge Ġrepresentation Ġof Ġa Ġdocument Ġis Ġeasily Ġconsistent Ġwith Ġboth Ġmanually Ġcurated Ġand Ġautomatically Ġextracted ĠKB s . </s> In Ġthis Ġpaper , Ġwe Ġpresent Ġknowledge Ġbased Ġmachine Ġreading Ġcomprehension , Ġwhich Ġrequires Ġreasoning Ġover Ġtriple t - style Ġknowledge Ġinvolved Ġin Ġa Ġdocument . ĠHowever , Ġwe Ġfind Ġpublished Ġdataset Ġdo Ġnot Ġsufficiently Ġsupport Ġthis Ġtask . ĠWe Ġconduct Ġpreliminary Ġexploration Ġon ĠS Qu AD ĠB IB REF 0 Ġ. ĠWe Ġuse Ġa Ġstrong Ġopen ĠIE Ġalgorithm ĠB IB REF 9 Ġto Ġextract Ġtriple ts Ġfrom Ġthe Ġdocuments Ġand Ġobserve Ġthat Ġonly Ġ15 % Ġof Ġinstances Ġhave Ġan Ġanswer Ġthat Ġis Ġexactly Ġthe Ġsame Ġas Ġthe Ġcorresponding Ġsubject / object Ġin Ġthe Ġextracted Ġtriple ts . ĠTo Ġdo Ġknowledge - based ĠM RC , ĠWe Ġbuild Ġa Ġnew Ġdataset Ġconsisting Ġof Ġ40 , 047 Ġexamples Ġfor Ġthe Ġknowledge Ġbased ĠM RC Ġtask . ĠThe Ġannotation Ġof Ġthis Ġdataset Ġis Ġdesigned Ġso Ġthat Ġsuccessfully Ġanswering Ġthe Ġquestions Ġrequires Ġunderstanding Ġand Ġthe Ġknowledge Ġinvolved Ġin Ġa Ġdocument . ĠEach Ġinstance Ġis Ġcomposed Ġof Ġa Ġquestion , Ġa Ġset Ġof Ġtriple ts Ġderived Ġfrom Ġa Ġdocument , Ġand Ġthe Ġanswer . </s> We Ġimplement Ġa Ġframework Ġconsisting Ġof Ġboth Ġa Ġquestion Ġanswering Ġmodel Ġand Ġa Ġquestion Ġgeneration Ġmodel , Ġboth Ġof Ġwhich Ġtake Ġthe Ġknowledge Ġextracted Ġfrom Ġthe Ġdocument Ġas Ġwell Ġas Ġrelevant Ġfacts Ġfrom Ġan Ġexternal Ġknowledge Ġbase Ġsuch Ġas ĠFree base / Pro Base / R ever b / NE LL . ĠThe Ġquestion Ġanswering Ġmodel Ġgives Ġeach Ġcandidate Ġanswer Ġa Ġscore Ġby Ġmeasuring Ġthe Ġsemantic Ġrelevance Ġbetween Ġrepresentation Ġand Ġthe Ġcandidate Ġanswer Ġrepresentation Ġin Ġvector Ġspace . ĠThe Ġquestion Ġgeneration Ġmodel Ġprovides Ġeach Ġcandidate Ġanswer Ġwith Ġa Ġscore Ġby Ġmeasuring Ġsemantic Ġrelevance Ġbetween Ġthe Ġquestion Ġand Ġthe Ġgenerated Ġquestion Ġbased Ġon Ġthe Ġsemantics Ġof Ġthe Ġcandidate Ġanswer . ĠWe Ġimplement Ġan ĠM RC Ġmodel ĠBi DA F ĠB IB REF 10 Ġas Ġa Ġbaseline Ġfor Ġthe Ġproposed Ġdataset . ĠTo Ġtest Ġthe Ġscal ability Ġof Ġour Ġapproach Ġin Ġleveraging Ġexternal ĠKB s , Ġwe Ġuse Ġboth Ġmanually Ġcreated Ġand Ġautomatically Ġextracted ĠKB s , Ġincluding ĠFree base ĠB IB REF 11 Ġ, ĠPro Base ĠB IB REF 12 Ġ, ĠNE LL ĠB IB REF 13 Ġand ĠRe verb ĠB IB REF 14 Ġ. ĠExper iments Ġshow Ġthat Ġincorporating Ġevidence Ġfrom Ġexternal ĠKB s Ġimproves Ġboth Ġthe Ġmatching - based Ġand Ġquestion Ġgeneration - based Ġapproaches . ĠQual itative Ġanalysis Ġshows Ġthe Ġadvantages Ġand Ġlimitations Ġof Ġour Ġapproaches , Ġas Ġwell Ġas Ġthe Ġremaining Ġchallenges . </s> Task ĠDefinition Ġand ĠDat as et </s> We Ġformulate Ġthe Ġtask Ġof Ġknowledge Ġbased Ġmachine Ġreading Ġcomprehension , Ġwhich Ġis Ġabbrevi ated Ġas ĠK BM RC , Ġand Ġdescribe Ġthe Ġdataset Ġbuilt Ġfor ĠK BM RC Ġand Ġthe Ġexternal Ġopen ĠKB s Ġlever aged Ġin Ġthis Ġwork . </s> App roach ĠOverview </s> Our Ġframework Ġconsists Ġof Ġa Ġquestion Ġanswering Ġmodel Ġand Ġa Ġquestion Ġgeneration Ġmodel . </s> We Ġimplement Ġtwo Ġquestion Ġanswering Ġmodels , Ġwhich Ġdirectly Ġmeasure Ġthe Ġsemantic Ġsimilarity Ġbetween Ġquestions Ġand Ġcandidate Ġanswers Ġin Ġthe Ġsemantic Ġspace . ĠFirst , Ġto Ġmake Ġthe Ġmodel 's Ġprediction Ġmore Ġexplain able , Ġwe Ġimplement Ġa Ġpath Ġbased ĠQ A Ġmodel ĠPC Net . ĠIn Ġthis Ġmodel , Ġto Ġget Ġcandidate Ġanswers Ġfrom Ġthe Ġtriple ts Ġbased Ġdocument Ġfor Ġa Ġgiven Ġquestion . ĠWe Ġfirst Ġretrieve Ġthe Ġan ĠâĢ ľ anch or âĢ Ŀ Ġpoint Ġ$ arg 1 _ i $ Ġor Ġ$ arg 2 _ i $ Ġin Ġthe Ġdocument Ġfact Ġ$ f _ i $ Ġ. ĠThese Ġanchors Ġare Ġselected Ġbased Ġon Ġedit Ġdistance Ġbetween Ġthe Ġwords Ġin Ġthe Ġquestions Ġand Ġthe Ġarguments . ĠThen Ġwe Ġregard Ġall Ġarguments Ġin Ġthe Ġ1 - hops Ġand Ġ2 - hops Ġfact Ġof Ġthe Ġanchors Ġas Ġanswers . ĠHowever , Ġthe Ġcoverage Ġof Ġthe Ġcandidate Ġanswers Ġcan Ġbe Ġ100 % Ġin Ġthe Ġmodel . ĠWe Ġthen Ġimplement Ġa Ġsecond Ġend - to - end Ġneural Ġmodel ĠK V Men Net Ġwhich Ġcovers Ġall Ġthe Ġanswers Ġbut Ġwith Ġless Ġinterpret ability . ĠBoth Ġmodels Ġgenerate Ġa Ġscore Ġ$ f _{ qa } ( q , Ġa ) $ Ġof Ġeach Ġcandidate Ġanswer . </s> We Ġthen Ġimplement Ġa Ġgeneration - based Ġmodel . ĠThe Ġmotivation Ġto Ġdesign Ġthis Ġmodel Ġis Ġthat Ġwe Ġwant Ġto Ġassociate Ġnatural Ġlanguage Ġphrases Ġwith Ġknowledge Ġbased Ġrepresentation . ĠIt Ġtakes Ġsemantics Ġof Ġa Ġcandidate Ġanswer Ġas Ġthe Ġinput Ġand Ġgenerates Ġa Ġquestion Ġ$\ hat { q } $ Ġ. ĠThen Ġa Ġparaph r asing Ġmodel Ġgives Ġa Ġscore Ġ$ f _{ q g } ( q , \ hat { q }) $ Ġ, Ġwhich Ġis Ġcomputed Ġbetween Ġthe Ġgenerated Ġquestion Ġ$\ hat { q } $ Ġand Ġthe Ġoriginal Ġquestion Ġ$ q $ Ġ, Ġas Ġthe Ġranking Ġscore . </s> We Ġget Ġthe Ġfinal Ġscores Ġ$ S ( q , a ) $ Ġused Ġfor Ġranking Ġas Ġfollows . </s> $$ S ( q , a ) Ġ= Ġ\ lambda Ġf _{ qa } ( q , Ġa ) Ġ+ Ġ( 1 - \ lambda Ġ) Ġf _{ q g } ( q , \ hat { q }) $$ Ġ Ġ Ġ( E q . Ġ7 ) </s> Moreover , Ġwe Ġincorporate Ġside Ġinformation Ġfrom Ġexternal ĠKB s Ġinto Ġthe Ġthree Ġmodels . ĠThe Ġdetails Ġof Ġhow Ġwe Ġuse Ġexternal ĠKB s Ġto Ġenhance Ġthe Ġrepresentation Ġof Ġelements Ġin Ġthe Ġdocument ĠKB Ġwill Ġbe Ġdescribed Ġin Ġsection ĠInc orpor ating ĠExternal ĠKnowledge . </s> The ĠQuestion ĠAn sw ering ĠModel </s> In Ġthis Ġsection , Ġwe Ġpresent Ġtwo Ġmatching Ġmodels Ġto Ġmeasure Ġthe Ġsemantic Ġrelated ness Ġbetween Ġthe Ġquestion Ġand Ġthe Ġcandidate Ġanswer Ġin Ġthe Ġvector Ġsemantic Ġspace . ĠAfterwards , Ġwe Ġintroduce Ġour Ġstrategy Ġthat Ġincorporates Ġopen ĠKB Ġas Ġexternal Ġknowledge Ġto Ġenhance Ġboth Ġmodels . </s> Q A ĠModel Ġ1 : ĠPC Net </s> We Ġfollow ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġb ordes 2014 question , d ong - E t Al : 2015 : AC L - IJ CN LP 1 Ġand Ġdevelop ĠPC Net , Ġwhich Ġis Ġshort Ġfor Ġpath - Ġand Ġcontext - Ġbased Ġneural Ġnetwork . ĠIn ĠPC Net , Ġcandidate Ġanswers Ġcome Ġfrom Ġarguments Ġof Ġthe Ġdocument ĠKB , Ġand Ġeach Ġcandidate Ġanswer Ġis Ġrepresented Ġwith Ġits Ġneighboring Ġarguments Ġand Ġpred icates Ġas Ġwell Ġas Ġits Ġpath Ġfrom Ġthe Ġanchor Ġin Ġthe Ġdocument ĠKB . ĠWe Ġuse Ġa Ġrule - based Ġapproach Ġbased Ġon Ġstring Ġfuzzy Ġmatch Ġto Ġdetect Ġthe Ġanchor . ĠEach Ġargument Ġis Ġmeasured Ġby Ġ$\ sum Ġ_ { i , j } ^{ }\ math bb Ġ{ I } ( arg _ i , Ġq _ j ) $ Ġ, Ġwhere Ġ$ i $ Ġand Ġ$ j $ Ġiter ate Ġacross Ġargument Ġwords Ġand Ġquestion Ġwords , Ġrespectively . Ġ$\ math bb Ġ{ I } ( x , y ) $ Ġis Ġan Ġindicator Ġfunction Ġwhose Ġvalue Ġis Ġ1 Ġif Ġthe Ġminimum Ġedit Ġdistance Ġbetween Ġ$ x $ Ġand Ġ$ y $ Ġis Ġno Ġmore Ġthan Ġ1 , Ġotherwise Ġit Ġis Ġ0 . ĠThe Ġarguments Ġlinked Ġto Ġthe Ġanchor Ġwith Ġ1 - hop Ġand Ġ2 - hop Ġpaths Ġare Ġregarded Ġas Ġcandidate Ġanswers . ĠSince Ġan Ġargument Ġmight Ġinclude Ġmultiple Ġwords , Ġsuch Ġas ĠâĢ ľ the Ġpopular Ġangry Ġbird Ġgame âĢ Ŀ , Ġwe Ġuse ĠGR U Ġbased ĠR NN Ġto Ġget Ġthe Ġvector Ġrepresentation Ġof Ġeach Ġargument / pred icate . ĠThe Ġpath Ġrepresentation Ġ$ v _{ p } $ Ġis Ġcomputed Ġby Ġaveraging Ġthe Ġvectors Ġof Ġelements Ġin Ġthe Ġpath . ĠSimilarly , Ġwe Ġuse Ġanother ĠR NN Ġto Ġget Ġthe Ġvector Ġof Ġeach Ġneighboring Ġargument / pred icate , Ġand Ġaverage Ġthem Ġto Ġget Ġthe Ġcontext Ġvector Ġ$ v _{ c } $ Ġ. </s> We Ġrepresent Ġthe Ġquestion Ġ$ q $ Ġusing Ġa Ġbid irection al ĠR NN Ġlayer Ġbased Ġon Ġthe ĠGR U Ġunit . ĠThe Ġconc aten ation Ġof Ġthe Ġlast Ġhidden Ġvectors Ġfrom Ġboth Ġdirections Ġis Ġused Ġas Ġthe Ġquestion Ġvector Ġ$ v _{ q } $ Ġ. ĠThe Ġdot Ġproduct Ġis Ġused Ġto Ġmeasure Ġthe Ġsemantic Ġrelevance Ġbetween Ġthe Ġb Ġb Ġquestion Ġand Ġtwo Ġtypes Ġof Ġevidence . </s> $$ f _{ qa } ( q , Ġa ) Ġ= Ġv _{ q } ^{ T } v _{ p } Ġ+ Ġv _ q ^{ T } v _{ c } $$ Ġ Ġ Ġ( E q . Ġ10 ) </s> Q A ĠModel Ġ2 : ĠK V Mem Net </s> Despite Ġthe Ġinterpret ability Ġof ĠPC Net , Ġthe Ġcoverage Ġof Ġanchor Ġdetection Ġlimits Ġthe Ġupper Ġbound Ġof Ġthe Ġapproach . ĠWe Ġimplement Ġa Ġmore Ġpowerful Ġmethod Ġbased Ġon Ġthe Ġkey - value Ġmemory Ġnetwork ĠB IB REF 19 Ġ, ĠK V Mem Net Ġfor Ġshort , Ġwhich Ġhas Ġproven Ġpowerful Ġin ĠKB - based Ġquestion Ġanswering . </s> The ĠK V Men Net Ġcould Ġbe Ġviewed Ġas Ġa ĠâĢ ľ soft âĢ Ŀ Ġmatching Ġapproach , Ġwhich Ġincludes Ġa Ġparameter ized Ġmemory Ġconsisting Ġof Ġkey - value Ġpairs . ĠInt uitive ly , Ġkeys Ġare Ġused Ġfor Ġmatching Ġwith Ġthe Ġquestion , Ġand Ġvalues Ġare Ġused Ġfor Ġmatching Ġto Ġthe Ġcandidate Ġanswer . ĠGiven Ġa ĠKB Ġfact Ġ( Ġ$ sub j , Ġpred , Ġobj $ Ġ), ĠWe Ġconsider Ġboth Ġdirections Ġand Ġadd Ġtwo Ġkey - value Ġpairs Ġin Ġthe Ġmemory , Ġnamely Ġ( Ġ$ key Ġ= Ġsub j Ġ+ pred , Ġvalue = obj $ Ġ) Ġand Ġ( Ġ$ key Ġ= obj Ġ+ Ġpred , Ġvalue = sub j $ Ġ). ĠThe Ġvectors Ġof Ġarguments / pred icates Ġare Ġcalculated Ġthe Ġsame Ġway Ġas Ġdescribed Ġin ĠPC Net . ĠThe Ġconc aten ation Ġof Ġtwo Ġvectors Ġis Ġused Ġas Ġthe Ġkey Ġvector Ġ$ v _{ key } $ Ġ. </s> Each Ġmemory Ġitem Ġis Ġassigned Ġa Ġrelevance Ġprobability Ġby Ġcomparing Ġthe Ġquestion Ġto Ġeach Ġkey . </s> $$ \ alpha Ġ_ { key _ i } Ġ= Ġ{ soft max } ( v _ q Ġ\ cd ot Ġv _{ key _ i }) $$ Ġ Ġ Ġ( E q . Ġ12 ) </s> After wards , Ġvectors Ġin Ġmemory Ġ( Ġ$ v _{ value } $ Ġ) Ġare Ġweighted Ġsummed Ġaccording Ġto Ġtheir Ġaddressing Ġprobabilities , Ġand Ġthe Ġvector Ġ$ v _ o $ Ġis Ġreturned . </s> $$ v _ o Ġ= Ġ\ sum Ġ_ i Ġ\ alpha Ġ_ { key _ i } v _{ value _ i } $$ Ġ Ġ Ġ( E q . Ġ13 ) </s> In ĠPC Net , Ġreasoning Ġover Ġtwo Ġfacts Ġis Ġachieved Ġby Ġincorporating Ġ2 - hop Ġpaths , Ġwhile Ġin ĠK V Mem Net Ġthis Ġis Ġachieved Ġby Ġrepeating Ġthe Ġmemory Ġaccess Ġprocess Ġtwice . ĠAfter Ġreceiving Ġthe Ġresult Ġ$ v _ o $ Ġ, Ġwe Ġupdate Ġthe Ġquery Ġwith Ġ$ q _ 2 Ġ= ĠR ( v _ q Ġ+ Ġv _ o ) $ Ġ, Ġwhere Ġ$ R $ Ġis Ġmodel Ġparameter . ĠFinally , Ġafter Ġa Ġfixed Ġnumber Ġ$ n $ Ġhops Ġ( Ġ$ n = 2 $ Ġin Ġthis Ġwork ), Ġthe Ġresulting Ġvector Ġis Ġused Ġto Ġmeasure Ġthe Ġrelevance Ġto Ġcandidate Ġanswers Ġvia Ġthe Ġdot Ġproduct . </s> Training Ġand ĠIn ference </s> Let Ġ$\ math cal Ġ{ D } Ġ= Ġ\ l brace Ġ( q _ i , Ġa _ i ); Ġi Ġ= Ġ1 , Ġ\ ld ots Ġ, Ġ| \ math cal Ġ{ D } | \ r brace Ġ$ Ġbe Ġthe Ġtraining Ġdata Ġconsisting Ġof Ġquestions Ġ$ q _ i $ Ġpaired Ġwith Ġtheir Ġcorrect Ġanswer Ġ$ a _ i $ Ġ. ĠWe Ġtrain Ġboth Ġmatching Ġmodels Ġwith Ġa Ġmargin - based Ġranking Ġloss Ġfunction , Ġwhich Ġis Ġcalculated Ġas Ġfollows , Ġwhere Ġ$ m $ Ġis Ġthe Ġmargin Ġ( fixed Ġto Ġ$ 0 . 1 $ Ġ) Ġand Ġ$\ bar { a } $ Ġis Ġrandomly Ġsampled Ġfrom Ġa Ġset Ġof Ġincorrect Ġcandidates Ġ$\ bar {\ math cal Ġ{ A }} $ Ġ. </s> $$ Ġ Ġ\ sum Ġ_ { i = 1 } ^{ | \ math cal Ġ{ D } | } Ġ\ sum Ġ_ {\ tiny Ġ{\ bar { a } Ġ\ in Ġ\ bar {\ math cal Ġ{ A }} ( q _ i ) }} Ġ\ max Ġ\ l brace Ġ0 , Ġm Ġ- ĠS ( q _ i , a _ i ) Ġ+ ĠS ( q _ i , \ bar { a } )\ r brace Ġ, $$ Ġ Ġ Ġ( E q . Ġ15 ) </s> For Ġtesting , Ġgiven Ġa Ġquestion Ġ$ q $ Ġ, Ġthe Ġmodel Ġpredicts Ġthe Ġanswer Ġbased Ġon Ġthe Ġfollowing Ġequation , Ġwhere Ġ$ Ġ\ math cal Ġ{ A } ( q ) $ Ġis Ġthe Ġcandidate Ġanswer Ġset . </s> $$ \ hat { a } Ġ= Ġ{\ mb ox { arg max }} _{ a ^ {\ prime Ġ} Ġ\ in Ġ\ math cal Ġ{ A } ( q )} ĠS ( q , a ^ {\ prime Ġ}) $$ Ġ Ġ Ġ( E q . Ġ16 ) </s> The ĠQuestion ĠGeneration ĠModel </s> In Ġthis Ġsection , Ġwe Ġpresent Ġthe Ġgeneration Ġmodel Ġwhich Ġgenerates Ġa Ġquestion Ġbased Ġon Ġthe Ġsemantics Ġof Ġa Ġcandidate Ġanswer . ĠAfter ward , Ġwe Ġintroduce Ġhow Ġour Ġparaph r asing Ġmodel , Ġwhich Ġmeasures Ġthe Ġsemantic Ġrelevance Ġbetween Ġthe Ġgenerated Ġquestion Ġand Ġthe Ġoriginal Ġquestion , Ġis Ġpret rained . </s> H ier arch ical Ġseq 2 seq ĠGeneration ĠModel </s> Our Ġquestion Ġgeneration Ġmodel Ġtakes Ġthe Ġpath Ġbetween Ġan ĠâĢ ľ anch or âĢ Ŀ Ġand Ġthe Ġcandidate Ġanswer , Ġand Ġoutputs Ġa Ġquestion . ĠWe Ġadopt Ġthe Ġsequence Ġto Ġsequence Ġarchitecture Ġas Ġour Ġbasic Ġquestion Ġgeneration Ġmodel Ġdue Ġto Ġits Ġeffectiveness Ġin Ġnatural Ġlanguage Ġgeneration Ġtasks . ĠThe Ġhierarchical Ġenc oder Ġand Ġthe Ġhierarchical Ġattention Ġmechanism Ġare Ġintroduced Ġto Ġtake Ġinto Ġaccount Ġthe Ġstructure Ġof Ġthe Ġinput Ġfacts . ĠFacts Ġfrom Ġexternal ĠKB s Ġare Ġconvention ally Ġintegrated Ġinto Ġthe Ġmodel Ġusing Ġthe Ġsame Ġway Ġas Ġdescribed Ġin Ġthe Ġmatching Ġmodel . </s> As Ġillustrated Ġin ĠFigure Ġ3 Ġ, Ġthe Ġquestion Ġgeneration Ġmodel Ġcontains Ġan Ġenc oder Ġand Ġa Ġdec oder . ĠWe Ġuse Ġa Ġhierarchical Ġenc oder Ġconsisting Ġof Ġtwo Ġlayers Ġto Ġmodel Ġthe Ġmeaning Ġof Ġeach Ġelement Ġ( subject , Ġpredicate Ġor Ġobject ) Ġand Ġthe Ġrelationship Ġbetween Ġthem . ĠSince Ġeach Ġelement Ġmight Ġcontain Ġmultiple Ġwords , Ġwe Ġuse ĠR NN Ġas Ġthe Ġword Ġlayer Ġenc oder Ġto Ġget Ġthe Ġrepresentation Ġof Ġeach Ġelement . ĠWe Ġdefine Ġthe Ġfact Ġlevel Ġsequence Ġas Ġa Ġpath Ġstarting Ġfrom Ġthe Ġanchor Ġand Ġending Ġat Ġthe Ġcandidate Ġanswer . ĠAnother ĠR NN Ġis Ġused Ġas Ġthe Ġfact Ġlevel Ġenc oder . ĠThe Ġlast Ġhidden Ġstate Ġat Ġthe Ġfact Ġlayer Ġis Ġfed Ġto Ġthe Ġdec oder . </s> We Ġdevelop Ġa Ġhierarchical Ġattention Ġmechanism Ġin Ġthe Ġdec oder , Ġwhich Ġfirst Ġmakes Ġsoft Ġalignment Ġover Ġthe Ġhidden Ġstates Ġat Ġthe Ġfact Ġlayer , Ġthe Ġoutput Ġof Ġwhich Ġis Ġfurther Ġused Ġto Ġattend Ġto Ġhidden Ġstates Ġat Ġthe Ġword Ġlevel . ĠSpecifically , Ġgiven Ġthe Ġdec oder Ġhidden Ġstate Ġ$ h ^{ dec } _{ t - 1 } $ Ġ, Ġwe Ġuse Ġa Ġfact - level Ġattention Ġmodel Ġ$ Att _{ f ct } $ Ġto Ġcalculate Ġthe Ġcontextual Ġvector , Ġwhich Ġis Ġfurther Ġcombined Ġwith Ġthe Ġcurrent Ġhidden Ġstate Ġ$ h ^{ dec } _{ t } $ Ġ, Ġresulting Ġin Ġ$ c _{ f ct } $ Ġ. ĠThe Ġcontextual Ġvector Ġis Ġcalculated Ġthrough Ġweighted Ġaveraging Ġover Ġhidden Ġvectors Ġat Ġthe Ġfact Ġlevel , Ġwhich Ġis Ġgiven Ġas Ġfollows , Ġwhere Ġ$\ od ot Ġ$ Ġis Ġdot Ġproduct , Ġ$ h ^{ f ld } _ j $ Ġis Ġthe Ġ$ j $ Ġ- th Ġhidden Ġstate Ġat Ġthe Ġfact Ġlayer Ġfrom Ġthe Ġenc oder , Ġand Ġ$ l _ f $ Ġis Ġthe Ġnumber Ġof Ġhidden Ġstates Ġat Ġthe Ġfact Ġlevel . </s> $$ c _{ f ct } Ġ& = ĠGR U ( h ^{ dec } _{ t }, Ġ\ sum Ġ_ { j = 1 } ^ {{ l _ f }} Ġ\ alpha Ġ_ { t j } h ^{ f ct } _ j ) Ġ\\ Ġ\ alpha Ġ_ { t j } Ġ& = Ġ\ frac { Ġexp ( h ^{ dec } _{ t - 1 } Ġ\ od ot Ġh ^{ f ct } _ j ) }{ \ sum Ġ_ { k = 1 } ^ {{ l _ f }} exp ( h ^{ dec } _{ t - 1 } Ġ\ od ot Ġh ^{ f ct } _ k )} $$ Ġ Ġ Ġ( E q . Ġ19 ) </s> Similarly , Ġwe Ġfeed Ġ$ c _{ f ct } $ Ġto Ġthe Ġword - level Ġattention Ġfunction Ġ$ Att _{ w rd } $ Ġand Ġcalculate Ġover Ġhidden Ġvectors Ġat Ġthe Ġword - level . ĠThe Ġoutput Ġ$ c _{ w rd } $ Ġwill Ġbe Ġconc aten ated Ġwith Ġ$ h ^{ dec } _{ t } $ Ġto Ġpredict Ġthe Ġnext Ġword . </s> Since Ġmany Ġentity Ġnames Ġof Ġgreat Ġimportance Ġare Ġrare Ġwords Ġfrom Ġthe Ġinput , Ġwe Ġuse Ġthe Ġcopying Ġmechanism ĠB IB REF 20 Ġthat Ġlearns Ġwhen Ġto Ġreplicate Ġwords Ġfrom Ġthe Ġinput Ġor Ġto Ġpredict Ġwords Ġfrom Ġthe Ġtarget Ġvocabulary . ĠThe Ġprobability Ġdistribution Ġof Ġgenerating Ġthe Ġword Ġ$ y $ Ġis Ġcalculated Ġas Ġfollows , Ġin Ġwhich Ġthe Ġ$ soft max $ Ġfunction Ġis Ġcalculated Ġover Ġa Ġcombined Ġlog its Ġfrom Ġboth Ġsides . </s> $$ \ begin { split } Ġ& p ( y )= \ frac { exp ({ e _ y Ġ\ od ot ĠW _ g [ h _ t ^{ dec }; c _{ w rd } ] }) + exp ( s _ c ( y )) }{ Z } \\ Ġ& s _ c ( y ) Ġ= Ġc _{ w rd } Ġ\ od ot Ġ{ tan h ( W _ ch _ t ^{ w rd }) } Ġ\ end { split } $$ Ġ Ġ Ġ( E q . Ġ20 ) </s> We Ġtrain Ġour Ġquestion Ġgeneration Ġmodel Ġwith Ġmaximum Ġlikelihood Ġestimation . ĠThe Ġloss Ġfunction Ġis Ġgiven Ġas Ġfollows , Ġwhere Ġ$ D $ Ġis Ġthe Ġtraining Ġcorpus . ĠWe Ġuse Ġbeam Ġsearch Ġin Ġthe Ġinference Ġprocess . </s> $$ l Ġ= Ġ- \ sum Ġ_ { ( x , y )\ in ĠD } Ġ\ sum Ġ_ t Ġlog Ġp ( y _ t Ġ| y _{ < t }, Ġx ) $$ Ġ Ġ Ġ( E q . Ġ21 ) </s> An Ġadvantage Ġof Ġthe Ġmodel Ġis Ġthat Ġexternal Ġknowledge Ġcould Ġbe Ġeasily Ġincorporated Ġwith Ġthe Ġsame Ġmechanism Ġas Ġwe Ġhave Ġdescribed Ġin Ġsection ĠInc orpor ating ĠExternal ĠKnowledge . ĠWe Ġenhance Ġthe Ġrepresentation Ġof Ġan Ġargument Ġor Ġa Ġpredicate Ġby Ġconc aten ating Ġopen ĠKB Ġvectors Ġinto Ġenc oder Ġhidden Ġstates . </s> The ĠPar aph r asing ĠModel </s> The Ġparaph r asing Ġmodel Ġis Ġused Ġto Ġmeasure Ġthe Ġsemantic Ġrelevance Ġbetween Ġthe Ġoriginal Ġquestion Ġand Ġthe Ġquestion Ġgenerated Ġfrom Ġthe ĠQ G Ġmodel . ĠWe Ġuse Ġbid irection al ĠR NN Ġwith Ġg ated Ġrecurrent Ġunit Ġto Ġrepresent Ġtwo Ġquestions , Ġand Ġcompose Ġthem Ġwith Ġelement - wise Ġmultiplication . ĠThe Ġresults Ġare Ġfollowed Ġby Ġa Ġ$ soft max $ Ġlayer , Ġwhose Ġoutput Ġlength Ġis Ġ2 . ĠThe Ġmodel Ġis Ġtrained Ġby Ġminimizing Ġthe Ġcross - ent ropy Ġerror , Ġin Ġwhich Ġthe Ġsupervision Ġis Ġprovided Ġin Ġthe Ġtraining Ġdata . </s> We Ġcollect Ġtwo Ġdatasets Ġto Ġtrain Ġthe Ġparaph r asing Ġmodel . ĠThe Ġfirst Ġdataset Ġis Ġfrom ĠQu ora Ġdataset , Ġwhich Ġis Ġbuilt Ġfor Ġdetecting Ġwhether Ġor Ġnot Ġa Ġpair Ġof Ġquestions Ġare Ġsem antically Ġequivalent . Ġ345 , 989 Ġpositive Ġquestion Ġpairs Ġand Ġ255 , 027 Ġnegative Ġpairs Ġare Ġincluded Ġin Ġthe Ġfirst Ġdataset . ĠThe Ġsecond Ġdataset Ġincludes Ġweb Ġqueries Ġfrom Ġquery Ġlogs , Ġwhich Ġare Ġobtained Ġby Ġclust ering Ġthe Ġweb Ġqueries Ġthat Ġclick Ġthe Ġsame Ġweb Ġpage . ĠIn Ġthis Ġway Ġwe Ġobtain Ġ6 , 118 , 023 Ġpositive Ġquery Ġpairs . ĠWe Ġimplement Ġa Ġhe uristic Ġrule Ġto Ġget Ġ6 , 118 , 023 Ġnegative Ġinstances Ġfor Ġthe Ġquery Ġdataset . ĠFor Ġeach Ġpair Ġof Ġquery Ġtext , Ġwe Ġclamp Ġthe Ġfirst Ġquery Ġand Ġretrieve Ġa Ġquery Ġthat Ġis Ġmostly Ġsimilar Ġto Ġthe Ġsecond Ġquery . ĠTo Ġimprove Ġthe Ġefficiency Ġof Ġthis Ġprocess , Ġwe Ġrandomly Ġsample Ġ10 , 000 Ġqueries Ġand Ġdefine Ġthe ĠâĢ ľ similar ity âĢ Ŀ Ġas Ġthe Ġnumber Ġof Ġco - occ urred Ġwords Ġbetween Ġtwo Ġquestions . ĠDuring Ġtraining , Ġwe Ġinitialize Ġthe Ġvalues Ġof Ġword Ġembed d ings Ġwith Ġ$ 300 d $ ĠGl ove Ġvectors , Ġwhich Ġis Ġlearned Ġon ĠWikipedia Ġtexts . ĠWe Ġuse Ġa Ġheld - out Ġdata Ġconsisting Ġof Ġ20 K Ġquery Ġpairs Ġto Ġcheck Ġthe Ġperformance Ġof Ġthe Ġparaph r asing Ġmodel . ĠThe Ġaccuracy Ġof Ġthe Ġparaph r asing Ġmodel Ġon Ġthe Ġheld - out Ġdataset Ġis Ġ87 . 36 %. </s> In cor por ating ĠExternal ĠKnowledge </s> There Ġare Ġmany Ġpossible Ġways Ġto Ġimplement Ġthe Ġidea Ġof Ġimproving Ġquestion Ġanswering Ġwith Ġexternal ĠKB . ĠIn Ġthis Ġwork , Ġwe Ġuse Ġexternal ĠKB s Ġ( such Ġas ĠNE LL Ġand ĠPro Base ) Ġto Ġenhance Ġthe Ġrepresentations Ġof Ġelements Ġin Ġthe Ġdocument ĠKB . ĠFor Ġinstance , Ġthe Ġargument ĠâĢ ľ the Ġsequence Ġof Ġamino Ġacids âĢ Ŀ Ġin ĠFigure Ġ1 Ġfrom Ġthe Ġdocument ĠKB Ġretrie ves Ġ( âĢ ľ am ino Ġacids âĢ Ŀ , Ġ` is âĢ Ŀ , ĠâĢ ľ protein âĢ Ŀ ) Ġfrom ĠNE LL . ĠEnhanced Ġwith Ġthis Ġadditional Ġclue , Ġthe Ġoriginal Ġargument Ġis Ġa Ġbetter Ġmatch Ġto Ġthe Ġquestion . </s> Similar Ġto ĠB IB REF 21 Ġk hot 2017 ans w ering , Ġwe Ġuse ĠElastic Search Ġto Ġretrieve Ġfacts Ġfrom Ġopen ĠKB s . ĠWe Ġremove Ġstop Ġwords Ġfrom Ġtokens Ġof Ġeach Ġargument Ġand Ġpredicate Ġin Ġa Ġdocument ĠKB Ġand Ġregard Ġthe Ġremained Ġwords Ġas ĠElastic Search Ġqueries . ĠWe Ġset Ġdifferent Ġsearch Ġoption Ġfor Ġarguments Ġand Ġpred icates , Ġnamely Ġsetting Ġarguments Ġas Ġthe Ġdominant Ġsearch able Ġfields Ġfor Ġargument Ġqueries Ġand Ġsetting Ġpred icates Ġas Ġthe Ġdominant Ġsearch able Ġfields Ġfor Ġpredicate Ġqueries . ĠWe Ġsave Ġthe Ġtop Ġ10 Ġhints Ġand Ġselectively Ġuse Ġthem Ġin Ġthe Ġexperiment . </s> We Ġregard Ġthe Ġretrieved Ġfacts Ġfrom Ġthe Ġexternal ĠKB Ġas Ġneighbors Ġto Ġthe Ġarguments Ġto Ġbe Ġenhanced . ĠInspired Ġby ĠB IB REF 22 Ġscar sell i 2009 graph , Ġwe Ġupdate Ġthe Ġvector Ġof Ġan Ġelement Ġ$ v _ e $ Ġas Ġfollows , Ġwhere Ġ$( e ) $ Ġand Ġ$( e ) $ Ġrepresent Ġadjacent Ġarguments Ġfrom Ġthe Ġfacts Ġretrieved Ġby Ġobject Ġand Ġsubject , Ġrespectively . ĠIn Ġthis Ġwork , Ġ$ f (\ cd ot Ġ) $ Ġis Ġimplemented Ġby Ġaveraging Ġthe Ġvectors Ġof Ġtwo Ġarguments . </s> $$ v _ e = v _ e Ġ+ Ġ\ sum Ġ_ { o ^ {\ prime Ġ} Ġ\ in Ġ( e )} ^{ } f ( v _{ p ^ {\ prime Ġ} }, v _{ s ^ {\ prime Ġ} }) + \ sum Ġ_ { s ^ {\ prime Ġ} Ġ\ in Ġ( e )} ^{ } f ( v _{ p ^ {\ prime Ġ} }, v _{ o ^ {\ prime Ġ} }) $$ Ġ Ġ Ġ( E q . Ġ26 ) </s> Related ĠWork </s> The Ġtask Ġof ĠK BM RC Ġdiffers Ġfrom Ġmachine Ġreading Ġcomprehension Ġ( M RC ) Ġin Ġboth Ġinput Ġand Ġoutput Ġaspects . ĠThe Ġinput Ġof ĠK BM RC Ġis Ġthe Ġknowledge Ġincluding Ġboth Ġword Ġknowledge Ġextracted Ġfrom Ġthe Ġdocument Ġand Ġworld Ġknowledge Ġretrieved Ġfrom Ġexternal Ġknowledge Ġbase , Ġwhile Ġthe Ġinput Ġof ĠM RC Ġis Ġthe Ġun struct ured Ġtext Ġof Ġa Ġdocument . ĠThe Ġoutput Ġof ĠK BM RC Ġis Ġa Ġsubject Ġor Ġan Ġargument , Ġwhile Ġthe Ġoutput Ġin ĠM RC Ġis Ġa Ġtext Ġspan Ġof Ġthe Ġdocument . ĠMeanwhile , ĠK BM RC Ġfacilitates Ġthe Ġaccessing Ġand Ġleveraging Ġof Ġknowledge Ġfrom Ġexternal ĠKB s Ġbecause Ġthe Ġdocument ĠKB Ġis Ġconsistent Ġwith Ġthe Ġrepresentation Ġof Ġfacts Ġin Ġexternal ĠKB s . </s> K BM RC Ġalso Ġrelates Ġto Ġknowledge - base Ġquestion Ġanswering Ġ( KB Q A ) ĠB IB REF 23 Ġ, Ġwhich Ġaims Ġto Ġanswer Ġquestions Ġbased Ġon Ġan Ġexternal Ġlarge - scale ĠKB Ġsuch Ġas ĠFree base Ġor ĠPro Base . ĠK BM RC Ġdiffers Ġfrom ĠKB Q A Ġin Ġthat Ġthe Ġoriginal ĠKB Ġcomes Ġfrom Ġthe Ġcontent Ġof Ġa Ġdocument . ĠExternal ĠKB Ġis Ġused Ġin Ġthis Ġwork Ġto Ġenhance Ġthe Ġdocument ĠKB . ĠMoreover , Ġexisting Ġbenchmark Ġdatasets Ġfor ĠKB Q A Ġsuch Ġas ĠWeb Questions ĠB IB REF 24 Ġare Ġtypically Ġlimited Ġto Ġsimple Ġquestions . ĠThe ĠK BM RC Ġtask Ġrequires Ġreasoning Ġover Ġtwo Ġfacts Ġfrom Ġthe Ġdocument ĠKB . </s> Our Ġapproach Ġdraws Ġinspiration Ġfrom Ġtwo Ġmain Ġclasses Ġin Ġexisting Ġapproaches Ġof ĠKB Q A , Ġnamely Ġranking Ġbased Ġand Ġparsing Ġbased . ĠRanking Ġbased Ġapproaches ĠB IB REF 17 Ġ, ĠB IB REF 25 Ġare Ġbottom - up , Ġwhich Ġtypically Ġfirst Ġfind Ġa Ġset Ġof Ġcandidate Ġanswers Ġand Ġthen Ġrank Ġbetween Ġthe Ġcandidates Ġwith Ġfeatures Ġat Ġdifferent Ġlevels Ġto Ġget Ġthe Ġanswer . ĠPars ing - based Ġapproaches ĠB IB REF 16 Ġare Ġtop - down , Ġwhich Ġfirst Ġinterpret Ġlogical Ġform Ġfrom Ġa Ġnatural Ġlanguage Ġutter ance , Ġand Ġthen Ġdo Ġexecution Ġto Ġyield Ġthe Ġanswer . ĠRanking - based Ġapproaches Ġachieve Ġbetter Ġperformances Ġthan Ġparsing - based Ġapproaches Ġon ĠWeb Questions , Ġa Ġbenchmark Ġdataset Ġfor ĠKB Q A . ĠWe Ġfollow Ġranking - based Ġapproaches , Ġand Ġdevelop Ġboth Ġa Ġmatching - based Ġmodel Ġwith Ġfeatures Ġat Ġdifferent Ġlevels Ġand Ġa Ġquestion Ġgeneration Ġmodel . ĠMore Ġreferences Ġcan Ġbe Ġfound Ġat Ġhttps :// acl web . org / acl wiki / Question _ An sw ering _( State _ of _ the _ art ). </s> Our Ġwork Ġalso Ġrelates Ġto ĠB IB REF 21 Ġk hot 2017 ans w ering , Ġwhich Ġuses Ġopen ĠIE Ġoutputs Ġfrom Ġexternal Ġtext Ġcorpor a Ġto Ġimprove Ġmulti - choice Ġquestion Ġanswering . ĠHowever , Ġour Ġwork Ġdiffers Ġfrom Ġthem Ġin Ġthat Ġtheir Ġtask Ġdoes Ġnot Ġcontain Ġdocument Ġinformation . ĠFurthermore , Ġwe Ġdevelop Ġa Ġquestion Ġgeneration Ġapproach Ġwhile Ġthey Ġregard Ġthe ĠQ A Ġtask Ġas Ġsub graph Ġsearch Ġbased Ġon Ġan Ġinteger Ġlinear Ġprogramming Ġ( IL P ) Ġapproach . ĠOur Ġwork Ġalso Ġrelates Ġto ĠB IB REF 26 Ġk hash abi 2018 question , Ġwhich Ġfocuses Ġon Ġmulti - choice Ġquestion Ġanswering Ġbased Ġon Ġthe Ġsemantics Ġof Ġa Ġdocument . ĠThey Ġuse Ġsemantic Ġrole Ġlabeling Ġand Ġshallow Ġparsing Ġof Ġa Ġdocument Ġto Ġconstruct Ġa Ġsemantic Ġgraph , Ġbased Ġon Ġwhich Ġan ĠIL P Ġbased Ġapproach Ġis Ġdeveloped Ġto Ġfind Ġthe Ġsupporting Ġsub graph . ĠThe Ġdifference Ġof Ġour Ġapproach Ġis Ġthat Ġpred icates Ġfrom Ġour Ġdocument ĠKB Ġform Ġare Ġnot Ġlimited Ġto Ġa Ġpred efined Ġset , Ġso Ġthat Ġthey Ġdo Ġnot Ġtake Ġinto Ġconsideration Ġthe Ġknowledge Ġfrom Ġexternal ĠKB s , Ġand Ġalso Ġthe Ġdifference Ġin Ġterms Ġof Ġmethodology . ĠB IB REF 19 Ġmill er 2016 key Ġanswer Ġquestions Ġbased Ġon ĠKB s Ġin Ġthe Ġmovie Ġdomain Ġor Ġinformation Ġextraction Ġresults Ġfrom ĠWikipedia Ġdocuments . ĠUnlike Ġthis Ġmethod , Ġour Ġapproach Ġfocuses Ġon Ġentities Ġfrom Ġan Ġexternal ĠKB , Ġour Ġdoc ĠKB Ġis Ġobtained Ġvia Ġopen ĠIE , Ġand Ġwe Ġcombine Ġthe Ġdocument ĠKB Ġwith Ġan Ġopen ĠKB Ġfor Ġquestion Ġanswering . </s> Exper iments </s> We Ġdescribe Ġexperiment Ġsettings Ġand Ġreport Ġfigures Ġand Ġanalysis Ġin Ġthis Ġsection . </s> Settings </s> In Ġour Ġexperiments , Ġwe Ġtune Ġmodel Ġparameters Ġon Ġthe Ġdevelopment Ġset Ġand Ġreport Ġresults Ġon Ġthe Ġtest Ġset . ĠWe Ġdesign Ġexperiments Ġfrom Ġboth Ġranking - based Ġdirection Ġand Ġquestion Ġgeneration - based Ġdirection . ĠThe Ġevaluation Ġmetric Ġis Ġprecision Ġ@ 1 ĠB IB REF 17 Ġ, Ġwhich Ġindicates Ġwhether Ġthe Ġtop Ġranked Ġresult Ġis Ġthe Ġcorrect Ġanswer . ĠWe Ġfurther Ġreport ĠB LE U Ġscore ĠB IB REF 27 Ġfor Ġthe Ġquestion Ġgeneration Ġapproach . </s> We Ġalso Ġadapt ĠBi DA F ĠB IB REF 10 Ġ, Ġa Ġtop - performing Ġreading Ġcomprehension Ġmodel Ġon Ġthe ĠS Qu AD Ġdataset ĠB IB REF 0 Ġas Ġa Ġstrong Ġbaseline . ĠAs ĠBi DA F Ġoutput Ġa Ġspan Ġfrom Ġthe Ġinput Ġdocument Ġas Ġthe Ġanswer Ġto Ġthe Ġgiven Ġquestion , Ġwe Ġadapt Ġit Ġto ĠK BM RC Ġas Ġa Ġranking Ġmodel Ġsimilarly Ġas Ġthe Ġapproach Ġused Ġin Ġprevious Ġresearch ĠB IB REF 28 Ġ. ĠWe Ġuse ĠBi DA F Ġto Ġselect Ġan Ġanswer Ġspan Ġfrom Ġa Ġcorresponding Ġdocument Ġbased Ġon Ġa Ġgiven Ġquestion Ġand Ġselect Ġthe Ġcandidate Ġanswer Ġthat Ġhas Ġmaximum Ġoverlap Ġwith Ġthe Ġanswer Ġspan Ġas Ġthe Ġfinal Ġanswer . </s> Analysis : ĠQuestion ĠAn sw ering ĠModels </s> Table Ġ3 Ġshows Ġthe Ġresults Ġof Ġour Ġtwo Ġquestion Ġanswering Ġmodels . ĠIt Ġis Ġclear Ġthat ĠK V Mem Net Ġachieves Ġbetter ĠP @ 1 Ġscores Ġon Ġboth Ġdev Ġand Ġtest Ġsets Ġthan ĠPC Net . ĠThe Ġreason Ġis Ġthat Ġcandidate Ġanswers Ġof ĠPC Net Ġcome Ġfrom Ġthe ĠâĢ ľ anch or âĢ Ŀ Ġpoint Ġalong Ġ1 - hop Ġor Ġ2 - hop Ġpaths . ĠHowever , Ġthe Ġcorrect Ġanswer Ġmight Ġnot Ġbe Ġconnected Ġdue Ġto Ġthe Ġquality Ġof Ġanchor Ġdetection . ĠOn Ġthe Ġdev Ġset , Ġwe Ġobserve Ġthat Ġonly Ġ69 . 6 % Ġof Ġcorrect Ġanswers Ġcan Ġbe Ġcovered Ġby Ġthe Ġset Ġof Ġcandidate Ġanswers Ġin ĠPC Net , Ġwhich Ġapparently Ġlimits Ġthe Ġupper Ġbound Ġof Ġthe Ġapproach . ĠThis Ġis Ġaddressed Ġin ĠK V Mem Net Ġbecause Ġall Ġthe Ġarguments Ġare Ġcandidate Ġanswers . ĠBoth ĠPC Net Ġand ĠK V Mem Net Ġoutper form Ġour Ġimplementation Ġof ĠB IB REF 17 Ġb ordes 2014 question , Ġsince Ġthe Ġlatter Ġignores Ġword Ġorder . ĠWe Ġincorporate Ġeach Ġof Ġthe Ġfour ĠKB s Ġseparately Ġinto ĠPC Net Ġand ĠK V Mem Net , Ġand Ġfind Ġthat Ġincorporating Ġexternal ĠKB s Ġcould Ġbring Ġimprovements . </s> From ĠFigure Ġ4 Ġ, Ġwe Ġcan Ġsee Ġthat Ġthe ĠK V Mem Net Ġmodel Ġattends Ġto Ġthe Ġkey ĠâĢ ľ ( st . Ġjohn s , Ġis Ġlocated ) âĢ Ŀ Ġfor Ġthe Ġquestion ĠâĢ ľ Where Ġis Ġst Ġjohn s Ġmi Ġlocated ? âĢ Ŀ . ĠThus , Ġthe Ġmodel Ġhas Ġhigher Ġconfidence Ġin Ġregarding Ġvalue ĠâĢ ľ in Ġcl inton Ġcounty âĢ Ŀ Ġas Ġthe Ġanswer . </s> Analysis : ĠGener ative ĠModels </s> Table Ġ4 Ġshows Ġthe Ġresults Ġof Ġdifferent Ġquestion Ġgeneration Ġmodels . ĠOur Ġapproach Ġis Ġabbrevi ated Ġas ĠQ G Net , Ġwhich Ġstands Ġfor Ġthe Ġuse Ġof Ġa Ġparaph r asing Ġmodel Ġplus Ġour Ġquestion Ġgeneration Ġmodel . ĠWe Ġcan Ġsee Ġthat ĠQ G Net Ġperforms Ġbetter Ġthan ĠSe q 2 Se q Ġin Ġterms Ġof ĠB LE U Ġscore Ġbecause Ġmany Ġimportant Ġwords Ġof Ġlow Ġfrequency Ġfrom Ġthe Ġinput Ġare Ġreplicated Ġto Ġthe Ġtarget Ġsequence . ĠHowever , Ġthe Ġimprovement Ġis Ġnot Ġsignificant Ġfor Ġthe ĠQ A Ġtask . ĠWe Ġalso Ġincorporate Ġeach Ġof Ġthe Ġfour ĠKB s Ġinto ĠQ G Net , Ġand Ġobserve Ġslight Ġimprovements Ġon ĠNE LL Ġand ĠRe verb . ĠDespite Ġthe Ġoverall Ġaccuracy Ġof ĠQ G Net Ġbeing Ġlower Ġthan ĠPC Net Ġand ĠK V Mem Net , Ġcombining Ġoutcomes Ġwith Ġthem Ġcould Ġgenerates Ġ1 . 5 % Ġand Ġ0 . 8 % Ġabsolute Ġgains , Ġrespectively . </s> We Ġshow Ġexamples Ġgenerated Ġby Ġour ĠQ G Ġmodel Ġin ĠFigure Ġ5 Ġ, Ġin Ġwhich Ġthe Ġpaths Ġof Ġtwo Ġcandidate Ġanswers Ġare Ġregarded Ġas Ġthe Ġinput Ġto Ġthe ĠQ G Ġmodel . ĠWe Ġcan Ġsee Ġthat Ġthe Ġoriginal Ġquestion Ġis Ġcloser Ġto Ġthe Ġfirst Ġgenerated Ġresult Ġthan Ġthe Ġsecond Ġone . ĠAccordingly , Ġthe Ġfirst Ġcandidate Ġ($ Ġ61 , 300 ) Ġwould Ġbe Ġassigned Ġwith Ġa Ġlarger Ġprobability Ġas Ġthe Ġanswer . </s> Future ĠOpportun ities Ġfor ĠFuture ĠResearch </s> We Ġconduct Ġerror Ġanalysis Ġfrom Ġmultiple Ġperspectives Ġto Ġshow Ġthe Ġlimitations Ġand Ġfuture Ġopportunities Ġof Ġdifferent Ġcomponents Ġof Ġour Ġapproach . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġfocus Ġon Ġknowledge Ġbased Ġmachine Ġreading Ġcomprehension . ĠWe Ġcreate Ġa Ġmanually Ġlabeled Ġdataset Ġfor Ġthe Ġtask , Ġand Ġdevelop Ġa Ġframework Ġconsisting Ġof Ġboth Ġquestion Ġanswering Ġmodel Ġand Ġquestion Ġgeneration Ġmodel . ĠWe Ġfurther Ġincorporate Ġfour Ġopen ĠKB s Ġas Ġexternal Ġknowledge Ġinto Ġboth Ġquestion Ġanswering Ġand Ġgener ative Ġapproaches , Ġand Ġdemonstrate Ġthat Ġincorporating Ġadditional Ġevidence Ġfrom Ġopen ĠKB s Ġimproves Ġtotal Ġaccuracy . ĠWe Ġconduct Ġextensive Ġmodel Ġanalysis Ġand Ġerror Ġanalysis Ġto Ġshow Ġthe Ġadvantages Ġand Ġlimitations Ġof Ġour Ġapproaches .
Introduction In recent years, there has been an increasing interest in Machine reading comprehension (MRC), which plays a vital role in the assessment of how well a machine could understand natural language. Several datasets BIBREF0 , BIBREF1 , BIBREF2 for machine reading comprehension have been released in recent years and have driven the evolution of powerful neural models. However, much of the research up to now has been dominated by answering questions that can be well solved solved using superficial information, yet struggles to do accurate natural language understanding and reasoning. For example, BIBREF3 jia2017Adversarial show that existing machine learning systems for MRC perform poorly under adversarial evaluation. Recent developments in MRC datasets BIBREF4 , BIBREF5 , BIBREF6 have heightened the need for deep understanding. Knowledge has a pivotal role in accurately understanding and reasoning natural language in MRC. Previous research BIBREF7 , BIBREF8 has established that human reading comprehension requires both words and world knowledge. In this paper, we consider words and world knowledge in the format of triplets (subject, predicate, object). Specifically, we believe the advantages of using knowledge in MRC are three-fold. First, utilizing knowledge in MRC supports reasoning over multiple triplets because a single triplet may not cover the entire question. Multi-hop reasoning is also a long-standing goal in question answering. Second, building a question answering system based on triplet-style knowledge facilitates the interpretability of the decision making process. Triplets organize the document together with KBs as a graph, where a well-designed model such as PCNet, which we will describe in a later section, expressly reveal rationales for their predictions. Third, representing the documents as knowledge allows for ease of accessing and leveraging the knowledge from external/background knowledge because the knowledge representation of a document is easily consistent with both manually curated and automatically extracted KBs. In this paper, we present knowledge based machine reading comprehension, which requires reasoning over triplet-style knowledge involved in a document. However, we find published dataset do not sufficiently support this task. We conduct preliminary exploration on SQuAD BIBREF0 . We use a strong open IE algorithm BIBREF9 to extract triplets from the documents and observe that only 15% of instances have an answer that is exactly the same as the corresponding subject/object in the extracted triplets. To do knowledge-based MRC, We build a new dataset consisting of 40,047 examples for the knowledge based MRC task. The annotation of this dataset is designed so that successfully answering the questions requires understanding and the knowledge involved in a document. Each instance is composed of a question, a set of triplets derived from a document, and the answer. We implement a framework consisting of both a question answering model and a question generation model, both of which take the knowledge extracted from the document as well as relevant facts from an external knowledge base such as Freebase/ProBase/Reverb/NELL. The question answering model gives each candidate answer a score by measuring the semantic relevance between representation and the candidate answer representation in vector space. The question generation model provides each candidate answer with a score by measuring semantic relevance between the question and the generated question based on the semantics of the candidate answer. We implement an MRC model BiDAF BIBREF10 as a baseline for the proposed dataset. To test the scalability of our approach in leveraging external KBs, we use both manually created and automatically extracted KBs, including Freebase BIBREF11 , ProBase BIBREF12 , NELL BIBREF13 and Reverb BIBREF14 . Experiments show that incorporating evidence from external KBs improves both the matching-based and question generation-based approaches. Qualitative analysis shows the advantages and limitations of our approaches, as well as the remaining challenges. Task Definition and Dataset We formulate the task of knowledge based machine reading comprehension, which is abbreviated as KBMRC, and describe the dataset built for KBMRC and the external open KBs leveraged in this work. Approach Overview Our framework consists of a question answering model and a question generation model. We implement two question answering models, which directly measure the semantic similarity between questions and candidate answers in the semantic space. First, to make the model's prediction more explainable, we implement a path based QA model PCNet. In this model, to get candidate answers from the triplets based document for a given question. We first retrieve the an “anchor” point $arg1_i$ or $arg2_i$ in the document fact $f_i$ . These anchors are selected based on edit distance between the words in the questions and the arguments. Then we regard all arguments in the 1-hops and 2-hops fact of the anchors as answers. However, the coverage of the candidate answers can be 100% in the model. We then implement a second end-to-end neural model KVMenNet which covers all the answers but with less interpretability. Both models generate a score $f_{qa}(q, a)$ of each candidate answer. We then implement a generation-based model. The motivation to design this model is that we want to associate natural language phrases with knowledge based representation. It takes semantics of a candidate answer as the input and generates a question $\hat{q}$ . Then a paraphrasing model gives a score $f_{qg}(q,\hat{q})$ , which is computed between the generated question $\hat{q}$ and the original question $q$ , as the ranking score. We get the final scores $S(q,a)$ used for ranking as follows. $$S(q,a) = \lambda f_{qa}(q, a) + (1-\lambda ) f_{qg}(q,\hat{q})$$   (Eq. 7) Moreover, we incorporate side information from external KBs into the three models. The details of how we use external KBs to enhance the representation of elements in the document KB will be described in section Incorporating External Knowledge. The Question Answering Model In this section, we present two matching models to measure the semantic relatedness between the question and the candidate answer in the vector semantic space. Afterwards, we introduce our strategy that incorporates open KB as external knowledge to enhance both models. QA Model 1: PCNet We follow BIBREF17 , BIBREF18 bordes2014question,dong-EtAl:2015:ACL-IJCNLP1 and develop PCNet, which is short for path- and context- based neural network. In PCNet, candidate answers come from arguments of the document KB, and each candidate answer is represented with its neighboring arguments and predicates as well as its path from the anchor in the document KB. We use a rule-based approach based on string fuzzy match to detect the anchor. Each argument is measured by $\sum _{i,j}^{}\mathbb {I}(arg_i, q_j)$ , where $i$ and $j$ iterate across argument words and question words, respectively. $\mathbb {I}(x,y)$ is an indicator function whose value is 1 if the minimum edit distance between $x$ and $y$ is no more than 1, otherwise it is 0. The arguments linked to the anchor with 1-hop and 2-hop paths are regarded as candidate answers. Since an argument might include multiple words, such as “the popular angry bird game”, we use GRU based RNN to get the vector representation of each argument/predicate. The path representation $v_{p}$ is computed by averaging the vectors of elements in the path. Similarly, we use another RNN to get the vector of each neighboring argument/predicate, and average them to get the context vector $v_{c}$ . We represent the question $q$ using a bidirectional RNN layer based on the GRU unit. The concatenation of the last hidden vectors from both directions is used as the question vector $v_{q}$ . The dot product is used to measure the semantic relevance between the b b question and two types of evidence. $$f_{qa}(q, a) = v_{q}^{T}v_{p} + v_q^{T}v_{c}$$   (Eq. 10) QA Model 2: KVMemNet Despite the interpretability of PCNet, the coverage of anchor detection limits the upper bound of the approach. We implement a more powerful method based on the key-value memory network BIBREF19 , KVMemNet for short, which has proven powerful in KB-based question answering. The KVMenNet could be viewed as a “soft” matching approach, which includes a parameterized memory consisting of key-value pairs. Intuitively, keys are used for matching with the question, and values are used for matching to the candidate answer. Given a KB fact ( $subj, pred, obj$ ), We consider both directions and add two key-value pairs in the memory, namely ( $key = subj +pred, value=obj$ ) and ( $key =obj + pred, value=subj$ ). The vectors of arguments/predicates are calculated the same way as described in PCNet. The concatenation of two vectors is used as the key vector $v_{key}$ . Each memory item is assigned a relevance probability by comparing the question to each key. $$\alpha _{key_i} = {softmax}(v_q \cdot v_{key_i})$$   (Eq. 12) Afterwards, vectors in memory ( $v_{value}$ ) are weighted summed according to their addressing probabilities, and the vector $v_o$ is returned. $$v_o = \sum _i \alpha _{key_i}v_{value_i}$$   (Eq. 13) In PCNet, reasoning over two facts is achieved by incorporating 2-hop paths, while in KVMemNet this is achieved by repeating the memory access process twice. After receiving the result $v_o$ , we update the query with $q_2 = R(v_q + v_o)$ , where $R$ is model parameter. Finally, after a fixed number $n$ hops ( $n=2$ in this work), the resulting vector is used to measure the relevance to candidate answers via the dot product. Training and Inference Let $\mathcal {D} = \lbrace (q_i, a_i); i = 1, \ldots , |\mathcal {D}|\rbrace $ be the training data consisting of questions $q_i$ paired with their correct answer $a_i$ . We train both matching models with a margin-based ranking loss function, which is calculated as follows, where $m$ is the margin (fixed to $0.1$ ) and $\bar{a}$ is randomly sampled from a set of incorrect candidates $\bar{\mathcal {A}}$ . $$  \sum _{i=1}^{|\mathcal {D}|} \sum _{\tiny {\bar{a} \in \bar{\mathcal {A}}(q_i)}} \max \lbrace 0, m - S(q_i,a_i) + S(q_i,\bar{a})\rbrace ,$$   (Eq. 15) For testing, given a question $q$ , the model predicts the answer based on the following equation, where $ \mathcal {A}(q)$ is the candidate answer set. $$\hat{a} = {\mbox{argmax}}_{a^{\prime } \in \mathcal {A}(q)} S(q,a^{\prime })$$   (Eq. 16) The Question Generation Model In this section, we present the generation model which generates a question based on the semantics of a candidate answer. Afterward, we introduce how our paraphrasing model, which measures the semantic relevance between the generated question and the original question, is pretrained. Hierarchical seq2seq Generation Model Our question generation model takes the path between an “anchor” and the candidate answer, and outputs a question. We adopt the sequence to sequence architecture as our basic question generation model due to its effectiveness in natural language generation tasks. The hierarchical encoder and the hierarchical attention mechanism are introduced to take into account the structure of the input facts. Facts from external KBs are conventionally integrated into the model using the same way as described in the matching model. As illustrated in Figure 3 , the question generation model contains an encoder and a decoder. We use a hierarchical encoder consisting of two layers to model the meaning of each element (subject, predicate or object) and the relationship between them. Since each element might contain multiple words, we use RNN as the word layer encoder to get the representation of each element. We define the fact level sequence as a path starting from the anchor and ending at the candidate answer. Another RNN is used as the fact level encoder. The last hidden state at the fact layer is fed to the decoder. We develop a hierarchical attention mechanism in the decoder, which first makes soft alignment over the hidden states at the fact layer, the output of which is further used to attend to hidden states at the word level. Specifically, given the decoder hidden state $h^{dec}_{t-1}$ , we use a fact-level attention model $Att_{fct}$ to calculate the contextual vector, which is further combined with the current hidden state $h^{dec}_{t}$ , resulting in $c_{fct}$ . The contextual vector is calculated through weighted averaging over hidden vectors at the fact level, which is given as follows, where $\odot $ is dot product, $h^{fld}_j$ is the $j$ -th hidden state at the fact layer from the encoder, and $l_f$ is the number of hidden states at the fact level. $$c_{fct} &= GRU(h^{dec}_{t}, \sum _{j=1}^{{l_f}} \alpha _{tj}h^{fct}_j) \\ \alpha _{tj} &= \frac{ exp(h^{dec}_{t-1} \odot h^{fct}_j)}{\sum _{k=1}^{{l_f}}exp(h^{dec}_{t-1} \odot h^{fct}_k)}$$   (Eq. 19) Similarly, we feed $c_{fct}$ to the word-level attention function $Att_{wrd}$ and calculate over hidden vectors at the word-level. The output $c_{wrd}$ will be concatenated with $h^{dec}_{t}$ to predict the next word. Since many entity names of great importance are rare words from the input, we use the copying mechanism BIBREF20 that learns when to replicate words from the input or to predict words from the target vocabulary. The probability distribution of generating the word $y$ is calculated as follows, in which the $softmax$ function is calculated over a combined logits from both sides. $$\begin{split} &p(y)=\frac{exp({e_y \odot W_g[h_t^{dec};c_{wrd}]})+exp(s_c(y))}{Z}\\ &s_c(y) = c_{wrd} \odot {tanh(W_ch_t^{wrd})} \end{split}$$   (Eq. 20) We train our question generation model with maximum likelihood estimation. The loss function is given as follows, where $D$ is the training corpus. We use beam search in the inference process. $$l = -\sum _{(x,y)\in D} \sum _t log p(y_t |y_{<t}, x)$$   (Eq. 21) An advantage of the model is that external knowledge could be easily incorporated with the same mechanism as we have described in section Incorporating External Knowledge. We enhance the representation of an argument or a predicate by concatenating open KB vectors into encoder hidden states. The Paraphrasing Model The paraphrasing model is used to measure the semantic relevance between the original question and the question generated from the QG model. We use bidirectional RNN with gated recurrent unit to represent two questions, and compose them with element-wise multiplication. The results are followed by a $softmax$ layer, whose output length is 2. The model is trained by minimizing the cross-entropy error, in which the supervision is provided in the training data. We collect two datasets to train the paraphrasing model. The first dataset is from Quora dataset, which is built for detecting whether or not a pair of questions are semantically equivalent. 345,989 positive question pairs and 255,027 negative pairs are included in the first dataset. The second dataset includes web queries from query logs, which are obtained by clustering the web queries that click the same web page. In this way we obtain 6,118,023 positive query pairs. We implement a heuristic rule to get 6,118,023 negative instances for the query dataset. For each pair of query text, we clamp the first query and retrieve a query that is mostly similar to the second query. To improve the efficiency of this process, we randomly sample 10,000 queries and define the “similarity” as the number of co-occurred words between two questions. During training, we initialize the values of word embeddings with $300d$ Glove vectors, which is learned on Wikipedia texts. We use a held-out data consisting of 20K query pairs to check the performance of the paraphrasing model. The accuracy of the paraphrasing model on the held-out dataset is 87.36%. Incorporating External Knowledge There are many possible ways to implement the idea of improving question answering with external KB. In this work, we use external KBs (such as NELL and ProBase) to enhance the representations of elements in the document KB. For instance, the argument “the sequence of amino acids” in Figure 1 from the document KB retrieves (“amino acids”, `is”, “protein”) from NELL. Enhanced with this additional clue, the original argument is a better match to the question. Similar to BIBREF21 khot2017answering, we use ElasticSearch to retrieve facts from open KBs. We remove stop words from tokens of each argument and predicate in a document KB and regard the remained words as ElasticSearch queries. We set different search option for arguments and predicates, namely setting arguments as the dominant searchable fields for argument queries and setting predicates as the dominant searchable fields for predicate queries. We save the top 10 hints and selectively use them in the experiment. We regard the retrieved facts from the external KB as neighbors to the arguments to be enhanced. Inspired by BIBREF22 scarselli2009graph, we update the vector of an element $v_e$ as follows, where $(e)$ and $(e)$ represent adjacent arguments from the facts retrieved by object and subject, respectively. In this work, $f(\cdot )$ is implemented by averaging the vectors of two arguments. $$v_e=v_e + \sum _{o^{\prime } \in (e)}^{}f(v_{p^{\prime }},v_{s^{\prime }})+\sum _{s^{\prime } \in (e)}^{}f(v_{p^{\prime }},v_{o^{\prime }})$$   (Eq. 26) Related Work The task of KBMRC differs from machine reading comprehension (MRC) in both input and output aspects. The input of KBMRC is the knowledge including both word knowledge extracted from the document and world knowledge retrieved from external knowledge base, while the input of MRC is the unstructured text of a document. The output of KBMRC is a subject or an argument, while the output in MRC is a text span of the document. Meanwhile, KBMRC facilitates the accessing and leveraging of knowledge from external KBs because the document KB is consistent with the representation of facts in external KBs. KBMRC also relates to knowledge-base question answering (KBQA) BIBREF23 , which aims to answer questions based on an external large-scale KB such as Freebase or ProBase. KBMRC differs from KBQA in that the original KB comes from the content of a document. External KB is used in this work to enhance the document KB. Moreover, existing benchmark datasets for KBQA such as WebQuestions BIBREF24 are typically limited to simple questions. The KBMRC task requires reasoning over two facts from the document KB. Our approach draws inspiration from two main classes in existing approaches of KBQA, namely ranking based and parsing based. Ranking based approaches BIBREF17 , BIBREF25 are bottom-up, which typically first find a set of candidate answers and then rank between the candidates with features at different levels to get the answer. Parsing-based approaches BIBREF16 are top-down, which first interpret logical form from a natural language utterance, and then do execution to yield the answer. Ranking-based approaches achieve better performances than parsing-based approaches on WebQuestions, a benchmark dataset for KBQA. We follow ranking-based approaches, and develop both a matching-based model with features at different levels and a question generation model. More references can be found at https://aclweb.org/aclwiki/Question_Answering_(State_of_the_art). Our work also relates to BIBREF21 khot2017answering, which uses open IE outputs from external text corpora to improve multi-choice question answering. However, our work differs from them in that their task does not contain document information. Furthermore, we develop a question generation approach while they regard the QA task as subgraph search based on an integer linear programming (ILP) approach. Our work also relates to BIBREF26 khashabi2018question, which focuses on multi-choice question answering based on the semantics of a document. They use semantic role labeling and shallow parsing of a document to construct a semantic graph, based on which an ILP based approach is developed to find the supporting subgraph. The difference of our approach is that predicates from our document KB form are not limited to a predefined set, so that they do not take into consideration the knowledge from external KBs, and also the difference in terms of methodology. BIBREF19 miller2016key answer questions based on KBs in the movie domain or information extraction results from Wikipedia documents. Unlike this method, our approach focuses on entities from an external KB, our doc KB is obtained via open IE, and we combine the document KB with an open KB for question answering. Experiments We describe experiment settings and report figures and analysis in this section. Settings In our experiments, we tune model parameters on the development set and report results on the test set. We design experiments from both ranking-based direction and question generation-based direction. The evaluation metric is precision @1 BIBREF17 , which indicates whether the top ranked result is the correct answer. We further report BLEU score BIBREF27 for the question generation approach. We also adapt BiDAF BIBREF10 , a top-performing reading comprehension model on the SQuAD dataset BIBREF0 as a strong baseline. As BiDAF output a span from the input document as the answer to the given question, we adapt it to KBMRC as a ranking model similarly as the approach used in previous research BIBREF28 . We use BiDAF to select an answer span from a corresponding document based on a given question and select the candidate answer that has maximum overlap with the answer span as the final answer. Analysis: Question Answering Models Table 3 shows the results of our two question answering models. It is clear that KVMemNet achieves better P@1 scores on both dev and test sets than PCNet. The reason is that candidate answers of PCNet come from the “anchor” point along 1-hop or 2-hop paths. However, the correct answer might not be connected due to the quality of anchor detection. On the dev set, we observe that only 69.6% of correct answers can be covered by the set of candidate answers in PCNet, which apparently limits the upper bound of the approach. This is addressed in KVMemNet because all the arguments are candidate answers. Both PCNet and KVMemNet outperform our implementation of BIBREF17 bordes2014question, since the latter ignores word order. We incorporate each of the four KBs separately into PCNet and KVMemNet, and find that incorporating external KBs could bring improvements. From Figure 4 , we can see that the KVMemNet model attends to the key “(st. johns, is located)” for the question “Where is st johns mi located?”. Thus, the model has higher confidence in regarding value “in clinton county” as the answer. Analysis: Generative Models Table 4 shows the results of different question generation models. Our approach is abbreviated as QGNet, which stands for the use of a paraphrasing model plus our question generation model. We can see that QGNet performs better than Seq2Seq in terms of BLEU score because many important words of low frequency from the input are replicated to the target sequence. However, the improvement is not significant for the QA task. We also incorporate each of the four KBs into QGNet, and observe slight improvements on NELL and Reverb. Despite the overall accuracy of QGNet being lower than PCNet and KVMemNet, combining outcomes with them could generates 1.5% and 0.8% absolute gains, respectively. We show examples generated by our QG model in Figure 5 , in which the paths of two candidate answers are regarded as the input to the QG model. We can see that the original question is closer to the first generated result than the second one. Accordingly, the first candidate ($ 61,300) would be assigned with a larger probability as the answer. Future Opportunities for Future Research We conduct error analysis from multiple perspectives to show the limitations and future opportunities of different components of our approach. Conclusion In this paper, we focus on knowledge based machine reading comprehension. We create a manually labeled dataset for the task, and develop a framework consisting of both question answering model and question generation model. We further incorporate four open KBs as external knowledge into both question answering and generative approaches, and demonstrate that incorporating additional evidence from open KBs improves total accuracy. We conduct extensive model analysis and error analysis to show the advantages and limitations of our approaches.
Where is a question generation model used?
d9b6c61fc6d29ad399d27b931b6cb7b1117b314a
Where Ġis Ġa Ġquestion Ġgeneration Ġmodel Ġused ?
The Ġquestion Ġgeneration Ġmodel Ġprovides Ġeach Ġcandidate Ġanswer Ġwith Ġa Ġscore Ġby Ġmeasuring Ġsemantic Ġrelevance Ġbetween Ġthe Ġquestion Ġand Ġthe Ġgenerated Ġquestion Ġbased Ġon Ġthe Ġsemantics Ġof Ġthe Ġcandidate Ġanswer . Ġ
<s> Where Ġis Ġa Ġquestion Ġgeneration Ġmodel Ġused ? </s> Introduction </s> In Ġrecent Ġyears , Ġthere Ġhas Ġbeen Ġan Ġincreasing Ġinterest Ġin ĠMachine Ġreading Ġcomprehension Ġ( M RC ), Ġwhich Ġplays Ġa Ġvital Ġrole Ġin Ġthe Ġassessment Ġof Ġhow Ġwell Ġa Ġmachine Ġcould Ġunderstand Ġnatural Ġlanguage . ĠSeveral Ġdatasets ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġfor Ġmachine Ġreading Ġcomprehension Ġhave Ġbeen Ġreleased Ġin Ġrecent Ġyears Ġand Ġhave Ġdriven Ġthe Ġevolution Ġof Ġpowerful Ġneural Ġmodels . ĠHowever , Ġmuch Ġof Ġthe Ġresearch Ġup Ġto Ġnow Ġhas Ġbeen Ġdominated Ġby Ġanswering Ġquestions Ġthat Ġcan Ġbe Ġwell Ġsolved Ġsolved Ġusing Ġsuperficial Ġinformation , Ġyet Ġstruggles Ġto Ġdo Ġaccurate Ġnatural Ġlanguage Ġunderstanding Ġand Ġreasoning . ĠFor Ġexample , ĠB IB REF 3 Ġj ia 2017 Ad vers arial Ġshow Ġthat Ġexisting Ġmachine Ġlearning Ġsystems Ġfor ĠM RC Ġperform Ġpoorly Ġunder Ġadvers arial Ġevaluation . ĠRecent Ġdevelopments Ġin ĠM RC Ġdatasets ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġhave Ġheightened Ġthe Ġneed Ġfor Ġdeep Ġunderstanding . </s> Know ledge Ġhas Ġa Ġpivotal Ġrole Ġin Ġaccurately Ġunderstanding Ġand Ġreasoning Ġnatural Ġlanguage Ġin ĠM RC . ĠPrevious Ġresearch ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġhas Ġestablished Ġthat Ġhuman Ġreading Ġcomprehension Ġrequires Ġboth Ġwords Ġand Ġworld Ġknowledge . ĠIn Ġthis Ġpaper , Ġwe Ġconsider Ġwords Ġand Ġworld Ġknowledge Ġin Ġthe Ġformat Ġof Ġtriple ts Ġ( subject , Ġpredicate , Ġobject ). ĠSpecifically , Ġwe Ġbelieve Ġthe Ġadvantages Ġof Ġusing Ġknowledge Ġin ĠM RC Ġare Ġthree - fold . ĠFirst , Ġutilizing Ġknowledge Ġin ĠM RC Ġsupports Ġreasoning Ġover Ġmultiple Ġtriple ts Ġbecause Ġa Ġsingle Ġtriple t Ġmay Ġnot Ġcover Ġthe Ġentire Ġquestion . ĠMulti - hop Ġreasoning Ġis Ġalso Ġa Ġlong - standing Ġgoal Ġin Ġquestion Ġanswering . ĠSecond , Ġbuilding Ġa Ġquestion Ġanswering Ġsystem Ġbased Ġon Ġtriple t - style Ġknowledge Ġfacilitates Ġthe Ġinterpret ability Ġof Ġthe Ġdecision Ġmaking Ġprocess . ĠTriple ts Ġorganize Ġthe Ġdocument Ġtogether Ġwith ĠKB s Ġas Ġa Ġgraph , Ġwhere Ġa Ġwell - designed Ġmodel Ġsuch Ġas ĠPC Net , Ġwhich Ġwe Ġwill Ġdescribe Ġin Ġa Ġlater Ġsection , Ġexpressly Ġreveal Ġrational es Ġfor Ġtheir Ġpredictions . ĠThird , Ġrepresenting Ġthe Ġdocuments Ġas Ġknowledge Ġallows Ġfor Ġease Ġof Ġaccessing Ġand Ġleveraging Ġthe Ġknowledge Ġfrom Ġexternal / background Ġknowledge Ġbecause Ġthe Ġknowledge Ġrepresentation Ġof Ġa Ġdocument Ġis Ġeasily Ġconsistent Ġwith Ġboth Ġmanually Ġcurated Ġand Ġautomatically Ġextracted ĠKB s . </s> In Ġthis Ġpaper , Ġwe Ġpresent Ġknowledge Ġbased Ġmachine Ġreading Ġcomprehension , Ġwhich Ġrequires Ġreasoning Ġover Ġtriple t - style Ġknowledge Ġinvolved Ġin Ġa Ġdocument . ĠHowever , Ġwe Ġfind Ġpublished Ġdataset Ġdo Ġnot Ġsufficiently Ġsupport Ġthis Ġtask . ĠWe Ġconduct Ġpreliminary Ġexploration Ġon ĠS Qu AD ĠB IB REF 0 Ġ. ĠWe Ġuse Ġa Ġstrong Ġopen ĠIE Ġalgorithm ĠB IB REF 9 Ġto Ġextract Ġtriple ts Ġfrom Ġthe Ġdocuments Ġand Ġobserve Ġthat Ġonly Ġ15 % Ġof Ġinstances Ġhave Ġan Ġanswer Ġthat Ġis Ġexactly Ġthe Ġsame Ġas Ġthe Ġcorresponding Ġsubject / object Ġin Ġthe Ġextracted Ġtriple ts . ĠTo Ġdo Ġknowledge - based ĠM RC , ĠWe Ġbuild Ġa Ġnew Ġdataset Ġconsisting Ġof Ġ40 , 047 Ġexamples Ġfor Ġthe Ġknowledge Ġbased ĠM RC Ġtask . ĠThe Ġannotation Ġof Ġthis Ġdataset Ġis Ġdesigned Ġso Ġthat Ġsuccessfully Ġanswering Ġthe Ġquestions Ġrequires Ġunderstanding Ġand Ġthe Ġknowledge Ġinvolved Ġin Ġa Ġdocument . ĠEach Ġinstance Ġis Ġcomposed Ġof Ġa Ġquestion , Ġa Ġset Ġof Ġtriple ts Ġderived Ġfrom Ġa Ġdocument , Ġand Ġthe Ġanswer . </s> We Ġimplement Ġa Ġframework Ġconsisting Ġof Ġboth Ġa Ġquestion Ġanswering Ġmodel Ġand Ġa Ġquestion Ġgeneration Ġmodel , Ġboth Ġof Ġwhich Ġtake Ġthe Ġknowledge Ġextracted Ġfrom Ġthe Ġdocument Ġas Ġwell Ġas Ġrelevant Ġfacts Ġfrom Ġan Ġexternal Ġknowledge Ġbase Ġsuch Ġas ĠFree base / Pro Base / R ever b / NE LL . ĠThe Ġquestion Ġanswering Ġmodel Ġgives Ġeach Ġcandidate Ġanswer Ġa Ġscore Ġby Ġmeasuring Ġthe Ġsemantic Ġrelevance Ġbetween Ġrepresentation Ġand Ġthe Ġcandidate Ġanswer Ġrepresentation Ġin Ġvector Ġspace . ĠThe Ġquestion Ġgeneration Ġmodel Ġprovides Ġeach Ġcandidate Ġanswer Ġwith Ġa Ġscore Ġby Ġmeasuring Ġsemantic Ġrelevance Ġbetween Ġthe Ġquestion Ġand Ġthe Ġgenerated Ġquestion Ġbased Ġon Ġthe Ġsemantics Ġof Ġthe Ġcandidate Ġanswer . ĠWe Ġimplement Ġan ĠM RC Ġmodel ĠBi DA F ĠB IB REF 10 Ġas Ġa Ġbaseline Ġfor Ġthe Ġproposed Ġdataset . ĠTo Ġtest Ġthe Ġscal ability Ġof Ġour Ġapproach Ġin Ġleveraging Ġexternal ĠKB s , Ġwe Ġuse Ġboth Ġmanually Ġcreated Ġand Ġautomatically Ġextracted ĠKB s , Ġincluding ĠFree base ĠB IB REF 11 Ġ, ĠPro Base ĠB IB REF 12 Ġ, ĠNE LL ĠB IB REF 13 Ġand ĠRe verb ĠB IB REF 14 Ġ. ĠExper iments Ġshow Ġthat Ġincorporating Ġevidence Ġfrom Ġexternal ĠKB s Ġimproves Ġboth Ġthe Ġmatching - based Ġand Ġquestion Ġgeneration - based Ġapproaches . ĠQual itative Ġanalysis Ġshows Ġthe Ġadvantages Ġand Ġlimitations Ġof Ġour Ġapproaches , Ġas Ġwell Ġas Ġthe Ġremaining Ġchallenges . </s> Task ĠDefinition Ġand ĠDat as et </s> We Ġformulate Ġthe Ġtask Ġof Ġknowledge Ġbased Ġmachine Ġreading Ġcomprehension , Ġwhich Ġis Ġabbrevi ated Ġas ĠK BM RC , Ġand Ġdescribe Ġthe Ġdataset Ġbuilt Ġfor ĠK BM RC Ġand Ġthe Ġexternal Ġopen ĠKB s Ġlever aged Ġin Ġthis Ġwork . </s> App roach ĠOverview </s> Our Ġframework Ġconsists Ġof Ġa Ġquestion Ġanswering Ġmodel Ġand Ġa Ġquestion Ġgeneration Ġmodel . </s> We Ġimplement Ġtwo Ġquestion Ġanswering Ġmodels , Ġwhich Ġdirectly Ġmeasure Ġthe Ġsemantic Ġsimilarity Ġbetween Ġquestions Ġand Ġcandidate Ġanswers Ġin Ġthe Ġsemantic Ġspace . ĠFirst , Ġto Ġmake Ġthe Ġmodel 's Ġprediction Ġmore Ġexplain able , Ġwe Ġimplement Ġa Ġpath Ġbased ĠQ A Ġmodel ĠPC Net . ĠIn Ġthis Ġmodel , Ġto Ġget Ġcandidate Ġanswers Ġfrom Ġthe Ġtriple ts Ġbased Ġdocument Ġfor Ġa Ġgiven Ġquestion . ĠWe Ġfirst Ġretrieve Ġthe Ġan ĠâĢ ľ anch or âĢ Ŀ Ġpoint Ġ$ arg 1 _ i $ Ġor Ġ$ arg 2 _ i $ Ġin Ġthe Ġdocument Ġfact Ġ$ f _ i $ Ġ. ĠThese Ġanchors Ġare Ġselected Ġbased Ġon Ġedit Ġdistance Ġbetween Ġthe Ġwords Ġin Ġthe Ġquestions Ġand Ġthe Ġarguments . ĠThen Ġwe Ġregard Ġall Ġarguments Ġin Ġthe Ġ1 - hops Ġand Ġ2 - hops Ġfact Ġof Ġthe Ġanchors Ġas Ġanswers . ĠHowever , Ġthe Ġcoverage Ġof Ġthe Ġcandidate Ġanswers Ġcan Ġbe Ġ100 % Ġin Ġthe Ġmodel . ĠWe Ġthen Ġimplement Ġa Ġsecond Ġend - to - end Ġneural Ġmodel ĠK V Men Net Ġwhich Ġcovers Ġall Ġthe Ġanswers Ġbut Ġwith Ġless Ġinterpret ability . ĠBoth Ġmodels Ġgenerate Ġa Ġscore Ġ$ f _{ qa } ( q , Ġa ) $ Ġof Ġeach Ġcandidate Ġanswer . </s> We Ġthen Ġimplement Ġa Ġgeneration - based Ġmodel . ĠThe Ġmotivation Ġto Ġdesign Ġthis Ġmodel Ġis Ġthat Ġwe Ġwant Ġto Ġassociate Ġnatural Ġlanguage Ġphrases Ġwith Ġknowledge Ġbased Ġrepresentation . ĠIt Ġtakes Ġsemantics Ġof Ġa Ġcandidate Ġanswer Ġas Ġthe Ġinput Ġand Ġgenerates Ġa Ġquestion Ġ$\ hat { q } $ Ġ. ĠThen Ġa Ġparaph r asing Ġmodel Ġgives Ġa Ġscore Ġ$ f _{ q g } ( q , \ hat { q }) $ Ġ, Ġwhich Ġis Ġcomputed Ġbetween Ġthe Ġgenerated Ġquestion Ġ$\ hat { q } $ Ġand Ġthe Ġoriginal Ġquestion Ġ$ q $ Ġ, Ġas Ġthe Ġranking Ġscore . </s> We Ġget Ġthe Ġfinal Ġscores Ġ$ S ( q , a ) $ Ġused Ġfor Ġranking Ġas Ġfollows . </s> $$ S ( q , a ) Ġ= Ġ\ lambda Ġf _{ qa } ( q , Ġa ) Ġ+ Ġ( 1 - \ lambda Ġ) Ġf _{ q g } ( q , \ hat { q }) $$ Ġ Ġ Ġ( E q . Ġ7 ) </s> Moreover , Ġwe Ġincorporate Ġside Ġinformation Ġfrom Ġexternal ĠKB s Ġinto Ġthe Ġthree Ġmodels . ĠThe Ġdetails Ġof Ġhow Ġwe Ġuse Ġexternal ĠKB s Ġto Ġenhance Ġthe Ġrepresentation Ġof Ġelements Ġin Ġthe Ġdocument ĠKB Ġwill Ġbe Ġdescribed Ġin Ġsection ĠInc orpor ating ĠExternal ĠKnowledge . </s> The ĠQuestion ĠAn sw ering ĠModel </s> In Ġthis Ġsection , Ġwe Ġpresent Ġtwo Ġmatching Ġmodels Ġto Ġmeasure Ġthe Ġsemantic Ġrelated ness Ġbetween Ġthe Ġquestion Ġand Ġthe Ġcandidate Ġanswer Ġin Ġthe Ġvector Ġsemantic Ġspace . ĠAfterwards , Ġwe Ġintroduce Ġour Ġstrategy Ġthat Ġincorporates Ġopen ĠKB Ġas Ġexternal Ġknowledge Ġto Ġenhance Ġboth Ġmodels . </s> Q A ĠModel Ġ1 : ĠPC Net </s> We Ġfollow ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġb ordes 2014 question , d ong - E t Al : 2015 : AC L - IJ CN LP 1 Ġand Ġdevelop ĠPC Net , Ġwhich Ġis Ġshort Ġfor Ġpath - Ġand Ġcontext - Ġbased Ġneural Ġnetwork . ĠIn ĠPC Net , Ġcandidate Ġanswers Ġcome Ġfrom Ġarguments Ġof Ġthe Ġdocument ĠKB , Ġand Ġeach Ġcandidate Ġanswer Ġis Ġrepresented Ġwith Ġits Ġneighboring Ġarguments Ġand Ġpred icates Ġas Ġwell Ġas Ġits Ġpath Ġfrom Ġthe Ġanchor Ġin Ġthe Ġdocument ĠKB . ĠWe Ġuse Ġa Ġrule - based Ġapproach Ġbased Ġon Ġstring Ġfuzzy Ġmatch Ġto Ġdetect Ġthe Ġanchor . ĠEach Ġargument Ġis Ġmeasured Ġby Ġ$\ sum Ġ_ { i , j } ^{ }\ math bb Ġ{ I } ( arg _ i , Ġq _ j ) $ Ġ, Ġwhere Ġ$ i $ Ġand Ġ$ j $ Ġiter ate Ġacross Ġargument Ġwords Ġand Ġquestion Ġwords , Ġrespectively . Ġ$\ math bb Ġ{ I } ( x , y ) $ Ġis Ġan Ġindicator Ġfunction Ġwhose Ġvalue Ġis Ġ1 Ġif Ġthe Ġminimum Ġedit Ġdistance Ġbetween Ġ$ x $ Ġand Ġ$ y $ Ġis Ġno Ġmore Ġthan Ġ1 , Ġotherwise Ġit Ġis Ġ0 . ĠThe Ġarguments Ġlinked Ġto Ġthe Ġanchor Ġwith Ġ1 - hop Ġand Ġ2 - hop Ġpaths Ġare Ġregarded Ġas Ġcandidate Ġanswers . ĠSince Ġan Ġargument Ġmight Ġinclude Ġmultiple Ġwords , Ġsuch Ġas ĠâĢ ľ the Ġpopular Ġangry Ġbird Ġgame âĢ Ŀ , Ġwe Ġuse ĠGR U Ġbased ĠR NN Ġto Ġget Ġthe Ġvector Ġrepresentation Ġof Ġeach Ġargument / pred icate . ĠThe Ġpath Ġrepresentation Ġ$ v _{ p } $ Ġis Ġcomputed Ġby Ġaveraging Ġthe Ġvectors Ġof Ġelements Ġin Ġthe Ġpath . ĠSimilarly , Ġwe Ġuse Ġanother ĠR NN Ġto Ġget Ġthe Ġvector Ġof Ġeach Ġneighboring Ġargument / pred icate , Ġand Ġaverage Ġthem Ġto Ġget Ġthe Ġcontext Ġvector Ġ$ v _{ c } $ Ġ. </s> We Ġrepresent Ġthe Ġquestion Ġ$ q $ Ġusing Ġa Ġbid irection al ĠR NN Ġlayer Ġbased Ġon Ġthe ĠGR U Ġunit . ĠThe Ġconc aten ation Ġof Ġthe Ġlast Ġhidden Ġvectors Ġfrom Ġboth Ġdirections Ġis Ġused Ġas Ġthe Ġquestion Ġvector Ġ$ v _{ q } $ Ġ. ĠThe Ġdot Ġproduct Ġis Ġused Ġto Ġmeasure Ġthe Ġsemantic Ġrelevance Ġbetween Ġthe Ġb Ġb Ġquestion Ġand Ġtwo Ġtypes Ġof Ġevidence . </s> $$ f _{ qa } ( q , Ġa ) Ġ= Ġv _{ q } ^{ T } v _{ p } Ġ+ Ġv _ q ^{ T } v _{ c } $$ Ġ Ġ Ġ( E q . Ġ10 ) </s> Q A ĠModel Ġ2 : ĠK V Mem Net </s> Despite Ġthe Ġinterpret ability Ġof ĠPC Net , Ġthe Ġcoverage Ġof Ġanchor Ġdetection Ġlimits Ġthe Ġupper Ġbound Ġof Ġthe Ġapproach . ĠWe Ġimplement Ġa Ġmore Ġpowerful Ġmethod Ġbased Ġon Ġthe Ġkey - value Ġmemory Ġnetwork ĠB IB REF 19 Ġ, ĠK V Mem Net Ġfor Ġshort , Ġwhich Ġhas Ġproven Ġpowerful Ġin ĠKB - based Ġquestion Ġanswering . </s> The ĠK V Men Net Ġcould Ġbe Ġviewed Ġas Ġa ĠâĢ ľ soft âĢ Ŀ Ġmatching Ġapproach , Ġwhich Ġincludes Ġa Ġparameter ized Ġmemory Ġconsisting Ġof Ġkey - value Ġpairs . ĠInt uitive ly , Ġkeys Ġare Ġused Ġfor Ġmatching Ġwith Ġthe Ġquestion , Ġand Ġvalues Ġare Ġused Ġfor Ġmatching Ġto Ġthe Ġcandidate Ġanswer . ĠGiven Ġa ĠKB Ġfact Ġ( Ġ$ sub j , Ġpred , Ġobj $ Ġ), ĠWe Ġconsider Ġboth Ġdirections Ġand Ġadd Ġtwo Ġkey - value Ġpairs Ġin Ġthe Ġmemory , Ġnamely Ġ( Ġ$ key Ġ= Ġsub j Ġ+ pred , Ġvalue = obj $ Ġ) Ġand Ġ( Ġ$ key Ġ= obj Ġ+ Ġpred , Ġvalue = sub j $ Ġ). ĠThe Ġvectors Ġof Ġarguments / pred icates Ġare Ġcalculated Ġthe Ġsame Ġway Ġas Ġdescribed Ġin ĠPC Net . ĠThe Ġconc aten ation Ġof Ġtwo Ġvectors Ġis Ġused Ġas Ġthe Ġkey Ġvector Ġ$ v _{ key } $ Ġ. </s> Each Ġmemory Ġitem Ġis Ġassigned Ġa Ġrelevance Ġprobability Ġby Ġcomparing Ġthe Ġquestion Ġto Ġeach Ġkey . </s> $$ \ alpha Ġ_ { key _ i } Ġ= Ġ{ soft max } ( v _ q Ġ\ cd ot Ġv _{ key _ i }) $$ Ġ Ġ Ġ( E q . Ġ12 ) </s> After wards , Ġvectors Ġin Ġmemory Ġ( Ġ$ v _{ value } $ Ġ) Ġare Ġweighted Ġsummed Ġaccording Ġto Ġtheir Ġaddressing Ġprobabilities , Ġand Ġthe Ġvector Ġ$ v _ o $ Ġis Ġreturned . </s> $$ v _ o Ġ= Ġ\ sum Ġ_ i Ġ\ alpha Ġ_ { key _ i } v _{ value _ i } $$ Ġ Ġ Ġ( E q . Ġ13 ) </s> In ĠPC Net , Ġreasoning Ġover Ġtwo Ġfacts Ġis Ġachieved Ġby Ġincorporating Ġ2 - hop Ġpaths , Ġwhile Ġin ĠK V Mem Net Ġthis Ġis Ġachieved Ġby Ġrepeating Ġthe Ġmemory Ġaccess Ġprocess Ġtwice . ĠAfter Ġreceiving Ġthe Ġresult Ġ$ v _ o $ Ġ, Ġwe Ġupdate Ġthe Ġquery Ġwith Ġ$ q _ 2 Ġ= ĠR ( v _ q Ġ+ Ġv _ o ) $ Ġ, Ġwhere Ġ$ R $ Ġis Ġmodel Ġparameter . ĠFinally , Ġafter Ġa Ġfixed Ġnumber Ġ$ n $ Ġhops Ġ( Ġ$ n = 2 $ Ġin Ġthis Ġwork ), Ġthe Ġresulting Ġvector Ġis Ġused Ġto Ġmeasure Ġthe Ġrelevance Ġto Ġcandidate Ġanswers Ġvia Ġthe Ġdot Ġproduct . </s> Training Ġand ĠIn ference </s> Let Ġ$\ math cal Ġ{ D } Ġ= Ġ\ l brace Ġ( q _ i , Ġa _ i ); Ġi Ġ= Ġ1 , Ġ\ ld ots Ġ, Ġ| \ math cal Ġ{ D } | \ r brace Ġ$ Ġbe Ġthe Ġtraining Ġdata Ġconsisting Ġof Ġquestions Ġ$ q _ i $ Ġpaired Ġwith Ġtheir Ġcorrect Ġanswer Ġ$ a _ i $ Ġ. ĠWe Ġtrain Ġboth Ġmatching Ġmodels Ġwith Ġa Ġmargin - based Ġranking Ġloss Ġfunction , Ġwhich Ġis Ġcalculated Ġas Ġfollows , Ġwhere Ġ$ m $ Ġis Ġthe Ġmargin Ġ( fixed Ġto Ġ$ 0 . 1 $ Ġ) Ġand Ġ$\ bar { a } $ Ġis Ġrandomly Ġsampled Ġfrom Ġa Ġset Ġof Ġincorrect Ġcandidates Ġ$\ bar {\ math cal Ġ{ A }} $ Ġ. </s> $$ Ġ Ġ\ sum Ġ_ { i = 1 } ^{ | \ math cal Ġ{ D } | } Ġ\ sum Ġ_ {\ tiny Ġ{\ bar { a } Ġ\ in Ġ\ bar {\ math cal Ġ{ A }} ( q _ i ) }} Ġ\ max Ġ\ l brace Ġ0 , Ġm Ġ- ĠS ( q _ i , a _ i ) Ġ+ ĠS ( q _ i , \ bar { a } )\ r brace Ġ, $$ Ġ Ġ Ġ( E q . Ġ15 ) </s> For Ġtesting , Ġgiven Ġa Ġquestion Ġ$ q $ Ġ, Ġthe Ġmodel Ġpredicts Ġthe Ġanswer Ġbased Ġon Ġthe Ġfollowing Ġequation , Ġwhere Ġ$ Ġ\ math cal Ġ{ A } ( q ) $ Ġis Ġthe Ġcandidate Ġanswer Ġset . </s> $$ \ hat { a } Ġ= Ġ{\ mb ox { arg max }} _{ a ^ {\ prime Ġ} Ġ\ in Ġ\ math cal Ġ{ A } ( q )} ĠS ( q , a ^ {\ prime Ġ}) $$ Ġ Ġ Ġ( E q . Ġ16 ) </s> The ĠQuestion ĠGeneration ĠModel </s> In Ġthis Ġsection , Ġwe Ġpresent Ġthe Ġgeneration Ġmodel Ġwhich Ġgenerates Ġa Ġquestion Ġbased Ġon Ġthe Ġsemantics Ġof Ġa Ġcandidate Ġanswer . ĠAfter ward , Ġwe Ġintroduce Ġhow Ġour Ġparaph r asing Ġmodel , Ġwhich Ġmeasures Ġthe Ġsemantic Ġrelevance Ġbetween Ġthe Ġgenerated Ġquestion Ġand Ġthe Ġoriginal Ġquestion , Ġis Ġpret rained . </s> H ier arch ical Ġseq 2 seq ĠGeneration ĠModel </s> Our Ġquestion Ġgeneration Ġmodel Ġtakes Ġthe Ġpath Ġbetween Ġan ĠâĢ ľ anch or âĢ Ŀ Ġand Ġthe Ġcandidate Ġanswer , Ġand Ġoutputs Ġa Ġquestion . ĠWe Ġadopt Ġthe Ġsequence Ġto Ġsequence Ġarchitecture Ġas Ġour Ġbasic Ġquestion Ġgeneration Ġmodel Ġdue Ġto Ġits Ġeffectiveness Ġin Ġnatural Ġlanguage Ġgeneration Ġtasks . ĠThe Ġhierarchical Ġenc oder Ġand Ġthe Ġhierarchical Ġattention Ġmechanism Ġare Ġintroduced Ġto Ġtake Ġinto Ġaccount Ġthe Ġstructure Ġof Ġthe Ġinput Ġfacts . ĠFacts Ġfrom Ġexternal ĠKB s Ġare Ġconvention ally Ġintegrated Ġinto Ġthe Ġmodel Ġusing Ġthe Ġsame Ġway Ġas Ġdescribed Ġin Ġthe Ġmatching Ġmodel . </s> As Ġillustrated Ġin ĠFigure Ġ3 Ġ, Ġthe Ġquestion Ġgeneration Ġmodel Ġcontains Ġan Ġenc oder Ġand Ġa Ġdec oder . ĠWe Ġuse Ġa Ġhierarchical Ġenc oder Ġconsisting Ġof Ġtwo Ġlayers Ġto Ġmodel Ġthe Ġmeaning Ġof Ġeach Ġelement Ġ( subject , Ġpredicate Ġor Ġobject ) Ġand Ġthe Ġrelationship Ġbetween Ġthem . ĠSince Ġeach Ġelement Ġmight Ġcontain Ġmultiple Ġwords , Ġwe Ġuse ĠR NN Ġas Ġthe Ġword Ġlayer Ġenc oder Ġto Ġget Ġthe Ġrepresentation Ġof Ġeach Ġelement . ĠWe Ġdefine Ġthe Ġfact Ġlevel Ġsequence Ġas Ġa Ġpath Ġstarting Ġfrom Ġthe Ġanchor Ġand Ġending Ġat Ġthe Ġcandidate Ġanswer . ĠAnother ĠR NN Ġis Ġused Ġas Ġthe Ġfact Ġlevel Ġenc oder . ĠThe Ġlast Ġhidden Ġstate Ġat Ġthe Ġfact Ġlayer Ġis Ġfed Ġto Ġthe Ġdec oder . </s> We Ġdevelop Ġa Ġhierarchical Ġattention Ġmechanism Ġin Ġthe Ġdec oder , Ġwhich Ġfirst Ġmakes Ġsoft Ġalignment Ġover Ġthe Ġhidden Ġstates Ġat Ġthe Ġfact Ġlayer , Ġthe Ġoutput Ġof Ġwhich Ġis Ġfurther Ġused Ġto Ġattend Ġto Ġhidden Ġstates Ġat Ġthe Ġword Ġlevel . ĠSpecifically , Ġgiven Ġthe Ġdec oder Ġhidden Ġstate Ġ$ h ^{ dec } _{ t - 1 } $ Ġ, Ġwe Ġuse Ġa Ġfact - level Ġattention Ġmodel Ġ$ Att _{ f ct } $ Ġto Ġcalculate Ġthe Ġcontextual Ġvector , Ġwhich Ġis Ġfurther Ġcombined Ġwith Ġthe Ġcurrent Ġhidden Ġstate Ġ$ h ^{ dec } _{ t } $ Ġ, Ġresulting Ġin Ġ$ c _{ f ct } $ Ġ. ĠThe Ġcontextual Ġvector Ġis Ġcalculated Ġthrough Ġweighted Ġaveraging Ġover Ġhidden Ġvectors Ġat Ġthe Ġfact Ġlevel , Ġwhich Ġis Ġgiven Ġas Ġfollows , Ġwhere Ġ$\ od ot Ġ$ Ġis Ġdot Ġproduct , Ġ$ h ^{ f ld } _ j $ Ġis Ġthe Ġ$ j $ Ġ- th Ġhidden Ġstate Ġat Ġthe Ġfact Ġlayer Ġfrom Ġthe Ġenc oder , Ġand Ġ$ l _ f $ Ġis Ġthe Ġnumber Ġof Ġhidden Ġstates Ġat Ġthe Ġfact Ġlevel . </s> $$ c _{ f ct } Ġ& = ĠGR U ( h ^{ dec } _{ t }, Ġ\ sum Ġ_ { j = 1 } ^ {{ l _ f }} Ġ\ alpha Ġ_ { t j } h ^{ f ct } _ j ) Ġ\\ Ġ\ alpha Ġ_ { t j } Ġ& = Ġ\ frac { Ġexp ( h ^{ dec } _{ t - 1 } Ġ\ od ot Ġh ^{ f ct } _ j ) }{ \ sum Ġ_ { k = 1 } ^ {{ l _ f }} exp ( h ^{ dec } _{ t - 1 } Ġ\ od ot Ġh ^{ f ct } _ k )} $$ Ġ Ġ Ġ( E q . Ġ19 ) </s> Similarly , Ġwe Ġfeed Ġ$ c _{ f ct } $ Ġto Ġthe Ġword - level Ġattention Ġfunction Ġ$ Att _{ w rd } $ Ġand Ġcalculate Ġover Ġhidden Ġvectors Ġat Ġthe Ġword - level . ĠThe Ġoutput Ġ$ c _{ w rd } $ Ġwill Ġbe Ġconc aten ated Ġwith Ġ$ h ^{ dec } _{ t } $ Ġto Ġpredict Ġthe Ġnext Ġword . </s> Since Ġmany Ġentity Ġnames Ġof Ġgreat Ġimportance Ġare Ġrare Ġwords Ġfrom Ġthe Ġinput , Ġwe Ġuse Ġthe Ġcopying Ġmechanism ĠB IB REF 20 Ġthat Ġlearns Ġwhen Ġto Ġreplicate Ġwords Ġfrom Ġthe Ġinput Ġor Ġto Ġpredict Ġwords Ġfrom Ġthe Ġtarget Ġvocabulary . ĠThe Ġprobability Ġdistribution Ġof Ġgenerating Ġthe Ġword Ġ$ y $ Ġis Ġcalculated Ġas Ġfollows , Ġin Ġwhich Ġthe Ġ$ soft max $ Ġfunction Ġis Ġcalculated Ġover Ġa Ġcombined Ġlog its Ġfrom Ġboth Ġsides . </s> $$ \ begin { split } Ġ& p ( y )= \ frac { exp ({ e _ y Ġ\ od ot ĠW _ g [ h _ t ^{ dec }; c _{ w rd } ] }) + exp ( s _ c ( y )) }{ Z } \\ Ġ& s _ c ( y ) Ġ= Ġc _{ w rd } Ġ\ od ot Ġ{ tan h ( W _ ch _ t ^{ w rd }) } Ġ\ end { split } $$ Ġ Ġ Ġ( E q . Ġ20 ) </s> We Ġtrain Ġour Ġquestion Ġgeneration Ġmodel Ġwith Ġmaximum Ġlikelihood Ġestimation . ĠThe Ġloss Ġfunction Ġis Ġgiven Ġas Ġfollows , Ġwhere Ġ$ D $ Ġis Ġthe Ġtraining Ġcorpus . ĠWe Ġuse Ġbeam Ġsearch Ġin Ġthe Ġinference Ġprocess . </s> $$ l Ġ= Ġ- \ sum Ġ_ { ( x , y )\ in ĠD } Ġ\ sum Ġ_ t Ġlog Ġp ( y _ t Ġ| y _{ < t }, Ġx ) $$ Ġ Ġ Ġ( E q . Ġ21 ) </s> An Ġadvantage Ġof Ġthe Ġmodel Ġis Ġthat Ġexternal Ġknowledge Ġcould Ġbe Ġeasily Ġincorporated Ġwith Ġthe Ġsame Ġmechanism Ġas Ġwe Ġhave Ġdescribed Ġin Ġsection ĠInc orpor ating ĠExternal ĠKnowledge . ĠWe Ġenhance Ġthe Ġrepresentation Ġof Ġan Ġargument Ġor Ġa Ġpredicate Ġby Ġconc aten ating Ġopen ĠKB Ġvectors Ġinto Ġenc oder Ġhidden Ġstates . </s> The ĠPar aph r asing ĠModel </s> The Ġparaph r asing Ġmodel Ġis Ġused Ġto Ġmeasure Ġthe Ġsemantic Ġrelevance Ġbetween Ġthe Ġoriginal Ġquestion Ġand Ġthe Ġquestion Ġgenerated Ġfrom Ġthe ĠQ G Ġmodel . ĠWe Ġuse Ġbid irection al ĠR NN Ġwith Ġg ated Ġrecurrent Ġunit Ġto Ġrepresent Ġtwo Ġquestions , Ġand Ġcompose Ġthem Ġwith Ġelement - wise Ġmultiplication . ĠThe Ġresults Ġare Ġfollowed Ġby Ġa Ġ$ soft max $ Ġlayer , Ġwhose Ġoutput Ġlength Ġis Ġ2 . ĠThe Ġmodel Ġis Ġtrained Ġby Ġminimizing Ġthe Ġcross - ent ropy Ġerror , Ġin Ġwhich Ġthe Ġsupervision Ġis Ġprovided Ġin Ġthe Ġtraining Ġdata . </s> We Ġcollect Ġtwo Ġdatasets Ġto Ġtrain Ġthe Ġparaph r asing Ġmodel . ĠThe Ġfirst Ġdataset Ġis Ġfrom ĠQu ora Ġdataset , Ġwhich Ġis Ġbuilt Ġfor Ġdetecting Ġwhether Ġor Ġnot Ġa Ġpair Ġof Ġquestions Ġare Ġsem antically Ġequivalent . Ġ345 , 989 Ġpositive Ġquestion Ġpairs Ġand Ġ255 , 027 Ġnegative Ġpairs Ġare Ġincluded Ġin Ġthe Ġfirst Ġdataset . ĠThe Ġsecond Ġdataset Ġincludes Ġweb Ġqueries Ġfrom Ġquery Ġlogs , Ġwhich Ġare Ġobtained Ġby Ġclust ering Ġthe Ġweb Ġqueries Ġthat Ġclick Ġthe Ġsame Ġweb Ġpage . ĠIn Ġthis Ġway Ġwe Ġobtain Ġ6 , 118 , 023 Ġpositive Ġquery Ġpairs . ĠWe Ġimplement Ġa Ġhe uristic Ġrule Ġto Ġget Ġ6 , 118 , 023 Ġnegative Ġinstances Ġfor Ġthe Ġquery Ġdataset . ĠFor Ġeach Ġpair Ġof Ġquery Ġtext , Ġwe Ġclamp Ġthe Ġfirst Ġquery Ġand Ġretrieve Ġa Ġquery Ġthat Ġis Ġmostly Ġsimilar Ġto Ġthe Ġsecond Ġquery . ĠTo Ġimprove Ġthe Ġefficiency Ġof Ġthis Ġprocess , Ġwe Ġrandomly Ġsample Ġ10 , 000 Ġqueries Ġand Ġdefine Ġthe ĠâĢ ľ similar ity âĢ Ŀ Ġas Ġthe Ġnumber Ġof Ġco - occ urred Ġwords Ġbetween Ġtwo Ġquestions . ĠDuring Ġtraining , Ġwe Ġinitialize Ġthe Ġvalues Ġof Ġword Ġembed d ings Ġwith Ġ$ 300 d $ ĠGl ove Ġvectors , Ġwhich Ġis Ġlearned Ġon ĠWikipedia Ġtexts . ĠWe Ġuse Ġa Ġheld - out Ġdata Ġconsisting Ġof Ġ20 K Ġquery Ġpairs Ġto Ġcheck Ġthe Ġperformance Ġof Ġthe Ġparaph r asing Ġmodel . ĠThe Ġaccuracy Ġof Ġthe Ġparaph r asing Ġmodel Ġon Ġthe Ġheld - out Ġdataset Ġis Ġ87 . 36 %. </s> In cor por ating ĠExternal ĠKnowledge </s> There Ġare Ġmany Ġpossible Ġways Ġto Ġimplement Ġthe Ġidea Ġof Ġimproving Ġquestion Ġanswering Ġwith Ġexternal ĠKB . ĠIn Ġthis Ġwork , Ġwe Ġuse Ġexternal ĠKB s Ġ( such Ġas ĠNE LL Ġand ĠPro Base ) Ġto Ġenhance Ġthe Ġrepresentations Ġof Ġelements Ġin Ġthe Ġdocument ĠKB . ĠFor Ġinstance , Ġthe Ġargument ĠâĢ ľ the Ġsequence Ġof Ġamino Ġacids âĢ Ŀ Ġin ĠFigure Ġ1 Ġfrom Ġthe Ġdocument ĠKB Ġretrie ves Ġ( âĢ ľ am ino Ġacids âĢ Ŀ , Ġ` is âĢ Ŀ , ĠâĢ ľ protein âĢ Ŀ ) Ġfrom ĠNE LL . ĠEnhanced Ġwith Ġthis Ġadditional Ġclue , Ġthe Ġoriginal Ġargument Ġis Ġa Ġbetter Ġmatch Ġto Ġthe Ġquestion . </s> Similar Ġto ĠB IB REF 21 Ġk hot 2017 ans w ering , Ġwe Ġuse ĠElastic Search Ġto Ġretrieve Ġfacts Ġfrom Ġopen ĠKB s . ĠWe Ġremove Ġstop Ġwords Ġfrom Ġtokens Ġof Ġeach Ġargument Ġand Ġpredicate Ġin Ġa Ġdocument ĠKB Ġand Ġregard Ġthe Ġremained Ġwords Ġas ĠElastic Search Ġqueries . ĠWe Ġset Ġdifferent Ġsearch Ġoption Ġfor Ġarguments Ġand Ġpred icates , Ġnamely Ġsetting Ġarguments Ġas Ġthe Ġdominant Ġsearch able Ġfields Ġfor Ġargument Ġqueries Ġand Ġsetting Ġpred icates Ġas Ġthe Ġdominant Ġsearch able Ġfields Ġfor Ġpredicate Ġqueries . ĠWe Ġsave Ġthe Ġtop Ġ10 Ġhints Ġand Ġselectively Ġuse Ġthem Ġin Ġthe Ġexperiment . </s> We Ġregard Ġthe Ġretrieved Ġfacts Ġfrom Ġthe Ġexternal ĠKB Ġas Ġneighbors Ġto Ġthe Ġarguments Ġto Ġbe Ġenhanced . ĠInspired Ġby ĠB IB REF 22 Ġscar sell i 2009 graph , Ġwe Ġupdate Ġthe Ġvector Ġof Ġan Ġelement Ġ$ v _ e $ Ġas Ġfollows , Ġwhere Ġ$( e ) $ Ġand Ġ$( e ) $ Ġrepresent Ġadjacent Ġarguments Ġfrom Ġthe Ġfacts Ġretrieved Ġby Ġobject Ġand Ġsubject , Ġrespectively . ĠIn Ġthis Ġwork , Ġ$ f (\ cd ot Ġ) $ Ġis Ġimplemented Ġby Ġaveraging Ġthe Ġvectors Ġof Ġtwo Ġarguments . </s> $$ v _ e = v _ e Ġ+ Ġ\ sum Ġ_ { o ^ {\ prime Ġ} Ġ\ in Ġ( e )} ^{ } f ( v _{ p ^ {\ prime Ġ} }, v _{ s ^ {\ prime Ġ} }) + \ sum Ġ_ { s ^ {\ prime Ġ} Ġ\ in Ġ( e )} ^{ } f ( v _{ p ^ {\ prime Ġ} }, v _{ o ^ {\ prime Ġ} }) $$ Ġ Ġ Ġ( E q . Ġ26 ) </s> Related ĠWork </s> The Ġtask Ġof ĠK BM RC Ġdiffers Ġfrom Ġmachine Ġreading Ġcomprehension Ġ( M RC ) Ġin Ġboth Ġinput Ġand Ġoutput Ġaspects . ĠThe Ġinput Ġof ĠK BM RC Ġis Ġthe Ġknowledge Ġincluding Ġboth Ġword Ġknowledge Ġextracted Ġfrom Ġthe Ġdocument Ġand Ġworld Ġknowledge Ġretrieved Ġfrom Ġexternal Ġknowledge Ġbase , Ġwhile Ġthe Ġinput Ġof ĠM RC Ġis Ġthe Ġun struct ured Ġtext Ġof Ġa Ġdocument . ĠThe Ġoutput Ġof ĠK BM RC Ġis Ġa Ġsubject Ġor Ġan Ġargument , Ġwhile Ġthe Ġoutput Ġin ĠM RC Ġis Ġa Ġtext Ġspan Ġof Ġthe Ġdocument . ĠMeanwhile , ĠK BM RC Ġfacilitates Ġthe Ġaccessing Ġand Ġleveraging Ġof Ġknowledge Ġfrom Ġexternal ĠKB s Ġbecause Ġthe Ġdocument ĠKB Ġis Ġconsistent Ġwith Ġthe Ġrepresentation Ġof Ġfacts Ġin Ġexternal ĠKB s . </s> K BM RC Ġalso Ġrelates Ġto Ġknowledge - base Ġquestion Ġanswering Ġ( KB Q A ) ĠB IB REF 23 Ġ, Ġwhich Ġaims Ġto Ġanswer Ġquestions Ġbased Ġon Ġan Ġexternal Ġlarge - scale ĠKB Ġsuch Ġas ĠFree base Ġor ĠPro Base . ĠK BM RC Ġdiffers Ġfrom ĠKB Q A Ġin Ġthat Ġthe Ġoriginal ĠKB Ġcomes Ġfrom Ġthe Ġcontent Ġof Ġa Ġdocument . ĠExternal ĠKB Ġis Ġused Ġin Ġthis Ġwork Ġto Ġenhance Ġthe Ġdocument ĠKB . ĠMoreover , Ġexisting Ġbenchmark Ġdatasets Ġfor ĠKB Q A Ġsuch Ġas ĠWeb Questions ĠB IB REF 24 Ġare Ġtypically Ġlimited Ġto Ġsimple Ġquestions . ĠThe ĠK BM RC Ġtask Ġrequires Ġreasoning Ġover Ġtwo Ġfacts Ġfrom Ġthe Ġdocument ĠKB . </s> Our Ġapproach Ġdraws Ġinspiration Ġfrom Ġtwo Ġmain Ġclasses Ġin Ġexisting Ġapproaches Ġof ĠKB Q A , Ġnamely Ġranking Ġbased Ġand Ġparsing Ġbased . ĠRanking Ġbased Ġapproaches ĠB IB REF 17 Ġ, ĠB IB REF 25 Ġare Ġbottom - up , Ġwhich Ġtypically Ġfirst Ġfind Ġa Ġset Ġof Ġcandidate Ġanswers Ġand Ġthen Ġrank Ġbetween Ġthe Ġcandidates Ġwith Ġfeatures Ġat Ġdifferent Ġlevels Ġto Ġget Ġthe Ġanswer . ĠPars ing - based Ġapproaches ĠB IB REF 16 Ġare Ġtop - down , Ġwhich Ġfirst Ġinterpret Ġlogical Ġform Ġfrom Ġa Ġnatural Ġlanguage Ġutter ance , Ġand Ġthen Ġdo Ġexecution Ġto Ġyield Ġthe Ġanswer . ĠRanking - based Ġapproaches Ġachieve Ġbetter Ġperformances Ġthan Ġparsing - based Ġapproaches Ġon ĠWeb Questions , Ġa Ġbenchmark Ġdataset Ġfor ĠKB Q A . ĠWe Ġfollow Ġranking - based Ġapproaches , Ġand Ġdevelop Ġboth Ġa Ġmatching - based Ġmodel Ġwith Ġfeatures Ġat Ġdifferent Ġlevels Ġand Ġa Ġquestion Ġgeneration Ġmodel . ĠMore Ġreferences Ġcan Ġbe Ġfound Ġat Ġhttps :// acl web . org / acl wiki / Question _ An sw ering _( State _ of _ the _ art ). </s> Our Ġwork Ġalso Ġrelates Ġto ĠB IB REF 21 Ġk hot 2017 ans w ering , Ġwhich Ġuses Ġopen ĠIE Ġoutputs Ġfrom Ġexternal Ġtext Ġcorpor a Ġto Ġimprove Ġmulti - choice Ġquestion Ġanswering . ĠHowever , Ġour Ġwork Ġdiffers Ġfrom Ġthem Ġin Ġthat Ġtheir Ġtask Ġdoes Ġnot Ġcontain Ġdocument Ġinformation . ĠFurthermore , Ġwe Ġdevelop Ġa Ġquestion Ġgeneration Ġapproach Ġwhile Ġthey Ġregard Ġthe ĠQ A Ġtask Ġas Ġsub graph Ġsearch Ġbased Ġon Ġan Ġinteger Ġlinear Ġprogramming Ġ( IL P ) Ġapproach . ĠOur Ġwork Ġalso Ġrelates Ġto ĠB IB REF 26 Ġk hash abi 2018 question , Ġwhich Ġfocuses Ġon Ġmulti - choice Ġquestion Ġanswering Ġbased Ġon Ġthe Ġsemantics Ġof Ġa Ġdocument . ĠThey Ġuse Ġsemantic Ġrole Ġlabeling Ġand Ġshallow Ġparsing Ġof Ġa Ġdocument Ġto Ġconstruct Ġa Ġsemantic Ġgraph , Ġbased Ġon Ġwhich Ġan ĠIL P Ġbased Ġapproach Ġis Ġdeveloped Ġto Ġfind Ġthe Ġsupporting Ġsub graph . ĠThe Ġdifference Ġof Ġour Ġapproach Ġis Ġthat Ġpred icates Ġfrom Ġour Ġdocument ĠKB Ġform Ġare Ġnot Ġlimited Ġto Ġa Ġpred efined Ġset , Ġso Ġthat Ġthey Ġdo Ġnot Ġtake Ġinto Ġconsideration Ġthe Ġknowledge Ġfrom Ġexternal ĠKB s , Ġand Ġalso Ġthe Ġdifference Ġin Ġterms Ġof Ġmethodology . ĠB IB REF 19 Ġmill er 2016 key Ġanswer Ġquestions Ġbased Ġon ĠKB s Ġin Ġthe Ġmovie Ġdomain Ġor Ġinformation Ġextraction Ġresults Ġfrom ĠWikipedia Ġdocuments . ĠUnlike Ġthis Ġmethod , Ġour Ġapproach Ġfocuses Ġon Ġentities Ġfrom Ġan Ġexternal ĠKB , Ġour Ġdoc ĠKB Ġis Ġobtained Ġvia Ġopen ĠIE , Ġand Ġwe Ġcombine Ġthe Ġdocument ĠKB Ġwith Ġan Ġopen ĠKB Ġfor Ġquestion Ġanswering . </s> Exper iments </s> We Ġdescribe Ġexperiment Ġsettings Ġand Ġreport Ġfigures Ġand Ġanalysis Ġin Ġthis Ġsection . </s> Settings </s> In Ġour Ġexperiments , Ġwe Ġtune Ġmodel Ġparameters Ġon Ġthe Ġdevelopment Ġset Ġand Ġreport Ġresults Ġon Ġthe Ġtest Ġset . ĠWe Ġdesign Ġexperiments Ġfrom Ġboth Ġranking - based Ġdirection Ġand Ġquestion Ġgeneration - based Ġdirection . ĠThe Ġevaluation Ġmetric Ġis Ġprecision Ġ@ 1 ĠB IB REF 17 Ġ, Ġwhich Ġindicates Ġwhether Ġthe Ġtop Ġranked Ġresult Ġis Ġthe Ġcorrect Ġanswer . ĠWe Ġfurther Ġreport ĠB LE U Ġscore ĠB IB REF 27 Ġfor Ġthe Ġquestion Ġgeneration Ġapproach . </s> We Ġalso Ġadapt ĠBi DA F ĠB IB REF 10 Ġ, Ġa Ġtop - performing Ġreading Ġcomprehension Ġmodel Ġon Ġthe ĠS Qu AD Ġdataset ĠB IB REF 0 Ġas Ġa Ġstrong Ġbaseline . ĠAs ĠBi DA F Ġoutput Ġa Ġspan Ġfrom Ġthe Ġinput Ġdocument Ġas Ġthe Ġanswer Ġto Ġthe Ġgiven Ġquestion , Ġwe Ġadapt Ġit Ġto ĠK BM RC Ġas Ġa Ġranking Ġmodel Ġsimilarly Ġas Ġthe Ġapproach Ġused Ġin Ġprevious Ġresearch ĠB IB REF 28 Ġ. ĠWe Ġuse ĠBi DA F Ġto Ġselect Ġan Ġanswer Ġspan Ġfrom Ġa Ġcorresponding Ġdocument Ġbased Ġon Ġa Ġgiven Ġquestion Ġand Ġselect Ġthe Ġcandidate Ġanswer Ġthat Ġhas Ġmaximum Ġoverlap Ġwith Ġthe Ġanswer Ġspan Ġas Ġthe Ġfinal Ġanswer . </s> Analysis : ĠQuestion ĠAn sw ering ĠModels </s> Table Ġ3 Ġshows Ġthe Ġresults Ġof Ġour Ġtwo Ġquestion Ġanswering Ġmodels . ĠIt Ġis Ġclear Ġthat ĠK V Mem Net Ġachieves Ġbetter ĠP @ 1 Ġscores Ġon Ġboth Ġdev Ġand Ġtest Ġsets Ġthan ĠPC Net . ĠThe Ġreason Ġis Ġthat Ġcandidate Ġanswers Ġof ĠPC Net Ġcome Ġfrom Ġthe ĠâĢ ľ anch or âĢ Ŀ Ġpoint Ġalong Ġ1 - hop Ġor Ġ2 - hop Ġpaths . ĠHowever , Ġthe Ġcorrect Ġanswer Ġmight Ġnot Ġbe Ġconnected Ġdue Ġto Ġthe Ġquality Ġof Ġanchor Ġdetection . ĠOn Ġthe Ġdev Ġset , Ġwe Ġobserve Ġthat Ġonly Ġ69 . 6 % Ġof Ġcorrect Ġanswers Ġcan Ġbe Ġcovered Ġby Ġthe Ġset Ġof Ġcandidate Ġanswers Ġin ĠPC Net , Ġwhich Ġapparently Ġlimits Ġthe Ġupper Ġbound Ġof Ġthe Ġapproach . ĠThis Ġis Ġaddressed Ġin ĠK V Mem Net Ġbecause Ġall Ġthe Ġarguments Ġare Ġcandidate Ġanswers . ĠBoth ĠPC Net Ġand ĠK V Mem Net Ġoutper form Ġour Ġimplementation Ġof ĠB IB REF 17 Ġb ordes 2014 question , Ġsince Ġthe Ġlatter Ġignores Ġword Ġorder . ĠWe Ġincorporate Ġeach Ġof Ġthe Ġfour ĠKB s Ġseparately Ġinto ĠPC Net Ġand ĠK V Mem Net , Ġand Ġfind Ġthat Ġincorporating Ġexternal ĠKB s Ġcould Ġbring Ġimprovements . </s> From ĠFigure Ġ4 Ġ, Ġwe Ġcan Ġsee Ġthat Ġthe ĠK V Mem Net Ġmodel Ġattends Ġto Ġthe Ġkey ĠâĢ ľ ( st . Ġjohn s , Ġis Ġlocated ) âĢ Ŀ Ġfor Ġthe Ġquestion ĠâĢ ľ Where Ġis Ġst Ġjohn s Ġmi Ġlocated ? âĢ Ŀ . ĠThus , Ġthe Ġmodel Ġhas Ġhigher Ġconfidence Ġin Ġregarding Ġvalue ĠâĢ ľ in Ġcl inton Ġcounty âĢ Ŀ Ġas Ġthe Ġanswer . </s> Analysis : ĠGener ative ĠModels </s> Table Ġ4 Ġshows Ġthe Ġresults Ġof Ġdifferent Ġquestion Ġgeneration Ġmodels . ĠOur Ġapproach Ġis Ġabbrevi ated Ġas ĠQ G Net , Ġwhich Ġstands Ġfor Ġthe Ġuse Ġof Ġa Ġparaph r asing Ġmodel Ġplus Ġour Ġquestion Ġgeneration Ġmodel . ĠWe Ġcan Ġsee Ġthat ĠQ G Net Ġperforms Ġbetter Ġthan ĠSe q 2 Se q Ġin Ġterms Ġof ĠB LE U Ġscore Ġbecause Ġmany Ġimportant Ġwords Ġof Ġlow Ġfrequency Ġfrom Ġthe Ġinput Ġare Ġreplicated Ġto Ġthe Ġtarget Ġsequence . ĠHowever , Ġthe Ġimprovement Ġis Ġnot Ġsignificant Ġfor Ġthe ĠQ A Ġtask . ĠWe Ġalso Ġincorporate Ġeach Ġof Ġthe Ġfour ĠKB s Ġinto ĠQ G Net , Ġand Ġobserve Ġslight Ġimprovements Ġon ĠNE LL Ġand ĠRe verb . ĠDespite Ġthe Ġoverall Ġaccuracy Ġof ĠQ G Net Ġbeing Ġlower Ġthan ĠPC Net Ġand ĠK V Mem Net , Ġcombining Ġoutcomes Ġwith Ġthem Ġcould Ġgenerates Ġ1 . 5 % Ġand Ġ0 . 8 % Ġabsolute Ġgains , Ġrespectively . </s> We Ġshow Ġexamples Ġgenerated Ġby Ġour ĠQ G Ġmodel Ġin ĠFigure Ġ5 Ġ, Ġin Ġwhich Ġthe Ġpaths Ġof Ġtwo Ġcandidate Ġanswers Ġare Ġregarded Ġas Ġthe Ġinput Ġto Ġthe ĠQ G Ġmodel . ĠWe Ġcan Ġsee Ġthat Ġthe Ġoriginal Ġquestion Ġis Ġcloser Ġto Ġthe Ġfirst Ġgenerated Ġresult Ġthan Ġthe Ġsecond Ġone . ĠAccordingly , Ġthe Ġfirst Ġcandidate Ġ($ Ġ61 , 300 ) Ġwould Ġbe Ġassigned Ġwith Ġa Ġlarger Ġprobability Ġas Ġthe Ġanswer . </s> Future ĠOpportun ities Ġfor ĠFuture ĠResearch </s> We Ġconduct Ġerror Ġanalysis Ġfrom Ġmultiple Ġperspectives Ġto Ġshow Ġthe Ġlimitations Ġand Ġfuture Ġopportunities Ġof Ġdifferent Ġcomponents Ġof Ġour Ġapproach . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġfocus Ġon Ġknowledge Ġbased Ġmachine Ġreading Ġcomprehension . ĠWe Ġcreate Ġa Ġmanually Ġlabeled Ġdataset Ġfor Ġthe Ġtask , Ġand Ġdevelop Ġa Ġframework Ġconsisting Ġof Ġboth Ġquestion Ġanswering Ġmodel Ġand Ġquestion Ġgeneration Ġmodel . ĠWe Ġfurther Ġincorporate Ġfour Ġopen ĠKB s Ġas Ġexternal Ġknowledge Ġinto Ġboth Ġquestion Ġanswering Ġand Ġgener ative Ġapproaches , Ġand Ġdemonstrate Ġthat Ġincorporating Ġadditional Ġevidence Ġfrom Ġopen ĠKB s Ġimproves Ġtotal Ġaccuracy . ĠWe Ġconduct Ġextensive Ġmodel Ġanalysis Ġand Ġerror Ġanalysis Ġto Ġshow Ġthe Ġadvantages Ġand Ġlimitations Ġof Ġour Ġapproaches .