answer questions by obtaining information from KB tuples 
EXTRACTIVE
hierarchical matching between questions and relations with residual learning
EXTRACTIVE
Knowledge Base Question Answering (KBQA) systems answer questions by obtaining information from KB tuples BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 . For an input question, these systems typically generate a KB query, which can be executed to retrieve the answers from a KB. Figure 1 illustrates the process used to parse two sample questions in a KBQA system: (a) a single-relation question, which can be answered with a single $<$ head-entity, relation, tail-entity $>$ KB tuple BIBREF6 , BIBREF7 , BIBREF2 ; and (b) a more complex case, where some constraints need to be handled for multiple entities in the question. The KBQA system in the figure performs two key tasks: (1) entity linking, which links $n$ -grams in questions to KB entities, and (2) relation detection, which identifies the KB relation(s) a question refers to.
Our main contributions include: (i) An improved relation detection model by hierarchical matching between questions and relations with residual learning; (ii) We demonstrate that the improved relation detector enables our simple KBQA system to achieve state-of-the-art results on both single-relation and multi-relation KBQA tasks.
1704.06194
Introduction </s> Know ledge ĠBase ĠQuestion ĠAn sw ering Ġ( KB Q A ) Ġsystems Ġanswer Ġquestions Ġby Ġobtaining Ġinformation Ġfrom ĠKB Ġtu ples ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ. ĠFor Ġan Ġinput Ġquestion , Ġthese Ġsystems Ġtypically Ġgenerate Ġa ĠKB Ġquery , Ġwhich Ġcan Ġbe Ġexecuted Ġto Ġretrieve Ġthe Ġanswers Ġfrom Ġa ĠKB . ĠFigure Ġ1 Ġillustrates Ġthe Ġprocess Ġused Ġto Ġparse Ġtwo Ġsample Ġquestions Ġin Ġa ĠKB Q A Ġsystem : Ġ( a ) Ġa Ġsingle - relation Ġquestion , Ġwhich Ġcan Ġbe Ġanswered Ġwith Ġa Ġsingle Ġ$ < $ Ġhead - entity , Ġrelation , Ġtail - entity Ġ$ > $ ĠKB Ġtuple ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 2 Ġ; Ġand Ġ( b ) Ġa Ġmore Ġcomplex Ġcase , Ġwhere Ġsome Ġconstraints Ġneed Ġto Ġbe Ġhandled Ġfor Ġmultiple Ġentities Ġin Ġthe Ġquestion . ĠThe ĠKB Q A Ġsystem Ġin Ġthe Ġfigure Ġperforms Ġtwo Ġkey Ġtasks : Ġ( 1 ) Ġentity Ġlinking , Ġwhich Ġlinks Ġ$ n $ Ġ- gram s Ġin Ġquestions Ġto ĠKB Ġentities , Ġand Ġ( 2 ) Ġrelation Ġdetection , Ġwhich Ġidentifies Ġthe ĠKB Ġrelation ( s ) Ġa Ġquestion Ġrefers Ġto . </s> The Ġmain Ġfocus Ġof Ġthis Ġwork Ġis Ġto Ġimprove Ġthe Ġrelation Ġdetection Ġsubt ask Ġand Ġfurther Ġexplore Ġhow Ġit Ġcan Ġcontribute Ġto Ġthe ĠKB Q A Ġsystem . ĠAlthough Ġgeneral Ġrelation Ġdetection Ġmethods Ġare Ġwell Ġstudied Ġin Ġthe ĠN LP Ġcommunity , Ġsuch Ġstudies Ġusually Ġdo Ġnot Ġtake Ġthe Ġend Ġtask Ġof ĠKB Q A Ġinto Ġconsideration . ĠAs Ġa Ġresult , Ġthere Ġis Ġa Ġsignificant Ġgap Ġbetween Ġgeneral Ġrelation Ġdetection Ġstudies Ġand ĠKB - specific Ġrelation Ġdetection . ĠFirst , Ġin Ġmost Ġgeneral Ġrelation Ġdetection Ġtasks , Ġthe Ġnumber Ġof Ġtarget Ġrelations Ġis Ġlimited , Ġnormally Ġsmaller Ġthan Ġ100 . ĠIn Ġcontrast , Ġin ĠKB Q A Ġeven Ġa Ġsmall ĠKB , Ġlike ĠFree base 2 M ĠB IB REF 2 Ġ, Ġcontains Ġmore Ġthan Ġ6 , 000 Ġrelation Ġtypes . ĠSecond , Ġrelation Ġdetection Ġfor ĠKB Q A Ġoften Ġbecomes Ġa Ġzero - shot Ġlearning Ġtask , Ġsince Ġsome Ġtest Ġinstances Ġmay Ġhave Ġunseen Ġrelations Ġin Ġthe Ġtraining Ġdata . ĠFor Ġexample , Ġthe ĠSimple Questions ĠB IB REF 2 Ġdata Ġset Ġhas Ġ14 % Ġof Ġthe Ġgolden Ġtest Ġrelations Ġnot Ġobserved Ġin Ġgolden Ġtraining Ġtu ples . ĠThird , Ġas Ġshown Ġin ĠFigure Ġ1 Ġ( b ), Ġfor Ġsome ĠKB Q A Ġtasks Ġlike ĠWeb Questions ĠB IB REF 0 Ġ, Ġwe Ġneed Ġto Ġpredict Ġa Ġchain Ġof Ġrelations Ġinstead Ġof Ġa Ġsingle Ġrelation . ĠThis Ġincreases Ġthe Ġnumber Ġof Ġtarget Ġrelation Ġtypes Ġand Ġthe Ġsizes Ġof Ġcandidate Ġrelation Ġpools , Ġfurther Ġincreasing Ġthe Ġdifficulty Ġof ĠKB Ġrelation Ġdetection . ĠO wing Ġto Ġthese Ġreasons , ĠKB Ġrelation Ġdetection Ġis Ġsignificantly Ġmore Ġchallenging Ġcompared Ġto Ġgeneral Ġrelation Ġdetection Ġtasks . </s> This Ġpaper Ġimproves ĠKB Ġrelation Ġdetection Ġto Ġcope Ġwith Ġthe Ġproblems Ġmentioned Ġabove . ĠFirst , Ġin Ġorder Ġto Ġdeal Ġwith Ġthe Ġunseen Ġrelations , Ġwe Ġpropose Ġto Ġbreak Ġthe Ġrelation Ġnames Ġinto Ġword Ġsequences Ġfor Ġquestion - relation Ġmatching . ĠSecond , Ġnoticing Ġthat Ġoriginal Ġrelation Ġnames Ġcan Ġsometimes Ġhelp Ġto Ġmatch Ġlonger Ġquestion Ġcontexts , Ġwe Ġpropose Ġto Ġbuild Ġboth Ġrelation - level Ġand Ġword - level Ġrelation Ġrepresentations . ĠThird , Ġwe Ġuse Ġdeep Ġbid irection al ĠL ST Ms Ġ( Bi L ST Ms ) Ġto Ġlearn Ġdifferent Ġlevels Ġof Ġquestion Ġrepresentations Ġin Ġorder Ġto Ġmatch Ġthe Ġdifferent Ġlevels Ġof Ġrelation Ġinformation . ĠFinally , Ġwe Ġpropose Ġa Ġresidual Ġlearning Ġmethod Ġfor Ġsequence Ġmatching , Ġwhich Ġmakes Ġthe Ġmodel Ġtraining Ġeasier Ġand Ġresults Ġin Ġmore Ġabstract Ġ( d eeper ) Ġquestion Ġrepresentations , Ġthus Ġimproves Ġhierarchical Ġmatching . </s> In Ġorder Ġto Ġassess Ġhow Ġthe Ġproposed Ġimproved Ġrelation Ġdetection Ġcould Ġbenefit Ġthe ĠKB Q A Ġend Ġtask , Ġwe Ġalso Ġpropose Ġa Ġsimple ĠKB Q A Ġimplementation Ġcomposed Ġof Ġtwo - step Ġrelation Ġdetection . ĠGiven Ġan Ġinput Ġquestion Ġand Ġa Ġset Ġof Ġcandidate Ġentities Ġretrieved Ġby Ġan Ġentity Ġlink er Ġbased Ġon Ġthe Ġquestion , Ġour Ġproposed Ġrelation Ġdetection Ġmodel Ġplays Ġa Ġkey Ġrole Ġin Ġthe ĠKB Q A Ġprocess : Ġ( 1 ) ĠRe - ranking Ġthe Ġentity Ġcandidates Ġaccording Ġto Ġwhether Ġthey Ġconnect Ġto Ġhigh Ġconfident Ġrelations Ġdetected Ġfrom Ġthe Ġraw Ġquestion Ġtext Ġby Ġthe Ġrelation Ġdetection Ġmodel . ĠThis Ġstep Ġis Ġimportant Ġto Ġdeal Ġwith Ġthe Ġambig u ities Ġnormally Ġpresent Ġin Ġentity Ġlinking Ġresults . Ġ( 2 ) ĠFinding Ġthe Ġcore Ġrelation Ġ( chains ) Ġfor Ġeach Ġtopic Ġentity Ġselection Ġfrom Ġa Ġmuch Ġsmaller Ġcandidate Ġentity Ġset Ġafter Ġre - ranking . ĠThe Ġabove Ġsteps Ġare Ġfollowed Ġby Ġan Ġoptional Ġconstraint Ġdetection Ġstep , Ġwhen Ġthe Ġquestion Ġcannot Ġbe Ġanswered Ġby Ġsingle Ġrelations Ġ( e . g ., Ġmultiple Ġentities Ġin Ġthe Ġquestion ). ĠFinally Ġthe Ġhighest Ġscored Ġquery Ġfrom Ġthe Ġabove Ġsteps Ġis Ġused Ġto Ġquery Ġthe ĠKB Ġfor Ġanswers . </s> Our Ġmain Ġcontributions Ġinclude : Ġ( i ) ĠAn Ġimproved Ġrelation Ġdetection Ġmodel Ġby Ġhierarchical Ġmatching Ġbetween Ġquestions Ġand Ġrelations Ġwith Ġresidual Ġlearning ; Ġ( ii ) ĠWe Ġdemonstrate Ġthat Ġthe Ġimproved Ġrelation Ġdetector Ġenables Ġour Ġsimple ĠKB Q A Ġsystem Ġto Ġachieve Ġstate - of - the - art Ġresults Ġon Ġboth Ġsingle - relation Ġand Ġmulti - relation ĠKB Q A Ġtasks . </s> Background : ĠDifferent ĠGran ularity Ġin ĠKB ĠRelations </s> Previous Ġresearch ĠB IB REF 4 Ġ, ĠB IB REF 20 Ġform ulates ĠKB Ġrelation Ġdetection Ġas Ġa Ġsequence Ġmatching Ġproblem . ĠHowever , Ġwhile Ġthe Ġquestions Ġare Ġnatural Ġword Ġsequences , Ġhow Ġto Ġrepresent Ġrelations Ġas Ġsequences Ġremains Ġa Ġchallenging Ġproblem . ĠHere Ġwe Ġgive Ġan Ġoverview Ġof Ġtwo Ġtypes Ġof Ġrelation Ġsequence Ġrepresentations Ġcommonly Ġused Ġin Ġprevious Ġwork . </s> ( 1 ) ĠRel ation ĠName Ġas Ġa ĠSingle ĠToken Ġ( relation - level ). ĠIn Ġthis Ġcase , Ġeach Ġrelation Ġname Ġis Ġtreated Ġas Ġa Ġunique Ġtoken . ĠThe Ġproblem Ġwith Ġthis Ġapproach Ġis Ġthat Ġit Ġsuffers Ġfrom Ġthe Ġlow Ġrelation Ġcoverage Ġdue Ġto Ġlimited Ġamount Ġof Ġtraining Ġdata , Ġthus Ġcannot Ġgeneral ize Ġwell Ġto Ġlarge Ġnumber Ġof Ġopen - domain Ġrelations . ĠFor Ġexample , Ġin ĠFigure Ġ1 Ġ, Ġwhen Ġtreating Ġrelation Ġnames Ġas Ġsingle Ġtokens , Ġit Ġwill Ġbe Ġdifficult Ġto Ġmatch Ġthe Ġquestions Ġto Ġrelation Ġnames ĠâĢ ľ ep isodes _ written âĢ Ŀ Ġand ĠâĢ ľ star ring _ ro les âĢ Ŀ Ġif Ġthese Ġnames Ġdo Ġnot Ġappear Ġin Ġtraining Ġdata ĠâĢĵ Ġtheir Ġrelation Ġembed d ings Ġ$\ math bf Ġ{ h } ^ r $ Ġs Ġwill Ġbe Ġrandom Ġvectors Ġthus Ġare Ġnot Ġcomparable Ġto Ġquestion Ġembed d ings Ġ$\ math bf Ġ{ h } ^ q $ Ġs . </s> ( 2 ) ĠRel ation Ġas ĠWord ĠSequence Ġ( word - level ). ĠIn Ġthis Ġcase , Ġthe Ġrelation Ġis Ġtreated Ġas Ġa Ġsequence Ġof Ġwords Ġfrom Ġthe Ġtoken ized Ġrelation Ġname . ĠIt Ġhas Ġbetter Ġgeneral ization , Ġbut Ġsuffers Ġfrom Ġthe Ġlack Ġof Ġglobal Ġinformation Ġfrom Ġthe Ġoriginal Ġrelation Ġnames . ĠFor Ġexample Ġin ĠFigure Ġ1 Ġ( b ), Ġwhen Ġdoing Ġonly Ġword - level Ġmatching , Ġit Ġis Ġdifficult Ġto Ġrank Ġthe Ġtarget Ġrelation ĠâĢ ľ star ring _ ro les âĢ Ŀ Ġhigher Ġcompared Ġto Ġthe Ġincorrect Ġrelation ĠâĢ ľ plays _ produced âĢ Ŀ . ĠThis Ġis Ġbecause Ġthe Ġincorrect Ġrelation Ġcontains Ġword ĠâĢ ľ plays âĢ Ŀ , Ġwhich Ġis Ġmore Ġsimilar Ġto Ġthe Ġquestion Ġ( containing Ġword ĠâĢ ľ play âĢ Ŀ ) Ġin Ġthe Ġembed ding Ġspace . ĠOn Ġthe Ġother Ġhand , Ġif Ġthe Ġtarget Ġrelation Ġco - occ urs Ġwith Ġquestions Ġrelated Ġto ĠâĢ ľ tv Ġappearance âĢ Ŀ Ġin Ġtraining , Ġby Ġtreating Ġthe Ġwhole Ġrelation Ġas Ġa Ġtoken Ġ( i . e . Ġrelation Ġid ), Ġwe Ġcould Ġbetter Ġlearn Ġthe Ġcorrespondence Ġbetween Ġthis Ġtoken Ġand Ġphrases Ġlike ĠâĢ ľ tv Ġshow âĢ Ŀ Ġand ĠâĢ ľ play Ġon âĢ Ŀ . </s> The Ġtwo Ġtypes Ġof Ġrelation Ġrepresentation Ġcontain Ġdifferent Ġlevels Ġof Ġabstraction . ĠAs Ġshown Ġin ĠTable Ġ1 Ġ, Ġthe Ġword - level Ġfocuses Ġmore Ġon Ġlocal Ġinformation Ġ( words Ġand Ġshort Ġphrases ), Ġand Ġthe Ġrelation - level Ġfocus Ġmore Ġon Ġglobal Ġinformation Ġ( long Ġphrases Ġand Ġskip - gram s ) Ġbut Ġsuffer Ġfrom Ġdata Ġsp arsity . ĠSince Ġboth Ġthese Ġlevels Ġof Ġgran ularity Ġhave Ġtheir Ġown Ġpros Ġand Ġcons , Ġwe Ġpropose Ġa Ġhierarchical Ġmatching Ġapproach Ġfor ĠKB Ġrelation Ġdetection : Ġfor Ġa Ġcandidate Ġrelation , Ġour Ġapproach Ġmatches Ġthe Ġinput Ġquestion Ġto Ġboth Ġword - level Ġand Ġrelation - level Ġrepresentations Ġto Ġget Ġthe Ġfinal Ġranking Ġscore . ĠSection Ġ" Improved ĠKB ĠRel ation ĠDetection " Ġgives Ġthe Ġdetails Ġof Ġour Ġproposed Ġapproach . </s> Improved ĠKB ĠRel ation ĠDetection </s> This Ġsection Ġdescribes Ġour Ġhierarchical Ġsequence Ġmatching Ġwith Ġresidual Ġlearning Ġapproach Ġfor Ġrelation Ġdetection . ĠIn Ġorder Ġto Ġmatch Ġthe Ġquestion Ġto Ġdifferent Ġaspects Ġof Ġa Ġrelation Ġ( with Ġdifferent Ġabstraction Ġlevels ), Ġwe Ġdeal Ġwith Ġthree Ġproblems Ġas Ġfollows Ġon Ġlearning Ġquestion / relation Ġrepresentations . </s> Rel ation ĠRepresent ations Ġfrom ĠDifferent ĠGran ularity </s> We Ġprovide Ġour Ġmodel Ġwith Ġboth Ġtypes Ġof Ġrelation Ġrepresentation : Ġword - level Ġand Ġrelation - level . ĠTherefore , Ġthe Ġinput Ġrelation Ġbecomes Ġ$\ math bf Ġ{ r } = \ l brace Ġr ^{ word } _ 1 , \ cd ots Ġ, r ^{ word } _{ M _ 1 }\ r brace Ġ Ġ\ cup Ġ\ l brace Ġr ^{ rel } _ 1 , \ cd ots Ġ, r ^{ rel } _{ M _ 2 }\ r brace Ġ$ Ġ, Ġwhere Ġthe Ġfirst Ġ$ M _ 1 $ Ġtokens Ġare Ġwords Ġ( e . g . Ġ{ episode , Ġwritten } ), Ġand Ġthe Ġlast Ġ$ M _ 2 $ Ġtokens Ġare Ġrelation Ġnames , Ġe . g ., Ġ{ episode _ written } Ġor Ġ{ star ring _ ro les , Ġseries } Ġ( when Ġthe Ġtarget Ġis Ġa Ġchain Ġlike Ġin ĠFigure Ġ1 Ġ( b )). ĠWe Ġtransform Ġeach Ġtoken Ġabove Ġto Ġits Ġword Ġembed ding Ġthen Ġuse Ġtwo ĠBi L ST Ms Ġ( with Ġshared Ġparameters ) Ġto Ġget Ġtheir Ġhidden Ġrepresentations Ġ$ [ \ math bf Ġ{ B } ^{ word } _{ 1 : M _ 1 } :\ math bf Ġ{ B } ^{ rel } _{ 1 : M _ 2 } ] $ Ġ( each Ġrow Ġvector Ġ$\ math bf Ġ{\ beta Ġ} _ i $ Ġis Ġthe Ġconc aten ation Ġbetween Ġforward / back ward Ġrepresentations Ġat Ġ$ i $ Ġ). ĠWe Ġinitialize Ġthe Ġrelation Ġsequence ĠL ST Ms Ġwith Ġthe Ġfinal Ġstate Ġrepresentations Ġof Ġthe Ġword Ġsequence , Ġas Ġa Ġback - off Ġfor Ġunseen Ġrelations . ĠWe Ġapply Ġone Ġmax - pool ing Ġon Ġthese Ġtwo Ġsets Ġof Ġvectors Ġand Ġget Ġthe Ġfinal Ġrelation Ġrepresentation Ġ$\ math bf Ġ{ h } ^ r $ Ġ. </s> Different ĠAbstract ions Ġof ĠQuestions ĠRepresent ations </s> From ĠTable Ġ1 Ġ, Ġwe Ġcan Ġsee Ġthat Ġdifferent Ġparts Ġof Ġa Ġrelation Ġcould Ġmatch Ġdifferent Ġcontexts Ġof Ġquestion Ġtexts . ĠUsually Ġrelation Ġnames Ġcould Ġmatch Ġlonger Ġphrases Ġin Ġthe Ġquestion Ġand Ġrelation Ġwords Ġcould Ġmatch Ġshort Ġphrases . ĠYet Ġdifferent Ġwords Ġmight Ġmatch Ġphrases Ġof Ġdifferent Ġlengths . </s> As Ġa Ġresult , Ġwe Ġhope Ġthe Ġquestion Ġrepresentations Ġcould Ġalso Ġcomprise Ġvectors Ġthat Ġsummarize Ġvarious Ġlengths Ġof Ġphrase Ġinformation Ġ( different Ġlevels Ġof Ġabstraction ), Ġin Ġorder Ġto Ġmatch Ġrelation Ġrepresentations Ġof Ġdifferent Ġgran ularity . ĠWe Ġdeal Ġwith Ġthis Ġproblem Ġby Ġapplying Ġdeep ĠBi L ST Ms Ġon Ġquestions . ĠThe Ġfirst - layer Ġof ĠBi L ST M Ġworks Ġon Ġthe Ġword Ġembed d ings Ġof Ġquestion Ġwords Ġ$\ math bf Ġ{ q } = \ l brace Ġq _ 1 , \ cd ots Ġ, q _ N \ r brace Ġ$ Ġand Ġgets Ġhidden Ġrepresentations Ġ$\ math bf Ġ{\ Gam ma Ġ} ^{ ( 1 )} _{ 1 : N } =[ \ math bf Ġ{\ gam ma Ġ} ^{ ( 1 )} _ 1 ; \ cd ots Ġ; \ math bf Ġ{\ gam ma Ġ} ^{ ( 1 )} _ N ] $ Ġ. ĠThe Ġsecond - layer ĠBi L ST M Ġworks Ġon Ġ$\ math bf Ġ{\ Gam ma Ġ} ^{ ( 1 )} _{ 1 : N } $ Ġto Ġget Ġthe Ġsecond Ġset Ġof Ġhidden Ġrepresentations Ġ$\ math bf Ġ{\ Gam ma Ġ} ^{ ( 2 )} _{ 1 : N } $ Ġ. ĠSince Ġthe Ġsecond ĠBi L ST M Ġstarts Ġwith Ġthe Ġhidden Ġvectors Ġfrom Ġthe Ġfirst Ġlayer , Ġintuitive ly Ġit Ġcould Ġlearn Ġmore Ġgeneral Ġand Ġabstract Ġinformation Ġcompared Ġto Ġthe Ġfirst Ġlayer . </s> Note Ġthat Ġthe Ġfirst ( second )- layer Ġof Ġquestion Ġrepresentations Ġdoes Ġnot Ġnecessarily Ġcorrespond Ġto Ġthe Ġword ( relation )- level Ġrelation Ġrepresentations , Ġinstead Ġeither Ġlayer Ġof Ġquestion Ġrepresentations Ġcould Ġpotentially Ġmatch Ġto Ġeither Ġlevel Ġof Ġrelation Ġrepresentations . ĠThis Ġraises Ġthe Ġdifficulty Ġof Ġmatching Ġbetween Ġdifferent Ġlevels Ġof Ġrelation / question Ġrepresentations ; Ġthe Ġfollowing Ġsection Ġgives Ġour Ġproposal Ġto Ġdeal Ġwith Ġsuch Ġproblem . </s> H ier arch ical ĠMatch ing Ġbetween ĠRel ation Ġand ĠQuestion </s> Now Ġwe Ġhave Ġquestion Ġcontexts Ġof Ġdifferent Ġlengths Ġencoded Ġin Ġ$\ math bf Ġ{\ Gam ma Ġ} ^{ ( 1 )} _{ 1 : N } $ Ġand Ġ$\ math bf Ġ{\ Gam ma Ġ} ^{ ( 2 )} _{ 1 : N } $ Ġ. ĠUnlike Ġthe Ġstandard Ġusage Ġof Ġdeep ĠBi L ST Ms Ġthat Ġemploys Ġthe Ġrepresentations Ġin Ġthe Ġfinal Ġlayer Ġfor Ġprediction , Ġhere Ġwe Ġexpect Ġthat Ġtwo Ġlayers Ġof Ġquestion Ġrepresentations Ġcan Ġbe Ġcomplementary Ġto Ġeach Ġother Ġand Ġboth Ġshould Ġbe Ġcompared Ġto Ġthe Ġrelation Ġrepresentation Ġspace Ġ( H ier arch ical ĠMatch ing ). ĠThis Ġis Ġimportant Ġfor Ġour Ġtask Ġsince Ġeach Ġrelation Ġtoken Ġcan Ġcorrespond Ġto Ġphrases Ġof Ġdifferent Ġlengths , Ġmainly Ġbecause Ġof Ġsynt actic Ġvariations . ĠFor Ġexample Ġin ĠTable Ġ1 Ġ, Ġthe Ġrelation Ġword Ġwritten Ġcould Ġbe Ġmatched Ġto Ġeither Ġthe Ġsame Ġsingle Ġword Ġin Ġthe Ġquestion Ġor Ġa Ġmuch Ġlonger Ġphrase Ġbe Ġthe Ġwriter Ġof . </s> We Ġcould Ġperform Ġthe Ġabove Ġhierarchical Ġmatching Ġby Ġcomputing Ġthe Ġsimilarity Ġbetween Ġeach Ġlayer Ġof Ġ$\ math bf Ġ{\ Gam ma Ġ} $ Ġand Ġ$\ math bf Ġ{ h } ^ r $ Ġseparately Ġand Ġdoing Ġthe Ġ( weight ed ) Ġsum Ġbetween Ġthe Ġtwo Ġscores . ĠHowever Ġthis Ġdoes Ġnot Ġgive Ġsignificant Ġimprovement Ġ( see ĠTable Ġ2 Ġ). ĠOur Ġanalysis Ġin ĠSection Ġ" Rel ation ĠDetection ĠResults " Ġshows Ġthat Ġthis Ġnaive Ġmethod Ġsuffers Ġfrom Ġthe Ġtraining Ġdifficulty , Ġevidenced Ġby Ġthat Ġthe Ġconver ged Ġtraining Ġloss Ġof Ġthis Ġmodel Ġis Ġmuch Ġhigher Ġthan Ġthat Ġof Ġa Ġsingle - layer Ġbaseline Ġmodel . ĠThis Ġis Ġmainly Ġbecause Ġ( 1 ) ĠDeep ĠBi L ST Ms Ġdo Ġnot Ġguarantee Ġthat Ġthe Ġtwo - levels Ġof Ġquestion Ġhidden Ġrepresentations Ġare Ġcomparable , Ġthe Ġtraining Ġusually Ġfalls Ġto Ġlocal Ġoptim a Ġwhere Ġone Ġlayer Ġhas Ġgood Ġmatching Ġscores Ġand Ġthe Ġother Ġalways Ġhas Ġweight Ġclose Ġto Ġ0 . Ġ( 2 ) ĠThe Ġtraining Ġof Ġdeeper Ġarchitectures Ġitself Ġis Ġmore Ġdifficult . </s> To Ġovercome Ġthe Ġabove Ġdifficulties , Ġwe Ġadopt Ġthe Ġidea Ġfrom ĠRes id ual ĠNetworks ĠB IB REF 23 Ġfor Ġhierarchical Ġmatching Ġby Ġadding Ġshortcut Ġconnections Ġbetween Ġtwo ĠBi L ST M Ġlayers . ĠWe Ġproposed Ġtwo Ġways Ġof Ġsuch ĠHier arch ical ĠRes id ual ĠMatch ing : Ġ( 1 ) ĠConnect ing Ġeach Ġ$\ math bf Ġ{\ gam ma Ġ} ^{ ( 1 )} _ i $ Ġand Ġ$\ math bf Ġ{\ gam ma Ġ} ^{ ( 2 )} _ i $ Ġ, Ġresulting Ġin Ġa Ġ$\ math bf Ġ{\ gam ma Ġ} ^{ ^ {\ prime Ġ}} _ i = \ math bf Ġ{\ gam ma Ġ} ^{ ( 1 )} _ i Ġ+ Ġ\ math bf Ġ{\ gam ma Ġ} ^{ ( 2 )} _ i $ Ġfor Ġeach Ġposition Ġ$ i $ Ġ. ĠThen Ġthe Ġfinal Ġquestion Ġrepresentation Ġ$\ math bf Ġ{ h } ^ q $ Ġbecomes Ġa Ġmax - pool ing Ġover Ġall Ġ$\ math bf Ġ{\ gam ma Ġ} ^{ ^ {\ prime Ġ}} _ i $ Ġs , Ġ1 Ġ$\ le Ġ$ Ġi Ġ$\ le Ġ$ Ġ$ N $ Ġ. Ġ( 2 ) ĠApp lying Ġmax - pool ing Ġon Ġ$\ math bf Ġ{\ Gam ma Ġ} ^{ ( 1 )} _{ 1 : N } $ Ġand Ġ$\ math bf Ġ{\ gam ma Ġ} ^{ ( 2 )} _ i $ 0 Ġto Ġget Ġ$\ math bf Ġ{\ gam ma Ġ} ^{ ( 2 )} _ i $ 1 Ġand Ġ$\ math bf Ġ{\ gam ma Ġ} ^{ ( 2 )} _ i $ 2 Ġ, Ġrespectively , Ġthen Ġsetting Ġ$\ math bf Ġ{\ gam ma Ġ} ^{ ( 2 )} _ i $ 3 Ġ. ĠFinally Ġwe Ġcompute Ġthe Ġmatching Ġscore Ġof Ġ$\ math bf Ġ{\ gam ma Ġ} ^{ ( 2 )} _ i $ 4 Ġgiven Ġ$\ math bf Ġ{\ gam ma Ġ} ^{ ( 2 )} _ i $ 5 Ġas Ġ$\ math bf Ġ{\ gam ma Ġ} ^{ ( 2 )} _ i $ 6 Ġ. </s> Int uitive ly , Ġthe Ġproposed Ġmethod Ġshould Ġbenefit Ġfrom Ġhierarchical Ġtraining Ġsince Ġthe Ġsecond Ġlayer Ġis Ġfitting Ġthe Ġresidues Ġfrom Ġthe Ġfirst Ġlayer Ġof Ġmatching , Ġso Ġthe Ġtwo Ġlayers Ġof Ġrepresentations Ġare Ġmore Ġlikely Ġto Ġbe Ġcomplementary Ġto Ġeach Ġother . ĠThis Ġalso Ġensures Ġthe Ġvector Ġspaces Ġof Ġtwo Ġlayers Ġare Ġcomparable Ġand Ġmakes Ġthe Ġsecond - layer Ġtraining Ġeasier . </s> During Ġtraining Ġwe Ġadopt Ġa Ġranking Ġloss Ġto Ġmaximizing Ġthe Ġmargin Ġbetween Ġthe Ġgold Ġrelation Ġ$\ math bf Ġ{ r } ^ + $ Ġand Ġother Ġrelations Ġ$\ math bf Ġ{ r } ^ -$ Ġin Ġthe Ġcandidate Ġpool Ġ$ R $ Ġ. </s> $$ l _{ \ math rm Ġ{ rel }} Ġ= Ġ\ max Ġ\ l brace Ġ0 , Ġ\ gam ma Ġ- Ġs _{ \ math rm Ġ{ rel }} (\ math bf Ġ{ r } ^ + ; Ġ\ math bf Ġ{ q }) Ġ+ Ġs _{ \ math rm Ġ{ rel }} (\ math bf Ġ{ r } ^ - ; Ġ\ math bf Ġ{ q } )\ r brace Ġ Ġ\ non umber Ġ$$ Ġ Ġ Ġ( E q . Ġ12 ) </s> where Ġ$\ gam ma Ġ$ Ġis Ġa Ġconstant Ġparameter . ĠFig Ġ2 Ġsummarizes Ġthe Ġabove ĠHier arch ical ĠRes id ual ĠBi L ST M Ġ( HR - Bi L ST M ) Ġmodel . </s> Another Ġway Ġof Ġhierarchical Ġmatching Ġconsists Ġin Ġrelying Ġon Ġattention Ġmechanism , Ġe . g . ĠB IB REF 24 Ġ, Ġto Ġfind Ġthe Ġcorrespondence Ġbetween Ġdifferent Ġlevels Ġof Ġrepresentations . ĠThis Ġperforms Ġbelow Ġthe ĠHR - Bi L ST M Ġ( see ĠTable Ġ2 Ġ). </s> KB Q A ĠEnhanced Ġby ĠRel ation ĠDetection </s> This Ġsection Ġdescribes Ġour ĠKB Q A Ġpipeline Ġsystem . ĠWe Ġmake Ġminimal Ġefforts Ġbeyond Ġthe Ġtraining Ġof Ġthe Ġrelation Ġdetection Ġmodel , Ġmaking Ġthe Ġwhole Ġsystem Ġeasy Ġto Ġbuild . </s> Following Ġprevious Ġwork ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, Ġour ĠKB Q A Ġsystem Ġtakes Ġan Ġexisting Ġentity Ġlink er Ġto Ġproduce Ġthe Ġtop - Ġ$ K $ Ġlinked Ġentities , Ġ$ EL _ K ( q ) $ Ġ, Ġfor Ġa Ġquestion Ġ$ q $ Ġ( âĢ ľ initial Ġentity Ġlinking âĢ Ŀ ). ĠThen Ġwe Ġgenerate Ġthe ĠKB Ġqueries Ġfor Ġ$ q $ Ġfollowing Ġthe Ġfour Ġsteps Ġillustrated Ġin ĠAl gorithm Ġ" KB Q A ĠEnhanced Ġby ĠRel ation ĠDetection " Ġ. </s> [ ht bp ] ĠInput Input ĠOutput Output ĠTop Ġquery Ġtuple Ġ$( \ hat { e }, \ hat { r }, Ġ\ l brace Ġ( c , Ġr _ c )\ r brace Ġ) $ ĠEntity ĠRe - R anking Ġ( first - step Ġrelation Ġdetection ): ĠUse Ġthe Ġraw Ġquestion Ġtext Ġas Ġinput Ġfor Ġa Ġrelation Ġdetector Ġto Ġscore Ġall Ġrelations Ġin Ġthe ĠKB Ġthat Ġare Ġassociated Ġto Ġthe Ġentities Ġin Ġ$ EL _ K ( q ) $ Ġ; Ġuse Ġthe Ġrelation Ġscores Ġto Ġre - rank Ġ$ EL _ K ( q ) $ Ġand Ġgenerate Ġa Ġshorter Ġlist Ġ$ EL ^ {\ prime Ġ} _{ K ^ {\ prime Ġ}} ( q ) $ Ġcontaining Ġthe Ġtop - Ġ$ K ^ {\ prime Ġ} $ Ġentity Ġcandidates Ġ( Section Ġ" Entity ĠRe - R anking " Ġ) ĠRel ation ĠDetection : ĠDetect Ġrelation ( s ) Ġusing Ġthe Ġreform atted Ġquestion Ġtext Ġin Ġwhich Ġthe Ġtopic Ġentity Ġis Ġreplaced Ġby Ġa Ġspecial Ġtoken Ġ$ < $ Ġe Ġ$ > $ Ġ( Section Ġ" Rel ation ĠDetection " Ġ) ĠQuery ĠGeneration : ĠCombine Ġthe Ġscores Ġfrom Ġstep Ġ1 Ġand Ġ2 , Ġand Ġselect Ġthe Ġtop Ġpair Ġ$( \ hat { e }, \ hat { r }) $ Ġ( Section Ġ" Query ĠGeneration " Ġ) ĠCon str aint ĠDetection Ġ( optional ): ĠComp ute Ġsimilarity Ġbetween Ġ$ q $ Ġand Ġany Ġneighbor Ġentity Ġ$ c $ Ġof Ġthe Ġentities Ġalong Ġ$ EL _ K ( q ) $ 0 Ġ( connect ing Ġby Ġa Ġrelation Ġ$ EL _ K ( q ) $ 1 Ġ) Ġ, Ġadd Ġthe Ġhigh Ġscoring Ġ$ EL _ K ( q ) $ 2 Ġand Ġ$ EL _ K ( q ) $ 3 Ġto Ġthe Ġquery Ġ( Section Ġ" Con str aint ĠDetection " Ġ). ĠKB Q A Ġwith Ġtwo - step Ġrelation Ġdetection </s> Compared Ġto Ġprevious Ġapproaches , Ġthe Ġmain Ġdifference Ġis Ġthat Ġwe Ġhave Ġan Ġadditional Ġentity Ġre - ranking Ġstep Ġafter Ġthe Ġinitial Ġentity Ġlinking . ĠWe Ġhave Ġthis Ġstep Ġbecause Ġwe Ġhave Ġobserved Ġthat Ġentity Ġlinking Ġsometimes Ġbecomes Ġa Ġbottleneck Ġin ĠKB Q A Ġsystems . ĠFor Ġexample , Ġon ĠSimple Questions Ġthe Ġbest Ġreported Ġlink er Ġcould Ġonly Ġget Ġ72 . 7 % Ġtop - 1 Ġaccuracy Ġon Ġidentifying Ġtopic Ġentities . ĠThis Ġis Ġusually Ġdue Ġto Ġthe Ġambig u ities Ġof Ġentity Ġnames , Ġe . g . Ġin ĠFig Ġ1 Ġ( a ), Ġthere Ġare ĠTV Ġwriter Ġand Ġbaseball Ġplayer ĠâĢ ľ Mike ĠKelley âĢ Ŀ , Ġwhich Ġis Ġimpossible Ġto Ġdistinguish Ġwith Ġonly Ġentity Ġname Ġmatching . </s> Having Ġobserved Ġthat Ġdifferent Ġentity Ġcandidates Ġusually Ġconnect Ġto Ġdifferent Ġrelations , Ġhere Ġwe Ġpropose Ġto Ġhelp Ġentity Ġdis amb ig uation Ġin Ġthe Ġinitial Ġentity Ġlinking Ġwith Ġrelations Ġdetected Ġin Ġquestions . </s> S ect ions Ġ" Entity ĠRe - R anking " Ġand Ġ" Rel ation ĠDetection " Ġelaborate Ġhow Ġour Ġrelation Ġdetection Ġhelp Ġto Ġre - rank Ġentities Ġin Ġthe Ġinitial Ġentity Ġlinking , Ġand Ġthen Ġthose Ġre - ranked Ġentities Ġenable Ġmore Ġaccurate Ġrelation Ġdetection . ĠThe ĠKB Q A Ġend Ġtask , Ġas Ġa Ġresult , Ġbenefits Ġfrom Ġthis Ġprocess . </s> Entity ĠRe - R anking </s> In Ġthis Ġstep , Ġwe Ġuse Ġthe Ġraw Ġquestion Ġtext Ġas Ġinput Ġfor Ġa Ġrelation Ġdetector Ġto Ġscore Ġall Ġrelations Ġin Ġthe ĠKB Ġwith Ġconnections Ġto Ġat Ġleast Ġone Ġof Ġthe Ġentity Ġcandidates Ġin Ġ$ EL _ K ( q ) $ Ġ. ĠWe Ġcall Ġthis Ġstep Ġrelation Ġdetection Ġon Ġentity Ġset Ġsince Ġit Ġdoes Ġnot Ġwork Ġon Ġa Ġsingle Ġtopic Ġentity Ġas Ġthe Ġusual Ġsettings . ĠWe Ġuse Ġthe ĠHR - Bi L ST M Ġas Ġdescribed Ġin ĠSec . Ġ" Improved ĠKB ĠRel ation ĠDetection " Ġ. ĠFor Ġeach Ġquestion Ġ$ q $ Ġ, Ġafter Ġgenerating Ġa Ġscore Ġ$ s _{ rel } ( r ; q ) $ Ġfor Ġeach Ġrelation Ġusing ĠHR - Bi L ST M , Ġwe Ġuse Ġthe Ġtop Ġ$ l $ Ġbest Ġscoring Ġrelations Ġ( Ġ$ R ^{ l } _ q $ Ġ) Ġto Ġre - rank Ġthe Ġoriginal Ġentity Ġcandidates . ĠCon crete ly , Ġfor Ġeach Ġentity Ġ$ e $ Ġand Ġits Ġassociated Ġrelations Ġ$ R _ e $ Ġ, Ġgiven Ġthe Ġoriginal Ġentity Ġlink er Ġscore Ġ$ s _{ link er } $ Ġ, Ġand Ġthe Ġscore Ġof Ġthe Ġmost Ġconfident Ġrelation Ġ$ r \ in ĠR _ q ^{ l } Ġ\ cap ĠR _ e $ Ġ, Ġwe Ġsum Ġthese Ġtwo Ġscores Ġto Ġre - rank Ġthe Ġentities : </s> $$ s _{ \ math rm Ġ{ rer ank }} ( e ; q ) Ġ= & Ġ\ alpha Ġ\ cd ot Ġs _{ \ math rm Ġ{ link er }} ( e ; q ) Ġ\ non umber Ġ\\ Ġ+ Ġ& Ġ( 1 - \ alpha Ġ) Ġ\ cd ot Ġ\ max Ġ_ { r Ġ\ in ĠR _ q ^{ l } Ġ\ cap ĠR _ e } Ġs _{ \ math rm Ġ{ rel }} ( r ; q ). \ non umber Ġ$$ Ġ Ġ Ġ( E q . Ġ15 ) </s> Finally , Ġwe Ġselect Ġtop Ġ$ K ^ {\ prime Ġ} $ Ġ$ < $ Ġ$ K $ Ġentities Ġaccording Ġto Ġscore Ġ$ s _{ rer ank } $ Ġto Ġform Ġthe Ġre - ranked Ġlist Ġ$ EL _{ K ^ {\ prime Ġ}} ^{ ^ {\ prime Ġ}} ( q ) $ Ġ. </s> We Ġuse Ġthe Ġsame Ġexample Ġin ĠFig Ġ1 Ġ( a ) Ġto Ġillustrate Ġthe Ġidea . ĠGiven Ġthe Ġinput Ġquestion Ġin Ġthe Ġexample , Ġa Ġrelation Ġdetector Ġis Ġvery Ġlikely Ġto Ġassign Ġhigh Ġscores Ġto Ġrelations Ġsuch Ġas ĠâĢ ľ ep isodes _ written âĢ Ŀ , ĠâĢ ľ author _ of âĢ Ŀ Ġand ĠâĢ ľ prof ession âĢ Ŀ . ĠThen , Ġaccording Ġto Ġthe Ġconnections Ġof Ġentity Ġcandidates Ġin ĠKB , Ġwe Ġfind Ġthat Ġthe ĠTV Ġwriter ĠâĢ ľ Mike ĠKelley âĢ Ŀ Ġwill Ġbe Ġscored Ġhigher Ġthan Ġthe Ġbaseball Ġplayer ĠâĢ ľ Mike ĠKelley âĢ Ŀ , Ġbecause Ġthe Ġformer Ġhas Ġthe Ġrelations ĠâĢ ľ ep isodes _ written âĢ Ŀ Ġand ĠâĢ ľ prof ession âĢ Ŀ . ĠThis Ġmethod Ġcan Ġbe Ġviewed Ġas Ġexploiting Ġentity - relation Ġcoll ocation Ġfor Ġentity Ġlinking . </s> Rel ation ĠDetection </s> In Ġthis Ġstep , Ġfor Ġeach Ġcandidate Ġentity Ġ$ e Ġ\ in ĠEL _ K ^ {\ prime Ġ} ( q ) $ Ġ, Ġwe Ġuse Ġthe Ġquestion Ġtext Ġas Ġthe Ġinput Ġto Ġa Ġrelation Ġdetector Ġto Ġscore Ġall Ġthe Ġrelations Ġ$ r Ġ\ in ĠR _ e $ Ġthat Ġare Ġassociated Ġto Ġthe Ġentity Ġ$ e $ Ġin Ġthe ĠKB . ĠBecause Ġwe Ġhave Ġa Ġsingle Ġtopic Ġentity Ġinput Ġin Ġthis Ġstep , Ġwe Ġdo Ġthe Ġfollowing Ġquestion Ġreform at ting : Ġwe Ġreplace Ġthe Ġthe Ġcandidate Ġ$ e $ Ġ' s Ġentity Ġmention Ġin Ġ$ q $ Ġwith Ġa Ġtoken ĠâĢ ľ Ġ$ < $ Ġe Ġ$ > $ ĠâĢ Ŀ . ĠThis Ġhelps Ġthe Ġmodel Ġbetter Ġdistinguish Ġthe Ġrelative Ġposition Ġof Ġeach Ġword Ġcompared Ġto Ġthe Ġentity . ĠWe Ġuse Ġthe ĠHR - Bi L ST M Ġmodel Ġto Ġpredict Ġthe Ġscore Ġof Ġeach Ġrelation Ġ$ r Ġ\ in ĠR _ e $ Ġ: Ġ$ s _{ rel } Ġ( r ; e , q ) $ Ġ. </s> Query ĠGeneration </s> Finally , Ġthe Ġsystem Ġoutputs Ġthe Ġ$ < $ Ġentity , Ġrelation Ġ( or Ġcore - chain ) Ġ$ > $ Ġpair Ġ$( \ hat { e }, Ġ\ hat { r }) $ Ġaccording Ġto : </s> $$ s (\ hat { e }, Ġ\ hat { r }; Ġq ) Ġ= & Ġ\ max Ġ_ { e Ġ\ in ĠEL _{ K ^ {\ prime Ġ}} ^{ ^ {\ prime Ġ}} ( q ), Ġr Ġ\ in ĠR _ e } Ġ\ left ( Ġ\ beta Ġ\ cd ot Ġs _{ \ math rm Ġ{ rer ank }} ( e ; q ) Ġ\ right . Ġ\ non umber Ġ\\ Ġ& \ left . + Ġ( 1 - \ beta Ġ) Ġ\ cd ot Ġs _{ \ math rm Ġ{ rel }} Ġ( r ; e , q ) Ġ\ right ), Ġ\ non umber Ġ$$ Ġ Ġ Ġ( E q . Ġ19 ) </s> where Ġ$\ beta Ġ$ Ġis Ġa Ġhyper param eter Ġto Ġbe Ġtuned . </s> Con str aint ĠDetection </s> Similar Ġto ĠB IB REF 4 Ġ, Ġwe Ġadopt Ġan Ġadditional Ġconstraint Ġdetection Ġstep Ġbased Ġon Ġtext Ġmatching . ĠOur Ġmethod Ġcan Ġbe Ġviewed Ġas Ġentity - l inking Ġon Ġa ĠKB Ġsub - graph . ĠIt Ġcontains Ġtwo Ġsteps : Ġ( 1 ) ĠSub - graph Ġgeneration : Ġgiven Ġthe Ġtop Ġscored Ġquery Ġgenerated Ġby Ġthe Ġprevious Ġ3 Ġsteps , Ġfor Ġeach Ġnode Ġ$ v $ Ġ( answer Ġnode Ġor Ġthe ĠCV T Ġnode Ġlike Ġin ĠFigure Ġ1 Ġ( b )), Ġwe Ġcollect Ġall Ġthe Ġnodes Ġ$ c $ Ġconnecting Ġto Ġ$ v $ Ġ( with Ġrelation Ġ$ r _ c $ Ġ) Ġwith Ġany Ġrelation , Ġand Ġgenerate Ġa Ġsub - graph Ġassociated Ġto Ġthe Ġoriginal Ġquery . Ġ( 2 ) ĠEntity - l inking Ġon Ġsub - graph Ġnodes : Ġwe Ġcompute Ġa Ġmatching Ġscore Ġbetween Ġeach Ġ$ n $ Ġ- gram Ġin Ġthe Ġinput Ġquestion Ġ( without Ġoverlapping Ġthe Ġtopic Ġentity ) Ġand Ġentity Ġname Ġof Ġ$ c $ Ġ( except Ġfor Ġthe Ġnode Ġin Ġthe Ġoriginal Ġquery ) Ġby Ġtaking Ġinto Ġaccount Ġthe Ġmaximum Ġoverlapping Ġsequence Ġof Ġcharacters Ġbetween Ġthem Ġ( see ĠAppendix ĠA Ġfor Ġdetails Ġand ĠB Ġfor Ġspecial Ġrules Ġdealing Ġwith Ġdate / answer Ġtype Ġconstraints ). ĠIf Ġthe Ġmatching Ġscore Ġis Ġlarger Ġthan Ġa Ġthreshold Ġ$\ the ta Ġ$ Ġ( tun ed Ġon Ġtraining Ġset ), Ġwe Ġwill Ġadd Ġthe Ġconstraint Ġentity Ġ$ c $ Ġ( and Ġ$ r _ c $ Ġ) Ġto Ġthe Ġquery Ġby Ġattaching Ġit Ġto Ġthe Ġcorresponding Ġnode Ġ$ v $ Ġon Ġthe Ġcore - chain . </s> Exper iments </s> Task ĠIntroduction Ġ& ĠSettings </s> We Ġuse Ġthe ĠSimple Questions ĠB IB REF 2 Ġand ĠWeb Q SP ĠB IB REF 25 Ġdatasets . ĠEach Ġquestion Ġin Ġthese Ġdatasets Ġis Ġlabeled Ġwith Ġthe Ġgold Ġsemantic Ġparse . ĠHence Ġwe Ġcan Ġdirectly Ġevaluate Ġrelation Ġdetection Ġperformance Ġindependently Ġas Ġwell Ġas Ġevaluate Ġon Ġthe ĠKB Q A Ġend Ġtask . </s> Simple Questions Ġ( S Q ): ĠIt Ġis Ġa Ġsingle - relation ĠKB Q A Ġtask . ĠThe ĠKB Ġwe Ġuse Ġconsists Ġof Ġa ĠFree base Ġsubset Ġwith Ġ2 M Ġentities Ġ( FB 2 M ) ĠB IB REF 2 Ġ, Ġin Ġorder Ġto Ġcompare Ġwith Ġprevious Ġresearch . Ġy in 2016 simple Ġalso Ġevaluated Ġtheir Ġrelation Ġextract or Ġon Ġthis Ġdata Ġset Ġand Ġreleased Ġtheir Ġproposed Ġquestion - relation Ġpairs , Ġso Ġwe Ġrun Ġour Ġrelation Ġdetection Ġmodel Ġon Ġtheir Ġdata Ġset . ĠFor Ġthe ĠKB Q A Ġevaluation , Ġwe Ġalso Ġstart Ġwith Ġtheir Ġentity Ġlinking Ġresults . ĠTherefore , Ġour Ġresults Ġcan Ġbe Ġcompared Ġwith Ġtheir Ġreported Ġresults Ġon Ġboth Ġtasks . </s> Web Q SP Ġ( W Q ): ĠA Ġmulti - relation ĠKB Q A Ġtask . ĠWe Ġuse Ġthe Ġentire ĠFree base ĠKB Ġfor Ġevaluation Ġpurposes . ĠFollowing Ġy ih - E t Al : 2016 : P 16 - 2 , Ġwe Ġuse ĠS - M ART ĠB IB REF 26 Ġentity - l inking Ġoutputs . ĠIn Ġorder Ġto Ġevaluate Ġthe Ġrelation Ġdetection Ġmodels , Ġwe Ġcreate Ġa Ġnew Ġrelation Ġdetection Ġtask Ġfrom Ġthe ĠWeb Q SP Ġdata Ġset . ĠFor Ġeach Ġquestion Ġand Ġits Ġlabeled Ġsemantic Ġparse : Ġ( 1 ) Ġwe Ġfirst Ġselect Ġthe Ġtopic Ġentity Ġfrom Ġthe Ġparse ; Ġand Ġthen Ġ( 2 ) Ġselect Ġall Ġthe Ġrelations Ġand Ġrelation Ġchains Ġ( length Ġ$\ le Ġ$ Ġ2 ) Ġconnected Ġto Ġthe Ġtopic Ġentity , Ġand Ġset Ġthe Ġcore - chain Ġlabeled Ġin Ġthe Ġparse Ġas Ġthe Ġpositive Ġlabel Ġand Ġall Ġthe Ġothers Ġas Ġthe Ġnegative Ġexamples . </s> We Ġtune Ġthe Ġfollowing Ġhyper - param eters Ġon Ġdevelopment Ġsets : Ġ( 1 ) Ġthe Ġsize Ġof Ġhidden Ġstates Ġfor ĠL ST Ms Ġ({ 50 , Ġ100 , Ġ200 , Ġ400 }); Ġ( 2 ) Ġlearning Ġrate Ġ({ 0 . 1 , Ġ0 . 5 , Ġ1 . 0 , Ġ2 . 0 }); Ġ( 3 ) Ġwhether Ġthe Ġshortcut Ġconnections Ġare Ġbetween Ġhidden Ġstates Ġor Ġbetween Ġmax - pool ing Ġresults Ġ( see ĠSection Ġ" H ier arch ical ĠMatch ing Ġbetween ĠRel ation Ġand ĠQuestion " Ġ); Ġand Ġ( 4 ) Ġthe Ġnumber Ġof Ġtraining Ġepoch s . </s> For Ġboth Ġthe Ġrelation Ġdetection Ġexperiments Ġand Ġthe Ġsecond - step Ġrelation Ġdetection Ġin ĠKB Q A , Ġwe Ġhave Ġentity Ġreplacement Ġfirst Ġ( see ĠSection Ġ" Rel ation ĠDetection " Ġand ĠFigure Ġ1 Ġ). ĠAll Ġword Ġvectors Ġare Ġinitialized Ġwith Ġ300 - Ġ$ d $ Ġpret rained Ġword Ġembed d ings ĠB IB REF 27 Ġ. ĠThe Ġembed d ings Ġof Ġrelation Ġnames Ġare Ġrandomly Ġinitialized , Ġsince Ġexisting Ġpre - trained Ġrelation Ġembed d ings Ġ( e . g . ĠTrans E ) Ġusually Ġsupport Ġlimited Ġsets Ġof Ġrelation Ġnames . ĠWe Ġleave Ġthe Ġusage Ġof Ġpre - trained Ġrelation Ġembed d ings Ġto Ġfuture Ġwork . </s> Rel ation ĠDetection ĠResults </s> Table Ġ2 Ġshows Ġthe Ġresults Ġon Ġtwo Ġrelation Ġdetection Ġtasks . ĠThe ĠAM PC NN Ġresult Ġis Ġfrom ĠB IB REF 20 Ġ, Ġwhich Ġyielded Ġstate - of - the - art Ġscores Ġby Ġoutper forming Ġseveral Ġattention - based Ġmethods . ĠWe Ġre - im ple mented Ġthe ĠBi CNN Ġmodel Ġfrom ĠB IB REF 4 Ġ, Ġwhere Ġboth Ġquestions Ġand Ġrelations Ġare Ġrepresented Ġwith Ġthe Ġword Ġhash Ġtrick Ġon Ġcharacter Ġtri - gram s . ĠThe Ġbaseline ĠBi L ST M Ġwith Ġrelation Ġword Ġsequence Ġappears Ġto Ġbe Ġthe Ġbest Ġbaseline Ġon ĠWeb Q SP Ġand Ġis Ġclose Ġto Ġthe Ġprevious Ġbest Ġresult Ġof ĠAM PC NN Ġon ĠSimple Questions . ĠOur Ġproposed ĠHR - Bi L ST M Ġoutper formed Ġthe Ġbest Ġbas elines Ġon Ġboth Ġtasks Ġby Ġmargins Ġof Ġ2 - 3 % Ġ( p Ġ$ < $ Ġ0 . 001 Ġand Ġ0 . 01 Ġcompared Ġto Ġthe Ġbest Ġbaseline ĠBi L ST M Ġw / Ġwords Ġon ĠSQ Ġand ĠW Q Ġrespectively ). </s> Note Ġthat Ġusing Ġonly Ġrelation Ġnames Ġinstead Ġof Ġwords Ġresults Ġin Ġa Ġweaker Ġbaseline ĠBi L ST M Ġmodel . ĠThe Ġmodel Ġyields Ġa Ġsignificant Ġperformance Ġdrop Ġon ĠSimple Questions Ġ( 91 . 2 % Ġto Ġ88 . 9 %). ĠHowever , Ġthe Ġdrop Ġis Ġmuch Ġsmaller Ġon ĠWeb Q SP , Ġand Ġit Ġsuggests Ġthat Ġunseen Ġrelations Ġhave Ġa Ġmuch Ġbigger Ġimpact Ġon ĠSimple Questions . </s> The Ġbottom Ġof ĠTable Ġ2 Ġshows Ġab lation Ġresults Ġof Ġthe Ġproposed ĠHR - Bi L ST M . ĠFirst , Ġhierarchical Ġmatching Ġbetween Ġquestions Ġand Ġboth Ġrelation Ġnames Ġand Ġrelation Ġwords Ġyields Ġimprovement Ġon Ġboth Ġdatasets , Ġespecially Ġfor ĠSimple Questions Ġ( 93 . 3 % Ġvs . Ġ91 . 2 / 88 . 8 %). ĠSecond , Ġresidual Ġlearning Ġhelps Ġhierarchical Ġmatching Ġcompared Ġto Ġweighted - sum Ġand Ġattention - based Ġbas elines Ġ( see ĠSection Ġ" H ier arch ical ĠMatch ing Ġbetween ĠRel ation Ġand ĠQuestion " Ġ). ĠFor Ġthe Ġattention - based Ġbaseline , Ġwe Ġtried Ġthe Ġmodel Ġfrom ĠB IB REF 24 Ġand Ġits Ġone - way Ġvariations , Ġwhere Ġthe Ġone - way Ġmodel Ġgives Ġbetter Ġresults . ĠNote Ġthat Ġresidual Ġlearning Ġsignificantly Ġhelps Ġon ĠWeb Q SP Ġ( 80 . 65 % Ġto Ġ82 . 53 %), Ġwhile Ġit Ġdoes Ġnot Ġhelp Ġas Ġmuch Ġon ĠSimple Questions . ĠOn ĠSimple Questions , Ġeven Ġremoving Ġthe Ġdeep Ġlayers Ġonly Ġcauses Ġa Ġsmall Ġdrop Ġin Ġperformance . ĠWeb Q SP Ġbenefits Ġmore Ġfrom Ġresidual Ġand Ġdeeper Ġarchitecture , Ġpossibly Ġbecause Ġin Ġthis Ġdataset Ġit Ġis Ġmore Ġimportant Ġto Ġhandle Ġlarger Ġscope Ġof Ġcontext Ġmatching . </s> Finally , Ġon ĠWeb Q SP , Ġreplacing ĠBi L ST M Ġwith ĠCNN Ġin Ġour Ġhierarchical Ġmatching Ġframework Ġresults Ġin Ġa Ġlarge Ġperformance Ġdrop . ĠYet Ġon ĠSimple Questions Ġthe Ġgap Ġis Ġmuch Ġsmaller . ĠWe Ġbelieve Ġthis Ġis Ġbecause Ġthe ĠL ST M Ġrelation Ġenc oder Ġcan Ġbetter Ġlearn Ġthe Ġcomposition Ġof Ġchains Ġof Ġrelations Ġin ĠWeb Q SP , Ġas Ġit Ġis Ġbetter Ġat Ġdealing Ġwith Ġlonger Ġdependencies . </s> Next , Ġwe Ġpresent Ġempirical Ġevid ences , Ġwhich Ġshow Ġwhy Ġour ĠHR - Bi L ST M Ġmodel Ġachieves Ġthe Ġbest Ġscores . ĠWe Ġuse ĠWeb Q SP Ġfor Ġthe Ġanalysis Ġpurposes . ĠFirst , Ġwe Ġhave Ġthe Ġhypothesis Ġthat Ġtraining Ġof Ġthe Ġweighted - sum Ġmodel Ġusually Ġfalls Ġto Ġlocal Ġoptim a , Ġsince Ġdeep ĠBi L ST Ms Ġdo Ġnot Ġguarantee Ġthat Ġthe Ġtwo - levels Ġof Ġquestion Ġhidden Ġrepresentations Ġare Ġcomparable . ĠThis Ġis Ġevidenced Ġby Ġthat Ġduring Ġtraining Ġone Ġlayer Ġusually Ġgets Ġa Ġweight Ġclose Ġto Ġ0 Ġthus Ġis Ġignored . ĠFor Ġexample , Ġone Ġrun Ġgives Ġus Ġweights Ġof Ġ- 75 . 39 / 0 . 14 Ġfor Ġthe Ġtwo Ġlayers Ġ( we Ġtake Ġexponential Ġfor Ġthe Ġfinal Ġweighted Ġsum ). ĠIt Ġalso Ġgives Ġmuch Ġlower Ġtraining Ġaccuracy Ġ( 91 . 94 %) Ġcompared Ġto ĠHR - Bi L ST M Ġ( 95 . 67 %), Ġsuffering Ġfrom Ġtraining Ġdifficulty . </s> Second , Ġcompared Ġto Ġour Ġdeep ĠBi L ST M Ġwith Ġshortcut Ġconnections , Ġwe Ġhave Ġthe Ġhypothesis Ġthat Ġfor ĠKB Ġrelation Ġdetection , Ġtraining Ġdeep ĠBi L ST Ms Ġis Ġmore Ġdifficult Ġwithout Ġshortcut Ġconnections . ĠOur Ġexperiments Ġsuggest Ġthat Ġdeeper ĠBi L ST M Ġdoes Ġnot Ġalways Ġresult Ġin Ġlower Ġtraining Ġaccuracy . ĠIn Ġthe Ġexperiments Ġa Ġtwo - layer ĠBi L ST M Ġconver ges Ġto Ġ94 . 99 %, Ġeven Ġlower Ġthan Ġthe Ġ95 . 25 % Ġachieved Ġby Ġa Ġsingle - layer ĠBi L ST M . ĠUnder Ġour Ġsetting Ġthe Ġtwo - layer Ġmodel Ġcaptures Ġthe Ġsingle - layer Ġmodel Ġas Ġa Ġspecial Ġcase Ġ( so Ġit Ġcould Ġpotentially Ġbetter Ġfit Ġthe Ġtraining Ġdata ), Ġthis Ġresult Ġsuggests Ġthat Ġthe Ġdeep ĠBi L ST M Ġwithout Ġshortcut Ġconnections Ġmight Ġsuffers Ġmore Ġfrom Ġtraining Ġdifficulty . </s> Finally , Ġwe Ġhypothes ize Ġthat ĠHR - Bi L ST M Ġis Ġmore Ġthan Ġcombination Ġof Ġtwo ĠBi L ST Ms Ġwith Ġresidual Ġconnections , Ġbecause Ġit Ġencourages Ġthe Ġhierarchical Ġarchitecture Ġto Ġlearn Ġdifferent Ġlevels Ġof Ġabstraction . ĠTo Ġverify Ġthis , Ġwe Ġreplace Ġthe Ġdeep ĠBi L ST M Ġquestion Ġenc oder Ġwith Ġtwo Ġsingle - layer ĠBi L ST Ms Ġ( both Ġon Ġwords ) Ġwith Ġshortcut Ġconnections Ġbetween Ġtheir Ġhidden Ġstates . ĠThis Ġdecreases Ġtest Ġaccuracy Ġto Ġ76 . 11 %. ĠIt Ġgives Ġsimilar Ġtraining Ġaccuracy Ġcompared Ġto ĠHR - Bi L ST M , Ġindicating Ġa Ġmore Ġserious Ġover - fitting Ġproblem . ĠThis Ġproves Ġthat Ġthe Ġresidual Ġand Ġdeep Ġstructures Ġboth Ġcontribute Ġto Ġthe Ġgood Ġperformance Ġof ĠHR - Bi L ST M . </s> KB Q A ĠEnd - Task ĠResults </s> Table Ġ3 Ġcompares Ġour Ġsystem Ġwith Ġtwo Ġpublished Ġbas elines Ġ( 1 ) ĠST AG G ĠB IB REF 4 Ġ, Ġthe Ġstate - of - the - art Ġon ĠWeb Q SP Ġand Ġ( 2 ) ĠAM PC NN ĠB IB REF 20 Ġ, Ġthe Ġstate - of - the - art Ġon ĠSimple Questions . ĠSince Ġthese Ġtwo Ġbas elines Ġare Ġspecially Ġdesigned / tun ed Ġfor Ġone Ġparticular Ġdataset , Ġthey Ġdo Ġnot Ġgeneral ize Ġwell Ġwhen Ġapplied Ġto Ġthe Ġother Ġdataset . ĠIn Ġorder Ġto Ġhighlight Ġthe Ġeffect Ġof Ġdifferent Ġrelation Ġdetection Ġmodels Ġon Ġthe ĠKB Q A Ġend - task , Ġwe Ġalso Ġimplemented Ġanother Ġbaseline Ġthat Ġuses Ġour ĠKB Q A Ġsystem Ġbut Ġreplaces ĠHR - Bi L ST M Ġwith Ġour Ġimplementation Ġof ĠAM PC NN Ġ( for ĠSimple Questions ) Ġor Ġthe Ġchar - 3 - gram ĠBi CNN Ġ( for ĠWeb Q SP ) Ġrelation Ġdetectors Ġ( second Ġblock Ġin ĠTable Ġ3 Ġ). </s> Compared Ġto Ġthe Ġbaseline Ġrelation Ġdetector Ġ( 3 rd Ġrow Ġof Ġresults ), Ġour Ġmethod , Ġwhich Ġincludes Ġan Ġimproved Ġrelation Ġdetector Ġ( HR - Bi L ST M ), Ġimproves Ġthe ĠKB Q A Ġend Ġtask Ġby Ġ2 - 3 % Ġ( 4 th Ġrow ). ĠNote Ġthat Ġin Ġcontrast Ġto Ġprevious ĠKB Q A Ġsystems , Ġour Ġsystem Ġdoes Ġnot Ġuse Ġjoint - in ference Ġor Ġfeature - based Ġre - ranking Ġstep , Ġnevertheless Ġit Ġstill Ġachieves Ġbetter Ġor Ġcomparable Ġresults Ġto Ġthe Ġstate - of - the - art . </s> The Ġthird Ġblock Ġof Ġthe Ġtable Ġdetails Ġtwo Ġab lation Ġtests Ġfor Ġthe Ġproposed Ġcomponents Ġin Ġour ĠKB Q A Ġsystems : Ġ( 1 ) ĠRem oving Ġthe Ġentity Ġre - ranking Ġstep Ġsignificantly Ġdecreases Ġthe Ġscores . ĠSince Ġthe Ġre - ranking Ġstep Ġrelies Ġon Ġthe Ġrelation Ġdetection Ġmodels , Ġthis Ġshows Ġthat Ġour ĠHR - Bi L ST M Ġmodel Ġcontributes Ġto Ġthe Ġgood Ġperformance Ġin Ġmultiple Ġways . ĠAppendix ĠC Ġgives Ġthe Ġdetailed Ġperformance Ġof Ġthe Ġre - ranking Ġstep . Ġ( 2 ) ĠIn Ġcontrast Ġto Ġthe Ġconclusion Ġin ĠB IB REF 4 Ġ, Ġconstraint Ġdetection Ġis Ġcrucial Ġfor Ġour Ġsystem . ĠThis Ġis Ġprobably Ġbecause Ġour Ġjoint Ġperformance Ġon Ġtopic Ġentity Ġand Ġcore - chain Ġdetection Ġis Ġmore Ġaccurate Ġ( 77 . 5 % Ġtop - 1 Ġaccuracy ), Ġleaving Ġa Ġhuge Ġpotential Ġ( 77 . 5 % Ġvs . Ġ58 . 0 %) Ġfor Ġthe Ġconstraint Ġdetection Ġmodule Ġto Ġimprove . </s> Finally , Ġlike ĠST AG G , Ġwhich Ġuses Ġmultiple Ġrelation Ġdetectors Ġ( see Ġy ih 2015 sem antic Ġfor Ġthe Ġthree Ġmodels Ġused ), Ġwe Ġalso Ġtry Ġto Ġuse Ġthe Ġtop - 3 Ġrelation Ġdetectors Ġfrom ĠSection Ġ" Rel ation ĠDetection ĠResults " Ġ. ĠAs Ġshown Ġon Ġthe Ġlast Ġrow Ġof ĠTable Ġ3 Ġ, Ġthis Ġgives Ġa Ġsignificant Ġperformance Ġboost , Ġresulting Ġin Ġa Ġnew Ġstate - of - the - art Ġresult Ġon ĠSimple Questions Ġand Ġa Ġresult Ġcomparable Ġto Ġthe Ġstate - of - the - art Ġon ĠWeb Q SP . </s> Conclusion </s> KB Ġrelation Ġdetection Ġis Ġa Ġkey Ġstep Ġin ĠKB Q A Ġand Ġis Ġsignificantly Ġdifferent Ġfrom Ġgeneral Ġrelation Ġextraction Ġtasks . ĠWe Ġpropose Ġa Ġnovel ĠKB Ġrelation Ġdetection Ġmodel , ĠHR - Bi L ST M , Ġthat Ġperforms Ġhierarchical Ġmatching Ġbetween Ġquestions Ġand ĠKB Ġrelations . ĠOur Ġmodel Ġoutper forms Ġthe Ġprevious Ġmethods Ġon ĠKB Ġrelation Ġdetection Ġtasks Ġand Ġallows Ġour ĠKB Q A Ġsystem Ġto Ġachieve Ġstate - of - the - arts . ĠFor Ġfuture Ġwork , Ġwe Ġwill Ġinvestigate Ġthe Ġintegration Ġof Ġour ĠHR - Bi L ST M Ġinto Ġend - to - end Ġsystems . ĠFor Ġexample , Ġour Ġmodel Ġcould Ġbe Ġintegrated Ġinto Ġthe Ġdec oder Ġin ĠB IB REF 31 Ġ, Ġto Ġprovide Ġbetter Ġsequence Ġprediction . ĠWe Ġwill Ġalso Ġinvestigate Ġnew Ġemerging Ġdatasets Ġlike ĠGraph Questions ĠB IB REF 32 Ġand ĠComplex Questions ĠB IB REF 30 Ġto Ġhandle Ġmore Ġcharacteristics Ġof Ġgeneral ĠQ A .
Introduction Knowledge Base Question Answering (KBQA) systems answer questions by obtaining information from KB tuples BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 . For an input question, these systems typically generate a KB query, which can be executed to retrieve the answers from a KB. Figure 1 illustrates the process used to parse two sample questions in a KBQA system: (a) a single-relation question, which can be answered with a single $<$ head-entity, relation, tail-entity $>$ KB tuple BIBREF6 , BIBREF7 , BIBREF2 ; and (b) a more complex case, where some constraints need to be handled for multiple entities in the question. The KBQA system in the figure performs two key tasks: (1) entity linking, which links $n$ -grams in questions to KB entities, and (2) relation detection, which identifies the KB relation(s) a question refers to. The main focus of this work is to improve the relation detection subtask and further explore how it can contribute to the KBQA system. Although general relation detection methods are well studied in the NLP community, such studies usually do not take the end task of KBQA into consideration. As a result, there is a significant gap between general relation detection studies and KB-specific relation detection. First, in most general relation detection tasks, the number of target relations is limited, normally smaller than 100. In contrast, in KBQA even a small KB, like Freebase2M BIBREF2 , contains more than 6,000 relation types. Second, relation detection for KBQA often becomes a zero-shot learning task, since some test instances may have unseen relations in the training data. For example, the SimpleQuestions BIBREF2 data set has 14% of the golden test relations not observed in golden training tuples. Third, as shown in Figure 1 (b), for some KBQA tasks like WebQuestions BIBREF0 , we need to predict a chain of relations instead of a single relation. This increases the number of target relation types and the sizes of candidate relation pools, further increasing the difficulty of KB relation detection. Owing to these reasons, KB relation detection is significantly more challenging compared to general relation detection tasks. This paper improves KB relation detection to cope with the problems mentioned above. First, in order to deal with the unseen relations, we propose to break the relation names into word sequences for question-relation matching. Second, noticing that original relation names can sometimes help to match longer question contexts, we propose to build both relation-level and word-level relation representations. Third, we use deep bidirectional LSTMs (BiLSTMs) to learn different levels of question representations in order to match the different levels of relation information. Finally, we propose a residual learning method for sequence matching, which makes the model training easier and results in more abstract (deeper) question representations, thus improves hierarchical matching. In order to assess how the proposed improved relation detection could benefit the KBQA end task, we also propose a simple KBQA implementation composed of two-step relation detection. Given an input question and a set of candidate entities retrieved by an entity linker based on the question, our proposed relation detection model plays a key role in the KBQA process: (1) Re-ranking the entity candidates according to whether they connect to high confident relations detected from the raw question text by the relation detection model. This step is important to deal with the ambiguities normally present in entity linking results. (2) Finding the core relation (chains) for each topic entity selection from a much smaller candidate entity set after re-ranking. The above steps are followed by an optional constraint detection step, when the question cannot be answered by single relations (e.g., multiple entities in the question). Finally the highest scored query from the above steps is used to query the KB for answers. Our main contributions include: (i) An improved relation detection model by hierarchical matching between questions and relations with residual learning; (ii) We demonstrate that the improved relation detector enables our simple KBQA system to achieve state-of-the-art results on both single-relation and multi-relation KBQA tasks. Background: Different Granularity in KB Relations Previous research BIBREF4 , BIBREF20 formulates KB relation detection as a sequence matching problem. However, while the questions are natural word sequences, how to represent relations as sequences remains a challenging problem. Here we give an overview of two types of relation sequence representations commonly used in previous work. (1) Relation Name as a Single Token (relation-level). In this case, each relation name is treated as a unique token. The problem with this approach is that it suffers from the low relation coverage due to limited amount of training data, thus cannot generalize well to large number of open-domain relations. For example, in Figure 1 , when treating relation names as single tokens, it will be difficult to match the questions to relation names “episodes_written” and “starring_roles” if these names do not appear in training data – their relation embeddings $\mathbf {h}^r$ s will be random vectors thus are not comparable to question embeddings $\mathbf {h}^q$ s. (2) Relation as Word Sequence (word-level). In this case, the relation is treated as a sequence of words from the tokenized relation name. It has better generalization, but suffers from the lack of global information from the original relation names. For example in Figure 1 (b), when doing only word-level matching, it is difficult to rank the target relation “starring_roles” higher compared to the incorrect relation “plays_produced”. This is because the incorrect relation contains word “plays”, which is more similar to the question (containing word “play”) in the embedding space. On the other hand, if the target relation co-occurs with questions related to “tv appearance” in training, by treating the whole relation as a token (i.e. relation id), we could better learn the correspondence between this token and phrases like “tv show” and “play on”. The two types of relation representation contain different levels of abstraction. As shown in Table 1 , the word-level focuses more on local information (words and short phrases), and the relation-level focus more on global information (long phrases and skip-grams) but suffer from data sparsity. Since both these levels of granularity have their own pros and cons, we propose a hierarchical matching approach for KB relation detection: for a candidate relation, our approach matches the input question to both word-level and relation-level representations to get the final ranking score. Section "Improved KB Relation Detection" gives the details of our proposed approach. Improved KB Relation Detection This section describes our hierarchical sequence matching with residual learning approach for relation detection. In order to match the question to different aspects of a relation (with different abstraction levels), we deal with three problems as follows on learning question/relation representations. Relation Representations from Different Granularity We provide our model with both types of relation representation: word-level and relation-level. Therefore, the input relation becomes $\mathbf {r}=\lbrace r^{word}_1,\cdots ,r^{word}_{M_1}\rbrace  \cup \lbrace r^{rel}_1,\cdots ,r^{rel}_{M_2}\rbrace $ , where the first $M_1$ tokens are words (e.g. {episode, written}), and the last $M_2$ tokens are relation names, e.g., {episode_written} or {starring_roles, series} (when the target is a chain like in Figure 1 (b)). We transform each token above to its word embedding then use two BiLSTMs (with shared parameters) to get their hidden representations $[\mathbf {B}^{word}_{1:M_1}:\mathbf {B}^{rel}_{1:M_2}]$ (each row vector $\mathbf {\beta }_i$ is the concatenation between forward/backward representations at $i$ ). We initialize the relation sequence LSTMs with the final state representations of the word sequence, as a back-off for unseen relations. We apply one max-pooling on these two sets of vectors and get the final relation representation $\mathbf {h}^r$ . Different Abstractions of Questions Representations From Table 1 , we can see that different parts of a relation could match different contexts of question texts. Usually relation names could match longer phrases in the question and relation words could match short phrases. Yet different words might match phrases of different lengths. As a result, we hope the question representations could also comprise vectors that summarize various lengths of phrase information (different levels of abstraction), in order to match relation representations of different granularity. We deal with this problem by applying deep BiLSTMs on questions. The first-layer of BiLSTM works on the word embeddings of question words $\mathbf {q}=\lbrace q_1,\cdots ,q_N\rbrace $ and gets hidden representations $\mathbf {\Gamma }^{(1)}_{1:N}=[\mathbf {\gamma }^{(1)}_1;\cdots ;\mathbf {\gamma }^{(1)}_N]$ . The second-layer BiLSTM works on $\mathbf {\Gamma }^{(1)}_{1:N}$ to get the second set of hidden representations $\mathbf {\Gamma }^{(2)}_{1:N}$ . Since the second BiLSTM starts with the hidden vectors from the first layer, intuitively it could learn more general and abstract information compared to the first layer. Note that the first(second)-layer of question representations does not necessarily correspond to the word(relation)-level relation representations, instead either layer of question representations could potentially match to either level of relation representations. This raises the difficulty of matching between different levels of relation/question representations; the following section gives our proposal to deal with such problem. Hierarchical Matching between Relation and Question Now we have question contexts of different lengths encoded in $\mathbf {\Gamma }^{(1)}_{1:N}$ and $\mathbf {\Gamma }^{(2)}_{1:N}$ . Unlike the standard usage of deep BiLSTMs that employs the representations in the final layer for prediction, here we expect that two layers of question representations can be complementary to each other and both should be compared to the relation representation space (Hierarchical Matching). This is important for our task since each relation token can correspond to phrases of different lengths, mainly because of syntactic variations. For example in Table 1 , the relation word written could be matched to either the same single word in the question or a much longer phrase be the writer of. We could perform the above hierarchical matching by computing the similarity between each layer of $\mathbf {\Gamma }$ and $\mathbf {h}^r$ separately and doing the (weighted) sum between the two scores. However this does not give significant improvement (see Table 2 ). Our analysis in Section "Relation Detection Results" shows that this naive method suffers from the training difficulty, evidenced by that the converged training loss of this model is much higher than that of a single-layer baseline model. This is mainly because (1) Deep BiLSTMs do not guarantee that the two-levels of question hidden representations are comparable, the training usually falls to local optima where one layer has good matching scores and the other always has weight close to 0. (2) The training of deeper architectures itself is more difficult. To overcome the above difficulties, we adopt the idea from Residual Networks BIBREF23 for hierarchical matching by adding shortcut connections between two BiLSTM layers. We proposed two ways of such Hierarchical Residual Matching: (1) Connecting each $\mathbf {\gamma }^{(1)}_i$ and $\mathbf {\gamma }^{(2)}_i$ , resulting in a $\mathbf {\gamma }^{^{\prime }}_i=\mathbf {\gamma }^{(1)}_i + \mathbf {\gamma }^{(2)}_i$ for each position $i$ . Then the final question representation $\mathbf {h}^q$ becomes a max-pooling over all $\mathbf {\gamma }^{^{\prime }}_i$ s, 1 $\le $ i $\le $ $N$ . (2) Applying max-pooling on $\mathbf {\Gamma }^{(1)}_{1:N}$ and $\mathbf {\gamma }^{(2)}_i$0 to get $\mathbf {\gamma }^{(2)}_i$1 and $\mathbf {\gamma }^{(2)}_i$2 , respectively, then setting $\mathbf {\gamma }^{(2)}_i$3 . Finally we compute the matching score of $\mathbf {\gamma }^{(2)}_i$4 given $\mathbf {\gamma }^{(2)}_i$5 as $\mathbf {\gamma }^{(2)}_i$6 . Intuitively, the proposed method should benefit from hierarchical training since the second layer is fitting the residues from the first layer of matching, so the two layers of representations are more likely to be complementary to each other. This also ensures the vector spaces of two layers are comparable and makes the second-layer training easier. During training we adopt a ranking loss to maximizing the margin between the gold relation $\mathbf {r}^+$ and other relations $\mathbf {r}^-$ in the candidate pool $R$ . $$l_{\mathrm {rel}} = \max \lbrace 0, \gamma - s_{\mathrm {rel}}(\mathbf {r}^+; \mathbf {q}) + s_{\mathrm {rel}}(\mathbf {r}^-; \mathbf {q})\rbrace  \nonumber $$   (Eq. 12) where $\gamma $ is a constant parameter. Fig 2 summarizes the above Hierarchical Residual BiLSTM (HR-BiLSTM) model. Another way of hierarchical matching consists in relying on attention mechanism, e.g. BIBREF24 , to find the correspondence between different levels of representations. This performs below the HR-BiLSTM (see Table 2 ). KBQA Enhanced by Relation Detection This section describes our KBQA pipeline system. We make minimal efforts beyond the training of the relation detection model, making the whole system easy to build. Following previous work BIBREF4 , BIBREF5 , our KBQA system takes an existing entity linker to produce the top- $K$ linked entities, $EL_K(q)$ , for a question $q$ (“initial entity linking”). Then we generate the KB queries for $q$ following the four steps illustrated in Algorithm "KBQA Enhanced by Relation Detection" . [htbp] InputInput OutputOutput Top query tuple $(\hat{e},\hat{r}, \lbrace (c, r_c)\rbrace )$ Entity Re-Ranking (first-step relation detection): Use the raw question text as input for a relation detector to score all relations in the KB that are associated to the entities in $EL_K(q)$ ; use the relation scores to re-rank $EL_K(q)$ and generate a shorter list $EL^{\prime }_{K^{\prime }}(q)$ containing the top- $K^{\prime }$ entity candidates (Section "Entity Re-Ranking" ) Relation Detection: Detect relation(s) using the reformatted question text in which the topic entity is replaced by a special token $<$ e $>$ (Section "Relation Detection" ) Query Generation: Combine the scores from step 1 and 2, and select the top pair $(\hat{e},\hat{r})$ (Section "Query Generation" ) Constraint Detection (optional): Compute similarity between $q$ and any neighbor entity $c$ of the entities along $EL_K(q)$0 (connecting by a relation $EL_K(q)$1 ) , add the high scoring $EL_K(q)$2 and $EL_K(q)$3 to the query (Section "Constraint Detection" ). KBQA with two-step relation detection Compared to previous approaches, the main difference is that we have an additional entity re-ranking step after the initial entity linking. We have this step because we have observed that entity linking sometimes becomes a bottleneck in KBQA systems. For example, on SimpleQuestions the best reported linker could only get 72.7% top-1 accuracy on identifying topic entities. This is usually due to the ambiguities of entity names, e.g. in Fig 1 (a), there are TV writer and baseball player “Mike Kelley”, which is impossible to distinguish with only entity name matching. Having observed that different entity candidates usually connect to different relations, here we propose to help entity disambiguation in the initial entity linking with relations detected in questions. Sections "Entity Re-Ranking" and "Relation Detection" elaborate how our relation detection help to re-rank entities in the initial entity linking, and then those re-ranked entities enable more accurate relation detection. The KBQA end task, as a result, benefits from this process. Entity Re-Ranking In this step, we use the raw question text as input for a relation detector to score all relations in the KB with connections to at least one of the entity candidates in $EL_K(q)$ . We call this step relation detection on entity set since it does not work on a single topic entity as the usual settings. We use the HR-BiLSTM as described in Sec. "Improved KB Relation Detection" . For each question $q$ , after generating a score $s_{rel}(r;q)$ for each relation using HR-BiLSTM, we use the top $l$ best scoring relations ( $R^{l}_q$ ) to re-rank the original entity candidates. Concretely, for each entity $e$ and its associated relations $R_e$ , given the original entity linker score $s_{linker}$ , and the score of the most confident relation $r\in R_q^{l} \cap R_e$ , we sum these two scores to re-rank the entities: $$s_{\mathrm {rerank}}(e;q) =& \alpha \cdot s_{\mathrm {linker}}(e;q) \nonumber \\ + & (1-\alpha ) \cdot \max _{r \in R_q^{l} \cap R_e} s_{\mathrm {rel}}(r;q).\nonumber $$   (Eq. 15) Finally, we select top $K^{\prime }$ $<$ $K$ entities according to score $s_{rerank}$ to form the re-ranked list $EL_{K^{\prime }}^{^{\prime }}(q)$ . We use the same example in Fig 1 (a) to illustrate the idea. Given the input question in the example, a relation detector is very likely to assign high scores to relations such as “episodes_written”, “author_of” and “profession”. Then, according to the connections of entity candidates in KB, we find that the TV writer “Mike Kelley” will be scored higher than the baseball player “Mike Kelley”, because the former has the relations “episodes_written” and “profession”. This method can be viewed as exploiting entity-relation collocation for entity linking. Relation Detection In this step, for each candidate entity $e \in EL_K^{\prime }(q)$ , we use the question text as the input to a relation detector to score all the relations $r \in R_e$ that are associated to the entity $e$ in the KB. Because we have a single topic entity input in this step, we do the following question reformatting: we replace the the candidate $e$ 's entity mention in $q$ with a token “ $<$ e $>$ ”. This helps the model better distinguish the relative position of each word compared to the entity. We use the HR-BiLSTM model to predict the score of each relation $r \in R_e$ : $s_{rel} (r;e,q)$ . Query Generation Finally, the system outputs the $<$ entity, relation (or core-chain) $>$ pair $(\hat{e}, \hat{r})$ according to: $$s(\hat{e}, \hat{r}; q) =& \max _{e \in EL_{K^{\prime }}^{^{\prime }}(q), r \in R_e} \left( \beta \cdot s_{\mathrm {rerank}}(e;q) \right. \nonumber \\ &\left.+ (1-\beta ) \cdot s_{\mathrm {rel}} (r;e,q) \right), \nonumber $$   (Eq. 19) where $\beta $ is a hyperparameter to be tuned. Constraint Detection Similar to BIBREF4 , we adopt an additional constraint detection step based on text matching. Our method can be viewed as entity-linking on a KB sub-graph. It contains two steps: (1) Sub-graph generation: given the top scored query generated by the previous 3 steps, for each node $v$ (answer node or the CVT node like in Figure 1 (b)), we collect all the nodes $c$ connecting to $v$ (with relation $r_c$ ) with any relation, and generate a sub-graph associated to the original query. (2) Entity-linking on sub-graph nodes: we compute a matching score between each $n$ -gram in the input question (without overlapping the topic entity) and entity name of $c$ (except for the node in the original query) by taking into account the maximum overlapping sequence of characters between them (see Appendix A for details and B for special rules dealing with date/answer type constraints). If the matching score is larger than a threshold $\theta $ (tuned on training set), we will add the constraint entity $c$ (and $r_c$ ) to the query by attaching it to the corresponding node $v$ on the core-chain. Experiments Task Introduction & Settings We use the SimpleQuestions BIBREF2 and WebQSP BIBREF25 datasets. Each question in these datasets is labeled with the gold semantic parse. Hence we can directly evaluate relation detection performance independently as well as evaluate on the KBQA end task. SimpleQuestions (SQ): It is a single-relation KBQA task. The KB we use consists of a Freebase subset with 2M entities (FB2M) BIBREF2 , in order to compare with previous research. yin2016simple also evaluated their relation extractor on this data set and released their proposed question-relation pairs, so we run our relation detection model on their data set. For the KBQA evaluation, we also start with their entity linking results. Therefore, our results can be compared with their reported results on both tasks. WebQSP (WQ): A multi-relation KBQA task. We use the entire Freebase KB for evaluation purposes. Following yih-EtAl:2016:P16-2, we use S-MART BIBREF26 entity-linking outputs. In order to evaluate the relation detection models, we create a new relation detection task from the WebQSP data set. For each question and its labeled semantic parse: (1) we first select the topic entity from the parse; and then (2) select all the relations and relation chains (length $\le $ 2) connected to the topic entity, and set the core-chain labeled in the parse as the positive label and all the others as the negative examples. We tune the following hyper-parameters on development sets: (1) the size of hidden states for LSTMs ({50, 100, 200, 400}); (2) learning rate ({0.1, 0.5, 1.0, 2.0}); (3) whether the shortcut connections are between hidden states or between max-pooling results (see Section "Hierarchical Matching between Relation and Question" ); and (4) the number of training epochs. For both the relation detection experiments and the second-step relation detection in KBQA, we have entity replacement first (see Section "Relation Detection" and Figure 1 ). All word vectors are initialized with 300- $d$ pretrained word embeddings BIBREF27 . The embeddings of relation names are randomly initialized, since existing pre-trained relation embeddings (e.g. TransE) usually support limited sets of relation names. We leave the usage of pre-trained relation embeddings to future work. Relation Detection Results Table 2 shows the results on two relation detection tasks. The AMPCNN result is from BIBREF20 , which yielded state-of-the-art scores by outperforming several attention-based methods. We re-implemented the BiCNN model from BIBREF4 , where both questions and relations are represented with the word hash trick on character tri-grams. The baseline BiLSTM with relation word sequence appears to be the best baseline on WebQSP and is close to the previous best result of AMPCNN on SimpleQuestions. Our proposed HR-BiLSTM outperformed the best baselines on both tasks by margins of 2-3% (p $<$ 0.001 and 0.01 compared to the best baseline BiLSTM w/ words on SQ and WQ respectively). Note that using only relation names instead of words results in a weaker baseline BiLSTM model. The model yields a significant performance drop on SimpleQuestions (91.2% to 88.9%). However, the drop is much smaller on WebQSP, and it suggests that unseen relations have a much bigger impact on SimpleQuestions. The bottom of Table 2 shows ablation results of the proposed HR-BiLSTM. First, hierarchical matching between questions and both relation names and relation words yields improvement on both datasets, especially for SimpleQuestions (93.3% vs. 91.2/88.8%). Second, residual learning helps hierarchical matching compared to weighted-sum and attention-based baselines (see Section "Hierarchical Matching between Relation and Question" ). For the attention-based baseline, we tried the model from BIBREF24 and its one-way variations, where the one-way model gives better results. Note that residual learning significantly helps on WebQSP (80.65% to 82.53%), while it does not help as much on SimpleQuestions. On SimpleQuestions, even removing the deep layers only causes a small drop in performance. WebQSP benefits more from residual and deeper architecture, possibly because in this dataset it is more important to handle larger scope of context matching. Finally, on WebQSP, replacing BiLSTM with CNN in our hierarchical matching framework results in a large performance drop. Yet on SimpleQuestions the gap is much smaller. We believe this is because the LSTM relation encoder can better learn the composition of chains of relations in WebQSP, as it is better at dealing with longer dependencies. Next, we present empirical evidences, which show why our HR-BiLSTM model achieves the best scores. We use WebQSP for the analysis purposes. First, we have the hypothesis that training of the weighted-sum model usually falls to local optima, since deep BiLSTMs do not guarantee that the two-levels of question hidden representations are comparable. This is evidenced by that during training one layer usually gets a weight close to 0 thus is ignored. For example, one run gives us weights of -75.39/0.14 for the two layers (we take exponential for the final weighted sum). It also gives much lower training accuracy (91.94%) compared to HR-BiLSTM (95.67%), suffering from training difficulty. Second, compared to our deep BiLSTM with shortcut connections, we have the hypothesis that for KB relation detection, training deep BiLSTMs is more difficult without shortcut connections. Our experiments suggest that deeper BiLSTM does not always result in lower training accuracy. In the experiments a two-layer BiLSTM converges to 94.99%, even lower than the 95.25% achieved by a single-layer BiLSTM. Under our setting the two-layer model captures the single-layer model as a special case (so it could potentially better fit the training data), this result suggests that the deep BiLSTM without shortcut connections might suffers more from training difficulty. Finally, we hypothesize that HR-BiLSTM is more than combination of two BiLSTMs with residual connections, because it encourages the hierarchical architecture to learn different levels of abstraction. To verify this, we replace the deep BiLSTM question encoder with two single-layer BiLSTMs (both on words) with shortcut connections between their hidden states. This decreases test accuracy to 76.11%. It gives similar training accuracy compared to HR-BiLSTM, indicating a more serious over-fitting problem. This proves that the residual and deep structures both contribute to the good performance of HR-BiLSTM. KBQA End-Task Results Table 3 compares our system with two published baselines (1) STAGG BIBREF4 , the state-of-the-art on WebQSP and (2) AMPCNN BIBREF20 , the state-of-the-art on SimpleQuestions. Since these two baselines are specially designed/tuned for one particular dataset, they do not generalize well when applied to the other dataset. In order to highlight the effect of different relation detection models on the KBQA end-task, we also implemented another baseline that uses our KBQA system but replaces HR-BiLSTM with our implementation of AMPCNN (for SimpleQuestions) or the char-3-gram BiCNN (for WebQSP) relation detectors (second block in Table 3 ). Compared to the baseline relation detector (3rd row of results), our method, which includes an improved relation detector (HR-BiLSTM), improves the KBQA end task by 2-3% (4th row). Note that in contrast to previous KBQA systems, our system does not use joint-inference or feature-based re-ranking step, nevertheless it still achieves better or comparable results to the state-of-the-art. The third block of the table details two ablation tests for the proposed components in our KBQA systems: (1) Removing the entity re-ranking step significantly decreases the scores. Since the re-ranking step relies on the relation detection models, this shows that our HR-BiLSTM model contributes to the good performance in multiple ways. Appendix C gives the detailed performance of the re-ranking step. (2) In contrast to the conclusion in BIBREF4 , constraint detection is crucial for our system. This is probably because our joint performance on topic entity and core-chain detection is more accurate (77.5% top-1 accuracy), leaving a huge potential (77.5% vs. 58.0%) for the constraint detection module to improve. Finally, like STAGG, which uses multiple relation detectors (see yih2015semantic for the three models used), we also try to use the top-3 relation detectors from Section "Relation Detection Results" . As shown on the last row of Table 3 , this gives a significant performance boost, resulting in a new state-of-the-art result on SimpleQuestions and a result comparable to the state-of-the-art on WebQSP. Conclusion KB relation detection is a key step in KBQA and is significantly different from general relation extraction tasks. We propose a novel KB relation detection model, HR-BiLSTM, that performs hierarchical matching between questions and KB relations. Our model outperforms the previous methods on KB relation detection tasks and allows our KBQA system to achieve state-of-the-arts. For future work, we will investigate the integration of our HR-BiLSTM into end-to-end systems. For example, our model could be integrated into the decoder in BIBREF31 , to provide better sequence prediction. We will also investigate new emerging datasets like GraphQuestions BIBREF32 and ComplexQuestions BIBREF30 to handle more characteristics of general QA.
What is te core component for KBQA?
c0e341c4d2253eb42c8840381b082aae274eddad
What Ġis Ġte Ġcore Ġcomponent Ġfor ĠKB Q A ?
answer Ġquestions Ġby Ġobtaining Ġinformation Ġfrom ĠKB Ġtu ples Ġ
<s> What Ġis Ġte Ġcore Ġcomponent Ġfor ĠKB Q A ? </s> Introduction </s> Know ledge ĠBase ĠQuestion ĠAn sw ering Ġ( KB Q A ) Ġsystems Ġanswer Ġquestions Ġby Ġobtaining Ġinformation Ġfrom ĠKB Ġtu ples ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ. ĠFor Ġan Ġinput Ġquestion , Ġthese Ġsystems Ġtypically Ġgenerate Ġa ĠKB Ġquery , Ġwhich Ġcan Ġbe Ġexecuted Ġto Ġretrieve Ġthe Ġanswers Ġfrom Ġa ĠKB . ĠFigure Ġ1 Ġillustrates Ġthe Ġprocess Ġused Ġto Ġparse Ġtwo Ġsample Ġquestions Ġin Ġa ĠKB Q A Ġsystem : Ġ( a ) Ġa Ġsingle - relation Ġquestion , Ġwhich Ġcan Ġbe Ġanswered Ġwith Ġa Ġsingle Ġ$ < $ Ġhead - entity , Ġrelation , Ġtail - entity Ġ$ > $ ĠKB Ġtuple ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 2 Ġ; Ġand Ġ( b ) Ġa Ġmore Ġcomplex Ġcase , Ġwhere Ġsome Ġconstraints Ġneed Ġto Ġbe Ġhandled Ġfor Ġmultiple Ġentities Ġin Ġthe Ġquestion . ĠThe ĠKB Q A Ġsystem Ġin Ġthe Ġfigure Ġperforms Ġtwo Ġkey Ġtasks : Ġ( 1 ) Ġentity Ġlinking , Ġwhich Ġlinks Ġ$ n $ Ġ- gram s Ġin Ġquestions Ġto ĠKB Ġentities , Ġand Ġ( 2 ) Ġrelation Ġdetection , Ġwhich Ġidentifies Ġthe ĠKB Ġrelation ( s ) Ġa Ġquestion Ġrefers Ġto . </s> The Ġmain Ġfocus Ġof Ġthis Ġwork Ġis Ġto Ġimprove Ġthe Ġrelation Ġdetection Ġsubt ask Ġand Ġfurther Ġexplore Ġhow Ġit Ġcan Ġcontribute Ġto Ġthe ĠKB Q A Ġsystem . ĠAlthough Ġgeneral Ġrelation Ġdetection Ġmethods Ġare Ġwell Ġstudied Ġin Ġthe ĠN LP Ġcommunity , Ġsuch Ġstudies Ġusually Ġdo Ġnot Ġtake Ġthe Ġend Ġtask Ġof ĠKB Q A Ġinto Ġconsideration . ĠAs Ġa Ġresult , Ġthere Ġis Ġa Ġsignificant Ġgap Ġbetween Ġgeneral Ġrelation Ġdetection Ġstudies Ġand ĠKB - specific Ġrelation Ġdetection . ĠFirst , Ġin Ġmost Ġgeneral Ġrelation Ġdetection Ġtasks , Ġthe Ġnumber Ġof Ġtarget Ġrelations Ġis Ġlimited , Ġnormally Ġsmaller Ġthan Ġ100 . ĠIn Ġcontrast , Ġin ĠKB Q A Ġeven Ġa Ġsmall ĠKB , Ġlike ĠFree base 2 M ĠB IB REF 2 Ġ, Ġcontains Ġmore Ġthan Ġ6 , 000 Ġrelation Ġtypes . ĠSecond , Ġrelation Ġdetection Ġfor ĠKB Q A Ġoften Ġbecomes Ġa Ġzero - shot Ġlearning Ġtask , Ġsince Ġsome Ġtest Ġinstances Ġmay Ġhave Ġunseen Ġrelations Ġin Ġthe Ġtraining Ġdata . ĠFor Ġexample , Ġthe ĠSimple Questions ĠB IB REF 2 Ġdata Ġset Ġhas Ġ14 % Ġof Ġthe Ġgolden Ġtest Ġrelations Ġnot Ġobserved Ġin Ġgolden Ġtraining Ġtu ples . ĠThird , Ġas Ġshown Ġin ĠFigure Ġ1 Ġ( b ), Ġfor Ġsome ĠKB Q A Ġtasks Ġlike ĠWeb Questions ĠB IB REF 0 Ġ, Ġwe Ġneed Ġto Ġpredict Ġa Ġchain Ġof Ġrelations Ġinstead Ġof Ġa Ġsingle Ġrelation . ĠThis Ġincreases Ġthe Ġnumber Ġof Ġtarget Ġrelation Ġtypes Ġand Ġthe Ġsizes Ġof Ġcandidate Ġrelation Ġpools , Ġfurther Ġincreasing Ġthe Ġdifficulty Ġof ĠKB Ġrelation Ġdetection . ĠO wing Ġto Ġthese Ġreasons , ĠKB Ġrelation Ġdetection Ġis Ġsignificantly Ġmore Ġchallenging Ġcompared Ġto Ġgeneral Ġrelation Ġdetection Ġtasks . </s> This Ġpaper Ġimproves ĠKB Ġrelation Ġdetection Ġto Ġcope Ġwith Ġthe Ġproblems Ġmentioned Ġabove . ĠFirst , Ġin Ġorder Ġto Ġdeal Ġwith Ġthe Ġunseen Ġrelations , Ġwe Ġpropose Ġto Ġbreak Ġthe Ġrelation Ġnames Ġinto Ġword Ġsequences Ġfor Ġquestion - relation Ġmatching . ĠSecond , Ġnoticing Ġthat Ġoriginal Ġrelation Ġnames Ġcan Ġsometimes Ġhelp Ġto Ġmatch Ġlonger Ġquestion Ġcontexts , Ġwe Ġpropose Ġto Ġbuild Ġboth Ġrelation - level Ġand Ġword - level Ġrelation Ġrepresentations . ĠThird , Ġwe Ġuse Ġdeep Ġbid irection al ĠL ST Ms Ġ( Bi L ST Ms ) Ġto Ġlearn Ġdifferent Ġlevels Ġof Ġquestion Ġrepresentations Ġin Ġorder Ġto Ġmatch Ġthe Ġdifferent Ġlevels Ġof Ġrelation Ġinformation . ĠFinally , Ġwe Ġpropose Ġa Ġresidual Ġlearning Ġmethod Ġfor Ġsequence Ġmatching , Ġwhich Ġmakes Ġthe Ġmodel Ġtraining Ġeasier Ġand Ġresults Ġin Ġmore Ġabstract Ġ( d eeper ) Ġquestion Ġrepresentations , Ġthus Ġimproves Ġhierarchical Ġmatching . </s> In Ġorder Ġto Ġassess Ġhow Ġthe Ġproposed Ġimproved Ġrelation Ġdetection Ġcould Ġbenefit Ġthe ĠKB Q A Ġend Ġtask , Ġwe Ġalso Ġpropose Ġa Ġsimple ĠKB Q A Ġimplementation Ġcomposed Ġof Ġtwo - step Ġrelation Ġdetection . ĠGiven Ġan Ġinput Ġquestion Ġand Ġa Ġset Ġof Ġcandidate Ġentities Ġretrieved Ġby Ġan Ġentity Ġlink er Ġbased Ġon Ġthe Ġquestion , Ġour Ġproposed Ġrelation Ġdetection Ġmodel Ġplays Ġa Ġkey Ġrole Ġin Ġthe ĠKB Q A Ġprocess : Ġ( 1 ) ĠRe - ranking Ġthe Ġentity Ġcandidates Ġaccording Ġto Ġwhether Ġthey Ġconnect Ġto Ġhigh Ġconfident Ġrelations Ġdetected Ġfrom Ġthe Ġraw Ġquestion Ġtext Ġby Ġthe Ġrelation Ġdetection Ġmodel . ĠThis Ġstep Ġis Ġimportant Ġto Ġdeal Ġwith Ġthe Ġambig u ities Ġnormally Ġpresent Ġin Ġentity Ġlinking Ġresults . Ġ( 2 ) ĠFinding Ġthe Ġcore Ġrelation Ġ( chains ) Ġfor Ġeach Ġtopic Ġentity Ġselection Ġfrom Ġa Ġmuch Ġsmaller Ġcandidate Ġentity Ġset Ġafter Ġre - ranking . ĠThe Ġabove Ġsteps Ġare Ġfollowed Ġby Ġan Ġoptional Ġconstraint Ġdetection Ġstep , Ġwhen Ġthe Ġquestion Ġcannot Ġbe Ġanswered Ġby Ġsingle Ġrelations Ġ( e . g ., Ġmultiple Ġentities Ġin Ġthe Ġquestion ). ĠFinally Ġthe Ġhighest Ġscored Ġquery Ġfrom Ġthe Ġabove Ġsteps Ġis Ġused Ġto Ġquery Ġthe ĠKB Ġfor Ġanswers . </s> Our Ġmain Ġcontributions Ġinclude : Ġ( i ) ĠAn Ġimproved Ġrelation Ġdetection Ġmodel Ġby Ġhierarchical Ġmatching Ġbetween Ġquestions Ġand Ġrelations Ġwith Ġresidual Ġlearning ; Ġ( ii ) ĠWe Ġdemonstrate Ġthat Ġthe Ġimproved Ġrelation Ġdetector Ġenables Ġour Ġsimple ĠKB Q A Ġsystem Ġto Ġachieve Ġstate - of - the - art Ġresults Ġon Ġboth Ġsingle - relation Ġand Ġmulti - relation ĠKB Q A Ġtasks . </s> Background : ĠDifferent ĠGran ularity Ġin ĠKB ĠRelations </s> Previous Ġresearch ĠB IB REF 4 Ġ, ĠB IB REF 20 Ġform ulates ĠKB Ġrelation Ġdetection Ġas Ġa Ġsequence Ġmatching Ġproblem . ĠHowever , Ġwhile Ġthe Ġquestions Ġare Ġnatural Ġword Ġsequences , Ġhow Ġto Ġrepresent Ġrelations Ġas Ġsequences Ġremains Ġa Ġchallenging Ġproblem . ĠHere Ġwe Ġgive Ġan Ġoverview Ġof Ġtwo Ġtypes Ġof Ġrelation Ġsequence Ġrepresentations Ġcommonly Ġused Ġin Ġprevious Ġwork . </s> ( 1 ) ĠRel ation ĠName Ġas Ġa ĠSingle ĠToken Ġ( relation - level ). ĠIn Ġthis Ġcase , Ġeach Ġrelation Ġname Ġis Ġtreated Ġas Ġa Ġunique Ġtoken . ĠThe Ġproblem Ġwith Ġthis Ġapproach Ġis Ġthat Ġit Ġsuffers Ġfrom Ġthe Ġlow Ġrelation Ġcoverage Ġdue Ġto Ġlimited Ġamount Ġof Ġtraining Ġdata , Ġthus Ġcannot Ġgeneral ize Ġwell Ġto Ġlarge Ġnumber Ġof Ġopen - domain Ġrelations . ĠFor Ġexample , Ġin ĠFigure Ġ1 Ġ, Ġwhen Ġtreating Ġrelation Ġnames Ġas Ġsingle Ġtokens , Ġit Ġwill Ġbe Ġdifficult Ġto Ġmatch Ġthe Ġquestions Ġto Ġrelation Ġnames ĠâĢ ľ ep isodes _ written âĢ Ŀ Ġand ĠâĢ ľ star ring _ ro les âĢ Ŀ Ġif Ġthese Ġnames Ġdo Ġnot Ġappear Ġin Ġtraining Ġdata ĠâĢĵ Ġtheir Ġrelation Ġembed d ings Ġ$\ math bf Ġ{ h } ^ r $ Ġs Ġwill Ġbe Ġrandom Ġvectors Ġthus Ġare Ġnot Ġcomparable Ġto Ġquestion Ġembed d ings Ġ$\ math bf Ġ{ h } ^ q $ Ġs . </s> ( 2 ) ĠRel ation Ġas ĠWord ĠSequence Ġ( word - level ). ĠIn Ġthis Ġcase , Ġthe Ġrelation Ġis Ġtreated Ġas Ġa Ġsequence Ġof Ġwords Ġfrom Ġthe Ġtoken ized Ġrelation Ġname . ĠIt Ġhas Ġbetter Ġgeneral ization , Ġbut Ġsuffers Ġfrom Ġthe Ġlack Ġof Ġglobal Ġinformation Ġfrom Ġthe Ġoriginal Ġrelation Ġnames . ĠFor Ġexample Ġin ĠFigure Ġ1 Ġ( b ), Ġwhen Ġdoing Ġonly Ġword - level Ġmatching , Ġit Ġis Ġdifficult Ġto Ġrank Ġthe Ġtarget Ġrelation ĠâĢ ľ star ring _ ro les âĢ Ŀ Ġhigher Ġcompared Ġto Ġthe Ġincorrect Ġrelation ĠâĢ ľ plays _ produced âĢ Ŀ . ĠThis Ġis Ġbecause Ġthe Ġincorrect Ġrelation Ġcontains Ġword ĠâĢ ľ plays âĢ Ŀ , Ġwhich Ġis Ġmore Ġsimilar Ġto Ġthe Ġquestion Ġ( containing Ġword ĠâĢ ľ play âĢ Ŀ ) Ġin Ġthe Ġembed ding Ġspace . ĠOn Ġthe Ġother Ġhand , Ġif Ġthe Ġtarget Ġrelation Ġco - occ urs Ġwith Ġquestions Ġrelated Ġto ĠâĢ ľ tv Ġappearance âĢ Ŀ Ġin Ġtraining , Ġby Ġtreating Ġthe Ġwhole Ġrelation Ġas Ġa Ġtoken Ġ( i . e . Ġrelation Ġid ), Ġwe Ġcould Ġbetter Ġlearn Ġthe Ġcorrespondence Ġbetween Ġthis Ġtoken Ġand Ġphrases Ġlike ĠâĢ ľ tv Ġshow âĢ Ŀ Ġand ĠâĢ ľ play Ġon âĢ Ŀ . </s> The Ġtwo Ġtypes Ġof Ġrelation Ġrepresentation Ġcontain Ġdifferent Ġlevels Ġof Ġabstraction . ĠAs Ġshown Ġin ĠTable Ġ1 Ġ, Ġthe Ġword - level Ġfocuses Ġmore Ġon Ġlocal Ġinformation Ġ( words Ġand Ġshort Ġphrases ), Ġand Ġthe Ġrelation - level Ġfocus Ġmore Ġon Ġglobal Ġinformation Ġ( long Ġphrases Ġand Ġskip - gram s ) Ġbut Ġsuffer Ġfrom Ġdata Ġsp arsity . ĠSince Ġboth Ġthese Ġlevels Ġof Ġgran ularity Ġhave Ġtheir Ġown Ġpros Ġand Ġcons , Ġwe Ġpropose Ġa Ġhierarchical Ġmatching Ġapproach Ġfor ĠKB Ġrelation Ġdetection : Ġfor Ġa Ġcandidate Ġrelation , Ġour Ġapproach Ġmatches Ġthe Ġinput Ġquestion Ġto Ġboth Ġword - level Ġand Ġrelation - level Ġrepresentations Ġto Ġget Ġthe Ġfinal Ġranking Ġscore . ĠSection Ġ" Improved ĠKB ĠRel ation ĠDetection " Ġgives Ġthe Ġdetails Ġof Ġour Ġproposed Ġapproach . </s> Improved ĠKB ĠRel ation ĠDetection </s> This Ġsection Ġdescribes Ġour Ġhierarchical Ġsequence Ġmatching Ġwith Ġresidual Ġlearning Ġapproach Ġfor Ġrelation Ġdetection . ĠIn Ġorder Ġto Ġmatch Ġthe Ġquestion Ġto Ġdifferent Ġaspects Ġof Ġa Ġrelation Ġ( with Ġdifferent Ġabstraction Ġlevels ), Ġwe Ġdeal Ġwith Ġthree Ġproblems Ġas Ġfollows Ġon Ġlearning Ġquestion / relation Ġrepresentations . </s> Rel ation ĠRepresent ations Ġfrom ĠDifferent ĠGran ularity </s> We Ġprovide Ġour Ġmodel Ġwith Ġboth Ġtypes Ġof Ġrelation Ġrepresentation : Ġword - level Ġand Ġrelation - level . ĠTherefore , Ġthe Ġinput Ġrelation Ġbecomes Ġ$\ math bf Ġ{ r } = \ l brace Ġr ^{ word } _ 1 , \ cd ots Ġ, r ^{ word } _{ M _ 1 }\ r brace Ġ Ġ\ cup Ġ\ l brace Ġr ^{ rel } _ 1 , \ cd ots Ġ, r ^{ rel } _{ M _ 2 }\ r brace Ġ$ Ġ, Ġwhere Ġthe Ġfirst Ġ$ M _ 1 $ Ġtokens Ġare Ġwords Ġ( e . g . Ġ{ episode , Ġwritten } ), Ġand Ġthe Ġlast Ġ$ M _ 2 $ Ġtokens Ġare Ġrelation Ġnames , Ġe . g ., Ġ{ episode _ written } Ġor Ġ{ star ring _ ro les , Ġseries } Ġ( when Ġthe Ġtarget Ġis Ġa Ġchain Ġlike Ġin ĠFigure Ġ1 Ġ( b )). ĠWe Ġtransform Ġeach Ġtoken Ġabove Ġto Ġits Ġword Ġembed ding Ġthen Ġuse Ġtwo ĠBi L ST Ms Ġ( with Ġshared Ġparameters ) Ġto Ġget Ġtheir Ġhidden Ġrepresentations Ġ$ [ \ math bf Ġ{ B } ^{ word } _{ 1 : M _ 1 } :\ math bf Ġ{ B } ^{ rel } _{ 1 : M _ 2 } ] $ Ġ( each Ġrow Ġvector Ġ$\ math bf Ġ{\ beta Ġ} _ i $ Ġis Ġthe Ġconc aten ation Ġbetween Ġforward / back ward Ġrepresentations Ġat Ġ$ i $ Ġ). ĠWe Ġinitialize Ġthe Ġrelation Ġsequence ĠL ST Ms Ġwith Ġthe Ġfinal Ġstate Ġrepresentations Ġof Ġthe Ġword Ġsequence , Ġas Ġa Ġback - off Ġfor Ġunseen Ġrelations . ĠWe Ġapply Ġone Ġmax - pool ing Ġon Ġthese Ġtwo Ġsets Ġof Ġvectors Ġand Ġget Ġthe Ġfinal Ġrelation Ġrepresentation Ġ$\ math bf Ġ{ h } ^ r $ Ġ. </s> Different ĠAbstract ions Ġof ĠQuestions ĠRepresent ations </s> From ĠTable Ġ1 Ġ, Ġwe Ġcan Ġsee Ġthat Ġdifferent Ġparts Ġof Ġa Ġrelation Ġcould Ġmatch Ġdifferent Ġcontexts Ġof Ġquestion Ġtexts . ĠUsually Ġrelation Ġnames Ġcould Ġmatch Ġlonger Ġphrases Ġin Ġthe Ġquestion Ġand Ġrelation Ġwords Ġcould Ġmatch Ġshort Ġphrases . ĠYet Ġdifferent Ġwords Ġmight Ġmatch Ġphrases Ġof Ġdifferent Ġlengths . </s> As Ġa Ġresult , Ġwe Ġhope Ġthe Ġquestion Ġrepresentations Ġcould Ġalso Ġcomprise Ġvectors Ġthat Ġsummarize Ġvarious Ġlengths Ġof Ġphrase Ġinformation Ġ( different Ġlevels Ġof Ġabstraction ), Ġin Ġorder Ġto Ġmatch Ġrelation Ġrepresentations Ġof Ġdifferent Ġgran ularity . ĠWe Ġdeal Ġwith Ġthis Ġproblem Ġby Ġapplying Ġdeep ĠBi L ST Ms Ġon Ġquestions . ĠThe Ġfirst - layer Ġof ĠBi L ST M Ġworks Ġon Ġthe Ġword Ġembed d ings Ġof Ġquestion Ġwords Ġ$\ math bf Ġ{ q } = \ l brace Ġq _ 1 , \ cd ots Ġ, q _ N \ r brace Ġ$ Ġand Ġgets Ġhidden Ġrepresentations Ġ$\ math bf Ġ{\ Gam ma Ġ} ^{ ( 1 )} _{ 1 : N } =[ \ math bf Ġ{\ gam ma Ġ} ^{ ( 1 )} _ 1 ; \ cd ots Ġ; \ math bf Ġ{\ gam ma Ġ} ^{ ( 1 )} _ N ] $ Ġ. ĠThe Ġsecond - layer ĠBi L ST M Ġworks Ġon Ġ$\ math bf Ġ{\ Gam ma Ġ} ^{ ( 1 )} _{ 1 : N } $ Ġto Ġget Ġthe Ġsecond Ġset Ġof Ġhidden Ġrepresentations Ġ$\ math bf Ġ{\ Gam ma Ġ} ^{ ( 2 )} _{ 1 : N } $ Ġ. ĠSince Ġthe Ġsecond ĠBi L ST M Ġstarts Ġwith Ġthe Ġhidden Ġvectors Ġfrom Ġthe Ġfirst Ġlayer , Ġintuitive ly Ġit Ġcould Ġlearn Ġmore Ġgeneral Ġand Ġabstract Ġinformation Ġcompared Ġto Ġthe Ġfirst Ġlayer . </s> Note Ġthat Ġthe Ġfirst ( second )- layer Ġof Ġquestion Ġrepresentations Ġdoes Ġnot Ġnecessarily Ġcorrespond Ġto Ġthe Ġword ( relation )- level Ġrelation Ġrepresentations , Ġinstead Ġeither Ġlayer Ġof Ġquestion Ġrepresentations Ġcould Ġpotentially Ġmatch Ġto Ġeither Ġlevel Ġof Ġrelation Ġrepresentations . ĠThis Ġraises Ġthe Ġdifficulty Ġof Ġmatching Ġbetween Ġdifferent Ġlevels Ġof Ġrelation / question Ġrepresentations ; Ġthe Ġfollowing Ġsection Ġgives Ġour Ġproposal Ġto Ġdeal Ġwith Ġsuch Ġproblem . </s> H ier arch ical ĠMatch ing Ġbetween ĠRel ation Ġand ĠQuestion </s> Now Ġwe Ġhave Ġquestion Ġcontexts Ġof Ġdifferent Ġlengths Ġencoded Ġin Ġ$\ math bf Ġ{\ Gam ma Ġ} ^{ ( 1 )} _{ 1 : N } $ Ġand Ġ$\ math bf Ġ{\ Gam ma Ġ} ^{ ( 2 )} _{ 1 : N } $ Ġ. ĠUnlike Ġthe Ġstandard Ġusage Ġof Ġdeep ĠBi L ST Ms Ġthat Ġemploys Ġthe Ġrepresentations Ġin Ġthe Ġfinal Ġlayer Ġfor Ġprediction , Ġhere Ġwe Ġexpect Ġthat Ġtwo Ġlayers Ġof Ġquestion Ġrepresentations Ġcan Ġbe Ġcomplementary Ġto Ġeach Ġother Ġand Ġboth Ġshould Ġbe Ġcompared Ġto Ġthe Ġrelation Ġrepresentation Ġspace Ġ( H ier arch ical ĠMatch ing ). ĠThis Ġis Ġimportant Ġfor Ġour Ġtask Ġsince Ġeach Ġrelation Ġtoken Ġcan Ġcorrespond Ġto Ġphrases Ġof Ġdifferent Ġlengths , Ġmainly Ġbecause Ġof Ġsynt actic Ġvariations . ĠFor Ġexample Ġin ĠTable Ġ1 Ġ, Ġthe Ġrelation Ġword Ġwritten Ġcould Ġbe Ġmatched Ġto Ġeither Ġthe Ġsame Ġsingle Ġword Ġin Ġthe Ġquestion Ġor Ġa Ġmuch Ġlonger Ġphrase Ġbe Ġthe Ġwriter Ġof . </s> We Ġcould Ġperform Ġthe Ġabove Ġhierarchical Ġmatching Ġby Ġcomputing Ġthe Ġsimilarity Ġbetween Ġeach Ġlayer Ġof Ġ$\ math bf Ġ{\ Gam ma Ġ} $ Ġand Ġ$\ math bf Ġ{ h } ^ r $ Ġseparately Ġand Ġdoing Ġthe Ġ( weight ed ) Ġsum Ġbetween Ġthe Ġtwo Ġscores . ĠHowever Ġthis Ġdoes Ġnot Ġgive Ġsignificant Ġimprovement Ġ( see ĠTable Ġ2 Ġ). ĠOur Ġanalysis Ġin ĠSection Ġ" Rel ation ĠDetection ĠResults " Ġshows Ġthat Ġthis Ġnaive Ġmethod Ġsuffers Ġfrom Ġthe Ġtraining Ġdifficulty , Ġevidenced Ġby Ġthat Ġthe Ġconver ged Ġtraining Ġloss Ġof Ġthis Ġmodel Ġis Ġmuch Ġhigher Ġthan Ġthat Ġof Ġa Ġsingle - layer Ġbaseline Ġmodel . ĠThis Ġis Ġmainly Ġbecause Ġ( 1 ) ĠDeep ĠBi L ST Ms Ġdo Ġnot Ġguarantee Ġthat Ġthe Ġtwo - levels Ġof Ġquestion Ġhidden Ġrepresentations Ġare Ġcomparable , Ġthe Ġtraining Ġusually Ġfalls Ġto Ġlocal Ġoptim a Ġwhere Ġone Ġlayer Ġhas Ġgood Ġmatching Ġscores Ġand Ġthe Ġother Ġalways Ġhas Ġweight Ġclose Ġto Ġ0 . Ġ( 2 ) ĠThe Ġtraining Ġof Ġdeeper Ġarchitectures Ġitself Ġis Ġmore Ġdifficult . </s> To Ġovercome Ġthe Ġabove Ġdifficulties , Ġwe Ġadopt Ġthe Ġidea Ġfrom ĠRes id ual ĠNetworks ĠB IB REF 23 Ġfor Ġhierarchical Ġmatching Ġby Ġadding Ġshortcut Ġconnections Ġbetween Ġtwo ĠBi L ST M Ġlayers . ĠWe Ġproposed Ġtwo Ġways Ġof Ġsuch ĠHier arch ical ĠRes id ual ĠMatch ing : Ġ( 1 ) ĠConnect ing Ġeach Ġ$\ math bf Ġ{\ gam ma Ġ} ^{ ( 1 )} _ i $ Ġand Ġ$\ math bf Ġ{\ gam ma Ġ} ^{ ( 2 )} _ i $ Ġ, Ġresulting Ġin Ġa Ġ$\ math bf Ġ{\ gam ma Ġ} ^{ ^ {\ prime Ġ}} _ i = \ math bf Ġ{\ gam ma Ġ} ^{ ( 1 )} _ i Ġ+ Ġ\ math bf Ġ{\ gam ma Ġ} ^{ ( 2 )} _ i $ Ġfor Ġeach Ġposition Ġ$ i $ Ġ. ĠThen Ġthe Ġfinal Ġquestion Ġrepresentation Ġ$\ math bf Ġ{ h } ^ q $ Ġbecomes Ġa Ġmax - pool ing Ġover Ġall Ġ$\ math bf Ġ{\ gam ma Ġ} ^{ ^ {\ prime Ġ}} _ i $ Ġs , Ġ1 Ġ$\ le Ġ$ Ġi Ġ$\ le Ġ$ Ġ$ N $ Ġ. Ġ( 2 ) ĠApp lying Ġmax - pool ing Ġon Ġ$\ math bf Ġ{\ Gam ma Ġ} ^{ ( 1 )} _{ 1 : N } $ Ġand Ġ$\ math bf Ġ{\ gam ma Ġ} ^{ ( 2 )} _ i $ 0 Ġto Ġget Ġ$\ math bf Ġ{\ gam ma Ġ} ^{ ( 2 )} _ i $ 1 Ġand Ġ$\ math bf Ġ{\ gam ma Ġ} ^{ ( 2 )} _ i $ 2 Ġ, Ġrespectively , Ġthen Ġsetting Ġ$\ math bf Ġ{\ gam ma Ġ} ^{ ( 2 )} _ i $ 3 Ġ. ĠFinally Ġwe Ġcompute Ġthe Ġmatching Ġscore Ġof Ġ$\ math bf Ġ{\ gam ma Ġ} ^{ ( 2 )} _ i $ 4 Ġgiven Ġ$\ math bf Ġ{\ gam ma Ġ} ^{ ( 2 )} _ i $ 5 Ġas Ġ$\ math bf Ġ{\ gam ma Ġ} ^{ ( 2 )} _ i $ 6 Ġ. </s> Int uitive ly , Ġthe Ġproposed Ġmethod Ġshould Ġbenefit Ġfrom Ġhierarchical Ġtraining Ġsince Ġthe Ġsecond Ġlayer Ġis Ġfitting Ġthe Ġresidues Ġfrom Ġthe Ġfirst Ġlayer Ġof Ġmatching , Ġso Ġthe Ġtwo Ġlayers Ġof Ġrepresentations Ġare Ġmore Ġlikely Ġto Ġbe Ġcomplementary Ġto Ġeach Ġother . ĠThis Ġalso Ġensures Ġthe Ġvector Ġspaces Ġof Ġtwo Ġlayers Ġare Ġcomparable Ġand Ġmakes Ġthe Ġsecond - layer Ġtraining Ġeasier . </s> During Ġtraining Ġwe Ġadopt Ġa Ġranking Ġloss Ġto Ġmaximizing Ġthe Ġmargin Ġbetween Ġthe Ġgold Ġrelation Ġ$\ math bf Ġ{ r } ^ + $ Ġand Ġother Ġrelations Ġ$\ math bf Ġ{ r } ^ -$ Ġin Ġthe Ġcandidate Ġpool Ġ$ R $ Ġ. </s> $$ l _{ \ math rm Ġ{ rel }} Ġ= Ġ\ max Ġ\ l brace Ġ0 , Ġ\ gam ma Ġ- Ġs _{ \ math rm Ġ{ rel }} (\ math bf Ġ{ r } ^ + ; Ġ\ math bf Ġ{ q }) Ġ+ Ġs _{ \ math rm Ġ{ rel }} (\ math bf Ġ{ r } ^ - ; Ġ\ math bf Ġ{ q } )\ r brace Ġ Ġ\ non umber Ġ$$ Ġ Ġ Ġ( E q . Ġ12 ) </s> where Ġ$\ gam ma Ġ$ Ġis Ġa Ġconstant Ġparameter . ĠFig Ġ2 Ġsummarizes Ġthe Ġabove ĠHier arch ical ĠRes id ual ĠBi L ST M Ġ( HR - Bi L ST M ) Ġmodel . </s> Another Ġway Ġof Ġhierarchical Ġmatching Ġconsists Ġin Ġrelying Ġon Ġattention Ġmechanism , Ġe . g . ĠB IB REF 24 Ġ, Ġto Ġfind Ġthe Ġcorrespondence Ġbetween Ġdifferent Ġlevels Ġof Ġrepresentations . ĠThis Ġperforms Ġbelow Ġthe ĠHR - Bi L ST M Ġ( see ĠTable Ġ2 Ġ). </s> KB Q A ĠEnhanced Ġby ĠRel ation ĠDetection </s> This Ġsection Ġdescribes Ġour ĠKB Q A Ġpipeline Ġsystem . ĠWe Ġmake Ġminimal Ġefforts Ġbeyond Ġthe Ġtraining Ġof Ġthe Ġrelation Ġdetection Ġmodel , Ġmaking Ġthe Ġwhole Ġsystem Ġeasy Ġto Ġbuild . </s> Following Ġprevious Ġwork ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, Ġour ĠKB Q A Ġsystem Ġtakes Ġan Ġexisting Ġentity Ġlink er Ġto Ġproduce Ġthe Ġtop - Ġ$ K $ Ġlinked Ġentities , Ġ$ EL _ K ( q ) $ Ġ, Ġfor Ġa Ġquestion Ġ$ q $ Ġ( âĢ ľ initial Ġentity Ġlinking âĢ Ŀ ). ĠThen Ġwe Ġgenerate Ġthe ĠKB Ġqueries Ġfor Ġ$ q $ Ġfollowing Ġthe Ġfour Ġsteps Ġillustrated Ġin ĠAl gorithm Ġ" KB Q A ĠEnhanced Ġby ĠRel ation ĠDetection " Ġ. </s> [ ht bp ] ĠInput Input ĠOutput Output ĠTop Ġquery Ġtuple Ġ$( \ hat { e }, \ hat { r }, Ġ\ l brace Ġ( c , Ġr _ c )\ r brace Ġ) $ ĠEntity ĠRe - R anking Ġ( first - step Ġrelation Ġdetection ): ĠUse Ġthe Ġraw Ġquestion Ġtext Ġas Ġinput Ġfor Ġa Ġrelation Ġdetector Ġto Ġscore Ġall Ġrelations Ġin Ġthe ĠKB Ġthat Ġare Ġassociated Ġto Ġthe Ġentities Ġin Ġ$ EL _ K ( q ) $ Ġ; Ġuse Ġthe Ġrelation Ġscores Ġto Ġre - rank Ġ$ EL _ K ( q ) $ Ġand Ġgenerate Ġa Ġshorter Ġlist Ġ$ EL ^ {\ prime Ġ} _{ K ^ {\ prime Ġ}} ( q ) $ Ġcontaining Ġthe Ġtop - Ġ$ K ^ {\ prime Ġ} $ Ġentity Ġcandidates Ġ( Section Ġ" Entity ĠRe - R anking " Ġ) ĠRel ation ĠDetection : ĠDetect Ġrelation ( s ) Ġusing Ġthe Ġreform atted Ġquestion Ġtext Ġin Ġwhich Ġthe Ġtopic Ġentity Ġis Ġreplaced Ġby Ġa Ġspecial Ġtoken Ġ$ < $ Ġe Ġ$ > $ Ġ( Section Ġ" Rel ation ĠDetection " Ġ) ĠQuery ĠGeneration : ĠCombine Ġthe Ġscores Ġfrom Ġstep Ġ1 Ġand Ġ2 , Ġand Ġselect Ġthe Ġtop Ġpair Ġ$( \ hat { e }, \ hat { r }) $ Ġ( Section Ġ" Query ĠGeneration " Ġ) ĠCon str aint ĠDetection Ġ( optional ): ĠComp ute Ġsimilarity Ġbetween Ġ$ q $ Ġand Ġany Ġneighbor Ġentity Ġ$ c $ Ġof Ġthe Ġentities Ġalong Ġ$ EL _ K ( q ) $ 0 Ġ( connect ing Ġby Ġa Ġrelation Ġ$ EL _ K ( q ) $ 1 Ġ) Ġ, Ġadd Ġthe Ġhigh Ġscoring Ġ$ EL _ K ( q ) $ 2 Ġand Ġ$ EL _ K ( q ) $ 3 Ġto Ġthe Ġquery Ġ( Section Ġ" Con str aint ĠDetection " Ġ). ĠKB Q A Ġwith Ġtwo - step Ġrelation Ġdetection </s> Compared Ġto Ġprevious Ġapproaches , Ġthe Ġmain Ġdifference Ġis Ġthat Ġwe Ġhave Ġan Ġadditional Ġentity Ġre - ranking Ġstep Ġafter Ġthe Ġinitial Ġentity Ġlinking . ĠWe Ġhave Ġthis Ġstep Ġbecause Ġwe Ġhave Ġobserved Ġthat Ġentity Ġlinking Ġsometimes Ġbecomes Ġa Ġbottleneck Ġin ĠKB Q A Ġsystems . ĠFor Ġexample , Ġon ĠSimple Questions Ġthe Ġbest Ġreported Ġlink er Ġcould Ġonly Ġget Ġ72 . 7 % Ġtop - 1 Ġaccuracy Ġon Ġidentifying Ġtopic Ġentities . ĠThis Ġis Ġusually Ġdue Ġto Ġthe Ġambig u ities Ġof Ġentity Ġnames , Ġe . g . Ġin ĠFig Ġ1 Ġ( a ), Ġthere Ġare ĠTV Ġwriter Ġand Ġbaseball Ġplayer ĠâĢ ľ Mike ĠKelley âĢ Ŀ , Ġwhich Ġis Ġimpossible Ġto Ġdistinguish Ġwith Ġonly Ġentity Ġname Ġmatching . </s> Having Ġobserved Ġthat Ġdifferent Ġentity Ġcandidates Ġusually Ġconnect Ġto Ġdifferent Ġrelations , Ġhere Ġwe Ġpropose Ġto Ġhelp Ġentity Ġdis amb ig uation Ġin Ġthe Ġinitial Ġentity Ġlinking Ġwith Ġrelations Ġdetected Ġin Ġquestions . </s> S ect ions Ġ" Entity ĠRe - R anking " Ġand Ġ" Rel ation ĠDetection " Ġelaborate Ġhow Ġour Ġrelation Ġdetection Ġhelp Ġto Ġre - rank Ġentities Ġin Ġthe Ġinitial Ġentity Ġlinking , Ġand Ġthen Ġthose Ġre - ranked Ġentities Ġenable Ġmore Ġaccurate Ġrelation Ġdetection . ĠThe ĠKB Q A Ġend Ġtask , Ġas Ġa Ġresult , Ġbenefits Ġfrom Ġthis Ġprocess . </s> Entity ĠRe - R anking </s> In Ġthis Ġstep , Ġwe Ġuse Ġthe Ġraw Ġquestion Ġtext Ġas Ġinput Ġfor Ġa Ġrelation Ġdetector Ġto Ġscore Ġall Ġrelations Ġin Ġthe ĠKB Ġwith Ġconnections Ġto Ġat Ġleast Ġone Ġof Ġthe Ġentity Ġcandidates Ġin Ġ$ EL _ K ( q ) $ Ġ. ĠWe Ġcall Ġthis Ġstep Ġrelation Ġdetection Ġon Ġentity Ġset Ġsince Ġit Ġdoes Ġnot Ġwork Ġon Ġa Ġsingle Ġtopic Ġentity Ġas Ġthe Ġusual Ġsettings . ĠWe Ġuse Ġthe ĠHR - Bi L ST M Ġas Ġdescribed Ġin ĠSec . Ġ" Improved ĠKB ĠRel ation ĠDetection " Ġ. ĠFor Ġeach Ġquestion Ġ$ q $ Ġ, Ġafter Ġgenerating Ġa Ġscore Ġ$ s _{ rel } ( r ; q ) $ Ġfor Ġeach Ġrelation Ġusing ĠHR - Bi L ST M , Ġwe Ġuse Ġthe Ġtop Ġ$ l $ Ġbest Ġscoring Ġrelations Ġ( Ġ$ R ^{ l } _ q $ Ġ) Ġto Ġre - rank Ġthe Ġoriginal Ġentity Ġcandidates . ĠCon crete ly , Ġfor Ġeach Ġentity Ġ$ e $ Ġand Ġits Ġassociated Ġrelations Ġ$ R _ e $ Ġ, Ġgiven Ġthe Ġoriginal Ġentity Ġlink er Ġscore Ġ$ s _{ link er } $ Ġ, Ġand Ġthe Ġscore Ġof Ġthe Ġmost Ġconfident Ġrelation Ġ$ r \ in ĠR _ q ^{ l } Ġ\ cap ĠR _ e $ Ġ, Ġwe Ġsum Ġthese Ġtwo Ġscores Ġto Ġre - rank Ġthe Ġentities : </s> $$ s _{ \ math rm Ġ{ rer ank }} ( e ; q ) Ġ= & Ġ\ alpha Ġ\ cd ot Ġs _{ \ math rm Ġ{ link er }} ( e ; q ) Ġ\ non umber Ġ\\ Ġ+ Ġ& Ġ( 1 - \ alpha Ġ) Ġ\ cd ot Ġ\ max Ġ_ { r Ġ\ in ĠR _ q ^{ l } Ġ\ cap ĠR _ e } Ġs _{ \ math rm Ġ{ rel }} ( r ; q ). \ non umber Ġ$$ Ġ Ġ Ġ( E q . Ġ15 ) </s> Finally , Ġwe Ġselect Ġtop Ġ$ K ^ {\ prime Ġ} $ Ġ$ < $ Ġ$ K $ Ġentities Ġaccording Ġto Ġscore Ġ$ s _{ rer ank } $ Ġto Ġform Ġthe Ġre - ranked Ġlist Ġ$ EL _{ K ^ {\ prime Ġ}} ^{ ^ {\ prime Ġ}} ( q ) $ Ġ. </s> We Ġuse Ġthe Ġsame Ġexample Ġin ĠFig Ġ1 Ġ( a ) Ġto Ġillustrate Ġthe Ġidea . ĠGiven Ġthe Ġinput Ġquestion Ġin Ġthe Ġexample , Ġa Ġrelation Ġdetector Ġis Ġvery Ġlikely Ġto Ġassign Ġhigh Ġscores Ġto Ġrelations Ġsuch Ġas ĠâĢ ľ ep isodes _ written âĢ Ŀ , ĠâĢ ľ author _ of âĢ Ŀ Ġand ĠâĢ ľ prof ession âĢ Ŀ . ĠThen , Ġaccording Ġto Ġthe Ġconnections Ġof Ġentity Ġcandidates Ġin ĠKB , Ġwe Ġfind Ġthat Ġthe ĠTV Ġwriter ĠâĢ ľ Mike ĠKelley âĢ Ŀ Ġwill Ġbe Ġscored Ġhigher Ġthan Ġthe Ġbaseball Ġplayer ĠâĢ ľ Mike ĠKelley âĢ Ŀ , Ġbecause Ġthe Ġformer Ġhas Ġthe Ġrelations ĠâĢ ľ ep isodes _ written âĢ Ŀ Ġand ĠâĢ ľ prof ession âĢ Ŀ . ĠThis Ġmethod Ġcan Ġbe Ġviewed Ġas Ġexploiting Ġentity - relation Ġcoll ocation Ġfor Ġentity Ġlinking . </s> Rel ation ĠDetection </s> In Ġthis Ġstep , Ġfor Ġeach Ġcandidate Ġentity Ġ$ e Ġ\ in ĠEL _ K ^ {\ prime Ġ} ( q ) $ Ġ, Ġwe Ġuse Ġthe Ġquestion Ġtext Ġas Ġthe Ġinput Ġto Ġa Ġrelation Ġdetector Ġto Ġscore Ġall Ġthe Ġrelations Ġ$ r Ġ\ in ĠR _ e $ Ġthat Ġare Ġassociated Ġto Ġthe Ġentity Ġ$ e $ Ġin Ġthe ĠKB . ĠBecause Ġwe Ġhave Ġa Ġsingle Ġtopic Ġentity Ġinput Ġin Ġthis Ġstep , Ġwe Ġdo Ġthe Ġfollowing Ġquestion Ġreform at ting : Ġwe Ġreplace Ġthe Ġthe Ġcandidate Ġ$ e $ Ġ' s Ġentity Ġmention Ġin Ġ$ q $ Ġwith Ġa Ġtoken ĠâĢ ľ Ġ$ < $ Ġe Ġ$ > $ ĠâĢ Ŀ . ĠThis Ġhelps Ġthe Ġmodel Ġbetter Ġdistinguish Ġthe Ġrelative Ġposition Ġof Ġeach Ġword Ġcompared Ġto Ġthe Ġentity . ĠWe Ġuse Ġthe ĠHR - Bi L ST M Ġmodel Ġto Ġpredict Ġthe Ġscore Ġof Ġeach Ġrelation Ġ$ r Ġ\ in ĠR _ e $ Ġ: Ġ$ s _{ rel } Ġ( r ; e , q ) $ Ġ. </s> Query ĠGeneration </s> Finally , Ġthe Ġsystem Ġoutputs Ġthe Ġ$ < $ Ġentity , Ġrelation Ġ( or Ġcore - chain ) Ġ$ > $ Ġpair Ġ$( \ hat { e }, Ġ\ hat { r }) $ Ġaccording Ġto : </s> $$ s (\ hat { e }, Ġ\ hat { r }; Ġq ) Ġ= & Ġ\ max Ġ_ { e Ġ\ in ĠEL _{ K ^ {\ prime Ġ}} ^{ ^ {\ prime Ġ}} ( q ), Ġr Ġ\ in ĠR _ e } Ġ\ left ( Ġ\ beta Ġ\ cd ot Ġs _{ \ math rm Ġ{ rer ank }} ( e ; q ) Ġ\ right . Ġ\ non umber Ġ\\ Ġ& \ left . + Ġ( 1 - \ beta Ġ) Ġ\ cd ot Ġs _{ \ math rm Ġ{ rel }} Ġ( r ; e , q ) Ġ\ right ), Ġ\ non umber Ġ$$ Ġ Ġ Ġ( E q . Ġ19 ) </s> where Ġ$\ beta Ġ$ Ġis Ġa Ġhyper param eter Ġto Ġbe Ġtuned . </s> Con str aint ĠDetection </s> Similar Ġto ĠB IB REF 4 Ġ, Ġwe Ġadopt Ġan Ġadditional Ġconstraint Ġdetection Ġstep Ġbased Ġon Ġtext Ġmatching . ĠOur Ġmethod Ġcan Ġbe Ġviewed Ġas Ġentity - l inking Ġon Ġa ĠKB Ġsub - graph . ĠIt Ġcontains Ġtwo Ġsteps : Ġ( 1 ) ĠSub - graph Ġgeneration : Ġgiven Ġthe Ġtop Ġscored Ġquery Ġgenerated Ġby Ġthe Ġprevious Ġ3 Ġsteps , Ġfor Ġeach Ġnode Ġ$ v $ Ġ( answer Ġnode Ġor Ġthe ĠCV T Ġnode Ġlike Ġin ĠFigure Ġ1 Ġ( b )), Ġwe Ġcollect Ġall Ġthe Ġnodes Ġ$ c $ Ġconnecting Ġto Ġ$ v $ Ġ( with Ġrelation Ġ$ r _ c $ Ġ) Ġwith Ġany Ġrelation , Ġand Ġgenerate Ġa Ġsub - graph Ġassociated Ġto Ġthe Ġoriginal Ġquery . Ġ( 2 ) ĠEntity - l inking Ġon Ġsub - graph Ġnodes : Ġwe Ġcompute Ġa Ġmatching Ġscore Ġbetween Ġeach Ġ$ n $ Ġ- gram Ġin Ġthe Ġinput Ġquestion Ġ( without Ġoverlapping Ġthe Ġtopic Ġentity ) Ġand Ġentity Ġname Ġof Ġ$ c $ Ġ( except Ġfor Ġthe Ġnode Ġin Ġthe Ġoriginal Ġquery ) Ġby Ġtaking Ġinto Ġaccount Ġthe Ġmaximum Ġoverlapping Ġsequence Ġof Ġcharacters Ġbetween Ġthem Ġ( see ĠAppendix ĠA Ġfor Ġdetails Ġand ĠB Ġfor Ġspecial Ġrules Ġdealing Ġwith Ġdate / answer Ġtype Ġconstraints ). ĠIf Ġthe Ġmatching Ġscore Ġis Ġlarger Ġthan Ġa Ġthreshold Ġ$\ the ta Ġ$ Ġ( tun ed Ġon Ġtraining Ġset ), Ġwe Ġwill Ġadd Ġthe Ġconstraint Ġentity Ġ$ c $ Ġ( and Ġ$ r _ c $ Ġ) Ġto Ġthe Ġquery Ġby Ġattaching Ġit Ġto Ġthe Ġcorresponding Ġnode Ġ$ v $ Ġon Ġthe Ġcore - chain . </s> Exper iments </s> Task ĠIntroduction Ġ& ĠSettings </s> We Ġuse Ġthe ĠSimple Questions ĠB IB REF 2 Ġand ĠWeb Q SP ĠB IB REF 25 Ġdatasets . ĠEach Ġquestion Ġin Ġthese Ġdatasets Ġis Ġlabeled Ġwith Ġthe Ġgold Ġsemantic Ġparse . ĠHence Ġwe Ġcan Ġdirectly Ġevaluate Ġrelation Ġdetection Ġperformance Ġindependently Ġas Ġwell Ġas Ġevaluate Ġon Ġthe ĠKB Q A Ġend Ġtask . </s> Simple Questions Ġ( S Q ): ĠIt Ġis Ġa Ġsingle - relation ĠKB Q A Ġtask . ĠThe ĠKB Ġwe Ġuse Ġconsists Ġof Ġa ĠFree base Ġsubset Ġwith Ġ2 M Ġentities Ġ( FB 2 M ) ĠB IB REF 2 Ġ, Ġin Ġorder Ġto Ġcompare Ġwith Ġprevious Ġresearch . Ġy in 2016 simple Ġalso Ġevaluated Ġtheir Ġrelation Ġextract or Ġon Ġthis Ġdata Ġset Ġand Ġreleased Ġtheir Ġproposed Ġquestion - relation Ġpairs , Ġso Ġwe Ġrun Ġour Ġrelation Ġdetection Ġmodel Ġon Ġtheir Ġdata Ġset . ĠFor Ġthe ĠKB Q A Ġevaluation , Ġwe Ġalso Ġstart Ġwith Ġtheir Ġentity Ġlinking Ġresults . ĠTherefore , Ġour Ġresults Ġcan Ġbe Ġcompared Ġwith Ġtheir Ġreported Ġresults Ġon Ġboth Ġtasks . </s> Web Q SP Ġ( W Q ): ĠA Ġmulti - relation ĠKB Q A Ġtask . ĠWe Ġuse Ġthe Ġentire ĠFree base ĠKB Ġfor Ġevaluation Ġpurposes . ĠFollowing Ġy ih - E t Al : 2016 : P 16 - 2 , Ġwe Ġuse ĠS - M ART ĠB IB REF 26 Ġentity - l inking Ġoutputs . ĠIn Ġorder Ġto Ġevaluate Ġthe Ġrelation Ġdetection Ġmodels , Ġwe Ġcreate Ġa Ġnew Ġrelation Ġdetection Ġtask Ġfrom Ġthe ĠWeb Q SP Ġdata Ġset . ĠFor Ġeach Ġquestion Ġand Ġits Ġlabeled Ġsemantic Ġparse : Ġ( 1 ) Ġwe Ġfirst Ġselect Ġthe Ġtopic Ġentity Ġfrom Ġthe Ġparse ; Ġand Ġthen Ġ( 2 ) Ġselect Ġall Ġthe Ġrelations Ġand Ġrelation Ġchains Ġ( length Ġ$\ le Ġ$ Ġ2 ) Ġconnected Ġto Ġthe Ġtopic Ġentity , Ġand Ġset Ġthe Ġcore - chain Ġlabeled Ġin Ġthe Ġparse Ġas Ġthe Ġpositive Ġlabel Ġand Ġall Ġthe Ġothers Ġas Ġthe Ġnegative Ġexamples . </s> We Ġtune Ġthe Ġfollowing Ġhyper - param eters Ġon Ġdevelopment Ġsets : Ġ( 1 ) Ġthe Ġsize Ġof Ġhidden Ġstates Ġfor ĠL ST Ms Ġ({ 50 , Ġ100 , Ġ200 , Ġ400 }); Ġ( 2 ) Ġlearning Ġrate Ġ({ 0 . 1 , Ġ0 . 5 , Ġ1 . 0 , Ġ2 . 0 }); Ġ( 3 ) Ġwhether Ġthe Ġshortcut Ġconnections Ġare Ġbetween Ġhidden Ġstates Ġor Ġbetween Ġmax - pool ing Ġresults Ġ( see ĠSection Ġ" H ier arch ical ĠMatch ing Ġbetween ĠRel ation Ġand ĠQuestion " Ġ); Ġand Ġ( 4 ) Ġthe Ġnumber Ġof Ġtraining Ġepoch s . </s> For Ġboth Ġthe Ġrelation Ġdetection Ġexperiments Ġand Ġthe Ġsecond - step Ġrelation Ġdetection Ġin ĠKB Q A , Ġwe Ġhave Ġentity Ġreplacement Ġfirst Ġ( see ĠSection Ġ" Rel ation ĠDetection " Ġand ĠFigure Ġ1 Ġ). ĠAll Ġword Ġvectors Ġare Ġinitialized Ġwith Ġ300 - Ġ$ d $ Ġpret rained Ġword Ġembed d ings ĠB IB REF 27 Ġ. ĠThe Ġembed d ings Ġof Ġrelation Ġnames Ġare Ġrandomly Ġinitialized , Ġsince Ġexisting Ġpre - trained Ġrelation Ġembed d ings Ġ( e . g . ĠTrans E ) Ġusually Ġsupport Ġlimited Ġsets Ġof Ġrelation Ġnames . ĠWe Ġleave Ġthe Ġusage Ġof Ġpre - trained Ġrelation Ġembed d ings Ġto Ġfuture Ġwork . </s> Rel ation ĠDetection ĠResults </s> Table Ġ2 Ġshows Ġthe Ġresults Ġon Ġtwo Ġrelation Ġdetection Ġtasks . ĠThe ĠAM PC NN Ġresult Ġis Ġfrom ĠB IB REF 20 Ġ, Ġwhich Ġyielded Ġstate - of - the - art Ġscores Ġby Ġoutper forming Ġseveral Ġattention - based Ġmethods . ĠWe Ġre - im ple mented Ġthe ĠBi CNN Ġmodel Ġfrom ĠB IB REF 4 Ġ, Ġwhere Ġboth Ġquestions Ġand Ġrelations Ġare Ġrepresented Ġwith Ġthe Ġword Ġhash Ġtrick Ġon Ġcharacter Ġtri - gram s . ĠThe Ġbaseline ĠBi L ST M Ġwith Ġrelation Ġword Ġsequence Ġappears Ġto Ġbe Ġthe Ġbest Ġbaseline Ġon ĠWeb Q SP Ġand Ġis Ġclose Ġto Ġthe Ġprevious Ġbest Ġresult Ġof ĠAM PC NN Ġon ĠSimple Questions . ĠOur Ġproposed ĠHR - Bi L ST M Ġoutper formed Ġthe Ġbest Ġbas elines Ġon Ġboth Ġtasks Ġby Ġmargins Ġof Ġ2 - 3 % Ġ( p Ġ$ < $ Ġ0 . 001 Ġand Ġ0 . 01 Ġcompared Ġto Ġthe Ġbest Ġbaseline ĠBi L ST M Ġw / Ġwords Ġon ĠSQ Ġand ĠW Q Ġrespectively ). </s> Note Ġthat Ġusing Ġonly Ġrelation Ġnames Ġinstead Ġof Ġwords Ġresults Ġin Ġa Ġweaker Ġbaseline ĠBi L ST M Ġmodel . ĠThe Ġmodel Ġyields Ġa Ġsignificant Ġperformance Ġdrop Ġon ĠSimple Questions Ġ( 91 . 2 % Ġto Ġ88 . 9 %). ĠHowever , Ġthe Ġdrop Ġis Ġmuch Ġsmaller Ġon ĠWeb Q SP , Ġand Ġit Ġsuggests Ġthat Ġunseen Ġrelations Ġhave Ġa Ġmuch Ġbigger Ġimpact Ġon ĠSimple Questions . </s> The Ġbottom Ġof ĠTable Ġ2 Ġshows Ġab lation Ġresults Ġof Ġthe Ġproposed ĠHR - Bi L ST M . ĠFirst , Ġhierarchical Ġmatching Ġbetween Ġquestions Ġand Ġboth Ġrelation Ġnames Ġand Ġrelation Ġwords Ġyields Ġimprovement Ġon Ġboth Ġdatasets , Ġespecially Ġfor ĠSimple Questions Ġ( 93 . 3 % Ġvs . Ġ91 . 2 / 88 . 8 %). ĠSecond , Ġresidual Ġlearning Ġhelps Ġhierarchical Ġmatching Ġcompared Ġto Ġweighted - sum Ġand Ġattention - based Ġbas elines Ġ( see ĠSection Ġ" H ier arch ical ĠMatch ing Ġbetween ĠRel ation Ġand ĠQuestion " Ġ). ĠFor Ġthe Ġattention - based Ġbaseline , Ġwe Ġtried Ġthe Ġmodel Ġfrom ĠB IB REF 24 Ġand Ġits Ġone - way Ġvariations , Ġwhere Ġthe Ġone - way Ġmodel Ġgives Ġbetter Ġresults . ĠNote Ġthat Ġresidual Ġlearning Ġsignificantly Ġhelps Ġon ĠWeb Q SP Ġ( 80 . 65 % Ġto Ġ82 . 53 %), Ġwhile Ġit Ġdoes Ġnot Ġhelp Ġas Ġmuch Ġon ĠSimple Questions . ĠOn ĠSimple Questions , Ġeven Ġremoving Ġthe Ġdeep Ġlayers Ġonly Ġcauses Ġa Ġsmall Ġdrop Ġin Ġperformance . ĠWeb Q SP Ġbenefits Ġmore Ġfrom Ġresidual Ġand Ġdeeper Ġarchitecture , Ġpossibly Ġbecause Ġin Ġthis Ġdataset Ġit Ġis Ġmore Ġimportant Ġto Ġhandle Ġlarger Ġscope Ġof Ġcontext Ġmatching . </s> Finally , Ġon ĠWeb Q SP , Ġreplacing ĠBi L ST M Ġwith ĠCNN Ġin Ġour Ġhierarchical Ġmatching Ġframework Ġresults Ġin Ġa Ġlarge Ġperformance Ġdrop . ĠYet Ġon ĠSimple Questions Ġthe Ġgap Ġis Ġmuch Ġsmaller . ĠWe Ġbelieve Ġthis Ġis Ġbecause Ġthe ĠL ST M Ġrelation Ġenc oder Ġcan Ġbetter Ġlearn Ġthe Ġcomposition Ġof Ġchains Ġof Ġrelations Ġin ĠWeb Q SP , Ġas Ġit Ġis Ġbetter Ġat Ġdealing Ġwith Ġlonger Ġdependencies . </s> Next , Ġwe Ġpresent Ġempirical Ġevid ences , Ġwhich Ġshow Ġwhy Ġour ĠHR - Bi L ST M Ġmodel Ġachieves Ġthe Ġbest Ġscores . ĠWe Ġuse ĠWeb Q SP Ġfor Ġthe Ġanalysis Ġpurposes . ĠFirst , Ġwe Ġhave Ġthe Ġhypothesis Ġthat Ġtraining Ġof Ġthe Ġweighted - sum Ġmodel Ġusually Ġfalls Ġto Ġlocal Ġoptim a , Ġsince Ġdeep ĠBi L ST Ms Ġdo Ġnot Ġguarantee Ġthat Ġthe Ġtwo - levels Ġof Ġquestion Ġhidden Ġrepresentations Ġare Ġcomparable . ĠThis Ġis Ġevidenced Ġby Ġthat Ġduring Ġtraining Ġone Ġlayer Ġusually Ġgets Ġa Ġweight Ġclose Ġto Ġ0 Ġthus Ġis Ġignored . ĠFor Ġexample , Ġone Ġrun Ġgives Ġus Ġweights Ġof Ġ- 75 . 39 / 0 . 14 Ġfor Ġthe Ġtwo Ġlayers Ġ( we Ġtake Ġexponential Ġfor Ġthe Ġfinal Ġweighted Ġsum ). ĠIt Ġalso Ġgives Ġmuch Ġlower Ġtraining Ġaccuracy Ġ( 91 . 94 %) Ġcompared Ġto ĠHR - Bi L ST M Ġ( 95 . 67 %), Ġsuffering Ġfrom Ġtraining Ġdifficulty . </s> Second , Ġcompared Ġto Ġour Ġdeep ĠBi L ST M Ġwith Ġshortcut Ġconnections , Ġwe Ġhave Ġthe Ġhypothesis Ġthat Ġfor ĠKB Ġrelation Ġdetection , Ġtraining Ġdeep ĠBi L ST Ms Ġis Ġmore Ġdifficult Ġwithout Ġshortcut Ġconnections . ĠOur Ġexperiments Ġsuggest Ġthat Ġdeeper ĠBi L ST M Ġdoes Ġnot Ġalways Ġresult Ġin Ġlower Ġtraining Ġaccuracy . ĠIn Ġthe Ġexperiments Ġa Ġtwo - layer ĠBi L ST M Ġconver ges Ġto Ġ94 . 99 %, Ġeven Ġlower Ġthan Ġthe Ġ95 . 25 % Ġachieved Ġby Ġa Ġsingle - layer ĠBi L ST M . ĠUnder Ġour Ġsetting Ġthe Ġtwo - layer Ġmodel Ġcaptures Ġthe Ġsingle - layer Ġmodel Ġas Ġa Ġspecial Ġcase Ġ( so Ġit Ġcould Ġpotentially Ġbetter Ġfit Ġthe Ġtraining Ġdata ), Ġthis Ġresult Ġsuggests Ġthat Ġthe Ġdeep ĠBi L ST M Ġwithout Ġshortcut Ġconnections Ġmight Ġsuffers Ġmore Ġfrom Ġtraining Ġdifficulty . </s> Finally , Ġwe Ġhypothes ize Ġthat ĠHR - Bi L ST M Ġis Ġmore Ġthan Ġcombination Ġof Ġtwo ĠBi L ST Ms Ġwith Ġresidual Ġconnections , Ġbecause Ġit Ġencourages Ġthe Ġhierarchical Ġarchitecture Ġto Ġlearn Ġdifferent Ġlevels Ġof Ġabstraction . ĠTo Ġverify Ġthis , Ġwe Ġreplace Ġthe Ġdeep ĠBi L ST M Ġquestion Ġenc oder Ġwith Ġtwo Ġsingle - layer ĠBi L ST Ms Ġ( both Ġon Ġwords ) Ġwith Ġshortcut Ġconnections Ġbetween Ġtheir Ġhidden Ġstates . ĠThis Ġdecreases Ġtest Ġaccuracy Ġto Ġ76 . 11 %. ĠIt Ġgives Ġsimilar Ġtraining Ġaccuracy Ġcompared Ġto ĠHR - Bi L ST M , Ġindicating Ġa Ġmore Ġserious Ġover - fitting Ġproblem . ĠThis Ġproves Ġthat Ġthe Ġresidual Ġand Ġdeep Ġstructures Ġboth Ġcontribute Ġto Ġthe Ġgood Ġperformance Ġof ĠHR - Bi L ST M . </s> KB Q A ĠEnd - Task ĠResults </s> Table Ġ3 Ġcompares Ġour Ġsystem Ġwith Ġtwo Ġpublished Ġbas elines Ġ( 1 ) ĠST AG G ĠB IB REF 4 Ġ, Ġthe Ġstate - of - the - art Ġon ĠWeb Q SP Ġand Ġ( 2 ) ĠAM PC NN ĠB IB REF 20 Ġ, Ġthe Ġstate - of - the - art Ġon ĠSimple Questions . ĠSince Ġthese Ġtwo Ġbas elines Ġare Ġspecially Ġdesigned / tun ed Ġfor Ġone Ġparticular Ġdataset , Ġthey Ġdo Ġnot Ġgeneral ize Ġwell Ġwhen Ġapplied Ġto Ġthe Ġother Ġdataset . ĠIn Ġorder Ġto Ġhighlight Ġthe Ġeffect Ġof Ġdifferent Ġrelation Ġdetection Ġmodels Ġon Ġthe ĠKB Q A Ġend - task , Ġwe Ġalso Ġimplemented Ġanother Ġbaseline Ġthat Ġuses Ġour ĠKB Q A Ġsystem Ġbut Ġreplaces ĠHR - Bi L ST M Ġwith Ġour Ġimplementation Ġof ĠAM PC NN Ġ( for ĠSimple Questions ) Ġor Ġthe Ġchar - 3 - gram ĠBi CNN Ġ( for ĠWeb Q SP ) Ġrelation Ġdetectors Ġ( second Ġblock Ġin ĠTable Ġ3 Ġ). </s> Compared Ġto Ġthe Ġbaseline Ġrelation Ġdetector Ġ( 3 rd Ġrow Ġof Ġresults ), Ġour Ġmethod , Ġwhich Ġincludes Ġan Ġimproved Ġrelation Ġdetector Ġ( HR - Bi L ST M ), Ġimproves Ġthe ĠKB Q A Ġend Ġtask Ġby Ġ2 - 3 % Ġ( 4 th Ġrow ). ĠNote Ġthat Ġin Ġcontrast Ġto Ġprevious ĠKB Q A Ġsystems , Ġour Ġsystem Ġdoes Ġnot Ġuse Ġjoint - in ference Ġor Ġfeature - based Ġre - ranking Ġstep , Ġnevertheless Ġit Ġstill Ġachieves Ġbetter Ġor Ġcomparable Ġresults Ġto Ġthe Ġstate - of - the - art . </s> The Ġthird Ġblock Ġof Ġthe Ġtable Ġdetails Ġtwo Ġab lation Ġtests Ġfor Ġthe Ġproposed Ġcomponents Ġin Ġour ĠKB Q A Ġsystems : Ġ( 1 ) ĠRem oving Ġthe Ġentity Ġre - ranking Ġstep Ġsignificantly Ġdecreases Ġthe Ġscores . ĠSince Ġthe Ġre - ranking Ġstep Ġrelies Ġon Ġthe Ġrelation Ġdetection Ġmodels , Ġthis Ġshows Ġthat Ġour ĠHR - Bi L ST M Ġmodel Ġcontributes Ġto Ġthe Ġgood Ġperformance Ġin Ġmultiple Ġways . ĠAppendix ĠC Ġgives Ġthe Ġdetailed Ġperformance Ġof Ġthe Ġre - ranking Ġstep . Ġ( 2 ) ĠIn Ġcontrast Ġto Ġthe Ġconclusion Ġin ĠB IB REF 4 Ġ, Ġconstraint Ġdetection Ġis Ġcrucial Ġfor Ġour Ġsystem . ĠThis Ġis Ġprobably Ġbecause Ġour Ġjoint Ġperformance Ġon Ġtopic Ġentity Ġand Ġcore - chain Ġdetection Ġis Ġmore Ġaccurate Ġ( 77 . 5 % Ġtop - 1 Ġaccuracy ), Ġleaving Ġa Ġhuge Ġpotential Ġ( 77 . 5 % Ġvs . Ġ58 . 0 %) Ġfor Ġthe Ġconstraint Ġdetection Ġmodule Ġto Ġimprove . </s> Finally , Ġlike ĠST AG G , Ġwhich Ġuses Ġmultiple Ġrelation Ġdetectors Ġ( see Ġy ih 2015 sem antic Ġfor Ġthe Ġthree Ġmodels Ġused ), Ġwe Ġalso Ġtry Ġto Ġuse Ġthe Ġtop - 3 Ġrelation Ġdetectors Ġfrom ĠSection Ġ" Rel ation ĠDetection ĠResults " Ġ. ĠAs Ġshown Ġon Ġthe Ġlast Ġrow Ġof ĠTable Ġ3 Ġ, Ġthis Ġgives Ġa Ġsignificant Ġperformance Ġboost , Ġresulting Ġin Ġa Ġnew Ġstate - of - the - art Ġresult Ġon ĠSimple Questions Ġand Ġa Ġresult Ġcomparable Ġto Ġthe Ġstate - of - the - art Ġon ĠWeb Q SP . </s> Conclusion </s> KB Ġrelation Ġdetection Ġis Ġa Ġkey Ġstep Ġin ĠKB Q A Ġand Ġis Ġsignificantly Ġdifferent Ġfrom Ġgeneral Ġrelation Ġextraction Ġtasks . ĠWe Ġpropose Ġa Ġnovel ĠKB Ġrelation Ġdetection Ġmodel , ĠHR - Bi L ST M , Ġthat Ġperforms Ġhierarchical Ġmatching Ġbetween Ġquestions Ġand ĠKB Ġrelations . ĠOur Ġmodel Ġoutper forms Ġthe Ġprevious Ġmethods Ġon ĠKB Ġrelation Ġdetection Ġtasks Ġand Ġallows Ġour ĠKB Q A Ġsystem Ġto Ġachieve Ġstate - of - the - arts . ĠFor Ġfuture Ġwork , Ġwe Ġwill Ġinvestigate Ġthe Ġintegration Ġof Ġour ĠHR - Bi L ST M Ġinto Ġend - to - end Ġsystems . ĠFor Ġexample , Ġour Ġmodel Ġcould Ġbe Ġintegrated Ġinto Ġthe Ġdec oder Ġin ĠB IB REF 31 Ġ, Ġto Ġprovide Ġbetter Ġsequence Ġprediction . ĠWe Ġwill Ġalso Ġinvestigate Ġnew Ġemerging Ġdatasets Ġlike ĠGraph Questions ĠB IB REF 32 Ġand ĠComplex Questions ĠB IB REF 30 Ġto Ġhandle Ġmore Ġcharacteristics Ġof Ġgeneral ĠQ A .