None
ABSTRACTIVE
Word-level Memory Neural Networks (MemNNs) proposed in Bordes et al. (2015)
ABSTRACTIVE
In our experiments, the Memory Neural Networks (MemNNs) proposed in babidataset serve as the baselines. For training, in addition to the 76K questions in the training set, the MemNNs use 3K training questions from WebQuestions BIBREF27 , 15M paraphrases from WikiAnswers BIBREF2 , and 11M and 12M automatically generated questions from the KB for the FB2M and FB5M settings, respectively. In contrast, our models are trained only on the 76K questions in the training set.
In our experiments, the Memory Neural Networks (MemNNs) proposed in babidataset serve as the baselines. For training, in addition to the 76K questions in the training set, the MemNNs use 3K training questions from WebQuestions BIBREF27 , 15M paraphrases from WikiAnswers BIBREF2 , and 11M and 12M automatically generated questions from the KB for the FB2M and FB5M settings, respectively. In contrast, our models are trained only on the 76K questions in the training set. We evaluate the proposed model on the SimpleQuestions dataset BIBREF0 . The dataset consists of 108,442 single-relation questions and their corresponding (topic entity, predicate, answer entity) triples from Freebase. It is split into 75,910 train, 10,845 validation, and 21,687 test questions. Only 10,843 of the 45,335 unique words in entity aliases and 886 out of 1,034 unique predicates in the test set were present in the train set. For the proposed dataset, there are two evaluation settings, called FB2M and FB5M, respectively. The former uses a KB for candidate generation which is a subset of Freebase and contains 2M entities, while the latter uses subset of Freebase with 5M entities.
1604.00727
Introduction </s> Single - relation Ġfact oid Ġquestions Ġare Ġthe Ġmost Ġcommon Ġform Ġof Ġquestions Ġfound Ġin Ġsearch Ġquery Ġlogs Ġand Ġcommunity Ġquestion Ġanswering Ġwebsites ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ. ĠA Ġknowledge - base Ġ( KB ) Ġsuch Ġas ĠFree base , ĠDB pedia , Ġor ĠWik id ata Ġcan Ġhelp Ġanswer Ġsuch Ġquestions Ġafter Ġusers Ġreform ulate Ġthem Ġas Ġqueries . ĠFor Ġinstance , Ġthe Ġquestion ĠWhere Ġwas ĠBarack ĠObama Ġborn ? Ġcan Ġbe Ġanswered Ġby Ġissuing Ġthe Ġfollowing ĠKB Ġquery : Ġ$ Ġ\ lambda Ġ( x ). place \ _ of \ _ birth ( Bar ack \ _ Obama , Ġx ) Ġ$ </s> However , Ġautomatically Ġmapping Ġa Ġnatural Ġlanguage Ġquestion Ġsuch Ġas ĠWhere Ġwas ĠBarack ĠObama Ġborn ? Ġto Ġits Ġcorresponding ĠKB Ġquery Ġremains Ġa Ġchallenging Ġtask . </s> There Ġare Ġthree Ġkey Ġissues Ġthat Ġmake Ġlearning Ġthis Ġmapping Ġnon - t riv ial . ĠFirst , Ġthere Ġare Ġmany Ġparaph r ases Ġof Ġthe Ġsame Ġquestion . ĠSecond , Ġmany Ġof Ġthe ĠKB Ġentries Ġare Ġunseen Ġduring Ġtraining Ġtime ; Ġhowever , Ġwe Ġstill Ġneed Ġto Ġcorrectly Ġpredict Ġthem Ġat Ġtest Ġtime . ĠThird , Ġa ĠKB Ġsuch Ġas ĠFree base Ġtypically Ġcontains Ġmillions Ġof Ġentities Ġand Ġthousands Ġof Ġpred icates , Ġmaking Ġit Ġdifficult Ġfor Ġa Ġsystem Ġto Ġpredict Ġthese Ġentities Ġat Ġscale ĠB IB REF 1 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 0 Ġ. ĠIn Ġthis Ġpaper , Ġwe Ġaddress Ġall Ġthree Ġof Ġthese Ġissues Ġwith Ġa Ġcharacter - level Ġenc oder - dec oder Ġframework Ġthat Ġsignificantly Ġimproves Ġperformance Ġover Ġstate - of - the - art Ġword - level Ġneural Ġmodels , Ġwhile Ġalso Ġproviding Ġa Ġmuch Ġmore Ġcompact Ġmodel Ġthat Ġcan Ġbe Ġlearned Ġfrom Ġless Ġdata . </s> First , Ġwe Ġuse Ġa Ġlong Ġshort - term Ġmemory Ġ( L ST M ) ĠB IB REF 4 Ġenc oder Ġto Ġembed Ġthe Ġquestion . ĠSecond , Ġto Ġmake Ġour Ġmodel Ġrobust Ġto Ġunseen ĠKB Ġentries , Ġwe Ġextract Ġembed d ings Ġfor Ġquestions , Ġpred icates Ġand Ġentities Ġpurely Ġfrom Ġtheir Ġcharacter - level Ġrepresentations . ĠCharacter - level Ġmodeling Ġhas Ġbeen Ġpreviously Ġshown Ġto Ġgeneral ize Ġwell Ġto Ġnew Ġwords Ġnot Ġseen Ġduring Ġtraining ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, Ġwhich Ġmakes Ġit Ġideal Ġfor Ġthis Ġtask . ĠThird , Ġto Ġscale Ġour Ġmodel Ġto Ġhandle Ġthe Ġmillions Ġof Ġentities Ġand Ġthousands Ġof Ġpred icates Ġin Ġthe ĠKB , Ġinstead Ġof Ġusing Ġa Ġlarge Ġoutput Ġlayer Ġin Ġthe Ġdec oder Ġto Ġdirectly Ġpredict Ġthe Ġentity Ġand Ġpredicate , Ġwe Ġuse Ġa Ġgeneral Ġinteraction Ġfunction Ġbetween Ġthe Ġquestion Ġembed d ings Ġand ĠKB Ġembed d ings Ġthat Ġmeasures Ġtheir Ġsemantic Ġrelevance Ġto Ġdetermine Ġthe Ġoutput . ĠThe Ġcombined Ġuse Ġof Ġcharacter - level Ġmodeling Ġand Ġa Ġsemantic Ġrelevance Ġfunction Ġallows Ġus Ġto Ġsuccessfully Ġproduce Ġlikelihood Ġscores Ġfor Ġthe ĠKB Ġentries Ġthat Ġare Ġnot Ġpresent Ġin Ġour Ġvocabulary , Ġa Ġchallenging Ġtask Ġfor Ġstandard Ġenc oder - dec oder Ġframeworks . </s> Our Ġnovel , Ġcharacter - level Ġenc oder - dec oder Ġmodel Ġis Ġcompact , Ġrequires Ġsignificantly Ġless Ġdata Ġto Ġtrain Ġthan Ġprevious Ġwork , Ġand Ġis Ġable Ġto Ġgeneral ize Ġwell Ġto Ġunseen Ġentities Ġin Ġtest Ġtime . ĠIn Ġparticular , Ġwithout Ġuse Ġof Ġen semb les , Ġwe Ġachieve Ġ70 . 9 % Ġaccuracy Ġin Ġthe ĠFree base 2 M Ġsetting Ġand Ġ70 . 3 % Ġaccuracy Ġin Ġthe ĠFree base 5 M Ġsetting Ġon Ġthe ĠSimple Questions Ġdataset , Ġoutper forming Ġthe Ġprevious Ġstate - of - arts Ġof Ġ62 . 7 % Ġand Ġ63 . 9 % ĠB IB REF 0 Ġby Ġ8 . 2 % Ġand Ġ6 . 4 % Ġrespectively . ĠMoreover , Ġwe Ġonly Ġuse Ġthe Ġtraining Ġquestions Ġprovided Ġin ĠSimple Questions Ġto Ġtrain Ġour Ġmodel , Ġwhich Ġcover Ġabout Ġ24 % Ġof Ġwords Ġin Ġentity Ġaliases Ġon Ġthe Ġtest Ġset . ĠThis Ġdemonstrates Ġthe Ġrobust ness Ġof Ġthe Ġcharacter - level Ġmodel Ġto Ġunseen Ġentities . ĠIn Ġcontrast , Ġdata Ġaug mentation Ġis Ġusually Ġnecessary Ġto Ġprovide Ġmore Ġcoverage Ġfor Ġunseen Ġentities Ġand Ġpred icates , Ġas Ġdone Ġin Ġprevious Ġwork ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ. </s> Related ĠWork </s> Our Ġwork Ġis Ġmotivated Ġby Ġthree Ġmajor Ġthreads Ġof Ġresearch Ġin Ġmachine Ġlearning Ġand Ġnatural Ġlanguage Ġprocessing : Ġsemantic - p ars ing Ġfor Ġopen - domain Ġquestion Ġanswering , Ġcharacter - level Ġlanguage Ġmodeling , Ġand Ġenc oder - dec oder Ġmethods . </s> Sem antic Ġparsing Ġfor Ġopen - domain Ġquestion Ġanswering , Ġwhich Ġtranslates Ġa Ġquestion Ġinto Ġa Ġstructured ĠKB Ġquery , Ġis Ġa Ġkey Ġcomponent Ġin Ġquestion Ġanswering Ġwith Ġa ĠKB . ĠWhile Ġearly Ġapproaches Ġrelied Ġon Ġbuilding Ġhigh - quality Ġlex icons Ġfor Ġdomain - specific Ġdatabases Ġsuch Ġas ĠGeo Query ĠB IB REF 7 Ġ, Ġrecent Ġwork Ġhas Ġfocused Ġon Ġbuilding Ġsemantic Ġparsing Ġframeworks Ġfor Ġgeneral Ġknowledge Ġbases Ġsuch Ġas ĠFree base ĠB IB REF 1 Ġ, ĠB IB REF 8 Ġ, ĠB IB REF 0 Ġ, ĠB IB REF 9 Ġ, ĠB IB REF 2 Ġ. </s> Sem antic Ġparsing Ġframeworks Ġfor Ġlarge - scale Ġknowledge Ġbases Ġhave Ġto Ġbe Ġable Ġto Ġsuccessfully Ġgenerate Ġqueries Ġfor Ġthe Ġmillions Ġof Ġentities Ġand Ġthousands Ġof Ġpred icates Ġin Ġthe ĠKB , Ġmany Ġof Ġwhich Ġare Ġunseen Ġduring Ġtraining . ĠTo Ġaddress Ġthis Ġissue , Ġrecent Ġwork Ġrelies Ġon Ġproducing Ġembed d ings Ġfor Ġpred icates Ġand Ġentities Ġin Ġa ĠKB Ġbased Ġon Ġtheir Ġtextual Ġdescriptions ĠB IB REF 8 Ġ, ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 10 Ġ. ĠA Ġgeneral Ġinteraction Ġfunction Ġcan Ġthen Ġbe Ġused Ġto Ġmeasure Ġthe Ġsemantic Ġrelevance Ġof Ġthese Ġembedded ĠKB Ġentries Ġto Ġthe Ġquestion Ġand Ġdetermine Ġthe Ġmost Ġlikely ĠKB Ġquery . </s> Most Ġof Ġthese Ġapproaches Ġuse Ġword - level Ġembed d ings Ġto Ġencode Ġentities Ġand Ġpred icates , Ġand Ġtherefore Ġmight Ġsuffer Ġfrom Ġthe Ġout - of - voc abulary Ġ( OO V ) Ġproblem Ġwhen Ġthey Ġencounter Ġunseen Ġwords Ġduring Ġtest Ġtime . ĠConsequently , Ġthey Ġoften Ġrely Ġon Ġsignificant Ġdata Ġaug mentation Ġfrom Ġsources Ġsuch Ġas ĠPar ale x ĠB IB REF 2 Ġ, Ġwhich Ġcontains Ġ18 Ġmillion Ġquestion - par aph rase Ġpairs Ġscrap ed Ġfrom ĠWiki An swers , Ġto Ġhave Ġsufficient Ġexamples Ġfor Ġeach Ġword Ġthey Ġencounter ĠB IB REF 11 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 0 Ġ. </s> As Ġopposed Ġto Ġword - level Ġmodeling , Ġcharacter - level Ġmodeling Ġcan Ġbe Ġused Ġto Ġhandle Ġthe ĠO OV Ġissue . ĠWhile Ġcharacter - level Ġmodeling Ġhas Ġnot Ġbeen Ġapplied Ġto Ġfact oid Ġquestion Ġanswering Ġbefore , Ġit Ġhas Ġbeen Ġsuccessfully Ġapplied Ġto Ġinformation Ġretrieval , Ġmachine Ġtranslation , Ġsentiment Ġanalysis , Ġclassification , Ġand Ġnamed Ġentity Ġrecognition ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ, ĠB IB REF 19 Ġ. ĠMoreover , Ġg fl st m Ġdemonstrate Ġthat Ġg ated - feed back ĠL ST Ms Ġon Ġtop Ġof Ġcharacter - level Ġembed d ings Ġcan Ġcapture Ġlong - term Ġdependencies Ġin Ġlanguage Ġmodeling . </s> Lastly , Ġenc oder - dec oder Ġnetworks Ġhave Ġbeen Ġapplied Ġto Ġmany Ġstructured Ġmachine Ġlearning Ġtasks . ĠFirst Ġintroduced Ġin Ġs uts ke ver 2014 sequence , Ġin Ġan Ġenc oder - dec oder Ġnetwork , Ġa Ġsource Ġsequence Ġis Ġfirst Ġencoded Ġwith Ġa Ġrecurrent Ġneural Ġnetwork Ġ( R NN ) Ġinto Ġa Ġfixed - length Ġvector Ġwhich Ġintuitive ly Ġcaptures Ġits Ġmeaning , Ġand Ġthen Ġdec oded Ġinto Ġa Ġdesired Ġtarget Ġsequence . ĠThis Ġapproach Ġand Ġrelated Ġmemory - based Ġor Ġattention - based Ġapproaches Ġhave Ġbeen Ġsuccessfully Ġapplied Ġin Ġdiverse Ġdomains Ġsuch Ġas Ġspeech Ġrecognition , Ġmachine Ġtranslation , Ġimage Ġcaption ing , Ġparsing , Ġexecuting Ġprograms , Ġand Ġconvers ational Ġdialog ues ĠB IB REF 20 Ġ, ĠB IB REF 21 Ġ, ĠB IB REF 22 Ġ, ĠB IB REF 23 Ġ, ĠB IB REF 24 Ġ, ĠB IB REF 25 Ġ, ĠB IB REF 26 Ġ. </s> Unlike Ġprevious Ġwork , Ġwe Ġformulate Ġquestion Ġanswering Ġas Ġa Ġproblem Ġof Ġdecoding Ġthe ĠKB Ġquery Ġgiven Ġthe Ġquestion Ġand ĠKB Ġentries Ġwhich Ġare Ġencoded Ġin Ġembed ding Ġspaces . ĠWe Ġtherefore Ġintegrate Ġthe Ġlearning Ġof Ġquestion Ġand ĠKB Ġembed d ings Ġin Ġa Ġunified Ġenc oder - dec oder Ġframework , Ġwhere Ġthe Ġwhole Ġsystem Ġis Ġoptimized Ġend - to - end . </s> Model </s> Since Ġwe Ġfocus Ġon Ġsingle - relation Ġquestion Ġanswering Ġin Ġthis Ġwork , Ġour Ġmodel Ġdec odes Ġevery Ġquestion Ġinto Ġa ĠKB Ġquery Ġthat Ġconsists Ġof Ġexactly Ġtwo Ġelements âĢĵ the Ġtopic Ġentity , Ġand Ġthe Ġpredicate . ĠMore Ġformally , Ġour Ġmodel Ġis Ġa Ġfunction Ġ$ f ( q , Ġ\ l brace Ġe \ r brace Ġ, Ġ\ l brace Ġp \ r brace Ġ) $ Ġthat Ġtakes Ġas Ġinput Ġa Ġquestion Ġ$ q $ Ġ, Ġa Ġset Ġof Ġcandidate Ġentities Ġ$\ l brace Ġe \ r brace Ġ= e _ 1 , Ġ... , e _ n $ Ġ, Ġa Ġset Ġof Ġcandidate Ġpred icates Ġ$\ l brace Ġp \ r brace Ġ= p _ 1 ,... , Ġp _ m $ Ġ, Ġand Ġproduces Ġa Ġlikelihood Ġscore Ġ$ p ( e _ i , Ġp _ j | q ) $ Ġof Ġgenerating Ġentity Ġ$ e _ i $ Ġand Ġpredicate Ġ$ p _ j $ Ġgiven Ġquestion Ġ$ q $ Ġfor Ġall Ġ$ i \ in Ġ{ 1 ... n }, Ġj \ in Ġ{ 1 ... m } $ Ġ. </s> As Ġillustrated Ġin ĠFigure Ġ1 , Ġour Ġmodel Ġconsists Ġof Ġthree Ġcomponents : </s> The Ġdetails Ġof Ġeach Ġcomponent Ġare Ġdescribed Ġin Ġthe Ġfollowing Ġsections . </s> Enc oding Ġthe ĠQuestion </s> To Ġencode Ġthe Ġquestion , Ġwe Ġtake Ġtwo Ġsteps : </s> We Ġfirst Ġextract Ġone - hot Ġencoding Ġvectors Ġfor Ġcharacters Ġin Ġthe Ġquestion , Ġ$ x _ 1 ,... , x _ n $ Ġ, Ġwhere Ġ$ x _ i $ Ġrepresents Ġthe Ġone - hot Ġencoding Ġvector Ġfor Ġthe Ġ$ i ^{ th } $ Ġcharacter Ġin Ġthe Ġquestion . ĠWe Ġkeep Ġthe Ġspace , Ġpunct uation Ġand Ġoriginal Ġcases Ġwithout Ġtoken ization . </s> We Ġfeed Ġ$ x _ 1 ,... , x _ n $ Ġfrom Ġleft Ġto Ġright Ġinto Ġa Ġtwo - layer Ġg ated - feed back ĠL ST M , Ġand Ġkeep Ġthe Ġoutputs Ġat Ġall Ġtime Ġsteps Ġas Ġthe Ġembed d ings Ġfor Ġthe Ġquestion , Ġi . e ., Ġthese Ġare Ġthe Ġvectors Ġ$ s _ 1 ,... , s _ n $ Ġ. </s> Enc oding ĠEnt ities Ġand ĠPred icates Ġin Ġthe ĠKB </s> To Ġencode Ġan Ġentity Ġor Ġpredicate Ġin Ġthe ĠKB , Ġwe Ġtake Ġtwo Ġsteps : </s> We Ġfirst Ġextract Ġone - hot Ġencoding Ġvectors Ġfor Ġcharacters Ġin Ġits ĠEnglish Ġalias , Ġ$ x _ 1 ,... , x _ n $ Ġ, Ġwhere Ġ$ x _ i $ Ġrepresents Ġthe Ġone - hot Ġencoding Ġvector Ġfor Ġthe Ġ$ i ^{ th } $ Ġcharacter Ġin Ġthe Ġalias . </s> We Ġthen Ġfeed Ġ$ x _ 1 ,... , x _ n $ Ġinto Ġa Ġtemporal ĠCNN Ġwith Ġtwo Ġalternating Ġconv olution al Ġand Ġfully - connected Ġlayers , Ġfollowed Ġby Ġone Ġfully - connected Ġlayer : Ġ$ Ġf ( x _ 1 ,... , x _ n ) Ġ= Ġtan h ( W _{ 3 } Ġ\ times Ġmax ( tan h Ġ( W _{ 2 } Ġ\ times Ġ\\ Ġconv ( tan h ({ W _{ 1 } Ġ\ times Ġconv ( x _ 1 ,... , x _ n )} )) ))) Ġ$ </s> where Ġ$ f ( x _{ 1 ... n }) Ġ$ Ġis Ġan Ġembed ding Ġvector Ġof Ġsize Ġ$ N $ Ġ, Ġ$ W _{ 3 } $ Ġhas Ġsize Ġ$ R ^{ N Ġ\ times Ġh } $ Ġ, Ġ$ conv $ Ġrepresents Ġa Ġtemporal Ġconv olution al Ġneural Ġnetwork , Ġand Ġ$ max $ Ġrepresents Ġa Ġmax Ġpool ing Ġlayer Ġin Ġthe Ġtemporal Ġdirection . </s> We Ġuse Ġa ĠCNN Ġas Ġopposed Ġto Ġan ĠL ST M Ġto Ġembed ĠKB Ġentries Ġprimarily Ġfor Ġcomputational Ġefficiency . ĠAlso , Ġwe Ġuse Ġtwo Ġdifferent ĠCNN s Ġto Ġencode Ġentities Ġand Ġpred icates Ġbecause Ġthey Ġtypically Ġhave Ġsignificantly Ġdifferent Ġstyles Ġ( e . g ., ĠBarack ĠObama Ġvs . Ġ/ people / person / place _ of _ birth ). </s> Dec oding Ġthe ĠKB ĠQuery </s> To Ġgenerate Ġthe Ġsingle Ġtopic Ġentity Ġand Ġpredicate Ġto Ġform Ġthe ĠKB Ġquery , Ġwe Ġuse Ġa Ġdec oder Ġwith Ġtwo Ġkey Ġcomponents : </s> An ĠL ST M - based Ġdec oder Ġwith Ġattention . ĠIts Ġhidden Ġstates Ġat Ġeach Ġtime Ġstep Ġ$ i $ Ġ, Ġ$ h _{ i } $ Ġ, Ġhave Ġthe Ġsame Ġdimension ality Ġ$ N $ Ġas Ġthe Ġembed d ings Ġof Ġentities / pred icates . ĠThe Ġinitial Ġhidden Ġstate Ġ$ h _ 0 $ Ġis Ġset Ġto Ġthe Ġzero Ġvector : Ġ$\ vec { 0 } $ Ġ. </s> A Ġpair wise Ġsemantic Ġrelevance Ġfunction Ġthat Ġmeasures Ġthe Ġsimilarity Ġbetween Ġthe Ġhidden Ġunits Ġof Ġthe ĠL ST M Ġand Ġthe Ġembed ding Ġof Ġan Ġentity Ġor Ġpredicate Ġcandidate . ĠIt Ġthen Ġreturns Ġthe Ġmostly Ġlikely Ġentity Ġor Ġpredicate Ġbased Ġon Ġthe Ġsimilarity Ġscore . </s> In Ġthe Ġfollowing Ġtwo Ġsections , Ġwe Ġwill Ġfirst Ġdescribe Ġthe ĠL ST M Ġdec oder Ġwith Ġattention , Ġfollowed Ġby Ġthe Ġsemantic Ġrelevance Ġfunction . </s> The Ġattention - based ĠL ST M Ġdec oder Ġuses Ġa Ġsimilar Ġarchitecture Ġas Ġthe Ġone Ġdescribed Ġin Ġal ignt rans late . ĠAt Ġeach Ġtime Ġstep Ġ$ i $ Ġ, Ġwe Ġfeed Ġin Ġa Ġcontext Ġvector Ġ$ c _ i $ Ġand Ġan Ġinput Ġvector Ġ$ v _ i $ Ġinto Ġthe ĠL ST M . ĠAt Ġtime Ġ$ i = 1 $ Ġwe Ġfeed Ġa Ġspecial Ġinput Ġvector Ġ$ v _{ < { S } > } = \ vec { 0 } $ Ġinto Ġthe ĠL ST M . ĠAt Ġtime Ġ$ i = 2 $ Ġ, Ġduring Ġtraining , Ġthe Ġinput Ġvector Ġis Ġthe Ġembed ding Ġof Ġthe Ġtrue Ġentity , Ġwhile Ġduring Ġtesting , Ġit Ġis Ġthe Ġembed ding Ġof Ġthe Ġmost Ġlikely Ġentity Ġas Ġdetermined Ġat Ġthe Ġprevious Ġtime Ġstep . </s> We Ġnow Ġdescribe Ġhow Ġwe Ġproduce Ġthe Ġcontext Ġvector Ġ$ c _ i $ Ġ. ĠLet Ġ$ h _{ i - 1 } $ Ġbe Ġthe Ġhidden Ġstate Ġof Ġthe ĠL ST M Ġat Ġtime Ġ$ i - 1 $ Ġ, Ġ$ s _ j $ Ġbe Ġthe Ġ$ j ^{ th } $ Ġquestion Ġcharacter Ġembed ding , Ġ$ n $ Ġbe Ġthe Ġnumber Ġof Ġcharacters Ġin Ġthe Ġquestion , Ġ$ r $ Ġbe Ġthe Ġsize Ġof Ġ$ s _ j $ Ġ, Ġand Ġ$ m $ Ġbe Ġa Ġhyper param eter . ĠThen Ġthe Ġcontext Ġvector Ġ$ c _ i $ Ġ, Ġwhich Ġrepresents Ġthe Ġattention - weight ed Ġcontent Ġof Ġthe Ġquestion , Ġis Ġrec om puted Ġat Ġeach Ġtime Ġstep Ġ$ h _{ i - 1 } $ 0 Ġas Ġfollows : Ġ$ h _{ i - 1 } $ 1 Ġ$ h _{ i - 1 } $ 2 </s> where Ġ$\ l brace Ġ\ alpha Ġ\ r brace Ġ$ Ġis Ġthe Ġattention Ġdistribution Ġthat Ġis Ġapplied Ġover Ġeach Ġhidden Ġunit Ġ$ s _ j $ Ġ, Ġ$ W _ a Ġ\ in ĠR ^{ m Ġ\ times ĠN }, ĠU _ a Ġ\ in ĠR ^{ m Ġ\ times Ġr }, $ Ġand Ġ$ v _ a Ġ\ in Ġ{ R } ^{ 1 Ġ\ times Ġm } $ Ġ. </s> Unlike Ġmachine Ġtranslation Ġand Ġlanguage Ġmodeling Ġwhere Ġthe Ġvocabulary Ġis Ġrelatively Ġsmall , Ġthere Ġare Ġmillions Ġof Ġentries Ġin Ġthe ĠKB . ĠIf Ġwe Ġtry Ġto Ġdirectly Ġpredict Ġthe ĠKB Ġentries , Ġthe Ġdec oder Ġwill Ġneed Ġan Ġoutput Ġlayer Ġwith Ġmillions Ġof Ġnodes , Ġwhich Ġis Ġcomput ationally Ġprohib itive . ĠTherefore , Ġwe Ġresort Ġto Ġa Ġrelevance Ġfunction Ġthat Ġmeasures Ġthe Ġsemantic Ġsimilarity Ġbetween Ġthe Ġdec oder 's Ġhidden Ġstate Ġand Ġthe Ġembed d ings Ġof ĠKB Ġentries . ĠOur Ġsemantic Ġrelevance Ġfunction Ġtakes Ġtwo Ġvectors Ġ$ x _ 1 $ Ġ, Ġ$ x _ 2 $ Ġand Ġreturns Ġa Ġdistance Ġmeasure Ġof Ġhow Ġsimilar Ġthey Ġare Ġto Ġeach Ġother . ĠIn Ġcurrent Ġexperiments Ġwe Ġuse Ġa Ġsimple Ġcos ine - similar ity Ġmetric : Ġ$ cos ( x _ 1 , Ġx _ 2 ) $ Ġ. </s> Using Ġthis Ġsimilarity Ġmetric , Ġthe Ġlikelihood s Ġof Ġgenerating Ġentity Ġ$ e _ j $ Ġand Ġpredicate Ġ$ p _ k $ Ġare : Ġ$ Ġ\ h space * { 0 . 0 pt } ĠP ( e _ j ) Ġ= Ġ\ frac { exp (\ lambda Ġcos ( h _ 1 , e _{ j } )) }{ \ sum Ġ_ { i = 1 } ^{ n } Ġexp (\ lambda Ġcos ( h _ 1 , e _ i )) } Ġ\\ ĠP ( p _ k ) Ġ= Ġ\ frac { exp (\ lambda Ġcos ( h _ 2 , p _{ k } )) }{ \ sum Ġ_ { i = 1 } ^{ m } Ġexp (\ lambda Ġcos ( h _ 2 , p _{ i } )) } Ġ$ </s> where Ġ$\ lambda Ġ$ Ġis Ġa Ġconstant , Ġ$ h _ 1 , Ġh _ 2 $ Ġare Ġthe Ġhidden Ġstates Ġof Ġthe ĠL ST M Ġat Ġtimes Ġ$ t = 1 $ Ġand Ġ$ t = 2 $ Ġ, Ġ$ e _ 1 ,... , e _ n $ Ġare Ġthe Ġentity Ġembed d ings , Ġand Ġ$ p _ 1 ,... , p _ m $ Ġare Ġthe Ġpredicate Ġembed d ings . ĠA Ġsimilar Ġlikelihood Ġfunction Ġwas Ġused Ġto Ġtrain Ġthe Ġsemantic Ġsimilarity Ġmodules Ġproposed Ġin Ġq a acl Ġand ĠY ih 2015 Sem antic P V . </s> During Ġinference , Ġ$ e _ 1 ,... , e _ n $ Ġand Ġ$ p _ 1 ,... , p _ m $ Ġare Ġthe Ġembed d ings Ġof Ġcandidate Ġentities Ġand Ġpred icates . ĠDuring Ġtraining Ġ$ e _ 1 ,... , e _ n $ Ġ, Ġ$ p _ 1 ,... , p _ m $ Ġare Ġthe Ġembed d ings Ġof Ġthe Ġtrue Ġentity Ġand Ġ50 Ġrandomly - sam pled Ġentities , Ġand Ġthe Ġtrue Ġpredicate Ġand Ġ50 Ġrandomly - sam pled Ġpred icates , Ġrespectively . </s> In ference </s> For Ġeach Ġquestion Ġ$ q $ Ġ, Ġwe Ġgenerate Ġa Ġcandidate Ġset Ġof Ġentities Ġand Ġpred icates , Ġ$\ l brace Ġe \ r brace Ġ$ Ġand Ġ$\ l brace Ġp \ r brace Ġ$ Ġ, Ġand Ġfeed Ġit Ġthrough Ġthe Ġmodel Ġ$ f ( q , Ġ\ l brace Ġe \ r brace Ġ, Ġ\ l brace Ġp \ r brace Ġ) $ Ġ. ĠWe Ġthen Ġdecode Ġthe Ġmost Ġlikely Ġ( entity , Ġpredicate ) Ġpair : Ġ$ Ġ( e ^ *, Ġp ^ *) Ġ= Ġarg max _{ e _ i , Ġp _ j } Ġ( P ( e _ i )* P ( p _ j )) Ġ$ </s> which Ġbecomes Ġour Ġsemantic Ġparse . </s> We Ġuse Ġa Ġsimilar Ġprocedure Ġas Ġthe Ġone Ġdescribed Ġin Ġbab id at as et Ġto Ġgenerate Ġcandidate Ġentities Ġ$\ l brace Ġe \ r brace Ġ$ Ġand Ġpred icates Ġ$\ l brace Ġp \ r brace Ġ$ Ġ. ĠName ly , Ġwe Ġtake Ġall Ġentities Ġwhose ĠEnglish Ġalias Ġis Ġa Ġsubst ring Ġof Ġthe Ġquestion , Ġand Ġremove Ġall Ġentities Ġwhose Ġalias Ġis Ġa Ġsubst ring Ġof Ġanother Ġentity . ĠFor Ġeach ĠEnglish Ġalias , Ġwe Ġsort Ġeach Ġentity Ġwith Ġthis Ġalias Ġby Ġthe Ġnumber Ġof Ġfacts Ġthat Ġit Ġhas Ġin Ġthe ĠKB , Ġand Ġappend Ġthe Ġtop Ġ10 Ġentities Ġfrom Ġthis Ġlist Ġto Ġour Ġset Ġof Ġcandidate Ġentities . ĠAll Ġpred icates Ġ${ p _ j } $ Ġfor Ġeach Ġentity Ġin Ġour Ġcandidate Ġentity Ġset Ġbecome Ġthe Ġset Ġof Ġcandidate Ġpred icates . </s> Learning </s> Our Ġgoal Ġin Ġlearning Ġis Ġto Ġmaximize Ġthe Ġjoint Ġlikelihood Ġ$ P ( e _ c ) Ġ\ cd ot ĠP ( p _ c ) $ Ġof Ġpredicting Ġthe Ġcorrect Ġentity Ġ$ e _ c $ Ġand Ġpredicate Ġ$ p _ c $ Ġpair Ġfrom Ġa Ġset Ġof Ġrandomly Ġsampled Ġentities Ġand Ġpred icates . ĠWe Ġuse Ġback - prop ag ation Ġto Ġlearn Ġall Ġof Ġthe Ġweights Ġin Ġour Ġmodel . </s> All Ġthe Ġparameters Ġof Ġour Ġmodel Ġare Ġlearned Ġjointly Ġwithout Ġpre - training . ĠThese Ġparameters Ġinclude Ġthe Ġweights Ġof Ġthe Ġcharacter - level Ġembed d ings , ĠCNN s , Ġand ĠL ST Ms . ĠWe ights Ġare Ġrandomly Ġinitialized Ġbefore Ġtraining . ĠFor Ġthe Ġ$ i ^{ th } $ Ġlayer Ġin Ġour Ġnetwork , Ġeach Ġweight Ġis Ġsampled Ġfrom Ġa Ġuniform Ġdistribution Ġbetween Ġ$ - \ frac { 1 }{ | l ^ i | } $ Ġand Ġ$\ frac { 1 }{ | l ^ i | } $ Ġ, Ġwhere Ġ$ | l ^ i | $ Ġis Ġthe Ġnumber Ġof Ġweights Ġin Ġlayer Ġ$ i $ Ġ. </s> Dat as et Ġand ĠExperimental ĠSettings </s> We Ġevaluate Ġthe Ġproposed Ġmodel Ġon Ġthe ĠSimple Questions Ġdataset ĠB IB REF 0 Ġ. ĠThe Ġdataset Ġconsists Ġof Ġ108 , 442 Ġsingle - relation Ġquestions Ġand Ġtheir Ġcorresponding Ġ( topic Ġentity , Ġpredicate , Ġanswer Ġentity ) Ġtri ples Ġfrom ĠFree base . ĠIt Ġis Ġsplit Ġinto Ġ75 , 910 Ġtrain , Ġ10 , 8 45 Ġvalidation , Ġand Ġ21 , 687 Ġtest Ġquestions . ĠOnly Ġ10 , 8 43 Ġof Ġthe Ġ45 , 335 Ġunique Ġwords Ġin Ġentity Ġaliases Ġand Ġ8 86 Ġout Ġof Ġ1 , 034 Ġunique Ġpred icates Ġin Ġthe Ġtest Ġset Ġwere Ġpresent Ġin Ġthe Ġtrain Ġset . ĠFor Ġthe Ġproposed Ġdataset , Ġthere Ġare Ġtwo Ġevaluation Ġsettings , Ġcalled ĠFB 2 M Ġand ĠFB 5 M , Ġrespectively . ĠThe Ġformer Ġuses Ġa ĠKB Ġfor Ġcandidate Ġgeneration Ġwhich Ġis Ġa Ġsubset Ġof ĠFree base Ġand Ġcontains Ġ2 M Ġentities , Ġwhile Ġthe Ġlatter Ġuses Ġsubset Ġof ĠFree base Ġwith Ġ5 M Ġentities . </s> In Ġour Ġexperiments , Ġthe ĠMemory ĠNeural ĠNetworks Ġ( Mem NN s ) Ġproposed Ġin Ġbab id at as et Ġserve Ġas Ġthe Ġbas elines . ĠFor Ġtraining , Ġin Ġaddition Ġto Ġthe Ġ76 K Ġquestions Ġin Ġthe Ġtraining Ġset , Ġthe ĠMem NN s Ġuse Ġ3 K Ġtraining Ġquestions Ġfrom ĠWeb Questions ĠB IB REF 27 Ġ, Ġ15 M Ġparaph r ases Ġfrom ĠWiki An swers ĠB IB REF 2 Ġ, Ġand Ġ11 M Ġand Ġ12 M Ġautomatically Ġgenerated Ġquestions Ġfrom Ġthe ĠKB Ġfor Ġthe ĠFB 2 M Ġand ĠFB 5 M Ġsettings , Ġrespectively . ĠIn Ġcontrast , Ġour Ġmodels Ġare Ġtrained Ġonly Ġon Ġthe Ġ76 K Ġquestions Ġin Ġthe Ġtraining Ġset . </s> For Ġour Ġmodel , Ġboth Ġlayers Ġof Ġthe ĠL ST M - based Ġquestion Ġenc oder Ġhave Ġsize Ġ200 . ĠThe Ġhidden Ġlayers Ġof Ġthe ĠL ST M - based Ġdec oder Ġhave Ġsize Ġ100 , Ġand Ġthe ĠCNN s Ġfor Ġentity Ġand Ġpredicate Ġembed d ings Ġhave Ġa Ġhidden Ġlayer Ġof Ġsize Ġ200 Ġand Ġan Ġoutput Ġlayer Ġof Ġsize Ġ100 . ĠThe ĠCNN s Ġfor Ġentity Ġand Ġpredicate Ġembed d ings Ġuse Ġa Ġreceptive Ġfield Ġof Ġsize Ġ4 , Ġ$\ lambda Ġ= 5 $ Ġ, Ġand Ġ$ m = 100 $ Ġ. ĠWe Ġtrain Ġthe Ġmodels Ġusing ĠRM SP rop Ġwith Ġa Ġlearning Ġrate Ġof Ġ$ 1 e ^{ - 4 } $ Ġ. </s> In Ġorder Ġto Ġmake Ġthe Ġinput Ġcharacter Ġsequence Ġlong Ġenough Ġto Ġfill Ġup Ġthe Ġreceptive Ġfields Ġof Ġmultiple ĠCNN Ġlayers , Ġwe Ġpad Ġeach Ġpredicate Ġor Ġentity Ġusing Ġthree Ġpadding Ġsymbols Ġ$ P $ Ġ, Ġa Ġspecial Ġstart Ġsymbol , Ġand Ġa Ġspecial Ġend Ġsymbol . ĠFor Ġinstance , Ġ$ Obama $ Ġwould Ġbecome Ġ$ S _{ start } PP P ĠObama PP PS _{ end } $ Ġ. ĠFor Ġconsistency , Ġwe Ġapply Ġthe Ġsame Ġpadding Ġto Ġthe Ġquestions . </s> End - to - end ĠResults Ġon ĠSimple Questions </s> Following Ġbab id at as et , Ġwe Ġreport Ġresults Ġon Ġthe ĠSimple Questions Ġdataset Ġin Ġterms Ġof ĠSQ Ġaccuracy , Ġfor Ġboth ĠFB 2 M Ġand ĠFB 5 M Ġsettings Ġin ĠTable Ġ1 . ĠSQ Ġaccuracy Ġis Ġdefined Ġas Ġthe Ġpercentage Ġof Ġquestions Ġfor Ġwhich Ġthe Ġmodel Ġgenerates Ġa Ġcorrect ĠKB Ġquery Ġ( i . e ., Ġboth Ġthe Ġtopic Ġentity Ġand Ġpredicate Ġare Ġcorrect ). ĠOur Ġsingle Ġcharacter - level Ġmodel Ġachieves ĠSQ Ġaccur acies Ġof Ġ70 . 9 % Ġand Ġ70 . 3 % Ġon Ġthe ĠFB 2 M Ġand ĠFB 5 M Ġsettings , Ġoutper forming Ġthe Ġprevious Ġstate - of - art Ġresults Ġby Ġ8 . 2 % Ġand Ġ6 . 4 %, Ġrespectively . ĠCompared Ġto Ġthe Ġcharacter - level Ġmodel , Ġwhich Ġonly Ġhas Ġ1 . 2 M Ġparameters , Ġour Ġword - level Ġmodel Ġhas Ġ19 . 9 M Ġparameters , Ġand Ġonly Ġachieves Ġa Ġbest ĠSQ Ġaccuracy Ġof Ġ53 . 9 %. ĠIn Ġaddition , Ġin Ġcontrast Ġto Ġprevious Ġwork , Ġthe ĠO OV Ġissue Ġis Ġmuch Ġmore Ġsevere Ġfor Ġour Ġword - level Ġmodel , Ġsince Ġwe Ġuse Ġno Ġdata Ġaug mentation Ġto Ġcover Ġentities Ġunseen Ġin Ġthe Ġtrain Ġset . </s> A bl ation Ġand ĠEmb edd ing ĠExper iments </s> We Ġcarry Ġout Ġab lation Ġstudies Ġin ĠSections Ġ5 . 2 . 1 Ġand Ġ5 . 2 . 2 Ġthrough Ġa Ġset Ġof Ġrandom - sam pling Ġexperiments . ĠIn Ġthese Ġexperiments , Ġfor Ġeach Ġquestion , Ġwe Ġrandomly Ġsample Ġ200 Ġentities Ġand Ġpred icates Ġfrom Ġthe Ġtest Ġset Ġas Ġnoise Ġsamples . ĠWe Ġthen Ġmix Ġthe Ġgold Ġentity Ġand Ġpredicate Ġinto Ġthese Ġnegative Ġsamples , Ġand Ġevaluate Ġthe Ġaccuracy Ġof Ġour Ġmodel Ġin Ġpredicting Ġthe Ġgold Ġpredicate Ġor Ġentity Ġfrom Ġthis Ġmixed Ġset . </s> We Ġfirst Ġexplore Ġusing Ġword - level Ġmodels Ġas Ġan Ġalternative Ġto Ġcharacter - level Ġmodels Ġto Ġconstruct Ġembed d ings Ġfor Ġquestions , Ġentities Ġand Ġpred icates . </s> Both Ġword - level Ġand Ġcharacter - level Ġmodels Ġperform Ġcompar ably Ġwell Ġwhen Ġpredicting Ġthe Ġpredicate , Ġreaching Ġan Ġaccuracy Ġof Ġaround Ġ80 % Ġ( Table Ġ3 ). ĠHowever , Ġthe Ġword - level Ġmodel Ġhas Ġconsiderable Ġdifficulty Ġgeneral izing Ġto Ġunseen Ġentities , Ġand Ġis Ġonly Ġable Ġto Ġpredict Ġ45 % Ġof Ġthe Ġentities Ġaccurately Ġfrom Ġthe Ġmixed Ġset . ĠThese Ġresults Ġclearly Ġdemonstrate Ġthat Ġthe ĠO OV Ġissue Ġis Ġmuch Ġmore Ġsevere Ġfor Ġentities Ġthan Ġpred icates , Ġand Ġthe Ġdifficulty Ġword - level Ġmodels Ġhave Ġwhen Ġgeneral izing Ġto Ġnew Ġentities . </s> In Ġcontrast , Ġcharacter - level Ġmodels Ġhave Ġno Ġsuch Ġissues , Ġand Ġachieve Ġa Ġ96 . 6 % Ġaccuracy Ġin Ġpredicting Ġthe Ġcorrect Ġentity Ġon Ġthe Ġmixed Ġset . ĠThis Ġdemonstrates Ġthat Ġcharacter - level Ġmodels Ġencode Ġthe Ġsemantic Ġrepresentation Ġof Ġentities Ġand Ġcan Ġmatch Ġentity Ġaliases Ġin Ġa ĠKB Ġwith Ġtheir Ġmentions Ġin Ġnatural Ġlanguage Ġquestions . </s> We Ġalso Ġstudy Ġthe Ġimpact Ġof Ġthe Ġdepth Ġof Ġneural Ġnetworks Ġin Ġour Ġmodel . ĠThe Ġresults Ġare Ġpresented Ġin ĠTable Ġ2 . ĠIn Ġthe Ġab lation Ġexperiments Ġwe Ġcompare Ġthe Ġperformance Ġof Ġa Ġsingle - layer ĠL ST M Ġto Ġa Ġtwo - layer ĠL ST M Ġto Ġencode Ġthe Ġquestion , Ġand Ġa Ġsingle - layer Ġvs . Ġtwo - layer ĠCNN Ġto Ġencode Ġthe ĠKB Ġentries . ĠWe Ġfind Ġthat Ġa Ġtwo - layer ĠL ST M Ġboosts Ġjoint Ġaccuracy Ġby Ġover Ġ6 %. ĠThe Ġmajority Ġof Ġaccuracy Ġgains Ġare Ġa Ġresult Ġof Ġimproved Ġpredicate Ġpredictions , Ġpossibly Ġbecause Ġentity Ġaccuracy Ġis Ġalready Ġsaturated Ġin Ġthis Ġexperimental Ġsetup . </s> Att ention ĠMechan isms </s> In Ġorder Ġto Ġfurther Ġunderstand Ġhow Ġthe Ġmodel Ġperforms Ġquestion Ġanswering , Ġwe Ġvisualize Ġthe Ġattention Ġdistribution Ġover Ġquestion Ġcharacters Ġin Ġthe Ġdecoding Ġprocess . ĠIn Ġeach Ġsub - figure Ġof ĠFigure Ġ2 , Ġthe Ġx - axis Ġis Ġthe Ġcharacter Ġsequence Ġof Ġthe Ġquestion , Ġand Ġthe Ġy - axis Ġis Ġthe Ġattention Ġweight Ġdistribution Ġ$\ l brace Ġ\ alpha Ġ_ i \ r brace Ġ$ Ġ. ĠThe Ġblue Ġcurve Ġis Ġthe Ġattention Ġdistribution Ġwhen Ġgenerating Ġthe Ġentity , Ġand Ġgreen Ġcurve Ġis Ġthe Ġattention Ġdistribution Ġwhen Ġgenerating Ġthe Ġpredicate . </s> Interestingly , Ġas Ġthe Ġexamples Ġshow , Ġthe Ġattention Ġdistribution Ġtypically Ġpeaks Ġat Ġempty Ġspaces . ĠThis Ġindicates Ġthat Ġthe Ġcharacter - level Ġmodel Ġlearns Ġthat Ġa Ġspace Ġdefines Ġan Ġending Ġpoint Ġof Ġa Ġcomplete Ġlinguistic Ġunit . ĠThat Ġis , Ġthe Ġhidden Ġstate Ġof Ġthe ĠL ST M Ġenc oder Ġat Ġa Ġspace Ġlikely Ġsummarizes Ġcontent Ġabout Ġthe Ġcharacter Ġsequence Ġbefore Ġthat Ġspace , Ġand Ġtherefore Ġcontains Ġimportant Ġsemantic Ġinformation Ġthat Ġthe Ġdec oder Ġneeds Ġto Ġattend Ġto . </s> Also , Ġwe Ġobserve Ġthat Ġentity Ġattention Ġdistributions Ġare Ġusually Ġless Ġsharp Ġand Ġspan Ġlonger Ġportions Ġof Ġwords , Ġsuch Ġas Ġjohn Ġor Ġr utters , Ġthan Ġpredicate Ġattention Ġdistributions Ġ( e . g ., ĠFigure Ġ2 a ). ĠFor Ġentities , Ġsemantic Ġinformation Ġmay Ġaccumulate Ġgradually Ġwhen Ġseeing Ġmore Ġand Ġmore Ġcharacters , Ġwhile Ġfor Ġpred icates , Ġsemantic Ġinformation Ġwill Ġbecome Ġclear Ġonly Ġafter Ġseeing Ġthe Ġcomplete Ġword . ĠFor Ġexample , Ġit Ġmay Ġonly Ġbe Ġclear Ġthat Ġcharacters Ġsuch Ġas Ġsong Ġby Ġrefer Ġto Ġa Ġpredicate Ġafter Ġa Ġspace , Ġas Ġopposed Ġto Ġthe Ġname Ġof Ġa Ġsong Ġsuch Ġas Ġsong Ġbye Ġbye Ġlove Ġ( Fig ures Ġ2 a , Ġ2 b ). ĠIn Ġcontrast , Ġa Ġsequence Ġof Ġcharacters Ġstarts Ġto Ġbecome Ġa Ġlikely Ġentity Ġafter Ġseeing Ġan Ġincomplete Ġname Ġsuch Ġas Ġj oh Ġor Ġr utt . </s> In Ġaddition , Ġa Ġcharacter - level Ġmodel Ġcan Ġidentify Ġentities Ġwhose ĠEnglish Ġaliases Ġwere Ġnever Ġseen Ġin Ġtraining , Ġsuch Ġas Ġph ren ology Ġ( Figure Ġ2 d ). ĠThe Ġmodel Ġapparently Ġlearns Ġthat Ġwords Ġending Ġwith Ġthe Ġsuffix Ġn ology Ġare Ġlikely Ġentity Ġmentions , Ġwhich Ġis Ġinteresting Ġbecause Ġit Ġreads Ġin Ġthe Ġinput Ġone Ġcharacter Ġat Ġa Ġtime . </s> Furthermore , Ġas Ġobserved Ġin ĠFigure Ġ2 d , Ġthe Ġattention Ġmodel Ġis Ġcapable Ġof Ġattending Ġdis j oint Ġregions Ġof Ġthe Ġquestion Ġand Ġcapture Ġthe Ġmention Ġof Ġa Ġpredicate Ġthat Ġis Ġinterrupted Ġby Ġentity Ġmentions . ĠWe Ġalso Ġnote Ġthat Ġpredicate Ġattention Ġoften Ġpeaks Ġat Ġthe Ġpadding Ġsymbols Ġafter Ġthe Ġlast Ġcharacter Ġof Ġthe Ġquestion , Ġpossibly Ġbecause Ġsentence Ġendings Ġcarry Ġextra Ġinformation Ġthat Ġfurther Ġhelp Ġdis amb ig uate Ġpredicate Ġmentions . ĠIn Ġcertain Ġscenarios , Ġthe Ġnetwork Ġmay Ġonly Ġhave Ġsufficient Ġinformation Ġto Ġbuild Ġa Ġsemantic Ġrepresentation Ġof Ġthe Ġpredicate Ġafter Ġbeing Ġensured Ġthat Ġit Ġreached Ġthe Ġend Ġof Ġa Ġsentence . ĠFinally , Ġcertain Ġwords Ġin Ġthe Ġquestion Ġhelp Ġidentify Ġboth Ġthe Ġentity Ġand Ġthe Ġpredicate . ĠFor Ġexample , Ġconsider Ġthe Ġword Ġuniversity Ġin Ġthe Ġquestion ĠWhat Ġtype Ġof Ġeducational Ġinstitution Ġis Ġeastern Ġnew Ġme x ico Ġuniversity Ġ( Figure Ġ2 c ). ĠAlthough Ġit Ġis Ġa Ġpart Ġof Ġthe Ġentity Ġmention , Ġit Ġalso Ġhelps Ġdis amb ig uate Ġthe Ġpredicate . ĠHowever , Ġprevious Ġsemantic Ġparsing - based ĠQ A Ġapproaches ĠB IB REF 10 Ġ, ĠB IB REF 1 Ġassume Ġthat Ġthere Ġis Ġa Ġclear Ġseparation Ġbetween Ġthe Ġpredicate Ġand Ġentity Ġmentions Ġin Ġthe Ġquestion . ĠIn Ġcontrast , Ġthe Ġproposed Ġmodel Ġdoes Ġnot Ġneed Ġto Ġmake Ġthis Ġhard Ġcategor ization , Ġand Ġattends Ġthe Ġword Ġuniversity Ġwhen Ġpredicting Ġboth Ġthe Ġentity Ġand Ġpredicate . </s> Error ĠAnalysis </s> We Ġrandomly Ġsampled Ġ50 Ġquestions Ġwhere Ġthe Ġbest - performing Ġmodel Ġgenerated Ġthe Ġwrong ĠKB Ġquery Ġand Ġcategorized Ġthe Ġerrors . ĠFor Ġ46 Ġout Ġof Ġthe Ġ50 Ġexamples , Ġthe Ġmodel Ġpredicted Ġa Ġpredicate Ġwith Ġa Ġvery Ġsimilar Ġalias Ġto Ġthe Ġtrue Ġpredicate , Ġi . e . Ġ/ music / release / track Ġvs . Ġ/ music / release / track _ list . ĠFor Ġ21 Ġout Ġof Ġthe Ġ50 Ġexamples , Ġthe Ġmodel Ġpredicted Ġthe Ġwrong Ġentity , Ġe . g ., ĠAlbum Ġvs . ĠStill ĠHere Ġfor Ġthe Ġquestion ĠWhat Ġtype Ġof Ġalbum Ġis Ġstill Ġhere ? . ĠFinally , Ġfor Ġ18 Ġof Ġthe Ġ50 Ġexamples , Ġthe Ġmodel Ġpredicted Ġthe Ġwrong Ġentity Ġand Ġpredicate , Ġi . e . Ġ( Play , Ġ/ free base / equ ivalent _ topic / equ ivalent _ type ) Ġfor Ġthe Ġquestion Ġwhich Ġinstrument Ġdoes Ġam ap ola Ġc abase Ġplay ? ĠTraining Ġon Ġmore Ġdata , Ġaugment ing Ġthe Ġnegative Ġsample Ġset Ġwith Ġwords Ġfrom Ġthe Ġquestion Ġthat Ġare Ġnot Ġan Ġentity Ġmention , Ġand Ġhaving Ġmore Ġexamples Ġthat Ġdis amb ig uate Ġbetween Ġsimilar Ġpred icates Ġmay Ġam el ior ate Ġmany Ġof Ġthese Ġerrors . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġproposed Ġa Ġnew Ġcharacter - level , Ġattention - based Ġenc oder - dec oder Ġmodel Ġfor Ġquestion Ġanswering . ĠIn Ġour Ġapproach , Ġembed d ings Ġof Ġquestions , Ġentities , Ġand Ġpred icates Ġare Ġall Ġjointly Ġlearned Ġto Ġdirectly Ġoptimize Ġthe Ġlikelihood Ġof Ġgenerating Ġthe Ġcorrect ĠKB Ġquery . ĠOur Ġapproach Ġimproved Ġthe Ġstate - of - the - art Ġaccuracy Ġon Ġthe ĠSimple Questions Ġbenchmark Ġsignificantly , Ġusing Ġmuch Ġless Ġdata Ġthan Ġprevious Ġwork . ĠFurthermore , Ġthanks Ġto Ġcharacter - level Ġmodeling , Ġwe Ġhave Ġa Ġcompact Ġmodel Ġthat Ġis Ġrobust Ġto Ġunseen Ġentities . ĠVisual izations Ġof Ġthe Ġattention Ġdistribution Ġreveal Ġthat Ġour Ġmodel , Ġalthough Ġbuilt Ġon Ġcharacter - level Ġinputs , Ġcan Ġlearn Ġhigher - level Ġsemantic Ġconcepts Ġrequired Ġto Ġanswer Ġa Ġnatural Ġlanguage Ġquestion Ġwith Ġa Ġstructured ĠKB . ĠIn Ġthe Ġfuture Ġwe Ġwould Ġlike Ġto Ġextend Ġour Ġsystem Ġto Ġhandle Ġmulti - relation Ġquestions .
Introduction Single-relation factoid questions are the most common form of questions found in search query logs and community question answering websites BIBREF1 , BIBREF2 . A knowledge-base (KB) such as Freebase, DBpedia, or Wikidata can help answer such questions after users reformulate them as queries. For instance, the question Where was Barack Obama born? can be answered by issuing the following KB query: $ \lambda (x).place\_of\_birth(Barack\_Obama, x) $ However, automatically mapping a natural language question such as Where was Barack Obama born? to its corresponding KB query remains a challenging task. There are three key issues that make learning this mapping non-trivial. First, there are many paraphrases of the same question. Second, many of the KB entries are unseen during training time; however, we still need to correctly predict them at test time. Third, a KB such as Freebase typically contains millions of entities and thousands of predicates, making it difficult for a system to predict these entities at scale BIBREF1 , BIBREF3 , BIBREF0 . In this paper, we address all three of these issues with a character-level encoder-decoder framework that significantly improves performance over state-of-the-art word-level neural models, while also providing a much more compact model that can be learned from less data. First, we use a long short-term memory (LSTM) BIBREF4 encoder to embed the question. Second, to make our model robust to unseen KB entries, we extract embeddings for questions, predicates and entities purely from their character-level representations. Character-level modeling has been previously shown to generalize well to new words not seen during training BIBREF5 , BIBREF6 , which makes it ideal for this task. Third, to scale our model to handle the millions of entities and thousands of predicates in the KB, instead of using a large output layer in the decoder to directly predict the entity and predicate, we use a general interaction function between the question embeddings and KB embeddings that measures their semantic relevance to determine the output. The combined use of character-level modeling and a semantic relevance function allows us to successfully produce likelihood scores for the KB entries that are not present in our vocabulary, a challenging task for standard encoder-decoder frameworks. Our novel, character-level encoder-decoder model is compact, requires significantly less data to train than previous work, and is able to generalize well to unseen entities in test time. In particular, without use of ensembles, we achieve 70.9% accuracy in the Freebase2M setting and 70.3% accuracy in the Freebase5M setting on the SimpleQuestions dataset, outperforming the previous state-of-arts of 62.7% and 63.9% BIBREF0 by 8.2% and 6.4% respectively. Moreover, we only use the training questions provided in SimpleQuestions to train our model, which cover about 24% of words in entity aliases on the test set. This demonstrates the robustness of the character-level model to unseen entities. In contrast, data augmentation is usually necessary to provide more coverage for unseen entities and predicates, as done in previous work BIBREF0 , BIBREF1 . Related Work Our work is motivated by three major threads of research in machine learning and natural language processing: semantic-parsing for open-domain question answering, character-level language modeling, and encoder-decoder methods. Semantic parsing for open-domain question answering, which translates a question into a structured KB query, is a key component in question answering with a KB. While early approaches relied on building high-quality lexicons for domain-specific databases such as GeoQuery BIBREF7 , recent work has focused on building semantic parsing frameworks for general knowledge bases such as Freebase BIBREF1 , BIBREF8 , BIBREF0 , BIBREF9 , BIBREF2 . Semantic parsing frameworks for large-scale knowledge bases have to be able to successfully generate queries for the millions of entities and thousands of predicates in the KB, many of which are unseen during training. To address this issue, recent work relies on producing embeddings for predicates and entities in a KB based on their textual descriptions BIBREF8 , BIBREF0 , BIBREF1 , BIBREF10 . A general interaction function can then be used to measure the semantic relevance of these embedded KB entries to the question and determine the most likely KB query. Most of these approaches use word-level embeddings to encode entities and predicates, and therefore might suffer from the out-of-vocabulary (OOV) problem when they encounter unseen words during test time. Consequently, they often rely on significant data augmentation from sources such as Paralex BIBREF2 , which contains 18 million question-paraphrase pairs scraped from WikiAnswers, to have sufficient examples for each word they encounter BIBREF11 , BIBREF1 , BIBREF0 . As opposed to word-level modeling, character-level modeling can be used to handle the OOV issue. While character-level modeling has not been applied to factoid question answering before, it has been successfully applied to information retrieval, machine translation, sentiment analysis, classification, and named entity recognition BIBREF12 , BIBREF13 , BIBREF6 , BIBREF14 , BIBREF15 , BIBREF16 , BIBREF17 , BIBREF18 , BIBREF19 . Moreover, gflstm demonstrate that gated-feedback LSTMs on top of character-level embeddings can capture long-term dependencies in language modeling. Lastly, encoder-decoder networks have been applied to many structured machine learning tasks. First introduced in sutskever2014sequence, in an encoder-decoder network, a source sequence is first encoded with a recurrent neural network (RNN) into a fixed-length vector which intuitively captures its meaning, and then decoded into a desired target sequence. This approach and related memory-based or attention-based approaches have been successfully applied in diverse domains such as speech recognition, machine translation, image captioning, parsing, executing programs, and conversational dialogues BIBREF20 , BIBREF21 , BIBREF22 , BIBREF23 , BIBREF24 , BIBREF25 , BIBREF26 . Unlike previous work, we formulate question answering as a problem of decoding the KB query given the question and KB entries which are encoded in embedding spaces. We therefore integrate the learning of question and KB embeddings in a unified encoder-decoder framework, where the whole system is optimized end-to-end. Model Since we focus on single-relation question answering in this work, our model decodes every question into a KB query that consists of exactly two elements–the topic entity, and the predicate. More formally, our model is a function $f(q, \lbrace e\rbrace , \lbrace p\rbrace )$ that takes as input a question $q$ , a set of candidate entities $\lbrace e\rbrace =e_1, ...,e_n$ , a set of candidate predicates $\lbrace p\rbrace =p_1,..., p_m$ , and produces a likelihood score $p(e_i, p_j|q)$ of generating entity $e_i$ and predicate $p_j$ given question $q$ for all $i\in {1...n}, j\in {1...m}$ . As illustrated in Figure 1, our model consists of three components: The details of each component are described in the following sections. Encoding the Question To encode the question, we take two steps: We first extract one-hot encoding vectors for characters in the question, $x_1,...,x_n$ , where $x_i$ represents the one-hot encoding vector for the $i^{th}$ character in the question. We keep the space, punctuation and original cases without tokenization. We feed $x_1,...,x_n$ from left to right into a two-layer gated-feedback LSTM, and keep the outputs at all time steps as the embeddings for the question, i.e., these are the vectors $s_1,...,s_n$ . Encoding Entities and Predicates in the KB To encode an entity or predicate in the KB, we take two steps: We first extract one-hot encoding vectors for characters in its English alias, $x_1,...,x_n$ , where $x_i$ represents the one-hot encoding vector for the $i^{th}$ character in the alias. We then feed $x_1,...,x_n$ into a temporal CNN with two alternating convolutional and fully-connected layers, followed by one fully-connected layer: $ f(x_1,...,x_n) = tanh(W_{3} \times max(tanh (W_{2} \times \\ conv(tanh({W_{1} \times conv(x_1,...,x_n)}))))) $ where $f(x_{1...n}) $ is an embedding vector of size $N$ , $W_{3}$ has size $R^{N \times h}$ , $conv$ represents a temporal convolutional neural network, and $max$ represents a max pooling layer in the temporal direction. We use a CNN as opposed to an LSTM to embed KB entries primarily for computational efficiency. Also, we use two different CNNs to encode entities and predicates because they typically have significantly different styles (e.g., Barack Obama vs. /people/person/place_of_birth). Decoding the KB Query To generate the single topic entity and predicate to form the KB query, we use a decoder with two key components: An LSTM-based decoder with attention. Its hidden states at each time step $i$ , $h_{i}$ , have the same dimensionality $N$ as the embeddings of entities/predicates. The initial hidden state $h_0$ is set to the zero vector: $\vec{0}$ . A pairwise semantic relevance function that measures the similarity between the hidden units of the LSTM and the embedding of an entity or predicate candidate. It then returns the mostly likely entity or predicate based on the similarity score. In the following two sections, we will first describe the LSTM decoder with attention, followed by the semantic relevance function. The attention-based LSTM decoder uses a similar architecture as the one described in aligntranslate. At each time step $i$ , we feed in a context vector $c_i$ and an input vector $v_i$ into the LSTM. At time $i=1$ we feed a special input vector $v_{<{S}>}=\vec{0}$ into the LSTM. At time $i=2$ , during training, the input vector is the embedding of the true entity, while during testing, it is the embedding of the most likely entity as determined at the previous time step. We now describe how we produce the context vector $c_i$ . Let $h_{i-1}$ be the hidden state of the LSTM at time $i-1$ , $s_j$ be the $j^{th}$ question character embedding, $n$ be the number of characters in the question, $r$ be the size of $s_j$ , and $m$ be a hyperparameter. Then the context vector $c_i$ , which represents the attention-weighted content of the question, is recomputed at each time step $h_{i-1}$0 as follows: $h_{i-1}$1 $h_{i-1}$2 where $\lbrace \alpha \rbrace $ is the attention distribution that is applied over each hidden unit $s_j$ , $W_a \in R^{m \times N}, U_a \in R^{m \times r},$ and $v_a \in {R}^{1 \times m}$ . Unlike machine translation and language modeling where the vocabulary is relatively small, there are millions of entries in the KB. If we try to directly predict the KB entries, the decoder will need an output layer with millions of nodes, which is computationally prohibitive. Therefore, we resort to a relevance function that measures the semantic similarity between the decoder's hidden state and the embeddings of KB entries. Our semantic relevance function takes two vectors $x_1$ , $x_2$ and returns a distance measure of how similar they are to each other. In current experiments we use a simple cosine-similarity metric: $cos(x_1, x_2)$ . Using this similarity metric, the likelihoods of generating entity $e_j$ and predicate $p_k$ are: $ \hspace*{0.0pt} P(e_j) = \frac{exp(\lambda cos(h_1,e_{j}))}{\sum _{i=1}^{n} exp(\lambda cos(h_1,e_i))} \\ P(p_k) = \frac{exp(\lambda cos(h_2,p_{k}))}{\sum _{i=1}^{m} exp(\lambda cos(h_2,p_{i}))} $ where $\lambda $ is a constant, $h_1, h_2$ are the hidden states of the LSTM at times $t=1$ and $t=2$ , $e_1,...,e_n$ are the entity embeddings, and $p_1,...,p_m$ are the predicate embeddings. A similar likelihood function was used to train the semantic similarity modules proposed in qaacl and Yih2015SemanticPV. During inference, $e_1,...,e_n$ and $p_1,...,p_m$ are the embeddings of candidate entities and predicates. During training $e_1,...,e_n$ , $p_1,...,p_m$ are the embeddings of the true entity and 50 randomly-sampled entities, and the true predicate and 50 randomly-sampled predicates, respectively. Inference For each question $q$ , we generate a candidate set of entities and predicates, $\lbrace e\rbrace $ and $\lbrace p\rbrace $ , and feed it through the model $f(q, \lbrace e\rbrace , \lbrace p\rbrace )$ . We then decode the most likely (entity, predicate) pair: $ (e^*, p^*) = argmax_{e_i, p_j} (P(e_i)*P(p_j)) $ which becomes our semantic parse. We use a similar procedure as the one described in babidataset to generate candidate entities $\lbrace e\rbrace $ and predicates $\lbrace p\rbrace $ . Namely, we take all entities whose English alias is a substring of the question, and remove all entities whose alias is a substring of another entity. For each English alias, we sort each entity with this alias by the number of facts that it has in the KB, and append the top 10 entities from this list to our set of candidate entities. All predicates ${p_j}$ for each entity in our candidate entity set become the set of candidate predicates. Learning Our goal in learning is to maximize the joint likelihood $P(e_c) \cdot P(p_c)$ of predicting the correct entity $e_c$ and predicate $p_c$ pair from a set of randomly sampled entities and predicates. We use back-propagation to learn all of the weights in our model. All the parameters of our model are learned jointly without pre-training. These parameters include the weights of the character-level embeddings, CNNs, and LSTMs. Weights are randomly initialized before training. For the $i^{th}$ layer in our network, each weight is sampled from a uniform distribution between $-\frac{1}{|l^i|}$ and $\frac{1}{|l^i|}$ , where $|l^i|$ is the number of weights in layer $i$ . Dataset and Experimental Settings We evaluate the proposed model on the SimpleQuestions dataset BIBREF0 . The dataset consists of 108,442 single-relation questions and their corresponding (topic entity, predicate, answer entity) triples from Freebase. It is split into 75,910 train, 10,845 validation, and 21,687 test questions. Only 10,843 of the 45,335 unique words in entity aliases and 886 out of 1,034 unique predicates in the test set were present in the train set. For the proposed dataset, there are two evaluation settings, called FB2M and FB5M, respectively. The former uses a KB for candidate generation which is a subset of Freebase and contains 2M entities, while the latter uses subset of Freebase with 5M entities. In our experiments, the Memory Neural Networks (MemNNs) proposed in babidataset serve as the baselines. For training, in addition to the 76K questions in the training set, the MemNNs use 3K training questions from WebQuestions BIBREF27 , 15M paraphrases from WikiAnswers BIBREF2 , and 11M and 12M automatically generated questions from the KB for the FB2M and FB5M settings, respectively. In contrast, our models are trained only on the 76K questions in the training set. For our model, both layers of the LSTM-based question encoder have size 200. The hidden layers of the LSTM-based decoder have size 100, and the CNNs for entity and predicate embeddings have a hidden layer of size 200 and an output layer of size 100. The CNNs for entity and predicate embeddings use a receptive field of size 4, $\lambda =5$ , and $m=100$ . We train the models using RMSProp with a learning rate of $1e^{-4}$ . In order to make the input character sequence long enough to fill up the receptive fields of multiple CNN layers, we pad each predicate or entity using three padding symbols $P$ , a special start symbol, and a special end symbol. For instance, $Obama$ would become $S_{start}PPP ObamaPPPS_{end}$ . For consistency, we apply the same padding to the questions. End-to-end Results on SimpleQuestions Following babidataset, we report results on the SimpleQuestions dataset in terms of SQ accuracy, for both FB2M and FB5M settings in Table 1. SQ accuracy is defined as the percentage of questions for which the model generates a correct KB query (i.e., both the topic entity and predicate are correct). Our single character-level model achieves SQ accuracies of 70.9% and 70.3% on the FB2M and FB5M settings, outperforming the previous state-of-art results by 8.2% and 6.4%, respectively. Compared to the character-level model, which only has 1.2M parameters, our word-level model has 19.9M parameters, and only achieves a best SQ accuracy of 53.9%. In addition, in contrast to previous work, the OOV issue is much more severe for our word-level model, since we use no data augmentation to cover entities unseen in the train set. Ablation and Embedding Experiments We carry out ablation studies in Sections 5.2.1 and 5.2.2 through a set of random-sampling experiments. In these experiments, for each question, we randomly sample 200 entities and predicates from the test set as noise samples. We then mix the gold entity and predicate into these negative samples, and evaluate the accuracy of our model in predicting the gold predicate or entity from this mixed set. We first explore using word-level models as an alternative to character-level models to construct embeddings for questions, entities and predicates. Both word-level and character-level models perform comparably well when predicting the predicate, reaching an accuracy of around 80% (Table 3). However, the word-level model has considerable difficulty generalizing to unseen entities, and is only able to predict 45% of the entities accurately from the mixed set. These results clearly demonstrate that the OOV issue is much more severe for entities than predicates, and the difficulty word-level models have when generalizing to new entities. In contrast, character-level models have no such issues, and achieve a 96.6% accuracy in predicting the correct entity on the mixed set. This demonstrates that character-level models encode the semantic representation of entities and can match entity aliases in a KB with their mentions in natural language questions. We also study the impact of the depth of neural networks in our model. The results are presented in Table 2. In the ablation experiments we compare the performance of a single-layer LSTM to a two-layer LSTM to encode the question, and a single-layer vs. two-layer CNN to encode the KB entries. We find that a two-layer LSTM boosts joint accuracy by over 6%. The majority of accuracy gains are a result of improved predicate predictions, possibly because entity accuracy is already saturated in this experimental setup. Attention Mechanisms In order to further understand how the model performs question answering, we visualize the attention distribution over question characters in the decoding process. In each sub-figure of Figure 2, the x-axis is the character sequence of the question, and the y-axis is the attention weight distribution $\lbrace \alpha _i\rbrace $ . The blue curve is the attention distribution when generating the entity, and green curve is the attention distribution when generating the predicate. Interestingly, as the examples show, the attention distribution typically peaks at empty spaces. This indicates that the character-level model learns that a space defines an ending point of a complete linguistic unit. That is, the hidden state of the LSTM encoder at a space likely summarizes content about the character sequence before that space, and therefore contains important semantic information that the decoder needs to attend to. Also, we observe that entity attention distributions are usually less sharp and span longer portions of words, such as john or rutters, than predicate attention distributions (e.g., Figure 2a). For entities, semantic information may accumulate gradually when seeing more and more characters, while for predicates, semantic information will become clear only after seeing the complete word. For example, it may only be clear that characters such as song by refer to a predicate after a space, as opposed to the name of a song such as song bye bye love (Figures 2a, 2b). In contrast, a sequence of characters starts to become a likely entity after seeing an incomplete name such as joh or rutt. In addition, a character-level model can identify entities whose English aliases were never seen in training, such as phrenology (Figure 2d). The model apparently learns that words ending with the suffix nology are likely entity mentions, which is interesting because it reads in the input one character at a time. Furthermore, as observed in Figure 2d, the attention model is capable of attending disjoint regions of the question and capture the mention of a predicate that is interrupted by entity mentions. We also note that predicate attention often peaks at the padding symbols after the last character of the question, possibly because sentence endings carry extra information that further help disambiguate predicate mentions. In certain scenarios, the network may only have sufficient information to build a semantic representation of the predicate after being ensured that it reached the end of a sentence. Finally, certain words in the question help identify both the entity and the predicate. For example, consider the word university in the question What type of educational institution is eastern new mexico university (Figure 2c). Although it is a part of the entity mention, it also helps disambiguate the predicate. However, previous semantic parsing-based QA approaches BIBREF10 , BIBREF1 assume that there is a clear separation between the predicate and entity mentions in the question. In contrast, the proposed model does not need to make this hard categorization, and attends the word university when predicting both the entity and predicate. Error Analysis We randomly sampled 50 questions where the best-performing model generated the wrong KB query and categorized the errors. For 46 out of the 50 examples, the model predicted a predicate with a very similar alias to the true predicate, i.e. /music/release/track vs. /music/release/track_list. For 21 out of the 50 examples, the model predicted the wrong entity, e.g., Album vs. Still Here for the question What type of album is still here?. Finally, for 18 of the 50 examples, the model predicted the wrong entity and predicate, i.e. (Play, /freebase/equivalent_topic/equivalent_type) for the question which instrument does amapola cabase play? Training on more data, augmenting the negative sample set with words from the question that are not an entity mention, and having more examples that disambiguate between similar predicates may ameliorate many of these errors. Conclusion In this paper, we proposed a new character-level, attention-based encoder-decoder model for question answering. In our approach, embeddings of questions, entities, and predicates are all jointly learned to directly optimize the likelihood of generating the correct KB query. Our approach improved the state-of-the-art accuracy on the SimpleQuestions benchmark significantly, using much less data than previous work. Furthermore, thanks to character-level modeling, we have a compact model that is robust to unseen entities. Visualizations of the attention distribution reveal that our model, although built on character-level inputs, can learn higher-level semantic concepts required to answer a natural language question with a structured KB. In the future we would like to extend our system to handle multi-relation questions.
What word level and character level model baselines are used?
7705dd04acedaefee30d8b2c9978537afb2040dc
What Ġword Ġlevel Ġand Ġcharacter Ġlevel Ġmodel Ġbas elines Ġare Ġused ?
None
<s> What Ġword Ġlevel Ġand Ġcharacter Ġlevel Ġmodel Ġbas elines Ġare Ġused ? </s> Introduction </s> Single - relation Ġfact oid Ġquestions Ġare Ġthe Ġmost Ġcommon Ġform Ġof Ġquestions Ġfound Ġin Ġsearch Ġquery Ġlogs Ġand Ġcommunity Ġquestion Ġanswering Ġwebsites ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ. ĠA Ġknowledge - base Ġ( KB ) Ġsuch Ġas ĠFree base , ĠDB pedia , Ġor ĠWik id ata Ġcan Ġhelp Ġanswer Ġsuch Ġquestions Ġafter Ġusers Ġreform ulate Ġthem Ġas Ġqueries . ĠFor Ġinstance , Ġthe Ġquestion ĠWhere Ġwas ĠBarack ĠObama Ġborn ? Ġcan Ġbe Ġanswered Ġby Ġissuing Ġthe Ġfollowing ĠKB Ġquery : Ġ$ Ġ\ lambda Ġ( x ). place \ _ of \ _ birth ( Bar ack \ _ Obama , Ġx ) Ġ$ </s> However , Ġautomatically Ġmapping Ġa Ġnatural Ġlanguage Ġquestion Ġsuch Ġas ĠWhere Ġwas ĠBarack ĠObama Ġborn ? Ġto Ġits Ġcorresponding ĠKB Ġquery Ġremains Ġa Ġchallenging Ġtask . </s> There Ġare Ġthree Ġkey Ġissues Ġthat Ġmake Ġlearning Ġthis Ġmapping Ġnon - t riv ial . ĠFirst , Ġthere Ġare Ġmany Ġparaph r ases Ġof Ġthe Ġsame Ġquestion . ĠSecond , Ġmany Ġof Ġthe ĠKB Ġentries Ġare Ġunseen Ġduring Ġtraining Ġtime ; Ġhowever , Ġwe Ġstill Ġneed Ġto Ġcorrectly Ġpredict Ġthem Ġat Ġtest Ġtime . ĠThird , Ġa ĠKB Ġsuch Ġas ĠFree base Ġtypically Ġcontains Ġmillions Ġof Ġentities Ġand Ġthousands Ġof Ġpred icates , Ġmaking Ġit Ġdifficult Ġfor Ġa Ġsystem Ġto Ġpredict Ġthese Ġentities Ġat Ġscale ĠB IB REF 1 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 0 Ġ. ĠIn Ġthis Ġpaper , Ġwe Ġaddress Ġall Ġthree Ġof Ġthese Ġissues Ġwith Ġa Ġcharacter - level Ġenc oder - dec oder Ġframework Ġthat Ġsignificantly Ġimproves Ġperformance Ġover Ġstate - of - the - art Ġword - level Ġneural Ġmodels , Ġwhile Ġalso Ġproviding Ġa Ġmuch Ġmore Ġcompact Ġmodel Ġthat Ġcan Ġbe Ġlearned Ġfrom Ġless Ġdata . </s> First , Ġwe Ġuse Ġa Ġlong Ġshort - term Ġmemory Ġ( L ST M ) ĠB IB REF 4 Ġenc oder Ġto Ġembed Ġthe Ġquestion . ĠSecond , Ġto Ġmake Ġour Ġmodel Ġrobust Ġto Ġunseen ĠKB Ġentries , Ġwe Ġextract Ġembed d ings Ġfor Ġquestions , Ġpred icates Ġand Ġentities Ġpurely Ġfrom Ġtheir Ġcharacter - level Ġrepresentations . ĠCharacter - level Ġmodeling Ġhas Ġbeen Ġpreviously Ġshown Ġto Ġgeneral ize Ġwell Ġto Ġnew Ġwords Ġnot Ġseen Ġduring Ġtraining ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, Ġwhich Ġmakes Ġit Ġideal Ġfor Ġthis Ġtask . ĠThird , Ġto Ġscale Ġour Ġmodel Ġto Ġhandle Ġthe Ġmillions Ġof Ġentities Ġand Ġthousands Ġof Ġpred icates Ġin Ġthe ĠKB , Ġinstead Ġof Ġusing Ġa Ġlarge Ġoutput Ġlayer Ġin Ġthe Ġdec oder Ġto Ġdirectly Ġpredict Ġthe Ġentity Ġand Ġpredicate , Ġwe Ġuse Ġa Ġgeneral Ġinteraction Ġfunction Ġbetween Ġthe Ġquestion Ġembed d ings Ġand ĠKB Ġembed d ings Ġthat Ġmeasures Ġtheir Ġsemantic Ġrelevance Ġto Ġdetermine Ġthe Ġoutput . ĠThe Ġcombined Ġuse Ġof Ġcharacter - level Ġmodeling Ġand Ġa Ġsemantic Ġrelevance Ġfunction Ġallows Ġus Ġto Ġsuccessfully Ġproduce Ġlikelihood Ġscores Ġfor Ġthe ĠKB Ġentries Ġthat Ġare Ġnot Ġpresent Ġin Ġour Ġvocabulary , Ġa Ġchallenging Ġtask Ġfor Ġstandard Ġenc oder - dec oder Ġframeworks . </s> Our Ġnovel , Ġcharacter - level Ġenc oder - dec oder Ġmodel Ġis Ġcompact , Ġrequires Ġsignificantly Ġless Ġdata Ġto Ġtrain Ġthan Ġprevious Ġwork , Ġand Ġis Ġable Ġto Ġgeneral ize Ġwell Ġto Ġunseen Ġentities Ġin Ġtest Ġtime . ĠIn Ġparticular , Ġwithout Ġuse Ġof Ġen semb les , Ġwe Ġachieve Ġ70 . 9 % Ġaccuracy Ġin Ġthe ĠFree base 2 M Ġsetting Ġand Ġ70 . 3 % Ġaccuracy Ġin Ġthe ĠFree base 5 M Ġsetting Ġon Ġthe ĠSimple Questions Ġdataset , Ġoutper forming Ġthe Ġprevious Ġstate - of - arts Ġof Ġ62 . 7 % Ġand Ġ63 . 9 % ĠB IB REF 0 Ġby Ġ8 . 2 % Ġand Ġ6 . 4 % Ġrespectively . ĠMoreover , Ġwe Ġonly Ġuse Ġthe Ġtraining Ġquestions Ġprovided Ġin ĠSimple Questions Ġto Ġtrain Ġour Ġmodel , Ġwhich Ġcover Ġabout Ġ24 % Ġof Ġwords Ġin Ġentity Ġaliases Ġon Ġthe Ġtest Ġset . ĠThis Ġdemonstrates Ġthe Ġrobust ness Ġof Ġthe Ġcharacter - level Ġmodel Ġto Ġunseen Ġentities . ĠIn Ġcontrast , Ġdata Ġaug mentation Ġis Ġusually Ġnecessary Ġto Ġprovide Ġmore Ġcoverage Ġfor Ġunseen Ġentities Ġand Ġpred icates , Ġas Ġdone Ġin Ġprevious Ġwork ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ. </s> Related ĠWork </s> Our Ġwork Ġis Ġmotivated Ġby Ġthree Ġmajor Ġthreads Ġof Ġresearch Ġin Ġmachine Ġlearning Ġand Ġnatural Ġlanguage Ġprocessing : Ġsemantic - p ars ing Ġfor Ġopen - domain Ġquestion Ġanswering , Ġcharacter - level Ġlanguage Ġmodeling , Ġand Ġenc oder - dec oder Ġmethods . </s> Sem antic Ġparsing Ġfor Ġopen - domain Ġquestion Ġanswering , Ġwhich Ġtranslates Ġa Ġquestion Ġinto Ġa Ġstructured ĠKB Ġquery , Ġis Ġa Ġkey Ġcomponent Ġin Ġquestion Ġanswering Ġwith Ġa ĠKB . ĠWhile Ġearly Ġapproaches Ġrelied Ġon Ġbuilding Ġhigh - quality Ġlex icons Ġfor Ġdomain - specific Ġdatabases Ġsuch Ġas ĠGeo Query ĠB IB REF 7 Ġ, Ġrecent Ġwork Ġhas Ġfocused Ġon Ġbuilding Ġsemantic Ġparsing Ġframeworks Ġfor Ġgeneral Ġknowledge Ġbases Ġsuch Ġas ĠFree base ĠB IB REF 1 Ġ, ĠB IB REF 8 Ġ, ĠB IB REF 0 Ġ, ĠB IB REF 9 Ġ, ĠB IB REF 2 Ġ. </s> Sem antic Ġparsing Ġframeworks Ġfor Ġlarge - scale Ġknowledge Ġbases Ġhave Ġto Ġbe Ġable Ġto Ġsuccessfully Ġgenerate Ġqueries Ġfor Ġthe Ġmillions Ġof Ġentities Ġand Ġthousands Ġof Ġpred icates Ġin Ġthe ĠKB , Ġmany Ġof Ġwhich Ġare Ġunseen Ġduring Ġtraining . ĠTo Ġaddress Ġthis Ġissue , Ġrecent Ġwork Ġrelies Ġon Ġproducing Ġembed d ings Ġfor Ġpred icates Ġand Ġentities Ġin Ġa ĠKB Ġbased Ġon Ġtheir Ġtextual Ġdescriptions ĠB IB REF 8 Ġ, ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 10 Ġ. ĠA Ġgeneral Ġinteraction Ġfunction Ġcan Ġthen Ġbe Ġused Ġto Ġmeasure Ġthe Ġsemantic Ġrelevance Ġof Ġthese Ġembedded ĠKB Ġentries Ġto Ġthe Ġquestion Ġand Ġdetermine Ġthe Ġmost Ġlikely ĠKB Ġquery . </s> Most Ġof Ġthese Ġapproaches Ġuse Ġword - level Ġembed d ings Ġto Ġencode Ġentities Ġand Ġpred icates , Ġand Ġtherefore Ġmight Ġsuffer Ġfrom Ġthe Ġout - of - voc abulary Ġ( OO V ) Ġproblem Ġwhen Ġthey Ġencounter Ġunseen Ġwords Ġduring Ġtest Ġtime . ĠConsequently , Ġthey Ġoften Ġrely Ġon Ġsignificant Ġdata Ġaug mentation Ġfrom Ġsources Ġsuch Ġas ĠPar ale x ĠB IB REF 2 Ġ, Ġwhich Ġcontains Ġ18 Ġmillion Ġquestion - par aph rase Ġpairs Ġscrap ed Ġfrom ĠWiki An swers , Ġto Ġhave Ġsufficient Ġexamples Ġfor Ġeach Ġword Ġthey Ġencounter ĠB IB REF 11 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 0 Ġ. </s> As Ġopposed Ġto Ġword - level Ġmodeling , Ġcharacter - level Ġmodeling Ġcan Ġbe Ġused Ġto Ġhandle Ġthe ĠO OV Ġissue . ĠWhile Ġcharacter - level Ġmodeling Ġhas Ġnot Ġbeen Ġapplied Ġto Ġfact oid Ġquestion Ġanswering Ġbefore , Ġit Ġhas Ġbeen Ġsuccessfully Ġapplied Ġto Ġinformation Ġretrieval , Ġmachine Ġtranslation , Ġsentiment Ġanalysis , Ġclassification , Ġand Ġnamed Ġentity Ġrecognition ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ, ĠB IB REF 19 Ġ. ĠMoreover , Ġg fl st m Ġdemonstrate Ġthat Ġg ated - feed back ĠL ST Ms Ġon Ġtop Ġof Ġcharacter - level Ġembed d ings Ġcan Ġcapture Ġlong - term Ġdependencies Ġin Ġlanguage Ġmodeling . </s> Lastly , Ġenc oder - dec oder Ġnetworks Ġhave Ġbeen Ġapplied Ġto Ġmany Ġstructured Ġmachine Ġlearning Ġtasks . ĠFirst Ġintroduced Ġin Ġs uts ke ver 2014 sequence , Ġin Ġan Ġenc oder - dec oder Ġnetwork , Ġa Ġsource Ġsequence Ġis Ġfirst Ġencoded Ġwith Ġa Ġrecurrent Ġneural Ġnetwork Ġ( R NN ) Ġinto Ġa Ġfixed - length Ġvector Ġwhich Ġintuitive ly Ġcaptures Ġits Ġmeaning , Ġand Ġthen Ġdec oded Ġinto Ġa Ġdesired Ġtarget Ġsequence . ĠThis Ġapproach Ġand Ġrelated Ġmemory - based Ġor Ġattention - based Ġapproaches Ġhave Ġbeen Ġsuccessfully Ġapplied Ġin Ġdiverse Ġdomains Ġsuch Ġas Ġspeech Ġrecognition , Ġmachine Ġtranslation , Ġimage Ġcaption ing , Ġparsing , Ġexecuting Ġprograms , Ġand Ġconvers ational Ġdialog ues ĠB IB REF 20 Ġ, ĠB IB REF 21 Ġ, ĠB IB REF 22 Ġ, ĠB IB REF 23 Ġ, ĠB IB REF 24 Ġ, ĠB IB REF 25 Ġ, ĠB IB REF 26 Ġ. </s> Unlike Ġprevious Ġwork , Ġwe Ġformulate Ġquestion Ġanswering Ġas Ġa Ġproblem Ġof Ġdecoding Ġthe ĠKB Ġquery Ġgiven Ġthe Ġquestion Ġand ĠKB Ġentries Ġwhich Ġare Ġencoded Ġin Ġembed ding Ġspaces . ĠWe Ġtherefore Ġintegrate Ġthe Ġlearning Ġof Ġquestion Ġand ĠKB Ġembed d ings Ġin Ġa Ġunified Ġenc oder - dec oder Ġframework , Ġwhere Ġthe Ġwhole Ġsystem Ġis Ġoptimized Ġend - to - end . </s> Model </s> Since Ġwe Ġfocus Ġon Ġsingle - relation Ġquestion Ġanswering Ġin Ġthis Ġwork , Ġour Ġmodel Ġdec odes Ġevery Ġquestion Ġinto Ġa ĠKB Ġquery Ġthat Ġconsists Ġof Ġexactly Ġtwo Ġelements âĢĵ the Ġtopic Ġentity , Ġand Ġthe Ġpredicate . ĠMore Ġformally , Ġour Ġmodel Ġis Ġa Ġfunction Ġ$ f ( q , Ġ\ l brace Ġe \ r brace Ġ, Ġ\ l brace Ġp \ r brace Ġ) $ Ġthat Ġtakes Ġas Ġinput Ġa Ġquestion Ġ$ q $ Ġ, Ġa Ġset Ġof Ġcandidate Ġentities Ġ$\ l brace Ġe \ r brace Ġ= e _ 1 , Ġ... , e _ n $ Ġ, Ġa Ġset Ġof Ġcandidate Ġpred icates Ġ$\ l brace Ġp \ r brace Ġ= p _ 1 ,... , Ġp _ m $ Ġ, Ġand Ġproduces Ġa Ġlikelihood Ġscore Ġ$ p ( e _ i , Ġp _ j | q ) $ Ġof Ġgenerating Ġentity Ġ$ e _ i $ Ġand Ġpredicate Ġ$ p _ j $ Ġgiven Ġquestion Ġ$ q $ Ġfor Ġall Ġ$ i \ in Ġ{ 1 ... n }, Ġj \ in Ġ{ 1 ... m } $ Ġ. </s> As Ġillustrated Ġin ĠFigure Ġ1 , Ġour Ġmodel Ġconsists Ġof Ġthree Ġcomponents : </s> The Ġdetails Ġof Ġeach Ġcomponent Ġare Ġdescribed Ġin Ġthe Ġfollowing Ġsections . </s> Enc oding Ġthe ĠQuestion </s> To Ġencode Ġthe Ġquestion , Ġwe Ġtake Ġtwo Ġsteps : </s> We Ġfirst Ġextract Ġone - hot Ġencoding Ġvectors Ġfor Ġcharacters Ġin Ġthe Ġquestion , Ġ$ x _ 1 ,... , x _ n $ Ġ, Ġwhere Ġ$ x _ i $ Ġrepresents Ġthe Ġone - hot Ġencoding Ġvector Ġfor Ġthe Ġ$ i ^{ th } $ Ġcharacter Ġin Ġthe Ġquestion . ĠWe Ġkeep Ġthe Ġspace , Ġpunct uation Ġand Ġoriginal Ġcases Ġwithout Ġtoken ization . </s> We Ġfeed Ġ$ x _ 1 ,... , x _ n $ Ġfrom Ġleft Ġto Ġright Ġinto Ġa Ġtwo - layer Ġg ated - feed back ĠL ST M , Ġand Ġkeep Ġthe Ġoutputs Ġat Ġall Ġtime Ġsteps Ġas Ġthe Ġembed d ings Ġfor Ġthe Ġquestion , Ġi . e ., Ġthese Ġare Ġthe Ġvectors Ġ$ s _ 1 ,... , s _ n $ Ġ. </s> Enc oding ĠEnt ities Ġand ĠPred icates Ġin Ġthe ĠKB </s> To Ġencode Ġan Ġentity Ġor Ġpredicate Ġin Ġthe ĠKB , Ġwe Ġtake Ġtwo Ġsteps : </s> We Ġfirst Ġextract Ġone - hot Ġencoding Ġvectors Ġfor Ġcharacters Ġin Ġits ĠEnglish Ġalias , Ġ$ x _ 1 ,... , x _ n $ Ġ, Ġwhere Ġ$ x _ i $ Ġrepresents Ġthe Ġone - hot Ġencoding Ġvector Ġfor Ġthe Ġ$ i ^{ th } $ Ġcharacter Ġin Ġthe Ġalias . </s> We Ġthen Ġfeed Ġ$ x _ 1 ,... , x _ n $ Ġinto Ġa Ġtemporal ĠCNN Ġwith Ġtwo Ġalternating Ġconv olution al Ġand Ġfully - connected Ġlayers , Ġfollowed Ġby Ġone Ġfully - connected Ġlayer : Ġ$ Ġf ( x _ 1 ,... , x _ n ) Ġ= Ġtan h ( W _{ 3 } Ġ\ times Ġmax ( tan h Ġ( W _{ 2 } Ġ\ times Ġ\\ Ġconv ( tan h ({ W _{ 1 } Ġ\ times Ġconv ( x _ 1 ,... , x _ n )} )) ))) Ġ$ </s> where Ġ$ f ( x _{ 1 ... n }) Ġ$ Ġis Ġan Ġembed ding Ġvector Ġof Ġsize Ġ$ N $ Ġ, Ġ$ W _{ 3 } $ Ġhas Ġsize Ġ$ R ^{ N Ġ\ times Ġh } $ Ġ, Ġ$ conv $ Ġrepresents Ġa Ġtemporal Ġconv olution al Ġneural Ġnetwork , Ġand Ġ$ max $ Ġrepresents Ġa Ġmax Ġpool ing Ġlayer Ġin Ġthe Ġtemporal Ġdirection . </s> We Ġuse Ġa ĠCNN Ġas Ġopposed Ġto Ġan ĠL ST M Ġto Ġembed ĠKB Ġentries Ġprimarily Ġfor Ġcomputational Ġefficiency . ĠAlso , Ġwe Ġuse Ġtwo Ġdifferent ĠCNN s Ġto Ġencode Ġentities Ġand Ġpred icates Ġbecause Ġthey Ġtypically Ġhave Ġsignificantly Ġdifferent Ġstyles Ġ( e . g ., ĠBarack ĠObama Ġvs . Ġ/ people / person / place _ of _ birth ). </s> Dec oding Ġthe ĠKB ĠQuery </s> To Ġgenerate Ġthe Ġsingle Ġtopic Ġentity Ġand Ġpredicate Ġto Ġform Ġthe ĠKB Ġquery , Ġwe Ġuse Ġa Ġdec oder Ġwith Ġtwo Ġkey Ġcomponents : </s> An ĠL ST M - based Ġdec oder Ġwith Ġattention . ĠIts Ġhidden Ġstates Ġat Ġeach Ġtime Ġstep Ġ$ i $ Ġ, Ġ$ h _{ i } $ Ġ, Ġhave Ġthe Ġsame Ġdimension ality Ġ$ N $ Ġas Ġthe Ġembed d ings Ġof Ġentities / pred icates . ĠThe Ġinitial Ġhidden Ġstate Ġ$ h _ 0 $ Ġis Ġset Ġto Ġthe Ġzero Ġvector : Ġ$\ vec { 0 } $ Ġ. </s> A Ġpair wise Ġsemantic Ġrelevance Ġfunction Ġthat Ġmeasures Ġthe Ġsimilarity Ġbetween Ġthe Ġhidden Ġunits Ġof Ġthe ĠL ST M Ġand Ġthe Ġembed ding Ġof Ġan Ġentity Ġor Ġpredicate Ġcandidate . ĠIt Ġthen Ġreturns Ġthe Ġmostly Ġlikely Ġentity Ġor Ġpredicate Ġbased Ġon Ġthe Ġsimilarity Ġscore . </s> In Ġthe Ġfollowing Ġtwo Ġsections , Ġwe Ġwill Ġfirst Ġdescribe Ġthe ĠL ST M Ġdec oder Ġwith Ġattention , Ġfollowed Ġby Ġthe Ġsemantic Ġrelevance Ġfunction . </s> The Ġattention - based ĠL ST M Ġdec oder Ġuses Ġa Ġsimilar Ġarchitecture Ġas Ġthe Ġone Ġdescribed Ġin Ġal ignt rans late . ĠAt Ġeach Ġtime Ġstep Ġ$ i $ Ġ, Ġwe Ġfeed Ġin Ġa Ġcontext Ġvector Ġ$ c _ i $ Ġand Ġan Ġinput Ġvector Ġ$ v _ i $ Ġinto Ġthe ĠL ST M . ĠAt Ġtime Ġ$ i = 1 $ Ġwe Ġfeed Ġa Ġspecial Ġinput Ġvector Ġ$ v _{ < { S } > } = \ vec { 0 } $ Ġinto Ġthe ĠL ST M . ĠAt Ġtime Ġ$ i = 2 $ Ġ, Ġduring Ġtraining , Ġthe Ġinput Ġvector Ġis Ġthe Ġembed ding Ġof Ġthe Ġtrue Ġentity , Ġwhile Ġduring Ġtesting , Ġit Ġis Ġthe Ġembed ding Ġof Ġthe Ġmost Ġlikely Ġentity Ġas Ġdetermined Ġat Ġthe Ġprevious Ġtime Ġstep . </s> We Ġnow Ġdescribe Ġhow Ġwe Ġproduce Ġthe Ġcontext Ġvector Ġ$ c _ i $ Ġ. ĠLet Ġ$ h _{ i - 1 } $ Ġbe Ġthe Ġhidden Ġstate Ġof Ġthe ĠL ST M Ġat Ġtime Ġ$ i - 1 $ Ġ, Ġ$ s _ j $ Ġbe Ġthe Ġ$ j ^{ th } $ Ġquestion Ġcharacter Ġembed ding , Ġ$ n $ Ġbe Ġthe Ġnumber Ġof Ġcharacters Ġin Ġthe Ġquestion , Ġ$ r $ Ġbe Ġthe Ġsize Ġof Ġ$ s _ j $ Ġ, Ġand Ġ$ m $ Ġbe Ġa Ġhyper param eter . ĠThen Ġthe Ġcontext Ġvector Ġ$ c _ i $ Ġ, Ġwhich Ġrepresents Ġthe Ġattention - weight ed Ġcontent Ġof Ġthe Ġquestion , Ġis Ġrec om puted Ġat Ġeach Ġtime Ġstep Ġ$ h _{ i - 1 } $ 0 Ġas Ġfollows : Ġ$ h _{ i - 1 } $ 1 Ġ$ h _{ i - 1 } $ 2 </s> where Ġ$\ l brace Ġ\ alpha Ġ\ r brace Ġ$ Ġis Ġthe Ġattention Ġdistribution Ġthat Ġis Ġapplied Ġover Ġeach Ġhidden Ġunit Ġ$ s _ j $ Ġ, Ġ$ W _ a Ġ\ in ĠR ^{ m Ġ\ times ĠN }, ĠU _ a Ġ\ in ĠR ^{ m Ġ\ times Ġr }, $ Ġand Ġ$ v _ a Ġ\ in Ġ{ R } ^{ 1 Ġ\ times Ġm } $ Ġ. </s> Unlike Ġmachine Ġtranslation Ġand Ġlanguage Ġmodeling Ġwhere Ġthe Ġvocabulary Ġis Ġrelatively Ġsmall , Ġthere Ġare Ġmillions Ġof Ġentries Ġin Ġthe ĠKB . ĠIf Ġwe Ġtry Ġto Ġdirectly Ġpredict Ġthe ĠKB Ġentries , Ġthe Ġdec oder Ġwill Ġneed Ġan Ġoutput Ġlayer Ġwith Ġmillions Ġof Ġnodes , Ġwhich Ġis Ġcomput ationally Ġprohib itive . ĠTherefore , Ġwe Ġresort Ġto Ġa Ġrelevance Ġfunction Ġthat Ġmeasures Ġthe Ġsemantic Ġsimilarity Ġbetween Ġthe Ġdec oder 's Ġhidden Ġstate Ġand Ġthe Ġembed d ings Ġof ĠKB Ġentries . ĠOur Ġsemantic Ġrelevance Ġfunction Ġtakes Ġtwo Ġvectors Ġ$ x _ 1 $ Ġ, Ġ$ x _ 2 $ Ġand Ġreturns Ġa Ġdistance Ġmeasure Ġof Ġhow Ġsimilar Ġthey Ġare Ġto Ġeach Ġother . ĠIn Ġcurrent Ġexperiments Ġwe Ġuse Ġa Ġsimple Ġcos ine - similar ity Ġmetric : Ġ$ cos ( x _ 1 , Ġx _ 2 ) $ Ġ. </s> Using Ġthis Ġsimilarity Ġmetric , Ġthe Ġlikelihood s Ġof Ġgenerating Ġentity Ġ$ e _ j $ Ġand Ġpredicate Ġ$ p _ k $ Ġare : Ġ$ Ġ\ h space * { 0 . 0 pt } ĠP ( e _ j ) Ġ= Ġ\ frac { exp (\ lambda Ġcos ( h _ 1 , e _{ j } )) }{ \ sum Ġ_ { i = 1 } ^{ n } Ġexp (\ lambda Ġcos ( h _ 1 , e _ i )) } Ġ\\ ĠP ( p _ k ) Ġ= Ġ\ frac { exp (\ lambda Ġcos ( h _ 2 , p _{ k } )) }{ \ sum Ġ_ { i = 1 } ^{ m } Ġexp (\ lambda Ġcos ( h _ 2 , p _{ i } )) } Ġ$ </s> where Ġ$\ lambda Ġ$ Ġis Ġa Ġconstant , Ġ$ h _ 1 , Ġh _ 2 $ Ġare Ġthe Ġhidden Ġstates Ġof Ġthe ĠL ST M Ġat Ġtimes Ġ$ t = 1 $ Ġand Ġ$ t = 2 $ Ġ, Ġ$ e _ 1 ,... , e _ n $ Ġare Ġthe Ġentity Ġembed d ings , Ġand Ġ$ p _ 1 ,... , p _ m $ Ġare Ġthe Ġpredicate Ġembed d ings . ĠA Ġsimilar Ġlikelihood Ġfunction Ġwas Ġused Ġto Ġtrain Ġthe Ġsemantic Ġsimilarity Ġmodules Ġproposed Ġin Ġq a acl Ġand ĠY ih 2015 Sem antic P V . </s> During Ġinference , Ġ$ e _ 1 ,... , e _ n $ Ġand Ġ$ p _ 1 ,... , p _ m $ Ġare Ġthe Ġembed d ings Ġof Ġcandidate Ġentities Ġand Ġpred icates . ĠDuring Ġtraining Ġ$ e _ 1 ,... , e _ n $ Ġ, Ġ$ p _ 1 ,... , p _ m $ Ġare Ġthe Ġembed d ings Ġof Ġthe Ġtrue Ġentity Ġand Ġ50 Ġrandomly - sam pled Ġentities , Ġand Ġthe Ġtrue Ġpredicate Ġand Ġ50 Ġrandomly - sam pled Ġpred icates , Ġrespectively . </s> In ference </s> For Ġeach Ġquestion Ġ$ q $ Ġ, Ġwe Ġgenerate Ġa Ġcandidate Ġset Ġof Ġentities Ġand Ġpred icates , Ġ$\ l brace Ġe \ r brace Ġ$ Ġand Ġ$\ l brace Ġp \ r brace Ġ$ Ġ, Ġand Ġfeed Ġit Ġthrough Ġthe Ġmodel Ġ$ f ( q , Ġ\ l brace Ġe \ r brace Ġ, Ġ\ l brace Ġp \ r brace Ġ) $ Ġ. ĠWe Ġthen Ġdecode Ġthe Ġmost Ġlikely Ġ( entity , Ġpredicate ) Ġpair : Ġ$ Ġ( e ^ *, Ġp ^ *) Ġ= Ġarg max _{ e _ i , Ġp _ j } Ġ( P ( e _ i )* P ( p _ j )) Ġ$ </s> which Ġbecomes Ġour Ġsemantic Ġparse . </s> We Ġuse Ġa Ġsimilar Ġprocedure Ġas Ġthe Ġone Ġdescribed Ġin Ġbab id at as et Ġto Ġgenerate Ġcandidate Ġentities Ġ$\ l brace Ġe \ r brace Ġ$ Ġand Ġpred icates Ġ$\ l brace Ġp \ r brace Ġ$ Ġ. ĠName ly , Ġwe Ġtake Ġall Ġentities Ġwhose ĠEnglish Ġalias Ġis Ġa Ġsubst ring Ġof Ġthe Ġquestion , Ġand Ġremove Ġall Ġentities Ġwhose Ġalias Ġis Ġa Ġsubst ring Ġof Ġanother Ġentity . ĠFor Ġeach ĠEnglish Ġalias , Ġwe Ġsort Ġeach Ġentity Ġwith Ġthis Ġalias Ġby Ġthe Ġnumber Ġof Ġfacts Ġthat Ġit Ġhas Ġin Ġthe ĠKB , Ġand Ġappend Ġthe Ġtop Ġ10 Ġentities Ġfrom Ġthis Ġlist Ġto Ġour Ġset Ġof Ġcandidate Ġentities . ĠAll Ġpred icates Ġ${ p _ j } $ Ġfor Ġeach Ġentity Ġin Ġour Ġcandidate Ġentity Ġset Ġbecome Ġthe Ġset Ġof Ġcandidate Ġpred icates . </s> Learning </s> Our Ġgoal Ġin Ġlearning Ġis Ġto Ġmaximize Ġthe Ġjoint Ġlikelihood Ġ$ P ( e _ c ) Ġ\ cd ot ĠP ( p _ c ) $ Ġof Ġpredicting Ġthe Ġcorrect Ġentity Ġ$ e _ c $ Ġand Ġpredicate Ġ$ p _ c $ Ġpair Ġfrom Ġa Ġset Ġof Ġrandomly Ġsampled Ġentities Ġand Ġpred icates . ĠWe Ġuse Ġback - prop ag ation Ġto Ġlearn Ġall Ġof Ġthe Ġweights Ġin Ġour Ġmodel . </s> All Ġthe Ġparameters Ġof Ġour Ġmodel Ġare Ġlearned Ġjointly Ġwithout Ġpre - training . ĠThese Ġparameters Ġinclude Ġthe Ġweights Ġof Ġthe Ġcharacter - level Ġembed d ings , ĠCNN s , Ġand ĠL ST Ms . ĠWe ights Ġare Ġrandomly Ġinitialized Ġbefore Ġtraining . ĠFor Ġthe Ġ$ i ^{ th } $ Ġlayer Ġin Ġour Ġnetwork , Ġeach Ġweight Ġis Ġsampled Ġfrom Ġa Ġuniform Ġdistribution Ġbetween Ġ$ - \ frac { 1 }{ | l ^ i | } $ Ġand Ġ$\ frac { 1 }{ | l ^ i | } $ Ġ, Ġwhere Ġ$ | l ^ i | $ Ġis Ġthe Ġnumber Ġof Ġweights Ġin Ġlayer Ġ$ i $ Ġ. </s> Dat as et Ġand ĠExperimental ĠSettings </s> We Ġevaluate Ġthe Ġproposed Ġmodel Ġon Ġthe ĠSimple Questions Ġdataset ĠB IB REF 0 Ġ. ĠThe Ġdataset Ġconsists Ġof Ġ108 , 442 Ġsingle - relation Ġquestions Ġand Ġtheir Ġcorresponding Ġ( topic Ġentity , Ġpredicate , Ġanswer Ġentity ) Ġtri ples Ġfrom ĠFree base . ĠIt Ġis Ġsplit Ġinto Ġ75 , 910 Ġtrain , Ġ10 , 8 45 Ġvalidation , Ġand Ġ21 , 687 Ġtest Ġquestions . ĠOnly Ġ10 , 8 43 Ġof Ġthe Ġ45 , 335 Ġunique Ġwords Ġin Ġentity Ġaliases Ġand Ġ8 86 Ġout Ġof Ġ1 , 034 Ġunique Ġpred icates Ġin Ġthe Ġtest Ġset Ġwere Ġpresent Ġin Ġthe Ġtrain Ġset . ĠFor Ġthe Ġproposed Ġdataset , Ġthere Ġare Ġtwo Ġevaluation Ġsettings , Ġcalled ĠFB 2 M Ġand ĠFB 5 M , Ġrespectively . ĠThe Ġformer Ġuses Ġa ĠKB Ġfor Ġcandidate Ġgeneration Ġwhich Ġis Ġa Ġsubset Ġof ĠFree base Ġand Ġcontains Ġ2 M Ġentities , Ġwhile Ġthe Ġlatter Ġuses Ġsubset Ġof ĠFree base Ġwith Ġ5 M Ġentities . </s> In Ġour Ġexperiments , Ġthe ĠMemory ĠNeural ĠNetworks Ġ( Mem NN s ) Ġproposed Ġin Ġbab id at as et Ġserve Ġas Ġthe Ġbas elines . ĠFor Ġtraining , Ġin Ġaddition Ġto Ġthe Ġ76 K Ġquestions Ġin Ġthe Ġtraining Ġset , Ġthe ĠMem NN s Ġuse Ġ3 K Ġtraining Ġquestions Ġfrom ĠWeb Questions ĠB IB REF 27 Ġ, Ġ15 M Ġparaph r ases Ġfrom ĠWiki An swers ĠB IB REF 2 Ġ, Ġand Ġ11 M Ġand Ġ12 M Ġautomatically Ġgenerated Ġquestions Ġfrom Ġthe ĠKB Ġfor Ġthe ĠFB 2 M Ġand ĠFB 5 M Ġsettings , Ġrespectively . ĠIn Ġcontrast , Ġour Ġmodels Ġare Ġtrained Ġonly Ġon Ġthe Ġ76 K Ġquestions Ġin Ġthe Ġtraining Ġset . </s> For Ġour Ġmodel , Ġboth Ġlayers Ġof Ġthe ĠL ST M - based Ġquestion Ġenc oder Ġhave Ġsize Ġ200 . ĠThe Ġhidden Ġlayers Ġof Ġthe ĠL ST M - based Ġdec oder Ġhave Ġsize Ġ100 , Ġand Ġthe ĠCNN s Ġfor Ġentity Ġand Ġpredicate Ġembed d ings Ġhave Ġa Ġhidden Ġlayer Ġof Ġsize Ġ200 Ġand Ġan Ġoutput Ġlayer Ġof Ġsize Ġ100 . ĠThe ĠCNN s Ġfor Ġentity Ġand Ġpredicate Ġembed d ings Ġuse Ġa Ġreceptive Ġfield Ġof Ġsize Ġ4 , Ġ$\ lambda Ġ= 5 $ Ġ, Ġand Ġ$ m = 100 $ Ġ. ĠWe Ġtrain Ġthe Ġmodels Ġusing ĠRM SP rop Ġwith Ġa Ġlearning Ġrate Ġof Ġ$ 1 e ^{ - 4 } $ Ġ. </s> In Ġorder Ġto Ġmake Ġthe Ġinput Ġcharacter Ġsequence Ġlong Ġenough Ġto Ġfill Ġup Ġthe Ġreceptive Ġfields Ġof Ġmultiple ĠCNN Ġlayers , Ġwe Ġpad Ġeach Ġpredicate Ġor Ġentity Ġusing Ġthree Ġpadding Ġsymbols Ġ$ P $ Ġ, Ġa Ġspecial Ġstart Ġsymbol , Ġand Ġa Ġspecial Ġend Ġsymbol . ĠFor Ġinstance , Ġ$ Obama $ Ġwould Ġbecome Ġ$ S _{ start } PP P ĠObama PP PS _{ end } $ Ġ. ĠFor Ġconsistency , Ġwe Ġapply Ġthe Ġsame Ġpadding Ġto Ġthe Ġquestions . </s> End - to - end ĠResults Ġon ĠSimple Questions </s> Following Ġbab id at as et , Ġwe Ġreport Ġresults Ġon Ġthe ĠSimple Questions Ġdataset Ġin Ġterms Ġof ĠSQ Ġaccuracy , Ġfor Ġboth ĠFB 2 M Ġand ĠFB 5 M Ġsettings Ġin ĠTable Ġ1 . ĠSQ Ġaccuracy Ġis Ġdefined Ġas Ġthe Ġpercentage Ġof Ġquestions Ġfor Ġwhich Ġthe Ġmodel Ġgenerates Ġa Ġcorrect ĠKB Ġquery Ġ( i . e ., Ġboth Ġthe Ġtopic Ġentity Ġand Ġpredicate Ġare Ġcorrect ). ĠOur Ġsingle Ġcharacter - level Ġmodel Ġachieves ĠSQ Ġaccur acies Ġof Ġ70 . 9 % Ġand Ġ70 . 3 % Ġon Ġthe ĠFB 2 M Ġand ĠFB 5 M Ġsettings , Ġoutper forming Ġthe Ġprevious Ġstate - of - art Ġresults Ġby Ġ8 . 2 % Ġand Ġ6 . 4 %, Ġrespectively . ĠCompared Ġto Ġthe Ġcharacter - level Ġmodel , Ġwhich Ġonly Ġhas Ġ1 . 2 M Ġparameters , Ġour Ġword - level Ġmodel Ġhas Ġ19 . 9 M Ġparameters , Ġand Ġonly Ġachieves Ġa Ġbest ĠSQ Ġaccuracy Ġof Ġ53 . 9 %. ĠIn Ġaddition , Ġin Ġcontrast Ġto Ġprevious Ġwork , Ġthe ĠO OV Ġissue Ġis Ġmuch Ġmore Ġsevere Ġfor Ġour Ġword - level Ġmodel , Ġsince Ġwe Ġuse Ġno Ġdata Ġaug mentation Ġto Ġcover Ġentities Ġunseen Ġin Ġthe Ġtrain Ġset . </s> A bl ation Ġand ĠEmb edd ing ĠExper iments </s> We Ġcarry Ġout Ġab lation Ġstudies Ġin ĠSections Ġ5 . 2 . 1 Ġand Ġ5 . 2 . 2 Ġthrough Ġa Ġset Ġof Ġrandom - sam pling Ġexperiments . ĠIn Ġthese Ġexperiments , Ġfor Ġeach Ġquestion , Ġwe Ġrandomly Ġsample Ġ200 Ġentities Ġand Ġpred icates Ġfrom Ġthe Ġtest Ġset Ġas Ġnoise Ġsamples . ĠWe Ġthen Ġmix Ġthe Ġgold Ġentity Ġand Ġpredicate Ġinto Ġthese Ġnegative Ġsamples , Ġand Ġevaluate Ġthe Ġaccuracy Ġof Ġour Ġmodel Ġin Ġpredicting Ġthe Ġgold Ġpredicate Ġor Ġentity Ġfrom Ġthis Ġmixed Ġset . </s> We Ġfirst Ġexplore Ġusing Ġword - level Ġmodels Ġas Ġan Ġalternative Ġto Ġcharacter - level Ġmodels Ġto Ġconstruct Ġembed d ings Ġfor Ġquestions , Ġentities Ġand Ġpred icates . </s> Both Ġword - level Ġand Ġcharacter - level Ġmodels Ġperform Ġcompar ably Ġwell Ġwhen Ġpredicting Ġthe Ġpredicate , Ġreaching Ġan Ġaccuracy Ġof Ġaround Ġ80 % Ġ( Table Ġ3 ). ĠHowever , Ġthe Ġword - level Ġmodel Ġhas Ġconsiderable Ġdifficulty Ġgeneral izing Ġto Ġunseen Ġentities , Ġand Ġis Ġonly Ġable Ġto Ġpredict Ġ45 % Ġof Ġthe Ġentities Ġaccurately Ġfrom Ġthe Ġmixed Ġset . ĠThese Ġresults Ġclearly Ġdemonstrate Ġthat Ġthe ĠO OV Ġissue Ġis Ġmuch Ġmore Ġsevere Ġfor Ġentities Ġthan Ġpred icates , Ġand Ġthe Ġdifficulty Ġword - level Ġmodels Ġhave Ġwhen Ġgeneral izing Ġto Ġnew Ġentities . </s> In Ġcontrast , Ġcharacter - level Ġmodels Ġhave Ġno Ġsuch Ġissues , Ġand Ġachieve Ġa Ġ96 . 6 % Ġaccuracy Ġin Ġpredicting Ġthe Ġcorrect Ġentity Ġon Ġthe Ġmixed Ġset . ĠThis Ġdemonstrates Ġthat Ġcharacter - level Ġmodels Ġencode Ġthe Ġsemantic Ġrepresentation Ġof Ġentities Ġand Ġcan Ġmatch Ġentity Ġaliases Ġin Ġa ĠKB Ġwith Ġtheir Ġmentions Ġin Ġnatural Ġlanguage Ġquestions . </s> We Ġalso Ġstudy Ġthe Ġimpact Ġof Ġthe Ġdepth Ġof Ġneural Ġnetworks Ġin Ġour Ġmodel . ĠThe Ġresults Ġare Ġpresented Ġin ĠTable Ġ2 . ĠIn Ġthe Ġab lation Ġexperiments Ġwe Ġcompare Ġthe Ġperformance Ġof Ġa Ġsingle - layer ĠL ST M Ġto Ġa Ġtwo - layer ĠL ST M Ġto Ġencode Ġthe Ġquestion , Ġand Ġa Ġsingle - layer Ġvs . Ġtwo - layer ĠCNN Ġto Ġencode Ġthe ĠKB Ġentries . ĠWe Ġfind Ġthat Ġa Ġtwo - layer ĠL ST M Ġboosts Ġjoint Ġaccuracy Ġby Ġover Ġ6 %. ĠThe Ġmajority Ġof Ġaccuracy Ġgains Ġare Ġa Ġresult Ġof Ġimproved Ġpredicate Ġpredictions , Ġpossibly Ġbecause Ġentity Ġaccuracy Ġis Ġalready Ġsaturated Ġin Ġthis Ġexperimental Ġsetup . </s> Att ention ĠMechan isms </s> In Ġorder Ġto Ġfurther Ġunderstand Ġhow Ġthe Ġmodel Ġperforms Ġquestion Ġanswering , Ġwe Ġvisualize Ġthe Ġattention Ġdistribution Ġover Ġquestion Ġcharacters Ġin Ġthe Ġdecoding Ġprocess . ĠIn Ġeach Ġsub - figure Ġof ĠFigure Ġ2 , Ġthe Ġx - axis Ġis Ġthe Ġcharacter Ġsequence Ġof Ġthe Ġquestion , Ġand Ġthe Ġy - axis Ġis Ġthe Ġattention Ġweight Ġdistribution Ġ$\ l brace Ġ\ alpha Ġ_ i \ r brace Ġ$ Ġ. ĠThe Ġblue Ġcurve Ġis Ġthe Ġattention Ġdistribution Ġwhen Ġgenerating Ġthe Ġentity , Ġand Ġgreen Ġcurve Ġis Ġthe Ġattention Ġdistribution Ġwhen Ġgenerating Ġthe Ġpredicate . </s> Interestingly , Ġas Ġthe Ġexamples Ġshow , Ġthe Ġattention Ġdistribution Ġtypically Ġpeaks Ġat Ġempty Ġspaces . ĠThis Ġindicates Ġthat Ġthe Ġcharacter - level Ġmodel Ġlearns Ġthat Ġa Ġspace Ġdefines Ġan Ġending Ġpoint Ġof Ġa Ġcomplete Ġlinguistic Ġunit . ĠThat Ġis , Ġthe Ġhidden Ġstate Ġof Ġthe ĠL ST M Ġenc oder Ġat Ġa Ġspace Ġlikely Ġsummarizes Ġcontent Ġabout Ġthe Ġcharacter Ġsequence Ġbefore Ġthat Ġspace , Ġand Ġtherefore Ġcontains Ġimportant Ġsemantic Ġinformation Ġthat Ġthe Ġdec oder Ġneeds Ġto Ġattend Ġto . </s> Also , Ġwe Ġobserve Ġthat Ġentity Ġattention Ġdistributions Ġare Ġusually Ġless Ġsharp Ġand Ġspan Ġlonger Ġportions Ġof Ġwords , Ġsuch Ġas Ġjohn Ġor Ġr utters , Ġthan Ġpredicate Ġattention Ġdistributions Ġ( e . g ., ĠFigure Ġ2 a ). ĠFor Ġentities , Ġsemantic Ġinformation Ġmay Ġaccumulate Ġgradually Ġwhen Ġseeing Ġmore Ġand Ġmore Ġcharacters , Ġwhile Ġfor Ġpred icates , Ġsemantic Ġinformation Ġwill Ġbecome Ġclear Ġonly Ġafter Ġseeing Ġthe Ġcomplete Ġword . ĠFor Ġexample , Ġit Ġmay Ġonly Ġbe Ġclear Ġthat Ġcharacters Ġsuch Ġas Ġsong Ġby Ġrefer Ġto Ġa Ġpredicate Ġafter Ġa Ġspace , Ġas Ġopposed Ġto Ġthe Ġname Ġof Ġa Ġsong Ġsuch Ġas Ġsong Ġbye Ġbye Ġlove Ġ( Fig ures Ġ2 a , Ġ2 b ). ĠIn Ġcontrast , Ġa Ġsequence Ġof Ġcharacters Ġstarts Ġto Ġbecome Ġa Ġlikely Ġentity Ġafter Ġseeing Ġan Ġincomplete Ġname Ġsuch Ġas Ġj oh Ġor Ġr utt . </s> In Ġaddition , Ġa Ġcharacter - level Ġmodel Ġcan Ġidentify Ġentities Ġwhose ĠEnglish Ġaliases Ġwere Ġnever Ġseen Ġin Ġtraining , Ġsuch Ġas Ġph ren ology Ġ( Figure Ġ2 d ). ĠThe Ġmodel Ġapparently Ġlearns Ġthat Ġwords Ġending Ġwith Ġthe Ġsuffix Ġn ology Ġare Ġlikely Ġentity Ġmentions , Ġwhich Ġis Ġinteresting Ġbecause Ġit Ġreads Ġin Ġthe Ġinput Ġone Ġcharacter Ġat Ġa Ġtime . </s> Furthermore , Ġas Ġobserved Ġin ĠFigure Ġ2 d , Ġthe Ġattention Ġmodel Ġis Ġcapable Ġof Ġattending Ġdis j oint Ġregions Ġof Ġthe Ġquestion Ġand Ġcapture Ġthe Ġmention Ġof Ġa Ġpredicate Ġthat Ġis Ġinterrupted Ġby Ġentity Ġmentions . ĠWe Ġalso Ġnote Ġthat Ġpredicate Ġattention Ġoften Ġpeaks Ġat Ġthe Ġpadding Ġsymbols Ġafter Ġthe Ġlast Ġcharacter Ġof Ġthe Ġquestion , Ġpossibly Ġbecause Ġsentence Ġendings Ġcarry Ġextra Ġinformation Ġthat Ġfurther Ġhelp Ġdis amb ig uate Ġpredicate Ġmentions . ĠIn Ġcertain Ġscenarios , Ġthe Ġnetwork Ġmay Ġonly Ġhave Ġsufficient Ġinformation Ġto Ġbuild Ġa Ġsemantic Ġrepresentation Ġof Ġthe Ġpredicate Ġafter Ġbeing Ġensured Ġthat Ġit Ġreached Ġthe Ġend Ġof Ġa Ġsentence . ĠFinally , Ġcertain Ġwords Ġin Ġthe Ġquestion Ġhelp Ġidentify Ġboth Ġthe Ġentity Ġand Ġthe Ġpredicate . ĠFor Ġexample , Ġconsider Ġthe Ġword Ġuniversity Ġin Ġthe Ġquestion ĠWhat Ġtype Ġof Ġeducational Ġinstitution Ġis Ġeastern Ġnew Ġme x ico Ġuniversity Ġ( Figure Ġ2 c ). ĠAlthough Ġit Ġis Ġa Ġpart Ġof Ġthe Ġentity Ġmention , Ġit Ġalso Ġhelps Ġdis amb ig uate Ġthe Ġpredicate . ĠHowever , Ġprevious Ġsemantic Ġparsing - based ĠQ A Ġapproaches ĠB IB REF 10 Ġ, ĠB IB REF 1 Ġassume Ġthat Ġthere Ġis Ġa Ġclear Ġseparation Ġbetween Ġthe Ġpredicate Ġand Ġentity Ġmentions Ġin Ġthe Ġquestion . ĠIn Ġcontrast , Ġthe Ġproposed Ġmodel Ġdoes Ġnot Ġneed Ġto Ġmake Ġthis Ġhard Ġcategor ization , Ġand Ġattends Ġthe Ġword Ġuniversity Ġwhen Ġpredicting Ġboth Ġthe Ġentity Ġand Ġpredicate . </s> Error ĠAnalysis </s> We Ġrandomly Ġsampled Ġ50 Ġquestions Ġwhere Ġthe Ġbest - performing Ġmodel Ġgenerated Ġthe Ġwrong ĠKB Ġquery Ġand Ġcategorized Ġthe Ġerrors . ĠFor Ġ46 Ġout Ġof Ġthe Ġ50 Ġexamples , Ġthe Ġmodel Ġpredicted Ġa Ġpredicate Ġwith Ġa Ġvery Ġsimilar Ġalias Ġto Ġthe Ġtrue Ġpredicate , Ġi . e . Ġ/ music / release / track Ġvs . Ġ/ music / release / track _ list . ĠFor Ġ21 Ġout Ġof Ġthe Ġ50 Ġexamples , Ġthe Ġmodel Ġpredicted Ġthe Ġwrong Ġentity , Ġe . g ., ĠAlbum Ġvs . ĠStill ĠHere Ġfor Ġthe Ġquestion ĠWhat Ġtype Ġof Ġalbum Ġis Ġstill Ġhere ? . ĠFinally , Ġfor Ġ18 Ġof Ġthe Ġ50 Ġexamples , Ġthe Ġmodel Ġpredicted Ġthe Ġwrong Ġentity Ġand Ġpredicate , Ġi . e . Ġ( Play , Ġ/ free base / equ ivalent _ topic / equ ivalent _ type ) Ġfor Ġthe Ġquestion Ġwhich Ġinstrument Ġdoes Ġam ap ola Ġc abase Ġplay ? ĠTraining Ġon Ġmore Ġdata , Ġaugment ing Ġthe Ġnegative Ġsample Ġset Ġwith Ġwords Ġfrom Ġthe Ġquestion Ġthat Ġare Ġnot Ġan Ġentity Ġmention , Ġand Ġhaving Ġmore Ġexamples Ġthat Ġdis amb ig uate Ġbetween Ġsimilar Ġpred icates Ġmay Ġam el ior ate Ġmany Ġof Ġthese Ġerrors . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġproposed Ġa Ġnew Ġcharacter - level , Ġattention - based Ġenc oder - dec oder Ġmodel Ġfor Ġquestion Ġanswering . ĠIn Ġour Ġapproach , Ġembed d ings Ġof Ġquestions , Ġentities , Ġand Ġpred icates Ġare Ġall Ġjointly Ġlearned Ġto Ġdirectly Ġoptimize Ġthe Ġlikelihood Ġof Ġgenerating Ġthe Ġcorrect ĠKB Ġquery . ĠOur Ġapproach Ġimproved Ġthe Ġstate - of - the - art Ġaccuracy Ġon Ġthe ĠSimple Questions Ġbenchmark Ġsignificantly , Ġusing Ġmuch Ġless Ġdata Ġthan Ġprevious Ġwork . ĠFurthermore , Ġthanks Ġto Ġcharacter - level Ġmodeling , Ġwe Ġhave Ġa Ġcompact Ġmodel Ġthat Ġis Ġrobust Ġto Ġunseen Ġentities . ĠVisual izations Ġof Ġthe Ġattention Ġdistribution Ġreveal Ġthat Ġour Ġmodel , Ġalthough Ġbuilt Ġon Ġcharacter - level Ġinputs , Ġcan Ġlearn Ġhigher - level Ġsemantic Ġconcepts Ġrequired Ġto Ġanswer Ġa Ġnatural Ġlanguage Ġquestion Ġwith Ġa Ġstructured ĠKB . ĠIn Ġthe Ġfuture Ġwe Ġwould Ġlike Ġto Ġextend Ġour Ġsystem Ġto Ġhandle Ġmulti - relation Ġquestions .