WordNet BIBREF0, Freebase BIBREF1, WN18 (a subset of WordNet) BIBREF24 , FB15K (a subset of Freebase) BIBREF2
EXTRACTIVE
We use two popular knowledge bases: WordNet BIBREF0 and Freebase BIBREF1 in this paper. Specifically, we use WN18 (a subset of WordNet) BIBREF24 and FB15K (a subset of Freebase) BIBREF2 since their text descriptions are easily publicly available. Table 1 lists statistics of the two datasets.
1611.08661
Introduction </s> Know ledge Ġgraphs Ġhave Ġbeen Ġproved Ġto Ġbenefit Ġmany Ġartificial Ġintelligence Ġapplications , Ġsuch Ġas Ġrelation Ġextraction , Ġquestion Ġanswering Ġand Ġso Ġon . ĠA Ġknowledge Ġgraph Ġconsists Ġof Ġmulti - rel ational Ġdata , Ġhaving Ġentities Ġas Ġnodes Ġand Ġrelations Ġas Ġedges . ĠAn Ġinstance Ġof Ġfact Ġis Ġrepresented Ġas Ġa Ġtriple t Ġ( Head ĠEntity , ĠRel ation , ĠTail ĠEntity ), Ġwhere Ġthe ĠRel ation Ġindicates Ġa Ġrelationship Ġbetween Ġthese Ġtwo Ġentities . ĠIn Ġthe Ġpast Ġdecades , Ġgreat Ġprogress Ġhas Ġbeen Ġmade Ġin Ġbuilding Ġlarge Ġscale Ġknowledge Ġgraphs , Ġsuch Ġas ĠWord Net ĠB IB REF 0 Ġ, ĠFree base ĠB IB REF 1 Ġ. ĠHowever , Ġmost Ġof Ġthem Ġhave Ġbeen Ġbuilt Ġeither Ġcollabor atively Ġor Ġsemi - aut om atically Ġand Ġas Ġa Ġresult , Ġthey Ġoften Ġsuffer Ġfrom Ġincom ple teness Ġand Ġsp ars eness . </s> The Ġknowledge Ġgraph Ġcompletion Ġis Ġto Ġpredict Ġrelations Ġbetween Ġentities Ġbased Ġon Ġexisting Ġtriple ts Ġin Ġa Ġknowledge Ġgraph . ĠRecently , Ġa Ġnew Ġpowerful Ġparadigm Ġhas Ġbeen Ġproposed Ġto Ġencode Ġevery Ġelement Ġ( entity Ġor Ġrelation ) Ġof Ġa Ġknowledge Ġgraph Ġinto Ġa Ġlow - dimensional Ġvector Ġspace ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ. ĠThe Ġrepresentations Ġof Ġentities Ġand Ġrelations Ġare Ġobtained Ġby Ġminimizing Ġa Ġglobal Ġloss Ġfunction Ġinvolving Ġall Ġentities Ġand Ġrelations . ĠTherefore , Ġwe Ġcan Ġdo Ġreasoning Ġover Ġknowledge Ġgraphs Ġthrough Ġalgebra ic Ġcomput ations . </s> Although Ġexisting Ġmethods Ġhave Ġgood Ġcapability Ġto Ġlearn Ġknowledge Ġgraph Ġembed d ings , Ġit Ġremains Ġchallenging Ġfor Ġentities Ġwith Ġfew Ġor Ġno Ġfacts ĠB IB REF 4 Ġ. ĠTo Ġsolve Ġthe Ġissue Ġof ĠKB Ġsp arsity , Ġmany Ġmethods Ġhave Ġbeen Ġproposed Ġto Ġlearn Ġknowledge Ġgraph Ġembed d ings Ġby Ġutilizing Ġrelated Ġtext Ġinformation ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ. ĠThese Ġmethods Ġlearn Ġjoint Ġembed ding Ġof Ġentities , Ġrelations , Ġand Ġwords Ġ( or Ġphrases , Ġsentences ) Ġinto Ġthe Ġsame Ġvector Ġspace . ĠHowever , Ġthere Ġare Ġstill Ġthree Ġproblems Ġto Ġbe Ġsolved . Ġ( 1 ) ĠThe Ġcombination Ġmethods Ġof Ġthe Ġstructural Ġand Ġtextual Ġrepresentations Ġare Ġnot Ġwell Ġstudied Ġin Ġthese Ġmethods , Ġin Ġwhich Ġtwo Ġkinds Ġof Ġrepresentations Ġare Ġmerely Ġaligned Ġon Ġword Ġlevel Ġor Ġseparate Ġloss Ġfunction . Ġ( 2 ) ĠThe Ġtext Ġdescription Ġmay Ġrepresent Ġan Ġentity Ġfrom Ġvarious Ġaspects , Ġand Ġvarious Ġrelations Ġonly Ġfocus Ġon Ġfraction al Ġaspects Ġof Ġthe Ġdescription . ĠA Ġgood Ġenc oder Ġshould Ġselect Ġthe Ġinformation Ġfrom Ġtext Ġin Ġaccordance Ġwith Ġcertain Ġcontexts Ġof Ġrelations . ĠFigure Ġ1 Ġillustrates Ġthe Ġfact Ġthat Ġnot Ġall Ġinformation Ġprovided Ġin Ġits Ġdescription Ġare Ġuseful Ġto Ġpredict Ġthe Ġlinked Ġentities Ġgiven Ġa Ġspecific Ġrelation . Ġ( 3 ) ĠInt uitive ly , Ġentities Ġwith Ġmany Ġfacts Ġdepend Ġmore Ġon Ġwell - trained Ġstructured Ġrepresentation Ġwhile Ġthose Ġwith Ġfew Ġor Ġno Ġfacts Ġmight Ġbe Ġlargely Ġdetermined Ġby Ġtext Ġdescriptions . ĠA Ġgood Ġrepresentation Ġshould Ġlearn Ġthe Ġmost Ġvaluable Ġinformation Ġby Ġbalancing Ġboth Ġsides . </s> In Ġthis Ġpaper , Ġwe Ġpropose Ġa Ġnew Ġdeep Ġarchitecture Ġto Ġlearn Ġthe Ġknowledge Ġrepresentation Ġby Ġutilizing Ġthe Ġexisting Ġtext Ġdescriptions Ġof Ġentities . ĠSpecifically , Ġwe Ġlearn Ġa Ġjoint Ġrepresentation Ġof Ġeach Ġentity Ġfrom Ġtwo Ġinformation Ġsources : Ġone Ġis Ġstructure Ġinformation , Ġand Ġanother Ġis Ġits Ġtext Ġdescription . ĠThe Ġjoint Ġrepresentation Ġis Ġthe Ġcombination Ġof Ġthe Ġstructure Ġand Ġtext Ġrepresentations Ġwith Ġa Ġg ating Ġmechanism . ĠThe Ġgate Ġdecides Ġhow Ġmuch Ġinformation Ġfrom Ġthe Ġstructure Ġor Ġtext Ġrepresentation Ġwill Ġcarry Ġover Ġto Ġthe Ġfinal Ġjoint Ġrepresentation . ĠIn Ġaddition , Ġwe Ġalso Ġintroduce Ġan Ġattention Ġmechanism Ġto Ġselect Ġthe Ġmost Ġrelated Ġinformation Ġfrom Ġtext Ġdescription Ġunder Ġdifferent Ġcontexts . ĠExperimental Ġresults Ġon Ġlink Ġprediction Ġand Ġtriple t Ġclassification Ġshow Ġthat Ġour Ġjoint Ġmodels Ġcan Ġhandle Ġthe Ġsp arsity Ġproblem Ġwell Ġand Ġoutper form Ġthe Ġbaseline Ġmethod Ġon Ġall Ġmetrics Ġwith Ġa Ġlarge Ġmargin . </s> Our Ġcontributions Ġin Ġthis Ġpaper Ġare Ġsummarized Ġas Ġfollows . </s> Know ledge ĠGraph ĠEmb edd ing </s> In Ġthis Ġsection , Ġwe Ġbriefly Ġintroduce Ġthe Ġbackground Ġknowledge Ġabout Ġthe Ġknowledge Ġgraph Ġembed ding . </s> Know ledge Ġgraph Ġembed ding Ġaims Ġto Ġmodel Ġmulti - rel ational Ġdata Ġ( ent ities Ġand Ġrelations ) Ġinto Ġa Ġcontinuous Ġlow - dimensional Ġvector Ġspace . ĠGiven Ġa Ġpair Ġof Ġentities Ġ$( h , t ) $ Ġand Ġtheir Ġrelation Ġ$ r $ Ġ, Ġwe Ġcan Ġrepresent Ġthem Ġwith Ġa Ġtriple Ġ$( h , r , t ) $ Ġ. ĠA Ġscore Ġfunction Ġ$ f ( h , r , Ġt ) $ Ġis Ġdefined Ġto Ġmodel Ġthe Ġcorrectness Ġof Ġthe Ġtriple Ġ$( h , r , t ) $ Ġ, Ġthus Ġto Ġdistinguish Ġwhether Ġtwo Ġentities Ġ$ h $ Ġand Ġ$ t $ Ġare Ġin Ġa Ġcertain Ġrelationship Ġ$ r $ Ġ. Ġ$ f ( h , r , Ġt ) $ Ġshould Ġbe Ġlarger Ġfor Ġa Ġgolden Ġtriple t Ġ$( h , Ġr , Ġt ) $ Ġthat Ġcorresponds Ġto Ġa Ġtrue Ġfact Ġin Ġreal Ġworld , Ġotherwise Ġ$ r $ 0 Ġshould Ġbe Ġlower Ġfor Ġan Ġnegative Ġtriple t . </s> The Ġdifference Ġamong Ġthe Ġexisting Ġmethods Ġvaries Ġbetween Ġlinear ĠB IB REF 2 Ġ, ĠB IB REF 8 Ġand Ġnon linear ĠB IB REF 3 Ġscore Ġfunctions Ġin Ġthe Ġlow - dimensional Ġvector Ġspace . </s> Among Ġthese Ġmethods , ĠTrans E ĠB IB REF 2 Ġis Ġa Ġsimple Ġand Ġeffective Ġapproach , Ġwhich Ġlearns Ġthe Ġvector Ġembed d ings Ġfor Ġboth Ġentities Ġand Ġrelationships . ĠIts Ġbasic Ġidea Ġis Ġthat Ġthe Ġrelationship Ġbetween Ġtwo Ġentities Ġis Ġsupposed Ġto Ġcorrespond Ġto Ġa Ġtranslation Ġbetween Ġthe Ġembed d ings Ġof Ġentities , Ġthat Ġis , Ġ$\ text bf Ġ{ h } + Ġ\ math bf Ġ{ r }\ app rox Ġ\ math bf Ġ{ t } $ Ġwhen Ġ$( h , r , t ) $ Ġholds . </s> Trans E 's Ġscore Ġfunction Ġis Ġdefined Ġas : </s> $$ f ( h , r , t )) Ġ& = Ġ- \ Vert Ġ\ text bf Ġ{ h } + \ math bf Ġ{ r } - \ math bf Ġ{ t }\ Vert Ġ_ { 2 } ^ 2 $$ Ġ Ġ Ġ( E q . Ġ5 ) </s> where Ġ$\ text bf Ġ{ h }, \ math bf Ġ{ t }, \ math bf Ġ{ r }\ in Ġ\ math bb Ġ{ R } ^ d $ Ġare Ġembed d ings Ġof Ġ$ h , t , r $ Ġrespectively , Ġand Ġsatisfy Ġ$\ Vert Ġ\ text bf Ġ{ h }\ Vert Ġ^ 2 _ 2 = \ Vert Ġ\ math bf Ġ{ t }\ Vert Ġ^ 2 _ 2 = 1 $ Ġ. ĠThe Ġ$\ text bf Ġ{ h }, Ġ\ math bf Ġ{ r }, Ġ\ math bf Ġ{ t } $ Ġare Ġindexed Ġby Ġa Ġlookup Ġtable Ġrespectively . </s> Ne ural ĠText ĠEnc oding </s> Given Ġan Ġentity Ġin Ġmost Ġof Ġthe Ġexisting Ġknowledge Ġbases , Ġthere Ġis Ġalways Ġan Ġavailable Ġcorresponding Ġtext Ġdescription Ġwith Ġvaluable Ġsemantic Ġinformation Ġfor Ġthis Ġentity , Ġwhich Ġcan Ġprovide Ġbeneficial Ġsupplement Ġfor Ġentity Ġrepresentation . </s> To Ġencode Ġthe Ġrepresentation Ġof Ġa Ġentity Ġfrom Ġits Ġtext Ġdescription , Ġwe Ġneed Ġto Ġencode Ġthe Ġvariable - length Ġsentence Ġto Ġa Ġfixed - length Ġvector . ĠThere Ġare Ġseveral Ġkinds Ġof Ġneural Ġmodels Ġused Ġin Ġsentence Ġmodeling . ĠThese Ġmodels Ġgenerally Ġconsist Ġof Ġa Ġprojection Ġlayer Ġthat Ġmaps Ġwords , Ġsub - word Ġunits Ġor Ġn - gram s Ġto Ġvector Ġrepresentations Ġ( often Ġtrained Ġbeforehand Ġwith Ġun super vised Ġmethods ), Ġand Ġthen Ġcombine Ġthem Ġwith Ġthe Ġdifferent Ġarchitectures Ġof Ġneural Ġnetworks , Ġsuch Ġas Ġneural Ġbag - of - words Ġ( NB OW ), Ġrecurrent Ġneural Ġnetwork Ġ( R NN ) ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġand Ġconv olution al Ġneural Ġnetwork Ġ( CNN ) ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ. </s> In Ġthis Ġpaper , Ġwe Ġuse Ġthree Ġenc od ers Ġ( NB OW , ĠL ST M Ġand Ġattentive ĠL ST M ) Ġto Ġmodel Ġthe Ġtext Ġdescriptions . </s> B ag - of - Words ĠEnc oder </s> A Ġsimple Ġand Ġintuitive Ġmethod Ġis Ġthe Ġneural Ġbag - of - words Ġ( NB OW ) Ġmodel , Ġin Ġwhich Ġthe Ġrepresentation Ġof Ġtext Ġcan Ġbe Ġgenerated Ġby Ġsum ming Ġup Ġits Ġconstituent Ġword Ġrepresentations . </s> We Ġdenote Ġthe Ġtext Ġdescription Ġas Ġword Ġsequence Ġ$ x _{ 1 : n } Ġ= Ġx _ 1 , \ cd ots Ġ, x _ n $ Ġ, Ġwhere Ġ$ x _ i $ Ġis Ġthe Ġword Ġat Ġposition Ġ$ i $ Ġ. ĠThe ĠNB OW Ġenc oder Ġis </s> $$ \ math rm Ġ{ enc _ 1 } ( x _{ 1 : n }) Ġ= Ġ\ sum Ġ_ { i = 1 } ^{ n } Ġ\ math bf Ġ{ x } _ i , $$ Ġ Ġ Ġ( E q . Ġ7 ) </s> where Ġ$\ math bf Ġ{ x } _ i Ġ\ in Ġ\ math bb Ġ{ R } ^ d $ Ġis Ġthe Ġword Ġembed ding Ġof Ġ$ x _ i $ Ġ. </s> L ST M ĠEnc oder </s> To Ġaddress Ġsome Ġof Ġthe Ġmodelling Ġissues Ġwith ĠNB OW , Ġwe Ġconsider Ġusing Ġa Ġbid irection al Ġlong Ġshort - term Ġmemory Ġnetwork Ġ( L ST M ) ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġto Ġmodel Ġthe Ġtext Ġdescription . </s> L ST M Ġwas Ġproposed Ġby ĠB IB REF 16 Ġto Ġspecifically Ġaddress Ġthis Ġissue Ġof Ġlearning Ġlong - term Ġdependencies ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ, ĠB IB REF 16 Ġin ĠR NN . ĠThe ĠL ST M Ġmaintains Ġa Ġseparate Ġmemory Ġcell Ġinside Ġit Ġthat Ġupdates Ġand Ġexposes Ġits Ġcontent Ġonly Ġwhen Ġdeemed Ġnecessary . </s> B id irection al ĠL ST M Ġ( BL ST M ) Ġcan Ġbe Ġregarded Ġas Ġtwo Ġseparate ĠL ST Ms Ġwith Ġdifferent Ġdirections . ĠOne ĠL ST M Ġmodels Ġthe Ġtext Ġdescription Ġfrom Ġleft Ġto Ġright , Ġand Ġanother ĠL ST M Ġmodels Ġtext Ġdescription Ġfrom Ġright Ġto Ġleft Ġrespectively . ĠWe Ġdefine Ġthe Ġoutputs Ġof Ġtwo ĠL ST M Ġat Ġtime Ġstep Ġ$ i $ Ġare Ġ$\ over right arrow {\ math bf Ġ{ z }} _ i $ Ġand Ġ$\ over left arrow {\ math bf Ġ{ z }} _ i $ Ġrespectively . </s> The Ġcombined Ġoutput Ġof ĠBL ST M Ġat Ġposition Ġ$ i $ Ġis Ġ${ \ math bf Ġ{ z } _ i } Ġ= Ġ\ over right arrow {\ math bf Ġ{ z }} _ i Ġ\ opl us Ġ\ over left arrow {\ math bf Ġ{ z }} _ i $ Ġ, Ġwhere Ġ$\ opl us Ġ$ Ġdenotes Ġthe Ġconc aten ation Ġoperation . </s> The ĠL ST M Ġenc oder Ġcombines Ġall Ġthe Ġoutputs Ġ$\ math bf Ġ{ z } _ i Ġ\ in Ġ\ math bb Ġ{ R } ^ d $ Ġof ĠBL ST M Ġat Ġdifferent Ġposition . </s> $$ \ math rm Ġ{ enc _ 2 } ( x _{ 1 : n }) Ġ= Ġ\ sum Ġ_ { i = 1 } ^{ n } Ġ{\ math bf Ġ{ z } _ i }. $$ Ġ Ġ Ġ( E q . Ġ9 ) </s> Att ent ive ĠL ST M ĠEnc oder </s> While Ġthe ĠL ST M Ġenc oder Ġhas Ġricher Ġcapacity Ġthan ĠNB OW , Ġit Ġproduces Ġthe Ġsame Ġrepresentation Ġfor Ġthe Ġentire Ġtext Ġdescription Ġregardless Ġof Ġits Ġcontexts . ĠHowever , Ġthe Ġtext Ġdescription Ġmay Ġpresent Ġan Ġentity Ġfrom Ġvarious Ġaspects , Ġand Ġvarious Ġrelations Ġonly Ġfocus Ġon Ġfraction al Ġaspects Ġof Ġthe Ġdescription . ĠThis Ġphenomenon Ġalso Ġoccurs Ġin Ġstructure Ġembed ding Ġfor Ġan Ġentity ĠB IB REF 8 Ġ, ĠB IB REF 19 Ġ. </s> Given Ġa Ġrelation Ġfor Ġan Ġentity , Ġnot Ġall Ġof Ġwords / ph r ases Ġin Ġits Ġtext Ġdescription Ġare Ġuseful Ġto Ġmodel Ġa Ġspecific Ġfact . ĠSome Ġof Ġthem Ġmay Ġbe Ġimportant Ġfor Ġthe Ġgiven Ġrelation , Ġbut Ġmay Ġbe Ġuseless Ġfor Ġother Ġrelations . ĠTherefore , Ġwe Ġintroduce Ġan Ġattention Ġmechanism ĠB IB REF 20 Ġto Ġutilize Ġan Ġattention - based Ġenc oder Ġthat Ġconstructs Ġcontextual Ġtext Ġenc od ings Ġaccording Ġto Ġdifferent Ġrelations . </s> For Ġeach Ġposition Ġ$ i $ Ġof Ġthe Ġtext Ġdescription , Ġthe Ġattention Ġfor Ġa Ġgiven Ġrelation Ġ$ r $ Ġis Ġdefined Ġas Ġ$\ alpha Ġ_ i ( r ) $ Ġ, Ġwhich Ġis </s> $$ e _ i ( r ) Ġ& = Ġ\ math bf Ġ{ v } _ a ^ T Ġ\ tan h Ġ( \ math bf Ġ{ W } _ a Ġ{\ math bf Ġ{ z }} _ i Ġ+ Ġ\ math bf Ġ{ U } _ a Ġ\ math bf Ġ{ r } ), Ġ\\ Ġ\ alpha Ġ_ i ( r ) & = \ oper at orn ame {\ math bf Ġ{ soft max }} ( e _ i ( r )) \ non umber Ġ\\ Ġ& = \ frac {\ exp Ġ( e _ i ( r )) }{ \ sum Ġ^ { n } _{ j = 1 } Ġ\ exp Ġ( e _ j ( r )) }, $$ Ġ Ġ Ġ( E q . Ġ12 ) </s> where Ġ$\ math bf Ġ{ r }\ in Ġ\ math bb Ġ{ R } ^ d $ Ġis Ġthe Ġrelation Ġembed ding ; Ġ${ \ math bf Ġ{ z }} _ i Ġ\ in Ġ\ math bb Ġ{ R } ^ d $ Ġis Ġthe Ġoutput Ġof ĠBL ST M Ġat Ġposition Ġ$ i $ Ġ; Ġ$\ math bf Ġ{ W } _ a , \ math bf Ġ{ U } _ a Ġ\ in Ġ\ math bb Ġ{ R } ^{ d \ times Ġd } $ Ġare Ġparameters Ġmat rices ; Ġ$\ math bf Ġ{ v } _ a Ġ\ in Ġ\ math bb Ġ{ R } ^{ d } $ Ġis Ġa Ġparameter Ġvector . </s> The Ġattention Ġ$\ alpha Ġ_ i ( r ) $ Ġis Ġinterpreted Ġas Ġthe Ġdegree Ġto Ġwhich Ġthe Ġnetwork Ġattends Ġto Ġpartial Ġrepresentation Ġ$\ math bf Ġ{ z } _{ i } $ Ġfor Ġgiven Ġrelation Ġ$ r $ Ġ. </s> The Ġcontextual Ġencoding Ġof Ġtext Ġdescription Ġcan Ġbe Ġformed Ġby Ġa Ġweighted Ġsum Ġof Ġthe Ġencoding Ġ$\ math bf Ġ{ z } _{ i } $ Ġwith Ġattention . </s> $$ \ math bf Ġ{ enc _ 3 } ( x _{ 1 : n }; r ) Ġ& = Ġ\ sum Ġ_ { i = 1 } ^{ n } Ġ\ alpha Ġ_ i ( r ) Ġ* Ġ\ math bf Ġ{ z } _ i . $$ Ġ Ġ Ġ( E q . Ġ13 ) </s> J oint ĠStructure Ġand ĠText ĠEnc oder </s> Since Ġboth Ġthe Ġstructure Ġand Ġtext Ġdescription Ġprovide Ġvaluable Ġinformation Ġfor Ġan Ġentity Ġ, Ġwe Ġwish Ġto Ġintegrate Ġall Ġthese Ġinformation Ġinto Ġa Ġjoint Ġrepresentation . </s> We Ġpropose Ġa Ġunited Ġmodel Ġto Ġlearn Ġa Ġjoint Ġrepresentation Ġof Ġboth Ġstructure Ġand Ġtext Ġinformation . ĠThe Ġwhole Ġmodel Ġcan Ġbe Ġend - to - end Ġtrained . </s> For Ġan Ġentity Ġ$ e $ Ġ, Ġwe Ġdenote Ġ$\ math bf Ġ{ e } _ s $ Ġto Ġbe Ġits Ġembed ding Ġof Ġstructure Ġinformation , Ġ$\ math bf Ġ{ e } _ d $ Ġto Ġbe Ġencoding Ġof Ġits Ġtext Ġdescriptions . ĠThe Ġmain Ġconcern Ġis Ġhow Ġto Ġcombine Ġ$\ math bf Ġ{ e } _ s $ Ġand Ġ$\ math bf Ġ{ e } _ d $ Ġ. </s> To Ġintegrate Ġtwo Ġkinds Ġof Ġrepresentations Ġof Ġentities , Ġwe Ġuse Ġg ating Ġmechanism Ġto Ġdecide Ġhow Ġmuch Ġthe Ġjoint Ġrepresentation Ġdepends Ġon Ġstructure Ġor Ġtext . </s> The Ġjoint Ġrepresentation Ġ$\ math bf Ġ{ e } $ Ġis Ġa Ġlinear Ġinterpol ation Ġbetween Ġthe Ġ$\ math bf Ġ{ e } _ s $ Ġand Ġ$\ math bf Ġ{ e } _ d $ Ġ. </s> $$ \ math bf Ġ{ e } = Ġ\ text bf Ġ{ g } _ e Ġ\ od ot Ġ\ math bf Ġ{ e } _ s Ġ+ Ġ( 1 - \ text bf Ġ{ g } _ e )\ od ot Ġ\ math bf Ġ{ e } _ d , $$ Ġ Ġ Ġ( E q . Ġ14 ) </s> where Ġ$\ text bf Ġ{ g } _ e $ Ġis Ġa Ġgate Ġto Ġbalance Ġtwo Ġsources Ġinformation Ġand Ġits Ġelements Ġare Ġin Ġ$ [ 0 , 1 ] $ Ġ, Ġand Ġ$\ od ot Ġ$ Ġis Ġan Ġelement - wise Ġmultiplication . ĠInt uitive ly , Ġwhen Ġthe Ġgate Ġis Ġclose Ġto Ġ0 , Ġthe Ġjoint Ġrepresentation Ġis Ġforced Ġto Ġignore Ġthe Ġstructure Ġinformation Ġand Ġis Ġthe Ġtext Ġrepresentation Ġonly . </s> Training </s> We Ġuse Ġthe Ġcontrast ive Ġmax - margin Ġcriterion ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġto Ġtrain Ġour Ġmodel . ĠInt uitive ly , Ġthe Ġmax - margin Ġcriterion Ġprovides Ġan Ġalternative Ġto Ġprob abil istic , Ġlikelihood - based Ġestimation Ġmethods Ġby Ġconcentrating Ġdirectly Ġon Ġthe Ġrobust ness Ġof Ġthe Ġdecision Ġboundary Ġof Ġa Ġmodel ĠB IB REF 23 Ġ. ĠThe Ġmain Ġidea Ġis Ġthat Ġeach Ġtriple t Ġ$( h , r , t ) $ Ġcoming Ġfrom Ġthe Ġtraining Ġcorpus Ġshould Ġreceives Ġa Ġhigher Ġscore Ġthan Ġa Ġtriple t Ġin Ġwhich Ġone Ġof Ġthe Ġelements Ġis Ġreplaced Ġwith Ġa Ġrandom Ġelements . </s> We Ġassume Ġthat Ġthere Ġare Ġ$ n _ t $ Ġtriple ts Ġin Ġtraining Ġset Ġand Ġdenote Ġthe Ġ$ i $ Ġth Ġtriple t Ġby Ġ$( h _ i , Ġr _ i , Ġt _ i ), ( i Ġ= Ġ1 , Ġ2 , Ġ\ cd ots Ġ, n _ t ) $ Ġ. ĠEach Ġtriple t Ġhas Ġa Ġlabel Ġ$ y _ i $ Ġto Ġindicate Ġthe Ġtriple t Ġis Ġpositive Ġ( Ġ$ y _ i Ġ= Ġ1 $ Ġ) Ġor Ġnegative Ġ( Ġ$ y _ i Ġ= Ġ0 $ Ġ). </s> Then Ġthe Ġgolden Ġand Ġnegative Ġtriple ts Ġare Ġden oted Ġby Ġ$\ math cal Ġ{ D } Ġ= Ġ\ l brace Ġ( h _ j , Ġr _ j , Ġt _ j ) Ġ| Ġy _ j Ġ= Ġ1 \ r brace Ġ$ Ġand Ġ$\ math cal Ġ{\ hat { D }} Ġ= Ġ\ l brace Ġ( h _ j , Ġr _ j , Ġt _ j ) Ġ| Ġy _ j Ġ= Ġ0 \ r brace Ġ$ Ġ, Ġrespectively . ĠThe Ġpositive Ġexample Ġare Ġthe Ġtriple ts Ġfrom Ġtraining Ġdataset , Ġand Ġthe Ġnegative Ġexamples Ġare Ġgenerated Ġas Ġfollows : Ġ$ Ġ\ math cal Ġ{\ hat { D }} Ġ= Ġ\ l brace Ġ( h _ l , Ġr _ k , Ġt _ k ) Ġ| Ġh _ l Ġ\ ne Ġh _ k Ġ\ w edge Ġy _ k Ġ= Ġ1 \ r brace Ġ\ cup Ġ\ l brace Ġ( h _ k , Ġr _ k , Ġt _ l ) Ġ| Ġt _ l Ġ\ ne Ġt _ k Ġ\ w edge Ġy _ k Ġ= Ġ1 \ r brace Ġ\ cup Ġ\ l brace Ġ( h _ k , Ġr _ l , Ġt _ k ) Ġ| Ġr _ l Ġ\ ne Ġr _ k Ġ\ w edge Ġy _ k Ġ= Ġ1 \ r brace Ġ$ Ġ. ĠThe Ġsampling Ġstrategy Ġis ĠBern ou lli Ġdistribution Ġdescribed Ġin ĠB IB REF 8 Ġ. ĠLet Ġthe Ġset Ġof Ġall Ġparameters Ġbe Ġ$\ The ta Ġ$ Ġ, Ġwe Ġminimize Ġthe Ġfollowing Ġobjective : </s> $$ J (\ The ta Ġ) = \ sum Ġ_ { ( h , r , t ) Ġ\ in Ġ\ math cal Ġ{ D }} \ sum Ġ_ { ( Ġ\ hat { h }, \ hat { r }, \ hat { t }) Ġ\ in Ġ\ math cal Ġ{\ hat { D }}} Ġ\ max Ġ\ left ( 0 , \ gam ma Ġ- Ġ\ right . Ġ\ non umber Ġ\\ Ġf ( Ġh , r , t )+ f (\ hat { h }, \ hat { r }, \ hat { t } )\ left . \ right )+ Ġ\ eta Ġ\ Vert Ġ\ The ta Ġ\ Vert Ġ_ 2 ^ 2 , $$ Ġ Ġ Ġ( E q . Ġ22 ) </s> where Ġ$\ gam ma Ġ> Ġ0 $ Ġis Ġa Ġmargin Ġbetween Ġgolden Ġtriple ts Ġand Ġnegative Ġtriple ts ., Ġ$ f ( h , Ġr , Ġt ) $ Ġis Ġthe Ġscore Ġfunction . ĠWe Ġuse Ġthe Ġstandard Ġ$ L _ 2 $ Ġregular ization Ġof Ġall Ġthe Ġparameters , Ġweighted Ġby Ġthe Ġhyper param eter Ġ$\ eta Ġ$ Ġ. </s> Exper iment </s> In Ġthis Ġsection , Ġwe Ġstudy Ġthe Ġempirical Ġperformance Ġof Ġour Ġproposed Ġmodels Ġon Ġtwo Ġbenchmark Ġtasks : Ġtriple t Ġclassification Ġand Ġlink Ġprediction . </s> Dat as ets </s> We Ġuse Ġtwo Ġpopular Ġknowledge Ġbases : ĠWord Net ĠB IB REF 0 Ġand ĠFree base ĠB IB REF 1 Ġin Ġthis Ġpaper . ĠSpecifically , Ġwe Ġuse ĠW N 18 Ġ( a Ġsubset Ġof ĠWord Net ) ĠB IB REF 24 Ġand ĠFB 15 K Ġ( a Ġsubset Ġof ĠFree base ) ĠB IB REF 2 Ġsince Ġtheir Ġtext Ġdescriptions Ġare Ġeasily Ġpublicly Ġavailable . ĠTable Ġ1 Ġlists Ġstatistics Ġof Ġthe Ġtwo Ġdatasets . </s> Link ĠPrediction </s> Link Ġprediction Ġis Ġa Ġsubt ask Ġof Ġknowledge Ġgraph Ġcompletion Ġto Ġcomplete Ġa Ġtriple t Ġ$( h , Ġr , Ġt ) $ Ġwith Ġ$ h $ Ġor Ġ$ t $ Ġmissing , Ġi . e ., Ġpredict Ġ$ t $ Ġgiven Ġ$( h , Ġr ) $ Ġor Ġpredict Ġ$ h $ Ġgiven Ġ$( r , Ġt ) $ Ġ. ĠRather Ġthan Ġrequiring Ġone Ġbest Ġanswer , Ġthis Ġtask Ġemphasizes Ġmore Ġon Ġranking Ġa Ġset Ġof Ġcandidate Ġentities Ġfrom Ġthe Ġknowledge Ġgraph . </s> Similar Ġto ĠB IB REF 2 Ġ, Ġwe Ġuse Ġtwo Ġmeasures Ġas Ġour Ġevaluation Ġmetrics . Ġ( 1 ) ĠMean ĠRank : Ġthe Ġaveraged Ġrank Ġof Ġcorrect Ġentities Ġor Ġrelations ; Ġ( 2 ) ĠHits @ p : Ġthe Ġproportion Ġof Ġvalid Ġentities Ġor Ġrelations Ġranked Ġin Ġtop Ġ$ p $ Ġpredictions . ĠHere , Ġwe Ġset Ġ$ p = 10 $ Ġfor Ġentities Ġand Ġ$ p = 1 $ Ġfor Ġrelations . ĠA Ġlower ĠMean ĠRank Ġand Ġa Ġhigher ĠHits @ p Ġshould Ġbe Ġachieved Ġby Ġa Ġgood Ġembed ding Ġmodel . ĠWe Ġcall Ġthis Ġevaluation Ġsetting ĠâĢ ľ Raw âĢ Ŀ . ĠSince Ġa Ġfalse Ġpredicted Ġtriple t Ġmay Ġalso Ġexist Ġin Ġknowledge Ġgraphs , Ġit Ġshould Ġbe Ġregard Ġas Ġa Ġvalid Ġtriple t . ĠHence , Ġwe Ġshould Ġremove Ġthe Ġfalse Ġpredicted Ġtriple ts Ġincluded Ġin Ġtraining , Ġvalidation Ġand Ġtest Ġsets Ġbefore Ġranking Ġ( except Ġthe Ġtest Ġtriple t Ġof Ġinterest ). ĠWe Ġcall Ġthis Ġevaluation Ġsetting ĠâĢ ľ Filter âĢ Ŀ . ĠThe Ġevaluation Ġresults Ġare Ġreported Ġunder Ġthese Ġtwo Ġsettings . </s> We Ġselect Ġthe Ġmargin Ġ$\ gam ma Ġ$ Ġamong Ġ$\ l brace Ġ1 , Ġ2 \ r brace Ġ$ Ġ, Ġthe Ġembed ding Ġdimension Ġ$ d $ Ġamong Ġ$\ l brace Ġ20 , Ġ50 , Ġ100 \ r brace Ġ$ Ġ, Ġthe Ġregular ization Ġ$\ eta Ġ$ Ġamong Ġ$\ l brace Ġ0 , Ġ1 E { - 5 }, Ġ1 E { - 6 }\ r brace Ġ$ Ġ, Ġtwo Ġlearning Ġrates Ġ$\ lambda Ġ_ s $ Ġand Ġ$\ lambda Ġ_ t $ Ġamong Ġ$\ l brace Ġ0 . 001 , Ġ0 . 01 , Ġ0 . 05 \ r brace Ġ$ Ġto Ġlearn Ġthe Ġparameters Ġof Ġstructure Ġand Ġtext Ġencoding . ĠThe Ġdiss imilar ity Ġmeasure Ġis Ġset Ġto Ġeither Ġ$ L _ 1 $ Ġor Ġ$\ l brace Ġ1 , Ġ2 \ r brace Ġ$ 0 Ġdistance . </s> In Ġorder Ġto Ġspeed Ġup Ġthe Ġconvergence Ġand Ġavoid Ġover fitting , Ġwe Ġinitiate Ġthe Ġstructure Ġembed d ings Ġof Ġentity Ġand Ġrelation Ġwith Ġthe Ġresults Ġof ĠTrans E . ĠThe Ġembed ding Ġof Ġa Ġword Ġis Ġinitialized Ġby Ġaveraging Ġthe Ġlinked Ġentity Ġembed d ings Ġwhose Ġdescription Ġinclude Ġthis Ġword . ĠThe Ġrest Ġparameters Ġare Ġinitialized Ġby Ġrandomly Ġsampling Ġfrom Ġuniform Ġdistribution Ġin Ġ$ [ - 0 . 1 , Ġ0 . 1 ] $ Ġ. </s> The Ġfinal Ġoptimal Ġconfigurations Ġare : Ġ$\ gam ma Ġ= Ġ2 $ Ġ, Ġ$ d = 20 $ Ġ, Ġ$\ eta Ġ= 1 E { - 5 } $ Ġ, Ġ$\ lambda Ġ_ s Ġ= Ġ0 . 01 $ Ġ, Ġ$\ lambda Ġ_ t Ġ= Ġ0 . 1 $ Ġ, Ġand Ġ$ L _ 1 $ Ġdistance Ġon ĠW N 18 ; Ġ$\ gam ma Ġ= 2 $ Ġ, Ġ$ d = 100 $ Ġ, Ġ$\ eta Ġ= 1 E { - 5 } $ Ġ, Ġ$\ lambda Ġ_ s Ġ= Ġ0 . 01 $ Ġ, Ġ$ d = 20 $ 0 Ġ, Ġand Ġ$ d = 20 $ 1 Ġdistance Ġon ĠFB 15 K . </s> Exper imental Ġresults Ġon Ġboth ĠW N 18 Ġand ĠFB 15 k Ġare Ġshown Ġin ĠTable Ġ2 Ġ, Ġwhere Ġwe Ġuse ĠâĢ ľ J oint ly ( CB OW ) âĢ Ŀ , ĠâĢ ľ J oint ly ( L ST M ) âĢ Ŀ Ġand ĠâĢ ľ J oint ly ( A - L ST M ) âĢ Ŀ Ġto Ġrepresent Ġour Ġjointly Ġencoding Ġmodels Ġwith ĠCB OW , ĠL ST M Ġand Ġattentive ĠL ST M Ġtext Ġenc od ers . ĠOur Ġbaseline Ġis ĠTrans E Ġsince Ġthat Ġthe Ġscore Ġfunction Ġof Ġour Ġmodels Ġis Ġbased Ġon ĠTrans E . </s> From Ġthe Ġresults , Ġwe Ġobserve Ġthat Ġproposed Ġmodels Ġsurpass Ġthe Ġbaseline , ĠTrans E , Ġon Ġall Ġmetrics , Ġwhich Ġindicates Ġthat Ġknowledge Ġrepresentation Ġcan Ġbenefit Ġgreatly Ġfrom Ġtext Ġdescription . </s> On ĠW N 18 , Ġthe Ġreason Ġwhy ĠâĢ ľ J oint ly ( A - L ST M ) âĢ Ŀ Ġis Ġslightly Ġworse Ġthan ĠâĢ ľ J oint ly ( L ST M ) âĢ Ŀ Ġis Ġprobably Ġbecause Ġthe Ġnumber Ġof Ġrelations Ġis Ġlimited . ĠTherefore , Ġthe Ġattention Ġmechanism Ġdoes Ġnot Ġhave Ġobvious Ġadvantage . ĠOn ĠFB 15 K , ĠâĢ ľ J oint ly ( A - L ST M ) âĢ Ŀ Ġachieves Ġthe Ġbest Ġperformance Ġand Ġis Ġsignificantly Ġhigher Ġthan Ġbaseline Ġmethods Ġon Ġmean Ġrank . </s> Although Ġthe ĠHits @ 10 Ġof Ġour Ġmodels Ġare Ġworse Ġthan Ġthe Ġbest Ġstate - of - the - art Ġmethod , ĠTrans D , Ġit Ġis Ġworth Ġnoticing Ġthat Ġthe Ġscore Ġfunction Ġof Ġour Ġmodels Ġis Ġbased Ġon ĠTrans E , Ġnot ĠTrans D . ĠOur Ġmodels Ġare Ġcompatible Ġwith Ġother Ġstate - of - the - art Ġknowledge Ġembed ding Ġmodels . ĠWe Ġbelieve Ġthat Ġour Ġmodel Ġcan Ġbe Ġfurther Ġimproved Ġby Ġadopting Ġthe Ġscore Ġfunctions Ġof Ġother Ġstate - of - the - art Ġmethods , Ġsuch Ġas ĠTrans D . </s> Besides , Ġtextual Ġinformation Ġlargely Ġallev iates Ġthe Ġissue Ġof Ġsp arsity Ġand Ġour Ġmodel Ġachieves Ġsubstantial Ġimprovement Ġon ĠMean ĠRank Ġcomparing Ġwith ĠTrans D . ĠHowever , Ġtextual Ġinformation Ġmay Ġslightly Ġdegrade Ġthe Ġrepresentation Ġof Ġfrequent Ġentities Ġwhich Ġhave Ġbeen Ġwell - trained . ĠThis Ġmay Ġbe Ġanother Ġreason Ġwhy Ġour ĠHits @ 10 Ġis Ġworse Ġthan ĠTrans D Ġwhich Ġonly Ġutilizes Ġstructural Ġinformation . </s> For Ġthe Ġcomparison Ġof ĠHits @ 10 Ġof Ġdifferent Ġkinds Ġof Ġrelations , Ġwe Ġcategorized Ġthe Ġrelationships Ġaccording Ġto Ġthe Ġcardinal ities Ġof Ġtheir Ġhead Ġand Ġtail Ġarguments Ġinto Ġfour Ġclasses : Ġ1 - to - 1 , Ġ1 - to - many , Ġmany - to - 1 , Ġmany - to - many . ĠM apping Ġproperties Ġof Ġrelations Ġfollows Ġthe Ġsame Ġrules Ġin ĠB IB REF 2 Ġ. </s> Table Ġ3 Ġshows Ġthe Ġdetailed Ġresults Ġby Ġmapping Ġproperties Ġof Ġrelations Ġon ĠFB 15 k . ĠWe Ġcan Ġsee Ġthat Ġour Ġmodels Ġoutper form Ġbaseline ĠTrans E Ġin Ġall Ġtypes Ġof Ġrelations Ġ( 1 - to - 1 , Ġ1 - to - N , ĠN - to - 1 Ġand ĠN - to - N ), Ġespecially Ġwhen Ġ( 1 ) Ġpredicting ĠâĢ ľ 1 - to - 1 âĢ Ŀ Ġrelations Ġand Ġ( 2 ) Ġpredicting Ġthe Ġ1 Ġside Ġfor ĠâĢ ľ 1 - to - N âĢ Ŀ Ġand ĠâĢ ľ N - to - 1 âĢ Ŀ Ġrelations . </s> To Ġget Ġmore Ġinsights Ġinto Ġhow Ġthe Ġjoint Ġrepresentation Ġis Ġinfluenced Ġby Ġthe Ġstructure Ġand Ġtext Ġinformation . ĠWe Ġobserve Ġthe Ġactiv ations Ġof Ġgates , Ġwhich Ġcontrol Ġthe Ġbalance Ġbetween Ġtwo Ġsources Ġof Ġinformation , Ġto Ġunderstand Ġthe Ġbehavior Ġof Ġneurons . ĠWe Ġsort Ġthe Ġentities Ġby Ġtheir Ġfrequencies Ġand Ġdivide Ġthem Ġinto Ġ50 Ġequal - size Ġgroups Ġof Ġdifferent Ġfrequencies , Ġand Ġaverage Ġthe Ġvalues Ġof Ġall Ġgates Ġin Ġeach Ġgroup . </s> Figure Ġ3 Ġgives Ġthe Ġaverage Ġof Ġgates Ġin Ġten Ġgroups Ġfrom Ġhigh - Ġto Ġlow - frequency . ĠWe Ġobserve Ġthat Ġthe Ġtext Ġinformation Ġplay Ġmore Ġimportant Ġrole Ġfor Ġthe Ġlow - frequency Ġentities . </s> Tri plet ĠClassification </s> Tri plet Ġclassification Ġis Ġa Ġbinary Ġclassification Ġtask , Ġwhich Ġaims Ġto Ġjudge Ġwhether Ġa Ġgiven Ġtriple t Ġ$( h , Ġr , Ġt ) $ Ġis Ġa Ġcorrect Ġfact Ġor Ġnot . ĠSince Ġour Ġused Ġtest Ġsets Ġ( WN 18 Ġand ĠFB 15 K ) Ġonly Ġcontain Ġcorrect Ġtriple ts , Ġwe Ġconstruct Ġnegative Ġtriple ts Ġfollowing Ġthe Ġsame Ġsetting Ġused Ġin ĠB IB REF 3 Ġ. </s> For Ġtriple ts Ġclassification , Ġwe Ġset Ġa Ġthreshold Ġ$\ d elta Ġ_ r $ Ġfor Ġeach Ġrelation Ġ$ r $ Ġ. Ġ$\ d elta Ġ_ r $ Ġis Ġobtained Ġby Ġmaximizing Ġthe Ġclassification Ġaccur acies Ġon Ġthe Ġvalid Ġset . ĠFor Ġa Ġgiven Ġtriple t Ġ$( h , Ġr , Ġt ) $ Ġ, Ġif Ġits Ġscore Ġis Ġlarger Ġthan Ġ$\ d elta Ġ_ r $ Ġ, Ġit Ġwill Ġbe Ġclassified Ġas Ġpositive , Ġotherwise Ġnegative . </s> Table Ġ4 Ġshows Ġthe Ġevaluation Ġresults Ġof Ġtriple ts Ġclassification . ĠThe Ġresults Ġreveal Ġthat Ġour Ġjoint Ġencoding Ġmodels Ġis Ġeffective Ġand Ġalso Ġoutper form Ġthe Ġbaseline Ġmethod . </s> On ĠW N 18 , ĠâĢ ľ J oint ly ( A - L ST M ) âĢ Ŀ Ġachieves Ġthe Ġbest Ġperformance , Ġand Ġthe ĠâĢ ľ J oint ly ( L ST M ) âĢ Ŀ Ġis Ġslightly Ġworse Ġthan ĠâĢ ľ J oint ly ( A - L ST M ) âĢ Ŀ . ĠThe Ġreason Ġis Ġthat Ġthe Ġnumber Ġof Ġrelations Ġis Ġrelatively Ġsmall . ĠTherefore , Ġthe Ġattention Ġmechanism Ġdoes Ġnot Ġshow Ġobvious Ġadvantage . ĠOn ĠFB 15 K , Ġthe Ġclassification Ġaccuracy Ġof ĠâĢ ľ J oint ly ( A - L ST M ) âĢ Ŀ Ġachieves Ġ91 . 5 %, Ġwhich Ġis Ġthe Ġbest Ġand Ġsignificantly Ġhigher Ġthan Ġthat Ġof Ġstate - of - the - art Ġmethods . </s> Related ĠWork </s> Recently , Ġit Ġhas Ġgained Ġlots Ġof Ġinterests Ġto Ġjointly Ġlearn Ġthe Ġembed d ings Ġof Ġknowledge Ġgraph Ġand Ġtext Ġinformation . ĠThere Ġare Ġseveral Ġmethods Ġusing Ġtextual Ġinformation Ġto Ġhelp ĠK G Ġrepresentation Ġlearning . </s> BI B REF 3 Ġrepresent Ġan Ġentity Ġas Ġthe Ġaverage Ġof Ġits Ġword Ġembed d ings Ġin Ġentity Ġname , Ġallowing Ġthe Ġsharing Ġof Ġtextual Ġinformation Ġlocated Ġin Ġsimilar Ġentity Ġnames . </s> BI B REF 5 Ġjointly Ġembed Ġknowledge Ġand Ġtext Ġinto Ġthe Ġsame Ġspace Ġby Ġalign ing Ġthe Ġentity Ġname Ġand Ġits ĠWikipedia Ġanchor , Ġwhich Ġbrings Ġpromising Ġimprovements Ġto Ġthe Ġaccuracy Ġof Ġpredicting Ġfacts . ĠB IB REF 6 Ġextend Ġthe Ġjoint Ġmodel Ġand Ġalign s Ġknowledge Ġand Ġwords Ġin Ġthe Ġentity Ġdescriptions . ĠHowever , Ġthese Ġtwo Ġworks Ġalign Ġthe Ġtwo Ġkinds Ġof Ġembed d ings Ġon Ġword Ġlevel , Ġwhich Ġcan Ġlose Ġsome Ġsemantic Ġinformation Ġon Ġphrase Ġor Ġsentence Ġlevel . </s> BI B REF 25 Ġalso Ġrepresent Ġentities Ġwith Ġentity Ġnames Ġor Ġthe Ġaverage Ġof Ġword Ġembed d ings Ġin Ġdescriptions . ĠHowever , Ġtheir Ġuse Ġof Ġdescriptions Ġneglect s Ġword Ġorders , Ġand Ġthe Ġuse Ġof Ġentity Ġnames Ġstruggles Ġwith Ġambiguity . ĠB IB REF 7 Ġjointly Ġlearn Ġknowledge Ġgraph Ġembed d ings Ġwith Ġentity Ġdescriptions . ĠThey Ġuse Ġcontinuous Ġbag - of - words Ġand Ġconv olution al Ġneural Ġnetwork Ġto Ġencode Ġsemantics Ġof Ġentity Ġdescriptions . ĠHowever , Ġthey Ġseparate Ġthe Ġobjective Ġfunctions Ġinto Ġtwo Ġenergy Ġfunctions Ġof Ġstructure - based Ġand Ġdescription - based Ġrepresentations . ĠB IB REF 26 Ġembed s Ġboth Ġentity Ġand Ġrelation Ġembed d ings Ġby Ġtaking ĠK G Ġand Ġtext Ġinto Ġconsideration Ġusing ĠCNN . ĠTo Ġutilize Ġboth Ġrepresentations , Ġthey Ġneed Ġfurther Ġestimate Ġan Ġoptimum Ġweight Ġcoefficients Ġto Ġcombine Ġthem Ġtogether Ġin Ġthe Ġspecific Ġtasks . </s> Besides Ġentity Ġrepresentation , Ġthere Ġare Ġalso Ġa Ġlot Ġof Ġworks ĠB IB REF 27 Ġ, ĠB IB REF 28 Ġ, ĠB IB REF 29 Ġto Ġmap Ġtextual Ġrelations Ġand Ġknowledge Ġbase Ġrelations Ġto Ġthe Ġsame Ġvector Ġspace Ġand Ġobtained Ġsubstantial Ġimprovements . </s> While Ġreleasing Ġthe Ġcurrent Ġpaper Ġwe Ġdiscovered Ġa Ġpaper Ġby ĠB IB REF 30 Ġproposing Ġa Ġsimilar Ġmodel Ġwith Ġattention Ġmechanism Ġwhich Ġis Ġevaluated Ġon Ġlink Ġprediction Ġand Ġtriple t Ġclassification . ĠHowever , Ġour Ġwork Ġenc odes Ġtext Ġdescription Ġas Ġa Ġwhole Ġwithout Ġexplicit Ġsegment ation Ġof Ġsentences , Ġwhich Ġbreaks Ġthe Ġorder Ġand Ġco herence Ġamong Ġsentences . </s> Conclusion </s> We Ġpropose Ġa Ġunited Ġrepresentation Ġfor Ġknowledge Ġgraph , Ġutilizing Ġboth Ġstructure Ġand Ġtext Ġdescription Ġinformation Ġof Ġthe Ġentities . ĠExper iments Ġshow Ġthat Ġour Ġproposed Ġjointly Ġrepresentation Ġlearning Ġwith Ġg ating Ġmechanism Ġis Ġeffective , Ġwhich Ġbenefits Ġto Ġmodeling Ġthe Ġmeaning Ġof Ġan Ġentity . </s> In Ġthe Ġfuture , Ġwe Ġwill Ġconsider Ġthe Ġfollowing Ġresearch Ġdirections Ġto Ġimprove Ġour Ġmodel :
Introduction Knowledge graphs have been proved to benefit many artificial intelligence applications, such as relation extraction, question answering and so on. A knowledge graph consists of multi-relational data, having entities as nodes and relations as edges. An instance of fact is represented as a triplet (Head Entity, Relation, Tail Entity), where the Relation indicates a relationship between these two entities. In the past decades, great progress has been made in building large scale knowledge graphs, such as WordNet BIBREF0 , Freebase BIBREF1 . However, most of them have been built either collaboratively or semi-automatically and as a result, they often suffer from incompleteness and sparseness. The knowledge graph completion is to predict relations between entities based on existing triplets in a knowledge graph. Recently, a new powerful paradigm has been proposed to encode every element (entity or relation) of a knowledge graph into a low-dimensional vector space BIBREF2 , BIBREF3 . The representations of entities and relations are obtained by minimizing a global loss function involving all entities and relations. Therefore, we can do reasoning over knowledge graphs through algebraic computations. Although existing methods have good capability to learn knowledge graph embeddings, it remains challenging for entities with few or no facts BIBREF4 . To solve the issue of KB sparsity, many methods have been proposed to learn knowledge graph embeddings by utilizing related text information BIBREF5 , BIBREF6 , BIBREF7 . These methods learn joint embedding of entities, relations, and words (or phrases, sentences) into the same vector space. However, there are still three problems to be solved. (1) The combination methods of the structural and textual representations are not well studied in these methods, in which two kinds of representations are merely aligned on word level or separate loss function. (2) The text description may represent an entity from various aspects, and various relations only focus on fractional aspects of the description. A good encoder should select the information from text in accordance with certain contexts of relations. Figure 1 illustrates the fact that not all information provided in its description are useful to predict the linked entities given a specific relation. (3) Intuitively, entities with many facts depend more on well-trained structured representation while those with few or no facts might be largely determined by text descriptions. A good representation should learn the most valuable information by balancing both sides. In this paper, we propose a new deep architecture to learn the knowledge representation by utilizing the existing text descriptions of entities. Specifically, we learn a joint representation of each entity from two information sources: one is structure information, and another is its text description. The joint representation is the combination of the structure and text representations with a gating mechanism. The gate decides how much information from the structure or text representation will carry over to the final joint representation. In addition, we also introduce an attention mechanism to select the most related information from text description under different contexts. Experimental results on link prediction and triplet classification show that our joint models can handle the sparsity problem well and outperform the baseline method on all metrics with a large margin. Our contributions in this paper are summarized as follows. Knowledge Graph Embedding In this section, we briefly introduce the background knowledge about the knowledge graph embedding. Knowledge graph embedding aims to model multi-relational data (entities and relations) into a continuous low-dimensional vector space. Given a pair of entities $(h,t)$ and their relation $r$ , we can represent them with a triple $(h,r,t)$ . A score function $f(h,r, t)$ is defined to model the correctness of the triple $(h,r,t)$ , thus to distinguish whether two entities $h$ and $t$ are in a certain relationship $r$ . $f(h,r, t)$ should be larger for a golden triplet $(h, r, t)$ that corresponds to a true fact in real world, otherwise $r$0 should be lower for an negative triplet. The difference among the existing methods varies between linear BIBREF2 , BIBREF8 and nonlinear BIBREF3 score functions in the low-dimensional vector space. Among these methods, TransE BIBREF2 is a simple and effective approach, which learns the vector embeddings for both entities and relationships. Its basic idea is that the relationship between two entities is supposed to correspond to a translation between the embeddings of entities, that is, $\textbf {h}+ \mathbf {r}\approx \mathbf {t}$ when $(h,r,t)$ holds. TransE's score function is defined as: $$f(h,r,t)) &= -\Vert \textbf {h}+\mathbf {r}-\mathbf {t}\Vert _{2}^2$$   (Eq. 5) where $\textbf {h},\mathbf {t},\mathbf {r}\in \mathbb {R}^d$ are embeddings of $h,t,r$ respectively, and satisfy $\Vert \textbf {h}\Vert ^2_2=\Vert \mathbf {t}\Vert ^2_2=1$ . The $\textbf {h}, \mathbf {r}, \mathbf {t}$ are indexed by a lookup table respectively. Neural Text Encoding Given an entity in most of the existing knowledge bases, there is always an available corresponding text description with valuable semantic information for this entity, which can provide beneficial supplement for entity representation. To encode the representation of a entity from its text description, we need to encode the variable-length sentence to a fixed-length vector. There are several kinds of neural models used in sentence modeling. These models generally consist of a projection layer that maps words, sub-word units or n-grams to vector representations (often trained beforehand with unsupervised methods), and then combine them with the different architectures of neural networks, such as neural bag-of-words (NBOW), recurrent neural network (RNN) BIBREF9 , BIBREF10 , BIBREF11 and convolutional neural network (CNN) BIBREF12 , BIBREF13 . In this paper, we use three encoders (NBOW, LSTM and attentive LSTM) to model the text descriptions. Bag-of-Words Encoder A simple and intuitive method is the neural bag-of-words (NBOW) model, in which the representation of text can be generated by summing up its constituent word representations. We denote the text description as word sequence $x_{1:n} = x_1,\cdots ,x_n$ , where $x_i$ is the word at position $i$ . The NBOW encoder is $$\mathrm {enc_1}(x_{1:n}) = \sum _{i=1}^{n} \mathbf {x}_i,$$   (Eq. 7) where $\mathbf {x}_i \in \mathbb {R}^d$ is the word embedding of $x_i$ . LSTM Encoder To address some of the modelling issues with NBOW, we consider using a bidirectional long short-term memory network (LSTM) BIBREF14 , BIBREF15 to model the text description. LSTM was proposed by BIBREF16 to specifically address this issue of learning long-term dependencies BIBREF17 , BIBREF18 , BIBREF16 in RNN. The LSTM maintains a separate memory cell inside it that updates and exposes its content only when deemed necessary. Bidirectional LSTM (BLSTM) can be regarded as two separate LSTMs with different directions. One LSTM models the text description from left to right, and another LSTM models text description from right to left respectively. We define the outputs of two LSTM at time step $i$ are $\overrightarrow{\mathbf {z}}_i$ and $\overleftarrow{\mathbf {z}}_i$ respectively. The combined output of BLSTM at position $i$ is ${\mathbf {z}_i} = \overrightarrow{\mathbf {z}}_i \oplus \overleftarrow{\mathbf {z}}_i$ , where $\oplus $ denotes the concatenation operation. The LSTM encoder combines all the outputs $\mathbf {z}_i \in \mathbb {R}^d$ of BLSTM at different position. $$\mathrm {enc_2}(x_{1:n}) = \sum _{i=1}^{n} {\mathbf {z}_i}.$$   (Eq. 9) Attentive LSTM Encoder While the LSTM encoder has richer capacity than NBOW, it produces the same representation for the entire text description regardless of its contexts. However, the text description may present an entity from various aspects, and various relations only focus on fractional aspects of the description. This phenomenon also occurs in structure embedding for an entity BIBREF8 , BIBREF19 . Given a relation for an entity, not all of words/phrases in its text description are useful to model a specific fact. Some of them may be important for the given relation, but may be useless for other relations. Therefore, we introduce an attention mechanism BIBREF20 to utilize an attention-based encoder that constructs contextual text encodings according to different relations. For each position $i$ of the text description, the attention for a given relation $r$ is defined as $\alpha _i(r)$ , which is $$e_i(r) &= \mathbf {v}_a^T \tanh (\mathbf {W}_a {\mathbf {z}}_i + \mathbf {U}_a \mathbf {r}), \\ \alpha _i(r)&=\operatorname{\mathbf {softmax}}(e_i(r))\nonumber \\ &=\frac{\exp (e_i(r))}{\sum ^{n}_{j=1} \exp (e_j(r))},$$   (Eq. 12) where $\mathbf {r}\in \mathbb {R}^d$ is the relation embedding; ${\mathbf {z}}_i \in \mathbb {R}^d$ is the output of BLSTM at position $i$ ; $\mathbf {W}_a,\mathbf {U}_a \in \mathbb {R}^{d\times d}$ are parameters matrices; $\mathbf {v}_a \in \mathbb {R}^{d}$ is a parameter vector. The attention $\alpha _i(r)$ is interpreted as the degree to which the network attends to partial representation $\mathbf {z}_{i}$ for given relation $r$ . The contextual encoding of text description can be formed by a weighted sum of the encoding $\mathbf {z}_{i}$ with attention. $$\mathbf {enc_3}(x_{1:n};r) &= \sum _{i=1}^{n} \alpha _i(r) * \mathbf {z}_i.$$   (Eq. 13) Joint Structure and Text Encoder Since both the structure and text description provide valuable information for an entity , we wish to integrate all these information into a joint representation. We propose a united model to learn a joint representation of both structure and text information. The whole model can be end-to-end trained. For an entity $e$ , we denote $\mathbf {e}_s$ to be its embedding of structure information, $\mathbf {e}_d$ to be encoding of its text descriptions. The main concern is how to combine $\mathbf {e}_s$ and $\mathbf {e}_d$ . To integrate two kinds of representations of entities, we use gating mechanism to decide how much the joint representation depends on structure or text. The joint representation $\mathbf {e}$ is a linear interpolation between the $\mathbf {e}_s$ and $\mathbf {e}_d$ . $$\mathbf {e}= \textbf {g}_e \odot \mathbf {e}_s + (1-\textbf {g}_e)\odot \mathbf {e}_d,$$   (Eq. 14) where $\textbf {g}_e$ is a gate to balance two sources information and its elements are in $[0,1]$ , and $\odot $ is an element-wise multiplication. Intuitively, when the gate is close to 0, the joint representation is forced to ignore the structure information and is the text representation only. Training We use the contrastive max-margin criterion BIBREF2 , BIBREF3 to train our model. Intuitively, the max-margin criterion provides an alternative to probabilistic, likelihood-based estimation methods by concentrating directly on the robustness of the decision boundary of a model BIBREF23 . The main idea is that each triplet $(h,r,t)$ coming from the training corpus should receives a higher score than a triplet in which one of the elements is replaced with a random elements. We assume that there are $n_t$ triplets in training set and denote the $i$ th triplet by $(h_i, r_i, t_i),(i = 1, 2, \cdots ,n_t)$ . Each triplet has a label $y_i$ to indicate the triplet is positive ( $y_i = 1$ ) or negative ( $y_i = 0$ ). Then the golden and negative triplets are denoted by $\mathcal {D} = \lbrace (h_j, r_j, t_j) | y_j = 1\rbrace $ and $\mathcal {\hat{D}} = \lbrace (h_j, r_j, t_j) | y_j = 0\rbrace $ , respectively. The positive example are the triplets from training dataset, and the negative examples are generated as follows: $ \mathcal {\hat{D}} = \lbrace (h_l, r_k, t_k) | h_l \ne h_k \wedge y_k = 1\rbrace \cup \lbrace (h_k, r_k, t_l) | t_l \ne t_k \wedge y_k = 1\rbrace \cup \lbrace (h_k, r_l, t_k) | r_l \ne r_k \wedge y_k = 1\rbrace $ . The sampling strategy is Bernoulli distribution described in BIBREF8 . Let the set of all parameters be $\Theta $ , we minimize the following objective: $$J(\Theta )=\sum _{(h,r,t) \in \mathcal {D}}\sum _{( \hat{h},\hat{r},\hat{t}) \in \mathcal {\hat{D}}} \max \left(0,\gamma - \right. \nonumber \\ f( h,r,t)+f(\hat{h},\hat{r},\hat{t})\left.\right)+ \eta \Vert \Theta \Vert _2^2,$$   (Eq. 22) where $\gamma > 0$ is a margin between golden triplets and negative triplets., $f(h, r, t)$ is the score function. We use the standard $L_2$ regularization of all the parameters, weighted by the hyperparameter $\eta $ . Experiment In this section, we study the empirical performance of our proposed models on two benchmark tasks: triplet classification and link prediction. Datasets We use two popular knowledge bases: WordNet BIBREF0 and Freebase BIBREF1 in this paper. Specifically, we use WN18 (a subset of WordNet) BIBREF24 and FB15K (a subset of Freebase) BIBREF2 since their text descriptions are easily publicly available. Table 1 lists statistics of the two datasets. Link Prediction Link prediction is a subtask of knowledge graph completion to complete a triplet $(h, r, t)$ with $h$ or $t$ missing, i.e., predict $t$ given $(h, r)$ or predict $h$ given $(r, t)$ . Rather than requiring one best answer, this task emphasizes more on ranking a set of candidate entities from the knowledge graph. Similar to BIBREF2 , we use two measures as our evaluation metrics. (1) Mean Rank: the averaged rank of correct entities or relations; (2) Hits@p: the proportion of valid entities or relations ranked in top $p$ predictions. Here, we set $p=10$ for entities and $p=1$ for relations. A lower Mean Rank and a higher Hits@p should be achieved by a good embedding model. We call this evaluation setting “Raw”. Since a false predicted triplet may also exist in knowledge graphs, it should be regard as a valid triplet. Hence, we should remove the false predicted triplets included in training, validation and test sets before ranking (except the test triplet of interest). We call this evaluation setting “Filter”. The evaluation results are reported under these two settings. We select the margin $\gamma $ among $\lbrace 1, 2\rbrace $ , the embedding dimension $d$ among $\lbrace 20, 50, 100\rbrace $ , the regularization $\eta $ among $\lbrace 0, 1E{-5}, 1E{-6}\rbrace $ , two learning rates $\lambda _s$ and $\lambda _t$ among $\lbrace 0.001, 0.01, 0.05\rbrace $ to learn the parameters of structure and text encoding. The dissimilarity measure is set to either $L_1$ or $\lbrace 1, 2\rbrace $0 distance. In order to speed up the convergence and avoid overfitting, we initiate the structure embeddings of entity and relation with the results of TransE. The embedding of a word is initialized by averaging the linked entity embeddings whose description include this word. The rest parameters are initialized by randomly sampling from uniform distribution in $[-0.1, 0.1]$ . The final optimal configurations are: $\gamma = 2$ , $d=20$ , $\eta =1E{-5}$ , $\lambda _s = 0.01$ , $\lambda _t = 0.1$ , and $L_1$ distance on WN18; $\gamma =2$ , $d=100$ , $\eta =1E{-5}$ , $\lambda _s = 0.01$ , $d=20$0 , and $d=20$1 distance on FB15K. Experimental results on both WN18 and FB15k are shown in Table 2 , where we use “Jointly(CBOW)”, “Jointly(LSTM)” and “Jointly(A-LSTM)” to represent our jointly encoding models with CBOW, LSTM and attentive LSTM text encoders. Our baseline is TransE since that the score function of our models is based on TransE. From the results, we observe that proposed models surpass the baseline, TransE, on all metrics, which indicates that knowledge representation can benefit greatly from text description. On WN18, the reason why “Jointly(A-LSTM)” is slightly worse than “Jointly(LSTM)” is probably because the number of relations is limited. Therefore, the attention mechanism does not have obvious advantage. On FB15K, “Jointly(A-LSTM)” achieves the best performance and is significantly higher than baseline methods on mean rank. Although the Hits@10 of our models are worse than the best state-of-the-art method, TransD, it is worth noticing that the score function of our models is based on TransE, not TransD. Our models are compatible with other state-of-the-art knowledge embedding models. We believe that our model can be further improved by adopting the score functions of other state-of-the-art methods, such as TransD. Besides, textual information largely alleviates the issue of sparsity and our model achieves substantial improvement on Mean Rank comparing with TransD. However, textual information may slightly degrade the representation of frequent entities which have been well-trained. This may be another reason why our Hits@10 is worse than TransD which only utilizes structural information. For the comparison of Hits@10 of different kinds of relations, we categorized the relationships according to the cardinalities of their head and tail arguments into four classes: 1-to-1, 1-to-many, many-to-1, many-to-many. Mapping properties of relations follows the same rules in BIBREF2 . Table 3 shows the detailed results by mapping properties of relations on FB15k. We can see that our models outperform baseline TransE in all types of relations (1-to-1, 1-to-N, N-to-1 and N-to-N), especially when (1) predicting “1-to-1” relations and (2) predicting the 1 side for “1-to-N” and “N-to-1” relations. To get more insights into how the joint representation is influenced by the structure and text information. We observe the activations of gates, which control the balance between two sources of information, to understand the behavior of neurons. We sort the entities by their frequencies and divide them into 50 equal-size groups of different frequencies, and average the values of all gates in each group. Figure 3 gives the average of gates in ten groups from high- to low-frequency. We observe that the text information play more important role for the low-frequency entities. Triplet Classification Triplet classification is a binary classification task, which aims to judge whether a given triplet $(h, r, t)$ is a correct fact or not. Since our used test sets (WN18 and FB15K) only contain correct triplets, we construct negative triplets following the same setting used in BIBREF3 . For triplets classification, we set a threshold $\delta _r$ for each relation $r$ . $\delta _r$ is obtained by maximizing the classification accuracies on the valid set. For a given triplet $(h, r, t)$ , if its score is larger than $\delta _r$ , it will be classified as positive, otherwise negative. Table 4 shows the evaluation results of triplets classification. The results reveal that our joint encoding models is effective and also outperform the baseline method. On WN18, “Jointly(A-LSTM)” achieves the best performance, and the “Jointly(LSTM)” is slightly worse than “Jointly(A-LSTM)”. The reason is that the number of relations is relatively small. Therefore, the attention mechanism does not show obvious advantage. On FB15K, the classification accuracy of “Jointly(A-LSTM)” achieves 91.5%, which is the best and significantly higher than that of state-of-the-art methods. Related Work Recently, it has gained lots of interests to jointly learn the embeddings of knowledge graph and text information. There are several methods using textual information to help KG representation learning. BIBREF3 represent an entity as the average of its word embeddings in entity name, allowing the sharing of textual information located in similar entity names. BIBREF5 jointly embed knowledge and text into the same space by aligning the entity name and its Wikipedia anchor, which brings promising improvements to the accuracy of predicting facts. BIBREF6 extend the joint model and aligns knowledge and words in the entity descriptions. However, these two works align the two kinds of embeddings on word level, which can lose some semantic information on phrase or sentence level. BIBREF25 also represent entities with entity names or the average of word embeddings in descriptions. However, their use of descriptions neglects word orders, and the use of entity names struggles with ambiguity. BIBREF7 jointly learn knowledge graph embeddings with entity descriptions. They use continuous bag-of-words and convolutional neural network to encode semantics of entity descriptions. However, they separate the objective functions into two energy functions of structure-based and description-based representations. BIBREF26 embeds both entity and relation embeddings by taking KG and text into consideration using CNN. To utilize both representations, they need further estimate an optimum weight coefficients to combine them together in the specific tasks. Besides entity representation, there are also a lot of works BIBREF27 , BIBREF28 , BIBREF29 to map textual relations and knowledge base relations to the same vector space and obtained substantial improvements. While releasing the current paper we discovered a paper by BIBREF30 proposing a similar model with attention mechanism which is evaluated on link prediction and triplet classification. However, our work encodes text description as a whole without explicit segmentation of sentences, which breaks the order and coherence among sentences. Conclusion We propose a united representation for knowledge graph, utilizing both structure and text description information of the entities. Experiments show that our proposed jointly representation learning with gating mechanism is effective, which benefits to modeling the meaning of an entity. In the future, we will consider the following research directions to improve our model:
What datasets are used to evaluate this paper?
b36f867fcda5ad62c46d23513369337352aa01d2
What Ġdatasets Ġare Ġused Ġto Ġevaluate Ġthis Ġpaper ?
Word Net ĠB IB REF 0 , ĠFree base ĠB IB REF 1 , ĠW N 18 Ġ( a Ġsubset Ġof ĠWord Net ) ĠB IB REF 24 Ġ, ĠFB 15 K Ġ( a Ġsubset Ġof ĠFree base ) ĠB IB REF 2
<s> What Ġdatasets Ġare Ġused Ġto Ġevaluate Ġthis Ġpaper ? </s> Introduction </s> Know ledge Ġgraphs Ġhave Ġbeen Ġproved Ġto Ġbenefit Ġmany Ġartificial Ġintelligence Ġapplications , Ġsuch Ġas Ġrelation Ġextraction , Ġquestion Ġanswering Ġand Ġso Ġon . ĠA Ġknowledge Ġgraph Ġconsists Ġof Ġmulti - rel ational Ġdata , Ġhaving Ġentities Ġas Ġnodes Ġand Ġrelations Ġas Ġedges . ĠAn Ġinstance Ġof Ġfact Ġis Ġrepresented Ġas Ġa Ġtriple t Ġ( Head ĠEntity , ĠRel ation , ĠTail ĠEntity ), Ġwhere Ġthe ĠRel ation Ġindicates Ġa Ġrelationship Ġbetween Ġthese Ġtwo Ġentities . ĠIn Ġthe Ġpast Ġdecades , Ġgreat Ġprogress Ġhas Ġbeen Ġmade Ġin Ġbuilding Ġlarge Ġscale Ġknowledge Ġgraphs , Ġsuch Ġas ĠWord Net ĠB IB REF 0 Ġ, ĠFree base ĠB IB REF 1 Ġ. ĠHowever , Ġmost Ġof Ġthem Ġhave Ġbeen Ġbuilt Ġeither Ġcollabor atively Ġor Ġsemi - aut om atically Ġand Ġas Ġa Ġresult , Ġthey Ġoften Ġsuffer Ġfrom Ġincom ple teness Ġand Ġsp ars eness . </s> The Ġknowledge Ġgraph Ġcompletion Ġis Ġto Ġpredict Ġrelations Ġbetween Ġentities Ġbased Ġon Ġexisting Ġtriple ts Ġin Ġa Ġknowledge Ġgraph . ĠRecently , Ġa Ġnew Ġpowerful Ġparadigm Ġhas Ġbeen Ġproposed Ġto Ġencode Ġevery Ġelement Ġ( entity Ġor Ġrelation ) Ġof Ġa Ġknowledge Ġgraph Ġinto Ġa Ġlow - dimensional Ġvector Ġspace ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ. ĠThe Ġrepresentations Ġof Ġentities Ġand Ġrelations Ġare Ġobtained Ġby Ġminimizing Ġa Ġglobal Ġloss Ġfunction Ġinvolving Ġall Ġentities Ġand Ġrelations . ĠTherefore , Ġwe Ġcan Ġdo Ġreasoning Ġover Ġknowledge Ġgraphs Ġthrough Ġalgebra ic Ġcomput ations . </s> Although Ġexisting Ġmethods Ġhave Ġgood Ġcapability Ġto Ġlearn Ġknowledge Ġgraph Ġembed d ings , Ġit Ġremains Ġchallenging Ġfor Ġentities Ġwith Ġfew Ġor Ġno Ġfacts ĠB IB REF 4 Ġ. ĠTo Ġsolve Ġthe Ġissue Ġof ĠKB Ġsp arsity , Ġmany Ġmethods Ġhave Ġbeen Ġproposed Ġto Ġlearn Ġknowledge Ġgraph Ġembed d ings Ġby Ġutilizing Ġrelated Ġtext Ġinformation ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ. ĠThese Ġmethods Ġlearn Ġjoint Ġembed ding Ġof Ġentities , Ġrelations , Ġand Ġwords Ġ( or Ġphrases , Ġsentences ) Ġinto Ġthe Ġsame Ġvector Ġspace . ĠHowever , Ġthere Ġare Ġstill Ġthree Ġproblems Ġto Ġbe Ġsolved . Ġ( 1 ) ĠThe Ġcombination Ġmethods Ġof Ġthe Ġstructural Ġand Ġtextual Ġrepresentations Ġare Ġnot Ġwell Ġstudied Ġin Ġthese Ġmethods , Ġin Ġwhich Ġtwo Ġkinds Ġof Ġrepresentations Ġare Ġmerely Ġaligned Ġon Ġword Ġlevel Ġor Ġseparate Ġloss Ġfunction . Ġ( 2 ) ĠThe Ġtext Ġdescription Ġmay Ġrepresent Ġan Ġentity Ġfrom Ġvarious Ġaspects , Ġand Ġvarious Ġrelations Ġonly Ġfocus Ġon Ġfraction al Ġaspects Ġof Ġthe Ġdescription . ĠA Ġgood Ġenc oder Ġshould Ġselect Ġthe Ġinformation Ġfrom Ġtext Ġin Ġaccordance Ġwith Ġcertain Ġcontexts Ġof Ġrelations . ĠFigure Ġ1 Ġillustrates Ġthe Ġfact Ġthat Ġnot Ġall Ġinformation Ġprovided Ġin Ġits Ġdescription Ġare Ġuseful Ġto Ġpredict Ġthe Ġlinked Ġentities Ġgiven Ġa Ġspecific Ġrelation . Ġ( 3 ) ĠInt uitive ly , Ġentities Ġwith Ġmany Ġfacts Ġdepend Ġmore Ġon Ġwell - trained Ġstructured Ġrepresentation Ġwhile Ġthose Ġwith Ġfew Ġor Ġno Ġfacts Ġmight Ġbe Ġlargely Ġdetermined Ġby Ġtext Ġdescriptions . ĠA Ġgood Ġrepresentation Ġshould Ġlearn Ġthe Ġmost Ġvaluable Ġinformation Ġby Ġbalancing Ġboth Ġsides . </s> In Ġthis Ġpaper , Ġwe Ġpropose Ġa Ġnew Ġdeep Ġarchitecture Ġto Ġlearn Ġthe Ġknowledge Ġrepresentation Ġby Ġutilizing Ġthe Ġexisting Ġtext Ġdescriptions Ġof Ġentities . ĠSpecifically , Ġwe Ġlearn Ġa Ġjoint Ġrepresentation Ġof Ġeach Ġentity Ġfrom Ġtwo Ġinformation Ġsources : Ġone Ġis Ġstructure Ġinformation , Ġand Ġanother Ġis Ġits Ġtext Ġdescription . ĠThe Ġjoint Ġrepresentation Ġis Ġthe Ġcombination Ġof Ġthe Ġstructure Ġand Ġtext Ġrepresentations Ġwith Ġa Ġg ating Ġmechanism . ĠThe Ġgate Ġdecides Ġhow Ġmuch Ġinformation Ġfrom Ġthe Ġstructure Ġor Ġtext Ġrepresentation Ġwill Ġcarry Ġover Ġto Ġthe Ġfinal Ġjoint Ġrepresentation . ĠIn Ġaddition , Ġwe Ġalso Ġintroduce Ġan Ġattention Ġmechanism Ġto Ġselect Ġthe Ġmost Ġrelated Ġinformation Ġfrom Ġtext Ġdescription Ġunder Ġdifferent Ġcontexts . ĠExperimental Ġresults Ġon Ġlink Ġprediction Ġand Ġtriple t Ġclassification Ġshow Ġthat Ġour Ġjoint Ġmodels Ġcan Ġhandle Ġthe Ġsp arsity Ġproblem Ġwell Ġand Ġoutper form Ġthe Ġbaseline Ġmethod Ġon Ġall Ġmetrics Ġwith Ġa Ġlarge Ġmargin . </s> Our Ġcontributions Ġin Ġthis Ġpaper Ġare Ġsummarized Ġas Ġfollows . </s> Know ledge ĠGraph ĠEmb edd ing </s> In Ġthis Ġsection , Ġwe Ġbriefly Ġintroduce Ġthe Ġbackground Ġknowledge Ġabout Ġthe Ġknowledge Ġgraph Ġembed ding . </s> Know ledge Ġgraph Ġembed ding Ġaims Ġto Ġmodel Ġmulti - rel ational Ġdata Ġ( ent ities Ġand Ġrelations ) Ġinto Ġa Ġcontinuous Ġlow - dimensional Ġvector Ġspace . ĠGiven Ġa Ġpair Ġof Ġentities Ġ$( h , t ) $ Ġand Ġtheir Ġrelation Ġ$ r $ Ġ, Ġwe Ġcan Ġrepresent Ġthem Ġwith Ġa Ġtriple Ġ$( h , r , t ) $ Ġ. ĠA Ġscore Ġfunction Ġ$ f ( h , r , Ġt ) $ Ġis Ġdefined Ġto Ġmodel Ġthe Ġcorrectness Ġof Ġthe Ġtriple Ġ$( h , r , t ) $ Ġ, Ġthus Ġto Ġdistinguish Ġwhether Ġtwo Ġentities Ġ$ h $ Ġand Ġ$ t $ Ġare Ġin Ġa Ġcertain Ġrelationship Ġ$ r $ Ġ. Ġ$ f ( h , r , Ġt ) $ Ġshould Ġbe Ġlarger Ġfor Ġa Ġgolden Ġtriple t Ġ$( h , Ġr , Ġt ) $ Ġthat Ġcorresponds Ġto Ġa Ġtrue Ġfact Ġin Ġreal Ġworld , Ġotherwise Ġ$ r $ 0 Ġshould Ġbe Ġlower Ġfor Ġan Ġnegative Ġtriple t . </s> The Ġdifference Ġamong Ġthe Ġexisting Ġmethods Ġvaries Ġbetween Ġlinear ĠB IB REF 2 Ġ, ĠB IB REF 8 Ġand Ġnon linear ĠB IB REF 3 Ġscore Ġfunctions Ġin Ġthe Ġlow - dimensional Ġvector Ġspace . </s> Among Ġthese Ġmethods , ĠTrans E ĠB IB REF 2 Ġis Ġa Ġsimple Ġand Ġeffective Ġapproach , Ġwhich Ġlearns Ġthe Ġvector Ġembed d ings Ġfor Ġboth Ġentities Ġand Ġrelationships . ĠIts Ġbasic Ġidea Ġis Ġthat Ġthe Ġrelationship Ġbetween Ġtwo Ġentities Ġis Ġsupposed Ġto Ġcorrespond Ġto Ġa Ġtranslation Ġbetween Ġthe Ġembed d ings Ġof Ġentities , Ġthat Ġis , Ġ$\ text bf Ġ{ h } + Ġ\ math bf Ġ{ r }\ app rox Ġ\ math bf Ġ{ t } $ Ġwhen Ġ$( h , r , t ) $ Ġholds . </s> Trans E 's Ġscore Ġfunction Ġis Ġdefined Ġas : </s> $$ f ( h , r , t )) Ġ& = Ġ- \ Vert Ġ\ text bf Ġ{ h } + \ math bf Ġ{ r } - \ math bf Ġ{ t }\ Vert Ġ_ { 2 } ^ 2 $$ Ġ Ġ Ġ( E q . Ġ5 ) </s> where Ġ$\ text bf Ġ{ h }, \ math bf Ġ{ t }, \ math bf Ġ{ r }\ in Ġ\ math bb Ġ{ R } ^ d $ Ġare Ġembed d ings Ġof Ġ$ h , t , r $ Ġrespectively , Ġand Ġsatisfy Ġ$\ Vert Ġ\ text bf Ġ{ h }\ Vert Ġ^ 2 _ 2 = \ Vert Ġ\ math bf Ġ{ t }\ Vert Ġ^ 2 _ 2 = 1 $ Ġ. ĠThe Ġ$\ text bf Ġ{ h }, Ġ\ math bf Ġ{ r }, Ġ\ math bf Ġ{ t } $ Ġare Ġindexed Ġby Ġa Ġlookup Ġtable Ġrespectively . </s> Ne ural ĠText ĠEnc oding </s> Given Ġan Ġentity Ġin Ġmost Ġof Ġthe Ġexisting Ġknowledge Ġbases , Ġthere Ġis Ġalways Ġan Ġavailable Ġcorresponding Ġtext Ġdescription Ġwith Ġvaluable Ġsemantic Ġinformation Ġfor Ġthis Ġentity , Ġwhich Ġcan Ġprovide Ġbeneficial Ġsupplement Ġfor Ġentity Ġrepresentation . </s> To Ġencode Ġthe Ġrepresentation Ġof Ġa Ġentity Ġfrom Ġits Ġtext Ġdescription , Ġwe Ġneed Ġto Ġencode Ġthe Ġvariable - length Ġsentence Ġto Ġa Ġfixed - length Ġvector . ĠThere Ġare Ġseveral Ġkinds Ġof Ġneural Ġmodels Ġused Ġin Ġsentence Ġmodeling . ĠThese Ġmodels Ġgenerally Ġconsist Ġof Ġa Ġprojection Ġlayer Ġthat Ġmaps Ġwords , Ġsub - word Ġunits Ġor Ġn - gram s Ġto Ġvector Ġrepresentations Ġ( often Ġtrained Ġbeforehand Ġwith Ġun super vised Ġmethods ), Ġand Ġthen Ġcombine Ġthem Ġwith Ġthe Ġdifferent Ġarchitectures Ġof Ġneural Ġnetworks , Ġsuch Ġas Ġneural Ġbag - of - words Ġ( NB OW ), Ġrecurrent Ġneural Ġnetwork Ġ( R NN ) ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġand Ġconv olution al Ġneural Ġnetwork Ġ( CNN ) ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ. </s> In Ġthis Ġpaper , Ġwe Ġuse Ġthree Ġenc od ers Ġ( NB OW , ĠL ST M Ġand Ġattentive ĠL ST M ) Ġto Ġmodel Ġthe Ġtext Ġdescriptions . </s> B ag - of - Words ĠEnc oder </s> A Ġsimple Ġand Ġintuitive Ġmethod Ġis Ġthe Ġneural Ġbag - of - words Ġ( NB OW ) Ġmodel , Ġin Ġwhich Ġthe Ġrepresentation Ġof Ġtext Ġcan Ġbe Ġgenerated Ġby Ġsum ming Ġup Ġits Ġconstituent Ġword Ġrepresentations . </s> We Ġdenote Ġthe Ġtext Ġdescription Ġas Ġword Ġsequence Ġ$ x _{ 1 : n } Ġ= Ġx _ 1 , \ cd ots Ġ, x _ n $ Ġ, Ġwhere Ġ$ x _ i $ Ġis Ġthe Ġword Ġat Ġposition Ġ$ i $ Ġ. ĠThe ĠNB OW Ġenc oder Ġis </s> $$ \ math rm Ġ{ enc _ 1 } ( x _{ 1 : n }) Ġ= Ġ\ sum Ġ_ { i = 1 } ^{ n } Ġ\ math bf Ġ{ x } _ i , $$ Ġ Ġ Ġ( E q . Ġ7 ) </s> where Ġ$\ math bf Ġ{ x } _ i Ġ\ in Ġ\ math bb Ġ{ R } ^ d $ Ġis Ġthe Ġword Ġembed ding Ġof Ġ$ x _ i $ Ġ. </s> L ST M ĠEnc oder </s> To Ġaddress Ġsome Ġof Ġthe Ġmodelling Ġissues Ġwith ĠNB OW , Ġwe Ġconsider Ġusing Ġa Ġbid irection al Ġlong Ġshort - term Ġmemory Ġnetwork Ġ( L ST M ) ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġto Ġmodel Ġthe Ġtext Ġdescription . </s> L ST M Ġwas Ġproposed Ġby ĠB IB REF 16 Ġto Ġspecifically Ġaddress Ġthis Ġissue Ġof Ġlearning Ġlong - term Ġdependencies ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ, ĠB IB REF 16 Ġin ĠR NN . ĠThe ĠL ST M Ġmaintains Ġa Ġseparate Ġmemory Ġcell Ġinside Ġit Ġthat Ġupdates Ġand Ġexposes Ġits Ġcontent Ġonly Ġwhen Ġdeemed Ġnecessary . </s> B id irection al ĠL ST M Ġ( BL ST M ) Ġcan Ġbe Ġregarded Ġas Ġtwo Ġseparate ĠL ST Ms Ġwith Ġdifferent Ġdirections . ĠOne ĠL ST M Ġmodels Ġthe Ġtext Ġdescription Ġfrom Ġleft Ġto Ġright , Ġand Ġanother ĠL ST M Ġmodels Ġtext Ġdescription Ġfrom Ġright Ġto Ġleft Ġrespectively . ĠWe Ġdefine Ġthe Ġoutputs Ġof Ġtwo ĠL ST M Ġat Ġtime Ġstep Ġ$ i $ Ġare Ġ$\ over right arrow {\ math bf Ġ{ z }} _ i $ Ġand Ġ$\ over left arrow {\ math bf Ġ{ z }} _ i $ Ġrespectively . </s> The Ġcombined Ġoutput Ġof ĠBL ST M Ġat Ġposition Ġ$ i $ Ġis Ġ${ \ math bf Ġ{ z } _ i } Ġ= Ġ\ over right arrow {\ math bf Ġ{ z }} _ i Ġ\ opl us Ġ\ over left arrow {\ math bf Ġ{ z }} _ i $ Ġ, Ġwhere Ġ$\ opl us Ġ$ Ġdenotes Ġthe Ġconc aten ation Ġoperation . </s> The ĠL ST M Ġenc oder Ġcombines Ġall Ġthe Ġoutputs Ġ$\ math bf Ġ{ z } _ i Ġ\ in Ġ\ math bb Ġ{ R } ^ d $ Ġof ĠBL ST M Ġat Ġdifferent Ġposition . </s> $$ \ math rm Ġ{ enc _ 2 } ( x _{ 1 : n }) Ġ= Ġ\ sum Ġ_ { i = 1 } ^{ n } Ġ{\ math bf Ġ{ z } _ i }. $$ Ġ Ġ Ġ( E q . Ġ9 ) </s> Att ent ive ĠL ST M ĠEnc oder </s> While Ġthe ĠL ST M Ġenc oder Ġhas Ġricher Ġcapacity Ġthan ĠNB OW , Ġit Ġproduces Ġthe Ġsame Ġrepresentation Ġfor Ġthe Ġentire Ġtext Ġdescription Ġregardless Ġof Ġits Ġcontexts . ĠHowever , Ġthe Ġtext Ġdescription Ġmay Ġpresent Ġan Ġentity Ġfrom Ġvarious Ġaspects , Ġand Ġvarious Ġrelations Ġonly Ġfocus Ġon Ġfraction al Ġaspects Ġof Ġthe Ġdescription . ĠThis Ġphenomenon Ġalso Ġoccurs Ġin Ġstructure Ġembed ding Ġfor Ġan Ġentity ĠB IB REF 8 Ġ, ĠB IB REF 19 Ġ. </s> Given Ġa Ġrelation Ġfor Ġan Ġentity , Ġnot Ġall Ġof Ġwords / ph r ases Ġin Ġits Ġtext Ġdescription Ġare Ġuseful Ġto Ġmodel Ġa Ġspecific Ġfact . ĠSome Ġof Ġthem Ġmay Ġbe Ġimportant Ġfor Ġthe Ġgiven Ġrelation , Ġbut Ġmay Ġbe Ġuseless Ġfor Ġother Ġrelations . ĠTherefore , Ġwe Ġintroduce Ġan Ġattention Ġmechanism ĠB IB REF 20 Ġto Ġutilize Ġan Ġattention - based Ġenc oder Ġthat Ġconstructs Ġcontextual Ġtext Ġenc od ings Ġaccording Ġto Ġdifferent Ġrelations . </s> For Ġeach Ġposition Ġ$ i $ Ġof Ġthe Ġtext Ġdescription , Ġthe Ġattention Ġfor Ġa Ġgiven Ġrelation Ġ$ r $ Ġis Ġdefined Ġas Ġ$\ alpha Ġ_ i ( r ) $ Ġ, Ġwhich Ġis </s> $$ e _ i ( r ) Ġ& = Ġ\ math bf Ġ{ v } _ a ^ T Ġ\ tan h Ġ( \ math bf Ġ{ W } _ a Ġ{\ math bf Ġ{ z }} _ i Ġ+ Ġ\ math bf Ġ{ U } _ a Ġ\ math bf Ġ{ r } ), Ġ\\ Ġ\ alpha Ġ_ i ( r ) & = \ oper at orn ame {\ math bf Ġ{ soft max }} ( e _ i ( r )) \ non umber Ġ\\ Ġ& = \ frac {\ exp Ġ( e _ i ( r )) }{ \ sum Ġ^ { n } _{ j = 1 } Ġ\ exp Ġ( e _ j ( r )) }, $$ Ġ Ġ Ġ( E q . Ġ12 ) </s> where Ġ$\ math bf Ġ{ r }\ in Ġ\ math bb Ġ{ R } ^ d $ Ġis Ġthe Ġrelation Ġembed ding ; Ġ${ \ math bf Ġ{ z }} _ i Ġ\ in Ġ\ math bb Ġ{ R } ^ d $ Ġis Ġthe Ġoutput Ġof ĠBL ST M Ġat Ġposition Ġ$ i $ Ġ; Ġ$\ math bf Ġ{ W } _ a , \ math bf Ġ{ U } _ a Ġ\ in Ġ\ math bb Ġ{ R } ^{ d \ times Ġd } $ Ġare Ġparameters Ġmat rices ; Ġ$\ math bf Ġ{ v } _ a Ġ\ in Ġ\ math bb Ġ{ R } ^{ d } $ Ġis Ġa Ġparameter Ġvector . </s> The Ġattention Ġ$\ alpha Ġ_ i ( r ) $ Ġis Ġinterpreted Ġas Ġthe Ġdegree Ġto Ġwhich Ġthe Ġnetwork Ġattends Ġto Ġpartial Ġrepresentation Ġ$\ math bf Ġ{ z } _{ i } $ Ġfor Ġgiven Ġrelation Ġ$ r $ Ġ. </s> The Ġcontextual Ġencoding Ġof Ġtext Ġdescription Ġcan Ġbe Ġformed Ġby Ġa Ġweighted Ġsum Ġof Ġthe Ġencoding Ġ$\ math bf Ġ{ z } _{ i } $ Ġwith Ġattention . </s> $$ \ math bf Ġ{ enc _ 3 } ( x _{ 1 : n }; r ) Ġ& = Ġ\ sum Ġ_ { i = 1 } ^{ n } Ġ\ alpha Ġ_ i ( r ) Ġ* Ġ\ math bf Ġ{ z } _ i . $$ Ġ Ġ Ġ( E q . Ġ13 ) </s> J oint ĠStructure Ġand ĠText ĠEnc oder </s> Since Ġboth Ġthe Ġstructure Ġand Ġtext Ġdescription Ġprovide Ġvaluable Ġinformation Ġfor Ġan Ġentity Ġ, Ġwe Ġwish Ġto Ġintegrate Ġall Ġthese Ġinformation Ġinto Ġa Ġjoint Ġrepresentation . </s> We Ġpropose Ġa Ġunited Ġmodel Ġto Ġlearn Ġa Ġjoint Ġrepresentation Ġof Ġboth Ġstructure Ġand Ġtext Ġinformation . ĠThe Ġwhole Ġmodel Ġcan Ġbe Ġend - to - end Ġtrained . </s> For Ġan Ġentity Ġ$ e $ Ġ, Ġwe Ġdenote Ġ$\ math bf Ġ{ e } _ s $ Ġto Ġbe Ġits Ġembed ding Ġof Ġstructure Ġinformation , Ġ$\ math bf Ġ{ e } _ d $ Ġto Ġbe Ġencoding Ġof Ġits Ġtext Ġdescriptions . ĠThe Ġmain Ġconcern Ġis Ġhow Ġto Ġcombine Ġ$\ math bf Ġ{ e } _ s $ Ġand Ġ$\ math bf Ġ{ e } _ d $ Ġ. </s> To Ġintegrate Ġtwo Ġkinds Ġof Ġrepresentations Ġof Ġentities , Ġwe Ġuse Ġg ating Ġmechanism Ġto Ġdecide Ġhow Ġmuch Ġthe Ġjoint Ġrepresentation Ġdepends Ġon Ġstructure Ġor Ġtext . </s> The Ġjoint Ġrepresentation Ġ$\ math bf Ġ{ e } $ Ġis Ġa Ġlinear Ġinterpol ation Ġbetween Ġthe Ġ$\ math bf Ġ{ e } _ s $ Ġand Ġ$\ math bf Ġ{ e } _ d $ Ġ. </s> $$ \ math bf Ġ{ e } = Ġ\ text bf Ġ{ g } _ e Ġ\ od ot Ġ\ math bf Ġ{ e } _ s Ġ+ Ġ( 1 - \ text bf Ġ{ g } _ e )\ od ot Ġ\ math bf Ġ{ e } _ d , $$ Ġ Ġ Ġ( E q . Ġ14 ) </s> where Ġ$\ text bf Ġ{ g } _ e $ Ġis Ġa Ġgate Ġto Ġbalance Ġtwo Ġsources Ġinformation Ġand Ġits Ġelements Ġare Ġin Ġ$ [ 0 , 1 ] $ Ġ, Ġand Ġ$\ od ot Ġ$ Ġis Ġan Ġelement - wise Ġmultiplication . ĠInt uitive ly , Ġwhen Ġthe Ġgate Ġis Ġclose Ġto Ġ0 , Ġthe Ġjoint Ġrepresentation Ġis Ġforced Ġto Ġignore Ġthe Ġstructure Ġinformation Ġand Ġis Ġthe Ġtext Ġrepresentation Ġonly . </s> Training </s> We Ġuse Ġthe Ġcontrast ive Ġmax - margin Ġcriterion ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġto Ġtrain Ġour Ġmodel . ĠInt uitive ly , Ġthe Ġmax - margin Ġcriterion Ġprovides Ġan Ġalternative Ġto Ġprob abil istic , Ġlikelihood - based Ġestimation Ġmethods Ġby Ġconcentrating Ġdirectly Ġon Ġthe Ġrobust ness Ġof Ġthe Ġdecision Ġboundary Ġof Ġa Ġmodel ĠB IB REF 23 Ġ. ĠThe Ġmain Ġidea Ġis Ġthat Ġeach Ġtriple t Ġ$( h , r , t ) $ Ġcoming Ġfrom Ġthe Ġtraining Ġcorpus Ġshould Ġreceives Ġa Ġhigher Ġscore Ġthan Ġa Ġtriple t Ġin Ġwhich Ġone Ġof Ġthe Ġelements Ġis Ġreplaced Ġwith Ġa Ġrandom Ġelements . </s> We Ġassume Ġthat Ġthere Ġare Ġ$ n _ t $ Ġtriple ts Ġin Ġtraining Ġset Ġand Ġdenote Ġthe Ġ$ i $ Ġth Ġtriple t Ġby Ġ$( h _ i , Ġr _ i , Ġt _ i ), ( i Ġ= Ġ1 , Ġ2 , Ġ\ cd ots Ġ, n _ t ) $ Ġ. ĠEach Ġtriple t Ġhas Ġa Ġlabel Ġ$ y _ i $ Ġto Ġindicate Ġthe Ġtriple t Ġis Ġpositive Ġ( Ġ$ y _ i Ġ= Ġ1 $ Ġ) Ġor Ġnegative Ġ( Ġ$ y _ i Ġ= Ġ0 $ Ġ). </s> Then Ġthe Ġgolden Ġand Ġnegative Ġtriple ts Ġare Ġden oted Ġby Ġ$\ math cal Ġ{ D } Ġ= Ġ\ l brace Ġ( h _ j , Ġr _ j , Ġt _ j ) Ġ| Ġy _ j Ġ= Ġ1 \ r brace Ġ$ Ġand Ġ$\ math cal Ġ{\ hat { D }} Ġ= Ġ\ l brace Ġ( h _ j , Ġr _ j , Ġt _ j ) Ġ| Ġy _ j Ġ= Ġ0 \ r brace Ġ$ Ġ, Ġrespectively . ĠThe Ġpositive Ġexample Ġare Ġthe Ġtriple ts Ġfrom Ġtraining Ġdataset , Ġand Ġthe Ġnegative Ġexamples Ġare Ġgenerated Ġas Ġfollows : Ġ$ Ġ\ math cal Ġ{\ hat { D }} Ġ= Ġ\ l brace Ġ( h _ l , Ġr _ k , Ġt _ k ) Ġ| Ġh _ l Ġ\ ne Ġh _ k Ġ\ w edge Ġy _ k Ġ= Ġ1 \ r brace Ġ\ cup Ġ\ l brace Ġ( h _ k , Ġr _ k , Ġt _ l ) Ġ| Ġt _ l Ġ\ ne Ġt _ k Ġ\ w edge Ġy _ k Ġ= Ġ1 \ r brace Ġ\ cup Ġ\ l brace Ġ( h _ k , Ġr _ l , Ġt _ k ) Ġ| Ġr _ l Ġ\ ne Ġr _ k Ġ\ w edge Ġy _ k Ġ= Ġ1 \ r brace Ġ$ Ġ. ĠThe Ġsampling Ġstrategy Ġis ĠBern ou lli Ġdistribution Ġdescribed Ġin ĠB IB REF 8 Ġ. ĠLet Ġthe Ġset Ġof Ġall Ġparameters Ġbe Ġ$\ The ta Ġ$ Ġ, Ġwe Ġminimize Ġthe Ġfollowing Ġobjective : </s> $$ J (\ The ta Ġ) = \ sum Ġ_ { ( h , r , t ) Ġ\ in Ġ\ math cal Ġ{ D }} \ sum Ġ_ { ( Ġ\ hat { h }, \ hat { r }, \ hat { t }) Ġ\ in Ġ\ math cal Ġ{\ hat { D }}} Ġ\ max Ġ\ left ( 0 , \ gam ma Ġ- Ġ\ right . Ġ\ non umber Ġ\\ Ġf ( Ġh , r , t )+ f (\ hat { h }, \ hat { r }, \ hat { t } )\ left . \ right )+ Ġ\ eta Ġ\ Vert Ġ\ The ta Ġ\ Vert Ġ_ 2 ^ 2 , $$ Ġ Ġ Ġ( E q . Ġ22 ) </s> where Ġ$\ gam ma Ġ> Ġ0 $ Ġis Ġa Ġmargin Ġbetween Ġgolden Ġtriple ts Ġand Ġnegative Ġtriple ts ., Ġ$ f ( h , Ġr , Ġt ) $ Ġis Ġthe Ġscore Ġfunction . ĠWe Ġuse Ġthe Ġstandard Ġ$ L _ 2 $ Ġregular ization Ġof Ġall Ġthe Ġparameters , Ġweighted Ġby Ġthe Ġhyper param eter Ġ$\ eta Ġ$ Ġ. </s> Exper iment </s> In Ġthis Ġsection , Ġwe Ġstudy Ġthe Ġempirical Ġperformance Ġof Ġour Ġproposed Ġmodels Ġon Ġtwo Ġbenchmark Ġtasks : Ġtriple t Ġclassification Ġand Ġlink Ġprediction . </s> Dat as ets </s> We Ġuse Ġtwo Ġpopular Ġknowledge Ġbases : ĠWord Net ĠB IB REF 0 Ġand ĠFree base ĠB IB REF 1 Ġin Ġthis Ġpaper . ĠSpecifically , Ġwe Ġuse ĠW N 18 Ġ( a Ġsubset Ġof ĠWord Net ) ĠB IB REF 24 Ġand ĠFB 15 K Ġ( a Ġsubset Ġof ĠFree base ) ĠB IB REF 2 Ġsince Ġtheir Ġtext Ġdescriptions Ġare Ġeasily Ġpublicly Ġavailable . ĠTable Ġ1 Ġlists Ġstatistics Ġof Ġthe Ġtwo Ġdatasets . </s> Link ĠPrediction </s> Link Ġprediction Ġis Ġa Ġsubt ask Ġof Ġknowledge Ġgraph Ġcompletion Ġto Ġcomplete Ġa Ġtriple t Ġ$( h , Ġr , Ġt ) $ Ġwith Ġ$ h $ Ġor Ġ$ t $ Ġmissing , Ġi . e ., Ġpredict Ġ$ t $ Ġgiven Ġ$( h , Ġr ) $ Ġor Ġpredict Ġ$ h $ Ġgiven Ġ$( r , Ġt ) $ Ġ. ĠRather Ġthan Ġrequiring Ġone Ġbest Ġanswer , Ġthis Ġtask Ġemphasizes Ġmore Ġon Ġranking Ġa Ġset Ġof Ġcandidate Ġentities Ġfrom Ġthe Ġknowledge Ġgraph . </s> Similar Ġto ĠB IB REF 2 Ġ, Ġwe Ġuse Ġtwo Ġmeasures Ġas Ġour Ġevaluation Ġmetrics . Ġ( 1 ) ĠMean ĠRank : Ġthe Ġaveraged Ġrank Ġof Ġcorrect Ġentities Ġor Ġrelations ; Ġ( 2 ) ĠHits @ p : Ġthe Ġproportion Ġof Ġvalid Ġentities Ġor Ġrelations Ġranked Ġin Ġtop Ġ$ p $ Ġpredictions . ĠHere , Ġwe Ġset Ġ$ p = 10 $ Ġfor Ġentities Ġand Ġ$ p = 1 $ Ġfor Ġrelations . ĠA Ġlower ĠMean ĠRank Ġand Ġa Ġhigher ĠHits @ p Ġshould Ġbe Ġachieved Ġby Ġa Ġgood Ġembed ding Ġmodel . ĠWe Ġcall Ġthis Ġevaluation Ġsetting ĠâĢ ľ Raw âĢ Ŀ . ĠSince Ġa Ġfalse Ġpredicted Ġtriple t Ġmay Ġalso Ġexist Ġin Ġknowledge Ġgraphs , Ġit Ġshould Ġbe Ġregard Ġas Ġa Ġvalid Ġtriple t . ĠHence , Ġwe Ġshould Ġremove Ġthe Ġfalse Ġpredicted Ġtriple ts Ġincluded Ġin Ġtraining , Ġvalidation Ġand Ġtest Ġsets Ġbefore Ġranking Ġ( except Ġthe Ġtest Ġtriple t Ġof Ġinterest ). ĠWe Ġcall Ġthis Ġevaluation Ġsetting ĠâĢ ľ Filter âĢ Ŀ . ĠThe Ġevaluation Ġresults Ġare Ġreported Ġunder Ġthese Ġtwo Ġsettings . </s> We Ġselect Ġthe Ġmargin Ġ$\ gam ma Ġ$ Ġamong Ġ$\ l brace Ġ1 , Ġ2 \ r brace Ġ$ Ġ, Ġthe Ġembed ding Ġdimension Ġ$ d $ Ġamong Ġ$\ l brace Ġ20 , Ġ50 , Ġ100 \ r brace Ġ$ Ġ, Ġthe Ġregular ization Ġ$\ eta Ġ$ Ġamong Ġ$\ l brace Ġ0 , Ġ1 E { - 5 }, Ġ1 E { - 6 }\ r brace Ġ$ Ġ, Ġtwo Ġlearning Ġrates Ġ$\ lambda Ġ_ s $ Ġand Ġ$\ lambda Ġ_ t $ Ġamong Ġ$\ l brace Ġ0 . 001 , Ġ0 . 01 , Ġ0 . 05 \ r brace Ġ$ Ġto Ġlearn Ġthe Ġparameters Ġof Ġstructure Ġand Ġtext Ġencoding . ĠThe Ġdiss imilar ity Ġmeasure Ġis Ġset Ġto Ġeither Ġ$ L _ 1 $ Ġor Ġ$\ l brace Ġ1 , Ġ2 \ r brace Ġ$ 0 Ġdistance . </s> In Ġorder Ġto Ġspeed Ġup Ġthe Ġconvergence Ġand Ġavoid Ġover fitting , Ġwe Ġinitiate Ġthe Ġstructure Ġembed d ings Ġof Ġentity Ġand Ġrelation Ġwith Ġthe Ġresults Ġof ĠTrans E . ĠThe Ġembed ding Ġof Ġa Ġword Ġis Ġinitialized Ġby Ġaveraging Ġthe Ġlinked Ġentity Ġembed d ings Ġwhose Ġdescription Ġinclude Ġthis Ġword . ĠThe Ġrest Ġparameters Ġare Ġinitialized Ġby Ġrandomly Ġsampling Ġfrom Ġuniform Ġdistribution Ġin Ġ$ [ - 0 . 1 , Ġ0 . 1 ] $ Ġ. </s> The Ġfinal Ġoptimal Ġconfigurations Ġare : Ġ$\ gam ma Ġ= Ġ2 $ Ġ, Ġ$ d = 20 $ Ġ, Ġ$\ eta Ġ= 1 E { - 5 } $ Ġ, Ġ$\ lambda Ġ_ s Ġ= Ġ0 . 01 $ Ġ, Ġ$\ lambda Ġ_ t Ġ= Ġ0 . 1 $ Ġ, Ġand Ġ$ L _ 1 $ Ġdistance Ġon ĠW N 18 ; Ġ$\ gam ma Ġ= 2 $ Ġ, Ġ$ d = 100 $ Ġ, Ġ$\ eta Ġ= 1 E { - 5 } $ Ġ, Ġ$\ lambda Ġ_ s Ġ= Ġ0 . 01 $ Ġ, Ġ$ d = 20 $ 0 Ġ, Ġand Ġ$ d = 20 $ 1 Ġdistance Ġon ĠFB 15 K . </s> Exper imental Ġresults Ġon Ġboth ĠW N 18 Ġand ĠFB 15 k Ġare Ġshown Ġin ĠTable Ġ2 Ġ, Ġwhere Ġwe Ġuse ĠâĢ ľ J oint ly ( CB OW ) âĢ Ŀ , ĠâĢ ľ J oint ly ( L ST M ) âĢ Ŀ Ġand ĠâĢ ľ J oint ly ( A - L ST M ) âĢ Ŀ Ġto Ġrepresent Ġour Ġjointly Ġencoding Ġmodels Ġwith ĠCB OW , ĠL ST M Ġand Ġattentive ĠL ST M Ġtext Ġenc od ers . ĠOur Ġbaseline Ġis ĠTrans E Ġsince Ġthat Ġthe Ġscore Ġfunction Ġof Ġour Ġmodels Ġis Ġbased Ġon ĠTrans E . </s> From Ġthe Ġresults , Ġwe Ġobserve Ġthat Ġproposed Ġmodels Ġsurpass Ġthe Ġbaseline , ĠTrans E , Ġon Ġall Ġmetrics , Ġwhich Ġindicates Ġthat Ġknowledge Ġrepresentation Ġcan Ġbenefit Ġgreatly Ġfrom Ġtext Ġdescription . </s> On ĠW N 18 , Ġthe Ġreason Ġwhy ĠâĢ ľ J oint ly ( A - L ST M ) âĢ Ŀ Ġis Ġslightly Ġworse Ġthan ĠâĢ ľ J oint ly ( L ST M ) âĢ Ŀ Ġis Ġprobably Ġbecause Ġthe Ġnumber Ġof Ġrelations Ġis Ġlimited . ĠTherefore , Ġthe Ġattention Ġmechanism Ġdoes Ġnot Ġhave Ġobvious Ġadvantage . ĠOn ĠFB 15 K , ĠâĢ ľ J oint ly ( A - L ST M ) âĢ Ŀ Ġachieves Ġthe Ġbest Ġperformance Ġand Ġis Ġsignificantly Ġhigher Ġthan Ġbaseline Ġmethods Ġon Ġmean Ġrank . </s> Although Ġthe ĠHits @ 10 Ġof Ġour Ġmodels Ġare Ġworse Ġthan Ġthe Ġbest Ġstate - of - the - art Ġmethod , ĠTrans D , Ġit Ġis Ġworth Ġnoticing Ġthat Ġthe Ġscore Ġfunction Ġof Ġour Ġmodels Ġis Ġbased Ġon ĠTrans E , Ġnot ĠTrans D . ĠOur Ġmodels Ġare Ġcompatible Ġwith Ġother Ġstate - of - the - art Ġknowledge Ġembed ding Ġmodels . ĠWe Ġbelieve Ġthat Ġour Ġmodel Ġcan Ġbe Ġfurther Ġimproved Ġby Ġadopting Ġthe Ġscore Ġfunctions Ġof Ġother Ġstate - of - the - art Ġmethods , Ġsuch Ġas ĠTrans D . </s> Besides , Ġtextual Ġinformation Ġlargely Ġallev iates Ġthe Ġissue Ġof Ġsp arsity Ġand Ġour Ġmodel Ġachieves Ġsubstantial Ġimprovement Ġon ĠMean ĠRank Ġcomparing Ġwith ĠTrans D . ĠHowever , Ġtextual Ġinformation Ġmay Ġslightly Ġdegrade Ġthe Ġrepresentation Ġof Ġfrequent Ġentities Ġwhich Ġhave Ġbeen Ġwell - trained . ĠThis Ġmay Ġbe Ġanother Ġreason Ġwhy Ġour ĠHits @ 10 Ġis Ġworse Ġthan ĠTrans D Ġwhich Ġonly Ġutilizes Ġstructural Ġinformation . </s> For Ġthe Ġcomparison Ġof ĠHits @ 10 Ġof Ġdifferent Ġkinds Ġof Ġrelations , Ġwe Ġcategorized Ġthe Ġrelationships Ġaccording Ġto Ġthe Ġcardinal ities Ġof Ġtheir Ġhead Ġand Ġtail Ġarguments Ġinto Ġfour Ġclasses : Ġ1 - to - 1 , Ġ1 - to - many , Ġmany - to - 1 , Ġmany - to - many . ĠM apping Ġproperties Ġof Ġrelations Ġfollows Ġthe Ġsame Ġrules Ġin ĠB IB REF 2 Ġ. </s> Table Ġ3 Ġshows Ġthe Ġdetailed Ġresults Ġby Ġmapping Ġproperties Ġof Ġrelations Ġon ĠFB 15 k . ĠWe Ġcan Ġsee Ġthat Ġour Ġmodels Ġoutper form Ġbaseline ĠTrans E Ġin Ġall Ġtypes Ġof Ġrelations Ġ( 1 - to - 1 , Ġ1 - to - N , ĠN - to - 1 Ġand ĠN - to - N ), Ġespecially Ġwhen Ġ( 1 ) Ġpredicting ĠâĢ ľ 1 - to - 1 âĢ Ŀ Ġrelations Ġand Ġ( 2 ) Ġpredicting Ġthe Ġ1 Ġside Ġfor ĠâĢ ľ 1 - to - N âĢ Ŀ Ġand ĠâĢ ľ N - to - 1 âĢ Ŀ Ġrelations . </s> To Ġget Ġmore Ġinsights Ġinto Ġhow Ġthe Ġjoint Ġrepresentation Ġis Ġinfluenced Ġby Ġthe Ġstructure Ġand Ġtext Ġinformation . ĠWe Ġobserve Ġthe Ġactiv ations Ġof Ġgates , Ġwhich Ġcontrol Ġthe Ġbalance Ġbetween Ġtwo Ġsources Ġof Ġinformation , Ġto Ġunderstand Ġthe Ġbehavior Ġof Ġneurons . ĠWe Ġsort Ġthe Ġentities Ġby Ġtheir Ġfrequencies Ġand Ġdivide Ġthem Ġinto Ġ50 Ġequal - size Ġgroups Ġof Ġdifferent Ġfrequencies , Ġand Ġaverage Ġthe Ġvalues Ġof Ġall Ġgates Ġin Ġeach Ġgroup . </s> Figure Ġ3 Ġgives Ġthe Ġaverage Ġof Ġgates Ġin Ġten Ġgroups Ġfrom Ġhigh - Ġto Ġlow - frequency . ĠWe Ġobserve Ġthat Ġthe Ġtext Ġinformation Ġplay Ġmore Ġimportant Ġrole Ġfor Ġthe Ġlow - frequency Ġentities . </s> Tri plet ĠClassification </s> Tri plet Ġclassification Ġis Ġa Ġbinary Ġclassification Ġtask , Ġwhich Ġaims Ġto Ġjudge Ġwhether Ġa Ġgiven Ġtriple t Ġ$( h , Ġr , Ġt ) $ Ġis Ġa Ġcorrect Ġfact Ġor Ġnot . ĠSince Ġour Ġused Ġtest Ġsets Ġ( WN 18 Ġand ĠFB 15 K ) Ġonly Ġcontain Ġcorrect Ġtriple ts , Ġwe Ġconstruct Ġnegative Ġtriple ts Ġfollowing Ġthe Ġsame Ġsetting Ġused Ġin ĠB IB REF 3 Ġ. </s> For Ġtriple ts Ġclassification , Ġwe Ġset Ġa Ġthreshold Ġ$\ d elta Ġ_ r $ Ġfor Ġeach Ġrelation Ġ$ r $ Ġ. Ġ$\ d elta Ġ_ r $ Ġis Ġobtained Ġby Ġmaximizing Ġthe Ġclassification Ġaccur acies Ġon Ġthe Ġvalid Ġset . ĠFor Ġa Ġgiven Ġtriple t Ġ$( h , Ġr , Ġt ) $ Ġ, Ġif Ġits Ġscore Ġis Ġlarger Ġthan Ġ$\ d elta Ġ_ r $ Ġ, Ġit Ġwill Ġbe Ġclassified Ġas Ġpositive , Ġotherwise Ġnegative . </s> Table Ġ4 Ġshows Ġthe Ġevaluation Ġresults Ġof Ġtriple ts Ġclassification . ĠThe Ġresults Ġreveal Ġthat Ġour Ġjoint Ġencoding Ġmodels Ġis Ġeffective Ġand Ġalso Ġoutper form Ġthe Ġbaseline Ġmethod . </s> On ĠW N 18 , ĠâĢ ľ J oint ly ( A - L ST M ) âĢ Ŀ Ġachieves Ġthe Ġbest Ġperformance , Ġand Ġthe ĠâĢ ľ J oint ly ( L ST M ) âĢ Ŀ Ġis Ġslightly Ġworse Ġthan ĠâĢ ľ J oint ly ( A - L ST M ) âĢ Ŀ . ĠThe Ġreason Ġis Ġthat Ġthe Ġnumber Ġof Ġrelations Ġis Ġrelatively Ġsmall . ĠTherefore , Ġthe Ġattention Ġmechanism Ġdoes Ġnot Ġshow Ġobvious Ġadvantage . ĠOn ĠFB 15 K , Ġthe Ġclassification Ġaccuracy Ġof ĠâĢ ľ J oint ly ( A - L ST M ) âĢ Ŀ Ġachieves Ġ91 . 5 %, Ġwhich Ġis Ġthe Ġbest Ġand Ġsignificantly Ġhigher Ġthan Ġthat Ġof Ġstate - of - the - art Ġmethods . </s> Related ĠWork </s> Recently , Ġit Ġhas Ġgained Ġlots Ġof Ġinterests Ġto Ġjointly Ġlearn Ġthe Ġembed d ings Ġof Ġknowledge Ġgraph Ġand Ġtext Ġinformation . ĠThere Ġare Ġseveral Ġmethods Ġusing Ġtextual Ġinformation Ġto Ġhelp ĠK G Ġrepresentation Ġlearning . </s> BI B REF 3 Ġrepresent Ġan Ġentity Ġas Ġthe Ġaverage Ġof Ġits Ġword Ġembed d ings Ġin Ġentity Ġname , Ġallowing Ġthe Ġsharing Ġof Ġtextual Ġinformation Ġlocated Ġin Ġsimilar Ġentity Ġnames . </s> BI B REF 5 Ġjointly Ġembed Ġknowledge Ġand Ġtext Ġinto Ġthe Ġsame Ġspace Ġby Ġalign ing Ġthe Ġentity Ġname Ġand Ġits ĠWikipedia Ġanchor , Ġwhich Ġbrings Ġpromising Ġimprovements Ġto Ġthe Ġaccuracy Ġof Ġpredicting Ġfacts . ĠB IB REF 6 Ġextend Ġthe Ġjoint Ġmodel Ġand Ġalign s Ġknowledge Ġand Ġwords Ġin Ġthe Ġentity Ġdescriptions . ĠHowever , Ġthese Ġtwo Ġworks Ġalign Ġthe Ġtwo Ġkinds Ġof Ġembed d ings Ġon Ġword Ġlevel , Ġwhich Ġcan Ġlose Ġsome Ġsemantic Ġinformation Ġon Ġphrase Ġor Ġsentence Ġlevel . </s> BI B REF 25 Ġalso Ġrepresent Ġentities Ġwith Ġentity Ġnames Ġor Ġthe Ġaverage Ġof Ġword Ġembed d ings Ġin Ġdescriptions . ĠHowever , Ġtheir Ġuse Ġof Ġdescriptions Ġneglect s Ġword Ġorders , Ġand Ġthe Ġuse Ġof Ġentity Ġnames Ġstruggles Ġwith Ġambiguity . ĠB IB REF 7 Ġjointly Ġlearn Ġknowledge Ġgraph Ġembed d ings Ġwith Ġentity Ġdescriptions . ĠThey Ġuse Ġcontinuous Ġbag - of - words Ġand Ġconv olution al Ġneural Ġnetwork Ġto Ġencode Ġsemantics Ġof Ġentity Ġdescriptions . ĠHowever , Ġthey Ġseparate Ġthe Ġobjective Ġfunctions Ġinto Ġtwo Ġenergy Ġfunctions Ġof Ġstructure - based Ġand Ġdescription - based Ġrepresentations . ĠB IB REF 26 Ġembed s Ġboth Ġentity Ġand Ġrelation Ġembed d ings Ġby Ġtaking ĠK G Ġand Ġtext Ġinto Ġconsideration Ġusing ĠCNN . ĠTo Ġutilize Ġboth Ġrepresentations , Ġthey Ġneed Ġfurther Ġestimate Ġan Ġoptimum Ġweight Ġcoefficients Ġto Ġcombine Ġthem Ġtogether Ġin Ġthe Ġspecific Ġtasks . </s> Besides Ġentity Ġrepresentation , Ġthere Ġare Ġalso Ġa Ġlot Ġof Ġworks ĠB IB REF 27 Ġ, ĠB IB REF 28 Ġ, ĠB IB REF 29 Ġto Ġmap Ġtextual Ġrelations Ġand Ġknowledge Ġbase Ġrelations Ġto Ġthe Ġsame Ġvector Ġspace Ġand Ġobtained Ġsubstantial Ġimprovements . </s> While Ġreleasing Ġthe Ġcurrent Ġpaper Ġwe Ġdiscovered Ġa Ġpaper Ġby ĠB IB REF 30 Ġproposing Ġa Ġsimilar Ġmodel Ġwith Ġattention Ġmechanism Ġwhich Ġis Ġevaluated Ġon Ġlink Ġprediction Ġand Ġtriple t Ġclassification . ĠHowever , Ġour Ġwork Ġenc odes Ġtext Ġdescription Ġas Ġa Ġwhole Ġwithout Ġexplicit Ġsegment ation Ġof Ġsentences , Ġwhich Ġbreaks Ġthe Ġorder Ġand Ġco herence Ġamong Ġsentences . </s> Conclusion </s> We Ġpropose Ġa Ġunited Ġrepresentation Ġfor Ġknowledge Ġgraph , Ġutilizing Ġboth Ġstructure Ġand Ġtext Ġdescription Ġinformation Ġof Ġthe Ġentities . ĠExper iments Ġshow Ġthat Ġour Ġproposed Ġjointly Ġrepresentation Ġlearning Ġwith Ġg ating Ġmechanism Ġis Ġeffective , Ġwhich Ġbenefits Ġto Ġmodeling Ġthe Ġmeaning Ġof Ġan Ġentity . </s> In Ġthe Ġfuture , Ġwe Ġwill Ġconsider Ġthe Ġfollowing Ġresearch Ġdirections Ġto Ġimprove Ġour Ġmodel :