No
BOOLEAN
No
BOOLEAN
While it is obvious that our embeddings can be used as features for new predictive models, it is also very easy to incorporate our learned Dolores embeddings into existing predictive models on knowledge graphs. The only requirement is that the model accepts as input, an embedding layer (for entities and relations). If a model fulfills this requirement (which a large number of neural models on knowledge graphs do), we can just use Dolores embeddings as a drop-in replacement. We just initialize the corresponding embedding layer with Dolores embeddings. In our evaluation below, we show how to improve several state-of-the-art models on various tasks simply by incorporating Dolores as a drop-in replacement to the original embedding layer.
While it is obvious that our embeddings can be used as features for new predictive models, it is also very easy to incorporate our learned Dolores embeddings into existing predictive models on knowledge graphs. The only requirement is that the model accepts as input, an embedding layer (for entities and relations). If a model fulfills this requirement (which a large number of neural models on knowledge graphs do), we can just use Dolores embeddings as a drop-in replacement. We just initialize the corresponding embedding layer with Dolores embeddings. In our evaluation below, we show how to improve several state-of-the-art models on various tasks simply by incorporating Dolores as a drop-in replacement to the original embedding layer.
1811.00147
Introduction </s> Know ledge Ġgraphs ĠB IB REF 0 Ġenable Ġstructured Ġaccess Ġto Ġworld Ġknowledge Ġand Ġform Ġa Ġkey Ġcomponent Ġof Ġseveral Ġapplications Ġlike Ġsearch Ġengines , Ġquestion Ġanswering Ġsystems Ġand Ġconvers ational Ġassistants . ĠKnowledge Ġgraphs Ġare Ġtypically Ġinterpreted Ġas Ġcomprising Ġof Ġdiscrete Ġtri ples Ġof Ġthe Ġform Ġ( entity A , Ġrelation X , Ġentity B ) Ġthus Ġrepresenting Ġa Ġrelation Ġ( relation X ) Ġbetween Ġentity A Ġand Ġentity B . ĠHowever , Ġone Ġlimitation Ġof Ġonly Ġa Ġdiscrete Ġrepresentation Ġof Ġtri ples Ġis Ġthat Ġit Ġdoes Ġnot Ġeasily Ġenable Ġone Ġto Ġinfer Ġsimilarities Ġand Ġpotential Ġrelations Ġamong Ġentities Ġwhich Ġmay Ġbe Ġmissing Ġin Ġthe Ġknowledge Ġgraph . ĠConsequently , Ġone Ġpopular Ġalternative Ġis Ġto Ġlearn Ġdense Ġcontinuous Ġrepresentations Ġof Ġentities Ġand Ġrelations Ġby Ġembed ding Ġthem Ġin Ġlatent Ġcontinuous Ġvector Ġspaces , Ġwhile Ġseeking Ġto Ġmodel Ġthe Ġinherent Ġstructure Ġof Ġthe Ġknowledge Ġgraph . ĠMost Ġknowledge Ġgraph Ġembed ding Ġmethods Ġcan Ġbe Ġclassified Ġinto Ġtwo Ġmajor Ġclasses : Ġone Ġclass Ġwhich Ġoperates Ġpurely Ġon Ġtri ples Ġlike ĠRES C AL ĠB IB REF 1 Ġ, ĠTrans E ĠB IB REF 2 Ġ, ĠDist Mult ĠB IB REF 3 Ġ, ĠTrans D ĠB IB REF 4 Ġ, ĠCompl Ex ĠB IB REF 5 Ġ, ĠConv E ĠB IB REF 6 Ġand Ġthe Ġsecond Ġclass Ġwhich Ġseeks Ġto Ġincorporate Ġadditional Ġinformation Ġ( like Ġmulti - hops ) ĠB IB REF 7 Ġ. ĠLearning Ġhigh - quality Ġknowledge Ġgraph Ġembed d ings Ġcan Ġbe Ġquite Ġchallenging Ġgiven Ġthat Ġ( a ) Ġthey Ġneed Ġto Ġeffectively Ġmodel Ġthe Ġcontextual Ġus ages Ġof Ġentities Ġand Ġrelations Ġ( b ) Ġthey Ġwould Ġneed Ġto Ġbe Ġuseful Ġfor Ġa Ġvariety Ġof Ġpredictive Ġtasks Ġon Ġknowledge Ġgraphs . </s> In Ġthis Ġpaper , Ġwe Ġpresent Ġa Ġnew Ġtype Ġof Ġknowledge Ġgraph Ġembed d ings Ġcalled ĠDol ores Ġthat Ġare Ġboth Ġdeep Ġand Ġcontextual ized . ĠDol ores Ġlearns Ġboth Ġcontext - independent Ġand Ġcontext - dependent Ġembed d ings Ġof Ġentities Ġand Ġrelations Ġthrough Ġa Ġdeep Ġneural Ġsequential Ġmodel . ĠFigure Ġ1 Ġillustrates Ġthe Ġdeep Ġcontextual ized Ġrepresentations Ġlearned . ĠNote Ġthat Ġthe Ġcontext ually Ġindependent Ġentity Ġembed d ings Ġ( see ĠFigure Ġ1 Ġ) Ġreveal Ġthree Ġclusters Ġof Ġentities : Ġwriters , Ġphilosophers , Ġand Ġmusicians . ĠThe Ġcontextual Ġdependent Ġembed d ings Ġin Ġturn Ġeffectively Ġaccount Ġfor Ġspecific Ġrelations . ĠIn Ġparticular , Ġthe Ġcontext - dependent Ġrepresentations Ġunder Ġthe Ġrelation Ġnationality Ġnow Ġnicely Ġcluster Ġthe Ġabove Ġentities Ġby Ġnationality Ġnamely ĠAust rians , ĠGermans , Ġand ĠBritish / Irish . ĠSimilarly ĠFigure Ġ1 Ġshows Ġcontextual Ġembed d ings Ġgiven Ġthe Ġrelation Ġplace - lived . ĠNote Ġthat Ġthese Ġembed d ings Ġcorrectly Ġcapture Ġthat Ġeven Ġthough ĠBe eth oven Ġand ĠBrah ms Ġbeing ĠGermans , Ġthey Ġlived Ġin ĠVienna Ġand Ġare Ġcloser Ġto Ġother ĠAustrian Ġmusicians Ġlike ĠSch u bert . </s> Unlike Ġmost Ġknowledge Ġgraph Ġembed d ings Ġlike ĠTrans D , ĠTrans E ĠB IB REF 2 Ġ, ĠB IB REF 4 Ġetc . Ġwhich Ġare Ġtypically Ġlearned Ġusing Ġshallow Ġmodels , Ġthe Ġrepresentations Ġlearned Ġby ĠDol ores Ġare Ġdeep : Ġdependent Ġon Ġan Ġentire Ġpath Ġ( rather Ġthan Ġjust Ġa Ġtriple ), Ġare Ġfunctions Ġof Ġinternal Ġstates Ġof Ġa ĠBi - D irection al ĠL ST M Ġand Ġcomposed Ġof Ġrepresentations Ġlearned Ġat Ġvarious Ġlayers Ġpotentially Ġcapturing Ġvarying Ġdegrees Ġof Ġabstract ions . ĠDol ores Ġis Ġinspired Ġby Ġrecent Ġadvances Ġin Ġlearning Ġword Ġrepresentations Ġ( word Ġembed d ings ) Ġfrom Ġdeep Ġneural Ġlanguage Ġmodels Ġusing ĠBi - D irection al ĠL ST Ms ĠB IB REF 8 Ġ. ĠIn Ġparticular , Ġwe Ġderive Ġconnections Ġbetween Ġthe Ġwork Ġof ĠPeters Ġet Ġal . Ġ( ĠB IB REF 8 Ġ) Ġwho Ġlearn Ġdeep Ġcontextual ized Ġword Ġembed d ings Ġfrom Ġsentences Ġusing Ġa ĠBi - D irection al ĠL ST M Ġbased Ġlanguage Ġmodel Ġand Ġrandom Ġwalks Ġon Ġknowledge Ġgraphs . ĠThese Ġconnections Ġenable Ġus Ġto Ġpropose Ġnew ĠâĢ ľ deep Ġcontextual ized âĢ Ŀ Ġknowledge Ġgraph Ġembed d ings Ġwhich Ġwe Ġcall ĠDol ores Ġembed d ings . </s> Know ledge ĠEmb edd ings Ġlearned Ġusing ĠDol ores Ġcan Ġeasily Ġbe Ġused Ġas Ġinput Ġrepresentations Ġfor Ġpredictive Ġmodels Ġon Ġknowledge Ġgraphs . ĠMore Ġimportantly , Ġwhen Ġexisting Ġpredictive Ġmodels Ġuse Ġinput Ġrepresentations Ġfor Ġentities Ġand Ġrelations , Ġwe Ġcan Ġeasily Ġreplace Ġthose Ġrepresentations Ġwith ĠDol ores Ġrepresentations Ġand Ġsignificantly Ġimprove Ġthe Ġperformance Ġof Ġexisting Ġmodels . ĠSpecifically , Ġwe Ġshow Ġthat ĠDol ores Ġembed d ings Ġadvance Ġthe Ġstate - of - the - art Ġmodels Ġon Ġvarious Ġtasks Ġlike Ġlink Ġprediction , Ġtriple Ġclassification Ġand Ġmissing Ġrelation Ġtype Ġprediction . </s> To Ġsummarize , Ġour Ġcontributions Ġare Ġas Ġfollows : </s> Related ĠWork </s> Ext ensive Ġwork Ġexists Ġon Ġknowledge Ġgraph Ġembed d ings Ġdating Ġback Ġto ĠNickel , ĠT resp , Ġand ĠK rieg el Ġ( ĠB IB REF 1 Ġ) Ġwho Ġfirst Ġproposed ĠRes cal Ġbased Ġon Ġa Ġmatrix Ġfactor ization Ġapproach . ĠB ordes Ġet Ġal . Ġ( ĠB IB REF 2 Ġ) Ġadvanced Ġthis Ġline Ġof Ġwork Ġby Ġproposing Ġthe Ġfirst Ġtransl ational Ġmodel ĠTrans E Ġwhich Ġseeks Ġto Ġrelate Ġthe Ġhead Ġand Ġtail Ġentity Ġembed d ings Ġby Ġmodeling Ġthe Ġrelation Ġas Ġa Ġtransl ational Ġvector . ĠThis Ġculminated Ġin Ġa Ġlong Ġseries Ġof Ġnew Ġknowledge Ġgraph Ġembed d ings Ġall Ġbased Ġon Ġthe Ġtransl ational Ġprinciple Ġwith Ġvarious Ġrefin ements ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġ. ĠSome Ġrecently Ġproposed Ġmodels Ġlike ĠMan i F old E ĠB IB REF 17 Ġattempt Ġto Ġlearn Ġknowledge Ġgraph Ġembed d ings Ġas Ġa Ġmanifold Ġwhile Ġembed d ings Ġlike ĠHol E ĠB IB REF 1 Ġderive Ġinspiration Ġfrom Ġassoci ative Ġmemories . ĠFurthermore , Ġwith Ġthe Ġsuccess Ġof Ġneural Ġmodels , Ġmodels Ġbased Ġon Ġconv olution al Ġneural Ġnetworks Ġhave Ġbeen Ġproposed Ġlike ĠB IB REF 6 Ġ, ĠB IB REF 18 Ġto Ġlearn Ġknowledge Ġgraph Ġembed d ings . ĠOther Ġmodels Ġin Ġthis Ġclass Ġof Ġmodels Ġinclude ĠConv KB ĠB IB REF 19 Ġand ĠKB GAN ĠB IB REF 20 Ġ. ĠThere Ġhas Ġbeen Ġsome Ġwork Ġon Ġincorporating Ġadditional Ġinformation Ġlike Ġentity Ġtypes , Ġrelation Ġpaths Ġetc . Ġto Ġlearn Ġknowledge Ġgraph Ġrepresentations . ĠPal umbo Ġet Ġal . Ġ( ĠB IB REF 21 Ġ) Ġuse Ġnode 2 vec Ġto Ġlearn Ġembed d ings Ġof Ġentities Ġand Ġitems Ġin Ġa Ġknowledge Ġgraph . ĠA Ġnotable Ġclass Ġof Ġmethods Ġcalled ĠâĢ ľ path - ranking âĢ Ŀ Ġbased Ġmodels Ġdirectly Ġmodel Ġpaths Ġbetween Ġentities Ġas Ġfeatures . ĠExamples Ġinclude ĠPath ĠRanking ĠAl gorithm Ġ( P RA ) ĠB IB REF 22 Ġ, ĠP Trans E ĠB IB REF 10 Ġand Ġmodels Ġbased Ġon Ġrecurrent Ġneural Ġnetworks ĠB IB REF 23 Ġ, ĠB IB REF 24 Ġ. ĠBesides , ĠDas Ġet Ġal . Ġ( ĠB IB REF 25 Ġ) Ġpropose Ġa Ġreinforcement Ġlearning Ġmethod Ġthat Ġaddresses Ġpractical Ġtask Ġof Ġanswering Ġquestions Ġwhere Ġthe Ġrelation Ġis Ġknown , Ġbut Ġonly Ġone Ġentity . ĠHartford Ġet Ġal . Ġ( ĠB IB REF 26 Ġ) Ġmodel Ġinteractions Ġacross Ġtwo Ġor Ġmore Ġsets Ġof Ġobjects Ġusing Ġa Ġparameter - sharing Ġscheme . ĠWhile Ġmost Ġof Ġthe Ġabove Ġmodels Ġexcept Ġfor Ġthe Ġrecurrent - ne ural Ġnet Ġab ased Ġmodels Ġabove Ġare Ġshallow Ġour Ġmodel ĠDol ores Ġdiffers Ġfrom Ġall Ġof Ġthese Ġworks Ġand Ġespecially Ġthat Ġof ĠPal umbo Ġet Ġal . Ġ( ĠB IB REF 21 Ġ) Ġin Ġthat Ġwe Ġlearn Ġdeep Ġcontextual ized Ġknowledge Ġgraph Ġrepresentations Ġof Ġentities Ġand Ġrelations Ġusing Ġa Ġdeep Ġneural Ġsequential Ġmodel . ĠThe Ġwork Ġthat Ġis Ġclosest Ġto Ġour Ġwork Ġis Ġthat Ġof ĠDas Ġet Ġal . Ġ( ĠB IB REF 24 Ġ) Ġwho Ġdirectly Ġuse Ġan ĠR NN - based Ġarchitecture Ġto Ġmodel Ġpaths Ġto Ġpredict Ġmissing Ġlinks . ĠWe Ġdistinguish Ġour Ġwork Ġfrom Ġthis Ġin Ġthe Ġfollowing Ġkey Ġways : Ġ( a ) ĠFirst , Ġunlike ĠDas Ġet Ġal . Ġ( ĠB IB REF 24 Ġ), Ġour Ġfocus Ġis Ġnot Ġon Ġpath Ġreasoning Ġbut Ġon Ġlearning Ġrich Ġknowledge Ġgraph Ġembed d ings Ġuseful Ġfor Ġa Ġvariety Ġof Ġpredictive Ġtasks . ĠMoreover Ġwhile ĠDas Ġet Ġal . Ġ( ĠB IB REF 24 Ġ) Ġneed Ġto Ġuse Ġpaths Ġgenerated Ġfrom ĠP RA Ġthat Ġtypically Ġcorrelate Ġwith Ġrelations , Ġour Ġmethod Ġhas Ġno Ġsuch Ġrestriction Ġand Ġonly Ġuses Ġpaths Ġgenerated Ġby Ġgeneric Ġrandom Ġwalks Ġgreatly Ġenhancing Ġthe Ġscal ability Ġof Ġour Ġmethod . ĠIn Ġfact , Ġwe Ġincorporate ĠDol ores Ġembed d ings Ġto Ġimprove Ġthe Ġperformance Ġof Ġthe Ġmodel Ġproposed Ġby ĠDas Ġet Ġal . Ġ( ĠB IB REF 24 Ġ). Ġ( b ) ĠSecond , Ġand Ġmost Ġimportantly Ġwe Ġlearn Ġknowledge Ġgraph Ġembed d ings Ġat Ġmultiple Ġlayers Ġeach Ġpotentially Ġcapturing Ġdifferent Ġlevels Ġof Ġabstraction . Ġ( c ) ĠFinally , Ġwhile Ġwe Ġare Ġinspired Ġby Ġthe Ġwork Ġof ĠPeters Ġet Ġal . Ġ( ĠB IB REF 8 Ġ) Ġin Ġlearning Ġdeep Ġword Ġrepresentations , Ġwe Ġbuild Ġon Ġtheir Ġideas Ġby Ġdrawing Ġconnections Ġbetween Ġknowledge Ġgraphs Ġand Ġlanguage Ġmodeling ĠB IB REF 8 Ġ. ĠIn Ġparticular , Ġwe Ġpropose Ġmethods Ġto Ġuse Ġrandom Ġwalks Ġover Ġknowledge Ġgraphs Ġin Ġconjunction Ġwith Ġthe Ġmachinery Ġof Ġdeep Ġneural Ġlanguage Ġmodeling Ġto Ġlearn Ġpowerful Ġdeep Ġcontextual ized Ġknowledge Ġgraph Ġembed d ings Ġthat Ġimprove Ġthe Ġstate Ġof Ġthe Ġart Ġon Ġvarious Ġknowledge Ġgraph Ġtasks . </s> Problem ĠForm ulation </s> Given Ġa Ġknowledge Ġgraph Ġ$ G =( E , ĠR ) $ Ġwhere Ġ$ E $ Ġdenotes Ġthe Ġset Ġof Ġentities Ġand Ġ$ R $ Ġdenotes Ġthe Ġset Ġof Ġrelations Ġamong Ġthose Ġentities , Ġwe Ġseek Ġto Ġlearn Ġ$ d $ Ġ- dimensional Ġembed d ings Ġof Ġthe Ġentities Ġand Ġrelations . ĠIn Ġcontrast Ġto Ġprevious Ġknowledge Ġgraph Ġembed ding Ġmethods Ġlike ĠB IB REF 2 Ġ, ĠB IB REF 9 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 5 Ġwhich Ġare Ġbased Ġon Ġshallow Ġmodels Ġand Ġoperates Ġprimarily Ġon Ġtri ples , Ġour Ġmethod ĠDol ores Ġuses Ġa Ġdeep Ġneural Ġmodel Ġto Ġlearn ĠâĢ ľ deep âĢ Ŀ Ġand ĠâĢ ľ context ual ized âĢ Ŀ Ġknowledge Ġgraph Ġembed d ings . </s> Having Ġformulated Ġthe Ġproblem , Ġwe Ġnow Ġdescribe ĠDol ores . ĠDol ores Ġconsists Ġof Ġtwo Ġmain Ġcomponents : </s> Path ĠGenerator ĠThis Ġcomponent Ġis Ġresponsible Ġfor Ġgenerating Ġa Ġlarge Ġset Ġof Ġentity - relation Ġchains Ġthat Ġreflect Ġthe Ġvarying Ġcontextual Ġus ages Ġof Ġentities Ġand Ġrelations Ġin Ġthe Ġknowledge Ġgraph . </s> Lear ner ĠThis Ġcomponent Ġis Ġa Ġdeep Ġneural Ġmodel Ġthat Ġtakes Ġas Ġinput Ġentity - relation Ġchains Ġand Ġlearns Ġentity Ġand Ġrelation Ġembed d ings Ġwhich Ġare Ġweighted Ġlinear Ġcombination Ġof Ġinternal Ġstates Ġof Ġthe Ġmodel Ġthus Ġcapturing Ġcontext Ġdependence . </s> Both Ġof Ġthe Ġabove Ġcomponents Ġare Ġmotivated Ġby Ġrecent Ġadvances Ġin Ġlearning Ġdeep Ġrepresentations Ġof Ġwords Ġin Ġlanguage Ġmodeling . ĠWe Ġmotivate Ġthis Ġbelow Ġand Ġalso Ġhighlight Ġkey Ġconnections Ġthat Ġenable Ġus Ġto Ġbuild Ġon Ġthese Ġadvances Ġto Ġlearn ĠDol ores Ġknowledge Ġgraph Ġembed d ings . </s> P rel im in aries </s> Rec all Ġthat Ġthe Ġgoal Ġof Ġa Ġlanguage Ġmodel Ġis Ġto Ġestimate Ġthe Ġlikelihood Ġof Ġa Ġsequence Ġof Ġwords : Ġ$ w _{ 1 }, Ġw _{ 2 }, Ġ\ cd ots Ġ, w _{ n } $ Ġwhere Ġeach Ġword Ġ$ w _{ i } $ Ġis Ġfrom Ġa Ġfinite Ġvocabulary Ġ$\ math cal Ġ{ V } $ Ġ. ĠSpecifically , Ġthe Ġgoal Ġof Ġa Ġforward Ġlanguage Ġmodel Ġis Ġto Ġmodel Ġ$\ text { Pr } ( w _{ i } | w _{ 1 }, Ġw _{ 2 }, Ġ\ cd ots Ġ, w _{ i - 1 }) $ Ġ. ĠWhile , Ġtraditionally Ġthis Ġhas Ġbeen Ġmodeled Ġusing Ġcount - based ĠâĢ ľ n - gram Ġbased " Ġmodels ĠB IB REF 27 Ġ, ĠB IB REF 28 Ġ, Ġrecently Ġdeep Ġneural Ġmodels Ġlike ĠL ST Ms Ġand ĠR NN 's Ġhave Ġbeen Ġused Ġto Ġbuild Ġsuch Ġlanguage Ġmodels . ĠAs Ġnoted Ġby ĠPeters Ġet Ġal . Ġ( ĠB IB REF 8 Ġ), Ġa Ġforward Ġlanguage Ġmodel Ġimplemented Ġusing Ġan ĠL ST M Ġof ĠâĢ ľ L âĢ Ŀ Ġlayers Ġworks Ġas Ġfollows : ĠAt Ġeach Ġposition Ġ$ k $ Ġ, Ġeach ĠL ST M Ġlayer Ġoutputs Ġa Ġcontext - dependent Ġrepresentation Ġden oted Ġby Ġ$\ over right arrow { h _{ k , j }} $ Ġcorresponding Ġto Ġthe Ġ$ j ^{ th } $ Ġlayer Ġof Ġthe ĠL ST M . ĠThe Ġtop - most Ġlayer Ġof Ġthe ĠL ST M Ġis Ġthen Ġfed Ġas Ġinput Ġto Ġa Ġsoft max Ġlayer Ġof Ġsize Ġ$ | \ math cal Ġ{ V } | $ Ġto Ġpredict Ġthe Ġnext Ġtoken . ĠSimilarly , Ġa Ġbackward Ġlanguage Ġmodel Ġwhich Ġmodels Ġ$\ text { Pr } ( w _{ i } | w _{ i + 1 }, Ġw _{ i + 2 }, Ġ\ cd ots Ġ, w _{ n }) $ Ġcan Ġbe Ġimplemented Ġusing Ġa ĠâĢ ľ back ward âĢ Ŀ ĠL ST M Ġproducing Ġsimilar Ġrepresentations . ĠA ĠBi - D irection al ĠL ST M Ġjust Ġcombines Ġboth Ġforward Ġand Ġbackward Ġdirections Ġand Ġseeks Ġto Ġjointly Ġmaximize Ġthe Ġlog - like lihood Ġof Ġthe Ġforward Ġand Ġbackward Ġdirectional Ġlanguage Ġmodel Ġobjectives . </s> We Ġnote Ġthat Ġthese Ġcontext - dependent Ġrepresentations Ġlearned Ġby Ġthe ĠL ST M Ġat Ġeach Ġlayer Ġhave Ġbeen Ġshown Ġto Ġbe Ġuseful Ġas ĠâĢ ľ deep Ġcontextual âĢ Ŀ Ġword Ġrepresentations Ġin Ġvarious Ġpredictive Ġtasks Ġin Ġnatural Ġlanguage Ġprocessing ĠB IB REF 8 Ġ. ĠIn Ġline Ġwith Ġthis Ġtrend , Ġwe Ġwill Ġalso Ġuse Ġdeep Ġneural Ġsequential Ġmodels Ġmore Ġspecifically ĠBi - D irection al ĠL ST Ms Ġto Ġlearn ĠDol ores Ġembed d ings . ĠWe Ġdo Ġthis Ġby Ġgeneral izing Ġthis Ġapproach Ġto Ġgraphs Ġby Ġnoting Ġconnections Ġfirst Ġnoted Ġby ĠPer o zzi , ĠAl - R f ou , Ġand ĠSk ien a Ġ( ĠB IB REF 29 Ġ). </s> Since Ġthe Ġinput Ġto Ġa Ġlanguage Ġmodel Ġis Ġa Ġlarge Ġcorpus Ġor Ġset Ġof Ġsentences , Ġone Ġcan Ġgeneral ize Ġlanguage Ġmodeling Ġapproaches Ġto Ġgraphs Ġby Ġnoting Ġthat Ġthe Ġanalog Ġof Ġa Ġsentence Ġin Ġgraphs Ġis Ġa ĠâĢ ľ random Ġwalk âĢ Ŀ . ĠMore Ġspecifically , Ġnote Ġthat Ġa Ġtrunc ated Ġrandom Ġwalk Ġof Ġlength ĠT Ġstarting Ġfrom Ġa Ġnode ĠâĢ ľ v âĢ Ŀ Ġis Ġanalogous Ġto Ġa Ġsentence Ġand Ġeffectively Ġcaptures Ġthe Ġcontext Ġof ĠâĢ ľ v " Ġin Ġthe Ġnetwork . ĠMore Ġprecisely , Ġthe Ġsame Ġmachinery Ġused Ġto Ġlearn Ġrepresentations Ġof Ġwords Ġin Ġlanguage Ġmodels Ġcan Ġnow Ġbe Ġadapted Ġto Ġlearn Ġdeep Ġcontextual ized Ġrepresentations Ġof Ġknowledge Ġgraphs . </s> This Ġcan Ġeasily Ġbe Ġadapted Ġto Ġknowledge Ġgraphs Ġby Ġconstructing Ġpaths Ġof Ġentities Ġand Ġrelations . ĠIn Ġparticular , Ġa Ġrandom Ġwalk Ġon Ġa Ġknowledge Ġgraph Ġstarting Ġat Ġentity Ġ$ e _{ 1 } $ Ġand Ġending Ġat Ġentity Ġ$ e _{ k } $ Ġis Ġa Ġsequence Ġof Ġthe Ġform Ġ$ e _{ 1 }, Ġr _{ 1 }, Ġe _{ 2 }, Ġr _{ 2 }, Ġ\ cd ots Ġ, e _{ k } $ Ġrepresenting Ġthe Ġentities Ġand Ġthe Ġcorresponding Ġrelations Ġbetween Ġ$ e _{ 1 } $ Ġand Ġ$ e _{ k } $ Ġin Ġthe Ġknowledge Ġgraph . ĠMoving Ġforward Ġwe Ġdenote Ġsuch Ġa Ġpath Ġof Ġentities Ġand Ġrelations Ġby Ġ$\ math bf Ġ{ q } =( e _{ 1 }, r _{ 1 }, Ġe _{ 2 }, Ġr _{ 2 }, Ġ\ cd ots Ġ, e _{ k }) $ Ġ. ĠWe Ġgenerate Ġa Ġlarge Ġset Ġof Ġsuch Ġpaths Ġfrom Ġthe Ġknowledge Ġgraph ĠG Ġby Ġperforming Ġseveral Ġrandom Ġwalks Ġon Ġit Ġwhich Ġin Ġturn Ġyields Ġa Ġcorpus Ġof ĠâĢ ľ sent ences âĢ Ŀ ĠS Ġneeded Ġfor ĠâĢ ľ language Ġmodeling âĢ Ŀ . </s> D ol ores : ĠPath ĠGenerator </s> Having Ġmotivated Ġthe Ġmodel Ġand Ġdiscussed Ġprelim in aries , Ġwe Ġnow Ġdescribe Ġthe Ġfirst Ġcomponent Ġof ĠDol ores ĠâĢĵ Ġthe Ġpath Ġgenerator . </s> Let ĠS Ġdenote Ġthe Ġset Ġof Ġentity - relation Ġchains Ġobtained Ġby Ġdoing Ġrandom Ġwalks Ġin Ġthe Ġknowledge Ġgraph . ĠWe Ġadopt Ġa Ġcomponent Ġof Ġnode 2 vec ĠB IB REF 30 Ġto Ġconstruct ĠS . ĠIn Ġparticular , Ġwe Ġperform Ġa Ġ2 nd Ġorder Ġrandom Ġwalk Ġwith Ġtwo Ġparameters Ġp Ġand Ġq Ġthat Ġdetermine Ġthe Ġdegree Ġof Ġbreadth - first Ġsampling Ġand Ġdepth - first Ġsampling . ĠSpecifically Ġas ĠGro ver Ġand ĠLes k ove c Ġ( ĠB IB REF 30 Ġ) Ġdescribed , Ġp Ġcontrols Ġthe Ġlikelihood Ġof Ġimmediately Ġrevis iting Ġa Ġnode Ġin Ġthe Ġwalk Ġwhereas Ġq Ġcontrols Ġwhether Ġthe Ġwalk Ġis Ġbiased Ġtowards Ġnodes Ġclose Ġto Ġstarting Ġnode Ġor Ġaway Ġfrom Ġstarting Ġnode . ĠWe Ġemphasize Ġthat Ġwhile Ġnode 2 vec Ġhas Ġadditional Ġsteps Ġto Ġlearn Ġdense Ġcontinuous Ġrepresentations Ġof Ġnodes , Ġwe Ġadopt Ġonly Ġits Ġfirst Ġcomponent Ġto Ġgenerate Ġa Ġcorpus Ġof Ġrandom Ġwalks Ġrepresenting Ġpaths Ġin Ġknowledge Ġgraphs . </s> D ol ores : ĠLear ner </s> Having Ġgenerated Ġa Ġset Ġof Ġpaths Ġon Ġknowledge Ġgraphs Ġrepresenting Ġlocal Ġcontexts Ġof Ġentities Ġand Ġrelations , Ġwe Ġare Ġnow Ġready Ġto Ġutilize Ġthe Ġmachinery Ġof Ġlanguage Ġmodeling Ġusing Ġdeep Ġneural Ġnetworks Ġto Ġlearn ĠDol ores Ġembed d ings . </s> While Ġtraditional Ġlanguage Ġmodels Ġmodel Ġa Ġsentence Ġas Ġa Ġsequence Ġof Ġwords , Ġwe Ġadopt Ġthe Ġsame Ġmachinery Ġto Ġmodel Ġknowledge Ġgraph Ġembed d ings Ġas Ġfollows : Ġ( a ) ĠA Ġword Ġis Ġan Ġ( entity , Ġrelation ) Ġtuple , Ġ( b ) Ġwe Ġmodel Ġa Ġsentence Ġas Ġa Ġpath Ġconsisting Ġof Ġ( entity , Ġrelation ) Ġtu ples . ĠNote Ġthat Ġwe Ġhave Ġalready Ġestablished Ġhow Ġto Ġgenerate Ġsuch Ġpaths Ġfrom Ġthe Ġknowledge Ġgraph Ġusing Ġthe Ġpath Ġgenerator Ġcomponent . </s> Given Ġsuch Ġpaths , Ġwe Ġwould Ġlike Ġto Ġmodel Ġthe Ġprobability Ġof Ġan Ġentity - relation Ġpair Ġgiven Ġthe Ġhistory Ġand Ġfuture Ġcontext Ġby Ġa ĠBi - D irection al ĠLong ĠShort - Term ĠMemory Ġnetwork . ĠIn Ġparticular , Ġthe Ġforward Ġdirection ĠL ST M Ġmodels : Ġ$ Ġ\ text { Pr } ([ e _ 1 , r _ 1 ], Ġ[ e _ 2 , r _ 2 ], Ġ\ cd ots Ġ, Ġ[ e _ N , Ġr _ N ]) Ġ= Ġ$ </s> $$ \ pro d Ġ_ { t = 1 } ^{ N } Ġ\ text { Pr } ( Ġ[ e _ t , Ġr _ t ] Ġ\ mid Ġ[ e _ 1 , Ġr _ 1 ], Ġ[ e _ 2 , Ġr _ 2 ], Ġ\ cd ots Ġ, Ġ[ e _{ t - 1 }, Ġr _{ t - 1 } ] Ġ). $$ Ġ Ġ Ġ( E q . Ġ17 ) </s> Similarly , Ġthe Ġbackward Ġdirection ĠL ST M Ġmodels : Ġ$ Ġ\ text { Pr } ([ e _ 1 , r _ 1 ], Ġ[ e _ 2 , r _ 2 ], Ġ\ cd ots Ġ, Ġ[ e _ N , Ġr _ N ]) Ġ= Ġ$ </s> $$ \ pro d Ġ_ { t = 1 } ^{ N } Ġ\ text { Pr } ( Ġ[ e _ t , Ġr _ t ] Ġ\ mid Ġ[ e _{ t + 1 }, Ġr _{ t + 1 } ], Ġ\ cd ots Ġ, Ġ[ e _{ N }, Ġr _{ N } ] Ġ). $$ Ġ Ġ Ġ( E q . Ġ18 ) </s> Figure Ġ2 Ġillustrates Ġthis Ġsuccinct ly . ĠAt Ġeach Ġtime - step Ġt , Ġwe Ġdeal Ġwith Ġan Ġentity - relation Ġpair Ġ[ Ġ$ e _ t $ Ġ, Ġ$ r _ t $ Ġ]. ĠWe Ġfirst Ġmap Ġone - hot Ġvectors Ġof Ġthe Ġ$ e _ t $ Ġand Ġ$ r _ t $ Ġto Ġan Ġembed ding Ġlayer , Ġconc aten ate Ġthem Ġto Ġobtain Ġcontext - independent Ġrepresentations Ġwhich Ġare Ġthen Ġpassed Ġthrough ĠL Ġlayers Ġof Ġa ĠBi - D irection al ĠL ST M . ĠEach Ġlayer Ġof ĠL ST M Ġoutputs Ġthe Ġpair 's Ġcontext - dependent Ġrepresentation Ġ$\ over right arrow { h _{ t , i }} $ Ġ, Ġ$\ over left arrow { h _{ t , i }} $ Ġ, Ġwhere Ġi = 1 , Ġ2 , Ġ$\ cd ots Ġ$ Ġ, ĠL . ĠFinally , Ġthe Ġoutput Ġof Ġthe Ġtop Ġlayer Ġof ĠL ST M , Ġ$\ over right arrow { h _{ t , L }} $ Ġ, Ġ$\ over left arrow { h _{ t , L }} $ Ġ, Ġis Ġused Ġto Ġpredict Ġthe Ġnext Ġpair Ġ[ Ġ$ e _{ t + 1 } $ Ġ, Ġ$ r _ t $ 0 Ġ] Ġand Ġ[ Ġ$ r _ t $ 1 Ġ, Ġ$ r _ t $ 2 Ġ] Ġrespectively Ġusing Ġa Ġsoft max Ġlayer . ĠForm ally , Ġwe Ġjointly Ġmaximize Ġthe Ġlog Ġlikelihood Ġof Ġthe Ġforward Ġand Ġbackward Ġdirections : </s> $$ \ begin { split } Ġ\ sum Ġ_ { t = 1 } ^{ N }\ log Ġ\ text { Pr } ([ e _ t , Ġr _ t ] \ mid Ġ[ e _ 1 , Ġr _ 1 ], \ cd ots Ġ, [ e _{ t - 1 }, Ġr _{ t - 1 } ]; \ math bf Ġ{\ The ta Ġ_ { F }} )+ \\ Ġ\ sum Ġ_ { t = 1 } ^{ N }\ log Ġ\ text { Pr } ([ e _ t , Ġr _ t ] \ mid Ġ[ e _{ t + 1 }, Ġr _{ t + 1 } ], \ cd ots Ġ, [ e _{ N }, Ġr _{ N } ]; \ math bf Ġ{\ The ta Ġ_ { B }} ), Ġ\ end { split } $$ Ġ Ġ Ġ( E q . Ġ19 ) </s> where Ġ$\ math bf Ġ{\ The ta Ġ_ { F }} $ Ġ= Ġ$( \ the ta Ġ_ x , Ġ\ over right arrow {\ the ta Ġ_ { L ST M } }, Ġ\ the ta Ġ_ s ) $ Ġcorresponds Ġto Ġthe Ġparameters Ġof Ġthe Ġembed ding Ġlayer , Ġforward - direction ĠL ST M Ġand Ġthe Ġsoft max Ġlayer Ġrespectively . ĠSimilarly Ġ$\ math bf Ġ{\ The ta Ġ_ { B }} $ Ġ= Ġ$( \ the ta Ġ_ x , Ġ\ over left arrow {\ the ta Ġ_ { L ST M } }, Ġ\ the ta Ġ_ s ) $ Ġcorresponds Ġto Ġthe Ġsimilar Ġset Ġof Ġparameters Ġfor Ġthe Ġbackward Ġdirection . ĠSpecifically , Ġnote Ġthat Ġwe Ġshare Ġthe Ġparameters Ġfor Ġthe Ġembed ding Ġand Ġsoft max Ġlayer Ġacross Ġboth Ġdirections . ĠWe Ġmaximize ĠEqu ation Ġ19 Ġby Ġtraining Ġthe ĠBi - direction al ĠL ST Ms Ġusing Ġback - prop ag ation Ġthrough Ġtime . </s> After Ġhaving Ġestimated Ġthe Ġparameters Ġof Ġthe ĠDol ores Ġlear ner , Ġwe Ġnow Ġextract Ġthe Ġcontext - independent Ġand Ġcontext - dependent Ġrepresentations Ġfor Ġeach Ġentity Ġand Ġrelation Ġand Ġcombine Ġthem Ġto Ġobtain ĠDol ores Ġembed d ings . ĠMore Ġspecifically , ĠDol ores Ġembed d ings Ġare Ġtask Ġspecific Ġcombination Ġof Ġthe Ġcontext - dependent Ġand Ġcontext - independent Ġrepresentations Ġlearned Ġby Ġour Ġlear ner . ĠNote Ġthat Ġour Ġlear ner Ġ( which Ġis Ġan Ġ$ L $ Ġ- layer ĠBi - D irection al ĠL ST M ) Ġcomp utes Ġa Ġset Ġof Ġ$ 2 L Ġ+ Ġ1 $ Ġrepresentations Ġfor Ġeach Ġentity - relation Ġpair Ġwhich Ġwe Ġdenote Ġby : Ġ$ ĠR _ t Ġ= Ġ[ Ġx _ t , Ġ\ over right arrow { h _{ t , i } }, Ġ\ over left arrow { h _{ t , i }} Ġ\ mid Ġi Ġ= Ġ1 , Ġ2 , Ġ\ cd ots Ġ, Ġ\ text it Ġ{ L } Ġ], Ġ$ </s> where Ġ$ x _ t $ Ġis Ġthe Ġcontext - independent Ġembed ding Ġand Ġ$\ over right arrow { h _{ t , i } }, Ġ\ over left arrow { h _{ t , i }} $ Ġcorrespond Ġto Ġthe Ġcontext - dependent Ġembed d ings Ġfrom Ġlayer Ġ$ i $ Ġ. </s> Given Ġa Ġdownstream Ġmodel , ĠDol ores Ġlearns Ġa Ġweighted Ġlinear Ġcombination Ġof Ġthe Ġcomponents Ġof Ġ$ R _ t $ Ġto Ġyield Ġa Ġsingle Ġvector Ġfor Ġuse Ġin Ġthe Ġembed ding Ġlayer Ġof Ġthe Ġdownstream Ġmodel . ĠIn Ġparticular </s> $$ \ text tt Ġ{\ text sc Ġ{ D ol ores }} _ t Ġ= Ġ[ Ġx _ t Ġ, Ġ\ sum Ġ_ { i = 1 } ^{ L } Ġ\ lambda Ġ_ i Ġh _{ t , i } Ġ], $$ Ġ Ġ Ġ( E q . Ġ23 ) </s> where Ġwe Ġdenote Ġ$ h _{ t , i } $ Ġ= Ġ[ Ġ$\ over right arrow { h _{ t , i }} $ Ġ, Ġ$\ over left arrow { h _{ t , i }} $ Ġ] Ġand Ġ$\ lambda Ġ_ { i } $ Ġdenote Ġtask Ġspecific Ġlearn able Ġweights Ġof Ġthe Ġlinear Ġcombination . </s> While Ġit Ġis Ġobvious Ġthat Ġour Ġembed d ings Ġcan Ġbe Ġused Ġas Ġfeatures Ġfor Ġnew Ġpredictive Ġmodels , Ġit Ġis Ġalso Ġvery Ġeasy Ġto Ġincorporate Ġour Ġlearned ĠDol ores Ġembed d ings Ġinto Ġexisting Ġpredictive Ġmodels Ġon Ġknowledge Ġgraphs . ĠThe Ġonly Ġrequirement Ġis Ġthat Ġthe Ġmodel Ġaccepts Ġas Ġinput , Ġan Ġembed ding Ġlayer Ġ( for Ġentities Ġand Ġrelations ). ĠIf Ġa Ġmodel Ġfulf ills Ġthis Ġrequirement Ġ( which Ġa Ġlarge Ġnumber Ġof Ġneural Ġmodels Ġon Ġknowledge Ġgraphs Ġdo ), Ġwe Ġcan Ġjust Ġuse ĠDol ores Ġembed d ings Ġas Ġa Ġdrop - in Ġreplacement . ĠWe Ġjust Ġinitialize Ġthe Ġcorresponding Ġembed ding Ġlayer Ġwith ĠDol ores Ġembed d ings . ĠIn Ġour Ġevaluation Ġbelow , Ġwe Ġshow Ġhow Ġto Ġimprove Ġseveral Ġstate - of - the - art Ġmodels Ġon Ġvarious Ġtasks Ġsimply Ġby Ġincorporating ĠDol ores Ġas Ġa Ġdrop - in Ġreplacement Ġto Ġthe Ġoriginal Ġembed ding Ġlayer . </s> Exper iments </s> We Ġevaluate ĠDol ores Ġon Ġa Ġset Ġof Ġ4 Ġdifferent Ġprediction Ġtasks Ġon Ġknowledge Ġgraphs . ĠIn Ġeach Ġcase , Ġsimply Ġadding ĠDol ores Ġto Ġexisting Ġstate - of - the - art Ġmodels Ġimproves Ġthe Ġstate Ġof Ġthe Ġart Ġperformance Ġsignificantly Ġ( in Ġsome Ġcases Ġby Ġat Ġleast Ġ$ 9 . 5 \ % $ Ġ) Ġwhich Ġwe Ġshow Ġin ĠTable Ġ1 Ġ. ĠWhile Ġwe Ġprimarily Ġshow Ġthat Ġwe Ġcan Ġadvance Ġthe Ġstate - of - the - art Ġmodel Ġby Ġincorporating ĠDol ores Ġembed d ings Ġas Ġa ĠâĢ ľ drop - in âĢ Ŀ Ġreplacement , Ġfor Ġthe Ġsake Ġof Ġcomple teness , Ġwe Ġalso Ġreport Ġraw Ġnumbers Ġof Ġother Ġstrong Ġbaseline Ġmethods Ġ( like ĠTrans D , ĠTrans E , ĠKB GAN , ĠConv E , Ġand ĠConv KB ) Ġto Ġplace Ġthe Ġresults Ġin Ġcontext . ĠWe Ġemphasize Ġthat Ġour Ġmethod Ġis Ġvery Ġgeneric Ġand Ġcan Ġbe Ġused Ġto Ġimprove Ġthe Ġperformance Ġof Ġa Ġlarge Ġclass Ġof Ġknowledge Ġgraph Ġprediction Ġmodels . ĠIn Ġthe Ġremainder Ġof Ġthe Ġsection , Ġwe Ġbriefly Ġprovide Ġhigh - level Ġoverview s Ġof Ġeach Ġtask Ġand Ġsummarize Ġresults Ġfor Ġall Ġtasks Ġconsidered . </s> Exper imental ĠSettings Ġfor ĠDol ores </s> Here , Ġwe Ġoutline Ġour Ġmodel Ġsettings Ġfor Ġlearning ĠDol ores Ġembed d ings . ĠWe Ġgenerate Ġ20 Ġchains Ġfor Ġeach Ġnode Ġin Ġthe Ġknowledge Ġgraph , Ġwith Ġthe Ġlength Ġof Ġeach Ġchain Ġbeing Ġ21 Ġ( 10 Ġrelations Ġand Ġ11 Ġentities Ġappear Ġaltern ately ). ĠOur Ġmodel Ġuses ĠL Ġ= Ġ4 ĠL ST M Ġlayers Ġwith Ġ512 Ġunits Ġand Ġ32 Ġdimension Ġprojections Ġ( project ed Ġvalues Ġare Ġclipped Ġelement - wise Ġto Ġwithin Ġ$ [ - 3 , Ġ3 ] $ Ġ). ĠWe Ġuse Ġresidual Ġconnections Ġbetween Ġlayers Ġand Ġthe Ġbatch Ġsize Ġis Ġset Ġto Ġ1024 Ġduring Ġthe Ġtraining Ġprocess . ĠWe Ġtrain ĠDol ores Ġfor Ġ200 Ġepoch s Ġon Ġcorresponding Ġdatasets Ġwith Ġdrop out Ġ( with Ġthe Ġdrop out Ġprobability Ġis Ġset Ġ$ 0 . 1 $ Ġ). ĠFinally , Ġwe Ġuse ĠAdam Ġas Ġthe Ġoptim izer Ġwith Ġappropriately Ġchosen Ġlearning Ġrates Ġbased Ġon Ġa Ġvalidation Ġset . </s> E val uation ĠT asks </s> We Ġconsider Ġthree Ġtasks , Ġlink Ġprediction , Ġtriple Ġclassification , Ġand Ġpredicting Ġmissing Ġrelation Ġtypes ĠB IB REF 24 Ġ: </s> Link ĠPrediction ĠA Ġcommon Ġtask Ġfor Ġknowledge Ġgraph Ġcompletion Ġis Ġlink Ġprediction , Ġaiming Ġto Ġpredict Ġthe Ġmissing Ġentity Ġwhen Ġthe Ġother Ġtwo Ġparts Ġof Ġa Ġtriple t Ġ( h , Ġr , Ġt ) Ġare Ġgiven . ĠIn Ġother Ġwords , Ġwe Ġneed Ġto Ġpredict Ġt Ġgiven Ġ( h , Ġr ) ĠâĢĵ Ġtail - entity Ġprediction Ġor Ġpredict Ġh Ġgiven Ġ( r , Ġt ) ĠâĢĵ Ġhead Ġentity Ġprediction . ĠIn - line Ġwith Ġprior Ġwork ĠB IB REF 6 Ġ, Ġwe Ġreport Ġresults Ġon Ġlink Ġprediction Ġin Ġterms Ġof ĠMean ĠRec ip rocal ĠRank Ġ( MR R ), ĠMean ĠRank Ġ( MR ) Ġand ĠHits @ 10 Ġon Ġthe ĠFB 15 K - 237 Ġdataset Ġin Ġthe Ġfiltered Ġsetting Ġon Ġboth Ġsub - t asks : Ġ( a ) Ġhead Ġentity Ġprediction Ġand Ġ( b ) Ġtail Ġentity Ġprediction . ĠOur Ġresults Ġare Ġshown Ġin ĠTable Ġ2 Ġ. ĠNote Ġthat Ġthe Ġpresent Ġstate - of - the - art Ġmodel , ĠConv KB Ġachieves Ġan ĠMR R Ġof Ġ$( 0 . 375 , Ġ0 . 487 ) $ Ġon Ġthe Ġhead Ġand Ġtail Ġlink Ġprediction Ġtasks . ĠObs erve Ġthat Ġsimply Ġincorporating ĠDol ores Ġsignificantly Ġimproves Ġthe Ġhead Ġand Ġtail Ġentity Ġprediction Ġperformance Ġby Ġ$ 3 . 10 \ % $ Ġand Ġ$ 7 . 90 \ % $ Ġrespectively . ĠSimilar Ġimprovements Ġare Ġalso Ġobserved Ġon Ġother Ġmetrics Ġlike ĠMean ĠRank Ġ( MR : Ġlower Ġis Ġbetter ) Ġand ĠHits @ 10 Ġ( higher Ġis Ġbetter ). </s> Tri ple ĠClassification ĠThe Ġtask Ġof Ġtriple Ġclassification Ġis Ġto Ġpredict Ġwhether Ġa Ġtriple Ġ( h , Ġr , Ġt ) Ġis Ġcorrect Ġor Ġnot . ĠTriple Ġclassification Ġis Ġa Ġbinary Ġclassification Ġtask Ġwidely Ġexplored Ġby Ġprevious Ġwork ĠB IB REF 2 Ġ, ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ. ĠSince Ġevaluation Ġof Ġclassification Ġneeds Ġnegative Ġtri ples , Ġwe Ġchoose ĠW N 11 Ġand ĠFB 13 , Ġtwo Ġbenchmark Ġdatasets Ġand Ġreport Ġthe Ġresults Ġof Ġour Ġevaluation Ġin ĠTable Ġ3 Ġ. ĠWe Ġnote Ġthat Ġthe Ġpresent Ġstate Ġof Ġthe Ġart Ġis Ġthe ĠConv KB Ġmodel . ĠWhen Ġwe Ġadd ĠDol ores Ġto Ġthe ĠConv KB Ġmodel Ġwith Ġour Ġembed d ings , Ġobserve Ġthat Ġwe Ġimprove Ġthe Ġaverage Ġperformance Ġof Ġthe Ġstate - of - the - art Ġmodel ĠConv KB Ġslightly Ġby Ġ$ 0 . 20 $ Ġpoints Ġ( from Ġ$ 88 . 20 $ Ġto Ġ$ 88 . 40 $ Ġ). ĠWe Ġbelieve Ġthe Ġimprovement Ġachieved Ġby Ġadding ĠDol ores Ġis Ġsmaller Ġin Ġterms Ġof Ġabsolute Ġsize Ġbecause Ġthe Ġstate - of - the - art Ġmodel Ġalready Ġhas Ġexcellent Ġperformance Ġon Ġthis Ġtask Ġ( Ġ$ 88 . 20 $ Ġ) Ġsuggesting Ġa Ġmuch Ġslower Ġimprovement Ġcurve . </s> Missing ĠRel ation ĠTypes ĠThe Ġgoal Ġof Ġthe Ġthird Ġtask Ġis Ġto Ġreason Ġon Ġthe Ġpaths Ġconnecting Ġan Ġentity Ġpair Ġto Ġpredict Ġmissing Ġrelation Ġtypes . ĠWe Ġfollow ĠDas Ġet Ġal . Ġ( ĠB IB REF 24 Ġ) Ġand Ġuse Ġthe Ġsame Ġdataset Ġreleased Ġby ĠNe el ak ant an , ĠRoth , Ġand ĠMcC all um Ġ( ĠB IB REF 23 Ġ) Ġwhich Ġis Ġa Ġsubset Ġof ĠFree Base Ġenriched Ġwith Ġinformation Ġfrom ĠCl ue Web . ĠThe Ġdataset Ġconsists Ġof Ġa Ġset Ġof Ġtri ples Ġ( Ġ$ e _ 1 $ Ġ, Ġr , Ġ$ e _ 2 $ Ġ) Ġand Ġthe Ġset Ġof Ġpaths Ġconnecting Ġthe Ġentity Ġpair Ġ( Ġ$ e _ 1 $ Ġ, Ġ$ e _ 2 $ Ġ) Ġin Ġthe Ġknowledge Ġgraph . ĠThese Ġtri ples Ġare Ġcollected Ġfrom ĠCl ue Web Ġby Ġconsidering Ġsentences Ġthat Ġcontain Ġthe Ġentity Ġpair Ġin ĠFree base . ĠNe el ak ant an Ġet Ġal . Ġ( ĠB IB REF 23 Ġ) Ġinfer Ġthe Ġrelation Ġtype Ġby Ġexamining Ġthe Ġphrase Ġbetween Ġtwo Ġentities . ĠWe Ġuse Ġthe Ġsame Ġevaluation Ġcriterion Ġas Ġused Ġby ĠDas Ġet Ġal . Ġ( ĠB IB REF 24 Ġ) Ġand Ġreport Ġour Ġresults Ġin ĠTable Ġ4 Ġ. ĠNote Ġthat Ġthe Ġcurrent Ġstate - of - the - art Ġmodel Ġfrom ĠDas Ġet Ġal . Ġ( ĠB IB REF 24 Ġ) Ġyields Ġa Ġscore Ġof Ġ$ 71 . 74 $ Ġ. ĠAdding ĠDol ores Ġto Ġthe Ġmodel Ġimproves Ġthe Ġscore Ġto Ġ$ 74 . 42 $ Ġyielding Ġa Ġ$\ text bf Ġ{ 9 . 5 \ % } $ Ġimprovement . </s> Alt ogether , Ġviewing Ġthe Ġresults Ġon Ġvarious Ġtasks Ġhol istically , Ġwe Ġconclude Ġthat Ġsimply Ġincorporating ĠDol ores Ġinto Ġexisting Ġstate - of - the - art Ġmodels Ġimproves Ġtheir Ġperformance Ġand Ġadvances Ġthe Ġstate Ġof Ġthe Ġart Ġon Ġeach Ġof Ġthese Ġtasks Ġand Ġsuggests Ġthat Ġour Ġembed d ings Ġcan Ġbe Ġeffective Ġin Ġyielding Ġperformance Ġgains Ġon Ġa Ġvariety Ġof Ġpredictive Ġtasks . </s> Error ĠAnalysis </s> In Ġthis Ġsection , Ġwe Ġconduct Ġan Ġanalysis Ġof Ġthe Ġpredictions Ġmade Ġwhen Ġusing ĠDol ores Ġon Ġthe Ġlink Ġprediction Ġtasks Ġto Ġgain Ġinsights Ġinto Ġour Ġlearned Ġembed d ings . ĠTo Ġdo Ġso , Ġwe Ġgroup Ġthe Ġtest Ġtri ples Ġby Ġthe Ġfirst Ġcomponent Ġof Ġtheir Ġrelation Ġ( the Ġmost Ġabstract Ġconcept Ġor Ġlevel ) Ġand Ġcompute Ġthe Ġmean Ġrank Ġof Ġthe Ġtail Ġentity Ġoutput Ġover Ġeach Ġgroup . ĠWe Ġcompare Ġthis Ġmetric Ġagainst Ġwhat Ġa Ġsimple Ġbaseline Ġlike ĠTrans E Ġob tains . ĠThis Ġenables Ġus Ġto Ġidentify Ġoverall Ġstatistical Ġpatterns Ġthat Ġdistinguish ĠDol ores Ġembed d ings Ġfrom Ġbaseline Ġembed d ings Ġlike ĠTrans E , Ġwhich Ġin Ġturn Ġboosts Ġthe Ġperformance Ġof Ġlink Ġprediction . </s> Figure Ġ3 Ġshows Ġthe Ġrelation Ġcategories Ġfor Ġwhich ĠDol ores Ġperformed Ġthe Ġbest Ġand Ġthe Ġworst Ġrelative Ġto Ġbaseline ĠTrans E Ġembed d ings Ġin Ġterms Ġof Ġmean Ġrank Ġ( lower Ġis Ġbetter ). ĠIn Ġparticular , ĠFigure Ġ3 Ġshows Ġthe Ġcategories Ġwhere ĠDol ores Ġperforms Ġthe Ġbest Ġand Ġreveals Ġcategories Ġlike Ġfood , Ġuser , Ġo lymp ics , Ġorganization . ĠNote Ġfor Ġexample , Ġthat Ġfor Ġtri ples Ġbelonging Ġunder Ġthe Ġfood Ġrelation , ĠDol ores Ġoutper forms Ġbaseline Ġby ĠTrans E Ġby Ġa Ġfactor Ġof Ġ4 Ġin Ġterms Ġof Ġmean Ġrank . ĠTo Ġillustrate Ġthis Ġpoint , Ġwe Ġshow Ġa Ġfew Ġinstances Ġof Ġsuch Ġtri ples Ġbelow : </s> ( A Ġserious Ġman , Ġfilm - release - region , ĠHong ĠKong ) </s> ( C ab aret , Ġfilm - genre , ĠFilm ĠAdapt ation ) </s> ( Lou ĠCost ello , Ġpeople - prof ession , ĠComed ian ) </s> Note Ġthat Ġwhen Ġthe Ġhead Ġentity Ġis Ġa Ġvery Ġspecific Ġentity Ġlike ĠLouis ĠCost ello , Ġour Ġmethod Ġis Ġvery Ġaccurate Ġat Ġpredicting Ġthe Ġcorrect Ġtail Ġentity Ġ( in Ġthis Ġcase ĠComed ian ). ĠTrans E , Ġon Ġthe Ġother Ġhand , Ġmakes Ġvery Ġpoor Ġpredictions Ġon Ġsuch Ġcases . ĠWe Ġbelieve Ġthat Ġour Ġmethod Ġis Ġable Ġto Ġmodel Ġsuch Ġcases Ġbetter Ġbecause Ġour Ġembed d ings , Ġespecially Ġfor Ġsuch Ġspecific Ġentities , Ġhave Ġcaptured Ġthe Ġrich Ġcontext Ġassociated Ġwith Ġthem Ġfrom Ġentire Ġpaths . </s> In Ġcontrast , ĠFigure Ġ3 Ġshows Ġthe Ġrelation Ġcategories Ġthat ĠDol ores Ġperforms Ġthe Ġworst Ġrelative Ġto ĠTrans E . ĠWe Ġnote Ġthat Ġthese Ġcorrespond Ġto Ġvery Ġbroad Ġrelation Ġcategories Ġlike Ġcommon , base , media _ common Ġetc . ĠWe Ġlist Ġa Ġfew Ġtri ples Ġbelow Ġto Ġillustrate Ġthis Ġpoint : </s> ( Psych ology , Ġstudents Ġmajor ing , ĠY anni ) </s> ( De cca ĠRecords , Ġmusic - record - label - art </s> ist , ĠJes sey e ĠNorman ) </s> ( Prior ity ĠRecords , Ġmusic - record - label - </s> artist , ĠCar ole ĠKing ) </s> Note Ġthat Ġsuch Ġinstances Ġare Ġindeed Ġvery Ġdifficult . ĠFor Ġinstance , Ġgiven Ġa Ġvery Ġgeneric Ġentity Ġlike ĠPsychology Ġit Ġis Ġvery Ġdifficult Ġto Ġguess Ġthat ĠY anni Ġwould Ġbe Ġthe Ġexpected Ġtail Ġentity . ĠAlt ogether , Ġour Ġmethod Ġis Ġable Ġto Ġbetter Ġmodel Ġtri ples Ġwhere Ġthe Ġhead Ġentity Ġis Ġmore Ġspecific Ġcompared Ġto Ġhead Ġentities Ġwhich Ġare Ġvery Ġbroad Ġand Ġgeneral . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġintroduce ĠDol ores , Ġa Ġnew Ġparadigm Ġof Ġlearning Ġknowledge Ġgraph Ġembed d ings Ġwhere Ġwe Ġlearn Ġnot Ġonly Ġcontextual Ġindependent Ġembed d ings Ġof Ġentities Ġand Ġrelations Ġbut Ġalso Ġmultiple Ġcontext - dependent Ġembed d ings Ġeach Ġcapturing Ġa Ġdifferent Ġlayer Ġof Ġabstraction . ĠWe Ġdemonstrate Ġthat Ġby Ġleveraging Ġconnections Ġbetween Ġthree Ġseemingly Ġdistinct Ġfields Ġnamely : Ġ( a ) Ġlarge - scale Ġnetwork Ġanalysis , Ġ( b ) Ġnatural Ġlanguage Ġprocessing Ġand Ġ( c ) Ġknowledge Ġgraphs Ġwe Ġare Ġable Ġto Ġlearn Ġrich Ġknowledge Ġgraph Ġembed d ings Ġthat Ġare Ġdeep Ġand Ġcontextual ized Ġin Ġcontrast Ġto Ġprior Ġmodels Ġthat Ġare Ġtypically Ġshallow . ĠMoreover , Ġour Ġlearned Ġembed d ings Ġcan Ġbe Ġeasily Ġincorporated Ġinto Ġexisting Ġknowledge Ġgraph Ġprediction Ġmodels Ġto Ġimprove Ġtheir Ġperformance . ĠSpecifically , Ġwe Ġshow Ġthat Ġour Ġembed d ings Ġare Ġnot Ġonly Ġa ĠâĢ ľ drop - in âĢ Ŀ Ġreplacement Ġfor Ġexisting Ġmodels Ġthat Ġuse Ġembed ding Ġlayers Ġbut Ġalso Ġsignificantly Ġimprove Ġthe Ġstate - of - the - art Ġmodels Ġon Ġa Ġvariety Ġof Ġtasks , Ġsometimes Ġby Ġalmost Ġ$ 9 . 5 \ % $ Ġ. ĠFurthermore , Ġour Ġmethod Ġis Ġinherently Ġonline Ġand Ġscales Ġto Ġlarge Ġdata - sets . </s> Our Ġwork Ġalso Ġnaturally Ġsuggests Ġnew Ġdirections Ġof Ġinvestigation Ġand Ġresearch Ġinto Ġknowledge Ġgraph Ġembed d ings . ĠOne Ġavenue Ġof Ġresearch Ġis Ġto Ġinvestigate Ġthe Ġutility Ġof Ġeach Ġlayer 's Ġentity Ġand Ġrelation Ġembed d ings Ġin Ġspecific Ġlearning Ġtasks . ĠAs Ġwas Ġnoted Ġby Ġresearch Ġin Ġcomputer Ġvision , Ġdeep Ġrepresentations Ġlearned Ġon Ġone Ġdataset Ġare Ġeffective Ġand Ġvery Ġuseful Ġin Ġtransfer - learning Ġtasks ĠB IB REF 33 Ġ. ĠA Ġnatural Ġline Ġof Ġinvestigation Ġthus Ġrevolves Ġaround Ġprecisely Ġquant ifying Ġthe Ġeffectiveness Ġof Ġthese Ġlearned Ġembed d ings Ġacross Ġmodels Ġand Ġdata - sets . ĠLastly , Ġit Ġwould Ġbe Ġinteresting Ġto Ġsee Ġif Ġsuch Ġknowledge Ġgraph Ġembed d ings Ġcan Ġbe Ġused Ġin Ġconjunction Ġwith Ġnatural Ġlanguage Ġprocessing Ġmodels Ġused Ġfor Ġrelation Ġextraction Ġand Ġnamed Ġentity Ġrecognition Ġfrom Ġraw Ġtextual Ġdata . </s> Finally , Ġin Ġa Ġbroader Ġperspective , Ġour Ġwork Ġintroduces Ġtwo Ġnew Ġparad ig ms Ġof Ġmodeling Ġknowledge Ġgraphs . ĠFirst , Ġrather Ġthan Ġview Ġa Ġknowledge Ġgraph Ġas Ġa Ġcollection Ġof Ġtri ples , Ġwe Ġview Ġit Ġas Ġa Ġset Ġof Ġpaths Ġbetween Ġentities . ĠThese Ġpaths Ġcan Ġbe Ġrepresented Ġas Ġa Ġcollection Ġof Ġtrunc ated Ġrandom Ġwalks Ġover Ġthe Ġknowledge Ġgraph Ġand Ġencode Ġrich Ġcontextual Ġinformation Ġbetween Ġentities Ġand Ġrelations . ĠSecond , Ġdeparting Ġfrom Ġthe Ġhitherto Ġwell - established Ġparadigm Ġof Ġmapping Ġentities Ġand Ġrelations Ġto Ġvectors Ġin Ġ$\ math cal Ġ{ R } ^{ d } $ Ġvia Ġa Ġmapping Ġfunction , Ġwe Ġlearn Ġmultiple Ġrepresentations Ġfor Ġentities Ġand Ġrelations Ġdetermined Ġby Ġthe Ġnumber Ġof Ġlayers Ġin Ġa Ġdeep Ġneural Ġnetwork . ĠThis Ġenables Ġus Ġto Ġlearn Ġknowledge Ġgraph Ġembed d ings Ġthat Ġcapture Ġdifferent Ġlayers Ġof Ġabstraction ĠâĢĵ Ġboth Ġcontext - independent Ġand Ġcontext - dependent Ġallowing Ġfor Ġthe Ġdevelopment Ġof Ġvery Ġpowerful Ġprediction Ġmodels Ġto Ġyield Ġsuperior Ġperformance Ġon Ġa Ġvariety Ġof Ġprediction Ġtasks .
Introduction Knowledge graphs BIBREF0 enable structured access to world knowledge and form a key component of several applications like search engines, question answering systems and conversational assistants. Knowledge graphs are typically interpreted as comprising of discrete triples of the form (entityA, relationX, entityB) thus representing a relation (relationX) between entityA and entityB. However, one limitation of only a discrete representation of triples is that it does not easily enable one to infer similarities and potential relations among entities which may be missing in the knowledge graph. Consequently, one popular alternative is to learn dense continuous representations of entities and relations by embedding them in latent continuous vector spaces, while seeking to model the inherent structure of the knowledge graph. Most knowledge graph embedding methods can be classified into two major classes: one class which operates purely on triples like RESCAL BIBREF1 , TransE BIBREF2 , DistMult BIBREF3 , TransD BIBREF4 , ComplEx BIBREF5 , ConvE BIBREF6 and the second class which seeks to incorporate additional information (like multi-hops) BIBREF7 . Learning high-quality knowledge graph embeddings can be quite challenging given that (a) they need to effectively model the contextual usages of entities and relations (b) they would need to be useful for a variety of predictive tasks on knowledge graphs. In this paper, we present a new type of knowledge graph embeddings called Dolores that are both deep and contextualized. Dolores learns both context-independent and context-dependent embeddings of entities and relations through a deep neural sequential model. Figure 1 illustrates the deep contextualized representations learned. Note that the contextually independent entity embeddings (see Figure 1 ) reveal three clusters of entities: writers, philosophers, and musicians. The contextual dependent embeddings in turn effectively account for specific relations. In particular, the context-dependent representations under the relation nationality now nicely cluster the above entities by nationality namely Austrians, Germans, and British/Irish. Similarly Figure 1 shows contextual embeddings given the relation place-lived. Note that these embeddings correctly capture that even though Beethoven and Brahms being Germans, they lived in Vienna and are closer to other Austrian musicians like Schubert. Unlike most knowledge graph embeddings like TransD, TransE BIBREF2 , BIBREF4 etc. which are typically learned using shallow models, the representations learned by Dolores are deep: dependent on an entire path (rather than just a triple), are functions of internal states of a Bi-Directional LSTM and composed of representations learned at various layers potentially capturing varying degrees of abstractions. Dolores is inspired by recent advances in learning word representations (word embeddings) from deep neural language models using Bi-Directional LSTMs BIBREF8 . In particular, we derive connections between the work of Peters et al. ( BIBREF8 ) who learn deep contextualized word embeddings from sentences using a Bi-Directional LSTM based language model and random walks on knowledge graphs. These connections enable us to propose new “deep contextualized” knowledge graph embeddings which we call Dolores embeddings. Knowledge Embeddings learned using Dolores can easily be used as input representations for predictive models on knowledge graphs. More importantly, when existing predictive models use input representations for entities and relations, we can easily replace those representations with Dolores representations and significantly improve the performance of existing models. Specifically, we show that Dolores embeddings advance the state-of-the-art models on various tasks like link prediction, triple classification and missing relation type prediction. To summarize, our contributions are as follows: Related Work Extensive work exists on knowledge graph embeddings dating back to Nickel, Tresp, and Kriegel ( BIBREF1 ) who first proposed Rescal based on a matrix factorization approach. Bordes et al. ( BIBREF2 ) advanced this line of work by proposing the first translational model TransE which seeks to relate the head and tail entity embeddings by modeling the relation as a translational vector. This culminated in a long series of new knowledge graph embeddings all based on the translational principle with various refinements BIBREF9 , BIBREF10 , BIBREF4 , BIBREF3 , BIBREF5 , BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14 , BIBREF15 , BIBREF16 . Some recently proposed models like ManiFoldE BIBREF17 attempt to learn knowledge graph embeddings as a manifold while embeddings like HolE BIBREF1 derive inspiration from associative memories. Furthermore, with the success of neural models, models based on convolutional neural networks have been proposed like BIBREF6 , BIBREF18 to learn knowledge graph embeddings. Other models in this class of models include ConvKB BIBREF19 and KBGAN BIBREF20 . There has been some work on incorporating additional information like entity types, relation paths etc. to learn knowledge graph representations. Palumbo et al. ( BIBREF21 ) use node2vec to learn embeddings of entities and items in a knowledge graph. A notable class of methods called “path-ranking” based models directly model paths between entities as features. Examples include Path Ranking Algorithm (PRA) BIBREF22 , PTransE BIBREF10 and models based on recurrent neural networks BIBREF23 , BIBREF24 . Besides, Das et al. ( BIBREF25 ) propose a reinforcement learning method that addresses practical task of answering questions where the relation is known, but only one entity. Hartford et al. ( BIBREF26 ) model interactions across two or more sets of objects using a parameter-sharing scheme. While most of the above models except for the recurrent-neural net abased models above are shallow our model Dolores differs from all of these works and especially that of Palumbo et al. ( BIBREF21 ) in that we learn deep contextualized knowledge graph representations of entities and relations using a deep neural sequential model. The work that is closest to our work is that of Das et al. ( BIBREF24 ) who directly use an RNN-based architecture to model paths to predict missing links. We distinguish our work from this in the following key ways: (a) First, unlike Das et al. ( BIBREF24 ), our focus is not on path reasoning but on learning rich knowledge graph embeddings useful for a variety of predictive tasks. Moreover while Das et al. ( BIBREF24 ) need to use paths generated from PRA that typically correlate with relations, our method has no such restriction and only uses paths generated by generic random walks greatly enhancing the scalability of our method. In fact, we incorporate Dolores embeddings to improve the performance of the model proposed by Das et al. ( BIBREF24 ). (b) Second, and most importantly we learn knowledge graph embeddings at multiple layers each potentially capturing different levels of abstraction. (c) Finally, while we are inspired by the work of Peters et al. ( BIBREF8 ) in learning deep word representations, we build on their ideas by drawing connections between knowledge graphs and language modeling BIBREF8 . In particular, we propose methods to use random walks over knowledge graphs in conjunction with the machinery of deep neural language modeling to learn powerful deep contextualized knowledge graph embeddings that improve the state of the art on various knowledge graph tasks. Problem Formulation Given a knowledge graph $G=(E, R)$ where $E$ denotes the set of entities and $R$ denotes the set of relations among those entities, we seek to learn $d$ -dimensional embeddings of the entities and relations. In contrast to previous knowledge graph embedding methods like BIBREF2 , BIBREF9 , BIBREF4 , BIBREF10 , BIBREF5 which are based on shallow models and operates primarily on triples, our method Dolores uses a deep neural model to learn “deep” and “contextualized” knowledge graph embeddings. Having formulated the problem, we now describe Dolores. Dolores consists of two main components: Path Generator This component is responsible for generating a large set of entity-relation chains that reflect the varying contextual usages of entities and relations in the knowledge graph. Learner This component is a deep neural model that takes as input entity-relation chains and learns entity and relation embeddings which are weighted linear combination of internal states of the model thus capturing context dependence. Both of the above components are motivated by recent advances in learning deep representations of words in language modeling. We motivate this below and also highlight key connections that enable us to build on these advances to learn Dolores knowledge graph embeddings. Preliminaries Recall that the goal of a language model is to estimate the likelihood of a sequence of words: $w_{1}, w_{2}, \cdots ,w_{n}$ where each word $w_{i}$ is from a finite vocabulary $\mathcal {V}$ . Specifically, the goal of a forward language model is to model $\text{Pr}(w_{i}|w_{1}, w_{2}, \cdots ,w_{i-1})$ . While, traditionally this has been modeled using count-based “n-gram based" models BIBREF27 , BIBREF28 , recently deep neural models like LSTMs and RNN's have been used to build such language models. As noted by Peters et al. ( BIBREF8 ), a forward language model implemented using an LSTM of “L” layers works as follows: At each position $k$ , each LSTM layer outputs a context-dependent representation denoted by $\overrightarrow{h_{k,j}}$ corresponding to the $j^{th}$ layer of the LSTM. The top-most layer of the LSTM is then fed as input to a softmax layer of size $|\mathcal {V}|$ to predict the next token. Similarly, a backward language model which models $\text{Pr}(w_{i}|w_{i+1}, w_{i+2}, \cdots ,w_{n})$ can be implemented using a “backward” LSTM producing similar representations. A Bi-Directional LSTM just combines both forward and backward directions and seeks to jointly maximize the log-likelihood of the forward and backward directional language model objectives. We note that these context-dependent representations learned by the LSTM at each layer have been shown to be useful as “deep contextual” word representations in various predictive tasks in natural language processing BIBREF8 . In line with this trend, we will also use deep neural sequential models more specifically Bi-Directional LSTMs to learn Dolores embeddings. We do this by generalizing this approach to graphs by noting connections first noted by Perozzi, Al-Rfou, and Skiena ( BIBREF29 ). Since the input to a language model is a large corpus or set of sentences, one can generalize language modeling approaches to graphs by noting that the analog of a sentence in graphs is a “random walk”. More specifically, note that a truncated random walk of length T starting from a node “v” is analogous to a sentence and effectively captures the context of “v" in the network. More precisely, the same machinery used to learn representations of words in language models can now be adapted to learn deep contextualized representations of knowledge graphs. This can easily be adapted to knowledge graphs by constructing paths of entities and relations. In particular, a random walk on a knowledge graph starting at entity $e_{1}$ and ending at entity $e_{k}$ is a sequence of the form $e_{1}, r_{1}, e_{2}, r_{2}, \cdots ,e_{k}$ representing the entities and the corresponding relations between $e_{1}$ and $e_{k}$ in the knowledge graph. Moving forward we denote such a path of entities and relations by $\mathbf {q}=(e_{1},r_{1}, e_{2}, r_{2}, \cdots ,e_{k})$ . We generate a large set of such paths from the knowledge graph G by performing several random walks on it which in turn yields a corpus of “sentences” S needed for “language modeling”. Dolores: Path Generator Having motivated the model and discussed preliminaries, we now describe the first component of Dolores – the path generator. Let S denote the set of entity-relation chains obtained by doing random walks in the knowledge graph. We adopt a component of node2vec BIBREF30 to construct S. In particular, we perform a 2nd order random walk with two parameters p and q that determine the degree of breadth-first sampling and depth-first sampling. Specifically as Grover and Leskovec ( BIBREF30 ) described, p controls the likelihood of immediately revisiting a node in the walk whereas q controls whether the walk is biased towards nodes close to starting node or away from starting node. We emphasize that while node2vec has additional steps to learn dense continuous representations of nodes, we adopt only its first component to generate a corpus of random walks representing paths in knowledge graphs. Dolores: Learner Having generated a set of paths on knowledge graphs representing local contexts of entities and relations, we are now ready to utilize the machinery of language modeling using deep neural networks to learn Dolores embeddings. While traditional language models model a sentence as a sequence of words, we adopt the same machinery to model knowledge graph embeddings as follows: (a) A word is an (entity, relation) tuple, (b) we model a sentence as a path consisting of (entity, relation) tuples. Note that we have already established how to generate such paths from the knowledge graph using the path generator component. Given such paths, we would like to model the probability of an entity-relation pair given the history and future context by a Bi-Directional Long Short-Term Memory network. In particular, the forward direction LSTM models: $ \text{Pr}([e_1,r_1], [e_2,r_2], \cdots , [e_N, r_N]) = $ $$\prod _{t=1}^{N} \text{Pr}( [e_t, r_t] \mid [e_1, r_1], [e_2, r_2], \cdots , [e_{t-1}, r_{t-1}] ).$$   (Eq. 17) Similarly, the backward direction LSTM models: $ \text{Pr}([e_1,r_1], [e_2,r_2], \cdots , [e_N, r_N]) = $ $$\prod _{t=1}^{N} \text{Pr}( [e_t, r_t] \mid [e_{t+1}, r_{t+1}], \cdots , [e_{N}, r_{N}] ).$$   (Eq. 18) Figure 2 illustrates this succinctly. At each time-step t, we deal with an entity-relation pair [ $e_t$ , $r_t$ ]. We first map one-hot vectors of the $e_t$ and $r_t$ to an embedding layer, concatenate them to obtain context-independent representations which are then passed through L layers of a Bi-Directional LSTM. Each layer of LSTM outputs the pair's context-dependent representation $\overrightarrow{h_{t,i}}$ , $\overleftarrow{h_{t,i}}$ , where i=1, 2, $\cdots $ , L. Finally, the output of the top layer of LSTM, $\overrightarrow{h_{t,L}}$ , $\overleftarrow{h_{t,L}}$ , is used to predict the next pair [ $e_{t+1}$ , $r_t$0 ] and [ $r_t$1 , $r_t$2 ] respectively using a softmax layer. Formally, we jointly maximize the log likelihood of the forward and backward directions: $$\begin{split} \sum _{t=1}^{N}\log \text{Pr}([e_t, r_t]\mid [e_1, r_1],\cdots ,[e_{t-1}, r_{t-1}];\mathbf {\Theta _{F}})+\\ \sum _{t=1}^{N}\log \text{Pr}([e_t, r_t]\mid [e_{t+1}, r_{t+1}],\cdots ,[e_{N}, r_{N}];\mathbf {\Theta _{B}}), \end{split}$$   (Eq. 19) where $\mathbf {\Theta _{F}}$ = $(\theta _x, \overrightarrow{\theta _{LSTM}}, \theta _s)$ corresponds to the parameters of the embedding layer, forward-direction LSTM and the softmax layer respectively. Similarly $\mathbf {\Theta _{B}}$ = $(\theta _x, \overleftarrow{\theta _{LSTM}}, \theta _s)$ corresponds to the similar set of parameters for the backward direction. Specifically, note that we share the parameters for the embedding and softmax layer across both directions. We maximize Equation 19 by training the Bi-directional LSTMs using back-propagation through time. After having estimated the parameters of the Dolores learner, we now extract the context-independent and context-dependent representations for each entity and relation and combine them to obtain Dolores embeddings. More specifically, Dolores embeddings are task specific combination of the context-dependent and context-independent representations learned by our learner. Note that our learner (which is an $L$ -layer Bi-Directional LSTM) computes a set of $2L + 1$ representations for each entity-relation pair which we denote by: $ R_t = [ x_t, \overrightarrow{h_{t,i}}, \overleftarrow{h_{t,i}} \mid i = 1, 2, \cdots , \textit {L} ], $ where $x_t$ is the context-independent embedding and $\overrightarrow{h_{t,i}}, \overleftarrow{h_{t,i}}$ correspond to the context-dependent embeddings from layer $i$ . Given a downstream model, Dolores learns a weighted linear combination of the components of $R_t$ to yield a single vector for use in the embedding layer of the downstream model. In particular $$\texttt {\textsc {Dolores}}_t = [ x_t , \sum _{i=1}^{L} \lambda _i h_{t,i} ],$$   (Eq. 23) where we denote $h_{t,i}$ = [ $\overrightarrow{h_{t,i}}$ , $\overleftarrow{h_{t,i}}$ ] and $\lambda _{i}$ denote task specific learnable weights of the linear combination. While it is obvious that our embeddings can be used as features for new predictive models, it is also very easy to incorporate our learned Dolores embeddings into existing predictive models on knowledge graphs. The only requirement is that the model accepts as input, an embedding layer (for entities and relations). If a model fulfills this requirement (which a large number of neural models on knowledge graphs do), we can just use Dolores embeddings as a drop-in replacement. We just initialize the corresponding embedding layer with Dolores embeddings. In our evaluation below, we show how to improve several state-of-the-art models on various tasks simply by incorporating Dolores as a drop-in replacement to the original embedding layer. Experiments We evaluate Dolores on a set of 4 different prediction tasks on knowledge graphs. In each case, simply adding Dolores to existing state-of-the-art models improves the state of the art performance significantly (in some cases by at least $9.5\%$ ) which we show in Table 1 . While we primarily show that we can advance the state-of-the-art model by incorporating Dolores embeddings as a “drop-in” replacement, for the sake of completeness, we also report raw numbers of other strong baseline methods (like TransD, TransE, KBGAN, ConvE, and ConvKB) to place the results in context. We emphasize that our method is very generic and can be used to improve the performance of a large class of knowledge graph prediction models. In the remainder of the section, we briefly provide high-level overviews of each task and summarize results for all tasks considered. Experimental Settings for Dolores Here, we outline our model settings for learning Dolores embeddings. We generate 20 chains for each node in the knowledge graph, with the length of each chain being 21 (10 relations and 11 entities appear alternately). Our model uses L = 4 LSTM layers with 512 units and 32 dimension projections (projected values are clipped element-wise to within $[-3, 3]$ ). We use residual connections between layers and the batch size is set to 1024 during the training process. We train Dolores for 200 epochs on corresponding datasets with dropout (with the dropout probability is set $0.1$ ). Finally, we use Adam as the optimizer with appropriately chosen learning rates based on a validation set. Evaluation Tasks We consider three tasks, link prediction, triple classification, and predicting missing relation types BIBREF24 : Link Prediction A common task for knowledge graph completion is link prediction, aiming to predict the missing entity when the other two parts of a triplet (h, r, t) are given. In other words, we need to predict t given (h, r) – tail-entity prediction or predict h given (r, t) – head entity prediction. In-line with prior work BIBREF6 , we report results on link prediction in terms of Mean Reciprocal Rank (MRR), Mean Rank (MR) and Hits@10 on the FB15K-237 dataset in the filtered setting on both sub-tasks: (a) head entity prediction and (b) tail entity prediction. Our results are shown in Table 2 . Note that the present state-of-the-art model, ConvKB achieves an MRR of $(0.375, 0.487)$ on the head and tail link prediction tasks. Observe that simply incorporating Dolores significantly improves the head and tail entity prediction performance by $3.10\%$ and $7.90\%$ respectively. Similar improvements are also observed on other metrics like Mean Rank (MR: lower is better) and Hits@10 (higher is better). Triple Classification The task of triple classification is to predict whether a triple (h, r, t) is correct or not. Triple classification is a binary classification task widely explored by previous work BIBREF2 , BIBREF9 , BIBREF10 . Since evaluation of classification needs negative triples, we choose WN11 and FB13, two benchmark datasets and report the results of our evaluation in Table 3 . We note that the present state of the art is the ConvKB model. When we add Dolores to the ConvKB model with our embeddings, observe that we improve the average performance of the state-of-the-art model ConvKB slightly by $0.20$ points (from $88.20$ to $88.40$ ). We believe the improvement achieved by adding Dolores is smaller in terms of absolute size because the state-of-the-art model already has excellent performance on this task ( $88.20$ ) suggesting a much slower improvement curve. Missing Relation Types The goal of the third task is to reason on the paths connecting an entity pair to predict missing relation types. We follow Das et al. ( BIBREF24 ) and use the same dataset released by Neelakantan, Roth, and McCallum ( BIBREF23 ) which is a subset of FreeBase enriched with information from ClueWeb. The dataset consists of a set of triples ( $e_1$ , r, $e_2$ ) and the set of paths connecting the entity pair ( $e_1$ , $e_2$ ) in the knowledge graph. These triples are collected from ClueWeb by considering sentences that contain the entity pair in Freebase. Neelakantan et al. ( BIBREF23 ) infer the relation type by examining the phrase between two entities. We use the same evaluation criterion as used by Das et al. ( BIBREF24 ) and report our results in Table 4 . Note that the current state-of-the-art model from Das et al. ( BIBREF24 ) yields a score of $71.74$ . Adding Dolores to the model improves the score to $74.42$ yielding a $\textbf {9.5\%}$ improvement. Altogether, viewing the results on various tasks holistically, we conclude that simply incorporating Dolores into existing state-of-the-art models improves their performance and advances the state of the art on each of these tasks and suggests that our embeddings can be effective in yielding performance gains on a variety of predictive tasks. Error Analysis In this section, we conduct an analysis of the predictions made when using Dolores on the link prediction tasks to gain insights into our learned embeddings. To do so, we group the test triples by the first component of their relation (the most abstract concept or level) and compute the mean rank of the tail entity output over each group. We compare this metric against what a simple baseline like TransE obtains. This enables us to identify overall statistical patterns that distinguish Dolores embeddings from baseline embeddings like TransE, which in turn boosts the performance of link prediction. Figure 3 shows the relation categories for which Dolores performed the best and the worst relative to baseline TransE embeddings in terms of mean rank (lower is better). In particular, Figure 3 shows the categories where Dolores performs the best and reveals categories like food, user, olympics, organization. Note for example, that for triples belonging under the food relation, Dolores outperforms baseline by TransE by a factor of 4 in terms of mean rank. To illustrate this point, we show a few instances of such triples below: (A serious man, film-release-region, Hong Kong) (Cabaret, film-genre, Film Adaptation) (Lou Costello, people-profession, Comedian) Note that when the head entity is a very specific entity like Louis Costello, our method is very accurate at predicting the correct tail entity (in this case Comedian). TransE, on the other hand, makes very poor predictions on such cases. We believe that our method is able to model such cases better because our embeddings, especially for such specific entities, have captured the rich context associated with them from entire paths. In contrast, Figure 3 shows the relation categories that Dolores performs the worst relative to TransE. We note that these correspond to very broad relation categories like common,base,media_common etc. We list a few triples below to illustrate this point: (Psychology, students majoring, Yanni) (Decca Records, music-record-label-art ist, Jesseye Norman) (Priority Records, music-record-label- artist, Carole King) Note that such instances are indeed very difficult. For instance, given a very generic entity like Psychology it is very difficult to guess that Yanni would be the expected tail entity. Altogether, our method is able to better model triples where the head entity is more specific compared to head entities which are very broad and general. Conclusion In this paper, we introduce Dolores, a new paradigm of learning knowledge graph embeddings where we learn not only contextual independent embeddings of entities and relations but also multiple context-dependent embeddings each capturing a different layer of abstraction. We demonstrate that by leveraging connections between three seemingly distinct fields namely: (a) large-scale network analysis, (b) natural language processing and (c) knowledge graphs we are able to learn rich knowledge graph embeddings that are deep and contextualized in contrast to prior models that are typically shallow. Moreover, our learned embeddings can be easily incorporated into existing knowledge graph prediction models to improve their performance. Specifically, we show that our embeddings are not only a “drop-in” replacement for existing models that use embedding layers but also significantly improve the state-of-the-art models on a variety of tasks, sometimes by almost $9.5\%$ . Furthermore, our method is inherently online and scales to large data-sets. Our work also naturally suggests new directions of investigation and research into knowledge graph embeddings. One avenue of research is to investigate the utility of each layer's entity and relation embeddings in specific learning tasks. As was noted by research in computer vision, deep representations learned on one dataset are effective and very useful in transfer-learning tasks BIBREF33 . A natural line of investigation thus revolves around precisely quantifying the effectiveness of these learned embeddings across models and data-sets. Lastly, it would be interesting to see if such knowledge graph embeddings can be used in conjunction with natural language processing models used for relation extraction and named entity recognition from raw textual data. Finally, in a broader perspective, our work introduces two new paradigms of modeling knowledge graphs. First, rather than view a knowledge graph as a collection of triples, we view it as a set of paths between entities. These paths can be represented as a collection of truncated random walks over the knowledge graph and encode rich contextual information between entities and relations. Second, departing from the hitherto well-established paradigm of mapping entities and relations to vectors in $\mathcal {R}^{d}$ via a mapping function, we learn multiple representations for entities and relations determined by the number of layers in a deep neural network. This enables us to learn knowledge graph embeddings that capture different layers of abstraction – both context-independent and context-dependent allowing for the development of very powerful prediction models to yield superior performance on a variety of prediction tasks.
Is fine-tuning required to incorporate these embeddings into existing models?
b06512c17d99f9339ffdab12cedbc63501ff527e
Is Ġfine - tun ing Ġrequired Ġto Ġincorporate Ġthese Ġembed d ings Ġinto Ġexisting Ġmodels ?
No
<s> Is Ġfine - tun ing Ġrequired Ġto Ġincorporate Ġthese Ġembed d ings Ġinto Ġexisting Ġmodels ? </s> Introduction </s> Know ledge Ġgraphs ĠB IB REF 0 Ġenable Ġstructured Ġaccess Ġto Ġworld Ġknowledge Ġand Ġform Ġa Ġkey Ġcomponent Ġof Ġseveral Ġapplications Ġlike Ġsearch Ġengines , Ġquestion Ġanswering Ġsystems Ġand Ġconvers ational Ġassistants . ĠKnowledge Ġgraphs Ġare Ġtypically Ġinterpreted Ġas Ġcomprising Ġof Ġdiscrete Ġtri ples Ġof Ġthe Ġform Ġ( entity A , Ġrelation X , Ġentity B ) Ġthus Ġrepresenting Ġa Ġrelation Ġ( relation X ) Ġbetween Ġentity A Ġand Ġentity B . ĠHowever , Ġone Ġlimitation Ġof Ġonly Ġa Ġdiscrete Ġrepresentation Ġof Ġtri ples Ġis Ġthat Ġit Ġdoes Ġnot Ġeasily Ġenable Ġone Ġto Ġinfer Ġsimilarities Ġand Ġpotential Ġrelations Ġamong Ġentities Ġwhich Ġmay Ġbe Ġmissing Ġin Ġthe Ġknowledge Ġgraph . ĠConsequently , Ġone Ġpopular Ġalternative Ġis Ġto Ġlearn Ġdense Ġcontinuous Ġrepresentations Ġof Ġentities Ġand Ġrelations Ġby Ġembed ding Ġthem Ġin Ġlatent Ġcontinuous Ġvector Ġspaces , Ġwhile Ġseeking Ġto Ġmodel Ġthe Ġinherent Ġstructure Ġof Ġthe Ġknowledge Ġgraph . ĠMost Ġknowledge Ġgraph Ġembed ding Ġmethods Ġcan Ġbe Ġclassified Ġinto Ġtwo Ġmajor Ġclasses : Ġone Ġclass Ġwhich Ġoperates Ġpurely Ġon Ġtri ples Ġlike ĠRES C AL ĠB IB REF 1 Ġ, ĠTrans E ĠB IB REF 2 Ġ, ĠDist Mult ĠB IB REF 3 Ġ, ĠTrans D ĠB IB REF 4 Ġ, ĠCompl Ex ĠB IB REF 5 Ġ, ĠConv E ĠB IB REF 6 Ġand Ġthe Ġsecond Ġclass Ġwhich Ġseeks Ġto Ġincorporate Ġadditional Ġinformation Ġ( like Ġmulti - hops ) ĠB IB REF 7 Ġ. ĠLearning Ġhigh - quality Ġknowledge Ġgraph Ġembed d ings Ġcan Ġbe Ġquite Ġchallenging Ġgiven Ġthat Ġ( a ) Ġthey Ġneed Ġto Ġeffectively Ġmodel Ġthe Ġcontextual Ġus ages Ġof Ġentities Ġand Ġrelations Ġ( b ) Ġthey Ġwould Ġneed Ġto Ġbe Ġuseful Ġfor Ġa Ġvariety Ġof Ġpredictive Ġtasks Ġon Ġknowledge Ġgraphs . </s> In Ġthis Ġpaper , Ġwe Ġpresent Ġa Ġnew Ġtype Ġof Ġknowledge Ġgraph Ġembed d ings Ġcalled ĠDol ores Ġthat Ġare Ġboth Ġdeep Ġand Ġcontextual ized . ĠDol ores Ġlearns Ġboth Ġcontext - independent Ġand Ġcontext - dependent Ġembed d ings Ġof Ġentities Ġand Ġrelations Ġthrough Ġa Ġdeep Ġneural Ġsequential Ġmodel . ĠFigure Ġ1 Ġillustrates Ġthe Ġdeep Ġcontextual ized Ġrepresentations Ġlearned . ĠNote Ġthat Ġthe Ġcontext ually Ġindependent Ġentity Ġembed d ings Ġ( see ĠFigure Ġ1 Ġ) Ġreveal Ġthree Ġclusters Ġof Ġentities : Ġwriters , Ġphilosophers , Ġand Ġmusicians . ĠThe Ġcontextual Ġdependent Ġembed d ings Ġin Ġturn Ġeffectively Ġaccount Ġfor Ġspecific Ġrelations . ĠIn Ġparticular , Ġthe Ġcontext - dependent Ġrepresentations Ġunder Ġthe Ġrelation Ġnationality Ġnow Ġnicely Ġcluster Ġthe Ġabove Ġentities Ġby Ġnationality Ġnamely ĠAust rians , ĠGermans , Ġand ĠBritish / Irish . ĠSimilarly ĠFigure Ġ1 Ġshows Ġcontextual Ġembed d ings Ġgiven Ġthe Ġrelation Ġplace - lived . ĠNote Ġthat Ġthese Ġembed d ings Ġcorrectly Ġcapture Ġthat Ġeven Ġthough ĠBe eth oven Ġand ĠBrah ms Ġbeing ĠGermans , Ġthey Ġlived Ġin ĠVienna Ġand Ġare Ġcloser Ġto Ġother ĠAustrian Ġmusicians Ġlike ĠSch u bert . </s> Unlike Ġmost Ġknowledge Ġgraph Ġembed d ings Ġlike ĠTrans D , ĠTrans E ĠB IB REF 2 Ġ, ĠB IB REF 4 Ġetc . Ġwhich Ġare Ġtypically Ġlearned Ġusing Ġshallow Ġmodels , Ġthe Ġrepresentations Ġlearned Ġby ĠDol ores Ġare Ġdeep : Ġdependent Ġon Ġan Ġentire Ġpath Ġ( rather Ġthan Ġjust Ġa Ġtriple ), Ġare Ġfunctions Ġof Ġinternal Ġstates Ġof Ġa ĠBi - D irection al ĠL ST M Ġand Ġcomposed Ġof Ġrepresentations Ġlearned Ġat Ġvarious Ġlayers Ġpotentially Ġcapturing Ġvarying Ġdegrees Ġof Ġabstract ions . ĠDol ores Ġis Ġinspired Ġby Ġrecent Ġadvances Ġin Ġlearning Ġword Ġrepresentations Ġ( word Ġembed d ings ) Ġfrom Ġdeep Ġneural Ġlanguage Ġmodels Ġusing ĠBi - D irection al ĠL ST Ms ĠB IB REF 8 Ġ. ĠIn Ġparticular , Ġwe Ġderive Ġconnections Ġbetween Ġthe Ġwork Ġof ĠPeters Ġet Ġal . Ġ( ĠB IB REF 8 Ġ) Ġwho Ġlearn Ġdeep Ġcontextual ized Ġword Ġembed d ings Ġfrom Ġsentences Ġusing Ġa ĠBi - D irection al ĠL ST M Ġbased Ġlanguage Ġmodel Ġand Ġrandom Ġwalks Ġon Ġknowledge Ġgraphs . ĠThese Ġconnections Ġenable Ġus Ġto Ġpropose Ġnew ĠâĢ ľ deep Ġcontextual ized âĢ Ŀ Ġknowledge Ġgraph Ġembed d ings Ġwhich Ġwe Ġcall ĠDol ores Ġembed d ings . </s> Know ledge ĠEmb edd ings Ġlearned Ġusing ĠDol ores Ġcan Ġeasily Ġbe Ġused Ġas Ġinput Ġrepresentations Ġfor Ġpredictive Ġmodels Ġon Ġknowledge Ġgraphs . ĠMore Ġimportantly , Ġwhen Ġexisting Ġpredictive Ġmodels Ġuse Ġinput Ġrepresentations Ġfor Ġentities Ġand Ġrelations , Ġwe Ġcan Ġeasily Ġreplace Ġthose Ġrepresentations Ġwith ĠDol ores Ġrepresentations Ġand Ġsignificantly Ġimprove Ġthe Ġperformance Ġof Ġexisting Ġmodels . ĠSpecifically , Ġwe Ġshow Ġthat ĠDol ores Ġembed d ings Ġadvance Ġthe Ġstate - of - the - art Ġmodels Ġon Ġvarious Ġtasks Ġlike Ġlink Ġprediction , Ġtriple Ġclassification Ġand Ġmissing Ġrelation Ġtype Ġprediction . </s> To Ġsummarize , Ġour Ġcontributions Ġare Ġas Ġfollows : </s> Related ĠWork </s> Ext ensive Ġwork Ġexists Ġon Ġknowledge Ġgraph Ġembed d ings Ġdating Ġback Ġto ĠNickel , ĠT resp , Ġand ĠK rieg el Ġ( ĠB IB REF 1 Ġ) Ġwho Ġfirst Ġproposed ĠRes cal Ġbased Ġon Ġa Ġmatrix Ġfactor ization Ġapproach . ĠB ordes Ġet Ġal . Ġ( ĠB IB REF 2 Ġ) Ġadvanced Ġthis Ġline Ġof Ġwork Ġby Ġproposing Ġthe Ġfirst Ġtransl ational Ġmodel ĠTrans E Ġwhich Ġseeks Ġto Ġrelate Ġthe Ġhead Ġand Ġtail Ġentity Ġembed d ings Ġby Ġmodeling Ġthe Ġrelation Ġas Ġa Ġtransl ational Ġvector . ĠThis Ġculminated Ġin Ġa Ġlong Ġseries Ġof Ġnew Ġknowledge Ġgraph Ġembed d ings Ġall Ġbased Ġon Ġthe Ġtransl ational Ġprinciple Ġwith Ġvarious Ġrefin ements ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġ. ĠSome Ġrecently Ġproposed Ġmodels Ġlike ĠMan i F old E ĠB IB REF 17 Ġattempt Ġto Ġlearn Ġknowledge Ġgraph Ġembed d ings Ġas Ġa Ġmanifold Ġwhile Ġembed d ings Ġlike ĠHol E ĠB IB REF 1 Ġderive Ġinspiration Ġfrom Ġassoci ative Ġmemories . ĠFurthermore , Ġwith Ġthe Ġsuccess Ġof Ġneural Ġmodels , Ġmodels Ġbased Ġon Ġconv olution al Ġneural Ġnetworks Ġhave Ġbeen Ġproposed Ġlike ĠB IB REF 6 Ġ, ĠB IB REF 18 Ġto Ġlearn Ġknowledge Ġgraph Ġembed d ings . ĠOther Ġmodels Ġin Ġthis Ġclass Ġof Ġmodels Ġinclude ĠConv KB ĠB IB REF 19 Ġand ĠKB GAN ĠB IB REF 20 Ġ. ĠThere Ġhas Ġbeen Ġsome Ġwork Ġon Ġincorporating Ġadditional Ġinformation Ġlike Ġentity Ġtypes , Ġrelation Ġpaths Ġetc . Ġto Ġlearn Ġknowledge Ġgraph Ġrepresentations . ĠPal umbo Ġet Ġal . Ġ( ĠB IB REF 21 Ġ) Ġuse Ġnode 2 vec Ġto Ġlearn Ġembed d ings Ġof Ġentities Ġand Ġitems Ġin Ġa Ġknowledge Ġgraph . ĠA Ġnotable Ġclass Ġof Ġmethods Ġcalled ĠâĢ ľ path - ranking âĢ Ŀ Ġbased Ġmodels Ġdirectly Ġmodel Ġpaths Ġbetween Ġentities Ġas Ġfeatures . ĠExamples Ġinclude ĠPath ĠRanking ĠAl gorithm Ġ( P RA ) ĠB IB REF 22 Ġ, ĠP Trans E ĠB IB REF 10 Ġand Ġmodels Ġbased Ġon Ġrecurrent Ġneural Ġnetworks ĠB IB REF 23 Ġ, ĠB IB REF 24 Ġ. ĠBesides , ĠDas Ġet Ġal . Ġ( ĠB IB REF 25 Ġ) Ġpropose Ġa Ġreinforcement Ġlearning Ġmethod Ġthat Ġaddresses Ġpractical Ġtask Ġof Ġanswering Ġquestions Ġwhere Ġthe Ġrelation Ġis Ġknown , Ġbut Ġonly Ġone Ġentity . ĠHartford Ġet Ġal . Ġ( ĠB IB REF 26 Ġ) Ġmodel Ġinteractions Ġacross Ġtwo Ġor Ġmore Ġsets Ġof Ġobjects Ġusing Ġa Ġparameter - sharing Ġscheme . ĠWhile Ġmost Ġof Ġthe Ġabove Ġmodels Ġexcept Ġfor Ġthe Ġrecurrent - ne ural Ġnet Ġab ased Ġmodels Ġabove Ġare Ġshallow Ġour Ġmodel ĠDol ores Ġdiffers Ġfrom Ġall Ġof Ġthese Ġworks Ġand Ġespecially Ġthat Ġof ĠPal umbo Ġet Ġal . Ġ( ĠB IB REF 21 Ġ) Ġin Ġthat Ġwe Ġlearn Ġdeep Ġcontextual ized Ġknowledge Ġgraph Ġrepresentations Ġof Ġentities Ġand Ġrelations Ġusing Ġa Ġdeep Ġneural Ġsequential Ġmodel . ĠThe Ġwork Ġthat Ġis Ġclosest Ġto Ġour Ġwork Ġis Ġthat Ġof ĠDas Ġet Ġal . Ġ( ĠB IB REF 24 Ġ) Ġwho Ġdirectly Ġuse Ġan ĠR NN - based Ġarchitecture Ġto Ġmodel Ġpaths Ġto Ġpredict Ġmissing Ġlinks . ĠWe Ġdistinguish Ġour Ġwork Ġfrom Ġthis Ġin Ġthe Ġfollowing Ġkey Ġways : Ġ( a ) ĠFirst , Ġunlike ĠDas Ġet Ġal . Ġ( ĠB IB REF 24 Ġ), Ġour Ġfocus Ġis Ġnot Ġon Ġpath Ġreasoning Ġbut Ġon Ġlearning Ġrich Ġknowledge Ġgraph Ġembed d ings Ġuseful Ġfor Ġa Ġvariety Ġof Ġpredictive Ġtasks . ĠMoreover Ġwhile ĠDas Ġet Ġal . Ġ( ĠB IB REF 24 Ġ) Ġneed Ġto Ġuse Ġpaths Ġgenerated Ġfrom ĠP RA Ġthat Ġtypically Ġcorrelate Ġwith Ġrelations , Ġour Ġmethod Ġhas Ġno Ġsuch Ġrestriction Ġand Ġonly Ġuses Ġpaths Ġgenerated Ġby Ġgeneric Ġrandom Ġwalks Ġgreatly Ġenhancing Ġthe Ġscal ability Ġof Ġour Ġmethod . ĠIn Ġfact , Ġwe Ġincorporate ĠDol ores Ġembed d ings Ġto Ġimprove Ġthe Ġperformance Ġof Ġthe Ġmodel Ġproposed Ġby ĠDas Ġet Ġal . Ġ( ĠB IB REF 24 Ġ). Ġ( b ) ĠSecond , Ġand Ġmost Ġimportantly Ġwe Ġlearn Ġknowledge Ġgraph Ġembed d ings Ġat Ġmultiple Ġlayers Ġeach Ġpotentially Ġcapturing Ġdifferent Ġlevels Ġof Ġabstraction . Ġ( c ) ĠFinally , Ġwhile Ġwe Ġare Ġinspired Ġby Ġthe Ġwork Ġof ĠPeters Ġet Ġal . Ġ( ĠB IB REF 8 Ġ) Ġin Ġlearning Ġdeep Ġword Ġrepresentations , Ġwe Ġbuild Ġon Ġtheir Ġideas Ġby Ġdrawing Ġconnections Ġbetween Ġknowledge Ġgraphs Ġand Ġlanguage Ġmodeling ĠB IB REF 8 Ġ. ĠIn Ġparticular , Ġwe Ġpropose Ġmethods Ġto Ġuse Ġrandom Ġwalks Ġover Ġknowledge Ġgraphs Ġin Ġconjunction Ġwith Ġthe Ġmachinery Ġof Ġdeep Ġneural Ġlanguage Ġmodeling Ġto Ġlearn Ġpowerful Ġdeep Ġcontextual ized Ġknowledge Ġgraph Ġembed d ings Ġthat Ġimprove Ġthe Ġstate Ġof Ġthe Ġart Ġon Ġvarious Ġknowledge Ġgraph Ġtasks . </s> Problem ĠForm ulation </s> Given Ġa Ġknowledge Ġgraph Ġ$ G =( E , ĠR ) $ Ġwhere Ġ$ E $ Ġdenotes Ġthe Ġset Ġof Ġentities Ġand Ġ$ R $ Ġdenotes Ġthe Ġset Ġof Ġrelations Ġamong Ġthose Ġentities , Ġwe Ġseek Ġto Ġlearn Ġ$ d $ Ġ- dimensional Ġembed d ings Ġof Ġthe Ġentities Ġand Ġrelations . ĠIn Ġcontrast Ġto Ġprevious Ġknowledge Ġgraph Ġembed ding Ġmethods Ġlike ĠB IB REF 2 Ġ, ĠB IB REF 9 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 5 Ġwhich Ġare Ġbased Ġon Ġshallow Ġmodels Ġand Ġoperates Ġprimarily Ġon Ġtri ples , Ġour Ġmethod ĠDol ores Ġuses Ġa Ġdeep Ġneural Ġmodel Ġto Ġlearn ĠâĢ ľ deep âĢ Ŀ Ġand ĠâĢ ľ context ual ized âĢ Ŀ Ġknowledge Ġgraph Ġembed d ings . </s> Having Ġformulated Ġthe Ġproblem , Ġwe Ġnow Ġdescribe ĠDol ores . ĠDol ores Ġconsists Ġof Ġtwo Ġmain Ġcomponents : </s> Path ĠGenerator ĠThis Ġcomponent Ġis Ġresponsible Ġfor Ġgenerating Ġa Ġlarge Ġset Ġof Ġentity - relation Ġchains Ġthat Ġreflect Ġthe Ġvarying Ġcontextual Ġus ages Ġof Ġentities Ġand Ġrelations Ġin Ġthe Ġknowledge Ġgraph . </s> Lear ner ĠThis Ġcomponent Ġis Ġa Ġdeep Ġneural Ġmodel Ġthat Ġtakes Ġas Ġinput Ġentity - relation Ġchains Ġand Ġlearns Ġentity Ġand Ġrelation Ġembed d ings Ġwhich Ġare Ġweighted Ġlinear Ġcombination Ġof Ġinternal Ġstates Ġof Ġthe Ġmodel Ġthus Ġcapturing Ġcontext Ġdependence . </s> Both Ġof Ġthe Ġabove Ġcomponents Ġare Ġmotivated Ġby Ġrecent Ġadvances Ġin Ġlearning Ġdeep Ġrepresentations Ġof Ġwords Ġin Ġlanguage Ġmodeling . ĠWe Ġmotivate Ġthis Ġbelow Ġand Ġalso Ġhighlight Ġkey Ġconnections Ġthat Ġenable Ġus Ġto Ġbuild Ġon Ġthese Ġadvances Ġto Ġlearn ĠDol ores Ġknowledge Ġgraph Ġembed d ings . </s> P rel im in aries </s> Rec all Ġthat Ġthe Ġgoal Ġof Ġa Ġlanguage Ġmodel Ġis Ġto Ġestimate Ġthe Ġlikelihood Ġof Ġa Ġsequence Ġof Ġwords : Ġ$ w _{ 1 }, Ġw _{ 2 }, Ġ\ cd ots Ġ, w _{ n } $ Ġwhere Ġeach Ġword Ġ$ w _{ i } $ Ġis Ġfrom Ġa Ġfinite Ġvocabulary Ġ$\ math cal Ġ{ V } $ Ġ. ĠSpecifically , Ġthe Ġgoal Ġof Ġa Ġforward Ġlanguage Ġmodel Ġis Ġto Ġmodel Ġ$\ text { Pr } ( w _{ i } | w _{ 1 }, Ġw _{ 2 }, Ġ\ cd ots Ġ, w _{ i - 1 }) $ Ġ. ĠWhile , Ġtraditionally Ġthis Ġhas Ġbeen Ġmodeled Ġusing Ġcount - based ĠâĢ ľ n - gram Ġbased " Ġmodels ĠB IB REF 27 Ġ, ĠB IB REF 28 Ġ, Ġrecently Ġdeep Ġneural Ġmodels Ġlike ĠL ST Ms Ġand ĠR NN 's Ġhave Ġbeen Ġused Ġto Ġbuild Ġsuch Ġlanguage Ġmodels . ĠAs Ġnoted Ġby ĠPeters Ġet Ġal . Ġ( ĠB IB REF 8 Ġ), Ġa Ġforward Ġlanguage Ġmodel Ġimplemented Ġusing Ġan ĠL ST M Ġof ĠâĢ ľ L âĢ Ŀ Ġlayers Ġworks Ġas Ġfollows : ĠAt Ġeach Ġposition Ġ$ k $ Ġ, Ġeach ĠL ST M Ġlayer Ġoutputs Ġa Ġcontext - dependent Ġrepresentation Ġden oted Ġby Ġ$\ over right arrow { h _{ k , j }} $ Ġcorresponding Ġto Ġthe Ġ$ j ^{ th } $ Ġlayer Ġof Ġthe ĠL ST M . ĠThe Ġtop - most Ġlayer Ġof Ġthe ĠL ST M Ġis Ġthen Ġfed Ġas Ġinput Ġto Ġa Ġsoft max Ġlayer Ġof Ġsize Ġ$ | \ math cal Ġ{ V } | $ Ġto Ġpredict Ġthe Ġnext Ġtoken . ĠSimilarly , Ġa Ġbackward Ġlanguage Ġmodel Ġwhich Ġmodels Ġ$\ text { Pr } ( w _{ i } | w _{ i + 1 }, Ġw _{ i + 2 }, Ġ\ cd ots Ġ, w _{ n }) $ Ġcan Ġbe Ġimplemented Ġusing Ġa ĠâĢ ľ back ward âĢ Ŀ ĠL ST M Ġproducing Ġsimilar Ġrepresentations . ĠA ĠBi - D irection al ĠL ST M Ġjust Ġcombines Ġboth Ġforward Ġand Ġbackward Ġdirections Ġand Ġseeks Ġto Ġjointly Ġmaximize Ġthe Ġlog - like lihood Ġof Ġthe Ġforward Ġand Ġbackward Ġdirectional Ġlanguage Ġmodel Ġobjectives . </s> We Ġnote Ġthat Ġthese Ġcontext - dependent Ġrepresentations Ġlearned Ġby Ġthe ĠL ST M Ġat Ġeach Ġlayer Ġhave Ġbeen Ġshown Ġto Ġbe Ġuseful Ġas ĠâĢ ľ deep Ġcontextual âĢ Ŀ Ġword Ġrepresentations Ġin Ġvarious Ġpredictive Ġtasks Ġin Ġnatural Ġlanguage Ġprocessing ĠB IB REF 8 Ġ. ĠIn Ġline Ġwith Ġthis Ġtrend , Ġwe Ġwill Ġalso Ġuse Ġdeep Ġneural Ġsequential Ġmodels Ġmore Ġspecifically ĠBi - D irection al ĠL ST Ms Ġto Ġlearn ĠDol ores Ġembed d ings . ĠWe Ġdo Ġthis Ġby Ġgeneral izing Ġthis Ġapproach Ġto Ġgraphs Ġby Ġnoting Ġconnections Ġfirst Ġnoted Ġby ĠPer o zzi , ĠAl - R f ou , Ġand ĠSk ien a Ġ( ĠB IB REF 29 Ġ). </s> Since Ġthe Ġinput Ġto Ġa Ġlanguage Ġmodel Ġis Ġa Ġlarge Ġcorpus Ġor Ġset Ġof Ġsentences , Ġone Ġcan Ġgeneral ize Ġlanguage Ġmodeling Ġapproaches Ġto Ġgraphs Ġby Ġnoting Ġthat Ġthe Ġanalog Ġof Ġa Ġsentence Ġin Ġgraphs Ġis Ġa ĠâĢ ľ random Ġwalk âĢ Ŀ . ĠMore Ġspecifically , Ġnote Ġthat Ġa Ġtrunc ated Ġrandom Ġwalk Ġof Ġlength ĠT Ġstarting Ġfrom Ġa Ġnode ĠâĢ ľ v âĢ Ŀ Ġis Ġanalogous Ġto Ġa Ġsentence Ġand Ġeffectively Ġcaptures Ġthe Ġcontext Ġof ĠâĢ ľ v " Ġin Ġthe Ġnetwork . ĠMore Ġprecisely , Ġthe Ġsame Ġmachinery Ġused Ġto Ġlearn Ġrepresentations Ġof Ġwords Ġin Ġlanguage Ġmodels Ġcan Ġnow Ġbe Ġadapted Ġto Ġlearn Ġdeep Ġcontextual ized Ġrepresentations Ġof Ġknowledge Ġgraphs . </s> This Ġcan Ġeasily Ġbe Ġadapted Ġto Ġknowledge Ġgraphs Ġby Ġconstructing Ġpaths Ġof Ġentities Ġand Ġrelations . ĠIn Ġparticular , Ġa Ġrandom Ġwalk Ġon Ġa Ġknowledge Ġgraph Ġstarting Ġat Ġentity Ġ$ e _{ 1 } $ Ġand Ġending Ġat Ġentity Ġ$ e _{ k } $ Ġis Ġa Ġsequence Ġof Ġthe Ġform Ġ$ e _{ 1 }, Ġr _{ 1 }, Ġe _{ 2 }, Ġr _{ 2 }, Ġ\ cd ots Ġ, e _{ k } $ Ġrepresenting Ġthe Ġentities Ġand Ġthe Ġcorresponding Ġrelations Ġbetween Ġ$ e _{ 1 } $ Ġand Ġ$ e _{ k } $ Ġin Ġthe Ġknowledge Ġgraph . ĠMoving Ġforward Ġwe Ġdenote Ġsuch Ġa Ġpath Ġof Ġentities Ġand Ġrelations Ġby Ġ$\ math bf Ġ{ q } =( e _{ 1 }, r _{ 1 }, Ġe _{ 2 }, Ġr _{ 2 }, Ġ\ cd ots Ġ, e _{ k }) $ Ġ. ĠWe Ġgenerate Ġa Ġlarge Ġset Ġof Ġsuch Ġpaths Ġfrom Ġthe Ġknowledge Ġgraph ĠG Ġby Ġperforming Ġseveral Ġrandom Ġwalks Ġon Ġit Ġwhich Ġin Ġturn Ġyields Ġa Ġcorpus Ġof ĠâĢ ľ sent ences âĢ Ŀ ĠS Ġneeded Ġfor ĠâĢ ľ language Ġmodeling âĢ Ŀ . </s> D ol ores : ĠPath ĠGenerator </s> Having Ġmotivated Ġthe Ġmodel Ġand Ġdiscussed Ġprelim in aries , Ġwe Ġnow Ġdescribe Ġthe Ġfirst Ġcomponent Ġof ĠDol ores ĠâĢĵ Ġthe Ġpath Ġgenerator . </s> Let ĠS Ġdenote Ġthe Ġset Ġof Ġentity - relation Ġchains Ġobtained Ġby Ġdoing Ġrandom Ġwalks Ġin Ġthe Ġknowledge Ġgraph . ĠWe Ġadopt Ġa Ġcomponent Ġof Ġnode 2 vec ĠB IB REF 30 Ġto Ġconstruct ĠS . ĠIn Ġparticular , Ġwe Ġperform Ġa Ġ2 nd Ġorder Ġrandom Ġwalk Ġwith Ġtwo Ġparameters Ġp Ġand Ġq Ġthat Ġdetermine Ġthe Ġdegree Ġof Ġbreadth - first Ġsampling Ġand Ġdepth - first Ġsampling . ĠSpecifically Ġas ĠGro ver Ġand ĠLes k ove c Ġ( ĠB IB REF 30 Ġ) Ġdescribed , Ġp Ġcontrols Ġthe Ġlikelihood Ġof Ġimmediately Ġrevis iting Ġa Ġnode Ġin Ġthe Ġwalk Ġwhereas Ġq Ġcontrols Ġwhether Ġthe Ġwalk Ġis Ġbiased Ġtowards Ġnodes Ġclose Ġto Ġstarting Ġnode Ġor Ġaway Ġfrom Ġstarting Ġnode . ĠWe Ġemphasize Ġthat Ġwhile Ġnode 2 vec Ġhas Ġadditional Ġsteps Ġto Ġlearn Ġdense Ġcontinuous Ġrepresentations Ġof Ġnodes , Ġwe Ġadopt Ġonly Ġits Ġfirst Ġcomponent Ġto Ġgenerate Ġa Ġcorpus Ġof Ġrandom Ġwalks Ġrepresenting Ġpaths Ġin Ġknowledge Ġgraphs . </s> D ol ores : ĠLear ner </s> Having Ġgenerated Ġa Ġset Ġof Ġpaths Ġon Ġknowledge Ġgraphs Ġrepresenting Ġlocal Ġcontexts Ġof Ġentities Ġand Ġrelations , Ġwe Ġare Ġnow Ġready Ġto Ġutilize Ġthe Ġmachinery Ġof Ġlanguage Ġmodeling Ġusing Ġdeep Ġneural Ġnetworks Ġto Ġlearn ĠDol ores Ġembed d ings . </s> While Ġtraditional Ġlanguage Ġmodels Ġmodel Ġa Ġsentence Ġas Ġa Ġsequence Ġof Ġwords , Ġwe Ġadopt Ġthe Ġsame Ġmachinery Ġto Ġmodel Ġknowledge Ġgraph Ġembed d ings Ġas Ġfollows : Ġ( a ) ĠA Ġword Ġis Ġan Ġ( entity , Ġrelation ) Ġtuple , Ġ( b ) Ġwe Ġmodel Ġa Ġsentence Ġas Ġa Ġpath Ġconsisting Ġof Ġ( entity , Ġrelation ) Ġtu ples . ĠNote Ġthat Ġwe Ġhave Ġalready Ġestablished Ġhow Ġto Ġgenerate Ġsuch Ġpaths Ġfrom Ġthe Ġknowledge Ġgraph Ġusing Ġthe Ġpath Ġgenerator Ġcomponent . </s> Given Ġsuch Ġpaths , Ġwe Ġwould Ġlike Ġto Ġmodel Ġthe Ġprobability Ġof Ġan Ġentity - relation Ġpair Ġgiven Ġthe Ġhistory Ġand Ġfuture Ġcontext Ġby Ġa ĠBi - D irection al ĠLong ĠShort - Term ĠMemory Ġnetwork . ĠIn Ġparticular , Ġthe Ġforward Ġdirection ĠL ST M Ġmodels : Ġ$ Ġ\ text { Pr } ([ e _ 1 , r _ 1 ], Ġ[ e _ 2 , r _ 2 ], Ġ\ cd ots Ġ, Ġ[ e _ N , Ġr _ N ]) Ġ= Ġ$ </s> $$ \ pro d Ġ_ { t = 1 } ^{ N } Ġ\ text { Pr } ( Ġ[ e _ t , Ġr _ t ] Ġ\ mid Ġ[ e _ 1 , Ġr _ 1 ], Ġ[ e _ 2 , Ġr _ 2 ], Ġ\ cd ots Ġ, Ġ[ e _{ t - 1 }, Ġr _{ t - 1 } ] Ġ). $$ Ġ Ġ Ġ( E q . Ġ17 ) </s> Similarly , Ġthe Ġbackward Ġdirection ĠL ST M Ġmodels : Ġ$ Ġ\ text { Pr } ([ e _ 1 , r _ 1 ], Ġ[ e _ 2 , r _ 2 ], Ġ\ cd ots Ġ, Ġ[ e _ N , Ġr _ N ]) Ġ= Ġ$ </s> $$ \ pro d Ġ_ { t = 1 } ^{ N } Ġ\ text { Pr } ( Ġ[ e _ t , Ġr _ t ] Ġ\ mid Ġ[ e _{ t + 1 }, Ġr _{ t + 1 } ], Ġ\ cd ots Ġ, Ġ[ e _{ N }, Ġr _{ N } ] Ġ). $$ Ġ Ġ Ġ( E q . Ġ18 ) </s> Figure Ġ2 Ġillustrates Ġthis Ġsuccinct ly . ĠAt Ġeach Ġtime - step Ġt , Ġwe Ġdeal Ġwith Ġan Ġentity - relation Ġpair Ġ[ Ġ$ e _ t $ Ġ, Ġ$ r _ t $ Ġ]. ĠWe Ġfirst Ġmap Ġone - hot Ġvectors Ġof Ġthe Ġ$ e _ t $ Ġand Ġ$ r _ t $ Ġto Ġan Ġembed ding Ġlayer , Ġconc aten ate Ġthem Ġto Ġobtain Ġcontext - independent Ġrepresentations Ġwhich Ġare Ġthen Ġpassed Ġthrough ĠL Ġlayers Ġof Ġa ĠBi - D irection al ĠL ST M . ĠEach Ġlayer Ġof ĠL ST M Ġoutputs Ġthe Ġpair 's Ġcontext - dependent Ġrepresentation Ġ$\ over right arrow { h _{ t , i }} $ Ġ, Ġ$\ over left arrow { h _{ t , i }} $ Ġ, Ġwhere Ġi = 1 , Ġ2 , Ġ$\ cd ots Ġ$ Ġ, ĠL . ĠFinally , Ġthe Ġoutput Ġof Ġthe Ġtop Ġlayer Ġof ĠL ST M , Ġ$\ over right arrow { h _{ t , L }} $ Ġ, Ġ$\ over left arrow { h _{ t , L }} $ Ġ, Ġis Ġused Ġto Ġpredict Ġthe Ġnext Ġpair Ġ[ Ġ$ e _{ t + 1 } $ Ġ, Ġ$ r _ t $ 0 Ġ] Ġand Ġ[ Ġ$ r _ t $ 1 Ġ, Ġ$ r _ t $ 2 Ġ] Ġrespectively Ġusing Ġa Ġsoft max Ġlayer . ĠForm ally , Ġwe Ġjointly Ġmaximize Ġthe Ġlog Ġlikelihood Ġof Ġthe Ġforward Ġand Ġbackward Ġdirections : </s> $$ \ begin { split } Ġ\ sum Ġ_ { t = 1 } ^{ N }\ log Ġ\ text { Pr } ([ e _ t , Ġr _ t ] \ mid Ġ[ e _ 1 , Ġr _ 1 ], \ cd ots Ġ, [ e _{ t - 1 }, Ġr _{ t - 1 } ]; \ math bf Ġ{\ The ta Ġ_ { F }} )+ \\ Ġ\ sum Ġ_ { t = 1 } ^{ N }\ log Ġ\ text { Pr } ([ e _ t , Ġr _ t ] \ mid Ġ[ e _{ t + 1 }, Ġr _{ t + 1 } ], \ cd ots Ġ, [ e _{ N }, Ġr _{ N } ]; \ math bf Ġ{\ The ta Ġ_ { B }} ), Ġ\ end { split } $$ Ġ Ġ Ġ( E q . Ġ19 ) </s> where Ġ$\ math bf Ġ{\ The ta Ġ_ { F }} $ Ġ= Ġ$( \ the ta Ġ_ x , Ġ\ over right arrow {\ the ta Ġ_ { L ST M } }, Ġ\ the ta Ġ_ s ) $ Ġcorresponds Ġto Ġthe Ġparameters Ġof Ġthe Ġembed ding Ġlayer , Ġforward - direction ĠL ST M Ġand Ġthe Ġsoft max Ġlayer Ġrespectively . ĠSimilarly Ġ$\ math bf Ġ{\ The ta Ġ_ { B }} $ Ġ= Ġ$( \ the ta Ġ_ x , Ġ\ over left arrow {\ the ta Ġ_ { L ST M } }, Ġ\ the ta Ġ_ s ) $ Ġcorresponds Ġto Ġthe Ġsimilar Ġset Ġof Ġparameters Ġfor Ġthe Ġbackward Ġdirection . ĠSpecifically , Ġnote Ġthat Ġwe Ġshare Ġthe Ġparameters Ġfor Ġthe Ġembed ding Ġand Ġsoft max Ġlayer Ġacross Ġboth Ġdirections . ĠWe Ġmaximize ĠEqu ation Ġ19 Ġby Ġtraining Ġthe ĠBi - direction al ĠL ST Ms Ġusing Ġback - prop ag ation Ġthrough Ġtime . </s> After Ġhaving Ġestimated Ġthe Ġparameters Ġof Ġthe ĠDol ores Ġlear ner , Ġwe Ġnow Ġextract Ġthe Ġcontext - independent Ġand Ġcontext - dependent Ġrepresentations Ġfor Ġeach Ġentity Ġand Ġrelation Ġand Ġcombine Ġthem Ġto Ġobtain ĠDol ores Ġembed d ings . ĠMore Ġspecifically , ĠDol ores Ġembed d ings Ġare Ġtask Ġspecific Ġcombination Ġof Ġthe Ġcontext - dependent Ġand Ġcontext - independent Ġrepresentations Ġlearned Ġby Ġour Ġlear ner . ĠNote Ġthat Ġour Ġlear ner Ġ( which Ġis Ġan Ġ$ L $ Ġ- layer ĠBi - D irection al ĠL ST M ) Ġcomp utes Ġa Ġset Ġof Ġ$ 2 L Ġ+ Ġ1 $ Ġrepresentations Ġfor Ġeach Ġentity - relation Ġpair Ġwhich Ġwe Ġdenote Ġby : Ġ$ ĠR _ t Ġ= Ġ[ Ġx _ t , Ġ\ over right arrow { h _{ t , i } }, Ġ\ over left arrow { h _{ t , i }} Ġ\ mid Ġi Ġ= Ġ1 , Ġ2 , Ġ\ cd ots Ġ, Ġ\ text it Ġ{ L } Ġ], Ġ$ </s> where Ġ$ x _ t $ Ġis Ġthe Ġcontext - independent Ġembed ding Ġand Ġ$\ over right arrow { h _{ t , i } }, Ġ\ over left arrow { h _{ t , i }} $ Ġcorrespond Ġto Ġthe Ġcontext - dependent Ġembed d ings Ġfrom Ġlayer Ġ$ i $ Ġ. </s> Given Ġa Ġdownstream Ġmodel , ĠDol ores Ġlearns Ġa Ġweighted Ġlinear Ġcombination Ġof Ġthe Ġcomponents Ġof Ġ$ R _ t $ Ġto Ġyield Ġa Ġsingle Ġvector Ġfor Ġuse Ġin Ġthe Ġembed ding Ġlayer Ġof Ġthe Ġdownstream Ġmodel . ĠIn Ġparticular </s> $$ \ text tt Ġ{\ text sc Ġ{ D ol ores }} _ t Ġ= Ġ[ Ġx _ t Ġ, Ġ\ sum Ġ_ { i = 1 } ^{ L } Ġ\ lambda Ġ_ i Ġh _{ t , i } Ġ], $$ Ġ Ġ Ġ( E q . Ġ23 ) </s> where Ġwe Ġdenote Ġ$ h _{ t , i } $ Ġ= Ġ[ Ġ$\ over right arrow { h _{ t , i }} $ Ġ, Ġ$\ over left arrow { h _{ t , i }} $ Ġ] Ġand Ġ$\ lambda Ġ_ { i } $ Ġdenote Ġtask Ġspecific Ġlearn able Ġweights Ġof Ġthe Ġlinear Ġcombination . </s> While Ġit Ġis Ġobvious Ġthat Ġour Ġembed d ings Ġcan Ġbe Ġused Ġas Ġfeatures Ġfor Ġnew Ġpredictive Ġmodels , Ġit Ġis Ġalso Ġvery Ġeasy Ġto Ġincorporate Ġour Ġlearned ĠDol ores Ġembed d ings Ġinto Ġexisting Ġpredictive Ġmodels Ġon Ġknowledge Ġgraphs . ĠThe Ġonly Ġrequirement Ġis Ġthat Ġthe Ġmodel Ġaccepts Ġas Ġinput , Ġan Ġembed ding Ġlayer Ġ( for Ġentities Ġand Ġrelations ). ĠIf Ġa Ġmodel Ġfulf ills Ġthis Ġrequirement Ġ( which Ġa Ġlarge Ġnumber Ġof Ġneural Ġmodels Ġon Ġknowledge Ġgraphs Ġdo ), Ġwe Ġcan Ġjust Ġuse ĠDol ores Ġembed d ings Ġas Ġa Ġdrop - in Ġreplacement . ĠWe Ġjust Ġinitialize Ġthe Ġcorresponding Ġembed ding Ġlayer Ġwith ĠDol ores Ġembed d ings . ĠIn Ġour Ġevaluation Ġbelow , Ġwe Ġshow Ġhow Ġto Ġimprove Ġseveral Ġstate - of - the - art Ġmodels Ġon Ġvarious Ġtasks Ġsimply Ġby Ġincorporating ĠDol ores Ġas Ġa Ġdrop - in Ġreplacement Ġto Ġthe Ġoriginal Ġembed ding Ġlayer . </s> Exper iments </s> We Ġevaluate ĠDol ores Ġon Ġa Ġset Ġof Ġ4 Ġdifferent Ġprediction Ġtasks Ġon Ġknowledge Ġgraphs . ĠIn Ġeach Ġcase , Ġsimply Ġadding ĠDol ores Ġto Ġexisting Ġstate - of - the - art Ġmodels Ġimproves Ġthe Ġstate Ġof Ġthe Ġart Ġperformance Ġsignificantly Ġ( in Ġsome Ġcases Ġby Ġat Ġleast Ġ$ 9 . 5 \ % $ Ġ) Ġwhich Ġwe Ġshow Ġin ĠTable Ġ1 Ġ. ĠWhile Ġwe Ġprimarily Ġshow Ġthat Ġwe Ġcan Ġadvance Ġthe Ġstate - of - the - art Ġmodel Ġby Ġincorporating ĠDol ores Ġembed d ings Ġas Ġa ĠâĢ ľ drop - in âĢ Ŀ Ġreplacement , Ġfor Ġthe Ġsake Ġof Ġcomple teness , Ġwe Ġalso Ġreport Ġraw Ġnumbers Ġof Ġother Ġstrong Ġbaseline Ġmethods Ġ( like ĠTrans D , ĠTrans E , ĠKB GAN , ĠConv E , Ġand ĠConv KB ) Ġto Ġplace Ġthe Ġresults Ġin Ġcontext . ĠWe Ġemphasize Ġthat Ġour Ġmethod Ġis Ġvery Ġgeneric Ġand Ġcan Ġbe Ġused Ġto Ġimprove Ġthe Ġperformance Ġof Ġa Ġlarge Ġclass Ġof Ġknowledge Ġgraph Ġprediction Ġmodels . ĠIn Ġthe Ġremainder Ġof Ġthe Ġsection , Ġwe Ġbriefly Ġprovide Ġhigh - level Ġoverview s Ġof Ġeach Ġtask Ġand Ġsummarize Ġresults Ġfor Ġall Ġtasks Ġconsidered . </s> Exper imental ĠSettings Ġfor ĠDol ores </s> Here , Ġwe Ġoutline Ġour Ġmodel Ġsettings Ġfor Ġlearning ĠDol ores Ġembed d ings . ĠWe Ġgenerate Ġ20 Ġchains Ġfor Ġeach Ġnode Ġin Ġthe Ġknowledge Ġgraph , Ġwith Ġthe Ġlength Ġof Ġeach Ġchain Ġbeing Ġ21 Ġ( 10 Ġrelations Ġand Ġ11 Ġentities Ġappear Ġaltern ately ). ĠOur Ġmodel Ġuses ĠL Ġ= Ġ4 ĠL ST M Ġlayers Ġwith Ġ512 Ġunits Ġand Ġ32 Ġdimension Ġprojections Ġ( project ed Ġvalues Ġare Ġclipped Ġelement - wise Ġto Ġwithin Ġ$ [ - 3 , Ġ3 ] $ Ġ). ĠWe Ġuse Ġresidual Ġconnections Ġbetween Ġlayers Ġand Ġthe Ġbatch Ġsize Ġis Ġset Ġto Ġ1024 Ġduring Ġthe Ġtraining Ġprocess . ĠWe Ġtrain ĠDol ores Ġfor Ġ200 Ġepoch s Ġon Ġcorresponding Ġdatasets Ġwith Ġdrop out Ġ( with Ġthe Ġdrop out Ġprobability Ġis Ġset Ġ$ 0 . 1 $ Ġ). ĠFinally , Ġwe Ġuse ĠAdam Ġas Ġthe Ġoptim izer Ġwith Ġappropriately Ġchosen Ġlearning Ġrates Ġbased Ġon Ġa Ġvalidation Ġset . </s> E val uation ĠT asks </s> We Ġconsider Ġthree Ġtasks , Ġlink Ġprediction , Ġtriple Ġclassification , Ġand Ġpredicting Ġmissing Ġrelation Ġtypes ĠB IB REF 24 Ġ: </s> Link ĠPrediction ĠA Ġcommon Ġtask Ġfor Ġknowledge Ġgraph Ġcompletion Ġis Ġlink Ġprediction , Ġaiming Ġto Ġpredict Ġthe Ġmissing Ġentity Ġwhen Ġthe Ġother Ġtwo Ġparts Ġof Ġa Ġtriple t Ġ( h , Ġr , Ġt ) Ġare Ġgiven . ĠIn Ġother Ġwords , Ġwe Ġneed Ġto Ġpredict Ġt Ġgiven Ġ( h , Ġr ) ĠâĢĵ Ġtail - entity Ġprediction Ġor Ġpredict Ġh Ġgiven Ġ( r , Ġt ) ĠâĢĵ Ġhead Ġentity Ġprediction . ĠIn - line Ġwith Ġprior Ġwork ĠB IB REF 6 Ġ, Ġwe Ġreport Ġresults Ġon Ġlink Ġprediction Ġin Ġterms Ġof ĠMean ĠRec ip rocal ĠRank Ġ( MR R ), ĠMean ĠRank Ġ( MR ) Ġand ĠHits @ 10 Ġon Ġthe ĠFB 15 K - 237 Ġdataset Ġin Ġthe Ġfiltered Ġsetting Ġon Ġboth Ġsub - t asks : Ġ( a ) Ġhead Ġentity Ġprediction Ġand Ġ( b ) Ġtail Ġentity Ġprediction . ĠOur Ġresults Ġare Ġshown Ġin ĠTable Ġ2 Ġ. ĠNote Ġthat Ġthe Ġpresent Ġstate - of - the - art Ġmodel , ĠConv KB Ġachieves Ġan ĠMR R Ġof Ġ$( 0 . 375 , Ġ0 . 487 ) $ Ġon Ġthe Ġhead Ġand Ġtail Ġlink Ġprediction Ġtasks . ĠObs erve Ġthat Ġsimply Ġincorporating ĠDol ores Ġsignificantly Ġimproves Ġthe Ġhead Ġand Ġtail Ġentity Ġprediction Ġperformance Ġby Ġ$ 3 . 10 \ % $ Ġand Ġ$ 7 . 90 \ % $ Ġrespectively . ĠSimilar Ġimprovements Ġare Ġalso Ġobserved Ġon Ġother Ġmetrics Ġlike ĠMean ĠRank Ġ( MR : Ġlower Ġis Ġbetter ) Ġand ĠHits @ 10 Ġ( higher Ġis Ġbetter ). </s> Tri ple ĠClassification ĠThe Ġtask Ġof Ġtriple Ġclassification Ġis Ġto Ġpredict Ġwhether Ġa Ġtriple Ġ( h , Ġr , Ġt ) Ġis Ġcorrect Ġor Ġnot . ĠTriple Ġclassification Ġis Ġa Ġbinary Ġclassification Ġtask Ġwidely Ġexplored Ġby Ġprevious Ġwork ĠB IB REF 2 Ġ, ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ. ĠSince Ġevaluation Ġof Ġclassification Ġneeds Ġnegative Ġtri ples , Ġwe Ġchoose ĠW N 11 Ġand ĠFB 13 , Ġtwo Ġbenchmark Ġdatasets Ġand Ġreport Ġthe Ġresults Ġof Ġour Ġevaluation Ġin ĠTable Ġ3 Ġ. ĠWe Ġnote Ġthat Ġthe Ġpresent Ġstate Ġof Ġthe Ġart Ġis Ġthe ĠConv KB Ġmodel . ĠWhen Ġwe Ġadd ĠDol ores Ġto Ġthe ĠConv KB Ġmodel Ġwith Ġour Ġembed d ings , Ġobserve Ġthat Ġwe Ġimprove Ġthe Ġaverage Ġperformance Ġof Ġthe Ġstate - of - the - art Ġmodel ĠConv KB Ġslightly Ġby Ġ$ 0 . 20 $ Ġpoints Ġ( from Ġ$ 88 . 20 $ Ġto Ġ$ 88 . 40 $ Ġ). ĠWe Ġbelieve Ġthe Ġimprovement Ġachieved Ġby Ġadding ĠDol ores Ġis Ġsmaller Ġin Ġterms Ġof Ġabsolute Ġsize Ġbecause Ġthe Ġstate - of - the - art Ġmodel Ġalready Ġhas Ġexcellent Ġperformance Ġon Ġthis Ġtask Ġ( Ġ$ 88 . 20 $ Ġ) Ġsuggesting Ġa Ġmuch Ġslower Ġimprovement Ġcurve . </s> Missing ĠRel ation ĠTypes ĠThe Ġgoal Ġof Ġthe Ġthird Ġtask Ġis Ġto Ġreason Ġon Ġthe Ġpaths Ġconnecting Ġan Ġentity Ġpair Ġto Ġpredict Ġmissing Ġrelation Ġtypes . ĠWe Ġfollow ĠDas Ġet Ġal . Ġ( ĠB IB REF 24 Ġ) Ġand Ġuse Ġthe Ġsame Ġdataset Ġreleased Ġby ĠNe el ak ant an , ĠRoth , Ġand ĠMcC all um Ġ( ĠB IB REF 23 Ġ) Ġwhich Ġis Ġa Ġsubset Ġof ĠFree Base Ġenriched Ġwith Ġinformation Ġfrom ĠCl ue Web . ĠThe Ġdataset Ġconsists Ġof Ġa Ġset Ġof Ġtri ples Ġ( Ġ$ e _ 1 $ Ġ, Ġr , Ġ$ e _ 2 $ Ġ) Ġand Ġthe Ġset Ġof Ġpaths Ġconnecting Ġthe Ġentity Ġpair Ġ( Ġ$ e _ 1 $ Ġ, Ġ$ e _ 2 $ Ġ) Ġin Ġthe Ġknowledge Ġgraph . ĠThese Ġtri ples Ġare Ġcollected Ġfrom ĠCl ue Web Ġby Ġconsidering Ġsentences Ġthat Ġcontain Ġthe Ġentity Ġpair Ġin ĠFree base . ĠNe el ak ant an Ġet Ġal . Ġ( ĠB IB REF 23 Ġ) Ġinfer Ġthe Ġrelation Ġtype Ġby Ġexamining Ġthe Ġphrase Ġbetween Ġtwo Ġentities . ĠWe Ġuse Ġthe Ġsame Ġevaluation Ġcriterion Ġas Ġused Ġby ĠDas Ġet Ġal . Ġ( ĠB IB REF 24 Ġ) Ġand Ġreport Ġour Ġresults Ġin ĠTable Ġ4 Ġ. ĠNote Ġthat Ġthe Ġcurrent Ġstate - of - the - art Ġmodel Ġfrom ĠDas Ġet Ġal . Ġ( ĠB IB REF 24 Ġ) Ġyields Ġa Ġscore Ġof Ġ$ 71 . 74 $ Ġ. ĠAdding ĠDol ores Ġto Ġthe Ġmodel Ġimproves Ġthe Ġscore Ġto Ġ$ 74 . 42 $ Ġyielding Ġa Ġ$\ text bf Ġ{ 9 . 5 \ % } $ Ġimprovement . </s> Alt ogether , Ġviewing Ġthe Ġresults Ġon Ġvarious Ġtasks Ġhol istically , Ġwe Ġconclude Ġthat Ġsimply Ġincorporating ĠDol ores Ġinto Ġexisting Ġstate - of - the - art Ġmodels Ġimproves Ġtheir Ġperformance Ġand Ġadvances Ġthe Ġstate Ġof Ġthe Ġart Ġon Ġeach Ġof Ġthese Ġtasks Ġand Ġsuggests Ġthat Ġour Ġembed d ings Ġcan Ġbe Ġeffective Ġin Ġyielding Ġperformance Ġgains Ġon Ġa Ġvariety Ġof Ġpredictive Ġtasks . </s> Error ĠAnalysis </s> In Ġthis Ġsection , Ġwe Ġconduct Ġan Ġanalysis Ġof Ġthe Ġpredictions Ġmade Ġwhen Ġusing ĠDol ores Ġon Ġthe Ġlink Ġprediction Ġtasks Ġto Ġgain Ġinsights Ġinto Ġour Ġlearned Ġembed d ings . ĠTo Ġdo Ġso , Ġwe Ġgroup Ġthe Ġtest Ġtri ples Ġby Ġthe Ġfirst Ġcomponent Ġof Ġtheir Ġrelation Ġ( the Ġmost Ġabstract Ġconcept Ġor Ġlevel ) Ġand Ġcompute Ġthe Ġmean Ġrank Ġof Ġthe Ġtail Ġentity Ġoutput Ġover Ġeach Ġgroup . ĠWe Ġcompare Ġthis Ġmetric Ġagainst Ġwhat Ġa Ġsimple Ġbaseline Ġlike ĠTrans E Ġob tains . ĠThis Ġenables Ġus Ġto Ġidentify Ġoverall Ġstatistical Ġpatterns Ġthat Ġdistinguish ĠDol ores Ġembed d ings Ġfrom Ġbaseline Ġembed d ings Ġlike ĠTrans E , Ġwhich Ġin Ġturn Ġboosts Ġthe Ġperformance Ġof Ġlink Ġprediction . </s> Figure Ġ3 Ġshows Ġthe Ġrelation Ġcategories Ġfor Ġwhich ĠDol ores Ġperformed Ġthe Ġbest Ġand Ġthe Ġworst Ġrelative Ġto Ġbaseline ĠTrans E Ġembed d ings Ġin Ġterms Ġof Ġmean Ġrank Ġ( lower Ġis Ġbetter ). ĠIn Ġparticular , ĠFigure Ġ3 Ġshows Ġthe Ġcategories Ġwhere ĠDol ores Ġperforms Ġthe Ġbest Ġand Ġreveals Ġcategories Ġlike Ġfood , Ġuser , Ġo lymp ics , Ġorganization . ĠNote Ġfor Ġexample , Ġthat Ġfor Ġtri ples Ġbelonging Ġunder Ġthe Ġfood Ġrelation , ĠDol ores Ġoutper forms Ġbaseline Ġby ĠTrans E Ġby Ġa Ġfactor Ġof Ġ4 Ġin Ġterms Ġof Ġmean Ġrank . ĠTo Ġillustrate Ġthis Ġpoint , Ġwe Ġshow Ġa Ġfew Ġinstances Ġof Ġsuch Ġtri ples Ġbelow : </s> ( A Ġserious Ġman , Ġfilm - release - region , ĠHong ĠKong ) </s> ( C ab aret , Ġfilm - genre , ĠFilm ĠAdapt ation ) </s> ( Lou ĠCost ello , Ġpeople - prof ession , ĠComed ian ) </s> Note Ġthat Ġwhen Ġthe Ġhead Ġentity Ġis Ġa Ġvery Ġspecific Ġentity Ġlike ĠLouis ĠCost ello , Ġour Ġmethod Ġis Ġvery Ġaccurate Ġat Ġpredicting Ġthe Ġcorrect Ġtail Ġentity Ġ( in Ġthis Ġcase ĠComed ian ). ĠTrans E , Ġon Ġthe Ġother Ġhand , Ġmakes Ġvery Ġpoor Ġpredictions Ġon Ġsuch Ġcases . ĠWe Ġbelieve Ġthat Ġour Ġmethod Ġis Ġable Ġto Ġmodel Ġsuch Ġcases Ġbetter Ġbecause Ġour Ġembed d ings , Ġespecially Ġfor Ġsuch Ġspecific Ġentities , Ġhave Ġcaptured Ġthe Ġrich Ġcontext Ġassociated Ġwith Ġthem Ġfrom Ġentire Ġpaths . </s> In Ġcontrast , ĠFigure Ġ3 Ġshows Ġthe Ġrelation Ġcategories Ġthat ĠDol ores Ġperforms Ġthe Ġworst Ġrelative Ġto ĠTrans E . ĠWe Ġnote Ġthat Ġthese Ġcorrespond Ġto Ġvery Ġbroad Ġrelation Ġcategories Ġlike Ġcommon , base , media _ common Ġetc . ĠWe Ġlist Ġa Ġfew Ġtri ples Ġbelow Ġto Ġillustrate Ġthis Ġpoint : </s> ( Psych ology , Ġstudents Ġmajor ing , ĠY anni ) </s> ( De cca ĠRecords , Ġmusic - record - label - art </s> ist , ĠJes sey e ĠNorman ) </s> ( Prior ity ĠRecords , Ġmusic - record - label - </s> artist , ĠCar ole ĠKing ) </s> Note Ġthat Ġsuch Ġinstances Ġare Ġindeed Ġvery Ġdifficult . ĠFor Ġinstance , Ġgiven Ġa Ġvery Ġgeneric Ġentity Ġlike ĠPsychology Ġit Ġis Ġvery Ġdifficult Ġto Ġguess Ġthat ĠY anni Ġwould Ġbe Ġthe Ġexpected Ġtail Ġentity . ĠAlt ogether , Ġour Ġmethod Ġis Ġable Ġto Ġbetter Ġmodel Ġtri ples Ġwhere Ġthe Ġhead Ġentity Ġis Ġmore Ġspecific Ġcompared Ġto Ġhead Ġentities Ġwhich Ġare Ġvery Ġbroad Ġand Ġgeneral . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġintroduce ĠDol ores , Ġa Ġnew Ġparadigm Ġof Ġlearning Ġknowledge Ġgraph Ġembed d ings Ġwhere Ġwe Ġlearn Ġnot Ġonly Ġcontextual Ġindependent Ġembed d ings Ġof Ġentities Ġand Ġrelations Ġbut Ġalso Ġmultiple Ġcontext - dependent Ġembed d ings Ġeach Ġcapturing Ġa Ġdifferent Ġlayer Ġof Ġabstraction . ĠWe Ġdemonstrate Ġthat Ġby Ġleveraging Ġconnections Ġbetween Ġthree Ġseemingly Ġdistinct Ġfields Ġnamely : Ġ( a ) Ġlarge - scale Ġnetwork Ġanalysis , Ġ( b ) Ġnatural Ġlanguage Ġprocessing Ġand Ġ( c ) Ġknowledge Ġgraphs Ġwe Ġare Ġable Ġto Ġlearn Ġrich Ġknowledge Ġgraph Ġembed d ings Ġthat Ġare Ġdeep Ġand Ġcontextual ized Ġin Ġcontrast Ġto Ġprior Ġmodels Ġthat Ġare Ġtypically Ġshallow . ĠMoreover , Ġour Ġlearned Ġembed d ings Ġcan Ġbe Ġeasily Ġincorporated Ġinto Ġexisting Ġknowledge Ġgraph Ġprediction Ġmodels Ġto Ġimprove Ġtheir Ġperformance . ĠSpecifically , Ġwe Ġshow Ġthat Ġour Ġembed d ings Ġare Ġnot Ġonly Ġa ĠâĢ ľ drop - in âĢ Ŀ Ġreplacement Ġfor Ġexisting Ġmodels Ġthat Ġuse Ġembed ding Ġlayers Ġbut Ġalso Ġsignificantly Ġimprove Ġthe Ġstate - of - the - art Ġmodels Ġon Ġa Ġvariety Ġof Ġtasks , Ġsometimes Ġby Ġalmost Ġ$ 9 . 5 \ % $ Ġ. ĠFurthermore , Ġour Ġmethod Ġis Ġinherently Ġonline Ġand Ġscales Ġto Ġlarge Ġdata - sets . </s> Our Ġwork Ġalso Ġnaturally Ġsuggests Ġnew Ġdirections Ġof Ġinvestigation Ġand Ġresearch Ġinto Ġknowledge Ġgraph Ġembed d ings . ĠOne Ġavenue Ġof Ġresearch Ġis Ġto Ġinvestigate Ġthe Ġutility Ġof Ġeach Ġlayer 's Ġentity Ġand Ġrelation Ġembed d ings Ġin Ġspecific Ġlearning Ġtasks . ĠAs Ġwas Ġnoted Ġby Ġresearch Ġin Ġcomputer Ġvision , Ġdeep Ġrepresentations Ġlearned Ġon Ġone Ġdataset Ġare Ġeffective Ġand Ġvery Ġuseful Ġin Ġtransfer - learning Ġtasks ĠB IB REF 33 Ġ. ĠA Ġnatural Ġline Ġof Ġinvestigation Ġthus Ġrevolves Ġaround Ġprecisely Ġquant ifying Ġthe Ġeffectiveness Ġof Ġthese Ġlearned Ġembed d ings Ġacross Ġmodels Ġand Ġdata - sets . ĠLastly , Ġit Ġwould Ġbe Ġinteresting Ġto Ġsee Ġif Ġsuch Ġknowledge Ġgraph Ġembed d ings Ġcan Ġbe Ġused Ġin Ġconjunction Ġwith Ġnatural Ġlanguage Ġprocessing Ġmodels Ġused Ġfor Ġrelation Ġextraction Ġand Ġnamed Ġentity Ġrecognition Ġfrom Ġraw Ġtextual Ġdata . </s> Finally , Ġin Ġa Ġbroader Ġperspective , Ġour Ġwork Ġintroduces Ġtwo Ġnew Ġparad ig ms Ġof Ġmodeling Ġknowledge Ġgraphs . ĠFirst , Ġrather Ġthan Ġview Ġa Ġknowledge Ġgraph Ġas Ġa Ġcollection Ġof Ġtri ples , Ġwe Ġview Ġit Ġas Ġa Ġset Ġof Ġpaths Ġbetween Ġentities . ĠThese Ġpaths Ġcan Ġbe Ġrepresented Ġas Ġa Ġcollection Ġof Ġtrunc ated Ġrandom Ġwalks Ġover Ġthe Ġknowledge Ġgraph Ġand Ġencode Ġrich Ġcontextual Ġinformation Ġbetween Ġentities Ġand Ġrelations . ĠSecond , Ġdeparting Ġfrom Ġthe Ġhitherto Ġwell - established Ġparadigm Ġof Ġmapping Ġentities Ġand Ġrelations Ġto Ġvectors Ġin Ġ$\ math cal Ġ{ R } ^{ d } $ Ġvia Ġa Ġmapping Ġfunction , Ġwe Ġlearn Ġmultiple Ġrepresentations Ġfor Ġentities Ġand Ġrelations Ġdetermined Ġby Ġthe Ġnumber Ġof Ġlayers Ġin Ġa Ġdeep Ġneural Ġnetwork . ĠThis Ġenables Ġus Ġto Ġlearn Ġknowledge Ġgraph Ġembed d ings Ġthat Ġcapture Ġdifferent Ġlayers Ġof Ġabstraction ĠâĢĵ Ġboth Ġcontext - independent Ġand Ġcontext - dependent Ġallowing Ġfor Ġthe Ġdevelopment Ġof Ġvery Ġpowerful Ġprediction Ġmodels Ġto Ġyield Ġsuperior Ġperformance Ġon Ġa Ġvariety Ġof Ġprediction Ġtasks .