In case of Freebase knowledge base, LiLi model had better F1 score than the single model by 0.20 , 0.01, 0.159 for kwn, unk, and all test Rel type.  The values for WordNet are 0.25, 0.1, 0.2. 

ABSTRACTIVE
Unanswerable
NONE
Baselines. As none of the existing KBC methods can solve the OKBC problem, we choose various versions of LiLi as baselines. Single: Version of LiLi where we train a single prediction model INLINEFORM0 for all test relations. Sep: We do not transfer (past learned) weights for initializing INLINEFORM0 , i.e., we disable LL. F-th): Here, we use a fixed prediction threshold 0.5 instead of relation-specific threshold INLINEFORM0 . BG: The missing or connecting links (when the user does not respond) are filled with “@-RelatedTo-@" blindly, no guessing mechanism. w/o PTS: LiLi does not ask for additional clues via past task selection for skillset improvement. Evaluation-I: Strategy Formulation Ability. Table 5 shows the list of inference strategies formulated by LiLi for various INLINEFORM0 and INLINEFORM1 , which control the strategy formulation of LiLi. When INLINEFORM2 , LiLi cannot interact with user and works like a closed-world method. Thus, INLINEFORM3 drops significantly (0.47). When INLINEFORM4 , i.e. with only one interaction per query, LiLi acquires knowledge well for instances where either of the entities or relation is unknown. However, as one unknown entity may appear in multiple test triples, once the entity becomes known, LiLi doesn’t need to ask for it again and can perform inference on future triples causing significant increase in INLINEFORM5 (0.97). When INLINEFORM6 , LiLi is able to perform inference on all instances and INLINEFORM7 becomes 1. For INLINEFORM8 , LiLi uses INLINEFORM9 only once (as only one MLQ satisfies INLINEFORM10 ) compared to INLINEFORM11 . In summary, LiLi’s RL-model can effectively formulate query-specific inference strategies (based on specified parameter values). Evaluation-II: Predictive Performance. Table 6 shows the comparative performance of LiLi with baselines. To judge the overall improvements, we performed paired t-test considering +ve F1 scores on each relation as paired data. Considering both KBs and all relation types, LiLi outperforms Sep with INLINEFORM12 . If we set INLINEFORM13 (training with very few clues), LiLi outperforms Sep with INLINEFORM14 on Freebase considering MCC. Thus, the lifelong learning mechanism is effective in transferring helpful knowledge. Single model performs better than Sep for unknown relations due to the sharing of knowledge (weights) across tasks. However, for known relations, performance drops because, as a new relation arrives to the system, old weights get corrupted and catastrophic forgetting occurs. For unknown relations, as the relations are evaluated just after training, there is no chance for catastrophic forgetting. The performance improvement ( INLINEFORM15 ) of LiLi over F-th on Freebase signifies that the relation-specific threshold INLINEFORM16 works better than fixed threshold 0.5 because, if all prediction values for test instances lie above (or below) 0.5, F-th predicts all instances as +ve (-ve) which degrades its performance. Due to the utilization of contextual similarity (highly correlated with class labels) of entity-pairs, LiLi’s guessing mechanism works better ( INLINEFORM17 ) than blind guessing (BG). The past task selection mechanism of LiLi also improves its performance over w/o PTS, as it acquires more clues during testing for poorly performed tasks (evaluated on validation set). For Freebase, due to a large number of past tasks [9 (25% of 38)], the performance difference is more significant ( INLINEFORM18 ). For WordNet, the number is relatively small [3 (25% of 14)] and hence, the difference is not significant. FLOAT SELECTED: Table 6: Comparison of predictive performance of various versions of LiLi [kwn = known, unk = unknown, all = overall].
1802.06024
Introduction </s> Chat bots Ġsuch Ġas Ġdialog Ġand Ġquestion - ans w ering Ġsystems Ġhave Ġa Ġlong Ġhistory Ġin ĠAI Ġand Ġnatural Ġlanguage Ġprocessing . ĠEarly Ġsuch Ġsystems Ġwere Ġmostly Ġbuilt Ġusing Ġmarkup Ġlanguages Ġsuch Ġas ĠA IM L , Ġhand crafted Ġconversation Ġgeneration Ġrules , Ġand / or Ġinformation Ġretrieval Ġtechniques ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ. ĠRecent Ġneural Ġconversation Ġmodels ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġare Ġeven Ġable Ġto Ġperform Ġopen - ended Ġconversations . ĠHowever , Ġsince Ġthey Ġdo Ġnot Ġuse Ġexplicit Ġknowledge Ġbases Ġand Ġdo Ġnot Ġperform Ġinference , Ġthey Ġoften Ġsuffer Ġfrom Ġgeneric Ġand Ġdull Ġresponses ĠB IB REF 5 Ġ, ĠB IB REF 7 Ġ. ĠMore Ġrecently , ĠB IB REF 8 Ġand ĠB IB REF 9 Ġproposed Ġto Ġuse Ġknowledge Ġbases Ġ( KB s ) Ġto Ġhelp Ġgenerate Ġresponses Ġfor Ġknowledge - ground ed Ġconversation . ĠHowever , Ġone Ġmajor Ġweakness Ġof Ġall Ġexisting Ġchat Ġsystems Ġis Ġthat Ġthey Ġdo Ġnot Ġexplicitly Ġor Ġimplicitly Ġlearn Ġnew Ġknowledge Ġin Ġthe Ġconversation Ġprocess . ĠThis Ġseriously Ġlimits Ġthe Ġscope Ġof Ġtheir Ġapplications . ĠIn Ġcontrast , Ġwe Ġhumans Ġconstantly Ġlearn Ġnew Ġknowledge Ġin Ġour Ġconversations . ĠEven Ġif Ġsome Ġexisting Ġsystems Ġcan Ġuse Ġvery Ġlarge Ġknowledge Ġbases Ġeither Ġharvested Ġfrom Ġa Ġlarge Ġdata Ġsource Ġsuch Ġas Ġthe ĠWeb Ġor Ġbuilt Ġmanually , Ġthese ĠKB s Ġstill Ġmiss Ġa Ġlarge Ġnumber Ġof Ġfacts Ġ( knowledge ) ĠB IB REF 10 Ġ. ĠIt Ġis Ġthus Ġimportant Ġfor Ġa Ġchat bot Ġto Ġcontinuously Ġlearn Ġnew Ġknowledge Ġin Ġthe Ġconversation Ġprocess Ġto Ġexpand Ġits ĠKB Ġand Ġto Ġimprove Ġits Ġconversation Ġability . </s> In Ġrecent Ġyears , Ġresearchers Ġhave Ġstudied Ġthe Ġproblem Ġof ĠKB Ġcompletion , Ġi . e ., Ġinfer ring Ġnew Ġfacts Ġ( knowledge ) Ġautomatically Ġfrom Ġexisting Ġfacts Ġin Ġa ĠKB . ĠKB Ġcompletion Ġ( K BC ) Ġis Ġdefined Ġas Ġa Ġbinary Ġclassification Ġproblem : ĠGiven Ġa Ġquery Ġtriple , Ġ( ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġ), Ġwe Ġwant Ġto Ġpredict Ġwhether Ġthe Ġsource Ġentity ĠIN LINE FORM 3 Ġand Ġtarget Ġentity ĠIN LINE FORM 4 Ġcan Ġbe Ġlinked Ġby Ġthe Ġrelation ĠIN LINE FORM 5 Ġ. ĠHowever , Ġexisting Ġapproaches ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġsolve Ġthis Ġproblem Ġunder Ġthe Ġclosed - world Ġassumption , Ġi . e ., ĠIN LINE FORM 6 Ġ, ĠIN LINE FORM 7 Ġand ĠIN LINE FORM 8 Ġare Ġall Ġknown Ġto Ġexist Ġin Ġthe ĠKB . ĠThis Ġis Ġa Ġmajor Ġweakness Ġbecause Ġit Ġmeans Ġthat Ġno Ġnew Ġknowledge Ġor Ġfacts Ġmay Ġcontain Ġunknown Ġentities Ġor Ġrelations . ĠDue Ġto Ġthis Ġlimitation , ĠK BC Ġis Ġclearly Ġnot Ġsufficient Ġfor Ġknowledge Ġlearning Ġin Ġconversations Ġbecause Ġin Ġa Ġconversation , Ġthe Ġuser Ġcan Ġsay Ġanything , Ġwhich Ġmay Ġcontain Ġentities Ġand Ġrelations Ġthat Ġare Ġnot Ġalready Ġin Ġthe ĠKB . </s> In Ġthis Ġpaper , Ġwe Ġremove Ġthis Ġassumption Ġof ĠK BC , Ġand Ġallow Ġall ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġto Ġbe Ġunknown . ĠWe Ġcall Ġthe Ġnew Ġproblem Ġopen - world Ġknowledge Ġbase Ġcompletion Ġ( OK BC ). ĠOK BC Ġgeneral izes ĠK BC . ĠBelow , Ġwe Ġshow Ġthat Ġsolving ĠOK BC Ġnaturally Ġprovides Ġthe Ġground Ġfor Ġknowledge Ġlearning Ġand Ġinference Ġin Ġconversations . ĠIn Ġessence , Ġwe Ġformulate Ġan Ġabstract Ġproblem Ġof Ġknowledge Ġlearning Ġand Ġinference Ġin Ġconversations Ġas Ġa Ġwell - defined ĠOK BC Ġproblem Ġin Ġthe Ġinteractive Ġsetting . </s> From Ġthe Ġperspective Ġof Ġknowledge Ġlearning Ġin Ġconversations , Ġessentially Ġwe Ġcan Ġextract Ġtwo Ġkey Ġtypes Ġof Ġinformation , Ġtrue Ġfacts Ġand Ġqueries , Ġfrom Ġthe Ġuser Ġutter ances . ĠQu eries Ġare Ġfacts Ġwhose Ġtruth Ġvalues Ġneed Ġto Ġbe Ġdetermined . ĠNote Ġthat Ġwe Ġdo Ġnot Ġstudy Ġfact Ġor Ġrelation Ġextraction Ġin Ġthis Ġpaper Ġas Ġthere Ġis Ġan Ġextensive Ġwork Ġon Ġthe Ġtopic . Ġ( 1 ) ĠFor Ġa Ġtrue Ġfact , Ġwe Ġwill Ġincorporate Ġit Ġinto Ġthe ĠKB . ĠHere Ġwe Ġneed Ġto Ġmake Ġsure Ġthat Ġit Ġis Ġnot Ġalready Ġin Ġthe ĠKB , Ġwhich Ġinvolves Ġrelation Ġresolution Ġand Ġentity Ġlinking . ĠAfter Ġa Ġfact Ġis Ġadded Ġto Ġthe ĠKB , Ġwe Ġmay Ġpredict Ġthat Ġsome Ġrelated Ġfacts Ġinvolving Ġsome Ġexisting Ġrelations Ġin Ġthe ĠKB Ġmay Ġalso Ġbe Ġtrue Ġ( not Ġlogical Ġimplications Ġas Ġthey Ġcan Ġbe Ġautomatically Ġinferred ). ĠFor Ġexample , Ġif Ġthe Ġuser Ġsays ĠâĢ ľ Obama Ġwas Ġborn Ġin ĠUSA , âĢ Ŀ Ġthe Ġsystem Ġmay Ġguess Ġthat Ġ( Obama , ĠCitizen Of , ĠUSA ) Ġ( meaning Ġthat ĠObama Ġis Ġa Ġcitizen Ġof ĠUSA ) Ġcould Ġalso Ġbe Ġtrue Ġbased Ġon Ġthe Ġcurrent ĠKB . ĠTo Ġverify Ġthis Ġfact , Ġit Ġneeds Ġto Ġsolve Ġa ĠK BC Ġproblem Ġby Ġtreating Ġ( Obama , ĠCitizen Of , ĠUSA ) Ġas Ġa Ġquery . ĠThis Ġis Ġa ĠK BC Ġproblem Ġbecause Ġthe Ġfact Ġ( Obama , ĠBorn In , ĠUSA ) Ġextracted Ġfrom Ġthe Ġoriginal Ġsentence Ġhas Ġbeen Ġadded Ġto Ġthe ĠKB . ĠThen ĠObama Ġand ĠUSA Ġare Ġin Ġthe ĠKB . ĠIf Ġthe ĠK BC Ġproblem Ġis Ġsolved , Ġit Ġlearns Ġa Ġnew Ġfact Ġ( Obama , ĠCitizen Of , ĠUSA ) Ġin Ġaddition Ġto Ġthe Ġextracted Ġfact Ġ( Obama , ĠBorn In , ĠUSA ). Ġ( 2 ) ĠFor Ġa Ġquery Ġfact , Ġe . g ., Ġ( Obama , ĠBorn In , ĠUSA ) Ġextracted Ġfrom Ġthe Ġuser Ġquestion ĠâĢ ľ Was ĠObama Ġborn Ġin ĠUSA ? âĢ Ŀ Ġwe Ġneed Ġto Ġsolve Ġthe ĠOK BC Ġproblem Ġif Ġany Ġof ĠâĢ ľ Obama , ĠâĢ ľ Born In âĢ Ŀ , Ġor ĠâĢ ľ USA " Ġis Ġnot Ġalready Ġin Ġthe ĠKB . </s> We Ġcan Ġsee Ġthat ĠOK BC Ġis Ġthe Ġcore Ġof Ġa Ġknowledge Ġlearning Ġengine Ġfor Ġconversation . ĠThus , Ġin Ġthis Ġpaper , Ġwe Ġfocus Ġon Ġsolving Ġit . ĠWe Ġassume Ġthat Ġother Ġtasks Ġsuch Ġas Ġfact / relation Ġextraction Ġand Ġresolution Ġand Ġguessing Ġof Ġrelated Ġfacts Ġof Ġan Ġextracted Ġfact Ġare Ġsolved Ġby Ġother Ġsub - system s . </s> We Ġsolve Ġthe ĠOK BC Ġproblem Ġby Ġmim icking Ġhow Ġhumans Ġacquire Ġknowledge Ġand Ġperform Ġreasoning Ġin Ġan Ġinteractive Ġconversation . ĠWhenever Ġwe Ġencounter Ġan Ġunknown Ġconcept Ġor Ġrelation Ġwhile Ġanswering Ġa Ġquery , Ġwe Ġperform Ġinference Ġusing Ġour Ġexisting Ġknowledge . ĠIf Ġour Ġknowledge Ġdoes Ġnot Ġallow Ġus Ġto Ġdraw Ġa Ġconclusion , Ġwe Ġtypically Ġask Ġquestions Ġto Ġothers Ġto Ġacquire Ġrelated Ġknowledge Ġand Ġuse Ġit Ġin Ġinference . ĠThe Ġprocess Ġtypically Ġinvolves Ġan Ġinference Ġstrategy Ġ( a Ġsequence Ġof Ġactions ), Ġwhich Ġinter le aves Ġa Ġsequence Ġof Ġprocessing Ġand Ġinteractive Ġactions . ĠA Ġprocessing Ġaction Ġcan Ġbe Ġthe Ġselection Ġof Ġrelated Ġfacts , Ġder iving Ġinference Ġchain , Ġetc ., Ġthat Ġadvances Ġthe Ġinference Ġprocess . ĠAn Ġinteractive Ġaction Ġcan Ġbe Ġdeciding Ġwhat Ġto Ġask , Ġform ulating Ġa Ġsuitable Ġquestion , Ġetc ., Ġthat Ġenable Ġus Ġto Ġinteract . ĠThe Ġprocess Ġhelps Ġgrow Ġthe Ġknowledge Ġover Ġtime Ġand Ġthe Ġgained Ġknowledge Ġenables Ġus Ġto Ġcommunicate Ġbetter Ġin Ġthe Ġfuture . ĠWe Ġcall Ġthis Ġlifelong Ġinteractive Ġlearning Ġand Ġinference Ġ( Li Li ). ĠLif elong Ġlearning Ġis Ġreflected Ġby Ġthe Ġfacts Ġthat Ġthe Ġnewly Ġacquired Ġfacts Ġare Ġretained Ġin Ġthe ĠKB Ġand Ġused Ġin Ġinference Ġfor Ġfuture Ġqueries , Ġand Ġthat Ġthe Ġaccumulated Ġknowledge Ġin Ġaddition Ġto Ġthe Ġupdated ĠKB Ġincluding Ġpast Ġinference Ġperformances Ġare Ġlever aged Ġto Ġguide Ġfuture Ġinteraction Ġand Ġlearning . ĠLi Li Ġshould Ġhave Ġthe Ġfollowing Ġcapabilities : </s> This Ġsetting Ġis Ġideal Ġfor Ġmany ĠN LP Ġapplications Ġlike Ġdialog Ġand Ġquestion - ans w ering Ġsystems Ġthat Ġnaturally Ġprovide Ġthe Ġscope Ġfor Ġhuman Ġinteraction Ġand Ġdemand Ġreal - time Ġinference . </s> Li Li Ġstarts Ġwith Ġthe Ġclosed - world ĠK BC Ġapproach Ġpath - ranking Ġ( PR ) ĠB IB REF 11 Ġ, ĠB IB REF 17 Ġand Ġextends ĠK BC Ġin Ġa Ġmajor Ġway Ġto Ġopen - world Ġknowledge Ġbase Ġcompletion Ġ( OK BC ). ĠFor Ġa Ġrelation ĠIN LINE FORM 0 Ġ, ĠPR Ġworks Ġby Ġenumer ating Ġpaths Ġ( except Ġsingle - link Ġpath ĠIN LINE FORM 1 Ġ) Ġbetween Ġentity - p airs Ġlinked Ġby ĠIN LINE FORM 2 Ġin Ġthe ĠKB Ġand Ġuse Ġthem Ġas Ġfeatures Ġto Ġtrain Ġa Ġbinary Ġclass ifier Ġto Ġpredict Ġwhether Ġa Ġquery ĠIN LINE FORM 3 Ġshould Ġbe Ġin Ġthe ĠKB . ĠHere , Ġa Ġpath Ġbetween Ġtwo Ġentities Ġis Ġa Ġsequence Ġof Ġrelations Ġlinking Ġthem . ĠIn Ġour Ġwork , Ġwe Ġadopt Ġthe Ġlatest ĠPR Ġmethod , ĠC - PR ĠB IB REF 16 Ġand Ġextend Ġit Ġto Ġmake Ġit Ġwork Ġin Ġthe Ġopen - world Ġsetting . ĠC - PR Ġenumer ates Ġpaths Ġby Ġperforming Ġbid irection al Ġrandom Ġwalks Ġover Ġthe ĠKB Ġgraph Ġwhile Ġleveraging Ġthe Ġcontext Ġof Ġthe Ġsource - target Ġentity - pair . ĠWe Ġalso Ġadopt Ġand Ġextend Ġthe Ġcompos itional Ġvector Ġspace Ġmodel ĠB IB REF 20 Ġ, ĠB IB REF 21 Ġwith Ġcontinual Ġlearning Ġcapability Ġfor Ġprediction . </s> Given Ġan ĠOK BC Ġquery Ġ( ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġ) Ġ( e . g ., ĠâĢ ľ ( Obama , ĠCitizen Of , ĠUSA ), Ġwhich Ġmeans Ġwhether ĠObama Ġa Ġcitizen Ġof ĠUSA ), ĠLi Li Ġinteracts Ġwith Ġthe Ġuser Ġ( if Ġneeded ) Ġby Ġdynamically Ġform ulating Ġquestions Ġ( see Ġthe Ġinteraction Ġexample Ġin ĠFigure Ġ1 , Ġwhich Ġwill Ġbe Ġfurther Ġexplained Ġin ĠÂ§ 3 ) Ġand Ġlever ages Ġthe Ġinteract ively Ġacquired Ġknowledge Ġ( support ing Ġfacts Ġ( SF s ) Ġin Ġthe Ġfigure ) Ġfor Ġcontinued Ġinference . ĠTo Ġdo Ġso , ĠLi Li Ġform ulates Ġa Ġquery - specific Ġinference Ġstrategy Ġand Ġexecutes Ġit . ĠWe Ġdesign ĠLi Li Ġin Ġa ĠRein forcement ĠLearning Ġ( RL ) Ġsetting Ġthat Ġperforms Ġsub - t asks Ġlike Ġform ulating Ġand Ġexecuting Ġstrategy , Ġtraining Ġa Ġprediction Ġmodel Ġfor Ġinference , Ġand Ġknowledge Ġretention Ġfor Ġfuture Ġuse . ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge , Ġour Ġwork Ġis Ġthe Ġfirst Ġto Ġaddress Ġthe ĠOK BC Ġproblem Ġand Ġto Ġpropose Ġan Ġinteractive Ġlearning Ġmechanism Ġto Ġsolve Ġit Ġin Ġa Ġcontinuous Ġor Ġlifelong Ġmanner . ĠWe Ġempir ically Ġverify Ġthe Ġeffectiveness Ġof ĠLi Li Ġon Ġtwo Ġstandard Ġreal - world ĠKB s : ĠFree base Ġand ĠWord Net . ĠExperimental Ġresults Ġshow Ġthat ĠLi Li Ġis Ġhighly Ġeffective Ġin Ġterms Ġof Ġits Ġpredictive Ġperformance Ġand Ġstrategy Ġformulation Ġability . </s> Related ĠWork </s> To Ġthe Ġbest Ġof Ġour Ġknowledge , Ġwe Ġare Ġnot Ġaware Ġof Ġany Ġknowledge Ġlearning Ġsystem Ġthat Ġcan Ġlearn Ġnew Ġknowledge Ġin Ġthe Ġconversation Ġprocess . ĠThis Ġsection Ġthus Ġdiscusses Ġother Ġrelated Ġwork . </s> Among Ġexisting ĠKB Ġcompletion Ġapproaches , ĠB IB REF 20 Ġextended Ġthe Ġvector Ġspace Ġmodel Ġfor Ġzero - shot ĠKB Ġinference . ĠHowever , Ġthe Ġmodel Ġcannot Ġhandle Ġunknown Ġentities Ġand Ġcan Ġonly Ġwork Ġon Ġfixed Ġset Ġof Ġunknown Ġrelations Ġwith Ġknown Ġembed d ings . ĠRecently , ĠB IB REF 22 Ġproposed Ġa Ġmethod Ġusing Ġexternal Ġtext Ġcorpus Ġto Ġperform Ġinference Ġon Ġunknown Ġentities . ĠHowever , Ġthe Ġmethod Ġcannot Ġhandle Ġunknown Ġrelations . ĠThus , Ġthese Ġmethods Ġare Ġnot Ġsuitable Ġfor Ġour Ġopen - world Ġsetting . ĠNone Ġof Ġthe Ġexisting ĠKB Ġinference Ġmethods Ġperform Ġinteractive Ġknowledge Ġlearning Ġlike ĠLi Li . ĠNE LL ĠB IB REF 23 Ġcontinuously Ġupdates Ġits ĠKB Ġusing Ġfacts Ġextracted Ġfrom Ġthe ĠWeb . ĠOur Ġtask Ġis Ġvery Ġdifferent Ġas Ġwe Ġdo Ġnot Ġdo ĠWeb Ġfact Ġextraction Ġ( which Ġis Ġalso Ġuseful ). ĠWe Ġfocus Ġon Ġuser Ġinteractions Ġin Ġthis Ġpaper . ĠOur Ġwork Ġis Ġrelated Ġto Ġinteractive Ġlanguage Ġlearning Ġ( ILL ) ĠB IB REF 24 Ġ, ĠB IB REF 25 Ġ, Ġbut Ġthese Ġare Ġnot Ġabout ĠKB Ġcompletion . ĠThe Ġwork Ġin ĠB IB REF 26 Ġallows Ġa Ġlear ner Ġto Ġask Ġquestions Ġin Ġdialogue . ĠHowever , Ġthis Ġwork Ġused ĠRL Ġto Ġlearn Ġabout Ġwhether Ġto Ġask Ġthe Ġuser Ġor Ġnot . ĠThe ĠâĢ ľ what Ġto Ġask Ġaspect " Ġwas Ġmanually Ġdesigned Ġby Ġmodeling Ġsynthetic Ġtasks . ĠLi Li Ġform ulates Ġquery - specific Ġinference Ġstrategies Ġwhich Ġembed Ġinteraction Ġbehaviors . ĠAlso , Ġno Ġexisting Ġdialogue Ġsystems ĠB IB REF 4 Ġ, ĠB IB REF 27 Ġ, ĠB IB REF 28 Ġ, ĠB IB REF 29 Ġ, ĠB IB REF 30 Ġemploy Ġlifelong Ġlearning Ġto Ġtrain Ġprediction Ġmodels Ġby Ġusing Ġinformation / knowledge Ġretained Ġin Ġthe Ġpast . </s> Our Ġwork Ġis Ġrelated Ġto Ġgeneral Ġlifelong Ġlearning Ġin ĠB IB REF 31 Ġ, ĠB IB REF 32 Ġ, ĠB IB REF 33 Ġ, ĠB IB REF 34 Ġ, ĠB IB REF 35 Ġ, ĠB IB REF 36 Ġ. ĠHowever , Ġthey Ġlearn Ġonly Ġone Ġtype Ġof Ġtasks , Ġe . g ., Ġsupervised , Ġtopic Ġmodeling Ġor Ġreinforcement Ġlearning Ġ( RL ) Ġtasks . ĠNone Ġof Ġthem Ġis Ġsuitable Ġfor Ġour Ġsetting , Ġwhich Ġinvolves Ġinter le aving Ġof ĠRL , Ġsupervised Ġand Ġinteractive Ġlearning . ĠMore Ġdetails Ġabout Ġlifelong Ġlearning Ġcan Ġbe Ġfound Ġin Ġthe Ġbook ĠB IB REF 31 Ġ. </s> Inter active ĠKnowledge ĠLearning Ġ( Li Li ) </s> We Ġdesign ĠLi Li Ġas Ġa Ġcombination Ġof Ġtwo Ġinterconnected Ġmodels : Ġ( 1 ) Ġa ĠRL Ġmodel Ġthat Ġlearns Ġto Ġformulate Ġa Ġquery - specific Ġinference Ġstrategy Ġfor Ġperforming Ġthe ĠOK BC Ġtask , Ġand Ġ( 2 ) Ġa Ġlifelong Ġprediction Ġmodel Ġto Ġpredict Ġwhether Ġa Ġtriple Ġshould Ġbe Ġin Ġthe ĠKB , Ġwhich Ġis Ġinvoked Ġby Ġan Ġaction Ġwhile Ġexecuting Ġthe Ġinference Ġstrategy Ġand Ġis Ġlearned Ġfor Ġeach Ġrelation Ġas Ġin ĠC - PR . ĠThe Ġframework Ġimproves Ġits Ġperformance Ġover Ġtime Ġthrough Ġuser Ġinteraction Ġand Ġknowledge Ġretention . ĠCompared Ġto Ġthe Ġexisting ĠKB Ġinference Ġmethods , ĠLi Li Ġover comes Ġthe Ġfollowing Ġthree Ġchallenges Ġfor ĠOK BC : </s> 1 . ĠM apping Ġopen - world Ġto Ġclose - world . ĠBeing Ġa Ġclosed - world Ġmethod , ĠC - PR Ġcannot Ġextract Ġpath Ġfeatures Ġand Ġlearn Ġa Ġprediction Ġmodel Ġwhen Ġany Ġof ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġor ĠIN LINE FORM 2 Ġis Ġunknown . ĠLi Li Ġsolves Ġthis Ġproblem Ġthrough Ġinteractive Ġknowledge Ġacquisition . ĠIf ĠIN LINE FORM 3 Ġis Ġunknown , ĠLi Li Ġasks Ġthe Ġuser Ġto Ġprovide Ġa Ġclue Ġ( an Ġexample Ġof ĠIN LINE FORM 4 Ġ). ĠAnd Ġif ĠIN LINE FORM 5 Ġor ĠIN LINE FORM 6 Ġis Ġunknown , ĠLi Li Ġasks Ġthe Ġuser Ġto Ġprovide Ġa Ġlink Ġ( relation ) Ġto Ġconnect Ġthe Ġunknown Ġentity Ġwith Ġan Ġexisting Ġentity Ġ( aut om atically Ġselected ) Ġin Ġthe ĠKB . ĠWe Ġrefer Ġto Ġsuch Ġa Ġquery Ġas Ġa Ġconnecting Ġlink Ġquery Ġ( CL Q ). ĠThe Ġacquired Ġknowledge Ġreduces ĠOK BC Ġto ĠK BC Ġand Ġmakes Ġthe Ġinference Ġtask Ġfeasible . </s> 2 . ĠSp ares eness Ġof ĠKB . ĠA Ġmain Ġissue Ġof Ġall ĠPR Ġmethods Ġlike ĠC - PR Ġis Ġthe Ġconnectivity Ġof Ġthe ĠKB Ġgraph . ĠIf Ġthere Ġis Ġno Ġpath Ġconnecting ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġin Ġthe Ġgraph , Ġpath Ġenumer ation Ġof ĠC - PR Ġgets Ġstuck Ġand Ġinference Ġbecomes Ġinf eas ible . ĠIn Ġsuch Ġcases , ĠLi Li Ġuses Ġa Ġtemplate Ġrelation Ġ( âĢ ľ @ - ? - @ ") Ġas Ġthe Ġmissing Ġlink Ġmarker Ġto Ġconnect Ġentity - p airs Ġand Ġcontinues Ġfeature Ġextraction . ĠA Ġpath Ġcontaining ĠâĢ ľ @ - ? - @ " Ġis Ġcalled Ġan Ġincomplete Ġpath . ĠThus , Ġthe Ġextracted Ġfeature Ġset Ġcontains Ġboth Ġcomplete Ġ( no Ġmissing Ġlink ) Ġand Ġincomplete Ġpaths . ĠNext , ĠLi Li Ġselects Ġan Ġincomplete Ġpath Ġfrom Ġthe Ġfeature Ġset Ġand Ġasks Ġthe Ġuser Ġto Ġprovide Ġa Ġlink Ġfor Ġpath Ġcompletion . ĠWe Ġrefer Ġto Ġsuch Ġa Ġquery Ġas Ġmissing Ġlink Ġquery Ġ( ML Q ). </s> 3 . ĠLim itation Ġin Ġuser Ġknowledge . ĠIf Ġthe Ġuser Ġis Ġunable Ġto Ġrespond Ġto ĠML Q s Ġor ĠCL Q s , ĠLi Li Ġuses Ġa Ġguessing Ġmechanism Ġ( disc ussed Ġlater ) Ġto Ġfill Ġthe Ġgap . ĠThis Ġenables ĠLi Li Ġto Ġcontinue Ġits Ġinference Ġeven Ġif Ġthe Ġuser Ġcannot Ġanswer Ġa Ġsystem Ġquestion . </s> Comp onents Ġof ĠLi Li </s> As Ġlifelong Ġlearning Ġneeds Ġto Ġretain Ġknowledge Ġlearned Ġfrom Ġpast Ġtasks Ġand Ġuse Ġit Ġto Ġhelp Ġfuture Ġlearning ĠB IB REF 31 Ġ, ĠLi Li Ġuses Ġa ĠKnowledge ĠStore Ġ( KS ) Ġfor Ġknowledge Ġretention . ĠKS Ġhas Ġfour Ġcomponents : Ġ( i ) ĠKnowledge ĠGraph Ġ( ĠIN LINE FORM 0 Ġ): ĠIN LINE FORM 1 Ġ( the ĠKB ) Ġis Ġinitialized Ġwith Ġbase ĠKB Ġtri ples Ġ( see ĠÂ§ 4 ) Ġand Ġgets Ġupdated Ġover Ġtime Ġwith Ġthe Ġacquired Ġknowledge . Ġ( ii ) ĠRel ation - Entity ĠMatrix Ġ( ĠIN LINE FORM 2 Ġ): ĠIN LINE FORM 3 Ġis Ġa Ġsparse Ġmatrix , Ġwith Ġrows Ġas Ġrelations Ġand Ġcolumns Ġas Ġentity - p airs Ġand Ġis Ġused Ġby Ġthe Ġprediction Ġmodel . ĠGiven Ġa Ġtriple Ġ( ĠIN LINE FORM 4 Ġ, ĠIN LINE FORM 5 Ġ, ĠIN LINE FORM 6 Ġ) ĠIN LINE FORM 7 Ġ, Ġwe Ġset ĠIN LINE FORM 8 Ġ[ ĠIN LINE FORM 9 Ġ, Ġ( ĠIN LINE FORM 10 Ġ, ĠIN LINE FORM 11 Ġ)] Ġ= Ġ1 Ġindicating ĠIN LINE FORM 12 Ġoccurs Ġfor Ġpair Ġ( ĠIN LINE FORM 13 Ġ, ĠIN LINE FORM 14 Ġ). Ġ( iii ) ĠTask ĠExperience ĠStore Ġ( ĠIN LINE FORM 15 Ġ): ĠIN LINE FORM 16 Ġstores Ġthe Ġpredictive Ġperformance Ġof ĠLi Li Ġon Ġpast Ġlearned Ġtasks Ġin Ġterms Ġof ĠMatthews Ġcorrelation Ġcoefficient Ġ( M CC ) Ġthat Ġmeasures Ġthe Ġquality Ġof Ġbinary Ġclassification . ĠSo , Ġfor Ġtwo Ġtasks ĠIN LINE FORM 17 Ġand ĠIN LINE FORM 18 Ġ( each Ġrelation Ġis Ġa Ġtask ), Ġif ĠIN LINE FORM 19 Ġ[ ĠIN LINE FORM 20 Ġ] ĠIN LINE FORM 21 ĠIN LINE FORM 22 Ġ[ ĠIN LINE FORM 23 Ġ] Ġ[ where ĠIN LINE FORM 24 Ġ[ ĠIN LINE FORM 25 Ġ] = M CC ( ĠIN LINE FORM 26 Ġ) ], Ġwe Ġsay ĠC - PR Ġhas Ġlearned ĠIN LINE FORM 27 Ġwell Ġcompared Ġto ĠIN LINE FORM 28 Ġ. Ġ( iv ) ĠIn complete ĠFeature ĠDB Ġ( ĠIN LINE FORM 29 Ġ): ĠIN LINE FORM 30 Ġstores Ġthe Ġfrequency Ġof Ġan Ġincomplete Ġpath ĠIN LINE FORM 31 Ġin Ġthe Ġform Ġof Ġa Ġtuple Ġ( ĠIN LINE FORM 32 Ġ, ĠIN LINE FORM 33 Ġ, ĠIN LINE FORM 34 Ġ) Ġand Ġis Ġused Ġin Ġform ulating ĠML Q s . ĠIN LINE FORM 35 Ġ[( ĠIN LINE FORM 36 Ġ, ĠIN LINE FORM 37 Ġ, ĠIN LINE FORM 38 Ġ)] Ġ= ĠIN LINE FORM 39 Ġimplies ĠLi Li Ġhas Ġextracted Ġincomplete Ġpath ĠIN LINE FORM 40 ĠIN LINE FORM 41 Ġtimes Ġinvolving Ġentity - pair ĠIN LINE FORM 42 Ġ[( ĠIN LINE FORM 43 Ġ, ĠIN LINE FORM 44 Ġ)] Ġfor Ġquery Ġrelation ĠIN LINE FORM 45 Ġ. </s> The ĠRL Ġmodel Ġlearns Ġeven Ġafter Ġtraining Ġwhenever Ġit Ġencounters Ġan Ġunseen Ġstate Ġ( in Ġtesting ) Ġand Ġthus , Ġgets Ġupdated Ġover Ġtime . ĠKS Ġis Ġupdated Ġcontinuously Ġover Ġtime Ġas Ġa Ġresult Ġof Ġthe Ġexecution Ġof ĠLi Li Ġand Ġtakes Ġpart Ġin Ġfuture Ġlearning . ĠThe Ġprediction Ġmodel Ġuses Ġlifelong Ġlearning Ġ( LL ), Ġwhere Ġwe Ġtransfer Ġknowledge Ġ( param eter Ġvalues ) Ġfrom Ġthe Ġmodel Ġfor Ġa Ġpast Ġmost Ġsimilar Ġtask Ġto Ġhelp Ġlearn Ġfor Ġthe Ġcurrent Ġtask . ĠSimilar Ġtasks Ġare Ġidentified Ġby Ġfactor izing ĠIN LINE FORM 0 Ġand Ġcomputing Ġa Ġtask Ġsimilarity Ġmatrix ĠIN LINE FORM 1 Ġ. ĠBesides ĠLL , ĠLi Li Ġuses ĠIN LINE FORM 2 Ġto Ġidentify Ġpoorly Ġlearned Ġpast Ġtasks Ġand Ġacquire Ġmore Ġclues Ġfor Ġthem Ġto Ġimprove Ġits Ġskills et Ġover Ġtime . </s> Li Li Ġalso Ġuses Ġa Ġstack , Ġcalled ĠIn ference ĠStack Ġ( ĠIN LINE FORM 0 Ġ) Ġto Ġhold Ġquery Ġand Ġits Ġstate Ġinformation Ġfor ĠRL . ĠLi Li Ġalways Ġprocesses Ġstack Ġtop Ġ( ĠIN LINE FORM 1 Ġ[ top ]). ĠThe Ġclues Ġfrom Ġthe Ġuser Ġget Ġstored Ġin ĠIN LINE FORM 2 Ġon Ġtop Ġof Ġthe Ġquery Ġduring Ġstrategy Ġexecution Ġand Ġprocessed Ġfirst . ĠThus , Ġthe Ġprediction Ġmodel Ġfor ĠIN LINE FORM 3 Ġis Ġlearned Ġbefore Ġperforming Ġinference Ġon Ġquery , Ġtransforming ĠOK BC Ġto Ġa ĠK BC Ġproblem . ĠTable Ġ1 Ġshows Ġthe Ġparameters Ġof ĠLi Li Ġused Ġin Ġthe Ġfollowing Ġsections . </s> Working Ġof ĠLi Li </s> Given Ġan ĠOK BC Ġquery Ġ( ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġ), Ġwe Ġrepresent Ġit Ġas Ġa Ġdata Ġinstance ĠIN LINE FORM 3 Ġ. ĠIN LINE FORM 4 Ġconsists Ġof ĠIN LINE FORM 5 Ġ( the Ġquery Ġtriple ), ĠIN LINE FORM 6 Ġ( inter action Ġlimit Ġset Ġfor ĠIN LINE FORM 7 Ġ), ĠIN LINE FORM 8 Ġ( exper ience Ġlist Ġstoring Ġthe Ġtransition Ġhistory Ġof ĠM DP Ġfor ĠIN LINE FORM 9 Ġin ĠRL ) Ġand ĠIN LINE FORM 10 Ġ( mode Ġof ĠIN LINE FORM 11 Ġ) Ġden oting Ġif ĠIN LINE FORM 12 Ġis Ġ` ĠIN LINE FORM 13 Ġ' Ġ( training ), Ġ` ĠIN LINE FORM 14 Ġ' Ġ( valid ation ), Ġ` ĠIN LINE FORM 15 Ġ' Ġ( eval uation ) Ġor Ġ` ĠIN LINE FORM 16 Ġ' Ġ( cl ue ) Ġinstance Ġand ĠIN LINE FORM 17 Ġ( feature Ġset ). ĠWe Ġdenote ĠIN LINE FORM 18 Ġ( ĠIN LINE FORM 19 Ġ) Ġas Ġthe Ġset Ġof Ġall Ġcomplete Ġ( in complete ) Ġpath Ġfeatures Ġin ĠIN LINE FORM 20 Ġ. ĠGiven Ġa Ġdata Ġinstance ĠIN LINE FORM 21 Ġ, ĠLi Li Ġstarts Ġits Ġinitialization Ġas Ġfollows : Ġit Ġsets Ġthe Ġstate Ġas ĠIN LINE FORM 22 Ġ( based Ġon ĠIN LINE FORM 23 Ġ, Ġexplained Ġlater ), Ġpushes Ġthe Ġquery Ġtuple Ġ( ĠIN LINE FORM 24 Ġ, ĠIN LINE FORM 25 Ġ) Ġinto ĠIN LINE FORM 26 Ġand Ġfeeds ĠIN LINE FORM 27 Ġ[ top ] Ġto Ġthe ĠRL - model Ġfor Ġstrategy Ġformulation Ġfrom ĠIN LINE FORM 28 Ġ. </s> In ference ĠStrategy ĠForm ulation . ĠWe Ġview Ġsolving Ġthe Ġstrategy Ġformulation Ġproblem Ġas Ġlearning Ġto Ġplay Ġan Ġinference Ġgame , Ġwhere Ġthe Ġgoal Ġis Ġto Ġformulate Ġa Ġstrategy Ġthat Ġ" makes Ġthe Ġinference Ġtask Ġpossible ". ĠConsidering ĠPR Ġmethods , Ġinference Ġis Ġpossible , Ġif f Ġ( 1 ) ĠIN LINE FORM 0 Ġbecomes Ġknown Ġto Ġits ĠKB Ġ( by Ġacquiring Ġclues Ġwhen ĠIN LINE FORM 1 Ġis Ġunknown ) Ġand Ġ( 2 ) Ġpath Ġfeatures Ġare Ġextracted Ġbetween ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġ( which Ġint urn Ġrequires ĠIN LINE FORM 4 Ġand ĠIN LINE FORM 5 Ġto Ġbe Ġknown Ġto ĠKB ). ĠIf Ġthese Ġconditions Ġare Ġmet Ġat Ġthe Ġend Ġof Ġan Ġepisode Ġ( when Ġstrategy Ġformulation Ġfinishes Ġfor Ġa Ġgiven Ġquery ) Ġof Ġthe Ġgame , ĠLi Li Ġwins Ġand Ġthus , Ġit Ġtrains Ġthe Ġprediction Ġmodel Ġfor ĠIN LINE FORM 6 Ġand Ġuses Ġit Ġfor Ġinference . </s> Li Li 's Ġstrategy Ġformulation Ġis Ġmodeled Ġas Ġa ĠMark ov ĠDecision ĠProcess Ġ( M DP ) Ġwith Ġfinite Ġstate Ġ( ĠIN LINE FORM 0 Ġ) Ġand Ġaction Ġ( ĠIN LINE FORM 1 Ġ) Ġspaces . ĠA Ġstate ĠIN LINE FORM 2 Ġconsists Ġof Ġ10 Ġbinary Ġstate Ġvariables Ġ( Table Ġ2 ), Ġeach Ġof Ġwhich Ġkeeps Ġtrack Ġof Ġresults Ġof Ġan Ġaction ĠIN LINE FORM 3 Ġtaken Ġby ĠLi Li Ġand Ġthus , Ġrecords Ġthe Ġprogress Ġin Ġinference Ġprocess Ġmade Ġso Ġfar . ĠIN LINE FORM 4 Ġis Ġthe Ġinitial Ġstate Ġwith Ġall Ġstate Ġbits Ġset Ġas Ġ0 . ĠIf Ġthe Ġdata Ġinstance Ġ( query ) Ġis Ġa Ġclue Ġ[ ĠIN LINE FORM 5 Ġ], ĠIN LINE FORM 6 Ġ[ CL UE ] Ġis Ġset Ġas Ġ1 . ĠIN LINE FORM 7 Ġconsists Ġof Ġ6 Ġactions Ġ( Table Ġ3 ). ĠIN LINE FORM 8 Ġ, ĠIN LINE FORM 9 Ġ, ĠIN LINE FORM 10 Ġare Ġprocessing Ġactions Ġand ĠIN LINE FORM 11 Ġ, ĠIN LINE FORM 12 Ġ, ĠIN LINE FORM 13 Ġare Ġinteractive Ġactions . ĠWhenever ĠIN LINE FORM 14 Ġis Ġexecuted , Ġthe ĠM DP Ġreaches Ġthe Ġterminal Ġstate . ĠGiven Ġan Ġaction ĠIN LINE FORM 15 Ġin Ġstate ĠIN LINE FORM 16 Ġ, Ġif ĠIN LINE FORM 17 Ġis Ġinvalid Ġin ĠIN LINE FORM 21 Ġor Ġthe Ġobjective Ġof ĠIN LINE FORM 22 Ġis Ġunsatisf ied Ġ(* Ġmarked Ġthe Ġcondition Ġin ĠIN LINE FORM 23 Ġ), ĠRL Ġreceives Ġa Ġnegative Ġreward Ġ( em pir ically Ġset ); Ġelse Ġreceives Ġa Ġpositive Ġreward .. ĠWe Ġuse ĠQ - learning ĠB IB REF 38 Ġwith ĠIN LINE FORM 24 Ġ- gre edy Ġstrategy Ġto Ġlearn Ġthe Ġoptimal Ġpolicy Ġfor Ġtraining Ġthe ĠRL Ġmodel . ĠNote Ġthat , Ġthe Ġinference Ġstrategy Ġis Ġindependent Ġof ĠKB Ġtype Ġand Ġcorrectness Ġof Ġprediction . ĠThus , Ġthe ĠRL - model Ġis Ġtrained Ġonly Ġonce Ġfrom Ġscratch Ġ( re used Ġthereafter Ġfor Ġother ĠKB s ) Ġand Ġalso , Ġindependently Ġof Ġthe Ġprediction Ġmodel . </s> Sometimes Ġthe Ġtraining Ġdataset Ġmay Ġnot Ġbe Ġenough Ġto Ġlearn Ġoptimal Ġpolicy Ġfor Ġall ĠIN LINE FORM 0 Ġ. ĠThus , Ġencountering Ġan Ġunseen Ġstate Ġduring Ġtest Ġcan Ġmake ĠRL - model Ġclueless Ġabout Ġthe Ġaction . ĠGiven Ġa Ġstate ĠIN LINE FORM 1 Ġ, Ġwhenever Ġan Ġinvalid ĠIN LINE FORM 2 Ġis Ġchosen , ĠLi Li Ġremains Ġin ĠIN LINE FORM 3 Ġ. ĠFor ĠIN LINE FORM 4 Ġ, ĠLi Li Ġremains Ġin ĠIN LINE FORM 5 Ġunt ill ĠIN LINE FORM 6 Ġ( see ĠTable Ġ1 Ġfor ĠIN LINE FORM 7 Ġ). ĠSo , Ġif Ġthe Ġstate Ġremains Ġthe Ġsame Ġfor Ġ( ĠIN LINE FORM 8 Ġ+ 1 ) Ġtimes , Ġit Ġimplies ĠLi Li Ġhas Ġencountered Ġa Ġfault Ġ( an Ġunseen Ġstate ). ĠRL - model Ġinstantly Ġswitches Ġto Ġthe Ġtraining Ġmode Ġand Ġrandomly Ġexplores ĠIN LINE FORM 9 Ġto Ġlearn Ġthe Ġoptimal Ġaction Ġ( f ault - t oler ant Ġlearning ). ĠWhile Ġexploring ĠIN LINE FORM 10 Ġ, Ġthe Ġmodel Ġchooses ĠIN LINE FORM 11 Ġonly Ġwhen Ġit Ġhas Ġtried Ġall Ġother ĠIN LINE FORM 12 Ġto Ġavoid Ġabrupt Ġend Ġof Ġepisode . </s> Exec ution Ġof ĠActions . ĠAt Ġany Ġgiven Ġpoint Ġin Ġtime , Ġlet Ġ( ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ) Ġbe Ġthe Ġcurrent ĠIN LINE FORM 2 Ġ[ top ], ĠIN LINE FORM 3 Ġis Ġthe Ġchosen Ġaction Ġand Ġthe Ġcurrent Ġversion Ġof ĠKS Ġcomponents Ġare ĠIN LINE FORM 4 Ġ, ĠIN LINE FORM 5 Ġ, ĠIN LINE FORM 6 Ġand ĠIN LINE FORM 7 Ġ. ĠThen , Ġif ĠIN LINE FORM 8 Ġis Ġinvalid Ġin ĠIN LINE FORM 9 Ġ, ĠLi Li Ġonly Ġupdates ĠIN LINE FORM 10 Ġ[ top ] Ġwith Ġ( ĠIN LINE FORM 11 Ġ, ĠIN LINE FORM 12 Ġ) Ġand Ġreturns ĠIN LINE FORM 13 Ġ[ top ] Ġto ĠRL - model . ĠIn Ġthis Ġprocess , ĠLi Li Ġadds Ġexperience Ġ( ĠIN LINE FORM 14 Ġ, ĠIN LINE FORM 15 Ġ, ĠIN LINE FORM 16 Ġ, ĠIN LINE FORM 17 Ġ) Ġin ĠIN LINE FORM 18 Ġand Ġthen , Ġreplaces ĠIN LINE FORM 19 Ġ[ top ] Ġwith Ġ( ĠIN LINE FORM 20 Ġ, ĠIN LINE FORM 21 Ġ). ĠIf ĠIN LINE FORM 22 Ġis Ġvalid Ġin ĠIN LINE FORM 23 Ġ, ĠLi Li Ġfirst Ġsets Ġthe Ġnext Ġstate ĠIN LINE FORM 24 Ġand Ġperforms Ġa Ġsequence Ġof Ġoperations ĠIN LINE FORM 25 Ġbased Ġon ĠIN LINE FORM 26 Ġ( disc ussed Ġbelow ). ĠUnless Ġspecified , Ġin ĠIN LINE FORM 27 Ġ, ĠLi Li Ġalways Ġmonitors ĠIN LINE FORM 28 Ġand Ġif ĠIN LINE FORM 29 Ġbecomes Ġ0 , ĠLi Li Ġsets ĠIN LINE FORM 30 Ġ. ĠAlso , Ġwhenever ĠLi Li Ġasks Ġthe Ġuser Ġa Ġquery , ĠIN LINE FORM 31 Ġis Ġdecre mented Ġby Ġ1 . ĠOnce ĠIN LINE FORM 32 Ġends , ĠLi Li Ġupdates ĠIN LINE FORM 33 Ġ[ top ] Ġwith Ġ( ĠIN LINE FORM 34 Ġ, ĠIN LINE FORM 35 Ġ) Ġand Ġreturns ĠIN LINE FORM 36 Ġ[ top ] Ġto ĠRL - model Ġfor Ġchoosing Ġthe Ġnext Ġaction . </s> In ĠIN LINE FORM 0 Ġ, ĠLi Li Ġsearches ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġ, ĠIN LINE FORM 3 Ġin ĠIN LINE FORM 4 Ġand Ġsets Ġappropriate Ġbits Ġin ĠIN LINE FORM 5 Ġ( see ĠTable Ġ2 ). ĠIf ĠIN LINE FORM 6 Ġwas Ġunknown Ġbefore Ġand Ġis Ġjust Ġadded Ġto ĠIN LINE FORM 7 Ġor Ġis Ġin Ġthe Ġbottom ĠIN LINE FORM 8 Ġ% Ġ( see ĠTable Ġ1 Ġfor ĠIN LINE FORM 9 Ġ) Ġof ĠIN LINE FORM 10 Ġ, ĠLi Li Ġrandomly Ġsets ĠIN LINE FORM 14 Ġwith Ġprobability ĠIN LINE FORM 15 Ġ. ĠIf ĠIN LINE FORM 16 Ġis Ġa Ġclue Ġand ĠIN LINE FORM 17 Ġ, ĠLi Li Ġupdates ĠKS Ġwith Ġtriple ĠIN LINE FORM 18 Ġ, Ġwhere Ġ( ĠIN LINE FORM 19 Ġ, ĠIN LINE FORM 20 Ġ, ĠIN LINE FORM 21 Ġ) Ġand Ġ( ĠIN LINE FORM 22 Ġ, ĠIN LINE FORM 23 Ġ, ĠIN LINE FORM 24 Ġ) Ġgets Ġadded Ġto ĠIN LINE FORM 25 Ġand ĠIN LINE FORM 26 Ġ, ĠIN LINE FORM 27 Ġare Ġset Ġas Ġ1 . </s> In ĠIN LINE FORM 0 Ġ, ĠLi Li Ġasks Ġthe Ġuser Ġto Ġprovide Ġa Ġclue Ġ(+ ve Ġinstance ) Ġfor ĠIN LINE FORM 1 Ġand Ġcorrupt s ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġof Ġthe Ġclue Ġonce Ġat Ġa Ġtime , Ġto Ġgenerate Ġ- ve Ġinstances Ġby Ġsampling Ġnodes Ġfrom ĠIN LINE FORM 4 Ġ. ĠThese Ġinstances Ġhelp Ġin Ġtraining Ġprediction Ġmodel Ġfor ĠIN LINE FORM 5 Ġwhile Ġexecuting ĠIN LINE FORM 6 Ġ. </s> In ĠIN LINE FORM 0 Ġ, ĠLi Li Ġselects Ġan Ġincomplete Ġpath ĠIN LINE FORM 1 Ġfrom ĠIN LINE FORM 2 Ġto Ġformulate ĠML Q , Ġsuch Ġthat ĠIN LINE FORM 3 Ġis Ġmost Ġfrequently Ġobserved Ġfor ĠIN LINE FORM 4 Ġand ĠIN LINE FORM 5 Ġis Ġhigh , Ġgiven Ġby ĠIN LINE FORM 6 Ġ. ĠHere , ĠIN LINE FORM 7 Ġdenotes Ġthe Ġcontextual Ġsimilarity ĠB IB REF 16 Ġof Ġentity - pair ĠIN LINE FORM 8 Ġ. ĠIf ĠIN LINE FORM 9 Ġis Ġhigh , ĠIN LINE FORM 10 Ġis Ġmore Ġlikely Ġto Ġpossess Ġa Ġrelation Ġbetween Ġthem Ġand Ġso , Ġis Ġa Ġgood Ġcandidate Ġfor Ġform ulating ĠML Q . ĠWhen Ġthe Ġuser Ġdoes Ġnot Ġrespond Ġto ĠML Q Ġ( or ĠCL Q Ġin ĠIN LINE FORM 11 Ġ), Ġthe Ġguessing Ġmechanism Ġis Ġused , Ġwhich Ġworks Ġas Ġfollows : ĠSince Ġcontextual Ġsimilarity Ġof Ġentity - p airs Ġis Ġhighly Ġcorrelated Ġwith Ġtheir Ġclass Ġlabels ĠB IB REF 16 Ġ, ĠLi Li Ġdivides Ġthe Ġsimilarity Ġrange Ġ[- 1 , Ġ1 ] Ġinto Ġthree Ġsegments , Ġusing Ġa Ġlow Ġ( ĠIN LINE FORM 12 Ġ) Ġand Ġhigh Ġ( ĠIN LINE FORM 13 Ġ) Ġsimilarity Ġthreshold Ġand Ġreplaces Ġthe Ġmissing Ġlink Ġwith ĠIN LINE FORM 14 Ġin ĠIN LINE FORM 15 Ġto Ġmake Ġit Ġcomplete Ġas Ġfollows : ĠIf ĠIN LINE FORM 16 Ġ, ĠIN LINE FORM 17 Ġ= ĠâĢ ľ @ - Lo ose ly Related To - @ "; Ġelse Ġif ĠIN LINE FORM 18 Ġ, ĠIN LINE FORM 19 Ġ= âĢ ľ @ - Not Related To - @ "; ĠOtherwise , ĠIN LINE FORM 20 Ġ= âĢ ľ @ - Related To - @ ". </s> In ĠIN LINE FORM 0 Ġ, ĠLi Li Ġasks ĠCL Q s Ġfor Ġconnecting Ġunknown Ġentities ĠIN LINE FORM 1 Ġand / or ĠIN LINE FORM 2 Ġwith ĠIN LINE FORM 3 Ġby Ġselecting Ġthe Ġmost Ġcontext ually Ġrelevant Ġnode Ġ( w rt ĠIN LINE FORM 4 Ġ, ĠIN LINE FORM 5 Ġ) Ġfrom ĠIN LINE FORM 6 Ġ, Ġgiven Ġby Ġlink ĠIN LINE FORM 7 Ġ. ĠWe Ġadopt Ġthe Ġcontextual Ġrelevance Ġidea Ġin ĠB IB REF 16 Ġwhich Ġis Ġcomputed Ġusing Ġword Ġembed ding ĠB IB REF 39 </s> In ĠIN LINE FORM 0 Ġ, ĠLi Li Ġextracts Ġpath Ġfeatures ĠIN LINE FORM 1 Ġbetween Ġ( ĠIN LINE FORM 2 Ġ, ĠIN LINE FORM 3 Ġ) Ġand Ġupdates ĠIN LINE FORM 4 Ġwith Ġincomplete Ġfeatures Ġfrom ĠIN LINE FORM 5 Ġ. ĠLi Li Ġalways Ġtrains Ġthe Ġprediction Ġmodel Ġwith Ġcomplete Ġfeatures ĠIN LINE FORM 6 Ġand Ġonce ĠIN LINE FORM 7 Ġor ĠIN LINE FORM 8 Ġ, ĠLi Li Ġstops Ġasking ĠML Q s . ĠThus , Ġin Ġboth ĠIN LINE FORM 9 Ġand ĠIN LINE FORM 10 Ġ, ĠLi Li Ġalways Ġmonitors ĠIN LINE FORM 11 Ġto Ġcheck Ġfor Ġthe Ġsaid Ġrequirements Ġand Ġsets ĠIN LINE FORM 12 Ġto Ġcontrol Ġinteractions . </s> In ĠIN LINE FORM 0 Ġ, Ġif ĠLi Li Ġwins Ġthe Ġepisode , Ġit Ġadds ĠIN LINE FORM 1 Ġin Ġone Ġof Ġdata Ġbuffers ĠIN LINE FORM 2 Ġbased Ġon Ġits Ġmode ĠIN LINE FORM 3 Ġ. ĠE . g ., Ġif ĠIN LINE FORM 4 Ġor ĠIN LINE FORM 5 Ġ, ĠIN LINE FORM 6 Ġis Ġused Ġfor Ġtraining Ġand Ġadded Ġto ĠIN LINE FORM 7 Ġ. ĠSimilarly Ġvalidation Ġbuffer ĠIN LINE FORM 8 Ġand Ġevaluation Ġbuffer ĠIN LINE FORM 9 Ġare Ġpopulated . ĠIf ĠIN LINE FORM 10 Ġ, ĠLi Li Ġinv okes Ġthe Ġprediction Ġmodel Ġfor ĠIN LINE FORM 11 Ġ. </s> L if elong ĠRel ation ĠPrediction . ĠGiven Ġa Ġrelation ĠIN LINE FORM 0 Ġ, ĠLi Li Ġuses ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġ( see ĠIN LINE FORM 3 Ġ) Ġto Ġtrain Ġa Ġprediction Ġmodel Ġ( say , ĠIN LINE FORM 4 Ġ) Ġwith Ġparameters ĠIN LINE FORM 5 Ġ. ĠFor Ġa Ġunknown ĠIN LINE FORM 6 Ġ, Ġthe Ġclue Ġinstances Ġget Ġstored Ġin ĠIN LINE FORM 7 Ġand ĠIN LINE FORM 8 Ġ. ĠThus , ĠLi Li Ġpop ulates ĠIN LINE FORM 9 Ġby Ġtaking Ġ10 % Ġ( see ĠÂ§ 4 ) Ġof Ġthe Ġinstances Ġfrom ĠIN LINE FORM 10 Ġand Ġstarts Ġthe Ġtraining . ĠFor ĠIN LINE FORM 11 Ġ, ĠLi Li Ġuses Ġa ĠL ST M ĠB IB REF 40 Ġto Ġcompose Ġthe Ġvector Ġrepresentation Ġof Ġeach Ġfeature ĠIN LINE FORM 12 Ġas ĠIN LINE FORM 13 Ġand Ġvector Ġrepresentation Ġof ĠIN LINE FORM 14 Ġas ĠIN LINE FORM 15 Ġ. ĠNext , ĠLi Li Ġcomp utes Ġthe Ġprediction Ġvalue , ĠIN LINE FORM 16 Ġas Ġs igm oid Ġof Ġthe Ġmean Ġcos ine Ġsimilarity Ġof Ġall Ġfeatures Ġand ĠIN LINE FORM 17 Ġ, Ġgiven Ġby ĠIN LINE FORM 18 Ġ) Ġand Ġmaximize Ġthe Ġlog - like lihood Ġof ĠIN LINE FORM 19 Ġfor Ġtraining . ĠOnce ĠIN LINE FORM 20 Ġis Ġtrained , ĠLi Li Ġupdates ĠIN LINE FORM 21 Ġ[ ĠIN LINE FORM 22 Ġ] Ġusing ĠIN LINE FORM 23 Ġ. ĠWe Ġalso Ġtrain Ġan Ġinverse Ġmodel Ġfor ĠIN LINE FORM 24 Ġ, ĠIN LINE FORM 25 Ġby Ġreversing Ġthe Ġpath Ġfeatures Ġin ĠIN LINE FORM 26 Ġand ĠIN LINE FORM 27 Ġwhich Ġhelp Ġin Ġlifelong Ġlearning Ġ( disc ussed Ġbelow ). ĠUnlike ĠB IB REF 20 Ġ, ĠB IB REF 21 Ġ, Ġwhile Ġpredicting Ġthe Ġlabel Ġfor ĠIN LINE FORM 28 Ġ, Ġwe Ġcompute Ġa Ġrelation - specific Ġprediction Ġthreshold ĠIN LINE FORM 29 Ġcorresponding Ġto ĠIN LINE FORM 30 Ġusing ĠIN LINE FORM 31 Ġas : ĠIN LINE FORM 32 Ġand Ġinfer ĠIN LINE FORM 33 Ġas Ġ+ ve Ġif ĠIN LINE FORM 34 Ġand Ġ- ve Ġotherwise . ĠHere , ĠIN LINE FORM 35 Ġ( ĠIN LINE FORM 36 Ġ) Ġis Ġthe Ġmean Ġprediction Ġvalue Ġfor Ġall Ġ+ ve Ġ(- ve ) Ġexamples Ġin ĠIN LINE FORM 37 Ġ. </s> Mod els Ġtrained Ġon Ġa Ġfew Ġexamples Ġ( e . g ., Ġclues Ġacquired Ġfor Ġunknown ĠIN LINE FORM 0 Ġ) Ġwith Ġrandomly Ġinitialized Ġweights Ġoften Ġperform Ġpoorly Ġdue Ġto Ġunder fitting . ĠThus , Ġwe Ġtransfer Ġknowledge Ġ( weights ) Ġfrom Ġthe Ġpast Ġmost Ġsimilar Ġ( w rt ĠIN LINE FORM 1 Ġ) Ġtask Ġin Ġa Ġlifelong Ġlearning Ġmanner ĠB IB REF 31 Ġ. ĠLi Li Ġuses ĠIN LINE FORM 2 Ġto Ġfind Ġthe Ġpast Ġmost Ġsimilar Ġtask Ġfor ĠIN LINE FORM 3 Ġas Ġfollows : ĠLi Li Ġcomp utes Ġtr anc ated ĠS VD Ġof ĠIN LINE FORM 4 Ġas ĠIN LINE FORM 5 Ġand Ġthen , Ġthe Ġsimilarity Ġmatrix ĠIN LINE FORM 6 Ġ. ĠIN LINE FORM 7 Ġprovides Ġthe Ġsimilarity Ġbetween Ġrelations ĠIN LINE FORM 8 Ġand ĠIN LINE FORM 9 Ġin ĠIN LINE FORM 10 Ġ. ĠThus , ĠLi Li Ġchooses Ġa Ġsource Ġrelation ĠIN LINE FORM 11 Ġto Ġtransfer Ġweights . ĠHere , ĠIN LINE FORM 12 Ġis Ġthe Ġset Ġof Ġall ĠIN LINE FORM 13 Ġand ĠIN LINE FORM 14 Ġfor Ġwhich ĠLi Li Ġhas Ġalready Ġlearned Ġa Ġprediction Ġmodel . ĠNow , Ġif ĠIN LINE FORM 15 Ġor ĠIN LINE FORM 16 Ġ, ĠLi Li Ġrandomly Ġinitial izes Ġthe Ġweights ĠIN LINE FORM 17 Ġfor ĠIN LINE FORM 18 Ġand Ġproceeds Ġwith Ġthe Ġtraining . ĠOtherwise , ĠLi Li Ġuses ĠIN LINE FORM 19 Ġas Ġinitial Ġweights Ġand Ġfine - t unes ĠIN LINE FORM 20 Ġwith Ġa Ġlow Ġlearning Ġrate . </s> A ĠRunning ĠExample . ĠConsidering Ġthe Ġexample Ġshown Ġin ĠFigure Ġ1 , ĠLi Li Ġworks Ġas Ġfollows : Ġfirst , ĠLi Li Ġexecutes ĠIN LINE FORM 0 Ġand Ġdetects Ġthat Ġthe Ġsource Ġentity ĠâĢ ľ Obama " Ġand Ġquery Ġrelation ĠâĢ ľ C itizen Of " Ġare Ġunknown . ĠThus , ĠLi Li Ġexecutes ĠIN LINE FORM 1 Ġto Ġacquire Ġclue Ġ( SF 1 ) Ġfor ĠâĢ ľ C itizen Of " Ġand Ġpushes Ġthe Ġclue Ġ(+ ve Ġexample ) Ġand Ġtwo Ġgenerated Ġ- ve Ġexamples Ġinto ĠIN LINE FORM 2 Ġ. ĠOnce Ġthe Ġclues Ġare Ġprocessed Ġand Ġa Ġprediction Ġmodel Ġis Ġtrained Ġfor ĠâĢ ľ C itizen Of " Ġby Ġform ulating Ġseparate Ġstrategies Ġfor Ġthem , ĠLi Li Ġbecomes Ġaware Ġof ĠâĢ ľ C itizen Of ". ĠNow , Ġas Ġthe Ġclues Ġhave Ġalready Ġbeen Ġpopped Ġfrom ĠIN LINE FORM 3 Ġ, Ġthe Ġquery Ġbecomes ĠIN LINE FORM 4 Ġand Ġthe Ġstrategy Ġformulation Ġprocess Ġfor Ġthe Ġquery Ġresumes . ĠNext , ĠLi Li Ġasks Ġuser Ġto Ġprovide Ġa Ġconnecting Ġlink Ġfor ĠâĢ ľ Obama " Ġby Ġperforming ĠIN LINE FORM 5 Ġ. ĠNow , Ġthe Ġquery Ġentities Ġand Ġrelation Ġbeing Ġknown , ĠLi Li Ġenumer ates Ġpaths Ġbetween ĠâĢ ľ Obama " Ġand ĠâĢ ľ USA " Ġby Ġperforming ĠIN LINE FORM 6 Ġ. ĠLet Ġan Ġextracted Ġpath Ġbe ĠâĢ ľ ĠIN LINE FORM 7 Ġ" Ġwith Ġmissing Ġlink Ġbetween Ġ( ĠIN LINE FORM 8 Ġ, ĠIN LINE FORM 9 Ġ). ĠLi Li Ġasks Ġthe Ġuser Ġto Ġfill Ġthe Ġlink Ġby Ġperforming ĠIN LINE FORM 10 Ġand Ġthen , Ġextracts Ġthe Ġcomplete Ġfeature ĠâĢ ľ ĠIN LINE FORM 11 Ġ". ĠThe Ġfeature Ġset Ġis Ġthen Ġfed Ġto Ġthe Ġprediction Ġmodel Ġand Ġinference Ġis Ġmade Ġas Ġa Ġresult Ġof ĠIN LINE FORM 12 Ġ. ĠThus , Ġthe Ġformulated Ġinference Ġstrategy Ġis : ĠâĢ ľ ĠIN LINE FORM 13 Ġ". </s> Exper iments </s> We Ġnow Ġevaluate ĠLi Li Ġin Ġterms Ġof Ġits Ġpredictive Ġperformance Ġand Ġstrategy Ġformulation Ġabilities . </s> Data : ĠWe Ġuse Ġtwo Ġstandard Ġdatasets Ġ( see ĠTable Ġ4 ): Ġ( 1 ) ĠFree base ĠFB 15 k , Ġand Ġ( 2 ) ĠWord Net ĠIN LINE FORM 0 Ġ. ĠUsing Ġeach Ġdataset , Ġwe Ġbuild Ġa Ġfairly Ġlarge Ġgraph Ġand Ġuse Ġit Ġas Ġthe Ġoriginal ĠKB Ġ( ĠIN LINE FORM 1 Ġ) Ġfor Ġevaluation . ĠWe Ġalso Ġaugment ĠIN LINE FORM 2 Ġwith Ġinverse Ġtri ples Ġ( ĠIN LINE FORM 3 Ġ, ĠIN LINE FORM 4 Ġ, ĠIN LINE FORM 5 Ġ) Ġfor Ġeach Ġ( ĠIN LINE FORM 6 Ġ, ĠIN LINE FORM 7 Ġ, ĠIN LINE FORM 8 Ġ) Ġfollowing Ġexisting ĠK BC Ġmethods . </s> Parameter ĠSettings . ĠUnless Ġspecified , Ġthe Ġempir ically Ġset Ġparameters Ġ( see ĠTable Ġ1 ) Ġof ĠLi Li Ġare : ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġ, ĠIN LINE FORM 3 Ġ, ĠIN LINE FORM 4 Ġ, ĠIN LINE FORM 5 Ġ, ĠIN LINE FORM 6 Ġ, ĠIN LINE FORM 7 Ġ, ĠIN LINE FORM 8 Ġ, ĠIN LINE FORM 9 Ġ, ĠIN LINE FORM 10 Ġ. ĠFor Ġtraining ĠRL - model Ġwith ĠIN LINE FORM 11 Ġ- gre edy Ġstrategy , Ġwe Ġuse ĠIN LINE FORM 12 Ġ, ĠIN LINE FORM 13 Ġ, Ġpre - training Ġsteps = 5 0000 . ĠWe Ġused ĠKer as Ġdeep Ġlearning Ġlibrary Ġto Ġimplement Ġand Ġtrain Ġthe Ġprediction Ġmodel . ĠWe Ġset Ġbatch - size Ġas Ġ128 , Ġmax . Ġtraining Ġepoch Ġas Ġ150 , Ġdrop out Ġas Ġ0 . 2 , Ġhidden Ġunits Ġand Ġembed ding Ġsize Ġas Ġ300 Ġand Ġlearning Ġrate Ġas Ġ5 e - 3 Ġwhich Ġis Ġreduced Ġgradually Ġon Ġplateau Ġwith Ġfactor Ġ0 . 5 Ġand Ġpatience Ġ5 . ĠAdam Ġoptim izer Ġand Ġearly Ġstopping Ġwere Ġused Ġin Ġtraining . ĠWe Ġalso Ġshuffle ĠIN LINE FORM 14 Ġin Ġeach Ġepoch Ġand Ġadjust Ġclass Ġweights Ġin versely Ġproportional Ġto Ġclass Ġfrequencies Ġin ĠIN LINE FORM 15 Ġ. </s> Label ed ĠDat as et ĠGeneration Ġand ĠSim ulated ĠUser ĠCreation . ĠWe Ġcreate Ġa Ġsimulated Ġuser Ġfor Ġeach ĠKB Ġto Ġevaluate ĠLi Li . ĠWe Ġcreate Ġthe Ġlabeled Ġdatasets , Ġthe Ġsimulated Ġuser âĢ Ļ s Ġknowledge Ġbase Ġ( ĠIN LINE FORM 0 Ġ), Ġand Ġthe Ġbase ĠKB Ġ( ĠIN LINE FORM 1 Ġ) Ġfrom ĠIN LINE FORM 2 Ġ. ĠIN LINE FORM 3 Ġused Ġas Ġthe Ġinitial ĠKB Ġgraph Ġ( ĠIN LINE FORM 4 Ġ) Ġof ĠLi Li . </s> We Ġfollowed ĠB IB REF 16 Ġfor Ġlabeled Ġdataset Ġgeneration . ĠFor ĠFree base , Ġwe Ġfound Ġ86 Ġrelations Ġwith ĠIN LINE FORM 0 Ġtri ples Ġand Ġrandomly Ġselected Ġ50 Ġfrom Ġvarious Ġdomains . ĠWe Ġrandomly Ġshuffle Ġthe Ġlist Ġof Ġ50 Ġrelations , Ġselect Ġ25 % Ġof Ġthem Ġas Ġunknown Ġrelations Ġand Ġconsider Ġthe Ġrest Ġ( 75 %) Ġas Ġknown Ġrelations . ĠFor Ġeach Ġknown Ġrelation ĠIN LINE FORM 1 Ġ, Ġwe Ġrandomly Ġshuffle Ġthe Ġlist Ġof Ġdistinct Ġtri ples Ġfor ĠIN LINE FORM 2 Ġ, Ġchoose Ġ1000 Ġtri ples Ġand Ġsplit Ġthem Ġinto Ġ60 % Ġtraining , Ġ10 % Ġvalidation Ġand Ġ20 % Ġtest . ĠRest Ġ10 % Ġalong Ġwith Ġthe Ġleftover Ġ( not Ġincluded Ġin Ġthe Ġlist Ġof Ġ1000 ) Ġtri ples Ġare Ġadded Ġto ĠIN LINE FORM 3 Ġ. ĠFor Ġeach Ġunknown Ġrelation ĠIN LINE FORM 4 Ġ, Ġwe Ġremove Ġall Ġtri ples Ġof ĠIN LINE FORM 5 Ġfrom ĠIN LINE FORM 6 Ġand Ġadd Ġthem Ġto ĠIN LINE FORM 7 Ġ. ĠIn Ġthis Ġprocess , Ġwe Ġalso Ġrandomly Ġchoose Ġ20 % Ġtri ples Ġas Ġtest Ġinstances Ġfor Ġunknown ĠIN LINE FORM 8 Ġwhich Ġare Ġexcluded Ġfrom ĠIN LINE FORM 9 Ġ. ĠNote Ġthat , Ġnow ĠIN LINE FORM 10 Ġhas Ġat Ġleast Ġ10 % Ġof Ġchosen Ġtri ples Ġfor Ġeach ĠIN LINE FORM 11 Ġ( known Ġand Ġunknown ) Ġand Ġso , Ġuser Ġis Ġalways Ġable Ġto Ġprovide Ġclues Ġfor Ġboth Ġcases . ĠFor Ġeach Ġlabeled Ġdataset , Ġwe Ġrandomly Ġchoose Ġ10 % Ġof Ġthe Ġentities Ġpresent Ġin Ġdataset Ġtri ples , Ġremove Ġtri ples Ġinvolving Ġthose Ġentities Ġfrom ĠIN LINE FORM 12 Ġand Ġadd Ġto ĠIN LINE FORM 13 Ġ. ĠAt Ġthis Ġpoint , ĠIN LINE FORM 14 Ġgets Ġreduced Ġto ĠIN LINE FORM 15 Ġand Ġis Ġused Ġas ĠIN LINE FORM 16 Ġfor ĠLi Li . ĠThe Ġdataset Ġstats Ġin ĠTable Ġ4 Ġshows Ġthat Ġthe Ġbase ĠKB Ġ( 60 % Ġtri ples Ġof ĠIN LINE FORM 17 Ġ) Ġis Ġhighly Ġsparse Ġ( comp ared Ġto Ġoriginal ĠKB ) Ġwhich Ġmakes Ġthe Ġinference Ġtask Ġmuch Ġharder . ĠWord Net Ġdataset Ġbeing Ġsmall , Ġwe Ġselect Ġall Ġ18 Ġrelations Ġfor Ġevaluation Ġand Ġcreate Ġlabeled Ġdataset , ĠIN LINE FORM 18 Ġand ĠIN LINE FORM 19 Ġfollowing ĠFree base . ĠAlthough Ġthe Ġuser Ġmay Ġprovide Ġclues Ġ100 % Ġof Ġthe Ġtime , Ġit Ġoften Ġcannot Ġrespond Ġto ĠML Q s Ġand ĠCL Q s Ġ( due Ġto Ġlack Ġof Ġrequired Ġtri ples / facts ). ĠThus , Ġwe Ġfurther Ġenrich ĠIN LINE FORM 20 Ġwith Ġexternal ĠKB Ġtri ples . </s> Given Ġa Ġrelation ĠIN LINE FORM 0 Ġand Ġan Ġobserved Ġtriple Ġ( ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġ, ĠIN LINE FORM 3 Ġ) Ġin Ġtraining Ġor Ġtesting , Ġthe Ġpair Ġ( ĠIN LINE FORM 4 Ġ, ĠIN LINE FORM 5 Ġ) Ġis Ġregarded Ġas Ġa Ġ+ ve Ġinstance Ġfor ĠIN LINE FORM 6 Ġ. ĠFollowing ĠB IB REF 18 Ġ, Ġfor Ġeach Ġ+ ve Ġinstance Ġ( ĠIN LINE FORM 7 Ġ, ĠIN LINE FORM 8 Ġ), Ġwe Ġgenerate Ġtwo Ġnegative Ġones , Ġone Ġby Ġrandomly Ġcorrupt ing Ġthe Ġsource ĠIN LINE FORM 9 Ġ, Ġand Ġthe Ġother Ġby Ġcorrupt ing Ġthe Ġtarget ĠIN LINE FORM 10 Ġ. ĠNote Ġthat , Ġthe Ġtest Ġtri ples Ġare Ġnot Ġin ĠIN LINE FORM 11 Ġor ĠIN LINE FORM 12 Ġand Ġnone Ġof Ġthe Ġ- ve Ġinstances Ġoverlap Ġwith Ġthe Ġ+ ve Ġones . </s> Bas elines . ĠAs Ġnone Ġof Ġthe Ġexisting ĠK BC Ġmethods Ġcan Ġsolve Ġthe ĠOK BC Ġproblem , Ġwe Ġchoose Ġvarious Ġversions Ġof ĠLi Li Ġas Ġbas elines . </s> Single : ĠVersion Ġof ĠLi Li Ġwhere Ġwe Ġtrain Ġa Ġsingle Ġprediction Ġmodel ĠIN LINE FORM 0 Ġfor Ġall Ġtest Ġrelations . </s> Sep : ĠWe Ġdo Ġnot Ġtransfer Ġ( past Ġlearned ) Ġweights Ġfor Ġinitial izing ĠIN LINE FORM 0 Ġ, Ġi . e ., Ġwe Ġdisable ĠLL . </s> F - th ): ĠHere , Ġwe Ġuse Ġa Ġfixed Ġprediction Ġthreshold Ġ0 . 5 Ġinstead Ġof Ġrelation - specific Ġthreshold ĠIN LINE FORM 0 Ġ. </s> BG : ĠThe Ġmissing Ġor Ġconnecting Ġlinks Ġ( when Ġthe Ġuser Ġdoes Ġnot Ġrespond ) Ġare Ġfilled Ġwith ĠâĢ ľ @ - Related To - @ " Ġblindly , Ġno Ġguessing Ġmechanism . </s> w / o ĠPTS : ĠLi Li Ġdoes Ġnot Ġask Ġfor Ġadditional Ġclues Ġvia Ġpast Ġtask Ġselection Ġfor Ġskills et Ġimprovement . </s> E val uation ĠMet rics . ĠTo Ġevaluate Ġthe Ġstrategy Ġformulation Ġability , Ġwe Ġintroduce Ġa Ġmeasure Ġcalled ĠCoverage ( ĠIN LINE FORM 0 Ġ), Ġdefined Ġas Ġthe Ġfraction Ġof Ġtotal Ġquery Ġdata Ġinstances , Ġfor Ġwhich ĠLi Li Ġhas Ġsuccessfully Ġformulated Ġstrategies Ġthat Ġlead Ġto Ġwinning . ĠIf ĠLi Li Ġwins Ġon Ġall Ġepisodes Ġfor Ġa Ġgiven Ġdataset , ĠIN LINE FORM 1 Ġis Ġ1 . 0 . ĠTo Ġevaluate Ġthe Ġpredictive Ġperformance , Ġwe Ġuse ĠAvg . ĠM CC Ġand Ġavg . Ġ+ ve ĠF 1 Ġscore . </s> Results Ġand ĠAnalysis </s> E val uation - I : ĠStrategy ĠForm ulation ĠAbility . ĠTable Ġ5 Ġshows Ġthe Ġlist Ġof Ġinference Ġstrategies Ġformulated Ġby ĠLi Li Ġfor Ġvarious ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġ, Ġwhich Ġcontrol Ġthe Ġstrategy Ġformulation Ġof ĠLi Li . ĠWhen ĠIN LINE FORM 2 Ġ, ĠLi Li Ġcannot Ġinteract Ġwith Ġuser Ġand Ġworks Ġlike Ġa Ġclosed - world Ġmethod . ĠThus , ĠIN LINE FORM 3 Ġdrops Ġsignificantly Ġ( 0 . 47 ). ĠWhen ĠIN LINE FORM 4 Ġ, Ġi . e . Ġwith Ġonly Ġone Ġinteraction Ġper Ġquery , ĠLi Li Ġacqu ires Ġknowledge Ġwell Ġfor Ġinstances Ġwhere Ġeither Ġof Ġthe Ġentities Ġor Ġrelation Ġis Ġunknown . ĠHowever , Ġas Ġone Ġunknown Ġentity Ġmay Ġappear Ġin Ġmultiple Ġtest Ġtri ples , Ġonce Ġthe Ġentity Ġbecomes Ġknown , ĠLi Li Ġdoesn âĢ Ļ t Ġneed Ġto Ġask Ġfor Ġit Ġagain Ġand Ġcan Ġperform Ġinference Ġon Ġfuture Ġtri ples Ġcausing Ġsignificant Ġincrease Ġin ĠIN LINE FORM 5 Ġ( 0 . 97 ). ĠWhen ĠIN LINE FORM 6 Ġ, ĠLi Li Ġis Ġable Ġto Ġperform Ġinference Ġon Ġall Ġinstances Ġand ĠIN LINE FORM 7 Ġbecomes Ġ1 . ĠFor ĠIN LINE FORM 8 Ġ, ĠLi Li Ġuses ĠIN LINE FORM 9 Ġonly Ġonce Ġ( as Ġonly Ġone ĠML Q Ġsatisfies ĠIN LINE FORM 10 Ġ) Ġcompared Ġto ĠIN LINE FORM 11 Ġ. ĠIn Ġsummary , ĠLi Li âĢ Ļ s ĠRL - model Ġcan Ġeffectively Ġformulate Ġquery - specific Ġinference Ġstrategies Ġ( based Ġon Ġspecified Ġparameter Ġvalues ). ĠEvaluation - II : ĠPred ictive ĠPerformance . ĠTable Ġ6 Ġshows Ġthe Ġcomparative Ġperformance Ġof ĠLi Li Ġwith Ġbas elines . ĠTo Ġjudge Ġthe Ġoverall Ġimprovements , Ġwe Ġperformed Ġpaired Ġt - test Ġconsidering Ġ+ ve ĠF 1 Ġscores Ġon Ġeach Ġrelation Ġas Ġpaired Ġdata . ĠConsidering Ġboth ĠKB s Ġand Ġall Ġrelation Ġtypes , ĠLi Li Ġoutper forms ĠSep Ġwith ĠIN LINE FORM 12 Ġ. ĠIf Ġwe Ġset ĠIN LINE FORM 13 Ġ( training Ġwith Ġvery Ġfew Ġclues ), ĠLi Li Ġoutper forms ĠSep Ġwith ĠIN LINE FORM 14 Ġon ĠFree base Ġconsidering ĠM CC . ĠThus , Ġthe Ġlifelong Ġlearning Ġmechanism Ġis Ġeffective Ġin Ġtransferring Ġhelpful Ġknowledge . ĠSingle Ġmodel Ġperforms Ġbetter Ġthan ĠSep Ġfor Ġunknown Ġrelations Ġdue Ġto Ġthe Ġsharing Ġof Ġknowledge Ġ( weights ) Ġacross Ġtasks . ĠHowever , Ġfor Ġknown Ġrelations , Ġperformance Ġdrops Ġbecause , Ġas Ġa Ġnew Ġrelation Ġarrives Ġto Ġthe Ġsystem , Ġold Ġweights Ġget Ġcorrupted Ġand Ġcatastrophic Ġforgetting Ġoccurs . ĠFor Ġunknown Ġrelations , Ġas Ġthe Ġrelations Ġare Ġevaluated Ġjust Ġafter Ġtraining , Ġthere Ġis Ġno Ġchance Ġfor Ġcatastrophic Ġforgetting . ĠThe Ġperformance Ġimprovement Ġ( ĠIN LINE FORM 15 Ġ) Ġof ĠLi Li Ġover ĠF - th Ġon ĠFree base Ġsignifies Ġthat Ġthe Ġrelation - specific Ġthreshold ĠIN LINE FORM 16 Ġworks Ġbetter Ġthan Ġfixed Ġthreshold Ġ0 . 5 Ġbecause , Ġif Ġall Ġprediction Ġvalues Ġfor Ġtest Ġinstances Ġlie Ġabove Ġ( or Ġbelow ) Ġ0 . 5 , ĠF - th Ġpredicts Ġall Ġinstances Ġas Ġ+ ve Ġ(- ve ) Ġwhich Ġde grades Ġits Ġperformance . ĠDue Ġto Ġthe Ġutilization Ġof Ġcontextual Ġsimilarity Ġ( highly Ġcorrelated Ġwith Ġclass Ġlabels ) Ġof Ġentity - p airs , ĠLi Li âĢ Ļ s Ġguessing Ġmechanism Ġworks Ġbetter Ġ( ĠIN LINE FORM 17 Ġ) Ġthan Ġblind Ġguessing Ġ( BG ). ĠThe Ġpast Ġtask Ġselection Ġmechanism Ġof ĠLi Li Ġalso Ġimproves Ġits Ġperformance Ġover Ġw / o ĠPTS , Ġas Ġit Ġacqu ires Ġmore Ġclues Ġduring Ġtesting Ġfor Ġpoorly Ġperformed Ġtasks Ġ( eval uated Ġon Ġvalidation Ġset ). ĠFor ĠFree base , Ġdue Ġto Ġa Ġlarge Ġnumber Ġof Ġpast Ġtasks Ġ[ 9 Ġ( 25 % Ġof Ġ38 ) ], Ġthe Ġperformance Ġdifference Ġis Ġmore Ġsignificant Ġ( ĠIN LINE FORM 18 Ġ). ĠFor ĠWord Net , Ġthe Ġnumber Ġis Ġrelatively Ġsmall Ġ[ 3 Ġ( 25 % Ġof Ġ14 )] Ġand Ġhence , Ġthe Ġdifference Ġis Ġnot Ġsignificant . </s> E val uation - III : ĠUser ĠInter action Ġvs . ĠPerformance . ĠTable Ġ7 Ġshows Ġthe Ġresults Ġof ĠLi Li Ġby Ġvarying Ġclue Ġacquisition Ġrate Ġ( ĠIN LINE FORM 0 Ġ). ĠWe Ġuse ĠFree base Ġfor Ġtuning ĠIN LINE FORM 1 Ġdue Ġto Ġits Ġhigher Ġnumber Ġof Ġunknown Ġtest Ġrelations Ġcompared Ġto ĠWord Net . ĠLi Li âĢ Ļ s Ġperformance Ġimproves Ġsignificantly Ġas Ġit Ġacqu ires Ġmore Ġclues Ġfrom Ġthe Ġuser . ĠThe Ġresults Ġon ĠIN LINE FORM 2 Ġoutper forms Ġ( ĠIN LINE FORM 3 Ġ) Ġthat Ġon ĠIN LINE FORM 4 Ġ. ĠTable Ġ8 Ġshows Ġthe Ġresults Ġof ĠLi Li Ġon Ġuser Ġresponses Ġto ĠML Q âĢ Ļ s Ġand ĠCL Q âĢ Ļ s . ĠAn sw ering ĠML Q âĢ Ļ s Ġand ĠCL Q âĢ Ļ s Ġis Ġvery Ġhard Ġfor Ġsimulated Ġusers Ġ( un like Ġcrowd - s ourcing ) Ġas Ġoften ĠIN LINE FORM 5 Ġlacks Ġthe Ġrequired Ġtriple . ĠThus , Ġwe Ġattempt Ġto Ġanalyze Ġhow Ġthe Ġperformance Ġis Ġeffected Ġif Ġthe Ġuser Ġdoes Ġnot Ġrespond Ġat Ġall . ĠThe Ġresults Ġshow Ġa Ġclear Ġtrend Ġin Ġoverall Ġperformance Ġimprovement Ġwhen Ġthe Ġuser Ġresponds . ĠHowever , Ġthe Ġimprovement Ġis Ġnot Ġsignificant Ġas Ġthe Ġsimulated Ġuser âĢ Ļ s Ġquery Ġsatisfaction Ġrate Ġ( 1 % ĠML Q s Ġand Ġ10 % ĠCL Q s ) Ġis Ġvery Ġsmall . ĠBut , Ġthe Ġanalysis Ġshows Ġthe Ġeffectiveness Ġof ĠLi Li âĢ Ļ s Ġguessing Ġmechanism Ġand Ġcontinual Ġlearning Ġability Ġthat Ġhelp Ġin Ġachieving Ġavg . Ġ+ ve ĠF 1 Ġof Ġ0 . 57 Ġand Ġ0 . 62 Ġon ĠFB Ġand ĠW N Ġrespectively Ġwith Ġminimal Ġparticipation Ġof Ġthe Ġuser . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġare Ġinterested Ġin Ġbuilding Ġa Ġgeneric Ġengine Ġfor Ġcontinuous Ġknowledge Ġlearning Ġin Ġhuman - machine Ġconversations . ĠWe Ġfirst Ġshowed Ġthat Ġthe Ġproblem Ġunderlying Ġthe Ġengine Ġcan Ġbe Ġformulated Ġas Ġan Ġopen - world Ġknowledge Ġbase Ġcompletion Ġ( OK BC ) Ġproblem . ĠWe Ġthen Ġproposed Ġan Ġlifelong Ġinteractive Ġlearning Ġand Ġinference Ġ( Li Li ) Ġapproach Ġto Ġsolving Ġthe ĠOK BC Ġproblem . ĠOK BC Ġis Ġa Ġgeneral ization Ġof ĠK BC . ĠLi Li Ġsolves Ġthe ĠOK BC Ġproblem Ġby Ġfirst Ġform ulating Ġa Ġquery - specific Ġinference Ġstrategy Ġusing ĠRL Ġand Ġthen Ġexecuting Ġit Ġto Ġsolve Ġthe Ġproblem Ġby Ġinteracting Ġwith Ġthe Ġuser Ġin Ġa Ġlifelong Ġlearning Ġmanner . ĠExperimental Ġresults Ġshowed Ġthe Ġeffectiveness Ġof ĠLi Li Ġin Ġterms Ġof Ġboth Ġpredictive Ġquality Ġand Ġstrategy Ġformulation Ġability . ĠWe Ġbelieve Ġthat Ġa Ġsystem Ġwith Ġthe ĠLi Li Ġapproach Ġcan Ġserve Ġas Ġa Ġknowledge Ġlearning Ġengine Ġfor Ġconversations . ĠOur Ġfuture Ġwork Ġwill Ġimprove ĠLi Li Ġto Ġmake Ġmore Ġaccurate . </s> Acknowled gments </s> This Ġwork Ġwas Ġsupported Ġin Ġpart Ġby ĠNational ĠScience ĠFoundation Ġ( NS F ) Ġunder Ġgrant Ġno . ĠI IS - 14 07 9 27 Ġand ĠI IS - 16 50 900 , Ġand Ġa Ġgift Ġfrom ĠHuawei ĠTechnologies ĠCo ĠLtd .
Introduction Chatbots such as dialog and question-answering systems have a long history in AI and natural language processing. Early such systems were mostly built using markup languages such as AIML, handcrafted conversation generation rules, and/or information retrieval techniques BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 . Recent neural conversation models BIBREF4 , BIBREF5 , BIBREF6 are even able to perform open-ended conversations. However, since they do not use explicit knowledge bases and do not perform inference, they often suffer from generic and dull responses BIBREF5 , BIBREF7 . More recently, BIBREF8 and BIBREF9 proposed to use knowledge bases (KBs) to help generate responses for knowledge-grounded conversation. However, one major weakness of all existing chat systems is that they do not explicitly or implicitly learn new knowledge in the conversation process. This seriously limits the scope of their applications. In contrast, we humans constantly learn new knowledge in our conversations. Even if some existing systems can use very large knowledge bases either harvested from a large data source such as the Web or built manually, these KBs still miss a large number of facts (knowledge) BIBREF10 . It is thus important for a chatbot to continuously learn new knowledge in the conversation process to expand its KB and to improve its conversation ability. In recent years, researchers have studied the problem of KB completion, i.e., inferring new facts (knowledge) automatically from existing facts in a KB. KB completion (KBC) is defined as a binary classification problem: Given a query triple, ( INLINEFORM0 , INLINEFORM1 , INLINEFORM2 ), we want to predict whether the source entity INLINEFORM3 and target entity INLINEFORM4 can be linked by the relation INLINEFORM5 . However, existing approaches BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14 , BIBREF15 , BIBREF16 solve this problem under the closed-world assumption, i.e., INLINEFORM6 , INLINEFORM7 and INLINEFORM8 are all known to exist in the KB. This is a major weakness because it means that no new knowledge or facts may contain unknown entities or relations. Due to this limitation, KBC is clearly not sufficient for knowledge learning in conversations because in a conversation, the user can say anything, which may contain entities and relations that are not already in the KB. In this paper, we remove this assumption of KBC, and allow all INLINEFORM0 , INLINEFORM1 and INLINEFORM2 to be unknown. We call the new problem open-world knowledge base completion (OKBC). OKBC generalizes KBC. Below, we show that solving OKBC naturally provides the ground for knowledge learning and inference in conversations. In essence, we formulate an abstract problem of knowledge learning and inference in conversations as a well-defined OKBC problem in the interactive setting. From the perspective of knowledge learning in conversations, essentially we can extract two key types of information, true facts and queries, from the user utterances. Queries are facts whose truth values need to be determined. Note that we do not study fact or relation extraction in this paper as there is an extensive work on the topic. (1) For a true fact, we will incorporate it into the KB. Here we need to make sure that it is not already in the KB, which involves relation resolution and entity linking. After a fact is added to the KB, we may predict that some related facts involving some existing relations in the KB may also be true (not logical implications as they can be automatically inferred). For example, if the user says “Obama was born in USA,” the system may guess that (Obama, CitizenOf, USA) (meaning that Obama is a citizen of USA) could also be true based on the current KB. To verify this fact, it needs to solve a KBC problem by treating (Obama, CitizenOf, USA) as a query. This is a KBC problem because the fact (Obama, BornIn, USA) extracted from the original sentence has been added to the KB. Then Obama and USA are in the KB. If the KBC problem is solved, it learns a new fact (Obama, CitizenOf, USA) in addition to the extracted fact (Obama, BornIn, USA). (2) For a query fact, e.g., (Obama, BornIn, USA) extracted from the user question “Was Obama born in USA?” we need to solve the OKBC problem if any of “Obama, “BornIn”, or “USA" is not already in the KB. We can see that OKBC is the core of a knowledge learning engine for conversation. Thus, in this paper, we focus on solving it. We assume that other tasks such as fact/relation extraction and resolution and guessing of related facts of an extracted fact are solved by other sub-systems. We solve the OKBC problem by mimicking how humans acquire knowledge and perform reasoning in an interactive conversation. Whenever we encounter an unknown concept or relation while answering a query, we perform inference using our existing knowledge. If our knowledge does not allow us to draw a conclusion, we typically ask questions to others to acquire related knowledge and use it in inference. The process typically involves an inference strategy (a sequence of actions), which interleaves a sequence of processing and interactive actions. A processing action can be the selection of related facts, deriving inference chain, etc., that advances the inference process. An interactive action can be deciding what to ask, formulating a suitable question, etc., that enable us to interact. The process helps grow the knowledge over time and the gained knowledge enables us to communicate better in the future. We call this lifelong interactive learning and inference (LiLi). Lifelong learning is reflected by the facts that the newly acquired facts are retained in the KB and used in inference for future queries, and that the accumulated knowledge in addition to the updated KB including past inference performances are leveraged to guide future interaction and learning. LiLi should have the following capabilities: This setting is ideal for many NLP applications like dialog and question-answering systems that naturally provide the scope for human interaction and demand real-time inference. LiLi starts with the closed-world KBC approach path-ranking (PR) BIBREF11 , BIBREF17 and extends KBC in a major way to open-world knowledge base completion (OKBC). For a relation INLINEFORM0 , PR works by enumerating paths (except single-link path INLINEFORM1 ) between entity-pairs linked by INLINEFORM2 in the KB and use them as features to train a binary classifier to predict whether a query INLINEFORM3 should be in the KB. Here, a path between two entities is a sequence of relations linking them. In our work, we adopt the latest PR method, C-PR BIBREF16 and extend it to make it work in the open-world setting. C-PR enumerates paths by performing bidirectional random walks over the KB graph while leveraging the context of the source-target entity-pair. We also adopt and extend the compositional vector space model BIBREF20 , BIBREF21 with continual learning capability for prediction. Given an OKBC query ( INLINEFORM0 , INLINEFORM1 , INLINEFORM2 ) (e.g., “(Obama, CitizenOf, USA), which means whether Obama a citizen of USA), LiLi interacts with the user (if needed) by dynamically formulating questions (see the interaction example in Figure 1, which will be further explained in §3) and leverages the interactively acquired knowledge (supporting facts (SFs) in the figure) for continued inference. To do so, LiLi formulates a query-specific inference strategy and executes it. We design LiLi in a Reinforcement Learning (RL) setting that performs sub-tasks like formulating and executing strategy, training a prediction model for inference, and knowledge retention for future use. To the best of our knowledge, our work is the first to address the OKBC problem and to propose an interactive learning mechanism to solve it in a continuous or lifelong manner. We empirically verify the effectiveness of LiLi on two standard real-world KBs: Freebase and WordNet. Experimental results show that LiLi is highly effective in terms of its predictive performance and strategy formulation ability. Related Work To the best of our knowledge, we are not aware of any knowledge learning system that can learn new knowledge in the conversation process. This section thus discusses other related work. Among existing KB completion approaches, BIBREF20 extended the vector space model for zero-shot KB inference. However, the model cannot handle unknown entities and can only work on fixed set of unknown relations with known embeddings. Recently, BIBREF22 proposed a method using external text corpus to perform inference on unknown entities. However, the method cannot handle unknown relations. Thus, these methods are not suitable for our open-world setting. None of the existing KB inference methods perform interactive knowledge learning like LiLi. NELL BIBREF23 continuously updates its KB using facts extracted from the Web. Our task is very different as we do not do Web fact extraction (which is also useful). We focus on user interactions in this paper. Our work is related to interactive language learning (ILL) BIBREF24 , BIBREF25 , but these are not about KB completion. The work in BIBREF26 allows a learner to ask questions in dialogue. However, this work used RL to learn about whether to ask the user or not. The “what to ask aspect" was manually designed by modeling synthetic tasks. LiLi formulates query-specific inference strategies which embed interaction behaviors. Also, no existing dialogue systems BIBREF4 , BIBREF27 , BIBREF28 , BIBREF29 , BIBREF30 employ lifelong learning to train prediction models by using information/knowledge retained in the past. Our work is related to general lifelong learning in BIBREF31 , BIBREF32 , BIBREF33 , BIBREF34 , BIBREF35 , BIBREF36 . However, they learn only one type of tasks, e.g., supervised, topic modeling or reinforcement learning (RL) tasks. None of them is suitable for our setting, which involves interleaving of RL, supervised and interactive learning. More details about lifelong learning can be found in the book BIBREF31 . Interactive Knowledge Learning (LiLi) We design LiLi as a combination of two interconnected models: (1) a RL model that learns to formulate a query-specific inference strategy for performing the OKBC task, and (2) a lifelong prediction model to predict whether a triple should be in the KB, which is invoked by an action while executing the inference strategy and is learned for each relation as in C-PR. The framework improves its performance over time through user interaction and knowledge retention. Compared to the existing KB inference methods, LiLi overcomes the following three challenges for OKBC: 1. Mapping open-world to close-world. Being a closed-world method, C-PR cannot extract path features and learn a prediction model when any of INLINEFORM0 , INLINEFORM1 or INLINEFORM2 is unknown. LiLi solves this problem through interactive knowledge acquisition. If INLINEFORM3 is unknown, LiLi asks the user to provide a clue (an example of INLINEFORM4 ). And if INLINEFORM5 or INLINEFORM6 is unknown, LiLi asks the user to provide a link (relation) to connect the unknown entity with an existing entity (automatically selected) in the KB. We refer to such a query as a connecting link query (CLQ). The acquired knowledge reduces OKBC to KBC and makes the inference task feasible. 2. Spareseness of KB. A main issue of all PR methods like C-PR is the connectivity of the KB graph. If there is no path connecting INLINEFORM0 and INLINEFORM1 in the graph, path enumeration of C-PR gets stuck and inference becomes infeasible. In such cases, LiLi uses a template relation (“@-?-@") as the missing link marker to connect entity-pairs and continues feature extraction. A path containing “@-?-@" is called an incomplete path. Thus, the extracted feature set contains both complete (no missing link) and incomplete paths. Next, LiLi selects an incomplete path from the feature set and asks the user to provide a link for path completion. We refer to such a query as missing link query (MLQ). 3. Limitation in user knowledge. If the user is unable to respond to MLQs or CLQs, LiLi uses a guessing mechanism (discussed later) to fill the gap. This enables LiLi to continue its inference even if the user cannot answer a system question. Components of LiLi As lifelong learning needs to retain knowledge learned from past tasks and use it to help future learning BIBREF31 , LiLi uses a Knowledge Store (KS) for knowledge retention. KS has four components: (i) Knowledge Graph ( INLINEFORM0 ): INLINEFORM1 (the KB) is initialized with base KB triples (see §4) and gets updated over time with the acquired knowledge. (ii) Relation-Entity Matrix ( INLINEFORM2 ): INLINEFORM3 is a sparse matrix, with rows as relations and columns as entity-pairs and is used by the prediction model. Given a triple ( INLINEFORM4 , INLINEFORM5 , INLINEFORM6 ) INLINEFORM7 , we set INLINEFORM8 [ INLINEFORM9 , ( INLINEFORM10 , INLINEFORM11 )] = 1 indicating INLINEFORM12 occurs for pair ( INLINEFORM13 , INLINEFORM14 ). (iii) Task Experience Store ( INLINEFORM15 ): INLINEFORM16 stores the predictive performance of LiLi on past learned tasks in terms of Matthews correlation coefficient (MCC) that measures the quality of binary classification. So, for two tasks INLINEFORM17 and INLINEFORM18 (each relation is a task), if INLINEFORM19 [ INLINEFORM20 ] INLINEFORM21 INLINEFORM22 [ INLINEFORM23 ] [where INLINEFORM24 [ INLINEFORM25 ]=MCC( INLINEFORM26 )], we say C-PR has learned INLINEFORM27 well compared to INLINEFORM28 . (iv) Incomplete Feature DB ( INLINEFORM29 ): INLINEFORM30 stores the frequency of an incomplete path INLINEFORM31 in the form of a tuple ( INLINEFORM32 , INLINEFORM33 , INLINEFORM34 ) and is used in formulating MLQs. INLINEFORM35 [( INLINEFORM36 , INLINEFORM37 , INLINEFORM38 )] = INLINEFORM39 implies LiLi has extracted incomplete path INLINEFORM40 INLINEFORM41 times involving entity-pair INLINEFORM42 [( INLINEFORM43 , INLINEFORM44 )] for query relation INLINEFORM45 . The RL model learns even after training whenever it encounters an unseen state (in testing) and thus, gets updated over time. KS is updated continuously over time as a result of the execution of LiLi and takes part in future learning. The prediction model uses lifelong learning (LL), where we transfer knowledge (parameter values) from the model for a past most similar task to help learn for the current task. Similar tasks are identified by factorizing INLINEFORM0 and computing a task similarity matrix INLINEFORM1 . Besides LL, LiLi uses INLINEFORM2 to identify poorly learned past tasks and acquire more clues for them to improve its skillset over time. LiLi also uses a stack, called Inference Stack ( INLINEFORM0 ) to hold query and its state information for RL. LiLi always processes stack top ( INLINEFORM1 [top]). The clues from the user get stored in INLINEFORM2 on top of the query during strategy execution and processed first. Thus, the prediction model for INLINEFORM3 is learned before performing inference on query, transforming OKBC to a KBC problem. Table 1 shows the parameters of LiLi used in the following sections. Working of LiLi Given an OKBC query ( INLINEFORM0 , INLINEFORM1 , INLINEFORM2 ), we represent it as a data instance INLINEFORM3 . INLINEFORM4 consists of INLINEFORM5 (the query triple), INLINEFORM6 (interaction limit set for INLINEFORM7 ), INLINEFORM8 (experience list storing the transition history of MDP for INLINEFORM9 in RL) and INLINEFORM10 (mode of INLINEFORM11 ) denoting if INLINEFORM12 is ` INLINEFORM13 ' (training), ` INLINEFORM14 ' (validation), ` INLINEFORM15 ' (evaluation) or ` INLINEFORM16 ' (clue) instance and INLINEFORM17 (feature set). We denote INLINEFORM18 ( INLINEFORM19 ) as the set of all complete (incomplete) path features in INLINEFORM20 . Given a data instance INLINEFORM21 , LiLi starts its initialization as follows: it sets the state as INLINEFORM22 (based on INLINEFORM23 , explained later), pushes the query tuple ( INLINEFORM24 , INLINEFORM25 ) into INLINEFORM26 and feeds INLINEFORM27 [top] to the RL-model for strategy formulation from INLINEFORM28 . Inference Strategy Formulation. We view solving the strategy formulation problem as learning to play an inference game, where the goal is to formulate a strategy that "makes the inference task possible". Considering PR methods, inference is possible, iff (1) INLINEFORM0 becomes known to its KB (by acquiring clues when INLINEFORM1 is unknown) and (2) path features are extracted between INLINEFORM2 and INLINEFORM3 (which inturn requires INLINEFORM4 and INLINEFORM5 to be known to KB). If these conditions are met at the end of an episode (when strategy formulation finishes for a given query) of the game, LiLi wins and thus, it trains the prediction model for INLINEFORM6 and uses it for inference. LiLi's strategy formulation is modeled as a Markov Decision Process (MDP) with finite state ( INLINEFORM0 ) and action ( INLINEFORM1 ) spaces. A state INLINEFORM2 consists of 10 binary state variables (Table 2), each of which keeps track of results of an action INLINEFORM3 taken by LiLi and thus, records the progress in inference process made so far. INLINEFORM4 is the initial state with all state bits set as 0. If the data instance (query) is a clue [ INLINEFORM5 ], INLINEFORM6 [CLUE] is set as 1. INLINEFORM7 consists of 6 actions (Table 3). INLINEFORM8 , INLINEFORM9 , INLINEFORM10 are processing actions and INLINEFORM11 , INLINEFORM12 , INLINEFORM13 are interactive actions. Whenever INLINEFORM14 is executed, the MDP reaches the terminal state. Given an action INLINEFORM15 in state INLINEFORM16 , if INLINEFORM17 is invalid in INLINEFORM21 or the objective of INLINEFORM22 is unsatisfied (* marked the condition in INLINEFORM23 ), RL receives a negative reward (empirically set); else receives a positive reward.. We use Q-learning BIBREF38 with INLINEFORM24 -greedy strategy to learn the optimal policy for training the RL model. Note that, the inference strategy is independent of KB type and correctness of prediction. Thus, the RL-model is trained only once from scratch (reused thereafter for other KBs) and also, independently of the prediction model. Sometimes the training dataset may not be enough to learn optimal policy for all INLINEFORM0 . Thus, encountering an unseen state during test can make RL-model clueless about the action. Given a state INLINEFORM1 , whenever an invalid INLINEFORM2 is chosen, LiLi remains in INLINEFORM3 . For INLINEFORM4 , LiLi remains in INLINEFORM5 untill INLINEFORM6 (see Table 1 for INLINEFORM7 ). So, if the state remains the same for ( INLINEFORM8 +1) times, it implies LiLi has encountered a fault (an unseen state). RL-model instantly switches to the training mode and randomly explores INLINEFORM9 to learn the optimal action (fault-tolerant learning). While exploring INLINEFORM10 , the model chooses INLINEFORM11 only when it has tried all other INLINEFORM12 to avoid abrupt end of episode. Execution of Actions. At any given point in time, let ( INLINEFORM0 , INLINEFORM1 ) be the current INLINEFORM2 [top], INLINEFORM3 is the chosen action and the current version of KS components are INLINEFORM4 , INLINEFORM5 , INLINEFORM6 and INLINEFORM7 . Then, if INLINEFORM8 is invalid in INLINEFORM9 , LiLi only updates INLINEFORM10 [top] with ( INLINEFORM11 , INLINEFORM12 ) and returns INLINEFORM13 [top] to RL-model. In this process, LiLi adds experience ( INLINEFORM14 , INLINEFORM15 , INLINEFORM16 , INLINEFORM17 ) in INLINEFORM18 and then, replaces INLINEFORM19 [top] with ( INLINEFORM20 , INLINEFORM21 ). If INLINEFORM22 is valid in INLINEFORM23 , LiLi first sets the next state INLINEFORM24 and performs a sequence of operations INLINEFORM25 based on INLINEFORM26 (discussed below). Unless specified, in INLINEFORM27 , LiLi always monitors INLINEFORM28 and if INLINEFORM29 becomes 0, LiLi sets INLINEFORM30 . Also, whenever LiLi asks the user a query, INLINEFORM31 is decremented by 1. Once INLINEFORM32 ends, LiLi updates INLINEFORM33 [top] with ( INLINEFORM34 , INLINEFORM35 ) and returns INLINEFORM36 [top] to RL-model for choosing the next action. In INLINEFORM0 , LiLi searches INLINEFORM1 , INLINEFORM2 , INLINEFORM3 in INLINEFORM4 and sets appropriate bits in INLINEFORM5 (see Table 2). If INLINEFORM6 was unknown before and is just added to INLINEFORM7 or is in the bottom INLINEFORM8 % (see Table 1 for INLINEFORM9 ) of INLINEFORM10 , LiLi randomly sets INLINEFORM14 with probability INLINEFORM15 . If INLINEFORM16 is a clue and INLINEFORM17 , LiLi updates KS with triple INLINEFORM18 , where ( INLINEFORM19 , INLINEFORM20 , INLINEFORM21 ) and ( INLINEFORM22 , INLINEFORM23 , INLINEFORM24 ) gets added to INLINEFORM25 and INLINEFORM26 , INLINEFORM27 are set as 1. In INLINEFORM0 , LiLi asks the user to provide a clue (+ve instance) for INLINEFORM1 and corrupts INLINEFORM2 and INLINEFORM3 of the clue once at a time, to generate -ve instances by sampling nodes from INLINEFORM4 . These instances help in training prediction model for INLINEFORM5 while executing INLINEFORM6 . In INLINEFORM0 , LiLi selects an incomplete path INLINEFORM1 from INLINEFORM2 to formulate MLQ, such that INLINEFORM3 is most frequently observed for INLINEFORM4 and INLINEFORM5 is high, given by INLINEFORM6 . Here, INLINEFORM7 denotes the contextual similarity BIBREF16 of entity-pair INLINEFORM8 . If INLINEFORM9 is high, INLINEFORM10 is more likely to possess a relation between them and so, is a good candidate for formulating MLQ. When the user does not respond to MLQ (or CLQ in INLINEFORM11 ), the guessing mechanism is used, which works as follows: Since contextual similarity of entity-pairs is highly correlated with their class labels BIBREF16 , LiLi divides the similarity range [-1, 1] into three segments, using a low ( INLINEFORM12 ) and high ( INLINEFORM13 ) similarity threshold and replaces the missing link with INLINEFORM14 in INLINEFORM15 to make it complete as follows: If INLINEFORM16 , INLINEFORM17 = “@-LooselyRelatedTo-@"; else if INLINEFORM18 , INLINEFORM19 =“@-NotRelatedTo-@"; Otherwise, INLINEFORM20 =“@-RelatedTo-@". In INLINEFORM0 , LiLi asks CLQs for connecting unknown entities INLINEFORM1 and/or INLINEFORM2 with INLINEFORM3 by selecting the most contextually relevant node (wrt INLINEFORM4 , INLINEFORM5 ) from INLINEFORM6 , given by link INLINEFORM7 . We adopt the contextual relevance idea in BIBREF16 which is computed using word embedding BIBREF39 In INLINEFORM0 , LiLi extracts path features INLINEFORM1 between ( INLINEFORM2 , INLINEFORM3 ) and updates INLINEFORM4 with incomplete features from INLINEFORM5 . LiLi always trains the prediction model with complete features INLINEFORM6 and once INLINEFORM7 or INLINEFORM8 , LiLi stops asking MLQs. Thus, in both INLINEFORM9 and INLINEFORM10 , LiLi always monitors INLINEFORM11 to check for the said requirements and sets INLINEFORM12 to control interactions. In INLINEFORM0 , if LiLi wins the episode, it adds INLINEFORM1 in one of data buffers INLINEFORM2 based on its mode INLINEFORM3 . E.g., if INLINEFORM4 or INLINEFORM5 , INLINEFORM6 is used for training and added to INLINEFORM7 . Similarly validation buffer INLINEFORM8 and evaluation buffer INLINEFORM9 are populated. If INLINEFORM10 , LiLi invokes the prediction model for INLINEFORM11 . Lifelong Relation Prediction. Given a relation INLINEFORM0 , LiLi uses INLINEFORM1 and INLINEFORM2 (see INLINEFORM3 ) to train a prediction model (say, INLINEFORM4 ) with parameters INLINEFORM5 . For a unknown INLINEFORM6 , the clue instances get stored in INLINEFORM7 and INLINEFORM8 . Thus, LiLi populates INLINEFORM9 by taking 10% (see §4) of the instances from INLINEFORM10 and starts the training. For INLINEFORM11 , LiLi uses a LSTM BIBREF40 to compose the vector representation of each feature INLINEFORM12 as INLINEFORM13 and vector representation of INLINEFORM14 as INLINEFORM15 . Next, LiLi computes the prediction value, INLINEFORM16 as sigmoid of the mean cosine similarity of all features and INLINEFORM17 , given by INLINEFORM18 ) and maximize the log-likelihood of INLINEFORM19 for training. Once INLINEFORM20 is trained, LiLi updates INLINEFORM21 [ INLINEFORM22 ] using INLINEFORM23 . We also train an inverse model for INLINEFORM24 , INLINEFORM25 by reversing the path features in INLINEFORM26 and INLINEFORM27 which help in lifelong learning (discussed below). Unlike BIBREF20 , BIBREF21 , while predicting the label for INLINEFORM28 , we compute a relation-specific prediction threshold INLINEFORM29 corresponding to INLINEFORM30 using INLINEFORM31 as: INLINEFORM32 and infer INLINEFORM33 as +ve if INLINEFORM34 and -ve otherwise. Here, INLINEFORM35 ( INLINEFORM36 ) is the mean prediction value for all +ve (-ve) examples in INLINEFORM37 . Models trained on a few examples (e.g., clues acquired for unknown INLINEFORM0 ) with randomly initialized weights often perform poorly due to underfitting. Thus, we transfer knowledge (weights) from the past most similar (wrt INLINEFORM1 ) task in a lifelong learning manner BIBREF31 . LiLi uses INLINEFORM2 to find the past most similar task for INLINEFORM3 as follows: LiLi computes trancated SVD of INLINEFORM4 as INLINEFORM5 and then, the similarity matrix INLINEFORM6 . INLINEFORM7 provides the similarity between relations INLINEFORM8 and INLINEFORM9 in INLINEFORM10 . Thus, LiLi chooses a source relation INLINEFORM11 to transfer weights. Here, INLINEFORM12 is the set of all INLINEFORM13 and INLINEFORM14 for which LiLi has already learned a prediction model. Now, if INLINEFORM15 or INLINEFORM16 , LiLi randomly initializes the weights INLINEFORM17 for INLINEFORM18 and proceeds with the training. Otherwise, LiLi uses INLINEFORM19 as initial weights and fine-tunes INLINEFORM20 with a low learning rate. A Running Example. Considering the example shown in Figure 1, LiLi works as follows: first, LiLi executes INLINEFORM0 and detects that the source entity “Obama" and query relation “CitizenOf" are unknown. Thus, LiLi executes INLINEFORM1 to acquire clue (SF1) for “CitizenOf" and pushes the clue (+ve example) and two generated -ve examples into INLINEFORM2 . Once the clues are processed and a prediction model is trained for “CitizenOf" by formulating separate strategies for them, LiLi becomes aware of “CitizenOf". Now, as the clues have already been popped from INLINEFORM3 , the query becomes INLINEFORM4 and the strategy formulation process for the query resumes. Next, LiLi asks user to provide a connecting link for “Obama" by performing INLINEFORM5 . Now, the query entities and relation being known, LiLi enumerates paths between “Obama" and “USA" by performing INLINEFORM6 . Let an extracted path be “ INLINEFORM7 " with missing link between ( INLINEFORM8 , INLINEFORM9 ). LiLi asks the user to fill the link by performing INLINEFORM10 and then, extracts the complete feature “ INLINEFORM11 ". The feature set is then fed to the prediction model and inference is made as a result of INLINEFORM12 . Thus, the formulated inference strategy is: “ INLINEFORM13 ". Experiments We now evaluate LiLi in terms of its predictive performance and strategy formulation abilities. Data: We use two standard datasets (see Table 4): (1) Freebase FB15k, and (2) WordNet INLINEFORM0 . Using each dataset, we build a fairly large graph and use it as the original KB ( INLINEFORM1 ) for evaluation. We also augment INLINEFORM2 with inverse triples ( INLINEFORM3 , INLINEFORM4 , INLINEFORM5 ) for each ( INLINEFORM6 , INLINEFORM7 , INLINEFORM8 ) following existing KBC methods. Parameter Settings. Unless specified, the empirically set parameters (see Table 1) of LiLi are: INLINEFORM0 , INLINEFORM1 , INLINEFORM2 , INLINEFORM3 , INLINEFORM4 , INLINEFORM5 , INLINEFORM6 , INLINEFORM7 , INLINEFORM8 , INLINEFORM9 , INLINEFORM10 . For training RL-model with INLINEFORM11 -greedy strategy, we use INLINEFORM12 , INLINEFORM13 , pre-training steps=50000. We used Keras deep learning library to implement and train the prediction model. We set batch-size as 128, max. training epoch as 150, dropout as 0.2, hidden units and embedding size as 300 and learning rate as 5e-3 which is reduced gradually on plateau with factor 0.5 and patience 5. Adam optimizer and early stopping were used in training. We also shuffle INLINEFORM14 in each epoch and adjust class weights inversely proportional to class frequencies in INLINEFORM15 . Labeled Dataset Generation and Simulated User Creation. We create a simulated user for each KB to evaluate LiLi. We create the labeled datasets, the simulated user’s knowledge base ( INLINEFORM0 ), and the base KB ( INLINEFORM1 ) from INLINEFORM2 . INLINEFORM3 used as the initial KB graph ( INLINEFORM4 ) of LiLi. We followed BIBREF16 for labeled dataset generation. For Freebase, we found 86 relations with INLINEFORM0 triples and randomly selected 50 from various domains. We randomly shuffle the list of 50 relations, select 25% of them as unknown relations and consider the rest (75%) as known relations. For each known relation INLINEFORM1 , we randomly shuffle the list of distinct triples for INLINEFORM2 , choose 1000 triples and split them into 60% training, 10% validation and 20% test. Rest 10% along with the leftover (not included in the list of 1000) triples are added to INLINEFORM3 . For each unknown relation INLINEFORM4 , we remove all triples of INLINEFORM5 from INLINEFORM6 and add them to INLINEFORM7 . In this process, we also randomly choose 20% triples as test instances for unknown INLINEFORM8 which are excluded from INLINEFORM9 . Note that, now INLINEFORM10 has at least 10% of chosen triples for each INLINEFORM11 (known and unknown) and so, user is always able to provide clues for both cases. For each labeled dataset, we randomly choose 10% of the entities present in dataset triples, remove triples involving those entities from INLINEFORM12 and add to INLINEFORM13 . At this point, INLINEFORM14 gets reduced to INLINEFORM15 and is used as INLINEFORM16 for LiLi. The dataset stats in Table 4 shows that the base KB (60% triples of INLINEFORM17 ) is highly sparse (compared to original KB) which makes the inference task much harder. WordNet dataset being small, we select all 18 relations for evaluation and create labeled dataset, INLINEFORM18 and INLINEFORM19 following Freebase. Although the user may provide clues 100% of the time, it often cannot respond to MLQs and CLQs (due to lack of required triples/facts). Thus, we further enrich INLINEFORM20 with external KB triples. Given a relation INLINEFORM0 and an observed triple ( INLINEFORM1 , INLINEFORM2 , INLINEFORM3 ) in training or testing, the pair ( INLINEFORM4 , INLINEFORM5 ) is regarded as a +ve instance for INLINEFORM6 . Following BIBREF18 , for each +ve instance ( INLINEFORM7 , INLINEFORM8 ), we generate two negative ones, one by randomly corrupting the source INLINEFORM9 , and the other by corrupting the target INLINEFORM10 . Note that, the test triples are not in INLINEFORM11 or INLINEFORM12 and none of the -ve instances overlap with the +ve ones. Baselines. As none of the existing KBC methods can solve the OKBC problem, we choose various versions of LiLi as baselines. Single: Version of LiLi where we train a single prediction model INLINEFORM0 for all test relations. Sep: We do not transfer (past learned) weights for initializing INLINEFORM0 , i.e., we disable LL. F-th): Here, we use a fixed prediction threshold 0.5 instead of relation-specific threshold INLINEFORM0 . BG: The missing or connecting links (when the user does not respond) are filled with “@-RelatedTo-@" blindly, no guessing mechanism. w/o PTS: LiLi does not ask for additional clues via past task selection for skillset improvement. Evaluation Metrics. To evaluate the strategy formulation ability, we introduce a measure called Coverage( INLINEFORM0 ), defined as the fraction of total query data instances, for which LiLi has successfully formulated strategies that lead to winning. If LiLi wins on all episodes for a given dataset, INLINEFORM1 is 1.0. To evaluate the predictive performance, we use Avg. MCC and avg. +ve F1 score. Results and Analysis Evaluation-I: Strategy Formulation Ability. Table 5 shows the list of inference strategies formulated by LiLi for various INLINEFORM0 and INLINEFORM1 , which control the strategy formulation of LiLi. When INLINEFORM2 , LiLi cannot interact with user and works like a closed-world method. Thus, INLINEFORM3 drops significantly (0.47). When INLINEFORM4 , i.e. with only one interaction per query, LiLi acquires knowledge well for instances where either of the entities or relation is unknown. However, as one unknown entity may appear in multiple test triples, once the entity becomes known, LiLi doesn’t need to ask for it again and can perform inference on future triples causing significant increase in INLINEFORM5 (0.97). When INLINEFORM6 , LiLi is able to perform inference on all instances and INLINEFORM7 becomes 1. For INLINEFORM8 , LiLi uses INLINEFORM9 only once (as only one MLQ satisfies INLINEFORM10 ) compared to INLINEFORM11 . In summary, LiLi’s RL-model can effectively formulate query-specific inference strategies (based on specified parameter values). Evaluation-II: Predictive Performance. Table 6 shows the comparative performance of LiLi with baselines. To judge the overall improvements, we performed paired t-test considering +ve F1 scores on each relation as paired data. Considering both KBs and all relation types, LiLi outperforms Sep with INLINEFORM12 . If we set INLINEFORM13 (training with very few clues), LiLi outperforms Sep with INLINEFORM14 on Freebase considering MCC. Thus, the lifelong learning mechanism is effective in transferring helpful knowledge. Single model performs better than Sep for unknown relations due to the sharing of knowledge (weights) across tasks. However, for known relations, performance drops because, as a new relation arrives to the system, old weights get corrupted and catastrophic forgetting occurs. For unknown relations, as the relations are evaluated just after training, there is no chance for catastrophic forgetting. The performance improvement ( INLINEFORM15 ) of LiLi over F-th on Freebase signifies that the relation-specific threshold INLINEFORM16 works better than fixed threshold 0.5 because, if all prediction values for test instances lie above (or below) 0.5, F-th predicts all instances as +ve (-ve) which degrades its performance. Due to the utilization of contextual similarity (highly correlated with class labels) of entity-pairs, LiLi’s guessing mechanism works better ( INLINEFORM17 ) than blind guessing (BG). The past task selection mechanism of LiLi also improves its performance over w/o PTS, as it acquires more clues during testing for poorly performed tasks (evaluated on validation set). For Freebase, due to a large number of past tasks [9 (25% of 38)], the performance difference is more significant ( INLINEFORM18 ). For WordNet, the number is relatively small [3 (25% of 14)] and hence, the difference is not significant. Evaluation-III: User Interaction vs. Performance. Table 7 shows the results of LiLi by varying clue acquisition rate ( INLINEFORM0 ). We use Freebase for tuning INLINEFORM1 due to its higher number of unknown test relations compared to WordNet. LiLi’s performance improves significantly as it acquires more clues from the user. The results on INLINEFORM2 outperforms ( INLINEFORM3 ) that on INLINEFORM4 . Table 8 shows the results of LiLi on user responses to MLQ’s and CLQ’s. Answering MLQ’s and CLQ’s is very hard for simulated users (unlike crowd-sourcing) as often INLINEFORM5 lacks the required triple. Thus, we attempt to analyze how the performance is effected if the user does not respond at all. The results show a clear trend in overall performance improvement when the user responds. However, the improvement is not significant as the simulated user’s query satisfaction rate (1% MLQs and 10% CLQs) is very small. But, the analysis shows the effectiveness of LiLi’s guessing mechanism and continual learning ability that help in achieving avg. +ve F1 of 0.57 and 0.62 on FB and WN respectively with minimal participation of the user. Conclusion In this paper, we are interested in building a generic engine for continuous knowledge learning in human-machine conversations. We first showed that the problem underlying the engine can be formulated as an open-world knowledge base completion (OKBC) problem. We then proposed an lifelong interactive learning and inference (LiLi) approach to solving the OKBC problem. OKBC is a generalization of KBC. LiLi solves the OKBC problem by first formulating a query-specific inference strategy using RL and then executing it to solve the problem by interacting with the user in a lifelong learning manner. Experimental results showed the effectiveness of LiLi in terms of both predictive quality and strategy formulation ability. We believe that a system with the LiLi approach can serve as a knowledge learning engine for conversations. Our future work will improve LiLi to make more accurate. Acknowledgments This work was supported in part by National Science Foundation (NSF) under grant no. IIS-1407927 and IIS-1650900, and a gift from Huawei Technologies Co Ltd.
How much better than the baseline is LiLi?
286078813136943dfafb5155ee15d2429e7601d9
How Ġmuch Ġbetter Ġthan Ġthe Ġbaseline Ġis ĠLi Li ?
In Ġcase Ġof ĠFree base Ġknowledge Ġbase , ĠLi Li Ġmodel Ġhad Ġbetter ĠF 1 Ġscore Ġthan Ġthe Ġsingle Ġmodel Ġby Ġ0 . 20 Ġ, Ġ0 . 01 , Ġ0 . 159 Ġfor Ġk wn , Ġun k , Ġand Ġall Ġtest ĠRel Ġtype . Ġ ĠThe Ġvalues Ġfor ĠWord Net Ġare Ġ0 . 25 , Ġ0 . 1 , Ġ0 . 2 . Ġ Ċ
<s> How Ġmuch Ġbetter Ġthan Ġthe Ġbaseline Ġis ĠLi Li ? </s> Introduction </s> Chat bots Ġsuch Ġas Ġdialog Ġand Ġquestion - ans w ering Ġsystems Ġhave Ġa Ġlong Ġhistory Ġin ĠAI Ġand Ġnatural Ġlanguage Ġprocessing . ĠEarly Ġsuch Ġsystems Ġwere Ġmostly Ġbuilt Ġusing Ġmarkup Ġlanguages Ġsuch Ġas ĠA IM L , Ġhand crafted Ġconversation Ġgeneration Ġrules , Ġand / or Ġinformation Ġretrieval Ġtechniques ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ. ĠRecent Ġneural Ġconversation Ġmodels ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġare Ġeven Ġable Ġto Ġperform Ġopen - ended Ġconversations . ĠHowever , Ġsince Ġthey Ġdo Ġnot Ġuse Ġexplicit Ġknowledge Ġbases Ġand Ġdo Ġnot Ġperform Ġinference , Ġthey Ġoften Ġsuffer Ġfrom Ġgeneric Ġand Ġdull Ġresponses ĠB IB REF 5 Ġ, ĠB IB REF 7 Ġ. ĠMore Ġrecently , ĠB IB REF 8 Ġand ĠB IB REF 9 Ġproposed Ġto Ġuse Ġknowledge Ġbases Ġ( KB s ) Ġto Ġhelp Ġgenerate Ġresponses Ġfor Ġknowledge - ground ed Ġconversation . ĠHowever , Ġone Ġmajor Ġweakness Ġof Ġall Ġexisting Ġchat Ġsystems Ġis Ġthat Ġthey Ġdo Ġnot Ġexplicitly Ġor Ġimplicitly Ġlearn Ġnew Ġknowledge Ġin Ġthe Ġconversation Ġprocess . ĠThis Ġseriously Ġlimits Ġthe Ġscope Ġof Ġtheir Ġapplications . ĠIn Ġcontrast , Ġwe Ġhumans Ġconstantly Ġlearn Ġnew Ġknowledge Ġin Ġour Ġconversations . ĠEven Ġif Ġsome Ġexisting Ġsystems Ġcan Ġuse Ġvery Ġlarge Ġknowledge Ġbases Ġeither Ġharvested Ġfrom Ġa Ġlarge Ġdata Ġsource Ġsuch Ġas Ġthe ĠWeb Ġor Ġbuilt Ġmanually , Ġthese ĠKB s Ġstill Ġmiss Ġa Ġlarge Ġnumber Ġof Ġfacts Ġ( knowledge ) ĠB IB REF 10 Ġ. ĠIt Ġis Ġthus Ġimportant Ġfor Ġa Ġchat bot Ġto Ġcontinuously Ġlearn Ġnew Ġknowledge Ġin Ġthe Ġconversation Ġprocess Ġto Ġexpand Ġits ĠKB Ġand Ġto Ġimprove Ġits Ġconversation Ġability . </s> In Ġrecent Ġyears , Ġresearchers Ġhave Ġstudied Ġthe Ġproblem Ġof ĠKB Ġcompletion , Ġi . e ., Ġinfer ring Ġnew Ġfacts Ġ( knowledge ) Ġautomatically Ġfrom Ġexisting Ġfacts Ġin Ġa ĠKB . ĠKB Ġcompletion Ġ( K BC ) Ġis Ġdefined Ġas Ġa Ġbinary Ġclassification Ġproblem : ĠGiven Ġa Ġquery Ġtriple , Ġ( ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġ), Ġwe Ġwant Ġto Ġpredict Ġwhether Ġthe Ġsource Ġentity ĠIN LINE FORM 3 Ġand Ġtarget Ġentity ĠIN LINE FORM 4 Ġcan Ġbe Ġlinked Ġby Ġthe Ġrelation ĠIN LINE FORM 5 Ġ. ĠHowever , Ġexisting Ġapproaches ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġsolve Ġthis Ġproblem Ġunder Ġthe Ġclosed - world Ġassumption , Ġi . e ., ĠIN LINE FORM 6 Ġ, ĠIN LINE FORM 7 Ġand ĠIN LINE FORM 8 Ġare Ġall Ġknown Ġto Ġexist Ġin Ġthe ĠKB . ĠThis Ġis Ġa Ġmajor Ġweakness Ġbecause Ġit Ġmeans Ġthat Ġno Ġnew Ġknowledge Ġor Ġfacts Ġmay Ġcontain Ġunknown Ġentities Ġor Ġrelations . ĠDue Ġto Ġthis Ġlimitation , ĠK BC Ġis Ġclearly Ġnot Ġsufficient Ġfor Ġknowledge Ġlearning Ġin Ġconversations Ġbecause Ġin Ġa Ġconversation , Ġthe Ġuser Ġcan Ġsay Ġanything , Ġwhich Ġmay Ġcontain Ġentities Ġand Ġrelations Ġthat Ġare Ġnot Ġalready Ġin Ġthe ĠKB . </s> In Ġthis Ġpaper , Ġwe Ġremove Ġthis Ġassumption Ġof ĠK BC , Ġand Ġallow Ġall ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġto Ġbe Ġunknown . ĠWe Ġcall Ġthe Ġnew Ġproblem Ġopen - world Ġknowledge Ġbase Ġcompletion Ġ( OK BC ). ĠOK BC Ġgeneral izes ĠK BC . ĠBelow , Ġwe Ġshow Ġthat Ġsolving ĠOK BC Ġnaturally Ġprovides Ġthe Ġground Ġfor Ġknowledge Ġlearning Ġand Ġinference Ġin Ġconversations . ĠIn Ġessence , Ġwe Ġformulate Ġan Ġabstract Ġproblem Ġof Ġknowledge Ġlearning Ġand Ġinference Ġin Ġconversations Ġas Ġa Ġwell - defined ĠOK BC Ġproblem Ġin Ġthe Ġinteractive Ġsetting . </s> From Ġthe Ġperspective Ġof Ġknowledge Ġlearning Ġin Ġconversations , Ġessentially Ġwe Ġcan Ġextract Ġtwo Ġkey Ġtypes Ġof Ġinformation , Ġtrue Ġfacts Ġand Ġqueries , Ġfrom Ġthe Ġuser Ġutter ances . ĠQu eries Ġare Ġfacts Ġwhose Ġtruth Ġvalues Ġneed Ġto Ġbe Ġdetermined . ĠNote Ġthat Ġwe Ġdo Ġnot Ġstudy Ġfact Ġor Ġrelation Ġextraction Ġin Ġthis Ġpaper Ġas Ġthere Ġis Ġan Ġextensive Ġwork Ġon Ġthe Ġtopic . Ġ( 1 ) ĠFor Ġa Ġtrue Ġfact , Ġwe Ġwill Ġincorporate Ġit Ġinto Ġthe ĠKB . ĠHere Ġwe Ġneed Ġto Ġmake Ġsure Ġthat Ġit Ġis Ġnot Ġalready Ġin Ġthe ĠKB , Ġwhich Ġinvolves Ġrelation Ġresolution Ġand Ġentity Ġlinking . ĠAfter Ġa Ġfact Ġis Ġadded Ġto Ġthe ĠKB , Ġwe Ġmay Ġpredict Ġthat Ġsome Ġrelated Ġfacts Ġinvolving Ġsome Ġexisting Ġrelations Ġin Ġthe ĠKB Ġmay Ġalso Ġbe Ġtrue Ġ( not Ġlogical Ġimplications Ġas Ġthey Ġcan Ġbe Ġautomatically Ġinferred ). ĠFor Ġexample , Ġif Ġthe Ġuser Ġsays ĠâĢ ľ Obama Ġwas Ġborn Ġin ĠUSA , âĢ Ŀ Ġthe Ġsystem Ġmay Ġguess Ġthat Ġ( Obama , ĠCitizen Of , ĠUSA ) Ġ( meaning Ġthat ĠObama Ġis Ġa Ġcitizen Ġof ĠUSA ) Ġcould Ġalso Ġbe Ġtrue Ġbased Ġon Ġthe Ġcurrent ĠKB . ĠTo Ġverify Ġthis Ġfact , Ġit Ġneeds Ġto Ġsolve Ġa ĠK BC Ġproblem Ġby Ġtreating Ġ( Obama , ĠCitizen Of , ĠUSA ) Ġas Ġa Ġquery . ĠThis Ġis Ġa ĠK BC Ġproblem Ġbecause Ġthe Ġfact Ġ( Obama , ĠBorn In , ĠUSA ) Ġextracted Ġfrom Ġthe Ġoriginal Ġsentence Ġhas Ġbeen Ġadded Ġto Ġthe ĠKB . ĠThen ĠObama Ġand ĠUSA Ġare Ġin Ġthe ĠKB . ĠIf Ġthe ĠK BC Ġproblem Ġis Ġsolved , Ġit Ġlearns Ġa Ġnew Ġfact Ġ( Obama , ĠCitizen Of , ĠUSA ) Ġin Ġaddition Ġto Ġthe Ġextracted Ġfact Ġ( Obama , ĠBorn In , ĠUSA ). Ġ( 2 ) ĠFor Ġa Ġquery Ġfact , Ġe . g ., Ġ( Obama , ĠBorn In , ĠUSA ) Ġextracted Ġfrom Ġthe Ġuser Ġquestion ĠâĢ ľ Was ĠObama Ġborn Ġin ĠUSA ? âĢ Ŀ Ġwe Ġneed Ġto Ġsolve Ġthe ĠOK BC Ġproblem Ġif Ġany Ġof ĠâĢ ľ Obama , ĠâĢ ľ Born In âĢ Ŀ , Ġor ĠâĢ ľ USA " Ġis Ġnot Ġalready Ġin Ġthe ĠKB . </s> We Ġcan Ġsee Ġthat ĠOK BC Ġis Ġthe Ġcore Ġof Ġa Ġknowledge Ġlearning Ġengine Ġfor Ġconversation . ĠThus , Ġin Ġthis Ġpaper , Ġwe Ġfocus Ġon Ġsolving Ġit . ĠWe Ġassume Ġthat Ġother Ġtasks Ġsuch Ġas Ġfact / relation Ġextraction Ġand Ġresolution Ġand Ġguessing Ġof Ġrelated Ġfacts Ġof Ġan Ġextracted Ġfact Ġare Ġsolved Ġby Ġother Ġsub - system s . </s> We Ġsolve Ġthe ĠOK BC Ġproblem Ġby Ġmim icking Ġhow Ġhumans Ġacquire Ġknowledge Ġand Ġperform Ġreasoning Ġin Ġan Ġinteractive Ġconversation . ĠWhenever Ġwe Ġencounter Ġan Ġunknown Ġconcept Ġor Ġrelation Ġwhile Ġanswering Ġa Ġquery , Ġwe Ġperform Ġinference Ġusing Ġour Ġexisting Ġknowledge . ĠIf Ġour Ġknowledge Ġdoes Ġnot Ġallow Ġus Ġto Ġdraw Ġa Ġconclusion , Ġwe Ġtypically Ġask Ġquestions Ġto Ġothers Ġto Ġacquire Ġrelated Ġknowledge Ġand Ġuse Ġit Ġin Ġinference . ĠThe Ġprocess Ġtypically Ġinvolves Ġan Ġinference Ġstrategy Ġ( a Ġsequence Ġof Ġactions ), Ġwhich Ġinter le aves Ġa Ġsequence Ġof Ġprocessing Ġand Ġinteractive Ġactions . ĠA Ġprocessing Ġaction Ġcan Ġbe Ġthe Ġselection Ġof Ġrelated Ġfacts , Ġder iving Ġinference Ġchain , Ġetc ., Ġthat Ġadvances Ġthe Ġinference Ġprocess . ĠAn Ġinteractive Ġaction Ġcan Ġbe Ġdeciding Ġwhat Ġto Ġask , Ġform ulating Ġa Ġsuitable Ġquestion , Ġetc ., Ġthat Ġenable Ġus Ġto Ġinteract . ĠThe Ġprocess Ġhelps Ġgrow Ġthe Ġknowledge Ġover Ġtime Ġand Ġthe Ġgained Ġknowledge Ġenables Ġus Ġto Ġcommunicate Ġbetter Ġin Ġthe Ġfuture . ĠWe Ġcall Ġthis Ġlifelong Ġinteractive Ġlearning Ġand Ġinference Ġ( Li Li ). ĠLif elong Ġlearning Ġis Ġreflected Ġby Ġthe Ġfacts Ġthat Ġthe Ġnewly Ġacquired Ġfacts Ġare Ġretained Ġin Ġthe ĠKB Ġand Ġused Ġin Ġinference Ġfor Ġfuture Ġqueries , Ġand Ġthat Ġthe Ġaccumulated Ġknowledge Ġin Ġaddition Ġto Ġthe Ġupdated ĠKB Ġincluding Ġpast Ġinference Ġperformances Ġare Ġlever aged Ġto Ġguide Ġfuture Ġinteraction Ġand Ġlearning . ĠLi Li Ġshould Ġhave Ġthe Ġfollowing Ġcapabilities : </s> This Ġsetting Ġis Ġideal Ġfor Ġmany ĠN LP Ġapplications Ġlike Ġdialog Ġand Ġquestion - ans w ering Ġsystems Ġthat Ġnaturally Ġprovide Ġthe Ġscope Ġfor Ġhuman Ġinteraction Ġand Ġdemand Ġreal - time Ġinference . </s> Li Li Ġstarts Ġwith Ġthe Ġclosed - world ĠK BC Ġapproach Ġpath - ranking Ġ( PR ) ĠB IB REF 11 Ġ, ĠB IB REF 17 Ġand Ġextends ĠK BC Ġin Ġa Ġmajor Ġway Ġto Ġopen - world Ġknowledge Ġbase Ġcompletion Ġ( OK BC ). ĠFor Ġa Ġrelation ĠIN LINE FORM 0 Ġ, ĠPR Ġworks Ġby Ġenumer ating Ġpaths Ġ( except Ġsingle - link Ġpath ĠIN LINE FORM 1 Ġ) Ġbetween Ġentity - p airs Ġlinked Ġby ĠIN LINE FORM 2 Ġin Ġthe ĠKB Ġand Ġuse Ġthem Ġas Ġfeatures Ġto Ġtrain Ġa Ġbinary Ġclass ifier Ġto Ġpredict Ġwhether Ġa Ġquery ĠIN LINE FORM 3 Ġshould Ġbe Ġin Ġthe ĠKB . ĠHere , Ġa Ġpath Ġbetween Ġtwo Ġentities Ġis Ġa Ġsequence Ġof Ġrelations Ġlinking Ġthem . ĠIn Ġour Ġwork , Ġwe Ġadopt Ġthe Ġlatest ĠPR Ġmethod , ĠC - PR ĠB IB REF 16 Ġand Ġextend Ġit Ġto Ġmake Ġit Ġwork Ġin Ġthe Ġopen - world Ġsetting . ĠC - PR Ġenumer ates Ġpaths Ġby Ġperforming Ġbid irection al Ġrandom Ġwalks Ġover Ġthe ĠKB Ġgraph Ġwhile Ġleveraging Ġthe Ġcontext Ġof Ġthe Ġsource - target Ġentity - pair . ĠWe Ġalso Ġadopt Ġand Ġextend Ġthe Ġcompos itional Ġvector Ġspace Ġmodel ĠB IB REF 20 Ġ, ĠB IB REF 21 Ġwith Ġcontinual Ġlearning Ġcapability Ġfor Ġprediction . </s> Given Ġan ĠOK BC Ġquery Ġ( ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġ) Ġ( e . g ., ĠâĢ ľ ( Obama , ĠCitizen Of , ĠUSA ), Ġwhich Ġmeans Ġwhether ĠObama Ġa Ġcitizen Ġof ĠUSA ), ĠLi Li Ġinteracts Ġwith Ġthe Ġuser Ġ( if Ġneeded ) Ġby Ġdynamically Ġform ulating Ġquestions Ġ( see Ġthe Ġinteraction Ġexample Ġin ĠFigure Ġ1 , Ġwhich Ġwill Ġbe Ġfurther Ġexplained Ġin ĠÂ§ 3 ) Ġand Ġlever ages Ġthe Ġinteract ively Ġacquired Ġknowledge Ġ( support ing Ġfacts Ġ( SF s ) Ġin Ġthe Ġfigure ) Ġfor Ġcontinued Ġinference . ĠTo Ġdo Ġso , ĠLi Li Ġform ulates Ġa Ġquery - specific Ġinference Ġstrategy Ġand Ġexecutes Ġit . ĠWe Ġdesign ĠLi Li Ġin Ġa ĠRein forcement ĠLearning Ġ( RL ) Ġsetting Ġthat Ġperforms Ġsub - t asks Ġlike Ġform ulating Ġand Ġexecuting Ġstrategy , Ġtraining Ġa Ġprediction Ġmodel Ġfor Ġinference , Ġand Ġknowledge Ġretention Ġfor Ġfuture Ġuse . ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge , Ġour Ġwork Ġis Ġthe Ġfirst Ġto Ġaddress Ġthe ĠOK BC Ġproblem Ġand Ġto Ġpropose Ġan Ġinteractive Ġlearning Ġmechanism Ġto Ġsolve Ġit Ġin Ġa Ġcontinuous Ġor Ġlifelong Ġmanner . ĠWe Ġempir ically Ġverify Ġthe Ġeffectiveness Ġof ĠLi Li Ġon Ġtwo Ġstandard Ġreal - world ĠKB s : ĠFree base Ġand ĠWord Net . ĠExperimental Ġresults Ġshow Ġthat ĠLi Li Ġis Ġhighly Ġeffective Ġin Ġterms Ġof Ġits Ġpredictive Ġperformance Ġand Ġstrategy Ġformulation Ġability . </s> Related ĠWork </s> To Ġthe Ġbest Ġof Ġour Ġknowledge , Ġwe Ġare Ġnot Ġaware Ġof Ġany Ġknowledge Ġlearning Ġsystem Ġthat Ġcan Ġlearn Ġnew Ġknowledge Ġin Ġthe Ġconversation Ġprocess . ĠThis Ġsection Ġthus Ġdiscusses Ġother Ġrelated Ġwork . </s> Among Ġexisting ĠKB Ġcompletion Ġapproaches , ĠB IB REF 20 Ġextended Ġthe Ġvector Ġspace Ġmodel Ġfor Ġzero - shot ĠKB Ġinference . ĠHowever , Ġthe Ġmodel Ġcannot Ġhandle Ġunknown Ġentities Ġand Ġcan Ġonly Ġwork Ġon Ġfixed Ġset Ġof Ġunknown Ġrelations Ġwith Ġknown Ġembed d ings . ĠRecently , ĠB IB REF 22 Ġproposed Ġa Ġmethod Ġusing Ġexternal Ġtext Ġcorpus Ġto Ġperform Ġinference Ġon Ġunknown Ġentities . ĠHowever , Ġthe Ġmethod Ġcannot Ġhandle Ġunknown Ġrelations . ĠThus , Ġthese Ġmethods Ġare Ġnot Ġsuitable Ġfor Ġour Ġopen - world Ġsetting . ĠNone Ġof Ġthe Ġexisting ĠKB Ġinference Ġmethods Ġperform Ġinteractive Ġknowledge Ġlearning Ġlike ĠLi Li . ĠNE LL ĠB IB REF 23 Ġcontinuously Ġupdates Ġits ĠKB Ġusing Ġfacts Ġextracted Ġfrom Ġthe ĠWeb . ĠOur Ġtask Ġis Ġvery Ġdifferent Ġas Ġwe Ġdo Ġnot Ġdo ĠWeb Ġfact Ġextraction Ġ( which Ġis Ġalso Ġuseful ). ĠWe Ġfocus Ġon Ġuser Ġinteractions Ġin Ġthis Ġpaper . ĠOur Ġwork Ġis Ġrelated Ġto Ġinteractive Ġlanguage Ġlearning Ġ( ILL ) ĠB IB REF 24 Ġ, ĠB IB REF 25 Ġ, Ġbut Ġthese Ġare Ġnot Ġabout ĠKB Ġcompletion . ĠThe Ġwork Ġin ĠB IB REF 26 Ġallows Ġa Ġlear ner Ġto Ġask Ġquestions Ġin Ġdialogue . ĠHowever , Ġthis Ġwork Ġused ĠRL Ġto Ġlearn Ġabout Ġwhether Ġto Ġask Ġthe Ġuser Ġor Ġnot . ĠThe ĠâĢ ľ what Ġto Ġask Ġaspect " Ġwas Ġmanually Ġdesigned Ġby Ġmodeling Ġsynthetic Ġtasks . ĠLi Li Ġform ulates Ġquery - specific Ġinference Ġstrategies Ġwhich Ġembed Ġinteraction Ġbehaviors . ĠAlso , Ġno Ġexisting Ġdialogue Ġsystems ĠB IB REF 4 Ġ, ĠB IB REF 27 Ġ, ĠB IB REF 28 Ġ, ĠB IB REF 29 Ġ, ĠB IB REF 30 Ġemploy Ġlifelong Ġlearning Ġto Ġtrain Ġprediction Ġmodels Ġby Ġusing Ġinformation / knowledge Ġretained Ġin Ġthe Ġpast . </s> Our Ġwork Ġis Ġrelated Ġto Ġgeneral Ġlifelong Ġlearning Ġin ĠB IB REF 31 Ġ, ĠB IB REF 32 Ġ, ĠB IB REF 33 Ġ, ĠB IB REF 34 Ġ, ĠB IB REF 35 Ġ, ĠB IB REF 36 Ġ. ĠHowever , Ġthey Ġlearn Ġonly Ġone Ġtype Ġof Ġtasks , Ġe . g ., Ġsupervised , Ġtopic Ġmodeling Ġor Ġreinforcement Ġlearning Ġ( RL ) Ġtasks . ĠNone Ġof Ġthem Ġis Ġsuitable Ġfor Ġour Ġsetting , Ġwhich Ġinvolves Ġinter le aving Ġof ĠRL , Ġsupervised Ġand Ġinteractive Ġlearning . ĠMore Ġdetails Ġabout Ġlifelong Ġlearning Ġcan Ġbe Ġfound Ġin Ġthe Ġbook ĠB IB REF 31 Ġ. </s> Inter active ĠKnowledge ĠLearning Ġ( Li Li ) </s> We Ġdesign ĠLi Li Ġas Ġa Ġcombination Ġof Ġtwo Ġinterconnected Ġmodels : Ġ( 1 ) Ġa ĠRL Ġmodel Ġthat Ġlearns Ġto Ġformulate Ġa Ġquery - specific Ġinference Ġstrategy Ġfor Ġperforming Ġthe ĠOK BC Ġtask , Ġand Ġ( 2 ) Ġa Ġlifelong Ġprediction Ġmodel Ġto Ġpredict Ġwhether Ġa Ġtriple Ġshould Ġbe Ġin Ġthe ĠKB , Ġwhich Ġis Ġinvoked Ġby Ġan Ġaction Ġwhile Ġexecuting Ġthe Ġinference Ġstrategy Ġand Ġis Ġlearned Ġfor Ġeach Ġrelation Ġas Ġin ĠC - PR . ĠThe Ġframework Ġimproves Ġits Ġperformance Ġover Ġtime Ġthrough Ġuser Ġinteraction Ġand Ġknowledge Ġretention . ĠCompared Ġto Ġthe Ġexisting ĠKB Ġinference Ġmethods , ĠLi Li Ġover comes Ġthe Ġfollowing Ġthree Ġchallenges Ġfor ĠOK BC : </s> 1 . ĠM apping Ġopen - world Ġto Ġclose - world . ĠBeing Ġa Ġclosed - world Ġmethod , ĠC - PR Ġcannot Ġextract Ġpath Ġfeatures Ġand Ġlearn Ġa Ġprediction Ġmodel Ġwhen Ġany Ġof ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġor ĠIN LINE FORM 2 Ġis Ġunknown . ĠLi Li Ġsolves Ġthis Ġproblem Ġthrough Ġinteractive Ġknowledge Ġacquisition . ĠIf ĠIN LINE FORM 3 Ġis Ġunknown , ĠLi Li Ġasks Ġthe Ġuser Ġto Ġprovide Ġa Ġclue Ġ( an Ġexample Ġof ĠIN LINE FORM 4 Ġ). ĠAnd Ġif ĠIN LINE FORM 5 Ġor ĠIN LINE FORM 6 Ġis Ġunknown , ĠLi Li Ġasks Ġthe Ġuser Ġto Ġprovide Ġa Ġlink Ġ( relation ) Ġto Ġconnect Ġthe Ġunknown Ġentity Ġwith Ġan Ġexisting Ġentity Ġ( aut om atically Ġselected ) Ġin Ġthe ĠKB . ĠWe Ġrefer Ġto Ġsuch Ġa Ġquery Ġas Ġa Ġconnecting Ġlink Ġquery Ġ( CL Q ). ĠThe Ġacquired Ġknowledge Ġreduces ĠOK BC Ġto ĠK BC Ġand Ġmakes Ġthe Ġinference Ġtask Ġfeasible . </s> 2 . ĠSp ares eness Ġof ĠKB . ĠA Ġmain Ġissue Ġof Ġall ĠPR Ġmethods Ġlike ĠC - PR Ġis Ġthe Ġconnectivity Ġof Ġthe ĠKB Ġgraph . ĠIf Ġthere Ġis Ġno Ġpath Ġconnecting ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġin Ġthe Ġgraph , Ġpath Ġenumer ation Ġof ĠC - PR Ġgets Ġstuck Ġand Ġinference Ġbecomes Ġinf eas ible . ĠIn Ġsuch Ġcases , ĠLi Li Ġuses Ġa Ġtemplate Ġrelation Ġ( âĢ ľ @ - ? - @ ") Ġas Ġthe Ġmissing Ġlink Ġmarker Ġto Ġconnect Ġentity - p airs Ġand Ġcontinues Ġfeature Ġextraction . ĠA Ġpath Ġcontaining ĠâĢ ľ @ - ? - @ " Ġis Ġcalled Ġan Ġincomplete Ġpath . ĠThus , Ġthe Ġextracted Ġfeature Ġset Ġcontains Ġboth Ġcomplete Ġ( no Ġmissing Ġlink ) Ġand Ġincomplete Ġpaths . ĠNext , ĠLi Li Ġselects Ġan Ġincomplete Ġpath Ġfrom Ġthe Ġfeature Ġset Ġand Ġasks Ġthe Ġuser Ġto Ġprovide Ġa Ġlink Ġfor Ġpath Ġcompletion . ĠWe Ġrefer Ġto Ġsuch Ġa Ġquery Ġas Ġmissing Ġlink Ġquery Ġ( ML Q ). </s> 3 . ĠLim itation Ġin Ġuser Ġknowledge . ĠIf Ġthe Ġuser Ġis Ġunable Ġto Ġrespond Ġto ĠML Q s Ġor ĠCL Q s , ĠLi Li Ġuses Ġa Ġguessing Ġmechanism Ġ( disc ussed Ġlater ) Ġto Ġfill Ġthe Ġgap . ĠThis Ġenables ĠLi Li Ġto Ġcontinue Ġits Ġinference Ġeven Ġif Ġthe Ġuser Ġcannot Ġanswer Ġa Ġsystem Ġquestion . </s> Comp onents Ġof ĠLi Li </s> As Ġlifelong Ġlearning Ġneeds Ġto Ġretain Ġknowledge Ġlearned Ġfrom Ġpast Ġtasks Ġand Ġuse Ġit Ġto Ġhelp Ġfuture Ġlearning ĠB IB REF 31 Ġ, ĠLi Li Ġuses Ġa ĠKnowledge ĠStore Ġ( KS ) Ġfor Ġknowledge Ġretention . ĠKS Ġhas Ġfour Ġcomponents : Ġ( i ) ĠKnowledge ĠGraph Ġ( ĠIN LINE FORM 0 Ġ): ĠIN LINE FORM 1 Ġ( the ĠKB ) Ġis Ġinitialized Ġwith Ġbase ĠKB Ġtri ples Ġ( see ĠÂ§ 4 ) Ġand Ġgets Ġupdated Ġover Ġtime Ġwith Ġthe Ġacquired Ġknowledge . Ġ( ii ) ĠRel ation - Entity ĠMatrix Ġ( ĠIN LINE FORM 2 Ġ): ĠIN LINE FORM 3 Ġis Ġa Ġsparse Ġmatrix , Ġwith Ġrows Ġas Ġrelations Ġand Ġcolumns Ġas Ġentity - p airs Ġand Ġis Ġused Ġby Ġthe Ġprediction Ġmodel . ĠGiven Ġa Ġtriple Ġ( ĠIN LINE FORM 4 Ġ, ĠIN LINE FORM 5 Ġ, ĠIN LINE FORM 6 Ġ) ĠIN LINE FORM 7 Ġ, Ġwe Ġset ĠIN LINE FORM 8 Ġ[ ĠIN LINE FORM 9 Ġ, Ġ( ĠIN LINE FORM 10 Ġ, ĠIN LINE FORM 11 Ġ)] Ġ= Ġ1 Ġindicating ĠIN LINE FORM 12 Ġoccurs Ġfor Ġpair Ġ( ĠIN LINE FORM 13 Ġ, ĠIN LINE FORM 14 Ġ). Ġ( iii ) ĠTask ĠExperience ĠStore Ġ( ĠIN LINE FORM 15 Ġ): ĠIN LINE FORM 16 Ġstores Ġthe Ġpredictive Ġperformance Ġof ĠLi Li Ġon Ġpast Ġlearned Ġtasks Ġin Ġterms Ġof ĠMatthews Ġcorrelation Ġcoefficient Ġ( M CC ) Ġthat Ġmeasures Ġthe Ġquality Ġof Ġbinary Ġclassification . ĠSo , Ġfor Ġtwo Ġtasks ĠIN LINE FORM 17 Ġand ĠIN LINE FORM 18 Ġ( each Ġrelation Ġis Ġa Ġtask ), Ġif ĠIN LINE FORM 19 Ġ[ ĠIN LINE FORM 20 Ġ] ĠIN LINE FORM 21 ĠIN LINE FORM 22 Ġ[ ĠIN LINE FORM 23 Ġ] Ġ[ where ĠIN LINE FORM 24 Ġ[ ĠIN LINE FORM 25 Ġ] = M CC ( ĠIN LINE FORM 26 Ġ) ], Ġwe Ġsay ĠC - PR Ġhas Ġlearned ĠIN LINE FORM 27 Ġwell Ġcompared Ġto ĠIN LINE FORM 28 Ġ. Ġ( iv ) ĠIn complete ĠFeature ĠDB Ġ( ĠIN LINE FORM 29 Ġ): ĠIN LINE FORM 30 Ġstores Ġthe Ġfrequency Ġof Ġan Ġincomplete Ġpath ĠIN LINE FORM 31 Ġin Ġthe Ġform Ġof Ġa Ġtuple Ġ( ĠIN LINE FORM 32 Ġ, ĠIN LINE FORM 33 Ġ, ĠIN LINE FORM 34 Ġ) Ġand Ġis Ġused Ġin Ġform ulating ĠML Q s . ĠIN LINE FORM 35 Ġ[( ĠIN LINE FORM 36 Ġ, ĠIN LINE FORM 37 Ġ, ĠIN LINE FORM 38 Ġ)] Ġ= ĠIN LINE FORM 39 Ġimplies ĠLi Li Ġhas Ġextracted Ġincomplete Ġpath ĠIN LINE FORM 40 ĠIN LINE FORM 41 Ġtimes Ġinvolving Ġentity - pair ĠIN LINE FORM 42 Ġ[( ĠIN LINE FORM 43 Ġ, ĠIN LINE FORM 44 Ġ)] Ġfor Ġquery Ġrelation ĠIN LINE FORM 45 Ġ. </s> The ĠRL Ġmodel Ġlearns Ġeven Ġafter Ġtraining Ġwhenever Ġit Ġencounters Ġan Ġunseen Ġstate Ġ( in Ġtesting ) Ġand Ġthus , Ġgets Ġupdated Ġover Ġtime . ĠKS Ġis Ġupdated Ġcontinuously Ġover Ġtime Ġas Ġa Ġresult Ġof Ġthe Ġexecution Ġof ĠLi Li Ġand Ġtakes Ġpart Ġin Ġfuture Ġlearning . ĠThe Ġprediction Ġmodel Ġuses Ġlifelong Ġlearning Ġ( LL ), Ġwhere Ġwe Ġtransfer Ġknowledge Ġ( param eter Ġvalues ) Ġfrom Ġthe Ġmodel Ġfor Ġa Ġpast Ġmost Ġsimilar Ġtask Ġto Ġhelp Ġlearn Ġfor Ġthe Ġcurrent Ġtask . ĠSimilar Ġtasks Ġare Ġidentified Ġby Ġfactor izing ĠIN LINE FORM 0 Ġand Ġcomputing Ġa Ġtask Ġsimilarity Ġmatrix ĠIN LINE FORM 1 Ġ. ĠBesides ĠLL , ĠLi Li Ġuses ĠIN LINE FORM 2 Ġto Ġidentify Ġpoorly Ġlearned Ġpast Ġtasks Ġand Ġacquire Ġmore Ġclues Ġfor Ġthem Ġto Ġimprove Ġits Ġskills et Ġover Ġtime . </s> Li Li Ġalso Ġuses Ġa Ġstack , Ġcalled ĠIn ference ĠStack Ġ( ĠIN LINE FORM 0 Ġ) Ġto Ġhold Ġquery Ġand Ġits Ġstate Ġinformation Ġfor ĠRL . ĠLi Li Ġalways Ġprocesses Ġstack Ġtop Ġ( ĠIN LINE FORM 1 Ġ[ top ]). ĠThe Ġclues Ġfrom Ġthe Ġuser Ġget Ġstored Ġin ĠIN LINE FORM 2 Ġon Ġtop Ġof Ġthe Ġquery Ġduring Ġstrategy Ġexecution Ġand Ġprocessed Ġfirst . ĠThus , Ġthe Ġprediction Ġmodel Ġfor ĠIN LINE FORM 3 Ġis Ġlearned Ġbefore Ġperforming Ġinference Ġon Ġquery , Ġtransforming ĠOK BC Ġto Ġa ĠK BC Ġproblem . ĠTable Ġ1 Ġshows Ġthe Ġparameters Ġof ĠLi Li Ġused Ġin Ġthe Ġfollowing Ġsections . </s> Working Ġof ĠLi Li </s> Given Ġan ĠOK BC Ġquery Ġ( ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġ), Ġwe Ġrepresent Ġit Ġas Ġa Ġdata Ġinstance ĠIN LINE FORM 3 Ġ. ĠIN LINE FORM 4 Ġconsists Ġof ĠIN LINE FORM 5 Ġ( the Ġquery Ġtriple ), ĠIN LINE FORM 6 Ġ( inter action Ġlimit Ġset Ġfor ĠIN LINE FORM 7 Ġ), ĠIN LINE FORM 8 Ġ( exper ience Ġlist Ġstoring Ġthe Ġtransition Ġhistory Ġof ĠM DP Ġfor ĠIN LINE FORM 9 Ġin ĠRL ) Ġand ĠIN LINE FORM 10 Ġ( mode Ġof ĠIN LINE FORM 11 Ġ) Ġden oting Ġif ĠIN LINE FORM 12 Ġis Ġ` ĠIN LINE FORM 13 Ġ' Ġ( training ), Ġ` ĠIN LINE FORM 14 Ġ' Ġ( valid ation ), Ġ` ĠIN LINE FORM 15 Ġ' Ġ( eval uation ) Ġor Ġ` ĠIN LINE FORM 16 Ġ' Ġ( cl ue ) Ġinstance Ġand ĠIN LINE FORM 17 Ġ( feature Ġset ). ĠWe Ġdenote ĠIN LINE FORM 18 Ġ( ĠIN LINE FORM 19 Ġ) Ġas Ġthe Ġset Ġof Ġall Ġcomplete Ġ( in complete ) Ġpath Ġfeatures Ġin ĠIN LINE FORM 20 Ġ. ĠGiven Ġa Ġdata Ġinstance ĠIN LINE FORM 21 Ġ, ĠLi Li Ġstarts Ġits Ġinitialization Ġas Ġfollows : Ġit Ġsets Ġthe Ġstate Ġas ĠIN LINE FORM 22 Ġ( based Ġon ĠIN LINE FORM 23 Ġ, Ġexplained Ġlater ), Ġpushes Ġthe Ġquery Ġtuple Ġ( ĠIN LINE FORM 24 Ġ, ĠIN LINE FORM 25 Ġ) Ġinto ĠIN LINE FORM 26 Ġand Ġfeeds ĠIN LINE FORM 27 Ġ[ top ] Ġto Ġthe ĠRL - model Ġfor Ġstrategy Ġformulation Ġfrom ĠIN LINE FORM 28 Ġ. </s> In ference ĠStrategy ĠForm ulation . ĠWe Ġview Ġsolving Ġthe Ġstrategy Ġformulation Ġproblem Ġas Ġlearning Ġto Ġplay Ġan Ġinference Ġgame , Ġwhere Ġthe Ġgoal Ġis Ġto Ġformulate Ġa Ġstrategy Ġthat Ġ" makes Ġthe Ġinference Ġtask Ġpossible ". ĠConsidering ĠPR Ġmethods , Ġinference Ġis Ġpossible , Ġif f Ġ( 1 ) ĠIN LINE FORM 0 Ġbecomes Ġknown Ġto Ġits ĠKB Ġ( by Ġacquiring Ġclues Ġwhen ĠIN LINE FORM 1 Ġis Ġunknown ) Ġand Ġ( 2 ) Ġpath Ġfeatures Ġare Ġextracted Ġbetween ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġ( which Ġint urn Ġrequires ĠIN LINE FORM 4 Ġand ĠIN LINE FORM 5 Ġto Ġbe Ġknown Ġto ĠKB ). ĠIf Ġthese Ġconditions Ġare Ġmet Ġat Ġthe Ġend Ġof Ġan Ġepisode Ġ( when Ġstrategy Ġformulation Ġfinishes Ġfor Ġa Ġgiven Ġquery ) Ġof Ġthe Ġgame , ĠLi Li Ġwins Ġand Ġthus , Ġit Ġtrains Ġthe Ġprediction Ġmodel Ġfor ĠIN LINE FORM 6 Ġand Ġuses Ġit Ġfor Ġinference . </s> Li Li 's Ġstrategy Ġformulation Ġis Ġmodeled Ġas Ġa ĠMark ov ĠDecision ĠProcess Ġ( M DP ) Ġwith Ġfinite Ġstate Ġ( ĠIN LINE FORM 0 Ġ) Ġand Ġaction Ġ( ĠIN LINE FORM 1 Ġ) Ġspaces . ĠA Ġstate ĠIN LINE FORM 2 Ġconsists Ġof Ġ10 Ġbinary Ġstate Ġvariables Ġ( Table Ġ2 ), Ġeach Ġof Ġwhich Ġkeeps Ġtrack Ġof Ġresults Ġof Ġan Ġaction ĠIN LINE FORM 3 Ġtaken Ġby ĠLi Li Ġand Ġthus , Ġrecords Ġthe Ġprogress Ġin Ġinference Ġprocess Ġmade Ġso Ġfar . ĠIN LINE FORM 4 Ġis Ġthe Ġinitial Ġstate Ġwith Ġall Ġstate Ġbits Ġset Ġas Ġ0 . ĠIf Ġthe Ġdata Ġinstance Ġ( query ) Ġis Ġa Ġclue Ġ[ ĠIN LINE FORM 5 Ġ], ĠIN LINE FORM 6 Ġ[ CL UE ] Ġis Ġset Ġas Ġ1 . ĠIN LINE FORM 7 Ġconsists Ġof Ġ6 Ġactions Ġ( Table Ġ3 ). ĠIN LINE FORM 8 Ġ, ĠIN LINE FORM 9 Ġ, ĠIN LINE FORM 10 Ġare Ġprocessing Ġactions Ġand ĠIN LINE FORM 11 Ġ, ĠIN LINE FORM 12 Ġ, ĠIN LINE FORM 13 Ġare Ġinteractive Ġactions . ĠWhenever ĠIN LINE FORM 14 Ġis Ġexecuted , Ġthe ĠM DP Ġreaches Ġthe Ġterminal Ġstate . ĠGiven Ġan Ġaction ĠIN LINE FORM 15 Ġin Ġstate ĠIN LINE FORM 16 Ġ, Ġif ĠIN LINE FORM 17 Ġis Ġinvalid Ġin ĠIN LINE FORM 21 Ġor Ġthe Ġobjective Ġof ĠIN LINE FORM 22 Ġis Ġunsatisf ied Ġ(* Ġmarked Ġthe Ġcondition Ġin ĠIN LINE FORM 23 Ġ), ĠRL Ġreceives Ġa Ġnegative Ġreward Ġ( em pir ically Ġset ); Ġelse Ġreceives Ġa Ġpositive Ġreward .. ĠWe Ġuse ĠQ - learning ĠB IB REF 38 Ġwith ĠIN LINE FORM 24 Ġ- gre edy Ġstrategy Ġto Ġlearn Ġthe Ġoptimal Ġpolicy Ġfor Ġtraining Ġthe ĠRL Ġmodel . ĠNote Ġthat , Ġthe Ġinference Ġstrategy Ġis Ġindependent Ġof ĠKB Ġtype Ġand Ġcorrectness Ġof Ġprediction . ĠThus , Ġthe ĠRL - model Ġis Ġtrained Ġonly Ġonce Ġfrom Ġscratch Ġ( re used Ġthereafter Ġfor Ġother ĠKB s ) Ġand Ġalso , Ġindependently Ġof Ġthe Ġprediction Ġmodel . </s> Sometimes Ġthe Ġtraining Ġdataset Ġmay Ġnot Ġbe Ġenough Ġto Ġlearn Ġoptimal Ġpolicy Ġfor Ġall ĠIN LINE FORM 0 Ġ. ĠThus , Ġencountering Ġan Ġunseen Ġstate Ġduring Ġtest Ġcan Ġmake ĠRL - model Ġclueless Ġabout Ġthe Ġaction . ĠGiven Ġa Ġstate ĠIN LINE FORM 1 Ġ, Ġwhenever Ġan Ġinvalid ĠIN LINE FORM 2 Ġis Ġchosen , ĠLi Li Ġremains Ġin ĠIN LINE FORM 3 Ġ. ĠFor ĠIN LINE FORM 4 Ġ, ĠLi Li Ġremains Ġin ĠIN LINE FORM 5 Ġunt ill ĠIN LINE FORM 6 Ġ( see ĠTable Ġ1 Ġfor ĠIN LINE FORM 7 Ġ). ĠSo , Ġif Ġthe Ġstate Ġremains Ġthe Ġsame Ġfor Ġ( ĠIN LINE FORM 8 Ġ+ 1 ) Ġtimes , Ġit Ġimplies ĠLi Li Ġhas Ġencountered Ġa Ġfault Ġ( an Ġunseen Ġstate ). ĠRL - model Ġinstantly Ġswitches Ġto Ġthe Ġtraining Ġmode Ġand Ġrandomly Ġexplores ĠIN LINE FORM 9 Ġto Ġlearn Ġthe Ġoptimal Ġaction Ġ( f ault - t oler ant Ġlearning ). ĠWhile Ġexploring ĠIN LINE FORM 10 Ġ, Ġthe Ġmodel Ġchooses ĠIN LINE FORM 11 Ġonly Ġwhen Ġit Ġhas Ġtried Ġall Ġother ĠIN LINE FORM 12 Ġto Ġavoid Ġabrupt Ġend Ġof Ġepisode . </s> Exec ution Ġof ĠActions . ĠAt Ġany Ġgiven Ġpoint Ġin Ġtime , Ġlet Ġ( ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ) Ġbe Ġthe Ġcurrent ĠIN LINE FORM 2 Ġ[ top ], ĠIN LINE FORM 3 Ġis Ġthe Ġchosen Ġaction Ġand Ġthe Ġcurrent Ġversion Ġof ĠKS Ġcomponents Ġare ĠIN LINE FORM 4 Ġ, ĠIN LINE FORM 5 Ġ, ĠIN LINE FORM 6 Ġand ĠIN LINE FORM 7 Ġ. ĠThen , Ġif ĠIN LINE FORM 8 Ġis Ġinvalid Ġin ĠIN LINE FORM 9 Ġ, ĠLi Li Ġonly Ġupdates ĠIN LINE FORM 10 Ġ[ top ] Ġwith Ġ( ĠIN LINE FORM 11 Ġ, ĠIN LINE FORM 12 Ġ) Ġand Ġreturns ĠIN LINE FORM 13 Ġ[ top ] Ġto ĠRL - model . ĠIn Ġthis Ġprocess , ĠLi Li Ġadds Ġexperience Ġ( ĠIN LINE FORM 14 Ġ, ĠIN LINE FORM 15 Ġ, ĠIN LINE FORM 16 Ġ, ĠIN LINE FORM 17 Ġ) Ġin ĠIN LINE FORM 18 Ġand Ġthen , Ġreplaces ĠIN LINE FORM 19 Ġ[ top ] Ġwith Ġ( ĠIN LINE FORM 20 Ġ, ĠIN LINE FORM 21 Ġ). ĠIf ĠIN LINE FORM 22 Ġis Ġvalid Ġin ĠIN LINE FORM 23 Ġ, ĠLi Li Ġfirst Ġsets Ġthe Ġnext Ġstate ĠIN LINE FORM 24 Ġand Ġperforms Ġa Ġsequence Ġof Ġoperations ĠIN LINE FORM 25 Ġbased Ġon ĠIN LINE FORM 26 Ġ( disc ussed Ġbelow ). ĠUnless Ġspecified , Ġin ĠIN LINE FORM 27 Ġ, ĠLi Li Ġalways Ġmonitors ĠIN LINE FORM 28 Ġand Ġif ĠIN LINE FORM 29 Ġbecomes Ġ0 , ĠLi Li Ġsets ĠIN LINE FORM 30 Ġ. ĠAlso , Ġwhenever ĠLi Li Ġasks Ġthe Ġuser Ġa Ġquery , ĠIN LINE FORM 31 Ġis Ġdecre mented Ġby Ġ1 . ĠOnce ĠIN LINE FORM 32 Ġends , ĠLi Li Ġupdates ĠIN LINE FORM 33 Ġ[ top ] Ġwith Ġ( ĠIN LINE FORM 34 Ġ, ĠIN LINE FORM 35 Ġ) Ġand Ġreturns ĠIN LINE FORM 36 Ġ[ top ] Ġto ĠRL - model Ġfor Ġchoosing Ġthe Ġnext Ġaction . </s> In ĠIN LINE FORM 0 Ġ, ĠLi Li Ġsearches ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġ, ĠIN LINE FORM 3 Ġin ĠIN LINE FORM 4 Ġand Ġsets Ġappropriate Ġbits Ġin ĠIN LINE FORM 5 Ġ( see ĠTable Ġ2 ). ĠIf ĠIN LINE FORM 6 Ġwas Ġunknown Ġbefore Ġand Ġis Ġjust Ġadded Ġto ĠIN LINE FORM 7 Ġor Ġis Ġin Ġthe Ġbottom ĠIN LINE FORM 8 Ġ% Ġ( see ĠTable Ġ1 Ġfor ĠIN LINE FORM 9 Ġ) Ġof ĠIN LINE FORM 10 Ġ, ĠLi Li Ġrandomly Ġsets ĠIN LINE FORM 14 Ġwith Ġprobability ĠIN LINE FORM 15 Ġ. ĠIf ĠIN LINE FORM 16 Ġis Ġa Ġclue Ġand ĠIN LINE FORM 17 Ġ, ĠLi Li Ġupdates ĠKS Ġwith Ġtriple ĠIN LINE FORM 18 Ġ, Ġwhere Ġ( ĠIN LINE FORM 19 Ġ, ĠIN LINE FORM 20 Ġ, ĠIN LINE FORM 21 Ġ) Ġand Ġ( ĠIN LINE FORM 22 Ġ, ĠIN LINE FORM 23 Ġ, ĠIN LINE FORM 24 Ġ) Ġgets Ġadded Ġto ĠIN LINE FORM 25 Ġand ĠIN LINE FORM 26 Ġ, ĠIN LINE FORM 27 Ġare Ġset Ġas Ġ1 . </s> In ĠIN LINE FORM 0 Ġ, ĠLi Li Ġasks Ġthe Ġuser Ġto Ġprovide Ġa Ġclue Ġ(+ ve Ġinstance ) Ġfor ĠIN LINE FORM 1 Ġand Ġcorrupt s ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġof Ġthe Ġclue Ġonce Ġat Ġa Ġtime , Ġto Ġgenerate Ġ- ve Ġinstances Ġby Ġsampling Ġnodes Ġfrom ĠIN LINE FORM 4 Ġ. ĠThese Ġinstances Ġhelp Ġin Ġtraining Ġprediction Ġmodel Ġfor ĠIN LINE FORM 5 Ġwhile Ġexecuting ĠIN LINE FORM 6 Ġ. </s> In ĠIN LINE FORM 0 Ġ, ĠLi Li Ġselects Ġan Ġincomplete Ġpath ĠIN LINE FORM 1 Ġfrom ĠIN LINE FORM 2 Ġto Ġformulate ĠML Q , Ġsuch Ġthat ĠIN LINE FORM 3 Ġis Ġmost Ġfrequently Ġobserved Ġfor ĠIN LINE FORM 4 Ġand ĠIN LINE FORM 5 Ġis Ġhigh , Ġgiven Ġby ĠIN LINE FORM 6 Ġ. ĠHere , ĠIN LINE FORM 7 Ġdenotes Ġthe Ġcontextual Ġsimilarity ĠB IB REF 16 Ġof Ġentity - pair ĠIN LINE FORM 8 Ġ. ĠIf ĠIN LINE FORM 9 Ġis Ġhigh , ĠIN LINE FORM 10 Ġis Ġmore Ġlikely Ġto Ġpossess Ġa Ġrelation Ġbetween Ġthem Ġand Ġso , Ġis Ġa Ġgood Ġcandidate Ġfor Ġform ulating ĠML Q . ĠWhen Ġthe Ġuser Ġdoes Ġnot Ġrespond Ġto ĠML Q Ġ( or ĠCL Q Ġin ĠIN LINE FORM 11 Ġ), Ġthe Ġguessing Ġmechanism Ġis Ġused , Ġwhich Ġworks Ġas Ġfollows : ĠSince Ġcontextual Ġsimilarity Ġof Ġentity - p airs Ġis Ġhighly Ġcorrelated Ġwith Ġtheir Ġclass Ġlabels ĠB IB REF 16 Ġ, ĠLi Li Ġdivides Ġthe Ġsimilarity Ġrange Ġ[- 1 , Ġ1 ] Ġinto Ġthree Ġsegments , Ġusing Ġa Ġlow Ġ( ĠIN LINE FORM 12 Ġ) Ġand Ġhigh Ġ( ĠIN LINE FORM 13 Ġ) Ġsimilarity Ġthreshold Ġand Ġreplaces Ġthe Ġmissing Ġlink Ġwith ĠIN LINE FORM 14 Ġin ĠIN LINE FORM 15 Ġto Ġmake Ġit Ġcomplete Ġas Ġfollows : ĠIf ĠIN LINE FORM 16 Ġ, ĠIN LINE FORM 17 Ġ= ĠâĢ ľ @ - Lo ose ly Related To - @ "; Ġelse Ġif ĠIN LINE FORM 18 Ġ, ĠIN LINE FORM 19 Ġ= âĢ ľ @ - Not Related To - @ "; ĠOtherwise , ĠIN LINE FORM 20 Ġ= âĢ ľ @ - Related To - @ ". </s> In ĠIN LINE FORM 0 Ġ, ĠLi Li Ġasks ĠCL Q s Ġfor Ġconnecting Ġunknown Ġentities ĠIN LINE FORM 1 Ġand / or ĠIN LINE FORM 2 Ġwith ĠIN LINE FORM 3 Ġby Ġselecting Ġthe Ġmost Ġcontext ually Ġrelevant Ġnode Ġ( w rt ĠIN LINE FORM 4 Ġ, ĠIN LINE FORM 5 Ġ) Ġfrom ĠIN LINE FORM 6 Ġ, Ġgiven Ġby Ġlink ĠIN LINE FORM 7 Ġ. ĠWe Ġadopt Ġthe Ġcontextual Ġrelevance Ġidea Ġin ĠB IB REF 16 Ġwhich Ġis Ġcomputed Ġusing Ġword Ġembed ding ĠB IB REF 39 </s> In ĠIN LINE FORM 0 Ġ, ĠLi Li Ġextracts Ġpath Ġfeatures ĠIN LINE FORM 1 Ġbetween Ġ( ĠIN LINE FORM 2 Ġ, ĠIN LINE FORM 3 Ġ) Ġand Ġupdates ĠIN LINE FORM 4 Ġwith Ġincomplete Ġfeatures Ġfrom ĠIN LINE FORM 5 Ġ. ĠLi Li Ġalways Ġtrains Ġthe Ġprediction Ġmodel Ġwith Ġcomplete Ġfeatures ĠIN LINE FORM 6 Ġand Ġonce ĠIN LINE FORM 7 Ġor ĠIN LINE FORM 8 Ġ, ĠLi Li Ġstops Ġasking ĠML Q s . ĠThus , Ġin Ġboth ĠIN LINE FORM 9 Ġand ĠIN LINE FORM 10 Ġ, ĠLi Li Ġalways Ġmonitors ĠIN LINE FORM 11 Ġto Ġcheck Ġfor Ġthe Ġsaid Ġrequirements Ġand Ġsets ĠIN LINE FORM 12 Ġto Ġcontrol Ġinteractions . </s> In ĠIN LINE FORM 0 Ġ, Ġif ĠLi Li Ġwins Ġthe Ġepisode , Ġit Ġadds ĠIN LINE FORM 1 Ġin Ġone Ġof Ġdata Ġbuffers ĠIN LINE FORM 2 Ġbased Ġon Ġits Ġmode ĠIN LINE FORM 3 Ġ. ĠE . g ., Ġif ĠIN LINE FORM 4 Ġor ĠIN LINE FORM 5 Ġ, ĠIN LINE FORM 6 Ġis Ġused Ġfor Ġtraining Ġand Ġadded Ġto ĠIN LINE FORM 7 Ġ. ĠSimilarly Ġvalidation Ġbuffer ĠIN LINE FORM 8 Ġand Ġevaluation Ġbuffer ĠIN LINE FORM 9 Ġare Ġpopulated . ĠIf ĠIN LINE FORM 10 Ġ, ĠLi Li Ġinv okes Ġthe Ġprediction Ġmodel Ġfor ĠIN LINE FORM 11 Ġ. </s> L if elong ĠRel ation ĠPrediction . ĠGiven Ġa Ġrelation ĠIN LINE FORM 0 Ġ, ĠLi Li Ġuses ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġ( see ĠIN LINE FORM 3 Ġ) Ġto Ġtrain Ġa Ġprediction Ġmodel Ġ( say , ĠIN LINE FORM 4 Ġ) Ġwith Ġparameters ĠIN LINE FORM 5 Ġ. ĠFor Ġa Ġunknown ĠIN LINE FORM 6 Ġ, Ġthe Ġclue Ġinstances Ġget Ġstored Ġin ĠIN LINE FORM 7 Ġand ĠIN LINE FORM 8 Ġ. ĠThus , ĠLi Li Ġpop ulates ĠIN LINE FORM 9 Ġby Ġtaking Ġ10 % Ġ( see ĠÂ§ 4 ) Ġof Ġthe Ġinstances Ġfrom ĠIN LINE FORM 10 Ġand Ġstarts Ġthe Ġtraining . ĠFor ĠIN LINE FORM 11 Ġ, ĠLi Li Ġuses Ġa ĠL ST M ĠB IB REF 40 Ġto Ġcompose Ġthe Ġvector Ġrepresentation Ġof Ġeach Ġfeature ĠIN LINE FORM 12 Ġas ĠIN LINE FORM 13 Ġand Ġvector Ġrepresentation Ġof ĠIN LINE FORM 14 Ġas ĠIN LINE FORM 15 Ġ. ĠNext , ĠLi Li Ġcomp utes Ġthe Ġprediction Ġvalue , ĠIN LINE FORM 16 Ġas Ġs igm oid Ġof Ġthe Ġmean Ġcos ine Ġsimilarity Ġof Ġall Ġfeatures Ġand ĠIN LINE FORM 17 Ġ, Ġgiven Ġby ĠIN LINE FORM 18 Ġ) Ġand Ġmaximize Ġthe Ġlog - like lihood Ġof ĠIN LINE FORM 19 Ġfor Ġtraining . ĠOnce ĠIN LINE FORM 20 Ġis Ġtrained , ĠLi Li Ġupdates ĠIN LINE FORM 21 Ġ[ ĠIN LINE FORM 22 Ġ] Ġusing ĠIN LINE FORM 23 Ġ. ĠWe Ġalso Ġtrain Ġan Ġinverse Ġmodel Ġfor ĠIN LINE FORM 24 Ġ, ĠIN LINE FORM 25 Ġby Ġreversing Ġthe Ġpath Ġfeatures Ġin ĠIN LINE FORM 26 Ġand ĠIN LINE FORM 27 Ġwhich Ġhelp Ġin Ġlifelong Ġlearning Ġ( disc ussed Ġbelow ). ĠUnlike ĠB IB REF 20 Ġ, ĠB IB REF 21 Ġ, Ġwhile Ġpredicting Ġthe Ġlabel Ġfor ĠIN LINE FORM 28 Ġ, Ġwe Ġcompute Ġa Ġrelation - specific Ġprediction Ġthreshold ĠIN LINE FORM 29 Ġcorresponding Ġto ĠIN LINE FORM 30 Ġusing ĠIN LINE FORM 31 Ġas : ĠIN LINE FORM 32 Ġand Ġinfer ĠIN LINE FORM 33 Ġas Ġ+ ve Ġif ĠIN LINE FORM 34 Ġand Ġ- ve Ġotherwise . ĠHere , ĠIN LINE FORM 35 Ġ( ĠIN LINE FORM 36 Ġ) Ġis Ġthe Ġmean Ġprediction Ġvalue Ġfor Ġall Ġ+ ve Ġ(- ve ) Ġexamples Ġin ĠIN LINE FORM 37 Ġ. </s> Mod els Ġtrained Ġon Ġa Ġfew Ġexamples Ġ( e . g ., Ġclues Ġacquired Ġfor Ġunknown ĠIN LINE FORM 0 Ġ) Ġwith Ġrandomly Ġinitialized Ġweights Ġoften Ġperform Ġpoorly Ġdue Ġto Ġunder fitting . ĠThus , Ġwe Ġtransfer Ġknowledge Ġ( weights ) Ġfrom Ġthe Ġpast Ġmost Ġsimilar Ġ( w rt ĠIN LINE FORM 1 Ġ) Ġtask Ġin Ġa Ġlifelong Ġlearning Ġmanner ĠB IB REF 31 Ġ. ĠLi Li Ġuses ĠIN LINE FORM 2 Ġto Ġfind Ġthe Ġpast Ġmost Ġsimilar Ġtask Ġfor ĠIN LINE FORM 3 Ġas Ġfollows : ĠLi Li Ġcomp utes Ġtr anc ated ĠS VD Ġof ĠIN LINE FORM 4 Ġas ĠIN LINE FORM 5 Ġand Ġthen , Ġthe Ġsimilarity Ġmatrix ĠIN LINE FORM 6 Ġ. ĠIN LINE FORM 7 Ġprovides Ġthe Ġsimilarity Ġbetween Ġrelations ĠIN LINE FORM 8 Ġand ĠIN LINE FORM 9 Ġin ĠIN LINE FORM 10 Ġ. ĠThus , ĠLi Li Ġchooses Ġa Ġsource Ġrelation ĠIN LINE FORM 11 Ġto Ġtransfer Ġweights . ĠHere , ĠIN LINE FORM 12 Ġis Ġthe Ġset Ġof Ġall ĠIN LINE FORM 13 Ġand ĠIN LINE FORM 14 Ġfor Ġwhich ĠLi Li Ġhas Ġalready Ġlearned Ġa Ġprediction Ġmodel . ĠNow , Ġif ĠIN LINE FORM 15 Ġor ĠIN LINE FORM 16 Ġ, ĠLi Li Ġrandomly Ġinitial izes Ġthe Ġweights ĠIN LINE FORM 17 Ġfor ĠIN LINE FORM 18 Ġand Ġproceeds Ġwith Ġthe Ġtraining . ĠOtherwise , ĠLi Li Ġuses ĠIN LINE FORM 19 Ġas Ġinitial Ġweights Ġand Ġfine - t unes ĠIN LINE FORM 20 Ġwith Ġa Ġlow Ġlearning Ġrate . </s> A ĠRunning ĠExample . ĠConsidering Ġthe Ġexample Ġshown Ġin ĠFigure Ġ1 , ĠLi Li Ġworks Ġas Ġfollows : Ġfirst , ĠLi Li Ġexecutes ĠIN LINE FORM 0 Ġand Ġdetects Ġthat Ġthe Ġsource Ġentity ĠâĢ ľ Obama " Ġand Ġquery Ġrelation ĠâĢ ľ C itizen Of " Ġare Ġunknown . ĠThus , ĠLi Li Ġexecutes ĠIN LINE FORM 1 Ġto Ġacquire Ġclue Ġ( SF 1 ) Ġfor ĠâĢ ľ C itizen Of " Ġand Ġpushes Ġthe Ġclue Ġ(+ ve Ġexample ) Ġand Ġtwo Ġgenerated Ġ- ve Ġexamples Ġinto ĠIN LINE FORM 2 Ġ. ĠOnce Ġthe Ġclues Ġare Ġprocessed Ġand Ġa Ġprediction Ġmodel Ġis Ġtrained Ġfor ĠâĢ ľ C itizen Of " Ġby Ġform ulating Ġseparate Ġstrategies Ġfor Ġthem , ĠLi Li Ġbecomes Ġaware Ġof ĠâĢ ľ C itizen Of ". ĠNow , Ġas Ġthe Ġclues Ġhave Ġalready Ġbeen Ġpopped Ġfrom ĠIN LINE FORM 3 Ġ, Ġthe Ġquery Ġbecomes ĠIN LINE FORM 4 Ġand Ġthe Ġstrategy Ġformulation Ġprocess Ġfor Ġthe Ġquery Ġresumes . ĠNext , ĠLi Li Ġasks Ġuser Ġto Ġprovide Ġa Ġconnecting Ġlink Ġfor ĠâĢ ľ Obama " Ġby Ġperforming ĠIN LINE FORM 5 Ġ. ĠNow , Ġthe Ġquery Ġentities Ġand Ġrelation Ġbeing Ġknown , ĠLi Li Ġenumer ates Ġpaths Ġbetween ĠâĢ ľ Obama " Ġand ĠâĢ ľ USA " Ġby Ġperforming ĠIN LINE FORM 6 Ġ. ĠLet Ġan Ġextracted Ġpath Ġbe ĠâĢ ľ ĠIN LINE FORM 7 Ġ" Ġwith Ġmissing Ġlink Ġbetween Ġ( ĠIN LINE FORM 8 Ġ, ĠIN LINE FORM 9 Ġ). ĠLi Li Ġasks Ġthe Ġuser Ġto Ġfill Ġthe Ġlink Ġby Ġperforming ĠIN LINE FORM 10 Ġand Ġthen , Ġextracts Ġthe Ġcomplete Ġfeature ĠâĢ ľ ĠIN LINE FORM 11 Ġ". ĠThe Ġfeature Ġset Ġis Ġthen Ġfed Ġto Ġthe Ġprediction Ġmodel Ġand Ġinference Ġis Ġmade Ġas Ġa Ġresult Ġof ĠIN LINE FORM 12 Ġ. ĠThus , Ġthe Ġformulated Ġinference Ġstrategy Ġis : ĠâĢ ľ ĠIN LINE FORM 13 Ġ". </s> Exper iments </s> We Ġnow Ġevaluate ĠLi Li Ġin Ġterms Ġof Ġits Ġpredictive Ġperformance Ġand Ġstrategy Ġformulation Ġabilities . </s> Data : ĠWe Ġuse Ġtwo Ġstandard Ġdatasets Ġ( see ĠTable Ġ4 ): Ġ( 1 ) ĠFree base ĠFB 15 k , Ġand Ġ( 2 ) ĠWord Net ĠIN LINE FORM 0 Ġ. ĠUsing Ġeach Ġdataset , Ġwe Ġbuild Ġa Ġfairly Ġlarge Ġgraph Ġand Ġuse Ġit Ġas Ġthe Ġoriginal ĠKB Ġ( ĠIN LINE FORM 1 Ġ) Ġfor Ġevaluation . ĠWe Ġalso Ġaugment ĠIN LINE FORM 2 Ġwith Ġinverse Ġtri ples Ġ( ĠIN LINE FORM 3 Ġ, ĠIN LINE FORM 4 Ġ, ĠIN LINE FORM 5 Ġ) Ġfor Ġeach Ġ( ĠIN LINE FORM 6 Ġ, ĠIN LINE FORM 7 Ġ, ĠIN LINE FORM 8 Ġ) Ġfollowing Ġexisting ĠK BC Ġmethods . </s> Parameter ĠSettings . ĠUnless Ġspecified , Ġthe Ġempir ically Ġset Ġparameters Ġ( see ĠTable Ġ1 ) Ġof ĠLi Li Ġare : ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġ, ĠIN LINE FORM 3 Ġ, ĠIN LINE FORM 4 Ġ, ĠIN LINE FORM 5 Ġ, ĠIN LINE FORM 6 Ġ, ĠIN LINE FORM 7 Ġ, ĠIN LINE FORM 8 Ġ, ĠIN LINE FORM 9 Ġ, ĠIN LINE FORM 10 Ġ. ĠFor Ġtraining ĠRL - model Ġwith ĠIN LINE FORM 11 Ġ- gre edy Ġstrategy , Ġwe Ġuse ĠIN LINE FORM 12 Ġ, ĠIN LINE FORM 13 Ġ, Ġpre - training Ġsteps = 5 0000 . ĠWe Ġused ĠKer as Ġdeep Ġlearning Ġlibrary Ġto Ġimplement Ġand Ġtrain Ġthe Ġprediction Ġmodel . ĠWe Ġset Ġbatch - size Ġas Ġ128 , Ġmax . Ġtraining Ġepoch Ġas Ġ150 , Ġdrop out Ġas Ġ0 . 2 , Ġhidden Ġunits Ġand Ġembed ding Ġsize Ġas Ġ300 Ġand Ġlearning Ġrate Ġas Ġ5 e - 3 Ġwhich Ġis Ġreduced Ġgradually Ġon Ġplateau Ġwith Ġfactor Ġ0 . 5 Ġand Ġpatience Ġ5 . ĠAdam Ġoptim izer Ġand Ġearly Ġstopping Ġwere Ġused Ġin Ġtraining . ĠWe Ġalso Ġshuffle ĠIN LINE FORM 14 Ġin Ġeach Ġepoch Ġand Ġadjust Ġclass Ġweights Ġin versely Ġproportional Ġto Ġclass Ġfrequencies Ġin ĠIN LINE FORM 15 Ġ. </s> Label ed ĠDat as et ĠGeneration Ġand ĠSim ulated ĠUser ĠCreation . ĠWe Ġcreate Ġa Ġsimulated Ġuser Ġfor Ġeach ĠKB Ġto Ġevaluate ĠLi Li . ĠWe Ġcreate Ġthe Ġlabeled Ġdatasets , Ġthe Ġsimulated Ġuser âĢ Ļ s Ġknowledge Ġbase Ġ( ĠIN LINE FORM 0 Ġ), Ġand Ġthe Ġbase ĠKB Ġ( ĠIN LINE FORM 1 Ġ) Ġfrom ĠIN LINE FORM 2 Ġ. ĠIN LINE FORM 3 Ġused Ġas Ġthe Ġinitial ĠKB Ġgraph Ġ( ĠIN LINE FORM 4 Ġ) Ġof ĠLi Li . </s> We Ġfollowed ĠB IB REF 16 Ġfor Ġlabeled Ġdataset Ġgeneration . ĠFor ĠFree base , Ġwe Ġfound Ġ86 Ġrelations Ġwith ĠIN LINE FORM 0 Ġtri ples Ġand Ġrandomly Ġselected Ġ50 Ġfrom Ġvarious Ġdomains . ĠWe Ġrandomly Ġshuffle Ġthe Ġlist Ġof Ġ50 Ġrelations , Ġselect Ġ25 % Ġof Ġthem Ġas Ġunknown Ġrelations Ġand Ġconsider Ġthe Ġrest Ġ( 75 %) Ġas Ġknown Ġrelations . ĠFor Ġeach Ġknown Ġrelation ĠIN LINE FORM 1 Ġ, Ġwe Ġrandomly Ġshuffle Ġthe Ġlist Ġof Ġdistinct Ġtri ples Ġfor ĠIN LINE FORM 2 Ġ, Ġchoose Ġ1000 Ġtri ples Ġand Ġsplit Ġthem Ġinto Ġ60 % Ġtraining , Ġ10 % Ġvalidation Ġand Ġ20 % Ġtest . ĠRest Ġ10 % Ġalong Ġwith Ġthe Ġleftover Ġ( not Ġincluded Ġin Ġthe Ġlist Ġof Ġ1000 ) Ġtri ples Ġare Ġadded Ġto ĠIN LINE FORM 3 Ġ. ĠFor Ġeach Ġunknown Ġrelation ĠIN LINE FORM 4 Ġ, Ġwe Ġremove Ġall Ġtri ples Ġof ĠIN LINE FORM 5 Ġfrom ĠIN LINE FORM 6 Ġand Ġadd Ġthem Ġto ĠIN LINE FORM 7 Ġ. ĠIn Ġthis Ġprocess , Ġwe Ġalso Ġrandomly Ġchoose Ġ20 % Ġtri ples Ġas Ġtest Ġinstances Ġfor Ġunknown ĠIN LINE FORM 8 Ġwhich Ġare Ġexcluded Ġfrom ĠIN LINE FORM 9 Ġ. ĠNote Ġthat , Ġnow ĠIN LINE FORM 10 Ġhas Ġat Ġleast Ġ10 % Ġof Ġchosen Ġtri ples Ġfor Ġeach ĠIN LINE FORM 11 Ġ( known Ġand Ġunknown ) Ġand Ġso , Ġuser Ġis Ġalways Ġable Ġto Ġprovide Ġclues Ġfor Ġboth Ġcases . ĠFor Ġeach Ġlabeled Ġdataset , Ġwe Ġrandomly Ġchoose Ġ10 % Ġof Ġthe Ġentities Ġpresent Ġin Ġdataset Ġtri ples , Ġremove Ġtri ples Ġinvolving Ġthose Ġentities Ġfrom ĠIN LINE FORM 12 Ġand Ġadd Ġto ĠIN LINE FORM 13 Ġ. ĠAt Ġthis Ġpoint , ĠIN LINE FORM 14 Ġgets Ġreduced Ġto ĠIN LINE FORM 15 Ġand Ġis Ġused Ġas ĠIN LINE FORM 16 Ġfor ĠLi Li . ĠThe Ġdataset Ġstats Ġin ĠTable Ġ4 Ġshows Ġthat Ġthe Ġbase ĠKB Ġ( 60 % Ġtri ples Ġof ĠIN LINE FORM 17 Ġ) Ġis Ġhighly Ġsparse Ġ( comp ared Ġto Ġoriginal ĠKB ) Ġwhich Ġmakes Ġthe Ġinference Ġtask Ġmuch Ġharder . ĠWord Net Ġdataset Ġbeing Ġsmall , Ġwe Ġselect Ġall Ġ18 Ġrelations Ġfor Ġevaluation Ġand Ġcreate Ġlabeled Ġdataset , ĠIN LINE FORM 18 Ġand ĠIN LINE FORM 19 Ġfollowing ĠFree base . ĠAlthough Ġthe Ġuser Ġmay Ġprovide Ġclues Ġ100 % Ġof Ġthe Ġtime , Ġit Ġoften Ġcannot Ġrespond Ġto ĠML Q s Ġand ĠCL Q s Ġ( due Ġto Ġlack Ġof Ġrequired Ġtri ples / facts ). ĠThus , Ġwe Ġfurther Ġenrich ĠIN LINE FORM 20 Ġwith Ġexternal ĠKB Ġtri ples . </s> Given Ġa Ġrelation ĠIN LINE FORM 0 Ġand Ġan Ġobserved Ġtriple Ġ( ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġ, ĠIN LINE FORM 3 Ġ) Ġin Ġtraining Ġor Ġtesting , Ġthe Ġpair Ġ( ĠIN LINE FORM 4 Ġ, ĠIN LINE FORM 5 Ġ) Ġis Ġregarded Ġas Ġa Ġ+ ve Ġinstance Ġfor ĠIN LINE FORM 6 Ġ. ĠFollowing ĠB IB REF 18 Ġ, Ġfor Ġeach Ġ+ ve Ġinstance Ġ( ĠIN LINE FORM 7 Ġ, ĠIN LINE FORM 8 Ġ), Ġwe Ġgenerate Ġtwo Ġnegative Ġones , Ġone Ġby Ġrandomly Ġcorrupt ing Ġthe Ġsource ĠIN LINE FORM 9 Ġ, Ġand Ġthe Ġother Ġby Ġcorrupt ing Ġthe Ġtarget ĠIN LINE FORM 10 Ġ. ĠNote Ġthat , Ġthe Ġtest Ġtri ples Ġare Ġnot Ġin ĠIN LINE FORM 11 Ġor ĠIN LINE FORM 12 Ġand Ġnone Ġof Ġthe Ġ- ve Ġinstances Ġoverlap Ġwith Ġthe Ġ+ ve Ġones . </s> Bas elines . ĠAs Ġnone Ġof Ġthe Ġexisting ĠK BC Ġmethods Ġcan Ġsolve Ġthe ĠOK BC Ġproblem , Ġwe Ġchoose Ġvarious Ġversions Ġof ĠLi Li Ġas Ġbas elines . </s> Single : ĠVersion Ġof ĠLi Li Ġwhere Ġwe Ġtrain Ġa Ġsingle Ġprediction Ġmodel ĠIN LINE FORM 0 Ġfor Ġall Ġtest Ġrelations . </s> Sep : ĠWe Ġdo Ġnot Ġtransfer Ġ( past Ġlearned ) Ġweights Ġfor Ġinitial izing ĠIN LINE FORM 0 Ġ, Ġi . e ., Ġwe Ġdisable ĠLL . </s> F - th ): ĠHere , Ġwe Ġuse Ġa Ġfixed Ġprediction Ġthreshold Ġ0 . 5 Ġinstead Ġof Ġrelation - specific Ġthreshold ĠIN LINE FORM 0 Ġ. </s> BG : ĠThe Ġmissing Ġor Ġconnecting Ġlinks Ġ( when Ġthe Ġuser Ġdoes Ġnot Ġrespond ) Ġare Ġfilled Ġwith ĠâĢ ľ @ - Related To - @ " Ġblindly , Ġno Ġguessing Ġmechanism . </s> w / o ĠPTS : ĠLi Li Ġdoes Ġnot Ġask Ġfor Ġadditional Ġclues Ġvia Ġpast Ġtask Ġselection Ġfor Ġskills et Ġimprovement . </s> E val uation ĠMet rics . ĠTo Ġevaluate Ġthe Ġstrategy Ġformulation Ġability , Ġwe Ġintroduce Ġa Ġmeasure Ġcalled ĠCoverage ( ĠIN LINE FORM 0 Ġ), Ġdefined Ġas Ġthe Ġfraction Ġof Ġtotal Ġquery Ġdata Ġinstances , Ġfor Ġwhich ĠLi Li Ġhas Ġsuccessfully Ġformulated Ġstrategies Ġthat Ġlead Ġto Ġwinning . ĠIf ĠLi Li Ġwins Ġon Ġall Ġepisodes Ġfor Ġa Ġgiven Ġdataset , ĠIN LINE FORM 1 Ġis Ġ1 . 0 . ĠTo Ġevaluate Ġthe Ġpredictive Ġperformance , Ġwe Ġuse ĠAvg . ĠM CC Ġand Ġavg . Ġ+ ve ĠF 1 Ġscore . </s> Results Ġand ĠAnalysis </s> E val uation - I : ĠStrategy ĠForm ulation ĠAbility . ĠTable Ġ5 Ġshows Ġthe Ġlist Ġof Ġinference Ġstrategies Ġformulated Ġby ĠLi Li Ġfor Ġvarious ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġ, Ġwhich Ġcontrol Ġthe Ġstrategy Ġformulation Ġof ĠLi Li . ĠWhen ĠIN LINE FORM 2 Ġ, ĠLi Li Ġcannot Ġinteract Ġwith Ġuser Ġand Ġworks Ġlike Ġa Ġclosed - world Ġmethod . ĠThus , ĠIN LINE FORM 3 Ġdrops Ġsignificantly Ġ( 0 . 47 ). ĠWhen ĠIN LINE FORM 4 Ġ, Ġi . e . Ġwith Ġonly Ġone Ġinteraction Ġper Ġquery , ĠLi Li Ġacqu ires Ġknowledge Ġwell Ġfor Ġinstances Ġwhere Ġeither Ġof Ġthe Ġentities Ġor Ġrelation Ġis Ġunknown . ĠHowever , Ġas Ġone Ġunknown Ġentity Ġmay Ġappear Ġin Ġmultiple Ġtest Ġtri ples , Ġonce Ġthe Ġentity Ġbecomes Ġknown , ĠLi Li Ġdoesn âĢ Ļ t Ġneed Ġto Ġask Ġfor Ġit Ġagain Ġand Ġcan Ġperform Ġinference Ġon Ġfuture Ġtri ples Ġcausing Ġsignificant Ġincrease Ġin ĠIN LINE FORM 5 Ġ( 0 . 97 ). ĠWhen ĠIN LINE FORM 6 Ġ, ĠLi Li Ġis Ġable Ġto Ġperform Ġinference Ġon Ġall Ġinstances Ġand ĠIN LINE FORM 7 Ġbecomes Ġ1 . ĠFor ĠIN LINE FORM 8 Ġ, ĠLi Li Ġuses ĠIN LINE FORM 9 Ġonly Ġonce Ġ( as Ġonly Ġone ĠML Q Ġsatisfies ĠIN LINE FORM 10 Ġ) Ġcompared Ġto ĠIN LINE FORM 11 Ġ. ĠIn Ġsummary , ĠLi Li âĢ Ļ s ĠRL - model Ġcan Ġeffectively Ġformulate Ġquery - specific Ġinference Ġstrategies Ġ( based Ġon Ġspecified Ġparameter Ġvalues ). ĠEvaluation - II : ĠPred ictive ĠPerformance . ĠTable Ġ6 Ġshows Ġthe Ġcomparative Ġperformance Ġof ĠLi Li Ġwith Ġbas elines . ĠTo Ġjudge Ġthe Ġoverall Ġimprovements , Ġwe Ġperformed Ġpaired Ġt - test Ġconsidering Ġ+ ve ĠF 1 Ġscores Ġon Ġeach Ġrelation Ġas Ġpaired Ġdata . ĠConsidering Ġboth ĠKB s Ġand Ġall Ġrelation Ġtypes , ĠLi Li Ġoutper forms ĠSep Ġwith ĠIN LINE FORM 12 Ġ. ĠIf Ġwe Ġset ĠIN LINE FORM 13 Ġ( training Ġwith Ġvery Ġfew Ġclues ), ĠLi Li Ġoutper forms ĠSep Ġwith ĠIN LINE FORM 14 Ġon ĠFree base Ġconsidering ĠM CC . ĠThus , Ġthe Ġlifelong Ġlearning Ġmechanism Ġis Ġeffective Ġin Ġtransferring Ġhelpful Ġknowledge . ĠSingle Ġmodel Ġperforms Ġbetter Ġthan ĠSep Ġfor Ġunknown Ġrelations Ġdue Ġto Ġthe Ġsharing Ġof Ġknowledge Ġ( weights ) Ġacross Ġtasks . ĠHowever , Ġfor Ġknown Ġrelations , Ġperformance Ġdrops Ġbecause , Ġas Ġa Ġnew Ġrelation Ġarrives Ġto Ġthe Ġsystem , Ġold Ġweights Ġget Ġcorrupted Ġand Ġcatastrophic Ġforgetting Ġoccurs . ĠFor Ġunknown Ġrelations , Ġas Ġthe Ġrelations Ġare Ġevaluated Ġjust Ġafter Ġtraining , Ġthere Ġis Ġno Ġchance Ġfor Ġcatastrophic Ġforgetting . ĠThe Ġperformance Ġimprovement Ġ( ĠIN LINE FORM 15 Ġ) Ġof ĠLi Li Ġover ĠF - th Ġon ĠFree base Ġsignifies Ġthat Ġthe Ġrelation - specific Ġthreshold ĠIN LINE FORM 16 Ġworks Ġbetter Ġthan Ġfixed Ġthreshold Ġ0 . 5 Ġbecause , Ġif Ġall Ġprediction Ġvalues Ġfor Ġtest Ġinstances Ġlie Ġabove Ġ( or Ġbelow ) Ġ0 . 5 , ĠF - th Ġpredicts Ġall Ġinstances Ġas Ġ+ ve Ġ(- ve ) Ġwhich Ġde grades Ġits Ġperformance . ĠDue Ġto Ġthe Ġutilization Ġof Ġcontextual Ġsimilarity Ġ( highly Ġcorrelated Ġwith Ġclass Ġlabels ) Ġof Ġentity - p airs , ĠLi Li âĢ Ļ s Ġguessing Ġmechanism Ġworks Ġbetter Ġ( ĠIN LINE FORM 17 Ġ) Ġthan Ġblind Ġguessing Ġ( BG ). ĠThe Ġpast Ġtask Ġselection Ġmechanism Ġof ĠLi Li Ġalso Ġimproves Ġits Ġperformance Ġover Ġw / o ĠPTS , Ġas Ġit Ġacqu ires Ġmore Ġclues Ġduring Ġtesting Ġfor Ġpoorly Ġperformed Ġtasks Ġ( eval uated Ġon Ġvalidation Ġset ). ĠFor ĠFree base , Ġdue Ġto Ġa Ġlarge Ġnumber Ġof Ġpast Ġtasks Ġ[ 9 Ġ( 25 % Ġof Ġ38 ) ], Ġthe Ġperformance Ġdifference Ġis Ġmore Ġsignificant Ġ( ĠIN LINE FORM 18 Ġ). ĠFor ĠWord Net , Ġthe Ġnumber Ġis Ġrelatively Ġsmall Ġ[ 3 Ġ( 25 % Ġof Ġ14 )] Ġand Ġhence , Ġthe Ġdifference Ġis Ġnot Ġsignificant . </s> E val uation - III : ĠUser ĠInter action Ġvs . ĠPerformance . ĠTable Ġ7 Ġshows Ġthe Ġresults Ġof ĠLi Li Ġby Ġvarying Ġclue Ġacquisition Ġrate Ġ( ĠIN LINE FORM 0 Ġ). ĠWe Ġuse ĠFree base Ġfor Ġtuning ĠIN LINE FORM 1 Ġdue Ġto Ġits Ġhigher Ġnumber Ġof Ġunknown Ġtest Ġrelations Ġcompared Ġto ĠWord Net . ĠLi Li âĢ Ļ s Ġperformance Ġimproves Ġsignificantly Ġas Ġit Ġacqu ires Ġmore Ġclues Ġfrom Ġthe Ġuser . ĠThe Ġresults Ġon ĠIN LINE FORM 2 Ġoutper forms Ġ( ĠIN LINE FORM 3 Ġ) Ġthat Ġon ĠIN LINE FORM 4 Ġ. ĠTable Ġ8 Ġshows Ġthe Ġresults Ġof ĠLi Li Ġon Ġuser Ġresponses Ġto ĠML Q âĢ Ļ s Ġand ĠCL Q âĢ Ļ s . ĠAn sw ering ĠML Q âĢ Ļ s Ġand ĠCL Q âĢ Ļ s Ġis Ġvery Ġhard Ġfor Ġsimulated Ġusers Ġ( un like Ġcrowd - s ourcing ) Ġas Ġoften ĠIN LINE FORM 5 Ġlacks Ġthe Ġrequired Ġtriple . ĠThus , Ġwe Ġattempt Ġto Ġanalyze Ġhow Ġthe Ġperformance Ġis Ġeffected Ġif Ġthe Ġuser Ġdoes Ġnot Ġrespond Ġat Ġall . ĠThe Ġresults Ġshow Ġa Ġclear Ġtrend Ġin Ġoverall Ġperformance Ġimprovement Ġwhen Ġthe Ġuser Ġresponds . ĠHowever , Ġthe Ġimprovement Ġis Ġnot Ġsignificant Ġas Ġthe Ġsimulated Ġuser âĢ Ļ s Ġquery Ġsatisfaction Ġrate Ġ( 1 % ĠML Q s Ġand Ġ10 % ĠCL Q s ) Ġis Ġvery Ġsmall . ĠBut , Ġthe Ġanalysis Ġshows Ġthe Ġeffectiveness Ġof ĠLi Li âĢ Ļ s Ġguessing Ġmechanism Ġand Ġcontinual Ġlearning Ġability Ġthat Ġhelp Ġin Ġachieving Ġavg . Ġ+ ve ĠF 1 Ġof Ġ0 . 57 Ġand Ġ0 . 62 Ġon ĠFB Ġand ĠW N Ġrespectively Ġwith Ġminimal Ġparticipation Ġof Ġthe Ġuser . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġare Ġinterested Ġin Ġbuilding Ġa Ġgeneric Ġengine Ġfor Ġcontinuous Ġknowledge Ġlearning Ġin Ġhuman - machine Ġconversations . ĠWe Ġfirst Ġshowed Ġthat Ġthe Ġproblem Ġunderlying Ġthe Ġengine Ġcan Ġbe Ġformulated Ġas Ġan Ġopen - world Ġknowledge Ġbase Ġcompletion Ġ( OK BC ) Ġproblem . ĠWe Ġthen Ġproposed Ġan Ġlifelong Ġinteractive Ġlearning Ġand Ġinference Ġ( Li Li ) Ġapproach Ġto Ġsolving Ġthe ĠOK BC Ġproblem . ĠOK BC Ġis Ġa Ġgeneral ization Ġof ĠK BC . ĠLi Li Ġsolves Ġthe ĠOK BC Ġproblem Ġby Ġfirst Ġform ulating Ġa Ġquery - specific Ġinference Ġstrategy Ġusing ĠRL Ġand Ġthen Ġexecuting Ġit Ġto Ġsolve Ġthe Ġproblem Ġby Ġinteracting Ġwith Ġthe Ġuser Ġin Ġa Ġlifelong Ġlearning Ġmanner . ĠExperimental Ġresults Ġshowed Ġthe Ġeffectiveness Ġof ĠLi Li Ġin Ġterms Ġof Ġboth Ġpredictive Ġquality Ġand Ġstrategy Ġformulation Ġability . ĠWe Ġbelieve Ġthat Ġa Ġsystem Ġwith Ġthe ĠLi Li Ġapproach Ġcan Ġserve Ġas Ġa Ġknowledge Ġlearning Ġengine Ġfor Ġconversations . ĠOur Ġfuture Ġwork Ġwill Ġimprove ĠLi Li Ġto Ġmake Ġmore Ġaccurate . </s> Acknowled gments </s> This Ġwork Ġwas Ġsupported Ġin Ġpart Ġby ĠNational ĠScience ĠFoundation Ġ( NS F ) Ġunder Ġgrant Ġno . ĠI IS - 14 07 9 27 Ġand ĠI IS - 16 50 900 , Ġand Ġa Ġgift Ġfrom ĠHuawei ĠTechnologies ĠCo ĠLtd .