4.49 turns
ABSTRACTIVE
4.5 turns per dialog (8533 turns / 1900 dialogs)
ABSTRACTIVE
We collected the dataset with 1900 dialogs and 8533 turns. Topics discussed in dialogs are questions randomly chosen from training examples of Simple questions BIBREF7 dataset. From this dataset we also took the correct answers in form of Freebase entities.
We collected the dataset with 1900 dialogs and 8533 turns. Topics discussed in dialogs are questions randomly chosen from training examples of Simple questions BIBREF7 dataset. From this dataset we also took the correct answers in form of Freebase entities.
1603.09631
Introduction </s> Now adays , Ġdialog Ġsystems Ġare Ġusually Ġdesigned Ġfor Ġa Ġsingle Ġdomain ĠB IB REF 0 Ġ. ĠThey Ġstore Ġdata Ġin Ġa Ġwell - defined Ġformat Ġwith Ġa Ġfixed Ġnumber Ġof Ġattributes Ġfor Ġentities Ġthat Ġthe Ġsystem Ġcan Ġprovide . ĠBecause Ġdata Ġin Ġthis Ġformat Ġcan Ġbe Ġstored Ġas Ġa Ġtwo - dimensional Ġtable Ġwithin Ġa Ġrelational Ġdatabase , Ġwe Ġcall Ġthe Ġdata Ġflat . ĠThis Ġdata Ġrepresentation Ġallows Ġthe Ġsystem Ġto Ġquery Ġthe Ġdatabase Ġin Ġa Ġsimple Ġand Ġefficient Ġway . ĠIt Ġalso Ġallows Ġto Ġkeep Ġthe Ġdialog Ġstate Ġin Ġthe Ġform Ġof Ġslots Ġ( which Ġusually Ġcorrespond Ġto Ġcolumns Ġin Ġthe Ġtable ) Ġand Ġtrack Ġit Ġthrough Ġthe Ġdialog Ġusing Ġprob abil istic Ġbelief Ġtracking ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ. </s> However , Ġthe Ġwell - defined Ġstructure Ġof Ġthe Ġdatabase Ġof Ġa Ġtypical Ġdialog Ġsystem Ġcomes Ġwith Ġa Ġhigh Ġcost Ġof Ġextending Ġit Ġas Ġevery Ġpiece Ġof Ġnew Ġinformation Ġhas Ġto Ġfit Ġthe Ġformat . ĠThis Ġis Ġespecially Ġa Ġproblem Ġwhen Ġwe Ġone Ġis Ġadapting Ġthe Ġsystem Ġfor Ġa Ġnew Ġdomain Ġbecause Ġits Ġentities Ġcould Ġhave Ġdifferent Ġattributes . </s> A Ġdialog Ġsystem Ġbased Ġon Ġknowledge Ġbases Ġoffers Ġmany Ġadvantages . ĠFirst , Ġthe Ġknowledge Ġbase , Ġwhich Ġcan Ġbe Ġrepresented Ġas Ġknowledge Ġgraph Ġcontaining Ġentities Ġconnected Ġby Ġrelations , Ġis Ġmuch Ġmore Ġflexible Ġthan Ġthe Ġrelational Ġdatabase . ĠSecond , Ġfreely Ġavailable Ġknowledge Ġbases , Ġsuch Ġas ĠFree base , ĠWik id ata , Ġetc . Ġcontain Ġan Ġenormous Ġamount Ġof Ġstructured Ġinformation , Ġand Ġare Ġstill Ġgrowing . ĠA Ġdialog Ġsystem Ġwhich Ġis Ġcapable Ġof Ġworking Ġwith Ġthis Ġtype Ġof Ġinformation Ġwould Ġbe Ġtherefore Ġvery Ġuseful . </s> In Ġthis Ġpaper Ġwe Ġpropose Ġa Ġdataset Ġaiming Ġto Ġhelp Ġdevelop Ġand Ġevaluate Ġdialog Ġsystems Ġbased Ġon Ġknowledge Ġbases Ġby Ġinteractive Ġlearning Ġmotivated Ġin ĠSection Ġ" Mot ivation " ĠSection Ġ" Dialog Ġpolicies " Ġdescribes Ġpolicies Ġthat Ġcan Ġbe Ġused Ġfor Ġretrieving Ġinformation Ġfrom Ġknowledge Ġbases . ĠIn ĠSection Ġ" Dialog ĠSimulation " Ġis Ġintroduced Ġa Ġdialog Ġsimulation Ġfrom Ġnatural Ġconversations Ġwhich Ġwe Ġuse Ġfor Ġevaluation Ġof Ġinteractive Ġlearning . ĠThe Ġdataset Ġcollection Ġprocess Ġallowing Ġthe Ġdialog Ġsimulation Ġis Ġdescribed Ġin ĠSection Ġ" Dat as et ĠCollection ĠProcess " Ġand Ġis Ġfollowed Ġby Ġproperties Ġof Ġthe Ġresulting Ġdataset Ġin ĠSection Ġ" Dat as et ĠProperties " ĠEvaluation Ġguidelines Ġwith Ġproposed Ġmetrics Ġcan Ġbe Ġfound Ġin ĠSection Ġ" Inter active ĠLearning ĠEvaluation " ĠThe Ġplanned Ġfuture Ġwork Ġis Ġsummarized Ġin ĠSection Ġ" Future ĠWork " ĠWe Ġconclude Ġthe Ġpaper Ġwith ĠSection Ġ" Conclusion " </s> Mot ivation </s> From Ġthe Ġpoint Ġof Ġview Ġof Ġdialog Ġsystems Ġproviding Ġgeneral Ġinformation Ġfrom Ġa Ġknowledge Ġbase , Ġthe Ġmost Ġlimiting Ġfactor Ġis Ġthat Ġa Ġlarge Ġportion Ġof Ġthe Ġquestions Ġis Ġunderstood Ġpoorly . </s> Current Ġapproaches ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġcan Ġonly Ġachieve Ġaround Ġ50 % Ġaccuracy Ġon Ġsome Ġquestion Ġanswering Ġdatasets . ĠTherefore , Ġwe Ġthink Ġthat Ġthere Ġis Ġa Ġroom Ġfor Ġimprovements Ġwhich Ġcan Ġbe Ġachieved Ġby Ġinteract ively Ġasking Ġfor Ġadditional Ġinformation Ġin Ġconvers ational Ġdialog s Ġwith Ġusers . ĠThis Ġextra Ġinformation Ġcan Ġbe Ġused Ġfor Ġimproving Ġpolicies Ġof Ġdialog Ġsystems . ĠWe Ġcall Ġthis Ġapproach Ġthe Ġinteractive Ġlearning Ġfrom Ġdialog s . </s> We Ġcan Ġimprove Ġdialog Ġsystems Ġin Ġseveral Ġaspects Ġthrough Ġinteractive Ġlearning Ġin Ġa Ġdirect Ġinteraction Ġwith Ġusers . ĠFirst , Ġthe Ġmost Ġstraightforward Ġway Ġobviously Ġis Ġgetting Ġthe Ġcorrect Ġanswer Ġfor Ġquestions Ġthat Ġthe Ġsystem Ġdoes Ġnot Ġknow . ĠWe Ġcan Ġtry Ġto Ġask Ġusers Ġfor Ġanswers Ġon Ġquestions Ġthat Ġthe Ġsystem Ġencountered Ġin Ġa Ġconversation Ġwith Ġa Ġdifferent Ġuser Ġand Ġdid Ġnot Ġunderstand Ġit . ĠSecond , Ġthe Ġsystem Ġcan Ġask Ġthe Ġuser Ġfor Ġa Ġbroader Ġexplanation Ġof Ġa Ġquestion . ĠThis Ġexplanation Ġcould Ġhelp Ġthe Ġsystem Ġto Ġunderstand Ġthe Ġquestion Ġand Ġprovide Ġthe Ġcorrect Ġanswer . ĠIn Ġaddition , Ġthe Ġsystem Ġcan Ġlearn Ġcorrect Ġpolicy Ġfor Ġthe Ġquestion Ġwhich Ġallows Ġproviding Ġanswers Ġwithout Ġasking Ġany Ġextra Ġinformation Ġfor Ġsimilar Ġquestions Ġnext Ġtime . ĠWe Ġhypothes ize Ġthat Ġusers Ġare Ġwilling Ġto Ġgive Ġsuch Ġexplanations Ġbecause Ġit Ġcould Ġhelp Ġthem Ġto Ġfind Ġanswers Ġfor Ġtheir Ġown Ġquestions . ĠThe Ġlast Ġsource Ġof Ġinformation Ġthat Ġwe Ġconsider Ġfor Ġinteractive Ġlearning Ġis Ġre ph r asing , Ġwhich Ġcould Ġhelp Ġwhen Ġthe Ġsystem Ġdoes Ġknow Ġthe Ġconcept Ġbut Ġdoes Ġnot Ġknow Ġthe Ġcorrect Ġwording . ĠThis Ġarea Ġis Ġextensively Ġstudied Ġfor Ġthe Ġpurposes Ġof Ġinformation Ġretrieval ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ. </s> The Ġmain Ġpurpose Ġof Ġthe Ġcollected Ġdataset Ġis Ġto Ġenable Ġinteractive Ġlearning Ġusing Ġthe Ġsteps Ġproposed Ġabove Ġand Ġpotentially Ġto Ġevaluate Ġhow Ġdifferent Ġsystems Ġperform Ġon Ġthis Ġtask . </s> Dialog Ġpolicies </s> The Ġobvious Ġdifficulty Ġwhen Ġdeveloping Ġa Ġdialog Ġsystem Ġis Ġfinding Ġa Ġway Ġhow Ġto Ġidentify Ġthe Ġpiece Ġof Ġinformation Ġthat Ġthe Ġuser Ġis Ġinterested Ġin . ĠThis Ġis Ġespecially Ġa Ġproblem Ġfor Ġdialog Ġsystems Ġbased Ġon Ġknowledge Ġgraphs Ġcontaining Ġa Ġlarge Ġamount Ġof Ġcomplex Ġstructured Ġinformation . ĠWhile Ġa Ġsimilar Ġproblem Ġis Ġbeing Ġsolved Ġin Ġa Ġtask Ġof Ġquestion Ġanswering , Ġdialog Ġsystems Ġhave Ġmore Ġpossibilities Ġof Ġidentifying Ġthe Ġreal Ġintention Ġof Ġthe Ġuser . ĠFor Ġexample , Ġa Ġdialog Ġsystem Ġcan Ġask Ġfor Ġadditional Ġinformation Ġduring Ġthe Ġdialog . </s> We Ġdistinguish Ġthree Ġdifferent Ġbasic Ġapproaches Ġto Ġrequesting Ġknowledge Ġbases : </s> A Ġcombination Ġof Ġthe Ġabove Ġapproaches Ġis Ġalso Ġpossible . ĠFor Ġexample , Ġwe Ġcan Ġimagine Ġscenarios Ġwhere Ġthe Ġdialog Ġsystem Ġstarts Ġwith Ġhand - crafted Ġrules , Ġwhich Ġare Ġsubsequently Ġinteract ively Ġimproved Ġthrough Ġdialog s Ġwith Ġits Ġusers . ĠWith Ġa Ġgrowing Ġdemand Ġfor Ġopen Ġdomain Ġdialog Ġsystems , Ġit Ġshows Ġthat Ġcreating Ġhand - crafted Ġpolicies Ġdoes Ġnot Ġscale Ġwell Ġ- Ġtherefore , Ġmachine Ġlearning Ġapproaches Ġare Ġgaining Ġon Ġpopularity . ĠMany Ġpublic Ġdatasets Ġfor Ġoffline Ġlearning Ġhave Ġbeen Ġpublished ĠB IB REF 8 Ġ, ĠB IB REF 7 Ġ. ĠHowever , Ġto Ġour Ġknowledge , Ġno Ġpublic Ġdatasets Ġfor Ġinteractive Ġlearning Ġare Ġavailable . ĠTo Ġfill Ġthis Ġgap , Ġwe Ġcollected Ġa Ġdataset Ġwhich Ġenables Ġto Ġtrain Ġinteract ively Ġlearned Ġpolicies Ġthrough Ġa Ġsimulated Ġinteraction Ġwith Ġusers . </s> Dialog ĠSimulation </s> Offline Ġevaluation Ġof Ġinteractive Ġdialog s Ġon Ġreal Ġdata Ġis Ġdifficult Ġbecause Ġdifferent Ġpolicies Ġcan Ġlead Ġto Ġdifferent Ġvariants Ġof Ġthe Ġdialog . ĠOur Ġsolution Ġto Ġthis Ġissue Ġis Ġto Ġcollect Ġdata Ġin Ġa Ġway Ġthat Ġallows Ġus Ġto Ġsimulate Ġall Ġdialog Ġvariants Ġpossible Ġaccording Ġto Ġany Ġpolicy . </s> The Ġdialog Ġvariants Ġwe Ġare Ġconsidering Ġfor Ġinteractive Ġlearning Ġdiffer Ġonly Ġin Ġpresence Ġof Ġseveral Ġparts Ġof Ġthe Ġdialog . ĠTherefore , Ġwe Ġcan Ġcollect Ġdialog s Ġcontaining Ġall Ġinformation Ġused Ġfor Ġinteractive Ġlearning Ġand Ġomit Ġthose Ġparts Ġthat Ġwere Ġnot Ġrequested Ġby Ġthe Ġpolicy . </s> We Ġcollected Ġthe Ġdataset Ġ( see ĠSection Ġ" Dat as et ĠCollection ĠProcess " Ġ) Ġthat Ġenables Ġsimulation Ġwhere Ġthe Ġpolicy Ġcan Ġdecide Ġhow Ġmuch Ġextra Ġinformation Ġto Ġthe Ġquestion Ġit Ġrequests . ĠIf Ġthe Ġquestion Ġis Ġclear Ġto Ġthe Ġsystem Ġit Ġcan Ġattempt Ġto Ġanswer Ġthe Ġquestion Ġwithout Ġany Ġother Ġinformation . ĠIt Ġcan Ġalso Ġask Ġfor Ġa Ġbroader Ġexplanation Ġwith Ġa Ġpossibility Ġto Ġanswer Ġthe Ġquestion Ġafterwards . ĠIf Ġthe Ġsystem Ġdecides Ġnot Ġto Ġanswer Ġthe Ġquestion , Ġwe Ġcan Ġsimulate Ġre r outing Ġthe Ġquestion Ġto Ġanother Ġuser , Ġto Ġtry Ġto Ġobtain Ġthe Ġanswer Ġfrom Ġthem . ĠThe Ġprinciple Ġof Ġsimulated Ġuser 's Ġanswer Ġis Ġshown Ġin Ġthe ĠFigure Ġ1 Ġ. </s> Note Ġthat Ġthe Ġsimulated Ġuser âĢ Ļ s Ġanswer Ġcan Ġbe Ġincorrect Ġbecause Ġhuman Ġusers Ġnaturally Ġmade Ġmistakes . ĠWe Ġintentionally Ġkeep Ġthese Ġmistakes Ġin Ġthe Ġdataset Ġbecause Ġreal Ġsystems Ġmust Ġaddress Ġthem Ġas Ġwell . </s> Dat as et ĠCollection ĠProcess </s> A Ġperfect Ġdata Ġcollection Ġscenario Ġfor Ġour Ġdataset Ġwould Ġuse Ġreal Ġrunning Ġdialog Ġsystem Ġproviding Ġgeneral Ġinformation Ġfrom Ġthe Ġknowledge Ġbase Ġto Ġreal Ġusers . ĠThis Ġsystem Ġcould Ġthen Ġask Ġfor Ġexplanations Ġand Ġanswers Ġfor Ġquestions Ġwhich Ġit Ġis Ġnot Ġable Ġto Ġanswer . </s> However , Ġgetting Ġaccess Ġto Ġsystems Ġwith Ġreal Ġusers Ġis Ġusually Ġhard . ĠTherefore , Ġwe Ġused Ġthe Ġcrowds ourcing Ġplatform ĠCrowd Fl ower Ġ( CF ) Ġfor Ġour Ġdata Ġcollection . </s> A ĠCF Ġworker Ġgets Ġa Ġtask Ġinstruct ing Ġthem Ġto Ġuse Ġour Ġchat - like Ġinterface Ġto Ġhelp Ġthe Ġsystem Ġwith Ġa Ġquestion Ġwhich Ġis Ġrandomly Ġselected Ġfrom Ġtraining Ġexamples Ġof ĠSimple Ġquestions ĠB IB REF 7 Ġdataset . ĠTo Ġcomplete Ġthe Ġtask Ġuser Ġhas Ġto Ġcommunicate Ġwith Ġthe Ġsystem Ġthrough Ġthe Ġthree Ġphase Ġdialog Ġdiscussing Ġquestion Ġparaph rase Ġ( see ĠSection Ġ" Inter active ĠLearning ĠEvaluation " Ġ), Ġexplanation Ġ( see ĠSection Ġ" Future ĠWork " Ġ) Ġand Ġanswer Ġof Ġthe Ġquestion Ġ( see ĠSection Ġ" Conclusion " Ġ). ĠTo Ġavoid Ġpoor ĠEnglish Ġlevel Ġof Ġdialog s Ġwe Ġinvolved ĠCF Ġworkers Ġfrom ĠEnglish Ġspeaking Ġcountries Ġonly . ĠThe Ġcollected Ġdialog s Ġhas Ġbeen Ġannot ated Ġ( see ĠSection Ġ" Acknowled gments " Ġ) Ġby Ġexpert Ġannot ators Ġafterwards . </s> The Ġdescribed Ġprocedure Ġleads Ġto Ġdialog s Ġlike Ġthe Ġone Ġshown Ġin Ġthe ĠFigure Ġ2 Ġ. </s> Question ĠPar aph r asing </s> At Ġbeginning Ġof Ġthe Ġdialog , Ġthe Ġsystem Ġis Ġrequesting Ġthe Ġuser Ġto Ġparaph rase Ġquestion Ġthat Ġthe Ġsystem Ġdoes Ġnot Ġunderstand . ĠThe Ġmain Ġgoal Ġof Ġthis Ġfirst Ġphase Ġis Ġto Ġlet Ġthe Ġuser Ġget Ġfamiliar Ġwith Ġthe Ġpresented Ġquestion Ġand Ġto Ġget Ġalternative Ġword ings Ġof Ġthe Ġposed Ġquestion . </s> Question ĠExplan ation </s> In Ġthe Ġsecond Ġphase , Ġthe Ġuser Ġis Ġasked Ġfor Ġan Ġexplanation Ġof Ġthe Ġquestion . ĠWe Ġexpect Ġthe Ġexplanation Ġto Ġbe Ġdifferent Ġenough Ġfrom Ġthe Ġoriginal Ġquestion Ġ( in Ġterms Ġof Ġthe Ġnumber Ġof Ġcommon Ġwords Ġbetween Ġthe Ġquestion Ġand Ġthe Ġexplanation ). ĠIf Ġthe Ġexplanation Ġis Ġtoo Ġsimilar Ġto Ġthe Ġquestion , Ġthe Ġuser Ġis Ġnotified Ġthat Ġtheir Ġexplanation Ġis Ġnot Ġbroad Ġenough Ġand Ġthey Ġmust Ġprovide Ġa Ġbetter Ġone . </s> Question ĠAnswer </s> With Ġthe Ġvalid Ġexplanation Ġthe Ġdialog Ġturns Ġinto Ġthe Ġlast Ġphase Ġwhere Ġthe Ġuser Ġis Ġasked Ġfor Ġa Ġcorrect Ġanswer Ġto Ġthe Ġoriginal Ġquestion . ĠThe Ġsystem Ġrequires Ġthe Ġuser Ġto Ġanswer Ġwith Ġa Ġfull Ġsentence . ĠIn Ġpractical Ġexperiments Ġthis Ġhas Ġshown Ġas Ġa Ġuseful Ġdecision Ġbecause Ġit Ġimproves Ġsystem 's Ġability Ġto Ġreveal Ġche aters . ĠWe Ġcan Ġsimply Ġmeasure Ġthe Ġconnection Ġ( in Ġterms Ġof Ġcommon Ġwords Ġ) Ġbetween Ġquestion Ġand Ġthe Ġanswer Ġsentence . ĠThis Ġallows Ġto Ġreject Ġcompletely Ġirrelevant Ġanswers . </s> An notation </s> The Ġcorrect Ġanswer Ġfor Ġquestion Ġin Ġeach Ġdialog Ġis Ġavailable Ġfrom ĠSimple Ġquestions Ġdataset . ĠAnswers Ġare Ġin Ġform Ġof ĠFree base Ġentities Ġidentified Ġby Ġunique Ġid . ĠFor Ġevaluation Ġpurposes Ġwe Ġneed Ġinformation Ġwhether Ġdialog Ġcontains Ġthe Ġanswer Ġwhich Ġis Ġconsistent Ġwith Ġthe Ġentity Ġfrom ĠSimple Ġquestions , Ġthe Ġanswer Ġwith Ġanother Ġentity Ġor Ġwhether Ġthe Ġdialog Ġdoes Ġnot Ġcontain Ġany Ġanswer . ĠWhile Ġthe Ġannotation Ġprocess Ġis Ġquite Ġsimple , Ġwe Ġdid Ġnot Ġneed Ġcrowds ourcing Ġfor Ġthe Ġprocess . </s> Natural ĠLanguage ĠUnderstanding Ġ( NL U ) </s> The Ġcollection Ġsystem Ġneeds Ġto Ġrecognize Ġfollowing Ġdialog Ġacts Ġfrom Ġuser Ġutter ances Ġduring Ġall Ġphases Ġof Ġthe Ġdialog : </s> âĢĵ Ġuser Ġdoes Ġnot Ġwant Ġto Ġprovide Ġrequested Ġinformation , </s> âĢĵ Ġuser Ġagrees Ġto Ġprovide Ġrequested Ġinformation , </s> âĢĵ Ġuser Ġdoes Ġnot Ġknow Ġthe Ġrequested Ġinformation , </s> âĢĵ Ġuser Ġtries Ġch it Ġchat Ġwith Ġthe Ġsystem Ġ( hello , Ġbye , Ġwho Ġare Ġyou ... ), </s> âĢĵ Ġnone Ġof Ġthe Ġabove , Ġinterpreted Ġas Ġuser Ġis Ġgiving Ġinformation Ġrequested Ġby Ġthe Ġsystem . </s> P ars ing Ġof Ġthe Ġdialog Ġacts Ġis Ġmade Ġby Ġhand Ġwritten Ġrules Ġusing Ġtemplates Ġand Ġkeyword Ġspotting . ĠThe Ġtemplates Ġand Ġkeywords Ġwere Ġmanually Ġcollected Ġfrom Ġfrequent Ġexpressions Ġused Ġby ĠCF Ġworkers Ġduring Ġpreparation Ġruns Ġof Ġthe Ġdataset Ġcollection Ġprocess Ġ( google Ġit , Ġcheck Ġwik ipedia , ĠI Ġwould Ġneed ... Ġ$\ right arrow Ġ$ ĠNeg ate ). </s> Dat as et ĠProperties </s> We Ġcollected Ġthe Ġdataset Ġwith Ġ1900 Ġdialog s Ġand Ġ85 33 Ġturns . ĠTopics Ġdiscussed Ġin Ġdialog s Ġare Ġquestions Ġrandomly Ġchosen Ġfrom Ġtraining Ġexamples Ġof ĠSimple Ġquestions ĠB IB REF 7 Ġdataset . ĠFrom Ġthis Ġdataset Ġwe Ġalso Ġtook Ġthe Ġcorrect Ġanswers Ġin Ġform Ġof ĠFree base Ġentities . </s> Our Ġdataset Ġconsists Ġof Ġstandard Ġdata Ġsplit Ġinto Ġtraining , Ġdevelopment Ġand Ġtest Ġfiles . ĠThe Ġbasic Ġproperties Ġof Ġthose Ġfiles Ġare Ġas Ġfollows : </s> Each Ġfile Ġcontains Ġcomplete Ġdialog s Ġenriched Ġby Ġoutputs Ġof ĠNL U Ġ( see ĠSection Ġ" Natural ĠLanguage ĠUnderstanding Ġ( NL U )" Ġ) Ġthat Ġwere Ġused Ġduring Ġthe Ġdata Ġcollection . ĠOn Ġtop Ġof Ġthat , Ġeach Ġdialog Ġis Ġlabeled Ġby Ġthe Ġcorrect Ġanswer Ġfor Ġthe Ġquestion Ġand Ġexpert Ġannotation Ġof Ġthe Ġuser Ġanswer Ġhint Ġwhich Ġtells Ġwhether Ġthe Ġhint Ġpoints Ġto Ġthe Ġcorrect Ġanswer , Ġincorrect Ġanswer , Ġor Ġno Ġanswer Ġat Ġall . </s> 351 Ġof Ġall Ġcollected Ġdialog s Ġcontain Ġcorrect Ġanswer Ġprovided Ġby Ġusers Ġand Ġ702 Ġdialog s Ġhave Ġincorrect Ġanswer . ĠIn Ġthe Ġremaining Ġ8 47 Ġdialog s Ġusers Ġdid Ġnot Ġwant Ġto Ġanswer Ġthe Ġquestion . ĠThe Ġcollected Ġdialog s Ġalso Ġcontain Ġ18 28 Ġparaph r ases Ġand Ġ15 39 Ġexplanations Ġfor Ġ1870 Ġquestions . </s> An Ġanswer Ġfor Ġa Ġquestion Ġwas Ġlabeled Ġas Ġcorrect Ġby Ġannot ators Ġonly Ġwhen Ġit Ġwas Ġevident Ġto Ġthem Ġthat Ġthe Ġanswer Ġpoints Ġto Ġthe Ġsame ĠFree base Ġentity Ġthat Ġwas Ġpresent Ġin ĠSimple Ġquestions Ġdataset Ġfor Ġthat Ġparticular Ġquestion . ĠHowever , Ġa Ġlarge Ġamount Ġof Ġquestions Ġfrom Ġthat Ġdataset Ġis Ġquite Ġgeneral Ġ- Ġwith Ġmany Ġpossible Ġanswers . ĠTherefore Ġlot Ġof Ġanswers Ġfrom Ġusers Ġwere Ġlabeled Ġas Ġincorrect Ġeven Ġthough Ġthose Ġanswers Ġperfectly Ġfit Ġthe Ġquestion . ĠOur Ġannot ators Ġidentified Ġthat Ġ285 Ġof Ġthe Ġincorrect Ġanswers Ġwere Ġanswers Ġfor Ġsuch Ġgeneral Ġquestions . ĠExample Ġof Ġthis Ġsituation Ġcan Ġbe Ġdemonstrated Ġby Ġquestion Ġ' Name Ġan Ġactor ' Ġwhich Ġwas Ġcorrectly Ġanswered Ġby Ġ' Brad ĠPitt Ġis Ġan Ġactor ', Ġhowever , Ġto Ġbe Ġconsistent Ġwith ĠSimple Ġquestions Ġannotation , Ġwhich Ġis Ġ' Kelly ĠAt wood ', Ġannot ators Ġwere Ġforced Ġto Ġmark Ġit Ġas Ġan Ġincorrect Ġanswer . </s> Inter active ĠLearning ĠEvaluation </s> A Ġperfect Ġinteractive Ġlearning Ġmodel Ġwould Ġbe Ġable Ġto Ġlearn Ġanything Ġinteract ively Ġfrom Ġtest Ġdialog s Ġduring Ġtesting , Ġwhich Ġwould Ġallow Ġus Ġto Ġmeasure Ġprogress Ġof Ġthe Ġmodel Ġfrom Ġscratch Ġover Ġthe Ġcourse Ġof Ġtime . ĠHowever , Ġa Ġdevelopment Ġof Ġsuch Ġmodel Ġwould Ġbe Ġunnecessarily Ġhard , Ġtherefore Ġwe Ġprovide Ġtraining Ġdialog s Ġwhich Ġcan Ġbe Ġused Ġfor Ġfeature Ġextraction Ġand Ġother Ġengineering Ġrelated Ġto Ġinteractive Ġlearning Ġfrom Ġdialog s Ġin Ġnatural Ġlanguage . ĠModel Ġdevelopment Ġis Ġfurther Ġsupported Ġwith Ġlabeled Ġvalidation Ġdata Ġfor Ġparameter Ġtuning . </s> We Ġpropose Ġtwo Ġevaluation Ġmetrics Ġfor Ġcomparing Ġinteractive Ġlearning Ġmodels . ĠFirst Ġmetric Ġ( see ĠSection Ġ" E fficiency ĠScore " Ġ) Ġscores Ġamount Ġof Ġinformation Ġrequired Ġby Ġthe Ġmodel , Ġsecond Ġmetric Ġ( see ĠSection Ġ" Answer ĠExt raction ĠAccuracy " Ġ) Ġis Ġaccuracy Ġof Ġanswer Ġextraction Ġfrom Ġuser Ġutter ances . ĠAll Ġmodels Ġmust Ġbase Ġtheir Ġanswers Ġonly Ġon Ġinformation Ġgained Ġfrom Ġtraining Ġdialog s Ġand Ġtesting Ġdialog s Ġseen Ġduring Ġthe Ġsimulation Ġso Ġfar , Ġto Ġensure Ġthat Ġthe Ġscore Ġwill Ġreflect Ġthe Ġinteractive Ġlearning Ġof Ġthe Ġmodel Ġinstead Ġof Ġgeneral Ġquestion Ġanswering . </s> E fficiency ĠScore </s> The Ġsimulation Ġof Ġdialog s Ġfrom Ġour Ġdataset Ġallows Ġto Ġevaluate Ġhow Ġefficient Ġa Ġdialog Ġsystem Ġis Ġin Ġusing Ġinformation Ġgained Ġfrom Ġusers . ĠThe Ġdialog Ġsystem Ġshould Ġmaximize Ġthe Ġnumber Ġof Ġcorrectly Ġanswered Ġquestions Ġwithout Ġrequesting Ġtoo Ġmany Ġexplanations Ġand Ġanswers Ġfrom Ġusers . ĠTo Ġevaluate Ġdifferent Ġsystems Ġusing Ġthe Ġcollected Ġdata , Ġwe Ġpropose Ġthe Ġfollowing Ġevaluation Ġmeasure : </s> $$ Ġ ĠS _ D Ġ= Ġ\ frac { n _ c Ġ- Ġw _ i Ġn _ i Ġ- Ġw _ e Ġn _ e Ġ- Ġw _ a Ġn _ a }{ | D | } $$ Ġ Ġ Ġ( E q . Ġ20 ) </s> Here , Ġ$ n _ c $ Ġdenotes Ġthe Ġnumber Ġof Ġcorrectly Ġanswered Ġquestions , Ġ$ n _ i $ Ġdenotes Ġthe Ġnumber Ġof Ġincorrectly Ġanswered Ġquestions , Ġ$ n _ e $ Ġdenotes Ġthe Ġnumber Ġof Ġrequested Ġexplanations , Ġ$ n _ a $ Ġdenotes Ġthe Ġnumber Ġof Ġrequested Ġanswers Ġand Ġ$ | D | $ Ġdenotes Ġthe Ġnumber Ġof Ġsimulated Ġdialog s Ġin Ġthe Ġdataset . Ġ$ w _ i $ Ġ, Ġ$ w _ e $ Ġ, Ġ$ w _ a $ Ġare Ġpenal ization Ġweights . </s> The Ġpenal ization Ġweights Ġare Ġused Ġto Ġcompensate Ġfor Ġdifferent Ġcosts Ġof Ġobtaining Ġdifferent Ġtypes Ġof Ġinformation Ġfrom Ġthe Ġuser . ĠFor Ġexample , Ġgaining Ġbroader Ġexplanation Ġfrom Ġthe Ġuser Ġis Ġrelatively Ġsimple Ġbecause Ġit Ġis Ġin Ġtheir Ġfavor Ġto Ġcooperate Ġwith Ġthe Ġsystem Ġon Ġa Ġquestion Ġthey Ġare Ġinterested Ġin . ĠHowever , Ġobtaining Ġcorrect Ġanswers Ġfrom Ġusers Ġis Ġsignificantly Ġmore Ġdifficult Ġbecause Ġthe Ġsystem Ġdoes Ġnot Ġalways Ġhave Ġthe Ġchance Ġto Ġask Ġthe Ġquestion Ġand Ġthe Ġuser Ġdoes Ġnot Ġhave Ġto Ġknow Ġthe Ġcorrect Ġanswer Ġfor Ġit . </s> To Ġmake Ġthe Ġevaluations Ġcomparable Ġbetween Ġdifferent Ġsystems Ġwe Ġrecommend Ġusing Ġour Ġevaluation Ġscripts Ġincluded Ġwith Ġthe Ġdataset Ġwith Ġfollowing Ġpenal ization Ġweights Ġthat Ġreflect Ġour Ġintuition Ġfor Ġgaining Ġinformation Ġfrom Ġusers : </s> âĢĵ Ġincorrect Ġanswers Ġare Ġpenal ized Ġsignificantly , </s> âĢĵ Ġexplanations Ġare Ġquite Ġcheap ; Ġtherefore , Ġwe Ġwill Ġpenal ize Ġthem Ġjust Ġslightly , </s> âĢĵ Ġgaining Ġquestion âĢ Ļ s Ġanswer Ġfrom Ġusers Ġis Ġharder Ġthan Ġgaining Ġexplanations . </s> Answer ĠExt raction ĠAccuracy </s> It Ġis Ġquite Ġchallenging Ġto Ġfind Ġappropriate Ġentity Ġin Ġthe Ġknowledge Ġbase Ġeven Ġthough Ġthe Ġuser Ġprovided Ġthe Ġcorrect Ġanswer . ĠTherefore , Ġwe Ġpropose Ġanother Ġmetric Ġrelevant Ġto Ġour Ġdataset . ĠThis Ġmetric Ġis Ġthe Ġaccuracy Ġof Ġentity Ġextraction Ġwhich Ġmeasures Ġhow Ġmany Ġtimes Ġwas Ġextracted Ġa Ġcorrect Ġanswer Ġfrom Ġanswer Ġhints Ġprovided Ġby Ġthe Ġuser Ġin Ġdialog s Ġannot ated Ġas Ġcorrectly Ġanswered . </s> Future ĠWork </s> Our Ġfuture Ġwork Ġwill Ġbe Ġmainly Ġfocused Ġon Ġproviding Ġa Ġbaseline Ġsystem Ġfor Ġinteractive Ġlearning Ġwhich Ġwill Ġbe Ġevaluated Ġon Ġthe Ġdataset . ĠWe Ġare Ġalso Ġplanning Ġimprovements Ġfor Ġdialog Ġmanagement Ġthat Ġis Ġused Ġto Ġgain Ġexplanations Ġduring Ġthe Ġdata Ġcollection . ĠWe Ġbelieve Ġthat Ġwith Ġconversation Ġabout Ġspecific Ġaspects Ġof Ġthe Ġdiscussed Ġquestion Ġit Ġwill Ġbe Ġpossible Ġto Ġgain Ġeven Ġmore Ġinteresting Ġinformation Ġfrom Ġusers . ĠThe Ġother Ġarea Ġof Ġour Ġinterest Ġis Ġin Ġpossibilities Ġto Ġimprove Ġquestion Ġanswering Ġaccuracy Ġon Ġtest Ġquestions Ġof ĠSimple Ġquestion Ġdataset Ġwith Ġthe Ġextra Ġinformation Ġcontained Ġin Ġthe Ġcollected Ġdialog s . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġpresented Ġa Ġnovel Ġway Ġhow Ġto Ġevaluate Ġdifferent Ġinteractive Ġlearning Ġapproaches Ġfor Ġdialog Ġmodels . ĠThe Ġevaluation Ġcovers Ġtwo Ġchallenging Ġaspects Ġof Ġinteractive Ġlearning . ĠFirst , Ġit Ġscores Ġefficiency Ġof Ġusing Ġinformation Ġgained Ġfrom Ġusers Ġin Ġsimulated Ġquestion Ġanswering Ġdialog s . ĠSecond , Ġit Ġmeasures Ġaccuracy Ġon Ġanswer Ġhints Ġunderstanding . </s> For Ġpurposes Ġof Ġevaluation Ġwe Ġcollected Ġa Ġdataset Ġfrom Ġconvers ational Ġdialog s Ġwith Ġworkers Ġon Ġcrowds ourcing Ġplatform ĠCrowd Fl ower . ĠThose Ġdialog s Ġwere Ġannot ated Ġwith Ġexpert Ġannot ators Ġand Ġpublished Ġunder ĠCreative ĠCommons Ġ4 . 0 ĠBY - SA Ġlicense Ġon Ġl ind at . ĠWe Ġalso Ġprovide Ġevaluation Ġscripts Ġwith Ġthe Ġdataset Ġthat Ġshould Ġensure Ġcomparable Ġevaluation Ġof Ġdifferent Ġinteractive Ġlearning Ġapproaches . </s> Acknowled gments </s> This Ġwork Ġwas Ġfunded Ġby Ġthe ĠMinistry Ġof ĠEducation , ĠYouth Ġand ĠSports Ġof Ġthe ĠCzech ĠRepublic Ġunder Ġthe Ġgrant Ġagreement ĠL K 112 21 Ġand Ġcore Ġresearch Ġfunding , ĠSV V Ġproject Ġ260 Ġ224 , Ġand ĠGA UK Ġgrant Ġ11 705 16 Ġof ĠCharles ĠUniversity Ġin ĠPrague . ĠIt Ġused Ġlanguage Ġresources Ġstored Ġand Ġdistributed Ġby Ġthe ĠL IND AT / CL AR IN Ġproject Ġof Ġthe ĠMinistry Ġof ĠEducation , ĠYouth Ġand ĠSports Ġof Ġthe ĠCzech ĠRepublic Ġ( project ĠLM 201 50 71 ).
Introduction Nowadays, dialog systems are usually designed for a single domain BIBREF0 . They store data in a well-defined format with a fixed number of attributes for entities that the system can provide. Because data in this format can be stored as a two-dimensional table within a relational database, we call the data flat. This data representation allows the system to query the database in a simple and efficient way. It also allows to keep the dialog state in the form of slots (which usually correspond to columns in the table) and track it through the dialog using probabilistic belief tracking BIBREF1 , BIBREF2 . However, the well-defined structure of the database of a typical dialog system comes with a high cost of extending it as every piece of new information has to fit the format. This is especially a problem when we one is adapting the system for a new domain because its entities could have different attributes. A dialog system based on knowledge bases offers many advantages. First, the knowledge base, which can be represented as knowledge graph containing entities connected by relations, is much more flexible than the relational database. Second, freely available knowledge bases, such as Freebase, Wikidata, etc. contain an enormous amount of structured information, and are still growing. A dialog system which is capable of working with this type of information would be therefore very useful. In this paper we propose a dataset aiming to help develop and evaluate dialog systems based on knowledge bases by interactive learning motivated in Section "Motivation" Section "Dialog policies" describes policies that can be used for retrieving information from knowledge bases. In Section "Dialog Simulation" is introduced a dialog simulation from natural conversations which we use for evaluation of interactive learning. The dataset collection process allowing the dialog simulation is described in Section "Dataset Collection Process" and is followed by properties of the resulting dataset in Section "Dataset Properties" Evaluation guidelines with proposed metrics can be found in Section "Interactive Learning Evaluation" The planned future work is summarized in Section "Future Work" We conclude the paper with Section "Conclusion" Motivation From the point of view of dialog systems providing general information from a knowledge base, the most limiting factor is that a large portion of the questions is understood poorly. Current approaches BIBREF3 , BIBREF4 can only achieve around 50% accuracy on some question answering datasets. Therefore, we think that there is a room for improvements which can be achieved by interactively asking for additional information in conversational dialogs with users. This extra information can be used for improving policies of dialog systems. We call this approach the interactive learning from dialogs. We can improve dialog systems in several aspects through interactive learning in a direct interaction with users. First, the most straightforward way obviously is getting the correct answer for questions that the system does not know. We can try to ask users for answers on questions that the system encountered in a conversation with a different user and did not understand it. Second, the system can ask the user for a broader explanation of a question. This explanation could help the system to understand the question and provide the correct answer. In addition, the system can learn correct policy for the question which allows providing answers without asking any extra information for similar questions next time. We hypothesize that users are willing to give such explanations because it could help them to find answers for their own questions. The last source of information that we consider for interactive learning is rephrasing, which could help when the system does know the concept but does not know the correct wording. This area is extensively studied for the purposes of information retrieval BIBREF5 , BIBREF6 . The main purpose of the collected dataset is to enable interactive learning using the steps proposed above and potentially to evaluate how different systems perform on this task. Dialog policies The obvious difficulty when developing a dialog system is finding a way how to identify the piece of information that the user is interested in. This is especially a problem for dialog systems based on knowledge graphs containing a large amount of complex structured information. While a similar problem is being solved in a task of question answering, dialog systems have more possibilities of identifying the real intention of the user. For example, a dialog system can ask for additional information during the dialog. We distinguish three different basic approaches to requesting knowledge bases: A combination of the above approaches is also possible. For example, we can imagine scenarios where the dialog system starts with hand-crafted rules, which are subsequently interactively improved through dialogs with its users. With a growing demand for open domain dialog systems, it shows that creating hand-crafted policies does not scale well - therefore, machine learning approaches are gaining on popularity. Many public datasets for offline learning have been published BIBREF8 , BIBREF7 . However, to our knowledge, no public datasets for interactive learning are available. To fill this gap, we collected a dataset which enables to train interactively learned policies through a simulated interaction with users. Dialog Simulation Offline evaluation of interactive dialogs on real data is difficult because different policies can lead to different variants of the dialog. Our solution to this issue is to collect data in a way that allows us to simulate all dialog variants possible according to any policy. The dialog variants we are considering for interactive learning differ only in presence of several parts of the dialog. Therefore, we can collect dialogs containing all information used for interactive learning and omit those parts that were not requested by the policy. We collected the dataset (see Section "Dataset Collection Process" ) that enables simulation where the policy can decide how much extra information to the question it requests. If the question is clear to the system it can attempt to answer the question without any other information. It can also ask for a broader explanation with a possibility to answer the question afterwards. If the system decides not to answer the question, we can simulate rerouting the question to another user, to try to obtain the answer from them. The principle of simulated user's answer is shown in the Figure 1 . Note that the simulated user’s answer can be incorrect because human users naturally made mistakes. We intentionally keep these mistakes in the dataset because real systems must address them as well. Dataset Collection Process A perfect data collection scenario for our dataset would use real running dialog system providing general information from the knowledge base to real users. This system could then ask for explanations and answers for questions which it is not able to answer. However, getting access to systems with real users is usually hard. Therefore, we used the crowdsourcing platform CrowdFlower (CF) for our data collection. A CF worker gets a task instructing them to use our chat-like interface to help the system with a question which is randomly selected from training examples of Simple questions BIBREF7 dataset. To complete the task user has to communicate with the system through the three phase dialog discussing question paraphrase (see Section "Interactive Learning Evaluation" ), explanation (see Section "Future Work" ) and answer of the question (see Section "Conclusion" ). To avoid poor English level of dialogs we involved CF workers from English speaking countries only. The collected dialogs has been annotated (see Section "Acknowledgments" ) by expert annotators afterwards. The described procedure leads to dialogs like the one shown in the Figure 2 . Question Paraphrasing At beginning of the dialog, the system is requesting the user to paraphrase question that the system does not understand. The main goal of this first phase is to let the user get familiar with the presented question and to get alternative wordings of the posed question. Question Explanation In the second phase, the user is asked for an explanation of the question. We expect the explanation to be different enough from the original question (in terms of the number of common words between the question and the explanation). If the explanation is too similar to the question, the user is notified that their explanation is not broad enough and they must provide a better one. Question Answer With the valid explanation the dialog turns into the last phase where the user is asked for a correct answer to the original question. The system requires the user to answer with a full sentence. In practical experiments this has shown as a useful decision because it improves system's ability to reveal cheaters. We can simply measure the connection (in terms of common words ) between question and the answer sentence. This allows to reject completely irrelevant answers. Annotation The correct answer for question in each dialog is available from Simple questions dataset. Answers are in form of Freebase entities identified by unique id. For evaluation purposes we need information whether dialog contains the answer which is consistent with the entity from Simple questions, the answer with another entity or whether the dialog does not contain any answer. While the annotation process is quite simple, we did not need crowdsourcing for the process. Natural Language Understanding (NLU) The collection system needs to recognize following dialog acts from user utterances during all phases of the dialog: – user does not want to provide requested information, – user agrees to provide requested information, – user does not know the requested information, – user tries chit chat with the system (hello, bye, who are you...), – none of the above, interpreted as user is giving information requested by the system. Parsing of the dialog acts is made by hand written rules using templates and keyword spotting. The templates and keywords were manually collected from frequent expressions used by CF workers during preparation runs of the dataset collection process (google it, check wikipedia, I would need... $\rightarrow $ Negate). Dataset Properties We collected the dataset with 1900 dialogs and 8533 turns. Topics discussed in dialogs are questions randomly chosen from training examples of Simple questions BIBREF7 dataset. From this dataset we also took the correct answers in form of Freebase entities. Our dataset consists of standard data split into training, development and test files. The basic properties of those files are as follows: Each file contains complete dialogs enriched by outputs of NLU (see Section "Natural Language Understanding (NLU)" ) that were used during the data collection. On top of that, each dialog is labeled by the correct answer for the question and expert annotation of the user answer hint which tells whether the hint points to the correct answer, incorrect answer, or no answer at all. 351 of all collected dialogs contain correct answer provided by users and 702 dialogs have incorrect answer. In the remaining 847 dialogs users did not want to answer the question. The collected dialogs also contain 1828 paraphrases and 1539 explanations for 1870 questions. An answer for a question was labeled as correct by annotators only when it was evident to them that the answer points to the same Freebase entity that was present in Simple questions dataset for that particular question. However, a large amount of questions from that dataset is quite general - with many possible answers. Therefore lot of answers from users were labeled as incorrect even though those answers perfectly fit the question. Our annotators identified that 285 of the incorrect answers were answers for such general questions. Example of this situation can be demonstrated by question 'Name an actor' which was correctly answered by 'Brad Pitt is an actor', however, to be consistent with Simple questions annotation, which is 'Kelly Atwood', annotators were forced to mark it as an incorrect answer. Interactive Learning Evaluation A perfect interactive learning model would be able to learn anything interactively from test dialogs during testing, which would allow us to measure progress of the model from scratch over the course of time. However, a development of such model would be unnecessarily hard, therefore we provide training dialogs which can be used for feature extraction and other engineering related to interactive learning from dialogs in natural language. Model development is further supported with labeled validation data for parameter tuning. We propose two evaluation metrics for comparing interactive learning models. First metric (see Section "Efficiency Score" ) scores amount of information required by the model, second metric (see Section "Answer Extraction Accuracy" ) is accuracy of answer extraction from user utterances. All models must base their answers only on information gained from training dialogs and testing dialogs seen during the simulation so far, to ensure that the score will reflect the interactive learning of the model instead of general question answering. Efficiency Score The simulation of dialogs from our dataset allows to evaluate how efficient a dialog system is in using information gained from users. The dialog system should maximize the number of correctly answered questions without requesting too many explanations and answers from users. To evaluate different systems using the collected data, we propose the following evaluation measure: $$  S_D = \frac{n_c - w_i n_i - w_e n_e - w_a n_a}{|D|}$$   (Eq. 20) Here, $n_c$ denotes the number of correctly answered questions, $n_i$ denotes the number of incorrectly answered questions, $n_e$ denotes the number of requested explanations, $n_a$ denotes the number of requested answers and $|D|$ denotes the number of simulated dialogs in the dataset. $w_i$ , $w_e$ , $w_a$ are penalization weights. The penalization weights are used to compensate for different costs of obtaining different types of information from the user. For example, gaining broader explanation from the user is relatively simple because it is in their favor to cooperate with the system on a question they are interested in. However, obtaining correct answers from users is significantly more difficult because the system does not always have the chance to ask the question and the user does not have to know the correct answer for it. To make the evaluations comparable between different systems we recommend using our evaluation scripts included with the dataset with following penalization weights that reflect our intuition for gaining information from users: – incorrect answers are penalized significantly, – explanations are quite cheap; therefore, we will penalize them just slightly, – gaining question’s answer from users is harder than gaining explanations. Answer Extraction Accuracy It is quite challenging to find appropriate entity in the knowledge base even though the user provided the correct answer. Therefore, we propose another metric relevant to our dataset. This metric is the accuracy of entity extraction which measures how many times was extracted a correct answer from answer hints provided by the user in dialogs annotated as correctly answered. Future Work Our future work will be mainly focused on providing a baseline system for interactive learning which will be evaluated on the dataset. We are also planning improvements for dialog management that is used to gain explanations during the data collection. We believe that with conversation about specific aspects of the discussed question it will be possible to gain even more interesting information from users. The other area of our interest is in possibilities to improve question answering accuracy on test questions of Simple question dataset with the extra information contained in the collected dialogs. Conclusion In this paper, we presented a novel way how to evaluate different interactive learning approaches for dialog models. The evaluation covers two challenging aspects of interactive learning. First, it scores efficiency of using information gained from users in simulated question answering dialogs. Second, it measures accuracy on answer hints understanding. For purposes of evaluation we collected a dataset from conversational dialogs with workers on crowdsourcing platform CrowdFlower. Those dialogs were annotated with expert annotators and published under Creative Commons 4.0 BY-SA license on lindat. We also provide evaluation scripts with the dataset that should ensure comparable evaluation of different interactive learning approaches. Acknowledgments This work was funded by the Ministry of Education, Youth and Sports of the Czech Republic under the grant agreement LK11221 and core research funding, SVV project 260 224, and GAUK grant 1170516 of Charles University in Prague. It used language resources stored and distributed by the LINDAT/CLARIN project of the Ministry of Education, Youth and Sports of the Czech Republic (project LM2015071).
What is the average length of dialog?
bc16ce6e9c61ae13d46970ebe6c4728a47f8f425
What Ġis Ġthe Ġaverage Ġlength Ġof Ġdialog ?
4 . 49 Ġturns
<s> What Ġis Ġthe Ġaverage Ġlength Ġof Ġdialog ? </s> Introduction </s> Now adays , Ġdialog Ġsystems Ġare Ġusually Ġdesigned Ġfor Ġa Ġsingle Ġdomain ĠB IB REF 0 Ġ. ĠThey Ġstore Ġdata Ġin Ġa Ġwell - defined Ġformat Ġwith Ġa Ġfixed Ġnumber Ġof Ġattributes Ġfor Ġentities Ġthat Ġthe Ġsystem Ġcan Ġprovide . ĠBecause Ġdata Ġin Ġthis Ġformat Ġcan Ġbe Ġstored Ġas Ġa Ġtwo - dimensional Ġtable Ġwithin Ġa Ġrelational Ġdatabase , Ġwe Ġcall Ġthe Ġdata Ġflat . ĠThis Ġdata Ġrepresentation Ġallows Ġthe Ġsystem Ġto Ġquery Ġthe Ġdatabase Ġin Ġa Ġsimple Ġand Ġefficient Ġway . ĠIt Ġalso Ġallows Ġto Ġkeep Ġthe Ġdialog Ġstate Ġin Ġthe Ġform Ġof Ġslots Ġ( which Ġusually Ġcorrespond Ġto Ġcolumns Ġin Ġthe Ġtable ) Ġand Ġtrack Ġit Ġthrough Ġthe Ġdialog Ġusing Ġprob abil istic Ġbelief Ġtracking ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ. </s> However , Ġthe Ġwell - defined Ġstructure Ġof Ġthe Ġdatabase Ġof Ġa Ġtypical Ġdialog Ġsystem Ġcomes Ġwith Ġa Ġhigh Ġcost Ġof Ġextending Ġit Ġas Ġevery Ġpiece Ġof Ġnew Ġinformation Ġhas Ġto Ġfit Ġthe Ġformat . ĠThis Ġis Ġespecially Ġa Ġproblem Ġwhen Ġwe Ġone Ġis Ġadapting Ġthe Ġsystem Ġfor Ġa Ġnew Ġdomain Ġbecause Ġits Ġentities Ġcould Ġhave Ġdifferent Ġattributes . </s> A Ġdialog Ġsystem Ġbased Ġon Ġknowledge Ġbases Ġoffers Ġmany Ġadvantages . ĠFirst , Ġthe Ġknowledge Ġbase , Ġwhich Ġcan Ġbe Ġrepresented Ġas Ġknowledge Ġgraph Ġcontaining Ġentities Ġconnected Ġby Ġrelations , Ġis Ġmuch Ġmore Ġflexible Ġthan Ġthe Ġrelational Ġdatabase . ĠSecond , Ġfreely Ġavailable Ġknowledge Ġbases , Ġsuch Ġas ĠFree base , ĠWik id ata , Ġetc . Ġcontain Ġan Ġenormous Ġamount Ġof Ġstructured Ġinformation , Ġand Ġare Ġstill Ġgrowing . ĠA Ġdialog Ġsystem Ġwhich Ġis Ġcapable Ġof Ġworking Ġwith Ġthis Ġtype Ġof Ġinformation Ġwould Ġbe Ġtherefore Ġvery Ġuseful . </s> In Ġthis Ġpaper Ġwe Ġpropose Ġa Ġdataset Ġaiming Ġto Ġhelp Ġdevelop Ġand Ġevaluate Ġdialog Ġsystems Ġbased Ġon Ġknowledge Ġbases Ġby Ġinteractive Ġlearning Ġmotivated Ġin ĠSection Ġ" Mot ivation " ĠSection Ġ" Dialog Ġpolicies " Ġdescribes Ġpolicies Ġthat Ġcan Ġbe Ġused Ġfor Ġretrieving Ġinformation Ġfrom Ġknowledge Ġbases . ĠIn ĠSection Ġ" Dialog ĠSimulation " Ġis Ġintroduced Ġa Ġdialog Ġsimulation Ġfrom Ġnatural Ġconversations Ġwhich Ġwe Ġuse Ġfor Ġevaluation Ġof Ġinteractive Ġlearning . ĠThe Ġdataset Ġcollection Ġprocess Ġallowing Ġthe Ġdialog Ġsimulation Ġis Ġdescribed Ġin ĠSection Ġ" Dat as et ĠCollection ĠProcess " Ġand Ġis Ġfollowed Ġby Ġproperties Ġof Ġthe Ġresulting Ġdataset Ġin ĠSection Ġ" Dat as et ĠProperties " ĠEvaluation Ġguidelines Ġwith Ġproposed Ġmetrics Ġcan Ġbe Ġfound Ġin ĠSection Ġ" Inter active ĠLearning ĠEvaluation " ĠThe Ġplanned Ġfuture Ġwork Ġis Ġsummarized Ġin ĠSection Ġ" Future ĠWork " ĠWe Ġconclude Ġthe Ġpaper Ġwith ĠSection Ġ" Conclusion " </s> Mot ivation </s> From Ġthe Ġpoint Ġof Ġview Ġof Ġdialog Ġsystems Ġproviding Ġgeneral Ġinformation Ġfrom Ġa Ġknowledge Ġbase , Ġthe Ġmost Ġlimiting Ġfactor Ġis Ġthat Ġa Ġlarge Ġportion Ġof Ġthe Ġquestions Ġis Ġunderstood Ġpoorly . </s> Current Ġapproaches ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġcan Ġonly Ġachieve Ġaround Ġ50 % Ġaccuracy Ġon Ġsome Ġquestion Ġanswering Ġdatasets . ĠTherefore , Ġwe Ġthink Ġthat Ġthere Ġis Ġa Ġroom Ġfor Ġimprovements Ġwhich Ġcan Ġbe Ġachieved Ġby Ġinteract ively Ġasking Ġfor Ġadditional Ġinformation Ġin Ġconvers ational Ġdialog s Ġwith Ġusers . ĠThis Ġextra Ġinformation Ġcan Ġbe Ġused Ġfor Ġimproving Ġpolicies Ġof Ġdialog Ġsystems . ĠWe Ġcall Ġthis Ġapproach Ġthe Ġinteractive Ġlearning Ġfrom Ġdialog s . </s> We Ġcan Ġimprove Ġdialog Ġsystems Ġin Ġseveral Ġaspects Ġthrough Ġinteractive Ġlearning Ġin Ġa Ġdirect Ġinteraction Ġwith Ġusers . ĠFirst , Ġthe Ġmost Ġstraightforward Ġway Ġobviously Ġis Ġgetting Ġthe Ġcorrect Ġanswer Ġfor Ġquestions Ġthat Ġthe Ġsystem Ġdoes Ġnot Ġknow . ĠWe Ġcan Ġtry Ġto Ġask Ġusers Ġfor Ġanswers Ġon Ġquestions Ġthat Ġthe Ġsystem Ġencountered Ġin Ġa Ġconversation Ġwith Ġa Ġdifferent Ġuser Ġand Ġdid Ġnot Ġunderstand Ġit . ĠSecond , Ġthe Ġsystem Ġcan Ġask Ġthe Ġuser Ġfor Ġa Ġbroader Ġexplanation Ġof Ġa Ġquestion . ĠThis Ġexplanation Ġcould Ġhelp Ġthe Ġsystem Ġto Ġunderstand Ġthe Ġquestion Ġand Ġprovide Ġthe Ġcorrect Ġanswer . ĠIn Ġaddition , Ġthe Ġsystem Ġcan Ġlearn Ġcorrect Ġpolicy Ġfor Ġthe Ġquestion Ġwhich Ġallows Ġproviding Ġanswers Ġwithout Ġasking Ġany Ġextra Ġinformation Ġfor Ġsimilar Ġquestions Ġnext Ġtime . ĠWe Ġhypothes ize Ġthat Ġusers Ġare Ġwilling Ġto Ġgive Ġsuch Ġexplanations Ġbecause Ġit Ġcould Ġhelp Ġthem Ġto Ġfind Ġanswers Ġfor Ġtheir Ġown Ġquestions . ĠThe Ġlast Ġsource Ġof Ġinformation Ġthat Ġwe Ġconsider Ġfor Ġinteractive Ġlearning Ġis Ġre ph r asing , Ġwhich Ġcould Ġhelp Ġwhen Ġthe Ġsystem Ġdoes Ġknow Ġthe Ġconcept Ġbut Ġdoes Ġnot Ġknow Ġthe Ġcorrect Ġwording . ĠThis Ġarea Ġis Ġextensively Ġstudied Ġfor Ġthe Ġpurposes Ġof Ġinformation Ġretrieval ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ. </s> The Ġmain Ġpurpose Ġof Ġthe Ġcollected Ġdataset Ġis Ġto Ġenable Ġinteractive Ġlearning Ġusing Ġthe Ġsteps Ġproposed Ġabove Ġand Ġpotentially Ġto Ġevaluate Ġhow Ġdifferent Ġsystems Ġperform Ġon Ġthis Ġtask . </s> Dialog Ġpolicies </s> The Ġobvious Ġdifficulty Ġwhen Ġdeveloping Ġa Ġdialog Ġsystem Ġis Ġfinding Ġa Ġway Ġhow Ġto Ġidentify Ġthe Ġpiece Ġof Ġinformation Ġthat Ġthe Ġuser Ġis Ġinterested Ġin . ĠThis Ġis Ġespecially Ġa Ġproblem Ġfor Ġdialog Ġsystems Ġbased Ġon Ġknowledge Ġgraphs Ġcontaining Ġa Ġlarge Ġamount Ġof Ġcomplex Ġstructured Ġinformation . ĠWhile Ġa Ġsimilar Ġproblem Ġis Ġbeing Ġsolved Ġin Ġa Ġtask Ġof Ġquestion Ġanswering , Ġdialog Ġsystems Ġhave Ġmore Ġpossibilities Ġof Ġidentifying Ġthe Ġreal Ġintention Ġof Ġthe Ġuser . ĠFor Ġexample , Ġa Ġdialog Ġsystem Ġcan Ġask Ġfor Ġadditional Ġinformation Ġduring Ġthe Ġdialog . </s> We Ġdistinguish Ġthree Ġdifferent Ġbasic Ġapproaches Ġto Ġrequesting Ġknowledge Ġbases : </s> A Ġcombination Ġof Ġthe Ġabove Ġapproaches Ġis Ġalso Ġpossible . ĠFor Ġexample , Ġwe Ġcan Ġimagine Ġscenarios Ġwhere Ġthe Ġdialog Ġsystem Ġstarts Ġwith Ġhand - crafted Ġrules , Ġwhich Ġare Ġsubsequently Ġinteract ively Ġimproved Ġthrough Ġdialog s Ġwith Ġits Ġusers . ĠWith Ġa Ġgrowing Ġdemand Ġfor Ġopen Ġdomain Ġdialog Ġsystems , Ġit Ġshows Ġthat Ġcreating Ġhand - crafted Ġpolicies Ġdoes Ġnot Ġscale Ġwell Ġ- Ġtherefore , Ġmachine Ġlearning Ġapproaches Ġare Ġgaining Ġon Ġpopularity . ĠMany Ġpublic Ġdatasets Ġfor Ġoffline Ġlearning Ġhave Ġbeen Ġpublished ĠB IB REF 8 Ġ, ĠB IB REF 7 Ġ. ĠHowever , Ġto Ġour Ġknowledge , Ġno Ġpublic Ġdatasets Ġfor Ġinteractive Ġlearning Ġare Ġavailable . ĠTo Ġfill Ġthis Ġgap , Ġwe Ġcollected Ġa Ġdataset Ġwhich Ġenables Ġto Ġtrain Ġinteract ively Ġlearned Ġpolicies Ġthrough Ġa Ġsimulated Ġinteraction Ġwith Ġusers . </s> Dialog ĠSimulation </s> Offline Ġevaluation Ġof Ġinteractive Ġdialog s Ġon Ġreal Ġdata Ġis Ġdifficult Ġbecause Ġdifferent Ġpolicies Ġcan Ġlead Ġto Ġdifferent Ġvariants Ġof Ġthe Ġdialog . ĠOur Ġsolution Ġto Ġthis Ġissue Ġis Ġto Ġcollect Ġdata Ġin Ġa Ġway Ġthat Ġallows Ġus Ġto Ġsimulate Ġall Ġdialog Ġvariants Ġpossible Ġaccording Ġto Ġany Ġpolicy . </s> The Ġdialog Ġvariants Ġwe Ġare Ġconsidering Ġfor Ġinteractive Ġlearning Ġdiffer Ġonly Ġin Ġpresence Ġof Ġseveral Ġparts Ġof Ġthe Ġdialog . ĠTherefore , Ġwe Ġcan Ġcollect Ġdialog s Ġcontaining Ġall Ġinformation Ġused Ġfor Ġinteractive Ġlearning Ġand Ġomit Ġthose Ġparts Ġthat Ġwere Ġnot Ġrequested Ġby Ġthe Ġpolicy . </s> We Ġcollected Ġthe Ġdataset Ġ( see ĠSection Ġ" Dat as et ĠCollection ĠProcess " Ġ) Ġthat Ġenables Ġsimulation Ġwhere Ġthe Ġpolicy Ġcan Ġdecide Ġhow Ġmuch Ġextra Ġinformation Ġto Ġthe Ġquestion Ġit Ġrequests . ĠIf Ġthe Ġquestion Ġis Ġclear Ġto Ġthe Ġsystem Ġit Ġcan Ġattempt Ġto Ġanswer Ġthe Ġquestion Ġwithout Ġany Ġother Ġinformation . ĠIt Ġcan Ġalso Ġask Ġfor Ġa Ġbroader Ġexplanation Ġwith Ġa Ġpossibility Ġto Ġanswer Ġthe Ġquestion Ġafterwards . ĠIf Ġthe Ġsystem Ġdecides Ġnot Ġto Ġanswer Ġthe Ġquestion , Ġwe Ġcan Ġsimulate Ġre r outing Ġthe Ġquestion Ġto Ġanother Ġuser , Ġto Ġtry Ġto Ġobtain Ġthe Ġanswer Ġfrom Ġthem . ĠThe Ġprinciple Ġof Ġsimulated Ġuser 's Ġanswer Ġis Ġshown Ġin Ġthe ĠFigure Ġ1 Ġ. </s> Note Ġthat Ġthe Ġsimulated Ġuser âĢ Ļ s Ġanswer Ġcan Ġbe Ġincorrect Ġbecause Ġhuman Ġusers Ġnaturally Ġmade Ġmistakes . ĠWe Ġintentionally Ġkeep Ġthese Ġmistakes Ġin Ġthe Ġdataset Ġbecause Ġreal Ġsystems Ġmust Ġaddress Ġthem Ġas Ġwell . </s> Dat as et ĠCollection ĠProcess </s> A Ġperfect Ġdata Ġcollection Ġscenario Ġfor Ġour Ġdataset Ġwould Ġuse Ġreal Ġrunning Ġdialog Ġsystem Ġproviding Ġgeneral Ġinformation Ġfrom Ġthe Ġknowledge Ġbase Ġto Ġreal Ġusers . ĠThis Ġsystem Ġcould Ġthen Ġask Ġfor Ġexplanations Ġand Ġanswers Ġfor Ġquestions Ġwhich Ġit Ġis Ġnot Ġable Ġto Ġanswer . </s> However , Ġgetting Ġaccess Ġto Ġsystems Ġwith Ġreal Ġusers Ġis Ġusually Ġhard . ĠTherefore , Ġwe Ġused Ġthe Ġcrowds ourcing Ġplatform ĠCrowd Fl ower Ġ( CF ) Ġfor Ġour Ġdata Ġcollection . </s> A ĠCF Ġworker Ġgets Ġa Ġtask Ġinstruct ing Ġthem Ġto Ġuse Ġour Ġchat - like Ġinterface Ġto Ġhelp Ġthe Ġsystem Ġwith Ġa Ġquestion Ġwhich Ġis Ġrandomly Ġselected Ġfrom Ġtraining Ġexamples Ġof ĠSimple Ġquestions ĠB IB REF 7 Ġdataset . ĠTo Ġcomplete Ġthe Ġtask Ġuser Ġhas Ġto Ġcommunicate Ġwith Ġthe Ġsystem Ġthrough Ġthe Ġthree Ġphase Ġdialog Ġdiscussing Ġquestion Ġparaph rase Ġ( see ĠSection Ġ" Inter active ĠLearning ĠEvaluation " Ġ), Ġexplanation Ġ( see ĠSection Ġ" Future ĠWork " Ġ) Ġand Ġanswer Ġof Ġthe Ġquestion Ġ( see ĠSection Ġ" Conclusion " Ġ). ĠTo Ġavoid Ġpoor ĠEnglish Ġlevel Ġof Ġdialog s Ġwe Ġinvolved ĠCF Ġworkers Ġfrom ĠEnglish Ġspeaking Ġcountries Ġonly . ĠThe Ġcollected Ġdialog s Ġhas Ġbeen Ġannot ated Ġ( see ĠSection Ġ" Acknowled gments " Ġ) Ġby Ġexpert Ġannot ators Ġafterwards . </s> The Ġdescribed Ġprocedure Ġleads Ġto Ġdialog s Ġlike Ġthe Ġone Ġshown Ġin Ġthe ĠFigure Ġ2 Ġ. </s> Question ĠPar aph r asing </s> At Ġbeginning Ġof Ġthe Ġdialog , Ġthe Ġsystem Ġis Ġrequesting Ġthe Ġuser Ġto Ġparaph rase Ġquestion Ġthat Ġthe Ġsystem Ġdoes Ġnot Ġunderstand . ĠThe Ġmain Ġgoal Ġof Ġthis Ġfirst Ġphase Ġis Ġto Ġlet Ġthe Ġuser Ġget Ġfamiliar Ġwith Ġthe Ġpresented Ġquestion Ġand Ġto Ġget Ġalternative Ġword ings Ġof Ġthe Ġposed Ġquestion . </s> Question ĠExplan ation </s> In Ġthe Ġsecond Ġphase , Ġthe Ġuser Ġis Ġasked Ġfor Ġan Ġexplanation Ġof Ġthe Ġquestion . ĠWe Ġexpect Ġthe Ġexplanation Ġto Ġbe Ġdifferent Ġenough Ġfrom Ġthe Ġoriginal Ġquestion Ġ( in Ġterms Ġof Ġthe Ġnumber Ġof Ġcommon Ġwords Ġbetween Ġthe Ġquestion Ġand Ġthe Ġexplanation ). ĠIf Ġthe Ġexplanation Ġis Ġtoo Ġsimilar Ġto Ġthe Ġquestion , Ġthe Ġuser Ġis Ġnotified Ġthat Ġtheir Ġexplanation Ġis Ġnot Ġbroad Ġenough Ġand Ġthey Ġmust Ġprovide Ġa Ġbetter Ġone . </s> Question ĠAnswer </s> With Ġthe Ġvalid Ġexplanation Ġthe Ġdialog Ġturns Ġinto Ġthe Ġlast Ġphase Ġwhere Ġthe Ġuser Ġis Ġasked Ġfor Ġa Ġcorrect Ġanswer Ġto Ġthe Ġoriginal Ġquestion . ĠThe Ġsystem Ġrequires Ġthe Ġuser Ġto Ġanswer Ġwith Ġa Ġfull Ġsentence . ĠIn Ġpractical Ġexperiments Ġthis Ġhas Ġshown Ġas Ġa Ġuseful Ġdecision Ġbecause Ġit Ġimproves Ġsystem 's Ġability Ġto Ġreveal Ġche aters . ĠWe Ġcan Ġsimply Ġmeasure Ġthe Ġconnection Ġ( in Ġterms Ġof Ġcommon Ġwords Ġ) Ġbetween Ġquestion Ġand Ġthe Ġanswer Ġsentence . ĠThis Ġallows Ġto Ġreject Ġcompletely Ġirrelevant Ġanswers . </s> An notation </s> The Ġcorrect Ġanswer Ġfor Ġquestion Ġin Ġeach Ġdialog Ġis Ġavailable Ġfrom ĠSimple Ġquestions Ġdataset . ĠAnswers Ġare Ġin Ġform Ġof ĠFree base Ġentities Ġidentified Ġby Ġunique Ġid . ĠFor Ġevaluation Ġpurposes Ġwe Ġneed Ġinformation Ġwhether Ġdialog Ġcontains Ġthe Ġanswer Ġwhich Ġis Ġconsistent Ġwith Ġthe Ġentity Ġfrom ĠSimple Ġquestions , Ġthe Ġanswer Ġwith Ġanother Ġentity Ġor Ġwhether Ġthe Ġdialog Ġdoes Ġnot Ġcontain Ġany Ġanswer . ĠWhile Ġthe Ġannotation Ġprocess Ġis Ġquite Ġsimple , Ġwe Ġdid Ġnot Ġneed Ġcrowds ourcing Ġfor Ġthe Ġprocess . </s> Natural ĠLanguage ĠUnderstanding Ġ( NL U ) </s> The Ġcollection Ġsystem Ġneeds Ġto Ġrecognize Ġfollowing Ġdialog Ġacts Ġfrom Ġuser Ġutter ances Ġduring Ġall Ġphases Ġof Ġthe Ġdialog : </s> âĢĵ Ġuser Ġdoes Ġnot Ġwant Ġto Ġprovide Ġrequested Ġinformation , </s> âĢĵ Ġuser Ġagrees Ġto Ġprovide Ġrequested Ġinformation , </s> âĢĵ Ġuser Ġdoes Ġnot Ġknow Ġthe Ġrequested Ġinformation , </s> âĢĵ Ġuser Ġtries Ġch it Ġchat Ġwith Ġthe Ġsystem Ġ( hello , Ġbye , Ġwho Ġare Ġyou ... ), </s> âĢĵ Ġnone Ġof Ġthe Ġabove , Ġinterpreted Ġas Ġuser Ġis Ġgiving Ġinformation Ġrequested Ġby Ġthe Ġsystem . </s> P ars ing Ġof Ġthe Ġdialog Ġacts Ġis Ġmade Ġby Ġhand Ġwritten Ġrules Ġusing Ġtemplates Ġand Ġkeyword Ġspotting . ĠThe Ġtemplates Ġand Ġkeywords Ġwere Ġmanually Ġcollected Ġfrom Ġfrequent Ġexpressions Ġused Ġby ĠCF Ġworkers Ġduring Ġpreparation Ġruns Ġof Ġthe Ġdataset Ġcollection Ġprocess Ġ( google Ġit , Ġcheck Ġwik ipedia , ĠI Ġwould Ġneed ... Ġ$\ right arrow Ġ$ ĠNeg ate ). </s> Dat as et ĠProperties </s> We Ġcollected Ġthe Ġdataset Ġwith Ġ1900 Ġdialog s Ġand Ġ85 33 Ġturns . ĠTopics Ġdiscussed Ġin Ġdialog s Ġare Ġquestions Ġrandomly Ġchosen Ġfrom Ġtraining Ġexamples Ġof ĠSimple Ġquestions ĠB IB REF 7 Ġdataset . ĠFrom Ġthis Ġdataset Ġwe Ġalso Ġtook Ġthe Ġcorrect Ġanswers Ġin Ġform Ġof ĠFree base Ġentities . </s> Our Ġdataset Ġconsists Ġof Ġstandard Ġdata Ġsplit Ġinto Ġtraining , Ġdevelopment Ġand Ġtest Ġfiles . ĠThe Ġbasic Ġproperties Ġof Ġthose Ġfiles Ġare Ġas Ġfollows : </s> Each Ġfile Ġcontains Ġcomplete Ġdialog s Ġenriched Ġby Ġoutputs Ġof ĠNL U Ġ( see ĠSection Ġ" Natural ĠLanguage ĠUnderstanding Ġ( NL U )" Ġ) Ġthat Ġwere Ġused Ġduring Ġthe Ġdata Ġcollection . ĠOn Ġtop Ġof Ġthat , Ġeach Ġdialog Ġis Ġlabeled Ġby Ġthe Ġcorrect Ġanswer Ġfor Ġthe Ġquestion Ġand Ġexpert Ġannotation Ġof Ġthe Ġuser Ġanswer Ġhint Ġwhich Ġtells Ġwhether Ġthe Ġhint Ġpoints Ġto Ġthe Ġcorrect Ġanswer , Ġincorrect Ġanswer , Ġor Ġno Ġanswer Ġat Ġall . </s> 351 Ġof Ġall Ġcollected Ġdialog s Ġcontain Ġcorrect Ġanswer Ġprovided Ġby Ġusers Ġand Ġ702 Ġdialog s Ġhave Ġincorrect Ġanswer . ĠIn Ġthe Ġremaining Ġ8 47 Ġdialog s Ġusers Ġdid Ġnot Ġwant Ġto Ġanswer Ġthe Ġquestion . ĠThe Ġcollected Ġdialog s Ġalso Ġcontain Ġ18 28 Ġparaph r ases Ġand Ġ15 39 Ġexplanations Ġfor Ġ1870 Ġquestions . </s> An Ġanswer Ġfor Ġa Ġquestion Ġwas Ġlabeled Ġas Ġcorrect Ġby Ġannot ators Ġonly Ġwhen Ġit Ġwas Ġevident Ġto Ġthem Ġthat Ġthe Ġanswer Ġpoints Ġto Ġthe Ġsame ĠFree base Ġentity Ġthat Ġwas Ġpresent Ġin ĠSimple Ġquestions Ġdataset Ġfor Ġthat Ġparticular Ġquestion . ĠHowever , Ġa Ġlarge Ġamount Ġof Ġquestions Ġfrom Ġthat Ġdataset Ġis Ġquite Ġgeneral Ġ- Ġwith Ġmany Ġpossible Ġanswers . ĠTherefore Ġlot Ġof Ġanswers Ġfrom Ġusers Ġwere Ġlabeled Ġas Ġincorrect Ġeven Ġthough Ġthose Ġanswers Ġperfectly Ġfit Ġthe Ġquestion . ĠOur Ġannot ators Ġidentified Ġthat Ġ285 Ġof Ġthe Ġincorrect Ġanswers Ġwere Ġanswers Ġfor Ġsuch Ġgeneral Ġquestions . ĠExample Ġof Ġthis Ġsituation Ġcan Ġbe Ġdemonstrated Ġby Ġquestion Ġ' Name Ġan Ġactor ' Ġwhich Ġwas Ġcorrectly Ġanswered Ġby Ġ' Brad ĠPitt Ġis Ġan Ġactor ', Ġhowever , Ġto Ġbe Ġconsistent Ġwith ĠSimple Ġquestions Ġannotation , Ġwhich Ġis Ġ' Kelly ĠAt wood ', Ġannot ators Ġwere Ġforced Ġto Ġmark Ġit Ġas Ġan Ġincorrect Ġanswer . </s> Inter active ĠLearning ĠEvaluation </s> A Ġperfect Ġinteractive Ġlearning Ġmodel Ġwould Ġbe Ġable Ġto Ġlearn Ġanything Ġinteract ively Ġfrom Ġtest Ġdialog s Ġduring Ġtesting , Ġwhich Ġwould Ġallow Ġus Ġto Ġmeasure Ġprogress Ġof Ġthe Ġmodel Ġfrom Ġscratch Ġover Ġthe Ġcourse Ġof Ġtime . ĠHowever , Ġa Ġdevelopment Ġof Ġsuch Ġmodel Ġwould Ġbe Ġunnecessarily Ġhard , Ġtherefore Ġwe Ġprovide Ġtraining Ġdialog s Ġwhich Ġcan Ġbe Ġused Ġfor Ġfeature Ġextraction Ġand Ġother Ġengineering Ġrelated Ġto Ġinteractive Ġlearning Ġfrom Ġdialog s Ġin Ġnatural Ġlanguage . ĠModel Ġdevelopment Ġis Ġfurther Ġsupported Ġwith Ġlabeled Ġvalidation Ġdata Ġfor Ġparameter Ġtuning . </s> We Ġpropose Ġtwo Ġevaluation Ġmetrics Ġfor Ġcomparing Ġinteractive Ġlearning Ġmodels . ĠFirst Ġmetric Ġ( see ĠSection Ġ" E fficiency ĠScore " Ġ) Ġscores Ġamount Ġof Ġinformation Ġrequired Ġby Ġthe Ġmodel , Ġsecond Ġmetric Ġ( see ĠSection Ġ" Answer ĠExt raction ĠAccuracy " Ġ) Ġis Ġaccuracy Ġof Ġanswer Ġextraction Ġfrom Ġuser Ġutter ances . ĠAll Ġmodels Ġmust Ġbase Ġtheir Ġanswers Ġonly Ġon Ġinformation Ġgained Ġfrom Ġtraining Ġdialog s Ġand Ġtesting Ġdialog s Ġseen Ġduring Ġthe Ġsimulation Ġso Ġfar , Ġto Ġensure Ġthat Ġthe Ġscore Ġwill Ġreflect Ġthe Ġinteractive Ġlearning Ġof Ġthe Ġmodel Ġinstead Ġof Ġgeneral Ġquestion Ġanswering . </s> E fficiency ĠScore </s> The Ġsimulation Ġof Ġdialog s Ġfrom Ġour Ġdataset Ġallows Ġto Ġevaluate Ġhow Ġefficient Ġa Ġdialog Ġsystem Ġis Ġin Ġusing Ġinformation Ġgained Ġfrom Ġusers . ĠThe Ġdialog Ġsystem Ġshould Ġmaximize Ġthe Ġnumber Ġof Ġcorrectly Ġanswered Ġquestions Ġwithout Ġrequesting Ġtoo Ġmany Ġexplanations Ġand Ġanswers Ġfrom Ġusers . ĠTo Ġevaluate Ġdifferent Ġsystems Ġusing Ġthe Ġcollected Ġdata , Ġwe Ġpropose Ġthe Ġfollowing Ġevaluation Ġmeasure : </s> $$ Ġ ĠS _ D Ġ= Ġ\ frac { n _ c Ġ- Ġw _ i Ġn _ i Ġ- Ġw _ e Ġn _ e Ġ- Ġw _ a Ġn _ a }{ | D | } $$ Ġ Ġ Ġ( E q . Ġ20 ) </s> Here , Ġ$ n _ c $ Ġdenotes Ġthe Ġnumber Ġof Ġcorrectly Ġanswered Ġquestions , Ġ$ n _ i $ Ġdenotes Ġthe Ġnumber Ġof Ġincorrectly Ġanswered Ġquestions , Ġ$ n _ e $ Ġdenotes Ġthe Ġnumber Ġof Ġrequested Ġexplanations , Ġ$ n _ a $ Ġdenotes Ġthe Ġnumber Ġof Ġrequested Ġanswers Ġand Ġ$ | D | $ Ġdenotes Ġthe Ġnumber Ġof Ġsimulated Ġdialog s Ġin Ġthe Ġdataset . Ġ$ w _ i $ Ġ, Ġ$ w _ e $ Ġ, Ġ$ w _ a $ Ġare Ġpenal ization Ġweights . </s> The Ġpenal ization Ġweights Ġare Ġused Ġto Ġcompensate Ġfor Ġdifferent Ġcosts Ġof Ġobtaining Ġdifferent Ġtypes Ġof Ġinformation Ġfrom Ġthe Ġuser . ĠFor Ġexample , Ġgaining Ġbroader Ġexplanation Ġfrom Ġthe Ġuser Ġis Ġrelatively Ġsimple Ġbecause Ġit Ġis Ġin Ġtheir Ġfavor Ġto Ġcooperate Ġwith Ġthe Ġsystem Ġon Ġa Ġquestion Ġthey Ġare Ġinterested Ġin . ĠHowever , Ġobtaining Ġcorrect Ġanswers Ġfrom Ġusers Ġis Ġsignificantly Ġmore Ġdifficult Ġbecause Ġthe Ġsystem Ġdoes Ġnot Ġalways Ġhave Ġthe Ġchance Ġto Ġask Ġthe Ġquestion Ġand Ġthe Ġuser Ġdoes Ġnot Ġhave Ġto Ġknow Ġthe Ġcorrect Ġanswer Ġfor Ġit . </s> To Ġmake Ġthe Ġevaluations Ġcomparable Ġbetween Ġdifferent Ġsystems Ġwe Ġrecommend Ġusing Ġour Ġevaluation Ġscripts Ġincluded Ġwith Ġthe Ġdataset Ġwith Ġfollowing Ġpenal ization Ġweights Ġthat Ġreflect Ġour Ġintuition Ġfor Ġgaining Ġinformation Ġfrom Ġusers : </s> âĢĵ Ġincorrect Ġanswers Ġare Ġpenal ized Ġsignificantly , </s> âĢĵ Ġexplanations Ġare Ġquite Ġcheap ; Ġtherefore , Ġwe Ġwill Ġpenal ize Ġthem Ġjust Ġslightly , </s> âĢĵ Ġgaining Ġquestion âĢ Ļ s Ġanswer Ġfrom Ġusers Ġis Ġharder Ġthan Ġgaining Ġexplanations . </s> Answer ĠExt raction ĠAccuracy </s> It Ġis Ġquite Ġchallenging Ġto Ġfind Ġappropriate Ġentity Ġin Ġthe Ġknowledge Ġbase Ġeven Ġthough Ġthe Ġuser Ġprovided Ġthe Ġcorrect Ġanswer . ĠTherefore , Ġwe Ġpropose Ġanother Ġmetric Ġrelevant Ġto Ġour Ġdataset . ĠThis Ġmetric Ġis Ġthe Ġaccuracy Ġof Ġentity Ġextraction Ġwhich Ġmeasures Ġhow Ġmany Ġtimes Ġwas Ġextracted Ġa Ġcorrect Ġanswer Ġfrom Ġanswer Ġhints Ġprovided Ġby Ġthe Ġuser Ġin Ġdialog s Ġannot ated Ġas Ġcorrectly Ġanswered . </s> Future ĠWork </s> Our Ġfuture Ġwork Ġwill Ġbe Ġmainly Ġfocused Ġon Ġproviding Ġa Ġbaseline Ġsystem Ġfor Ġinteractive Ġlearning Ġwhich Ġwill Ġbe Ġevaluated Ġon Ġthe Ġdataset . ĠWe Ġare Ġalso Ġplanning Ġimprovements Ġfor Ġdialog Ġmanagement Ġthat Ġis Ġused Ġto Ġgain Ġexplanations Ġduring Ġthe Ġdata Ġcollection . ĠWe Ġbelieve Ġthat Ġwith Ġconversation Ġabout Ġspecific Ġaspects Ġof Ġthe Ġdiscussed Ġquestion Ġit Ġwill Ġbe Ġpossible Ġto Ġgain Ġeven Ġmore Ġinteresting Ġinformation Ġfrom Ġusers . ĠThe Ġother Ġarea Ġof Ġour Ġinterest Ġis Ġin Ġpossibilities Ġto Ġimprove Ġquestion Ġanswering Ġaccuracy Ġon Ġtest Ġquestions Ġof ĠSimple Ġquestion Ġdataset Ġwith Ġthe Ġextra Ġinformation Ġcontained Ġin Ġthe Ġcollected Ġdialog s . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġpresented Ġa Ġnovel Ġway Ġhow Ġto Ġevaluate Ġdifferent Ġinteractive Ġlearning Ġapproaches Ġfor Ġdialog Ġmodels . ĠThe Ġevaluation Ġcovers Ġtwo Ġchallenging Ġaspects Ġof Ġinteractive Ġlearning . ĠFirst , Ġit Ġscores Ġefficiency Ġof Ġusing Ġinformation Ġgained Ġfrom Ġusers Ġin Ġsimulated Ġquestion Ġanswering Ġdialog s . ĠSecond , Ġit Ġmeasures Ġaccuracy Ġon Ġanswer Ġhints Ġunderstanding . </s> For Ġpurposes Ġof Ġevaluation Ġwe Ġcollected Ġa Ġdataset Ġfrom Ġconvers ational Ġdialog s Ġwith Ġworkers Ġon Ġcrowds ourcing Ġplatform ĠCrowd Fl ower . ĠThose Ġdialog s Ġwere Ġannot ated Ġwith Ġexpert Ġannot ators Ġand Ġpublished Ġunder ĠCreative ĠCommons Ġ4 . 0 ĠBY - SA Ġlicense Ġon Ġl ind at . ĠWe Ġalso Ġprovide Ġevaluation Ġscripts Ġwith Ġthe Ġdataset Ġthat Ġshould Ġensure Ġcomparable Ġevaluation Ġof Ġdifferent Ġinteractive Ġlearning Ġapproaches . </s> Acknowled gments </s> This Ġwork Ġwas Ġfunded Ġby Ġthe ĠMinistry Ġof ĠEducation , ĠYouth Ġand ĠSports Ġof Ġthe ĠCzech ĠRepublic Ġunder Ġthe Ġgrant Ġagreement ĠL K 112 21 Ġand Ġcore Ġresearch Ġfunding , ĠSV V Ġproject Ġ260 Ġ224 , Ġand ĠGA UK Ġgrant Ġ11 705 16 Ġof ĠCharles ĠUniversity Ġin ĠPrague . ĠIt Ġused Ġlanguage Ġresources Ġstored Ġand Ġdistributed Ġby Ġthe ĠL IND AT / CL AR IN Ġproject Ġof Ġthe ĠMinistry Ġof ĠEducation , ĠYouth Ġand ĠSports Ġof Ġthe ĠCzech ĠRepublic Ġ( project ĠLM 201 50 71 ).