SVM, MLP, FastText, CNN, BERT, Google's DialogFlow, Rasa NLU
EXTRACTIVE
SVM, MLP, FastText, CNN, BERT, DialogFlow, Rasa NLU
EXTRACTIVE
Benchmark Evaluation ::: Classifier Models SVM: A linear support vector machine with bag-of-words sentence representations. MLP: A multi-layer perceptron with USE embeddings BIBREF4 as input. FastText: A shallow neural network that averages embeddings of n-grams BIBREF5. CNN: A convolutional neural network with non-static word embeddings initialized with GloVe BIBREF6. BERT: A neural network that is trained to predict elided words in text and then fine-tuned on our data BIBREF1. Platforms: Several platforms exist for the development of task-oriented agents. We consider Google's DialogFlow and Rasa NLU with spacy-sklearn.
Benchmark Evaluation ::: Classifier Models SVM: A linear support vector machine with bag-of-words sentence representations. MLP: A multi-layer perceptron with USE embeddings BIBREF4 as input. FastText: A shallow neural network that averages embeddings of n-grams BIBREF5. CNN: A convolutional neural network with non-static word embeddings initialized with GloVe BIBREF6. BERT: A neural network that is trained to predict elided words in text and then fine-tuned on our data BIBREF1. Platforms: Several platforms exist for the development of task-oriented agents. We consider Google's DialogFlow and Rasa NLU with spacy-sklearn.
1909.02027
Introduction </s> Task - oriented Ġdialog Ġsystems Ġhave Ġbecome Ġubiquitous , Ġproviding Ġa Ġmeans Ġfor Ġbillions Ġof Ġpeople Ġto Ġinteract Ġwith Ġcomputers Ġusing Ġnatural Ġlanguage . ĠMoreover , Ġthe Ġrecent Ġinflux Ġof Ġplatforms Ġand Ġtools Ġsuch Ġas ĠGoogle 's ĠDial og Flow Ġor ĠAmazon 's ĠLex Ġfor Ġbuilding Ġand Ġdeploying Ġsuch Ġsystems Ġmakes Ġthem Ġeven Ġmore Ġaccessible Ġto Ġvarious Ġindustries Ġand Ġdemographics Ġacross Ġthe Ġglobe . </s> Tools Ġfor Ġdeveloping Ġsuch Ġsystems Ġstart Ġby Ġguiding Ġdevelopers Ġto Ġcollect Ġtraining Ġdata Ġfor Ġintent Ġclassification : Ġthe Ġtask Ġof Ġidentifying Ġwhich Ġof Ġa Ġfixed Ġset Ġof Ġactions Ġthe Ġuser Ġwishes Ġto Ġtake Ġbased Ġon Ġtheir Ġquery . ĠRel atively Ġfew Ġpublic Ġdatasets Ġexist Ġfor Ġevaluating Ġperformance Ġon Ġthis Ġtask , Ġand Ġthose Ġthat Ġdo Ġexist Ġtypically Ġcover Ġonly Ġa Ġvery Ġsmall Ġnumber Ġof Ġint ents Ġ( e . g . ĠB IB REF 0 , Ġwhich Ġhas Ġ7 Ġint ents ). ĠFurthermore , Ġsuch Ġresources Ġdo Ġnot Ġfacilitate Ġanalysis Ġof Ġout - of - scope Ġqueries : Ġqueries Ġthat Ġusers Ġmay Ġreasonably Ġmake , Ġbut Ġfall Ġoutside Ġof Ġthe Ġscope Ġof Ġthe Ġsystem - supported Ġint ents . </s> Figure ĠFIG REF 1 Ġshows Ġexample Ġquery - response Ġexchanges Ġbetween Ġa Ġuser Ġand Ġa Ġtask - driven Ġdialog Ġsystem Ġfor Ġpersonal Ġfinance . ĠIn Ġthe Ġfirst Ġuser - system Ġexchange , Ġthe Ġsystem Ġcorrectly Ġidentifies Ġthe Ġuser 's Ġintent Ġas Ġan Ġin - scope Ġbalance Ġquery . ĠIn Ġthe Ġsecond Ġand Ġthird Ġexchanges , Ġthe Ġuser Ġqueries Ġwith Ġout - of - scope Ġinputs . ĠIn Ġthe Ġsecond Ġexchange , Ġthe Ġsystem Ġincorrectly Ġidentifies Ġthe Ġquery Ġas Ġin - scope Ġand Ġyields Ġan Ġunrelated Ġresponse . ĠIn Ġthe Ġthird Ġexchange , Ġthe Ġsystem Ġcorrectly Ġclass ifies Ġthe Ġuser 's Ġquery Ġas Ġout - of - scope , Ġand Ġyields Ġa Ġfall back Ġresponse . </s> Out - of - scope Ġqueries Ġare Ġinevitable Ġfor Ġa Ġtask - oriented Ġdialog Ġsystem , Ġas Ġmost Ġusers Ġwill Ġnot Ġbe Ġfully Ġcogn iz ant Ġof Ġthe Ġsystem 's Ġcapabilities , Ġwhich Ġare Ġlimited Ġby Ġthe Ġfixed Ġnumber Ġof Ġintent Ġclasses . ĠCorrect ly Ġidentifying Ġout - of - scope Ġcases Ġis Ġthus Ġcrucial Ġin Ġdeployed Ġsystems âĢĶ both Ġto Ġavoid Ġperforming Ġthe Ġwrong Ġaction Ġand Ġalso Ġto Ġidentify Ġpotential Ġfuture Ġdirections Ġfor Ġdevelopment . ĠHowever , Ġthis Ġproblem Ġhas Ġseen Ġlittle Ġattention Ġin Ġanalyses Ġand Ġevaluations Ġof Ġintent Ġclassification Ġsystems . </s> This Ġpaper Ġfills Ġthis Ġgap Ġby Ġanalyzing Ġintent Ġclassification Ġperformance Ġwith Ġa Ġfocus Ġon Ġout - of - scope Ġhandling . ĠTo Ġdo Ġso , Ġwe Ġconstructed Ġa Ġnew Ġdataset Ġwith Ġ23 , 700 Ġqueries Ġthat Ġare Ġshort Ġand Ġun struct ured , Ġin Ġthe Ġsame Ġstyle Ġmade Ġby Ġreal Ġusers Ġof Ġtask - oriented Ġsystems . ĠThe Ġqueries Ġcover Ġ150 Ġint ents , Ġplus Ġout - of - scope Ġqueries Ġthat Ġdo Ġnot Ġfall Ġwithin Ġany Ġof Ġthe Ġ150 Ġin - scope Ġint ents . </s> We Ġevaluate Ġa Ġrange Ġof Ġbenchmark Ġclass ifiers Ġand Ġout - of - scope Ġhandling Ġmethods Ġon Ġour Ġdataset . ĠB ERT ĠB IB REF 1 Ġyields Ġthe Ġbest Ġin - scope Ġaccuracy , Ġscoring Ġ96 % Ġor Ġabove Ġeven Ġwhen Ġwe Ġlimit Ġthe Ġtraining Ġdata Ġor Ġintroduce Ġclass Ġimbalance . ĠHowever , Ġall Ġmethods Ġstruggle Ġwith Ġidentifying Ġout - of - scope Ġqueries . ĠEven Ġwhen Ġa Ġlarge Ġnumber Ġof Ġout - of - scope Ġexamples Ġare Ġprovided Ġfor Ġtraining , Ġthere Ġis Ġa Ġmajor Ġperformance Ġgap , Ġwith Ġthe Ġbest Ġsystem Ġscoring Ġ66 % Ġout - of - scope Ġrecall . ĠOur Ġresults Ġshow Ġthat Ġwhile Ġcurrent Ġmodels Ġwork Ġon Ġknown Ġclasses , Ġthey Ġhave Ġdifficulty Ġon Ġout - of - scope Ġqueries , Ġparticularly Ġwhen Ġdata Ġis Ġnot Ġplentiful . ĠThis Ġdataset Ġwill Ġenable Ġfuture Ġwork Ġto Ġaddress Ġthis Ġkey Ġgap Ġin Ġthe Ġresearch Ġand Ġdevelopment Ġof Ġdialog Ġsystems . ĠAll Ġdata Ġintroduced Ġin Ġthis Ġpaper Ġcan Ġbe Ġfound Ġat Ġhttps :// github . com / cl inc / oos - eval . </s> Dat as et </s> We Ġintroduce Ġa Ġnew Ġcrowds ourced Ġdataset Ġof Ġ23 , 700 Ġqueries , Ġincluding Ġ22 , 500 Ġin - scope Ġqueries Ġcovering Ġ150 Ġint ents , Ġwhich Ġcan Ġbe Ġgrouped Ġinto Ġ10 Ġgeneral Ġdomains . ĠThe Ġdataset Ġalso Ġincludes Ġ1 , 200 Ġout - of - scope Ġqueries . ĠTable ĠT AB REF 2 Ġshows Ġexamples Ġof Ġthe Ġdata . </s> Dat as et Ġ: :: ĠIn - Scope ĠData ĠCollection </s> We Ġdefined Ġthe Ġint ents Ġwith Ġguidance Ġfrom Ġqueries Ġcollected Ġusing Ġa Ġsc oping Ġcrowds ourcing Ġtask , Ġwhich Ġprompted Ġcrowd Ġworkers Ġto Ġprovide Ġquestions Ġand Ġcommands Ġrelated Ġto Ġtopic Ġdomains Ġin Ġthe Ġmanner Ġthey Ġwould Ġinteract Ġwith Ġan Ġartificially Ġintelligent Ġassistant . ĠWe Ġmanually Ġgrouped Ġdata Ġgenerated Ġby Ġsc oping Ġtasks Ġinto Ġint ents . ĠTo Ġcollect Ġadditional Ġdata Ġfor Ġeach Ġintent , Ġwe Ġused Ġthe Ġre phrase Ġand Ġscenario Ġcrowds ourcing Ġtasks Ġproposed Ġby ĠB IB REF 2 . ĠFor Ġeach Ġintent , Ġthere Ġare Ġ100 Ġtraining Ġqueries , Ġwhich Ġis Ġrepresentative Ġof Ġwhat Ġa Ġteam Ġwith Ġa Ġlimited Ġbudget Ġcould Ġgather Ġwhile Ġdeveloping Ġa Ġtask - driven Ġdialog Ġsystem . ĠAlong Ġwith Ġthe Ġ100 Ġtraining Ġqueries , Ġthere Ġare Ġ20 Ġvalidation Ġand Ġ30 Ġtesting Ġqueries Ġper Ġintent . </s> Dat as et Ġ: :: ĠOut - of - Scope ĠData ĠCollection </s> Out - of - scope Ġqueries Ġwere Ġcollected Ġin Ġtwo Ġways . ĠFirst , Ġusing Ġworker Ġmistakes : Ġqueries Ġwritten Ġfor Ġone Ġof Ġthe Ġ150 Ġint ents Ġthat Ġdid Ġnot Ġactually Ġmatch Ġany Ġof Ġthe Ġint ents . ĠSecond , Ġusing Ġsc oping Ġand Ġscenario Ġtasks Ġwith Ġprompts Ġbased Ġon Ġtopic Ġareas Ġfound Ġon ĠQu ora , ĠWikipedia , Ġand Ġelsewhere . ĠTo Ġhelp Ġensure Ġthe Ġrichness Ġof Ġthis Ġadditional Ġout - of - scope Ġdata , Ġeach Ġof Ġthese Ġtask Ġprompts Ġcontributed Ġto Ġat Ġmost Ġfour Ġqueries . ĠSince Ġwe Ġuse Ġthe Ġsame Ġcrowds ourcing Ġmethod Ġfor Ġcollecting Ġout - of - scope Ġdata , Ġthese Ġqueries Ġare Ġsimilar Ġin Ġstyle Ġto Ġtheir Ġin - scope Ġcounterparts . </s> The Ġout - of - scope Ġdata Ġis Ġdifficult Ġto Ġcollect , Ġrequiring Ġexpert Ġknowledge Ġof Ġthe Ġin - scope Ġint ents Ġto Ġpainstaking ly Ġensure Ġthat Ġno Ġout - of - scope Ġquery Ġsample Ġis Ġmistakenly Ġlabeled Ġas Ġin - scope Ġ( and Ġvice Ġversa ). ĠIndeed , Ġroughly Ġonly Ġ69 % Ġof Ġqueries Ġcollected Ġwith Ġprompts Ġtargeting Ġout - of - scope Ġyielded Ġout - of - scope Ġqueries . ĠOf Ġthe Ġ1 , 200 Ġout - of - scope Ġqueries Ġcollected , Ġ100 Ġare Ġused Ġfor Ġvalidation Ġand Ġ100 Ġare Ġused Ġfor Ġtraining , Ġleaving Ġ1 , 000 Ġfor Ġtesting . </s> Dat as et Ġ: :: ĠData ĠPre processing Ġand ĠPart ition ing </s> For Ġall Ġqueries Ġcollected , Ġall Ġtokens Ġwere Ġdown - c ased , Ġand Ġall Ġend - of - sent ence Ġpunct uation Ġwas Ġremoved . ĠAdditionally , Ġall Ġduplicate Ġqueries Ġwere Ġremoved Ġand Ġreplaced . </s> In Ġan Ġeffort Ġto Ġreduce Ġbias Ġin Ġthe Ġin - scope Ġdata , Ġwe Ġplaced Ġall Ġqueries Ġfrom Ġa Ġgiven Ġcrowd Ġworker Ġin Ġa Ġsingle Ġsplit Ġ( train , Ġvalidation , Ġor Ġtest ). ĠThis Ġavoids Ġthe Ġpotential Ġissue Ġof Ġsimilar Ġqueries Ġfrom Ġa Ġcrowd Ġworker Ġending Ġup Ġin Ġboth Ġthe Ġtrain Ġand Ġtest Ġsets , Ġfor Ġinstance , Ġwhich Ġwould Ġmake Ġthe Ġtrain Ġand Ġtest Ġdistributions Ġunreal istically Ġsimilar . ĠWe Ġnote Ġthat Ġthis Ġis Ġa Ġrecommendation Ġfrom Ġconcurrent Ġwork Ġby ĠB IB REF 3 . ĠWe Ġalso Ġused Ġthis Ġprocedure Ġfor Ġthe Ġout - of - scope Ġset , Ġexcept Ġthat Ġwe Ġsplit Ġthe Ġdata Ġinto Ġtrain / valid ation / test Ġbased Ġon Ġtask Ġprompt Ġinstead Ġof Ġworker . </s> Dat as et Ġ: :: ĠDat as et ĠVari ants </s> In Ġaddition Ġto Ġthe Ġfull Ġdataset , Ġwe Ġconsider Ġthree Ġvariations . ĠFirst , ĠSmall , Ġin Ġwhich Ġthere Ġare Ġonly Ġ50 Ġtraining Ġqueries Ġper Ġeach Ġin - scope Ġintent , Ġrather Ġthan Ġ100 . ĠSecond , ĠIm balanced , Ġin Ġwhich Ġint ents Ġhave Ġeither Ġ25 , Ġ50 , Ġ75 , Ġor Ġ100 Ġtraining Ġqueries . ĠThird , ĠO OS +, Ġin Ġwhich Ġthere Ġare Ġ250 Ġout - of - scope Ġtraining Ġexamples , Ġrather Ġthan Ġ100 . ĠThese Ġare Ġintended Ġto Ġrepresent Ġproduction Ġscenarios Ġwhere Ġdata Ġmay Ġbe Ġin Ġlimited Ġor Ġuneven Ġsupply . </s> Bench mark ĠEvaluation </s> To Ġquantify Ġthe Ġchallenges Ġthat Ġour Ġnew Ġdataset Ġpresents , Ġwe Ġevaluated Ġthe Ġperformance Ġof Ġa Ġrange Ġof Ġclass ifier Ġmodels Ġand Ġout - of - scope Ġprediction Ġschemes . </s> Bench mark ĠEvaluation Ġ: :: ĠClass ifier ĠModels </s> S VM : ĠA Ġlinear Ġsupport Ġvector Ġmachine Ġwith Ġbag - of - words Ġsentence Ġrepresentations . </s> ML P : ĠA Ġmulti - layer Ġpercept ron Ġwith ĠUSE Ġembed d ings ĠB IB REF 4 Ġas Ġinput . </s> Fast Text : ĠA Ġshallow Ġneural Ġnetwork Ġthat Ġaverages Ġembed d ings Ġof Ġn - gram s ĠB IB REF 5 . </s> CNN : ĠA Ġconv olution al Ġneural Ġnetwork Ġwith Ġnon - static Ġword Ġembed d ings Ġinitialized Ġwith ĠGl o Ve ĠB IB REF 6 . </s> BER T : ĠA Ġneural Ġnetwork Ġthat Ġis Ġtrained Ġto Ġpredict Ġel ided Ġwords Ġin Ġtext Ġand Ġthen Ġfine - tun ed Ġon Ġour Ġdata ĠB IB REF 1 . </s> Platform s : ĠSeveral Ġplatforms Ġexist Ġfor Ġthe Ġdevelopment Ġof Ġtask - oriented Ġagents . ĠWe Ġconsider ĠGoogle 's ĠDial og Flow Ġand ĠR asa ĠNL U Ġwith Ġsp acy - sk learn . </s> Bench mark ĠEvaluation Ġ: :: ĠOut - of - Scope ĠPrediction </s> We Ġuse Ġthree Ġbaseline Ġapproaches Ġfor Ġthe Ġtask Ġof Ġpredicting Ġwhether Ġa Ġquery Ġis Ġout - of - scope : Ġ( 1 ) Ġo os - train , Ġwhere Ġwe Ġtrain Ġan Ġadditional Ġ( i . e . Ġ151 st ) Ġintent Ġon Ġout - of - scope Ġtraining Ġdata ; Ġ( 2 ) Ġo os - th reshold , Ġwhere Ġwe Ġuse Ġa Ġthreshold Ġon Ġthe Ġclass ifier 's Ġprobability Ġestimate ; Ġand Ġ( 3 ) Ġo os - binary , Ġa Ġtwo - stage Ġprocess Ġwhere Ġwe Ġfirst Ġclassify Ġa Ġquery Ġas Ġin - Ġor Ġout - of - scope , Ġthen Ġclassify Ġit Ġinto Ġone Ġof Ġthe Ġ150 Ġint ents Ġif Ġclassified Ġas Ġin - scope . </s> To Ġreduce Ġthe Ġseverity Ġof Ġthe Ġclass Ġimbalance Ġbetween Ġin - scope Ġversus Ġout - of - scope Ġquery Ġsamples Ġ( i . e ., Ġ15 , 000 Ġversus Ġ250 Ġqueries Ġfor ĠO OS + ), Ġwe Ġinvestigate Ġtwo Ġstrategies Ġwhen Ġusing Ġo os - binary : Ġone Ġwhere Ġwe Ġunders ample Ġthe Ġin - scope Ġdata Ġand Ġtrain Ġusing Ġ1 , 000 Ġin - scope Ġqueries Ġsampled Ġevenly Ġacross Ġall Ġint ents Ġ( vers us Ġ250 Ġout - of - scope ), Ġand Ġanother Ġwhere Ġwe Ġaugment Ġthe Ġ250 ĠO OS + Ġout - of - scope Ġtraining Ġqueries Ġwith Ġ14 , 750 Ġsentences Ġsampled Ġfrom ĠWikipedia . </s> From Ġa Ġdevelopment Ġpoint Ġof Ġview , Ġthe Ġo os - train Ġand Ġo os - binary Ġmethods Ġboth Ġrequire Ġcareful Ġcur ation Ġof Ġan Ġout - of - scope Ġtraining Ġset , Ġand Ġthis Ġset Ġcan Ġbe Ġtailored Ġto Ġindividual Ġsystems . ĠThe Ġo os - th reshold Ġmethod Ġis Ġa Ġmore Ġgeneral Ġdecision Ġrule Ġthat Ġcan Ġbe Ġapplied Ġto Ġany Ġmodel Ġthat Ġproduces Ġa Ġprobability . ĠIn Ġour Ġevaluation , Ġthe Ġout - of - scope Ġthreshold Ġwas Ġchosen Ġto Ġbe Ġthe Ġvalue Ġwhich Ġyielded Ġthe Ġhighest Ġvalidation Ġscore Ġacross Ġall Ġint ents , Ġtreating Ġout - of - scope Ġas Ġits Ġown Ġintent . </s> Bench mark ĠEvaluation Ġ: :: ĠMet rics </s> We Ġconsider Ġtwo Ġperformance Ġmetrics Ġfor Ġall Ġscenarios : Ġ( 1 ) Ġaccuracy Ġover Ġthe Ġ150 Ġint ents , Ġand Ġ( 2 ) Ġrecall Ġon Ġout - of - scope Ġqueries . ĠWe Ġuse Ġrecall Ġto Ġevaluate Ġout - of - scope Ġsince Ġwe Ġare Ġmore Ġinterested Ġin Ġcases Ġwhere Ġsuch Ġqueries Ġare Ġpredicted Ġas Ġin - scope , Ġas Ġthis Ġwould Ġmean Ġa Ġsystem Ġgives Ġthe Ġuser Ġa Ġresponse Ġthat Ġis Ġcompletely Ġwrong . ĠPrecision Ġerrors Ġare Ġless Ġproblematic Ġas Ġthe Ġfall back Ġresponse Ġwill Ġprompt Ġthe Ġuser Ġto Ġtry Ġagain , Ġor Ġinform Ġthe Ġuser Ġof Ġthe Ġsystem 's Ġscope Ġof Ġsupported Ġdomains . </s> Results Ġ: :: ĠResults Ġwith Ġo os - train </s> Table ĠT AB REF 14 Ġpresents Ġresults Ġfor Ġall Ġmodels Ġacross Ġthe Ġfour Ġvariations Ġof Ġthe Ġdataset . ĠFirst , ĠB ERT Ġis Ġconsistently Ġthe Ġbest Ġapproach Ġfor Ġin - scope , Ġfollowed Ġby ĠML P . ĠSecond , Ġout - of - scope Ġquery Ġperformance Ġis Ġmuch Ġlower Ġthan Ġin - scope Ġacross Ġall Ġmethods . ĠTraining Ġon Ġless Ġdata Ġ( Small Ġand ĠIm balanced ) Ġyields Ġmodels Ġthat Ġperform Ġslightly Ġworse Ġon Ġin - scope Ġqueries . ĠThe Ġtrend Ġis Ġmostly Ġthe Ġopposite Ġwhen Ġevaluating Ġout - of - scope , Ġwhere Ġrecall Ġincreases Ġunder Ġthe ĠSmall Ġand ĠIm balanced Ġtraining Ġconditions . ĠUnder Ġthese Ġtwo Ġconditions , Ġthe Ġsize Ġof Ġthe Ġin - scope Ġtraining Ġset Ġwas Ġdecreased , Ġwhile Ġthe Ġnumber Ġof Ġout - of - scope Ġtraining Ġqueries Ġremained Ġconstant . ĠThis Ġindicates Ġthat Ġout - of - scope Ġperformance Ġcan Ġbe Ġincreased Ġby Ġincreasing Ġthe Ġrelative Ġnumber Ġof Ġout - of - scope Ġtraining Ġqueries . ĠWe Ġdo Ġjust Ġthat Ġin Ġthe ĠO OS + Ġsetting âĢĶ where Ġthe Ġmodels Ġwere Ġtrained Ġon Ġthe Ġfull Ġtraining Ġset Ġas Ġwell Ġas Ġ150 Ġadditional Ġout - of - scope Ġqueries âĢĶ and Ġsee Ġthat Ġperformance Ġon Ġout - of - scope Ġincreases Ġsubstantially , Ġyet Ġstill Ġremains Ġlow Ġrelative Ġto Ġin - scope Ġaccuracy . </s> Results Ġ: :: ĠResults Ġwith Ġo os - th reshold </s> In - scope Ġaccuracy Ġusing Ġthe Ġo os - th reshold Ġapproach Ġis Ġlargely Ġcomparable Ġto Ġo os - train . ĠOut - of - scope Ġrecall Ġtends Ġto Ġbe Ġmuch Ġhigher Ġon ĠFull , Ġbut Ġseveral Ġmodels Ġsuffer Ġgreatly Ġon Ġthe Ġlimited Ġdatasets . ĠB ERT Ġand ĠML P Ġare Ġthe Ġtop Ġo os - th reshold Ġperformers , Ġand Ġfor Ġseveral Ġmodels Ġthe Ġthreshold Ġapproach Ġprovided Ġerratic Ġresults , Ġparticularly ĠFast Text Ġand ĠR asa . </s> Results Ġ: :: ĠResults Ġwith Ġo os - binary </s> Table ĠT AB REF 19 Ġcompares Ġclass ifier Ġperformance Ġusing Ġthe Ġo os - binary Ġscheme . ĠIn - scope Ġaccuracy Ġsuffers Ġfor Ġall Ġmodels Ġusing Ġthe Ġunders am pling Ġscheme Ġwhen Ġcompared Ġto Ġtraining Ġon Ġthe Ġfull Ġdataset Ġusing Ġthe Ġo os - train Ġand Ġo os - th reshold Ġapproaches Ġshown Ġin ĠTable ĠT AB REF 14 . ĠHowever , Ġout - of - scope Ġrecall Ġimproves Ġcompared Ġto Ġo os - train Ġon ĠFull Ġbut Ġnot ĠO OS +. ĠAug ment ing Ġthe Ġout - of - scope Ġtraining Ġset Ġappears Ġto Ġhelp Ġimprove Ġboth Ġin - scope Ġand Ġout - of - scope Ġperformance Ġcompared Ġto Ġunders am pling , Ġbut Ġout - of - scope Ġperformance Ġremains Ġweak . </s> Prior ĠWork </s> In Ġmost Ġother Ġanalyses Ġand Ġdatasets , Ġthe Ġidea Ġof Ġout - of - scope Ġdata Ġis Ġnot Ġconsidered , Ġand Ġinstead Ġthe Ġoutput Ġclasses Ġare Ġintended Ġto Ġcover Ġall Ġpossible Ġqueries Ġ( e . g ., ĠT REC ĠB IB REF 7 ). ĠRecent Ġwork Ġby ĠB IB REF 8 Ġconsiders Ġa Ġsimilar Ġproblem Ġthey Ġcall Ġout - of - dist ribution Ġdetection . ĠThey Ġuse Ġother Ġdatasets Ġor Ġclasses Ġexcluded Ġduring Ġtraining Ġto Ġform Ġthe Ġout - of - dist ribution Ġsamples . ĠThis Ġmeans Ġthat Ġthe Ġout - of - scope Ġsamples Ġare Ġfrom Ġa Ġsmall Ġset Ġof Ġcoherent Ġclasses Ġthat Ġdiffer Ġsubstantially Ġfrom Ġthe Ġin - dist ribution Ġsamples . ĠSimilar Ġexperiments Ġwere Ġconducted Ġfor Ġevaluating Ġunknown Ġintent Ġdiscovery Ġmodels Ġin ĠB IB REF 9 . ĠIn Ġcontrast , Ġour Ġout - of - scope Ġqueries Ġcover Ġa Ġbroad Ġrange Ġof Ġphenomena Ġand Ġare Ġsimilar Ġin Ġstyle Ġand Ġoften Ġsimilar Ġin Ġtopic Ġto Ġin - scope Ġqueries , Ġrepresenting Ġthings Ġa Ġuser Ġmight Ġsay Ġgiven Ġpartial Ġknowledge Ġof Ġthe Ġcapabilities Ġof Ġa Ġsystem . </s> Table ĠT AB REF 20 Ġcompares Ġour Ġdataset Ġwith Ġother Ġshort - query Ġintent Ġclassification Ġdatasets . ĠThe ĠSn ips ĠB IB REF 0 Ġdataset Ġand Ġthe Ġdataset Ġpresented Ġin ĠB IB REF 10 Ġare Ġthe Ġmost Ġsimilar Ġto Ġthe Ġin - scope Ġpart Ġof Ġour Ġwork , Ġwith Ġthe Ġsame Ġtype Ġof Ġconvers ational Ġagent Ġrequests . ĠLike Ġour Ġwork , Ġboth Ġof Ġthese Ġdatasets Ġwere Ġboot stra pped Ġusing Ġcrowds ourcing . ĠHowever , Ġthe ĠSn ips Ġdataset Ġhas Ġonly Ġa Ġsmall Ġnumber Ġof Ġint ents Ġand Ġan Ġenormous Ġnumber Ġof Ġexamples Ġof Ġeach . ĠSn ips Ġdoes Ġpresent Ġa Ġlow - data Ġvariation , Ġwith Ġ70 Ġtraining Ġqueries Ġper Ġintent , Ġin Ġwhich Ġperformance Ġdrops Ġslightly . ĠThe Ġdataset Ġpresented Ġin ĠB IB REF 10 Ġhas Ġa Ġlarge Ġnumber Ġof Ġintent Ġclasses , Ġyet Ġalso Ġcontains Ġa Ġwide Ġrange Ġof Ġsamples Ġper Ġintent Ġclass Ġ( ranging Ġfrom Ġ24 Ġto Ġ5 , 98 1 Ġqueries Ġper Ġintent , Ġand Ġso Ġis Ġnot Ġconstrained Ġin Ġall Ġcases ). </s> BI B REF 11 Ġcreated Ġdatasets Ġwith Ġconstrained Ġtraining Ġdata , Ġbut Ġwith Ġvery Ġfew Ġint ents , Ġpresenting Ġa Ġvery Ġdifferent Ġtype Ġof Ġchallenge . ĠWe Ġalso Ġinclude Ġthe ĠT REC Ġquery Ġclassification Ġdatasets ĠB IB REF 7 , Ġwhich Ġhave Ġa Ġlarge Ġset Ġof Ġlabels , Ġbut Ġthey Ġdescribe Ġthe Ġdesired Ġresponse Ġtype Ġ( e . g ., Ġdistance , Ġcity , Ġabbre viation ) Ġrather Ġthan Ġthe Ġaction Ġint ents Ġwe Ġconsider . ĠMoreover , ĠT REC Ġcontains Ġonly Ġquestions Ġand Ġno Ġcommands . ĠCru cially , Ġnone Ġof Ġthe Ġother Ġdatasets Ġsummarized Ġin ĠTable ĠT AB REF 20 Ġoffer Ġa Ġfeasible Ġway Ġto Ġevaluate Ġout - of - scope Ġperformance . </s> The ĠDial og ĠState ĠTracking ĠChallenge Ġ( D ST C ) Ġdatasets Ġare Ġanother Ġrelated Ġresource . ĠSpecifically , ĠD ST C Ġ1 ĠB IB REF 12 , ĠD ST C Ġ2 ĠB IB REF 13 , Ġand ĠD ST C Ġ3 ĠB IB REF 14 Ġcontain ĠâĢ ľ chat bot Ġstyle " Ġqueries , Ġbut Ġthe Ġdatasets Ġare Ġfocused Ġon Ġstate Ġtracking . ĠMoreover , Ġmost Ġif Ġnot Ġall Ġqueries Ġin Ġthese Ġdatasets Ġare Ġin - scope . ĠIn Ġcontrast , Ġthe Ġfocus Ġof Ġour Ġanalysis Ġis Ġon Ġboth Ġin - Ġand Ġout - of - scope Ġqueries Ġthat Ġchallenge Ġa Ġvirtual Ġassistant Ġto Ġdetermine Ġwhether Ġit Ġcan Ġprovide Ġan Ġacceptable Ġresponse . </s> Conclusion </s> This Ġpaper Ġanalyzed Ġintent Ġclassification Ġand Ġout - of - scope Ġprediction Ġmethods Ġwith Ġa Ġnew Ġdataset Ġconsisting Ġof Ġcarefully Ġcollected Ġout - of - scope Ġdata . ĠOur Ġfindings Ġindicate Ġthat Ġcertain Ġmodels Ġlike ĠB ERT Ġperform Ġbetter Ġon Ġin - scope Ġclassification , Ġbut Ġall Ġmethods Ġinvestigated Ġstruggle Ġwith Ġidentifying Ġout - of - scope Ġqueries . ĠModels Ġthat Ġincorporate Ġmore Ġout - of - scope Ġtraining Ġdata Ġtend Ġto Ġimprove Ġon Ġout - of - scope Ġperformance , Ġyet Ġsuch Ġdata Ġis Ġexpensive Ġand Ġdifficult Ġto Ġgenerate . ĠWe Ġbelieve Ġour Ġanalysis Ġand Ġdataset Ġwill Ġlead Ġto Ġdeveloping Ġbetter , Ġmore Ġrobust Ġdialog Ġsystems . </s> All Ġdatasets Ġintroduced Ġin Ġthis Ġpaper Ġcan Ġbe Ġfound Ġat Ġhttps :// github . com / cl inc / oos - eval .
Introduction Task-oriented dialog systems have become ubiquitous, providing a means for billions of people to interact with computers using natural language. Moreover, the recent influx of platforms and tools such as Google's DialogFlow or Amazon's Lex for building and deploying such systems makes them even more accessible to various industries and demographics across the globe. Tools for developing such systems start by guiding developers to collect training data for intent classification: the task of identifying which of a fixed set of actions the user wishes to take based on their query. Relatively few public datasets exist for evaluating performance on this task, and those that do exist typically cover only a very small number of intents (e.g. BIBREF0, which has 7 intents). Furthermore, such resources do not facilitate analysis of out-of-scope queries: queries that users may reasonably make, but fall outside of the scope of the system-supported intents. Figure FIGREF1 shows example query-response exchanges between a user and a task-driven dialog system for personal finance. In the first user-system exchange, the system correctly identifies the user's intent as an in-scope balance query. In the second and third exchanges, the user queries with out-of-scope inputs. In the second exchange, the system incorrectly identifies the query as in-scope and yields an unrelated response. In the third exchange, the system correctly classifies the user's query as out-of-scope, and yields a fallback response. Out-of-scope queries are inevitable for a task-oriented dialog system, as most users will not be fully cognizant of the system's capabilities, which are limited by the fixed number of intent classes. Correctly identifying out-of-scope cases is thus crucial in deployed systems—both to avoid performing the wrong action and also to identify potential future directions for development. However, this problem has seen little attention in analyses and evaluations of intent classification systems. This paper fills this gap by analyzing intent classification performance with a focus on out-of-scope handling. To do so, we constructed a new dataset with 23,700 queries that are short and unstructured, in the same style made by real users of task-oriented systems. The queries cover 150 intents, plus out-of-scope queries that do not fall within any of the 150 in-scope intents. We evaluate a range of benchmark classifiers and out-of-scope handling methods on our dataset. BERT BIBREF1 yields the best in-scope accuracy, scoring 96% or above even when we limit the training data or introduce class imbalance. However, all methods struggle with identifying out-of-scope queries. Even when a large number of out-of-scope examples are provided for training, there is a major performance gap, with the best system scoring 66% out-of-scope recall. Our results show that while current models work on known classes, they have difficulty on out-of-scope queries, particularly when data is not plentiful. This dataset will enable future work to address this key gap in the research and development of dialog systems. All data introduced in this paper can be found at https://github.com/clinc/oos-eval. Dataset We introduce a new crowdsourced dataset of 23,700 queries, including 22,500 in-scope queries covering 150 intents, which can be grouped into 10 general domains. The dataset also includes 1,200 out-of-scope queries. Table TABREF2 shows examples of the data. Dataset ::: In-Scope Data Collection We defined the intents with guidance from queries collected using a scoping crowdsourcing task, which prompted crowd workers to provide questions and commands related to topic domains in the manner they would interact with an artificially intelligent assistant. We manually grouped data generated by scoping tasks into intents. To collect additional data for each intent, we used the rephrase and scenario crowdsourcing tasks proposed by BIBREF2. For each intent, there are 100 training queries, which is representative of what a team with a limited budget could gather while developing a task-driven dialog system. Along with the 100 training queries, there are 20 validation and 30 testing queries per intent. Dataset ::: Out-of-Scope Data Collection Out-of-scope queries were collected in two ways. First, using worker mistakes: queries written for one of the 150 intents that did not actually match any of the intents. Second, using scoping and scenario tasks with prompts based on topic areas found on Quora, Wikipedia, and elsewhere. To help ensure the richness of this additional out-of-scope data, each of these task prompts contributed to at most four queries. Since we use the same crowdsourcing method for collecting out-of-scope data, these queries are similar in style to their in-scope counterparts. The out-of-scope data is difficult to collect, requiring expert knowledge of the in-scope intents to painstakingly ensure that no out-of-scope query sample is mistakenly labeled as in-scope (and vice versa). Indeed, roughly only 69% of queries collected with prompts targeting out-of-scope yielded out-of-scope queries. Of the 1,200 out-of-scope queries collected, 100 are used for validation and 100 are used for training, leaving 1,000 for testing. Dataset ::: Data Preprocessing and Partitioning For all queries collected, all tokens were down-cased, and all end-of-sentence punctuation was removed. Additionally, all duplicate queries were removed and replaced. In an effort to reduce bias in the in-scope data, we placed all queries from a given crowd worker in a single split (train, validation, or test). This avoids the potential issue of similar queries from a crowd worker ending up in both the train and test sets, for instance, which would make the train and test distributions unrealistically similar. We note that this is a recommendation from concurrent work by BIBREF3. We also used this procedure for the out-of-scope set, except that we split the data into train/validation/test based on task prompt instead of worker. Dataset ::: Dataset Variants In addition to the full dataset, we consider three variations. First, Small, in which there are only 50 training queries per each in-scope intent, rather than 100. Second, Imbalanced, in which intents have either 25, 50, 75, or 100 training queries. Third, OOS+, in which there are 250 out-of-scope training examples, rather than 100. These are intended to represent production scenarios where data may be in limited or uneven supply. Benchmark Evaluation To quantify the challenges that our new dataset presents, we evaluated the performance of a range of classifier models and out-of-scope prediction schemes. Benchmark Evaluation ::: Classifier Models SVM: A linear support vector machine with bag-of-words sentence representations. MLP: A multi-layer perceptron with USE embeddings BIBREF4 as input. FastText: A shallow neural network that averages embeddings of n-grams BIBREF5. CNN: A convolutional neural network with non-static word embeddings initialized with GloVe BIBREF6. BERT: A neural network that is trained to predict elided words in text and then fine-tuned on our data BIBREF1. Platforms: Several platforms exist for the development of task-oriented agents. We consider Google's DialogFlow and Rasa NLU with spacy-sklearn. Benchmark Evaluation ::: Out-of-Scope Prediction We use three baseline approaches for the task of predicting whether a query is out-of-scope: (1) oos-train, where we train an additional (i.e. 151st) intent on out-of-scope training data; (2) oos-threshold, where we use a threshold on the classifier's probability estimate; and (3) oos-binary, a two-stage process where we first classify a query as in- or out-of-scope, then classify it into one of the 150 intents if classified as in-scope. To reduce the severity of the class imbalance between in-scope versus out-of-scope query samples (i.e., 15,000 versus 250 queries for OOS+), we investigate two strategies when using oos-binary: one where we undersample the in-scope data and train using 1,000 in-scope queries sampled evenly across all intents (versus 250 out-of-scope), and another where we augment the 250 OOS+ out-of-scope training queries with 14,750 sentences sampled from Wikipedia. From a development point of view, the oos-train and oos-binary methods both require careful curation of an out-of-scope training set, and this set can be tailored to individual systems. The oos-threshold method is a more general decision rule that can be applied to any model that produces a probability. In our evaluation, the out-of-scope threshold was chosen to be the value which yielded the highest validation score across all intents, treating out-of-scope as its own intent. Benchmark Evaluation ::: Metrics We consider two performance metrics for all scenarios: (1) accuracy over the 150 intents, and (2) recall on out-of-scope queries. We use recall to evaluate out-of-scope since we are more interested in cases where such queries are predicted as in-scope, as this would mean a system gives the user a response that is completely wrong. Precision errors are less problematic as the fallback response will prompt the user to try again, or inform the user of the system's scope of supported domains. Results ::: Results with oos-train Table TABREF14 presents results for all models across the four variations of the dataset. First, BERT is consistently the best approach for in-scope, followed by MLP. Second, out-of-scope query performance is much lower than in-scope across all methods. Training on less data (Small and Imbalanced) yields models that perform slightly worse on in-scope queries. The trend is mostly the opposite when evaluating out-of-scope, where recall increases under the Small and Imbalanced training conditions. Under these two conditions, the size of the in-scope training set was decreased, while the number of out-of-scope training queries remained constant. This indicates that out-of-scope performance can be increased by increasing the relative number of out-of-scope training queries. We do just that in the OOS+ setting—where the models were trained on the full training set as well as 150 additional out-of-scope queries—and see that performance on out-of-scope increases substantially, yet still remains low relative to in-scope accuracy. Results ::: Results with oos-threshold In-scope accuracy using the oos-threshold approach is largely comparable to oos-train. Out-of-scope recall tends to be much higher on Full, but several models suffer greatly on the limited datasets. BERT and MLP are the top oos-threshold performers, and for several models the threshold approach provided erratic results, particularly FastText and Rasa. Results ::: Results with oos-binary Table TABREF19 compares classifier performance using the oos-binary scheme. In-scope accuracy suffers for all models using the undersampling scheme when compared to training on the full dataset using the oos-train and oos-threshold approaches shown in Table TABREF14. However, out-of-scope recall improves compared to oos-train on Full but not OOS+. Augmenting the out-of-scope training set appears to help improve both in-scope and out-of-scope performance compared to undersampling, but out-of-scope performance remains weak. Prior Work In most other analyses and datasets, the idea of out-of-scope data is not considered, and instead the output classes are intended to cover all possible queries (e.g., TREC BIBREF7). Recent work by BIBREF8 considers a similar problem they call out-of-distribution detection. They use other datasets or classes excluded during training to form the out-of-distribution samples. This means that the out-of-scope samples are from a small set of coherent classes that differ substantially from the in-distribution samples. Similar experiments were conducted for evaluating unknown intent discovery models in BIBREF9. In contrast, our out-of-scope queries cover a broad range of phenomena and are similar in style and often similar in topic to in-scope queries, representing things a user might say given partial knowledge of the capabilities of a system. Table TABREF20 compares our dataset with other short-query intent classification datasets. The Snips BIBREF0 dataset and the dataset presented in BIBREF10 are the most similar to the in-scope part of our work, with the same type of conversational agent requests. Like our work, both of these datasets were bootstrapped using crowdsourcing. However, the Snips dataset has only a small number of intents and an enormous number of examples of each. Snips does present a low-data variation, with 70 training queries per intent, in which performance drops slightly. The dataset presented in BIBREF10 has a large number of intent classes, yet also contains a wide range of samples per intent class (ranging from 24 to 5,981 queries per intent, and so is not constrained in all cases). BIBREF11 created datasets with constrained training data, but with very few intents, presenting a very different type of challenge. We also include the TREC query classification datasets BIBREF7, which have a large set of labels, but they describe the desired response type (e.g., distance, city, abbreviation) rather than the action intents we consider. Moreover, TREC contains only questions and no commands. Crucially, none of the other datasets summarized in Table TABREF20 offer a feasible way to evaluate out-of-scope performance. The Dialog State Tracking Challenge (DSTC) datasets are another related resource. Specifically, DSTC 1 BIBREF12, DSTC 2 BIBREF13, and DSTC 3 BIBREF14 contain “chatbot style" queries, but the datasets are focused on state tracking. Moreover, most if not all queries in these datasets are in-scope. In contrast, the focus of our analysis is on both in- and out-of-scope queries that challenge a virtual assistant to determine whether it can provide an acceptable response. Conclusion This paper analyzed intent classification and out-of-scope prediction methods with a new dataset consisting of carefully collected out-of-scope data. Our findings indicate that certain models like BERT perform better on in-scope classification, but all methods investigated struggle with identifying out-of-scope queries. Models that incorporate more out-of-scope training data tend to improve on out-of-scope performance, yet such data is expensive and difficult to generate. We believe our analysis and dataset will lead to developing better, more robust dialog systems. All datasets introduced in this paper can be found at https://github.com/clinc/oos-eval.
Which classifiers are evaluated?
149da739b1c19a157880d9d4827f0b692006aa2c
Which Ġclass ifiers Ġare Ġevaluated ?
S VM , ĠML P , ĠFast Text , ĠCNN , ĠB ERT , ĠGoogle 's ĠDial og Flow , ĠR asa ĠNL U
<s> Which Ġclass ifiers Ġare Ġevaluated ? </s> Introduction </s> Task - oriented Ġdialog Ġsystems Ġhave Ġbecome Ġubiquitous , Ġproviding Ġa Ġmeans Ġfor Ġbillions Ġof Ġpeople Ġto Ġinteract Ġwith Ġcomputers Ġusing Ġnatural Ġlanguage . ĠMoreover , Ġthe Ġrecent Ġinflux Ġof Ġplatforms Ġand Ġtools Ġsuch Ġas ĠGoogle 's ĠDial og Flow Ġor ĠAmazon 's ĠLex Ġfor Ġbuilding Ġand Ġdeploying Ġsuch Ġsystems Ġmakes Ġthem Ġeven Ġmore Ġaccessible Ġto Ġvarious Ġindustries Ġand Ġdemographics Ġacross Ġthe Ġglobe . </s> Tools Ġfor Ġdeveloping Ġsuch Ġsystems Ġstart Ġby Ġguiding Ġdevelopers Ġto Ġcollect Ġtraining Ġdata Ġfor Ġintent Ġclassification : Ġthe Ġtask Ġof Ġidentifying Ġwhich Ġof Ġa Ġfixed Ġset Ġof Ġactions Ġthe Ġuser Ġwishes Ġto Ġtake Ġbased Ġon Ġtheir Ġquery . ĠRel atively Ġfew Ġpublic Ġdatasets Ġexist Ġfor Ġevaluating Ġperformance Ġon Ġthis Ġtask , Ġand Ġthose Ġthat Ġdo Ġexist Ġtypically Ġcover Ġonly Ġa Ġvery Ġsmall Ġnumber Ġof Ġint ents Ġ( e . g . ĠB IB REF 0 , Ġwhich Ġhas Ġ7 Ġint ents ). ĠFurthermore , Ġsuch Ġresources Ġdo Ġnot Ġfacilitate Ġanalysis Ġof Ġout - of - scope Ġqueries : Ġqueries Ġthat Ġusers Ġmay Ġreasonably Ġmake , Ġbut Ġfall Ġoutside Ġof Ġthe Ġscope Ġof Ġthe Ġsystem - supported Ġint ents . </s> Figure ĠFIG REF 1 Ġshows Ġexample Ġquery - response Ġexchanges Ġbetween Ġa Ġuser Ġand Ġa Ġtask - driven Ġdialog Ġsystem Ġfor Ġpersonal Ġfinance . ĠIn Ġthe Ġfirst Ġuser - system Ġexchange , Ġthe Ġsystem Ġcorrectly Ġidentifies Ġthe Ġuser 's Ġintent Ġas Ġan Ġin - scope Ġbalance Ġquery . ĠIn Ġthe Ġsecond Ġand Ġthird Ġexchanges , Ġthe Ġuser Ġqueries Ġwith Ġout - of - scope Ġinputs . ĠIn Ġthe Ġsecond Ġexchange , Ġthe Ġsystem Ġincorrectly Ġidentifies Ġthe Ġquery Ġas Ġin - scope Ġand Ġyields Ġan Ġunrelated Ġresponse . ĠIn Ġthe Ġthird Ġexchange , Ġthe Ġsystem Ġcorrectly Ġclass ifies Ġthe Ġuser 's Ġquery Ġas Ġout - of - scope , Ġand Ġyields Ġa Ġfall back Ġresponse . </s> Out - of - scope Ġqueries Ġare Ġinevitable Ġfor Ġa Ġtask - oriented Ġdialog Ġsystem , Ġas Ġmost Ġusers Ġwill Ġnot Ġbe Ġfully Ġcogn iz ant Ġof Ġthe Ġsystem 's Ġcapabilities , Ġwhich Ġare Ġlimited Ġby Ġthe Ġfixed Ġnumber Ġof Ġintent Ġclasses . ĠCorrect ly Ġidentifying Ġout - of - scope Ġcases Ġis Ġthus Ġcrucial Ġin Ġdeployed Ġsystems âĢĶ both Ġto Ġavoid Ġperforming Ġthe Ġwrong Ġaction Ġand Ġalso Ġto Ġidentify Ġpotential Ġfuture Ġdirections Ġfor Ġdevelopment . ĠHowever , Ġthis Ġproblem Ġhas Ġseen Ġlittle Ġattention Ġin Ġanalyses Ġand Ġevaluations Ġof Ġintent Ġclassification Ġsystems . </s> This Ġpaper Ġfills Ġthis Ġgap Ġby Ġanalyzing Ġintent Ġclassification Ġperformance Ġwith Ġa Ġfocus Ġon Ġout - of - scope Ġhandling . ĠTo Ġdo Ġso , Ġwe Ġconstructed Ġa Ġnew Ġdataset Ġwith Ġ23 , 700 Ġqueries Ġthat Ġare Ġshort Ġand Ġun struct ured , Ġin Ġthe Ġsame Ġstyle Ġmade Ġby Ġreal Ġusers Ġof Ġtask - oriented Ġsystems . ĠThe Ġqueries Ġcover Ġ150 Ġint ents , Ġplus Ġout - of - scope Ġqueries Ġthat Ġdo Ġnot Ġfall Ġwithin Ġany Ġof Ġthe Ġ150 Ġin - scope Ġint ents . </s> We Ġevaluate Ġa Ġrange Ġof Ġbenchmark Ġclass ifiers Ġand Ġout - of - scope Ġhandling Ġmethods Ġon Ġour Ġdataset . ĠB ERT ĠB IB REF 1 Ġyields Ġthe Ġbest Ġin - scope Ġaccuracy , Ġscoring Ġ96 % Ġor Ġabove Ġeven Ġwhen Ġwe Ġlimit Ġthe Ġtraining Ġdata Ġor Ġintroduce Ġclass Ġimbalance . ĠHowever , Ġall Ġmethods Ġstruggle Ġwith Ġidentifying Ġout - of - scope Ġqueries . ĠEven Ġwhen Ġa Ġlarge Ġnumber Ġof Ġout - of - scope Ġexamples Ġare Ġprovided Ġfor Ġtraining , Ġthere Ġis Ġa Ġmajor Ġperformance Ġgap , Ġwith Ġthe Ġbest Ġsystem Ġscoring Ġ66 % Ġout - of - scope Ġrecall . ĠOur Ġresults Ġshow Ġthat Ġwhile Ġcurrent Ġmodels Ġwork Ġon Ġknown Ġclasses , Ġthey Ġhave Ġdifficulty Ġon Ġout - of - scope Ġqueries , Ġparticularly Ġwhen Ġdata Ġis Ġnot Ġplentiful . ĠThis Ġdataset Ġwill Ġenable Ġfuture Ġwork Ġto Ġaddress Ġthis Ġkey Ġgap Ġin Ġthe Ġresearch Ġand Ġdevelopment Ġof Ġdialog Ġsystems . ĠAll Ġdata Ġintroduced Ġin Ġthis Ġpaper Ġcan Ġbe Ġfound Ġat Ġhttps :// github . com / cl inc / oos - eval . </s> Dat as et </s> We Ġintroduce Ġa Ġnew Ġcrowds ourced Ġdataset Ġof Ġ23 , 700 Ġqueries , Ġincluding Ġ22 , 500 Ġin - scope Ġqueries Ġcovering Ġ150 Ġint ents , Ġwhich Ġcan Ġbe Ġgrouped Ġinto Ġ10 Ġgeneral Ġdomains . ĠThe Ġdataset Ġalso Ġincludes Ġ1 , 200 Ġout - of - scope Ġqueries . ĠTable ĠT AB REF 2 Ġshows Ġexamples Ġof Ġthe Ġdata . </s> Dat as et Ġ: :: ĠIn - Scope ĠData ĠCollection </s> We Ġdefined Ġthe Ġint ents Ġwith Ġguidance Ġfrom Ġqueries Ġcollected Ġusing Ġa Ġsc oping Ġcrowds ourcing Ġtask , Ġwhich Ġprompted Ġcrowd Ġworkers Ġto Ġprovide Ġquestions Ġand Ġcommands Ġrelated Ġto Ġtopic Ġdomains Ġin Ġthe Ġmanner Ġthey Ġwould Ġinteract Ġwith Ġan Ġartificially Ġintelligent Ġassistant . ĠWe Ġmanually Ġgrouped Ġdata Ġgenerated Ġby Ġsc oping Ġtasks Ġinto Ġint ents . ĠTo Ġcollect Ġadditional Ġdata Ġfor Ġeach Ġintent , Ġwe Ġused Ġthe Ġre phrase Ġand Ġscenario Ġcrowds ourcing Ġtasks Ġproposed Ġby ĠB IB REF 2 . ĠFor Ġeach Ġintent , Ġthere Ġare Ġ100 Ġtraining Ġqueries , Ġwhich Ġis Ġrepresentative Ġof Ġwhat Ġa Ġteam Ġwith Ġa Ġlimited Ġbudget Ġcould Ġgather Ġwhile Ġdeveloping Ġa Ġtask - driven Ġdialog Ġsystem . ĠAlong Ġwith Ġthe Ġ100 Ġtraining Ġqueries , Ġthere Ġare Ġ20 Ġvalidation Ġand Ġ30 Ġtesting Ġqueries Ġper Ġintent . </s> Dat as et Ġ: :: ĠOut - of - Scope ĠData ĠCollection </s> Out - of - scope Ġqueries Ġwere Ġcollected Ġin Ġtwo Ġways . ĠFirst , Ġusing Ġworker Ġmistakes : Ġqueries Ġwritten Ġfor Ġone Ġof Ġthe Ġ150 Ġint ents Ġthat Ġdid Ġnot Ġactually Ġmatch Ġany Ġof Ġthe Ġint ents . ĠSecond , Ġusing Ġsc oping Ġand Ġscenario Ġtasks Ġwith Ġprompts Ġbased Ġon Ġtopic Ġareas Ġfound Ġon ĠQu ora , ĠWikipedia , Ġand Ġelsewhere . ĠTo Ġhelp Ġensure Ġthe Ġrichness Ġof Ġthis Ġadditional Ġout - of - scope Ġdata , Ġeach Ġof Ġthese Ġtask Ġprompts Ġcontributed Ġto Ġat Ġmost Ġfour Ġqueries . ĠSince Ġwe Ġuse Ġthe Ġsame Ġcrowds ourcing Ġmethod Ġfor Ġcollecting Ġout - of - scope Ġdata , Ġthese Ġqueries Ġare Ġsimilar Ġin Ġstyle Ġto Ġtheir Ġin - scope Ġcounterparts . </s> The Ġout - of - scope Ġdata Ġis Ġdifficult Ġto Ġcollect , Ġrequiring Ġexpert Ġknowledge Ġof Ġthe Ġin - scope Ġint ents Ġto Ġpainstaking ly Ġensure Ġthat Ġno Ġout - of - scope Ġquery Ġsample Ġis Ġmistakenly Ġlabeled Ġas Ġin - scope Ġ( and Ġvice Ġversa ). ĠIndeed , Ġroughly Ġonly Ġ69 % Ġof Ġqueries Ġcollected Ġwith Ġprompts Ġtargeting Ġout - of - scope Ġyielded Ġout - of - scope Ġqueries . ĠOf Ġthe Ġ1 , 200 Ġout - of - scope Ġqueries Ġcollected , Ġ100 Ġare Ġused Ġfor Ġvalidation Ġand Ġ100 Ġare Ġused Ġfor Ġtraining , Ġleaving Ġ1 , 000 Ġfor Ġtesting . </s> Dat as et Ġ: :: ĠData ĠPre processing Ġand ĠPart ition ing </s> For Ġall Ġqueries Ġcollected , Ġall Ġtokens Ġwere Ġdown - c ased , Ġand Ġall Ġend - of - sent ence Ġpunct uation Ġwas Ġremoved . ĠAdditionally , Ġall Ġduplicate Ġqueries Ġwere Ġremoved Ġand Ġreplaced . </s> In Ġan Ġeffort Ġto Ġreduce Ġbias Ġin Ġthe Ġin - scope Ġdata , Ġwe Ġplaced Ġall Ġqueries Ġfrom Ġa Ġgiven Ġcrowd Ġworker Ġin Ġa Ġsingle Ġsplit Ġ( train , Ġvalidation , Ġor Ġtest ). ĠThis Ġavoids Ġthe Ġpotential Ġissue Ġof Ġsimilar Ġqueries Ġfrom Ġa Ġcrowd Ġworker Ġending Ġup Ġin Ġboth Ġthe Ġtrain Ġand Ġtest Ġsets , Ġfor Ġinstance , Ġwhich Ġwould Ġmake Ġthe Ġtrain Ġand Ġtest Ġdistributions Ġunreal istically Ġsimilar . ĠWe Ġnote Ġthat Ġthis Ġis Ġa Ġrecommendation Ġfrom Ġconcurrent Ġwork Ġby ĠB IB REF 3 . ĠWe Ġalso Ġused Ġthis Ġprocedure Ġfor Ġthe Ġout - of - scope Ġset , Ġexcept Ġthat Ġwe Ġsplit Ġthe Ġdata Ġinto Ġtrain / valid ation / test Ġbased Ġon Ġtask Ġprompt Ġinstead Ġof Ġworker . </s> Dat as et Ġ: :: ĠDat as et ĠVari ants </s> In Ġaddition Ġto Ġthe Ġfull Ġdataset , Ġwe Ġconsider Ġthree Ġvariations . ĠFirst , ĠSmall , Ġin Ġwhich Ġthere Ġare Ġonly Ġ50 Ġtraining Ġqueries Ġper Ġeach Ġin - scope Ġintent , Ġrather Ġthan Ġ100 . ĠSecond , ĠIm balanced , Ġin Ġwhich Ġint ents Ġhave Ġeither Ġ25 , Ġ50 , Ġ75 , Ġor Ġ100 Ġtraining Ġqueries . ĠThird , ĠO OS +, Ġin Ġwhich Ġthere Ġare Ġ250 Ġout - of - scope Ġtraining Ġexamples , Ġrather Ġthan Ġ100 . ĠThese Ġare Ġintended Ġto Ġrepresent Ġproduction Ġscenarios Ġwhere Ġdata Ġmay Ġbe Ġin Ġlimited Ġor Ġuneven Ġsupply . </s> Bench mark ĠEvaluation </s> To Ġquantify Ġthe Ġchallenges Ġthat Ġour Ġnew Ġdataset Ġpresents , Ġwe Ġevaluated Ġthe Ġperformance Ġof Ġa Ġrange Ġof Ġclass ifier Ġmodels Ġand Ġout - of - scope Ġprediction Ġschemes . </s> Bench mark ĠEvaluation Ġ: :: ĠClass ifier ĠModels </s> S VM : ĠA Ġlinear Ġsupport Ġvector Ġmachine Ġwith Ġbag - of - words Ġsentence Ġrepresentations . </s> ML P : ĠA Ġmulti - layer Ġpercept ron Ġwith ĠUSE Ġembed d ings ĠB IB REF 4 Ġas Ġinput . </s> Fast Text : ĠA Ġshallow Ġneural Ġnetwork Ġthat Ġaverages Ġembed d ings Ġof Ġn - gram s ĠB IB REF 5 . </s> CNN : ĠA Ġconv olution al Ġneural Ġnetwork Ġwith Ġnon - static Ġword Ġembed d ings Ġinitialized Ġwith ĠGl o Ve ĠB IB REF 6 . </s> BER T : ĠA Ġneural Ġnetwork Ġthat Ġis Ġtrained Ġto Ġpredict Ġel ided Ġwords Ġin Ġtext Ġand Ġthen Ġfine - tun ed Ġon Ġour Ġdata ĠB IB REF 1 . </s> Platform s : ĠSeveral Ġplatforms Ġexist Ġfor Ġthe Ġdevelopment Ġof Ġtask - oriented Ġagents . ĠWe Ġconsider ĠGoogle 's ĠDial og Flow Ġand ĠR asa ĠNL U Ġwith Ġsp acy - sk learn . </s> Bench mark ĠEvaluation Ġ: :: ĠOut - of - Scope ĠPrediction </s> We Ġuse Ġthree Ġbaseline Ġapproaches Ġfor Ġthe Ġtask Ġof Ġpredicting Ġwhether Ġa Ġquery Ġis Ġout - of - scope : Ġ( 1 ) Ġo os - train , Ġwhere Ġwe Ġtrain Ġan Ġadditional Ġ( i . e . Ġ151 st ) Ġintent Ġon Ġout - of - scope Ġtraining Ġdata ; Ġ( 2 ) Ġo os - th reshold , Ġwhere Ġwe Ġuse Ġa Ġthreshold Ġon Ġthe Ġclass ifier 's Ġprobability Ġestimate ; Ġand Ġ( 3 ) Ġo os - binary , Ġa Ġtwo - stage Ġprocess Ġwhere Ġwe Ġfirst Ġclassify Ġa Ġquery Ġas Ġin - Ġor Ġout - of - scope , Ġthen Ġclassify Ġit Ġinto Ġone Ġof Ġthe Ġ150 Ġint ents Ġif Ġclassified Ġas Ġin - scope . </s> To Ġreduce Ġthe Ġseverity Ġof Ġthe Ġclass Ġimbalance Ġbetween Ġin - scope Ġversus Ġout - of - scope Ġquery Ġsamples Ġ( i . e ., Ġ15 , 000 Ġversus Ġ250 Ġqueries Ġfor ĠO OS + ), Ġwe Ġinvestigate Ġtwo Ġstrategies Ġwhen Ġusing Ġo os - binary : Ġone Ġwhere Ġwe Ġunders ample Ġthe Ġin - scope Ġdata Ġand Ġtrain Ġusing Ġ1 , 000 Ġin - scope Ġqueries Ġsampled Ġevenly Ġacross Ġall Ġint ents Ġ( vers us Ġ250 Ġout - of - scope ), Ġand Ġanother Ġwhere Ġwe Ġaugment Ġthe Ġ250 ĠO OS + Ġout - of - scope Ġtraining Ġqueries Ġwith Ġ14 , 750 Ġsentences Ġsampled Ġfrom ĠWikipedia . </s> From Ġa Ġdevelopment Ġpoint Ġof Ġview , Ġthe Ġo os - train Ġand Ġo os - binary Ġmethods Ġboth Ġrequire Ġcareful Ġcur ation Ġof Ġan Ġout - of - scope Ġtraining Ġset , Ġand Ġthis Ġset Ġcan Ġbe Ġtailored Ġto Ġindividual Ġsystems . ĠThe Ġo os - th reshold Ġmethod Ġis Ġa Ġmore Ġgeneral Ġdecision Ġrule Ġthat Ġcan Ġbe Ġapplied Ġto Ġany Ġmodel Ġthat Ġproduces Ġa Ġprobability . ĠIn Ġour Ġevaluation , Ġthe Ġout - of - scope Ġthreshold Ġwas Ġchosen Ġto Ġbe Ġthe Ġvalue Ġwhich Ġyielded Ġthe Ġhighest Ġvalidation Ġscore Ġacross Ġall Ġint ents , Ġtreating Ġout - of - scope Ġas Ġits Ġown Ġintent . </s> Bench mark ĠEvaluation Ġ: :: ĠMet rics </s> We Ġconsider Ġtwo Ġperformance Ġmetrics Ġfor Ġall Ġscenarios : Ġ( 1 ) Ġaccuracy Ġover Ġthe Ġ150 Ġint ents , Ġand Ġ( 2 ) Ġrecall Ġon Ġout - of - scope Ġqueries . ĠWe Ġuse Ġrecall Ġto Ġevaluate Ġout - of - scope Ġsince Ġwe Ġare Ġmore Ġinterested Ġin Ġcases Ġwhere Ġsuch Ġqueries Ġare Ġpredicted Ġas Ġin - scope , Ġas Ġthis Ġwould Ġmean Ġa Ġsystem Ġgives Ġthe Ġuser Ġa Ġresponse Ġthat Ġis Ġcompletely Ġwrong . ĠPrecision Ġerrors Ġare Ġless Ġproblematic Ġas Ġthe Ġfall back Ġresponse Ġwill Ġprompt Ġthe Ġuser Ġto Ġtry Ġagain , Ġor Ġinform Ġthe Ġuser Ġof Ġthe Ġsystem 's Ġscope Ġof Ġsupported Ġdomains . </s> Results Ġ: :: ĠResults Ġwith Ġo os - train </s> Table ĠT AB REF 14 Ġpresents Ġresults Ġfor Ġall Ġmodels Ġacross Ġthe Ġfour Ġvariations Ġof Ġthe Ġdataset . ĠFirst , ĠB ERT Ġis Ġconsistently Ġthe Ġbest Ġapproach Ġfor Ġin - scope , Ġfollowed Ġby ĠML P . ĠSecond , Ġout - of - scope Ġquery Ġperformance Ġis Ġmuch Ġlower Ġthan Ġin - scope Ġacross Ġall Ġmethods . ĠTraining Ġon Ġless Ġdata Ġ( Small Ġand ĠIm balanced ) Ġyields Ġmodels Ġthat Ġperform Ġslightly Ġworse Ġon Ġin - scope Ġqueries . ĠThe Ġtrend Ġis Ġmostly Ġthe Ġopposite Ġwhen Ġevaluating Ġout - of - scope , Ġwhere Ġrecall Ġincreases Ġunder Ġthe ĠSmall Ġand ĠIm balanced Ġtraining Ġconditions . ĠUnder Ġthese Ġtwo Ġconditions , Ġthe Ġsize Ġof Ġthe Ġin - scope Ġtraining Ġset Ġwas Ġdecreased , Ġwhile Ġthe Ġnumber Ġof Ġout - of - scope Ġtraining Ġqueries Ġremained Ġconstant . ĠThis Ġindicates Ġthat Ġout - of - scope Ġperformance Ġcan Ġbe Ġincreased Ġby Ġincreasing Ġthe Ġrelative Ġnumber Ġof Ġout - of - scope Ġtraining Ġqueries . ĠWe Ġdo Ġjust Ġthat Ġin Ġthe ĠO OS + Ġsetting âĢĶ where Ġthe Ġmodels Ġwere Ġtrained Ġon Ġthe Ġfull Ġtraining Ġset Ġas Ġwell Ġas Ġ150 Ġadditional Ġout - of - scope Ġqueries âĢĶ and Ġsee Ġthat Ġperformance Ġon Ġout - of - scope Ġincreases Ġsubstantially , Ġyet Ġstill Ġremains Ġlow Ġrelative Ġto Ġin - scope Ġaccuracy . </s> Results Ġ: :: ĠResults Ġwith Ġo os - th reshold </s> In - scope Ġaccuracy Ġusing Ġthe Ġo os - th reshold Ġapproach Ġis Ġlargely Ġcomparable Ġto Ġo os - train . ĠOut - of - scope Ġrecall Ġtends Ġto Ġbe Ġmuch Ġhigher Ġon ĠFull , Ġbut Ġseveral Ġmodels Ġsuffer Ġgreatly Ġon Ġthe Ġlimited Ġdatasets . ĠB ERT Ġand ĠML P Ġare Ġthe Ġtop Ġo os - th reshold Ġperformers , Ġand Ġfor Ġseveral Ġmodels Ġthe Ġthreshold Ġapproach Ġprovided Ġerratic Ġresults , Ġparticularly ĠFast Text Ġand ĠR asa . </s> Results Ġ: :: ĠResults Ġwith Ġo os - binary </s> Table ĠT AB REF 19 Ġcompares Ġclass ifier Ġperformance Ġusing Ġthe Ġo os - binary Ġscheme . ĠIn - scope Ġaccuracy Ġsuffers Ġfor Ġall Ġmodels Ġusing Ġthe Ġunders am pling Ġscheme Ġwhen Ġcompared Ġto Ġtraining Ġon Ġthe Ġfull Ġdataset Ġusing Ġthe Ġo os - train Ġand Ġo os - th reshold Ġapproaches Ġshown Ġin ĠTable ĠT AB REF 14 . ĠHowever , Ġout - of - scope Ġrecall Ġimproves Ġcompared Ġto Ġo os - train Ġon ĠFull Ġbut Ġnot ĠO OS +. ĠAug ment ing Ġthe Ġout - of - scope Ġtraining Ġset Ġappears Ġto Ġhelp Ġimprove Ġboth Ġin - scope Ġand Ġout - of - scope Ġperformance Ġcompared Ġto Ġunders am pling , Ġbut Ġout - of - scope Ġperformance Ġremains Ġweak . </s> Prior ĠWork </s> In Ġmost Ġother Ġanalyses Ġand Ġdatasets , Ġthe Ġidea Ġof Ġout - of - scope Ġdata Ġis Ġnot Ġconsidered , Ġand Ġinstead Ġthe Ġoutput Ġclasses Ġare Ġintended Ġto Ġcover Ġall Ġpossible Ġqueries Ġ( e . g ., ĠT REC ĠB IB REF 7 ). ĠRecent Ġwork Ġby ĠB IB REF 8 Ġconsiders Ġa Ġsimilar Ġproblem Ġthey Ġcall Ġout - of - dist ribution Ġdetection . ĠThey Ġuse Ġother Ġdatasets Ġor Ġclasses Ġexcluded Ġduring Ġtraining Ġto Ġform Ġthe Ġout - of - dist ribution Ġsamples . ĠThis Ġmeans Ġthat Ġthe Ġout - of - scope Ġsamples Ġare Ġfrom Ġa Ġsmall Ġset Ġof Ġcoherent Ġclasses Ġthat Ġdiffer Ġsubstantially Ġfrom Ġthe Ġin - dist ribution Ġsamples . ĠSimilar Ġexperiments Ġwere Ġconducted Ġfor Ġevaluating Ġunknown Ġintent Ġdiscovery Ġmodels Ġin ĠB IB REF 9 . ĠIn Ġcontrast , Ġour Ġout - of - scope Ġqueries Ġcover Ġa Ġbroad Ġrange Ġof Ġphenomena Ġand Ġare Ġsimilar Ġin Ġstyle Ġand Ġoften Ġsimilar Ġin Ġtopic Ġto Ġin - scope Ġqueries , Ġrepresenting Ġthings Ġa Ġuser Ġmight Ġsay Ġgiven Ġpartial Ġknowledge Ġof Ġthe Ġcapabilities Ġof Ġa Ġsystem . </s> Table ĠT AB REF 20 Ġcompares Ġour Ġdataset Ġwith Ġother Ġshort - query Ġintent Ġclassification Ġdatasets . ĠThe ĠSn ips ĠB IB REF 0 Ġdataset Ġand Ġthe Ġdataset Ġpresented Ġin ĠB IB REF 10 Ġare Ġthe Ġmost Ġsimilar Ġto Ġthe Ġin - scope Ġpart Ġof Ġour Ġwork , Ġwith Ġthe Ġsame Ġtype Ġof Ġconvers ational Ġagent Ġrequests . ĠLike Ġour Ġwork , Ġboth Ġof Ġthese Ġdatasets Ġwere Ġboot stra pped Ġusing Ġcrowds ourcing . ĠHowever , Ġthe ĠSn ips Ġdataset Ġhas Ġonly Ġa Ġsmall Ġnumber Ġof Ġint ents Ġand Ġan Ġenormous Ġnumber Ġof Ġexamples Ġof Ġeach . ĠSn ips Ġdoes Ġpresent Ġa Ġlow - data Ġvariation , Ġwith Ġ70 Ġtraining Ġqueries Ġper Ġintent , Ġin Ġwhich Ġperformance Ġdrops Ġslightly . ĠThe Ġdataset Ġpresented Ġin ĠB IB REF 10 Ġhas Ġa Ġlarge Ġnumber Ġof Ġintent Ġclasses , Ġyet Ġalso Ġcontains Ġa Ġwide Ġrange Ġof Ġsamples Ġper Ġintent Ġclass Ġ( ranging Ġfrom Ġ24 Ġto Ġ5 , 98 1 Ġqueries Ġper Ġintent , Ġand Ġso Ġis Ġnot Ġconstrained Ġin Ġall Ġcases ). </s> BI B REF 11 Ġcreated Ġdatasets Ġwith Ġconstrained Ġtraining Ġdata , Ġbut Ġwith Ġvery Ġfew Ġint ents , Ġpresenting Ġa Ġvery Ġdifferent Ġtype Ġof Ġchallenge . ĠWe Ġalso Ġinclude Ġthe ĠT REC Ġquery Ġclassification Ġdatasets ĠB IB REF 7 , Ġwhich Ġhave Ġa Ġlarge Ġset Ġof Ġlabels , Ġbut Ġthey Ġdescribe Ġthe Ġdesired Ġresponse Ġtype Ġ( e . g ., Ġdistance , Ġcity , Ġabbre viation ) Ġrather Ġthan Ġthe Ġaction Ġint ents Ġwe Ġconsider . ĠMoreover , ĠT REC Ġcontains Ġonly Ġquestions Ġand Ġno Ġcommands . ĠCru cially , Ġnone Ġof Ġthe Ġother Ġdatasets Ġsummarized Ġin ĠTable ĠT AB REF 20 Ġoffer Ġa Ġfeasible Ġway Ġto Ġevaluate Ġout - of - scope Ġperformance . </s> The ĠDial og ĠState ĠTracking ĠChallenge Ġ( D ST C ) Ġdatasets Ġare Ġanother Ġrelated Ġresource . ĠSpecifically , ĠD ST C Ġ1 ĠB IB REF 12 , ĠD ST C Ġ2 ĠB IB REF 13 , Ġand ĠD ST C Ġ3 ĠB IB REF 14 Ġcontain ĠâĢ ľ chat bot Ġstyle " Ġqueries , Ġbut Ġthe Ġdatasets Ġare Ġfocused Ġon Ġstate Ġtracking . ĠMoreover , Ġmost Ġif Ġnot Ġall Ġqueries Ġin Ġthese Ġdatasets Ġare Ġin - scope . ĠIn Ġcontrast , Ġthe Ġfocus Ġof Ġour Ġanalysis Ġis Ġon Ġboth Ġin - Ġand Ġout - of - scope Ġqueries Ġthat Ġchallenge Ġa Ġvirtual Ġassistant Ġto Ġdetermine Ġwhether Ġit Ġcan Ġprovide Ġan Ġacceptable Ġresponse . </s> Conclusion </s> This Ġpaper Ġanalyzed Ġintent Ġclassification Ġand Ġout - of - scope Ġprediction Ġmethods Ġwith Ġa Ġnew Ġdataset Ġconsisting Ġof Ġcarefully Ġcollected Ġout - of - scope Ġdata . ĠOur Ġfindings Ġindicate Ġthat Ġcertain Ġmodels Ġlike ĠB ERT Ġperform Ġbetter Ġon Ġin - scope Ġclassification , Ġbut Ġall Ġmethods Ġinvestigated Ġstruggle Ġwith Ġidentifying Ġout - of - scope Ġqueries . ĠModels Ġthat Ġincorporate Ġmore Ġout - of - scope Ġtraining Ġdata Ġtend Ġto Ġimprove Ġon Ġout - of - scope Ġperformance , Ġyet Ġsuch Ġdata Ġis Ġexpensive Ġand Ġdifficult Ġto Ġgenerate . ĠWe Ġbelieve Ġour Ġanalysis Ġand Ġdataset Ġwill Ġlead Ġto Ġdeveloping Ġbetter , Ġmore Ġrobust Ġdialog Ġsystems . </s> All Ġdatasets Ġintroduced Ġin Ġthis Ġpaper Ġcan Ġbe Ġfound Ġat Ġhttps :// github . com / cl inc / oos - eval .