DSTC2
EXTRACTIVE
The manual transcriptions of the DSTC2 training set 
EXTRACTIVE
In this paper the Neural User Simulator (NUS) is introduced which outputs natural language and whose behaviour is learned from a corpus. The main component, inspired by BIBREF4 , consists of a feature extractor and a neural network based sequence-to-sequence model BIBREF5 . The sequence-to-sequence model consists of a recurrent neural network (RNN) encoder that encodes the dialogue history and a decoder RNN which outputs natural language. Furthermore, the NUS generates its own goal and possibly changes it during a dialogue. This allows the model to be deployed for training more sophisticated DM policies. To achieve this, a method is proposed that transforms the goal-labels of the used dataset (DSTC2) into labels whose behaviour can be replicated during deployment.
The manual transcriptions of the DSTC2 training set (not the ASR output) were used to train the sequence-to-sequence model. Since the transcriptions were done manually they contained spelling errors. These were manually corrected to ensure proper delexicalization. Some dialogues were discarded due to transcriptions errors being too large. After cleaning the dataset the training set consisted of 1609 dialogues with a total of 11638 dialogue turns. The validation set had 505 dialogues with 3896 dialogue turns. The maximum sequence length of the delexicalized turns was 22, including the end of sentence character. The maximum dialogue length was 30 turns.
1805.06966
Introduction </s> Sp oken ĠDialogue ĠSystems Ġ( S DS ) Ġallow Ġhuman - computer Ġinteraction Ġusing Ġnatural Ġspeech . ĠTask - oriented Ġdialogue Ġsystems , Ġthe Ġfocus Ġof Ġthis Ġwork , Ġhelp Ġusers Ġachieve Ġgoals Ġsuch Ġas Ġfinding Ġrestaurants Ġor Ġbooking Ġflights ĠB IB REF 0 Ġ. </s> Te aching Ġa Ġsystem Ġhow Ġto Ġrespond Ġappropriately Ġin Ġa Ġtask - oriented Ġsetting Ġis Ġnon - t riv ial . ĠIn Ġstate - of - the - art Ġsystems Ġthis Ġdialogue Ġmanagement Ġtask Ġis Ġoften Ġformulated Ġas Ġa Ġreinforcement Ġlearning Ġ( RL ) Ġproblem ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ. ĠIn Ġthis Ġframework , Ġthe Ġsystem Ġlearns Ġby Ġa Ġtrial Ġand Ġerror Ġprocess Ġgoverned Ġby Ġa Ġreward Ġfunction . ĠUser ĠSim ulators Ġcan Ġbe Ġused Ġto Ġtrain Ġthe Ġpolicy Ġof Ġa Ġdialogue Ġmanager Ġ( DM ) Ġwithout Ġreal Ġuser Ġinteractions . ĠFurthermore , Ġthey Ġallow Ġan Ġunlimited Ġnumber Ġof Ġdialog ues Ġto Ġbe Ġcreated Ġwith Ġeach Ġdialogue Ġbeing Ġfaster Ġthan Ġa Ġdialogue Ġwith Ġa Ġhuman . </s> In Ġthis Ġpaper Ġthe ĠNeural ĠUser ĠSimulator Ġ( N US ) Ġis Ġintroduced Ġwhich Ġoutputs Ġnatural Ġlanguage Ġand Ġwhose Ġbehaviour Ġis Ġlearned Ġfrom Ġa Ġcorpus . ĠThe Ġmain Ġcomponent , Ġinspired Ġby ĠB IB REF 4 Ġ, Ġconsists Ġof Ġa Ġfeature Ġextract or Ġand Ġa Ġneural Ġnetwork Ġbased Ġsequence - to - sequence Ġmodel ĠB IB REF 5 Ġ. ĠThe Ġsequence - to - sequence Ġmodel Ġconsists Ġof Ġa Ġrecurrent Ġneural Ġnetwork Ġ( R NN ) Ġenc oder Ġthat Ġenc odes Ġthe Ġdialogue Ġhistory Ġand Ġa Ġdec oder ĠR NN Ġwhich Ġoutputs Ġnatural Ġlanguage . ĠFurthermore , Ġthe ĠN US Ġgenerates Ġits Ġown Ġgoal Ġand Ġpossibly Ġchanges Ġit Ġduring Ġa Ġdialogue . ĠThis Ġallows Ġthe Ġmodel Ġto Ġbe Ġdeployed Ġfor Ġtraining Ġmore Ġsophisticated ĠDM Ġpolicies . ĠTo Ġachieve Ġthis , Ġa Ġmethod Ġis Ġproposed Ġthat Ġtransforms Ġthe Ġgoal - lab els Ġof Ġthe Ġused Ġdataset Ġ( D ST C 2 ) Ġinto Ġlabels Ġwhose Ġbehaviour Ġcan Ġbe Ġreplicated Ġduring Ġdeployment . </s> The ĠN US Ġis Ġtrained Ġon Ġdialog ues Ġbetween Ġreal Ġusers Ġand Ġan ĠS DS Ġin Ġa Ġrestaurant Ġrecommendation Ġdomain . ĠCompared Ġto Ġmuch Ġof Ġthe Ġrelated Ġwork Ġon Ġuser Ġsimulation , Ġwe Ġuse Ġthe Ġtrained ĠN US Ġto Ġtrain Ġthe Ġpolicy Ġof Ġa Ġreinforcement Ġlearning Ġbased ĠS DS . ĠIn Ġorder Ġto Ġevaluate Ġthe ĠN US , Ġan ĠAgenda - Based ĠUser - Sim ulator Ġ( AB US ) ĠB IB REF 6 Ġis Ġused Ġto Ġtrain Ġanother Ġpolicy . ĠThe Ġtwo Ġpolicies Ġare Ġcompared Ġagainst Ġeach Ġother Ġby Ġusing Ġcross - model Ġevaluation ĠB IB REF 7 Ġ. ĠThis Ġmeans Ġto Ġtrain Ġon Ġone Ġmodel Ġand Ġto Ġtest Ġon Ġthe Ġother . ĠFurthermore , Ġboth Ġtrained Ġpolicies Ġare Ġtested Ġon Ġreal Ġusers . ĠOn Ġboth Ġevaluation Ġtasks Ġthe ĠN US Ġoutper forms Ġthe ĠAB US , Ġwhich Ġis Ġcurrently Ġone Ġof Ġthe Ġmost Ġpopular Ġoff - line Ġtraining Ġtools Ġfor Ġreinforcement Ġlearning Ġbased ĠSp oken ĠDialogue ĠSystems ĠB IB REF 8 Ġ, ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ. </s> The Ġremainder Ġof Ġthis Ġpaper Ġis Ġorganised Ġas Ġfollows . ĠSection Ġ2 Ġbriefly Ġdescribes Ġtask - oriented Ġdialogue . ĠSection Ġ3 Ġdescribes Ġthe Ġmotivation Ġfor Ġthe ĠN US Ġand Ġdiscusses Ġrelated Ġwork . ĠSection Ġ4 Ġexplains Ġthe Ġstructure Ġof Ġthe ĠN US , Ġhow Ġit Ġis Ġtrained Ġand Ġhow Ġit Ġis Ġdeployed Ġfor Ġtraining Ġa ĠDM 's Ġpolicy . ĠSections Ġ5 Ġand Ġ6 Ġpresent Ġthe Ġexperimental Ġsetup Ġand Ġresults . ĠFinally , ĠSection Ġ7 Ġgives Ġconclusions . </s> Task - O ri ented ĠDialogue </s> A ĠTask - O ri ented ĠS DS Ġis Ġtypically Ġdesigned Ġaccording Ġto Ġa Ġstructured Ġont ology , Ġwhich Ġdefines Ġwhat Ġthe Ġsystem Ġcan Ġtalk Ġabout . ĠIn Ġa Ġsystem Ġrecommending Ġrestaurants Ġthe Ġont ology Ġdefines Ġthose Ġattributes Ġof Ġa Ġrestaurant Ġthat Ġthe Ġuser Ġcan Ġchoose , Ġcalled Ġinform able Ġslots Ġ( e . g . Ġdifferent Ġfood Ġtypes , Ġareas Ġand Ġprice Ġranges ), Ġthe Ġattributes Ġthat Ġthe Ġuser Ġcan Ġrequest , Ġcalled Ġrequest able Ġslots Ġ( e . g . Ġphone Ġnumber Ġor Ġaddress ) Ġand Ġthe Ġrestaurants Ġthat Ġit Ġhas Ġdata Ġabout . ĠAn Ġattribute Ġis Ġreferred Ġto Ġas Ġa Ġslot Ġand Ġhas Ġa Ġcorresponding Ġvalue . ĠTogether Ġthese Ġare Ġreferred Ġto Ġas Ġa Ġslot - value Ġpair Ġ( e . g . Ġarea = north ). </s> Using ĠRL Ġthe ĠDM Ġis Ġtrained Ġto Ġact Ġsuch Ġthat Ġis Ġmaxim ises Ġthe Ġcumulative Ġfuture Ġreward . ĠThe Ġprocess Ġby Ġwhich Ġthe ĠDM Ġchooses Ġits Ġnext Ġaction Ġis Ġcalled Ġits Ġpolicy . ĠA Ġtypical Ġapproach Ġto Ġdefining Ġthe Ġreward Ġfunction Ġfor Ġa Ġtask - oriented ĠS DS Ġis Ġto Ġapply Ġa Ġsmall Ġper - turn Ġpenalty Ġto Ġencourage Ġshort Ġdialog ues Ġand Ġto Ġgive Ġa Ġlarge Ġpositive Ġreward Ġat Ġthe Ġend Ġof Ġeach Ġsuccessful Ġinteraction . </s> Mot ivation Ġand ĠRelated ĠWork </s> Ide ally Ġthe ĠDM 's Ġpolicy Ġwould Ġbe Ġtrained Ġby Ġinteracting Ġwith Ġreal Ġusers . ĠAlthough Ġthere Ġare Ġmodels Ġthat Ġsupport Ġon - line Ġlearning ĠB IB REF 15 Ġ, Ġfor Ġthe Ġmajority Ġof ĠRL Ġalgorithms , Ġwhich Ġrequire Ġa Ġlot Ġof Ġinteractions , Ġthis Ġis Ġimpractical . ĠFurthermore , Ġa Ġset Ġof Ġusers Ġneeds Ġto Ġbe Ġrecruited Ġevery Ġtime Ġa Ġpolicy Ġis Ġtrained . ĠThis Ġmakes Ġcommon Ġpractices Ġsuch Ġas Ġhyper - param eter Ġoptimization Ġprohib itive ly Ġexpensive . ĠThus , Ġit Ġis Ġnatural Ġto Ġtry Ġto Ġlearn Ġfrom Ġa Ġdataset Ġwhich Ġneeds Ġto Ġbe Ġrecorded Ġonly Ġonce , Ġbut Ġcan Ġbe Ġused Ġover Ġand Ġover Ġagain . </s> A Ġproblem Ġwith Ġlearning Ġdirectly Ġfrom Ġrecorded Ġdialogue Ġcorpor a Ġis Ġthat Ġthe Ġstate Ġspace Ġthat Ġwas Ġvisited Ġduring Ġthe Ġcollection Ġof Ġthe Ġdata Ġis Ġlimited ; Ġthe Ġsize Ġof Ġthe Ġrecorded Ġcorpus Ġusually Ġfalls Ġshort Ġof Ġthe Ġrequirements Ġfor Ġtraining Ġa Ġstatistical ĠDM . ĠHowever , Ġeven Ġif Ġthe Ġsize Ġof Ġthe Ġcorpus Ġis Ġlarge Ġenough Ġthe Ġoptimal Ġdialogue Ġstrategy Ġis Ġlikely Ġnot Ġto Ġbe Ġcontained Ġwithin Ġit . </s> A Ġsolution Ġis Ġto Ġtransform Ġthe Ġstatic Ġcorpus Ġinto Ġa Ġdynamic Ġtool : Ġa Ġuser Ġsimulator . ĠThe Ġuser Ġsimulator Ġ( US ) Ġis Ġtrained Ġon Ġa Ġdialogue Ġcorpus Ġto Ġlearn Ġwhat Ġresponses Ġa Ġreal Ġuser Ġwould Ġprovide Ġin Ġa Ġgiven Ġdialogue Ġcontext . ĠThe ĠUS Ġis Ġtrained Ġusing Ġsupervised Ġlearning Ġsince Ġthe Ġaim Ġis Ġfor Ġit Ġto Ġlearn Ġtypical Ġuser Ġbehaviour . ĠFor Ġthe ĠDM , Ġhowever , Ġwe Ġwant Ġoptimal Ġbehaviour Ġwhich Ġis Ġwhy Ġsupervised Ġlearning Ġcannot Ġbe Ġused . ĠBy Ġinteracting Ġwith Ġthe ĠS DS , Ġthe Ġtrained ĠUS Ġcan Ġbe Ġused Ġto Ġtrain Ġthe ĠDM 's Ġpolicy . ĠThe ĠDM 's Ġpolicy Ġis Ġoptim ised Ġusing Ġthe Ġfeedback Ġgiven Ġby Ġeither Ġthe Ġuser Ġsimulator Ġor Ġa Ġseparate Ġeval u ator . ĠAny Ġnumber Ġof Ġdialog ues Ġcan Ġbe Ġgenerated Ġusing Ġthe ĠUS Ġand Ġdialogue Ġstrategies Ġthat Ġare Ġnot Ġin Ġthe Ġrecorded Ġcorpus Ġcan Ġbe Ġexplored . </s> Most Ġuser - sim ulators Ġwork Ġon Ġthe Ġlevel Ġof Ġuser Ġsemantics . ĠThese Ġusually Ġconsist Ġof Ġa Ġuser Ġdialogue Ġact Ġ( e . g . Ġinform , Ġor Ġrequest ) Ġand Ġa Ġcorresponding Ġslot - value Ġpair . ĠThe Ġfirst Ġstatistical Ġuser Ġsimulator ĠB IB REF 16 Ġused Ġa Ġsimple Ġbi - gram Ġmodel ĠIN LINE FORM 0 Ġto Ġpredict Ġthe Ġnext Ġuser Ġact ĠIN LINE FORM 1 Ġgiven Ġthe Ġlast Ġsystem Ġact ĠIN LINE FORM 2 Ġ. ĠIt Ġhas Ġthe Ġadvantage Ġof Ġbeing Ġpurely Ġprob abil istic Ġand Ġdomain - independent . ĠHowever , Ġit Ġdoes Ġnot Ġtake Ġthe Ġfull Ġdialogue Ġhistory Ġinto Ġaccount Ġand Ġis Ġnot Ġconditioned Ġon Ġa Ġgoal , Ġleading Ġto Ġinco herent Ġuser Ġbehaviour Ġthroughout Ġa Ġdialogue . ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġattempted Ġto Ġovercome Ġgoal Ġinconsistency Ġby Ġproposing Ġa Ġgraph - based Ġmodel . ĠHowever , Ġdeveloping Ġthe Ġgraph Ġstructure Ġrequires Ġextensive Ġdomain - specific Ġknowledge . ĠB IB REF 19 Ġcombined Ġfeatures Ġfrom ĠShe ff ler Ġand ĠYoung 's Ġwork Ġwith ĠEck ert 's ĠModel , Ġby Ġconditioning Ġa Ġset Ġof Ġprobabilities Ġon Ġan Ġexplicit Ġrepresentation Ġof Ġthe Ġuser Ġgoal Ġand Ġmemory . ĠA ĠMark ov ĠModel Ġis Ġalso Ġused Ġby ĠB IB REF 20 Ġ. ĠIt Ġuses Ġa Ġlarge Ġfeature Ġvector Ġto Ġdescribe Ġthe Ġuser 's Ġcurrent Ġstate , Ġwhich Ġhelps Ġto Ġcompensate Ġfor Ġthe ĠMark ov Ġassumption . ĠHowever , Ġthe Ġmodel Ġis Ġnot Ġconditioned Ġon Ġany Ġgoal . ĠTherefore , Ġit Ġis Ġnot Ġused Ġto Ġtrain Ġa Ġdialogue Ġpolicy Ġsince Ġit Ġis Ġimpossible Ġto Ġdetermine Ġwhether Ġthe Ġuser Ġgoal Ġwas Ġfulfilled . ĠA Ġhidden ĠMark ov Ġmodel Ġwas Ġproposed Ġby ĠB IB REF 21 Ġ, Ġwhich Ġwas Ġalso Ġnot Ġused Ġto Ġtrain Ġa Ġpolicy . ĠB IB REF 22 Ġcast Ġuser Ġsimulation Ġas Ġan Ġinverse Ġreinforcement Ġlearning Ġproblem Ġwhere Ġthe Ġuser Ġis Ġmod elled Ġas Ġa Ġdecision - making Ġagent . ĠThe Ġmodel Ġdid Ġnot Ġincorporate Ġa Ġuser Ġgoal Ġand Ġwas Ġhence Ġnot Ġused Ġto Ġtrain Ġa Ġpolicy . ĠThe Ġmost Ġprominent Ġuser Ġmodel Ġfor Ġpolicy Ġoptim isation Ġis Ġthe ĠAgenda - Based ĠUser ĠSimulator ĠB IB REF 6 Ġ, Ġwhich Ġrepresents Ġthe Ġuser Ġstate Ġeleg antly Ġas Ġa Ġstack Ġof Ġnecessary Ġuser Ġactions , Ġcalled Ġthe Ġagenda . ĠThe Ġmechanism Ġthat Ġgenerates Ġthe Ġuser Ġresponse Ġand Ġupdates Ġthe Ġagenda Ġdoes Ġnot Ġrequire Ġany Ġdata , Ġthough Ġit Ġcan Ġbe Ġimproved Ġusing Ġdata . ĠThe Ġmodel Ġis Ġconditioned Ġon Ġa Ġgoal Ġfor Ġwhich Ġit Ġhas Ġupdate Ġrules Ġin Ġcase Ġthe Ġdialogue Ġsystem Ġexpresses Ġthat Ġit Ġcannot Ġfulfil Ġthe Ġgoal . ĠB IB REF 4 Ġmod elled Ġuser Ġsimulation Ġas Ġa Ġsequence - to - sequence Ġtask . ĠThe Ġmodel Ġcan Ġkeep Ġtrack Ġof Ġthe Ġdialogue Ġhistory Ġand Ġuser Ġbehaviour Ġis Ġlearned Ġentirely Ġfrom Ġdata . ĠHowever , Ġgoal Ġchanges Ġwere Ġnot Ġmod elled , Ġeven Ġthough Ġa Ġlarge Ġproportion Ġof Ġdialog ues Ġwithin Ġtheir Ġdataset Ġ( D ST C 2 ) Ġcontains Ġgoal Ġchanges . ĠTheir Ġmodel Ġoutper formed Ġthe ĠAB US Ġon Ġstatistical Ġmetrics , Ġwhich Ġis Ġnot Ġsurprising Ġgiven Ġthat Ġit Ġwas Ġtrained Ġby Ġoptim ising Ġa Ġstatistical Ġmetric Ġand Ġthe ĠAB US Ġwas Ġnot . </s> The Ġaforementioned Ġwork Ġfocuses Ġon Ġuser Ġsimulation Ġat Ġthe Ġsemantic Ġlevel . ĠMultiple Ġissues Ġarise Ġfrom Ġthis Ġapproach . ĠFirstly , Ġannot ating Ġthe Ġuser - response Ġwith Ġthe Ġcorrect Ġsemantics Ġis Ġcostly . ĠMore Ġdata Ġcould Ġbe Ġcollected , Ġif Ġthe ĠUS Ġwere Ġto Ġoutput Ġnatural Ġlanguage . ĠSecondly , Ġresearch Ġsuggests Ġthat Ġthe Ġtwo Ġmodules Ġof Ġan ĠS DS Ġperforming ĠSp oken ĠLanguage ĠUnderstanding Ġ( SL U ) Ġand Ġbelief Ġtracking Ġshould Ġbe Ġjointly Ġtrained Ġas Ġa Ġsingle Ġentity ĠB IB REF 23 Ġ, ĠB IB REF 24 Ġ, ĠB IB REF 25 Ġ, ĠB IB REF 26 Ġ, ĠB IB REF 27 Ġ. ĠIn Ġfact Ġin Ġthe Ġsecond ĠDialogue ĠState ĠTracking ĠChallenge Ġ( D ST C 2 ) ĠB IB REF 28 Ġ, Ġthe Ġdata Ġof Ġwhich Ġthis Ġwork Ġuses , Ġsystems Ġwhich Ġused Ġno Ġexternal ĠSL U Ġmodule Ġoutper formed Ġall Ġsystems Ġthat Ġonly Ġused Ġan Ġexternal ĠSL U ĠModule . ĠTraining Ġthe Ġpolicy Ġof Ġa ĠDM Ġin Ġa Ġsimulated Ġenvironment , Ġwhen Ġalso Ġusing Ġa Ġjoint Ġsystem Ġfor ĠSL U Ġand Ġbelief Ġtracking Ġis Ġnot Ġpossible Ġwithout Ġa ĠUS Ġthat Ġproduces Ġnatural Ġlanguage . ĠThird ly , Ġa ĠUS Ġis Ġsometimes Ġaugmented Ġwith Ġan Ġerror Ġmodel Ġwhich Ġgenerates Ġa Ġset Ġof Ġcompeting Ġhypotheses Ġwith Ġassociated Ġconfidence Ġscores Ġtrying Ġto Ġreplicate Ġthe Ġerrors Ġof Ġthe Ġspeech Ġrecogn iser . ĠWhen Ġthe Ġerror Ġmodel Ġmatches Ġthe Ġcharacteristics Ġof Ġthe Ġspeech Ġrecogn iser Ġmore Ġaccurately , Ġthe ĠS DS Ġperforms Ġbetter ĠB IB REF 29 Ġ. ĠHowever , Ġspeech Ġrecognition Ġerrors Ġare Ġbadly Ġmod elled Ġbased Ġon Ġuser Ġsemantics Ġsince Ġthey Ġarise Ġ( mostly ) Ġdue Ġto Ġthe Ġphon etics Ġof Ġthe Ġspoken Ġwords Ġand Ġnot Ġtheir Ġsemantics ĠB IB REF 30 Ġ. ĠThus , Ġan ĠS DS Ġthat Ġis Ġtrained Ġwith Ġa Ġnatural Ġlanguage Ġbased Ġerror Ġmodel Ġis Ġlikely Ġto Ġoutper form Ġone Ġtrained Ġwith Ġa Ġsemantic Ġerror Ġmodel Ġwhen Ġtested Ġon Ġreal Ġusers . ĠSequence - to - sequence Ġlearning Ġfor Ġword - level Ġuser Ġsimulation Ġis Ġperformed Ġin ĠB IB REF 31 Ġ, Ġthough Ġthe Ġmodel Ġis Ġnot Ġconditioned Ġon Ġany Ġgoal Ġand Ġhence Ġnot Ġused Ġfor Ġpolicy Ġoptim isation . ĠA Ġword - level Ġuser Ġsimulator Ġwas Ġalso Ġused Ġin ĠB IB REF 32 Ġwhere Ġit Ġwas Ġbuilt Ġby Ġaugment ing Ġthe ĠAB US Ġwith Ġa Ġnatural Ġlanguage Ġgenerator . </s> Ne ural ĠUser ĠSimulator </s> An Ġoverview Ġof Ġthe ĠN US Ġis Ġgiven Ġin ĠFigure ĠFIG REF 2 Ġ. ĠAt Ġthe Ġstart Ġof Ġa Ġdialogue Ġa Ġrandom Ġgoal ĠIN LINE FORM 0 Ġis Ġgenerated Ġby Ġthe ĠGoal ĠGenerator . ĠThe Ġpossibilities Ġfor ĠIN LINE FORM 1 Ġare Ġdefined Ġby Ġthe Ġont ology . ĠIn Ġdialogue Ġturn ĠIN LINE FORM 2 Ġ, Ġthe Ġoutput Ġof Ġthe ĠS DS Ġ( ĠIN LINE FORM 3 Ġ) Ġis Ġpassed Ġto Ġthe ĠN US 's ĠFeature ĠExtract or , Ġwhich Ġgenerates Ġa Ġfeature Ġvector ĠIN LINE FORM 4 Ġbased Ġon ĠIN LINE FORM 5 Ġ, Ġthe Ġcurrent Ġuser Ġgoal , ĠIN LINE FORM 6 Ġ, Ġand Ġparts Ġof Ġthe Ġdialogue Ġhistory . ĠThis Ġvector Ġis Ġapp ended Ġto Ġthe ĠFeature ĠHistory ĠIN LINE FORM 7 Ġ. ĠThis Ġsequence Ġis Ġpassed Ġto Ġthe Ġsequence - to - sequence Ġmodel Ġ( Fig . ĠFIG REF 7 Ġ), Ġwhich Ġwill Ġgenerate Ġthe Ġuser 's Ġlength ĠIN LINE FORM 8 Ġutter ance ĠIN LINE FORM 9 Ġ. ĠAs Ġin ĠFigure ĠFIG REF 7 Ġ, Ġwords Ġin ĠIN LINE FORM 10 Ġcorresponding Ġto Ġa Ġslot Ġare Ġreplaced Ġby Ġa Ġslot Ġtoken ; Ġa Ġprocess Ġcalled Ġde lex ical isation . ĠIf Ġthe ĠS DS Ġexpresses Ġto Ġthe ĠN US Ġthat Ġthere Ġis Ġno Ġvenue Ġmatching Ġthe ĠN US 's Ġconstraints , Ġthe Ġgoal Ġwill Ġbe Ġaltered Ġby Ġthe ĠGoal ĠGenerator . </s> Goal ĠGenerator </s> The ĠGoal ĠGenerator Ġgenerates Ġa Ġrandom Ġgoal ĠIN LINE FORM 0 Ġat Ġthe Ġstart Ġof Ġthe Ġdialogue . ĠIt Ġconsists Ġof Ġa Ġset Ġof Ġconstraints , ĠIN LINE FORM 1 Ġ, Ġwhich Ġspecify Ġthe Ġrequired Ġvenue Ġe . g . Ġ( food = Spanish , Ġarea = north ) Ġand Ġa Ġnumber Ġof Ġrequests , ĠIN LINE FORM 2 Ġ, Ġthat Ġspecify Ġthe Ġinformation Ġthat Ġthe ĠN US Ġwants Ġabout Ġthe Ġfinal Ġvenue Ġe . g . Ġthe Ġaddress Ġor Ġthe Ġphone Ġnumber . ĠThe Ġpossibilities Ġfor ĠIN LINE FORM 3 Ġand ĠIN LINE FORM 4 Ġare Ġdefined Ġby Ġthe Ġont ology . ĠIn ĠD ST C 2 ĠIN LINE FORM 5 Ġcan Ġconsist Ġof Ġa Ġmaximum Ġof Ġthree Ġconstraints ; Ġfood , Ġarea Ġand Ġpr icer ange . ĠWhether Ġeach Ġof Ġthe Ġthree Ġis Ġpresent Ġis Ġindependently Ġsampled Ġwith Ġa Ġprobability Ġof Ġ0 . 66 , Ġ0 . 62 Ġand Ġ0 . 58 Ġrespectively . ĠThese Ġprobabilities Ġwere Ġestimated Ġfrom Ġthe ĠD ST C 2 Ġdata Ġset . ĠIf Ġno Ġconstraint Ġis Ġsampled Ġthen Ġthe Ġgoal Ġis Ġre - sam pled . ĠFor Ġeach Ġslot Ġin ĠIN LINE FORM 6 Ġa Ġvalue Ġ( e . g . Ġnorth Ġfor Ġarea ) Ġis Ġsampled Ġuniformly Ġfrom Ġthe Ġont ology . ĠSimilarly , Ġthe Ġpresence Ġof Ġa Ġrequest Ġis Ġindependently Ġsampled , Ġfollowed Ġby Ġre - sam pling Ġif Ġzero Ġrequests Ġwere Ġchosen . </s> When Ġtraining Ġthe Ġsequence - to - sequence Ġmodel , Ġthe ĠGoal ĠGenerator Ġis Ġnot Ġused , Ġbut Ġinstead Ġthe Ġgoal Ġlabels Ġfrom Ġthe ĠD ST C 2 Ġdataset Ġare Ġused . ĠIn ĠD ST C 2 Ġone Ġgoal - label Ġis Ġgiven Ġto Ġthe Ġentire Ġdialogue . ĠThis Ġgoal Ġis Ġalways Ġthe Ġfinal Ġgoal . ĠIf Ġthe Ġuser 's Ġgoal Ġat Ġthe Ġstart Ġof Ġthe Ġdialogue Ġis Ġ( food = er it re an , Ġarea = south ), Ġwhich Ġis Ġchanged Ġto Ġ( food = sp anish , Ġarea = south ), Ġdue Ġto Ġthe Ġnon - existence Ġof Ġan ĠErit re an Ġrestaurant Ġin Ġthe Ġsouth , Ġusing Ġonly Ġthe Ġfinal Ġgoal Ġis Ġinsufficient Ġto Ġmodel Ġthe Ġdialogue . ĠThe Ġfinal Ġgoal Ġcan Ġonly Ġbe Ġused Ġfor Ġthe Ġrequests Ġas Ġthey Ġare Ġnot Ġaltered Ġduring Ġa Ġdialogue . ĠD ST C 2 Ġalso Ġprovides Ġturn - specific Ġlabels . ĠThese Ġcontain Ġthe Ġconstraints Ġand Ġrequests Ġexpressed Ġby Ġthe Ġuser Ġup Ġuntil Ġand Ġincluding Ġthe Ġcurrent Ġturn . ĠWhen Ġtraining Ġa Ġpolicy Ġwith Ġthe ĠN US , Ġsuch Ġlabels Ġwould Ġnot Ġbe Ġavailable Ġas Ġthey ĠâĢ ľ p redict Ġthe Ġfuture ", Ġi . e . Ġwhen Ġthe Ġturn - specific Ġconstraints Ġchange Ġfrom Ġ( area = south ) Ġto Ġ( food = er it re an , Ġarea = south ) Ġit Ġmeans Ġthat Ġthe Ġuser Ġwill Ġinform Ġthe Ġsystem Ġabout Ġher Ġdesire Ġto Ġeat ĠErit re an Ġfood Ġin Ġthe Ġcurrent Ġturn . </s> In Ġrelated Ġwork Ġon Ġuser - sim ulation Ġfor Ġwhich Ġthe ĠD ST C 2 Ġdataset Ġwas Ġused , Ġthe Ġfinal Ġgoal Ġwas Ġused Ġfor Ġthe Ġentire Ġdialogue ĠB IB REF 4 Ġ, ĠB IB REF 33 Ġ, ĠB IB REF 34 Ġ. ĠAs Ġstated Ġabove , Ġwe Ġdo Ġnot Ġbelieve Ġthis Ġto Ġbe Ġsufficient . ĠThe Ġfollowing Ġdescribes Ġhow Ġto Ġupdate Ġthe Ġturn - specific Ġconstraint Ġlabels Ġsuch Ġthat Ġtheir Ġbehaviour Ġcan Ġbe Ġreplicated Ġwhen Ġtraining Ġa ĠDM 's Ġpolicy , Ġwhilst Ġallowing Ġgoal Ġchanges Ġto Ġbe Ġmod elled . ĠThe Ġupdate Ġstrategy Ġis Ġillustrated Ġin ĠTable ĠT AB REF 4 Ġwith Ġan Ġexample . ĠThe Ġfinal Ġturn Ġkeeps Ġits Ġconstraints , Ġfrom Ġwhich Ġwe Ġiter ate Ġbackwards Ġthrough Ġthe Ġlist Ġof ĠD ST C 2 's Ġturn - specific Ġconstraints . ĠThe Ġconstraints Ġof Ġa Ġturn Ġwill Ġbe Ġset Ġto Ġthe Ġupdated Ġconstraints Ġof Ġthe Ġsucceeding Ġturn , Ġbesides Ġif Ġthe Ġsame Ġslot Ġis Ġpresent Ġwith Ġa Ġdifferent Ġvalue . ĠIn Ġthat Ġcase Ġthe Ġvalue Ġwill Ġbe Ġkept . ĠThe Ġbehaviour Ġof Ġthe Ġupdated Ġturn - specific Ġgoal - lab els Ġcan Ġbe Ġreplicated Ġwhen Ġthe ĠN US Ġis Ġused Ġto Ġtrain Ġa ĠDM 's Ġpolicy . ĠIn Ġthe Ġexample , Ġthe Ġfood Ġtype Ġchanged Ġdue Ġto Ġthe ĠS DS Ġexpressing Ġthat Ġthere Ġis Ġno Ġrestaurant Ġserving ĠErit re an Ġfood Ġin Ġthe Ġsouth . ĠWhen Ġdeploying Ġthe ĠN US Ġto Ġtrain Ġa Ġpolicy , Ġthe Ġgoal Ġis Ġupdated Ġwhen Ġthe ĠS DS Ġoutputs Ġthe Ġcant help Ġdialogue Ġact . </s> Feature ĠExtract or </s> The ĠFeature ĠExtract or Ġgenerates Ġthe Ġfeature Ġvector Ġthat Ġis Ġapp ended Ġto Ġthe Ġsequence Ġof Ġfeature Ġvectors , Ġhere Ġcalled ĠFeature ĠHistory , Ġthat Ġis Ġpassed Ġto Ġthe Ġsequence - to - sequence Ġmodel . ĠThe Ġinput Ġto Ġthe ĠFeature ĠExtract or Ġis Ġthe Ġoutput Ġof Ġthe ĠDM Ġand Ġthe Ġcurrent Ġgoal ĠIN LINE FORM 0 Ġ. ĠFurthermore , Ġas Ġindicated Ġin ĠFigure ĠFIG REF 2 Ġ, Ġthe ĠFeature ĠExtract or Ġkeeps Ġtrack Ġof Ġthe Ġcurrently Ġaccepted Ġvenue Ġas Ġwell Ġas Ġthe Ġcurrent Ġand Ġinitial Ġrequest - vector , Ġwhich Ġis Ġexplained Ġbelow . </s> The Ġfeature Ġvector ĠIN LINE FORM 0 Ġis Ġmade Ġup Ġof Ġfour Ġsub - ve ctors . ĠThe Ġmotivation Ġbehind Ġthe Ġway Ġin Ġwhich Ġthese Ġfour Ġvectors Ġwere Ġdesigned Ġis Ġto Ġprovide Ġan Ġembed ding Ġfor Ġthe Ġsystem Ġresponse Ġthat Ġpreserves Ġall Ġnecessary Ġvalue - independent Ġinformation . </s> The Ġfirst Ġvector , Ġmachine - act Ġvector ĠIN LINE FORM 0 Ġ, Ġenc odes Ġthe Ġdialogue Ġacts Ġof Ġthe Ġsystem Ġresponse Ġand Ġconsists Ġof Ġtwo Ġparts ; ĠIN LINE FORM 1 Ġ. ĠIN LINE FORM 2 Ġis Ġa Ġbinary Ġrepresentation Ġof Ġthe Ġsystem Ġdialogue Ġacts Ġpresent Ġin Ġthe Ġinput . ĠIts Ġlength Ġis Ġthus Ġthe Ġnumber Ġof Ġpossible Ġsystem Ġdialogue Ġacts . ĠIt Ġis Ġbinary Ġand Ġnot Ġone - hot Ġsince Ġin ĠD ST C 2 Ġmultiple Ġdialogue Ġacts Ġcan Ġbe Ġin Ġthe Ġsystem 's Ġresponse . ĠIN LINE FORM 3 Ġis Ġa Ġbinary Ġrepresentation Ġof Ġthe Ġslot Ġif Ġthe Ġdialogue Ġact Ġis Ġrequest Ġor Ġselect Ġand Ġif Ġit Ġis Ġinform Ġor Ġexpl - conf Ġtogether Ġwith Ġa Ġcorrect Ġslot - value Ġpair Ġfor Ġan Ġinform able Ġslot . ĠThe Ġlength Ġis Ġfour Ġtimes Ġthe Ġnumber Ġof Ġinform able Ġslots . ĠIN LINE FORM 4 Ġis Ġnecessary Ġdue Ġto Ġthe Ġdependence Ġof Ġthe Ġsentence Ġstructure Ġon Ġthe Ġexact Ġslot Ġmentioned Ġby Ġthe Ġsystem . ĠThe Ġutter ances Ġof Ġa Ġuser Ġin Ġresponse Ġto Ġrequest ( food ) Ġand Ġrequest ( area ) Ġare Ġoften Ġvery Ġdifferent . </s> The Ġsecond Ġvector , Ġrequest - vector ĠIN LINE FORM 0 Ġ, Ġis Ġa Ġbinary Ġrepresentation Ġof Ġthe Ġrequests Ġthat Ġhave Ġnot Ġyet Ġbeen Ġfulfilled . ĠIt 's Ġlength Ġis Ġthus Ġthe Ġnumber Ġof Ġrequest able Ġslots . ĠIn Ġcomparison Ġto Ġthe Ġother Ġthree Ġvectors Ġthe Ġfeature Ġextract or Ġneeds Ġto Ġremember Ġit Ġfor Ġthe Ġnext Ġturn . ĠAt Ġthe Ġstart Ġof Ġthe Ġdialogue Ġthe Ġindices Ġcorresponding Ġto Ġrequests Ġthat Ġare Ġin ĠIN LINE FORM 1 Ġare Ġset Ġto Ġ1 Ġand Ġthe Ġrest Ġto Ġ0 . ĠWhenever Ġthe Ġsystem Ġinforms Ġa Ġcertain Ġrequest Ġthe Ġcorresponding Ġindex Ġin ĠIN LINE FORM 2 Ġis Ġset Ġto Ġ0 . ĠWhen Ġa Ġnew Ġvenue Ġis Ġproposed ĠIN LINE FORM 3 Ġis Ġreset Ġto Ġthe Ġoriginal Ġrequest Ġvector , Ġwhich Ġis Ġwhy Ġthe ĠFeature ĠExtract or Ġkeeps Ġtrack Ġof Ġit . </s> The Ġthird Ġvector , Ġinconsistency - vector ĠIN LINE FORM 0 Ġ, Ġrepresents Ġthe Ġinconsistency Ġbetween Ġthe Ġsystem 's Ġresponse Ġand ĠIN LINE FORM 1 Ġ. ĠEvery Ġtime Ġa Ġslot Ġis Ġmentioned Ġby Ġthe Ġsystem , Ġwhen Ġdescribing Ġa Ġvenue Ġ( in form ) Ġor Ġconfirming Ġa Ġslot - value Ġpair Ġ( expl - conf Ġor Ġimpl - conf ), Ġthe Ġindices Ġcorresponding Ġto Ġthe Ġslots Ġthat Ġhave Ġbeen Ġmisunderstood Ġare Ġset Ġto Ġ1 . ĠThe Ġlength Ġof ĠIN LINE FORM 2 Ġis Ġthe Ġnumber Ġof Ġinform able Ġslots . ĠThis Ġvector Ġis Ġnecessary Ġin Ġorder Ġfor Ġthe ĠN US Ġto Ġcorrect Ġthe Ġsystem . </s> The Ġfourth Ġvector , ĠIN LINE FORM 0 Ġ, Ġis Ġa Ġbinary Ġrepresentation Ġof Ġthe Ġslots Ġthat Ġare Ġin Ġthe Ġconstraints ĠIN LINE FORM 1 Ġ. ĠIt 's Ġlength Ġis Ġthus Ġthe Ġnumber Ġof Ġinform able Ġslots . ĠThis Ġvector Ġis Ġnecessary Ġin Ġorder Ġfor Ġthe ĠN US Ġto Ġbe Ġable Ġto Ġinform Ġabout Ġits Ġpreferred Ġvenue . </s> Sequ ence - To - Sequ ence ĠModel </s> The Ġsequence - to - sequence Ġmodel Ġ( Figure ĠFIG REF 7 Ġ) Ġconsists Ġof Ġan ĠR NN Ġenc oder , Ġfollowed Ġby Ġa Ġfully - connect Ġlayer Ġand Ġan ĠR NN Ġdec oder . ĠAn ĠR NN Ġcan Ġbe Ġdefined Ġas : ĠDIS PLAY FORM 0 </s> At Ġtime - step ĠIN LINE FORM 0 Ġ, Ġan ĠR NN Ġuses Ġan Ġinput ĠIN LINE FORM 1 Ġand Ġan Ġinternal Ġstate ĠIN LINE FORM 2 Ġto Ġproduce Ġits Ġoutput ĠIN LINE FORM 3 Ġand Ġits Ġnew Ġinternal Ġstate ĠIN LINE FORM 4 Ġ. ĠA Ġspecific ĠR NN - design Ġis Ġusually Ġdefined Ġusing Ġmatrix Ġmultipl ications , Ġelement - wise Ġadditions Ġand Ġmultipl ications Ġas Ġwell Ġas Ġelement - wise Ġnon - linear Ġfunctions . ĠThere Ġare Ġa Ġplethora Ġof Ġdifferent ĠR NN Ġarchitectures Ġthat Ġcould Ġbe Ġused Ġand Ġexplored . ĠGiven Ġthat Ġsuch Ġexploration Ġis Ġnot Ġthe Ġfocus Ġof Ġthis Ġwork Ġa Ġsingle Ġlayer ĠL ST M ĠB IB REF 35 Ġis Ġused Ġfor Ġboth Ġthe ĠR NN Ġenc oder Ġand Ġdec oder . ĠThe Ġexact ĠL ST M Ġversion Ġused Ġin Ġthis Ġwork Ġuses Ġa Ġforget Ġgate Ġwithout Ġbias Ġand Ġdoes Ġnot Ġuse Ġpe ep - holes . </s> The Ġfirst ĠR NN Ġ( shown Ġas Ġwhite Ġblocks Ġin ĠFig . ĠFIG REF 7 Ġ) Ġtakes Ġone Ġfeature Ġvector ĠIN LINE FORM 0 Ġat Ġa Ġtime Ġas Ġits Ġinput Ġ( ĠIN LINE FORM 1 Ġ). ĠIf Ġthe Ġcurrent Ġdialogue Ġturn Ġis Ġturn ĠIN LINE FORM 2 Ġthen Ġthe Ġfinal Ġoutput Ġof Ġthe ĠR NN Ġenc oder Ġis Ġgiven Ġby ĠIN LINE FORM 3 Ġ, Ġwhich Ġis Ġpassed Ġthrough Ġa Ġfully - connected Ġlayer Ġ( shown Ġas Ġthe Ġlight - grey Ġblock ) Ġwith Ġlinear Ġactivation Ġfunction : ĠDIS PLAY FORM 0 </s> For Ġa Ġcertain Ġencoding ĠIN LINE FORM 0 Ġthe Ġsequence - to - sequence Ġmodel Ġshould Ġdefine Ġa Ġprobability Ġdistribution Ġover Ġdifferent Ġsequences . ĠBy Ġsampling Ġfrom Ġthis Ġdistribution Ġthe ĠN US Ġcan Ġgenerate Ġa Ġdiverse Ġset Ġof Ġsentences Ġcorresponding Ġto Ġthe Ġsame Ġdialogue Ġcontext . ĠThe Ġconditional Ġprobability Ġdistribution Ġof Ġa Ġlength ĠIN LINE FORM 1 Ġsequence Ġis Ġdefined Ġas : ĠDIS PLAY FORM 0 </s> The Ġdec oder ĠR NN Ġ( shown Ġas Ġdark Ġblocks ) Ġwill Ġbe Ġused Ġto Ġmodel ĠIN LINE FORM 0 Ġ. ĠIt 's Ġinput Ġat Ġeach Ġtime - step Ġis Ġthe Ġconc aten ation Ġof Ġan Ġembed ding ĠIN LINE FORM 1 Ġ( we Ġused Ġ1 - hot ) Ġof Ġthe Ġprevious Ġword ĠIN LINE FORM 2 Ġ( ĠIN LINE FORM 3 Ġ). ĠFor ĠIN LINE FORM 4 Ġa Ġstart - of - sent ence Ġ(< S OS >) Ġtoken Ġis Ġused Ġas ĠIN LINE FORM 5 Ġ. ĠThe Ġend Ġof Ġthe Ġutter ance Ġis Ġmod elled Ġusing Ġan Ġend - of - sent ence Ġ(< E OS >) Ġtoken . ĠWhen Ġthe Ġdec oder ĠR NN Ġgenerates Ġthe Ġend - of - sent ence Ġtoken , Ġthe Ġdecoding Ġprocess Ġis Ġterminated . ĠThe Ġoutput Ġof Ġthe Ġdec oder ĠR NN , ĠIN LINE FORM 6 Ġ, Ġis Ġpassed Ġthrough Ġan Ġaff ine Ġtransform Ġfollowed Ġby Ġthe Ġsoft max Ġfunction , ĠSM , Ġto Ġform ĠIN LINE FORM 7 Ġ. ĠA Ġword ĠIN LINE FORM 8 Ġcan Ġbe Ġobtained Ġby Ġeither Ġtaking Ġthe Ġword Ġwith Ġthe Ġhighest Ġprobability Ġor Ġsampling Ġfrom Ġthe Ġdistribution : ĠDIS PLAY FORM 0 </s> During Ġtraining Ġthe Ġwords Ġare Ġnot Ġsampled Ġfrom Ġthe Ġoutput Ġdistribution , Ġbut Ġinstead Ġthe Ġtrue Ġwords Ġfrom Ġthe Ġdataset Ġare Ġused . ĠThis Ġa Ġcommon Ġtechnique Ġthat Ġis Ġoften Ġreferred Ġto Ġas Ġteacher - forcing , Ġthough Ġit Ġalso Ġdirectly Ġfollows Ġfrom Ġequation ĠEQ REF 10 Ġ. </s> To Ġgenerate Ġa Ġsequence Ġusing Ġan ĠR NN , Ġbeam - search Ġis Ġoften Ġused . ĠUsing Ġbeam - search Ġwith ĠIN LINE FORM 0 Ġbeams , Ġthe Ġwords Ġcorresponding Ġto Ġthe Ġtop ĠIN LINE FORM 1 Ġprobabilities Ġof ĠIN LINE FORM 2 Ġare Ġthe Ġfirst ĠIN LINE FORM 3 Ġbeams . ĠFor Ġeach Ġsucceeding ĠIN LINE FORM 4 Ġ, Ġthe ĠIN LINE FORM 5 Ġwords Ġcorresponding Ġto Ġthe Ġtop ĠIN LINE FORM 6 Ġprobabilities Ġof ĠIN LINE FORM 7 Ġare Ġtaken Ġfor Ġeach Ġof Ġthe ĠIN LINE FORM 8 Ġbeams . ĠThis Ġis Ġfollowed Ġby Ġreducing Ġthe Ġnumber Ġof Ġbeams Ġfrom Ġnow ĠIN LINE FORM 9 Ġdown Ġto ĠIN LINE FORM 10 Ġ, Ġby Ġtaking Ġthe ĠIN LINE FORM 11 Ġbeams Ġwith Ġthe Ġhighest Ġprobability ĠIN LINE FORM 12 Ġ. ĠThis Ġis Ġa Ġdeter ministic Ġprocess . ĠHowever , Ġfor Ġthe ĠN US Ġto Ġalways Ġgive Ġthe Ġsame Ġresponse Ġin Ġthe Ġsame Ġcontext Ġis Ġnot Ġrealistic . ĠThus , Ġthe ĠN US Ġcannot Ġcover Ġthe Ġfull Ġbreadth Ġof Ġuser Ġbehaviour Ġif Ġbeam - search Ġis Ġused . ĠTo Ġsolve Ġthis Ġissue Ġwhile Ġkeeping Ġthe Ġbenefit Ġof Ġrejecting Ġsequences Ġwith Ġlow Ġprobability , Ġa Ġtype Ġof Ġbeam - search Ġwith Ġsampling Ġis Ġused . ĠThe Ġprocess Ġis Ġidentical Ġto Ġthe Ġabove , Ġbut ĠIN LINE FORM 13 Ġwords Ġper Ġbeam Ġare Ġsampled Ġfrom Ġthe Ġprobability Ġdistribution . ĠThe ĠN US Ġis Ġnow Ġnon - d eter ministic Ġresulting Ġin Ġa Ġdiverse ĠUS . ĠUsing Ġ2 Ġbeams Ġgave Ġa Ġgood Ġtrade - off Ġbetween Ġreasonable Ġresponses Ġand Ġdiversity . </s> Training </s> The Ġneural Ġsequence - to - sequence Ġmodel Ġis Ġtrained Ġto Ġmaximize Ġthe Ġlog Ġprobability Ġthat Ġit Ġassigns Ġto Ġthe Ġuser Ġutter ances Ġof Ġthe Ġtraining Ġdata Ġset : ĠDIS PLAY FORM 0 </s> The Ġnetwork Ġwas Ġimplemented Ġin ĠT ensor flow ĠB IB REF 36 Ġand Ġoptimized Ġusing ĠT ensor flow 's Ġdefault Ġsetup Ġof Ġthe ĠAdam Ġoptim izer ĠB IB REF 37 Ġ. ĠThe ĠL ST M Ġlayers Ġand Ġthe Ġfully - connected Ġlayer Ġhad Ġwidth s Ġof Ġ100 Ġeach Ġto Ġgive Ġa Ġreasonable Ġnumber Ġof Ġoverall Ġparameters . ĠThe Ġwidth Ġwas Ġnot Ġtuned . ĠThe Ġlearning Ġrate Ġwas Ġoptim ised Ġon Ġa Ġheld Ġout Ġvalidation Ġset Ġand Ġno Ġregular ization Ġmethods Ġused . ĠThe Ġtraining Ġset Ġwas Ġshuff led Ġat Ġthe Ġdialogue Ġturn Ġlevel . </s> The Ġmanual Ġtranscript ions Ġof Ġthe ĠD ST C 2 Ġtraining Ġset Ġ( not Ġthe ĠAS R Ġoutput ) Ġwere Ġused Ġto Ġtrain Ġthe Ġsequence - to - sequence Ġmodel . ĠSince Ġthe Ġtranscript ions Ġwere Ġdone Ġmanually Ġthey Ġcontained Ġspelling Ġerrors . ĠThese Ġwere Ġmanually Ġcorrected Ġto Ġensure Ġproper Ġde lex ical ization . ĠSome Ġdialog ues Ġwere Ġdiscarded Ġdue Ġto Ġtranscript ions Ġerrors Ġbeing Ġtoo Ġlarge . ĠAfter Ġcleaning Ġthe Ġdataset Ġthe Ġtraining Ġset Ġconsisted Ġof Ġ16 09 Ġdialog ues Ġwith Ġa Ġtotal Ġof Ġ116 38 Ġdialogue Ġturns . ĠThe Ġvalidation Ġset Ġhad Ġ505 Ġdialog ues Ġwith Ġ38 96 Ġdialogue Ġturns . ĠThe Ġmaximum Ġsequence Ġlength Ġof Ġthe Ġde lex ical ized Ġturns Ġwas Ġ22 , Ġincluding Ġthe Ġend Ġof Ġsentence Ġcharacter . ĠThe Ġmaximum Ġdialogue Ġlength Ġwas Ġ30 Ġturns . </s> All Ġdialogue Ġpolicies Ġwere Ġtrained Ġwith Ġthe ĠPy Dial Ġtool kit ĠB IB REF 38 Ġ, Ġby Ġinteracting Ġwith Ġeither Ġthe ĠN US Ġor ĠAB US . ĠThe ĠRL Ġalgorithm Ġused Ġis ĠGP - S AR SA ĠB IB REF 3 Ġwith Ġhyper param eters Ġtaken Ġfrom ĠB IB REF 39 Ġ. ĠThe Ġreward Ġfunction Ġused Ġgives Ġa Ġreward Ġof Ġ20 Ġto Ġa Ġsuccessfully Ġcompleted Ġdialogue Ġand Ġof Ġ- 1 Ġfor Ġeach Ġdialogue Ġturn . ĠThe Ġmaximum Ġdialogue Ġlength Ġwas Ġ25 Ġturns . ĠThe Ġpresented Ġmetrics Ġare Ġsuccess Ġrate Ġ( SR ) Ġand Ġaverage Ġreward Ġover Ġtest Ġdialog ues . ĠSR Ġis Ġthe Ġpercentage Ġof Ġdialog ues Ġfor Ġwhich Ġthe Ġsystem Ġsatisfied Ġboth Ġthe Ġuser 's Ġconstraints Ġand Ġrequests . ĠThe Ġfinal Ġgoal , Ġafter Ġpossible Ġgoal Ġchanges , Ġwas Ġused Ġfor Ġthis Ġevaluation . ĠWhen Ġpolicies Ġare Ġtrained Ġusing Ġthe ĠN US , Ġits Ġoutput Ġis Ġparsed Ġusing ĠPy Dial 's Ġregular Ġexpression Ġbased Ġsemantic Ġdec oder . ĠThe Ġpolicies Ġwere Ġtrained Ġfor Ġ4000 Ġdialog ues . </s> Exper imental ĠSetup </s> The Ġevaluation Ġof Ġuser Ġsim ulators Ġis Ġan Ġongoing Ġarea Ġof Ġresearch Ġand Ġa Ġvariety Ġof Ġtechniques Ġcan Ġbe Ġfound Ġin Ġthe Ġliterature . ĠMost Ġpapers Ġpublished Ġon Ġuser Ġsimulation Ġevaluate Ġtheir ĠUS Ġusing Ġdirect Ġmethods . ĠThese Ġmethods Ġevaluate Ġthe ĠUS Ġthrough Ġa Ġstatistical Ġmeasure Ġof Ġsimilarity Ġbetween Ġthe Ġoutputs Ġof Ġthe ĠUS Ġand Ġa Ġreal Ġuser Ġon Ġa Ġtest Ġset . ĠMultiple Ġmodels Ġcan Ġoutper form Ġthe ĠAB US Ġon Ġthese Ġmetrics . ĠHowever , Ġthis Ġis Ġunsur prising Ġsince Ġthese Ġuser Ġsim ulators Ġwere Ġtrained Ġon Ġthe Ġsame Ġor Ġsimilar Ġmetrics . ĠThe ĠAB US Ġwas Ġexplicitly Ġproposed Ġas Ġa Ġtool Ġto Ġtrain Ġthe Ġpolicy Ġof Ġa Ġdialogue Ġmanager Ġand Ġit Ġis Ġstill Ġthe Ġdominant Ġform Ġof ĠUS Ġused Ġfor Ġthis Ġtask . ĠTherefore , Ġthe Ġonly Ġfair Ġcomparison Ġbetween Ġa Ġnew ĠUS Ġmodel Ġand Ġthe ĠAB US Ġis Ġto Ġuse Ġthe Ġindirect Ġmethod Ġof Ġevaluating Ġthe Ġpolicies Ġthat Ġwere Ġobtained Ġby Ġtraining Ġwith Ġeach ĠUS . </s> Testing Ġwith Ġa Ġsimulated Ġuser </s> In ĠSch atz mann Ġet . Ġal Ġsch atz t mann 2005 effects Ġcross - model Ġevaluation Ġis Ġproposed Ġto Ġcompare Ġuser Ġsim ulators . ĠFirst , Ġthe Ġuser Ġsim ulators Ġto Ġbe Ġevaluated Ġare Ġused Ġto Ġtrain ĠIN LINE FORM 0 Ġpolicy Ġeach . ĠThen Ġthese Ġpolicies Ġare Ġtested Ġusing Ġthe Ġdifferent Ġuser Ġsim ulators Ġand Ġthe Ġresults Ġaveraged . ĠB IB REF 7 Ġshowed Ġthat Ġa Ġstrategy Ġlearned Ġwith Ġa Ġgood Ġuser Ġmodel Ġstill Ġperforms Ġwell Ġwhen Ġtested Ġon Ġpoor Ġuser Ġmodels . ĠIf Ġa Ġpolicy Ġperforms Ġwell Ġon Ġall Ġuser Ġsim ulators Ġand Ġnot Ġjust Ġon Ġthe Ġone Ġthat Ġit Ġwas Ġtrained Ġon , Ġit Ġindicates Ġthat Ġthe ĠUS Ġwith Ġwhich Ġit Ġwas Ġtrained Ġis Ġdiverse Ġand Ġrealistic , Ġand Ġthus Ġthe Ġpolicy Ġis Ġlikely Ġto Ġperform Ġbetter Ġon Ġreal Ġusers . ĠFor Ġeach ĠUS Ġfive Ġpolicies Ġ( ĠIN LINE FORM 1 Ġ), Ġeach Ġusing Ġa Ġdifferent Ġrandom Ġseed Ġfor Ġinitial isation , Ġare Ġtrained . ĠResults Ġare Ġreported Ġfor Ġboth Ġthe Ġbest Ġand Ġthe Ġaverage Ġperformance Ġon Ġ1000 Ġtest Ġdialog ues . ĠThe ĠAB US Ġis Ġprogrammed Ġto Ġalways Ġmention Ġthe Ġnew Ġgoal Ġafter Ġa Ġgoal Ġchange . ĠIn Ġorder Ġto Ġnot Ġlet Ġthis Ġaffect Ġour Ġresults Ġwe Ġimplement Ġthe Ġsame Ġfor Ġthe ĠN US Ġby Ġre - sam pling Ġa Ġsentence Ġif Ġthe Ġnew Ġgoal Ġis Ġnot Ġmentioned . </s> Testing Ġwith Ġreal Ġusers </s> Though Ġthe Ġabove Ġtest Ġis Ġalready Ġmore Ġindicative Ġof Ġpolicy Ġperformance Ġon Ġreal Ġusers Ġthan Ġmeasuring Ġstatistical Ġmetrics Ġof Ġuser Ġbehaviour , Ġa Ġbetter Ġtest Ġis Ġto Ġtest Ġwith Ġhuman Ġusers . ĠFor Ġthe Ġtest Ġon Ġhuman Ġusers , Ġtwo Ġpolicies Ġfor Ġeach ĠUS Ġthat Ġwas Ġused Ġfor Ġtraining Ġare Ġchosen Ġfrom Ġthe Ġfive Ġpolicies . ĠThe Ġfirst Ġpolicy Ġis Ġthe Ġone Ġthat Ġperformed Ġbest Ġwhen Ġtested Ġon Ġthe ĠN US . ĠThe Ġsecond Ġis Ġthe Ġone Ġthat Ġperformed Ġbest Ġwhen Ġtested Ġon Ġthe ĠAB US . ĠThis Ġchoice Ġof Ġpolicies Ġis Ġmotivated Ġby Ġa Ġtype Ġof Ġover fitting Ġto Ġbe Ġseen Ġin ĠSec . ĠSEC REF 17 Ġ. ĠThe Ġevaluation Ġof Ġthe Ġtrained Ġdialogue Ġpolicies Ġin Ġinteraction Ġwith Ġreal Ġusers Ġfollows Ġa Ġsimilar Ġset - up Ġto ĠB IB REF 40 Ġ. ĠUsers Ġare Ġrecruited Ġthrough Ġthe ĠAmazon ĠMechanical ĠTurk Ġ( AM T ) Ġservice . Ġ1000 Ġdialog ues Ġ( 250 Ġper Ġpolicy ) Ġwere Ġgathered . ĠThe Ġlearnt Ġpolicies Ġwere Ġincorporated Ġinto Ġan ĠS DS Ġpipeline Ġwith Ġa Ġcommercial ĠAS R Ġsystem . ĠThe ĠAM T Ġusers Ġwere Ġasked Ġto Ġfind Ġa Ġrestaurant Ġthat Ġmatches Ġcertain Ġconstraints Ġand Ġfind Ġcertain Ġrequests . ĠSubjects Ġwere Ġrandomly Ġallocated Ġto Ġone Ġof Ġthe Ġfour Ġanalysed Ġsystems . ĠAfter Ġeach Ġdialogue Ġthe Ġusers Ġwere Ġasked Ġwhether Ġthey Ġjudged Ġthe Ġdialogue Ġto Ġbe Ġsuccessful Ġor Ġnot Ġwhich Ġwas Ġthen Ġtranslated Ġto Ġthe Ġreward Ġmeasure . </s> Cross - Model ĠEvaluation </s> Table ĠT AB REF 18 Ġshows Ġthe Ġresults Ġof Ġthe Ġcross - model Ġevaluation Ġafter Ġ4000 Ġtraining Ġdialog ues . ĠThe Ġpolicies Ġtrained Ġwith Ġthe ĠN US Ġachieved Ġan Ġaverage Ġsuccess Ġrate Ġ( SR ) Ġof Ġ94 . 0 % Ġand Ġof Ġ96 . 6 % Ġwhen Ġtested Ġon Ġthe ĠAB US Ġand Ġthe ĠN US , Ġrespectively . ĠBy Ġcomparison , Ġthe Ġpolicies Ġtrained Ġwith Ġthe ĠAB US Ġachieved Ġaverage ĠSR s Ġof Ġ99 . 5 % Ġand Ġ45 . 5 % Ġrespectively . ĠThus , Ġtraining Ġwith Ġthe ĠN US Ġleads Ġto Ġpolicies Ġthat Ġcan Ġperform Ġwell Ġon Ġboth ĠUS s , Ġwhich Ġis Ġnot Ġthe Ġcase Ġfor Ġtraining Ġwith Ġthe ĠAB US . ĠFurthermore , Ġthe Ġbest ĠSR s Ġwhen Ġtested Ġon Ġthe ĠAB US Ġare Ġsimilar Ġat Ġ99 . 9 % Ġ( AB US ) Ġand Ġ99 . 8 % Ġ( N US ). ĠWhen Ġtested Ġon Ġthe ĠN US Ġthe Ġbest ĠSR s Ġwere Ġ71 . 5 % Ġ( AB US ) Ġand Ġ98 . 0 % Ġ( N US ). ĠThis Ġshows Ġthat Ġthe Ġbehaviour Ġof Ġthe ĠNeural ĠUser ĠSimulator Ġis Ġrealistic Ġand Ġdiverse Ġenough Ġto Ġtrain Ġpolicies Ġthat Ġcan Ġalso Ġperform Ġvery Ġwell Ġon Ġthe ĠAgenda - Based ĠUser ĠSimulator . </s> Of Ġthe Ġfive Ġpolicies , Ġfor Ġeach ĠUS , Ġthe Ġpolicy Ġperforming Ġbest Ġon Ġthe ĠN US Ġwas Ġnot Ġthe Ġbest Ġperforming Ġpolicy Ġon Ġthe ĠAB US . ĠThis Ġcould Ġindicate Ġthat Ġthe Ġpolicy ĠâĢ ľ over fits âĢ Ŀ Ġto Ġa Ġparticular Ġuser Ġsimulator . ĠOver fitting Ġusually Ġmanifests Ġitself Ġin Ġworse Ġresults Ġas Ġthe Ġmodel Ġis Ġtrained Ġfor Ġlonger . ĠFive Ġpolicies Ġtrained Ġon Ġeach ĠUS Ġfor Ġonly Ġ1000 Ġdialog ues Ġwere Ġalso Ġevaluated , Ġthe Ġresults Ġof Ġwhich Ġcan Ġbe Ġseen Ġin ĠTable ĠT AB REF 19 Ġ. ĠAfter Ġtraining Ġfor Ġ1000 Ġdialog ues , Ġthe Ġaverage ĠSR Ġof Ġthe Ġpolicies Ġtrained Ġon Ġthe ĠN US Ġwhen Ġtested Ġon Ġthe ĠAB US Ġwas Ġ97 . 3 % Ġin Ġcomparison Ġto Ġ94 . 0 % Ġafter Ġ4000 Ġdialog ues . ĠThis Ġbehaviour Ġwas Ġobserved Ġfor Ġall Ġfive Ġseeds , Ġwhich Ġindicates Ġthat Ġthe Ġpolicy Ġindeed Ġover fits Ġto Ġthe ĠN US . ĠFor Ġthe Ġpolicies Ġtrained Ġwith Ġthe ĠAB US Ġthis Ġwas Ġnot Ġobserved . ĠThis Ġcould Ġindicate Ġthat Ġthe Ġpolicy Ġcan Ġlearn Ġto Ġexploit Ġsome Ġof Ġthe Ġshortcomings Ġof Ġthe Ġtrained ĠN US . </s> Human ĠEvaluation </s> The Ġresults Ġof Ġthe Ġhuman Ġevaluation Ġare Ġshown Ġin ĠTable ĠT AB REF 21 Ġfor Ġ250 Ġdialog ues Ġper Ġpolicy . ĠIn ĠTable ĠT AB REF 21 Ġpolicies Ġare Ġmarked Ġusing Ġan ĠID Ġ( ĠIN LINE FORM 0 Ġ) Ġthat Ġtranslates Ġto Ġresults Ġin ĠTables ĠT AB REF 18 Ġand ĠT AB REF 19 Ġ. ĠBoth Ġpolicies Ġtrained Ġwith Ġthe ĠN US Ġoutper formed Ġthose Ġtrained Ġon Ġthe ĠAB US Ġin Ġterms Ġof Ġboth Ġreward Ġand Ġsuccess Ġrate . ĠThe Ġbest Ġperforming Ġpolicy Ġtrained Ġon Ġthe ĠN US Ġachieves Ġa Ġ93 . 4 % Ġsuccess Ġrate Ġand Ġ13 . 8 Ġaverage Ġrewards Ġwhilst Ġthe Ġbest Ġperforming Ġpolicy Ġtrained Ġwith Ġthe ĠAB US Ġachieves Ġonly Ġa Ġ90 . 0 % Ġsuccess Ġrate Ġand Ġ13 . 3 Ġaverage Ġreward . ĠThis Ġshows Ġthat Ġthe Ġgood Ġperformance Ġof Ġthe ĠN US Ġon Ġthe Ġcross - model Ġevaluation Ġtransfers Ġto Ġreal Ġusers . ĠFurthermore , Ġthe Ġover fitting Ġto Ġa Ġparticular ĠUS Ġis Ġalso Ġobserved Ġin Ġthe Ġreal Ġuser Ġevaluation . ĠFor Ġnot Ġonly Ġthe Ġpolicies Ġtrained Ġon Ġthe ĠN US , Ġbut Ġalso Ġthose Ġtrained Ġon Ġthe ĠAB US , Ġthe Ġbest Ġperforming Ġpolicy Ġwas Ġthe Ġpolicy Ġthat Ġperformed Ġbest Ġon Ġthe Ġother ĠUS . </s> Conclusion </s> We Ġintroduced Ġthe ĠNeural ĠUser ĠSimulator Ġ( N US ), Ġwhich Ġuses Ġthe Ġsystem 's Ġresponse Ġin Ġits Ġsemantic Ġform Ġas Ġinput Ġand Ġgives Ġa Ġnatural Ġlanguage Ġresponse . ĠIt Ġthus Ġneeds Ġless Ġlab elling Ġof Ġthe Ġtraining Ġdata Ġthan ĠUser ĠSim ulators Ġthat Ġgenerate Ġa Ġresponse Ġin Ġsemantic Ġform . ĠIt Ġwas Ġshown Ġthat Ġthe ĠN US Ġlearns Ġrealistic Ġuser Ġbehaviour Ġfrom Ġa Ġcorpus Ġof Ġrecorded Ġdialog ues Ġsuch Ġthat Ġit Ġcan Ġbe Ġused Ġto Ġoptim ise Ġthe Ġpolicy Ġof Ġthe Ġdialogue Ġmanager Ġof Ġa Ġspoken Ġdialogue Ġsystem . ĠThe ĠN US Ġwas Ġcompared Ġto Ġthe ĠAgenda - Based ĠUser ĠSimulator Ġby Ġevaluating Ġpolicies Ġtrained Ġwith Ġthese Ġuser Ġsim ulators . ĠThe Ġtrained Ġpolicies Ġwere Ġcompared Ġboth Ġby Ġtesting Ġthem Ġwith Ġsimulated Ġusers Ġand Ġalso Ġwith Ġreal Ġusers . ĠThe ĠN US Ġexce lled Ġon Ġboth Ġevaluation Ġtasks . </s> Acknowled gements </s> This Ġresearch Ġwas Ġpartly Ġfunded Ġby Ġthe ĠEPS RC Ġgrant ĠEP / M 01 89 46 / 1 ĠOpen ĠDomain ĠStatistical ĠSp oken ĠDialogue ĠSystems . ĠFlor ian ĠK rey ss ig Ġis Ġsupported Ġby Ġthe ĠStud ien st ift ung Ġdes ĠDe ut sc hen ĠVol kes . ĠPaw e ÅĤ ĠBud z ian owski Ġis Ġsupported Ġby Ġthe ĠEPS RC Ġand ĠTos hiba ĠResearch ĠEurope ĠLtd .
Introduction Spoken Dialogue Systems (SDS) allow human-computer interaction using natural speech. Task-oriented dialogue systems, the focus of this work, help users achieve goals such as finding restaurants or booking flights BIBREF0 . Teaching a system how to respond appropriately in a task-oriented setting is non-trivial. In state-of-the-art systems this dialogue management task is often formulated as a reinforcement learning (RL) problem BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 . In this framework, the system learns by a trial and error process governed by a reward function. User Simulators can be used to train the policy of a dialogue manager (DM) without real user interactions. Furthermore, they allow an unlimited number of dialogues to be created with each dialogue being faster than a dialogue with a human. In this paper the Neural User Simulator (NUS) is introduced which outputs natural language and whose behaviour is learned from a corpus. The main component, inspired by BIBREF4 , consists of a feature extractor and a neural network based sequence-to-sequence model BIBREF5 . The sequence-to-sequence model consists of a recurrent neural network (RNN) encoder that encodes the dialogue history and a decoder RNN which outputs natural language. Furthermore, the NUS generates its own goal and possibly changes it during a dialogue. This allows the model to be deployed for training more sophisticated DM policies. To achieve this, a method is proposed that transforms the goal-labels of the used dataset (DSTC2) into labels whose behaviour can be replicated during deployment. The NUS is trained on dialogues between real users and an SDS in a restaurant recommendation domain. Compared to much of the related work on user simulation, we use the trained NUS to train the policy of a reinforcement learning based SDS. In order to evaluate the NUS, an Agenda-Based User-Simulator (ABUS) BIBREF6 is used to train another policy. The two policies are compared against each other by using cross-model evaluation BIBREF7 . This means to train on one model and to test on the other. Furthermore, both trained policies are tested on real users. On both evaluation tasks the NUS outperforms the ABUS, which is currently one of the most popular off-line training tools for reinforcement learning based Spoken Dialogue Systems BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14 . The remainder of this paper is organised as follows. Section 2 briefly describes task-oriented dialogue. Section 3 describes the motivation for the NUS and discusses related work. Section 4 explains the structure of the NUS, how it is trained and how it is deployed for training a DM's policy. Sections 5 and 6 present the experimental setup and results. Finally, Section 7 gives conclusions. Task-Oriented Dialogue A Task-Oriented SDS is typically designed according to a structured ontology, which defines what the system can talk about. In a system recommending restaurants the ontology defines those attributes of a restaurant that the user can choose, called informable slots (e.g. different food types, areas and price ranges), the attributes that the user can request, called requestable slots (e.g. phone number or address) and the restaurants that it has data about. An attribute is referred to as a slot and has a corresponding value. Together these are referred to as a slot-value pair (e.g. area=north). Using RL the DM is trained to act such that is maximises the cumulative future reward. The process by which the DM chooses its next action is called its policy. A typical approach to defining the reward function for a task-oriented SDS is to apply a small per-turn penalty to encourage short dialogues and to give a large positive reward at the end of each successful interaction. Motivation and Related Work Ideally the DM's policy would be trained by interacting with real users. Although there are models that support on-line learning BIBREF15 , for the majority of RL algorithms, which require a lot of interactions, this is impractical. Furthermore, a set of users needs to be recruited every time a policy is trained. This makes common practices such as hyper-parameter optimization prohibitively expensive. Thus, it is natural to try to learn from a dataset which needs to be recorded only once, but can be used over and over again. A problem with learning directly from recorded dialogue corpora is that the state space that was visited during the collection of the data is limited; the size of the recorded corpus usually falls short of the requirements for training a statistical DM. However, even if the size of the corpus is large enough the optimal dialogue strategy is likely not to be contained within it. A solution is to transform the static corpus into a dynamic tool: a user simulator. The user simulator (US) is trained on a dialogue corpus to learn what responses a real user would provide in a given dialogue context. The US is trained using supervised learning since the aim is for it to learn typical user behaviour. For the DM, however, we want optimal behaviour which is why supervised learning cannot be used. By interacting with the SDS, the trained US can be used to train the DM's policy. The DM's policy is optimised using the feedback given by either the user simulator or a separate evaluator. Any number of dialogues can be generated using the US and dialogue strategies that are not in the recorded corpus can be explored. Most user-simulators work on the level of user semantics. These usually consist of a user dialogue act (e.g. inform, or request) and a corresponding slot-value pair. The first statistical user simulator BIBREF16 used a simple bi-gram model INLINEFORM0 to predict the next user act INLINEFORM1 given the last system act INLINEFORM2 . It has the advantage of being purely probabilistic and domain-independent. However, it does not take the full dialogue history into account and is not conditioned on a goal, leading to incoherent user behaviour throughout a dialogue. BIBREF17 , BIBREF18 attempted to overcome goal inconsistency by proposing a graph-based model. However, developing the graph structure requires extensive domain-specific knowledge. BIBREF19 combined features from Sheffler and Young's work with Eckert's Model, by conditioning a set of probabilities on an explicit representation of the user goal and memory. A Markov Model is also used by BIBREF20 . It uses a large feature vector to describe the user's current state, which helps to compensate for the Markov assumption. However, the model is not conditioned on any goal. Therefore, it is not used to train a dialogue policy since it is impossible to determine whether the user goal was fulfilled. A hidden Markov model was proposed by BIBREF21 , which was also not used to train a policy. BIBREF22 cast user simulation as an inverse reinforcement learning problem where the user is modelled as a decision-making agent. The model did not incorporate a user goal and was hence not used to train a policy. The most prominent user model for policy optimisation is the Agenda-Based User Simulator BIBREF6 , which represents the user state elegantly as a stack of necessary user actions, called the agenda. The mechanism that generates the user response and updates the agenda does not require any data, though it can be improved using data. The model is conditioned on a goal for which it has update rules in case the dialogue system expresses that it cannot fulfil the goal. BIBREF4 modelled user simulation as a sequence-to-sequence task. The model can keep track of the dialogue history and user behaviour is learned entirely from data. However, goal changes were not modelled, even though a large proportion of dialogues within their dataset (DSTC2) contains goal changes. Their model outperformed the ABUS on statistical metrics, which is not surprising given that it was trained by optimising a statistical metric and the ABUS was not. The aforementioned work focuses on user simulation at the semantic level. Multiple issues arise from this approach. Firstly, annotating the user-response with the correct semantics is costly. More data could be collected, if the US were to output natural language. Secondly, research suggests that the two modules of an SDS performing Spoken Language Understanding (SLU) and belief tracking should be jointly trained as a single entity BIBREF23 , BIBREF24 , BIBREF25 , BIBREF26 , BIBREF27 . In fact in the second Dialogue State Tracking Challenge (DSTC2) BIBREF28 , the data of which this work uses, systems which used no external SLU module outperformed all systems that only used an external SLU Module. Training the policy of a DM in a simulated environment, when also using a joint system for SLU and belief tracking is not possible without a US that produces natural language. Thirdly, a US is sometimes augmented with an error model which generates a set of competing hypotheses with associated confidence scores trying to replicate the errors of the speech recogniser. When the error model matches the characteristics of the speech recogniser more accurately, the SDS performs better BIBREF29 . However, speech recognition errors are badly modelled based on user semantics since they arise (mostly) due to the phonetics of the spoken words and not their semantics BIBREF30 . Thus, an SDS that is trained with a natural language based error model is likely to outperform one trained with a semantic error model when tested on real users. Sequence-to-sequence learning for word-level user simulation is performed in BIBREF31 , though the model is not conditioned on any goal and hence not used for policy optimisation. A word-level user simulator was also used in BIBREF32 where it was built by augmenting the ABUS with a natural language generator. Neural User Simulator An overview of the NUS is given in Figure FIGREF2 . At the start of a dialogue a random goal INLINEFORM0 is generated by the Goal Generator. The possibilities for INLINEFORM1 are defined by the ontology. In dialogue turn INLINEFORM2 , the output of the SDS ( INLINEFORM3 ) is passed to the NUS's Feature Extractor, which generates a feature vector INLINEFORM4 based on INLINEFORM5 , the current user goal, INLINEFORM6 , and parts of the dialogue history. This vector is appended to the Feature History INLINEFORM7 . This sequence is passed to the sequence-to-sequence model (Fig. FIGREF7 ), which will generate the user's length INLINEFORM8 utterance INLINEFORM9 . As in Figure FIGREF7 , words in INLINEFORM10 corresponding to a slot are replaced by a slot token; a process called delexicalisation. If the SDS expresses to the NUS that there is no venue matching the NUS's constraints, the goal will be altered by the Goal Generator. Goal Generator The Goal Generator generates a random goal INLINEFORM0 at the start of the dialogue. It consists of a set of constraints, INLINEFORM1 , which specify the required venue e.g. (food=Spanish, area=north) and a number of requests, INLINEFORM2 , that specify the information that the NUS wants about the final venue e.g. the address or the phone number. The possibilities for INLINEFORM3 and INLINEFORM4 are defined by the ontology. In DSTC2 INLINEFORM5 can consist of a maximum of three constraints; food, area and pricerange. Whether each of the three is present is independently sampled with a probability of 0.66, 0.62 and 0.58 respectively. These probabilities were estimated from the DSTC2 data set. If no constraint is sampled then the goal is re-sampled. For each slot in INLINEFORM6 a value (e.g. north for area) is sampled uniformly from the ontology. Similarly, the presence of a request is independently sampled, followed by re-sampling if zero requests were chosen. When training the sequence-to-sequence model, the Goal Generator is not used, but instead the goal labels from the DSTC2 dataset are used. In DSTC2 one goal-label is given to the entire dialogue. This goal is always the final goal. If the user's goal at the start of the dialogue is (food=eritrean, area=south), which is changed to (food=spanish, area=south), due to the non-existence of an Eritrean restaurant in the south, using only the final goal is insufficient to model the dialogue. The final goal can only be used for the requests as they are not altered during a dialogue. DSTC2 also provides turn-specific labels. These contain the constraints and requests expressed by the user up until and including the current turn. When training a policy with the NUS, such labels would not be available as they “predict the future", i.e. when the turn-specific constraints change from (area=south) to (food=eritrean, area=south) it means that the user will inform the system about her desire to eat Eritrean food in the current turn. In related work on user-simulation for which the DSTC2 dataset was used, the final goal was used for the entire dialogue BIBREF4 , BIBREF33 , BIBREF34 . As stated above, we do not believe this to be sufficient. The following describes how to update the turn-specific constraint labels such that their behaviour can be replicated when training a DM's policy, whilst allowing goal changes to be modelled. The update strategy is illustrated in Table TABREF4 with an example. The final turn keeps its constraints, from which we iterate backwards through the list of DSTC2's turn-specific constraints. The constraints of a turn will be set to the updated constraints of the succeeding turn, besides if the same slot is present with a different value. In that case the value will be kept. The behaviour of the updated turn-specific goal-labels can be replicated when the NUS is used to train a DM's policy. In the example, the food type changed due to the SDS expressing that there is no restaurant serving Eritrean food in the south. When deploying the NUS to train a policy, the goal is updated when the SDS outputs the canthelp dialogue act. Feature Extractor The Feature Extractor generates the feature vector that is appended to the sequence of feature vectors, here called Feature History, that is passed to the sequence-to-sequence model. The input to the Feature Extractor is the output of the DM and the current goal INLINEFORM0 . Furthermore, as indicated in Figure FIGREF2 , the Feature Extractor keeps track of the currently accepted venue as well as the current and initial request-vector, which is explained below. The feature vector INLINEFORM0 is made up of four sub-vectors. The motivation behind the way in which these four vectors were designed is to provide an embedding for the system response that preserves all necessary value-independent information. The first vector, machine-act vector INLINEFORM0 , encodes the dialogue acts of the system response and consists of two parts; INLINEFORM1 . INLINEFORM2 is a binary representation of the system dialogue acts present in the input. Its length is thus the number of possible system dialogue acts. It is binary and not one-hot since in DSTC2 multiple dialogue acts can be in the system's response. INLINEFORM3 is a binary representation of the slot if the dialogue act is request or select and if it is inform or expl-conf together with a correct slot-value pair for an informable slot. The length is four times the number of informable slots. INLINEFORM4 is necessary due to the dependence of the sentence structure on the exact slot mentioned by the system. The utterances of a user in response to request(food) and request(area) are often very different. The second vector, request-vector INLINEFORM0 , is a binary representation of the requests that have not yet been fulfilled. It's length is thus the number of requestable slots. In comparison to the other three vectors the feature extractor needs to remember it for the next turn. At the start of the dialogue the indices corresponding to requests that are in INLINEFORM1 are set to 1 and the rest to 0. Whenever the system informs a certain request the corresponding index in INLINEFORM2 is set to 0. When a new venue is proposed INLINEFORM3 is reset to the original request vector, which is why the Feature Extractor keeps track of it. The third vector, inconsistency-vector INLINEFORM0 , represents the inconsistency between the system's response and INLINEFORM1 . Every time a slot is mentioned by the system, when describing a venue (inform) or confirming a slot-value pair (expl-conf or impl-conf), the indices corresponding to the slots that have been misunderstood are set to 1. The length of INLINEFORM2 is the number of informable slots. This vector is necessary in order for the NUS to correct the system. The fourth vector, INLINEFORM0 , is a binary representation of the slots that are in the constraints INLINEFORM1 . It's length is thus the number of informable slots. This vector is necessary in order for the NUS to be able to inform about its preferred venue. Sequence-To-Sequence Model The sequence-to-sequence model (Figure FIGREF7 ) consists of an RNN encoder, followed by a fully-connect layer and an RNN decoder. An RNN can be defined as: DISPLAYFORM0 At time-step INLINEFORM0 , an RNN uses an input INLINEFORM1 and an internal state INLINEFORM2 to produce its output INLINEFORM3 and its new internal state INLINEFORM4 . A specific RNN-design is usually defined using matrix multiplications, element-wise additions and multiplications as well as element-wise non-linear functions. There are a plethora of different RNN architectures that could be used and explored. Given that such exploration is not the focus of this work a single layer LSTM BIBREF35 is used for both the RNN encoder and decoder. The exact LSTM version used in this work uses a forget gate without bias and does not use peep-holes. The first RNN (shown as white blocks in Fig. FIGREF7 ) takes one feature vector INLINEFORM0 at a time as its input ( INLINEFORM1 ). If the current dialogue turn is turn INLINEFORM2 then the final output of the RNN encoder is given by INLINEFORM3 , which is passed through a fully-connected layer (shown as the light-grey block) with linear activation function: DISPLAYFORM0 For a certain encoding INLINEFORM0 the sequence-to-sequence model should define a probability distribution over different sequences. By sampling from this distribution the NUS can generate a diverse set of sentences corresponding to the same dialogue context. The conditional probability distribution of a length INLINEFORM1 sequence is defined as: DISPLAYFORM0 The decoder RNN (shown as dark blocks) will be used to model INLINEFORM0 . It's input at each time-step is the concatenation of an embedding INLINEFORM1 (we used 1-hot) of the previous word INLINEFORM2 ( INLINEFORM3 ). For INLINEFORM4 a start-of-sentence (<SOS>) token is used as INLINEFORM5 . The end of the utterance is modelled using an end-of-sentence (<EOS>) token. When the decoder RNN generates the end-of-sentence token, the decoding process is terminated. The output of the decoder RNN, INLINEFORM6 , is passed through an affine transform followed by the softmax function, SM, to form INLINEFORM7 . A word INLINEFORM8 can be obtained by either taking the word with the highest probability or sampling from the distribution: DISPLAYFORM0 During training the words are not sampled from the output distribution, but instead the true words from the dataset are used. This a common technique that is often referred to as teacher-forcing, though it also directly follows from equation EQREF10 . To generate a sequence using an RNN, beam-search is often used. Using beam-search with INLINEFORM0 beams, the words corresponding to the top INLINEFORM1 probabilities of INLINEFORM2 are the first INLINEFORM3 beams. For each succeeding INLINEFORM4 , the INLINEFORM5 words corresponding to the top INLINEFORM6 probabilities of INLINEFORM7 are taken for each of the INLINEFORM8 beams. This is followed by reducing the number of beams from now INLINEFORM9 down to INLINEFORM10 , by taking the INLINEFORM11 beams with the highest probability INLINEFORM12 . This is a deterministic process. However, for the NUS to always give the same response in the same context is not realistic. Thus, the NUS cannot cover the full breadth of user behaviour if beam-search is used. To solve this issue while keeping the benefit of rejecting sequences with low probability, a type of beam-search with sampling is used. The process is identical to the above, but INLINEFORM13 words per beam are sampled from the probability distribution. The NUS is now non-deterministic resulting in a diverse US. Using 2 beams gave a good trade-off between reasonable responses and diversity. Training The neural sequence-to-sequence model is trained to maximize the log probability that it assigns to the user utterances of the training data set: DISPLAYFORM0 The network was implemented in Tensorflow BIBREF36 and optimized using Tensorflow's default setup of the Adam optimizer BIBREF37 . The LSTM layers and the fully-connected layer had widths of 100 each to give a reasonable number of overall parameters. The width was not tuned. The learning rate was optimised on a held out validation set and no regularization methods used. The training set was shuffled at the dialogue turn level. The manual transcriptions of the DSTC2 training set (not the ASR output) were used to train the sequence-to-sequence model. Since the transcriptions were done manually they contained spelling errors. These were manually corrected to ensure proper delexicalization. Some dialogues were discarded due to transcriptions errors being too large. After cleaning the dataset the training set consisted of 1609 dialogues with a total of 11638 dialogue turns. The validation set had 505 dialogues with 3896 dialogue turns. The maximum sequence length of the delexicalized turns was 22, including the end of sentence character. The maximum dialogue length was 30 turns. All dialogue policies were trained with the PyDial toolkit BIBREF38 , by interacting with either the NUS or ABUS. The RL algorithm used is GP-SARSA BIBREF3 with hyperparameters taken from BIBREF39 . The reward function used gives a reward of 20 to a successfully completed dialogue and of -1 for each dialogue turn. The maximum dialogue length was 25 turns. The presented metrics are success rate (SR) and average reward over test dialogues. SR is the percentage of dialogues for which the system satisfied both the user's constraints and requests. The final goal, after possible goal changes, was used for this evaluation. When policies are trained using the NUS, its output is parsed using PyDial's regular expression based semantic decoder. The policies were trained for 4000 dialogues. Experimental Setup The evaluation of user simulators is an ongoing area of research and a variety of techniques can be found in the literature. Most papers published on user simulation evaluate their US using direct methods. These methods evaluate the US through a statistical measure of similarity between the outputs of the US and a real user on a test set. Multiple models can outperform the ABUS on these metrics. However, this is unsurprising since these user simulators were trained on the same or similar metrics. The ABUS was explicitly proposed as a tool to train the policy of a dialogue manager and it is still the dominant form of US used for this task. Therefore, the only fair comparison between a new US model and the ABUS is to use the indirect method of evaluating the policies that were obtained by training with each US. Testing with a simulated user In Schatzmann et. al schatztmann2005effects cross-model evaluation is proposed to compare user simulators. First, the user simulators to be evaluated are used to train INLINEFORM0 policy each. Then these policies are tested using the different user simulators and the results averaged. BIBREF7 showed that a strategy learned with a good user model still performs well when tested on poor user models. If a policy performs well on all user simulators and not just on the one that it was trained on, it indicates that the US with which it was trained is diverse and realistic, and thus the policy is likely to perform better on real users. For each US five policies ( INLINEFORM1 ), each using a different random seed for initialisation, are trained. Results are reported for both the best and the average performance on 1000 test dialogues. The ABUS is programmed to always mention the new goal after a goal change. In order to not let this affect our results we implement the same for the NUS by re-sampling a sentence if the new goal is not mentioned. Testing with real users Though the above test is already more indicative of policy performance on real users than measuring statistical metrics of user behaviour, a better test is to test with human users. For the test on human users, two policies for each US that was used for training are chosen from the five policies. The first policy is the one that performed best when tested on the NUS. The second is the one that performed best when tested on the ABUS. This choice of policies is motivated by a type of overfitting to be seen in Sec. SECREF17 . The evaluation of the trained dialogue policies in interaction with real users follows a similar set-up to BIBREF40 . Users are recruited through the Amazon Mechanical Turk (AMT) service. 1000 dialogues (250 per policy) were gathered. The learnt policies were incorporated into an SDS pipeline with a commercial ASR system. The AMT users were asked to find a restaurant that matches certain constraints and find certain requests. Subjects were randomly allocated to one of the four analysed systems. After each dialogue the users were asked whether they judged the dialogue to be successful or not which was then translated to the reward measure. Cross-Model Evaluation Table TABREF18 shows the results of the cross-model evaluation after 4000 training dialogues. The policies trained with the NUS achieved an average success rate (SR) of 94.0% and of 96.6% when tested on the ABUS and the NUS, respectively. By comparison, the policies trained with the ABUS achieved average SRs of 99.5% and 45.5% respectively. Thus, training with the NUS leads to policies that can perform well on both USs, which is not the case for training with the ABUS. Furthermore, the best SRs when tested on the ABUS are similar at 99.9% (ABUS) and 99.8% (NUS). When tested on the NUS the best SRs were 71.5% (ABUS) and 98.0% (NUS). This shows that the behaviour of the Neural User Simulator is realistic and diverse enough to train policies that can also perform very well on the Agenda-Based User Simulator. Of the five policies, for each US, the policy performing best on the NUS was not the best performing policy on the ABUS. This could indicate that the policy “overfits” to a particular user simulator. Overfitting usually manifests itself in worse results as the model is trained for longer. Five policies trained on each US for only 1000 dialogues were also evaluated, the results of which can be seen in Table TABREF19 . After training for 1000 dialogues, the average SR of the policies trained on the NUS when tested on the ABUS was 97.3% in comparison to 94.0% after 4000 dialogues. This behaviour was observed for all five seeds, which indicates that the policy indeed overfits to the NUS. For the policies trained with the ABUS this was not observed. This could indicate that the policy can learn to exploit some of the shortcomings of the trained NUS. Human Evaluation The results of the human evaluation are shown in Table TABREF21 for 250 dialogues per policy. In Table TABREF21 policies are marked using an ID ( INLINEFORM0 ) that translates to results in Tables TABREF18 and TABREF19 . Both policies trained with the NUS outperformed those trained on the ABUS in terms of both reward and success rate. The best performing policy trained on the NUS achieves a 93.4% success rate and 13.8 average rewards whilst the best performing policy trained with the ABUS achieves only a 90.0% success rate and 13.3 average reward. This shows that the good performance of the NUS on the cross-model evaluation transfers to real users. Furthermore, the overfitting to a particular US is also observed in the real user evaluation. For not only the policies trained on the NUS, but also those trained on the ABUS, the best performing policy was the policy that performed best on the other US. Conclusion We introduced the Neural User Simulator (NUS), which uses the system's response in its semantic form as input and gives a natural language response. It thus needs less labelling of the training data than User Simulators that generate a response in semantic form. It was shown that the NUS learns realistic user behaviour from a corpus of recorded dialogues such that it can be used to optimise the policy of the dialogue manager of a spoken dialogue system. The NUS was compared to the Agenda-Based User Simulator by evaluating policies trained with these user simulators. The trained policies were compared both by testing them with simulated users and also with real users. The NUS excelled on both evaluation tasks. Acknowledgements This research was partly funded by the EPSRC grant EP/M018946/1 Open Domain Statistical Spoken Dialogue Systems. Florian Kreyssig is supported by the Studienstiftung des Deutschen Volkes. Paweł Budzianowski is supported by the EPSRC and Toshiba Research Europe Ltd.
what corpus is used to learn behavior?
a15bc19674d48cd9919ad1cf152bf49c88f4417d
what Ġcorpus Ġis Ġused Ġto Ġlearn Ġbehavior ?
D ST C 2
<s> what Ġcorpus Ġis Ġused Ġto Ġlearn Ġbehavior ? </s> Introduction </s> Sp oken ĠDialogue ĠSystems Ġ( S DS ) Ġallow Ġhuman - computer Ġinteraction Ġusing Ġnatural Ġspeech . ĠTask - oriented Ġdialogue Ġsystems , Ġthe Ġfocus Ġof Ġthis Ġwork , Ġhelp Ġusers Ġachieve Ġgoals Ġsuch Ġas Ġfinding Ġrestaurants Ġor Ġbooking Ġflights ĠB IB REF 0 Ġ. </s> Te aching Ġa Ġsystem Ġhow Ġto Ġrespond Ġappropriately Ġin Ġa Ġtask - oriented Ġsetting Ġis Ġnon - t riv ial . ĠIn Ġstate - of - the - art Ġsystems Ġthis Ġdialogue Ġmanagement Ġtask Ġis Ġoften Ġformulated Ġas Ġa Ġreinforcement Ġlearning Ġ( RL ) Ġproblem ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ. ĠIn Ġthis Ġframework , Ġthe Ġsystem Ġlearns Ġby Ġa Ġtrial Ġand Ġerror Ġprocess Ġgoverned Ġby Ġa Ġreward Ġfunction . ĠUser ĠSim ulators Ġcan Ġbe Ġused Ġto Ġtrain Ġthe Ġpolicy Ġof Ġa Ġdialogue Ġmanager Ġ( DM ) Ġwithout Ġreal Ġuser Ġinteractions . ĠFurthermore , Ġthey Ġallow Ġan Ġunlimited Ġnumber Ġof Ġdialog ues Ġto Ġbe Ġcreated Ġwith Ġeach Ġdialogue Ġbeing Ġfaster Ġthan Ġa Ġdialogue Ġwith Ġa Ġhuman . </s> In Ġthis Ġpaper Ġthe ĠNeural ĠUser ĠSimulator Ġ( N US ) Ġis Ġintroduced Ġwhich Ġoutputs Ġnatural Ġlanguage Ġand Ġwhose Ġbehaviour Ġis Ġlearned Ġfrom Ġa Ġcorpus . ĠThe Ġmain Ġcomponent , Ġinspired Ġby ĠB IB REF 4 Ġ, Ġconsists Ġof Ġa Ġfeature Ġextract or Ġand Ġa Ġneural Ġnetwork Ġbased Ġsequence - to - sequence Ġmodel ĠB IB REF 5 Ġ. ĠThe Ġsequence - to - sequence Ġmodel Ġconsists Ġof Ġa Ġrecurrent Ġneural Ġnetwork Ġ( R NN ) Ġenc oder Ġthat Ġenc odes Ġthe Ġdialogue Ġhistory Ġand Ġa Ġdec oder ĠR NN Ġwhich Ġoutputs Ġnatural Ġlanguage . ĠFurthermore , Ġthe ĠN US Ġgenerates Ġits Ġown Ġgoal Ġand Ġpossibly Ġchanges Ġit Ġduring Ġa Ġdialogue . ĠThis Ġallows Ġthe Ġmodel Ġto Ġbe Ġdeployed Ġfor Ġtraining Ġmore Ġsophisticated ĠDM Ġpolicies . ĠTo Ġachieve Ġthis , Ġa Ġmethod Ġis Ġproposed Ġthat Ġtransforms Ġthe Ġgoal - lab els Ġof Ġthe Ġused Ġdataset Ġ( D ST C 2 ) Ġinto Ġlabels Ġwhose Ġbehaviour Ġcan Ġbe Ġreplicated Ġduring Ġdeployment . </s> The ĠN US Ġis Ġtrained Ġon Ġdialog ues Ġbetween Ġreal Ġusers Ġand Ġan ĠS DS Ġin Ġa Ġrestaurant Ġrecommendation Ġdomain . ĠCompared Ġto Ġmuch Ġof Ġthe Ġrelated Ġwork Ġon Ġuser Ġsimulation , Ġwe Ġuse Ġthe Ġtrained ĠN US Ġto Ġtrain Ġthe Ġpolicy Ġof Ġa Ġreinforcement Ġlearning Ġbased ĠS DS . ĠIn Ġorder Ġto Ġevaluate Ġthe ĠN US , Ġan ĠAgenda - Based ĠUser - Sim ulator Ġ( AB US ) ĠB IB REF 6 Ġis Ġused Ġto Ġtrain Ġanother Ġpolicy . ĠThe Ġtwo Ġpolicies Ġare Ġcompared Ġagainst Ġeach Ġother Ġby Ġusing Ġcross - model Ġevaluation ĠB IB REF 7 Ġ. ĠThis Ġmeans Ġto Ġtrain Ġon Ġone Ġmodel Ġand Ġto Ġtest Ġon Ġthe Ġother . ĠFurthermore , Ġboth Ġtrained Ġpolicies Ġare Ġtested Ġon Ġreal Ġusers . ĠOn Ġboth Ġevaluation Ġtasks Ġthe ĠN US Ġoutper forms Ġthe ĠAB US , Ġwhich Ġis Ġcurrently Ġone Ġof Ġthe Ġmost Ġpopular Ġoff - line Ġtraining Ġtools Ġfor Ġreinforcement Ġlearning Ġbased ĠSp oken ĠDialogue ĠSystems ĠB IB REF 8 Ġ, ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ. </s> The Ġremainder Ġof Ġthis Ġpaper Ġis Ġorganised Ġas Ġfollows . ĠSection Ġ2 Ġbriefly Ġdescribes Ġtask - oriented Ġdialogue . ĠSection Ġ3 Ġdescribes Ġthe Ġmotivation Ġfor Ġthe ĠN US Ġand Ġdiscusses Ġrelated Ġwork . ĠSection Ġ4 Ġexplains Ġthe Ġstructure Ġof Ġthe ĠN US , Ġhow Ġit Ġis Ġtrained Ġand Ġhow Ġit Ġis Ġdeployed Ġfor Ġtraining Ġa ĠDM 's Ġpolicy . ĠSections Ġ5 Ġand Ġ6 Ġpresent Ġthe Ġexperimental Ġsetup Ġand Ġresults . ĠFinally , ĠSection Ġ7 Ġgives Ġconclusions . </s> Task - O ri ented ĠDialogue </s> A ĠTask - O ri ented ĠS DS Ġis Ġtypically Ġdesigned Ġaccording Ġto Ġa Ġstructured Ġont ology , Ġwhich Ġdefines Ġwhat Ġthe Ġsystem Ġcan Ġtalk Ġabout . ĠIn Ġa Ġsystem Ġrecommending Ġrestaurants Ġthe Ġont ology Ġdefines Ġthose Ġattributes Ġof Ġa Ġrestaurant Ġthat Ġthe Ġuser Ġcan Ġchoose , Ġcalled Ġinform able Ġslots Ġ( e . g . Ġdifferent Ġfood Ġtypes , Ġareas Ġand Ġprice Ġranges ), Ġthe Ġattributes Ġthat Ġthe Ġuser Ġcan Ġrequest , Ġcalled Ġrequest able Ġslots Ġ( e . g . Ġphone Ġnumber Ġor Ġaddress ) Ġand Ġthe Ġrestaurants Ġthat Ġit Ġhas Ġdata Ġabout . ĠAn Ġattribute Ġis Ġreferred Ġto Ġas Ġa Ġslot Ġand Ġhas Ġa Ġcorresponding Ġvalue . ĠTogether Ġthese Ġare Ġreferred Ġto Ġas Ġa Ġslot - value Ġpair Ġ( e . g . Ġarea = north ). </s> Using ĠRL Ġthe ĠDM Ġis Ġtrained Ġto Ġact Ġsuch Ġthat Ġis Ġmaxim ises Ġthe Ġcumulative Ġfuture Ġreward . ĠThe Ġprocess Ġby Ġwhich Ġthe ĠDM Ġchooses Ġits Ġnext Ġaction Ġis Ġcalled Ġits Ġpolicy . ĠA Ġtypical Ġapproach Ġto Ġdefining Ġthe Ġreward Ġfunction Ġfor Ġa Ġtask - oriented ĠS DS Ġis Ġto Ġapply Ġa Ġsmall Ġper - turn Ġpenalty Ġto Ġencourage Ġshort Ġdialog ues Ġand Ġto Ġgive Ġa Ġlarge Ġpositive Ġreward Ġat Ġthe Ġend Ġof Ġeach Ġsuccessful Ġinteraction . </s> Mot ivation Ġand ĠRelated ĠWork </s> Ide ally Ġthe ĠDM 's Ġpolicy Ġwould Ġbe Ġtrained Ġby Ġinteracting Ġwith Ġreal Ġusers . ĠAlthough Ġthere Ġare Ġmodels Ġthat Ġsupport Ġon - line Ġlearning ĠB IB REF 15 Ġ, Ġfor Ġthe Ġmajority Ġof ĠRL Ġalgorithms , Ġwhich Ġrequire Ġa Ġlot Ġof Ġinteractions , Ġthis Ġis Ġimpractical . ĠFurthermore , Ġa Ġset Ġof Ġusers Ġneeds Ġto Ġbe Ġrecruited Ġevery Ġtime Ġa Ġpolicy Ġis Ġtrained . ĠThis Ġmakes Ġcommon Ġpractices Ġsuch Ġas Ġhyper - param eter Ġoptimization Ġprohib itive ly Ġexpensive . ĠThus , Ġit Ġis Ġnatural Ġto Ġtry Ġto Ġlearn Ġfrom Ġa Ġdataset Ġwhich Ġneeds Ġto Ġbe Ġrecorded Ġonly Ġonce , Ġbut Ġcan Ġbe Ġused Ġover Ġand Ġover Ġagain . </s> A Ġproblem Ġwith Ġlearning Ġdirectly Ġfrom Ġrecorded Ġdialogue Ġcorpor a Ġis Ġthat Ġthe Ġstate Ġspace Ġthat Ġwas Ġvisited Ġduring Ġthe Ġcollection Ġof Ġthe Ġdata Ġis Ġlimited ; Ġthe Ġsize Ġof Ġthe Ġrecorded Ġcorpus Ġusually Ġfalls Ġshort Ġof Ġthe Ġrequirements Ġfor Ġtraining Ġa Ġstatistical ĠDM . ĠHowever , Ġeven Ġif Ġthe Ġsize Ġof Ġthe Ġcorpus Ġis Ġlarge Ġenough Ġthe Ġoptimal Ġdialogue Ġstrategy Ġis Ġlikely Ġnot Ġto Ġbe Ġcontained Ġwithin Ġit . </s> A Ġsolution Ġis Ġto Ġtransform Ġthe Ġstatic Ġcorpus Ġinto Ġa Ġdynamic Ġtool : Ġa Ġuser Ġsimulator . ĠThe Ġuser Ġsimulator Ġ( US ) Ġis Ġtrained Ġon Ġa Ġdialogue Ġcorpus Ġto Ġlearn Ġwhat Ġresponses Ġa Ġreal Ġuser Ġwould Ġprovide Ġin Ġa Ġgiven Ġdialogue Ġcontext . ĠThe ĠUS Ġis Ġtrained Ġusing Ġsupervised Ġlearning Ġsince Ġthe Ġaim Ġis Ġfor Ġit Ġto Ġlearn Ġtypical Ġuser Ġbehaviour . ĠFor Ġthe ĠDM , Ġhowever , Ġwe Ġwant Ġoptimal Ġbehaviour Ġwhich Ġis Ġwhy Ġsupervised Ġlearning Ġcannot Ġbe Ġused . ĠBy Ġinteracting Ġwith Ġthe ĠS DS , Ġthe Ġtrained ĠUS Ġcan Ġbe Ġused Ġto Ġtrain Ġthe ĠDM 's Ġpolicy . ĠThe ĠDM 's Ġpolicy Ġis Ġoptim ised Ġusing Ġthe Ġfeedback Ġgiven Ġby Ġeither Ġthe Ġuser Ġsimulator Ġor Ġa Ġseparate Ġeval u ator . ĠAny Ġnumber Ġof Ġdialog ues Ġcan Ġbe Ġgenerated Ġusing Ġthe ĠUS Ġand Ġdialogue Ġstrategies Ġthat Ġare Ġnot Ġin Ġthe Ġrecorded Ġcorpus Ġcan Ġbe Ġexplored . </s> Most Ġuser - sim ulators Ġwork Ġon Ġthe Ġlevel Ġof Ġuser Ġsemantics . ĠThese Ġusually Ġconsist Ġof Ġa Ġuser Ġdialogue Ġact Ġ( e . g . Ġinform , Ġor Ġrequest ) Ġand Ġa Ġcorresponding Ġslot - value Ġpair . ĠThe Ġfirst Ġstatistical Ġuser Ġsimulator ĠB IB REF 16 Ġused Ġa Ġsimple Ġbi - gram Ġmodel ĠIN LINE FORM 0 Ġto Ġpredict Ġthe Ġnext Ġuser Ġact ĠIN LINE FORM 1 Ġgiven Ġthe Ġlast Ġsystem Ġact ĠIN LINE FORM 2 Ġ. ĠIt Ġhas Ġthe Ġadvantage Ġof Ġbeing Ġpurely Ġprob abil istic Ġand Ġdomain - independent . ĠHowever , Ġit Ġdoes Ġnot Ġtake Ġthe Ġfull Ġdialogue Ġhistory Ġinto Ġaccount Ġand Ġis Ġnot Ġconditioned Ġon Ġa Ġgoal , Ġleading Ġto Ġinco herent Ġuser Ġbehaviour Ġthroughout Ġa Ġdialogue . ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġattempted Ġto Ġovercome Ġgoal Ġinconsistency Ġby Ġproposing Ġa Ġgraph - based Ġmodel . ĠHowever , Ġdeveloping Ġthe Ġgraph Ġstructure Ġrequires Ġextensive Ġdomain - specific Ġknowledge . ĠB IB REF 19 Ġcombined Ġfeatures Ġfrom ĠShe ff ler Ġand ĠYoung 's Ġwork Ġwith ĠEck ert 's ĠModel , Ġby Ġconditioning Ġa Ġset Ġof Ġprobabilities Ġon Ġan Ġexplicit Ġrepresentation Ġof Ġthe Ġuser Ġgoal Ġand Ġmemory . ĠA ĠMark ov ĠModel Ġis Ġalso Ġused Ġby ĠB IB REF 20 Ġ. ĠIt Ġuses Ġa Ġlarge Ġfeature Ġvector Ġto Ġdescribe Ġthe Ġuser 's Ġcurrent Ġstate , Ġwhich Ġhelps Ġto Ġcompensate Ġfor Ġthe ĠMark ov Ġassumption . ĠHowever , Ġthe Ġmodel Ġis Ġnot Ġconditioned Ġon Ġany Ġgoal . ĠTherefore , Ġit Ġis Ġnot Ġused Ġto Ġtrain Ġa Ġdialogue Ġpolicy Ġsince Ġit Ġis Ġimpossible Ġto Ġdetermine Ġwhether Ġthe Ġuser Ġgoal Ġwas Ġfulfilled . ĠA Ġhidden ĠMark ov Ġmodel Ġwas Ġproposed Ġby ĠB IB REF 21 Ġ, Ġwhich Ġwas Ġalso Ġnot Ġused Ġto Ġtrain Ġa Ġpolicy . ĠB IB REF 22 Ġcast Ġuser Ġsimulation Ġas Ġan Ġinverse Ġreinforcement Ġlearning Ġproblem Ġwhere Ġthe Ġuser Ġis Ġmod elled Ġas Ġa Ġdecision - making Ġagent . ĠThe Ġmodel Ġdid Ġnot Ġincorporate Ġa Ġuser Ġgoal Ġand Ġwas Ġhence Ġnot Ġused Ġto Ġtrain Ġa Ġpolicy . ĠThe Ġmost Ġprominent Ġuser Ġmodel Ġfor Ġpolicy Ġoptim isation Ġis Ġthe ĠAgenda - Based ĠUser ĠSimulator ĠB IB REF 6 Ġ, Ġwhich Ġrepresents Ġthe Ġuser Ġstate Ġeleg antly Ġas Ġa Ġstack Ġof Ġnecessary Ġuser Ġactions , Ġcalled Ġthe Ġagenda . ĠThe Ġmechanism Ġthat Ġgenerates Ġthe Ġuser Ġresponse Ġand Ġupdates Ġthe Ġagenda Ġdoes Ġnot Ġrequire Ġany Ġdata , Ġthough Ġit Ġcan Ġbe Ġimproved Ġusing Ġdata . ĠThe Ġmodel Ġis Ġconditioned Ġon Ġa Ġgoal Ġfor Ġwhich Ġit Ġhas Ġupdate Ġrules Ġin Ġcase Ġthe Ġdialogue Ġsystem Ġexpresses Ġthat Ġit Ġcannot Ġfulfil Ġthe Ġgoal . ĠB IB REF 4 Ġmod elled Ġuser Ġsimulation Ġas Ġa Ġsequence - to - sequence Ġtask . ĠThe Ġmodel Ġcan Ġkeep Ġtrack Ġof Ġthe Ġdialogue Ġhistory Ġand Ġuser Ġbehaviour Ġis Ġlearned Ġentirely Ġfrom Ġdata . ĠHowever , Ġgoal Ġchanges Ġwere Ġnot Ġmod elled , Ġeven Ġthough Ġa Ġlarge Ġproportion Ġof Ġdialog ues Ġwithin Ġtheir Ġdataset Ġ( D ST C 2 ) Ġcontains Ġgoal Ġchanges . ĠTheir Ġmodel Ġoutper formed Ġthe ĠAB US Ġon Ġstatistical Ġmetrics , Ġwhich Ġis Ġnot Ġsurprising Ġgiven Ġthat Ġit Ġwas Ġtrained Ġby Ġoptim ising Ġa Ġstatistical Ġmetric Ġand Ġthe ĠAB US Ġwas Ġnot . </s> The Ġaforementioned Ġwork Ġfocuses Ġon Ġuser Ġsimulation Ġat Ġthe Ġsemantic Ġlevel . ĠMultiple Ġissues Ġarise Ġfrom Ġthis Ġapproach . ĠFirstly , Ġannot ating Ġthe Ġuser - response Ġwith Ġthe Ġcorrect Ġsemantics Ġis Ġcostly . ĠMore Ġdata Ġcould Ġbe Ġcollected , Ġif Ġthe ĠUS Ġwere Ġto Ġoutput Ġnatural Ġlanguage . ĠSecondly , Ġresearch Ġsuggests Ġthat Ġthe Ġtwo Ġmodules Ġof Ġan ĠS DS Ġperforming ĠSp oken ĠLanguage ĠUnderstanding Ġ( SL U ) Ġand Ġbelief Ġtracking Ġshould Ġbe Ġjointly Ġtrained Ġas Ġa Ġsingle Ġentity ĠB IB REF 23 Ġ, ĠB IB REF 24 Ġ, ĠB IB REF 25 Ġ, ĠB IB REF 26 Ġ, ĠB IB REF 27 Ġ. ĠIn Ġfact Ġin Ġthe Ġsecond ĠDialogue ĠState ĠTracking ĠChallenge Ġ( D ST C 2 ) ĠB IB REF 28 Ġ, Ġthe Ġdata Ġof Ġwhich Ġthis Ġwork Ġuses , Ġsystems Ġwhich Ġused Ġno Ġexternal ĠSL U Ġmodule Ġoutper formed Ġall Ġsystems Ġthat Ġonly Ġused Ġan Ġexternal ĠSL U ĠModule . ĠTraining Ġthe Ġpolicy Ġof Ġa ĠDM Ġin Ġa Ġsimulated Ġenvironment , Ġwhen Ġalso Ġusing Ġa Ġjoint Ġsystem Ġfor ĠSL U Ġand Ġbelief Ġtracking Ġis Ġnot Ġpossible Ġwithout Ġa ĠUS Ġthat Ġproduces Ġnatural Ġlanguage . ĠThird ly , Ġa ĠUS Ġis Ġsometimes Ġaugmented Ġwith Ġan Ġerror Ġmodel Ġwhich Ġgenerates Ġa Ġset Ġof Ġcompeting Ġhypotheses Ġwith Ġassociated Ġconfidence Ġscores Ġtrying Ġto Ġreplicate Ġthe Ġerrors Ġof Ġthe Ġspeech Ġrecogn iser . ĠWhen Ġthe Ġerror Ġmodel Ġmatches Ġthe Ġcharacteristics Ġof Ġthe Ġspeech Ġrecogn iser Ġmore Ġaccurately , Ġthe ĠS DS Ġperforms Ġbetter ĠB IB REF 29 Ġ. ĠHowever , Ġspeech Ġrecognition Ġerrors Ġare Ġbadly Ġmod elled Ġbased Ġon Ġuser Ġsemantics Ġsince Ġthey Ġarise Ġ( mostly ) Ġdue Ġto Ġthe Ġphon etics Ġof Ġthe Ġspoken Ġwords Ġand Ġnot Ġtheir Ġsemantics ĠB IB REF 30 Ġ. ĠThus , Ġan ĠS DS Ġthat Ġis Ġtrained Ġwith Ġa Ġnatural Ġlanguage Ġbased Ġerror Ġmodel Ġis Ġlikely Ġto Ġoutper form Ġone Ġtrained Ġwith Ġa Ġsemantic Ġerror Ġmodel Ġwhen Ġtested Ġon Ġreal Ġusers . ĠSequence - to - sequence Ġlearning Ġfor Ġword - level Ġuser Ġsimulation Ġis Ġperformed Ġin ĠB IB REF 31 Ġ, Ġthough Ġthe Ġmodel Ġis Ġnot Ġconditioned Ġon Ġany Ġgoal Ġand Ġhence Ġnot Ġused Ġfor Ġpolicy Ġoptim isation . ĠA Ġword - level Ġuser Ġsimulator Ġwas Ġalso Ġused Ġin ĠB IB REF 32 Ġwhere Ġit Ġwas Ġbuilt Ġby Ġaugment ing Ġthe ĠAB US Ġwith Ġa Ġnatural Ġlanguage Ġgenerator . </s> Ne ural ĠUser ĠSimulator </s> An Ġoverview Ġof Ġthe ĠN US Ġis Ġgiven Ġin ĠFigure ĠFIG REF 2 Ġ. ĠAt Ġthe Ġstart Ġof Ġa Ġdialogue Ġa Ġrandom Ġgoal ĠIN LINE FORM 0 Ġis Ġgenerated Ġby Ġthe ĠGoal ĠGenerator . ĠThe Ġpossibilities Ġfor ĠIN LINE FORM 1 Ġare Ġdefined Ġby Ġthe Ġont ology . ĠIn Ġdialogue Ġturn ĠIN LINE FORM 2 Ġ, Ġthe Ġoutput Ġof Ġthe ĠS DS Ġ( ĠIN LINE FORM 3 Ġ) Ġis Ġpassed Ġto Ġthe ĠN US 's ĠFeature ĠExtract or , Ġwhich Ġgenerates Ġa Ġfeature Ġvector ĠIN LINE FORM 4 Ġbased Ġon ĠIN LINE FORM 5 Ġ, Ġthe Ġcurrent Ġuser Ġgoal , ĠIN LINE FORM 6 Ġ, Ġand Ġparts Ġof Ġthe Ġdialogue Ġhistory . ĠThis Ġvector Ġis Ġapp ended Ġto Ġthe ĠFeature ĠHistory ĠIN LINE FORM 7 Ġ. ĠThis Ġsequence Ġis Ġpassed Ġto Ġthe Ġsequence - to - sequence Ġmodel Ġ( Fig . ĠFIG REF 7 Ġ), Ġwhich Ġwill Ġgenerate Ġthe Ġuser 's Ġlength ĠIN LINE FORM 8 Ġutter ance ĠIN LINE FORM 9 Ġ. ĠAs Ġin ĠFigure ĠFIG REF 7 Ġ, Ġwords Ġin ĠIN LINE FORM 10 Ġcorresponding Ġto Ġa Ġslot Ġare Ġreplaced Ġby Ġa Ġslot Ġtoken ; Ġa Ġprocess Ġcalled Ġde lex ical isation . ĠIf Ġthe ĠS DS Ġexpresses Ġto Ġthe ĠN US Ġthat Ġthere Ġis Ġno Ġvenue Ġmatching Ġthe ĠN US 's Ġconstraints , Ġthe Ġgoal Ġwill Ġbe Ġaltered Ġby Ġthe ĠGoal ĠGenerator . </s> Goal ĠGenerator </s> The ĠGoal ĠGenerator Ġgenerates Ġa Ġrandom Ġgoal ĠIN LINE FORM 0 Ġat Ġthe Ġstart Ġof Ġthe Ġdialogue . ĠIt Ġconsists Ġof Ġa Ġset Ġof Ġconstraints , ĠIN LINE FORM 1 Ġ, Ġwhich Ġspecify Ġthe Ġrequired Ġvenue Ġe . g . Ġ( food = Spanish , Ġarea = north ) Ġand Ġa Ġnumber Ġof Ġrequests , ĠIN LINE FORM 2 Ġ, Ġthat Ġspecify Ġthe Ġinformation Ġthat Ġthe ĠN US Ġwants Ġabout Ġthe Ġfinal Ġvenue Ġe . g . Ġthe Ġaddress Ġor Ġthe Ġphone Ġnumber . ĠThe Ġpossibilities Ġfor ĠIN LINE FORM 3 Ġand ĠIN LINE FORM 4 Ġare Ġdefined Ġby Ġthe Ġont ology . ĠIn ĠD ST C 2 ĠIN LINE FORM 5 Ġcan Ġconsist Ġof Ġa Ġmaximum Ġof Ġthree Ġconstraints ; Ġfood , Ġarea Ġand Ġpr icer ange . ĠWhether Ġeach Ġof Ġthe Ġthree Ġis Ġpresent Ġis Ġindependently Ġsampled Ġwith Ġa Ġprobability Ġof Ġ0 . 66 , Ġ0 . 62 Ġand Ġ0 . 58 Ġrespectively . ĠThese Ġprobabilities Ġwere Ġestimated Ġfrom Ġthe ĠD ST C 2 Ġdata Ġset . ĠIf Ġno Ġconstraint Ġis Ġsampled Ġthen Ġthe Ġgoal Ġis Ġre - sam pled . ĠFor Ġeach Ġslot Ġin ĠIN LINE FORM 6 Ġa Ġvalue Ġ( e . g . Ġnorth Ġfor Ġarea ) Ġis Ġsampled Ġuniformly Ġfrom Ġthe Ġont ology . ĠSimilarly , Ġthe Ġpresence Ġof Ġa Ġrequest Ġis Ġindependently Ġsampled , Ġfollowed Ġby Ġre - sam pling Ġif Ġzero Ġrequests Ġwere Ġchosen . </s> When Ġtraining Ġthe Ġsequence - to - sequence Ġmodel , Ġthe ĠGoal ĠGenerator Ġis Ġnot Ġused , Ġbut Ġinstead Ġthe Ġgoal Ġlabels Ġfrom Ġthe ĠD ST C 2 Ġdataset Ġare Ġused . ĠIn ĠD ST C 2 Ġone Ġgoal - label Ġis Ġgiven Ġto Ġthe Ġentire Ġdialogue . ĠThis Ġgoal Ġis Ġalways Ġthe Ġfinal Ġgoal . ĠIf Ġthe Ġuser 's Ġgoal Ġat Ġthe Ġstart Ġof Ġthe Ġdialogue Ġis Ġ( food = er it re an , Ġarea = south ), Ġwhich Ġis Ġchanged Ġto Ġ( food = sp anish , Ġarea = south ), Ġdue Ġto Ġthe Ġnon - existence Ġof Ġan ĠErit re an Ġrestaurant Ġin Ġthe Ġsouth , Ġusing Ġonly Ġthe Ġfinal Ġgoal Ġis Ġinsufficient Ġto Ġmodel Ġthe Ġdialogue . ĠThe Ġfinal Ġgoal Ġcan Ġonly Ġbe Ġused Ġfor Ġthe Ġrequests Ġas Ġthey Ġare Ġnot Ġaltered Ġduring Ġa Ġdialogue . ĠD ST C 2 Ġalso Ġprovides Ġturn - specific Ġlabels . ĠThese Ġcontain Ġthe Ġconstraints Ġand Ġrequests Ġexpressed Ġby Ġthe Ġuser Ġup Ġuntil Ġand Ġincluding Ġthe Ġcurrent Ġturn . ĠWhen Ġtraining Ġa Ġpolicy Ġwith Ġthe ĠN US , Ġsuch Ġlabels Ġwould Ġnot Ġbe Ġavailable Ġas Ġthey ĠâĢ ľ p redict Ġthe Ġfuture ", Ġi . e . Ġwhen Ġthe Ġturn - specific Ġconstraints Ġchange Ġfrom Ġ( area = south ) Ġto Ġ( food = er it re an , Ġarea = south ) Ġit Ġmeans Ġthat Ġthe Ġuser Ġwill Ġinform Ġthe Ġsystem Ġabout Ġher Ġdesire Ġto Ġeat ĠErit re an Ġfood Ġin Ġthe Ġcurrent Ġturn . </s> In Ġrelated Ġwork Ġon Ġuser - sim ulation Ġfor Ġwhich Ġthe ĠD ST C 2 Ġdataset Ġwas Ġused , Ġthe Ġfinal Ġgoal Ġwas Ġused Ġfor Ġthe Ġentire Ġdialogue ĠB IB REF 4 Ġ, ĠB IB REF 33 Ġ, ĠB IB REF 34 Ġ. ĠAs Ġstated Ġabove , Ġwe Ġdo Ġnot Ġbelieve Ġthis Ġto Ġbe Ġsufficient . ĠThe Ġfollowing Ġdescribes Ġhow Ġto Ġupdate Ġthe Ġturn - specific Ġconstraint Ġlabels Ġsuch Ġthat Ġtheir Ġbehaviour Ġcan Ġbe Ġreplicated Ġwhen Ġtraining Ġa ĠDM 's Ġpolicy , Ġwhilst Ġallowing Ġgoal Ġchanges Ġto Ġbe Ġmod elled . ĠThe Ġupdate Ġstrategy Ġis Ġillustrated Ġin ĠTable ĠT AB REF 4 Ġwith Ġan Ġexample . ĠThe Ġfinal Ġturn Ġkeeps Ġits Ġconstraints , Ġfrom Ġwhich Ġwe Ġiter ate Ġbackwards Ġthrough Ġthe Ġlist Ġof ĠD ST C 2 's Ġturn - specific Ġconstraints . ĠThe Ġconstraints Ġof Ġa Ġturn Ġwill Ġbe Ġset Ġto Ġthe Ġupdated Ġconstraints Ġof Ġthe Ġsucceeding Ġturn , Ġbesides Ġif Ġthe Ġsame Ġslot Ġis Ġpresent Ġwith Ġa Ġdifferent Ġvalue . ĠIn Ġthat Ġcase Ġthe Ġvalue Ġwill Ġbe Ġkept . ĠThe Ġbehaviour Ġof Ġthe Ġupdated Ġturn - specific Ġgoal - lab els Ġcan Ġbe Ġreplicated Ġwhen Ġthe ĠN US Ġis Ġused Ġto Ġtrain Ġa ĠDM 's Ġpolicy . ĠIn Ġthe Ġexample , Ġthe Ġfood Ġtype Ġchanged Ġdue Ġto Ġthe ĠS DS Ġexpressing Ġthat Ġthere Ġis Ġno Ġrestaurant Ġserving ĠErit re an Ġfood Ġin Ġthe Ġsouth . ĠWhen Ġdeploying Ġthe ĠN US Ġto Ġtrain Ġa Ġpolicy , Ġthe Ġgoal Ġis Ġupdated Ġwhen Ġthe ĠS DS Ġoutputs Ġthe Ġcant help Ġdialogue Ġact . </s> Feature ĠExtract or </s> The ĠFeature ĠExtract or Ġgenerates Ġthe Ġfeature Ġvector Ġthat Ġis Ġapp ended Ġto Ġthe Ġsequence Ġof Ġfeature Ġvectors , Ġhere Ġcalled ĠFeature ĠHistory , Ġthat Ġis Ġpassed Ġto Ġthe Ġsequence - to - sequence Ġmodel . ĠThe Ġinput Ġto Ġthe ĠFeature ĠExtract or Ġis Ġthe Ġoutput Ġof Ġthe ĠDM Ġand Ġthe Ġcurrent Ġgoal ĠIN LINE FORM 0 Ġ. ĠFurthermore , Ġas Ġindicated Ġin ĠFigure ĠFIG REF 2 Ġ, Ġthe ĠFeature ĠExtract or Ġkeeps Ġtrack Ġof Ġthe Ġcurrently Ġaccepted Ġvenue Ġas Ġwell Ġas Ġthe Ġcurrent Ġand Ġinitial Ġrequest - vector , Ġwhich Ġis Ġexplained Ġbelow . </s> The Ġfeature Ġvector ĠIN LINE FORM 0 Ġis Ġmade Ġup Ġof Ġfour Ġsub - ve ctors . ĠThe Ġmotivation Ġbehind Ġthe Ġway Ġin Ġwhich Ġthese Ġfour Ġvectors Ġwere Ġdesigned Ġis Ġto Ġprovide Ġan Ġembed ding Ġfor Ġthe Ġsystem Ġresponse Ġthat Ġpreserves Ġall Ġnecessary Ġvalue - independent Ġinformation . </s> The Ġfirst Ġvector , Ġmachine - act Ġvector ĠIN LINE FORM 0 Ġ, Ġenc odes Ġthe Ġdialogue Ġacts Ġof Ġthe Ġsystem Ġresponse Ġand Ġconsists Ġof Ġtwo Ġparts ; ĠIN LINE FORM 1 Ġ. ĠIN LINE FORM 2 Ġis Ġa Ġbinary Ġrepresentation Ġof Ġthe Ġsystem Ġdialogue Ġacts Ġpresent Ġin Ġthe Ġinput . ĠIts Ġlength Ġis Ġthus Ġthe Ġnumber Ġof Ġpossible Ġsystem Ġdialogue Ġacts . ĠIt Ġis Ġbinary Ġand Ġnot Ġone - hot Ġsince Ġin ĠD ST C 2 Ġmultiple Ġdialogue Ġacts Ġcan Ġbe Ġin Ġthe Ġsystem 's Ġresponse . ĠIN LINE FORM 3 Ġis Ġa Ġbinary Ġrepresentation Ġof Ġthe Ġslot Ġif Ġthe Ġdialogue Ġact Ġis Ġrequest Ġor Ġselect Ġand Ġif Ġit Ġis Ġinform Ġor Ġexpl - conf Ġtogether Ġwith Ġa Ġcorrect Ġslot - value Ġpair Ġfor Ġan Ġinform able Ġslot . ĠThe Ġlength Ġis Ġfour Ġtimes Ġthe Ġnumber Ġof Ġinform able Ġslots . ĠIN LINE FORM 4 Ġis Ġnecessary Ġdue Ġto Ġthe Ġdependence Ġof Ġthe Ġsentence Ġstructure Ġon Ġthe Ġexact Ġslot Ġmentioned Ġby Ġthe Ġsystem . ĠThe Ġutter ances Ġof Ġa Ġuser Ġin Ġresponse Ġto Ġrequest ( food ) Ġand Ġrequest ( area ) Ġare Ġoften Ġvery Ġdifferent . </s> The Ġsecond Ġvector , Ġrequest - vector ĠIN LINE FORM 0 Ġ, Ġis Ġa Ġbinary Ġrepresentation Ġof Ġthe Ġrequests Ġthat Ġhave Ġnot Ġyet Ġbeen Ġfulfilled . ĠIt 's Ġlength Ġis Ġthus Ġthe Ġnumber Ġof Ġrequest able Ġslots . ĠIn Ġcomparison Ġto Ġthe Ġother Ġthree Ġvectors Ġthe Ġfeature Ġextract or Ġneeds Ġto Ġremember Ġit Ġfor Ġthe Ġnext Ġturn . ĠAt Ġthe Ġstart Ġof Ġthe Ġdialogue Ġthe Ġindices Ġcorresponding Ġto Ġrequests Ġthat Ġare Ġin ĠIN LINE FORM 1 Ġare Ġset Ġto Ġ1 Ġand Ġthe Ġrest Ġto Ġ0 . ĠWhenever Ġthe Ġsystem Ġinforms Ġa Ġcertain Ġrequest Ġthe Ġcorresponding Ġindex Ġin ĠIN LINE FORM 2 Ġis Ġset Ġto Ġ0 . ĠWhen Ġa Ġnew Ġvenue Ġis Ġproposed ĠIN LINE FORM 3 Ġis Ġreset Ġto Ġthe Ġoriginal Ġrequest Ġvector , Ġwhich Ġis Ġwhy Ġthe ĠFeature ĠExtract or Ġkeeps Ġtrack Ġof Ġit . </s> The Ġthird Ġvector , Ġinconsistency - vector ĠIN LINE FORM 0 Ġ, Ġrepresents Ġthe Ġinconsistency Ġbetween Ġthe Ġsystem 's Ġresponse Ġand ĠIN LINE FORM 1 Ġ. ĠEvery Ġtime Ġa Ġslot Ġis Ġmentioned Ġby Ġthe Ġsystem , Ġwhen Ġdescribing Ġa Ġvenue Ġ( in form ) Ġor Ġconfirming Ġa Ġslot - value Ġpair Ġ( expl - conf Ġor Ġimpl - conf ), Ġthe Ġindices Ġcorresponding Ġto Ġthe Ġslots Ġthat Ġhave Ġbeen Ġmisunderstood Ġare Ġset Ġto Ġ1 . ĠThe Ġlength Ġof ĠIN LINE FORM 2 Ġis Ġthe Ġnumber Ġof Ġinform able Ġslots . ĠThis Ġvector Ġis Ġnecessary Ġin Ġorder Ġfor Ġthe ĠN US Ġto Ġcorrect Ġthe Ġsystem . </s> The Ġfourth Ġvector , ĠIN LINE FORM 0 Ġ, Ġis Ġa Ġbinary Ġrepresentation Ġof Ġthe Ġslots Ġthat Ġare Ġin Ġthe Ġconstraints ĠIN LINE FORM 1 Ġ. ĠIt 's Ġlength Ġis Ġthus Ġthe Ġnumber Ġof Ġinform able Ġslots . ĠThis Ġvector Ġis Ġnecessary Ġin Ġorder Ġfor Ġthe ĠN US Ġto Ġbe Ġable Ġto Ġinform Ġabout Ġits Ġpreferred Ġvenue . </s> Sequ ence - To - Sequ ence ĠModel </s> The Ġsequence - to - sequence Ġmodel Ġ( Figure ĠFIG REF 7 Ġ) Ġconsists Ġof Ġan ĠR NN Ġenc oder , Ġfollowed Ġby Ġa Ġfully - connect Ġlayer Ġand Ġan ĠR NN Ġdec oder . ĠAn ĠR NN Ġcan Ġbe Ġdefined Ġas : ĠDIS PLAY FORM 0 </s> At Ġtime - step ĠIN LINE FORM 0 Ġ, Ġan ĠR NN Ġuses Ġan Ġinput ĠIN LINE FORM 1 Ġand Ġan Ġinternal Ġstate ĠIN LINE FORM 2 Ġto Ġproduce Ġits Ġoutput ĠIN LINE FORM 3 Ġand Ġits Ġnew Ġinternal Ġstate ĠIN LINE FORM 4 Ġ. ĠA Ġspecific ĠR NN - design Ġis Ġusually Ġdefined Ġusing Ġmatrix Ġmultipl ications , Ġelement - wise Ġadditions Ġand Ġmultipl ications Ġas Ġwell Ġas Ġelement - wise Ġnon - linear Ġfunctions . ĠThere Ġare Ġa Ġplethora Ġof Ġdifferent ĠR NN Ġarchitectures Ġthat Ġcould Ġbe Ġused Ġand Ġexplored . ĠGiven Ġthat Ġsuch Ġexploration Ġis Ġnot Ġthe Ġfocus Ġof Ġthis Ġwork Ġa Ġsingle Ġlayer ĠL ST M ĠB IB REF 35 Ġis Ġused Ġfor Ġboth Ġthe ĠR NN Ġenc oder Ġand Ġdec oder . ĠThe Ġexact ĠL ST M Ġversion Ġused Ġin Ġthis Ġwork Ġuses Ġa Ġforget Ġgate Ġwithout Ġbias Ġand Ġdoes Ġnot Ġuse Ġpe ep - holes . </s> The Ġfirst ĠR NN Ġ( shown Ġas Ġwhite Ġblocks Ġin ĠFig . ĠFIG REF 7 Ġ) Ġtakes Ġone Ġfeature Ġvector ĠIN LINE FORM 0 Ġat Ġa Ġtime Ġas Ġits Ġinput Ġ( ĠIN LINE FORM 1 Ġ). ĠIf Ġthe Ġcurrent Ġdialogue Ġturn Ġis Ġturn ĠIN LINE FORM 2 Ġthen Ġthe Ġfinal Ġoutput Ġof Ġthe ĠR NN Ġenc oder Ġis Ġgiven Ġby ĠIN LINE FORM 3 Ġ, Ġwhich Ġis Ġpassed Ġthrough Ġa Ġfully - connected Ġlayer Ġ( shown Ġas Ġthe Ġlight - grey Ġblock ) Ġwith Ġlinear Ġactivation Ġfunction : ĠDIS PLAY FORM 0 </s> For Ġa Ġcertain Ġencoding ĠIN LINE FORM 0 Ġthe Ġsequence - to - sequence Ġmodel Ġshould Ġdefine Ġa Ġprobability Ġdistribution Ġover Ġdifferent Ġsequences . ĠBy Ġsampling Ġfrom Ġthis Ġdistribution Ġthe ĠN US Ġcan Ġgenerate Ġa Ġdiverse Ġset Ġof Ġsentences Ġcorresponding Ġto Ġthe Ġsame Ġdialogue Ġcontext . ĠThe Ġconditional Ġprobability Ġdistribution Ġof Ġa Ġlength ĠIN LINE FORM 1 Ġsequence Ġis Ġdefined Ġas : ĠDIS PLAY FORM 0 </s> The Ġdec oder ĠR NN Ġ( shown Ġas Ġdark Ġblocks ) Ġwill Ġbe Ġused Ġto Ġmodel ĠIN LINE FORM 0 Ġ. ĠIt 's Ġinput Ġat Ġeach Ġtime - step Ġis Ġthe Ġconc aten ation Ġof Ġan Ġembed ding ĠIN LINE FORM 1 Ġ( we Ġused Ġ1 - hot ) Ġof Ġthe Ġprevious Ġword ĠIN LINE FORM 2 Ġ( ĠIN LINE FORM 3 Ġ). ĠFor ĠIN LINE FORM 4 Ġa Ġstart - of - sent ence Ġ(< S OS >) Ġtoken Ġis Ġused Ġas ĠIN LINE FORM 5 Ġ. ĠThe Ġend Ġof Ġthe Ġutter ance Ġis Ġmod elled Ġusing Ġan Ġend - of - sent ence Ġ(< E OS >) Ġtoken . ĠWhen Ġthe Ġdec oder ĠR NN Ġgenerates Ġthe Ġend - of - sent ence Ġtoken , Ġthe Ġdecoding Ġprocess Ġis Ġterminated . ĠThe Ġoutput Ġof Ġthe Ġdec oder ĠR NN , ĠIN LINE FORM 6 Ġ, Ġis Ġpassed Ġthrough Ġan Ġaff ine Ġtransform Ġfollowed Ġby Ġthe Ġsoft max Ġfunction , ĠSM , Ġto Ġform ĠIN LINE FORM 7 Ġ. ĠA Ġword ĠIN LINE FORM 8 Ġcan Ġbe Ġobtained Ġby Ġeither Ġtaking Ġthe Ġword Ġwith Ġthe Ġhighest Ġprobability Ġor Ġsampling Ġfrom Ġthe Ġdistribution : ĠDIS PLAY FORM 0 </s> During Ġtraining Ġthe Ġwords Ġare Ġnot Ġsampled Ġfrom Ġthe Ġoutput Ġdistribution , Ġbut Ġinstead Ġthe Ġtrue Ġwords Ġfrom Ġthe Ġdataset Ġare Ġused . ĠThis Ġa Ġcommon Ġtechnique Ġthat Ġis Ġoften Ġreferred Ġto Ġas Ġteacher - forcing , Ġthough Ġit Ġalso Ġdirectly Ġfollows Ġfrom Ġequation ĠEQ REF 10 Ġ. </s> To Ġgenerate Ġa Ġsequence Ġusing Ġan ĠR NN , Ġbeam - search Ġis Ġoften Ġused . ĠUsing Ġbeam - search Ġwith ĠIN LINE FORM 0 Ġbeams , Ġthe Ġwords Ġcorresponding Ġto Ġthe Ġtop ĠIN LINE FORM 1 Ġprobabilities Ġof ĠIN LINE FORM 2 Ġare Ġthe Ġfirst ĠIN LINE FORM 3 Ġbeams . ĠFor Ġeach Ġsucceeding ĠIN LINE FORM 4 Ġ, Ġthe ĠIN LINE FORM 5 Ġwords Ġcorresponding Ġto Ġthe Ġtop ĠIN LINE FORM 6 Ġprobabilities Ġof ĠIN LINE FORM 7 Ġare Ġtaken Ġfor Ġeach Ġof Ġthe ĠIN LINE FORM 8 Ġbeams . ĠThis Ġis Ġfollowed Ġby Ġreducing Ġthe Ġnumber Ġof Ġbeams Ġfrom Ġnow ĠIN LINE FORM 9 Ġdown Ġto ĠIN LINE FORM 10 Ġ, Ġby Ġtaking Ġthe ĠIN LINE FORM 11 Ġbeams Ġwith Ġthe Ġhighest Ġprobability ĠIN LINE FORM 12 Ġ. ĠThis Ġis Ġa Ġdeter ministic Ġprocess . ĠHowever , Ġfor Ġthe ĠN US Ġto Ġalways Ġgive Ġthe Ġsame Ġresponse Ġin Ġthe Ġsame Ġcontext Ġis Ġnot Ġrealistic . ĠThus , Ġthe ĠN US Ġcannot Ġcover Ġthe Ġfull Ġbreadth Ġof Ġuser Ġbehaviour Ġif Ġbeam - search Ġis Ġused . ĠTo Ġsolve Ġthis Ġissue Ġwhile Ġkeeping Ġthe Ġbenefit Ġof Ġrejecting Ġsequences Ġwith Ġlow Ġprobability , Ġa Ġtype Ġof Ġbeam - search Ġwith Ġsampling Ġis Ġused . ĠThe Ġprocess Ġis Ġidentical Ġto Ġthe Ġabove , Ġbut ĠIN LINE FORM 13 Ġwords Ġper Ġbeam Ġare Ġsampled Ġfrom Ġthe Ġprobability Ġdistribution . ĠThe ĠN US Ġis Ġnow Ġnon - d eter ministic Ġresulting Ġin Ġa Ġdiverse ĠUS . ĠUsing Ġ2 Ġbeams Ġgave Ġa Ġgood Ġtrade - off Ġbetween Ġreasonable Ġresponses Ġand Ġdiversity . </s> Training </s> The Ġneural Ġsequence - to - sequence Ġmodel Ġis Ġtrained Ġto Ġmaximize Ġthe Ġlog Ġprobability Ġthat Ġit Ġassigns Ġto Ġthe Ġuser Ġutter ances Ġof Ġthe Ġtraining Ġdata Ġset : ĠDIS PLAY FORM 0 </s> The Ġnetwork Ġwas Ġimplemented Ġin ĠT ensor flow ĠB IB REF 36 Ġand Ġoptimized Ġusing ĠT ensor flow 's Ġdefault Ġsetup Ġof Ġthe ĠAdam Ġoptim izer ĠB IB REF 37 Ġ. ĠThe ĠL ST M Ġlayers Ġand Ġthe Ġfully - connected Ġlayer Ġhad Ġwidth s Ġof Ġ100 Ġeach Ġto Ġgive Ġa Ġreasonable Ġnumber Ġof Ġoverall Ġparameters . ĠThe Ġwidth Ġwas Ġnot Ġtuned . ĠThe Ġlearning Ġrate Ġwas Ġoptim ised Ġon Ġa Ġheld Ġout Ġvalidation Ġset Ġand Ġno Ġregular ization Ġmethods Ġused . ĠThe Ġtraining Ġset Ġwas Ġshuff led Ġat Ġthe Ġdialogue Ġturn Ġlevel . </s> The Ġmanual Ġtranscript ions Ġof Ġthe ĠD ST C 2 Ġtraining Ġset Ġ( not Ġthe ĠAS R Ġoutput ) Ġwere Ġused Ġto Ġtrain Ġthe Ġsequence - to - sequence Ġmodel . ĠSince Ġthe Ġtranscript ions Ġwere Ġdone Ġmanually Ġthey Ġcontained Ġspelling Ġerrors . ĠThese Ġwere Ġmanually Ġcorrected Ġto Ġensure Ġproper Ġde lex ical ization . ĠSome Ġdialog ues Ġwere Ġdiscarded Ġdue Ġto Ġtranscript ions Ġerrors Ġbeing Ġtoo Ġlarge . ĠAfter Ġcleaning Ġthe Ġdataset Ġthe Ġtraining Ġset Ġconsisted Ġof Ġ16 09 Ġdialog ues Ġwith Ġa Ġtotal Ġof Ġ116 38 Ġdialogue Ġturns . ĠThe Ġvalidation Ġset Ġhad Ġ505 Ġdialog ues Ġwith Ġ38 96 Ġdialogue Ġturns . ĠThe Ġmaximum Ġsequence Ġlength Ġof Ġthe Ġde lex ical ized Ġturns Ġwas Ġ22 , Ġincluding Ġthe Ġend Ġof Ġsentence Ġcharacter . ĠThe Ġmaximum Ġdialogue Ġlength Ġwas Ġ30 Ġturns . </s> All Ġdialogue Ġpolicies Ġwere Ġtrained Ġwith Ġthe ĠPy Dial Ġtool kit ĠB IB REF 38 Ġ, Ġby Ġinteracting Ġwith Ġeither Ġthe ĠN US Ġor ĠAB US . ĠThe ĠRL Ġalgorithm Ġused Ġis ĠGP - S AR SA ĠB IB REF 3 Ġwith Ġhyper param eters Ġtaken Ġfrom ĠB IB REF 39 Ġ. ĠThe Ġreward Ġfunction Ġused Ġgives Ġa Ġreward Ġof Ġ20 Ġto Ġa Ġsuccessfully Ġcompleted Ġdialogue Ġand Ġof Ġ- 1 Ġfor Ġeach Ġdialogue Ġturn . ĠThe Ġmaximum Ġdialogue Ġlength Ġwas Ġ25 Ġturns . ĠThe Ġpresented Ġmetrics Ġare Ġsuccess Ġrate Ġ( SR ) Ġand Ġaverage Ġreward Ġover Ġtest Ġdialog ues . ĠSR Ġis Ġthe Ġpercentage Ġof Ġdialog ues Ġfor Ġwhich Ġthe Ġsystem Ġsatisfied Ġboth Ġthe Ġuser 's Ġconstraints Ġand Ġrequests . ĠThe Ġfinal Ġgoal , Ġafter Ġpossible Ġgoal Ġchanges , Ġwas Ġused Ġfor Ġthis Ġevaluation . ĠWhen Ġpolicies Ġare Ġtrained Ġusing Ġthe ĠN US , Ġits Ġoutput Ġis Ġparsed Ġusing ĠPy Dial 's Ġregular Ġexpression Ġbased Ġsemantic Ġdec oder . ĠThe Ġpolicies Ġwere Ġtrained Ġfor Ġ4000 Ġdialog ues . </s> Exper imental ĠSetup </s> The Ġevaluation Ġof Ġuser Ġsim ulators Ġis Ġan Ġongoing Ġarea Ġof Ġresearch Ġand Ġa Ġvariety Ġof Ġtechniques Ġcan Ġbe Ġfound Ġin Ġthe Ġliterature . ĠMost Ġpapers Ġpublished Ġon Ġuser Ġsimulation Ġevaluate Ġtheir ĠUS Ġusing Ġdirect Ġmethods . ĠThese Ġmethods Ġevaluate Ġthe ĠUS Ġthrough Ġa Ġstatistical Ġmeasure Ġof Ġsimilarity Ġbetween Ġthe Ġoutputs Ġof Ġthe ĠUS Ġand Ġa Ġreal Ġuser Ġon Ġa Ġtest Ġset . ĠMultiple Ġmodels Ġcan Ġoutper form Ġthe ĠAB US Ġon Ġthese Ġmetrics . ĠHowever , Ġthis Ġis Ġunsur prising Ġsince Ġthese Ġuser Ġsim ulators Ġwere Ġtrained Ġon Ġthe Ġsame Ġor Ġsimilar Ġmetrics . ĠThe ĠAB US Ġwas Ġexplicitly Ġproposed Ġas Ġa Ġtool Ġto Ġtrain Ġthe Ġpolicy Ġof Ġa Ġdialogue Ġmanager Ġand Ġit Ġis Ġstill Ġthe Ġdominant Ġform Ġof ĠUS Ġused Ġfor Ġthis Ġtask . ĠTherefore , Ġthe Ġonly Ġfair Ġcomparison Ġbetween Ġa Ġnew ĠUS Ġmodel Ġand Ġthe ĠAB US Ġis Ġto Ġuse Ġthe Ġindirect Ġmethod Ġof Ġevaluating Ġthe Ġpolicies Ġthat Ġwere Ġobtained Ġby Ġtraining Ġwith Ġeach ĠUS . </s> Testing Ġwith Ġa Ġsimulated Ġuser </s> In ĠSch atz mann Ġet . Ġal Ġsch atz t mann 2005 effects Ġcross - model Ġevaluation Ġis Ġproposed Ġto Ġcompare Ġuser Ġsim ulators . ĠFirst , Ġthe Ġuser Ġsim ulators Ġto Ġbe Ġevaluated Ġare Ġused Ġto Ġtrain ĠIN LINE FORM 0 Ġpolicy Ġeach . ĠThen Ġthese Ġpolicies Ġare Ġtested Ġusing Ġthe Ġdifferent Ġuser Ġsim ulators Ġand Ġthe Ġresults Ġaveraged . ĠB IB REF 7 Ġshowed Ġthat Ġa Ġstrategy Ġlearned Ġwith Ġa Ġgood Ġuser Ġmodel Ġstill Ġperforms Ġwell Ġwhen Ġtested Ġon Ġpoor Ġuser Ġmodels . ĠIf Ġa Ġpolicy Ġperforms Ġwell Ġon Ġall Ġuser Ġsim ulators Ġand Ġnot Ġjust Ġon Ġthe Ġone Ġthat Ġit Ġwas Ġtrained Ġon , Ġit Ġindicates Ġthat Ġthe ĠUS Ġwith Ġwhich Ġit Ġwas Ġtrained Ġis Ġdiverse Ġand Ġrealistic , Ġand Ġthus Ġthe Ġpolicy Ġis Ġlikely Ġto Ġperform Ġbetter Ġon Ġreal Ġusers . ĠFor Ġeach ĠUS Ġfive Ġpolicies Ġ( ĠIN LINE FORM 1 Ġ), Ġeach Ġusing Ġa Ġdifferent Ġrandom Ġseed Ġfor Ġinitial isation , Ġare Ġtrained . ĠResults Ġare Ġreported Ġfor Ġboth Ġthe Ġbest Ġand Ġthe Ġaverage Ġperformance Ġon Ġ1000 Ġtest Ġdialog ues . ĠThe ĠAB US Ġis Ġprogrammed Ġto Ġalways Ġmention Ġthe Ġnew Ġgoal Ġafter Ġa Ġgoal Ġchange . ĠIn Ġorder Ġto Ġnot Ġlet Ġthis Ġaffect Ġour Ġresults Ġwe Ġimplement Ġthe Ġsame Ġfor Ġthe ĠN US Ġby Ġre - sam pling Ġa Ġsentence Ġif Ġthe Ġnew Ġgoal Ġis Ġnot Ġmentioned . </s> Testing Ġwith Ġreal Ġusers </s> Though Ġthe Ġabove Ġtest Ġis Ġalready Ġmore Ġindicative Ġof Ġpolicy Ġperformance Ġon Ġreal Ġusers Ġthan Ġmeasuring Ġstatistical Ġmetrics Ġof Ġuser Ġbehaviour , Ġa Ġbetter Ġtest Ġis Ġto Ġtest Ġwith Ġhuman Ġusers . ĠFor Ġthe Ġtest Ġon Ġhuman Ġusers , Ġtwo Ġpolicies Ġfor Ġeach ĠUS Ġthat Ġwas Ġused Ġfor Ġtraining Ġare Ġchosen Ġfrom Ġthe Ġfive Ġpolicies . ĠThe Ġfirst Ġpolicy Ġis Ġthe Ġone Ġthat Ġperformed Ġbest Ġwhen Ġtested Ġon Ġthe ĠN US . ĠThe Ġsecond Ġis Ġthe Ġone Ġthat Ġperformed Ġbest Ġwhen Ġtested Ġon Ġthe ĠAB US . ĠThis Ġchoice Ġof Ġpolicies Ġis Ġmotivated Ġby Ġa Ġtype Ġof Ġover fitting Ġto Ġbe Ġseen Ġin ĠSec . ĠSEC REF 17 Ġ. ĠThe Ġevaluation Ġof Ġthe Ġtrained Ġdialogue Ġpolicies Ġin Ġinteraction Ġwith Ġreal Ġusers Ġfollows Ġa Ġsimilar Ġset - up Ġto ĠB IB REF 40 Ġ. ĠUsers Ġare Ġrecruited Ġthrough Ġthe ĠAmazon ĠMechanical ĠTurk Ġ( AM T ) Ġservice . Ġ1000 Ġdialog ues Ġ( 250 Ġper Ġpolicy ) Ġwere Ġgathered . ĠThe Ġlearnt Ġpolicies Ġwere Ġincorporated Ġinto Ġan ĠS DS Ġpipeline Ġwith Ġa Ġcommercial ĠAS R Ġsystem . ĠThe ĠAM T Ġusers Ġwere Ġasked Ġto Ġfind Ġa Ġrestaurant Ġthat Ġmatches Ġcertain Ġconstraints Ġand Ġfind Ġcertain Ġrequests . ĠSubjects Ġwere Ġrandomly Ġallocated Ġto Ġone Ġof Ġthe Ġfour Ġanalysed Ġsystems . ĠAfter Ġeach Ġdialogue Ġthe Ġusers Ġwere Ġasked Ġwhether Ġthey Ġjudged Ġthe Ġdialogue Ġto Ġbe Ġsuccessful Ġor Ġnot Ġwhich Ġwas Ġthen Ġtranslated Ġto Ġthe Ġreward Ġmeasure . </s> Cross - Model ĠEvaluation </s> Table ĠT AB REF 18 Ġshows Ġthe Ġresults Ġof Ġthe Ġcross - model Ġevaluation Ġafter Ġ4000 Ġtraining Ġdialog ues . ĠThe Ġpolicies Ġtrained Ġwith Ġthe ĠN US Ġachieved Ġan Ġaverage Ġsuccess Ġrate Ġ( SR ) Ġof Ġ94 . 0 % Ġand Ġof Ġ96 . 6 % Ġwhen Ġtested Ġon Ġthe ĠAB US Ġand Ġthe ĠN US , Ġrespectively . ĠBy Ġcomparison , Ġthe Ġpolicies Ġtrained Ġwith Ġthe ĠAB US Ġachieved Ġaverage ĠSR s Ġof Ġ99 . 5 % Ġand Ġ45 . 5 % Ġrespectively . ĠThus , Ġtraining Ġwith Ġthe ĠN US Ġleads Ġto Ġpolicies Ġthat Ġcan Ġperform Ġwell Ġon Ġboth ĠUS s , Ġwhich Ġis Ġnot Ġthe Ġcase Ġfor Ġtraining Ġwith Ġthe ĠAB US . ĠFurthermore , Ġthe Ġbest ĠSR s Ġwhen Ġtested Ġon Ġthe ĠAB US Ġare Ġsimilar Ġat Ġ99 . 9 % Ġ( AB US ) Ġand Ġ99 . 8 % Ġ( N US ). ĠWhen Ġtested Ġon Ġthe ĠN US Ġthe Ġbest ĠSR s Ġwere Ġ71 . 5 % Ġ( AB US ) Ġand Ġ98 . 0 % Ġ( N US ). ĠThis Ġshows Ġthat Ġthe Ġbehaviour Ġof Ġthe ĠNeural ĠUser ĠSimulator Ġis Ġrealistic Ġand Ġdiverse Ġenough Ġto Ġtrain Ġpolicies Ġthat Ġcan Ġalso Ġperform Ġvery Ġwell Ġon Ġthe ĠAgenda - Based ĠUser ĠSimulator . </s> Of Ġthe Ġfive Ġpolicies , Ġfor Ġeach ĠUS , Ġthe Ġpolicy Ġperforming Ġbest Ġon Ġthe ĠN US Ġwas Ġnot Ġthe Ġbest Ġperforming Ġpolicy Ġon Ġthe ĠAB US . ĠThis Ġcould Ġindicate Ġthat Ġthe Ġpolicy ĠâĢ ľ over fits âĢ Ŀ Ġto Ġa Ġparticular Ġuser Ġsimulator . ĠOver fitting Ġusually Ġmanifests Ġitself Ġin Ġworse Ġresults Ġas Ġthe Ġmodel Ġis Ġtrained Ġfor Ġlonger . ĠFive Ġpolicies Ġtrained Ġon Ġeach ĠUS Ġfor Ġonly Ġ1000 Ġdialog ues Ġwere Ġalso Ġevaluated , Ġthe Ġresults Ġof Ġwhich Ġcan Ġbe Ġseen Ġin ĠTable ĠT AB REF 19 Ġ. ĠAfter Ġtraining Ġfor Ġ1000 Ġdialog ues , Ġthe Ġaverage ĠSR Ġof Ġthe Ġpolicies Ġtrained Ġon Ġthe ĠN US Ġwhen Ġtested Ġon Ġthe ĠAB US Ġwas Ġ97 . 3 % Ġin Ġcomparison Ġto Ġ94 . 0 % Ġafter Ġ4000 Ġdialog ues . ĠThis Ġbehaviour Ġwas Ġobserved Ġfor Ġall Ġfive Ġseeds , Ġwhich Ġindicates Ġthat Ġthe Ġpolicy Ġindeed Ġover fits Ġto Ġthe ĠN US . ĠFor Ġthe Ġpolicies Ġtrained Ġwith Ġthe ĠAB US Ġthis Ġwas Ġnot Ġobserved . ĠThis Ġcould Ġindicate Ġthat Ġthe Ġpolicy Ġcan Ġlearn Ġto Ġexploit Ġsome Ġof Ġthe Ġshortcomings Ġof Ġthe Ġtrained ĠN US . </s> Human ĠEvaluation </s> The Ġresults Ġof Ġthe Ġhuman Ġevaluation Ġare Ġshown Ġin ĠTable ĠT AB REF 21 Ġfor Ġ250 Ġdialog ues Ġper Ġpolicy . ĠIn ĠTable ĠT AB REF 21 Ġpolicies Ġare Ġmarked Ġusing Ġan ĠID Ġ( ĠIN LINE FORM 0 Ġ) Ġthat Ġtranslates Ġto Ġresults Ġin ĠTables ĠT AB REF 18 Ġand ĠT AB REF 19 Ġ. ĠBoth Ġpolicies Ġtrained Ġwith Ġthe ĠN US Ġoutper formed Ġthose Ġtrained Ġon Ġthe ĠAB US Ġin Ġterms Ġof Ġboth Ġreward Ġand Ġsuccess Ġrate . ĠThe Ġbest Ġperforming Ġpolicy Ġtrained Ġon Ġthe ĠN US Ġachieves Ġa Ġ93 . 4 % Ġsuccess Ġrate Ġand Ġ13 . 8 Ġaverage Ġrewards Ġwhilst Ġthe Ġbest Ġperforming Ġpolicy Ġtrained Ġwith Ġthe ĠAB US Ġachieves Ġonly Ġa Ġ90 . 0 % Ġsuccess Ġrate Ġand Ġ13 . 3 Ġaverage Ġreward . ĠThis Ġshows Ġthat Ġthe Ġgood Ġperformance Ġof Ġthe ĠN US Ġon Ġthe Ġcross - model Ġevaluation Ġtransfers Ġto Ġreal Ġusers . ĠFurthermore , Ġthe Ġover fitting Ġto Ġa Ġparticular ĠUS Ġis Ġalso Ġobserved Ġin Ġthe Ġreal Ġuser Ġevaluation . ĠFor Ġnot Ġonly Ġthe Ġpolicies Ġtrained Ġon Ġthe ĠN US , Ġbut Ġalso Ġthose Ġtrained Ġon Ġthe ĠAB US , Ġthe Ġbest Ġperforming Ġpolicy Ġwas Ġthe Ġpolicy Ġthat Ġperformed Ġbest Ġon Ġthe Ġother ĠUS . </s> Conclusion </s> We Ġintroduced Ġthe ĠNeural ĠUser ĠSimulator Ġ( N US ), Ġwhich Ġuses Ġthe Ġsystem 's Ġresponse Ġin Ġits Ġsemantic Ġform Ġas Ġinput Ġand Ġgives Ġa Ġnatural Ġlanguage Ġresponse . ĠIt Ġthus Ġneeds Ġless Ġlab elling Ġof Ġthe Ġtraining Ġdata Ġthan ĠUser ĠSim ulators Ġthat Ġgenerate Ġa Ġresponse Ġin Ġsemantic Ġform . ĠIt Ġwas Ġshown Ġthat Ġthe ĠN US Ġlearns Ġrealistic Ġuser Ġbehaviour Ġfrom Ġa Ġcorpus Ġof Ġrecorded Ġdialog ues Ġsuch Ġthat Ġit Ġcan Ġbe Ġused Ġto Ġoptim ise Ġthe Ġpolicy Ġof Ġthe Ġdialogue Ġmanager Ġof Ġa Ġspoken Ġdialogue Ġsystem . ĠThe ĠN US Ġwas Ġcompared Ġto Ġthe ĠAgenda - Based ĠUser ĠSimulator Ġby Ġevaluating Ġpolicies Ġtrained Ġwith Ġthese Ġuser Ġsim ulators . ĠThe Ġtrained Ġpolicies Ġwere Ġcompared Ġboth Ġby Ġtesting Ġthem Ġwith Ġsimulated Ġusers Ġand Ġalso Ġwith Ġreal Ġusers . ĠThe ĠN US Ġexce lled Ġon Ġboth Ġevaluation Ġtasks . </s> Acknowled gements </s> This Ġresearch Ġwas Ġpartly Ġfunded Ġby Ġthe ĠEPS RC Ġgrant ĠEP / M 01 89 46 / 1 ĠOpen ĠDomain ĠStatistical ĠSp oken ĠDialogue ĠSystems . ĠFlor ian ĠK rey ss ig Ġis Ġsupported Ġby Ġthe ĠStud ien st ift ung Ġdes ĠDe ut sc hen ĠVol kes . ĠPaw e ÅĤ ĠBud z ian owski Ġis Ġsupported Ġby Ġthe ĠEPS RC Ġand ĠTos hiba ĠResearch ĠEurope ĠLtd .