two previous turns
ABSTRACTIVE
160
EXTRACTIVE
The previously proposed contextual language models, such as DRNNLM and CCDCLM, treat dialog history as a sequence of inputs, without modeling dialog interactions. A dialog turn from one speaker may not only be a direct response to the other speaker's query, but also likely to be a continuation of his own previous statement. Thus, when modeling turn $k$ in a dialog, we propose to connect the last RNN state of turn $k-2$ directly to the starting RNN state of turn $k$ , instead of letting it to propagate through the RNN for turn $k-1$ . The last RNN state of turn $k-1$ serves as the context vector to turn $k$ , which is fed to turn $k$ 's RNN hidden state at each time step together with the word input. The model architecture is as shown in Figure 2 . The context vector $c$ and the initial RNN hidden state for the $k$ th turn $h^{\mathbf {U}_k}_{0}$ are defined as:
We use the Switchboard Dialog Act Corpus (SwDA) in evaluating our contextual langauge models. The SwDA corpus extends the Switchboard-1 Telephone Speech Corpus with turn and utterance-level dialog act tags. The utterances are also tagged with part-of-speech (POS) tags. We split the data in folder sw00 to sw09 as training set, folder sw10 as test set, and folder sw11 to sw13 as validation set. The training, validation, and test sets contain 98.7K turns (190.0K utterances), 5.7K turns (11.3K utterances), and 11.9K turns (22.2K utterances) respectively. Maximum turn length is set to 160. The vocabulary is defined with the top frequent 10K words.
1701.04056
Introduction </s> Language Ġmodel Ġplays Ġan Ġimportant Ġrole Ġin Ġmany Ġnatural Ġlanguage Ġprocessing Ġsystems , Ġsuch Ġas Ġin Ġautomatic Ġspeech Ġrecognition ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġand Ġmachine Ġtranslation Ġsystems ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ. ĠRec urrent Ġneural Ġnetwork Ġ( R NN ) Ġbased Ġmodels ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġhave Ġrecently Ġshown Ġsuccess Ġin Ġlanguage Ġmodeling , Ġoutper forming Ġconventional Ġn - gram Ġbased Ġmodels . ĠLong Ġshort - term Ġmemory ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġis Ġa Ġwidely Ġused ĠR NN Ġvariant Ġfor Ġlanguage Ġmodeling Ġdue Ġto Ġits Ġsuperior Ġperformance Ġin Ġcapturing Ġlonger Ġterm Ġdependencies . </s> Con ventional ĠR NN Ġbased Ġlanguage Ġmodel Ġuses Ġa Ġhidden Ġstate Ġto Ġrepresent Ġthe Ġsummary Ġof Ġthe Ġpreceding Ġwords Ġin Ġa Ġsentence Ġwithout Ġconsidering Ġcontext Ġsignals . ĠMik ol ov Ġet Ġal . Ġproposed Ġa Ġcontext Ġdependent ĠR NN Ġlanguage Ġmodel ĠB IB REF 8 Ġby Ġconnecting Ġa Ġcontextual Ġvector Ġto Ġthe ĠR NN Ġhidden Ġstate . ĠThis Ġcontextual Ġvector Ġis Ġproduced Ġby Ġapplying ĠLat ent ĠDir ich let ĠAll ocation ĠB IB REF 9 Ġon Ġpreceding Ġtext . ĠSeveral Ġother Ġcontextual Ġlanguage Ġmodels Ġwere Ġlater Ġproposed Ġby Ġusing Ġbag - of - word ĠB IB REF 10 Ġand ĠR NN Ġmethods ĠB IB REF 11 Ġto Ġlearn Ġlarger Ġcontext Ġrepresentation Ġthat Ġbeyond Ġthe Ġtarget Ġsentence . </s> The Ġpreviously Ġproposed Ġcontextual Ġlanguage Ġmodels Ġtreat Ġpreceding Ġsentences Ġas Ġa Ġsequence Ġof Ġinputs , Ġand Ġthey Ġare Ġsuitable Ġfor Ġdocument Ġlevel Ġcontext Ġmodeling . ĠIn Ġdialog Ġmodeling , Ġhowever , Ġdialog Ġinteractions Ġbetween Ġspeakers Ġplay Ġan Ġimportant Ġrole . ĠModel ing Ġutter ances Ġin Ġa Ġdialog Ġas Ġa Ġsequence Ġof Ġinputs Ġmight Ġnot Ġwell Ġcapture Ġthe Ġpauses , Ġturn - taking , Ġand Ġgrounding Ġphenomena ĠB IB REF 12 Ġin Ġa Ġdialog . ĠIn Ġthis Ġwork , Ġwe Ġpropose Ġcontextual ĠR NN Ġlanguage Ġmodels Ġthat Ġspecially Ġtrack Ġthe Ġinteractions Ġbetween Ġspeakers . ĠWe Ġexpect Ġsuch Ġmodels Ġto Ġgenerate Ġbetter Ġrepresentations Ġof Ġthe Ġdialog Ġcontext . </s> The Ġremainder Ġof Ġthe Ġpaper Ġis Ġorganized Ġas Ġfollows . ĠIn Ġsection Ġ2 , Ġwe Ġintroduce Ġthe Ġbackground Ġon Ġcontextual Ġlanguage Ġmodeling . ĠIn Ġsection Ġ3 , Ġwe Ġdescribe Ġthe Ġproposed Ġdialog Ġcontext Ġlanguage Ġmodels . ĠSection Ġ4 Ġdiscusses Ġthe Ġevaluation Ġprocedures Ġand Ġresults . ĠSection Ġ5 Ġconcludes Ġthe Ġwork . </s> R NN ĠLanguage ĠModel </s> A Ġlanguage Ġmodel Ġassigns Ġa Ġprobability Ġto Ġa Ġsequence Ġof Ġwords Ġ$\ math bf Ġ{ w } =( w _ 1 , Ġw _ 2 , Ġ... , Ġw _{ T }) $ Ġfollowing Ġprobability Ġdistribution . ĠUsing Ġthe Ġchain Ġrule , Ġthe Ġlikelihood Ġof Ġthe Ġword Ġsequence Ġ$\ math bf Ġ{ w } $ Ġcan Ġbe Ġfactor ized Ġas : </s> $$ P (\ math bf Ġ{ w }) Ġ= ĠP ( w _ 1 , Ġw _ 2 , Ġ... , Ġw _{ T }) Ġ= Ġ\ pro d Ġ_ { t = 1 } ^{ T } P ( w _{ t } | w _{ < Ġt }) Ġ\\ $$ Ġ Ġ Ġ( E q . Ġ2 ) </s> At Ġtime Ġstep Ġ$ t $ Ġ, Ġthe Ġsystem Ġinput Ġis Ġthe Ġembed ding Ġof Ġthe Ġword Ġat Ġindex Ġ$ t $ Ġ, Ġand Ġthe Ġsystem Ġoutput Ġis Ġthe Ġprobability Ġdistribution Ġof Ġthe Ġword Ġat Ġindex Ġ$ t + 1 $ Ġ. ĠThe ĠR NN Ġhidden Ġstate Ġ$ h _ t $ Ġenc odes Ġthe Ġinformation Ġof Ġthe Ġword Ġsequence Ġup Ġtill Ġcurrent Ġstep : </s> $$ & h _ t Ġ= Ġ\ oper at orn ame { R NN } ( h _{ t - 1 }, Ġw _ t ) Ġ\\ Ġ& P ( w _{ t + 1 } | w _{ < Ġt + 1 }) Ġ= Ġ\ oper at orn ame { soft max } ( W _{ o } h _{ t } Ġ+ Ġb _{ o }) $$ Ġ Ġ Ġ( E q . Ġ3 ) </s> where Ġ$ W _{ o } $ Ġand Ġ$ b _{ o } $ Ġare Ġthe Ġoutput Ġlayer Ġweights Ġand Ġbiases . </s> Context ual ĠR NN ĠLanguage ĠModel </s> A Ġnumber Ġof Ġmethods Ġhave Ġbeen Ġproposed Ġto Ġintroduce Ġcontextual Ġinformation Ġto Ġthe Ġlanguage Ġmodel . ĠMik ol ov Ġand ĠZ we ig ĠB IB REF 8 Ġproposed Ġa Ġtopic - condition ed ĠR NN LM Ġby Ġintroducing Ġa Ġcontextual Ġreal - valued Ġvector Ġto ĠR NN Ġhidden Ġstate . ĠThe Ġcontextual Ġvector Ġwas Ġcreated Ġby Ġperforming ĠL DA ĠB IB REF 9 Ġon Ġpreceding Ġtext . ĠWang Ġand ĠCho ĠB IB REF 10 Ġstudied Ġintroducing Ġcorpus - level Ġdiscourse Ġinformation Ġinto Ġlanguage Ġmodeling . ĠA Ġnumber Ġof Ġcontext Ġrepresentation Ġmethods Ġwere Ġexplored , Ġincluding Ġbag - of - words , Ġsequence Ġof Ġbag - of - words , Ġand Ġsequence Ġof Ġbag - of - words Ġwith Ġattention . ĠLin Ġet Ġal . ĠB IB REF 13 Ġproposed Ġusing Ġhierarchical ĠR NN Ġfor Ġdocument Ġmodeling . ĠComp aring Ġto Ġusing Ġbag - of - words Ġand Ġsequence Ġof Ġbag - of - words Ġfor Ġdocument Ġcontext Ġrepresentation , Ġusing Ġhierarchical ĠR NN Ġcan Ġbetter Ġmodel Ġthe Ġorder Ġof Ġwords Ġin Ġpreceding Ġtext , Ġat Ġthe Ġcost Ġof Ġthe Ġincreased Ġcomputational Ġcomplexity . ĠThese Ġcontextual Ġlanguage Ġmodels Ġfocused Ġon Ġcontextual Ġinformation Ġat Ġthe Ġdocument Ġlevel . ĠTr an Ġet Ġal . ĠB IB REF 14 Ġfurther Ġproposed Ġa Ġcontextual Ġlanguage Ġmodel Ġthat Ġconsider Ġinformation Ġat Ġinter - document Ġlevel . ĠThey Ġclaimed Ġthat Ġby Ġutilizing Ġthe Ġstructural Ġinformation Ġfrom Ġa Ġtree - struct ured Ġdocument Ġset , Ġlanguage Ġmodeling Ġperformance Ġwas Ġlargely Ġimproved . </s> Methods </s> The Ġpreviously Ġproposed Ġcontextual Ġlanguage Ġmodels Ġfocus Ġon Ġapplying Ġcontext Ġby Ġencoding Ġpreceding Ġtext , Ġwithout Ġconsidering Ġinteractions Ġin Ġdialog s . ĠThese Ġmodels Ġmay Ġnot Ġbe Ġwell Ġsuited Ġfor Ġdialog Ġlanguage Ġmodeling , Ġas Ġthey Ġare Ġnot Ġdesigned Ġto Ġcapture Ġdialog Ġinteractions , Ġsuch Ġas Ġclar ifications Ġand Ġconfirm ations . ĠBy Ġmaking Ġspecial Ġdesign Ġin Ġlearning Ġdialog Ġinteractions , Ġwe Ġexpect Ġthe Ġmodels Ġto Ġgenerate Ġbetter Ġrepresentations Ġof Ġthe Ġdialog Ġcontext , Ġand Ġthus Ġlower Ġperplex ity Ġof Ġthe Ġtarget Ġdialog Ġturn Ġor Ġutter ance . </s> In Ġthis Ġsection , Ġwe Ġfirst Ġexplain Ġthe Ġcontext Ġdependent ĠR NN Ġlanguage Ġmodel Ġthat Ġoperates Ġon Ġutter ance Ġor Ġturn Ġlevel . ĠFollowing Ġthat , Ġwe Ġdescribe Ġthe Ġtwo Ġproposed Ġcontextual Ġlanguage Ġmodels Ġthat Ġutilize Ġthe Ġdialog Ġlevel Ġcontext . </s> Context ĠDepend ent ĠR NN LM </s> Let Ġ$\ math bf Ġ{ D } Ġ= Ġ( \ math bf Ġ{ U } _ 1 , Ġ\ math bf Ġ{ U } _ 2 , Ġ... , Ġ\ math bf Ġ{ U } _ K ) $ Ġbe Ġa Ġdialog Ġthat Ġhas Ġ$ K $ Ġturns Ġand Ġinvolves Ġtwo Ġspeakers . ĠEach Ġturn Ġmay Ġhave Ġone Ġor Ġmore Ġutter ances . ĠThe Ġ$ k $ Ġth Ġturn Ġ$\ math bf Ġ{ U } _ k Ġ= Ġ( w _ 1 , Ġw _ 2 , Ġ... , Ġw _{ T _ k }) $ Ġis Ġrepresented Ġas Ġa Ġsequence Ġof Ġ$ T _ k $ Ġwords . ĠCondition ing Ġon Ġinformation Ġof Ġthe Ġpreceding Ġtext Ġin Ġthe Ġdialog , Ġprobability Ġof Ġthe Ġtarget Ġturn Ġ$\ math bf Ġ{ U } _ k $ Ġcan Ġbe Ġcalculated Ġas : </s> $$ P (\ math bf Ġ{ U } _ k | \ math bf Ġ{ U } _{ < k }) Ġ= Ġ\ pro d Ġ_ { t = 1 } ^{ T _ k } P ( w ^ {\ math bf Ġ{ U } _{ k }} _{ t } | w ^ {\ math bf Ġ{ U } _{ k }} _{ < Ġt }, Ġ\ math bf Ġ{ U } _{ < k }) Ġ\\ $$ Ġ Ġ Ġ( E q . Ġ6 ) </s> where Ġ$\ math bf Ġ{ U } _{ < k } $ Ġdenotes Ġall Ġprevious Ġturns Ġbefore Ġ$\ math bf Ġ{ U } _ k $ Ġ, Ġand Ġ$ w ^ {\ math bf Ġ{ U } _{ k }} _{ < Ġt } $ Ġdenotes Ġall Ġprevious Ġwords Ġbefore Ġthe Ġ$ t $ Ġth Ġword Ġin Ġturn Ġ$\ math bf Ġ{ U } _ k $ Ġ. </s> In Ġcontext Ġdependent ĠR NN Ġlanguage Ġmodel , Ġthe Ġcontext Ġvector Ġ$ c $ Ġis Ġconnected Ġto Ġthe ĠR NN Ġhidden Ġstate Ġtogether Ġwith Ġthe Ġinput Ġword Ġembed ding Ġat Ġeach Ġtime Ġstep Ġ( Figure Ġ1 Ġ). ĠThis Ġis Ġsimilar Ġto Ġthe Ġcontext Ġdependent ĠR NN Ġlanguage Ġmodel Ġproposed Ġin ĠB IB REF 8 Ġ, Ġother Ġthan Ġthat Ġthe Ġcontext Ġvector Ġis Ġnot Ġconnected Ġdirectly Ġto Ġthe ĠR NN Ġoutput Ġlayer . ĠWith Ġthe Ġadditional Ġcontext Ġvector Ġinput Ġ$ c $ Ġ, Ġthe ĠR NN Ġstate Ġ$ h _ t $ Ġis Ġupdated Ġas : </s> $$ h _ t Ġ= Ġ\ oper at orn ame { R NN } ( h _{ t - 1 }, Ġ[ w _ t , Ġc ]) $$ Ġ Ġ Ġ( E q . Ġ8 ) </s> Context ĠRepresent ations </s> In Ġneural Ġnetwork Ġbased Ġlanguage Ġmodels , Ġthe Ġdialog Ġcontext Ġcan Ġbe Ġrepresented Ġas Ġa Ġdense Ġcontinuous Ġvector . ĠThis Ġcontext Ġvector Ġcan Ġbe Ġproduced Ġin Ġa Ġnumber Ġof Ġways . </s> One Ġsimple Ġapproach Ġis Ġto Ġuse Ġbag Ġof Ġword Ġembed d ings . ĠHowever , Ġbag Ġof Ġword Ġembed ding Ġcontext Ġrepresentation Ġdoes Ġnot Ġtake Ġword Ġorder Ġinto Ġconsideration . ĠAn Ġalternative Ġapproach Ġis Ġto Ġuse Ġan ĠR NN Ġto Ġread Ġthe Ġpreceding Ġtext . ĠThe Ġlast Ġhidden Ġstate Ġof Ġthe ĠR NN Ġenc oder Ġcan Ġbe Ġseen Ġas Ġthe Ġrepresentation Ġof Ġthe Ġtext Ġand Ġbe Ġused Ġas Ġthe Ġcontext Ġvector Ġfor Ġthe Ġnext Ġturn . ĠTo Ġgenerate Ġdocument Ġlevel Ġcontext Ġrepresentation , Ġone Ġmay Ġcascade Ġall Ġsentences Ġin Ġa Ġdocument Ġby Ġremoving Ġthe Ġsentence Ġboundaries . ĠThe Ġlast ĠR NN Ġhidden Ġstate Ġof Ġthe Ġprevious Ġutter ance Ġserves Ġas Ġthe Ġinitial ĠR NN Ġstate Ġof Ġthe Ġnext Ġutter ance . ĠAs Ġin ĠB IB REF 11 Ġ, Ġwe Ġrefer Ġto Ġthis Ġmodel Ġas ĠDR NN LM . ĠAlternatively , Ġin Ġthe ĠC CD CL M Ġmodel Ġproposed Ġin ĠB IB REF 11 Ġ, Ġthe Ġlast ĠR NN Ġhidden Ġstate Ġof Ġthe Ġprevious Ġutter ance Ġis Ġfed Ġto Ġthe ĠR NN Ġhidden Ġstate Ġof Ġthe Ġtarget Ġutter ance Ġat Ġeach Ġtime Ġstep . </s> Inter active ĠDial og ĠContext ĠLM </s> The Ġpreviously Ġproposed Ġcontextual Ġlanguage Ġmodels , Ġsuch Ġas ĠDR NN LM Ġand ĠC CD CL M , Ġtreat Ġdialog Ġhistory Ġas Ġa Ġsequence Ġof Ġinputs , Ġwithout Ġmodeling Ġdialog Ġinteractions . ĠA Ġdialog Ġturn Ġfrom Ġone Ġspeaker Ġmay Ġnot Ġonly Ġbe Ġa Ġdirect Ġresponse Ġto Ġthe Ġother Ġspeaker 's Ġquery , Ġbut Ġalso Ġlikely Ġto Ġbe Ġa Ġcontinuation Ġof Ġhis Ġown Ġprevious Ġstatement . ĠThus , Ġwhen Ġmodeling Ġturn Ġ$ k $ Ġin Ġa Ġdialog , Ġwe Ġpropose Ġto Ġconnect Ġthe Ġlast ĠR NN Ġstate Ġof Ġturn Ġ$ k - 2 $ Ġdirectly Ġto Ġthe Ġstarting ĠR NN Ġstate Ġof Ġturn Ġ$ k $ Ġ, Ġinstead Ġof Ġletting Ġit Ġto Ġpropagate Ġthrough Ġthe ĠR NN Ġfor Ġturn Ġ$ k - 1 $ Ġ. ĠThe Ġlast ĠR NN Ġstate Ġof Ġturn Ġ$ k - 1 $ Ġserves Ġas Ġthe Ġcontext Ġvector Ġto Ġturn Ġ$ k $ Ġ, Ġwhich Ġis Ġfed Ġto Ġturn Ġ$ k $ Ġ' s ĠR NN Ġhidden Ġstate Ġat Ġeach Ġtime Ġstep Ġtogether Ġwith Ġthe Ġword Ġinput . ĠThe Ġmodel Ġarchitecture Ġis Ġas Ġshown Ġin ĠFigure Ġ2 Ġ. ĠThe Ġcontext Ġvector Ġ$ c $ Ġand Ġthe Ġinitial ĠR NN Ġhidden Ġstate Ġfor Ġthe Ġ$ k $ Ġth Ġturn Ġ$ h ^ {\ math bf Ġ{ U } _ k } _{ 0 } $ Ġare Ġdefined Ġas : </s> $$ c Ġ= Ġh ^ {\ math bf Ġ{ U } _{ k - 1 }} _{ T _{ k - 1 } }, Ġ\ ; Ġh ^ {\ math bf Ġ{ U } _ k } _{ 0 } Ġ= Ġh ^ {\ math bf Ġ{ U } _{ k - 2 }} _{ T _{ k - 2 }} $$ Ġ Ġ Ġ( E q . Ġ11 ) </s> where Ġ$ h ^ {\ math bf Ġ{ U } _{ k - 1 }} _{ T _{ k - 1 }} $ Ġrepresents Ġthe Ġlast ĠR NN Ġhidden Ġstate Ġof Ġturn Ġ$ k - 1 $ Ġ. ĠThis Ġmodel Ġalso Ġallows Ġthe Ġcontext Ġsignal Ġfrom Ġprevious Ġturns Ġto Ġpropagate Ġthrough Ġthe Ġnetwork Ġin Ġfewer Ġsteps , Ġwhich Ġhelps Ġto Ġreduce Ġinformation Ġloss Ġalong Ġthe Ġpropagation . ĠWe Ġrefer Ġto Ġthis Ġmodel Ġas ĠInteractive ĠDial og ĠContext ĠLanguage ĠModel Ġ( ID CL M ). </s> External ĠState ĠInteractive ĠDial og ĠContext ĠLM </s> The Ġpropagation Ġof Ġdialog Ġcontext Ġcan Ġbe Ġseen Ġas Ġa Ġseries Ġof Ġupdates Ġof Ġa Ġhidden Ġdialog Ġcontext Ġstate Ġalong Ġthe Ġgrowing Ġdialog . ĠID CL M Ġmodels Ġthis Ġhidden Ġdialog Ġcontext Ġstate Ġchanges Ġimplicitly Ġin Ġthe Ġturn Ġlevel ĠR NN Ġstate . ĠSuch Ġdialog Ġcontext Ġstate Ġupdates Ġcan Ġalso Ġbe Ġmodeled Ġin Ġa Ġseparated ĠR NN . ĠAs Ġshown Ġin Ġthe Ġarchitecture Ġin ĠFigure Ġ3 Ġ, Ġwe Ġuse Ġan Ġexternal ĠR NN Ġto Ġmodel Ġthe Ġcontext Ġchanges Ġexplicitly . ĠInput Ġto Ġthe Ġexternal Ġstate ĠR NN Ġis Ġthe Ġvector Ġrepresentation Ġof Ġthe Ġprevious Ġdialog Ġturns . ĠThe Ġexternal Ġstate ĠR NN Ġoutput Ġserves Ġas Ġthe Ġdialog Ġcontext Ġfor Ġnext Ġturn : </s> $$ s _{ k - 1 } Ġ= Ġ\ oper at orn ame { R NN } _{ ES } ( s _{ k - 2 }, Ġh ^ {\ math bf Ġ{ U } _{ k - 1 }} _{ T _{ k - 1 }} ) $$ Ġ Ġ Ġ( E q . Ġ14 ) </s> where Ġ$ s _{ k - 1 } $ Ġis Ġthe Ġoutput Ġof Ġthe Ġexternal Ġstate ĠR NN Ġafter Ġthe Ġprocessing Ġof Ġturn Ġ$ k - 1 $ Ġ. ĠThe Ġcontext Ġvector Ġ$ c $ Ġand Ġthe Ġinitial ĠR NN Ġhidden Ġstate Ġfor Ġthe Ġ$ k $ Ġth Ġturn Ġ$ h ^ {\ math bf Ġ{ U } _ k } _{ 0 } $ Ġare Ġthen Ġdefined Ġas : </s> $$ c Ġ= Ġs _{ k - 1 }, Ġ\ ; Ġh ^ {\ math bf Ġ{ U } _ k } _{ 0 } Ġ= Ġh ^ {\ math bf Ġ{ U } _{ k - 2 }} _{ T _{ k - 2 }} $$ Ġ Ġ Ġ( E q . Ġ15 ) </s> We Ġrefer Ġto Ġthis Ġmodel Ġas ĠExternal ĠState ĠInteractive ĠDial og ĠContext ĠLanguage ĠModel Ġ( ES ID CL M ). </s> Comp aring Ġto ĠID CL M , ĠES ID CL M Ġreleases Ġthe Ġburden Ġof Ġturn Ġlevel ĠR NN Ġby Ġusing Ġan Ġexternal ĠR NN Ġto Ġmodel Ġdialog Ġcontext Ġstate Ġchanges . ĠOne Ġdrawback Ġof ĠES ID CL M Ġis Ġthat Ġthere Ġare Ġadditional ĠR NN Ġmodel Ġparameters Ġto Ġbe Ġlearned Ġduring Ġmodel Ġtraining , Ġwhich Ġmay Ġmake Ġthe Ġmodel Ġmore Ġprone Ġto Ġover fitting Ġwhen Ġtraining Ġdata Ġsize Ġis Ġlimited . </s> Data ĠSet </s> We Ġuse Ġthe ĠSwitch board ĠDial og ĠAct ĠCorpus Ġ( Sw DA ) Ġin Ġevaluating Ġour Ġcontextual Ġlang au ge Ġmodels . ĠThe ĠSw DA Ġcorpus Ġextends Ġthe ĠSwitch board - 1 ĠTelephone ĠSpeech ĠCorpus Ġwith Ġturn Ġand Ġutter ance - level Ġdialog Ġact Ġtags . ĠThe Ġutter ances Ġare Ġalso Ġtagged Ġwith Ġpart - of - speech Ġ( POS ) Ġtags . ĠWe Ġsplit Ġthe Ġdata Ġin Ġfolder Ġsw 00 Ġto Ġsw 09 Ġas Ġtraining Ġset , Ġfolder Ġsw 10 Ġas Ġtest Ġset , Ġand Ġfolder Ġsw 11 Ġto Ġsw 13 Ġas Ġvalidation Ġset . ĠThe Ġtraining , Ġvalidation , Ġand Ġtest Ġsets Ġcontain Ġ98 . 7 K Ġturns Ġ( 190 . 0 K Ġutter ances ), Ġ5 . 7 K Ġturns Ġ( 11 . 3 K Ġutter ances ), Ġand Ġ11 . 9 K Ġturns Ġ( 22 . 2 K Ġutter ances ) Ġrespectively . ĠMaximum Ġturn Ġlength Ġis Ġset Ġto Ġ160 . ĠThe Ġvocabulary Ġis Ġdefined Ġwith Ġthe Ġtop Ġfrequent Ġ10 K Ġwords . </s> Bas elines </s> We Ġcompare ĠID CL M Ġand ĠES ID CL M Ġto Ġseveral Ġbaseline Ġmethods , Ġincluding Ġn - gram Ġbased Ġmodel , Ġsingle Ġturn ĠR NN LM , Ġand Ġvarious Ġcontext Ġdependent ĠR NN L Ms . </s> 5 - gram ĠKN ĠA Ġ5 - gram Ġlanguage Ġmodel Ġwith Ġmodified ĠK nes er - N ey Ġsmoot hing ĠB IB REF 15 Ġ. </s> Single - Turn - R NN LM ĠCon ventional ĠR NN LM Ġthat Ġoperates Ġon Ġsingle Ġturn Ġlevel Ġwith Ġno Ġcontext Ġinformation . </s> Bo W - Context - R NN LM ĠContext ual ĠR NN LM Ġwith ĠBo W Ġrepresentation Ġof Ġpreceding Ġtext Ġas Ġcontext . </s> DR NN LM ĠContext ual ĠR NN LM Ġwith Ġturn Ġlevel Ġcontext Ġvector Ġconnected Ġto Ġinitial ĠR NN Ġstate Ġof Ġthe Ġtarget Ġturn . </s> CC D CL M ĠContext ual ĠR NN LM Ġwith Ġturn Ġlevel Ġcontext Ġvector Ġconnected Ġto ĠR NN Ġhidden Ġstate Ġof Ġthe Ġtarget Ġturn Ġat Ġeach Ġtime Ġstep . ĠWe Ġimplement Ġthis Ġmodel Ġfollowing Ġthe Ġdesign Ġin ĠB IB REF 11 Ġ. </s> In Ġorder Ġto Ġinvestigate Ġthe Ġpotential Ġperformance Ġgain Ġthat Ġcan Ġbe Ġachieved Ġby Ġintroducing Ġcontext , Ġwe Ġalso Ġcompare Ġthe Ġproposed Ġmethods Ġto ĠR NN L Ms Ġthat Ġuse Ġtrue Ġdialog Ġact Ġtags Ġas Ġcontext . ĠAlthough Ġhuman Ġlabeled Ġdialog Ġact Ġmight Ġnot Ġbe Ġthe Ġbest Ġoption Ġfor Ġmodeling Ġthe Ġdialog Ġcontext Ġstate , Ġit Ġprovides Ġa Ġreasonable Ġestimation Ġof Ġthe Ġbest Ġgain Ġthat Ġcan Ġbe Ġachieved Ġby Ġintroducing Ġlinguistic Ġcontext . ĠThe Ġdialog Ġact Ġsequence Ġis Ġmodeled Ġby Ġa Ġseparated ĠR NN , Ġsimilar Ġto Ġthe Ġexternal Ġstate ĠR NN Ġused Ġin ĠES ID CL M . ĠWe Ġrefer Ġto Ġthis Ġmodel Ġas ĠDial og ĠAct ĠContext ĠLanguage ĠModel Ġ( D AC LM ). </s> D AC LM ĠR NN LM Ġwith Ġtrue Ġdialog Ġact Ġcontext Ġvector Ġconnected Ġto ĠR NN Ġstate Ġof Ġthe Ġtarget Ġturn Ġat Ġeach Ġtime Ġstep . </s> Model ĠConfiguration Ġand ĠTraining </s> In Ġthis Ġwork , Ġwe Ġuse ĠL ST M Ġcell ĠB IB REF 6 Ġas Ġthe Ġbasic ĠR NN Ġunit Ġfor Ġits Ġstronger Ġcapability Ġin Ġcapturing Ġlong - range Ġdependencies Ġin Ġa Ġword Ġsequence Ġcomparing Ġto Ġsimple ĠR NN . ĠWe Ġuse Ġpre - trained Ġword Ġvectors ĠB IB REF 16 Ġthat Ġare Ġtrained Ġon ĠGoogle ĠNews Ġdataset Ġto Ġinitialize Ġthe Ġword Ġembed d ings . ĠThese Ġword Ġembed d ings Ġare Ġfine - tun ed Ġduring Ġmodel Ġtraining . ĠWe Ġconduct Ġmini - batch Ġtraining Ġusing ĠAdam Ġoptimization Ġmethod Ġfollowing Ġthe Ġsuggested Ġparameter Ġsetup Ġin ĠB IB REF 17 Ġ. ĠMaximum Ġnorm Ġis Ġset Ġto Ġ5 Ġfor Ġgradient Ġclipping Ġ. ĠFor Ġregular ization , Ġwe Ġapply Ġdrop out Ġ( Ġ$ p = 0 . 8 $ Ġ) Ġon Ġthe Ġnon - rec urrent Ġconnections ĠB IB REF 18 Ġof ĠL ST M . ĠIn Ġaddition , Ġwe Ġapply Ġ$ L _ 2 $ Ġregular ization Ġ( Ġ$\ lambda Ġ= Ġ10 ^{ - 4 } $ Ġ) Ġon Ġthe Ġweights Ġand Ġbiases Ġof Ġthe ĠR NN Ġoutput Ġlayer . </s> Results Ġand ĠAnalysis </s> The Ġexperiment Ġresults Ġon Ġlanguage Ġmodeling Ġperplex ity Ġfor Ġmodels Ġusing Ġdifferent Ġdialog Ġturn Ġsize Ġare Ġshown Ġin ĠTable Ġ1 Ġ. Ġ$ K $ Ġvalue Ġindicates Ġthe Ġnumber Ġof Ġturns Ġin Ġthe Ġdialog . ĠPer plex ity Ġis Ġcalculated Ġon Ġthe Ġlast Ġturn , Ġwith Ġpreceding Ġturns Ġused Ġas Ġcontext Ġto Ġthe Ġmodel . </s> As Ġcan Ġbe Ġseen Ġfrom Ġthe Ġresults , Ġall ĠR NN Ġbased Ġmodels Ġoutper form Ġthe Ġn - gram Ġmodel Ġby Ġlarge Ġmargin . ĠThe ĠBo W - Context - R NN LM Ġand ĠDR NN LM Ġbeat Ġthe ĠSingle - Turn - R NN LM Ġconsistently . ĠOur Ġimplementation Ġof Ġthe Ġcontext Ġdependent ĠC CD CL M Ġperforms Ġworse Ġthan ĠSingle - Turn - R NN LM . ĠThis Ġmight Ġdue Ġto Ġfact Ġthat Ġthe Ġtarget Ġturn Ġword Ġprediction Ġdepends Ġtoo Ġmuch Ġon Ġthe Ġprevious Ġturn Ġcontext Ġvector , Ġwhich Ġconnects Ġdirectly Ġto Ġthe Ġhidden Ġstate Ġof Ġcurrent Ġturn ĠR NN Ġat Ġeach Ġtime Ġstep . ĠThe Ġmodel Ġperformance Ġon Ġtraining Ġset Ġmight Ġnot Ġgeneral ize Ġwell Ġduring Ġinference Ġgiven Ġthe Ġlimited Ġsize Ġof Ġthe Ġtraining Ġset . </s> The Ġproposed ĠID CL M Ġand ĠES ID CL M Ġbeat Ġthe Ġsingle Ġturn ĠR NN LM Ġconsistently Ġunder Ġdifferent Ġcontext Ġturn Ġsizes . ĠES ID CL M Ġshows Ġthe Ġbest Ġlanguage Ġmodeling Ġperformance Ġunder Ġdialog Ġturn Ġsize Ġof Ġ3 Ġand Ġ5 , Ġoutper forming ĠID CL M Ġby Ġa Ġsmall Ġmargin . ĠID CL M Ġbeats Ġall Ġbaseline Ġmodels Ġwhen Ġusing Ġdialog Ġturn Ġsize Ġof Ġ5 , Ġand Ġproduces Ġslightly Ġworse Ġperplex ity Ġthan ĠDR NN LM Ġwhen Ġusing Ġdialog Ġturn Ġsize Ġof Ġ3 . </s> To Ġanalyze Ġthe Ġbest Ġpotential Ġgain Ġthat Ġmay Ġbe Ġachieved Ġby Ġintroducing Ġlinguistic Ġcontext , Ġwe Ġcompare Ġthe Ġproposed Ġcontextual Ġmodels Ġto ĠDAC LM , Ġthe Ġmodel Ġthat Ġuses Ġtrue Ġdialog Ġact Ġhistory Ġfor Ġdialog Ġcontext Ġmodeling . ĠAs Ġshown Ġin ĠTable Ġ1 Ġ, Ġthe Ġgap Ġbetween Ġour Ġproposed Ġmodels Ġand ĠDAC LM Ġis Ġnot Ġwide . ĠThis Ġgives Ġa Ġpositive Ġhint Ġthat Ġthe Ġproposed Ġcontextual Ġmodels Ġmay Ġimplicitly Ġcapture Ġthe Ġdialog Ġcontext Ġstate Ġchanges . </s> For Ġfine - gr ained Ġanalyses Ġof Ġthe Ġmodel Ġperformance , Ġwe Ġfurther Ġcompute Ġthe Ġtest Ġset Ġperplex ity Ġper ĠPOS Ġtag Ġand Ġper Ġdialog Ġact Ġtag . ĠWe Ġselected Ġthe Ġmost Ġfrequent ĠPOS Ġtags Ġand Ġdialog Ġact Ġtags Ġin ĠSw DA Ġcorpus , Ġand Ġreport Ġthe Ġtag Ġbased Ġperplex ity Ġrelative Ġchanges Ġ( Ġ$\ % $ Ġ) Ġof Ġthe Ġproposed Ġmodels Ġcomparing Ġto ĠSingle - Turn - R NN LM . ĠA Ġnegative Ġnumber Ġindicates Ġperformance Ġgain . </s> Table Ġ2 Ġshows Ġthe Ġmodel Ġperplex ity Ġper ĠPOS Ġtag . ĠAll Ġthe Ġthree Ġcontext Ġdependent Ġmodels Ġproduce Ġconsistent Ġperformance Ġgain Ġover Ġthe ĠSingle - Turn - R NN LM Ġfor Ġpronouns , Ġpre pos itions , Ġand Ġad verbs , Ġwith Ġpronouns Ġhaving Ġthe Ġlargest Ġperplex ity Ġimprovement . ĠHowever , Ġthe Ġproposed Ġcontextual Ġmodels Ġare Ġless Ġeffective Ġin Ġcapturing Ġnoun s . ĠThis Ġsuggests Ġthat Ġthe Ġproposed Ġcontextual ĠR NN Ġlanguage Ġmodels Ġexploit Ġthe Ġcontext Ġto Ġachieve Ġsuperior Ġprediction Ġon Ġcertain Ġbut Ġnot Ġall ĠPOS Ġtypes . ĠFurther Ġexploration Ġon Ġthe Ġmodel Ġdesign Ġis Ġrequired Ġif Ġwe Ġwant Ġto Ġbetter Ġcapture Ġwords Ġof Ġa Ġspecific Ġtype . </s> For Ġthe Ġdialog Ġact Ġtag Ġbased Ġresults Ġin ĠTable Ġ3 Ġ, Ġthe Ġthree Ġcontextual Ġmodels Ġshow Ġconsistent Ġperformance Ġgain Ġon ĠStatement - non - op inion Ġtype Ġutter ances . ĠThe Ġperplex ity Ġchanges Ġfor Ġother Ġdialog Ġact Ġtags Ġvary Ġfor Ġdifferent Ġmodels . </s> Con clusions </s> In Ġthis Ġwork , Ġwe Ġpropose Ġtwo Ġdialog Ġcontext Ġlanguage Ġmodels Ġthat Ġwith Ġspecial Ġdesign Ġto Ġmodel Ġdialog Ġinteractions . ĠOur Ġevaluation Ġresults Ġon ĠSwitch board ĠDial og ĠAct ĠCorpus Ġshow Ġthat Ġthe Ġproposed Ġmodel Ġoutper form Ġconventional ĠR NN Ġlanguage Ġmodel Ġby Ġ3 . 3 %. ĠThe Ġproposed Ġmodels Ġalso Ġillustrate Ġadvantageous Ġperformance Ġover Ġseveral Ġcompetitive Ġcontextual Ġlanguage Ġmodels . ĠPer plex ity Ġof Ġthe Ġproposed Ġdialog Ġcontext Ġlanguage Ġmodels Ġis Ġhigher Ġthan Ġthat Ġof Ġthe Ġmodel Ġusing Ġtrue Ġdialog Ġact Ġtags Ġas Ġcontext Ġby Ġa Ġsmall Ġmargin . ĠThis Ġindicates Ġthat Ġthe Ġproposed Ġmodel Ġmay Ġimplicitly Ġcapture Ġthe Ġdialog Ġcontext Ġstate Ġfor Ġlanguage Ġmodeling .
Introduction Language model plays an important role in many natural language processing systems, such as in automatic speech recognition BIBREF0 , BIBREF1 and machine translation systems BIBREF2 , BIBREF3 . Recurrent neural network (RNN) based models BIBREF4 , BIBREF5 have recently shown success in language modeling, outperforming conventional n-gram based models. Long short-term memory BIBREF6 , BIBREF7 is a widely used RNN variant for language modeling due to its superior performance in capturing longer term dependencies. Conventional RNN based language model uses a hidden state to represent the summary of the preceding words in a sentence without considering context signals. Mikolov et al. proposed a context dependent RNN language model BIBREF8 by connecting a contextual vector to the RNN hidden state. This contextual vector is produced by applying Latent Dirichlet Allocation BIBREF9 on preceding text. Several other contextual language models were later proposed by using bag-of-word BIBREF10 and RNN methods BIBREF11 to learn larger context representation that beyond the target sentence. The previously proposed contextual language models treat preceding sentences as a sequence of inputs, and they are suitable for document level context modeling. In dialog modeling, however, dialog interactions between speakers play an important role. Modeling utterances in a dialog as a sequence of inputs might not well capture the pauses, turn-taking, and grounding phenomena BIBREF12 in a dialog. In this work, we propose contextual RNN language models that specially track the interactions between speakers. We expect such models to generate better representations of the dialog context. The remainder of the paper is organized as follows. In section 2, we introduce the background on contextual language modeling. In section 3, we describe the proposed dialog context language models. Section 4 discusses the evaluation procedures and results. Section 5 concludes the work. RNN Language Model A language model assigns a probability to a sequence of words $\mathbf {w}=(w_1, w_2, ..., w_{T})$ following probability distribution. Using the chain rule, the likelihood of the word sequence $\mathbf {w}$ can be factorized as: $$P(\mathbf {w}) = P(w_1, w_2, ..., w_{T}) = \prod _{t=1}^{T}P(w_{t}|w_{< t}) \\$$   (Eq. 2) At time step $t$ , the system input is the embedding of the word at index $t$ , and the system output is the probability distribution of the word at index $t+1$ . The RNN hidden state $h_t$ encodes the information of the word sequence up till current step: $$&h_t = \operatorname{RNN}(h_{t-1}, w_t) \\ &P(w_{t+1}|w_{< t+1}) = \operatorname{softmax}(W_{o}h_{t} + b_{o})$$   (Eq. 3) where $W_{o}$ and $b_{o}$ are the output layer weights and biases. Contextual RNN Language Model A number of methods have been proposed to introduce contextual information to the language model. Mikolov and Zweig BIBREF8 proposed a topic-conditioned RNNLM by introducing a contextual real-valued vector to RNN hidden state. The contextual vector was created by performing LDA BIBREF9 on preceding text. Wang and Cho BIBREF10 studied introducing corpus-level discourse information into language modeling. A number of context representation methods were explored, including bag-of-words, sequence of bag-of-words, and sequence of bag-of-words with attention. Lin et al. BIBREF13 proposed using hierarchical RNN for document modeling. Comparing to using bag-of-words and sequence of bag-of-words for document context representation, using hierarchical RNN can better model the order of words in preceding text, at the cost of the increased computational complexity. These contextual language models focused on contextual information at the document level. Tran et al. BIBREF14 further proposed a contextual language model that consider information at inter-document level. They claimed that by utilizing the structural information from a tree-structured document set, language modeling performance was largely improved. Methods The previously proposed contextual language models focus on applying context by encoding preceding text, without considering interactions in dialogs. These models may not be well suited for dialog language modeling, as they are not designed to capture dialog interactions, such as clarifications and confirmations. By making special design in learning dialog interactions, we expect the models to generate better representations of the dialog context, and thus lower perplexity of the target dialog turn or utterance. In this section, we first explain the context dependent RNN language model that operates on utterance or turn level. Following that, we describe the two proposed contextual language models that utilize the dialog level context. Context Dependent RNNLM Let $\mathbf {D} = (\mathbf {U}_1, \mathbf {U}_2, ..., \mathbf {U}_K)$ be a dialog that has $K$ turns and involves two speakers. Each turn may have one or more utterances. The $k$ th turn $\mathbf {U}_k = (w_1, w_2, ..., w_{T_k})$ is represented as a sequence of $T_k$ words. Conditioning on information of the preceding text in the dialog, probability of the target turn $\mathbf {U}_k$ can be calculated as: $$P(\mathbf {U}_k|\mathbf {U}_{<k}) = \prod _{t=1}^{T_k}P(w^{\mathbf {U}_{k}}_{t}|w^{\mathbf {U}_{k}}_{< t}, \mathbf {U}_{<k}) \\$$   (Eq. 6) where $\mathbf {U}_{<k}$ denotes all previous turns before $\mathbf {U}_k$ , and $w^{\mathbf {U}_{k}}_{< t}$ denotes all previous words before the $t$ th word in turn $\mathbf {U}_k$ . In context dependent RNN language model, the context vector $c$ is connected to the RNN hidden state together with the input word embedding at each time step (Figure 1 ). This is similar to the context dependent RNN language model proposed in BIBREF8 , other than that the context vector is not connected directly to the RNN output layer. With the additional context vector input $c$ , the RNN state $h_t$ is updated as: $$h_t = \operatorname{RNN}(h_{t-1}, [w_t, c])$$   (Eq. 8) Context Representations In neural network based language models, the dialog context can be represented as a dense continuous vector. This context vector can be produced in a number of ways. One simple approach is to use bag of word embeddings. However, bag of word embedding context representation does not take word order into consideration. An alternative approach is to use an RNN to read the preceding text. The last hidden state of the RNN encoder can be seen as the representation of the text and be used as the context vector for the next turn. To generate document level context representation, one may cascade all sentences in a document by removing the sentence boundaries. The last RNN hidden state of the previous utterance serves as the initial RNN state of the next utterance. As in BIBREF11 , we refer to this model as DRNNLM. Alternatively, in the CCDCLM model proposed in BIBREF11 , the last RNN hidden state of the previous utterance is fed to the RNN hidden state of the target utterance at each time step. Interactive Dialog Context LM The previously proposed contextual language models, such as DRNNLM and CCDCLM, treat dialog history as a sequence of inputs, without modeling dialog interactions. A dialog turn from one speaker may not only be a direct response to the other speaker's query, but also likely to be a continuation of his own previous statement. Thus, when modeling turn $k$ in a dialog, we propose to connect the last RNN state of turn $k-2$ directly to the starting RNN state of turn $k$ , instead of letting it to propagate through the RNN for turn $k-1$ . The last RNN state of turn $k-1$ serves as the context vector to turn $k$ , which is fed to turn $k$ 's RNN hidden state at each time step together with the word input. The model architecture is as shown in Figure 2 . The context vector $c$ and the initial RNN hidden state for the $k$ th turn $h^{\mathbf {U}_k}_{0}$ are defined as: $$c = h^{\mathbf {U}_{k-1}}_{T_{k-1}}, \; h^{\mathbf {U}_k}_{0} = h^{\mathbf {U}_{k-2}}_{T_{k-2}}$$   (Eq. 11) where $h^{\mathbf {U}_{k-1}}_{T_{k-1}}$ represents the last RNN hidden state of turn $k-1$ . This model also allows the context signal from previous turns to propagate through the network in fewer steps, which helps to reduce information loss along the propagation. We refer to this model as Interactive Dialog Context Language Model (IDCLM). External State Interactive Dialog Context LM The propagation of dialog context can be seen as a series of updates of a hidden dialog context state along the growing dialog. IDCLM models this hidden dialog context state changes implicitly in the turn level RNN state. Such dialog context state updates can also be modeled in a separated RNN. As shown in the architecture in Figure 3 , we use an external RNN to model the context changes explicitly. Input to the external state RNN is the vector representation of the previous dialog turns. The external state RNN output serves as the dialog context for next turn: $$s_{k-1} = \operatorname{RNN}_{ES}(s_{k-2}, h^{\mathbf {U}_{k-1}}_{T_{k-1}})$$   (Eq. 14) where $s_{k-1}$ is the output of the external state RNN after the processing of turn $k-1$ . The context vector $c$ and the initial RNN hidden state for the $k$ th turn $h^{\mathbf {U}_k}_{0}$ are then defined as: $$c = s_{k-1}, \; h^{\mathbf {U}_k}_{0} = h^{\mathbf {U}_{k-2}}_{T_{k-2}}$$   (Eq. 15) We refer to this model as External State Interactive Dialog Context Language Model (ESIDCLM). Comparing to IDCLM, ESIDCLM releases the burden of turn level RNN by using an external RNN to model dialog context state changes. One drawback of ESIDCLM is that there are additional RNN model parameters to be learned during model training, which may make the model more prone to overfitting when training data size is limited. Data Set We use the Switchboard Dialog Act Corpus (SwDA) in evaluating our contextual langauge models. The SwDA corpus extends the Switchboard-1 Telephone Speech Corpus with turn and utterance-level dialog act tags. The utterances are also tagged with part-of-speech (POS) tags. We split the data in folder sw00 to sw09 as training set, folder sw10 as test set, and folder sw11 to sw13 as validation set. The training, validation, and test sets contain 98.7K turns (190.0K utterances), 5.7K turns (11.3K utterances), and 11.9K turns (22.2K utterances) respectively. Maximum turn length is set to 160. The vocabulary is defined with the top frequent 10K words. Baselines We compare IDCLM and ESIDCLM to several baseline methods, including n-gram based model, single turn RNNLM, and various context dependent RNNLMs. 5-gram KN A 5-gram language model with modified Kneser-Ney smoothing BIBREF15 . Single-Turn-RNNLM Conventional RNNLM that operates on single turn level with no context information. BoW-Context-RNNLM Contextual RNNLM with BoW representation of preceding text as context. DRNNLM Contextual RNNLM with turn level context vector connected to initial RNN state of the target turn. CCDCLM Contextual RNNLM with turn level context vector connected to RNN hidden state of the target turn at each time step. We implement this model following the design in BIBREF11 . In order to investigate the potential performance gain that can be achieved by introducing context, we also compare the proposed methods to RNNLMs that use true dialog act tags as context. Although human labeled dialog act might not be the best option for modeling the dialog context state, it provides a reasonable estimation of the best gain that can be achieved by introducing linguistic context. The dialog act sequence is modeled by a separated RNN, similar to the external state RNN used in ESIDCLM. We refer to this model as Dialog Act Context Language Model (DACLM). DACLM RNNLM with true dialog act context vector connected to RNN state of the target turn at each time step. Model Configuration and Training In this work, we use LSTM cell BIBREF6 as the basic RNN unit for its stronger capability in capturing long-range dependencies in a word sequence comparing to simple RNN. We use pre-trained word vectors BIBREF16 that are trained on Google News dataset to initialize the word embeddings. These word embeddings are fine-tuned during model training. We conduct mini-batch training using Adam optimization method following the suggested parameter setup in BIBREF17 . Maximum norm is set to 5 for gradient clipping . For regularization, we apply dropout ( $p=0.8$ ) on the non-recurrent connections BIBREF18 of LSTM. In addition, we apply $L_2$ regularization ( $\lambda = 10^{-4}$ ) on the weights and biases of the RNN output layer. Results and Analysis The experiment results on language modeling perplexity for models using different dialog turn size are shown in Table 1 . $K$ value indicates the number of turns in the dialog. Perplexity is calculated on the last turn, with preceding turns used as context to the model. As can be seen from the results, all RNN based models outperform the n-gram model by large margin. The BoW-Context-RNNLM and DRNNLM beat the Single-Turn-RNNLM consistently. Our implementation of the context dependent CCDCLM performs worse than Single-Turn-RNNLM. This might due to fact that the target turn word prediction depends too much on the previous turn context vector, which connects directly to the hidden state of current turn RNN at each time step. The model performance on training set might not generalize well during inference given the limited size of the training set. The proposed IDCLM and ESIDCLM beat the single turn RNNLM consistently under different context turn sizes. ESIDCLM shows the best language modeling performance under dialog turn size of 3 and 5, outperforming IDCLM by a small margin. IDCLM beats all baseline models when using dialog turn size of 5, and produces slightly worse perplexity than DRNNLM when using dialog turn size of 3. To analyze the best potential gain that may be achieved by introducing linguistic context, we compare the proposed contextual models to DACLM, the model that uses true dialog act history for dialog context modeling. As shown in Table 1 , the gap between our proposed models and DACLM is not wide. This gives a positive hint that the proposed contextual models may implicitly capture the dialog context state changes. For fine-grained analyses of the model performance, we further compute the test set perplexity per POS tag and per dialog act tag. We selected the most frequent POS tags and dialog act tags in SwDA corpus, and report the tag based perplexity relative changes ( $\%$ ) of the proposed models comparing to Single-Turn-RNNLM. A negative number indicates performance gain. Table 2 shows the model perplexity per POS tag. All the three context dependent models produce consistent performance gain over the Single-Turn-RNNLM for pronouns, prepositions, and adverbs, with pronouns having the largest perplexity improvement. However, the proposed contextual models are less effective in capturing nouns. This suggests that the proposed contextual RNN language models exploit the context to achieve superior prediction on certain but not all POS types. Further exploration on the model design is required if we want to better capture words of a specific type. For the dialog act tag based results in Table 3 , the three contextual models show consistent performance gain on Statement-non-opinion type utterances. The perplexity changes for other dialog act tags vary for different models. Conclusions In this work, we propose two dialog context language models that with special design to model dialog interactions. Our evaluation results on Switchboard Dialog Act Corpus show that the proposed model outperform conventional RNN language model by 3.3%. The proposed models also illustrate advantageous performance over several competitive contextual language models. Perplexity of the proposed dialog context language models is higher than that of the model using true dialog act tags as context by a small margin. This indicates that the proposed model may implicitly capture the dialog context state for language modeling.
How long of dialog history is captured?
5260cb56b7d127772425583c5c28958c37cb9bea
How Ġlong Ġof Ġdialog Ġhistory Ġis Ġcaptured ?
two Ġprevious Ġturns
<s> How Ġlong Ġof Ġdialog Ġhistory Ġis Ġcaptured ? </s> Introduction </s> Language Ġmodel Ġplays Ġan Ġimportant Ġrole Ġin Ġmany Ġnatural Ġlanguage Ġprocessing Ġsystems , Ġsuch Ġas Ġin Ġautomatic Ġspeech Ġrecognition ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġand Ġmachine Ġtranslation Ġsystems ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ. ĠRec urrent Ġneural Ġnetwork Ġ( R NN ) Ġbased Ġmodels ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġhave Ġrecently Ġshown Ġsuccess Ġin Ġlanguage Ġmodeling , Ġoutper forming Ġconventional Ġn - gram Ġbased Ġmodels . ĠLong Ġshort - term Ġmemory ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġis Ġa Ġwidely Ġused ĠR NN Ġvariant Ġfor Ġlanguage Ġmodeling Ġdue Ġto Ġits Ġsuperior Ġperformance Ġin Ġcapturing Ġlonger Ġterm Ġdependencies . </s> Con ventional ĠR NN Ġbased Ġlanguage Ġmodel Ġuses Ġa Ġhidden Ġstate Ġto Ġrepresent Ġthe Ġsummary Ġof Ġthe Ġpreceding Ġwords Ġin Ġa Ġsentence Ġwithout Ġconsidering Ġcontext Ġsignals . ĠMik ol ov Ġet Ġal . Ġproposed Ġa Ġcontext Ġdependent ĠR NN Ġlanguage Ġmodel ĠB IB REF 8 Ġby Ġconnecting Ġa Ġcontextual Ġvector Ġto Ġthe ĠR NN Ġhidden Ġstate . ĠThis Ġcontextual Ġvector Ġis Ġproduced Ġby Ġapplying ĠLat ent ĠDir ich let ĠAll ocation ĠB IB REF 9 Ġon Ġpreceding Ġtext . ĠSeveral Ġother Ġcontextual Ġlanguage Ġmodels Ġwere Ġlater Ġproposed Ġby Ġusing Ġbag - of - word ĠB IB REF 10 Ġand ĠR NN Ġmethods ĠB IB REF 11 Ġto Ġlearn Ġlarger Ġcontext Ġrepresentation Ġthat Ġbeyond Ġthe Ġtarget Ġsentence . </s> The Ġpreviously Ġproposed Ġcontextual Ġlanguage Ġmodels Ġtreat Ġpreceding Ġsentences Ġas Ġa Ġsequence Ġof Ġinputs , Ġand Ġthey Ġare Ġsuitable Ġfor Ġdocument Ġlevel Ġcontext Ġmodeling . ĠIn Ġdialog Ġmodeling , Ġhowever , Ġdialog Ġinteractions Ġbetween Ġspeakers Ġplay Ġan Ġimportant Ġrole . ĠModel ing Ġutter ances Ġin Ġa Ġdialog Ġas Ġa Ġsequence Ġof Ġinputs Ġmight Ġnot Ġwell Ġcapture Ġthe Ġpauses , Ġturn - taking , Ġand Ġgrounding Ġphenomena ĠB IB REF 12 Ġin Ġa Ġdialog . ĠIn Ġthis Ġwork , Ġwe Ġpropose Ġcontextual ĠR NN Ġlanguage Ġmodels Ġthat Ġspecially Ġtrack Ġthe Ġinteractions Ġbetween Ġspeakers . ĠWe Ġexpect Ġsuch Ġmodels Ġto Ġgenerate Ġbetter Ġrepresentations Ġof Ġthe Ġdialog Ġcontext . </s> The Ġremainder Ġof Ġthe Ġpaper Ġis Ġorganized Ġas Ġfollows . ĠIn Ġsection Ġ2 , Ġwe Ġintroduce Ġthe Ġbackground Ġon Ġcontextual Ġlanguage Ġmodeling . ĠIn Ġsection Ġ3 , Ġwe Ġdescribe Ġthe Ġproposed Ġdialog Ġcontext Ġlanguage Ġmodels . ĠSection Ġ4 Ġdiscusses Ġthe Ġevaluation Ġprocedures Ġand Ġresults . ĠSection Ġ5 Ġconcludes Ġthe Ġwork . </s> R NN ĠLanguage ĠModel </s> A Ġlanguage Ġmodel Ġassigns Ġa Ġprobability Ġto Ġa Ġsequence Ġof Ġwords Ġ$\ math bf Ġ{ w } =( w _ 1 , Ġw _ 2 , Ġ... , Ġw _{ T }) $ Ġfollowing Ġprobability Ġdistribution . ĠUsing Ġthe Ġchain Ġrule , Ġthe Ġlikelihood Ġof Ġthe Ġword Ġsequence Ġ$\ math bf Ġ{ w } $ Ġcan Ġbe Ġfactor ized Ġas : </s> $$ P (\ math bf Ġ{ w }) Ġ= ĠP ( w _ 1 , Ġw _ 2 , Ġ... , Ġw _{ T }) Ġ= Ġ\ pro d Ġ_ { t = 1 } ^{ T } P ( w _{ t } | w _{ < Ġt }) Ġ\\ $$ Ġ Ġ Ġ( E q . Ġ2 ) </s> At Ġtime Ġstep Ġ$ t $ Ġ, Ġthe Ġsystem Ġinput Ġis Ġthe Ġembed ding Ġof Ġthe Ġword Ġat Ġindex Ġ$ t $ Ġ, Ġand Ġthe Ġsystem Ġoutput Ġis Ġthe Ġprobability Ġdistribution Ġof Ġthe Ġword Ġat Ġindex Ġ$ t + 1 $ Ġ. ĠThe ĠR NN Ġhidden Ġstate Ġ$ h _ t $ Ġenc odes Ġthe Ġinformation Ġof Ġthe Ġword Ġsequence Ġup Ġtill Ġcurrent Ġstep : </s> $$ & h _ t Ġ= Ġ\ oper at orn ame { R NN } ( h _{ t - 1 }, Ġw _ t ) Ġ\\ Ġ& P ( w _{ t + 1 } | w _{ < Ġt + 1 }) Ġ= Ġ\ oper at orn ame { soft max } ( W _{ o } h _{ t } Ġ+ Ġb _{ o }) $$ Ġ Ġ Ġ( E q . Ġ3 ) </s> where Ġ$ W _{ o } $ Ġand Ġ$ b _{ o } $ Ġare Ġthe Ġoutput Ġlayer Ġweights Ġand Ġbiases . </s> Context ual ĠR NN ĠLanguage ĠModel </s> A Ġnumber Ġof Ġmethods Ġhave Ġbeen Ġproposed Ġto Ġintroduce Ġcontextual Ġinformation Ġto Ġthe Ġlanguage Ġmodel . ĠMik ol ov Ġand ĠZ we ig ĠB IB REF 8 Ġproposed Ġa Ġtopic - condition ed ĠR NN LM Ġby Ġintroducing Ġa Ġcontextual Ġreal - valued Ġvector Ġto ĠR NN Ġhidden Ġstate . ĠThe Ġcontextual Ġvector Ġwas Ġcreated Ġby Ġperforming ĠL DA ĠB IB REF 9 Ġon Ġpreceding Ġtext . ĠWang Ġand ĠCho ĠB IB REF 10 Ġstudied Ġintroducing Ġcorpus - level Ġdiscourse Ġinformation Ġinto Ġlanguage Ġmodeling . ĠA Ġnumber Ġof Ġcontext Ġrepresentation Ġmethods Ġwere Ġexplored , Ġincluding Ġbag - of - words , Ġsequence Ġof Ġbag - of - words , Ġand Ġsequence Ġof Ġbag - of - words Ġwith Ġattention . ĠLin Ġet Ġal . ĠB IB REF 13 Ġproposed Ġusing Ġhierarchical ĠR NN Ġfor Ġdocument Ġmodeling . ĠComp aring Ġto Ġusing Ġbag - of - words Ġand Ġsequence Ġof Ġbag - of - words Ġfor Ġdocument Ġcontext Ġrepresentation , Ġusing Ġhierarchical ĠR NN Ġcan Ġbetter Ġmodel Ġthe Ġorder Ġof Ġwords Ġin Ġpreceding Ġtext , Ġat Ġthe Ġcost Ġof Ġthe Ġincreased Ġcomputational Ġcomplexity . ĠThese Ġcontextual Ġlanguage Ġmodels Ġfocused Ġon Ġcontextual Ġinformation Ġat Ġthe Ġdocument Ġlevel . ĠTr an Ġet Ġal . ĠB IB REF 14 Ġfurther Ġproposed Ġa Ġcontextual Ġlanguage Ġmodel Ġthat Ġconsider Ġinformation Ġat Ġinter - document Ġlevel . ĠThey Ġclaimed Ġthat Ġby Ġutilizing Ġthe Ġstructural Ġinformation Ġfrom Ġa Ġtree - struct ured Ġdocument Ġset , Ġlanguage Ġmodeling Ġperformance Ġwas Ġlargely Ġimproved . </s> Methods </s> The Ġpreviously Ġproposed Ġcontextual Ġlanguage Ġmodels Ġfocus Ġon Ġapplying Ġcontext Ġby Ġencoding Ġpreceding Ġtext , Ġwithout Ġconsidering Ġinteractions Ġin Ġdialog s . ĠThese Ġmodels Ġmay Ġnot Ġbe Ġwell Ġsuited Ġfor Ġdialog Ġlanguage Ġmodeling , Ġas Ġthey Ġare Ġnot Ġdesigned Ġto Ġcapture Ġdialog Ġinteractions , Ġsuch Ġas Ġclar ifications Ġand Ġconfirm ations . ĠBy Ġmaking Ġspecial Ġdesign Ġin Ġlearning Ġdialog Ġinteractions , Ġwe Ġexpect Ġthe Ġmodels Ġto Ġgenerate Ġbetter Ġrepresentations Ġof Ġthe Ġdialog Ġcontext , Ġand Ġthus Ġlower Ġperplex ity Ġof Ġthe Ġtarget Ġdialog Ġturn Ġor Ġutter ance . </s> In Ġthis Ġsection , Ġwe Ġfirst Ġexplain Ġthe Ġcontext Ġdependent ĠR NN Ġlanguage Ġmodel Ġthat Ġoperates Ġon Ġutter ance Ġor Ġturn Ġlevel . ĠFollowing Ġthat , Ġwe Ġdescribe Ġthe Ġtwo Ġproposed Ġcontextual Ġlanguage Ġmodels Ġthat Ġutilize Ġthe Ġdialog Ġlevel Ġcontext . </s> Context ĠDepend ent ĠR NN LM </s> Let Ġ$\ math bf Ġ{ D } Ġ= Ġ( \ math bf Ġ{ U } _ 1 , Ġ\ math bf Ġ{ U } _ 2 , Ġ... , Ġ\ math bf Ġ{ U } _ K ) $ Ġbe Ġa Ġdialog Ġthat Ġhas Ġ$ K $ Ġturns Ġand Ġinvolves Ġtwo Ġspeakers . ĠEach Ġturn Ġmay Ġhave Ġone Ġor Ġmore Ġutter ances . ĠThe Ġ$ k $ Ġth Ġturn Ġ$\ math bf Ġ{ U } _ k Ġ= Ġ( w _ 1 , Ġw _ 2 , Ġ... , Ġw _{ T _ k }) $ Ġis Ġrepresented Ġas Ġa Ġsequence Ġof Ġ$ T _ k $ Ġwords . ĠCondition ing Ġon Ġinformation Ġof Ġthe Ġpreceding Ġtext Ġin Ġthe Ġdialog , Ġprobability Ġof Ġthe Ġtarget Ġturn Ġ$\ math bf Ġ{ U } _ k $ Ġcan Ġbe Ġcalculated Ġas : </s> $$ P (\ math bf Ġ{ U } _ k | \ math bf Ġ{ U } _{ < k }) Ġ= Ġ\ pro d Ġ_ { t = 1 } ^{ T _ k } P ( w ^ {\ math bf Ġ{ U } _{ k }} _{ t } | w ^ {\ math bf Ġ{ U } _{ k }} _{ < Ġt }, Ġ\ math bf Ġ{ U } _{ < k }) Ġ\\ $$ Ġ Ġ Ġ( E q . Ġ6 ) </s> where Ġ$\ math bf Ġ{ U } _{ < k } $ Ġdenotes Ġall Ġprevious Ġturns Ġbefore Ġ$\ math bf Ġ{ U } _ k $ Ġ, Ġand Ġ$ w ^ {\ math bf Ġ{ U } _{ k }} _{ < Ġt } $ Ġdenotes Ġall Ġprevious Ġwords Ġbefore Ġthe Ġ$ t $ Ġth Ġword Ġin Ġturn Ġ$\ math bf Ġ{ U } _ k $ Ġ. </s> In Ġcontext Ġdependent ĠR NN Ġlanguage Ġmodel , Ġthe Ġcontext Ġvector Ġ$ c $ Ġis Ġconnected Ġto Ġthe ĠR NN Ġhidden Ġstate Ġtogether Ġwith Ġthe Ġinput Ġword Ġembed ding Ġat Ġeach Ġtime Ġstep Ġ( Figure Ġ1 Ġ). ĠThis Ġis Ġsimilar Ġto Ġthe Ġcontext Ġdependent ĠR NN Ġlanguage Ġmodel Ġproposed Ġin ĠB IB REF 8 Ġ, Ġother Ġthan Ġthat Ġthe Ġcontext Ġvector Ġis Ġnot Ġconnected Ġdirectly Ġto Ġthe ĠR NN Ġoutput Ġlayer . ĠWith Ġthe Ġadditional Ġcontext Ġvector Ġinput Ġ$ c $ Ġ, Ġthe ĠR NN Ġstate Ġ$ h _ t $ Ġis Ġupdated Ġas : </s> $$ h _ t Ġ= Ġ\ oper at orn ame { R NN } ( h _{ t - 1 }, Ġ[ w _ t , Ġc ]) $$ Ġ Ġ Ġ( E q . Ġ8 ) </s> Context ĠRepresent ations </s> In Ġneural Ġnetwork Ġbased Ġlanguage Ġmodels , Ġthe Ġdialog Ġcontext Ġcan Ġbe Ġrepresented Ġas Ġa Ġdense Ġcontinuous Ġvector . ĠThis Ġcontext Ġvector Ġcan Ġbe Ġproduced Ġin Ġa Ġnumber Ġof Ġways . </s> One Ġsimple Ġapproach Ġis Ġto Ġuse Ġbag Ġof Ġword Ġembed d ings . ĠHowever , Ġbag Ġof Ġword Ġembed ding Ġcontext Ġrepresentation Ġdoes Ġnot Ġtake Ġword Ġorder Ġinto Ġconsideration . ĠAn Ġalternative Ġapproach Ġis Ġto Ġuse Ġan ĠR NN Ġto Ġread Ġthe Ġpreceding Ġtext . ĠThe Ġlast Ġhidden Ġstate Ġof Ġthe ĠR NN Ġenc oder Ġcan Ġbe Ġseen Ġas Ġthe Ġrepresentation Ġof Ġthe Ġtext Ġand Ġbe Ġused Ġas Ġthe Ġcontext Ġvector Ġfor Ġthe Ġnext Ġturn . ĠTo Ġgenerate Ġdocument Ġlevel Ġcontext Ġrepresentation , Ġone Ġmay Ġcascade Ġall Ġsentences Ġin Ġa Ġdocument Ġby Ġremoving Ġthe Ġsentence Ġboundaries . ĠThe Ġlast ĠR NN Ġhidden Ġstate Ġof Ġthe Ġprevious Ġutter ance Ġserves Ġas Ġthe Ġinitial ĠR NN Ġstate Ġof Ġthe Ġnext Ġutter ance . ĠAs Ġin ĠB IB REF 11 Ġ, Ġwe Ġrefer Ġto Ġthis Ġmodel Ġas ĠDR NN LM . ĠAlternatively , Ġin Ġthe ĠC CD CL M Ġmodel Ġproposed Ġin ĠB IB REF 11 Ġ, Ġthe Ġlast ĠR NN Ġhidden Ġstate Ġof Ġthe Ġprevious Ġutter ance Ġis Ġfed Ġto Ġthe ĠR NN Ġhidden Ġstate Ġof Ġthe Ġtarget Ġutter ance Ġat Ġeach Ġtime Ġstep . </s> Inter active ĠDial og ĠContext ĠLM </s> The Ġpreviously Ġproposed Ġcontextual Ġlanguage Ġmodels , Ġsuch Ġas ĠDR NN LM Ġand ĠC CD CL M , Ġtreat Ġdialog Ġhistory Ġas Ġa Ġsequence Ġof Ġinputs , Ġwithout Ġmodeling Ġdialog Ġinteractions . ĠA Ġdialog Ġturn Ġfrom Ġone Ġspeaker Ġmay Ġnot Ġonly Ġbe Ġa Ġdirect Ġresponse Ġto Ġthe Ġother Ġspeaker 's Ġquery , Ġbut Ġalso Ġlikely Ġto Ġbe Ġa Ġcontinuation Ġof Ġhis Ġown Ġprevious Ġstatement . ĠThus , Ġwhen Ġmodeling Ġturn Ġ$ k $ Ġin Ġa Ġdialog , Ġwe Ġpropose Ġto Ġconnect Ġthe Ġlast ĠR NN Ġstate Ġof Ġturn Ġ$ k - 2 $ Ġdirectly Ġto Ġthe Ġstarting ĠR NN Ġstate Ġof Ġturn Ġ$ k $ Ġ, Ġinstead Ġof Ġletting Ġit Ġto Ġpropagate Ġthrough Ġthe ĠR NN Ġfor Ġturn Ġ$ k - 1 $ Ġ. ĠThe Ġlast ĠR NN Ġstate Ġof Ġturn Ġ$ k - 1 $ Ġserves Ġas Ġthe Ġcontext Ġvector Ġto Ġturn Ġ$ k $ Ġ, Ġwhich Ġis Ġfed Ġto Ġturn Ġ$ k $ Ġ' s ĠR NN Ġhidden Ġstate Ġat Ġeach Ġtime Ġstep Ġtogether Ġwith Ġthe Ġword Ġinput . ĠThe Ġmodel Ġarchitecture Ġis Ġas Ġshown Ġin ĠFigure Ġ2 Ġ. ĠThe Ġcontext Ġvector Ġ$ c $ Ġand Ġthe Ġinitial ĠR NN Ġhidden Ġstate Ġfor Ġthe Ġ$ k $ Ġth Ġturn Ġ$ h ^ {\ math bf Ġ{ U } _ k } _{ 0 } $ Ġare Ġdefined Ġas : </s> $$ c Ġ= Ġh ^ {\ math bf Ġ{ U } _{ k - 1 }} _{ T _{ k - 1 } }, Ġ\ ; Ġh ^ {\ math bf Ġ{ U } _ k } _{ 0 } Ġ= Ġh ^ {\ math bf Ġ{ U } _{ k - 2 }} _{ T _{ k - 2 }} $$ Ġ Ġ Ġ( E q . Ġ11 ) </s> where Ġ$ h ^ {\ math bf Ġ{ U } _{ k - 1 }} _{ T _{ k - 1 }} $ Ġrepresents Ġthe Ġlast ĠR NN Ġhidden Ġstate Ġof Ġturn Ġ$ k - 1 $ Ġ. ĠThis Ġmodel Ġalso Ġallows Ġthe Ġcontext Ġsignal Ġfrom Ġprevious Ġturns Ġto Ġpropagate Ġthrough Ġthe Ġnetwork Ġin Ġfewer Ġsteps , Ġwhich Ġhelps Ġto Ġreduce Ġinformation Ġloss Ġalong Ġthe Ġpropagation . ĠWe Ġrefer Ġto Ġthis Ġmodel Ġas ĠInteractive ĠDial og ĠContext ĠLanguage ĠModel Ġ( ID CL M ). </s> External ĠState ĠInteractive ĠDial og ĠContext ĠLM </s> The Ġpropagation Ġof Ġdialog Ġcontext Ġcan Ġbe Ġseen Ġas Ġa Ġseries Ġof Ġupdates Ġof Ġa Ġhidden Ġdialog Ġcontext Ġstate Ġalong Ġthe Ġgrowing Ġdialog . ĠID CL M Ġmodels Ġthis Ġhidden Ġdialog Ġcontext Ġstate Ġchanges Ġimplicitly Ġin Ġthe Ġturn Ġlevel ĠR NN Ġstate . ĠSuch Ġdialog Ġcontext Ġstate Ġupdates Ġcan Ġalso Ġbe Ġmodeled Ġin Ġa Ġseparated ĠR NN . ĠAs Ġshown Ġin Ġthe Ġarchitecture Ġin ĠFigure Ġ3 Ġ, Ġwe Ġuse Ġan Ġexternal ĠR NN Ġto Ġmodel Ġthe Ġcontext Ġchanges Ġexplicitly . ĠInput Ġto Ġthe Ġexternal Ġstate ĠR NN Ġis Ġthe Ġvector Ġrepresentation Ġof Ġthe Ġprevious Ġdialog Ġturns . ĠThe Ġexternal Ġstate ĠR NN Ġoutput Ġserves Ġas Ġthe Ġdialog Ġcontext Ġfor Ġnext Ġturn : </s> $$ s _{ k - 1 } Ġ= Ġ\ oper at orn ame { R NN } _{ ES } ( s _{ k - 2 }, Ġh ^ {\ math bf Ġ{ U } _{ k - 1 }} _{ T _{ k - 1 }} ) $$ Ġ Ġ Ġ( E q . Ġ14 ) </s> where Ġ$ s _{ k - 1 } $ Ġis Ġthe Ġoutput Ġof Ġthe Ġexternal Ġstate ĠR NN Ġafter Ġthe Ġprocessing Ġof Ġturn Ġ$ k - 1 $ Ġ. ĠThe Ġcontext Ġvector Ġ$ c $ Ġand Ġthe Ġinitial ĠR NN Ġhidden Ġstate Ġfor Ġthe Ġ$ k $ Ġth Ġturn Ġ$ h ^ {\ math bf Ġ{ U } _ k } _{ 0 } $ Ġare Ġthen Ġdefined Ġas : </s> $$ c Ġ= Ġs _{ k - 1 }, Ġ\ ; Ġh ^ {\ math bf Ġ{ U } _ k } _{ 0 } Ġ= Ġh ^ {\ math bf Ġ{ U } _{ k - 2 }} _{ T _{ k - 2 }} $$ Ġ Ġ Ġ( E q . Ġ15 ) </s> We Ġrefer Ġto Ġthis Ġmodel Ġas ĠExternal ĠState ĠInteractive ĠDial og ĠContext ĠLanguage ĠModel Ġ( ES ID CL M ). </s> Comp aring Ġto ĠID CL M , ĠES ID CL M Ġreleases Ġthe Ġburden Ġof Ġturn Ġlevel ĠR NN Ġby Ġusing Ġan Ġexternal ĠR NN Ġto Ġmodel Ġdialog Ġcontext Ġstate Ġchanges . ĠOne Ġdrawback Ġof ĠES ID CL M Ġis Ġthat Ġthere Ġare Ġadditional ĠR NN Ġmodel Ġparameters Ġto Ġbe Ġlearned Ġduring Ġmodel Ġtraining , Ġwhich Ġmay Ġmake Ġthe Ġmodel Ġmore Ġprone Ġto Ġover fitting Ġwhen Ġtraining Ġdata Ġsize Ġis Ġlimited . </s> Data ĠSet </s> We Ġuse Ġthe ĠSwitch board ĠDial og ĠAct ĠCorpus Ġ( Sw DA ) Ġin Ġevaluating Ġour Ġcontextual Ġlang au ge Ġmodels . ĠThe ĠSw DA Ġcorpus Ġextends Ġthe ĠSwitch board - 1 ĠTelephone ĠSpeech ĠCorpus Ġwith Ġturn Ġand Ġutter ance - level Ġdialog Ġact Ġtags . ĠThe Ġutter ances Ġare Ġalso Ġtagged Ġwith Ġpart - of - speech Ġ( POS ) Ġtags . ĠWe Ġsplit Ġthe Ġdata Ġin Ġfolder Ġsw 00 Ġto Ġsw 09 Ġas Ġtraining Ġset , Ġfolder Ġsw 10 Ġas Ġtest Ġset , Ġand Ġfolder Ġsw 11 Ġto Ġsw 13 Ġas Ġvalidation Ġset . ĠThe Ġtraining , Ġvalidation , Ġand Ġtest Ġsets Ġcontain Ġ98 . 7 K Ġturns Ġ( 190 . 0 K Ġutter ances ), Ġ5 . 7 K Ġturns Ġ( 11 . 3 K Ġutter ances ), Ġand Ġ11 . 9 K Ġturns Ġ( 22 . 2 K Ġutter ances ) Ġrespectively . ĠMaximum Ġturn Ġlength Ġis Ġset Ġto Ġ160 . ĠThe Ġvocabulary Ġis Ġdefined Ġwith Ġthe Ġtop Ġfrequent Ġ10 K Ġwords . </s> Bas elines </s> We Ġcompare ĠID CL M Ġand ĠES ID CL M Ġto Ġseveral Ġbaseline Ġmethods , Ġincluding Ġn - gram Ġbased Ġmodel , Ġsingle Ġturn ĠR NN LM , Ġand Ġvarious Ġcontext Ġdependent ĠR NN L Ms . </s> 5 - gram ĠKN ĠA Ġ5 - gram Ġlanguage Ġmodel Ġwith Ġmodified ĠK nes er - N ey Ġsmoot hing ĠB IB REF 15 Ġ. </s> Single - Turn - R NN LM ĠCon ventional ĠR NN LM Ġthat Ġoperates Ġon Ġsingle Ġturn Ġlevel Ġwith Ġno Ġcontext Ġinformation . </s> Bo W - Context - R NN LM ĠContext ual ĠR NN LM Ġwith ĠBo W Ġrepresentation Ġof Ġpreceding Ġtext Ġas Ġcontext . </s> DR NN LM ĠContext ual ĠR NN LM Ġwith Ġturn Ġlevel Ġcontext Ġvector Ġconnected Ġto Ġinitial ĠR NN Ġstate Ġof Ġthe Ġtarget Ġturn . </s> CC D CL M ĠContext ual ĠR NN LM Ġwith Ġturn Ġlevel Ġcontext Ġvector Ġconnected Ġto ĠR NN Ġhidden Ġstate Ġof Ġthe Ġtarget Ġturn Ġat Ġeach Ġtime Ġstep . ĠWe Ġimplement Ġthis Ġmodel Ġfollowing Ġthe Ġdesign Ġin ĠB IB REF 11 Ġ. </s> In Ġorder Ġto Ġinvestigate Ġthe Ġpotential Ġperformance Ġgain Ġthat Ġcan Ġbe Ġachieved Ġby Ġintroducing Ġcontext , Ġwe Ġalso Ġcompare Ġthe Ġproposed Ġmethods Ġto ĠR NN L Ms Ġthat Ġuse Ġtrue Ġdialog Ġact Ġtags Ġas Ġcontext . ĠAlthough Ġhuman Ġlabeled Ġdialog Ġact Ġmight Ġnot Ġbe Ġthe Ġbest Ġoption Ġfor Ġmodeling Ġthe Ġdialog Ġcontext Ġstate , Ġit Ġprovides Ġa Ġreasonable Ġestimation Ġof Ġthe Ġbest Ġgain Ġthat Ġcan Ġbe Ġachieved Ġby Ġintroducing Ġlinguistic Ġcontext . ĠThe Ġdialog Ġact Ġsequence Ġis Ġmodeled Ġby Ġa Ġseparated ĠR NN , Ġsimilar Ġto Ġthe Ġexternal Ġstate ĠR NN Ġused Ġin ĠES ID CL M . ĠWe Ġrefer Ġto Ġthis Ġmodel Ġas ĠDial og ĠAct ĠContext ĠLanguage ĠModel Ġ( D AC LM ). </s> D AC LM ĠR NN LM Ġwith Ġtrue Ġdialog Ġact Ġcontext Ġvector Ġconnected Ġto ĠR NN Ġstate Ġof Ġthe Ġtarget Ġturn Ġat Ġeach Ġtime Ġstep . </s> Model ĠConfiguration Ġand ĠTraining </s> In Ġthis Ġwork , Ġwe Ġuse ĠL ST M Ġcell ĠB IB REF 6 Ġas Ġthe Ġbasic ĠR NN Ġunit Ġfor Ġits Ġstronger Ġcapability Ġin Ġcapturing Ġlong - range Ġdependencies Ġin Ġa Ġword Ġsequence Ġcomparing Ġto Ġsimple ĠR NN . ĠWe Ġuse Ġpre - trained Ġword Ġvectors ĠB IB REF 16 Ġthat Ġare Ġtrained Ġon ĠGoogle ĠNews Ġdataset Ġto Ġinitialize Ġthe Ġword Ġembed d ings . ĠThese Ġword Ġembed d ings Ġare Ġfine - tun ed Ġduring Ġmodel Ġtraining . ĠWe Ġconduct Ġmini - batch Ġtraining Ġusing ĠAdam Ġoptimization Ġmethod Ġfollowing Ġthe Ġsuggested Ġparameter Ġsetup Ġin ĠB IB REF 17 Ġ. ĠMaximum Ġnorm Ġis Ġset Ġto Ġ5 Ġfor Ġgradient Ġclipping Ġ. ĠFor Ġregular ization , Ġwe Ġapply Ġdrop out Ġ( Ġ$ p = 0 . 8 $ Ġ) Ġon Ġthe Ġnon - rec urrent Ġconnections ĠB IB REF 18 Ġof ĠL ST M . ĠIn Ġaddition , Ġwe Ġapply Ġ$ L _ 2 $ Ġregular ization Ġ( Ġ$\ lambda Ġ= Ġ10 ^{ - 4 } $ Ġ) Ġon Ġthe Ġweights Ġand Ġbiases Ġof Ġthe ĠR NN Ġoutput Ġlayer . </s> Results Ġand ĠAnalysis </s> The Ġexperiment Ġresults Ġon Ġlanguage Ġmodeling Ġperplex ity Ġfor Ġmodels Ġusing Ġdifferent Ġdialog Ġturn Ġsize Ġare Ġshown Ġin ĠTable Ġ1 Ġ. Ġ$ K $ Ġvalue Ġindicates Ġthe Ġnumber Ġof Ġturns Ġin Ġthe Ġdialog . ĠPer plex ity Ġis Ġcalculated Ġon Ġthe Ġlast Ġturn , Ġwith Ġpreceding Ġturns Ġused Ġas Ġcontext Ġto Ġthe Ġmodel . </s> As Ġcan Ġbe Ġseen Ġfrom Ġthe Ġresults , Ġall ĠR NN Ġbased Ġmodels Ġoutper form Ġthe Ġn - gram Ġmodel Ġby Ġlarge Ġmargin . ĠThe ĠBo W - Context - R NN LM Ġand ĠDR NN LM Ġbeat Ġthe ĠSingle - Turn - R NN LM Ġconsistently . ĠOur Ġimplementation Ġof Ġthe Ġcontext Ġdependent ĠC CD CL M Ġperforms Ġworse Ġthan ĠSingle - Turn - R NN LM . ĠThis Ġmight Ġdue Ġto Ġfact Ġthat Ġthe Ġtarget Ġturn Ġword Ġprediction Ġdepends Ġtoo Ġmuch Ġon Ġthe Ġprevious Ġturn Ġcontext Ġvector , Ġwhich Ġconnects Ġdirectly Ġto Ġthe Ġhidden Ġstate Ġof Ġcurrent Ġturn ĠR NN Ġat Ġeach Ġtime Ġstep . ĠThe Ġmodel Ġperformance Ġon Ġtraining Ġset Ġmight Ġnot Ġgeneral ize Ġwell Ġduring Ġinference Ġgiven Ġthe Ġlimited Ġsize Ġof Ġthe Ġtraining Ġset . </s> The Ġproposed ĠID CL M Ġand ĠES ID CL M Ġbeat Ġthe Ġsingle Ġturn ĠR NN LM Ġconsistently Ġunder Ġdifferent Ġcontext Ġturn Ġsizes . ĠES ID CL M Ġshows Ġthe Ġbest Ġlanguage Ġmodeling Ġperformance Ġunder Ġdialog Ġturn Ġsize Ġof Ġ3 Ġand Ġ5 , Ġoutper forming ĠID CL M Ġby Ġa Ġsmall Ġmargin . ĠID CL M Ġbeats Ġall Ġbaseline Ġmodels Ġwhen Ġusing Ġdialog Ġturn Ġsize Ġof Ġ5 , Ġand Ġproduces Ġslightly Ġworse Ġperplex ity Ġthan ĠDR NN LM Ġwhen Ġusing Ġdialog Ġturn Ġsize Ġof Ġ3 . </s> To Ġanalyze Ġthe Ġbest Ġpotential Ġgain Ġthat Ġmay Ġbe Ġachieved Ġby Ġintroducing Ġlinguistic Ġcontext , Ġwe Ġcompare Ġthe Ġproposed Ġcontextual Ġmodels Ġto ĠDAC LM , Ġthe Ġmodel Ġthat Ġuses Ġtrue Ġdialog Ġact Ġhistory Ġfor Ġdialog Ġcontext Ġmodeling . ĠAs Ġshown Ġin ĠTable Ġ1 Ġ, Ġthe Ġgap Ġbetween Ġour Ġproposed Ġmodels Ġand ĠDAC LM Ġis Ġnot Ġwide . ĠThis Ġgives Ġa Ġpositive Ġhint Ġthat Ġthe Ġproposed Ġcontextual Ġmodels Ġmay Ġimplicitly Ġcapture Ġthe Ġdialog Ġcontext Ġstate Ġchanges . </s> For Ġfine - gr ained Ġanalyses Ġof Ġthe Ġmodel Ġperformance , Ġwe Ġfurther Ġcompute Ġthe Ġtest Ġset Ġperplex ity Ġper ĠPOS Ġtag Ġand Ġper Ġdialog Ġact Ġtag . ĠWe Ġselected Ġthe Ġmost Ġfrequent ĠPOS Ġtags Ġand Ġdialog Ġact Ġtags Ġin ĠSw DA Ġcorpus , Ġand Ġreport Ġthe Ġtag Ġbased Ġperplex ity Ġrelative Ġchanges Ġ( Ġ$\ % $ Ġ) Ġof Ġthe Ġproposed Ġmodels Ġcomparing Ġto ĠSingle - Turn - R NN LM . ĠA Ġnegative Ġnumber Ġindicates Ġperformance Ġgain . </s> Table Ġ2 Ġshows Ġthe Ġmodel Ġperplex ity Ġper ĠPOS Ġtag . ĠAll Ġthe Ġthree Ġcontext Ġdependent Ġmodels Ġproduce Ġconsistent Ġperformance Ġgain Ġover Ġthe ĠSingle - Turn - R NN LM Ġfor Ġpronouns , Ġpre pos itions , Ġand Ġad verbs , Ġwith Ġpronouns Ġhaving Ġthe Ġlargest Ġperplex ity Ġimprovement . ĠHowever , Ġthe Ġproposed Ġcontextual Ġmodels Ġare Ġless Ġeffective Ġin Ġcapturing Ġnoun s . ĠThis Ġsuggests Ġthat Ġthe Ġproposed Ġcontextual ĠR NN Ġlanguage Ġmodels Ġexploit Ġthe Ġcontext Ġto Ġachieve Ġsuperior Ġprediction Ġon Ġcertain Ġbut Ġnot Ġall ĠPOS Ġtypes . ĠFurther Ġexploration Ġon Ġthe Ġmodel Ġdesign Ġis Ġrequired Ġif Ġwe Ġwant Ġto Ġbetter Ġcapture Ġwords Ġof Ġa Ġspecific Ġtype . </s> For Ġthe Ġdialog Ġact Ġtag Ġbased Ġresults Ġin ĠTable Ġ3 Ġ, Ġthe Ġthree Ġcontextual Ġmodels Ġshow Ġconsistent Ġperformance Ġgain Ġon ĠStatement - non - op inion Ġtype Ġutter ances . ĠThe Ġperplex ity Ġchanges Ġfor Ġother Ġdialog Ġact Ġtags Ġvary Ġfor Ġdifferent Ġmodels . </s> Con clusions </s> In Ġthis Ġwork , Ġwe Ġpropose Ġtwo Ġdialog Ġcontext Ġlanguage Ġmodels Ġthat Ġwith Ġspecial Ġdesign Ġto Ġmodel Ġdialog Ġinteractions . ĠOur Ġevaluation Ġresults Ġon ĠSwitch board ĠDial og ĠAct ĠCorpus Ġshow Ġthat Ġthe Ġproposed Ġmodel Ġoutper form Ġconventional ĠR NN Ġlanguage Ġmodel Ġby Ġ3 . 3 %. ĠThe Ġproposed Ġmodels Ġalso Ġillustrate Ġadvantageous Ġperformance Ġover Ġseveral Ġcompetitive Ġcontextual Ġlanguage Ġmodels . ĠPer plex ity Ġof Ġthe Ġproposed Ġdialog Ġcontext Ġlanguage Ġmodels Ġis Ġhigher Ġthan Ġthat Ġof Ġthe Ġmodel Ġusing Ġtrue Ġdialog Ġact Ġtags Ġas Ġcontext Ġby Ġa Ġsmall Ġmargin . ĠThis Ġindicates Ġthat Ġthe Ġproposed Ġmodel Ġmay Ġimplicitly Ġcapture Ġthe Ġdialog Ġcontext Ġstate Ġfor Ġlanguage Ġmodeling .