Henderson:2017, MobileNet model
EXTRACTIVE
PolyResponse: Conversational Search ::: Text Representation. The model, implemented as a deep neural network, learns to respond by training on hundreds of millions context-reply $(c,r)$ pairs. First, similar to Henderson:2017arxiv, raw text from both $c$ and $r$ is converted to unigrams and bigrams. All input text is first lower-cased and tokenised, numbers with 5 or more digits get their digits replaced by a wildcard symbol #, while words longer than 16 characters are replaced by a wildcard token LONGWORD. Sentence boundary tokens are added to each sentence. The vocabulary consists of the unigrams that occur at least 10 times in a random 10M subset of the Reddit training set (see Figure FIGREF2) plus the 200K most frequent bigrams in the same random subset. PolyResponse: Conversational Search ::: Photo Representation. Photos are represented using convolutional neural net (CNN) models pretrained on ImageNet BIBREF17. We use a MobileNet model with a depth multiplier of 1.4, and an input dimension of $224 \times 224$ pixels as in BIBREF18. This provides a $1,280 \times 1.4 = 1,792$-dimensional representation of a photo, which is then passed through a single hidden layer of dimensionality $1,024$ with ReLU activation, before being passed to a hidden layer of dimensionality 512 with no activation to provide the final representation $h_p$.
1909.01296
Introduction Ġand ĠBackground </s> Task - oriented Ġdialogue Ġsystems Ġare Ġprimarily Ġdesigned Ġto Ġsearch Ġand Ġinteract Ġwith Ġlarge Ġdatabases Ġwhich Ġcontain Ġinformation Ġpertaining Ġto Ġa Ġcertain Ġdialogue Ġdomain : Ġthe Ġmain Ġpurpose Ġof Ġsuch Ġsystems Ġis Ġto Ġassist Ġthe Ġusers Ġin Ġaccompl ishing Ġa Ġwell - defined Ġtask Ġsuch Ġas Ġflight Ġbooking ĠB IB REF 0 , Ġtourist Ġinformation ĠB IB REF 1 , Ġrestaurant Ġsearch ĠB IB REF 2 , Ġor Ġbooking Ġa Ġtaxi ĠB IB REF 3 . ĠThese Ġsystems Ġare Ġtypically Ġconstructed Ġaround Ġrigid Ġtask - specific Ġont ologies ĠB IB REF 1 , ĠB IB REF 4 Ġwhich Ġenumer ate Ġthe Ġconstraints Ġthe Ġusers Ġcan Ġexpress Ġusing Ġa Ġcollection Ġof Ġslots Ġ( e . g ., Ġprice Ġrange Ġfor Ġrestaurant Ġsearch ) Ġand Ġtheir Ġslot Ġvalues Ġ( e . g ., Ġcheap , Ġexpensive Ġfor Ġthe Ġaforementioned Ġslots ). ĠConvers ations Ġare Ġthen Ġmod elled Ġas Ġa Ġsequence Ġof Ġactions Ġthat Ġconst rain Ġslots Ġto Ġparticular Ġvalues . ĠThis Ġexplicit Ġsemantic Ġspace Ġis Ġmanually Ġengineered Ġby Ġthe Ġsystem Ġdesigner . ĠIt Ġserves Ġas Ġthe Ġoutput Ġof Ġthe Ġnatural Ġlanguage Ġunderstanding Ġcomponent Ġas Ġwell Ġas Ġthe Ġinput Ġto Ġthe Ġlanguage Ġgeneration Ġcomponent Ġboth Ġin Ġtraditional Ġmodular Ġsystems ĠB IB REF 5 , ĠB IB REF 6 Ġand Ġin Ġmore Ġrecent Ġend - to - end Ġtask - oriented Ġdialogue Ġsystems ĠB IB REF 7 , ĠB IB REF 8 , ĠB IB REF 9 , ĠB IB REF 3 . </s> Working Ġwith Ġsuch Ġexplicit Ġsemantics Ġfor Ġtask - oriented Ġdialogue Ġsystems Ġposes Ġseveral Ġcritical Ġchallenges Ġon Ġtop Ġof Ġthe Ġmanual Ġtime - consuming Ġdomain Ġont ology Ġdesign . ĠFirst , Ġit Ġis Ġdifficult Ġto Ġcollect Ġdomain - specific Ġdata Ġlabelled Ġwith Ġexplicit Ġsemantic Ġrepresentations . ĠAs Ġa Ġconsequence , Ġdespite Ġrecent Ġdata Ġcollection Ġefforts Ġto Ġenable Ġtraining Ġof Ġtask - oriented Ġsystems Ġacross Ġmultiple Ġdomains ĠB IB REF 0 , ĠB IB REF 3 , Ġannot ated Ġdatasets Ġare Ġstill Ġfew Ġand Ġfar Ġbetween , Ġas Ġwell Ġas Ġlimited Ġin Ġsize Ġand Ġthe Ġnumber Ġof Ġdomains Ġcovered . ĠSecond , Ġthe Ġcurrent Ġapproach Ġconst r ains Ġthe Ġtypes Ġof Ġdialogue Ġthe Ġsystem Ġcan Ġsupport , Ġresulting Ġin Ġartificial Ġconversations , Ġand Ġbreakdown s Ġwhen Ġthe Ġuser Ġdoes Ġnot Ġunderstand Ġwhat Ġthe Ġsystem Ġcan Ġand Ġcannot Ġsupport . ĠIn Ġother Ġwords , Ġtraining Ġa Ġtask - based Ġdialogue Ġsystem Ġfor Ġvoice - controlled Ġsearch Ġin Ġa Ġnew Ġdomain Ġalways Ġimplies Ġthe Ġcomplex , Ġexpensive , Ġand Ġtime - consuming Ġprocess Ġof Ġcollecting Ġand Ġannot ating Ġsufficient Ġamounts Ġof Ġin - domain Ġdialogue Ġdata . </s> In Ġthis Ġpaper , Ġwe Ġpresent Ġa Ġdemo Ġsystem Ġbased Ġon Ġan Ġalternative Ġapproach Ġto Ġtask - oriented Ġdialogue . ĠRe lying Ġon Ġnon - gener ative Ġresponse Ġretrieval Ġwe Ġdescribe Ġthe ĠPoly Response Ġconvers ational Ġsearch Ġengine Ġand Ġits Ġapplication Ġin Ġthe Ġtask Ġof Ġrestaurant Ġsearch Ġand Ġbooking . ĠThe Ġengine Ġis Ġtrained Ġon Ġhundreds Ġof Ġmillions Ġof Ġreal Ġconversations Ġfrom Ġa Ġgeneral Ġdomain Ġ( i . e ., ĠReddit ), Ġusing Ġan Ġimplicit Ġrepresentation Ġof Ġsemantics Ġthat Ġdirectly Ġoptim izes Ġthe Ġtask Ġat Ġhand . ĠIt Ġlearns Ġwhat Ġresponses Ġare Ġappropriate Ġin Ġdifferent Ġconvers ational Ġcontexts , Ġand Ġconsequently Ġranks Ġa Ġlarge Ġpool Ġof Ġresponses Ġaccording Ġto Ġtheir Ġrelevance Ġto Ġthe Ġcurrent Ġuser Ġutter ance Ġand Ġprevious Ġdialogue Ġhistory Ġ( i . e ., Ġdialogue Ġcontext ). </s> The Ġtechnical Ġaspects Ġof Ġthe Ġunderlying Ġconvers ational Ġsearch Ġengine Ġare Ġexplained Ġin Ġdetail Ġin Ġour Ġrecent Ġwork ĠB IB REF 11 , Ġwhile Ġthe Ġdetails Ġconcerning Ġthe ĠReddit Ġtraining Ġdata Ġare Ġalso Ġavailable Ġin Ġanother Ġrecent Ġpublication ĠB IB REF 12 . ĠIn Ġthis Ġdemo , Ġwe Ġput Ġfocus Ġon Ġthe Ġactual Ġpractical Ġusefulness Ġof Ġthe Ġsearch Ġengine Ġby Ġdemonstrating Ġits Ġpotential Ġin Ġthe Ġtask Ġof Ġrestaurant Ġsearch , Ġand Ġextending Ġit Ġto Ġdeal Ġwith Ġmulti - mod al Ġdata . ĠWe Ġdescribe Ġa ĠPoly Rep onse Ġsystem Ġthat Ġassists Ġthe Ġusers Ġin Ġfinding Ġa Ġrelevant Ġrestaurant Ġaccording Ġto Ġtheir Ġpreference , Ġand Ġthen Ġadditionally Ġhelps Ġthem Ġto Ġmake Ġa Ġbooking Ġin Ġthe Ġselected Ġrestaurant . ĠDue Ġto Ġits Ġretrieval - based Ġdesign , Ġwith Ġthe ĠPoly Response Ġengine Ġthere Ġis Ġno Ġneed Ġto Ġengineer Ġa Ġstructured Ġont ology , Ġor Ġto Ġsolve Ġthe Ġdifficult Ġtask Ġof Ġgeneral Ġlanguage Ġgeneration . ĠThis Ġdesign Ġalso Ġbypass es Ġthe Ġconstruction Ġof Ġdedicated Ġdecision - making Ġpolicy Ġmodules . ĠThe Ġlarge Ġranking Ġmodel Ġalready Ġencaps ulates Ġa Ġlot Ġof Ġknowledge Ġabout Ġnatural Ġlanguage Ġand Ġconvers ational Ġflow . </s> Since Ġretrieved Ġsystem Ġresponses Ġare Ġpresented Ġvisually Ġto Ġthe Ġuser , Ġthe ĠPoly Response Ġrestaurant Ġsearch Ġengine Ġis Ġable Ġto Ġcombine Ġtext Ġresponses Ġwith Ġrelevant Ġvisual Ġinformation Ġ( e . g ., Ġphotos Ġfrom Ġsocial Ġmedia Ġassociated Ġwith Ġthe Ġcurrent Ġrestaurant Ġand Ġrelated Ġto Ġthe Ġuser Ġutter ance ), Ġeffectively Ġyielding Ġa Ġmulti - mod al Ġresponse . ĠThis Ġsetup Ġof Ġusing Ġvoice Ġas Ġinput , Ġand Ġresponding Ġvisually Ġis Ġbecoming Ġmore Ġand Ġmore Ġprevalent Ġwith Ġthe Ġrise Ġof Ġsmart Ġscreens Ġlike ĠEcho ĠShow Ġand Ġeven Ġmixed Ġreality . ĠFinally , Ġthe ĠPoly Response Ġrestaurant Ġsearch Ġengine Ġis Ġmult ilingual : Ġit Ġis Ġcurrently Ġdeployed Ġin Ġ8 Ġlanguages Ġenabling Ġsearch Ġover Ġrestaurants Ġin Ġ8 Ġcities Ġaround Ġthe Ġworld . ĠSystem Ġsnapshots Ġin Ġfour Ġdifferent Ġlanguages Ġare Ġpresented Ġin ĠFigure ĠFIG REF 16 , Ġwhile Ġscreen cast Ġvideos Ġthat Ġillustrate Ġthe Ġdialogue Ġflow Ġwith Ġthe ĠPoly Response Ġengine Ġare Ġavailable Ġat : Ġhttps :// tiny url . com / y 3 ev k cf z . </s> Poly Response : ĠConvers ational ĠSearch </s> The ĠPoly Response Ġsystem Ġis Ġpowered Ġby Ġa Ġsingle Ġlarge Ġconvers ational Ġsearch Ġengine , Ġtrained Ġon Ġa Ġlarge Ġamount Ġof Ġconvers ational Ġand Ġimage Ġdata , Ġas Ġshown Ġin ĠFigure ĠFIG REF 2 . ĠIn Ġsimple Ġwords , Ġit Ġis Ġa Ġranking Ġmodel Ġthat Ġlearns Ġto Ġscore Ġconvers ational Ġreplies Ġand Ġimages Ġin Ġa Ġgiven Ġconvers ational Ġcontext . ĠThe Ġhighest - scoring Ġresponses Ġare Ġthen Ġretrieved Ġas Ġsystem Ġoutputs . ĠThe Ġsystem Ġcomp utes Ġtwo Ġsets Ġof Ġsimilarity Ġscores : Ġ1 ) Ġ$ S ( r , c ) $ Ġis Ġthe Ġscore Ġof Ġa Ġcandidate Ġreply Ġ$ r $ Ġgiven Ġa Ġconvers ational Ġcontext Ġ$ c $, Ġand Ġ2 ) Ġ$ S ( p , c ) $ Ġis Ġthe Ġscore Ġof Ġa Ġcandidate Ġphoto Ġ$ p $ Ġgiven Ġa Ġconvers ational Ġcontext Ġ$ c $. ĠThese Ġscores Ġare Ġcomputed Ġas Ġa Ġscaled Ġcos ine Ġsimilarity Ġof Ġa Ġvector Ġthat Ġrepresents Ġthe Ġcontext Ġ($ h _ c $ ), Ġand Ġa Ġvector Ġthat Ġrepresents Ġthe Ġcandidate Ġresponse : Ġa Ġtext Ġreply Ġ($ h _ r $ ) Ġor Ġa Ġphoto Ġ($ h _ p $ ). ĠFor Ġinstance , Ġ$ S ( r , c ) $ Ġis Ġcomputed Ġas Ġ$ S ( r , c )= C Ġcos ( h _ r , h _ c ) $, Ġwhere Ġ$ C $ Ġis Ġa Ġlearned Ġconstant . ĠThe Ġpart Ġof Ġthe Ġmodel Ġdealing Ġwith Ġtext Ġinput Ġ( i . e ., Ġobtaining Ġthe Ġenc od ings Ġ$ h _ c $ Ġand Ġ$ h _ r $ ) Ġfollows Ġthe Ġarchitecture Ġintroduced Ġrecently Ġby ĠHenderson : 2019 acl . ĠWe Ġprovide Ġonly Ġa Ġbrief Ġrecap Ġhere ; Ġsee Ġthe Ġoriginal Ġpaper Ġfor Ġfurther Ġdetails . </s> Poly Response : ĠConvers ational ĠSearch Ġ: :: ĠText ĠRepresent ation . </s> The Ġmodel , Ġimplemented Ġas Ġa Ġdeep Ġneural Ġnetwork , Ġlearns Ġto Ġrespond Ġby Ġtraining Ġon Ġhundreds Ġof Ġmillions Ġcontext - reply Ġ$( c , r ) $ Ġpairs . ĠFirst , Ġsimilar Ġto ĠHenderson : 2017 ar x iv , Ġraw Ġtext Ġfrom Ġboth Ġ$ c $ Ġand Ġ$ r $ Ġis Ġconverted Ġto Ġun ig rams Ġand Ġbig rams . ĠAll Ġinput Ġtext Ġis Ġfirst Ġlower - c ased Ġand Ġtoken ised , Ġnumbers Ġwith Ġ5 Ġor Ġmore Ġdigits Ġget Ġtheir Ġdigits Ġreplaced Ġby Ġa Ġwild card Ġsymbol Ġ# , Ġwhile Ġwords Ġlonger Ġthan Ġ16 Ġcharacters Ġare Ġreplaced Ġby Ġa Ġwild card Ġtoken ĠLONG W ORD . ĠSent ence Ġboundary Ġtokens Ġare Ġadded Ġto Ġeach Ġsentence . ĠThe Ġvocabulary Ġconsists Ġof Ġthe Ġun ig rams Ġthat Ġoccur Ġat Ġleast Ġ10 Ġtimes Ġin Ġa Ġrandom Ġ10 M Ġsubset Ġof Ġthe ĠReddit Ġtraining Ġset Ġ( see ĠFigure ĠFIG REF 2 ) Ġplus Ġthe Ġ200 K Ġmost Ġfrequent Ġbig rams Ġin Ġthe Ġsame Ġrandom Ġsubset . </s> During Ġtraining , Ġwe Ġobtain Ġ$ d $ - dimensional Ġfeature Ġrepresentations Ġ($ d = 320 $ ) Ġshared Ġbetween Ġcontexts Ġand Ġreplies Ġfor Ġeach Ġun ig ram Ġand Ġbig ram Ġjointly Ġwith Ġother Ġneural Ġnet Ġparameters . ĠA Ġstate - of - the - art Ġarchitecture Ġbased Ġon Ġtransform ers ĠB IB REF 13 Ġis Ġthen Ġapplied Ġto Ġun ig ram Ġand Ġbig ram Ġvectors Ġseparately , Ġwhich Ġare Ġthen Ġaveraged Ġto Ġform Ġthe Ġfinal Ġ320 - dimensional Ġencoding . ĠThat Ġencoding Ġis Ġthen Ġpassed Ġthrough Ġthree Ġfully - connected Ġnon - linear Ġhidden Ġlayers Ġof Ġdimension ality Ġ$ 1 , 024 $. ĠThe Ġfinal Ġlayer Ġis Ġlinear Ġand Ġmaps Ġthe Ġtext Ġinto Ġthe Ġfinal Ġ$ l $ - dimensional Ġ($ l = 512 $ ) Ġrepresentation : Ġ$ h _ c $ Ġand Ġ$ h _ r $. ĠOther Ġstandard Ġand Ġmore Ġsophisticated Ġenc oder Ġmodels Ġcan Ġalso Ġbe Ġused Ġto Ġprovide Ġfinal Ġenc od ings Ġ$ h _ c $ Ġand Ġ$ h _ r $, Ġbut Ġthe Ġcurrent Ġarchitecture Ġshows Ġa Ġgood Ġtrade - off Ġbetween Ġspeed Ġand Ġefficacy Ġwith Ġstrong Ġand Ġrobust Ġperformance Ġin Ġour Ġempirical Ġevaluations Ġon Ġthe Ġresponse Ġretrieval Ġtask Ġusing ĠReddit ĠB IB REF 14 , ĠOpen Sub t itles ĠB IB REF 15 , Ġand ĠAmazon Q A ĠB IB REF 16 Ġconvers ational Ġtest Ġdata , Ġsee ĠB IB REF 12 Ġfor Ġfurther Ġdetails . </s> In Ġtraining Ġthe Ġconstant Ġ$ C $ Ġis Ġconstrained Ġto Ġlie Ġbetween Ġ0 Ġand Ġ$\ sq rt { l } $. ĠFollowing ĠHenderson : 2017 ar x iv , Ġthe Ġscoring Ġfunction Ġin Ġthe Ġtraining Ġobjective Ġaims Ġto Ġmaxim ise Ġthe Ġsimilarity Ġscore Ġof Ġcontext - reply Ġpairs Ġthat Ġgo Ġtogether , Ġwhile Ġminim ising Ġthe Ġscore Ġof Ġrandom Ġpair ings : Ġnegative Ġexamples . ĠTraining Ġproceeds Ġvia ĠSG D Ġwith Ġbatches Ġcomprising Ġ500 Ġpairs Ġ( 1 Ġpositive Ġand Ġ499 Ġnegatives ). </s> Poly Response : ĠConvers ational ĠSearch Ġ: :: ĠPhoto ĠRepresent ation . </s> Photos Ġare Ġrepresented Ġusing Ġconv olution al Ġneural Ġnet Ġ( CNN ) Ġmodels Ġpret rained Ġon ĠImage Net ĠB IB REF 17 . ĠWe Ġuse Ġa ĠMobile Net Ġmodel Ġwith Ġa Ġdepth Ġmultiplier Ġof Ġ1 . 4 , Ġand Ġan Ġinput Ġdimension Ġof Ġ$ 224 Ġ\ times Ġ224 $ Ġpixels Ġas Ġin ĠB IB REF 18 . ĠThis Ġprovides Ġa Ġ$ 1 , 280 Ġ\ times Ġ1 . 4 Ġ= Ġ1 , 792 $ - dimensional Ġrepresentation Ġof Ġa Ġphoto , Ġwhich Ġis Ġthen Ġpassed Ġthrough Ġa Ġsingle Ġhidden Ġlayer Ġof Ġdimension ality Ġ$ 1 , 024 $ Ġwith ĠRe LU Ġactivation , Ġbefore Ġbeing Ġpassed Ġto Ġa Ġhidden Ġlayer Ġof Ġdimension ality Ġ512 Ġwith Ġno Ġactivation Ġto Ġprovide Ġthe Ġfinal Ġrepresentation Ġ$ h _ p $. </s> Poly Response : ĠConvers ational ĠSearch Ġ: :: ĠData ĠSource Ġ1 : ĠReddit . </s> For Ġtraining Ġtext Ġrepresentations Ġwe Ġuse Ġa ĠReddit Ġdataset Ġsimilar Ġto ĠAl R f ou : 2016 ar x iv . ĠOur Ġdataset Ġis Ġlarge Ġand Ġprovides Ġnatural Ġconvers ational Ġstructure : Ġall ĠReddit Ġdata Ġfrom ĠJanuary Ġ2015 Ġto ĠDecember Ġ2018 , Ġavailable Ġas Ġa Ġpublic ĠBig Query Ġdataset , Ġspan Ġalmost Ġ3 . 7 B Ġcomments ĠB IB REF 12 . ĠWe Ġpre process Ġthe Ġdataset Ġto Ġremove Ġunin form ative Ġand Ġlong Ġcomments Ġby Ġretaining Ġonly Ġsentences Ġcontaining Ġmore Ġthan Ġ8 Ġand Ġless Ġthan Ġ128 Ġword Ġtokens . ĠAfter Ġpairing Ġall Ġcomments / context s Ġ$ c $ Ġwith Ġtheir Ġreplies Ġ$ r $, Ġwe Ġobtain Ġmore Ġthan Ġ7 27 M Ġcontext - reply Ġ$( c , r ) $ Ġpairs Ġfor Ġtraining , Ġsee ĠFigure ĠFIG REF 2 . </s> Poly Response : ĠConvers ational ĠSearch Ġ: :: ĠData ĠSource Ġ2 : ĠYelp . </s> Once Ġthe Ġtext Ġencoding Ġsub - net works Ġare Ġtrained , Ġa Ġphoto Ġenc oder Ġis Ġlearned Ġon Ġtop Ġof Ġa Ġpret rained ĠMobile Net ĠCNN , Ġusing Ġdata Ġtaken Ġfrom Ġthe ĠYelp ĠOpen Ġdataset : Ġit Ġcontains Ġaround Ġ200 K Ġphotos Ġand Ġtheir Ġcapt ions . ĠTraining Ġof Ġthe Ġmulti - mod al Ġsub - network Ġthen Ġmaxim ises Ġthe Ġsimilarity Ġof Ġcapt ions Ġencoded Ġwith Ġthe Ġresponse Ġenc oder Ġ$ h _ r $ Ġto Ġthe Ġphoto Ġrepresentation Ġ$ h _ p $. ĠAs Ġa Ġresult , Ġwe Ġcan Ġcompute Ġthe Ġscore Ġof Ġa Ġphoto Ġgiven Ġa Ġcontext Ġusing Ġthe Ġcos ine Ġsimilarity Ġof Ġthe Ġrespective Ġvectors . ĠA Ġphoto Ġwill Ġbe Ġscored Ġhighly Ġif Ġit Ġlooks Ġlike Ġits Ġcaption Ġwould Ġbe Ġa Ġgood Ġresponse Ġto Ġthe Ġcurrent Ġcontext . </s> Poly Response : ĠConvers ational ĠSearch Ġ: :: ĠIndex Ġof ĠRespons es . </s> The ĠYelp Ġdataset Ġis Ġused Ġat Ġinference Ġtime Ġto Ġprovide Ġtext Ġand Ġphoto Ġcandidates Ġto Ġdisplay Ġto Ġthe Ġuser Ġat Ġeach Ġstep Ġin Ġthe Ġconversation . ĠOur Ġrestaurant Ġsearch Ġis Ġcurrently Ġdeployed Ġseparately Ġfor Ġeach Ġcity , Ġand Ġwe Ġlimit Ġthe Ġresponses Ġto Ġa Ġgiven Ġcity . ĠFor Ġinstance , Ġfor Ġour ĠEnglish Ġsystem Ġfor ĠEdinburgh Ġwe Ġwork Ġwith Ġ396 Ġrestaurants , Ġ4 , 225 Ġphotos Ġ( these Ġinclude Ġadditional Ġphotos Ġobtained Ġusing Ġthe ĠGoogle ĠPlaces ĠAPI Ġwithout Ġcapt ions ), Ġ6 , 725 Ġresponses Ġcreated Ġfrom Ġthe Ġstructured Ġinformation Ġabout Ġrestaurants Ġthat ĠYelp Ġprovides , Ġconverted Ġusing Ġsimple Ġtemplates Ġto Ġsentences Ġof Ġthe Ġform Ġsuch Ġas ĠâĢ ľ Rest aur ant ĠX Ġaccepts Ġcredit Ġcards . âĢ Ŀ , Ġ125 , 830 Ġsentences Ġextracted Ġfrom Ġonline Ġreviews . </s> Poly Response : ĠConvers ational ĠSearch Ġ: :: ĠPoly Response Ġin Ġa ĠN uts hell . </s> The Ġsystem Ġjointly Ġtrains Ġtwo Ġencoding Ġfunctions Ġ( with Ġshared Ġword Ġembed d ings ) Ġ$ f ( context ) $ Ġand Ġ$ g ( reply ) $ Ġwhich Ġproduce Ġenc od ings Ġ$ h _ c $ Ġand Ġ$ h _ r $, Ġso Ġthat Ġthe Ġsimilarity Ġ$ S ( c , r ) $ Ġis Ġhigh Ġfor Ġall Ġ$( c , r ) $ Ġpairs Ġfrom Ġthe ĠReddit Ġtraining Ġdata Ġand Ġlow Ġfor Ġrandom Ġpairs . ĠThe Ġencoding Ġfunction Ġ$ g () $ Ġis Ġthen Ġfrozen , Ġand Ġan Ġencoding Ġfunction Ġ$ t ( photo ) $ Ġis Ġlearnt Ġwhich Ġmakes Ġthe Ġsimilarity Ġbetween Ġa Ġphoto Ġand Ġits Ġassociated Ġcaption Ġhigh Ġfor Ġall Ġ( photo , Ġcaption ) Ġpairs Ġfrom Ġthe ĠYelp Ġdataset , Ġand Ġlow Ġfor Ġrandom Ġpairs . Ġ$ t $ Ġis Ġa ĠCNN Ġpret rained Ġon ĠImage Net , Ġwith Ġa Ġshallow Ġone - layer ĠD NN Ġon Ġtop . ĠGiven Ġa Ġnew Ġcontext / query , Ġwe Ġthen Ġprovide Ġits Ġencoding Ġ$ h _ c $ Ġby Ġapplying Ġ$ f () $, Ġand Ġfind Ġplausible Ġtext Ġreplies Ġand Ġphoto Ġresponses Ġaccording Ġto Ġfunctions Ġ$ g () $ Ġand Ġ$ t () $, Ġrespectively . ĠThese Ġshould Ġbe Ġresponses Ġthat Ġlook Ġlike Ġanswers Ġto Ġthe Ġquery , Ġand Ġphotos Ġthat Ġlook Ġlike Ġthey Ġwould Ġhave Ġcapt ions Ġthat Ġwould Ġbe Ġanswers Ġto Ġthe Ġprovided Ġquery . </s> At Ġinference , Ġfinding Ġrelevant Ġcandidates Ġgiven Ġa Ġcontext Ġreduces Ġto Ġcomputing Ġ$ h _ c $ Ġfor Ġthe Ġcontext Ġ$ c $ Ġ, Ġand Ġfinding Ġnearby Ġ$ h _ r $ Ġand Ġ$ h _ p $ Ġvectors . ĠThe Ġresponse Ġvectors Ġcan Ġall Ġbe Ġpre - com puted , Ġand Ġthe Ġnearest Ġneighbour Ġsearch Ġcan Ġbe Ġfurther Ġoptim ised Ġusing Ġstandard Ġlibraries Ġsuch Ġas ĠFa iss ĠB IB REF 19 Ġor Ġapproximate Ġnearest Ġneighbour Ġretrieval ĠB IB REF 20 , Ġgiving Ġan Ġefficient Ġsearch Ġthat Ġscales Ġto Ġbillions Ġof Ġcandidate Ġresponses . </s> The Ġsystem Ġoffers Ġboth Ġvoice Ġand Ġtext Ġinput Ġand Ġoutput . ĠSpeech - to - text Ġand Ġtext - to - speech Ġconversion Ġin Ġthe ĠPoly Response Ġsystem Ġis Ġcurrently Ġsupported Ġby Ġthe Ġoff - the - she lf ĠGoogle ĠCloud Ġtools . </s> Dialogue ĠFlow </s> The Ġranking Ġmodel Ġlends Ġitself Ġto Ġthe Ġone - shot Ġtask Ġof Ġfinding Ġthe Ġmost Ġrelevant Ġresponses Ġin Ġa Ġgiven Ġcontext . ĠHowever , Ġa Ġrestaurant - b rows ing Ġsystem Ġneeds Ġto Ġsupport Ġa Ġdialogue Ġflow Ġwhere Ġthe Ġuser Ġfinds Ġa Ġrestaurant , Ġand Ġthen Ġasks Ġquestions Ġabout Ġit . ĠThe Ġdialogue Ġstate Ġfor Ġeach Ġsearch Ġscenario Ġis Ġrepresented Ġas Ġthe Ġset Ġof Ġrestaurants Ġthat Ġare Ġconsidered Ġrelevant . ĠThis Ġstarts Ġoff Ġas Ġall Ġthe Ġrestaurants Ġin Ġthe Ġgiven Ġcity , Ġand Ġis Ġassumed Ġto Ġmon oton ically Ġdecrease Ġin Ġsize Ġas Ġthe Ġconversation Ġprogresses Ġuntil Ġthe Ġuser Ġconver ges Ġto Ġa Ġsingle Ġrestaurant . ĠA Ġrestaurant Ġis Ġonly Ġconsidered Ġvalid Ġin Ġthe Ġcontext Ġof Ġa Ġnew Ġuser Ġinput Ġif Ġit Ġhas Ġrelevant Ġresponses Ġcorresponding Ġto Ġit . ĠThis Ġflow Ġis Ġsummar ised Ġhere : </s> S 1 . ĠInitial ise Ġ$ R $ Ġas Ġthe Ġset Ġof Ġall Ġrestaurants Ġin Ġthe Ġcity . ĠGiven Ġthe Ġuser 's Ġinput , Ġrank Ġall Ġthe Ġresponses Ġin Ġthe Ġresponse Ġpool Ġpertaining Ġto Ġrestaurants Ġin Ġ$ R $. </s> S 2 . ĠRet rieve Ġthe Ġtop Ġ$ N $ Ġresponses Ġ$ r _ 1 , Ġr _ 2 , Ġ\ ld ots Ġ, Ġr _ N $ Ġwith Ġcorresponding Ġ( s orted ) Ġcos ine Ġsimilarity Ġscores : Ġ$ s _ 1 Ġ\ ge Ġs _ 2 Ġ\ ge Ġ\ ld ots Ġ\ ge Ġs _ N $. </s> S 3 . ĠComp ute Ġprobability Ġscores Ġ$ p _ i Ġ\ pro pt o Ġ\ exp Ġ( a Ġ\ cd ot Ġs _ i ) $ Ġwith Ġ$\ sum Ġ_ { i = 1 } ^ N Ġp _ i $, Ġwhere Ġ$ a > 0 $ Ġis Ġa Ġtun able Ġconstant . </s> S 4 . ĠComp ute Ġa Ġscore Ġ$ q _ e $ Ġfor Ġeach Ġrestaurant / entity Ġ$ e Ġ\ in ĠR $, Ġ$ q _ e Ġ= Ġ\ sum Ġ_ { i : Ġr _ i Ġ\ in Ġe } Ġp _ i $. </s> S 5 . ĠUpdate Ġ$ R $ Ġto Ġthe Ġsmallest Ġset Ġof Ġrestaurants Ġwith Ġhighest Ġ$ q $ Ġwhose Ġ$ q $ - values Ġsum Ġup Ġto Ġmore Ġthan Ġa Ġpred efined Ġthreshold Ġ$ t $. </s> S 6 . ĠDisplay Ġthe Ġmost Ġrelevant Ġresponses Ġassociated Ġwith Ġthe Ġupdated Ġ$ R $, Ġand Ġreturn Ġto ĠS 2 . </s> If Ġthere Ġare Ġmultiple Ġrelevant Ġrestaurants , Ġone Ġresponse Ġis Ġshown Ġfrom Ġeach . ĠWhen Ġonly Ġone Ġrestaurant Ġis Ġrelevant , Ġthe Ġtop Ġ$ N $ Ġresponses Ġare Ġall Ġshown , Ġand Ġrelevant Ġphotos Ġare Ġalso Ġdisplayed . ĠThe Ġsystem Ġdoes Ġnot Ġrequire Ġdedicated Ġunderstanding , Ġdecision - making , Ġand Ġgeneration Ġmodules , Ġand Ġthis Ġdialogue Ġflow Ġdoes Ġnot Ġrely Ġon Ġexplicit Ġtask - tail ored Ġsemantics . ĠThe Ġset Ġof Ġrelevant Ġrestaurants Ġis Ġkept Ġinternally Ġwhile Ġthe Ġsystem Ġnarrow s Ġit Ġdown Ġacross Ġmultiple Ġdialogue Ġturns . ĠA Ġsimple Ġset Ġof Ġpred efined Ġrules Ġis Ġused Ġto Ġprovide Ġa Ġtem pl atic Ġspoken Ġsystem Ġresponse : Ġe . g ., Ġan Ġexample Ġrule Ġis ĠâĢ ľ One Ġreview Ġof Ġ$ e $ Ġsaid Ġ$ r $ âĢ Ŀ , Ġwhere Ġ$ e $ Ġrefers Ġto Ġthe Ġrestaurant , Ġand Ġ$ r $ Ġto Ġa Ġrelevant Ġresponse Ġassociated Ġwith Ġ$ e $. ĠNote Ġthat Ġwhile Ġthe Ġdemo Ġis Ġcurrently Ġfocused Ġon Ġthe Ġrestaurant Ġsearch Ġtask , Ġthe Ġdescribed ĠâĢ ľ n arrow ing Ġdown âĢ Ŀ Ġdialogue Ġflow Ġis Ġgeneric Ġand Ġapplicable Ġto Ġa Ġvariety Ġof Ġapplications Ġdealing Ġwith Ġsimilar Ġentity Ġsearch . </s> The Ġsystem Ġcan Ġuse Ġa Ġset Ġof Ġintent Ġclass ifiers Ġto Ġallow Ġreset ting Ġthe Ġdialogue Ġstate , Ġor Ġto Ġactivate Ġthe Ġseparate Ġrestaurant Ġbooking Ġdialogue Ġflow . ĠThese Ġclass ifiers Ġare Ġbriefly Ġdiscussed Ġin ĠÂ§ SEC REF 4 . </s> Other ĠFunction ality Ġ: :: ĠMult iling uality . </s> The ĠPoly Response Ġrestaurant Ġsearch Ġis Ġcurrently Ġavailable Ġin Ġ8 Ġlanguages Ġand Ġfor Ġ8 Ġcities Ġaround Ġthe Ġworld : ĠEnglish Ġ( Ed inburgh ), ĠGerman Ġ( Ber lin ), ĠSpanish Ġ( Mad rid ), ĠMandarin Ġ( Tai pei ), ĠPolish Ġ( Wars aw ), ĠRussian Ġ( Moscow ), ĠKorean Ġ( Se oul ), Ġand ĠSerbian Ġ( Bel grade ). ĠSelected Ġsnapshots Ġare Ġshown Ġin ĠFigure ĠFIG REF 16 , Ġwhile Ġwe Ġalso Ġprovide Ġvideos Ġdemonstrating Ġthe Ġuse Ġand Ġbehaviour Ġof Ġthe Ġsystems Ġat : Ġhttps :// tiny url . com / y 3 ev k cf z . ĠA Ġsimple ĠMT - based Ġtranslate - to - source Ġapproach Ġat Ġinference Ġtime Ġis Ġcurrently Ġused Ġto Ġenable Ġthe Ġdeployment Ġof Ġthe Ġsystem Ġin Ġother Ġlanguages : Ġ1 ) Ġthe Ġpool Ġof Ġresponses Ġin Ġeach Ġlanguage Ġis Ġtranslated Ġto ĠEnglish Ġby ĠGoogle ĠTrans late Ġbeforehand , Ġand Ġpre - com puted Ġenc od ings Ġof Ġtheir ĠEnglish Ġtranslations Ġare Ġused Ġas Ġrepresentations Ġof Ġeach Ġforeign Ġlanguage Ġresponse ; Ġ2 ) Ġa Ġprovided Ġuser Ġutter ance Ġ( i . e ., Ġcontext ) Ġis Ġtranslated Ġto ĠEnglish Ġon - the - fly Ġand Ġits Ġencoding Ġ$ h _ c $ Ġis Ġthen Ġlearned . ĠWe Ġplan Ġto Ġexperiment Ġwith Ġmore Ġsophisticated Ġmult ilingual Ġmodels Ġin Ġfuture Ġwork . </s> Other ĠFunction ality Ġ: :: ĠVoice - Cont rolled ĠMenu ĠSearch . </s> An Ġadditional Ġfunctionality Ġenables Ġthe Ġuser Ġto Ġget Ġparts Ġof Ġthe Ġrestaurant Ġmenu Ġrelevant Ġto Ġthe Ġcurrent Ġuser Ġutter ance Ġas Ġresponses . ĠThis Ġis Ġachieved Ġby Ġperforming Ġan Ġadditional Ġranking Ġstep Ġof Ġavailable Ġmenu Ġitems Ġand Ġretrieving Ġthe Ġones Ġthat Ġare Ġsem antically Ġrelevant Ġto Ġthe Ġuser Ġutter ance Ġusing Ġexactly Ġthe Ġsame Ġmethodology Ġas Ġwith Ġranking Ġother Ġresponses . ĠAn Ġexample Ġof Ġthis Ġfunctionality Ġis Ġshown Ġin ĠFigure ĠFIG REF 21 . </s> Other ĠFunction ality Ġ: :: ĠReset ting Ġand ĠSwitch ing Ġto ĠBook ing . </s> The Ġrestaurant Ġsearch Ġsystem Ġneeds Ġto Ġsupport Ġthe Ġdiscrete Ġactions Ġof Ġrestart ing Ġthe Ġconversation Ġ( i . e ., Ġreset ting Ġthe Ġset Ġ$ R $ ), Ġand Ġshould Ġenable Ġtransferring Ġto Ġthe Ġslot - based Ġtable Ġbooking Ġflow . ĠThis Ġis Ġachieved Ġusing Ġtwo Ġbinary Ġintent Ġclass ifiers , Ġthat Ġare Ġrun Ġat Ġeach Ġstep Ġin Ġthe Ġdialogue . ĠThese Ġclass ifiers Ġmake Ġuse Ġof Ġthe Ġalready - com puted Ġ$ h _ c $ Ġvector Ġthat Ġrepresents Ġthe Ġuser 's Ġlatest Ġtext . ĠA Ġsingle - layer Ġneural Ġnet Ġis Ġlearned Ġon Ġtop Ġof Ġthe Ġ512 - dimensional Ġencoding , Ġwith Ġa ĠRe LU Ġactivation Ġand Ġ100 Ġhidden Ġnodes . ĠTo Ġtrain Ġthe Ġclass ifiers , Ġsets Ġof Ġ20 Ġrelevant Ġparaph r ases Ġ( e . g ., ĠâĢ ľ Start Ġagain âĢ Ŀ ) Ġare Ġprovided Ġas Ġpositive Ġexamples . ĠFinally , Ġwhen Ġthe Ġsystem Ġsuccessfully Ġswitches Ġto Ġthe Ġbooking Ġscenario , Ġit Ġproceeds Ġto Ġthe Ġslot Ġfilling Ġtask : Ġit Ġaims Ġto Ġextract Ġall Ġthe Ġrelevant Ġbooking Ġinformation Ġfrom Ġthe Ġuser Ġ( e . g ., Ġdate , Ġtime , Ġnumber Ġof Ġpeople Ġto Ġd ine ). ĠThe Ġentire Ġflow Ġof Ġthe Ġsystem Ġillustrating Ġboth Ġthe Ġsearch Ġphase Ġand Ġthe Ġbooking Ġphase Ġis Ġprovided Ġas Ġthe Ġsupplemental Ġvideo Ġmaterial . </s> Conclusion Ġand ĠFuture ĠWork </s> This Ġpaper Ġhas Ġpresented Ġa Ġgeneral Ġapproach Ġto Ġsearch - based Ġdialogue Ġthat Ġdoes Ġnot Ġrely Ġon Ġexplicit Ġsemantic Ġrepresentations Ġsuch Ġas Ġdialogue Ġacts Ġor Ġslot - value Ġont ologies , Ġand Ġallows Ġfor Ġmulti - mod al Ġresponses . ĠIn Ġfuture Ġwork , Ġwe Ġwill Ġextend Ġthe Ġcurrent Ġdemo Ġsystem Ġto Ġmore Ġtasks Ġand Ġlanguages , Ġand Ġwork Ġwith Ġmore Ġsophisticated Ġenc od ers Ġand Ġranking Ġfunctions . ĠBesides Ġthe Ġinitial Ġdialogue Ġflow Ġfrom Ġthis Ġwork Ġ( Â§ SEC REF 3 ), Ġwe Ġwill Ġalso Ġwork Ġwith Ġmore Ġcomplex Ġflows Ġdealing , Ġe . g ., Ġwith Ġuser Ġintent Ġshifts .
Introduction and Background Task-oriented dialogue systems are primarily designed to search and interact with large databases which contain information pertaining to a certain dialogue domain: the main purpose of such systems is to assist the users in accomplishing a well-defined task such as flight booking BIBREF0, tourist information BIBREF1, restaurant search BIBREF2, or booking a taxi BIBREF3. These systems are typically constructed around rigid task-specific ontologies BIBREF1, BIBREF4 which enumerate the constraints the users can express using a collection of slots (e.g., price range for restaurant search) and their slot values (e.g., cheap, expensive for the aforementioned slots). Conversations are then modelled as a sequence of actions that constrain slots to particular values. This explicit semantic space is manually engineered by the system designer. It serves as the output of the natural language understanding component as well as the input to the language generation component both in traditional modular systems BIBREF5, BIBREF6 and in more recent end-to-end task-oriented dialogue systems BIBREF7, BIBREF8, BIBREF9, BIBREF3. Working with such explicit semantics for task-oriented dialogue systems poses several critical challenges on top of the manual time-consuming domain ontology design. First, it is difficult to collect domain-specific data labelled with explicit semantic representations. As a consequence, despite recent data collection efforts to enable training of task-oriented systems across multiple domains BIBREF0, BIBREF3, annotated datasets are still few and far between, as well as limited in size and the number of domains covered. Second, the current approach constrains the types of dialogue the system can support, resulting in artificial conversations, and breakdowns when the user does not understand what the system can and cannot support. In other words, training a task-based dialogue system for voice-controlled search in a new domain always implies the complex, expensive, and time-consuming process of collecting and annotating sufficient amounts of in-domain dialogue data. In this paper, we present a demo system based on an alternative approach to task-oriented dialogue. Relying on non-generative response retrieval we describe the PolyResponse conversational search engine and its application in the task of restaurant search and booking. The engine is trained on hundreds of millions of real conversations from a general domain (i.e., Reddit), using an implicit representation of semantics that directly optimizes the task at hand. It learns what responses are appropriate in different conversational contexts, and consequently ranks a large pool of responses according to their relevance to the current user utterance and previous dialogue history (i.e., dialogue context). The technical aspects of the underlying conversational search engine are explained in detail in our recent work BIBREF11, while the details concerning the Reddit training data are also available in another recent publication BIBREF12. In this demo, we put focus on the actual practical usefulness of the search engine by demonstrating its potential in the task of restaurant search, and extending it to deal with multi-modal data. We describe a PolyReponse system that assists the users in finding a relevant restaurant according to their preference, and then additionally helps them to make a booking in the selected restaurant. Due to its retrieval-based design, with the PolyResponse engine there is no need to engineer a structured ontology, or to solve the difficult task of general language generation. This design also bypasses the construction of dedicated decision-making policy modules. The large ranking model already encapsulates a lot of knowledge about natural language and conversational flow. Since retrieved system responses are presented visually to the user, the PolyResponse restaurant search engine is able to combine text responses with relevant visual information (e.g., photos from social media associated with the current restaurant and related to the user utterance), effectively yielding a multi-modal response. This setup of using voice as input, and responding visually is becoming more and more prevalent with the rise of smart screens like Echo Show and even mixed reality. Finally, the PolyResponse restaurant search engine is multilingual: it is currently deployed in 8 languages enabling search over restaurants in 8 cities around the world. System snapshots in four different languages are presented in Figure FIGREF16, while screencast videos that illustrate the dialogue flow with the PolyResponse engine are available at: https://tinyurl.com/y3evkcfz. PolyResponse: Conversational Search The PolyResponse system is powered by a single large conversational search engine, trained on a large amount of conversational and image data, as shown in Figure FIGREF2. In simple words, it is a ranking model that learns to score conversational replies and images in a given conversational context. The highest-scoring responses are then retrieved as system outputs. The system computes two sets of similarity scores: 1) $S(r,c)$ is the score of a candidate reply $r$ given a conversational context $c$, and 2) $S(p,c)$ is the score of a candidate photo $p$ given a conversational context $c$. These scores are computed as a scaled cosine similarity of a vector that represents the context ($h_c$), and a vector that represents the candidate response: a text reply ($h_r$) or a photo ($h_p$). For instance, $S(r,c)$ is computed as $S(r,c)=C cos(h_r,h_c)$, where $C$ is a learned constant. The part of the model dealing with text input (i.e., obtaining the encodings $h_c$ and $h_r$) follows the architecture introduced recently by Henderson:2019acl. We provide only a brief recap here; see the original paper for further details. PolyResponse: Conversational Search ::: Text Representation. The model, implemented as a deep neural network, learns to respond by training on hundreds of millions context-reply $(c,r)$ pairs. First, similar to Henderson:2017arxiv, raw text from both $c$ and $r$ is converted to unigrams and bigrams. All input text is first lower-cased and tokenised, numbers with 5 or more digits get their digits replaced by a wildcard symbol #, while words longer than 16 characters are replaced by a wildcard token LONGWORD. Sentence boundary tokens are added to each sentence. The vocabulary consists of the unigrams that occur at least 10 times in a random 10M subset of the Reddit training set (see Figure FIGREF2) plus the 200K most frequent bigrams in the same random subset. During training, we obtain $d$-dimensional feature representations ($d=320$) shared between contexts and replies for each unigram and bigram jointly with other neural net parameters. A state-of-the-art architecture based on transformers BIBREF13 is then applied to unigram and bigram vectors separately, which are then averaged to form the final 320-dimensional encoding. That encoding is then passed through three fully-connected non-linear hidden layers of dimensionality $1,024$. The final layer is linear and maps the text into the final $l$-dimensional ($l=512$) representation: $h_c$ and $h_r$. Other standard and more sophisticated encoder models can also be used to provide final encodings $h_c$ and $h_r$, but the current architecture shows a good trade-off between speed and efficacy with strong and robust performance in our empirical evaluations on the response retrieval task using Reddit BIBREF14, OpenSubtitles BIBREF15, and AmazonQA BIBREF16 conversational test data, see BIBREF12 for further details. In training the constant $C$ is constrained to lie between 0 and $\sqrt{l}$. Following Henderson:2017arxiv, the scoring function in the training objective aims to maximise the similarity score of context-reply pairs that go together, while minimising the score of random pairings: negative examples. Training proceeds via SGD with batches comprising 500 pairs (1 positive and 499 negatives). PolyResponse: Conversational Search ::: Photo Representation. Photos are represented using convolutional neural net (CNN) models pretrained on ImageNet BIBREF17. We use a MobileNet model with a depth multiplier of 1.4, and an input dimension of $224 \times 224$ pixels as in BIBREF18. This provides a $1,280 \times 1.4 = 1,792$-dimensional representation of a photo, which is then passed through a single hidden layer of dimensionality $1,024$ with ReLU activation, before being passed to a hidden layer of dimensionality 512 with no activation to provide the final representation $h_p$. PolyResponse: Conversational Search ::: Data Source 1: Reddit. For training text representations we use a Reddit dataset similar to AlRfou:2016arxiv. Our dataset is large and provides natural conversational structure: all Reddit data from January 2015 to December 2018, available as a public BigQuery dataset, span almost 3.7B comments BIBREF12. We preprocess the dataset to remove uninformative and long comments by retaining only sentences containing more than 8 and less than 128 word tokens. After pairing all comments/contexts $c$ with their replies $r$, we obtain more than 727M context-reply $(c,r)$ pairs for training, see Figure FIGREF2. PolyResponse: Conversational Search ::: Data Source 2: Yelp. Once the text encoding sub-networks are trained, a photo encoder is learned on top of a pretrained MobileNet CNN, using data taken from the Yelp Open dataset: it contains around 200K photos and their captions. Training of the multi-modal sub-network then maximises the similarity of captions encoded with the response encoder $h_r$ to the photo representation $h_p$. As a result, we can compute the score of a photo given a context using the cosine similarity of the respective vectors. A photo will be scored highly if it looks like its caption would be a good response to the current context. PolyResponse: Conversational Search ::: Index of Responses. The Yelp dataset is used at inference time to provide text and photo candidates to display to the user at each step in the conversation. Our restaurant search is currently deployed separately for each city, and we limit the responses to a given city. For instance, for our English system for Edinburgh we work with 396 restaurants, 4,225 photos (these include additional photos obtained using the Google Places API without captions), 6,725 responses created from the structured information about restaurants that Yelp provides, converted using simple templates to sentences of the form such as “Restaurant X accepts credit cards.”, 125,830 sentences extracted from online reviews. PolyResponse: Conversational Search ::: PolyResponse in a Nutshell. The system jointly trains two encoding functions (with shared word embeddings) $f(context)$ and $g(reply)$ which produce encodings $h_c$ and $h_r$, so that the similarity $S(c,r)$ is high for all $(c,r)$ pairs from the Reddit training data and low for random pairs. The encoding function $g()$ is then frozen, and an encoding function $t(photo)$ is learnt which makes the similarity between a photo and its associated caption high for all (photo, caption) pairs from the Yelp dataset, and low for random pairs. $t$ is a CNN pretrained on ImageNet, with a shallow one-layer DNN on top. Given a new context/query, we then provide its encoding $h_c$ by applying $f()$, and find plausible text replies and photo responses according to functions $g()$ and $t()$, respectively. These should be responses that look like answers to the query, and photos that look like they would have captions that would be answers to the provided query. At inference, finding relevant candidates given a context reduces to computing $h_c$ for the context $c$ , and finding nearby $h_r$ and $h_p$ vectors. The response vectors can all be pre-computed, and the nearest neighbour search can be further optimised using standard libraries such as Faiss BIBREF19 or approximate nearest neighbour retrieval BIBREF20, giving an efficient search that scales to billions of candidate responses. The system offers both voice and text input and output. Speech-to-text and text-to-speech conversion in the PolyResponse system is currently supported by the off-the-shelf Google Cloud tools. Dialogue Flow The ranking model lends itself to the one-shot task of finding the most relevant responses in a given context. However, a restaurant-browsing system needs to support a dialogue flow where the user finds a restaurant, and then asks questions about it. The dialogue state for each search scenario is represented as the set of restaurants that are considered relevant. This starts off as all the restaurants in the given city, and is assumed to monotonically decrease in size as the conversation progresses until the user converges to a single restaurant. A restaurant is only considered valid in the context of a new user input if it has relevant responses corresponding to it. This flow is summarised here: S1. Initialise $R$ as the set of all restaurants in the city. Given the user's input, rank all the responses in the response pool pertaining to restaurants in $R$. S2. Retrieve the top $N$ responses $r_1, r_2, \ldots , r_N$ with corresponding (sorted) cosine similarity scores: $s_1 \ge s_2 \ge \ldots \ge s_N$. S3. Compute probability scores $p_i \propto \exp (a \cdot s_i)$ with $\sum _{i=1}^N p_i$, where $a>0$ is a tunable constant. S4. Compute a score $q_e$ for each restaurant/entity $e \in R$, $q_e = \sum _{i: r_i \in e} p_i$. S5. Update $R$ to the smallest set of restaurants with highest $q$ whose $q$-values sum up to more than a predefined threshold $t$. S6. Display the most relevant responses associated with the updated $R$, and return to S2. If there are multiple relevant restaurants, one response is shown from each. When only one restaurant is relevant, the top $N$ responses are all shown, and relevant photos are also displayed. The system does not require dedicated understanding, decision-making, and generation modules, and this dialogue flow does not rely on explicit task-tailored semantics. The set of relevant restaurants is kept internally while the system narrows it down across multiple dialogue turns. A simple set of predefined rules is used to provide a templatic spoken system response: e.g., an example rule is “One review of $e$ said $r$”, where $e$ refers to the restaurant, and $r$ to a relevant response associated with $e$. Note that while the demo is currently focused on the restaurant search task, the described “narrowing down” dialogue flow is generic and applicable to a variety of applications dealing with similar entity search. The system can use a set of intent classifiers to allow resetting the dialogue state, or to activate the separate restaurant booking dialogue flow. These classifiers are briefly discussed in §SECREF4. Other Functionality ::: Multilinguality. The PolyResponse restaurant search is currently available in 8 languages and for 8 cities around the world: English (Edinburgh), German (Berlin), Spanish (Madrid), Mandarin (Taipei), Polish (Warsaw), Russian (Moscow), Korean (Seoul), and Serbian (Belgrade). Selected snapshots are shown in Figure FIGREF16, while we also provide videos demonstrating the use and behaviour of the systems at: https://tinyurl.com/y3evkcfz. A simple MT-based translate-to-source approach at inference time is currently used to enable the deployment of the system in other languages: 1) the pool of responses in each language is translated to English by Google Translate beforehand, and pre-computed encodings of their English translations are used as representations of each foreign language response; 2) a provided user utterance (i.e., context) is translated to English on-the-fly and its encoding $h_c$ is then learned. We plan to experiment with more sophisticated multilingual models in future work. Other Functionality ::: Voice-Controlled Menu Search. An additional functionality enables the user to get parts of the restaurant menu relevant to the current user utterance as responses. This is achieved by performing an additional ranking step of available menu items and retrieving the ones that are semantically relevant to the user utterance using exactly the same methodology as with ranking other responses. An example of this functionality is shown in Figure FIGREF21. Other Functionality ::: Resetting and Switching to Booking. The restaurant search system needs to support the discrete actions of restarting the conversation (i.e., resetting the set $R$), and should enable transferring to the slot-based table booking flow. This is achieved using two binary intent classifiers, that are run at each step in the dialogue. These classifiers make use of the already-computed $h_c$ vector that represents the user's latest text. A single-layer neural net is learned on top of the 512-dimensional encoding, with a ReLU activation and 100 hidden nodes. To train the classifiers, sets of 20 relevant paraphrases (e.g., “Start again”) are provided as positive examples. Finally, when the system successfully switches to the booking scenario, it proceeds to the slot filling task: it aims to extract all the relevant booking information from the user (e.g., date, time, number of people to dine). The entire flow of the system illustrating both the search phase and the booking phase is provided as the supplemental video material. Conclusion and Future Work This paper has presented a general approach to search-based dialogue that does not rely on explicit semantic representations such as dialogue acts or slot-value ontologies, and allows for multi-modal responses. In future work, we will extend the current demo system to more tasks and languages, and work with more sophisticated encoders and ranking functions. Besides the initial dialogue flow from this work (§SECREF3), we will also work with more complex flows dealing, e.g., with user intent shifts.
How does PolyResponse architecture look like?
e4a315e9c190cf96493eefe04ce4ba6ae6894550
How Ġdoes ĠPoly Response Ġarchitecture Ġlook Ġlike ?
H end erson : 2017 , ĠMobile Net Ġmodel
<s> How Ġdoes ĠPoly Response Ġarchitecture Ġlook Ġlike ? </s> Introduction Ġand ĠBackground </s> Task - oriented Ġdialogue Ġsystems Ġare Ġprimarily Ġdesigned Ġto Ġsearch Ġand Ġinteract Ġwith Ġlarge Ġdatabases Ġwhich Ġcontain Ġinformation Ġpertaining Ġto Ġa Ġcertain Ġdialogue Ġdomain : Ġthe Ġmain Ġpurpose Ġof Ġsuch Ġsystems Ġis Ġto Ġassist Ġthe Ġusers Ġin Ġaccompl ishing Ġa Ġwell - defined Ġtask Ġsuch Ġas Ġflight Ġbooking ĠB IB REF 0 , Ġtourist Ġinformation ĠB IB REF 1 , Ġrestaurant Ġsearch ĠB IB REF 2 , Ġor Ġbooking Ġa Ġtaxi ĠB IB REF 3 . ĠThese Ġsystems Ġare Ġtypically Ġconstructed Ġaround Ġrigid Ġtask - specific Ġont ologies ĠB IB REF 1 , ĠB IB REF 4 Ġwhich Ġenumer ate Ġthe Ġconstraints Ġthe Ġusers Ġcan Ġexpress Ġusing Ġa Ġcollection Ġof Ġslots Ġ( e . g ., Ġprice Ġrange Ġfor Ġrestaurant Ġsearch ) Ġand Ġtheir Ġslot Ġvalues Ġ( e . g ., Ġcheap , Ġexpensive Ġfor Ġthe Ġaforementioned Ġslots ). ĠConvers ations Ġare Ġthen Ġmod elled Ġas Ġa Ġsequence Ġof Ġactions Ġthat Ġconst rain Ġslots Ġto Ġparticular Ġvalues . ĠThis Ġexplicit Ġsemantic Ġspace Ġis Ġmanually Ġengineered Ġby Ġthe Ġsystem Ġdesigner . ĠIt Ġserves Ġas Ġthe Ġoutput Ġof Ġthe Ġnatural Ġlanguage Ġunderstanding Ġcomponent Ġas Ġwell Ġas Ġthe Ġinput Ġto Ġthe Ġlanguage Ġgeneration Ġcomponent Ġboth Ġin Ġtraditional Ġmodular Ġsystems ĠB IB REF 5 , ĠB IB REF 6 Ġand Ġin Ġmore Ġrecent Ġend - to - end Ġtask - oriented Ġdialogue Ġsystems ĠB IB REF 7 , ĠB IB REF 8 , ĠB IB REF 9 , ĠB IB REF 3 . </s> Working Ġwith Ġsuch Ġexplicit Ġsemantics Ġfor Ġtask - oriented Ġdialogue Ġsystems Ġposes Ġseveral Ġcritical Ġchallenges Ġon Ġtop Ġof Ġthe Ġmanual Ġtime - consuming Ġdomain Ġont ology Ġdesign . ĠFirst , Ġit Ġis Ġdifficult Ġto Ġcollect Ġdomain - specific Ġdata Ġlabelled Ġwith Ġexplicit Ġsemantic Ġrepresentations . ĠAs Ġa Ġconsequence , Ġdespite Ġrecent Ġdata Ġcollection Ġefforts Ġto Ġenable Ġtraining Ġof Ġtask - oriented Ġsystems Ġacross Ġmultiple Ġdomains ĠB IB REF 0 , ĠB IB REF 3 , Ġannot ated Ġdatasets Ġare Ġstill Ġfew Ġand Ġfar Ġbetween , Ġas Ġwell Ġas Ġlimited Ġin Ġsize Ġand Ġthe Ġnumber Ġof Ġdomains Ġcovered . ĠSecond , Ġthe Ġcurrent Ġapproach Ġconst r ains Ġthe Ġtypes Ġof Ġdialogue Ġthe Ġsystem Ġcan Ġsupport , Ġresulting Ġin Ġartificial Ġconversations , Ġand Ġbreakdown s Ġwhen Ġthe Ġuser Ġdoes Ġnot Ġunderstand Ġwhat Ġthe Ġsystem Ġcan Ġand Ġcannot Ġsupport . ĠIn Ġother Ġwords , Ġtraining Ġa Ġtask - based Ġdialogue Ġsystem Ġfor Ġvoice - controlled Ġsearch Ġin Ġa Ġnew Ġdomain Ġalways Ġimplies Ġthe Ġcomplex , Ġexpensive , Ġand Ġtime - consuming Ġprocess Ġof Ġcollecting Ġand Ġannot ating Ġsufficient Ġamounts Ġof Ġin - domain Ġdialogue Ġdata . </s> In Ġthis Ġpaper , Ġwe Ġpresent Ġa Ġdemo Ġsystem Ġbased Ġon Ġan Ġalternative Ġapproach Ġto Ġtask - oriented Ġdialogue . ĠRe lying Ġon Ġnon - gener ative Ġresponse Ġretrieval Ġwe Ġdescribe Ġthe ĠPoly Response Ġconvers ational Ġsearch Ġengine Ġand Ġits Ġapplication Ġin Ġthe Ġtask Ġof Ġrestaurant Ġsearch Ġand Ġbooking . ĠThe Ġengine Ġis Ġtrained Ġon Ġhundreds Ġof Ġmillions Ġof Ġreal Ġconversations Ġfrom Ġa Ġgeneral Ġdomain Ġ( i . e ., ĠReddit ), Ġusing Ġan Ġimplicit Ġrepresentation Ġof Ġsemantics Ġthat Ġdirectly Ġoptim izes Ġthe Ġtask Ġat Ġhand . ĠIt Ġlearns Ġwhat Ġresponses Ġare Ġappropriate Ġin Ġdifferent Ġconvers ational Ġcontexts , Ġand Ġconsequently Ġranks Ġa Ġlarge Ġpool Ġof Ġresponses Ġaccording Ġto Ġtheir Ġrelevance Ġto Ġthe Ġcurrent Ġuser Ġutter ance Ġand Ġprevious Ġdialogue Ġhistory Ġ( i . e ., Ġdialogue Ġcontext ). </s> The Ġtechnical Ġaspects Ġof Ġthe Ġunderlying Ġconvers ational Ġsearch Ġengine Ġare Ġexplained Ġin Ġdetail Ġin Ġour Ġrecent Ġwork ĠB IB REF 11 , Ġwhile Ġthe Ġdetails Ġconcerning Ġthe ĠReddit Ġtraining Ġdata Ġare Ġalso Ġavailable Ġin Ġanother Ġrecent Ġpublication ĠB IB REF 12 . ĠIn Ġthis Ġdemo , Ġwe Ġput Ġfocus Ġon Ġthe Ġactual Ġpractical Ġusefulness Ġof Ġthe Ġsearch Ġengine Ġby Ġdemonstrating Ġits Ġpotential Ġin Ġthe Ġtask Ġof Ġrestaurant Ġsearch , Ġand Ġextending Ġit Ġto Ġdeal Ġwith Ġmulti - mod al Ġdata . ĠWe Ġdescribe Ġa ĠPoly Rep onse Ġsystem Ġthat Ġassists Ġthe Ġusers Ġin Ġfinding Ġa Ġrelevant Ġrestaurant Ġaccording Ġto Ġtheir Ġpreference , Ġand Ġthen Ġadditionally Ġhelps Ġthem Ġto Ġmake Ġa Ġbooking Ġin Ġthe Ġselected Ġrestaurant . ĠDue Ġto Ġits Ġretrieval - based Ġdesign , Ġwith Ġthe ĠPoly Response Ġengine Ġthere Ġis Ġno Ġneed Ġto Ġengineer Ġa Ġstructured Ġont ology , Ġor Ġto Ġsolve Ġthe Ġdifficult Ġtask Ġof Ġgeneral Ġlanguage Ġgeneration . ĠThis Ġdesign Ġalso Ġbypass es Ġthe Ġconstruction Ġof Ġdedicated Ġdecision - making Ġpolicy Ġmodules . ĠThe Ġlarge Ġranking Ġmodel Ġalready Ġencaps ulates Ġa Ġlot Ġof Ġknowledge Ġabout Ġnatural Ġlanguage Ġand Ġconvers ational Ġflow . </s> Since Ġretrieved Ġsystem Ġresponses Ġare Ġpresented Ġvisually Ġto Ġthe Ġuser , Ġthe ĠPoly Response Ġrestaurant Ġsearch Ġengine Ġis Ġable Ġto Ġcombine Ġtext Ġresponses Ġwith Ġrelevant Ġvisual Ġinformation Ġ( e . g ., Ġphotos Ġfrom Ġsocial Ġmedia Ġassociated Ġwith Ġthe Ġcurrent Ġrestaurant Ġand Ġrelated Ġto Ġthe Ġuser Ġutter ance ), Ġeffectively Ġyielding Ġa Ġmulti - mod al Ġresponse . ĠThis Ġsetup Ġof Ġusing Ġvoice Ġas Ġinput , Ġand Ġresponding Ġvisually Ġis Ġbecoming Ġmore Ġand Ġmore Ġprevalent Ġwith Ġthe Ġrise Ġof Ġsmart Ġscreens Ġlike ĠEcho ĠShow Ġand Ġeven Ġmixed Ġreality . ĠFinally , Ġthe ĠPoly Response Ġrestaurant Ġsearch Ġengine Ġis Ġmult ilingual : Ġit Ġis Ġcurrently Ġdeployed Ġin Ġ8 Ġlanguages Ġenabling Ġsearch Ġover Ġrestaurants Ġin Ġ8 Ġcities Ġaround Ġthe Ġworld . ĠSystem Ġsnapshots Ġin Ġfour Ġdifferent Ġlanguages Ġare Ġpresented Ġin ĠFigure ĠFIG REF 16 , Ġwhile Ġscreen cast Ġvideos Ġthat Ġillustrate Ġthe Ġdialogue Ġflow Ġwith Ġthe ĠPoly Response Ġengine Ġare Ġavailable Ġat : Ġhttps :// tiny url . com / y 3 ev k cf z . </s> Poly Response : ĠConvers ational ĠSearch </s> The ĠPoly Response Ġsystem Ġis Ġpowered Ġby Ġa Ġsingle Ġlarge Ġconvers ational Ġsearch Ġengine , Ġtrained Ġon Ġa Ġlarge Ġamount Ġof Ġconvers ational Ġand Ġimage Ġdata , Ġas Ġshown Ġin ĠFigure ĠFIG REF 2 . ĠIn Ġsimple Ġwords , Ġit Ġis Ġa Ġranking Ġmodel Ġthat Ġlearns Ġto Ġscore Ġconvers ational Ġreplies Ġand Ġimages Ġin Ġa Ġgiven Ġconvers ational Ġcontext . ĠThe Ġhighest - scoring Ġresponses Ġare Ġthen Ġretrieved Ġas Ġsystem Ġoutputs . ĠThe Ġsystem Ġcomp utes Ġtwo Ġsets Ġof Ġsimilarity Ġscores : Ġ1 ) Ġ$ S ( r , c ) $ Ġis Ġthe Ġscore Ġof Ġa Ġcandidate Ġreply Ġ$ r $ Ġgiven Ġa Ġconvers ational Ġcontext Ġ$ c $, Ġand Ġ2 ) Ġ$ S ( p , c ) $ Ġis Ġthe Ġscore Ġof Ġa Ġcandidate Ġphoto Ġ$ p $ Ġgiven Ġa Ġconvers ational Ġcontext Ġ$ c $. ĠThese Ġscores Ġare Ġcomputed Ġas Ġa Ġscaled Ġcos ine Ġsimilarity Ġof Ġa Ġvector Ġthat Ġrepresents Ġthe Ġcontext Ġ($ h _ c $ ), Ġand Ġa Ġvector Ġthat Ġrepresents Ġthe Ġcandidate Ġresponse : Ġa Ġtext Ġreply Ġ($ h _ r $ ) Ġor Ġa Ġphoto Ġ($ h _ p $ ). ĠFor Ġinstance , Ġ$ S ( r , c ) $ Ġis Ġcomputed Ġas Ġ$ S ( r , c )= C Ġcos ( h _ r , h _ c ) $, Ġwhere Ġ$ C $ Ġis Ġa Ġlearned Ġconstant . ĠThe Ġpart Ġof Ġthe Ġmodel Ġdealing Ġwith Ġtext Ġinput Ġ( i . e ., Ġobtaining Ġthe Ġenc od ings Ġ$ h _ c $ Ġand Ġ$ h _ r $ ) Ġfollows Ġthe Ġarchitecture Ġintroduced Ġrecently Ġby ĠHenderson : 2019 acl . ĠWe Ġprovide Ġonly Ġa Ġbrief Ġrecap Ġhere ; Ġsee Ġthe Ġoriginal Ġpaper Ġfor Ġfurther Ġdetails . </s> Poly Response : ĠConvers ational ĠSearch Ġ: :: ĠText ĠRepresent ation . </s> The Ġmodel , Ġimplemented Ġas Ġa Ġdeep Ġneural Ġnetwork , Ġlearns Ġto Ġrespond Ġby Ġtraining Ġon Ġhundreds Ġof Ġmillions Ġcontext - reply Ġ$( c , r ) $ Ġpairs . ĠFirst , Ġsimilar Ġto ĠHenderson : 2017 ar x iv , Ġraw Ġtext Ġfrom Ġboth Ġ$ c $ Ġand Ġ$ r $ Ġis Ġconverted Ġto Ġun ig rams Ġand Ġbig rams . ĠAll Ġinput Ġtext Ġis Ġfirst Ġlower - c ased Ġand Ġtoken ised , Ġnumbers Ġwith Ġ5 Ġor Ġmore Ġdigits Ġget Ġtheir Ġdigits Ġreplaced Ġby Ġa Ġwild card Ġsymbol Ġ# , Ġwhile Ġwords Ġlonger Ġthan Ġ16 Ġcharacters Ġare Ġreplaced Ġby Ġa Ġwild card Ġtoken ĠLONG W ORD . ĠSent ence Ġboundary Ġtokens Ġare Ġadded Ġto Ġeach Ġsentence . ĠThe Ġvocabulary Ġconsists Ġof Ġthe Ġun ig rams Ġthat Ġoccur Ġat Ġleast Ġ10 Ġtimes Ġin Ġa Ġrandom Ġ10 M Ġsubset Ġof Ġthe ĠReddit Ġtraining Ġset Ġ( see ĠFigure ĠFIG REF 2 ) Ġplus Ġthe Ġ200 K Ġmost Ġfrequent Ġbig rams Ġin Ġthe Ġsame Ġrandom Ġsubset . </s> During Ġtraining , Ġwe Ġobtain Ġ$ d $ - dimensional Ġfeature Ġrepresentations Ġ($ d = 320 $ ) Ġshared Ġbetween Ġcontexts Ġand Ġreplies Ġfor Ġeach Ġun ig ram Ġand Ġbig ram Ġjointly Ġwith Ġother Ġneural Ġnet Ġparameters . ĠA Ġstate - of - the - art Ġarchitecture Ġbased Ġon Ġtransform ers ĠB IB REF 13 Ġis Ġthen Ġapplied Ġto Ġun ig ram Ġand Ġbig ram Ġvectors Ġseparately , Ġwhich Ġare Ġthen Ġaveraged Ġto Ġform Ġthe Ġfinal Ġ320 - dimensional Ġencoding . ĠThat Ġencoding Ġis Ġthen Ġpassed Ġthrough Ġthree Ġfully - connected Ġnon - linear Ġhidden Ġlayers Ġof Ġdimension ality Ġ$ 1 , 024 $. ĠThe Ġfinal Ġlayer Ġis Ġlinear Ġand Ġmaps Ġthe Ġtext Ġinto Ġthe Ġfinal Ġ$ l $ - dimensional Ġ($ l = 512 $ ) Ġrepresentation : Ġ$ h _ c $ Ġand Ġ$ h _ r $. ĠOther Ġstandard Ġand Ġmore Ġsophisticated Ġenc oder Ġmodels Ġcan Ġalso Ġbe Ġused Ġto Ġprovide Ġfinal Ġenc od ings Ġ$ h _ c $ Ġand Ġ$ h _ r $, Ġbut Ġthe Ġcurrent Ġarchitecture Ġshows Ġa Ġgood Ġtrade - off Ġbetween Ġspeed Ġand Ġefficacy Ġwith Ġstrong Ġand Ġrobust Ġperformance Ġin Ġour Ġempirical Ġevaluations Ġon Ġthe Ġresponse Ġretrieval Ġtask Ġusing ĠReddit ĠB IB REF 14 , ĠOpen Sub t itles ĠB IB REF 15 , Ġand ĠAmazon Q A ĠB IB REF 16 Ġconvers ational Ġtest Ġdata , Ġsee ĠB IB REF 12 Ġfor Ġfurther Ġdetails . </s> In Ġtraining Ġthe Ġconstant Ġ$ C $ Ġis Ġconstrained Ġto Ġlie Ġbetween Ġ0 Ġand Ġ$\ sq rt { l } $. ĠFollowing ĠHenderson : 2017 ar x iv , Ġthe Ġscoring Ġfunction Ġin Ġthe Ġtraining Ġobjective Ġaims Ġto Ġmaxim ise Ġthe Ġsimilarity Ġscore Ġof Ġcontext - reply Ġpairs Ġthat Ġgo Ġtogether , Ġwhile Ġminim ising Ġthe Ġscore Ġof Ġrandom Ġpair ings : Ġnegative Ġexamples . ĠTraining Ġproceeds Ġvia ĠSG D Ġwith Ġbatches Ġcomprising Ġ500 Ġpairs Ġ( 1 Ġpositive Ġand Ġ499 Ġnegatives ). </s> Poly Response : ĠConvers ational ĠSearch Ġ: :: ĠPhoto ĠRepresent ation . </s> Photos Ġare Ġrepresented Ġusing Ġconv olution al Ġneural Ġnet Ġ( CNN ) Ġmodels Ġpret rained Ġon ĠImage Net ĠB IB REF 17 . ĠWe Ġuse Ġa ĠMobile Net Ġmodel Ġwith Ġa Ġdepth Ġmultiplier Ġof Ġ1 . 4 , Ġand Ġan Ġinput Ġdimension Ġof Ġ$ 224 Ġ\ times Ġ224 $ Ġpixels Ġas Ġin ĠB IB REF 18 . ĠThis Ġprovides Ġa Ġ$ 1 , 280 Ġ\ times Ġ1 . 4 Ġ= Ġ1 , 792 $ - dimensional Ġrepresentation Ġof Ġa Ġphoto , Ġwhich Ġis Ġthen Ġpassed Ġthrough Ġa Ġsingle Ġhidden Ġlayer Ġof Ġdimension ality Ġ$ 1 , 024 $ Ġwith ĠRe LU Ġactivation , Ġbefore Ġbeing Ġpassed Ġto Ġa Ġhidden Ġlayer Ġof Ġdimension ality Ġ512 Ġwith Ġno Ġactivation Ġto Ġprovide Ġthe Ġfinal Ġrepresentation Ġ$ h _ p $. </s> Poly Response : ĠConvers ational ĠSearch Ġ: :: ĠData ĠSource Ġ1 : ĠReddit . </s> For Ġtraining Ġtext Ġrepresentations Ġwe Ġuse Ġa ĠReddit Ġdataset Ġsimilar Ġto ĠAl R f ou : 2016 ar x iv . ĠOur Ġdataset Ġis Ġlarge Ġand Ġprovides Ġnatural Ġconvers ational Ġstructure : Ġall ĠReddit Ġdata Ġfrom ĠJanuary Ġ2015 Ġto ĠDecember Ġ2018 , Ġavailable Ġas Ġa Ġpublic ĠBig Query Ġdataset , Ġspan Ġalmost Ġ3 . 7 B Ġcomments ĠB IB REF 12 . ĠWe Ġpre process Ġthe Ġdataset Ġto Ġremove Ġunin form ative Ġand Ġlong Ġcomments Ġby Ġretaining Ġonly Ġsentences Ġcontaining Ġmore Ġthan Ġ8 Ġand Ġless Ġthan Ġ128 Ġword Ġtokens . ĠAfter Ġpairing Ġall Ġcomments / context s Ġ$ c $ Ġwith Ġtheir Ġreplies Ġ$ r $, Ġwe Ġobtain Ġmore Ġthan Ġ7 27 M Ġcontext - reply Ġ$( c , r ) $ Ġpairs Ġfor Ġtraining , Ġsee ĠFigure ĠFIG REF 2 . </s> Poly Response : ĠConvers ational ĠSearch Ġ: :: ĠData ĠSource Ġ2 : ĠYelp . </s> Once Ġthe Ġtext Ġencoding Ġsub - net works Ġare Ġtrained , Ġa Ġphoto Ġenc oder Ġis Ġlearned Ġon Ġtop Ġof Ġa Ġpret rained ĠMobile Net ĠCNN , Ġusing Ġdata Ġtaken Ġfrom Ġthe ĠYelp ĠOpen Ġdataset : Ġit Ġcontains Ġaround Ġ200 K Ġphotos Ġand Ġtheir Ġcapt ions . ĠTraining Ġof Ġthe Ġmulti - mod al Ġsub - network Ġthen Ġmaxim ises Ġthe Ġsimilarity Ġof Ġcapt ions Ġencoded Ġwith Ġthe Ġresponse Ġenc oder Ġ$ h _ r $ Ġto Ġthe Ġphoto Ġrepresentation Ġ$ h _ p $. ĠAs Ġa Ġresult , Ġwe Ġcan Ġcompute Ġthe Ġscore Ġof Ġa Ġphoto Ġgiven Ġa Ġcontext Ġusing Ġthe Ġcos ine Ġsimilarity Ġof Ġthe Ġrespective Ġvectors . ĠA Ġphoto Ġwill Ġbe Ġscored Ġhighly Ġif Ġit Ġlooks Ġlike Ġits Ġcaption Ġwould Ġbe Ġa Ġgood Ġresponse Ġto Ġthe Ġcurrent Ġcontext . </s> Poly Response : ĠConvers ational ĠSearch Ġ: :: ĠIndex Ġof ĠRespons es . </s> The ĠYelp Ġdataset Ġis Ġused Ġat Ġinference Ġtime Ġto Ġprovide Ġtext Ġand Ġphoto Ġcandidates Ġto Ġdisplay Ġto Ġthe Ġuser Ġat Ġeach Ġstep Ġin Ġthe Ġconversation . ĠOur Ġrestaurant Ġsearch Ġis Ġcurrently Ġdeployed Ġseparately Ġfor Ġeach Ġcity , Ġand Ġwe Ġlimit Ġthe Ġresponses Ġto Ġa Ġgiven Ġcity . ĠFor Ġinstance , Ġfor Ġour ĠEnglish Ġsystem Ġfor ĠEdinburgh Ġwe Ġwork Ġwith Ġ396 Ġrestaurants , Ġ4 , 225 Ġphotos Ġ( these Ġinclude Ġadditional Ġphotos Ġobtained Ġusing Ġthe ĠGoogle ĠPlaces ĠAPI Ġwithout Ġcapt ions ), Ġ6 , 725 Ġresponses Ġcreated Ġfrom Ġthe Ġstructured Ġinformation Ġabout Ġrestaurants Ġthat ĠYelp Ġprovides , Ġconverted Ġusing Ġsimple Ġtemplates Ġto Ġsentences Ġof Ġthe Ġform Ġsuch Ġas ĠâĢ ľ Rest aur ant ĠX Ġaccepts Ġcredit Ġcards . âĢ Ŀ , Ġ125 , 830 Ġsentences Ġextracted Ġfrom Ġonline Ġreviews . </s> Poly Response : ĠConvers ational ĠSearch Ġ: :: ĠPoly Response Ġin Ġa ĠN uts hell . </s> The Ġsystem Ġjointly Ġtrains Ġtwo Ġencoding Ġfunctions Ġ( with Ġshared Ġword Ġembed d ings ) Ġ$ f ( context ) $ Ġand Ġ$ g ( reply ) $ Ġwhich Ġproduce Ġenc od ings Ġ$ h _ c $ Ġand Ġ$ h _ r $, Ġso Ġthat Ġthe Ġsimilarity Ġ$ S ( c , r ) $ Ġis Ġhigh Ġfor Ġall Ġ$( c , r ) $ Ġpairs Ġfrom Ġthe ĠReddit Ġtraining Ġdata Ġand Ġlow Ġfor Ġrandom Ġpairs . ĠThe Ġencoding Ġfunction Ġ$ g () $ Ġis Ġthen Ġfrozen , Ġand Ġan Ġencoding Ġfunction Ġ$ t ( photo ) $ Ġis Ġlearnt Ġwhich Ġmakes Ġthe Ġsimilarity Ġbetween Ġa Ġphoto Ġand Ġits Ġassociated Ġcaption Ġhigh Ġfor Ġall Ġ( photo , Ġcaption ) Ġpairs Ġfrom Ġthe ĠYelp Ġdataset , Ġand Ġlow Ġfor Ġrandom Ġpairs . Ġ$ t $ Ġis Ġa ĠCNN Ġpret rained Ġon ĠImage Net , Ġwith Ġa Ġshallow Ġone - layer ĠD NN Ġon Ġtop . ĠGiven Ġa Ġnew Ġcontext / query , Ġwe Ġthen Ġprovide Ġits Ġencoding Ġ$ h _ c $ Ġby Ġapplying Ġ$ f () $, Ġand Ġfind Ġplausible Ġtext Ġreplies Ġand Ġphoto Ġresponses Ġaccording Ġto Ġfunctions Ġ$ g () $ Ġand Ġ$ t () $, Ġrespectively . ĠThese Ġshould Ġbe Ġresponses Ġthat Ġlook Ġlike Ġanswers Ġto Ġthe Ġquery , Ġand Ġphotos Ġthat Ġlook Ġlike Ġthey Ġwould Ġhave Ġcapt ions Ġthat Ġwould Ġbe Ġanswers Ġto Ġthe Ġprovided Ġquery . </s> At Ġinference , Ġfinding Ġrelevant Ġcandidates Ġgiven Ġa Ġcontext Ġreduces Ġto Ġcomputing Ġ$ h _ c $ Ġfor Ġthe Ġcontext Ġ$ c $ Ġ, Ġand Ġfinding Ġnearby Ġ$ h _ r $ Ġand Ġ$ h _ p $ Ġvectors . ĠThe Ġresponse Ġvectors Ġcan Ġall Ġbe Ġpre - com puted , Ġand Ġthe Ġnearest Ġneighbour Ġsearch Ġcan Ġbe Ġfurther Ġoptim ised Ġusing Ġstandard Ġlibraries Ġsuch Ġas ĠFa iss ĠB IB REF 19 Ġor Ġapproximate Ġnearest Ġneighbour Ġretrieval ĠB IB REF 20 , Ġgiving Ġan Ġefficient Ġsearch Ġthat Ġscales Ġto Ġbillions Ġof Ġcandidate Ġresponses . </s> The Ġsystem Ġoffers Ġboth Ġvoice Ġand Ġtext Ġinput Ġand Ġoutput . ĠSpeech - to - text Ġand Ġtext - to - speech Ġconversion Ġin Ġthe ĠPoly Response Ġsystem Ġis Ġcurrently Ġsupported Ġby Ġthe Ġoff - the - she lf ĠGoogle ĠCloud Ġtools . </s> Dialogue ĠFlow </s> The Ġranking Ġmodel Ġlends Ġitself Ġto Ġthe Ġone - shot Ġtask Ġof Ġfinding Ġthe Ġmost Ġrelevant Ġresponses Ġin Ġa Ġgiven Ġcontext . ĠHowever , Ġa Ġrestaurant - b rows ing Ġsystem Ġneeds Ġto Ġsupport Ġa Ġdialogue Ġflow Ġwhere Ġthe Ġuser Ġfinds Ġa Ġrestaurant , Ġand Ġthen Ġasks Ġquestions Ġabout Ġit . ĠThe Ġdialogue Ġstate Ġfor Ġeach Ġsearch Ġscenario Ġis Ġrepresented Ġas Ġthe Ġset Ġof Ġrestaurants Ġthat Ġare Ġconsidered Ġrelevant . ĠThis Ġstarts Ġoff Ġas Ġall Ġthe Ġrestaurants Ġin Ġthe Ġgiven Ġcity , Ġand Ġis Ġassumed Ġto Ġmon oton ically Ġdecrease Ġin Ġsize Ġas Ġthe Ġconversation Ġprogresses Ġuntil Ġthe Ġuser Ġconver ges Ġto Ġa Ġsingle Ġrestaurant . ĠA Ġrestaurant Ġis Ġonly Ġconsidered Ġvalid Ġin Ġthe Ġcontext Ġof Ġa Ġnew Ġuser Ġinput Ġif Ġit Ġhas Ġrelevant Ġresponses Ġcorresponding Ġto Ġit . ĠThis Ġflow Ġis Ġsummar ised Ġhere : </s> S 1 . ĠInitial ise Ġ$ R $ Ġas Ġthe Ġset Ġof Ġall Ġrestaurants Ġin Ġthe Ġcity . ĠGiven Ġthe Ġuser 's Ġinput , Ġrank Ġall Ġthe Ġresponses Ġin Ġthe Ġresponse Ġpool Ġpertaining Ġto Ġrestaurants Ġin Ġ$ R $. </s> S 2 . ĠRet rieve Ġthe Ġtop Ġ$ N $ Ġresponses Ġ$ r _ 1 , Ġr _ 2 , Ġ\ ld ots Ġ, Ġr _ N $ Ġwith Ġcorresponding Ġ( s orted ) Ġcos ine Ġsimilarity Ġscores : Ġ$ s _ 1 Ġ\ ge Ġs _ 2 Ġ\ ge Ġ\ ld ots Ġ\ ge Ġs _ N $. </s> S 3 . ĠComp ute Ġprobability Ġscores Ġ$ p _ i Ġ\ pro pt o Ġ\ exp Ġ( a Ġ\ cd ot Ġs _ i ) $ Ġwith Ġ$\ sum Ġ_ { i = 1 } ^ N Ġp _ i $, Ġwhere Ġ$ a > 0 $ Ġis Ġa Ġtun able Ġconstant . </s> S 4 . ĠComp ute Ġa Ġscore Ġ$ q _ e $ Ġfor Ġeach Ġrestaurant / entity Ġ$ e Ġ\ in ĠR $, Ġ$ q _ e Ġ= Ġ\ sum Ġ_ { i : Ġr _ i Ġ\ in Ġe } Ġp _ i $. </s> S 5 . ĠUpdate Ġ$ R $ Ġto Ġthe Ġsmallest Ġset Ġof Ġrestaurants Ġwith Ġhighest Ġ$ q $ Ġwhose Ġ$ q $ - values Ġsum Ġup Ġto Ġmore Ġthan Ġa Ġpred efined Ġthreshold Ġ$ t $. </s> S 6 . ĠDisplay Ġthe Ġmost Ġrelevant Ġresponses Ġassociated Ġwith Ġthe Ġupdated Ġ$ R $, Ġand Ġreturn Ġto ĠS 2 . </s> If Ġthere Ġare Ġmultiple Ġrelevant Ġrestaurants , Ġone Ġresponse Ġis Ġshown Ġfrom Ġeach . ĠWhen Ġonly Ġone Ġrestaurant Ġis Ġrelevant , Ġthe Ġtop Ġ$ N $ Ġresponses Ġare Ġall Ġshown , Ġand Ġrelevant Ġphotos Ġare Ġalso Ġdisplayed . ĠThe Ġsystem Ġdoes Ġnot Ġrequire Ġdedicated Ġunderstanding , Ġdecision - making , Ġand Ġgeneration Ġmodules , Ġand Ġthis Ġdialogue Ġflow Ġdoes Ġnot Ġrely Ġon Ġexplicit Ġtask - tail ored Ġsemantics . ĠThe Ġset Ġof Ġrelevant Ġrestaurants Ġis Ġkept Ġinternally Ġwhile Ġthe Ġsystem Ġnarrow s Ġit Ġdown Ġacross Ġmultiple Ġdialogue Ġturns . ĠA Ġsimple Ġset Ġof Ġpred efined Ġrules Ġis Ġused Ġto Ġprovide Ġa Ġtem pl atic Ġspoken Ġsystem Ġresponse : Ġe . g ., Ġan Ġexample Ġrule Ġis ĠâĢ ľ One Ġreview Ġof Ġ$ e $ Ġsaid Ġ$ r $ âĢ Ŀ , Ġwhere Ġ$ e $ Ġrefers Ġto Ġthe Ġrestaurant , Ġand Ġ$ r $ Ġto Ġa Ġrelevant Ġresponse Ġassociated Ġwith Ġ$ e $. ĠNote Ġthat Ġwhile Ġthe Ġdemo Ġis Ġcurrently Ġfocused Ġon Ġthe Ġrestaurant Ġsearch Ġtask , Ġthe Ġdescribed ĠâĢ ľ n arrow ing Ġdown âĢ Ŀ Ġdialogue Ġflow Ġis Ġgeneric Ġand Ġapplicable Ġto Ġa Ġvariety Ġof Ġapplications Ġdealing Ġwith Ġsimilar Ġentity Ġsearch . </s> The Ġsystem Ġcan Ġuse Ġa Ġset Ġof Ġintent Ġclass ifiers Ġto Ġallow Ġreset ting Ġthe Ġdialogue Ġstate , Ġor Ġto Ġactivate Ġthe Ġseparate Ġrestaurant Ġbooking Ġdialogue Ġflow . ĠThese Ġclass ifiers Ġare Ġbriefly Ġdiscussed Ġin ĠÂ§ SEC REF 4 . </s> Other ĠFunction ality Ġ: :: ĠMult iling uality . </s> The ĠPoly Response Ġrestaurant Ġsearch Ġis Ġcurrently Ġavailable Ġin Ġ8 Ġlanguages Ġand Ġfor Ġ8 Ġcities Ġaround Ġthe Ġworld : ĠEnglish Ġ( Ed inburgh ), ĠGerman Ġ( Ber lin ), ĠSpanish Ġ( Mad rid ), ĠMandarin Ġ( Tai pei ), ĠPolish Ġ( Wars aw ), ĠRussian Ġ( Moscow ), ĠKorean Ġ( Se oul ), Ġand ĠSerbian Ġ( Bel grade ). ĠSelected Ġsnapshots Ġare Ġshown Ġin ĠFigure ĠFIG REF 16 , Ġwhile Ġwe Ġalso Ġprovide Ġvideos Ġdemonstrating Ġthe Ġuse Ġand Ġbehaviour Ġof Ġthe Ġsystems Ġat : Ġhttps :// tiny url . com / y 3 ev k cf z . ĠA Ġsimple ĠMT - based Ġtranslate - to - source Ġapproach Ġat Ġinference Ġtime Ġis Ġcurrently Ġused Ġto Ġenable Ġthe Ġdeployment Ġof Ġthe Ġsystem Ġin Ġother Ġlanguages : Ġ1 ) Ġthe Ġpool Ġof Ġresponses Ġin Ġeach Ġlanguage Ġis Ġtranslated Ġto ĠEnglish Ġby ĠGoogle ĠTrans late Ġbeforehand , Ġand Ġpre - com puted Ġenc od ings Ġof Ġtheir ĠEnglish Ġtranslations Ġare Ġused Ġas Ġrepresentations Ġof Ġeach Ġforeign Ġlanguage Ġresponse ; Ġ2 ) Ġa Ġprovided Ġuser Ġutter ance Ġ( i . e ., Ġcontext ) Ġis Ġtranslated Ġto ĠEnglish Ġon - the - fly Ġand Ġits Ġencoding Ġ$ h _ c $ Ġis Ġthen Ġlearned . ĠWe Ġplan Ġto Ġexperiment Ġwith Ġmore Ġsophisticated Ġmult ilingual Ġmodels Ġin Ġfuture Ġwork . </s> Other ĠFunction ality Ġ: :: ĠVoice - Cont rolled ĠMenu ĠSearch . </s> An Ġadditional Ġfunctionality Ġenables Ġthe Ġuser Ġto Ġget Ġparts Ġof Ġthe Ġrestaurant Ġmenu Ġrelevant Ġto Ġthe Ġcurrent Ġuser Ġutter ance Ġas Ġresponses . ĠThis Ġis Ġachieved Ġby Ġperforming Ġan Ġadditional Ġranking Ġstep Ġof Ġavailable Ġmenu Ġitems Ġand Ġretrieving Ġthe Ġones Ġthat Ġare Ġsem antically Ġrelevant Ġto Ġthe Ġuser Ġutter ance Ġusing Ġexactly Ġthe Ġsame Ġmethodology Ġas Ġwith Ġranking Ġother Ġresponses . ĠAn Ġexample Ġof Ġthis Ġfunctionality Ġis Ġshown Ġin ĠFigure ĠFIG REF 21 . </s> Other ĠFunction ality Ġ: :: ĠReset ting Ġand ĠSwitch ing Ġto ĠBook ing . </s> The Ġrestaurant Ġsearch Ġsystem Ġneeds Ġto Ġsupport Ġthe Ġdiscrete Ġactions Ġof Ġrestart ing Ġthe Ġconversation Ġ( i . e ., Ġreset ting Ġthe Ġset Ġ$ R $ ), Ġand Ġshould Ġenable Ġtransferring Ġto Ġthe Ġslot - based Ġtable Ġbooking Ġflow . ĠThis Ġis Ġachieved Ġusing Ġtwo Ġbinary Ġintent Ġclass ifiers , Ġthat Ġare Ġrun Ġat Ġeach Ġstep Ġin Ġthe Ġdialogue . ĠThese Ġclass ifiers Ġmake Ġuse Ġof Ġthe Ġalready - com puted Ġ$ h _ c $ Ġvector Ġthat Ġrepresents Ġthe Ġuser 's Ġlatest Ġtext . ĠA Ġsingle - layer Ġneural Ġnet Ġis Ġlearned Ġon Ġtop Ġof Ġthe Ġ512 - dimensional Ġencoding , Ġwith Ġa ĠRe LU Ġactivation Ġand Ġ100 Ġhidden Ġnodes . ĠTo Ġtrain Ġthe Ġclass ifiers , Ġsets Ġof Ġ20 Ġrelevant Ġparaph r ases Ġ( e . g ., ĠâĢ ľ Start Ġagain âĢ Ŀ ) Ġare Ġprovided Ġas Ġpositive Ġexamples . ĠFinally , Ġwhen Ġthe Ġsystem Ġsuccessfully Ġswitches Ġto Ġthe Ġbooking Ġscenario , Ġit Ġproceeds Ġto Ġthe Ġslot Ġfilling Ġtask : Ġit Ġaims Ġto Ġextract Ġall Ġthe Ġrelevant Ġbooking Ġinformation Ġfrom Ġthe Ġuser Ġ( e . g ., Ġdate , Ġtime , Ġnumber Ġof Ġpeople Ġto Ġd ine ). ĠThe Ġentire Ġflow Ġof Ġthe Ġsystem Ġillustrating Ġboth Ġthe Ġsearch Ġphase Ġand Ġthe Ġbooking Ġphase Ġis Ġprovided Ġas Ġthe Ġsupplemental Ġvideo Ġmaterial . </s> Conclusion Ġand ĠFuture ĠWork </s> This Ġpaper Ġhas Ġpresented Ġa Ġgeneral Ġapproach Ġto Ġsearch - based Ġdialogue Ġthat Ġdoes Ġnot Ġrely Ġon Ġexplicit Ġsemantic Ġrepresentations Ġsuch Ġas Ġdialogue Ġacts Ġor Ġslot - value Ġont ologies , Ġand Ġallows Ġfor Ġmulti - mod al Ġresponses . ĠIn Ġfuture Ġwork , Ġwe Ġwill Ġextend Ġthe Ġcurrent Ġdemo Ġsystem Ġto Ġmore Ġtasks Ġand Ġlanguages , Ġand Ġwork Ġwith Ġmore Ġsophisticated Ġenc od ers Ġand Ġranking Ġfunctions . ĠBesides Ġthe Ġinitial Ġdialogue Ġflow Ġfrom Ġthis Ġwork Ġ( Â§ SEC REF 3 ), Ġwe Ġwill Ġalso Ġwork Ġwith Ġmore Ġcomplex Ġflows Ġdealing , Ġe . g ., Ġwith Ġuser Ġintent Ġshifts .