Unif and Stopword
EXTRACTIVE
Unif and Stopword
EXTRACTIVE
We quantify the efficiency-accuracy tradeoff compared to two rule-based baselines: Unif and Stopword. The Unif encoder randomly keeps tokens to generate keywords with the probability $\delta $. The Stopword encoder keeps all tokens but drops stop words (e.g. `the', `a', `or') all the time ($\delta =0$) or half of the time ($\delta =0.5$). The corresponding decoders for these encoders are optimized using gradient descent to minimize the reconstruction error (i.e. $\mathrm {loss}(x, \alpha , \beta )$).
We quantify the efficiency-accuracy tradeoff compared to two rule-based baselines: Unif and Stopword. The Unif encoder randomly keeps tokens to generate keywords with the probability $\delta $. The Stopword encoder keeps all tokens but drops stop words (e.g. `the', `a', `or') all the time ($\delta =0$) or half of the time ($\delta =0.5$). The corresponding decoders for these encoders are optimized using gradient descent to minimize the reconstruction error (i.e. $\mathrm {loss}(x, \alpha , \beta )$).
1911.06964
Introduction </s> Supp ose Ġa Ġuser Ġwants Ġto Ġwrite Ġa Ġsentence ĠâĢ ľ I Ġwill Ġbe Ġ10 Ġminutes Ġlate . âĢ Ŀ ĠIdeally , Ġshe Ġwould Ġtype Ġjust Ġa Ġfew Ġkeywords Ġsuch Ġas ĠâĢ ľ 10 Ġminutes Ġlate âĢ Ŀ Ġand Ġan Ġaut ocom plete Ġsystem Ġwould Ġbe Ġable Ġto Ġinfer Ġthe Ġintended Ġsentence Ġ( Figure ĠFIG REF 1 ). ĠEx isting Ġleft - to - right Ġaut ocom plete Ġsystems ĠB IB REF 0 , ĠB IB REF 1 Ġcan Ġoften Ġbe Ġinefficient , Ġas Ġthe Ġprefix Ġof Ġa Ġsentence Ġ( e . g . ĠâĢ ľ I Ġwill Ġbe âĢ Ŀ ) Ġfails Ġto Ġcapture Ġthe Ġcore Ġmeaning Ġof Ġthe Ġsentence . ĠBesides Ġthe Ġpractical Ġgoal Ġof Ġbuilding Ġa Ġbetter Ġaut ocom plete Ġsystem , Ġwe Ġare Ġinterested Ġin Ġexploring Ġthe Ġtrade offs Ġinherent Ġto Ġsuch Ġcommunication Ġschemes Ġbetween Ġthe Ġefficiency Ġof Ġtyping Ġkeywords , Ġaccuracy Ġof Ġreconstruction , Ġand Ġinterpret ability Ġof Ġkeywords . </s> One Ġapproach Ġto Ġlearn Ġsuch Ġschemes Ġis Ġto Ġcollect Ġa Ġsupervised Ġdataset Ġof Ġkeywords - sent ence Ġpairs Ġas Ġa Ġtraining Ġset , Ġbut Ġ( i ) Ġit Ġwould Ġbe Ġexpensive Ġto Ġcollect Ġsuch Ġdata Ġfrom Ġusers , Ġand Ġ( ii ) Ġa Ġstatic Ġdataset Ġwould Ġnot Ġcapture Ġa Ġreal Ġuser 's Ġnatural Ġpred ile ction Ġto Ġadapt Ġto Ġthe Ġsystem ĠB IB REF 2 . ĠAnother Ġapproach Ġis Ġto Ġavoid Ġsupervision Ġand Ġjointly Ġlearn Ġa Ġuser - system Ġcommunication Ġscheme Ġto Ġdirectly Ġoptimize Ġthe Ġcombination Ġof Ġefficiency Ġand Ġaccuracy . ĠHowever , Ġlearning Ġin Ġthis Ġway Ġcan Ġlead Ġto Ġcommunication Ġschemes Ġthat Ġare Ġun interpret able Ġto Ġhumans ĠB IB REF 3 , ĠB IB REF 4 Ġ( see ĠAppendix Ġfor Ġadditional Ġrelated Ġwork ). </s> In Ġthis Ġwork , Ġwe Ġpropose Ġa Ġsimple , Ġun super vised Ġapproach Ġto Ġan Ġaut ocom plete Ġsystem Ġthat Ġis Ġefficient , Ġaccurate , Ġand Ġinterpret able . ĠFor Ġinterpret ability , Ġwe Ġrestrict Ġkeywords Ġto Ġbe Ġsubsequ ences Ġof Ġtheir Ġsource Ġsentences Ġbased Ġon Ġthe Ġintuition Ġthat Ġhumans Ġcan Ġinfer Ġmost Ġof Ġthe Ġoriginal Ġmeaning Ġfrom Ġa Ġfew Ġkeywords . ĠWe Ġthen Ġapply Ġmulti - object ive Ġoptimization Ġapproaches Ġto Ġdirectly Ġcontrol Ġand Ġachieve Ġdesirable Ġtrade offs Ġbetween Ġefficiency Ġand Ġaccuracy . </s> We Ġobserve Ġthat Ġna ively Ġoptimizing Ġa Ġlinear Ġcombination Ġof Ġefficiency Ġand Ġaccuracy Ġterms Ġis Ġunstable Ġand Ġleads Ġto Ġsub opt imal Ġschemes . ĠThus , Ġwe Ġpropose Ġa Ġnew Ġobjective Ġwhich Ġoptim izes Ġfor Ġcommunication Ġefficiency Ġunder Ġan Ġaccuracy Ġconstraint . ĠWe Ġshow Ġthis Ġnew Ġobjective Ġis Ġmore Ġstable Ġand Ġefficient Ġthan Ġthe Ġlinear Ġobjective Ġat Ġall Ġaccuracy Ġlevels . </s> As Ġa Ġproof - of - concept , Ġwe Ġbuild Ġan Ġaut ocom plete Ġsystem Ġwithin Ġthis Ġframework Ġwhich Ġallows Ġa Ġuser Ġto Ġwrite Ġsentences Ġby Ġspecifying Ġkeywords . ĠWe Ġempir ically Ġshow Ġthat Ġour Ġframework Ġproduces Ġcommunication Ġschemes Ġthat Ġare Ġ52 . 16 % Ġmore Ġaccurate Ġthan Ġrule - based Ġbas elines Ġwhen Ġspecifying Ġ77 . 37 % Ġof Ġsentences , Ġand Ġ11 . 73 % Ġmore Ġaccurate Ġthan Ġa Ġnaive , Ġweighted Ġoptimization Ġapproach Ġwhen Ġspecifying Ġ53 . 38 % Ġof Ġsentences . ĠFinally , Ġwe Ġdemonstrate Ġthat Ġhumans Ġcan Ġeasily Ġadapt Ġto Ġthe Ġkeyword - based Ġaut ocom plete Ġsystem Ġand Ġsave Ġnearly Ġ50 % Ġof Ġtime Ġcompared Ġto Ġtyping Ġa Ġfull Ġsentence Ġin Ġour Ġuser Ġstudy . </s> App roach </s> Consider Ġa Ġcommunication Ġgame Ġin Ġwhich Ġthe Ġgoal Ġis Ġfor Ġa Ġuser Ġto Ġcommunicate Ġa Ġtarget Ġsequence Ġ$ x = Ġ( x _ 1 , Ġ... , Ġx _ m ) $ Ġto Ġa Ġsystem Ġby Ġpassing Ġa Ġsequence Ġof Ġkeywords Ġ$ z = Ġ( z _ 1 , Ġ... , Ġz _ n ) $. ĠThe Ġuser Ġgenerates Ġkeywords Ġ$ z $ Ġusing Ġan Ġencoding Ġstrategy Ġ$ q _{ \ alpha Ġ} ( z \ mid Ġx ) $, Ġand Ġthe Ġsystem Ġattempts Ġto Ġguess Ġthe Ġtarget Ġsequence Ġ$ x $ Ġvia Ġa Ġdecoding Ġstrategy Ġ$ p _{ \ beta Ġ} ( x \ mid Ġz ) $. </s> A Ġgood Ġcommunication Ġscheme Ġ$( q _{ \ alpha Ġ}, Ġp _{ \ beta Ġ}) $ Ġshould Ġbe Ġboth Ġefficient Ġand Ġaccurate . ĠSpecifically , Ġwe Ġprefer Ġschemes Ġthat Ġuse Ġfewer Ġkeywords Ġ( cost ), Ġand Ġthe Ġtarget Ġsentence Ġ$ x $ Ġto Ġbe Ġreconstructed Ġwith Ġhigh Ġprobability Ġ( loss ) Ġwhere </s> Based Ġon Ġour Ġassumption Ġthat Ġhumans Ġhave Ġan Ġintuitive Ġsense Ġof Ġretaining Ġimportant Ġkeywords , Ġwe Ġrestrict Ġthe Ġset Ġof Ġschemes Ġto Ġbe Ġa Ġ( pot entially Ġnon cont iguous ) Ġsubsequ ence Ġof Ġthe Ġtarget Ġsentence . ĠOur Ġhypothesis Ġis Ġthat Ġsuch Ġsubsequ ence Ġschemes Ġnaturally Ġensure Ġinterpret ability , Ġas Ġefficient Ġhuman Ġand Ġmachine Ġcommunication Ġschemes Ġare Ġboth Ġlikely Ġto Ġinvolve Ġkeeping Ġimportant Ġcontent Ġwords . </s> App roach Ġ: :: ĠModel ing Ġwith Ġaut oen cod ers . </s> To Ġlearn Ġcommunication Ġschemes Ġwithout Ġsupervision , Ġwe Ġmodel Ġthe Ġcooperative Ġcommunication Ġbetween Ġa Ġuser Ġand Ġsystem Ġthrough Ġan Ġenc oder - dec oder Ġframework . ĠCon crete ly , Ġwe Ġmodel Ġthe Ġuser 's Ġencoding Ġstrategy Ġ$ q _{ \ alpha Ġ} ( z \ mid Ġx ) $ Ġwith Ġan Ġenc oder Ġwhich Ġenc odes Ġthe Ġtarget Ġsentence Ġ$ x $ Ġinto Ġthe Ġkeywords Ġ$ z $ Ġby Ġkeeping Ġa Ġsubset Ġof Ġthe Ġtokens . ĠThis Ġsto ch astic Ġenc oder Ġ$ q _{ \ alpha Ġ} ( z \ mid Ġx ) $ Ġis Ġdefined Ġby Ġa Ġmodel Ġwhich Ġreturns Ġthe Ġprobability Ġof Ġeach Ġtoken Ġretained Ġin Ġthe Ġfinal Ġsubsequ ence Ġ$ z $. ĠThen , Ġwe Ġsample Ġfrom ĠBern ou lli Ġdistributions Ġaccording Ġto Ġthese Ġprobabilities Ġto Ġeither Ġkeep Ġor Ġdrop Ġthe Ġtokens Ġindependently Ġ( see ĠAppendix Ġfor Ġan Ġexample ). </s> We Ġmodel Ġthe Ġaut ocom plete Ġsystem 's Ġdecoding Ġstrategy Ġ$ p _{ \ beta Ġ} ( x \ mid Ġz ) $ Ġas Ġa Ġprob abil istic Ġmodel Ġwhich Ġconditions Ġon Ġthe Ġkeywords Ġ$ z $ Ġand Ġreturns Ġa Ġdistribution Ġover Ġpredictions Ġ$ x $. ĠWe Ġuse Ġa Ġstandard Ġsequence - to - sequence Ġmodel Ġwith Ġattention Ġand Ġcopying Ġfor Ġthe Ġdec oder , Ġbut Ġany Ġmodel Ġarchitecture Ġcan Ġbe Ġused Ġ( see ĠAppendix Ġfor Ġdetails ). </s> App roach Ġ: :: ĠMulti - object ive Ġoptimization . </s> Our Ġgoal Ġnow Ġis Ġto Ġlearn Ġenc oder - dec oder Ġpairs Ġwhich Ġoptim ally Ġbalance Ġthe Ġcommunication Ġcost Ġand Ġreconstruction Ġloss . ĠThe Ġsimplest Ġapproach Ġto Ġbalancing Ġefficiency Ġand Ġaccuracy Ġis Ġto Ġweight Ġ$\ math rm Ġ{ cost } ( x , Ġ\ alpha Ġ) $ Ġand Ġ$\ math rm Ġ{ loss } ( x , Ġ\ alpha Ġ, Ġ\ beta Ġ) $ Ġlin early Ġusing Ġa Ġweight Ġ$\ lambda Ġ$ Ġas Ġfollows , </s> where Ġthe Ġexpectation Ġis Ġtaken Ġover Ġthe Ġpopulation Ġdistribution Ġof Ġsource Ġsentences Ġ$ x $, Ġwhich Ġis Ġomitted Ġto Ġsimplify Ġnotation . ĠHowever , Ġwe Ġobserve Ġthat Ġna ively Ġweight ing Ġand Ġsearching Ġover Ġ$\ lambda Ġ$ Ġis Ġsub opt imal Ġand Ġhighly Ġunstable âĢĶ even Ġslight Ġchanges Ġto Ġthe Ġweight ing Ġresults Ġin Ġdegener ate Ġschemes Ġwhich Ġkeep Ġall Ġor Ġnone Ġof Ġits Ġtokens . ĠThis Ġinstability Ġmotiv ates Ġus Ġto Ġdevelop Ġa Ġnew Ġstable Ġobjective . </s> Our Ġmain Ġtechnical Ġcontribution Ġis Ġto Ġdraw Ġinspiration Ġfrom Ġthe Ġmulti - object ive Ġoptimization Ġliterature Ġand Ġview Ġthe Ġtrade off Ġas Ġa Ġsequence Ġof Ġconstrained Ġoptimization Ġproblems , Ġwhere Ġwe Ġminimize Ġthe Ġexpected Ġcost Ġsubject Ġto Ġvarying Ġexpected Ġreconstruction Ġerror Ġconstraints Ġ$\ ep sil on Ġ$ , </s> This Ġgreatly Ġimproves Ġthe Ġstability Ġof Ġthe Ġtraining Ġprocedure . ĠWe Ġempir ically Ġobserve Ġthat Ġthe Ġmodel Ġinitially Ġkeeps Ġmost Ġof Ġthe Ġtokens Ġto Ġmeet Ġthe Ġconstraints , Ġand Ġslowly Ġlearns Ġto Ġdrop Ġunin form ative Ġwords Ġfrom Ġthe Ġkeywords Ġto Ġminimize Ġthe Ġcost . ĠFurthermore , Ġ$\ ep sil on Ġ$ Ġin ĠE q Ġ( DIS PLAY _ FORM 6 ) Ġallows Ġus Ġto Ġdirectly Ġcontrol Ġthe Ġmaximum Ġreconstruction Ġerror Ġof Ġresulting Ġschemes , Ġwhereas Ġ$\ lambda Ġ$ Ġin ĠE q Ġ( DIS PLAY _ FORM 5 ) Ġis Ġnot Ġdirectly Ġrelated Ġto Ġany Ġof Ġour Ġdes ider ata . </s> To Ġoptimize Ġthe Ġconstrained Ġobjective , Ġwe Ġconsider Ġthe ĠLag rang ian Ġof ĠE q Ġ( DIS PLAY _ FORM 6 ), </s> Much Ġlike Ġthe Ġobjective Ġin ĠE q Ġ( DIS PLAY _ FORM 5 ) Ġwe Ġcan Ġcompute Ġunbiased Ġgrad ients Ġby Ġreplacing Ġthe Ġexpectations Ġwith Ġtheir Ġaverages Ġover Ġrandom Ġmin ib atches . ĠAlthough Ġgradient Ġdescent Ġguarantees Ġconvergence Ġon ĠE q Ġ( DIS PLAY _ FORM 7 ) Ġonly Ġwhen Ġthe Ġobjective Ġis Ġconve x , Ġwe Ġfind Ġthat Ġnot Ġonly Ġis Ġthe Ġoptimization Ġstable , Ġthe Ġresulting Ġsolution Ġachieves Ġbetter Ġperformance Ġthan Ġthe Ġweight ing Ġapproach Ġin ĠE q Ġ( DIS PLAY _ FORM 5 ). </s> App roach Ġ: :: ĠOptim ization . </s> Opt im ization Ġwith Ġrespect Ġto Ġ$ q _{ \ alpha Ġ} ( z \ mid Ġx ) $ Ġis Ġchallenging Ġas Ġ$ z $ Ġis Ġdiscrete , Ġand Ġthus , Ġwe Ġcannot Ġdifferentiate Ġ$\ alpha Ġ$ Ġthrough Ġ$ z $ Ġvia Ġthe Ġchain Ġrule . ĠBecause Ġof Ġthis , Ġwe Ġuse Ġthe Ġsto ch astic ĠRE IN FOR CE Ġestimate ĠB IB REF 5 Ġas Ġfollows : </s> We Ġperform Ġjoint Ġupdates Ġon Ġ$( \ alpha Ġ, Ġ\ beta Ġ, Ġ\ lambda Ġ) $, Ġwhere Ġ$\ beta Ġ$ Ġand Ġ$\ lambda Ġ$ Ġare Ġupdated Ġvia Ġstandard Ġgradient Ġcomput ations , Ġwhile Ġ$\ alpha Ġ$ Ġuses Ġan Ġunbiased , Ġsto ch astic Ġgradient Ġestimate Ġwhere Ġwe Ġapproximate Ġthe Ġexpectation Ġin ĠE q Ġ( DIS PLAY _ FORM 9 ). ĠWe Ġuse Ġa Ġsingle Ġsample Ġfrom Ġ$ q _{ \ alpha Ġ} ( z \ mid Ġx ) $ Ġand Ġmoving - average Ġof Ġrewards Ġas Ġa Ġbaseline Ġto Ġreduce Ġvariance . </s> Exper iments </s> We Ġevaluate Ġour Ġapproach Ġby Ġtraining Ġan Ġaut ocom plete Ġsystem Ġon Ġ500 K Ġrandomly Ġsampled Ġsentences Ġfrom ĠYelp Ġreviews ĠB IB REF 6 Ġ( see ĠAppendix Ġfor Ġdetails ). ĠWe Ġquantify Ġthe Ġefficiency Ġof Ġa Ġcommunication Ġscheme Ġ$( q _{ \ alpha Ġ}, p _{ \ beta Ġ}) $ Ġby Ġthe Ġretention Ġrate Ġof Ġtokens , Ġwhich Ġis Ġmeasured Ġas Ġthe Ġfraction Ġof Ġtokens Ġthat Ġare Ġkept Ġin Ġthe Ġkeywords . ĠThe Ġaccuracy Ġof Ġa Ġscheme Ġis Ġmeasured Ġas Ġthe Ġfraction Ġof Ġsentences Ġgenerated Ġby Ġgreed ily Ġdecoding Ġthe Ġmodel Ġthat Ġexactly Ġmatches Ġthe Ġtarget Ġsentence . </s> Exper iments Ġ: :: ĠEffect iveness Ġof Ġconstrained Ġobjective . </s> We Ġfirst Ġshow Ġthat Ġthe Ġlinear Ġobjective Ġin ĠE q Ġ( DIS PLAY _ FORM 5 ) Ġis Ġsub opt imal Ġcompared Ġto Ġthe Ġconstrained Ġobjective Ġin ĠE q Ġ( DIS PLAY _ FORM 6 ). ĠFigure ĠFIG REF 10 Ġcompares Ġthe Ġachievable Ġaccuracy Ġand Ġefficiency Ġtrade offs Ġfor Ġthe Ġtwo Ġobjectives , Ġwhich Ġshows Ġthat Ġthe Ġconstrained Ġobjective Ġresults Ġin Ġmore Ġefficient Ġschemes Ġthan Ġthe Ġlinear Ġobjective Ġat Ġevery Ġaccuracy Ġlevel Ġ( e . g . Ġ11 . 73 % Ġmore Ġaccurate Ġat Ġa Ġ53 . 38 % Ġretention Ġrate ). </s> We Ġalso Ġobserve Ġthat Ġthe Ġlinear Ġobjective Ġis Ġhighly Ġunstable Ġas Ġa Ġfunction Ġof Ġthe Ġtrade off Ġparameter Ġ$\ lambda Ġ$ Ġand Ġrequires Ġcareful Ġtuning . ĠEven Ġslight Ġchanges Ġto Ġ$\ lambda Ġ$ Ġresults Ġin Ġdegener ate Ġschemes Ġthat Ġkeep Ġall Ġor Ġnone Ġof Ġthe Ġtokens Ġ( e . g . Ġ$\ lambda Ġ\ le Ġ4 . 2 $ Ġand Ġ$\ lambda Ġ\ ge Ġ4 . 4 $ ). ĠOn Ġthe Ġother Ġhand , Ġthe Ġconstrained Ġobjective Ġis Ġsubstantially Ġmore Ġstable Ġas Ġa Ġfunction Ġof Ġ$\ ep sil on Ġ$ Ġ( e . g . Ġpoints Ġfor Ġ$\ ep sil on Ġ$ Ġare Ġmore Ġevenly Ġspaced Ġthan Ġ$\ lambda Ġ$ ). </s> Exper iments Ġ: :: ĠEfficiency - acc uracy Ġtrade off . </s> We Ġquantify Ġthe Ġefficiency - acc uracy Ġtrade off Ġcompared Ġto Ġtwo Ġrule - based Ġbas elines : ĠUn if Ġand ĠStop word . ĠThe ĠUn if Ġenc oder Ġrandomly Ġkeeps Ġtokens Ġto Ġgenerate Ġkeywords Ġwith Ġthe Ġprobability Ġ$\ d elta Ġ$ . ĠThe ĠStop word Ġenc oder Ġkeeps Ġall Ġtokens Ġbut Ġdrops Ġstop Ġwords Ġ( e . g . Ġ` the ', Ġ` a ', Ġ` or ') Ġall Ġthe Ġtime Ġ($ \ d elta Ġ= 0 $ ) Ġor Ġhalf Ġof Ġthe Ġtime Ġ($ \ d elta Ġ= 0 . 5 $ ). ĠThe Ġcorresponding Ġdec od ers Ġfor Ġthese Ġenc od ers Ġare Ġoptimized Ġusing Ġgradient Ġdescent Ġto Ġminimize Ġthe Ġreconstruction Ġerror Ġ( i . e . Ġ$\ math rm Ġ{ loss } ( x , Ġ\ alpha Ġ, Ġ\ beta Ġ) $ ). </s> Figure ĠFIG REF 10 Ġshows Ġthat Ġtwo Ġbas elines Ġachieve Ġsimilar Ġtrade off Ġcurves , Ġwhile Ġthe Ġconstrained Ġmodel Ġachieves Ġa Ġsubstantial Ġ52 . 16 % Ġimprovement Ġin Ġaccuracy Ġat Ġa Ġ77 . 37 % Ġretention Ġrate Ġcompared Ġto ĠUn if , Ġthereby Ġshowing Ġthe Ġbenefits Ġof Ġjointly Ġtraining Ġthe Ġenc oder Ġand Ġdec oder . </s> Exper iments Ġ: :: ĠRob ust ness Ġand Ġanalysis . </s> We Ġprovide Ġadditional Ġexperimental Ġresults Ġon Ġthe Ġrobust ness Ġof Ġlearned Ġcommunication Ġschemes Ġas Ġwell Ġas Ġin - depth Ġanalysis Ġon Ġthe Ġcorrelation Ġbetween Ġthe Ġretention Ġrates Ġof Ġtokens Ġand Ġtheir Ġproperties , Ġwhich Ġwe Ġdefer Ġto ĠAppendix Ġand Ġfor Ġspace . </s> Exper iments Ġ: :: ĠUser Ġstudy . </s> We Ġrecruited Ġ100 Ġcrowd workers Ġon ĠAmazon ĠMechanical ĠTurk Ġ( AM T ) Ġand Ġmeasured Ġcompletion Ġtimes Ġand Ġaccur acies Ġfor Ġtyping Ġrandomly Ġsampled Ġsentences Ġfrom Ġthe ĠYelp Ġcorpus . ĠEach Ġuser Ġwas Ġshown Ġalternating Ġaut ocom plete Ġand Ġwriting Ġtasks Ġacross Ġ50 Ġsentences Ġ( see ĠAppendix Ġfor Ġuser Ġinterface ). ĠFor Ġthe Ġaut ocom plete Ġtask , Ġwe Ġgave Ġusers Ġa Ġtarget Ġsentence Ġand Ġasked Ġthem Ġto Ġtype Ġa Ġset Ġof Ġkeywords Ġinto Ġthe Ġsystem . ĠThe Ġusers Ġwere Ġshown Ġthe Ġtop Ġthree Ġsuggestions Ġfrom Ġthe Ġaut ocom plete Ġsystem , Ġand Ġwere Ġasked Ġto Ġmark Ġwhether Ġeach Ġof Ġthese Ġthree Ġsuggestions Ġwas Ġsem antically Ġequivalent Ġto Ġthe Ġtarget Ġsentence . ĠFor Ġthe Ġwriting Ġtask , Ġwe Ġgave Ġusers Ġa Ġtarget Ġsentence Ġand Ġasked Ġthem Ġto Ġeither Ġtype Ġthe Ġsentence Ġver bat im Ġor Ġa Ġsentence Ġthat Ġpreserves Ġthe Ġmeaning Ġof Ġthe Ġtarget Ġsentence . </s> Table ĠT AB REF 13 Ġshows Ġtwo Ġexamples Ġof Ġthe Ġaut ocom plete Ġtask Ġand Ġactual Ġuser - provided Ġkeywords . ĠEach Ġcolumn Ġcontains Ġa Ġset Ġof Ġkeywords Ġand Ġits Ġcorresponding Ġtop Ġthree Ġsuggestions Ġgenerated Ġby Ġthe Ġaut ocom plete Ġsystem Ġwith Ġbeam Ġsearch . ĠWe Ġobserve Ġthat Ġthe Ġsystem Ġis Ġlikely Ġto Ġpropose Ġgeneric Ġsentences Ġfor Ġunder - specified Ġkeywords Ġ( left Ġcolumn ) Ġand Ġalmost Ġthe Ġsame Ġsentences Ġfor Ġover - specified Ġkeywords Ġ( right Ġcolumn ). ĠFor Ġproperly Ġspecified Ġkeywords Ġ( middle Ġcolumn ), Ġthe Ġsystem Ġcompletes Ġsentences Ġaccordingly Ġby Ġadding Ġa Ġverb , Ġad verb , Ġadjective , Ġpre position , Ġcapital ization , Ġand Ġpunct uation . </s> Overall , Ġthe Ġaut ocom plete Ġsystem Ġachieved Ġhigh Ġaccuracy Ġin Ġreconstruct ing Ġthe Ġkeywords . ĠUsers Ġmarked Ġthe Ġtop Ġsuggestion Ġfrom Ġthe Ġaut ocom plete Ġsystem Ġto Ġbe Ġsem antically Ġequivalent Ġto Ġthe Ġtarget Ġ$ 80 . 6 $ % Ġof Ġthe Ġtime , Ġand Ġone Ġof Ġthe Ġtop Ġ3 Ġwas Ġsem antically Ġequivalent Ġ$ 90 . 11 $ % Ġof Ġthe Ġtime . ĠThe Ġmodel Ġalso Ġachieved Ġa Ġhigh Ġexact Ġmatch Ġaccuracy Ġof Ġ18 . 39 %. ĠFurthermore , Ġthe Ġsystem Ġwas Ġefficient , Ġas Ġusers Ġspent Ġ$ 3 . 86 $ Ġseconds Ġtyping Ġkeywords Ġcompared Ġto Ġ$ 5 . 76 $ Ġseconds Ġfor Ġfull Ġsentences Ġon Ġaverage . ĠThe Ġvariance Ġof Ġthe Ġtyping Ġtime Ġwas Ġ$ 0 . 08 $ Ġsecond Ġfor Ġkeywords Ġand Ġ$ 0 . 12 $ Ġsecond Ġfor Ġfull Ġsentences , Ġindicating Ġthat Ġchoosing Ġand Ġtyping Ġkeywords Ġfor Ġthe Ġsystem Ġdid Ġnot Ġincur Ġmuch Ġoverhead . </s> Exper iments Ġ: :: ĠA cknowled gments </s> We Ġthank Ġthe Ġreviewers Ġand ĠYun se ok ĠJ ang Ġfor Ġtheir Ġinsightful Ġcomments . ĠThis Ġwork Ġwas Ġsupported Ġby ĠNS F ĠCARE ER ĠAward ĠI IS - 155 26 35 Ġand Ġan ĠInt uit ĠResearch ĠAward . </s> Exper iments Ġ: :: ĠReprodu c ibility </s> All Ġcode , Ġdata Ġand Ġexperiments Ġare Ġavailable Ġon ĠC oda Lab Ġat Ġhttps :// bit . ly / 353 f byn .
Introduction Suppose a user wants to write a sentence “I will be 10 minutes late.” Ideally, she would type just a few keywords such as “10 minutes late” and an autocomplete system would be able to infer the intended sentence (Figure FIGREF1). Existing left-to-right autocomplete systems BIBREF0, BIBREF1 can often be inefficient, as the prefix of a sentence (e.g. “I will be”) fails to capture the core meaning of the sentence. Besides the practical goal of building a better autocomplete system, we are interested in exploring the tradeoffs inherent to such communication schemes between the efficiency of typing keywords, accuracy of reconstruction, and interpretability of keywords. One approach to learn such schemes is to collect a supervised dataset of keywords-sentence pairs as a training set, but (i) it would be expensive to collect such data from users, and (ii) a static dataset would not capture a real user's natural predilection to adapt to the system BIBREF2. Another approach is to avoid supervision and jointly learn a user-system communication scheme to directly optimize the combination of efficiency and accuracy. However, learning in this way can lead to communication schemes that are uninterpretable to humans BIBREF3, BIBREF4 (see Appendix for additional related work). In this work, we propose a simple, unsupervised approach to an autocomplete system that is efficient, accurate, and interpretable. For interpretability, we restrict keywords to be subsequences of their source sentences based on the intuition that humans can infer most of the original meaning from a few keywords. We then apply multi-objective optimization approaches to directly control and achieve desirable tradeoffs between efficiency and accuracy. We observe that naively optimizing a linear combination of efficiency and accuracy terms is unstable and leads to suboptimal schemes. Thus, we propose a new objective which optimizes for communication efficiency under an accuracy constraint. We show this new objective is more stable and efficient than the linear objective at all accuracy levels. As a proof-of-concept, we build an autocomplete system within this framework which allows a user to write sentences by specifying keywords. We empirically show that our framework produces communication schemes that are 52.16% more accurate than rule-based baselines when specifying 77.37% of sentences, and 11.73% more accurate than a naive, weighted optimization approach when specifying 53.38% of sentences. Finally, we demonstrate that humans can easily adapt to the keyword-based autocomplete system and save nearly 50% of time compared to typing a full sentence in our user study. Approach Consider a communication game in which the goal is for a user to communicate a target sequence $x= (x_1, ..., x_m)$ to a system by passing a sequence of keywords $z= (z_1, ..., z_n)$. The user generates keywords $z$ using an encoding strategy $q_{\alpha }(z\mid x)$, and the system attempts to guess the target sequence $x$ via a decoding strategy $p_{\beta }(x\mid z)$. A good communication scheme $(q_{\alpha }, p_{\beta })$ should be both efficient and accurate. Specifically, we prefer schemes that use fewer keywords (cost), and the target sentence $x$ to be reconstructed with high probability (loss) where Based on our assumption that humans have an intuitive sense of retaining important keywords, we restrict the set of schemes to be a (potentially noncontiguous) subsequence of the target sentence. Our hypothesis is that such subsequence schemes naturally ensure interpretability, as efficient human and machine communication schemes are both likely to involve keeping important content words. Approach ::: Modeling with autoencoders. To learn communication schemes without supervision, we model the cooperative communication between a user and system through an encoder-decoder framework. Concretely, we model the user's encoding strategy $q_{\alpha }(z\mid x)$ with an encoder which encodes the target sentence $x$ into the keywords $z$ by keeping a subset of the tokens. This stochastic encoder $q_{\alpha }(z\mid x)$ is defined by a model which returns the probability of each token retained in the final subsequence $z$. Then, we sample from Bernoulli distributions according to these probabilities to either keep or drop the tokens independently (see Appendix for an example). We model the autocomplete system's decoding strategy $p_{\beta }(x\mid z)$ as a probabilistic model which conditions on the keywords $z$ and returns a distribution over predictions $x$. We use a standard sequence-to-sequence model with attention and copying for the decoder, but any model architecture can be used (see Appendix for details). Approach ::: Multi-objective optimization. Our goal now is to learn encoder-decoder pairs which optimally balance the communication cost and reconstruction loss. The simplest approach to balancing efficiency and accuracy is to weight $\mathrm {cost}(x, \alpha )$ and $\mathrm {loss}(x, \alpha , \beta )$ linearly using a weight $\lambda $ as follows, where the expectation is taken over the population distribution of source sentences $x$, which is omitted to simplify notation. However, we observe that naively weighting and searching over $\lambda $ is suboptimal and highly unstable—even slight changes to the weighting results in degenerate schemes which keep all or none of its tokens. This instability motivates us to develop a new stable objective. Our main technical contribution is to draw inspiration from the multi-objective optimization literature and view the tradeoff as a sequence of constrained optimization problems, where we minimize the expected cost subject to varying expected reconstruction error constraints $\epsilon $, This greatly improves the stability of the training procedure. We empirically observe that the model initially keeps most of the tokens to meet the constraints, and slowly learns to drop uninformative words from the keywords to minimize the cost. Furthermore, $\epsilon $ in Eq (DISPLAY_FORM6) allows us to directly control the maximum reconstruction error of resulting schemes, whereas $\lambda $ in Eq (DISPLAY_FORM5) is not directly related to any of our desiderata. To optimize the constrained objective, we consider the Lagrangian of Eq (DISPLAY_FORM6), Much like the objective in Eq (DISPLAY_FORM5) we can compute unbiased gradients by replacing the expectations with their averages over random minibatches. Although gradient descent guarantees convergence on Eq (DISPLAY_FORM7) only when the objective is convex, we find that not only is the optimization stable, the resulting solution achieves better performance than the weighting approach in Eq (DISPLAY_FORM5). Approach ::: Optimization. Optimization with respect to $q_{\alpha }(z\mid x)$ is challenging as $z$ is discrete, and thus, we cannot differentiate $\alpha $ through $z$ via the chain rule. Because of this, we use the stochastic REINFORCE estimate BIBREF5 as follows: We perform joint updates on $(\alpha , \beta , \lambda )$, where $\beta $ and $\lambda $ are updated via standard gradient computations, while $\alpha $ uses an unbiased, stochastic gradient estimate where we approximate the expectation in Eq (DISPLAY_FORM9). We use a single sample from $q_{\alpha }(z\mid x)$ and moving-average of rewards as a baseline to reduce variance. Experiments We evaluate our approach by training an autocomplete system on 500K randomly sampled sentences from Yelp reviews BIBREF6 (see Appendix for details). We quantify the efficiency of a communication scheme $(q_{\alpha },p_{\beta })$ by the retention rate of tokens, which is measured as the fraction of tokens that are kept in the keywords. The accuracy of a scheme is measured as the fraction of sentences generated by greedily decoding the model that exactly matches the target sentence. Experiments ::: Effectiveness of constrained objective. We first show that the linear objective in Eq (DISPLAY_FORM5) is suboptimal compared to the constrained objective in Eq (DISPLAY_FORM6). Figure FIGREF10 compares the achievable accuracy and efficiency tradeoffs for the two objectives, which shows that the constrained objective results in more efficient schemes than the linear objective at every accuracy level (e.g. 11.73% more accurate at a 53.38% retention rate). We also observe that the linear objective is highly unstable as a function of the tradeoff parameter $\lambda $ and requires careful tuning. Even slight changes to $\lambda $ results in degenerate schemes that keep all or none of the tokens (e.g. $\lambda \le 4.2$ and $\lambda \ge 4.4$). On the other hand, the constrained objective is substantially more stable as a function of $\epsilon $ (e.g. points for $\epsilon $ are more evenly spaced than $\lambda $). Experiments ::: Efficiency-accuracy tradeoff. We quantify the efficiency-accuracy tradeoff compared to two rule-based baselines: Unif and Stopword. The Unif encoder randomly keeps tokens to generate keywords with the probability $\delta $. The Stopword encoder keeps all tokens but drops stop words (e.g. `the', `a', `or') all the time ($\delta =0$) or half of the time ($\delta =0.5$). The corresponding decoders for these encoders are optimized using gradient descent to minimize the reconstruction error (i.e. $\mathrm {loss}(x, \alpha , \beta )$). Figure FIGREF10 shows that two baselines achieve similar tradeoff curves, while the constrained model achieves a substantial 52.16% improvement in accuracy at a 77.37% retention rate compared to Unif, thereby showing the benefits of jointly training the encoder and decoder. Experiments ::: Robustness and analysis. We provide additional experimental results on the robustness of learned communication schemes as well as in-depth analysis on the correlation between the retention rates of tokens and their properties, which we defer to Appendix and for space. Experiments ::: User study. We recruited 100 crowdworkers on Amazon Mechanical Turk (AMT) and measured completion times and accuracies for typing randomly sampled sentences from the Yelp corpus. Each user was shown alternating autocomplete and writing tasks across 50 sentences (see Appendix for user interface). For the autocomplete task, we gave users a target sentence and asked them to type a set of keywords into the system. The users were shown the top three suggestions from the autocomplete system, and were asked to mark whether each of these three suggestions was semantically equivalent to the target sentence. For the writing task, we gave users a target sentence and asked them to either type the sentence verbatim or a sentence that preserves the meaning of the target sentence. Table TABREF13 shows two examples of the autocomplete task and actual user-provided keywords. Each column contains a set of keywords and its corresponding top three suggestions generated by the autocomplete system with beam search. We observe that the system is likely to propose generic sentences for under-specified keywords (left column) and almost the same sentences for over-specified keywords (right column). For properly specified keywords (middle column), the system completes sentences accordingly by adding a verb, adverb, adjective, preposition, capitalization, and punctuation. Overall, the autocomplete system achieved high accuracy in reconstructing the keywords. Users marked the top suggestion from the autocomplete system to be semantically equivalent to the target $80.6$% of the time, and one of the top 3 was semantically equivalent $90.11$% of the time. The model also achieved a high exact match accuracy of 18.39%. Furthermore, the system was efficient, as users spent $3.86$ seconds typing keywords compared to $5.76$ seconds for full sentences on average. The variance of the typing time was $0.08$ second for keywords and $0.12$ second for full sentences, indicating that choosing and typing keywords for the system did not incur much overhead. Experiments ::: Acknowledgments We thank the reviewers and Yunseok Jang for their insightful comments. This work was supported by NSF CAREER Award IIS-1552635 and an Intuit Research Award. Experiments ::: Reproducibility All code, data and experiments are available on CodaLab at https://bit.ly/353fbyn.
What are the baselines used?
3fad42be0fb2052bb404b989cc7d58b440cd23a0
What Ġare Ġthe Ġbas elines Ġused ?
Un if Ġand ĠStop word
<s> What Ġare Ġthe Ġbas elines Ġused ? </s> Introduction </s> Supp ose Ġa Ġuser Ġwants Ġto Ġwrite Ġa Ġsentence ĠâĢ ľ I Ġwill Ġbe Ġ10 Ġminutes Ġlate . âĢ Ŀ ĠIdeally , Ġshe Ġwould Ġtype Ġjust Ġa Ġfew Ġkeywords Ġsuch Ġas ĠâĢ ľ 10 Ġminutes Ġlate âĢ Ŀ Ġand Ġan Ġaut ocom plete Ġsystem Ġwould Ġbe Ġable Ġto Ġinfer Ġthe Ġintended Ġsentence Ġ( Figure ĠFIG REF 1 ). ĠEx isting Ġleft - to - right Ġaut ocom plete Ġsystems ĠB IB REF 0 , ĠB IB REF 1 Ġcan Ġoften Ġbe Ġinefficient , Ġas Ġthe Ġprefix Ġof Ġa Ġsentence Ġ( e . g . ĠâĢ ľ I Ġwill Ġbe âĢ Ŀ ) Ġfails Ġto Ġcapture Ġthe Ġcore Ġmeaning Ġof Ġthe Ġsentence . ĠBesides Ġthe Ġpractical Ġgoal Ġof Ġbuilding Ġa Ġbetter Ġaut ocom plete Ġsystem , Ġwe Ġare Ġinterested Ġin Ġexploring Ġthe Ġtrade offs Ġinherent Ġto Ġsuch Ġcommunication Ġschemes Ġbetween Ġthe Ġefficiency Ġof Ġtyping Ġkeywords , Ġaccuracy Ġof Ġreconstruction , Ġand Ġinterpret ability Ġof Ġkeywords . </s> One Ġapproach Ġto Ġlearn Ġsuch Ġschemes Ġis Ġto Ġcollect Ġa Ġsupervised Ġdataset Ġof Ġkeywords - sent ence Ġpairs Ġas Ġa Ġtraining Ġset , Ġbut Ġ( i ) Ġit Ġwould Ġbe Ġexpensive Ġto Ġcollect Ġsuch Ġdata Ġfrom Ġusers , Ġand Ġ( ii ) Ġa Ġstatic Ġdataset Ġwould Ġnot Ġcapture Ġa Ġreal Ġuser 's Ġnatural Ġpred ile ction Ġto Ġadapt Ġto Ġthe Ġsystem ĠB IB REF 2 . ĠAnother Ġapproach Ġis Ġto Ġavoid Ġsupervision Ġand Ġjointly Ġlearn Ġa Ġuser - system Ġcommunication Ġscheme Ġto Ġdirectly Ġoptimize Ġthe Ġcombination Ġof Ġefficiency Ġand Ġaccuracy . ĠHowever , Ġlearning Ġin Ġthis Ġway Ġcan Ġlead Ġto Ġcommunication Ġschemes Ġthat Ġare Ġun interpret able Ġto Ġhumans ĠB IB REF 3 , ĠB IB REF 4 Ġ( see ĠAppendix Ġfor Ġadditional Ġrelated Ġwork ). </s> In Ġthis Ġwork , Ġwe Ġpropose Ġa Ġsimple , Ġun super vised Ġapproach Ġto Ġan Ġaut ocom plete Ġsystem Ġthat Ġis Ġefficient , Ġaccurate , Ġand Ġinterpret able . ĠFor Ġinterpret ability , Ġwe Ġrestrict Ġkeywords Ġto Ġbe Ġsubsequ ences Ġof Ġtheir Ġsource Ġsentences Ġbased Ġon Ġthe Ġintuition Ġthat Ġhumans Ġcan Ġinfer Ġmost Ġof Ġthe Ġoriginal Ġmeaning Ġfrom Ġa Ġfew Ġkeywords . ĠWe Ġthen Ġapply Ġmulti - object ive Ġoptimization Ġapproaches Ġto Ġdirectly Ġcontrol Ġand Ġachieve Ġdesirable Ġtrade offs Ġbetween Ġefficiency Ġand Ġaccuracy . </s> We Ġobserve Ġthat Ġna ively Ġoptimizing Ġa Ġlinear Ġcombination Ġof Ġefficiency Ġand Ġaccuracy Ġterms Ġis Ġunstable Ġand Ġleads Ġto Ġsub opt imal Ġschemes . ĠThus , Ġwe Ġpropose Ġa Ġnew Ġobjective Ġwhich Ġoptim izes Ġfor Ġcommunication Ġefficiency Ġunder Ġan Ġaccuracy Ġconstraint . ĠWe Ġshow Ġthis Ġnew Ġobjective Ġis Ġmore Ġstable Ġand Ġefficient Ġthan Ġthe Ġlinear Ġobjective Ġat Ġall Ġaccuracy Ġlevels . </s> As Ġa Ġproof - of - concept , Ġwe Ġbuild Ġan Ġaut ocom plete Ġsystem Ġwithin Ġthis Ġframework Ġwhich Ġallows Ġa Ġuser Ġto Ġwrite Ġsentences Ġby Ġspecifying Ġkeywords . ĠWe Ġempir ically Ġshow Ġthat Ġour Ġframework Ġproduces Ġcommunication Ġschemes Ġthat Ġare Ġ52 . 16 % Ġmore Ġaccurate Ġthan Ġrule - based Ġbas elines Ġwhen Ġspecifying Ġ77 . 37 % Ġof Ġsentences , Ġand Ġ11 . 73 % Ġmore Ġaccurate Ġthan Ġa Ġnaive , Ġweighted Ġoptimization Ġapproach Ġwhen Ġspecifying Ġ53 . 38 % Ġof Ġsentences . ĠFinally , Ġwe Ġdemonstrate Ġthat Ġhumans Ġcan Ġeasily Ġadapt Ġto Ġthe Ġkeyword - based Ġaut ocom plete Ġsystem Ġand Ġsave Ġnearly Ġ50 % Ġof Ġtime Ġcompared Ġto Ġtyping Ġa Ġfull Ġsentence Ġin Ġour Ġuser Ġstudy . </s> App roach </s> Consider Ġa Ġcommunication Ġgame Ġin Ġwhich Ġthe Ġgoal Ġis Ġfor Ġa Ġuser Ġto Ġcommunicate Ġa Ġtarget Ġsequence Ġ$ x = Ġ( x _ 1 , Ġ... , Ġx _ m ) $ Ġto Ġa Ġsystem Ġby Ġpassing Ġa Ġsequence Ġof Ġkeywords Ġ$ z = Ġ( z _ 1 , Ġ... , Ġz _ n ) $. ĠThe Ġuser Ġgenerates Ġkeywords Ġ$ z $ Ġusing Ġan Ġencoding Ġstrategy Ġ$ q _{ \ alpha Ġ} ( z \ mid Ġx ) $, Ġand Ġthe Ġsystem Ġattempts Ġto Ġguess Ġthe Ġtarget Ġsequence Ġ$ x $ Ġvia Ġa Ġdecoding Ġstrategy Ġ$ p _{ \ beta Ġ} ( x \ mid Ġz ) $. </s> A Ġgood Ġcommunication Ġscheme Ġ$( q _{ \ alpha Ġ}, Ġp _{ \ beta Ġ}) $ Ġshould Ġbe Ġboth Ġefficient Ġand Ġaccurate . ĠSpecifically , Ġwe Ġprefer Ġschemes Ġthat Ġuse Ġfewer Ġkeywords Ġ( cost ), Ġand Ġthe Ġtarget Ġsentence Ġ$ x $ Ġto Ġbe Ġreconstructed Ġwith Ġhigh Ġprobability Ġ( loss ) Ġwhere </s> Based Ġon Ġour Ġassumption Ġthat Ġhumans Ġhave Ġan Ġintuitive Ġsense Ġof Ġretaining Ġimportant Ġkeywords , Ġwe Ġrestrict Ġthe Ġset Ġof Ġschemes Ġto Ġbe Ġa Ġ( pot entially Ġnon cont iguous ) Ġsubsequ ence Ġof Ġthe Ġtarget Ġsentence . ĠOur Ġhypothesis Ġis Ġthat Ġsuch Ġsubsequ ence Ġschemes Ġnaturally Ġensure Ġinterpret ability , Ġas Ġefficient Ġhuman Ġand Ġmachine Ġcommunication Ġschemes Ġare Ġboth Ġlikely Ġto Ġinvolve Ġkeeping Ġimportant Ġcontent Ġwords . </s> App roach Ġ: :: ĠModel ing Ġwith Ġaut oen cod ers . </s> To Ġlearn Ġcommunication Ġschemes Ġwithout Ġsupervision , Ġwe Ġmodel Ġthe Ġcooperative Ġcommunication Ġbetween Ġa Ġuser Ġand Ġsystem Ġthrough Ġan Ġenc oder - dec oder Ġframework . ĠCon crete ly , Ġwe Ġmodel Ġthe Ġuser 's Ġencoding Ġstrategy Ġ$ q _{ \ alpha Ġ} ( z \ mid Ġx ) $ Ġwith Ġan Ġenc oder Ġwhich Ġenc odes Ġthe Ġtarget Ġsentence Ġ$ x $ Ġinto Ġthe Ġkeywords Ġ$ z $ Ġby Ġkeeping Ġa Ġsubset Ġof Ġthe Ġtokens . ĠThis Ġsto ch astic Ġenc oder Ġ$ q _{ \ alpha Ġ} ( z \ mid Ġx ) $ Ġis Ġdefined Ġby Ġa Ġmodel Ġwhich Ġreturns Ġthe Ġprobability Ġof Ġeach Ġtoken Ġretained Ġin Ġthe Ġfinal Ġsubsequ ence Ġ$ z $. ĠThen , Ġwe Ġsample Ġfrom ĠBern ou lli Ġdistributions Ġaccording Ġto Ġthese Ġprobabilities Ġto Ġeither Ġkeep Ġor Ġdrop Ġthe Ġtokens Ġindependently Ġ( see ĠAppendix Ġfor Ġan Ġexample ). </s> We Ġmodel Ġthe Ġaut ocom plete Ġsystem 's Ġdecoding Ġstrategy Ġ$ p _{ \ beta Ġ} ( x \ mid Ġz ) $ Ġas Ġa Ġprob abil istic Ġmodel Ġwhich Ġconditions Ġon Ġthe Ġkeywords Ġ$ z $ Ġand Ġreturns Ġa Ġdistribution Ġover Ġpredictions Ġ$ x $. ĠWe Ġuse Ġa Ġstandard Ġsequence - to - sequence Ġmodel Ġwith Ġattention Ġand Ġcopying Ġfor Ġthe Ġdec oder , Ġbut Ġany Ġmodel Ġarchitecture Ġcan Ġbe Ġused Ġ( see ĠAppendix Ġfor Ġdetails ). </s> App roach Ġ: :: ĠMulti - object ive Ġoptimization . </s> Our Ġgoal Ġnow Ġis Ġto Ġlearn Ġenc oder - dec oder Ġpairs Ġwhich Ġoptim ally Ġbalance Ġthe Ġcommunication Ġcost Ġand Ġreconstruction Ġloss . ĠThe Ġsimplest Ġapproach Ġto Ġbalancing Ġefficiency Ġand Ġaccuracy Ġis Ġto Ġweight Ġ$\ math rm Ġ{ cost } ( x , Ġ\ alpha Ġ) $ Ġand Ġ$\ math rm Ġ{ loss } ( x , Ġ\ alpha Ġ, Ġ\ beta Ġ) $ Ġlin early Ġusing Ġa Ġweight Ġ$\ lambda Ġ$ Ġas Ġfollows , </s> where Ġthe Ġexpectation Ġis Ġtaken Ġover Ġthe Ġpopulation Ġdistribution Ġof Ġsource Ġsentences Ġ$ x $, Ġwhich Ġis Ġomitted Ġto Ġsimplify Ġnotation . ĠHowever , Ġwe Ġobserve Ġthat Ġna ively Ġweight ing Ġand Ġsearching Ġover Ġ$\ lambda Ġ$ Ġis Ġsub opt imal Ġand Ġhighly Ġunstable âĢĶ even Ġslight Ġchanges Ġto Ġthe Ġweight ing Ġresults Ġin Ġdegener ate Ġschemes Ġwhich Ġkeep Ġall Ġor Ġnone Ġof Ġits Ġtokens . ĠThis Ġinstability Ġmotiv ates Ġus Ġto Ġdevelop Ġa Ġnew Ġstable Ġobjective . </s> Our Ġmain Ġtechnical Ġcontribution Ġis Ġto Ġdraw Ġinspiration Ġfrom Ġthe Ġmulti - object ive Ġoptimization Ġliterature Ġand Ġview Ġthe Ġtrade off Ġas Ġa Ġsequence Ġof Ġconstrained Ġoptimization Ġproblems , Ġwhere Ġwe Ġminimize Ġthe Ġexpected Ġcost Ġsubject Ġto Ġvarying Ġexpected Ġreconstruction Ġerror Ġconstraints Ġ$\ ep sil on Ġ$ , </s> This Ġgreatly Ġimproves Ġthe Ġstability Ġof Ġthe Ġtraining Ġprocedure . ĠWe Ġempir ically Ġobserve Ġthat Ġthe Ġmodel Ġinitially Ġkeeps Ġmost Ġof Ġthe Ġtokens Ġto Ġmeet Ġthe Ġconstraints , Ġand Ġslowly Ġlearns Ġto Ġdrop Ġunin form ative Ġwords Ġfrom Ġthe Ġkeywords Ġto Ġminimize Ġthe Ġcost . ĠFurthermore , Ġ$\ ep sil on Ġ$ Ġin ĠE q Ġ( DIS PLAY _ FORM 6 ) Ġallows Ġus Ġto Ġdirectly Ġcontrol Ġthe Ġmaximum Ġreconstruction Ġerror Ġof Ġresulting Ġschemes , Ġwhereas Ġ$\ lambda Ġ$ Ġin ĠE q Ġ( DIS PLAY _ FORM 5 ) Ġis Ġnot Ġdirectly Ġrelated Ġto Ġany Ġof Ġour Ġdes ider ata . </s> To Ġoptimize Ġthe Ġconstrained Ġobjective , Ġwe Ġconsider Ġthe ĠLag rang ian Ġof ĠE q Ġ( DIS PLAY _ FORM 6 ), </s> Much Ġlike Ġthe Ġobjective Ġin ĠE q Ġ( DIS PLAY _ FORM 5 ) Ġwe Ġcan Ġcompute Ġunbiased Ġgrad ients Ġby Ġreplacing Ġthe Ġexpectations Ġwith Ġtheir Ġaverages Ġover Ġrandom Ġmin ib atches . ĠAlthough Ġgradient Ġdescent Ġguarantees Ġconvergence Ġon ĠE q Ġ( DIS PLAY _ FORM 7 ) Ġonly Ġwhen Ġthe Ġobjective Ġis Ġconve x , Ġwe Ġfind Ġthat Ġnot Ġonly Ġis Ġthe Ġoptimization Ġstable , Ġthe Ġresulting Ġsolution Ġachieves Ġbetter Ġperformance Ġthan Ġthe Ġweight ing Ġapproach Ġin ĠE q Ġ( DIS PLAY _ FORM 5 ). </s> App roach Ġ: :: ĠOptim ization . </s> Opt im ization Ġwith Ġrespect Ġto Ġ$ q _{ \ alpha Ġ} ( z \ mid Ġx ) $ Ġis Ġchallenging Ġas Ġ$ z $ Ġis Ġdiscrete , Ġand Ġthus , Ġwe Ġcannot Ġdifferentiate Ġ$\ alpha Ġ$ Ġthrough Ġ$ z $ Ġvia Ġthe Ġchain Ġrule . ĠBecause Ġof Ġthis , Ġwe Ġuse Ġthe Ġsto ch astic ĠRE IN FOR CE Ġestimate ĠB IB REF 5 Ġas Ġfollows : </s> We Ġperform Ġjoint Ġupdates Ġon Ġ$( \ alpha Ġ, Ġ\ beta Ġ, Ġ\ lambda Ġ) $, Ġwhere Ġ$\ beta Ġ$ Ġand Ġ$\ lambda Ġ$ Ġare Ġupdated Ġvia Ġstandard Ġgradient Ġcomput ations , Ġwhile Ġ$\ alpha Ġ$ Ġuses Ġan Ġunbiased , Ġsto ch astic Ġgradient Ġestimate Ġwhere Ġwe Ġapproximate Ġthe Ġexpectation Ġin ĠE q Ġ( DIS PLAY _ FORM 9 ). ĠWe Ġuse Ġa Ġsingle Ġsample Ġfrom Ġ$ q _{ \ alpha Ġ} ( z \ mid Ġx ) $ Ġand Ġmoving - average Ġof Ġrewards Ġas Ġa Ġbaseline Ġto Ġreduce Ġvariance . </s> Exper iments </s> We Ġevaluate Ġour Ġapproach Ġby Ġtraining Ġan Ġaut ocom plete Ġsystem Ġon Ġ500 K Ġrandomly Ġsampled Ġsentences Ġfrom ĠYelp Ġreviews ĠB IB REF 6 Ġ( see ĠAppendix Ġfor Ġdetails ). ĠWe Ġquantify Ġthe Ġefficiency Ġof Ġa Ġcommunication Ġscheme Ġ$( q _{ \ alpha Ġ}, p _{ \ beta Ġ}) $ Ġby Ġthe Ġretention Ġrate Ġof Ġtokens , Ġwhich Ġis Ġmeasured Ġas Ġthe Ġfraction Ġof Ġtokens Ġthat Ġare Ġkept Ġin Ġthe Ġkeywords . ĠThe Ġaccuracy Ġof Ġa Ġscheme Ġis Ġmeasured Ġas Ġthe Ġfraction Ġof Ġsentences Ġgenerated Ġby Ġgreed ily Ġdecoding Ġthe Ġmodel Ġthat Ġexactly Ġmatches Ġthe Ġtarget Ġsentence . </s> Exper iments Ġ: :: ĠEffect iveness Ġof Ġconstrained Ġobjective . </s> We Ġfirst Ġshow Ġthat Ġthe Ġlinear Ġobjective Ġin ĠE q Ġ( DIS PLAY _ FORM 5 ) Ġis Ġsub opt imal Ġcompared Ġto Ġthe Ġconstrained Ġobjective Ġin ĠE q Ġ( DIS PLAY _ FORM 6 ). ĠFigure ĠFIG REF 10 Ġcompares Ġthe Ġachievable Ġaccuracy Ġand Ġefficiency Ġtrade offs Ġfor Ġthe Ġtwo Ġobjectives , Ġwhich Ġshows Ġthat Ġthe Ġconstrained Ġobjective Ġresults Ġin Ġmore Ġefficient Ġschemes Ġthan Ġthe Ġlinear Ġobjective Ġat Ġevery Ġaccuracy Ġlevel Ġ( e . g . Ġ11 . 73 % Ġmore Ġaccurate Ġat Ġa Ġ53 . 38 % Ġretention Ġrate ). </s> We Ġalso Ġobserve Ġthat Ġthe Ġlinear Ġobjective Ġis Ġhighly Ġunstable Ġas Ġa Ġfunction Ġof Ġthe Ġtrade off Ġparameter Ġ$\ lambda Ġ$ Ġand Ġrequires Ġcareful Ġtuning . ĠEven Ġslight Ġchanges Ġto Ġ$\ lambda Ġ$ Ġresults Ġin Ġdegener ate Ġschemes Ġthat Ġkeep Ġall Ġor Ġnone Ġof Ġthe Ġtokens Ġ( e . g . Ġ$\ lambda Ġ\ le Ġ4 . 2 $ Ġand Ġ$\ lambda Ġ\ ge Ġ4 . 4 $ ). ĠOn Ġthe Ġother Ġhand , Ġthe Ġconstrained Ġobjective Ġis Ġsubstantially Ġmore Ġstable Ġas Ġa Ġfunction Ġof Ġ$\ ep sil on Ġ$ Ġ( e . g . Ġpoints Ġfor Ġ$\ ep sil on Ġ$ Ġare Ġmore Ġevenly Ġspaced Ġthan Ġ$\ lambda Ġ$ ). </s> Exper iments Ġ: :: ĠEfficiency - acc uracy Ġtrade off . </s> We Ġquantify Ġthe Ġefficiency - acc uracy Ġtrade off Ġcompared Ġto Ġtwo Ġrule - based Ġbas elines : ĠUn if Ġand ĠStop word . ĠThe ĠUn if Ġenc oder Ġrandomly Ġkeeps Ġtokens Ġto Ġgenerate Ġkeywords Ġwith Ġthe Ġprobability Ġ$\ d elta Ġ$ . ĠThe ĠStop word Ġenc oder Ġkeeps Ġall Ġtokens Ġbut Ġdrops Ġstop Ġwords Ġ( e . g . Ġ` the ', Ġ` a ', Ġ` or ') Ġall Ġthe Ġtime Ġ($ \ d elta Ġ= 0 $ ) Ġor Ġhalf Ġof Ġthe Ġtime Ġ($ \ d elta Ġ= 0 . 5 $ ). ĠThe Ġcorresponding Ġdec od ers Ġfor Ġthese Ġenc od ers Ġare Ġoptimized Ġusing Ġgradient Ġdescent Ġto Ġminimize Ġthe Ġreconstruction Ġerror Ġ( i . e . Ġ$\ math rm Ġ{ loss } ( x , Ġ\ alpha Ġ, Ġ\ beta Ġ) $ ). </s> Figure ĠFIG REF 10 Ġshows Ġthat Ġtwo Ġbas elines Ġachieve Ġsimilar Ġtrade off Ġcurves , Ġwhile Ġthe Ġconstrained Ġmodel Ġachieves Ġa Ġsubstantial Ġ52 . 16 % Ġimprovement Ġin Ġaccuracy Ġat Ġa Ġ77 . 37 % Ġretention Ġrate Ġcompared Ġto ĠUn if , Ġthereby Ġshowing Ġthe Ġbenefits Ġof Ġjointly Ġtraining Ġthe Ġenc oder Ġand Ġdec oder . </s> Exper iments Ġ: :: ĠRob ust ness Ġand Ġanalysis . </s> We Ġprovide Ġadditional Ġexperimental Ġresults Ġon Ġthe Ġrobust ness Ġof Ġlearned Ġcommunication Ġschemes Ġas Ġwell Ġas Ġin - depth Ġanalysis Ġon Ġthe Ġcorrelation Ġbetween Ġthe Ġretention Ġrates Ġof Ġtokens Ġand Ġtheir Ġproperties , Ġwhich Ġwe Ġdefer Ġto ĠAppendix Ġand Ġfor Ġspace . </s> Exper iments Ġ: :: ĠUser Ġstudy . </s> We Ġrecruited Ġ100 Ġcrowd workers Ġon ĠAmazon ĠMechanical ĠTurk Ġ( AM T ) Ġand Ġmeasured Ġcompletion Ġtimes Ġand Ġaccur acies Ġfor Ġtyping Ġrandomly Ġsampled Ġsentences Ġfrom Ġthe ĠYelp Ġcorpus . ĠEach Ġuser Ġwas Ġshown Ġalternating Ġaut ocom plete Ġand Ġwriting Ġtasks Ġacross Ġ50 Ġsentences Ġ( see ĠAppendix Ġfor Ġuser Ġinterface ). ĠFor Ġthe Ġaut ocom plete Ġtask , Ġwe Ġgave Ġusers Ġa Ġtarget Ġsentence Ġand Ġasked Ġthem Ġto Ġtype Ġa Ġset Ġof Ġkeywords Ġinto Ġthe Ġsystem . ĠThe Ġusers Ġwere Ġshown Ġthe Ġtop Ġthree Ġsuggestions Ġfrom Ġthe Ġaut ocom plete Ġsystem , Ġand Ġwere Ġasked Ġto Ġmark Ġwhether Ġeach Ġof Ġthese Ġthree Ġsuggestions Ġwas Ġsem antically Ġequivalent Ġto Ġthe Ġtarget Ġsentence . ĠFor Ġthe Ġwriting Ġtask , Ġwe Ġgave Ġusers Ġa Ġtarget Ġsentence Ġand Ġasked Ġthem Ġto Ġeither Ġtype Ġthe Ġsentence Ġver bat im Ġor Ġa Ġsentence Ġthat Ġpreserves Ġthe Ġmeaning Ġof Ġthe Ġtarget Ġsentence . </s> Table ĠT AB REF 13 Ġshows Ġtwo Ġexamples Ġof Ġthe Ġaut ocom plete Ġtask Ġand Ġactual Ġuser - provided Ġkeywords . ĠEach Ġcolumn Ġcontains Ġa Ġset Ġof Ġkeywords Ġand Ġits Ġcorresponding Ġtop Ġthree Ġsuggestions Ġgenerated Ġby Ġthe Ġaut ocom plete Ġsystem Ġwith Ġbeam Ġsearch . ĠWe Ġobserve Ġthat Ġthe Ġsystem Ġis Ġlikely Ġto Ġpropose Ġgeneric Ġsentences Ġfor Ġunder - specified Ġkeywords Ġ( left Ġcolumn ) Ġand Ġalmost Ġthe Ġsame Ġsentences Ġfor Ġover - specified Ġkeywords Ġ( right Ġcolumn ). ĠFor Ġproperly Ġspecified Ġkeywords Ġ( middle Ġcolumn ), Ġthe Ġsystem Ġcompletes Ġsentences Ġaccordingly Ġby Ġadding Ġa Ġverb , Ġad verb , Ġadjective , Ġpre position , Ġcapital ization , Ġand Ġpunct uation . </s> Overall , Ġthe Ġaut ocom plete Ġsystem Ġachieved Ġhigh Ġaccuracy Ġin Ġreconstruct ing Ġthe Ġkeywords . ĠUsers Ġmarked Ġthe Ġtop Ġsuggestion Ġfrom Ġthe Ġaut ocom plete Ġsystem Ġto Ġbe Ġsem antically Ġequivalent Ġto Ġthe Ġtarget Ġ$ 80 . 6 $ % Ġof Ġthe Ġtime , Ġand Ġone Ġof Ġthe Ġtop Ġ3 Ġwas Ġsem antically Ġequivalent Ġ$ 90 . 11 $ % Ġof Ġthe Ġtime . ĠThe Ġmodel Ġalso Ġachieved Ġa Ġhigh Ġexact Ġmatch Ġaccuracy Ġof Ġ18 . 39 %. ĠFurthermore , Ġthe Ġsystem Ġwas Ġefficient , Ġas Ġusers Ġspent Ġ$ 3 . 86 $ Ġseconds Ġtyping Ġkeywords Ġcompared Ġto Ġ$ 5 . 76 $ Ġseconds Ġfor Ġfull Ġsentences Ġon Ġaverage . ĠThe Ġvariance Ġof Ġthe Ġtyping Ġtime Ġwas Ġ$ 0 . 08 $ Ġsecond Ġfor Ġkeywords Ġand Ġ$ 0 . 12 $ Ġsecond Ġfor Ġfull Ġsentences , Ġindicating Ġthat Ġchoosing Ġand Ġtyping Ġkeywords Ġfor Ġthe Ġsystem Ġdid Ġnot Ġincur Ġmuch Ġoverhead . </s> Exper iments Ġ: :: ĠA cknowled gments </s> We Ġthank Ġthe Ġreviewers Ġand ĠYun se ok ĠJ ang Ġfor Ġtheir Ġinsightful Ġcomments . ĠThis Ġwork Ġwas Ġsupported Ġby ĠNS F ĠCARE ER ĠAward ĠI IS - 155 26 35 Ġand Ġan ĠInt uit ĠResearch ĠAward . </s> Exper iments Ġ: :: ĠReprodu c ibility </s> All Ġcode , Ġdata Ġand Ġexperiments Ġare Ġavailable Ġon ĠC oda Lab Ġat Ġhttps :// bit . ly / 353 f byn .