In pass-through, the recognizer passes on the possibly misspelled word, backoff to neutral word backs off to a word with similar distribution across classes and backoff to background model backs off to a more generic word recognition model trained with larger and less specialized corpus.
ABSTRACTIVE
Pass-through passes the possibly misspelled word as is, backoff to neutral word backs off to a word with similar distribution across classes and backoff to background model backs off to a more generic word recognition model trained with larger and less specialized corpus.
ABSTRACTIVE
Backoff to "a" when an UNK-predicted word is encountered, backoff to a more generic word recognition model when the model predicts UNK
ABSTRACTIVE
While BIBREF7 demonstrate strong word recognition performance, a drawback of their evaluation setup is that they only attack and evaluate on the subset of words that are a part of their training vocabulary. In such a setting, the word recognition performance is unreasonably dependent on the chosen vocabulary size. In principle, one can design models to predict (correctly) only a few chosen words, and ignore the remaining majority and still reach 100% accuracy. For the adversarial setting, rare and unseen words in the wild are particularly critical, as they provide opportunities for the attackers. A reliable word-recognizer should handle these cases gracefully. Below, we explore different ways to back off when the ScRNN predicts UNK (a frequent outcome for rare and unseen words): Pass-through: word-recognizer passes on the (possibly misspelled) word as is. Backoff to neutral word: Alternatively, noting that passing $\colorbox {gray!20}{\texttt {UNK}}$ -predicted words through unchanged exposes the downstream model to potentially corrupted text, we consider backing off to a neutral word like `a', which has a similar distribution across classes. Backoff to background model: We also consider falling back upon a more generic word recognition model trained upon a larger, less-specialized corpus whenever the foreground word recognition model predicts UNK. Figure 1 depicts this scenario pictorially.
While BIBREF7 demonstrate strong word recognition performance, a drawback of their evaluation setup is that they only attack and evaluate on the subset of words that are a part of their training vocabulary. In such a setting, the word recognition performance is unreasonably dependent on the chosen vocabulary size. In principle, one can design models to predict (correctly) only a few chosen words, and ignore the remaining majority and still reach 100% accuracy. For the adversarial setting, rare and unseen words in the wild are particularly critical, as they provide opportunities for the attackers. A reliable word-recognizer should handle these cases gracefully. Below, we explore different ways to back off when the ScRNN predicts UNK (a frequent outcome for rare and unseen words): Pass-through: word-recognizer passes on the (possibly misspelled) word as is. Backoff to neutral word: Alternatively, noting that passing $\colorbox {gray!20}{\texttt {UNK}}$ -predicted words through unchanged exposes the downstream model to potentially corrupted text, we consider backing off to a neutral word like `a', which has a similar distribution across classes. Backoff to background model: We also consider falling back upon a more generic word recognition model trained upon a larger, less-specialized corpus whenever the foreground word recognition model predicts UNK. Figure 1 depicts this scenario pictorially.
While BIBREF7 demonstrate strong word recognition performance, a drawback of their evaluation setup is that they only attack and evaluate on the subset of words that are a part of their training vocabulary. In such a setting, the word recognition performance is unreasonably dependent on the chosen vocabulary size. In principle, one can design models to predict (correctly) only a few chosen words, and ignore the remaining majority and still reach 100% accuracy. For the adversarial setting, rare and unseen words in the wild are particularly critical, as they provide opportunities for the attackers. A reliable word-recognizer should handle these cases gracefully. Below, we explore different ways to back off when the ScRNN predicts UNK (a frequent outcome for rare and unseen words): Pass-through: word-recognizer passes on the (possibly misspelled) word as is. Backoff to neutral word: Alternatively, noting that passing $\colorbox {gray!20}{\texttt {UNK}}$ -predicted words through unchanged exposes the downstream model to potentially corrupted text, we consider backing off to a neutral word like `a', which has a similar distribution across classes. Backoff to background model: We also consider falling back upon a more generic word recognition model trained upon a larger, less-specialized corpus whenever the foreground word recognition model predicts UNK. Figure 1 depicts this scenario pictorially.
1905.11268
Introduction </s> Despite Ġthe Ġrapid Ġprogress Ġof Ġdeep Ġlearning Ġtechniques Ġon Ġdiverse Ġsupervised Ġlearning Ġtasks , Ġthese Ġmodels Ġremain Ġbrittle Ġto Ġsubtle Ġshifts Ġin Ġthe Ġdata Ġdistribution . ĠEven Ġwhen Ġthe Ġpermissible Ġchanges Ġare Ġconfined Ġto Ġbarely - per cept ible Ġpert urb ations , Ġtraining Ġrobust Ġmodels Ġremains Ġan Ġopen Ġchallenge . ĠFollowing Ġthe Ġdiscovery Ġthat Ġimper cept ible Ġattacks Ġcould Ġcause Ġimage Ġrecognition Ġmodels Ġto Ġmis class ify Ġexamples ĠB IB REF 0 Ġ, Ġa Ġver itable Ġsub - field Ġhas Ġemerged Ġin Ġwhich Ġauthors Ġiter atively Ġpropose Ġattacks Ġand Ġcounter measures . </s> For Ġall Ġthe Ġinterest Ġin Ġadvers arial Ġcomputer Ġvision , Ġthese Ġattacks Ġare Ġrarely Ġencountered Ġoutside Ġof Ġacademic Ġresearch . ĠHowever , Ġadvers arial Ġmiss pell ings Ġconstitute Ġa Ġlongstanding Ġreal - world Ġproblem . ĠSp ammers Ġcontinually Ġbombard Ġemail Ġservers , Ġsubtly Ġmiss pelling Ġwords Ġin Ġefforts Ġto Ġevade Ġspam Ġdetection Ġwhile Ġpreserving Ġthe Ġemails ' Ġintended Ġmeaning ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ. ĠAs Ġanother Ġexample , Ġprogram matic Ġcensorship Ġon Ġthe ĠInternet Ġhas Ġspurred Ġcommunities Ġto Ġadopt Ġsimilar Ġmethods Ġto Ġcommunicate Ġsur re pt it iously ĠB IB REF 3 Ġ. </s> In Ġthis Ġpaper , Ġwe Ġfocus Ġon Ġadvers ar ially - ch osen Ġspelling Ġmistakes Ġin Ġthe Ġcontext Ġof Ġtext Ġclassification , Ġaddressing Ġthe Ġfollowing Ġattack Ġtypes : Ġdropping , Ġadding , Ġand Ġswapping Ġinternal Ġcharacters Ġwithin Ġwords . ĠThese Ġpert urb ations Ġare Ġinspired Ġby Ġpsych ol ingu istic Ġstudies ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġwhich Ġdemonstrated Ġthat Ġhumans Ġcan Ġcomprehend Ġtext Ġaltered Ġby Ġj umbling Ġinternal Ġcharacters , Ġprovided Ġthat Ġthe Ġfirst Ġand Ġlast Ġcharacters Ġof Ġeach Ġword Ġremain Ġun pert urbed . </s> First , Ġin Ġexperiments Ġaddressing Ġboth ĠBi L ST M Ġand Ġfine - tun ed ĠB ERT Ġmodels , Ġcomprising Ġfour Ġdifferent Ġinput Ġformats : Ġword - only , Ġchar - only , Ġword + char , Ġand Ġword - piece ĠB IB REF 6 Ġ, Ġwe Ġdemonstrate Ġthat Ġan Ġadversary Ġcan Ġdegrade Ġa Ġclass ifier 's Ġperformance Ġto Ġthat Ġachieved Ġby Ġrandom Ġguessing . ĠThis Ġrequires Ġaltering Ġjust Ġtwo Ġcharacters Ġper Ġsentence . ĠSuch Ġmodifications Ġmight Ġflip Ġwords Ġeither Ġto Ġa Ġdifferent Ġword Ġin Ġthe Ġvocabulary Ġor , Ġmore Ġoften , Ġto Ġthe Ġout - of - voc abulary Ġtoken ĠUN K . ĠConsequently , Ġadvers arial Ġedits Ġcan Ġdegrade Ġa Ġword - level Ġmodel Ġby Ġtransforming Ġthe Ġinformative Ġwords Ġto ĠUN K . ĠInt uitive ly , Ġone Ġmight Ġsuspect Ġthat Ġword - piece Ġand Ġcharacter - level Ġmodels Ġwould Ġbe Ġless Ġsusceptible Ġto Ġspelling Ġattacks Ġas Ġthey Ġcan Ġmake Ġuse Ġof Ġthe Ġresidual Ġword Ġcontext . ĠHowever , Ġour Ġexperiments Ġdemonstrate Ġthat Ġcharacter Ġand Ġword - piece Ġmodels Ġare Ġin Ġfact Ġmore Ġvulnerable . ĠWe Ġshow Ġthat Ġthis Ġis Ġdue Ġto Ġthe Ġadversary 's Ġeffective Ġcapacity Ġfor Ġfiner Ġgr ained Ġmanip ulations Ġon Ġthese Ġmodels . ĠWhile Ġagainst Ġa Ġword - level Ġmodel , Ġthe Ġadversary Ġis Ġmostly Ġlimited Ġto ĠUN K - ing Ġwords , Ġagainst Ġa Ġword - piece Ġor Ġcharacter - level Ġmodel , Ġeach Ġcharacter - level Ġadd , Ġdrop , Ġor Ġswap Ġproduces Ġa Ġdistinct Ġinput , Ġproviding Ġthe Ġadversary Ġwith Ġa Ġgreater Ġset Ġof Ġoptions . </s> Second , Ġwe Ġevaluate Ġfirst - line Ġtechniques Ġincluding Ġdata Ġaug mentation Ġand Ġadvers arial Ġtraining , Ġdemonstrating Ġthat Ġthey Ġoffer Ġonly Ġmarginal Ġbenefits Ġhere , Ġe . g ., Ġa ĠB ERT Ġmodel Ġachieving Ġ$ 90 . 3 $ Ġaccuracy Ġon Ġa Ġsentiment Ġclassification Ġtask , Ġis Ġdegraded Ġto Ġ$ 64 . 1 $ Ġby Ġan Ġadvers ar ially - ch osen Ġ1 - character Ġswap Ġin Ġthe Ġsentence , Ġwhich Ġcan Ġonly Ġbe Ġrestored Ġto Ġ$ 69 . 2 $ Ġby Ġadvers arial Ġtraining . </s> Third Ġ( our Ġprimary Ġcontribution ), Ġwe Ġpropose Ġa Ġtask - agn ostic Ġdefense , Ġattaching Ġa Ġword Ġrecognition Ġmodel Ġthat Ġpredicts Ġeach Ġword Ġin Ġa Ġsentence Ġgiven Ġa Ġfull Ġsequence Ġof Ġ( possibly Ġmiss pelled ) Ġinputs . ĠThe Ġword Ġrecognition Ġmodel 's Ġoutputs Ġform Ġthe Ġinput Ġto Ġa Ġdownstream Ġclassification Ġmodel . ĠOur Ġword Ġrecognition Ġmodels Ġbuild Ġupon Ġthe ĠR NN - based Ġsemi - character Ġword Ġrecognition Ġmodel Ġdue Ġto ĠB IB REF 7 Ġ. ĠWhile Ġour Ġword Ġrecogn izers Ġare Ġtrained Ġon Ġdomain - specific Ġtext Ġfrom Ġthe Ġtask Ġat Ġhand , Ġthey Ġoften Ġpredict ĠUN K Ġat Ġtest Ġtime , Ġowing Ġto Ġthe Ġsmall Ġdomain - specific Ġvocabulary . ĠTo Ġhandle Ġunob served Ġand Ġrare Ġwords , Ġwe Ġpropose Ġseveral Ġback off Ġstrategies Ġincluding Ġfalling Ġback Ġon Ġa Ġgeneric Ġword Ġrecogn izer Ġtrained Ġon Ġa Ġlarger Ġcorpus . ĠInc orpor ating Ġour Ġdefenses , ĠB ERT Ġmodels Ġsubject Ġto Ġ1 - character Ġattacks Ġare Ġrestored Ġto Ġ$ 88 . 3 $ Ġ, Ġ$ 81 . 1 $ Ġ, Ġ$ 78 . 0 $ Ġaccuracy Ġfor Ġswap , Ġdrop , Ġadd Ġattacks Ġrespectively , Ġas Ġcompared Ġto Ġ$ 69 . 2 $ Ġ, Ġ$ 63 . 6 $ Ġ, Ġand Ġ$ 50 . 0 $ Ġfor Ġadvers arial Ġtraining </s> Fourth , Ġwe Ġoffer Ġa Ġdetailed Ġqualitative Ġanalysis , Ġdemonstrating Ġthat Ġa Ġlow Ġword Ġerror Ġrate Ġalone Ġis Ġinsufficient Ġfor Ġa Ġword Ġrecogn izer Ġto Ġconfer Ġrobust ness Ġon Ġthe Ġdownstream Ġtask . ĠAdditionally , Ġwe Ġfind Ġthat Ġit Ġis Ġimportant Ġthat Ġthe Ġrecognition Ġmodel Ġsupply Ġfew Ġdegrees Ġof Ġfreedom Ġto Ġan Ġattacker . ĠWe Ġprovide Ġa Ġmetric Ġto Ġquantify Ġthis Ġnotion Ġof Ġsensitivity Ġin Ġword Ġrecognition Ġmodels Ġand Ġstudy Ġits Ġrelation Ġto Ġrobust ness Ġempir ically . ĠModels Ġwith Ġlow Ġsensitivity Ġand Ġword Ġerror Ġrate Ġare Ġmost Ġrobust . </s> Related ĠWork </s> Several Ġpapers Ġaddress Ġadvers arial Ġattacks Ġon ĠN LP Ġsystems . ĠChanges Ġto Ġtext , Ġwhether Ġword - Ġor Ġcharacter - level , Ġare Ġall Ġpercept ible , Ġraising Ġsome Ġquestions Ġabout Ġwhat Ġshould Ġrightly Ġbe Ġconsidered Ġan Ġadvers arial Ġexample ĠB IB REF 8 Ġ, ĠB IB REF 9 Ġ. ĠB IB REF 10 Ġaddress Ġthe Ġreading Ġcomprehension Ġtask , Ġshowing Ġthat Ġby Ġapp ending Ġdistract or Ġsentences Ġto Ġthe Ġend Ġof Ġstories Ġfrom Ġthe ĠS Qu AD Ġdataset ĠB IB REF 11 Ġ, Ġthey Ġcould Ġcause Ġmodels Ġto Ġoutput Ġincorrect Ġanswers . ĠInspired Ġby Ġthis Ġwork , ĠB IB REF 12 Ġdemonstrate Ġan Ġattack Ġthat Ġbreaks Ġentail ment Ġsystems Ġby Ġreplacing Ġa Ġsingle Ġword Ġwith Ġeither Ġa Ġsyn onym Ġor Ġits Ġhyper ny m . ĠRecently , ĠB IB REF 13 Ġinvestigated Ġthe Ġproblem Ġof Ġproducing Ġnatural - se eming Ġadvers arial Ġexamples , Ġnoting Ġthat Ġadvers arial Ġexamples Ġin ĠN LP Ġare Ġoften Ġun gram matical ĠB IB REF 14 Ġ. </s> In Ġrelated Ġwork Ġon Ġcharacter - level Ġattacks , ĠB IB REF 8 Ġ, ĠB IB REF 15 Ġexplored Ġgradient - based Ġmethods Ġto Ġgenerate Ġstring Ġedits Ġto Ġfool Ġclassification Ġand Ġtranslation Ġsystems , Ġrespectively . ĠWhile Ġtheir Ġfocus Ġis Ġon Ġefficient Ġmethods Ġfor Ġgenerating Ġadversaries , Ġours Ġis Ġon Ġimproving Ġthe Ġworst Ġcase Ġadvers arial Ġperformance . ĠSimilarly , ĠB IB REF 9 Ġstudied Ġhow Ġsynthetic Ġand Ġnatural Ġnoise Ġaffects Ġcharacter - level Ġmachine Ġtranslation . ĠThey Ġconsidered Ġstructure Ġinvari ant Ġrepresentations Ġand Ġadvers arial Ġtraining Ġas Ġdefenses Ġagainst Ġsuch Ġnoise . ĠHere , Ġwe Ġshow Ġthat Ġan Ġauxiliary Ġword Ġrecognition Ġmodel , Ġwhich Ġcan Ġbe Ġtrained Ġon Ġunl abel ed Ġdata , Ġprovides Ġa Ġstrong Ġdefense . </s> Spe lling Ġcorrection ĠB IB REF 16 Ġis Ġoften Ġviewed Ġas Ġa Ġsub - task Ġof Ġgram matical Ġerror Ġcorrection ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ. ĠClassic Ġmethods Ġrely Ġon Ġa Ġsource Ġlanguage Ġmodel Ġand Ġa Ġnoisy Ġchannel Ġmodel Ġto Ġfind Ġthe Ġmost Ġlikely Ġcorrection Ġfor Ġa Ġgiven Ġword ĠB IB REF 19 Ġ, ĠB IB REF 20 Ġ. ĠRecently , Ġneural Ġtechniques Ġhave Ġbeen Ġapplied Ġto Ġthe Ġtask ĠB IB REF 7 Ġ, ĠB IB REF 21 Ġ, Ġwhich Ġmodel Ġthe Ġcontext Ġand Ġorth ography Ġof Ġthe Ġinput Ġtogether . ĠOur Ġwork Ġextends Ġthe ĠSc R NN Ġmodel Ġof ĠB IB REF 7 Ġ. </s> Rob ust ĠWord ĠRecogn ition </s> To Ġtackle Ġcharacter - level Ġadvers arial Ġattacks , Ġwe Ġintroduce Ġa Ġsimple Ġtwo - stage Ġsolution , Ġplacing Ġa Ġword Ġrecognition Ġmodel Ġ( Ġ$ W $ Ġ) Ġbefore Ġthe Ġdownstream Ġclass ifier Ġ( Ġ$ C $ Ġ). ĠUnder Ġthis Ġscheme , Ġall Ġinputs Ġare Ġclassified Ġby Ġthe Ġcomposed Ġmodel Ġ$ C Ġ\ circ ĠW $ Ġ. ĠThis Ġmodular Ġapproach , Ġwith Ġ$ W $ Ġand Ġ$ C $ Ġtrained Ġseparately , Ġoffers Ġseveral Ġbenefits : Ġ( i ) Ġwe Ġcan Ġdeploy Ġthe Ġsame Ġword Ġrecognition Ġmodel Ġfor Ġmultiple Ġdownstream Ġclassification Ġtasks / models ; Ġand Ġ( ii ) Ġwe Ġcan Ġtrain Ġthe Ġword Ġrecognition Ġmodel Ġwith Ġlarger Ġunl abel ed Ġcorpor a . </s> Against Ġadvers arial Ġmistakes , Ġtwo Ġimportant Ġfactors Ġgovern Ġthe Ġrobust ness Ġof Ġthis Ġcombined Ġmodel : Ġ$ W $ Ġ' s Ġaccuracy Ġin Ġrecognizing Ġmiss pelled Ġwords Ġand Ġ$ W $ Ġ' s Ġsensitivity Ġto Ġadvers arial Ġpert urb ations Ġon Ġthe Ġsame Ġinput . ĠWe Ġdiscuss Ġthese Ġaspects Ġin Ġdetail Ġbelow . </s> Sc R NN Ġwith ĠBack off </s> We Ġnow Ġdescribe Ġsemi - character ĠR NN s Ġfor Ġword Ġrecognition , Ġexplain Ġtheir Ġlimitations , Ġand Ġsuggest Ġtechniques Ġto Ġimprove Ġthem . </s> Insp ired Ġby Ġthe Ġpsych ol ingu istic Ġstudies ĠB IB REF 5 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 7 Ġproposed Ġa Ġsemi - character Ġbased ĠR NN Ġ( Sc R NN ) Ġthat Ġprocesses Ġa Ġsentence Ġof Ġwords Ġwith Ġmiss pelled Ġcharacters , Ġpredicting Ġthe Ġcorrect Ġwords Ġat Ġeach Ġstep . ĠLet Ġ$ s Ġ= Ġ\ l brace Ġw _ 1 , Ġw _ 2 , Ġ\ d ots Ġ, Ġw _ n \ r brace Ġ$ Ġdenote Ġthe Ġinput Ġsentence , Ġa Ġsequence Ġof Ġconstituent Ġwords Ġ$ w _ i $ Ġ. ĠEach Ġinput Ġword Ġ( Ġ$ w _ i $ Ġ) Ġis Ġrepresented Ġby Ġconc aten ating Ġ( i ) Ġa Ġone Ġhot Ġvector Ġof Ġthe Ġfirst Ġcharacter Ġ( Ġ$\ math bf Ġ{ w _{ i 1 }} $ Ġ); Ġ( ii ) Ġa Ġone Ġhot Ġrepresentation Ġof Ġthe Ġlast Ġcharacter Ġ( Ġ$\ math bf Ġ{ w _{ il }} $ Ġ, Ġwhere Ġ$ l $ Ġis Ġthe Ġlength Ġof Ġword Ġ$ w _ i $ Ġ); Ġand Ġ( iii ) Ġa Ġbag Ġof Ġcharacters Ġrepresentation Ġof Ġthe Ġinternal Ġcharacters Ġ( Ġ$\ sum Ġ_ { j = 2 } ^{ l - 1 }\ math bf Ġ{ w _{ ij }} ) $ Ġ. ĠSc R NN Ġtreats Ġthe Ġfirst Ġand Ġthe Ġlast Ġcharacters Ġindividually , Ġand Ġis Ġag nostic Ġto Ġthe Ġordering Ġof Ġthe Ġinternal Ġcharacters . ĠEach Ġword , Ġrepresented Ġaccordingly , Ġis Ġthen Ġfed Ġinto Ġa ĠBi L ST M Ġcell . ĠAt Ġeach Ġsequence Ġstep , Ġthe Ġtraining Ġtarget Ġis Ġthe Ġcorrect Ġcorresponding Ġword Ġ( output Ġdimension Ġequal Ġto Ġvocabulary Ġsize ), Ġand Ġthe Ġmodel Ġis Ġoptimized Ġwith Ġcross - ent ropy Ġloss . </s> While ĠB IB REF 7 Ġdemonstrate Ġstrong Ġword Ġrecognition Ġperformance , Ġa Ġdrawback Ġof Ġtheir Ġevaluation Ġsetup Ġis Ġthat Ġthey Ġonly Ġattack Ġand Ġevaluate Ġon Ġthe Ġsubset Ġof Ġwords Ġthat Ġare Ġa Ġpart Ġof Ġtheir Ġtraining Ġvocabulary . ĠIn Ġsuch Ġa Ġsetting , Ġthe Ġword Ġrecognition Ġperformance Ġis Ġunre ason ably Ġdependent Ġon Ġthe Ġchosen Ġvocabulary Ġsize . ĠIn Ġprinciple , Ġone Ġcan Ġdesign Ġmodels Ġto Ġpredict Ġ( correct ly ) Ġonly Ġa Ġfew Ġchosen Ġwords , Ġand Ġignore Ġthe Ġremaining Ġmajority Ġand Ġstill Ġreach Ġ100 % Ġaccuracy . ĠFor Ġthe Ġadvers arial Ġsetting , Ġrare Ġand Ġunseen Ġwords Ġin Ġthe Ġwild Ġare Ġparticularly Ġcritical , Ġas Ġthey Ġprovide Ġopportunities Ġfor Ġthe Ġattackers . ĠA Ġreliable Ġword - recogn izer Ġshould Ġhandle Ġthese Ġcases Ġgrace fully . ĠBelow , Ġwe Ġexplore Ġdifferent Ġways Ġto Ġback Ġoff Ġwhen Ġthe ĠSc R NN Ġpredicts ĠUN K Ġ( a Ġfrequent Ġoutcome Ġfor Ġrare Ġand Ġunseen Ġwords ): </s> Pass - through : Ġword - recogn izer Ġpasses Ġon Ġthe Ġ( possibly Ġmiss pelled ) Ġword Ġas Ġis . </s> Back off Ġto Ġneutral Ġword : ĠAlternatively , Ġnoting Ġthat Ġpassing Ġ$\ color box Ġ{ gray ! 20 }{ \ text tt Ġ{ UN K }} $ Ġ- pred icted Ġwords Ġthrough Ġunchanged Ġexposes Ġthe Ġdownstream Ġmodel Ġto Ġpotentially Ġcorrupted Ġtext , Ġwe Ġconsider Ġbacking Ġoff Ġto Ġa Ġneutral Ġword Ġlike Ġ` a ', Ġwhich Ġhas Ġa Ġsimilar Ġdistribution Ġacross Ġclasses . </s> Back off Ġto Ġbackground Ġmodel : ĠWe Ġalso Ġconsider Ġfalling Ġback Ġupon Ġa Ġmore Ġgeneric Ġword Ġrecognition Ġmodel Ġtrained Ġupon Ġa Ġlarger , Ġless - special ized Ġcorpus Ġwhenever Ġthe Ġforeground Ġword Ġrecognition Ġmodel Ġpredicts ĠUN K . ĠFigure Ġ1 Ġdepicts Ġthis Ġscenario Ġpict or ially . </s> E mp ir ically , Ġwe Ġfind Ġthat Ġthe Ġbackground Ġmodel Ġ( by Ġitself ) Ġis Ġless Ġaccurate , Ġbecause Ġof Ġthe Ġlarge Ġnumber Ġof Ġwords Ġit Ġis Ġtrained Ġto Ġpredict . ĠThus , Ġit Ġis Ġbest Ġto Ġtrain Ġa Ġprecise Ġforeground Ġmodel Ġon Ġan Ġin - domain Ġcorpus Ġand Ġfocus Ġon Ġfrequent Ġwords , Ġand Ġthen Ġto Ġresort Ġto Ġa Ġgeneral - purpose Ġbackground Ġmodel Ġfor Ġrare Ġand Ġunob served Ġwords . ĠNext , Ġwe Ġdeline ate Ġour Ġsecond Ġconsideration Ġfor Ġbuilding Ġrobust Ġword - recogn izers . </s> Model ĠSens itivity </s> In Ġcomputer Ġvision , Ġan Ġimportant Ġfactor Ġdetermining Ġthe Ġsuccess Ġof Ġan Ġadversary Ġis Ġthe Ġnorm Ġconstraint Ġon Ġthe Ġpert urb ations Ġallowed Ġto Ġan Ġimage Ġ( Ġ$ || Ġ\ bf Ġx Ġ- Ġ\ bf Ġx ^ {\ prime Ġ} || _{ \ in fty Ġ} Ġ< Ġ\ ep sil on Ġ$ Ġ). ĠHigher Ġvalues Ġof Ġ$\ ep sil on Ġ$ Ġlead Ġto Ġa Ġhigher Ġchance Ġof Ġmis - class ification Ġfor Ġat Ġleast Ġone Ġ$\ bf Ġx ^ {\ prime Ġ} $ Ġ. ĠDefense Ġmethods Ġsuch Ġas Ġquant ization ĠB IB REF 22 Ġand Ġtherm ometer Ġencoding ĠB IB REF 23 Ġtry Ġto Ġreduce Ġthe Ġspace Ġof Ġpert urb ations Ġavailable Ġto Ġthe Ġadversary Ġby Ġmaking Ġthe Ġmodel Ġinvari ant Ġto Ġsmall Ġchanges Ġin Ġthe Ġinput . </s> In ĠN LP , Ġwe Ġoften Ġget Ġsuch Ġinvari ance Ġfor Ġfree , Ġe . g ., Ġfor Ġa Ġword - level Ġmodel , Ġmost Ġof Ġthe Ġpert urb ations Ġproduced Ġby Ġour Ġcharacter - level Ġadversary Ġlead Ġto Ġan ĠUN K Ġat Ġits Ġinput . ĠIf Ġthe Ġmodel Ġis Ġrobust Ġto Ġthe Ġpresence Ġof Ġthese ĠUN K Ġtokens , Ġthere Ġis Ġlittle Ġroom Ġfor Ġan Ġadversary Ġto Ġmanipulate Ġit . ĠCharacter - level Ġmodels , Ġon Ġthe Ġother Ġhand , Ġdespite Ġtheir Ġsuperior Ġperformance Ġin Ġmany Ġtasks , Ġdo Ġnot Ġenjoy Ġsuch Ġinvari ance . ĠThis Ġcharacteristic Ġinvari ance Ġcould Ġbe Ġexploited Ġby Ġan Ġattacker . ĠThus , Ġto Ġlimit Ġthe Ġnumber Ġof Ġdifferent Ġinputs Ġto Ġthe Ġclass ifier , Ġwe Ġwish Ġto Ġreduce Ġthe Ġnumber Ġof Ġdistinct Ġword Ġrecognition Ġoutputs Ġthat Ġan Ġattacker Ġcan Ġinduce , Ġnot Ġjust Ġthe Ġnumber Ġof Ġwords Ġon Ġwhich Ġthe Ġmodel Ġis ĠâĢ ľ f ool ed âĢ Ŀ . ĠWe Ġdenote Ġthis Ġproperty Ġof Ġa Ġmodel Ġas Ġits Ġsensitivity . </s> We Ġcan Ġquantify Ġthis Ġnotion Ġfor Ġa Ġword Ġrecognition Ġsystem Ġ$ W $ Ġas Ġthe Ġexpected Ġnumber Ġof Ġunique Ġoutputs Ġit Ġassigns Ġto Ġa Ġset Ġof Ġadvers arial Ġpert urb ations . ĠGiven Ġa Ġsentence Ġ$ s $ Ġfrom Ġthe Ġset Ġof Ġsentences Ġ$\ math cal Ġ{ S } $ Ġ, Ġlet Ġ$ A ( s ) Ġ= Ġ{ s _ 1 } ^ {\ prime Ġ} Ġ, Ġ{ s _ 2 } ^ {\ prime Ġ}, Ġ\ d ots Ġ, Ġ{ s _ n } ^ {\ prime Ġ} $ Ġdenote Ġthe Ġset Ġof Ġ$ n $ Ġpert urb ations Ġto Ġit Ġunder Ġattack Ġtype Ġ$ A $ Ġ, Ġand Ġlet Ġ$ V $ Ġbe Ġthe Ġfunction Ġthat Ġmaps Ġstrings Ġto Ġan Ġinput Ġrepresentation Ġfor Ġthe Ġdownstream Ġclass ifier . ĠFor Ġa Ġword Ġlevel Ġmodel , Ġ$ V $ Ġwould Ġtransform Ġsentences Ġto Ġa Ġsequence Ġof Ġword Ġ ids , Ġmapping ĠO OV Ġwords Ġto Ġthe Ġsame ĠUN K ĠID . ĠWhereas , Ġfor Ġa Ġchar Ġ( or Ġword + char , Ġword - piece ) Ġmodel , Ġ$ V $ Ġwould Ġmap Ġinputs Ġto Ġa Ġsequence Ġof Ġcharacter ĠIDs . ĠForm ally , Ġsensitivity Ġis Ġdefined Ġas </s> $$ S _{ W , V } ^ A = \ math bb Ġ{ E } _{ s }\ left [ \ frac {\ # _{ u } ( V Ġ\ circ ĠW ({ s _ 1 } ^ {\ prime Ġ} ), Ġ\ d ots Ġ, ĠV Ġ\ circ ĠW ({ s _ n } ^ {\ prime Ġ} )) }{ n }\ right ] Ġ, $$ Ġ Ġ Ġ( E q . Ġ12 ) </s> where Ġ$ V Ġ\ circ ĠW Ġ( s _ i ) $ Ġreturns Ġthe Ġinput Ġrepresentation Ġ( of Ġthe Ġdownstream Ġclass ifier ) Ġfor Ġthe Ġoutput Ġstring Ġproduced Ġby Ġthe Ġword - recogn izer Ġ$ W $ Ġusing Ġ$ s _ i $ Ġand Ġ$\ # _{ u } (\ cd ot Ġ) $ Ġcounts Ġthe Ġnumber Ġof Ġunique Ġarguments . </s> Int uitive ly , Ġwe Ġexpect Ġa Ġhigh Ġvalue Ġof Ġ$ S _{ W , ĠV } ^ A $ Ġto Ġlead Ġto Ġa Ġlower Ġrobust ness Ġof Ġthe Ġdownstream Ġclass ifier , Ġsince Ġthe Ġadversary Ġhas Ġmore Ġdegrees Ġof Ġfreedom Ġto Ġattack Ġthe Ġclass ifier . ĠThus , Ġwhen Ġusing Ġword Ġrecognition Ġas Ġa Ġdefense , Ġit Ġis Ġprudent Ġto Ġdesign Ġa Ġlow Ġsensitivity Ġsystem Ġwith Ġa Ġlow Ġerror Ġrate . ĠHowever , Ġas Ġwe Ġwill Ġdemonstrate , Ġthere Ġis Ġoften Ġa Ġtrade - off Ġbetween Ġsensitivity Ġand Ġerror Ġrate . </s> Sy nt hes izing ĠAd vers arial ĠAttacks </s> Supp ose Ġwe Ġare Ġgiven Ġa Ġclass ifier Ġ$ C : Ġ\ math cal Ġ{ S } Ġ\ right arrow Ġ\ math cal Ġ{ Y } $ Ġwhich Ġmaps Ġnatural Ġlanguage Ġsentences Ġ$ s Ġ\ in Ġ\ math cal Ġ{ S } $ Ġto Ġa Ġlabel Ġfrom Ġa Ġpred efined Ġset Ġ$ y Ġ\ in Ġ\ math cal Ġ{ Y } $ Ġ. ĠAn Ġadversary Ġfor Ġthis Ġclass ifier Ġis Ġa Ġfunction Ġ$ A $ Ġwhich Ġmaps Ġa Ġsentence Ġ$ s $ Ġto Ġits Ġpert urbed Ġversions Ġ$\ l brace Ġs ^ {\ prime Ġ} _ 1 , Ġs ^ {\ prime Ġ} _ 2 , Ġ\ ld ots Ġ, Ġs ^ {\ prime Ġ} _{ n }\ r brace Ġ$ Ġsuch Ġthat Ġeach Ġ$ s ^ {\ prime Ġ} _ i $ Ġis Ġclose Ġto Ġ$ s $ Ġunder Ġsome Ġnotion Ġof Ġdistance Ġbetween Ġsentences . ĠWe Ġdefine Ġthe Ġrobust ness Ġof Ġclass ifier Ġ$ C $ Ġto Ġthe Ġadversary Ġ$ A $ Ġas : </s> $$ R _{ C , A } Ġ= Ġ\ math bb Ġ{ E } _ s Ġ\ left [ \ min Ġ_ { s ^ {\ prime Ġ} Ġ\ in ĠA ( s )} Ġ\ math bb Ġ{ 1 } [ C ( s ^ {\ prime Ġ}) Ġ= Ġy ] \ right ], $$ Ġ Ġ Ġ( E q . Ġ14 ) </s> where Ġ$ y $ Ġrepresents Ġthe Ġground Ġtruth Ġlabel Ġfor Ġ$ s $ Ġ. ĠIn Ġpractice , Ġa Ġreal - world Ġadversary Ġmay Ġonly Ġbe Ġable Ġto Ġquery Ġthe Ġclass ifier Ġa Ġfew Ġtimes , Ġhence Ġ$ R _{ C , A } $ Ġrepresents Ġthe Ġworst - case Ġadvers arial Ġperformance Ġof Ġ$ C $ Ġ. ĠMethods Ġfor Ġgenerating Ġadvers arial Ġexamples , Ġsuch Ġas ĠHot Fl ip ĠB IB REF 8 Ġ, Ġfocus Ġon Ġefficient Ġalgorithms Ġfor Ġsearching Ġthe Ġ$\ min Ġ$ Ġabove . ĠImpro ving Ġ$ R _{ C , A } $ Ġwould Ġimply Ġbetter Ġrobust ness Ġagainst Ġall Ġthese Ġmethods . </s> We Ġexplore Ġadversaries Ġwhich Ġpert urb Ġsentences Ġwith Ġfour Ġtypes Ġof Ġcharacter - level Ġedits : </s> ( 1 ) ĠSwap : Ġswapping Ġtwo Ġadjacent Ġinternal Ġcharacters Ġof Ġa Ġword . Ġ( 2 ) ĠDrop : Ġremoving Ġan Ġinternal Ġcharacter Ġof Ġa Ġword . Ġ( 3 ) ĠKeyboard : Ġsubstit uting Ġan Ġinternal Ġcharacter Ġwith Ġadjacent Ġcharacters Ġof ĠQ WER TY Ġkeyboard Ġ( 4 ) ĠAdd : Ġinserting Ġa Ġnew Ġcharacter Ġinternally Ġin Ġa Ġword . ĠIn Ġline Ġwith Ġthe Ġpsych ol ingu istic Ġstudies ĠB IB REF 5 Ġ, ĠB IB REF 4 Ġ, Ġto Ġensure Ġthat Ġthe Ġpert urb ations Ġdo Ġnot Ġaffect Ġhuman Ġability Ġto Ġcomprehend Ġthe Ġsentence , Ġwe Ġonly Ġallow Ġthe Ġadversary Ġto Ġedit Ġthe Ġinternal Ġcharacters Ġof Ġa Ġword , Ġand Ġnot Ġedit Ġstop words Ġor Ġwords Ġshorter Ġthan Ġ4 Ġcharacters . </s> For Ġ1 - character Ġattacks , Ġwe Ġtry Ġall Ġpossible Ġpert urb ations Ġlisted Ġabove Ġuntil Ġwe Ġfind Ġan Ġadversary Ġthat Ġflips Ġthe Ġmodel Ġprediction . ĠFor Ġ2 - character Ġattacks , Ġwe Ġgreed ily Ġfix Ġthe Ġedit Ġwhich Ġhad Ġthe Ġleast Ġconfidence Ġamong Ġ1 - character Ġattacks , Ġand Ġthen Ġtry Ġall Ġthe Ġallowed Ġpert urb ations Ġon Ġthe Ġremaining Ġwords . ĠHigher Ġorder Ġattacks Ġcan Ġbe Ġperformed Ġin Ġa Ġsimilar Ġmanner . ĠThe Ġgreedy Ġstrategy Ġreduces Ġthe Ġcomputation Ġrequired Ġto Ġobtain Ġhigher Ġorder Ġattacks , Ġbut Ġalso Ġmeans Ġthat Ġthe Ġrobust ness Ġscore Ġis Ġan Ġupper Ġbound Ġon Ġthe Ġtrue Ġrobust ness Ġof Ġthe Ġclass ifier . </s> Exper iments Ġand ĠResults </s> In Ġthis Ġsection , Ġwe Ġfirst Ġdiscuss Ġour Ġexperiments Ġon Ġthe Ġword Ġrecognition Ġsystems . </s> Word ĠError ĠCorrection </s> Data : ĠWe Ġevaluate Ġthe Ġspell Ġcorrect ors Ġfrom ĠÂ§ Ġ" Rob ust ĠWord ĠRecogn ition " Ġon Ġmovie Ġreviews Ġfrom Ġthe ĠStanford ĠSent iment ĠTree bank Ġ( S ST ) ĠB IB REF 24 Ġ. ĠThe ĠS ST Ġdataset Ġconsists Ġof Ġ85 44 Ġmovie Ġreviews , Ġwith Ġa Ġvocabulary Ġof Ġover Ġ16 K Ġwords . ĠAs Ġa Ġbackground Ġcorpus , Ġwe Ġuse Ġthe ĠIM DB Ġmovie Ġreviews ĠB IB REF 25 Ġ, Ġwhich Ġcontain Ġ54 K Ġmovie Ġreviews , Ġand Ġa Ġvocabulary Ġof Ġover Ġ78 K Ġwords . ĠThe Ġtwo Ġdatasets Ġdo Ġnot Ġshare Ġany Ġreviews Ġin Ġcommon . ĠThe Ġspell - cor rection Ġmodels Ġare Ġevaluated Ġon Ġtheir Ġability Ġto Ġcorrect Ġmiss pell ings . ĠThe Ġtest Ġsetting Ġconsists Ġof Ġreviews Ġwhere Ġeach Ġword Ġ( with Ġlength Ġ$\ ge Ġ4 $ Ġ, Ġbarring Ġstop words ) Ġis Ġattacked Ġby Ġone Ġof Ġthe Ġattack Ġtypes Ġ( from Ġswap , Ġadd , Ġdrop Ġand Ġkeyboard Ġattacks ). ĠIn Ġthe Ġall Ġattack Ġsetting , Ġwe Ġmix Ġall Ġattacks Ġby Ġrandomly Ġchoosing Ġone Ġfor Ġeach Ġword . ĠThis Ġmost Ġclosely Ġresembles Ġa Ġreal Ġworld Ġattack Ġsetting . </s> In Ġaddition Ġto Ġour Ġword Ġrecognition Ġmodels , Ġwe Ġalso Ġcompare Ġto ĠAfter ĠThe ĠDeadline Ġ( AT D ), Ġan Ġopen - source Ġspell Ġcorrect or . ĠWe Ġfound ĠAT D Ġto Ġbe Ġthe Ġbest Ġfreely - available Ġcorrect or . ĠWe Ġrefer Ġthe Ġreader Ġto ĠB IB REF 7 Ġfor Ġcomparisons Ġof ĠSc R NN Ġto Ġother Ġanonym ized Ġcommercial Ġspell Ġcheck ers . </s> For Ġthe ĠSc R NN Ġmodel , Ġwe Ġuse Ġa Ġsingle - layer ĠBi - L ST M Ġwith Ġa Ġhidden Ġdimension Ġsize Ġof Ġ50 . ĠThe Ġinput Ġrepresentation Ġconsists Ġof Ġ198 Ġdimensions , Ġwhich Ġis Ġthr ice Ġthe Ġnumber Ġof Ġunique Ġcharacters Ġ( 66 ) Ġin Ġthe Ġvocabulary . ĠWe Ġcap Ġthe Ġvocabulary Ġsize Ġto Ġ10 K Ġwords , Ġwhereas Ġwe Ġuse Ġthe Ġentire Ġvocabulary Ġof Ġ7 84 70 Ġwords Ġwhen Ġwe Ġback off Ġto Ġthe Ġbackground Ġmodel . ĠFor Ġtraining Ġthese Ġnetworks , Ġwe Ġcorrupt Ġthe Ġmovie Ġreviews Ġaccording Ġto Ġall Ġattack Ġtypes , Ġi . e ., Ġapplying Ġone Ġof Ġthe Ġ4 Ġattack Ġtypes Ġto Ġeach Ġword , Ġand Ġtrying Ġto Ġreconstruct Ġthe Ġoriginal Ġwords Ġvia Ġcross Ġentropy Ġloss . </s> We Ġcalculate Ġthe Ġword Ġerror Ġrates Ġ( WER ) Ġof Ġeach Ġof Ġthe Ġmodels Ġfor Ġdifferent Ġattacks Ġand Ġpresent Ġour Ġfindings Ġin ĠTable Ġ2 Ġ. ĠNote Ġthat ĠAT D Ġincorrectly Ġpredicts Ġ$ 11 . 2 $ Ġwords Ġfor Ġevery Ġ100 Ġwords Ġ( in Ġthe Ġ` all ' Ġsetting ), Ġwhereas , Ġall Ġof Ġthe Ġback off Ġvariations Ġof Ġthe ĠSc R NN Ġreconstruct Ġbetter . ĠThe Ġmost Ġaccurate Ġvariant Ġinvolves Ġbacking Ġoff Ġto Ġthe Ġbackground Ġmodel , Ġresulting Ġin Ġa Ġlow Ġerror Ġrate Ġof Ġ$ 6 . 9 \ % $ Ġ, Ġleading Ġto Ġthe Ġbest Ġperformance Ġon Ġword Ġrecognition . ĠThis Ġis Ġa Ġ$ 32 \ % $ Ġrelative Ġerror Ġreduction Ġcompared Ġto Ġthe Ġvanilla ĠSc R NN Ġmodel Ġwith Ġa Ġpass - through Ġback off Ġstrategy . ĠWe Ġcan Ġattribute Ġthe Ġimproved Ġperformance Ġto Ġthe Ġfact Ġthat Ġthere Ġare Ġ$ 5 . 25 \ % $ Ġwords Ġin Ġthe Ġtest Ġcorpus Ġthat Ġare Ġunseen Ġin Ġthe Ġtraining Ġcorpus , Ġand Ġare Ġthus Ġonly Ġrecover able Ġby Ġbacking Ġoff Ġto Ġa Ġlarger Ġcorpus . ĠNotably , Ġonly Ġtraining Ġon Ġthe Ġlarger Ġbackground Ġcorpus Ġdoes Ġworse , Ġat Ġ$ 8 . 7 \ % $ Ġ, Ġsince Ġthe Ġdistribution Ġof Ġword Ġfrequencies Ġis Ġdifferent Ġin Ġthe Ġbackground Ġcorpus Ġcompared Ġto Ġthe Ġforeground Ġcorpus . </s> Rob ust ness Ġto Ġadvers arial Ġattacks </s> We Ġuse Ġsentiment Ġanalysis Ġand Ġparaph rase Ġdetection Ġas Ġdownstream Ġtasks , Ġas Ġfor Ġthese Ġtwo Ġtasks , Ġ1 - 2 Ġcharacter Ġedits Ġdo Ġnot Ġchange Ġthe Ġoutput Ġlabels . </s> For Ġsentiment Ġclassification , Ġwe Ġsystematically Ġstudy Ġthe Ġeffect Ġof Ġcharacter - level Ġadvers arial Ġattacks Ġon Ġtwo Ġarchitectures Ġand Ġfour Ġdifferent Ġinput Ġformats . ĠThe Ġfirst Ġarchitecture Ġenc odes Ġthe Ġinput Ġsentence Ġinto Ġa Ġsequence Ġof Ġembed d ings , Ġwhich Ġare Ġthen Ġsequ entially Ġprocessed Ġby Ġa ĠBi L ST M . ĠThe Ġfirst Ġand Ġlast Ġstates Ġof Ġthe ĠBi L ST M Ġare Ġthen Ġused Ġby Ġthe Ġsoft max Ġlayer Ġto Ġpredict Ġthe Ġsentiment Ġof Ġthe Ġinput . ĠWe Ġconsider Ġthree Ġinput Ġformats Ġfor Ġthis Ġarchitecture : Ġ( 1 ) ĠWord - only : Ġwhere Ġthe Ġinput Ġwords Ġare Ġencoded Ġusing Ġa Ġlookup Ġtable ; Ġ( 2 ) ĠChar - only : Ġwhere Ġthe Ġinput Ġwords Ġare Ġencoded Ġusing Ġa Ġseparate Ġsingle - lay ered ĠBi L ST M Ġover Ġtheir Ġcharacters ; Ġand Ġ( 3 ) ĠWord Ġ$ + $ ĠChar : Ġwhere Ġthe Ġinput Ġwords Ġare Ġencoded Ġusing Ġa Ġconc aten ation Ġof Ġ( 1 ) Ġand Ġ( 2 ) Ġ. </s> The Ġsecond Ġarchitecture Ġuses Ġthe Ġfine - tun ed ĠB ERT Ġmodel ĠB IB REF 26 Ġ, Ġwith Ġan Ġinput Ġformat Ġof Ġword - piece Ġtoken ization . ĠThis Ġmodel Ġhas Ġrecently Ġset Ġa Ġnew Ġstate - of - the - art Ġon Ġseveral ĠN LP Ġbenchmarks , Ġincluding Ġthe Ġsentiment Ġanalysis Ġtask Ġwe Ġconsider Ġhere . ĠAll Ġmodels Ġare Ġtrained Ġand Ġevaluated Ġon Ġthe Ġbinary Ġversion Ġof Ġthe Ġsentence - level ĠStanford ĠSent iment ĠTree bank ĠB IB REF 24 Ġdataset Ġwith Ġonly Ġpositive Ġand Ġnegative Ġreviews . </s> We Ġalso Ġconsider Ġthe Ġtask Ġof Ġparaph rase Ġdetection . ĠHere Ġtoo , Ġwe Ġmake Ġuse Ġof Ġthe Ġfine - tun ed ĠB ERT ĠB IB REF 26 Ġ, Ġwhich Ġis Ġtrained Ġand Ġevaluated Ġon Ġthe ĠMicrosoft ĠResearch ĠPar aph rase ĠCorpus Ġ( MR PC ) ĠB IB REF 27 Ġ. </s> Two Ġcommon Ġmethods Ġfor Ġdealing Ġwith Ġadvers arial Ġexamples Ġinclude : Ġ( 1 ) Ġdata Ġaug mentation Ġ( DA ) ĠB IB REF 28 Ġ; Ġand Ġ( 2 ) Ġadvers arial Ġtraining Ġ( Adv ) ĠB IB REF 29 Ġ. ĠIn ĠDA , Ġthe Ġtrained Ġmodel Ġis Ġfine - tun ed Ġafter Ġaugment ing Ġthe Ġtraining Ġset Ġwith Ġan Ġequal Ġnumber Ġof Ġexamples Ġrandomly Ġattacked Ġwith Ġa Ġ1 - character Ġedit . ĠIn ĠAdv , Ġthe Ġtrained Ġmodel Ġis Ġfine - tun ed Ġwith Ġadditional Ġadvers arial Ġexamples Ġ( selected Ġat Ġrandom ) Ġthat Ġproduce Ġincorrect Ġpredictions Ġfrom Ġthe Ġcurrent - state Ġclass ifier . ĠThe Ġprocess Ġis Ġrepeated Ġiter atively , Ġgenerating Ġand Ġadding Ġnewer Ġadvers arial Ġexamples Ġfrom Ġthe Ġupdated Ġclass ifier Ġmodel , Ġuntil Ġthe Ġadvers arial Ġaccuracy Ġon Ġdev Ġset Ġstops Ġimproving . </s> In ĠTable Ġ3 Ġ, Ġwe Ġexamine Ġthe Ġrobust ness Ġof Ġthe Ġsentiment Ġmodels Ġunder Ġeach Ġattack Ġand Ġdefense Ġmethod . ĠIn Ġthe Ġabsence Ġof Ġany Ġattack Ġor Ġdefense , ĠB ERT Ġ( a Ġword - piece Ġmodel ) Ġperforms Ġthe Ġbest Ġ( Ġ$ 90 . 3 \ % $ Ġ) Ġfollowed Ġby Ġword + char Ġmodels Ġ( Ġ$ 80 . 5 \ % $ Ġ), Ġword - only Ġmodels Ġ( Ġ$ 79 . 2 \ % $ Ġ) Ġand Ġthen Ġchar - only Ġmodels Ġ( Ġ$ 70 . 3 \ % $ Ġ). ĠHowever , Ġeven Ġsingle - character Ġattacks Ġ( ch osen Ġadvers ar ially ) Ġcan Ġbe Ġcatastrophic , Ġresulting Ġin Ġa Ġsignificantly Ġdegraded Ġperformance Ġof Ġ$ 46 \ % $ Ġ, Ġ$ 57 \ % $ Ġ, Ġ$ 59 \ % $ Ġand Ġ$ 33 \ % $ Ġ, Ġrespectively Ġunder Ġthe Ġ` all ' Ġsetting . </s> Int uitive ly , Ġone Ġmight Ġsuppose Ġthat Ġword - piece Ġand Ġcharacter - level Ġmodels Ġwould Ġbe Ġmore Ġrobust Ġto Ġsuch Ġattacks Ġgiven Ġthey Ġcan Ġmake Ġuse Ġof Ġthe Ġremaining Ġcontext . ĠHowever , Ġwe Ġfind Ġthat Ġthey Ġare Ġthe Ġmore Ġsusceptible . ĠTo Ġsee Ġwhy , Ġnote Ġthat Ġthe Ġword Ġ` beaut iful ' Ġcan Ġonly Ġbe Ġaltered Ġin Ġa Ġfew Ġways Ġfor Ġword - only Ġmodels , Ġeither Ġleading Ġto Ġan ĠUN K Ġor Ġan Ġexisting Ġvocabulary Ġword , Ġwhereas , Ġword - piece Ġand Ġcharacter - only Ġmodels Ġtreat Ġeach Ġunique Ġcharacter Ġcombination Ġdifferently . ĠThis Ġprovides Ġmore Ġvariations Ġthat Ġan Ġattacker Ġcan Ġexploit . ĠFollowing Ġsimilar Ġreasoning , Ġadd Ġand Ġkey Ġattacks Ġpose Ġa Ġgreater Ġthreat Ġthan Ġswap Ġand Ġdrop Ġattacks . ĠThe Ġrobust ness Ġof Ġdifferent Ġmodels Ġcan Ġbe Ġordered Ġas Ġword - only Ġ$ > $ Ġword + char Ġ$ > $ Ġchar - only Ġ$\ sim Ġ$ Ġword - piece , Ġand Ġthe Ġefficacy Ġof Ġdifferent Ġattacks Ġas Ġadd Ġ$ > $ Ġkey Ġ$ > $ Ġdrop Ġ$ > $ Ġswap . </s> Next , Ġwe Ġscrutin ize Ġthe Ġeffectiveness Ġof Ġdefense Ġmethods Ġwhen Ġfaced Ġagainst Ġadvers ar ially Ġchosen Ġattacks . ĠClearly Ġfrom Ġtable Ġ3 Ġ, ĠDA Ġand ĠAdv Ġare Ġnot Ġeffective Ġin Ġthis Ġcase . ĠWe Ġobserved Ġthat Ġdespite Ġa Ġlow Ġtraining Ġerror , Ġthese Ġmodels Ġwere Ġnot Ġable Ġto Ġgeneral ize Ġto Ġattacks Ġon Ġnewer Ġwords Ġat Ġtest Ġtime . ĠAT D Ġspell Ġcorrect or Ġis Ġthe Ġmost Ġeffective Ġon Ġkeyboard Ġattacks , Ġbut Ġperforms Ġpoorly Ġon Ġother Ġattack Ġtypes , Ġparticularly Ġthe Ġadd Ġattack Ġstrategy . </s> The ĠSc R NN Ġmodel Ġwith Ġpass - through Ġback off Ġoffers Ġbetter Ġprotection , Ġbringing Ġback Ġthe Ġadvers arial Ġaccuracy Ġwithin Ġ$ 5 \ % $ Ġrange Ġfor Ġthe Ġswap Ġattack . ĠIt Ġis Ġalso Ġeffective Ġunder Ġother Ġattack Ġclasses , Ġand Ġcan Ġmitigate Ġthe Ġadvers arial Ġeffect Ġin Ġword - piece Ġmodels Ġby Ġ$ 21 \ % $ Ġ, Ġcharacter - only Ġmodels Ġby Ġ$ 19 \ % $ Ġ, Ġand Ġin Ġword , Ġand Ġword + char Ġmodels Ġby Ġover Ġ$ 4 . 5 \ % $ Ġ. ĠThis Ġsuggests Ġthat Ġthe Ġdirect Ġtraining Ġsignal Ġof Ġword Ġerror Ġcorrection Ġis Ġmore Ġeffective Ġthan Ġthe Ġindirect Ġsignal Ġof Ġsentiment Ġclassification Ġavailable Ġto ĠDA Ġand ĠAdv Ġfor Ġmodel Ġrobust ness . </s> We Ġobserve Ġadditional Ġgains Ġby Ġusing Ġbackground Ġmodels Ġas Ġa Ġback off Ġalternative , Ġbecause Ġof Ġits Ġlower Ġword Ġerror Ġrate Ġ( WER ), Ġespecially , Ġunder Ġthe Ġswap Ġand Ġdrop Ġattacks . ĠHowever , Ġthese Ġgains Ġdo Ġnot Ġconsistently Ġtranslate Ġin Ġall Ġother Ġsettings , Ġas Ġlower ĠW ER Ġis Ġnecessary Ġbut Ġnot Ġsufficient . ĠBesides Ġlower Ġerror Ġrate , Ġwe Ġfind Ġthat Ġa Ġsolid Ġdefense Ġshould Ġfurnish Ġthe Ġattacker Ġthe Ġfew est Ġoptions Ġto Ġattack , Ġi . e . Ġit Ġshould Ġhave Ġa Ġlow Ġsensitivity . </s> As Ġwe Ġshall Ġsee Ġin Ġsection ĠÂ§ Ġ" Understanding ĠModel ĠSens itivity " Ġ, Ġthe Ġback off Ġneutral Ġvariation Ġhas Ġthe Ġlowest Ġsensitivity Ġdue Ġto Ġmapping ĠUN K Ġpredictions Ġto Ġa Ġfixed Ġneutral Ġword . ĠThus , Ġit Ġresults Ġin Ġthe Ġhighest Ġrobust ness Ġon Ġmost Ġof Ġthe Ġattack Ġtypes Ġfor Ġall Ġfour Ġmodel Ġclasses . </s> Table Ġ4 Ġshows Ġthe Ġaccuracy Ġof ĠB ERT Ġon Ġ200 Ġexamples Ġfrom Ġthe Ġdev Ġset Ġof Ġthe ĠMR PC Ġparaph rase Ġdetection Ġtask Ġunder Ġvarious Ġattack Ġand Ġdefense Ġsettings . ĠWe Ġre - trained Ġthe ĠSc R NN Ġmodel Ġvariants Ġon Ġthe ĠMR PC Ġtraining Ġset Ġfor Ġthese Ġexperiments . ĠAgain , Ġwe Ġfind Ġthat Ġsimple Ġ1 - 2 Ġcharacter Ġattacks Ġcan Ġbring Ġdown Ġthe Ġaccuracy Ġof ĠB ERT Ġsignificantly Ġ( Ġ$ 89 \ % $ Ġto Ġ$ 31 \ % $ Ġ). ĠWord Ġrecognition Ġmodels Ġcan Ġprovide Ġan Ġeffective Ġdefense , Ġwith Ġboth Ġour Ġpass - through Ġand Ġneutral Ġvariants Ġrecovering Ġmost Ġof Ġthe Ġaccuracy . ĠWhile Ġthe Ġneutral Ġback off Ġmodel Ġis Ġeffective Ġon Ġ2 - char Ġattacks , Ġit Ġhurts Ġperformance Ġin Ġthe Ġno Ġattack Ġsetting , Ġsince Ġit Ġincorrectly Ġmod ifies Ġcertain Ġcorrectly Ġspelled Ġentity Ġnames . ĠSince Ġthe Ġtwo Ġvariants Ġare Ġalready Ġeffective , Ġwe Ġdid Ġnot Ġtrain Ġa Ġbackground Ġmodel Ġfor Ġthis Ġtask . </s> Understanding ĠModel ĠSens itivity </s> To Ġstudy Ġmodel Ġsensitivity , Ġfor Ġeach Ġsentence , Ġwe Ġpert urb Ġone Ġrandomly - ch osen Ġword Ġand Ġreplace Ġit Ġwith Ġall Ġpossible Ġpert urb ations Ġunder Ġa Ġgiven Ġattack Ġtype . ĠThe Ġresulting Ġset Ġof Ġpert urbed Ġsentences Ġis Ġthen Ġfed Ġto Ġthe Ġword Ġrecogn izer Ġ( whose Ġsensitivity Ġis Ġto Ġbe Ġestimated ). ĠAs Ġdescribed Ġin Ġequation Ġ12 Ġ, Ġwe Ġcount Ġthe Ġnumber Ġof Ġunique Ġpredictions Ġfrom Ġthe Ġoutput Ġsentences . ĠTwo Ġcorrections Ġare Ġconsidered Ġunique Ġif Ġthey Ġare Ġmapped Ġdifferently Ġby Ġthe Ġdownstream Ġclass ifier . </s> The Ġneutral Ġback off Ġvariant Ġhas Ġthe Ġlowest Ġsensitivity Ġ( Table Ġ5 Ġ). ĠThis Ġis Ġexpected , Ġas Ġit Ġreturns Ġa Ġfixed Ġneutral Ġword Ġwhenever Ġthe ĠSc R NN Ġpredicts Ġan ĠUN K , Ġtherefore Ġreducing Ġthe Ġnumber Ġof Ġunique Ġoutputs Ġit Ġpredicts . ĠOpen Ġvocabulary Ġ( i . e . Ġchar - only , Ġword + char , Ġword - piece ) Ġdownstream Ġclass ifiers Ġconsider Ġevery Ġunique Ġcombination Ġof Ġcharacters Ġdifferently , Ġwhereas Ġword - only Ġclass ifiers Ġinternally Ġtreat Ġall Ġout Ġof Ġvocabulary Ġ( OO V ) Ġwords Ġalike . ĠHence , Ġfor Ġchar - only , Ġword + char , Ġand Ġword - piece Ġmodels , Ġthe Ġpass - through Ġversion Ġis Ġmore Ġsensitive Ġthan Ġthe Ġbackground Ġvariant , Ġas Ġit Ġpasses Ġwords Ġas Ġis Ġ( and Ġeach Ġcombination Ġis Ġconsidered Ġuniquely ). ĠHowever , Ġfor Ġword - only Ġmodels , Ġpass - through Ġis Ġless Ġsensitive Ġas Ġall Ġthe ĠO OV Ġcharacter Ġcombinations Ġare Ġrendered Ġidentical . </s> Ide ally , Ġa Ġpreferred Ġdefense Ġis Ġone Ġwith Ġlow Ġsensitivity Ġand Ġword Ġerror Ġrate . ĠIn Ġpractice , Ġhowever , Ġwe Ġsee Ġthat Ġa Ġlow Ġerror Ġrate Ġoften Ġcomes Ġat Ġthe Ġcost Ġof Ġsensitivity . ĠWe Ġsee Ġthis Ġtrade - off Ġin ĠFigure Ġ2 Ġ, Ġwhere Ġwe Ġplot ĠW ER Ġand Ġsensitivity Ġon Ġthe Ġtwo Ġaxes , Ġand Ġdepict Ġthe Ġrobust ness Ġwhen Ġusing Ġdifferent Ġback off Ġvariants . ĠGenerally , Ġsensitivity Ġis Ġthe Ġmore Ġdominant Ġfactor Ġout Ġof Ġthe Ġtwo , Ġas Ġthe Ġerror Ġrates Ġof Ġthe Ġconsidered Ġvariants Ġare Ġreasonably Ġlow . </s> We Ġverify Ġif Ġthe Ġsentiment Ġ( of Ġthe Ġreviews ) Ġis Ġpreserved Ġwith Ġchar - level Ġattacks . ĠIn Ġa Ġhuman Ġstudy Ġwith Ġ50 Ġattacked Ġ( and Ġsubsequently Ġmis classified ), Ġand Ġ50 Ġunchanged Ġreviews , Ġit Ġwas Ġnoted Ġthat Ġ48 Ġand Ġ49 , Ġrespectively , Ġpreserved Ġthe Ġsentiment . </s> Conclusion </s> As Ġcharacter Ġand Ġword - piece Ġinputs Ġbecome Ġcommonplace Ġin Ġmodern ĠN LP Ġpipelines , Ġit Ġis Ġworth Ġhighlighting Ġthe Ġvulnerability Ġthey Ġadd . ĠWe Ġshow Ġthat Ġminim ally - do ct ored Ġattacks Ġcan Ġbring Ġdown Ġaccuracy Ġof Ġclass ifiers Ġto Ġrandom Ġguessing . ĠWe Ġrecommend Ġword Ġrecognition Ġas Ġa Ġsafeguard Ġagainst Ġthis Ġand Ġbuild Ġupon ĠR NN - based Ġsemi - character Ġword Ġrecogn izers . ĠWe Ġdiscover Ġthat Ġwhen Ġused Ġas Ġa Ġdefense Ġmechanism , Ġthe Ġmost Ġaccurate Ġword Ġrecognition Ġmodels Ġare Ġnot Ġalways Ġthe Ġmost Ġrobust Ġagainst Ġadvers arial Ġattacks . ĠAdditionally , Ġwe Ġhighlight Ġthe Ġneed Ġto Ġcontrol Ġthe Ġsensitivity Ġof Ġthese Ġmodels Ġto Ġachieve Ġhigh Ġrobust ness . </s> Acknowled gements </s> The Ġauthors Ġare Ġgrateful Ġto ĠGraham ĠNe ub ig , ĠEdu ard ĠH ovy , ĠPaul ĠMichel , ĠMans i ĠGupta , Ġand ĠAnton ios ĠAn ast as opoulos Ġfor Ġsuggestions Ġand Ġfeedback .
Introduction Despite the rapid progress of deep learning techniques on diverse supervised learning tasks, these models remain brittle to subtle shifts in the data distribution. Even when the permissible changes are confined to barely-perceptible perturbations, training robust models remains an open challenge. Following the discovery that imperceptible attacks could cause image recognition models to misclassify examples BIBREF0 , a veritable sub-field has emerged in which authors iteratively propose attacks and countermeasures. For all the interest in adversarial computer vision, these attacks are rarely encountered outside of academic research. However, adversarial misspellings constitute a longstanding real-world problem. Spammers continually bombard email servers, subtly misspelling words in efforts to evade spam detection while preserving the emails' intended meaning BIBREF1 , BIBREF2 . As another example, programmatic censorship on the Internet has spurred communities to adopt similar methods to communicate surreptitiously BIBREF3 . In this paper, we focus on adversarially-chosen spelling mistakes in the context of text classification, addressing the following attack types: dropping, adding, and swapping internal characters within words. These perturbations are inspired by psycholinguistic studies BIBREF4 , BIBREF5 which demonstrated that humans can comprehend text altered by jumbling internal characters, provided that the first and last characters of each word remain unperturbed. First, in experiments addressing both BiLSTM and fine-tuned BERT models, comprising four different input formats: word-only, char-only, word+char, and word-piece BIBREF6 , we demonstrate that an adversary can degrade a classifier's performance to that achieved by random guessing. This requires altering just two characters per sentence. Such modifications might flip words either to a different word in the vocabulary or, more often, to the out-of-vocabulary token UNK. Consequently, adversarial edits can degrade a word-level model by transforming the informative words to UNK. Intuitively, one might suspect that word-piece and character-level models would be less susceptible to spelling attacks as they can make use of the residual word context. However, our experiments demonstrate that character and word-piece models are in fact more vulnerable. We show that this is due to the adversary's effective capacity for finer grained manipulations on these models. While against a word-level model, the adversary is mostly limited to UNK-ing words, against a word-piece or character-level model, each character-level add, drop, or swap produces a distinct input, providing the adversary with a greater set of options. Second, we evaluate first-line techniques including data augmentation and adversarial training, demonstrating that they offer only marginal benefits here, e.g., a BERT model achieving $90.3$ accuracy on a sentiment classification task, is degraded to $64.1$ by an adversarially-chosen 1-character swap in the sentence, which can only be restored to $69.2$ by adversarial training. Third (our primary contribution), we propose a task-agnostic defense, attaching a word recognition model that predicts each word in a sentence given a full sequence of (possibly misspelled) inputs. The word recognition model's outputs form the input to a downstream classification model. Our word recognition models build upon the RNN-based semi-character word recognition model due to BIBREF7 . While our word recognizers are trained on domain-specific text from the task at hand, they often predict UNK at test time, owing to the small domain-specific vocabulary. To handle unobserved and rare words, we propose several backoff strategies including falling back on a generic word recognizer trained on a larger corpus. Incorporating our defenses, BERT models subject to 1-character attacks are restored to $88.3$ , $81.1$ , $78.0$ accuracy for swap, drop, add attacks respectively, as compared to $69.2$ , $63.6$ , and $50.0$ for adversarial training Fourth, we offer a detailed qualitative analysis, demonstrating that a low word error rate alone is insufficient for a word recognizer to confer robustness on the downstream task. Additionally, we find that it is important that the recognition model supply few degrees of freedom to an attacker. We provide a metric to quantify this notion of sensitivity in word recognition models and study its relation to robustness empirically. Models with low sensitivity and word error rate are most robust. Related Work Several papers address adversarial attacks on NLP systems. Changes to text, whether word- or character-level, are all perceptible, raising some questions about what should rightly be considered an adversarial example BIBREF8 , BIBREF9 . BIBREF10 address the reading comprehension task, showing that by appending distractor sentences to the end of stories from the SQuAD dataset BIBREF11 , they could cause models to output incorrect answers. Inspired by this work, BIBREF12 demonstrate an attack that breaks entailment systems by replacing a single word with either a synonym or its hypernym. Recently, BIBREF13 investigated the problem of producing natural-seeming adversarial examples, noting that adversarial examples in NLP are often ungrammatical BIBREF14 . In related work on character-level attacks, BIBREF8 , BIBREF15 explored gradient-based methods to generate string edits to fool classification and translation systems, respectively. While their focus is on efficient methods for generating adversaries, ours is on improving the worst case adversarial performance. Similarly, BIBREF9 studied how synthetic and natural noise affects character-level machine translation. They considered structure invariant representations and adversarial training as defenses against such noise. Here, we show that an auxiliary word recognition model, which can be trained on unlabeled data, provides a strong defense. Spelling correction BIBREF16 is often viewed as a sub-task of grammatical error correction BIBREF17 , BIBREF18 . Classic methods rely on a source language model and a noisy channel model to find the most likely correction for a given word BIBREF19 , BIBREF20 . Recently, neural techniques have been applied to the task BIBREF7 , BIBREF21 , which model the context and orthography of the input together. Our work extends the ScRNN model of BIBREF7 . Robust Word Recognition To tackle character-level adversarial attacks, we introduce a simple two-stage solution, placing a word recognition model ( $W$ ) before the downstream classifier ( $C$ ). Under this scheme, all inputs are classified by the composed model $C \circ W$ . This modular approach, with $W$ and $C$ trained separately, offers several benefits: (i) we can deploy the same word recognition model for multiple downstream classification tasks/models; and (ii) we can train the word recognition model with larger unlabeled corpora. Against adversarial mistakes, two important factors govern the robustness of this combined model: $W$ 's accuracy in recognizing misspelled words and $W$ 's sensitivity to adversarial perturbations on the same input. We discuss these aspects in detail below. ScRNN with Backoff We now describe semi-character RNNs for word recognition, explain their limitations, and suggest techniques to improve them. Inspired by the psycholinguistic studies BIBREF5 , BIBREF4 , BIBREF7 proposed a semi-character based RNN (ScRNN) that processes a sentence of words with misspelled characters, predicting the correct words at each step. Let $s = \lbrace w_1, w_2, \dots , w_n\rbrace $ denote the input sentence, a sequence of constituent words $w_i$ . Each input word ( $w_i$ ) is represented by concatenating (i) a one hot vector of the first character ( $\mathbf {w_{i1}}$ ); (ii) a one hot representation of the last character ( $\mathbf {w_{il}}$ , where $l$ is the length of word $w_i$ ); and (iii) a bag of characters representation of the internal characters ( $\sum _{j=2}^{l-1}\mathbf {w_{ij}})$ . ScRNN treats the first and the last characters individually, and is agnostic to the ordering of the internal characters. Each word, represented accordingly, is then fed into a BiLSTM cell. At each sequence step, the training target is the correct corresponding word (output dimension equal to vocabulary size), and the model is optimized with cross-entropy loss. While BIBREF7 demonstrate strong word recognition performance, a drawback of their evaluation setup is that they only attack and evaluate on the subset of words that are a part of their training vocabulary. In such a setting, the word recognition performance is unreasonably dependent on the chosen vocabulary size. In principle, one can design models to predict (correctly) only a few chosen words, and ignore the remaining majority and still reach 100% accuracy. For the adversarial setting, rare and unseen words in the wild are particularly critical, as they provide opportunities for the attackers. A reliable word-recognizer should handle these cases gracefully. Below, we explore different ways to back off when the ScRNN predicts UNK (a frequent outcome for rare and unseen words): Pass-through: word-recognizer passes on the (possibly misspelled) word as is. Backoff to neutral word: Alternatively, noting that passing $\colorbox {gray!20}{\texttt {UNK}}$ -predicted words through unchanged exposes the downstream model to potentially corrupted text, we consider backing off to a neutral word like `a', which has a similar distribution across classes. Backoff to background model: We also consider falling back upon a more generic word recognition model trained upon a larger, less-specialized corpus whenever the foreground word recognition model predicts UNK. Figure 1 depicts this scenario pictorially. Empirically, we find that the background model (by itself) is less accurate, because of the large number of words it is trained to predict. Thus, it is best to train a precise foreground model on an in-domain corpus and focus on frequent words, and then to resort to a general-purpose background model for rare and unobserved words. Next, we delineate our second consideration for building robust word-recognizers. Model Sensitivity In computer vision, an important factor determining the success of an adversary is the norm constraint on the perturbations allowed to an image ( $|| \bf x - \bf x^{\prime }||_{\infty } < \epsilon $ ). Higher values of $\epsilon $ lead to a higher chance of mis-classification for at least one $\bf x^{\prime }$ . Defense methods such as quantization BIBREF22 and thermometer encoding BIBREF23 try to reduce the space of perturbations available to the adversary by making the model invariant to small changes in the input. In NLP, we often get such invariance for free, e.g., for a word-level model, most of the perturbations produced by our character-level adversary lead to an UNK at its input. If the model is robust to the presence of these UNK tokens, there is little room for an adversary to manipulate it. Character-level models, on the other hand, despite their superior performance in many tasks, do not enjoy such invariance. This characteristic invariance could be exploited by an attacker. Thus, to limit the number of different inputs to the classifier, we wish to reduce the number of distinct word recognition outputs that an attacker can induce, not just the number of words on which the model is “fooled”. We denote this property of a model as its sensitivity. We can quantify this notion for a word recognition system $W$ as the expected number of unique outputs it assigns to a set of adversarial perturbations. Given a sentence $s$ from the set of sentences $\mathcal {S}$ , let $A(s) = {s_1}^{\prime } , {s_2}^{\prime }, \dots , {s_n}^{\prime }$ denote the set of $n$ perturbations to it under attack type $A$ , and let $V$ be the function that maps strings to an input representation for the downstream classifier. For a word level model, $V$ would transform sentences to a sequence of word ids, mapping OOV words to the same UNK ID. Whereas, for a char (or word+char, word-piece) model, $V$ would map inputs to a sequence of character IDs. Formally, sensitivity is defined as $$S_{W,V}^A=\mathbb {E}_{s}\left[\frac{\#_{u}(V \circ W({s_1}^{\prime }), \dots , V \circ W({s_n}^{\prime }))}{n}\right] ,$$   (Eq. 12) where $V \circ W (s_i)$ returns the input representation (of the downstream classifier) for the output string produced by the word-recognizer $W$ using $s_i$ and $\#_{u}(\cdot )$ counts the number of unique arguments. Intuitively, we expect a high value of $S_{W, V}^A$ to lead to a lower robustness of the downstream classifier, since the adversary has more degrees of freedom to attack the classifier. Thus, when using word recognition as a defense, it is prudent to design a low sensitivity system with a low error rate. However, as we will demonstrate, there is often a trade-off between sensitivity and error rate. Synthesizing Adversarial Attacks Suppose we are given a classifier $C: \mathcal {S} \rightarrow \mathcal {Y}$ which maps natural language sentences $s \in \mathcal {S}$ to a label from a predefined set $y \in \mathcal {Y}$ . An adversary for this classifier is a function $A$ which maps a sentence $s$ to its perturbed versions $\lbrace s^{\prime }_1, s^{\prime }_2, \ldots , s^{\prime }_{n}\rbrace $ such that each $s^{\prime }_i$ is close to $s$ under some notion of distance between sentences. We define the robustness of classifier $C$ to the adversary $A$ as: $$R_{C,A} = \mathbb {E}_s \left[\min _{s^{\prime } \in A(s)} \mathbb {1}[C(s^{\prime }) = y]\right],$$   (Eq. 14) where $y$ represents the ground truth label for $s$ . In practice, a real-world adversary may only be able to query the classifier a few times, hence $R_{C,A}$ represents the worst-case adversarial performance of $C$ . Methods for generating adversarial examples, such as HotFlip BIBREF8 , focus on efficient algorithms for searching the $\min $ above. Improving $R_{C,A}$ would imply better robustness against all these methods. We explore adversaries which perturb sentences with four types of character-level edits: (1) Swap: swapping two adjacent internal characters of a word. (2) Drop: removing an internal character of a word. (3) Keyboard: substituting an internal character with adjacent characters of QWERTY keyboard (4) Add: inserting a new character internally in a word. In line with the psycholinguistic studies BIBREF5 , BIBREF4 , to ensure that the perturbations do not affect human ability to comprehend the sentence, we only allow the adversary to edit the internal characters of a word, and not edit stopwords or words shorter than 4 characters. For 1-character attacks, we try all possible perturbations listed above until we find an adversary that flips the model prediction. For 2-character attacks, we greedily fix the edit which had the least confidence among 1-character attacks, and then try all the allowed perturbations on the remaining words. Higher order attacks can be performed in a similar manner. The greedy strategy reduces the computation required to obtain higher order attacks, but also means that the robustness score is an upper bound on the true robustness of the classifier. Experiments and Results In this section, we first discuss our experiments on the word recognition systems. Word Error Correction Data: We evaluate the spell correctors from § "Robust Word Recognition" on movie reviews from the Stanford Sentiment Treebank (SST) BIBREF24 . The SST dataset consists of 8544 movie reviews, with a vocabulary of over 16K words. As a background corpus, we use the IMDB movie reviews BIBREF25 , which contain 54K movie reviews, and a vocabulary of over 78K words. The two datasets do not share any reviews in common. The spell-correction models are evaluated on their ability to correct misspellings. The test setting consists of reviews where each word (with length $\ge 4$ , barring stopwords) is attacked by one of the attack types (from swap, add, drop and keyboard attacks). In the all attack setting, we mix all attacks by randomly choosing one for each word. This most closely resembles a real world attack setting. In addition to our word recognition models, we also compare to After The Deadline (ATD), an open-source spell corrector. We found ATD to be the best freely-available corrector. We refer the reader to BIBREF7 for comparisons of ScRNN to other anonymized commercial spell checkers. For the ScRNN model, we use a single-layer Bi-LSTM with a hidden dimension size of 50. The input representation consists of 198 dimensions, which is thrice the number of unique characters (66) in the vocabulary. We cap the vocabulary size to 10K words, whereas we use the entire vocabulary of 78470 words when we backoff to the background model. For training these networks, we corrupt the movie reviews according to all attack types, i.e., applying one of the 4 attack types to each word, and trying to reconstruct the original words via cross entropy loss. We calculate the word error rates (WER) of each of the models for different attacks and present our findings in Table 2 . Note that ATD incorrectly predicts $11.2$ words for every 100 words (in the `all' setting), whereas, all of the backoff variations of the ScRNN reconstruct better. The most accurate variant involves backing off to the background model, resulting in a low error rate of $6.9\%$ , leading to the best performance on word recognition. This is a $32\%$ relative error reduction compared to the vanilla ScRNN model with a pass-through backoff strategy. We can attribute the improved performance to the fact that there are $5.25\%$ words in the test corpus that are unseen in the training corpus, and are thus only recoverable by backing off to a larger corpus. Notably, only training on the larger background corpus does worse, at $8.7\%$ , since the distribution of word frequencies is different in the background corpus compared to the foreground corpus. Robustness to adversarial attacks We use sentiment analysis and paraphrase detection as downstream tasks, as for these two tasks, 1-2 character edits do not change the output labels. For sentiment classification, we systematically study the effect of character-level adversarial attacks on two architectures and four different input formats. The first architecture encodes the input sentence into a sequence of embeddings, which are then sequentially processed by a BiLSTM. The first and last states of the BiLSTM are then used by the softmax layer to predict the sentiment of the input. We consider three input formats for this architecture: (1) Word-only: where the input words are encoded using a lookup table; (2) Char-only: where the input words are encoded using a separate single-layered BiLSTM over their characters; and (3) Word $+$ Char: where the input words are encoded using a concatenation of (1) and (2) . The second architecture uses the fine-tuned BERT model BIBREF26 , with an input format of word-piece tokenization. This model has recently set a new state-of-the-art on several NLP benchmarks, including the sentiment analysis task we consider here. All models are trained and evaluated on the binary version of the sentence-level Stanford Sentiment Treebank BIBREF24 dataset with only positive and negative reviews. We also consider the task of paraphrase detection. Here too, we make use of the fine-tuned BERT BIBREF26 , which is trained and evaluated on the Microsoft Research Paraphrase Corpus (MRPC) BIBREF27 . Two common methods for dealing with adversarial examples include: (1) data augmentation (DA) BIBREF28 ; and (2) adversarial training (Adv) BIBREF29 . In DA, the trained model is fine-tuned after augmenting the training set with an equal number of examples randomly attacked with a 1-character edit. In Adv, the trained model is fine-tuned with additional adversarial examples (selected at random) that produce incorrect predictions from the current-state classifier. The process is repeated iteratively, generating and adding newer adversarial examples from the updated classifier model, until the adversarial accuracy on dev set stops improving. In Table 3 , we examine the robustness of the sentiment models under each attack and defense method. In the absence of any attack or defense, BERT (a word-piece model) performs the best ( $90.3\%$ ) followed by word+char models ( $80.5\%$ ), word-only models ( $79.2\%$ ) and then char-only models ( $70.3\%$ ). However, even single-character attacks (chosen adversarially) can be catastrophic, resulting in a significantly degraded performance of $46\%$ , $57\%$ , $59\%$ and $33\%$ , respectively under the `all' setting. Intuitively, one might suppose that word-piece and character-level models would be more robust to such attacks given they can make use of the remaining context. However, we find that they are the more susceptible. To see why, note that the word `beautiful' can only be altered in a few ways for word-only models, either leading to an UNK or an existing vocabulary word, whereas, word-piece and character-only models treat each unique character combination differently. This provides more variations that an attacker can exploit. Following similar reasoning, add and key attacks pose a greater threat than swap and drop attacks. The robustness of different models can be ordered as word-only $>$ word+char $>$ char-only $\sim $ word-piece, and the efficacy of different attacks as add $>$ key $>$ drop $>$ swap. Next, we scrutinize the effectiveness of defense methods when faced against adversarially chosen attacks. Clearly from table 3 , DA and Adv are not effective in this case. We observed that despite a low training error, these models were not able to generalize to attacks on newer words at test time. ATD spell corrector is the most effective on keyboard attacks, but performs poorly on other attack types, particularly the add attack strategy. The ScRNN model with pass-through backoff offers better protection, bringing back the adversarial accuracy within $5\%$ range for the swap attack. It is also effective under other attack classes, and can mitigate the adversarial effect in word-piece models by $21\%$ , character-only models by $19\%$ , and in word, and word+char models by over $4.5\%$ . This suggests that the direct training signal of word error correction is more effective than the indirect signal of sentiment classification available to DA and Adv for model robustness. We observe additional gains by using background models as a backoff alternative, because of its lower word error rate (WER), especially, under the swap and drop attacks. However, these gains do not consistently translate in all other settings, as lower WER is necessary but not sufficient. Besides lower error rate, we find that a solid defense should furnish the attacker the fewest options to attack, i.e. it should have a low sensitivity. As we shall see in section § "Understanding Model Sensitivity" , the backoff neutral variation has the lowest sensitivity due to mapping UNK predictions to a fixed neutral word. Thus, it results in the highest robustness on most of the attack types for all four model classes. Table 4 shows the accuracy of BERT on 200 examples from the dev set of the MRPC paraphrase detection task under various attack and defense settings. We re-trained the ScRNN model variants on the MRPC training set for these experiments. Again, we find that simple 1-2 character attacks can bring down the accuracy of BERT significantly ( $89\%$ to $31\%$ ). Word recognition models can provide an effective defense, with both our pass-through and neutral variants recovering most of the accuracy. While the neutral backoff model is effective on 2-char attacks, it hurts performance in the no attack setting, since it incorrectly modifies certain correctly spelled entity names. Since the two variants are already effective, we did not train a background model for this task. Understanding Model Sensitivity To study model sensitivity, for each sentence, we perturb one randomly-chosen word and replace it with all possible perturbations under a given attack type. The resulting set of perturbed sentences is then fed to the word recognizer (whose sensitivity is to be estimated). As described in equation 12 , we count the number of unique predictions from the output sentences. Two corrections are considered unique if they are mapped differently by the downstream classifier. The neutral backoff variant has the lowest sensitivity (Table 5 ). This is expected, as it returns a fixed neutral word whenever the ScRNN predicts an UNK, therefore reducing the number of unique outputs it predicts. Open vocabulary (i.e. char-only, word+char, word-piece) downstream classifiers consider every unique combination of characters differently, whereas word-only classifiers internally treat all out of vocabulary (OOV) words alike. Hence, for char-only, word+char, and word-piece models, the pass-through version is more sensitive than the background variant, as it passes words as is (and each combination is considered uniquely). However, for word-only models, pass-through is less sensitive as all the OOV character combinations are rendered identical. Ideally, a preferred defense is one with low sensitivity and word error rate. In practice, however, we see that a low error rate often comes at the cost of sensitivity. We see this trade-off in Figure 2 , where we plot WER and sensitivity on the two axes, and depict the robustness when using different backoff variants. Generally, sensitivity is the more dominant factor out of the two, as the error rates of the considered variants are reasonably low. We verify if the sentiment (of the reviews) is preserved with char-level attacks. In a human study with 50 attacked (and subsequently misclassified), and 50 unchanged reviews, it was noted that 48 and 49, respectively, preserved the sentiment. Conclusion As character and word-piece inputs become commonplace in modern NLP pipelines, it is worth highlighting the vulnerability they add. We show that minimally-doctored attacks can bring down accuracy of classifiers to random guessing. We recommend word recognition as a safeguard against this and build upon RNN-based semi-character word recognizers. We discover that when used as a defense mechanism, the most accurate word recognition models are not always the most robust against adversarial attacks. Additionally, we highlight the need to control the sensitivity of these models to achieve high robustness. Acknowledgements The authors are grateful to Graham Neubig, Eduard Hovy, Paul Michel, Mansi Gupta, and Antonios Anastasopoulos for suggestions and feedback.
How do the backoff strategies work?
ba539cab80d25c3e20f39644415ed48b9e4e4185
How Ġdo Ġthe Ġback off Ġstrategies Ġwork ?
In Ġpass - through , Ġthe Ġrecogn izer Ġpasses Ġon Ġthe Ġpossibly Ġmiss pelled Ġword , Ġback off Ġto Ġneutral Ġword Ġbacks Ġoff Ġto Ġa Ġword Ġwith Ġsimilar Ġdistribution Ġacross Ġclasses Ġand Ġback off Ġto Ġbackground Ġmodel Ġbacks Ġoff Ġto Ġa Ġmore Ġgeneric Ġword Ġrecognition Ġmodel Ġtrained Ġwith Ġlarger Ġand Ġless Ġspecialized Ġcorpus .
<s> How Ġdo Ġthe Ġback off Ġstrategies Ġwork ? </s> Introduction </s> Despite Ġthe Ġrapid Ġprogress Ġof Ġdeep Ġlearning Ġtechniques Ġon Ġdiverse Ġsupervised Ġlearning Ġtasks , Ġthese Ġmodels Ġremain Ġbrittle Ġto Ġsubtle Ġshifts Ġin Ġthe Ġdata Ġdistribution . ĠEven Ġwhen Ġthe Ġpermissible Ġchanges Ġare Ġconfined Ġto Ġbarely - per cept ible Ġpert urb ations , Ġtraining Ġrobust Ġmodels Ġremains Ġan Ġopen Ġchallenge . ĠFollowing Ġthe Ġdiscovery Ġthat Ġimper cept ible Ġattacks Ġcould Ġcause Ġimage Ġrecognition Ġmodels Ġto Ġmis class ify Ġexamples ĠB IB REF 0 Ġ, Ġa Ġver itable Ġsub - field Ġhas Ġemerged Ġin Ġwhich Ġauthors Ġiter atively Ġpropose Ġattacks Ġand Ġcounter measures . </s> For Ġall Ġthe Ġinterest Ġin Ġadvers arial Ġcomputer Ġvision , Ġthese Ġattacks Ġare Ġrarely Ġencountered Ġoutside Ġof Ġacademic Ġresearch . ĠHowever , Ġadvers arial Ġmiss pell ings Ġconstitute Ġa Ġlongstanding Ġreal - world Ġproblem . ĠSp ammers Ġcontinually Ġbombard Ġemail Ġservers , Ġsubtly Ġmiss pelling Ġwords Ġin Ġefforts Ġto Ġevade Ġspam Ġdetection Ġwhile Ġpreserving Ġthe Ġemails ' Ġintended Ġmeaning ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ. ĠAs Ġanother Ġexample , Ġprogram matic Ġcensorship Ġon Ġthe ĠInternet Ġhas Ġspurred Ġcommunities Ġto Ġadopt Ġsimilar Ġmethods Ġto Ġcommunicate Ġsur re pt it iously ĠB IB REF 3 Ġ. </s> In Ġthis Ġpaper , Ġwe Ġfocus Ġon Ġadvers ar ially - ch osen Ġspelling Ġmistakes Ġin Ġthe Ġcontext Ġof Ġtext Ġclassification , Ġaddressing Ġthe Ġfollowing Ġattack Ġtypes : Ġdropping , Ġadding , Ġand Ġswapping Ġinternal Ġcharacters Ġwithin Ġwords . ĠThese Ġpert urb ations Ġare Ġinspired Ġby Ġpsych ol ingu istic Ġstudies ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġwhich Ġdemonstrated Ġthat Ġhumans Ġcan Ġcomprehend Ġtext Ġaltered Ġby Ġj umbling Ġinternal Ġcharacters , Ġprovided Ġthat Ġthe Ġfirst Ġand Ġlast Ġcharacters Ġof Ġeach Ġword Ġremain Ġun pert urbed . </s> First , Ġin Ġexperiments Ġaddressing Ġboth ĠBi L ST M Ġand Ġfine - tun ed ĠB ERT Ġmodels , Ġcomprising Ġfour Ġdifferent Ġinput Ġformats : Ġword - only , Ġchar - only , Ġword + char , Ġand Ġword - piece ĠB IB REF 6 Ġ, Ġwe Ġdemonstrate Ġthat Ġan Ġadversary Ġcan Ġdegrade Ġa Ġclass ifier 's Ġperformance Ġto Ġthat Ġachieved Ġby Ġrandom Ġguessing . ĠThis Ġrequires Ġaltering Ġjust Ġtwo Ġcharacters Ġper Ġsentence . ĠSuch Ġmodifications Ġmight Ġflip Ġwords Ġeither Ġto Ġa Ġdifferent Ġword Ġin Ġthe Ġvocabulary Ġor , Ġmore Ġoften , Ġto Ġthe Ġout - of - voc abulary Ġtoken ĠUN K . ĠConsequently , Ġadvers arial Ġedits Ġcan Ġdegrade Ġa Ġword - level Ġmodel Ġby Ġtransforming Ġthe Ġinformative Ġwords Ġto ĠUN K . ĠInt uitive ly , Ġone Ġmight Ġsuspect Ġthat Ġword - piece Ġand Ġcharacter - level Ġmodels Ġwould Ġbe Ġless Ġsusceptible Ġto Ġspelling Ġattacks Ġas Ġthey Ġcan Ġmake Ġuse Ġof Ġthe Ġresidual Ġword Ġcontext . ĠHowever , Ġour Ġexperiments Ġdemonstrate Ġthat Ġcharacter Ġand Ġword - piece Ġmodels Ġare Ġin Ġfact Ġmore Ġvulnerable . ĠWe Ġshow Ġthat Ġthis Ġis Ġdue Ġto Ġthe Ġadversary 's Ġeffective Ġcapacity Ġfor Ġfiner Ġgr ained Ġmanip ulations Ġon Ġthese Ġmodels . ĠWhile Ġagainst Ġa Ġword - level Ġmodel , Ġthe Ġadversary Ġis Ġmostly Ġlimited Ġto ĠUN K - ing Ġwords , Ġagainst Ġa Ġword - piece Ġor Ġcharacter - level Ġmodel , Ġeach Ġcharacter - level Ġadd , Ġdrop , Ġor Ġswap Ġproduces Ġa Ġdistinct Ġinput , Ġproviding Ġthe Ġadversary Ġwith Ġa Ġgreater Ġset Ġof Ġoptions . </s> Second , Ġwe Ġevaluate Ġfirst - line Ġtechniques Ġincluding Ġdata Ġaug mentation Ġand Ġadvers arial Ġtraining , Ġdemonstrating Ġthat Ġthey Ġoffer Ġonly Ġmarginal Ġbenefits Ġhere , Ġe . g ., Ġa ĠB ERT Ġmodel Ġachieving Ġ$ 90 . 3 $ Ġaccuracy Ġon Ġa Ġsentiment Ġclassification Ġtask , Ġis Ġdegraded Ġto Ġ$ 64 . 1 $ Ġby Ġan Ġadvers ar ially - ch osen Ġ1 - character Ġswap Ġin Ġthe Ġsentence , Ġwhich Ġcan Ġonly Ġbe Ġrestored Ġto Ġ$ 69 . 2 $ Ġby Ġadvers arial Ġtraining . </s> Third Ġ( our Ġprimary Ġcontribution ), Ġwe Ġpropose Ġa Ġtask - agn ostic Ġdefense , Ġattaching Ġa Ġword Ġrecognition Ġmodel Ġthat Ġpredicts Ġeach Ġword Ġin Ġa Ġsentence Ġgiven Ġa Ġfull Ġsequence Ġof Ġ( possibly Ġmiss pelled ) Ġinputs . ĠThe Ġword Ġrecognition Ġmodel 's Ġoutputs Ġform Ġthe Ġinput Ġto Ġa Ġdownstream Ġclassification Ġmodel . ĠOur Ġword Ġrecognition Ġmodels Ġbuild Ġupon Ġthe ĠR NN - based Ġsemi - character Ġword Ġrecognition Ġmodel Ġdue Ġto ĠB IB REF 7 Ġ. ĠWhile Ġour Ġword Ġrecogn izers Ġare Ġtrained Ġon Ġdomain - specific Ġtext Ġfrom Ġthe Ġtask Ġat Ġhand , Ġthey Ġoften Ġpredict ĠUN K Ġat Ġtest Ġtime , Ġowing Ġto Ġthe Ġsmall Ġdomain - specific Ġvocabulary . ĠTo Ġhandle Ġunob served Ġand Ġrare Ġwords , Ġwe Ġpropose Ġseveral Ġback off Ġstrategies Ġincluding Ġfalling Ġback Ġon Ġa Ġgeneric Ġword Ġrecogn izer Ġtrained Ġon Ġa Ġlarger Ġcorpus . ĠInc orpor ating Ġour Ġdefenses , ĠB ERT Ġmodels Ġsubject Ġto Ġ1 - character Ġattacks Ġare Ġrestored Ġto Ġ$ 88 . 3 $ Ġ, Ġ$ 81 . 1 $ Ġ, Ġ$ 78 . 0 $ Ġaccuracy Ġfor Ġswap , Ġdrop , Ġadd Ġattacks Ġrespectively , Ġas Ġcompared Ġto Ġ$ 69 . 2 $ Ġ, Ġ$ 63 . 6 $ Ġ, Ġand Ġ$ 50 . 0 $ Ġfor Ġadvers arial Ġtraining </s> Fourth , Ġwe Ġoffer Ġa Ġdetailed Ġqualitative Ġanalysis , Ġdemonstrating Ġthat Ġa Ġlow Ġword Ġerror Ġrate Ġalone Ġis Ġinsufficient Ġfor Ġa Ġword Ġrecogn izer Ġto Ġconfer Ġrobust ness Ġon Ġthe Ġdownstream Ġtask . ĠAdditionally , Ġwe Ġfind Ġthat Ġit Ġis Ġimportant Ġthat Ġthe Ġrecognition Ġmodel Ġsupply Ġfew Ġdegrees Ġof Ġfreedom Ġto Ġan Ġattacker . ĠWe Ġprovide Ġa Ġmetric Ġto Ġquantify Ġthis Ġnotion Ġof Ġsensitivity Ġin Ġword Ġrecognition Ġmodels Ġand Ġstudy Ġits Ġrelation Ġto Ġrobust ness Ġempir ically . ĠModels Ġwith Ġlow Ġsensitivity Ġand Ġword Ġerror Ġrate Ġare Ġmost Ġrobust . </s> Related ĠWork </s> Several Ġpapers Ġaddress Ġadvers arial Ġattacks Ġon ĠN LP Ġsystems . ĠChanges Ġto Ġtext , Ġwhether Ġword - Ġor Ġcharacter - level , Ġare Ġall Ġpercept ible , Ġraising Ġsome Ġquestions Ġabout Ġwhat Ġshould Ġrightly Ġbe Ġconsidered Ġan Ġadvers arial Ġexample ĠB IB REF 8 Ġ, ĠB IB REF 9 Ġ. ĠB IB REF 10 Ġaddress Ġthe Ġreading Ġcomprehension Ġtask , Ġshowing Ġthat Ġby Ġapp ending Ġdistract or Ġsentences Ġto Ġthe Ġend Ġof Ġstories Ġfrom Ġthe ĠS Qu AD Ġdataset ĠB IB REF 11 Ġ, Ġthey Ġcould Ġcause Ġmodels Ġto Ġoutput Ġincorrect Ġanswers . ĠInspired Ġby Ġthis Ġwork , ĠB IB REF 12 Ġdemonstrate Ġan Ġattack Ġthat Ġbreaks Ġentail ment Ġsystems Ġby Ġreplacing Ġa Ġsingle Ġword Ġwith Ġeither Ġa Ġsyn onym Ġor Ġits Ġhyper ny m . ĠRecently , ĠB IB REF 13 Ġinvestigated Ġthe Ġproblem Ġof Ġproducing Ġnatural - se eming Ġadvers arial Ġexamples , Ġnoting Ġthat Ġadvers arial Ġexamples Ġin ĠN LP Ġare Ġoften Ġun gram matical ĠB IB REF 14 Ġ. </s> In Ġrelated Ġwork Ġon Ġcharacter - level Ġattacks , ĠB IB REF 8 Ġ, ĠB IB REF 15 Ġexplored Ġgradient - based Ġmethods Ġto Ġgenerate Ġstring Ġedits Ġto Ġfool Ġclassification Ġand Ġtranslation Ġsystems , Ġrespectively . ĠWhile Ġtheir Ġfocus Ġis Ġon Ġefficient Ġmethods Ġfor Ġgenerating Ġadversaries , Ġours Ġis Ġon Ġimproving Ġthe Ġworst Ġcase Ġadvers arial Ġperformance . ĠSimilarly , ĠB IB REF 9 Ġstudied Ġhow Ġsynthetic Ġand Ġnatural Ġnoise Ġaffects Ġcharacter - level Ġmachine Ġtranslation . ĠThey Ġconsidered Ġstructure Ġinvari ant Ġrepresentations Ġand Ġadvers arial Ġtraining Ġas Ġdefenses Ġagainst Ġsuch Ġnoise . ĠHere , Ġwe Ġshow Ġthat Ġan Ġauxiliary Ġword Ġrecognition Ġmodel , Ġwhich Ġcan Ġbe Ġtrained Ġon Ġunl abel ed Ġdata , Ġprovides Ġa Ġstrong Ġdefense . </s> Spe lling Ġcorrection ĠB IB REF 16 Ġis Ġoften Ġviewed Ġas Ġa Ġsub - task Ġof Ġgram matical Ġerror Ġcorrection ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ. ĠClassic Ġmethods Ġrely Ġon Ġa Ġsource Ġlanguage Ġmodel Ġand Ġa Ġnoisy Ġchannel Ġmodel Ġto Ġfind Ġthe Ġmost Ġlikely Ġcorrection Ġfor Ġa Ġgiven Ġword ĠB IB REF 19 Ġ, ĠB IB REF 20 Ġ. ĠRecently , Ġneural Ġtechniques Ġhave Ġbeen Ġapplied Ġto Ġthe Ġtask ĠB IB REF 7 Ġ, ĠB IB REF 21 Ġ, Ġwhich Ġmodel Ġthe Ġcontext Ġand Ġorth ography Ġof Ġthe Ġinput Ġtogether . ĠOur Ġwork Ġextends Ġthe ĠSc R NN Ġmodel Ġof ĠB IB REF 7 Ġ. </s> Rob ust ĠWord ĠRecogn ition </s> To Ġtackle Ġcharacter - level Ġadvers arial Ġattacks , Ġwe Ġintroduce Ġa Ġsimple Ġtwo - stage Ġsolution , Ġplacing Ġa Ġword Ġrecognition Ġmodel Ġ( Ġ$ W $ Ġ) Ġbefore Ġthe Ġdownstream Ġclass ifier Ġ( Ġ$ C $ Ġ). ĠUnder Ġthis Ġscheme , Ġall Ġinputs Ġare Ġclassified Ġby Ġthe Ġcomposed Ġmodel Ġ$ C Ġ\ circ ĠW $ Ġ. ĠThis Ġmodular Ġapproach , Ġwith Ġ$ W $ Ġand Ġ$ C $ Ġtrained Ġseparately , Ġoffers Ġseveral Ġbenefits : Ġ( i ) Ġwe Ġcan Ġdeploy Ġthe Ġsame Ġword Ġrecognition Ġmodel Ġfor Ġmultiple Ġdownstream Ġclassification Ġtasks / models ; Ġand Ġ( ii ) Ġwe Ġcan Ġtrain Ġthe Ġword Ġrecognition Ġmodel Ġwith Ġlarger Ġunl abel ed Ġcorpor a . </s> Against Ġadvers arial Ġmistakes , Ġtwo Ġimportant Ġfactors Ġgovern Ġthe Ġrobust ness Ġof Ġthis Ġcombined Ġmodel : Ġ$ W $ Ġ' s Ġaccuracy Ġin Ġrecognizing Ġmiss pelled Ġwords Ġand Ġ$ W $ Ġ' s Ġsensitivity Ġto Ġadvers arial Ġpert urb ations Ġon Ġthe Ġsame Ġinput . ĠWe Ġdiscuss Ġthese Ġaspects Ġin Ġdetail Ġbelow . </s> Sc R NN Ġwith ĠBack off </s> We Ġnow Ġdescribe Ġsemi - character ĠR NN s Ġfor Ġword Ġrecognition , Ġexplain Ġtheir Ġlimitations , Ġand Ġsuggest Ġtechniques Ġto Ġimprove Ġthem . </s> Insp ired Ġby Ġthe Ġpsych ol ingu istic Ġstudies ĠB IB REF 5 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 7 Ġproposed Ġa Ġsemi - character Ġbased ĠR NN Ġ( Sc R NN ) Ġthat Ġprocesses Ġa Ġsentence Ġof Ġwords Ġwith Ġmiss pelled Ġcharacters , Ġpredicting Ġthe Ġcorrect Ġwords Ġat Ġeach Ġstep . ĠLet Ġ$ s Ġ= Ġ\ l brace Ġw _ 1 , Ġw _ 2 , Ġ\ d ots Ġ, Ġw _ n \ r brace Ġ$ Ġdenote Ġthe Ġinput Ġsentence , Ġa Ġsequence Ġof Ġconstituent Ġwords Ġ$ w _ i $ Ġ. ĠEach Ġinput Ġword Ġ( Ġ$ w _ i $ Ġ) Ġis Ġrepresented Ġby Ġconc aten ating Ġ( i ) Ġa Ġone Ġhot Ġvector Ġof Ġthe Ġfirst Ġcharacter Ġ( Ġ$\ math bf Ġ{ w _{ i 1 }} $ Ġ); Ġ( ii ) Ġa Ġone Ġhot Ġrepresentation Ġof Ġthe Ġlast Ġcharacter Ġ( Ġ$\ math bf Ġ{ w _{ il }} $ Ġ, Ġwhere Ġ$ l $ Ġis Ġthe Ġlength Ġof Ġword Ġ$ w _ i $ Ġ); Ġand Ġ( iii ) Ġa Ġbag Ġof Ġcharacters Ġrepresentation Ġof Ġthe Ġinternal Ġcharacters Ġ( Ġ$\ sum Ġ_ { j = 2 } ^{ l - 1 }\ math bf Ġ{ w _{ ij }} ) $ Ġ. ĠSc R NN Ġtreats Ġthe Ġfirst Ġand Ġthe Ġlast Ġcharacters Ġindividually , Ġand Ġis Ġag nostic Ġto Ġthe Ġordering Ġof Ġthe Ġinternal Ġcharacters . ĠEach Ġword , Ġrepresented Ġaccordingly , Ġis Ġthen Ġfed Ġinto Ġa ĠBi L ST M Ġcell . ĠAt Ġeach Ġsequence Ġstep , Ġthe Ġtraining Ġtarget Ġis Ġthe Ġcorrect Ġcorresponding Ġword Ġ( output Ġdimension Ġequal Ġto Ġvocabulary Ġsize ), Ġand Ġthe Ġmodel Ġis Ġoptimized Ġwith Ġcross - ent ropy Ġloss . </s> While ĠB IB REF 7 Ġdemonstrate Ġstrong Ġword Ġrecognition Ġperformance , Ġa Ġdrawback Ġof Ġtheir Ġevaluation Ġsetup Ġis Ġthat Ġthey Ġonly Ġattack Ġand Ġevaluate Ġon Ġthe Ġsubset Ġof Ġwords Ġthat Ġare Ġa Ġpart Ġof Ġtheir Ġtraining Ġvocabulary . ĠIn Ġsuch Ġa Ġsetting , Ġthe Ġword Ġrecognition Ġperformance Ġis Ġunre ason ably Ġdependent Ġon Ġthe Ġchosen Ġvocabulary Ġsize . ĠIn Ġprinciple , Ġone Ġcan Ġdesign Ġmodels Ġto Ġpredict Ġ( correct ly ) Ġonly Ġa Ġfew Ġchosen Ġwords , Ġand Ġignore Ġthe Ġremaining Ġmajority Ġand Ġstill Ġreach Ġ100 % Ġaccuracy . ĠFor Ġthe Ġadvers arial Ġsetting , Ġrare Ġand Ġunseen Ġwords Ġin Ġthe Ġwild Ġare Ġparticularly Ġcritical , Ġas Ġthey Ġprovide Ġopportunities Ġfor Ġthe Ġattackers . ĠA Ġreliable Ġword - recogn izer Ġshould Ġhandle Ġthese Ġcases Ġgrace fully . ĠBelow , Ġwe Ġexplore Ġdifferent Ġways Ġto Ġback Ġoff Ġwhen Ġthe ĠSc R NN Ġpredicts ĠUN K Ġ( a Ġfrequent Ġoutcome Ġfor Ġrare Ġand Ġunseen Ġwords ): </s> Pass - through : Ġword - recogn izer Ġpasses Ġon Ġthe Ġ( possibly Ġmiss pelled ) Ġword Ġas Ġis . </s> Back off Ġto Ġneutral Ġword : ĠAlternatively , Ġnoting Ġthat Ġpassing Ġ$\ color box Ġ{ gray ! 20 }{ \ text tt Ġ{ UN K }} $ Ġ- pred icted Ġwords Ġthrough Ġunchanged Ġexposes Ġthe Ġdownstream Ġmodel Ġto Ġpotentially Ġcorrupted Ġtext , Ġwe Ġconsider Ġbacking Ġoff Ġto Ġa Ġneutral Ġword Ġlike Ġ` a ', Ġwhich Ġhas Ġa Ġsimilar Ġdistribution Ġacross Ġclasses . </s> Back off Ġto Ġbackground Ġmodel : ĠWe Ġalso Ġconsider Ġfalling Ġback Ġupon Ġa Ġmore Ġgeneric Ġword Ġrecognition Ġmodel Ġtrained Ġupon Ġa Ġlarger , Ġless - special ized Ġcorpus Ġwhenever Ġthe Ġforeground Ġword Ġrecognition Ġmodel Ġpredicts ĠUN K . ĠFigure Ġ1 Ġdepicts Ġthis Ġscenario Ġpict or ially . </s> E mp ir ically , Ġwe Ġfind Ġthat Ġthe Ġbackground Ġmodel Ġ( by Ġitself ) Ġis Ġless Ġaccurate , Ġbecause Ġof Ġthe Ġlarge Ġnumber Ġof Ġwords Ġit Ġis Ġtrained Ġto Ġpredict . ĠThus , Ġit Ġis Ġbest Ġto Ġtrain Ġa Ġprecise Ġforeground Ġmodel Ġon Ġan Ġin - domain Ġcorpus Ġand Ġfocus Ġon Ġfrequent Ġwords , Ġand Ġthen Ġto Ġresort Ġto Ġa Ġgeneral - purpose Ġbackground Ġmodel Ġfor Ġrare Ġand Ġunob served Ġwords . ĠNext , Ġwe Ġdeline ate Ġour Ġsecond Ġconsideration Ġfor Ġbuilding Ġrobust Ġword - recogn izers . </s> Model ĠSens itivity </s> In Ġcomputer Ġvision , Ġan Ġimportant Ġfactor Ġdetermining Ġthe Ġsuccess Ġof Ġan Ġadversary Ġis Ġthe Ġnorm Ġconstraint Ġon Ġthe Ġpert urb ations Ġallowed Ġto Ġan Ġimage Ġ( Ġ$ || Ġ\ bf Ġx Ġ- Ġ\ bf Ġx ^ {\ prime Ġ} || _{ \ in fty Ġ} Ġ< Ġ\ ep sil on Ġ$ Ġ). ĠHigher Ġvalues Ġof Ġ$\ ep sil on Ġ$ Ġlead Ġto Ġa Ġhigher Ġchance Ġof Ġmis - class ification Ġfor Ġat Ġleast Ġone Ġ$\ bf Ġx ^ {\ prime Ġ} $ Ġ. ĠDefense Ġmethods Ġsuch Ġas Ġquant ization ĠB IB REF 22 Ġand Ġtherm ometer Ġencoding ĠB IB REF 23 Ġtry Ġto Ġreduce Ġthe Ġspace Ġof Ġpert urb ations Ġavailable Ġto Ġthe Ġadversary Ġby Ġmaking Ġthe Ġmodel Ġinvari ant Ġto Ġsmall Ġchanges Ġin Ġthe Ġinput . </s> In ĠN LP , Ġwe Ġoften Ġget Ġsuch Ġinvari ance Ġfor Ġfree , Ġe . g ., Ġfor Ġa Ġword - level Ġmodel , Ġmost Ġof Ġthe Ġpert urb ations Ġproduced Ġby Ġour Ġcharacter - level Ġadversary Ġlead Ġto Ġan ĠUN K Ġat Ġits Ġinput . ĠIf Ġthe Ġmodel Ġis Ġrobust Ġto Ġthe Ġpresence Ġof Ġthese ĠUN K Ġtokens , Ġthere Ġis Ġlittle Ġroom Ġfor Ġan Ġadversary Ġto Ġmanipulate Ġit . ĠCharacter - level Ġmodels , Ġon Ġthe Ġother Ġhand , Ġdespite Ġtheir Ġsuperior Ġperformance Ġin Ġmany Ġtasks , Ġdo Ġnot Ġenjoy Ġsuch Ġinvari ance . ĠThis Ġcharacteristic Ġinvari ance Ġcould Ġbe Ġexploited Ġby Ġan Ġattacker . ĠThus , Ġto Ġlimit Ġthe Ġnumber Ġof Ġdifferent Ġinputs Ġto Ġthe Ġclass ifier , Ġwe Ġwish Ġto Ġreduce Ġthe Ġnumber Ġof Ġdistinct Ġword Ġrecognition Ġoutputs Ġthat Ġan Ġattacker Ġcan Ġinduce , Ġnot Ġjust Ġthe Ġnumber Ġof Ġwords Ġon Ġwhich Ġthe Ġmodel Ġis ĠâĢ ľ f ool ed âĢ Ŀ . ĠWe Ġdenote Ġthis Ġproperty Ġof Ġa Ġmodel Ġas Ġits Ġsensitivity . </s> We Ġcan Ġquantify Ġthis Ġnotion Ġfor Ġa Ġword Ġrecognition Ġsystem Ġ$ W $ Ġas Ġthe Ġexpected Ġnumber Ġof Ġunique Ġoutputs Ġit Ġassigns Ġto Ġa Ġset Ġof Ġadvers arial Ġpert urb ations . ĠGiven Ġa Ġsentence Ġ$ s $ Ġfrom Ġthe Ġset Ġof Ġsentences Ġ$\ math cal Ġ{ S } $ Ġ, Ġlet Ġ$ A ( s ) Ġ= Ġ{ s _ 1 } ^ {\ prime Ġ} Ġ, Ġ{ s _ 2 } ^ {\ prime Ġ}, Ġ\ d ots Ġ, Ġ{ s _ n } ^ {\ prime Ġ} $ Ġdenote Ġthe Ġset Ġof Ġ$ n $ Ġpert urb ations Ġto Ġit Ġunder Ġattack Ġtype Ġ$ A $ Ġ, Ġand Ġlet Ġ$ V $ Ġbe Ġthe Ġfunction Ġthat Ġmaps Ġstrings Ġto Ġan Ġinput Ġrepresentation Ġfor Ġthe Ġdownstream Ġclass ifier . ĠFor Ġa Ġword Ġlevel Ġmodel , Ġ$ V $ Ġwould Ġtransform Ġsentences Ġto Ġa Ġsequence Ġof Ġword Ġ ids , Ġmapping ĠO OV Ġwords Ġto Ġthe Ġsame ĠUN K ĠID . ĠWhereas , Ġfor Ġa Ġchar Ġ( or Ġword + char , Ġword - piece ) Ġmodel , Ġ$ V $ Ġwould Ġmap Ġinputs Ġto Ġa Ġsequence Ġof Ġcharacter ĠIDs . ĠForm ally , Ġsensitivity Ġis Ġdefined Ġas </s> $$ S _{ W , V } ^ A = \ math bb Ġ{ E } _{ s }\ left [ \ frac {\ # _{ u } ( V Ġ\ circ ĠW ({ s _ 1 } ^ {\ prime Ġ} ), Ġ\ d ots Ġ, ĠV Ġ\ circ ĠW ({ s _ n } ^ {\ prime Ġ} )) }{ n }\ right ] Ġ, $$ Ġ Ġ Ġ( E q . Ġ12 ) </s> where Ġ$ V Ġ\ circ ĠW Ġ( s _ i ) $ Ġreturns Ġthe Ġinput Ġrepresentation Ġ( of Ġthe Ġdownstream Ġclass ifier ) Ġfor Ġthe Ġoutput Ġstring Ġproduced Ġby Ġthe Ġword - recogn izer Ġ$ W $ Ġusing Ġ$ s _ i $ Ġand Ġ$\ # _{ u } (\ cd ot Ġ) $ Ġcounts Ġthe Ġnumber Ġof Ġunique Ġarguments . </s> Int uitive ly , Ġwe Ġexpect Ġa Ġhigh Ġvalue Ġof Ġ$ S _{ W , ĠV } ^ A $ Ġto Ġlead Ġto Ġa Ġlower Ġrobust ness Ġof Ġthe Ġdownstream Ġclass ifier , Ġsince Ġthe Ġadversary Ġhas Ġmore Ġdegrees Ġof Ġfreedom Ġto Ġattack Ġthe Ġclass ifier . ĠThus , Ġwhen Ġusing Ġword Ġrecognition Ġas Ġa Ġdefense , Ġit Ġis Ġprudent Ġto Ġdesign Ġa Ġlow Ġsensitivity Ġsystem Ġwith Ġa Ġlow Ġerror Ġrate . ĠHowever , Ġas Ġwe Ġwill Ġdemonstrate , Ġthere Ġis Ġoften Ġa Ġtrade - off Ġbetween Ġsensitivity Ġand Ġerror Ġrate . </s> Sy nt hes izing ĠAd vers arial ĠAttacks </s> Supp ose Ġwe Ġare Ġgiven Ġa Ġclass ifier Ġ$ C : Ġ\ math cal Ġ{ S } Ġ\ right arrow Ġ\ math cal Ġ{ Y } $ Ġwhich Ġmaps Ġnatural Ġlanguage Ġsentences Ġ$ s Ġ\ in Ġ\ math cal Ġ{ S } $ Ġto Ġa Ġlabel Ġfrom Ġa Ġpred efined Ġset Ġ$ y Ġ\ in Ġ\ math cal Ġ{ Y } $ Ġ. ĠAn Ġadversary Ġfor Ġthis Ġclass ifier Ġis Ġa Ġfunction Ġ$ A $ Ġwhich Ġmaps Ġa Ġsentence Ġ$ s $ Ġto Ġits Ġpert urbed Ġversions Ġ$\ l brace Ġs ^ {\ prime Ġ} _ 1 , Ġs ^ {\ prime Ġ} _ 2 , Ġ\ ld ots Ġ, Ġs ^ {\ prime Ġ} _{ n }\ r brace Ġ$ Ġsuch Ġthat Ġeach Ġ$ s ^ {\ prime Ġ} _ i $ Ġis Ġclose Ġto Ġ$ s $ Ġunder Ġsome Ġnotion Ġof Ġdistance Ġbetween Ġsentences . ĠWe Ġdefine Ġthe Ġrobust ness Ġof Ġclass ifier Ġ$ C $ Ġto Ġthe Ġadversary Ġ$ A $ Ġas : </s> $$ R _{ C , A } Ġ= Ġ\ math bb Ġ{ E } _ s Ġ\ left [ \ min Ġ_ { s ^ {\ prime Ġ} Ġ\ in ĠA ( s )} Ġ\ math bb Ġ{ 1 } [ C ( s ^ {\ prime Ġ}) Ġ= Ġy ] \ right ], $$ Ġ Ġ Ġ( E q . Ġ14 ) </s> where Ġ$ y $ Ġrepresents Ġthe Ġground Ġtruth Ġlabel Ġfor Ġ$ s $ Ġ. ĠIn Ġpractice , Ġa Ġreal - world Ġadversary Ġmay Ġonly Ġbe Ġable Ġto Ġquery Ġthe Ġclass ifier Ġa Ġfew Ġtimes , Ġhence Ġ$ R _{ C , A } $ Ġrepresents Ġthe Ġworst - case Ġadvers arial Ġperformance Ġof Ġ$ C $ Ġ. ĠMethods Ġfor Ġgenerating Ġadvers arial Ġexamples , Ġsuch Ġas ĠHot Fl ip ĠB IB REF 8 Ġ, Ġfocus Ġon Ġefficient Ġalgorithms Ġfor Ġsearching Ġthe Ġ$\ min Ġ$ Ġabove . ĠImpro ving Ġ$ R _{ C , A } $ Ġwould Ġimply Ġbetter Ġrobust ness Ġagainst Ġall Ġthese Ġmethods . </s> We Ġexplore Ġadversaries Ġwhich Ġpert urb Ġsentences Ġwith Ġfour Ġtypes Ġof Ġcharacter - level Ġedits : </s> ( 1 ) ĠSwap : Ġswapping Ġtwo Ġadjacent Ġinternal Ġcharacters Ġof Ġa Ġword . Ġ( 2 ) ĠDrop : Ġremoving Ġan Ġinternal Ġcharacter Ġof Ġa Ġword . Ġ( 3 ) ĠKeyboard : Ġsubstit uting Ġan Ġinternal Ġcharacter Ġwith Ġadjacent Ġcharacters Ġof ĠQ WER TY Ġkeyboard Ġ( 4 ) ĠAdd : Ġinserting Ġa Ġnew Ġcharacter Ġinternally Ġin Ġa Ġword . ĠIn Ġline Ġwith Ġthe Ġpsych ol ingu istic Ġstudies ĠB IB REF 5 Ġ, ĠB IB REF 4 Ġ, Ġto Ġensure Ġthat Ġthe Ġpert urb ations Ġdo Ġnot Ġaffect Ġhuman Ġability Ġto Ġcomprehend Ġthe Ġsentence , Ġwe Ġonly Ġallow Ġthe Ġadversary Ġto Ġedit Ġthe Ġinternal Ġcharacters Ġof Ġa Ġword , Ġand Ġnot Ġedit Ġstop words Ġor Ġwords Ġshorter Ġthan Ġ4 Ġcharacters . </s> For Ġ1 - character Ġattacks , Ġwe Ġtry Ġall Ġpossible Ġpert urb ations Ġlisted Ġabove Ġuntil Ġwe Ġfind Ġan Ġadversary Ġthat Ġflips Ġthe Ġmodel Ġprediction . ĠFor Ġ2 - character Ġattacks , Ġwe Ġgreed ily Ġfix Ġthe Ġedit Ġwhich Ġhad Ġthe Ġleast Ġconfidence Ġamong Ġ1 - character Ġattacks , Ġand Ġthen Ġtry Ġall Ġthe Ġallowed Ġpert urb ations Ġon Ġthe Ġremaining Ġwords . ĠHigher Ġorder Ġattacks Ġcan Ġbe Ġperformed Ġin Ġa Ġsimilar Ġmanner . ĠThe Ġgreedy Ġstrategy Ġreduces Ġthe Ġcomputation Ġrequired Ġto Ġobtain Ġhigher Ġorder Ġattacks , Ġbut Ġalso Ġmeans Ġthat Ġthe Ġrobust ness Ġscore Ġis Ġan Ġupper Ġbound Ġon Ġthe Ġtrue Ġrobust ness Ġof Ġthe Ġclass ifier . </s> Exper iments Ġand ĠResults </s> In Ġthis Ġsection , Ġwe Ġfirst Ġdiscuss Ġour Ġexperiments Ġon Ġthe Ġword Ġrecognition Ġsystems . </s> Word ĠError ĠCorrection </s> Data : ĠWe Ġevaluate Ġthe Ġspell Ġcorrect ors Ġfrom ĠÂ§ Ġ" Rob ust ĠWord ĠRecogn ition " Ġon Ġmovie Ġreviews Ġfrom Ġthe ĠStanford ĠSent iment ĠTree bank Ġ( S ST ) ĠB IB REF 24 Ġ. ĠThe ĠS ST Ġdataset Ġconsists Ġof Ġ85 44 Ġmovie Ġreviews , Ġwith Ġa Ġvocabulary Ġof Ġover Ġ16 K Ġwords . ĠAs Ġa Ġbackground Ġcorpus , Ġwe Ġuse Ġthe ĠIM DB Ġmovie Ġreviews ĠB IB REF 25 Ġ, Ġwhich Ġcontain Ġ54 K Ġmovie Ġreviews , Ġand Ġa Ġvocabulary Ġof Ġover Ġ78 K Ġwords . ĠThe Ġtwo Ġdatasets Ġdo Ġnot Ġshare Ġany Ġreviews Ġin Ġcommon . ĠThe Ġspell - cor rection Ġmodels Ġare Ġevaluated Ġon Ġtheir Ġability Ġto Ġcorrect Ġmiss pell ings . ĠThe Ġtest Ġsetting Ġconsists Ġof Ġreviews Ġwhere Ġeach Ġword Ġ( with Ġlength Ġ$\ ge Ġ4 $ Ġ, Ġbarring Ġstop words ) Ġis Ġattacked Ġby Ġone Ġof Ġthe Ġattack Ġtypes Ġ( from Ġswap , Ġadd , Ġdrop Ġand Ġkeyboard Ġattacks ). ĠIn Ġthe Ġall Ġattack Ġsetting , Ġwe Ġmix Ġall Ġattacks Ġby Ġrandomly Ġchoosing Ġone Ġfor Ġeach Ġword . ĠThis Ġmost Ġclosely Ġresembles Ġa Ġreal Ġworld Ġattack Ġsetting . </s> In Ġaddition Ġto Ġour Ġword Ġrecognition Ġmodels , Ġwe Ġalso Ġcompare Ġto ĠAfter ĠThe ĠDeadline Ġ( AT D ), Ġan Ġopen - source Ġspell Ġcorrect or . ĠWe Ġfound ĠAT D Ġto Ġbe Ġthe Ġbest Ġfreely - available Ġcorrect or . ĠWe Ġrefer Ġthe Ġreader Ġto ĠB IB REF 7 Ġfor Ġcomparisons Ġof ĠSc R NN Ġto Ġother Ġanonym ized Ġcommercial Ġspell Ġcheck ers . </s> For Ġthe ĠSc R NN Ġmodel , Ġwe Ġuse Ġa Ġsingle - layer ĠBi - L ST M Ġwith Ġa Ġhidden Ġdimension Ġsize Ġof Ġ50 . ĠThe Ġinput Ġrepresentation Ġconsists Ġof Ġ198 Ġdimensions , Ġwhich Ġis Ġthr ice Ġthe Ġnumber Ġof Ġunique Ġcharacters Ġ( 66 ) Ġin Ġthe Ġvocabulary . ĠWe Ġcap Ġthe Ġvocabulary Ġsize Ġto Ġ10 K Ġwords , Ġwhereas Ġwe Ġuse Ġthe Ġentire Ġvocabulary Ġof Ġ7 84 70 Ġwords Ġwhen Ġwe Ġback off Ġto Ġthe Ġbackground Ġmodel . ĠFor Ġtraining Ġthese Ġnetworks , Ġwe Ġcorrupt Ġthe Ġmovie Ġreviews Ġaccording Ġto Ġall Ġattack Ġtypes , Ġi . e ., Ġapplying Ġone Ġof Ġthe Ġ4 Ġattack Ġtypes Ġto Ġeach Ġword , Ġand Ġtrying Ġto Ġreconstruct Ġthe Ġoriginal Ġwords Ġvia Ġcross Ġentropy Ġloss . </s> We Ġcalculate Ġthe Ġword Ġerror Ġrates Ġ( WER ) Ġof Ġeach Ġof Ġthe Ġmodels Ġfor Ġdifferent Ġattacks Ġand Ġpresent Ġour Ġfindings Ġin ĠTable Ġ2 Ġ. ĠNote Ġthat ĠAT D Ġincorrectly Ġpredicts Ġ$ 11 . 2 $ Ġwords Ġfor Ġevery Ġ100 Ġwords Ġ( in Ġthe Ġ` all ' Ġsetting ), Ġwhereas , Ġall Ġof Ġthe Ġback off Ġvariations Ġof Ġthe ĠSc R NN Ġreconstruct Ġbetter . ĠThe Ġmost Ġaccurate Ġvariant Ġinvolves Ġbacking Ġoff Ġto Ġthe Ġbackground Ġmodel , Ġresulting Ġin Ġa Ġlow Ġerror Ġrate Ġof Ġ$ 6 . 9 \ % $ Ġ, Ġleading Ġto Ġthe Ġbest Ġperformance Ġon Ġword Ġrecognition . ĠThis Ġis Ġa Ġ$ 32 \ % $ Ġrelative Ġerror Ġreduction Ġcompared Ġto Ġthe Ġvanilla ĠSc R NN Ġmodel Ġwith Ġa Ġpass - through Ġback off Ġstrategy . ĠWe Ġcan Ġattribute Ġthe Ġimproved Ġperformance Ġto Ġthe Ġfact Ġthat Ġthere Ġare Ġ$ 5 . 25 \ % $ Ġwords Ġin Ġthe Ġtest Ġcorpus Ġthat Ġare Ġunseen Ġin Ġthe Ġtraining Ġcorpus , Ġand Ġare Ġthus Ġonly Ġrecover able Ġby Ġbacking Ġoff Ġto Ġa Ġlarger Ġcorpus . ĠNotably , Ġonly Ġtraining Ġon Ġthe Ġlarger Ġbackground Ġcorpus Ġdoes Ġworse , Ġat Ġ$ 8 . 7 \ % $ Ġ, Ġsince Ġthe Ġdistribution Ġof Ġword Ġfrequencies Ġis Ġdifferent Ġin Ġthe Ġbackground Ġcorpus Ġcompared Ġto Ġthe Ġforeground Ġcorpus . </s> Rob ust ness Ġto Ġadvers arial Ġattacks </s> We Ġuse Ġsentiment Ġanalysis Ġand Ġparaph rase Ġdetection Ġas Ġdownstream Ġtasks , Ġas Ġfor Ġthese Ġtwo Ġtasks , Ġ1 - 2 Ġcharacter Ġedits Ġdo Ġnot Ġchange Ġthe Ġoutput Ġlabels . </s> For Ġsentiment Ġclassification , Ġwe Ġsystematically Ġstudy Ġthe Ġeffect Ġof Ġcharacter - level Ġadvers arial Ġattacks Ġon Ġtwo Ġarchitectures Ġand Ġfour Ġdifferent Ġinput Ġformats . ĠThe Ġfirst Ġarchitecture Ġenc odes Ġthe Ġinput Ġsentence Ġinto Ġa Ġsequence Ġof Ġembed d ings , Ġwhich Ġare Ġthen Ġsequ entially Ġprocessed Ġby Ġa ĠBi L ST M . ĠThe Ġfirst Ġand Ġlast Ġstates Ġof Ġthe ĠBi L ST M Ġare Ġthen Ġused Ġby Ġthe Ġsoft max Ġlayer Ġto Ġpredict Ġthe Ġsentiment Ġof Ġthe Ġinput . ĠWe Ġconsider Ġthree Ġinput Ġformats Ġfor Ġthis Ġarchitecture : Ġ( 1 ) ĠWord - only : Ġwhere Ġthe Ġinput Ġwords Ġare Ġencoded Ġusing Ġa Ġlookup Ġtable ; Ġ( 2 ) ĠChar - only : Ġwhere Ġthe Ġinput Ġwords Ġare Ġencoded Ġusing Ġa Ġseparate Ġsingle - lay ered ĠBi L ST M Ġover Ġtheir Ġcharacters ; Ġand Ġ( 3 ) ĠWord Ġ$ + $ ĠChar : Ġwhere Ġthe Ġinput Ġwords Ġare Ġencoded Ġusing Ġa Ġconc aten ation Ġof Ġ( 1 ) Ġand Ġ( 2 ) Ġ. </s> The Ġsecond Ġarchitecture Ġuses Ġthe Ġfine - tun ed ĠB ERT Ġmodel ĠB IB REF 26 Ġ, Ġwith Ġan Ġinput Ġformat Ġof Ġword - piece Ġtoken ization . ĠThis Ġmodel Ġhas Ġrecently Ġset Ġa Ġnew Ġstate - of - the - art Ġon Ġseveral ĠN LP Ġbenchmarks , Ġincluding Ġthe Ġsentiment Ġanalysis Ġtask Ġwe Ġconsider Ġhere . ĠAll Ġmodels Ġare Ġtrained Ġand Ġevaluated Ġon Ġthe Ġbinary Ġversion Ġof Ġthe Ġsentence - level ĠStanford ĠSent iment ĠTree bank ĠB IB REF 24 Ġdataset Ġwith Ġonly Ġpositive Ġand Ġnegative Ġreviews . </s> We Ġalso Ġconsider Ġthe Ġtask Ġof Ġparaph rase Ġdetection . ĠHere Ġtoo , Ġwe Ġmake Ġuse Ġof Ġthe Ġfine - tun ed ĠB ERT ĠB IB REF 26 Ġ, Ġwhich Ġis Ġtrained Ġand Ġevaluated Ġon Ġthe ĠMicrosoft ĠResearch ĠPar aph rase ĠCorpus Ġ( MR PC ) ĠB IB REF 27 Ġ. </s> Two Ġcommon Ġmethods Ġfor Ġdealing Ġwith Ġadvers arial Ġexamples Ġinclude : Ġ( 1 ) Ġdata Ġaug mentation Ġ( DA ) ĠB IB REF 28 Ġ; Ġand Ġ( 2 ) Ġadvers arial Ġtraining Ġ( Adv ) ĠB IB REF 29 Ġ. ĠIn ĠDA , Ġthe Ġtrained Ġmodel Ġis Ġfine - tun ed Ġafter Ġaugment ing Ġthe Ġtraining Ġset Ġwith Ġan Ġequal Ġnumber Ġof Ġexamples Ġrandomly Ġattacked Ġwith Ġa Ġ1 - character Ġedit . ĠIn ĠAdv , Ġthe Ġtrained Ġmodel Ġis Ġfine - tun ed Ġwith Ġadditional Ġadvers arial Ġexamples Ġ( selected Ġat Ġrandom ) Ġthat Ġproduce Ġincorrect Ġpredictions Ġfrom Ġthe Ġcurrent - state Ġclass ifier . ĠThe Ġprocess Ġis Ġrepeated Ġiter atively , Ġgenerating Ġand Ġadding Ġnewer Ġadvers arial Ġexamples Ġfrom Ġthe Ġupdated Ġclass ifier Ġmodel , Ġuntil Ġthe Ġadvers arial Ġaccuracy Ġon Ġdev Ġset Ġstops Ġimproving . </s> In ĠTable Ġ3 Ġ, Ġwe Ġexamine Ġthe Ġrobust ness Ġof Ġthe Ġsentiment Ġmodels Ġunder Ġeach Ġattack Ġand Ġdefense Ġmethod . ĠIn Ġthe Ġabsence Ġof Ġany Ġattack Ġor Ġdefense , ĠB ERT Ġ( a Ġword - piece Ġmodel ) Ġperforms Ġthe Ġbest Ġ( Ġ$ 90 . 3 \ % $ Ġ) Ġfollowed Ġby Ġword + char Ġmodels Ġ( Ġ$ 80 . 5 \ % $ Ġ), Ġword - only Ġmodels Ġ( Ġ$ 79 . 2 \ % $ Ġ) Ġand Ġthen Ġchar - only Ġmodels Ġ( Ġ$ 70 . 3 \ % $ Ġ). ĠHowever , Ġeven Ġsingle - character Ġattacks Ġ( ch osen Ġadvers ar ially ) Ġcan Ġbe Ġcatastrophic , Ġresulting Ġin Ġa Ġsignificantly Ġdegraded Ġperformance Ġof Ġ$ 46 \ % $ Ġ, Ġ$ 57 \ % $ Ġ, Ġ$ 59 \ % $ Ġand Ġ$ 33 \ % $ Ġ, Ġrespectively Ġunder Ġthe Ġ` all ' Ġsetting . </s> Int uitive ly , Ġone Ġmight Ġsuppose Ġthat Ġword - piece Ġand Ġcharacter - level Ġmodels Ġwould Ġbe Ġmore Ġrobust Ġto Ġsuch Ġattacks Ġgiven Ġthey Ġcan Ġmake Ġuse Ġof Ġthe Ġremaining Ġcontext . ĠHowever , Ġwe Ġfind Ġthat Ġthey Ġare Ġthe Ġmore Ġsusceptible . ĠTo Ġsee Ġwhy , Ġnote Ġthat Ġthe Ġword Ġ` beaut iful ' Ġcan Ġonly Ġbe Ġaltered Ġin Ġa Ġfew Ġways Ġfor Ġword - only Ġmodels , Ġeither Ġleading Ġto Ġan ĠUN K Ġor Ġan Ġexisting Ġvocabulary Ġword , Ġwhereas , Ġword - piece Ġand Ġcharacter - only Ġmodels Ġtreat Ġeach Ġunique Ġcharacter Ġcombination Ġdifferently . ĠThis Ġprovides Ġmore Ġvariations Ġthat Ġan Ġattacker Ġcan Ġexploit . ĠFollowing Ġsimilar Ġreasoning , Ġadd Ġand Ġkey Ġattacks Ġpose Ġa Ġgreater Ġthreat Ġthan Ġswap Ġand Ġdrop Ġattacks . ĠThe Ġrobust ness Ġof Ġdifferent Ġmodels Ġcan Ġbe Ġordered Ġas Ġword - only Ġ$ > $ Ġword + char Ġ$ > $ Ġchar - only Ġ$\ sim Ġ$ Ġword - piece , Ġand Ġthe Ġefficacy Ġof Ġdifferent Ġattacks Ġas Ġadd Ġ$ > $ Ġkey Ġ$ > $ Ġdrop Ġ$ > $ Ġswap . </s> Next , Ġwe Ġscrutin ize Ġthe Ġeffectiveness Ġof Ġdefense Ġmethods Ġwhen Ġfaced Ġagainst Ġadvers ar ially Ġchosen Ġattacks . ĠClearly Ġfrom Ġtable Ġ3 Ġ, ĠDA Ġand ĠAdv Ġare Ġnot Ġeffective Ġin Ġthis Ġcase . ĠWe Ġobserved Ġthat Ġdespite Ġa Ġlow Ġtraining Ġerror , Ġthese Ġmodels Ġwere Ġnot Ġable Ġto Ġgeneral ize Ġto Ġattacks Ġon Ġnewer Ġwords Ġat Ġtest Ġtime . ĠAT D Ġspell Ġcorrect or Ġis Ġthe Ġmost Ġeffective Ġon Ġkeyboard Ġattacks , Ġbut Ġperforms Ġpoorly Ġon Ġother Ġattack Ġtypes , Ġparticularly Ġthe Ġadd Ġattack Ġstrategy . </s> The ĠSc R NN Ġmodel Ġwith Ġpass - through Ġback off Ġoffers Ġbetter Ġprotection , Ġbringing Ġback Ġthe Ġadvers arial Ġaccuracy Ġwithin Ġ$ 5 \ % $ Ġrange Ġfor Ġthe Ġswap Ġattack . ĠIt Ġis Ġalso Ġeffective Ġunder Ġother Ġattack Ġclasses , Ġand Ġcan Ġmitigate Ġthe Ġadvers arial Ġeffect Ġin Ġword - piece Ġmodels Ġby Ġ$ 21 \ % $ Ġ, Ġcharacter - only Ġmodels Ġby Ġ$ 19 \ % $ Ġ, Ġand Ġin Ġword , Ġand Ġword + char Ġmodels Ġby Ġover Ġ$ 4 . 5 \ % $ Ġ. ĠThis Ġsuggests Ġthat Ġthe Ġdirect Ġtraining Ġsignal Ġof Ġword Ġerror Ġcorrection Ġis Ġmore Ġeffective Ġthan Ġthe Ġindirect Ġsignal Ġof Ġsentiment Ġclassification Ġavailable Ġto ĠDA Ġand ĠAdv Ġfor Ġmodel Ġrobust ness . </s> We Ġobserve Ġadditional Ġgains Ġby Ġusing Ġbackground Ġmodels Ġas Ġa Ġback off Ġalternative , Ġbecause Ġof Ġits Ġlower Ġword Ġerror Ġrate Ġ( WER ), Ġespecially , Ġunder Ġthe Ġswap Ġand Ġdrop Ġattacks . ĠHowever , Ġthese Ġgains Ġdo Ġnot Ġconsistently Ġtranslate Ġin Ġall Ġother Ġsettings , Ġas Ġlower ĠW ER Ġis Ġnecessary Ġbut Ġnot Ġsufficient . ĠBesides Ġlower Ġerror Ġrate , Ġwe Ġfind Ġthat Ġa Ġsolid Ġdefense Ġshould Ġfurnish Ġthe Ġattacker Ġthe Ġfew est Ġoptions Ġto Ġattack , Ġi . e . Ġit Ġshould Ġhave Ġa Ġlow Ġsensitivity . </s> As Ġwe Ġshall Ġsee Ġin Ġsection ĠÂ§ Ġ" Understanding ĠModel ĠSens itivity " Ġ, Ġthe Ġback off Ġneutral Ġvariation Ġhas Ġthe Ġlowest Ġsensitivity Ġdue Ġto Ġmapping ĠUN K Ġpredictions Ġto Ġa Ġfixed Ġneutral Ġword . ĠThus , Ġit Ġresults Ġin Ġthe Ġhighest Ġrobust ness Ġon Ġmost Ġof Ġthe Ġattack Ġtypes Ġfor Ġall Ġfour Ġmodel Ġclasses . </s> Table Ġ4 Ġshows Ġthe Ġaccuracy Ġof ĠB ERT Ġon Ġ200 Ġexamples Ġfrom Ġthe Ġdev Ġset Ġof Ġthe ĠMR PC Ġparaph rase Ġdetection Ġtask Ġunder Ġvarious Ġattack Ġand Ġdefense Ġsettings . ĠWe Ġre - trained Ġthe ĠSc R NN Ġmodel Ġvariants Ġon Ġthe ĠMR PC Ġtraining Ġset Ġfor Ġthese Ġexperiments . ĠAgain , Ġwe Ġfind Ġthat Ġsimple Ġ1 - 2 Ġcharacter Ġattacks Ġcan Ġbring Ġdown Ġthe Ġaccuracy Ġof ĠB ERT Ġsignificantly Ġ( Ġ$ 89 \ % $ Ġto Ġ$ 31 \ % $ Ġ). ĠWord Ġrecognition Ġmodels Ġcan Ġprovide Ġan Ġeffective Ġdefense , Ġwith Ġboth Ġour Ġpass - through Ġand Ġneutral Ġvariants Ġrecovering Ġmost Ġof Ġthe Ġaccuracy . ĠWhile Ġthe Ġneutral Ġback off Ġmodel Ġis Ġeffective Ġon Ġ2 - char Ġattacks , Ġit Ġhurts Ġperformance Ġin Ġthe Ġno Ġattack Ġsetting , Ġsince Ġit Ġincorrectly Ġmod ifies Ġcertain Ġcorrectly Ġspelled Ġentity Ġnames . ĠSince Ġthe Ġtwo Ġvariants Ġare Ġalready Ġeffective , Ġwe Ġdid Ġnot Ġtrain Ġa Ġbackground Ġmodel Ġfor Ġthis Ġtask . </s> Understanding ĠModel ĠSens itivity </s> To Ġstudy Ġmodel Ġsensitivity , Ġfor Ġeach Ġsentence , Ġwe Ġpert urb Ġone Ġrandomly - ch osen Ġword Ġand Ġreplace Ġit Ġwith Ġall Ġpossible Ġpert urb ations Ġunder Ġa Ġgiven Ġattack Ġtype . ĠThe Ġresulting Ġset Ġof Ġpert urbed Ġsentences Ġis Ġthen Ġfed Ġto Ġthe Ġword Ġrecogn izer Ġ( whose Ġsensitivity Ġis Ġto Ġbe Ġestimated ). ĠAs Ġdescribed Ġin Ġequation Ġ12 Ġ, Ġwe Ġcount Ġthe Ġnumber Ġof Ġunique Ġpredictions Ġfrom Ġthe Ġoutput Ġsentences . ĠTwo Ġcorrections Ġare Ġconsidered Ġunique Ġif Ġthey Ġare Ġmapped Ġdifferently Ġby Ġthe Ġdownstream Ġclass ifier . </s> The Ġneutral Ġback off Ġvariant Ġhas Ġthe Ġlowest Ġsensitivity Ġ( Table Ġ5 Ġ). ĠThis Ġis Ġexpected , Ġas Ġit Ġreturns Ġa Ġfixed Ġneutral Ġword Ġwhenever Ġthe ĠSc R NN Ġpredicts Ġan ĠUN K , Ġtherefore Ġreducing Ġthe Ġnumber Ġof Ġunique Ġoutputs Ġit Ġpredicts . ĠOpen Ġvocabulary Ġ( i . e . Ġchar - only , Ġword + char , Ġword - piece ) Ġdownstream Ġclass ifiers Ġconsider Ġevery Ġunique Ġcombination Ġof Ġcharacters Ġdifferently , Ġwhereas Ġword - only Ġclass ifiers Ġinternally Ġtreat Ġall Ġout Ġof Ġvocabulary Ġ( OO V ) Ġwords Ġalike . ĠHence , Ġfor Ġchar - only , Ġword + char , Ġand Ġword - piece Ġmodels , Ġthe Ġpass - through Ġversion Ġis Ġmore Ġsensitive Ġthan Ġthe Ġbackground Ġvariant , Ġas Ġit Ġpasses Ġwords Ġas Ġis Ġ( and Ġeach Ġcombination Ġis Ġconsidered Ġuniquely ). ĠHowever , Ġfor Ġword - only Ġmodels , Ġpass - through Ġis Ġless Ġsensitive Ġas Ġall Ġthe ĠO OV Ġcharacter Ġcombinations Ġare Ġrendered Ġidentical . </s> Ide ally , Ġa Ġpreferred Ġdefense Ġis Ġone Ġwith Ġlow Ġsensitivity Ġand Ġword Ġerror Ġrate . ĠIn Ġpractice , Ġhowever , Ġwe Ġsee Ġthat Ġa Ġlow Ġerror Ġrate Ġoften Ġcomes Ġat Ġthe Ġcost Ġof Ġsensitivity . ĠWe Ġsee Ġthis Ġtrade - off Ġin ĠFigure Ġ2 Ġ, Ġwhere Ġwe Ġplot ĠW ER Ġand Ġsensitivity Ġon Ġthe Ġtwo Ġaxes , Ġand Ġdepict Ġthe Ġrobust ness Ġwhen Ġusing Ġdifferent Ġback off Ġvariants . ĠGenerally , Ġsensitivity Ġis Ġthe Ġmore Ġdominant Ġfactor Ġout Ġof Ġthe Ġtwo , Ġas Ġthe Ġerror Ġrates Ġof Ġthe Ġconsidered Ġvariants Ġare Ġreasonably Ġlow . </s> We Ġverify Ġif Ġthe Ġsentiment Ġ( of Ġthe Ġreviews ) Ġis Ġpreserved Ġwith Ġchar - level Ġattacks . ĠIn Ġa Ġhuman Ġstudy Ġwith Ġ50 Ġattacked Ġ( and Ġsubsequently Ġmis classified ), Ġand Ġ50 Ġunchanged Ġreviews , Ġit Ġwas Ġnoted Ġthat Ġ48 Ġand Ġ49 , Ġrespectively , Ġpreserved Ġthe Ġsentiment . </s> Conclusion </s> As Ġcharacter Ġand Ġword - piece Ġinputs Ġbecome Ġcommonplace Ġin Ġmodern ĠN LP Ġpipelines , Ġit Ġis Ġworth Ġhighlighting Ġthe Ġvulnerability Ġthey Ġadd . ĠWe Ġshow Ġthat Ġminim ally - do ct ored Ġattacks Ġcan Ġbring Ġdown Ġaccuracy Ġof Ġclass ifiers Ġto Ġrandom Ġguessing . ĠWe Ġrecommend Ġword Ġrecognition Ġas Ġa Ġsafeguard Ġagainst Ġthis Ġand Ġbuild Ġupon ĠR NN - based Ġsemi - character Ġword Ġrecogn izers . ĠWe Ġdiscover Ġthat Ġwhen Ġused Ġas Ġa Ġdefense Ġmechanism , Ġthe Ġmost Ġaccurate Ġword Ġrecognition Ġmodels Ġare Ġnot Ġalways Ġthe Ġmost Ġrobust Ġagainst Ġadvers arial Ġattacks . ĠAdditionally , Ġwe Ġhighlight Ġthe Ġneed Ġto Ġcontrol Ġthe Ġsensitivity Ġof Ġthese Ġmodels Ġto Ġachieve Ġhigh Ġrobust ness . </s> Acknowled gements </s> The Ġauthors Ġare Ġgrateful Ġto ĠGraham ĠNe ub ig , ĠEdu ard ĠH ovy , ĠPaul ĠMichel , ĠMans i ĠGupta , Ġand ĠAnton ios ĠAn ast as opoulos Ġfor Ġsuggestions Ġand Ġfeedback .