Yes
BOOLEAN
No
BOOLEAN
In our setup, our starting point is a base model, trained on NLI data. Rather than employing automated adversarial methods, here the model's “adversary” is a human annotator. Given a context (also often called a “premise” in NLI), and a desired target label, we ask the human writer to provide a hypothesis that fools the model into misclassifying the label. One can think of the writer as a “white hat” hacker, trying to identify vulnerabilities in the system. For each human-generated example that is misclassified, we also ask the writer to provide a reason why they believe it was misclassified.
1910.14599
Introduction </s> Progress Ġin ĠAI Ġhas Ġbeen Ġdriven Ġby , Ġamong Ġother Ġthings , Ġthe Ġdevelopment Ġof Ġchallenging Ġlarge - scale Ġbenchmarks Ġlike ĠImage Net ĠB IB REF 0 Ġin Ġcomputer Ġvision , Ġand ĠSN LI ĠB IB REF 1 , ĠS Qu AD ĠB IB REF 2 , Ġand Ġothers Ġin Ġnatural Ġlanguage Ġprocessing Ġ( N LP ). ĠRecently , Ġfor Ġnatural Ġlanguage Ġunderstanding Ġ( NL U ) Ġin Ġparticular , Ġthe Ġfocus Ġhas Ġshifted Ġto Ġcombined Ġbenchmarks Ġlike ĠSent E val ĠB IB REF 3 Ġand ĠGL UE ĠB IB REF 4 , Ġwhich Ġtrack Ġmodel Ġperformance Ġon Ġmultiple Ġtasks Ġand Ġprovide Ġa Ġunified Ġplatform Ġfor Ġanalysis . </s> With Ġthe Ġrapid Ġpace Ġof Ġadvancement Ġin ĠAI , Ġhowever , ĠNL U Ġbenchmarks Ġstruggle Ġto Ġkeep Ġup Ġwith Ġmodel Ġimprovement . ĠWhereas Ġit Ġtook Ġaround Ġ15 Ġyears Ġto Ġachieve ĠâĢ ľ near - human Ġperformance âĢ Ŀ Ġon ĠMN IST ĠB IB REF 5 , ĠB IB REF 6 , ĠB IB REF 7 Ġand Ġapproximately Ġ7 Ġyears Ġto Ġsurpass Ġhumans Ġon ĠImage Net ĠB IB REF 8 , ĠB IB REF 0 , ĠB IB REF 9 , Ġthe ĠGL UE Ġbenchmark Ġdid Ġnot Ġlast Ġas Ġlong Ġas Ġwe Ġwould Ġhave Ġhoped Ġafter Ġthe Ġadvent Ġof ĠB ERT ĠB IB REF 10 , Ġand Ġrapidly Ġhad Ġto Ġbe Ġextended Ġinto ĠSuper GL UE ĠB IB REF 11 . ĠThis Ġraises Ġan Ġimportant Ġquestion : ĠCan Ġwe Ġcollect Ġa Ġlarge Ġbenchmark Ġdataset Ġthat Ġcan Ġlast Ġlonger ? </s> The Ġspeed Ġwith Ġwhich Ġbenchmarks Ġbecome Ġobsolete Ġraises Ġanother Ġimportant Ġquestion : Ġare Ġcurrent ĠNL U Ġmodels Ġgenuinely Ġas Ġgood Ġas Ġtheir Ġhigh Ġperformance Ġon Ġbenchmarks Ġsuggests ? ĠA Ġgrowing Ġbody Ġof Ġevidence Ġshows Ġthat Ġstate - of - the - art Ġmodels Ġlearn Ġto Ġexploit Ġspurious Ġstatistical Ġpatterns Ġin Ġdatasets ĠB IB REF 12 , ĠB IB REF 13 , ĠB IB REF 14 , ĠB IB REF 15 , ĠB IB REF 16 , ĠB IB REF 17 , Ġinstead Ġof Ġlearning Ġmeaning Ġin Ġthe Ġflexible Ġand Ġgeneral izable Ġway Ġthat Ġhumans Ġdo . ĠGiven Ġthis , Ġhuman Ġannot ators âĢĶ be Ġthey Ġseasoned ĠN LP Ġresearchers Ġor Ġnon - exper ts âĢĶ might Ġeasily Ġbe Ġable Ġto Ġconstruct Ġexamples Ġthat Ġexpose Ġmodel Ġbr itt leness . </s> We Ġpropose Ġan Ġiter ative , Ġadvers arial Ġhuman - and - model - in - the - loop Ġsolution Ġfor ĠNL U Ġdataset Ġcollection Ġthat Ġaddresses Ġboth Ġbenchmark Ġlongevity Ġand Ġrobust ness Ġissues . ĠIn Ġthe Ġfirst Ġstage , Ġhuman Ġannot ators Ġdevise Ġexamples Ġthat Ġour Ġcurrent Ġbest Ġmodels Ġcannot Ġdetermine Ġthe Ġcorrect Ġlabel Ġfor . ĠThese Ġresulting Ġhard Ġexamples âĢĶ which Ġshould Ġexpose Ġadditional Ġmodel Ġweaknesses âĢĶ can Ġbe Ġadded Ġto Ġthe Ġtraining Ġset Ġand Ġused Ġto Ġtrain Ġa Ġstronger Ġmodel . ĠWe Ġthen Ġsubject Ġthe Ġstrengthened Ġmodel Ġto Ġhuman Ġinterference Ġand Ġcollect Ġmore Ġweaknesses Ġover Ġseveral Ġrounds . ĠAfter Ġeach Ġround , Ġwe Ġboth Ġtrain Ġa Ġnew Ġmodel , Ġand Ġset Ġaside Ġa Ġnew Ġtest Ġset . ĠThe Ġprocess Ġcan Ġbe Ġiter atively Ġrepeated Ġin Ġa Ġnever - ending Ġlearning ĠB IB REF 18 Ġsetting , Ġwith Ġthe Ġmodel Ġgetting Ġstronger Ġand Ġthe Ġtest Ġset Ġgetting Ġharder Ġin Ġeach Ġnew Ġround . This Ġprocess Ġyields Ġa ĠâĢ ľ moving Ġpost âĢ Ŀ Ġdynamic Ġtarget Ġfor ĠNL U Ġsystems , Ġrather Ġthan Ġa Ġstatic Ġbenchmark Ġthat Ġwill Ġeventually Ġsatur ate . </s> Our Ġapproach Ġdraws Ġinspiration Ġfrom Ġrecent Ġefforts Ġthat Ġgam ify Ġcollaborative Ġtraining Ġof Ġmachine Ġlearning Ġagents Ġover Ġmultiple Ġrounds ĠB IB REF 19 Ġand Ġpit ĠâĢ ľ builders âĢ Ŀ Ġagainst ĠâĢ ľ breakers âĢ Ŀ Ġto Ġlearn Ġbetter Ġmodels ĠB IB REF 20 . ĠRecently , ĠDin an 2019 build Ġshowed Ġthat Ġa Ġsimilar Ġapproach Ġcan Ġbe Ġused Ġto Ġmake Ġdialogue Ġsafety Ġclass ifiers Ġmore Ġrobust . ĠHere , Ġwe Ġfocus Ġon Ġnatural Ġlanguage Ġinference Ġ( N LI ), Ġarguably Ġthe Ġmost Ġcanonical Ġtask Ġin ĠNL U . ĠWe Ġcollected Ġthree Ġrounds Ġof Ġdata , Ġand Ġcall Ġour Ġnew Ġdataset ĠAd vers arial ĠNL I Ġ( AN LI ). </s> Our Ġcontributions Ġare Ġas Ġfollows : Ġ1 ) ĠWe Ġintroduce Ġa Ġnovel Ġhuman - and - model - in - the - loop Ġdataset , Ġcurrently Ġconsisting Ġof Ġthree Ġrounds Ġthat Ġprogressively Ġincrease Ġin Ġdifficulty Ġand Ġcomplexity , Ġthat Ġincludes Ġannot ator - provided Ġexplanations . Ġ2 ) ĠWe Ġshow Ġthat Ġtraining Ġmodels Ġon Ġthis Ġnew Ġdataset Ġleads Ġto Ġstate - of - the - art Ġperformance Ġon Ġa Ġvariety Ġof Ġpopular ĠNL I Ġbenchmarks . Ġ3 ) ĠWe Ġprovide Ġa Ġdetailed Ġanalysis Ġof Ġthe Ġcollected Ġdata Ġthat Ġsheds Ġlight Ġon Ġthe Ġshortcomings Ġof Ġcurrent Ġmodels , Ġcategor izes Ġthe Ġdata Ġby Ġinference Ġtype Ġto Ġexamine Ġweaknesses , Ġand Ġdemonstrates Ġgood Ġperformance Ġon ĠNL I Ġstress Ġtests . ĠThe ĠAN LI Ġdataset Ġis Ġavailable Ġat Ġgithub . com / facebook research / an li /. ĠA Ġdemo Ġof Ġthe Ġannotation Ġprocedure Ġcan Ġbe Ġviewed Ġat Ġadvers arial n li . com . </s> Dat as et Ġcollection </s> The Ġprimary Ġaim Ġof Ġthis Ġwork Ġis Ġto Ġcreate Ġa Ġnew Ġlarge - scale ĠNL I Ġbenchmark Ġon Ġwhich Ġcurrent Ġstate - of - the - art Ġmodels Ġfail . ĠThis Ġconstitutes Ġa Ġnew Ġtarget Ġfor Ġthe Ġfield Ġto Ġwork Ġtowards , Ġand Ġcan Ġeluc idate Ġmodel Ġcapabilities Ġand Ġlimitations . ĠAs Ġnoted , Ġhowever , Ġstatic Ġbenchmarks Ġdo Ġnot Ġlast Ġvery Ġlong Ġthese Ġdays . ĠIf Ġcontinuously Ġdeployed , Ġthe Ġdata Ġcollection Ġprocedure Ġwe Ġintroduce Ġhere Ġcan Ġpose Ġa Ġdynamic Ġchallenge Ġthat Ġallows Ġfor Ġnever - ending Ġlearning . </s> Dat as et Ġcollection Ġ: :: ĠHAM LET </s> To Ġparaph rase Ġthe Ġgreat Ġb ard ĠB IB REF 21 , Ġthere Ġis Ġsomething Ġrotten Ġin Ġthe Ġstate Ġof Ġthe Ġart . ĠWe Ġpropose ĠHuman - And - Model - in - the - Loop ĠEnt ail ment ĠTraining Ġ( HAM LET ), Ġa Ġtraining Ġprocedure Ġto Ġautomatically Ġmitigate Ġproblems Ġwith Ġcurrent Ġdataset Ġcollection Ġprocedures Ġ( see ĠFigure ĠFIG REF 1 ). </s> In Ġour Ġsetup , Ġour Ġstarting Ġpoint Ġis Ġa Ġbase Ġmodel , Ġtrained Ġon ĠNL I Ġdata . ĠRather Ġthan Ġemploying Ġautomated Ġadvers arial Ġmethods , Ġhere Ġthe Ġmodel 's ĠâĢ ľ ad vers ary âĢ Ŀ Ġis Ġa Ġhuman Ġannot ator . ĠGiven Ġa Ġcontext Ġ( also Ġoften Ġcalled Ġa ĠâĢ ľ prem ise âĢ Ŀ Ġin ĠNL I ), Ġand Ġa Ġdesired Ġtarget Ġlabel , Ġwe Ġask Ġthe Ġhuman Ġwriter Ġto Ġprovide Ġa Ġhypothesis Ġthat Ġfools Ġthe Ġmodel Ġinto Ġmis class ifying Ġthe Ġlabel . ĠOne Ġcan Ġthink Ġof Ġthe Ġwriter Ġas Ġa ĠâĢ ľ white Ġhat âĢ Ŀ Ġhacker , Ġtrying Ġto Ġidentify Ġvulnerabilities Ġin Ġthe Ġsystem . ĠFor Ġeach Ġhuman - generated Ġexample Ġthat Ġis Ġmis classified , Ġwe Ġalso Ġask Ġthe Ġwriter Ġto Ġprovide Ġa Ġreason Ġwhy Ġthey Ġbelieve Ġit Ġwas Ġmis classified . </s> For Ġexamples Ġthat Ġthe Ġmodel Ġmis classified , Ġit Ġis Ġnecessary Ġto Ġverify Ġthat Ġthey Ġare Ġactually Ġcorrect ĠâĢĶ i . e ., Ġthat Ġthe Ġgiven Ġcontext - hyp ot hesis Ġpairs Ġgenuinely Ġhave Ġtheir Ġspecified Ġtarget Ġlabel . ĠThe Ġbest Ġway Ġto Ġdo Ġthis Ġis Ġto Ġhave Ġthem Ġchecked Ġby Ġanother Ġhuman . ĠHence , Ġwe Ġprovide Ġthe Ġexample Ġto Ġhuman Ġver ifiers . ĠIf Ġtwo Ġhuman Ġver ifiers Ġagree Ġwith Ġthe Ġwriter , Ġthe Ġexample Ġis Ġconsidered Ġa Ġgood Ġexample . ĠIf Ġthey Ġdisagree , Ġwe Ġask Ġa Ġthird Ġhuman Ġver ifier Ġto Ġbreak Ġthe Ġtie . ĠIf Ġthere Ġis Ġstill Ġdisagreement Ġbetween Ġthe Ġwriter Ġand Ġthe Ġver ifiers , Ġthe Ġexample Ġis Ġdiscarded . ĠOccasionally , Ġver ifiers Ġwill Ġover rule Ġthe Ġoriginal Ġlabel Ġof Ġthe Ġwriter . </s> Once Ġdata Ġcollection Ġfor Ġthe Ġcurrent Ġround Ġis Ġfinished , Ġwe Ġconstruct Ġa Ġnew Ġtraining Ġset Ġfrom Ġthe Ġcollected Ġdata , Ġwith Ġaccompanying Ġdevelopment Ġand Ġtest Ġsets . ĠWhile Ġthe Ġtraining Ġset Ġincludes Ġcorrectly Ġclassified Ġexamples , Ġthe Ġdevelopment Ġand Ġtests Ġsets Ġare Ġbuilt Ġsolely Ġfrom Ġthem . ĠThe Ġtest Ġset Ġwas Ġfurther Ġrestricted Ġso Ġas Ġto : Ġ1 ) Ġinclude Ġpairs Ġfrom ĠâĢ ľ exclusive âĢ Ŀ Ġannot ators Ġthat Ġare Ġnever Ġincluded Ġin Ġthe Ġtraining Ġdata ; Ġand Ġ2 ) Ġbe Ġbalanced Ġby Ġlabel Ġclasses Ġ( and Ġgenres , Ġwhere Ġapplicable ). ĠWe Ġsubsequently Ġtrain Ġa Ġnew Ġmodel Ġon Ġthis Ġand Ġother Ġexisting Ġdata , Ġand Ġrepeat Ġthe Ġprocedure Ġthree Ġtimes . </s> Dat as et Ġcollection Ġ: :: ĠAn notation Ġdetails </s> We Ġemployed Ġcrowds ourced Ġworkers Ġfrom ĠMechanical ĠTurk Ġwith Ġqualifications . ĠWe Ġcollected Ġhypotheses Ġvia Ġthe ĠPar l AI Ġframework . ĠAn not ators Ġare Ġpresented Ġwith Ġa Ġcontext Ġand Ġa Ġtarget Ġlabel âĢĶ either Ġ` ent ail ment ', Ġ` cont rad iction ', Ġor Ġ` neutral ' âĢĶ and Ġasked Ġto Ġwrite Ġa Ġhypothesis Ġthat Ġcorresponds Ġto Ġthe Ġlabel . ĠWe Ġphrase Ġthe Ġlabel Ġclasses Ġas ĠâĢ ľ def initely Ġcorrect âĢ Ŀ , ĠâĢ ľ def initely Ġincorrect âĢ Ŀ , Ġor ĠâĢ ľ ne ither Ġdefinitely Ġcorrect Ġnor Ġdefinitely Ġincorrect âĢ Ŀ Ġgiven Ġthe Ġcontext , Ġto Ġmake Ġthe Ġtask Ġeasier Ġto Ġgrasp . ĠSub mitted Ġhypotheses Ġare Ġgiven Ġto Ġthe Ġmodel Ġto Ġmake Ġa Ġprediction Ġfor Ġthe Ġcontext - hyp ot hesis Ġpair . ĠThe Ġprobability Ġof Ġeach Ġlabel Ġis Ġreturned Ġto Ġthe Ġworker Ġas Ġfeedback . ĠIf Ġthe Ġmodel Ġpredicts Ġthe Ġlabel Ġincorrectly , Ġthe Ġjob Ġis Ġcomplete . ĠIf Ġnot , Ġthe Ġworker Ġcontinues Ġto Ġwrite Ġhypotheses Ġfor Ġthe Ġgiven Ġ( context , Ġtarget - label ) Ġpair Ġuntil Ġthe Ġmodel Ġpredicts Ġthe Ġlabel Ġincorrectly Ġor Ġthe Ġnumber Ġof Ġtries Ġexceeds Ġa Ġthreshold Ġ( 5 Ġtries Ġin Ġthe Ġfirst Ġround , Ġ10 Ġtries Ġthereafter ). ĠTo Ġencourage Ġworkers , Ġpayments Ġincreased Ġas Ġrounds Ġbecame Ġharder . ĠFor Ġhypotheses Ġthat Ġthe Ġmodel Ġpredicted Ġthe Ġincorrect Ġlabel Ġfor , Ġbut Ġwere Ġverified Ġby Ġother Ġhumans , Ġwe Ġpaid Ġan Ġadditional Ġbonus Ġon Ġtop Ġof Ġthe Ġstandard Ġrate . </s> Dat as et Ġcollection Ġ: :: ĠRound Ġ1 </s> For Ġthe Ġfirst Ġround , Ġwe Ġused Ġa ĠB ERT - Large Ġmodel ĠB IB REF 10 Ġtrained Ġon Ġa Ġconc aten ation Ġof ĠSN LI ĠB IB REF 1 Ġand ĠMN LI ĠB IB REF 22 , Ġand Ġselected Ġthe Ġbest - performing Ġmodel Ġwe Ġcould Ġtrain Ġas Ġthe Ġstarting Ġpoint Ġfor Ġour Ġdataset Ġcollection Ġprocedure . ĠFor ĠRound Ġ1 Ġcontexts , Ġwe Ġrandomly Ġsampled Ġshort Ġmulti - sent ence Ġpassages Ġfrom ĠWikipedia Ġ( of Ġ250 - 600 Ġcharacters ) Ġfrom Ġthe Ġmanually Ġcurated ĠHot pot Q A Ġtraining Ġset ĠB IB REF 23 . ĠContext s Ġare Ġeither Ġground - truth Ġcontexts Ġfrom Ġthat Ġdataset , Ġor Ġthey Ġare ĠWikipedia Ġpassages Ġretrieved Ġusing ĠTF - ID F ĠB IB REF 24 Ġbased Ġon Ġa ĠHot pot Q A Ġquestion . </s> Dat as et Ġcollection Ġ: :: ĠRound Ġ2 </s> For Ġthe Ġsecond Ġround , Ġwe Ġused Ġa Ġmore Ġpowerful ĠRo BER Ta Ġmodel ĠB IB REF 25 Ġtrained Ġon ĠSN LI , ĠMN LI , Ġan ĠNL I - version Ġof ĠFE VER ĠB IB REF 26 , Ġand Ġthe Ġtraining Ġdata Ġfrom Ġthe Ġprevious Ġround Ġ( A 1 ). ĠAfter Ġa Ġhyper param eter Ġsearch , Ġwe Ġselected Ġthe Ġmodel Ġwith Ġthe Ġbest Ġperformance Ġon Ġthe ĠA 1 Ġdevelopment Ġset . ĠThen , Ġusing Ġthe Ġhyper param eters Ġselected Ġfrom Ġthis Ġsearch , Ġwe Ġcreated Ġa Ġfinal Ġset Ġof Ġmodels Ġby Ġtraining Ġseveral Ġmodels Ġwith Ġdifferent Ġrandom Ġseeds . ĠDuring Ġannotation , Ġwe Ġconstructed Ġan Ġensemble Ġby Ġrandomly Ġpicking Ġa Ġmodel Ġfrom Ġthe Ġmodel Ġset Ġas Ġthe Ġadversary Ġeach Ġturn . ĠThis Ġhelps Ġus Ġavoid Ġannot ators Ġexploiting Ġvulnerabilities Ġin Ġone Ġsingle Ġmodel . ĠA Ġnew Ġnon - over l apping Ġset Ġof Ġcontexts Ġwas Ġagain Ġconstructed Ġfrom ĠWikipedia Ġvia ĠHot pot Q A Ġusing Ġthe Ġsame Ġmethod Ġas ĠRound Ġ1 . </s> Dat as et Ġcollection Ġ: :: ĠRound Ġ3 </s> For Ġthe Ġthird Ġround , Ġwe Ġselected Ġa Ġmore Ġdiverse Ġset Ġof Ġcontexts , Ġin Ġorder Ġto Ġexplore Ġrobust ness Ġunder Ġdomain Ġtransfer . ĠIn Ġaddition Ġto Ġcontexts Ġfrom ĠWikipedia Ġfor ĠRound Ġ3 , Ġwe Ġalso Ġincluded Ġcontexts Ġfrom Ġthe Ġfollowing Ġdomains : ĠNews Ġ( ext racted Ġfrom ĠCommon ĠC rawl ), Ġfiction Ġ( ext racted Ġfrom ĠB IB REF 27 , Ġand ĠB IB REF 28 ), Ġformal Ġspoken Ġtext Ġ( ex cerpt ed Ġfrom Ġcourt Ġand Ġpresidential Ġdebate Ġtranscripts Ġin Ġthe ĠMan ually ĠAn not ated ĠSub - Corp us Ġ( MAS C ) Ġof Ġthe ĠOpen ĠAmerican ĠNational ĠCorpus ), Ġand Ġcausal Ġor Ġprocedural Ġtext , Ġwhich Ġdescribes Ġsequences Ġof Ġevents Ġor Ġactions , Ġextracted Ġfrom ĠWiki How . ĠFinally , Ġwe Ġalso Ġcollected Ġannotations Ġusing Ġthe Ġlonger Ġcontexts Ġpresent Ġin Ġthe ĠGL UE ĠR TE Ġtraining Ġdata , Ġwhich Ġcame Ġfrom Ġthe ĠR TE 5 Ġdataset ĠB IB REF 29 . ĠWe Ġtrained Ġan Ġeven Ġstronger ĠRo BER Ta Ġmodel Ġby Ġadding Ġthe Ġtraining Ġset Ġfrom Ġthe Ġsecond Ġround Ġ( A 2 ) Ġto Ġthe Ġtraining Ġdata . </s> Dat as et Ġcollection Ġ: :: ĠComp aring Ġwith Ġother Ġdatasets </s> The ĠAN LI Ġdataset Ġimproves Ġupon Ġprevious Ġwork Ġin Ġseveral Ġways . ĠFirst , Ġand Ġmost Ġobviously , Ġthe Ġdataset Ġis Ġcollected Ġto Ġbe Ġmore Ġdifficult Ġthan Ġprevious Ġdatasets , Ġby Ġdesign . ĠSecond , Ġit Ġremedies Ġa Ġproblem Ġwith ĠSN LI , Ġnamely Ġthat Ġits Ġcontexts Ġ( or Ġpremises ) Ġare Ġvery Ġshort , Ġbecause Ġthey Ġwere Ġselected Ġfrom Ġthe Ġimage Ġcaption ing Ġdomain . ĠWe Ġbelieve Ġlonger Ġcontexts Ġshould Ġnaturally Ġlead Ġto Ġharder Ġexamples , Ġand Ġso Ġwe Ġconstructed ĠAN LI Ġcontexts Ġfrom Ġlonger , Ġmulti - sent ence Ġsource Ġmaterial . </s> Following Ġprevious Ġobservations Ġthat Ġmodels Ġmight Ġexploit Ġspurious Ġbiases Ġin ĠNL I Ġhypotheses , ĠB IB REF 12 , ĠB IB REF 13 , Ġwe Ġconduct Ġa Ġstudy Ġof Ġthe Ġperformance Ġof Ġhypothesis - only Ġmodels Ġon Ġour Ġdataset . ĠWe Ġshow Ġthat Ġsuch Ġmodels Ġperform Ġpoorly Ġon Ġour Ġtest Ġsets . </s> With Ġrespect Ġto Ġdata Ġgeneration Ġwith ĠnaÃ¯ve Ġannot ators , ĠG eva 2019 task or annot ator Ġnoted Ġthat Ġmodels Ġmight Ġpick Ġup Ġon Ġannot ator Ġbias , Ġmodelling Ġthe Ġannot ators Ġthemselves Ġrather Ġthan Ġcapturing Ġthe Ġintended Ġreasoning Ġphenomenon . ĠTo Ġcounter Ġthis , Ġwe Ġselected Ġa Ġsubset Ġof Ġannot ators Ġ( i . e ., Ġthe ĠâĢ ľ exclusive âĢ Ŀ Ġworkers ) Ġwhose Ġdata Ġwould Ġonly Ġbe Ġincluded Ġin Ġthe Ġtest Ġset . ĠThis Ġenables Ġus Ġto Ġavoid Ġover fitting Ġto Ġthe Ġwriting Ġstyle Ġbiases Ġof Ġparticular Ġannot ators , Ġand Ġalso Ġto Ġdetermine Ġhow Ġmuch Ġindividual Ġannot ator Ġbias Ġis Ġpresent Ġfor Ġthe Ġmain Ġportion Ġof Ġthe Ġdata . ĠExamples Ġfrom Ġeach Ġround Ġof Ġdataset Ġcollection Ġare Ġprovided Ġin ĠTable ĠT AB REF 2 . </s> Furthermore , Ġour Ġdataset Ġposes Ġnew Ġchallenges Ġto Ġthe Ġcommunity Ġthat Ġwere Ġless Ġrelevant Ġfor Ġprevious Ġwork , Ġsuch Ġas : Ġcan Ġwe Ġimprove Ġperformance Ġonline Ġwithout Ġhaving Ġto Ġtrain Ġa Ġnew Ġmodel Ġfrom Ġscratch Ġevery Ġround , Ġhow Ġcan Ġwe Ġovercome Ġcatastrophic Ġforgetting , Ġhow Ġdo Ġwe Ġdeal Ġwith Ġmixed Ġmodel Ġbiases , Ġetc . ĠBecause Ġthe Ġtraining Ġset Ġincludes Ġexamples Ġthat Ġthe Ġmodel Ġgot Ġright Ġbut Ġwere Ġnot Ġverified , Ġit Ġmight Ġbe Ġnoisy , Ġposing Ġfiltering Ġas Ġan Ġadditional Ġinteresting Ġproblem . </s> Dat as et Ġstatistics </s> The Ġdataset Ġstatistics Ġcan Ġbe Ġfound Ġin ĠTable ĠT AB REF 7 . ĠThe Ġnumber Ġof Ġexamples Ġwe Ġcollected Ġincreases Ġper Ġround , Ġstarting Ġwith Ġapproximately Ġ19 k Ġexamples Ġfor ĠRound Ġ1 , Ġto Ġaround Ġ47 k Ġexamples Ġfor ĠRound Ġ2 , Ġto Ġover Ġ103 k Ġexamples Ġfor ĠRound Ġ3 . ĠWe Ġcollected Ġmore Ġdata Ġfor Ġlater Ġrounds Ġnot Ġonly Ġbecause Ġthat Ġdata Ġis Ġlikely Ġto Ġbe Ġmore Ġinteresting , Ġbut Ġalso Ġsimply Ġbecause Ġthe Ġbase Ġmodel Ġis Ġbetter Ġand Ġso Ġannotation Ġtook Ġlonger Ġto Ġcollect Ġgood , Ġverified Ġcorrect Ġexamples Ġof Ġmodel Ġvulnerabilities . </s> For Ġeach Ġround , Ġwe Ġreport Ġthe Ġmodel Ġerror Ġrate , Ġboth Ġon Ġverified Ġand Ġun verified Ġexamples . ĠThe Ġun verified Ġmodel Ġerror Ġrate Ġcaptures Ġthe Ġpercentage Ġof Ġexamples Ġwhere Ġthe Ġmodel Ġdisagreed Ġwith Ġthe Ġwriter 's Ġtarget Ġlabel , Ġbut Ġwhere Ġwe Ġare Ġnot Ġ( yet ) Ġsure Ġif Ġthe Ġexample Ġis Ġcorrect . ĠThe Ġverified Ġmodel Ġerror Ġrate Ġis Ġthe Ġpercentage Ġof Ġmodel Ġerrors Ġfrom Ġexample Ġpairs Ġthat Ġother Ġannot ators Ġwere Ġable Ġto Ġconfirm Ġthe Ġcorrect Ġlabel Ġfor . ĠNote Ġthat Ġthis Ġerror Ġrate Ġrepresents Ġa Ġstraightforward Ġway Ġto Ġevaluate Ġmodel Ġquality : Ġthe Ġlower Ġthe Ġmodel Ġerror Ġrate âĢĶ assuming Ġconstant Ġannot ator Ġquality Ġand Ġcontext - diff iculty âĢĶ the Ġbetter Ġthe Ġmodel . </s> We Ġobserve Ġthat Ġmodel Ġerror Ġrates Ġdecrease Ġas Ġwe Ġprogress Ġthrough Ġrounds . ĠIn ĠRound Ġ3 , Ġwhere Ġwe Ġincluded Ġa Ġmore Ġdiverse Ġrange Ġof Ġcontexts Ġfrom Ġvarious Ġdomains , Ġthe Ġoverall Ġerror Ġrate Ġwent Ġslightly Ġup Ġcompared Ġto Ġthe Ġpreceding Ġround , Ġbut Ġfor ĠWikipedia Ġcontexts Ġthe Ġerror Ġrate Ġdecreased Ġsubstantially . ĠWhile Ġfor Ġthe Ġfirst Ġround Ġroughly Ġ1 Ġin Ġevery Ġ5 Ġexamples Ġwere Ġverified Ġmodel Ġerrors , Ġthis Ġquickly Ġdropped Ġover Ġconsecutive Ġrounds , Ġand Ġthe Ġoverall Ġmodel Ġerror Ġrate Ġis Ġless Ġthan Ġ1 Ġin Ġ10 . ĠOn Ġthe Ġone Ġhand , Ġthis Ġis Ġimpressive , Ġand Ġshows Ġhow Ġfar Ġwe Ġhave Ġcome Ġwith Ġjust Ġthree Ġrounds . ĠOn Ġthe Ġother Ġhand , Ġit Ġshows Ġthat Ġwe Ġstill Ġhave Ġa Ġlong Ġway Ġto Ġgo Ġif Ġeven Ġunt rained Ġannot ators Ġcan Ġfool Ġen semb les Ġof Ġstate - of - the - art Ġmodels Ġwith Ġrelative Ġease . </s> Table ĠT AB REF 7 Ġalso Ġreports Ġthe Ġaverage Ġnumber Ġof ĠâĢ ľ t ries âĢ Ŀ , Ġi . e ., Ġattempts Ġmade Ġfor Ġeach Ġcontext Ġuntil Ġa Ġmodel Ġerror Ġwas Ġfound Ġ( or Ġthe Ġnumber Ġof Ġpossible Ġtries Ġis Ġexceeded ), Ġand Ġthe Ġaverage Ġtime Ġthis Ġtook Ġ( in Ġseconds ). ĠAgain , Ġthese Ġmetrics Ġrepresent Ġa Ġuseful Ġway Ġto Ġevaluate Ġmodel Ġquality . ĠWe Ġobserve Ġthat Ġthe Ġaverage Ġtries Ġand Ġaverage Ġtime Ġper Ġverified Ġerror Ġboth Ġgo Ġup Ġas Ġwe Ġprogress Ġthrough Ġthe Ġrounds . ĠThe Ġnumbers Ġclearly Ġdemonstrate Ġthat Ġthe Ġrounds Ġare Ġgetting Ġincreasingly Ġmore Ġdifficult . </s> Results </s> Table ĠT AB REF 13 Ġreports Ġthe Ġmain Ġresults . ĠIn Ġaddition Ġto ĠB ERT ĠB IB REF 10 Ġand ĠRo BER Ta ĠB IB REF 25 , Ġwe Ġalso Ġinclude ĠXL Net ĠB IB REF 30 Ġas Ġan Ġexample Ġof Ġa Ġstrong , Ġbut Ġdifferent , Ġmodel Ġarchitecture . ĠWe Ġshow Ġtest Ġset Ġperformance Ġon Ġthe ĠAN LI Ġtest Ġsets Ġper Ġround , Ġthe Ġtotal ĠAN LI Ġtest Ġset , Ġand Ġthe Ġexclusive Ġtest Ġsubset Ġ( ex amples Ġfrom Ġtest - set - exclusive Ġworkers ). ĠWe Ġalso Ġshow Ġaccuracy Ġon Ġthe ĠSN LI Ġtest Ġset Ġand Ġthe ĠMN LI Ġdevelopment Ġ( for Ġthe Ġpurpose Ġof Ġcomparing Ġbetween Ġdifferent Ġmodel Ġconfigurations Ġacross Ġtable Ġrows ) Ġset . ĠIn Ġwhat Ġfollows , Ġwe Ġbriefly Ġdiscuss Ġour Ġobservations . </s> Results Ġ: :: ĠBase Ġmodel Ġperformance Ġis Ġlow . </s> Notice Ġthat Ġthe Ġbase Ġmodel Ġfor Ġeach Ġround Ġperforms Ġvery Ġpoorly Ġon Ġthat Ġround 's Ġtest Ġset . ĠThis Ġis Ġthe Ġexpected Ġoutcome : ĠFor Ġround Ġ1 , Ġthe Ġbase Ġmodel Ġgets Ġthe Ġentire Ġtest Ġset Ġwrong , Ġby Ġdesign . ĠFor Ġrounds Ġ2 Ġand Ġ3 , Ġwe Ġused Ġan Ġensemble , Ġso Ġperformance Ġis Ġnot Ġnecessarily Ġzero . ĠHowever , Ġas Ġit Ġturns Ġout , Ġperformance Ġstill Ġfalls Ġwell Ġbelow Ġchance , Ġindicating Ġthat Ġworkers Ġdid Ġnot Ġfind Ġvulnerabilities Ġspecific Ġto Ġa Ġsingle Ġmodel , Ġbut Ġgenerally Ġapplicable Ġones Ġfor Ġthat Ġmodel Ġclass . </s> Results Ġ: :: ĠRounds Ġbecome Ġincreasingly Ġmore Ġdifficult . </s> As Ġalready Ġfore shadow ed Ġby Ġthe Ġdataset Ġstatistics , Ġround Ġ3 Ġis Ġmore Ġdifficult Ġ( y ield s Ġlower Ġperformance ) Ġthan Ġround Ġ2 , Ġand Ġround Ġ2 Ġis Ġmore Ġdifficult Ġthan Ġround Ġ1 . ĠThis Ġis Ġtrue Ġfor Ġall Ġmodel Ġarchitectures . </s> Results Ġ: :: ĠTraining Ġon Ġmore Ġrounds Ġimproves Ġrobust ness . </s> Generally , Ġour Ġresults Ġindicate Ġthat Ġtraining Ġon Ġmore Ġrounds Ġimproves Ġmodel Ġperformance . ĠThis Ġis Ġtrue Ġfor Ġall Ġmodel Ġarchitectures . ĠSimply Ġtraining Ġon Ġmore ĠâĢ ľ normal ĠNL I âĢ Ŀ Ġdata Ġwould Ġnot Ġhelp Ġa Ġmodel Ġbe Ġrobust Ġto Ġadvers arial Ġattacks , Ġbut Ġour Ġdata Ġactively Ġhelps Ġmitigate Ġthese . </s> Results Ġ: :: ĠRo BER Ta Ġachieves Ġstate - of - the - art Ġperformance ... </s> We Ġobtain Ġstate Ġof Ġthe Ġart Ġperformance Ġon Ġboth ĠSN LI Ġand ĠMN LI Ġwith Ġthe ĠRo BER Ta Ġmodel Ġfin et uned Ġon Ġour Ġnew Ġdata ĠThe ĠRo BER Ta Ġpaper ĠB IB REF 25 Ġreports Ġa Ġscore Ġof Ġ$ 90 . 2 $ Ġfor Ġboth ĠMN LI - matched Ġand Ġ- m ism atched Ġdev , Ġwhile Ġwe Ġobtain Ġ$ 91 . 0 $ Ġand Ġ$ 90 . 7 $. ĠThe Ġstate Ġof Ġthe Ġart Ġon ĠSN LI Ġis Ġcurrently Ġheld Ġby ĠMT - D NN ĠB IB REF 31 , Ġwhich Ġreports Ġ$ 91 . 6 $ Ġcompared Ġto Ġour Ġ$ 92 . 9 $. </s> Results Ġ: :: Ġ... but Ġis Ġoutper formed Ġwhen Ġit Ġis Ġbase Ġmodel . </s> However , Ġthe Ġbase Ġ( Ro BER Ta ) Ġmodels Ġfor Ġrounds Ġ2 Ġand Ġ3 Ġare Ġoutper formed Ġby Ġboth ĠB ERT Ġand ĠXL Net . ĠThis Ġshows Ġthat Ġannot ators Ġhave Ġmanaged Ġto Ġwrite Ġexamples Ġthat ĠRo BER Ta Ġgenerally Ġstruggles Ġwith , Ġand Ġmore Ġtraining Ġdata Ġalone Ġcannot Ġeasily Ġmit igated Ġthese Ġshortcomings . ĠIt Ġalso Ġimplies Ġthat ĠB ERT , ĠXL Net , Ġand ĠRo BER Ta Ġall Ġhave Ġdifferent Ġweaknesses , Ġpossibly Ġas Ġa Ġfunction Ġof Ġtheir Ġtraining Ġdata Ġ( BER T , ĠXL Net Ġand ĠRo BER Ta Ġwere Ġtrained Ġon Ġvery Ġdifferent Ġdata Ġsets , Ġwhich Ġmight Ġor Ġmight Ġnot Ġhave Ġcontained Ġinformation Ġrelevant Ġto Ġthe Ġweaknesses )âĢĶ an Ġadditional Ġround Ġwith Ġa Ġwider Ġmodel Ġvariety Ġwould Ġthus Ġbe Ġinteresting Ġto Ġinvestigate Ġas Ġa Ġnext Ġstep . </s> Results Ġ: :: ĠContin uously Ġaugment ing Ġtraining Ġdata Ġdoes Ġnot Ġdowngrade Ġperformance . </s> Even Ġthough ĠAN LI Ġtraining Ġdata Ġis Ġdifferent Ġfrom ĠSN LI Ġand ĠMN LI , Ġadding Ġthis Ġdata Ġto Ġthe Ġtraining Ġset Ġdoes Ġnot Ġharm Ġperformance Ġon Ġthose Ġtasks . ĠFurthermore , Ġas ĠTable ĠT AB REF 21 Ġshows , Ġtraining Ġonly Ġon ĠAN LI Ġis Ġtransfer able Ġto ĠSN LI Ġand ĠMN LI , Ġbut Ġnot Ġvice Ġversa . ĠThis Ġsuggests Ġthat Ġmethods Ġcould Ġsuccessfully Ġbe Ġapplied Ġfor Ġmany Ġmore Ġconsecutive Ġrounds . </s> Results Ġ: :: ĠExclusive Ġtest Ġsubset Ġdifference Ġis Ġsmall . </s> In Ġorder Ġto Ġavoid Ġthe Ġpossibility Ġthat Ġmodels Ġmight Ġpick Ġup Ġon Ġannot ator - specific Ġartifacts , Ġa Ġconcern Ġraised Ġby ĠG eva 2019 task or annot ator , Ġwe Ġincluded Ġan Ġexclusive Ġtest Ġsubset Ġwith Ġexamples Ġfrom Ġannot ators Ġnever Ġseen Ġin Ġthe Ġtraining Ġdata . ĠWe Ġfind Ġthat Ġthe Ġdifferences Ġbetween Ġthis Ġexclusive Ġsubset Ġand Ġthe Ġtest Ġset Ġare Ġsmall , Ġindicating Ġthat Ġour Ġmodels Ġdo Ġnot Ġover - rely Ġon Ġindividual Ġannot ator 's Ġwriting Ġstyles . </s> Results Ġ: :: ĠHyp ot hesis - only Ġresults </s> For ĠSN LI Ġand ĠMN LI , Ġconcerns Ġhave Ġbeen Ġraised Ġabout Ġthe Ġpropensity Ġof Ġmodels Ġto Ġpick Ġup Ġon Ġspurious Ġartifacts Ġthat Ġare Ġpresent Ġjust Ġin Ġthe Ġhypotheses ĠB IB REF 12 , ĠB IB REF 13 . ĠTo Ġstudy Ġthis Ġin Ġthe Ġcontext Ġof Ġour Ġresults Ġand Ġtask Ġdifficulty , Ġwe Ġcompare Ġmodels Ġtrained Ġon Ġ( context , Ġhypothesis ) Ġpairs Ġto Ġmodels Ġtrained Ġonly Ġon Ġthe Ġhypothesis Ġ( marked Ġ$ H $ ). ĠTable ĠT AB REF 21 Ġreports Ġresults Ġon Ġthe Ġthree Ġrounds Ġof ĠAN LI , Ġas Ġwell Ġas ĠSN LI Ġand ĠMN LI . ĠThe Ġtable Ġshows Ġsome Ġinteresting Ġtake - aways : </s> Results Ġ: :: ĠHyp ot hesis - only Ġresults Ġ: :: ĠHyp ot hesis - only Ġmodels Ġperform Ġpoorly Ġon ĠAN LI . </s> We Ġcorrobor ate Ġthat Ġhypothesis - only Ġmodels Ġobtain Ġgood Ġperformance Ġon ĠSN LI Ġand ĠMN LI . ĠPerformance Ġof Ġsuch Ġmodels Ġon ĠAN LI Ġis Ġsubstantially Ġlower , Ġand Ġdecreases Ġwith Ġmore Ġrounds . </s> Results Ġ: :: ĠHyp ot hesis - only Ġresults Ġ: :: ĠRo BER Ta Ġdoes Ġnot Ġoutper form Ġhypothesis - only Ġon Ġrounds Ġ2 Ġand Ġ3 . </s> On Ġthe Ġtwo Ġrounds Ġwhere ĠRo BER Ta Ġwas Ġused Ġas Ġthe Ġbase Ġmodel , Ġits Ġperformance Ġis Ġnot Ġmuch Ġbetter Ġthan Ġthe Ġhypothesis - only Ġmodel . ĠThis Ġcould Ġmean Ġtwo Ġthings : Ġeither Ġthe Ġtest Ġdata Ġis Ġvery Ġdifficult , Ġor Ġthe Ġtraining Ġdata Ġis Ġnot Ġgood . ĠTo Ġrule Ġout Ġthe Ġlatter , Ġwe Ġtrained Ġonly Ġon ĠAN LI Ġ($ \ sim Ġ$ 163 k Ġtraining Ġexamples ): Ġdoing Ġso Ġwith ĠRo BER Ta Ġmatches Ġthe Ġperformance Ġof ĠB ERT Ġon ĠMN LI Ġwhen Ġit Ġis Ġtrained Ġon Ġthe Ġmuch Ġlarger , Ġfully Ġin - domain ĠSN LI + MN LI Ġcombined Ġdataset Ġ( 9 43 k Ġtraining Ġexamples ), Ġwith Ġboth Ġgetting Ġ$\ sim Ġ$ 86 , Ġwhich Ġis Ġimpressive . ĠHence , Ġthis Ġshows Ġthat Ġour Ġnew Ġchallenge Ġtest Ġsets Ġare Ġso Ġdifficult Ġthat Ġthe Ġcurrent Ġstate - of - the - art Ġmodel Ġcannot Ġdo Ġbetter Ġthan Ġa Ġhypothesis - only Ġprior . </s> Analysis </s> We Ġperform Ġtwo Ġtypes Ġof Ġmodel Ġerror Ġanalysis . ĠFirst Ġwe Ġevaluate Ġtwo Ġpopular Ġexisting Ġtest Ġsets Ġthat Ġwere Ġcreated Ġto Ġexpose Ġmodel Ġweaknesses , Ġand Ġshow Ġthat Ġour Ġdataset Ġdiscour ages Ġmodels Ġfrom Ġlearning Ġspurious Ġstatistical Ġfacts , Ġrelative Ġto Ġother Ġlarge Ġpopular Ġdatasets Ġ( e . g ., ĠSN LI Ġand ĠMN LI ). ĠSecondly , Ġwe Ġexplore , Ġby Ġround , Ġthe Ġtypes Ġof Ġinf erences Ġour Ġwriters Ġsuccessfully Ġemployed Ġto Ġstump Ġmodels , Ġby Ġperforming Ġhand - ann otation Ġon Ġ500 Ġexamples Ġfrom Ġeach Ġround 's Ġdevelopment Ġset . </s> Analysis Ġ: :: ĠPerformance Ġon Ġchallenge Ġdatasets </s> Recently , Ġseveral Ġhard Ġtest Ġsets Ġhave Ġbeen Ġmade Ġavailable Ġfor Ġrevealing Ġthe Ġbiases ĠNL I Ġmodels Ġlearn Ġfrom Ġtheir Ġtraining Ġdatasets ĠB IB REF 32 , ĠB IB REF 17 , ĠB IB REF 12 , ĠB IB REF 33 . ĠWe Ġexamined Ġmodel Ġperformance Ġon Ġtwo Ġof Ġthese : Ġthe ĠSN LI - Hard ĠB IB REF 12 Ġtest Ġset , Ġwhich Ġconsists Ġof Ġexamples Ġthat Ġhypothesis - only Ġmodels Ġlabel Ġincorrectly , Ġand Ġthe ĠNL I Ġstress Ġtests ĠB IB REF 33 , Ġin Ġwhich Ġsentences Ġcontaining Ġan ton y ms Ġpairs , Ġneg ations , Ġhigh Ġword Ġoverlap , Ġi . a ., Ġare Ġhe ur istically Ġconstructed . ĠWe Ġtest Ġour Ġmodels Ġon Ġthese Ġstress Ġtests , Ġafter Ġtuning Ġon Ġeach Ġtest 's Ġrespective Ġdevelopment Ġset Ġto Ġaccount Ġfor Ġpotential Ġdomain Ġmism atches . ĠFor Ġcomparison , Ġwe Ġalso Ġreport Ġaccur acies Ġfrom Ġthe Ġoriginal Ġpapers : Ġfor ĠSN LI - Hard Ġwe Ġpresent Ġthe Ġresults Ġfrom ĠB IB REF 12 's Ġimplementation Ġof Ġthe Ġhierarchical Ġtens or - based ĠD ense ly ĠInteractive ĠIn ference ĠNetwork ĠB IB REF 34 Ġon ĠMN LI , Ġand Ġfor Ġthe ĠNL I Ġstress Ġtests , Ġwe Ġpresent Ġthe Ġperformance Ġof ĠB IB REF 33 's Ġimplementation Ġof ĠIn fer Sent ĠB IB REF 3 Ġtrained Ġon ĠSN LI . ĠOur Ġresults Ġare Ġin ĠTable ĠT AB REF 23 . </s> We Ġobserve Ġthat Ġall Ġof Ġour Ġmodels Ġfar Ġoutper form Ġthe Ġmodels Ġpresented Ġin Ġoriginal Ġpapers Ġfor Ġthese Ġcommon Ġstress Ġtests , Ġwith Ġour Ġtwo ĠRo BER Ta Ġmodels Ġperforming Ġbest . ĠBoth Ġperform Ġwell Ġon ĠSN LI - Hard Ġand Ġachieve Ġaccuracy Ġlevels Ġin Ġthe Ġhigh Ġ80 s Ġon Ġthe Ġ` ant onym ' Ġ( AT ), Ġ` n umer ical Ġreasoning ' Ġ( NR ), Ġ` length ' Ġ( L N ), Ġ` spe lling Ġerror ' ( SE ) Ġsub - dat as ets , Ġand Ġshow Ġmarked Ġimprovement Ġon Ġboth Ġ` neg ation ' Ġ( NG ), Ġand Ġ` word Ġoverlap ' Ġ( W O ). ĠTraining Ġa ĠRo BER Ta Ġmodel Ġalso Ġon ĠAN LI Ġappears Ġto Ġbe Ġparticularly Ġuseful Ġfor Ġthe ĠNR , ĠW O , ĠNG Ġand ĠAT ĠNL I Ġstress Ġtests . </s> Analysis Ġ: :: ĠReason ing Ġtypes </s> A Ġdynamically Ġevolving Ġdataset Ġoffers Ġthe Ġunique Ġopportunity Ġto Ġtrack Ġhow Ġmodel Ġerror Ġrates Ġchange Ġover Ġtime . ĠSince Ġeach Ġround 's Ġdevelopment Ġset Ġcontains Ġonly Ġverified Ġexamples , Ġwe Ġcan Ġinvestigate Ġtwo Ġinteresting Ġquestions : Ġwhich Ġtypes Ġof Ġinference Ġdo Ġwriters Ġemploy Ġto Ġfool Ġthe Ġmodels , Ġand Ġare Ġbase Ġmodels Ġdifferent ially Ġsensitive Ġto Ġdifferent Ġtypes Ġof Ġreasoning ? ĠThe Ġresults Ġare Ġsummarized Ġin ĠTable ĠT AB REF 27 . </s> We Ġemployed Ġan Ġexpert Ġlingu ist Ġannot ator Ġto Ġdevise Ġan Ġont ology Ġof Ġinference Ġtypes Ġthat Ġwould Ġbe Ġspecific Ġto ĠNL I . ĠWhile Ġdesigning Ġan Ġappropriate Ġont ology Ġof Ġtypes Ġof Ġinference Ġis Ġfar Ġfrom Ġstraightforward , Ġwe Ġfound Ġthat Ġa Ġunified Ġont ology Ġcould Ġbe Ġutilized Ġto Ġcharacterize Ġexamples Ġfrom Ġall Ġthree Ġrounds , Ġwhich Ġsuggests Ġthat Ġit Ġhas Ġat Ġleast Ġsome Ġgeneral ize able Ġapplic ability . ĠThe Ġont ology Ġwas Ġused Ġto Ġlabel Ġ500 Ġexamples Ġfrom Ġeach ĠAN LI Ġdevelopment Ġset . </s> The Ġinference Ġont ology Ġcontains Ġsix Ġtypes Ġof Ġinference : ĠN umer ical Ġ& ĠQuant itative Ġ( i . e ., Ġreasoning Ġabout Ġcardinal Ġand Ġord inal Ġnumbers , Ġinfer ring Ġdates Ġand Ġages Ġfrom Ġnumbers , Ġetc .), ĠReference Ġ& ĠNames Ġ( core f erences Ġbetween Ġpronouns Ġand Ġforms Ġof Ġproper Ġnames , Ġknowing Ġfacts Ġabout Ġname Ġgender , Ġetc .), ĠBasic ĠInf erences Ġ( con jun ctions , Ġneg ations , Ġcause - and - effect , Ġcompar atives Ġand Ġsuper l atives Ġetc .), ĠLex ical ĠIn ference Ġ( inf erences Ġmade Ġpossible Ġby Ġlex ical Ġinformation Ġabout Ġsyn onyms , Ġan ton y ms , Ġetc .), ĠTr icky ĠInf erences Ġ( word play , Ġlinguistic Ġstrategies Ġsuch Ġas Ġsynt actic Ġtransformations / re order ings , Ġor Ġinfer ring Ġwriter Ġintentions Ġfrom Ġcontexts ), Ġand Ġreasoning Ġfrom Ġoutside Ġknowledge Ġor Ġadditional Ġfacts Ġ( e . g ., ĠâĢ ľ You Ġcan 't Ġreach Ġthe Ġsea Ġdirectly Ġfrom ĠDj ib out i âĢ Ŀ ). ĠThe Ġquality Ġof Ġannotations Ġwas Ġalso Ġtracked ; Ġif Ġa Ġpair Ġwas Ġambiguous Ġor Ġhad Ġa Ġlabel Ġthat Ġseemed Ġincorrect Ġ( from Ġthe Ġexpert Ġannot ator 's Ġperspective ), Ġit Ġwas Ġflagged . ĠRound Ġ1 âĢĵ 3 Ġdevelopment Ġsets Ġcontained Ġfew Ġ` Quality ' Ġtags ; Ġthe Ġincidence Ġof Ġquality Ġissues Ġwas Ġstable Ġat Ġbetween Ġ3 % Ġand Ġ4 % Ġper Ġround . ĠAny Ġone Ġexample Ġcan Ġhave Ġmultiple Ġtypes , Ġand Ġevery Ġexample Ġcontained Ġat Ġleast Ġone Ġtag . </s> As Ġrounds Ġ1 Ġand Ġ2 Ġwere Ġboth Ġbuilt Ġwith Ġcontexts Ġfrom Ġthe Ġsame Ġgenre Ġ( Wikipedia ), Ġwe Ġmight Ġexpect Ġwriters Ġto Ġarrive Ġat Ġsimilar Ġstrategies . ĠHowever , Ġsince Ġthe Ġmodel Ġarchitectures Ġused Ġin Ġthe Ġfirst Ġtwo Ġrounds Ġdiffer , Ġwriters Ġmight Ġbe Ġsufficiently Ġcreative Ġin Ġfinding Ġdifferent Ġexploits Ġin Ġeach . ĠFor Ġround Ġ3 , Ġwe Ġexpect Ġsome Ġdifference Ġin Ġreasoning Ġtypes Ġto Ġbe Ġpresent , Ġbecause Ġwe Ġused Ġsource Ġmaterial Ġfrom Ġseveral Ġdomains Ġas Ġour Ġcontexts . ĠIn Ġsum , Ġany Ġchange Ġbetween Ġrounds Ġcould Ġbe Ġdue Ġto Ġany Ġof Ġthe Ġfollowing Ġfactors : Ġinherent Ġdifferences Ġbetween Ġdata Ġcollection , Ġmodel Ġarchitectures Ġand Ġmodel Ġtraining Ġdata , Ġrandom Ġselection Ġof Ġcontexts , Ġor Ġslight Ġdifferences Ġin Ġwriter Ġpool Ġor Ġwriter Ġpreferences . </s> We Ġobserve Ġthat Ġboth Ġround Ġ1 Ġand Ġ2 Ġwriters Ġrely Ġheavily Ġon Ġnumerical Ġand Ġquantitative Ġreasoning Ġin Ġover Ġ30 % Ġof Ġthe Ġdevelopment Ġset âĢĶ the Ġpercentage Ġin ĠA 2 Ġ( 32 %) Ġdropped Ġroughly Ġ6 % Ġfrom ĠA 1 Ġ( 38 %) âĢĶ while Ġround Ġ3 Ġwriters Ġuse Ġnumerical Ġor Ġquantitative Ġreasoning Ġfor Ġonly Ġ17 %. ĠThe Ġmajority Ġof Ġnumerical Ġreasoning Ġtypes Ġwere Ġreferences Ġto Ġcardinal Ġnumbers Ġthat Ġreferred Ġto Ġdates Ġand Ġages . ĠInf erences Ġpred icated Ġon Ġreferences Ġand Ġnames Ġwere Ġpresent Ġin Ġabout Ġ10 % Ġof Ġrounds Ġ1 Ġ& Ġ3 Ġdevelopment Ġsets , Ġand Ġreached Ġa Ġhigh Ġof Ġ20 % Ġin Ġround Ġ2 , Ġwith Ġcore ference Ġfeaturing Ġprominently . ĠBasic Ġinference Ġtypes Ġincreased Ġin Ġprevalence Ġas Ġthe Ġrounds Ġincreased , Ġranging Ġfrom Ġ18 % âĢĵ 30 %, Ġas Ġdid ĠLex ical Ġinf erences Ġ( increasing Ġfrom Ġ13 % âĢĵ 33 %). ĠThe Ġpercentage Ġof Ġsentences Ġrelying Ġon Ġreasoning Ġand Ġoutside Ġfacts Ġremains Ġroughly Ġthe Ġsame , Ġin Ġthe Ġmid - 50 s , Ġperhaps Ġslightly Ġincreasing Ġafter Ġround Ġ1 . ĠFor Ġround Ġ3 , Ġwe Ġobserve Ġthat Ġthe Ġmodel Ġused Ġto Ġcollect Ġit Ġappears Ġto Ġbe Ġmore Ġsusceptible Ġto ĠBasic , ĠLex ical , Ġand ĠTr icky Ġinference Ġtypes . ĠThis Ġfinding Ġis Ġcompatible Ġwith Ġthe Ġidea Ġthat Ġthe Ġmodels Ġtrained Ġon Ġadvers arial Ġdata Ġare Ġmore Ġimpressive Ġand Ġperform Ġbetter , Ġencouraging Ġwriters Ġto Ġdevise Ġmore Ġcreative Ġexamples Ġcontaining Ġharder Ġtypes Ġof Ġinference Ġin Ġorder Ġto Ġstump Ġthem . </s> Related Ġwork Ġ: :: ĠB ias Ġin Ġdatasets </s> Machine Ġlearning Ġmethods Ġare Ġwell - known Ġto Ġpick Ġup Ġon Ġspurious Ġstatistical Ġpatterns . ĠFor Ġinstance , Ġin Ġimage Ġcaption ing , Ġa Ġsimple Ġbaseline Ġof Ġutilizing Ġthe Ġcapt ions Ġof Ġnearest Ġneighbors Ġin Ġthe Ġtraining Ġset Ġwas Ġshown Ġto Ġyield Ġimpressive ĠB LE U Ġscores ĠB IB REF 35 . ĠIn Ġthe Ġfirst Ġvisual Ġquestion Ġanswering Ġdataset ĠB IB REF 36 , Ġbiases Ġlike ĠâĢ ľ 2 âĢ Ŀ Ġbeing Ġthe Ġcorrect Ġanswer Ġto Ġ39 % Ġof Ġthe Ġquestions Ġstarting Ġwith ĠâĢ ľ how Ġmany âĢ Ŀ Ġallowed Ġlearning Ġalgorithms Ġto Ġperform Ġwell Ġwhile Ġignoring Ġthe Ġvisual Ġmod ality Ġaltogether ĠB IB REF 37 , ĠB IB REF 38 . ĠThe Ġfield Ġhas Ġa Ġtendency Ġto Ġover fit Ġon Ġstatic Ġtargets , Ġeven Ġif Ġthat Ġdoes Ġnot Ġhappen Ġdeliberately ĠB IB REF 39 . </s> In ĠNL I , ĠGur ur angan 2018 ann otation , ĠPol iak 2018 hyp ot hesis Ġand ĠT such iya 2018 performance Ġshowed Ġthat Ġhypothesis - only Ġbas elines Ġoften Ġperform Ġfar Ġbetter Ġthan Ġchance . ĠIt Ġhas Ġbeen Ġshown Ġthat ĠNL I Ġsystems Ġcan Ġoften Ġbe Ġbroken Ġmerely Ġby Ġperforming Ġsimple Ġlex ical Ġsubstit utions ĠB IB REF 15 , Ġand Ġthat Ġthey Ġstruggle Ġwith Ġquant ifiers ĠB IB REF 40 Ġand Ġcertain Ġsuperficial Ġsynt actic Ġproperties ĠB IB REF 17 . ĠIn Ġreading Ġcomprehension Ġand Ġquestion Ġanswering , ĠKa ush ik 2018 how much Ġshowed Ġthat Ġquestion - Ġand Ġpassage - only Ġmodels Ġcan Ġperform Ġsurprisingly Ġwell , Ġwhile ĠJ ia 2017 ad vers arial Ġadded Ġadvers ar ially Ġconstructed Ġsentences Ġto Ġpassages , Ġleading Ġto Ġa Ġdrastic Ġdrop Ġin Ġperformance . ĠMany Ġtext Ġclassification Ġdatasets Ġdo Ġnot Ġrequire Ġsophisticated Ġlinguistic Ġreasoning , Ġas Ġshown Ġby Ġthe Ġsurprisingly Ġgood Ġperformance Ġof Ġrandom Ġenc od ers ĠB IB REF 41 . ĠSimilar Ġobservations Ġwere Ġmade Ġin Ġmachine Ġtranslation ĠB IB REF 42 Ġand Ġdialogue ĠB IB REF 43 . ĠIn Ġshort , Ġthe Ġfield Ġis Ġrife Ġwith Ġdataset Ġbias Ġand Ġpapers Ġtrying Ġaddress Ġthis Ġimportant Ġproblem . ĠThis Ġwork Ġcan Ġbe Ġviewed Ġas Ġa Ġnatural Ġextension : Ġif Ġsuch Ġbiases Ġexist , Ġthey Ġwill Ġallow Ġhumans Ġto Ġfool Ġthe Ġmodels , Ġadding Ġuseful Ġexamples Ġto Ġthe Ġtraining Ġdata Ġuntil Ġthe Ġbias Ġis Ġdynamically Ġmit igated . </s> Related Ġwork Ġ: :: ĠDynamic Ġdatasets . </s> Con currently Ġwith Ġthis Ġwork , ĠAnonymous 2020 ad vers arial fil ters Ġproposed ĠAFL ite , Ġan Ġiter ative Ġapproach Ġfor Ġfiltering Ġadvers arial Ġdata Ġpoints Ġto Ġavoid Ġspurious Ġbiases . ĠKa ush ik 2019 learning diff erence Ġoffer Ġa Ġcausal Ġaccount Ġof Ġspurious Ġpatterns , Ġand Ġcounter fact ually Ġaugment ĠNL I Ġdatasets Ġby Ġediting Ġexamples Ġto Ġbreak Ġthe Ġmodel . ĠThe Ġformer Ġis Ġan Ġexample Ġof Ġa Ġmodel - in - the - loop Ġsetting , Ġwhere Ġthe Ġmodel Ġis Ġiter atively Ġprob ed Ġand Ġimproved . ĠThe Ġlatter Ġis Ġhuman - in - the - loop Ġtraining , Ġwhere Ġhumans Ġare Ġused Ġto Ġfind Ġproblems Ġwith Ġone Ġsingle Ġmodel . ĠIn Ġthis Ġwork , Ġwe Ġemploy Ġboth Ġstrategies Ġiter atively , Ġin Ġa Ġform Ġof Ġhuman - and - model - in - the - loop Ġtraining , Ġto Ġcollect Ġcompletely Ġnew Ġexamples , Ġin Ġa Ġpotentially Ġnever - ending Ġloop ĠB IB REF 18 . ĠRelated ly , Ġlan 2017 pp db Ġpropose Ġa Ġmethod Ġfor Ġcontinuously Ġgrowing Ġa Ġdataset Ġof Ġparaph r ases . </s> Human - and - model - in - the - loop Ġtraining Ġis Ġnot Ġa Ġnew Ġidea . ĠMechanical ĠTur ker ĠDes cent Ġproposes Ġa Ġgam ified Ġenvironment Ġfor Ġthe Ġcollaborative Ġtraining Ġof Ġgrounded Ġlanguage Ġlearning Ġagents Ġover Ġmultiple Ġrounds ĠB IB REF 19 . ĠThe ĠâĢ ľ Build Ġit ĠBreak Ġit ĠFix Ġit âĢ Ŀ Ġstrategy Ġin Ġthe Ġsecurity Ġdomain ĠB IB REF 44 Ġhas Ġbeen Ġadapted Ġto ĠN LP ĠB IB REF 20 Ġas Ġwell Ġas Ġdialogue ĠB IB REF 45 . ĠThe ĠQ Ap edia Ġframework ĠB IB REF 46 Ġcontinuously Ġref ines Ġand Ġupdates Ġits Ġcontent Ġrepository Ġusing Ġhumans Ġin Ġthe Ġloop , Ġwhile Ġhuman Ġfeedback Ġloops Ġhave Ġbeen Ġused Ġto Ġimprove Ġimage Ġcaption ing Ġsystems ĠB IB REF 47 . Ġwall ace 2018 tr ick Ġleverage Ġtrivia Ġexperts Ġto Ġcreate Ġa Ġmodel - driven Ġadvers arial Ġquestion Ġwriting Ġprocedure Ġand Ġgenerate Ġa Ġsmall Ġset Ġof Ġchallenge Ġquestions Ġthat ĠQ A - models Ġfail Ġon . </s> There Ġhas Ġbeen Ġa Ġflurry Ġof Ġwork Ġin Ġconstructing Ġdatasets Ġwith Ġan Ġadvers arial Ġcomponent , Ġsuch Ġas ĠSw ag ĠB IB REF 48 Ġand ĠHell a Sw ag ĠB IB REF 49 , ĠC OD AH ĠB IB REF 50 , ĠAd vers arial ĠS Qu AD ĠB IB REF 51 , ĠLamb ada ĠB IB REF 52 Ġand Ġothers . ĠOur Ġdataset Ġis Ġnot Ġto Ġbe Ġconfused Ġwith Ġabduct ive ĠNL I ĠB IB REF 53 , Ġwhich Ġcalls Ġitself Ġ$\ alpha Ġ$ N LI , Ġor ĠART . </s> Discussion Ġ& ĠConclusion </s> In Ġthis Ġwork , Ġwe Ġused Ġa Ġhuman - and - model - in - the - loop Ġentail ment Ġtraining Ġmethod Ġto Ġcollect Ġa Ġnew Ġbenchmark Ġfor Ġnatural Ġlanguage Ġunderstanding . ĠThe Ġbenchmark Ġis Ġdesigned Ġto Ġbe Ġchallenging Ġto Ġcurrent Ġstate Ġof Ġthe Ġart Ġmodels . ĠAn not ators Ġwere Ġemployed Ġto Ġact Ġas Ġadversaries , Ġand Ġencouraged Ġto Ġfind Ġvulnerabilities Ġthat Ġfool Ġthe Ġmodel Ġinto Ġpredicting Ġthe Ġwrong Ġlabel , Ġbut Ġthat Ġanother Ġperson Ġwould Ġcorrectly Ġclassify . ĠWe Ġfound Ġthat Ġnon - ex pert Ġannot ators , Ġin Ġthis Ġgam ified Ġsetting Ġand Ġwith Ġappropriate Ġincentives Ġto Ġfool Ġthe Ġmodel , Ġare Ġremarkably Ġcreative Ġat Ġfinding Ġand Ġexploiting Ġweaknesses Ġin Ġmodels . ĠWe Ġcollected Ġthree Ġrounds , Ġand Ġas Ġthe Ġrounds Ġprogressed , Ġthe Ġmodels Ġbecame Ġmore Ġrobust Ġand Ġthe Ġtest Ġsets Ġfor Ġeach Ġround Ġbecame Ġmore Ġdifficult . ĠTraining Ġon Ġthis Ġnew Ġdata Ġyielded Ġthe Ġstate Ġof Ġthe Ġart Ġon Ġexisting ĠNL I Ġbenchmarks . </s> The ĠAN LI Ġbenchmark Ġpresents Ġa Ġnew Ġchallenge Ġto Ġthe Ġcommunity . ĠIt Ġwas Ġcarefully Ġconstructed Ġto Ġmitigate Ġissues Ġwith Ġprevious Ġdatasets , Ġand Ġwas Ġdesigned Ġfrom Ġfirst Ġprinciples Ġto Ġlast Ġlonger âĢĶ if Ġthe Ġtest Ġset Ġsatur ates , Ġthe Ġfield Ġcan Ġsimply Ġtrain Ġup Ġa Ġnew Ġmodel , Ġcollect Ġmore Ġdata Ġand Ġfind Ġitself Ġconfronted Ġyet Ġagain Ġwith Ġa Ġdifficult Ġchallenge . </s> The Ġdataset Ġalso Ġpresents Ġmany Ġopportunities Ġfor Ġfurther Ġstudy . ĠFor Ġinstance , Ġwe Ġcollected Ġannot ator - provided Ġexplanations Ġfor Ġeach Ġexample Ġthat Ġthe Ġmodel Ġgot Ġwrong . ĠWe Ġprovided Ġinference Ġlabels Ġfor Ġthe Ġdevelopment Ġset , Ġopening Ġup Ġpossibilities Ġfor Ġinteresting Ġmore Ġfine - gr ained Ġstudies Ġof ĠNL I Ġmodel Ġperformance . ĠWhile Ġwe Ġverified Ġthe Ġdevelopment Ġand Ġtest Ġexamples , Ġwe Ġdid Ġnot Ġverify Ġthe Ġcorrectness Ġof Ġeach Ġtraining Ġexample , Ġwhich Ġmeans Ġthere Ġis Ġprobably Ġsome Ġroom Ġfor Ġimprovement Ġthere . </s> The Ġbenchmark Ġis Ġmeant Ġto Ġbe Ġa Ġchallenge Ġfor Ġmeasuring ĠNL U Ġprogress , Ġeven Ġfor Ġas Ġyet Ġund iscovered Ġmodels Ġand Ġarchitectures . ĠWe Ġplan Ġfor Ġthe Ġbenchmark Ġitself Ġto Ġadapt Ġto Ġthese Ġnew Ġmodels Ġby Ġcontinuing Ġto Ġbuild Ġnew Ġchallenge Ġrounds . ĠAs Ġa Ġfirst Ġnext Ġstep , Ġit Ġwould Ġbe Ġinteresting Ġto Ġexamine Ġresults Ġwhen Ġannot ators Ġare Ġconfronted Ġwith Ġa Ġwide Ġvariety Ġof Ġmodel Ġarchitectures . ĠWe Ġhope Ġthat Ġthe Ġdataset Ġwill Ġprove Ġto Ġbe Ġan Ġinteresting Ġnew Ġchallenge Ġfor Ġthe Ġcommunity . ĠLuckily , Ġif Ġit Ġdoes Ġturn Ġout Ġto Ġsatur ate Ġquickly , Ġwe Ġwill Ġalways Ġbe Ġable Ġto Ġcollect Ġa Ġnew Ġround . </s> Acknowled gments </s> YN Ġand ĠMB Ġwere Ġsponsored Ġby ĠDAR PA ĠM CS ĠGrant Ġ# N 66 001 - 19 - 2 - 40 31 , ĠON R ĠGrant Ġ# N 0001 4 - 18 - 1 - 28 71 , Ġand ĠDAR PA ĠY FA 17 - D 17 AP 000 22 .
Introduction Progress in AI has been driven by, among other things, the development of challenging large-scale benchmarks like ImageNet BIBREF0 in computer vision, and SNLI BIBREF1, SQuAD BIBREF2, and others in natural language processing (NLP). Recently, for natural language understanding (NLU) in particular, the focus has shifted to combined benchmarks like SentEval BIBREF3 and GLUE BIBREF4, which track model performance on multiple tasks and provide a unified platform for analysis. With the rapid pace of advancement in AI, however, NLU benchmarks struggle to keep up with model improvement. Whereas it took around 15 years to achieve “near-human performance” on MNIST BIBREF5, BIBREF6, BIBREF7 and approximately 7 years to surpass humans on ImageNet BIBREF8, BIBREF0, BIBREF9, the GLUE benchmark did not last as long as we would have hoped after the advent of BERT BIBREF10, and rapidly had to be extended into SuperGLUE BIBREF11. This raises an important question: Can we collect a large benchmark dataset that can last longer? The speed with which benchmarks become obsolete raises another important question: are current NLU models genuinely as good as their high performance on benchmarks suggests? A growing body of evidence shows that state-of-the-art models learn to exploit spurious statistical patterns in datasets BIBREF12, BIBREF13, BIBREF14, BIBREF15, BIBREF16, BIBREF17, instead of learning meaning in the flexible and generalizable way that humans do. Given this, human annotators—be they seasoned NLP researchers or non-experts—might easily be able to construct examples that expose model brittleness. We propose an iterative, adversarial human-and-model-in-the-loop solution for NLU dataset collection that addresses both benchmark longevity and robustness issues. In the first stage, human annotators devise examples that our current best models cannot determine the correct label for. These resulting hard examples—which should expose additional model weaknesses—can be added to the training set and used to train a stronger model. We then subject the strengthened model to human interference and collect more weaknesses over several rounds. After each round, we both train a new model, and set aside a new test set. The process can be iteratively repeated in a never-ending learning BIBREF18 setting, with the model getting stronger and the test set getting harder in each new round.This process yields a “moving post” dynamic target for NLU systems, rather than a static benchmark that will eventually saturate. Our approach draws inspiration from recent efforts that gamify collaborative training of machine learning agents over multiple rounds BIBREF19 and pit “builders” against “breakers” to learn better models BIBREF20. Recently, Dinan2019build showed that a similar approach can be used to make dialogue safety classifiers more robust. Here, we focus on natural language inference (NLI), arguably the most canonical task in NLU. We collected three rounds of data, and call our new dataset Adversarial NLI (ANLI). Our contributions are as follows: 1) We introduce a novel human-and-model-in-the-loop dataset, currently consisting of three rounds that progressively increase in difficulty and complexity, that includes annotator-provided explanations. 2) We show that training models on this new dataset leads to state-of-the-art performance on a variety of popular NLI benchmarks. 3) We provide a detailed analysis of the collected data that sheds light on the shortcomings of current models, categorizes the data by inference type to examine weaknesses, and demonstrates good performance on NLI stress tests. The ANLI dataset is available at github.com/facebookresearch/anli/. A demo of the annotation procedure can be viewed at adversarialnli.com. Dataset collection The primary aim of this work is to create a new large-scale NLI benchmark on which current state-of-the-art models fail. This constitutes a new target for the field to work towards, and can elucidate model capabilities and limitations. As noted, however, static benchmarks do not last very long these days. If continuously deployed, the data collection procedure we introduce here can pose a dynamic challenge that allows for never-ending learning. Dataset collection ::: HAMLET To paraphrase the great bard BIBREF21, there is something rotten in the state of the art. We propose Human-And-Model-in-the-Loop Entailment Training (HAMLET), a training procedure to automatically mitigate problems with current dataset collection procedures (see Figure FIGREF1). In our setup, our starting point is a base model, trained on NLI data. Rather than employing automated adversarial methods, here the model's “adversary” is a human annotator. Given a context (also often called a “premise” in NLI), and a desired target label, we ask the human writer to provide a hypothesis that fools the model into misclassifying the label. One can think of the writer as a “white hat” hacker, trying to identify vulnerabilities in the system. For each human-generated example that is misclassified, we also ask the writer to provide a reason why they believe it was misclassified. For examples that the model misclassified, it is necessary to verify that they are actually correct —i.e., that the given context-hypothesis pairs genuinely have their specified target label. The best way to do this is to have them checked by another human. Hence, we provide the example to human verifiers. If two human verifiers agree with the writer, the example is considered a good example. If they disagree, we ask a third human verifier to break the tie. If there is still disagreement between the writer and the verifiers, the example is discarded. Occasionally, verifiers will overrule the original label of the writer. Once data collection for the current round is finished, we construct a new training set from the collected data, with accompanying development and test sets. While the training set includes correctly classified examples, the development and tests sets are built solely from them. The test set was further restricted so as to: 1) include pairs from “exclusive” annotators that are never included in the training data; and 2) be balanced by label classes (and genres, where applicable). We subsequently train a new model on this and other existing data, and repeat the procedure three times. Dataset collection ::: Annotation details We employed crowdsourced workers from Mechanical Turk with qualifications. We collected hypotheses via the ParlAI framework. Annotators are presented with a context and a target label—either `entailment', `contradiction', or `neutral'—and asked to write a hypothesis that corresponds to the label. We phrase the label classes as “definitely correct”, “definitely incorrect”, or “neither definitely correct nor definitely incorrect” given the context, to make the task easier to grasp. Submitted hypotheses are given to the model to make a prediction for the context-hypothesis pair. The probability of each label is returned to the worker as feedback. If the model predicts the label incorrectly, the job is complete. If not, the worker continues to write hypotheses for the given (context, target-label) pair until the model predicts the label incorrectly or the number of tries exceeds a threshold (5 tries in the first round, 10 tries thereafter). To encourage workers, payments increased as rounds became harder. For hypotheses that the model predicted the incorrect label for, but were verified by other humans, we paid an additional bonus on top of the standard rate. Dataset collection ::: Round 1 For the first round, we used a BERT-Large model BIBREF10 trained on a concatenation of SNLI BIBREF1 and MNLI BIBREF22, and selected the best-performing model we could train as the starting point for our dataset collection procedure. For Round 1 contexts, we randomly sampled short multi-sentence passages from Wikipedia (of 250-600 characters) from the manually curated HotpotQA training set BIBREF23. Contexts are either ground-truth contexts from that dataset, or they are Wikipedia passages retrieved using TF-IDF BIBREF24 based on a HotpotQA question. Dataset collection ::: Round 2 For the second round, we used a more powerful RoBERTa model BIBREF25 trained on SNLI, MNLI, an NLI-version of FEVER BIBREF26, and the training data from the previous round (A1). After a hyperparameter search, we selected the model with the best performance on the A1 development set. Then, using the hyperparameters selected from this search, we created a final set of models by training several models with different random seeds. During annotation, we constructed an ensemble by randomly picking a model from the model set as the adversary each turn. This helps us avoid annotators exploiting vulnerabilities in one single model. A new non-overlapping set of contexts was again constructed from Wikipedia via HotpotQA using the same method as Round 1. Dataset collection ::: Round 3 For the third round, we selected a more diverse set of contexts, in order to explore robustness under domain transfer. In addition to contexts from Wikipedia for Round 3, we also included contexts from the following domains: News (extracted from Common Crawl), fiction (extracted from BIBREF27, and BIBREF28), formal spoken text (excerpted from court and presidential debate transcripts in the Manually Annotated Sub-Corpus (MASC) of the Open American National Corpus), and causal or procedural text, which describes sequences of events or actions, extracted from WikiHow. Finally, we also collected annotations using the longer contexts present in the GLUE RTE training data, which came from the RTE5 dataset BIBREF29. We trained an even stronger RoBERTa model by adding the training set from the second round (A2) to the training data. Dataset collection ::: Comparing with other datasets The ANLI dataset improves upon previous work in several ways. First, and most obviously, the dataset is collected to be more difficult than previous datasets, by design. Second, it remedies a problem with SNLI, namely that its contexts (or premises) are very short, because they were selected from the image captioning domain. We believe longer contexts should naturally lead to harder examples, and so we constructed ANLI contexts from longer, multi-sentence source material. Following previous observations that models might exploit spurious biases in NLI hypotheses, BIBREF12, BIBREF13, we conduct a study of the performance of hypothesis-only models on our dataset. We show that such models perform poorly on our test sets. With respect to data generation with naïve annotators, Geva2019taskorannotator noted that models might pick up on annotator bias, modelling the annotators themselves rather than capturing the intended reasoning phenomenon. To counter this, we selected a subset of annotators (i.e., the “exclusive” workers) whose data would only be included in the test set. This enables us to avoid overfitting to the writing style biases of particular annotators, and also to determine how much individual annotator bias is present for the main portion of the data. Examples from each round of dataset collection are provided in Table TABREF2. Furthermore, our dataset poses new challenges to the community that were less relevant for previous work, such as: can we improve performance online without having to train a new model from scratch every round, how can we overcome catastrophic forgetting, how do we deal with mixed model biases, etc. Because the training set includes examples that the model got right but were not verified, it might be noisy, posing filtering as an additional interesting problem. Dataset statistics The dataset statistics can be found in Table TABREF7. The number of examples we collected increases per round, starting with approximately 19k examples for Round 1, to around 47k examples for Round 2, to over 103k examples for Round 3. We collected more data for later rounds not only because that data is likely to be more interesting, but also simply because the base model is better and so annotation took longer to collect good, verified correct examples of model vulnerabilities. For each round, we report the model error rate, both on verified and unverified examples. The unverified model error rate captures the percentage of examples where the model disagreed with the writer's target label, but where we are not (yet) sure if the example is correct. The verified model error rate is the percentage of model errors from example pairs that other annotators were able to confirm the correct label for. Note that this error rate represents a straightforward way to evaluate model quality: the lower the model error rate—assuming constant annotator quality and context-difficulty—the better the model. We observe that model error rates decrease as we progress through rounds. In Round 3, where we included a more diverse range of contexts from various domains, the overall error rate went slightly up compared to the preceding round, but for Wikipedia contexts the error rate decreased substantially. While for the first round roughly 1 in every 5 examples were verified model errors, this quickly dropped over consecutive rounds, and the overall model error rate is less than 1 in 10. On the one hand, this is impressive, and shows how far we have come with just three rounds. On the other hand, it shows that we still have a long way to go if even untrained annotators can fool ensembles of state-of-the-art models with relative ease. Table TABREF7 also reports the average number of “tries”, i.e., attempts made for each context until a model error was found (or the number of possible tries is exceeded), and the average time this took (in seconds). Again, these metrics represent a useful way to evaluate model quality. We observe that the average tries and average time per verified error both go up as we progress through the rounds. The numbers clearly demonstrate that the rounds are getting increasingly more difficult. Results Table TABREF13 reports the main results. In addition to BERT BIBREF10 and RoBERTa BIBREF25, we also include XLNet BIBREF30 as an example of a strong, but different, model architecture. We show test set performance on the ANLI test sets per round, the total ANLI test set, and the exclusive test subset (examples from test-set-exclusive workers). We also show accuracy on the SNLI test set and the MNLI development (for the purpose of comparing between different model configurations across table rows) set. In what follows, we briefly discuss our observations. Results ::: Base model performance is low. Notice that the base model for each round performs very poorly on that round's test set. This is the expected outcome: For round 1, the base model gets the entire test set wrong, by design. For rounds 2 and 3, we used an ensemble, so performance is not necessarily zero. However, as it turns out, performance still falls well below chance, indicating that workers did not find vulnerabilities specific to a single model, but generally applicable ones for that model class. Results ::: Rounds become increasingly more difficult. As already foreshadowed by the dataset statistics, round 3 is more difficult (yields lower performance) than round 2, and round 2 is more difficult than round 1. This is true for all model architectures. Results ::: Training on more rounds improves robustness. Generally, our results indicate that training on more rounds improves model performance. This is true for all model architectures. Simply training on more “normal NLI” data would not help a model be robust to adversarial attacks, but our data actively helps mitigate these. Results ::: RoBERTa achieves state-of-the-art performance... We obtain state of the art performance on both SNLI and MNLI with the RoBERTa model finetuned on our new data The RoBERTa paper BIBREF25 reports a score of $90.2$ for both MNLI-matched and -mismatched dev, while we obtain $91.0$ and $90.7$. The state of the art on SNLI is currently held by MT-DNN BIBREF31, which reports $91.6$ compared to our $92.9$. Results ::: ...but is outperformed when it is base model. However, the base (RoBERTa) models for rounds 2 and 3 are outperformed by both BERT and XLNet. This shows that annotators have managed to write examples that RoBERTa generally struggles with, and more training data alone cannot easily mitigated these shortcomings. It also implies that BERT, XLNet, and RoBERTa all have different weaknesses, possibly as a function of their training data (BERT, XLNet and RoBERTa were trained on very different data sets, which might or might not have contained information relevant to the weaknesses)—an additional round with a wider model variety would thus be interesting to investigate as a next step. Results ::: Continuously augmenting training data does not downgrade performance. Even though ANLI training data is different from SNLI and MNLI, adding this data to the training set does not harm performance on those tasks. Furthermore, as Table TABREF21 shows, training only on ANLI is transferable to SNLI and MNLI, but not vice versa. This suggests that methods could successfully be applied for many more consecutive rounds. Results ::: Exclusive test subset difference is small. In order to avoid the possibility that models might pick up on annotator-specific artifacts, a concern raised by Geva2019taskorannotator, we included an exclusive test subset with examples from annotators never seen in the training data. We find that the differences between this exclusive subset and the test set are small, indicating that our models do not over-rely on individual annotator's writing styles. Results ::: Hypothesis-only results For SNLI and MNLI, concerns have been raised about the propensity of models to pick up on spurious artifacts that are present just in the hypotheses BIBREF12, BIBREF13. To study this in the context of our results and task difficulty, we compare models trained on (context, hypothesis) pairs to models trained only on the hypothesis (marked $H$). Table TABREF21 reports results on the three rounds of ANLI, as well as SNLI and MNLI. The table shows some interesting take-aways: Results ::: Hypothesis-only results ::: Hypothesis-only models perform poorly on ANLI. We corroborate that hypothesis-only models obtain good performance on SNLI and MNLI. Performance of such models on ANLI is substantially lower, and decreases with more rounds. Results ::: Hypothesis-only results ::: RoBERTa does not outperform hypothesis-only on rounds 2 and 3. On the two rounds where RoBERTa was used as the base model, its performance is not much better than the hypothesis-only model. This could mean two things: either the test data is very difficult, or the training data is not good. To rule out the latter, we trained only on ANLI ($\sim $163k training examples): doing so with RoBERTa matches the performance of BERT on MNLI when it is trained on the much larger, fully in-domain SNLI+MNLI combined dataset (943k training examples), with both getting $\sim $86, which is impressive. Hence, this shows that our new challenge test sets are so difficult that the current state-of-the-art model cannot do better than a hypothesis-only prior. Analysis We perform two types of model error analysis. First we evaluate two popular existing test sets that were created to expose model weaknesses, and show that our dataset discourages models from learning spurious statistical facts, relative to other large popular datasets (e.g., SNLI and MNLI). Secondly, we explore, by round, the types of inferences our writers successfully employed to stump models, by performing hand-annotation on 500 examples from each round's development set. Analysis ::: Performance on challenge datasets Recently, several hard test sets have been made available for revealing the biases NLI models learn from their training datasets BIBREF32, BIBREF17, BIBREF12, BIBREF33. We examined model performance on two of these: the SNLI-Hard BIBREF12 test set, which consists of examples that hypothesis-only models label incorrectly, and the NLI stress tests BIBREF33, in which sentences containing antonyms pairs, negations, high word overlap, i.a., are heuristically constructed. We test our models on these stress tests, after tuning on each test's respective development set to account for potential domain mismatches. For comparison, we also report accuracies from the original papers: for SNLI-Hard we present the results from BIBREF12's implementation of the hierarchical tensor-based Densely Interactive Inference Network BIBREF34 on MNLI, and for the NLI stress tests, we present the performance of BIBREF33's implementation of InferSent BIBREF3 trained on SNLI. Our results are in Table TABREF23. We observe that all of our models far outperform the models presented in original papers for these common stress tests, with our two RoBERTa models performing best. Both perform well on SNLI-Hard and achieve accuracy levels in the high 80s on the `antonym' (AT), `numerical reasoning' (NR), `length' (LN), `spelling error'(SE) sub-datasets, and show marked improvement on both `negation' (NG), and `word overlap' (WO). Training a RoBERTa model also on ANLI appears to be particularly useful for the NR, WO, NG and AT NLI stress tests. Analysis ::: Reasoning types A dynamically evolving dataset offers the unique opportunity to track how model error rates change over time. Since each round's development set contains only verified examples, we can investigate two interesting questions: which types of inference do writers employ to fool the models, and are base models differentially sensitive to different types of reasoning? The results are summarized in Table TABREF27. We employed an expert linguist annotator to devise an ontology of inference types that would be specific to NLI. While designing an appropriate ontology of types of inference is far from straightforward, we found that a unified ontology could be utilized to characterize examples from all three rounds, which suggests that it has at least some generalizeable applicability. The ontology was used to label 500 examples from each ANLI development set. The inference ontology contains six types of inference: Numerical & Quantitative (i.e., reasoning about cardinal and ordinal numbers, inferring dates and ages from numbers, etc.), Reference & Names (coreferences between pronouns and forms of proper names, knowing facts about name gender, etc.), Basic Inferences (conjunctions, negations, cause-and-effect, comparatives and superlatives etc.), Lexical Inference (inferences made possible by lexical information about synonyms, antonyms, etc.), Tricky Inferences (wordplay, linguistic strategies such as syntactic transformations/reorderings, or inferring writer intentions from contexts), and reasoning from outside knowledge or additional facts (e.g., “You can't reach the sea directly from Djibouti”). The quality of annotations was also tracked; if a pair was ambiguous or had a label that seemed incorrect (from the expert annotator's perspective), it was flagged. Round 1–3 development sets contained few `Quality' tags; the incidence of quality issues was stable at between 3% and 4% per round. Any one example can have multiple types, and every example contained at least one tag. As rounds 1 and 2 were both built with contexts from the same genre (Wikipedia), we might expect writers to arrive at similar strategies. However, since the model architectures used in the first two rounds differ, writers might be sufficiently creative in finding different exploits in each. For round 3, we expect some difference in reasoning types to be present, because we used source material from several domains as our contexts. In sum, any change between rounds could be due to any of the following factors: inherent differences between data collection, model architectures and model training data, random selection of contexts, or slight differences in writer pool or writer preferences. We observe that both round 1 and 2 writers rely heavily on numerical and quantitative reasoning in over 30% of the development set—the percentage in A2 (32%) dropped roughly 6% from A1 (38%)—while round 3 writers use numerical or quantitative reasoning for only 17%. The majority of numerical reasoning types were references to cardinal numbers that referred to dates and ages. Inferences predicated on references and names were present in about 10% of rounds 1 & 3 development sets, and reached a high of 20% in round 2, with coreference featuring prominently. Basic inference types increased in prevalence as the rounds increased, ranging from 18%–30%, as did Lexical inferences (increasing from 13%–33%). The percentage of sentences relying on reasoning and outside facts remains roughly the same, in the mid-50s, perhaps slightly increasing after round 1. For round 3, we observe that the model used to collect it appears to be more susceptible to Basic, Lexical, and Tricky inference types. This finding is compatible with the idea that the models trained on adversarial data are more impressive and perform better, encouraging writers to devise more creative examples containing harder types of inference in order to stump them. Related work ::: Bias in datasets Machine learning methods are well-known to pick up on spurious statistical patterns. For instance, in image captioning, a simple baseline of utilizing the captions of nearest neighbors in the training set was shown to yield impressive BLEU scores BIBREF35. In the first visual question answering dataset BIBREF36, biases like “2” being the correct answer to 39% of the questions starting with “how many” allowed learning algorithms to perform well while ignoring the visual modality altogether BIBREF37, BIBREF38. The field has a tendency to overfit on static targets, even if that does not happen deliberately BIBREF39. In NLI, Gururangan2018annotation, Poliak2018hypothesis and Tsuchiya2018performance showed that hypothesis-only baselines often perform far better than chance. It has been shown that NLI systems can often be broken merely by performing simple lexical substitutions BIBREF15, and that they struggle with quantifiers BIBREF40 and certain superficial syntactic properties BIBREF17. In reading comprehension and question answering, Kaushik2018howmuch showed that question- and passage-only models can perform surprisingly well, while Jia2017adversarial added adversarially constructed sentences to passages, leading to a drastic drop in performance. Many text classification datasets do not require sophisticated linguistic reasoning, as shown by the surprisingly good performance of random encoders BIBREF41. Similar observations were made in machine translation BIBREF42 and dialogue BIBREF43. In short, the field is rife with dataset bias and papers trying address this important problem. This work can be viewed as a natural extension: if such biases exist, they will allow humans to fool the models, adding useful examples to the training data until the bias is dynamically mitigated. Related work ::: Dynamic datasets. Concurrently with this work, Anonymous2020adversarialfilters proposed AFLite, an iterative approach for filtering adversarial data points to avoid spurious biases. Kaushik2019learningdifference offer a causal account of spurious patterns, and counterfactually augment NLI datasets by editing examples to break the model. The former is an example of a model-in-the-loop setting, where the model is iteratively probed and improved. The latter is human-in-the-loop training, where humans are used to find problems with one single model. In this work, we employ both strategies iteratively, in a form of human-and-model-in-the-loop training, to collect completely new examples, in a potentially never-ending loop BIBREF18. Relatedly, lan2017ppdb propose a method for continuously growing a dataset of paraphrases. Human-and-model-in-the-loop training is not a new idea. Mechanical Turker Descent proposes a gamified environment for the collaborative training of grounded language learning agents over multiple rounds BIBREF19. The “Build it Break it Fix it” strategy in the security domain BIBREF44 has been adapted to NLP BIBREF20 as well as dialogue BIBREF45. The QApedia framework BIBREF46 continuously refines and updates its content repository using humans in the loop, while human feedback loops have been used to improve image captioning systems BIBREF47. wallace2018trick leverage trivia experts to create a model-driven adversarial question writing procedure and generate a small set of challenge questions that QA-models fail on. There has been a flurry of work in constructing datasets with an adversarial component, such as Swag BIBREF48 and HellaSwag BIBREF49, CODAH BIBREF50, Adversarial SQuAD BIBREF51, Lambada BIBREF52 and others. Our dataset is not to be confused with abductive NLI BIBREF53, which calls itself $\alpha $NLI, or ART. Discussion & Conclusion In this work, we used a human-and-model-in-the-loop entailment training method to collect a new benchmark for natural language understanding. The benchmark is designed to be challenging to current state of the art models. Annotators were employed to act as adversaries, and encouraged to find vulnerabilities that fool the model into predicting the wrong label, but that another person would correctly classify. We found that non-expert annotators, in this gamified setting and with appropriate incentives to fool the model, are remarkably creative at finding and exploiting weaknesses in models. We collected three rounds, and as the rounds progressed, the models became more robust and the test sets for each round became more difficult. Training on this new data yielded the state of the art on existing NLI benchmarks. The ANLI benchmark presents a new challenge to the community. It was carefully constructed to mitigate issues with previous datasets, and was designed from first principles to last longer—if the test set saturates, the field can simply train up a new model, collect more data and find itself confronted yet again with a difficult challenge. The dataset also presents many opportunities for further study. For instance, we collected annotator-provided explanations for each example that the model got wrong. We provided inference labels for the development set, opening up possibilities for interesting more fine-grained studies of NLI model performance. While we verified the development and test examples, we did not verify the correctness of each training example, which means there is probably some room for improvement there. The benchmark is meant to be a challenge for measuring NLU progress, even for as yet undiscovered models and architectures. We plan for the benchmark itself to adapt to these new models by continuing to build new challenge rounds. As a first next step, it would be interesting to examine results when annotators are confronted with a wide variety of model architectures. We hope that the dataset will prove to be an interesting new challenge for the community. Luckily, if it does turn out to saturate quickly, we will always be able to collect a new round. Acknowledgments YN and MB were sponsored by DARPA MCS Grant #N66001-19-2-4031, ONR Grant #N00014-18-1-2871, and DARPA YFA17-D17AP00022.
Do they use active learning to create their dataset?
ee2c9bc24d70daa0c87e38e0558e09ab97feb4f2
Do Ġthey Ġuse Ġactive Ġlearning Ġto Ġcreate Ġtheir Ġdataset ?
Yes
<s> Do Ġthey Ġuse Ġactive Ġlearning Ġto Ġcreate Ġtheir Ġdataset ? </s> Introduction </s> Progress Ġin ĠAI Ġhas Ġbeen Ġdriven Ġby , Ġamong Ġother Ġthings , Ġthe Ġdevelopment Ġof Ġchallenging Ġlarge - scale Ġbenchmarks Ġlike ĠImage Net ĠB IB REF 0 Ġin Ġcomputer Ġvision , Ġand ĠSN LI ĠB IB REF 1 , ĠS Qu AD ĠB IB REF 2 , Ġand Ġothers Ġin Ġnatural Ġlanguage Ġprocessing Ġ( N LP ). ĠRecently , Ġfor Ġnatural Ġlanguage Ġunderstanding Ġ( NL U ) Ġin Ġparticular , Ġthe Ġfocus Ġhas Ġshifted Ġto Ġcombined Ġbenchmarks Ġlike ĠSent E val ĠB IB REF 3 Ġand ĠGL UE ĠB IB REF 4 , Ġwhich Ġtrack Ġmodel Ġperformance Ġon Ġmultiple Ġtasks Ġand Ġprovide Ġa Ġunified Ġplatform Ġfor Ġanalysis . </s> With Ġthe Ġrapid Ġpace Ġof Ġadvancement Ġin ĠAI , Ġhowever , ĠNL U Ġbenchmarks Ġstruggle Ġto Ġkeep Ġup Ġwith Ġmodel Ġimprovement . ĠWhereas Ġit Ġtook Ġaround Ġ15 Ġyears Ġto Ġachieve ĠâĢ ľ near - human Ġperformance âĢ Ŀ Ġon ĠMN IST ĠB IB REF 5 , ĠB IB REF 6 , ĠB IB REF 7 Ġand Ġapproximately Ġ7 Ġyears Ġto Ġsurpass Ġhumans Ġon ĠImage Net ĠB IB REF 8 , ĠB IB REF 0 , ĠB IB REF 9 , Ġthe ĠGL UE Ġbenchmark Ġdid Ġnot Ġlast Ġas Ġlong Ġas Ġwe Ġwould Ġhave Ġhoped Ġafter Ġthe Ġadvent Ġof ĠB ERT ĠB IB REF 10 , Ġand Ġrapidly Ġhad Ġto Ġbe Ġextended Ġinto ĠSuper GL UE ĠB IB REF 11 . ĠThis Ġraises Ġan Ġimportant Ġquestion : ĠCan Ġwe Ġcollect Ġa Ġlarge Ġbenchmark Ġdataset Ġthat Ġcan Ġlast Ġlonger ? </s> The Ġspeed Ġwith Ġwhich Ġbenchmarks Ġbecome Ġobsolete Ġraises Ġanother Ġimportant Ġquestion : Ġare Ġcurrent ĠNL U Ġmodels Ġgenuinely Ġas Ġgood Ġas Ġtheir Ġhigh Ġperformance Ġon Ġbenchmarks Ġsuggests ? ĠA Ġgrowing Ġbody Ġof Ġevidence Ġshows Ġthat Ġstate - of - the - art Ġmodels Ġlearn Ġto Ġexploit Ġspurious Ġstatistical Ġpatterns Ġin Ġdatasets ĠB IB REF 12 , ĠB IB REF 13 , ĠB IB REF 14 , ĠB IB REF 15 , ĠB IB REF 16 , ĠB IB REF 17 , Ġinstead Ġof Ġlearning Ġmeaning Ġin Ġthe Ġflexible Ġand Ġgeneral izable Ġway Ġthat Ġhumans Ġdo . ĠGiven Ġthis , Ġhuman Ġannot ators âĢĶ be Ġthey Ġseasoned ĠN LP Ġresearchers Ġor Ġnon - exper ts âĢĶ might Ġeasily Ġbe Ġable Ġto Ġconstruct Ġexamples Ġthat Ġexpose Ġmodel Ġbr itt leness . </s> We Ġpropose Ġan Ġiter ative , Ġadvers arial Ġhuman - and - model - in - the - loop Ġsolution Ġfor ĠNL U Ġdataset Ġcollection Ġthat Ġaddresses Ġboth Ġbenchmark Ġlongevity Ġand Ġrobust ness Ġissues . ĠIn Ġthe Ġfirst Ġstage , Ġhuman Ġannot ators Ġdevise Ġexamples Ġthat Ġour Ġcurrent Ġbest Ġmodels Ġcannot Ġdetermine Ġthe Ġcorrect Ġlabel Ġfor . ĠThese Ġresulting Ġhard Ġexamples âĢĶ which Ġshould Ġexpose Ġadditional Ġmodel Ġweaknesses âĢĶ can Ġbe Ġadded Ġto Ġthe Ġtraining Ġset Ġand Ġused Ġto Ġtrain Ġa Ġstronger Ġmodel . ĠWe Ġthen Ġsubject Ġthe Ġstrengthened Ġmodel Ġto Ġhuman Ġinterference Ġand Ġcollect Ġmore Ġweaknesses Ġover Ġseveral Ġrounds . ĠAfter Ġeach Ġround , Ġwe Ġboth Ġtrain Ġa Ġnew Ġmodel , Ġand Ġset Ġaside Ġa Ġnew Ġtest Ġset . ĠThe Ġprocess Ġcan Ġbe Ġiter atively Ġrepeated Ġin Ġa Ġnever - ending Ġlearning ĠB IB REF 18 Ġsetting , Ġwith Ġthe Ġmodel Ġgetting Ġstronger Ġand Ġthe Ġtest Ġset Ġgetting Ġharder Ġin Ġeach Ġnew Ġround . This Ġprocess Ġyields Ġa ĠâĢ ľ moving Ġpost âĢ Ŀ Ġdynamic Ġtarget Ġfor ĠNL U Ġsystems , Ġrather Ġthan Ġa Ġstatic Ġbenchmark Ġthat Ġwill Ġeventually Ġsatur ate . </s> Our Ġapproach Ġdraws Ġinspiration Ġfrom Ġrecent Ġefforts Ġthat Ġgam ify Ġcollaborative Ġtraining Ġof Ġmachine Ġlearning Ġagents Ġover Ġmultiple Ġrounds ĠB IB REF 19 Ġand Ġpit ĠâĢ ľ builders âĢ Ŀ Ġagainst ĠâĢ ľ breakers âĢ Ŀ Ġto Ġlearn Ġbetter Ġmodels ĠB IB REF 20 . ĠRecently , ĠDin an 2019 build Ġshowed Ġthat Ġa Ġsimilar Ġapproach Ġcan Ġbe Ġused Ġto Ġmake Ġdialogue Ġsafety Ġclass ifiers Ġmore Ġrobust . ĠHere , Ġwe Ġfocus Ġon Ġnatural Ġlanguage Ġinference Ġ( N LI ), Ġarguably Ġthe Ġmost Ġcanonical Ġtask Ġin ĠNL U . ĠWe Ġcollected Ġthree Ġrounds Ġof Ġdata , Ġand Ġcall Ġour Ġnew Ġdataset ĠAd vers arial ĠNL I Ġ( AN LI ). </s> Our Ġcontributions Ġare Ġas Ġfollows : Ġ1 ) ĠWe Ġintroduce Ġa Ġnovel Ġhuman - and - model - in - the - loop Ġdataset , Ġcurrently Ġconsisting Ġof Ġthree Ġrounds Ġthat Ġprogressively Ġincrease Ġin Ġdifficulty Ġand Ġcomplexity , Ġthat Ġincludes Ġannot ator - provided Ġexplanations . Ġ2 ) ĠWe Ġshow Ġthat Ġtraining Ġmodels Ġon Ġthis Ġnew Ġdataset Ġleads Ġto Ġstate - of - the - art Ġperformance Ġon Ġa Ġvariety Ġof Ġpopular ĠNL I Ġbenchmarks . Ġ3 ) ĠWe Ġprovide Ġa Ġdetailed Ġanalysis Ġof Ġthe Ġcollected Ġdata Ġthat Ġsheds Ġlight Ġon Ġthe Ġshortcomings Ġof Ġcurrent Ġmodels , Ġcategor izes Ġthe Ġdata Ġby Ġinference Ġtype Ġto Ġexamine Ġweaknesses , Ġand Ġdemonstrates Ġgood Ġperformance Ġon ĠNL I Ġstress Ġtests . ĠThe ĠAN LI Ġdataset Ġis Ġavailable Ġat Ġgithub . com / facebook research / an li /. ĠA Ġdemo Ġof Ġthe Ġannotation Ġprocedure Ġcan Ġbe Ġviewed Ġat Ġadvers arial n li . com . </s> Dat as et Ġcollection </s> The Ġprimary Ġaim Ġof Ġthis Ġwork Ġis Ġto Ġcreate Ġa Ġnew Ġlarge - scale ĠNL I Ġbenchmark Ġon Ġwhich Ġcurrent Ġstate - of - the - art Ġmodels Ġfail . ĠThis Ġconstitutes Ġa Ġnew Ġtarget Ġfor Ġthe Ġfield Ġto Ġwork Ġtowards , Ġand Ġcan Ġeluc idate Ġmodel Ġcapabilities Ġand Ġlimitations . ĠAs Ġnoted , Ġhowever , Ġstatic Ġbenchmarks Ġdo Ġnot Ġlast Ġvery Ġlong Ġthese Ġdays . ĠIf Ġcontinuously Ġdeployed , Ġthe Ġdata Ġcollection Ġprocedure Ġwe Ġintroduce Ġhere Ġcan Ġpose Ġa Ġdynamic Ġchallenge Ġthat Ġallows Ġfor Ġnever - ending Ġlearning . </s> Dat as et Ġcollection Ġ: :: ĠHAM LET </s> To Ġparaph rase Ġthe Ġgreat Ġb ard ĠB IB REF 21 , Ġthere Ġis Ġsomething Ġrotten Ġin Ġthe Ġstate Ġof Ġthe Ġart . ĠWe Ġpropose ĠHuman - And - Model - in - the - Loop ĠEnt ail ment ĠTraining Ġ( HAM LET ), Ġa Ġtraining Ġprocedure Ġto Ġautomatically Ġmitigate Ġproblems Ġwith Ġcurrent Ġdataset Ġcollection Ġprocedures Ġ( see ĠFigure ĠFIG REF 1 ). </s> In Ġour Ġsetup , Ġour Ġstarting Ġpoint Ġis Ġa Ġbase Ġmodel , Ġtrained Ġon ĠNL I Ġdata . ĠRather Ġthan Ġemploying Ġautomated Ġadvers arial Ġmethods , Ġhere Ġthe Ġmodel 's ĠâĢ ľ ad vers ary âĢ Ŀ Ġis Ġa Ġhuman Ġannot ator . ĠGiven Ġa Ġcontext Ġ( also Ġoften Ġcalled Ġa ĠâĢ ľ prem ise âĢ Ŀ Ġin ĠNL I ), Ġand Ġa Ġdesired Ġtarget Ġlabel , Ġwe Ġask Ġthe Ġhuman Ġwriter Ġto Ġprovide Ġa Ġhypothesis Ġthat Ġfools Ġthe Ġmodel Ġinto Ġmis class ifying Ġthe Ġlabel . ĠOne Ġcan Ġthink Ġof Ġthe Ġwriter Ġas Ġa ĠâĢ ľ white Ġhat âĢ Ŀ Ġhacker , Ġtrying Ġto Ġidentify Ġvulnerabilities Ġin Ġthe Ġsystem . ĠFor Ġeach Ġhuman - generated Ġexample Ġthat Ġis Ġmis classified , Ġwe Ġalso Ġask Ġthe Ġwriter Ġto Ġprovide Ġa Ġreason Ġwhy Ġthey Ġbelieve Ġit Ġwas Ġmis classified . </s> For Ġexamples Ġthat Ġthe Ġmodel Ġmis classified , Ġit Ġis Ġnecessary Ġto Ġverify Ġthat Ġthey Ġare Ġactually Ġcorrect ĠâĢĶ i . e ., Ġthat Ġthe Ġgiven Ġcontext - hyp ot hesis Ġpairs Ġgenuinely Ġhave Ġtheir Ġspecified Ġtarget Ġlabel . ĠThe Ġbest Ġway Ġto Ġdo Ġthis Ġis Ġto Ġhave Ġthem Ġchecked Ġby Ġanother Ġhuman . ĠHence , Ġwe Ġprovide Ġthe Ġexample Ġto Ġhuman Ġver ifiers . ĠIf Ġtwo Ġhuman Ġver ifiers Ġagree Ġwith Ġthe Ġwriter , Ġthe Ġexample Ġis Ġconsidered Ġa Ġgood Ġexample . ĠIf Ġthey Ġdisagree , Ġwe Ġask Ġa Ġthird Ġhuman Ġver ifier Ġto Ġbreak Ġthe Ġtie . ĠIf Ġthere Ġis Ġstill Ġdisagreement Ġbetween Ġthe Ġwriter Ġand Ġthe Ġver ifiers , Ġthe Ġexample Ġis Ġdiscarded . ĠOccasionally , Ġver ifiers Ġwill Ġover rule Ġthe Ġoriginal Ġlabel Ġof Ġthe Ġwriter . </s> Once Ġdata Ġcollection Ġfor Ġthe Ġcurrent Ġround Ġis Ġfinished , Ġwe Ġconstruct Ġa Ġnew Ġtraining Ġset Ġfrom Ġthe Ġcollected Ġdata , Ġwith Ġaccompanying Ġdevelopment Ġand Ġtest Ġsets . ĠWhile Ġthe Ġtraining Ġset Ġincludes Ġcorrectly Ġclassified Ġexamples , Ġthe Ġdevelopment Ġand Ġtests Ġsets Ġare Ġbuilt Ġsolely Ġfrom Ġthem . ĠThe Ġtest Ġset Ġwas Ġfurther Ġrestricted Ġso Ġas Ġto : Ġ1 ) Ġinclude Ġpairs Ġfrom ĠâĢ ľ exclusive âĢ Ŀ Ġannot ators Ġthat Ġare Ġnever Ġincluded Ġin Ġthe Ġtraining Ġdata ; Ġand Ġ2 ) Ġbe Ġbalanced Ġby Ġlabel Ġclasses Ġ( and Ġgenres , Ġwhere Ġapplicable ). ĠWe Ġsubsequently Ġtrain Ġa Ġnew Ġmodel Ġon Ġthis Ġand Ġother Ġexisting Ġdata , Ġand Ġrepeat Ġthe Ġprocedure Ġthree Ġtimes . </s> Dat as et Ġcollection Ġ: :: ĠAn notation Ġdetails </s> We Ġemployed Ġcrowds ourced Ġworkers Ġfrom ĠMechanical ĠTurk Ġwith Ġqualifications . ĠWe Ġcollected Ġhypotheses Ġvia Ġthe ĠPar l AI Ġframework . ĠAn not ators Ġare Ġpresented Ġwith Ġa Ġcontext Ġand Ġa Ġtarget Ġlabel âĢĶ either Ġ` ent ail ment ', Ġ` cont rad iction ', Ġor Ġ` neutral ' âĢĶ and Ġasked Ġto Ġwrite Ġa Ġhypothesis Ġthat Ġcorresponds Ġto Ġthe Ġlabel . ĠWe Ġphrase Ġthe Ġlabel Ġclasses Ġas ĠâĢ ľ def initely Ġcorrect âĢ Ŀ , ĠâĢ ľ def initely Ġincorrect âĢ Ŀ , Ġor ĠâĢ ľ ne ither Ġdefinitely Ġcorrect Ġnor Ġdefinitely Ġincorrect âĢ Ŀ Ġgiven Ġthe Ġcontext , Ġto Ġmake Ġthe Ġtask Ġeasier Ġto Ġgrasp . ĠSub mitted Ġhypotheses Ġare Ġgiven Ġto Ġthe Ġmodel Ġto Ġmake Ġa Ġprediction Ġfor Ġthe Ġcontext - hyp ot hesis Ġpair . ĠThe Ġprobability Ġof Ġeach Ġlabel Ġis Ġreturned Ġto Ġthe Ġworker Ġas Ġfeedback . ĠIf Ġthe Ġmodel Ġpredicts Ġthe Ġlabel Ġincorrectly , Ġthe Ġjob Ġis Ġcomplete . ĠIf Ġnot , Ġthe Ġworker Ġcontinues Ġto Ġwrite Ġhypotheses Ġfor Ġthe Ġgiven Ġ( context , Ġtarget - label ) Ġpair Ġuntil Ġthe Ġmodel Ġpredicts Ġthe Ġlabel Ġincorrectly Ġor Ġthe Ġnumber Ġof Ġtries Ġexceeds Ġa Ġthreshold Ġ( 5 Ġtries Ġin Ġthe Ġfirst Ġround , Ġ10 Ġtries Ġthereafter ). ĠTo Ġencourage Ġworkers , Ġpayments Ġincreased Ġas Ġrounds Ġbecame Ġharder . ĠFor Ġhypotheses Ġthat Ġthe Ġmodel Ġpredicted Ġthe Ġincorrect Ġlabel Ġfor , Ġbut Ġwere Ġverified Ġby Ġother Ġhumans , Ġwe Ġpaid Ġan Ġadditional Ġbonus Ġon Ġtop Ġof Ġthe Ġstandard Ġrate . </s> Dat as et Ġcollection Ġ: :: ĠRound Ġ1 </s> For Ġthe Ġfirst Ġround , Ġwe Ġused Ġa ĠB ERT - Large Ġmodel ĠB IB REF 10 Ġtrained Ġon Ġa Ġconc aten ation Ġof ĠSN LI ĠB IB REF 1 Ġand ĠMN LI ĠB IB REF 22 , Ġand Ġselected Ġthe Ġbest - performing Ġmodel Ġwe Ġcould Ġtrain Ġas Ġthe Ġstarting Ġpoint Ġfor Ġour Ġdataset Ġcollection Ġprocedure . ĠFor ĠRound Ġ1 Ġcontexts , Ġwe Ġrandomly Ġsampled Ġshort Ġmulti - sent ence Ġpassages Ġfrom ĠWikipedia Ġ( of Ġ250 - 600 Ġcharacters ) Ġfrom Ġthe Ġmanually Ġcurated ĠHot pot Q A Ġtraining Ġset ĠB IB REF 23 . ĠContext s Ġare Ġeither Ġground - truth Ġcontexts Ġfrom Ġthat Ġdataset , Ġor Ġthey Ġare ĠWikipedia Ġpassages Ġretrieved Ġusing ĠTF - ID F ĠB IB REF 24 Ġbased Ġon Ġa ĠHot pot Q A Ġquestion . </s> Dat as et Ġcollection Ġ: :: ĠRound Ġ2 </s> For Ġthe Ġsecond Ġround , Ġwe Ġused Ġa Ġmore Ġpowerful ĠRo BER Ta Ġmodel ĠB IB REF 25 Ġtrained Ġon ĠSN LI , ĠMN LI , Ġan ĠNL I - version Ġof ĠFE VER ĠB IB REF 26 , Ġand Ġthe Ġtraining Ġdata Ġfrom Ġthe Ġprevious Ġround Ġ( A 1 ). ĠAfter Ġa Ġhyper param eter Ġsearch , Ġwe Ġselected Ġthe Ġmodel Ġwith Ġthe Ġbest Ġperformance Ġon Ġthe ĠA 1 Ġdevelopment Ġset . ĠThen , Ġusing Ġthe Ġhyper param eters Ġselected Ġfrom Ġthis Ġsearch , Ġwe Ġcreated Ġa Ġfinal Ġset Ġof Ġmodels Ġby Ġtraining Ġseveral Ġmodels Ġwith Ġdifferent Ġrandom Ġseeds . ĠDuring Ġannotation , Ġwe Ġconstructed Ġan Ġensemble Ġby Ġrandomly Ġpicking Ġa Ġmodel Ġfrom Ġthe Ġmodel Ġset Ġas Ġthe Ġadversary Ġeach Ġturn . ĠThis Ġhelps Ġus Ġavoid Ġannot ators Ġexploiting Ġvulnerabilities Ġin Ġone Ġsingle Ġmodel . ĠA Ġnew Ġnon - over l apping Ġset Ġof Ġcontexts Ġwas Ġagain Ġconstructed Ġfrom ĠWikipedia Ġvia ĠHot pot Q A Ġusing Ġthe Ġsame Ġmethod Ġas ĠRound Ġ1 . </s> Dat as et Ġcollection Ġ: :: ĠRound Ġ3 </s> For Ġthe Ġthird Ġround , Ġwe Ġselected Ġa Ġmore Ġdiverse Ġset Ġof Ġcontexts , Ġin Ġorder Ġto Ġexplore Ġrobust ness Ġunder Ġdomain Ġtransfer . ĠIn Ġaddition Ġto Ġcontexts Ġfrom ĠWikipedia Ġfor ĠRound Ġ3 , Ġwe Ġalso Ġincluded Ġcontexts Ġfrom Ġthe Ġfollowing Ġdomains : ĠNews Ġ( ext racted Ġfrom ĠCommon ĠC rawl ), Ġfiction Ġ( ext racted Ġfrom ĠB IB REF 27 , Ġand ĠB IB REF 28 ), Ġformal Ġspoken Ġtext Ġ( ex cerpt ed Ġfrom Ġcourt Ġand Ġpresidential Ġdebate Ġtranscripts Ġin Ġthe ĠMan ually ĠAn not ated ĠSub - Corp us Ġ( MAS C ) Ġof Ġthe ĠOpen ĠAmerican ĠNational ĠCorpus ), Ġand Ġcausal Ġor Ġprocedural Ġtext , Ġwhich Ġdescribes Ġsequences Ġof Ġevents Ġor Ġactions , Ġextracted Ġfrom ĠWiki How . ĠFinally , Ġwe Ġalso Ġcollected Ġannotations Ġusing Ġthe Ġlonger Ġcontexts Ġpresent Ġin Ġthe ĠGL UE ĠR TE Ġtraining Ġdata , Ġwhich Ġcame Ġfrom Ġthe ĠR TE 5 Ġdataset ĠB IB REF 29 . ĠWe Ġtrained Ġan Ġeven Ġstronger ĠRo BER Ta Ġmodel Ġby Ġadding Ġthe Ġtraining Ġset Ġfrom Ġthe Ġsecond Ġround Ġ( A 2 ) Ġto Ġthe Ġtraining Ġdata . </s> Dat as et Ġcollection Ġ: :: ĠComp aring Ġwith Ġother Ġdatasets </s> The ĠAN LI Ġdataset Ġimproves Ġupon Ġprevious Ġwork Ġin Ġseveral Ġways . ĠFirst , Ġand Ġmost Ġobviously , Ġthe Ġdataset Ġis Ġcollected Ġto Ġbe Ġmore Ġdifficult Ġthan Ġprevious Ġdatasets , Ġby Ġdesign . ĠSecond , Ġit Ġremedies Ġa Ġproblem Ġwith ĠSN LI , Ġnamely Ġthat Ġits Ġcontexts Ġ( or Ġpremises ) Ġare Ġvery Ġshort , Ġbecause Ġthey Ġwere Ġselected Ġfrom Ġthe Ġimage Ġcaption ing Ġdomain . ĠWe Ġbelieve Ġlonger Ġcontexts Ġshould Ġnaturally Ġlead Ġto Ġharder Ġexamples , Ġand Ġso Ġwe Ġconstructed ĠAN LI Ġcontexts Ġfrom Ġlonger , Ġmulti - sent ence Ġsource Ġmaterial . </s> Following Ġprevious Ġobservations Ġthat Ġmodels Ġmight Ġexploit Ġspurious Ġbiases Ġin ĠNL I Ġhypotheses , ĠB IB REF 12 , ĠB IB REF 13 , Ġwe Ġconduct Ġa Ġstudy Ġof Ġthe Ġperformance Ġof Ġhypothesis - only Ġmodels Ġon Ġour Ġdataset . ĠWe Ġshow Ġthat Ġsuch Ġmodels Ġperform Ġpoorly Ġon Ġour Ġtest Ġsets . </s> With Ġrespect Ġto Ġdata Ġgeneration Ġwith ĠnaÃ¯ve Ġannot ators , ĠG eva 2019 task or annot ator Ġnoted Ġthat Ġmodels Ġmight Ġpick Ġup Ġon Ġannot ator Ġbias , Ġmodelling Ġthe Ġannot ators Ġthemselves Ġrather Ġthan Ġcapturing Ġthe Ġintended Ġreasoning Ġphenomenon . ĠTo Ġcounter Ġthis , Ġwe Ġselected Ġa Ġsubset Ġof Ġannot ators Ġ( i . e ., Ġthe ĠâĢ ľ exclusive âĢ Ŀ Ġworkers ) Ġwhose Ġdata Ġwould Ġonly Ġbe Ġincluded Ġin Ġthe Ġtest Ġset . ĠThis Ġenables Ġus Ġto Ġavoid Ġover fitting Ġto Ġthe Ġwriting Ġstyle Ġbiases Ġof Ġparticular Ġannot ators , Ġand Ġalso Ġto Ġdetermine Ġhow Ġmuch Ġindividual Ġannot ator Ġbias Ġis Ġpresent Ġfor Ġthe Ġmain Ġportion Ġof Ġthe Ġdata . ĠExamples Ġfrom Ġeach Ġround Ġof Ġdataset Ġcollection Ġare Ġprovided Ġin ĠTable ĠT AB REF 2 . </s> Furthermore , Ġour Ġdataset Ġposes Ġnew Ġchallenges Ġto Ġthe Ġcommunity Ġthat Ġwere Ġless Ġrelevant Ġfor Ġprevious Ġwork , Ġsuch Ġas : Ġcan Ġwe Ġimprove Ġperformance Ġonline Ġwithout Ġhaving Ġto Ġtrain Ġa Ġnew Ġmodel Ġfrom Ġscratch Ġevery Ġround , Ġhow Ġcan Ġwe Ġovercome Ġcatastrophic Ġforgetting , Ġhow Ġdo Ġwe Ġdeal Ġwith Ġmixed Ġmodel Ġbiases , Ġetc . ĠBecause Ġthe Ġtraining Ġset Ġincludes Ġexamples Ġthat Ġthe Ġmodel Ġgot Ġright Ġbut Ġwere Ġnot Ġverified , Ġit Ġmight Ġbe Ġnoisy , Ġposing Ġfiltering Ġas Ġan Ġadditional Ġinteresting Ġproblem . </s> Dat as et Ġstatistics </s> The Ġdataset Ġstatistics Ġcan Ġbe Ġfound Ġin ĠTable ĠT AB REF 7 . ĠThe Ġnumber Ġof Ġexamples Ġwe Ġcollected Ġincreases Ġper Ġround , Ġstarting Ġwith Ġapproximately Ġ19 k Ġexamples Ġfor ĠRound Ġ1 , Ġto Ġaround Ġ47 k Ġexamples Ġfor ĠRound Ġ2 , Ġto Ġover Ġ103 k Ġexamples Ġfor ĠRound Ġ3 . ĠWe Ġcollected Ġmore Ġdata Ġfor Ġlater Ġrounds Ġnot Ġonly Ġbecause Ġthat Ġdata Ġis Ġlikely Ġto Ġbe Ġmore Ġinteresting , Ġbut Ġalso Ġsimply Ġbecause Ġthe Ġbase Ġmodel Ġis Ġbetter Ġand Ġso Ġannotation Ġtook Ġlonger Ġto Ġcollect Ġgood , Ġverified Ġcorrect Ġexamples Ġof Ġmodel Ġvulnerabilities . </s> For Ġeach Ġround , Ġwe Ġreport Ġthe Ġmodel Ġerror Ġrate , Ġboth Ġon Ġverified Ġand Ġun verified Ġexamples . ĠThe Ġun verified Ġmodel Ġerror Ġrate Ġcaptures Ġthe Ġpercentage Ġof Ġexamples Ġwhere Ġthe Ġmodel Ġdisagreed Ġwith Ġthe Ġwriter 's Ġtarget Ġlabel , Ġbut Ġwhere Ġwe Ġare Ġnot Ġ( yet ) Ġsure Ġif Ġthe Ġexample Ġis Ġcorrect . ĠThe Ġverified Ġmodel Ġerror Ġrate Ġis Ġthe Ġpercentage Ġof Ġmodel Ġerrors Ġfrom Ġexample Ġpairs Ġthat Ġother Ġannot ators Ġwere Ġable Ġto Ġconfirm Ġthe Ġcorrect Ġlabel Ġfor . ĠNote Ġthat Ġthis Ġerror Ġrate Ġrepresents Ġa Ġstraightforward Ġway Ġto Ġevaluate Ġmodel Ġquality : Ġthe Ġlower Ġthe Ġmodel Ġerror Ġrate âĢĶ assuming Ġconstant Ġannot ator Ġquality Ġand Ġcontext - diff iculty âĢĶ the Ġbetter Ġthe Ġmodel . </s> We Ġobserve Ġthat Ġmodel Ġerror Ġrates Ġdecrease Ġas Ġwe Ġprogress Ġthrough Ġrounds . ĠIn ĠRound Ġ3 , Ġwhere Ġwe Ġincluded Ġa Ġmore Ġdiverse Ġrange Ġof Ġcontexts Ġfrom Ġvarious Ġdomains , Ġthe Ġoverall Ġerror Ġrate Ġwent Ġslightly Ġup Ġcompared Ġto Ġthe Ġpreceding Ġround , Ġbut Ġfor ĠWikipedia Ġcontexts Ġthe Ġerror Ġrate Ġdecreased Ġsubstantially . ĠWhile Ġfor Ġthe Ġfirst Ġround Ġroughly Ġ1 Ġin Ġevery Ġ5 Ġexamples Ġwere Ġverified Ġmodel Ġerrors , Ġthis Ġquickly Ġdropped Ġover Ġconsecutive Ġrounds , Ġand Ġthe Ġoverall Ġmodel Ġerror Ġrate Ġis Ġless Ġthan Ġ1 Ġin Ġ10 . ĠOn Ġthe Ġone Ġhand , Ġthis Ġis Ġimpressive , Ġand Ġshows Ġhow Ġfar Ġwe Ġhave Ġcome Ġwith Ġjust Ġthree Ġrounds . ĠOn Ġthe Ġother Ġhand , Ġit Ġshows Ġthat Ġwe Ġstill Ġhave Ġa Ġlong Ġway Ġto Ġgo Ġif Ġeven Ġunt rained Ġannot ators Ġcan Ġfool Ġen semb les Ġof Ġstate - of - the - art Ġmodels Ġwith Ġrelative Ġease . </s> Table ĠT AB REF 7 Ġalso Ġreports Ġthe Ġaverage Ġnumber Ġof ĠâĢ ľ t ries âĢ Ŀ , Ġi . e ., Ġattempts Ġmade Ġfor Ġeach Ġcontext Ġuntil Ġa Ġmodel Ġerror Ġwas Ġfound Ġ( or Ġthe Ġnumber Ġof Ġpossible Ġtries Ġis Ġexceeded ), Ġand Ġthe Ġaverage Ġtime Ġthis Ġtook Ġ( in Ġseconds ). ĠAgain , Ġthese Ġmetrics Ġrepresent Ġa Ġuseful Ġway Ġto Ġevaluate Ġmodel Ġquality . ĠWe Ġobserve Ġthat Ġthe Ġaverage Ġtries Ġand Ġaverage Ġtime Ġper Ġverified Ġerror Ġboth Ġgo Ġup Ġas Ġwe Ġprogress Ġthrough Ġthe Ġrounds . ĠThe Ġnumbers Ġclearly Ġdemonstrate Ġthat Ġthe Ġrounds Ġare Ġgetting Ġincreasingly Ġmore Ġdifficult . </s> Results </s> Table ĠT AB REF 13 Ġreports Ġthe Ġmain Ġresults . ĠIn Ġaddition Ġto ĠB ERT ĠB IB REF 10 Ġand ĠRo BER Ta ĠB IB REF 25 , Ġwe Ġalso Ġinclude ĠXL Net ĠB IB REF 30 Ġas Ġan Ġexample Ġof Ġa Ġstrong , Ġbut Ġdifferent , Ġmodel Ġarchitecture . ĠWe Ġshow Ġtest Ġset Ġperformance Ġon Ġthe ĠAN LI Ġtest Ġsets Ġper Ġround , Ġthe Ġtotal ĠAN LI Ġtest Ġset , Ġand Ġthe Ġexclusive Ġtest Ġsubset Ġ( ex amples Ġfrom Ġtest - set - exclusive Ġworkers ). ĠWe Ġalso Ġshow Ġaccuracy Ġon Ġthe ĠSN LI Ġtest Ġset Ġand Ġthe ĠMN LI Ġdevelopment Ġ( for Ġthe Ġpurpose Ġof Ġcomparing Ġbetween Ġdifferent Ġmodel Ġconfigurations Ġacross Ġtable Ġrows ) Ġset . ĠIn Ġwhat Ġfollows , Ġwe Ġbriefly Ġdiscuss Ġour Ġobservations . </s> Results Ġ: :: ĠBase Ġmodel Ġperformance Ġis Ġlow . </s> Notice Ġthat Ġthe Ġbase Ġmodel Ġfor Ġeach Ġround Ġperforms Ġvery Ġpoorly Ġon Ġthat Ġround 's Ġtest Ġset . ĠThis Ġis Ġthe Ġexpected Ġoutcome : ĠFor Ġround Ġ1 , Ġthe Ġbase Ġmodel Ġgets Ġthe Ġentire Ġtest Ġset Ġwrong , Ġby Ġdesign . ĠFor Ġrounds Ġ2 Ġand Ġ3 , Ġwe Ġused Ġan Ġensemble , Ġso Ġperformance Ġis Ġnot Ġnecessarily Ġzero . ĠHowever , Ġas Ġit Ġturns Ġout , Ġperformance Ġstill Ġfalls Ġwell Ġbelow Ġchance , Ġindicating Ġthat Ġworkers Ġdid Ġnot Ġfind Ġvulnerabilities Ġspecific Ġto Ġa Ġsingle Ġmodel , Ġbut Ġgenerally Ġapplicable Ġones Ġfor Ġthat Ġmodel Ġclass . </s> Results Ġ: :: ĠRounds Ġbecome Ġincreasingly Ġmore Ġdifficult . </s> As Ġalready Ġfore shadow ed Ġby Ġthe Ġdataset Ġstatistics , Ġround Ġ3 Ġis Ġmore Ġdifficult Ġ( y ield s Ġlower Ġperformance ) Ġthan Ġround Ġ2 , Ġand Ġround Ġ2 Ġis Ġmore Ġdifficult Ġthan Ġround Ġ1 . ĠThis Ġis Ġtrue Ġfor Ġall Ġmodel Ġarchitectures . </s> Results Ġ: :: ĠTraining Ġon Ġmore Ġrounds Ġimproves Ġrobust ness . </s> Generally , Ġour Ġresults Ġindicate Ġthat Ġtraining Ġon Ġmore Ġrounds Ġimproves Ġmodel Ġperformance . ĠThis Ġis Ġtrue Ġfor Ġall Ġmodel Ġarchitectures . ĠSimply Ġtraining Ġon Ġmore ĠâĢ ľ normal ĠNL I âĢ Ŀ Ġdata Ġwould Ġnot Ġhelp Ġa Ġmodel Ġbe Ġrobust Ġto Ġadvers arial Ġattacks , Ġbut Ġour Ġdata Ġactively Ġhelps Ġmitigate Ġthese . </s> Results Ġ: :: ĠRo BER Ta Ġachieves Ġstate - of - the - art Ġperformance ... </s> We Ġobtain Ġstate Ġof Ġthe Ġart Ġperformance Ġon Ġboth ĠSN LI Ġand ĠMN LI Ġwith Ġthe ĠRo BER Ta Ġmodel Ġfin et uned Ġon Ġour Ġnew Ġdata ĠThe ĠRo BER Ta Ġpaper ĠB IB REF 25 Ġreports Ġa Ġscore Ġof Ġ$ 90 . 2 $ Ġfor Ġboth ĠMN LI - matched Ġand Ġ- m ism atched Ġdev , Ġwhile Ġwe Ġobtain Ġ$ 91 . 0 $ Ġand Ġ$ 90 . 7 $. ĠThe Ġstate Ġof Ġthe Ġart Ġon ĠSN LI Ġis Ġcurrently Ġheld Ġby ĠMT - D NN ĠB IB REF 31 , Ġwhich Ġreports Ġ$ 91 . 6 $ Ġcompared Ġto Ġour Ġ$ 92 . 9 $. </s> Results Ġ: :: Ġ... but Ġis Ġoutper formed Ġwhen Ġit Ġis Ġbase Ġmodel . </s> However , Ġthe Ġbase Ġ( Ro BER Ta ) Ġmodels Ġfor Ġrounds Ġ2 Ġand Ġ3 Ġare Ġoutper formed Ġby Ġboth ĠB ERT Ġand ĠXL Net . ĠThis Ġshows Ġthat Ġannot ators Ġhave Ġmanaged Ġto Ġwrite Ġexamples Ġthat ĠRo BER Ta Ġgenerally Ġstruggles Ġwith , Ġand Ġmore Ġtraining Ġdata Ġalone Ġcannot Ġeasily Ġmit igated Ġthese Ġshortcomings . ĠIt Ġalso Ġimplies Ġthat ĠB ERT , ĠXL Net , Ġand ĠRo BER Ta Ġall Ġhave Ġdifferent Ġweaknesses , Ġpossibly Ġas Ġa Ġfunction Ġof Ġtheir Ġtraining Ġdata Ġ( BER T , ĠXL Net Ġand ĠRo BER Ta Ġwere Ġtrained Ġon Ġvery Ġdifferent Ġdata Ġsets , Ġwhich Ġmight Ġor Ġmight Ġnot Ġhave Ġcontained Ġinformation Ġrelevant Ġto Ġthe Ġweaknesses )âĢĶ an Ġadditional Ġround Ġwith Ġa Ġwider Ġmodel Ġvariety Ġwould Ġthus Ġbe Ġinteresting Ġto Ġinvestigate Ġas Ġa Ġnext Ġstep . </s> Results Ġ: :: ĠContin uously Ġaugment ing Ġtraining Ġdata Ġdoes Ġnot Ġdowngrade Ġperformance . </s> Even Ġthough ĠAN LI Ġtraining Ġdata Ġis Ġdifferent Ġfrom ĠSN LI Ġand ĠMN LI , Ġadding Ġthis Ġdata Ġto Ġthe Ġtraining Ġset Ġdoes Ġnot Ġharm Ġperformance Ġon Ġthose Ġtasks . ĠFurthermore , Ġas ĠTable ĠT AB REF 21 Ġshows , Ġtraining Ġonly Ġon ĠAN LI Ġis Ġtransfer able Ġto ĠSN LI Ġand ĠMN LI , Ġbut Ġnot Ġvice Ġversa . ĠThis Ġsuggests Ġthat Ġmethods Ġcould Ġsuccessfully Ġbe Ġapplied Ġfor Ġmany Ġmore Ġconsecutive Ġrounds . </s> Results Ġ: :: ĠExclusive Ġtest Ġsubset Ġdifference Ġis Ġsmall . </s> In Ġorder Ġto Ġavoid Ġthe Ġpossibility Ġthat Ġmodels Ġmight Ġpick Ġup Ġon Ġannot ator - specific Ġartifacts , Ġa Ġconcern Ġraised Ġby ĠG eva 2019 task or annot ator , Ġwe Ġincluded Ġan Ġexclusive Ġtest Ġsubset Ġwith Ġexamples Ġfrom Ġannot ators Ġnever Ġseen Ġin Ġthe Ġtraining Ġdata . ĠWe Ġfind Ġthat Ġthe Ġdifferences Ġbetween Ġthis Ġexclusive Ġsubset Ġand Ġthe Ġtest Ġset Ġare Ġsmall , Ġindicating Ġthat Ġour Ġmodels Ġdo Ġnot Ġover - rely Ġon Ġindividual Ġannot ator 's Ġwriting Ġstyles . </s> Results Ġ: :: ĠHyp ot hesis - only Ġresults </s> For ĠSN LI Ġand ĠMN LI , Ġconcerns Ġhave Ġbeen Ġraised Ġabout Ġthe Ġpropensity Ġof Ġmodels Ġto Ġpick Ġup Ġon Ġspurious Ġartifacts Ġthat Ġare Ġpresent Ġjust Ġin Ġthe Ġhypotheses ĠB IB REF 12 , ĠB IB REF 13 . ĠTo Ġstudy Ġthis Ġin Ġthe Ġcontext Ġof Ġour Ġresults Ġand Ġtask Ġdifficulty , Ġwe Ġcompare Ġmodels Ġtrained Ġon Ġ( context , Ġhypothesis ) Ġpairs Ġto Ġmodels Ġtrained Ġonly Ġon Ġthe Ġhypothesis Ġ( marked Ġ$ H $ ). ĠTable ĠT AB REF 21 Ġreports Ġresults Ġon Ġthe Ġthree Ġrounds Ġof ĠAN LI , Ġas Ġwell Ġas ĠSN LI Ġand ĠMN LI . ĠThe Ġtable Ġshows Ġsome Ġinteresting Ġtake - aways : </s> Results Ġ: :: ĠHyp ot hesis - only Ġresults Ġ: :: ĠHyp ot hesis - only Ġmodels Ġperform Ġpoorly Ġon ĠAN LI . </s> We Ġcorrobor ate Ġthat Ġhypothesis - only Ġmodels Ġobtain Ġgood Ġperformance Ġon ĠSN LI Ġand ĠMN LI . ĠPerformance Ġof Ġsuch Ġmodels Ġon ĠAN LI Ġis Ġsubstantially Ġlower , Ġand Ġdecreases Ġwith Ġmore Ġrounds . </s> Results Ġ: :: ĠHyp ot hesis - only Ġresults Ġ: :: ĠRo BER Ta Ġdoes Ġnot Ġoutper form Ġhypothesis - only Ġon Ġrounds Ġ2 Ġand Ġ3 . </s> On Ġthe Ġtwo Ġrounds Ġwhere ĠRo BER Ta Ġwas Ġused Ġas Ġthe Ġbase Ġmodel , Ġits Ġperformance Ġis Ġnot Ġmuch Ġbetter Ġthan Ġthe Ġhypothesis - only Ġmodel . ĠThis Ġcould Ġmean Ġtwo Ġthings : Ġeither Ġthe Ġtest Ġdata Ġis Ġvery Ġdifficult , Ġor Ġthe Ġtraining Ġdata Ġis Ġnot Ġgood . ĠTo Ġrule Ġout Ġthe Ġlatter , Ġwe Ġtrained Ġonly Ġon ĠAN LI Ġ($ \ sim Ġ$ 163 k Ġtraining Ġexamples ): Ġdoing Ġso Ġwith ĠRo BER Ta Ġmatches Ġthe Ġperformance Ġof ĠB ERT Ġon ĠMN LI Ġwhen Ġit Ġis Ġtrained Ġon Ġthe Ġmuch Ġlarger , Ġfully Ġin - domain ĠSN LI + MN LI Ġcombined Ġdataset Ġ( 9 43 k Ġtraining Ġexamples ), Ġwith Ġboth Ġgetting Ġ$\ sim Ġ$ 86 , Ġwhich Ġis Ġimpressive . ĠHence , Ġthis Ġshows Ġthat Ġour Ġnew Ġchallenge Ġtest Ġsets Ġare Ġso Ġdifficult Ġthat Ġthe Ġcurrent Ġstate - of - the - art Ġmodel Ġcannot Ġdo Ġbetter Ġthan Ġa Ġhypothesis - only Ġprior . </s> Analysis </s> We Ġperform Ġtwo Ġtypes Ġof Ġmodel Ġerror Ġanalysis . ĠFirst Ġwe Ġevaluate Ġtwo Ġpopular Ġexisting Ġtest Ġsets Ġthat Ġwere Ġcreated Ġto Ġexpose Ġmodel Ġweaknesses , Ġand Ġshow Ġthat Ġour Ġdataset Ġdiscour ages Ġmodels Ġfrom Ġlearning Ġspurious Ġstatistical Ġfacts , Ġrelative Ġto Ġother Ġlarge Ġpopular Ġdatasets Ġ( e . g ., ĠSN LI Ġand ĠMN LI ). ĠSecondly , Ġwe Ġexplore , Ġby Ġround , Ġthe Ġtypes Ġof Ġinf erences Ġour Ġwriters Ġsuccessfully Ġemployed Ġto Ġstump Ġmodels , Ġby Ġperforming Ġhand - ann otation Ġon Ġ500 Ġexamples Ġfrom Ġeach Ġround 's Ġdevelopment Ġset . </s> Analysis Ġ: :: ĠPerformance Ġon Ġchallenge Ġdatasets </s> Recently , Ġseveral Ġhard Ġtest Ġsets Ġhave Ġbeen Ġmade Ġavailable Ġfor Ġrevealing Ġthe Ġbiases ĠNL I Ġmodels Ġlearn Ġfrom Ġtheir Ġtraining Ġdatasets ĠB IB REF 32 , ĠB IB REF 17 , ĠB IB REF 12 , ĠB IB REF 33 . ĠWe Ġexamined Ġmodel Ġperformance Ġon Ġtwo Ġof Ġthese : Ġthe ĠSN LI - Hard ĠB IB REF 12 Ġtest Ġset , Ġwhich Ġconsists Ġof Ġexamples Ġthat Ġhypothesis - only Ġmodels Ġlabel Ġincorrectly , Ġand Ġthe ĠNL I Ġstress Ġtests ĠB IB REF 33 , Ġin Ġwhich Ġsentences Ġcontaining Ġan ton y ms Ġpairs , Ġneg ations , Ġhigh Ġword Ġoverlap , Ġi . a ., Ġare Ġhe ur istically Ġconstructed . ĠWe Ġtest Ġour Ġmodels Ġon Ġthese Ġstress Ġtests , Ġafter Ġtuning Ġon Ġeach Ġtest 's Ġrespective Ġdevelopment Ġset Ġto Ġaccount Ġfor Ġpotential Ġdomain Ġmism atches . ĠFor Ġcomparison , Ġwe Ġalso Ġreport Ġaccur acies Ġfrom Ġthe Ġoriginal Ġpapers : Ġfor ĠSN LI - Hard Ġwe Ġpresent Ġthe Ġresults Ġfrom ĠB IB REF 12 's Ġimplementation Ġof Ġthe Ġhierarchical Ġtens or - based ĠD ense ly ĠInteractive ĠIn ference ĠNetwork ĠB IB REF 34 Ġon ĠMN LI , Ġand Ġfor Ġthe ĠNL I Ġstress Ġtests , Ġwe Ġpresent Ġthe Ġperformance Ġof ĠB IB REF 33 's Ġimplementation Ġof ĠIn fer Sent ĠB IB REF 3 Ġtrained Ġon ĠSN LI . ĠOur Ġresults Ġare Ġin ĠTable ĠT AB REF 23 . </s> We Ġobserve Ġthat Ġall Ġof Ġour Ġmodels Ġfar Ġoutper form Ġthe Ġmodels Ġpresented Ġin Ġoriginal Ġpapers Ġfor Ġthese Ġcommon Ġstress Ġtests , Ġwith Ġour Ġtwo ĠRo BER Ta Ġmodels Ġperforming Ġbest . ĠBoth Ġperform Ġwell Ġon ĠSN LI - Hard Ġand Ġachieve Ġaccuracy Ġlevels Ġin Ġthe Ġhigh Ġ80 s Ġon Ġthe Ġ` ant onym ' Ġ( AT ), Ġ` n umer ical Ġreasoning ' Ġ( NR ), Ġ` length ' Ġ( L N ), Ġ` spe lling Ġerror ' ( SE ) Ġsub - dat as ets , Ġand Ġshow Ġmarked Ġimprovement Ġon Ġboth Ġ` neg ation ' Ġ( NG ), Ġand Ġ` word Ġoverlap ' Ġ( W O ). ĠTraining Ġa ĠRo BER Ta Ġmodel Ġalso Ġon ĠAN LI Ġappears Ġto Ġbe Ġparticularly Ġuseful Ġfor Ġthe ĠNR , ĠW O , ĠNG Ġand ĠAT ĠNL I Ġstress Ġtests . </s> Analysis Ġ: :: ĠReason ing Ġtypes </s> A Ġdynamically Ġevolving Ġdataset Ġoffers Ġthe Ġunique Ġopportunity Ġto Ġtrack Ġhow Ġmodel Ġerror Ġrates Ġchange Ġover Ġtime . ĠSince Ġeach Ġround 's Ġdevelopment Ġset Ġcontains Ġonly Ġverified Ġexamples , Ġwe Ġcan Ġinvestigate Ġtwo Ġinteresting Ġquestions : Ġwhich Ġtypes Ġof Ġinference Ġdo Ġwriters Ġemploy Ġto Ġfool Ġthe Ġmodels , Ġand Ġare Ġbase Ġmodels Ġdifferent ially Ġsensitive Ġto Ġdifferent Ġtypes Ġof Ġreasoning ? ĠThe Ġresults Ġare Ġsummarized Ġin ĠTable ĠT AB REF 27 . </s> We Ġemployed Ġan Ġexpert Ġlingu ist Ġannot ator Ġto Ġdevise Ġan Ġont ology Ġof Ġinference Ġtypes Ġthat Ġwould Ġbe Ġspecific Ġto ĠNL I . ĠWhile Ġdesigning Ġan Ġappropriate Ġont ology Ġof Ġtypes Ġof Ġinference Ġis Ġfar Ġfrom Ġstraightforward , Ġwe Ġfound Ġthat Ġa Ġunified Ġont ology Ġcould Ġbe Ġutilized Ġto Ġcharacterize Ġexamples Ġfrom Ġall Ġthree Ġrounds , Ġwhich Ġsuggests Ġthat Ġit Ġhas Ġat Ġleast Ġsome Ġgeneral ize able Ġapplic ability . ĠThe Ġont ology Ġwas Ġused Ġto Ġlabel Ġ500 Ġexamples Ġfrom Ġeach ĠAN LI Ġdevelopment Ġset . </s> The Ġinference Ġont ology Ġcontains Ġsix Ġtypes Ġof Ġinference : ĠN umer ical Ġ& ĠQuant itative Ġ( i . e ., Ġreasoning Ġabout Ġcardinal Ġand Ġord inal Ġnumbers , Ġinfer ring Ġdates Ġand Ġages Ġfrom Ġnumbers , Ġetc .), ĠReference Ġ& ĠNames Ġ( core f erences Ġbetween Ġpronouns Ġand Ġforms Ġof Ġproper Ġnames , Ġknowing Ġfacts Ġabout Ġname Ġgender , Ġetc .), ĠBasic ĠInf erences Ġ( con jun ctions , Ġneg ations , Ġcause - and - effect , Ġcompar atives Ġand Ġsuper l atives Ġetc .), ĠLex ical ĠIn ference Ġ( inf erences Ġmade Ġpossible Ġby Ġlex ical Ġinformation Ġabout Ġsyn onyms , Ġan ton y ms , Ġetc .), ĠTr icky ĠInf erences Ġ( word play , Ġlinguistic Ġstrategies Ġsuch Ġas Ġsynt actic Ġtransformations / re order ings , Ġor Ġinfer ring Ġwriter Ġintentions Ġfrom Ġcontexts ), Ġand Ġreasoning Ġfrom Ġoutside Ġknowledge Ġor Ġadditional Ġfacts Ġ( e . g ., ĠâĢ ľ You Ġcan 't Ġreach Ġthe Ġsea Ġdirectly Ġfrom ĠDj ib out i âĢ Ŀ ). ĠThe Ġquality Ġof Ġannotations Ġwas Ġalso Ġtracked ; Ġif Ġa Ġpair Ġwas Ġambiguous Ġor Ġhad Ġa Ġlabel Ġthat Ġseemed Ġincorrect Ġ( from Ġthe Ġexpert Ġannot ator 's Ġperspective ), Ġit Ġwas Ġflagged . ĠRound Ġ1 âĢĵ 3 Ġdevelopment Ġsets Ġcontained Ġfew Ġ` Quality ' Ġtags ; Ġthe Ġincidence Ġof Ġquality Ġissues Ġwas Ġstable Ġat Ġbetween Ġ3 % Ġand Ġ4 % Ġper Ġround . ĠAny Ġone Ġexample Ġcan Ġhave Ġmultiple Ġtypes , Ġand Ġevery Ġexample Ġcontained Ġat Ġleast Ġone Ġtag . </s> As Ġrounds Ġ1 Ġand Ġ2 Ġwere Ġboth Ġbuilt Ġwith Ġcontexts Ġfrom Ġthe Ġsame Ġgenre Ġ( Wikipedia ), Ġwe Ġmight Ġexpect Ġwriters Ġto Ġarrive Ġat Ġsimilar Ġstrategies . ĠHowever , Ġsince Ġthe Ġmodel Ġarchitectures Ġused Ġin Ġthe Ġfirst Ġtwo Ġrounds Ġdiffer , Ġwriters Ġmight Ġbe Ġsufficiently Ġcreative Ġin Ġfinding Ġdifferent Ġexploits Ġin Ġeach . ĠFor Ġround Ġ3 , Ġwe Ġexpect Ġsome Ġdifference Ġin Ġreasoning Ġtypes Ġto Ġbe Ġpresent , Ġbecause Ġwe Ġused Ġsource Ġmaterial Ġfrom Ġseveral Ġdomains Ġas Ġour Ġcontexts . ĠIn Ġsum , Ġany Ġchange Ġbetween Ġrounds Ġcould Ġbe Ġdue Ġto Ġany Ġof Ġthe Ġfollowing Ġfactors : Ġinherent Ġdifferences Ġbetween Ġdata Ġcollection , Ġmodel Ġarchitectures Ġand Ġmodel Ġtraining Ġdata , Ġrandom Ġselection Ġof Ġcontexts , Ġor Ġslight Ġdifferences Ġin Ġwriter Ġpool Ġor Ġwriter Ġpreferences . </s> We Ġobserve Ġthat Ġboth Ġround Ġ1 Ġand Ġ2 Ġwriters Ġrely Ġheavily Ġon Ġnumerical Ġand Ġquantitative Ġreasoning Ġin Ġover Ġ30 % Ġof Ġthe Ġdevelopment Ġset âĢĶ the Ġpercentage Ġin ĠA 2 Ġ( 32 %) Ġdropped Ġroughly Ġ6 % Ġfrom ĠA 1 Ġ( 38 %) âĢĶ while Ġround Ġ3 Ġwriters Ġuse Ġnumerical Ġor Ġquantitative Ġreasoning Ġfor Ġonly Ġ17 %. ĠThe Ġmajority Ġof Ġnumerical Ġreasoning Ġtypes Ġwere Ġreferences Ġto Ġcardinal Ġnumbers Ġthat Ġreferred Ġto Ġdates Ġand Ġages . ĠInf erences Ġpred icated Ġon Ġreferences Ġand Ġnames Ġwere Ġpresent Ġin Ġabout Ġ10 % Ġof Ġrounds Ġ1 Ġ& Ġ3 Ġdevelopment Ġsets , Ġand Ġreached Ġa Ġhigh Ġof Ġ20 % Ġin Ġround Ġ2 , Ġwith Ġcore ference Ġfeaturing Ġprominently . ĠBasic Ġinference Ġtypes Ġincreased Ġin Ġprevalence Ġas Ġthe Ġrounds Ġincreased , Ġranging Ġfrom Ġ18 % âĢĵ 30 %, Ġas Ġdid ĠLex ical Ġinf erences Ġ( increasing Ġfrom Ġ13 % âĢĵ 33 %). ĠThe Ġpercentage Ġof Ġsentences Ġrelying Ġon Ġreasoning Ġand Ġoutside Ġfacts Ġremains Ġroughly Ġthe Ġsame , Ġin Ġthe Ġmid - 50 s , Ġperhaps Ġslightly Ġincreasing Ġafter Ġround Ġ1 . ĠFor Ġround Ġ3 , Ġwe Ġobserve Ġthat Ġthe Ġmodel Ġused Ġto Ġcollect Ġit Ġappears Ġto Ġbe Ġmore Ġsusceptible Ġto ĠBasic , ĠLex ical , Ġand ĠTr icky Ġinference Ġtypes . ĠThis Ġfinding Ġis Ġcompatible Ġwith Ġthe Ġidea Ġthat Ġthe Ġmodels Ġtrained Ġon Ġadvers arial Ġdata Ġare Ġmore Ġimpressive Ġand Ġperform Ġbetter , Ġencouraging Ġwriters Ġto Ġdevise Ġmore Ġcreative Ġexamples Ġcontaining Ġharder Ġtypes Ġof Ġinference Ġin Ġorder Ġto Ġstump Ġthem . </s> Related Ġwork Ġ: :: ĠB ias Ġin Ġdatasets </s> Machine Ġlearning Ġmethods Ġare Ġwell - known Ġto Ġpick Ġup Ġon Ġspurious Ġstatistical Ġpatterns . ĠFor Ġinstance , Ġin Ġimage Ġcaption ing , Ġa Ġsimple Ġbaseline Ġof Ġutilizing Ġthe Ġcapt ions Ġof Ġnearest Ġneighbors Ġin Ġthe Ġtraining Ġset Ġwas Ġshown Ġto Ġyield Ġimpressive ĠB LE U Ġscores ĠB IB REF 35 . ĠIn Ġthe Ġfirst Ġvisual Ġquestion Ġanswering Ġdataset ĠB IB REF 36 , Ġbiases Ġlike ĠâĢ ľ 2 âĢ Ŀ Ġbeing Ġthe Ġcorrect Ġanswer Ġto Ġ39 % Ġof Ġthe Ġquestions Ġstarting Ġwith ĠâĢ ľ how Ġmany âĢ Ŀ Ġallowed Ġlearning Ġalgorithms Ġto Ġperform Ġwell Ġwhile Ġignoring Ġthe Ġvisual Ġmod ality Ġaltogether ĠB IB REF 37 , ĠB IB REF 38 . ĠThe Ġfield Ġhas Ġa Ġtendency Ġto Ġover fit Ġon Ġstatic Ġtargets , Ġeven Ġif Ġthat Ġdoes Ġnot Ġhappen Ġdeliberately ĠB IB REF 39 . </s> In ĠNL I , ĠGur ur angan 2018 ann otation , ĠPol iak 2018 hyp ot hesis Ġand ĠT such iya 2018 performance Ġshowed Ġthat Ġhypothesis - only Ġbas elines Ġoften Ġperform Ġfar Ġbetter Ġthan Ġchance . ĠIt Ġhas Ġbeen Ġshown Ġthat ĠNL I Ġsystems Ġcan Ġoften Ġbe Ġbroken Ġmerely Ġby Ġperforming Ġsimple Ġlex ical Ġsubstit utions ĠB IB REF 15 , Ġand Ġthat Ġthey Ġstruggle Ġwith Ġquant ifiers ĠB IB REF 40 Ġand Ġcertain Ġsuperficial Ġsynt actic Ġproperties ĠB IB REF 17 . ĠIn Ġreading Ġcomprehension Ġand Ġquestion Ġanswering , ĠKa ush ik 2018 how much Ġshowed Ġthat Ġquestion - Ġand Ġpassage - only Ġmodels Ġcan Ġperform Ġsurprisingly Ġwell , Ġwhile ĠJ ia 2017 ad vers arial Ġadded Ġadvers ar ially Ġconstructed Ġsentences Ġto Ġpassages , Ġleading Ġto Ġa Ġdrastic Ġdrop Ġin Ġperformance . ĠMany Ġtext Ġclassification Ġdatasets Ġdo Ġnot Ġrequire Ġsophisticated Ġlinguistic Ġreasoning , Ġas Ġshown Ġby Ġthe Ġsurprisingly Ġgood Ġperformance Ġof Ġrandom Ġenc od ers ĠB IB REF 41 . ĠSimilar Ġobservations Ġwere Ġmade Ġin Ġmachine Ġtranslation ĠB IB REF 42 Ġand Ġdialogue ĠB IB REF 43 . ĠIn Ġshort , Ġthe Ġfield Ġis Ġrife Ġwith Ġdataset Ġbias Ġand Ġpapers Ġtrying Ġaddress Ġthis Ġimportant Ġproblem . ĠThis Ġwork Ġcan Ġbe Ġviewed Ġas Ġa Ġnatural Ġextension : Ġif Ġsuch Ġbiases Ġexist , Ġthey Ġwill Ġallow Ġhumans Ġto Ġfool Ġthe Ġmodels , Ġadding Ġuseful Ġexamples Ġto Ġthe Ġtraining Ġdata Ġuntil Ġthe Ġbias Ġis Ġdynamically Ġmit igated . </s> Related Ġwork Ġ: :: ĠDynamic Ġdatasets . </s> Con currently Ġwith Ġthis Ġwork , ĠAnonymous 2020 ad vers arial fil ters Ġproposed ĠAFL ite , Ġan Ġiter ative Ġapproach Ġfor Ġfiltering Ġadvers arial Ġdata Ġpoints Ġto Ġavoid Ġspurious Ġbiases . ĠKa ush ik 2019 learning diff erence Ġoffer Ġa Ġcausal Ġaccount Ġof Ġspurious Ġpatterns , Ġand Ġcounter fact ually Ġaugment ĠNL I Ġdatasets Ġby Ġediting Ġexamples Ġto Ġbreak Ġthe Ġmodel . ĠThe Ġformer Ġis Ġan Ġexample Ġof Ġa Ġmodel - in - the - loop Ġsetting , Ġwhere Ġthe Ġmodel Ġis Ġiter atively Ġprob ed Ġand Ġimproved . ĠThe Ġlatter Ġis Ġhuman - in - the - loop Ġtraining , Ġwhere Ġhumans Ġare Ġused Ġto Ġfind Ġproblems Ġwith Ġone Ġsingle Ġmodel . ĠIn Ġthis Ġwork , Ġwe Ġemploy Ġboth Ġstrategies Ġiter atively , Ġin Ġa Ġform Ġof Ġhuman - and - model - in - the - loop Ġtraining , Ġto Ġcollect Ġcompletely Ġnew Ġexamples , Ġin Ġa Ġpotentially Ġnever - ending Ġloop ĠB IB REF 18 . ĠRelated ly , Ġlan 2017 pp db Ġpropose Ġa Ġmethod Ġfor Ġcontinuously Ġgrowing Ġa Ġdataset Ġof Ġparaph r ases . </s> Human - and - model - in - the - loop Ġtraining Ġis Ġnot Ġa Ġnew Ġidea . ĠMechanical ĠTur ker ĠDes cent Ġproposes Ġa Ġgam ified Ġenvironment Ġfor Ġthe Ġcollaborative Ġtraining Ġof Ġgrounded Ġlanguage Ġlearning Ġagents Ġover Ġmultiple Ġrounds ĠB IB REF 19 . ĠThe ĠâĢ ľ Build Ġit ĠBreak Ġit ĠFix Ġit âĢ Ŀ Ġstrategy Ġin Ġthe Ġsecurity Ġdomain ĠB IB REF 44 Ġhas Ġbeen Ġadapted Ġto ĠN LP ĠB IB REF 20 Ġas Ġwell Ġas Ġdialogue ĠB IB REF 45 . ĠThe ĠQ Ap edia Ġframework ĠB IB REF 46 Ġcontinuously Ġref ines Ġand Ġupdates Ġits Ġcontent Ġrepository Ġusing Ġhumans Ġin Ġthe Ġloop , Ġwhile Ġhuman Ġfeedback Ġloops Ġhave Ġbeen Ġused Ġto Ġimprove Ġimage Ġcaption ing Ġsystems ĠB IB REF 47 . Ġwall ace 2018 tr ick Ġleverage Ġtrivia Ġexperts Ġto Ġcreate Ġa Ġmodel - driven Ġadvers arial Ġquestion Ġwriting Ġprocedure Ġand Ġgenerate Ġa Ġsmall Ġset Ġof Ġchallenge Ġquestions Ġthat ĠQ A - models Ġfail Ġon . </s> There Ġhas Ġbeen Ġa Ġflurry Ġof Ġwork Ġin Ġconstructing Ġdatasets Ġwith Ġan Ġadvers arial Ġcomponent , Ġsuch Ġas ĠSw ag ĠB IB REF 48 Ġand ĠHell a Sw ag ĠB IB REF 49 , ĠC OD AH ĠB IB REF 50 , ĠAd vers arial ĠS Qu AD ĠB IB REF 51 , ĠLamb ada ĠB IB REF 52 Ġand Ġothers . ĠOur Ġdataset Ġis Ġnot Ġto Ġbe Ġconfused Ġwith Ġabduct ive ĠNL I ĠB IB REF 53 , Ġwhich Ġcalls Ġitself Ġ$\ alpha Ġ$ N LI , Ġor ĠART . </s> Discussion Ġ& ĠConclusion </s> In Ġthis Ġwork , Ġwe Ġused Ġa Ġhuman - and - model - in - the - loop Ġentail ment Ġtraining Ġmethod Ġto Ġcollect Ġa Ġnew Ġbenchmark Ġfor Ġnatural Ġlanguage Ġunderstanding . ĠThe Ġbenchmark Ġis Ġdesigned Ġto Ġbe Ġchallenging Ġto Ġcurrent Ġstate Ġof Ġthe Ġart Ġmodels . ĠAn not ators Ġwere Ġemployed Ġto Ġact Ġas Ġadversaries , Ġand Ġencouraged Ġto Ġfind Ġvulnerabilities Ġthat Ġfool Ġthe Ġmodel Ġinto Ġpredicting Ġthe Ġwrong Ġlabel , Ġbut Ġthat Ġanother Ġperson Ġwould Ġcorrectly Ġclassify . ĠWe Ġfound Ġthat Ġnon - ex pert Ġannot ators , Ġin Ġthis Ġgam ified Ġsetting Ġand Ġwith Ġappropriate Ġincentives Ġto Ġfool Ġthe Ġmodel , Ġare Ġremarkably Ġcreative Ġat Ġfinding Ġand Ġexploiting Ġweaknesses Ġin Ġmodels . ĠWe Ġcollected Ġthree Ġrounds , Ġand Ġas Ġthe Ġrounds Ġprogressed , Ġthe Ġmodels Ġbecame Ġmore Ġrobust Ġand Ġthe Ġtest Ġsets Ġfor Ġeach Ġround Ġbecame Ġmore Ġdifficult . ĠTraining Ġon Ġthis Ġnew Ġdata Ġyielded Ġthe Ġstate Ġof Ġthe Ġart Ġon Ġexisting ĠNL I Ġbenchmarks . </s> The ĠAN LI Ġbenchmark Ġpresents Ġa Ġnew Ġchallenge Ġto Ġthe Ġcommunity . ĠIt Ġwas Ġcarefully Ġconstructed Ġto Ġmitigate Ġissues Ġwith Ġprevious Ġdatasets , Ġand Ġwas Ġdesigned Ġfrom Ġfirst Ġprinciples Ġto Ġlast Ġlonger âĢĶ if Ġthe Ġtest Ġset Ġsatur ates , Ġthe Ġfield Ġcan Ġsimply Ġtrain Ġup Ġa Ġnew Ġmodel , Ġcollect Ġmore Ġdata Ġand Ġfind Ġitself Ġconfronted Ġyet Ġagain Ġwith Ġa Ġdifficult Ġchallenge . </s> The Ġdataset Ġalso Ġpresents Ġmany Ġopportunities Ġfor Ġfurther Ġstudy . ĠFor Ġinstance , Ġwe Ġcollected Ġannot ator - provided Ġexplanations Ġfor Ġeach Ġexample Ġthat Ġthe Ġmodel Ġgot Ġwrong . ĠWe Ġprovided Ġinference Ġlabels Ġfor Ġthe Ġdevelopment Ġset , Ġopening Ġup Ġpossibilities Ġfor Ġinteresting Ġmore Ġfine - gr ained Ġstudies Ġof ĠNL I Ġmodel Ġperformance . ĠWhile Ġwe Ġverified Ġthe Ġdevelopment Ġand Ġtest Ġexamples , Ġwe Ġdid Ġnot Ġverify Ġthe Ġcorrectness Ġof Ġeach Ġtraining Ġexample , Ġwhich Ġmeans Ġthere Ġis Ġprobably Ġsome Ġroom Ġfor Ġimprovement Ġthere . </s> The Ġbenchmark Ġis Ġmeant Ġto Ġbe Ġa Ġchallenge Ġfor Ġmeasuring ĠNL U Ġprogress , Ġeven Ġfor Ġas Ġyet Ġund iscovered Ġmodels Ġand Ġarchitectures . ĠWe Ġplan Ġfor Ġthe Ġbenchmark Ġitself Ġto Ġadapt Ġto Ġthese Ġnew Ġmodels Ġby Ġcontinuing Ġto Ġbuild Ġnew Ġchallenge Ġrounds . ĠAs Ġa Ġfirst Ġnext Ġstep , Ġit Ġwould Ġbe Ġinteresting Ġto Ġexamine Ġresults Ġwhen Ġannot ators Ġare Ġconfronted Ġwith Ġa Ġwide Ġvariety Ġof Ġmodel Ġarchitectures . ĠWe Ġhope Ġthat Ġthe Ġdataset Ġwill Ġprove Ġto Ġbe Ġan Ġinteresting Ġnew Ġchallenge Ġfor Ġthe Ġcommunity . ĠLuckily , Ġif Ġit Ġdoes Ġturn Ġout Ġto Ġsatur ate Ġquickly , Ġwe Ġwill Ġalways Ġbe Ġable Ġto Ġcollect Ġa Ġnew Ġround . </s> Acknowled gments </s> YN Ġand ĠMB Ġwere Ġsponsored Ġby ĠDAR PA ĠM CS ĠGrant Ġ# N 66 001 - 19 - 2 - 40 31 , ĠON R ĠGrant Ġ# N 0001 4 - 18 - 1 - 28 71 , Ġand ĠDAR PA ĠY FA 17 - D 17 AP 000 22 .