RoBERTa
EXTRACTIVE
RoBERTa
EXTRACTIVE
Looking at the current WinoGrande leaderboard, it appears that the previous state of the art is based on RoBERTa BIBREF2, which can be characterized as an encoder-only transformer architecture. Since T5-3B is larger than RoBERTa, it cannot be ruled out that model size alone explains the performance gain. However, when coupled with the observations of Nogueira et al. BIBREF7, T5's “generative capability”, i.e., its ability to generate fluent text, honed through pretraining, seems to play an important role. The fact that the choice of target tokens affects prediction accuracy is consistent with this observation. How and why is the subject of ongoing work.
Looking at the current WinoGrande leaderboard, it appears that the previous state of the art is based on RoBERTa BIBREF2, which can be characterized as an encoder-only transformer architecture. Since T5-3B is larger than RoBERTa, it cannot be ruled out that model size alone explains the performance gain. However, when coupled with the observations of Nogueira et al. BIBREF7, T5's “generative capability”, i.e., its ability to generate fluent text, honed through pretraining, seems to play an important role. The fact that the choice of target tokens affects prediction accuracy is consistent with this observation. How and why is the subject of ongoing work.
2003.08380
Introduction </s> Other Ġthan Ġenc oder - only Ġpret rained Ġtransformer Ġarchitectures ĠB IB REF 2 , ĠB IB REF 3 , ĠB IB REF 4 , Ġenc oder âĢĵ dec oder Ġstyle Ġpret rained Ġtransform ers ĠB IB REF 0 , ĠB IB REF 5 Ġhave Ġbeen Ġproven Ġto Ġbe Ġeffective Ġin Ġtext Ġgeneration Ġtasks Ġas Ġwell Ġas Ġcomprehension Ġtasks . ĠThis Ġpaper Ġdescribes Ġour Ġsubmission Ġto Ġthe Ġcommons ense Ġreasoning Ġtask Ġleader board Ġof Ġthe ĠAI 2 ĠW ino Grand e ĠChallenge ĠB IB REF 1 , Ġwhich Ġuses Ġthe Ġtext - to - text Ġtransfer Ġtransformer Ġ( T 5 ); Ġour Ġapproach Ġcurrently Ġrepresents Ġthe Ġstate Ġof Ġthe Ġart . </s> In ĠT 5 ĠB IB REF 0 , ĠN LP Ġtasks Ġare Ġformulated Ġas Ġtext - to - text Ġproblems , Ġwhere Ġthe Ġinputs Ġare Ġcast Ġinto Ġnatural Ġlanguage Ġtemplates Ġthat Ġcontain Ġthe Ġtask Ġdescript ors . ĠCon crete ly , ĠR aff el Ġet Ġal . Ġprovide Ġthe Ġfollowing Ġexample Ġfor ĠMN LI ĠB IB REF 6 , Ġwhere Ġthe Ġgoal Ġis Ġto Ġpredict Ġwhether Ġa Ġpremise Ġimplies Ġ( âĢ ľ ent ail ment âĢ Ŀ ) Ġor Ġcontradicts Ġ( âĢ ľ cont rad iction âĢ Ŀ ) Ġa Ġhypothesis , Ġor Ġneither Ġ( âĢ ľ neutral âĢ Ŀ ). ĠThus , Ġa Ġtraining Ġexample Ġbecomes : </s> âĢ ľ mn li Ġpremise : ĠI Ġhate Ġpige ons . Ġhypothesis : ĠMy Ġfeelings Ġtowards Ġpige ons Ġare Ġfilled Ġwith Ġanimosity . âĢ Ŀ </s> with ĠâĢ ľ ent ail ment âĢ Ŀ Ġas Ġthe Ġcorresponding Ġground Ġtruth Ġtarget Ġoutput . ĠIn Ġother Ġwords , Ġa Ġtoken Ġrepresenting Ġeach Ġclass Ġis Ġdirectly Ġused Ġas Ġthe Ġprediction Ġtarget . </s> App roach </s> The Ġnatural Ġlanguage Ġtemplate Ġapproach Ġenables Ġvarious Ġoptions Ġto Ġformulate Ġthe ĠW ino Grand e Ġcommons ense Ġreasoning Ġtask Ġas Ġa Ġtext - to - text Ġproblem Ġwith ĠT 5 . ĠHere Ġwe Ġadopt Ġa Ġformulation Ġsimilar Ġto Ġthe ĠMN LI Ġtemplate . ĠConsider Ġa Ġconcrete Ġexample : </s> He Ġnever Ġcomes Ġto Ġmy Ġhome , Ġbut ĠI Ġalways Ġgo Ġto Ġhis Ġhouse Ġbecause Ġthe Ġ_ Ġis Ġsmaller . </s> Option 1 : Ġhome ; ĠOption 2 : Ġhouse </s> In Ġthis Ġcase , Ġthe Ġcorrect Ġreplacement Ġfor Ġ_ Ġis ĠOption 1 . ĠWe Ġdecom pose Ġthe Ġabove Ġproblem Ġinto Ġtwo Ġsource âĢĵ target Ġtraining Ġexamples , Ġwhere Ġ_ Ġis Ġreplaced Ġwith Ġeach Ġoption Ġand Ġannot ated Ġwith Ġthe Ġcorrect Ġanswer Ġas Ġthe Ġtarget Ġtoken , Ġas Ġshown Ġin ĠTable ĠT AB REF 2 . ĠIn Ġaddition , Ġwe Ġreform ulate Ġeach Ġexample Ġinto Ġa Ġcommons ense Ġreasoning ĠâĢ ľ template âĢ Ŀ Ġwith Ġtwo Ġstatements : Ġhypothesis Ġ( from Ġ_ Ġto Ġthe Ġend Ġof Ġthe Ġoriginal Ġproblem Ġstatement ) Ġand Ġpremise Ġ( the Ġremaining Ġpart Ġof Ġthe Ġoriginal Ġproblem Ġstatement ). ĠNote Ġthat Ġthe Ġbold Ġand Ġcolored Ġfonts Ġare Ġfor Ġclarity Ġonly ; Ġthose Ġtokens Ġare Ġnot Ġmarked Ġin Ġany Ġway Ġin Ġthe Ġmodel Ġinput . </s> At Ġinference Ġ( test ) Ġtime , Ġwe Ġalso Ġdecom pose Ġthe Ġproblem Ġinto Ġtwo Ġinputs , Ġwhere Ġeach Ġinput Ġis Ġformulated Ġin Ġexactly Ġthe Ġsame Ġmanner Ġas Ġin ĠTable ĠT AB REF 2 , Ġwith Ġeither Ġone Ġof Ġthe Ġanswer Ġoptions . ĠWe Ġthen Ġfeed Ġeach Ġinto ĠT 5 Ġto Ġpredict Ġa Ġtarget Ġtoken . ĠIn Ġthis Ġscenario , Ġthere Ġare Ġfour Ġpossible Ġoutcomes : </s> one Ġproduces ĠâĢ ľ ent ail ment âĢ Ŀ Ġand Ġthe Ġother ĠâĢ ľ cont rad iction âĢ Ŀ , </s> one Ġproduces ĠâĢ ľ ent ail ment âĢ Ŀ Ġor ĠâĢ ľ cont rad iction âĢ Ŀ Ġand Ġthe Ġother Ġsome Ġother Ġtoken , </s> both Ġproduce Ġsome Ġother Ġtokens , Ġand </s> both Ġproduce Ġthe Ġsame Ġtoken , Ġeither ĠâĢ ľ ent ail ment âĢ Ŀ Ġor ĠâĢ ľ cont rad iction âĢ Ŀ . </s> Ide ally , ĠT 5 Ġwould Ġproduce Ġcontrast ive Ġtokens Ġfor Ġeach Ġinput Ġpair , Ġas Ġin Ġcase Ġ( 1 ), Ġwhich Ġallows Ġus Ġto Ġunamb ig uously Ġselect Ġthe Ġfinal Ġanswer . ĠHowever , Ġthe Ġmodel Ġmight Ġproduce Ġthe Ġsame Ġtokens Ġfor Ġeach Ġinput , Ġor Ġeven Ġtokens Ġnot Ġin Ġthe Ġpred efined Ġset , Ġas Ġin Ġcases Ġ( 2 ) Ġto Ġ( 4 ). ĠTo Ġdeal Ġwith Ġthese Ġcases , Ġwe Ġapply Ġa Ġsoft max Ġover Ġthe Ġlog its Ġof Ġthe Ġpair Ġof Ġpred efined Ġtarget Ġtokens , Ġsimilar Ġto ĠN ogue ira Ġet Ġal . ĠB IB REF 7 . ĠFrom Ġthis , Ġwe Ġcan Ġcompute Ġthe Ġprobabilities Ġof Ġthe Ġpred efined Ġtarget Ġtokens Ġ( in Ġthe Ġcase Ġof ĠTable ĠT AB REF 2 , ĠâĢ ľ ent ail ment âĢ Ŀ Ġand ĠâĢ ľ cont rad iction âĢ Ŀ ). ĠThen , Ġwe Ġcompare Ġthe Ġprobabilities Ġacross Ġboth Ġinput Ġinstances , Ġand Ġin Ġcases Ġ( 2 ) Ġto Ġ( 4 ), Ġwe Ġselect Ġthe Ġinstance Ġthat Ġhas Ġa Ġhigher Ġprobability Ġas Ġthe Ġcorrect Ġanswer . </s> This Ġgeneral Ġproblem Ġsetup Ġallows Ġus Ġto Ġchoose Ġthe Ġtarget Ġtokens , Ġwhich Ġmay Ġhave Ġan Ġimpact Ġon Ġthe Ġprediction Ġaccuracy ĠB IB REF 7 . ĠIn Ġaddition Ġto Ġselecting ĠâĢ ľ ent ail ment âĢ Ŀ Ġvs . ĠâĢ ľ cont rad iction âĢ Ŀ Ġas Ġthe Ġtarget , Ġwe Ġalso Ġtried Ġthe Ġcontrast ive Ġpair ĠâĢ ľ true âĢ Ŀ Ġvs . ĠâĢ Ŀ false âĢ Ŀ . </s> In Ġour Ġexperiment , Ġwe Ġfine - t une ĠT 5 - 3 B Ġon ĠGoogle ĠCol ab 's ĠT PU Ġv 2 Ġwith Ġa Ġbatch Ġsize Ġof Ġ16 , Ġa Ġlearning Ġrate Ġof Ġ$ 2 Ġ\ cd ot Ġ10 ^{ - 4 } $, Ġand Ġsave Ġmodel Ġcheckpoints Ġevery Ġ5000 Ġsteps . ĠIt Ġtakes Ġ130 k Ġsteps Ġto Ġconverge Ġfor Ġthe ĠXL Ġdata Ġsize Ġ( see Ġbelow ). ĠAt Ġinference Ġtime , Ġwe Ġuse Ġgreedy Ġdecoding Ġand Ġselect Ġfor Ġevaluation Ġthe Ġmodel Ġcheckpoint Ġthat Ġachieves Ġthe Ġhighest Ġscore Ġon Ġthe Ġdevelopment Ġset . ĠWe Ġdid Ġnot Ġexperiment Ġwith ĠT 5 - 11 B Ġdue Ġto Ġlimited Ġcomputational Ġresources . </s> Results </s> Exper imental Ġresults Ġon Ġthe ĠW ino Grand e Ġdevelopment Ġset Ġare Ġreported Ġin ĠTable ĠT AB REF 7 Ġfor Ġdifferent Ġtraining Ġdata Ġsizes . ĠNote Ġthat Ġwe Ġfine - t une Ġthe Ġmodel Ġfor Ġeach Ġtraining Ġdata Ġsize Ġseparately . ĠA Ġunder Ġthe ĠâĢ ľ log it âĢ Ŀ Ġcolumn Ġindicates Ġthat Ġwe Ġused Ġthe Ġsoft max Ġover Ġthe Ġtarget Ġtokens Ġas Ġdescribed Ġabove . ĠWithout Ġthis Ġtechnique , Ġgiven Ġthe Ġoriginal Ġtwo - choice Ġquestion , Ġif ĠT 5 Ġoutputs Ġthe Ġsame Ġtokens Ġfor Ġthe Ġtwo Ġprocessed Ġinputs , Ġwe Ġsimply Ġassign ĠOption 1 Ġas Ġthe Ġanswer . ĠThe Ġtable Ġalso Ġreports ĠâĢ ľ zero - shot âĢ Ŀ Ġperformance , Ġi . e ., Ġperforming Ġinference Ġon Ġthe Ġdevelopment Ġset Ġwithout Ġany Ġmodel Ġfine Ġtuning . ĠCondition Ġ# 2 Ġrepresents Ġour Ġsubmission Ġto Ġthe Ġofficial Ġleader board , Ġwhich Ġachieves Ġ0 . 767 3 ĠA UC Ġon Ġthe Ġheld - out Ġtest Ġset . </s> From Ġthese Ġresults , Ġwe Ġsee Ġthat Ġthe Ġlog it Ġtrick Ġclearly Ġimproves Ġperformance , Ġwhich Ġis Ġconsistent Ġwith Ġthe Ġobservations Ġof ĠN ogue ira Ġet Ġal . ĠB IB REF 7 . ĠIn Ġfact , Ġapplying Ġthis Ġtechnique Ġin Ġthe Ġzero - shot Ġsetting Ġyields Ġperformance Ġthat Ġis Ġclearly Ġbetter Ġthan Ġrandom . ĠAnother Ġinteresting Ġfinding Ġis Ġthat Ġthe Ġchoice Ġof Ġtarget Ġtoken Ġappears Ġto Ġhave Ġan Ġimpact Ġon Ġperformance , Ġwhich Ġis Ġalso Ġconsistent Ġwith Ġthe Ġabove Ġwork . ĠSince Ġusing Ġtrue / false Ġas Ġthe Ġtarget Ġtoken Ġ( cond itions Ġ# 3 Ġand Ġ# 4 ) Ġdid Ġnot Ġimprove Ġperformance Ġmuch Ġover Ġconditions Ġwith Ġentail ment / cont rad iction , Ġwe Ġdid Ġnot Ġrun Ġall Ġdata Ġsize Ġconditions Ġgiven Ġour Ġlimited Ġcomputational Ġresources . </s> Looking Ġat Ġthe Ġcurrent ĠW ino Grand e Ġleader board , Ġit Ġappears Ġthat Ġthe Ġprevious Ġstate Ġof Ġthe Ġart Ġis Ġbased Ġon ĠRo BER Ta ĠB IB REF 2 , Ġwhich Ġcan Ġbe Ġcharacterized Ġas Ġan Ġenc oder - only Ġtransformer Ġarchitecture . ĠSince ĠT 5 - 3 B Ġis Ġlarger Ġthan ĠRo BER Ta , Ġit Ġcannot Ġbe Ġruled Ġout Ġthat Ġmodel Ġsize Ġalone Ġexplains Ġthe Ġperformance Ġgain . ĠHowever , Ġwhen Ġcoupled Ġwith Ġthe Ġobservations Ġof ĠN ogue ira Ġet Ġal . ĠB IB REF 7 , ĠT 5 's ĠâĢ ľ gener ative Ġcapability âĢ Ŀ , Ġi . e ., Ġits Ġability Ġto Ġgenerate Ġfluent Ġtext , Ġhon ed Ġthrough Ġpret raining , Ġseems Ġto Ġplay Ġan Ġimportant Ġrole . ĠThe Ġfact Ġthat Ġthe Ġchoice Ġof Ġtarget Ġtokens Ġaffects Ġprediction Ġaccuracy Ġis Ġconsistent Ġwith Ġthis Ġobservation . ĠHow Ġand Ġwhy Ġis Ġthe Ġsubject Ġof Ġongoing Ġwork . </s> Impl ications </s> Collect ively , Ġthe Ġsuccess Ġof Ġlarge Ġpret rained Ġneural Ġmodels , Ġboth Ġenc oder - only ĠB ERT - like Ġarchitectures Ġas Ġwell Ġas Ġenc oder âĢĵ dec oder Ġarchitectures Ġlike ĠT 5 , Ġraise Ġinteresting Ġquestions Ġfor Ġthe Ġpursuit Ġof Ġcommons ense Ġreasoning . ĠResearchers Ġhave Ġdiscovered Ġthat Ġprevious Ġmodels Ġperform Ġwell Ġon Ġbenchmark Ġdatasets Ġbecause Ġthey Ġpick Ġup Ġon Ġincidental Ġbiases Ġin Ġthe Ġdataset Ġthat Ġhave Ġnothing Ġto Ġdo Ġwith Ġthe Ġtask ; Ġin Ġcontrast , Ġthe ĠW ino Grand e Ġdataset Ġhas Ġdevoted Ġconsiderable Ġeffort Ġto Ġreducing Ġsuch Ġbiases , Ġwhich Ġmay Ġallow Ġmodels Ġto Ġ( in advert ently ) ĠâĢ ľ cheat âĢ Ŀ Ġ( for Ġexample , Ġusing Ġsimple Ġstatistical Ġassociations ). ĠWhile Ġit Ġis Ġcertainly Ġtrue Ġthat Ġdatasets Ġover - est imate Ġthe Ġcommons ense Ġreasoning Ġcapabilities Ġof Ġmodern Ġmodels ĠB IB REF 1 , Ġthere Ġare Ġalternative Ġand Ġcomplementary Ġexplanations Ġas Ġwell : </s> It Ġhas Ġbeen Ġa Ġfundamental Ġassumption Ġof Ġthe Ġresearch Ġcommunity Ġthat Ġcommons ense Ġreasoning Ġis Ġdifficult Ġbecause Ġit Ġcomprises Ġtacit Ġrather Ġthan Ġexplicit Ġknowledge ĠB IB REF 8 . ĠThat Ġis , Ġcommons ense Ġknowledge âĢĶ like Ġwater Ġis Ġwet Ġand Ġthat Ġa Ġtub a Ġis Ġusually Ġtoo Ġbig Ġto Ġfit Ġin Ġa Ġbackpack âĢĶ is Ġnot Ġwritten Ġdown Ġanywhere Ġ( un like , Ġsay , Ġfactual Ġknowledge , Ġwhich Ġcan Ġbe Ġmodeled Ġin Ġa Ġknowledge Ġgraph ). ĠAs Ġa Ġresult âĢĶ the Ġreasoning Ġgoes âĢĶ data - driven Ġtechniques Ġ( even Ġneural Ġmodels ) Ġwill Ġbe Ġof Ġlimited Ġuse Ġdue Ġto Ġthe Ġp auc ity Ġof Ġrelevant Ġcorpor a . </s> Yet , Ġprevious Ġenc oder - only Ġarchitectures Ġlike ĠRo BER Ta Ġthat Ġexploit Ġa Ġlanguage Ġmodeling Ġobjective Ġ( that Ġis , Ġrelying Ġonly Ġon Ġexplicit Ġtextual Ġknowledge ) Ġcan Ġclearly Ġmake Ġhead way Ġin Ġa Ġcommons ense Ġreasoning Ġtask , Ġand Ġwe Ġcan Ġfurther Ġimprove Ġupon Ġthese Ġapproaches Ġwith Ġa Ġsequence - to - sequence Ġmodel . ĠThis Ġleaves Ġus Ġwith Ġtwo Ġpossible Ġexplanations : Ġdespite Ġcareful Ġcontrols , Ġthe ĠW ino Grand e Ġchallenge Ġstill Ġcontains Ġincidental Ġbiases Ġthat Ġthese Ġmore Ġsophisticated Ġmodels Ġcan Ġexploit , Ġor Ġthat Ġwe Ġare Ġgenuinely Ġmaking Ġat Ġleast Ġsome Ġprogress Ġin Ġcommons ense Ġreasoning . ĠThe Ġlatter , Ġin Ġparticular , Ġchallenges Ġthe Ġnotion Ġthat Ġcommons ense Ġknowledge Ġis Ġ( mostly ) Ġtacit . ĠPerhaps Ġit Ġis Ġthe Ġcase Ġthat Ġin Ġa Ġhum ong ous Ġcorpus Ġof Ġnatural Ġlanguage Ġtext , Ġsomeone Ġreally Ġhas Ġwritten Ġabout Ġtrying Ġto Ġstuff Ġa Ġtub a Ġin Ġa Ġbackpack ? </s> Acknowled gments </s> This Ġresearch Ġwas Ġsupported Ġin Ġpart Ġby Ġthe ĠCanada ĠFirst ĠResearch ĠExcellence ĠFund Ġand Ġthe ĠNatural ĠSciences Ġand ĠEngineering ĠResearch ĠCouncil Ġ( NS ERC ) Ġof ĠCanada . ĠWe Ġwould Ġlike Ġto Ġthank ĠGoogle ĠCol ab Ġfor Ġproviding Ġsupport Ġin Ġterms Ġof Ġcomputational Ġresources .
Introduction Other than encoder-only pretrained transformer architectures BIBREF2, BIBREF3, BIBREF4, encoder–decoder style pretrained transformers BIBREF0, BIBREF5 have been proven to be effective in text generation tasks as well as comprehension tasks. This paper describes our submission to the commonsense reasoning task leaderboard of the AI2 WinoGrande Challenge BIBREF1, which uses the text-to-text transfer transformer (T5); our approach currently represents the state of the art. In T5 BIBREF0, NLP tasks are formulated as text-to-text problems, where the inputs are cast into natural language templates that contain the task descriptors. Concretely, Raffel et al. provide the following example for MNLI BIBREF6, where the goal is to predict whether a premise implies (“entailment”) or contradicts (“contradiction”) a hypothesis, or neither (“neutral”). Thus, a training example becomes: “mnli premise: I hate pigeons. hypothesis: My feelings towards pigeons are filled with animosity.” with “entailment” as the corresponding ground truth target output. In other words, a token representing each class is directly used as the prediction target. Approach The natural language template approach enables various options to formulate the WinoGrande commonsense reasoning task as a text-to-text problem with T5. Here we adopt a formulation similar to the MNLI template. Consider a concrete example: He never comes to my home, but I always go to his house because the _ is smaller. Option1: home; Option2: house In this case, the correct replacement for _ is Option1. We decompose the above problem into two source–target training examples, where _ is replaced with each option and annotated with the correct answer as the target token, as shown in Table TABREF2. In addition, we reformulate each example into a commonsense reasoning “template” with two statements: hypothesis (from _ to the end of the original problem statement) and premise (the remaining part of the original problem statement). Note that the bold and colored fonts are for clarity only; those tokens are not marked in any way in the model input. At inference (test) time, we also decompose the problem into two inputs, where each input is formulated in exactly the same manner as in Table TABREF2, with either one of the answer options. We then feed each into T5 to predict a target token. In this scenario, there are four possible outcomes: one produces “entailment” and the other “contradiction”, one produces “entailment” or “contradiction” and the other some other token, both produce some other tokens, and both produce the same token, either “entailment” or “contradiction”. Ideally, T5 would produce contrastive tokens for each input pair, as in case (1), which allows us to unambiguously select the final answer. However, the model might produce the same tokens for each input, or even tokens not in the predefined set, as in cases (2) to (4). To deal with these cases, we apply a softmax over the logits of the pair of predefined target tokens, similar to Nogueira et al. BIBREF7. From this, we can compute the probabilities of the predefined target tokens (in the case of Table TABREF2, “entailment” and “contradiction”). Then, we compare the probabilities across both input instances, and in cases (2) to (4), we select the instance that has a higher probability as the correct answer. This general problem setup allows us to choose the target tokens, which may have an impact on the prediction accuracy BIBREF7. In addition to selecting “entailment” vs. “contradiction” as the target, we also tried the contrastive pair “true” vs. ”false”. In our experiment, we fine-tune T5-3B on Google Colab's TPU v2 with a batch size of 16, a learning rate of $2 \cdot 10^{-4}$, and save model checkpoints every 5000 steps. It takes 130k steps to converge for the XL data size (see below). At inference time, we use greedy decoding and select for evaluation the model checkpoint that achieves the highest score on the development set. We did not experiment with T5-11B due to limited computational resources. Results Experimental results on the WinoGrande development set are reported in Table TABREF7 for different training data sizes. Note that we fine-tune the model for each training data size separately. A under the “logit” column indicates that we used the softmax over the target tokens as described above. Without this technique, given the original two-choice question, if T5 outputs the same tokens for the two processed inputs, we simply assign Option1 as the answer. The table also reports “zero-shot” performance, i.e., performing inference on the development set without any model fine tuning. Condition #2 represents our submission to the official leaderboard, which achieves 0.7673 AUC on the held-out test set. From these results, we see that the logit trick clearly improves performance, which is consistent with the observations of Nogueira et al. BIBREF7. In fact, applying this technique in the zero-shot setting yields performance that is clearly better than random. Another interesting finding is that the choice of target token appears to have an impact on performance, which is also consistent with the above work. Since using true/false as the target token (conditions #3 and #4) did not improve performance much over conditions with entailment/contradiction, we did not run all data size conditions given our limited computational resources. Looking at the current WinoGrande leaderboard, it appears that the previous state of the art is based on RoBERTa BIBREF2, which can be characterized as an encoder-only transformer architecture. Since T5-3B is larger than RoBERTa, it cannot be ruled out that model size alone explains the performance gain. However, when coupled with the observations of Nogueira et al. BIBREF7, T5's “generative capability”, i.e., its ability to generate fluent text, honed through pretraining, seems to play an important role. The fact that the choice of target tokens affects prediction accuracy is consistent with this observation. How and why is the subject of ongoing work. Implications Collectively, the success of large pretrained neural models, both encoder-only BERT-like architectures as well as encoder–decoder architectures like T5, raise interesting questions for the pursuit of commonsense reasoning. Researchers have discovered that previous models perform well on benchmark datasets because they pick up on incidental biases in the dataset that have nothing to do with the task; in contrast, the WinoGrande dataset has devoted considerable effort to reducing such biases, which may allow models to (inadvertently) “cheat” (for example, using simple statistical associations). While it is certainly true that datasets over-estimate the commonsense reasoning capabilities of modern models BIBREF1, there are alternative and complementary explanations as well: It has been a fundamental assumption of the research community that commonsense reasoning is difficult because it comprises tacit rather than explicit knowledge BIBREF8. That is, commonsense knowledge—like water is wet and that a tuba is usually too big to fit in a backpack—is not written down anywhere (unlike, say, factual knowledge, which can be modeled in a knowledge graph). As a result—the reasoning goes—data-driven techniques (even neural models) will be of limited use due to the paucity of relevant corpora. Yet, previous encoder-only architectures like RoBERTa that exploit a language modeling objective (that is, relying only on explicit textual knowledge) can clearly make headway in a commonsense reasoning task, and we can further improve upon these approaches with a sequence-to-sequence model. This leaves us with two possible explanations: despite careful controls, the WinoGrande challenge still contains incidental biases that these more sophisticated models can exploit, or that we are genuinely making at least some progress in commonsense reasoning. The latter, in particular, challenges the notion that commonsense knowledge is (mostly) tacit. Perhaps it is the case that in a humongous corpus of natural language text, someone really has written about trying to stuff a tuba in a backpack? Acknowledgments This research was supported in part by the Canada First Research Excellence Fund and the Natural Sciences and Engineering Research Council (NSERC) of Canada. We would like to thank Google Colab for providing support in terms of computational resources.
What is the previous state of the art?
7fb27d8d5a8bb351f97236a1f6dcd8b2613b16f1
What Ġis Ġthe Ġprevious Ġstate Ġof Ġthe Ġart ?
Ro BER Ta
<s> What Ġis Ġthe Ġprevious Ġstate Ġof Ġthe Ġart ? </s> Introduction </s> Other Ġthan Ġenc oder - only Ġpret rained Ġtransformer Ġarchitectures ĠB IB REF 2 , ĠB IB REF 3 , ĠB IB REF 4 , Ġenc oder âĢĵ dec oder Ġstyle Ġpret rained Ġtransform ers ĠB IB REF 0 , ĠB IB REF 5 Ġhave Ġbeen Ġproven Ġto Ġbe Ġeffective Ġin Ġtext Ġgeneration Ġtasks Ġas Ġwell Ġas Ġcomprehension Ġtasks . ĠThis Ġpaper Ġdescribes Ġour Ġsubmission Ġto Ġthe Ġcommons ense Ġreasoning Ġtask Ġleader board Ġof Ġthe ĠAI 2 ĠW ino Grand e ĠChallenge ĠB IB REF 1 , Ġwhich Ġuses Ġthe Ġtext - to - text Ġtransfer Ġtransformer Ġ( T 5 ); Ġour Ġapproach Ġcurrently Ġrepresents Ġthe Ġstate Ġof Ġthe Ġart . </s> In ĠT 5 ĠB IB REF 0 , ĠN LP Ġtasks Ġare Ġformulated Ġas Ġtext - to - text Ġproblems , Ġwhere Ġthe Ġinputs Ġare Ġcast Ġinto Ġnatural Ġlanguage Ġtemplates Ġthat Ġcontain Ġthe Ġtask Ġdescript ors . ĠCon crete ly , ĠR aff el Ġet Ġal . Ġprovide Ġthe Ġfollowing Ġexample Ġfor ĠMN LI ĠB IB REF 6 , Ġwhere Ġthe Ġgoal Ġis Ġto Ġpredict Ġwhether Ġa Ġpremise Ġimplies Ġ( âĢ ľ ent ail ment âĢ Ŀ ) Ġor Ġcontradicts Ġ( âĢ ľ cont rad iction âĢ Ŀ ) Ġa Ġhypothesis , Ġor Ġneither Ġ( âĢ ľ neutral âĢ Ŀ ). ĠThus , Ġa Ġtraining Ġexample Ġbecomes : </s> âĢ ľ mn li Ġpremise : ĠI Ġhate Ġpige ons . Ġhypothesis : ĠMy Ġfeelings Ġtowards Ġpige ons Ġare Ġfilled Ġwith Ġanimosity . âĢ Ŀ </s> with ĠâĢ ľ ent ail ment âĢ Ŀ Ġas Ġthe Ġcorresponding Ġground Ġtruth Ġtarget Ġoutput . ĠIn Ġother Ġwords , Ġa Ġtoken Ġrepresenting Ġeach Ġclass Ġis Ġdirectly Ġused Ġas Ġthe Ġprediction Ġtarget . </s> App roach </s> The Ġnatural Ġlanguage Ġtemplate Ġapproach Ġenables Ġvarious Ġoptions Ġto Ġformulate Ġthe ĠW ino Grand e Ġcommons ense Ġreasoning Ġtask Ġas Ġa Ġtext - to - text Ġproblem Ġwith ĠT 5 . ĠHere Ġwe Ġadopt Ġa Ġformulation Ġsimilar Ġto Ġthe ĠMN LI Ġtemplate . ĠConsider Ġa Ġconcrete Ġexample : </s> He Ġnever Ġcomes Ġto Ġmy Ġhome , Ġbut ĠI Ġalways Ġgo Ġto Ġhis Ġhouse Ġbecause Ġthe Ġ_ Ġis Ġsmaller . </s> Option 1 : Ġhome ; ĠOption 2 : Ġhouse </s> In Ġthis Ġcase , Ġthe Ġcorrect Ġreplacement Ġfor Ġ_ Ġis ĠOption 1 . ĠWe Ġdecom pose Ġthe Ġabove Ġproblem Ġinto Ġtwo Ġsource âĢĵ target Ġtraining Ġexamples , Ġwhere Ġ_ Ġis Ġreplaced Ġwith Ġeach Ġoption Ġand Ġannot ated Ġwith Ġthe Ġcorrect Ġanswer Ġas Ġthe Ġtarget Ġtoken , Ġas Ġshown Ġin ĠTable ĠT AB REF 2 . ĠIn Ġaddition , Ġwe Ġreform ulate Ġeach Ġexample Ġinto Ġa Ġcommons ense Ġreasoning ĠâĢ ľ template âĢ Ŀ Ġwith Ġtwo Ġstatements : Ġhypothesis Ġ( from Ġ_ Ġto Ġthe Ġend Ġof Ġthe Ġoriginal Ġproblem Ġstatement ) Ġand Ġpremise Ġ( the Ġremaining Ġpart Ġof Ġthe Ġoriginal Ġproblem Ġstatement ). ĠNote Ġthat Ġthe Ġbold Ġand Ġcolored Ġfonts Ġare Ġfor Ġclarity Ġonly ; Ġthose Ġtokens Ġare Ġnot Ġmarked Ġin Ġany Ġway Ġin Ġthe Ġmodel Ġinput . </s> At Ġinference Ġ( test ) Ġtime , Ġwe Ġalso Ġdecom pose Ġthe Ġproblem Ġinto Ġtwo Ġinputs , Ġwhere Ġeach Ġinput Ġis Ġformulated Ġin Ġexactly Ġthe Ġsame Ġmanner Ġas Ġin ĠTable ĠT AB REF 2 , Ġwith Ġeither Ġone Ġof Ġthe Ġanswer Ġoptions . ĠWe Ġthen Ġfeed Ġeach Ġinto ĠT 5 Ġto Ġpredict Ġa Ġtarget Ġtoken . ĠIn Ġthis Ġscenario , Ġthere Ġare Ġfour Ġpossible Ġoutcomes : </s> one Ġproduces ĠâĢ ľ ent ail ment âĢ Ŀ Ġand Ġthe Ġother ĠâĢ ľ cont rad iction âĢ Ŀ , </s> one Ġproduces ĠâĢ ľ ent ail ment âĢ Ŀ Ġor ĠâĢ ľ cont rad iction âĢ Ŀ Ġand Ġthe Ġother Ġsome Ġother Ġtoken , </s> both Ġproduce Ġsome Ġother Ġtokens , Ġand </s> both Ġproduce Ġthe Ġsame Ġtoken , Ġeither ĠâĢ ľ ent ail ment âĢ Ŀ Ġor ĠâĢ ľ cont rad iction âĢ Ŀ . </s> Ide ally , ĠT 5 Ġwould Ġproduce Ġcontrast ive Ġtokens Ġfor Ġeach Ġinput Ġpair , Ġas Ġin Ġcase Ġ( 1 ), Ġwhich Ġallows Ġus Ġto Ġunamb ig uously Ġselect Ġthe Ġfinal Ġanswer . ĠHowever , Ġthe Ġmodel Ġmight Ġproduce Ġthe Ġsame Ġtokens Ġfor Ġeach Ġinput , Ġor Ġeven Ġtokens Ġnot Ġin Ġthe Ġpred efined Ġset , Ġas Ġin Ġcases Ġ( 2 ) Ġto Ġ( 4 ). ĠTo Ġdeal Ġwith Ġthese Ġcases , Ġwe Ġapply Ġa Ġsoft max Ġover Ġthe Ġlog its Ġof Ġthe Ġpair Ġof Ġpred efined Ġtarget Ġtokens , Ġsimilar Ġto ĠN ogue ira Ġet Ġal . ĠB IB REF 7 . ĠFrom Ġthis , Ġwe Ġcan Ġcompute Ġthe Ġprobabilities Ġof Ġthe Ġpred efined Ġtarget Ġtokens Ġ( in Ġthe Ġcase Ġof ĠTable ĠT AB REF 2 , ĠâĢ ľ ent ail ment âĢ Ŀ Ġand ĠâĢ ľ cont rad iction âĢ Ŀ ). ĠThen , Ġwe Ġcompare Ġthe Ġprobabilities Ġacross Ġboth Ġinput Ġinstances , Ġand Ġin Ġcases Ġ( 2 ) Ġto Ġ( 4 ), Ġwe Ġselect Ġthe Ġinstance Ġthat Ġhas Ġa Ġhigher Ġprobability Ġas Ġthe Ġcorrect Ġanswer . </s> This Ġgeneral Ġproblem Ġsetup Ġallows Ġus Ġto Ġchoose Ġthe Ġtarget Ġtokens , Ġwhich Ġmay Ġhave Ġan Ġimpact Ġon Ġthe Ġprediction Ġaccuracy ĠB IB REF 7 . ĠIn Ġaddition Ġto Ġselecting ĠâĢ ľ ent ail ment âĢ Ŀ Ġvs . ĠâĢ ľ cont rad iction âĢ Ŀ Ġas Ġthe Ġtarget , Ġwe Ġalso Ġtried Ġthe Ġcontrast ive Ġpair ĠâĢ ľ true âĢ Ŀ Ġvs . ĠâĢ Ŀ false âĢ Ŀ . </s> In Ġour Ġexperiment , Ġwe Ġfine - t une ĠT 5 - 3 B Ġon ĠGoogle ĠCol ab 's ĠT PU Ġv 2 Ġwith Ġa Ġbatch Ġsize Ġof Ġ16 , Ġa Ġlearning Ġrate Ġof Ġ$ 2 Ġ\ cd ot Ġ10 ^{ - 4 } $, Ġand Ġsave Ġmodel Ġcheckpoints Ġevery Ġ5000 Ġsteps . ĠIt Ġtakes Ġ130 k Ġsteps Ġto Ġconverge Ġfor Ġthe ĠXL Ġdata Ġsize Ġ( see Ġbelow ). ĠAt Ġinference Ġtime , Ġwe Ġuse Ġgreedy Ġdecoding Ġand Ġselect Ġfor Ġevaluation Ġthe Ġmodel Ġcheckpoint Ġthat Ġachieves Ġthe Ġhighest Ġscore Ġon Ġthe Ġdevelopment Ġset . ĠWe Ġdid Ġnot Ġexperiment Ġwith ĠT 5 - 11 B Ġdue Ġto Ġlimited Ġcomputational Ġresources . </s> Results </s> Exper imental Ġresults Ġon Ġthe ĠW ino Grand e Ġdevelopment Ġset Ġare Ġreported Ġin ĠTable ĠT AB REF 7 Ġfor Ġdifferent Ġtraining Ġdata Ġsizes . ĠNote Ġthat Ġwe Ġfine - t une Ġthe Ġmodel Ġfor Ġeach Ġtraining Ġdata Ġsize Ġseparately . ĠA Ġunder Ġthe ĠâĢ ľ log it âĢ Ŀ Ġcolumn Ġindicates Ġthat Ġwe Ġused Ġthe Ġsoft max Ġover Ġthe Ġtarget Ġtokens Ġas Ġdescribed Ġabove . ĠWithout Ġthis Ġtechnique , Ġgiven Ġthe Ġoriginal Ġtwo - choice Ġquestion , Ġif ĠT 5 Ġoutputs Ġthe Ġsame Ġtokens Ġfor Ġthe Ġtwo Ġprocessed Ġinputs , Ġwe Ġsimply Ġassign ĠOption 1 Ġas Ġthe Ġanswer . ĠThe Ġtable Ġalso Ġreports ĠâĢ ľ zero - shot âĢ Ŀ Ġperformance , Ġi . e ., Ġperforming Ġinference Ġon Ġthe Ġdevelopment Ġset Ġwithout Ġany Ġmodel Ġfine Ġtuning . ĠCondition Ġ# 2 Ġrepresents Ġour Ġsubmission Ġto Ġthe Ġofficial Ġleader board , Ġwhich Ġachieves Ġ0 . 767 3 ĠA UC Ġon Ġthe Ġheld - out Ġtest Ġset . </s> From Ġthese Ġresults , Ġwe Ġsee Ġthat Ġthe Ġlog it Ġtrick Ġclearly Ġimproves Ġperformance , Ġwhich Ġis Ġconsistent Ġwith Ġthe Ġobservations Ġof ĠN ogue ira Ġet Ġal . ĠB IB REF 7 . ĠIn Ġfact , Ġapplying Ġthis Ġtechnique Ġin Ġthe Ġzero - shot Ġsetting Ġyields Ġperformance Ġthat Ġis Ġclearly Ġbetter Ġthan Ġrandom . ĠAnother Ġinteresting Ġfinding Ġis Ġthat Ġthe Ġchoice Ġof Ġtarget Ġtoken Ġappears Ġto Ġhave Ġan Ġimpact Ġon Ġperformance , Ġwhich Ġis Ġalso Ġconsistent Ġwith Ġthe Ġabove Ġwork . ĠSince Ġusing Ġtrue / false Ġas Ġthe Ġtarget Ġtoken Ġ( cond itions Ġ# 3 Ġand Ġ# 4 ) Ġdid Ġnot Ġimprove Ġperformance Ġmuch Ġover Ġconditions Ġwith Ġentail ment / cont rad iction , Ġwe Ġdid Ġnot Ġrun Ġall Ġdata Ġsize Ġconditions Ġgiven Ġour Ġlimited Ġcomputational Ġresources . </s> Looking Ġat Ġthe Ġcurrent ĠW ino Grand e Ġleader board , Ġit Ġappears Ġthat Ġthe Ġprevious Ġstate Ġof Ġthe Ġart Ġis Ġbased Ġon ĠRo BER Ta ĠB IB REF 2 , Ġwhich Ġcan Ġbe Ġcharacterized Ġas Ġan Ġenc oder - only Ġtransformer Ġarchitecture . ĠSince ĠT 5 - 3 B Ġis Ġlarger Ġthan ĠRo BER Ta , Ġit Ġcannot Ġbe Ġruled Ġout Ġthat Ġmodel Ġsize Ġalone Ġexplains Ġthe Ġperformance Ġgain . ĠHowever , Ġwhen Ġcoupled Ġwith Ġthe Ġobservations Ġof ĠN ogue ira Ġet Ġal . ĠB IB REF 7 , ĠT 5 's ĠâĢ ľ gener ative Ġcapability âĢ Ŀ , Ġi . e ., Ġits Ġability Ġto Ġgenerate Ġfluent Ġtext , Ġhon ed Ġthrough Ġpret raining , Ġseems Ġto Ġplay Ġan Ġimportant Ġrole . ĠThe Ġfact Ġthat Ġthe Ġchoice Ġof Ġtarget Ġtokens Ġaffects Ġprediction Ġaccuracy Ġis Ġconsistent Ġwith Ġthis Ġobservation . ĠHow Ġand Ġwhy Ġis Ġthe Ġsubject Ġof Ġongoing Ġwork . </s> Impl ications </s> Collect ively , Ġthe Ġsuccess Ġof Ġlarge Ġpret rained Ġneural Ġmodels , Ġboth Ġenc oder - only ĠB ERT - like Ġarchitectures Ġas Ġwell Ġas Ġenc oder âĢĵ dec oder Ġarchitectures Ġlike ĠT 5 , Ġraise Ġinteresting Ġquestions Ġfor Ġthe Ġpursuit Ġof Ġcommons ense Ġreasoning . ĠResearchers Ġhave Ġdiscovered Ġthat Ġprevious Ġmodels Ġperform Ġwell Ġon Ġbenchmark Ġdatasets Ġbecause Ġthey Ġpick Ġup Ġon Ġincidental Ġbiases Ġin Ġthe Ġdataset Ġthat Ġhave Ġnothing Ġto Ġdo Ġwith Ġthe Ġtask ; Ġin Ġcontrast , Ġthe ĠW ino Grand e Ġdataset Ġhas Ġdevoted Ġconsiderable Ġeffort Ġto Ġreducing Ġsuch Ġbiases , Ġwhich Ġmay Ġallow Ġmodels Ġto Ġ( in advert ently ) ĠâĢ ľ cheat âĢ Ŀ Ġ( for Ġexample , Ġusing Ġsimple Ġstatistical Ġassociations ). ĠWhile Ġit Ġis Ġcertainly Ġtrue Ġthat Ġdatasets Ġover - est imate Ġthe Ġcommons ense Ġreasoning Ġcapabilities Ġof Ġmodern Ġmodels ĠB IB REF 1 , Ġthere Ġare Ġalternative Ġand Ġcomplementary Ġexplanations Ġas Ġwell : </s> It Ġhas Ġbeen Ġa Ġfundamental Ġassumption Ġof Ġthe Ġresearch Ġcommunity Ġthat Ġcommons ense Ġreasoning Ġis Ġdifficult Ġbecause Ġit Ġcomprises Ġtacit Ġrather Ġthan Ġexplicit Ġknowledge ĠB IB REF 8 . ĠThat Ġis , Ġcommons ense Ġknowledge âĢĶ like Ġwater Ġis Ġwet Ġand Ġthat Ġa Ġtub a Ġis Ġusually Ġtoo Ġbig Ġto Ġfit Ġin Ġa Ġbackpack âĢĶ is Ġnot Ġwritten Ġdown Ġanywhere Ġ( un like , Ġsay , Ġfactual Ġknowledge , Ġwhich Ġcan Ġbe Ġmodeled Ġin Ġa Ġknowledge Ġgraph ). ĠAs Ġa Ġresult âĢĶ the Ġreasoning Ġgoes âĢĶ data - driven Ġtechniques Ġ( even Ġneural Ġmodels ) Ġwill Ġbe Ġof Ġlimited Ġuse Ġdue Ġto Ġthe Ġp auc ity Ġof Ġrelevant Ġcorpor a . </s> Yet , Ġprevious Ġenc oder - only Ġarchitectures Ġlike ĠRo BER Ta Ġthat Ġexploit Ġa Ġlanguage Ġmodeling Ġobjective Ġ( that Ġis , Ġrelying Ġonly Ġon Ġexplicit Ġtextual Ġknowledge ) Ġcan Ġclearly Ġmake Ġhead way Ġin Ġa Ġcommons ense Ġreasoning Ġtask , Ġand Ġwe Ġcan Ġfurther Ġimprove Ġupon Ġthese Ġapproaches Ġwith Ġa Ġsequence - to - sequence Ġmodel . ĠThis Ġleaves Ġus Ġwith Ġtwo Ġpossible Ġexplanations : Ġdespite Ġcareful Ġcontrols , Ġthe ĠW ino Grand e Ġchallenge Ġstill Ġcontains Ġincidental Ġbiases Ġthat Ġthese Ġmore Ġsophisticated Ġmodels Ġcan Ġexploit , Ġor Ġthat Ġwe Ġare Ġgenuinely Ġmaking Ġat Ġleast Ġsome Ġprogress Ġin Ġcommons ense Ġreasoning . ĠThe Ġlatter , Ġin Ġparticular , Ġchallenges Ġthe Ġnotion Ġthat Ġcommons ense Ġknowledge Ġis Ġ( mostly ) Ġtacit . ĠPerhaps Ġit Ġis Ġthe Ġcase Ġthat Ġin Ġa Ġhum ong ous Ġcorpus Ġof Ġnatural Ġlanguage Ġtext , Ġsomeone Ġreally Ġhas Ġwritten Ġabout Ġtrying Ġto Ġstuff Ġa Ġtub a Ġin Ġa Ġbackpack ? </s> Acknowled gments </s> This Ġresearch Ġwas Ġsupported Ġin Ġpart Ġby Ġthe ĠCanada ĠFirst ĠResearch ĠExcellence ĠFund Ġand Ġthe ĠNatural ĠSciences Ġand ĠEngineering ĠResearch ĠCouncil Ġ( NS ERC ) Ġof ĠCanada . ĠWe Ġwould Ġlike Ġto Ġthank ĠGoogle ĠCol ab Ġfor Ġproviding Ġsupport Ġin Ġterms Ġof Ġcomputational Ġresources .