Yes
BOOLEAN
Yes
BOOLEAN
Tables 1 , 2 and 3 show the results. All cases exhibit high scores—in the vast majority of the cases substantially higher than reported in previous work. As discussed above, the results are not directly comparable to previous work: the BERT models are trained on different (and larger) data, are allowed to access the suffix of the sentence in addition to its prefix, and are evaluated on somewhat different data due to discarding OOV items. Still, taken together, the high performance numbers indicate that the purely attention-based BERT models are likely capable of capturing the same kind of syntactic regularities that LSTM-based models are capable of capturing, at least as well as the LSTM models and probably better. Recent work examines the extent to which RNN-based models capture syntax-sensitive phenomena that are traditionally taken as evidence for the existence in hierarchical structure. In particular, in BIBREF1 we assess the ability of LSTMs to learn subject-verb agreement patterns in English, and evaluate on naturally occurring wikipedia sentences. BIBREF2 also consider subject-verb agreement, but in a “colorless green ideas” setting in which content words in naturally occurring sentences are replaced with random words with the same part-of-speech and inflection, thus ensuring a focus on syntax rather than on selectional-preferences based cues. BIBREF3 consider a wider range of syntactic phenomena (subject-verb agreement, reflexive anaphora, negative polarity items) using manually constructed stimuli, allowing for greater coverage and control than in the naturally occurring setting.
Recent work examines the extent to which RNN-based models capture syntax-sensitive phenomena that are traditionally taken as evidence for the existence in hierarchical structure. In particular, in BIBREF1 we assess the ability of LSTMs to learn subject-verb agreement patterns in English, and evaluate on naturally occurring wikipedia sentences. BIBREF2 also consider subject-verb agreement, but in a “colorless green ideas” setting in which content words in naturally occurring sentences are replaced with random words with the same part-of-speech and inflection, thus ensuring a focus on syntax rather than on selectional-preferences based cues. BIBREF3 consider a wider range of syntactic phenomena (subject-verb agreement, reflexive anaphora, negative polarity items) using manually constructed stimuli, allowing for greater coverage and control than in the naturally occurring setting. I use the stimuli provided by BIBREF1 , BIBREF2 , BIBREF3 , but change the experimental protocol to adapt it to the bidirectional nature of the BERT model. This requires discarding some of the stimuli, as described below. Thus, the numbers are not strictly comparable to those reported in previous work.
1901.05287
Introduction </s> The Ġrecently Ġintroduced ĠB ERT Ġmodel ĠB IB REF 0 Ġexhibits Ġstrong Ġperformance Ġon Ġseveral Ġlanguage Ġunderstanding Ġbenchmarks . ĠTo Ġwhat Ġextent Ġdoes Ġit Ġcapture Ġsyntax - sensitive Ġstructures ? </s> Recent Ġwork Ġexamines Ġthe Ġextent Ġto Ġwhich ĠR NN - based Ġmodels Ġcapture Ġsyntax - sensitive Ġphenomena Ġthat Ġare Ġtraditionally Ġtaken Ġas Ġevidence Ġfor Ġthe Ġexistence Ġin Ġhierarchical Ġstructure . ĠIn Ġparticular , Ġin ĠB IB REF 1 Ġwe Ġassess Ġthe Ġability Ġof ĠL ST Ms Ġto Ġlearn Ġsubject - verb Ġagreement Ġpatterns Ġin ĠEnglish , Ġand Ġevaluate Ġon Ġnaturally Ġoccurring Ġwik ipedia Ġsentences . ĠB IB REF 2 Ġalso Ġconsider Ġsubject - verb Ġagreement , Ġbut Ġin Ġa ĠâĢ ľ color less Ġgreen Ġideas âĢ Ŀ Ġsetting Ġin Ġwhich Ġcontent Ġwords Ġin Ġnaturally Ġoccurring Ġsentences Ġare Ġreplaced Ġwith Ġrandom Ġwords Ġwith Ġthe Ġsame Ġpart - of - speech Ġand Ġinf lection , Ġthus Ġensuring Ġa Ġfocus Ġon Ġsyntax Ġrather Ġthan Ġon Ġselection al - pre f erences Ġbased Ġcues . ĠB IB REF 3 Ġconsider Ġa Ġwider Ġrange Ġof Ġsynt actic Ġphenomena Ġ( subject - verb Ġagreement , Ġreflex ive Ġan aph ora , Ġnegative Ġpol arity Ġitems ) Ġusing Ġmanually Ġconstructed Ġstimuli , Ġallowing Ġfor Ġgreater Ġcoverage Ġand Ġcontrol Ġthan Ġin Ġthe Ġnaturally Ġoccurring Ġsetting . </s> The ĠB ERT Ġmodel Ġis Ġbased Ġon Ġthe ĠâĢ ľ Trans former âĢ Ŀ Ġarchitecture ĠB IB REF 4 Ġ, Ġwhich âĢĶ in Ġcontrast Ġto ĠR NN s âĢĶ rel ies Ġpurely Ġon Ġattention Ġmechanisms , Ġand Ġdoes Ġnot Ġhave Ġan Ġexplicit Ġnotion Ġof Ġword Ġorder Ġbeyond Ġmarking Ġeach Ġword Ġwith Ġits Ġabsolute - position Ġembed ding . ĠThis Ġreliance Ġon Ġattention Ġmay Ġlead Ġone Ġto Ġexpect Ġdecreased Ġperformance Ġon Ġsyntax - sensitive Ġtasks Ġcompared Ġto ĠR NN Ġ( L ST M ) Ġmodels Ġthat Ġdo Ġmodel Ġword Ġorder Ġdirectly , Ġand Ġexplicitly Ġtrack Ġstates Ġacross Ġthe Ġsentence . ĠIndeed , ĠB IB REF 5 Ġfinds Ġthat Ġtransformer - based Ġmodels Ġperform Ġworse Ġthan ĠL ST M Ġmodels Ġon Ġthe ĠB IB REF 1 Ġagreement Ġprediction Ġdataset . ĠIn Ġcontrast , ĠB IB REF 6 Ġfind Ġthat Ġself - att ention Ġperforms Ġon Ġpar Ġwith ĠL ST M Ġfor Ġsyntax Ġsensitive Ġdependencies Ġin Ġthe Ġcontext Ġof Ġmachine - translation , Ġand Ġperformance Ġon Ġsynt actic Ġtasks Ġis Ġcorrelated Ġwith Ġthe Ġnumber Ġof Ġattention Ġheads Ġin Ġmulti - head Ġattention . </s> I Ġadapt Ġthe Ġevaluation Ġprotocol Ġand Ġstimuli Ġof ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġand ĠB IB REF 3 Ġto Ġthe Ġbid irection al Ġsetting Ġrequired Ġby ĠB ERT , Ġand Ġevaluate Ġthe Ġpre - trained ĠB ERT Ġmodels Ġ( both Ġthe ĠLarge Ġand Ġthe ĠBase Ġmodels ). ĠSurprisingly Ġ( at Ġleast Ġto Ġme ), Ġthe Ġout - of - the - box Ġmodels Ġ( without Ġany Ġtask - specific Ġfine - tun ing ) Ġperform Ġvery Ġwell Ġon Ġall Ġthe Ġsynt actic Ġtasks . </s> Method ology </s> I Ġuse Ġthe Ġstimuli Ġprovided Ġby ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, Ġbut Ġchange Ġthe Ġexperimental Ġprotocol Ġto Ġadapt Ġit Ġto Ġthe Ġbid irection al Ġnature Ġof Ġthe ĠB ERT Ġmodel . ĠThis Ġrequires Ġdisc arding Ġsome Ġof Ġthe Ġstimuli , Ġas Ġdescribed Ġbelow . ĠThus , Ġthe Ġnumbers Ġare Ġnot Ġstrictly Ġcomparable Ġto Ġthose Ġreported Ġin Ġprevious Ġwork . </s> Previous Ġsetups </s> All Ġthree Ġprevious Ġwork Ġuse Ġun i - direction al Ġlanguage - model - like Ġmodels . </s> BI B REF 1 Ġstart Ġwith Ġexisting Ġsentences Ġfrom Ġwik ipedia Ġthat Ġcontain Ġa Ġpresent - t ense Ġverb . ĠThey Ġfeed Ġeach Ġsentence Ġword Ġby Ġword Ġinto Ġan ĠL ST M , Ġstop Ġright Ġbefore Ġthe Ġfocus Ġverb , Ġand Ġask Ġthe Ġmodel Ġto Ġpredict Ġa Ġbinary Ġplural / sing ular Ġdecision Ġ( super vised Ġsetup ) Ġor Ġcompare Ġthe Ġprobability Ġassigned Ġby Ġa Ġpre - trained Ġlanguage Ġmodel Ġ( LM ) Ġto Ġthe Ġplural Ġvs Ġsingular Ġforms Ġof Ġthe Ġverb Ġ( LM Ġsetup ). ĠThe Ġevaluation Ġis Ġthen Ġperformed Ġon Ġsentences Ġwith ĠâĢ ľ ag reement Ġattract ors âĢ Ŀ Ġin Ġwhich Ġat Ġthere Ġis Ġat Ġleast Ġone Ġnoun Ġbetween Ġthe Ġverb Ġand Ġits Ġsubject , Ġand Ġall Ġof Ġthe Ġnoun s Ġbetween Ġthe Ġverb Ġand Ġsubject Ġare Ġof Ġthe Ġopposite Ġnumber Ġfrom Ġthe Ġsubject . </s> BI B REF 2 Ġalso Ġstart Ġwith Ġexisting Ġsentences . ĠHowever , Ġin Ġorder Ġto Ġcontrol Ġfor Ġthe Ġposs ib ill ity Ġof Ġthe Ġmodel Ġlearning Ġto Ġrely Ġon ĠâĢ ľ sem antic âĢ Ŀ Ġselection al - pre f erences Ġcues Ġrather Ġthan Ġsynt actic Ġones , Ġthey Ġreplace Ġeach Ġcontent Ġword Ġwith Ġrandom Ġwords Ġfrom Ġthe Ġsame Ġpart - of - speech Ġand Ġinf lection . ĠThis Ġresults Ġin ĠâĢ ľ col ore less Ġgreen Ġideas âĢ Ŀ Ġnon ce Ġsentences . ĠThe Ġevaluation Ġis Ġthen Ġperformed Ġsimilarly Ġto Ġthe ĠLM Ġsetup Ġof ĠB IB REF 1 Ġ: Ġthe Ġsentence Ġis Ġfed Ġinto Ġa Ġpre - tra i end ĠL ST M ĠLM Ġup Ġto Ġthe Ġfocus Ġverb , Ġand Ġthe Ġmodel Ġis Ġconsidered Ġcorrect Ġif Ġthe Ġprobability Ġassigned Ġto Ġthe Ġcorrect Ġinf lection Ġof Ġthe Ġoriginal Ġverb Ġform Ġgiven Ġthe Ġprefix Ġis Ġlarger Ġthan Ġthat Ġassigned Ġto Ġthe Ġincorrect Ġinf lection . </s> BI B REF 3 Ġfocus Ġon Ġmanually Ġconstructed Ġand Ġcontrolled Ġstimuli , Ġthat Ġalso Ġemphasizes Ġlinguistic Ġstructure Ġover Ġselection al Ġpreferences . ĠThey Ġconstruct Ġminimal Ġpairs Ġof Ġgram matical Ġand Ġun gram matical Ġsentences , Ġfeed Ġeach Ġone Ġin Ġits Ġentirety Ġinto Ġa Ġpre - trained ĠL ST M - LM , Ġand Ġcompare Ġthe Ġperplex ity Ġassigned Ġby Ġthe Ġmodel Ġto Ġthe Ġgram matical Ġand Ġun gram matical Ġsentences . ĠThe Ġmodel Ġis ĠâĢ ľ correct âĢ Ŀ Ġif Ġit Ġassigns Ġthe Ġgram matical Ġsentence Ġa Ġhigher Ġprobability Ġthan Ġto Ġthe Ġun gram matical Ġone . ĠSince Ġthe Ġminimal Ġpairs Ġfor Ġmost Ġphenomena Ġdiffer Ġonly Ġin Ġa Ġsingle Ġword Ġ( the Ġfocus Ġverb ), Ġthis Ġscoring Ġis Ġvery Ġsimilar Ġto Ġthe Ġone Ġused Ġin Ġthe Ġtwo Ġprevious Ġworks . ĠHowever , Ġit Ġdoes Ġconsider Ġthe Ġcontinuation Ġof Ġthe Ġsentence Ġafter Ġthe Ġfocus Ġverb , Ġand Ġalso Ġallows Ġfor Ġassessing Ġphenomena Ġthat Ġrequire Ġchange Ġinto Ġtwo Ġor Ġmore Ġwords Ġ( like Ġnegative Ġpol arity Ġitems ). </s> Adapt ation Ġto Ġthe ĠB ERT Ġmodel </s> In Ġcontrast Ġto Ġthese Ġworks , Ġthe ĠB ERT Ġmodel Ġis Ġbi - direction al : Ġit Ġis Ġtrained Ġto Ġpredict Ġthe Ġidentity Ġof Ġmasked Ġwords Ġbased Ġon Ġboth Ġthe Ġprefix Ġand Ġsuffix Ġsurrounding Ġthese Ġwords . ĠI Ġadapt Ġthe Ġun i - direction al Ġsetup Ġby Ġfeeding Ġinto ĠB ERT Ġthe Ġcomplete Ġsentence , Ġwhile Ġmask ing Ġout Ġthe Ġsingle Ġfocus Ġverb . ĠI Ġthen Ġask ĠB ERT Ġfor Ġits Ġword Ġpredictions Ġfor Ġthe Ġmasked Ġposition , Ġand Ġcompare Ġthe Ġscore Ġassigned Ġto Ġthe Ġoriginal Ġcorrect Ġverb Ġto Ġthe Ġscore Ġassigned Ġto Ġthe Ġincorrect Ġone . </s> For Ġexample , Ġfor Ġthe Ġsentence : </s> a Ġ2002 Ġsystemic Ġreview Ġof Ġherbal Ġproducts Ġfound Ġthat Ġseveral Ġherbs Ġ, Ġincluding Ġpepper mint Ġand Ġcar away Ġ, Ġhave Ġanti - d ys pe ptic Ġeffects Ġfor Ġnon - ul cer Ġdys pe ps ia Ġwith ĠâĢ ľ Ġencouraging Ġsafety Ġprofiles ĠâĢ Ŀ Ġ. Ġ( from ĠB IB REF 1 Ġ) </s> I Ġfeed Ġinto ĠB ERT : </s> [ CL S ] Ġa Ġ2002 Ġsystemic Ġreview Ġof Ġherbal Ġproducts Ġfound Ġthat Ġseveral Ġherbs Ġ, Ġincluding Ġpepper mint Ġand Ġcar away Ġ, Ġ[ MAS K ] Ġanti - d ys pe ptic Ġeffects Ġfor Ġnon - ul cer Ġdys pe ps ia Ġwith ĠâĢ ľ Ġencouraging Ġsafety Ġprofiles ĠâĢ Ŀ Ġ. Ġand Ġlook Ġfor Ġthe Ġscore Ġassigned Ġto Ġthe Ġwords Ġhave Ġand Ġhas Ġat Ġthe Ġmasked Ġposition . </s> Similarly , Ġfor Ġthe Ġpair </s> the Ġgame Ġthat Ġthe Ġguard Ġhates Ġis Ġbad Ġ. </s> the Ġgame Ġthat Ġthe Ġguard Ġhates Ġare Ġbad Ġ. </s> ( from ĠB IB REF 3 Ġ), ĠI Ġfeed Ġinto ĠB ERT : </s> [ CL S ] Ġthe Ġgame Ġthat Ġthe Ġguard Ġhates Ġ[ MAS K ] Ġbad Ġ. </s> and Ġcompare Ġthe Ġscores Ġpredicted Ġfor Ġis Ġand Ġare . </s> This Ġdiffers Ġfrom ĠB IB REF 1 Ġand ĠB IB REF 2 Ġby Ġconsidering Ġthe Ġentire Ġsentence Ġ( excluding Ġthe Ġverb ) Ġand Ġnot Ġjust Ġits Ġprefix Ġleading Ġto Ġthe Ġverb , Ġand Ġdiffers Ġfrom ĠB IB REF 3 Ġby Ġconditioning Ġthe Ġfocus Ġverb Ġon Ġbid irection al Ġcontext . </s> I Ġuse Ġthe ĠPy Tor ch Ġimplementation Ġof ĠB ERT , Ġwith Ġthe Ġpre - trained Ġmodels Ġsupplied Ġby ĠGoogle . ĠI Ġexperiment Ġwith Ġthe Ġb ert - large - unc ased Ġand Ġb ert - base - unc ased Ġmodels . </s> The Ġbi - direction al Ġsetup Ġpre cludes Ġusing Ġusing Ġthe ĠN PI Ġstimuli Ġof ĠB IB REF 3 Ġ, Ġin Ġwhich Ġthe Ġminimal Ġpair Ġdiffers Ġin Ġtwo Ġwords Ġposition , Ġwhich ĠI Ġdiscard Ġfrom Ġthe Ġevaluation . ĠI Ġalso Ġdiscard Ġthe Ġagreement Ġcases Ġinvolving Ġthe Ġverbs Ġis Ġor Ġare Ġin ĠB IB REF 1 Ġand Ġin ĠB IB REF 2 Ġ, Ġbecause Ġsome Ġof Ġthem Ġare Ġcop ular Ġconstruction , Ġin Ġwhich Ġstrong Ġagreement Ġhints Ġcan Ġbe Ġfound Ġalso Ġon Ġthe Ġobject Ġfollowing Ġthe Ġverb . ĠThis Ġis Ġnot Ġan Ġissue Ġin Ġthe Ġmanually Ġconstructed ĠB IB REF 3 Ġstimuli Ġdue Ġto Ġthe Ġpatterns Ġthey Ġchose . </s> Finally , ĠI Ġdiscard Ġstimuli Ġin Ġwhich Ġthe Ġfocus Ġverb Ġor Ġits Ġplural / sing ular Ġinf lection Ġdoes Ġnot Ġappear Ġas Ġa Ġsingle Ġword Ġin Ġthe ĠB ERT Ġword - piece - based Ġvocabulary Ġ( and Ġhence Ġcannot Ġbe Ġpredicted Ġby Ġthe Ġmodel ). ĠThis Ġinclude Ġdisc arding ĠB IB REF 3 Ġstimuli Ġinvolving Ġthe Ġwords Ġswim s Ġor Ġadm ires , Ġresulting Ġin Ġ23 , 368 Ġdiscarded Ġpairs Ġ( out Ġof Ġ152 , 300 ). ĠI Ġsimilarly Ġdiscard Ġ680 Ġsentences Ġfrom ĠB IB REF 1 Ġwhere Ġthe Ġfocus Ġverb Ġor Ġits Ġinf lection Ġwere Ġone Ġof Ġ108 Ġout - of - voc abulary Ġtokens , Ġand Ġ28 Ġsentence - p airs Ġ( 8 Ġtokens ) Ġfrom ĠB IB REF 2 Ġ. </s> The ĠB ERT Ġresults Ġare Ġnot Ġdirectly Ġcomparable Ġto Ġthe Ġnumbers Ġreported Ġin Ġprevious Ġwork . ĠBeyond Ġthe Ġdifferences Ġdue Ġto Ġbid irection ality Ġand Ġthe Ġdiscarded Ġstimuli , Ġthe ĠB ERT Ġmodels Ġare Ġalso Ġtrained Ġon Ġa Ġdifferent Ġand Ġlarger Ġcorpus Ġ( cover ing Ġboth Ġwik ipedia Ġand Ġbooks ). </s> Code Ġis Ġavailable Ġat Ġhttps :// github . com / yo av g / bert - sy ntax . </s> Results </s> T ables Ġ1 Ġ, Ġ2 Ġand Ġ3 Ġshow Ġthe Ġresults . ĠAll Ġcases Ġexhibit Ġhigh Ġscores âĢĶ in Ġthe Ġvast Ġmajority Ġof Ġthe Ġcases Ġsubstantially Ġhigher Ġthan Ġreported Ġin Ġprevious Ġwork . ĠAs Ġdiscussed Ġabove , Ġthe Ġresults Ġare Ġnot Ġdirectly Ġcomparable Ġto Ġprevious Ġwork : Ġthe ĠB ERT Ġmodels Ġare Ġtrained Ġon Ġdifferent Ġ( and Ġlarger ) Ġdata , Ġare Ġallowed Ġto Ġaccess Ġthe Ġsuffix Ġof Ġthe Ġsentence Ġin Ġaddition Ġto Ġits Ġprefix , Ġand Ġare Ġevaluated Ġon Ġsomewhat Ġdifferent Ġdata Ġdue Ġto Ġdisc arding ĠO OV Ġitems . ĠStill , Ġtaken Ġtogether , Ġthe Ġhigh Ġperformance Ġnumbers Ġindicate Ġthat Ġthe Ġpurely Ġattention - based ĠB ERT Ġmodels Ġare Ġlikely Ġcapable Ġof Ġcapturing Ġthe Ġsame Ġkind Ġof Ġsynt actic Ġregular ities Ġthat ĠL ST M - based Ġmodels Ġare Ġcapable Ġof Ġcapturing , Ġat Ġleast Ġas Ġwell Ġas Ġthe ĠL ST M Ġmodels Ġand Ġprobably Ġbetter . </s> Another Ġnoticeable Ġand Ġinteresting Ġtrend Ġis Ġthat Ġlarger Ġis Ġnot Ġnecessarily Ġbetter : Ġthe ĠB ERT - Base Ġmodel Ġoutper forms Ġthe ĠB ERT - Large Ġmodel Ġon Ġmany Ġof Ġthe Ġsynt actic Ġconditions . </s> Discussion </s> The ĠB ERT Ġmodels Ġperform Ġremarkably Ġwell Ġon Ġall Ġthe Ġsynt actic Ġtest Ġcases . ĠI Ġexpected Ġthe Ġattention - based Ġmechanism Ġto Ġfail Ġon Ġthese Ġ( comp ared Ġto Ġthe ĠL ST M - based Ġmodels ), Ġand Ġam Ġsurprised Ġby Ġthese Ġresults . ĠThe ĠB IB REF 2 Ġand ĠB IB REF 3 Ġconditions Ġrule Ġout Ġthe Ġpossibility Ġof Ġoverly Ġrelying Ġon Ġselection al Ġpreference Ġcues Ġor Ġmemor izing Ġthe Ġwik ipedia Ġtraining Ġdata , Ġand Ġsuggest Ġreal Ġsynt actic Ġgeneral ization Ġis Ġtaking Ġplace . ĠExpl oring Ġthe Ġextent Ġto Ġwhich Ġdeep Ġpurely - att ention - based Ġarchitectures Ġsuch Ġas ĠB ERT Ġare Ġcapable Ġof Ġcapturing Ġhierarchy - sensitive Ġand Ġsynt actic Ġdependencies âĢĶ as Ġwell Ġas Ġthe Ġmechanisms Ġby Ġwhich Ġthis Ġis Ġachieved âĢĶ is Ġa Ġfascinating Ġarea Ġfor Ġfuture Ġresearch .
Introduction The recently introduced BERT model BIBREF0 exhibits strong performance on several language understanding benchmarks. To what extent does it capture syntax-sensitive structures? Recent work examines the extent to which RNN-based models capture syntax-sensitive phenomena that are traditionally taken as evidence for the existence in hierarchical structure. In particular, in BIBREF1 we assess the ability of LSTMs to learn subject-verb agreement patterns in English, and evaluate on naturally occurring wikipedia sentences. BIBREF2 also consider subject-verb agreement, but in a “colorless green ideas” setting in which content words in naturally occurring sentences are replaced with random words with the same part-of-speech and inflection, thus ensuring a focus on syntax rather than on selectional-preferences based cues. BIBREF3 consider a wider range of syntactic phenomena (subject-verb agreement, reflexive anaphora, negative polarity items) using manually constructed stimuli, allowing for greater coverage and control than in the naturally occurring setting. The BERT model is based on the “Transformer” architecture BIBREF4 , which—in contrast to RNNs—relies purely on attention mechanisms, and does not have an explicit notion of word order beyond marking each word with its absolute-position embedding. This reliance on attention may lead one to expect decreased performance on syntax-sensitive tasks compared to RNN (LSTM) models that do model word order directly, and explicitly track states across the sentence. Indeed, BIBREF5 finds that transformer-based models perform worse than LSTM models on the BIBREF1 agreement prediction dataset. In contrast, BIBREF6 find that self-attention performs on par with LSTM for syntax sensitive dependencies in the context of machine-translation, and performance on syntactic tasks is correlated with the number of attention heads in multi-head attention. I adapt the evaluation protocol and stimuli of BIBREF1 , BIBREF2 and BIBREF3 to the bidirectional setting required by BERT, and evaluate the pre-trained BERT models (both the Large and the Base models). Surprisingly (at least to me), the out-of-the-box models (without any task-specific fine-tuning) perform very well on all the syntactic tasks. Methodology I use the stimuli provided by BIBREF1 , BIBREF2 , BIBREF3 , but change the experimental protocol to adapt it to the bidirectional nature of the BERT model. This requires discarding some of the stimuli, as described below. Thus, the numbers are not strictly comparable to those reported in previous work. Previous setups All three previous work use uni-directional language-model-like models. BIBREF1 start with existing sentences from wikipedia that contain a present-tense verb. They feed each sentence word by word into an LSTM, stop right before the focus verb, and ask the model to predict a binary plural/singular decision (supervised setup) or compare the probability assigned by a pre-trained language model (LM) to the plural vs singular forms of the verb (LM setup). The evaluation is then performed on sentences with “agreement attractors” in which at there is at least one noun between the verb and its subject, and all of the nouns between the verb and subject are of the opposite number from the subject. BIBREF2 also start with existing sentences. However, in order to control for the possibillity of the model learning to rely on “semantic” selectional-preferences cues rather than syntactic ones, they replace each content word with random words from the same part-of-speech and inflection. This results in “coloreless green ideas” nonce sentences. The evaluation is then performed similarly to the LM setup of BIBREF1 : the sentence is fed into a pre-traiend LSTM LM up to the focus verb, and the model is considered correct if the probability assigned to the correct inflection of the original verb form given the prefix is larger than that assigned to the incorrect inflection. BIBREF3 focus on manually constructed and controlled stimuli, that also emphasizes linguistic structure over selectional preferences. They construct minimal pairs of grammatical and ungrammatical sentences, feed each one in its entirety into a pre-trained LSTM-LM, and compare the perplexity assigned by the model to the grammatical and ungrammatical sentences. The model is “correct” if it assigns the grammatical sentence a higher probability than to the ungrammatical one. Since the minimal pairs for most phenomena differ only in a single word (the focus verb), this scoring is very similar to the one used in the two previous works. However, it does consider the continuation of the sentence after the focus verb, and also allows for assessing phenomena that require change into two or more words (like negative polarity items). Adaptation to the BERT model In contrast to these works, the BERT model is bi-directional: it is trained to predict the identity of masked words based on both the prefix and suffix surrounding these words. I adapt the uni-directional setup by feeding into BERT the complete sentence, while masking out the single focus verb. I then ask BERT for its word predictions for the masked position, and compare the score assigned to the original correct verb to the score assigned to the incorrect one. For example, for the sentence: a 2002 systemic review of herbal products found that several herbs , including peppermint and caraway , have anti-dyspeptic effects for non-ulcer dyspepsia with “ encouraging safety profiles ” . (from BIBREF1 ) I feed into BERT: [CLS] a 2002 systemic review of herbal products found that several herbs , including peppermint and caraway , [MASK] anti-dyspeptic effects for non-ulcer dyspepsia with “ encouraging safety profiles ” . and look for the score assigned to the words have and has at the masked position. Similarly, for the pair the game that the guard hates is bad . the game that the guard hates are bad . (from BIBREF3 ), I feed into BERT: [CLS] the game that the guard hates [MASK] bad . and compare the scores predicted for is and are. This differs from BIBREF1 and BIBREF2 by considering the entire sentence (excluding the verb) and not just its prefix leading to the verb, and differs from BIBREF3 by conditioning the focus verb on bidirectional context. I use the PyTorch implementation of BERT, with the pre-trained models supplied by Google. I experiment with the bert-large-uncased and bert-base-uncased models. The bi-directional setup precludes using using the NPI stimuli of BIBREF3 , in which the minimal pair differs in two words position, which I discard from the evaluation. I also discard the agreement cases involving the verbs is or are in BIBREF1 and in BIBREF2 , because some of them are copular construction, in which strong agreement hints can be found also on the object following the verb. This is not an issue in the manually constructed BIBREF3 stimuli due to the patterns they chose. Finally, I discard stimuli in which the focus verb or its plural/singular inflection does not appear as a single word in the BERT word-piece-based vocabulary (and hence cannot be predicted by the model). This include discarding BIBREF3 stimuli involving the words swims or admires, resulting in 23,368 discarded pairs (out of 152,300). I similarly discard 680 sentences from BIBREF1 where the focus verb or its inflection were one of 108 out-of-vocabulary tokens, and 28 sentence-pairs (8 tokens) from BIBREF2 . The BERT results are not directly comparable to the numbers reported in previous work. Beyond the differences due to bidirectionality and the discarded stimuli, the BERT models are also trained on a different and larger corpus (covering both wikipedia and books). Code is available at https://github.com/yoavg/bert-syntax. Results Tables 1 , 2 and 3 show the results. All cases exhibit high scores—in the vast majority of the cases substantially higher than reported in previous work. As discussed above, the results are not directly comparable to previous work: the BERT models are trained on different (and larger) data, are allowed to access the suffix of the sentence in addition to its prefix, and are evaluated on somewhat different data due to discarding OOV items. Still, taken together, the high performance numbers indicate that the purely attention-based BERT models are likely capable of capturing the same kind of syntactic regularities that LSTM-based models are capable of capturing, at least as well as the LSTM models and probably better. Another noticeable and interesting trend is that larger is not necessarily better: the BERT-Base model outperforms the BERT-Large model on many of the syntactic conditions. Discussion The BERT models perform remarkably well on all the syntactic test cases. I expected the attention-based mechanism to fail on these (compared to the LSTM-based models), and am surprised by these results. The BIBREF2 and BIBREF3 conditions rule out the possibility of overly relying on selectional preference cues or memorizing the wikipedia training data, and suggest real syntactic generalization is taking place. Exploring the extent to which deep purely-attention-based architectures such as BERT are capable of capturing hierarchy-sensitive and syntactic dependencies—as well as the mechanisms by which this is achieved—is a fascinating area for future research.
Were any of these tasks evaluated in any previous work?
d27438b11bc70e706431dda0af2b1c0b0d209f96
Were Ġany Ġof Ġthese Ġtasks Ġevaluated Ġin Ġany Ġprevious Ġwork ?
Yes
<s> Were Ġany Ġof Ġthese Ġtasks Ġevaluated Ġin Ġany Ġprevious Ġwork ? </s> Introduction </s> The Ġrecently Ġintroduced ĠB ERT Ġmodel ĠB IB REF 0 Ġexhibits Ġstrong Ġperformance Ġon Ġseveral Ġlanguage Ġunderstanding Ġbenchmarks . ĠTo Ġwhat Ġextent Ġdoes Ġit Ġcapture Ġsyntax - sensitive Ġstructures ? </s> Recent Ġwork Ġexamines Ġthe Ġextent Ġto Ġwhich ĠR NN - based Ġmodels Ġcapture Ġsyntax - sensitive Ġphenomena Ġthat Ġare Ġtraditionally Ġtaken Ġas Ġevidence Ġfor Ġthe Ġexistence Ġin Ġhierarchical Ġstructure . ĠIn Ġparticular , Ġin ĠB IB REF 1 Ġwe Ġassess Ġthe Ġability Ġof ĠL ST Ms Ġto Ġlearn Ġsubject - verb Ġagreement Ġpatterns Ġin ĠEnglish , Ġand Ġevaluate Ġon Ġnaturally Ġoccurring Ġwik ipedia Ġsentences . ĠB IB REF 2 Ġalso Ġconsider Ġsubject - verb Ġagreement , Ġbut Ġin Ġa ĠâĢ ľ color less Ġgreen Ġideas âĢ Ŀ Ġsetting Ġin Ġwhich Ġcontent Ġwords Ġin Ġnaturally Ġoccurring Ġsentences Ġare Ġreplaced Ġwith Ġrandom Ġwords Ġwith Ġthe Ġsame Ġpart - of - speech Ġand Ġinf lection , Ġthus Ġensuring Ġa Ġfocus Ġon Ġsyntax Ġrather Ġthan Ġon Ġselection al - pre f erences Ġbased Ġcues . ĠB IB REF 3 Ġconsider Ġa Ġwider Ġrange Ġof Ġsynt actic Ġphenomena Ġ( subject - verb Ġagreement , Ġreflex ive Ġan aph ora , Ġnegative Ġpol arity Ġitems ) Ġusing Ġmanually Ġconstructed Ġstimuli , Ġallowing Ġfor Ġgreater Ġcoverage Ġand Ġcontrol Ġthan Ġin Ġthe Ġnaturally Ġoccurring Ġsetting . </s> The ĠB ERT Ġmodel Ġis Ġbased Ġon Ġthe ĠâĢ ľ Trans former âĢ Ŀ Ġarchitecture ĠB IB REF 4 Ġ, Ġwhich âĢĶ in Ġcontrast Ġto ĠR NN s âĢĶ rel ies Ġpurely Ġon Ġattention Ġmechanisms , Ġand Ġdoes Ġnot Ġhave Ġan Ġexplicit Ġnotion Ġof Ġword Ġorder Ġbeyond Ġmarking Ġeach Ġword Ġwith Ġits Ġabsolute - position Ġembed ding . ĠThis Ġreliance Ġon Ġattention Ġmay Ġlead Ġone Ġto Ġexpect Ġdecreased Ġperformance Ġon Ġsyntax - sensitive Ġtasks Ġcompared Ġto ĠR NN Ġ( L ST M ) Ġmodels Ġthat Ġdo Ġmodel Ġword Ġorder Ġdirectly , Ġand Ġexplicitly Ġtrack Ġstates Ġacross Ġthe Ġsentence . ĠIndeed , ĠB IB REF 5 Ġfinds Ġthat Ġtransformer - based Ġmodels Ġperform Ġworse Ġthan ĠL ST M Ġmodels Ġon Ġthe ĠB IB REF 1 Ġagreement Ġprediction Ġdataset . ĠIn Ġcontrast , ĠB IB REF 6 Ġfind Ġthat Ġself - att ention Ġperforms Ġon Ġpar Ġwith ĠL ST M Ġfor Ġsyntax Ġsensitive Ġdependencies Ġin Ġthe Ġcontext Ġof Ġmachine - translation , Ġand Ġperformance Ġon Ġsynt actic Ġtasks Ġis Ġcorrelated Ġwith Ġthe Ġnumber Ġof Ġattention Ġheads Ġin Ġmulti - head Ġattention . </s> I Ġadapt Ġthe Ġevaluation Ġprotocol Ġand Ġstimuli Ġof ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġand ĠB IB REF 3 Ġto Ġthe Ġbid irection al Ġsetting Ġrequired Ġby ĠB ERT , Ġand Ġevaluate Ġthe Ġpre - trained ĠB ERT Ġmodels Ġ( both Ġthe ĠLarge Ġand Ġthe ĠBase Ġmodels ). ĠSurprisingly Ġ( at Ġleast Ġto Ġme ), Ġthe Ġout - of - the - box Ġmodels Ġ( without Ġany Ġtask - specific Ġfine - tun ing ) Ġperform Ġvery Ġwell Ġon Ġall Ġthe Ġsynt actic Ġtasks . </s> Method ology </s> I Ġuse Ġthe Ġstimuli Ġprovided Ġby ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, Ġbut Ġchange Ġthe Ġexperimental Ġprotocol Ġto Ġadapt Ġit Ġto Ġthe Ġbid irection al Ġnature Ġof Ġthe ĠB ERT Ġmodel . ĠThis Ġrequires Ġdisc arding Ġsome Ġof Ġthe Ġstimuli , Ġas Ġdescribed Ġbelow . ĠThus , Ġthe Ġnumbers Ġare Ġnot Ġstrictly Ġcomparable Ġto Ġthose Ġreported Ġin Ġprevious Ġwork . </s> Previous Ġsetups </s> All Ġthree Ġprevious Ġwork Ġuse Ġun i - direction al Ġlanguage - model - like Ġmodels . </s> BI B REF 1 Ġstart Ġwith Ġexisting Ġsentences Ġfrom Ġwik ipedia Ġthat Ġcontain Ġa Ġpresent - t ense Ġverb . ĠThey Ġfeed Ġeach Ġsentence Ġword Ġby Ġword Ġinto Ġan ĠL ST M , Ġstop Ġright Ġbefore Ġthe Ġfocus Ġverb , Ġand Ġask Ġthe Ġmodel Ġto Ġpredict Ġa Ġbinary Ġplural / sing ular Ġdecision Ġ( super vised Ġsetup ) Ġor Ġcompare Ġthe Ġprobability Ġassigned Ġby Ġa Ġpre - trained Ġlanguage Ġmodel Ġ( LM ) Ġto Ġthe Ġplural Ġvs Ġsingular Ġforms Ġof Ġthe Ġverb Ġ( LM Ġsetup ). ĠThe Ġevaluation Ġis Ġthen Ġperformed Ġon Ġsentences Ġwith ĠâĢ ľ ag reement Ġattract ors âĢ Ŀ Ġin Ġwhich Ġat Ġthere Ġis Ġat Ġleast Ġone Ġnoun Ġbetween Ġthe Ġverb Ġand Ġits Ġsubject , Ġand Ġall Ġof Ġthe Ġnoun s Ġbetween Ġthe Ġverb Ġand Ġsubject Ġare Ġof Ġthe Ġopposite Ġnumber Ġfrom Ġthe Ġsubject . </s> BI B REF 2 Ġalso Ġstart Ġwith Ġexisting Ġsentences . ĠHowever , Ġin Ġorder Ġto Ġcontrol Ġfor Ġthe Ġposs ib ill ity Ġof Ġthe Ġmodel Ġlearning Ġto Ġrely Ġon ĠâĢ ľ sem antic âĢ Ŀ Ġselection al - pre f erences Ġcues Ġrather Ġthan Ġsynt actic Ġones , Ġthey Ġreplace Ġeach Ġcontent Ġword Ġwith Ġrandom Ġwords Ġfrom Ġthe Ġsame Ġpart - of - speech Ġand Ġinf lection . ĠThis Ġresults Ġin ĠâĢ ľ col ore less Ġgreen Ġideas âĢ Ŀ Ġnon ce Ġsentences . ĠThe Ġevaluation Ġis Ġthen Ġperformed Ġsimilarly Ġto Ġthe ĠLM Ġsetup Ġof ĠB IB REF 1 Ġ: Ġthe Ġsentence Ġis Ġfed Ġinto Ġa Ġpre - tra i end ĠL ST M ĠLM Ġup Ġto Ġthe Ġfocus Ġverb , Ġand Ġthe Ġmodel Ġis Ġconsidered Ġcorrect Ġif Ġthe Ġprobability Ġassigned Ġto Ġthe Ġcorrect Ġinf lection Ġof Ġthe Ġoriginal Ġverb Ġform Ġgiven Ġthe Ġprefix Ġis Ġlarger Ġthan Ġthat Ġassigned Ġto Ġthe Ġincorrect Ġinf lection . </s> BI B REF 3 Ġfocus Ġon Ġmanually Ġconstructed Ġand Ġcontrolled Ġstimuli , Ġthat Ġalso Ġemphasizes Ġlinguistic Ġstructure Ġover Ġselection al Ġpreferences . ĠThey Ġconstruct Ġminimal Ġpairs Ġof Ġgram matical Ġand Ġun gram matical Ġsentences , Ġfeed Ġeach Ġone Ġin Ġits Ġentirety Ġinto Ġa Ġpre - trained ĠL ST M - LM , Ġand Ġcompare Ġthe Ġperplex ity Ġassigned Ġby Ġthe Ġmodel Ġto Ġthe Ġgram matical Ġand Ġun gram matical Ġsentences . ĠThe Ġmodel Ġis ĠâĢ ľ correct âĢ Ŀ Ġif Ġit Ġassigns Ġthe Ġgram matical Ġsentence Ġa Ġhigher Ġprobability Ġthan Ġto Ġthe Ġun gram matical Ġone . ĠSince Ġthe Ġminimal Ġpairs Ġfor Ġmost Ġphenomena Ġdiffer Ġonly Ġin Ġa Ġsingle Ġword Ġ( the Ġfocus Ġverb ), Ġthis Ġscoring Ġis Ġvery Ġsimilar Ġto Ġthe Ġone Ġused Ġin Ġthe Ġtwo Ġprevious Ġworks . ĠHowever , Ġit Ġdoes Ġconsider Ġthe Ġcontinuation Ġof Ġthe Ġsentence Ġafter Ġthe Ġfocus Ġverb , Ġand Ġalso Ġallows Ġfor Ġassessing Ġphenomena Ġthat Ġrequire Ġchange Ġinto Ġtwo Ġor Ġmore Ġwords Ġ( like Ġnegative Ġpol arity Ġitems ). </s> Adapt ation Ġto Ġthe ĠB ERT Ġmodel </s> In Ġcontrast Ġto Ġthese Ġworks , Ġthe ĠB ERT Ġmodel Ġis Ġbi - direction al : Ġit Ġis Ġtrained Ġto Ġpredict Ġthe Ġidentity Ġof Ġmasked Ġwords Ġbased Ġon Ġboth Ġthe Ġprefix Ġand Ġsuffix Ġsurrounding Ġthese Ġwords . ĠI Ġadapt Ġthe Ġun i - direction al Ġsetup Ġby Ġfeeding Ġinto ĠB ERT Ġthe Ġcomplete Ġsentence , Ġwhile Ġmask ing Ġout Ġthe Ġsingle Ġfocus Ġverb . ĠI Ġthen Ġask ĠB ERT Ġfor Ġits Ġword Ġpredictions Ġfor Ġthe Ġmasked Ġposition , Ġand Ġcompare Ġthe Ġscore Ġassigned Ġto Ġthe Ġoriginal Ġcorrect Ġverb Ġto Ġthe Ġscore Ġassigned Ġto Ġthe Ġincorrect Ġone . </s> For Ġexample , Ġfor Ġthe Ġsentence : </s> a Ġ2002 Ġsystemic Ġreview Ġof Ġherbal Ġproducts Ġfound Ġthat Ġseveral Ġherbs Ġ, Ġincluding Ġpepper mint Ġand Ġcar away Ġ, Ġhave Ġanti - d ys pe ptic Ġeffects Ġfor Ġnon - ul cer Ġdys pe ps ia Ġwith ĠâĢ ľ Ġencouraging Ġsafety Ġprofiles ĠâĢ Ŀ Ġ. Ġ( from ĠB IB REF 1 Ġ) </s> I Ġfeed Ġinto ĠB ERT : </s> [ CL S ] Ġa Ġ2002 Ġsystemic Ġreview Ġof Ġherbal Ġproducts Ġfound Ġthat Ġseveral Ġherbs Ġ, Ġincluding Ġpepper mint Ġand Ġcar away Ġ, Ġ[ MAS K ] Ġanti - d ys pe ptic Ġeffects Ġfor Ġnon - ul cer Ġdys pe ps ia Ġwith ĠâĢ ľ Ġencouraging Ġsafety Ġprofiles ĠâĢ Ŀ Ġ. Ġand Ġlook Ġfor Ġthe Ġscore Ġassigned Ġto Ġthe Ġwords Ġhave Ġand Ġhas Ġat Ġthe Ġmasked Ġposition . </s> Similarly , Ġfor Ġthe Ġpair </s> the Ġgame Ġthat Ġthe Ġguard Ġhates Ġis Ġbad Ġ. </s> the Ġgame Ġthat Ġthe Ġguard Ġhates Ġare Ġbad Ġ. </s> ( from ĠB IB REF 3 Ġ), ĠI Ġfeed Ġinto ĠB ERT : </s> [ CL S ] Ġthe Ġgame Ġthat Ġthe Ġguard Ġhates Ġ[ MAS K ] Ġbad Ġ. </s> and Ġcompare Ġthe Ġscores Ġpredicted Ġfor Ġis Ġand Ġare . </s> This Ġdiffers Ġfrom ĠB IB REF 1 Ġand ĠB IB REF 2 Ġby Ġconsidering Ġthe Ġentire Ġsentence Ġ( excluding Ġthe Ġverb ) Ġand Ġnot Ġjust Ġits Ġprefix Ġleading Ġto Ġthe Ġverb , Ġand Ġdiffers Ġfrom ĠB IB REF 3 Ġby Ġconditioning Ġthe Ġfocus Ġverb Ġon Ġbid irection al Ġcontext . </s> I Ġuse Ġthe ĠPy Tor ch Ġimplementation Ġof ĠB ERT , Ġwith Ġthe Ġpre - trained Ġmodels Ġsupplied Ġby ĠGoogle . ĠI Ġexperiment Ġwith Ġthe Ġb ert - large - unc ased Ġand Ġb ert - base - unc ased Ġmodels . </s> The Ġbi - direction al Ġsetup Ġpre cludes Ġusing Ġusing Ġthe ĠN PI Ġstimuli Ġof ĠB IB REF 3 Ġ, Ġin Ġwhich Ġthe Ġminimal Ġpair Ġdiffers Ġin Ġtwo Ġwords Ġposition , Ġwhich ĠI Ġdiscard Ġfrom Ġthe Ġevaluation . ĠI Ġalso Ġdiscard Ġthe Ġagreement Ġcases Ġinvolving Ġthe Ġverbs Ġis Ġor Ġare Ġin ĠB IB REF 1 Ġand Ġin ĠB IB REF 2 Ġ, Ġbecause Ġsome Ġof Ġthem Ġare Ġcop ular Ġconstruction , Ġin Ġwhich Ġstrong Ġagreement Ġhints Ġcan Ġbe Ġfound Ġalso Ġon Ġthe Ġobject Ġfollowing Ġthe Ġverb . ĠThis Ġis Ġnot Ġan Ġissue Ġin Ġthe Ġmanually Ġconstructed ĠB IB REF 3 Ġstimuli Ġdue Ġto Ġthe Ġpatterns Ġthey Ġchose . </s> Finally , ĠI Ġdiscard Ġstimuli Ġin Ġwhich Ġthe Ġfocus Ġverb Ġor Ġits Ġplural / sing ular Ġinf lection Ġdoes Ġnot Ġappear Ġas Ġa Ġsingle Ġword Ġin Ġthe ĠB ERT Ġword - piece - based Ġvocabulary Ġ( and Ġhence Ġcannot Ġbe Ġpredicted Ġby Ġthe Ġmodel ). ĠThis Ġinclude Ġdisc arding ĠB IB REF 3 Ġstimuli Ġinvolving Ġthe Ġwords Ġswim s Ġor Ġadm ires , Ġresulting Ġin Ġ23 , 368 Ġdiscarded Ġpairs Ġ( out Ġof Ġ152 , 300 ). ĠI Ġsimilarly Ġdiscard Ġ680 Ġsentences Ġfrom ĠB IB REF 1 Ġwhere Ġthe Ġfocus Ġverb Ġor Ġits Ġinf lection Ġwere Ġone Ġof Ġ108 Ġout - of - voc abulary Ġtokens , Ġand Ġ28 Ġsentence - p airs Ġ( 8 Ġtokens ) Ġfrom ĠB IB REF 2 Ġ. </s> The ĠB ERT Ġresults Ġare Ġnot Ġdirectly Ġcomparable Ġto Ġthe Ġnumbers Ġreported Ġin Ġprevious Ġwork . ĠBeyond Ġthe Ġdifferences Ġdue Ġto Ġbid irection ality Ġand Ġthe Ġdiscarded Ġstimuli , Ġthe ĠB ERT Ġmodels Ġare Ġalso Ġtrained Ġon Ġa Ġdifferent Ġand Ġlarger Ġcorpus Ġ( cover ing Ġboth Ġwik ipedia Ġand Ġbooks ). </s> Code Ġis Ġavailable Ġat Ġhttps :// github . com / yo av g / bert - sy ntax . </s> Results </s> T ables Ġ1 Ġ, Ġ2 Ġand Ġ3 Ġshow Ġthe Ġresults . ĠAll Ġcases Ġexhibit Ġhigh Ġscores âĢĶ in Ġthe Ġvast Ġmajority Ġof Ġthe Ġcases Ġsubstantially Ġhigher Ġthan Ġreported Ġin Ġprevious Ġwork . ĠAs Ġdiscussed Ġabove , Ġthe Ġresults Ġare Ġnot Ġdirectly Ġcomparable Ġto Ġprevious Ġwork : Ġthe ĠB ERT Ġmodels Ġare Ġtrained Ġon Ġdifferent Ġ( and Ġlarger ) Ġdata , Ġare Ġallowed Ġto Ġaccess Ġthe Ġsuffix Ġof Ġthe Ġsentence Ġin Ġaddition Ġto Ġits Ġprefix , Ġand Ġare Ġevaluated Ġon Ġsomewhat Ġdifferent Ġdata Ġdue Ġto Ġdisc arding ĠO OV Ġitems . ĠStill , Ġtaken Ġtogether , Ġthe Ġhigh Ġperformance Ġnumbers Ġindicate Ġthat Ġthe Ġpurely Ġattention - based ĠB ERT Ġmodels Ġare Ġlikely Ġcapable Ġof Ġcapturing Ġthe Ġsame Ġkind Ġof Ġsynt actic Ġregular ities Ġthat ĠL ST M - based Ġmodels Ġare Ġcapable Ġof Ġcapturing , Ġat Ġleast Ġas Ġwell Ġas Ġthe ĠL ST M Ġmodels Ġand Ġprobably Ġbetter . </s> Another Ġnoticeable Ġand Ġinteresting Ġtrend Ġis Ġthat Ġlarger Ġis Ġnot Ġnecessarily Ġbetter : Ġthe ĠB ERT - Base Ġmodel Ġoutper forms Ġthe ĠB ERT - Large Ġmodel Ġon Ġmany Ġof Ġthe Ġsynt actic Ġconditions . </s> Discussion </s> The ĠB ERT Ġmodels Ġperform Ġremarkably Ġwell Ġon Ġall Ġthe Ġsynt actic Ġtest Ġcases . ĠI Ġexpected Ġthe Ġattention - based Ġmechanism Ġto Ġfail Ġon Ġthese Ġ( comp ared Ġto Ġthe ĠL ST M - based Ġmodels ), Ġand Ġam Ġsurprised Ġby Ġthese Ġresults . ĠThe ĠB IB REF 2 Ġand ĠB IB REF 3 Ġconditions Ġrule Ġout Ġthe Ġpossibility Ġof Ġoverly Ġrelying Ġon Ġselection al Ġpreference Ġcues Ġor Ġmemor izing Ġthe Ġwik ipedia Ġtraining Ġdata , Ġand Ġsuggest Ġreal Ġsynt actic Ġgeneral ization Ġis Ġtaking Ġplace . ĠExpl oring Ġthe Ġextent Ġto Ġwhich Ġdeep Ġpurely - att ention - based Ġarchitectures Ġsuch Ġas ĠB ERT Ġare Ġcapable Ġof Ġcapturing Ġhierarchy - sensitive Ġand Ġsynt actic Ġdependencies âĢĶ as Ġwell Ġas Ġthe Ġmechanisms Ġby Ġwhich Ġthis Ġis Ġachieved âĢĶ is Ġa Ġfascinating Ġarea Ġfor Ġfuture Ġresearch .