token-level chunk label embeddings,  chunk boundary information is passed into the task model via BIOUL encoding of the labels
EXTRACTIVE
Our second approach incorporates shallow syntactic information in downstream tasks via token-level chunk label embeddings. Task training (and test) data is automatically chunked, and chunk boundary information is passed into the task model via BIOUL encoding of the labels. We add randomly initialized chunk label embeddings to task-specific input encoders, which are then fine-tuned for task-specific objectives. This approach does not require a shallow syntactic encoder or chunk annotations for pretraining cwrs, only a chunker. Hence, this can more directly measure the impact of shallow syntax for a given task.
1908.11047
Introduction </s> The ĠN LP Ġcommunity Ġis Ġrevis iting Ġthe Ġrole Ġof Ġlinguistic Ġstructure Ġin Ġapplications Ġwith Ġthe Ġadvent Ġof Ġcontextual Ġword Ġrepresentations Ġ( c w rs ) Ġderived Ġfrom Ġpret raining Ġlanguage Ġmodels Ġon Ġlarge Ġcorpor a ĠB IB REF 2 , ĠB IB REF 3 , ĠB IB REF 4 , ĠB IB REF 5 . ĠRecent Ġwork Ġhas Ġshown Ġthat Ġdownstream Ġtask Ġperformance Ġmay Ġbenefit Ġfrom Ġexplicitly Ġinjecting Ġa Ġsynt actic Ġinduct ive Ġbias Ġinto Ġmodel Ġarchitectures ĠB IB REF 6 , Ġeven Ġwhen Ġc w rs Ġare Ġalso Ġused ĠB IB REF 7 . ĠHowever , Ġhigh Ġquality Ġlinguistic Ġstructure Ġannotation Ġat Ġa Ġlarge Ġscale Ġremains Ġexpensive âĢĶ a Ġtrade - off Ġneeds Ġto Ġbe Ġmade Ġbetween Ġthe Ġquality Ġof Ġthe Ġannotations Ġand Ġthe Ġcomputational Ġexpense Ġof Ġobtaining Ġthem . ĠSh allow Ġsynt actic Ġstructures Ġ( BI B REF 8 ; Ġalso Ġcalled Ġchunk Ġsequences ) Ġoffer Ġa Ġviable Ġmiddle Ġground , Ġby Ġproviding Ġa Ġflat , Ġnon - h ier arch ical Ġapproximation Ġto Ġphrase - sy nt actic Ġtrees Ġ( see ĠFig . ĠFIG REF 1 Ġfor Ġan Ġexample ). ĠThese Ġstructures Ġcan Ġbe Ġobtained Ġefficiently , Ġand Ġwith Ġhigh Ġaccuracy , Ġusing Ġsequence Ġlabel ers . ĠIn Ġthis Ġpaper Ġwe Ġconsider Ġshallow Ġsyntax Ġto Ġbe Ġa Ġproxy Ġfor Ġlinguistic Ġstructure . </s> While Ġshallow Ġsynt actic Ġchunks Ġare Ġalmost Ġas Ġubiquitous Ġas Ġpart - of - speech Ġtags Ġin Ġstandard ĠN LP Ġpipelines ĠB IB REF 9 , Ġtheir Ġrelative Ġmerits Ġin Ġthe Ġpresence Ġof Ġc w rs Ġremain Ġunclear . ĠWe Ġinvestigate Ġthe Ġrole Ġof Ġthese Ġstructures Ġusing Ġtwo Ġmethods . ĠFirst , Ġwe Ġenhance Ġthe ĠEL Mo Ġarchitecture ĠB IB REF 0 Ġto Ġallow Ġpret raining Ġon Ġpredicted Ġshallow Ġsynt actic Ġpars es , Ġinstead Ġof Ġjust Ġraw Ġtext , Ġso Ġthat Ġcontextual Ġembed d ings Ġmake Ġuse Ġof Ġshallow Ġsynt actic Ġcontext Ġ( Â§ SEC REF 2 ). ĠOur Ġsecond Ġmethod Ġinvolves Ġclassical Ġaddition Ġof Ġchunk Ġfeatures Ġto Ġc wr - inf used Ġarchitectures Ġfor Ġfour Ġdifferent Ġdownstream Ġtasks Ġ( Â§ SEC REF 3 ). ĠSh allow Ġsynt actic Ġinformation Ġis Ġobtained Ġautomatically Ġusing Ġa Ġhighly Ġaccurate Ġmodel Ġ( 97 % Ġ$ F _ 1 $ Ġon Ġstandard Ġbenchmarks ). ĠIn Ġboth Ġsettings , Ġwe Ġobserve Ġonly Ġmodest Ġgains Ġon Ġthree Ġof Ġthe Ġfour Ġdownstream Ġtasks Ġrelative Ġto ĠEL Mo - only Ġbas elines Ġ( Â§ SEC REF 4 ). </s> Recent Ġwork Ġhas Ġprob ed Ġthe Ġknowledge Ġencoded Ġin Ġc w rs Ġand Ġfound Ġthey Ġcapture Ġa Ġsurprisingly Ġlarge Ġamount Ġof Ġsyntax ĠB IB REF 10 , ĠB IB REF 1 , ĠB IB REF 11 . ĠWe Ġfurther Ġexamine Ġthe Ġcontextual Ġembed d ings Ġobtained Ġfrom Ġthe Ġenhanced Ġarchitecture Ġand Ġa Ġshallow Ġsynt actic Ġcontext , Ġusing Ġblack - box Ġprobes Ġfrom ĠB IB REF 1 . ĠOur Ġanalysis Ġindicates Ġthat Ġour Ġshallow - sy ntax - aware Ġcontextual Ġembed d ings Ġdo Ġnot Ġtransfer Ġto Ġlinguistic Ġtasks Ġany Ġmore Ġeasily Ġthan ĠEL Mo Ġembed d ings Ġ( Â§ SEC REF 18 ). </s> Overall , Ġour Ġfindings Ġshow Ġthat Ġwhile Ġshallow Ġsyntax Ġcan Ġbe Ġsomewhat Ġuseful , ĠEL Mo - style Ġpret raining Ġdiscovers Ġrepresentations Ġwhich Ġmake Ġadditional Ġawareness Ġof Ġshallow Ġsyntax Ġlargely Ġredundant . </s> P ret raining Ġwith ĠSh allow ĠSynt actic ĠAnnotations </s> We Ġbriefly Ġreview Ġthe Ġshallow Ġsynt actic Ġstructures Ġused Ġin Ġthis Ġwork , Ġand Ġthen Ġpresent Ġa Ġmodel Ġarchitecture Ġto Ġobtain Ġembed d ings Ġfrom Ġshallow ĠSynt actic ĠContext Ġ( m Syn C ). </s> P ret raining Ġwith ĠSh allow ĠSynt actic ĠAnnotations Ġ: :: ĠSh allow ĠSynt ax </s> Base Ġphrase Ġchunk ing Ġis Ġa Ġcheap Ġsequence - label ing âĢĵ based Ġalternative Ġto Ġfull Ġsynt actic Ġparsing , Ġwhere Ġthe Ġsequence Ġconsists Ġof Ġnon - over l apping Ġlabeled Ġsegments Ġ( Fig . ĠFIG REF 1 Ġincludes Ġan Ġexample .) ĠFull Ġsynt actic Ġtrees Ġcan Ġbe Ġconverted Ġinto Ġsuch Ġshallow Ġsynt actic Ġchunk Ġsequences Ġusing Ġa Ġdeter ministic Ġprocedure ĠB IB REF 9 . ĠB IB REF 12 Ġoffered Ġa Ġrule - based Ġtransformation Ġder iving Ġnon - over l apping Ġchunks Ġfrom Ġphrase - st ructure Ġtrees Ġas Ġfound Ġin Ġthe ĠPenn ĠTree bank ĠB IB REF 13 . ĠThe Ġprocedure Ġper col ates Ġsome Ġsynt actic Ġphrase Ġnodes Ġfrom Ġa Ġphrase - sy nt actic Ġtree Ġto Ġthe Ġphrase Ġin Ġthe Ġleaves Ġof Ġthe Ġtree . ĠAll Ġoverlapping Ġembedded Ġphrases Ġare Ġthen Ġremoved , Ġand Ġthe Ġremainder Ġof Ġthe Ġphrase Ġgets Ġthe Ġper col ated Ġlabel âĢĶ this Ġusually Ġcorresponds Ġto Ġthe Ġhead Ġword Ġof Ġthe Ġphrase . </s> In Ġorder Ġto Ġobtain Ġshallow Ġsynt actic Ġannotations Ġon Ġa Ġlarge Ġcorpus , Ġwe Ġtrain Ġa ĠBi L ST M - CR F Ġmodel ĠB IB REF 14 , ĠB IB REF 15 , Ġwhich Ġachieves Ġ97 % Ġ$ F _ 1 $ Ġon Ġthe ĠCo N LL Ġ2000 Ġbenchmark Ġtest Ġset . ĠThe Ġtraining Ġdata Ġis Ġobtained Ġfrom Ġthe ĠCo N LL Ġ2000 Ġshared Ġtask ĠB IB REF 12 , Ġas Ġwell Ġas Ġthe Ġremaining Ġsections Ġ( except ĠÂ§ 23 Ġand ĠÂ§ 20 ) Ġof Ġthe ĠPenn ĠTree bank , Ġusing Ġthe Ġofficial Ġscript Ġfor Ġchunk Ġgeneration . ĠThe Ġstandard Ġtask Ġdefinition Ġfrom Ġthe Ġshared Ġtask Ġincludes Ġeleven Ġchunk Ġlabels , Ġas Ġshown Ġin ĠTable ĠT AB REF 4 . </s> P ret raining Ġwith ĠSh allow ĠSynt actic ĠAnnotations Ġ: :: ĠPret raining ĠObjective </s> Traditional Ġlanguage Ġmodels Ġare Ġestimated Ġto Ġmaximize Ġthe Ġlikelihood Ġof Ġeach Ġword Ġ$ x _ i $ Ġgiven Ġthe Ġwords Ġthat Ġpreced e Ġit , Ġ$ p ( x _ i Ġ\ mid Ġx _{ < i }) $. ĠGiven Ġa Ġcorpus Ġthat Ġis Ġannot ated Ġwith Ġshallow Ġsyntax , Ġwe Ġpropose Ġto Ġcondition Ġon Ġboth Ġthe Ġpreceding Ġwords Ġand Ġtheir Ġannotations . </s> We Ġassociate Ġwith Ġeach Ġword Ġ$ x _ i $ Ġthree Ġadditional Ġvariables Ġ( den oted Ġ$ c _ i $ ): Ġthe Ġindices Ġof Ġthe Ġbeginning Ġand Ġend Ġof Ġthe Ġlast Ġcompleted Ġchunk Ġbefore Ġ$ x _ i $, Ġand Ġits Ġlabel . ĠFor Ġexample , Ġin ĠFig . ĠFIG REF 8 , Ġ$ c _ 4 = \ l angle Ġ3 , Ġ3 , Ġ\ text { VP }\ r angle Ġ$ Ġfor Ġ$ x _ 4 = \ text { the } $. ĠCh unks , Ġ$ c $ Ġare Ġonly Ġused Ġas Ġconditioning Ġcontext Ġvia Ġ$ p ( x _ i Ġ\ mid Ġx _{ < i }, Ġc _{ \ le q sl ant Ġi }) $ ; Ġthey Ġare Ġnot Ġpredicted . ĠBecause Ġthe Ġ$ c $ Ġlabels Ġdepend Ġon Ġthe Ġentire Ġsentence Ġthrough Ġthe ĠCR F Ġchunk er , Ġconditioning Ġeach Ġword 's Ġprobability Ġon Ġany Ġ$ c _ i $ Ġmeans Ġthat Ġour Ġmodel Ġis , Ġstrictly Ġspeaking , Ġnot Ġa Ġlanguage Ġmodel , Ġand Ġit Ġcan Ġno Ġlonger Ġbe Ġmeaning fully Ġevaluated Ġusing Ġperplex ity . </s> A Ġright - to - left Ġmodel Ġis Ġconstructed Ġanalog ously , Ġconditioning Ġon Ġ$ c _{ \ ge q sl ant Ġi } $ Ġalongside Ġ$ x _{ > i } $. ĠFollowing ĠB IB REF 2 , Ġwe Ġuse Ġa Ġjoint Ġobjective Ġmaximizing Ġdata Ġlikelihood Ġobjectives Ġin Ġboth Ġdirections , Ġwith Ġshared Ġsoft max Ġparameters . </s> P ret raining Ġwith ĠSh allow ĠSynt actic ĠAnnotations Ġ: :: ĠPret raining ĠModel ĠArchitecture </s> Our Ġmodel Ġuses Ġtwo Ġenc od ers : Ġ$ e _{ \ math it Ġ{ seq }} $ Ġfor Ġencoding Ġthe Ġsequential Ġhistory Ġ($ x _{ < i } $ ), Ġand Ġ$ e _{ \ math it Ġ{ syn }} $ Ġfor Ġshallow Ġsynt actic Ġ( ch unk ) Ġhistory Ġ($ c _{ \ le q sl ant Ġi } $ ). ĠFor Ġboth , Ġwe Ġuse Ġtransform ers ĠB IB REF 16 , Ġwhich Ġconsist Ġof Ġlarge Ġfeed forward Ġnetworks Ġequipped Ġwith Ġmulti headed Ġself - att ention Ġmechanisms . </s> As Ġinputs Ġto Ġ$ e _{ \ math it Ġ{ seq }} $, Ġwe Ġuse Ġa Ġcontext - independent Ġembed ding , Ġobtained Ġfrom Ġa ĠCNN Ġcharacter Ġenc oder ĠB IB REF 17 Ġfor Ġeach Ġtoken Ġ$ x _ i $. ĠThe Ġoutputs Ġ$ h _ i $ Ġfrom Ġ$ e _{ \ math it Ġ{ seq }} $ Ġrepresent Ġwords Ġin Ġcontext . </s> Next , Ġwe Ġbuild Ġrepresentations Ġfor Ġ( ob served ) Ġchunks Ġin Ġthe Ġsentence Ġby Ġconc aten ating Ġa Ġlearned Ġembed ding Ġfor Ġthe Ġchunk Ġlabel Ġwith Ġ$ h $ s Ġfor Ġthe Ġboundaries Ġand Ġapplying Ġa Ġlinear Ġprojection Ġ($ f _ \ math it Ġ{ pro j } $ ). ĠThe Ġoutput Ġfrom Ġ$ f _ \ math it Ġ{ pro j } $ Ġis Ġinput Ġto Ġ$ e _{ \ math it Ġ{ syn }} $, Ġthe Ġshallow Ġsynt actic Ġenc oder , Ġand Ġresults Ġin Ġcontextual ized Ġchunk Ġrepresentations , Ġ$ g $. ĠNote Ġthat Ġthe Ġnumber Ġof Ġchunks Ġin Ġthe Ġsentence Ġis Ġless Ġthan Ġor Ġequal Ġto Ġthe Ġnumber Ġof Ġtokens . </s> Each Ġ$ h _ i $ Ġis Ġnow Ġconc at ent ated Ġwith Ġ$ g _{ c _ i } $, Ġwhere Ġ$ g _{ c _ i } $ Ġcorresponds Ġto Ġ$ c _ i $, Ġthe Ġlast Ġchunk Ġbefore Ġposition Ġ$ i $. ĠFinally , Ġthe Ġoutput Ġis Ġgiven Ġby Ġ$\ mb ox {\ text bf Ġ{ m Syn C }} _ i Ġ= Ġ{ u } _ \ math it Ġ{ pro j } ( h _ i , Ġg _{ c _ i }) Ġ= ĠW ^ \ top Ġ[ h _ i ; Ġg _{ c _ i } ] $, Ġwhere Ġ$ W $ Ġis Ġa Ġmodel Ġparameter . ĠFor Ġtraining , Ġ$\ mb ox {\ text bf Ġ{ m Syn C }} _ i $ Ġis Ġused Ġto Ġcompute Ġthe Ġprobability Ġof Ġthe Ġnext Ġword , Ġusing Ġa Ġsampled Ġsoft max ĠB IB REF 18 . ĠFor Ġdownstream Ġtasks , Ġwe Ġuse Ġa Ġlearned Ġlinear Ġweight ing Ġof Ġall Ġlayers Ġin Ġthe Ġenc od ers Ġto Ġobtain Ġa Ġtask - specific Ġm Syn C , Ġfollowing ĠB IB REF 2 . </s> P ret raining Ġwith ĠSh allow ĠSynt actic ĠAnnotations Ġ: :: ĠPret raining ĠModel ĠArchitecture Ġ: :: ĠSt aged Ġparameter Ġupdates </s> J oint ly Ġtraining Ġboth Ġthe Ġsequential Ġenc oder Ġ$ e _{ \ math it Ġ{ seq }} $, Ġand Ġthe Ġsynt actic Ġenc oder Ġ$ e _{ \ math it Ġ{ syn }} $ Ġcan Ġbe Ġexpensive , Ġdue Ġto Ġthe Ġlarge Ġnumber Ġof Ġparameters Ġinvolved . ĠTo Ġreduce Ġcost , Ġwe Ġinitialize Ġour Ġsequential Ġc w rs Ġ$ h $, Ġusing Ġpret rained Ġembed d ings Ġfrom ĠEL Mo - trans former . ĠOnce Ġinitialized Ġas Ġsuch , Ġthe Ġenc oder Ġis Ġfine - tun ed Ġto Ġthe Ġdata Ġlikelihood Ġobjective Ġ( Â§ SEC REF 5 ). ĠThis Ġresults Ġin Ġa Ġstaged Ġparameter Ġupdate , Ġwhich Ġreduces Ġtraining Ġduration Ġby Ġa Ġfactor Ġof Ġ10 Ġin Ġour Ġexperiments . ĠWe Ġdiscuss Ġthe Ġempirical Ġeffect Ġof Ġthis Ġapproach Ġin ĠÂ§ SEC REF 20 . </s> Sh allow ĠSynt actic ĠFeatures </s> Our Ġsecond Ġapproach Ġincorporates Ġshallow Ġsynt actic Ġinformation Ġin Ġdownstream Ġtasks Ġvia Ġtoken - level Ġchunk Ġlabel Ġembed d ings . ĠTask Ġtraining Ġ( and Ġtest ) Ġdata Ġis Ġautomatically Ġchunk ed , Ġand Ġchunk Ġboundary Ġinformation Ġis Ġpassed Ġinto Ġthe Ġtask Ġmodel Ġvia ĠBI OU L Ġencoding Ġof Ġthe Ġlabels . ĠWe Ġadd Ġrandomly Ġinitialized Ġchunk Ġlabel Ġembed d ings Ġto Ġtask - specific Ġinput Ġenc od ers , Ġwhich Ġare Ġthen Ġfine - tun ed Ġfor Ġtask - specific Ġobjectives . ĠThis Ġapproach Ġdoes Ġnot Ġrequire Ġa Ġshallow Ġsynt actic Ġenc oder Ġor Ġchunk Ġannotations Ġfor Ġpret raining Ġc w rs , Ġonly Ġa Ġchunk er . ĠHence , Ġthis Ġcan Ġmore Ġdirectly Ġmeasure Ġthe Ġimpact Ġof Ġshallow Ġsyntax Ġfor Ġa Ġgiven Ġtask . </s> Exper iments </s> Our Ġexperiments Ġevaluate Ġthe Ġeffect Ġof Ġshallow Ġsyntax , Ġvia Ġcontextual ization Ġ( m Syn C , ĠÂ§ SEC REF 2 ) Ġand Ġfeatures Ġ( Â§ SEC REF 3 ). ĠWe Ġprovide Ġcomparisons Ġwith Ġfour Ġbas elines âĢĶ EL Mo - trans former ĠB IB REF 0 , Ġour Ġreim plementation Ġof Ġthe Ġsame , Ġas Ġwell Ġas Ġtwo Ġc wr - free Ġbas elines , Ġwith Ġand Ġwithout Ġshallow Ġsynt actic Ġfeatures . ĠBoth ĠEL Mo - trans former Ġand Ġm Syn C Ġare Ġtrained Ġon Ġthe Ġ1 B Ġword Ġbenchmark Ġcorpus ĠB IB REF 19 ; Ġthe Ġlatter Ġalso Ġemploys Ġchunk Ġannotations Ġ( Â§ SEC REF 2 ). ĠExperimental Ġsettings Ġare Ġdetailed Ġin ĠAppendix ĠÂ§ SEC REF 22 . </s> Exper iments Ġ: :: ĠDown stream ĠTask ĠTransfer </s> We Ġemploy Ġfour Ġtasks Ġto Ġtest Ġthe Ġimpact Ġof Ġshallow Ġsyntax . ĠThe Ġfirst Ġthree , Ġnamely , Ġcoarse Ġand Ġfine - gr ained Ġnamed Ġentity Ġrecognition Ġ( NER ), Ġand Ġconstituency Ġparsing , Ġare Ġspan - based ; Ġthe Ġfourth Ġis Ġa Ġsentence - level Ġsentiment Ġclassification Ġtask . ĠFollowing ĠB IB REF 2 , Ġwe Ġdo Ġnot Ġapply Ġfin et uning Ġto Ġtask - specific Ġarchitectures , Ġallowing Ġus Ġto Ġdo Ġa Ġcontrolled Ġcomparison Ġwith ĠEL Mo . ĠGiven Ġan Ġidentical Ġbase Ġarchitecture Ġacross Ġmodels Ġfor Ġeach Ġtask , Ġwe Ġcan Ġattribute Ġany Ġdifference Ġin Ġperformance Ġto Ġthe Ġincorporation Ġof Ġshallow Ġsyntax Ġor Ġcontextual ization . ĠDetails Ġof Ġdownstream Ġarchitectures Ġare Ġprovided Ġbelow , Ġand Ġoverall Ġdataset Ġstatistics Ġfor Ġall Ġtasks Ġis Ġshown Ġin Ġthe ĠAppendix , ĠTable ĠT AB REF 26 . </s> Exper iments Ġ: :: ĠDown stream ĠTask ĠTransfer Ġ: :: ĠN ER </s> We Ġuse Ġthe ĠEnglish Ġportion Ġof Ġthe ĠCo N LL Ġ2003 Ġdataset ĠB IB REF 20 , Ġwhich Ġprovides Ġnamed Ġentity Ġannotations Ġon Ġnews wire Ġdata Ġacross Ġfour Ġdifferent Ġentity Ġtypes Ġ( PER , ĠLOC , ĠOR G , ĠM ISC ). ĠA Ġbid irection al ĠL ST M - CR F Ġarchitecture ĠB IB REF 14 Ġand Ġa ĠBI OU L Ġtagging Ġscheme Ġwere Ġused . </s> Exper iments Ġ: :: ĠDown stream ĠTask ĠTransfer Ġ: :: ĠFine - gr ained ĠN ER </s> The Ġsame Ġarchitecture Ġand Ġtagging Ġscheme Ġfrom Ġabove Ġis Ġalso Ġused Ġto Ġpredict Ġfine - gr ained Ġentity Ġannotations Ġfrom ĠOnt o Notes Ġ5 . 0 ĠB IB REF 21 . ĠThere Ġare Ġ18 Ġfine - gr ained ĠN ER Ġlabels Ġin Ġthe Ġdataset , Ġincluding Ġregular Ġnamed Ġentit ities Ġas Ġwell Ġas Ġentities Ġsuch Ġas Ġdate , Ġtime Ġand Ġcommon Ġnumerical Ġentries . </s> Exper iments Ġ: :: ĠDown stream ĠTask ĠTransfer Ġ: :: ĠPh rase - st ructure Ġparsing </s> We Ġuse Ġthe Ġstandard ĠPenn ĠTree bank Ġsplits , Ġand Ġadopt Ġthe Ġspan - based Ġmodel Ġfrom ĠB IB REF 22 . ĠFollowing Ġtheir Ġapproach , Ġwe Ġused Ġpredicted Ġpart - of - speech Ġtags Ġfrom Ġthe ĠStanford Ġtag ger ĠB IB REF 23 Ġfor Ġtraining Ġand Ġtesting . ĠAbout Ġ51 % Ġof Ġphrase - sy nt actic Ġconstituents Ġalign Ġexactly Ġwith Ġthe Ġpredicted Ġchunks Ġused , Ġwith Ġa Ġmajority Ġbeing Ġsingle - width Ġnoun Ġphrases . ĠGiven Ġthat Ġthe Ġrule - based Ġprocedure Ġused Ġto Ġobtain Ġchunks Ġonly Ġpropag ates Ġthe Ġphrase Ġtype Ġto Ġthe Ġhead - word Ġand Ġremoves Ġall Ġoverlapping Ġphrases Ġto Ġthe Ġright , Ġthis Ġis Ġexpected . ĠWe Ġdid Ġnot Ġemploy Ġjack - kn if ing Ġto Ġobtain Ġpredicted Ġchunks Ġon ĠPT B Ġdata ; Ġas Ġa Ġresult Ġthere Ġmight Ġbe Ġdifferences Ġin Ġthe Ġquality Ġof Ġshallow Ġsyntax Ġannotations Ġbetween Ġthe Ġtrain Ġand Ġtest Ġportions Ġof Ġthe Ġdata . </s> Exper iments Ġ: :: ĠDown stream ĠTask ĠTransfer Ġ: :: ĠSent iment Ġanalysis </s> We Ġconsider Ġfine - gr ained Ġ( 5 - class ) Ġclassification Ġon ĠStanford ĠSent iment ĠTree bank ĠB IB REF 24 . ĠThe Ġlabels Ġare Ġnegative , Ġsomewhat _ negative , Ġneutral , Ġpositive Ġand Ġsomewhat _ positive . ĠOur Ġmodel Ġwas Ġbased Ġon Ġthe Ġbi att ent ive Ġclassification Ġnetwork ĠB IB REF 25 . ĠWe Ġused Ġall Ġphrase Ġlengths Ġin Ġthe Ġdataset Ġfor Ġtraining , Ġbut Ġtest Ġresults Ġare Ġreported Ġonly Ġon Ġfull Ġsentences , Ġfollowing Ġprior Ġwork . </s> Results Ġare Ġshown Ġin ĠTable ĠT AB REF 12 . ĠCons istent Ġwith Ġprevious Ġfindings , Ġc w rs Ġoffer Ġlarge Ġimprovements Ġacross Ġall Ġtasks . ĠThough Ġhelpful Ġto Ġspan - level Ġtask Ġmodels Ġwithout Ġc w rs , Ġshallow Ġsynt actic Ġfeatures Ġoffer Ġlittle Ġto Ġno Ġbenefit Ġto ĠEL Mo Ġmodels . Ġm Syn C 's Ġperformance Ġis Ġsimilar . ĠThis Ġholds Ġeven Ġfor Ġphrase - st ructure Ġparsing , Ġwhere Ġ( gold ) Ġchunks Ġalign Ġwith Ġsynt actic Ġphrases , Ġindicating Ġthat Ġtask - relevant Ġsignal Ġlearned Ġfrom Ġexposure Ġto Ġshallow Ġsyntax Ġis Ġalready Ġlearned Ġby ĠEL Mo . ĠOn Ġsentiment Ġclassification , Ġchunk Ġfeatures Ġare Ġslightly Ġharmful Ġon Ġaverage Ġ( but Ġvariance Ġis Ġhigh ); Ġm Syn C Ġagain Ġperforms Ġsimilarly Ġto ĠEL Mo - trans former . ĠOverall , Ġthe Ġperformance Ġdifferences Ġacross Ġall Ġtasks Ġare Ġsmall Ġenough Ġto Ġinfer Ġthat Ġshallow Ġsyntax Ġis Ġnot Ġparticularly Ġhelpful Ġwhen Ġusing Ġc w rs . </s> Exper iments Ġ: :: ĠL ingu istic ĠPro bes </s> We Ġfurther Ġanalyze Ġwhether Ġawareness Ġof Ġshallow Ġsyntax Ġcarries Ġover Ġto Ġother Ġlinguistic Ġtasks , Ġvia Ġprobes Ġfrom ĠB IB REF 1 . ĠPro bes Ġare Ġlinear Ġmodels Ġtrained Ġon Ġfrozen Ġc w rs Ġto Ġmake Ġpredictions Ġabout Ġlinguistic Ġ( sy nt actic Ġand Ġsemantic ) Ġproperties Ġof Ġwords Ġand Ġphrases . ĠUnlike ĠÂ§ SEC REF 11 , Ġthere Ġis Ġminimal Ġdownstream Ġtask Ġarchitecture , Ġbringing Ġinto Ġfocus Ġthe Ġtransfer ability Ġof Ġc w rs , Ġas Ġopposed Ġto Ġtask - specific Ġadaptation . </s> Exper iments Ġ: :: ĠL ingu istic ĠPro bes Ġ: :: ĠPro bing ĠT asks </s> The Ġten Ġdifferent Ġprobing Ġtasks Ġwe Ġused Ġinclude ĠCC G Ġsuper tag ging ĠB IB REF 26 , Ġpart - of - speech Ġtagging Ġfrom ĠPT B ĠB IB REF 13 Ġand ĠE WT Ġ( Universal ĠDep ed encies ĠB IB REF 27 ), Ġnamed Ġentity Ġrecognition ĠB IB REF 20 , Ġbase - phrase Ġchunk ing ĠB IB REF 12 , Ġgrammar Ġerror Ġdetection ĠB IB REF 28 , Ġsemantic Ġtagging ĠB IB REF 29 , Ġpre position Ġsupers ense Ġidentification ĠB IB REF 30 , Ġand Ġevent Ġfact uality Ġdetection ĠB IB REF 31 . ĠMet rics Ġand Ġreferences Ġfor Ġeach Ġare Ġsummarized Ġin ĠTable ĠT AB REF 27 . ĠFor Ġmore Ġdetails , Ġplease Ġsee ĠB IB REF 1 . </s> Results Ġin ĠTable ĠT AB REF 13 Ġshow Ġten Ġprobes . ĠAgain , Ġwe Ġsee Ġthe Ġperformance Ġof Ġbaseline ĠEL Mo - trans former Ġand Ġm Syn C Ġare Ġsimilar , Ġwith Ġm Syn C Ġdoing Ġslightly Ġworse Ġon Ġ7 Ġout Ġof Ġ9 Ġtasks . ĠAs Ġwe Ġwould Ġexpect , Ġon Ġthe Ġprobe Ġfor Ġpredicting Ġchunk Ġtags , Ġm Syn C Ġachieves Ġ96 . 9 Ġ$ F _ 1 $ Ġvs . Ġ92 . 2 Ġ$ F _ 1 $ Ġfor ĠEL Mo - trans former , Ġindicating Ġthat Ġm Syn C Ġis Ġindeed Ġencoding Ġshallow Ġsyntax . ĠOverall , Ġthe Ġresults Ġfurther Ġconfirm Ġthat Ġexplicit Ġshallow Ġsyntax Ġdoes Ġnot Ġoffer Ġany Ġbenefits Ġover ĠEL Mo - trans former . </s> Exper iments Ġ: :: ĠEffect Ġof ĠTraining ĠScheme </s> We Ġtest Ġwhether Ġour Ġstaged Ġparameter Ġtraining Ġ( Â§ SEC REF 9 ) Ġis Ġa Ġviable Ġalternative Ġto Ġan Ġend - to - end Ġtraining Ġof Ġboth Ġ$ e _{ \ math it Ġ{ syn }} $ Ġand Ġ$ e _{ \ math it Ġ{ seq }} $. ĠWe Ġmake Ġa Ġfurther Ġdistinction Ġbetween Ġfine - tun ing Ġ$ e _{ \ math it Ġ{ seq }} $ Ġvs . Ġnot Ġupdating Ġit Ġat Ġall Ġafter Ġinitialization Ġ( f rozen ). </s> Down stream Ġvalidation - set Ġ$ F _ 1 $ Ġon Ġfine - gr ained ĠN ER , Ġreported Ġin ĠTable ĠT AB REF 21 , Ġshows Ġthat Ġthe Ġend - to - end Ġstrategy Ġl ags Ġbehind Ġthe Ġothers , Ġperhaps Ġindicating Ġthe Ġneed Ġto Ġtrain Ġlonger Ġthan Ġ10 Ġepoch s . ĠHowever , Ġa Ġsingle Ġepoch Ġon Ġthe Ġ1 B - word Ġbenchmark Ġtakes Ġ36 Ġhours Ġon Ġ2 ĠTesla ĠV 100 s , Ġmaking Ġthis Ġprohib itive . ĠInterestingly , Ġthe Ġfrozen Ġstrategy , Ġwhich Ġtakes Ġthe Ġleast Ġamount Ġof Ġtime Ġto Ġconverge Ġ( 24 Ġhours Ġon Ġ1 ĠTesla ĠV 100 ), Ġalso Ġperforms Ġalmost Ġas Ġwell Ġas Ġfine - tun ing . </s> Conclusion </s> We Ġfind Ġthat Ġexposing Ġc wr - based Ġmodels Ġto Ġshallow Ġsyntax , Ġeither Ġthrough Ġnew Ġc wr Ġlearning Ġarchitectures Ġor Ġexplicit Ġpip el ined Ġfeatures , Ġhas Ġlittle Ġeffect Ġon Ġtheir Ġperformance , Ġacross Ġseveral Ġtasks . ĠL ingu istic Ġprobing Ġalso Ġshows Ġthat Ġc w rs Ġaware Ġof Ġsuch Ġstructures Ġdo Ġnot Ġimprove Ġtask Ġtransfer ability . ĠOur Ġarchitecture Ġand Ġmethods Ġare Ġgeneral Ġenough Ġto Ġbe Ġadapted Ġfor Ġricher Ġinduct ive Ġbiases , Ġsuch Ġas Ġthose Ġgiven Ġby Ġfull Ġsynt actic Ġtrees Ġ( R NN Gs ; ĠB IB REF 32 ), Ġor Ġto Ġdifferent Ġpret raining Ġobjectives , Ġsuch Ġas Ġmasked Ġlanguage Ġmodeling Ġ( BER T ; ĠB IB REF 5 ); Ġwe Ġleave Ġthis Ġpursuit Ġto Ġfuture Ġwork . </s> Supplement al ĠMaterial Ġ: :: ĠHyper param eters Ġ: :: ĠEL Mo - trans former </s> Our Ġbaseline Ġpret raining Ġmodel Ġwas Ġa Ġreim plementation Ġof Ġthat Ġgiven Ġin ĠB IB REF 0 . ĠHyper param eters Ġwere Ġgenerally Ġidentical , Ġbut Ġwe Ġtrained Ġon Ġonly Ġ2 ĠGPUs Ġwith Ġ( up Ġto ) Ġ4 , 000 Ġtokens Ġper Ġbatch . ĠThis Ġdifference Ġin Ġbatch Ġsize Ġmeant Ġwe Ġused Ġ6 , 000 Ġwarm Ġup Ġsteps Ġwith Ġthe Ġlearning Ġrate Ġschedule Ġof ĠB IB REF 16 . </s> Supplement al ĠMaterial Ġ: :: ĠHyper param eters Ġ: :: Ġm Syn C </s> The Ġfunction Ġ$ f _{ seq } $ Ġis Ġidentical Ġto Ġthe Ġ6 - layer Ġbi LM Ġused Ġin ĠEL Mo - trans former . Ġ$ f _{ syn } $, Ġon Ġthe Ġother Ġhand , Ġuses Ġonly Ġ2 Ġlayers . ĠThe Ġlearned Ġembed d ings Ġfor Ġthe Ġchunk Ġlabels Ġhave Ġ128 Ġdimensions Ġand Ġare Ġconc aten ated Ġwith Ġthe Ġtwo Ġboundary Ġ$ h $ Ġof Ġdimension Ġ512 . ĠThus Ġ$ f _{ pro j } $ Ġmaps Ġ$ 1024 Ġ+ Ġ128 $ Ġdimensions Ġto Ġ512 . ĠFurther , Ġwe Ġdid Ġnot Ġperform Ġweight Ġaveraging Ġover Ġseveral Ġcheckpoints . </s> Supplement al ĠMaterial Ġ: :: ĠHyper param eters Ġ: :: ĠSh allow ĠSynt ax </s> The Ġsize Ġof Ġthe Ġshallow Ġsynt actic Ġfeature Ġembed ding Ġwas Ġ50 Ġacross Ġall Ġexperiments , Ġinitialized Ġuniform Ġrandomly . </s> All Ġmodel Ġimplementations Ġare Ġbased Ġon Ġthe ĠAllen N LP Ġlibrary ĠB IB REF 33 .
Introduction The NLP community is revisiting the role of linguistic structure in applications with the advent of contextual word representations (cwrs) derived from pretraining language models on large corpora BIBREF2, BIBREF3, BIBREF4, BIBREF5. Recent work has shown that downstream task performance may benefit from explicitly injecting a syntactic inductive bias into model architectures BIBREF6, even when cwrs are also used BIBREF7. However, high quality linguistic structure annotation at a large scale remains expensive—a trade-off needs to be made between the quality of the annotations and the computational expense of obtaining them. Shallow syntactic structures (BIBREF8; also called chunk sequences) offer a viable middle ground, by providing a flat, non-hierarchical approximation to phrase-syntactic trees (see Fig. FIGREF1 for an example). These structures can be obtained efficiently, and with high accuracy, using sequence labelers. In this paper we consider shallow syntax to be a proxy for linguistic structure. While shallow syntactic chunks are almost as ubiquitous as part-of-speech tags in standard NLP pipelines BIBREF9, their relative merits in the presence of cwrs remain unclear. We investigate the role of these structures using two methods. First, we enhance the ELMo architecture BIBREF0 to allow pretraining on predicted shallow syntactic parses, instead of just raw text, so that contextual embeddings make use of shallow syntactic context (§SECREF2). Our second method involves classical addition of chunk features to cwr-infused architectures for four different downstream tasks (§SECREF3). Shallow syntactic information is obtained automatically using a highly accurate model (97% $F_1$ on standard benchmarks). In both settings, we observe only modest gains on three of the four downstream tasks relative to ELMo-only baselines (§SECREF4). Recent work has probed the knowledge encoded in cwrs and found they capture a surprisingly large amount of syntax BIBREF10, BIBREF1, BIBREF11. We further examine the contextual embeddings obtained from the enhanced architecture and a shallow syntactic context, using black-box probes from BIBREF1. Our analysis indicates that our shallow-syntax-aware contextual embeddings do not transfer to linguistic tasks any more easily than ELMo embeddings (§SECREF18). Overall, our findings show that while shallow syntax can be somewhat useful, ELMo-style pretraining discovers representations which make additional awareness of shallow syntax largely redundant. Pretraining with Shallow Syntactic Annotations We briefly review the shallow syntactic structures used in this work, and then present a model architecture to obtain embeddings from shallow Syntactic Context (mSynC). Pretraining with Shallow Syntactic Annotations ::: Shallow Syntax Base phrase chunking is a cheap sequence-labeling–based alternative to full syntactic parsing, where the sequence consists of non-overlapping labeled segments (Fig. FIGREF1 includes an example.) Full syntactic trees can be converted into such shallow syntactic chunk sequences using a deterministic procedure BIBREF9. BIBREF12 offered a rule-based transformation deriving non-overlapping chunks from phrase-structure trees as found in the Penn Treebank BIBREF13. The procedure percolates some syntactic phrase nodes from a phrase-syntactic tree to the phrase in the leaves of the tree. All overlapping embedded phrases are then removed, and the remainder of the phrase gets the percolated label—this usually corresponds to the head word of the phrase. In order to obtain shallow syntactic annotations on a large corpus, we train a BiLSTM-CRF model BIBREF14, BIBREF15, which achieves 97% $F_1$ on the CoNLL 2000 benchmark test set. The training data is obtained from the CoNLL 2000 shared task BIBREF12, as well as the remaining sections (except §23 and §20) of the Penn Treebank, using the official script for chunk generation. The standard task definition from the shared task includes eleven chunk labels, as shown in Table TABREF4. Pretraining with Shallow Syntactic Annotations ::: Pretraining Objective Traditional language models are estimated to maximize the likelihood of each word $x_i$ given the words that precede it, $p(x_i \mid x_{<i})$. Given a corpus that is annotated with shallow syntax, we propose to condition on both the preceding words and their annotations. We associate with each word $x_i$ three additional variables (denoted $c_i$): the indices of the beginning and end of the last completed chunk before $x_i$, and its label. For example, in Fig. FIGREF8, $c_4=\langle 3, 3, \text{VP}\rangle $ for $x_4=\text{the}$. Chunks, $c$ are only used as conditioning context via $p(x_i \mid x_{<i}, c_{\leqslant i})$; they are not predicted. Because the $c$ labels depend on the entire sentence through the CRF chunker, conditioning each word's probability on any $c_i$ means that our model is, strictly speaking, not a language model, and it can no longer be meaningfully evaluated using perplexity. A right-to-left model is constructed analogously, conditioning on $c_{\geqslant i}$ alongside $x_{>i}$. Following BIBREF2, we use a joint objective maximizing data likelihood objectives in both directions, with shared softmax parameters. Pretraining with Shallow Syntactic Annotations ::: Pretraining Model Architecture Our model uses two encoders: $e_{\mathit {seq}}$ for encoding the sequential history ($x_{<i}$), and $e_{\mathit {syn}}$ for shallow syntactic (chunk) history ($c_{\leqslant i}$). For both, we use transformers BIBREF16, which consist of large feedforward networks equipped with multiheaded self-attention mechanisms. As inputs to $e_{\mathit {seq}}$, we use a context-independent embedding, obtained from a CNN character encoder BIBREF17 for each token $x_i$. The outputs $h_i$ from $e_{\mathit {seq}}$ represent words in context. Next, we build representations for (observed) chunks in the sentence by concatenating a learned embedding for the chunk label with $h$s for the boundaries and applying a linear projection ($f_\mathit {proj}$). The output from $f_\mathit {proj}$ is input to $e_{\mathit {syn}}$, the shallow syntactic encoder, and results in contextualized chunk representations, $g$. Note that the number of chunks in the sentence is less than or equal to the number of tokens. Each $h_i$ is now concatentated with $g_{c_i}$, where $g_{c_i}$ corresponds to $c_i$, the last chunk before position $i$. Finally, the output is given by $\mbox{\textbf {mSynC}}_i = {u}_\mathit {proj}(h_i, g_{c_i}) = W^\top [h_i; g_{c_i}]$, where $W$ is a model parameter. For training, $\mbox{\textbf {mSynC}}_i$ is used to compute the probability of the next word, using a sampled softmax BIBREF18. For downstream tasks, we use a learned linear weighting of all layers in the encoders to obtain a task-specific mSynC, following BIBREF2. Pretraining with Shallow Syntactic Annotations ::: Pretraining Model Architecture ::: Staged parameter updates Jointly training both the sequential encoder $e_{\mathit {seq}}$, and the syntactic encoder $e_{\mathit {syn}}$ can be expensive, due to the large number of parameters involved. To reduce cost, we initialize our sequential cwrs $h$, using pretrained embeddings from ELMo-transformer. Once initialized as such, the encoder is fine-tuned to the data likelihood objective (§SECREF5). This results in a staged parameter update, which reduces training duration by a factor of 10 in our experiments. We discuss the empirical effect of this approach in §SECREF20. Shallow Syntactic Features Our second approach incorporates shallow syntactic information in downstream tasks via token-level chunk label embeddings. Task training (and test) data is automatically chunked, and chunk boundary information is passed into the task model via BIOUL encoding of the labels. We add randomly initialized chunk label embeddings to task-specific input encoders, which are then fine-tuned for task-specific objectives. This approach does not require a shallow syntactic encoder or chunk annotations for pretraining cwrs, only a chunker. Hence, this can more directly measure the impact of shallow syntax for a given task. Experiments Our experiments evaluate the effect of shallow syntax, via contextualization (mSynC, §SECREF2) and features (§SECREF3). We provide comparisons with four baselines—ELMo-transformer BIBREF0, our reimplementation of the same, as well as two cwr-free baselines, with and without shallow syntactic features. Both ELMo-transformer and mSynC are trained on the 1B word benchmark corpus BIBREF19; the latter also employs chunk annotations (§SECREF2). Experimental settings are detailed in Appendix §SECREF22. Experiments ::: Downstream Task Transfer We employ four tasks to test the impact of shallow syntax. The first three, namely, coarse and fine-grained named entity recognition (NER), and constituency parsing, are span-based; the fourth is a sentence-level sentiment classification task. Following BIBREF2, we do not apply finetuning to task-specific architectures, allowing us to do a controlled comparison with ELMo. Given an identical base architecture across models for each task, we can attribute any difference in performance to the incorporation of shallow syntax or contextualization. Details of downstream architectures are provided below, and overall dataset statistics for all tasks is shown in the Appendix, Table TABREF26. Experiments ::: Downstream Task Transfer ::: NER We use the English portion of the CoNLL 2003 dataset BIBREF20, which provides named entity annotations on newswire data across four different entity types (PER, LOC, ORG, MISC). A bidirectional LSTM-CRF architecture BIBREF14 and a BIOUL tagging scheme were used. Experiments ::: Downstream Task Transfer ::: Fine-grained NER The same architecture and tagging scheme from above is also used to predict fine-grained entity annotations from OntoNotes 5.0 BIBREF21. There are 18 fine-grained NER labels in the dataset, including regular named entitities as well as entities such as date, time and common numerical entries. Experiments ::: Downstream Task Transfer ::: Phrase-structure parsing We use the standard Penn Treebank splits, and adopt the span-based model from BIBREF22. Following their approach, we used predicted part-of-speech tags from the Stanford tagger BIBREF23 for training and testing. About 51% of phrase-syntactic constituents align exactly with the predicted chunks used, with a majority being single-width noun phrases. Given that the rule-based procedure used to obtain chunks only propagates the phrase type to the head-word and removes all overlapping phrases to the right, this is expected. We did not employ jack-knifing to obtain predicted chunks on PTB data; as a result there might be differences in the quality of shallow syntax annotations between the train and test portions of the data. Experiments ::: Downstream Task Transfer ::: Sentiment analysis We consider fine-grained (5-class) classification on Stanford Sentiment Treebank BIBREF24. The labels are negative, somewhat_negative, neutral, positive and somewhat_positive. Our model was based on the biattentive classification network BIBREF25. We used all phrase lengths in the dataset for training, but test results are reported only on full sentences, following prior work. Results are shown in Table TABREF12. Consistent with previous findings, cwrs offer large improvements across all tasks. Though helpful to span-level task models without cwrs, shallow syntactic features offer little to no benefit to ELMo models. mSynC's performance is similar. This holds even for phrase-structure parsing, where (gold) chunks align with syntactic phrases, indicating that task-relevant signal learned from exposure to shallow syntax is already learned by ELMo. On sentiment classification, chunk features are slightly harmful on average (but variance is high); mSynC again performs similarly to ELMo-transformer. Overall, the performance differences across all tasks are small enough to infer that shallow syntax is not particularly helpful when using cwrs. Experiments ::: Linguistic Probes We further analyze whether awareness of shallow syntax carries over to other linguistic tasks, via probes from BIBREF1. Probes are linear models trained on frozen cwrs to make predictions about linguistic (syntactic and semantic) properties of words and phrases. Unlike §SECREF11, there is minimal downstream task architecture, bringing into focus the transferability of cwrs, as opposed to task-specific adaptation. Experiments ::: Linguistic Probes ::: Probing Tasks The ten different probing tasks we used include CCG supertagging BIBREF26, part-of-speech tagging from PTB BIBREF13 and EWT (Universal Depedencies BIBREF27), named entity recognition BIBREF20, base-phrase chunking BIBREF12, grammar error detection BIBREF28, semantic tagging BIBREF29, preposition supersense identification BIBREF30, and event factuality detection BIBREF31. Metrics and references for each are summarized in Table TABREF27. For more details, please see BIBREF1. Results in Table TABREF13 show ten probes. Again, we see the performance of baseline ELMo-transformer and mSynC are similar, with mSynC doing slightly worse on 7 out of 9 tasks. As we would expect, on the probe for predicting chunk tags, mSynC achieves 96.9 $F_1$ vs. 92.2 $F_1$ for ELMo-transformer, indicating that mSynC is indeed encoding shallow syntax. Overall, the results further confirm that explicit shallow syntax does not offer any benefits over ELMo-transformer. Experiments ::: Effect of Training Scheme We test whether our staged parameter training (§SECREF9) is a viable alternative to an end-to-end training of both $e_{\mathit {syn}}$ and $e_{\mathit {seq}}$. We make a further distinction between fine-tuning $e_{\mathit {seq}}$ vs. not updating it at all after initialization (frozen). Downstream validation-set $F_1$ on fine-grained NER, reported in Table TABREF21, shows that the end-to-end strategy lags behind the others, perhaps indicating the need to train longer than 10 epochs. However, a single epoch on the 1B-word benchmark takes 36 hours on 2 Tesla V100s, making this prohibitive. Interestingly, the frozen strategy, which takes the least amount of time to converge (24 hours on 1 Tesla V100), also performs almost as well as fine-tuning. Conclusion We find that exposing cwr-based models to shallow syntax, either through new cwr learning architectures or explicit pipelined features, has little effect on their performance, across several tasks. Linguistic probing also shows that cwrs aware of such structures do not improve task transferability. Our architecture and methods are general enough to be adapted for richer inductive biases, such as those given by full syntactic trees (RNNGs; BIBREF32), or to different pretraining objectives, such as masked language modeling (BERT; BIBREF5); we leave this pursuit to future work. Supplemental Material ::: Hyperparameters ::: ELMo-transformer Our baseline pretraining model was a reimplementation of that given in BIBREF0. Hyperparameters were generally identical, but we trained on only 2 GPUs with (up to) 4,000 tokens per batch. This difference in batch size meant we used 6,000 warm up steps with the learning rate schedule of BIBREF16. Supplemental Material ::: Hyperparameters ::: mSynC The function $f_{seq}$ is identical to the 6-layer biLM used in ELMo-transformer. $f_{syn}$, on the other hand, uses only 2 layers. The learned embeddings for the chunk labels have 128 dimensions and are concatenated with the two boundary $h$ of dimension 512. Thus $f_{proj}$ maps $1024 + 128$ dimensions to 512. Further, we did not perform weight averaging over several checkpoints. Supplemental Material ::: Hyperparameters ::: Shallow Syntax The size of the shallow syntactic feature embedding was 50 across all experiments, initialized uniform randomly. All model implementations are based on the AllenNLP library BIBREF33.
Which syntactic features are obtained automatically on downstream task data?
9132d56e26844dc13b3355448d0f14b95bd2178a
Which Ġsynt actic Ġfeatures Ġare Ġobtained Ġautomatically Ġon Ġdownstream Ġtask Ġdata ?
token - level Ġchunk Ġlabel Ġembed d ings , Ġ Ġchunk Ġboundary Ġinformation Ġis Ġpassed Ġinto Ġthe Ġtask Ġmodel Ġvia ĠBI OU L Ġencoding Ġof Ġthe Ġlabels
<s> Which Ġsynt actic Ġfeatures Ġare Ġobtained Ġautomatically Ġon Ġdownstream Ġtask Ġdata ? </s> Introduction </s> The ĠN LP Ġcommunity Ġis Ġrevis iting Ġthe Ġrole Ġof Ġlinguistic Ġstructure Ġin Ġapplications Ġwith Ġthe Ġadvent Ġof Ġcontextual Ġword Ġrepresentations Ġ( c w rs ) Ġderived Ġfrom Ġpret raining Ġlanguage Ġmodels Ġon Ġlarge Ġcorpor a ĠB IB REF 2 , ĠB IB REF 3 , ĠB IB REF 4 , ĠB IB REF 5 . ĠRecent Ġwork Ġhas Ġshown Ġthat Ġdownstream Ġtask Ġperformance Ġmay Ġbenefit Ġfrom Ġexplicitly Ġinjecting Ġa Ġsynt actic Ġinduct ive Ġbias Ġinto Ġmodel Ġarchitectures ĠB IB REF 6 , Ġeven Ġwhen Ġc w rs Ġare Ġalso Ġused ĠB IB REF 7 . ĠHowever , Ġhigh Ġquality Ġlinguistic Ġstructure Ġannotation Ġat Ġa Ġlarge Ġscale Ġremains Ġexpensive âĢĶ a Ġtrade - off Ġneeds Ġto Ġbe Ġmade Ġbetween Ġthe Ġquality Ġof Ġthe Ġannotations Ġand Ġthe Ġcomputational Ġexpense Ġof Ġobtaining Ġthem . ĠSh allow Ġsynt actic Ġstructures Ġ( BI B REF 8 ; Ġalso Ġcalled Ġchunk Ġsequences ) Ġoffer Ġa Ġviable Ġmiddle Ġground , Ġby Ġproviding Ġa Ġflat , Ġnon - h ier arch ical Ġapproximation Ġto Ġphrase - sy nt actic Ġtrees Ġ( see ĠFig . ĠFIG REF 1 Ġfor Ġan Ġexample ). ĠThese Ġstructures Ġcan Ġbe Ġobtained Ġefficiently , Ġand Ġwith Ġhigh Ġaccuracy , Ġusing Ġsequence Ġlabel ers . ĠIn Ġthis Ġpaper Ġwe Ġconsider Ġshallow Ġsyntax Ġto Ġbe Ġa Ġproxy Ġfor Ġlinguistic Ġstructure . </s> While Ġshallow Ġsynt actic Ġchunks Ġare Ġalmost Ġas Ġubiquitous Ġas Ġpart - of - speech Ġtags Ġin Ġstandard ĠN LP Ġpipelines ĠB IB REF 9 , Ġtheir Ġrelative Ġmerits Ġin Ġthe Ġpresence Ġof Ġc w rs Ġremain Ġunclear . ĠWe Ġinvestigate Ġthe Ġrole Ġof Ġthese Ġstructures Ġusing Ġtwo Ġmethods . ĠFirst , Ġwe Ġenhance Ġthe ĠEL Mo Ġarchitecture ĠB IB REF 0 Ġto Ġallow Ġpret raining Ġon Ġpredicted Ġshallow Ġsynt actic Ġpars es , Ġinstead Ġof Ġjust Ġraw Ġtext , Ġso Ġthat Ġcontextual Ġembed d ings Ġmake Ġuse Ġof Ġshallow Ġsynt actic Ġcontext Ġ( Â§ SEC REF 2 ). ĠOur Ġsecond Ġmethod Ġinvolves Ġclassical Ġaddition Ġof Ġchunk Ġfeatures Ġto Ġc wr - inf used Ġarchitectures Ġfor Ġfour Ġdifferent Ġdownstream Ġtasks Ġ( Â§ SEC REF 3 ). ĠSh allow Ġsynt actic Ġinformation Ġis Ġobtained Ġautomatically Ġusing Ġa Ġhighly Ġaccurate Ġmodel Ġ( 97 % Ġ$ F _ 1 $ Ġon Ġstandard Ġbenchmarks ). ĠIn Ġboth Ġsettings , Ġwe Ġobserve Ġonly Ġmodest Ġgains Ġon Ġthree Ġof Ġthe Ġfour Ġdownstream Ġtasks Ġrelative Ġto ĠEL Mo - only Ġbas elines Ġ( Â§ SEC REF 4 ). </s> Recent Ġwork Ġhas Ġprob ed Ġthe Ġknowledge Ġencoded Ġin Ġc w rs Ġand Ġfound Ġthey Ġcapture Ġa Ġsurprisingly Ġlarge Ġamount Ġof Ġsyntax ĠB IB REF 10 , ĠB IB REF 1 , ĠB IB REF 11 . ĠWe Ġfurther Ġexamine Ġthe Ġcontextual Ġembed d ings Ġobtained Ġfrom Ġthe Ġenhanced Ġarchitecture Ġand Ġa Ġshallow Ġsynt actic Ġcontext , Ġusing Ġblack - box Ġprobes Ġfrom ĠB IB REF 1 . ĠOur Ġanalysis Ġindicates Ġthat Ġour Ġshallow - sy ntax - aware Ġcontextual Ġembed d ings Ġdo Ġnot Ġtransfer Ġto Ġlinguistic Ġtasks Ġany Ġmore Ġeasily Ġthan ĠEL Mo Ġembed d ings Ġ( Â§ SEC REF 18 ). </s> Overall , Ġour Ġfindings Ġshow Ġthat Ġwhile Ġshallow Ġsyntax Ġcan Ġbe Ġsomewhat Ġuseful , ĠEL Mo - style Ġpret raining Ġdiscovers Ġrepresentations Ġwhich Ġmake Ġadditional Ġawareness Ġof Ġshallow Ġsyntax Ġlargely Ġredundant . </s> P ret raining Ġwith ĠSh allow ĠSynt actic ĠAnnotations </s> We Ġbriefly Ġreview Ġthe Ġshallow Ġsynt actic Ġstructures Ġused Ġin Ġthis Ġwork , Ġand Ġthen Ġpresent Ġa Ġmodel Ġarchitecture Ġto Ġobtain Ġembed d ings Ġfrom Ġshallow ĠSynt actic ĠContext Ġ( m Syn C ). </s> P ret raining Ġwith ĠSh allow ĠSynt actic ĠAnnotations Ġ: :: ĠSh allow ĠSynt ax </s> Base Ġphrase Ġchunk ing Ġis Ġa Ġcheap Ġsequence - label ing âĢĵ based Ġalternative Ġto Ġfull Ġsynt actic Ġparsing , Ġwhere Ġthe Ġsequence Ġconsists Ġof Ġnon - over l apping Ġlabeled Ġsegments Ġ( Fig . ĠFIG REF 1 Ġincludes Ġan Ġexample .) ĠFull Ġsynt actic Ġtrees Ġcan Ġbe Ġconverted Ġinto Ġsuch Ġshallow Ġsynt actic Ġchunk Ġsequences Ġusing Ġa Ġdeter ministic Ġprocedure ĠB IB REF 9 . ĠB IB REF 12 Ġoffered Ġa Ġrule - based Ġtransformation Ġder iving Ġnon - over l apping Ġchunks Ġfrom Ġphrase - st ructure Ġtrees Ġas Ġfound Ġin Ġthe ĠPenn ĠTree bank ĠB IB REF 13 . ĠThe Ġprocedure Ġper col ates Ġsome Ġsynt actic Ġphrase Ġnodes Ġfrom Ġa Ġphrase - sy nt actic Ġtree Ġto Ġthe Ġphrase Ġin Ġthe Ġleaves Ġof Ġthe Ġtree . ĠAll Ġoverlapping Ġembedded Ġphrases Ġare Ġthen Ġremoved , Ġand Ġthe Ġremainder Ġof Ġthe Ġphrase Ġgets Ġthe Ġper col ated Ġlabel âĢĶ this Ġusually Ġcorresponds Ġto Ġthe Ġhead Ġword Ġof Ġthe Ġphrase . </s> In Ġorder Ġto Ġobtain Ġshallow Ġsynt actic Ġannotations Ġon Ġa Ġlarge Ġcorpus , Ġwe Ġtrain Ġa ĠBi L ST M - CR F Ġmodel ĠB IB REF 14 , ĠB IB REF 15 , Ġwhich Ġachieves Ġ97 % Ġ$ F _ 1 $ Ġon Ġthe ĠCo N LL Ġ2000 Ġbenchmark Ġtest Ġset . ĠThe Ġtraining Ġdata Ġis Ġobtained Ġfrom Ġthe ĠCo N LL Ġ2000 Ġshared Ġtask ĠB IB REF 12 , Ġas Ġwell Ġas Ġthe Ġremaining Ġsections Ġ( except ĠÂ§ 23 Ġand ĠÂ§ 20 ) Ġof Ġthe ĠPenn ĠTree bank , Ġusing Ġthe Ġofficial Ġscript Ġfor Ġchunk Ġgeneration . ĠThe Ġstandard Ġtask Ġdefinition Ġfrom Ġthe Ġshared Ġtask Ġincludes Ġeleven Ġchunk Ġlabels , Ġas Ġshown Ġin ĠTable ĠT AB REF 4 . </s> P ret raining Ġwith ĠSh allow ĠSynt actic ĠAnnotations Ġ: :: ĠPret raining ĠObjective </s> Traditional Ġlanguage Ġmodels Ġare Ġestimated Ġto Ġmaximize Ġthe Ġlikelihood Ġof Ġeach Ġword Ġ$ x _ i $ Ġgiven Ġthe Ġwords Ġthat Ġpreced e Ġit , Ġ$ p ( x _ i Ġ\ mid Ġx _{ < i }) $. ĠGiven Ġa Ġcorpus Ġthat Ġis Ġannot ated Ġwith Ġshallow Ġsyntax , Ġwe Ġpropose Ġto Ġcondition Ġon Ġboth Ġthe Ġpreceding Ġwords Ġand Ġtheir Ġannotations . </s> We Ġassociate Ġwith Ġeach Ġword Ġ$ x _ i $ Ġthree Ġadditional Ġvariables Ġ( den oted Ġ$ c _ i $ ): Ġthe Ġindices Ġof Ġthe Ġbeginning Ġand Ġend Ġof Ġthe Ġlast Ġcompleted Ġchunk Ġbefore Ġ$ x _ i $, Ġand Ġits Ġlabel . ĠFor Ġexample , Ġin ĠFig . ĠFIG REF 8 , Ġ$ c _ 4 = \ l angle Ġ3 , Ġ3 , Ġ\ text { VP }\ r angle Ġ$ Ġfor Ġ$ x _ 4 = \ text { the } $. ĠCh unks , Ġ$ c $ Ġare Ġonly Ġused Ġas Ġconditioning Ġcontext Ġvia Ġ$ p ( x _ i Ġ\ mid Ġx _{ < i }, Ġc _{ \ le q sl ant Ġi }) $ ; Ġthey Ġare Ġnot Ġpredicted . ĠBecause Ġthe Ġ$ c $ Ġlabels Ġdepend Ġon Ġthe Ġentire Ġsentence Ġthrough Ġthe ĠCR F Ġchunk er , Ġconditioning Ġeach Ġword 's Ġprobability Ġon Ġany Ġ$ c _ i $ Ġmeans Ġthat Ġour Ġmodel Ġis , Ġstrictly Ġspeaking , Ġnot Ġa Ġlanguage Ġmodel , Ġand Ġit Ġcan Ġno Ġlonger Ġbe Ġmeaning fully Ġevaluated Ġusing Ġperplex ity . </s> A Ġright - to - left Ġmodel Ġis Ġconstructed Ġanalog ously , Ġconditioning Ġon Ġ$ c _{ \ ge q sl ant Ġi } $ Ġalongside Ġ$ x _{ > i } $. ĠFollowing ĠB IB REF 2 , Ġwe Ġuse Ġa Ġjoint Ġobjective Ġmaximizing Ġdata Ġlikelihood Ġobjectives Ġin Ġboth Ġdirections , Ġwith Ġshared Ġsoft max Ġparameters . </s> P ret raining Ġwith ĠSh allow ĠSynt actic ĠAnnotations Ġ: :: ĠPret raining ĠModel ĠArchitecture </s> Our Ġmodel Ġuses Ġtwo Ġenc od ers : Ġ$ e _{ \ math it Ġ{ seq }} $ Ġfor Ġencoding Ġthe Ġsequential Ġhistory Ġ($ x _{ < i } $ ), Ġand Ġ$ e _{ \ math it Ġ{ syn }} $ Ġfor Ġshallow Ġsynt actic Ġ( ch unk ) Ġhistory Ġ($ c _{ \ le q sl ant Ġi } $ ). ĠFor Ġboth , Ġwe Ġuse Ġtransform ers ĠB IB REF 16 , Ġwhich Ġconsist Ġof Ġlarge Ġfeed forward Ġnetworks Ġequipped Ġwith Ġmulti headed Ġself - att ention Ġmechanisms . </s> As Ġinputs Ġto Ġ$ e _{ \ math it Ġ{ seq }} $, Ġwe Ġuse Ġa Ġcontext - independent Ġembed ding , Ġobtained Ġfrom Ġa ĠCNN Ġcharacter Ġenc oder ĠB IB REF 17 Ġfor Ġeach Ġtoken Ġ$ x _ i $. ĠThe Ġoutputs Ġ$ h _ i $ Ġfrom Ġ$ e _{ \ math it Ġ{ seq }} $ Ġrepresent Ġwords Ġin Ġcontext . </s> Next , Ġwe Ġbuild Ġrepresentations Ġfor Ġ( ob served ) Ġchunks Ġin Ġthe Ġsentence Ġby Ġconc aten ating Ġa Ġlearned Ġembed ding Ġfor Ġthe Ġchunk Ġlabel Ġwith Ġ$ h $ s Ġfor Ġthe Ġboundaries Ġand Ġapplying Ġa Ġlinear Ġprojection Ġ($ f _ \ math it Ġ{ pro j } $ ). ĠThe Ġoutput Ġfrom Ġ$ f _ \ math it Ġ{ pro j } $ Ġis Ġinput Ġto Ġ$ e _{ \ math it Ġ{ syn }} $, Ġthe Ġshallow Ġsynt actic Ġenc oder , Ġand Ġresults Ġin Ġcontextual ized Ġchunk Ġrepresentations , Ġ$ g $. ĠNote Ġthat Ġthe Ġnumber Ġof Ġchunks Ġin Ġthe Ġsentence Ġis Ġless Ġthan Ġor Ġequal Ġto Ġthe Ġnumber Ġof Ġtokens . </s> Each Ġ$ h _ i $ Ġis Ġnow Ġconc at ent ated Ġwith Ġ$ g _{ c _ i } $, Ġwhere Ġ$ g _{ c _ i } $ Ġcorresponds Ġto Ġ$ c _ i $, Ġthe Ġlast Ġchunk Ġbefore Ġposition Ġ$ i $. ĠFinally , Ġthe Ġoutput Ġis Ġgiven Ġby Ġ$\ mb ox {\ text bf Ġ{ m Syn C }} _ i Ġ= Ġ{ u } _ \ math it Ġ{ pro j } ( h _ i , Ġg _{ c _ i }) Ġ= ĠW ^ \ top Ġ[ h _ i ; Ġg _{ c _ i } ] $, Ġwhere Ġ$ W $ Ġis Ġa Ġmodel Ġparameter . ĠFor Ġtraining , Ġ$\ mb ox {\ text bf Ġ{ m Syn C }} _ i $ Ġis Ġused Ġto Ġcompute Ġthe Ġprobability Ġof Ġthe Ġnext Ġword , Ġusing Ġa Ġsampled Ġsoft max ĠB IB REF 18 . ĠFor Ġdownstream Ġtasks , Ġwe Ġuse Ġa Ġlearned Ġlinear Ġweight ing Ġof Ġall Ġlayers Ġin Ġthe Ġenc od ers Ġto Ġobtain Ġa Ġtask - specific Ġm Syn C , Ġfollowing ĠB IB REF 2 . </s> P ret raining Ġwith ĠSh allow ĠSynt actic ĠAnnotations Ġ: :: ĠPret raining ĠModel ĠArchitecture Ġ: :: ĠSt aged Ġparameter Ġupdates </s> J oint ly Ġtraining Ġboth Ġthe Ġsequential Ġenc oder Ġ$ e _{ \ math it Ġ{ seq }} $, Ġand Ġthe Ġsynt actic Ġenc oder Ġ$ e _{ \ math it Ġ{ syn }} $ Ġcan Ġbe Ġexpensive , Ġdue Ġto Ġthe Ġlarge Ġnumber Ġof Ġparameters Ġinvolved . ĠTo Ġreduce Ġcost , Ġwe Ġinitialize Ġour Ġsequential Ġc w rs Ġ$ h $, Ġusing Ġpret rained Ġembed d ings Ġfrom ĠEL Mo - trans former . ĠOnce Ġinitialized Ġas Ġsuch , Ġthe Ġenc oder Ġis Ġfine - tun ed Ġto Ġthe Ġdata Ġlikelihood Ġobjective Ġ( Â§ SEC REF 5 ). ĠThis Ġresults Ġin Ġa Ġstaged Ġparameter Ġupdate , Ġwhich Ġreduces Ġtraining Ġduration Ġby Ġa Ġfactor Ġof Ġ10 Ġin Ġour Ġexperiments . ĠWe Ġdiscuss Ġthe Ġempirical Ġeffect Ġof Ġthis Ġapproach Ġin ĠÂ§ SEC REF 20 . </s> Sh allow ĠSynt actic ĠFeatures </s> Our Ġsecond Ġapproach Ġincorporates Ġshallow Ġsynt actic Ġinformation Ġin Ġdownstream Ġtasks Ġvia Ġtoken - level Ġchunk Ġlabel Ġembed d ings . ĠTask Ġtraining Ġ( and Ġtest ) Ġdata Ġis Ġautomatically Ġchunk ed , Ġand Ġchunk Ġboundary Ġinformation Ġis Ġpassed Ġinto Ġthe Ġtask Ġmodel Ġvia ĠBI OU L Ġencoding Ġof Ġthe Ġlabels . ĠWe Ġadd Ġrandomly Ġinitialized Ġchunk Ġlabel Ġembed d ings Ġto Ġtask - specific Ġinput Ġenc od ers , Ġwhich Ġare Ġthen Ġfine - tun ed Ġfor Ġtask - specific Ġobjectives . ĠThis Ġapproach Ġdoes Ġnot Ġrequire Ġa Ġshallow Ġsynt actic Ġenc oder Ġor Ġchunk Ġannotations Ġfor Ġpret raining Ġc w rs , Ġonly Ġa Ġchunk er . ĠHence , Ġthis Ġcan Ġmore Ġdirectly Ġmeasure Ġthe Ġimpact Ġof Ġshallow Ġsyntax Ġfor Ġa Ġgiven Ġtask . </s> Exper iments </s> Our Ġexperiments Ġevaluate Ġthe Ġeffect Ġof Ġshallow Ġsyntax , Ġvia Ġcontextual ization Ġ( m Syn C , ĠÂ§ SEC REF 2 ) Ġand Ġfeatures Ġ( Â§ SEC REF 3 ). ĠWe Ġprovide Ġcomparisons Ġwith Ġfour Ġbas elines âĢĶ EL Mo - trans former ĠB IB REF 0 , Ġour Ġreim plementation Ġof Ġthe Ġsame , Ġas Ġwell Ġas Ġtwo Ġc wr - free Ġbas elines , Ġwith Ġand Ġwithout Ġshallow Ġsynt actic Ġfeatures . ĠBoth ĠEL Mo - trans former Ġand Ġm Syn C Ġare Ġtrained Ġon Ġthe Ġ1 B Ġword Ġbenchmark Ġcorpus ĠB IB REF 19 ; Ġthe Ġlatter Ġalso Ġemploys Ġchunk Ġannotations Ġ( Â§ SEC REF 2 ). ĠExperimental Ġsettings Ġare Ġdetailed Ġin ĠAppendix ĠÂ§ SEC REF 22 . </s> Exper iments Ġ: :: ĠDown stream ĠTask ĠTransfer </s> We Ġemploy Ġfour Ġtasks Ġto Ġtest Ġthe Ġimpact Ġof Ġshallow Ġsyntax . ĠThe Ġfirst Ġthree , Ġnamely , Ġcoarse Ġand Ġfine - gr ained Ġnamed Ġentity Ġrecognition Ġ( NER ), Ġand Ġconstituency Ġparsing , Ġare Ġspan - based ; Ġthe Ġfourth Ġis Ġa Ġsentence - level Ġsentiment Ġclassification Ġtask . ĠFollowing ĠB IB REF 2 , Ġwe Ġdo Ġnot Ġapply Ġfin et uning Ġto Ġtask - specific Ġarchitectures , Ġallowing Ġus Ġto Ġdo Ġa Ġcontrolled Ġcomparison Ġwith ĠEL Mo . ĠGiven Ġan Ġidentical Ġbase Ġarchitecture Ġacross Ġmodels Ġfor Ġeach Ġtask , Ġwe Ġcan Ġattribute Ġany Ġdifference Ġin Ġperformance Ġto Ġthe Ġincorporation Ġof Ġshallow Ġsyntax Ġor Ġcontextual ization . ĠDetails Ġof Ġdownstream Ġarchitectures Ġare Ġprovided Ġbelow , Ġand Ġoverall Ġdataset Ġstatistics Ġfor Ġall Ġtasks Ġis Ġshown Ġin Ġthe ĠAppendix , ĠTable ĠT AB REF 26 . </s> Exper iments Ġ: :: ĠDown stream ĠTask ĠTransfer Ġ: :: ĠN ER </s> We Ġuse Ġthe ĠEnglish Ġportion Ġof Ġthe ĠCo N LL Ġ2003 Ġdataset ĠB IB REF 20 , Ġwhich Ġprovides Ġnamed Ġentity Ġannotations Ġon Ġnews wire Ġdata Ġacross Ġfour Ġdifferent Ġentity Ġtypes Ġ( PER , ĠLOC , ĠOR G , ĠM ISC ). ĠA Ġbid irection al ĠL ST M - CR F Ġarchitecture ĠB IB REF 14 Ġand Ġa ĠBI OU L Ġtagging Ġscheme Ġwere Ġused . </s> Exper iments Ġ: :: ĠDown stream ĠTask ĠTransfer Ġ: :: ĠFine - gr ained ĠN ER </s> The Ġsame Ġarchitecture Ġand Ġtagging Ġscheme Ġfrom Ġabove Ġis Ġalso Ġused Ġto Ġpredict Ġfine - gr ained Ġentity Ġannotations Ġfrom ĠOnt o Notes Ġ5 . 0 ĠB IB REF 21 . ĠThere Ġare Ġ18 Ġfine - gr ained ĠN ER Ġlabels Ġin Ġthe Ġdataset , Ġincluding Ġregular Ġnamed Ġentit ities Ġas Ġwell Ġas Ġentities Ġsuch Ġas Ġdate , Ġtime Ġand Ġcommon Ġnumerical Ġentries . </s> Exper iments Ġ: :: ĠDown stream ĠTask ĠTransfer Ġ: :: ĠPh rase - st ructure Ġparsing </s> We Ġuse Ġthe Ġstandard ĠPenn ĠTree bank Ġsplits , Ġand Ġadopt Ġthe Ġspan - based Ġmodel Ġfrom ĠB IB REF 22 . ĠFollowing Ġtheir Ġapproach , Ġwe Ġused Ġpredicted Ġpart - of - speech Ġtags Ġfrom Ġthe ĠStanford Ġtag ger ĠB IB REF 23 Ġfor Ġtraining Ġand Ġtesting . ĠAbout Ġ51 % Ġof Ġphrase - sy nt actic Ġconstituents Ġalign Ġexactly Ġwith Ġthe Ġpredicted Ġchunks Ġused , Ġwith Ġa Ġmajority Ġbeing Ġsingle - width Ġnoun Ġphrases . ĠGiven Ġthat Ġthe Ġrule - based Ġprocedure Ġused Ġto Ġobtain Ġchunks Ġonly Ġpropag ates Ġthe Ġphrase Ġtype Ġto Ġthe Ġhead - word Ġand Ġremoves Ġall Ġoverlapping Ġphrases Ġto Ġthe Ġright , Ġthis Ġis Ġexpected . ĠWe Ġdid Ġnot Ġemploy Ġjack - kn if ing Ġto Ġobtain Ġpredicted Ġchunks Ġon ĠPT B Ġdata ; Ġas Ġa Ġresult Ġthere Ġmight Ġbe Ġdifferences Ġin Ġthe Ġquality Ġof Ġshallow Ġsyntax Ġannotations Ġbetween Ġthe Ġtrain Ġand Ġtest Ġportions Ġof Ġthe Ġdata . </s> Exper iments Ġ: :: ĠDown stream ĠTask ĠTransfer Ġ: :: ĠSent iment Ġanalysis </s> We Ġconsider Ġfine - gr ained Ġ( 5 - class ) Ġclassification Ġon ĠStanford ĠSent iment ĠTree bank ĠB IB REF 24 . ĠThe Ġlabels Ġare Ġnegative , Ġsomewhat _ negative , Ġneutral , Ġpositive Ġand Ġsomewhat _ positive . ĠOur Ġmodel Ġwas Ġbased Ġon Ġthe Ġbi att ent ive Ġclassification Ġnetwork ĠB IB REF 25 . ĠWe Ġused Ġall Ġphrase Ġlengths Ġin Ġthe Ġdataset Ġfor Ġtraining , Ġbut Ġtest Ġresults Ġare Ġreported Ġonly Ġon Ġfull Ġsentences , Ġfollowing Ġprior Ġwork . </s> Results Ġare Ġshown Ġin ĠTable ĠT AB REF 12 . ĠCons istent Ġwith Ġprevious Ġfindings , Ġc w rs Ġoffer Ġlarge Ġimprovements Ġacross Ġall Ġtasks . ĠThough Ġhelpful Ġto Ġspan - level Ġtask Ġmodels Ġwithout Ġc w rs , Ġshallow Ġsynt actic Ġfeatures Ġoffer Ġlittle Ġto Ġno Ġbenefit Ġto ĠEL Mo Ġmodels . Ġm Syn C 's Ġperformance Ġis Ġsimilar . ĠThis Ġholds Ġeven Ġfor Ġphrase - st ructure Ġparsing , Ġwhere Ġ( gold ) Ġchunks Ġalign Ġwith Ġsynt actic Ġphrases , Ġindicating Ġthat Ġtask - relevant Ġsignal Ġlearned Ġfrom Ġexposure Ġto Ġshallow Ġsyntax Ġis Ġalready Ġlearned Ġby ĠEL Mo . ĠOn Ġsentiment Ġclassification , Ġchunk Ġfeatures Ġare Ġslightly Ġharmful Ġon Ġaverage Ġ( but Ġvariance Ġis Ġhigh ); Ġm Syn C Ġagain Ġperforms Ġsimilarly Ġto ĠEL Mo - trans former . ĠOverall , Ġthe Ġperformance Ġdifferences Ġacross Ġall Ġtasks Ġare Ġsmall Ġenough Ġto Ġinfer Ġthat Ġshallow Ġsyntax Ġis Ġnot Ġparticularly Ġhelpful Ġwhen Ġusing Ġc w rs . </s> Exper iments Ġ: :: ĠL ingu istic ĠPro bes </s> We Ġfurther Ġanalyze Ġwhether Ġawareness Ġof Ġshallow Ġsyntax Ġcarries Ġover Ġto Ġother Ġlinguistic Ġtasks , Ġvia Ġprobes Ġfrom ĠB IB REF 1 . ĠPro bes Ġare Ġlinear Ġmodels Ġtrained Ġon Ġfrozen Ġc w rs Ġto Ġmake Ġpredictions Ġabout Ġlinguistic Ġ( sy nt actic Ġand Ġsemantic ) Ġproperties Ġof Ġwords Ġand Ġphrases . ĠUnlike ĠÂ§ SEC REF 11 , Ġthere Ġis Ġminimal Ġdownstream Ġtask Ġarchitecture , Ġbringing Ġinto Ġfocus Ġthe Ġtransfer ability Ġof Ġc w rs , Ġas Ġopposed Ġto Ġtask - specific Ġadaptation . </s> Exper iments Ġ: :: ĠL ingu istic ĠPro bes Ġ: :: ĠPro bing ĠT asks </s> The Ġten Ġdifferent Ġprobing Ġtasks Ġwe Ġused Ġinclude ĠCC G Ġsuper tag ging ĠB IB REF 26 , Ġpart - of - speech Ġtagging Ġfrom ĠPT B ĠB IB REF 13 Ġand ĠE WT Ġ( Universal ĠDep ed encies ĠB IB REF 27 ), Ġnamed Ġentity Ġrecognition ĠB IB REF 20 , Ġbase - phrase Ġchunk ing ĠB IB REF 12 , Ġgrammar Ġerror Ġdetection ĠB IB REF 28 , Ġsemantic Ġtagging ĠB IB REF 29 , Ġpre position Ġsupers ense Ġidentification ĠB IB REF 30 , Ġand Ġevent Ġfact uality Ġdetection ĠB IB REF 31 . ĠMet rics Ġand Ġreferences Ġfor Ġeach Ġare Ġsummarized Ġin ĠTable ĠT AB REF 27 . ĠFor Ġmore Ġdetails , Ġplease Ġsee ĠB IB REF 1 . </s> Results Ġin ĠTable ĠT AB REF 13 Ġshow Ġten Ġprobes . ĠAgain , Ġwe Ġsee Ġthe Ġperformance Ġof Ġbaseline ĠEL Mo - trans former Ġand Ġm Syn C Ġare Ġsimilar , Ġwith Ġm Syn C Ġdoing Ġslightly Ġworse Ġon Ġ7 Ġout Ġof Ġ9 Ġtasks . ĠAs Ġwe Ġwould Ġexpect , Ġon Ġthe Ġprobe Ġfor Ġpredicting Ġchunk Ġtags , Ġm Syn C Ġachieves Ġ96 . 9 Ġ$ F _ 1 $ Ġvs . Ġ92 . 2 Ġ$ F _ 1 $ Ġfor ĠEL Mo - trans former , Ġindicating Ġthat Ġm Syn C Ġis Ġindeed Ġencoding Ġshallow Ġsyntax . ĠOverall , Ġthe Ġresults Ġfurther Ġconfirm Ġthat Ġexplicit Ġshallow Ġsyntax Ġdoes Ġnot Ġoffer Ġany Ġbenefits Ġover ĠEL Mo - trans former . </s> Exper iments Ġ: :: ĠEffect Ġof ĠTraining ĠScheme </s> We Ġtest Ġwhether Ġour Ġstaged Ġparameter Ġtraining Ġ( Â§ SEC REF 9 ) Ġis Ġa Ġviable Ġalternative Ġto Ġan Ġend - to - end Ġtraining Ġof Ġboth Ġ$ e _{ \ math it Ġ{ syn }} $ Ġand Ġ$ e _{ \ math it Ġ{ seq }} $. ĠWe Ġmake Ġa Ġfurther Ġdistinction Ġbetween Ġfine - tun ing Ġ$ e _{ \ math it Ġ{ seq }} $ Ġvs . Ġnot Ġupdating Ġit Ġat Ġall Ġafter Ġinitialization Ġ( f rozen ). </s> Down stream Ġvalidation - set Ġ$ F _ 1 $ Ġon Ġfine - gr ained ĠN ER , Ġreported Ġin ĠTable ĠT AB REF 21 , Ġshows Ġthat Ġthe Ġend - to - end Ġstrategy Ġl ags Ġbehind Ġthe Ġothers , Ġperhaps Ġindicating Ġthe Ġneed Ġto Ġtrain Ġlonger Ġthan Ġ10 Ġepoch s . ĠHowever , Ġa Ġsingle Ġepoch Ġon Ġthe Ġ1 B - word Ġbenchmark Ġtakes Ġ36 Ġhours Ġon Ġ2 ĠTesla ĠV 100 s , Ġmaking Ġthis Ġprohib itive . ĠInterestingly , Ġthe Ġfrozen Ġstrategy , Ġwhich Ġtakes Ġthe Ġleast Ġamount Ġof Ġtime Ġto Ġconverge Ġ( 24 Ġhours Ġon Ġ1 ĠTesla ĠV 100 ), Ġalso Ġperforms Ġalmost Ġas Ġwell Ġas Ġfine - tun ing . </s> Conclusion </s> We Ġfind Ġthat Ġexposing Ġc wr - based Ġmodels Ġto Ġshallow Ġsyntax , Ġeither Ġthrough Ġnew Ġc wr Ġlearning Ġarchitectures Ġor Ġexplicit Ġpip el ined Ġfeatures , Ġhas Ġlittle Ġeffect Ġon Ġtheir Ġperformance , Ġacross Ġseveral Ġtasks . ĠL ingu istic Ġprobing Ġalso Ġshows Ġthat Ġc w rs Ġaware Ġof Ġsuch Ġstructures Ġdo Ġnot Ġimprove Ġtask Ġtransfer ability . ĠOur Ġarchitecture Ġand Ġmethods Ġare Ġgeneral Ġenough Ġto Ġbe Ġadapted Ġfor Ġricher Ġinduct ive Ġbiases , Ġsuch Ġas Ġthose Ġgiven Ġby Ġfull Ġsynt actic Ġtrees Ġ( R NN Gs ; ĠB IB REF 32 ), Ġor Ġto Ġdifferent Ġpret raining Ġobjectives , Ġsuch Ġas Ġmasked Ġlanguage Ġmodeling Ġ( BER T ; ĠB IB REF 5 ); Ġwe Ġleave Ġthis Ġpursuit Ġto Ġfuture Ġwork . </s> Supplement al ĠMaterial Ġ: :: ĠHyper param eters Ġ: :: ĠEL Mo - trans former </s> Our Ġbaseline Ġpret raining Ġmodel Ġwas Ġa Ġreim plementation Ġof Ġthat Ġgiven Ġin ĠB IB REF 0 . ĠHyper param eters Ġwere Ġgenerally Ġidentical , Ġbut Ġwe Ġtrained Ġon Ġonly Ġ2 ĠGPUs Ġwith Ġ( up Ġto ) Ġ4 , 000 Ġtokens Ġper Ġbatch . ĠThis Ġdifference Ġin Ġbatch Ġsize Ġmeant Ġwe Ġused Ġ6 , 000 Ġwarm Ġup Ġsteps Ġwith Ġthe Ġlearning Ġrate Ġschedule Ġof ĠB IB REF 16 . </s> Supplement al ĠMaterial Ġ: :: ĠHyper param eters Ġ: :: Ġm Syn C </s> The Ġfunction Ġ$ f _{ seq } $ Ġis Ġidentical Ġto Ġthe Ġ6 - layer Ġbi LM Ġused Ġin ĠEL Mo - trans former . Ġ$ f _{ syn } $, Ġon Ġthe Ġother Ġhand , Ġuses Ġonly Ġ2 Ġlayers . ĠThe Ġlearned Ġembed d ings Ġfor Ġthe Ġchunk Ġlabels Ġhave Ġ128 Ġdimensions Ġand Ġare Ġconc aten ated Ġwith Ġthe Ġtwo Ġboundary Ġ$ h $ Ġof Ġdimension Ġ512 . ĠThus Ġ$ f _{ pro j } $ Ġmaps Ġ$ 1024 Ġ+ Ġ128 $ Ġdimensions Ġto Ġ512 . ĠFurther , Ġwe Ġdid Ġnot Ġperform Ġweight Ġaveraging Ġover Ġseveral Ġcheckpoints . </s> Supplement al ĠMaterial Ġ: :: ĠHyper param eters Ġ: :: ĠSh allow ĠSynt ax </s> The Ġsize Ġof Ġthe Ġshallow Ġsynt actic Ġfeature Ġembed ding Ġwas Ġ50 Ġacross Ġall Ġexperiments , Ġinitialized Ġuniform Ġrandomly . </s> All Ġmodel Ġimplementations Ġare Ġbased Ġon Ġthe ĠAllen N LP Ġlibrary ĠB IB REF 33 .