The task is set up to mimic (albeit, in an oversimplified manner) the input-output symbol alignments and local syntactic properties that models must learn in many natural language tasks, such as translation, tagging and summarization.
EXTRACTIVE
Real world NLP tasks are complex, and as such, it can be difficult to precisely define what a model should and should not learn during training. As done in previous work BIBREF8 , BIBREF9 , we ease analysis by looking at a simple formal task. The task is set up to mimic (albeit, in an oversimplified manner) the input-output symbol alignments and local syntactic properties that models must learn in many natural language tasks, such as translation, tagging and summarization.
1805.01445
Introduction </s> It Ġis Ġwell Ġknown Ġthat Ġlanguage Ġhas Ġcertain Ġstructural Ġproperties Ġwhich Ġallows Ġnatural Ġlanguage Ġspeakers Ġto Ġmake ĠâĢ ľ inf inite Ġuse Ġof Ġfinite Ġmeans " ĠB IB REF 3 Ġ. ĠThis Ġstructure Ġallows Ġus Ġto Ġgeneral ize Ġbeyond Ġthe Ġtypical Ġmachine Ġlearning Ġdefinition Ġof Ġgeneral ization ĠB IB REF 4 Ġ( which Ġconsiders Ġperformance Ġon Ġthe Ġdistribution Ġthat Ġgenerated Ġthe Ġtraining Ġset ), Ġpermitting Ġthe Ġunderstanding Ġof Ġany Ġutter ance Ġsharing Ġthe Ġsame Ġstructure , Ġregardless Ġof Ġprobability . ĠFor Ġexample , Ġsentences Ġof Ġlength Ġ100 Ġtypically Ġdo Ġnot Ġappear Ġin Ġnatural Ġtext Ġor Ġspeech Ġ( our Ġpersonal Ġ' training Ġset '), Ġbut Ġcan Ġbe Ġunderstood Ġregardless Ġdue Ġto Ġtheir Ġstructure . ĠWe Ġrefer Ġto Ġthis Ġnotion Ġas Ġlinguistic Ġgeneral ization Ġ. </s> Many Ġproblems Ġin ĠN LP Ġare Ġtreated Ġas Ġsequence Ġto Ġsequence Ġtasks Ġwith Ġsolutions Ġbuilt Ġon Ġseq 2 seq - att ention Ġbased Ġmodels . ĠWhile Ġthese Ġmodels Ġperform Ġvery Ġwell Ġon Ġstandard Ġdatasets Ġand Ġalso Ġappear Ġto Ġcapture Ġsome Ġlinguistic Ġstructure ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, Ġthey Ġalso Ġcan Ġbe Ġquite Ġbrittle , Ġtypically Ġbreaking Ġon Ġun character istic Ġinputs ĠB IB REF 8 Ġ, ĠB IB REF 1 Ġ, Ġindicating Ġthat Ġthe Ġextent Ġof Ġlinguistic Ġgeneral ization Ġthese Ġmodels Ġachieve Ġis Ġstill Ġsomewhat Ġlacking . </s> Due Ġto Ġthe Ġhigh Ġcapacity Ġof Ġthese Ġmodels , Ġit Ġis Ġnot Ġunreasonable Ġto Ġexpect Ġthem Ġto Ġlearn Ġsome Ġstructure Ġfrom Ġthe Ġdata . ĠHowever , Ġlearning Ġstructure Ġis Ġnot Ġa Ġsufficient Ġcondition Ġto Ġachieving Ġlinguistic Ġgeneral ization . ĠIf Ġthis Ġstructure Ġis Ġto Ġbe Ġusable Ġon Ġdata Ġoutside Ġthe Ġtraining Ġdistribution , Ġthe Ġmodel Ġmust Ġlearn Ġthe Ġstructure Ġwithout Ġadditionally Ġlearning Ġ( over fitting Ġon ) Ġpatterns Ġspecific Ġto Ġthe Ġtraining Ġdata . ĠOne Ġmay Ġhope , Ġgiven Ġthe Ġright Ġhyper param eter Ġconfiguration Ġand Ġregular ization , Ġthat Ġa Ġmodel Ġconver ges Ġto Ġa Ġsolution Ġthat Ġcaptures Ġthe Ġreusable Ġstructure Ġwithout Ġover fitting Ġtoo Ġmuch Ġon Ġthe Ġtraining Ġset . ĠWhile Ġthis Ġsolution Ġexists Ġin Ġtheory , Ġin Ġpractice , Ġit Ġmay Ġbe Ġdifficult Ġto Ġfind . </s> In Ġthis Ġwork , Ġwe Ġlook Ġat Ġthe Ġfeasibility Ġof Ġtraining Ġand Ġtuning Ġseq 2 seq - att ention Ġmodels Ġtowards Ġa Ġsolution Ġthat Ġgeneral izes Ġin Ġthis Ġlinguistic Ġsense . ĠIn Ġparticular , Ġwe Ġtrain Ġmodels Ġon Ġa Ġsymbol Ġreplacement Ġtask Ġwith Ġa Ġwell Ġdefined Ġgeneral izable Ġstructure . ĠThe Ġtask Ġis Ġsimple Ġenough Ġthat Ġall Ġmodels Ġachieve Ġnear Ġperfect Ġaccuracy Ġon Ġthe Ġstandard Ġtest Ġset , Ġi . e ., Ġwhere Ġthe Ġinputs Ġare Ġdrawn Ġfrom Ġthe Ġsame Ġdistribution Ġas Ġthat Ġof Ġthe Ġtraining Ġset . ĠWe Ġthen Ġtest Ġthese Ġmodels Ġfor Ġlinguistic Ġgeneral ization Ġby Ġcreating Ġtest Ġsets Ġof Ġun character istic Ġinputs , Ġi . e ., Ġinputs Ġthat Ġare Ġnot Ġtypical Ġin Ġthe Ġtraining Ġdistribution Ġbut Ġstill Ġsol vable Ġgiven Ġthat Ġthe Ġgeneral izable Ġstructure Ġwas Ġlearned . ĠOur Ġresults Ġindicate Ġthat Ġgeneral ization Ġis Ġhighly Ġsensitive ; Ġsuch Ġthat Ġeven Ġchanges Ġin Ġthe Ġrandom Ġseed Ġcan Ġdrastically Ġaffect Ġthe Ġability Ġto Ġgeneral ize . ĠThis Ġdependence Ġon Ġan Ġelement Ġthat Ġis Ġnot Ġ( or Ġideally Ġshould Ġnot Ġbe ) Ġa Ġhyper param eter Ġsuggests Ġthat Ġthe Ġline Ġbetween Ġgeneral ization Ġand Ġfailure Ġis Ġquite Ġfine , Ġand Ġmay Ġnot Ġbe Ġfeasible Ġto Ġreach Ġsimply Ġby Ġhyper param eter Ġtuning Ġalone . </s> General ization Ġin Ġa ĠSymbol ĠRew rit ing ĠTask </s> Real Ġworld ĠN LP Ġtasks Ġare Ġcomplex , Ġand Ġas Ġsuch , Ġit Ġcan Ġbe Ġdifficult Ġto Ġprecisely Ġdefine Ġwhat Ġa Ġmodel Ġshould Ġand Ġshould Ġnot Ġlearn Ġduring Ġtraining . ĠAs Ġdone Ġin Ġprevious Ġwork ĠB IB REF 8 Ġ, ĠB IB REF 9 Ġ, Ġwe Ġease Ġanalysis Ġby Ġlooking Ġat Ġa Ġsimple Ġformal Ġtask . ĠThe Ġtask Ġis Ġset Ġup Ġto Ġmimic Ġ( albeit , Ġin Ġan Ġovers impl ified Ġmanner ) Ġthe Ġinput - output Ġsymbol Ġalign ments Ġand Ġlocal Ġsynt actic Ġproperties Ġthat Ġmodels Ġmust Ġlearn Ġin Ġmany Ġnatural Ġlanguage Ġtasks , Ġsuch Ġas Ġtranslation , Ġtagging Ġand Ġsummar ization . </s> The Ġtask Ġis Ġdefined Ġover Ġsequences Ġof Ġsymbols , Ġ$\ l brace Ġx _ 1 ,... x _ n Ġ| Ġx _ i Ġ\ in ĠX \ r brace Ġ$ Ġ, Ġwhere Ġ$ X $ Ġis Ġthe Ġinput Ġalphabet . ĠEach Ġsymbol Ġ$ x Ġ\ in ĠX $ Ġis Ġuniquely Ġassociated Ġwith Ġits Ġown Ġoutput Ġalphabet Ġ$ Y _ x $ Ġ. ĠOutput Ġis Ġcreated Ġby Ġtaking Ġeach Ġindividual Ġsymbol Ġ$ x _ i $ Ġin Ġthe Ġsequence Ġand Ġrewriting Ġit Ġas Ġany Ġsequence Ġof Ġ$ k $ Ġsymbols Ġfrom Ġ$ Y _{ x _ i } $ Ġ. ĠTo Ġdo Ġthe Ġtask , Ġthe Ġmodel Ġmust Ġlearn Ġalign ments Ġbetween Ġthe Ġinput Ġand Ġoutput Ġsymbols , Ġand Ġpreserve Ġthe Ġsimple Ġlocal Ġsynt actic Ġconditions Ġ( every Ġgroup Ġof Ġ$ k $ Ġsymbols Ġmust Ġcome Ġfrom Ġthe Ġsame Ġinput Ġalphabet Ġ$ Y _ x $ Ġ). </s> As Ġan Ġexample , Ġlet Ġ$ X Ġ= Ġ\ l brace ĠA , B \ r brace Ġ$ Ġ, Ġ$ Y _ A Ġ= Ġ\ l brace Ġ\ text { $ a _{ 1 }, Ġa _{ 2 } $, Ġ$ a _ 3 $ }\ r brace Ġ$ Ġ, Ġ$ Y _ B Ġ= Ġ\ l brace Ġ\ text { $ b _ 1 , Ġb _ 2 , $ Ġ$ b _ 3 $ }\ r brace Ġ$ Ġ, Ġand Ġ$ k = 3 $ Ġ. ĠEach Ġ$ a _ i $ Ġand Ġ$ b _ i $ Ġhas Ġ2 Ġpossible Ġvalues , Ġ$ a _{ i 1 } $ Ġor Ġ$ a _{ i 2 } $ Ġand Ġ$ b _{ i 1 } $ Ġor Ġ$ b _{ i 2 } $ Ġrespectively . ĠThus , Ġmapping Ġan Ġinput Ġsymbol Ġto Ġ48 Ġ( Ġ$ Y _ A Ġ= Ġ\ l brace Ġ\ text { $ a _{ 1 }, Ġa _{ 2 } $, Ġ$ a _ 3 $ }\ r brace Ġ$ 0 Ġ) Ġpossible Ġperm utations . ĠA Ġpossible Ġvalid Ġoutput Ġfor Ġthe Ġinput Ġ$ Y _ A Ġ= Ġ\ l brace Ġ\ text { $ a _{ 1 }, Ġa _{ 2 } $, Ġ$ a _ 3 $ }\ r brace Ġ$ 1 Ġis Ġ$ Y _ A Ġ= Ġ\ l brace Ġ\ text { $ a _{ 1 }, Ġa _{ 2 } $, Ġ$ a _ 3 $ }\ r brace Ġ$ 2 Ġ. ĠNote Ġthat Ġsuch Ġvalid Ġstrings Ġare Ġselected Ġat Ġrandom Ġwhen Ġgenerating Ġthe Ġdataset . ĠWe Ġallow Ġthis Ġsto ch astic ity Ġin Ġthe Ġoutputs Ġin Ġorder Ġto Ġprevent Ġthe Ġmodel Ġfrom Ġresort ing Ġto Ġpure Ġmemor ization . ĠFor Ġour Ġtask , Ġ$ Y _ A Ġ= Ġ\ l brace Ġ\ text { $ a _{ 1 }, Ġa _{ 2 } $, Ġ$ a _ 3 $ }\ r brace Ġ$ 3 Ġand Ġeach Ġ$ Y _ A Ġ= Ġ\ l brace Ġ\ text { $ a _{ 1 }, Ġa _{ 2 } $, Ġ$ a _ 3 $ }\ r brace Ġ$ 4 Ġhas Ġa Ġcorresponding Ġoutput Ġalphabet Ġ$ Y _ A Ġ= Ġ\ l brace Ġ\ text { $ a _{ 1 }, Ġa _{ 2 } $, Ġ$ a _ 3 $ }\ r brace Ġ$ 5 Ġof Ġsize Ġ16 . </s> Model Ġand ĠTraining ĠDetails </s> The Ġmodels Ġwe Ġuse Ġare Ġsingle Ġlayer , Ġun id irection al , Ġseq 2 seq ĠL ST Ms ĠB IB REF 10 Ġwith Ġbil in ear Ġattention ĠB IB REF 11 Ġand Ġtrained Ġwith Ġvanilla ĠSG D . ĠTo Ġdetermine Ġthe Ġepoch Ġto Ġstop Ġtraining Ġat , Ġwe Ġcreate Ġa Ġvalidation Ġset Ġof Ġ2000 Ġsamples Ġwith Ġthe Ġsame Ġcharacteristics Ġas Ġthe Ġtraining Ġset , Ġi . e ., Ġof Ġlength Ġ5 - 10 Ġwith Ġno Ġrepeated Ġsymbols . ĠTraining Ġis Ġstopped Ġonce Ġaccuracy Ġon Ġthe Ġvalidation Ġset Ġeither Ġdecreases Ġor Ġremains Ġunchanged . ĠThe Ġsize Ġof Ġthe Ġhidden Ġstate Ġand Ġembed d ings Ġwere Ġchosen Ġsuch Ġthat Ġthey Ġwere Ġas Ġsmall Ġas Ġpossible Ġwithout Ġreducing Ġvalidation Ġaccuracy , Ġgiving Ġa Ġsize Ġof Ġ32 . </s> T uning Ġhyper param eters Ġis Ġoften Ġdone Ġon Ġa Ġvalidation Ġset Ġdrawn Ġfrom Ġthe Ġsame Ġdistribution Ġas Ġthe Ġtraining Ġset Ġ( as Ġopposed Ġto Ġvalid ating Ġon Ġdata Ġfrom Ġa Ġdifferent Ġdistribution ) Ġwhich Ġmotivated Ġour Ġinitial Ġdecision Ġto Ġuse Ġa Ġvalidation Ġset Ġof Ġcharacteristic Ġinputs Ġto Ġdecide Ġthe Ġepoch Ġto Ġstop Ġat . ĠHowever , Ġwe Ġnoticed Ġonly Ġsmall Ġvariation Ġin Ġthe Ġvalidation Ġperformance Ġupon Ġusing Ġdifferent Ġlearning Ġrates Ġand Ġdrop out Ġprobabilities Ġ( where Ġdrop out Ġwas Ġapplied Ġto Ġthe Ġinput Ġand Ġoutput Ġlayers ). ĠIn Ġorder Ġto Ġfine Ġtune Ġthese Ġparameters Ġto Ġavoid Ġextreme Ġover fitting , Ġwe Ġcreated Ġanother Ġvalidation Ġset Ġconsisting Ġof Ġ5000 Ġsamples Ġof Ġ" unch ar acter istic " Ġinputs , Ġi . e ., Ġinputs Ġwith Ġrepeated Ġsymbols Ġand Ġvarying Ġfrom Ġlength Ġ3 - 12 . ĠThese Ġtwo Ġhyper param eter Ġvalues Ġwere Ġset Ġto Ġ0 . 125 Ġand Ġ0 . 1 , Ġrespectively , Ġaccording Ġto Ġthe Ġperformance Ġon Ġthis Ġvalidation Ġset , Ġaveraged Ġacross Ġa Ġset Ġof Ġrandomly Ġchosen Ġrandom Ġseeds . ĠAll Ġother Ġhyper param eters Ġwere Ġalso Ġdecided Ġusing Ġthis Ġvalidation Ġset . ĠFurther Ġtraining Ġdetails Ġare Ġlisted Ġin ĠTable Ġ1 Ġ. </s> Exper iments </s> To Ġgeneral ize Ġto Ġany Ġinput Ġsequence , Ġa Ġmodel Ġmust : Ġ( 1 ) Ġlearn Ġthe Ġgeneral izable Ġstructure Ġ- Ġthe Ġalign ments Ġbetween Ġinput Ġand Ġoutput Ġal ph ab ets , Ġand Ġ( 2 ) Ġnot Ġlearn Ġany Ġdependencies Ġamong Ġinput Ġsymbols Ġor Ġsequence Ġlength . ĠTo Ġtest Ġthe Ġextent Ġto Ġwhich Ġ( 2 ) Ġis Ġmet , Ġwe Ġtrain Ġseq 2 seq - att ention Ġmodels Ġwith Ġ100 , 000 Ġrandomly Ġgenerated Ġsamples Ġwith Ġinputs Ġuniformly Ġgenerated Ġwith Ġlengths Ġ5 - 10 Ġand Ġno Ġinput Ġsymbol Ġappearing Ġmore Ġthan Ġonce Ġin Ġa Ġsingle Ġsample . ĠIf Ġthe Ġmodel Ġlearned Ġalign ments Ġwithout Ġpicking Ġup Ġother Ġdependencies Ġamong Ġinput Ġsymbols Ġor Ġinput Ġlengths Ġthen Ġthe Ġresulting Ġmodel Ġshould Ġhave Ġlittle Ġproblem Ġin Ġhandling Ġinputs Ġwith Ġrepeated Ġsymbols Ġor Ġdifferent Ġlengths , Ġdespite Ġnever Ġseeing Ġsuch Ġstrings . </s> For Ġevaluation Ġwe Ġtrained Ġ50 Ġdifferent Ġmodels Ġwith Ġthe Ġsame Ġconfiguration , Ġchosen Ġwith Ġa Ġvalidation Ġset , Ġbut Ġwith Ġdifferent Ġrandom Ġseeds . ĠWe Ġcreated Ġ4 Ġdifferent Ġtest Ġsets , Ġeach Ġwith Ġ2000 Ġrandomly Ġgenerated Ġsamples . ĠThe Ġfirst Ġtest Ġset Ġconsists Ġof Ġsamples Ġthat Ġare Ġcharacteristic Ġof Ġthe Ġtraining Ġset , Ġhaving Ġlengths Ġ5 - 10 Ġand Ġno Ġrepeats Ġ( Standard ). ĠThe Ġsecond Ġset Ġtests Ġthe Ġmodel 's Ġability Ġto Ġgeneral ize Ġto Ġrepeated Ġsymbols Ġin Ġthe Ġinput Ġ( Repeat ). ĠThe Ġthird Ġand Ġfourth Ġsets Ġtest Ġits Ġability Ġto Ġgeneral ize Ġto Ġdifferent Ġinput Ġlengths , Ġstrings Ġof Ġlength Ġ1 - 4 Ġ( Short ) Ġand Ġ11 - 15 Ġ( Long ) Ġrespectively . ĠTable Ġ2 Ġprovides Ġfurther Ġinformation Ġon Ġthe Ġ4 Ġdifferent Ġtest Ġsets . </s> Results </s> The Ġdistribution Ġof Ġmodel Ġaccuracy Ġmeasured Ġat Ġinstance Ġlevel Ġon Ġthe Ġfour Ġtest Ġsets Ġacross Ġall Ġthe Ġ50 Ġseeds Ġis Ġgiven Ġin ĠFigure Ġ1 Ġ. ĠAll Ġmodels Ġperform Ġabove Ġ99 % Ġon Ġthe Ġstandard Ġset , Ġwith Ġa Ġdeviation Ġwell Ġbelow Ġ0 . 1 . ĠHowever , Ġthe Ġdeviation Ġon Ġthe Ġother Ġtwo Ġsets Ġis Ġmuch Ġlarger , Ġranging Ġfrom Ġ13 . 39 Ġfor Ġthe Ġrepeat Ġset Ġto Ġ20 . 63 Ġfor Ġthe Ġlong Ġset . ĠIn Ġgeneral , Ġthe Ġmodel Ġperforms Ġbetter Ġon Ġthe Ġrepeat Ġset Ġthan Ġon Ġthe Ġshort Ġand Ġlong Ġsets . ĠPerformance Ġon Ġthe Ġshort Ġand Ġlong Ġsets Ġis Ġnot Ġalways Ġbad , Ġsome Ġseeds Ġgiving Ġperformances Ġof Ġabove Ġ95 % Ġfor Ġeither Ġthe Ġshort Ġor Ġlong Ġset . ĠIdeally , Ġwe Ġwould Ġlike Ġa Ġseed Ġwhich Ġperforms Ġgood Ġon Ġall Ġthe Ġtest Ġsets ; Ġhowever , Ġthis Ġseems Ġhard Ġto Ġobtain . ĠThe Ġhighest Ġaverage Ġperformance Ġacross Ġthe Ġnon Ġstandard Ġtest Ġsets Ġfor Ġany Ġseed Ġwas Ġ79 . 52 %. ĠLearning Ġto Ġgeneral ize Ġfor Ġboth Ġthe Ġrepeated Ġand Ġlonger Ġinputs Ġseems Ġeven Ġharder , Ġwith Ġthe ĠPearson Ġcorrelation Ġbetween Ġperformance Ġon Ġthe Ġrepeat Ġand Ġlong Ġsets Ġbeing Ġ- 0 . 71 . </s> We Ġprovide Ġthe Ġsummary Ġstatistics Ġacross Ġall Ġ50 Ġruns Ġ( 50 Ġdifferent Ġrandom Ġseeds ) Ġin ĠTable Ġ4 Ġ, Ġwhich Ġgives Ġthe Ġmean , Ġstandard Ġdeviation , Ġminimum , Ġand Ġmaximum Ġaccur acies Ġacross Ġall Ġrandom Ġseeds . ĠWe Ġadditionally Ġprovide Ġa Ġsample Ġof Ġperformances Ġfor Ġsome Ġindividual Ġrandom Ġseeds Ġin ĠTable Ġ3 Ġ, Ġwith Ġthe Ġhighest Ġand Ġlowest Ġaccur acies Ġin Ġeach Ġcolumn Ġhighlighted . </s> Related ĠWork </s> BI B REF 12 Ġprovides Ġone Ġof Ġthe Ġearliest Ġworks Ġinvestigating Ġthe Ġability Ġof ĠR NN s Ġto Ġcapture Ġproperties Ġof Ġlanguage Ġnecessary Ġfor Ġgeneral ization . ĠFurther Ġwork Ġexplored Ġhow ĠR NN s Ġcan Ġlearn Ġcontext Ġfree Ġlanguages ĠB IB REF 13 Ġ, ĠB IB REF 9 Ġas Ġwell Ġsome Ġcontext Ġsensitive Ġlanguages ĠB IB REF 14 Ġ. ĠRecent Ġwork Ġhas Ġmoved Ġoutside Ġthe Ġformal Ġlanguage Ġspace , Ġwith Ġexperiments Ġthat Ġindicate ĠR NN s Ġmay Ġcapture Ġthe Ġhierarchical Ġstructure Ġof Ġnatural Ġlanguage Ġsyntax , Ġdespite Ġnot Ġhaving Ġany Ġhierarchical Ġbias Ġbuilt Ġin Ġthe Ġmodel Ġor Ġthe Ġdata ĠB IB REF 5 Ġ, ĠB IB REF 15 Ġ. </s> Though Ġthese Ġmodels Ġappear Ġto Ġcapture Ġsystematic Ġstructure Ġnecessary Ġfor Ġgeneral ization , Ġhow Ġthey Ġdo Ġso Ġis Ġoften Ġextremely Ġcounter Ġintuitive ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġ. ĠB IB REF 8 Ġhas Ġquestioned Ġthe Ġsystematic ity Ġwith Ġwhich Ġthese Ġmodels Ġlearn . ĠThe Ġsame Ġwork Ġalso Ġmakes Ġsimilar Ġobservations Ġabout Ġthe Ġdifficulty Ġof Ġgeneral izing Ġto Ġlonger Ġlength Ġstrings . ĠDespite Ġthe Ġdifficulty Ġmodels Ġhave Ġgeneralized Ġto Ġlonger Ġlength Ġstrings ; Ġthey Ġnote Ġthat Ġthe Ġmodel Ġonly Ġneeds Ġto Ġsee Ġa Ġrelatively Ġsmall Ġamount Ġof Ġlonger Ġlength Ġstrings Ġduring Ġtraining Ġin Ġorder Ġto Ġgeneral ize Ġ( up Ġto Ġthe Ġlength Ġof Ġthe Ġlongest Ġstring Ġshown Ġin Ġtraining ). </s> Our Ġexperiments Ġindicate Ġthat Ġthe Ġfinal Ġmodel 's Ġability Ġto Ġgeneral ize Ġis Ġhighly Ġdependent Ġon Ġthe Ġrandom Ġseed . ĠIn Ġshould Ġbe Ġnoted Ġhowever Ġthat Ġthe Ġrandom Ġseed Ġhas Ġan Ġeffect Ġon Ġseveral Ġcomponents Ġof Ġthe Ġtraining Ġprocess , Ġmost Ġnotably , Ġthe Ġexact Ġinitialization Ġof Ġthe Ġnetwork Ġand Ġthe Ġorder Ġin Ġwhich Ġtraining Ġsamples Ġare Ġshown . ĠThough Ġthe Ġexact Ġvalue Ġof Ġthe Ġinitialization Ġlikely Ġplays Ġa Ġfactor , Ġrecent Ġwork Ġby ĠB IB REF 18 Ġgives Ġevidence Ġthat Ġthe Ġorder Ġtraining Ġdata Ġis Ġpresented Ġmay Ġplay Ġan Ġequally Ġlarge Ġpart Ġin Ġdetermining Ġwhether Ġa Ġmodel Ġachieves Ġgeneral ization . </s> Con clusions </s> The Ġvariability Ġin Ġgeneral ization Ġon Ġun character istic Ġinputs Ġ( and Ġthus , Ġthe Ġextent Ġof Ġlinguistic Ġgeneral ization ) Ġgiven Ġdifferent Ġrandom Ġseeds Ġis Ġalarming , Ġparticularly Ġgiven Ġthe Ġfact Ġthat Ġthe Ġstandard Ġtest Ġset Ġperformance Ġremains Ġmostly Ġthe Ġsame Ġregardless . ĠThe Ġtask Ġpresented Ġhere Ġwas Ġeasy Ġand Ġsimple Ġto Ġanalyze , Ġhowever , Ġfuture Ġwork Ġmay Ġbe Ġdone Ġon Ġnatural Ġlanguage Ġtasks . ĠIf Ġthese Ġproperties Ġhold Ġit Ġmight Ġindicate Ġthat Ġa Ġnew Ġevaluation Ġparadigm Ġfor ĠN LP Ġshould Ġbe Ġpushed ; Ġone Ġthat Ġemphasizes Ġperformance Ġon Ġun character istic Ġ( but Ġstruct urally Ġsound ) Ġinputs Ġin Ġaddition Ġto Ġthe Ġdata Ġtypically Ġseen Ġin Ġtraining . </s> Acknowled gements </s> This Ġwork Ġis Ġsupported Ġin Ġpart Ġby Ġthe ĠNational ĠScience ĠFoundation Ġunder ĠGrant ĠI IS - 16 179 69 .
Introduction It is well known that language has certain structural properties which allows natural language speakers to make “infinite use of finite means" BIBREF3 . This structure allows us to generalize beyond the typical machine learning definition of generalization BIBREF4 (which considers performance on the distribution that generated the training set), permitting the understanding of any utterance sharing the same structure, regardless of probability. For example, sentences of length 100 typically do not appear in natural text or speech (our personal 'training set'), but can be understood regardless due to their structure. We refer to this notion as linguistic generalization . Many problems in NLP are treated as sequence to sequence tasks with solutions built on seq2seq-attention based models. While these models perform very well on standard datasets and also appear to capture some linguistic structure BIBREF5 , BIBREF6 , BIBREF7 , they also can be quite brittle, typically breaking on uncharacteristic inputs BIBREF8 , BIBREF1 , indicating that the extent of linguistic generalization these models achieve is still somewhat lacking. Due to the high capacity of these models, it is not unreasonable to expect them to learn some structure from the data. However, learning structure is not a sufficient condition to achieving linguistic generalization. If this structure is to be usable on data outside the training distribution, the model must learn the structure without additionally learning (overfitting on) patterns specific to the training data. One may hope, given the right hyperparameter configuration and regularization, that a model converges to a solution that captures the reusable structure without overfitting too much on the training set. While this solution exists in theory, in practice, it may be difficult to find. In this work, we look at the feasibility of training and tuning seq2seq-attention models towards a solution that generalizes in this linguistic sense. In particular, we train models on a symbol replacement task with a well defined generalizable structure. The task is simple enough that all models achieve near perfect accuracy on the standard test set, i.e., where the inputs are drawn from the same distribution as that of the training set. We then test these models for linguistic generalization by creating test sets of uncharacteristic inputs, i.e., inputs that are not typical in the training distribution but still solvable given that the generalizable structure was learned. Our results indicate that generalization is highly sensitive; such that even changes in the random seed can drastically affect the ability to generalize. This dependence on an element that is not (or ideally should not be) a hyperparameter suggests that the line between generalization and failure is quite fine, and may not be feasible to reach simply by hyperparameter tuning alone. Generalization in a Symbol Rewriting Task Real world NLP tasks are complex, and as such, it can be difficult to precisely define what a model should and should not learn during training. As done in previous work BIBREF8 , BIBREF9 , we ease analysis by looking at a simple formal task. The task is set up to mimic (albeit, in an oversimplified manner) the input-output symbol alignments and local syntactic properties that models must learn in many natural language tasks, such as translation, tagging and summarization. The task is defined over sequences of symbols, $\lbrace x_1,...x_n | x_i \in X\rbrace $ , where $X$ is the input alphabet. Each symbol $x \in X$ is uniquely associated with its own output alphabet $Y_x$ . Output is created by taking each individual symbol $x_i$ in the sequence and rewriting it as any sequence of $k$ symbols from $Y_{x_i}$ . To do the task, the model must learn alignments between the input and output symbols, and preserve the simple local syntactic conditions (every group of $k$ symbols must come from the same input alphabet $Y_x$ ). As an example, let $X = \lbrace A,B\rbrace $ , $Y_A = \lbrace \text{$a_{1}, a_{2}$, $a_3$}\rbrace $ , $Y_B = \lbrace \text{$b_1, b_2,$ $b_3$}\rbrace $ , and $k=3$ . Each $a_i$ and $b_i$ has 2 possible values, $a_{i1}$ or $a_{i2}$ and $b_{i1}$ or $b_{i2}$ respectively. Thus, mapping an input symbol to 48 ( $Y_A = \lbrace \text{$a_{1}, a_{2}$, $a_3$}\rbrace $0 ) possible permutations. A possible valid output for the input $Y_A = \lbrace \text{$a_{1}, a_{2}$, $a_3$}\rbrace $1 is $Y_A = \lbrace \text{$a_{1}, a_{2}$, $a_3$}\rbrace $2 . Note that such valid strings are selected at random when generating the dataset. We allow this stochasticity in the outputs in order to prevent the model from resorting to pure memorization. For our task, $Y_A = \lbrace \text{$a_{1}, a_{2}$, $a_3$}\rbrace $3 and each $Y_A = \lbrace \text{$a_{1}, a_{2}$, $a_3$}\rbrace $4 has a corresponding output alphabet $Y_A = \lbrace \text{$a_{1}, a_{2}$, $a_3$}\rbrace $5 of size 16. Model and Training Details The models we use are single layer, unidirectional, seq2seq LSTMs BIBREF10 with bilinear attention BIBREF11 and trained with vanilla SGD. To determine the epoch to stop training at, we create a validation set of 2000 samples with the same characteristics as the training set, i.e., of length 5-10 with no repeated symbols. Training is stopped once accuracy on the validation set either decreases or remains unchanged. The size of the hidden state and embeddings were chosen such that they were as small as possible without reducing validation accuracy, giving a size of 32. Tuning hyperparameters is often done on a validation set drawn from the same distribution as the training set (as opposed to validating on data from a different distribution) which motivated our initial decision to use a validation set of characteristic inputs to decide the epoch to stop at. However, we noticed only small variation in the validation performance upon using different learning rates and dropout probabilities (where dropout was applied to the input and output layers). In order to fine tune these parameters to avoid extreme overfitting, we created another validation set consisting of 5000 samples of "uncharacteristic" inputs, i.e., inputs with repeated symbols and varying from length 3-12. These two hyperparameter values were set to 0.125 and 0.1, respectively, according to the performance on this validation set, averaged across a set of randomly chosen random seeds. All other hyperparameters were also decided using this validation set. Further training details are listed in Table 1 . Experiments To generalize to any input sequence, a model must: (1) learn the generalizable structure - the alignments between input and output alphabets, and (2) not learn any dependencies among input symbols or sequence length. To test the extent to which (2) is met, we train seq2seq-attention models with 100,000 randomly generated samples with inputs uniformly generated with lengths 5-10 and no input symbol appearing more than once in a single sample. If the model learned alignments without picking up other dependencies among input symbols or input lengths then the resulting model should have little problem in handling inputs with repeated symbols or different lengths, despite never seeing such strings. For evaluation we trained 50 different models with the same configuration, chosen with a validation set, but with different random seeds. We created 4 different test sets, each with 2000 randomly generated samples. The first test set consists of samples that are characteristic of the training set, having lengths 5-10 and no repeats (Standard). The second set tests the model's ability to generalize to repeated symbols in the input (Repeat). The third and fourth sets test its ability to generalize to different input lengths, strings of length 1-4 (Short) and 11-15 (Long) respectively. Table 2 provides further information on the 4 different test sets. Results The distribution of model accuracy measured at instance level on the four test sets across all the 50 seeds is given in Figure 1 . All models perform above 99% on the standard set, with a deviation well below 0.1. However, the deviation on the other two sets is much larger, ranging from 13.39 for the repeat set to 20.63 for the long set. In general, the model performs better on the repeat set than on the short and long sets. Performance on the short and long sets is not always bad, some seeds giving performances of above 95% for either the short or long set. Ideally, we would like a seed which performs good on all the test sets; however, this seems hard to obtain. The highest average performance across the non standard test sets for any seed was 79.52%. Learning to generalize for both the repeated and longer inputs seems even harder, with the Pearson correlation between performance on the repeat and long sets being -0.71. We provide the summary statistics across all 50 runs (50 different random seeds) in Table 4 , which gives the mean, standard deviation, minimum, and maximum accuracies across all random seeds. We additionally provide a sample of performances for some individual random seeds in Table 3 , with the highest and lowest accuracies in each column highlighted. Related Work BIBREF12 provides one of the earliest works investigating the ability of RNNs to capture properties of language necessary for generalization. Further work explored how RNNs can learn context free languages BIBREF13 , BIBREF9 as well some context sensitive languages BIBREF14 . Recent work has moved outside the formal language space, with experiments that indicate RNNs may capture the hierarchical structure of natural language syntax, despite not having any hierarchical bias built in the model or the data BIBREF5 , BIBREF15 . Though these models appear to capture systematic structure necessary for generalization, how they do so is often extremely counter intuitive BIBREF16 , BIBREF17 . BIBREF8 has questioned the systematicity with which these models learn. The same work also makes similar observations about the difficulty of generalizing to longer length strings. Despite the difficulty models have generalized to longer length strings; they note that the model only needs to see a relatively small amount of longer length strings during training in order to generalize (up to the length of the longest string shown in training). Our experiments indicate that the final model's ability to generalize is highly dependent on the random seed. In should be noted however that the random seed has an effect on several components of the training process, most notably, the exact initialization of the network and the order in which training samples are shown. Though the exact value of the initialization likely plays a factor, recent work by BIBREF18 gives evidence that the order training data is presented may play an equally large part in determining whether a model achieves generalization. Conclusions The variability in generalization on uncharacteristic inputs (and thus, the extent of linguistic generalization) given different random seeds is alarming, particularly given the fact that the standard test set performance remains mostly the same regardless. The task presented here was easy and simple to analyze, however, future work may be done on natural language tasks. If these properties hold it might indicate that a new evaluation paradigm for NLP should be pushed; one that emphasizes performance on uncharacteristic (but structurally sound) inputs in addition to the data typically seen in training. Acknowledgements This work is supported in part by the National Science Foundation under Grant IIS-1617969.
Why does the proposed task a good proxy for the general-purpose sequence to sequence tasks?
dcea88698949da4a1bd00277c06df06c33f6a5ff
Why Ġdoes Ġthe Ġproposed Ġtask Ġa Ġgood Ġproxy Ġfor Ġthe Ġgeneral - purpose Ġsequence Ġto Ġsequence Ġtasks ?
The Ġtask Ġis Ġset Ġup Ġto Ġmimic Ġ( albeit , Ġin Ġan Ġovers impl ified Ġmanner ) Ġthe Ġinput - output Ġsymbol Ġalign ments Ġand Ġlocal Ġsynt actic Ġproperties Ġthat Ġmodels Ġmust Ġlearn Ġin Ġmany Ġnatural Ġlanguage Ġtasks , Ġsuch Ġas Ġtranslation , Ġtagging Ġand Ġsummar ization .
<s> Why Ġdoes Ġthe Ġproposed Ġtask Ġa Ġgood Ġproxy Ġfor Ġthe Ġgeneral - purpose Ġsequence Ġto Ġsequence Ġtasks ? </s> Introduction </s> It Ġis Ġwell Ġknown Ġthat Ġlanguage Ġhas Ġcertain Ġstructural Ġproperties Ġwhich Ġallows Ġnatural Ġlanguage Ġspeakers Ġto Ġmake ĠâĢ ľ inf inite Ġuse Ġof Ġfinite Ġmeans " ĠB IB REF 3 Ġ. ĠThis Ġstructure Ġallows Ġus Ġto Ġgeneral ize Ġbeyond Ġthe Ġtypical Ġmachine Ġlearning Ġdefinition Ġof Ġgeneral ization ĠB IB REF 4 Ġ( which Ġconsiders Ġperformance Ġon Ġthe Ġdistribution Ġthat Ġgenerated Ġthe Ġtraining Ġset ), Ġpermitting Ġthe Ġunderstanding Ġof Ġany Ġutter ance Ġsharing Ġthe Ġsame Ġstructure , Ġregardless Ġof Ġprobability . ĠFor Ġexample , Ġsentences Ġof Ġlength Ġ100 Ġtypically Ġdo Ġnot Ġappear Ġin Ġnatural Ġtext Ġor Ġspeech Ġ( our Ġpersonal Ġ' training Ġset '), Ġbut Ġcan Ġbe Ġunderstood Ġregardless Ġdue Ġto Ġtheir Ġstructure . ĠWe Ġrefer Ġto Ġthis Ġnotion Ġas Ġlinguistic Ġgeneral ization Ġ. </s> Many Ġproblems Ġin ĠN LP Ġare Ġtreated Ġas Ġsequence Ġto Ġsequence Ġtasks Ġwith Ġsolutions Ġbuilt Ġon Ġseq 2 seq - att ention Ġbased Ġmodels . ĠWhile Ġthese Ġmodels Ġperform Ġvery Ġwell Ġon Ġstandard Ġdatasets Ġand Ġalso Ġappear Ġto Ġcapture Ġsome Ġlinguistic Ġstructure ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, Ġthey Ġalso Ġcan Ġbe Ġquite Ġbrittle , Ġtypically Ġbreaking Ġon Ġun character istic Ġinputs ĠB IB REF 8 Ġ, ĠB IB REF 1 Ġ, Ġindicating Ġthat Ġthe Ġextent Ġof Ġlinguistic Ġgeneral ization Ġthese Ġmodels Ġachieve Ġis Ġstill Ġsomewhat Ġlacking . </s> Due Ġto Ġthe Ġhigh Ġcapacity Ġof Ġthese Ġmodels , Ġit Ġis Ġnot Ġunreasonable Ġto Ġexpect Ġthem Ġto Ġlearn Ġsome Ġstructure Ġfrom Ġthe Ġdata . ĠHowever , Ġlearning Ġstructure Ġis Ġnot Ġa Ġsufficient Ġcondition Ġto Ġachieving Ġlinguistic Ġgeneral ization . ĠIf Ġthis Ġstructure Ġis Ġto Ġbe Ġusable Ġon Ġdata Ġoutside Ġthe Ġtraining Ġdistribution , Ġthe Ġmodel Ġmust Ġlearn Ġthe Ġstructure Ġwithout Ġadditionally Ġlearning Ġ( over fitting Ġon ) Ġpatterns Ġspecific Ġto Ġthe Ġtraining Ġdata . ĠOne Ġmay Ġhope , Ġgiven Ġthe Ġright Ġhyper param eter Ġconfiguration Ġand Ġregular ization , Ġthat Ġa Ġmodel Ġconver ges Ġto Ġa Ġsolution Ġthat Ġcaptures Ġthe Ġreusable Ġstructure Ġwithout Ġover fitting Ġtoo Ġmuch Ġon Ġthe Ġtraining Ġset . ĠWhile Ġthis Ġsolution Ġexists Ġin Ġtheory , Ġin Ġpractice , Ġit Ġmay Ġbe Ġdifficult Ġto Ġfind . </s> In Ġthis Ġwork , Ġwe Ġlook Ġat Ġthe Ġfeasibility Ġof Ġtraining Ġand Ġtuning Ġseq 2 seq - att ention Ġmodels Ġtowards Ġa Ġsolution Ġthat Ġgeneral izes Ġin Ġthis Ġlinguistic Ġsense . ĠIn Ġparticular , Ġwe Ġtrain Ġmodels Ġon Ġa Ġsymbol Ġreplacement Ġtask Ġwith Ġa Ġwell Ġdefined Ġgeneral izable Ġstructure . ĠThe Ġtask Ġis Ġsimple Ġenough Ġthat Ġall Ġmodels Ġachieve Ġnear Ġperfect Ġaccuracy Ġon Ġthe Ġstandard Ġtest Ġset , Ġi . e ., Ġwhere Ġthe Ġinputs Ġare Ġdrawn Ġfrom Ġthe Ġsame Ġdistribution Ġas Ġthat Ġof Ġthe Ġtraining Ġset . ĠWe Ġthen Ġtest Ġthese Ġmodels Ġfor Ġlinguistic Ġgeneral ization Ġby Ġcreating Ġtest Ġsets Ġof Ġun character istic Ġinputs , Ġi . e ., Ġinputs Ġthat Ġare Ġnot Ġtypical Ġin Ġthe Ġtraining Ġdistribution Ġbut Ġstill Ġsol vable Ġgiven Ġthat Ġthe Ġgeneral izable Ġstructure Ġwas Ġlearned . ĠOur Ġresults Ġindicate Ġthat Ġgeneral ization Ġis Ġhighly Ġsensitive ; Ġsuch Ġthat Ġeven Ġchanges Ġin Ġthe Ġrandom Ġseed Ġcan Ġdrastically Ġaffect Ġthe Ġability Ġto Ġgeneral ize . ĠThis Ġdependence Ġon Ġan Ġelement Ġthat Ġis Ġnot Ġ( or Ġideally Ġshould Ġnot Ġbe ) Ġa Ġhyper param eter Ġsuggests Ġthat Ġthe Ġline Ġbetween Ġgeneral ization Ġand Ġfailure Ġis Ġquite Ġfine , Ġand Ġmay Ġnot Ġbe Ġfeasible Ġto Ġreach Ġsimply Ġby Ġhyper param eter Ġtuning Ġalone . </s> General ization Ġin Ġa ĠSymbol ĠRew rit ing ĠTask </s> Real Ġworld ĠN LP Ġtasks Ġare Ġcomplex , Ġand Ġas Ġsuch , Ġit Ġcan Ġbe Ġdifficult Ġto Ġprecisely Ġdefine Ġwhat Ġa Ġmodel Ġshould Ġand Ġshould Ġnot Ġlearn Ġduring Ġtraining . ĠAs Ġdone Ġin Ġprevious Ġwork ĠB IB REF 8 Ġ, ĠB IB REF 9 Ġ, Ġwe Ġease Ġanalysis Ġby Ġlooking Ġat Ġa Ġsimple Ġformal Ġtask . ĠThe Ġtask Ġis Ġset Ġup Ġto Ġmimic Ġ( albeit , Ġin Ġan Ġovers impl ified Ġmanner ) Ġthe Ġinput - output Ġsymbol Ġalign ments Ġand Ġlocal Ġsynt actic Ġproperties Ġthat Ġmodels Ġmust Ġlearn Ġin Ġmany Ġnatural Ġlanguage Ġtasks , Ġsuch Ġas Ġtranslation , Ġtagging Ġand Ġsummar ization . </s> The Ġtask Ġis Ġdefined Ġover Ġsequences Ġof Ġsymbols , Ġ$\ l brace Ġx _ 1 ,... x _ n Ġ| Ġx _ i Ġ\ in ĠX \ r brace Ġ$ Ġ, Ġwhere Ġ$ X $ Ġis Ġthe Ġinput Ġalphabet . ĠEach Ġsymbol Ġ$ x Ġ\ in ĠX $ Ġis Ġuniquely Ġassociated Ġwith Ġits Ġown Ġoutput Ġalphabet Ġ$ Y _ x $ Ġ. ĠOutput Ġis Ġcreated Ġby Ġtaking Ġeach Ġindividual Ġsymbol Ġ$ x _ i $ Ġin Ġthe Ġsequence Ġand Ġrewriting Ġit Ġas Ġany Ġsequence Ġof Ġ$ k $ Ġsymbols Ġfrom Ġ$ Y _{ x _ i } $ Ġ. ĠTo Ġdo Ġthe Ġtask , Ġthe Ġmodel Ġmust Ġlearn Ġalign ments Ġbetween Ġthe Ġinput Ġand Ġoutput Ġsymbols , Ġand Ġpreserve Ġthe Ġsimple Ġlocal Ġsynt actic Ġconditions Ġ( every Ġgroup Ġof Ġ$ k $ Ġsymbols Ġmust Ġcome Ġfrom Ġthe Ġsame Ġinput Ġalphabet Ġ$ Y _ x $ Ġ). </s> As Ġan Ġexample , Ġlet Ġ$ X Ġ= Ġ\ l brace ĠA , B \ r brace Ġ$ Ġ, Ġ$ Y _ A Ġ= Ġ\ l brace Ġ\ text { $ a _{ 1 }, Ġa _{ 2 } $, Ġ$ a _ 3 $ }\ r brace Ġ$ Ġ, Ġ$ Y _ B Ġ= Ġ\ l brace Ġ\ text { $ b _ 1 , Ġb _ 2 , $ Ġ$ b _ 3 $ }\ r brace Ġ$ Ġ, Ġand Ġ$ k = 3 $ Ġ. ĠEach Ġ$ a _ i $ Ġand Ġ$ b _ i $ Ġhas Ġ2 Ġpossible Ġvalues , Ġ$ a _{ i 1 } $ Ġor Ġ$ a _{ i 2 } $ Ġand Ġ$ b _{ i 1 } $ Ġor Ġ$ b _{ i 2 } $ Ġrespectively . ĠThus , Ġmapping Ġan Ġinput Ġsymbol Ġto Ġ48 Ġ( Ġ$ Y _ A Ġ= Ġ\ l brace Ġ\ text { $ a _{ 1 }, Ġa _{ 2 } $, Ġ$ a _ 3 $ }\ r brace Ġ$ 0 Ġ) Ġpossible Ġperm utations . ĠA Ġpossible Ġvalid Ġoutput Ġfor Ġthe Ġinput Ġ$ Y _ A Ġ= Ġ\ l brace Ġ\ text { $ a _{ 1 }, Ġa _{ 2 } $, Ġ$ a _ 3 $ }\ r brace Ġ$ 1 Ġis Ġ$ Y _ A Ġ= Ġ\ l brace Ġ\ text { $ a _{ 1 }, Ġa _{ 2 } $, Ġ$ a _ 3 $ }\ r brace Ġ$ 2 Ġ. ĠNote Ġthat Ġsuch Ġvalid Ġstrings Ġare Ġselected Ġat Ġrandom Ġwhen Ġgenerating Ġthe Ġdataset . ĠWe Ġallow Ġthis Ġsto ch astic ity Ġin Ġthe Ġoutputs Ġin Ġorder Ġto Ġprevent Ġthe Ġmodel Ġfrom Ġresort ing Ġto Ġpure Ġmemor ization . ĠFor Ġour Ġtask , Ġ$ Y _ A Ġ= Ġ\ l brace Ġ\ text { $ a _{ 1 }, Ġa _{ 2 } $, Ġ$ a _ 3 $ }\ r brace Ġ$ 3 Ġand Ġeach Ġ$ Y _ A Ġ= Ġ\ l brace Ġ\ text { $ a _{ 1 }, Ġa _{ 2 } $, Ġ$ a _ 3 $ }\ r brace Ġ$ 4 Ġhas Ġa Ġcorresponding Ġoutput Ġalphabet Ġ$ Y _ A Ġ= Ġ\ l brace Ġ\ text { $ a _{ 1 }, Ġa _{ 2 } $, Ġ$ a _ 3 $ }\ r brace Ġ$ 5 Ġof Ġsize Ġ16 . </s> Model Ġand ĠTraining ĠDetails </s> The Ġmodels Ġwe Ġuse Ġare Ġsingle Ġlayer , Ġun id irection al , Ġseq 2 seq ĠL ST Ms ĠB IB REF 10 Ġwith Ġbil in ear Ġattention ĠB IB REF 11 Ġand Ġtrained Ġwith Ġvanilla ĠSG D . ĠTo Ġdetermine Ġthe Ġepoch Ġto Ġstop Ġtraining Ġat , Ġwe Ġcreate Ġa Ġvalidation Ġset Ġof Ġ2000 Ġsamples Ġwith Ġthe Ġsame Ġcharacteristics Ġas Ġthe Ġtraining Ġset , Ġi . e ., Ġof Ġlength Ġ5 - 10 Ġwith Ġno Ġrepeated Ġsymbols . ĠTraining Ġis Ġstopped Ġonce Ġaccuracy Ġon Ġthe Ġvalidation Ġset Ġeither Ġdecreases Ġor Ġremains Ġunchanged . ĠThe Ġsize Ġof Ġthe Ġhidden Ġstate Ġand Ġembed d ings Ġwere Ġchosen Ġsuch Ġthat Ġthey Ġwere Ġas Ġsmall Ġas Ġpossible Ġwithout Ġreducing Ġvalidation Ġaccuracy , Ġgiving Ġa Ġsize Ġof Ġ32 . </s> T uning Ġhyper param eters Ġis Ġoften Ġdone Ġon Ġa Ġvalidation Ġset Ġdrawn Ġfrom Ġthe Ġsame Ġdistribution Ġas Ġthe Ġtraining Ġset Ġ( as Ġopposed Ġto Ġvalid ating Ġon Ġdata Ġfrom Ġa Ġdifferent Ġdistribution ) Ġwhich Ġmotivated Ġour Ġinitial Ġdecision Ġto Ġuse Ġa Ġvalidation Ġset Ġof Ġcharacteristic Ġinputs Ġto Ġdecide Ġthe Ġepoch Ġto Ġstop Ġat . ĠHowever , Ġwe Ġnoticed Ġonly Ġsmall Ġvariation Ġin Ġthe Ġvalidation Ġperformance Ġupon Ġusing Ġdifferent Ġlearning Ġrates Ġand Ġdrop out Ġprobabilities Ġ( where Ġdrop out Ġwas Ġapplied Ġto Ġthe Ġinput Ġand Ġoutput Ġlayers ). ĠIn Ġorder Ġto Ġfine Ġtune Ġthese Ġparameters Ġto Ġavoid Ġextreme Ġover fitting , Ġwe Ġcreated Ġanother Ġvalidation Ġset Ġconsisting Ġof Ġ5000 Ġsamples Ġof Ġ" unch ar acter istic " Ġinputs , Ġi . e ., Ġinputs Ġwith Ġrepeated Ġsymbols Ġand Ġvarying Ġfrom Ġlength Ġ3 - 12 . ĠThese Ġtwo Ġhyper param eter Ġvalues Ġwere Ġset Ġto Ġ0 . 125 Ġand Ġ0 . 1 , Ġrespectively , Ġaccording Ġto Ġthe Ġperformance Ġon Ġthis Ġvalidation Ġset , Ġaveraged Ġacross Ġa Ġset Ġof Ġrandomly Ġchosen Ġrandom Ġseeds . ĠAll Ġother Ġhyper param eters Ġwere Ġalso Ġdecided Ġusing Ġthis Ġvalidation Ġset . ĠFurther Ġtraining Ġdetails Ġare Ġlisted Ġin ĠTable Ġ1 Ġ. </s> Exper iments </s> To Ġgeneral ize Ġto Ġany Ġinput Ġsequence , Ġa Ġmodel Ġmust : Ġ( 1 ) Ġlearn Ġthe Ġgeneral izable Ġstructure Ġ- Ġthe Ġalign ments Ġbetween Ġinput Ġand Ġoutput Ġal ph ab ets , Ġand Ġ( 2 ) Ġnot Ġlearn Ġany Ġdependencies Ġamong Ġinput Ġsymbols Ġor Ġsequence Ġlength . ĠTo Ġtest Ġthe Ġextent Ġto Ġwhich Ġ( 2 ) Ġis Ġmet , Ġwe Ġtrain Ġseq 2 seq - att ention Ġmodels Ġwith Ġ100 , 000 Ġrandomly Ġgenerated Ġsamples Ġwith Ġinputs Ġuniformly Ġgenerated Ġwith Ġlengths Ġ5 - 10 Ġand Ġno Ġinput Ġsymbol Ġappearing Ġmore Ġthan Ġonce Ġin Ġa Ġsingle Ġsample . ĠIf Ġthe Ġmodel Ġlearned Ġalign ments Ġwithout Ġpicking Ġup Ġother Ġdependencies Ġamong Ġinput Ġsymbols Ġor Ġinput Ġlengths Ġthen Ġthe Ġresulting Ġmodel Ġshould Ġhave Ġlittle Ġproblem Ġin Ġhandling Ġinputs Ġwith Ġrepeated Ġsymbols Ġor Ġdifferent Ġlengths , Ġdespite Ġnever Ġseeing Ġsuch Ġstrings . </s> For Ġevaluation Ġwe Ġtrained Ġ50 Ġdifferent Ġmodels Ġwith Ġthe Ġsame Ġconfiguration , Ġchosen Ġwith Ġa Ġvalidation Ġset , Ġbut Ġwith Ġdifferent Ġrandom Ġseeds . ĠWe Ġcreated Ġ4 Ġdifferent Ġtest Ġsets , Ġeach Ġwith Ġ2000 Ġrandomly Ġgenerated Ġsamples . ĠThe Ġfirst Ġtest Ġset Ġconsists Ġof Ġsamples Ġthat Ġare Ġcharacteristic Ġof Ġthe Ġtraining Ġset , Ġhaving Ġlengths Ġ5 - 10 Ġand Ġno Ġrepeats Ġ( Standard ). ĠThe Ġsecond Ġset Ġtests Ġthe Ġmodel 's Ġability Ġto Ġgeneral ize Ġto Ġrepeated Ġsymbols Ġin Ġthe Ġinput Ġ( Repeat ). ĠThe Ġthird Ġand Ġfourth Ġsets Ġtest Ġits Ġability Ġto Ġgeneral ize Ġto Ġdifferent Ġinput Ġlengths , Ġstrings Ġof Ġlength Ġ1 - 4 Ġ( Short ) Ġand Ġ11 - 15 Ġ( Long ) Ġrespectively . ĠTable Ġ2 Ġprovides Ġfurther Ġinformation Ġon Ġthe Ġ4 Ġdifferent Ġtest Ġsets . </s> Results </s> The Ġdistribution Ġof Ġmodel Ġaccuracy Ġmeasured Ġat Ġinstance Ġlevel Ġon Ġthe Ġfour Ġtest Ġsets Ġacross Ġall Ġthe Ġ50 Ġseeds Ġis Ġgiven Ġin ĠFigure Ġ1 Ġ. ĠAll Ġmodels Ġperform Ġabove Ġ99 % Ġon Ġthe Ġstandard Ġset , Ġwith Ġa Ġdeviation Ġwell Ġbelow Ġ0 . 1 . ĠHowever , Ġthe Ġdeviation Ġon Ġthe Ġother Ġtwo Ġsets Ġis Ġmuch Ġlarger , Ġranging Ġfrom Ġ13 . 39 Ġfor Ġthe Ġrepeat Ġset Ġto Ġ20 . 63 Ġfor Ġthe Ġlong Ġset . ĠIn Ġgeneral , Ġthe Ġmodel Ġperforms Ġbetter Ġon Ġthe Ġrepeat Ġset Ġthan Ġon Ġthe Ġshort Ġand Ġlong Ġsets . ĠPerformance Ġon Ġthe Ġshort Ġand Ġlong Ġsets Ġis Ġnot Ġalways Ġbad , Ġsome Ġseeds Ġgiving Ġperformances Ġof Ġabove Ġ95 % Ġfor Ġeither Ġthe Ġshort Ġor Ġlong Ġset . ĠIdeally , Ġwe Ġwould Ġlike Ġa Ġseed Ġwhich Ġperforms Ġgood Ġon Ġall Ġthe Ġtest Ġsets ; Ġhowever , Ġthis Ġseems Ġhard Ġto Ġobtain . ĠThe Ġhighest Ġaverage Ġperformance Ġacross Ġthe Ġnon Ġstandard Ġtest Ġsets Ġfor Ġany Ġseed Ġwas Ġ79 . 52 %. ĠLearning Ġto Ġgeneral ize Ġfor Ġboth Ġthe Ġrepeated Ġand Ġlonger Ġinputs Ġseems Ġeven Ġharder , Ġwith Ġthe ĠPearson Ġcorrelation Ġbetween Ġperformance Ġon Ġthe Ġrepeat Ġand Ġlong Ġsets Ġbeing Ġ- 0 . 71 . </s> We Ġprovide Ġthe Ġsummary Ġstatistics Ġacross Ġall Ġ50 Ġruns Ġ( 50 Ġdifferent Ġrandom Ġseeds ) Ġin ĠTable Ġ4 Ġ, Ġwhich Ġgives Ġthe Ġmean , Ġstandard Ġdeviation , Ġminimum , Ġand Ġmaximum Ġaccur acies Ġacross Ġall Ġrandom Ġseeds . ĠWe Ġadditionally Ġprovide Ġa Ġsample Ġof Ġperformances Ġfor Ġsome Ġindividual Ġrandom Ġseeds Ġin ĠTable Ġ3 Ġ, Ġwith Ġthe Ġhighest Ġand Ġlowest Ġaccur acies Ġin Ġeach Ġcolumn Ġhighlighted . </s> Related ĠWork </s> BI B REF 12 Ġprovides Ġone Ġof Ġthe Ġearliest Ġworks Ġinvestigating Ġthe Ġability Ġof ĠR NN s Ġto Ġcapture Ġproperties Ġof Ġlanguage Ġnecessary Ġfor Ġgeneral ization . ĠFurther Ġwork Ġexplored Ġhow ĠR NN s Ġcan Ġlearn Ġcontext Ġfree Ġlanguages ĠB IB REF 13 Ġ, ĠB IB REF 9 Ġas Ġwell Ġsome Ġcontext Ġsensitive Ġlanguages ĠB IB REF 14 Ġ. ĠRecent Ġwork Ġhas Ġmoved Ġoutside Ġthe Ġformal Ġlanguage Ġspace , Ġwith Ġexperiments Ġthat Ġindicate ĠR NN s Ġmay Ġcapture Ġthe Ġhierarchical Ġstructure Ġof Ġnatural Ġlanguage Ġsyntax , Ġdespite Ġnot Ġhaving Ġany Ġhierarchical Ġbias Ġbuilt Ġin Ġthe Ġmodel Ġor Ġthe Ġdata ĠB IB REF 5 Ġ, ĠB IB REF 15 Ġ. </s> Though Ġthese Ġmodels Ġappear Ġto Ġcapture Ġsystematic Ġstructure Ġnecessary Ġfor Ġgeneral ization , Ġhow Ġthey Ġdo Ġso Ġis Ġoften Ġextremely Ġcounter Ġintuitive ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġ. ĠB IB REF 8 Ġhas Ġquestioned Ġthe Ġsystematic ity Ġwith Ġwhich Ġthese Ġmodels Ġlearn . ĠThe Ġsame Ġwork Ġalso Ġmakes Ġsimilar Ġobservations Ġabout Ġthe Ġdifficulty Ġof Ġgeneral izing Ġto Ġlonger Ġlength Ġstrings . ĠDespite Ġthe Ġdifficulty Ġmodels Ġhave Ġgeneralized Ġto Ġlonger Ġlength Ġstrings ; Ġthey Ġnote Ġthat Ġthe Ġmodel Ġonly Ġneeds Ġto Ġsee Ġa Ġrelatively Ġsmall Ġamount Ġof Ġlonger Ġlength Ġstrings Ġduring Ġtraining Ġin Ġorder Ġto Ġgeneral ize Ġ( up Ġto Ġthe Ġlength Ġof Ġthe Ġlongest Ġstring Ġshown Ġin Ġtraining ). </s> Our Ġexperiments Ġindicate Ġthat Ġthe Ġfinal Ġmodel 's Ġability Ġto Ġgeneral ize Ġis Ġhighly Ġdependent Ġon Ġthe Ġrandom Ġseed . ĠIn Ġshould Ġbe Ġnoted Ġhowever Ġthat Ġthe Ġrandom Ġseed Ġhas Ġan Ġeffect Ġon Ġseveral Ġcomponents Ġof Ġthe Ġtraining Ġprocess , Ġmost Ġnotably , Ġthe Ġexact Ġinitialization Ġof Ġthe Ġnetwork Ġand Ġthe Ġorder Ġin Ġwhich Ġtraining Ġsamples Ġare Ġshown . ĠThough Ġthe Ġexact Ġvalue Ġof Ġthe Ġinitialization Ġlikely Ġplays Ġa Ġfactor , Ġrecent Ġwork Ġby ĠB IB REF 18 Ġgives Ġevidence Ġthat Ġthe Ġorder Ġtraining Ġdata Ġis Ġpresented Ġmay Ġplay Ġan Ġequally Ġlarge Ġpart Ġin Ġdetermining Ġwhether Ġa Ġmodel Ġachieves Ġgeneral ization . </s> Con clusions </s> The Ġvariability Ġin Ġgeneral ization Ġon Ġun character istic Ġinputs Ġ( and Ġthus , Ġthe Ġextent Ġof Ġlinguistic Ġgeneral ization ) Ġgiven Ġdifferent Ġrandom Ġseeds Ġis Ġalarming , Ġparticularly Ġgiven Ġthe Ġfact Ġthat Ġthe Ġstandard Ġtest Ġset Ġperformance Ġremains Ġmostly Ġthe Ġsame Ġregardless . ĠThe Ġtask Ġpresented Ġhere Ġwas Ġeasy Ġand Ġsimple Ġto Ġanalyze , Ġhowever , Ġfuture Ġwork Ġmay Ġbe Ġdone Ġon Ġnatural Ġlanguage Ġtasks . ĠIf Ġthese Ġproperties Ġhold Ġit Ġmight Ġindicate Ġthat Ġa Ġnew Ġevaluation Ġparadigm Ġfor ĠN LP Ġshould Ġbe Ġpushed ; Ġone Ġthat Ġemphasizes Ġperformance Ġon Ġun character istic Ġ( but Ġstruct urally Ġsound ) Ġinputs Ġin Ġaddition Ġto Ġthe Ġdata Ġtypically Ġseen Ġin Ġtraining . </s> Acknowled gements </s> This Ġwork Ġis Ġsupported Ġin Ġpart Ġby Ġthe ĠNational ĠScience ĠFoundation Ġunder ĠGrant ĠI IS - 16 179 69 .