the IMDb movie review dataset BIBREF17
EXTRACTIVE
IMDb movie review
EXTRACTIVE
We evaluate the QRNN architecture on a popular document-level sentiment classification benchmark, the IMDb movie review dataset BIBREF17 . The dataset consists of a balanced sample of 25,000 positive and 25,000 negative reviews, divided into equal-size train and test sets, with an average document length of 231 words BIBREF18 . We compare only to other results that do not make use of additional unlabeled data (thus excluding e.g., BIBREF19 ).
We evaluate the QRNN architecture on a popular document-level sentiment classification benchmark, the IMDb movie review dataset BIBREF17 . The dataset consists of a balanced sample of 25,000 positive and 25,000 negative reviews, divided into equal-size train and test sets, with an average document length of 231 words BIBREF18 . We compare only to other results that do not make use of additional unlabeled data (thus excluding e.g., BIBREF19 ).
1611.01576
Introduction </s> Rec urrent Ġneural Ġnetworks Ġ( R NN s ), Ġincluding Ġg ated Ġvariants Ġsuch Ġas Ġthe Ġlong Ġshort - term Ġmemory Ġ( L ST M ) ĠB IB REF 0 Ġhave Ġbecome Ġthe Ġstandard Ġmodel Ġarchitecture Ġfor Ġdeep Ġlearning Ġapproaches Ġto Ġsequence Ġmodeling Ġtasks . ĠR NN s Ġrepeatedly Ġapply Ġa Ġfunction Ġwith Ġtrain able Ġparameters Ġto Ġa Ġhidden Ġstate . ĠRec urrent Ġlayers Ġcan Ġalso Ġbe Ġstacked , Ġincreasing Ġnetwork Ġdepth , Ġrepresent ational Ġpower Ġand Ġoften Ġaccuracy . ĠR NN Ġapplications Ġin Ġthe Ġnatural Ġlanguage Ġdomain Ġrange Ġfrom Ġsentence Ġclassification ĠB IB REF 1 Ġto Ġword - Ġand Ġcharacter - level Ġlanguage Ġmodeling ĠB IB REF 2 Ġ. ĠR NN s Ġare Ġalso Ġcommonly Ġthe Ġbasic Ġbuilding Ġblock Ġfor Ġmore Ġcomplex Ġmodels Ġfor Ġtasks Ġsuch Ġas Ġmachine Ġtranslation ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġor Ġquestion Ġanswering ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ. ĠUnfortunately Ġstandard ĠR NN s , Ġincluding ĠL ST Ms , Ġare Ġlimited Ġin Ġtheir Ġcapability Ġto Ġhandle Ġtasks Ġinvolving Ġvery Ġlong Ġsequences , Ġsuch Ġas Ġdocument Ġclassification Ġor Ġcharacter - level Ġmachine Ġtranslation , Ġas Ġthe Ġcomputation Ġof Ġfeatures Ġor Ġstates Ġfor Ġdifferent Ġparts Ġof Ġthe Ġdocument Ġcannot Ġoccur Ġin Ġparallel . </s> Con v olution al Ġneural Ġnetworks Ġ( CNN s ) ĠB IB REF 8 Ġ, Ġthough Ġmore Ġpopular Ġon Ġtasks Ġinvolving Ġimage Ġdata , Ġhave Ġalso Ġbeen Ġapplied Ġto Ġsequence Ġencoding Ġtasks ĠB IB REF 9 Ġ. ĠSuch Ġmodels Ġapply Ġtime - inv ari ant Ġfilter Ġfunctions Ġin Ġparallel Ġto Ġwindows Ġalong Ġthe Ġinput Ġsequence . ĠCNN s Ġpossess Ġseveral Ġadvantages Ġover Ġrecurrent Ġmodels , Ġincluding Ġincreased Ġparallel ism Ġand Ġbetter Ġscaling Ġto Ġlong Ġsequences Ġsuch Ġas Ġthose Ġoften Ġseen Ġwith Ġcharacter - level Ġlanguage Ġdata . ĠConv olution al Ġmodels Ġfor Ġsequence Ġprocessing Ġhave Ġbeen Ġmore Ġsuccessful Ġwhen Ġcombined Ġwith ĠR NN Ġlayers Ġin Ġa Ġhybrid Ġarchitecture ĠB IB REF 10 Ġ, Ġbecause Ġtraditional Ġmax - Ġand Ġaverage - pool ing Ġapproaches Ġto Ġcombining Ġconv olution al Ġfeatures Ġacross Ġtim est eps Ġassume Ġtime Ġinvari ance Ġand Ġhence Ġcannot Ġmake Ġfull Ġuse Ġof Ġlarge - scale Ġsequence Ġorder Ġinformation . </s> We Ġpresent Ġquasi - rec urrent Ġneural Ġnetworks Ġfor Ġneural Ġsequence Ġmodeling . ĠQR NN s Ġaddress Ġboth Ġdrawbacks Ġof Ġstandard Ġmodels : Ġlike ĠCNN s , ĠQR NN s Ġallow Ġfor Ġparallel Ġcomputation Ġacross Ġboth Ġtim est ep Ġand Ġmin ib atch Ġdimensions , Ġenabling Ġhigh Ġthroughput Ġand Ġgood Ġscaling Ġto Ġlong Ġsequences . ĠLike ĠR NN s , ĠQR NN s Ġallow Ġthe Ġoutput Ġto Ġdepend Ġon Ġthe Ġoverall Ġorder Ġof Ġelements Ġin Ġthe Ġsequence . ĠWe Ġdescribe ĠQR NN Ġvariants Ġtailored Ġto Ġseveral Ġnatural Ġlanguage Ġtasks , Ġincluding Ġdocument - level Ġsentiment Ġclassification , Ġlanguage Ġmodeling , Ġand Ġcharacter - level Ġmachine Ġtranslation . ĠThese Ġmodels Ġoutper form Ġstrong ĠL ST M Ġbas elines Ġon Ġall Ġthree Ġtasks Ġwhile Ġdramatically Ġreducing Ġcomputation Ġtime . </s> Model </s> Each Ġlayer Ġof Ġa Ġquasi - rec urrent Ġneural Ġnetwork Ġconsists Ġof Ġtwo Ġkinds Ġof Ġsub comp onents , Ġanalogous Ġto Ġconv olution Ġand Ġpool ing Ġlayers Ġin ĠCNN s . ĠThe Ġconv olution al Ġcomponent , Ġlike Ġconv olution al Ġlayers Ġin ĠCNN s , Ġallows Ġfully Ġparallel Ġcomputation Ġacross Ġboth Ġmin ib atches Ġand Ġspatial Ġdimensions , Ġin Ġthis Ġcase Ġthe Ġsequence Ġdimension . ĠThe Ġpool ing Ġcomponent , Ġlike Ġpool ing Ġlayers Ġin ĠCNN s , Ġlacks Ġtrain able Ġparameters Ġand Ġallows Ġfully Ġparallel Ġcomputation Ġacross Ġmin ib atch Ġand Ġfeature Ġdimensions . </s> Given Ġan Ġinput Ġsequence ĠIN LINE FORM 0 Ġof ĠIN LINE FORM 1 ĠIN LINE FORM 2 Ġ- dimensional Ġvectors ĠIN LINE FORM 3 Ġ, Ġthe Ġconv olution al Ġsub component Ġof Ġa ĠQR NN Ġperforms Ġconv olutions Ġin Ġthe Ġtim est ep Ġdimension Ġwith Ġa Ġbank Ġof ĠIN LINE FORM 4 Ġfilters , Ġproducing Ġa Ġsequence ĠIN LINE FORM 5 Ġof ĠIN LINE FORM 6 Ġ- dimensional Ġcandidate Ġvectors ĠIN LINE FORM 7 Ġ. ĠIn Ġorder Ġto Ġbe Ġuseful Ġfor Ġtasks Ġthat Ġinclude Ġprediction Ġof Ġthe Ġnext Ġtoken , Ġthe Ġfilters Ġmust Ġnot Ġallow Ġthe Ġcomputation Ġfor Ġany Ġgiven Ġtim est ep Ġto Ġaccess Ġinformation Ġfrom Ġfuture Ġtim est eps . ĠThat Ġis , Ġwith Ġfilters Ġof Ġwidth ĠIN LINE FORM 8 Ġ, Ġeach ĠIN LINE FORM 9 Ġdepends Ġonly Ġon ĠIN LINE FORM 10 Ġthrough ĠIN LINE FORM 11 Ġ. ĠThis Ġconcept , Ġknown Ġas Ġa Ġmasked Ġconv olution ĠB IB REF 11 Ġ, Ġis Ġimplemented Ġby Ġpadding Ġthe Ġinput Ġto Ġthe Ġleft Ġby Ġthe Ġconv olution 's Ġfilter Ġsize Ġminus Ġone . </s> We Ġapply Ġadditional Ġconv olutions Ġwith Ġseparate Ġfilter Ġbanks Ġto Ġobtain Ġsequences Ġof Ġvectors Ġfor Ġthe Ġelement wise Ġgates Ġthat Ġare Ġneeded Ġfor Ġthe Ġpool ing Ġfunction . ĠWhile Ġthe Ġcandidate Ġvectors Ġare Ġpassed Ġthrough Ġa ĠIN LINE FORM 0 Ġnon linear ity , Ġthe Ġgates Ġuse Ġan Ġelement wise Ġs igm oid . ĠIf Ġthe Ġpool ing Ġfunction Ġrequires Ġa Ġforget Ġgate ĠIN LINE FORM 1 Ġand Ġan Ġoutput Ġgate ĠIN LINE FORM 2 Ġat Ġeach Ġtim est ep , Ġthe Ġfull Ġset Ġof Ġcomput ations Ġin Ġthe Ġconv olution al Ġcomponent Ġis Ġthen : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, Ġand ĠIN LINE FORM 2 Ġ, Ġeach Ġin ĠIN LINE FORM 3 Ġ, Ġare Ġthe Ġconv olution al Ġfilter Ġbanks Ġand ĠIN LINE FORM 4 Ġdenotes Ġa Ġmasked Ġconv olution Ġalong Ġthe Ġtim est ep Ġdimension . ĠNote Ġthat Ġif Ġthe Ġfilter Ġwidth Ġis Ġ2 , Ġthese Ġequations Ġreduce Ġto Ġthe ĠL ST M - like ĠDIS PLAY FORM 0 </s> Con v olution Ġfilters Ġof Ġlarger Ġwidth Ġeffectively Ġcompute Ġhigher ĠIN LINE FORM 0 Ġ- gram Ġfeatures Ġat Ġeach Ġtim est ep ; Ġthus Ġlarger Ġwidth s Ġare Ġespecially Ġimportant Ġfor Ġcharacter - level Ġtasks . </s> Su itable Ġfunctions Ġfor Ġthe Ġpool ing Ġsub component Ġcan Ġbe Ġconstructed Ġfrom Ġthe Ġfamiliar Ġelement wise Ġgates Ġof Ġthe Ġtraditional ĠL ST M Ġcell . ĠWe Ġseek Ġa Ġfunction Ġcontrolled Ġby Ġgates Ġthat Ġcan Ġmix Ġstates Ġacross Ġtim est eps , Ġbut Ġwhich Ġacts Ġindependently Ġon Ġeach Ġchannel Ġof Ġthe Ġstate Ġvector . ĠThe Ġsimplest Ġoption , Ġwhich ĠB IB REF 12 Ġterm ĠâĢ ľ d ynamic Ġaverage Ġpool ing âĢ Ŀ , Ġuses Ġonly Ġa Ġforget Ġgate : ĠDIS PLAY FORM 0 </s> We Ġterm Ġthese Ġthree Ġoptions Ġf - pool ing , Ġfo - pool ing , Ġand Ġif o - pool ing Ġrespectively ; Ġin Ġeach Ġcase Ġwe Ġinitialize ĠIN LINE FORM 0 Ġor ĠIN LINE FORM 1 Ġto Ġzero . ĠAlthough Ġthe Ġrecurrent Ġparts Ġof Ġthese Ġfunctions Ġmust Ġbe Ġcalculated Ġfor Ġeach Ġtim est ep Ġin Ġsequence , Ġtheir Ġsimplicity Ġand Ġparallel ism Ġalong Ġfeature Ġdimensions Ġmeans Ġthat , Ġin Ġpractice , Ġevaluating Ġthem Ġover Ġeven Ġlong Ġsequences Ġrequires Ġa Ġnegligible Ġamount Ġof Ġcomputation Ġtime . </s> A Ġsingle ĠQR NN Ġlayer Ġthus Ġperforms Ġan Ġinput - dependent Ġpool ing , Ġfollowed Ġby Ġa Ġg ated Ġlinear Ġcombination Ġof Ġconv olution al Ġfeatures . ĠAs Ġwith Ġconv olution al Ġneural Ġnetworks , Ġtwo Ġor Ġmore ĠQR NN Ġlayers Ġshould Ġbe Ġstacked Ġto Ġcreate Ġa Ġmodel Ġwith Ġthe Ġcapacity Ġto Ġapproximate Ġmore Ġcomplex Ġfunctions . </s> Vari ants </s> Mot ivated Ġby Ġseveral Ġcommon Ġnatural Ġlanguage Ġtasks , Ġand Ġthe Ġlong Ġhistory Ġof Ġwork Ġon Ġrelated Ġarchitectures , Ġwe Ġintroduce Ġseveral Ġextensions Ġto Ġthe Ġstacked ĠQR NN Ġdescribed Ġabove . ĠNotably , Ġmany Ġextensions Ġto Ġboth Ġrecurrent Ġand Ġconv olution al Ġmodels Ġcan Ġbe Ġapplied Ġdirectly Ġto Ġthe ĠQR NN Ġas Ġit Ġcombines Ġelements Ġof Ġboth Ġmodel Ġtypes . </s> Regular ization ĠAn Ġimportant Ġextension Ġto Ġthe Ġstacked ĠQR NN Ġis Ġa Ġrobust Ġregular ization Ġscheme Ġinspired Ġby Ġrecent Ġwork Ġin Ġregular izing ĠL ST Ms . </s> The Ġneed Ġfor Ġan Ġeffective Ġregular ization Ġmethod Ġfor ĠL ST Ms , Ġand Ġdrop out 's Ġrelative Ġlack Ġof Ġefficacy Ġwhen Ġapplied Ġto Ġrecurrent Ġconnections , Ġled Ġto Ġthe Ġdevelopment Ġof Ġrecurrent Ġdrop out Ġschemes , Ġincluding Ġvari ational Ġinference âĢĵ based Ġdrop out ĠB IB REF 13 Ġand Ġzone out ĠB IB REF 14 Ġ. ĠThese Ġschemes Ġextend Ġdrop out Ġto Ġthe Ġrecurrent Ġsetting Ġby Ġtaking Ġadvantage Ġof Ġthe Ġrepeating Ġstructure Ġof Ġrecurrent Ġnetworks , Ġproviding Ġmore Ġpowerful Ġand Ġless Ġdestructive Ġregular ization . </s> Vari ational Ġinference âĢĵ based Ġdrop out Ġlocks Ġthe Ġdrop out Ġmask Ġused Ġfor Ġthe Ġrecurrent Ġconnections Ġacross Ġtim est eps , Ġso Ġa Ġsingle ĠR NN Ġpass Ġuses Ġa Ġsingle Ġsto ch astic Ġsubset Ġof Ġthe Ġrecurrent Ġweights . ĠZone out Ġsto ch astically Ġchooses Ġa Ġnew Ġsubset Ġof Ġchannels Ġto ĠâĢ ľ zone Ġout âĢ Ŀ Ġat Ġeach Ġtim est ep ; Ġfor Ġthese Ġchannels Ġthe Ġnetwork Ġcopies Ġstates Ġfrom Ġone Ġtim est ep Ġto Ġthe Ġnext Ġwithout Ġmodification . </s> As ĠQR NN s Ġlack Ġrecurrent Ġweights , Ġthe Ġvari ational Ġinference Ġapproach Ġdoes Ġnot Ġapply . ĠThus Ġwe Ġextended Ġzone out Ġto Ġthe ĠQR NN Ġarchitecture Ġby Ġmodifying Ġthe Ġpool ing Ġfunction Ġto Ġkeep Ġthe Ġprevious Ġpool ing Ġstate Ġfor Ġa Ġsto ch astic Ġsubset Ġof Ġchannels . ĠCon venient ly , Ġthis Ġis Ġequivalent Ġto Ġsto ch astically Ġsetting Ġa Ġsubset Ġof Ġthe ĠQR NN 's ĠIN LINE FORM 0 Ġgate Ġchannels Ġto Ġ1 , Ġor Ġapplying Ġdrop out Ġon ĠIN LINE FORM 1 Ġ: ĠDIS PLAY FORM 0 </s> Thus Ġthe Ġpool ing Ġfunction Ġitself Ġneed Ġnot Ġbe Ġmodified Ġat Ġall . ĠWe Ġnote Ġthat Ġwhen Ġusing Ġan Ġoff - the - she lf Ġdrop out Ġlayer Ġin Ġthis Ġcontext , Ġit Ġis Ġimportant Ġto Ġremove Ġautomatic Ġresc aling Ġfunctionality Ġfrom Ġthe Ġimplementation Ġif Ġit Ġis Ġpresent . ĠIn Ġmany Ġexperiments , Ġwe Ġalso Ġapply Ġordinary Ġdrop out Ġbetween Ġlayers , Ġincluding Ġbetween Ġword Ġembed d ings Ġand Ġthe Ġfirst ĠQR NN Ġlayer . </s> D ense ly - Connect ed ĠL ayers ĠWe Ġcan Ġalso Ġextend Ġthe ĠQR NN Ġarchitecture Ġusing Ġtechniques Ġintroduced Ġfor Ġconv olution al Ġnetworks . ĠFor Ġsequence Ġclassification Ġtasks , Ġwe Ġfound Ġit Ġhelpful Ġto Ġuse Ġskip - connect ions Ġbetween Ġevery ĠQR NN Ġlayer , Ġa Ġtechnique Ġtermed ĠâĢ ľ d ense Ġconv olution âĢ Ŀ Ġby ĠB IB REF 15 Ġ. ĠWhere Ġtraditional Ġfeed - forward Ġor Ġconv olution al Ġnetworks Ġhave Ġconnections Ġonly Ġbetween Ġsubsequent Ġlayers , Ġa ĠâĢ ľ D ense Net âĢ Ŀ Ġwith ĠIN LINE FORM 0 Ġlayers Ġhas Ġfeed - forward Ġor Ġconv olution al Ġconnections Ġbetween Ġevery Ġpair Ġof Ġlayers , Ġfor Ġa Ġtotal Ġof ĠIN LINE FORM 1 Ġ. ĠThis Ġcan Ġimprove Ġgradient Ġflow Ġand Ġconvergence Ġproperties , Ġespecially Ġin Ġdeeper Ġnetworks , Ġalthough Ġit Ġrequires Ġa Ġparameter Ġcount Ġthat Ġis Ġquad r atic Ġin Ġthe Ġnumber Ġof Ġlayers . </s> When Ġapplying Ġthis Ġtechnique Ġto Ġthe ĠQR NN , Ġwe Ġinclude Ġconnections Ġbetween Ġthe Ġinput Ġembed d ings Ġand Ġevery ĠQR NN Ġlayer Ġand Ġbetween Ġevery Ġpair Ġof ĠQR NN Ġlayers . ĠThis Ġis Ġequivalent Ġto Ġconc aten ating Ġeach ĠQR NN Ġlayer 's Ġinput Ġto Ġits Ġoutput Ġalong Ġthe Ġchannel Ġdimension Ġbefore Ġfeeding Ġthe Ġstate Ġinto Ġthe Ġnext Ġlayer . ĠThe Ġoutput Ġof Ġthe Ġlast Ġlayer Ġalone Ġis Ġthen Ġused Ġas Ġthe Ġoverall Ġencoding Ġresult . </s> Enc oder âĢĵ Dec oder ĠModels ĠTo Ġdemonstrate Ġthe Ġgener ality Ġof ĠQR NN s , Ġwe Ġextend Ġthe Ġmodel Ġarchitecture Ġto Ġsequence - to - sequence Ġtasks , Ġsuch Ġas Ġmachine Ġtranslation , Ġby Ġusing Ġa ĠQR NN Ġas Ġenc oder Ġand Ġa Ġmodified ĠQR NN , Ġenhanced Ġwith Ġattention , Ġas Ġdec oder . ĠThe Ġmotivation Ġfor Ġmodifying Ġthe Ġdec oder Ġis Ġthat Ġsimply Ġfeeding Ġthe Ġlast Ġenc oder Ġhidden Ġstate Ġ( the Ġoutput Ġof Ġthe Ġenc oder 's Ġpool ing Ġlayer ) Ġinto Ġthe Ġdec oder 's Ġrecurrent Ġpool ing Ġlayer , Ġanalog ously Ġto Ġconventional Ġrecurrent Ġenc oder âĢĵ dec oder Ġarchitectures , Ġwould Ġnot Ġallow Ġthe Ġenc oder Ġstate Ġto Ġaffect Ġthe Ġgate Ġor Ġupdate Ġvalues Ġthat Ġare Ġprovided Ġto Ġthe Ġdec oder 's Ġpool ing Ġlayer . ĠThis Ġwould Ġsubstantially Ġlimit Ġthe Ġrepresent ational Ġpower Ġof Ġthe Ġdec oder . </s> Instead , Ġthe Ġoutput Ġof Ġeach Ġdec oder ĠQR NN Ġlayer 's Ġconv olution Ġfunctions Ġis Ġsupplemented Ġat Ġevery Ġtim est ep Ġwith Ġthe Ġfinal Ġenc oder Ġhidden Ġstate . ĠThis Ġis Ġaccomplished Ġby Ġadding Ġthe Ġresult Ġof Ġthe Ġconv olution Ġfor Ġlayer ĠIN LINE FORM 0 Ġ( e . g ., ĠIN LINE FORM 1 Ġ, Ġin ĠIN LINE FORM 2 Ġ) Ġwith Ġbroadcasting Ġto Ġa Ġlin early Ġprojected Ġcopy Ġof Ġlayer ĠIN LINE FORM 3 Ġ' s Ġlast Ġenc oder Ġstate Ġ( e . g ., ĠIN LINE FORM 4 Ġ, Ġin ĠIN LINE FORM 5 Ġ): ĠDIS PLAY FORM 0 </s> where Ġthe Ġt ilde Ġdenotes Ġthat ĠIN LINE FORM 0 Ġis Ġan Ġenc oder Ġvariable . ĠEnc oder âĢĵ dec oder Ġmodels Ġwhich Ġoperate Ġon Ġlong Ġsequences Ġare Ġmade Ġsignificantly Ġmore Ġpowerful Ġwith Ġthe Ġaddition Ġof Ġsoft Ġattention ĠB IB REF 3 Ġ, Ġwhich Ġremoves Ġthe Ġneed Ġfor Ġthe Ġentire Ġinput Ġrepresentation Ġto Ġfit Ġinto Ġa Ġfixed - length Ġencoding Ġvector . ĠIn Ġour Ġexperiments , Ġwe Ġcomputed Ġan Ġattention al Ġsum Ġof Ġthe Ġenc oder 's Ġlast Ġlayer 's Ġhidden Ġstates . ĠWe Ġused Ġthe Ġdot Ġproducts Ġof Ġthese Ġenc oder Ġhidden Ġstates Ġwith Ġthe Ġdec oder 's Ġlast Ġlayer 's Ġun - g ated Ġhidden Ġstates , Ġapplying Ġa ĠIN LINE FORM 1 Ġalong Ġthe Ġenc oder Ġtim est eps , Ġto Ġweight Ġthe Ġenc oder Ġstates Ġinto Ġan Ġattention al Ġsum ĠIN LINE FORM 2 Ġfor Ġeach Ġdec oder Ġtim est ep . ĠThis Ġcontext , Ġand Ġthe Ġdec oder Ġstate , Ġare Ġthen Ġfed Ġinto Ġa Ġlinear Ġlayer Ġfollowed Ġby Ġthe Ġoutput Ġgate : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġlast Ġlayer . </s> While Ġthe Ġfirst Ġstep Ġof Ġthis Ġattention Ġprocedure Ġis Ġquad r atic Ġin Ġthe Ġsequence Ġlength , Ġin Ġpractice Ġit Ġtakes Ġsignificantly Ġless Ġcomputation Ġtime Ġthan Ġthe Ġmodel 's Ġlinear Ġand Ġconv olution al Ġlayers Ġdue Ġto Ġthe Ġsimple Ġand Ġhighly Ġparallel Ġdot - product Ġscoring Ġfunction . </s> Exper iments </s> We Ġevaluate Ġthe Ġperformance Ġof Ġthe ĠQR NN Ġon Ġthree Ġdifferent Ġnatural Ġlanguage Ġtasks : Ġdocument - level Ġsentiment Ġclassification , Ġlanguage Ġmodeling , Ġand Ġcharacter - based Ġneural Ġmachine Ġtranslation . ĠOur ĠQR NN Ġmodels Ġoutper form ĠL ST M - based Ġmodels Ġof Ġequal Ġhidden Ġsize Ġon Ġall Ġthree Ġtasks Ġwhile Ġdramatically Ġimproving Ġcomputation Ġspeed . ĠExper iments Ġwere Ġimplemented Ġin ĠCh ainer ĠB IB REF 16 Ġ. </s> Sent iment ĠClassification </s> We Ġevaluate Ġthe ĠQR NN Ġarchitecture Ġon Ġa Ġpopular Ġdocument - level Ġsentiment Ġclassification Ġbenchmark , Ġthe ĠIM Db Ġmovie Ġreview Ġdataset ĠB IB REF 17 Ġ. ĠThe Ġdataset Ġconsists Ġof Ġa Ġbalanced Ġsample Ġof Ġ25 , 000 Ġpositive Ġand Ġ25 , 000 Ġnegative Ġreviews , Ġdivided Ġinto Ġequal - size Ġtrain Ġand Ġtest Ġsets , Ġwith Ġan Ġaverage Ġdocument Ġlength Ġof Ġ231 Ġwords ĠB IB REF 18 Ġ. ĠWe Ġcompare Ġonly Ġto Ġother Ġresults Ġthat Ġdo Ġnot Ġmake Ġuse Ġof Ġadditional Ġunl abel ed Ġdata Ġ( thus Ġexcluding Ġe . g ., ĠB IB REF 19 Ġ). </s> Our Ġbest Ġperformance Ġon Ġa Ġheld - out Ġdevelopment Ġset Ġwas Ġachieved Ġusing Ġa Ġfour - layer Ġdensely - connected ĠQR NN Ġwith Ġ256 Ġunits Ġper Ġlayer Ġand Ġword Ġvectors Ġinitialized Ġusing Ġ300 - dimensional Ġc ased ĠGl o Ve Ġembed d ings ĠB IB REF 20 Ġ. ĠDrop out Ġof Ġ0 . 3 Ġwas Ġapplied Ġbetween Ġlayers , Ġand Ġwe Ġused ĠIN LINE FORM 0 Ġregular ization Ġof ĠIN LINE FORM 1 Ġ. ĠOptim ization Ġwas Ġperformed Ġon Ġmin ib atches Ġof Ġ24 Ġexamples Ġusing ĠR MS prop ĠB IB REF 21 Ġwith Ġlearning Ġrate Ġof ĠIN LINE FORM 2 Ġ, ĠIN LINE FORM 3 Ġ, Ġand ĠIN LINE FORM 4 Ġ. </s> Small Ġbatch Ġsizes Ġand Ġlong Ġsequence Ġlengths Ġprovide Ġan Ġideal Ġsituation Ġfor Ġdemonstrating Ġthe ĠQR NN 's Ġperformance Ġadvantages Ġover Ġtraditional Ġrecurrent Ġarchitectures . ĠWe Ġobserved Ġa Ġspeed up Ġof Ġ3 . 2 x Ġon ĠIM Db Ġtrain Ġtime Ġper Ġepoch Ġcompared Ġto Ġthe Ġoptimized ĠL ST M Ġimplementation Ġprovided Ġin ĠNVIDIA 's Ġcu D NN Ġlibrary . ĠFor Ġspecific Ġbatch Ġsizes Ġand Ġsequence Ġlengths , Ġa Ġ16 x Ġspeed Ġgain Ġis Ġpossible . ĠFigure ĠFIG REF 15 Ġprovides Ġextensive Ġspeed Ġcomparisons . </s> In ĠFigure ĠFIG REF 12 Ġ, Ġwe Ġvisualize Ġthe Ġhidden Ġstate Ġvectors ĠIN LINE FORM 0 Ġof Ġthe Ġfinal ĠQR NN Ġlayer Ġon Ġpart Ġof Ġan Ġexample Ġfrom Ġthe ĠIM Db Ġdataset . ĠEven Ġwithout Ġany Ġpost - processing , Ġchanges Ġin Ġthe Ġhidden Ġstate Ġare Ġvisible Ġand Ġinterpret able Ġin Ġregards Ġto Ġthe Ġinput . ĠThis Ġis Ġa Ġconsequence Ġof Ġthe Ġelement wise Ġnature Ġof Ġthe Ġrecurrent Ġpool ing Ġfunction , Ġwhich Ġdelays Ġdirect Ġinteraction Ġbetween Ġdifferent Ġchannels Ġof Ġthe Ġhidden Ġstate Ġuntil Ġthe Ġcomputation Ġof Ġthe Ġnext ĠQR NN Ġlayer . </s> Language ĠModel ing </s> We Ġreplicate Ġthe Ġlanguage Ġmodeling Ġexperiment Ġof ĠB IB REF 2 Ġand ĠB IB REF 13 Ġto Ġbenchmark Ġthe ĠQR NN Ġarchitecture Ġfor Ġnatural Ġlanguage Ġsequence Ġprediction . ĠThe Ġexperiment Ġuses Ġa Ġstandard Ġpre process ed Ġversion Ġof Ġthe ĠPenn ĠTree bank Ġ( PT B ) Ġby ĠB IB REF 25 Ġ. </s> We Ġimplemented Ġa Ġg ated ĠQR NN Ġmodel Ġwith Ġmedium Ġhidden Ġsize : Ġ2 Ġlayers Ġwith Ġ640 Ġunits Ġin Ġeach Ġlayer . ĠBoth ĠQR NN Ġlayers Ġuse Ġa Ġconv olution al Ġfilter Ġwidth ĠIN LINE FORM 0 Ġof Ġtwo Ġtim est eps . ĠWhile Ġthe ĠâĢ ľ medium âĢ Ŀ Ġmodels Ġused Ġin Ġother Ġwork ĠB IB REF 2 Ġ, ĠB IB REF 13 Ġconsist Ġof Ġ650 Ġunits Ġin Ġeach Ġlayer , Ġit Ġwas Ġmore Ġcomput ationally Ġconvenient Ġto Ġuse Ġa Ġmultiple Ġof Ġ32 . ĠAs Ġthe ĠPenn ĠTree bank Ġis Ġa Ġrelatively Ġsmall Ġdataset , Ġpreventing Ġover fitting Ġis Ġof Ġconsiderable Ġimportance Ġand Ġa Ġmajor Ġfocus Ġof Ġrecent Ġresearch . ĠIt Ġis Ġnot Ġobvious Ġin Ġadvance Ġwhich Ġof Ġthe Ġmany ĠR NN Ġregular ization Ġschemes Ġwould Ġperform Ġwell Ġwhen Ġapplied Ġto Ġthe ĠQR NN . ĠOur Ġtests Ġshowed Ġencouraging Ġresults Ġfrom Ġzone out Ġapplied Ġto Ġthe ĠQR NN 's Ġrecurrent Ġpool ing Ġlayer , Ġimplemented Ġas Ġdescribed Ġin ĠSection ĠSEC REF 5 Ġ. </s> The Ġexperimental Ġsettings Ġlargely Ġfollowed Ġthe ĠâĢ ľ medium âĢ Ŀ Ġsetup Ġof ĠB IB REF 2 Ġ. ĠOptim ization Ġwas Ġperformed Ġby Ġsto ch astic Ġgradient Ġdescent Ġ( SG D ) Ġwithout Ġmomentum . ĠThe Ġlearning Ġrate Ġwas Ġset Ġat Ġ1 Ġfor Ġsix Ġepoch s , Ġthen Ġdec ayed Ġby Ġ0 . 95 Ġfor Ġeach Ġsubsequent Ġepoch , Ġfor Ġa Ġtotal Ġof Ġ72 Ġepoch s . ĠWe Ġadditionally Ġused ĠIN LINE FORM 0 Ġregular ization Ġof ĠIN LINE FORM 1 Ġand Ġresc aled Ġgrad ients Ġwith Ġnorm Ġabove Ġ10 . ĠZone out Ġwas Ġapplied Ġby Ġperforming Ġdrop out Ġwith Ġratio Ġ0 . 1 Ġon Ġthe Ġforget Ġgates Ġof Ġthe ĠQR NN , Ġwithout Ġresc aling Ġthe Ġoutput Ġof Ġthe Ġdrop out Ġfunction . ĠBat ches Ġconsist Ġof Ġ20 Ġexamples , Ġeach Ġ105 Ġtim est eps . </s> Comp aring Ġour Ġresults Ġon Ġthe Ġg ated ĠQR NN Ġwith Ġzone out Ġto Ġthe Ġresults Ġof ĠL ST Ms Ġwith Ġboth Ġordinary Ġand Ġvari ational Ġdrop out Ġin ĠTable ĠT AB REF 14 Ġ, Ġwe Ġsee Ġthat Ġthe ĠQR NN Ġis Ġhighly Ġcompetitive . ĠThe ĠQR NN Ġwithout Ġzone out Ġstrongly Ġoutper forms Ġboth Ġour Ġmedium ĠL ST M Ġand Ġthe Ġmedium ĠL ST M Ġof ĠB IB REF 2 Ġwhich Ġdo Ġnot Ġuse Ġrecurrent Ġdrop out Ġand Ġis Ġeven Ġcompetitive Ġwith Ġvari ational ĠL ST Ms . ĠThis Ġmay Ġbe Ġdue Ġto Ġthe Ġlimited Ġcomputational Ġcapacity Ġthat Ġthe ĠQR NN 's Ġpool ing Ġlayer Ġhas Ġrelative Ġto Ġthe ĠL ST M 's Ġrecurrent Ġweights , Ġproviding Ġstructural Ġregular ization Ġover Ġthe Ġrec urrence . </s> Without Ġzone out , Ġearly Ġstopping Ġbased Ġupon Ġvalidation Ġloss Ġwas Ġrequired Ġas Ġthe ĠQR NN Ġwould Ġbegin Ġover fitting . ĠBy Ġapplying Ġa Ġsmall Ġamount Ġof Ġzone out Ġ( ĠIN LINE FORM 0 Ġ), Ġno Ġearly Ġstopping Ġis Ġrequired Ġand Ġthe ĠQR NN Ġachieves Ġcompetitive Ġlevels Ġof Ġperplex ity Ġto Ġthe Ġvari ational ĠL ST M Ġof ĠB IB REF 13 Ġ, Ġwhich Ġhad Ġvari ational Ġinference Ġbased Ġdrop out Ġof Ġ0 . 2 Ġapplied Ġrecurrent ly . ĠTheir Ġbest Ġperforming Ġvariation Ġalso Ġused ĠMonte ĠCarlo Ġ( MC ) Ġdrop out Ġaveraging Ġat Ġtest Ġtime Ġof Ġ1000 Ġdifferent Ġmasks , Ġmaking Ġit Ġcomput ationally Ġmore Ġexpensive Ġto Ġrun . </s> When Ġtraining Ġon Ġthe ĠPT B Ġdataset Ġwith Ġan ĠNVIDIA ĠK 40 ĠGPU , Ġwe Ġfound Ġthat Ġthe ĠQR NN Ġis Ġsubstantially Ġfaster Ġthan Ġa Ġstandard ĠL ST M , Ġeven Ġwhen Ġcomparing Ġagainst Ġthe Ġoptimized Ġcu D NN ĠL ST M . ĠIn ĠFigure ĠFIG REF 15 Ġwe Ġprovide Ġa Ġbreakdown Ġof Ġthe Ġtime Ġtaken Ġfor ĠCh ainer 's Ġdefault ĠL ST M , Ġthe Ġcu D NN ĠL ST M , Ġand ĠQR NN Ġto Ġperform Ġa Ġfull Ġforward Ġand Ġbackward Ġpass Ġon Ġa Ġsingle Ġbatch Ġduring Ġtraining Ġof Ġthe ĠR NN ĠLM Ġon ĠPT B . ĠFor Ġboth ĠL ST M Ġimplementations , Ġrunning Ġtime Ġwas Ġdominated Ġby Ġthe ĠR NN Ġcomput ations , Ġeven Ġwith Ġthe Ġhighly Ġoptimized Ġcu D NN Ġimplementation . ĠFor Ġthe ĠQR NN Ġimplementation , Ġhowever , Ġthe ĠâĢ ľ R NN âĢ Ŀ Ġlayers Ġare Ġno Ġlonger Ġthe Ġbottleneck . ĠIndeed , Ġthere Ġare Ġdiminishing Ġreturns Ġfrom Ġfurther Ġoptimization Ġof Ġthe ĠQR NN Ġitself Ġas Ġthe Ġsoft max Ġand Ġoptimization Ġoverhead Ġtake Ġequal Ġor Ġgreater Ġtime . ĠNote Ġthat Ġthe Ġsoft max , Ġover Ġa Ġvocabulary Ġsize Ġof Ġonly Ġ10 , 000 Ġwords , Ġis Ġrelatively Ġsmall ; Ġfor Ġtasks Ġwith Ġlarger Ġvoc ab ular ies , Ġthe Ġsoft max Ġwould Ġlikely Ġdominate Ġcomputation Ġtime . </s> It Ġis Ġalso Ġimportant Ġto Ġnote Ġthat Ġthe Ġcu D NN Ġlibrary 's ĠR NN Ġprim itives Ġdo Ġnot Ġnative ly Ġsupport Ġany Ġform Ġof Ġrecurrent Ġdrop out . ĠThat Ġis , Ġrunning Ġan ĠL ST M Ġthat Ġuses Ġa Ġstate - of - the - art Ġregular ization Ġscheme Ġat Ġcu D NN - like Ġspeeds Ġwould Ġlikely Ġrequire Ġan Ġentirely Ġcustom Ġkernel . </s> Character - level ĠNeural ĠMachine ĠTranslation </s> We Ġevaluate Ġthe Ġsequence - to - sequence ĠQR NN Ġarchitecture Ġdescribed Ġin ĠSEC REF 5 Ġon Ġa Ġchallenging Ġneural Ġmachine Ġtranslation Ġtask , ĠI W SL T ĠGerman âĢĵ English Ġspoken - domain Ġtranslation , Ġapplying Ġfully Ġcharacter - level Ġsegment ation . ĠThis Ġdataset Ġconsists Ġof Ġ209 , 772 Ġsentence Ġpairs Ġof Ġparallel Ġtraining Ġdata Ġfrom Ġtransc ribed ĠTED Ġand ĠTED x Ġpresentations , Ġwith Ġa Ġmean Ġsentence Ġlength Ġof Ġ103 Ġcharacters Ġfor ĠGerman Ġand Ġ93 Ġfor ĠEnglish . ĠWe Ġremove Ġtraining Ġsentences Ġwith Ġmore Ġthan Ġ300 Ġcharacters Ġin ĠEnglish Ġor ĠGerman , Ġand Ġuse Ġa Ġunified Ġvocabulary Ġof Ġ187 ĠUnicode Ġcode Ġpoints . </s> Our Ġbest Ġperformance Ġon Ġa Ġdevelopment Ġset Ġ( TED . t st 2013 ) Ġwas Ġachieved Ġusing Ġa Ġfour - layer Ġenc oder âĢĵ dec oder ĠQR NN Ġwith Ġ320 Ġunits Ġper Ġlayer , Ġno Ġdrop out Ġor ĠIN LINE FORM 0 Ġregular ization , Ġand Ġgradient Ġresc aling Ġto Ġa Ġmaximum Ġmagnitude Ġof Ġ5 . ĠInput s Ġwere Ġsupplied Ġto Ġthe Ġenc oder Ġreversed , Ġwhile Ġthe Ġenc oder Ġconv olutions Ġwere Ġnot Ġmasked . ĠThe Ġfirst Ġenc oder Ġlayer Ġused Ġconv olution al Ġfilter Ġwidth ĠIN LINE FORM 1 Ġ, Ġwhile Ġthe Ġother Ġenc oder Ġlayers Ġused ĠIN LINE FORM 2 Ġ. ĠOptim ization Ġwas Ġperformed Ġfor Ġ10 Ġepoch s Ġon Ġmin ib atches Ġof Ġ16 Ġexamples Ġusing ĠAdam ĠB IB REF 28 Ġwith ĠIN LINE FORM 3 Ġ, ĠIN LINE FORM 4 Ġ, ĠIN LINE FORM 5 Ġ, Ġand ĠIN LINE FORM 6 Ġ. ĠDec oding Ġwas Ġperformed Ġusing Ġbeam Ġsearch Ġwith Ġbeam Ġwidth Ġ8 Ġand Ġlength Ġnormal ization ĠIN LINE FORM 7 Ġ. ĠThe Ġmodified Ġlog - pro b ability Ġranking Ġcriterion Ġis Ġprovided Ġin Ġthe Ġappendix . </s> Results Ġusing Ġthis Ġarchitecture Ġwere Ġcompared Ġto Ġan Ġequal - sized Ġfour - layer Ġenc oder âĢĵ dec oder ĠL ST M Ġwith Ġattention , Ġapplying Ġdrop out Ġof Ġ0 . 2 . ĠWe Ġagain Ġoptimized Ġusing ĠAdam ; Ġother Ġhyper param eters Ġwere Ġequal Ġto Ġtheir Ġvalues Ġfor Ġthe ĠQR NN Ġand Ġthe Ġsame Ġbeam Ġsearch Ġprocedure Ġwas Ġapplied . ĠTable ĠT AB REF 17 Ġshows Ġthat Ġthe ĠQR NN Ġoutper formed Ġthe Ġcharacter - level ĠL ST M , Ġalmost Ġmatching Ġthe Ġperformance Ġof Ġa Ġword - level Ġattention al Ġbaseline . </s> Related ĠWork </s> Expl oring Ġalternatives Ġto Ġtraditional ĠR NN s Ġfor Ġsequence Ġtasks Ġis Ġa Ġmajor Ġarea Ġof Ġcurrent Ġresearch . ĠQu asi - rec urrent Ġneural Ġnetworks Ġare Ġrelated Ġto Ġseveral Ġsuch Ġrecently Ġdescribed Ġmodels , Ġespecially Ġthe Ġstrongly - ty ped Ġrecurrent Ġneural Ġnetworks Ġ( T - R NN ) Ġintroduced Ġby ĠB IB REF 12 Ġ. ĠWhile Ġthe Ġmotivation Ġand Ġconstraints Ġdescribed Ġin Ġthat Ġwork Ġare Ġdifferent , ĠB IB REF 12 Ġ' s Ġconcepts Ġof ĠâĢ ľ learn ware âĢ Ŀ Ġand ĠâĢ ľ f irm ware âĢ Ŀ Ġparallel Ġour Ġdiscussion Ġof Ġconv olution - like Ġand Ġpool ing - like Ġsub comp onents . ĠAs Ġthe Ġuse Ġof Ġa Ġfully Ġconnected Ġlayer Ġfor Ġrecurrent Ġconnections Ġviolates Ġthe Ġconstraint Ġof ĠâĢ ľ strong Ġtyping âĢ Ŀ , Ġall Ġstrongly - ty ped ĠR NN Ġarchitectures Ġ( including Ġthe ĠT - R NN , ĠT - GR U , Ġand ĠT - L ST M ) Ġare Ġalso Ġquasi - rec urrent . ĠHowever , Ġsome ĠQR NN Ġmodels Ġ( including Ġthose Ġwith Ġattention Ġor Ġskip - connect ions ) Ġare Ġnot ĠâĢ ľ strong ly Ġtyped âĢ Ŀ . ĠIn Ġparticular , Ġa ĠT - R NN Ġdiffers Ġfrom Ġa ĠQR NN Ġas Ġdescribed Ġin Ġthis Ġpaper Ġwith Ġfilter Ġsize Ġ1 Ġand Ġf - pool ing Ġonly Ġin Ġthe Ġabsence Ġof Ġan Ġactivation Ġfunction Ġon ĠIN LINE FORM 0 Ġ. ĠSimilarly , ĠT - GR Us Ġand ĠT - L ST Ms Ġdiffer Ġfrom ĠQR NN s Ġwith Ġfilter Ġsize Ġ2 Ġand Ġfo - Ġor Ġif o - pool ing Ġrespectively Ġin Ġthat Ġthey Ġlack ĠIN LINE FORM 1 Ġon ĠIN LINE FORM 2 Ġand Ġuse ĠIN LINE FORM 3 Ġrather Ġthan Ġs igm oid Ġon ĠIN LINE FORM 4 Ġ. </s> The ĠQR NN Ġis Ġalso Ġrelated Ġto Ġwork Ġin Ġhybrid Ġconv olution al âĢĵ rec urrent Ġmodels . ĠB IB REF 31 Ġapply ĠCNN s Ġat Ġthe Ġword Ġlevel Ġto Ġgenerate ĠIN LINE FORM 0 Ġ- gram Ġfeatures Ġused Ġby Ġan ĠL ST M Ġfor Ġtext Ġclassification . ĠB IB REF 32 Ġalso Ġtackle Ġtext Ġclassification Ġby Ġapplying Ġconv olutions Ġat Ġthe Ġcharacter Ġlevel , Ġwith Ġa Ġstride Ġto Ġreduce Ġsequence Ġlength , Ġthen Ġfeeding Ġthese Ġfeatures Ġinto Ġa Ġbid irection al ĠL ST M . ĠA Ġsimilar Ġapproach Ġwas Ġtaken Ġby ĠB IB REF 10 Ġfor Ġcharacter - level Ġmachine Ġtranslation . ĠTheir Ġmodel 's Ġenc oder Ġuses Ġa Ġconv olution al Ġlayer Ġfollowed Ġby Ġmax - pool ing Ġto Ġreduce Ġsequence Ġlength , Ġa Ġfour - layer Ġhighway Ġnetwork , Ġand Ġa Ġbid irection al ĠGR U . ĠThe Ġparallel ism Ġof Ġthe Ġconv olution al , Ġpool ing , Ġand Ġhighway Ġlayers Ġallows Ġtraining Ġspeed Ġcomparable Ġto Ġsub word - level Ġmodels Ġwithout Ġhard - coded Ġtext Ġsegment ation . </s> The ĠQR NN Ġenc oder âĢĵ dec oder Ġmodel Ġshares Ġthe Ġfavorable Ġparallel ism Ġand Ġpath - length Ġproperties Ġexhibited Ġby Ġthe ĠByte Net ĠB IB REF 33 Ġ, Ġan Ġarchitecture Ġfor Ġcharacter - level Ġmachine Ġtranslation Ġbased Ġon Ġresidual Ġconv olutions Ġover Ġbinary Ġtrees . ĠTheir Ġmodel Ġwas Ġconstructed Ġto Ġachieve Ġthree Ġdesired Ġproperties : Ġparallel ism , Ġlinear - time Ġcomputational Ġcomplexity , Ġand Ġshort Ġpaths Ġbetween Ġany Ġpair Ġof Ġwords Ġin Ġorder Ġto Ġbetter Ġpropagate Ġgradient Ġsignals . </s> Conclusion </s> Int uitive ly , Ġmany Ġaspects Ġof Ġthe Ġsemantics Ġof Ġlong Ġsequences Ġare Ġcontext - inv ari ant Ġand Ġcan Ġbe Ġcomputed Ġin Ġparallel Ġ( e . g ., Ġconv olution ally ), Ġbut Ġsome Ġaspects Ġrequire Ġlong - distance Ġcontext Ġand Ġmust Ġbe Ġcomputed Ġrecurrent ly . ĠMany Ġexisting Ġneural Ġnetwork Ġarchitectures Ġeither Ġfail Ġto Ġtake Ġadvantage Ġof Ġthe Ġcontextual Ġinformation Ġor Ġfail Ġto Ġtake Ġadvantage Ġof Ġthe Ġparallel ism . ĠQR NN s Ġexploit Ġboth Ġparallel ism Ġand Ġcontext , Ġexhibiting Ġadvantages Ġfrom Ġboth Ġconv olution al Ġand Ġrecurrent Ġneural Ġnetworks . ĠQR NN s Ġhave Ġbetter Ġpredictive Ġaccuracy Ġthan ĠL ST M - based Ġmodels Ġof Ġequal Ġhidden Ġsize , Ġeven Ġthough Ġthey Ġuse Ġfewer Ġparameters Ġand Ġrun Ġsubstantially Ġfaster . ĠOur Ġexperiments Ġshow Ġthat Ġthe Ġspeed Ġand Ġaccuracy Ġadvantages Ġremain Ġconsistent Ġacross Ġtasks Ġand Ġat Ġboth Ġword Ġand Ġcharacter Ġlevels . </s> Ext ensions Ġto Ġboth ĠCNN s Ġand ĠR NN s Ġare Ġoften Ġdirectly Ġapplicable Ġto Ġthe ĠQR NN , Ġwhile Ġthe Ġmodel 's Ġhidden Ġstates Ġare Ġmore Ġinterpret able Ġthan Ġthose Ġof Ġother Ġrecurrent Ġarchitectures Ġas Ġits Ġchannels Ġmaintain Ġtheir Ġindependence Ġacross Ġtim est eps . ĠWe Ġbelieve Ġthat ĠQR NN s Ġcan Ġserve Ġas Ġa Ġbuilding Ġblock Ġfor Ġlong - sequence Ġtasks Ġthat Ġwere Ġpreviously Ġimpractical Ġwith Ġtraditional ĠR NN s . </s> Be am Ġsearch Ġranking Ġcriterion </s> The Ġmodified Ġlog - pro b ability Ġranking Ġcriterion Ġwe Ġused Ġin Ġbeam Ġsearch Ġfor Ġtranslation Ġexperiments Ġis : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġa Ġlength Ġnormal ization Ġparameter ĠB IB REF 34 Ġ, ĠIN LINE FORM 1 Ġis Ġthe ĠIN LINE FORM 2 Ġth Ġoutput Ġcharacter , Ġand ĠIN LINE FORM 3 Ġis Ġa ĠâĢ ľ target Ġlength âĢ Ŀ Ġequal Ġto Ġthe Ġsource Ġsentence Ġlength Ġplus Ġfive Ġcharacters . ĠThis Ġreduces Ġat ĠIN LINE FORM 4 Ġto Ġordinary Ġbeam Ġsearch Ġwith Ġprobabilities : ĠDIS PLAY FORM 0 </s> and Ġat ĠIN LINE FORM 0 Ġto Ġbeam Ġsearch Ġwith Ġprobabilities Ġnormalized Ġby Ġlength Ġ( up Ġto Ġthe Ġtarget Ġlength ): ĠDIS PLAY FORM 0 </s> Con venient ly , Ġthis Ġranking Ġcriterion Ġcan Ġbe Ġcomputed Ġat Ġintermediate Ġbeam - search Ġtim est eps , Ġob vi ating Ġthe Ġneed Ġto Ġapply Ġa Ġseparate Ġre ranking Ġon Ġcomplete Ġhypotheses .
Introduction Recurrent neural networks (RNNs), including gated variants such as the long short-term memory (LSTM) BIBREF0 have become the standard model architecture for deep learning approaches to sequence modeling tasks. RNNs repeatedly apply a function with trainable parameters to a hidden state. Recurrent layers can also be stacked, increasing network depth, representational power and often accuracy. RNN applications in the natural language domain range from sentence classification BIBREF1 to word- and character-level language modeling BIBREF2 . RNNs are also commonly the basic building block for more complex models for tasks such as machine translation BIBREF3 , BIBREF4 , BIBREF5 or question answering BIBREF6 , BIBREF7 . Unfortunately standard RNNs, including LSTMs, are limited in their capability to handle tasks involving very long sequences, such as document classification or character-level machine translation, as the computation of features or states for different parts of the document cannot occur in parallel. Convolutional neural networks (CNNs) BIBREF8 , though more popular on tasks involving image data, have also been applied to sequence encoding tasks BIBREF9 . Such models apply time-invariant filter functions in parallel to windows along the input sequence. CNNs possess several advantages over recurrent models, including increased parallelism and better scaling to long sequences such as those often seen with character-level language data. Convolutional models for sequence processing have been more successful when combined with RNN layers in a hybrid architecture BIBREF10 , because traditional max- and average-pooling approaches to combining convolutional features across timesteps assume time invariance and hence cannot make full use of large-scale sequence order information. We present quasi-recurrent neural networks for neural sequence modeling. QRNNs address both drawbacks of standard models: like CNNs, QRNNs allow for parallel computation across both timestep and minibatch dimensions, enabling high throughput and good scaling to long sequences. Like RNNs, QRNNs allow the output to depend on the overall order of elements in the sequence. We describe QRNN variants tailored to several natural language tasks, including document-level sentiment classification, language modeling, and character-level machine translation. These models outperform strong LSTM baselines on all three tasks while dramatically reducing computation time. Model Each layer of a quasi-recurrent neural network consists of two kinds of subcomponents, analogous to convolution and pooling layers in CNNs. The convolutional component, like convolutional layers in CNNs, allows fully parallel computation across both minibatches and spatial dimensions, in this case the sequence dimension. The pooling component, like pooling layers in CNNs, lacks trainable parameters and allows fully parallel computation across minibatch and feature dimensions. Given an input sequence INLINEFORM0 of INLINEFORM1 INLINEFORM2 -dimensional vectors INLINEFORM3 , the convolutional subcomponent of a QRNN performs convolutions in the timestep dimension with a bank of INLINEFORM4 filters, producing a sequence INLINEFORM5 of INLINEFORM6 -dimensional candidate vectors INLINEFORM7 . In order to be useful for tasks that include prediction of the next token, the filters must not allow the computation for any given timestep to access information from future timesteps. That is, with filters of width INLINEFORM8 , each INLINEFORM9 depends only on INLINEFORM10 through INLINEFORM11 . This concept, known as a masked convolution BIBREF11 , is implemented by padding the input to the left by the convolution's filter size minus one. We apply additional convolutions with separate filter banks to obtain sequences of vectors for the elementwise gates that are needed for the pooling function. While the candidate vectors are passed through a INLINEFORM0 nonlinearity, the gates use an elementwise sigmoid. If the pooling function requires a forget gate INLINEFORM1 and an output gate INLINEFORM2 at each timestep, the full set of computations in the convolutional component is then: DISPLAYFORM0 where INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 , each in INLINEFORM3 , are the convolutional filter banks and INLINEFORM4 denotes a masked convolution along the timestep dimension. Note that if the filter width is 2, these equations reduce to the LSTM-like DISPLAYFORM0 Convolution filters of larger width effectively compute higher INLINEFORM0 -gram features at each timestep; thus larger widths are especially important for character-level tasks. Suitable functions for the pooling subcomponent can be constructed from the familiar elementwise gates of the traditional LSTM cell. We seek a function controlled by gates that can mix states across timesteps, but which acts independently on each channel of the state vector. The simplest option, which BIBREF12 term “dynamic average pooling”, uses only a forget gate: DISPLAYFORM0 We term these three options f-pooling, fo-pooling, and ifo-pooling respectively; in each case we initialize INLINEFORM0 or INLINEFORM1 to zero. Although the recurrent parts of these functions must be calculated for each timestep in sequence, their simplicity and parallelism along feature dimensions means that, in practice, evaluating them over even long sequences requires a negligible amount of computation time. A single QRNN layer thus performs an input-dependent pooling, followed by a gated linear combination of convolutional features. As with convolutional neural networks, two or more QRNN layers should be stacked to create a model with the capacity to approximate more complex functions. Variants Motivated by several common natural language tasks, and the long history of work on related architectures, we introduce several extensions to the stacked QRNN described above. Notably, many extensions to both recurrent and convolutional models can be applied directly to the QRNN as it combines elements of both model types. Regularization An important extension to the stacked QRNN is a robust regularization scheme inspired by recent work in regularizing LSTMs. The need for an effective regularization method for LSTMs, and dropout's relative lack of efficacy when applied to recurrent connections, led to the development of recurrent dropout schemes, including variational inference–based dropout BIBREF13 and zoneout BIBREF14 . These schemes extend dropout to the recurrent setting by taking advantage of the repeating structure of recurrent networks, providing more powerful and less destructive regularization. Variational inference–based dropout locks the dropout mask used for the recurrent connections across timesteps, so a single RNN pass uses a single stochastic subset of the recurrent weights. Zoneout stochastically chooses a new subset of channels to “zone out” at each timestep; for these channels the network copies states from one timestep to the next without modification. As QRNNs lack recurrent weights, the variational inference approach does not apply. Thus we extended zoneout to the QRNN architecture by modifying the pooling function to keep the previous pooling state for a stochastic subset of channels. Conveniently, this is equivalent to stochastically setting a subset of the QRNN's INLINEFORM0 gate channels to 1, or applying dropout on INLINEFORM1 : DISPLAYFORM0 Thus the pooling function itself need not be modified at all. We note that when using an off-the-shelf dropout layer in this context, it is important to remove automatic rescaling functionality from the implementation if it is present. In many experiments, we also apply ordinary dropout between layers, including between word embeddings and the first QRNN layer. Densely-Connected Layers We can also extend the QRNN architecture using techniques introduced for convolutional networks. For sequence classification tasks, we found it helpful to use skip-connections between every QRNN layer, a technique termed “dense convolution” by BIBREF15 . Where traditional feed-forward or convolutional networks have connections only between subsequent layers, a “DenseNet” with INLINEFORM0 layers has feed-forward or convolutional connections between every pair of layers, for a total of INLINEFORM1 . This can improve gradient flow and convergence properties, especially in deeper networks, although it requires a parameter count that is quadratic in the number of layers. When applying this technique to the QRNN, we include connections between the input embeddings and every QRNN layer and between every pair of QRNN layers. This is equivalent to concatenating each QRNN layer's input to its output along the channel dimension before feeding the state into the next layer. The output of the last layer alone is then used as the overall encoding result. Encoder–Decoder Models To demonstrate the generality of QRNNs, we extend the model architecture to sequence-to-sequence tasks, such as machine translation, by using a QRNN as encoder and a modified QRNN, enhanced with attention, as decoder. The motivation for modifying the decoder is that simply feeding the last encoder hidden state (the output of the encoder's pooling layer) into the decoder's recurrent pooling layer, analogously to conventional recurrent encoder–decoder architectures, would not allow the encoder state to affect the gate or update values that are provided to the decoder's pooling layer. This would substantially limit the representational power of the decoder. Instead, the output of each decoder QRNN layer's convolution functions is supplemented at every timestep with the final encoder hidden state. This is accomplished by adding the result of the convolution for layer INLINEFORM0 (e.g., INLINEFORM1 , in INLINEFORM2 ) with broadcasting to a linearly projected copy of layer INLINEFORM3 's last encoder state (e.g., INLINEFORM4 , in INLINEFORM5 ): DISPLAYFORM0 where the tilde denotes that INLINEFORM0 is an encoder variable. Encoder–decoder models which operate on long sequences are made significantly more powerful with the addition of soft attention BIBREF3 , which removes the need for the entire input representation to fit into a fixed-length encoding vector. In our experiments, we computed an attentional sum of the encoder's last layer's hidden states. We used the dot products of these encoder hidden states with the decoder's last layer's un-gated hidden states, applying a INLINEFORM1 along the encoder timesteps, to weight the encoder states into an attentional sum INLINEFORM2 for each decoder timestep. This context, and the decoder state, are then fed into a linear layer followed by the output gate: DISPLAYFORM0 where INLINEFORM0 is the last layer. While the first step of this attention procedure is quadratic in the sequence length, in practice it takes significantly less computation time than the model's linear and convolutional layers due to the simple and highly parallel dot-product scoring function. Experiments We evaluate the performance of the QRNN on three different natural language tasks: document-level sentiment classification, language modeling, and character-based neural machine translation. Our QRNN models outperform LSTM-based models of equal hidden size on all three tasks while dramatically improving computation speed. Experiments were implemented in Chainer BIBREF16 . Sentiment Classification We evaluate the QRNN architecture on a popular document-level sentiment classification benchmark, the IMDb movie review dataset BIBREF17 . The dataset consists of a balanced sample of 25,000 positive and 25,000 negative reviews, divided into equal-size train and test sets, with an average document length of 231 words BIBREF18 . We compare only to other results that do not make use of additional unlabeled data (thus excluding e.g., BIBREF19 ). Our best performance on a held-out development set was achieved using a four-layer densely-connected QRNN with 256 units per layer and word vectors initialized using 300-dimensional cased GloVe embeddings BIBREF20 . Dropout of 0.3 was applied between layers, and we used INLINEFORM0 regularization of INLINEFORM1 . Optimization was performed on minibatches of 24 examples using RMSprop BIBREF21 with learning rate of INLINEFORM2 , INLINEFORM3 , and INLINEFORM4 . Small batch sizes and long sequence lengths provide an ideal situation for demonstrating the QRNN's performance advantages over traditional recurrent architectures. We observed a speedup of 3.2x on IMDb train time per epoch compared to the optimized LSTM implementation provided in NVIDIA's cuDNN library. For specific batch sizes and sequence lengths, a 16x speed gain is possible. Figure FIGREF15 provides extensive speed comparisons. In Figure FIGREF12 , we visualize the hidden state vectors INLINEFORM0 of the final QRNN layer on part of an example from the IMDb dataset. Even without any post-processing, changes in the hidden state are visible and interpretable in regards to the input. This is a consequence of the elementwise nature of the recurrent pooling function, which delays direct interaction between different channels of the hidden state until the computation of the next QRNN layer. Language Modeling We replicate the language modeling experiment of BIBREF2 and BIBREF13 to benchmark the QRNN architecture for natural language sequence prediction. The experiment uses a standard preprocessed version of the Penn Treebank (PTB) by BIBREF25 . We implemented a gated QRNN model with medium hidden size: 2 layers with 640 units in each layer. Both QRNN layers use a convolutional filter width INLINEFORM0 of two timesteps. While the “medium” models used in other work BIBREF2 , BIBREF13 consist of 650 units in each layer, it was more computationally convenient to use a multiple of 32. As the Penn Treebank is a relatively small dataset, preventing overfitting is of considerable importance and a major focus of recent research. It is not obvious in advance which of the many RNN regularization schemes would perform well when applied to the QRNN. Our tests showed encouraging results from zoneout applied to the QRNN's recurrent pooling layer, implemented as described in Section SECREF5 . The experimental settings largely followed the “medium” setup of BIBREF2 . Optimization was performed by stochastic gradient descent (SGD) without momentum. The learning rate was set at 1 for six epochs, then decayed by 0.95 for each subsequent epoch, for a total of 72 epochs. We additionally used INLINEFORM0 regularization of INLINEFORM1 and rescaled gradients with norm above 10. Zoneout was applied by performing dropout with ratio 0.1 on the forget gates of the QRNN, without rescaling the output of the dropout function. Batches consist of 20 examples, each 105 timesteps. Comparing our results on the gated QRNN with zoneout to the results of LSTMs with both ordinary and variational dropout in Table TABREF14 , we see that the QRNN is highly competitive. The QRNN without zoneout strongly outperforms both our medium LSTM and the medium LSTM of BIBREF2 which do not use recurrent dropout and is even competitive with variational LSTMs. This may be due to the limited computational capacity that the QRNN's pooling layer has relative to the LSTM's recurrent weights, providing structural regularization over the recurrence. Without zoneout, early stopping based upon validation loss was required as the QRNN would begin overfitting. By applying a small amount of zoneout ( INLINEFORM0 ), no early stopping is required and the QRNN achieves competitive levels of perplexity to the variational LSTM of BIBREF13 , which had variational inference based dropout of 0.2 applied recurrently. Their best performing variation also used Monte Carlo (MC) dropout averaging at test time of 1000 different masks, making it computationally more expensive to run. When training on the PTB dataset with an NVIDIA K40 GPU, we found that the QRNN is substantially faster than a standard LSTM, even when comparing against the optimized cuDNN LSTM. In Figure FIGREF15 we provide a breakdown of the time taken for Chainer's default LSTM, the cuDNN LSTM, and QRNN to perform a full forward and backward pass on a single batch during training of the RNN LM on PTB. For both LSTM implementations, running time was dominated by the RNN computations, even with the highly optimized cuDNN implementation. For the QRNN implementation, however, the “RNN” layers are no longer the bottleneck. Indeed, there are diminishing returns from further optimization of the QRNN itself as the softmax and optimization overhead take equal or greater time. Note that the softmax, over a vocabulary size of only 10,000 words, is relatively small; for tasks with larger vocabularies, the softmax would likely dominate computation time. It is also important to note that the cuDNN library's RNN primitives do not natively support any form of recurrent dropout. That is, running an LSTM that uses a state-of-the-art regularization scheme at cuDNN-like speeds would likely require an entirely custom kernel. Character-level Neural Machine Translation We evaluate the sequence-to-sequence QRNN architecture described in SECREF5 on a challenging neural machine translation task, IWSLT German–English spoken-domain translation, applying fully character-level segmentation. This dataset consists of 209,772 sentence pairs of parallel training data from transcribed TED and TEDx presentations, with a mean sentence length of 103 characters for German and 93 for English. We remove training sentences with more than 300 characters in English or German, and use a unified vocabulary of 187 Unicode code points. Our best performance on a development set (TED.tst2013) was achieved using a four-layer encoder–decoder QRNN with 320 units per layer, no dropout or INLINEFORM0 regularization, and gradient rescaling to a maximum magnitude of 5. Inputs were supplied to the encoder reversed, while the encoder convolutions were not masked. The first encoder layer used convolutional filter width INLINEFORM1 , while the other encoder layers used INLINEFORM2 . Optimization was performed for 10 epochs on minibatches of 16 examples using Adam BIBREF28 with INLINEFORM3 , INLINEFORM4 , INLINEFORM5 , and INLINEFORM6 . Decoding was performed using beam search with beam width 8 and length normalization INLINEFORM7 . The modified log-probability ranking criterion is provided in the appendix. Results using this architecture were compared to an equal-sized four-layer encoder–decoder LSTM with attention, applying dropout of 0.2. We again optimized using Adam; other hyperparameters were equal to their values for the QRNN and the same beam search procedure was applied. Table TABREF17 shows that the QRNN outperformed the character-level LSTM, almost matching the performance of a word-level attentional baseline. Related Work Exploring alternatives to traditional RNNs for sequence tasks is a major area of current research. Quasi-recurrent neural networks are related to several such recently described models, especially the strongly-typed recurrent neural networks (T-RNN) introduced by BIBREF12 . While the motivation and constraints described in that work are different, BIBREF12 's concepts of “learnware” and “firmware” parallel our discussion of convolution-like and pooling-like subcomponents. As the use of a fully connected layer for recurrent connections violates the constraint of “strong typing”, all strongly-typed RNN architectures (including the T-RNN, T-GRU, and T-LSTM) are also quasi-recurrent. However, some QRNN models (including those with attention or skip-connections) are not “strongly typed”. In particular, a T-RNN differs from a QRNN as described in this paper with filter size 1 and f-pooling only in the absence of an activation function on INLINEFORM0 . Similarly, T-GRUs and T-LSTMs differ from QRNNs with filter size 2 and fo- or ifo-pooling respectively in that they lack INLINEFORM1 on INLINEFORM2 and use INLINEFORM3 rather than sigmoid on INLINEFORM4 . The QRNN is also related to work in hybrid convolutional–recurrent models. BIBREF31 apply CNNs at the word level to generate INLINEFORM0 -gram features used by an LSTM for text classification. BIBREF32 also tackle text classification by applying convolutions at the character level, with a stride to reduce sequence length, then feeding these features into a bidirectional LSTM. A similar approach was taken by BIBREF10 for character-level machine translation. Their model's encoder uses a convolutional layer followed by max-pooling to reduce sequence length, a four-layer highway network, and a bidirectional GRU. The parallelism of the convolutional, pooling, and highway layers allows training speed comparable to subword-level models without hard-coded text segmentation. The QRNN encoder–decoder model shares the favorable parallelism and path-length properties exhibited by the ByteNet BIBREF33 , an architecture for character-level machine translation based on residual convolutions over binary trees. Their model was constructed to achieve three desired properties: parallelism, linear-time computational complexity, and short paths between any pair of words in order to better propagate gradient signals. Conclusion Intuitively, many aspects of the semantics of long sequences are context-invariant and can be computed in parallel (e.g., convolutionally), but some aspects require long-distance context and must be computed recurrently. Many existing neural network architectures either fail to take advantage of the contextual information or fail to take advantage of the parallelism. QRNNs exploit both parallelism and context, exhibiting advantages from both convolutional and recurrent neural networks. QRNNs have better predictive accuracy than LSTM-based models of equal hidden size, even though they use fewer parameters and run substantially faster. Our experiments show that the speed and accuracy advantages remain consistent across tasks and at both word and character levels. Extensions to both CNNs and RNNs are often directly applicable to the QRNN, while the model's hidden states are more interpretable than those of other recurrent architectures as its channels maintain their independence across timesteps. We believe that QRNNs can serve as a building block for long-sequence tasks that were previously impractical with traditional RNNs. Beam search ranking criterion The modified log-probability ranking criterion we used in beam search for translation experiments is: DISPLAYFORM0 where INLINEFORM0 is a length normalization parameter BIBREF34 , INLINEFORM1 is the INLINEFORM2 th output character, and INLINEFORM3 is a “target length” equal to the source sentence length plus five characters. This reduces at INLINEFORM4 to ordinary beam search with probabilities: DISPLAYFORM0 and at INLINEFORM0 to beam search with probabilities normalized by length (up to the target length): DISPLAYFORM0 Conveniently, this ranking criterion can be computed at intermediate beam-search timesteps, obviating the need to apply a separate reranking on complete hypotheses.
What sentiment classification dataset is used?
b591853e938984e6069d738371500ebdec50d256
What Ġsentiment Ġclassification Ġdataset Ġis Ġused ?
the ĠIM Db Ġmovie Ġreview Ġdataset ĠB IB REF 17
<s> What Ġsentiment Ġclassification Ġdataset Ġis Ġused ? </s> Introduction </s> Rec urrent Ġneural Ġnetworks Ġ( R NN s ), Ġincluding Ġg ated Ġvariants Ġsuch Ġas Ġthe Ġlong Ġshort - term Ġmemory Ġ( L ST M ) ĠB IB REF 0 Ġhave Ġbecome Ġthe Ġstandard Ġmodel Ġarchitecture Ġfor Ġdeep Ġlearning Ġapproaches Ġto Ġsequence Ġmodeling Ġtasks . ĠR NN s Ġrepeatedly Ġapply Ġa Ġfunction Ġwith Ġtrain able Ġparameters Ġto Ġa Ġhidden Ġstate . ĠRec urrent Ġlayers Ġcan Ġalso Ġbe Ġstacked , Ġincreasing Ġnetwork Ġdepth , Ġrepresent ational Ġpower Ġand Ġoften Ġaccuracy . ĠR NN Ġapplications Ġin Ġthe Ġnatural Ġlanguage Ġdomain Ġrange Ġfrom Ġsentence Ġclassification ĠB IB REF 1 Ġto Ġword - Ġand Ġcharacter - level Ġlanguage Ġmodeling ĠB IB REF 2 Ġ. ĠR NN s Ġare Ġalso Ġcommonly Ġthe Ġbasic Ġbuilding Ġblock Ġfor Ġmore Ġcomplex Ġmodels Ġfor Ġtasks Ġsuch Ġas Ġmachine Ġtranslation ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġor Ġquestion Ġanswering ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ. ĠUnfortunately Ġstandard ĠR NN s , Ġincluding ĠL ST Ms , Ġare Ġlimited Ġin Ġtheir Ġcapability Ġto Ġhandle Ġtasks Ġinvolving Ġvery Ġlong Ġsequences , Ġsuch Ġas Ġdocument Ġclassification Ġor Ġcharacter - level Ġmachine Ġtranslation , Ġas Ġthe Ġcomputation Ġof Ġfeatures Ġor Ġstates Ġfor Ġdifferent Ġparts Ġof Ġthe Ġdocument Ġcannot Ġoccur Ġin Ġparallel . </s> Con v olution al Ġneural Ġnetworks Ġ( CNN s ) ĠB IB REF 8 Ġ, Ġthough Ġmore Ġpopular Ġon Ġtasks Ġinvolving Ġimage Ġdata , Ġhave Ġalso Ġbeen Ġapplied Ġto Ġsequence Ġencoding Ġtasks ĠB IB REF 9 Ġ. ĠSuch Ġmodels Ġapply Ġtime - inv ari ant Ġfilter Ġfunctions Ġin Ġparallel Ġto Ġwindows Ġalong Ġthe Ġinput Ġsequence . ĠCNN s Ġpossess Ġseveral Ġadvantages Ġover Ġrecurrent Ġmodels , Ġincluding Ġincreased Ġparallel ism Ġand Ġbetter Ġscaling Ġto Ġlong Ġsequences Ġsuch Ġas Ġthose Ġoften Ġseen Ġwith Ġcharacter - level Ġlanguage Ġdata . ĠConv olution al Ġmodels Ġfor Ġsequence Ġprocessing Ġhave Ġbeen Ġmore Ġsuccessful Ġwhen Ġcombined Ġwith ĠR NN Ġlayers Ġin Ġa Ġhybrid Ġarchitecture ĠB IB REF 10 Ġ, Ġbecause Ġtraditional Ġmax - Ġand Ġaverage - pool ing Ġapproaches Ġto Ġcombining Ġconv olution al Ġfeatures Ġacross Ġtim est eps Ġassume Ġtime Ġinvari ance Ġand Ġhence Ġcannot Ġmake Ġfull Ġuse Ġof Ġlarge - scale Ġsequence Ġorder Ġinformation . </s> We Ġpresent Ġquasi - rec urrent Ġneural Ġnetworks Ġfor Ġneural Ġsequence Ġmodeling . ĠQR NN s Ġaddress Ġboth Ġdrawbacks Ġof Ġstandard Ġmodels : Ġlike ĠCNN s , ĠQR NN s Ġallow Ġfor Ġparallel Ġcomputation Ġacross Ġboth Ġtim est ep Ġand Ġmin ib atch Ġdimensions , Ġenabling Ġhigh Ġthroughput Ġand Ġgood Ġscaling Ġto Ġlong Ġsequences . ĠLike ĠR NN s , ĠQR NN s Ġallow Ġthe Ġoutput Ġto Ġdepend Ġon Ġthe Ġoverall Ġorder Ġof Ġelements Ġin Ġthe Ġsequence . ĠWe Ġdescribe ĠQR NN Ġvariants Ġtailored Ġto Ġseveral Ġnatural Ġlanguage Ġtasks , Ġincluding Ġdocument - level Ġsentiment Ġclassification , Ġlanguage Ġmodeling , Ġand Ġcharacter - level Ġmachine Ġtranslation . ĠThese Ġmodels Ġoutper form Ġstrong ĠL ST M Ġbas elines Ġon Ġall Ġthree Ġtasks Ġwhile Ġdramatically Ġreducing Ġcomputation Ġtime . </s> Model </s> Each Ġlayer Ġof Ġa Ġquasi - rec urrent Ġneural Ġnetwork Ġconsists Ġof Ġtwo Ġkinds Ġof Ġsub comp onents , Ġanalogous Ġto Ġconv olution Ġand Ġpool ing Ġlayers Ġin ĠCNN s . ĠThe Ġconv olution al Ġcomponent , Ġlike Ġconv olution al Ġlayers Ġin ĠCNN s , Ġallows Ġfully Ġparallel Ġcomputation Ġacross Ġboth Ġmin ib atches Ġand Ġspatial Ġdimensions , Ġin Ġthis Ġcase Ġthe Ġsequence Ġdimension . ĠThe Ġpool ing Ġcomponent , Ġlike Ġpool ing Ġlayers Ġin ĠCNN s , Ġlacks Ġtrain able Ġparameters Ġand Ġallows Ġfully Ġparallel Ġcomputation Ġacross Ġmin ib atch Ġand Ġfeature Ġdimensions . </s> Given Ġan Ġinput Ġsequence ĠIN LINE FORM 0 Ġof ĠIN LINE FORM 1 ĠIN LINE FORM 2 Ġ- dimensional Ġvectors ĠIN LINE FORM 3 Ġ, Ġthe Ġconv olution al Ġsub component Ġof Ġa ĠQR NN Ġperforms Ġconv olutions Ġin Ġthe Ġtim est ep Ġdimension Ġwith Ġa Ġbank Ġof ĠIN LINE FORM 4 Ġfilters , Ġproducing Ġa Ġsequence ĠIN LINE FORM 5 Ġof ĠIN LINE FORM 6 Ġ- dimensional Ġcandidate Ġvectors ĠIN LINE FORM 7 Ġ. ĠIn Ġorder Ġto Ġbe Ġuseful Ġfor Ġtasks Ġthat Ġinclude Ġprediction Ġof Ġthe Ġnext Ġtoken , Ġthe Ġfilters Ġmust Ġnot Ġallow Ġthe Ġcomputation Ġfor Ġany Ġgiven Ġtim est ep Ġto Ġaccess Ġinformation Ġfrom Ġfuture Ġtim est eps . ĠThat Ġis , Ġwith Ġfilters Ġof Ġwidth ĠIN LINE FORM 8 Ġ, Ġeach ĠIN LINE FORM 9 Ġdepends Ġonly Ġon ĠIN LINE FORM 10 Ġthrough ĠIN LINE FORM 11 Ġ. ĠThis Ġconcept , Ġknown Ġas Ġa Ġmasked Ġconv olution ĠB IB REF 11 Ġ, Ġis Ġimplemented Ġby Ġpadding Ġthe Ġinput Ġto Ġthe Ġleft Ġby Ġthe Ġconv olution 's Ġfilter Ġsize Ġminus Ġone . </s> We Ġapply Ġadditional Ġconv olutions Ġwith Ġseparate Ġfilter Ġbanks Ġto Ġobtain Ġsequences Ġof Ġvectors Ġfor Ġthe Ġelement wise Ġgates Ġthat Ġare Ġneeded Ġfor Ġthe Ġpool ing Ġfunction . ĠWhile Ġthe Ġcandidate Ġvectors Ġare Ġpassed Ġthrough Ġa ĠIN LINE FORM 0 Ġnon linear ity , Ġthe Ġgates Ġuse Ġan Ġelement wise Ġs igm oid . ĠIf Ġthe Ġpool ing Ġfunction Ġrequires Ġa Ġforget Ġgate ĠIN LINE FORM 1 Ġand Ġan Ġoutput Ġgate ĠIN LINE FORM 2 Ġat Ġeach Ġtim est ep , Ġthe Ġfull Ġset Ġof Ġcomput ations Ġin Ġthe Ġconv olution al Ġcomponent Ġis Ġthen : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, Ġand ĠIN LINE FORM 2 Ġ, Ġeach Ġin ĠIN LINE FORM 3 Ġ, Ġare Ġthe Ġconv olution al Ġfilter Ġbanks Ġand ĠIN LINE FORM 4 Ġdenotes Ġa Ġmasked Ġconv olution Ġalong Ġthe Ġtim est ep Ġdimension . ĠNote Ġthat Ġif Ġthe Ġfilter Ġwidth Ġis Ġ2 , Ġthese Ġequations Ġreduce Ġto Ġthe ĠL ST M - like ĠDIS PLAY FORM 0 </s> Con v olution Ġfilters Ġof Ġlarger Ġwidth Ġeffectively Ġcompute Ġhigher ĠIN LINE FORM 0 Ġ- gram Ġfeatures Ġat Ġeach Ġtim est ep ; Ġthus Ġlarger Ġwidth s Ġare Ġespecially Ġimportant Ġfor Ġcharacter - level Ġtasks . </s> Su itable Ġfunctions Ġfor Ġthe Ġpool ing Ġsub component Ġcan Ġbe Ġconstructed Ġfrom Ġthe Ġfamiliar Ġelement wise Ġgates Ġof Ġthe Ġtraditional ĠL ST M Ġcell . ĠWe Ġseek Ġa Ġfunction Ġcontrolled Ġby Ġgates Ġthat Ġcan Ġmix Ġstates Ġacross Ġtim est eps , Ġbut Ġwhich Ġacts Ġindependently Ġon Ġeach Ġchannel Ġof Ġthe Ġstate Ġvector . ĠThe Ġsimplest Ġoption , Ġwhich ĠB IB REF 12 Ġterm ĠâĢ ľ d ynamic Ġaverage Ġpool ing âĢ Ŀ , Ġuses Ġonly Ġa Ġforget Ġgate : ĠDIS PLAY FORM 0 </s> We Ġterm Ġthese Ġthree Ġoptions Ġf - pool ing , Ġfo - pool ing , Ġand Ġif o - pool ing Ġrespectively ; Ġin Ġeach Ġcase Ġwe Ġinitialize ĠIN LINE FORM 0 Ġor ĠIN LINE FORM 1 Ġto Ġzero . ĠAlthough Ġthe Ġrecurrent Ġparts Ġof Ġthese Ġfunctions Ġmust Ġbe Ġcalculated Ġfor Ġeach Ġtim est ep Ġin Ġsequence , Ġtheir Ġsimplicity Ġand Ġparallel ism Ġalong Ġfeature Ġdimensions Ġmeans Ġthat , Ġin Ġpractice , Ġevaluating Ġthem Ġover Ġeven Ġlong Ġsequences Ġrequires Ġa Ġnegligible Ġamount Ġof Ġcomputation Ġtime . </s> A Ġsingle ĠQR NN Ġlayer Ġthus Ġperforms Ġan Ġinput - dependent Ġpool ing , Ġfollowed Ġby Ġa Ġg ated Ġlinear Ġcombination Ġof Ġconv olution al Ġfeatures . ĠAs Ġwith Ġconv olution al Ġneural Ġnetworks , Ġtwo Ġor Ġmore ĠQR NN Ġlayers Ġshould Ġbe Ġstacked Ġto Ġcreate Ġa Ġmodel Ġwith Ġthe Ġcapacity Ġto Ġapproximate Ġmore Ġcomplex Ġfunctions . </s> Vari ants </s> Mot ivated Ġby Ġseveral Ġcommon Ġnatural Ġlanguage Ġtasks , Ġand Ġthe Ġlong Ġhistory Ġof Ġwork Ġon Ġrelated Ġarchitectures , Ġwe Ġintroduce Ġseveral Ġextensions Ġto Ġthe Ġstacked ĠQR NN Ġdescribed Ġabove . ĠNotably , Ġmany Ġextensions Ġto Ġboth Ġrecurrent Ġand Ġconv olution al Ġmodels Ġcan Ġbe Ġapplied Ġdirectly Ġto Ġthe ĠQR NN Ġas Ġit Ġcombines Ġelements Ġof Ġboth Ġmodel Ġtypes . </s> Regular ization ĠAn Ġimportant Ġextension Ġto Ġthe Ġstacked ĠQR NN Ġis Ġa Ġrobust Ġregular ization Ġscheme Ġinspired Ġby Ġrecent Ġwork Ġin Ġregular izing ĠL ST Ms . </s> The Ġneed Ġfor Ġan Ġeffective Ġregular ization Ġmethod Ġfor ĠL ST Ms , Ġand Ġdrop out 's Ġrelative Ġlack Ġof Ġefficacy Ġwhen Ġapplied Ġto Ġrecurrent Ġconnections , Ġled Ġto Ġthe Ġdevelopment Ġof Ġrecurrent Ġdrop out Ġschemes , Ġincluding Ġvari ational Ġinference âĢĵ based Ġdrop out ĠB IB REF 13 Ġand Ġzone out ĠB IB REF 14 Ġ. ĠThese Ġschemes Ġextend Ġdrop out Ġto Ġthe Ġrecurrent Ġsetting Ġby Ġtaking Ġadvantage Ġof Ġthe Ġrepeating Ġstructure Ġof Ġrecurrent Ġnetworks , Ġproviding Ġmore Ġpowerful Ġand Ġless Ġdestructive Ġregular ization . </s> Vari ational Ġinference âĢĵ based Ġdrop out Ġlocks Ġthe Ġdrop out Ġmask Ġused Ġfor Ġthe Ġrecurrent Ġconnections Ġacross Ġtim est eps , Ġso Ġa Ġsingle ĠR NN Ġpass Ġuses Ġa Ġsingle Ġsto ch astic Ġsubset Ġof Ġthe Ġrecurrent Ġweights . ĠZone out Ġsto ch astically Ġchooses Ġa Ġnew Ġsubset Ġof Ġchannels Ġto ĠâĢ ľ zone Ġout âĢ Ŀ Ġat Ġeach Ġtim est ep ; Ġfor Ġthese Ġchannels Ġthe Ġnetwork Ġcopies Ġstates Ġfrom Ġone Ġtim est ep Ġto Ġthe Ġnext Ġwithout Ġmodification . </s> As ĠQR NN s Ġlack Ġrecurrent Ġweights , Ġthe Ġvari ational Ġinference Ġapproach Ġdoes Ġnot Ġapply . ĠThus Ġwe Ġextended Ġzone out Ġto Ġthe ĠQR NN Ġarchitecture Ġby Ġmodifying Ġthe Ġpool ing Ġfunction Ġto Ġkeep Ġthe Ġprevious Ġpool ing Ġstate Ġfor Ġa Ġsto ch astic Ġsubset Ġof Ġchannels . ĠCon venient ly , Ġthis Ġis Ġequivalent Ġto Ġsto ch astically Ġsetting Ġa Ġsubset Ġof Ġthe ĠQR NN 's ĠIN LINE FORM 0 Ġgate Ġchannels Ġto Ġ1 , Ġor Ġapplying Ġdrop out Ġon ĠIN LINE FORM 1 Ġ: ĠDIS PLAY FORM 0 </s> Thus Ġthe Ġpool ing Ġfunction Ġitself Ġneed Ġnot Ġbe Ġmodified Ġat Ġall . ĠWe Ġnote Ġthat Ġwhen Ġusing Ġan Ġoff - the - she lf Ġdrop out Ġlayer Ġin Ġthis Ġcontext , Ġit Ġis Ġimportant Ġto Ġremove Ġautomatic Ġresc aling Ġfunctionality Ġfrom Ġthe Ġimplementation Ġif Ġit Ġis Ġpresent . ĠIn Ġmany Ġexperiments , Ġwe Ġalso Ġapply Ġordinary Ġdrop out Ġbetween Ġlayers , Ġincluding Ġbetween Ġword Ġembed d ings Ġand Ġthe Ġfirst ĠQR NN Ġlayer . </s> D ense ly - Connect ed ĠL ayers ĠWe Ġcan Ġalso Ġextend Ġthe ĠQR NN Ġarchitecture Ġusing Ġtechniques Ġintroduced Ġfor Ġconv olution al Ġnetworks . ĠFor Ġsequence Ġclassification Ġtasks , Ġwe Ġfound Ġit Ġhelpful Ġto Ġuse Ġskip - connect ions Ġbetween Ġevery ĠQR NN Ġlayer , Ġa Ġtechnique Ġtermed ĠâĢ ľ d ense Ġconv olution âĢ Ŀ Ġby ĠB IB REF 15 Ġ. ĠWhere Ġtraditional Ġfeed - forward Ġor Ġconv olution al Ġnetworks Ġhave Ġconnections Ġonly Ġbetween Ġsubsequent Ġlayers , Ġa ĠâĢ ľ D ense Net âĢ Ŀ Ġwith ĠIN LINE FORM 0 Ġlayers Ġhas Ġfeed - forward Ġor Ġconv olution al Ġconnections Ġbetween Ġevery Ġpair Ġof Ġlayers , Ġfor Ġa Ġtotal Ġof ĠIN LINE FORM 1 Ġ. ĠThis Ġcan Ġimprove Ġgradient Ġflow Ġand Ġconvergence Ġproperties , Ġespecially Ġin Ġdeeper Ġnetworks , Ġalthough Ġit Ġrequires Ġa Ġparameter Ġcount Ġthat Ġis Ġquad r atic Ġin Ġthe Ġnumber Ġof Ġlayers . </s> When Ġapplying Ġthis Ġtechnique Ġto Ġthe ĠQR NN , Ġwe Ġinclude Ġconnections Ġbetween Ġthe Ġinput Ġembed d ings Ġand Ġevery ĠQR NN Ġlayer Ġand Ġbetween Ġevery Ġpair Ġof ĠQR NN Ġlayers . ĠThis Ġis Ġequivalent Ġto Ġconc aten ating Ġeach ĠQR NN Ġlayer 's Ġinput Ġto Ġits Ġoutput Ġalong Ġthe Ġchannel Ġdimension Ġbefore Ġfeeding Ġthe Ġstate Ġinto Ġthe Ġnext Ġlayer . ĠThe Ġoutput Ġof Ġthe Ġlast Ġlayer Ġalone Ġis Ġthen Ġused Ġas Ġthe Ġoverall Ġencoding Ġresult . </s> Enc oder âĢĵ Dec oder ĠModels ĠTo Ġdemonstrate Ġthe Ġgener ality Ġof ĠQR NN s , Ġwe Ġextend Ġthe Ġmodel Ġarchitecture Ġto Ġsequence - to - sequence Ġtasks , Ġsuch Ġas Ġmachine Ġtranslation , Ġby Ġusing Ġa ĠQR NN Ġas Ġenc oder Ġand Ġa Ġmodified ĠQR NN , Ġenhanced Ġwith Ġattention , Ġas Ġdec oder . ĠThe Ġmotivation Ġfor Ġmodifying Ġthe Ġdec oder Ġis Ġthat Ġsimply Ġfeeding Ġthe Ġlast Ġenc oder Ġhidden Ġstate Ġ( the Ġoutput Ġof Ġthe Ġenc oder 's Ġpool ing Ġlayer ) Ġinto Ġthe Ġdec oder 's Ġrecurrent Ġpool ing Ġlayer , Ġanalog ously Ġto Ġconventional Ġrecurrent Ġenc oder âĢĵ dec oder Ġarchitectures , Ġwould Ġnot Ġallow Ġthe Ġenc oder Ġstate Ġto Ġaffect Ġthe Ġgate Ġor Ġupdate Ġvalues Ġthat Ġare Ġprovided Ġto Ġthe Ġdec oder 's Ġpool ing Ġlayer . ĠThis Ġwould Ġsubstantially Ġlimit Ġthe Ġrepresent ational Ġpower Ġof Ġthe Ġdec oder . </s> Instead , Ġthe Ġoutput Ġof Ġeach Ġdec oder ĠQR NN Ġlayer 's Ġconv olution Ġfunctions Ġis Ġsupplemented Ġat Ġevery Ġtim est ep Ġwith Ġthe Ġfinal Ġenc oder Ġhidden Ġstate . ĠThis Ġis Ġaccomplished Ġby Ġadding Ġthe Ġresult Ġof Ġthe Ġconv olution Ġfor Ġlayer ĠIN LINE FORM 0 Ġ( e . g ., ĠIN LINE FORM 1 Ġ, Ġin ĠIN LINE FORM 2 Ġ) Ġwith Ġbroadcasting Ġto Ġa Ġlin early Ġprojected Ġcopy Ġof Ġlayer ĠIN LINE FORM 3 Ġ' s Ġlast Ġenc oder Ġstate Ġ( e . g ., ĠIN LINE FORM 4 Ġ, Ġin ĠIN LINE FORM 5 Ġ): ĠDIS PLAY FORM 0 </s> where Ġthe Ġt ilde Ġdenotes Ġthat ĠIN LINE FORM 0 Ġis Ġan Ġenc oder Ġvariable . ĠEnc oder âĢĵ dec oder Ġmodels Ġwhich Ġoperate Ġon Ġlong Ġsequences Ġare Ġmade Ġsignificantly Ġmore Ġpowerful Ġwith Ġthe Ġaddition Ġof Ġsoft Ġattention ĠB IB REF 3 Ġ, Ġwhich Ġremoves Ġthe Ġneed Ġfor Ġthe Ġentire Ġinput Ġrepresentation Ġto Ġfit Ġinto Ġa Ġfixed - length Ġencoding Ġvector . ĠIn Ġour Ġexperiments , Ġwe Ġcomputed Ġan Ġattention al Ġsum Ġof Ġthe Ġenc oder 's Ġlast Ġlayer 's Ġhidden Ġstates . ĠWe Ġused Ġthe Ġdot Ġproducts Ġof Ġthese Ġenc oder Ġhidden Ġstates Ġwith Ġthe Ġdec oder 's Ġlast Ġlayer 's Ġun - g ated Ġhidden Ġstates , Ġapplying Ġa ĠIN LINE FORM 1 Ġalong Ġthe Ġenc oder Ġtim est eps , Ġto Ġweight Ġthe Ġenc oder Ġstates Ġinto Ġan Ġattention al Ġsum ĠIN LINE FORM 2 Ġfor Ġeach Ġdec oder Ġtim est ep . ĠThis Ġcontext , Ġand Ġthe Ġdec oder Ġstate , Ġare Ġthen Ġfed Ġinto Ġa Ġlinear Ġlayer Ġfollowed Ġby Ġthe Ġoutput Ġgate : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġlast Ġlayer . </s> While Ġthe Ġfirst Ġstep Ġof Ġthis Ġattention Ġprocedure Ġis Ġquad r atic Ġin Ġthe Ġsequence Ġlength , Ġin Ġpractice Ġit Ġtakes Ġsignificantly Ġless Ġcomputation Ġtime Ġthan Ġthe Ġmodel 's Ġlinear Ġand Ġconv olution al Ġlayers Ġdue Ġto Ġthe Ġsimple Ġand Ġhighly Ġparallel Ġdot - product Ġscoring Ġfunction . </s> Exper iments </s> We Ġevaluate Ġthe Ġperformance Ġof Ġthe ĠQR NN Ġon Ġthree Ġdifferent Ġnatural Ġlanguage Ġtasks : Ġdocument - level Ġsentiment Ġclassification , Ġlanguage Ġmodeling , Ġand Ġcharacter - based Ġneural Ġmachine Ġtranslation . ĠOur ĠQR NN Ġmodels Ġoutper form ĠL ST M - based Ġmodels Ġof Ġequal Ġhidden Ġsize Ġon Ġall Ġthree Ġtasks Ġwhile Ġdramatically Ġimproving Ġcomputation Ġspeed . ĠExper iments Ġwere Ġimplemented Ġin ĠCh ainer ĠB IB REF 16 Ġ. </s> Sent iment ĠClassification </s> We Ġevaluate Ġthe ĠQR NN Ġarchitecture Ġon Ġa Ġpopular Ġdocument - level Ġsentiment Ġclassification Ġbenchmark , Ġthe ĠIM Db Ġmovie Ġreview Ġdataset ĠB IB REF 17 Ġ. ĠThe Ġdataset Ġconsists Ġof Ġa Ġbalanced Ġsample Ġof Ġ25 , 000 Ġpositive Ġand Ġ25 , 000 Ġnegative Ġreviews , Ġdivided Ġinto Ġequal - size Ġtrain Ġand Ġtest Ġsets , Ġwith Ġan Ġaverage Ġdocument Ġlength Ġof Ġ231 Ġwords ĠB IB REF 18 Ġ. ĠWe Ġcompare Ġonly Ġto Ġother Ġresults Ġthat Ġdo Ġnot Ġmake Ġuse Ġof Ġadditional Ġunl abel ed Ġdata Ġ( thus Ġexcluding Ġe . g ., ĠB IB REF 19 Ġ). </s> Our Ġbest Ġperformance Ġon Ġa Ġheld - out Ġdevelopment Ġset Ġwas Ġachieved Ġusing Ġa Ġfour - layer Ġdensely - connected ĠQR NN Ġwith Ġ256 Ġunits Ġper Ġlayer Ġand Ġword Ġvectors Ġinitialized Ġusing Ġ300 - dimensional Ġc ased ĠGl o Ve Ġembed d ings ĠB IB REF 20 Ġ. ĠDrop out Ġof Ġ0 . 3 Ġwas Ġapplied Ġbetween Ġlayers , Ġand Ġwe Ġused ĠIN LINE FORM 0 Ġregular ization Ġof ĠIN LINE FORM 1 Ġ. ĠOptim ization Ġwas Ġperformed Ġon Ġmin ib atches Ġof Ġ24 Ġexamples Ġusing ĠR MS prop ĠB IB REF 21 Ġwith Ġlearning Ġrate Ġof ĠIN LINE FORM 2 Ġ, ĠIN LINE FORM 3 Ġ, Ġand ĠIN LINE FORM 4 Ġ. </s> Small Ġbatch Ġsizes Ġand Ġlong Ġsequence Ġlengths Ġprovide Ġan Ġideal Ġsituation Ġfor Ġdemonstrating Ġthe ĠQR NN 's Ġperformance Ġadvantages Ġover Ġtraditional Ġrecurrent Ġarchitectures . ĠWe Ġobserved Ġa Ġspeed up Ġof Ġ3 . 2 x Ġon ĠIM Db Ġtrain Ġtime Ġper Ġepoch Ġcompared Ġto Ġthe Ġoptimized ĠL ST M Ġimplementation Ġprovided Ġin ĠNVIDIA 's Ġcu D NN Ġlibrary . ĠFor Ġspecific Ġbatch Ġsizes Ġand Ġsequence Ġlengths , Ġa Ġ16 x Ġspeed Ġgain Ġis Ġpossible . ĠFigure ĠFIG REF 15 Ġprovides Ġextensive Ġspeed Ġcomparisons . </s> In ĠFigure ĠFIG REF 12 Ġ, Ġwe Ġvisualize Ġthe Ġhidden Ġstate Ġvectors ĠIN LINE FORM 0 Ġof Ġthe Ġfinal ĠQR NN Ġlayer Ġon Ġpart Ġof Ġan Ġexample Ġfrom Ġthe ĠIM Db Ġdataset . ĠEven Ġwithout Ġany Ġpost - processing , Ġchanges Ġin Ġthe Ġhidden Ġstate Ġare Ġvisible Ġand Ġinterpret able Ġin Ġregards Ġto Ġthe Ġinput . ĠThis Ġis Ġa Ġconsequence Ġof Ġthe Ġelement wise Ġnature Ġof Ġthe Ġrecurrent Ġpool ing Ġfunction , Ġwhich Ġdelays Ġdirect Ġinteraction Ġbetween Ġdifferent Ġchannels Ġof Ġthe Ġhidden Ġstate Ġuntil Ġthe Ġcomputation Ġof Ġthe Ġnext ĠQR NN Ġlayer . </s> Language ĠModel ing </s> We Ġreplicate Ġthe Ġlanguage Ġmodeling Ġexperiment Ġof ĠB IB REF 2 Ġand ĠB IB REF 13 Ġto Ġbenchmark Ġthe ĠQR NN Ġarchitecture Ġfor Ġnatural Ġlanguage Ġsequence Ġprediction . ĠThe Ġexperiment Ġuses Ġa Ġstandard Ġpre process ed Ġversion Ġof Ġthe ĠPenn ĠTree bank Ġ( PT B ) Ġby ĠB IB REF 25 Ġ. </s> We Ġimplemented Ġa Ġg ated ĠQR NN Ġmodel Ġwith Ġmedium Ġhidden Ġsize : Ġ2 Ġlayers Ġwith Ġ640 Ġunits Ġin Ġeach Ġlayer . ĠBoth ĠQR NN Ġlayers Ġuse Ġa Ġconv olution al Ġfilter Ġwidth ĠIN LINE FORM 0 Ġof Ġtwo Ġtim est eps . ĠWhile Ġthe ĠâĢ ľ medium âĢ Ŀ Ġmodels Ġused Ġin Ġother Ġwork ĠB IB REF 2 Ġ, ĠB IB REF 13 Ġconsist Ġof Ġ650 Ġunits Ġin Ġeach Ġlayer , Ġit Ġwas Ġmore Ġcomput ationally Ġconvenient Ġto Ġuse Ġa Ġmultiple Ġof Ġ32 . ĠAs Ġthe ĠPenn ĠTree bank Ġis Ġa Ġrelatively Ġsmall Ġdataset , Ġpreventing Ġover fitting Ġis Ġof Ġconsiderable Ġimportance Ġand Ġa Ġmajor Ġfocus Ġof Ġrecent Ġresearch . ĠIt Ġis Ġnot Ġobvious Ġin Ġadvance Ġwhich Ġof Ġthe Ġmany ĠR NN Ġregular ization Ġschemes Ġwould Ġperform Ġwell Ġwhen Ġapplied Ġto Ġthe ĠQR NN . ĠOur Ġtests Ġshowed Ġencouraging Ġresults Ġfrom Ġzone out Ġapplied Ġto Ġthe ĠQR NN 's Ġrecurrent Ġpool ing Ġlayer , Ġimplemented Ġas Ġdescribed Ġin ĠSection ĠSEC REF 5 Ġ. </s> The Ġexperimental Ġsettings Ġlargely Ġfollowed Ġthe ĠâĢ ľ medium âĢ Ŀ Ġsetup Ġof ĠB IB REF 2 Ġ. ĠOptim ization Ġwas Ġperformed Ġby Ġsto ch astic Ġgradient Ġdescent Ġ( SG D ) Ġwithout Ġmomentum . ĠThe Ġlearning Ġrate Ġwas Ġset Ġat Ġ1 Ġfor Ġsix Ġepoch s , Ġthen Ġdec ayed Ġby Ġ0 . 95 Ġfor Ġeach Ġsubsequent Ġepoch , Ġfor Ġa Ġtotal Ġof Ġ72 Ġepoch s . ĠWe Ġadditionally Ġused ĠIN LINE FORM 0 Ġregular ization Ġof ĠIN LINE FORM 1 Ġand Ġresc aled Ġgrad ients Ġwith Ġnorm Ġabove Ġ10 . ĠZone out Ġwas Ġapplied Ġby Ġperforming Ġdrop out Ġwith Ġratio Ġ0 . 1 Ġon Ġthe Ġforget Ġgates Ġof Ġthe ĠQR NN , Ġwithout Ġresc aling Ġthe Ġoutput Ġof Ġthe Ġdrop out Ġfunction . ĠBat ches Ġconsist Ġof Ġ20 Ġexamples , Ġeach Ġ105 Ġtim est eps . </s> Comp aring Ġour Ġresults Ġon Ġthe Ġg ated ĠQR NN Ġwith Ġzone out Ġto Ġthe Ġresults Ġof ĠL ST Ms Ġwith Ġboth Ġordinary Ġand Ġvari ational Ġdrop out Ġin ĠTable ĠT AB REF 14 Ġ, Ġwe Ġsee Ġthat Ġthe ĠQR NN Ġis Ġhighly Ġcompetitive . ĠThe ĠQR NN Ġwithout Ġzone out Ġstrongly Ġoutper forms Ġboth Ġour Ġmedium ĠL ST M Ġand Ġthe Ġmedium ĠL ST M Ġof ĠB IB REF 2 Ġwhich Ġdo Ġnot Ġuse Ġrecurrent Ġdrop out Ġand Ġis Ġeven Ġcompetitive Ġwith Ġvari ational ĠL ST Ms . ĠThis Ġmay Ġbe Ġdue Ġto Ġthe Ġlimited Ġcomputational Ġcapacity Ġthat Ġthe ĠQR NN 's Ġpool ing Ġlayer Ġhas Ġrelative Ġto Ġthe ĠL ST M 's Ġrecurrent Ġweights , Ġproviding Ġstructural Ġregular ization Ġover Ġthe Ġrec urrence . </s> Without Ġzone out , Ġearly Ġstopping Ġbased Ġupon Ġvalidation Ġloss Ġwas Ġrequired Ġas Ġthe ĠQR NN Ġwould Ġbegin Ġover fitting . ĠBy Ġapplying Ġa Ġsmall Ġamount Ġof Ġzone out Ġ( ĠIN LINE FORM 0 Ġ), Ġno Ġearly Ġstopping Ġis Ġrequired Ġand Ġthe ĠQR NN Ġachieves Ġcompetitive Ġlevels Ġof Ġperplex ity Ġto Ġthe Ġvari ational ĠL ST M Ġof ĠB IB REF 13 Ġ, Ġwhich Ġhad Ġvari ational Ġinference Ġbased Ġdrop out Ġof Ġ0 . 2 Ġapplied Ġrecurrent ly . ĠTheir Ġbest Ġperforming Ġvariation Ġalso Ġused ĠMonte ĠCarlo Ġ( MC ) Ġdrop out Ġaveraging Ġat Ġtest Ġtime Ġof Ġ1000 Ġdifferent Ġmasks , Ġmaking Ġit Ġcomput ationally Ġmore Ġexpensive Ġto Ġrun . </s> When Ġtraining Ġon Ġthe ĠPT B Ġdataset Ġwith Ġan ĠNVIDIA ĠK 40 ĠGPU , Ġwe Ġfound Ġthat Ġthe ĠQR NN Ġis Ġsubstantially Ġfaster Ġthan Ġa Ġstandard ĠL ST M , Ġeven Ġwhen Ġcomparing Ġagainst Ġthe Ġoptimized Ġcu D NN ĠL ST M . ĠIn ĠFigure ĠFIG REF 15 Ġwe Ġprovide Ġa Ġbreakdown Ġof Ġthe Ġtime Ġtaken Ġfor ĠCh ainer 's Ġdefault ĠL ST M , Ġthe Ġcu D NN ĠL ST M , Ġand ĠQR NN Ġto Ġperform Ġa Ġfull Ġforward Ġand Ġbackward Ġpass Ġon Ġa Ġsingle Ġbatch Ġduring Ġtraining Ġof Ġthe ĠR NN ĠLM Ġon ĠPT B . ĠFor Ġboth ĠL ST M Ġimplementations , Ġrunning Ġtime Ġwas Ġdominated Ġby Ġthe ĠR NN Ġcomput ations , Ġeven Ġwith Ġthe Ġhighly Ġoptimized Ġcu D NN Ġimplementation . ĠFor Ġthe ĠQR NN Ġimplementation , Ġhowever , Ġthe ĠâĢ ľ R NN âĢ Ŀ Ġlayers Ġare Ġno Ġlonger Ġthe Ġbottleneck . ĠIndeed , Ġthere Ġare Ġdiminishing Ġreturns Ġfrom Ġfurther Ġoptimization Ġof Ġthe ĠQR NN Ġitself Ġas Ġthe Ġsoft max Ġand Ġoptimization Ġoverhead Ġtake Ġequal Ġor Ġgreater Ġtime . ĠNote Ġthat Ġthe Ġsoft max , Ġover Ġa Ġvocabulary Ġsize Ġof Ġonly Ġ10 , 000 Ġwords , Ġis Ġrelatively Ġsmall ; Ġfor Ġtasks Ġwith Ġlarger Ġvoc ab ular ies , Ġthe Ġsoft max Ġwould Ġlikely Ġdominate Ġcomputation Ġtime . </s> It Ġis Ġalso Ġimportant Ġto Ġnote Ġthat Ġthe Ġcu D NN Ġlibrary 's ĠR NN Ġprim itives Ġdo Ġnot Ġnative ly Ġsupport Ġany Ġform Ġof Ġrecurrent Ġdrop out . ĠThat Ġis , Ġrunning Ġan ĠL ST M Ġthat Ġuses Ġa Ġstate - of - the - art Ġregular ization Ġscheme Ġat Ġcu D NN - like Ġspeeds Ġwould Ġlikely Ġrequire Ġan Ġentirely Ġcustom Ġkernel . </s> Character - level ĠNeural ĠMachine ĠTranslation </s> We Ġevaluate Ġthe Ġsequence - to - sequence ĠQR NN Ġarchitecture Ġdescribed Ġin ĠSEC REF 5 Ġon Ġa Ġchallenging Ġneural Ġmachine Ġtranslation Ġtask , ĠI W SL T ĠGerman âĢĵ English Ġspoken - domain Ġtranslation , Ġapplying Ġfully Ġcharacter - level Ġsegment ation . ĠThis Ġdataset Ġconsists Ġof Ġ209 , 772 Ġsentence Ġpairs Ġof Ġparallel Ġtraining Ġdata Ġfrom Ġtransc ribed ĠTED Ġand ĠTED x Ġpresentations , Ġwith Ġa Ġmean Ġsentence Ġlength Ġof Ġ103 Ġcharacters Ġfor ĠGerman Ġand Ġ93 Ġfor ĠEnglish . ĠWe Ġremove Ġtraining Ġsentences Ġwith Ġmore Ġthan Ġ300 Ġcharacters Ġin ĠEnglish Ġor ĠGerman , Ġand Ġuse Ġa Ġunified Ġvocabulary Ġof Ġ187 ĠUnicode Ġcode Ġpoints . </s> Our Ġbest Ġperformance Ġon Ġa Ġdevelopment Ġset Ġ( TED . t st 2013 ) Ġwas Ġachieved Ġusing Ġa Ġfour - layer Ġenc oder âĢĵ dec oder ĠQR NN Ġwith Ġ320 Ġunits Ġper Ġlayer , Ġno Ġdrop out Ġor ĠIN LINE FORM 0 Ġregular ization , Ġand Ġgradient Ġresc aling Ġto Ġa Ġmaximum Ġmagnitude Ġof Ġ5 . ĠInput s Ġwere Ġsupplied Ġto Ġthe Ġenc oder Ġreversed , Ġwhile Ġthe Ġenc oder Ġconv olutions Ġwere Ġnot Ġmasked . ĠThe Ġfirst Ġenc oder Ġlayer Ġused Ġconv olution al Ġfilter Ġwidth ĠIN LINE FORM 1 Ġ, Ġwhile Ġthe Ġother Ġenc oder Ġlayers Ġused ĠIN LINE FORM 2 Ġ. ĠOptim ization Ġwas Ġperformed Ġfor Ġ10 Ġepoch s Ġon Ġmin ib atches Ġof Ġ16 Ġexamples Ġusing ĠAdam ĠB IB REF 28 Ġwith ĠIN LINE FORM 3 Ġ, ĠIN LINE FORM 4 Ġ, ĠIN LINE FORM 5 Ġ, Ġand ĠIN LINE FORM 6 Ġ. ĠDec oding Ġwas Ġperformed Ġusing Ġbeam Ġsearch Ġwith Ġbeam Ġwidth Ġ8 Ġand Ġlength Ġnormal ization ĠIN LINE FORM 7 Ġ. ĠThe Ġmodified Ġlog - pro b ability Ġranking Ġcriterion Ġis Ġprovided Ġin Ġthe Ġappendix . </s> Results Ġusing Ġthis Ġarchitecture Ġwere Ġcompared Ġto Ġan Ġequal - sized Ġfour - layer Ġenc oder âĢĵ dec oder ĠL ST M Ġwith Ġattention , Ġapplying Ġdrop out Ġof Ġ0 . 2 . ĠWe Ġagain Ġoptimized Ġusing ĠAdam ; Ġother Ġhyper param eters Ġwere Ġequal Ġto Ġtheir Ġvalues Ġfor Ġthe ĠQR NN Ġand Ġthe Ġsame Ġbeam Ġsearch Ġprocedure Ġwas Ġapplied . ĠTable ĠT AB REF 17 Ġshows Ġthat Ġthe ĠQR NN Ġoutper formed Ġthe Ġcharacter - level ĠL ST M , Ġalmost Ġmatching Ġthe Ġperformance Ġof Ġa Ġword - level Ġattention al Ġbaseline . </s> Related ĠWork </s> Expl oring Ġalternatives Ġto Ġtraditional ĠR NN s Ġfor Ġsequence Ġtasks Ġis Ġa Ġmajor Ġarea Ġof Ġcurrent Ġresearch . ĠQu asi - rec urrent Ġneural Ġnetworks Ġare Ġrelated Ġto Ġseveral Ġsuch Ġrecently Ġdescribed Ġmodels , Ġespecially Ġthe Ġstrongly - ty ped Ġrecurrent Ġneural Ġnetworks Ġ( T - R NN ) Ġintroduced Ġby ĠB IB REF 12 Ġ. ĠWhile Ġthe Ġmotivation Ġand Ġconstraints Ġdescribed Ġin Ġthat Ġwork Ġare Ġdifferent , ĠB IB REF 12 Ġ' s Ġconcepts Ġof ĠâĢ ľ learn ware âĢ Ŀ Ġand ĠâĢ ľ f irm ware âĢ Ŀ Ġparallel Ġour Ġdiscussion Ġof Ġconv olution - like Ġand Ġpool ing - like Ġsub comp onents . ĠAs Ġthe Ġuse Ġof Ġa Ġfully Ġconnected Ġlayer Ġfor Ġrecurrent Ġconnections Ġviolates Ġthe Ġconstraint Ġof ĠâĢ ľ strong Ġtyping âĢ Ŀ , Ġall Ġstrongly - ty ped ĠR NN Ġarchitectures Ġ( including Ġthe ĠT - R NN , ĠT - GR U , Ġand ĠT - L ST M ) Ġare Ġalso Ġquasi - rec urrent . ĠHowever , Ġsome ĠQR NN Ġmodels Ġ( including Ġthose Ġwith Ġattention Ġor Ġskip - connect ions ) Ġare Ġnot ĠâĢ ľ strong ly Ġtyped âĢ Ŀ . ĠIn Ġparticular , Ġa ĠT - R NN Ġdiffers Ġfrom Ġa ĠQR NN Ġas Ġdescribed Ġin Ġthis Ġpaper Ġwith Ġfilter Ġsize Ġ1 Ġand Ġf - pool ing Ġonly Ġin Ġthe Ġabsence Ġof Ġan Ġactivation Ġfunction Ġon ĠIN LINE FORM 0 Ġ. ĠSimilarly , ĠT - GR Us Ġand ĠT - L ST Ms Ġdiffer Ġfrom ĠQR NN s Ġwith Ġfilter Ġsize Ġ2 Ġand Ġfo - Ġor Ġif o - pool ing Ġrespectively Ġin Ġthat Ġthey Ġlack ĠIN LINE FORM 1 Ġon ĠIN LINE FORM 2 Ġand Ġuse ĠIN LINE FORM 3 Ġrather Ġthan Ġs igm oid Ġon ĠIN LINE FORM 4 Ġ. </s> The ĠQR NN Ġis Ġalso Ġrelated Ġto Ġwork Ġin Ġhybrid Ġconv olution al âĢĵ rec urrent Ġmodels . ĠB IB REF 31 Ġapply ĠCNN s Ġat Ġthe Ġword Ġlevel Ġto Ġgenerate ĠIN LINE FORM 0 Ġ- gram Ġfeatures Ġused Ġby Ġan ĠL ST M Ġfor Ġtext Ġclassification . ĠB IB REF 32 Ġalso Ġtackle Ġtext Ġclassification Ġby Ġapplying Ġconv olutions Ġat Ġthe Ġcharacter Ġlevel , Ġwith Ġa Ġstride Ġto Ġreduce Ġsequence Ġlength , Ġthen Ġfeeding Ġthese Ġfeatures Ġinto Ġa Ġbid irection al ĠL ST M . ĠA Ġsimilar Ġapproach Ġwas Ġtaken Ġby ĠB IB REF 10 Ġfor Ġcharacter - level Ġmachine Ġtranslation . ĠTheir Ġmodel 's Ġenc oder Ġuses Ġa Ġconv olution al Ġlayer Ġfollowed Ġby Ġmax - pool ing Ġto Ġreduce Ġsequence Ġlength , Ġa Ġfour - layer Ġhighway Ġnetwork , Ġand Ġa Ġbid irection al ĠGR U . ĠThe Ġparallel ism Ġof Ġthe Ġconv olution al , Ġpool ing , Ġand Ġhighway Ġlayers Ġallows Ġtraining Ġspeed Ġcomparable Ġto Ġsub word - level Ġmodels Ġwithout Ġhard - coded Ġtext Ġsegment ation . </s> The ĠQR NN Ġenc oder âĢĵ dec oder Ġmodel Ġshares Ġthe Ġfavorable Ġparallel ism Ġand Ġpath - length Ġproperties Ġexhibited Ġby Ġthe ĠByte Net ĠB IB REF 33 Ġ, Ġan Ġarchitecture Ġfor Ġcharacter - level Ġmachine Ġtranslation Ġbased Ġon Ġresidual Ġconv olutions Ġover Ġbinary Ġtrees . ĠTheir Ġmodel Ġwas Ġconstructed Ġto Ġachieve Ġthree Ġdesired Ġproperties : Ġparallel ism , Ġlinear - time Ġcomputational Ġcomplexity , Ġand Ġshort Ġpaths Ġbetween Ġany Ġpair Ġof Ġwords Ġin Ġorder Ġto Ġbetter Ġpropagate Ġgradient Ġsignals . </s> Conclusion </s> Int uitive ly , Ġmany Ġaspects Ġof Ġthe Ġsemantics Ġof Ġlong Ġsequences Ġare Ġcontext - inv ari ant Ġand Ġcan Ġbe Ġcomputed Ġin Ġparallel Ġ( e . g ., Ġconv olution ally ), Ġbut Ġsome Ġaspects Ġrequire Ġlong - distance Ġcontext Ġand Ġmust Ġbe Ġcomputed Ġrecurrent ly . ĠMany Ġexisting Ġneural Ġnetwork Ġarchitectures Ġeither Ġfail Ġto Ġtake Ġadvantage Ġof Ġthe Ġcontextual Ġinformation Ġor Ġfail Ġto Ġtake Ġadvantage Ġof Ġthe Ġparallel ism . ĠQR NN s Ġexploit Ġboth Ġparallel ism Ġand Ġcontext , Ġexhibiting Ġadvantages Ġfrom Ġboth Ġconv olution al Ġand Ġrecurrent Ġneural Ġnetworks . ĠQR NN s Ġhave Ġbetter Ġpredictive Ġaccuracy Ġthan ĠL ST M - based Ġmodels Ġof Ġequal Ġhidden Ġsize , Ġeven Ġthough Ġthey Ġuse Ġfewer Ġparameters Ġand Ġrun Ġsubstantially Ġfaster . ĠOur Ġexperiments Ġshow Ġthat Ġthe Ġspeed Ġand Ġaccuracy Ġadvantages Ġremain Ġconsistent Ġacross Ġtasks Ġand Ġat Ġboth Ġword Ġand Ġcharacter Ġlevels . </s> Ext ensions Ġto Ġboth ĠCNN s Ġand ĠR NN s Ġare Ġoften Ġdirectly Ġapplicable Ġto Ġthe ĠQR NN , Ġwhile Ġthe Ġmodel 's Ġhidden Ġstates Ġare Ġmore Ġinterpret able Ġthan Ġthose Ġof Ġother Ġrecurrent Ġarchitectures Ġas Ġits Ġchannels Ġmaintain Ġtheir Ġindependence Ġacross Ġtim est eps . ĠWe Ġbelieve Ġthat ĠQR NN s Ġcan Ġserve Ġas Ġa Ġbuilding Ġblock Ġfor Ġlong - sequence Ġtasks Ġthat Ġwere Ġpreviously Ġimpractical Ġwith Ġtraditional ĠR NN s . </s> Be am Ġsearch Ġranking Ġcriterion </s> The Ġmodified Ġlog - pro b ability Ġranking Ġcriterion Ġwe Ġused Ġin Ġbeam Ġsearch Ġfor Ġtranslation Ġexperiments Ġis : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġa Ġlength Ġnormal ization Ġparameter ĠB IB REF 34 Ġ, ĠIN LINE FORM 1 Ġis Ġthe ĠIN LINE FORM 2 Ġth Ġoutput Ġcharacter , Ġand ĠIN LINE FORM 3 Ġis Ġa ĠâĢ ľ target Ġlength âĢ Ŀ Ġequal Ġto Ġthe Ġsource Ġsentence Ġlength Ġplus Ġfive Ġcharacters . ĠThis Ġreduces Ġat ĠIN LINE FORM 4 Ġto Ġordinary Ġbeam Ġsearch Ġwith Ġprobabilities : ĠDIS PLAY FORM 0 </s> and Ġat ĠIN LINE FORM 0 Ġto Ġbeam Ġsearch Ġwith Ġprobabilities Ġnormalized Ġby Ġlength Ġ( up Ġto Ġthe Ġtarget Ġlength ): ĠDIS PLAY FORM 0 </s> Con venient ly , Ġthis Ġranking Ġcriterion Ġcan Ġbe Ġcomputed Ġat Ġintermediate Ġbeam - search Ġtim est eps , Ġob vi ating Ġthe Ġneed Ġto Ġapply Ġa Ġseparate Ġre ranking Ġon Ġcomplete Ġhypotheses .