(i) models that use plain stacked LSTMs, (ii) models with different INLINEFORM0, (iii) models without INLINEFORM1, (iv) models that integrate lower contexts via peephole connections
EXTRACTIVE
In this subsection, we see the influence of each component of a model on performance by removing or replacing its components. the SNLI dataset is used for experiments, and the best performing configuration is used as a baseline for modifications. We consider the following variants: (i) models that use plain stacked LSTMs, (ii) models with different INLINEFORM0 , (iii) models without INLINEFORM1 , and (iv) models that integrate lower contexts via peephole connections.
1809.02279
Introduction </s> In Ġthe Ġfield Ġof Ġnatural Ġlanguage Ġprocessing Ġ( N LP ), Ġthe Ġmost Ġprevalent Ġneural Ġapproach Ġto Ġobtaining Ġsentence Ġrepresentations Ġis Ġto Ġuse Ġrecurrent Ġneural Ġnetworks Ġ( R NN s ), Ġwhere Ġwords Ġin Ġa Ġsentence Ġare Ġprocessed Ġin Ġa Ġsequential Ġand Ġrecurrent Ġmanner . ĠAlong Ġwith Ġtheir Ġintuitive Ġdesign , ĠR NN s Ġhave Ġshown Ġoutstanding Ġperformance Ġacross Ġvarious ĠN LP Ġtasks Ġe . g . Ġlanguage Ġmodeling ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, Ġmachine Ġtranslation ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ, Ġtext Ġclassification ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, Ġand Ġparsing ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ. </s> Among Ġseveral Ġvariants Ġof Ġthe Ġoriginal ĠR NN ĠB IB REF 9 Ġ, Ġg ated Ġrecurrent Ġarchitectures Ġsuch Ġas Ġlong Ġshort - term Ġmemory Ġ( L ST M ) ĠB IB REF 10 Ġand Ġg ated Ġrecurrent Ġunit Ġ( GR U ) ĠB IB REF 2 Ġhave Ġbeen Ġaccepted Ġas Ġde - fact o Ġstandard Ġchoices Ġfor ĠR NN s Ġdue Ġto Ġtheir Ġcapability Ġof Ġaddressing Ġthe Ġvanishing Ġand Ġexploding Ġgradient Ġproblem Ġand Ġconsidering Ġlong - term Ġdependencies . ĠG ated ĠR NN s Ġachieve Ġthese Ġproperties Ġby Ġintroducing Ġadditional Ġg ating Ġunits Ġthat Ġlearn Ġto Ġcontrol Ġthe Ġamount Ġof Ġinformation Ġto Ġbe Ġtransferred Ġor Ġforgotten ĠB IB REF 11 Ġ, Ġand Ġare Ġproven Ġto Ġwork Ġwell Ġwithout Ġrelying Ġon Ġcomplex Ġoptimization Ġalgorithms Ġor Ġcareful Ġinitialization ĠB IB REF 12 Ġ. </s> Meanwhile , Ġthe Ġcommon Ġpractice Ġfor Ġfurther Ġenhancing Ġthe Ġexpress iveness Ġof ĠR NN s Ġis Ġto Ġstack Ġmultiple ĠR NN Ġlayers , Ġeach Ġof Ġwhich Ġhas Ġdistinct Ġparameter Ġsets Ġ( st acked ĠR NN ) ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ. ĠIn Ġstacked ĠR NN s , Ġthe Ġhidden Ġstates Ġof Ġa Ġlayer Ġare Ġfed Ġas Ġinput Ġto Ġthe Ġsubsequent Ġlayer , Ġand Ġthey Ġare Ġshown Ġto Ġwork Ġwell Ġdue Ġto Ġincreased Ġdepth ĠB IB REF 15 Ġor Ġtheir Ġability Ġto Ġcapture Ġhierarchical Ġtime Ġseries ĠB IB REF 16 Ġwhich Ġare Ġinherent Ġto Ġthe Ġnature Ġof Ġthe Ġproblem Ġbeing Ġmodeled . </s> However Ġthis Ġsetting Ġof Ġstacking ĠR NN s Ġmight Ġhinder Ġthe Ġpossibility Ġof Ġmore Ġsophisticated Ġrec urrence - based Ġstructures Ġsince Ġthe Ġinformation Ġfrom Ġlower Ġlayers Ġis Ġsimply Ġtreated Ġas Ġinput Ġto Ġthe Ġnext Ġlayer , Ġrather Ġthan Ġas Ġanother Ġclass Ġof Ġstate Ġthat Ġparticipates Ġin Ġcore ĠR NN Ġcomput ations . ĠEspecially Ġfor Ġg ated ĠR NN s Ġsuch Ġas ĠL ST Ms Ġand ĠGR Us , Ġthis Ġmeans Ġthat Ġlayer - to - layer Ġconnections Ġcannot Ġfully Ġbenefit Ġfrom Ġthe Ġcarefully Ġconstructed Ġg ating Ġmechanism Ġused Ġin Ġtemporal Ġtransitions . ĠSome Ġrecent Ġwork Ġon Ġstacking ĠR NN s Ġsuggests Ġalternative Ġmethods Ġthat Ġencourage Ġdirect Ġand Ġeffective Ġinteraction Ġbetween ĠR NN Ġlayers Ġby Ġadding Ġresidual Ġconnections ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ, Ġby Ġshortcut Ġconnections ĠB IB REF 18 Ġ, ĠB IB REF 19 Ġ, Ġor Ġby Ġusing Ġcell Ġstates Ġof ĠL ST Ms ĠB IB REF 20 Ġ, ĠB IB REF 21 Ġ. </s> In Ġthis Ġpaper , Ġwe Ġpropose Ġa Ġmethod Ġof Ġconstructing Ġmulti - layer ĠL ST Ms Ġwhere Ġcell Ġstates Ġare Ġused Ġin Ġcontrolling Ġthe Ġvertical Ġinformation Ġflow . ĠThis Ġsystem Ġutilizes Ġstates Ġfrom Ġthe Ġleft Ġand Ġthe Ġlower Ġcontext Ġequally Ġin Ġcomputation Ġof Ġthe Ġnew Ġstate , Ġthus Ġthe Ġinformation Ġfrom Ġlower Ġlayers Ġis Ġelabor ately Ġfiltered Ġand Ġreflected Ġthrough Ġa Ġsoft Ġg ating Ġmechanism . ĠOur Ġmethod Ġis Ġeasy - to - im plement , Ġeffective , Ġand Ġcan Ġreplace Ġconventional Ġstacked ĠL ST Ms Ġwithout Ġmuch Ġmodification Ġof Ġthe Ġoverall Ġarchitecture . </s> We Ġcall Ġthe Ġproposed Ġarchitecture ĠCell - aware ĠSt acked ĠL ST M , Ġor ĠCAS - L ST M , Ġand Ġevaluate Ġour Ġmethod Ġon Ġmultiple Ġbenchmark Ġdatasets : ĠSN LI ĠB IB REF 22 Ġ, ĠMulti N LI ĠB IB REF 23 Ġ, ĠQu ora ĠQuestion ĠP airs ĠB IB REF 24 Ġ, Ġand ĠS ST ĠB IB REF 25 Ġ. ĠFrom Ġexperiments Ġwe Ġshow Ġthat Ġthe ĠCAS - L ST Ms Ġconsistently Ġoutper form Ġtypical Ġstacked ĠL ST Ms , Ġopening Ġthe Ġpossibility Ġof Ġperformance Ġimprovement Ġof Ġarchitectures Ġthat Ġuse Ġstacked ĠL ST Ms . </s> Our Ġcontribution Ġis Ġsummarized Ġas Ġfollows . </s> This Ġpaper Ġis Ġorganized Ġas Ġfollows . ĠWe Ġgive Ġa Ġdetailed Ġdescription Ġabout Ġthe Ġproposed Ġmethod Ġin ĠÂ§ ĠSEC REF 2 Ġ. ĠExperimental Ġresults Ġare Ġgiven Ġin ĠÂ§ ĠSEC REF 3 Ġ. ĠWe Ġstudy Ġprior Ġwork Ġrelated Ġto Ġour Ġobjective Ġin ĠÂ§ ĠSEC REF 4 Ġand Ġconclude Ġin ĠÂ§ ĠSEC REF 5 Ġ. </s> Model ĠDescription </s> In Ġthis Ġsection , Ġwe Ġgive Ġa Ġdetailed Ġformulation Ġof Ġthe Ġarchitectures Ġused Ġin Ġexperiments . </s> Not ation </s> Throughout Ġthis Ġpaper , Ġwe Ġdenote Ġmat rices Ġas Ġbold face Ġcapital Ġletters Ġ( ĠIN LINE FORM 0 Ġ), Ġvectors Ġas Ġbold face Ġlower case Ġletters Ġ( ĠIN LINE FORM 1 Ġ), Ġand Ġscal ars Ġas Ġnormal Ġital ic Ġletters Ġ( ĠIN LINE FORM 2 Ġ). ĠFor ĠL ST M Ġstates , Ġwe Ġdenote Ġa Ġhidden Ġstate Ġas ĠIN LINE FORM 3 Ġand Ġa Ġcell Ġstate Ġas ĠIN LINE FORM 4 Ġ. ĠAlso , Ġa Ġlayer Ġindex Ġof ĠIN LINE FORM 5 Ġor ĠIN LINE FORM 6 Ġis Ġden oted Ġby Ġsupers cript Ġand Ġa Ġtime Ġindex Ġis Ġden oted Ġby Ġa Ġsubscript , Ġi . e . ĠIN LINE FORM 7 Ġindicates Ġthe Ġhidden Ġstate Ġat Ġtime ĠIN LINE FORM 8 Ġand Ġlayer ĠIN LINE FORM 9 Ġ. ĠIN LINE FORM 10 Ġmeans Ġthe Ġelement - wise Ġmultiplication Ġbetween Ġtwo Ġvectors . ĠWe Ġwrite ĠIN LINE FORM 11 Ġ- th Ġcomponent Ġof Ġvector ĠIN LINE FORM 12 Ġas ĠIN LINE FORM 13 Ġ. ĠAll Ġvectors Ġare Ġassumed Ġto Ġbe Ġcolumn Ġvectors . </s> St acked ĠL ST Ms </s> While Ġthere Ġexist Ġvarious Ġversions Ġof ĠL ST M Ġformulation , Ġin Ġthis Ġwork Ġwe Ġuse Ġthe Ġfollowing , Ġone Ġof Ġthe Ġmost Ġcommon Ġversions : ĠDIS PLAY FORM 0 ĠDIS PLAY FORM 1 </s> where ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, Ġand ĠIN LINE FORM 2 Ġ, ĠIN LINE FORM 3 Ġ, ĠIN LINE FORM 4 Ġare Ġtrain able Ġparameters . ĠIN LINE FORM 5 Ġand ĠIN LINE FORM 6 Ġare Ġthe Ġs igm oid Ġactivation Ġand Ġthe Ġhyper b olic Ġtang ent Ġactivation Ġfunction Ġrespectively . ĠAlso Ġwe Ġassume Ġthat ĠIN LINE FORM 7 Ġwhere ĠIN LINE FORM 8 Ġis Ġthe ĠIN LINE FORM 9 Ġ- th Ġinput Ġto Ġthe Ġnetwork . </s> The Ġinput Ġgate ĠIN LINE FORM 0 Ġand Ġthe Ġforget Ġgate ĠIN LINE FORM 1 Ġcontrol Ġthe Ġamount Ġof Ġinformation Ġtransmitted Ġfrom ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġ, Ġthe Ġcandidate Ġcell Ġstate Ġand Ġthe Ġprevious Ġcell Ġstate , Ġto Ġthe Ġnew Ġcell Ġstate ĠIN LINE FORM 4 Ġ. ĠSimilarly Ġthe Ġoutput Ġgate ĠIN LINE FORM 5 Ġsoft - select s Ġwhich Ġportion Ġof Ġthe Ġcell Ġstate ĠIN LINE FORM 6 Ġis Ġto Ġbe Ġused Ġin Ġthe Ġfinal Ġhidden Ġstate . </s> We Ġcan Ġclearly Ġsee Ġthat Ġcell Ġstates Ġ( ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġ) Ġplay Ġa Ġcrucial Ġrole Ġin Ġforming Ġhorizontal Ġrec urrence . ĠHowever Ġthe Ġcurrent Ġformulation Ġdoes Ġnot Ġconsider ĠIN LINE FORM 3 Ġ, Ġthe Ġcell Ġstate Ġfrom ĠIN LINE FORM 4 Ġ- th Ġlayer , Ġin Ġcomputation Ġand Ġthus Ġthe Ġlower Ġcontext Ġis Ġreflected Ġonly Ġthrough Ġthe Ġrudimentary Ġway , Ġhind ering Ġthe Ġpossibility Ġof Ġcontrolling Ġvertical Ġinformation Ġflow . </s> Cell - aware ĠSt acked ĠL ST Ms </s> Now Ġwe Ġextend Ġthe Ġstacked ĠL ST M Ġformulation Ġdefined Ġabove Ġto Ġaddress Ġthe Ġproblem Ġnoted Ġin Ġthe Ġprevious Ġsubsection . ĠTo Ġenhance Ġthe Ġinteraction Ġbetween Ġlayers Ġin Ġa Ġway Ġsimilar Ġto Ġhow ĠL ST Ms Ġkeep Ġand Ġforget Ġthe Ġinformation Ġfrom Ġthe Ġprevious Ġtime Ġstep , Ġwe Ġintroduce Ġthe Ġadditional Ġforget Ġgate ĠIN LINE FORM 0 Ġthat Ġdetermines Ġwhether Ġto Ġaccept Ġor Ġignore Ġthe Ġsignals Ġcoming Ġfrom Ġthe Ġprevious Ġlayer . ĠTherefore Ġthe Ġproposed ĠCell - aware ĠSt acked ĠL ST M Ġis Ġformulated Ġas Ġfollows : ĠDIS PLAY FORM 0 ĠDIS PLAY FORM 1 </s> where ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġ. ĠIN LINE FORM 2 Ġcan Ġeither Ġbe Ġa Ġvector Ġof Ġconstants Ġor Ġparameters . ĠWhen ĠIN LINE FORM 3 Ġ, Ġthe Ġequations Ġdefined Ġin Ġthe Ġprevious Ġsubsection Ġare Ġused . ĠTherefore , Ġit Ġcan Ġbe Ġsaid Ġthat Ġeach Ġnon - bottom Ġlayer Ġof ĠCAS - L ST M Ġaccepts Ġtwo Ġsets Ġof Ġhidden Ġand Ġcell Ġstates âĢĶ one Ġfrom Ġthe Ġleft Ġcontext Ġand Ġthe Ġother Ġfrom Ġthe Ġbelow Ġcontext . ĠThe Ġleft Ġand Ġthe Ġbelow Ġcontext Ġparticipate Ġin Ġcomputation Ġwith Ġthe Ġequivalent Ġprocedure Ġso Ġthat Ġthe Ġinformation Ġfrom Ġlower Ġlayers Ġcan Ġbe Ġefficiently Ġpropag ated . ĠFig . ĠFIG REF 1 Ġcompares ĠCAS - L ST M Ġto Ġthe Ġconventional Ġstacked ĠL ST M Ġarchitecture , Ġand ĠFig . ĠFIG REF 8 Ġdepicts Ġthe Ġcomputation Ġflow Ġof Ġthe ĠCAS - L ST M . </s> We Ġargue Ġthat Ġconsidering ĠIN LINE FORM 0 Ġin Ġcomputation Ġis Ġbeneficial Ġfor Ġthe Ġfollowing Ġreasons . ĠFirst , ĠIN LINE FORM 1 Ġcontains Ġadditional Ġinformation Ġcompared Ġto ĠIN LINE FORM 2 Ġsince Ġit Ġis Ġnot Ġfiltered Ġby ĠIN LINE FORM 3 Ġ. ĠThus Ġa Ġmodel Ġthat Ġdirectly Ġuses ĠIN LINE FORM 4 Ġdoes Ġnot Ġrely Ġsolely Ġon ĠIN LINE FORM 5 Ġfor Ġextracting Ġinformation , Ġdue Ġto Ġthe Ġfact Ġthat Ġit Ġhas Ġaccess Ġto Ġthe Ġraw Ġinformation ĠIN LINE FORM 6 Ġ, Ġas Ġin Ġtemporal Ġconnections . ĠIn Ġother Ġwords , ĠIN LINE FORM 7 Ġno Ġlonger Ġhas Ġto Ġtake Ġall Ġresponsibility Ġfor Ġselecting Ġuseful Ġfeatures Ġfor Ġboth Ġhorizontal Ġand Ġvertical Ġtransitions , Ġand Ġthe Ġburden Ġof Ġselecting Ġinformation Ġis Ġshared Ġwith ĠIN LINE FORM 8 Ġ. </s> Another Ġadvantage Ġof Ġusing Ġthe ĠIN LINE FORM 0 Ġlies Ġin Ġthe Ġfact Ġthat Ġit Ġdirectly Ġconnects ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġ. ĠThis Ġdirect Ġconnection Ġhelps Ġand Ġstabil izes Ġtraining , Ġsince Ġthe Ġterminal Ġerror Ġsignals Ġcan Ġbe Ġeasily Ġback prop ag ated Ġto Ġmodel Ġparameters . ĠFig . ĠFIG REF 23 Ġillustrates Ġpaths Ġbetween Ġthe Ġtwo Ġcell Ġstates . </s> We Ġfind Ġexperiment ally Ġthat Ġthere Ġis Ġlittle Ġdifference Ġbetween Ġletting ĠIN LINE FORM 0 Ġbe Ġconstant Ġand Ġletting Ġit Ġbe Ġtrain able Ġparameters , Ġthus Ġwe Ġset ĠIN LINE FORM 1 Ġin Ġall Ġexperiments . ĠWe Ġalso Ġexperimented Ġwith Ġthe Ġarchitecture Ġwithout ĠIN LINE FORM 2 Ġi . e . Ġtwo Ġcell Ġstates Ġare Ġcombined Ġby Ġun weight ed Ġsumm ation Ġsimilar Ġto Ġmult id imensional ĠR NN s ĠB IB REF 27 Ġ, Ġand Ġfound Ġthat Ġit Ġleads Ġto Ġperformance Ġdegradation Ġand Ġunstable Ġconvergence , Ġlikely Ġdue Ġto Ġmismatch Ġin Ġthe Ġrange Ġof Ġcell Ġstate Ġvalues Ġbetween Ġlayers Ġ( ĠIN LINE FORM 3 Ġfor Ġthe Ġfirst Ġlayer Ġand ĠIN LINE FORM 4 Ġfor Ġthe Ġothers ). ĠExperimental Ġresults Ġon Ġvarious ĠIN LINE FORM 5 Ġare Ġpresented Ġin ĠÂ§ ĠSEC REF 3 Ġ. </s> The Ġidea Ġof Ġhaving Ġmultiple Ġstates Ġis Ġalso Ġrelated Ġto Ġtree - struct ured ĠR NN s ĠB IB REF 29 Ġ, ĠB IB REF 30 Ġ. ĠAmong Ġthem , Ġtree - struct ured ĠL ST Ms Ġ( Tree - L ST Ms ) ĠB IB REF 31 Ġ, ĠB IB REF 32 Ġ, ĠB IB REF 33 Ġare Ġsimilar Ġto Ġours Ġin Ġthat Ġthey Ġuse Ġboth Ġhidden Ġand Ġcell Ġstates Ġfrom Ġchildren Ġnodes . ĠIn ĠTree - L ST Ms , Ġstates Ġfor Ġall Ġchildren Ġnodes Ġare Ġregarded Ġas Ġinput , Ġand Ġthey Ġparticipate Ġin Ġthe Ġcomputation Ġequally Ġthrough Ġweight - shared Ġ( in ĠChild - Sum ĠTree - L ST Ms ) Ġor Ġweight - un shared Ġ( in ĠIN LINE FORM 0 Ġ- ary ĠTree - L ST Ms ) Ġprojection . ĠFrom Ġthis Ġperspective , Ġeach ĠCAS - L ST M Ġlayer Ġ( where ĠIN LINE FORM 1 Ġ) Ġcan Ġbe Ġseen Ġas Ġa Ġbinary ĠTree - L ST M Ġwhere Ġthe Ġstructures Ġit Ġoperates Ġon Ġare Ġfixed Ġto Ġright - br anch ing Ġtrees . ĠThe Ġuse Ġof Ġcell Ġstate Ġin Ġcomputation Ġcould Ġbe Ġone Ġreason Ġthat ĠTree - L ST Ms Ġperform Ġbetter Ġthan Ġsequential ĠL ST Ms Ġeven Ġwhen Ġtrivial Ġtrees Ġ( st rict ly Ġleft - Ġor Ġright - br anch ing ) Ġare Ġgiven ĠB IB REF 34 Ġ. </s> Mult id imensional ĠR NN s Ġ( M DR NN ) Ġare Ġan Ġextension Ġof Ġ1 D Ġsequential ĠR NN s Ġthat Ġcan Ġaccept Ġmult id imensional Ġinput Ġe . g . Ġimages , Ġand Ġhave Ġbeen Ġsuccessfully Ġapplied Ġto Ġimage Ġsegment ation ĠB IB REF 26 Ġand Ġhandwriting Ġrecognition ĠB IB REF 27 Ġ. ĠNotably Ġmult id imensional ĠL ST Ms Ġ( MD L ST M ) ĠB IB REF 27 Ġhave Ġan Ġanalogous Ġformulation Ġto Ġours Ġexcept Ġthe ĠIN LINE FORM 0 Ġterm Ġand Ġthe Ġfact Ġthat Ġwe Ġuse Ġdistinct Ġweights Ġper Ġcolumn Ġ( or Ġ` layer ' Ġin Ġour Ġcase ). ĠFrom Ġthis Ġview , ĠCAS - L ST M Ġcan Ġbe Ġseen Ġas Ġa Ġcertain Ġkind Ġof ĠMD L ST M Ġthat Ġaccepts Ġa Ġ2 D Ġinput ĠIN LINE FORM 1 Ġ. ĠGrid ĠL ST Ms ĠB IB REF 21 Ġalso Ġtake ĠIN LINE FORM 2 Ġinputs Ġbut Ġemit ĠIN LINE FORM 3 Ġoutputs , Ġwhich Ġis Ġdifferent Ġfrom Ġour Ġcase Ġwhere Ġa Ġsingle Ġset Ġof Ġhidden Ġand Ġcell Ġstates Ġis Ġproduced . </s> Sent ence ĠEnc od ers </s> The Ġsentence Ġenc oder Ġnetwork Ġwe Ġuse Ġin Ġour Ġexperiments Ġtakes ĠIN LINE FORM 0 Ġwords Ġ( ass umed Ġto Ġbe Ġone - hot Ġvectors ) Ġas Ġinput . ĠThe Ġwords Ġare Ġprojected Ġto Ġcorresponding Ġword Ġrepresentations : ĠIN LINE FORM 1 Ġwhere ĠIN LINE FORM 2 Ġ. ĠThen ĠIN LINE FORM 3 Ġis Ġfed Ġto Ġa ĠIN LINE FORM 4 Ġ- layer ĠCAS - L ST M Ġmodel , Ġresulting Ġin Ġthe Ġrepresentations ĠIN LINE FORM 5 Ġ. ĠThe Ġsentence Ġrepresentation , ĠIN LINE FORM 6 Ġ, Ġis Ġcomputed Ġby Ġmax - pool ing ĠIN LINE FORM 7 Ġover Ġtime Ġas Ġin Ġthe Ġwork Ġof ĠB IB REF 35 Ġ. ĠSimilar Ġto Ġtheir Ġresults , Ġfrom Ġpreliminary Ġexperiments Ġwe Ġfound Ġthat Ġthe Ġmax - pool ing Ġperforms Ġconsistently Ġbetter Ġthan Ġmean - Ġand Ġlast - pool ing . </s> To Ġmake Ġmodels Ġmore Ġexpressive , Ġa Ġbid irection al ĠCAS - L ST M Ġnetwork Ġmay Ġalso Ġbe Ġused . ĠIn Ġthe Ġbid irection al Ġcase , Ġthe Ġforward Ġrepresentations ĠIN LINE FORM 0 Ġand Ġthe Ġbackward Ġrepresentations ĠIN LINE FORM 1 Ġare Ġconc aten ated Ġand Ġmax - pool ed Ġto Ġyield Ġthe Ġsentence Ġrepresentation ĠIN LINE FORM 2 Ġ. ĠWe Ġcall Ġthis Ġbid irection al Ġarchitecture ĠBi - C AS - L ST M Ġin Ġexperiments . </s> Top - layer ĠClass ifiers </s> For Ġthe Ġnatural Ġlanguage Ġinference Ġexperiments , Ġwe Ġuse Ġthe Ġfollowing Ġhe uristic Ġfunction Ġproposed Ġby ĠB IB REF 36 Ġin Ġfeature Ġextraction : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġmeans Ġvector Ġconc aten ation , Ġand ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġare Ġapplied Ġelement - wise . </s> And Ġwe Ġuse Ġthe Ġfollowing Ġfunction Ġin Ġparaph rase Ġidentification Ġexperiments : ĠDIS PLAY FORM 0 </s> as Ġin Ġthe Ġwork Ġof ĠB IB REF 37 Ġ. </s> For Ġsentiment Ġclassification , Ġwe Ġuse Ġthe Ġsentence Ġrepresentation Ġitself . ĠDIS PLAY FORM 0 </s> We Ġfeed Ġthe Ġfeature Ġextracted Ġfrom ĠIN LINE FORM 0 Ġas Ġinput Ġto Ġthe ĠML P Ġclass ifier Ġwith ĠRe LU Ġactivation Ġfollowed Ġby Ġthe Ġfully - connected Ġsoft max Ġlayer Ġto Ġpredict Ġthe Ġlabel Ġdistribution : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġis Ġthe Ġnumber Ġof Ġlabel Ġclasses , Ġand ĠIN LINE FORM 2 Ġthe Ġdimension Ġof Ġthe ĠML P Ġoutput , </s> Exper iments </s> We Ġevaluate Ġour Ġmethod Ġon Ġnatural Ġlanguage Ġinference Ġ( N LI ), Ġparaph rase Ġidentification Ġ( PI ), Ġand Ġsentiment Ġclassification . ĠWe Ġalso Ġconduct Ġanalysis Ġon Ġgate Ġvalues Ġand Ġexperiments Ġon Ġmodel Ġvariants . ĠFor Ġdetailed Ġexperimental Ġsettings , Ġwe Ġrefer Ġreaders Ġto Ġthe Ġsupplemental Ġmaterial . </s> For Ġthe ĠNL I Ġand ĠPI Ġtasks , Ġthere Ġexists Ġrecent Ġwork Ġspecializing Ġin Ġsentence Ġpair Ġclassification . ĠHowever Ġin Ġthis Ġwork Ġwe Ġconf ine Ġour Ġmodel Ġto Ġthe Ġarchitecture Ġthat Ġenc odes Ġeach Ġsentence Ġusing Ġa Ġshared Ġenc oder Ġwithout Ġany Ġinter - sent ence Ġinteraction , Ġin Ġorder Ġto Ġfocus Ġon Ġthe Ġeffectiveness Ġof Ġthe Ġmodels Ġin Ġextracting Ġsemantics . ĠBut Ġnote Ġthat Ġthe Ġapplic ability Ġof ĠCAS - L ST M Ġis Ġnot Ġlimited Ġto Ġsentence Ġencoding Ġbased Ġapproaches . </s> Natural ĠLanguage ĠIn ference </s> For Ġthe Ġevaluation Ġof Ġperformance Ġof Ġthe Ġproposed Ġmethod Ġon Ġthe ĠNL I Ġtask , ĠSN LI ĠB IB REF 22 Ġand ĠMulti N LI ĠB IB REF 23 Ġdatasets Ġare Ġused . ĠThe Ġobjective Ġof Ġboth Ġdatasets Ġis Ġto Ġpredict Ġthe Ġrelationship Ġbetween Ġa Ġpremise Ġand Ġa Ġhypothesis Ġsentence : Ġentail ment , Ġcontradiction , Ġand Ġneutral . ĠSN LI Ġand ĠMulti N LI Ġdatasets Ġare Ġcomposed Ġof Ġabout Ġ570 k Ġand Ġ430 k Ġpremise - hyp ot hesis Ġpairs Ġrespectively . </s> G lo Ve Ġpret rained Ġword Ġembed d ings ĠB IB REF 49 Ġare Ġused Ġand Ġremain Ġfixed Ġduring Ġtraining . ĠThe Ġdimension Ġof Ġenc oder Ġstates Ġ( ĠIN LINE FORM 0 Ġ) Ġis Ġset Ġto Ġ300 Ġand Ġa Ġ1024 D ĠML P Ġwith Ġone Ġor Ġtwo Ġhidden Ġlayers Ġis Ġused . ĠWe Ġapply Ġdrop out ĠB IB REF 50 Ġto Ġthe Ġword Ġembed d ings Ġand Ġthe ĠML P Ġlayers . ĠThe Ġfeatures Ġused Ġas Ġinput Ġto Ġthe ĠML P Ġclass ifier Ġare Ġextracted Ġfollowing ĠE q . ĠEQ REF 28 Ġ. </s> Table ĠT AB REF 32 Ġand ĠT AB REF 33 Ġcontain Ġresults Ġof Ġthe Ġmodels Ġon ĠSN LI Ġand ĠMulti N LI Ġdatasets . ĠIn ĠSN LI , Ġour Ġbest Ġmodel Ġachieves Ġthe Ġnew Ġstate - of - the - art Ġaccuracy Ġof Ġ87 . 0 % Ġwith Ġrelatively Ġfewer Ġparameters . ĠSimilarly Ġin ĠMulti N LI , Ġour Ġmodels Ġmatch Ġthe Ġaccuracy Ġof Ġstate - of - the - art Ġmodels Ġin Ġboth Ġin - domain Ġ( matched ) Ġand Ġcross - domain Ġ( m ism atched ) Ġtest Ġsets . ĠNote Ġthat Ġonly Ġthe ĠGl o Ve Ġword Ġvectors Ġare Ġused Ġas Ġword Ġrepresentations , Ġas Ġopposed Ġto Ġsome Ġmodels Ġthat Ġintroduce Ġcharacter - level Ġfeatures . ĠIt Ġis Ġalso Ġnotable Ġthat Ġour Ġproposed Ġarchitecture Ġdoes Ġnot Ġrestrict Ġthe Ġselection Ġof Ġpool ing Ġmethod ; Ġthe Ġperformance Ġcould Ġfurther Ġbe Ġimproved Ġby Ġreplacing Ġmax - pool ing Ġwith Ġother Ġadvanced Ġalgorithms Ġe . g . Ġintra - sent ence Ġattention ĠB IB REF 39 Ġand Ġgeneralized Ġpool ing ĠB IB REF 19 Ġ. </s> Par aph rase ĠIdentification </s> We Ġuse ĠQu ora ĠQuestion ĠP airs Ġdataset ĠB IB REF 24 Ġin Ġevaluating Ġthe Ġperformance Ġof Ġour Ġmethod Ġon Ġthe ĠPI Ġtask . ĠThe Ġdataset Ġconsists Ġof Ġover Ġ400 k Ġquestion Ġpairs , Ġand Ġeach Ġpair Ġis Ġannot ated Ġwith Ġwhether Ġthe Ġtwo Ġsentences Ġare Ġparaph rase Ġof Ġeach Ġother Ġor Ġnot . </s> Similar Ġto Ġthe ĠNL I Ġexperiments , ĠGl o Ve Ġpret rained Ġvectors , Ġ300 D Ġenc od ers , Ġand Ġ1024 D ĠML P Ġare Ġused . ĠThe Ġnumber Ġof ĠCAS - L ST M Ġlayers Ġis Ġfixed Ġto Ġ2 Ġin ĠPI Ġexperiments . ĠTwo Ġsentence Ġvectors Ġare Ġaggreg ated Ġusing ĠE q . ĠEQ REF 29 Ġand Ġfed Ġas Ġinput Ġto Ġthe ĠML P . ĠThe Ġresults Ġon Ġthe ĠQu ora ĠQuestion ĠP airs Ġdataset Ġare Ġsummarized Ġin ĠTable ĠT AB REF 34 Ġ. ĠAgain Ġwe Ġcan Ġsee Ġthat Ġour Ġmodels Ġoutper form Ġother Ġmodels Ġby Ġlarge Ġmargin , Ġachieving Ġthe Ġnew Ġstate Ġof Ġthe Ġart . </s> Sent iment ĠClassification </s> In Ġevaluating Ġsentiment Ġclassification Ġperformance , Ġthe ĠStanford ĠSent iment ĠTree bank Ġ( S ST ) ĠB IB REF 25 Ġis Ġused . ĠIt Ġconsists Ġof Ġabout Ġ12 , 000 Ġbinary - p ars ed Ġsentences Ġwhere Ġconstituents Ġ( ph r ases ) Ġof Ġeach Ġparse Ġtree Ġare Ġannot ated Ġwith Ġa Ġsentiment Ġlabel Ġ( very Ġpositive , Ġpositive , Ġneutral , Ġnegative , Ġvery Ġnegative ). ĠFollowing Ġthe Ġconvention Ġof Ġprior Ġwork , Ġall Ġphrases Ġand Ġtheir Ġlabels Ġare Ġused Ġin Ġtraining Ġbut Ġonly Ġthe Ġsentence - level Ġdata Ġare Ġused Ġin Ġevaluation . </s> In Ġevaluation Ġwe Ġconsider Ġtwo Ġsettings , Ġnamely ĠS ST - 2 Ġand ĠS ST - 5 , Ġthe Ġtwo Ġdiffering Ġonly Ġin Ġtheir Ġlevel Ġof Ġgran ularity Ġwith Ġregard Ġto Ġlabels . ĠIn ĠS ST - 2 , Ġdata Ġsamples Ġannot ated Ġwith Ġ` neutral ' Ġare Ġignored Ġfrom Ġtraining Ġand Ġevaluation . ĠThe Ġtwo Ġpositive Ġlabels Ġ( very Ġpositive , Ġpositive ) Ġare Ġconsidered Ġas Ġthe Ġsame Ġlabel , Ġand Ġsimilarly Ġfor Ġthe Ġtwo Ġnegative Ġlabels . ĠAs Ġa Ġresult Ġ98 , 794 / 8 72 / 1 , 8 21 Ġdata Ġsamples Ġare Ġused Ġin Ġtraining / valid ation / test , Ġand Ġthe Ġtask Ġis Ġconsidered Ġas Ġa Ġbinary Ġclassification Ġproblem . ĠIn ĠS ST - 5 , Ġdata Ġare Ġused Ġas - is Ġand Ġthus Ġthe Ġtask Ġis Ġa Ġ5 - class Ġclassification Ġproblem . ĠAll Ġ318 , 582 / 1 , 101 / 2 , 210 Ġdata Ġsamples Ġfor Ġtraining / valid ation / test Ġare Ġused Ġin Ġthe ĠS ST - 5 Ġsetting . </s> We Ġuse Ġ300 D ĠGl o Ve Ġvectors , Ġ2 - layer Ġ150 D Ġor Ġ300 D Ġenc od ers , Ġand Ġa Ġ300 D ĠML P Ġclass ifier Ġfor Ġthe Ġmodels , Ġhowever Ġunlike Ġprevious Ġexperiments Ġwe Ġtune Ġthe Ġword Ġembed d ings Ġduring Ġtraining . ĠThe Ġresults Ġon ĠS ST Ġare Ġlisted Ġin ĠTable ĠT AB REF 35 Ġ. ĠOur Ġmodels Ġachieve Ġthe Ġnew Ġstate - of - the - art Ġaccuracy Ġon ĠS ST - 2 Ġand Ġcompetitive Ġaccuracy Ġon ĠS ST - 5 , Ġwithout Ġutilizing Ġparse Ġtree Ġinformation . </s> For get ĠGate ĠAnalysis </s> To Ġinspect Ġthe Ġeffect Ġof Ġthe Ġadditional Ġforget Ġgate , Ġwe Ġinvestigate Ġhow Ġthe Ġvalues Ġof Ġvertical Ġforget Ġgates Ġare Ġdistributed . ĠWe Ġsample Ġ1 , 000 Ġrandom Ġsentences Ġfrom Ġthe Ġdevelopment Ġset Ġof Ġthe ĠSN LI Ġdataset , Ġand Ġuse Ġthe Ġ3 - layer ĠCAS - L ST M Ġmodel Ġtrained Ġon Ġthe ĠSN LI Ġdataset Ġto Ġcompute Ġgate Ġvalues . </s> If Ġall Ġvalues Ġfrom Ġa Ġvertical Ġforget Ġgate ĠIN LINE FORM 0 Ġwere Ġto Ġbe Ġ0 , Ġthis Ġwould Ġmean Ġthat Ġthe Ġintroduction Ġof Ġthe Ġadditional Ġforget Ġgate Ġis Ġmeaningless Ġand Ġthe Ġmodel Ġwould Ġreduce Ġto Ġa Ġplain Ġstacked ĠL ST M . ĠOn Ġthe Ġcontrary Ġif Ġall Ġvalues Ġwere Ġ1 , Ġmeaning Ġthat Ġthe Ġvertical Ġforget Ġgates Ġwere Ġalways Ġopen , Ġit Ġwould Ġbe Ġimpossible Ġto Ġsay Ġthat Ġthe Ġinformation Ġis Ġmod ulated Ġeffectively . </s> Fig . ĠFIG REF 40 Ġand ĠFIG REF 40 Ġrepresent Ġhist ograms Ġof Ġthe Ġvertical Ġforget Ġgate Ġvalues Ġfrom Ġthe Ġsecond Ġand Ġthe Ġthird Ġlayer . ĠFrom Ġthe Ġfigures Ġwe Ġcan Ġvalidate Ġthat Ġthe Ġtrained Ġmodel Ġdoes Ġnot Ġfall Ġinto Ġthe Ġdegener ate Ġcase Ġwhere Ġvertical Ġforget Ġgates Ġare Ġignored . ĠAlso Ġthe Ġfigures Ġshow Ġthat Ġthe Ġvalues Ġare Ġright - s ke wed , Ġwhich Ġwe Ġconjecture Ġto Ġbe Ġa Ġresult Ġof Ġfocusing Ġmore Ġon Ġa Ġstrong Ġinteraction Ġbetween Ġadjacent Ġlayers . </s> To Ġfurther Ġverify Ġthat Ġthe Ġgate Ġvalues Ġare Ġdiverse Ġenough Ġwithin Ġeach Ġtime Ġstep , Ġwe Ġcompute Ġthe Ġdistribution Ġof Ġthe Ġrange Ġof Ġvalues Ġper Ġtime Ġstep , ĠIN LINE FORM 0 Ġ, Ġwhere ĠIN LINE FORM 1 Ġ. ĠWe Ġplot Ġthe Ġhist ograms Ġin ĠFig . ĠFIG REF 40 Ġand ĠFIG REF 40 Ġ. ĠFrom Ġthe Ġfigure Ġwe Ġsee Ġthat Ġa Ġvertical Ġforget Ġgate Ġcontrols Ġthe Ġamount Ġof Ġinformation Ġflow Ġeffectively , Ġmaking Ġthe Ġdecision Ġof Ġretaining Ġor Ġdisc arding Ġsignals . </s> Finally , Ġto Ġinvestigate Ġthe Ġargument Ġpresented Ġin ĠÂ§ ĠSEC REF 2 Ġthat Ġthe Ġadditional Ġforget Ġgate Ġhelps Ġthe Ġprevious Ġoutput Ġgate Ġwith Ġreducing Ġthe Ġburden Ġof Ġextracting Ġall Ġneeded Ġinformation , Ġwe Ġinspect Ġthe Ġdistribution Ġof Ġthe Ġvalues Ġfrom ĠIN LINE FORM 0 Ġ. ĠThis Ġdistribution Ġindicates Ġhow Ġdifferently Ġthe Ġvertical Ġforget Ġgate Ġand Ġthe Ġprevious Ġoutput Ġgate Ġselect Ġinformation Ġfrom ĠIN LINE FORM 1 Ġ. ĠFrom ĠFig . ĠFIG REF 40 Ġand ĠFIG REF 40 Ġwe Ġcan Ġsee Ġthat Ġthe Ġtwo Ġgates Ġmake Ġfairly Ġdifferent Ġdecisions , Ġfrom Ġwhich Ġwe Ġdemonstrate Ġthat Ġthe Ġdirect Ġpath Ġbetween ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġenables Ġa Ġmodel Ġto Ġutilize Ġsignals Ġoverlooked Ġby ĠIN LINE FORM 4 Ġ. </s> Model ĠVari ations </s> In Ġthis Ġsubsection , Ġwe Ġsee Ġthe Ġinfluence Ġof Ġeach Ġcomponent Ġof Ġa Ġmodel Ġon Ġperformance Ġby Ġremoving Ġor Ġreplacing Ġits Ġcomponents . Ġthe ĠSN LI Ġdataset Ġis Ġused Ġfor Ġexperiments , Ġand Ġthe Ġbest Ġperforming Ġconfiguration Ġis Ġused Ġas Ġa Ġbaseline Ġfor Ġmodifications . ĠWe Ġconsider Ġthe Ġfollowing Ġvariants : Ġ( i ) Ġmodels Ġthat Ġuse Ġplain Ġstacked ĠL ST Ms , Ġ( ii ) Ġmodels Ġwith Ġdifferent ĠIN LINE FORM 0 Ġ, Ġ( iii ) Ġmodels Ġwithout ĠIN LINE FORM 1 Ġ, Ġand Ġ( iv ) Ġmodels Ġthat Ġintegrate Ġlower Ġcontexts Ġvia Ġpe ep hole Ġconnections . </s> Vari ant Ġ( iv ) Ġintegrates Ġlower Ġcontexts Ġvia Ġthe Ġfollowing Ġequations : ĠDIS PLAY FORM 0 ĠDIS PLAY FORM 1 </s> where ĠIN LINE FORM 0 Ġrepresent Ġpe ep hole Ġweights Ġthat Ġtake Ġcell Ġstates Ġinto Ġaccount . ĠAmong Ġthe Ġabove Ġequations , Ġthose Ġthat Ġuse Ġthe Ġlower Ġcell Ġstate ĠIN LINE FORM 1 Ġare ĠE q . ĠEQ REF 52 Ġand ĠEQ REF 55 Ġ. ĠWe Ġcan Ġsee Ġthat ĠIN LINE FORM 2 Ġaffects Ġthe Ġvalue Ġof ĠIN LINE FORM 3 Ġonly Ġvia Ġpe ep hole Ġconnections , Ġwhich Ġmakes ĠIN LINE FORM 4 Ġindependent Ġof ĠIN LINE FORM 5 Ġ. </s> Table ĠT AB REF 36 Ġsummarizes Ġthe Ġresults Ġof Ġmodel Ġvariants . ĠWe Ġcan Ġagain Ġsee Ġthat Ġthe Ġuse Ġof Ġcell Ġstates Ġclearly Ġimproves Ġsentence Ġmodeling Ġperformance Ġ( bas eline Ġvs . Ġ( i ) Ġand Ġ( iv ) Ġvs . Ġ( i )). ĠAlso Ġfrom Ġthe Ġresults Ġof Ġbaseline Ġand Ġ( ii ), Ġwe Ġvalidate Ġthat Ġthe Ġselection Ġof ĠIN LINE FORM 0 Ġdoes Ġnot Ġsignificantly Ġaffect Ġperformance Ġbut Ġintroducing ĠIN LINE FORM 1 Ġis Ġbeneficial Ġ( bas eline Ġvs . Ġ( iii )) Ġpossibly Ġdue Ġto Ġits Ġeffect Ġon Ġnormal izing Ġinformation Ġfrom Ġmultiple Ġsources , Ġas Ġmentioned Ġin ĠÂ§ ĠSEC REF 2 Ġ. ĠFinally , Ġfrom Ġthe Ġcomparison Ġbetween Ġbaseline Ġand Ġ( iv ), Ġwe Ġshow Ġthat Ġthe Ġproposed Ġway Ġof Ġcombining Ġthe Ġleft Ġand Ġthe Ġlower Ġcontexts Ġleads Ġto Ġbetter Ġmodeling Ġof Ġsentence Ġrepresentations Ġthan Ġthat Ġof ĠB IB REF 20 Ġin Ġencoding Ġsentences . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġproposed Ġa Ġmethod Ġof Ġstacking Ġmultiple ĠL ST M Ġlayers Ġfor Ġmodeling Ġsentences , Ġdubbed ĠCAS - L ST M . ĠIt Ġuses Ġnot Ġonly Ġhidden Ġstates Ġbut Ġalso Ġcell Ġstates Ġfrom Ġthe Ġprevious Ġlayer , Ġfor Ġthe Ġpurpose Ġof Ġcontrolling Ġthe Ġvertical Ġinformation Ġflow Ġin Ġa Ġmore Ġelaborate Ġway . ĠWe Ġevaluated Ġthe Ġproposed Ġmethod Ġon Ġvarious Ġbenchmark Ġtasks : Ġnatural Ġlanguage Ġinference , Ġparaph rase Ġidentification , Ġand Ġsentiment Ġclassification . ĠOur Ġmodels Ġachieve Ġthe Ġnew Ġstate - of - the - art Ġaccuracy Ġon ĠSN LI Ġand ĠQu ora ĠQuestion ĠP airs Ġdatasets Ġand Ġobtain Ġcomparable Ġresults Ġon ĠMulti N LI Ġand ĠS ST Ġdatasets . ĠThe Ġproposed Ġarchitecture Ġcan Ġreplace Ġany Ġstacked ĠL ST M Ġunder Ġone Ġweak Ġrestriction âĢĶ the Ġsize Ġof Ġstates Ġshould Ġbe Ġidentical Ġacross Ġall Ġlayers . </s> For Ġfuture Ġwork Ġwe Ġplan Ġto Ġapply Ġthe ĠCAS - L ST M Ġarchitecture Ġbeyond Ġsentence Ġmodeling Ġtasks . ĠVarious Ġproblems Ġe . g . Ġsequence Ġlabeling , Ġsequence Ġgeneration , Ġand Ġlanguage Ġmodeling Ġmight Ġbenefit Ġfrom Ġsophisticated Ġmodulation Ġon Ġcontext Ġintegration . ĠAgg reg ating Ġdiverse Ġcontexts Ġfrom Ġsequential Ġdata , Ġe . g . Ġthose Ġfrom Ġforward Ġand Ġbackward Ġreading Ġof Ġtext , Ġcould Ġalso Ġbe Ġan Ġintriguing Ġresearch Ġdirection . </s> Acknowled gments </s> We Ġthank ĠDan ĠEd m iston Ġfor Ġthe Ġreview Ġof Ġthe Ġmanuscript .
Introduction In the field of natural language processing (NLP), the most prevalent neural approach to obtaining sentence representations is to use recurrent neural networks (RNNs), where words in a sentence are processed in a sequential and recurrent manner. Along with their intuitive design, RNNs have shown outstanding performance across various NLP tasks e.g. language modeling BIBREF0 , BIBREF1 , machine translation BIBREF2 , BIBREF3 , BIBREF4 , text classification BIBREF5 , BIBREF6 , and parsing BIBREF7 , BIBREF8 . Among several variants of the original RNN BIBREF9 , gated recurrent architectures such as long short-term memory (LSTM) BIBREF10 and gated recurrent unit (GRU) BIBREF2 have been accepted as de-facto standard choices for RNNs due to their capability of addressing the vanishing and exploding gradient problem and considering long-term dependencies. Gated RNNs achieve these properties by introducing additional gating units that learn to control the amount of information to be transferred or forgotten BIBREF11 , and are proven to work well without relying on complex optimization algorithms or careful initialization BIBREF12 . Meanwhile, the common practice for further enhancing the expressiveness of RNNs is to stack multiple RNN layers, each of which has distinct parameter sets (stacked RNN) BIBREF13 , BIBREF14 . In stacked RNNs, the hidden states of a layer are fed as input to the subsequent layer, and they are shown to work well due to increased depth BIBREF15 or their ability to capture hierarchical time series BIBREF16 which are inherent to the nature of the problem being modeled. However this setting of stacking RNNs might hinder the possibility of more sophisticated recurrence-based structures since the information from lower layers is simply treated as input to the next layer, rather than as another class of state that participates in core RNN computations. Especially for gated RNNs such as LSTMs and GRUs, this means that layer-to-layer connections cannot fully benefit from the carefully constructed gating mechanism used in temporal transitions. Some recent work on stacking RNNs suggests alternative methods that encourage direct and effective interaction between RNN layers by adding residual connections BIBREF17 , BIBREF18 , by shortcut connections BIBREF18 , BIBREF19 , or by using cell states of LSTMs BIBREF20 , BIBREF21 . In this paper, we propose a method of constructing multi-layer LSTMs where cell states are used in controlling the vertical information flow. This system utilizes states from the left and the lower context equally in computation of the new state, thus the information from lower layers is elaborately filtered and reflected through a soft gating mechanism. Our method is easy-to-implement, effective, and can replace conventional stacked LSTMs without much modification of the overall architecture. We call the proposed architecture Cell-aware Stacked LSTM, or CAS-LSTM, and evaluate our method on multiple benchmark datasets: SNLI BIBREF22 , MultiNLI BIBREF23 , Quora Question Pairs BIBREF24 , and SST BIBREF25 . From experiments we show that the CAS-LSTMs consistently outperform typical stacked LSTMs, opening the possibility of performance improvement of architectures that use stacked LSTMs. Our contribution is summarized as follows. This paper is organized as follows. We give a detailed description about the proposed method in § SECREF2 . Experimental results are given in § SECREF3 . We study prior work related to our objective in § SECREF4 and conclude in § SECREF5 . Model Description In this section, we give a detailed formulation of the architectures used in experiments. Notation Throughout this paper, we denote matrices as boldface capital letters ( INLINEFORM0 ), vectors as boldface lowercase letters ( INLINEFORM1 ), and scalars as normal italic letters ( INLINEFORM2 ). For LSTM states, we denote a hidden state as INLINEFORM3 and a cell state as INLINEFORM4 . Also, a layer index of INLINEFORM5 or INLINEFORM6 is denoted by superscript and a time index is denoted by a subscript, i.e. INLINEFORM7 indicates the hidden state at time INLINEFORM8 and layer INLINEFORM9 . INLINEFORM10 means the element-wise multiplication between two vectors. We write INLINEFORM11 -th component of vector INLINEFORM12 as INLINEFORM13 . All vectors are assumed to be column vectors. Stacked LSTMs While there exist various versions of LSTM formulation, in this work we use the following, one of the most common versions: DISPLAYFORM0 DISPLAYFORM1 where INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 , INLINEFORM3 , INLINEFORM4 are trainable parameters. INLINEFORM5 and INLINEFORM6 are the sigmoid activation and the hyperbolic tangent activation function respectively. Also we assume that INLINEFORM7 where INLINEFORM8 is the INLINEFORM9 -th input to the network. The input gate INLINEFORM0 and the forget gate INLINEFORM1 control the amount of information transmitted from INLINEFORM2 and INLINEFORM3 , the candidate cell state and the previous cell state, to the new cell state INLINEFORM4 . Similarly the output gate INLINEFORM5 soft-selects which portion of the cell state INLINEFORM6 is to be used in the final hidden state. We can clearly see that cell states ( INLINEFORM0 , INLINEFORM1 , INLINEFORM2 ) play a crucial role in forming horizontal recurrence. However the current formulation does not consider INLINEFORM3 , the cell state from INLINEFORM4 -th layer, in computation and thus the lower context is reflected only through the rudimentary way, hindering the possibility of controlling vertical information flow. Cell-aware Stacked LSTMs Now we extend the stacked LSTM formulation defined above to address the problem noted in the previous subsection. To enhance the interaction between layers in a way similar to how LSTMs keep and forget the information from the previous time step, we introduce the additional forget gate INLINEFORM0 that determines whether to accept or ignore the signals coming from the previous layer. Therefore the proposed Cell-aware Stacked LSTM is formulated as follows: DISPLAYFORM0 DISPLAYFORM1 where INLINEFORM0 and INLINEFORM1 . INLINEFORM2 can either be a vector of constants or parameters. When INLINEFORM3 , the equations defined in the previous subsection are used. Therefore, it can be said that each non-bottom layer of CAS-LSTM accepts two sets of hidden and cell states—one from the left context and the other from the below context. The left and the below context participate in computation with the equivalent procedure so that the information from lower layers can be efficiently propagated. Fig. FIGREF1 compares CAS-LSTM to the conventional stacked LSTM architecture, and Fig. FIGREF8 depicts the computation flow of the CAS-LSTM. We argue that considering INLINEFORM0 in computation is beneficial for the following reasons. First, INLINEFORM1 contains additional information compared to INLINEFORM2 since it is not filtered by INLINEFORM3 . Thus a model that directly uses INLINEFORM4 does not rely solely on INLINEFORM5 for extracting information, due to the fact that it has access to the raw information INLINEFORM6 , as in temporal connections. In other words, INLINEFORM7 no longer has to take all responsibility for selecting useful features for both horizontal and vertical transitions, and the burden of selecting information is shared with INLINEFORM8 . Another advantage of using the INLINEFORM0 lies in the fact that it directly connects INLINEFORM1 and INLINEFORM2 . This direct connection helps and stabilizes training, since the terminal error signals can be easily backpropagated to model parameters. Fig. FIGREF23 illustrates paths between the two cell states. We find experimentally that there is little difference between letting INLINEFORM0 be constant and letting it be trainable parameters, thus we set INLINEFORM1 in all experiments. We also experimented with the architecture without INLINEFORM2 i.e. two cell states are combined by unweighted summation similar to multidimensional RNNs BIBREF27 , and found that it leads to performance degradation and unstable convergence, likely due to mismatch in the range of cell state values between layers ( INLINEFORM3 for the first layer and INLINEFORM4 for the others). Experimental results on various INLINEFORM5 are presented in § SECREF3 . The idea of having multiple states is also related to tree-structured RNNs BIBREF29 , BIBREF30 . Among them, tree-structured LSTMs (Tree-LSTMs) BIBREF31 , BIBREF32 , BIBREF33 are similar to ours in that they use both hidden and cell states from children nodes. In Tree-LSTMs, states for all children nodes are regarded as input, and they participate in the computation equally through weight-shared (in Child-Sum Tree-LSTMs) or weight-unshared (in INLINEFORM0 -ary Tree-LSTMs) projection. From this perspective, each CAS-LSTM layer (where INLINEFORM1 ) can be seen as a binary Tree-LSTM where the structures it operates on are fixed to right-branching trees. The use of cell state in computation could be one reason that Tree-LSTMs perform better than sequential LSTMs even when trivial trees (strictly left- or right-branching) are given BIBREF34 . Multidimensional RNNs (MDRNN) are an extension of 1D sequential RNNs that can accept multidimensional input e.g. images, and have been successfully applied to image segmentation BIBREF26 and handwriting recognition BIBREF27 . Notably multidimensional LSTMs (MDLSTM) BIBREF27 have an analogous formulation to ours except the INLINEFORM0 term and the fact that we use distinct weights per column (or `layer' in our case). From this view, CAS-LSTM can be seen as a certain kind of MDLSTM that accepts a 2D input INLINEFORM1 . Grid LSTMs BIBREF21 also take INLINEFORM2 inputs but emit INLINEFORM3 outputs, which is different from our case where a single set of hidden and cell states is produced. Sentence Encoders The sentence encoder network we use in our experiments takes INLINEFORM0 words (assumed to be one-hot vectors) as input. The words are projected to corresponding word representations: INLINEFORM1 where INLINEFORM2 . Then INLINEFORM3 is fed to a INLINEFORM4 -layer CAS-LSTM model, resulting in the representations INLINEFORM5 . The sentence representation, INLINEFORM6 , is computed by max-pooling INLINEFORM7 over time as in the work of BIBREF35 . Similar to their results, from preliminary experiments we found that the max-pooling performs consistently better than mean- and last-pooling. To make models more expressive, a bidirectional CAS-LSTM network may also be used. In the bidirectional case, the forward representations INLINEFORM0 and the backward representations INLINEFORM1 are concatenated and max-pooled to yield the sentence representation INLINEFORM2 . We call this bidirectional architecture Bi-CAS-LSTM in experiments. Top-layer Classifiers For the natural language inference experiments, we use the following heuristic function proposed by BIBREF36 in feature extraction: DISPLAYFORM0 where INLINEFORM0 means vector concatenation, and INLINEFORM1 and INLINEFORM2 are applied element-wise. And we use the following function in paraphrase identification experiments: DISPLAYFORM0 as in the work of BIBREF37 . For sentiment classification, we use the sentence representation itself. DISPLAYFORM0 We feed the feature extracted from INLINEFORM0 as input to the MLP classifier with ReLU activation followed by the fully-connected softmax layer to predict the label distribution: DISPLAYFORM0 where INLINEFORM0 , INLINEFORM1 is the number of label classes, and INLINEFORM2 the dimension of the MLP output, Experiments We evaluate our method on natural language inference (NLI), paraphrase identification (PI), and sentiment classification. We also conduct analysis on gate values and experiments on model variants. For detailed experimental settings, we refer readers to the supplemental material. For the NLI and PI tasks, there exists recent work specializing in sentence pair classification. However in this work we confine our model to the architecture that encodes each sentence using a shared encoder without any inter-sentence interaction, in order to focus on the effectiveness of the models in extracting semantics. But note that the applicability of CAS-LSTM is not limited to sentence encoding based approaches. Natural Language Inference For the evaluation of performance of the proposed method on the NLI task, SNLI BIBREF22 and MultiNLI BIBREF23 datasets are used. The objective of both datasets is to predict the relationship between a premise and a hypothesis sentence: entailment, contradiction, and neutral. SNLI and MultiNLI datasets are composed of about 570k and 430k premise-hypothesis pairs respectively. GloVe pretrained word embeddings BIBREF49 are used and remain fixed during training. The dimension of encoder states ( INLINEFORM0 ) is set to 300 and a 1024D MLP with one or two hidden layers is used. We apply dropout BIBREF50 to the word embeddings and the MLP layers. The features used as input to the MLP classifier are extracted following Eq. EQREF28 . Table TABREF32 and TABREF33 contain results of the models on SNLI and MultiNLI datasets. In SNLI, our best model achieves the new state-of-the-art accuracy of 87.0% with relatively fewer parameters. Similarly in MultiNLI, our models match the accuracy of state-of-the-art models in both in-domain (matched) and cross-domain (mismatched) test sets. Note that only the GloVe word vectors are used as word representations, as opposed to some models that introduce character-level features. It is also notable that our proposed architecture does not restrict the selection of pooling method; the performance could further be improved by replacing max-pooling with other advanced algorithms e.g. intra-sentence attention BIBREF39 and generalized pooling BIBREF19 . Paraphrase Identification We use Quora Question Pairs dataset BIBREF24 in evaluating the performance of our method on the PI task. The dataset consists of over 400k question pairs, and each pair is annotated with whether the two sentences are paraphrase of each other or not. Similar to the NLI experiments, GloVe pretrained vectors, 300D encoders, and 1024D MLP are used. The number of CAS-LSTM layers is fixed to 2 in PI experiments. Two sentence vectors are aggregated using Eq. EQREF29 and fed as input to the MLP. The results on the Quora Question Pairs dataset are summarized in Table TABREF34 . Again we can see that our models outperform other models by large margin, achieving the new state of the art. Sentiment Classification In evaluating sentiment classification performance, the Stanford Sentiment Treebank (SST) BIBREF25 is used. It consists of about 12,000 binary-parsed sentences where constituents (phrases) of each parse tree are annotated with a sentiment label (very positive, positive, neutral, negative, very negative). Following the convention of prior work, all phrases and their labels are used in training but only the sentence-level data are used in evaluation. In evaluation we consider two settings, namely SST-2 and SST-5, the two differing only in their level of granularity with regard to labels. In SST-2, data samples annotated with `neutral' are ignored from training and evaluation. The two positive labels (very positive, positive) are considered as the same label, and similarly for the two negative labels. As a result 98,794/872/1,821 data samples are used in training/validation/test, and the task is considered as a binary classification problem. In SST-5, data are used as-is and thus the task is a 5-class classification problem. All 318,582/1,101/2,210 data samples for training/validation/test are used in the SST-5 setting. We use 300D GloVe vectors, 2-layer 150D or 300D encoders, and a 300D MLP classifier for the models, however unlike previous experiments we tune the word embeddings during training. The results on SST are listed in Table TABREF35 . Our models achieve the new state-of-the-art accuracy on SST-2 and competitive accuracy on SST-5, without utilizing parse tree information. Forget Gate Analysis To inspect the effect of the additional forget gate, we investigate how the values of vertical forget gates are distributed. We sample 1,000 random sentences from the development set of the SNLI dataset, and use the 3-layer CAS-LSTM model trained on the SNLI dataset to compute gate values. If all values from a vertical forget gate INLINEFORM0 were to be 0, this would mean that the introduction of the additional forget gate is meaningless and the model would reduce to a plain stacked LSTM. On the contrary if all values were 1, meaning that the vertical forget gates were always open, it would be impossible to say that the information is modulated effectively. Fig. FIGREF40 and FIGREF40 represent histograms of the vertical forget gate values from the second and the third layer. From the figures we can validate that the trained model does not fall into the degenerate case where vertical forget gates are ignored. Also the figures show that the values are right-skewed, which we conjecture to be a result of focusing more on a strong interaction between adjacent layers. To further verify that the gate values are diverse enough within each time step, we compute the distribution of the range of values per time step, INLINEFORM0 , where INLINEFORM1 . We plot the histograms in Fig. FIGREF40 and FIGREF40 . From the figure we see that a vertical forget gate controls the amount of information flow effectively, making the decision of retaining or discarding signals. Finally, to investigate the argument presented in § SECREF2 that the additional forget gate helps the previous output gate with reducing the burden of extracting all needed information, we inspect the distribution of the values from INLINEFORM0 . This distribution indicates how differently the vertical forget gate and the previous output gate select information from INLINEFORM1 . From Fig. FIGREF40 and FIGREF40 we can see that the two gates make fairly different decisions, from which we demonstrate that the direct path between INLINEFORM2 and INLINEFORM3 enables a model to utilize signals overlooked by INLINEFORM4 . Model Variations In this subsection, we see the influence of each component of a model on performance by removing or replacing its components. the SNLI dataset is used for experiments, and the best performing configuration is used as a baseline for modifications. We consider the following variants: (i) models that use plain stacked LSTMs, (ii) models with different INLINEFORM0 , (iii) models without INLINEFORM1 , and (iv) models that integrate lower contexts via peephole connections. Variant (iv) integrates lower contexts via the following equations: DISPLAYFORM0 DISPLAYFORM1 where INLINEFORM0 represent peephole weights that take cell states into account. Among the above equations, those that use the lower cell state INLINEFORM1 are Eq. EQREF52 and EQREF55 . We can see that INLINEFORM2 affects the value of INLINEFORM3 only via peephole connections, which makes INLINEFORM4 independent of INLINEFORM5 . Table TABREF36 summarizes the results of model variants. We can again see that the use of cell states clearly improves sentence modeling performance (baseline vs. (i) and (iv) vs. (i)). Also from the results of baseline and (ii), we validate that the selection of INLINEFORM0 does not significantly affect performance but introducing INLINEFORM1 is beneficial (baseline vs. (iii)) possibly due to its effect on normalizing information from multiple sources, as mentioned in § SECREF2 . Finally, from the comparison between baseline and (iv), we show that the proposed way of combining the left and the lower contexts leads to better modeling of sentence representations than that of BIBREF20 in encoding sentences. Conclusion In this paper, we proposed a method of stacking multiple LSTM layers for modeling sentences, dubbed CAS-LSTM. It uses not only hidden states but also cell states from the previous layer, for the purpose of controlling the vertical information flow in a more elaborate way. We evaluated the proposed method on various benchmark tasks: natural language inference, paraphrase identification, and sentiment classification. Our models achieve the new state-of-the-art accuracy on SNLI and Quora Question Pairs datasets and obtain comparable results on MultiNLI and SST datasets. The proposed architecture can replace any stacked LSTM under one weak restriction—the size of states should be identical across all layers. For future work we plan to apply the CAS-LSTM architecture beyond sentence modeling tasks. Various problems e.g. sequence labeling, sequence generation, and language modeling might benefit from sophisticated modulation on context integration. Aggregating diverse contexts from sequential data, e.g. those from forward and backward reading of text, could also be an intriguing research direction. Acknowledgments We thank Dan Edmiston for the review of the manuscript.
What were the baselines?
f7d0fa52017a642a9f70091a252857fccca31f12
What Ġwere Ġthe Ġbas elines ?
( i ) Ġmodels Ġthat Ġuse Ġplain Ġstacked ĠL ST Ms , Ġ( ii ) Ġmodels Ġwith Ġdifferent ĠIN LINE FORM 0 , Ġ( iii ) Ġmodels Ġwithout ĠIN LINE FORM 1 , Ġ( iv ) Ġmodels Ġthat Ġintegrate Ġlower Ġcontexts Ġvia Ġpe ep hole Ġconnections
<s> What Ġwere Ġthe Ġbas elines ? </s> Introduction </s> In Ġthe Ġfield Ġof Ġnatural Ġlanguage Ġprocessing Ġ( N LP ), Ġthe Ġmost Ġprevalent Ġneural Ġapproach Ġto Ġobtaining Ġsentence Ġrepresentations Ġis Ġto Ġuse Ġrecurrent Ġneural Ġnetworks Ġ( R NN s ), Ġwhere Ġwords Ġin Ġa Ġsentence Ġare Ġprocessed Ġin Ġa Ġsequential Ġand Ġrecurrent Ġmanner . ĠAlong Ġwith Ġtheir Ġintuitive Ġdesign , ĠR NN s Ġhave Ġshown Ġoutstanding Ġperformance Ġacross Ġvarious ĠN LP Ġtasks Ġe . g . Ġlanguage Ġmodeling ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, Ġmachine Ġtranslation ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ, Ġtext Ġclassification ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, Ġand Ġparsing ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ. </s> Among Ġseveral Ġvariants Ġof Ġthe Ġoriginal ĠR NN ĠB IB REF 9 Ġ, Ġg ated Ġrecurrent Ġarchitectures Ġsuch Ġas Ġlong Ġshort - term Ġmemory Ġ( L ST M ) ĠB IB REF 10 Ġand Ġg ated Ġrecurrent Ġunit Ġ( GR U ) ĠB IB REF 2 Ġhave Ġbeen Ġaccepted Ġas Ġde - fact o Ġstandard Ġchoices Ġfor ĠR NN s Ġdue Ġto Ġtheir Ġcapability Ġof Ġaddressing Ġthe Ġvanishing Ġand Ġexploding Ġgradient Ġproblem Ġand Ġconsidering Ġlong - term Ġdependencies . ĠG ated ĠR NN s Ġachieve Ġthese Ġproperties Ġby Ġintroducing Ġadditional Ġg ating Ġunits Ġthat Ġlearn Ġto Ġcontrol Ġthe Ġamount Ġof Ġinformation Ġto Ġbe Ġtransferred Ġor Ġforgotten ĠB IB REF 11 Ġ, Ġand Ġare Ġproven Ġto Ġwork Ġwell Ġwithout Ġrelying Ġon Ġcomplex Ġoptimization Ġalgorithms Ġor Ġcareful Ġinitialization ĠB IB REF 12 Ġ. </s> Meanwhile , Ġthe Ġcommon Ġpractice Ġfor Ġfurther Ġenhancing Ġthe Ġexpress iveness Ġof ĠR NN s Ġis Ġto Ġstack Ġmultiple ĠR NN Ġlayers , Ġeach Ġof Ġwhich Ġhas Ġdistinct Ġparameter Ġsets Ġ( st acked ĠR NN ) ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ. ĠIn Ġstacked ĠR NN s , Ġthe Ġhidden Ġstates Ġof Ġa Ġlayer Ġare Ġfed Ġas Ġinput Ġto Ġthe Ġsubsequent Ġlayer , Ġand Ġthey Ġare Ġshown Ġto Ġwork Ġwell Ġdue Ġto Ġincreased Ġdepth ĠB IB REF 15 Ġor Ġtheir Ġability Ġto Ġcapture Ġhierarchical Ġtime Ġseries ĠB IB REF 16 Ġwhich Ġare Ġinherent Ġto Ġthe Ġnature Ġof Ġthe Ġproblem Ġbeing Ġmodeled . </s> However Ġthis Ġsetting Ġof Ġstacking ĠR NN s Ġmight Ġhinder Ġthe Ġpossibility Ġof Ġmore Ġsophisticated Ġrec urrence - based Ġstructures Ġsince Ġthe Ġinformation Ġfrom Ġlower Ġlayers Ġis Ġsimply Ġtreated Ġas Ġinput Ġto Ġthe Ġnext Ġlayer , Ġrather Ġthan Ġas Ġanother Ġclass Ġof Ġstate Ġthat Ġparticipates Ġin Ġcore ĠR NN Ġcomput ations . ĠEspecially Ġfor Ġg ated ĠR NN s Ġsuch Ġas ĠL ST Ms Ġand ĠGR Us , Ġthis Ġmeans Ġthat Ġlayer - to - layer Ġconnections Ġcannot Ġfully Ġbenefit Ġfrom Ġthe Ġcarefully Ġconstructed Ġg ating Ġmechanism Ġused Ġin Ġtemporal Ġtransitions . ĠSome Ġrecent Ġwork Ġon Ġstacking ĠR NN s Ġsuggests Ġalternative Ġmethods Ġthat Ġencourage Ġdirect Ġand Ġeffective Ġinteraction Ġbetween ĠR NN Ġlayers Ġby Ġadding Ġresidual Ġconnections ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ, Ġby Ġshortcut Ġconnections ĠB IB REF 18 Ġ, ĠB IB REF 19 Ġ, Ġor Ġby Ġusing Ġcell Ġstates Ġof ĠL ST Ms ĠB IB REF 20 Ġ, ĠB IB REF 21 Ġ. </s> In Ġthis Ġpaper , Ġwe Ġpropose Ġa Ġmethod Ġof Ġconstructing Ġmulti - layer ĠL ST Ms Ġwhere Ġcell Ġstates Ġare Ġused Ġin Ġcontrolling Ġthe Ġvertical Ġinformation Ġflow . ĠThis Ġsystem Ġutilizes Ġstates Ġfrom Ġthe Ġleft Ġand Ġthe Ġlower Ġcontext Ġequally Ġin Ġcomputation Ġof Ġthe Ġnew Ġstate , Ġthus Ġthe Ġinformation Ġfrom Ġlower Ġlayers Ġis Ġelabor ately Ġfiltered Ġand Ġreflected Ġthrough Ġa Ġsoft Ġg ating Ġmechanism . ĠOur Ġmethod Ġis Ġeasy - to - im plement , Ġeffective , Ġand Ġcan Ġreplace Ġconventional Ġstacked ĠL ST Ms Ġwithout Ġmuch Ġmodification Ġof Ġthe Ġoverall Ġarchitecture . </s> We Ġcall Ġthe Ġproposed Ġarchitecture ĠCell - aware ĠSt acked ĠL ST M , Ġor ĠCAS - L ST M , Ġand Ġevaluate Ġour Ġmethod Ġon Ġmultiple Ġbenchmark Ġdatasets : ĠSN LI ĠB IB REF 22 Ġ, ĠMulti N LI ĠB IB REF 23 Ġ, ĠQu ora ĠQuestion ĠP airs ĠB IB REF 24 Ġ, Ġand ĠS ST ĠB IB REF 25 Ġ. ĠFrom Ġexperiments Ġwe Ġshow Ġthat Ġthe ĠCAS - L ST Ms Ġconsistently Ġoutper form Ġtypical Ġstacked ĠL ST Ms , Ġopening Ġthe Ġpossibility Ġof Ġperformance Ġimprovement Ġof Ġarchitectures Ġthat Ġuse Ġstacked ĠL ST Ms . </s> Our Ġcontribution Ġis Ġsummarized Ġas Ġfollows . </s> This Ġpaper Ġis Ġorganized Ġas Ġfollows . ĠWe Ġgive Ġa Ġdetailed Ġdescription Ġabout Ġthe Ġproposed Ġmethod Ġin ĠÂ§ ĠSEC REF 2 Ġ. ĠExperimental Ġresults Ġare Ġgiven Ġin ĠÂ§ ĠSEC REF 3 Ġ. ĠWe Ġstudy Ġprior Ġwork Ġrelated Ġto Ġour Ġobjective Ġin ĠÂ§ ĠSEC REF 4 Ġand Ġconclude Ġin ĠÂ§ ĠSEC REF 5 Ġ. </s> Model ĠDescription </s> In Ġthis Ġsection , Ġwe Ġgive Ġa Ġdetailed Ġformulation Ġof Ġthe Ġarchitectures Ġused Ġin Ġexperiments . </s> Not ation </s> Throughout Ġthis Ġpaper , Ġwe Ġdenote Ġmat rices Ġas Ġbold face Ġcapital Ġletters Ġ( ĠIN LINE FORM 0 Ġ), Ġvectors Ġas Ġbold face Ġlower case Ġletters Ġ( ĠIN LINE FORM 1 Ġ), Ġand Ġscal ars Ġas Ġnormal Ġital ic Ġletters Ġ( ĠIN LINE FORM 2 Ġ). ĠFor ĠL ST M Ġstates , Ġwe Ġdenote Ġa Ġhidden Ġstate Ġas ĠIN LINE FORM 3 Ġand Ġa Ġcell Ġstate Ġas ĠIN LINE FORM 4 Ġ. ĠAlso , Ġa Ġlayer Ġindex Ġof ĠIN LINE FORM 5 Ġor ĠIN LINE FORM 6 Ġis Ġden oted Ġby Ġsupers cript Ġand Ġa Ġtime Ġindex Ġis Ġden oted Ġby Ġa Ġsubscript , Ġi . e . ĠIN LINE FORM 7 Ġindicates Ġthe Ġhidden Ġstate Ġat Ġtime ĠIN LINE FORM 8 Ġand Ġlayer ĠIN LINE FORM 9 Ġ. ĠIN LINE FORM 10 Ġmeans Ġthe Ġelement - wise Ġmultiplication Ġbetween Ġtwo Ġvectors . ĠWe Ġwrite ĠIN LINE FORM 11 Ġ- th Ġcomponent Ġof Ġvector ĠIN LINE FORM 12 Ġas ĠIN LINE FORM 13 Ġ. ĠAll Ġvectors Ġare Ġassumed Ġto Ġbe Ġcolumn Ġvectors . </s> St acked ĠL ST Ms </s> While Ġthere Ġexist Ġvarious Ġversions Ġof ĠL ST M Ġformulation , Ġin Ġthis Ġwork Ġwe Ġuse Ġthe Ġfollowing , Ġone Ġof Ġthe Ġmost Ġcommon Ġversions : ĠDIS PLAY FORM 0 ĠDIS PLAY FORM 1 </s> where ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, Ġand ĠIN LINE FORM 2 Ġ, ĠIN LINE FORM 3 Ġ, ĠIN LINE FORM 4 Ġare Ġtrain able Ġparameters . ĠIN LINE FORM 5 Ġand ĠIN LINE FORM 6 Ġare Ġthe Ġs igm oid Ġactivation Ġand Ġthe Ġhyper b olic Ġtang ent Ġactivation Ġfunction Ġrespectively . ĠAlso Ġwe Ġassume Ġthat ĠIN LINE FORM 7 Ġwhere ĠIN LINE FORM 8 Ġis Ġthe ĠIN LINE FORM 9 Ġ- th Ġinput Ġto Ġthe Ġnetwork . </s> The Ġinput Ġgate ĠIN LINE FORM 0 Ġand Ġthe Ġforget Ġgate ĠIN LINE FORM 1 Ġcontrol Ġthe Ġamount Ġof Ġinformation Ġtransmitted Ġfrom ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġ, Ġthe Ġcandidate Ġcell Ġstate Ġand Ġthe Ġprevious Ġcell Ġstate , Ġto Ġthe Ġnew Ġcell Ġstate ĠIN LINE FORM 4 Ġ. ĠSimilarly Ġthe Ġoutput Ġgate ĠIN LINE FORM 5 Ġsoft - select s Ġwhich Ġportion Ġof Ġthe Ġcell Ġstate ĠIN LINE FORM 6 Ġis Ġto Ġbe Ġused Ġin Ġthe Ġfinal Ġhidden Ġstate . </s> We Ġcan Ġclearly Ġsee Ġthat Ġcell Ġstates Ġ( ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġ) Ġplay Ġa Ġcrucial Ġrole Ġin Ġforming Ġhorizontal Ġrec urrence . ĠHowever Ġthe Ġcurrent Ġformulation Ġdoes Ġnot Ġconsider ĠIN LINE FORM 3 Ġ, Ġthe Ġcell Ġstate Ġfrom ĠIN LINE FORM 4 Ġ- th Ġlayer , Ġin Ġcomputation Ġand Ġthus Ġthe Ġlower Ġcontext Ġis Ġreflected Ġonly Ġthrough Ġthe Ġrudimentary Ġway , Ġhind ering Ġthe Ġpossibility Ġof Ġcontrolling Ġvertical Ġinformation Ġflow . </s> Cell - aware ĠSt acked ĠL ST Ms </s> Now Ġwe Ġextend Ġthe Ġstacked ĠL ST M Ġformulation Ġdefined Ġabove Ġto Ġaddress Ġthe Ġproblem Ġnoted Ġin Ġthe Ġprevious Ġsubsection . ĠTo Ġenhance Ġthe Ġinteraction Ġbetween Ġlayers Ġin Ġa Ġway Ġsimilar Ġto Ġhow ĠL ST Ms Ġkeep Ġand Ġforget Ġthe Ġinformation Ġfrom Ġthe Ġprevious Ġtime Ġstep , Ġwe Ġintroduce Ġthe Ġadditional Ġforget Ġgate ĠIN LINE FORM 0 Ġthat Ġdetermines Ġwhether Ġto Ġaccept Ġor Ġignore Ġthe Ġsignals Ġcoming Ġfrom Ġthe Ġprevious Ġlayer . ĠTherefore Ġthe Ġproposed ĠCell - aware ĠSt acked ĠL ST M Ġis Ġformulated Ġas Ġfollows : ĠDIS PLAY FORM 0 ĠDIS PLAY FORM 1 </s> where ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġ. ĠIN LINE FORM 2 Ġcan Ġeither Ġbe Ġa Ġvector Ġof Ġconstants Ġor Ġparameters . ĠWhen ĠIN LINE FORM 3 Ġ, Ġthe Ġequations Ġdefined Ġin Ġthe Ġprevious Ġsubsection Ġare Ġused . ĠTherefore , Ġit Ġcan Ġbe Ġsaid Ġthat Ġeach Ġnon - bottom Ġlayer Ġof ĠCAS - L ST M Ġaccepts Ġtwo Ġsets Ġof Ġhidden Ġand Ġcell Ġstates âĢĶ one Ġfrom Ġthe Ġleft Ġcontext Ġand Ġthe Ġother Ġfrom Ġthe Ġbelow Ġcontext . ĠThe Ġleft Ġand Ġthe Ġbelow Ġcontext Ġparticipate Ġin Ġcomputation Ġwith Ġthe Ġequivalent Ġprocedure Ġso Ġthat Ġthe Ġinformation Ġfrom Ġlower Ġlayers Ġcan Ġbe Ġefficiently Ġpropag ated . ĠFig . ĠFIG REF 1 Ġcompares ĠCAS - L ST M Ġto Ġthe Ġconventional Ġstacked ĠL ST M Ġarchitecture , Ġand ĠFig . ĠFIG REF 8 Ġdepicts Ġthe Ġcomputation Ġflow Ġof Ġthe ĠCAS - L ST M . </s> We Ġargue Ġthat Ġconsidering ĠIN LINE FORM 0 Ġin Ġcomputation Ġis Ġbeneficial Ġfor Ġthe Ġfollowing Ġreasons . ĠFirst , ĠIN LINE FORM 1 Ġcontains Ġadditional Ġinformation Ġcompared Ġto ĠIN LINE FORM 2 Ġsince Ġit Ġis Ġnot Ġfiltered Ġby ĠIN LINE FORM 3 Ġ. ĠThus Ġa Ġmodel Ġthat Ġdirectly Ġuses ĠIN LINE FORM 4 Ġdoes Ġnot Ġrely Ġsolely Ġon ĠIN LINE FORM 5 Ġfor Ġextracting Ġinformation , Ġdue Ġto Ġthe Ġfact Ġthat Ġit Ġhas Ġaccess Ġto Ġthe Ġraw Ġinformation ĠIN LINE FORM 6 Ġ, Ġas Ġin Ġtemporal Ġconnections . ĠIn Ġother Ġwords , ĠIN LINE FORM 7 Ġno Ġlonger Ġhas Ġto Ġtake Ġall Ġresponsibility Ġfor Ġselecting Ġuseful Ġfeatures Ġfor Ġboth Ġhorizontal Ġand Ġvertical Ġtransitions , Ġand Ġthe Ġburden Ġof Ġselecting Ġinformation Ġis Ġshared Ġwith ĠIN LINE FORM 8 Ġ. </s> Another Ġadvantage Ġof Ġusing Ġthe ĠIN LINE FORM 0 Ġlies Ġin Ġthe Ġfact Ġthat Ġit Ġdirectly Ġconnects ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġ. ĠThis Ġdirect Ġconnection Ġhelps Ġand Ġstabil izes Ġtraining , Ġsince Ġthe Ġterminal Ġerror Ġsignals Ġcan Ġbe Ġeasily Ġback prop ag ated Ġto Ġmodel Ġparameters . ĠFig . ĠFIG REF 23 Ġillustrates Ġpaths Ġbetween Ġthe Ġtwo Ġcell Ġstates . </s> We Ġfind Ġexperiment ally Ġthat Ġthere Ġis Ġlittle Ġdifference Ġbetween Ġletting ĠIN LINE FORM 0 Ġbe Ġconstant Ġand Ġletting Ġit Ġbe Ġtrain able Ġparameters , Ġthus Ġwe Ġset ĠIN LINE FORM 1 Ġin Ġall Ġexperiments . ĠWe Ġalso Ġexperimented Ġwith Ġthe Ġarchitecture Ġwithout ĠIN LINE FORM 2 Ġi . e . Ġtwo Ġcell Ġstates Ġare Ġcombined Ġby Ġun weight ed Ġsumm ation Ġsimilar Ġto Ġmult id imensional ĠR NN s ĠB IB REF 27 Ġ, Ġand Ġfound Ġthat Ġit Ġleads Ġto Ġperformance Ġdegradation Ġand Ġunstable Ġconvergence , Ġlikely Ġdue Ġto Ġmismatch Ġin Ġthe Ġrange Ġof Ġcell Ġstate Ġvalues Ġbetween Ġlayers Ġ( ĠIN LINE FORM 3 Ġfor Ġthe Ġfirst Ġlayer Ġand ĠIN LINE FORM 4 Ġfor Ġthe Ġothers ). ĠExperimental Ġresults Ġon Ġvarious ĠIN LINE FORM 5 Ġare Ġpresented Ġin ĠÂ§ ĠSEC REF 3 Ġ. </s> The Ġidea Ġof Ġhaving Ġmultiple Ġstates Ġis Ġalso Ġrelated Ġto Ġtree - struct ured ĠR NN s ĠB IB REF 29 Ġ, ĠB IB REF 30 Ġ. ĠAmong Ġthem , Ġtree - struct ured ĠL ST Ms Ġ( Tree - L ST Ms ) ĠB IB REF 31 Ġ, ĠB IB REF 32 Ġ, ĠB IB REF 33 Ġare Ġsimilar Ġto Ġours Ġin Ġthat Ġthey Ġuse Ġboth Ġhidden Ġand Ġcell Ġstates Ġfrom Ġchildren Ġnodes . ĠIn ĠTree - L ST Ms , Ġstates Ġfor Ġall Ġchildren Ġnodes Ġare Ġregarded Ġas Ġinput , Ġand Ġthey Ġparticipate Ġin Ġthe Ġcomputation Ġequally Ġthrough Ġweight - shared Ġ( in ĠChild - Sum ĠTree - L ST Ms ) Ġor Ġweight - un shared Ġ( in ĠIN LINE FORM 0 Ġ- ary ĠTree - L ST Ms ) Ġprojection . ĠFrom Ġthis Ġperspective , Ġeach ĠCAS - L ST M Ġlayer Ġ( where ĠIN LINE FORM 1 Ġ) Ġcan Ġbe Ġseen Ġas Ġa Ġbinary ĠTree - L ST M Ġwhere Ġthe Ġstructures Ġit Ġoperates Ġon Ġare Ġfixed Ġto Ġright - br anch ing Ġtrees . ĠThe Ġuse Ġof Ġcell Ġstate Ġin Ġcomputation Ġcould Ġbe Ġone Ġreason Ġthat ĠTree - L ST Ms Ġperform Ġbetter Ġthan Ġsequential ĠL ST Ms Ġeven Ġwhen Ġtrivial Ġtrees Ġ( st rict ly Ġleft - Ġor Ġright - br anch ing ) Ġare Ġgiven ĠB IB REF 34 Ġ. </s> Mult id imensional ĠR NN s Ġ( M DR NN ) Ġare Ġan Ġextension Ġof Ġ1 D Ġsequential ĠR NN s Ġthat Ġcan Ġaccept Ġmult id imensional Ġinput Ġe . g . Ġimages , Ġand Ġhave Ġbeen Ġsuccessfully Ġapplied Ġto Ġimage Ġsegment ation ĠB IB REF 26 Ġand Ġhandwriting Ġrecognition ĠB IB REF 27 Ġ. ĠNotably Ġmult id imensional ĠL ST Ms Ġ( MD L ST M ) ĠB IB REF 27 Ġhave Ġan Ġanalogous Ġformulation Ġto Ġours Ġexcept Ġthe ĠIN LINE FORM 0 Ġterm Ġand Ġthe Ġfact Ġthat Ġwe Ġuse Ġdistinct Ġweights Ġper Ġcolumn Ġ( or Ġ` layer ' Ġin Ġour Ġcase ). ĠFrom Ġthis Ġview , ĠCAS - L ST M Ġcan Ġbe Ġseen Ġas Ġa Ġcertain Ġkind Ġof ĠMD L ST M Ġthat Ġaccepts Ġa Ġ2 D Ġinput ĠIN LINE FORM 1 Ġ. ĠGrid ĠL ST Ms ĠB IB REF 21 Ġalso Ġtake ĠIN LINE FORM 2 Ġinputs Ġbut Ġemit ĠIN LINE FORM 3 Ġoutputs , Ġwhich Ġis Ġdifferent Ġfrom Ġour Ġcase Ġwhere Ġa Ġsingle Ġset Ġof Ġhidden Ġand Ġcell Ġstates Ġis Ġproduced . </s> Sent ence ĠEnc od ers </s> The Ġsentence Ġenc oder Ġnetwork Ġwe Ġuse Ġin Ġour Ġexperiments Ġtakes ĠIN LINE FORM 0 Ġwords Ġ( ass umed Ġto Ġbe Ġone - hot Ġvectors ) Ġas Ġinput . ĠThe Ġwords Ġare Ġprojected Ġto Ġcorresponding Ġword Ġrepresentations : ĠIN LINE FORM 1 Ġwhere ĠIN LINE FORM 2 Ġ. ĠThen ĠIN LINE FORM 3 Ġis Ġfed Ġto Ġa ĠIN LINE FORM 4 Ġ- layer ĠCAS - L ST M Ġmodel , Ġresulting Ġin Ġthe Ġrepresentations ĠIN LINE FORM 5 Ġ. ĠThe Ġsentence Ġrepresentation , ĠIN LINE FORM 6 Ġ, Ġis Ġcomputed Ġby Ġmax - pool ing ĠIN LINE FORM 7 Ġover Ġtime Ġas Ġin Ġthe Ġwork Ġof ĠB IB REF 35 Ġ. ĠSimilar Ġto Ġtheir Ġresults , Ġfrom Ġpreliminary Ġexperiments Ġwe Ġfound Ġthat Ġthe Ġmax - pool ing Ġperforms Ġconsistently Ġbetter Ġthan Ġmean - Ġand Ġlast - pool ing . </s> To Ġmake Ġmodels Ġmore Ġexpressive , Ġa Ġbid irection al ĠCAS - L ST M Ġnetwork Ġmay Ġalso Ġbe Ġused . ĠIn Ġthe Ġbid irection al Ġcase , Ġthe Ġforward Ġrepresentations ĠIN LINE FORM 0 Ġand Ġthe Ġbackward Ġrepresentations ĠIN LINE FORM 1 Ġare Ġconc aten ated Ġand Ġmax - pool ed Ġto Ġyield Ġthe Ġsentence Ġrepresentation ĠIN LINE FORM 2 Ġ. ĠWe Ġcall Ġthis Ġbid irection al Ġarchitecture ĠBi - C AS - L ST M Ġin Ġexperiments . </s> Top - layer ĠClass ifiers </s> For Ġthe Ġnatural Ġlanguage Ġinference Ġexperiments , Ġwe Ġuse Ġthe Ġfollowing Ġhe uristic Ġfunction Ġproposed Ġby ĠB IB REF 36 Ġin Ġfeature Ġextraction : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġmeans Ġvector Ġconc aten ation , Ġand ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġare Ġapplied Ġelement - wise . </s> And Ġwe Ġuse Ġthe Ġfollowing Ġfunction Ġin Ġparaph rase Ġidentification Ġexperiments : ĠDIS PLAY FORM 0 </s> as Ġin Ġthe Ġwork Ġof ĠB IB REF 37 Ġ. </s> For Ġsentiment Ġclassification , Ġwe Ġuse Ġthe Ġsentence Ġrepresentation Ġitself . ĠDIS PLAY FORM 0 </s> We Ġfeed Ġthe Ġfeature Ġextracted Ġfrom ĠIN LINE FORM 0 Ġas Ġinput Ġto Ġthe ĠML P Ġclass ifier Ġwith ĠRe LU Ġactivation Ġfollowed Ġby Ġthe Ġfully - connected Ġsoft max Ġlayer Ġto Ġpredict Ġthe Ġlabel Ġdistribution : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġis Ġthe Ġnumber Ġof Ġlabel Ġclasses , Ġand ĠIN LINE FORM 2 Ġthe Ġdimension Ġof Ġthe ĠML P Ġoutput , </s> Exper iments </s> We Ġevaluate Ġour Ġmethod Ġon Ġnatural Ġlanguage Ġinference Ġ( N LI ), Ġparaph rase Ġidentification Ġ( PI ), Ġand Ġsentiment Ġclassification . ĠWe Ġalso Ġconduct Ġanalysis Ġon Ġgate Ġvalues Ġand Ġexperiments Ġon Ġmodel Ġvariants . ĠFor Ġdetailed Ġexperimental Ġsettings , Ġwe Ġrefer Ġreaders Ġto Ġthe Ġsupplemental Ġmaterial . </s> For Ġthe ĠNL I Ġand ĠPI Ġtasks , Ġthere Ġexists Ġrecent Ġwork Ġspecializing Ġin Ġsentence Ġpair Ġclassification . ĠHowever Ġin Ġthis Ġwork Ġwe Ġconf ine Ġour Ġmodel Ġto Ġthe Ġarchitecture Ġthat Ġenc odes Ġeach Ġsentence Ġusing Ġa Ġshared Ġenc oder Ġwithout Ġany Ġinter - sent ence Ġinteraction , Ġin Ġorder Ġto Ġfocus Ġon Ġthe Ġeffectiveness Ġof Ġthe Ġmodels Ġin Ġextracting Ġsemantics . ĠBut Ġnote Ġthat Ġthe Ġapplic ability Ġof ĠCAS - L ST M Ġis Ġnot Ġlimited Ġto Ġsentence Ġencoding Ġbased Ġapproaches . </s> Natural ĠLanguage ĠIn ference </s> For Ġthe Ġevaluation Ġof Ġperformance Ġof Ġthe Ġproposed Ġmethod Ġon Ġthe ĠNL I Ġtask , ĠSN LI ĠB IB REF 22 Ġand ĠMulti N LI ĠB IB REF 23 Ġdatasets Ġare Ġused . ĠThe Ġobjective Ġof Ġboth Ġdatasets Ġis Ġto Ġpredict Ġthe Ġrelationship Ġbetween Ġa Ġpremise Ġand Ġa Ġhypothesis Ġsentence : Ġentail ment , Ġcontradiction , Ġand Ġneutral . ĠSN LI Ġand ĠMulti N LI Ġdatasets Ġare Ġcomposed Ġof Ġabout Ġ570 k Ġand Ġ430 k Ġpremise - hyp ot hesis Ġpairs Ġrespectively . </s> G lo Ve Ġpret rained Ġword Ġembed d ings ĠB IB REF 49 Ġare Ġused Ġand Ġremain Ġfixed Ġduring Ġtraining . ĠThe Ġdimension Ġof Ġenc oder Ġstates Ġ( ĠIN LINE FORM 0 Ġ) Ġis Ġset Ġto Ġ300 Ġand Ġa Ġ1024 D ĠML P Ġwith Ġone Ġor Ġtwo Ġhidden Ġlayers Ġis Ġused . ĠWe Ġapply Ġdrop out ĠB IB REF 50 Ġto Ġthe Ġword Ġembed d ings Ġand Ġthe ĠML P Ġlayers . ĠThe Ġfeatures Ġused Ġas Ġinput Ġto Ġthe ĠML P Ġclass ifier Ġare Ġextracted Ġfollowing ĠE q . ĠEQ REF 28 Ġ. </s> Table ĠT AB REF 32 Ġand ĠT AB REF 33 Ġcontain Ġresults Ġof Ġthe Ġmodels Ġon ĠSN LI Ġand ĠMulti N LI Ġdatasets . ĠIn ĠSN LI , Ġour Ġbest Ġmodel Ġachieves Ġthe Ġnew Ġstate - of - the - art Ġaccuracy Ġof Ġ87 . 0 % Ġwith Ġrelatively Ġfewer Ġparameters . ĠSimilarly Ġin ĠMulti N LI , Ġour Ġmodels Ġmatch Ġthe Ġaccuracy Ġof Ġstate - of - the - art Ġmodels Ġin Ġboth Ġin - domain Ġ( matched ) Ġand Ġcross - domain Ġ( m ism atched ) Ġtest Ġsets . ĠNote Ġthat Ġonly Ġthe ĠGl o Ve Ġword Ġvectors Ġare Ġused Ġas Ġword Ġrepresentations , Ġas Ġopposed Ġto Ġsome Ġmodels Ġthat Ġintroduce Ġcharacter - level Ġfeatures . ĠIt Ġis Ġalso Ġnotable Ġthat Ġour Ġproposed Ġarchitecture Ġdoes Ġnot Ġrestrict Ġthe Ġselection Ġof Ġpool ing Ġmethod ; Ġthe Ġperformance Ġcould Ġfurther Ġbe Ġimproved Ġby Ġreplacing Ġmax - pool ing Ġwith Ġother Ġadvanced Ġalgorithms Ġe . g . Ġintra - sent ence Ġattention ĠB IB REF 39 Ġand Ġgeneralized Ġpool ing ĠB IB REF 19 Ġ. </s> Par aph rase ĠIdentification </s> We Ġuse ĠQu ora ĠQuestion ĠP airs Ġdataset ĠB IB REF 24 Ġin Ġevaluating Ġthe Ġperformance Ġof Ġour Ġmethod Ġon Ġthe ĠPI Ġtask . ĠThe Ġdataset Ġconsists Ġof Ġover Ġ400 k Ġquestion Ġpairs , Ġand Ġeach Ġpair Ġis Ġannot ated Ġwith Ġwhether Ġthe Ġtwo Ġsentences Ġare Ġparaph rase Ġof Ġeach Ġother Ġor Ġnot . </s> Similar Ġto Ġthe ĠNL I Ġexperiments , ĠGl o Ve Ġpret rained Ġvectors , Ġ300 D Ġenc od ers , Ġand Ġ1024 D ĠML P Ġare Ġused . ĠThe Ġnumber Ġof ĠCAS - L ST M Ġlayers Ġis Ġfixed Ġto Ġ2 Ġin ĠPI Ġexperiments . ĠTwo Ġsentence Ġvectors Ġare Ġaggreg ated Ġusing ĠE q . ĠEQ REF 29 Ġand Ġfed Ġas Ġinput Ġto Ġthe ĠML P . ĠThe Ġresults Ġon Ġthe ĠQu ora ĠQuestion ĠP airs Ġdataset Ġare Ġsummarized Ġin ĠTable ĠT AB REF 34 Ġ. ĠAgain Ġwe Ġcan Ġsee Ġthat Ġour Ġmodels Ġoutper form Ġother Ġmodels Ġby Ġlarge Ġmargin , Ġachieving Ġthe Ġnew Ġstate Ġof Ġthe Ġart . </s> Sent iment ĠClassification </s> In Ġevaluating Ġsentiment Ġclassification Ġperformance , Ġthe ĠStanford ĠSent iment ĠTree bank Ġ( S ST ) ĠB IB REF 25 Ġis Ġused . ĠIt Ġconsists Ġof Ġabout Ġ12 , 000 Ġbinary - p ars ed Ġsentences Ġwhere Ġconstituents Ġ( ph r ases ) Ġof Ġeach Ġparse Ġtree Ġare Ġannot ated Ġwith Ġa Ġsentiment Ġlabel Ġ( very Ġpositive , Ġpositive , Ġneutral , Ġnegative , Ġvery Ġnegative ). ĠFollowing Ġthe Ġconvention Ġof Ġprior Ġwork , Ġall Ġphrases Ġand Ġtheir Ġlabels Ġare Ġused Ġin Ġtraining Ġbut Ġonly Ġthe Ġsentence - level Ġdata Ġare Ġused Ġin Ġevaluation . </s> In Ġevaluation Ġwe Ġconsider Ġtwo Ġsettings , Ġnamely ĠS ST - 2 Ġand ĠS ST - 5 , Ġthe Ġtwo Ġdiffering Ġonly Ġin Ġtheir Ġlevel Ġof Ġgran ularity Ġwith Ġregard Ġto Ġlabels . ĠIn ĠS ST - 2 , Ġdata Ġsamples Ġannot ated Ġwith Ġ` neutral ' Ġare Ġignored Ġfrom Ġtraining Ġand Ġevaluation . ĠThe Ġtwo Ġpositive Ġlabels Ġ( very Ġpositive , Ġpositive ) Ġare Ġconsidered Ġas Ġthe Ġsame Ġlabel , Ġand Ġsimilarly Ġfor Ġthe Ġtwo Ġnegative Ġlabels . ĠAs Ġa Ġresult Ġ98 , 794 / 8 72 / 1 , 8 21 Ġdata Ġsamples Ġare Ġused Ġin Ġtraining / valid ation / test , Ġand Ġthe Ġtask Ġis Ġconsidered Ġas Ġa Ġbinary Ġclassification Ġproblem . ĠIn ĠS ST - 5 , Ġdata Ġare Ġused Ġas - is Ġand Ġthus Ġthe Ġtask Ġis Ġa Ġ5 - class Ġclassification Ġproblem . ĠAll Ġ318 , 582 / 1 , 101 / 2 , 210 Ġdata Ġsamples Ġfor Ġtraining / valid ation / test Ġare Ġused Ġin Ġthe ĠS ST - 5 Ġsetting . </s> We Ġuse Ġ300 D ĠGl o Ve Ġvectors , Ġ2 - layer Ġ150 D Ġor Ġ300 D Ġenc od ers , Ġand Ġa Ġ300 D ĠML P Ġclass ifier Ġfor Ġthe Ġmodels , Ġhowever Ġunlike Ġprevious Ġexperiments Ġwe Ġtune Ġthe Ġword Ġembed d ings Ġduring Ġtraining . ĠThe Ġresults Ġon ĠS ST Ġare Ġlisted Ġin ĠTable ĠT AB REF 35 Ġ. ĠOur Ġmodels Ġachieve Ġthe Ġnew Ġstate - of - the - art Ġaccuracy Ġon ĠS ST - 2 Ġand Ġcompetitive Ġaccuracy Ġon ĠS ST - 5 , Ġwithout Ġutilizing Ġparse Ġtree Ġinformation . </s> For get ĠGate ĠAnalysis </s> To Ġinspect Ġthe Ġeffect Ġof Ġthe Ġadditional Ġforget Ġgate , Ġwe Ġinvestigate Ġhow Ġthe Ġvalues Ġof Ġvertical Ġforget Ġgates Ġare Ġdistributed . ĠWe Ġsample Ġ1 , 000 Ġrandom Ġsentences Ġfrom Ġthe Ġdevelopment Ġset Ġof Ġthe ĠSN LI Ġdataset , Ġand Ġuse Ġthe Ġ3 - layer ĠCAS - L ST M Ġmodel Ġtrained Ġon Ġthe ĠSN LI Ġdataset Ġto Ġcompute Ġgate Ġvalues . </s> If Ġall Ġvalues Ġfrom Ġa Ġvertical Ġforget Ġgate ĠIN LINE FORM 0 Ġwere Ġto Ġbe Ġ0 , Ġthis Ġwould Ġmean Ġthat Ġthe Ġintroduction Ġof Ġthe Ġadditional Ġforget Ġgate Ġis Ġmeaningless Ġand Ġthe Ġmodel Ġwould Ġreduce Ġto Ġa Ġplain Ġstacked ĠL ST M . ĠOn Ġthe Ġcontrary Ġif Ġall Ġvalues Ġwere Ġ1 , Ġmeaning Ġthat Ġthe Ġvertical Ġforget Ġgates Ġwere Ġalways Ġopen , Ġit Ġwould Ġbe Ġimpossible Ġto Ġsay Ġthat Ġthe Ġinformation Ġis Ġmod ulated Ġeffectively . </s> Fig . ĠFIG REF 40 Ġand ĠFIG REF 40 Ġrepresent Ġhist ograms Ġof Ġthe Ġvertical Ġforget Ġgate Ġvalues Ġfrom Ġthe Ġsecond Ġand Ġthe Ġthird Ġlayer . ĠFrom Ġthe Ġfigures Ġwe Ġcan Ġvalidate Ġthat Ġthe Ġtrained Ġmodel Ġdoes Ġnot Ġfall Ġinto Ġthe Ġdegener ate Ġcase Ġwhere Ġvertical Ġforget Ġgates Ġare Ġignored . ĠAlso Ġthe Ġfigures Ġshow Ġthat Ġthe Ġvalues Ġare Ġright - s ke wed , Ġwhich Ġwe Ġconjecture Ġto Ġbe Ġa Ġresult Ġof Ġfocusing Ġmore Ġon Ġa Ġstrong Ġinteraction Ġbetween Ġadjacent Ġlayers . </s> To Ġfurther Ġverify Ġthat Ġthe Ġgate Ġvalues Ġare Ġdiverse Ġenough Ġwithin Ġeach Ġtime Ġstep , Ġwe Ġcompute Ġthe Ġdistribution Ġof Ġthe Ġrange Ġof Ġvalues Ġper Ġtime Ġstep , ĠIN LINE FORM 0 Ġ, Ġwhere ĠIN LINE FORM 1 Ġ. ĠWe Ġplot Ġthe Ġhist ograms Ġin ĠFig . ĠFIG REF 40 Ġand ĠFIG REF 40 Ġ. ĠFrom Ġthe Ġfigure Ġwe Ġsee Ġthat Ġa Ġvertical Ġforget Ġgate Ġcontrols Ġthe Ġamount Ġof Ġinformation Ġflow Ġeffectively , Ġmaking Ġthe Ġdecision Ġof Ġretaining Ġor Ġdisc arding Ġsignals . </s> Finally , Ġto Ġinvestigate Ġthe Ġargument Ġpresented Ġin ĠÂ§ ĠSEC REF 2 Ġthat Ġthe Ġadditional Ġforget Ġgate Ġhelps Ġthe Ġprevious Ġoutput Ġgate Ġwith Ġreducing Ġthe Ġburden Ġof Ġextracting Ġall Ġneeded Ġinformation , Ġwe Ġinspect Ġthe Ġdistribution Ġof Ġthe Ġvalues Ġfrom ĠIN LINE FORM 0 Ġ. ĠThis Ġdistribution Ġindicates Ġhow Ġdifferently Ġthe Ġvertical Ġforget Ġgate Ġand Ġthe Ġprevious Ġoutput Ġgate Ġselect Ġinformation Ġfrom ĠIN LINE FORM 1 Ġ. ĠFrom ĠFig . ĠFIG REF 40 Ġand ĠFIG REF 40 Ġwe Ġcan Ġsee Ġthat Ġthe Ġtwo Ġgates Ġmake Ġfairly Ġdifferent Ġdecisions , Ġfrom Ġwhich Ġwe Ġdemonstrate Ġthat Ġthe Ġdirect Ġpath Ġbetween ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġenables Ġa Ġmodel Ġto Ġutilize Ġsignals Ġoverlooked Ġby ĠIN LINE FORM 4 Ġ. </s> Model ĠVari ations </s> In Ġthis Ġsubsection , Ġwe Ġsee Ġthe Ġinfluence Ġof Ġeach Ġcomponent Ġof Ġa Ġmodel Ġon Ġperformance Ġby Ġremoving Ġor Ġreplacing Ġits Ġcomponents . Ġthe ĠSN LI Ġdataset Ġis Ġused Ġfor Ġexperiments , Ġand Ġthe Ġbest Ġperforming Ġconfiguration Ġis Ġused Ġas Ġa Ġbaseline Ġfor Ġmodifications . ĠWe Ġconsider Ġthe Ġfollowing Ġvariants : Ġ( i ) Ġmodels Ġthat Ġuse Ġplain Ġstacked ĠL ST Ms , Ġ( ii ) Ġmodels Ġwith Ġdifferent ĠIN LINE FORM 0 Ġ, Ġ( iii ) Ġmodels Ġwithout ĠIN LINE FORM 1 Ġ, Ġand Ġ( iv ) Ġmodels Ġthat Ġintegrate Ġlower Ġcontexts Ġvia Ġpe ep hole Ġconnections . </s> Vari ant Ġ( iv ) Ġintegrates Ġlower Ġcontexts Ġvia Ġthe Ġfollowing Ġequations : ĠDIS PLAY FORM 0 ĠDIS PLAY FORM 1 </s> where ĠIN LINE FORM 0 Ġrepresent Ġpe ep hole Ġweights Ġthat Ġtake Ġcell Ġstates Ġinto Ġaccount . ĠAmong Ġthe Ġabove Ġequations , Ġthose Ġthat Ġuse Ġthe Ġlower Ġcell Ġstate ĠIN LINE FORM 1 Ġare ĠE q . ĠEQ REF 52 Ġand ĠEQ REF 55 Ġ. ĠWe Ġcan Ġsee Ġthat ĠIN LINE FORM 2 Ġaffects Ġthe Ġvalue Ġof ĠIN LINE FORM 3 Ġonly Ġvia Ġpe ep hole Ġconnections , Ġwhich Ġmakes ĠIN LINE FORM 4 Ġindependent Ġof ĠIN LINE FORM 5 Ġ. </s> Table ĠT AB REF 36 Ġsummarizes Ġthe Ġresults Ġof Ġmodel Ġvariants . ĠWe Ġcan Ġagain Ġsee Ġthat Ġthe Ġuse Ġof Ġcell Ġstates Ġclearly Ġimproves Ġsentence Ġmodeling Ġperformance Ġ( bas eline Ġvs . Ġ( i ) Ġand Ġ( iv ) Ġvs . Ġ( i )). ĠAlso Ġfrom Ġthe Ġresults Ġof Ġbaseline Ġand Ġ( ii ), Ġwe Ġvalidate Ġthat Ġthe Ġselection Ġof ĠIN LINE FORM 0 Ġdoes Ġnot Ġsignificantly Ġaffect Ġperformance Ġbut Ġintroducing ĠIN LINE FORM 1 Ġis Ġbeneficial Ġ( bas eline Ġvs . Ġ( iii )) Ġpossibly Ġdue Ġto Ġits Ġeffect Ġon Ġnormal izing Ġinformation Ġfrom Ġmultiple Ġsources , Ġas Ġmentioned Ġin ĠÂ§ ĠSEC REF 2 Ġ. ĠFinally , Ġfrom Ġthe Ġcomparison Ġbetween Ġbaseline Ġand Ġ( iv ), Ġwe Ġshow Ġthat Ġthe Ġproposed Ġway Ġof Ġcombining Ġthe Ġleft Ġand Ġthe Ġlower Ġcontexts Ġleads Ġto Ġbetter Ġmodeling Ġof Ġsentence Ġrepresentations Ġthan Ġthat Ġof ĠB IB REF 20 Ġin Ġencoding Ġsentences . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġproposed Ġa Ġmethod Ġof Ġstacking Ġmultiple ĠL ST M Ġlayers Ġfor Ġmodeling Ġsentences , Ġdubbed ĠCAS - L ST M . ĠIt Ġuses Ġnot Ġonly Ġhidden Ġstates Ġbut Ġalso Ġcell Ġstates Ġfrom Ġthe Ġprevious Ġlayer , Ġfor Ġthe Ġpurpose Ġof Ġcontrolling Ġthe Ġvertical Ġinformation Ġflow Ġin Ġa Ġmore Ġelaborate Ġway . ĠWe Ġevaluated Ġthe Ġproposed Ġmethod Ġon Ġvarious Ġbenchmark Ġtasks : Ġnatural Ġlanguage Ġinference , Ġparaph rase Ġidentification , Ġand Ġsentiment Ġclassification . ĠOur Ġmodels Ġachieve Ġthe Ġnew Ġstate - of - the - art Ġaccuracy Ġon ĠSN LI Ġand ĠQu ora ĠQuestion ĠP airs Ġdatasets Ġand Ġobtain Ġcomparable Ġresults Ġon ĠMulti N LI Ġand ĠS ST Ġdatasets . ĠThe Ġproposed Ġarchitecture Ġcan Ġreplace Ġany Ġstacked ĠL ST M Ġunder Ġone Ġweak Ġrestriction âĢĶ the Ġsize Ġof Ġstates Ġshould Ġbe Ġidentical Ġacross Ġall Ġlayers . </s> For Ġfuture Ġwork Ġwe Ġplan Ġto Ġapply Ġthe ĠCAS - L ST M Ġarchitecture Ġbeyond Ġsentence Ġmodeling Ġtasks . ĠVarious Ġproblems Ġe . g . Ġsequence Ġlabeling , Ġsequence Ġgeneration , Ġand Ġlanguage Ġmodeling Ġmight Ġbenefit Ġfrom Ġsophisticated Ġmodulation Ġon Ġcontext Ġintegration . ĠAgg reg ating Ġdiverse Ġcontexts Ġfrom Ġsequential Ġdata , Ġe . g . Ġthose Ġfrom Ġforward Ġand Ġbackward Ġreading Ġof Ġtext , Ġcould Ġalso Ġbe Ġan Ġintriguing Ġresearch Ġdirection . </s> Acknowled gments </s> We Ġthank ĠDan ĠEd m iston Ġfor Ġthe Ġreview Ġof Ġthe Ġmanuscript .