Unanswerable
NONE
1705.04153
Introduction </s> Learning Ġthe Ġdistributed Ġrepresentation Ġfor Ġlong Ġspans Ġof Ġtext Ġfrom Ġits Ġconstituents Ġhas Ġbeen Ġa Ġkey Ġstep Ġfor Ġvarious Ġnatural Ġlanguage Ġprocessing Ġ( N LP ) Ġtasks , Ġsuch Ġas Ġtext Ġclassification ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, Ġsemantic Ġmatching ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, Ġand Ġmachine Ġtranslation ĠB IB REF 4 Ġ. ĠEx isting Ġdeep Ġlearning Ġapproaches Ġtake Ġa Ġcompos itional Ġfunction Ġwith Ġdifferent Ġforms Ġto Ġcompose Ġword Ġvectors Ġrec urs ively Ġuntil Ġobtaining Ġa Ġsent ential Ġrepresentation . ĠTypically , Ġthese Ġcompos itional Ġfunctions Ġinvolve Ġrecurrent Ġneural Ġnetworks ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, Ġconv olution al Ġneural Ġnetworks ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ, Ġand Ġtree - struct ured Ġneural Ġnetworks ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ. </s> Among Ġthese Ġmethods , Ġtree - struct ured Ġneural Ġnetworks Ġ( Tree - NN s ) Ġshow Ġtheirs Ġsuperior Ġperformance Ġin Ġmany ĠN LP Ġtasks ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ. ĠFollowing Ġthe Ġsynt actic Ġtree Ġstructure , ĠTree - NN s Ġassign Ġa Ġfixed - length Ġvector Ġto Ġeach Ġword Ġat Ġthe Ġleaves Ġof Ġthe Ġtree , Ġand Ġcombine Ġword Ġand Ġphrase Ġpairs Ġrec urs ively Ġto Ġcreate Ġintermediate Ġnode Ġvectors , Ġeventually Ġobtaining Ġone Ġfinal Ġvector Ġto Ġrepresent Ġthe Ġwhole Ġsentence . </s> However , Ġthese Ġmodels Ġhave Ġa Ġmajor Ġlimitation Ġin Ġtheir Ġinability Ġto Ġfully Ġcapture Ġthe Ġrichness Ġof Ġcomposition ality ĠB IB REF 13 Ġ. ĠThe Ġsame Ġparameters Ġare Ġused Ġfor Ġall Ġkinds Ġof Ġsemantic Ġcompositions , Ġeven Ġthough Ġthe Ġcompositions Ġhave Ġdifferent Ġcharacteristics Ġin Ġnature . ĠFor Ġexample , Ġthe Ġcomposition Ġof Ġthe Ġadjective Ġand Ġthe Ġnoun Ġdiffers Ġsignificantly Ġfrom Ġthe Ġcomposition Ġof Ġthe Ġverb Ġand Ġthe Ġnoun . ĠMoreover , Ġmany Ġsemantic Ġphenomena , Ġsuch Ġas Ġsemantic Ġidi omatic ity Ġor Ġtransparency , Ġcall Ġfor Ġmore Ġpowerful Ġcompos itional Ġmechanisms ĠB IB REF 14 Ġ. ĠTherefore , ĠTree - NN s Ġsuffer Ġfrom Ġthe Ġunder fitting Ġproblem . </s> To Ġalleviate Ġthis Ġproblem , Ġsome Ġresearchers Ġpropose Ġto Ġuse Ġmultiple Ġcompos itional Ġfunctions , Ġwhich Ġare Ġarranged Ġbeforehand Ġaccording Ġto Ġsome Ġpartition Ġcriterion ĠB IB REF 11 Ġ, ĠB IB REF 13 Ġ, ĠB IB REF 15 Ġ. ĠInt uitive ly , Ġusing Ġdifferent Ġparameters Ġfor Ġdifferent Ġtypes Ġof Ġcompositions Ġhas Ġthe Ġpotential Ġto Ġgreatly Ġreduce Ġunder fitting . ĠB IB REF 13 Ġ[ ĠB IB REF 13 Ġ] Ġdefined Ġdifferent Ġcompos itional Ġfunctions Ġin Ġterms Ġof Ġsynt actic Ġcategories , Ġand Ġa Ġsuitable Ġcompos itional Ġfunction Ġis Ġselected Ġbased Ġon Ġthe Ġsynt actic Ġcategories . ĠB IB REF 15 Ġ[ ĠB IB REF 15 Ġ] Ġintroduced Ġmultiple Ġcompos itional Ġfunctions Ġand Ġduring Ġcompos itional Ġphase , Ġa Ġproper Ġone Ġis Ġselected Ġbased Ġon Ġthe Ġinput Ġinformation . ĠAlthough Ġthese Ġmodels Ġaccomplished Ġtheir Ġmission Ġto Ġa Ġcertain Ġextent , Ġthey Ġstill Ġsuffer Ġfrom Ġthe Ġfollowing Ġthree Ġchallenges . ĠFirst , Ġthe Ġpred efined Ġcompos itional Ġfunctions Ġcannot Ġcover Ġall Ġthe Ġcompos itional Ġrules ; ĠSecond , Ġthey Ġrequire Ġmore Ġlearn able Ġparameters , Ġsuffering Ġfrom Ġthe Ġproblem Ġof Ġover fitting ; ĠThird , Ġit Ġis Ġdifficult Ġto Ġdetermine Ġa Ġuniversal Ġcriterion Ġfor Ġsemantic Ġcomposition Ġbased Ġsolely Ġon Ġsynt actic Ġcategories . </s> In Ġthis Ġpaper , Ġwe Ġpropose Ġdynamic Ġcompos itional Ġneural Ġnetworks Ġover Ġtree Ġstructure , Ġin Ġwhich Ġa Ġmeta Ġnetwork Ġis Ġused Ġto Ġgenerate Ġthe Ġcontext - specific Ġparameters Ġof Ġa Ġdynamic Ġcompos itional Ġnetwork . ĠSpecifically , Ġwe Ġconstruct Ġour Ġmodels Ġbased Ġon Ġtwo Ġkinds Ġof Ġtree - struct ured Ġneural Ġnetworks : Ġrecursive Ġneural Ġnetwork Ġ( Tree - Rec NN ) ĠB IB REF 11 Ġand Ġtree - st ructure Ġlong Ġshort - term Ġmemory Ġneural Ġnetwork Ġ( Tree - L ST M ) ĠB IB REF 9 Ġ. ĠOur Ġwork Ġis Ġinspired Ġby Ġrecent Ġwork Ġon Ġdynamic Ġparameter Ġprediction ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ. ĠThe Ġmeta Ġnetwork Ġis Ġused Ġto Ġextract Ġthe Ġshared Ġmeta - knowledge Ġacross Ġdifferent Ġcompos itional Ġrules Ġand Ġto Ġdynamically Ġgenerate Ġthe Ġcontext - specific Ġcompos itional Ġfunction . ĠThus , Ġthe Ġcompos itional Ġfunction Ġof Ġour Ġmodels Ġvaries Ġwith Ġpositions , Ġcontexts Ġand Ġsamples . ĠThe Ġdynamic Ġcompos itional Ġnetwork Ġthen Ġapplies Ġthose Ġcontext - specific Ġparameters Ġto Ġthe Ġcurrent Ġinput Ġinformation . ĠBoth Ġmeta Ġand Ġdynamic Ġnetworks Ġare Ġdifferent iable Ġsuch Ġthat Ġthe Ġoverall Ġnetworks Ġcan Ġbe Ġtrained Ġin Ġan Ġend - to - end Ġfashion . ĠAdditional , Ġto Ġreduce Ġthe Ġcomplexity Ġof Ġthe Ġwhole Ġnetworks , Ġwe Ġdefine Ġthe Ġdynamic Ġweight Ġmatrix Ġin Ġa Ġmanner Ġsim ulating Ġlow - rank Ġmatrix Ġdecom position . </s> We Ġevaluate Ġour Ġmodels Ġon Ġtwo Ġtypical Ġtasks : Ġtext Ġclassification Ġand Ġtext Ġsemantic Ġmatching . ĠThe Ġresults Ġshow Ġthat Ġour Ġmodels Ġare Ġmore Ġexpressive Ġdue Ġto Ġtheir Ġlearning Ġto Ġlearn Ġnature , Ġyet Ġwithout Ġincreasing Ġthe Ġnumber Ġof Ġmodel 's Ġparameters . ĠMoreover , Ġwe Ġfind Ġcertain Ġcomposition Ġoperations Ġcan Ġbe Ġlearned Ġimplicitly Ġby Ġmeta ĠTree NN , Ġsuch Ġas Ġthe Ġcomposition Ġof Ġnoun Ġphrases Ġand Ġverb Ġphrases . </s> The Ġcontributions Ġof Ġthe Ġpaper Ġcan Ġbe Ġsummed Ġup Ġas Ġfollows . </s> Tree - Struct ured ĠNeural ĠNetwork </s> In Ġthis Ġsection , Ġwe Ġbriefly Ġdescribe Ġthe Ġtree - struct ured Ġneural Ġnetworks . </s> The Ġidea Ġof Ġtree - struct ured Ġneural Ġnetworks Ġfor Ġnatural Ġlanguage Ġprocessing Ġ( N LP ) Ġis Ġto Ġtrain Ġa Ġdeep Ġlearning Ġmodel Ġwith Ġa Ġgram matical Ġtree Ġstructure ĠB IB REF 19 Ġthat Ġcan Ġbe Ġapplied Ġto Ġphrases Ġand Ġsentences . ĠAt Ġevery Ġnode Ġin Ġthe Ġtree , Ġthe Ġcontexts Ġof Ġthe Ġleft Ġand Ġright Ġchildren Ġare Ġcombined Ġby Ġa Ġcompos itional Ġfunction . ĠThe Ġparameters Ġof Ġthe Ġcompos itional Ġfunction Ġare Ġshared Ġacross Ġall Ġnodes Ġin Ġthe Ġtree . ĠThe Ġlayer Ġcomputed Ġat Ġthe Ġtop Ġnode Ġgives Ġa Ġrepresentation Ġfor Ġthe Ġwhole Ġsentence . </s> Van illa ĠRec ursive ĠNeural ĠNetwork </s> The Ġsimplest Ġmember Ġof Ġtree - struct ured ĠN N Ġis Ġthe Ġvanilla Ġrecursive Ġneural Ġnetwork ĠB IB REF 13 Ġ, Ġin Ġwhich Ġthe Ġrepresentation Ġof Ġparent Ġis Ġcalculated Ġby Ġweighted Ġlinear Ġcombination Ġof Ġthe Ġchild Ġvectors . </s> Form ally , Ġgiven Ġa Ġbinary Ġconstituency Ġtree Ġ$ T $ Ġinduced Ġby Ġa Ġsentence , Ġeach Ġnon - leaf Ġnode Ġcorresponds Ġto Ġa Ġphrase . ĠWe Ġrefer Ġto Ġ$\ math bf Ġ{ h } _{ j } Ġ\ in Ġ\ math bb Ġ{ R } ^{ d } $ Ġas Ġthe Ġhidden Ġstate Ġof Ġeach Ġnode Ġ$ j $ Ġ, Ġand Ġlet Ġ$\ math bf Ġ{ h } _{ j } ^{ l } $ Ġ, Ġ$\ math bf Ġ{ h } _{ j } ^{ r } $ Ġdenote Ġthe Ġleft Ġand Ġright Ġchild Ġrepresentations Ġrespectively . </s> $$ \ math bf Ġ{ h } _{ j } Ġ= Ġ\ tan h Ġ\ left ( Ġ\ math bf Ġ{ W } Ġ\ begin { b mat rix } Ġ\ math bf Ġ{ h } _{ j } ^{ l } Ġ\\ Ġ\ math bf Ġ{ h } _{ j } ^{ r } Ġ\\ Ġ\ end { b mat rix } Ġ+ Ġ\ math bf Ġ{ b } Ġ\ right ), $$ Ġ Ġ Ġ( E q . Ġ5 ) </s> where Ġ$\ math bf Ġ{ W } Ġ\ in Ġ\ math bb Ġ{ R } ^{ d Ġ\ times Ġ2 d } $ Ġis Ġa Ġlearn able Ġcompos itional Ġmatrix , Ġ$\ math bf Ġ{ b } $ Ġis Ġthe Ġbias Ġvector . </s> Tree ĠL ST M </s> Tree ĠL ST M ĠB IB REF 9 Ġis Ġa Ġgeneral ization Ġof ĠL ST Ms Ġto Ġtree - struct ured Ġnetwork Ġtop ologies . ĠIn Ġthis Ġmodel , Ġthe Ġcompos itional Ġfunction Ġis Ġan ĠL ST M Ġunit , Ġand Ġthe Ġhidden Ġstate Ġ$ h _ j $ Ġof Ġeach Ġnode Ġcan Ġbe Ġcomputed Ġas Ġfollows : Ġwe Ġrefer Ġto Ġ$\ math bf Ġ{ h } _{ j } $ Ġand Ġ$\ math bf Ġ{ c } _{ j } $ Ġas Ġthe Ġhidden Ġstate Ġand Ġmemory Ġcell Ġof Ġeach Ġnode Ġ$ j $ Ġ. ĠThe Ġtransition Ġequations Ġof Ġnode Ġ$ j $ Ġare Ġas Ġfollows : </s> $$ \ begin { b mat rix } Ġ\ math bf Ġ{\ t ilde { c }} _{ j } Ġ\\ Ġ\ math bf Ġ{ o } _{ j } Ġ\\ Ġ\ math bf Ġ{ i } _{ j } Ġ\\ Ġ\ math bf Ġ{ f } _{ j } ^{ l } Ġ\\ Ġ\ math bf Ġ{ f } _{ j } ^{ r } Ġ\ end { b mat rix } Ġ& = Ġ\ begin { b mat rix } Ġ\ tan h Ġ\\ Ġ\ s igma Ġ\\ Ġ\ s igma Ġ\\ Ġ\ s igma Ġ\\ Ġ\ s igma Ġ\ end { b mat rix } Ġ\ left ( Ġ\ math bf Ġ{ W } Ġ\ begin { b mat rix } Ġ\ math bf Ġ{ x } _{ j } Ġ\\ Ġ\ math bf Ġ{ h } _{ j } ^{ l } Ġ\\ Ġ\ math bf Ġ{ h } _{ j } ^{ r } Ġ\\ Ġ\ end { b mat rix } Ġ+ Ġ\ math bf Ġ{ b } Ġ\ right ), \\ Ġ\ math bf Ġ{ c } _{ j } Ġ& = Ġ\ math bf Ġ{\ t ilde { c }} _{ j } Ġ\ od ot Ġ\ math bf Ġ{ i } _{ j } Ġ+ Ġ\ math bf Ġ{ c } _{ j } ^{ l } Ġ\ od ot Ġ\ math bf Ġ{ f } _{ j } ^{ l } + Ġ\ math bf Ġ{ c } _{ j } ^{ r } Ġ\ od ot Ġ\ math bf Ġ{ f } _{ j } ^{ r } Ġ, Ġ\\ Ġ\ math bf Ġ{ h } _{ j } Ġ& = Ġ\ math bf Ġ{ o } _{ j } Ġ\ od ot Ġ\ tan h Ġ\ left ( Ġ\ math bf Ġ{ c } _{ j } Ġ\ right ), $$ Ġ Ġ Ġ( E q . Ġ8 ) </s> where Ġ$\ math bf Ġ{ x } _ j Ġ\ in Ġ\ math bb Ġ{ R } ^{ d } $ Ġdenotes Ġthe Ġinput Ġvector Ġand Ġis Ġnon - zero Ġif Ġand Ġonly Ġif Ġit Ġis Ġa Ġleaf Ġnode . ĠThe Ġsupers cript Ġ$ l $ Ġand Ġ$ r $ Ġrepresent Ġthe Ġleft Ġchild Ġand Ġright Ġchild Ġrespectively . Ġ$\ s igma Ġ$ Ġrepresents Ġthe Ġlog istic Ġs igm oid Ġfunction Ġand Ġ$\ od ot Ġ$ Ġdenotes Ġelement - wise Ġmultiplication . Ġ$\ math bf Ġ{ W } Ġ\ in Ġ\ math bb Ġ{ R } ^{ 5 d Ġ\ times Ġ3 d } $ Ġand Ġ$\ math bf Ġ{ b } Ġ\ in Ġ\ math bb Ġ{ R } ^{ 5 d } $ Ġare Ġlearn able Ġparameters . </s> Dynamic ĠCompos itional ĠNeural ĠNetwork </s> In Ġthe Ġabove Ġtwo Ġtree - struct ured ĠN Ns , Ġthe Ġcompos itional Ġfunction Ġis Ġshared Ġacross Ġall Ġnodes Ġin Ġthe Ġtree , Ġwhich Ġresults Ġin Ġunder fitting Ġsince Ġthe Ġsemantic Ġcompositions Ġhave Ġgreat Ġdivers ities . ĠTo Ġaddress Ġthis Ġproblem , Ġwe Ġpropose Ġtwo Ġdynamic Ġcompos itional Ġneural Ġnetworks Ġover Ġtree Ġstructure , Ġwhich Ġdynamically Ġgenerate Ġdifferent Ġparameters Ġfor Ġdifferent Ġtypes Ġof Ġcompositions . ĠFigure Ġ2 Ġshows Ġan Ġillustration Ġof Ġthe Ġdynamic Ġcompos itional Ġneural Ġnetwork , Ġconsisting Ġof Ġtwo Ġcomponents : Ġ( 1 ) Ġmeta Ġnetwork Ġand Ġ( 2 ) Ġbasic Ġnetwork Ġwith Ġdynamic Ġparameters . </s> Specifically , Ġwe Ġpropose Ġtwo Ġmeta Ġnetworks Ġto Ġgenerate Ġthe Ġcontext - specific Ġcompos itional Ġfunctions Ġfor ĠRec NN Ġand ĠTree L ST M Ġrespectively . </s> Meta ĠNetwork Ġfor ĠRec NN </s> For ĠRec NN , Ġwe Ġreplace Ġthe Ġstatic Ġparameters Ġ$\ math bf Ġ{ W } $ Ġand Ġ$\ math bf Ġ{ b } $ Ġin ĠE q .( Ġ5 Ġ) Ġwith Ġthe Ġdynamic Ġparameters Ġ$\ math bf Ġ{ W } (\ math bf Ġ{ z } _{ j }) $ Ġand Ġ$\ math bf Ġ{ b } (\ math bf Ġ{ z } _{ j }) $ Ġ, Ġwhich Ġare Ġgenerated Ġby Ġa Ġmeta Ġnetwork . ĠThe Ġmeta Ġnetwork Ġis Ġa Ġsmaller ĠRec NN , Ġand Ġthe Ġhidden Ġstate Ġ$\ hat {\ math bf Ġ{ h }} _{ j }\ in Ġ\ math bb Ġ{ R } ^{ m } $ Ġof Ġnode Ġ$ j $ Ġin Ġmeta Ġnetwork Ġis Ġdefined Ġas </s> $$ \ hat {\ math bf Ġ{ h }} _{ j } Ġ& = Ġ\ tan h Ġ\ left ( Ġ\ math bf Ġ{ W } _{ m } Ġ\ math cal Ġ{ H } _{ j } Ġ+ Ġ\ math bf Ġ{ b } _{ m } Ġ\ right ), \\ Ġ\ math bf Ġ{ z } _{ j } Ġ& = Ġ\ math bf Ġ{ W } _{ z } Ġ\ hat {\ math bf Ġ{ h }} _{ j } $$ Ġ Ġ Ġ( E q . Ġ11 ) </s> where Ġ$\ math cal Ġ{ H } _{ j } Ġ= Ġ\ math bf Ġ{ h } _{ j } ^{ l }\ opl us Ġ\ math bf Ġ{ h } _{ j } ^{ r }\ opl us Ġ\ math bf Ġ{\ hat { h }} _{ j } ^{ l }\ opl us Ġ\ math bf Ġ{\ hat { h }} _{ j } ^{ r } Ġ\ in Ġ\ math bb Ġ{ R } ^{ 2 m + 2 d } $ Ġ, Ġ$\ math bf Ġ{ W } _{ m } Ġ\ in Ġ\ math bb Ġ{ R } ^{ m Ġ\ times Ġ( 2 d + 2 m )} $ Ġand Ġ$\ math bf Ġ{ b } _{ m } Ġ\ in Ġ\ math bb Ġ{ R } ^{ m } $ Ġare Ġparameters Ġof Ġmeta ĠRec NN ; Ġ$\ math bf Ġ{ W } _{ z } Ġ\ in Ġ\ math bb Ġ{ R } ^{ z Ġ\ times Ġm } $ Ġis Ġa Ġscale Ġmatrix . </s> To Ġreduce Ġthe Ġnumber Ġof Ġthe Ġparameters , Ġwe Ġdefine Ġthe Ġdynamic Ġparameters Ġwith Ġa Ġlow - rank Ġfactor ized Ġrepresentation Ġof Ġthe Ġweights , Ġanalogous Ġto Ġthe ĠSing ular ĠValue ĠDec om position . ĠThe Ġdynamic Ġparameters Ġ$\ math bf Ġ{ W } (\ math bf Ġ{ z } _{ j }) $ Ġand Ġ$\ math bf Ġ{ b } (\ math bf Ġ{ z } _{ j }) $ Ġof Ġthe Ġbasic ĠRec NN Ġare Ġcomputed Ġby : </s> $$ \ math bf Ġ{ W } (\ math bf Ġ{ z } _{ j }) Ġ& = Ġ\ begin { b mat rix } ĠP _ l Ġ\ math bf Ġ{ D } (\ math bf Ġ{ z } _ j ) Q _ l Ġ\\ ĠP _ r Ġ\ math bf Ġ{ D } (\ math bf Ġ{ z } _ j ) Q _ r Ġ\\ Ġ\ end { b mat rix } Ġ\\ Ġ\ math bf Ġ{ b } (\ math bf Ġ{ z } _{ j }) Ġ& = Ġ\ begin { b mat rix } ĠB _ l \ math bf Ġ{ z } _ j Ġ\\ ĠB _ r \ math bf Ġ{ z } _ j Ġ\\ Ġ\ end { b mat rix } $$ Ġ Ġ Ġ( E q . Ġ12 ) </s> where Ġ$ P Ġ\ in Ġ\ math bb Ġ{ R } ^{ d Ġ\ times Ġz } $ Ġ, Ġ$ Q Ġ\ in Ġ\ math bb Ġ{ R } ^{ z Ġ\ times Ġd } $ Ġ, Ġand Ġ$\ math bf Ġ{ D } (\ math bf Ġ{ z } _ t ) Ġ\ in Ġ\ math bb Ġ{ R } ^{ z Ġ\ times Ġz } $ Ġis Ġthe Ġdiagonal Ġmatrix Ġof Ġ$\ math bf Ġ{ z } $ Ġ. </s> Thus , Ġour Ġdynamic ĠRec NN Ġneeds Ġ$( 6 d z Ġ+ Ġm z ) $ Ġparameters , Ġwhile Ġthe Ġvanilla ĠRec NN Ġhas Ġ$( 2 d ^ 2 + d ) $ Ġparameters . ĠWith Ġa Ġsmall Ġ$ z $ Ġand Ġ$ m $ Ġ, Ġour Ġdynamic ĠRec NN Ġneeds Ġless Ġparameters Ġthan Ġthe Ġvanilla ĠRec NN . ĠFor Ġexample , Ġif Ġwe Ġset Ġ$ d Ġ= 100 $ Ġand Ġ$ z = m = 20 $ Ġ, Ġour Ġmodel Ġneeds Ġ$ 12 , 400 $ Ġparameters Ġwhile Ġthe Ġvanilla Ġmodel Ġneeds Ġ$ 20 , 100 $ Ġparameters . </s> Meta ĠNetwork Ġfor ĠTree L ST M </s> Likewise , Ġwe Ġalso Ġuse Ġa Ġsmaller Ġmeta Ġnetwork Ġto Ġgenerate Ġthe Ġstatic Ġparameters Ġ$\ math bf Ġ{ W } $ Ġand Ġ$\ math bf Ġ{ b } $ Ġin ĠE q .( Ġ8 Ġ) Ġwith Ġthe Ġdynamic Ġparameters Ġ$\ math bf Ġ{ W } (\ math bf Ġ{ z } _{ j }) $ Ġand Ġ$\ math bf Ġ{ b } (\ math bf Ġ{ z } _{ j }) $ Ġ. ĠThe Ġmeta Ġnetwork Ġis Ġa Ġsmaller ĠTree L ST M , Ġand Ġthe Ġhidden Ġstate Ġ$\ hat {\ math bf Ġ{ h }} _{ j }\ in Ġ\ math bb Ġ{ R } ^{ m } $ Ġof Ġnode Ġ$ j $ Ġin Ġmeta Ġnetwork Ġis Ġdefined Ġas </s> $$ \ begin { b mat rix } Ġ\ math bf Ġ{\ hat { g }} _{ j } Ġ\\ Ġ\ math bf Ġ{\ hat { o }} _{ j } Ġ\\ Ġ\ math bf Ġ{\ hat { i }} _{ j } Ġ\\ Ġ\ math bf Ġ{\ hat { f }} _{ j } ^{ l } Ġ\\ Ġ\ math bf Ġ{\ hat { f }} _{ j } ^{ r } Ġ\ end { b mat rix } Ġ& = Ġ\ begin { b mat rix } Ġ\ tan h Ġ\\ Ġ\ s igma Ġ\\ Ġ\ s igma Ġ\\ Ġ\ s igma Ġ\\ Ġ\ s igma Ġ\ end { b mat rix } Ġ\ left ( Ġ\ math bf Ġ{ W } _{ m } Ġ\ begin { b mat rix } Ġ\ math bf Ġ{ x } _{ j } Ġ\\ Ġ\ math cal Ġ{ H } _{ j } Ġ\\ Ġ\ end { b mat rix } Ġ+ Ġ\ math bf Ġ{ b } _{ m } Ġ\ right ) Ġ, \\ Ġ\ math bf Ġ{\ hat { c }} _{ j } Ġ& = Ġ\ math bf Ġ{\ hat { g }} _{ j } Ġ\ od ot Ġ\ math bf Ġ{\ hat { i }} _{ j } Ġ+ Ġ\ math bf Ġ{\ hat { g }} _{ j } ^{ l } Ġ\ od ot Ġ\ math bf Ġ{\ hat { f }} _{ j } ^{ l } + Ġ\ math bf Ġ{\ hat { g }} _{ j } ^{ r } Ġ\ od ot Ġ\ math bf Ġ{\ hat { f }} _{ j } ^{ r } Ġ, Ġ\\ Ġ\ math bf Ġ{\ hat { h }} _{ j } Ġ& = Ġ\ math bf Ġ{\ hat { o }} _{ j } Ġ\ od ot Ġ\ tan h Ġ\ left ( Ġ\ math bf Ġ{\ hat { c }} _{ j } Ġ\ right ), Ġ\\ Ġ\ math bf Ġ{ z } _{ j } Ġ& = Ġ\ math bf Ġ{ W } _{ z } Ġ\ hat {\ math bf Ġ{ h }} _{ j } $$ Ġ Ġ Ġ( E q . Ġ14 ) </s> where Ġ$\ math cal Ġ{ H } _{ j } Ġ= Ġ\ math bf Ġ{ h } _{ j } ^{ l }\ opl us Ġ\ math bf Ġ{ h } _{ j } ^{ r }\ opl us Ġ\ math bf Ġ{\ hat { h }} _{ j } ^{ l }\ opl us Ġ\ math bf Ġ{\ hat { h }} _{ j } ^{ r } Ġ\ in Ġ\ math bb Ġ{ R } ^{ 2 m + 2 d } $ Ġ; Ġ$\ math bf Ġ{ W } _{ m } Ġ\ in Ġ\ math bb Ġ{ R } ^{ 5 m Ġ\ times Ġ( 3 d + 2 m )} $ Ġand Ġ$\ math bf Ġ{ b } _{ m } Ġ\ in Ġ\ math bb Ġ{ R } ^{ m } $ Ġare Ġparameters Ġof Ġmeta ĠTree L ST M ; Ġ$\ math bf Ġ{ W } _{ z } Ġ\ in Ġ\ math bb Ġ{ R } ^{ z Ġ\ times Ġm } $ Ġis Ġa Ġscale Ġmatrix . </s> The Ġdynamic Ġparameters Ġ$\ math bf Ġ{ W } (\ math bf Ġ{ z } _{ j }) $ Ġand Ġ$\ math bf Ġ{ b } (\ math bf Ġ{ z } _{ j }) $ Ġof Ġbasic ĠTree L ST M Ġare Ġcomputed Ġby : </s> $$ \ math bf Ġ{ W } (\ math bf Ġ{ z } _{ j }) Ġ& = Ġ\ left [ \ math bf Ġ{ W } ^{ g }, Ġ\ math bf Ġ{ W } ^{ i }, Ġ\ math bf Ġ{ W } ^{ f ^ l }, \ math bf Ġ{ W } ^{ f ^ r }, \ math bf Ġ{ W } ^{ o }\ right ] Ġ\\ Ġ\ math bf Ġ{ b } (\ math bf Ġ{ z } _{ j }) Ġ& = Ġ\ left [ \ math bf Ġ{ b } ^{ g }, Ġ\ math bf Ġ{ b } ^{ i }, Ġ\ math bf Ġ{ b } ^{ f ^ l }, \ math bf Ġ{ b } ^{ f ^ r }, \ math bf Ġ{ b } ^{ o }\ right ], $$ Ġ Ġ Ġ( E q . Ġ15 ) </s> where Ġfor Ġ$ * \ in Ġ\ l brace Ġc , o , i , f ^ l , f ^ r \ r brace Ġ$ Ġ, </s> $$ \ math bf Ġ{ W } ^{ * } (\ math bf Ġ{ z } _{ j }) Ġ& = Ġ\ begin { b mat rix } ĠP _ x ^{ * } Ġ\ math bf Ġ{ D } (\ math bf Ġ{ z } _ t ) Q _ x ^{ * } Ġ\\ ĠP _ l ^{ * } Ġ\ math bf Ġ{ D } (\ math bf Ġ{ z } _ t ) Q _ l ^{ * } Ġ\\ ĠP _ r ^{ * } Ġ\ math bf Ġ{ D } (\ math bf Ġ{ z } _ t ) Q _ r ^{ * } Ġ\\ Ġ\ end { b mat rix }, Ġ\\ Ġ\ math bf Ġ{ b } ^{ * } (\ math bf Ġ{ z } _{ j }) Ġ& = Ġ\ begin { b mat rix } ĠB _ x ^{ * }\ math bf Ġ{ z } _ t Ġ\\ ĠB _ l ^{ * }\ math bf Ġ{ z } _ t Ġ\\ ĠB _ r ^{ * }\ math bf Ġ{ z } _ t Ġ\\ Ġ\ end { b mat rix }, $$ Ġ Ġ Ġ( E q . Ġ16 ) </s> where Ġ$ P Ġ\ in Ġ\ math bb Ġ{ R } ^{ 5 d Ġ\ times Ġz } $ Ġ, Ġ$ Q Ġ\ in Ġ\ math bb Ġ{ R } ^{ z Ġ\ times Ġ3 d } $ Ġ, Ġ$ B Ġ\ in Ġ\ math bb Ġ{ R } ^{ 5 d Ġ\ times Ġz } $ Ġ, Ġand Ġ$\ math bf Ġ{ D } (\ math bf Ġ{ z } _ t ) Ġ\ in Ġ\ math bb Ġ{ R } ^{ z Ġ\ times Ġz } $ Ġis Ġthe Ġdiagonal Ġmatrix Ġof Ġ$\ math bf Ġ{ z } $ Ġ. </s> With Ġa Ġsmall Ġ$ z $ Ġand Ġ$ m $ Ġ, Ġour Ġdynamic ĠTree L ST M Ġneeds Ġa Ġsimilar Ġamount Ġof Ġparameters Ġcompared Ġto Ġthe Ġstandard ĠTree L ST M . </s> Application Ġof ĠDynamic ĠCompos itional ĠNeural ĠNetworks </s> In Ġthis Ġsection , Ġwe Ġdescribe Ġtwo Ġspecific Ġmodels Ġto Ġshow Ġthe Ġapplications Ġof Ġdynamic Ġcompos itional Ġneural Ġnetworks Ġfor Ġtwo Ġtypical Ġtasks Ġin ĠN LP . </s> Text ĠClassification </s> The Ġpurpose Ġof Ġtext Ġclassification Ġis Ġthat , Ġgiven Ġa Ġsentence Ġ$ x $ Ġ, Ġthe Ġmodel Ġshould Ġpredict Ġlabels Ġ$\ hat { y } $ Ġfrom Ġa Ġpre - defined Ġlabel Ġset Ġ$\ math cal Ġ{ Y } $ Ġ. ĠFrom Ġthe Ġdescription Ġin Ġthe Ġprevious Ġsection , Ġwe Ġcan Ġcompute Ġthe Ġdistributed Ġrepresentation Ġ$\ math bf Ġ{ h } _ j $ Ġof Ġthe Ġphrase Ġat Ġnode Ġ$ j $ Ġof Ġa Ġtree : </s> $$ \ math bf Ġ{ h } _{ j } Ġ& = Ġ\ oper at orn am ew ith limits { DC - Tree NN } (\ math bf Ġ{ x } _ j , Ġ\ math bf Ġ{ h } _{ j } ^ l , Ġ\ math bf Ġ{ h } _{ j } ^ r , \ the ta Ġ) $$ Ġ Ġ Ġ( E q . Ġ18 ) </s> After Ġthis Ġrecursive Ġprocess , Ġthe Ġhidden Ġstate Ġ$\ math bf Ġ{ h } _ R $ Ġat Ġthe Ġroot Ġnode Ġis Ġused Ġas Ġthe Ġsent ential Ġrepresentation , Ġwhich Ġthen Ġfollowed Ġby Ġa Ġsoft max Ġclass ifier Ġto Ġpredict Ġthe Ġprobability Ġdistribution Ġover Ġclasses . </s> $$ {\ hat {\ math bf Ġ{ y }}} Ġ= Ġ\ oper at orn am ew ith limits { soft max } (\ math bf Ġ{ W } _ t Ġ\ math bf Ġ{ h } _ R Ġ+ Ġ\ math bf Ġ{ b } _ t ) $$ Ġ Ġ Ġ( E q . Ġ19 ) </s> where Ġ${ \ hat {\ math bf Ġ{ y }}} $ Ġis Ġprediction Ġprobabilities , Ġ$\ math bf Ġ{ W } _ t $ Ġand Ġ$\ math bf Ġ{ b } _ t $ Ġare Ġthe Ġparameters Ġof Ġthe Ġclass ifier . </s> We Ġevaluate Ġour Ġmodels Ġon Ġfive Ġdifferent Ġdatasets . ĠThe Ġdetailed Ġstatistics Ġabout Ġthe Ġfive Ġdatasets Ġare Ġlisted Ġin ĠTable Ġ2 Ġ. ĠEach Ġdataset Ġis Ġbriefly Ġdescribed Ġas Ġfollows . </s> S ST ĠThe Ġmovie Ġreviews Ġwith Ġtwo Ġclasses Ġ( negative , Ġpositive ) Ġin Ġthe ĠStanford ĠSent iment ĠTree bank ĠB IB REF 23 Ġ. </s> MR ĠThe Ġmovie Ġreviews Ġwith Ġtwo Ġclasses ĠB IB REF 24 Ġ. </s> Q C ĠThe ĠT REC Ġquestions Ġdataset Ġinvolves Ġsix Ġdifferent Ġquestion Ġtypes . ĠB IB REF 25 Ġ. </s> S UB J ĠSubject ivity Ġdataset Ġwhere Ġthe Ġgoal Ġis Ġto Ġclassify Ġeach Ġinstance Ġ( sn ipp et ) Ġas Ġbeing Ġsubjective Ġor Ġobjective . ĠB IB REF 26 </s> IE ĠId iom Ġenhanced Ġsentiment Ġclassification . ĠB IB REF 27 Ġ. ĠEach Ġsentence Ġcontains Ġat Ġleast Ġone Ġid iom . </s> As Ġshown Ġin ĠTable Ġ3 Ġ, ĠDC - Tree L ST M Ġconsistently Ġoutper forms ĠRec NN , ĠMV - Rec NN , ĠR NT N , Ġand ĠTree L ST M Ġby Ġa Ġlarge Ġmargin Ġwhile Ġachieving Ġcomparable Ġresults Ġto Ġthe ĠCNN Ġand Ġusing Ġmuch Ġfewer Ġparameters .( The Ġnumber Ġof Ġparameters Ġin Ġour Ġmodels Ġis Ġapproximately Ġ10 K Ġwhile Ġin ĠCNN Ġthe Ġnumber Ġof Ġparameters Ġis Ġabout Ġ400 K ). ĠCompared Ġwith ĠRec NN , ĠDC - Rec NN Ġperforms Ġbetter , Ġindicating Ġthe Ġeffectiveness Ġof Ġthe Ġdynamic Ġcompos itional Ġfunction . ĠAdditionally , Ġboth ĠDC - Rec NN Ġand ĠDC - Tree L ST M Ġachieve Ġsubstantial Ġimprovement Ġon ĠIE Ġdataset , Ġwhich Ġcovers Ġthe Ġrichness Ġof Ġcomposition ality Ġ( idi omatic ity ). ĠWe Ġattribute Ġthe Ġsuccess Ġon ĠIE Ġto Ġits Ġpower Ġin Ġmodeling Ġmore Ġcomplicated Ġcomposition ality . </s> Text ĠSem antic ĠMatch ing </s> Among Ġmany Ġnatural Ġlanguage Ġprocessing Ġ( N LP ) Ġtasks , Ġa Ġcommon Ġproblem Ġis Ġmodelling Ġthe Ġrelevance Ġof Ġa Ġpair Ġof Ġtexts . ĠIn Ġthis Ġsection , Ġwe Ġshow Ġhow Ġto Ġeffectively Ġuse Ġthe Ġdynamic Ġcompos itional Ġneural Ġnetworks Ġto Ġmodel Ġthe Ġsemantic Ġrelationship Ġbetween Ġtwo Ġsentences . </s> As Ġshown Ġin ĠFigure Ġ1 Ġ, Ġgiven Ġtwo Ġsentences Ġ$ x _ a $ Ġand Ġ$ x _ b $ Ġ, Ġthe Ġrepresentation Ġof Ġeach Ġsentence Ġ$\ math bf Ġ{ h } _{ R } $ Ġcan Ġbe Ġcomputed Ġby Ġone Ġbasic ĠTree NN . </s> $$ \ math bf Ġ{ h } _{ R } ^{ ( a )} Ġ& = Ġ\ oper at orn am ew ith limits { DC - Tree NN } (\ math bf Ġ{ x } _ R , Ġ\ math bf Ġ{ h } _{ R } ^ l , Ġ\ math bf Ġ{ h } _{ R } ^ r , \ the ta Ġ_ { a }) Ġ\\ Ġ\ math bf Ġ{ h } _{ R } ^{ ( b )} Ġ& = Ġ\ oper at orn am ew ith limits { DC - Tree NN } (\ math bf Ġ{ x } _ R , Ġ\ math bf Ġ{ h } _{ R } ^ l , Ġ\ math bf Ġ{ h } _{ R } ^ r , \ the ta Ġ_ { b }) $$ Ġ Ġ Ġ( E q . Ġ21 ) </s> where Ġ$ R $ Ġdenotes Ġthe Ġroot Ġnode Ġof Ġa Ġtree . Ġ$\ the ta Ġ_ { a } $ Ġand Ġ$\ the ta Ġ_ { b } $ Ġare Ġgenerated Ġby Ġa Ġshared Ġmeta ĠTree NN . ĠThen , Ġthe Ġrepresentation Ġof Ġeach Ġsentence Ġwill Ġbe Ġfed Ġinto Ġa Ġmulti - layer Ġpercept ron Ġto Ġobtain Ġa Ġunified Ġrepresentation Ġfor Ġthe Ġfinal Ġrelationship Ġclassification . </s> The Ġsample - specific Ġbut Ġshared Ġmeta ĠTree NN Ġensures Ġthat , Ġon Ġthe Ġone Ġhand Ġwe Ġcan Ġdynamically Ġmodel Ġthe Ġdiversity Ġof Ġsemantic Ġcomposition ality , Ġon Ġthe Ġother Ġhand Ġwe Ġcan Ġcapture Ġthe Ġgeneral Ġrules Ġacross Ġdifferent Ġsamples . </s> We Ġchoose Ġthe Ġdataset Ġof ĠSent ences ĠInv olving ĠCompos itional ĠKnowledge Ġ( S ICK ), Ġwhich Ġis Ġproposed Ġby ĠB IB REF 30 Ġ[ ĠB IB REF 30 Ġ] Ġaiming Ġat Ġevaluation Ġof Ġcompos itional Ġdistribution al Ġsemantic Ġmodels . ĠThe Ġdataset Ġconsists Ġof Ġ99 27 Ġsentence Ġpairs Ġin Ġa Ġ4 500 / 500 / 49 27 Ġtrain / dev / test Ġsplit , Ġin Ġwhich Ġeach Ġsentence Ġpairs Ġare Ġpre - defined Ġinto Ġthree Ġlabels : ĠâĢ ľ ent ail ment âĢ Ŀ , âĢ ľ cont rad iction âĢ Ŀ Ġand ĠâĢ ľ neutral âĢ Ŀ . </s> Our Ġresults Ġare Ġsummarized Ġin ĠTable Ġ4 Ġ, Ġwhere Ġthe Ġperformance Ġof ĠNB OW , ĠL ST M , ĠRec NN , Ġand ĠR NT N Ġare Ġreported Ġby ĠB IB REF 31 Ġ, ĠB IB REF 32 Ġ. ĠFor Ġfair Ġcomparison , Ġwe Ġtrain Ġour Ġmodels Ġwith Ġthe Ġsame Ġsetting . ĠWe Ġcan Ġsee Ġboth ĠDC - Rec NN Ġand ĠDC - Tree L ST M Ġoutper form Ġcompetitor Ġmodels , Ġin Ġwhich ĠDC - Rec NN Ġ( DC - Tree L ST M ) Ġachieves Ġ3 % Ġ( 2 . 7 %) Ġimprovements Ġthan ĠRec NN Ġ( Tree L ST M ). ĠWe Ġthink Ġthis Ġbreakthrough Ġis Ġbasically Ġattributed Ġto Ġthe Ġdynamic Ġcompos itional Ġmechanism , Ġwhich Ġenables Ġour Ġmodels Ġto Ġcapture Ġvarious Ġsynt actic Ġpatterns Ġ( As Ġwe Ġwill Ġdiscuss Ġlater ) Ġtherefore Ġcan Ġmore Ġaccurately Ġunderstand Ġsentences . </s> Exper iment </s> To Ġmake Ġa Ġcomprehensive Ġevaluation , Ġwe Ġassess Ġour Ġmodel Ġon Ġfive Ġtext Ġclassification Ġtasks Ġand Ġa Ġsemantic Ġmatching Ġtask . </s> Training Ġand ĠHyper param eters </s> Given Ġa Ġsentence Ġ( or Ġsentence Ġpair ) Ġand Ġits Ġlabel , Ġthe Ġoutput Ġof Ġa Ġneural Ġnetwork Ġis Ġthe Ġprobabilities Ġof Ġthe Ġdifferent Ġclasses . ĠThe Ġparameters Ġof Ġthe Ġnetwork Ġare Ġtrained Ġto Ġminim ise Ġthe Ġcross - ent ropy Ġof Ġthe Ġpredicted Ġand Ġtrue Ġlabel Ġdistributions . ĠTo Ġminimize Ġthe Ġobjective , Ġwe Ġuse Ġsto ch astic Ġgradient Ġdescent Ġwith Ġthe Ġdiagonal Ġvariant Ġof ĠAda Grad ĠB IB REF 20 Ġ. </s> The Ġword Ġembed d ings Ġfor Ġall Ġof Ġthe Ġmodels Ġare Ġinitialized Ġwith ĠGl o Ve Ġvectors ĠB IB REF 21 Ġ. ĠThe Ġother Ġparameters Ġare Ġinitialized Ġby Ġrandomly Ġsampling Ġfrom Ġuniform Ġdistribution Ġin Ġ$ [ - 0 . 1 , Ġ0 . 1 ] $ Ġ. </s> The Ġfinal Ġhyper - param eters Ġare Ġas Ġfollows . ĠThe Ġinitial Ġlearning Ġrate Ġis Ġ$ 0 . 1 $ Ġ. ĠThe Ġregular ization Ġweight Ġof Ġthe Ġparameters Ġis Ġ$ 1 E { - 5 } $ Ġand Ġthe Ġothers Ġare Ġlisted Ġas ĠTable Ġ1 Ġ. </s> For Ġall Ġthe Ġsentences Ġin Ġthe Ġdatasets , Ġwe Ġparse Ġthem Ġwith Ġconstituency Ġparser ĠB IB REF 22 Ġto Ġobtain Ġthe Ġtrees Ġfor Ġour Ġmodels Ġand Ġsome Ġcompetitor Ġmodels . </s> Comp et itor ĠMethods </s> Rec NN ĠB IB REF 11 Ġ: ĠRec ursive Ġneural Ġnetwork Ġwith Ġstandard Ġcompos itional Ġfunction . </s> R NT N ĠB IB REF 23 Ġ: ĠThe ĠR NT N Ġis Ġa Ġrecursive Ġneural Ġnetwork Ġwith Ġneural Ġtens or Ġlayer , Ġwhich Ġcan Ġmodel Ġstrong Ġinteractions Ġbetween Ġtwo Ġconstituents . </s> M V - Rec NN ĠB IB REF 11 Ġ: ĠThe ĠMV - Rec NN Ġis Ġto Ġrepresent Ġevery Ġword Ġand Ġlonger Ġphrase Ġin Ġa Ġparse Ġtree Ġas Ġboth Ġa Ġvector Ġand Ġa Ġmatrix Ġin Ġorder Ġto Ġmodel Ġrich Ġcomposition ality . </s> Tree L ST M ĠB IB REF 9 Ġ: ĠRec ursive Ġneural Ġnetwork Ġwith ĠLong ĠShort - Term ĠMemory Ġunit . </s> Discussion Ġand ĠQual itative ĠAnalysis </s> In Ġour Ġmodels , Ġthe Ġlatent Ġvector Ġ$\ math bf Ġ{ z } $ Ġcontrols Ġthe Ġprocess Ġof Ġpredicting Ġnetwork 's Ġparameters Ġand Ġits Ġdimension ality Ġdetermines Ġthe Ġnumber Ġof Ġmodel 's Ġparameters . ĠNext , Ġwe Ġwill Ġinvestigate Ġhow Ġthe Ġcontrolling Ġvector Ġ$\ math bf Ġ{ z } $ Ġinfluences Ġthe Ġperformance Ġof Ġour Ġmodels . </s> Figure Ġ4 Ġshows Ġthe Ġaccur acies Ġof ĠDC - Rec NN Ġacross Ġthe Ġdifferent Ġdimensions Ġof Ġ$ [ 5 , Ġ10 , Ġ\ d ots Ġ, Ġ50 ] $ Ġfor Ġthe Ġcontrolling Ġvector Ġ$\ math bf Ġ{ z } $ Ġon Ġfive Ġdatasets . ĠWe Ġget Ġthe Ġfollowing Ġfindings : </s> For Ġall Ġfive Ġdatasets , Ġthe Ġmodel Ġcan Ġachieve Ġconsiderable Ġperformances Ġeven Ġwhen Ġthe Ġsize Ġof Ġvector Ġ$\ math bf Ġ{ z } $ Ġis Ġreduced Ġto Ġ5 . ĠParticularly , Ġfor Ġthe Ġdataset ĠQC , Ġthe Ġmodel Ġob tains Ġ$ 87 . 0 \ % $ Ġaccuracy Ġwith Ġa Ġpretty Ġsmall Ġmeta ĠTree - Rec NN , Ġsuggesting Ġa Ġsmaller Ġmeta Ġnetwork Ġcan Ġbe Ġused Ġfor Ġgenerating Ġa Ġmore Ġpowerful Ġcompos itional Ġfunction Ġto Ġeffectively Ġmodel Ġsentence . </s> When Ġdealing Ġwith Ġthe Ġdataset Ġwith Ġmore Ġlabels , Ġlarger Ġvector Ġsize Ġleads Ġto Ġa Ġbetter Ġperformance . ĠFor Ġexample , Ġthe Ġperformance Ġon ĠIE Ġand ĠQC Ġdatasets Ġreaches Ġthe Ġmaximum Ġwhen Ġthe Ġsize Ġof Ġ$ z $ Ġequals Ġ40 , Ġwhile Ġfor Ġthe Ġother Ġthree Ġdatasets ĠMR , ĠS ST Ġand ĠSUB J , Ġthe Ġmodel Ġob tains Ġthe Ġbest Ġperformance Ġwith Ġthe Ġvalue Ġof Ġ30 , Ġ30 Ġand Ġ20 Ġrespectively . </s> As Ġdescribed Ġin Ġprevious Ġsections , Ġwe Ġknow Ġthe Ġcompos itional Ġfunction Ġis Ġchanged Ġcross Ġchild Ġnodes Ġover Ġa Ġtree , Ġwhich Ġis Ġcontrolled Ġby Ġa Ġlatent Ġvector Ġ$ z $ Ġ. ĠTo Ġget Ġan Ġintuitive Ġunderstanding Ġof Ġhow Ġthe Ġcontrolling Ġvector Ġ$ z $ Ġworks , Ġwe Ġdesign Ġan Ġexperiment Ġto Ġexamine Ġthe Ġneuron 's Ġbehaviours Ġof Ġ$\ math bf Ġ{ z } $ Ġon Ġeach Ġnode . ĠMore Ġconcrete ly , Ġwe Ġrefer Ġto Ġ$ z _{ j k } $ Ġas Ġthe Ġactivation Ġof Ġthe Ġ$ k $ Ġ- ne uron Ġat Ġnode Ġ$ j $ Ġ, Ġwhere Ġ$ j Ġ\ in Ġ\ l brace Ġ1 , \ ld ots Ġ, N \ r brace Ġ$ Ġand Ġ$ k Ġ\ in Ġ\ l brace Ġ1 , \ ld ots Ġ, Ġz \ r brace Ġ$ Ġ. ĠThen Ġwe Ġrandomly Ġsample Ġsome Ġsentences Ġon Ġthe Ġdevelopment Ġset Ġfrom Ġthe Ġdatasets Ġwe Ġused . ĠBy Ġvisual izing Ġthe Ġlatent Ġvector Ġ$\ math bf Ġ{ z } _{ j } $ Ġand Ġanalyzing Ġthe Ġmaximum Ġactivation , Ġwe Ġcan Ġfind Ġwhat Ġkinds Ġof Ġpatterns Ġthe Ġcurrent Ġneuron Ġfocuses Ġon . </s> Table Ġ5 Ġillustrates Ġmultiple Ġinterpret able Ġneurons Ġand Ġsome Ġrepresentative Ġwords Ġor Ġphrases Ġwhich Ġcan Ġactivate Ġthese Ġneurons . ĠWe Ġcan Ġobserve Ġthat : </s> For Ġsome Ġsimple Ġtasks Ġsuch Ġas Ġtext Ġclassification , Ġmeta Ġnetwork Ġwill Ġintegrate Ġuseful Ġsemantic Ġinformation Ġinto Ġthe Ġthe Ġgeneration Ġprocess Ġof Ġcompos itional Ġfunction . ĠThese Ġsemantic Ġbias Ġbefore Ġcomposition Ġare Ġtask - specific . </s> For Ġexample , Ġthe Ġ21 - Ġ$ st $ Ġneuron Ġis Ġmore Ġsensitive Ġto Ġemotional Ġterms , Ġwhich Ġcan Ġbe Ġunderstood Ġas Ġa Ġsent inel , Ġtelling Ġthe Ġbasic Ġneural Ġnetwork Ġthat Ġan Ġinformative Ġphrase Ġis Ġcoming , Ġmore Ġattention Ġshould Ġbe Ġpaid Ġin Ġthe Ġprocess Ġof Ġcomposition . ĠFigure Ġ5 Ġ- ( a ) Ġshows Ġa Ġvisualization . ĠWe Ġcan Ġsee Ġin Ġthis Ġsentence , Ġthe Ġneuron Ġhas Ġrealized Ġthat Ġthis Ġidi omatic Ġcoll ocation ĠâĢ ľ in Ġstitches âĢ Ŀ Ġis Ġa Ġkey Ġpattern , Ġwhich Ġis Ġcrucial Ġfor Ġthe Ġfinal Ġsentiment Ġprediction . </s> For Ġmore Ġcomplicated Ġtasks Ġsuch Ġas Ġsemantic Ġmatching , Ġa Ġwell - ground ed Ġunderstanding Ġof Ġthe Ġsynt actic Ġstructure Ġis Ġcrucial . ĠIn Ġthis Ġcontext , Ġwe Ġfind Ġthat Ġa Ġmeta Ġnetwork Ġcould Ġcapture Ġsome Ġsynt actic Ġinformation . ĠFor Ġexample , Ġthe Ġ27 - Ġ$ th $ Ġneuron Ġmonitors Ġphrases Ġconstructed Ġby Ġlight - verb . ĠAs Ġshown Ġin ĠFigure Ġ5 Ġ- ( b ), Ġthe Ġverb Ġphrase ĠâĢ ľ taking Ġoff âĢ Ŀ Ġhas Ġbeen Ġattended Ġfor Ġforthcoming Ġcompos itional Ġoperation , Ġwhich Ġis Ġmore Ġuseful Ġfor Ġjudging Ġthe Ġsemantic Ġrelation Ġbetween Ġthe Ġsentence Ġpair ĠâĢ ľ An Ġairplane Ġis Ġtaking Ġoff / A Ġplane Ġis Ġlanding âĢ Ŀ . </s> Related ĠWork </s> One Ġthread Ġof Ġrelated Ġwork Ġis Ġthe Ġexploration Ġof Ġdifferent Ġkinds Ġof Ġcompos itional Ġfunction Ġover Ġtree Ġstructures . ĠB IB REF 11 Ġ[ ĠB IB REF 11 Ġ] Ġproposed Ġthe Ġrecursive Ġneural Ġnetwork Ġwith Ġstandard Ġcompos itional Ġfunction . ĠAfter Ġthat , Ġsome Ġextensions Ġare Ġintroduced Ġto Ġenhance Ġthe Ġexpressive Ġpower Ġof Ġcompos itional Ġfunction , Ġsuch Ġas ĠMV - Rec NN ĠB IB REF 23 Ġ, ĠSU - R NN ĠB IB REF 13 Ġ, ĠR NT N ĠB IB REF 23 Ġ, Ġwhile Ġthese Ġmodels Ġsuffer Ġfrom Ġthe Ġproblem Ġof Ġhard - coded Ġcompos itional Ġoperations Ġand Ġover fitting . </s> Another Ġthread Ġof Ġwork Ġis Ġthe Ġidea Ġof Ġusing Ġone Ġnetwork Ġto Ġdirect Ġthe Ġlearning Ġof Ġanother Ġnetwork ĠB IB REF 16 Ġ. ĠB IB REF 33 Ġ[ ĠB IB REF 33 Ġ] Ġintroduce Ġa Ġmeta Ġneural Ġnetwork Ġto Ġprovide Ġanother Ġnetwork Ġwith Ġa Ġstep Ġsize Ġand Ġa Ġdirection Ġvector , Ġwhich Ġis Ġhelpful Ġfor Ġparameter Ġoptimization . ĠB IB REF 16 Ġ[ ĠB IB REF 16 Ġ] Ġpropose Ġthe Ġdynamic Ġfilter Ġnetwork Ġto Ġimplicitly Ġlearn Ġa Ġvariety Ġof Ġfiltering Ġoperations . ĠB IB REF 17 Ġ[ ĠB IB REF 17 Ġ] Ġintroduce Ġa Ġlearn et Ġfor Ġone - shot Ġlearning , Ġwhich Ġcan Ġpredict Ġthe Ġparameters Ġof Ġa Ġsecond Ġnetwork Ġgiven Ġa Ġsingle Ġexempl ar . ĠB IB REF 18 Ġ[ ĠB IB REF 18 Ġ] Ġpropose Ġthe Ġmodel Ġhyper network , Ġwhich Ġuses Ġa Ġsmall Ġnetwork Ġto Ġgenerate Ġthe Ġweights Ġfor Ġa Ġlarger Ġnetwork . </s> Different Ġfrom Ġthese Ġmodels , Ġwe Ġemploy Ġthe Ġidea Ġof Ġparameter Ġgeneration Ġto Ġaddress Ġthe Ġlimitation Ġof Ġweight - sharing Ġor Ġpartially Ġsharing Ġparadigm Ġof Ġtree - based Ġcompos itional Ġmodels . </s> Conclusion </s> In Ġthis Ġwork , Ġwe Ġintroduce Ġa Ġmeta Ġneural Ġnetwork , Ġwhich Ġcan Ġgenerate Ġa Ġcompos itional Ġnetwork Ġto Ġdynamically Ġcompose Ġconstituents Ġover Ġtree Ġstructure . ĠThe Ġparameters Ġof Ġcompos itional Ġfunction Ġvary Ġfrom Ġposition Ġto Ġposition Ġand Ġfrom Ġsample Ġto Ġsample , Ġallowing Ġfor Ġmore Ġsophisticated Ġoperations Ġon Ġthe Ġinput . </s> To Ġevaluate Ġour Ġmodels , Ġwe Ġchoose Ġtwo Ġtypical ĠN LP Ġtasks Ġinvolving Ġsix Ġdatasets . ĠThe Ġqualitative Ġand Ġquantitative Ġexperiment Ġresults Ġdemonstrate Ġthe Ġeffectiveness Ġof Ġour Ġmodels . </s> Acknowled gments </s> We Ġwould Ġlike Ġto Ġthank Ġthe Ġanonymous Ġreviewers Ġfor Ġtheir Ġvaluable Ġcomments Ġand Ġthank ĠKai yu ĠQian , ĠJ iac hen ĠXu , ĠJ if an ĠChen Ġfor Ġuseful Ġdiscussions . ĠThis Ġwork Ġwas Ġpartially Ġfunded Ġby ĠNational ĠNatural ĠScience ĠFoundation Ġof ĠChina Ġ( No . Ġ6 153 2011 Ġand Ġ6 16 72 162 ), ĠShanghai ĠMunicipal ĠScience Ġand ĠTechnology ĠCommission Ġ( No . Ġ16 JC 14 204 01 ).
Introduction Learning the distributed representation for long spans of text from its constituents has been a key step for various natural language processing (NLP) tasks, such as text classification BIBREF0 , BIBREF1 , semantic matching BIBREF2 , BIBREF3 , and machine translation BIBREF4 . Existing deep learning approaches take a compositional function with different forms to compose word vectors recursively until obtaining a sentential representation. Typically, these compositional functions involve recurrent neural networks BIBREF5 , BIBREF6 , convolutional neural networks BIBREF7 , BIBREF8 , and tree-structured neural networks BIBREF9 , BIBREF10 . Among these methods, tree-structured neural networks (Tree-NNs) show theirs superior performance in many NLP tasks BIBREF11 , BIBREF12 . Following the syntactic tree structure, Tree-NNs assign a fixed-length vector to each word at the leaves of the tree, and combine word and phrase pairs recursively to create intermediate node vectors, eventually obtaining one final vector to represent the whole sentence. However, these models have a major limitation in their inability to fully capture the richness of compositionality BIBREF13 . The same parameters are used for all kinds of semantic compositions, even though the compositions have different characteristics in nature. For example, the composition of the adjective and the noun differs significantly from the composition of the verb and the noun. Moreover, many semantic phenomena, such as semantic idiomaticity or transparency, call for more powerful compositional mechanisms BIBREF14 . Therefore, Tree-NNs suffer from the underfitting problem. To alleviate this problem, some researchers propose to use multiple compositional functions, which are arranged beforehand according to some partition criterion BIBREF11 , BIBREF13 , BIBREF15 . Intuitively, using different parameters for different types of compositions has the potential to greatly reduce underfitting. BIBREF13 [ BIBREF13 ] defined different compositional functions in terms of syntactic categories, and a suitable compositional function is selected based on the syntactic categories. BIBREF15 [ BIBREF15 ] introduced multiple compositional functions and during compositional phase, a proper one is selected based on the input information. Although these models accomplished their mission to a certain extent, they still suffer from the following three challenges. First, the predefined compositional functions cannot cover all the compositional rules; Second, they require more learnable parameters, suffering from the problem of overfitting; Third, it is difficult to determine a universal criterion for semantic composition based solely on syntactic categories. In this paper, we propose dynamic compositional neural networks over tree structure, in which a meta network is used to generate the context-specific parameters of a dynamic compositional network. Specifically, we construct our models based on two kinds of tree-structured neural networks: recursive neural network (Tree-RecNN) BIBREF11 and tree-structure long short-term memory neural network (Tree-LSTM) BIBREF9 . Our work is inspired by recent work on dynamic parameter prediction BIBREF16 , BIBREF17 , BIBREF18 . The meta network is used to extract the shared meta-knowledge across different compositional rules and to dynamically generate the context-specific compositional function. Thus, the compositional function of our models varies with positions, contexts and samples. The dynamic compositional network then applies those context-specific parameters to the current input information. Both meta and dynamic networks are differentiable such that the overall networks can be trained in an end-to-end fashion. Additional, to reduce the complexity of the whole networks, we define the dynamic weight matrix in a manner simulating low-rank matrix decomposition. We evaluate our models on two typical tasks: text classification and text semantic matching. The results show that our models are more expressive due to their learning to learn nature, yet without increasing the number of model's parameters. Moreover, we find certain composition operations can be learned implicitly by meta TreeNN, such as the composition of noun phrases and verb phrases. The contributions of the paper can be summed up as follows. Tree-Structured Neural Network In this section, we briefly describe the tree-structured neural networks. The idea of tree-structured neural networks for natural language processing (NLP) is to train a deep learning model with a grammatical tree structure BIBREF19 that can be applied to phrases and sentences. At every node in the tree, the contexts of the left and right children are combined by a compositional function. The parameters of the compositional function are shared across all nodes in the tree. The layer computed at the top node gives a representation for the whole sentence. Vanilla Recursive Neural Network The simplest member of tree-structured NN is the vanilla recursive neural network BIBREF13 , in which the representation of parent is calculated by weighted linear combination of the child vectors. Formally, given a binary constituency tree $T$ induced by a sentence, each non-leaf node corresponds to a phrase. We refer to $\mathbf {h}_{j} \in \mathbb {R}^{d}$ as the hidden state of each node $j$ , and let $\mathbf {h}_{j}^{l}$ , $\mathbf {h}_{j}^{r}$ denote the left and right child representations respectively. $$\mathbf {h}_{j} = \tanh \left( \mathbf {W} \begin{bmatrix} \mathbf {h}_{j}^{l} \\ \mathbf {h}_{j}^{r} \\ \end{bmatrix} + \mathbf {b} \right),$$   (Eq. 5) where $\mathbf {W} \in \mathbb {R}^{d \times 2d}$ is a learnable compositional matrix, $\mathbf {b}$ is the bias vector. Tree LSTM Tree LSTM BIBREF9 is a generalization of LSTMs to tree-structured network topologies. In this model, the compositional function is an LSTM unit, and the hidden state $h_j$ of each node can be computed as follows: we refer to $\mathbf {h}_{j}$ and $\mathbf {c}_{j}$ as the hidden state and memory cell of each node $j$ . The transition equations of node $j$ are as follows: $$\begin{bmatrix} \mathbf {\tilde{c}}_{j} \\ \mathbf {o}_{j} \\ \mathbf {i}_{j} \\ \mathbf {f}_{j}^{l} \\ \mathbf {f}_{j}^{r} \end{bmatrix} &= \begin{bmatrix} \tanh \\ \sigma \\ \sigma \\ \sigma \\ \sigma \end{bmatrix} \left( \mathbf {W} \begin{bmatrix} \mathbf {x}_{j} \\ \mathbf {h}_{j}^{l} \\ \mathbf {h}_{j}^{r} \\ \end{bmatrix} + \mathbf {b} \right),\\ \mathbf {c}_{j} &= \mathbf {\tilde{c}}_{j} \odot \mathbf {i}_{j} + \mathbf {c}_{j}^{l} \odot \mathbf {f}_{j}^{l}+ \mathbf {c}_{j}^{r} \odot \mathbf {f}_{j}^{r} , \\ \mathbf {h}_{j} &= \mathbf {o}_{j} \odot \tanh \left( \mathbf {c}_{j} \right),$$   (Eq. 8) where $\mathbf {x}_j \in \mathbb {R}^{d}$ denotes the input vector and is non-zero if and only if it is a leaf node. The superscript $l$ and $r$ represent the left child and right child respectively. $\sigma $ represents the logistic sigmoid function and $\odot $ denotes element-wise multiplication. $\mathbf {W} \in \mathbb {R}^{5d \times 3d}$ and $\mathbf {b} \in \mathbb {R}^{5d}$ are learnable parameters. Dynamic Compositional Neural Network In the above two tree-structured NNs, the compositional function is shared across all nodes in the tree, which results in underfitting since the semantic compositions have great diversities. To address this problem, we propose two dynamic compositional neural networks over tree structure, which dynamically generate different parameters for different types of compositions. Figure 2 shows an illustration of the dynamic compositional neural network, consisting of two components: (1) meta network and (2) basic network with dynamic parameters. Specifically, we propose two meta networks to generate the context-specific compositional functions for RecNN and TreeLSTM respectively. Meta Network for RecNN For RecNN, we replace the static parameters $\mathbf {W}$ and $\mathbf {b}$ in Eq.( 5 ) with the dynamic parameters $\mathbf {W}(\mathbf {z}_{j})$ and $\mathbf {b}(\mathbf {z}_{j})$ , which are generated by a meta network. The meta network is a smaller RecNN, and the hidden state $\hat{\mathbf {h}}_{j}\in \mathbb {R}^{m}$ of node $j$ in meta network is defined as $$\hat{\mathbf {h}}_{j} &= \tanh \left( \mathbf {W}_{m} \mathcal {H}_{j} + \mathbf {b}_{m} \right),\\ \mathbf {z}_{j} &= \mathbf {W}_{z} \hat{\mathbf {h}}_{j}$$   (Eq. 11) where $\mathcal {H}_{j} = \mathbf {h}_{j}^{l}\oplus \mathbf {h}_{j}^{r}\oplus \mathbf {\hat{h}}_{j}^{l}\oplus \mathbf {\hat{h}}_{j}^{r} \in \mathbb {R}^{2m+2d}$ , $\mathbf {W}_{m} \in \mathbb {R}^{m \times (2d+2m)}$ and $\mathbf {b}_{m} \in \mathbb {R}^{m}$ are parameters of meta RecNN; $\mathbf {W}_{z} \in \mathbb {R}^{z \times m}$ is a scale matrix. To reduce the number of the parameters, we define the dynamic parameters with a low-rank factorized representation of the weights, analogous to the Singular Value Decomposition. The dynamic parameters $\mathbf {W}(\mathbf {z}_{j})$ and $\mathbf {b}(\mathbf {z}_{j})$ of the basic RecNN are computed by: $$\mathbf {W}(\mathbf {z}_{j}) &= \begin{bmatrix} P_l \mathbf {D}(\mathbf {z}_j)Q_l \\ P_r \mathbf {D}(\mathbf {z}_j)Q_r \\ \end{bmatrix} \\ \mathbf {b}(\mathbf {z}_{j}) &= \begin{bmatrix} B_l\mathbf {z}_j \\ B_r\mathbf {z}_j \\ \end{bmatrix}$$   (Eq. 12) where $P \in \mathbb {R}^{d \times z}$ , $Q \in \mathbb {R}^{z \times d}$ , and $\mathbf {D}(\mathbf {z}_t) \in \mathbb {R}^{z \times z}$ is the diagonal matrix of $\mathbf {z}$ . Thus, our dynamic RecNN needs $(6dz + mz)$ parameters, while the vanilla RecNN has $(2d^2+d)$ parameters. With a small $z$ and $m$ , our dynamic RecNN needs less parameters than the vanilla RecNN. For example, if we set $d =100$ and $z=m=20$ , our model needs $12,400$ parameters while the vanilla model needs $20,100$ parameters. Meta Network for TreeLSTM Likewise, we also use a smaller meta network to generate the static parameters $\mathbf {W}$ and $\mathbf {b}$ in Eq.( 8 ) with the dynamic parameters $\mathbf {W}(\mathbf {z}_{j})$ and $\mathbf {b}(\mathbf {z}_{j})$ . The meta network is a smaller TreeLSTM, and the hidden state $\hat{\mathbf {h}}_{j}\in \mathbb {R}^{m}$ of node $j$ in meta network is defined as $$\begin{bmatrix} \mathbf {\hat{g}}_{j} \\ \mathbf {\hat{o}}_{j} \\ \mathbf {\hat{i}}_{j} \\ \mathbf {\hat{f}}_{j}^{l} \\ \mathbf {\hat{f}}_{j}^{r} \end{bmatrix} &= \begin{bmatrix} \tanh \\ \sigma \\ \sigma \\ \sigma \\ \sigma \end{bmatrix} \left( \mathbf {W}_{m} \begin{bmatrix} \mathbf {x}_{j} \\ \mathcal {H}_{j} \\ \end{bmatrix} + \mathbf {b}_{m} \right) ,\\ \mathbf {\hat{c}}_{j} &= \mathbf {\hat{g}}_{j} \odot \mathbf {\hat{i}}_{j} + \mathbf {\hat{g}}_{j}^{l} \odot \mathbf {\hat{f}}_{j}^{l}+ \mathbf {\hat{g}}_{j}^{r} \odot \mathbf {\hat{f}}_{j}^{r} , \\ \mathbf {\hat{h}}_{j} &= \mathbf {\hat{o}}_{j} \odot \tanh \left( \mathbf {\hat{c}}_{j} \right), \\ \mathbf {z}_{j} &= \mathbf {W}_{z} \hat{\mathbf {h}}_{j}$$   (Eq. 14) where $\mathcal {H}_{j} = \mathbf {h}_{j}^{l}\oplus \mathbf {h}_{j}^{r}\oplus \mathbf {\hat{h}}_{j}^{l}\oplus \mathbf {\hat{h}}_{j}^{r} \in \mathbb {R}^{2m+2d}$ ; $\mathbf {W}_{m} \in \mathbb {R}^{5m \times (3d+2m)}$ and $\mathbf {b}_{m} \in \mathbb {R}^{m}$ are parameters of meta TreeLSTM; $\mathbf {W}_{z} \in \mathbb {R}^{z \times m}$ is a scale matrix. The dynamic parameters $\mathbf {W}(\mathbf {z}_{j})$ and $\mathbf {b}(\mathbf {z}_{j})$ of basic TreeLSTM are computed by: $$\mathbf {W}(\mathbf {z}_{j}) &= \left[\mathbf {W}^{g}, \mathbf {W}^{i}, \mathbf {W}^{f^l},\mathbf {W}^{f^r},\mathbf {W}^{o}\right] \\ \mathbf {b}(\mathbf {z}_{j}) &= \left[\mathbf {b}^{g}, \mathbf {b}^{i}, \mathbf {b}^{f^l},\mathbf {b}^{f^r},\mathbf {b}^{o}\right],$$   (Eq. 15) where for $*\in \lbrace c,o,i,f^l,f^r\rbrace $ , $$\mathbf {W}^{*}(\mathbf {z}_{j}) &= \begin{bmatrix} P_x^{*} \mathbf {D}(\mathbf {z}_t)Q_x^{*} \\ P_l^{*} \mathbf {D}(\mathbf {z}_t)Q_l^{*} \\ P_r^{*} \mathbf {D}(\mathbf {z}_t)Q_r^{*} \\ \end{bmatrix}, \\ \mathbf {b}^{*}(\mathbf {z}_{j}) &= \begin{bmatrix} B_x^{*}\mathbf {z}_t \\ B_l^{*}\mathbf {z}_t \\ B_r^{*}\mathbf {z}_t \\ \end{bmatrix},$$   (Eq. 16) where $P \in \mathbb {R}^{5d \times z}$ , $Q \in \mathbb {R}^{z \times 3d}$ , $B \in \mathbb {R}^{5d \times z}$ , and $\mathbf {D}(\mathbf {z}_t) \in \mathbb {R}^{z \times z}$ is the diagonal matrix of $\mathbf {z}$ . With a small $z$ and $m$ , our dynamic TreeLSTM needs a similar amount of parameters compared to the standard TreeLSTM. Application of Dynamic Compositional Neural Networks In this section, we describe two specific models to show the applications of dynamic compositional neural networks for two typical tasks in NLP. Text Classification The purpose of text classification is that, given a sentence $x$ , the model should predict labels $\hat{y}$ from a pre-defined label set $\mathcal {Y}$ . From the description in the previous section, we can compute the distributed representation $\mathbf {h}_j$ of the phrase at node $j$ of a tree: $$\mathbf {h}_{j} &= \operatornamewithlimits{DC-TreeNN}(\mathbf {x}_j, \mathbf {h}_{j}^l, \mathbf {h}_{j}^r,\theta )$$   (Eq. 18) After this recursive process, the hidden state $\mathbf {h}_R$ at the root node is used as the sentential representation, which then followed by a softmax classifier to predict the probability distribution over classes. $${\hat{\mathbf {y}}} = \operatornamewithlimits{softmax}(\mathbf {W}_t \mathbf {h}_R + \mathbf {b}_t)$$   (Eq. 19) where ${\hat{\mathbf {y}}}$ is prediction probabilities, $\mathbf {W}_t$ and $\mathbf {b}_t$ are the parameters of the classifier. We evaluate our models on five different datasets. The detailed statistics about the five datasets are listed in Table 2 . Each dataset is briefly described as follows. SST The movie reviews with two classes (negative, positive) in the Stanford Sentiment Treebank BIBREF23 . MR The movie reviews with two classes BIBREF24 . QC The TREC questions dataset involves six different question types. BIBREF25 . SUBJ Subjectivity dataset where the goal is to classify each instance (snippet) as being subjective or objective. BIBREF26 IE Idiom enhanced sentiment classification. BIBREF27 . Each sentence contains at least one idiom. As shown in Table 3 , DC-TreeLSTM consistently outperforms RecNN, MV-RecNN, RNTN, and TreeLSTM by a large margin while achieving comparable results to the CNN and using much fewer parameters.(The number of parameters in our models is approximately 10K while in CNN the number of parameters is about 400K). Compared with RecNN, DC-RecNN performs better, indicating the effectiveness of the dynamic compositional function. Additionally, both DC-RecNN and DC-TreeLSTM achieve substantial improvement on IE dataset, which covers the richness of compositionality (idiomaticity). We attribute the success on IE to its power in modeling more complicated compositionality. Text Semantic Matching Among many natural language processing (NLP) tasks, a common problem is modelling the relevance of a pair of texts. In this section, we show how to effectively use the dynamic compositional neural networks to model the semantic relationship between two sentences. As shown in Figure 1 , given two sentences $x_a$ and $x_b$ , the representation of each sentence $\mathbf {h}_{R}$ can be computed by one basic TreeNN. $$\mathbf {h}_{R}^{(a)} &= \operatornamewithlimits{DC-TreeNN}(\mathbf {x}_R, \mathbf {h}_{R}^l, \mathbf {h}_{R}^r,\theta _{a}) \\ \mathbf {h}_{R}^{(b)} &= \operatornamewithlimits{DC-TreeNN}(\mathbf {x}_R, \mathbf {h}_{R}^l, \mathbf {h}_{R}^r,\theta _{b})$$   (Eq. 21) where $R$ denotes the root node of a tree. $\theta _{a}$ and $\theta _{b}$ are generated by a shared meta TreeNN. Then, the representation of each sentence will be fed into a multi-layer perceptron to obtain a unified representation for the final relationship classification. The sample-specific but shared meta TreeNN ensures that, on the one hand we can dynamically model the diversity of semantic compositionality, on the other hand we can capture the general rules across different samples. We choose the dataset of Sentences Involving Compositional Knowledge (SICK), which is proposed by BIBREF30 [ BIBREF30 ] aiming at evaluation of compositional distributional semantic models. The dataset consists of 9927 sentence pairs in a 4500/500/4927 train/dev/test split, in which each sentence pairs are pre-defined into three labels: “entailment”,“contradiction” and “neutral”. Our results are summarized in Table 4 , where the performance of NBOW, LSTM, RecNN, and RNTN are reported by BIBREF31 , BIBREF32 . For fair comparison, we train our models with the same setting. We can see both DC-RecNN and DC-TreeLSTM outperform competitor models, in which DC-RecNN (DC-TreeLSTM) achieves 3% (2.7%) improvements than RecNN (TreeLSTM). We think this breakthrough is basically attributed to the dynamic compositional mechanism, which enables our models to capture various syntactic patterns (As we will discuss later) therefore can more accurately understand sentences. Experiment To make a comprehensive evaluation, we assess our model on five text classification tasks and a semantic matching task. Training and Hyperparameters Given a sentence (or sentence pair) and its label, the output of a neural network is the probabilities of the different classes. The parameters of the network are trained to minimise the cross-entropy of the predicted and true label distributions. To minimize the objective, we use stochastic gradient descent with the diagonal variant of AdaGrad BIBREF20 . The word embeddings for all of the models are initialized with GloVe vectors BIBREF21 . The other parameters are initialized by randomly sampling from uniform distribution in $[-0.1, 0.1]$ . The final hyper-parameters are as follows. The initial learning rate is $0.1$ . The regularization weight of the parameters is $1E{-5}$ and the others are listed as Table 1 . For all the sentences in the datasets, we parse them with constituency parser BIBREF22 to obtain the trees for our models and some competitor models. Competitor Methods RecNN BIBREF11 : Recursive neural network with standard compositional function. RNTN BIBREF23 : The RNTN is a recursive neural network with neural tensor layer, which can model strong interactions between two constituents. MV-RecNN BIBREF11 : The MV-RecNN is to represent every word and longer phrase in a parse tree as both a vector and a matrix in order to model rich compositionality. TreeLSTM BIBREF9 : Recursive neural network with Long Short-Term Memory unit. Discussion and Qualitative Analysis In our models, the latent vector $\mathbf {z}$ controls the process of predicting network's parameters and its dimensionality determines the number of model's parameters. Next, we will investigate how the controlling vector $\mathbf {z}$ influences the performance of our models. Figure 4 shows the accuracies of DC-RecNN across the different dimensions of $[5, 10, \dots , 50]$ for the controlling vector $\mathbf {z}$ on five datasets. We get the following findings: For all five datasets, the model can achieve considerable performances even when the size of vector $\mathbf {z}$ is reduced to 5. Particularly, for the dataset QC, the model obtains $87.0\%$ accuracy with a pretty small meta Tree-RecNN, suggesting a smaller meta network can be used for generating a more powerful compositional function to effectively model sentence. When dealing with the dataset with more labels, larger vector size leads to a better performance. For example, the performance on IE and QC datasets reaches the maximum when the size of $z$ equals 40, while for the other three datasets MR, SST and SUBJ, the model obtains the best performance with the value of 30, 30 and 20 respectively. As described in previous sections, we know the compositional function is changed cross child nodes over a tree, which is controlled by a latent vector $z$ . To get an intuitive understanding of how the controlling vector $z$ works, we design an experiment to examine the neuron's behaviours of $\mathbf {z}$ on each node. More concretely, we refer to $z_{jk}$ as the activation of the $k$ -neuron at node $j$ , where $j \in \lbrace 1,\ldots ,N\rbrace $ and $k \in \lbrace 1,\ldots , z\rbrace $ . Then we randomly sample some sentences on the development set from the datasets we used. By visualizing the latent vector $\mathbf {z}_{j}$ and analyzing the maximum activation, we can find what kinds of patterns the current neuron focuses on. Table 5 illustrates multiple interpretable neurons and some representative words or phrases which can activate these neurons. We can observe that: For some simple tasks such as text classification, meta network will integrate useful semantic information into the the generation process of compositional function. These semantic bias before composition are task-specific. For example, the 21- $st$ neuron is more sensitive to emotional terms, which can be understood as a sentinel, telling the basic neural network that an informative phrase is coming, more attention should be paid in the process of composition. Figure 5 -(a) shows a visualization. We can see in this sentence, the neuron has realized that this idiomatic collocation “in stitches” is a key pattern, which is crucial for the final sentiment prediction. For more complicated tasks such as semantic matching, a well-grounded understanding of the syntactic structure is crucial. In this context, we find that a meta network could capture some syntactic information. For example, the 27- $th$ neuron monitors phrases constructed by light-verb. As shown in Figure 5 -(b), the verb phrase “taking off” has been attended for forthcoming compositional operation, which is more useful for judging the semantic relation between the sentence pair “An airplane is taking off/A plane is landing”. Related Work One thread of related work is the exploration of different kinds of compositional function over tree structures. BIBREF11 [ BIBREF11 ] proposed the recursive neural network with standard compositional function. After that, some extensions are introduced to enhance the expressive power of compositional function, such as MV-RecNN BIBREF23 , SU-RNN BIBREF13 , RNTN BIBREF23 , while these models suffer from the problem of hard-coded compositional operations and overfitting. Another thread of work is the idea of using one network to direct the learning of another network BIBREF16 . BIBREF33 [ BIBREF33 ] introduce a meta neural network to provide another network with a step size and a direction vector, which is helpful for parameter optimization. BIBREF16 [ BIBREF16 ] propose the dynamic filter network to implicitly learn a variety of filtering operations. BIBREF17 [ BIBREF17 ] introduce a learnet for one-shot learning, which can predict the parameters of a second network given a single exemplar. BIBREF18 [ BIBREF18 ] propose the model hypernetwork, which uses a small network to generate the weights for a larger network. Different from these models, we employ the idea of parameter generation to address the limitation of weight-sharing or partially sharing paradigm of tree-based compositional models. Conclusion In this work, we introduce a meta neural network, which can generate a compositional network to dynamically compose constituents over tree structure. The parameters of compositional function vary from position to position and from sample to sample, allowing for more sophisticated operations on the input. To evaluate our models, we choose two typical NLP tasks involving six datasets. The qualitative and quantitative experiment results demonstrate the effectiveness of our models. Acknowledgments We would like to thank the anonymous reviewers for their valuable comments and thank Kaiyu Qian, Jiachen Xu, Jifan Chen for useful discussions. This work was partially funded by National Natural Science Foundation of China (No. 61532011 and 61672162), Shanghai Municipal Science and Technology Commission (No. 16JC1420401).
What is the meta knowledge specifically?
04bde1d2b445f971e97bb46ade2d0290981c7a32
What Ġis Ġthe Ġmeta Ġknowledge Ġspecifically ?
Un answer able
<s> What Ġis Ġthe Ġmeta Ġknowledge Ġspecifically ? </s> Introduction </s> Learning Ġthe Ġdistributed Ġrepresentation Ġfor Ġlong Ġspans Ġof Ġtext Ġfrom Ġits Ġconstituents Ġhas Ġbeen Ġa Ġkey Ġstep Ġfor Ġvarious Ġnatural Ġlanguage Ġprocessing Ġ( N LP ) Ġtasks , Ġsuch Ġas Ġtext Ġclassification ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, Ġsemantic Ġmatching ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, Ġand Ġmachine Ġtranslation ĠB IB REF 4 Ġ. ĠEx isting Ġdeep Ġlearning Ġapproaches Ġtake Ġa Ġcompos itional Ġfunction Ġwith Ġdifferent Ġforms Ġto Ġcompose Ġword Ġvectors Ġrec urs ively Ġuntil Ġobtaining Ġa Ġsent ential Ġrepresentation . ĠTypically , Ġthese Ġcompos itional Ġfunctions Ġinvolve Ġrecurrent Ġneural Ġnetworks ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, Ġconv olution al Ġneural Ġnetworks ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ, Ġand Ġtree - struct ured Ġneural Ġnetworks ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ. </s> Among Ġthese Ġmethods , Ġtree - struct ured Ġneural Ġnetworks Ġ( Tree - NN s ) Ġshow Ġtheirs Ġsuperior Ġperformance Ġin Ġmany ĠN LP Ġtasks ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ. ĠFollowing Ġthe Ġsynt actic Ġtree Ġstructure , ĠTree - NN s Ġassign Ġa Ġfixed - length Ġvector Ġto Ġeach Ġword Ġat Ġthe Ġleaves Ġof Ġthe Ġtree , Ġand Ġcombine Ġword Ġand Ġphrase Ġpairs Ġrec urs ively Ġto Ġcreate Ġintermediate Ġnode Ġvectors , Ġeventually Ġobtaining Ġone Ġfinal Ġvector Ġto Ġrepresent Ġthe Ġwhole Ġsentence . </s> However , Ġthese Ġmodels Ġhave Ġa Ġmajor Ġlimitation Ġin Ġtheir Ġinability Ġto Ġfully Ġcapture Ġthe Ġrichness Ġof Ġcomposition ality ĠB IB REF 13 Ġ. ĠThe Ġsame Ġparameters Ġare Ġused Ġfor Ġall Ġkinds Ġof Ġsemantic Ġcompositions , Ġeven Ġthough Ġthe Ġcompositions Ġhave Ġdifferent Ġcharacteristics Ġin Ġnature . ĠFor Ġexample , Ġthe Ġcomposition Ġof Ġthe Ġadjective Ġand Ġthe Ġnoun Ġdiffers Ġsignificantly Ġfrom Ġthe Ġcomposition Ġof Ġthe Ġverb Ġand Ġthe Ġnoun . ĠMoreover , Ġmany Ġsemantic Ġphenomena , Ġsuch Ġas Ġsemantic Ġidi omatic ity Ġor Ġtransparency , Ġcall Ġfor Ġmore Ġpowerful Ġcompos itional Ġmechanisms ĠB IB REF 14 Ġ. ĠTherefore , ĠTree - NN s Ġsuffer Ġfrom Ġthe Ġunder fitting Ġproblem . </s> To Ġalleviate Ġthis Ġproblem , Ġsome Ġresearchers Ġpropose Ġto Ġuse Ġmultiple Ġcompos itional Ġfunctions , Ġwhich Ġare Ġarranged Ġbeforehand Ġaccording Ġto Ġsome Ġpartition Ġcriterion ĠB IB REF 11 Ġ, ĠB IB REF 13 Ġ, ĠB IB REF 15 Ġ. ĠInt uitive ly , Ġusing Ġdifferent Ġparameters Ġfor Ġdifferent Ġtypes Ġof Ġcompositions Ġhas Ġthe Ġpotential Ġto Ġgreatly Ġreduce Ġunder fitting . ĠB IB REF 13 Ġ[ ĠB IB REF 13 Ġ] Ġdefined Ġdifferent Ġcompos itional Ġfunctions Ġin Ġterms Ġof Ġsynt actic Ġcategories , Ġand Ġa Ġsuitable Ġcompos itional Ġfunction Ġis Ġselected Ġbased Ġon Ġthe Ġsynt actic Ġcategories . ĠB IB REF 15 Ġ[ ĠB IB REF 15 Ġ] Ġintroduced Ġmultiple Ġcompos itional Ġfunctions Ġand Ġduring Ġcompos itional Ġphase , Ġa Ġproper Ġone Ġis Ġselected Ġbased Ġon Ġthe Ġinput Ġinformation . ĠAlthough Ġthese Ġmodels Ġaccomplished Ġtheir Ġmission Ġto Ġa Ġcertain Ġextent , Ġthey Ġstill Ġsuffer Ġfrom Ġthe Ġfollowing Ġthree Ġchallenges . ĠFirst , Ġthe Ġpred efined Ġcompos itional Ġfunctions Ġcannot Ġcover Ġall Ġthe Ġcompos itional Ġrules ; ĠSecond , Ġthey Ġrequire Ġmore Ġlearn able Ġparameters , Ġsuffering Ġfrom Ġthe Ġproblem Ġof Ġover fitting ; ĠThird , Ġit Ġis Ġdifficult Ġto Ġdetermine Ġa Ġuniversal Ġcriterion Ġfor Ġsemantic Ġcomposition Ġbased Ġsolely Ġon Ġsynt actic Ġcategories . </s> In Ġthis Ġpaper , Ġwe Ġpropose Ġdynamic Ġcompos itional Ġneural Ġnetworks Ġover Ġtree Ġstructure , Ġin Ġwhich Ġa Ġmeta Ġnetwork Ġis Ġused Ġto Ġgenerate Ġthe Ġcontext - specific Ġparameters Ġof Ġa Ġdynamic Ġcompos itional Ġnetwork . ĠSpecifically , Ġwe Ġconstruct Ġour Ġmodels Ġbased Ġon Ġtwo Ġkinds Ġof Ġtree - struct ured Ġneural Ġnetworks : Ġrecursive Ġneural Ġnetwork Ġ( Tree - Rec NN ) ĠB IB REF 11 Ġand Ġtree - st ructure Ġlong Ġshort - term Ġmemory Ġneural Ġnetwork Ġ( Tree - L ST M ) ĠB IB REF 9 Ġ. ĠOur Ġwork Ġis Ġinspired Ġby Ġrecent Ġwork Ġon Ġdynamic Ġparameter Ġprediction ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ. ĠThe Ġmeta Ġnetwork Ġis Ġused Ġto Ġextract Ġthe Ġshared Ġmeta - knowledge Ġacross Ġdifferent Ġcompos itional Ġrules Ġand Ġto Ġdynamically Ġgenerate Ġthe Ġcontext - specific Ġcompos itional Ġfunction . ĠThus , Ġthe Ġcompos itional Ġfunction Ġof Ġour Ġmodels Ġvaries Ġwith Ġpositions , Ġcontexts Ġand Ġsamples . ĠThe Ġdynamic Ġcompos itional Ġnetwork Ġthen Ġapplies Ġthose Ġcontext - specific Ġparameters Ġto Ġthe Ġcurrent Ġinput Ġinformation . ĠBoth Ġmeta Ġand Ġdynamic Ġnetworks Ġare Ġdifferent iable Ġsuch Ġthat Ġthe Ġoverall Ġnetworks Ġcan Ġbe Ġtrained Ġin Ġan Ġend - to - end Ġfashion . ĠAdditional , Ġto Ġreduce Ġthe Ġcomplexity Ġof Ġthe Ġwhole Ġnetworks , Ġwe Ġdefine Ġthe Ġdynamic Ġweight Ġmatrix Ġin Ġa Ġmanner Ġsim ulating Ġlow - rank Ġmatrix Ġdecom position . </s> We Ġevaluate Ġour Ġmodels Ġon Ġtwo Ġtypical Ġtasks : Ġtext Ġclassification Ġand Ġtext Ġsemantic Ġmatching . ĠThe Ġresults Ġshow Ġthat Ġour Ġmodels Ġare Ġmore Ġexpressive Ġdue Ġto Ġtheir Ġlearning Ġto Ġlearn Ġnature , Ġyet Ġwithout Ġincreasing Ġthe Ġnumber Ġof Ġmodel 's Ġparameters . ĠMoreover , Ġwe Ġfind Ġcertain Ġcomposition Ġoperations Ġcan Ġbe Ġlearned Ġimplicitly Ġby Ġmeta ĠTree NN , Ġsuch Ġas Ġthe Ġcomposition Ġof Ġnoun Ġphrases Ġand Ġverb Ġphrases . </s> The Ġcontributions Ġof Ġthe Ġpaper Ġcan Ġbe Ġsummed Ġup Ġas Ġfollows . </s> Tree - Struct ured ĠNeural ĠNetwork </s> In Ġthis Ġsection , Ġwe Ġbriefly Ġdescribe Ġthe Ġtree - struct ured Ġneural Ġnetworks . </s> The Ġidea Ġof Ġtree - struct ured Ġneural Ġnetworks Ġfor Ġnatural Ġlanguage Ġprocessing Ġ( N LP ) Ġis Ġto Ġtrain Ġa Ġdeep Ġlearning Ġmodel Ġwith Ġa Ġgram matical Ġtree Ġstructure ĠB IB REF 19 Ġthat Ġcan Ġbe Ġapplied Ġto Ġphrases Ġand Ġsentences . ĠAt Ġevery Ġnode Ġin Ġthe Ġtree , Ġthe Ġcontexts Ġof Ġthe Ġleft Ġand Ġright Ġchildren Ġare Ġcombined Ġby Ġa Ġcompos itional Ġfunction . ĠThe Ġparameters Ġof Ġthe Ġcompos itional Ġfunction Ġare Ġshared Ġacross Ġall Ġnodes Ġin Ġthe Ġtree . ĠThe Ġlayer Ġcomputed Ġat Ġthe Ġtop Ġnode Ġgives Ġa Ġrepresentation Ġfor Ġthe Ġwhole Ġsentence . </s> Van illa ĠRec ursive ĠNeural ĠNetwork </s> The Ġsimplest Ġmember Ġof Ġtree - struct ured ĠN N Ġis Ġthe Ġvanilla Ġrecursive Ġneural Ġnetwork ĠB IB REF 13 Ġ, Ġin Ġwhich Ġthe Ġrepresentation Ġof Ġparent Ġis Ġcalculated Ġby Ġweighted Ġlinear Ġcombination Ġof Ġthe Ġchild Ġvectors . </s> Form ally , Ġgiven Ġa Ġbinary Ġconstituency Ġtree Ġ$ T $ Ġinduced Ġby Ġa Ġsentence , Ġeach Ġnon - leaf Ġnode Ġcorresponds Ġto Ġa Ġphrase . ĠWe Ġrefer Ġto Ġ$\ math bf Ġ{ h } _{ j } Ġ\ in Ġ\ math bb Ġ{ R } ^{ d } $ Ġas Ġthe Ġhidden Ġstate Ġof Ġeach Ġnode Ġ$ j $ Ġ, Ġand Ġlet Ġ$\ math bf Ġ{ h } _{ j } ^{ l } $ Ġ, Ġ$\ math bf Ġ{ h } _{ j } ^{ r } $ Ġdenote Ġthe Ġleft Ġand Ġright Ġchild Ġrepresentations Ġrespectively . </s> $$ \ math bf Ġ{ h } _{ j } Ġ= Ġ\ tan h Ġ\ left ( Ġ\ math bf Ġ{ W } Ġ\ begin { b mat rix } Ġ\ math bf Ġ{ h } _{ j } ^{ l } Ġ\\ Ġ\ math bf Ġ{ h } _{ j } ^{ r } Ġ\\ Ġ\ end { b mat rix } Ġ+ Ġ\ math bf Ġ{ b } Ġ\ right ), $$ Ġ Ġ Ġ( E q . Ġ5 ) </s> where Ġ$\ math bf Ġ{ W } Ġ\ in Ġ\ math bb Ġ{ R } ^{ d Ġ\ times Ġ2 d } $ Ġis Ġa Ġlearn able Ġcompos itional Ġmatrix , Ġ$\ math bf Ġ{ b } $ Ġis Ġthe Ġbias Ġvector . </s> Tree ĠL ST M </s> Tree ĠL ST M ĠB IB REF 9 Ġis Ġa Ġgeneral ization Ġof ĠL ST Ms Ġto Ġtree - struct ured Ġnetwork Ġtop ologies . ĠIn Ġthis Ġmodel , Ġthe Ġcompos itional Ġfunction Ġis Ġan ĠL ST M Ġunit , Ġand Ġthe Ġhidden Ġstate Ġ$ h _ j $ Ġof Ġeach Ġnode Ġcan Ġbe Ġcomputed Ġas Ġfollows : Ġwe Ġrefer Ġto Ġ$\ math bf Ġ{ h } _{ j } $ Ġand Ġ$\ math bf Ġ{ c } _{ j } $ Ġas Ġthe Ġhidden Ġstate Ġand Ġmemory Ġcell Ġof Ġeach Ġnode Ġ$ j $ Ġ. ĠThe Ġtransition Ġequations Ġof Ġnode Ġ$ j $ Ġare Ġas Ġfollows : </s> $$ \ begin { b mat rix } Ġ\ math bf Ġ{\ t ilde { c }} _{ j } Ġ\\ Ġ\ math bf Ġ{ o } _{ j } Ġ\\ Ġ\ math bf Ġ{ i } _{ j } Ġ\\ Ġ\ math bf Ġ{ f } _{ j } ^{ l } Ġ\\ Ġ\ math bf Ġ{ f } _{ j } ^{ r } Ġ\ end { b mat rix } Ġ& = Ġ\ begin { b mat rix } Ġ\ tan h Ġ\\ Ġ\ s igma Ġ\\ Ġ\ s igma Ġ\\ Ġ\ s igma Ġ\\ Ġ\ s igma Ġ\ end { b mat rix } Ġ\ left ( Ġ\ math bf Ġ{ W } Ġ\ begin { b mat rix } Ġ\ math bf Ġ{ x } _{ j } Ġ\\ Ġ\ math bf Ġ{ h } _{ j } ^{ l } Ġ\\ Ġ\ math bf Ġ{ h } _{ j } ^{ r } Ġ\\ Ġ\ end { b mat rix } Ġ+ Ġ\ math bf Ġ{ b } Ġ\ right ), \\ Ġ\ math bf Ġ{ c } _{ j } Ġ& = Ġ\ math bf Ġ{\ t ilde { c }} _{ j } Ġ\ od ot Ġ\ math bf Ġ{ i } _{ j } Ġ+ Ġ\ math bf Ġ{ c } _{ j } ^{ l } Ġ\ od ot Ġ\ math bf Ġ{ f } _{ j } ^{ l } + Ġ\ math bf Ġ{ c } _{ j } ^{ r } Ġ\ od ot Ġ\ math bf Ġ{ f } _{ j } ^{ r } Ġ, Ġ\\ Ġ\ math bf Ġ{ h } _{ j } Ġ& = Ġ\ math bf Ġ{ o } _{ j } Ġ\ od ot Ġ\ tan h Ġ\ left ( Ġ\ math bf Ġ{ c } _{ j } Ġ\ right ), $$ Ġ Ġ Ġ( E q . Ġ8 ) </s> where Ġ$\ math bf Ġ{ x } _ j Ġ\ in Ġ\ math bb Ġ{ R } ^{ d } $ Ġdenotes Ġthe Ġinput Ġvector Ġand Ġis Ġnon - zero Ġif Ġand Ġonly Ġif Ġit Ġis Ġa Ġleaf Ġnode . ĠThe Ġsupers cript Ġ$ l $ Ġand Ġ$ r $ Ġrepresent Ġthe Ġleft Ġchild Ġand Ġright Ġchild Ġrespectively . Ġ$\ s igma Ġ$ Ġrepresents Ġthe Ġlog istic Ġs igm oid Ġfunction Ġand Ġ$\ od ot Ġ$ Ġdenotes Ġelement - wise Ġmultiplication . Ġ$\ math bf Ġ{ W } Ġ\ in Ġ\ math bb Ġ{ R } ^{ 5 d Ġ\ times Ġ3 d } $ Ġand Ġ$\ math bf Ġ{ b } Ġ\ in Ġ\ math bb Ġ{ R } ^{ 5 d } $ Ġare Ġlearn able Ġparameters . </s> Dynamic ĠCompos itional ĠNeural ĠNetwork </s> In Ġthe Ġabove Ġtwo Ġtree - struct ured ĠN Ns , Ġthe Ġcompos itional Ġfunction Ġis Ġshared Ġacross Ġall Ġnodes Ġin Ġthe Ġtree , Ġwhich Ġresults Ġin Ġunder fitting Ġsince Ġthe Ġsemantic Ġcompositions Ġhave Ġgreat Ġdivers ities . ĠTo Ġaddress Ġthis Ġproblem , Ġwe Ġpropose Ġtwo Ġdynamic Ġcompos itional Ġneural Ġnetworks Ġover Ġtree Ġstructure , Ġwhich Ġdynamically Ġgenerate Ġdifferent Ġparameters Ġfor Ġdifferent Ġtypes Ġof Ġcompositions . ĠFigure Ġ2 Ġshows Ġan Ġillustration Ġof Ġthe Ġdynamic Ġcompos itional Ġneural Ġnetwork , Ġconsisting Ġof Ġtwo Ġcomponents : Ġ( 1 ) Ġmeta Ġnetwork Ġand Ġ( 2 ) Ġbasic Ġnetwork Ġwith Ġdynamic Ġparameters . </s> Specifically , Ġwe Ġpropose Ġtwo Ġmeta Ġnetworks Ġto Ġgenerate Ġthe Ġcontext - specific Ġcompos itional Ġfunctions Ġfor ĠRec NN Ġand ĠTree L ST M Ġrespectively . </s> Meta ĠNetwork Ġfor ĠRec NN </s> For ĠRec NN , Ġwe Ġreplace Ġthe Ġstatic Ġparameters Ġ$\ math bf Ġ{ W } $ Ġand Ġ$\ math bf Ġ{ b } $ Ġin ĠE q .( Ġ5 Ġ) Ġwith Ġthe Ġdynamic Ġparameters Ġ$\ math bf Ġ{ W } (\ math bf Ġ{ z } _{ j }) $ Ġand Ġ$\ math bf Ġ{ b } (\ math bf Ġ{ z } _{ j }) $ Ġ, Ġwhich Ġare Ġgenerated Ġby Ġa Ġmeta Ġnetwork . ĠThe Ġmeta Ġnetwork Ġis Ġa Ġsmaller ĠRec NN , Ġand Ġthe Ġhidden Ġstate Ġ$\ hat {\ math bf Ġ{ h }} _{ j }\ in Ġ\ math bb Ġ{ R } ^{ m } $ Ġof Ġnode Ġ$ j $ Ġin Ġmeta Ġnetwork Ġis Ġdefined Ġas </s> $$ \ hat {\ math bf Ġ{ h }} _{ j } Ġ& = Ġ\ tan h Ġ\ left ( Ġ\ math bf Ġ{ W } _{ m } Ġ\ math cal Ġ{ H } _{ j } Ġ+ Ġ\ math bf Ġ{ b } _{ m } Ġ\ right ), \\ Ġ\ math bf Ġ{ z } _{ j } Ġ& = Ġ\ math bf Ġ{ W } _{ z } Ġ\ hat {\ math bf Ġ{ h }} _{ j } $$ Ġ Ġ Ġ( E q . Ġ11 ) </s> where Ġ$\ math cal Ġ{ H } _{ j } Ġ= Ġ\ math bf Ġ{ h } _{ j } ^{ l }\ opl us Ġ\ math bf Ġ{ h } _{ j } ^{ r }\ opl us Ġ\ math bf Ġ{\ hat { h }} _{ j } ^{ l }\ opl us Ġ\ math bf Ġ{\ hat { h }} _{ j } ^{ r } Ġ\ in Ġ\ math bb Ġ{ R } ^{ 2 m + 2 d } $ Ġ, Ġ$\ math bf Ġ{ W } _{ m } Ġ\ in Ġ\ math bb Ġ{ R } ^{ m Ġ\ times Ġ( 2 d + 2 m )} $ Ġand Ġ$\ math bf Ġ{ b } _{ m } Ġ\ in Ġ\ math bb Ġ{ R } ^{ m } $ Ġare Ġparameters Ġof Ġmeta ĠRec NN ; Ġ$\ math bf Ġ{ W } _{ z } Ġ\ in Ġ\ math bb Ġ{ R } ^{ z Ġ\ times Ġm } $ Ġis Ġa Ġscale Ġmatrix . </s> To Ġreduce Ġthe Ġnumber Ġof Ġthe Ġparameters , Ġwe Ġdefine Ġthe Ġdynamic Ġparameters Ġwith Ġa Ġlow - rank Ġfactor ized Ġrepresentation Ġof Ġthe Ġweights , Ġanalogous Ġto Ġthe ĠSing ular ĠValue ĠDec om position . ĠThe Ġdynamic Ġparameters Ġ$\ math bf Ġ{ W } (\ math bf Ġ{ z } _{ j }) $ Ġand Ġ$\ math bf Ġ{ b } (\ math bf Ġ{ z } _{ j }) $ Ġof Ġthe Ġbasic ĠRec NN Ġare Ġcomputed Ġby : </s> $$ \ math bf Ġ{ W } (\ math bf Ġ{ z } _{ j }) Ġ& = Ġ\ begin { b mat rix } ĠP _ l Ġ\ math bf Ġ{ D } (\ math bf Ġ{ z } _ j ) Q _ l Ġ\\ ĠP _ r Ġ\ math bf Ġ{ D } (\ math bf Ġ{ z } _ j ) Q _ r Ġ\\ Ġ\ end { b mat rix } Ġ\\ Ġ\ math bf Ġ{ b } (\ math bf Ġ{ z } _{ j }) Ġ& = Ġ\ begin { b mat rix } ĠB _ l \ math bf Ġ{ z } _ j Ġ\\ ĠB _ r \ math bf Ġ{ z } _ j Ġ\\ Ġ\ end { b mat rix } $$ Ġ Ġ Ġ( E q . Ġ12 ) </s> where Ġ$ P Ġ\ in Ġ\ math bb Ġ{ R } ^{ d Ġ\ times Ġz } $ Ġ, Ġ$ Q Ġ\ in Ġ\ math bb Ġ{ R } ^{ z Ġ\ times Ġd } $ Ġ, Ġand Ġ$\ math bf Ġ{ D } (\ math bf Ġ{ z } _ t ) Ġ\ in Ġ\ math bb Ġ{ R } ^{ z Ġ\ times Ġz } $ Ġis Ġthe Ġdiagonal Ġmatrix Ġof Ġ$\ math bf Ġ{ z } $ Ġ. </s> Thus , Ġour Ġdynamic ĠRec NN Ġneeds Ġ$( 6 d z Ġ+ Ġm z ) $ Ġparameters , Ġwhile Ġthe Ġvanilla ĠRec NN Ġhas Ġ$( 2 d ^ 2 + d ) $ Ġparameters . ĠWith Ġa Ġsmall Ġ$ z $ Ġand Ġ$ m $ Ġ, Ġour Ġdynamic ĠRec NN Ġneeds Ġless Ġparameters Ġthan Ġthe Ġvanilla ĠRec NN . ĠFor Ġexample , Ġif Ġwe Ġset Ġ$ d Ġ= 100 $ Ġand Ġ$ z = m = 20 $ Ġ, Ġour Ġmodel Ġneeds Ġ$ 12 , 400 $ Ġparameters Ġwhile Ġthe Ġvanilla Ġmodel Ġneeds Ġ$ 20 , 100 $ Ġparameters . </s> Meta ĠNetwork Ġfor ĠTree L ST M </s> Likewise , Ġwe Ġalso Ġuse Ġa Ġsmaller Ġmeta Ġnetwork Ġto Ġgenerate Ġthe Ġstatic Ġparameters Ġ$\ math bf Ġ{ W } $ Ġand Ġ$\ math bf Ġ{ b } $ Ġin ĠE q .( Ġ8 Ġ) Ġwith Ġthe Ġdynamic Ġparameters Ġ$\ math bf Ġ{ W } (\ math bf Ġ{ z } _{ j }) $ Ġand Ġ$\ math bf Ġ{ b } (\ math bf Ġ{ z } _{ j }) $ Ġ. ĠThe Ġmeta Ġnetwork Ġis Ġa Ġsmaller ĠTree L ST M , Ġand Ġthe Ġhidden Ġstate Ġ$\ hat {\ math bf Ġ{ h }} _{ j }\ in Ġ\ math bb Ġ{ R } ^{ m } $ Ġof Ġnode Ġ$ j $ Ġin Ġmeta Ġnetwork Ġis Ġdefined Ġas </s> $$ \ begin { b mat rix } Ġ\ math bf Ġ{\ hat { g }} _{ j } Ġ\\ Ġ\ math bf Ġ{\ hat { o }} _{ j } Ġ\\ Ġ\ math bf Ġ{\ hat { i }} _{ j } Ġ\\ Ġ\ math bf Ġ{\ hat { f }} _{ j } ^{ l } Ġ\\ Ġ\ math bf Ġ{\ hat { f }} _{ j } ^{ r } Ġ\ end { b mat rix } Ġ& = Ġ\ begin { b mat rix } Ġ\ tan h Ġ\\ Ġ\ s igma Ġ\\ Ġ\ s igma Ġ\\ Ġ\ s igma Ġ\\ Ġ\ s igma Ġ\ end { b mat rix } Ġ\ left ( Ġ\ math bf Ġ{ W } _{ m } Ġ\ begin { b mat rix } Ġ\ math bf Ġ{ x } _{ j } Ġ\\ Ġ\ math cal Ġ{ H } _{ j } Ġ\\ Ġ\ end { b mat rix } Ġ+ Ġ\ math bf Ġ{ b } _{ m } Ġ\ right ) Ġ, \\ Ġ\ math bf Ġ{\ hat { c }} _{ j } Ġ& = Ġ\ math bf Ġ{\ hat { g }} _{ j } Ġ\ od ot Ġ\ math bf Ġ{\ hat { i }} _{ j } Ġ+ Ġ\ math bf Ġ{\ hat { g }} _{ j } ^{ l } Ġ\ od ot Ġ\ math bf Ġ{\ hat { f }} _{ j } ^{ l } + Ġ\ math bf Ġ{\ hat { g }} _{ j } ^{ r } Ġ\ od ot Ġ\ math bf Ġ{\ hat { f }} _{ j } ^{ r } Ġ, Ġ\\ Ġ\ math bf Ġ{\ hat { h }} _{ j } Ġ& = Ġ\ math bf Ġ{\ hat { o }} _{ j } Ġ\ od ot Ġ\ tan h Ġ\ left ( Ġ\ math bf Ġ{\ hat { c }} _{ j } Ġ\ right ), Ġ\\ Ġ\ math bf Ġ{ z } _{ j } Ġ& = Ġ\ math bf Ġ{ W } _{ z } Ġ\ hat {\ math bf Ġ{ h }} _{ j } $$ Ġ Ġ Ġ( E q . Ġ14 ) </s> where Ġ$\ math cal Ġ{ H } _{ j } Ġ= Ġ\ math bf Ġ{ h } _{ j } ^{ l }\ opl us Ġ\ math bf Ġ{ h } _{ j } ^{ r }\ opl us Ġ\ math bf Ġ{\ hat { h }} _{ j } ^{ l }\ opl us Ġ\ math bf Ġ{\ hat { h }} _{ j } ^{ r } Ġ\ in Ġ\ math bb Ġ{ R } ^{ 2 m + 2 d } $ Ġ; Ġ$\ math bf Ġ{ W } _{ m } Ġ\ in Ġ\ math bb Ġ{ R } ^{ 5 m Ġ\ times Ġ( 3 d + 2 m )} $ Ġand Ġ$\ math bf Ġ{ b } _{ m } Ġ\ in Ġ\ math bb Ġ{ R } ^{ m } $ Ġare Ġparameters Ġof Ġmeta ĠTree L ST M ; Ġ$\ math bf Ġ{ W } _{ z } Ġ\ in Ġ\ math bb Ġ{ R } ^{ z Ġ\ times Ġm } $ Ġis Ġa Ġscale Ġmatrix . </s> The Ġdynamic Ġparameters Ġ$\ math bf Ġ{ W } (\ math bf Ġ{ z } _{ j }) $ Ġand Ġ$\ math bf Ġ{ b } (\ math bf Ġ{ z } _{ j }) $ Ġof Ġbasic ĠTree L ST M Ġare Ġcomputed Ġby : </s> $$ \ math bf Ġ{ W } (\ math bf Ġ{ z } _{ j }) Ġ& = Ġ\ left [ \ math bf Ġ{ W } ^{ g }, Ġ\ math bf Ġ{ W } ^{ i }, Ġ\ math bf Ġ{ W } ^{ f ^ l }, \ math bf Ġ{ W } ^{ f ^ r }, \ math bf Ġ{ W } ^{ o }\ right ] Ġ\\ Ġ\ math bf Ġ{ b } (\ math bf Ġ{ z } _{ j }) Ġ& = Ġ\ left [ \ math bf Ġ{ b } ^{ g }, Ġ\ math bf Ġ{ b } ^{ i }, Ġ\ math bf Ġ{ b } ^{ f ^ l }, \ math bf Ġ{ b } ^{ f ^ r }, \ math bf Ġ{ b } ^{ o }\ right ], $$ Ġ Ġ Ġ( E q . Ġ15 ) </s> where Ġfor Ġ$ * \ in Ġ\ l brace Ġc , o , i , f ^ l , f ^ r \ r brace Ġ$ Ġ, </s> $$ \ math bf Ġ{ W } ^{ * } (\ math bf Ġ{ z } _{ j }) Ġ& = Ġ\ begin { b mat rix } ĠP _ x ^{ * } Ġ\ math bf Ġ{ D } (\ math bf Ġ{ z } _ t ) Q _ x ^{ * } Ġ\\ ĠP _ l ^{ * } Ġ\ math bf Ġ{ D } (\ math bf Ġ{ z } _ t ) Q _ l ^{ * } Ġ\\ ĠP _ r ^{ * } Ġ\ math bf Ġ{ D } (\ math bf Ġ{ z } _ t ) Q _ r ^{ * } Ġ\\ Ġ\ end { b mat rix }, Ġ\\ Ġ\ math bf Ġ{ b } ^{ * } (\ math bf Ġ{ z } _{ j }) Ġ& = Ġ\ begin { b mat rix } ĠB _ x ^{ * }\ math bf Ġ{ z } _ t Ġ\\ ĠB _ l ^{ * }\ math bf Ġ{ z } _ t Ġ\\ ĠB _ r ^{ * }\ math bf Ġ{ z } _ t Ġ\\ Ġ\ end { b mat rix }, $$ Ġ Ġ Ġ( E q . Ġ16 ) </s> where Ġ$ P Ġ\ in Ġ\ math bb Ġ{ R } ^{ 5 d Ġ\ times Ġz } $ Ġ, Ġ$ Q Ġ\ in Ġ\ math bb Ġ{ R } ^{ z Ġ\ times Ġ3 d } $ Ġ, Ġ$ B Ġ\ in Ġ\ math bb Ġ{ R } ^{ 5 d Ġ\ times Ġz } $ Ġ, Ġand Ġ$\ math bf Ġ{ D } (\ math bf Ġ{ z } _ t ) Ġ\ in Ġ\ math bb Ġ{ R } ^{ z Ġ\ times Ġz } $ Ġis Ġthe Ġdiagonal Ġmatrix Ġof Ġ$\ math bf Ġ{ z } $ Ġ. </s> With Ġa Ġsmall Ġ$ z $ Ġand Ġ$ m $ Ġ, Ġour Ġdynamic ĠTree L ST M Ġneeds Ġa Ġsimilar Ġamount Ġof Ġparameters Ġcompared Ġto Ġthe Ġstandard ĠTree L ST M . </s> Application Ġof ĠDynamic ĠCompos itional ĠNeural ĠNetworks </s> In Ġthis Ġsection , Ġwe Ġdescribe Ġtwo Ġspecific Ġmodels Ġto Ġshow Ġthe Ġapplications Ġof Ġdynamic Ġcompos itional Ġneural Ġnetworks Ġfor Ġtwo Ġtypical Ġtasks Ġin ĠN LP . </s> Text ĠClassification </s> The Ġpurpose Ġof Ġtext Ġclassification Ġis Ġthat , Ġgiven Ġa Ġsentence Ġ$ x $ Ġ, Ġthe Ġmodel Ġshould Ġpredict Ġlabels Ġ$\ hat { y } $ Ġfrom Ġa Ġpre - defined Ġlabel Ġset Ġ$\ math cal Ġ{ Y } $ Ġ. ĠFrom Ġthe Ġdescription Ġin Ġthe Ġprevious Ġsection , Ġwe Ġcan Ġcompute Ġthe Ġdistributed Ġrepresentation Ġ$\ math bf Ġ{ h } _ j $ Ġof Ġthe Ġphrase Ġat Ġnode Ġ$ j $ Ġof Ġa Ġtree : </s> $$ \ math bf Ġ{ h } _{ j } Ġ& = Ġ\ oper at orn am ew ith limits { DC - Tree NN } (\ math bf Ġ{ x } _ j , Ġ\ math bf Ġ{ h } _{ j } ^ l , Ġ\ math bf Ġ{ h } _{ j } ^ r , \ the ta Ġ) $$ Ġ Ġ Ġ( E q . Ġ18 ) </s> After Ġthis Ġrecursive Ġprocess , Ġthe Ġhidden Ġstate Ġ$\ math bf Ġ{ h } _ R $ Ġat Ġthe Ġroot Ġnode Ġis Ġused Ġas Ġthe Ġsent ential Ġrepresentation , Ġwhich Ġthen Ġfollowed Ġby Ġa Ġsoft max Ġclass ifier Ġto Ġpredict Ġthe Ġprobability Ġdistribution Ġover Ġclasses . </s> $$ {\ hat {\ math bf Ġ{ y }}} Ġ= Ġ\ oper at orn am ew ith limits { soft max } (\ math bf Ġ{ W } _ t Ġ\ math bf Ġ{ h } _ R Ġ+ Ġ\ math bf Ġ{ b } _ t ) $$ Ġ Ġ Ġ( E q . Ġ19 ) </s> where Ġ${ \ hat {\ math bf Ġ{ y }}} $ Ġis Ġprediction Ġprobabilities , Ġ$\ math bf Ġ{ W } _ t $ Ġand Ġ$\ math bf Ġ{ b } _ t $ Ġare Ġthe Ġparameters Ġof Ġthe Ġclass ifier . </s> We Ġevaluate Ġour Ġmodels Ġon Ġfive Ġdifferent Ġdatasets . ĠThe Ġdetailed Ġstatistics Ġabout Ġthe Ġfive Ġdatasets Ġare Ġlisted Ġin ĠTable Ġ2 Ġ. ĠEach Ġdataset Ġis Ġbriefly Ġdescribed Ġas Ġfollows . </s> S ST ĠThe Ġmovie Ġreviews Ġwith Ġtwo Ġclasses Ġ( negative , Ġpositive ) Ġin Ġthe ĠStanford ĠSent iment ĠTree bank ĠB IB REF 23 Ġ. </s> MR ĠThe Ġmovie Ġreviews Ġwith Ġtwo Ġclasses ĠB IB REF 24 Ġ. </s> Q C ĠThe ĠT REC Ġquestions Ġdataset Ġinvolves Ġsix Ġdifferent Ġquestion Ġtypes . ĠB IB REF 25 Ġ. </s> S UB J ĠSubject ivity Ġdataset Ġwhere Ġthe Ġgoal Ġis Ġto Ġclassify Ġeach Ġinstance Ġ( sn ipp et ) Ġas Ġbeing Ġsubjective Ġor Ġobjective . ĠB IB REF 26 </s> IE ĠId iom Ġenhanced Ġsentiment Ġclassification . ĠB IB REF 27 Ġ. ĠEach Ġsentence Ġcontains Ġat Ġleast Ġone Ġid iom . </s> As Ġshown Ġin ĠTable Ġ3 Ġ, ĠDC - Tree L ST M Ġconsistently Ġoutper forms ĠRec NN , ĠMV - Rec NN , ĠR NT N , Ġand ĠTree L ST M Ġby Ġa Ġlarge Ġmargin Ġwhile Ġachieving Ġcomparable Ġresults Ġto Ġthe ĠCNN Ġand Ġusing Ġmuch Ġfewer Ġparameters .( The Ġnumber Ġof Ġparameters Ġin Ġour Ġmodels Ġis Ġapproximately Ġ10 K Ġwhile Ġin ĠCNN Ġthe Ġnumber Ġof Ġparameters Ġis Ġabout Ġ400 K ). ĠCompared Ġwith ĠRec NN , ĠDC - Rec NN Ġperforms Ġbetter , Ġindicating Ġthe Ġeffectiveness Ġof Ġthe Ġdynamic Ġcompos itional Ġfunction . ĠAdditionally , Ġboth ĠDC - Rec NN Ġand ĠDC - Tree L ST M Ġachieve Ġsubstantial Ġimprovement Ġon ĠIE Ġdataset , Ġwhich Ġcovers Ġthe Ġrichness Ġof Ġcomposition ality Ġ( idi omatic ity ). ĠWe Ġattribute Ġthe Ġsuccess Ġon ĠIE Ġto Ġits Ġpower Ġin Ġmodeling Ġmore Ġcomplicated Ġcomposition ality . </s> Text ĠSem antic ĠMatch ing </s> Among Ġmany Ġnatural Ġlanguage Ġprocessing Ġ( N LP ) Ġtasks , Ġa Ġcommon Ġproblem Ġis Ġmodelling Ġthe Ġrelevance Ġof Ġa Ġpair Ġof Ġtexts . ĠIn Ġthis Ġsection , Ġwe Ġshow Ġhow Ġto Ġeffectively Ġuse Ġthe Ġdynamic Ġcompos itional Ġneural Ġnetworks Ġto Ġmodel Ġthe Ġsemantic Ġrelationship Ġbetween Ġtwo Ġsentences . </s> As Ġshown Ġin ĠFigure Ġ1 Ġ, Ġgiven Ġtwo Ġsentences Ġ$ x _ a $ Ġand Ġ$ x _ b $ Ġ, Ġthe Ġrepresentation Ġof Ġeach Ġsentence Ġ$\ math bf Ġ{ h } _{ R } $ Ġcan Ġbe Ġcomputed Ġby Ġone Ġbasic ĠTree NN . </s> $$ \ math bf Ġ{ h } _{ R } ^{ ( a )} Ġ& = Ġ\ oper at orn am ew ith limits { DC - Tree NN } (\ math bf Ġ{ x } _ R , Ġ\ math bf Ġ{ h } _{ R } ^ l , Ġ\ math bf Ġ{ h } _{ R } ^ r , \ the ta Ġ_ { a }) Ġ\\ Ġ\ math bf Ġ{ h } _{ R } ^{ ( b )} Ġ& = Ġ\ oper at orn am ew ith limits { DC - Tree NN } (\ math bf Ġ{ x } _ R , Ġ\ math bf Ġ{ h } _{ R } ^ l , Ġ\ math bf Ġ{ h } _{ R } ^ r , \ the ta Ġ_ { b }) $$ Ġ Ġ Ġ( E q . Ġ21 ) </s> where Ġ$ R $ Ġdenotes Ġthe Ġroot Ġnode Ġof Ġa Ġtree . Ġ$\ the ta Ġ_ { a } $ Ġand Ġ$\ the ta Ġ_ { b } $ Ġare Ġgenerated Ġby Ġa Ġshared Ġmeta ĠTree NN . ĠThen , Ġthe Ġrepresentation Ġof Ġeach Ġsentence Ġwill Ġbe Ġfed Ġinto Ġa Ġmulti - layer Ġpercept ron Ġto Ġobtain Ġa Ġunified Ġrepresentation Ġfor Ġthe Ġfinal Ġrelationship Ġclassification . </s> The Ġsample - specific Ġbut Ġshared Ġmeta ĠTree NN Ġensures Ġthat , Ġon Ġthe Ġone Ġhand Ġwe Ġcan Ġdynamically Ġmodel Ġthe Ġdiversity Ġof Ġsemantic Ġcomposition ality , Ġon Ġthe Ġother Ġhand Ġwe Ġcan Ġcapture Ġthe Ġgeneral Ġrules Ġacross Ġdifferent Ġsamples . </s> We Ġchoose Ġthe Ġdataset Ġof ĠSent ences ĠInv olving ĠCompos itional ĠKnowledge Ġ( S ICK ), Ġwhich Ġis Ġproposed Ġby ĠB IB REF 30 Ġ[ ĠB IB REF 30 Ġ] Ġaiming Ġat Ġevaluation Ġof Ġcompos itional Ġdistribution al Ġsemantic Ġmodels . ĠThe Ġdataset Ġconsists Ġof Ġ99 27 Ġsentence Ġpairs Ġin Ġa Ġ4 500 / 500 / 49 27 Ġtrain / dev / test Ġsplit , Ġin Ġwhich Ġeach Ġsentence Ġpairs Ġare Ġpre - defined Ġinto Ġthree Ġlabels : ĠâĢ ľ ent ail ment âĢ Ŀ , âĢ ľ cont rad iction âĢ Ŀ Ġand ĠâĢ ľ neutral âĢ Ŀ . </s> Our Ġresults Ġare Ġsummarized Ġin ĠTable Ġ4 Ġ, Ġwhere Ġthe Ġperformance Ġof ĠNB OW , ĠL ST M , ĠRec NN , Ġand ĠR NT N Ġare Ġreported Ġby ĠB IB REF 31 Ġ, ĠB IB REF 32 Ġ. ĠFor Ġfair Ġcomparison , Ġwe Ġtrain Ġour Ġmodels Ġwith Ġthe Ġsame Ġsetting . ĠWe Ġcan Ġsee Ġboth ĠDC - Rec NN Ġand ĠDC - Tree L ST M Ġoutper form Ġcompetitor Ġmodels , Ġin Ġwhich ĠDC - Rec NN Ġ( DC - Tree L ST M ) Ġachieves Ġ3 % Ġ( 2 . 7 %) Ġimprovements Ġthan ĠRec NN Ġ( Tree L ST M ). ĠWe Ġthink Ġthis Ġbreakthrough Ġis Ġbasically Ġattributed Ġto Ġthe Ġdynamic Ġcompos itional Ġmechanism , Ġwhich Ġenables Ġour Ġmodels Ġto Ġcapture Ġvarious Ġsynt actic Ġpatterns Ġ( As Ġwe Ġwill Ġdiscuss Ġlater ) Ġtherefore Ġcan Ġmore Ġaccurately Ġunderstand Ġsentences . </s> Exper iment </s> To Ġmake Ġa Ġcomprehensive Ġevaluation , Ġwe Ġassess Ġour Ġmodel Ġon Ġfive Ġtext Ġclassification Ġtasks Ġand Ġa Ġsemantic Ġmatching Ġtask . </s> Training Ġand ĠHyper param eters </s> Given Ġa Ġsentence Ġ( or Ġsentence Ġpair ) Ġand Ġits Ġlabel , Ġthe Ġoutput Ġof Ġa Ġneural Ġnetwork Ġis Ġthe Ġprobabilities Ġof Ġthe Ġdifferent Ġclasses . ĠThe Ġparameters Ġof Ġthe Ġnetwork Ġare Ġtrained Ġto Ġminim ise Ġthe Ġcross - ent ropy Ġof Ġthe Ġpredicted Ġand Ġtrue Ġlabel Ġdistributions . ĠTo Ġminimize Ġthe Ġobjective , Ġwe Ġuse Ġsto ch astic Ġgradient Ġdescent Ġwith Ġthe Ġdiagonal Ġvariant Ġof ĠAda Grad ĠB IB REF 20 Ġ. </s> The Ġword Ġembed d ings Ġfor Ġall Ġof Ġthe Ġmodels Ġare Ġinitialized Ġwith ĠGl o Ve Ġvectors ĠB IB REF 21 Ġ. ĠThe Ġother Ġparameters Ġare Ġinitialized Ġby Ġrandomly Ġsampling Ġfrom Ġuniform Ġdistribution Ġin Ġ$ [ - 0 . 1 , Ġ0 . 1 ] $ Ġ. </s> The Ġfinal Ġhyper - param eters Ġare Ġas Ġfollows . ĠThe Ġinitial Ġlearning Ġrate Ġis Ġ$ 0 . 1 $ Ġ. ĠThe Ġregular ization Ġweight Ġof Ġthe Ġparameters Ġis Ġ$ 1 E { - 5 } $ Ġand Ġthe Ġothers Ġare Ġlisted Ġas ĠTable Ġ1 Ġ. </s> For Ġall Ġthe Ġsentences Ġin Ġthe Ġdatasets , Ġwe Ġparse Ġthem Ġwith Ġconstituency Ġparser ĠB IB REF 22 Ġto Ġobtain Ġthe Ġtrees Ġfor Ġour Ġmodels Ġand Ġsome Ġcompetitor Ġmodels . </s> Comp et itor ĠMethods </s> Rec NN ĠB IB REF 11 Ġ: ĠRec ursive Ġneural Ġnetwork Ġwith Ġstandard Ġcompos itional Ġfunction . </s> R NT N ĠB IB REF 23 Ġ: ĠThe ĠR NT N Ġis Ġa Ġrecursive Ġneural Ġnetwork Ġwith Ġneural Ġtens or Ġlayer , Ġwhich Ġcan Ġmodel Ġstrong Ġinteractions Ġbetween Ġtwo Ġconstituents . </s> M V - Rec NN ĠB IB REF 11 Ġ: ĠThe ĠMV - Rec NN Ġis Ġto Ġrepresent Ġevery Ġword Ġand Ġlonger Ġphrase Ġin Ġa Ġparse Ġtree Ġas Ġboth Ġa Ġvector Ġand Ġa Ġmatrix Ġin Ġorder Ġto Ġmodel Ġrich Ġcomposition ality . </s> Tree L ST M ĠB IB REF 9 Ġ: ĠRec ursive Ġneural Ġnetwork Ġwith ĠLong ĠShort - Term ĠMemory Ġunit . </s> Discussion Ġand ĠQual itative ĠAnalysis </s> In Ġour Ġmodels , Ġthe Ġlatent Ġvector Ġ$\ math bf Ġ{ z } $ Ġcontrols Ġthe Ġprocess Ġof Ġpredicting Ġnetwork 's Ġparameters Ġand Ġits Ġdimension ality Ġdetermines Ġthe Ġnumber Ġof Ġmodel 's Ġparameters . ĠNext , Ġwe Ġwill Ġinvestigate Ġhow Ġthe Ġcontrolling Ġvector Ġ$\ math bf Ġ{ z } $ Ġinfluences Ġthe Ġperformance Ġof Ġour Ġmodels . </s> Figure Ġ4 Ġshows Ġthe Ġaccur acies Ġof ĠDC - Rec NN Ġacross Ġthe Ġdifferent Ġdimensions Ġof Ġ$ [ 5 , Ġ10 , Ġ\ d ots Ġ, Ġ50 ] $ Ġfor Ġthe Ġcontrolling Ġvector Ġ$\ math bf Ġ{ z } $ Ġon Ġfive Ġdatasets . ĠWe Ġget Ġthe Ġfollowing Ġfindings : </s> For Ġall Ġfive Ġdatasets , Ġthe Ġmodel Ġcan Ġachieve Ġconsiderable Ġperformances Ġeven Ġwhen Ġthe Ġsize Ġof Ġvector Ġ$\ math bf Ġ{ z } $ Ġis Ġreduced Ġto Ġ5 . ĠParticularly , Ġfor Ġthe Ġdataset ĠQC , Ġthe Ġmodel Ġob tains Ġ$ 87 . 0 \ % $ Ġaccuracy Ġwith Ġa Ġpretty Ġsmall Ġmeta ĠTree - Rec NN , Ġsuggesting Ġa Ġsmaller Ġmeta Ġnetwork Ġcan Ġbe Ġused Ġfor Ġgenerating Ġa Ġmore Ġpowerful Ġcompos itional Ġfunction Ġto Ġeffectively Ġmodel Ġsentence . </s> When Ġdealing Ġwith Ġthe Ġdataset Ġwith Ġmore Ġlabels , Ġlarger Ġvector Ġsize Ġleads Ġto Ġa Ġbetter Ġperformance . ĠFor Ġexample , Ġthe Ġperformance Ġon ĠIE Ġand ĠQC Ġdatasets Ġreaches Ġthe Ġmaximum Ġwhen Ġthe Ġsize Ġof Ġ$ z $ Ġequals Ġ40 , Ġwhile Ġfor Ġthe Ġother Ġthree Ġdatasets ĠMR , ĠS ST Ġand ĠSUB J , Ġthe Ġmodel Ġob tains Ġthe Ġbest Ġperformance Ġwith Ġthe Ġvalue Ġof Ġ30 , Ġ30 Ġand Ġ20 Ġrespectively . </s> As Ġdescribed Ġin Ġprevious Ġsections , Ġwe Ġknow Ġthe Ġcompos itional Ġfunction Ġis Ġchanged Ġcross Ġchild Ġnodes Ġover Ġa Ġtree , Ġwhich Ġis Ġcontrolled Ġby Ġa Ġlatent Ġvector Ġ$ z $ Ġ. ĠTo Ġget Ġan Ġintuitive Ġunderstanding Ġof Ġhow Ġthe Ġcontrolling Ġvector Ġ$ z $ Ġworks , Ġwe Ġdesign Ġan Ġexperiment Ġto Ġexamine Ġthe Ġneuron 's Ġbehaviours Ġof Ġ$\ math bf Ġ{ z } $ Ġon Ġeach Ġnode . ĠMore Ġconcrete ly , Ġwe Ġrefer Ġto Ġ$ z _{ j k } $ Ġas Ġthe Ġactivation Ġof Ġthe Ġ$ k $ Ġ- ne uron Ġat Ġnode Ġ$ j $ Ġ, Ġwhere Ġ$ j Ġ\ in Ġ\ l brace Ġ1 , \ ld ots Ġ, N \ r brace Ġ$ Ġand Ġ$ k Ġ\ in Ġ\ l brace Ġ1 , \ ld ots Ġ, Ġz \ r brace Ġ$ Ġ. ĠThen Ġwe Ġrandomly Ġsample Ġsome Ġsentences Ġon Ġthe Ġdevelopment Ġset Ġfrom Ġthe Ġdatasets Ġwe Ġused . ĠBy Ġvisual izing Ġthe Ġlatent Ġvector Ġ$\ math bf Ġ{ z } _{ j } $ Ġand Ġanalyzing Ġthe Ġmaximum Ġactivation , Ġwe Ġcan Ġfind Ġwhat Ġkinds Ġof Ġpatterns Ġthe Ġcurrent Ġneuron Ġfocuses Ġon . </s> Table Ġ5 Ġillustrates Ġmultiple Ġinterpret able Ġneurons Ġand Ġsome Ġrepresentative Ġwords Ġor Ġphrases Ġwhich Ġcan Ġactivate Ġthese Ġneurons . ĠWe Ġcan Ġobserve Ġthat : </s> For Ġsome Ġsimple Ġtasks Ġsuch Ġas Ġtext Ġclassification , Ġmeta Ġnetwork Ġwill Ġintegrate Ġuseful Ġsemantic Ġinformation Ġinto Ġthe Ġthe Ġgeneration Ġprocess Ġof Ġcompos itional Ġfunction . ĠThese Ġsemantic Ġbias Ġbefore Ġcomposition Ġare Ġtask - specific . </s> For Ġexample , Ġthe Ġ21 - Ġ$ st $ Ġneuron Ġis Ġmore Ġsensitive Ġto Ġemotional Ġterms , Ġwhich Ġcan Ġbe Ġunderstood Ġas Ġa Ġsent inel , Ġtelling Ġthe Ġbasic Ġneural Ġnetwork Ġthat Ġan Ġinformative Ġphrase Ġis Ġcoming , Ġmore Ġattention Ġshould Ġbe Ġpaid Ġin Ġthe Ġprocess Ġof Ġcomposition . ĠFigure Ġ5 Ġ- ( a ) Ġshows Ġa Ġvisualization . ĠWe Ġcan Ġsee Ġin Ġthis Ġsentence , Ġthe Ġneuron Ġhas Ġrealized Ġthat Ġthis Ġidi omatic Ġcoll ocation ĠâĢ ľ in Ġstitches âĢ Ŀ Ġis Ġa Ġkey Ġpattern , Ġwhich Ġis Ġcrucial Ġfor Ġthe Ġfinal Ġsentiment Ġprediction . </s> For Ġmore Ġcomplicated Ġtasks Ġsuch Ġas Ġsemantic Ġmatching , Ġa Ġwell - ground ed Ġunderstanding Ġof Ġthe Ġsynt actic Ġstructure Ġis Ġcrucial . ĠIn Ġthis Ġcontext , Ġwe Ġfind Ġthat Ġa Ġmeta Ġnetwork Ġcould Ġcapture Ġsome Ġsynt actic Ġinformation . ĠFor Ġexample , Ġthe Ġ27 - Ġ$ th $ Ġneuron Ġmonitors Ġphrases Ġconstructed Ġby Ġlight - verb . ĠAs Ġshown Ġin ĠFigure Ġ5 Ġ- ( b ), Ġthe Ġverb Ġphrase ĠâĢ ľ taking Ġoff âĢ Ŀ Ġhas Ġbeen Ġattended Ġfor Ġforthcoming Ġcompos itional Ġoperation , Ġwhich Ġis Ġmore Ġuseful Ġfor Ġjudging Ġthe Ġsemantic Ġrelation Ġbetween Ġthe Ġsentence Ġpair ĠâĢ ľ An Ġairplane Ġis Ġtaking Ġoff / A Ġplane Ġis Ġlanding âĢ Ŀ . </s> Related ĠWork </s> One Ġthread Ġof Ġrelated Ġwork Ġis Ġthe Ġexploration Ġof Ġdifferent Ġkinds Ġof Ġcompos itional Ġfunction Ġover Ġtree Ġstructures . ĠB IB REF 11 Ġ[ ĠB IB REF 11 Ġ] Ġproposed Ġthe Ġrecursive Ġneural Ġnetwork Ġwith Ġstandard Ġcompos itional Ġfunction . ĠAfter Ġthat , Ġsome Ġextensions Ġare Ġintroduced Ġto Ġenhance Ġthe Ġexpressive Ġpower Ġof Ġcompos itional Ġfunction , Ġsuch Ġas ĠMV - Rec NN ĠB IB REF 23 Ġ, ĠSU - R NN ĠB IB REF 13 Ġ, ĠR NT N ĠB IB REF 23 Ġ, Ġwhile Ġthese Ġmodels Ġsuffer Ġfrom Ġthe Ġproblem Ġof Ġhard - coded Ġcompos itional Ġoperations Ġand Ġover fitting . </s> Another Ġthread Ġof Ġwork Ġis Ġthe Ġidea Ġof Ġusing Ġone Ġnetwork Ġto Ġdirect Ġthe Ġlearning Ġof Ġanother Ġnetwork ĠB IB REF 16 Ġ. ĠB IB REF 33 Ġ[ ĠB IB REF 33 Ġ] Ġintroduce Ġa Ġmeta Ġneural Ġnetwork Ġto Ġprovide Ġanother Ġnetwork Ġwith Ġa Ġstep Ġsize Ġand Ġa Ġdirection Ġvector , Ġwhich Ġis Ġhelpful Ġfor Ġparameter Ġoptimization . ĠB IB REF 16 Ġ[ ĠB IB REF 16 Ġ] Ġpropose Ġthe Ġdynamic Ġfilter Ġnetwork Ġto Ġimplicitly Ġlearn Ġa Ġvariety Ġof Ġfiltering Ġoperations . ĠB IB REF 17 Ġ[ ĠB IB REF 17 Ġ] Ġintroduce Ġa Ġlearn et Ġfor Ġone - shot Ġlearning , Ġwhich Ġcan Ġpredict Ġthe Ġparameters Ġof Ġa Ġsecond Ġnetwork Ġgiven Ġa Ġsingle Ġexempl ar . ĠB IB REF 18 Ġ[ ĠB IB REF 18 Ġ] Ġpropose Ġthe Ġmodel Ġhyper network , Ġwhich Ġuses Ġa Ġsmall Ġnetwork Ġto Ġgenerate Ġthe Ġweights Ġfor Ġa Ġlarger Ġnetwork . </s> Different Ġfrom Ġthese Ġmodels , Ġwe Ġemploy Ġthe Ġidea Ġof Ġparameter Ġgeneration Ġto Ġaddress Ġthe Ġlimitation Ġof Ġweight - sharing Ġor Ġpartially Ġsharing Ġparadigm Ġof Ġtree - based Ġcompos itional Ġmodels . </s> Conclusion </s> In Ġthis Ġwork , Ġwe Ġintroduce Ġa Ġmeta Ġneural Ġnetwork , Ġwhich Ġcan Ġgenerate Ġa Ġcompos itional Ġnetwork Ġto Ġdynamically Ġcompose Ġconstituents Ġover Ġtree Ġstructure . ĠThe Ġparameters Ġof Ġcompos itional Ġfunction Ġvary Ġfrom Ġposition Ġto Ġposition Ġand Ġfrom Ġsample Ġto Ġsample , Ġallowing Ġfor Ġmore Ġsophisticated Ġoperations Ġon Ġthe Ġinput . </s> To Ġevaluate Ġour Ġmodels , Ġwe Ġchoose Ġtwo Ġtypical ĠN LP Ġtasks Ġinvolving Ġsix Ġdatasets . ĠThe Ġqualitative Ġand Ġquantitative Ġexperiment Ġresults Ġdemonstrate Ġthe Ġeffectiveness Ġof Ġour Ġmodels . </s> Acknowled gments </s> We Ġwould Ġlike Ġto Ġthank Ġthe Ġanonymous Ġreviewers Ġfor Ġtheir Ġvaluable Ġcomments Ġand Ġthank ĠKai yu ĠQian , ĠJ iac hen ĠXu , ĠJ if an ĠChen Ġfor Ġuseful Ġdiscussions . ĠThis Ġwork Ġwas Ġpartially Ġfunded Ġby ĠNational ĠNatural ĠScience ĠFoundation Ġof ĠChina Ġ( No . Ġ6 153 2011 Ġand Ġ6 16 72 162 ), ĠShanghai ĠMunicipal ĠScience Ġand ĠTechnology ĠCommission Ġ( No . Ġ16 JC 14 204 01 ).