Various tree structured neural networks including variants of Tree-LSTM, Tree-based CNN, RNTN, and non-tree models including variants of LSTMs, CNNs, residual, and self-attention based networks
ABSTRACTIVE
Sentence classification baselines: RNTN (Socher et al. 2013), AdaMC-RNTN (Dong et al. 2014), TE-RNTN (Qian et al. 2015), TBCNN (Mou et al. 2015), Tree-LSTM (Tai, Socher, and Manning 2015), AdaHT-LSTM-CM (Liu, Qiu, and Huang 2017), DC-TreeLSTM (Liu, Qiu, and Huang 2017), TE-LSTM (Huang, Qian, and Zhu 2017), BiConTree (Teng and Zhang 2017), Gumbel Tree-LSTM (Choi, Yoo, and Lee 2018), TreeNet (Cheng et al. 2018), CNN (Kim 2014), AdaSent (Zhao, Lu, and Poupart 2015), LSTM-CNN (Zhou et al. 2016), byte-mLSTM (Radford, Jozefowicz, and Sutskever 2017), BCN + Char + CoVe (McCann et al. 2017), BCN + Char + ELMo (Peters et al. 2018). 
Stanford Natural Language Inference baselines: Latent Syntax Tree-LSTM (Yogatama et al. 2017), Tree-based CNN (Mou et al. 2016), Gumbel Tree-LSTM (Choi, Yoo, and Lee 2018), NSE (Munkhdalai and Yu 2017), Reinforced Self- Attention Network (Shen et al. 2018), Residual stacked encoders: (Nie and Bansal 2017), BiLSTM with generalized pooling (Chen, Ling, and Zhu 2018).
ABSTRACTIVE
FLOAT SELECTED: Table 1: The comparison of various models on different sentence classification tasks. We report the test accuracy of each model in percentage. Our SATA Tree-LSTM shows superior or competitive performance on all tasks, compared to previous treestructured models as well as other sophisticated models. ?: Latent tree-structured models. †: Models which are pre-trained with large external corpora. Our experimental results on the SNLI dataset are shown in table 2 . In this table, we report the test accuracy and number of trainable parameters for each model. Our SATA-LSTM again demonstrates its decent performance compared against the neural models built on both syntactic trees and latent trees, as well as the non-tree models. (Latent Syntax Tree-LSTM: BIBREF10 ( BIBREF10 ), Tree-based CNN: BIBREF35 ( BIBREF35 ), Gumbel Tree-LSTM: BIBREF11 ( BIBREF11 ), NSE: BIBREF36 ( BIBREF36 ), Reinforced Self-Attention Network: BIBREF4 ( BIBREF4 ), Residual stacked encoders: BIBREF37 ( BIBREF37 ), BiLSTM with generalized pooling: BIBREF38 ( BIBREF38 ).) Note that the number of learned parameters in our model is also comparable to other sophisticated models, showing the efficiency of our model.
FLOAT SELECTED: Table 1: The comparison of various models on different sentence classification tasks. We report the test accuracy of each model in percentage. Our SATA Tree-LSTM shows superior or competitive performance on all tasks, compared to previous treestructured models as well as other sophisticated models. ?: Latent tree-structured models. †: Models which are pre-trained with large external corpora. Our experimental results on the SNLI dataset are shown in table 2 . In this table, we report the test accuracy and number of trainable parameters for each model. Our SATA-LSTM again demonstrates its decent performance compared against the neural models built on both syntactic trees and latent trees, as well as the non-tree models. (Latent Syntax Tree-LSTM: BIBREF10 ( BIBREF10 ), Tree-based CNN: BIBREF35 ( BIBREF35 ), Gumbel Tree-LSTM: BIBREF11 ( BIBREF11 ), NSE: BIBREF36 ( BIBREF36 ), Reinforced Self-Attention Network: BIBREF4 ( BIBREF4 ), Residual stacked encoders: BIBREF37 ( BIBREF37 ), BiLSTM with generalized pooling: BIBREF38 ( BIBREF38 ).) Note that the number of learned parameters in our model is also comparable to other sophisticated models, showing the efficiency of our model.
1809.02286
Introduction </s> One Ġof Ġthe Ġmost Ġfundamental Ġtopics Ġin Ġnatural Ġlanguage Ġprocessing Ġis Ġhow Ġbest Ġto Ġderive Ġhigh - level Ġrepresentations Ġfrom Ġconstituent Ġparts , Ġas Ġnatural Ġlanguage Ġmeanings Ġare Ġa Ġfunction Ġof Ġtheir Ġconstituent Ġparts . ĠHow Ġbest Ġto Ġconstruct Ġa Ġsentence Ġrepresentation Ġfrom Ġdistributed Ġword Ġembed d ings Ġis Ġan Ġexample Ġdomain Ġof Ġthis Ġlarger Ġissue . ĠEven Ġthough Ġsequential Ġneural Ġmodels Ġsuch Ġas Ġrecurrent Ġneural Ġnetworks Ġ( R NN ) ĠB IB REF 0 Ġand Ġtheir Ġvariants Ġincluding ĠLong ĠShort - Term ĠMemory Ġ( L ST M ) ĠB IB REF 1 Ġand ĠG ated ĠRec urrent ĠUnit Ġ( GR U ) ĠB IB REF 2 Ġhave Ġbecome Ġthe Ġde - fact o Ġstandard Ġfor Ġcond ensing Ġsentence - level Ġinformation Ġfrom Ġa Ġsequence Ġof Ġwords Ġinto Ġa Ġfixed Ġvector , Ġthere Ġhave Ġbeen Ġmany Ġlines Ġof Ġresearch Ġtowards Ġbetter Ġsentence Ġrepresentation Ġusing Ġother Ġneural Ġarchitectures , Ġe . g . Ġconv olution al Ġneural Ġnetworks Ġ( CNN ) ĠB IB REF 3 Ġor Ġself - att ention Ġbased Ġmodels ĠB IB REF 4 Ġ. </s> From Ġa Ġlinguistic Ġpoint Ġof Ġview , Ġthe Ġunderlying Ġtree Ġstructure âĢĶ as Ġexpressed Ġby Ġits Ġconstituency Ġand Ġdependency Ġtrees âĢĶ of Ġa Ġsentence Ġis Ġan Ġintegral Ġpart Ġof Ġits Ġmeaning . ĠInspired Ġby Ġthis Ġfact , Ġsome Ġrecursive Ġneural Ġnetwork Ġ( R v NN ) Ġmodels Ġare Ġdesigned Ġto Ġreflect Ġthe Ġsynt actic Ġtree Ġstructure , Ġachieving Ġimpressive Ġresults Ġon Ġseveral Ġsentence - level Ġtasks Ġsuch Ġas Ġsentiment Ġanalysis ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, Ġmachine Ġtranslation ĠB IB REF 7 Ġ, Ġnatural Ġlanguage Ġinference ĠB IB REF 8 Ġ, Ġand Ġdiscourse Ġrelation Ġclassification ĠB IB REF 9 Ġ. </s> However , Ġsome Ġrecent Ġworks Ġhave ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġproposed Ġlatent Ġtree Ġmodels , Ġwhich Ġlearn Ġto Ġconstruct Ġtask - specific Ġtree Ġstructures Ġwithout Ġexplicit Ġsupervision , Ġbringing Ġinto Ġquestion Ġthe Ġvalue Ġof Ġlingu istically - mot ivated Ġrecursive Ġneural Ġmodels . ĠWitness ing Ġthe Ġsurprising Ġperformance Ġof Ġthe Ġlatent Ġtree Ġmodels Ġon Ġsome Ġsentence - level Ġtasks , Ġthere Ġarises Ġa Ġnatural Ġquestion : ĠAre Ġlinguistic Ġtree Ġstructures Ġthe Ġoptimal Ġway Ġof Ġcomposing Ġsentence Ġrepresentations Ġfor ĠN LP Ġtasks ? </s> In Ġthis Ġpaper , Ġwe Ġdemonstrate Ġthat Ġlinguistic Ġpri ors Ġare Ġin Ġfact Ġuseful Ġfor Ġdev ising Ġeffective Ġneural Ġmodels Ġfor Ġsentence Ġrepresentations , Ġshowing Ġthat Ġour Ġnovel Ġarchitecture Ġbased Ġon Ġconstituency Ġtrees Ġand Ġtheir Ġtag Ġinformation Ġob tains Ġsuperior Ġperformance Ġon Ġseveral Ġsentence - level Ġtasks , Ġincluding Ġsentiment Ġanalysis Ġand Ġnatural Ġlanguage Ġinference . </s> A Ġchief Ġnovelty Ġof Ġour Ġapproach Ġis Ġthat Ġwe Ġintroduce Ġa Ġsmall Ġseparate Ġtag - level Ġtree - L ST M Ġto Ġcontrol Ġthe Ġcomposition Ġfunction Ġof Ġthe Ġexisting Ġword - level Ġtree - L ST M , Ġwhich Ġis Ġin Ġcharge Ġof Ġextracting Ġhelpful Ġsynt actic Ġsignals Ġfor Ġmeaningful Ġsemantic Ġcomposition Ġof Ġconstituents Ġby Ġconsidering Ġboth Ġthe Ġstructures Ġand Ġlinguistic Ġtags Ġof Ġconstituency Ġtrees Ġsimultaneously . ĠIn Ġaddition , Ġwe Ġdemonstrate Ġthat Ġapplying Ġa Ġtypical ĠL ST M Ġto Ġpre process Ġthe Ġleaf Ġnodes Ġof Ġa Ġtree - L ST M Ġgreatly Ġimproves Ġthe Ġperformance Ġof Ġthe Ġtree Ġmodels . ĠMoreover , Ġwe Ġpropose Ġa Ġclustered Ġtag Ġset Ġto Ġreplace Ġthe Ġexisting Ġtags Ġon Ġthe Ġassumption Ġthat Ġthe Ġoriginal Ġsynt actic Ġtags Ġare Ġtoo Ġfined - gr ained Ġto Ġbe Ġuseful Ġin Ġneural Ġmodels . </s> In Ġshort , Ġour Ġcontributions Ġin Ġthis Ġwork Ġare Ġas Ġfollows : </s> Related ĠWork </s> Rec ursive Ġneural Ġnetworks Ġ( R v NN ) Ġare Ġa Ġkind Ġof Ġneural Ġarchitecture Ġwhich Ġmodel Ġsentences Ġby Ġexploiting Ġsynt actic Ġstructure . ĠWhile Ġearlier ĠR v NN Ġmodels Ġproposed Ġutilizing Ġdiverse Ġcomposition Ġfunctions , Ġincluding Ġfeed - forward Ġneural Ġnetworks ĠB IB REF 12 Ġ, Ġmatrix - vector Ġmultiplication ĠB IB REF 5 Ġ, Ġand Ġtens or Ġcomputation ĠB IB REF 6 Ġ, Ġtree - L ST Ms ĠB IB REF 13 Ġremain Ġthe Ġstandard Ġfor Ġseveral Ġsentence - level Ġtasks . </s> Even Ġthough Ġclassic ĠR v NN s Ġhave Ġdemonstrated Ġsuperior Ġperformance Ġon Ġa Ġvariety Ġof Ġtasks , Ġtheir Ġinf lex ibility , Ġi . e . Ġtheir Ġinability Ġto Ġhandle Ġdynamic Ġcomposition ality Ġfor Ġdifferent Ġsynt actic Ġconfigurations , Ġis Ġa Ġconsiderable Ġweakness . ĠFor Ġinstance , Ġit Ġwould Ġbe Ġdesirable Ġif Ġour Ġmodel Ġcould Ġdistinguish Ġe . g . Ġadjective - n oun Ġcomposition Ġfrom Ġthat Ġof Ġverb - n oun Ġor Ġpre position - n oun Ġcomposition , Ġas Ġmodels Ġfailing Ġto Ġmake Ġsuch Ġa Ġdistinction Ġignore Ġreal - world Ġsynt actic Ġconsiderations Ġsuch Ġas Ġ` - arity ' Ġof Ġfunction Ġwords Ġ( i . e . Ġtypes ), Ġand Ġthe Ġadjunct / argument Ġdistinction . </s> To Ġenable Ġdynamic Ġcomposition ality Ġin Ġrecursive Ġneural Ġnetworks , Ġmany Ġprevious Ġworks ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġ, ĠB IB REF 9 Ġ, ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ, ĠB IB REF 19 Ġhave Ġproposed Ġvarious Ġmethods . </s> One Ġmain Ġdirection Ġof Ġresearch Ġlever ages Ġtag Ġinformation , Ġwhich Ġis Ġproduced Ġas Ġa Ġby - product Ġof Ġparsing . ĠIn Ġdetail , ĠB IB REF 16 Ġ( ĠB IB REF 16 Ġ) Ġsuggested ĠTG - R NN , Ġa Ġmodel Ġemploying Ġdifferent Ġcomposition Ġfunctions Ġaccording Ġto ĠPOS Ġtags , Ġand ĠTE - R NN / TE - R NT N , Ġmodels Ġwhich Ġleverage Ġtag Ġembed d ings Ġas Ġadditional Ġinputs Ġfor Ġthe Ġexisting Ġtree - struct ured Ġmodels . ĠDespite Ġthe Ġnovelty Ġof Ġutilizing Ġtag Ġinformation , Ġthe Ġexplosion Ġof Ġthe Ġnumber Ġof Ġparameters Ġ( in Ġcase Ġof Ġthe ĠTG - R NN ) Ġand Ġthe Ġlimited Ġperformance Ġof Ġthe Ġoriginal Ġmodels Ġ( in Ġcase Ġof Ġthe ĠTE - R NN / TE - R NT N ) Ġhave Ġprevented Ġthese Ġmodels Ġfrom Ġbeing Ġwidely Ġadopted . ĠMeanwhile , ĠB IB REF 9 Ġ( ĠB IB REF 9 Ġ) Ġand ĠB IB REF 18 Ġ( ĠB IB REF 18 Ġ) Ġproposed Ġmodels Ġbased Ġon Ġa Ġtree - L ST M Ġwhich Ġalso Ġuses Ġthe Ġtag Ġvectors Ġto Ġcontrol Ġthe Ġgate Ġfunctions Ġof Ġthe Ġtree - L ST M . ĠIn Ġspite Ġof Ġtheir Ġimpressive Ġresults , Ġthere Ġis Ġa Ġlimitation Ġthat Ġthe Ġtrained Ġtag Ġembed d ings Ġare Ġtoo Ġsimple Ġto Ġreflect Ġthe Ġrich Ġinformation Ġwhich Ġtags Ġprovide Ġin Ġdifferent Ġsynt actic Ġstructures . ĠTo Ġalleviate Ġthis Ġproblem , Ġwe Ġintroduce Ġstructure - aware Ġtag Ġrepresentations Ġin Ġthe Ġnext Ġsection . </s> Another Ġway Ġof Ġbuilding Ġdynamic Ġcomposition ality Ġinto ĠR v NN s Ġis Ġto Ġtake Ġadvantage Ġof Ġa Ġmeta - network Ġ( or Ġhyper - network ). ĠInspired Ġby Ġrecent Ġworks Ġon Ġdynamic Ġparameter Ġprediction , ĠDC - Tree L ST Ms ĠB IB REF 17 Ġdynamically Ġcreate Ġthe Ġparameters Ġfor Ġcompos itional Ġfunctions Ġin Ġa Ġtree - L ST M . ĠSpecifically , Ġthe Ġmodel Ġhas Ġtwo Ġseparate Ġtree - L ST M Ġnetworks Ġwhose Ġarchitectures Ġare Ġsimilar , Ġbut Ġthe Ġsmaller Ġof Ġthe Ġtwo Ġis Ġutilized Ġto Ġcalculate Ġthe Ġweights Ġof Ġthe Ġbigger Ġone . ĠA Ġpossible Ġproblem Ġfor Ġthis Ġmodel Ġis Ġthat Ġit Ġmay Ġbe Ġeasy Ġto Ġbe Ġtrained Ġsuch Ġthat Ġthe Ġrole Ġof Ġeach Ġtree - L ST M Ġis Ġambiguous , Ġas Ġthey Ġshare Ġthe Ġsame Ġinput , Ġi . e . Ġword Ġinformation . ĠTherefore , Ġwe Ġdesign Ġtwo Ġdis ent angled Ġtree - L ST Ms Ġin Ġour Ġmodel Ġso Ġthat Ġone Ġfocuses Ġon Ġextracting Ġuseful Ġfeatures Ġfrom Ġonly Ġsynt actic Ġinformation Ġwhile Ġthe Ġother Ġcomp oses Ġsemantic Ġunits Ġwith Ġthe Ġaid Ġof Ġthe Ġfeatures . ĠFurthermore , Ġour Ġmodel Ġreduces Ġthe Ġcomplexity Ġof Ġcomputation Ġby Ġutilizing Ġtypical Ġtree - L ST M Ġframeworks Ġinstead Ġof Ġcomputing Ġthe Ġweights Ġfor Ġeach Ġexample . </s> Finally , Ġsome Ġrecent Ġworks ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġhave Ġproposed Ġlatent Ġtree - struct ured Ġmodels Ġthat Ġlearn Ġhow Ġto Ġformulate Ġtree Ġstructures Ġfrom Ġonly Ġsequences Ġof Ġtokens , Ġwithout Ġthe Ġaid Ġof Ġsynt actic Ġtrees Ġor Ġlinguistic Ġinformation . ĠThe Ġlatent Ġtree Ġmodels Ġhave Ġthe Ġadvantage Ġof Ġbeing Ġable Ġto Ġfind Ġthe Ġoptimized Ġtask - specific Ġorder Ġof Ġcomposition Ġrather Ġthan Ġa Ġsequential Ġor Ġsynt actic Ġone . ĠIn Ġexperiments , Ġwe Ġcompare Ġour Ġmodel Ġwith Ġnot Ġonly Ġsynt actic Ġtree - based Ġmodels Ġbut Ġalso Ġlatent Ġtree Ġmodels , Ġdemonstrating Ġthat Ġmodeling Ġwith Ġexplicit Ġlinguistic Ġknowledge Ġcan Ġbe Ġan Ġattractive Ġoption . </s> Model </s> In Ġthis Ġsection , Ġwe Ġintroduce Ġa Ġnovel ĠR v NN Ġarchitecture , Ġcalled ĠSATA ĠTree - L ST M Ġ( St ructure - A ware ĠTag ĠAug mented ĠTree - L ST M ). ĠThis Ġmodel Ġis Ġsimilar Ġto Ġtypical ĠTree - L ST Ms , Ġbut Ġprovides Ġdynamic Ġcomposition ality Ġby Ġaugment ing Ġa Ġseparate Ġtag - level Ġtree - L ST M Ġwhich Ġproduces Ġstructure - aware Ġtag Ġrepresentations Ġfor Ġeach Ġnode Ġin Ġa Ġtree . ĠIn Ġother Ġwords , Ġour Ġmodel Ġhas Ġtwo Ġindependent Ġtree - struct ured Ġmodules Ġbased Ġon Ġthe Ġsame Ġconstituency Ġtree , Ġone Ġof Ġwhich Ġ( word - level Ġtree - L ST M ) Ġis Ġresponsible Ġfor Ġconstructing Ġsentence Ġrepresentations Ġgiven Ġa Ġsequence Ġof Ġwords Ġas Ġusual , Ġwhile Ġthe Ġother Ġ( tag - level Ġtree - L ST M ) Ġprovides Ġsupplementary Ġsynt actic Ġinformation Ġto Ġthe Ġformer . </s> In Ġsection Ġ3 . 1 , Ġwe Ġfirst Ġreview Ġtree - L ST M Ġarchitectures . ĠThen Ġin Ġsection Ġ3 . 2 , Ġwe Ġintroduce Ġa Ġtag - level Ġtree - L ST M Ġand Ġstructure - aware Ġtag Ġrepresentations . ĠIn Ġsection Ġ3 . 3 , Ġwe Ġdiscuss Ġan Ġadditional Ġtechnique Ġto Ġboost Ġthe Ġperformance Ġof Ġtree - struct ured Ġmodels , Ġand Ġin Ġsection Ġ3 . 4 , Ġwe Ġdescribe Ġthe Ġentire Ġarchitecture Ġof Ġour Ġmodel Ġin Ġdetail . </s> Tree - L ST M </s> The ĠL ST M ĠB IB REF 1 Ġarchitecture Ġwas Ġfirst Ġintroduced Ġas Ġan Ġextension Ġof Ġthe ĠR NN Ġarchitecture Ġto Ġmitigate Ġthe Ġvanishing Ġand Ġexploding Ġgradient Ġproblems . ĠIn Ġaddition , Ġseveral Ġworks Ġhave Ġdiscovered Ġthat Ġapplying Ġthe ĠL ST M Ġcell Ġinto Ġtree Ġstructures Ġcan Ġbe Ġan Ġeffective Ġmeans Ġof Ġmodeling Ġsentence Ġrepresentations . </s> To Ġbe Ġformal , Ġthe Ġcomposition Ġfunction Ġof Ġthe Ġcell Ġin Ġa Ġtree - L ST M Ġcan Ġbe Ġformulated Ġas Ġfollows : </s> $$ Ġ Ġ\ begin { b mat rix } Ġ\ math bf Ġ{ i } Ġ\\ Ġ\ math bf Ġ{ f } _ l Ġ\\ Ġ\ math bf Ġ{ f } _ r Ġ\\ Ġ\ math bf Ġ{ o } Ġ\\ Ġ\ math bf Ġ{ g } Ġ\ end { b mat rix } Ġ= Ġ\ begin { b mat rix } Ġ\ s igma Ġ\\ Ġ\ s igma Ġ\\ Ġ\ s igma Ġ\\ Ġ\ s igma Ġ\\ Ġ\ tan h Ġ\ end { b mat rix } Ġ\ B igg Ġ( Ġ\ math bf Ġ{ W } Ġ\ begin { b mat rix } Ġ\ math bf Ġ{ h } _ l Ġ\\ Ġ\ math bf Ġ{ h } _ r Ġ\\ Ġ\ end { b mat rix } Ġ+ Ġ\ math bf Ġ{ b } Ġ\ B igg Ġ) $$ Ġ Ġ Ġ( E q . Ġ8 ) </s> $$ Ġ Ġ\ math bf Ġ{ c } Ġ= Ġ\ math bf Ġ{ f } _ l Ġ\ od ot Ġ\ math bf Ġ{ c } _ l Ġ+ Ġ\ math bf Ġ{ f } _ r Ġ\ od ot Ġ\ math bf Ġ{ c } _ r Ġ+ Ġ\ math bf Ġ{ i } Ġ\ od ot Ġ\ math bf Ġ{ g } \\ $$ Ġ Ġ Ġ( E q . Ġ9 ) </s> where Ġ$\ math bf Ġ{ h }, Ġ\ math bf Ġ{ c } Ġ\ in Ġ\ math bb Ġ{ R } ^{ d } $ Ġindicate Ġthe Ġhidden Ġstate Ġand Ġcell Ġstate Ġof Ġthe ĠL ST M Ġcell , Ġand Ġ$\ math bf Ġ{ h } _ l , Ġ\ math bf Ġ{ h } _ r , Ġ\ math bf Ġ{ c } _ l , Ġ\ math bf Ġ{ c } _ r Ġ\ in Ġ\ math bb Ġ{ R } ^{ d } $ Ġthe Ġhidden Ġstates Ġand Ġcell Ġstates Ġof Ġa Ġleft Ġand Ġright Ġchild . Ġ$\ math bf Ġ{ g } Ġ\ in Ġ\ math bb Ġ{ R } ^{ d } $ Ġis Ġthe Ġnewly Ġcomposed Ġinput Ġfor Ġthe Ġcell Ġand Ġ$\ math bf Ġ{ i }, Ġ\ math bf Ġ{ f } _{ l }, Ġ\ math bf Ġ{ f } _{ r }, Ġ\ math bf Ġ{ o } Ġ\ in Ġ\ math bb Ġ{ R } ^{ d } $ Ġrepresent Ġan Ġinput Ġgate , Ġtwo Ġforget Ġgates Ġ( left , Ġright ), Ġand Ġan Ġoutput Ġgate Ġrespectively . Ġ$\ math bf Ġ{ W } Ġ\ in Ġ\ math bb Ġ{ R } ^{ 5 d \ times Ġ2 d } $ Ġand Ġ$\ math bf Ġ{ b } Ġ\ in Ġ\ math bb Ġ{ R } ^{ 5 d } $ Ġare Ġtrain able Ġparameters . Ġ$\ s igma Ġ$ Ġcorresponds Ġto Ġthe Ġs igm oid Ġfunction , Ġ$\ tan h Ġ$ Ġto Ġthe Ġhyper b olic Ġtang ent , Ġand Ġ$\ od ot Ġ$ Ġto Ġelement - wise Ġmultiplication . </s> Note Ġthe Ġequations Ġassume Ġthat Ġthere Ġare Ġonly Ġtwo Ġchildren Ġfor Ġeach Ġnode , Ġi . e . Ġbinary Ġor Ġbin ar ized Ġtrees , Ġfollowing Ġthe Ġstandard Ġin Ġthe Ġliterature . ĠWhile ĠR v NN Ġmodels Ġcan Ġbe Ġconstructed Ġon Ġany Ġtree Ġstructure , Ġin Ġthis Ġwork Ġwe Ġonly Ġconsider Ġconstituency Ġtrees Ġas Ġinputs . </s> In Ġspite Ġof Ġthe Ġobvious Ġupside Ġthat Ġrecursive Ġmodels Ġhave Ġin Ġbeing Ġso Ġflexible , Ġthey Ġare Ġknown Ġfor Ġbeing Ġdifficult Ġto Ġfully Ġutilize Ġwith Ġbatch Ġcomput ations Ġas Ġcompared Ġto Ġother Ġneural Ġarchitectures Ġbecause Ġof Ġthe Ġdiversity Ġof Ġstructure Ġfound Ġacross Ġsentences . ĠTo Ġalleviate Ġthis Ġproblem , ĠB IB REF 8 Ġ( ĠB IB REF 8 Ġ) Ġproposed Ġthe ĠSP IN N Ġmodel , Ġwhich Ġbrings Ġa Ġshift - red uce Ġalgorithm Ġto Ġthe Ġtree - L ST M . ĠAs ĠSP IN N Ġsimpl ifies Ġthe Ġprocess Ġof Ġconstructing Ġa Ġtree Ġinto Ġonly Ġtwo Ġoperations , Ġi . e . Ġshift Ġand Ġreduce , Ġit Ġcan Ġsupport Ġmore Ġeffective Ġparallel Ġcomput ations Ġwhile Ġenjoying Ġthe Ġadvantages Ġof Ġtree Ġstructures . ĠFor Ġefficiency , Ġour Ġmodel Ġalso Ġstarts Ġfrom Ġour Ġown ĠSP IN N Ġre - im plementation , Ġwhose Ġfunction Ġis Ġexactly Ġthe Ġsame Ġas Ġthat Ġof Ġthe Ġtree - L ST M . </s> St ructure - aware ĠTag ĠRepresent ation </s> In Ġmost Ġprevious Ġworks Ġusing Ġlinguistic Ġtag Ġinformation ĠB IB REF 16 Ġ, ĠB IB REF 9 Ġ, ĠB IB REF 18 Ġ, Ġtags Ġare Ġusually Ġrepresented Ġas Ġsimple Ġlow - dimensional Ġdense Ġvectors , Ġsimilar Ġto Ġword Ġembed d ings . ĠThis Ġapproach Ġseems Ġreasonable Ġin Ġthe Ġcase Ġof ĠPOS Ġtags Ġthat Ġare Ġattached Ġto Ġthe Ġcorresponding Ġwords , Ġbut Ġphrase - level Ġconstituent Ġtags Ġ( e . g . ĠNP , ĠVP , ĠAD JP ) Ġvary Ġgreatly Ġin Ġsize Ġand Ġshape , Ġmaking Ġthem Ġless Ġam enable Ġto Ġuniform Ġtreatment . ĠFor Ġinstance , Ġeven Ġthe Ġsame Ġphrase Ġtags Ġwithin Ġdifferent Ġsynt actic Ġcontexts Ġcan Ġvary Ġgreatly Ġin Ġsize Ġand Ġinternal Ġstructure , Ġas Ġthe Ġcase Ġof ĠNP Ġtags Ġin ĠFigure Ġ1 Ġshows . ĠHere , Ġthe ĠNP Ġconsisting Ġof ĠDT [ the ]- NN [ stories ] Ġhas Ġa Ġdifferent Ġinternal Ġstructure Ġthan Ġthe ĠNP Ġconsisting Ġof ĠNP [ the Ġfilm Ġ' s ]- NN S [ short comings ]. </s> One Ġway Ġof Ġder iving Ġstructure - aware Ġtag Ġrepresentations Ġfrom Ġthe Ġoriginal Ġtag Ġembed d ings Ġis Ġto Ġintroduce Ġa Ġseparate Ġtag - level Ġtree - L ST M Ġwhich Ġaccepts Ġthe Ġtypical Ġtag Ġembed d ings Ġat Ġeach Ġnode Ġof Ġa Ġtree Ġand Ġoutputs Ġthe Ġcomputed Ġstructure - aware Ġtag Ġrepresentations Ġfor Ġthe Ġnodes . ĠNote Ġthat Ġthe Ġmodule Ġconcent rates Ġon Ġextracting Ġuseful Ġsynt actic Ġfeatures Ġby Ġconsidering Ġonly Ġthe Ġtags Ġand Ġstructures Ġof Ġthe Ġtrees , Ġexcluding Ġword Ġinformation . </s> Form ally , Ġwe Ġdenote Ġa Ġtag Ġembed ding Ġfor Ġthe Ġtag Ġattached Ġto Ġeach Ġnode Ġin Ġa Ġtree Ġas Ġ$\ text bf Ġ{ e } Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ \ text { T }} $ Ġ. ĠThen , Ġthe Ġfunction Ġof Ġeach Ġcell Ġin Ġthe Ġtag Ġtree - L ST M Ġis Ġdefined Ġin Ġthe Ġfollowing Ġway . ĠLeaf Ġnodes Ġare Ġdefined Ġby Ġthe Ġfollowing : </s> $$ Ġ Ġ\ begin { b mat rix } Ġ\ hat {\ math bf Ġ{ c }} Ġ\\ Ġ\ hat {\ math bf Ġ{ h }} Ġ\\ Ġ\ end { b mat rix } Ġ= Ġ\ tan h Ġ{\ left (\ math bf Ġ{ U } _ \ text { T } Ġ\ math bf Ġ{ e } Ġ+ Ġ\ math bf Ġ{ a } _ \ text { T }\ right )} $$ Ġ Ġ Ġ( E q . Ġ13 ) </s> while Ġnon - leaf Ġnodes Ġare Ġdefined Ġby Ġthe Ġfollowing : </s> $$ Ġ Ġ\ begin { b mat rix } Ġ\ hat {\ math bf Ġ{ i }} Ġ\\ Ġ\ hat {\ math bf Ġ{ f }} _ l Ġ\\ Ġ\ hat {\ math bf Ġ{ f }} _ r Ġ\\ Ġ\ hat {\ math bf Ġ{ o }} Ġ\\ Ġ\ hat {\ math bf Ġ{ g }} Ġ\ end { b mat rix } Ġ= Ġ\ begin { b mat rix } Ġ\ s igma Ġ\\ Ġ\ s igma Ġ\\ Ġ\ s igma Ġ\\ Ġ\ s igma Ġ\\ Ġ\ tan h Ġ\ end { b mat rix } Ġ\ B igg Ġ( Ġ\ math bf Ġ{ W _ \ text { T }} Ġ\ begin { b mat rix } Ġ\ hat {\ math bf Ġ{ h }} _ l Ġ\\ Ġ\ hat {\ math bf Ġ{ h }} _ r Ġ\\ Ġ\ math bf Ġ{ e } Ġ\\ Ġ\ end { b mat rix } Ġ+ Ġ\ math bf Ġ{ b } _ \ text { T } Ġ\ B igg Ġ) $$ Ġ Ġ Ġ( E q . Ġ14 ) </s> $$ Ġ Ġ\ hat {\ math bf Ġ{ c }} Ġ= Ġ\ hat {\ math bf Ġ{ f }} _ l Ġ\ od ot Ġ\ hat {\ math bf Ġ{ c }} _ l Ġ+ Ġ\ hat {\ math bf Ġ{ f }} _ r Ġ\ od ot Ġ\ hat {\ math bf Ġ{ c }} _ r Ġ+ Ġ\ hat {\ math bf Ġ{ i }} Ġ\ od ot Ġ\ hat {\ math bf Ġ{ g }} \\ $$ Ġ Ġ Ġ( E q . Ġ15 ) </s> where Ġ$\ hat {\ math bf Ġ{ h } }, Ġ\ hat {\ math bf Ġ{ c }} Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ \ text { T }} $ Ġrepresent Ġthe Ġhidden Ġstate Ġand Ġcell Ġstate Ġof Ġeach Ġnode Ġin Ġthe Ġtag Ġtree - L ST M . ĠWe Ġregard Ġthe Ġhidden Ġstate Ġ( Ġ$\ hat {\ math bf Ġ{ h }} $ Ġ) Ġas Ġa Ġstructure - aware Ġtag Ġrepresentation Ġfor Ġthe Ġnode . Ġ$ Ġ\ math bf Ġ{ U } _ \ text { T } Ġ\ in Ġ\ math bb Ġ{ R } ^{ 2 d _ \ text { T } Ġ\ times Ġd _ \ text { T } }, Ġ\ text bf Ġ{ a } _ \ text { T } Ġ\ in Ġ\ math bb Ġ{ R } ^{ 2 d _ \ text { T } }, Ġ\ math bf Ġ{ W } _ \ text { T } Ġ\ in Ġ\ math bb Ġ{ R } ^{ 5 d _ \ text { T } Ġ\ times Ġ3 d _ \ text { T }} $ Ġ, Ġand Ġ$\ math bf Ġ{ b } _ \ text { T } Ġ\ in Ġ\ math bb Ġ{ R } ^{ 5 d _ \ text { T }} $ Ġare Ġtrain able Ġparameters . ĠThe Ġrest Ġof Ġthe Ġnotation Ġfollows Ġequations Ġ8 Ġ, Ġ9 Ġ, Ġand Ġ10 Ġ. ĠIn Ġcase Ġof Ġleaf Ġnodes , Ġthe Ġstates Ġare Ġcomputed Ġby Ġa Ġsimple Ġnon - linear Ġtransformation . ĠMeanwhile , Ġthe Ġcomposition Ġfunction Ġin Ġa Ġnon - leaf Ġnode Ġabsorbs Ġthe Ġtag Ġembed ding Ġ( Ġ$\ math bf Ġ{ e } $ Ġ) Ġas Ġan Ġadditional Ġinput Ġas Ġwell Ġas Ġthe Ġhidden Ġstates Ġof Ġthe Ġtwo Ġchildren Ġnodes . ĠThe Ġbenefit Ġof Ġrev ising Ġtag Ġrepresentations Ġaccording Ġto Ġthe Ġinternal Ġstructure Ġis Ġthat Ġthe Ġderived Ġembed ding Ġis Ġa Ġfunction Ġof Ġthe Ġcorresponding Ġmakeup Ġof Ġthe Ġnode , Ġrather Ġthan Ġa Ġmon olithic , Ġcateg orical Ġtag . </s> With Ġregard Ġto Ġthe Ġtags Ġthemselves , Ġwe Ġconjecture Ġthat Ġthe Ġtax onomy Ġof Ġthe Ġtags Ġcurrently Ġin Ġuse Ġin Ġmany ĠN LP Ġsystems Ġis Ġtoo Ġcomplex Ġto Ġbe Ġutilized Ġeffectively Ġin Ġdeep Ġneural Ġmodels , Ġconsidering Ġthe Ġspecificity Ġof Ġmany Ġtag Ġsets Ġand Ġthe Ġlimited Ġamount Ġof Ġdata Ġwith Ġwhich Ġto Ġtrain . ĠThus , Ġwe Ġcluster ĠPOS Ġ( word - level ) Ġtags Ġinto Ġ12 Ġgroups Ġfollowing Ġthe Ġuniversal ĠPOS Ġtag set ĠB IB REF 20 Ġand Ġphrase - level Ġtags Ġinto Ġ11 Ġgroups Ġaccording Ġto Ġcriteria Ġanalogous Ġto Ġthe Ġcase Ġof Ġwords , Ġresulting Ġin Ġ23 Ġtag Ġcategories Ġin Ġtotal . ĠIn Ġthis Ġwork , Ġwe Ġuse Ġthe Ġrevised Ġcoarse - gr ained Ġtags Ġinstead Ġof Ġthe Ġoriginal Ġones . ĠFor Ġmore Ġdetails , Ġwe Ġrefer Ġreaders Ġto Ġthe Ġsupplemental Ġmaterials . </s> Le af - L ST M </s> An Ġinherent Ġshort coming Ġof ĠR v NN s Ġrelative Ġto Ġsequential Ġmodels Ġis Ġthat Ġeach Ġintermediate Ġrepresentation Ġin Ġa Ġtree Ġis Ġunaware Ġof Ġits Ġexternal Ġcontext Ġuntil Ġall Ġthe Ġinformation Ġis Ġgathered Ġtogether Ġat Ġthe Ġroot Ġnode . ĠIn Ġother Ġwords , Ġeach Ġcomposition Ġprocess Ġis Ġprone Ġto Ġbe Ġlocally Ġoptimized Ġrather Ġthan Ġglobally Ġoptimized . </s> To Ġmitigate Ġthis Ġproblem , Ġwe Ġpropose Ġusing Ġa Ġleaf - L ST M Ġfollowing Ġthe Ġconvention Ġof Ġsome Ġprevious Ġworks ĠB IB REF 21 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 11 Ġ, Ġwhich Ġis Ġa Ġtypical ĠL ST M Ġthat Ġaccepts Ġa Ġsequence Ġof Ġwords Ġin Ġorder . ĠInstead Ġof Ġleveraging Ġword Ġembed d ings Ġdirectly , Ġwe Ġcan Ġuse Ġeach Ġhidden Ġstate Ġand Ġcell Ġstate Ġof Ġthe Ġleaf - L ST M Ġas Ġinput Ġtokens Ġfor Ġleaf Ġnodes Ġin Ġa Ġtree - L ST M , Ġanticipating Ġthe Ġproper Ġcontextual ization Ġof Ġthe Ġinput Ġsequence . </s> Form ally , Ġwe Ġdenote Ġa Ġsequence Ġof Ġwords Ġin Ġan Ġinput Ġsentence Ġas Ġ$ w _{ 1 : n } $ Ġ( Ġ$ n $ Ġ: Ġthe Ġlength Ġof Ġthe Ġsentence ), Ġand Ġthe Ġcorresponding Ġword Ġembed d ings Ġas Ġ$\ math bf Ġ{ x } _{ 1 : n } $ Ġ. ĠThen , Ġthe Ġoperation Ġof Ġthe Ġleaf - L ST M Ġat Ġtime Ġ$ t $ Ġcan Ġbe Ġformulated Ġas , </s> $$ Ġ Ġ\ begin { b mat rix } Ġ\ t ilde {\ math bf Ġ{ i }} Ġ\\ Ġ\ t ilde {\ math bf Ġ{ f }} Ġ\\ Ġ\ t ilde {\ math bf Ġ{ o }} Ġ\\ Ġ\ t ilde {\ math bf Ġ{ g }} Ġ\ end { b mat rix } Ġ= Ġ\ begin { b mat rix } Ġ\ s igma Ġ\\ Ġ\ s igma Ġ\\ Ġ\ s igma Ġ\\ Ġ\ tan h Ġ\ end { b mat rix } Ġ\ B igg Ġ( Ġ\ math bf Ġ{ W } _ \ text { L } Ġ\ begin { b mat rix } Ġ\ t ilde {\ math bf Ġ{ h }} _{ t - 1 } Ġ\\ Ġ\ math bf Ġ{ x } _ t Ġ\\ Ġ\ end { b mat rix } Ġ+ Ġ\ math bf Ġ{ b } _ \ text { L } Ġ\ B igg Ġ) $$ Ġ Ġ Ġ( E q . Ġ18 ) </s> $$ Ġ Ġ\ t ilde {\ math bf Ġ{ c }} _ t Ġ= Ġ\ t ilde {\ math bf Ġ{ f }} Ġ\ od ot Ġ\ t ilde {\ math bf Ġ{ c }} _{ t - 1 } Ġ+ Ġ\ t ilde {\ math bf Ġ{ i }} Ġ\ od ot Ġ\ t ilde {\ math bf Ġ{ g }} \\ $$ Ġ Ġ Ġ( E q . Ġ19 ) </s> where Ġ$\ math bf Ġ{ x } _ t Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ w } $ Ġindicates Ġan Ġinput Ġword Ġvector Ġand Ġ$\ t ilde {\ math bf Ġ{ h }} _ t $ Ġ, Ġ$\ t ilde {\ math bf Ġ{ c }} _ t Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ h } $ Ġrepresent Ġthe Ġhidden Ġand Ġcell Ġstate Ġof Ġthe ĠL ST M Ġat Ġtime Ġ$ t $ Ġ( Ġ$\ t ilde {\ math bf Ġ{ h }} _{ t - 1 } $ Ġcorresponds Ġto Ġthe Ġhidden Ġstate Ġat Ġtime Ġ$ t $ Ġ- 1 ). Ġ$\ math bf Ġ{ W } _ \ text { L } $ Ġand Ġ$\ math bf Ġ{ b } _ \ text { L } Ġ$ Ġare Ġlearn able Ġparameters . ĠThe Ġremaining Ġnotation Ġfollows Ġthat Ġof Ġthe Ġtree - L ST M Ġabove . </s> In Ġexperiments , Ġwe Ġdemonstrate Ġthat Ġintroducing Ġa Ġleaf - L ST M Ġfares Ġbetter Ġat Ġprocessing Ġthe Ġinput Ġwords Ġof Ġa Ġtree - L ST M Ġcompared Ġto Ġusing Ġa Ġfeed - forward Ġneural Ġnetwork . ĠWe Ġalso Ġexplore Ġthe Ġpossibility Ġof Ġits Ġbid irection al Ġsetting Ġin Ġab lation Ġstudy . </s> S ATA ĠTree - L ST M </s> In Ġthis Ġsection , Ġwe Ġdefine ĠSATA ĠTree - L ST M Ġ( St ructure - A ware ĠTag ĠAug mented ĠTree - L ST M , Ġsee ĠFigure Ġ2 Ġ) Ġwhich Ġjoins Ġa Ġtag - level Ġtree - L ST M Ġ( section Ġ3 . 2 ), Ġa Ġleaf - L ST M Ġ( section Ġ3 . 3 ), Ġand Ġthe Ġoriginal Ġword Ġtree - L ST M Ġtogether . </s> As Ġabove Ġwe Ġdenote Ġa Ġsequence Ġof Ġwords Ġin Ġan Ġinput Ġsentence Ġas Ġ$ w _{ 1 : n } $ Ġand Ġthe Ġcorresponding Ġword Ġembed d ings Ġas Ġ$\ math bf Ġ{ x } _{ 1 : n } $ Ġ. ĠIn Ġaddition , Ġa Ġtag Ġembed ding Ġfor Ġthe Ġtag Ġattached Ġto Ġeach Ġnode Ġin Ġa Ġtree Ġis Ġden oted Ġby Ġ$\ text bf Ġ{ e } Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ \ text { T }} $ Ġ. ĠThen , Ġwe Ġderive Ġthe Ġfinal Ġsentence Ġrepresentation Ġfor Ġthe Ġinput Ġsentence Ġwith Ġour Ġmodel Ġin Ġtwo Ġsteps . </s> First , Ġwe Ġcompute Ġstructure - aware Ġtag Ġrepresentations Ġ( Ġ$\ hat {\ math bf Ġ{ h }} $ Ġ) Ġfor Ġeach Ġnode Ġof Ġa Ġtree Ġusing Ġthe Ġtag Ġtree - L ST M Ġ( the Ġright Ġside Ġof ĠFigure Ġ2 Ġ) Ġas Ġfollows : </s> $$ Ġ Ġ\ begin { b mat rix } Ġ\ hat {\ math bf Ġ{ c }} Ġ\\ Ġ\ hat {\ math bf Ġ{ h }} Ġ\\ Ġ\ end { b mat rix } Ġ= Ġ{\ left \ l brace Ġ\ begin { array }{ ll } Ġ\ text { Tag - Tree - L ST M } (\ math bf Ġ{ e }) Ġ& Ġ\ text { if Ġa Ġleaf Ġnode } Ġ\\ Ġ\ text { Tag - Tree - L ST M } (\ hat {\ math bf Ġ{ h }} _ l , Ġ\ hat {\ math bf Ġ{ h }} _ r , Ġ\ math bf Ġ{ e }) Ġ& Ġ\ text { other wise } Ġ\ end { array }\ right .} $$ Ġ Ġ Ġ( E q . Ġ23 ) </s> where ĠTag - Tree - L ST M Ġindicates Ġthe Ġmodule Ġwe Ġdescribed Ġin Ġsection Ġ3 . 2 . </s> Second , Ġwe Ġcombine Ġsemantic Ġunits Ġrec urs ively Ġon Ġthe Ġword Ġtree - L ST M Ġin Ġa Ġbottom - up Ġfashion . ĠFor Ġleaf Ġnodes , Ġwe Ġleverage Ġthe ĠLeaf - L ST M Ġ( the Ġbottom - left Ġof ĠFigure Ġ2 Ġ, Ġexplained Ġin Ġsection Ġ3 . 3 ) Ġto Ġcompute Ġ$\ t ilde {\ math bf Ġ{ c }} _{ t } $ Ġand Ġ$\ t ilde {\ math bf Ġ{ h }} _{ t } $ Ġin Ġsequential Ġorder , Ġwith Ġthe Ġcorresponding Ġinput Ġ$\ math bf Ġ{ x } _ t $ Ġ. </s> $$ Ġ Ġ\ begin { b mat rix } Ġ\ t ilde {\ math bf Ġ{ c }} _{ t } Ġ\\ Ġ\ t ilde {\ math bf Ġ{ h }} _{ t } Ġ\\ Ġ\ end { b mat rix } Ġ= Ġ\ text { Le af - L ST M } (\ t ilde {\ text bf Ġ{ h }} _{ t - 1 }, Ġ\ text bf Ġ{ x } _ t ) $$ Ġ Ġ Ġ( E q . Ġ24 ) </s> Then , Ġthe Ġ$\ t ilde {\ math bf Ġ{ c }} _{ t } $ Ġand Ġ$\ t ilde {\ math bf Ġ{ h }} _{ t } $ Ġcan Ġbe Ġutilized Ġas Ġinput Ġtokens Ġto Ġthe Ġword Ġtree - L ST M , Ġwith Ġthe Ġleft Ġ( right ) Ġchild Ġof Ġthe Ġtarget Ġnode Ġcorresponding Ġto Ġthe Ġ$ t $ Ġth Ġword Ġin Ġthe Ġinput Ġsentence . </s> $$ Ġ Ġ\ begin { b mat rix } Ġ\ check {\ text bf Ġ{ c }} _{ \ l brace Ġl , Ġr \ r brace Ġ} Ġ\\ Ġ\ check {\ text bf Ġ{ h }} _{ \ l brace Ġl , Ġr \ r brace Ġ} Ġ\ end { b mat rix } Ġ= Ġ\ begin { b mat rix } Ġ\ t ilde {\ text bf Ġ{ c }} _{ t } Ġ\\ Ġ\ t ilde {\ text bf Ġ{ h }} _{ t } Ġ\ end { b mat rix } $$ Ġ Ġ Ġ( E q . Ġ25 ) </s> In Ġthe Ġnon - leaf Ġnode Ġcase , Ġwe Ġcalculate Ġphrase Ġrepresentations Ġfor Ġeach Ġnode Ġin Ġthe Ġword Ġtree - L ST M Ġ( the Ġupper - left Ġof ĠFigure Ġ2 Ġ) Ġrec urs ively Ġas Ġfollows : </s> $$ Ġ Ġ\ check {\ math bf Ġ{ g }} Ġ= Ġ\ tan h Ġ{\ left ( Ġ\ math bf Ġ{ U } _ \ text { w } Ġ\ begin { b mat rix } Ġ\ check {\ math bf Ġ{ h }} _ l Ġ\\ Ġ\ check {\ math bf Ġ{ h }} _ r Ġ\\ Ġ\ end { b mat rix } Ġ+ Ġ\ math bf Ġ{ a } _ \ text { w } Ġ\ right )} $$ Ġ Ġ Ġ( E q . Ġ26 ) </s> $$ Ġ Ġ\ begin { b mat rix } Ġ\ check {\ math bf Ġ{ i }} Ġ\\ Ġ\ check {\ math bf Ġ{ f }} _ l Ġ\\ Ġ\ check {\ math bf Ġ{ f }} _ r Ġ\\ Ġ\ check {\ math bf Ġ{ o }} Ġ\ end { b mat rix } Ġ= Ġ\ begin { b mat rix } Ġ\ s igma Ġ\\ Ġ\ s igma Ġ\\ Ġ\ s igma Ġ\\ Ġ\ s igma Ġ\ end { b mat rix } Ġ\ B igg Ġ( Ġ\ math bf Ġ{ W _ \ text { w }} Ġ\ begin { b mat rix } Ġ\ check {\ math bf Ġ{ h }} _ l Ġ\\ Ġ\ check {\ math bf Ġ{ h }} _ r Ġ\\ Ġ\ hat {\ math bf Ġ{ h }} Ġ\\ Ġ\ end { b mat rix } Ġ+ Ġ\ math bf Ġ{ b } _ \ text { w } Ġ\ B igg Ġ) $$ Ġ Ġ Ġ( E q . Ġ27 ) </s> where Ġ$\ check {\ math bf Ġ{ h }} $ Ġ, Ġ$\ check {\ math bf Ġ{ c }} Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ h } $ Ġrepresent Ġthe Ġhidden Ġand Ġcell Ġstate Ġof Ġeach Ġnode Ġin Ġthe Ġword Ġtree - L ST M . Ġ$\ math bf Ġ{ U } _ \ text { w } Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ h Ġ\ times Ġ2 d _ h } $ Ġ, Ġ$\ math bf Ġ{ W } _ \ text { w } Ġ\ in Ġ\ math bb Ġ{ R } ^{ 4 d _ h Ġ\ times Ġ\ left ( 2 d _ h + d _ \ text { T }\ right )} $ Ġ, Ġ$\ math bf Ġ{ a } _ \ text { w } Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ h } $ Ġ, Ġ$\ math bf Ġ{ b } _ \ text { w } Ġ\ in Ġ\ math bb Ġ{ R } ^{ 4 d _ h } $ Ġare Ġlearned Ġparameters . ĠThe Ġremaining Ġnotation Ġfollows Ġthose Ġof Ġthe Ġprevious Ġsections . ĠNote Ġthat Ġthe Ġstructure - aware Ġtag Ġrepresentations Ġ( Ġ$\ hat {\ math bf Ġ{ h }} $ Ġ) Ġare Ġonly Ġutilized Ġto Ġcontrol Ġthe Ġgate Ġfunctions Ġof Ġthe Ġword Ġtree - L ST M Ġin Ġthe Ġform Ġof Ġadditional Ġinputs , Ġand Ġare Ġnot Ġinvolved Ġin Ġthe Ġsemantic Ġcomposition Ġ( Ġ$\ check {\ math bf Ġ{ g }} $ Ġ) Ġdirectly . </s> Finally , Ġthe Ġhidden Ġstate Ġof Ġthe Ġroot Ġnode Ġ( Ġ$\ check {\ math bf Ġ{ h }} _ \ text { root } $ Ġ) Ġin Ġthe Ġword - level Ġtree - L ST M Ġbecomes Ġthe Ġfinal Ġsentence Ġrepresentation Ġof Ġthe Ġinput Ġsentence . </s> Quant itative ĠAnalysis </s> One Ġof Ġthe Ġmost Ġbasic Ġapproaches Ġto Ġevaluate Ġa Ġsentence Ġenc oder Ġis Ġto Ġmeasure Ġthe Ġclassification Ġperformance Ġwith Ġthe Ġsentence Ġrepresentations Ġmade Ġby Ġthe Ġenc oder . ĠThus , Ġwe Ġconduct Ġexperiments Ġon Ġthe Ġfollowing Ġfive Ġdatasets . Ġ( Summary Ġstatistics Ġfor Ġthe Ġdatasets Ġare Ġreported Ġin Ġthe Ġsupplemental Ġmaterials .) </s> MR : ĠA Ġgroup Ġof Ġmovie Ġreviews Ġwith Ġbinary Ġ( positive Ġ/ Ġnegative ) Ġclasses . ĠB IB REF 22 </s> S ST - 2 : ĠStanford ĠSent iment ĠTree bank ĠB IB REF 6 Ġ. ĠSimilar Ġto ĠMR , Ġbut Ġeach Ġreview Ġis Ġprovided Ġin Ġthe Ġform Ġof Ġa Ġbinary Ġparse Ġtree Ġwhose Ġnodes Ġare Ġannot ated Ġwith Ġnumeric Ġsentiment Ġvalues . ĠFor ĠS ST - 2 , Ġwe Ġonly Ġconsider Ġbinary Ġ( positive Ġ/ Ġnegative ) Ġclasses . </s> S ST - 5 : ĠIdent ical Ġto ĠS ST - 2 , Ġbut Ġthe Ġreviews Ġare Ġgrouped Ġinto Ġfine - gr ained Ġ( very Ġnegative , Ġnegative , Ġneutral , Ġpositive , Ġvery Ġpositive ) Ġclasses . </s> S UB J : ĠSent ences Ġgrouped Ġas Ġbeing Ġeither Ġsubjective Ġor Ġobjective Ġ( binary Ġclasses ). ĠB IB REF 23 </s> T REC : ĠA Ġdataset Ġwhich Ġgroups Ġquestions Ġinto Ġsix Ġdifferent Ġquestion Ġtypes Ġ( classes ). ĠB IB REF 24 </s> As Ġa Ġpre processing Ġstep , Ġwe Ġconstruct Ġparse Ġtrees Ġfor Ġthe Ġsentences Ġin Ġthe Ġdatasets Ġusing Ġthe ĠStanford ĠPC FG Ġparser ĠB IB REF 25 Ġ. ĠBecause Ġsynt actic Ġtags Ġare Ġby - products Ġof Ġconstituency Ġparsing , Ġwe Ġdo Ġnot Ġneed Ġfurther Ġpre processing . </s> To Ġclassify Ġthe Ġsentence Ġgiven Ġour Ġsentence Ġrepresentation Ġ( Ġ$\ check {\ math bf Ġ{ h }} _ \ text { root } $ Ġ), Ġwe Ġuse Ġone Ġfully - connected Ġlayer Ġwith Ġa ĠRe LU Ġactivation , Ġfollowed Ġby Ġa Ġsoft max Ġclass ifier . ĠThe Ġfinal Ġpredicted Ġprobability Ġdistribution Ġof Ġthe Ġclass Ġ$ y $ Ġgiven Ġthe Ġsentence Ġ$ w _{ 1 : n } $ Ġis Ġdefined Ġas Ġfollows , </s> $$ \ math bf Ġ{ s } Ġ= Ġ\ text { Re LU } (\ math bf Ġ{ W } _ \ text { s } Ġ\ check {\ math bf Ġ{ h }} _ \ text { root } + Ġ\ math bf Ġ{ b } _ \ text { s }) $$ Ġ Ġ Ġ( E q . Ġ37 ) </s> $$ p ( y | w _{ 1 : n }) Ġ= Ġ\ text { soft max } (\ math bf Ġ{ W } _ \ text { c }\ math bf Ġ{ s } Ġ+ Ġ\ math bf Ġ{ b } _ \ text { c }) $$ Ġ Ġ Ġ( E q . Ġ38 ) </s> where Ġ$\ text bf Ġ{ s } Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ \ text { s }} $ Ġis Ġthe Ġcomputed Ġtask - specific Ġsentence Ġrepresentation Ġfor Ġthe Ġclass ifier , Ġand Ġ$\ text bf Ġ{ W } _ \ text { s } Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ \ text { s } Ġ\ times Ġd _ h } $ Ġ, Ġ$\ text bf Ġ{ W } _ \ text { c } Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ \ text { c } Ġ\ times Ġd _ s } $ Ġ, Ġ$\ text bf Ġ{ b } _ \ text { s } Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ s } $ Ġ, Ġ$\ text bf Ġ{ b } _ \ text { c } Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ c } $ Ġare Ġtrain able Ġparameters . ĠAs Ġan Ġobjective Ġfunction , Ġwe Ġuse Ġthe Ġcross Ġentropy Ġof Ġthe Ġpredicted Ġand Ġtrue Ġclass Ġdistributions . </s> The Ġresults Ġof Ġthe Ġexperiments Ġon Ġthe Ġfive Ġdatasets Ġare Ġshown Ġin Ġtable Ġ1 Ġ. ĠIn Ġthis Ġtable , Ġwe Ġreport Ġthe Ġtest Ġaccuracy Ġof Ġour Ġmodel Ġand Ġvarious Ġother Ġmodels Ġon Ġeach Ġdataset Ġin Ġterms Ġof Ġpercentage . ĠTo Ġconsider Ġthe Ġeffects Ġof Ġrandom Ġinitialization , Ġwe Ġreport Ġthe Ġbest Ġnumbers Ġobtained Ġfrom Ġeach Ġseveral Ġruns Ġwith Ġhyper - param eters Ġfixed . </s> Compared Ġwith Ġthe Ġprevious Ġsynt actic Ġtree - based Ġmodels Ġas Ġwell Ġas Ġother Ġneural Ġmodels , Ġour ĠSATA ĠTree - L ST M Ġshows Ġsuperior Ġor Ġcompetitive Ġperformance Ġon Ġall Ġtasks . ĠSpecifically , Ġour Ġmodel Ġachieves Ġnew Ġstate - of - the - art Ġresults Ġwithin Ġthe Ġtree - struct ured Ġmodel Ġclass Ġon Ġ4 Ġout Ġof Ġ5 Ġsentence Ġclassification Ġtasks âĢĶ S ST - 2 , ĠS ST - 5 , ĠMR , Ġand ĠT REC . ĠThe Ġmodel Ġshows Ġits Ġstrength , Ġin Ġparticular , Ġwhen Ġthe Ġdatasets Ġprovide Ġphrase - level Ġsupervision Ġto Ġfacilitate Ġtree Ġstructure Ġlearning Ġ( i . e . ĠS ST - 2 , ĠS ST - 5 ). ĠMoreover , Ġthe Ġnumbers Ġwe Ġreport Ġfor ĠS ST - 5 Ġand ĠT REC Ġare Ġcompetitive Ġto Ġthe Ġexisting Ġstate - of - the - art Ġresults Ġincluding Ġones Ġfrom Ġstruct urally Ġpre - trained Ġmodels Ġsuch Ġas ĠEL Mo ĠB IB REF 26 Ġ, Ġproving Ġour Ġmodel 's Ġsuperiority . ĠNote Ġthat Ġthe ĠSATA ĠTree - L ST M Ġalso Ġoutper forms Ġthe Ġrecent Ġlatent Ġtree - based Ġmodel , Ġindicating Ġthat Ġmodeling Ġa Ġneural Ġmodel Ġwith Ġexplicit Ġlinguistic Ġknowledge Ġcan Ġbe Ġan Ġattractive Ġoption . </s> On Ġthe Ġother Ġhand , Ġa Ġremaining Ġconcern Ġis Ġthat Ġour ĠSATA ĠTree - L ST M Ġis Ġnot Ġrobust Ġto Ġrandom Ġseeds Ġwhen Ġthe Ġsize Ġof Ġa Ġdataset Ġis Ġrelatively Ġsmall , Ġas Ġtag Ġembed d ings Ġare Ġrandomly Ġinitialized Ġrather Ġthan Ġrelying Ġon Ġpre - trained Ġones Ġin Ġcontrast Ġwith Ġthe Ġcase Ġof Ġwords . ĠFrom Ġthis Ġobservation , Ġwe Ġcould Ġfind Ġout Ġthere Ġneeds Ġa Ġdirection Ġof Ġresearch Ġtowards Ġpre - trained Ġtag Ġembed d ings . </s> To Ġestimate Ġthe Ġperformance Ġof Ġour Ġmodel Ġbeyond Ġthe Ġtasks Ġrequiring Ġonly Ġone Ġsentence Ġat Ġa Ġtime , Ġwe Ġconduct Ġan Ġexperiment Ġon Ġthe ĠStanford ĠNatural ĠLanguage ĠIn ference ĠB IB REF 34 Ġdataset , Ġeach Ġexample Ġof Ġwhich Ġconsists Ġof Ġtwo Ġsentences , Ġthe Ġpremise Ġand Ġthe Ġhypothesis . ĠOur Ġobjective Ġgiven Ġthe Ġdata Ġis Ġto Ġpredict Ġthe Ġcorrect Ġrelationship Ġbetween Ġthe Ġtwo Ġsentences Ġamong Ġthree Ġoptions âĢĶ Ġcontradiction , Ġneutral , Ġor Ġentail ment . </s> We Ġuse Ġthe Ġsi ames e Ġarchitecture Ġto Ġencode Ġboth Ġthe Ġpremise Ġ( Ġ$ p _{ 1 : m } $ Ġ) Ġand Ġhypothesis Ġ( Ġ$ h _{ 1 : n } $ Ġ) Ġfollowing Ġthe Ġstandard Ġof Ġsentence - enc oding Ġmodels Ġin Ġthe Ġliterature . Ġ( Specifically , Ġ$ p _{ 1 : m } $ Ġis Ġencoded Ġas Ġ$\ check {\ math bf Ġ{ h }} _ \ text { root } ^ p Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ h } $ Ġand Ġ$ h _{ 1 : n } $ Ġis Ġencoded Ġas Ġ$\ check {\ math bf Ġ{ h }} _ \ text { root } ^ h Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ h } $ Ġwith Ġthe Ġsame Ġenc oder .) ĠThen , Ġwe Ġleverage Ġsome Ġhe ur istics ĠB IB REF 35 Ġ, Ġfollowed Ġby Ġone Ġfully - connected Ġlayer Ġwith Ġa ĠRe LU Ġactivation Ġand Ġa Ġsoft max Ġclass ifier . ĠSpecifically , </s> $$ \ math bf Ġ{ z } Ġ= Ġ\ left [ Ġ\ check {\ math bf Ġ{ h }} _ \ text { root } ^ p ; Ġ\ check {\ math bf Ġ{ h }} _ \ text { root } ^ h ; Ġ| Ġ\ check {\ math bf Ġ{ h }} _ \ text { root } ^ p Ġ- Ġ\ check {\ math bf Ġ{ h }} _ \ text { root } ^ h Ġ| ; Ġ\ check {\ math bf Ġ{ h }} _ \ text { root } ^ p Ġ\ od ot Ġ\ check {\ math bf Ġ{ h }} _ \ text { root } ^ h Ġ\ right ] $$ Ġ Ġ Ġ( E q . Ġ41 ) </s> $$ \ math bf Ġ{ s } Ġ= Ġ\ text { Re LU } (\ math bf Ġ{ W } _ \ text { s } Ġ\ math bf Ġ{ z } Ġ+ Ġ\ math bf Ġ{ b } _ \ text { s }) $$ Ġ Ġ Ġ( E q . Ġ42 ) </s> where Ġ$\ text bf Ġ{ z } Ġ\ in Ġ\ math bb Ġ{ R } ^{ 4 d _ h } $ Ġ, Ġ$\ text bf Ġ{ s } Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ s } $ Ġare Ġintermediate Ġfeatures Ġfor Ġthe Ġclass ifier Ġand Ġ$\ text bf Ġ{ W } _ \ text { s } Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ \ text { s } Ġ\ times Ġ4 d _ h } $ Ġ, Ġ$\ text bf Ġ{ W } _ \ text { c } Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ \ text { c } Ġ\ times Ġd _ s } $ Ġ, Ġ$\ text bf Ġ{ b } _ \ text { s } Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ s } $ Ġ, Ġ$\ text bf Ġ{ b } _ \ text { c } Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ c } $ Ġare Ġagain Ġtrain able Ġparameters . </s> Our Ġexperimental Ġresults Ġon Ġthe ĠSN LI Ġdataset Ġare Ġshown Ġin Ġtable Ġ2 Ġ. ĠIn Ġthis Ġtable , Ġwe Ġreport Ġthe Ġtest Ġaccuracy Ġand Ġnumber Ġof Ġtrain able Ġparameters Ġfor Ġeach Ġmodel . ĠOur ĠSATA - L ST M Ġagain Ġdemonstrates Ġits Ġdecent Ġperformance Ġcompared Ġagainst Ġthe Ġneural Ġmodels Ġbuilt Ġon Ġboth Ġsynt actic Ġtrees Ġand Ġlatent Ġtrees , Ġas Ġwell Ġas Ġthe Ġnon - tree Ġmodels . Ġ( Lat ent ĠSynt ax ĠTree - L ST M : ĠB IB REF 10 Ġ( ĠB IB REF 10 Ġ), ĠTree - based ĠCNN : ĠB IB REF 35 Ġ( ĠB IB REF 35 Ġ), ĠG umb el ĠTree - L ST M : ĠB IB REF 11 Ġ( ĠB IB REF 11 Ġ), ĠN SE : ĠB IB REF 36 Ġ( ĠB IB REF 36 Ġ), ĠRein forced ĠSelf - Att ention ĠNetwork : ĠB IB REF 4 Ġ( ĠB IB REF 4 Ġ), ĠRes id ual Ġstacked Ġenc od ers : ĠB IB REF 37 Ġ( ĠB IB REF 37 Ġ), ĠBi L ST M Ġwith Ġgeneralized Ġpool ing : ĠB IB REF 38 Ġ( ĠB IB REF 38 Ġ). ) ĠNote Ġthat Ġthe Ġnumber Ġof Ġlearned Ġparameters Ġin Ġour Ġmodel Ġis Ġalso Ġcomparable Ġto Ġother Ġsophisticated Ġmodels , Ġshowing Ġthe Ġefficiency Ġof Ġour Ġmodel . </s> Even Ġthough Ġour Ġmodel Ġhas Ġproven Ġits Ġmet tle , Ġthe Ġeffect Ġof Ġtag Ġinformation Ġseems Ġrelatively Ġweak Ġin Ġthe Ġcase Ġof ĠSN LI , Ġwhich Ġcontains Ġa Ġlarge Ġamount Ġof Ġdata Ġcompared Ġto Ġthe Ġothers . ĠOne Ġpossible Ġexplanation Ġis Ġthat Ġneural Ġmodels Ġmay Ġlearn Ġsome Ġsynt actic Ġrules Ġfrom Ġlarge Ġamounts Ġof Ġtext Ġwhen Ġthe Ġtext Ġsize Ġis Ġlarge Ġenough , Ġreducing Ġthe Ġnecessity Ġof Ġexternal Ġlinguistic Ġknowledge . ĠWe Ġleave Ġthe Ġexploration Ġof Ġthe Ġeffectiveness Ġof Ġtags Ġrelative Ġto Ġdata Ġsize Ġfor Ġfuture Ġwork . </s> Here Ġwe Ġgo Ġover Ġthe Ġsettings Ġcommon Ġacross Ġour Ġmodels Ġduring Ġexperimentation . ĠFor Ġmore Ġtask - specific Ġdetails , Ġrefer Ġto Ġthe Ġsupplemental Ġmaterials . </s> For Ġour Ġinput Ġembed d ings , Ġwe Ġused Ġ300 Ġdimensional Ġ840 B ĠGl o Ve ĠB IB REF 39 Ġas Ġpre - trained Ġword Ġembed d ings , Ġand Ġtag Ġrepresentations Ġwere Ġrandomly Ġsampled Ġfrom Ġthe Ġuniform Ġdistribution Ġ[- 0 . 005 , Ġ0 . 005 ]. ĠTag Ġvectors Ġare Ġrevised Ġduring Ġtraining Ġwhile Ġthe Ġfine - tun ing Ġof Ġthe Ġword Ġembed ding Ġdepends Ġon Ġthe Ġtask . ĠOur Ġmodels Ġwere Ġtrained Ġusing Ġthe ĠAdam ĠB IB REF 40 Ġor ĠAd ad elta ĠB IB REF 41 Ġoptim izer , Ġdepending Ġon Ġtask . ĠFor Ġregular ization , Ġweight Ġdecay Ġis Ġadded Ġto Ġthe Ġloss Ġfunction Ġexcept Ġfor ĠSN LI Ġfollowing ĠB IB REF 42 Ġ( ĠB IB REF 42 Ġ) Ġand ĠDrop out ĠB IB REF 43 Ġis Ġalso Ġapplied Ġfor Ġthe Ġword Ġembed d ings Ġand Ġtask - specific Ġclass ifiers . ĠMoreover , Ġbatch Ġnormal ization ĠB IB REF 44 Ġis Ġadopted Ġfor Ġthe Ġclass ifiers . ĠAs Ġa Ġdefault , Ġall Ġthe Ġweights Ġin Ġthe Ġmodel Ġare Ġinitialized Ġfollowing ĠB IB REF 45 Ġ( ĠB IB REF 45 Ġ) Ġand Ġthe Ġbiases Ġare Ġset Ġto Ġ0 . ĠThe Ġtotal Ġnorm Ġof Ġthe Ġgrad ients Ġof Ġthe Ġparameters Ġis Ġclipped Ġnot Ġto Ġbe Ġover Ġ5 Ġduring Ġtraining . </s> Our Ġbest Ġmodels Ġfor Ġeach Ġdataset Ġwere Ġchosen Ġby Ġvalidation Ġaccuracy Ġin Ġcases Ġwhere Ġa Ġvalidation Ġset Ġwas Ġprovided Ġas Ġa Ġpart Ġof Ġthe Ġdataset . ĠOtherwise , Ġwe Ġperform Ġa Ġgrid Ġsearch Ġon Ġprobable Ġhyper - param eter Ġsettings , Ġor Ġrun Ġ10 - fold Ġcross - valid ation Ġin Ġcases Ġwhere Ġeven Ġa Ġtest Ġset Ġdoes Ġnot Ġexist . </s> A bl ation ĠStudy </s> In Ġthis Ġsection , Ġwe Ġdesign Ġan Ġab lation Ġstudy Ġon Ġthe Ġcore Ġmodules Ġof Ġour Ġmodel Ġto Ġexplore Ġtheir Ġeffectiveness . ĠThe Ġdataset Ġused Ġin Ġthis Ġexperiment Ġis ĠS ST - 2 . ĠTo Ġconduct Ġthe Ġexperiment , Ġwe Ġonly Ġreplace Ġthe Ġtarget Ġmodule Ġwith Ġother Ġcandidates Ġwhile Ġmaintaining Ġthe Ġother Ġsettings . ĠTo Ġbe Ġspecific , Ġwe Ġfocus Ġon Ġtwo Ġmodules , Ġthe Ġleaf - L ST M Ġand Ġstructure - aware Ġtag Ġembed d ings Ġ( tag - level Ġtree - L ST M ). ĠIn Ġthe Ġfirst Ġcase , Ġthe Ġleaf - L ST M Ġis Ġreplaced Ġwith Ġa Ġfully - connected Ġlayer Ġwith Ġa Ġ$\ tan h Ġ$ Ġactivation Ġor ĠBi - L ST M . ĠIn Ġthe Ġsecond Ġcase , Ġwe Ġreplace Ġthe Ġstructure - aware Ġtag Ġembed d ings Ġwith Ġnaive Ġtag Ġembed d ings Ġor Ġdo Ġnot Ġemploy Ġthem Ġat Ġall . </s> The Ġexperimental Ġresults Ġare Ġdepicted Ġin ĠFigure Ġ3 Ġ. ĠAs Ġthe Ġchart Ġshows , Ġour Ġmodel Ġoutper forms Ġall Ġthe Ġother Ġoptions Ġwe Ġhave Ġconsidered . ĠIn Ġdetail , Ġthe Ġleft Ġpart Ġof Ġthe Ġchart Ġshows Ġthat Ġthe Ġleaf - L ST M Ġis Ġthe Ġmost Ġeffective Ġoption Ġcompared Ġto Ġits Ġcompetitors . ĠNote Ġthat Ġthe Ġsequential Ġleaf - L ST M Ġis Ġsomewhat Ġsuperior Ġor Ġcompetitive Ġthan Ġthe Ġbid irection al Ġleaf - L ST M Ġwhen Ġboth Ġhave Ġa Ġcomparable Ġnumber Ġof Ġparameters . ĠWe Ġconjecture Ġthis Ġmay Ġbecause Ġa Ġbackward ĠL ST M Ġdoes Ġnot Ġadd Ġadditional Ġuseful Ġknowledge Ġwhen Ġthe Ġstructure Ġof Ġa Ġsentence Ġis Ġalready Ġknown . ĠIn Ġconclusion , Ġwe Ġuse Ġthe Ġun i - direction al ĠL ST M Ġas Ġa Ġleaf Ġmodule Ġbecause Ġof Ġits Ġsimplicity Ġand Ġremarkable Ġperformance . </s> Meanwhile , Ġthe Ġright Ġpart Ġof Ġthe Ġfigure Ġdemonstrates Ġthat Ġour Ġnewly Ġintroduced Ġstructure - aware Ġembed d ings Ġhave Ġa Ġreal Ġimpact Ġon Ġimproving Ġthe Ġmodel Ġperformance . ĠInterestingly , Ġemploying Ġthe Ġnaive Ġtag Ġembed d ings Ġmade Ġno Ġdifference Ġin Ġterms Ġof Ġthe Ġtest Ġaccuracy , Ġeven Ġthough Ġthe Ġabsolute Ġvalidation Ġaccuracy Ġincreased Ġ( not Ġreported Ġin Ġthe Ġfigure ). ĠThis Ġresult Ġsupports Ġour Ġassumption Ġthat Ġtag Ġinformation Ġshould Ġbe Ġconsidered Ġin Ġthe Ġstructure . </s> Qual itative ĠAnalysis </s> In Ġprevious Ġsections , Ġwe Ġhave Ġnumer ically Ġdemonstrated Ġthat Ġour Ġmodel Ġis Ġeffective Ġin Ġencouraging Ġuseful Ġcomposition Ġof Ġsemantic Ġunits . ĠHere , Ġwe Ġdirectly Ġinvestigate Ġthe Ġcomputed Ġrepresentations Ġfor Ġeach Ġnode Ġof Ġa Ġtree , Ġshowing Ġthat Ġthe Ġremarkable Ġperformance Ġof Ġour Ġmodel Ġis Ġmainly Ġdue Ġto Ġthe Ġgradual Ġand Ġrecursive Ġcomposition Ġof Ġthe Ġintermediate Ġrepresentations Ġon Ġthe Ġsynt actic Ġstructure . </s> To Ġobserve Ġthe Ġphrase - level Ġembed d ings Ġat Ġa Ġglance , Ġwe Ġdraw Ġa Ġscatter Ġplot Ġin Ġwhich Ġa Ġpoint Ġrepresents Ġthe Ġcorresponding Ġintermediate Ġrepresentation . ĠWe Ġutilize ĠPC A Ġ( Princ ipal ĠComponent ĠAnalysis ) Ġto Ġproject Ġthe Ġrepresentations Ġinto Ġa Ġtwo - dimensional Ġvector Ġspace . ĠAs Ġa Ġtarget Ġparse Ġtree , Ġwe Ġreuse Ġthe Ġone Ġseen Ġin ĠFigure Ġ1 Ġ. ĠThe Ġresult Ġis Ġshown Ġin ĠFigure Ġ4 Ġ. </s> From Ġthis Ġfigure , Ġwe Ġconfirm Ġthat Ġthe Ġintermediate Ġrepresentations Ġhave Ġa Ġhierarchy Ġin Ġthe Ġsemantic Ġspace , Ġwhich Ġis Ġvery Ġsimilar Ġto Ġthat Ġof Ġthe Ġparse Ġtree . ĠIn Ġother Ġwords , Ġas Ġmany Ġtree - struct ured Ġmodels Ġpursue , Ġwe Ġcan Ġsee Ġthe Ġtendency Ġof Ġconstructing Ġthe Ġrepresentations Ġfrom Ġthe Ġlow - level Ġ( the Ġbottom Ġof Ġthe Ġfigure ) Ġto Ġthe Ġhigh - level Ġ( the Ġtop - left Ġand Ġtop - right Ġof Ġthe Ġfigure ), Ġintegrating Ġthe Ġmeaning Ġof Ġthe Ġconstituents Ġrec urs ively . ĠAn Ġinteresting Ġthing Ġto Ġnote Ġis Ġthat Ġthe Ġfinal Ġsentence Ġrepresentation Ġis Ġnear Ġthat Ġof Ġthe Ġphrase Ġ` , Ġthe Ġstories Ġare Ġquietly Ġmoving .' Ġrather Ġthan Ġthat Ġof Ġ` Despite Ġthe Ġfilm 's Ġshortcomings ', Ġcatching Ġthe Ġmain Ġmeaning Ġof Ġthe Ġsentence . </s> Conclusion </s> We Ġhave Ġproposed Ġa Ġnovel ĠR v NN Ġarchitecture Ġto Ġfully Ġutilize Ġlinguistic Ġpri ors . ĠA Ġnewly Ġintroduced Ġtag - level Ġtree - L ST M Ġdemonstrates Ġthat Ġit Ġcan Ġeffectively Ġcontrol Ġthe Ġcomposition Ġfunction Ġof Ġthe Ġcorresponding Ġword - level Ġtree - L ST M . ĠIn Ġaddition , Ġthe Ġproper Ġcontextual ization Ġof Ġthe Ġinput Ġword Ġvectors Ġresults Ġin Ġsignificant Ġperformance Ġimprovements Ġon Ġseveral Ġsentence - level Ġtasks . ĠFor Ġfuture Ġwork , Ġwe Ġplan Ġto Ġexplore Ġa Ġnew Ġway Ġof Ġexploiting Ġdependency Ġtrees Ġeffectively , Ġsimilar Ġto Ġthe Ġcase Ġof Ġconstituency Ġtrees . </s> Acknowled gments </s> We Ġthank Ġanonymous Ġreviewers Ġfor Ġtheir Ġconstructive Ġand Ġfruitful Ġcomments . ĠThis Ġwork Ġwas Ġsupported Ġby Ġthe ĠNational ĠResearch ĠFoundation Ġof ĠKorea Ġ( NR F ) Ġgrant Ġfunded Ġby Ġthe ĠKorea Ġgovernment Ġ( MS IT ) Ġ( NR F 2016 M 3 C 4 A 79 525 87 ).
Introduction One of the most fundamental topics in natural language processing is how best to derive high-level representations from constituent parts, as natural language meanings are a function of their constituent parts. How best to construct a sentence representation from distributed word embeddings is an example domain of this larger issue. Even though sequential neural models such as recurrent neural networks (RNN) BIBREF0 and their variants including Long Short-Term Memory (LSTM) BIBREF1 and Gated Recurrent Unit (GRU) BIBREF2 have become the de-facto standard for condensing sentence-level information from a sequence of words into a fixed vector, there have been many lines of research towards better sentence representation using other neural architectures, e.g. convolutional neural networks (CNN) BIBREF3 or self-attention based models BIBREF4 . From a linguistic point of view, the underlying tree structure—as expressed by its constituency and dependency trees—of a sentence is an integral part of its meaning. Inspired by this fact, some recursive neural network (RvNN) models are designed to reflect the syntactic tree structure, achieving impressive results on several sentence-level tasks such as sentiment analysis BIBREF5 , BIBREF6 , machine translation BIBREF7 , natural language inference BIBREF8 , and discourse relation classification BIBREF9 . However, some recent works have BIBREF10 , BIBREF11 proposed latent tree models, which learn to construct task-specific tree structures without explicit supervision, bringing into question the value of linguistically-motivated recursive neural models. Witnessing the surprising performance of the latent tree models on some sentence-level tasks, there arises a natural question: Are linguistic tree structures the optimal way of composing sentence representations for NLP tasks? In this paper, we demonstrate that linguistic priors are in fact useful for devising effective neural models for sentence representations, showing that our novel architecture based on constituency trees and their tag information obtains superior performance on several sentence-level tasks, including sentiment analysis and natural language inference. A chief novelty of our approach is that we introduce a small separate tag-level tree-LSTM to control the composition function of the existing word-level tree-LSTM, which is in charge of extracting helpful syntactic signals for meaningful semantic composition of constituents by considering both the structures and linguistic tags of constituency trees simultaneously. In addition, we demonstrate that applying a typical LSTM to preprocess the leaf nodes of a tree-LSTM greatly improves the performance of the tree models. Moreover, we propose a clustered tag set to replace the existing tags on the assumption that the original syntactic tags are too fined-grained to be useful in neural models. In short, our contributions in this work are as follows: Related Work Recursive neural networks (RvNN) are a kind of neural architecture which model sentences by exploiting syntactic structure. While earlier RvNN models proposed utilizing diverse composition functions, including feed-forward neural networks BIBREF12 , matrix-vector multiplication BIBREF5 , and tensor computation BIBREF6 , tree-LSTMs BIBREF13 remain the standard for several sentence-level tasks. Even though classic RvNNs have demonstrated superior performance on a variety of tasks, their inflexibility, i.e. their inability to handle dynamic compositionality for different syntactic configurations, is a considerable weakness. For instance, it would be desirable if our model could distinguish e.g. adjective-noun composition from that of verb-noun or preposition-noun composition, as models failing to make such a distinction ignore real-world syntactic considerations such as `-arity' of function words (i.e. types), and the adjunct/argument distinction. To enable dynamic compositionality in recursive neural networks, many previous works BIBREF14 , BIBREF15 , BIBREF16 , BIBREF9 , BIBREF17 , BIBREF18 , BIBREF19 have proposed various methods. One main direction of research leverages tag information, which is produced as a by-product of parsing. In detail, BIBREF16 ( BIBREF16 ) suggested TG-RNN, a model employing different composition functions according to POS tags, and TE-RNN/TE-RNTN, models which leverage tag embeddings as additional inputs for the existing tree-structured models. Despite the novelty of utilizing tag information, the explosion of the number of parameters (in case of the TG-RNN) and the limited performance of the original models (in case of the TE-RNN/TE-RNTN) have prevented these models from being widely adopted. Meanwhile, BIBREF9 ( BIBREF9 ) and BIBREF18 ( BIBREF18 ) proposed models based on a tree-LSTM which also uses the tag vectors to control the gate functions of the tree-LSTM. In spite of their impressive results, there is a limitation that the trained tag embeddings are too simple to reflect the rich information which tags provide in different syntactic structures. To alleviate this problem, we introduce structure-aware tag representations in the next section. Another way of building dynamic compositionality into RvNNs is to take advantage of a meta-network (or hyper-network). Inspired by recent works on dynamic parameter prediction, DC-TreeLSTMs BIBREF17 dynamically create the parameters for compositional functions in a tree-LSTM. Specifically, the model has two separate tree-LSTM networks whose architectures are similar, but the smaller of the two is utilized to calculate the weights of the bigger one. A possible problem for this model is that it may be easy to be trained such that the role of each tree-LSTM is ambiguous, as they share the same input, i.e. word information. Therefore, we design two disentangled tree-LSTMs in our model so that one focuses on extracting useful features from only syntactic information while the other composes semantic units with the aid of the features. Furthermore, our model reduces the complexity of computation by utilizing typical tree-LSTM frameworks instead of computing the weights for each example. Finally, some recent works BIBREF10 , BIBREF11 have proposed latent tree-structured models that learn how to formulate tree structures from only sequences of tokens, without the aid of syntactic trees or linguistic information. The latent tree models have the advantage of being able to find the optimized task-specific order of composition rather than a sequential or syntactic one. In experiments, we compare our model with not only syntactic tree-based models but also latent tree models, demonstrating that modeling with explicit linguistic knowledge can be an attractive option. Model In this section, we introduce a novel RvNN architecture, called SATA Tree-LSTM (Structure-Aware Tag Augmented Tree-LSTM). This model is similar to typical Tree-LSTMs, but provides dynamic compositionality by augmenting a separate tag-level tree-LSTM which produces structure-aware tag representations for each node in a tree. In other words, our model has two independent tree-structured modules based on the same constituency tree, one of which (word-level tree-LSTM) is responsible for constructing sentence representations given a sequence of words as usual, while the other (tag-level tree-LSTM) provides supplementary syntactic information to the former. In section 3.1, we first review tree-LSTM architectures. Then in section 3.2, we introduce a tag-level tree-LSTM and structure-aware tag representations. In section 3.3, we discuss an additional technique to boost the performance of tree-structured models, and in section 3.4, we describe the entire architecture of our model in detail. Tree-LSTM The LSTM BIBREF1 architecture was first introduced as an extension of the RNN architecture to mitigate the vanishing and exploding gradient problems. In addition, several works have discovered that applying the LSTM cell into tree structures can be an effective means of modeling sentence representations. To be formal, the composition function of the cell in a tree-LSTM can be formulated as follows: $$  \begin{bmatrix} \mathbf {i} \\ \mathbf {f}_l \\ \mathbf {f}_r \\ \mathbf {o} \\ \mathbf {g} \end{bmatrix} = \begin{bmatrix} \sigma \\ \sigma \\ \sigma \\ \sigma \\ \tanh \end{bmatrix} \Bigg ( \mathbf {W} \begin{bmatrix} \mathbf {h}_l \\ \mathbf {h}_r \\ \end{bmatrix} + \mathbf {b} \Bigg )$$   (Eq. 8) $$  \mathbf {c} = \mathbf {f}_l \odot \mathbf {c}_l + \mathbf {f}_r \odot \mathbf {c}_r + \mathbf {i} \odot \mathbf {g}\\$$   (Eq. 9) where $\mathbf {h}, \mathbf {c} \in \mathbb {R}^{d}$ indicate the hidden state and cell state of the LSTM cell, and $\mathbf {h}_l, \mathbf {h}_r, \mathbf {c}_l, \mathbf {c}_r \in \mathbb {R}^{d}$ the hidden states and cell states of a left and right child. $\mathbf {g} \in \mathbb {R}^{d}$ is the newly composed input for the cell and $\mathbf {i}, \mathbf {f}_{l}, \mathbf {f}_{r}, \mathbf {o} \in \mathbb {R}^{d}$ represent an input gate, two forget gates (left, right), and an output gate respectively. $\mathbf {W} \in \mathbb {R}^{5d\times 2d}$ and $\mathbf {b} \in \mathbb {R}^{5d}$ are trainable parameters. $\sigma $ corresponds to the sigmoid function, $\tanh $ to the hyperbolic tangent, and $\odot $ to element-wise multiplication. Note the equations assume that there are only two children for each node, i.e. binary or binarized trees, following the standard in the literature. While RvNN models can be constructed on any tree structure, in this work we only consider constituency trees as inputs. In spite of the obvious upside that recursive models have in being so flexible, they are known for being difficult to fully utilize with batch computations as compared to other neural architectures because of the diversity of structure found across sentences. To alleviate this problem, BIBREF8 ( BIBREF8 ) proposed the SPINN model, which brings a shift-reduce algorithm to the tree-LSTM. As SPINN simplifies the process of constructing a tree into only two operations, i.e. shift and reduce, it can support more effective parallel computations while enjoying the advantages of tree structures. For efficiency, our model also starts from our own SPINN re-implementation, whose function is exactly the same as that of the tree-LSTM. Structure-aware Tag Representation In most previous works using linguistic tag information BIBREF16 , BIBREF9 , BIBREF18 , tags are usually represented as simple low-dimensional dense vectors, similar to word embeddings. This approach seems reasonable in the case of POS tags that are attached to the corresponding words, but phrase-level constituent tags (e.g. NP, VP, ADJP) vary greatly in size and shape, making them less amenable to uniform treatment. For instance, even the same phrase tags within different syntactic contexts can vary greatly in size and internal structure, as the case of NP tags in Figure 1 shows. Here, the NP consisting of DT[the]-NN[stories] has a different internal structure than the NP consisting of NP[the film 's]-NNS[shortcomings]. One way of deriving structure-aware tag representations from the original tag embeddings is to introduce a separate tag-level tree-LSTM which accepts the typical tag embeddings at each node of a tree and outputs the computed structure-aware tag representations for the nodes. Note that the module concentrates on extracting useful syntactic features by considering only the tags and structures of the trees, excluding word information. Formally, we denote a tag embedding for the tag attached to each node in a tree as $\textbf {e} \in \mathbb {R}^{d_\text{T}}$ . Then, the function of each cell in the tag tree-LSTM is defined in the following way. Leaf nodes are defined by the following: $$  \begin{bmatrix} \hat{\mathbf {c}} \\ \hat{\mathbf {h}} \\ \end{bmatrix} = \tanh {\left(\mathbf {U}_\text{T} \mathbf {e} + \mathbf {a}_\text{T}\right)}$$   (Eq. 13) while non-leaf nodes are defined by the following: $$  \begin{bmatrix} \hat{\mathbf {i}} \\ \hat{\mathbf {f}}_l \\ \hat{\mathbf {f}}_r \\ \hat{\mathbf {o}} \\ \hat{\mathbf {g}} \end{bmatrix} = \begin{bmatrix} \sigma \\ \sigma \\ \sigma \\ \sigma \\ \tanh \end{bmatrix} \Bigg ( \mathbf {W_\text{T}} \begin{bmatrix} \hat{\mathbf {h}}_l \\ \hat{\mathbf {h}}_r \\ \mathbf {e} \\ \end{bmatrix} + \mathbf {b}_\text{T} \Bigg )$$   (Eq. 14) $$  \hat{\mathbf {c}} = \hat{\mathbf {f}}_l \odot \hat{\mathbf {c}}_l + \hat{\mathbf {f}}_r \odot \hat{\mathbf {c}}_r + \hat{\mathbf {i}} \odot \hat{\mathbf {g}}\\$$   (Eq. 15) where $\hat{\mathbf {h}}, \hat{\mathbf {c}} \in \mathbb {R}^{d_\text{T}}$ represent the hidden state and cell state of each node in the tag tree-LSTM. We regard the hidden state ( $\hat{\mathbf {h}}$ ) as a structure-aware tag representation for the node. $ \mathbf {U}_\text{T} \in \mathbb {R}^{2d_\text{T} \times d_\text{T}}, \textbf {a}_\text{T} \in \mathbb {R}^{2d_\text{T}}, \mathbf {W}_\text{T} \in \mathbb {R}^{5d_\text{T} \times 3d_\text{T}}$ , and $\mathbf {b}_\text{T} \in \mathbb {R}^{5d_\text{T}}$ are trainable parameters. The rest of the notation follows equations 8 , 9 , and 10 . In case of leaf nodes, the states are computed by a simple non-linear transformation. Meanwhile, the composition function in a non-leaf node absorbs the tag embedding ( $\mathbf {e}$ ) as an additional input as well as the hidden states of the two children nodes. The benefit of revising tag representations according to the internal structure is that the derived embedding is a function of the corresponding makeup of the node, rather than a monolithic, categorical tag. With regard to the tags themselves, we conjecture that the taxonomy of the tags currently in use in many NLP systems is too complex to be utilized effectively in deep neural models, considering the specificity of many tag sets and the limited amount of data with which to train. Thus, we cluster POS (word-level) tags into 12 groups following the universal POS tagset BIBREF20 and phrase-level tags into 11 groups according to criteria analogous to the case of words, resulting in 23 tag categories in total. In this work, we use the revised coarse-grained tags instead of the original ones. For more details, we refer readers to the supplemental materials. Leaf-LSTM An inherent shortcoming of RvNNs relative to sequential models is that each intermediate representation in a tree is unaware of its external context until all the information is gathered together at the root node. In other words, each composition process is prone to be locally optimized rather than globally optimized. To mitigate this problem, we propose using a leaf-LSTM following the convention of some previous works BIBREF21 , BIBREF7 , BIBREF11 , which is a typical LSTM that accepts a sequence of words in order. Instead of leveraging word embeddings directly, we can use each hidden state and cell state of the leaf-LSTM as input tokens for leaf nodes in a tree-LSTM, anticipating the proper contextualization of the input sequence. Formally, we denote a sequence of words in an input sentence as $w_{1:n}$ ( $n$ : the length of the sentence), and the corresponding word embeddings as $\mathbf {x}_{1:n}$ . Then, the operation of the leaf-LSTM at time $t$ can be formulated as, $$  \begin{bmatrix} \tilde{\mathbf {i}} \\ \tilde{\mathbf {f}} \\ \tilde{\mathbf {o}} \\ \tilde{\mathbf {g}} \end{bmatrix} = \begin{bmatrix} \sigma \\ \sigma \\ \sigma \\ \tanh \end{bmatrix} \Bigg ( \mathbf {W}_\text{L} \begin{bmatrix} \tilde{\mathbf {h}}_{t-1} \\ \mathbf {x}_t \\ \end{bmatrix} + \mathbf {b}_\text{L} \Bigg )$$   (Eq. 18) $$  \tilde{\mathbf {c}}_t = \tilde{\mathbf {f}} \odot \tilde{\mathbf {c}}_{t-1} + \tilde{\mathbf {i}} \odot \tilde{\mathbf {g}}\\$$   (Eq. 19) where $\mathbf {x}_t \in \mathbb {R}^{d_w}$ indicates an input word vector and $\tilde{\mathbf {h}}_t$ , $\tilde{\mathbf {c}}_t \in \mathbb {R}^{d_h}$ represent the hidden and cell state of the LSTM at time $t$ ( $\tilde{\mathbf {h}}_{t-1}$ corresponds to the hidden state at time $t$ -1). $\mathbf {W}_\text{L}$ and $\mathbf {b}_\text{L} $ are learnable parameters. The remaining notation follows that of the tree-LSTM above. In experiments, we demonstrate that introducing a leaf-LSTM fares better at processing the input words of a tree-LSTM compared to using a feed-forward neural network. We also explore the possibility of its bidirectional setting in ablation study. SATA Tree-LSTM In this section, we define SATA Tree-LSTM (Structure-Aware Tag Augmented Tree-LSTM, see Figure 2 ) which joins a tag-level tree-LSTM (section 3.2), a leaf-LSTM (section 3.3), and the original word tree-LSTM together. As above we denote a sequence of words in an input sentence as $w_{1:n}$ and the corresponding word embeddings as $\mathbf {x}_{1:n}$ . In addition, a tag embedding for the tag attached to each node in a tree is denoted by $\textbf {e} \in \mathbb {R}^{d_\text{T}}$ . Then, we derive the final sentence representation for the input sentence with our model in two steps. First, we compute structure-aware tag representations ( $\hat{\mathbf {h}}$ ) for each node of a tree using the tag tree-LSTM (the right side of Figure 2 ) as follows: $$  \begin{bmatrix} \hat{\mathbf {c}} \\ \hat{\mathbf {h}} \\ \end{bmatrix} = {\left\lbrace \begin{array}{ll} \text{Tag-Tree-LSTM}(\mathbf {e}) & \text{if a leaf node} \\ \text{Tag-Tree-LSTM}(\hat{\mathbf {h}}_l, \hat{\mathbf {h}}_r, \mathbf {e}) & \text{otherwise} \end{array}\right.}$$   (Eq. 23) where Tag-Tree-LSTM indicates the module we described in section 3.2. Second, we combine semantic units recursively on the word tree-LSTM in a bottom-up fashion. For leaf nodes, we leverage the Leaf-LSTM (the bottom-left of Figure 2 , explained in section 3.3) to compute $\tilde{\mathbf {c}}_{t}$ and $\tilde{\mathbf {h}}_{t}$ in sequential order, with the corresponding input $\mathbf {x}_t$ . $$  \begin{bmatrix} \tilde{\mathbf {c}}_{t} \\ \tilde{\mathbf {h}}_{t} \\ \end{bmatrix} = \text{Leaf-LSTM}(\tilde{\textbf {h}}_{t-1}, \textbf {x}_t)$$   (Eq. 24) Then, the $\tilde{\mathbf {c}}_{t}$ and $\tilde{\mathbf {h}}_{t}$ can be utilized as input tokens to the word tree-LSTM, with the left (right) child of the target node corresponding to the $t$ th word in the input sentence. $$  \begin{bmatrix} \check{\textbf {c}}_{\lbrace l, r\rbrace } \\ \check{\textbf {h}}_{\lbrace l, r\rbrace } \end{bmatrix} = \begin{bmatrix} \tilde{\textbf {c}}_{t} \\ \tilde{\textbf {h}}_{t} \end{bmatrix}$$   (Eq. 25) In the non-leaf node case, we calculate phrase representations for each node in the word tree-LSTM (the upper-left of Figure 2 ) recursively as follows: $$  \check{\mathbf {g}} = \tanh {\left( \mathbf {U}_\text{w} \begin{bmatrix} \check{\mathbf {h}}_l \\ \check{\mathbf {h}}_r \\ \end{bmatrix} + \mathbf {a}_\text{w} \right)}$$   (Eq. 26) $$  \begin{bmatrix} \check{\mathbf {i}} \\ \check{\mathbf {f}}_l \\ \check{\mathbf {f}}_r \\ \check{\mathbf {o}} \end{bmatrix} = \begin{bmatrix} \sigma \\ \sigma \\ \sigma \\ \sigma \end{bmatrix} \Bigg ( \mathbf {W_\text{w}} \begin{bmatrix} \check{\mathbf {h}}_l \\ \check{\mathbf {h}}_r \\ \hat{\mathbf {h}} \\ \end{bmatrix} + \mathbf {b}_\text{w} \Bigg )$$   (Eq. 27) where $\check{\mathbf {h}}$ , $\check{\mathbf {c}} \in \mathbb {R}^{d_h}$ represent the hidden and cell state of each node in the word tree-LSTM. $\mathbf {U}_\text{w} \in \mathbb {R}^{d_h \times 2d_h}$ , $\mathbf {W}_\text{w} \in \mathbb {R}^{4d_h \times \left(2d_h+d_\text{T}\right)}$ , $\mathbf {a}_\text{w} \in \mathbb {R}^{d_h}$ , $\mathbf {b}_\text{w} \in \mathbb {R}^{4d_h}$ are learned parameters. The remaining notation follows those of the previous sections. Note that the structure-aware tag representations ( $\hat{\mathbf {h}}$ ) are only utilized to control the gate functions of the word tree-LSTM in the form of additional inputs, and are not involved in the semantic composition ( $\check{\mathbf {g}}$ ) directly. Finally, the hidden state of the root node ( $\check{\mathbf {h}}_\text{root}$ ) in the word-level tree-LSTM becomes the final sentence representation of the input sentence. Quantitative Analysis One of the most basic approaches to evaluate a sentence encoder is to measure the classification performance with the sentence representations made by the encoder. Thus, we conduct experiments on the following five datasets. (Summary statistics for the datasets are reported in the supplemental materials.) MR: A group of movie reviews with binary (positive / negative) classes. BIBREF22 SST-2: Stanford Sentiment Treebank BIBREF6 . Similar to MR, but each review is provided in the form of a binary parse tree whose nodes are annotated with numeric sentiment values. For SST-2, we only consider binary (positive / negative) classes. SST-5: Identical to SST-2, but the reviews are grouped into fine-grained (very negative, negative, neutral, positive, very positive) classes. SUBJ: Sentences grouped as being either subjective or objective (binary classes). BIBREF23 TREC: A dataset which groups questions into six different question types (classes). BIBREF24 As a preprocessing step, we construct parse trees for the sentences in the datasets using the Stanford PCFG parser BIBREF25 . Because syntactic tags are by-products of constituency parsing, we do not need further preprocessing. To classify the sentence given our sentence representation ( $\check{\mathbf {h}}_\text{root}$ ), we use one fully-connected layer with a ReLU activation, followed by a softmax classifier. The final predicted probability distribution of the class $y$ given the sentence $w_{1:n}$ is defined as follows, $$\mathbf {s} = \text{ReLU}(\mathbf {W}_\text{s} \check{\mathbf {h}}_\text{root}+ \mathbf {b}_\text{s})$$   (Eq. 37) $$p(y|w_{1:n}) = \text{softmax}(\mathbf {W}_\text{c}\mathbf {s} + \mathbf {b}_\text{c})$$   (Eq. 38) where $\textbf {s} \in \mathbb {R}^{d_\text{s}}$ is the computed task-specific sentence representation for the classifier, and $\textbf {W}_\text{s} \in \mathbb {R}^{d_\text{s} \times d_h}$ , $\textbf {W}_\text{c} \in \mathbb {R}^{d_\text{c} \times d_s}$ , $\textbf {b}_\text{s} \in \mathbb {R}^{d_s}$ , $\textbf {b}_\text{c} \in \mathbb {R}^{d_c}$ are trainable parameters. As an objective function, we use the cross entropy of the predicted and true class distributions. The results of the experiments on the five datasets are shown in table 1 . In this table, we report the test accuracy of our model and various other models on each dataset in terms of percentage. To consider the effects of random initialization, we report the best numbers obtained from each several runs with hyper-parameters fixed. Compared with the previous syntactic tree-based models as well as other neural models, our SATA Tree-LSTM shows superior or competitive performance on all tasks. Specifically, our model achieves new state-of-the-art results within the tree-structured model class on 4 out of 5 sentence classification tasks—SST-2, SST-5, MR, and TREC. The model shows its strength, in particular, when the datasets provide phrase-level supervision to facilitate tree structure learning (i.e. SST-2, SST-5). Moreover, the numbers we report for SST-5 and TREC are competitive to the existing state-of-the-art results including ones from structurally pre-trained models such as ELMo BIBREF26 , proving our model's superiority. Note that the SATA Tree-LSTM also outperforms the recent latent tree-based model, indicating that modeling a neural model with explicit linguistic knowledge can be an attractive option. On the other hand, a remaining concern is that our SATA Tree-LSTM is not robust to random seeds when the size of a dataset is relatively small, as tag embeddings are randomly initialized rather than relying on pre-trained ones in contrast with the case of words. From this observation, we could find out there needs a direction of research towards pre-trained tag embeddings. To estimate the performance of our model beyond the tasks requiring only one sentence at a time, we conduct an experiment on the Stanford Natural Language Inference BIBREF34 dataset, each example of which consists of two sentences, the premise and the hypothesis. Our objective given the data is to predict the correct relationship between the two sentences among three options— contradiction, neutral, or entailment. We use the siamese architecture to encode both the premise ( $p_{1:m}$ ) and hypothesis ( $h_{1:n}$ ) following the standard of sentence-encoding models in the literature. (Specifically, $p_{1:m}$ is encoded as $\check{\mathbf {h}}_\text{root}^p \in \mathbb {R}^{d_h}$ and $h_{1:n}$ is encoded as $\check{\mathbf {h}}_\text{root}^h \in \mathbb {R}^{d_h}$ with the same encoder.) Then, we leverage some heuristics BIBREF35 , followed by one fully-connected layer with a ReLU activation and a softmax classifier. Specifically, $$\mathbf {z} = \left[ \check{\mathbf {h}}_\text{root}^p; \check{\mathbf {h}}_\text{root}^h; | \check{\mathbf {h}}_\text{root}^p - \check{\mathbf {h}}_\text{root}^h |; \check{\mathbf {h}}_\text{root}^p \odot \check{\mathbf {h}}_\text{root}^h \right]$$   (Eq. 41) $$\mathbf {s} = \text{ReLU}(\mathbf {W}_\text{s} \mathbf {z} + \mathbf {b}_\text{s})$$   (Eq. 42) where $\textbf {z} \in \mathbb {R}^{4d_h}$ , $\textbf {s} \in \mathbb {R}^{d_s}$ are intermediate features for the classifier and $\textbf {W}_\text{s} \in \mathbb {R}^{d_\text{s} \times 4d_h}$ , $\textbf {W}_\text{c} \in \mathbb {R}^{d_\text{c} \times d_s}$ , $\textbf {b}_\text{s} \in \mathbb {R}^{d_s}$ , $\textbf {b}_\text{c} \in \mathbb {R}^{d_c}$ are again trainable parameters. Our experimental results on the SNLI dataset are shown in table 2 . In this table, we report the test accuracy and number of trainable parameters for each model. Our SATA-LSTM again demonstrates its decent performance compared against the neural models built on both syntactic trees and latent trees, as well as the non-tree models. (Latent Syntax Tree-LSTM: BIBREF10 ( BIBREF10 ), Tree-based CNN: BIBREF35 ( BIBREF35 ), Gumbel Tree-LSTM: BIBREF11 ( BIBREF11 ), NSE: BIBREF36 ( BIBREF36 ), Reinforced Self-Attention Network: BIBREF4 ( BIBREF4 ), Residual stacked encoders: BIBREF37 ( BIBREF37 ), BiLSTM with generalized pooling: BIBREF38 ( BIBREF38 ).) Note that the number of learned parameters in our model is also comparable to other sophisticated models, showing the efficiency of our model. Even though our model has proven its mettle, the effect of tag information seems relatively weak in the case of SNLI, which contains a large amount of data compared to the others. One possible explanation is that neural models may learn some syntactic rules from large amounts of text when the text size is large enough, reducing the necessity of external linguistic knowledge. We leave the exploration of the effectiveness of tags relative to data size for future work. Here we go over the settings common across our models during experimentation. For more task-specific details, refer to the supplemental materials. For our input embeddings, we used 300 dimensional 840B GloVe BIBREF39 as pre-trained word embeddings, and tag representations were randomly sampled from the uniform distribution [-0.005, 0.005]. Tag vectors are revised during training while the fine-tuning of the word embedding depends on the task. Our models were trained using the Adam BIBREF40 or Adadelta BIBREF41 optimizer, depending on task. For regularization, weight decay is added to the loss function except for SNLI following BIBREF42 ( BIBREF42 ) and Dropout BIBREF43 is also applied for the word embeddings and task-specific classifiers. Moreover, batch normalization BIBREF44 is adopted for the classifiers. As a default, all the weights in the model are initialized following BIBREF45 ( BIBREF45 ) and the biases are set to 0. The total norm of the gradients of the parameters is clipped not to be over 5 during training. Our best models for each dataset were chosen by validation accuracy in cases where a validation set was provided as a part of the dataset. Otherwise, we perform a grid search on probable hyper-parameter settings, or run 10-fold cross-validation in cases where even a test set does not exist. Ablation Study In this section, we design an ablation study on the core modules of our model to explore their effectiveness. The dataset used in this experiment is SST-2. To conduct the experiment, we only replace the target module with other candidates while maintaining the other settings. To be specific, we focus on two modules, the leaf-LSTM and structure-aware tag embeddings (tag-level tree-LSTM). In the first case, the leaf-LSTM is replaced with a fully-connected layer with a $\tanh $ activation or Bi-LSTM. In the second case, we replace the structure-aware tag embeddings with naive tag embeddings or do not employ them at all. The experimental results are depicted in Figure 3 . As the chart shows, our model outperforms all the other options we have considered. In detail, the left part of the chart shows that the leaf-LSTM is the most effective option compared to its competitors. Note that the sequential leaf-LSTM is somewhat superior or competitive than the bidirectional leaf-LSTM when both have a comparable number of parameters. We conjecture this may because a backward LSTM does not add additional useful knowledge when the structure of a sentence is already known. In conclusion, we use the uni-directional LSTM as a leaf module because of its simplicity and remarkable performance. Meanwhile, the right part of the figure demonstrates that our newly introduced structure-aware embeddings have a real impact on improving the model performance. Interestingly, employing the naive tag embeddings made no difference in terms of the test accuracy, even though the absolute validation accuracy increased (not reported in the figure). This result supports our assumption that tag information should be considered in the structure. Qualitative Analysis In previous sections, we have numerically demonstrated that our model is effective in encouraging useful composition of semantic units. Here, we directly investigate the computed representations for each node of a tree, showing that the remarkable performance of our model is mainly due to the gradual and recursive composition of the intermediate representations on the syntactic structure. To observe the phrase-level embeddings at a glance, we draw a scatter plot in which a point represents the corresponding intermediate representation. We utilize PCA (Principal Component Analysis) to project the representations into a two-dimensional vector space. As a target parse tree, we reuse the one seen in Figure 1 . The result is shown in Figure 4 . From this figure, we confirm that the intermediate representations have a hierarchy in the semantic space, which is very similar to that of the parse tree. In other words, as many tree-structured models pursue, we can see the tendency of constructing the representations from the low-level (the bottom of the figure) to the high-level (the top-left and top-right of the figure), integrating the meaning of the constituents recursively. An interesting thing to note is that the final sentence representation is near that of the phrase `, the stories are quietly moving.' rather than that of `Despite the film's shortcomings', catching the main meaning of the sentence. Conclusion We have proposed a novel RvNN architecture to fully utilize linguistic priors. A newly introduced tag-level tree-LSTM demonstrates that it can effectively control the composition function of the corresponding word-level tree-LSTM. In addition, the proper contextualization of the input word vectors results in significant performance improvements on several sentence-level tasks. For future work, we plan to explore a new way of exploiting dependency trees effectively, similar to the case of constituency trees. Acknowledgments We thank anonymous reviewers for their constructive and fruitful comments. This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (NRF2016M3C4A7952587).
Which baselines did they compare against?
0ad4359e3e7e5e5f261c2668fe84c12bc762b3b8
Which Ġbas elines Ġdid Ġthey Ġcompare Ġagainst ?
Various Ġtree Ġstructured Ġneural Ġnetworks Ġincluding Ġvariants Ġof ĠTree - L ST M , ĠTree - based ĠCNN , ĠR NT N , Ġand Ġnon - tree Ġmodels Ġincluding Ġvariants Ġof ĠL ST Ms , ĠCNN s , Ġresidual , Ġand Ġself - att ention Ġbased Ġnetworks
<s> Which Ġbas elines Ġdid Ġthey Ġcompare Ġagainst ? </s> Introduction </s> One Ġof Ġthe Ġmost Ġfundamental Ġtopics Ġin Ġnatural Ġlanguage Ġprocessing Ġis Ġhow Ġbest Ġto Ġderive Ġhigh - level Ġrepresentations Ġfrom Ġconstituent Ġparts , Ġas Ġnatural Ġlanguage Ġmeanings Ġare Ġa Ġfunction Ġof Ġtheir Ġconstituent Ġparts . ĠHow Ġbest Ġto Ġconstruct Ġa Ġsentence Ġrepresentation Ġfrom Ġdistributed Ġword Ġembed d ings Ġis Ġan Ġexample Ġdomain Ġof Ġthis Ġlarger Ġissue . ĠEven Ġthough Ġsequential Ġneural Ġmodels Ġsuch Ġas Ġrecurrent Ġneural Ġnetworks Ġ( R NN ) ĠB IB REF 0 Ġand Ġtheir Ġvariants Ġincluding ĠLong ĠShort - Term ĠMemory Ġ( L ST M ) ĠB IB REF 1 Ġand ĠG ated ĠRec urrent ĠUnit Ġ( GR U ) ĠB IB REF 2 Ġhave Ġbecome Ġthe Ġde - fact o Ġstandard Ġfor Ġcond ensing Ġsentence - level Ġinformation Ġfrom Ġa Ġsequence Ġof Ġwords Ġinto Ġa Ġfixed Ġvector , Ġthere Ġhave Ġbeen Ġmany Ġlines Ġof Ġresearch Ġtowards Ġbetter Ġsentence Ġrepresentation Ġusing Ġother Ġneural Ġarchitectures , Ġe . g . Ġconv olution al Ġneural Ġnetworks Ġ( CNN ) ĠB IB REF 3 Ġor Ġself - att ention Ġbased Ġmodels ĠB IB REF 4 Ġ. </s> From Ġa Ġlinguistic Ġpoint Ġof Ġview , Ġthe Ġunderlying Ġtree Ġstructure âĢĶ as Ġexpressed Ġby Ġits Ġconstituency Ġand Ġdependency Ġtrees âĢĶ of Ġa Ġsentence Ġis Ġan Ġintegral Ġpart Ġof Ġits Ġmeaning . ĠInspired Ġby Ġthis Ġfact , Ġsome Ġrecursive Ġneural Ġnetwork Ġ( R v NN ) Ġmodels Ġare Ġdesigned Ġto Ġreflect Ġthe Ġsynt actic Ġtree Ġstructure , Ġachieving Ġimpressive Ġresults Ġon Ġseveral Ġsentence - level Ġtasks Ġsuch Ġas Ġsentiment Ġanalysis ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, Ġmachine Ġtranslation ĠB IB REF 7 Ġ, Ġnatural Ġlanguage Ġinference ĠB IB REF 8 Ġ, Ġand Ġdiscourse Ġrelation Ġclassification ĠB IB REF 9 Ġ. </s> However , Ġsome Ġrecent Ġworks Ġhave ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġproposed Ġlatent Ġtree Ġmodels , Ġwhich Ġlearn Ġto Ġconstruct Ġtask - specific Ġtree Ġstructures Ġwithout Ġexplicit Ġsupervision , Ġbringing Ġinto Ġquestion Ġthe Ġvalue Ġof Ġlingu istically - mot ivated Ġrecursive Ġneural Ġmodels . ĠWitness ing Ġthe Ġsurprising Ġperformance Ġof Ġthe Ġlatent Ġtree Ġmodels Ġon Ġsome Ġsentence - level Ġtasks , Ġthere Ġarises Ġa Ġnatural Ġquestion : ĠAre Ġlinguistic Ġtree Ġstructures Ġthe Ġoptimal Ġway Ġof Ġcomposing Ġsentence Ġrepresentations Ġfor ĠN LP Ġtasks ? </s> In Ġthis Ġpaper , Ġwe Ġdemonstrate Ġthat Ġlinguistic Ġpri ors Ġare Ġin Ġfact Ġuseful Ġfor Ġdev ising Ġeffective Ġneural Ġmodels Ġfor Ġsentence Ġrepresentations , Ġshowing Ġthat Ġour Ġnovel Ġarchitecture Ġbased Ġon Ġconstituency Ġtrees Ġand Ġtheir Ġtag Ġinformation Ġob tains Ġsuperior Ġperformance Ġon Ġseveral Ġsentence - level Ġtasks , Ġincluding Ġsentiment Ġanalysis Ġand Ġnatural Ġlanguage Ġinference . </s> A Ġchief Ġnovelty Ġof Ġour Ġapproach Ġis Ġthat Ġwe Ġintroduce Ġa Ġsmall Ġseparate Ġtag - level Ġtree - L ST M Ġto Ġcontrol Ġthe Ġcomposition Ġfunction Ġof Ġthe Ġexisting Ġword - level Ġtree - L ST M , Ġwhich Ġis Ġin Ġcharge Ġof Ġextracting Ġhelpful Ġsynt actic Ġsignals Ġfor Ġmeaningful Ġsemantic Ġcomposition Ġof Ġconstituents Ġby Ġconsidering Ġboth Ġthe Ġstructures Ġand Ġlinguistic Ġtags Ġof Ġconstituency Ġtrees Ġsimultaneously . ĠIn Ġaddition , Ġwe Ġdemonstrate Ġthat Ġapplying Ġa Ġtypical ĠL ST M Ġto Ġpre process Ġthe Ġleaf Ġnodes Ġof Ġa Ġtree - L ST M Ġgreatly Ġimproves Ġthe Ġperformance Ġof Ġthe Ġtree Ġmodels . ĠMoreover , Ġwe Ġpropose Ġa Ġclustered Ġtag Ġset Ġto Ġreplace Ġthe Ġexisting Ġtags Ġon Ġthe Ġassumption Ġthat Ġthe Ġoriginal Ġsynt actic Ġtags Ġare Ġtoo Ġfined - gr ained Ġto Ġbe Ġuseful Ġin Ġneural Ġmodels . </s> In Ġshort , Ġour Ġcontributions Ġin Ġthis Ġwork Ġare Ġas Ġfollows : </s> Related ĠWork </s> Rec ursive Ġneural Ġnetworks Ġ( R v NN ) Ġare Ġa Ġkind Ġof Ġneural Ġarchitecture Ġwhich Ġmodel Ġsentences Ġby Ġexploiting Ġsynt actic Ġstructure . ĠWhile Ġearlier ĠR v NN Ġmodels Ġproposed Ġutilizing Ġdiverse Ġcomposition Ġfunctions , Ġincluding Ġfeed - forward Ġneural Ġnetworks ĠB IB REF 12 Ġ, Ġmatrix - vector Ġmultiplication ĠB IB REF 5 Ġ, Ġand Ġtens or Ġcomputation ĠB IB REF 6 Ġ, Ġtree - L ST Ms ĠB IB REF 13 Ġremain Ġthe Ġstandard Ġfor Ġseveral Ġsentence - level Ġtasks . </s> Even Ġthough Ġclassic ĠR v NN s Ġhave Ġdemonstrated Ġsuperior Ġperformance Ġon Ġa Ġvariety Ġof Ġtasks , Ġtheir Ġinf lex ibility , Ġi . e . Ġtheir Ġinability Ġto Ġhandle Ġdynamic Ġcomposition ality Ġfor Ġdifferent Ġsynt actic Ġconfigurations , Ġis Ġa Ġconsiderable Ġweakness . ĠFor Ġinstance , Ġit Ġwould Ġbe Ġdesirable Ġif Ġour Ġmodel Ġcould Ġdistinguish Ġe . g . Ġadjective - n oun Ġcomposition Ġfrom Ġthat Ġof Ġverb - n oun Ġor Ġpre position - n oun Ġcomposition , Ġas Ġmodels Ġfailing Ġto Ġmake Ġsuch Ġa Ġdistinction Ġignore Ġreal - world Ġsynt actic Ġconsiderations Ġsuch Ġas Ġ` - arity ' Ġof Ġfunction Ġwords Ġ( i . e . Ġtypes ), Ġand Ġthe Ġadjunct / argument Ġdistinction . </s> To Ġenable Ġdynamic Ġcomposition ality Ġin Ġrecursive Ġneural Ġnetworks , Ġmany Ġprevious Ġworks ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġ, ĠB IB REF 9 Ġ, ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ, ĠB IB REF 19 Ġhave Ġproposed Ġvarious Ġmethods . </s> One Ġmain Ġdirection Ġof Ġresearch Ġlever ages Ġtag Ġinformation , Ġwhich Ġis Ġproduced Ġas Ġa Ġby - product Ġof Ġparsing . ĠIn Ġdetail , ĠB IB REF 16 Ġ( ĠB IB REF 16 Ġ) Ġsuggested ĠTG - R NN , Ġa Ġmodel Ġemploying Ġdifferent Ġcomposition Ġfunctions Ġaccording Ġto ĠPOS Ġtags , Ġand ĠTE - R NN / TE - R NT N , Ġmodels Ġwhich Ġleverage Ġtag Ġembed d ings Ġas Ġadditional Ġinputs Ġfor Ġthe Ġexisting Ġtree - struct ured Ġmodels . ĠDespite Ġthe Ġnovelty Ġof Ġutilizing Ġtag Ġinformation , Ġthe Ġexplosion Ġof Ġthe Ġnumber Ġof Ġparameters Ġ( in Ġcase Ġof Ġthe ĠTG - R NN ) Ġand Ġthe Ġlimited Ġperformance Ġof Ġthe Ġoriginal Ġmodels Ġ( in Ġcase Ġof Ġthe ĠTE - R NN / TE - R NT N ) Ġhave Ġprevented Ġthese Ġmodels Ġfrom Ġbeing Ġwidely Ġadopted . ĠMeanwhile , ĠB IB REF 9 Ġ( ĠB IB REF 9 Ġ) Ġand ĠB IB REF 18 Ġ( ĠB IB REF 18 Ġ) Ġproposed Ġmodels Ġbased Ġon Ġa Ġtree - L ST M Ġwhich Ġalso Ġuses Ġthe Ġtag Ġvectors Ġto Ġcontrol Ġthe Ġgate Ġfunctions Ġof Ġthe Ġtree - L ST M . ĠIn Ġspite Ġof Ġtheir Ġimpressive Ġresults , Ġthere Ġis Ġa Ġlimitation Ġthat Ġthe Ġtrained Ġtag Ġembed d ings Ġare Ġtoo Ġsimple Ġto Ġreflect Ġthe Ġrich Ġinformation Ġwhich Ġtags Ġprovide Ġin Ġdifferent Ġsynt actic Ġstructures . ĠTo Ġalleviate Ġthis Ġproblem , Ġwe Ġintroduce Ġstructure - aware Ġtag Ġrepresentations Ġin Ġthe Ġnext Ġsection . </s> Another Ġway Ġof Ġbuilding Ġdynamic Ġcomposition ality Ġinto ĠR v NN s Ġis Ġto Ġtake Ġadvantage Ġof Ġa Ġmeta - network Ġ( or Ġhyper - network ). ĠInspired Ġby Ġrecent Ġworks Ġon Ġdynamic Ġparameter Ġprediction , ĠDC - Tree L ST Ms ĠB IB REF 17 Ġdynamically Ġcreate Ġthe Ġparameters Ġfor Ġcompos itional Ġfunctions Ġin Ġa Ġtree - L ST M . ĠSpecifically , Ġthe Ġmodel Ġhas Ġtwo Ġseparate Ġtree - L ST M Ġnetworks Ġwhose Ġarchitectures Ġare Ġsimilar , Ġbut Ġthe Ġsmaller Ġof Ġthe Ġtwo Ġis Ġutilized Ġto Ġcalculate Ġthe Ġweights Ġof Ġthe Ġbigger Ġone . ĠA Ġpossible Ġproblem Ġfor Ġthis Ġmodel Ġis Ġthat Ġit Ġmay Ġbe Ġeasy Ġto Ġbe Ġtrained Ġsuch Ġthat Ġthe Ġrole Ġof Ġeach Ġtree - L ST M Ġis Ġambiguous , Ġas Ġthey Ġshare Ġthe Ġsame Ġinput , Ġi . e . Ġword Ġinformation . ĠTherefore , Ġwe Ġdesign Ġtwo Ġdis ent angled Ġtree - L ST Ms Ġin Ġour Ġmodel Ġso Ġthat Ġone Ġfocuses Ġon Ġextracting Ġuseful Ġfeatures Ġfrom Ġonly Ġsynt actic Ġinformation Ġwhile Ġthe Ġother Ġcomp oses Ġsemantic Ġunits Ġwith Ġthe Ġaid Ġof Ġthe Ġfeatures . ĠFurthermore , Ġour Ġmodel Ġreduces Ġthe Ġcomplexity Ġof Ġcomputation Ġby Ġutilizing Ġtypical Ġtree - L ST M Ġframeworks Ġinstead Ġof Ġcomputing Ġthe Ġweights Ġfor Ġeach Ġexample . </s> Finally , Ġsome Ġrecent Ġworks ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġhave Ġproposed Ġlatent Ġtree - struct ured Ġmodels Ġthat Ġlearn Ġhow Ġto Ġformulate Ġtree Ġstructures Ġfrom Ġonly Ġsequences Ġof Ġtokens , Ġwithout Ġthe Ġaid Ġof Ġsynt actic Ġtrees Ġor Ġlinguistic Ġinformation . ĠThe Ġlatent Ġtree Ġmodels Ġhave Ġthe Ġadvantage Ġof Ġbeing Ġable Ġto Ġfind Ġthe Ġoptimized Ġtask - specific Ġorder Ġof Ġcomposition Ġrather Ġthan Ġa Ġsequential Ġor Ġsynt actic Ġone . ĠIn Ġexperiments , Ġwe Ġcompare Ġour Ġmodel Ġwith Ġnot Ġonly Ġsynt actic Ġtree - based Ġmodels Ġbut Ġalso Ġlatent Ġtree Ġmodels , Ġdemonstrating Ġthat Ġmodeling Ġwith Ġexplicit Ġlinguistic Ġknowledge Ġcan Ġbe Ġan Ġattractive Ġoption . </s> Model </s> In Ġthis Ġsection , Ġwe Ġintroduce Ġa Ġnovel ĠR v NN Ġarchitecture , Ġcalled ĠSATA ĠTree - L ST M Ġ( St ructure - A ware ĠTag ĠAug mented ĠTree - L ST M ). ĠThis Ġmodel Ġis Ġsimilar Ġto Ġtypical ĠTree - L ST Ms , Ġbut Ġprovides Ġdynamic Ġcomposition ality Ġby Ġaugment ing Ġa Ġseparate Ġtag - level Ġtree - L ST M Ġwhich Ġproduces Ġstructure - aware Ġtag Ġrepresentations Ġfor Ġeach Ġnode Ġin Ġa Ġtree . ĠIn Ġother Ġwords , Ġour Ġmodel Ġhas Ġtwo Ġindependent Ġtree - struct ured Ġmodules Ġbased Ġon Ġthe Ġsame Ġconstituency Ġtree , Ġone Ġof Ġwhich Ġ( word - level Ġtree - L ST M ) Ġis Ġresponsible Ġfor Ġconstructing Ġsentence Ġrepresentations Ġgiven Ġa Ġsequence Ġof Ġwords Ġas Ġusual , Ġwhile Ġthe Ġother Ġ( tag - level Ġtree - L ST M ) Ġprovides Ġsupplementary Ġsynt actic Ġinformation Ġto Ġthe Ġformer . </s> In Ġsection Ġ3 . 1 , Ġwe Ġfirst Ġreview Ġtree - L ST M Ġarchitectures . ĠThen Ġin Ġsection Ġ3 . 2 , Ġwe Ġintroduce Ġa Ġtag - level Ġtree - L ST M Ġand Ġstructure - aware Ġtag Ġrepresentations . ĠIn Ġsection Ġ3 . 3 , Ġwe Ġdiscuss Ġan Ġadditional Ġtechnique Ġto Ġboost Ġthe Ġperformance Ġof Ġtree - struct ured Ġmodels , Ġand Ġin Ġsection Ġ3 . 4 , Ġwe Ġdescribe Ġthe Ġentire Ġarchitecture Ġof Ġour Ġmodel Ġin Ġdetail . </s> Tree - L ST M </s> The ĠL ST M ĠB IB REF 1 Ġarchitecture Ġwas Ġfirst Ġintroduced Ġas Ġan Ġextension Ġof Ġthe ĠR NN Ġarchitecture Ġto Ġmitigate Ġthe Ġvanishing Ġand Ġexploding Ġgradient Ġproblems . ĠIn Ġaddition , Ġseveral Ġworks Ġhave Ġdiscovered Ġthat Ġapplying Ġthe ĠL ST M Ġcell Ġinto Ġtree Ġstructures Ġcan Ġbe Ġan Ġeffective Ġmeans Ġof Ġmodeling Ġsentence Ġrepresentations . </s> To Ġbe Ġformal , Ġthe Ġcomposition Ġfunction Ġof Ġthe Ġcell Ġin Ġa Ġtree - L ST M Ġcan Ġbe Ġformulated Ġas Ġfollows : </s> $$ Ġ Ġ\ begin { b mat rix } Ġ\ math bf Ġ{ i } Ġ\\ Ġ\ math bf Ġ{ f } _ l Ġ\\ Ġ\ math bf Ġ{ f } _ r Ġ\\ Ġ\ math bf Ġ{ o } Ġ\\ Ġ\ math bf Ġ{ g } Ġ\ end { b mat rix } Ġ= Ġ\ begin { b mat rix } Ġ\ s igma Ġ\\ Ġ\ s igma Ġ\\ Ġ\ s igma Ġ\\ Ġ\ s igma Ġ\\ Ġ\ tan h Ġ\ end { b mat rix } Ġ\ B igg Ġ( Ġ\ math bf Ġ{ W } Ġ\ begin { b mat rix } Ġ\ math bf Ġ{ h } _ l Ġ\\ Ġ\ math bf Ġ{ h } _ r Ġ\\ Ġ\ end { b mat rix } Ġ+ Ġ\ math bf Ġ{ b } Ġ\ B igg Ġ) $$ Ġ Ġ Ġ( E q . Ġ8 ) </s> $$ Ġ Ġ\ math bf Ġ{ c } Ġ= Ġ\ math bf Ġ{ f } _ l Ġ\ od ot Ġ\ math bf Ġ{ c } _ l Ġ+ Ġ\ math bf Ġ{ f } _ r Ġ\ od ot Ġ\ math bf Ġ{ c } _ r Ġ+ Ġ\ math bf Ġ{ i } Ġ\ od ot Ġ\ math bf Ġ{ g } \\ $$ Ġ Ġ Ġ( E q . Ġ9 ) </s> where Ġ$\ math bf Ġ{ h }, Ġ\ math bf Ġ{ c } Ġ\ in Ġ\ math bb Ġ{ R } ^{ d } $ Ġindicate Ġthe Ġhidden Ġstate Ġand Ġcell Ġstate Ġof Ġthe ĠL ST M Ġcell , Ġand Ġ$\ math bf Ġ{ h } _ l , Ġ\ math bf Ġ{ h } _ r , Ġ\ math bf Ġ{ c } _ l , Ġ\ math bf Ġ{ c } _ r Ġ\ in Ġ\ math bb Ġ{ R } ^{ d } $ Ġthe Ġhidden Ġstates Ġand Ġcell Ġstates Ġof Ġa Ġleft Ġand Ġright Ġchild . Ġ$\ math bf Ġ{ g } Ġ\ in Ġ\ math bb Ġ{ R } ^{ d } $ Ġis Ġthe Ġnewly Ġcomposed Ġinput Ġfor Ġthe Ġcell Ġand Ġ$\ math bf Ġ{ i }, Ġ\ math bf Ġ{ f } _{ l }, Ġ\ math bf Ġ{ f } _{ r }, Ġ\ math bf Ġ{ o } Ġ\ in Ġ\ math bb Ġ{ R } ^{ d } $ Ġrepresent Ġan Ġinput Ġgate , Ġtwo Ġforget Ġgates Ġ( left , Ġright ), Ġand Ġan Ġoutput Ġgate Ġrespectively . Ġ$\ math bf Ġ{ W } Ġ\ in Ġ\ math bb Ġ{ R } ^{ 5 d \ times Ġ2 d } $ Ġand Ġ$\ math bf Ġ{ b } Ġ\ in Ġ\ math bb Ġ{ R } ^{ 5 d } $ Ġare Ġtrain able Ġparameters . Ġ$\ s igma Ġ$ Ġcorresponds Ġto Ġthe Ġs igm oid Ġfunction , Ġ$\ tan h Ġ$ Ġto Ġthe Ġhyper b olic Ġtang ent , Ġand Ġ$\ od ot Ġ$ Ġto Ġelement - wise Ġmultiplication . </s> Note Ġthe Ġequations Ġassume Ġthat Ġthere Ġare Ġonly Ġtwo Ġchildren Ġfor Ġeach Ġnode , Ġi . e . Ġbinary Ġor Ġbin ar ized Ġtrees , Ġfollowing Ġthe Ġstandard Ġin Ġthe Ġliterature . ĠWhile ĠR v NN Ġmodels Ġcan Ġbe Ġconstructed Ġon Ġany Ġtree Ġstructure , Ġin Ġthis Ġwork Ġwe Ġonly Ġconsider Ġconstituency Ġtrees Ġas Ġinputs . </s> In Ġspite Ġof Ġthe Ġobvious Ġupside Ġthat Ġrecursive Ġmodels Ġhave Ġin Ġbeing Ġso Ġflexible , Ġthey Ġare Ġknown Ġfor Ġbeing Ġdifficult Ġto Ġfully Ġutilize Ġwith Ġbatch Ġcomput ations Ġas Ġcompared Ġto Ġother Ġneural Ġarchitectures Ġbecause Ġof Ġthe Ġdiversity Ġof Ġstructure Ġfound Ġacross Ġsentences . ĠTo Ġalleviate Ġthis Ġproblem , ĠB IB REF 8 Ġ( ĠB IB REF 8 Ġ) Ġproposed Ġthe ĠSP IN N Ġmodel , Ġwhich Ġbrings Ġa Ġshift - red uce Ġalgorithm Ġto Ġthe Ġtree - L ST M . ĠAs ĠSP IN N Ġsimpl ifies Ġthe Ġprocess Ġof Ġconstructing Ġa Ġtree Ġinto Ġonly Ġtwo Ġoperations , Ġi . e . Ġshift Ġand Ġreduce , Ġit Ġcan Ġsupport Ġmore Ġeffective Ġparallel Ġcomput ations Ġwhile Ġenjoying Ġthe Ġadvantages Ġof Ġtree Ġstructures . ĠFor Ġefficiency , Ġour Ġmodel Ġalso Ġstarts Ġfrom Ġour Ġown ĠSP IN N Ġre - im plementation , Ġwhose Ġfunction Ġis Ġexactly Ġthe Ġsame Ġas Ġthat Ġof Ġthe Ġtree - L ST M . </s> St ructure - aware ĠTag ĠRepresent ation </s> In Ġmost Ġprevious Ġworks Ġusing Ġlinguistic Ġtag Ġinformation ĠB IB REF 16 Ġ, ĠB IB REF 9 Ġ, ĠB IB REF 18 Ġ, Ġtags Ġare Ġusually Ġrepresented Ġas Ġsimple Ġlow - dimensional Ġdense Ġvectors , Ġsimilar Ġto Ġword Ġembed d ings . ĠThis Ġapproach Ġseems Ġreasonable Ġin Ġthe Ġcase Ġof ĠPOS Ġtags Ġthat Ġare Ġattached Ġto Ġthe Ġcorresponding Ġwords , Ġbut Ġphrase - level Ġconstituent Ġtags Ġ( e . g . ĠNP , ĠVP , ĠAD JP ) Ġvary Ġgreatly Ġin Ġsize Ġand Ġshape , Ġmaking Ġthem Ġless Ġam enable Ġto Ġuniform Ġtreatment . ĠFor Ġinstance , Ġeven Ġthe Ġsame Ġphrase Ġtags Ġwithin Ġdifferent Ġsynt actic Ġcontexts Ġcan Ġvary Ġgreatly Ġin Ġsize Ġand Ġinternal Ġstructure , Ġas Ġthe Ġcase Ġof ĠNP Ġtags Ġin ĠFigure Ġ1 Ġshows . ĠHere , Ġthe ĠNP Ġconsisting Ġof ĠDT [ the ]- NN [ stories ] Ġhas Ġa Ġdifferent Ġinternal Ġstructure Ġthan Ġthe ĠNP Ġconsisting Ġof ĠNP [ the Ġfilm Ġ' s ]- NN S [ short comings ]. </s> One Ġway Ġof Ġder iving Ġstructure - aware Ġtag Ġrepresentations Ġfrom Ġthe Ġoriginal Ġtag Ġembed d ings Ġis Ġto Ġintroduce Ġa Ġseparate Ġtag - level Ġtree - L ST M Ġwhich Ġaccepts Ġthe Ġtypical Ġtag Ġembed d ings Ġat Ġeach Ġnode Ġof Ġa Ġtree Ġand Ġoutputs Ġthe Ġcomputed Ġstructure - aware Ġtag Ġrepresentations Ġfor Ġthe Ġnodes . ĠNote Ġthat Ġthe Ġmodule Ġconcent rates Ġon Ġextracting Ġuseful Ġsynt actic Ġfeatures Ġby Ġconsidering Ġonly Ġthe Ġtags Ġand Ġstructures Ġof Ġthe Ġtrees , Ġexcluding Ġword Ġinformation . </s> Form ally , Ġwe Ġdenote Ġa Ġtag Ġembed ding Ġfor Ġthe Ġtag Ġattached Ġto Ġeach Ġnode Ġin Ġa Ġtree Ġas Ġ$\ text bf Ġ{ e } Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ \ text { T }} $ Ġ. ĠThen , Ġthe Ġfunction Ġof Ġeach Ġcell Ġin Ġthe Ġtag Ġtree - L ST M Ġis Ġdefined Ġin Ġthe Ġfollowing Ġway . ĠLeaf Ġnodes Ġare Ġdefined Ġby Ġthe Ġfollowing : </s> $$ Ġ Ġ\ begin { b mat rix } Ġ\ hat {\ math bf Ġ{ c }} Ġ\\ Ġ\ hat {\ math bf Ġ{ h }} Ġ\\ Ġ\ end { b mat rix } Ġ= Ġ\ tan h Ġ{\ left (\ math bf Ġ{ U } _ \ text { T } Ġ\ math bf Ġ{ e } Ġ+ Ġ\ math bf Ġ{ a } _ \ text { T }\ right )} $$ Ġ Ġ Ġ( E q . Ġ13 ) </s> while Ġnon - leaf Ġnodes Ġare Ġdefined Ġby Ġthe Ġfollowing : </s> $$ Ġ Ġ\ begin { b mat rix } Ġ\ hat {\ math bf Ġ{ i }} Ġ\\ Ġ\ hat {\ math bf Ġ{ f }} _ l Ġ\\ Ġ\ hat {\ math bf Ġ{ f }} _ r Ġ\\ Ġ\ hat {\ math bf Ġ{ o }} Ġ\\ Ġ\ hat {\ math bf Ġ{ g }} Ġ\ end { b mat rix } Ġ= Ġ\ begin { b mat rix } Ġ\ s igma Ġ\\ Ġ\ s igma Ġ\\ Ġ\ s igma Ġ\\ Ġ\ s igma Ġ\\ Ġ\ tan h Ġ\ end { b mat rix } Ġ\ B igg Ġ( Ġ\ math bf Ġ{ W _ \ text { T }} Ġ\ begin { b mat rix } Ġ\ hat {\ math bf Ġ{ h }} _ l Ġ\\ Ġ\ hat {\ math bf Ġ{ h }} _ r Ġ\\ Ġ\ math bf Ġ{ e } Ġ\\ Ġ\ end { b mat rix } Ġ+ Ġ\ math bf Ġ{ b } _ \ text { T } Ġ\ B igg Ġ) $$ Ġ Ġ Ġ( E q . Ġ14 ) </s> $$ Ġ Ġ\ hat {\ math bf Ġ{ c }} Ġ= Ġ\ hat {\ math bf Ġ{ f }} _ l Ġ\ od ot Ġ\ hat {\ math bf Ġ{ c }} _ l Ġ+ Ġ\ hat {\ math bf Ġ{ f }} _ r Ġ\ od ot Ġ\ hat {\ math bf Ġ{ c }} _ r Ġ+ Ġ\ hat {\ math bf Ġ{ i }} Ġ\ od ot Ġ\ hat {\ math bf Ġ{ g }} \\ $$ Ġ Ġ Ġ( E q . Ġ15 ) </s> where Ġ$\ hat {\ math bf Ġ{ h } }, Ġ\ hat {\ math bf Ġ{ c }} Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ \ text { T }} $ Ġrepresent Ġthe Ġhidden Ġstate Ġand Ġcell Ġstate Ġof Ġeach Ġnode Ġin Ġthe Ġtag Ġtree - L ST M . ĠWe Ġregard Ġthe Ġhidden Ġstate Ġ( Ġ$\ hat {\ math bf Ġ{ h }} $ Ġ) Ġas Ġa Ġstructure - aware Ġtag Ġrepresentation Ġfor Ġthe Ġnode . Ġ$ Ġ\ math bf Ġ{ U } _ \ text { T } Ġ\ in Ġ\ math bb Ġ{ R } ^{ 2 d _ \ text { T } Ġ\ times Ġd _ \ text { T } }, Ġ\ text bf Ġ{ a } _ \ text { T } Ġ\ in Ġ\ math bb Ġ{ R } ^{ 2 d _ \ text { T } }, Ġ\ math bf Ġ{ W } _ \ text { T } Ġ\ in Ġ\ math bb Ġ{ R } ^{ 5 d _ \ text { T } Ġ\ times Ġ3 d _ \ text { T }} $ Ġ, Ġand Ġ$\ math bf Ġ{ b } _ \ text { T } Ġ\ in Ġ\ math bb Ġ{ R } ^{ 5 d _ \ text { T }} $ Ġare Ġtrain able Ġparameters . ĠThe Ġrest Ġof Ġthe Ġnotation Ġfollows Ġequations Ġ8 Ġ, Ġ9 Ġ, Ġand Ġ10 Ġ. ĠIn Ġcase Ġof Ġleaf Ġnodes , Ġthe Ġstates Ġare Ġcomputed Ġby Ġa Ġsimple Ġnon - linear Ġtransformation . ĠMeanwhile , Ġthe Ġcomposition Ġfunction Ġin Ġa Ġnon - leaf Ġnode Ġabsorbs Ġthe Ġtag Ġembed ding Ġ( Ġ$\ math bf Ġ{ e } $ Ġ) Ġas Ġan Ġadditional Ġinput Ġas Ġwell Ġas Ġthe Ġhidden Ġstates Ġof Ġthe Ġtwo Ġchildren Ġnodes . ĠThe Ġbenefit Ġof Ġrev ising Ġtag Ġrepresentations Ġaccording Ġto Ġthe Ġinternal Ġstructure Ġis Ġthat Ġthe Ġderived Ġembed ding Ġis Ġa Ġfunction Ġof Ġthe Ġcorresponding Ġmakeup Ġof Ġthe Ġnode , Ġrather Ġthan Ġa Ġmon olithic , Ġcateg orical Ġtag . </s> With Ġregard Ġto Ġthe Ġtags Ġthemselves , Ġwe Ġconjecture Ġthat Ġthe Ġtax onomy Ġof Ġthe Ġtags Ġcurrently Ġin Ġuse Ġin Ġmany ĠN LP Ġsystems Ġis Ġtoo Ġcomplex Ġto Ġbe Ġutilized Ġeffectively Ġin Ġdeep Ġneural Ġmodels , Ġconsidering Ġthe Ġspecificity Ġof Ġmany Ġtag Ġsets Ġand Ġthe Ġlimited Ġamount Ġof Ġdata Ġwith Ġwhich Ġto Ġtrain . ĠThus , Ġwe Ġcluster ĠPOS Ġ( word - level ) Ġtags Ġinto Ġ12 Ġgroups Ġfollowing Ġthe Ġuniversal ĠPOS Ġtag set ĠB IB REF 20 Ġand Ġphrase - level Ġtags Ġinto Ġ11 Ġgroups Ġaccording Ġto Ġcriteria Ġanalogous Ġto Ġthe Ġcase Ġof Ġwords , Ġresulting Ġin Ġ23 Ġtag Ġcategories Ġin Ġtotal . ĠIn Ġthis Ġwork , Ġwe Ġuse Ġthe Ġrevised Ġcoarse - gr ained Ġtags Ġinstead Ġof Ġthe Ġoriginal Ġones . ĠFor Ġmore Ġdetails , Ġwe Ġrefer Ġreaders Ġto Ġthe Ġsupplemental Ġmaterials . </s> Le af - L ST M </s> An Ġinherent Ġshort coming Ġof ĠR v NN s Ġrelative Ġto Ġsequential Ġmodels Ġis Ġthat Ġeach Ġintermediate Ġrepresentation Ġin Ġa Ġtree Ġis Ġunaware Ġof Ġits Ġexternal Ġcontext Ġuntil Ġall Ġthe Ġinformation Ġis Ġgathered Ġtogether Ġat Ġthe Ġroot Ġnode . ĠIn Ġother Ġwords , Ġeach Ġcomposition Ġprocess Ġis Ġprone Ġto Ġbe Ġlocally Ġoptimized Ġrather Ġthan Ġglobally Ġoptimized . </s> To Ġmitigate Ġthis Ġproblem , Ġwe Ġpropose Ġusing Ġa Ġleaf - L ST M Ġfollowing Ġthe Ġconvention Ġof Ġsome Ġprevious Ġworks ĠB IB REF 21 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 11 Ġ, Ġwhich Ġis Ġa Ġtypical ĠL ST M Ġthat Ġaccepts Ġa Ġsequence Ġof Ġwords Ġin Ġorder . ĠInstead Ġof Ġleveraging Ġword Ġembed d ings Ġdirectly , Ġwe Ġcan Ġuse Ġeach Ġhidden Ġstate Ġand Ġcell Ġstate Ġof Ġthe Ġleaf - L ST M Ġas Ġinput Ġtokens Ġfor Ġleaf Ġnodes Ġin Ġa Ġtree - L ST M , Ġanticipating Ġthe Ġproper Ġcontextual ization Ġof Ġthe Ġinput Ġsequence . </s> Form ally , Ġwe Ġdenote Ġa Ġsequence Ġof Ġwords Ġin Ġan Ġinput Ġsentence Ġas Ġ$ w _{ 1 : n } $ Ġ( Ġ$ n $ Ġ: Ġthe Ġlength Ġof Ġthe Ġsentence ), Ġand Ġthe Ġcorresponding Ġword Ġembed d ings Ġas Ġ$\ math bf Ġ{ x } _{ 1 : n } $ Ġ. ĠThen , Ġthe Ġoperation Ġof Ġthe Ġleaf - L ST M Ġat Ġtime Ġ$ t $ Ġcan Ġbe Ġformulated Ġas , </s> $$ Ġ Ġ\ begin { b mat rix } Ġ\ t ilde {\ math bf Ġ{ i }} Ġ\\ Ġ\ t ilde {\ math bf Ġ{ f }} Ġ\\ Ġ\ t ilde {\ math bf Ġ{ o }} Ġ\\ Ġ\ t ilde {\ math bf Ġ{ g }} Ġ\ end { b mat rix } Ġ= Ġ\ begin { b mat rix } Ġ\ s igma Ġ\\ Ġ\ s igma Ġ\\ Ġ\ s igma Ġ\\ Ġ\ tan h Ġ\ end { b mat rix } Ġ\ B igg Ġ( Ġ\ math bf Ġ{ W } _ \ text { L } Ġ\ begin { b mat rix } Ġ\ t ilde {\ math bf Ġ{ h }} _{ t - 1 } Ġ\\ Ġ\ math bf Ġ{ x } _ t Ġ\\ Ġ\ end { b mat rix } Ġ+ Ġ\ math bf Ġ{ b } _ \ text { L } Ġ\ B igg Ġ) $$ Ġ Ġ Ġ( E q . Ġ18 ) </s> $$ Ġ Ġ\ t ilde {\ math bf Ġ{ c }} _ t Ġ= Ġ\ t ilde {\ math bf Ġ{ f }} Ġ\ od ot Ġ\ t ilde {\ math bf Ġ{ c }} _{ t - 1 } Ġ+ Ġ\ t ilde {\ math bf Ġ{ i }} Ġ\ od ot Ġ\ t ilde {\ math bf Ġ{ g }} \\ $$ Ġ Ġ Ġ( E q . Ġ19 ) </s> where Ġ$\ math bf Ġ{ x } _ t Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ w } $ Ġindicates Ġan Ġinput Ġword Ġvector Ġand Ġ$\ t ilde {\ math bf Ġ{ h }} _ t $ Ġ, Ġ$\ t ilde {\ math bf Ġ{ c }} _ t Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ h } $ Ġrepresent Ġthe Ġhidden Ġand Ġcell Ġstate Ġof Ġthe ĠL ST M Ġat Ġtime Ġ$ t $ Ġ( Ġ$\ t ilde {\ math bf Ġ{ h }} _{ t - 1 } $ Ġcorresponds Ġto Ġthe Ġhidden Ġstate Ġat Ġtime Ġ$ t $ Ġ- 1 ). Ġ$\ math bf Ġ{ W } _ \ text { L } $ Ġand Ġ$\ math bf Ġ{ b } _ \ text { L } Ġ$ Ġare Ġlearn able Ġparameters . ĠThe Ġremaining Ġnotation Ġfollows Ġthat Ġof Ġthe Ġtree - L ST M Ġabove . </s> In Ġexperiments , Ġwe Ġdemonstrate Ġthat Ġintroducing Ġa Ġleaf - L ST M Ġfares Ġbetter Ġat Ġprocessing Ġthe Ġinput Ġwords Ġof Ġa Ġtree - L ST M Ġcompared Ġto Ġusing Ġa Ġfeed - forward Ġneural Ġnetwork . ĠWe Ġalso Ġexplore Ġthe Ġpossibility Ġof Ġits Ġbid irection al Ġsetting Ġin Ġab lation Ġstudy . </s> S ATA ĠTree - L ST M </s> In Ġthis Ġsection , Ġwe Ġdefine ĠSATA ĠTree - L ST M Ġ( St ructure - A ware ĠTag ĠAug mented ĠTree - L ST M , Ġsee ĠFigure Ġ2 Ġ) Ġwhich Ġjoins Ġa Ġtag - level Ġtree - L ST M Ġ( section Ġ3 . 2 ), Ġa Ġleaf - L ST M Ġ( section Ġ3 . 3 ), Ġand Ġthe Ġoriginal Ġword Ġtree - L ST M Ġtogether . </s> As Ġabove Ġwe Ġdenote Ġa Ġsequence Ġof Ġwords Ġin Ġan Ġinput Ġsentence Ġas Ġ$ w _{ 1 : n } $ Ġand Ġthe Ġcorresponding Ġword Ġembed d ings Ġas Ġ$\ math bf Ġ{ x } _{ 1 : n } $ Ġ. ĠIn Ġaddition , Ġa Ġtag Ġembed ding Ġfor Ġthe Ġtag Ġattached Ġto Ġeach Ġnode Ġin Ġa Ġtree Ġis Ġden oted Ġby Ġ$\ text bf Ġ{ e } Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ \ text { T }} $ Ġ. ĠThen , Ġwe Ġderive Ġthe Ġfinal Ġsentence Ġrepresentation Ġfor Ġthe Ġinput Ġsentence Ġwith Ġour Ġmodel Ġin Ġtwo Ġsteps . </s> First , Ġwe Ġcompute Ġstructure - aware Ġtag Ġrepresentations Ġ( Ġ$\ hat {\ math bf Ġ{ h }} $ Ġ) Ġfor Ġeach Ġnode Ġof Ġa Ġtree Ġusing Ġthe Ġtag Ġtree - L ST M Ġ( the Ġright Ġside Ġof ĠFigure Ġ2 Ġ) Ġas Ġfollows : </s> $$ Ġ Ġ\ begin { b mat rix } Ġ\ hat {\ math bf Ġ{ c }} Ġ\\ Ġ\ hat {\ math bf Ġ{ h }} Ġ\\ Ġ\ end { b mat rix } Ġ= Ġ{\ left \ l brace Ġ\ begin { array }{ ll } Ġ\ text { Tag - Tree - L ST M } (\ math bf Ġ{ e }) Ġ& Ġ\ text { if Ġa Ġleaf Ġnode } Ġ\\ Ġ\ text { Tag - Tree - L ST M } (\ hat {\ math bf Ġ{ h }} _ l , Ġ\ hat {\ math bf Ġ{ h }} _ r , Ġ\ math bf Ġ{ e }) Ġ& Ġ\ text { other wise } Ġ\ end { array }\ right .} $$ Ġ Ġ Ġ( E q . Ġ23 ) </s> where ĠTag - Tree - L ST M Ġindicates Ġthe Ġmodule Ġwe Ġdescribed Ġin Ġsection Ġ3 . 2 . </s> Second , Ġwe Ġcombine Ġsemantic Ġunits Ġrec urs ively Ġon Ġthe Ġword Ġtree - L ST M Ġin Ġa Ġbottom - up Ġfashion . ĠFor Ġleaf Ġnodes , Ġwe Ġleverage Ġthe ĠLeaf - L ST M Ġ( the Ġbottom - left Ġof ĠFigure Ġ2 Ġ, Ġexplained Ġin Ġsection Ġ3 . 3 ) Ġto Ġcompute Ġ$\ t ilde {\ math bf Ġ{ c }} _{ t } $ Ġand Ġ$\ t ilde {\ math bf Ġ{ h }} _{ t } $ Ġin Ġsequential Ġorder , Ġwith Ġthe Ġcorresponding Ġinput Ġ$\ math bf Ġ{ x } _ t $ Ġ. </s> $$ Ġ Ġ\ begin { b mat rix } Ġ\ t ilde {\ math bf Ġ{ c }} _{ t } Ġ\\ Ġ\ t ilde {\ math bf Ġ{ h }} _{ t } Ġ\\ Ġ\ end { b mat rix } Ġ= Ġ\ text { Le af - L ST M } (\ t ilde {\ text bf Ġ{ h }} _{ t - 1 }, Ġ\ text bf Ġ{ x } _ t ) $$ Ġ Ġ Ġ( E q . Ġ24 ) </s> Then , Ġthe Ġ$\ t ilde {\ math bf Ġ{ c }} _{ t } $ Ġand Ġ$\ t ilde {\ math bf Ġ{ h }} _{ t } $ Ġcan Ġbe Ġutilized Ġas Ġinput Ġtokens Ġto Ġthe Ġword Ġtree - L ST M , Ġwith Ġthe Ġleft Ġ( right ) Ġchild Ġof Ġthe Ġtarget Ġnode Ġcorresponding Ġto Ġthe Ġ$ t $ Ġth Ġword Ġin Ġthe Ġinput Ġsentence . </s> $$ Ġ Ġ\ begin { b mat rix } Ġ\ check {\ text bf Ġ{ c }} _{ \ l brace Ġl , Ġr \ r brace Ġ} Ġ\\ Ġ\ check {\ text bf Ġ{ h }} _{ \ l brace Ġl , Ġr \ r brace Ġ} Ġ\ end { b mat rix } Ġ= Ġ\ begin { b mat rix } Ġ\ t ilde {\ text bf Ġ{ c }} _{ t } Ġ\\ Ġ\ t ilde {\ text bf Ġ{ h }} _{ t } Ġ\ end { b mat rix } $$ Ġ Ġ Ġ( E q . Ġ25 ) </s> In Ġthe Ġnon - leaf Ġnode Ġcase , Ġwe Ġcalculate Ġphrase Ġrepresentations Ġfor Ġeach Ġnode Ġin Ġthe Ġword Ġtree - L ST M Ġ( the Ġupper - left Ġof ĠFigure Ġ2 Ġ) Ġrec urs ively Ġas Ġfollows : </s> $$ Ġ Ġ\ check {\ math bf Ġ{ g }} Ġ= Ġ\ tan h Ġ{\ left ( Ġ\ math bf Ġ{ U } _ \ text { w } Ġ\ begin { b mat rix } Ġ\ check {\ math bf Ġ{ h }} _ l Ġ\\ Ġ\ check {\ math bf Ġ{ h }} _ r Ġ\\ Ġ\ end { b mat rix } Ġ+ Ġ\ math bf Ġ{ a } _ \ text { w } Ġ\ right )} $$ Ġ Ġ Ġ( E q . Ġ26 ) </s> $$ Ġ Ġ\ begin { b mat rix } Ġ\ check {\ math bf Ġ{ i }} Ġ\\ Ġ\ check {\ math bf Ġ{ f }} _ l Ġ\\ Ġ\ check {\ math bf Ġ{ f }} _ r Ġ\\ Ġ\ check {\ math bf Ġ{ o }} Ġ\ end { b mat rix } Ġ= Ġ\ begin { b mat rix } Ġ\ s igma Ġ\\ Ġ\ s igma Ġ\\ Ġ\ s igma Ġ\\ Ġ\ s igma Ġ\ end { b mat rix } Ġ\ B igg Ġ( Ġ\ math bf Ġ{ W _ \ text { w }} Ġ\ begin { b mat rix } Ġ\ check {\ math bf Ġ{ h }} _ l Ġ\\ Ġ\ check {\ math bf Ġ{ h }} _ r Ġ\\ Ġ\ hat {\ math bf Ġ{ h }} Ġ\\ Ġ\ end { b mat rix } Ġ+ Ġ\ math bf Ġ{ b } _ \ text { w } Ġ\ B igg Ġ) $$ Ġ Ġ Ġ( E q . Ġ27 ) </s> where Ġ$\ check {\ math bf Ġ{ h }} $ Ġ, Ġ$\ check {\ math bf Ġ{ c }} Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ h } $ Ġrepresent Ġthe Ġhidden Ġand Ġcell Ġstate Ġof Ġeach Ġnode Ġin Ġthe Ġword Ġtree - L ST M . Ġ$\ math bf Ġ{ U } _ \ text { w } Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ h Ġ\ times Ġ2 d _ h } $ Ġ, Ġ$\ math bf Ġ{ W } _ \ text { w } Ġ\ in Ġ\ math bb Ġ{ R } ^{ 4 d _ h Ġ\ times Ġ\ left ( 2 d _ h + d _ \ text { T }\ right )} $ Ġ, Ġ$\ math bf Ġ{ a } _ \ text { w } Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ h } $ Ġ, Ġ$\ math bf Ġ{ b } _ \ text { w } Ġ\ in Ġ\ math bb Ġ{ R } ^{ 4 d _ h } $ Ġare Ġlearned Ġparameters . ĠThe Ġremaining Ġnotation Ġfollows Ġthose Ġof Ġthe Ġprevious Ġsections . ĠNote Ġthat Ġthe Ġstructure - aware Ġtag Ġrepresentations Ġ( Ġ$\ hat {\ math bf Ġ{ h }} $ Ġ) Ġare Ġonly Ġutilized Ġto Ġcontrol Ġthe Ġgate Ġfunctions Ġof Ġthe Ġword Ġtree - L ST M Ġin Ġthe Ġform Ġof Ġadditional Ġinputs , Ġand Ġare Ġnot Ġinvolved Ġin Ġthe Ġsemantic Ġcomposition Ġ( Ġ$\ check {\ math bf Ġ{ g }} $ Ġ) Ġdirectly . </s> Finally , Ġthe Ġhidden Ġstate Ġof Ġthe Ġroot Ġnode Ġ( Ġ$\ check {\ math bf Ġ{ h }} _ \ text { root } $ Ġ) Ġin Ġthe Ġword - level Ġtree - L ST M Ġbecomes Ġthe Ġfinal Ġsentence Ġrepresentation Ġof Ġthe Ġinput Ġsentence . </s> Quant itative ĠAnalysis </s> One Ġof Ġthe Ġmost Ġbasic Ġapproaches Ġto Ġevaluate Ġa Ġsentence Ġenc oder Ġis Ġto Ġmeasure Ġthe Ġclassification Ġperformance Ġwith Ġthe Ġsentence Ġrepresentations Ġmade Ġby Ġthe Ġenc oder . ĠThus , Ġwe Ġconduct Ġexperiments Ġon Ġthe Ġfollowing Ġfive Ġdatasets . Ġ( Summary Ġstatistics Ġfor Ġthe Ġdatasets Ġare Ġreported Ġin Ġthe Ġsupplemental Ġmaterials .) </s> MR : ĠA Ġgroup Ġof Ġmovie Ġreviews Ġwith Ġbinary Ġ( positive Ġ/ Ġnegative ) Ġclasses . ĠB IB REF 22 </s> S ST - 2 : ĠStanford ĠSent iment ĠTree bank ĠB IB REF 6 Ġ. ĠSimilar Ġto ĠMR , Ġbut Ġeach Ġreview Ġis Ġprovided Ġin Ġthe Ġform Ġof Ġa Ġbinary Ġparse Ġtree Ġwhose Ġnodes Ġare Ġannot ated Ġwith Ġnumeric Ġsentiment Ġvalues . ĠFor ĠS ST - 2 , Ġwe Ġonly Ġconsider Ġbinary Ġ( positive Ġ/ Ġnegative ) Ġclasses . </s> S ST - 5 : ĠIdent ical Ġto ĠS ST - 2 , Ġbut Ġthe Ġreviews Ġare Ġgrouped Ġinto Ġfine - gr ained Ġ( very Ġnegative , Ġnegative , Ġneutral , Ġpositive , Ġvery Ġpositive ) Ġclasses . </s> S UB J : ĠSent ences Ġgrouped Ġas Ġbeing Ġeither Ġsubjective Ġor Ġobjective Ġ( binary Ġclasses ). ĠB IB REF 23 </s> T REC : ĠA Ġdataset Ġwhich Ġgroups Ġquestions Ġinto Ġsix Ġdifferent Ġquestion Ġtypes Ġ( classes ). ĠB IB REF 24 </s> As Ġa Ġpre processing Ġstep , Ġwe Ġconstruct Ġparse Ġtrees Ġfor Ġthe Ġsentences Ġin Ġthe Ġdatasets Ġusing Ġthe ĠStanford ĠPC FG Ġparser ĠB IB REF 25 Ġ. ĠBecause Ġsynt actic Ġtags Ġare Ġby - products Ġof Ġconstituency Ġparsing , Ġwe Ġdo Ġnot Ġneed Ġfurther Ġpre processing . </s> To Ġclassify Ġthe Ġsentence Ġgiven Ġour Ġsentence Ġrepresentation Ġ( Ġ$\ check {\ math bf Ġ{ h }} _ \ text { root } $ Ġ), Ġwe Ġuse Ġone Ġfully - connected Ġlayer Ġwith Ġa ĠRe LU Ġactivation , Ġfollowed Ġby Ġa Ġsoft max Ġclass ifier . ĠThe Ġfinal Ġpredicted Ġprobability Ġdistribution Ġof Ġthe Ġclass Ġ$ y $ Ġgiven Ġthe Ġsentence Ġ$ w _{ 1 : n } $ Ġis Ġdefined Ġas Ġfollows , </s> $$ \ math bf Ġ{ s } Ġ= Ġ\ text { Re LU } (\ math bf Ġ{ W } _ \ text { s } Ġ\ check {\ math bf Ġ{ h }} _ \ text { root } + Ġ\ math bf Ġ{ b } _ \ text { s }) $$ Ġ Ġ Ġ( E q . Ġ37 ) </s> $$ p ( y | w _{ 1 : n }) Ġ= Ġ\ text { soft max } (\ math bf Ġ{ W } _ \ text { c }\ math bf Ġ{ s } Ġ+ Ġ\ math bf Ġ{ b } _ \ text { c }) $$ Ġ Ġ Ġ( E q . Ġ38 ) </s> where Ġ$\ text bf Ġ{ s } Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ \ text { s }} $ Ġis Ġthe Ġcomputed Ġtask - specific Ġsentence Ġrepresentation Ġfor Ġthe Ġclass ifier , Ġand Ġ$\ text bf Ġ{ W } _ \ text { s } Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ \ text { s } Ġ\ times Ġd _ h } $ Ġ, Ġ$\ text bf Ġ{ W } _ \ text { c } Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ \ text { c } Ġ\ times Ġd _ s } $ Ġ, Ġ$\ text bf Ġ{ b } _ \ text { s } Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ s } $ Ġ, Ġ$\ text bf Ġ{ b } _ \ text { c } Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ c } $ Ġare Ġtrain able Ġparameters . ĠAs Ġan Ġobjective Ġfunction , Ġwe Ġuse Ġthe Ġcross Ġentropy Ġof Ġthe Ġpredicted Ġand Ġtrue Ġclass Ġdistributions . </s> The Ġresults Ġof Ġthe Ġexperiments Ġon Ġthe Ġfive Ġdatasets Ġare Ġshown Ġin Ġtable Ġ1 Ġ. ĠIn Ġthis Ġtable , Ġwe Ġreport Ġthe Ġtest Ġaccuracy Ġof Ġour Ġmodel Ġand Ġvarious Ġother Ġmodels Ġon Ġeach Ġdataset Ġin Ġterms Ġof Ġpercentage . ĠTo Ġconsider Ġthe Ġeffects Ġof Ġrandom Ġinitialization , Ġwe Ġreport Ġthe Ġbest Ġnumbers Ġobtained Ġfrom Ġeach Ġseveral Ġruns Ġwith Ġhyper - param eters Ġfixed . </s> Compared Ġwith Ġthe Ġprevious Ġsynt actic Ġtree - based Ġmodels Ġas Ġwell Ġas Ġother Ġneural Ġmodels , Ġour ĠSATA ĠTree - L ST M Ġshows Ġsuperior Ġor Ġcompetitive Ġperformance Ġon Ġall Ġtasks . ĠSpecifically , Ġour Ġmodel Ġachieves Ġnew Ġstate - of - the - art Ġresults Ġwithin Ġthe Ġtree - struct ured Ġmodel Ġclass Ġon Ġ4 Ġout Ġof Ġ5 Ġsentence Ġclassification Ġtasks âĢĶ S ST - 2 , ĠS ST - 5 , ĠMR , Ġand ĠT REC . ĠThe Ġmodel Ġshows Ġits Ġstrength , Ġin Ġparticular , Ġwhen Ġthe Ġdatasets Ġprovide Ġphrase - level Ġsupervision Ġto Ġfacilitate Ġtree Ġstructure Ġlearning Ġ( i . e . ĠS ST - 2 , ĠS ST - 5 ). ĠMoreover , Ġthe Ġnumbers Ġwe Ġreport Ġfor ĠS ST - 5 Ġand ĠT REC Ġare Ġcompetitive Ġto Ġthe Ġexisting Ġstate - of - the - art Ġresults Ġincluding Ġones Ġfrom Ġstruct urally Ġpre - trained Ġmodels Ġsuch Ġas ĠEL Mo ĠB IB REF 26 Ġ, Ġproving Ġour Ġmodel 's Ġsuperiority . ĠNote Ġthat Ġthe ĠSATA ĠTree - L ST M Ġalso Ġoutper forms Ġthe Ġrecent Ġlatent Ġtree - based Ġmodel , Ġindicating Ġthat Ġmodeling Ġa Ġneural Ġmodel Ġwith Ġexplicit Ġlinguistic Ġknowledge Ġcan Ġbe Ġan Ġattractive Ġoption . </s> On Ġthe Ġother Ġhand , Ġa Ġremaining Ġconcern Ġis Ġthat Ġour ĠSATA ĠTree - L ST M Ġis Ġnot Ġrobust Ġto Ġrandom Ġseeds Ġwhen Ġthe Ġsize Ġof Ġa Ġdataset Ġis Ġrelatively Ġsmall , Ġas Ġtag Ġembed d ings Ġare Ġrandomly Ġinitialized Ġrather Ġthan Ġrelying Ġon Ġpre - trained Ġones Ġin Ġcontrast Ġwith Ġthe Ġcase Ġof Ġwords . ĠFrom Ġthis Ġobservation , Ġwe Ġcould Ġfind Ġout Ġthere Ġneeds Ġa Ġdirection Ġof Ġresearch Ġtowards Ġpre - trained Ġtag Ġembed d ings . </s> To Ġestimate Ġthe Ġperformance Ġof Ġour Ġmodel Ġbeyond Ġthe Ġtasks Ġrequiring Ġonly Ġone Ġsentence Ġat Ġa Ġtime , Ġwe Ġconduct Ġan Ġexperiment Ġon Ġthe ĠStanford ĠNatural ĠLanguage ĠIn ference ĠB IB REF 34 Ġdataset , Ġeach Ġexample Ġof Ġwhich Ġconsists Ġof Ġtwo Ġsentences , Ġthe Ġpremise Ġand Ġthe Ġhypothesis . ĠOur Ġobjective Ġgiven Ġthe Ġdata Ġis Ġto Ġpredict Ġthe Ġcorrect Ġrelationship Ġbetween Ġthe Ġtwo Ġsentences Ġamong Ġthree Ġoptions âĢĶ Ġcontradiction , Ġneutral , Ġor Ġentail ment . </s> We Ġuse Ġthe Ġsi ames e Ġarchitecture Ġto Ġencode Ġboth Ġthe Ġpremise Ġ( Ġ$ p _{ 1 : m } $ Ġ) Ġand Ġhypothesis Ġ( Ġ$ h _{ 1 : n } $ Ġ) Ġfollowing Ġthe Ġstandard Ġof Ġsentence - enc oding Ġmodels Ġin Ġthe Ġliterature . Ġ( Specifically , Ġ$ p _{ 1 : m } $ Ġis Ġencoded Ġas Ġ$\ check {\ math bf Ġ{ h }} _ \ text { root } ^ p Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ h } $ Ġand Ġ$ h _{ 1 : n } $ Ġis Ġencoded Ġas Ġ$\ check {\ math bf Ġ{ h }} _ \ text { root } ^ h Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ h } $ Ġwith Ġthe Ġsame Ġenc oder .) ĠThen , Ġwe Ġleverage Ġsome Ġhe ur istics ĠB IB REF 35 Ġ, Ġfollowed Ġby Ġone Ġfully - connected Ġlayer Ġwith Ġa ĠRe LU Ġactivation Ġand Ġa Ġsoft max Ġclass ifier . ĠSpecifically , </s> $$ \ math bf Ġ{ z } Ġ= Ġ\ left [ Ġ\ check {\ math bf Ġ{ h }} _ \ text { root } ^ p ; Ġ\ check {\ math bf Ġ{ h }} _ \ text { root } ^ h ; Ġ| Ġ\ check {\ math bf Ġ{ h }} _ \ text { root } ^ p Ġ- Ġ\ check {\ math bf Ġ{ h }} _ \ text { root } ^ h Ġ| ; Ġ\ check {\ math bf Ġ{ h }} _ \ text { root } ^ p Ġ\ od ot Ġ\ check {\ math bf Ġ{ h }} _ \ text { root } ^ h Ġ\ right ] $$ Ġ Ġ Ġ( E q . Ġ41 ) </s> $$ \ math bf Ġ{ s } Ġ= Ġ\ text { Re LU } (\ math bf Ġ{ W } _ \ text { s } Ġ\ math bf Ġ{ z } Ġ+ Ġ\ math bf Ġ{ b } _ \ text { s }) $$ Ġ Ġ Ġ( E q . Ġ42 ) </s> where Ġ$\ text bf Ġ{ z } Ġ\ in Ġ\ math bb Ġ{ R } ^{ 4 d _ h } $ Ġ, Ġ$\ text bf Ġ{ s } Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ s } $ Ġare Ġintermediate Ġfeatures Ġfor Ġthe Ġclass ifier Ġand Ġ$\ text bf Ġ{ W } _ \ text { s } Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ \ text { s } Ġ\ times Ġ4 d _ h } $ Ġ, Ġ$\ text bf Ġ{ W } _ \ text { c } Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ \ text { c } Ġ\ times Ġd _ s } $ Ġ, Ġ$\ text bf Ġ{ b } _ \ text { s } Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ s } $ Ġ, Ġ$\ text bf Ġ{ b } _ \ text { c } Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ c } $ Ġare Ġagain Ġtrain able Ġparameters . </s> Our Ġexperimental Ġresults Ġon Ġthe ĠSN LI Ġdataset Ġare Ġshown Ġin Ġtable Ġ2 Ġ. ĠIn Ġthis Ġtable , Ġwe Ġreport Ġthe Ġtest Ġaccuracy Ġand Ġnumber Ġof Ġtrain able Ġparameters Ġfor Ġeach Ġmodel . ĠOur ĠSATA - L ST M Ġagain Ġdemonstrates Ġits Ġdecent Ġperformance Ġcompared Ġagainst Ġthe Ġneural Ġmodels Ġbuilt Ġon Ġboth Ġsynt actic Ġtrees Ġand Ġlatent Ġtrees , Ġas Ġwell Ġas Ġthe Ġnon - tree Ġmodels . Ġ( Lat ent ĠSynt ax ĠTree - L ST M : ĠB IB REF 10 Ġ( ĠB IB REF 10 Ġ), ĠTree - based ĠCNN : ĠB IB REF 35 Ġ( ĠB IB REF 35 Ġ), ĠG umb el ĠTree - L ST M : ĠB IB REF 11 Ġ( ĠB IB REF 11 Ġ), ĠN SE : ĠB IB REF 36 Ġ( ĠB IB REF 36 Ġ), ĠRein forced ĠSelf - Att ention ĠNetwork : ĠB IB REF 4 Ġ( ĠB IB REF 4 Ġ), ĠRes id ual Ġstacked Ġenc od ers : ĠB IB REF 37 Ġ( ĠB IB REF 37 Ġ), ĠBi L ST M Ġwith Ġgeneralized Ġpool ing : ĠB IB REF 38 Ġ( ĠB IB REF 38 Ġ). ) ĠNote Ġthat Ġthe Ġnumber Ġof Ġlearned Ġparameters Ġin Ġour Ġmodel Ġis Ġalso Ġcomparable Ġto Ġother Ġsophisticated Ġmodels , Ġshowing Ġthe Ġefficiency Ġof Ġour Ġmodel . </s> Even Ġthough Ġour Ġmodel Ġhas Ġproven Ġits Ġmet tle , Ġthe Ġeffect Ġof Ġtag Ġinformation Ġseems Ġrelatively Ġweak Ġin Ġthe Ġcase Ġof ĠSN LI , Ġwhich Ġcontains Ġa Ġlarge Ġamount Ġof Ġdata Ġcompared Ġto Ġthe Ġothers . ĠOne Ġpossible Ġexplanation Ġis Ġthat Ġneural Ġmodels Ġmay Ġlearn Ġsome Ġsynt actic Ġrules Ġfrom Ġlarge Ġamounts Ġof Ġtext Ġwhen Ġthe Ġtext Ġsize Ġis Ġlarge Ġenough , Ġreducing Ġthe Ġnecessity Ġof Ġexternal Ġlinguistic Ġknowledge . ĠWe Ġleave Ġthe Ġexploration Ġof Ġthe Ġeffectiveness Ġof Ġtags Ġrelative Ġto Ġdata Ġsize Ġfor Ġfuture Ġwork . </s> Here Ġwe Ġgo Ġover Ġthe Ġsettings Ġcommon Ġacross Ġour Ġmodels Ġduring Ġexperimentation . ĠFor Ġmore Ġtask - specific Ġdetails , Ġrefer Ġto Ġthe Ġsupplemental Ġmaterials . </s> For Ġour Ġinput Ġembed d ings , Ġwe Ġused Ġ300 Ġdimensional Ġ840 B ĠGl o Ve ĠB IB REF 39 Ġas Ġpre - trained Ġword Ġembed d ings , Ġand Ġtag Ġrepresentations Ġwere Ġrandomly Ġsampled Ġfrom Ġthe Ġuniform Ġdistribution Ġ[- 0 . 005 , Ġ0 . 005 ]. ĠTag Ġvectors Ġare Ġrevised Ġduring Ġtraining Ġwhile Ġthe Ġfine - tun ing Ġof Ġthe Ġword Ġembed ding Ġdepends Ġon Ġthe Ġtask . ĠOur Ġmodels Ġwere Ġtrained Ġusing Ġthe ĠAdam ĠB IB REF 40 Ġor ĠAd ad elta ĠB IB REF 41 Ġoptim izer , Ġdepending Ġon Ġtask . ĠFor Ġregular ization , Ġweight Ġdecay Ġis Ġadded Ġto Ġthe Ġloss Ġfunction Ġexcept Ġfor ĠSN LI Ġfollowing ĠB IB REF 42 Ġ( ĠB IB REF 42 Ġ) Ġand ĠDrop out ĠB IB REF 43 Ġis Ġalso Ġapplied Ġfor Ġthe Ġword Ġembed d ings Ġand Ġtask - specific Ġclass ifiers . ĠMoreover , Ġbatch Ġnormal ization ĠB IB REF 44 Ġis Ġadopted Ġfor Ġthe Ġclass ifiers . ĠAs Ġa Ġdefault , Ġall Ġthe Ġweights Ġin Ġthe Ġmodel Ġare Ġinitialized Ġfollowing ĠB IB REF 45 Ġ( ĠB IB REF 45 Ġ) Ġand Ġthe Ġbiases Ġare Ġset Ġto Ġ0 . ĠThe Ġtotal Ġnorm Ġof Ġthe Ġgrad ients Ġof Ġthe Ġparameters Ġis Ġclipped Ġnot Ġto Ġbe Ġover Ġ5 Ġduring Ġtraining . </s> Our Ġbest Ġmodels Ġfor Ġeach Ġdataset Ġwere Ġchosen Ġby Ġvalidation Ġaccuracy Ġin Ġcases Ġwhere Ġa Ġvalidation Ġset Ġwas Ġprovided Ġas Ġa Ġpart Ġof Ġthe Ġdataset . ĠOtherwise , Ġwe Ġperform Ġa Ġgrid Ġsearch Ġon Ġprobable Ġhyper - param eter Ġsettings , Ġor Ġrun Ġ10 - fold Ġcross - valid ation Ġin Ġcases Ġwhere Ġeven Ġa Ġtest Ġset Ġdoes Ġnot Ġexist . </s> A bl ation ĠStudy </s> In Ġthis Ġsection , Ġwe Ġdesign Ġan Ġab lation Ġstudy Ġon Ġthe Ġcore Ġmodules Ġof Ġour Ġmodel Ġto Ġexplore Ġtheir Ġeffectiveness . ĠThe Ġdataset Ġused Ġin Ġthis Ġexperiment Ġis ĠS ST - 2 . ĠTo Ġconduct Ġthe Ġexperiment , Ġwe Ġonly Ġreplace Ġthe Ġtarget Ġmodule Ġwith Ġother Ġcandidates Ġwhile Ġmaintaining Ġthe Ġother Ġsettings . ĠTo Ġbe Ġspecific , Ġwe Ġfocus Ġon Ġtwo Ġmodules , Ġthe Ġleaf - L ST M Ġand Ġstructure - aware Ġtag Ġembed d ings Ġ( tag - level Ġtree - L ST M ). ĠIn Ġthe Ġfirst Ġcase , Ġthe Ġleaf - L ST M Ġis Ġreplaced Ġwith Ġa Ġfully - connected Ġlayer Ġwith Ġa Ġ$\ tan h Ġ$ Ġactivation Ġor ĠBi - L ST M . ĠIn Ġthe Ġsecond Ġcase , Ġwe Ġreplace Ġthe Ġstructure - aware Ġtag Ġembed d ings Ġwith Ġnaive Ġtag Ġembed d ings Ġor Ġdo Ġnot Ġemploy Ġthem Ġat Ġall . </s> The Ġexperimental Ġresults Ġare Ġdepicted Ġin ĠFigure Ġ3 Ġ. ĠAs Ġthe Ġchart Ġshows , Ġour Ġmodel Ġoutper forms Ġall Ġthe Ġother Ġoptions Ġwe Ġhave Ġconsidered . ĠIn Ġdetail , Ġthe Ġleft Ġpart Ġof Ġthe Ġchart Ġshows Ġthat Ġthe Ġleaf - L ST M Ġis Ġthe Ġmost Ġeffective Ġoption Ġcompared Ġto Ġits Ġcompetitors . ĠNote Ġthat Ġthe Ġsequential Ġleaf - L ST M Ġis Ġsomewhat Ġsuperior Ġor Ġcompetitive Ġthan Ġthe Ġbid irection al Ġleaf - L ST M Ġwhen Ġboth Ġhave Ġa Ġcomparable Ġnumber Ġof Ġparameters . ĠWe Ġconjecture Ġthis Ġmay Ġbecause Ġa Ġbackward ĠL ST M Ġdoes Ġnot Ġadd Ġadditional Ġuseful Ġknowledge Ġwhen Ġthe Ġstructure Ġof Ġa Ġsentence Ġis Ġalready Ġknown . ĠIn Ġconclusion , Ġwe Ġuse Ġthe Ġun i - direction al ĠL ST M Ġas Ġa Ġleaf Ġmodule Ġbecause Ġof Ġits Ġsimplicity Ġand Ġremarkable Ġperformance . </s> Meanwhile , Ġthe Ġright Ġpart Ġof Ġthe Ġfigure Ġdemonstrates Ġthat Ġour Ġnewly Ġintroduced Ġstructure - aware Ġembed d ings Ġhave Ġa Ġreal Ġimpact Ġon Ġimproving Ġthe Ġmodel Ġperformance . ĠInterestingly , Ġemploying Ġthe Ġnaive Ġtag Ġembed d ings Ġmade Ġno Ġdifference Ġin Ġterms Ġof Ġthe Ġtest Ġaccuracy , Ġeven Ġthough Ġthe Ġabsolute Ġvalidation Ġaccuracy Ġincreased Ġ( not Ġreported Ġin Ġthe Ġfigure ). ĠThis Ġresult Ġsupports Ġour Ġassumption Ġthat Ġtag Ġinformation Ġshould Ġbe Ġconsidered Ġin Ġthe Ġstructure . </s> Qual itative ĠAnalysis </s> In Ġprevious Ġsections , Ġwe Ġhave Ġnumer ically Ġdemonstrated Ġthat Ġour Ġmodel Ġis Ġeffective Ġin Ġencouraging Ġuseful Ġcomposition Ġof Ġsemantic Ġunits . ĠHere , Ġwe Ġdirectly Ġinvestigate Ġthe Ġcomputed Ġrepresentations Ġfor Ġeach Ġnode Ġof Ġa Ġtree , Ġshowing Ġthat Ġthe Ġremarkable Ġperformance Ġof Ġour Ġmodel Ġis Ġmainly Ġdue Ġto Ġthe Ġgradual Ġand Ġrecursive Ġcomposition Ġof Ġthe Ġintermediate Ġrepresentations Ġon Ġthe Ġsynt actic Ġstructure . </s> To Ġobserve Ġthe Ġphrase - level Ġembed d ings Ġat Ġa Ġglance , Ġwe Ġdraw Ġa Ġscatter Ġplot Ġin Ġwhich Ġa Ġpoint Ġrepresents Ġthe Ġcorresponding Ġintermediate Ġrepresentation . ĠWe Ġutilize ĠPC A Ġ( Princ ipal ĠComponent ĠAnalysis ) Ġto Ġproject Ġthe Ġrepresentations Ġinto Ġa Ġtwo - dimensional Ġvector Ġspace . ĠAs Ġa Ġtarget Ġparse Ġtree , Ġwe Ġreuse Ġthe Ġone Ġseen Ġin ĠFigure Ġ1 Ġ. ĠThe Ġresult Ġis Ġshown Ġin ĠFigure Ġ4 Ġ. </s> From Ġthis Ġfigure , Ġwe Ġconfirm Ġthat Ġthe Ġintermediate Ġrepresentations Ġhave Ġa Ġhierarchy Ġin Ġthe Ġsemantic Ġspace , Ġwhich Ġis Ġvery Ġsimilar Ġto Ġthat Ġof Ġthe Ġparse Ġtree . ĠIn Ġother Ġwords , Ġas Ġmany Ġtree - struct ured Ġmodels Ġpursue , Ġwe Ġcan Ġsee Ġthe Ġtendency Ġof Ġconstructing Ġthe Ġrepresentations Ġfrom Ġthe Ġlow - level Ġ( the Ġbottom Ġof Ġthe Ġfigure ) Ġto Ġthe Ġhigh - level Ġ( the Ġtop - left Ġand Ġtop - right Ġof Ġthe Ġfigure ), Ġintegrating Ġthe Ġmeaning Ġof Ġthe Ġconstituents Ġrec urs ively . ĠAn Ġinteresting Ġthing Ġto Ġnote Ġis Ġthat Ġthe Ġfinal Ġsentence Ġrepresentation Ġis Ġnear Ġthat Ġof Ġthe Ġphrase Ġ` , Ġthe Ġstories Ġare Ġquietly Ġmoving .' Ġrather Ġthan Ġthat Ġof Ġ` Despite Ġthe Ġfilm 's Ġshortcomings ', Ġcatching Ġthe Ġmain Ġmeaning Ġof Ġthe Ġsentence . </s> Conclusion </s> We Ġhave Ġproposed Ġa Ġnovel ĠR v NN Ġarchitecture Ġto Ġfully Ġutilize Ġlinguistic Ġpri ors . ĠA Ġnewly Ġintroduced Ġtag - level Ġtree - L ST M Ġdemonstrates Ġthat Ġit Ġcan Ġeffectively Ġcontrol Ġthe Ġcomposition Ġfunction Ġof Ġthe Ġcorresponding Ġword - level Ġtree - L ST M . ĠIn Ġaddition , Ġthe Ġproper Ġcontextual ization Ġof Ġthe Ġinput Ġword Ġvectors Ġresults Ġin Ġsignificant Ġperformance Ġimprovements Ġon Ġseveral Ġsentence - level Ġtasks . ĠFor Ġfuture Ġwork , Ġwe Ġplan Ġto Ġexplore Ġa Ġnew Ġway Ġof Ġexploiting Ġdependency Ġtrees Ġeffectively , Ġsimilar Ġto Ġthe Ġcase Ġof Ġconstituency Ġtrees . </s> Acknowled gments </s> We Ġthank Ġanonymous Ġreviewers Ġfor Ġtheir Ġconstructive Ġand Ġfruitful Ġcomments . ĠThis Ġwork Ġwas Ġsupported Ġby Ġthe ĠNational ĠResearch ĠFoundation Ġof ĠKorea Ġ( NR F ) Ġgrant Ġfunded Ġby Ġthe ĠKorea Ġgovernment Ġ( MS IT ) Ġ( NR F 2016 M 3 C 4 A 79 525 87 ).