doc2vec , CNN, DAN, Tree-LSTM, DRNN, LSTMN, C-LSTM, SPGK, WMD, S-WMD, Semantic-CNN, LSTM-GRNN, HN-ATT
EXTRACTIVE
Experiments ::: Baselines We evaluate MPAD against multiple state-of-the-art baseline models, including hierarchical ones, to enable fair comparison with the hierarchical MPAD variants. doc2vec BIBREF37. Doc2vec (or paragraph vector) is an extension of word2vec that learns vectors for documents in a fully unsupervised manner. Document embeddings are then fed to a logistic regression classifier. CNN BIBREF38. The convolutional neural network architecture, well-known in computer vision, is applied to text. There is one spatial dimension and the word embeddings are used as channels (depth dimensions). DAN BIBREF39. The Deep Averaging Network passes the unweighted average of the embeddings of the input words through multiple dense layers and a final softmax. Tree-LSTM BIBREF40 is a generalization of the standard LSTM architecture to constituency and dependency parse trees. DRNN BIBREF41. Recursive neural networks are stacked and applied to parse trees. LSTMN BIBREF42 is an extension of the LSTM model where the memory cell is replaced by a memory network which stores word representations. C-LSTM BIBREF43 combines convolutional and recurrent neural networks. The region embeddings provided by a CNN are fed to a LSTM. SPGK BIBREF44 also models documents as word co-occurrence networks. It computes a graph kernel that compares shortest paths extracted from the word co-occurrence networks and then uses a SVM to categorize documents. WMD BIBREF45 is an application of the well-known Earth Mover's Distance to text. A k-nearest neighbor classifier is used. S-WMD BIBREF46 is a supervised extension of the Word Mover's Distance. Semantic-CNN BIBREF47. Here, a CNN is applied to semantic units obtained by clustering words in the embedding space. LSTM-GRNN BIBREF26 is a hierarchical model where sentence embeddings are obtained with a CNN and a GRU-RNN is fed the sentence representations to obtain a document vector. HN-ATT BIBREF27 is another hierarchical model, where the same encoder architecture (a bidirectional GRU-RNN) is used for both sentences and documents, with different parameters. A self-attention mechanism is applied to the RNN annotations at each level.
1908.06267
Introduction </s> The Ġconcept Ġof Ġmessage Ġpassing Ġover Ġgraphs Ġhas Ġbeen Ġaround Ġfor Ġmany Ġyears ĠB IB REF 0 , ĠB IB REF 1 , Ġas Ġwell Ġas Ġthat Ġof Ġgraph Ġneural Ġnetworks Ġ( G NN s ) ĠB IB REF 2 , ĠB IB REF 3 . ĠHowever , ĠG NN s Ġhave Ġonly Ġrecently Ġstarted Ġto Ġbe Ġclosely Ġinvestigated , Ġfollowing Ġthe Ġadvent Ġof Ġdeep Ġlearning . ĠSome Ġnotable Ġexamples Ġinclude ĠB IB REF 4 , ĠB IB REF 5 , ĠB IB REF 6 , ĠB IB REF 7 , ĠB IB REF 8 , ĠB IB REF 9 , ĠB IB REF 10 , ĠB IB REF 11 , ĠB IB REF 12 . ĠThese Ġapproaches Ġare Ġknown Ġas Ġspectral . ĠTheir Ġsimilarity Ġwith Ġmessage Ġpassing Ġ( MP ) Ġwas Ġobserved Ġby ĠB IB REF 9 Ġand Ġformal ized Ġby ĠB IB REF 13 Ġand ĠB IB REF 14 . </s> The ĠMP Ġframework Ġis Ġbased Ġon Ġthe Ġcore Ġidea Ġof Ġrecursive Ġneighborhood Ġaggregation . ĠThat Ġis , Ġat Ġevery Ġiteration , Ġthe Ġrepresentation Ġof Ġeach Ġvertex Ġis Ġupdated Ġbased Ġon Ġmessages Ġreceived Ġfrom Ġits Ġneighbors . ĠAll Ġspectral ĠG NN s Ġcan Ġbe Ġdescribed Ġin Ġterms Ġof Ġthe ĠMP Ġframework . </s> G NN s Ġhave Ġbeen Ġapplied Ġwith Ġgreat Ġsuccess Ġto Ġbio in format ics Ġand Ġsocial Ġnetwork Ġdata , Ġfor Ġnode Ġclassification , Ġlink Ġprediction , Ġand Ġgraph Ġclassification . ĠHowever , Ġa Ġfew Ġstudies Ġonly Ġhave Ġfocused Ġon Ġthe Ġapplication Ġof Ġthe ĠMP Ġframework Ġto Ġrepresentation Ġlearning Ġon Ġtext . ĠThis Ġpaper Ġproposes Ġone Ġsuch Ġapplication . ĠMore Ġprecisely , Ġwe Ġrepresent Ġdocuments Ġas Ġword Ġco - occ urrence Ġnetworks , Ġand Ġdevelop Ġan Ġexpressive ĠMP ĠG NN Ġtailored Ġto Ġdocument Ġunderstanding , Ġthe ĠMessage ĠPassing ĠAttention Ġnetwork Ġfor ĠDocument Ġunderstanding Ġ( MP AD ). ĠWe Ġalso Ġpropose Ġseveral Ġhierarchical Ġvariants Ġof ĠMP AD . ĠEvaluation Ġon Ġ10 Ġdocument Ġclassification Ġdatasets Ġshow Ġthat Ġour Ġarchitectures Ġlearn Ġrepresentations Ġthat Ġare Ġcompetitive Ġwith Ġthe Ġstate - of - the - art . ĠFurthermore , Ġab lation Ġexperiments Ġshed Ġlight Ġon Ġthe Ġimpact Ġof Ġvarious Ġarchitectural Ġchoices . </s> In Ġwhat Ġfollows , Ġwe Ġfirst Ġprovide Ġsome Ġbackground Ġabout Ġthe ĠMP Ġframework Ġ( in Ġsec . ĠSEC REF 2 ), Ġthoroughly Ġdescribe Ġand Ġexplain ĠMP AD Ġ( sec . ĠSEC REF 3 ), Ġpresent Ġour Ġexperimental Ġframework Ġ( sec . ĠSEC REF 4 ), Ġreport Ġand Ġinterpret Ġour Ġresults Ġ( sec . ĠSEC REF 5 ), Ġand Ġprovide Ġa Ġreview Ġof Ġthe Ġrelevant Ġliterature Ġ( sec . ĠSEC REF 6 ). </s> Message ĠPassing ĠNeural ĠNetworks </s> BI B REF 13 Ġproposed Ġa ĠMP Ġframework Ġunder Ġwhich Ġmany Ġof Ġthe Ġrecently Ġintroduced ĠG NN s Ġcan Ġbe Ġreform ulated . ĠMP Ġconsists Ġin Ġan Ġaggregation Ġphase Ġfollowed Ġby Ġa Ġcombination Ġphase ĠB IB REF 14 . ĠMore Ġprecisely , Ġlet Ġ$ G ( V , E ) $ Ġbe Ġa Ġgraph , Ġand Ġlet Ġus Ġconsider Ġ$ v Ġ\ in ĠV $. ĠAt Ġtime Ġ$ t + 1 $, Ġa Ġmessage Ġvector Ġ$\ math bf Ġ{ m } _ v ^{ t + 1 } $ Ġis Ġcomputed Ġfrom Ġthe Ġrepresentations Ġof Ġthe Ġneighbors Ġ$\ math cal Ġ{ N } ( v ) $ Ġof Ġ$ v $ : </s> The Ġnew Ġrepresentation Ġ$\ math bf Ġ{ h } ^{ t + 1 } _ v $ Ġof Ġ$ v $ Ġis Ġthen Ġcomputed Ġby Ġcombining Ġits Ġcurrent Ġfeature Ġvector Ġ$\ math bf Ġ{ h } ^{ t } _ v $ Ġwith Ġthe Ġmessage Ġvector Ġ$\ math bf Ġ{ m } _ v ^{ t + 1 } $ : </s> Mess ages Ġare Ġpassed Ġfor Ġ$ T $ Ġtime Ġsteps . ĠEach Ġstep Ġis Ġimplemented Ġby Ġa Ġdifferent Ġlayer Ġof Ġthe ĠMP Ġnetwork . ĠHence , Ġiterations Ġcorrespond Ġto Ġnetwork Ġdepth . ĠThe Ġfinal Ġfeature Ġvector Ġ$\ math bf Ġ{ h } _ v ^ T $ Ġof Ġ$ v $ Ġis Ġbased Ġon Ġmessages Ġpropag ated Ġfrom Ġall Ġthe Ġnodes Ġin Ġthe Ġsubt ree Ġof Ġheight Ġ$ T $ Ġrooted Ġat Ġ$ v $. ĠIt Ġcaptures Ġboth Ġthe Ġtop ology Ġof Ġthe Ġneighborhood Ġof Ġ$ v $ Ġand Ġthe Ġdistribution Ġof Ġthe Ġvertex Ġrepresentations Ġin Ġit . </s> If Ġa Ġgraph - level Ġfeature Ġvector Ġis Ġneeded , Ġe . g ., Ġfor Ġclassification Ġor Ġregression , Ġa ĠREAD OUT Ġpool ing Ġfunction , Ġthat Ġmust Ġbe Ġinvari ant Ġto Ġperm utations , Ġis Ġapplied : </s> Next , Ġwe Ġpresent Ġthe ĠMP Ġnetwork Ġwe Ġdeveloped Ġfor Ġdocument Ġunderstanding . </s> Message ĠPassing ĠAttention Ġnetwork Ġfor ĠDocument Ġunderstanding Ġ( MP AD ) Ġ: :: ĠWord Ġco - occ urrence Ġnetworks </s> We Ġrepresent Ġa Ġdocument Ġas Ġa Ġstatistical Ġword Ġco - occ urrence Ġnetwork ĠB IB REF 18 , ĠB IB REF 19 Ġwith Ġa Ġsliding Ġwindow Ġof Ġsize Ġ2 Ġover span ning Ġsentences . ĠLet Ġus Ġdenote Ġthat Ġgraph Ġ$ G ( V , E ) $. ĠEach Ġunique Ġword Ġin Ġthe Ġpre process ed Ġdocument Ġis Ġrepresented Ġby Ġa Ġnode Ġin Ġ$ G $, Ġand Ġan Ġedge Ġis Ġadded Ġbetween Ġtwo Ġnodes Ġif Ġthey Ġare Ġfound Ġtogether Ġin Ġat Ġleast Ġone Ġinstant iation Ġof Ġthe Ġwindow . Ġ$ G $ Ġis Ġdirected Ġand Ġweighted : Ġedge Ġdirections Ġand Ġweights Ġrespectively Ġcapture Ġtext Ġflow Ġand Ġco - occ urrence Ġcounts . </s> $ G $ Ġis Ġa Ġcompact Ġrepresentation Ġof Ġits Ġdocument . ĠIn Ġ$ G $, Ġimmediate Ġneighbors Ġare Ġconsecutive Ġwords Ġin Ġthe Ġsame Ġsentence . ĠThat Ġis , Ġpaths Ġof Ġlength Ġ2 Ġcorrespond Ġto Ġbig rams . ĠPath s Ġof Ġlength Ġmore Ġthan Ġ2 Ġcan Ġcorrespond Ġeither Ġto Ġtraditional Ġ$ n $ - gram s Ġor Ġto Ġrelaxed Ġ$ n $ - gram s , Ġthat Ġis , Ġwords Ġthat Ġnever Ġappear Ġin Ġthe Ġsame Ġsentence Ġbut Ġco - occ ur Ġwith Ġthe Ġsame Ġword ( s ). ĠSuch Ġnodes Ġare Ġlinked Ġthrough Ġcommon Ġneighbors . </s> Master Ġnode . ĠInspired Ġby ĠB IB REF 3 , Ġour Ġ$ G $ Ġalso Ġincludes Ġa Ġspecial Ġdocument Ġnode , Ġlinked Ġto Ġall Ġother Ġnodes Ġvia Ġunit Ġweight Ġbi - direction al Ġedges . ĠIn Ġwhat Ġfollows , Ġlet Ġus Ġdenote Ġby Ġ$ n $ Ġthe Ġnumber Ġof Ġnodes Ġin Ġ$ G $, Ġincluding Ġthe Ġmaster Ġnode . </s> Message ĠPassing ĠAttention Ġnetwork Ġfor ĠDocument Ġunderstanding Ġ( MP AD ) Ġ: :: ĠMessage Ġpassing </s> We Ġformulate Ġour ĠA GG REG ATE Ġfunction Ġas : </s> where Ġ$\ math bf Ġ{ H } ^ t Ġ\ in Ġ\ math bb Ġ{ R } ^{ n Ġ\ times Ġd } $ Ġcontains Ġnode Ġfeatures Ġ($ d $ Ġis Ġa Ġhyper param eter ), Ġand Ġ$\ math bf Ġ{ A } Ġ\ in Ġ\ math bb Ġ{ R } ^{ n Ġ\ times Ġn } $ Ġis Ġthe Ġadj ac ency Ġmatrix Ġof Ġ$ G $. ĠSince Ġ$ G $ Ġis Ġdirected , Ġ$\ math bf Ġ{ A } $ Ġis Ġasymm etric . ĠAlso , Ġ$\ math bf Ġ{ A } $ Ġhas Ġzero Ġdiagonal Ġas Ġwe Ġchoose Ġnot Ġto Ġconsider Ġthe Ġfeature Ġof Ġthe Ġnode Ġitself , Ġonly Ġthat Ġof Ġits Ġincoming Ġneighbors , Ġwhen Ġupdating Ġits Ġrepresentation . ĠSince Ġ$ G $ Ġis Ġweighted , Ġthe Ġ$ i ^{ th } $ Ġrow Ġof Ġ$ A $ Ġcontains Ġthe Ġweights Ġof Ġthe Ġedges Ġincoming Ġon Ġnode Ġ$ v _ i $. Ġ$\ math bf Ġ{ D } Ġ\ in Ġ\ math bb Ġ{ R } ^{ n Ġ\ times Ġn } $ Ġis Ġthe Ġdiagonal Ġin - degree Ġmatrix Ġof Ġ$ G $. ĠML P Ġdenotes Ġa Ġmulti - layer Ġpercept ron , Ġand Ġ$\ math bf Ġ{ M } ^{ t + 1 } Ġ\ in Ġ\ math bb Ġ{ R } ^{ n Ġ\ times Ġd } $ Ġis Ġthe Ġmessage Ġmatrix . </s> The Ġuse Ġof Ġa ĠML P Ġwas Ġmotivated Ġby Ġthe Ġobservation Ġthat Ġfor Ġgraph Ġclassification , ĠMP Ġneural Ġnets Ġwith Ġ1 - layer Ġpercept rons Ġare Ġinferior Ġto Ġtheir ĠML P Ġcounterparts ĠB IB REF 14 . ĠIndeed , Ġ1 - layer Ġpercept rons Ġare Ġnot Ġuniversal Ġapprox im ators Ġof Ġmult is et Ġfunctions . ĠNote Ġthat Ġlike Ġin ĠB IB REF 14 , Ġwe Ġuse Ġa Ġdifferent ĠML P Ġat Ġeach Ġlayer . </s> Ren ormal ization . ĠThe Ġrows Ġof Ġ$\ math bf Ġ{ D } ^{ - 1 }\ math bf Ġ{ A } $ Ġsum Ġto Ġ1 . ĠThis Ġis Ġequivalent Ġto Ġthe Ġren ormal ization Ġtrick Ġof ĠB IB REF 9 , Ġbut Ġusing Ġonly Ġthe Ġin - deg rees . ĠThat Ġis , Ġinstead Ġof Ġcomputing Ġa Ġweighted Ġsum Ġof Ġthe Ġincoming Ġneighbors ' Ġfeature Ġvectors , Ġwe Ġcompute Ġa Ġweighted Ġaverage Ġof Ġthem . ĠThe Ġcoefficients Ġare Ġproportional Ġto Ġthe Ġstrength Ġof Ġco - occ urrence Ġbetween Ġwords . ĠOne Ġshould Ġnote Ġthat Ġby Ġaveraging , Ġwe Ġlose Ġthe Ġability Ġto Ġdistinguish Ġbetween Ġdifferent Ġneighborhood Ġstructures Ġin Ġsome Ġspecial Ġcases , Ġthat Ġis , Ġwe Ġlose Ġinject ivity . ĠSuch Ġcases Ġinclude Ġneighborhoods Ġin Ġwhich Ġall Ġnodes Ġhave Ġthe Ġsame Ġrepresentations , Ġand Ġneighborhoods Ġof Ġdifferent Ġsizes Ġcontaining Ġvarious Ġrepresentations Ġin Ġequal Ġproportions ĠB IB REF 14 . ĠAs Ġsuggested Ġby Ġthe Ġresults Ġof Ġan Ġab lation Ġexperiment , Ġaveraging Ġis Ġbetter Ġthan Ġsum ming Ġin Ġour Ġapplication Ġ( see Ġsubsection ĠSEC REF 30 ). ĠNote Ġthat Ġinstead Ġof Ġsimply Ġsum ming / aver aging , Ġwe Ġalso Ġtried Ġusing ĠG AT - like Ġattention ĠB IB REF 11 Ġin Ġearly Ġexperiments , Ġwithout Ġobtaining Ġbetter Ġresults . </s> As Ġfar Ġas Ġour ĠCOM B INE Ġfunction , Ġwe Ġuse Ġthe ĠG ated ĠRec urrent ĠUnit ĠB IB REF 20 , ĠB IB REF 21 : </s> O mitting Ġbiases Ġfor Ġread ability , Ġwe Ġhave : </s> where Ġthe Ġ$\ math bf Ġ{ W } $ Ġand Ġ$\ math bf Ġ{ U } $ Ġmat rices Ġare Ġtrain able Ġweight Ġmat rices Ġnot Ġshared Ġacross Ġtime Ġsteps , Ġ$\ s igma Ġ( \ math bf Ġ{ x }) Ġ= Ġ1 /( 1 + \ exp Ġ(- \ math bf Ġ{ x } )) $ Ġis Ġthe Ġs igm oid Ġfunction , Ġand Ġ$\ math bf Ġ{ R } $ Ġand Ġ$\ math bf Ġ{ Z } $ Ġare Ġthe Ġparameters Ġof Ġthe Ġreset Ġand Ġupdate Ġgates . ĠThe Ġreset Ġgate Ġcontrols Ġthe Ġamount Ġof Ġinformation Ġfrom Ġthe Ġprevious Ġtime Ġstep Ġ( in Ġ$\ math bf Ġ{ H } ^ t $ ) Ġthat Ġshould Ġpropagate Ġto Ġthe Ġcandidate Ġrepresentations , Ġ$\ t ilde {\ math bf Ġ{ H }} ^{ t + 1 } $. ĠThe Ġnew Ġrepresentations Ġ$\ math bf Ġ{ H } ^{ t + 1 } $ Ġare Ġfinally Ġobtained Ġby Ġlin early Ġinterpol ating Ġbetween Ġthe Ġprevious Ġand Ġthe Ġcandidate Ġones , Ġusing Ġthe Ġcoefficients Ġreturned Ġby Ġthe Ġupdate Ġgate . </s> Inter pret ation . ĠUp dating Ġnode Ġrepresentations Ġthrough Ġa ĠGR U Ġshould Ġin Ġprinciple Ġallow Ġnodes Ġto Ġencode Ġa Ġcombination Ġof Ġlocal Ġand Ġglobal Ġsignals Ġ( low Ġand Ġhigh Ġvalues Ġof Ġ$ t $, Ġresp .), Ġby Ġallowing Ġthem Ġto Ġremember Ġabout Ġpast Ġiterations . ĠIn Ġaddition , Ġwe Ġalso Ġexplicitly Ġconsider Ġnode Ġrepresentations Ġat Ġall Ġiterations Ġwhen Ġreading Ġout Ġ( see ĠE q . ĠDIS PLAY _ FORM 18 ). </s> Message ĠPassing ĠAttention Ġnetwork Ġfor ĠDocument Ġunderstanding Ġ( MP AD ) Ġ: :: ĠRead out </s> After Ġpassing Ġmessages Ġand Ġperforming Ġupdates Ġfor Ġ$ T $ Ġiterations , Ġwe Ġobtain Ġa Ġmatrix Ġ$\ math bf Ġ{ H } ^ T Ġ\ in Ġ\ math bb Ġ{ R } ^{ n Ġ\ times Ġd } $ Ġcontaining Ġthe Ġfinal Ġvertex Ġrepresentations . ĠLet Ġ$\ hat { G } $ Ġbe Ġgraph Ġ$ G $ Ġwithout Ġthe Ġspecial Ġdocument Ġnode , Ġand Ġmatrix Ġ$\ math bf Ġ{\ hat { H }} ^ T Ġ\ in Ġ\ math bb Ġ{ R } ^{ ( n - 1 ) Ġ\ times Ġd } $ Ġbe Ġthe Ġcorresponding Ġrepresentation Ġmatrix Ġ( i . e ., Ġ$\ math bf Ġ{ H } ^ T $ Ġwithout Ġthe Ġrow Ġof Ġthe Ġdocument Ġnode ). </s> We Ġuse Ġas Ġour ĠREAD OUT Ġfunction Ġthe Ġconc aten ation Ġof Ġself - att ention Ġapplied Ġto Ġ$\ math bf Ġ{\ hat { H }} ^ T $ Ġwith Ġthe Ġfinal Ġdocument Ġnode Ġrepresentation . ĠMore Ġprecisely , Ġwe Ġapply Ġa Ġglobal Ġself - att ention Ġmechanism ĠB IB REF 22 Ġto Ġthe Ġrows Ġof Ġ$\ math bf Ġ{\ hat { H }} ^ T $. ĠAs Ġshown Ġin ĠE q . ĠDIS PLAY _ FORM 17 , Ġ$\ math bf Ġ{\ hat { H }} ^ T $ Ġis Ġfirst Ġpassed Ġto Ġa Ġdense Ġlayer Ġparameter ized Ġby Ġmatrix Ġ$\ math bf Ġ{ W } _ A ^ T Ġ\ in Ġ\ math bb Ġ{ R } ^{ d Ġ\ times Ġd } $. ĠAn Ġalignment Ġvector Ġ$\ math bf Ġ{ a } $ Ġis Ġthen Ġderived Ġby Ġcomparing , Ġvia Ġdot Ġproducts , Ġthe Ġrows Ġof Ġthe Ġoutput Ġof Ġthe Ġdense Ġlayer Ġ$\ math bf Ġ{ Y } ^ T Ġ\ in Ġ\ math bb Ġ{ R } ^{ ( n - 1 ) Ġ\ times Ġd } $ Ġwith Ġa Ġtrain able Ġvector Ġ$\ math bf Ġ{ v } ^ T Ġ\ in Ġ\ math bb Ġ{ R } ^ d $ Ġ( initialized Ġrandomly ) Ġand Ġnormal izing Ġwith Ġa Ġsoft max . ĠThe Ġnormalized Ġalignment Ġcoefficients Ġare Ġfinally Ġused Ġto Ġcompute Ġthe Ġattention al Ġvector Ġ$\ math bf Ġ{ u } ^ T Ġ\ in Ġ\ math bb Ġ{ R } ^ d $ Ġas Ġa Ġweighted Ġsum Ġof Ġthe Ġfinal Ġrepresentations Ġ$\ math bf Ġ{\ hat { H }} ^ T $. </s> Note Ġthat Ġwe Ġtried Ġwith Ġmultiple Ġcontext Ġvectors , Ġi . e ., Ġwith Ġa Ġmatrix Ġ$\ math bf Ġ{ V } ^ T $ Ġinstead Ġof Ġa Ġvector Ġ$\ math bf Ġ{ v } ^ T $, Ġlike Ġin ĠB IB REF 22 , Ġbut Ġresults Ġwere Ġnot Ġconvincing , Ġeven Ġwhen Ġadding Ġa Ġregular ization Ġterm Ġto Ġthe Ġloss Ġto Ġfavor Ġdiversity Ġamong Ġthe Ġrows Ġof Ġ$\ math bf Ġ{ V } ^ T $. </s> Master Ġnode Ġskip Ġconnection . Ġ$\ math bf Ġ{ h } _ G ^ T Ġ\ in Ġ\ math bb Ġ{ R } ^{ 2 d } $ Ġis Ġobtained Ġby Ġconc aten ating Ġ$\ math bf Ġ{ u } ^ T $ Ġand Ġthe Ġfinal Ġmaster Ġnode Ġrepresentation . ĠThat Ġis , Ġthe Ġmaster Ġnode Ġvector Ġbypass es Ġthe Ġattention Ġmechanism . ĠThis Ġis Ġequivalent Ġto Ġa Ġskip Ġor Ġshortcut Ġconnection ĠB IB REF 23 . ĠThe Ġreason Ġbehind Ġthis Ġchoice Ġis Ġthat Ġwe Ġexpect Ġthe Ġspecial Ġdocument Ġnode Ġto Ġlearn Ġa Ġhigh - level Ġsummary Ġabout Ġthe Ġdocument , Ġsuch Ġas Ġits Ġsize , Ġvocabulary , Ġetc . Ġ( more Ġdetails Ġare Ġgiven Ġin Ġsubsection ĠSEC REF 30 ). ĠTherefore , Ġby Ġmaking Ġthe Ġmaster Ġnode Ġbypass Ġthe Ġattention Ġlayer , Ġwe Ġdirectly Ġinject Ġglobal Ġinformation Ġabout Ġthe Ġdocument Ġinto Ġits Ġfinal Ġrepresentation . </s> Multi - read out . ĠB IB REF 14 , Ġinspired Ġby ĠJump ing ĠKnowledge ĠNetworks ĠB IB REF 12 , Ġrecommend Ġto Ġnot Ġonly Ġuse Ġthe Ġfinal Ġrepresentations Ġwhen Ġperforming Ġread out , Ġbut Ġalso Ġthat Ġof Ġthe Ġearlier Ġsteps . ĠIndeed , Ġas Ġone Ġiter ates , Ġnode Ġfeatures Ġcapture Ġmore Ġand Ġmore Ġglobal Ġinformation . ĠHowever , Ġretaining Ġmore Ġlocal , Ġintermediary Ġinformation Ġmight Ġbe Ġuseful Ġtoo . ĠThus , Ġinstead Ġof Ġapplying Ġthe Ġread out Ġfunction Ġonly Ġto Ġ$ t = T $, Ġwe Ġapply Ġit Ġto Ġall Ġtime Ġsteps Ġand Ġconc aten ate Ġthe Ġresults , Ġfinally Ġobtaining Ġ$\ math bf Ġ{ h } _ G Ġ\ in Ġ\ math bb Ġ{ R } ^{ T Ġ\ times Ġ2 d } $ Ġ: </s> In Ġeffect , Ġwith Ġthis Ġmodification , Ġwe Ġtake Ġinto Ġaccount Ġfeatures Ġbased Ġon Ġinformation Ġaggreg ated Ġfrom Ġsubt rees Ġof Ġdifferent Ġheights Ġ( from Ġ1 Ġto Ġ$ T $ ), Ġcorresponding Ġto Ġlocal Ġand Ġglobal Ġfeatures . </s> Message ĠPassing ĠAttention Ġnetwork Ġfor ĠDocument Ġunderstanding Ġ( MP AD ) Ġ: :: ĠHier arch ical Ġvariants Ġof ĠMP AD </s> Through Ġthe Ġsuccessive ĠMP Ġiterations , Ġit Ġcould Ġbe Ġargued Ġthat ĠMP AD Ġimplicitly Ġcaptures Ġsome Ġsoft Ġnotion Ġof Ġthe Ġhierarchical Ġstructure Ġof Ġdocuments Ġ( words Ġ$\ right arrow Ġ$ Ġbig rams Ġ$\ right arrow Ġ$ Ġcompositions Ġof Ġbig rams , Ġetc .). ĠHowever , Ġit Ġmight Ġbe Ġbeneficial Ġto Ġexplicitly Ġcapture Ġdocument Ġhierarchy . ĠHier arch ical Ġarchitectures Ġhave Ġbrought Ġsignificant Ġimprovements Ġto Ġmany ĠN LP Ġtasks , Ġsuch Ġas Ġlanguage Ġmodeling Ġand Ġgeneration ĠB IB REF 24 , ĠB IB REF 25 , Ġsentiment Ġand Ġtopic Ġclassification ĠB IB REF 26 , ĠB IB REF 27 , Ġand Ġspoken Ġlanguage Ġunderstanding ĠB IB REF 28 , ĠB IB REF 29 . ĠInspired Ġby Ġthis Ġline Ġof Ġresearch , Ġwe Ġpropose Ġseveral Ġhierarchical Ġvariants Ġof ĠMP AD , Ġdetailed Ġin Ġwhat Ġfollows . ĠIn Ġall Ġof Ġthem , Ġwe Ġrepresent Ġeach Ġsentence Ġin Ġthe Ġdocument Ġas Ġa Ġword Ġco - occ urrence Ġnetwork , Ġand Ġobtain Ġan Ġembed ding Ġfor Ġit Ġby Ġapplying ĠMP AD Ġas Ġpreviously Ġdescribed . </s> MP AD - sent ence - att . ĠHere , Ġthe Ġsentence Ġembed d ings Ġare Ġsimply Ġcombined Ġthrough Ġself - att ention . </s> MP AD - cl ique . ĠIn Ġthis Ġvariant , Ġwe Ġbuild Ġa Ġcomplete Ġgraph Ġwhere Ġeach Ġnode Ġrepresents Ġa Ġsentence . ĠWe Ġthen Ġfeed Ġthat Ġgraph Ġto ĠMP AD , Ġwhere Ġthe Ġfeature Ġvectors Ġof Ġthe Ġnodes Ġare Ġinitialized Ġwith Ġthe Ġsentence Ġembed d ings Ġpreviously Ġobtained . </s> MP AD - path . ĠThis Ġvariant Ġis Ġsimilar Ġto Ġthe Ġcl ique Ġone , Ġexcept Ġthat Ġinstead Ġof Ġa Ġcomplete Ġgraph , Ġwe Ġbuild Ġa Ġpath Ġaccording Ġto Ġthe Ġnatural Ġflow Ġof Ġthe Ġtext . ĠThat Ġis , Ġtwo Ġnodes Ġare Ġlinked Ġby Ġa Ġdirected Ġedge Ġif Ġthe Ġtwo Ġsentences Ġthey Ġrepresent Ġfollow Ġeach Ġother Ġin Ġthe Ġdocument . </s> Exper iments Ġ: :: ĠDat as ets </s> We Ġevaluate Ġthe Ġquality Ġof Ġthe Ġdocument Ġembed d ings Ġlearned Ġby ĠMP AD Ġon Ġ10 Ġdocument Ġclassification Ġdatasets , Ġcovering Ġthe Ġtopic Ġidentification , Ġcoarse Ġand Ġfine Ġsentiment Ġanalysis Ġand Ġopinion Ġmining , Ġand Ġsubject ivity Ġdetection Ġtasks . ĠWe Ġbriefly Ġintroduce Ġthe Ġdatasets Ġnext . ĠTheir Ġstatistics Ġare Ġreported Ġin ĠTable ĠT AB REF 21 . </s> ( 1 ) ĠReuters . ĠThis Ġdataset Ġcontains Ġstories Ġcollected Ġfrom Ġthe ĠReuters Ġnews Ġagency Ġin Ġ1987 . ĠFollowing Ġcommon Ġpractice , Ġwe Ġused Ġthe ĠMod A pt e Ġsplit Ġand Ġconsidered Ġonly Ġthe Ġ10 Ġclasses Ġwith Ġthe Ġhighest Ġnumber Ġof Ġpositive Ġtraining Ġexamples . ĠWe Ġalso Ġremoved Ġdocuments Ġbelonging Ġto Ġmore Ġthan Ġone Ġclass Ġand Ġthen Ġclasses Ġleft Ġwith Ġno Ġdocument Ġ( 2 Ġclasses ). </s> ( 2 ) ĠBBC Sport ĠB IB REF 30 Ġcontains Ġdocuments Ġfrom Ġthe ĠBBC ĠSport Ġwebsite Ġcorresponding Ġto Ġ2004 - 2005 Ġsports Ġnews Ġarticles . </s> ( 3 ) ĠPol arity ĠB IB REF 31 Ġfeatures Ġpositive Ġand Ġnegative Ġlabeled Ġsnippets Ġfrom ĠR otten ĠTom atoes . </s> ( 4 ) ĠSubject ivity ĠB IB REF 32 Ġcontains Ġmovie Ġreview Ġsnippets Ġfrom ĠR otten ĠTom atoes Ġ( subject ive Ġsentences ), Ġand ĠInternet ĠMovie ĠDatabase Ġplot Ġsumm aries Ġ( object ive Ġsentences ). </s> ( 5 ) ĠMP Q A ĠB IB REF 33 Ġis Ġmade Ġof Ġpositive Ġand Ġnegative Ġphrases , Ġannot ated Ġas Ġpart Ġof Ġthe Ġsummer Ġ2002 ĠNR RC ĠWorkshop Ġon ĠMulti - Pers pect ive ĠQuestion ĠAn sw ering . </s> ( 6 ) ĠIM DB ĠB IB REF 34 Ġis Ġa Ġcollection Ġof Ġhighly Ġpolarized Ġmovie Ġreviews Ġfrom ĠIM DB Ġ( positive Ġand Ġnegative ). ĠThere Ġare Ġat Ġmost Ġ30 Ġreviews Ġfor Ġeach Ġmovie . </s> ( 7 ) ĠT REC ĠB IB REF 35 Ġconsists Ġof Ġquestions Ġthat Ġare Ġclassified Ġinto Ġ6 Ġdifferent Ġcategories . </s> ( 8 ) ĠS ST - 1 ĠB IB REF 36 Ġcontains Ġthe Ġsame Ġsnippets Ġas ĠPol arity . ĠThe Ġauthors Ġused Ġthe ĠStanford ĠPars er Ġto Ġparse Ġthe Ġsnippets Ġand Ġsplit Ġthem Ġinto Ġmultiple Ġsentences . ĠThey Ġthen Ġused ĠAmazon ĠMechanical ĠTurk Ġto Ġannot ate Ġthe Ġresulting Ġphrases Ġaccording Ġto Ġtheir Ġpol arity Ġ( very Ġnegative , Ġnegative , Ġneutral , Ġpositive , Ġvery Ġpositive ). </s> ( 9 ) ĠS ST - 2 ĠB IB REF 36 Ġis Ġthe Ġsame Ġas ĠS ST - 1 Ġbut Ġwith Ġneutral Ġreviews Ġremoved Ġand Ġsnippets Ġclassified Ġas Ġpositive Ġor Ġnegative . </s> ( 10 ) ĠYelp 2013 ĠB IB REF 26 Ġfeatures Ġreviews Ġobtained Ġfrom Ġthe Ġ2013 ĠYelp ĠDat as et ĠChallenge . </s> Exper iments Ġ: :: ĠBas elines </s> We Ġevaluate ĠMP AD Ġagainst Ġmultiple Ġstate - of - the - art Ġbaseline Ġmodels , Ġincluding Ġhierarchical Ġones , Ġto Ġenable Ġfair Ġcomparison Ġwith Ġthe Ġhierarchical ĠMP AD Ġvariants . </s> doc 2 vec ĠB IB REF 37 . ĠDoc 2 vec Ġ( or Ġparagraph Ġvector ) Ġis Ġan Ġextension Ġof Ġword 2 vec Ġthat Ġlearns Ġvectors Ġfor Ġdocuments Ġin Ġa Ġfully Ġun super vised Ġmanner . ĠDocument Ġembed d ings Ġare Ġthen Ġfed Ġto Ġa Ġlog istic Ġregression Ġclass ifier . </s> CNN ĠB IB REF 38 . ĠThe Ġconv olution al Ġneural Ġnetwork Ġarchitecture , Ġwell - known Ġin Ġcomputer Ġvision , Ġis Ġapplied Ġto Ġtext . ĠThere Ġis Ġone Ġspatial Ġdimension Ġand Ġthe Ġword Ġembed d ings Ġare Ġused Ġas Ġchannels Ġ( depth Ġdimensions ). </s> D AN ĠB IB REF 39 . ĠThe ĠDeep ĠA ver aging ĠNetwork Ġpasses Ġthe Ġun weight ed Ġaverage Ġof Ġthe Ġembed d ings Ġof Ġthe Ġinput Ġwords Ġthrough Ġmultiple Ġdense Ġlayers Ġand Ġa Ġfinal Ġsoft max . </s> Tree - L ST M ĠB IB REF 40 Ġis Ġa Ġgeneral ization Ġof Ġthe Ġstandard ĠL ST M Ġarchitecture Ġto Ġconstituency Ġand Ġdependency Ġparse Ġtrees . </s> DR NN ĠB IB REF 41 . ĠRec ursive Ġneural Ġnetworks Ġare Ġstacked Ġand Ġapplied Ġto Ġparse Ġtrees . </s> L ST MN ĠB IB REF 42 Ġis Ġan Ġextension Ġof Ġthe ĠL ST M Ġmodel Ġwhere Ġthe Ġmemory Ġcell Ġis Ġreplaced Ġby Ġa Ġmemory Ġnetwork Ġwhich Ġstores Ġword Ġrepresentations . </s> C - L ST M ĠB IB REF 43 Ġcombines Ġconv olution al Ġand Ġrecurrent Ġneural Ġnetworks . ĠThe Ġregion Ġembed d ings Ġprovided Ġby Ġa ĠCNN Ġare Ġfed Ġto Ġa ĠL ST M . </s> SP G K ĠB IB REF 44 Ġalso Ġmodels Ġdocuments Ġas Ġword Ġco - occ urrence Ġnetworks . ĠIt Ġcomp utes Ġa Ġgraph Ġkernel Ġthat Ġcompares Ġshortest Ġpaths Ġextracted Ġfrom Ġthe Ġword Ġco - occ urrence Ġnetworks Ġand Ġthen Ġuses Ġa ĠS VM Ġto Ġcategor ize Ġdocuments . </s> W MD ĠB IB REF 45 Ġis Ġan Ġapplication Ġof Ġthe Ġwell - known ĠEarth ĠM over 's ĠDistance Ġto Ġtext . ĠA Ġk - ne arest Ġneighbor Ġclass ifier Ġis Ġused . </s> S - W MD ĠB IB REF 46 Ġis Ġa Ġsupervised Ġextension Ġof Ġthe ĠWord ĠM over 's ĠDistance . </s> Sem antic - CNN ĠB IB REF 47 . ĠHere , Ġa ĠCNN Ġis Ġapplied Ġto Ġsemantic Ġunits Ġobtained Ġby Ġclust ering Ġwords Ġin Ġthe Ġembed ding Ġspace . </s> L ST M - GR NN ĠB IB REF 26 Ġis Ġa Ġhierarchical Ġmodel Ġwhere Ġsentence Ġembed d ings Ġare Ġobtained Ġwith Ġa ĠCNN Ġand Ġa ĠGR U - R NN Ġis Ġfed Ġthe Ġsentence Ġrepresentations Ġto Ġobtain Ġa Ġdocument Ġvector . </s> H N - ATT ĠB IB REF 27 Ġis Ġanother Ġhierarchical Ġmodel , Ġwhere Ġthe Ġsame Ġenc oder Ġarchitecture Ġ( a Ġbid irection al ĠGR U - R NN ) Ġis Ġused Ġfor Ġboth Ġsentences Ġand Ġdocuments , Ġwith Ġdifferent Ġparameters . ĠA Ġself - att ention Ġmechanism Ġis Ġapplied Ġto Ġthe ĠR NN Ġannotations Ġat Ġeach Ġlevel . </s> Exper iments Ġ: :: ĠModel Ġconfiguration Ġand Ġtraining </s> We Ġpre process Ġall Ġdatasets Ġusing Ġthe Ġcode Ġof ĠB IB REF 38 . ĠOn ĠYelp 2013 , Ġwe Ġalso Ġreplace Ġall Ġtokens Ġappearing Ġstrictly Ġless Ġthan Ġ6 Ġtimes Ġwith Ġa Ġspecial ĠUN K Ġtoken , Ġlike Ġin ĠB IB REF 27 . ĠWe Ġthen Ġbuild Ġa Ġdirected Ġword Ġco - occ urrence Ġnetwork Ġfrom Ġeach Ġdocument , Ġwith Ġa Ġwindow Ġof Ġsize Ġ2 . </s> We Ġuse Ġtwo ĠMP Ġiterations Ġ($ T $ = 2 ) Ġfor Ġthe Ġbasic ĠMP AD , Ġand Ġtwo ĠMP Ġiterations Ġat Ġeach Ġlevel , Ġfor Ġthe Ġhierarchical Ġvariants . ĠWe Ġset Ġ$ d $ Ġto Ġ64 , Ġexcept Ġon ĠIM DB Ġand ĠYelp Ġon Ġwhich Ġ$ d = 128 $, Ġand Ġuse Ġa Ġtwo - layer ĠML P . ĠThe Ġfinal Ġgraph Ġrepresentations Ġare Ġpassed Ġthrough Ġa Ġsoft max Ġfor Ġclassification . ĠWe Ġtrain ĠMP AD Ġin Ġan Ġend - to - end Ġfashion Ġby Ġminimizing Ġthe Ġcross - ent ropy Ġloss Ġfunction Ġwith Ġthe ĠAdam Ġoptim izer ĠB IB REF 48 Ġand Ġan Ġinitial Ġlearning Ġrate Ġof Ġ0 . 001 . </s> To Ġregulate Ġpotential Ġdifferences Ġin Ġmagnitude , Ġwe Ġapply Ġbatch Ġnormal ization Ġafter Ġconc aten ating Ġthe Ġfeature Ġvector Ġof Ġthe Ġmaster Ġnode Ġwith Ġthe Ġself - att ention al Ġvector , Ġthat Ġis , Ġafter Ġthe Ġskip Ġconnection Ġ( see Ġsubsection ĠSEC REF 16 ). ĠTo Ġprevent Ġover fitting , Ġwe Ġuse Ġdrop out ĠB IB REF 49 Ġwith Ġa Ġrate Ġof Ġ0 . 5 . ĠWe Ġselect Ġthe Ġbest Ġepoch , Ġcapped Ġat Ġ200 , Ġbased Ġon Ġthe Ġvalidation Ġaccuracy . ĠWhen Ġcross - valid ation Ġis Ġused Ġ( see Ġ3 rd Ġcolumn Ġof ĠTable ĠT AB REF 21 ), Ġwe Ġconstruct Ġa Ġvalidation Ġset Ġby Ġrandomly Ġsampling Ġ10 % Ġof Ġthe Ġtraining Ġset Ġof Ġeach Ġfold . </s> On Ġall Ġdatasets Ġexcept ĠYelp 2013 , Ġwe Ġuse Ġthe Ġpublicly Ġavailable Ġ300 - dimensional Ġpre - trained ĠGoogle ĠNews Ġvectors Ġ($ D $ = 300 ) ĠB IB REF 50 Ġto Ġinitialize Ġthe Ġnode Ġrepresentations Ġ$\ math bf Ġ{ H } ^ 0 $. ĠOn ĠYelp 2013 , Ġwe Ġfollow ĠB IB REF 27 Ġand Ġlearn Ġour Ġown Ġword Ġvectors Ġfrom Ġthe Ġtraining Ġand Ġvalidation Ġsets Ġwith Ġthe Ġg ens im Ġimplementation Ġof Ġword 2 vec ĠB IB REF 51 . </s> MP AD Ġwas Ġimplemented Ġin ĠPython Ġ3 . 6 Ġusing Ġthe ĠPy Tor ch Ġlibrary ĠB IB REF 52 . ĠAll Ġexperiments Ġwere Ġrun Ġon Ġa Ġsingle Ġmachine Ġconsisting Ġof Ġa Ġ3 . 4 ĠGHz ĠIntel ĠCore Ġi 7 ĠCPU Ġwith Ġ16 ĠGB Ġof ĠRAM Ġand Ġan ĠNV idia ĠGeForce ĠTitan ĠX p ĠGPU . </s> Results Ġand Ġabl ations Ġ: :: ĠResults </s> Exper imental Ġresults Ġare Ġshown Ġin ĠTable ĠT AB REF 28 . ĠFor Ġthe Ġbas elines , Ġthe Ġbest Ġscores Ġreported Ġin Ġeach Ġoriginal Ġpaper Ġare Ġshown . ĠMP AD Ġreaches Ġbest Ġperformance Ġon Ġ7 Ġout Ġof Ġ10 Ġdatasets , Ġand Ġis Ġclose Ġsecond Ġelsewhere . ĠMoreover , Ġthe Ġ7 Ġdatasets Ġon Ġwhich ĠMP AD Ġranks Ġfirst Ġwidely Ġdiffer Ġin Ġtraining Ġset Ġsize , Ġnumber Ġof Ġcategories , Ġand Ġprediction Ġtask Ġ( topic , Ġsentiment , Ġsubject ivity ), Ġwhich Ġindicates Ġthat ĠMP AD Ġcan Ġperform Ġwell Ġin Ġdifferent Ġsettings . </s> MP AD Ġvs . Ġhierarchical Ġvariants . ĠOn Ġ9 Ġdatasets Ġout Ġof Ġ10 , Ġone Ġor Ġmore Ġof Ġthe Ġhierarchical Ġvariants Ġoutper form Ġthe Ġvanilla ĠMP AD Ġarchitecture , Ġhighlighting Ġthe Ġbenefit Ġof Ġexplicitly Ġmodeling Ġthe Ġhierarchical Ġnature Ġof Ġdocuments . </s> However , Ġon ĠSubject ivity , Ġstandard ĠMP AD Ġoutper forms Ġall Ġhierarchical Ġvariants . ĠOn ĠT REC , Ġit Ġreaches Ġthe Ġsame Ġaccuracy . ĠWe Ġhypothes ize Ġthat Ġin Ġsome Ġcases , Ġusing Ġa Ġdifferent Ġgraph Ġto Ġseparately Ġencode Ġeach Ġsentence Ġmight Ġbe Ġworse Ġthan Ġusing Ġone Ġsingle Ġgraph Ġto Ġdirectly Ġencode Ġthe Ġdocument . ĠIndeed , Ġin Ġthe Ġsingle Ġdocument Ġgraph , Ġsome Ġwords Ġthat Ġnever Ġappear Ġin Ġthe Ġsame Ġsentence Ġcan Ġbe Ġconnected Ġthrough Ġcommon Ġneighbors , Ġas Ġwas Ġexplained Ġin Ġsubsection ĠSEC REF 7 . ĠSo , Ġthis Ġway , Ġsome Ġnotion Ġof Ġcross - sent ence Ġcontext Ġis Ġcaptured Ġwhile Ġlearning Ġrepresentations Ġof Ġwords , Ġbig rams , Ġetc . Ġat Ġeach ĠMP Ġiteration . ĠThis Ġcreates Ġbetter Ġinformed Ġrepresentations , Ġresulting Ġin Ġa Ġbetter Ġdocument Ġembed ding . ĠWith Ġthe Ġhierarchical Ġvariants , Ġon Ġthe Ġother Ġhand , Ġeach Ġsentence Ġvector Ġis Ġproduced Ġin Ġisolation , Ġwithout Ġany Ġcontextual Ġinformation Ġabout Ġthe Ġother Ġsentences Ġin Ġthe Ġdocument . ĠTherefore , Ġthe Ġfinal Ġsentence Ġembed d ings Ġmight Ġbe Ġof Ġlower Ġquality , Ġand Ġas Ġa Ġgroup Ġmight Ġalso Ġcontain Ġredundant / repe ated Ġinformation . ĠWhen Ġthe Ġsentence Ġvectors Ġare Ġfinally Ġcombined Ġinto Ġa Ġdocument Ġrepresentation , Ġit Ġis Ġtoo Ġlate Ġto Ġtake Ġcontext Ġinto Ġaccount . </s> Results Ġand Ġabl ations Ġ: :: ĠAb lation Ġstudies </s> To Ġunderstand Ġthe Ġimpact Ġof Ġsome Ġhyper param eters Ġon Ġperformance , Ġwe Ġconducted Ġadditional Ġexperiments Ġon Ġthe ĠReuters , ĠPol arity , Ġand ĠIM DB Ġdatasets , Ġwith Ġthe Ġnon - h ier arch ical Ġversion Ġof ĠMP AD . ĠResults Ġare Ġshown Ġin ĠTable ĠT AB REF 29 . </s> Number Ġof ĠMP Ġiterations . ĠFirst , Ġwe Ġvaried Ġthe Ġnumber Ġof Ġmessage Ġpassing Ġiterations Ġfrom Ġ1 Ġto Ġ4 . ĠWe Ġcan Ġclearly Ġsee Ġin ĠTable ĠT AB REF 29 Ġthat Ġhaving Ġmore Ġiterations Ġimproves Ġperformance . ĠWe Ġattribute Ġthis Ġto Ġthe Ġfact Ġthat Ġwe Ġare Ġreading Ġout Ġat Ġeach Ġiteration Ġfrom Ġ1 Ġto Ġ$ T $ Ġ( see ĠE q . ĠDIS PLAY _ FORM 18 ), Ġwhich Ġenables Ġthe Ġfinal Ġgraph Ġrepresentation Ġto Ġencode Ġa Ġmixture Ġof Ġlow - level Ġand Ġhigh - level Ġfeatures . ĠIndeed , Ġin Ġinitial Ġexperiments Ġinvolving Ġread out Ġat Ġ$ t $ =$ T $ Ġonly , Ġsetting Ġ$ T \ ge Ġ2 $ Ġwas Ġalways Ġdecreasing Ġperformance , Ġdespite Ġthe ĠGR U - based Ġupdates Ġ( E q . ĠDIS PLAY _ FORM 14 ). ĠThese Ġresults Ġwere Ġconsistent Ġwith Ġthat Ġof ĠB IB REF 53 Ġand ĠB IB REF 9 , Ġwho Ġboth Ġare Ġreading Ġout Ġonly Ġat Ġ$ t $ =$ T $ Ġtoo . ĠWe Ġhypothes ize Ġthat Ġnode Ġfeatures Ġat Ġ$ T \ ge Ġ2 $ Ġare Ġtoo Ġdiffuse Ġto Ġbe Ġentirely Ġrelied Ġupon Ġduring Ġread out . ĠMore Ġprecisely , Ġinitially Ġat Ġ$ t $ = 0 , Ġnode Ġrepresentations Ġcapture Ġinformation Ġabout Ġwords , Ġat Ġ$ t $ = 1 , Ġabout Ġtheir Ġ1 - hop Ġneighborhood Ġ( big rams ), Ġat Ġ$ t $ = 2 , Ġabout Ġcompositions Ġof Ġbig rams , Ġetc . ĠThus , Ġpretty Ġquickly , Ġnode Ġfeatures Ġbecome Ġgeneral Ġand Ġdiffuse . ĠIn Ġsuch Ġcases , Ġconsidering Ġalso Ġthe Ġlower - level , Ġmore Ġprecise Ġfeatures Ġof Ġthe Ġearlier Ġiterations Ġwhen Ġreading Ġout Ġmay Ġbe Ġnecessary . </s> Und irect ed Ġedges . ĠOn ĠReuters , Ġusing Ġan Ġund irect ed Ġgraph Ġleads Ġto Ġbetter Ġperformance , Ġwhile Ġon ĠPol arity Ġand ĠIM DB , Ġit Ġis Ġthe Ġopposite . ĠThis Ġcan Ġbe Ġexplained Ġby Ġthe Ġfact Ġthat ĠReuters Ġis Ġa Ġtopic Ġclassification Ġtask , Ġfor Ġwhich Ġthe Ġpresence Ġor Ġabsence Ġof Ġsome Ġpatterns Ġis Ġimportant , Ġbut Ġnot Ġnecessarily Ġthe Ġorder Ġin Ġwhich Ġthey Ġappear , Ġwhile ĠPol arity Ġand ĠIM DB Ġare Ġsentiment Ġanalysis Ġtasks . ĠTo Ġcapture Ġsentiment , Ġmodeling Ġword Ġorder Ġis Ġcrucial , Ġe . g ., Ġin Ġdetecting Ġneg ation . </s> No Ġmaster Ġnode . ĠRem oving Ġthe Ġmaster Ġnode Ġdeterior ates Ġperformance Ġacross Ġall Ġdatasets , Ġclearly Ġshowing Ġthe Ġvalue Ġof Ġhaving Ġsuch Ġa Ġnode . ĠWe Ġhypothes ize Ġthat Ġsince Ġthe Ġspecial Ġdocument Ġnode Ġis Ġconnected Ġto Ġall Ġother Ġnodes , Ġit Ġis Ġable Ġto Ġencode Ġduring Ġmessage Ġpassing Ġa Ġsummary Ġof Ġthe Ġdocument . </s> No Ġren ormal ization . ĠHere , Ġwe Ġdo Ġnot Ġuse Ġthe Ġren ormal ization Ġtrick Ġof ĠB IB REF 9 Ġduring ĠMP Ġ( see Ġsubsection ĠSEC REF 10 ). ĠThat Ġis , ĠE q . ĠDIS PLAY _ FORM 11 Ġbecomes Ġ$\ math bf Ġ{ M } ^{ t + 1 } Ġ= Ġ\ text sc Ġ{ ML P } ^{ t + 1 }\ big Ġ( \ math bf Ġ{ A }\ math bf Ġ{ H } ^{ t }\ big Ġ) $. ĠIn Ġother Ġwords , Ġinstead Ġof Ġcomputing Ġa Ġweighted Ġaverage Ġof Ġthe Ġincoming Ġneighbors ' Ġfeature Ġvectors , Ġwe Ġcompute Ġa Ġweighted Ġsum Ġof Ġthem . ĠUnlike Ġthe Ġmean , Ġwhich Ġcaptures Ġdistributions , Ġthe Ġsum Ġcaptures Ġstructural Ġinformation ĠB IB REF 14 . ĠAs Ġshown Ġin ĠTable ĠT AB REF 29 , Ġusing Ġsum Ġinstead Ġof Ġmean Ġdecreases Ġperformance Ġeverywhere , Ġsuggesting Ġthat Ġin Ġour Ġapplication , Ġcapturing Ġthe Ġdistribution Ġof Ġneighbor Ġrepresentations Ġis Ġmore Ġimportant Ġthat Ġcapturing Ġtheir Ġstructure . ĠWe Ġhypothes ize Ġthat Ġthis Ġis Ġthe Ġcase Ġbecause Ġstatistical Ġword Ġco - occ urrence Ġnetworks Ġtend Ġto Ġhave Ġsimilar Ġstructural Ġproperties , Ġregardless Ġof Ġthe Ġtopic , Ġpol arity , Ġsentiment , Ġetc . Ġof Ġthe Ġcorresponding Ġdocuments . </s> Neigh bors - only . ĠIn Ġthis Ġexperiment , Ġwe Ġreplaced Ġthe ĠGR U Ġcombine Ġfunction Ġ( see ĠE q . ĠDIS PLAY _ FORM 14 ) Ġwith Ġthe Ġidentity Ġfunction . ĠThat Ġis , Ġwe Ġsimply Ġhave Ġ$\ math bf Ġ{ H } ^{ t + 1 } $ =$ \ math bf Ġ{ M } ^{ t + 1 } $. ĠSince Ġ$\ math bf Ġ{ A } $ Ġhas Ġzero Ġdiagonal , Ġby Ġdoing Ġso , Ġwe Ġcompletely Ġignore Ġthe Ġprevious Ġfeature Ġof Ġthe Ġnode Ġitself Ġwhen Ġupdating Ġits Ġrepresentation . ĠThat Ġis , Ġthe Ġupdate Ġis Ġbased Ġentirely Ġon Ġits Ġneighbors . ĠExcept Ġon ĠReuters Ġ( almost Ġno Ġchange ), Ġperformance Ġalways Ġsuffers , Ġstressing Ġthe Ġneed Ġto Ġtake Ġinto Ġaccount Ġthe Ġroot Ġnode Ġduring Ġupdates , Ġnot Ġonly Ġits Ġneighborhood . </s> Related Ġwork </s> In Ġwhat Ġfollows , Ġwe Ġoffer Ġa Ġbrief Ġreview Ġof Ġrelevant Ġstudies , Ġranked Ġby Ġincreasing Ġorder Ġof Ġsimilarity Ġwith Ġour Ġwork . </s> BI B REF 9 , ĠB IB REF 54 , ĠB IB REF 11 , ĠB IB REF 10 Ġconduct Ġsome Ġnode Ġclassification Ġexperiments Ġon Ġcitation Ġnetworks , Ġwhere Ġnodes Ġare Ġscientific Ġpapers , Ġi . e ., Ġtextual Ġdata . ĠHowever , Ġtext Ġis Ġonly Ġused Ġto Ġderive Ġnode Ġfeature Ġvectors . ĠThe Ġexternal Ġgraph Ġstructure , Ġwhich Ġplays Ġa Ġcentral Ġrole Ġin Ġdetermining Ġnode Ġlabels , Ġis Ġcompletely Ġunrelated Ġto Ġtext . </s> On Ġthe Ġother Ġhand , ĠB IB REF 55 , ĠB IB REF 7 Ġexperiment Ġon Ġtraditional Ġdocument Ġclassification Ġtasks . ĠThey Ġboth Ġbuild Ġ$ k $ - ne arest Ġneighbor Ġsimilarity Ġgraphs Ġbased Ġon Ġthe ĠGa ussian Ġdiffusion Ġkernel . ĠMore Ġprecisely , ĠB IB REF 55 Ġbuild Ġone Ġsingle Ġgraph Ġwhere Ġnodes Ġare Ġdocuments Ġand Ġdistance Ġis Ġcomputed Ġin Ġthe ĠBo W Ġspace . ĠNode Ġfeatures Ġare Ġthen Ġused Ġfor Ġclassification . ĠCl oser Ġto Ġour Ġwork , ĠB IB REF 7 Ġrepresent Ġeach Ġdocument Ġas Ġa Ġgraph . ĠAll Ġdocument Ġgraphs Ġare Ġderived Ġfrom Ġthe Ġsame Ġunderlying Ġstructure . ĠOnly Ġnode Ġfeatures , Ġcorresponding Ġto Ġthe Ġentries Ġof Ġthe Ġdocuments ' ĠBo W Ġvectors , Ġvary . ĠThe Ġunderlying , Ġshared Ġstructure Ġis Ġthat Ġof Ġa Ġ$ k $ - NN Ġgraph Ġwhere Ġnodes Ġare Ġvocabulary Ġterms Ġand Ġsimilarity Ġis Ġthe Ġcos ine Ġof Ġthe Ġword Ġembed ding Ġvectors . ĠB IB REF 7 Ġthen Ġperform Ġgraph Ġclassification . ĠHowever Ġthey Ġfound Ġperformance Ġto Ġbe Ġlower Ġthan Ġthat Ġof Ġa Ġnaive ĠBay es Ġclass ifier . </s> BI B REF 56 Ġuse Ġa ĠG NN Ġfor Ġhierarchical Ġclassification Ġinto Ġa Ġlarge Ġtax onomy Ġof Ġtopics . ĠThis Ġtask Ġdiffers Ġfrom Ġtraditional Ġdocument Ġclassification . ĠThe Ġauthors Ġrepresent Ġdocuments Ġas Ġun weight ed , Ġund irect ed Ġword Ġco - occ urrence Ġnetworks Ġwith Ġword Ġembed d ings Ġas Ġnode Ġfeatures . ĠThey Ġthen Ġuse Ġthe Ġspatial ĠG NN Ġof ĠB IB REF 15 Ġto Ġperform Ġgraph Ġclassification . </s> The Ġwork Ġclosest Ġto Ġours Ġis Ġprobably Ġthat Ġof ĠB IB REF 53 . ĠThe Ġauthors Ġadopt Ġthe Ġsemi - super vised Ġnode Ġclassification Ġapproach Ġof ĠB IB REF 9 . ĠThey Ġbuild Ġone Ġsingle Ġund irect ed Ġgraph Ġfrom Ġthe Ġentire Ġdataset , Ġwith Ġboth Ġword Ġand Ġdocument Ġnodes . ĠDocument - word Ġedges Ġare Ġweighted Ġby ĠTF - ID F Ġand Ġword - word Ġedges Ġare Ġweighted Ġby Ġpoint wise Ġmutual Ġinformation Ġderived Ġfrom Ġco - occ urrence Ġwithin Ġa Ġsliding Ġwindow . ĠThere Ġare Ġno Ġdocument - document Ġedges . ĠThe ĠG NN Ġis Ġtrained Ġbased Ġon Ġthe Ġcross - ent ropy Ġloss Ġcomputed Ġonly Ġfor Ġthe Ġlabeled Ġnodes , Ġthat Ġis , Ġthe Ġdocuments Ġin Ġthe Ġtraining Ġset . ĠWhen Ġthe Ġfinal Ġnode Ġrepresentations Ġare Ġobtained , Ġone Ġcan Ġuse Ġthat Ġof Ġthe Ġtest Ġdocuments Ġto Ġclassify Ġthem Ġand Ġevaluate Ġprediction Ġperformance . </s> There Ġare Ġsignificant Ġdifferences Ġbetween ĠB IB REF 53 Ġand Ġour Ġwork . ĠFirst , Ġour Ġapproach Ġis Ġinduct ive , Ġnot Ġtrans duct ive . ĠIndeed , Ġwhile Ġthe Ġnode Ġclassification Ġapproach Ġof ĠB IB REF 53 Ġrequires Ġall Ġtest Ġdocuments Ġat Ġtraining Ġtime , Ġour Ġgraph Ġclassification Ġmodel Ġis Ġable Ġto Ġperform Ġinference Ġon Ġnew , Ġnever - seen Ġdocuments . ĠThe Ġdownside Ġof Ġrepresenting Ġdocuments Ġas Ġseparate Ġgraphs , Ġhowever , Ġis Ġthat Ġwe Ġlose Ġthe Ġability Ġto Ġcapture Ġcorpus - level Ġdependencies . ĠAlso , Ġour Ġdirected Ġgraphs Ġcapture Ġword Ġordering , Ġwhich Ġis Ġignored Ġby ĠB IB REF 53 . ĠFinally , Ġthe Ġapproach Ġof ĠB IB REF 53 Ġrequires Ġcomputing Ġthe ĠPM I Ġfor Ġevery Ġword Ġpair Ġin Ġthe Ġvocabulary , Ġwhich Ġmay Ġbe Ġprohib itive Ġon Ġdatasets Ġwith Ġvery Ġlarge Ġvoc ab ular ies . ĠOn Ġthe Ġother Ġhand , Ġthe Ġcomplexity Ġof ĠMP AD Ġdoes Ġnot Ġdepend Ġon Ġvocabulary Ġsize . </s> Conclusion </s> We Ġhave Ġproposed Ġan Ġapplication Ġof Ġthe Ġmessage Ġpassing Ġframework Ġto ĠN LP , Ġthe ĠMessage ĠPassing ĠAttention Ġnetwork Ġfor ĠDocument Ġunderstanding Ġ( MP AD ). ĠExper iments Ġconducted Ġon Ġ10 Ġstandard Ġtext Ġclassification Ġdatasets Ġshow Ġthat Ġour Ġarchitecture Ġis Ġcompetitive Ġwith Ġthe Ġstate - of - the - art . ĠBy Ġprocessing Ġweighted , Ġdirected Ġword Ġco - occ urrence Ġnetworks , ĠMP AD Ġis Ġsensitive Ġto Ġword Ġorder Ġand Ġword - word Ġrelationship Ġstrength . ĠTo Ġexplicitly Ġcapture Ġthe Ġhierarchical Ġstructure Ġof Ġdocuments , Ġwe Ġalso Ġpropose Ġthree Ġhierarchical Ġvariants Ġof ĠMP AD , Ġthat Ġwe Ġshow Ġbring Ġimprovements Ġover Ġthe Ġvanilla Ġarchitecture . </s> Acknowled gments </s> We Ġthank Ġthe ĠNV idia Ġcorporation Ġfor Ġthe Ġdonation Ġof Ġa ĠGPU Ġas Ġpart Ġof Ġtheir ĠGPU Ġgrant Ġprogram .
Introduction The concept of message passing over graphs has been around for many years BIBREF0, BIBREF1, as well as that of graph neural networks (GNNs) BIBREF2, BIBREF3. However, GNNs have only recently started to be closely investigated, following the advent of deep learning. Some notable examples include BIBREF4, BIBREF5, BIBREF6, BIBREF7, BIBREF8, BIBREF9, BIBREF10, BIBREF11, BIBREF12. These approaches are known as spectral. Their similarity with message passing (MP) was observed by BIBREF9 and formalized by BIBREF13 and BIBREF14. The MP framework is based on the core idea of recursive neighborhood aggregation. That is, at every iteration, the representation of each vertex is updated based on messages received from its neighbors. All spectral GNNs can be described in terms of the MP framework. GNNs have been applied with great success to bioinformatics and social network data, for node classification, link prediction, and graph classification. However, a few studies only have focused on the application of the MP framework to representation learning on text. This paper proposes one such application. More precisely, we represent documents as word co-occurrence networks, and develop an expressive MP GNN tailored to document understanding, the Message Passing Attention network for Document understanding (MPAD). We also propose several hierarchical variants of MPAD. Evaluation on 10 document classification datasets show that our architectures learn representations that are competitive with the state-of-the-art. Furthermore, ablation experiments shed light on the impact of various architectural choices. In what follows, we first provide some background about the MP framework (in sec. SECREF2), thoroughly describe and explain MPAD (sec. SECREF3), present our experimental framework (sec. SECREF4), report and interpret our results (sec. SECREF5), and provide a review of the relevant literature (sec. SECREF6). Message Passing Neural Networks BIBREF13 proposed a MP framework under which many of the recently introduced GNNs can be reformulated. MP consists in an aggregation phase followed by a combination phase BIBREF14. More precisely, let $G(V,E)$ be a graph, and let us consider $v \in V$. At time $t+1$, a message vector $\mathbf {m}_v^{t+1}$ is computed from the representations of the neighbors $\mathcal {N}(v)$ of $v$: The new representation $\mathbf {h}^{t+1}_v$ of $v$ is then computed by combining its current feature vector $\mathbf {h}^{t}_v$ with the message vector $\mathbf {m}_v^{t+1}$: Messages are passed for $T$ time steps. Each step is implemented by a different layer of the MP network. Hence, iterations correspond to network depth. The final feature vector $\mathbf {h}_v^T$ of $v$ is based on messages propagated from all the nodes in the subtree of height $T$ rooted at $v$. It captures both the topology of the neighborhood of $v$ and the distribution of the vertex representations in it. If a graph-level feature vector is needed, e.g., for classification or regression, a READOUT pooling function, that must be invariant to permutations, is applied: Next, we present the MP network we developed for document understanding. Message Passing Attention network for Document understanding (MPAD) ::: Word co-occurrence networks We represent a document as a statistical word co-occurrence network BIBREF18, BIBREF19 with a sliding window of size 2 overspanning sentences. Let us denote that graph $G(V,E)$. Each unique word in the preprocessed document is represented by a node in $G$, and an edge is added between two nodes if they are found together in at least one instantiation of the window. $G$ is directed and weighted: edge directions and weights respectively capture text flow and co-occurrence counts. $G$ is a compact representation of its document. In $G$, immediate neighbors are consecutive words in the same sentence. That is, paths of length 2 correspond to bigrams. Paths of length more than 2 can correspond either to traditional $n$-grams or to relaxed $n$-grams, that is, words that never appear in the same sentence but co-occur with the same word(s). Such nodes are linked through common neighbors. Master node. Inspired by BIBREF3, our $G$ also includes a special document node, linked to all other nodes via unit weight bi-directional edges. In what follows, let us denote by $n$ the number of nodes in $G$, including the master node. Message Passing Attention network for Document understanding (MPAD) ::: Message passing We formulate our AGGREGATE function as: where $\mathbf {H}^t \in \mathbb {R}^{n \times d}$ contains node features ($d$ is a hyperparameter), and $\mathbf {A} \in \mathbb {R}^{n \times n}$ is the adjacency matrix of $G$. Since $G$ is directed, $\mathbf {A}$ is asymmetric. Also, $\mathbf {A}$ has zero diagonal as we choose not to consider the feature of the node itself, only that of its incoming neighbors, when updating its representation. Since $G$ is weighted, the $i^{th}$ row of $A$ contains the weights of the edges incoming on node $v_i$. $\mathbf {D} \in \mathbb {R}^{n \times n}$ is the diagonal in-degree matrix of $G$. MLP denotes a multi-layer perceptron, and $\mathbf {M}^{t+1} \in \mathbb {R}^{n \times d}$ is the message matrix. The use of a MLP was motivated by the observation that for graph classification, MP neural nets with 1-layer perceptrons are inferior to their MLP counterparts BIBREF14. Indeed, 1-layer perceptrons are not universal approximators of multiset functions. Note that like in BIBREF14, we use a different MLP at each layer. Renormalization. The rows of $\mathbf {D}^{-1}\mathbf {A}$ sum to 1. This is equivalent to the renormalization trick of BIBREF9, but using only the in-degrees. That is, instead of computing a weighted sum of the incoming neighbors' feature vectors, we compute a weighted average of them. The coefficients are proportional to the strength of co-occurrence between words. One should note that by averaging, we lose the ability to distinguish between different neighborhood structures in some special cases, that is, we lose injectivity. Such cases include neighborhoods in which all nodes have the same representations, and neighborhoods of different sizes containing various representations in equal proportions BIBREF14. As suggested by the results of an ablation experiment, averaging is better than summing in our application (see subsection SECREF30). Note that instead of simply summing/averaging, we also tried using GAT-like attention BIBREF11 in early experiments, without obtaining better results. As far as our COMBINE function, we use the Gated Recurrent Unit BIBREF20, BIBREF21: Omitting biases for readability, we have: where the $\mathbf {W}$ and $\mathbf {U}$ matrices are trainable weight matrices not shared across time steps, $\sigma (\mathbf {x}) = 1/(1+\exp (-\mathbf {x}))$ is the sigmoid function, and $\mathbf {R}$ and $\mathbf {Z}$ are the parameters of the reset and update gates. The reset gate controls the amount of information from the previous time step (in $\mathbf {H}^t$) that should propagate to the candidate representations, $\tilde{\mathbf {H}}^{t+1}$. The new representations $\mathbf {H}^{t+1}$ are finally obtained by linearly interpolating between the previous and the candidate ones, using the coefficients returned by the update gate. Interpretation. Updating node representations through a GRU should in principle allow nodes to encode a combination of local and global signals (low and high values of $t$, resp.), by allowing them to remember about past iterations. In addition, we also explicitly consider node representations at all iterations when reading out (see Eq. DISPLAY_FORM18). Message Passing Attention network for Document understanding (MPAD) ::: Readout After passing messages and performing updates for $T$ iterations, we obtain a matrix $\mathbf {H}^T \in \mathbb {R}^{n \times d}$ containing the final vertex representations. Let $\hat{G}$ be graph $G$ without the special document node, and matrix $\mathbf {\hat{H}}^T \in \mathbb {R}^{(n-1) \times d}$ be the corresponding representation matrix (i.e., $\mathbf {H}^T$ without the row of the document node). We use as our READOUT function the concatenation of self-attention applied to $\mathbf {\hat{H}}^T$ with the final document node representation. More precisely, we apply a global self-attention mechanism BIBREF22 to the rows of $\mathbf {\hat{H}}^T$. As shown in Eq. DISPLAY_FORM17, $\mathbf {\hat{H}}^T$ is first passed to a dense layer parameterized by matrix $\mathbf {W}_A^T \in \mathbb {R}^{d \times d}$. An alignment vector $\mathbf {a}$ is then derived by comparing, via dot products, the rows of the output of the dense layer $\mathbf {Y}^T \in \mathbb {R}^{(n-1) \times d}$ with a trainable vector $\mathbf {v}^T \in \mathbb {R}^d$ (initialized randomly) and normalizing with a softmax. The normalized alignment coefficients are finally used to compute the attentional vector $\mathbf {u}^T \in \mathbb {R}^d$ as a weighted sum of the final representations $\mathbf {\hat{H}}^T$. Note that we tried with multiple context vectors, i.e., with a matrix $\mathbf {V}^T$ instead of a vector $\mathbf {v}^T$, like in BIBREF22, but results were not convincing, even when adding a regularization term to the loss to favor diversity among the rows of $\mathbf {V}^T$. Master node skip connection. $\mathbf {h}_G^T \in \mathbb {R}^{2d}$ is obtained by concatenating $\mathbf {u}^T$ and the final master node representation. That is, the master node vector bypasses the attention mechanism. This is equivalent to a skip or shortcut connection BIBREF23. The reason behind this choice is that we expect the special document node to learn a high-level summary about the document, such as its size, vocabulary, etc. (more details are given in subsection SECREF30). Therefore, by making the master node bypass the attention layer, we directly inject global information about the document into its final representation. Multi-readout. BIBREF14, inspired by Jumping Knowledge Networks BIBREF12, recommend to not only use the final representations when performing readout, but also that of the earlier steps. Indeed, as one iterates, node features capture more and more global information. However, retaining more local, intermediary information might be useful too. Thus, instead of applying the readout function only to $t=T$, we apply it to all time steps and concatenate the results, finally obtaining $\mathbf {h}_G \in \mathbb {R}^{T \times 2d}$ : In effect, with this modification, we take into account features based on information aggregated from subtrees of different heights (from 1 to $T$), corresponding to local and global features. Message Passing Attention network for Document understanding (MPAD) ::: Hierarchical variants of MPAD Through the successive MP iterations, it could be argued that MPAD implicitly captures some soft notion of the hierarchical structure of documents (words $\rightarrow $ bigrams $\rightarrow $ compositions of bigrams, etc.). However, it might be beneficial to explicitly capture document hierarchy. Hierarchical architectures have brought significant improvements to many NLP tasks, such as language modeling and generation BIBREF24, BIBREF25, sentiment and topic classification BIBREF26, BIBREF27, and spoken language understanding BIBREF28, BIBREF29. Inspired by this line of research, we propose several hierarchical variants of MPAD, detailed in what follows. In all of them, we represent each sentence in the document as a word co-occurrence network, and obtain an embedding for it by applying MPAD as previously described. MPAD-sentence-att. Here, the sentence embeddings are simply combined through self-attention. MPAD-clique. In this variant, we build a complete graph where each node represents a sentence. We then feed that graph to MPAD, where the feature vectors of the nodes are initialized with the sentence embeddings previously obtained. MPAD-path. This variant is similar to the clique one, except that instead of a complete graph, we build a path according to the natural flow of the text. That is, two nodes are linked by a directed edge if the two sentences they represent follow each other in the document. Experiments ::: Datasets We evaluate the quality of the document embeddings learned by MPAD on 10 document classification datasets, covering the topic identification, coarse and fine sentiment analysis and opinion mining, and subjectivity detection tasks. We briefly introduce the datasets next. Their statistics are reported in Table TABREF21. (1) Reuters. This dataset contains stories collected from the Reuters news agency in 1987. Following common practice, we used the ModApte split and considered only the 10 classes with the highest number of positive training examples. We also removed documents belonging to more than one class and then classes left with no document (2 classes). (2) BBCSport BIBREF30 contains documents from the BBC Sport website corresponding to 2004-2005 sports news articles. (3) Polarity BIBREF31 features positive and negative labeled snippets from Rotten Tomatoes. (4) Subjectivity BIBREF32 contains movie review snippets from Rotten Tomatoes (subjective sentences), and Internet Movie Database plot summaries (objective sentences). (5) MPQA BIBREF33 is made of positive and negative phrases, annotated as part of the summer 2002 NRRC Workshop on Multi-Perspective Question Answering. (6) IMDB BIBREF34 is a collection of highly polarized movie reviews from IMDB (positive and negative). There are at most 30 reviews for each movie. (7) TREC BIBREF35 consists of questions that are classified into 6 different categories. (8) SST-1 BIBREF36 contains the same snippets as Polarity. The authors used the Stanford Parser to parse the snippets and split them into multiple sentences. They then used Amazon Mechanical Turk to annotate the resulting phrases according to their polarity (very negative, negative, neutral, positive, very positive). (9) SST-2 BIBREF36 is the same as SST-1 but with neutral reviews removed and snippets classified as positive or negative. (10) Yelp2013 BIBREF26 features reviews obtained from the 2013 Yelp Dataset Challenge. Experiments ::: Baselines We evaluate MPAD against multiple state-of-the-art baseline models, including hierarchical ones, to enable fair comparison with the hierarchical MPAD variants. doc2vec BIBREF37. Doc2vec (or paragraph vector) is an extension of word2vec that learns vectors for documents in a fully unsupervised manner. Document embeddings are then fed to a logistic regression classifier. CNN BIBREF38. The convolutional neural network architecture, well-known in computer vision, is applied to text. There is one spatial dimension and the word embeddings are used as channels (depth dimensions). DAN BIBREF39. The Deep Averaging Network passes the unweighted average of the embeddings of the input words through multiple dense layers and a final softmax. Tree-LSTM BIBREF40 is a generalization of the standard LSTM architecture to constituency and dependency parse trees. DRNN BIBREF41. Recursive neural networks are stacked and applied to parse trees. LSTMN BIBREF42 is an extension of the LSTM model where the memory cell is replaced by a memory network which stores word representations. C-LSTM BIBREF43 combines convolutional and recurrent neural networks. The region embeddings provided by a CNN are fed to a LSTM. SPGK BIBREF44 also models documents as word co-occurrence networks. It computes a graph kernel that compares shortest paths extracted from the word co-occurrence networks and then uses a SVM to categorize documents. WMD BIBREF45 is an application of the well-known Earth Mover's Distance to text. A k-nearest neighbor classifier is used. S-WMD BIBREF46 is a supervised extension of the Word Mover's Distance. Semantic-CNN BIBREF47. Here, a CNN is applied to semantic units obtained by clustering words in the embedding space. LSTM-GRNN BIBREF26 is a hierarchical model where sentence embeddings are obtained with a CNN and a GRU-RNN is fed the sentence representations to obtain a document vector. HN-ATT BIBREF27 is another hierarchical model, where the same encoder architecture (a bidirectional GRU-RNN) is used for both sentences and documents, with different parameters. A self-attention mechanism is applied to the RNN annotations at each level. Experiments ::: Model configuration and training We preprocess all datasets using the code of BIBREF38. On Yelp2013, we also replace all tokens appearing strictly less than 6 times with a special UNK token, like in BIBREF27. We then build a directed word co-occurrence network from each document, with a window of size 2. We use two MP iterations ($T$=2) for the basic MPAD, and two MP iterations at each level, for the hierarchical variants. We set $d$ to 64, except on IMDB and Yelp on which $d=128$, and use a two-layer MLP. The final graph representations are passed through a softmax for classification. We train MPAD in an end-to-end fashion by minimizing the cross-entropy loss function with the Adam optimizer BIBREF48 and an initial learning rate of 0.001. To regulate potential differences in magnitude, we apply batch normalization after concatenating the feature vector of the master node with the self-attentional vector, that is, after the skip connection (see subsection SECREF16). To prevent overfitting, we use dropout BIBREF49 with a rate of 0.5. We select the best epoch, capped at 200, based on the validation accuracy. When cross-validation is used (see 3rd column of Table TABREF21), we construct a validation set by randomly sampling 10% of the training set of each fold. On all datasets except Yelp2013, we use the publicly available 300-dimensional pre-trained Google News vectors ($D$=300) BIBREF50 to initialize the node representations $\mathbf {H}^0$. On Yelp2013, we follow BIBREF27 and learn our own word vectors from the training and validation sets with the gensim implementation of word2vec BIBREF51. MPAD was implemented in Python 3.6 using the PyTorch library BIBREF52. All experiments were run on a single machine consisting of a 3.4 GHz Intel Core i7 CPU with 16 GB of RAM and an NVidia GeForce Titan Xp GPU. Results and ablations ::: Results Experimental results are shown in Table TABREF28. For the baselines, the best scores reported in each original paper are shown. MPAD reaches best performance on 7 out of 10 datasets, and is close second elsewhere. Moreover, the 7 datasets on which MPAD ranks first widely differ in training set size, number of categories, and prediction task (topic, sentiment, subjectivity), which indicates that MPAD can perform well in different settings. MPAD vs. hierarchical variants. On 9 datasets out of 10, one or more of the hierarchical variants outperform the vanilla MPAD architecture, highlighting the benefit of explicitly modeling the hierarchical nature of documents. However, on Subjectivity, standard MPAD outperforms all hierarchical variants. On TREC, it reaches the same accuracy. We hypothesize that in some cases, using a different graph to separately encode each sentence might be worse than using one single graph to directly encode the document. Indeed, in the single document graph, some words that never appear in the same sentence can be connected through common neighbors, as was explained in subsection SECREF7. So, this way, some notion of cross-sentence context is captured while learning representations of words, bigrams, etc. at each MP iteration. This creates better informed representations, resulting in a better document embedding. With the hierarchical variants, on the other hand, each sentence vector is produced in isolation, without any contextual information about the other sentences in the document. Therefore, the final sentence embeddings might be of lower quality, and as a group might also contain redundant/repeated information. When the sentence vectors are finally combined into a document representation, it is too late to take context into account. Results and ablations ::: Ablation studies To understand the impact of some hyperparameters on performance, we conducted additional experiments on the Reuters, Polarity, and IMDB datasets, with the non-hierarchical version of MPAD. Results are shown in Table TABREF29. Number of MP iterations. First, we varied the number of message passing iterations from 1 to 4. We can clearly see in Table TABREF29 that having more iterations improves performance. We attribute this to the fact that we are reading out at each iteration from 1 to $T$ (see Eq. DISPLAY_FORM18), which enables the final graph representation to encode a mixture of low-level and high-level features. Indeed, in initial experiments involving readout at $t$=$T$ only, setting $T\ge 2$ was always decreasing performance, despite the GRU-based updates (Eq. DISPLAY_FORM14). These results were consistent with that of BIBREF53 and BIBREF9, who both are reading out only at $t$=$T$ too. We hypothesize that node features at $T\ge 2$ are too diffuse to be entirely relied upon during readout. More precisely, initially at $t$=0, node representations capture information about words, at $t$=1, about their 1-hop neighborhood (bigrams), at $t$=2, about compositions of bigrams, etc. Thus, pretty quickly, node features become general and diffuse. In such cases, considering also the lower-level, more precise features of the earlier iterations when reading out may be necessary. Undirected edges. On Reuters, using an undirected graph leads to better performance, while on Polarity and IMDB, it is the opposite. This can be explained by the fact that Reuters is a topic classification task, for which the presence or absence of some patterns is important, but not necessarily the order in which they appear, while Polarity and IMDB are sentiment analysis tasks. To capture sentiment, modeling word order is crucial, e.g., in detecting negation. No master node. Removing the master node deteriorates performance across all datasets, clearly showing the value of having such a node. We hypothesize that since the special document node is connected to all other nodes, it is able to encode during message passing a summary of the document. No renormalization. Here, we do not use the renormalization trick of BIBREF9 during MP (see subsection SECREF10). That is, Eq. DISPLAY_FORM11 becomes $\mathbf {M}^{t+1} = \textsc {MLP}^{t+1}\big (\mathbf {A}\mathbf {H}^{t}\big )$. In other words, instead of computing a weighted average of the incoming neighbors' feature vectors, we compute a weighted sum of them. Unlike the mean, which captures distributions, the sum captures structural information BIBREF14. As shown in Table TABREF29, using sum instead of mean decreases performance everywhere, suggesting that in our application, capturing the distribution of neighbor representations is more important that capturing their structure. We hypothesize that this is the case because statistical word co-occurrence networks tend to have similar structural properties, regardless of the topic, polarity, sentiment, etc. of the corresponding documents. Neighbors-only. In this experiment, we replaced the GRU combine function (see Eq. DISPLAY_FORM14) with the identity function. That is, we simply have $\mathbf {H}^{t+1}$=$\mathbf {M}^{t+1}$. Since $\mathbf {A}$ has zero diagonal, by doing so, we completely ignore the previous feature of the node itself when updating its representation. That is, the update is based entirely on its neighbors. Except on Reuters (almost no change), performance always suffers, stressing the need to take into account the root node during updates, not only its neighborhood. Related work In what follows, we offer a brief review of relevant studies, ranked by increasing order of similarity with our work. BIBREF9, BIBREF54, BIBREF11, BIBREF10 conduct some node classification experiments on citation networks, where nodes are scientific papers, i.e., textual data. However, text is only used to derive node feature vectors. The external graph structure, which plays a central role in determining node labels, is completely unrelated to text. On the other hand, BIBREF55, BIBREF7 experiment on traditional document classification tasks. They both build $k$-nearest neighbor similarity graphs based on the Gaussian diffusion kernel. More precisely, BIBREF55 build one single graph where nodes are documents and distance is computed in the BoW space. Node features are then used for classification. Closer to our work, BIBREF7 represent each document as a graph. All document graphs are derived from the same underlying structure. Only node features, corresponding to the entries of the documents' BoW vectors, vary. The underlying, shared structure is that of a $k$-NN graph where nodes are vocabulary terms and similarity is the cosine of the word embedding vectors. BIBREF7 then perform graph classification. However they found performance to be lower than that of a naive Bayes classifier. BIBREF56 use a GNN for hierarchical classification into a large taxonomy of topics. This task differs from traditional document classification. The authors represent documents as unweighted, undirected word co-occurrence networks with word embeddings as node features. They then use the spatial GNN of BIBREF15 to perform graph classification. The work closest to ours is probably that of BIBREF53. The authors adopt the semi-supervised node classification approach of BIBREF9. They build one single undirected graph from the entire dataset, with both word and document nodes. Document-word edges are weighted by TF-IDF and word-word edges are weighted by pointwise mutual information derived from co-occurrence within a sliding window. There are no document-document edges. The GNN is trained based on the cross-entropy loss computed only for the labeled nodes, that is, the documents in the training set. When the final node representations are obtained, one can use that of the test documents to classify them and evaluate prediction performance. There are significant differences between BIBREF53 and our work. First, our approach is inductive, not transductive. Indeed, while the node classification approach of BIBREF53 requires all test documents at training time, our graph classification model is able to perform inference on new, never-seen documents. The downside of representing documents as separate graphs, however, is that we lose the ability to capture corpus-level dependencies. Also, our directed graphs capture word ordering, which is ignored by BIBREF53. Finally, the approach of BIBREF53 requires computing the PMI for every word pair in the vocabulary, which may be prohibitive on datasets with very large vocabularies. On the other hand, the complexity of MPAD does not depend on vocabulary size. Conclusion We have proposed an application of the message passing framework to NLP, the Message Passing Attention network for Document understanding (MPAD). Experiments conducted on 10 standard text classification datasets show that our architecture is competitive with the state-of-the-art. By processing weighted, directed word co-occurrence networks, MPAD is sensitive to word order and word-word relationship strength. To explicitly capture the hierarchical structure of documents, we also propose three hierarchical variants of MPAD, that we show bring improvements over the vanilla architecture. Acknowledgments We thank the NVidia corporation for the donation of a GPU as part of their GPU grant program.
What is the state-of-the-art system?
cb12c19f9d14bef7b2f778892d9071eea2d6c63d
What Ġis Ġthe Ġstate - of - the - art Ġsystem ?
doc 2 vec Ġ, ĠCNN , ĠD AN , ĠTree - L ST M , ĠDR NN , ĠL ST MN , ĠC - L ST M , ĠSP G K , ĠW MD , ĠS - W MD , ĠSem antic - CNN , ĠL ST M - GR NN , ĠH N - ATT
<s> What Ġis Ġthe Ġstate - of - the - art Ġsystem ? </s> Introduction </s> The Ġconcept Ġof Ġmessage Ġpassing Ġover Ġgraphs Ġhas Ġbeen Ġaround Ġfor Ġmany Ġyears ĠB IB REF 0 , ĠB IB REF 1 , Ġas Ġwell Ġas Ġthat Ġof Ġgraph Ġneural Ġnetworks Ġ( G NN s ) ĠB IB REF 2 , ĠB IB REF 3 . ĠHowever , ĠG NN s Ġhave Ġonly Ġrecently Ġstarted Ġto Ġbe Ġclosely Ġinvestigated , Ġfollowing Ġthe Ġadvent Ġof Ġdeep Ġlearning . ĠSome Ġnotable Ġexamples Ġinclude ĠB IB REF 4 , ĠB IB REF 5 , ĠB IB REF 6 , ĠB IB REF 7 , ĠB IB REF 8 , ĠB IB REF 9 , ĠB IB REF 10 , ĠB IB REF 11 , ĠB IB REF 12 . ĠThese Ġapproaches Ġare Ġknown Ġas Ġspectral . ĠTheir Ġsimilarity Ġwith Ġmessage Ġpassing Ġ( MP ) Ġwas Ġobserved Ġby ĠB IB REF 9 Ġand Ġformal ized Ġby ĠB IB REF 13 Ġand ĠB IB REF 14 . </s> The ĠMP Ġframework Ġis Ġbased Ġon Ġthe Ġcore Ġidea Ġof Ġrecursive Ġneighborhood Ġaggregation . ĠThat Ġis , Ġat Ġevery Ġiteration , Ġthe Ġrepresentation Ġof Ġeach Ġvertex Ġis Ġupdated Ġbased Ġon Ġmessages Ġreceived Ġfrom Ġits Ġneighbors . ĠAll Ġspectral ĠG NN s Ġcan Ġbe Ġdescribed Ġin Ġterms Ġof Ġthe ĠMP Ġframework . </s> G NN s Ġhave Ġbeen Ġapplied Ġwith Ġgreat Ġsuccess Ġto Ġbio in format ics Ġand Ġsocial Ġnetwork Ġdata , Ġfor Ġnode Ġclassification , Ġlink Ġprediction , Ġand Ġgraph Ġclassification . ĠHowever , Ġa Ġfew Ġstudies Ġonly Ġhave Ġfocused Ġon Ġthe Ġapplication Ġof Ġthe ĠMP Ġframework Ġto Ġrepresentation Ġlearning Ġon Ġtext . ĠThis Ġpaper Ġproposes Ġone Ġsuch Ġapplication . ĠMore Ġprecisely , Ġwe Ġrepresent Ġdocuments Ġas Ġword Ġco - occ urrence Ġnetworks , Ġand Ġdevelop Ġan Ġexpressive ĠMP ĠG NN Ġtailored Ġto Ġdocument Ġunderstanding , Ġthe ĠMessage ĠPassing ĠAttention Ġnetwork Ġfor ĠDocument Ġunderstanding Ġ( MP AD ). ĠWe Ġalso Ġpropose Ġseveral Ġhierarchical Ġvariants Ġof ĠMP AD . ĠEvaluation Ġon Ġ10 Ġdocument Ġclassification Ġdatasets Ġshow Ġthat Ġour Ġarchitectures Ġlearn Ġrepresentations Ġthat Ġare Ġcompetitive Ġwith Ġthe Ġstate - of - the - art . ĠFurthermore , Ġab lation Ġexperiments Ġshed Ġlight Ġon Ġthe Ġimpact Ġof Ġvarious Ġarchitectural Ġchoices . </s> In Ġwhat Ġfollows , Ġwe Ġfirst Ġprovide Ġsome Ġbackground Ġabout Ġthe ĠMP Ġframework Ġ( in Ġsec . ĠSEC REF 2 ), Ġthoroughly Ġdescribe Ġand Ġexplain ĠMP AD Ġ( sec . ĠSEC REF 3 ), Ġpresent Ġour Ġexperimental Ġframework Ġ( sec . ĠSEC REF 4 ), Ġreport Ġand Ġinterpret Ġour Ġresults Ġ( sec . ĠSEC REF 5 ), Ġand Ġprovide Ġa Ġreview Ġof Ġthe Ġrelevant Ġliterature Ġ( sec . ĠSEC REF 6 ). </s> Message ĠPassing ĠNeural ĠNetworks </s> BI B REF 13 Ġproposed Ġa ĠMP Ġframework Ġunder Ġwhich Ġmany Ġof Ġthe Ġrecently Ġintroduced ĠG NN s Ġcan Ġbe Ġreform ulated . ĠMP Ġconsists Ġin Ġan Ġaggregation Ġphase Ġfollowed Ġby Ġa Ġcombination Ġphase ĠB IB REF 14 . ĠMore Ġprecisely , Ġlet Ġ$ G ( V , E ) $ Ġbe Ġa Ġgraph , Ġand Ġlet Ġus Ġconsider Ġ$ v Ġ\ in ĠV $. ĠAt Ġtime Ġ$ t + 1 $, Ġa Ġmessage Ġvector Ġ$\ math bf Ġ{ m } _ v ^{ t + 1 } $ Ġis Ġcomputed Ġfrom Ġthe Ġrepresentations Ġof Ġthe Ġneighbors Ġ$\ math cal Ġ{ N } ( v ) $ Ġof Ġ$ v $ : </s> The Ġnew Ġrepresentation Ġ$\ math bf Ġ{ h } ^{ t + 1 } _ v $ Ġof Ġ$ v $ Ġis Ġthen Ġcomputed Ġby Ġcombining Ġits Ġcurrent Ġfeature Ġvector Ġ$\ math bf Ġ{ h } ^{ t } _ v $ Ġwith Ġthe Ġmessage Ġvector Ġ$\ math bf Ġ{ m } _ v ^{ t + 1 } $ : </s> Mess ages Ġare Ġpassed Ġfor Ġ$ T $ Ġtime Ġsteps . ĠEach Ġstep Ġis Ġimplemented Ġby Ġa Ġdifferent Ġlayer Ġof Ġthe ĠMP Ġnetwork . ĠHence , Ġiterations Ġcorrespond Ġto Ġnetwork Ġdepth . ĠThe Ġfinal Ġfeature Ġvector Ġ$\ math bf Ġ{ h } _ v ^ T $ Ġof Ġ$ v $ Ġis Ġbased Ġon Ġmessages Ġpropag ated Ġfrom Ġall Ġthe Ġnodes Ġin Ġthe Ġsubt ree Ġof Ġheight Ġ$ T $ Ġrooted Ġat Ġ$ v $. ĠIt Ġcaptures Ġboth Ġthe Ġtop ology Ġof Ġthe Ġneighborhood Ġof Ġ$ v $ Ġand Ġthe Ġdistribution Ġof Ġthe Ġvertex Ġrepresentations Ġin Ġit . </s> If Ġa Ġgraph - level Ġfeature Ġvector Ġis Ġneeded , Ġe . g ., Ġfor Ġclassification Ġor Ġregression , Ġa ĠREAD OUT Ġpool ing Ġfunction , Ġthat Ġmust Ġbe Ġinvari ant Ġto Ġperm utations , Ġis Ġapplied : </s> Next , Ġwe Ġpresent Ġthe ĠMP Ġnetwork Ġwe Ġdeveloped Ġfor Ġdocument Ġunderstanding . </s> Message ĠPassing ĠAttention Ġnetwork Ġfor ĠDocument Ġunderstanding Ġ( MP AD ) Ġ: :: ĠWord Ġco - occ urrence Ġnetworks </s> We Ġrepresent Ġa Ġdocument Ġas Ġa Ġstatistical Ġword Ġco - occ urrence Ġnetwork ĠB IB REF 18 , ĠB IB REF 19 Ġwith Ġa Ġsliding Ġwindow Ġof Ġsize Ġ2 Ġover span ning Ġsentences . ĠLet Ġus Ġdenote Ġthat Ġgraph Ġ$ G ( V , E ) $. ĠEach Ġunique Ġword Ġin Ġthe Ġpre process ed Ġdocument Ġis Ġrepresented Ġby Ġa Ġnode Ġin Ġ$ G $, Ġand Ġan Ġedge Ġis Ġadded Ġbetween Ġtwo Ġnodes Ġif Ġthey Ġare Ġfound Ġtogether Ġin Ġat Ġleast Ġone Ġinstant iation Ġof Ġthe Ġwindow . Ġ$ G $ Ġis Ġdirected Ġand Ġweighted : Ġedge Ġdirections Ġand Ġweights Ġrespectively Ġcapture Ġtext Ġflow Ġand Ġco - occ urrence Ġcounts . </s> $ G $ Ġis Ġa Ġcompact Ġrepresentation Ġof Ġits Ġdocument . ĠIn Ġ$ G $, Ġimmediate Ġneighbors Ġare Ġconsecutive Ġwords Ġin Ġthe Ġsame Ġsentence . ĠThat Ġis , Ġpaths Ġof Ġlength Ġ2 Ġcorrespond Ġto Ġbig rams . ĠPath s Ġof Ġlength Ġmore Ġthan Ġ2 Ġcan Ġcorrespond Ġeither Ġto Ġtraditional Ġ$ n $ - gram s Ġor Ġto Ġrelaxed Ġ$ n $ - gram s , Ġthat Ġis , Ġwords Ġthat Ġnever Ġappear Ġin Ġthe Ġsame Ġsentence Ġbut Ġco - occ ur Ġwith Ġthe Ġsame Ġword ( s ). ĠSuch Ġnodes Ġare Ġlinked Ġthrough Ġcommon Ġneighbors . </s> Master Ġnode . ĠInspired Ġby ĠB IB REF 3 , Ġour Ġ$ G $ Ġalso Ġincludes Ġa Ġspecial Ġdocument Ġnode , Ġlinked Ġto Ġall Ġother Ġnodes Ġvia Ġunit Ġweight Ġbi - direction al Ġedges . ĠIn Ġwhat Ġfollows , Ġlet Ġus Ġdenote Ġby Ġ$ n $ Ġthe Ġnumber Ġof Ġnodes Ġin Ġ$ G $, Ġincluding Ġthe Ġmaster Ġnode . </s> Message ĠPassing ĠAttention Ġnetwork Ġfor ĠDocument Ġunderstanding Ġ( MP AD ) Ġ: :: ĠMessage Ġpassing </s> We Ġformulate Ġour ĠA GG REG ATE Ġfunction Ġas : </s> where Ġ$\ math bf Ġ{ H } ^ t Ġ\ in Ġ\ math bb Ġ{ R } ^{ n Ġ\ times Ġd } $ Ġcontains Ġnode Ġfeatures Ġ($ d $ Ġis Ġa Ġhyper param eter ), Ġand Ġ$\ math bf Ġ{ A } Ġ\ in Ġ\ math bb Ġ{ R } ^{ n Ġ\ times Ġn } $ Ġis Ġthe Ġadj ac ency Ġmatrix Ġof Ġ$ G $. ĠSince Ġ$ G $ Ġis Ġdirected , Ġ$\ math bf Ġ{ A } $ Ġis Ġasymm etric . ĠAlso , Ġ$\ math bf Ġ{ A } $ Ġhas Ġzero Ġdiagonal Ġas Ġwe Ġchoose Ġnot Ġto Ġconsider Ġthe Ġfeature Ġof Ġthe Ġnode Ġitself , Ġonly Ġthat Ġof Ġits Ġincoming Ġneighbors , Ġwhen Ġupdating Ġits Ġrepresentation . ĠSince Ġ$ G $ Ġis Ġweighted , Ġthe Ġ$ i ^{ th } $ Ġrow Ġof Ġ$ A $ Ġcontains Ġthe Ġweights Ġof Ġthe Ġedges Ġincoming Ġon Ġnode Ġ$ v _ i $. Ġ$\ math bf Ġ{ D } Ġ\ in Ġ\ math bb Ġ{ R } ^{ n Ġ\ times Ġn } $ Ġis Ġthe Ġdiagonal Ġin - degree Ġmatrix Ġof Ġ$ G $. ĠML P Ġdenotes Ġa Ġmulti - layer Ġpercept ron , Ġand Ġ$\ math bf Ġ{ M } ^{ t + 1 } Ġ\ in Ġ\ math bb Ġ{ R } ^{ n Ġ\ times Ġd } $ Ġis Ġthe Ġmessage Ġmatrix . </s> The Ġuse Ġof Ġa ĠML P Ġwas Ġmotivated Ġby Ġthe Ġobservation Ġthat Ġfor Ġgraph Ġclassification , ĠMP Ġneural Ġnets Ġwith Ġ1 - layer Ġpercept rons Ġare Ġinferior Ġto Ġtheir ĠML P Ġcounterparts ĠB IB REF 14 . ĠIndeed , Ġ1 - layer Ġpercept rons Ġare Ġnot Ġuniversal Ġapprox im ators Ġof Ġmult is et Ġfunctions . ĠNote Ġthat Ġlike Ġin ĠB IB REF 14 , Ġwe Ġuse Ġa Ġdifferent ĠML P Ġat Ġeach Ġlayer . </s> Ren ormal ization . ĠThe Ġrows Ġof Ġ$\ math bf Ġ{ D } ^{ - 1 }\ math bf Ġ{ A } $ Ġsum Ġto Ġ1 . ĠThis Ġis Ġequivalent Ġto Ġthe Ġren ormal ization Ġtrick Ġof ĠB IB REF 9 , Ġbut Ġusing Ġonly Ġthe Ġin - deg rees . ĠThat Ġis , Ġinstead Ġof Ġcomputing Ġa Ġweighted Ġsum Ġof Ġthe Ġincoming Ġneighbors ' Ġfeature Ġvectors , Ġwe Ġcompute Ġa Ġweighted Ġaverage Ġof Ġthem . ĠThe Ġcoefficients Ġare Ġproportional Ġto Ġthe Ġstrength Ġof Ġco - occ urrence Ġbetween Ġwords . ĠOne Ġshould Ġnote Ġthat Ġby Ġaveraging , Ġwe Ġlose Ġthe Ġability Ġto Ġdistinguish Ġbetween Ġdifferent Ġneighborhood Ġstructures Ġin Ġsome Ġspecial Ġcases , Ġthat Ġis , Ġwe Ġlose Ġinject ivity . ĠSuch Ġcases Ġinclude Ġneighborhoods Ġin Ġwhich Ġall Ġnodes Ġhave Ġthe Ġsame Ġrepresentations , Ġand Ġneighborhoods Ġof Ġdifferent Ġsizes Ġcontaining Ġvarious Ġrepresentations Ġin Ġequal Ġproportions ĠB IB REF 14 . ĠAs Ġsuggested Ġby Ġthe Ġresults Ġof Ġan Ġab lation Ġexperiment , Ġaveraging Ġis Ġbetter Ġthan Ġsum ming Ġin Ġour Ġapplication Ġ( see Ġsubsection ĠSEC REF 30 ). ĠNote Ġthat Ġinstead Ġof Ġsimply Ġsum ming / aver aging , Ġwe Ġalso Ġtried Ġusing ĠG AT - like Ġattention ĠB IB REF 11 Ġin Ġearly Ġexperiments , Ġwithout Ġobtaining Ġbetter Ġresults . </s> As Ġfar Ġas Ġour ĠCOM B INE Ġfunction , Ġwe Ġuse Ġthe ĠG ated ĠRec urrent ĠUnit ĠB IB REF 20 , ĠB IB REF 21 : </s> O mitting Ġbiases Ġfor Ġread ability , Ġwe Ġhave : </s> where Ġthe Ġ$\ math bf Ġ{ W } $ Ġand Ġ$\ math bf Ġ{ U } $ Ġmat rices Ġare Ġtrain able Ġweight Ġmat rices Ġnot Ġshared Ġacross Ġtime Ġsteps , Ġ$\ s igma Ġ( \ math bf Ġ{ x }) Ġ= Ġ1 /( 1 + \ exp Ġ(- \ math bf Ġ{ x } )) $ Ġis Ġthe Ġs igm oid Ġfunction , Ġand Ġ$\ math bf Ġ{ R } $ Ġand Ġ$\ math bf Ġ{ Z } $ Ġare Ġthe Ġparameters Ġof Ġthe Ġreset Ġand Ġupdate Ġgates . ĠThe Ġreset Ġgate Ġcontrols Ġthe Ġamount Ġof Ġinformation Ġfrom Ġthe Ġprevious Ġtime Ġstep Ġ( in Ġ$\ math bf Ġ{ H } ^ t $ ) Ġthat Ġshould Ġpropagate Ġto Ġthe Ġcandidate Ġrepresentations , Ġ$\ t ilde {\ math bf Ġ{ H }} ^{ t + 1 } $. ĠThe Ġnew Ġrepresentations Ġ$\ math bf Ġ{ H } ^{ t + 1 } $ Ġare Ġfinally Ġobtained Ġby Ġlin early Ġinterpol ating Ġbetween Ġthe Ġprevious Ġand Ġthe Ġcandidate Ġones , Ġusing Ġthe Ġcoefficients Ġreturned Ġby Ġthe Ġupdate Ġgate . </s> Inter pret ation . ĠUp dating Ġnode Ġrepresentations Ġthrough Ġa ĠGR U Ġshould Ġin Ġprinciple Ġallow Ġnodes Ġto Ġencode Ġa Ġcombination Ġof Ġlocal Ġand Ġglobal Ġsignals Ġ( low Ġand Ġhigh Ġvalues Ġof Ġ$ t $, Ġresp .), Ġby Ġallowing Ġthem Ġto Ġremember Ġabout Ġpast Ġiterations . ĠIn Ġaddition , Ġwe Ġalso Ġexplicitly Ġconsider Ġnode Ġrepresentations Ġat Ġall Ġiterations Ġwhen Ġreading Ġout Ġ( see ĠE q . ĠDIS PLAY _ FORM 18 ). </s> Message ĠPassing ĠAttention Ġnetwork Ġfor ĠDocument Ġunderstanding Ġ( MP AD ) Ġ: :: ĠRead out </s> After Ġpassing Ġmessages Ġand Ġperforming Ġupdates Ġfor Ġ$ T $ Ġiterations , Ġwe Ġobtain Ġa Ġmatrix Ġ$\ math bf Ġ{ H } ^ T Ġ\ in Ġ\ math bb Ġ{ R } ^{ n Ġ\ times Ġd } $ Ġcontaining Ġthe Ġfinal Ġvertex Ġrepresentations . ĠLet Ġ$\ hat { G } $ Ġbe Ġgraph Ġ$ G $ Ġwithout Ġthe Ġspecial Ġdocument Ġnode , Ġand Ġmatrix Ġ$\ math bf Ġ{\ hat { H }} ^ T Ġ\ in Ġ\ math bb Ġ{ R } ^{ ( n - 1 ) Ġ\ times Ġd } $ Ġbe Ġthe Ġcorresponding Ġrepresentation Ġmatrix Ġ( i . e ., Ġ$\ math bf Ġ{ H } ^ T $ Ġwithout Ġthe Ġrow Ġof Ġthe Ġdocument Ġnode ). </s> We Ġuse Ġas Ġour ĠREAD OUT Ġfunction Ġthe Ġconc aten ation Ġof Ġself - att ention Ġapplied Ġto Ġ$\ math bf Ġ{\ hat { H }} ^ T $ Ġwith Ġthe Ġfinal Ġdocument Ġnode Ġrepresentation . ĠMore Ġprecisely , Ġwe Ġapply Ġa Ġglobal Ġself - att ention Ġmechanism ĠB IB REF 22 Ġto Ġthe Ġrows Ġof Ġ$\ math bf Ġ{\ hat { H }} ^ T $. ĠAs Ġshown Ġin ĠE q . ĠDIS PLAY _ FORM 17 , Ġ$\ math bf Ġ{\ hat { H }} ^ T $ Ġis Ġfirst Ġpassed Ġto Ġa Ġdense Ġlayer Ġparameter ized Ġby Ġmatrix Ġ$\ math bf Ġ{ W } _ A ^ T Ġ\ in Ġ\ math bb Ġ{ R } ^{ d Ġ\ times Ġd } $. ĠAn Ġalignment Ġvector Ġ$\ math bf Ġ{ a } $ Ġis Ġthen Ġderived Ġby Ġcomparing , Ġvia Ġdot Ġproducts , Ġthe Ġrows Ġof Ġthe Ġoutput Ġof Ġthe Ġdense Ġlayer Ġ$\ math bf Ġ{ Y } ^ T Ġ\ in Ġ\ math bb Ġ{ R } ^{ ( n - 1 ) Ġ\ times Ġd } $ Ġwith Ġa Ġtrain able Ġvector Ġ$\ math bf Ġ{ v } ^ T Ġ\ in Ġ\ math bb Ġ{ R } ^ d $ Ġ( initialized Ġrandomly ) Ġand Ġnormal izing Ġwith Ġa Ġsoft max . ĠThe Ġnormalized Ġalignment Ġcoefficients Ġare Ġfinally Ġused Ġto Ġcompute Ġthe Ġattention al Ġvector Ġ$\ math bf Ġ{ u } ^ T Ġ\ in Ġ\ math bb Ġ{ R } ^ d $ Ġas Ġa Ġweighted Ġsum Ġof Ġthe Ġfinal Ġrepresentations Ġ$\ math bf Ġ{\ hat { H }} ^ T $. </s> Note Ġthat Ġwe Ġtried Ġwith Ġmultiple Ġcontext Ġvectors , Ġi . e ., Ġwith Ġa Ġmatrix Ġ$\ math bf Ġ{ V } ^ T $ Ġinstead Ġof Ġa Ġvector Ġ$\ math bf Ġ{ v } ^ T $, Ġlike Ġin ĠB IB REF 22 , Ġbut Ġresults Ġwere Ġnot Ġconvincing , Ġeven Ġwhen Ġadding Ġa Ġregular ization Ġterm Ġto Ġthe Ġloss Ġto Ġfavor Ġdiversity Ġamong Ġthe Ġrows Ġof Ġ$\ math bf Ġ{ V } ^ T $. </s> Master Ġnode Ġskip Ġconnection . Ġ$\ math bf Ġ{ h } _ G ^ T Ġ\ in Ġ\ math bb Ġ{ R } ^{ 2 d } $ Ġis Ġobtained Ġby Ġconc aten ating Ġ$\ math bf Ġ{ u } ^ T $ Ġand Ġthe Ġfinal Ġmaster Ġnode Ġrepresentation . ĠThat Ġis , Ġthe Ġmaster Ġnode Ġvector Ġbypass es Ġthe Ġattention Ġmechanism . ĠThis Ġis Ġequivalent Ġto Ġa Ġskip Ġor Ġshortcut Ġconnection ĠB IB REF 23 . ĠThe Ġreason Ġbehind Ġthis Ġchoice Ġis Ġthat Ġwe Ġexpect Ġthe Ġspecial Ġdocument Ġnode Ġto Ġlearn Ġa Ġhigh - level Ġsummary Ġabout Ġthe Ġdocument , Ġsuch Ġas Ġits Ġsize , Ġvocabulary , Ġetc . Ġ( more Ġdetails Ġare Ġgiven Ġin Ġsubsection ĠSEC REF 30 ). ĠTherefore , Ġby Ġmaking Ġthe Ġmaster Ġnode Ġbypass Ġthe Ġattention Ġlayer , Ġwe Ġdirectly Ġinject Ġglobal Ġinformation Ġabout Ġthe Ġdocument Ġinto Ġits Ġfinal Ġrepresentation . </s> Multi - read out . ĠB IB REF 14 , Ġinspired Ġby ĠJump ing ĠKnowledge ĠNetworks ĠB IB REF 12 , Ġrecommend Ġto Ġnot Ġonly Ġuse Ġthe Ġfinal Ġrepresentations Ġwhen Ġperforming Ġread out , Ġbut Ġalso Ġthat Ġof Ġthe Ġearlier Ġsteps . ĠIndeed , Ġas Ġone Ġiter ates , Ġnode Ġfeatures Ġcapture Ġmore Ġand Ġmore Ġglobal Ġinformation . ĠHowever , Ġretaining Ġmore Ġlocal , Ġintermediary Ġinformation Ġmight Ġbe Ġuseful Ġtoo . ĠThus , Ġinstead Ġof Ġapplying Ġthe Ġread out Ġfunction Ġonly Ġto Ġ$ t = T $, Ġwe Ġapply Ġit Ġto Ġall Ġtime Ġsteps Ġand Ġconc aten ate Ġthe Ġresults , Ġfinally Ġobtaining Ġ$\ math bf Ġ{ h } _ G Ġ\ in Ġ\ math bb Ġ{ R } ^{ T Ġ\ times Ġ2 d } $ Ġ: </s> In Ġeffect , Ġwith Ġthis Ġmodification , Ġwe Ġtake Ġinto Ġaccount Ġfeatures Ġbased Ġon Ġinformation Ġaggreg ated Ġfrom Ġsubt rees Ġof Ġdifferent Ġheights Ġ( from Ġ1 Ġto Ġ$ T $ ), Ġcorresponding Ġto Ġlocal Ġand Ġglobal Ġfeatures . </s> Message ĠPassing ĠAttention Ġnetwork Ġfor ĠDocument Ġunderstanding Ġ( MP AD ) Ġ: :: ĠHier arch ical Ġvariants Ġof ĠMP AD </s> Through Ġthe Ġsuccessive ĠMP Ġiterations , Ġit Ġcould Ġbe Ġargued Ġthat ĠMP AD Ġimplicitly Ġcaptures Ġsome Ġsoft Ġnotion Ġof Ġthe Ġhierarchical Ġstructure Ġof Ġdocuments Ġ( words Ġ$\ right arrow Ġ$ Ġbig rams Ġ$\ right arrow Ġ$ Ġcompositions Ġof Ġbig rams , Ġetc .). ĠHowever , Ġit Ġmight Ġbe Ġbeneficial Ġto Ġexplicitly Ġcapture Ġdocument Ġhierarchy . ĠHier arch ical Ġarchitectures Ġhave Ġbrought Ġsignificant Ġimprovements Ġto Ġmany ĠN LP Ġtasks , Ġsuch Ġas Ġlanguage Ġmodeling Ġand Ġgeneration ĠB IB REF 24 , ĠB IB REF 25 , Ġsentiment Ġand Ġtopic Ġclassification ĠB IB REF 26 , ĠB IB REF 27 , Ġand Ġspoken Ġlanguage Ġunderstanding ĠB IB REF 28 , ĠB IB REF 29 . ĠInspired Ġby Ġthis Ġline Ġof Ġresearch , Ġwe Ġpropose Ġseveral Ġhierarchical Ġvariants Ġof ĠMP AD , Ġdetailed Ġin Ġwhat Ġfollows . ĠIn Ġall Ġof Ġthem , Ġwe Ġrepresent Ġeach Ġsentence Ġin Ġthe Ġdocument Ġas Ġa Ġword Ġco - occ urrence Ġnetwork , Ġand Ġobtain Ġan Ġembed ding Ġfor Ġit Ġby Ġapplying ĠMP AD Ġas Ġpreviously Ġdescribed . </s> MP AD - sent ence - att . ĠHere , Ġthe Ġsentence Ġembed d ings Ġare Ġsimply Ġcombined Ġthrough Ġself - att ention . </s> MP AD - cl ique . ĠIn Ġthis Ġvariant , Ġwe Ġbuild Ġa Ġcomplete Ġgraph Ġwhere Ġeach Ġnode Ġrepresents Ġa Ġsentence . ĠWe Ġthen Ġfeed Ġthat Ġgraph Ġto ĠMP AD , Ġwhere Ġthe Ġfeature Ġvectors Ġof Ġthe Ġnodes Ġare Ġinitialized Ġwith Ġthe Ġsentence Ġembed d ings Ġpreviously Ġobtained . </s> MP AD - path . ĠThis Ġvariant Ġis Ġsimilar Ġto Ġthe Ġcl ique Ġone , Ġexcept Ġthat Ġinstead Ġof Ġa Ġcomplete Ġgraph , Ġwe Ġbuild Ġa Ġpath Ġaccording Ġto Ġthe Ġnatural Ġflow Ġof Ġthe Ġtext . ĠThat Ġis , Ġtwo Ġnodes Ġare Ġlinked Ġby Ġa Ġdirected Ġedge Ġif Ġthe Ġtwo Ġsentences Ġthey Ġrepresent Ġfollow Ġeach Ġother Ġin Ġthe Ġdocument . </s> Exper iments Ġ: :: ĠDat as ets </s> We Ġevaluate Ġthe Ġquality Ġof Ġthe Ġdocument Ġembed d ings Ġlearned Ġby ĠMP AD Ġon Ġ10 Ġdocument Ġclassification Ġdatasets , Ġcovering Ġthe Ġtopic Ġidentification , Ġcoarse Ġand Ġfine Ġsentiment Ġanalysis Ġand Ġopinion Ġmining , Ġand Ġsubject ivity Ġdetection Ġtasks . ĠWe Ġbriefly Ġintroduce Ġthe Ġdatasets Ġnext . ĠTheir Ġstatistics Ġare Ġreported Ġin ĠTable ĠT AB REF 21 . </s> ( 1 ) ĠReuters . ĠThis Ġdataset Ġcontains Ġstories Ġcollected Ġfrom Ġthe ĠReuters Ġnews Ġagency Ġin Ġ1987 . ĠFollowing Ġcommon Ġpractice , Ġwe Ġused Ġthe ĠMod A pt e Ġsplit Ġand Ġconsidered Ġonly Ġthe Ġ10 Ġclasses Ġwith Ġthe Ġhighest Ġnumber Ġof Ġpositive Ġtraining Ġexamples . ĠWe Ġalso Ġremoved Ġdocuments Ġbelonging Ġto Ġmore Ġthan Ġone Ġclass Ġand Ġthen Ġclasses Ġleft Ġwith Ġno Ġdocument Ġ( 2 Ġclasses ). </s> ( 2 ) ĠBBC Sport ĠB IB REF 30 Ġcontains Ġdocuments Ġfrom Ġthe ĠBBC ĠSport Ġwebsite Ġcorresponding Ġto Ġ2004 - 2005 Ġsports Ġnews Ġarticles . </s> ( 3 ) ĠPol arity ĠB IB REF 31 Ġfeatures Ġpositive Ġand Ġnegative Ġlabeled Ġsnippets Ġfrom ĠR otten ĠTom atoes . </s> ( 4 ) ĠSubject ivity ĠB IB REF 32 Ġcontains Ġmovie Ġreview Ġsnippets Ġfrom ĠR otten ĠTom atoes Ġ( subject ive Ġsentences ), Ġand ĠInternet ĠMovie ĠDatabase Ġplot Ġsumm aries Ġ( object ive Ġsentences ). </s> ( 5 ) ĠMP Q A ĠB IB REF 33 Ġis Ġmade Ġof Ġpositive Ġand Ġnegative Ġphrases , Ġannot ated Ġas Ġpart Ġof Ġthe Ġsummer Ġ2002 ĠNR RC ĠWorkshop Ġon ĠMulti - Pers pect ive ĠQuestion ĠAn sw ering . </s> ( 6 ) ĠIM DB ĠB IB REF 34 Ġis Ġa Ġcollection Ġof Ġhighly Ġpolarized Ġmovie Ġreviews Ġfrom ĠIM DB Ġ( positive Ġand Ġnegative ). ĠThere Ġare Ġat Ġmost Ġ30 Ġreviews Ġfor Ġeach Ġmovie . </s> ( 7 ) ĠT REC ĠB IB REF 35 Ġconsists Ġof Ġquestions Ġthat Ġare Ġclassified Ġinto Ġ6 Ġdifferent Ġcategories . </s> ( 8 ) ĠS ST - 1 ĠB IB REF 36 Ġcontains Ġthe Ġsame Ġsnippets Ġas ĠPol arity . ĠThe Ġauthors Ġused Ġthe ĠStanford ĠPars er Ġto Ġparse Ġthe Ġsnippets Ġand Ġsplit Ġthem Ġinto Ġmultiple Ġsentences . ĠThey Ġthen Ġused ĠAmazon ĠMechanical ĠTurk Ġto Ġannot ate Ġthe Ġresulting Ġphrases Ġaccording Ġto Ġtheir Ġpol arity Ġ( very Ġnegative , Ġnegative , Ġneutral , Ġpositive , Ġvery Ġpositive ). </s> ( 9 ) ĠS ST - 2 ĠB IB REF 36 Ġis Ġthe Ġsame Ġas ĠS ST - 1 Ġbut Ġwith Ġneutral Ġreviews Ġremoved Ġand Ġsnippets Ġclassified Ġas Ġpositive Ġor Ġnegative . </s> ( 10 ) ĠYelp 2013 ĠB IB REF 26 Ġfeatures Ġreviews Ġobtained Ġfrom Ġthe Ġ2013 ĠYelp ĠDat as et ĠChallenge . </s> Exper iments Ġ: :: ĠBas elines </s> We Ġevaluate ĠMP AD Ġagainst Ġmultiple Ġstate - of - the - art Ġbaseline Ġmodels , Ġincluding Ġhierarchical Ġones , Ġto Ġenable Ġfair Ġcomparison Ġwith Ġthe Ġhierarchical ĠMP AD Ġvariants . </s> doc 2 vec ĠB IB REF 37 . ĠDoc 2 vec Ġ( or Ġparagraph Ġvector ) Ġis Ġan Ġextension Ġof Ġword 2 vec Ġthat Ġlearns Ġvectors Ġfor Ġdocuments Ġin Ġa Ġfully Ġun super vised Ġmanner . ĠDocument Ġembed d ings Ġare Ġthen Ġfed Ġto Ġa Ġlog istic Ġregression Ġclass ifier . </s> CNN ĠB IB REF 38 . ĠThe Ġconv olution al Ġneural Ġnetwork Ġarchitecture , Ġwell - known Ġin Ġcomputer Ġvision , Ġis Ġapplied Ġto Ġtext . ĠThere Ġis Ġone Ġspatial Ġdimension Ġand Ġthe Ġword Ġembed d ings Ġare Ġused Ġas Ġchannels Ġ( depth Ġdimensions ). </s> D AN ĠB IB REF 39 . ĠThe ĠDeep ĠA ver aging ĠNetwork Ġpasses Ġthe Ġun weight ed Ġaverage Ġof Ġthe Ġembed d ings Ġof Ġthe Ġinput Ġwords Ġthrough Ġmultiple Ġdense Ġlayers Ġand Ġa Ġfinal Ġsoft max . </s> Tree - L ST M ĠB IB REF 40 Ġis Ġa Ġgeneral ization Ġof Ġthe Ġstandard ĠL ST M Ġarchitecture Ġto Ġconstituency Ġand Ġdependency Ġparse Ġtrees . </s> DR NN ĠB IB REF 41 . ĠRec ursive Ġneural Ġnetworks Ġare Ġstacked Ġand Ġapplied Ġto Ġparse Ġtrees . </s> L ST MN ĠB IB REF 42 Ġis Ġan Ġextension Ġof Ġthe ĠL ST M Ġmodel Ġwhere Ġthe Ġmemory Ġcell Ġis Ġreplaced Ġby Ġa Ġmemory Ġnetwork Ġwhich Ġstores Ġword Ġrepresentations . </s> C - L ST M ĠB IB REF 43 Ġcombines Ġconv olution al Ġand Ġrecurrent Ġneural Ġnetworks . ĠThe Ġregion Ġembed d ings Ġprovided Ġby Ġa ĠCNN Ġare Ġfed Ġto Ġa ĠL ST M . </s> SP G K ĠB IB REF 44 Ġalso Ġmodels Ġdocuments Ġas Ġword Ġco - occ urrence Ġnetworks . ĠIt Ġcomp utes Ġa Ġgraph Ġkernel Ġthat Ġcompares Ġshortest Ġpaths Ġextracted Ġfrom Ġthe Ġword Ġco - occ urrence Ġnetworks Ġand Ġthen Ġuses Ġa ĠS VM Ġto Ġcategor ize Ġdocuments . </s> W MD ĠB IB REF 45 Ġis Ġan Ġapplication Ġof Ġthe Ġwell - known ĠEarth ĠM over 's ĠDistance Ġto Ġtext . ĠA Ġk - ne arest Ġneighbor Ġclass ifier Ġis Ġused . </s> S - W MD ĠB IB REF 46 Ġis Ġa Ġsupervised Ġextension Ġof Ġthe ĠWord ĠM over 's ĠDistance . </s> Sem antic - CNN ĠB IB REF 47 . ĠHere , Ġa ĠCNN Ġis Ġapplied Ġto Ġsemantic Ġunits Ġobtained Ġby Ġclust ering Ġwords Ġin Ġthe Ġembed ding Ġspace . </s> L ST M - GR NN ĠB IB REF 26 Ġis Ġa Ġhierarchical Ġmodel Ġwhere Ġsentence Ġembed d ings Ġare Ġobtained Ġwith Ġa ĠCNN Ġand Ġa ĠGR U - R NN Ġis Ġfed Ġthe Ġsentence Ġrepresentations Ġto Ġobtain Ġa Ġdocument Ġvector . </s> H N - ATT ĠB IB REF 27 Ġis Ġanother Ġhierarchical Ġmodel , Ġwhere Ġthe Ġsame Ġenc oder Ġarchitecture Ġ( a Ġbid irection al ĠGR U - R NN ) Ġis Ġused Ġfor Ġboth Ġsentences Ġand Ġdocuments , Ġwith Ġdifferent Ġparameters . ĠA Ġself - att ention Ġmechanism Ġis Ġapplied Ġto Ġthe ĠR NN Ġannotations Ġat Ġeach Ġlevel . </s> Exper iments Ġ: :: ĠModel Ġconfiguration Ġand Ġtraining </s> We Ġpre process Ġall Ġdatasets Ġusing Ġthe Ġcode Ġof ĠB IB REF 38 . ĠOn ĠYelp 2013 , Ġwe Ġalso Ġreplace Ġall Ġtokens Ġappearing Ġstrictly Ġless Ġthan Ġ6 Ġtimes Ġwith Ġa Ġspecial ĠUN K Ġtoken , Ġlike Ġin ĠB IB REF 27 . ĠWe Ġthen Ġbuild Ġa Ġdirected Ġword Ġco - occ urrence Ġnetwork Ġfrom Ġeach Ġdocument , Ġwith Ġa Ġwindow Ġof Ġsize Ġ2 . </s> We Ġuse Ġtwo ĠMP Ġiterations Ġ($ T $ = 2 ) Ġfor Ġthe Ġbasic ĠMP AD , Ġand Ġtwo ĠMP Ġiterations Ġat Ġeach Ġlevel , Ġfor Ġthe Ġhierarchical Ġvariants . ĠWe Ġset Ġ$ d $ Ġto Ġ64 , Ġexcept Ġon ĠIM DB Ġand ĠYelp Ġon Ġwhich Ġ$ d = 128 $, Ġand Ġuse Ġa Ġtwo - layer ĠML P . ĠThe Ġfinal Ġgraph Ġrepresentations Ġare Ġpassed Ġthrough Ġa Ġsoft max Ġfor Ġclassification . ĠWe Ġtrain ĠMP AD Ġin Ġan Ġend - to - end Ġfashion Ġby Ġminimizing Ġthe Ġcross - ent ropy Ġloss Ġfunction Ġwith Ġthe ĠAdam Ġoptim izer ĠB IB REF 48 Ġand Ġan Ġinitial Ġlearning Ġrate Ġof Ġ0 . 001 . </s> To Ġregulate Ġpotential Ġdifferences Ġin Ġmagnitude , Ġwe Ġapply Ġbatch Ġnormal ization Ġafter Ġconc aten ating Ġthe Ġfeature Ġvector Ġof Ġthe Ġmaster Ġnode Ġwith Ġthe Ġself - att ention al Ġvector , Ġthat Ġis , Ġafter Ġthe Ġskip Ġconnection Ġ( see Ġsubsection ĠSEC REF 16 ). ĠTo Ġprevent Ġover fitting , Ġwe Ġuse Ġdrop out ĠB IB REF 49 Ġwith Ġa Ġrate Ġof Ġ0 . 5 . ĠWe Ġselect Ġthe Ġbest Ġepoch , Ġcapped Ġat Ġ200 , Ġbased Ġon Ġthe Ġvalidation Ġaccuracy . ĠWhen Ġcross - valid ation Ġis Ġused Ġ( see Ġ3 rd Ġcolumn Ġof ĠTable ĠT AB REF 21 ), Ġwe Ġconstruct Ġa Ġvalidation Ġset Ġby Ġrandomly Ġsampling Ġ10 % Ġof Ġthe Ġtraining Ġset Ġof Ġeach Ġfold . </s> On Ġall Ġdatasets Ġexcept ĠYelp 2013 , Ġwe Ġuse Ġthe Ġpublicly Ġavailable Ġ300 - dimensional Ġpre - trained ĠGoogle ĠNews Ġvectors Ġ($ D $ = 300 ) ĠB IB REF 50 Ġto Ġinitialize Ġthe Ġnode Ġrepresentations Ġ$\ math bf Ġ{ H } ^ 0 $. ĠOn ĠYelp 2013 , Ġwe Ġfollow ĠB IB REF 27 Ġand Ġlearn Ġour Ġown Ġword Ġvectors Ġfrom Ġthe Ġtraining Ġand Ġvalidation Ġsets Ġwith Ġthe Ġg ens im Ġimplementation Ġof Ġword 2 vec ĠB IB REF 51 . </s> MP AD Ġwas Ġimplemented Ġin ĠPython Ġ3 . 6 Ġusing Ġthe ĠPy Tor ch Ġlibrary ĠB IB REF 52 . ĠAll Ġexperiments Ġwere Ġrun Ġon Ġa Ġsingle Ġmachine Ġconsisting Ġof Ġa Ġ3 . 4 ĠGHz ĠIntel ĠCore Ġi 7 ĠCPU Ġwith Ġ16 ĠGB Ġof ĠRAM Ġand Ġan ĠNV idia ĠGeForce ĠTitan ĠX p ĠGPU . </s> Results Ġand Ġabl ations Ġ: :: ĠResults </s> Exper imental Ġresults Ġare Ġshown Ġin ĠTable ĠT AB REF 28 . ĠFor Ġthe Ġbas elines , Ġthe Ġbest Ġscores Ġreported Ġin Ġeach Ġoriginal Ġpaper Ġare Ġshown . ĠMP AD Ġreaches Ġbest Ġperformance Ġon Ġ7 Ġout Ġof Ġ10 Ġdatasets , Ġand Ġis Ġclose Ġsecond Ġelsewhere . ĠMoreover , Ġthe Ġ7 Ġdatasets Ġon Ġwhich ĠMP AD Ġranks Ġfirst Ġwidely Ġdiffer Ġin Ġtraining Ġset Ġsize , Ġnumber Ġof Ġcategories , Ġand Ġprediction Ġtask Ġ( topic , Ġsentiment , Ġsubject ivity ), Ġwhich Ġindicates Ġthat ĠMP AD Ġcan Ġperform Ġwell Ġin Ġdifferent Ġsettings . </s> MP AD Ġvs . Ġhierarchical Ġvariants . ĠOn Ġ9 Ġdatasets Ġout Ġof Ġ10 , Ġone Ġor Ġmore Ġof Ġthe Ġhierarchical Ġvariants Ġoutper form Ġthe Ġvanilla ĠMP AD Ġarchitecture , Ġhighlighting Ġthe Ġbenefit Ġof Ġexplicitly Ġmodeling Ġthe Ġhierarchical Ġnature Ġof Ġdocuments . </s> However , Ġon ĠSubject ivity , Ġstandard ĠMP AD Ġoutper forms Ġall Ġhierarchical Ġvariants . ĠOn ĠT REC , Ġit Ġreaches Ġthe Ġsame Ġaccuracy . ĠWe Ġhypothes ize Ġthat Ġin Ġsome Ġcases , Ġusing Ġa Ġdifferent Ġgraph Ġto Ġseparately Ġencode Ġeach Ġsentence Ġmight Ġbe Ġworse Ġthan Ġusing Ġone Ġsingle Ġgraph Ġto Ġdirectly Ġencode Ġthe Ġdocument . ĠIndeed , Ġin Ġthe Ġsingle Ġdocument Ġgraph , Ġsome Ġwords Ġthat Ġnever Ġappear Ġin Ġthe Ġsame Ġsentence Ġcan Ġbe Ġconnected Ġthrough Ġcommon Ġneighbors , Ġas Ġwas Ġexplained Ġin Ġsubsection ĠSEC REF 7 . ĠSo , Ġthis Ġway , Ġsome Ġnotion Ġof Ġcross - sent ence Ġcontext Ġis Ġcaptured Ġwhile Ġlearning Ġrepresentations Ġof Ġwords , Ġbig rams , Ġetc . Ġat Ġeach ĠMP Ġiteration . ĠThis Ġcreates Ġbetter Ġinformed Ġrepresentations , Ġresulting Ġin Ġa Ġbetter Ġdocument Ġembed ding . ĠWith Ġthe Ġhierarchical Ġvariants , Ġon Ġthe Ġother Ġhand , Ġeach Ġsentence Ġvector Ġis Ġproduced Ġin Ġisolation , Ġwithout Ġany Ġcontextual Ġinformation Ġabout Ġthe Ġother Ġsentences Ġin Ġthe Ġdocument . ĠTherefore , Ġthe Ġfinal Ġsentence Ġembed d ings Ġmight Ġbe Ġof Ġlower Ġquality , Ġand Ġas Ġa Ġgroup Ġmight Ġalso Ġcontain Ġredundant / repe ated Ġinformation . ĠWhen Ġthe Ġsentence Ġvectors Ġare Ġfinally Ġcombined Ġinto Ġa Ġdocument Ġrepresentation , Ġit Ġis Ġtoo Ġlate Ġto Ġtake Ġcontext Ġinto Ġaccount . </s> Results Ġand Ġabl ations Ġ: :: ĠAb lation Ġstudies </s> To Ġunderstand Ġthe Ġimpact Ġof Ġsome Ġhyper param eters Ġon Ġperformance , Ġwe Ġconducted Ġadditional Ġexperiments Ġon Ġthe ĠReuters , ĠPol arity , Ġand ĠIM DB Ġdatasets , Ġwith Ġthe Ġnon - h ier arch ical Ġversion Ġof ĠMP AD . ĠResults Ġare Ġshown Ġin ĠTable ĠT AB REF 29 . </s> Number Ġof ĠMP Ġiterations . ĠFirst , Ġwe Ġvaried Ġthe Ġnumber Ġof Ġmessage Ġpassing Ġiterations Ġfrom Ġ1 Ġto Ġ4 . ĠWe Ġcan Ġclearly Ġsee Ġin ĠTable ĠT AB REF 29 Ġthat Ġhaving Ġmore Ġiterations Ġimproves Ġperformance . ĠWe Ġattribute Ġthis Ġto Ġthe Ġfact Ġthat Ġwe Ġare Ġreading Ġout Ġat Ġeach Ġiteration Ġfrom Ġ1 Ġto Ġ$ T $ Ġ( see ĠE q . ĠDIS PLAY _ FORM 18 ), Ġwhich Ġenables Ġthe Ġfinal Ġgraph Ġrepresentation Ġto Ġencode Ġa Ġmixture Ġof Ġlow - level Ġand Ġhigh - level Ġfeatures . ĠIndeed , Ġin Ġinitial Ġexperiments Ġinvolving Ġread out Ġat Ġ$ t $ =$ T $ Ġonly , Ġsetting Ġ$ T \ ge Ġ2 $ Ġwas Ġalways Ġdecreasing Ġperformance , Ġdespite Ġthe ĠGR U - based Ġupdates Ġ( E q . ĠDIS PLAY _ FORM 14 ). ĠThese Ġresults Ġwere Ġconsistent Ġwith Ġthat Ġof ĠB IB REF 53 Ġand ĠB IB REF 9 , Ġwho Ġboth Ġare Ġreading Ġout Ġonly Ġat Ġ$ t $ =$ T $ Ġtoo . ĠWe Ġhypothes ize Ġthat Ġnode Ġfeatures Ġat Ġ$ T \ ge Ġ2 $ Ġare Ġtoo Ġdiffuse Ġto Ġbe Ġentirely Ġrelied Ġupon Ġduring Ġread out . ĠMore Ġprecisely , Ġinitially Ġat Ġ$ t $ = 0 , Ġnode Ġrepresentations Ġcapture Ġinformation Ġabout Ġwords , Ġat Ġ$ t $ = 1 , Ġabout Ġtheir Ġ1 - hop Ġneighborhood Ġ( big rams ), Ġat Ġ$ t $ = 2 , Ġabout Ġcompositions Ġof Ġbig rams , Ġetc . ĠThus , Ġpretty Ġquickly , Ġnode Ġfeatures Ġbecome Ġgeneral Ġand Ġdiffuse . ĠIn Ġsuch Ġcases , Ġconsidering Ġalso Ġthe Ġlower - level , Ġmore Ġprecise Ġfeatures Ġof Ġthe Ġearlier Ġiterations Ġwhen Ġreading Ġout Ġmay Ġbe Ġnecessary . </s> Und irect ed Ġedges . ĠOn ĠReuters , Ġusing Ġan Ġund irect ed Ġgraph Ġleads Ġto Ġbetter Ġperformance , Ġwhile Ġon ĠPol arity Ġand ĠIM DB , Ġit Ġis Ġthe Ġopposite . ĠThis Ġcan Ġbe Ġexplained Ġby Ġthe Ġfact Ġthat ĠReuters Ġis Ġa Ġtopic Ġclassification Ġtask , Ġfor Ġwhich Ġthe Ġpresence Ġor Ġabsence Ġof Ġsome Ġpatterns Ġis Ġimportant , Ġbut Ġnot Ġnecessarily Ġthe Ġorder Ġin Ġwhich Ġthey Ġappear , Ġwhile ĠPol arity Ġand ĠIM DB Ġare Ġsentiment Ġanalysis Ġtasks . ĠTo Ġcapture Ġsentiment , Ġmodeling Ġword Ġorder Ġis Ġcrucial , Ġe . g ., Ġin Ġdetecting Ġneg ation . </s> No Ġmaster Ġnode . ĠRem oving Ġthe Ġmaster Ġnode Ġdeterior ates Ġperformance Ġacross Ġall Ġdatasets , Ġclearly Ġshowing Ġthe Ġvalue Ġof Ġhaving Ġsuch Ġa Ġnode . ĠWe Ġhypothes ize Ġthat Ġsince Ġthe Ġspecial Ġdocument Ġnode Ġis Ġconnected Ġto Ġall Ġother Ġnodes , Ġit Ġis Ġable Ġto Ġencode Ġduring Ġmessage Ġpassing Ġa Ġsummary Ġof Ġthe Ġdocument . </s> No Ġren ormal ization . ĠHere , Ġwe Ġdo Ġnot Ġuse Ġthe Ġren ormal ization Ġtrick Ġof ĠB IB REF 9 Ġduring ĠMP Ġ( see Ġsubsection ĠSEC REF 10 ). ĠThat Ġis , ĠE q . ĠDIS PLAY _ FORM 11 Ġbecomes Ġ$\ math bf Ġ{ M } ^{ t + 1 } Ġ= Ġ\ text sc Ġ{ ML P } ^{ t + 1 }\ big Ġ( \ math bf Ġ{ A }\ math bf Ġ{ H } ^{ t }\ big Ġ) $. ĠIn Ġother Ġwords , Ġinstead Ġof Ġcomputing Ġa Ġweighted Ġaverage Ġof Ġthe Ġincoming Ġneighbors ' Ġfeature Ġvectors , Ġwe Ġcompute Ġa Ġweighted Ġsum Ġof Ġthem . ĠUnlike Ġthe Ġmean , Ġwhich Ġcaptures Ġdistributions , Ġthe Ġsum Ġcaptures Ġstructural Ġinformation ĠB IB REF 14 . ĠAs Ġshown Ġin ĠTable ĠT AB REF 29 , Ġusing Ġsum Ġinstead Ġof Ġmean Ġdecreases Ġperformance Ġeverywhere , Ġsuggesting Ġthat Ġin Ġour Ġapplication , Ġcapturing Ġthe Ġdistribution Ġof Ġneighbor Ġrepresentations Ġis Ġmore Ġimportant Ġthat Ġcapturing Ġtheir Ġstructure . ĠWe Ġhypothes ize Ġthat Ġthis Ġis Ġthe Ġcase Ġbecause Ġstatistical Ġword Ġco - occ urrence Ġnetworks Ġtend Ġto Ġhave Ġsimilar Ġstructural Ġproperties , Ġregardless Ġof Ġthe Ġtopic , Ġpol arity , Ġsentiment , Ġetc . Ġof Ġthe Ġcorresponding Ġdocuments . </s> Neigh bors - only . ĠIn Ġthis Ġexperiment , Ġwe Ġreplaced Ġthe ĠGR U Ġcombine Ġfunction Ġ( see ĠE q . ĠDIS PLAY _ FORM 14 ) Ġwith Ġthe Ġidentity Ġfunction . ĠThat Ġis , Ġwe Ġsimply Ġhave Ġ$\ math bf Ġ{ H } ^{ t + 1 } $ =$ \ math bf Ġ{ M } ^{ t + 1 } $. ĠSince Ġ$\ math bf Ġ{ A } $ Ġhas Ġzero Ġdiagonal , Ġby Ġdoing Ġso , Ġwe Ġcompletely Ġignore Ġthe Ġprevious Ġfeature Ġof Ġthe Ġnode Ġitself Ġwhen Ġupdating Ġits Ġrepresentation . ĠThat Ġis , Ġthe Ġupdate Ġis Ġbased Ġentirely Ġon Ġits Ġneighbors . ĠExcept Ġon ĠReuters Ġ( almost Ġno Ġchange ), Ġperformance Ġalways Ġsuffers , Ġstressing Ġthe Ġneed Ġto Ġtake Ġinto Ġaccount Ġthe Ġroot Ġnode Ġduring Ġupdates , Ġnot Ġonly Ġits Ġneighborhood . </s> Related Ġwork </s> In Ġwhat Ġfollows , Ġwe Ġoffer Ġa Ġbrief Ġreview Ġof Ġrelevant Ġstudies , Ġranked Ġby Ġincreasing Ġorder Ġof Ġsimilarity Ġwith Ġour Ġwork . </s> BI B REF 9 , ĠB IB REF 54 , ĠB IB REF 11 , ĠB IB REF 10 Ġconduct Ġsome Ġnode Ġclassification Ġexperiments Ġon Ġcitation Ġnetworks , Ġwhere Ġnodes Ġare Ġscientific Ġpapers , Ġi . e ., Ġtextual Ġdata . ĠHowever , Ġtext Ġis Ġonly Ġused Ġto Ġderive Ġnode Ġfeature Ġvectors . ĠThe Ġexternal Ġgraph Ġstructure , Ġwhich Ġplays Ġa Ġcentral Ġrole Ġin Ġdetermining Ġnode Ġlabels , Ġis Ġcompletely Ġunrelated Ġto Ġtext . </s> On Ġthe Ġother Ġhand , ĠB IB REF 55 , ĠB IB REF 7 Ġexperiment Ġon Ġtraditional Ġdocument Ġclassification Ġtasks . ĠThey Ġboth Ġbuild Ġ$ k $ - ne arest Ġneighbor Ġsimilarity Ġgraphs Ġbased Ġon Ġthe ĠGa ussian Ġdiffusion Ġkernel . ĠMore Ġprecisely , ĠB IB REF 55 Ġbuild Ġone Ġsingle Ġgraph Ġwhere Ġnodes Ġare Ġdocuments Ġand Ġdistance Ġis Ġcomputed Ġin Ġthe ĠBo W Ġspace . ĠNode Ġfeatures Ġare Ġthen Ġused Ġfor Ġclassification . ĠCl oser Ġto Ġour Ġwork , ĠB IB REF 7 Ġrepresent Ġeach Ġdocument Ġas Ġa Ġgraph . ĠAll Ġdocument Ġgraphs Ġare Ġderived Ġfrom Ġthe Ġsame Ġunderlying Ġstructure . ĠOnly Ġnode Ġfeatures , Ġcorresponding Ġto Ġthe Ġentries Ġof Ġthe Ġdocuments ' ĠBo W Ġvectors , Ġvary . ĠThe Ġunderlying , Ġshared Ġstructure Ġis Ġthat Ġof Ġa Ġ$ k $ - NN Ġgraph Ġwhere Ġnodes Ġare Ġvocabulary Ġterms Ġand Ġsimilarity Ġis Ġthe Ġcos ine Ġof Ġthe Ġword Ġembed ding Ġvectors . ĠB IB REF 7 Ġthen Ġperform Ġgraph Ġclassification . ĠHowever Ġthey Ġfound Ġperformance Ġto Ġbe Ġlower Ġthan Ġthat Ġof Ġa Ġnaive ĠBay es Ġclass ifier . </s> BI B REF 56 Ġuse Ġa ĠG NN Ġfor Ġhierarchical Ġclassification Ġinto Ġa Ġlarge Ġtax onomy Ġof Ġtopics . ĠThis Ġtask Ġdiffers Ġfrom Ġtraditional Ġdocument Ġclassification . ĠThe Ġauthors Ġrepresent Ġdocuments Ġas Ġun weight ed , Ġund irect ed Ġword Ġco - occ urrence Ġnetworks Ġwith Ġword Ġembed d ings Ġas Ġnode Ġfeatures . ĠThey Ġthen Ġuse Ġthe Ġspatial ĠG NN Ġof ĠB IB REF 15 Ġto Ġperform Ġgraph Ġclassification . </s> The Ġwork Ġclosest Ġto Ġours Ġis Ġprobably Ġthat Ġof ĠB IB REF 53 . ĠThe Ġauthors Ġadopt Ġthe Ġsemi - super vised Ġnode Ġclassification Ġapproach Ġof ĠB IB REF 9 . ĠThey Ġbuild Ġone Ġsingle Ġund irect ed Ġgraph Ġfrom Ġthe Ġentire Ġdataset , Ġwith Ġboth Ġword Ġand Ġdocument Ġnodes . ĠDocument - word Ġedges Ġare Ġweighted Ġby ĠTF - ID F Ġand Ġword - word Ġedges Ġare Ġweighted Ġby Ġpoint wise Ġmutual Ġinformation Ġderived Ġfrom Ġco - occ urrence Ġwithin Ġa Ġsliding Ġwindow . ĠThere Ġare Ġno Ġdocument - document Ġedges . ĠThe ĠG NN Ġis Ġtrained Ġbased Ġon Ġthe Ġcross - ent ropy Ġloss Ġcomputed Ġonly Ġfor Ġthe Ġlabeled Ġnodes , Ġthat Ġis , Ġthe Ġdocuments Ġin Ġthe Ġtraining Ġset . ĠWhen Ġthe Ġfinal Ġnode Ġrepresentations Ġare Ġobtained , Ġone Ġcan Ġuse Ġthat Ġof Ġthe Ġtest Ġdocuments Ġto Ġclassify Ġthem Ġand Ġevaluate Ġprediction Ġperformance . </s> There Ġare Ġsignificant Ġdifferences Ġbetween ĠB IB REF 53 Ġand Ġour Ġwork . ĠFirst , Ġour Ġapproach Ġis Ġinduct ive , Ġnot Ġtrans duct ive . ĠIndeed , Ġwhile Ġthe Ġnode Ġclassification Ġapproach Ġof ĠB IB REF 53 Ġrequires Ġall Ġtest Ġdocuments Ġat Ġtraining Ġtime , Ġour Ġgraph Ġclassification Ġmodel Ġis Ġable Ġto Ġperform Ġinference Ġon Ġnew , Ġnever - seen Ġdocuments . ĠThe Ġdownside Ġof Ġrepresenting Ġdocuments Ġas Ġseparate Ġgraphs , Ġhowever , Ġis Ġthat Ġwe Ġlose Ġthe Ġability Ġto Ġcapture Ġcorpus - level Ġdependencies . ĠAlso , Ġour Ġdirected Ġgraphs Ġcapture Ġword Ġordering , Ġwhich Ġis Ġignored Ġby ĠB IB REF 53 . ĠFinally , Ġthe Ġapproach Ġof ĠB IB REF 53 Ġrequires Ġcomputing Ġthe ĠPM I Ġfor Ġevery Ġword Ġpair Ġin Ġthe Ġvocabulary , Ġwhich Ġmay Ġbe Ġprohib itive Ġon Ġdatasets Ġwith Ġvery Ġlarge Ġvoc ab ular ies . ĠOn Ġthe Ġother Ġhand , Ġthe Ġcomplexity Ġof ĠMP AD Ġdoes Ġnot Ġdepend Ġon Ġvocabulary Ġsize . </s> Conclusion </s> We Ġhave Ġproposed Ġan Ġapplication Ġof Ġthe Ġmessage Ġpassing Ġframework Ġto ĠN LP , Ġthe ĠMessage ĠPassing ĠAttention Ġnetwork Ġfor ĠDocument Ġunderstanding Ġ( MP AD ). ĠExper iments Ġconducted Ġon Ġ10 Ġstandard Ġtext Ġclassification Ġdatasets Ġshow Ġthat Ġour Ġarchitecture Ġis Ġcompetitive Ġwith Ġthe Ġstate - of - the - art . ĠBy Ġprocessing Ġweighted , Ġdirected Ġword Ġco - occ urrence Ġnetworks , ĠMP AD Ġis Ġsensitive Ġto Ġword Ġorder Ġand Ġword - word Ġrelationship Ġstrength . ĠTo Ġexplicitly Ġcapture Ġthe Ġhierarchical Ġstructure Ġof Ġdocuments , Ġwe Ġalso Ġpropose Ġthree Ġhierarchical Ġvariants Ġof ĠMP AD , Ġthat Ġwe Ġshow Ġbring Ġimprovements Ġover Ġthe Ġvanilla Ġarchitecture . </s> Acknowled gments </s> We Ġthank Ġthe ĠNV idia Ġcorporation Ġfor Ġthe Ġdonation Ġof Ġa ĠGPU Ġas Ġpart Ġof Ġtheir ĠGPU Ġgrant Ġprogram .