the basic model yields good performance for recognizing explicit discourse relations as well, which is comparable with previous best result (92.05% macro F1-score and 93.09% accuracy as reported in BIBREF11 )., full paragraph-level neural network model achieves the best macro-average F1-score of 48.82% in predicting implicit discourse relations, which outperforms previous neural tensor network models (e.g., BIBREF18 ) by more than 2 percents and outperforms the best previous system BIBREF19 by 1 percent., Then we also created ensemble models by applying majority voting to combine results of ten runs. From table 5 , each ensemble model obtains performance improvements compared with single model. The full model achieves performance boosting of (51.84 - 48.82 = 3.02) and (94.17 - 93.21 = 0.96) in macro F1-scores for predicting implicit and explicit discourse relations respectively. 
EXTRACTIVE
1 percent
EXTRACTIVE
The second row shows the performance of our basic paragraph-level model which predicts both implicit and explicit discourse relations in a paragraph. Compared to the variant system (the first row), the basic model further improved the classification performance on the first three implicit relations. Especially on the contingency relation, the classification performance was improved by another 1.42 percents. Moreover, the basic model yields good performance for recognizing explicit discourse relations as well, which is comparable with previous best result (92.05% macro F1-score and 93.09% accuracy as reported in BIBREF11 ). After untying parameters in the softmax prediction layer, implicit discourse relation classification performance was improved across all four relations, meanwhile, the explicit discourse relation classification performance was also improved. The CRF layer further improved implicit discourse relation recognition performance on the three small classes. In summary, our full paragraph-level neural network model achieves the best macro-average F1-score of 48.82% in predicting implicit discourse relations, which outperforms previous neural tensor network models (e.g., BIBREF18 ) by more than 2 percents and outperforms the best previous system BIBREF19 by 1 percent. As we explained in section 4.2, we ran our models for 10 times to obtain stable average performance. Then we also created ensemble models by applying majority voting to combine results of ten runs. From table 5 , each ensemble model obtains performance improvements compared with single model. The full model achieves performance boosting of (51.84 - 48.82 = 3.02) and (94.17 - 93.21 = 0.96) in macro F1-scores for predicting implicit and explicit discourse relations respectively. Furthermore, the ensemble model achieves the best performance for predicting both implicit and explicit discourse relations simultaneously.
After untying parameters in the softmax prediction layer, implicit discourse relation classification performance was improved across all four relations, meanwhile, the explicit discourse relation classification performance was also improved. The CRF layer further improved implicit discourse relation recognition performance on the three small classes. In summary, our full paragraph-level neural network model achieves the best macro-average F1-score of 48.82% in predicting implicit discourse relations, which outperforms previous neural tensor network models (e.g., BIBREF18 ) by more than 2 percents and outperforms the best previous system BIBREF19 by 1 percent.
1804.05918
Introduction </s> PD TB - style Ġdiscourse Ġrelations , Ġmostly Ġdefined Ġbetween Ġtwo Ġadjacent Ġtext Ġspans Ġ( i . e ., Ġdiscourse Ġunits , Ġeither Ġclauses Ġor Ġsentences ), Ġspecify Ġhow Ġtwo Ġdiscourse Ġunits Ġare Ġlogically Ġconnected Ġ( e . g ., Ġcausal , Ġcontrast ). ĠRecogn izing Ġdiscourse Ġrelations Ġis Ġone Ġcrucial Ġstep Ġin Ġdiscourse Ġanalysis Ġand Ġcan Ġbe Ġbeneficial Ġfor Ġmany Ġdownstream ĠN LP Ġapplications Ġsuch Ġas Ġinformation Ġextraction , Ġmachine Ġtranslation Ġand Ġnatural Ġlanguage Ġgeneration . </s> Comm only , Ġexplicit Ġdiscourse Ġrelations Ġwere Ġdistinguished Ġfrom Ġimplicit Ġones , Ġdepending Ġon Ġwhether Ġa Ġdiscourse Ġconnect ive Ġ( e . g ., ĠâĢ ľ because âĢ Ŀ Ġand ĠâĢ ľ after âĢ Ŀ ) Ġappears Ġbetween Ġtwo Ġdiscourse Ġunits ĠB IB REF 0 Ġ. ĠWhile Ġexplicit Ġdiscourse Ġrelation Ġdetection Ġcan Ġbe Ġframed Ġas Ġa Ġdiscourse Ġconnect ive Ġdis amb ig uation Ġproblem ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġand Ġhas Ġachieved Ġreasonable Ġperformance Ġ( F 1 Ġscore Ġ$ > $ Ġ90 %), Ġimplicit Ġdiscourse Ġrelations Ġhave Ġno Ġdiscourse Ġconnect ive Ġand Ġare Ġespecially Ġdifficult Ġto Ġidentify ĠB IB REF 3 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 4 Ġ. ĠTo Ġfill Ġthe Ġgap , Ġimplicit Ġdiscourse Ġrelation Ġprediction Ġhas Ġdrawn Ġsignificant Ġresearch Ġinterest Ġrecently Ġand Ġprogress Ġhas Ġbeen Ġmade ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġby Ġmodeling Ġcompos itional Ġmeanings Ġof Ġtwo Ġdiscourse Ġunits Ġand Ġexploiting Ġword Ġinteractions Ġbetween Ġdiscourse Ġunits Ġusing Ġneural Ġtens or Ġnetworks Ġor Ġattention Ġmechanisms Ġin Ġneural Ġnets . ĠHowever , Ġmost Ġof Ġexisting Ġapproaches Ġignore Ġwider Ġparagraph - level Ġcontexts Ġbeyond Ġthe Ġtwo Ġdiscourse Ġunits Ġthat Ġare Ġexamined Ġfor Ġpredicting Ġa Ġdiscourse Ġrelation Ġin Ġbetween . </s> To Ġfurther Ġimprove Ġimplicit Ġdiscourse Ġrelation Ġprediction , Ġwe Ġaim Ġto Ġimprove Ġdiscourse Ġunit Ġrepresentations Ġby Ġpositioning Ġa Ġdiscourse Ġunit Ġ( D U ) Ġin Ġits Ġwider Ġcontext Ġof Ġa Ġparagraph . ĠThe Ġkey Ġobservation Ġis Ġthat Ġsemantic Ġmeaning Ġof Ġa ĠDU Ġcan Ġnot Ġbe Ġinterpreted Ġindependently Ġfrom Ġthe Ġrest Ġof Ġthe Ġparagraph Ġthat Ġcontains Ġit , Ġor Ġindependently Ġfrom Ġthe Ġoverall Ġparagraph - level Ġdiscourse Ġstructure Ġthat Ġinvolve Ġthe ĠDU . ĠConsidering Ġthe Ġfollowing Ġparagraph Ġwith Ġfour Ġdiscourse Ġrelations , Ġone Ġrelation Ġbetween Ġeach Ġtwo Ġadjacent ĠD Us : </s> ( 1 ): Ġ[ The ĠButler , ĠWis ., Ġmanufacturer Ġwent Ġpublic Ġat Ġ$ 15 . 75 Ġa Ġshare Ġin ĠAugust Ġ1987 , ] Ġ$ _{ D U 1 } $ Ġand Ġ( Expl icit - Exp ansion ) Ġ[ Mr . ĠSim 's Ġgoal Ġthen Ġwas Ġa Ġ$ 29 Ġper - share Ġprice Ġby Ġ1992 .] Ġ$ _{ D U 2 } $ Ġ( Impl icit - Exp ansion ) Ġ[ Strong Ġearnings Ġgrowth Ġhelped Ġachieve Ġthat Ġprice Ġfar Ġahead Ġof Ġschedule , Ġin ĠAugust Ġ1988 .] Ġ$ _{ D U 3 } $ Ġ( Impl icit - Compar ison ) Ġ[ The Ġstock Ġhas Ġsince Ġsoftened , Ġtrading Ġaround Ġ$ 25 Ġa Ġshare Ġlast Ġweek Ġand Ġclosing Ġyesterday Ġat Ġ$ 23 Ġin Ġnational Ġover - the - counter Ġtrading .] Ġ$ _{ D U 4 } $ ĠBut Ġ( Expl icit - Compar ison ) Ġ[ Mr . ĠSim Ġhas Ġset Ġa Ġfresh Ġtarget Ġof Ġ$ 50 Ġa Ġshare Ġby Ġthe Ġend Ġof Ġreaching Ġthat Ġgoal .] Ġ$ _{ D U 5 } $ </s> Clearly , Ġeach ĠDU Ġis Ġan Ġintegral Ġpart Ġof Ġthe Ġparagraph Ġand Ġnot Ġindependent Ġfrom Ġother Ġunits . ĠFirst , Ġpredicting Ġa Ġdiscourse Ġrelation Ġmay Ġrequire Ġunderstanding Ġwider Ġparagraph - level Ġcontexts Ġbeyond Ġtwo Ġrelevant ĠD Us Ġand Ġthe Ġoverall Ġdiscourse Ġstructure Ġof Ġa Ġparagraph . ĠFor Ġexample , Ġthe Ġimplicit ĠâĢ ľ Compar ison âĢ Ŀ Ġdiscourse Ġrelation Ġbetween ĠDU 3 Ġand ĠDU 4 Ġis Ġdifficult Ġto Ġidentify Ġwithout Ġthe Ġbackground Ġinformation Ġ( the Ġhistory Ġof Ġper - share Ġprice ) Ġintroduced Ġin ĠDU 1 Ġand ĠDU 2 . ĠSecond , Ġa ĠDU Ġmay Ġbe Ġinvolved Ġin Ġmultiple Ġdiscourse Ġrelations Ġ( e . g ., ĠDU 4 Ġis Ġconnected Ġwith Ġboth ĠDU 3 Ġand ĠDU 5 Ġwith Ġa ĠâĢ ľ Compar ison âĢ Ŀ Ġrelation ), Ġtherefore Ġthe Ġpragmatic Ġmeaning Ġrepresentation Ġof Ġa ĠDU Ġshould Ġreflect Ġall Ġthe Ġdiscourse Ġrelations Ġthe Ġunit Ġwas Ġinvolved Ġin . ĠThird , Ġimplicit Ġdiscourse Ġrelation Ġprediction Ġshould Ġbenefit Ġfrom Ġmodeling Ġdiscourse Ġrelation Ġcontinuity Ġand Ġpatterns Ġin Ġa Ġparagraph Ġthat Ġinvolve Ġeasy - to - ident ify Ġexplicit Ġdiscourse Ġrelations Ġ( e . g ., ĠâĢ ľ Impl icit - Compar ison âĢ Ŀ Ġrelation Ġis Ġfollowed Ġby ĠâĢ ľ Expl icit - Compar ison âĢ Ŀ Ġin Ġthe Ġabove Ġexample ). </s> Following Ġthese Ġobservations , Ġwe Ġconstruct Ġa Ġneural Ġnet Ġmodel Ġto Ġprocess Ġa Ġparagraph Ġeach Ġtime Ġand Ġjointly Ġbuild Ġmeaning Ġrepresentations Ġfor Ġall ĠD Us Ġin Ġthe Ġparagraph . ĠThe Ġlearned ĠDU Ġrepresentations Ġare Ġused Ġto Ġpredict Ġa Ġsequence Ġof Ġdiscourse Ġrelations Ġin Ġthe Ġparagraph , Ġincluding Ġboth Ġimplicit Ġand Ġexplicit Ġrelations . ĠAlthough Ġexplicit Ġrelations Ġare Ġnot Ġour Ġfocus , Ġpredicting Ġan Ġexplicit Ġrelation Ġwill Ġhelp Ġto Ġreveal Ġthe Ġpragmatic Ġroles Ġof Ġits Ġtwo ĠD Us Ġand Ġreconstruct Ġtheir Ġrepresentations , Ġwhich Ġwill Ġfacilitate Ġpredicting Ġneighboring Ġimplicit Ġdiscourse Ġrelations Ġthat Ġinvolve Ġone Ġof Ġthe ĠD Us . </s> In Ġaddition , Ġwe Ġintroduce Ġtwo Ġnovel Ġdesigns Ġto Ġfurther Ġimprove Ġdiscourse Ġrelation Ġclassification Ġperformance Ġof Ġour Ġparagraph - level Ġneural Ġnet Ġmodel . ĠFirst , Ġprevious Ġwork Ġhas Ġindicated Ġthat Ġrecognizing Ġexplicit Ġand Ġimplicit Ġdiscourse Ġrelations Ġrequires Ġdifferent Ġstrategies , Ġwe Ġtherefore Ġunt ie Ġparameters Ġin Ġthe Ġdiscourse Ġrelation Ġprediction Ġlayer Ġof Ġthe Ġneural Ġnetworks Ġand Ġtrain Ġtwo Ġseparate Ġclass ifiers Ġfor Ġpredicting Ġexplicit Ġand Ġimplicit Ġdiscourse Ġrelations Ġrespectively . ĠThis Ġunique Ġdesign Ġhas Ġimproved Ġboth Ġimplicit Ġand Ġexplicit Ġdiscourse Ġrelation Ġidentification Ġperformance . ĠSecond , Ġwe Ġadd Ġa ĠCR F Ġlayer Ġon Ġtop Ġof Ġthe Ġdiscourse Ġrelation Ġprediction Ġlayer Ġto Ġfine - t une Ġa Ġsequence Ġof Ġpredicted Ġdiscourse Ġrelations Ġby Ġmodeling Ġdiscourse Ġrelation Ġcontinuity Ġand Ġpatterns Ġin Ġa Ġparagraph . </s> Exper imental Ġresults Ġshow Ġthat Ġthe Ġintuitive Ġparagraph - level Ġdiscourse Ġrelation Ġprediction Ġmodel Ġachieves Ġimproved Ġperformance Ġon ĠPD TB Ġfor Ġboth Ġimplicit Ġdiscourse Ġrelation Ġclassification Ġand Ġexplicit Ġdiscourse Ġrelation Ġclassification . </s> Impl icit ĠDisc ourse ĠRel ation ĠRecogn ition </s> Since Ġthe ĠPD TB ĠB IB REF 7 Ġcorpus Ġwas Ġcreated , Ġa Ġsurge Ġof Ġstudies ĠB IB REF 8 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġhave Ġbeen Ġconducted Ġfor Ġpredicting Ġdiscourse Ġrelations , Ġprimarily Ġfocusing Ġon Ġthe Ġchallenging Ġtask Ġof Ġimplicit Ġdiscourse Ġrelation Ġclassification Ġwhen Ġno Ġexplicit Ġdiscourse Ġconnect ive Ġphrase Ġwas Ġpresented . ĠEarly Ġstudies ĠB IB REF 11 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 12 Ġfocused Ġon Ġextracting Ġlinguistic Ġand Ġsemantic Ġfeatures Ġfrom Ġtwo Ġdiscourse Ġunits . ĠRecent Ġresearch ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġtried Ġto Ġmodel Ġcompos itional Ġmeanings Ġof Ġtwo Ġdiscourse Ġunits Ġby Ġexploiting Ġinteractions Ġbetween Ġwords Ġin Ġtwo Ġunits Ġwith Ġmore Ġand Ġmore Ġcomplicated Ġneural Ġnetwork Ġmodels , Ġincluding Ġthe Ġones Ġusing Ġneural Ġtens or ĠB IB REF 5 Ġ, ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġand Ġattention Ġmechanisms ĠB IB REF 6 Ġ, ĠB IB REF 19 Ġ, ĠB IB REF 20 Ġ. ĠAnother Ġtrend Ġis Ġto Ġalleviate Ġthe Ġshortage Ġof Ġannot ated Ġdata Ġby Ġleveraging Ġrelated Ġexternal Ġdata , Ġsuch Ġas Ġexplicit Ġdiscourse Ġrelations Ġin ĠPD TB ĠB IB REF 9 Ġ, ĠB IB REF 19 Ġ, ĠB IB REF 21 Ġand Ġunl abel ed Ġdata Ġobtained Ġelsewhere ĠB IB REF 12 Ġ, ĠB IB REF 19 Ġ, Ġoften Ġin Ġa Ġmulti - task Ġjoint Ġlearning Ġframework . </s> However , Ġnearly Ġall Ġthe Ġprevious Ġworks Ġassume Ġthat Ġa Ġpair Ġof Ġdiscourse Ġunits Ġis Ġindependent Ġfrom Ġits Ġwider Ġparagraph - level Ġcontexts Ġand Ġbuild Ġtheir Ġdiscourse Ġrelation Ġprediction Ġmodels Ġbased Ġon Ġonly Ġtwo Ġrelevant Ġdiscourse Ġunits . ĠIn Ġcontrast , Ġwe Ġmodel Ġinter - depend encies Ġof Ġdiscourse Ġunits Ġin Ġa Ġparagraph Ġwhen Ġbuilding Ġdiscourse Ġunit Ġrepresentations ; Ġin Ġaddition , Ġwe Ġmodel Ġglobal Ġcontinuity Ġand Ġpatterns Ġin Ġa Ġsequence Ġof Ġdiscourse Ġrelations , Ġincluding Ġboth Ġimplicit Ġand Ġexplicit Ġrelations . </s> H ier arch ical Ġneural Ġnetwork Ġmodels ĠB IB REF 22 Ġ, ĠB IB REF 23 Ġhave Ġbeen Ġapplied Ġto ĠR ST - style Ġdiscourse Ġparsing ĠB IB REF 24 Ġmainly Ġfor Ġthe Ġpurpose Ġof Ġgenerating Ġtext - level Ġhierarchical Ġdiscourse Ġstructures . ĠIn Ġcontrast , Ġwe Ġuse Ġhierarchical Ġneural Ġnetwork Ġmodels Ġto Ġbuild Ġcontext - aware Ġsentence Ġrepresentations Ġin Ġorder Ġto Ġimprove Ġimplicit Ġdiscourse Ġrelation Ġprediction . </s> Par agraph ĠEnc oding </s> Abstract ing Ġlatent Ġrepresentations Ġfrom Ġa Ġlong Ġsequence Ġof Ġwords , Ġsuch Ġas Ġa Ġparagraph , Ġis Ġa Ġchallenging Ġtask . ĠWhile Ġseveral Ġnovel Ġneural Ġnetwork Ġmodels ĠB IB REF 25 Ġ, ĠB IB REF 26 Ġhave Ġbeen Ġintroduced Ġin Ġrecent Ġyears Ġfor Ġencoding Ġa Ġparagraph , ĠRec urrent ĠNeural ĠNetwork Ġ( R NN )- based Ġmethods Ġremain Ġthe Ġmost Ġeffective Ġapproaches . ĠR NN s , Ġespecially Ġthe Ġlong - short Ġterm Ġmemory Ġ( L ST M ) ĠB IB REF 27 Ġmodels , Ġhave Ġbeen Ġwidely Ġused Ġto Ġencode Ġa Ġparagraph Ġfor Ġmachine Ġtranslation ĠB IB REF 28 Ġ, Ġdialogue Ġsystems ĠB IB REF 29 Ġand Ġtext Ġsummar ization ĠB IB REF 30 Ġbecause Ġof Ġits Ġability Ġin Ġmodeling Ġlong - distance Ġdependencies Ġbetween Ġwords . ĠIn Ġaddition , Ġamong Ġfour Ġtypical Ġpool ing Ġmethods Ġ( sum , Ġmean , Ġlast Ġand Ġmax ) Ġfor Ġcalculating Ġsentence Ġrepresentations Ġfrom ĠR NN - enc oded Ġhidden Ġstates Ġfor Ġindividual Ġwords , Ġmax - pool ing Ġalong Ġwith Ġbid irection al ĠL ST M Ġ( Bi - L ST M ) ĠB IB REF 31 Ġyields Ġthe Ġcurrent Ġbest Ġuniversal Ġsentence Ġrepresentation Ġmethod ĠB IB REF 32 Ġ. ĠWe Ġadopted Ġa Ġsimilar Ġneural Ġnetwork Ġarchitecture Ġfor Ġparagraph Ġencoding . </s> The ĠBasic ĠModel ĠArchitecture </s> Figure Ġ1 Ġillustrates Ġthe Ġoverall Ġarchitecture Ġof Ġthe Ġdiscourse - level Ġneural Ġnetwork Ġmodel Ġthat Ġconsists Ġof Ġtwo ĠBi - L ST M Ġlayers , Ġone Ġmax - pool ing Ġlayer Ġin Ġbetween Ġand Ġone Ġsoft max Ġprediction Ġlayer . ĠThe Ġinput Ġof Ġthe Ġneural Ġnetwork Ġmodel Ġis Ġa Ġparagraph Ġcontaining Ġa Ġsequence Ġof Ġdiscourse Ġunits , Ġwhile Ġthe Ġoutput Ġis Ġa Ġsequence Ġof Ġdiscourse Ġrelations Ġwith Ġone Ġrelation Ġbetween Ġeach Ġpair Ġof Ġadjacent Ġdiscourse Ġunits . </s> Given Ġthe Ġwords Ġsequence Ġof Ġone Ġparagraph Ġas Ġinput , Ġthe Ġlower ĠBi - L ST M Ġlayer Ġwill Ġread Ġthe Ġwhole Ġparagraph Ġand Ġcalculate Ġhidden Ġstates Ġas Ġword Ġrepresentations , Ġand Ġa Ġmax - pool ing Ġlayer Ġwill Ġbe Ġapplied Ġto Ġabstract Ġthe Ġrepresentation Ġof Ġeach Ġdiscourse Ġunit Ġbased Ġon Ġindividual Ġword Ġrepresentations . ĠThen Ġanother ĠBi - L ST M Ġlayer Ġwill Ġrun Ġover Ġthe Ġsequence Ġof Ġdiscourse Ġunit Ġrepresentations Ġand Ġcompute Ġnew Ġrepresentations Ġby Ġfurther Ġmodeling Ġsemantic Ġdependencies Ġbetween Ġdiscourse Ġunits Ġwithin Ġparagraph . ĠThe Ġfinal Ġsoft max Ġprediction Ġlayer Ġwill Ġconc aten ate Ġrepresentations Ġof Ġtwo Ġadjacent Ġdiscourse Ġunits Ġand Ġpredict Ġthe Ġdiscourse Ġrelation Ġbetween Ġthem . </s> Word ĠV ect ors Ġas ĠInput : ĠThe Ġinput Ġof Ġthe Ġparagraph - level Ġdiscourse Ġrelation Ġprediction Ġmodel Ġis Ġa Ġsequence Ġof Ġword Ġvectors , Ġone Ġvector Ġper Ġword Ġin Ġthe Ġparagraph . ĠIn Ġthis Ġwork , Ġwe Ġused Ġthe Ġpre - trained Ġ300 - dimension ĠGoogle ĠEnglish Ġword 2 vec Ġembed d ings . ĠFor Ġeach Ġword Ġthat Ġis Ġnot Ġin Ġthe Ġvocabulary Ġof ĠGoogle Ġword 2 vec , Ġwe Ġwill Ġrandomly Ġinitialize Ġa Ġvector Ġwith Ġeach Ġdimension Ġsampled Ġfrom Ġthe Ġrange Ġ$ [ - 0 . 25 , Ġ0 . 25 ] $ Ġ. ĠIn Ġaddition , Ġrecognizing Ġkey Ġentities Ġand Ġdiscourse Ġconnect ive Ġphrases Ġis Ġimportant Ġfor Ġdiscourse Ġrelation Ġrecognition , Ġtherefore , Ġwe Ġconc aten ate Ġthe Ġraw Ġword Ġembed d ings Ġwith Ġextra Ġlinguistic Ġfeatures , Ġspecifically Ġone - hot ĠPart - Of - Spe ech Ġtag Ġembed d ings Ġand Ġone - hot Ġnamed Ġentity Ġtag Ġembed d ings . </s> Building ĠDisc ourse ĠUnit ĠRepresent ations : ĠWe Ġaim Ġto Ġbuild Ġdiscourse Ġunit Ġ( D U ) Ġrepresentations Ġthat Ġsufficiently Ġleverage Ġcues Ġfor Ġdiscourse Ġrelation Ġprediction Ġfrom Ġparagraph - wide Ġcontexts , Ġincluding Ġthe Ġpreceding Ġand Ġfollowing Ġdiscourse Ġunits Ġin Ġa Ġparagraph . ĠTo Ġprocess Ġlong Ġparagraph - wide Ġcontexts , Ġwe Ġtake Ġa Ġbottom - up Ġtwo - level Ġabstraction Ġapproach Ġand Ġprogressively Ġgenerate Ġa Ġcompos itional Ġrepresentation Ġof Ġeach Ġword Ġfirst Ġ( low Ġlevel ) Ġand Ġthen Ġgenerate Ġa Ġcompos itional Ġrepresentation Ġof Ġeach Ġdiscourse Ġunit Ġ( high Ġlevel ), Ġwith Ġa Ġmax - pool ing Ġoperation Ġin Ġbetween . ĠAt Ġboth Ġword - level Ġand ĠDU - level , Ġwe Ġchoose ĠBi - L ST M Ġas Ġour Ġbasic Ġcomponent Ġfor Ġgenerating Ġcompos itional Ġrepresentations , Ġmainly Ġconsidering Ġits Ġcapability Ġto Ġcapture Ġlong - distance Ġdependencies Ġbetween Ġwords Ġ( disc ourse Ġunits ) Ġand Ġto Ġincorporate Ġinfluences Ġof Ġcontext Ġwords Ġ( disc ourse Ġunits ) Ġin Ġeach Ġside . </s> Given Ġa Ġvariable - length Ġwords Ġsequence Ġ$ X Ġ= Ġ( x _ 1 , x _ 2 ,... , x _ L ) $ Ġin Ġa Ġparagraph , Ġthe Ġword - level ĠBi - L ST M Ġwill Ġprocess Ġthe Ġinput Ġsequence Ġby Ġusing Ġtwo Ġseparate ĠL ST Ms , Ġone Ġprocess Ġthe Ġword Ġsequence Ġfrom Ġthe Ġleft Ġto Ġright Ġwhile Ġthe Ġother Ġfollows Ġthe Ġreversed Ġdirection . ĠTherefore , Ġat Ġeach Ġword Ġposition Ġ$ t $ Ġ, Ġwe Ġobtain Ġtwo Ġhidden Ġstates Ġ$\ over right arrow { h _ t }, Ġ\ over left arrow { h _ t } $ Ġ. ĠWe Ġconc aten ate Ġthem Ġto Ġget Ġthe Ġword Ġrepresentation Ġ$ h _ t Ġ= Ġ[ \ over right arrow { h _ t }, Ġ\ over left arrow { h _ t } ] $ Ġ. ĠThen Ġwe Ġapply Ġmax - pool ing Ġover Ġthe Ġsequence Ġof Ġword Ġrepresentations Ġfor Ġwords Ġin Ġa Ġdiscourse Ġunit Ġin Ġorder Ġto Ġget Ġthe Ġdiscourse Ġunit Ġembed ding : </s> $$ MP _{ D U } [ j ] Ġ= Ġ\ max Ġ_ { i = D U \ _ start } ^{ D U \ _ end } h _ i [ j ] \ quad Ġ\\ Ġwhere , Ġ1 Ġ\ le Ġj Ġ\ le Ġhidden \ _ node \ _ size $$ Ġ Ġ Ġ( E q . Ġ8 ) </s> Next , Ġthe ĠDU - level ĠBi - L ST M Ġwill Ġprocess Ġthe Ġsequence Ġof Ġdiscourse Ġunit Ġembed d ings Ġin Ġa Ġparagraph Ġand Ġgenerate Ġtwo Ġhidden Ġstates Ġ$\ over right arrow { h D U _ t } $ Ġand Ġ$\ over left arrow { h D U _ t } $ Ġat Ġeach Ġdiscourse Ġunit Ġposition . ĠWe Ġconc aten ate Ġthem Ġto Ġget Ġthe Ġdiscourse Ġunit Ġrepresentation Ġ$ h D U _ t Ġ= Ġ[ \ over right arrow { h D U _ t }, Ġ\ over left arrow { h D U _ t } ] $ Ġ. </s> The ĠSoft max ĠPrediction ĠLayer : ĠFinally , Ġwe Ġconc aten ate Ġtwo Ġadjacent Ġdiscourse Ġunit Ġrepresentations Ġ$ h D U _{ t - 1 } $ Ġand Ġ$ h D U _ t $ Ġand Ġpredict Ġthe Ġdiscourse Ġrelation Ġbetween Ġthem Ġusing Ġa Ġsoft max Ġfunction : </s> $$ y _{ t - 1 } Ġ= Ġsoft max ( W _ y * [ h D U _{ t - 1 }, h D U _ t ]+ b _ y ) $$ Ġ Ġ Ġ( E q . Ġ9 ) </s> Unt ie ĠParameters Ġin Ġthe ĠSoft max ĠPrediction ĠLayer Ġ( Impl icit Ġvs . ĠExplicit ) </s> Previous Ġwork ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 10 Ġhas Ġrevealed Ġthat Ġrecognizing Ġexplicit Ġvs . Ġimplicit Ġdiscourse Ġrelations Ġrequires Ġdifferent Ġstrategies . ĠNote Ġthat Ġin Ġthe ĠPD TB Ġdataset , Ġexplicit Ġdiscourse Ġrelations Ġwere Ġdistinguished Ġfrom Ġimplicit Ġones , Ġdepending Ġon Ġwhether Ġa Ġdiscourse Ġconnect ive Ġexists Ġbetween Ġtwo Ġdiscourse Ġunits . ĠTherefore , Ġexplicit Ġdiscourse Ġrelation Ġdetection Ġcan Ġbe Ġsimplified Ġas Ġa Ġdiscourse Ġconnect ive Ġphrase Ġdis amb ig uation Ġproblem ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ. ĠOn Ġthe Ġcontrary , Ġpredicting Ġan Ġimplicit Ġdiscourse Ġrelation Ġshould Ġrely Ġon Ġunderstanding Ġthe Ġoverall Ġcontents Ġof Ġits Ġtwo Ġdiscourse Ġunits ĠB IB REF 2 Ġ, ĠB IB REF 10 Ġ. </s> Considering Ġthe Ġdifferent Ġn atures Ġof Ġexplicit Ġvs . Ġimplicit Ġdiscourse Ġrelation Ġprediction , Ġwe Ġdecide Ġto Ġunt ie Ġparameters Ġat Ġthe Ġfinal Ġdiscourse Ġrelation Ġprediction Ġlayer Ġand Ġtrain Ġtwo Ġsoft max Ġclass ifiers , Ġas Ġillustrated Ġin ĠFigure Ġ2 Ġ. ĠThe Ġtwo Ġclass ifiers Ġhave Ġdifferent Ġsets Ġof Ġparameters , Ġwith Ġone Ġclass ifier Ġfor Ġonly Ġimplicit Ġdiscourse Ġrelations Ġand Ġthe Ġother Ġfor Ġonly Ġexplicit Ġdiscourse Ġrelations . </s> $$ y _{ t - 1 } Ġ= Ġ{\ left \ l brace Ġ\ begin { array }{ ll } Ġsoft max ( W _{ exp } [ h D U _{ t - 1 }, h D U _ t ]+ b _{ exp } ), & exp \\ Ġsoft max ( W _{ imp } [ h D U _{ t - 1 }, h D U _ t ]+ b _{ imp } ), & imp Ġ\ end { array }\ right .} $$ Ġ Ġ Ġ( E q . Ġ12 ) </s> The Ġloss Ġfunction Ġused Ġfor Ġthe Ġneural Ġnetwork Ġtraining Ġconsiders Ġloss Ġinduced Ġby Ġboth Ġimplicit Ġrelation Ġprediction Ġand Ġexplicit Ġrelation Ġprediction : </s> $$ L oss Ġ= ĠLoss _{ imp } Ġ+ Ġ\ alpha Ġ* L oss _{ exp } $$ Ġ Ġ Ġ( E q . Ġ13 ) </s> The Ġ$\ alpha Ġ$ Ġ, Ġin Ġthe Ġfull Ġsystem , Ġis Ġset Ġto Ġbe Ġ1 , Ġwhich Ġmeans Ġthat Ġminimizing Ġthe Ġloss Ġin Ġpredicting Ġeither Ġtype Ġof Ġdiscourse Ġrelations Ġis Ġequally Ġimportant . ĠIn Ġthe Ġevaluation , Ġwe Ġwill Ġalso Ġevaluate Ġa Ġsystem Ġvariant , Ġwhere Ġwe Ġwill Ġset Ġ$\ alpha Ġ= Ġ0 $ Ġ, Ġwhich Ġmeans Ġthat Ġthe Ġneural Ġnetwork Ġwill Ġnot Ġattempt Ġto Ġpredict Ġexplicit Ġdiscourse Ġrelations Ġand Ġimplicit Ġdiscourse Ġrelation Ġprediction Ġwill Ġnot Ġbe Ġinfluenced Ġby Ġpredicting Ġneighboring Ġexplicit Ġdiscourse Ġrelations . </s> Fine - t une ĠDisc ourse ĠRel ation ĠPred ictions ĠUsing Ġa ĠCR F ĠLayer </s> Data Ġanalysis Ġand Ġmany Ġlinguistic Ġstudies ĠB IB REF 11 Ġ, ĠB IB REF 33 Ġ, ĠB IB REF 34 Ġ, ĠB IB REF 35 Ġhave Ġrepeatedly Ġshown Ġthat Ġdiscourse Ġrelations Ġfeature Ġcontinuity Ġand Ġpatterns Ġ( e . g ., Ġa Ġtemporal Ġrelation Ġis Ġlikely Ġto Ġbe Ġfollowed Ġby Ġanother Ġtemporal Ġrelation ). ĠEspecially , ĠB IB REF 11 Ġfirst ly Ġreported Ġthat Ġpatterns Ġexist Ġbetween Ġimplicit Ġdiscourse Ġrelations Ġand Ġtheir Ġneighboring Ġexplicit Ġdiscourse Ġrelations . </s> Mot ivated Ġby Ġthese Ġobservations , Ġwe Ġaim Ġto Ġimprove Ġimplicit Ġdiscourse Ġrelation Ġdetection Ġby Ġmaking Ġuse Ġof Ġeasily Ġidentifiable Ġexplicit Ġdiscourse Ġrelations Ġand Ġtaking Ġinto Ġaccount Ġglobal Ġpatterns Ġof Ġdiscourse Ġrelation Ġdistributions . ĠSpecifically , Ġwe Ġadd Ġan Ġextra ĠCR F Ġlayer Ġat Ġthe Ġtop Ġof Ġthe Ġsoft max Ġprediction Ġlayer Ġ( shown Ġin Ġfigure Ġ3 Ġ) Ġto Ġfine - t une Ġpredicted Ġdiscourse Ġrelations Ġby Ġconsidering Ġtheir Ġinter - depend encies . </s> The ĠCond itional ĠRandom ĠFields ĠB IB REF 36 Ġ( CR F ) Ġlayer Ġupdates Ġa Ġstate Ġtransition Ġmatrix , Ġwhich Ġcan Ġeffectively Ġadjust Ġthe Ġcurrent Ġlabel Ġdepending Ġon Ġproceeding Ġand Ġfollowing Ġlabels . ĠBoth Ġtraining Ġand Ġdecoding Ġof Ġthe ĠCR F Ġlayer Ġcan Ġbe Ġsolved Ġefficiently Ġby Ġusing Ġthe ĠV iter bi Ġalgorithm . ĠWith Ġthe ĠCR F Ġlayer , Ġthe Ġmodel Ġjointly Ġassigns Ġa Ġsequence Ġof Ġdiscourse Ġrelations Ġbetween Ġeach Ġtwo Ġadjacent Ġdiscourse Ġunits Ġin Ġa Ġparagraph , Ġincluding Ġboth Ġimplicit Ġand Ġexplicit Ġrelations , Ġby Ġconsidering Ġrelevant Ġdiscourse Ġunit Ġrepresentations Ġas Ġwell Ġas Ġglobal Ġdiscourse Ġrelation Ġpatterns . </s> Dat as et Ġand ĠPre processing </s> The ĠPenn ĠDisc ourse ĠTree bank Ġ( PD TB ): ĠWe Ġexperimented Ġwith ĠPD TB Ġv 2 . 0 ĠB IB REF 7 Ġwhich Ġis Ġthe Ġlargest Ġannot ated Ġcorpus Ġcontaining Ġ36 k Ġdiscourse Ġrelations Ġin Ġ2 , 159 ĠWall ĠStreet ĠJournal Ġ( WS J ) Ġarticles . ĠIn Ġthis Ġwork , Ġwe Ġfocus Ġon Ġthe Ġtop - level Ġdiscourse Ġrelation Ġsenses Ġwhich Ġare Ġconsist Ġof Ġfour Ġmajor Ġsemantic Ġclasses : ĠComparison Ġ( Comp ), ĠCont ing ency Ġ( Cont ), ĠExpansion Ġ( Exp ) Ġand ĠTem poral Ġ( Temp ). ĠWe Ġfollowed Ġthe Ġsame ĠPD TB Ġsection Ġpartition ĠB IB REF 12 Ġas Ġprevious Ġwork Ġand Ġused Ġsections Ġ2 - 20 Ġas Ġtraining Ġset , Ġsections Ġ21 - 22 Ġas Ġtest Ġset , Ġand Ġsections Ġ0 - 1 Ġas Ġdevelopment Ġset . ĠTable Ġ1 Ġpresents Ġthe Ġdata Ġdistributions Ġwe Ġcollected Ġfrom ĠPD TB . </s> Pre processing : ĠThe ĠPD TB Ġdataset Ġdocuments Ġits Ġannotations Ġas Ġa Ġlist Ġof Ġdiscourse Ġrelations , Ġwith Ġeach Ġrelation Ġassociated Ġwith Ġits Ġtwo Ġdiscourse Ġunits . ĠTo Ġrecover Ġthe Ġparagraph Ġcontext Ġfor Ġa Ġdiscourse Ġrelation , Ġwe Ġmatch Ġcontents Ġof Ġits Ġtwo Ġannot ated Ġdiscourse Ġunits Ġwith Ġall Ġparagraphs Ġin Ġcorresponding Ġraw ĠWS J Ġarticle . ĠWhen Ġall Ġthe Ġmatching Ġwas Ġcompleted , Ġeach Ġparagraph Ġwas Ġsplit Ġinto Ġa Ġsequence Ġof Ġdiscourse Ġunits , Ġwith Ġone Ġdiscourse Ġrelation Ġ( impl icit Ġor Ġexplicit ) Ġbetween Ġeach Ġtwo Ġadjacent Ġdiscourse Ġunits . ĠFollowing Ġthis Ġmethod , Ġwe Ġobtained Ġ14 , 309 Ġparagraphs Ġin Ġtotal , Ġeach Ġcontains Ġ3 . 2 Ġdiscourse Ġunits Ġon Ġaverage . ĠTable Ġ2 Ġshows Ġthe Ġdistribution Ġof Ġparagraphs Ġbased Ġon Ġthe Ġnumber Ġof Ġdiscourse Ġunits Ġin Ġa Ġparagraph . </s> Parameter ĠSettings Ġand ĠModel ĠTraining </s> We Ġtuned Ġthe Ġparameters Ġbased Ġon Ġthe Ġbest Ġperformance Ġon Ġthe Ġdevelopment Ġset . ĠWe Ġfixed Ġthe Ġweights Ġof Ġword Ġembed d ings Ġduring Ġtraining . ĠAll Ġthe ĠL ST Ms Ġin Ġour Ġneural Ġnetwork Ġuse Ġthe Ġhidden Ġstate Ġsize Ġof Ġ300 . ĠTo Ġavoid Ġover fitting , Ġwe Ġapplied Ġdrop out ĠB IB REF 37 Ġwith Ġdrop out Ġratio Ġof Ġ0 . 5 Ġto Ġboth Ġinput Ġand Ġoutput Ġof ĠL ST M Ġlayers . ĠTo Ġprevent Ġthe Ġexploding Ġgradient Ġproblem Ġin Ġtraining ĠL ST Ms , Ġwe Ġadopt Ġgradient Ġclipping Ġwith Ġgradient ĠL 2 - norm Ġthreshold Ġof Ġ5 . 0 . ĠThese Ġparameters Ġremain Ġthe Ġsame Ġfor Ġall Ġour Ġproposed Ġmodels Ġas Ġwell Ġas Ġour Ġown Ġbaseline Ġmodels . </s> We Ġchose Ġthe Ġstandard Ġcross - ent ropy Ġloss Ġfunction Ġfor Ġtraining Ġour Ġneural Ġnetwork Ġmodel Ġand Ġadopted ĠAdam ĠB IB REF 38 Ġoptim izer Ġwith Ġthe Ġinitial Ġlearning Ġrate Ġof Ġ5 e - 4 Ġand Ġa Ġmini - batch Ġsize Ġof Ġ128 . ĠIf Ġone Ġinstance Ġis Ġannot ated Ġwith Ġtwo Ġlabels Ġ( 4 % Ġof Ġall Ġinstances ), Ġwe Ġuse Ġboth Ġof Ġthem Ġin Ġloss Ġcalculation Ġand Ġregard Ġthe Ġprediction Ġas Ġcorrect Ġif Ġmodel Ġpredicts Ġone Ġof Ġthe Ġannot ated Ġlabels . ĠAll Ġthe Ġproposed Ġmodels Ġwere Ġimplemented Ġwith ĠPy tor ch Ġand Ġconver ged Ġto Ġthe Ġbest Ġperformance Ġwithin Ġ20 - 40 Ġepoch s . </s> To Ġalleviate Ġthe Ġinfluence Ġof Ġrandom ness Ġin Ġneural Ġnetwork Ġmodel Ġtraining Ġand Ġobtain Ġstable Ġexperimental Ġresults , Ġwe Ġran Ġeach Ġof Ġthe Ġproposed Ġmodels Ġand Ġour Ġown Ġbaseline Ġmodels Ġten Ġtimes Ġand Ġreport Ġthe Ġaverage Ġperformance Ġof Ġeach Ġmodel Ġinstead Ġof Ġthe Ġbest Ġperformance Ġas Ġreported Ġin Ġmany Ġprevious Ġworks . </s> Bas eline ĠModels Ġand ĠSystems </s> We Ġcompare Ġthe Ġperformance Ġof Ġour Ġneural Ġnetwork Ġmodel Ġwith Ġseveral Ġrecent Ġdiscourse Ġrelation Ġrecognition Ġsystems Ġthat Ġonly Ġconsider Ġtwo Ġrelevant Ġdiscourse Ġunits . </s> BI B REF 12 Ġ: Ġimproves Ġimplicit Ġdiscourse Ġrelation Ġprediction Ġby Ġcreating Ġmore Ġtraining Ġinstances Ġfrom Ġthe ĠGig aw ord Ġcorpus Ġutilizing Ġexplicitly Ġmentioned Ġdiscourse Ġconnect ive Ġphrases . </s> BI B REF 5 Ġ: Ġa Ġg ated Ġrelevance Ġnetwork Ġ( GR N ) Ġmodel Ġwith Ġtens ors Ġto Ġcapture Ġsemantic Ġinteractions Ġbetween Ġwords Ġfrom Ġtwo Ġdiscourse Ġunits . </s> BI B REF 9 Ġ: Ġa Ġconv olution al Ġneural Ġnetwork Ġmodel Ġthat Ġlever ages Ġrelations Ġbetween Ġdifferent Ġstyles Ġof Ġdiscourse Ġrelations Ġannotations Ġ( PD TB Ġand ĠR ST ĠB IB REF 24 Ġ) Ġin Ġa Ġmulti - task Ġjoint Ġlearning Ġframework . </s> BI B REF 6 Ġ: Ġa Ġmulti - level Ġattention - over - att ention Ġmodel Ġto Ġdynamically Ġexploit Ġfeatures Ġfrom Ġtwo Ġdiscourse Ġunits Ġfor Ġrecognizing Ġan Ġimplicit Ġdiscourse Ġrelation . </s> BI B REF 21 Ġ: Ġa Ġnovel Ġpip el ined Ġadvers arial Ġframework Ġto Ġenable Ġan Ġadaptive Ġimitation Ġcompetition Ġbetween Ġthe Ġimplicit Ġnetwork Ġand Ġa Ġrival Ġfeature Ġdiscrim inator Ġwith Ġaccess Ġto Ġconnect ives . </s> BI B REF 18 Ġ: Ġa ĠSimple ĠWord ĠInter action ĠModel Ġ( SW IM ) Ġwith Ġtens ors Ġthat Ġcaptures Ġboth Ġlinear Ġand Ġquad r atic Ġrelations Ġbetween Ġwords Ġfrom Ġtwo Ġdiscourse Ġunits . </s> BI B REF 19 Ġ: Ġan Ġattention - based ĠL ST M Ġneural Ġnetwork Ġthat Ġlever ages Ġexplicit Ġdiscourse Ġrelations Ġin ĠPD TB Ġand Ġun annot ated Ġexternal Ġdata Ġin Ġa Ġmulti - task Ġjoint Ġlearning Ġframework . </s> E val uation ĠSettings </s> On Ġthe ĠPD TB Ġcorpus , Ġboth Ġbinary Ġclassification Ġand Ġmulti - way Ġclassification Ġsettings Ġare Ġcommonly Ġused Ġto Ġevaluate Ġthe Ġimplicit Ġdiscourse Ġrelation Ġrecognition Ġperformance . ĠWe Ġnoticed Ġthat Ġall Ġthe Ġrecent Ġworks Ġreport Ġclass - wise Ġimplicit Ġrelation Ġprediction Ġperformance Ġin Ġthe Ġbinary Ġclassification Ġsetting , Ġwhile Ġnone Ġof Ġthem Ġreport Ġdetailed Ġperformance Ġin Ġthe Ġmulti - way Ġclassification Ġsetting . ĠIn Ġthe Ġbinary Ġclassification Ġsetting , Ġseparate ĠâĢ ľ one - vers us - all âĢ Ŀ Ġbinary Ġclass ifiers Ġwere Ġtrained , Ġand Ġeach Ġclass ifier Ġis Ġto Ġidentify Ġone Ġclass Ġof Ġdiscourse Ġrelations . ĠAlthough Ġseparate Ġclass ifiers Ġare Ġgenerally Ġmore Ġflexible Ġin Ġcombating Ġwith Ġim balanced Ġdistributions Ġof Ġdiscourse Ġrelation Ġclasses Ġand Ġobtain Ġhigher Ġclass - wise Ġprediction Ġperformance , Ġone Ġpair Ġof Ġdiscourse Ġunits Ġmay Ġbe Ġtagged Ġwith Ġall Ġfour Ġdiscourse Ġrelations Ġwithout Ġproper Ġconflict Ġresolution . ĠTherefore , Ġthe Ġmulti - way Ġclassification Ġsetting Ġis Ġmore Ġappropriate Ġand Ġnatural Ġin Ġevaluating Ġa Ġpractical Ġend - to - end Ġdiscourse Ġparser , Ġand Ġwe Ġmainly Ġevaluate Ġour Ġproposed Ġmodels Ġusing Ġthe Ġfour - way Ġmulti - class Ġclassification Ġsetting . </s> Since Ġnone Ġof Ġthe Ġrecent Ġprevious Ġwork Ġreported Ġclass - wise Ġimplicit Ġrelation Ġclassification Ġperformance Ġin Ġthe Ġmulti - way Ġclassification Ġsetting , Ġfor Ġbetter Ġcomparisons , Ġwe Ġre - im ple mented Ġthe Ġneural Ġtens or Ġnetwork Ġarchitecture Ġ( so - called ĠSW IM Ġin ĠB IB REF 18 Ġ) Ġwhich Ġis Ġessentially Ġa ĠBi - L ST M Ġmodel Ġwith Ġtens ors Ġand Ġreport Ġits Ġdetailed Ġevaluation Ġresult Ġin Ġthe Ġmulti - way Ġclassification Ġsetting . ĠAs Ġanother Ġbaseline , Ġwe Ġreport Ġthe Ġperformance Ġof Ġa ĠBi - L ST M Ġmodel Ġwithout Ġtens ors Ġas Ġwell . ĠBoth Ġbaseline Ġmodels Ġtake Ġtwo Ġrelevant Ġdiscourse Ġunits Ġas Ġthe Ġonly Ġinput . </s> For Ġadditional Ġcomparisons , ĠWe Ġalso Ġreport Ġthe Ġperformance Ġof Ġour Ġproposed Ġmodels Ġin Ġthe Ġbinary Ġclassification Ġsetting . </s> Exper imental ĠResults </s> Multi - way ĠClassification : ĠThe Ġfirst Ġsection Ġof Ġtable Ġ3 Ġshows Ġmacro Ġaverage ĠF 1 - sc ores Ġand Ġaccur acies Ġof Ġprevious Ġworks . ĠThe Ġsecond Ġsection Ġof Ġtable Ġ3 Ġshows Ġthe Ġmulti - class Ġclassification Ġresults Ġof Ġour Ġimplemented Ġbaseline Ġsystems . ĠCons istent Ġwith Ġresults Ġof Ġprevious Ġworks , Ġneural Ġtens ors , Ġwhen Ġapplied Ġto ĠBi - L ST Ms , Ġimproved Ġimplicit Ġdiscourse Ġrelation Ġprediction Ġperformance . ĠHowever , Ġthe Ġperformance Ġon Ġthe Ġthree Ġsmall Ġclasses Ġ( Comp , ĠCont Ġand ĠTemp ) Ġremains Ġlow . </s> The Ġthird Ġsection Ġof Ġtable Ġ3 Ġshows Ġthe Ġmulti - class Ġclassification Ġresults Ġof Ġour Ġproposed Ġparagraph - level Ġneural Ġnetwork Ġmodels Ġthat Ġcapture Ġinter - depend encies Ġamong Ġdiscourse Ġunits . ĠThe Ġfirst Ġrow Ġshows Ġthe Ġperformance Ġof Ġa Ġvariant Ġof Ġour Ġbasic Ġmodel , Ġwhere Ġwe Ġonly Ġidentify Ġimplicit Ġrelations Ġand Ġignore Ġidentifying Ġexplicit Ġrelations Ġby Ġsetting Ġthe Ġ$\ alpha Ġ$ Ġin Ġequation Ġ( 5 ) Ġto Ġbe Ġ0 . ĠCompared Ġwith Ġthe Ġbaseline ĠBi - L ST M Ġmodel , Ġthe Ġonly Ġdifference Ġis Ġthat Ġthis Ġmodel Ġconsiders Ġparagraph - wide Ġcontexts Ġand Ġmodel Ġinter - depend encies Ġamong Ġdiscourse Ġunits Ġwhen Ġbuilding Ġrepresentation Ġfor Ġindividual ĠDU . ĠWe Ġcan Ġsee Ġthat Ġthis Ġmodel Ġhas Ġgreatly Ġimproved Ġimplicit Ġrelation Ġclassification Ġperformance Ġacross Ġall Ġthe Ġfour Ġrelations Ġand Ġimproved Ġthe Ġmacro - average ĠF 1 - score Ġby Ġover Ġ7 Ġper c ents . ĠIn Ġaddition , Ġcompared Ġwith Ġthe Ġbaseline ĠBi - L ST M Ġmodel Ġwith Ġtens or , Ġthis Ġmodel Ġimproved Ġimplicit Ġrelation Ġclassification Ġperformance Ġacross Ġthe Ġthree Ġsmall Ġclasses , Ġwith Ġclear Ġperformance Ġgains Ġof Ġaround Ġ2 Ġand Ġ8 Ġper c ents Ġon Ġcontingency Ġand Ġtemporal Ġrelations Ġrespectively , Ġand Ġoverall Ġimproved Ġthe Ġmacro - average ĠF 1 - score Ġby Ġ2 . 2 Ġper c ents . </s> The Ġsecond Ġrow Ġshows Ġthe Ġperformance Ġof Ġour Ġbasic Ġparagraph - level Ġmodel Ġwhich Ġpredicts Ġboth Ġimplicit Ġand Ġexplicit Ġdiscourse Ġrelations Ġin Ġa Ġparagraph . ĠCompared Ġto Ġthe Ġvariant Ġsystem Ġ( the Ġfirst Ġrow ), Ġthe Ġbasic Ġmodel Ġfurther Ġimproved Ġthe Ġclassification Ġperformance Ġon Ġthe Ġfirst Ġthree Ġimplicit Ġrelations . ĠEspecially Ġon Ġthe Ġcontingency Ġrelation , Ġthe Ġclassification Ġperformance Ġwas Ġimproved Ġby Ġanother Ġ1 . 42 Ġper c ents . ĠMoreover , Ġthe Ġbasic Ġmodel Ġyields Ġgood Ġperformance Ġfor Ġrecognizing Ġexplicit Ġdiscourse Ġrelations Ġas Ġwell , Ġwhich Ġis Ġcomparable Ġwith Ġprevious Ġbest Ġresult Ġ( 92 . 05 % Ġmacro ĠF 1 - score Ġand Ġ93 . 09 % Ġaccuracy Ġas Ġreported Ġin ĠB IB REF 11 Ġ). </s> After Ġunt ying Ġparameters Ġin Ġthe Ġsoft max Ġprediction Ġlayer , Ġimplicit Ġdiscourse Ġrelation Ġclassification Ġperformance Ġwas Ġimproved Ġacross Ġall Ġfour Ġrelations , Ġmeanwhile , Ġthe Ġexplicit Ġdiscourse Ġrelation Ġclassification Ġperformance Ġwas Ġalso Ġimproved . ĠThe ĠCR F Ġlayer Ġfurther Ġimproved Ġimplicit Ġdiscourse Ġrelation Ġrecognition Ġperformance Ġon Ġthe Ġthree Ġsmall Ġclasses . ĠIn Ġsummary , Ġour Ġfull Ġparagraph - level Ġneural Ġnetwork Ġmodel Ġachieves Ġthe Ġbest Ġmacro - average ĠF 1 - score Ġof Ġ48 . 82 % Ġin Ġpredicting Ġimplicit Ġdiscourse Ġrelations , Ġwhich Ġoutper forms Ġprevious Ġneural Ġtens or Ġnetwork Ġmodels Ġ( e . g ., ĠB IB REF 18 Ġ) Ġby Ġmore Ġthan Ġ2 Ġper c ents Ġand Ġoutper forms Ġthe Ġbest Ġprevious Ġsystem ĠB IB REF 19 Ġby Ġ1 Ġpercent . </s> B inary ĠClassification : ĠFrom Ġtable Ġ4 Ġ, Ġwe Ġcan Ġsee Ġthat Ġcompared Ġagainst Ġthe Ġbest Ġprevious Ġsystems , Ġour Ġparagraph - level Ġmodel Ġwith Ġunt ied Ġparameters Ġin Ġthe Ġprediction Ġlayer Ġachieves ĠF 1 - score Ġimprovements Ġof Ġ6 Ġpoints Ġon ĠComparison Ġand Ġ7 Ġpoints Ġon ĠTem poral , Ġwhich Ġdemonstrates Ġthat Ġparagraph - wide Ġcontexts Ġare Ġimportant Ġin Ġdetecting Ġminority Ġdiscourse Ġrelations . ĠNote Ġthat Ġthe ĠCR F Ġlayer Ġof Ġthe Ġmodel Ġis Ġnot Ġsuitable Ġfor Ġbinary Ġclassification . </s> En semble ĠModel </s> As Ġwe Ġexplained Ġin Ġsection Ġ4 . 2 , Ġwe Ġran Ġour Ġmodels Ġfor Ġ10 Ġtimes Ġto Ġobtain Ġstable Ġaverage Ġperformance . ĠThen Ġwe Ġalso Ġcreated Ġensemble Ġmodels Ġby Ġapplying Ġmajority Ġvoting Ġto Ġcombine Ġresults Ġof Ġten Ġruns . ĠFrom Ġtable Ġ5 Ġ, Ġeach Ġensemble Ġmodel Ġob tains Ġperformance Ġimprovements Ġcompared Ġwith Ġsingle Ġmodel . ĠThe Ġfull Ġmodel Ġachieves Ġperformance Ġboosting Ġof Ġ( 51 . 84 Ġ- Ġ48 . 82 Ġ= Ġ3 . 02 ) Ġand Ġ( 94 . 17 Ġ- Ġ93 . 21 Ġ= Ġ0 . 96 ) Ġin Ġmacro ĠF 1 - sc ores Ġfor Ġpredicting Ġimplicit Ġand Ġexplicit Ġdiscourse Ġrelations Ġrespectively . ĠFurthermore , Ġthe Ġensemble Ġmodel Ġachieves Ġthe Ġbest Ġperformance Ġfor Ġpredicting Ġboth Ġimplicit Ġand Ġexplicit Ġdiscourse Ġrelations Ġsimultaneously . </s> Imp act Ġof ĠPar agraph ĠLength </s> To Ġunderstand Ġthe Ġinfluence Ġof Ġparagraph Ġlengths Ġto Ġour Ġparagraph - level Ġmodels , Ġwe Ġdivide Ġparagraphs Ġin Ġthe ĠPD TB Ġtest Ġset Ġinto Ġseveral Ġsubs ets Ġbased Ġon Ġthe Ġnumber Ġof ĠD Us Ġin Ġa Ġparagraph , Ġand Ġthen Ġevaluate Ġour Ġproposed Ġmodels Ġon Ġeach Ġsubset Ġseparately . ĠFrom ĠFigure Ġ4 Ġ, Ġwe Ġcan Ġsee Ġthat Ġour Ġparagraph - level Ġmodels Ġ( the Ġlatter Ġthree ) Ġoverall Ġoutper form ĠDU - pair Ġbas elines Ġacross Ġall Ġthe Ġsubs ets . ĠAs Ġexpected , Ġthe Ġparagraph - level Ġmodels Ġachieve Ġclear Ġperformance Ġgains Ġon Ġlong Ġparagraphs Ġ( with Ġmore Ġthan Ġ5 ĠD Us ) Ġby Ġextensively Ġmodeling Ġmutual Ġinfluences Ġof ĠD Us Ġin Ġa Ġparagraph . ĠBut Ġsomewhat Ġsurprisingly , Ġthe Ġparagraph - level Ġmodels Ġachieve Ġnoticeable Ġperformance Ġgains Ġon Ġshort Ġparagraphs Ġ( with Ġ2 Ġor Ġ3 ĠD Us ) Ġas Ġwell . ĠWe Ġhypothes ize Ġthat Ġby Ġlearning Ġmore Ġappropriate Ġdiscourse - aware ĠDU Ġrepresentations Ġin Ġlong Ġparagraphs , Ġour Ġparagraph - level Ġmodels Ġreduce Ġbias Ġof Ġusing ĠDU Ġrepresentations Ġin Ġpredicting Ġdiscourse Ġrelations , Ġwhich Ġbenefits Ġdiscourse Ġrelation Ġprediction Ġin Ġshort Ġparagraphs Ġas Ġwell . </s> Example ĠAnalysis </s> For Ġthe Ġexample Ġ( Ġ" Impl icit ĠDisc ourse ĠRel ation ĠRecogn ition " Ġ), Ġthe Ġbaseline Ġneural Ġtens or Ġmodel Ġpredicted Ġboth Ġimplicit Ġrelations Ġwrongly Ġ( âĢ ľ Impl icit - Cont ing ency âĢ Ŀ Ġbetween ĠDU 2 Ġand ĠDU 3 ; ĠâĢ ľ Impl icit - Exp ansion âĢ Ŀ Ġbetween ĠDU 3 Ġand ĠDU 4 ), Ġwhile Ġour Ġparagraph - level Ġmodel Ġpredicted Ġall Ġthe Ġfour Ġdiscourse Ġrelations Ġcorrectly , Ġwhich Ġindicates Ġthat Ġparagraph - wide Ġcontexts Ġplay Ġa Ġkey Ġrole Ġin Ġimplicit Ġdiscourse Ġrelation Ġprediction . </s> For Ġanother Ġexample : </s> ( 2 ): Ġ[ Marsh all Ġcame Ġcl anking Ġin Ġlike ĠMar ley 's Ġghost Ġdragging Ġthose Ġchains Ġof Ġbrig ades Ġand Ġair Ġwings Ġand Ġlinks Ġwith ĠArab Ġdesp ots .] Ġ$ _{ D U 1 } $ Ġ( Impl icit - Tem poral ) Ġ[ He Ġwouldn 't Ġleave ] Ġ$ _{ D U 2 } $ Ġuntil Ġ( Expl icit - Tem poral ) Ġ[ Mr . ĠCheney Ġpromised Ġto Ġdo Ġwhatever Ġthe ĠPentagon Ġsystems Ġanalysts Ġtold Ġhim .] Ġ$ _{ D U 3 } $ </s> Our Ġbasic Ġparagraph - level Ġmodel Ġwrongly Ġpredicted Ġthe Ġimplicit Ġdiscourse Ġrelation Ġbetween ĠDU 1 Ġand ĠDU 2 Ġto Ġbe ĠâĢ ľ Impl icit - Compar ison âĢ Ŀ , Ġwithout Ġbeing Ġable Ġto Ġeffectively Ġuse Ġthe Ġsucceeding ĠâĢ ľ Expl icit - Tem poral âĢ Ŀ Ġrelation . ĠOn Ġthe Ġcontrary , Ġthe Ġfull Ġmodel Ġcorrected Ġthis Ġmistake Ġby Ġmodeling Ġdiscourse Ġrelation Ġpatterns Ġwith Ġthe ĠCR F Ġlayer . </s> Conclusion </s> We Ġhave Ġpresented Ġa Ġparagraph - level Ġneural Ġnetwork Ġmodel Ġthat Ġtakes Ġa Ġsequence Ġof Ġdiscourse Ġunits Ġas Ġinput , Ġmodels Ġinter - depend encies Ġbetween Ġdiscourse Ġunits Ġas Ġwell Ġas Ġdiscourse Ġrelation Ġcontinuity Ġand Ġpatterns , Ġand Ġpredicts Ġa Ġsequence Ġof Ġdiscourse Ġrelations Ġin Ġa Ġparagraph . ĠBy Ġbuilding Ġwider - context Ġinformed Ġdiscourse Ġunit Ġrepresentations Ġand Ġcapturing Ġthe Ġoverall Ġdiscourse Ġstructure , Ġthe Ġparagraph - level Ġneural Ġnetwork Ġmodel Ġoutper forms Ġthe Ġbest Ġprevious Ġmodels Ġfor Ġimplicit Ġdiscourse Ġrelation Ġrecognition Ġon Ġthe ĠPD TB Ġdataset . </s> Acknowled gments </s> We Ġacknowledge Ġthe Ġsupport Ġof ĠNVIDIA ĠCorporation Ġfor Ġtheir Ġdonation Ġof Ġone ĠGeForce ĠGTX ĠTIT AN ĠX ĠGPU Ġused Ġfor Ġthis Ġresearch .
Introduction PDTB-style discourse relations, mostly defined between two adjacent text spans (i.e., discourse units, either clauses or sentences), specify how two discourse units are logically connected (e.g., causal, contrast). Recognizing discourse relations is one crucial step in discourse analysis and can be beneficial for many downstream NLP applications such as information extraction, machine translation and natural language generation. Commonly, explicit discourse relations were distinguished from implicit ones, depending on whether a discourse connective (e.g., “because” and “after”) appears between two discourse units BIBREF0 . While explicit discourse relation detection can be framed as a discourse connective disambiguation problem BIBREF1 , BIBREF2 and has achieved reasonable performance (F1 score $>$ 90%), implicit discourse relations have no discourse connective and are especially difficult to identify BIBREF3 , BIBREF2 , BIBREF4 . To fill the gap, implicit discourse relation prediction has drawn significant research interest recently and progress has been made BIBREF5 , BIBREF6 by modeling compositional meanings of two discourse units and exploiting word interactions between discourse units using neural tensor networks or attention mechanisms in neural nets. However, most of existing approaches ignore wider paragraph-level contexts beyond the two discourse units that are examined for predicting a discourse relation in between. To further improve implicit discourse relation prediction, we aim to improve discourse unit representations by positioning a discourse unit (DU) in its wider context of a paragraph. The key observation is that semantic meaning of a DU can not be interpreted independently from the rest of the paragraph that contains it, or independently from the overall paragraph-level discourse structure that involve the DU. Considering the following paragraph with four discourse relations, one relation between each two adjacent DUs: (1): [The Butler, Wis., manufacturer went public at $15.75 a share in August 1987,] $_{DU1}$ and (Explicit-Expansion) [Mr. Sim's goal then was a $29 per-share price by 1992.] $_{DU2}$ (Implicit-Expansion) [Strong earnings growth helped achieve that price far ahead of schedule, in August 1988.] $_{DU3}$ (Implicit-Comparison) [The stock has since softened, trading around $25 a share last week and closing yesterday at $23 in national over-the-counter trading.] $_{DU4}$ But (Explicit-Comparison) [Mr. Sim has set a fresh target of $50 a share by the end of reaching that goal.] $_{DU5}$ Clearly, each DU is an integral part of the paragraph and not independent from other units. First, predicting a discourse relation may require understanding wider paragraph-level contexts beyond two relevant DUs and the overall discourse structure of a paragraph. For example, the implicit “Comparison” discourse relation between DU3 and DU4 is difficult to identify without the background information (the history of per-share price) introduced in DU1 and DU2. Second, a DU may be involved in multiple discourse relations (e.g., DU4 is connected with both DU3 and DU5 with a “Comparison” relation), therefore the pragmatic meaning representation of a DU should reflect all the discourse relations the unit was involved in. Third, implicit discourse relation prediction should benefit from modeling discourse relation continuity and patterns in a paragraph that involve easy-to-identify explicit discourse relations (e.g., “Implicit-Comparison” relation is followed by “Explicit-Comparison” in the above example). Following these observations, we construct a neural net model to process a paragraph each time and jointly build meaning representations for all DUs in the paragraph. The learned DU representations are used to predict a sequence of discourse relations in the paragraph, including both implicit and explicit relations. Although explicit relations are not our focus, predicting an explicit relation will help to reveal the pragmatic roles of its two DUs and reconstruct their representations, which will facilitate predicting neighboring implicit discourse relations that involve one of the DUs. In addition, we introduce two novel designs to further improve discourse relation classification performance of our paragraph-level neural net model. First, previous work has indicated that recognizing explicit and implicit discourse relations requires different strategies, we therefore untie parameters in the discourse relation prediction layer of the neural networks and train two separate classifiers for predicting explicit and implicit discourse relations respectively. This unique design has improved both implicit and explicit discourse relation identification performance. Second, we add a CRF layer on top of the discourse relation prediction layer to fine-tune a sequence of predicted discourse relations by modeling discourse relation continuity and patterns in a paragraph. Experimental results show that the intuitive paragraph-level discourse relation prediction model achieves improved performance on PDTB for both implicit discourse relation classification and explicit discourse relation classification. Implicit Discourse Relation Recognition Since the PDTB BIBREF7 corpus was created, a surge of studies BIBREF8 , BIBREF3 , BIBREF9 , BIBREF10 have been conducted for predicting discourse relations, primarily focusing on the challenging task of implicit discourse relation classification when no explicit discourse connective phrase was presented. Early studies BIBREF11 , BIBREF3 , BIBREF2 , BIBREF12 focused on extracting linguistic and semantic features from two discourse units. Recent research BIBREF13 , BIBREF14 , BIBREF15 , BIBREF16 tried to model compositional meanings of two discourse units by exploiting interactions between words in two units with more and more complicated neural network models, including the ones using neural tensor BIBREF5 , BIBREF17 , BIBREF18 and attention mechanisms BIBREF6 , BIBREF19 , BIBREF20 . Another trend is to alleviate the shortage of annotated data by leveraging related external data, such as explicit discourse relations in PDTB BIBREF9 , BIBREF19 , BIBREF21 and unlabeled data obtained elsewhere BIBREF12 , BIBREF19 , often in a multi-task joint learning framework. However, nearly all the previous works assume that a pair of discourse units is independent from its wider paragraph-level contexts and build their discourse relation prediction models based on only two relevant discourse units. In contrast, we model inter-dependencies of discourse units in a paragraph when building discourse unit representations; in addition, we model global continuity and patterns in a sequence of discourse relations, including both implicit and explicit relations. Hierarchical neural network models BIBREF22 , BIBREF23 have been applied to RST-style discourse parsing BIBREF24 mainly for the purpose of generating text-level hierarchical discourse structures. In contrast, we use hierarchical neural network models to build context-aware sentence representations in order to improve implicit discourse relation prediction. Paragraph Encoding Abstracting latent representations from a long sequence of words, such as a paragraph, is a challenging task. While several novel neural network models BIBREF25 , BIBREF26 have been introduced in recent years for encoding a paragraph, Recurrent Neural Network (RNN)-based methods remain the most effective approaches. RNNs, especially the long-short term memory (LSTM) BIBREF27 models, have been widely used to encode a paragraph for machine translation BIBREF28 , dialogue systems BIBREF29 and text summarization BIBREF30 because of its ability in modeling long-distance dependencies between words. In addition, among four typical pooling methods (sum, mean, last and max) for calculating sentence representations from RNN-encoded hidden states for individual words, max-pooling along with bidirectional LSTM (Bi-LSTM) BIBREF31 yields the current best universal sentence representation method BIBREF32 . We adopted a similar neural network architecture for paragraph encoding. The Basic Model Architecture Figure 1 illustrates the overall architecture of the discourse-level neural network model that consists of two Bi-LSTM layers, one max-pooling layer in between and one softmax prediction layer. The input of the neural network model is a paragraph containing a sequence of discourse units, while the output is a sequence of discourse relations with one relation between each pair of adjacent discourse units. Given the words sequence of one paragraph as input, the lower Bi-LSTM layer will read the whole paragraph and calculate hidden states as word representations, and a max-pooling layer will be applied to abstract the representation of each discourse unit based on individual word representations. Then another Bi-LSTM layer will run over the sequence of discourse unit representations and compute new representations by further modeling semantic dependencies between discourse units within paragraph. The final softmax prediction layer will concatenate representations of two adjacent discourse units and predict the discourse relation between them. Word Vectors as Input: The input of the paragraph-level discourse relation prediction model is a sequence of word vectors, one vector per word in the paragraph. In this work, we used the pre-trained 300-dimension Google English word2vec embeddings. For each word that is not in the vocabulary of Google word2vec, we will randomly initialize a vector with each dimension sampled from the range $[-0.25, 0.25]$ . In addition, recognizing key entities and discourse connective phrases is important for discourse relation recognition, therefore, we concatenate the raw word embeddings with extra linguistic features, specifically one-hot Part-Of-Speech tag embeddings and one-hot named entity tag embeddings. Building Discourse Unit Representations: We aim to build discourse unit (DU) representations that sufficiently leverage cues for discourse relation prediction from paragraph-wide contexts, including the preceding and following discourse units in a paragraph. To process long paragraph-wide contexts, we take a bottom-up two-level abstraction approach and progressively generate a compositional representation of each word first (low level) and then generate a compositional representation of each discourse unit (high level), with a max-pooling operation in between. At both word-level and DU-level, we choose Bi-LSTM as our basic component for generating compositional representations, mainly considering its capability to capture long-distance dependencies between words (discourse units) and to incorporate influences of context words (discourse units) in each side. Given a variable-length words sequence $X = (x_1,x_2,...,x_L)$ in a paragraph, the word-level Bi-LSTM will process the input sequence by using two separate LSTMs, one process the word sequence from the left to right while the other follows the reversed direction. Therefore, at each word position $t$ , we obtain two hidden states $\overrightarrow{h_t}, \overleftarrow{h_t}$ . We concatenate them to get the word representation $h_t = [\overrightarrow{h_t}, \overleftarrow{h_t}]$ . Then we apply max-pooling over the sequence of word representations for words in a discourse unit in order to get the discourse unit embedding: $$MP_{DU}[j] = \max _{i=DU\_start}^{DU\_end}h_i[j]\quad \\ where, 1 \le j \le hidden\_node\_size$$   (Eq. 8) Next, the DU-level Bi-LSTM will process the sequence of discourse unit embeddings in a paragraph and generate two hidden states $\overrightarrow{hDU_t}$ and $\overleftarrow{hDU_t}$ at each discourse unit position. We concatenate them to get the discourse unit representation $hDU_t = [\overrightarrow{hDU_t}, \overleftarrow{hDU_t}]$ . The Softmax Prediction Layer: Finally, we concatenate two adjacent discourse unit representations $hDU_{t-1}$ and $hDU_t$ and predict the discourse relation between them using a softmax function: $$y_{t-1} = softmax(W_y*[hDU_{t-1},hDU_t]+b_y)$$   (Eq. 9) Untie Parameters in the Softmax Prediction Layer (Implicit vs. Explicit) Previous work BIBREF1 , BIBREF2 , BIBREF10 has revealed that recognizing explicit vs. implicit discourse relations requires different strategies. Note that in the PDTB dataset, explicit discourse relations were distinguished from implicit ones, depending on whether a discourse connective exists between two discourse units. Therefore, explicit discourse relation detection can be simplified as a discourse connective phrase disambiguation problem BIBREF1 , BIBREF2 . On the contrary, predicting an implicit discourse relation should rely on understanding the overall contents of its two discourse units BIBREF2 , BIBREF10 . Considering the different natures of explicit vs. implicit discourse relation prediction, we decide to untie parameters at the final discourse relation prediction layer and train two softmax classifiers, as illustrated in Figure 2 . The two classifiers have different sets of parameters, with one classifier for only implicit discourse relations and the other for only explicit discourse relations. $$y_{t-1} = {\left\lbrace \begin{array}{ll} softmax(W_{exp}[hDU_{t-1},hDU_t]+b_{exp}),&exp\\ softmax(W_{imp}[hDU_{t-1},hDU_t]+b_{imp}),&imp \end{array}\right.}$$   (Eq. 12) The loss function used for the neural network training considers loss induced by both implicit relation prediction and explicit relation prediction: $$Loss = Loss_{imp} + \alpha *Loss_{exp}$$   (Eq. 13) The $\alpha $ , in the full system, is set to be 1, which means that minimizing the loss in predicting either type of discourse relations is equally important. In the evaluation, we will also evaluate a system variant, where we will set $\alpha = 0$ , which means that the neural network will not attempt to predict explicit discourse relations and implicit discourse relation prediction will not be influenced by predicting neighboring explicit discourse relations. Fine-tune Discourse Relation Predictions Using a CRF Layer Data analysis and many linguistic studies BIBREF11 , BIBREF33 , BIBREF34 , BIBREF35 have repeatedly shown that discourse relations feature continuity and patterns (e.g., a temporal relation is likely to be followed by another temporal relation). Especially, BIBREF11 firstly reported that patterns exist between implicit discourse relations and their neighboring explicit discourse relations. Motivated by these observations, we aim to improve implicit discourse relation detection by making use of easily identifiable explicit discourse relations and taking into account global patterns of discourse relation distributions. Specifically, we add an extra CRF layer at the top of the softmax prediction layer (shown in figure 3 ) to fine-tune predicted discourse relations by considering their inter-dependencies. The Conditional Random Fields BIBREF36 (CRF) layer updates a state transition matrix, which can effectively adjust the current label depending on proceeding and following labels. Both training and decoding of the CRF layer can be solved efficiently by using the Viterbi algorithm. With the CRF layer, the model jointly assigns a sequence of discourse relations between each two adjacent discourse units in a paragraph, including both implicit and explicit relations, by considering relevant discourse unit representations as well as global discourse relation patterns. Dataset and Preprocessing The Penn Discourse Treebank (PDTB): We experimented with PDTB v2.0 BIBREF7 which is the largest annotated corpus containing 36k discourse relations in 2,159 Wall Street Journal (WSJ) articles. In this work, we focus on the top-level discourse relation senses which are consist of four major semantic classes: Comparison (Comp), Contingency (Cont), Expansion (Exp) and Temporal (Temp). We followed the same PDTB section partition BIBREF12 as previous work and used sections 2-20 as training set, sections 21-22 as test set, and sections 0-1 as development set. Table 1 presents the data distributions we collected from PDTB. Preprocessing: The PDTB dataset documents its annotations as a list of discourse relations, with each relation associated with its two discourse units. To recover the paragraph context for a discourse relation, we match contents of its two annotated discourse units with all paragraphs in corresponding raw WSJ article. When all the matching was completed, each paragraph was split into a sequence of discourse units, with one discourse relation (implicit or explicit) between each two adjacent discourse units. Following this method, we obtained 14,309 paragraphs in total, each contains 3.2 discourse units on average. Table 2 shows the distribution of paragraphs based on the number of discourse units in a paragraph. Parameter Settings and Model Training We tuned the parameters based on the best performance on the development set. We fixed the weights of word embeddings during training. All the LSTMs in our neural network use the hidden state size of 300. To avoid overfitting, we applied dropout BIBREF37 with dropout ratio of 0.5 to both input and output of LSTM layers. To prevent the exploding gradient problem in training LSTMs, we adopt gradient clipping with gradient L2-norm threshold of 5.0. These parameters remain the same for all our proposed models as well as our own baseline models. We chose the standard cross-entropy loss function for training our neural network model and adopted Adam BIBREF38 optimizer with the initial learning rate of 5e-4 and a mini-batch size of 128. If one instance is annotated with two labels (4% of all instances), we use both of them in loss calculation and regard the prediction as correct if model predicts one of the annotated labels. All the proposed models were implemented with Pytorch and converged to the best performance within 20-40 epochs. To alleviate the influence of randomness in neural network model training and obtain stable experimental results, we ran each of the proposed models and our own baseline models ten times and report the average performance of each model instead of the best performance as reported in many previous works. Baseline Models and Systems We compare the performance of our neural network model with several recent discourse relation recognition systems that only consider two relevant discourse units. BIBREF12 : improves implicit discourse relation prediction by creating more training instances from the Gigaword corpus utilizing explicitly mentioned discourse connective phrases. BIBREF5 : a gated relevance network (GRN) model with tensors to capture semantic interactions between words from two discourse units. BIBREF9 : a convolutional neural network model that leverages relations between different styles of discourse relations annotations (PDTB and RST BIBREF24 ) in a multi-task joint learning framework. BIBREF6 : a multi-level attention-over-attention model to dynamically exploit features from two discourse units for recognizing an implicit discourse relation. BIBREF21 : a novel pipelined adversarial framework to enable an adaptive imitation competition between the implicit network and a rival feature discriminator with access to connectives. BIBREF18 : a Simple Word Interaction Model (SWIM) with tensors that captures both linear and quadratic relations between words from two discourse units. BIBREF19 : an attention-based LSTM neural network that leverages explicit discourse relations in PDTB and unannotated external data in a multi-task joint learning framework. Evaluation Settings On the PDTB corpus, both binary classification and multi-way classification settings are commonly used to evaluate the implicit discourse relation recognition performance. We noticed that all the recent works report class-wise implicit relation prediction performance in the binary classification setting, while none of them report detailed performance in the multi-way classification setting. In the binary classification setting, separate “one-versus-all” binary classifiers were trained, and each classifier is to identify one class of discourse relations. Although separate classifiers are generally more flexible in combating with imbalanced distributions of discourse relation classes and obtain higher class-wise prediction performance, one pair of discourse units may be tagged with all four discourse relations without proper conflict resolution. Therefore, the multi-way classification setting is more appropriate and natural in evaluating a practical end-to-end discourse parser, and we mainly evaluate our proposed models using the four-way multi-class classification setting. Since none of the recent previous work reported class-wise implicit relation classification performance in the multi-way classification setting, for better comparisons, we re-implemented the neural tensor network architecture (so-called SWIM in BIBREF18 ) which is essentially a Bi-LSTM model with tensors and report its detailed evaluation result in the multi-way classification setting. As another baseline, we report the performance of a Bi-LSTM model without tensors as well. Both baseline models take two relevant discourse units as the only input. For additional comparisons, We also report the performance of our proposed models in the binary classification setting. Experimental Results Multi-way Classification: The first section of table 3 shows macro average F1-scores and accuracies of previous works. The second section of table 3 shows the multi-class classification results of our implemented baseline systems. Consistent with results of previous works, neural tensors, when applied to Bi-LSTMs, improved implicit discourse relation prediction performance. However, the performance on the three small classes (Comp, Cont and Temp) remains low. The third section of table 3 shows the multi-class classification results of our proposed paragraph-level neural network models that capture inter-dependencies among discourse units. The first row shows the performance of a variant of our basic model, where we only identify implicit relations and ignore identifying explicit relations by setting the $\alpha $ in equation (5) to be 0. Compared with the baseline Bi-LSTM model, the only difference is that this model considers paragraph-wide contexts and model inter-dependencies among discourse units when building representation for individual DU. We can see that this model has greatly improved implicit relation classification performance across all the four relations and improved the macro-average F1-score by over 7 percents. In addition, compared with the baseline Bi-LSTM model with tensor, this model improved implicit relation classification performance across the three small classes, with clear performance gains of around 2 and 8 percents on contingency and temporal relations respectively, and overall improved the macro-average F1-score by 2.2 percents. The second row shows the performance of our basic paragraph-level model which predicts both implicit and explicit discourse relations in a paragraph. Compared to the variant system (the first row), the basic model further improved the classification performance on the first three implicit relations. Especially on the contingency relation, the classification performance was improved by another 1.42 percents. Moreover, the basic model yields good performance for recognizing explicit discourse relations as well, which is comparable with previous best result (92.05% macro F1-score and 93.09% accuracy as reported in BIBREF11 ). After untying parameters in the softmax prediction layer, implicit discourse relation classification performance was improved across all four relations, meanwhile, the explicit discourse relation classification performance was also improved. The CRF layer further improved implicit discourse relation recognition performance on the three small classes. In summary, our full paragraph-level neural network model achieves the best macro-average F1-score of 48.82% in predicting implicit discourse relations, which outperforms previous neural tensor network models (e.g., BIBREF18 ) by more than 2 percents and outperforms the best previous system BIBREF19 by 1 percent. Binary Classification: From table 4 , we can see that compared against the best previous systems, our paragraph-level model with untied parameters in the prediction layer achieves F1-score improvements of 6 points on Comparison and 7 points on Temporal, which demonstrates that paragraph-wide contexts are important in detecting minority discourse relations. Note that the CRF layer of the model is not suitable for binary classification. Ensemble Model As we explained in section 4.2, we ran our models for 10 times to obtain stable average performance. Then we also created ensemble models by applying majority voting to combine results of ten runs. From table 5 , each ensemble model obtains performance improvements compared with single model. The full model achieves performance boosting of (51.84 - 48.82 = 3.02) and (94.17 - 93.21 = 0.96) in macro F1-scores for predicting implicit and explicit discourse relations respectively. Furthermore, the ensemble model achieves the best performance for predicting both implicit and explicit discourse relations simultaneously. Impact of Paragraph Length To understand the influence of paragraph lengths to our paragraph-level models, we divide paragraphs in the PDTB test set into several subsets based on the number of DUs in a paragraph, and then evaluate our proposed models on each subset separately. From Figure 4 , we can see that our paragraph-level models (the latter three) overall outperform DU-pair baselines across all the subsets. As expected, the paragraph-level models achieve clear performance gains on long paragraphs (with more than 5 DUs) by extensively modeling mutual influences of DUs in a paragraph. But somewhat surprisingly, the paragraph-level models achieve noticeable performance gains on short paragraphs (with 2 or 3 DUs) as well. We hypothesize that by learning more appropriate discourse-aware DU representations in long paragraphs, our paragraph-level models reduce bias of using DU representations in predicting discourse relations, which benefits discourse relation prediction in short paragraphs as well. Example Analysis For the example ( "Implicit Discourse Relation Recognition" ), the baseline neural tensor model predicted both implicit relations wrongly (“Implicit-Contingency” between DU2 and DU3; “Implicit-Expansion” between DU3 and DU4), while our paragraph-level model predicted all the four discourse relations correctly, which indicates that paragraph-wide contexts play a key role in implicit discourse relation prediction. For another example: (2): [Marshall came clanking in like Marley's ghost dragging those chains of brigades and air wings and links with Arab despots.] $_{DU1}$ (Implicit-Temporal) [He wouldn't leave] $_{DU2}$ until (Explicit-Temporal) [Mr. Cheney promised to do whatever the Pentagon systems analysts told him.] $_{DU3}$ Our basic paragraph-level model wrongly predicted the implicit discourse relation between DU1 and DU2 to be “Implicit-Comparison”, without being able to effectively use the succeeding “Explicit-Temporal” relation. On the contrary, the full model corrected this mistake by modeling discourse relation patterns with the CRF layer. Conclusion We have presented a paragraph-level neural network model that takes a sequence of discourse units as input, models inter-dependencies between discourse units as well as discourse relation continuity and patterns, and predicts a sequence of discourse relations in a paragraph. By building wider-context informed discourse unit representations and capturing the overall discourse structure, the paragraph-level neural network model outperforms the best previous models for implicit discourse relation recognition on the PDTB dataset. Acknowledgments We acknowledge the support of NVIDIA Corporation for their donation of one GeForce GTX TITAN X GPU used for this research.
How much does this model improve state-of-the-art?
bd5bd1765362c2d972a762ca12675108754aa437
How Ġmuch Ġdoes Ġthis Ġmodel Ġimprove Ġstate - of - the - art ?
the Ġbasic Ġmodel Ġyields Ġgood Ġperformance Ġfor Ġrecognizing Ġexplicit Ġdiscourse Ġrelations Ġas Ġwell , Ġwhich Ġis Ġcomparable Ġwith Ġprevious Ġbest Ġresult Ġ( 92 . 05 % Ġmacro ĠF 1 - score Ġand Ġ93 . 09 % Ġaccuracy Ġas Ġreported Ġin ĠB IB REF 11 Ġ). , Ġfull Ġparagraph - level Ġneural Ġnetwork Ġmodel Ġachieves Ġthe Ġbest Ġmacro - average ĠF 1 - score Ġof Ġ48 . 82 % Ġin Ġpredicting Ġimplicit Ġdiscourse Ġrelations , Ġwhich Ġoutper forms Ġprevious Ġneural Ġtens or Ġnetwork Ġmodels Ġ( e . g ., ĠB IB REF 18 Ġ) Ġby Ġmore Ġthan Ġ2 Ġper c ents Ġand Ġoutper forms Ġthe Ġbest Ġprevious Ġsystem ĠB IB REF 19 Ġby Ġ1 Ġpercent ., ĠThen Ġwe Ġalso Ġcreated Ġensemble Ġmodels Ġby Ġapplying Ġmajority Ġvoting Ġto Ġcombine Ġresults Ġof Ġten Ġruns . ĠFrom Ġtable Ġ5 Ġ, Ġeach Ġensemble Ġmodel Ġob tains Ġperformance Ġimprovements Ġcompared Ġwith Ġsingle Ġmodel . ĠThe Ġfull Ġmodel Ġachieves Ġperformance Ġboosting Ġof Ġ( 51 . 84 Ġ- Ġ48 . 82 Ġ= Ġ3 . 02 ) Ġand Ġ( 94 . 17 Ġ- Ġ93 . 21 Ġ= Ġ0 . 96 ) Ġin Ġmacro ĠF 1 - sc ores Ġfor Ġpredicting Ġimplicit Ġand Ġexplicit Ġdiscourse Ġrelations Ġrespectively . Ġ
<s> How Ġmuch Ġdoes Ġthis Ġmodel Ġimprove Ġstate - of - the - art ? </s> Introduction </s> PD TB - style Ġdiscourse Ġrelations , Ġmostly Ġdefined Ġbetween Ġtwo Ġadjacent Ġtext Ġspans Ġ( i . e ., Ġdiscourse Ġunits , Ġeither Ġclauses Ġor Ġsentences ), Ġspecify Ġhow Ġtwo Ġdiscourse Ġunits Ġare Ġlogically Ġconnected Ġ( e . g ., Ġcausal , Ġcontrast ). ĠRecogn izing Ġdiscourse Ġrelations Ġis Ġone Ġcrucial Ġstep Ġin Ġdiscourse Ġanalysis Ġand Ġcan Ġbe Ġbeneficial Ġfor Ġmany Ġdownstream ĠN LP Ġapplications Ġsuch Ġas Ġinformation Ġextraction , Ġmachine Ġtranslation Ġand Ġnatural Ġlanguage Ġgeneration . </s> Comm only , Ġexplicit Ġdiscourse Ġrelations Ġwere Ġdistinguished Ġfrom Ġimplicit Ġones , Ġdepending Ġon Ġwhether Ġa Ġdiscourse Ġconnect ive Ġ( e . g ., ĠâĢ ľ because âĢ Ŀ Ġand ĠâĢ ľ after âĢ Ŀ ) Ġappears Ġbetween Ġtwo Ġdiscourse Ġunits ĠB IB REF 0 Ġ. ĠWhile Ġexplicit Ġdiscourse Ġrelation Ġdetection Ġcan Ġbe Ġframed Ġas Ġa Ġdiscourse Ġconnect ive Ġdis amb ig uation Ġproblem ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġand Ġhas Ġachieved Ġreasonable Ġperformance Ġ( F 1 Ġscore Ġ$ > $ Ġ90 %), Ġimplicit Ġdiscourse Ġrelations Ġhave Ġno Ġdiscourse Ġconnect ive Ġand Ġare Ġespecially Ġdifficult Ġto Ġidentify ĠB IB REF 3 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 4 Ġ. ĠTo Ġfill Ġthe Ġgap , Ġimplicit Ġdiscourse Ġrelation Ġprediction Ġhas Ġdrawn Ġsignificant Ġresearch Ġinterest Ġrecently Ġand Ġprogress Ġhas Ġbeen Ġmade ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġby Ġmodeling Ġcompos itional Ġmeanings Ġof Ġtwo Ġdiscourse Ġunits Ġand Ġexploiting Ġword Ġinteractions Ġbetween Ġdiscourse Ġunits Ġusing Ġneural Ġtens or Ġnetworks Ġor Ġattention Ġmechanisms Ġin Ġneural Ġnets . ĠHowever , Ġmost Ġof Ġexisting Ġapproaches Ġignore Ġwider Ġparagraph - level Ġcontexts Ġbeyond Ġthe Ġtwo Ġdiscourse Ġunits Ġthat Ġare Ġexamined Ġfor Ġpredicting Ġa Ġdiscourse Ġrelation Ġin Ġbetween . </s> To Ġfurther Ġimprove Ġimplicit Ġdiscourse Ġrelation Ġprediction , Ġwe Ġaim Ġto Ġimprove Ġdiscourse Ġunit Ġrepresentations Ġby Ġpositioning Ġa Ġdiscourse Ġunit Ġ( D U ) Ġin Ġits Ġwider Ġcontext Ġof Ġa Ġparagraph . ĠThe Ġkey Ġobservation Ġis Ġthat Ġsemantic Ġmeaning Ġof Ġa ĠDU Ġcan Ġnot Ġbe Ġinterpreted Ġindependently Ġfrom Ġthe Ġrest Ġof Ġthe Ġparagraph Ġthat Ġcontains Ġit , Ġor Ġindependently Ġfrom Ġthe Ġoverall Ġparagraph - level Ġdiscourse Ġstructure Ġthat Ġinvolve Ġthe ĠDU . ĠConsidering Ġthe Ġfollowing Ġparagraph Ġwith Ġfour Ġdiscourse Ġrelations , Ġone Ġrelation Ġbetween Ġeach Ġtwo Ġadjacent ĠD Us : </s> ( 1 ): Ġ[ The ĠButler , ĠWis ., Ġmanufacturer Ġwent Ġpublic Ġat Ġ$ 15 . 75 Ġa Ġshare Ġin ĠAugust Ġ1987 , ] Ġ$ _{ D U 1 } $ Ġand Ġ( Expl icit - Exp ansion ) Ġ[ Mr . ĠSim 's Ġgoal Ġthen Ġwas Ġa Ġ$ 29 Ġper - share Ġprice Ġby Ġ1992 .] Ġ$ _{ D U 2 } $ Ġ( Impl icit - Exp ansion ) Ġ[ Strong Ġearnings Ġgrowth Ġhelped Ġachieve Ġthat Ġprice Ġfar Ġahead Ġof Ġschedule , Ġin ĠAugust Ġ1988 .] Ġ$ _{ D U 3 } $ Ġ( Impl icit - Compar ison ) Ġ[ The Ġstock Ġhas Ġsince Ġsoftened , Ġtrading Ġaround Ġ$ 25 Ġa Ġshare Ġlast Ġweek Ġand Ġclosing Ġyesterday Ġat Ġ$ 23 Ġin Ġnational Ġover - the - counter Ġtrading .] Ġ$ _{ D U 4 } $ ĠBut Ġ( Expl icit - Compar ison ) Ġ[ Mr . ĠSim Ġhas Ġset Ġa Ġfresh Ġtarget Ġof Ġ$ 50 Ġa Ġshare Ġby Ġthe Ġend Ġof Ġreaching Ġthat Ġgoal .] Ġ$ _{ D U 5 } $ </s> Clearly , Ġeach ĠDU Ġis Ġan Ġintegral Ġpart Ġof Ġthe Ġparagraph Ġand Ġnot Ġindependent Ġfrom Ġother Ġunits . ĠFirst , Ġpredicting Ġa Ġdiscourse Ġrelation Ġmay Ġrequire Ġunderstanding Ġwider Ġparagraph - level Ġcontexts Ġbeyond Ġtwo Ġrelevant ĠD Us Ġand Ġthe Ġoverall Ġdiscourse Ġstructure Ġof Ġa Ġparagraph . ĠFor Ġexample , Ġthe Ġimplicit ĠâĢ ľ Compar ison âĢ Ŀ Ġdiscourse Ġrelation Ġbetween ĠDU 3 Ġand ĠDU 4 Ġis Ġdifficult Ġto Ġidentify Ġwithout Ġthe Ġbackground Ġinformation Ġ( the Ġhistory Ġof Ġper - share Ġprice ) Ġintroduced Ġin ĠDU 1 Ġand ĠDU 2 . ĠSecond , Ġa ĠDU Ġmay Ġbe Ġinvolved Ġin Ġmultiple Ġdiscourse Ġrelations Ġ( e . g ., ĠDU 4 Ġis Ġconnected Ġwith Ġboth ĠDU 3 Ġand ĠDU 5 Ġwith Ġa ĠâĢ ľ Compar ison âĢ Ŀ Ġrelation ), Ġtherefore Ġthe Ġpragmatic Ġmeaning Ġrepresentation Ġof Ġa ĠDU Ġshould Ġreflect Ġall Ġthe Ġdiscourse Ġrelations Ġthe Ġunit Ġwas Ġinvolved Ġin . ĠThird , Ġimplicit Ġdiscourse Ġrelation Ġprediction Ġshould Ġbenefit Ġfrom Ġmodeling Ġdiscourse Ġrelation Ġcontinuity Ġand Ġpatterns Ġin Ġa Ġparagraph Ġthat Ġinvolve Ġeasy - to - ident ify Ġexplicit Ġdiscourse Ġrelations Ġ( e . g ., ĠâĢ ľ Impl icit - Compar ison âĢ Ŀ Ġrelation Ġis Ġfollowed Ġby ĠâĢ ľ Expl icit - Compar ison âĢ Ŀ Ġin Ġthe Ġabove Ġexample ). </s> Following Ġthese Ġobservations , Ġwe Ġconstruct Ġa Ġneural Ġnet Ġmodel Ġto Ġprocess Ġa Ġparagraph Ġeach Ġtime Ġand Ġjointly Ġbuild Ġmeaning Ġrepresentations Ġfor Ġall ĠD Us Ġin Ġthe Ġparagraph . ĠThe Ġlearned ĠDU Ġrepresentations Ġare Ġused Ġto Ġpredict Ġa Ġsequence Ġof Ġdiscourse Ġrelations Ġin Ġthe Ġparagraph , Ġincluding Ġboth Ġimplicit Ġand Ġexplicit Ġrelations . ĠAlthough Ġexplicit Ġrelations Ġare Ġnot Ġour Ġfocus , Ġpredicting Ġan Ġexplicit Ġrelation Ġwill Ġhelp Ġto Ġreveal Ġthe Ġpragmatic Ġroles Ġof Ġits Ġtwo ĠD Us Ġand Ġreconstruct Ġtheir Ġrepresentations , Ġwhich Ġwill Ġfacilitate Ġpredicting Ġneighboring Ġimplicit Ġdiscourse Ġrelations Ġthat Ġinvolve Ġone Ġof Ġthe ĠD Us . </s> In Ġaddition , Ġwe Ġintroduce Ġtwo Ġnovel Ġdesigns Ġto Ġfurther Ġimprove Ġdiscourse Ġrelation Ġclassification Ġperformance Ġof Ġour Ġparagraph - level Ġneural Ġnet Ġmodel . ĠFirst , Ġprevious Ġwork Ġhas Ġindicated Ġthat Ġrecognizing Ġexplicit Ġand Ġimplicit Ġdiscourse Ġrelations Ġrequires Ġdifferent Ġstrategies , Ġwe Ġtherefore Ġunt ie Ġparameters Ġin Ġthe Ġdiscourse Ġrelation Ġprediction Ġlayer Ġof Ġthe Ġneural Ġnetworks Ġand Ġtrain Ġtwo Ġseparate Ġclass ifiers Ġfor Ġpredicting Ġexplicit Ġand Ġimplicit Ġdiscourse Ġrelations Ġrespectively . ĠThis Ġunique Ġdesign Ġhas Ġimproved Ġboth Ġimplicit Ġand Ġexplicit Ġdiscourse Ġrelation Ġidentification Ġperformance . ĠSecond , Ġwe Ġadd Ġa ĠCR F Ġlayer Ġon Ġtop Ġof Ġthe Ġdiscourse Ġrelation Ġprediction Ġlayer Ġto Ġfine - t une Ġa Ġsequence Ġof Ġpredicted Ġdiscourse Ġrelations Ġby Ġmodeling Ġdiscourse Ġrelation Ġcontinuity Ġand Ġpatterns Ġin Ġa Ġparagraph . </s> Exper imental Ġresults Ġshow Ġthat Ġthe Ġintuitive Ġparagraph - level Ġdiscourse Ġrelation Ġprediction Ġmodel Ġachieves Ġimproved Ġperformance Ġon ĠPD TB Ġfor Ġboth Ġimplicit Ġdiscourse Ġrelation Ġclassification Ġand Ġexplicit Ġdiscourse Ġrelation Ġclassification . </s> Impl icit ĠDisc ourse ĠRel ation ĠRecogn ition </s> Since Ġthe ĠPD TB ĠB IB REF 7 Ġcorpus Ġwas Ġcreated , Ġa Ġsurge Ġof Ġstudies ĠB IB REF 8 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġhave Ġbeen Ġconducted Ġfor Ġpredicting Ġdiscourse Ġrelations , Ġprimarily Ġfocusing Ġon Ġthe Ġchallenging Ġtask Ġof Ġimplicit Ġdiscourse Ġrelation Ġclassification Ġwhen Ġno Ġexplicit Ġdiscourse Ġconnect ive Ġphrase Ġwas Ġpresented . ĠEarly Ġstudies ĠB IB REF 11 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 12 Ġfocused Ġon Ġextracting Ġlinguistic Ġand Ġsemantic Ġfeatures Ġfrom Ġtwo Ġdiscourse Ġunits . ĠRecent Ġresearch ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġtried Ġto Ġmodel Ġcompos itional Ġmeanings Ġof Ġtwo Ġdiscourse Ġunits Ġby Ġexploiting Ġinteractions Ġbetween Ġwords Ġin Ġtwo Ġunits Ġwith Ġmore Ġand Ġmore Ġcomplicated Ġneural Ġnetwork Ġmodels , Ġincluding Ġthe Ġones Ġusing Ġneural Ġtens or ĠB IB REF 5 Ġ, ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġand Ġattention Ġmechanisms ĠB IB REF 6 Ġ, ĠB IB REF 19 Ġ, ĠB IB REF 20 Ġ. ĠAnother Ġtrend Ġis Ġto Ġalleviate Ġthe Ġshortage Ġof Ġannot ated Ġdata Ġby Ġleveraging Ġrelated Ġexternal Ġdata , Ġsuch Ġas Ġexplicit Ġdiscourse Ġrelations Ġin ĠPD TB ĠB IB REF 9 Ġ, ĠB IB REF 19 Ġ, ĠB IB REF 21 Ġand Ġunl abel ed Ġdata Ġobtained Ġelsewhere ĠB IB REF 12 Ġ, ĠB IB REF 19 Ġ, Ġoften Ġin Ġa Ġmulti - task Ġjoint Ġlearning Ġframework . </s> However , Ġnearly Ġall Ġthe Ġprevious Ġworks Ġassume Ġthat Ġa Ġpair Ġof Ġdiscourse Ġunits Ġis Ġindependent Ġfrom Ġits Ġwider Ġparagraph - level Ġcontexts Ġand Ġbuild Ġtheir Ġdiscourse Ġrelation Ġprediction Ġmodels Ġbased Ġon Ġonly Ġtwo Ġrelevant Ġdiscourse Ġunits . ĠIn Ġcontrast , Ġwe Ġmodel Ġinter - depend encies Ġof Ġdiscourse Ġunits Ġin Ġa Ġparagraph Ġwhen Ġbuilding Ġdiscourse Ġunit Ġrepresentations ; Ġin Ġaddition , Ġwe Ġmodel Ġglobal Ġcontinuity Ġand Ġpatterns Ġin Ġa Ġsequence Ġof Ġdiscourse Ġrelations , Ġincluding Ġboth Ġimplicit Ġand Ġexplicit Ġrelations . </s> H ier arch ical Ġneural Ġnetwork Ġmodels ĠB IB REF 22 Ġ, ĠB IB REF 23 Ġhave Ġbeen Ġapplied Ġto ĠR ST - style Ġdiscourse Ġparsing ĠB IB REF 24 Ġmainly Ġfor Ġthe Ġpurpose Ġof Ġgenerating Ġtext - level Ġhierarchical Ġdiscourse Ġstructures . ĠIn Ġcontrast , Ġwe Ġuse Ġhierarchical Ġneural Ġnetwork Ġmodels Ġto Ġbuild Ġcontext - aware Ġsentence Ġrepresentations Ġin Ġorder Ġto Ġimprove Ġimplicit Ġdiscourse Ġrelation Ġprediction . </s> Par agraph ĠEnc oding </s> Abstract ing Ġlatent Ġrepresentations Ġfrom Ġa Ġlong Ġsequence Ġof Ġwords , Ġsuch Ġas Ġa Ġparagraph , Ġis Ġa Ġchallenging Ġtask . ĠWhile Ġseveral Ġnovel Ġneural Ġnetwork Ġmodels ĠB IB REF 25 Ġ, ĠB IB REF 26 Ġhave Ġbeen Ġintroduced Ġin Ġrecent Ġyears Ġfor Ġencoding Ġa Ġparagraph , ĠRec urrent ĠNeural ĠNetwork Ġ( R NN )- based Ġmethods Ġremain Ġthe Ġmost Ġeffective Ġapproaches . ĠR NN s , Ġespecially Ġthe Ġlong - short Ġterm Ġmemory Ġ( L ST M ) ĠB IB REF 27 Ġmodels , Ġhave Ġbeen Ġwidely Ġused Ġto Ġencode Ġa Ġparagraph Ġfor Ġmachine Ġtranslation ĠB IB REF 28 Ġ, Ġdialogue Ġsystems ĠB IB REF 29 Ġand Ġtext Ġsummar ization ĠB IB REF 30 Ġbecause Ġof Ġits Ġability Ġin Ġmodeling Ġlong - distance Ġdependencies Ġbetween Ġwords . ĠIn Ġaddition , Ġamong Ġfour Ġtypical Ġpool ing Ġmethods Ġ( sum , Ġmean , Ġlast Ġand Ġmax ) Ġfor Ġcalculating Ġsentence Ġrepresentations Ġfrom ĠR NN - enc oded Ġhidden Ġstates Ġfor Ġindividual Ġwords , Ġmax - pool ing Ġalong Ġwith Ġbid irection al ĠL ST M Ġ( Bi - L ST M ) ĠB IB REF 31 Ġyields Ġthe Ġcurrent Ġbest Ġuniversal Ġsentence Ġrepresentation Ġmethod ĠB IB REF 32 Ġ. ĠWe Ġadopted Ġa Ġsimilar Ġneural Ġnetwork Ġarchitecture Ġfor Ġparagraph Ġencoding . </s> The ĠBasic ĠModel ĠArchitecture </s> Figure Ġ1 Ġillustrates Ġthe Ġoverall Ġarchitecture Ġof Ġthe Ġdiscourse - level Ġneural Ġnetwork Ġmodel Ġthat Ġconsists Ġof Ġtwo ĠBi - L ST M Ġlayers , Ġone Ġmax - pool ing Ġlayer Ġin Ġbetween Ġand Ġone Ġsoft max Ġprediction Ġlayer . ĠThe Ġinput Ġof Ġthe Ġneural Ġnetwork Ġmodel Ġis Ġa Ġparagraph Ġcontaining Ġa Ġsequence Ġof Ġdiscourse Ġunits , Ġwhile Ġthe Ġoutput Ġis Ġa Ġsequence Ġof Ġdiscourse Ġrelations Ġwith Ġone Ġrelation Ġbetween Ġeach Ġpair Ġof Ġadjacent Ġdiscourse Ġunits . </s> Given Ġthe Ġwords Ġsequence Ġof Ġone Ġparagraph Ġas Ġinput , Ġthe Ġlower ĠBi - L ST M Ġlayer Ġwill Ġread Ġthe Ġwhole Ġparagraph Ġand Ġcalculate Ġhidden Ġstates Ġas Ġword Ġrepresentations , Ġand Ġa Ġmax - pool ing Ġlayer Ġwill Ġbe Ġapplied Ġto Ġabstract Ġthe Ġrepresentation Ġof Ġeach Ġdiscourse Ġunit Ġbased Ġon Ġindividual Ġword Ġrepresentations . ĠThen Ġanother ĠBi - L ST M Ġlayer Ġwill Ġrun Ġover Ġthe Ġsequence Ġof Ġdiscourse Ġunit Ġrepresentations Ġand Ġcompute Ġnew Ġrepresentations Ġby Ġfurther Ġmodeling Ġsemantic Ġdependencies Ġbetween Ġdiscourse Ġunits Ġwithin Ġparagraph . ĠThe Ġfinal Ġsoft max Ġprediction Ġlayer Ġwill Ġconc aten ate Ġrepresentations Ġof Ġtwo Ġadjacent Ġdiscourse Ġunits Ġand Ġpredict Ġthe Ġdiscourse Ġrelation Ġbetween Ġthem . </s> Word ĠV ect ors Ġas ĠInput : ĠThe Ġinput Ġof Ġthe Ġparagraph - level Ġdiscourse Ġrelation Ġprediction Ġmodel Ġis Ġa Ġsequence Ġof Ġword Ġvectors , Ġone Ġvector Ġper Ġword Ġin Ġthe Ġparagraph . ĠIn Ġthis Ġwork , Ġwe Ġused Ġthe Ġpre - trained Ġ300 - dimension ĠGoogle ĠEnglish Ġword 2 vec Ġembed d ings . ĠFor Ġeach Ġword Ġthat Ġis Ġnot Ġin Ġthe Ġvocabulary Ġof ĠGoogle Ġword 2 vec , Ġwe Ġwill Ġrandomly Ġinitialize Ġa Ġvector Ġwith Ġeach Ġdimension Ġsampled Ġfrom Ġthe Ġrange Ġ$ [ - 0 . 25 , Ġ0 . 25 ] $ Ġ. ĠIn Ġaddition , Ġrecognizing Ġkey Ġentities Ġand Ġdiscourse Ġconnect ive Ġphrases Ġis Ġimportant Ġfor Ġdiscourse Ġrelation Ġrecognition , Ġtherefore , Ġwe Ġconc aten ate Ġthe Ġraw Ġword Ġembed d ings Ġwith Ġextra Ġlinguistic Ġfeatures , Ġspecifically Ġone - hot ĠPart - Of - Spe ech Ġtag Ġembed d ings Ġand Ġone - hot Ġnamed Ġentity Ġtag Ġembed d ings . </s> Building ĠDisc ourse ĠUnit ĠRepresent ations : ĠWe Ġaim Ġto Ġbuild Ġdiscourse Ġunit Ġ( D U ) Ġrepresentations Ġthat Ġsufficiently Ġleverage Ġcues Ġfor Ġdiscourse Ġrelation Ġprediction Ġfrom Ġparagraph - wide Ġcontexts , Ġincluding Ġthe Ġpreceding Ġand Ġfollowing Ġdiscourse Ġunits Ġin Ġa Ġparagraph . ĠTo Ġprocess Ġlong Ġparagraph - wide Ġcontexts , Ġwe Ġtake Ġa Ġbottom - up Ġtwo - level Ġabstraction Ġapproach Ġand Ġprogressively Ġgenerate Ġa Ġcompos itional Ġrepresentation Ġof Ġeach Ġword Ġfirst Ġ( low Ġlevel ) Ġand Ġthen Ġgenerate Ġa Ġcompos itional Ġrepresentation Ġof Ġeach Ġdiscourse Ġunit Ġ( high Ġlevel ), Ġwith Ġa Ġmax - pool ing Ġoperation Ġin Ġbetween . ĠAt Ġboth Ġword - level Ġand ĠDU - level , Ġwe Ġchoose ĠBi - L ST M Ġas Ġour Ġbasic Ġcomponent Ġfor Ġgenerating Ġcompos itional Ġrepresentations , Ġmainly Ġconsidering Ġits Ġcapability Ġto Ġcapture Ġlong - distance Ġdependencies Ġbetween Ġwords Ġ( disc ourse Ġunits ) Ġand Ġto Ġincorporate Ġinfluences Ġof Ġcontext Ġwords Ġ( disc ourse Ġunits ) Ġin Ġeach Ġside . </s> Given Ġa Ġvariable - length Ġwords Ġsequence Ġ$ X Ġ= Ġ( x _ 1 , x _ 2 ,... , x _ L ) $ Ġin Ġa Ġparagraph , Ġthe Ġword - level ĠBi - L ST M Ġwill Ġprocess Ġthe Ġinput Ġsequence Ġby Ġusing Ġtwo Ġseparate ĠL ST Ms , Ġone Ġprocess Ġthe Ġword Ġsequence Ġfrom Ġthe Ġleft Ġto Ġright Ġwhile Ġthe Ġother Ġfollows Ġthe Ġreversed Ġdirection . ĠTherefore , Ġat Ġeach Ġword Ġposition Ġ$ t $ Ġ, Ġwe Ġobtain Ġtwo Ġhidden Ġstates Ġ$\ over right arrow { h _ t }, Ġ\ over left arrow { h _ t } $ Ġ. ĠWe Ġconc aten ate Ġthem Ġto Ġget Ġthe Ġword Ġrepresentation Ġ$ h _ t Ġ= Ġ[ \ over right arrow { h _ t }, Ġ\ over left arrow { h _ t } ] $ Ġ. ĠThen Ġwe Ġapply Ġmax - pool ing Ġover Ġthe Ġsequence Ġof Ġword Ġrepresentations Ġfor Ġwords Ġin Ġa Ġdiscourse Ġunit Ġin Ġorder Ġto Ġget Ġthe Ġdiscourse Ġunit Ġembed ding : </s> $$ MP _{ D U } [ j ] Ġ= Ġ\ max Ġ_ { i = D U \ _ start } ^{ D U \ _ end } h _ i [ j ] \ quad Ġ\\ Ġwhere , Ġ1 Ġ\ le Ġj Ġ\ le Ġhidden \ _ node \ _ size $$ Ġ Ġ Ġ( E q . Ġ8 ) </s> Next , Ġthe ĠDU - level ĠBi - L ST M Ġwill Ġprocess Ġthe Ġsequence Ġof Ġdiscourse Ġunit Ġembed d ings Ġin Ġa Ġparagraph Ġand Ġgenerate Ġtwo Ġhidden Ġstates Ġ$\ over right arrow { h D U _ t } $ Ġand Ġ$\ over left arrow { h D U _ t } $ Ġat Ġeach Ġdiscourse Ġunit Ġposition . ĠWe Ġconc aten ate Ġthem Ġto Ġget Ġthe Ġdiscourse Ġunit Ġrepresentation Ġ$ h D U _ t Ġ= Ġ[ \ over right arrow { h D U _ t }, Ġ\ over left arrow { h D U _ t } ] $ Ġ. </s> The ĠSoft max ĠPrediction ĠLayer : ĠFinally , Ġwe Ġconc aten ate Ġtwo Ġadjacent Ġdiscourse Ġunit Ġrepresentations Ġ$ h D U _{ t - 1 } $ Ġand Ġ$ h D U _ t $ Ġand Ġpredict Ġthe Ġdiscourse Ġrelation Ġbetween Ġthem Ġusing Ġa Ġsoft max Ġfunction : </s> $$ y _{ t - 1 } Ġ= Ġsoft max ( W _ y * [ h D U _{ t - 1 }, h D U _ t ]+ b _ y ) $$ Ġ Ġ Ġ( E q . Ġ9 ) </s> Unt ie ĠParameters Ġin Ġthe ĠSoft max ĠPrediction ĠLayer Ġ( Impl icit Ġvs . ĠExplicit ) </s> Previous Ġwork ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 10 Ġhas Ġrevealed Ġthat Ġrecognizing Ġexplicit Ġvs . Ġimplicit Ġdiscourse Ġrelations Ġrequires Ġdifferent Ġstrategies . ĠNote Ġthat Ġin Ġthe ĠPD TB Ġdataset , Ġexplicit Ġdiscourse Ġrelations Ġwere Ġdistinguished Ġfrom Ġimplicit Ġones , Ġdepending Ġon Ġwhether Ġa Ġdiscourse Ġconnect ive Ġexists Ġbetween Ġtwo Ġdiscourse Ġunits . ĠTherefore , Ġexplicit Ġdiscourse Ġrelation Ġdetection Ġcan Ġbe Ġsimplified Ġas Ġa Ġdiscourse Ġconnect ive Ġphrase Ġdis amb ig uation Ġproblem ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ. ĠOn Ġthe Ġcontrary , Ġpredicting Ġan Ġimplicit Ġdiscourse Ġrelation Ġshould Ġrely Ġon Ġunderstanding Ġthe Ġoverall Ġcontents Ġof Ġits Ġtwo Ġdiscourse Ġunits ĠB IB REF 2 Ġ, ĠB IB REF 10 Ġ. </s> Considering Ġthe Ġdifferent Ġn atures Ġof Ġexplicit Ġvs . Ġimplicit Ġdiscourse Ġrelation Ġprediction , Ġwe Ġdecide Ġto Ġunt ie Ġparameters Ġat Ġthe Ġfinal Ġdiscourse Ġrelation Ġprediction Ġlayer Ġand Ġtrain Ġtwo Ġsoft max Ġclass ifiers , Ġas Ġillustrated Ġin ĠFigure Ġ2 Ġ. ĠThe Ġtwo Ġclass ifiers Ġhave Ġdifferent Ġsets Ġof Ġparameters , Ġwith Ġone Ġclass ifier Ġfor Ġonly Ġimplicit Ġdiscourse Ġrelations Ġand Ġthe Ġother Ġfor Ġonly Ġexplicit Ġdiscourse Ġrelations . </s> $$ y _{ t - 1 } Ġ= Ġ{\ left \ l brace Ġ\ begin { array }{ ll } Ġsoft max ( W _{ exp } [ h D U _{ t - 1 }, h D U _ t ]+ b _{ exp } ), & exp \\ Ġsoft max ( W _{ imp } [ h D U _{ t - 1 }, h D U _ t ]+ b _{ imp } ), & imp Ġ\ end { array }\ right .} $$ Ġ Ġ Ġ( E q . Ġ12 ) </s> The Ġloss Ġfunction Ġused Ġfor Ġthe Ġneural Ġnetwork Ġtraining Ġconsiders Ġloss Ġinduced Ġby Ġboth Ġimplicit Ġrelation Ġprediction Ġand Ġexplicit Ġrelation Ġprediction : </s> $$ L oss Ġ= ĠLoss _{ imp } Ġ+ Ġ\ alpha Ġ* L oss _{ exp } $$ Ġ Ġ Ġ( E q . Ġ13 ) </s> The Ġ$\ alpha Ġ$ Ġ, Ġin Ġthe Ġfull Ġsystem , Ġis Ġset Ġto Ġbe Ġ1 , Ġwhich Ġmeans Ġthat Ġminimizing Ġthe Ġloss Ġin Ġpredicting Ġeither Ġtype Ġof Ġdiscourse Ġrelations Ġis Ġequally Ġimportant . ĠIn Ġthe Ġevaluation , Ġwe Ġwill Ġalso Ġevaluate Ġa Ġsystem Ġvariant , Ġwhere Ġwe Ġwill Ġset Ġ$\ alpha Ġ= Ġ0 $ Ġ, Ġwhich Ġmeans Ġthat Ġthe Ġneural Ġnetwork Ġwill Ġnot Ġattempt Ġto Ġpredict Ġexplicit Ġdiscourse Ġrelations Ġand Ġimplicit Ġdiscourse Ġrelation Ġprediction Ġwill Ġnot Ġbe Ġinfluenced Ġby Ġpredicting Ġneighboring Ġexplicit Ġdiscourse Ġrelations . </s> Fine - t une ĠDisc ourse ĠRel ation ĠPred ictions ĠUsing Ġa ĠCR F ĠLayer </s> Data Ġanalysis Ġand Ġmany Ġlinguistic Ġstudies ĠB IB REF 11 Ġ, ĠB IB REF 33 Ġ, ĠB IB REF 34 Ġ, ĠB IB REF 35 Ġhave Ġrepeatedly Ġshown Ġthat Ġdiscourse Ġrelations Ġfeature Ġcontinuity Ġand Ġpatterns Ġ( e . g ., Ġa Ġtemporal Ġrelation Ġis Ġlikely Ġto Ġbe Ġfollowed Ġby Ġanother Ġtemporal Ġrelation ). ĠEspecially , ĠB IB REF 11 Ġfirst ly Ġreported Ġthat Ġpatterns Ġexist Ġbetween Ġimplicit Ġdiscourse Ġrelations Ġand Ġtheir Ġneighboring Ġexplicit Ġdiscourse Ġrelations . </s> Mot ivated Ġby Ġthese Ġobservations , Ġwe Ġaim Ġto Ġimprove Ġimplicit Ġdiscourse Ġrelation Ġdetection Ġby Ġmaking Ġuse Ġof Ġeasily Ġidentifiable Ġexplicit Ġdiscourse Ġrelations Ġand Ġtaking Ġinto Ġaccount Ġglobal Ġpatterns Ġof Ġdiscourse Ġrelation Ġdistributions . ĠSpecifically , Ġwe Ġadd Ġan Ġextra ĠCR F Ġlayer Ġat Ġthe Ġtop Ġof Ġthe Ġsoft max Ġprediction Ġlayer Ġ( shown Ġin Ġfigure Ġ3 Ġ) Ġto Ġfine - t une Ġpredicted Ġdiscourse Ġrelations Ġby Ġconsidering Ġtheir Ġinter - depend encies . </s> The ĠCond itional ĠRandom ĠFields ĠB IB REF 36 Ġ( CR F ) Ġlayer Ġupdates Ġa Ġstate Ġtransition Ġmatrix , Ġwhich Ġcan Ġeffectively Ġadjust Ġthe Ġcurrent Ġlabel Ġdepending Ġon Ġproceeding Ġand Ġfollowing Ġlabels . ĠBoth Ġtraining Ġand Ġdecoding Ġof Ġthe ĠCR F Ġlayer Ġcan Ġbe Ġsolved Ġefficiently Ġby Ġusing Ġthe ĠV iter bi Ġalgorithm . ĠWith Ġthe ĠCR F Ġlayer , Ġthe Ġmodel Ġjointly Ġassigns Ġa Ġsequence Ġof Ġdiscourse Ġrelations Ġbetween Ġeach Ġtwo Ġadjacent Ġdiscourse Ġunits Ġin Ġa Ġparagraph , Ġincluding Ġboth Ġimplicit Ġand Ġexplicit Ġrelations , Ġby Ġconsidering Ġrelevant Ġdiscourse Ġunit Ġrepresentations Ġas Ġwell Ġas Ġglobal Ġdiscourse Ġrelation Ġpatterns . </s> Dat as et Ġand ĠPre processing </s> The ĠPenn ĠDisc ourse ĠTree bank Ġ( PD TB ): ĠWe Ġexperimented Ġwith ĠPD TB Ġv 2 . 0 ĠB IB REF 7 Ġwhich Ġis Ġthe Ġlargest Ġannot ated Ġcorpus Ġcontaining Ġ36 k Ġdiscourse Ġrelations Ġin Ġ2 , 159 ĠWall ĠStreet ĠJournal Ġ( WS J ) Ġarticles . ĠIn Ġthis Ġwork , Ġwe Ġfocus Ġon Ġthe Ġtop - level Ġdiscourse Ġrelation Ġsenses Ġwhich Ġare Ġconsist Ġof Ġfour Ġmajor Ġsemantic Ġclasses : ĠComparison Ġ( Comp ), ĠCont ing ency Ġ( Cont ), ĠExpansion Ġ( Exp ) Ġand ĠTem poral Ġ( Temp ). ĠWe Ġfollowed Ġthe Ġsame ĠPD TB Ġsection Ġpartition ĠB IB REF 12 Ġas Ġprevious Ġwork Ġand Ġused Ġsections Ġ2 - 20 Ġas Ġtraining Ġset , Ġsections Ġ21 - 22 Ġas Ġtest Ġset , Ġand Ġsections Ġ0 - 1 Ġas Ġdevelopment Ġset . ĠTable Ġ1 Ġpresents Ġthe Ġdata Ġdistributions Ġwe Ġcollected Ġfrom ĠPD TB . </s> Pre processing : ĠThe ĠPD TB Ġdataset Ġdocuments Ġits Ġannotations Ġas Ġa Ġlist Ġof Ġdiscourse Ġrelations , Ġwith Ġeach Ġrelation Ġassociated Ġwith Ġits Ġtwo Ġdiscourse Ġunits . ĠTo Ġrecover Ġthe Ġparagraph Ġcontext Ġfor Ġa Ġdiscourse Ġrelation , Ġwe Ġmatch Ġcontents Ġof Ġits Ġtwo Ġannot ated Ġdiscourse Ġunits Ġwith Ġall Ġparagraphs Ġin Ġcorresponding Ġraw ĠWS J Ġarticle . ĠWhen Ġall Ġthe Ġmatching Ġwas Ġcompleted , Ġeach Ġparagraph Ġwas Ġsplit Ġinto Ġa Ġsequence Ġof Ġdiscourse Ġunits , Ġwith Ġone Ġdiscourse Ġrelation Ġ( impl icit Ġor Ġexplicit ) Ġbetween Ġeach Ġtwo Ġadjacent Ġdiscourse Ġunits . ĠFollowing Ġthis Ġmethod , Ġwe Ġobtained Ġ14 , 309 Ġparagraphs Ġin Ġtotal , Ġeach Ġcontains Ġ3 . 2 Ġdiscourse Ġunits Ġon Ġaverage . ĠTable Ġ2 Ġshows Ġthe Ġdistribution Ġof Ġparagraphs Ġbased Ġon Ġthe Ġnumber Ġof Ġdiscourse Ġunits Ġin Ġa Ġparagraph . </s> Parameter ĠSettings Ġand ĠModel ĠTraining </s> We Ġtuned Ġthe Ġparameters Ġbased Ġon Ġthe Ġbest Ġperformance Ġon Ġthe Ġdevelopment Ġset . ĠWe Ġfixed Ġthe Ġweights Ġof Ġword Ġembed d ings Ġduring Ġtraining . ĠAll Ġthe ĠL ST Ms Ġin Ġour Ġneural Ġnetwork Ġuse Ġthe Ġhidden Ġstate Ġsize Ġof Ġ300 . ĠTo Ġavoid Ġover fitting , Ġwe Ġapplied Ġdrop out ĠB IB REF 37 Ġwith Ġdrop out Ġratio Ġof Ġ0 . 5 Ġto Ġboth Ġinput Ġand Ġoutput Ġof ĠL ST M Ġlayers . ĠTo Ġprevent Ġthe Ġexploding Ġgradient Ġproblem Ġin Ġtraining ĠL ST Ms , Ġwe Ġadopt Ġgradient Ġclipping Ġwith Ġgradient ĠL 2 - norm Ġthreshold Ġof Ġ5 . 0 . ĠThese Ġparameters Ġremain Ġthe Ġsame Ġfor Ġall Ġour Ġproposed Ġmodels Ġas Ġwell Ġas Ġour Ġown Ġbaseline Ġmodels . </s> We Ġchose Ġthe Ġstandard Ġcross - ent ropy Ġloss Ġfunction Ġfor Ġtraining Ġour Ġneural Ġnetwork Ġmodel Ġand Ġadopted ĠAdam ĠB IB REF 38 Ġoptim izer Ġwith Ġthe Ġinitial Ġlearning Ġrate Ġof Ġ5 e - 4 Ġand Ġa Ġmini - batch Ġsize Ġof Ġ128 . ĠIf Ġone Ġinstance Ġis Ġannot ated Ġwith Ġtwo Ġlabels Ġ( 4 % Ġof Ġall Ġinstances ), Ġwe Ġuse Ġboth Ġof Ġthem Ġin Ġloss Ġcalculation Ġand Ġregard Ġthe Ġprediction Ġas Ġcorrect Ġif Ġmodel Ġpredicts Ġone Ġof Ġthe Ġannot ated Ġlabels . ĠAll Ġthe Ġproposed Ġmodels Ġwere Ġimplemented Ġwith ĠPy tor ch Ġand Ġconver ged Ġto Ġthe Ġbest Ġperformance Ġwithin Ġ20 - 40 Ġepoch s . </s> To Ġalleviate Ġthe Ġinfluence Ġof Ġrandom ness Ġin Ġneural Ġnetwork Ġmodel Ġtraining Ġand Ġobtain Ġstable Ġexperimental Ġresults , Ġwe Ġran Ġeach Ġof Ġthe Ġproposed Ġmodels Ġand Ġour Ġown Ġbaseline Ġmodels Ġten Ġtimes Ġand Ġreport Ġthe Ġaverage Ġperformance Ġof Ġeach Ġmodel Ġinstead Ġof Ġthe Ġbest Ġperformance Ġas Ġreported Ġin Ġmany Ġprevious Ġworks . </s> Bas eline ĠModels Ġand ĠSystems </s> We Ġcompare Ġthe Ġperformance Ġof Ġour Ġneural Ġnetwork Ġmodel Ġwith Ġseveral Ġrecent Ġdiscourse Ġrelation Ġrecognition Ġsystems Ġthat Ġonly Ġconsider Ġtwo Ġrelevant Ġdiscourse Ġunits . </s> BI B REF 12 Ġ: Ġimproves Ġimplicit Ġdiscourse Ġrelation Ġprediction Ġby Ġcreating Ġmore Ġtraining Ġinstances Ġfrom Ġthe ĠGig aw ord Ġcorpus Ġutilizing Ġexplicitly Ġmentioned Ġdiscourse Ġconnect ive Ġphrases . </s> BI B REF 5 Ġ: Ġa Ġg ated Ġrelevance Ġnetwork Ġ( GR N ) Ġmodel Ġwith Ġtens ors Ġto Ġcapture Ġsemantic Ġinteractions Ġbetween Ġwords Ġfrom Ġtwo Ġdiscourse Ġunits . </s> BI B REF 9 Ġ: Ġa Ġconv olution al Ġneural Ġnetwork Ġmodel Ġthat Ġlever ages Ġrelations Ġbetween Ġdifferent Ġstyles Ġof Ġdiscourse Ġrelations Ġannotations Ġ( PD TB Ġand ĠR ST ĠB IB REF 24 Ġ) Ġin Ġa Ġmulti - task Ġjoint Ġlearning Ġframework . </s> BI B REF 6 Ġ: Ġa Ġmulti - level Ġattention - over - att ention Ġmodel Ġto Ġdynamically Ġexploit Ġfeatures Ġfrom Ġtwo Ġdiscourse Ġunits Ġfor Ġrecognizing Ġan Ġimplicit Ġdiscourse Ġrelation . </s> BI B REF 21 Ġ: Ġa Ġnovel Ġpip el ined Ġadvers arial Ġframework Ġto Ġenable Ġan Ġadaptive Ġimitation Ġcompetition Ġbetween Ġthe Ġimplicit Ġnetwork Ġand Ġa Ġrival Ġfeature Ġdiscrim inator Ġwith Ġaccess Ġto Ġconnect ives . </s> BI B REF 18 Ġ: Ġa ĠSimple ĠWord ĠInter action ĠModel Ġ( SW IM ) Ġwith Ġtens ors Ġthat Ġcaptures Ġboth Ġlinear Ġand Ġquad r atic Ġrelations Ġbetween Ġwords Ġfrom Ġtwo Ġdiscourse Ġunits . </s> BI B REF 19 Ġ: Ġan Ġattention - based ĠL ST M Ġneural Ġnetwork Ġthat Ġlever ages Ġexplicit Ġdiscourse Ġrelations Ġin ĠPD TB Ġand Ġun annot ated Ġexternal Ġdata Ġin Ġa Ġmulti - task Ġjoint Ġlearning Ġframework . </s> E val uation ĠSettings </s> On Ġthe ĠPD TB Ġcorpus , Ġboth Ġbinary Ġclassification Ġand Ġmulti - way Ġclassification Ġsettings Ġare Ġcommonly Ġused Ġto Ġevaluate Ġthe Ġimplicit Ġdiscourse Ġrelation Ġrecognition Ġperformance . ĠWe Ġnoticed Ġthat Ġall Ġthe Ġrecent Ġworks Ġreport Ġclass - wise Ġimplicit Ġrelation Ġprediction Ġperformance Ġin Ġthe Ġbinary Ġclassification Ġsetting , Ġwhile Ġnone Ġof Ġthem Ġreport Ġdetailed Ġperformance Ġin Ġthe Ġmulti - way Ġclassification Ġsetting . ĠIn Ġthe Ġbinary Ġclassification Ġsetting , Ġseparate ĠâĢ ľ one - vers us - all âĢ Ŀ Ġbinary Ġclass ifiers Ġwere Ġtrained , Ġand Ġeach Ġclass ifier Ġis Ġto Ġidentify Ġone Ġclass Ġof Ġdiscourse Ġrelations . ĠAlthough Ġseparate Ġclass ifiers Ġare Ġgenerally Ġmore Ġflexible Ġin Ġcombating Ġwith Ġim balanced Ġdistributions Ġof Ġdiscourse Ġrelation Ġclasses Ġand Ġobtain Ġhigher Ġclass - wise Ġprediction Ġperformance , Ġone Ġpair Ġof Ġdiscourse Ġunits Ġmay Ġbe Ġtagged Ġwith Ġall Ġfour Ġdiscourse Ġrelations Ġwithout Ġproper Ġconflict Ġresolution . ĠTherefore , Ġthe Ġmulti - way Ġclassification Ġsetting Ġis Ġmore Ġappropriate Ġand Ġnatural Ġin Ġevaluating Ġa Ġpractical Ġend - to - end Ġdiscourse Ġparser , Ġand Ġwe Ġmainly Ġevaluate Ġour Ġproposed Ġmodels Ġusing Ġthe Ġfour - way Ġmulti - class Ġclassification Ġsetting . </s> Since Ġnone Ġof Ġthe Ġrecent Ġprevious Ġwork Ġreported Ġclass - wise Ġimplicit Ġrelation Ġclassification Ġperformance Ġin Ġthe Ġmulti - way Ġclassification Ġsetting , Ġfor Ġbetter Ġcomparisons , Ġwe Ġre - im ple mented Ġthe Ġneural Ġtens or Ġnetwork Ġarchitecture Ġ( so - called ĠSW IM Ġin ĠB IB REF 18 Ġ) Ġwhich Ġis Ġessentially Ġa ĠBi - L ST M Ġmodel Ġwith Ġtens ors Ġand Ġreport Ġits Ġdetailed Ġevaluation Ġresult Ġin Ġthe Ġmulti - way Ġclassification Ġsetting . ĠAs Ġanother Ġbaseline , Ġwe Ġreport Ġthe Ġperformance Ġof Ġa ĠBi - L ST M Ġmodel Ġwithout Ġtens ors Ġas Ġwell . ĠBoth Ġbaseline Ġmodels Ġtake Ġtwo Ġrelevant Ġdiscourse Ġunits Ġas Ġthe Ġonly Ġinput . </s> For Ġadditional Ġcomparisons , ĠWe Ġalso Ġreport Ġthe Ġperformance Ġof Ġour Ġproposed Ġmodels Ġin Ġthe Ġbinary Ġclassification Ġsetting . </s> Exper imental ĠResults </s> Multi - way ĠClassification : ĠThe Ġfirst Ġsection Ġof Ġtable Ġ3 Ġshows Ġmacro Ġaverage ĠF 1 - sc ores Ġand Ġaccur acies Ġof Ġprevious Ġworks . ĠThe Ġsecond Ġsection Ġof Ġtable Ġ3 Ġshows Ġthe Ġmulti - class Ġclassification Ġresults Ġof Ġour Ġimplemented Ġbaseline Ġsystems . ĠCons istent Ġwith Ġresults Ġof Ġprevious Ġworks , Ġneural Ġtens ors , Ġwhen Ġapplied Ġto ĠBi - L ST Ms , Ġimproved Ġimplicit Ġdiscourse Ġrelation Ġprediction Ġperformance . ĠHowever , Ġthe Ġperformance Ġon Ġthe Ġthree Ġsmall Ġclasses Ġ( Comp , ĠCont Ġand ĠTemp ) Ġremains Ġlow . </s> The Ġthird Ġsection Ġof Ġtable Ġ3 Ġshows Ġthe Ġmulti - class Ġclassification Ġresults Ġof Ġour Ġproposed Ġparagraph - level Ġneural Ġnetwork Ġmodels Ġthat Ġcapture Ġinter - depend encies Ġamong Ġdiscourse Ġunits . ĠThe Ġfirst Ġrow Ġshows Ġthe Ġperformance Ġof Ġa Ġvariant Ġof Ġour Ġbasic Ġmodel , Ġwhere Ġwe Ġonly Ġidentify Ġimplicit Ġrelations Ġand Ġignore Ġidentifying Ġexplicit Ġrelations Ġby Ġsetting Ġthe Ġ$\ alpha Ġ$ Ġin Ġequation Ġ( 5 ) Ġto Ġbe Ġ0 . ĠCompared Ġwith Ġthe Ġbaseline ĠBi - L ST M Ġmodel , Ġthe Ġonly Ġdifference Ġis Ġthat Ġthis Ġmodel Ġconsiders Ġparagraph - wide Ġcontexts Ġand Ġmodel Ġinter - depend encies Ġamong Ġdiscourse Ġunits Ġwhen Ġbuilding Ġrepresentation Ġfor Ġindividual ĠDU . ĠWe Ġcan Ġsee Ġthat Ġthis Ġmodel Ġhas Ġgreatly Ġimproved Ġimplicit Ġrelation Ġclassification Ġperformance Ġacross Ġall Ġthe Ġfour Ġrelations Ġand Ġimproved Ġthe Ġmacro - average ĠF 1 - score Ġby Ġover Ġ7 Ġper c ents . ĠIn Ġaddition , Ġcompared Ġwith Ġthe Ġbaseline ĠBi - L ST M Ġmodel Ġwith Ġtens or , Ġthis Ġmodel Ġimproved Ġimplicit Ġrelation Ġclassification Ġperformance Ġacross Ġthe Ġthree Ġsmall Ġclasses , Ġwith Ġclear Ġperformance Ġgains Ġof Ġaround Ġ2 Ġand Ġ8 Ġper c ents Ġon Ġcontingency Ġand Ġtemporal Ġrelations Ġrespectively , Ġand Ġoverall Ġimproved Ġthe Ġmacro - average ĠF 1 - score Ġby Ġ2 . 2 Ġper c ents . </s> The Ġsecond Ġrow Ġshows Ġthe Ġperformance Ġof Ġour Ġbasic Ġparagraph - level Ġmodel Ġwhich Ġpredicts Ġboth Ġimplicit Ġand Ġexplicit Ġdiscourse Ġrelations Ġin Ġa Ġparagraph . ĠCompared Ġto Ġthe Ġvariant Ġsystem Ġ( the Ġfirst Ġrow ), Ġthe Ġbasic Ġmodel Ġfurther Ġimproved Ġthe Ġclassification Ġperformance Ġon Ġthe Ġfirst Ġthree Ġimplicit Ġrelations . ĠEspecially Ġon Ġthe Ġcontingency Ġrelation , Ġthe Ġclassification Ġperformance Ġwas Ġimproved Ġby Ġanother Ġ1 . 42 Ġper c ents . ĠMoreover , Ġthe Ġbasic Ġmodel Ġyields Ġgood Ġperformance Ġfor Ġrecognizing Ġexplicit Ġdiscourse Ġrelations Ġas Ġwell , Ġwhich Ġis Ġcomparable Ġwith Ġprevious Ġbest Ġresult Ġ( 92 . 05 % Ġmacro ĠF 1 - score Ġand Ġ93 . 09 % Ġaccuracy Ġas Ġreported Ġin ĠB IB REF 11 Ġ). </s> After Ġunt ying Ġparameters Ġin Ġthe Ġsoft max Ġprediction Ġlayer , Ġimplicit Ġdiscourse Ġrelation Ġclassification Ġperformance Ġwas Ġimproved Ġacross Ġall Ġfour Ġrelations , Ġmeanwhile , Ġthe Ġexplicit Ġdiscourse Ġrelation Ġclassification Ġperformance Ġwas Ġalso Ġimproved . ĠThe ĠCR F Ġlayer Ġfurther Ġimproved Ġimplicit Ġdiscourse Ġrelation Ġrecognition Ġperformance Ġon Ġthe Ġthree Ġsmall Ġclasses . ĠIn Ġsummary , Ġour Ġfull Ġparagraph - level Ġneural Ġnetwork Ġmodel Ġachieves Ġthe Ġbest Ġmacro - average ĠF 1 - score Ġof Ġ48 . 82 % Ġin Ġpredicting Ġimplicit Ġdiscourse Ġrelations , Ġwhich Ġoutper forms Ġprevious Ġneural Ġtens or Ġnetwork Ġmodels Ġ( e . g ., ĠB IB REF 18 Ġ) Ġby Ġmore Ġthan Ġ2 Ġper c ents Ġand Ġoutper forms Ġthe Ġbest Ġprevious Ġsystem ĠB IB REF 19 Ġby Ġ1 Ġpercent . </s> B inary ĠClassification : ĠFrom Ġtable Ġ4 Ġ, Ġwe Ġcan Ġsee Ġthat Ġcompared Ġagainst Ġthe Ġbest Ġprevious Ġsystems , Ġour Ġparagraph - level Ġmodel Ġwith Ġunt ied Ġparameters Ġin Ġthe Ġprediction Ġlayer Ġachieves ĠF 1 - score Ġimprovements Ġof Ġ6 Ġpoints Ġon ĠComparison Ġand Ġ7 Ġpoints Ġon ĠTem poral , Ġwhich Ġdemonstrates Ġthat Ġparagraph - wide Ġcontexts Ġare Ġimportant Ġin Ġdetecting Ġminority Ġdiscourse Ġrelations . ĠNote Ġthat Ġthe ĠCR F Ġlayer Ġof Ġthe Ġmodel Ġis Ġnot Ġsuitable Ġfor Ġbinary Ġclassification . </s> En semble ĠModel </s> As Ġwe Ġexplained Ġin Ġsection Ġ4 . 2 , Ġwe Ġran Ġour Ġmodels Ġfor Ġ10 Ġtimes Ġto Ġobtain Ġstable Ġaverage Ġperformance . ĠThen Ġwe Ġalso Ġcreated Ġensemble Ġmodels Ġby Ġapplying Ġmajority Ġvoting Ġto Ġcombine Ġresults Ġof Ġten Ġruns . ĠFrom Ġtable Ġ5 Ġ, Ġeach Ġensemble Ġmodel Ġob tains Ġperformance Ġimprovements Ġcompared Ġwith Ġsingle Ġmodel . ĠThe Ġfull Ġmodel Ġachieves Ġperformance Ġboosting Ġof Ġ( 51 . 84 Ġ- Ġ48 . 82 Ġ= Ġ3 . 02 ) Ġand Ġ( 94 . 17 Ġ- Ġ93 . 21 Ġ= Ġ0 . 96 ) Ġin Ġmacro ĠF 1 - sc ores Ġfor Ġpredicting Ġimplicit Ġand Ġexplicit Ġdiscourse Ġrelations Ġrespectively . ĠFurthermore , Ġthe Ġensemble Ġmodel Ġachieves Ġthe Ġbest Ġperformance Ġfor Ġpredicting Ġboth Ġimplicit Ġand Ġexplicit Ġdiscourse Ġrelations Ġsimultaneously . </s> Imp act Ġof ĠPar agraph ĠLength </s> To Ġunderstand Ġthe Ġinfluence Ġof Ġparagraph Ġlengths Ġto Ġour Ġparagraph - level Ġmodels , Ġwe Ġdivide Ġparagraphs Ġin Ġthe ĠPD TB Ġtest Ġset Ġinto Ġseveral Ġsubs ets Ġbased Ġon Ġthe Ġnumber Ġof ĠD Us Ġin Ġa Ġparagraph , Ġand Ġthen Ġevaluate Ġour Ġproposed Ġmodels Ġon Ġeach Ġsubset Ġseparately . ĠFrom ĠFigure Ġ4 Ġ, Ġwe Ġcan Ġsee Ġthat Ġour Ġparagraph - level Ġmodels Ġ( the Ġlatter Ġthree ) Ġoverall Ġoutper form ĠDU - pair Ġbas elines Ġacross Ġall Ġthe Ġsubs ets . ĠAs Ġexpected , Ġthe Ġparagraph - level Ġmodels Ġachieve Ġclear Ġperformance Ġgains Ġon Ġlong Ġparagraphs Ġ( with Ġmore Ġthan Ġ5 ĠD Us ) Ġby Ġextensively Ġmodeling Ġmutual Ġinfluences Ġof ĠD Us Ġin Ġa Ġparagraph . ĠBut Ġsomewhat Ġsurprisingly , Ġthe Ġparagraph - level Ġmodels Ġachieve Ġnoticeable Ġperformance Ġgains Ġon Ġshort Ġparagraphs Ġ( with Ġ2 Ġor Ġ3 ĠD Us ) Ġas Ġwell . ĠWe Ġhypothes ize Ġthat Ġby Ġlearning Ġmore Ġappropriate Ġdiscourse - aware ĠDU Ġrepresentations Ġin Ġlong Ġparagraphs , Ġour Ġparagraph - level Ġmodels Ġreduce Ġbias Ġof Ġusing ĠDU Ġrepresentations Ġin Ġpredicting Ġdiscourse Ġrelations , Ġwhich Ġbenefits Ġdiscourse Ġrelation Ġprediction Ġin Ġshort Ġparagraphs Ġas Ġwell . </s> Example ĠAnalysis </s> For Ġthe Ġexample Ġ( Ġ" Impl icit ĠDisc ourse ĠRel ation ĠRecogn ition " Ġ), Ġthe Ġbaseline Ġneural Ġtens or Ġmodel Ġpredicted Ġboth Ġimplicit Ġrelations Ġwrongly Ġ( âĢ ľ Impl icit - Cont ing ency âĢ Ŀ Ġbetween ĠDU 2 Ġand ĠDU 3 ; ĠâĢ ľ Impl icit - Exp ansion âĢ Ŀ Ġbetween ĠDU 3 Ġand ĠDU 4 ), Ġwhile Ġour Ġparagraph - level Ġmodel Ġpredicted Ġall Ġthe Ġfour Ġdiscourse Ġrelations Ġcorrectly , Ġwhich Ġindicates Ġthat Ġparagraph - wide Ġcontexts Ġplay Ġa Ġkey Ġrole Ġin Ġimplicit Ġdiscourse Ġrelation Ġprediction . </s> For Ġanother Ġexample : </s> ( 2 ): Ġ[ Marsh all Ġcame Ġcl anking Ġin Ġlike ĠMar ley 's Ġghost Ġdragging Ġthose Ġchains Ġof Ġbrig ades Ġand Ġair Ġwings Ġand Ġlinks Ġwith ĠArab Ġdesp ots .] Ġ$ _{ D U 1 } $ Ġ( Impl icit - Tem poral ) Ġ[ He Ġwouldn 't Ġleave ] Ġ$ _{ D U 2 } $ Ġuntil Ġ( Expl icit - Tem poral ) Ġ[ Mr . ĠCheney Ġpromised Ġto Ġdo Ġwhatever Ġthe ĠPentagon Ġsystems Ġanalysts Ġtold Ġhim .] Ġ$ _{ D U 3 } $ </s> Our Ġbasic Ġparagraph - level Ġmodel Ġwrongly Ġpredicted Ġthe Ġimplicit Ġdiscourse Ġrelation Ġbetween ĠDU 1 Ġand ĠDU 2 Ġto Ġbe ĠâĢ ľ Impl icit - Compar ison âĢ Ŀ , Ġwithout Ġbeing Ġable Ġto Ġeffectively Ġuse Ġthe Ġsucceeding ĠâĢ ľ Expl icit - Tem poral âĢ Ŀ Ġrelation . ĠOn Ġthe Ġcontrary , Ġthe Ġfull Ġmodel Ġcorrected Ġthis Ġmistake Ġby Ġmodeling Ġdiscourse Ġrelation Ġpatterns Ġwith Ġthe ĠCR F Ġlayer . </s> Conclusion </s> We Ġhave Ġpresented Ġa Ġparagraph - level Ġneural Ġnetwork Ġmodel Ġthat Ġtakes Ġa Ġsequence Ġof Ġdiscourse Ġunits Ġas Ġinput , Ġmodels Ġinter - depend encies Ġbetween Ġdiscourse Ġunits Ġas Ġwell Ġas Ġdiscourse Ġrelation Ġcontinuity Ġand Ġpatterns , Ġand Ġpredicts Ġa Ġsequence Ġof Ġdiscourse Ġrelations Ġin Ġa Ġparagraph . ĠBy Ġbuilding Ġwider - context Ġinformed Ġdiscourse Ġunit Ġrepresentations Ġand Ġcapturing Ġthe Ġoverall Ġdiscourse Ġstructure , Ġthe Ġparagraph - level Ġneural Ġnetwork Ġmodel Ġoutper forms Ġthe Ġbest Ġprevious Ġmodels Ġfor Ġimplicit Ġdiscourse Ġrelation Ġrecognition Ġon Ġthe ĠPD TB Ġdataset . </s> Acknowled gments </s> We Ġacknowledge Ġthe Ġsupport Ġof ĠNVIDIA ĠCorporation Ġfor Ġtheir Ġdonation Ġof Ġone ĠGeForce ĠGTX ĠTIT AN ĠX ĠGPU Ġused Ġfor Ġthis Ġresearch .