Entity-Augmented Distributional Semantics for Discourse Relations

Discourse relations bind smaller linguistic elements into coherent texts. However, automatically identifying discourse relations is difficult, because it requires understanding the semantics of the linked sentences. A more subtle challenge is that it is not enough to represent the meaning of each sentence of a discourse relation, because the relation may depend on links between lower-level elements, such as entity mentions. Our solution computes distributional meaning representations by composition up the syntactic parse tree. A key difference from previous work on compositional distributional semantics is that we also compute representations for entity mentions, using a novel downward compositional pass. Discourse relations are predicted not only from the distributional representations of the sentences, but also of their coreferent entity mentions. The resulting system obtains substantial improvements over the previous state-of-the-art in predicting implicit discourse relations in the Penn Discourse Treebank.


INTRODUCTION
The high-level organization of text can be characterized in terms of discourse relations between adjacent spans of text (Knott, 1996;Mann, 1984;Webber et al., 1999). Identifying these relations has been shown to be relevant to tasks such as summarization (Louis et al., 2010;Yoshida et al., 2014), sentiment analysis (Somasundaran et al., 2009), and coherence evaluation . While the Penn Discourse Treebank (PDTB) now provides a large dataset annotated for discourse relations (Prasad et al., 2008), the automatic identification of implicit discourse relations is a difficult task, with state-of-the-art performance at roughly 40% (Lin et al., 2009).
One reason for this poor performance is that predicting implicit discourse relations is a fundamentally semantic task, and the relevant semantics may be difficult to recover from surface level features. For example, consider the discourse relation between the following two sentences in Example (1), where a discourse connector like "because" seems appropriate to indicate the relationship. However, without discourse connector, there is little surface information to signal the relationship. We address this issue by applying a discriminatively-trained model of compositional distributional semantics to discourse relation classification (Socher et al., 2013;Baroni et al., 2014). The meaning of each sentence is represented as a vector (Turney et al., 2010), which is computed through a series of compositional operations over the parse tree.
She was hungry. He was hungry.
We further argue that purely vector-based representations on sentences are insufficiently expressive to capture discourse relations. To see why, consider what happens in Example (2), where a tiny change is made based on Example (1). After changing the subject of the second sentence to Bob, the original discourse relation seems no longer holding in Example (2). But despite the radical difference in meaning, the distributional representation of the second sentence will be almost unchanged: the syntactic structure remains identical, and the words "he" and "she" have very similar word representations (Ji & Eisenstein, 2014). We address this issue by computing vector representations not only for each sentence, but also for each coreferent entity mention within the sentences.
These representations are meant to capture the role played by the entity in the text. We compute entity-role representations using a novel feed-forward compositional model, which combines upward and downward passes through the syntactic structure. Representations for these coreferent mentions are then combined into a classification model, and help to predict the implicit discourse relation. In combination, our approach achieves a 3% improvement in accuracy over the best previous work (Lin et al., 2009) on the second-level discourse relation identification in the PDTB 1 .

ENTITY AUGMENTED DISTRIBUTIONAL SEMANTICS FOR RELATION IDENTIFICATION
We briefly describe our approach to entity-augmented distributional semantics and to discourse relation identification. Our relation identification model is named as DISCO2, since it is a distributional compositional approach to discourse relations.

ENTITY AUGMENTED DISTRIBUTIONAL SEMANTICS
The entity-augmented distributional semantics includes two passes in composition procedure: the upward pass for distributional representation of sentence, while the downward pass for distributional representation of entities shared between sentences.
Upward pass Distributional representations for sentences are computed in a feed-forward upward pass: each non-terminal in the binarized syntactic parse tree has a K-dimensional distributional representation that is computed from the distributional representations of its children, bottoming out in representations of individual words. We follow the Recursive Neural Network (RNN) model proposed by Socher et al. (2011). Specifically, for a given parent node i, we denote the left child as ℓ(i), and the right child as r(i). We compose their representations to obtain, where tanh (·) is the element-wise hyperbolic tangent function, and U ∈ R K×2K is the upward composition matrix. We apply this compositional procedure from the bottom up, ultimately obtaining the sentence-level representation u 0 . (1) and (2), a model that uses a single vector representation for each sentence would find little to distinguish between "she was hungry" and "he was hungry". It would therefore almost certainly fail to identify the correct discourse relation for at least one of these cases, which requires tracking the roles played by the entities that are coreferent in each pair of sentences. To address this issue, we augment the representation of each sentence with additional vectors, representing the semantics of the role played by each coreferent entity in each sentence. Rather than represent this information in a logical form -which would require robust parsing to a logical representation -we represent it through additional distributional vectors. The role of a constituent i can be viewed as a combination of information from two neighboring nodes in the parse tree: its parent ρ(i), and its sibling s(i). We can make a downward pass, computing the downward vector d i from the downward vector of the parent d ρ(i) , and the upward vector of the sibling u s(i) :

Downward pass As seen in the contrast between Examples
The base case of this recursive procedure occurs at the root of the parse tree, which is set equal to the upward representation, d 0 u 0 .

RELATION IDENTIFICATION MODEL
To predict the discourse relation between an sentence pair (m, n), the decision function is a sum of bilinear products, where the predicted relation is given byŷ = arg max y∈Y ψ(y), and A y , B y ∈ R K×K are the classification parameters for relation y. A scalar b y is used as the bias term for relation y, and A(m, n) is the set of coreferent entity mentions shared among the sentence pair (m, n). For the Model +Entity semantics +Surface features K Accuracy(%)  Table 1: Experimental results on multiclass classification of second-level discourse relations. The results of Lin et al. (2009) are shown in line 1; the results for our reimplementation of this system are shown in line 2.
cases where there are no coreferent entity mentions between two sentences, A(m, n) = ∅, the classification model considers only the upward vectors at the root. We also use the surface features vector φ (m,n) in the decision function, as we find that, this approach outperforms prior work on the classification of implicit discourse relations in the PDTB, when combined with a small number of surface features.

EXPERIMENTS
We evaluate our approach on the implicit discourse relation identification in the Penn Discourse Treebank (PDTB). PDTB relations may be explicit, meaning that they are signaled by discourse connectives (e.g., because); alternatively, they may be implicit, meaning that the connective is absent. We focus on the more challenging problem of classifying implicit discourse relations. Aiming to build a discourse parser in future, we follow the same experimental setting proposed by Lin et al. (2009), and evaluate our relation identification model on the second-level relation types.
We run the Stanford parser (Klein & Manning, 2003) and the Berkeley coreference system (Durrett & Klein, 2013) to obtain syntactic trees and coreference results respectively. In the PDTB, each discourse relation is annotated between two argument spans. For non-sentence argument span, we identify the syntactic subtrees with the span, and construct a right-branching superstructure to unify them into a tree. Table 1 presents results for multiclass identification of second-level PDTB relations. As shown in lines 5 and 6, DISCO2 outperforms the prior state-of-the-art (line 1). The strongest performance is obtained by including the entity distributional semantics, with a 3.4% improvement over the accuracy reported by Lin et al. (2009) (p < .05). The improvement over our reimplementation of this work (line 2) is even greater, which shows how the distributional representation provides additional value over the surface features. The contribution of entity semantics is shown in Table 1 by the accuracy differences between lines 3 and 4, and between lines 5 and 6. More experimental results and discussion are in (Ji & Eisenstein, 2014).