Q: What tasks do they experiment with?
Text: Introduction
Learning the distributed representation for long spans of text from its constituents has been a key step for various natural language processing (NLP) tasks, such as text classification BIBREF0 , BIBREF1 , semantic matching BIBREF2 , BIBREF3 , and machine translation BIBREF4 . Existing deep learning approaches take a compositional function with different forms to compose word vectors recursively until obtaining a sentential representation. Typically, these compositional functions involve recurrent neural networks BIBREF5 , BIBREF6 , convolutional neural networks BIBREF7 , BIBREF8 , and tree-structured neural networks BIBREF9 , BIBREF10 .
Among these methods, tree-structured neural networks (Tree-NNs) show theirs superior performance in many NLP tasks BIBREF11 , BIBREF12 . Following the syntactic tree structure, Tree-NNs assign a fixed-length vector to each word at the leaves of the tree, and combine word and phrase pairs recursively to create intermediate node vectors, eventually obtaining one final vector to represent the whole sentence.
However, these models have a major limitation in their inability to fully capture the richness of compositionality BIBREF13 . The same parameters are used for all kinds of semantic compositions, even though the compositions have different characteristics in nature. For example, the composition of the adjective and the noun differs significantly from the composition of the verb and the noun. Moreover, many semantic phenomena, such as semantic idiomaticity or transparency, call for more powerful compositional mechanisms BIBREF14 . Therefore, Tree-NNs suffer from the underfitting problem.
To alleviate this problem, some researchers propose to use multiple compositional functions, which are arranged beforehand according to some partition criterion BIBREF11 , BIBREF13 , BIBREF15 . Intuitively, using different parameters for different types of compositions has the potential to greatly reduce underfitting. BIBREF13 [ BIBREF13 ] defined different compositional functions in terms of syntactic categories, and a suitable compositional function is selected based on the syntactic categories. BIBREF15 [ BIBREF15 ] introduced multiple compositional functions and during compositional phase, a proper one is selected based on the input information. Although these models accomplished their mission to a certain extent, they still suffer from the following three challenges. First, the predefined compositional functions cannot cover all the compositional rules; Second, they require more learnable parameters, suffering from the problem of overfitting; Third, it is difficult to determine a universal criterion for semantic composition based solely on syntactic categories.
In this paper, we propose dynamic compositional neural networks over tree structure, in which a meta network is used to generate the context-specific parameters of a dynamic compositional network. Specifically, we construct our models based on two kinds of tree-structured neural networks: recursive neural network (Tree-RecNN) BIBREF11 and tree-structure long short-term memory neural network (Tree-LSTM) BIBREF9 . Our work is inspired by recent work on dynamic parameter prediction BIBREF16 , BIBREF17 , BIBREF18 . The meta network is used to extract the shared meta-knowledge across different compositional rules and to dynamically generate the context-specific compositional function. Thus, the compositional function of our models varies with positions, contexts and samples. The dynamic compositional network then applies those context-specific parameters to the current input information. Both meta and dynamic networks are differentiable such that the overall networks can be trained in an end-to-end fashion. Additional, to reduce the complexity of the whole networks, we define the dynamic weight matrix in a manner simulating low-rank matrix decomposition.
We evaluate our models on two typical tasks: text classification and text semantic matching. The results show that our models are more expressive due to their learning to learn nature, yet without increasing the number of model's parameters. Moreover, we find certain composition operations can be learned implicitly by meta TreeNN, such as the composition of noun phrases and verb phrases.
The contributions of the paper can be summed up as follows.
Tree-Structured Neural Network
In this section, we briefly describe the tree-structured neural networks.
The idea of tree-structured neural networks for natural language processing (NLP) is to train a deep learning model with a grammatical tree structure BIBREF19 that can be applied to phrases and sentences. At every node in the tree, the contexts of the left and right children are combined by a compositional function. The parameters of the compositional function are shared across all nodes in the tree. The layer computed at the top node gives a representation for the whole sentence.
Vanilla Recursive Neural Network
The simplest member of tree-structured NN is the vanilla recursive neural network BIBREF13 , in which the representation of parent is calculated by weighted linear combination of the child vectors.
Formally, given a binary constituency tree $T$ induced by a sentence, each non-leaf node corresponds to a phrase. We refer to $\mathbf {h}_{j} \in \mathbb {R}^{d}$ as the hidden state of each node $j$ , and let $\mathbf {h}_{j}^{l}$ , $\mathbf {h}_{j}^{r}$ denote the left and right child representations respectively.
$$\mathbf {h}_{j} = \tanh \left( \mathbf {W} \begin{bmatrix} \mathbf {h}_{j}^{l} \\ \mathbf {h}_{j}^{r} \\ \end{bmatrix} + \mathbf {b} \right),$$   (Eq. 5)
where $\mathbf {W} \in \mathbb {R}^{d \times 2d}$ is a learnable compositional matrix, $\mathbf {b}$ is the bias vector.
Tree LSTM
Tree LSTM BIBREF9 is a generalization of LSTMs to tree-structured network topologies. In this model, the compositional function is an LSTM unit, and the hidden state $h_j$ of each node can be computed as follows: we refer to $\mathbf {h}_{j}$ and $\mathbf {c}_{j}$ as the hidden state and memory cell of each node $j$ . The transition equations of node $j$ are as follows:
$$\begin{bmatrix} \mathbf {\tilde{c}}_{j} \\ \mathbf {o}_{j} \\ \mathbf {i}_{j} \\ \mathbf {f}_{j}^{l} \\ \mathbf {f}_{j}^{r} \end{bmatrix} &= \begin{bmatrix} \tanh \\ \sigma \\ \sigma \\ \sigma \\ \sigma \end{bmatrix} \left( \mathbf {W} \begin{bmatrix} \mathbf {x}_{j} \\ \mathbf {h}_{j}^{l} \\ \mathbf {h}_{j}^{r} \\ \end{bmatrix} + \mathbf {b} \right),\\ \mathbf {c}_{j} &= \mathbf {\tilde{c}}_{j} \odot \mathbf {i}_{j} + \mathbf {c}_{j}^{l} \odot \mathbf {f}_{j}^{l}+ \mathbf {c}_{j}^{r} \odot \mathbf {f}_{j}^{r} , \\ \mathbf {h}_{j} &= \mathbf {o}_{j} \odot \tanh \left( \mathbf {c}_{j} \right),$$   (Eq. 8)
where $\mathbf {x}_j \in \mathbb {R}^{d}$ denotes the input vector and is non-zero if and only if it is a leaf node. The superscript $l$ and $r$ represent the left child and right child respectively. $\sigma $ represents the logistic sigmoid function and $\odot $ denotes element-wise multiplication. $\mathbf {W} \in \mathbb {R}^{5d \times 3d}$ and $\mathbf {b} \in \mathbb {R}^{5d}$ are learnable parameters.
Dynamic Compositional Neural Network
In the above two tree-structured NNs, the compositional function is shared across all nodes in the tree, which results in underfitting since the semantic compositions have great diversities. To address this problem, we propose two dynamic compositional neural networks over tree structure, which dynamically generate different parameters for different types of compositions. Figure 2 shows an illustration of the dynamic compositional neural network, consisting of two components: (1) meta network and (2) basic network with dynamic parameters.
Specifically, we propose two meta networks to generate the context-specific compositional functions for RecNN and TreeLSTM respectively.
Meta Network for RecNN
For RecNN, we replace the static parameters $\mathbf {W}$ and $\mathbf {b}$ in Eq.( 5 ) with the dynamic parameters $\mathbf {W}(\mathbf {z}_{j})$ and $\mathbf {b}(\mathbf {z}_{j})$ , which are generated by a meta network. The meta network is a smaller RecNN, and the hidden state $\hat{\mathbf {h}}_{j}\in \mathbb {R}^{m}$ of node $j$ in meta network is defined as
$$\hat{\mathbf {h}}_{j} &= \tanh \left( \mathbf {W}_{m} \mathcal {H}_{j} + \mathbf {b}_{m} \right),\\ \mathbf {z}_{j} &= \mathbf {W}_{z} \hat{\mathbf {h}}_{j}$$   (Eq. 11)
where $\mathcal {H}_{j} = \mathbf {h}_{j}^{l}\oplus \mathbf {h}_{j}^{r}\oplus \mathbf {\hat{h}}_{j}^{l}\oplus \mathbf {\hat{h}}_{j}^{r} \in \mathbb {R}^{2m+2d}$ , $\mathbf {W}_{m} \in \mathbb {R}^{m \times (2d+2m)}$ and $\mathbf {b}_{m} \in \mathbb {R}^{m}$ are parameters of meta RecNN; $\mathbf {W}_{z} \in \mathbb {R}^{z \times m}$ is a scale matrix.
To reduce the number of the parameters, we define the dynamic parameters with a low-rank factorized representation of the weights, analogous to the Singular Value Decomposition. The dynamic parameters $\mathbf {W}(\mathbf {z}_{j})$ and $\mathbf {b}(\mathbf {z}_{j})$ of the basic RecNN are computed by:
$$\mathbf {W}(\mathbf {z}_{j}) &= \begin{bmatrix} P_l \mathbf {D}(\mathbf {z}_j)Q_l \\ P_r \mathbf {D}(\mathbf {z}_j)Q_r \\ \end{bmatrix} \\ \mathbf {b}(\mathbf {z}_{j}) &= \begin{bmatrix} B_l\mathbf {z}_j \\ B_r\mathbf {z}_j \\ \end{bmatrix}$$   (Eq. 12)
where $P \in \mathbb {R}^{d \times z}$ , $Q \in \mathbb {R}^{z \times d}$ , and $\mathbf {D}(\mathbf {z}_t) \in \mathbb {R}^{z \times z}$ is the diagonal matrix of $\mathbf {z}$ .
Thus, our dynamic RecNN needs $(6dz + mz)$ parameters, while the vanilla RecNN has $(2d^2+d)$ parameters. With a small $z$ and $m$ , our dynamic RecNN needs less parameters than the vanilla RecNN. For example, if we set $d =100$ and $z=m=20$ , our model needs $12,400$ parameters while the vanilla model needs $20,100$ parameters.
Meta Network for TreeLSTM
Likewise, we also use a smaller meta network to generate the static parameters $\mathbf {W}$ and $\mathbf {b}$ in Eq.( 8 ) with the dynamic parameters $\mathbf {W}(\mathbf {z}_{j})$ and $\mathbf {b}(\mathbf {z}_{j})$ . The meta network is a smaller TreeLSTM, and the hidden state $\hat{\mathbf {h}}_{j}\in \mathbb {R}^{m}$ of node $j$ in meta network is defined as
$$\begin{bmatrix} \mathbf {\hat{g}}_{j} \\ \mathbf {\hat{o}}_{j} \\ \mathbf {\hat{i}}_{j} \\ \mathbf {\hat{f}}_{j}^{l} \\ \mathbf {\hat{f}}_{j}^{r} \end{bmatrix} &= \begin{bmatrix} \tanh \\ \sigma \\ \sigma \\ \sigma \\ \sigma \end{bmatrix} \left( \mathbf {W}_{m} \begin{bmatrix} \mathbf {x}_{j} \\ \mathcal {H}_{j} \\ \end{bmatrix} + \mathbf {b}_{m} \right) ,\\ \mathbf {\hat{c}}_{j} &= \mathbf {\hat{g}}_{j} \odot \mathbf {\hat{i}}_{j} + \mathbf {\hat{g}}_{j}^{l} \odot \mathbf {\hat{f}}_{j}^{l}+ \mathbf {\hat{g}}_{j}^{r} \odot \mathbf {\hat{f}}_{j}^{r} , \\ \mathbf {\hat{h}}_{j} &= \mathbf {\hat{o}}_{j} \odot \tanh \left( \mathbf {\hat{c}}_{j} \right), \\ \mathbf {z}_{j} &= \mathbf {W}_{z} \hat{\mathbf {h}}_{j}$$   (Eq. 14)
where $\mathcal {H}_{j} = \mathbf {h}_{j}^{l}\oplus \mathbf {h}_{j}^{r}\oplus \mathbf {\hat{h}}_{j}^{l}\oplus \mathbf {\hat{h}}_{j}^{r} \in \mathbb {R}^{2m+2d}$ ; $\mathbf {W}_{m} \in \mathbb {R}^{5m \times (3d+2m)}$ and $\mathbf {b}_{m} \in \mathbb {R}^{m}$ are parameters of meta TreeLSTM; $\mathbf {W}_{z} \in \mathbb {R}^{z \times m}$ is a scale matrix.
The dynamic parameters $\mathbf {W}(\mathbf {z}_{j})$ and $\mathbf {b}(\mathbf {z}_{j})$ of basic TreeLSTM are computed by:
$$\mathbf {W}(\mathbf {z}_{j}) &= \left[\mathbf {W}^{g}, \mathbf {W}^{i}, \mathbf {W}^{f^l},\mathbf {W}^{f^r},\mathbf {W}^{o}\right] \\ \mathbf {b}(\mathbf {z}_{j}) &= \left[\mathbf {b}^{g}, \mathbf {b}^{i}, \mathbf {b}^{f^l},\mathbf {b}^{f^r},\mathbf {b}^{o}\right],$$   (Eq. 15)
where for $*\in \lbrace c,o,i,f^l,f^r\rbrace $ ,
$$\mathbf {W}^{*}(\mathbf {z}_{j}) &= \begin{bmatrix} P_x^{*} \mathbf {D}(\mathbf {z}_t)Q_x^{*} \\ P_l^{*} \mathbf {D}(\mathbf {z}_t)Q_l^{*} \\ P_r^{*} \mathbf {D}(\mathbf {z}_t)Q_r^{*} \\ \end{bmatrix}, \\ \mathbf {b}^{*}(\mathbf {z}_{j}) &= \begin{bmatrix} B_x^{*}\mathbf {z}_t \\ B_l^{*}\mathbf {z}_t \\ B_r^{*}\mathbf {z}_t \\ \end{bmatrix},$$   (Eq. 16)
where $P \in \mathbb {R}^{5d \times z}$ , $Q \in \mathbb {R}^{z \times 3d}$ , $B \in \mathbb {R}^{5d \times z}$ , and $\mathbf {D}(\mathbf {z}_t) \in \mathbb {R}^{z \times z}$ is the diagonal matrix of $\mathbf {z}$ .
With a small $z$ and $m$ , our dynamic TreeLSTM needs a similar amount of parameters compared to the standard TreeLSTM.
Application of Dynamic Compositional Neural Networks
In this section, we describe two specific models to show the applications of dynamic compositional neural networks for two typical tasks in NLP.
Text Classification
The purpose of text classification is that, given a sentence $x$ , the model should predict labels $\hat{y}$ from a pre-defined label set $\mathcal {Y}$ . From the description in the previous section, we can compute the distributed representation $\mathbf {h}_j$ of the phrase at node $j$ of a tree:
$$\mathbf {h}_{j} &= \operatornamewithlimits{DC-TreeNN}(\mathbf {x}_j, \mathbf {h}_{j}^l, \mathbf {h}_{j}^r,\theta )$$   (Eq. 18)
After this recursive process, the hidden state $\mathbf {h}_R$ at the root node is used as the sentential representation, which then followed by a softmax classifier to predict the probability distribution over classes.
$${\hat{\mathbf {y}}} = \operatornamewithlimits{softmax}(\mathbf {W}_t \mathbf {h}_R + \mathbf {b}_t)$$   (Eq. 19)
where ${\hat{\mathbf {y}}}$ is prediction probabilities, $\mathbf {W}_t$ and $\mathbf {b}_t$ are the parameters of the classifier.
We evaluate our models on five different datasets. The detailed statistics about the five datasets are listed in Table 2 . Each dataset is briefly described as follows.
SST The movie reviews with two classes (negative, positive) in the Stanford Sentiment Treebank BIBREF23 .
MR The movie reviews with two classes BIBREF24 .
QC The TREC questions dataset involves six different question types. BIBREF25 .
SUBJ Subjectivity dataset where the goal is to classify each instance (snippet) as being subjective or objective. BIBREF26
IE Idiom enhanced sentiment classification. BIBREF27 . Each sentence contains at least one idiom.
As shown in Table 3 , DC-TreeLSTM consistently outperforms RecNN, MV-RecNN, RNTN, and TreeLSTM by a large margin while achieving comparable results to the CNN and using much fewer parameters.(The number of parameters in our models is approximately 10K while in CNN the number of parameters is about 400K). Compared with RecNN, DC-RecNN performs better, indicating the effectiveness of the dynamic compositional function. Additionally, both DC-RecNN and DC-TreeLSTM achieve substantial improvement on IE dataset, which covers the richness of compositionality (idiomaticity). We attribute the success on IE to its power in modeling more complicated compositionality.
Text Semantic Matching
Among many natural language processing (NLP) tasks, a common problem is modelling the relevance of a pair of texts. In this section, we show how to effectively use the dynamic compositional neural networks to model the semantic relationship between two sentences.
As shown in Figure 1 , given two sentences $x_a$ and $x_b$ , the representation of each sentence $\mathbf {h}_{R}$ can be computed by one basic TreeNN.
$$\mathbf {h}_{R}^{(a)} &= \operatornamewithlimits{DC-TreeNN}(\mathbf {x}_R, \mathbf {h}_{R}^l, \mathbf {h}_{R}^r,\theta _{a}) \\ \mathbf {h}_{R}^{(b)} &= \operatornamewithlimits{DC-TreeNN}(\mathbf {x}_R, \mathbf {h}_{R}^l, \mathbf {h}_{R}^r,\theta _{b})$$   (Eq. 21)
where $R$ denotes the root node of a tree. $\theta _{a}$ and $\theta _{b}$ are generated by a shared meta TreeNN. Then, the representation of each sentence will be fed into a multi-layer perceptron to obtain a unified representation for the final relationship classification.
The sample-specific but shared meta TreeNN ensures that, on the one hand we can dynamically model the diversity of semantic compositionality, on the other hand we can capture the general rules across different samples.
We choose the dataset of Sentences Involving Compositional Knowledge (SICK), which is proposed by BIBREF30 [ BIBREF30 ] aiming at evaluation of compositional distributional semantic models. The dataset consists of 9927 sentence pairs in a 4500/500/4927 train/dev/test split, in which each sentence pairs are pre-defined into three labels: “entailment”,“contradiction” and “neutral”.
Our results are summarized in Table 4 , where the performance of NBOW, LSTM, RecNN, and RNTN are reported by BIBREF31 , BIBREF32 . For fair comparison, we train our models with the same setting. We can see both DC-RecNN and DC-TreeLSTM outperform competitor models, in which DC-RecNN (DC-TreeLSTM) achieves 3% (2.7%) improvements than RecNN (TreeLSTM). We think this breakthrough is basically attributed to the dynamic compositional mechanism, which enables our models to capture various syntactic patterns (As we will discuss later) therefore can more accurately understand sentences.
Experiment
To make a comprehensive evaluation, we assess our model on five text classification tasks and a semantic matching task.
Training and Hyperparameters
Given a sentence (or sentence pair) and its label, the output of a neural network is the probabilities of the different classes. The parameters of the network are trained to minimise the cross-entropy of the predicted and true label distributions. To minimize the objective, we use stochastic gradient descent with the diagonal variant of AdaGrad BIBREF20 .
The word embeddings for all of the models are initialized with GloVe vectors BIBREF21 . The other parameters are initialized by randomly sampling from uniform distribution in $[-0.1, 0.1]$ .
The final hyper-parameters are as follows. The initial learning rate is $0.1$ . The regularization weight of the parameters is $1E{-5}$ and the others are listed as Table 1 .
For all the sentences in the datasets, we parse them with constituency parser BIBREF22 to obtain the trees for our models and some competitor models.
Competitor Methods
RecNN BIBREF11 : Recursive neural network with standard compositional function.
RNTN BIBREF23 : The RNTN is a recursive neural network with neural tensor layer, which can model strong interactions between two constituents.
MV-RecNN BIBREF11 : The MV-RecNN is to represent every word and longer phrase in a parse tree as both a vector and a matrix in order to model rich compositionality.
TreeLSTM BIBREF9 : Recursive neural network with Long Short-Term Memory unit.
Discussion and Qualitative Analysis
In our models, the latent vector $\mathbf {z}$ controls the process of predicting network's parameters and its dimensionality determines the number of model's parameters. Next, we will investigate how the controlling vector $\mathbf {z}$ influences the performance of our models.
Figure 4 shows the accuracies of DC-RecNN across the different dimensions of $[5, 10, \dots , 50]$ for the controlling vector $\mathbf {z}$ on five datasets. We get the following findings:
For all five datasets, the model can achieve considerable performances even when the size of vector $\mathbf {z}$ is reduced to 5. Particularly, for the dataset QC, the model obtains $87.0\%$ accuracy with a pretty small meta Tree-RecNN, suggesting a smaller meta network can be used for generating a more powerful compositional function to effectively model sentence.
When dealing with the dataset with more labels, larger vector size leads to a better performance. For example, the performance on IE and QC datasets reaches the maximum when the size of $z$ equals 40, while for the other three datasets MR, SST and SUBJ, the model obtains the best performance with the value of 30, 30 and 20 respectively.
As described in previous sections, we know the compositional function is changed cross child nodes over a tree, which is controlled by a latent vector $z$ . To get an intuitive understanding of how the controlling vector $z$ works, we design an experiment to examine the neuron's behaviours of $\mathbf {z}$ on each node. More concretely, we refer to $z_{jk}$ as the activation of the $k$ -neuron at node $j$ , where $j \in \lbrace 1,\ldots ,N\rbrace $ and $k \in \lbrace 1,\ldots , z\rbrace $ . Then we randomly sample some sentences on the development set from the datasets we used. By visualizing the latent vector $\mathbf {z}_{j}$ and analyzing the maximum activation, we can find what kinds of patterns the current neuron focuses on.
Table 5 illustrates multiple interpretable neurons and some representative words or phrases which can activate these neurons. We can observe that:
For some simple tasks such as text classification, meta network will integrate useful semantic information into the the generation process of compositional function. These semantic bias before composition are task-specific.
For example, the 21- $st$ neuron is more sensitive to emotional terms, which can be understood as a sentinel, telling the basic neural network that an informative phrase is coming, more attention should be paid in the process of composition. Figure 5 -(a) shows a visualization. We can see in this sentence, the neuron has realized that this idiomatic collocation “in stitches” is a key pattern, which is crucial for the final sentiment prediction.
For more complicated tasks such as semantic matching, a well-grounded understanding of the syntactic structure is crucial. In this context, we find that a meta network could capture some syntactic information. For example, the 27- $th$ neuron monitors phrases constructed by light-verb. As shown in Figure 5 -(b), the verb phrase “taking off” has been attended for forthcoming compositional operation, which is more useful for judging the semantic relation between the sentence pair “An airplane is taking off/A plane is landing”.
Related Work
One thread of related work is the exploration of different kinds of compositional function over tree structures. BIBREF11 [ BIBREF11 ] proposed the recursive neural network with standard compositional function. After that, some extensions are introduced to enhance the expressive power of compositional function, such as MV-RecNN BIBREF23 , SU-RNN BIBREF13 , RNTN BIBREF23 , while these models suffer from the problem of hard-coded compositional operations and overfitting.
Another thread of work is the idea of using one network to direct the learning of another network BIBREF16 . BIBREF33 [ BIBREF33 ] introduce a meta neural network to provide another network with a step size and a direction vector, which is helpful for parameter optimization. BIBREF16 [ BIBREF16 ] propose the dynamic filter network to implicitly learn a variety of filtering operations. BIBREF17 [ BIBREF17 ] introduce a learnet for one-shot learning, which can predict the parameters of a second network given a single exemplar. BIBREF18 [ BIBREF18 ] propose the model hypernetwork, which uses a small network to generate the weights for a larger network.
Different from these models, we employ the idea of parameter generation to address the limitation of weight-sharing or partially sharing paradigm of tree-based compositional models.
Conclusion
In this work, we introduce a meta neural network, which can generate a compositional network to dynamically compose constituents over tree structure. The parameters of compositional function vary from position to position and from sample to sample, allowing for more sophisticated operations on the input.
To evaluate our models, we choose two typical NLP tasks involving six datasets. The qualitative and quantitative experiment results demonstrate the effectiveness of our models.
Acknowledgments
We would like to thank the anonymous reviewers for their valuable comments and thank Kaiyu Qian, Jiachen Xu, Jifan Chen for useful discussions. This work was partially funded by National Natural Science Foundation of China (No. 61532011 and 61672162), Shanghai Municipal Science and Technology Commission (No. 16JC1420401).
text classification and text semantic matching