We introduce the task of predicting adverbial presupposition triggers such as also and again . Solving such a task requires detecting recurring or similar events in the discourse context , and has applications in natural language generation tasks such as summarization and dialogue systems . We create two new datasets for the task , derived from the Penn Treebank and the Annotated English Gigaword corpora , as well as a novel attention mechanism tailored to this task . Our attention mechanism augments a baseline recurrent neural network without the need for additional trainable parameters , minimizing the added computational cost of our mechanism . We demonstrate that our model statistically outperforms a number of baselines , including an LSTM-based language model . In pragmatics , presuppositions are assumptions or beliefs in the common ground between discourse participants when an utterance is made ( Frege , 1892 ; Strawson , 1950 ; Stalnaker , 1973 Stalnaker , , 1998 ) ) , and are ubiquitous in naturally occurring discourses ( Beaver and Geurts , 2014 ) . Presuppositions underly spoken statements and written sentences and understanding them facilitates smooth communication . We refer to expressions that indicate the presence of presuppositions as presupposition triggers . These include definite descriptions , factive verbs and certain adverbs , among others . For example , consider the following statements : ( 1 ) John is going to the restaurant again . * Authors ( listed in alphabetical order ) contributed equally . ( 2 ) John has been to the restaurant . ( 1 ) is only appropriate in the context where ( 2 ) is held to be true because of the presence of the presupposition trigger again . One distinguishing characteristic of presupposition is that it is unaffected by negation of the presupposing context , unlike other semantic phenomena such as entailment and implicature . The negation of ( 1 ) , John is not going to the restaurant again . , also presupposes ( 2 ) . Our focus in this paper is on adverbial presupposition triggers such as again , also and still . Adverbial presupposition triggers indicate the recurrence , continuation , or termination of an event in the discourse context , or the presence of a similar event . In one study of presuppositional triggers in English journalistic texts ( Khaleel , 2010 ) , adverbial triggers were found to be the most commonly occurring presupposition triggers after existential triggers . 1 Despite their frequency , there has been little work on these triggers in the computational literature from a statistical , corpus-driven perspective . As a first step towards language technology systems capable of understanding and using presuppositions , we propose to investigate the detection of contexts in which these triggers can be used . This task constitutes an interesting testing ground for pragmatic reasoning , because the cues that are indicative of contexts containing recurring or similar events are complex and often span more than one sentence , as illustrated in Sentences ( 1 ) and ( 2 ) . Moreover , such a task has immediate practical consequences . For example , in language generation applications such as summarization and dialogue systems , adding presuppositional triggers in contextually appropriate loca-1 Presupposition of existence are triggered by possessive constructions , names or definite noun phrases . tions can improve the readability and coherence of the generated output . We create two datasets based on the Penn Treebank corpus ( Marcus et al . , 1993 ) and the English Gigaword corpus ( Graff et al . , 2007 ) , extracting contexts that include presupposition triggers as well as other similar contexts that do not , in order to form a binary classification task . In creating our datasets , we consider a set of five target adverbs : too , again , also , still , and yet . We focus on these adverbs in our investigation because these triggers are well known in the existing linguistic literature and commonly triggering presuppositions . We control for a number of potential confounding factors , such as class balance , and the syntactic governor of the triggering adverb , so that models can not exploit these correlating factors without any actual understanding of the presuppositional properties of the context . We test a number of standard baseline classifiers on these datasets , including a logistic regression model and deep learning methods based on recurrent neural networks ( RNN ) and convolutional neural networks ( CNN ) . In addition , we investigate the potential of attention-based deep learning models for detecting adverbial triggers . Attention is a promising approach to this task because it allows a model to weigh information from multiple points in the previous context and infer long-range dependencies in the data ( Bahdanau et al . , 2015 ) . For example , the model could learn to detect multiple instances involving John and restaurants , which would be a good indication that again is appropriate in that context . Also , an attention-based RNN has achieved success in predicting article definiteness , which involves another class of presupposition triggers ( Kabbara et al . , 2016 ) . As another contribution , we introduce a new weighted pooling attention mechanism designed for predicting adverbial presupposition triggers . Our attention mechanism allows for a weighted averaging of our RNN hidden states where the weights are informed by the inputs , as opposed to a simple unweighted averaging . Our model uses a form of self-attention ( Paulus et al . , 2018 ; Vaswani et al . , 2017 ) , where the input sequence acts as both the attention mechanism 's query and key / value . Unlike other attention models , instead of simply averaging the scores to be weighted , our approach aggregates ( learned ) attention scores by learning a reweighting scheme of those scores through another level ( dimension ) of attention . Additionally , our mechanism does not introduce any new parameters when compared to our LSTM baseline , reducing its computational impact . We compare our model using the novel attention mechanism against the baseline classifiers in terms of prediction accuracy . Our model outperforms these baselines for most of the triggers on the two datasets , achieving 82.42 % accuracy on predicting the adverb " also " on the Gigaword dataset . The contributions of this work are as follows : 1 . We introduce the task of predicting adverbial presupposition triggers . 2 . We present new datasets for the task of detecting adverbial presupposition triggers , with a data extraction method that can be applied to other similar pre-processing tasks . 3 . We develop a new attention mechanism in an RNN architecture that is appropriate for the prediction of adverbial presupposition triggers , and show that its use results in better prediction performance over a number of baselines without introducing additional parameters . 2 Related Work In this work , we have investigated the task of predicting adverbial presupposition triggers and introduced several datasets for the task . Additionally , we have presented a novel weighted-pooling attention mechanism which is incorporated into a recurrent neural network model for predicting the presence of an adverbial presuppositional trigger . Our results show that the model outperforms the CNN and LSTM , and does not add any additional parameters over the standard LSTM model . This shows its promise in classification tasks involving capturing and combining relevant information from multiple points in the previous context . In future work , we would like to focus more on designing models that can deal with and be optimized for scenarios with severe data imbalance . We would like to also explore various applications of presupposition trigger prediction in language generation applications , as well as additional attention-based neural network architectures .
Adverbaial triggers indicate the event recurrence, continuation, or termination in the discourse context and are frequently found in English but there are few related works.
They introduce an adverbial presupposition trigger prediction task and datasets and propose an attention mechanism that augments a recurrent neural network without additional trainable parameters.
The proposed model outperforms baselines including an LSTM-based language model on most of the triggers on the two datasets.