The current state-of-the-art end-to-end semantic role labeling (SRL) model is a deep neural network architecture with no explicit linguistic features. 
 However, prior work has shown that gold syntax trees can dramatically improve SRL, suggesting that neural network models could see great improvements from explicit modeling of syntax.
 In this work, we present linguistically-informed self-attention (LISA): a new neural network model that combines 
 multi-head self-attention with multi-task learning across dependency parsing, part-of-speech, predicate detection and SRL. For example, syntax is incorporated by training one of the attention heads to attend to syntactic parents for each token. Our model can predict all of the above tasks, but it is also trained such that if a high-quality syntactic parse is already available, it can be beneficially injected at test time without re-training our SRL model.
 In experiments on the CoNLL-2005 SRL dataset LISA achieves an increase of 2.5 F1 absolute over the previous state-of-the-art on newswire with predicted predicates and more than 2.0 F1 on out-of-domain data. On ConLL-2012 English SRL we also show an improvement of more than 3.0 F1, a 13% reduction in error. Semantic role labeling (SRL) extracts a high-level representation of meaning from a sentence, labeling e.g. who did what to whom. Explicit representations of such semantic information have been shown to improve results in challenging downstream tasks such as dialog systems BID63 BID14 , machine reading BID8 BID65 and machine translation BID36 BID5 .Though syntax was long considered an obvious prerequisite for SRL systems BID34 BID51 , recently deep neural network architectures have surpassed syntacticallyinformed models BID69 BID25 BID60 , achieving state-of-the art SRL performance with no explicit modeling of syntax.Still, recent work BID53 BID25 indicates that neural network models could see even higher performance gains by leveraging syntactic information rather than ignoring it. BID25 indicate that many of the errors made by a strong syntax-free neural-network on SRL are tied to certain syntactic confusions such as prepositional phrase attachment, and show that while constrained inference using a relatively low-accuracy predicted parse can provide small improvements in SRL accuracy, providing a gold-quality parse leads to very significant gains. incorporate syntax from a highquality parser BID31 using graph convolutional neural networks BID32 , but like BID25 they attain only small increases over a model with no syntactic parse, and even perform worse than a syntax-free model on out-of-domain data. These works suggest that though syntax has the potential to improve neural network SRL models, we have not yet designed an architecture which maximizes the benefits of auxiliary syntactic information.In response, we propose linguistically-informed self-attention (LISA): a model which combines multi-task learning BID12 with stacked layers of multi-head self-attention BID64 trained to act as an oracle providing syntactic parses to downstream parameters tasked with predicting semantic role labels. Our model is endto-end: earlier layers are trained to predict prerequisite parts-of-speech and predicates, which are supplied to later layers for scoring. The model is trained such that, as syntactic parsing models improve, providing high-quality parses at test time can only improve its performance, allowing the model to benefit maximally from improved parsing models without requiring re-training. Unlike previous work, we encode each sentence only once, predict its predicates, part-of-speech tags and syntactic parse, then predict the semantic roles for all predicates in the sentence in parallel, leading to exceptionally fast training and decoding speeds: our model matches state-of-the art accuracy in less than one quarter the training time.In extensive experiments on the CoNLL-2005 and CoNLL-2012 datasets, we show that our linguistically-informed models consistently outperform the syntax-free state-of-the-art for SRL models with predicted predicates. On CoNLL-2005, our single model out-performs the previous state-of-the-art single model on the WSJ test set by nearly 1.5 F1 points absolute using its own predicted parses, and by 2.5 points using a stateof-the-art parse (Dozat and Manning, 2017) . On the challenging out-of-domain Brown test set, our model also improves over the previous state-ofthe-art by more than 2.0 F1. On CoNLL-2012, our model gains 1.4 points with its own parses and more than 3.0 points absolute over previous work: 13% reduction in error. Our single models also out-perform state-of-the-art ensembles across all datasets, up to more than 1.4 F1 over a strong fivemodel ensemble on CoNLL-2012. We present linguistically-informed self-attention: a new multi-task neural network model that effectively incorporates rich linguistic information for semantic role labeling. LISA out-performs the state-of-the-art on two benchmark SRL datasets, including out-of-domain, while training more than 4× faster. Future work will explore improving LISA's parsing accuracy, developing better training techniques and adapting to more tasks. Following previous work BID25 , we evaluate our models on the CoNLL-2012 data split BID49 of OntoNotes 5.0 BID27 . 10 This dataset is drawn from seven domains: newswire, web, broadcast news and conversation, magazines, telephone conversations, and text from the bible. The text is annotated with gold part-of-speech, syntactic constituencies, named entities, word sense, speaker, co-reference and semantic role labels based on the PropBank guidelines BID44 . Propositions may be verbal or nominal, and there are 41 distinct semantic role labels, excluding continuation roles and including the predicate.We processed the data as follows: We convert the semantic proposition and role segmentations to BIO boundary-encoded tags, resulting in 129 distinct BIO-encoded tags (including continuation roles). We initialize word embeddings with 100d pre-trained GloVe embeddings trained on 6 billion tokens of Wikipedia and Gigaword BID47 . Following the experimental setup for parsing from Choi et al. FORMULA1 , we convert constituency structure to dependencies using the ClearNLP dependency converter BID17 , use automatic part-of-speech tags assigned by the ClearNLP tagger BID16 , and exclude single-token sentences in our parsing evaluation. 10 We constructed the data split following instructions at: BID11 ) is based on the original PropBank corpus BID44 , which labels the Wall Street Journal portion of the Penn TreeBank corpus (PTB) BID42 with predicateargument structures, plus a challenging out-ofdomain test set derived from the Brown corpus (Francis and Kučera, 1964) . This dataset contains only verbal predicates, though some are multiword verbs, and 28 distinct role label types. We obtain 105 SRL labels including continuations after encoding predicate argument segment boundaries with BIO tags. We evaluate the SRL performance of our models using the srl-eval.pl script provided by the CoNLL-2005 shared task, 11 which computes segment-level precision, recall and F1 score. We also report the predicate detection scores output by this script.For CoNLL-2005 we train the same parser as for CoNLL-2012 except on the typical split of the WSJ portion of the PTB using Stanford dependencies (de Marneffe and Manning, 2008) and POS tags from the Stanford CoreNLP left3words model BID62 . We train on WSJ sections 02-21, use section 22 for development and section 23 for test. This corresponds to the same train/test split used for propositions in the CoNLL-2005 dataset, except that section 24 is used for development rather than section 22.
Our combination of multi-task learning and self-attention, training the model to attend to parents in a syntactic parse tree, achieves state-of-the-art CoNLL-2005 and CoNLL-2012 SRL results for models using predicted predicates.
{"src": "scitldr-AIC_train", "paper_id": "Bk7GGldiz"}