disjoint with any mention.

The latter two (negative instances) significantly outnumber the first exact match ones (positive instances).
This class imbalance problem may result in low predictive accuracy.
To solve this problem,  and  use a down-sampling strategy. That is, they fix the total number of candidate spans in each training batch, including all positive spans and sampled negative spans.
[1]} remove those negative spans that highly overlap with spans corresponding to gold mentions. The negative span \(b\)  is used for training, only if
\(\max \bigg ( \bigg [ \textnormal {IoU} (b, g) \, \textnormal {for} \, g \, \textnormal {in} \, G \bigg ] \bigg ) \le \Gamma ,\) 
where \(G\)  is the set of gold entity mentions. \(\textnormal {IoU} (b, g)\)  is a function measuring how many tokens are shared between two spans:
\(\textnormal {IoU} (b, g) = \frac{\textnormal {length} (b \cap g)}{\textnormal {length} (b \cup g)}\) 
and \(\Gamma \)  is a hyperparameter tuned on different datasets.
Reducing search space
Instead of exhaustive classifying over all possible spans, a two-stage paradigm is investigated to reduce the size of candidate mentions.  propose a boundary-aware model, where first sequence labelling models are used to detect possible span boundaries, and then the span based models are used to predict entity categories of a small number of candidate spans. Similarly,  separate the task into two stages: deciding whether the candidate span is an entity mention or not via a detector, and then classifying detected candidates into predefined entity categories via a classifier.
[2]} detect entity mentions by using what they call head-driven phrase structures. They first identify possible head words of entity mentions, and then recognise the mention boundaries by exploiting phrase structures. They argue that although entity mentions might nest with each other, they cannot share the same head words, and the head words are informative to decide the entity category. They also propose an objective function—bag loss—which does not require gold head word annotations. This is done by exploiting the association between words and entity categories.

Modelling surrounding mentions
To take the surrounding mentions of a given span into consideration,  introduce a 2nd-pass mechanism. They train two models: one standard model, and the other model using outputs from the first model, where the predicted entity categories are used to replace the entity mentions. During inference, the span score is the linear interpolation between scores from these two models.
[3]} propose a multi-task learning framework where entity recognition, relation extraction, and coreference resolution are treated as classification problems with shared span representations. By sharing low-level LSTM encoder, information about relation types with surrounding mentions and coreferences can be used to create input span representations to entity classifier. Instead of sharing only LSTM encoder, [4]} further extend the multi-task model using dynamically constructed span (node) graphs. At each training step, the most confident entity spans are treated as nodes in a graph structure, and arcs are confidence-weighted relation types and coreferences. Then, the span representations are refined using updates which are propagated from neighbouring relation types and co-refeerred entities.

Sentence-level Approach 
Instead of predicting whether a token belongs to an entity mention and its role in the mention (token-level approach) or whether a consecutive sequence of tokens form an entity mention (span-level approach), some methods predict directly a combination of entity mentions within a sentence. We call these methods sentence-level approach.
[5]} consider NER as a structured multi-label classification. Instead of starting and ending indices, they represent each entity mention using the set of token positions that belong to the mention. An example of this representation, with each token tagged using an I/O schema is shown in Figure REF . The advantage of this method is that the representation is very flexible as it allows entity mentions consisting of discontinuous tokens and does not require mentions to exclude each other. Using this representation, the NER problem is converted into the multi-label classification problem of finding up to \(k\)  correct labels among all possible \((T+1)^n\)  labels, where \(k\)  is a hyper-parameter of the model, \(T\)  is the number of entity categories, and \(n\)  is the length of the sentence. Note that labels do not come from a pre-defined category but depend on the sentence being processed. [5]} use large-margin online learning algorithms to train the model, so that the scores of the correct labels (entity mentions) are higher than those of all other possible incorrect mentions. Another advantage of this method is that the outputs of the model are unambiguous for all kinds of complex entity mentions and easy to be decoded. However, the method suffers from a \(O(n^3T)\)  inference complexity.
<FIGURE>[7]} use a discriminative constituency parser to recognise nested entity mentions. They represent each sentence as a constituency tree, where each mention corresponds to a phrase in the tree. In addition, each node needs to be annotated with its parent and grandparent labels, so that the parser can learn how entity mentions nest. [8]} also employ a joint model using the Berkeley parser [9]}, and show that it performs well even without specialised NER features. However, one disadvantage of these parsing based models, as in [5]}, is that their time complexity is cubic in the number of tokens in the sentence. Furthermore, the high quality parse training data, which is not always available, plays a crucial role in the success of the joint model [11]}.
[12]} propose a novel hypergraph to represent exponentially many possible nested mentions in one sentence, and one sub-hypergraph of the complete hypergraph can therefore be used to represent a combination of mentions in the sentence. The mention hypergraph consist of five types of nodes:
\(A^k\)  nodes
represent all mentions whose left boundaries are exactly at or after the \(k\) -th token;
\(E^k\)  nodes
represent all mentions whose left boundaries are exactly at the \(k\) -th token;
\(T^k_j\)  nodes
represent all mentions whose left boundaries are exactly at the \(k\) -th token and have the mention type \(j\) ;
\(I^k_j\)  nodes
represent all mentions which contain the \(k\) -th token and have the mention type \(j\) ; and
\(X\)  nodes
indicate the completion of a path.

Hyper-edges, each of which consists of a parent node and an ordered list of child nodes, are used to connect nodes. Figure REF  is an example of such a sub-hypergraph, which represents two nested entity mentions.
<FIGURE>The training objective of this hypergraph-based model is to maximise the log-likelihood of training instances consisting of the sentence and mention-encoded hypergraph. During inference, the model first predicts a sub-hypergraph among all possible sub-hypergraphs of the complete hypergraph, and predicted mentions can be decoded from the output sub-hypergraph. Different to [12]} who build hand-crafted features defined over the input sentence and the output hypergraph structure, [14]} learn the hypergraph representation using features extracted from a recurrent network.
Although this hypergraph-based model enjoys a time complexity that is linear in the number of tokens in the input sentence, it suffers from some degree of ambiguity during decoding stage. For example, when one mention is contained by another mention with the same entity category and their boundaries are all different, the hypergraph can be decoded in different ways. This ambiguity comes from the fact that, if one node has multiple parent nodes and multiple child nodes, there is no mechanism to decide which of the parent node is paired with which child node. Therefore, [15]} propose an extension of the I node where they use \(I^k_{i,n}\)  nodes to represent all mentions of type \(k\)  which contain the \(j\) -th token and start with the \(i\) -th token.
To represent discontinuous mentions, [16]} expand the node types in the hypergraph representation to capture discontinuous mentions. That is, they add two new node types: B for tokens within the mention, and O for tokens belonging to part of the gap. Figure REF  is an example of the sub-hypergraph, which encodes two mentions: `muscle pain' and `muscle fatigue'. [17]} propose a two-stage approach that all spans are first identified using the hypergraph representation and then a classifier is used to predict whether two spans form a discontinuous mention.
<FIGURE>
Summary
Despite the potential applications of complex NER recognition, there is comparatively few studies on recognising discontinuous and overlapping mentions. The span-based approach focuses on building effective span representations which are used to predict whether the span is an entity mention and its entity category. However, it cannot be directed used for discontinuous NER, because discontinuous mentions consist of multiple spans.
We observe that there is usually a trade-of between the expressiveness power and the modelling difficulty. In other words, the more flexible (less constraints) the representation is, the more interactions are ignored, therefore the model might be more difficult to train. For example, the multi-label representation proposed by [5]} is the most flexible representation (Figure REF ); however, it does not take any interactions between different mentions into consideration. [19]} empirically show that a less flexible representation—BIO variant tagging based model—can outperform the multi-label representation when the training data are limited.
Methods belonging to token-level and sentence-level approaches first predict intermediate representations, and then the intermediate representations are decoded into entity mentions. That is, token level representations use a sequence of tags [20]}, [21]} as the intermediate representation, and sentence-level approach uses a graph structure [12]}, [14]}. Inspired by these approaches, in Chapter , we propose a transition-based model for discontinuous NER. Similar to token-level and sentence-level approaches, our transition-based model first predicts the intermediate representation, i.e., a sequence of actions, but greatly reduce the ambiguity problem in these two approaches. Our method also employs effective methods to build span representations, which are inspired by span-based approach.

Data Augmentation for NER 
Data augmentation, expanding the training set by transforming training instances without changing their labels, is heavily studied in the field of computer vision and sentence level NLP tasks. Inspired by these efforts, we design several easy to use data augmentation methods for NER. Through experiments on two English datasets from the biomedical domain, we demonstrate that our proposed augmentation methods can boost performance over a strong baseline where large scale pre-trained models are used, especially when the original labelled training set is small.

Overview
Modern deep learning techniques typically require a large amount of labelled data for training [24]}, [25]}. However, in real world applications, such large labelled data sets are not always available. This is especially true in some specific domains, such as the biomedical domain, where annotating data requires expert knowledge and is usually time-consuming [26]}, [27]}.
Different approaches have been investigated to solve this low-resource problem [28]}. For example, transfer learning pre-trains language representations on self-supervised or rich-resource source tasks and then adapts these representations to the target task [29]}, [30]}. Data augmentation expands the training set by applying transformations to training instances without changing their original labels [31]}.
Recently, there has been an increased interest on applying data augmentation techniques on sentence-level NLP tasks, such as text classification [32]}, , natural language inference [33]}, and machine translation [34]}. Augmentation methods explored for these tasks include creating augmented instances by manipulating a few words in the original instance, such as word replacement , [35]}, [36]}, random deletion [32]}, and word position swap [38]}, [33]}; or creating entirely artificial instances via generative models, such as variational autoencoders , [40]} and back-translation models , [41]}.
Different from these sentence-level NLP tasks, NER is usually regarded as a token-level NLP task. That is, for each token in the sentence, an NER model predicts a label indicating whether the token belongs to an entity mention and which entity category the mention belongs to. Therefore, applying transformations to individual tokens may also change their labels. Due to such a difficulty, data augmentation for NER is relatively less studied. In this chapter, we describe our efforts to fill this gap by exploring data augmentation techniques for NER, solved as a sequence tagging problem.

Proposed Data Augmentation Methods
We surveyed the existing data augmentation techniques for sentence-level NLP tasks in Section . Inspired by these efforts, we design several easy to use data augmentation methods for NER. Note that our proposed methods do not rely on any external trained models, such as machine translation models , [41]} or syntactic parsing models [38]}, which are by themselves difficult to train in low-resource domain specific scenarios.
Given an original training instance, consisting of a sequence of tokens and the corresponding sequence of labels, we use the following transformations to create augmented instances.
<TABLE>Label-wise Token Replacement (LwTR)
For each token which is not a stop word, we use a binomial distribution to randomly decide whether it should be replaced. If yes, we then use a label-wise token distribution, built from the original training set, to randomly select another token with the same label. Thus, we keep the original label sequence unchanged. Taking the instance in Table REF  as an example, there are five tokens replaced by other tokens which share the same label as the original tokens.

Synonym Replacement (SR)
Our second approach is similar to LwTR, except that we replace the token with one of its synonyms retrieved from WordNet  [44]}. Note that the retrieved synonym may consist of more than one token. Its BIO labels can be derived using a straightforward rule: If the replaced token is the first token within a mention (i.e., the corresponding label is `B-Entity'), we assign the same label to the first token of the retrieved multi-word synonym, and `I-Entity' to the other tokens.

Mention Replacement (MR)
For each mention in the instance, we use a binomial distribution to randomly decide whether it should be replaced. If yes, we randomly select another mention from the original training set which has the same entity category as the replacement. The corresponding sequence of BIO labels can be changed accordingly. For example, in Table REF , the mention `headache [B-problem]' is replaced by another problem mention `neuropathic pain syndrome [B-problem I-problem I-problem]'.

Shuffle within Segments (SiS)
We first split the token sequence into segments of the same entity category. Thus, each segment corresponds to either an entity mention or a sequence of tokens that does not belong to any mention. For example, the original instance in Table REF  is split into five segments: `She did not complain of [Out-of-Mention]', `headache [Problem]', `or [Out-of-Mention]', `any other neurological symptoms [Problem]', `. [Out-of-Mention]'. Then for each segment, we use a binomial distribution to randomly decide whether it should be shuffled. If yes, the order of the tokens within the segment is shuffled, while the label order is kept unchanged.

All
We also explore the augmentation of the training set using all aforementioned augmentation methods. That is, for each training instance, we create multiple augmented instances, one per transformation.

Evaluation
We present an empirical analysis of the data augmentation methods described in Section  on two English datasets from the biomedical domain In [45]}, we also evaluate these methods on a dataset from the materials science domain.: i2b2-2010 [46]} and NCBI-Disease  [47]}.
We use a BERT-CRF model [48]}, [49]} as the backbone model, and we investigate the impact of applying data augmentation on training data of different sizes.
Datasets
<TABLE>i2b2-2010 focuses on the identification of three entity types of problem, treatment and test from patient reports. We use the train-test split from its corresponding shared task and randomly select 15% of sentences from the training set as the development set. NCBI-Disease contains scholarly articles annotated with disease names. We use the train-dev-test split provided by the authors. Descriptive statistics of these two datasets is listed in Table REF .
To simulate a low-resource setting, we select the first 50, 150, 500 sentences which contain at least one mention from the complete training set to create the corresponding small, medium, and large subsets (denoted as S, M, L in Table REF , whereas the complete training set is denoted as F). Note that we apply data augmentation only on the training set, without changing the development and test sets.

Backbone model
We regard the NER task as a token-level sequence tagging problem, where each token in the sentence is assigned a tag. The tag can be used to infer whether the token is the first token within a mention, inside a mention or does not belong to any mention.
The backbone model, illustrated in Figure REF , is a BERT-CRF model [48]}, [49]}. It takes advantage of large scale pre-trained language models—using BERT-based encoder to create contextual representations for each token, and a probabilistic graphical model—using conditional random fields [52]} to capture dependencies between neighbouring tags.
<FIGURE>BERT-based encoder
Given a sentence, the tokenizer, coupled with the pre-trained BERT-based model, first converts each token in the sentence into word pieces. That is, if the original token does not exist in the vocabulary, it will be segmented into several pieces from the vocabulary [53]}. Then the word pieces are mapped to dense vectors—token embeddings—via a lookup table. Finally, the sum of token embeddings and positional embeddings, which indicate the position of each token in the sequence, are fed into a stack of multi-head self-attention and fully-connected feed-forward layers [54]}. Following the study in [55]}, we use the final outputs corresponding to the first word piece within each token as the token representation.
Recent studies on domain-specific BERT models show that effectiveness on downstream tasks can be improved when the BERT models are further pre-trained on in-domain data [30]}. We thus choose SciBERT [48]}, which is pre-trained on full text of scholarly articles about biology and computer science, and fine-tune it on the target NER task. In our preliminary experiments, we observe that SciBERT achieves significant better results than vanilla BERT [55]} and slightly better results than BioBERT [59]}.

Conditional random fields (CRF)
Instead of assigning a tag to each token independently, we model them jointly using a conditional random fields. That is, given a sequence of token representations \(\mathbf {R}=\left(\mathbf {r}_{1}, \mathbf {r}_{2}, \ldots , \mathbf {r}_{n}\right)\) , we aim to predict a sequence of tags \(\mathbf {y}=\left(y_{1}, y_{2}, \ldots , y_{n}\right)\)  which has the maximum probability over all possible tag sequences. This conditional probability can be calculated using:
\(p(\mathbf {y} \mid \mathbf {R})=\frac{e^{s(\mathbf {R}, \mathbf {y})}}{\sum _{\widetilde{\mathbf {y}} \in \mathbf {Y}_{\mathbf {R}}} e^{s(\mathbf {R}, \widetilde{\mathbf {y}})}}\) 
and
\(s(\mathbf {R}, \mathbf {y})=\sum _{i=0}^{n} A_{y_{i}, y_{i+1}}+\sum _{i=1}^{n} P_{i, y_{i}},\) 
where \(A_{i,j}\)  is the compatibility score of a transition from the tag i to tag j, and \(P_{i, j}\)  is the score of the tag \(j\)  given token representation \(r_i\) .
The parameters, of both the SciBERT encoder and CRFs, are trained jointly to maximise the conditional probability of gold tag sequence given the training sentences.

Experimental results
The evaluation results on the effectiveness of data augmentation methods are shown in Table REF . We use the Micro-average string match \(F_1\)  score to evaluate the effectiveness of the models. The model which is most effective on the development set, measured using the \(F_1\)  score, is finally evaluated on the test set. All experiments are repeated five times with different random seeds. Mean values and standard deviations are reported. The \(\Delta \)  row shows the averaged improvement due to data augmentation, comparing against the baseline without using data augmentation. In general, we find that all data augmentation methods improve over the baseline, and synonym replacement outperforms other augmentation on average.
<TABLE>Another observation is that the data augmentation methods are more effective when the original training sets are small. For example, all data augmentation methods achieve improvements when the training set contains only 50 training instances. In contrast, when the complete training sets are used, only synonym replacement and mention replacement achieve improvements. This has also been observed in previous work on applying data augmentation on other NLP tasks [60]}, [38]}, .

Analysis
After demonstrating the effectiveness of proposed data augmentation methods, we present an analysis of the best two performing transformations: synonym replacement and mention replacement. We aim to provide practical suggestions on hyperparameter settings as well as understandings about how they improve the performance.
The impact of hyperparameters
For each augmentation method, we tune the number of augmented instances per original training instance from a list of numbers: {1, 3, 6, 10}. We also tune the \(p\)  value of the binomial distribution which is used to decide whether a token or a mention should be replaced. It is searched over the range from 0.1 to 0.7, with an incremental step of 0.2. We perform grid search to find the best combination of these two hyperparameters on the development set.
The main question we aim to answer is how much augmentation is enough? More specifically, how the number of augmented instances per original training instance affects performance, and how the ratio a token, or a mention, is replaced affects performance?
<FIGURE>Figure REF  shows the impact of the number of augmented instances per original training instance on the performance gain of synonym replacement and mention replacement. We use the improvement of absolute \(F_1\)  score over the baseline without using data augmentation as the performance gain.
In general, we find that larger number of augmented instances can bring larger performance gain, especially when the training sets are small (i.e., 50 training instances). However, the performance gain becomes relatively small when the number of augmented instances per original training instance is greater than 6. The second observation is that when the training sets are medium (i.e., 150 training instances) or large (i.e., 500 training instances), the benefits of more augmented instances become small. On i2b2-2010, creating more augmented instances using synonym replacement on large training set even decreases the performance.
<FIGURE>Figure REF  A and B shows the impact of the ratio a token is replaced with one of its synonyms on the performance gain. We note a moderate ratio (e.g., \(0.3\)  or \(0.6\) ) performs well across different setups. If the ratio is too small, the augmented instances may be very similar to the original one. Training on these augmented instances may have a similar effect as training on the original training instances for more epochs. In contrast, a large ratio is more likely to create syntactically invalid instance. These syntactically invalid instances are noisy, and training on such a combination of small amount of clean data and large amount of noisy data may underperform training on clean data only (more discussions in Section REF ).
The pattern with mention replacement is different from the one with synonyms replacement (Figure REF  C and D). When the training sets are small, increasing the ratio a mention is replaced with another mention of the same entity category always enlarges the performance gain. We find that most of these entity mentions have similar part of speech patterns. That is, most of them are either nouns or noun phrases. Because of this feature, the risk of creating syntactically invalid instance is much lower using mention replacement than using synonyms replacement. It can be the reason why a moderate ratio for synonyms replacement is best, whereas mention replacement can benefit from a large replace ratio.

A closer look at errors
The next question we aim to answer is how data augmentation improves the performance. Put another way, are data augmentation methods guaranteed to fix some particular errors predicted by the baseline model without using data augmentation, and if yes, which types of errors are more likely to get rectified.
<TABLE>To answer this question, we train three models—one baseline model without using data augmentation, two models using synonym replacement and mention replacement, respectively—and then compare the error predictions by these three models.
From Table REF , we find data augmentations are more likely to reduce false positives than false negatives. In other words, if the baseline model fails to recall some entity mentions, the model trained using data augmentation usually fails to recall them as well. However, data augmentation can fix those mistakenly predicted entity mentions. On one hand, we believe this improvement can be linked to the over-fitting problem. That is, the model trained without using data augmentation may overfit some patterns observed in the small training set, and data augmentation can relieve this problem, by creating a new combination of mention and context. On the other hand, training model using data augmentation provides very little improvement on fixing those false negatives. Note that data augmentation may also make large amount of new false positives. In other words, there is no guarantee data augmentation can fix some particular errors predicted by models without using data augmentation, since they may provide a mechanism to prevent the training from over-fitting, but not help the learning algorithm to discover new regularities.

Summary
We design several easy to use data augmentation methods for NER: label-wise token replacement, synonym replacement, mention replacement, and shuffle within segments. Through experiments on two datasets from the biomedical domain, we find that all proposed data augmentation methods can improve over the strong baseline, where large scale pre-trained models are used, and synonym replacement outperforms other augmentation on average.

Cost-effective Selection of Pre-training Data 
Pre-training language representation models on unlabelled data and then adapting them to downstream supervised tasks has become a standard practice in NLP. However, the selection of pre-training data usually resorts to intuition, which varies across NLP practitioners. We make use of similarity measures to nominate in-domain pre-training data. Experimental results suggest that simple similarity measures are good predictors of the usefulness of pre-trained language representation models on downstream NER tasks.

Overview
Sequential transfer learning—which pre-trains a model from a source task and then adapts it to a different target task—has demonstrated its effectiveness on a range of NLP tasks [62]}, [63]}, [29]}. There are two stages in this procedure: pre-training, and adaptation. Researchers who work on low-resource NLP usually spend a considerable amount of efforts and resources on choosing useful external data sources and investigating how to transfer knowledge to their target tasks.
[65]}, [66]}, [55]} make the most of limited labelled data by incorporating language representation models which are pre-trained on a large amount of unlabelled data. This benefits a range of NLP tasks where appropriate unlabelled data is available, and has become a standard practice in NLP.
However, there is still a lack of systematic study on how to select appropriate data to pre-train language representation models. We observe two heuristic strategies in the literature:

collecting as large as possible generic data, such as news [65]}, [66]}, [70]}, web crawl [71]}, [72]}, and Wikipedia [73]}, [55]}; and,

selecting moderate size data focusing on a specific domain. The resulting pre-trained models are called domain-specific models [75]}, [76]}, [77]}, [78]}, [59]}.

The advantage of the first strategy is that the pre-trained generic models can be re-used in various domains, however, the corresponding training cost is high and unbearable to many academic labs. For example, [70]} pre-train the RoBERTa model using 1024 V100 GPUs, which are only accessible by large companies. Therefore, we focus on studying the second strategy, and we aim to pre-train domain-specific models, optimising the performance on downstream biomedical NER datasets.
Studies on domain-specific language representation models empirically show that target task performance can be improved, when in-domain data is used for pre-training [81]}, [59]}, [48]}. These publicly available domain-specific models are valuable to the NLP community. However, the selection of in-domain data usually resorts to intuition, which varies across NLP practitioners (Section ). According to [84]}, the context specific usage of language is affected by three factors: field (the subject matter being discussed), tenor (the relationship between the participants in the discourse and their purpose) and mode (communication medium, such as `spoken' or `written'). Generally, the selection of pre-training data in existing domain-specific models is mainly based on the field rather than the tenor. For example, BioBERT [59]} and SciBERT [48]} are both pre-trained on scholar articles, but on different fields (biology and computer science).
We first show in Section  that human intuition regarding selecting pre-training data varies across practitioners, motivating our work on employing quantitative measures to nominate in-domain pre-training data. We then describe several measures which can quantify similarity between two datasets in Section . We pre-train several domain-specific language representation models on different sources, and investigate their effectiveness on various downstream NER datasets, respectively (Section ). Finally, through correlation analysis, we show that simple similarity measures can be used to nominate in-domain pre-training data (Section REF ).

What Human Intuition Indicates
We surveyed 30 NLP or machine learning practitioners to learn the human intuition regarding selection of pre-training data. Participants were provided short descriptions of the target data T, and two possible source data S1 and S2 as

T: Online forum posts about medications;

S1: Research papers about biology and health;

S2: Online reviews about restaurants, hotels, barbers, mechanics, etc.

A screenshot is shown in Figure REF .
<FIGURE>We constructed each of the descriptions as `\(t\)  about \(f\) ' where \(t\)  is intended to indicate the tenor and \(f\)  the field. Each participant rated both sources on a five-point Likert, indicating agreement with the statement “Unsupervised pre-training on S would be useful for supervised named entity recognition learning on T”.
<FIGURE>Survey results show that 73% of the participants agreed or strongly agreed that S1—sharing similar biomedical field with the target—would be useful. Only 27% agreed that S2—sharing similar social media tenor with the target—would be useful (Figure REF ). On the one hand, a Wilcoxon signed-rank test indicates that scores are significantly higher for S1 than for S2 (\(Z = 43.0, p < 0.001\) ). On the other hand, these results show the variety across practitioners, motivating our work on employing quantitative measures to nominate pre-training data. Our empirical investigations (detailed in Section ) also suggest that human intuition maybe unreliable regarding selecting pre-training data. That is, practitioners favour field over tenor when selecting pre-training data, and this would be detrimental to accuracy of the target NER tasks.

Similarity Measures
Recall that the context specific usage of language is affected by three factors: field, tenor and mode [84]}. Researchers who select pre-training data from a similar field believe that, if the source data has a similar field to the target data, they tend to share similar topical vocabulary. Conversely, vocabularies are different from each other if source and target are from different fields. Imagine datasets about medications and restaurants. Those who select pre-training data from a similar tenor believe that tenor may impact the writing style of text. Imagine the participants in online reviews and scientific papers, their relationships to each other, their purposes and how these affect text style, including punctuation, lexical normalisation, politeness, emotiveness and so on [88]}, [89]}. We do not explicitly consider mode, because all of the datasets studied in this thesis are written text.
Below, we detail different measures based on these intuitions to quantify different aspects of similarity between two datasets.
Target vocabulary covered
The first measure is simply the percentage of the target vocabulary that is also present in the source data. An extremely dissimilar example is that of different languages. They have a totally different vocabulary and are considered dissimilar, even if they are written in a similar style and talking about the same subject. Note that our focus is on transferring through pre-trained models using one single source and we do not consider multilingual similarity. We propose Target Vocabulary Covered (TVC) as a measure of field, calculated as
\(TVC (D_S, D_T) = \frac{|V_{D_S} \cap V_{D_T}|}{|V_{D_T}|},\) 
where \(V_{D_S}\)  and \(V_{D_T}\)  are sets of unique content words (nouns, verbs, adjectives) in source and target datasets respectively.

Jaccard similarity of vocabularies
Jaccard similarity coefficient [90]}, is a statistic used for estimating the similarity and diversity of two sets. By calculating
\(JSC (D_S, D_T) = \frac{|V_{D_S} \cap V_{D_T}|}{|V_{D_S} \cup V_{D_T}|},\) 
Jaccard Similarity of Vocabularies (JSV) can be used to measure the similarity between source and target vocabularies, meanwhile, factoring out the source vocabulary size.

Language model perplexity
A language model [91]} assigns a probability to any sequence of words \([w_1,\cdots ,w_N]\)  using chain rule of probability:
\(p(w_1, w_2, \cdots , w_N) = \prod _{i=1}^N p(w_i|w_1^{w-1}),\) 
where \(N\)  is the length of the sequence and \(w_1^{i-1}\)  are all words before word \(w_i\) . In practice, this equation can be simplified by n-gram models based on Markov Assumption:
\(p(w_1, w_2, \cdots , w_N) = \prod _{i=1}^N p(w_i|w_{i-n+1}^{i-1}),\) 
where \(w_{i-n+1}^{i-1}\)  represents only \(n\)  preceding words of \(w_i\) . To make the model generalise better, smoothing techniques can be used to assign non-zero probabilities to unseen events. We use Kneser-Ney smoothed 3-gram models [92]} to measure the similarity between two datasets. Specifically, we first train the language model on the source data, then evaluate it on the target data using perplexity to represent the degree of similarity. The intuition is that, if the model finds a sentence very unlikely (dissimilar from the data where this language model is trained on), it will assign a low probability and therefore high perplexity. The summed up perplexity (PPL) is then:
\(PPL(D_S, D_T) = \sum _{i=1}^{m} P(D_T^i)^{-\frac{1}{N_i}},\) 
where \(m\)  is the number of sentences in the target data set, and \(P(D_T^i)\)  is the probability assigned by the language model trained on the source data to the \(i\) -th sentence from the target data set, whose sentence length is \(N_i\) .
Similar to TVC, PPL is token-based but also captures surface structure. We therefore propose PPL as a proxy to measure tenor as well as field.

Jensen-Shannon divergence
Jensen-Shannon divergence (JSD), based on term distributions, has been successfully used for domain adaptation [93]}. We first measure the probability of each term (up to 3-gram) in source \(S\)  and target data \(T\) , separately. Then, we use the Jensen-Shannon divergence [94]} between
these two probability distributions
\(JSD(S || T) = \frac{1}{2} KL (S || M) + \frac{1}{2} KL (T || M),\) 
where
\(KL (P || Q) = \sum _{x \in X} P(x) \log (\frac{P(x)}{Q(x)})\) 
and
\(M = \frac{1}{2} (S + T)\) 
as a proxy to measure tenor as well as field.

Datasets
We use six datasets as source data, covering a range of fields and tenors:
News
The original one billion word language model benchmark data [95]}, produced from News Crawl data. Popular reporting usually involves one or several writers, and a large number of readers. The text is usually edited several times for easy understanding.
Books
A corpus of 11,038 books in 16 different genres, e.g., Romance, Fantasy, Science fiction, etc., collected by . There books are all free books written by yet unpublished authors. Fiction books usually involves one writer, and a moderate size of readers. The text is usually edited many times, reflecting writer personality.
MIMIC
A clinical database comprising over 58,000 hospital admissions for intensive care unit (ICU) patients [96]}. Clinical notes are usually written by doctors and nurses under time pressure, and read by their colleagues. The text is seldom carefully edited, so it is syntactically noisy and usually contains a lot of jargon for efficient communication.
PubMed
Titles and abstracts of biomedical scholar articles. Scholar articles are usually written by a small groups of writers, and the readers usually have similar knowledge background with authors. The text is usually edited many times for more comprehensible and less ambiguous.
Yelp
Crowd-sourced reviews about local businesses, including restaurants, hotels, barbers, mechanics, etc. Online review is usually written by a single customer, and read by a small group of people who are interested in the business. The text is usually edited once, and writer tends to use descriptive language to share their experiences.
Wikipedia
A free online encyclopedia. Online encyclopedia is created and edited by volunteers around the world, and it has around 250 million page views every day Siteviews Analysis: shorturl.at/ayQR5. Accessed data: 2021-Jan-31.. Another important feature of Wikipedia as an open-collaborative website is that articles are edited all the time by human users and bots, reflecting the newest development of the world knowledge.

To isolate the impact of source data size, we randomly sample all source data to approximately 700 million tokens. The only exception is on MIMIC. Although all text from MIMIC data set has been used, it is still relatively small, comparing to other sources. The data statistics of source data is listed in Table REF . Based on the number of unique tokens and average sentence length, we can see these sources are roughly split into two categorise: formal text—PubMed, Wikipedia and News—with large vocabulary and long sentences, and informal text—Books, MIMIC, and Yelp—with small vocabulary and short sentences.
<TABLE>Ten NER datasets are used as target data: BC2GM (BioCreative II Gene Mention Recognition) [97]}, BTC (Broad Twitter Corpus) [98]}, CADEC (CSIRO Adverse Drug Event Corpus) [26]}), CoNLL 2003 [100]}, EBM (Evidence Based Medicine) [27]}, i2b2 2010 [46]}, JNLPBA [103]}, NCBI-Disease  [47]}, SciERC [3]}, WetLab [106]}, and W-NUT 2016 [107]}.
Details of these target data are listed in Table REF .
<TABLE>
Experimental Results
Similarity Between Source and Target Datasets
The results shown in Table REF  and REF  indicate that PubMed is the most similar source to most of these target datasets from the Biomedical domain. It achieves lower language model perplexity, higher target vocabulary covered, and Jensen-Shannon Divergence when evaluated against BC2GM, EBM, JNLPBA, NCBI-disease, SciERC and Wetlab compared to other sources. On one hand, it is expected that PubMed is similar to BC2GM, EBM, JNLPBA, NCBI-disease and Wetlab, since they are all scientific writing about biology and health, thus being similar in terms of both field and tenor. On the other hand, although SciERC does not have the same field as PubMed (computer science, material and physics versus biology and health), they are similar because they share a similar tenor (scholarly publications). On i2b2-2010 (clinical notes), only the target vocabulary covered measure indicates PubMed is the most similar source, whereas other three metrics indicate MIMIC as the most similar source.
<TABLE><TABLE>The second observation is that tenor might be reflected more than field by these measures. Source data Yelp is more similar to CADEC than PubMed and MIMIC from both language model perplexity and Jensen-Shannon Divergence perspectives. CADEC is a data set focusing on recognising drugs, diseases and adverse drug events. The field of CADEC is therefore more similar to PubMed which includes journal articles in health discipline and MIMIC which contains clinical notes. However, CADEC is written by patients, and can be considered as `drug reviews'. The tenor is therefore closer to the one in Yelp, where customers use informal language to describe their experiences. Target vocabulary covered nominates News as the most similar source. Note that News has a moderate size vocabulary, 2.1 millions unique tokens, whereas the vocabulary size of PubMed is 4.2 million.
The last observation is that using different measures can lead to almost the same answer regarding which source is the most similar one to a given target, except for the Jaccard similarity of vocabularies. Using Jaccard similarity of vocabularies measure, MIMIC source is nominated as the most similar source against target sets, except for BTC. This might be explained by the fact that the vocabulary size of MIMIC is the smallest one in all sources, and Jaccard similarity of vocabularies measure favours sources with small vocabulary size than the ones with large vocabulary size.

Effectiveness of domain-specific models on downstream NER tasks
After we quantify the similarity between source and target datasets, the next step is to investigate the impact of source data on pre-trained language representation models. We pre-train ELECTRA [108]}—a sample-efficient variant of BERT—on different sources separately, then observe how the effectiveness of these pre-trained models varies in different downstream NER datasets.
The most common approach of training domain-specific models is continue pre-training, which is used by BioBERT [59]}, ClinicalBERT [81]}, and BERTweet [78]}. Continue pre-training approach starts from an existing pre-trained model—usually pre-trained on large size generic data set—and continues training on a domain-specific corpus. The main advantage of continue pre-training is that they can inherit knowledge from language representation models pre-trained on generic data, and thus be considered as capturing both generic domain and domain-specific knowledge. However, we aim to investigate the impact of pre-training data, therefore, we use the learning from scratch approach, eschewing the potential impact of generic data. We follow the hyper-parameter setting in [108]}, shown in Table REF  to train the domain-specific models. Training of each model took four days using 1 Nvidia Tesla v100 GPU.
<TABLE>Evaluation results using these domain-specific models on downstream NER tasks show that the effectiveness varies in different target datasets (Table REF ). In other words, no single source is suitable for all target NER datasets. It is worthy note that most of these results (for example on BC2GM, BTC, CADEC, SciERC) are lower than state-of-the-art results on these datasets with large margin. This is mainly because our pre-trained models are smaller—smaller hidden size, less number of attention heads, smaller embedding size—than the ones in other studies.
The best performing models on each target data are all pre-trained on the most similar source which is nominated by at least one similarity measure, except for SciERC (Table REF  and REF ).
<TABLE>
Predictiveness of similarity measures 
To analyse how proposed similarity measures can be used to nominate the best pre-training data option, we investigate the correlation between these similarity values and the effectiveness of pre-trained models on target tasks. Specifically, we employ Spearman rank-order correlation coefficient to measure the relationship between the ranking of similarity values and NER results. For example, given the target data set NCBI-disease, the rank of sources is PubMed, Wikipedia, MIMIC, Yelp, News, and Book, if they are sorted based on the effectiveness of different domain-specific models. Similarity, if they are sorted based on the target vocabulary covered measure, the rank of sources is PubMed, Wikipedia, News, MIMIC, Books, and Yelp. The Spearman rank-order correlation coefficient between these two rankings is 0.71.
The results in Figure REF  show that these proposed similarity measures are predictive of the effectiveness of the pre-training data, except for Jaccard similarity of vocabularies.
<FIGURE>
Comparison to publicly available pre-trained models
Literature shows substantial improvements are sometimes possible when pre-training on large generic corpora [70]}, [49]}. Given that pre-trained models are freely available, is it even necessary to pre-train on similar data as proposed above? We compare to publicly available ELECTRA models trained on 3.3 Billion tokens generic data. Note that the publicly available model we choose has the same model size. It is also pre-trained using the same hyper-parameters as ours, except that it is pre-trained longer than ours (1.45M vs.0.8M).
<TABLE>These results, shown in Table REF  indicate that a small similar source reduces the computational cost without sacrificing the performance. This is especially important in practice, because collecting data and pre-training models are expensive, in terms of both computational and environmental cost [115]}.

Summary
This chapter focuses on whether there are cost-effective methods to nominate datasets to pretrain language representation models that are building blocks of NER models.
We propose using different measures to measure different aspects of similarity between source and target data. We investigate how these measures correlate with the effectiveness of pre-trained models for NER tasks.
While different NLP tasks may rely on different aspects of language, our study is a step towards systematically guiding researchers on their choice of data for pre-training, and models pre-trained on small size domain-specific corpus can outperform the one pre-trained on large size generic domain data.

Transition-based Model for Discontinuous NER 
Discontinuous mentions represent compositional concepts, for example disorders or symptoms, that differ from concepts represented by individual components, for example body locations or general feelings. In downstream applications such as pharmacovigilance and summarization, recognising these discontinuous mentions is more useful than recognising separate components. We propose a transition-based model that can effectively recognise discontinuous mentions without sacrificing the accuracy on continuous mentions.

Overview
NER is a critical component of biomedical text mining applications. In pharmacovigilance, it can be used to identify adverse drug events in consumer reviews in online medication forums, alerting medication developers, regulators, and clinicians [116]}, [117]}, [118]}. In clinical settings, NER can be used to extract and summarise key information from electronic medical records such as conditions hidden in unstructured doctors' notes [119]}, [120]}. These applications require identification of complex entity mentions, discontinuous and overlapping mentions, not seen in generic domains.
Widely used sequence tagging techniques encode two assumptions that do not always hold: (1) mentions do not overlap, therefore each token can belong to at most one mention; and, (2) mentions comprise continuous sequences of tokens. Nested entity recognition addresses violations of the first assumption (more discussions in Section ). However, the violation of the second assumption is comparatively less studied and requires handling discontinuous mentions (see examples in Figure REF ).
<FIGURE>In contrast to continuous mentions which are often short spans of text, discontinuous mentions consist of components that are separated by intervals. Recognising discontinuous mentions is particularly challenging as exhaustive enumeration of possible mentions, including discontinuous and overlapping spans, is exponential to sentence length. Existing approaches for discontinuous NER either suffer from high time complexity [5]} or ambiguity in translating intermediate representations into mentions [20]}, [21]}, [16]}. In addition, current arts use traditional approaches that rely on manually designed features, which are tailored to recognise specific entity categories. Also, these features usually do not generalise well in different types of text [125]}.
Motivations
The main motivation for recognising discontinuous mentions is that they usually represent compositional concepts that differ from concepts represented by individual components. For example, the mention `left atrium dilated' in the first example of Figure REF  describes a disorder which has its own CUI (Concept Unique Identifier) in UMLS (Unified Medical Language System), whereas both `left atrium' and `dilated' also have their own CUIs. In downstream applications such as pharmacovigilance and summarization, recognising these discontinuous mentions that refer to disorders or symptoms is more useful than recognising separate components which may refer to body locations or general feelings.
Another motivation for discontinuous NER is that discontinuous mentions usually overlap, and separating these overlapping mentions rather than identifying them as a single mention is important for downstream tasks, such as entity linking where the assumption is that the input mention refers to one entity [126]}.
In this chapter, we first characterise three datasets, from the biomedical domain, with a substantial number of discontinuous mentions (Section ). Then, we introduce a transition-based model that can recognise discontinuous mentions (Section ). Through experiments, we show that our model can effectively recognise discontinuous mentions without sacrificing the accuracy on continuous mentions (Section ). Analysis also suggests that our model is better than existing discontinuous NER models at handling long mentions and mentions that do not overlap or overlap at left, resulting in higher recall (Section ).

Datasets 
Although some text annotation tools, such as BRAT [127]}, allow discontinuous annotations, corpora annotated with large number of discontinuous mentions are still rare because they are hard to annotate. We describe three datasets from the biomedical domain that include a substantial number of discontinuous mentions: CADEC [26]}, ShARe/CLEF 2013 [129]} and ShARe/CLEF 2014 [130]}. We then motivate the discontinuous NER task.
CADEC corpus is sourced from posts from AskaPatienthttps://www.askapatient.com/. Accessed data: 2022/12/09 20:22:09, a forum where patients can discuss their experiences with medications. The entity categories annotated in CADEC include drug, Adverse Drug Event (ADE), disease and symptom. In this work, we only consider the ADE annotations because only the ADEs involve discontinuous mentions. Note that ADEs in CADEC are defined as span of text that are clearly associated with a drug and should have the corresponding MedDRA (Medical Dictionary for Regulatory Activities) term. ShARe/CLEF 2013, specifically Task 1 of the ShARe/CLEF eHealth evaluation lab 2013, focuses on identification of disorder mentions in clinical reports. The corpus is sourced from de-identified clinical reports, including discharge summaries, electrocardiogram, echocardiogram, and radiology reports [96]}. A disorder mention is defined as any span of text which can be mapped onto a concept in the Disorder semantic group of SNOMED-CT (Systematised Nomenclature of Medicine – Clinical Terms). ShARe/CLEF 2014, an extension of the ShARe/CLEF 2013 task, focuses on template filling of disorder attributes. That is, given a disorder mention and its surrounding words, recognise the attributes of the disorder mention from its context, including subject class, severity indicator, uncertainty indicator. In this work, we frame ShARe/CLEF 2014 as a disorder-NER dataset. That is, we consider only disorder annotations, without taking their attributes into consideration.
<TABLE>Descriptive statistics of these three datasets is listed in Table REF . On average, discontinuous mentions are longer than continuous mentions, because they consist of several components, and the intervals between different components make the total length of span even longer. Another important characteristic of discontinuous mentions is that they usually overlap. That is, several mentions may share components that refer to the same body location (e.g., `muscle' in `muscle pain and fatigue'), or the same feeling (e.g., `Pain' in `Pain in knee and foot'). From this perspective, we also categorise discontinuous mentions into the following groups:

No overlap: in such cases, the discontinuous mention can be intervened by severity indicators (e.g., `is mildly' in sentence `left atrium is mildly dilated'), preposition (e.g., `on my' in sentence `...rough on my stomach...') and so on. This category accounts for half of discontinuous mentions in the ShARe/CLEF datasets but only 12% in CADEC.

Left overlap: the discontinuous mention shares one component with other mentions, and the shared component is at the beginning of the discontinuous mention. This is usually accompanied with coordination structure (e.g., the shared component `muscle' in `muscle pain and fatigue'). Conjunctions (e.g., `and', `or') are clear indicators of the coordination structure. However, clinical notes (ShARe/CLEF datasets) are usually written by practitioners under time pressure. They often use commas or slashes rather than conjunctions. This category accounts for more than half of discontinuous mentions in CADEC and one third in ShARe/CLEF.

Right overlap: similar to left overlap, although the shared component is at the end.
For example, `hip/leg/foot pain' contains three mentions that share the token `pain'.

Multi-overlap: the discontinuous mention shares multiple components with the others, which usually forms crossing compositions. For example, the sentence `Joint and Muscle Pain / Stiffness' contains four mentions: `Joint Pain', `Joint Stiffness', `Muscle Stiffness' and `Muscle Pain', where each discontinuous mention share two components with the others.

Although these three datasets – CADEC, ShARe/CLEF 2013 and ShARe/CLEF 2014 – share similar field (the subject matter of the content being discussed), the tenor (the participants in the discourse, their relationships to each other, and their purposes) of CADEC is very different from the ShARe/CLEF datasets. Specially, laymen authors (CADEC) tend to use idioms or ungrammatical phrases to describe their feelings, whereas professional practitioners (ShARe/CLEF) tend to use compact terms for efficient communications. This difference of tenor results in different features of mentions between these datasets. That is, the mentions in CADEC are overall longer than those in ShARe/CLEF datasets, and larger ratio of discontinuous mentions in CADEC are involved in overlapping structure (Table REF ).

Proposed Model 
We propose a transition-based model based on the shift-reduce parser [132]}, [133]} that employs a stack to store partially processed spans and a buffer to store unprocessed tokens. The learning problem is then framed as: given the state of the parser, predict an action which is applied to change the state of the parser. This process is repeated until the parser reaches the end state, which is the stack and buffer are both empty.
Similar to prior work [21]}, [16]} that first predict an intermediate representation of mentions, which are then decoded into the final mentions, our proposed transition-based model uses a sequence of actions as the intermediate representation (refer to Section ).
The main difference between our model and the ones in [132]}, [133]} is the set of transition actions. [132]} use SHIFT, REDUCE, UNARY, FINISH, and IDEA for the constituent parsing system. [133]} use SHIFT, REDUCE, OUT for the flat NER system. Inspired by these models, we design a set of actions specifically for recognising discontinuous and overlapping structure. There is a total of six actions in our model:

SHIFT moves the first token from the buffer to the stack; it implies this token is part of an entity mention.

OUT pops the first token of the buffer, indicating it does not belong to any mention.

COMPLETE pops the top span of the stack, outputting it as an entity mention. If we are interested in multiple entity categories, we can extend this action to COMPLETE-\(y\)  which labels the mention with entity category \(y\) .

REDUCE pops the top two spans \(s_0\)  and \(s_1\)  from the stack and concatenates them as a new span which is then pushed back to the stack.

LEFT-REDUCE is similar to the REDUCE action, except that the span \(s_1\)  is kept in the stack. This action indicates the span \(s_1\)  is involved in multiple mentions. In other words, several mentions share \(s_1\)  which could be a single token or several tokens.

RIGHT-REDUCE is the same as LEFT-REDUCE, except that \(s_0\)  is kept in the stack.

Figure REF  shows an example of how the proposed parser recognises entity mentions from a sentence.
<FIGURE>Representation of the parser state 
Given a sequence of \(N\)  tokens, we first run a bi-directional LSTM  [140]} to derive the contextual representation of each token. Specifically, for the \(i\) -th token in the sequence, its representation can be denoted as:
\(\mathbf {\tilde{c_i}} = \left[ \overrightarrow{\textsc {LSTM}} (\mathbf {t_0}, \dots , \mathbf {t_{i}}); \overleftarrow{\textsc {LSTM}} (\mathbf {t_{i}}, \dots , \mathbf {t_{N - 1}}) \right],\) 
where \(\mathbf {t_i}\)  is the concatenation of the embeddings for the \(i\) -th token, its character level representation learned using a CNN network [141]}. Pretrained contextual word representations have shown its usefulness on improving various NLP tasks. Here, we can also concatenate pretrained contextual word representations using ELMo [66]} with \(\mathbf {\tilde{c_i}}\) , resulting in:
\(\mathbf {c_i} = \big [ \mathbf {\tilde{c_i}}; \textsc {ELMo} _i \big ],\) 
where \(\textsc {ELMo} _i\)  is the output representation of pretrained ELMo models (frozen) for the \(i\) -th token. These token representations \(\mathbf {c}\)  are directly used to represent tokens in the buffer.
Following the work in [143]}, we use StackLSTM to represent spans in the stack. That is, if a token is moved from the buffer to the stack, its representation is learned using:
\(\mathbf {s_0} = \textsc {StackLSTM} (\mathbf {s_D} \dots \mathbf {s_1};\mathbf {c}_{\text{SHIFT}}),\) 
where \(D\)  is the number of spans in the stack. Once REDUCE related actions are applied, we use a multi-layer perceptron to learn the representation of the concatenated span. For example, the REDUCE action takes the representation of the top two spans in the stack: \(\mathbf {s_0}\)  and \(\mathbf {s_1}\) , and produces a new span representation:
\(\mathbf {\tilde{s}} = \mathbf {W}^T \big [ \mathbf {s_0}; \mathbf {s_1} \big ] + \mathbf {b},\) 
where \(\mathbf {W}\)  and \(\mathbf {b}\)  denote the parameters for the composition function.
The new span representation \(\mathbf {\tilde{s}}\)  is pushed back to the stack to replace the original two spans: \(\mathbf {s_0}\)  and \(\mathbf {s_1}\) .

Capturing discontinuous dependencies
We hypothesise that the interactions between spans in the stack and tokens in the buffer are important factors in recognising discontinuous mentions. Considering the example in Figure REF , a span in the stack (e.g., `muscle') may need to combine with a future token in the buffer (e.g., `fatigue'). To capture this interaction, we use multiplicative attention [144]} to let the span in the stack \(\mathbf {s_i}\)  learn which token in the buffer to attend, and thus a weighted sum of the representation of tokens in the buffer \(\mathbf {B}\) :
\(\begin{split}\mathbf {s^a_i} & = \textsc {Attention} (\mathbf {s_i}, \mathbf {B}, \mathbf {B}) \\& = \textsc {softmax} (\mathbf {s_i^TW^a_iB})\mathbf {B}.\end{split}\) 
We use distinct \(\mathbf {W}_i^a\)  for spans in different positions \(\mathbf {s_i}\)  separately.

Selecting an action
Finally, we build the parser representation as the concatenation of the representation of top three spans from the stack (\(\mathbf {s_0}, \mathbf {s_1}, \mathbf {s_2}\) ) and its attended representation (\(\mathbf {s^a_0}\) , \(\mathbf {s^a_1}\) , \(\mathbf {s^a_2}\) ), as well as the representation of the previous action \(\mathbf {a}\) , which is learned using a simple unidirectional LSTM. If there are less than 3 spans in the stack or no previous action, we use randomly initialised vectors \(\mathbf {s_{empty}}\)  or \(\mathbf {a_{empty}}\)  to replace the corresponding vector. This parser representation is used as input for the final softmax prediction layer to select the next action.
Note that, given one parser state, not all types of actions are valid. For example, if the stack does not contain any span, only SHIFT and OUT actions are valid because all other actions involve popping spans from the stack. We employ hard constraints that we only select the most likely action from valid actions.

Experimental Results 
To evaluate the effectiveness of our proposed model, we run experiments on previous described three datasets: CADEC, ShARe/CLEF 2013 and ShARe/CLEF 2014, and compare the effectiveness of our model against several baselines.
Baseline models
We choose one flat NER model which is strong at recognising continuous mentions, and two discontinuous NER models as our baseline models:
Flat model
To train the flat model on our datasets, we use an off-the-shelf framework: Flair  [145]}, which achieves the state-of-the-art performance on CoNLL 2003 dataset. Recall that the flat model cannot be directly applied to datasets containing discontinuous mentions. Following the practice in [146]}, we replace the discontinuous mention with the shortest span that fully covers it, and merge overlapping mentions into a single mention that covers both. Different from [146]}, we apply these changes only on the training set, and not on the development and the test sets.

BIO extension model
The original implementation in [21]} used a CRF model with manually designed features. We report their results on CADEC in Table REF  and re-implement a BiLSTM-CRF-ELMo model using their tag schema (denoted as `BIO extension' in Table REF ).

Graph-based model
The original paper of [16]} only reported the evaluation results on sentences which contain at least one discontinuous mention. We use their implementation to train the model and report evaluation results on the whole test set (denoted as `Graph' in Table REF ). We argue that it is important to see how a discontinuous NER model works not only on the discontinuous mentions but also on all the mentions, especially since, in real datasets, the ratio of discontinuous mentions cannot be made a priori.

Experimental setup
As CADEC does not have an official train-test split, we follow [21]} and randomly assign 70% of the posts as the training set, 15% as the development set, and the remaining posts as the test set. The train-test splits of ShARe/CLEF 13 and 14 are both from their corresponding shared task settings, except that we randomly select 10% of documents from each training set as the development set. The original ShARe/CLEF 14 task focuses on template filling of disorder attributes: that is, given a disorder mention, recognise the attribute from its context. In this work, we use its mention annotations and frame the task as a discontinuous NER task. Micro average strict match \(F_1\)  score is used to evaluate the effectiveness of the model. The trained model which is most effective on the development set, measured using the \(F_1\)  score, is used to evaluate the test set. All experiments are repeated five times using different random seeds and averaged results are reported.

Results
When evaluated on the whole test set, our model outperforms three baseline models, as well as over previous reported results in the literature, in terms of recall and \(F_1\)  scores (Table REF ).
<TABLE>The graph-based model achieves highest precision, but with substantially lower recall, therefore obtaining lowest \(F_1\)  scores. In contrast, our model improves recall over flat and BIO extension models as well as previously reported results, without sacrificing precision. This results in more balanced precision and recall. Improved recall is especially encouraging for our motivating pharmacovigilance and medical record summarization applications, where recall is at least as important as precision. Note that most of these previous models are tailored for specific entity categories, and utilise domain-specific resources. For example, [151]}, the best-performing system participated in the ShARe/CLEF 2013 shared task, utilise several external domain-specific resources, such as MetaMap, cTAKES and UMLS. We avoid these tailored resources in our model. We argue that this makes our model more generic and robust, especially since we apply the hyper-parameters tuned on CADEC directly to ShARe/CLEF datasets and obtain similar improvements with respect to the benchmarks.
Effectiveness on recognising discontinuous mentions
Recall that only 10% of mentions in these three datasets are discontinuous. To evaluate the effectiveness of our proposed model on recognising discontinuous mentions, we follow the evaluation approach in [16]} where we construct a subset of test set where only sentences with at least one discontinuous mention are included (Table REF ). We also report the evaluation results when only discontinuous mentions are considered (Table REF ). Note that sentences in the former setting usually contain continuous mentions as well, including those involved in overlapping structure (e.g., `muscle pain' in the sentence `muscle pain and fatigue'). Therefore, the flat model, which cannot predict any discontinuous mentions, still achieves \(38\%\)  \(F_1\)  on average when evaluated on these sentences with at least one discontinuous mention, but fails to recognise discontinuous mentions.
<TABLE><TABLE>Our model again achieves the highest \(F_1\)  and recall in all three datasets under both settings. The comparison between these two evaluation results also shows the necessity of comprehensive evaluation settings. The BIO extension model outperforms the graph-based model in terms of \(F_1\)  score on CADEC, when evaluated on sentences with discontinuous mentions. However, it achieves only \(1.8\)  \(F_1\)  when evaluated on discontinuous mentions only. The main reason is that most of discontinuous mentions in CADEC are involved in overlapping structure (\(88\%\) , cf. Table REF ), and the BIO extension model is better than the graph-based model at recognising these continuous mentions. On ShARe/CLEF 2013 and 2014, where the portion of discontinuous mentions involved in overlapping is much less than on CADEC, the graph-based model clearly outperforms BIO extension model in both evaluation settings.
Graph based model again achieves highest precision on all three datasets. It also outperforms BIO extension model on ShARe/CLEF 2013 and 2014 in terms of \(F_1\)  score, but not on CADEC. Graph based model employs lots of handcrafted features for clinical notes (e.g., note type, section name, word-level semantic category extracted from UMLS). These handcrafted features usually lead to high precision but are not general enough to recall unseen mentions. In addition, they usually do not generalise well in different types of text (i.e., online posts in CADEC). In contrast, we avoid these handcrafted features in our model.

Analysis
Impact of mention and interval length
Discontinuous mentions usually represent compositional concepts that consist of multiple components. Therefore, these mentions are usually longer than continuous mentions (Table REF ). In addition, intervals between components make the total length of span involved even longer. Previous work shows that flat NER performance degrades when applied on long mentions [153]}, , [154]}.
<FIGURE>We experiment to measure the ability of different models on recalling mentions of different lengths, and to observe the impact of interval lengths. We find that the recall of all models decreases with the increase of mention length in general (Figure REF  (a – c)), which is similar to previous observations in the literature on flat mentions [154]}. However, the impact of interval length is not straightforward. Mentions with very short interval lengths are as difficult as those with very long interval lengths to be recognised (Figure REF  (d – f)). On CADEC, discontinuous mentions with interval length of two are easiest to be recognised (Figure REF  (d)), whereas those with interval length of three are easiest on ShARe/CLEF 2013 and 2014. We hypothesise this also relates to annotation inconsistency, because very short intervals may be overlooked by annotators.
Our method achieves highest recall among all models in most settings. This demonstrates our model is effective to recognise both continuous and discontinuous mentions with various lengths. In contrast, the BIO extension model is only strong at recalling continuous mentions (outperforming the graph-based model), but fails on discontinuous mentions (interval lengths larger than zero).

Impact of overlapping structure
Another characteristic of discontinuous mentions is that they usually overlap (Section ). Previous study shows that the intervals between components can be problematic for coordination boundary detection ([156]}). Conversely, we want to observe whether the overlapping structure may help or hinder discontinuous entity recognition. We categorise discontinuous mentions into different subsets, described in Section , and measure the effectiveness of different discontinuous NER models on each category.
<TABLE>From Table REF , we find that our model achieves better results on discontinuous mentions belonging to No overlap category on ShARe/CLEF 2013 and 2014, and Overlap at left category on CADEC and ShARe/CLEF 2014. Note that No overlap category accounts for half of discontinuous mentions in ShARe/CLEF 2013 and 2014, whereas Overlap at left accounts for half in CADEC (Table REF ). Graph-based model achieves better results on Overlap at right category. On the Multiple overlaps category, no models is effective Our model cannot recognise all mentions belonging to this category in theory. For example, if two mentions overlap at both the left and the right, our model can predict only one of them., which emphasises the challenges of dealing with this syntactic phenomena (examples can be found in Section REF ). We note, however, the portion of discontinuous mentions belonging to this category is very small in all three datasets.
Although our model achieves better results on No overlap category on ShARe/CLEF 2013 and 2014, it does not predict correctly any discontinuous mention belonging to this category on CADEC. The ineffectiveness of our model, as well as other discontinuous NER models, on CADEC No overlap category can be attributed to two reasons: 1) the number of discontinuous mentions belonging to this category in CADEC is small (around 12%), rending the learning process more difficult. 2) the gold annotations belonging to this category are inconsistent from a linguistic perspective. For example, severity indicators are annotated as the interval of the discontinuous mention sometimes, but not often. Note that this may be reasonable from a medical perspective, as some symptoms are roughly grouped together no matter their severity, whereas some symptoms are linked to different concepts based on their severity and severe adverse drug reactions are especially on the radar.

Example predictions 
We find that previous models often fail to identify discontinuous mentions that involve long and overlapping spans. For example, the sentence ‘Severe joint pain in the shoulders and knees.’ contains two mentions: ‘Severe joint pain in the shoulders’ and ‘Severe joint pain in the knees’. Graph-based model does not identify any mention from this sentence, resulting in a low recall. The BIO extension model predicts most of these tags (8 out of 9) correctly, but fails to decode into correct mentions (predict ‘Severe joint pain in the’, resulting in a false positive, while it misses ‘Severe joint pain in the shoulders’). In contrast, our model correctly identifies both of these two mentions.
<TABLE>Another observation is that no model can fully recognise mentions which form crossing compositions. For example, the sentence ‘Joint and Muscle Pain / Stiffness’ contains four mentions: ‘Joint Pain’, ‘Joint Stiffness’, ‘Muscle Stiffness’ and ‘Muscle Pain’, all of which share multiple components with the others. Our model correctly predicts ‘Joint Pain’ and ‘Muscle Pain’, but it mistakenly predicts ‘Stiffness’ itself as a mention (Table REF ).

Ablation studies
To empirically evaluate the importance of attention and ELMo components, we test the performance of model variants where attention and ELMo are removed separately on CADEC and ShARe/CLEF 2013 datasets.
<TABLE>The results in Table REF  show that removing attention hurts the performance when evaluated on sentences with discontinuous mentions (w. Disc. columns), but have little impact on the complete test set where continuous mentions are prevalent. Since we use BiLSTM to derive contextual representation for each token, we believe these contextual representations are effective at recognising continuous mentions, but have trouble identifying intervals within discontinuous mentions. Attention mechanism, via allowing tokens interacting with distant tokens, can capture additional discontinuous dependencies which are not captured by BiLSTM. In terms of the ELMo component, we find that it contributes approximately 2 \(F_1\)  score when evaluated on the complete test set and around 4 \(F_1\)  when evaluated on sentences with discontinuous mentions, demonstrating the usefulness of pretrained word representations.

Summary
Recognising discontinuous mentions that represent compositional concepts is important for downstream applications such as pharmacovigilance. We propose an end-to-end transition-based model for discontinuous NER. It makes use of specialised actions and attention mechanism to determine whether a span is the component of a discontinuous mention or not. We evaluate our model on three biomedical datasets with a substantial number of discontinuous mentions and demonstrate that our model can effectively recognise discontinuous mentions without sacrificing the accuracy on continuous mentions. Analysis also suggests that our model is better than existing discontinuous NER models at handling long mentions, resulting in higher recall.

Conclusions 
Recognising biomedical names from scholarly articles, clinical notes, and social media data is a fundamental NLP task that can benefit many downstream biomedical NLP and information retrieval applications. However, due to the unique characteristics of biomedical names and the stylistic variation in biomedical language—used by biomedical researchers, practitioners, patients and other participants—biomedical NER needs to solve challenges comparatively less studied in the generic domain NER applications.
In this thesis, we first identified challenges of applying standard sequence tagger to recognise biomedical names. Although sequence tagging techniques have demonstrated their effectiveness in generic domain NER, achieving state-of-the-art performance in many benchmarks, they suffer from three problems when being applied in the biomedical domain:

Biomedical names may consist of non-consecutive spans and they may overlap with each other. The main reason of this complex structure in biomedical names is that many biomedical concepts are compositional. For example, a symptom description may consist of several components: body location, severity indicator, and general feeling, and these components may locate far away from each other.

Training of neural based sequence taggers usually requires large training set, which is difficult to obtain in the biomedical domain. Annotating biomedical NER datasets usually requires domain-knowledge, and sometimes even unlabelled data are unavailable due to legal reasons.

State-of-the-art sequence taggers are usually enhanced by language representation models pre-trained on large set of generic domain unlabelled data. Domain shift between these out-of-domain pre-training data and the target biomedical data usually results in a performance drop.

Targeting these three problems, we explored the corresponding research directions.
We proposed a transition-based model for discontinuous NER. The proposed model is an end-to-end model with generic neural encoding that allows us to leverage specialised actions and attention mechanism to determine whether a span is the component of a discontinuous mention or not. We evaluate our model on three biomedical datasets with a substantial number of discontinuous mentions and demonstrate that our model can effectively recognise discontinuous mentions without sacrificing the accuracy on continuous mentions.
We designed several easy to use data augmentation methods for the NER task: Label-wise token replacement, Synonym replacement, Mention replacement and Shuffle within segments. These augmentations do not rely on any externally trained models, such as machine translation models or syntactic parsing models, which are by themselves difficult to train in a low-resource domain-specific scenario. Through experiments on two biomedical datasets, we show that simple data augmentation can improve performance even over strong baselines, where large scale pre-trained language representation models are used. We leave the exploration of combining these data augmentation methods with other NER models, such as the transition-based model we proposed, for future work.
We analysed different aspects of similarity between domains, and employed cost-effective measures to quantify domain similarity. We demonstrated that these measures are good predictors of the usefulness of pre-trained language representation models on downstream NER task. We find that human intuition favour field (the subject matter being discussed) over tenor (the participants of the discourse and their purpose) when they select in-domain pre-training data. Results suggest that this intuition may be unreliable when the target data set locates in the intersection of several domains.
Based on the discoveries presented in this thesis, we see two future directions worth exploring.

The first one is on incorporating existing biomedical knowledge base. In Chapter , we explored the data augmentation methods that make use of the original training set and a generic lexical database of English. Similar augmentation methods can also be applied to biomedical knowledge base. For example, a biomedical concept—defined using CUI in UMLS—can have several aliases from various vocabularies. These aliases can be used to create augmented sentences in the data augmentation settings. In Chapter , we investigated the impact of pre-training data on the effectiveness of pre-trained language representation models, where we pre-train models using only the unlabelled text. We believe more sophisticated pre-training tasks based on biomedical knowledge base can create pre-trained models that capture both language and biomedical knowledge.

The second direction is on investigating the impact of NER performance on downstream tasks, such as entity linking, relation extraction, or biomedical literature search. In Chapter , we showed that our proposed model can effectively recognise discontinuous biomedical names without sacrificing the performance of continuous ones. It worth investing how this improvement can benefit downstream tasks whose results are directly presented to end users.

