Memory, Show the Way: Memory Based Few Shot Word Representation Learning

Distributional semantic models (DSMs) generally require sufficient examples for a word to learn a high quality representation. This is in stark contrast with human who can guess the meaning of a word from one or a few referents only. In this paper, we propose Mem2Vec, a memory based embedding learning method capable of acquiring high quality word representations from fairly limited context. Our method directly adapts the representations produced by a DSM with a longterm memory to guide its guess of a novel word. Based on a pre-trained embedding space, the proposed method delivers impressive performance on two challenging few-shot word similarity tasks. Embeddings learned with our method also lead to considerable improvements over strong baselines on NER and sentiment classification.

Introduction
Humans can learn a new word quickly from minimal exposure to its context, as in the following example: The Labrador runs happily towards me, barking and wagging its tail.
Even this is the first time one hears about Labrador, we can guess it should be an animal or even further a dog easily, since it runs, barks and has a tail. Such ability to efficiently acquire representation from small data, namely fast mapping, is thought to be the hallmark of human intelligence that a cognitive plausible agent should strive to reach (Xu and Tenenbaum, 2007;Lake et al., 2015).
However, as the mainstream of text representation learning in NLP, most distributed semantic models (DSMs) don't fare well in tiny data (Lazaridou et al., 2017;Herbelot and Baroni, 2017;Wang et al., 2016). Even if they have learned a lot of words, they still need sufficient examples to acquire a high-quality representation for a novel word. This not only constitutes a blow to DSM's cognitive plausibility but also limits its practical usage in NLP. Because plentiful enough data is not always available, especially in domain specific tasks. Even if a large corpora is at hand, low-frequency words in it are still more than highly frequent ones, according to the Zipfian distribution of natural language.
Given the above reasons, it's desirable to build a word embedding method capable of acquiring high quality representations with limited contexts, i.e., few shot word representation learning. We take lessons from hypothesis constraint (HC) theory to achieve this goal. HC is an influential proposal for human's fast mapping (Xu and Tenenbaum, 2007). It indicates that people learn a new word by eliminating incorrect hypotheses about the word meaning, based on usage of the target word and prior knowledge of context words. This is instructive to us since embedding a word in the high-dimensional vector space also faces nearly unlimited candidate hypotheses (Wang et al., 2018) . General DSMs can't efficiently handle these candidates, so they fall back on multiple context examples to find the path, while we propose to let a memory show the way. We augment DSM with an longterm memory to transfer knowledge from a large general domain corpora to adapt the representation learning on the small text. In context of the HC theory, we directly constrain the hypothesis a DSM makes about the target word by its current usage and prior knowledge acquired from a large corpora. Experiments show our method makes educated guess of a novel word efficiently with fairly limited examples, just as humans do in the fast mapping.
It's worth noting that us attaching importance to few-shot word representation learning doesn't mean we need to learn words, no matter frequent or rare, all in the few-shot way. DSMs have done pretty well in frequent words learning with large corpus. We augment a DSM with an external memory for few-shot representation learning, under the assumption that gradual acquisition of frequent words plus fast learning of rare words make an integrated word representation learning scheme. Our ultimate goal is certainly an allround architecture that learns text representations from any amount of data. We believe Mem2Vec, which bridges the word representation learning from big data to small text, will be a building block of that ideal architecture. The primary contribution of this work is a memory augmented word embedding model with a fast adaptation mechanism, capable of learning representations efficiently from tiny data. Experimental results show that the proposed Mem2Vec learns high quality target word representation with both single informative sentence and a few casual sentences as contexts. To show its performance in downstream applications, Mem2Vec is used to pre-train embeddings for three NER tasks and also surpasses strong baselines. Since our model transfers from general domain corpus to a target small text, it is highly possible to face the problem of domain shift. Mem2vec is impressively competent in tackling domain shift, as demonstrated in a series of cross-domain sentiment classification tasks.

Related Work
Rare Word Embedding Acquiring representations for rare words has long been a well-known challenge of natural language understanding (Herbelot and Baroni, 2017;. Khodak et al. (2018) learn a linear transformation with pretrained word vectors and linear regression, which can be efficient adapted for novel words. Lazaridou et al. (2017) directly sum the context embedding of a novel word as its representation, based on a pre-trained embedding space. Though not explicitly stated, their idea actually matches the HC theory. They constrain the hypothesis solely within the current context of the target word which we think is not enough. We constrain the hypothesis with memory and the context. Another strand of solutions rely on auxiliary information, such as morphological structure (Luong et al., 2013;Kisselew et al., 2015) and external knowledge (Long et al., 2017). Lazaridou et al. (2013) derive morphologically complex words from sub-word parts with phrase composition methods. Ling et al. (2015) read characters of the rare word with a bidirectional LSTM to deal with open vocabulary problem in language modeling and NER. Hill et al. (2016) learn an embedding of a dictionary definition to match the pre-trained headword vector, while Weissenborn (2017) refines the word embeddings with explicit background knowledge from a commonsense knowledge base. Different from this strand of work, our method doesn't fall back on auxiliary information. We acquire knowledge from a large unlabeled general domain corpora which is widely available.
Cross Domain Word Embedding The knowledge accumulation phase of our model aims to learn an embedding space from a large general domain corpora. This is partially in line with cross domain word embedding work. Among these work, a strand of approach hypothesizes that a word frequent in multiple domains should mean nearly across these domains. Bollegala et al. (2015) call such word pivot, share its embeddings across domains and use them to predict the surrounding non-pivots. Yang et al. (2017) selectively incorporate source domain information to target domain word embeddings with a word-frequencybased regularization. These pivot-based methods have delivered improvements on sentiment analysis and NER. However, they have a defect that only limited target domain words benefit from the knowledge transfer.
Memory based Meta Learning Memory augmented neural networks (MANN) are widely used in different tasks for efficient recall of experience and fast adaptation to new knowledge (Bahdanau et al., 2014;Merity et al., 2017;Miller et al., 2016;Grave et al., 2017;Sprechmann et al., 2018;. Intuitively, Meta-learning, which aims to train a model that quickly adapts to a new task, should benefit from memory architecture, and empirically it does do (Santoro et al., 2016;Duan et al., 2016;Wang et al., 2016;Munkhdalai and Yu, 2017;Kaiser et al., 2017). The memory we use is closely related to (Kaiser et al., 2017), but still get three major differences. First,they only retrieve the single nearest neighbor from the memory while we retrieve an average of the K nearest neighbors weighted by how they match the current context. Second, they focus on supervised learn-  ing and don't have a fast adaptation mechanism for acquiring representation. Third, they update the memory according to whether the returned value is strictly the same as the target. However, synonyms are common in natural language text. We thus take a softer criterion and update the memory according to vector similarity between the addressed value and the target word embedding.

Methods
Our model in brief is a neural network based DSM augmented by a longterm memory. As showed in Fig.1, it operates in two consecutive phases, first accumulating knowledge and then doing fast adaptation on new words, just as the human learning process goes. In the knowledge acquisition phase, we train the memory augmented DSM to learn a semantic space. We also accumulate similar contexts of target words in the memory and gradually form "prototype" representations. The pre-trained embedding space and the saved prototypes are just the knowledge acquired. The fast adaption phase occurs when we need to learn a new word from minimal context. In this phase, we directly combine the context embedding and retrieve content from the memory to form the target word representation.
In the following sections, we will first introduce the memory architecture and the content based addressing. We then detail how exactly our model runs respectively in the knowledge accumulation and fast adaption phase.

Memory Addressing
M is a non-parametric key-value memory which stores a key-value pair (k i , v i ) in each memory slot i. Inspired by (Kaiser et al., 2017), we keep an additional vector A tracking the age of slots. The initial age of all is zero. So the whole memory M looks like (K m×ks , V m , A m ) where m denotes memory size and ks denotes key vector size. Given a normalized query q, its nearest neighbor in M is defined as any of the keys that maximize the cosine similarity with q: During training, a query to the memory M searches k nearest neighbors which is a natural extension to (1): We take an average weighted by how the addressed memory slots match the query: This is actually a dot-product attention on the k nearest neighbors. R K , R V are the final output of the memory. Note that here we use softmax with temperature T : T is normally set to 1. Using a higher value for T produces a softer probability distribution. We set different temperatures in the knowledge accumulation and fast adaptation phase, which will be detailed in the following subsections.

Knowledge Accumulation
Given a target word embedding t j with its context embedding c j as input, we query the memory with c j as (1)-(3) and retrieve R k , R V . The semantic relation between the current example and the retrieved content from memory is hoped to be consistent from context to target words, so we derive the following loss: We also hope the target word representation fully incorporates context information and stay far from negative examples, so we also inherit the loss from (Mikolov et al., 2013): where D denotes the whole corpora, and #(t j , c j ) means times the target and the context word cooccur. The word p i is a negative sample sampled from the distribution P (t j ), as Mikolov et al. (2013) do. We minimize the sum of L m and L s : Memory Update. After each query, we update a memory slot according to how frequently the key is addressed and how useful the addressed value is. The update is done piecewise according to similarity between the addressed values (V n 1 , ..., V n k ) and the target word. For all the addressed values V n i whose similarity to the target word is higher than the threshold β, we only update its corresponding key by taking a weighted average of the current key and the query: Otherwise, it means no addressed value correlates enough with the target, we then choose memory slots n with maximum age and rewrite the stored items in it: The age of each updated slot will be reset to zero while all other non-updated slots get incremented by 1 in age. Memory updated in this way gradually accumulates similar contexts of a word into the same slot, which in another word, forms the prototype representation of a word.

Fast Adaptation
Now we show how to poll the memory to efficiently learn a new word representation from limited context. This is where the hypothesis constraint takes place. To be specific, given a new word embedding t * j to be learned and its context embedding c * j , we retrieve memory relevant to c * j as (2)-(3) and get R * K . Then we adapt context embedding with the retrieved memory to form the target word representation: where α can be tuned a hyper-parameter or learned with regression models. Actually we also try to incorporate R * V , but the aggregated prototype R * K seems to continuously perform better. We here pay additional attention to the softmax temperature T . T is emphasized since it conditions how the model "treats" the retrieved memory. Contexts are fairly limited in the few-shot case, so how the retrieved memory is treated crucially affects the quality of the learned representation. A higher temperature leads to a softer attention distribution , which means the model will be more likely to sample from all retrieved contents. A lower temperature makes the model focus more on the memory with highest similarity to the query. We predict a slightly higher temperature will generally be better in the fast adaptation phase. Since the HC theory points out that hypotheses are not in strict mutual-exclusions, they overlap with each other which corresponds to the higher-temperature condition. We will test this in the experiments.

Few-shot Word Similarity Tasks
We test the proposed method on two few-shot word similarity tasks. Fig.2 gives examples of the two tasks. In the following subsections we will introduce these datasets in detail and show the performance of tested methods on these tasks.

Tasks and Datasets
Definitional Nonce Task We evaluate on the Definitional Nonce dataset (Herbelot and Baroni, Nonce Definition Provided Context : ___ international inc is an american multinational conglomerate company that produces a variety of commercial and consumer products engineering services and aerospace systems for a wide variety of customers from private consumers to major corporations and governments Ground Truth Word: Honeywell Chimera-l2 Provided Context : 1. Canned sardines and ____ between two slices of whole meal bread and thinly spread Flora Original. 2. Erm, ____, low fat dairy products, incidents of heart disease for those who have an olive oil rich diet. Probe Words: rhubarb, onion, pear, strawberry, limousine, cushion Human Response: 2.57, 4.43 , 3.86, 3.71, 1.43, 2.14 Figure 2: Examples of the Nonce Definition and Chimera Task 2017) to simulate the process where a competent speaker learns a novel word from one informative sentence. 1000 words are included in the dataset as targets, with 700 for training and 300 for testing. Each target word corresponds to only one sentence extracted from its Wikipedia definition as context. All context sentences have been manually checked to be definitional enough to describe the corresponding target words. After tuning parameters on training data, the model is required to learn the target word representation with the provided context in test set. Learned representations are assessed by similarity to ground truth vectors produced in exposure to the whole corpora. We use the Reciprocal Rank (RR) of the ground vector in all nearest neighbors to the learnt representation for fair comparison of different methods, following Herbelot and Baroni (2017)'s settings. The mean value of RR over all test instances in the dataset is calculated as the final score.
Chimera Task Our second evaluation on the Chimera dataset (Lazaridou et al., 2017) means to simulate the case where a speaker learns the new word in a more casual multi-sentence context, not as highly informative as definitions in the Nonce dataset. There are 3 sub-tasks in Chimera:L2, L4 and L6, respectively providing 2, 4, 6 sentences as context to for each of the 330 instances in the dataset. The tested model needs to learn target word representation from the provided contexts. The similarity between learned embeddings and each of the probe words is measured and compared to human judgments by Spearman correlation. The final score is the average Spearman across all test pairs.

Baselines
Our model is compared to several baselines, including Word2Vec (Mikolov et al., 2013), GloVe (Pennington et al., 2014), SUM (Lazaridou et al., 2017) and N2V (Herbelot and Baroni, 2017). Glove and Word2Vec are representatives of traditional DSMs. With them we want to test how exactly traditional DSMs perform in the few shot representation learning without any additional mechanism for small data. SUM and N2V are proposed especially for learning on small corpus. They adapt Word2Vec's skip-gram structure for incremental learning and show improvements on the Chimera dataset. They partially match the HC theory which Mem2Vec is based on. Note that several rare word learning methods (Long et al., 2017;Xu et al., 2014;Lazaridou et al., 2013) that rely on auxiliary information don't apply with most of our task settings. In the Nonce and Chimera task, context for target word learning is strictly limited for fair comparison, so external knowledge is banned. And the target word, as showed in Fig.2, is just a slot which doesn't provide any morphological hints, so sub-word methods are also excluded.
Both the above baselines and the proposed Mem2Vec use a dump of Wikipedia to learn a fundamental semantic space. To be specific, N2V and SUM use embeddings pre-trained by Word2Vec, while Mem2Vec acquires prior-knowledge, all from that Wiki corpora. We calculate correlation with the similarity ratings in the MEN and SIM-LEX dataset to verify if the pre-trained semantic space is ready for use.

Results
Nonce Definition Task We show the results of Nonce Definition Task in Table 1. Before analyzing the results we need to explain that MRR achieved by the tested models seems pretty low. This is no odd since matching the ground truth word in a vocabulary of 210,512 sets a potentially very large denominator in the reciprocal rank calculation. Our model achieves an MRR of 0.05416, which means the median rank of the true vector is 518 in the challenging two hundred thousand neighbors, surpassing all the baselines. N2V and SUM also deliver satisfactory performance with N2V working better. We are sorry to find that the naive Word2Vec and GloVe totally fail in the Nonce task, supporting the importance of adapting traditional DSMs for few-shot word representation learning. Chimera Task The results on 3 chimera tasks are shown in Table 1, too. Mem2vec out-performs baselines in all the 3 context length settings. SUM performs steadily well from Nonce to the Chimera task, suggesting the effectiveness of constraining hypothesis space with contexts. But the continuous improvement of Mem2Vec over SUM confirms the advantage of our model, which incorporates "global" semantic information from the memory with the local contexts. N2V also works here but not as well as in the Nonce task, probably because the contexts in chimera are not as informative. Such performance drop may indicate N2V's limited scalability to downstream NLP tasks since not all real world texts are as informative as in Nonce Definition Task. We will test this speculation with NER tasks in section 5.

Memory Parameter Analysis
We are interested to know how the two key parameters of memory, the softmax temperature T , and the memory size influence the quality of learned representations. We use the Nonce Definition task as the testbed. While studying the influence of one parameter, the other parameters are fixed. We run the model for 3 times with each candidate parameter and calculate the average precision as the final score. Softmax Temperature Fig.3 (left) shows task performance under different softmax temperatures in semilog coordinate. We are a little bit surprised to find that it roughly fits a normal distribution and a mid-high temperature leads to best performance. A mid-high T means the model doesn't give too large or too small weights to certain retrieved memory. This meets the HC theory about how humans weight the constrained hypothesis. We don't just trust a single hypothesis, nor do we treat all the hypotheses equally. The experiments shows similar principle also applies to our model. Memory Size Fig.3 (right) shows task performance under different memory size. We find that increasing the memory size does lead to improved performance . But the improvements tend to be minor after the memory size is larger than 20,000. We owe it to the fact that we does not save specific examples, we accumulate similar contexts together in the memory to form prototypes. While we retrieve the memory, prototypes can be combined in different ways to represent multiple examples, thus a smaller memory can also work as well as the bigger one.

Extrinsic Tasks
We hope that the learned representations not only perform well on word similarity tasks but also apply to downstream NLP tasks. NER on domain specific datasets is an ideal benchmark. Named entities in these datasets are relatively low in frequency and not well covered by general domain corpus, tough for a traditional DSM to learn. Besides, while transferring from general domain corpus to a target small text, domain shift is a highly possible issue. We test if Mem2Vec could tackle the domain shift with a series of cross-domain sentiment analysis tasks.

Tasks and datasets
Domain Specific NER We use BioNLP11species (Kim et al., 2011), AnatEMs (Pyysalo and Ananiadou, 2013) and NCBI-disease (Dogan et al., 2014) dataset, respectively from taxonomy, anatomy and pathology literatures. We train embeddings with tested methods to initialize the recognizer, whose performance then demonstrates whether the tested models learn representations well for rare words. Cross Domain Sentiment Classification cross domain sentiment classification on Amazon Review dataset (Blitzer et al., 2007) is chosen as a benchmark. This dataset includes reviews from 4 product categories: books, DVDs, kitchens and electronics, suitable for the cross-domain setting. Using one as source domain and one as the target, we get 16 pairs for experiments. We train the classifier with source domain data and directly test it on the target domain, using the pre-trained embed-dings as input feature. Note that through this task we also test how Mem2Vec performs when transferring from a small text, since in all the above experiments we learn prior knowledge from a large corpora.

Baselines
Except for the four baselines considered in word similarity tasks, we also compare with DAREP (Bollegala et al., 2015) and CRE (Yang et al., 2017) in the NER and sentiment classification tasks. They are both pivot-based methods for cross domain embedding learning which fare well in some downstream tasks. Besides we introduce SCL (Blitzer et al., 2006), a well-cited crossdomain sentiment analyser, as a baseline only for the sentiment classification task. For NER, we use pre-trained embeddings by the tested methods as only input features for a LSTM-CRF recognition model (Lample et al., 2016). We simply mix the Wikipedia corpora with a dump of PubMed as our source corpora. Note that N2V and SUM can't be directly used to pre-train embeddings for downstream tasks since they focus on novel word learning. We thus explicitly divide words which occur less than 5 times as rare words while others as frequent words. N2V and SUM learn the frequent words with Word2Vec and learn the rare words in their own way. This setting also applies to the sentiment classification task.
For sentiment classification, we use a multilayer perceptron (MLP) as the classifier, with one hidden layer of 400 nodes, ReLu activation and softmax output function. Table 2 shows the results of domain specific named entity recognition. Used for pre-training embeddings, Mem2Vec achieves higher F1-score than all the baselines. It first surpasses CRE and DAREP that only bring slight improvements over Word2Vec. CRE and DAREP are both methods which relies on words with cooccurance patterns in source and target domain as the pivots for cross-domain transfer. This indicates the advantage of Mem2Vec over the traditional word frequency based methods in fast mapping cases where word cooccurrence pattern is not clear.

Named Entity Recognition
Our improvements over the N2V and SUM are more obvious than in the two word similarity   tasks. This again affirms our speculation that constraining hypothesis solely with the context is not enough. In the setting of NER, the context of one named entity is likely to be filled with other named entities which are also low in frequency. Directly summing the context as SUM does or taking risk to enlarge the window size as N2V may lead to over-fitting. While every training step of our method incorporates relative information from all the experienced examples stored in the memory, alleviating the danger of learn representations that over fits the local contexts.
In addition, it's worth noting that parameter tuning for N2V is no picnic. In our experiments, the original settings: high learning rate, large window size and short iteration span don't lead to satisfactory performance. More conservative parameter selection gets N2V back in track but departs from its fast mapping intention. Fig.4 shows the results of Amazon Review sentiment classification. Mem2Vec delivers impressive performance, beating all the baselines in 10 of the total 12 pairs, including CRE and DAREP. This demonstrates the advantage of the memory as a transfer medium over the word-frequency based transfer of CRE and DAREP. But CRE and DAREP are still strong baselines in the cross domain task, surpassing SCL by a large margin. N2V and SUM are built for learning representation from small data, but they don't consider the possible domain discrepancy when using pre-trained embeddings on the target small text. So they don't bring much improvements over Word2Vec and GloVe. This also reminds us that to get the few-shot word representation learning methods in practical use, domain shift should be properly addressed.

Conclusion
We presented an integrated representation learning scheme which gradually learns from a big corpora and quickly adapt on tiny data. It accumulates knowledge with a long-term memory to adapt the representation learning of a novel word, in the few-shot learning case. Such adaptation means to constrain the "guess" of a DSM for the novel word according to the most relevant representation learning experience, inspired by hypothesis constraint theory for fast mapping. Experiments show the proposed method learns high quality representation from both highly informative and less definitional contexts in limited size. Pre-trained embeddings with our model also lead to improvements in Named Entity Recognition and sentiment analysis.
This work is our effort towards an ideal word representation learning scheme which learns from any amount of data. In the future work, we will explore more effective memory addressing and updating approaches to boost the few-shot representation learning. We believe not all examples are equally important and worth memorizing. Learning to memorize core examples should alleviate the data-hungry of representation learning methods.


extfieldsofstudy:
Computer Science
Medicine

text:
Semantic annotation of consumer health questions

Background Consumers increasingly use online resources for their health information needs. While current search engines can address these needs to some extent, they generally do not take into account that most health information needs are complex and can only fully be expressed in natural language. Consumer health question answering (QA) systems aim to fill this gap. A major challenge in developing consumer health QA systems is extracting relevant semantic content from the natural language questions (question understanding). To develop effective question understanding tools, question corpora semantically annotated for relevant question elements are needed. In this paper, we present a two-part consumer health question corpus annotated with several semantic categories: named entities, question triggers/types, question frames, and question topic. The first part (CHQA-email) consists of relatively long email requests received by the U.S. National Library of Medicine (NLM) customer service, while the second part (CHQA-web) consists of shorter questions posed to MedlinePlus search engine as queries. Each question has been annotated by two annotators. The annotation methodology is largely the same between the two parts of the corpus; however, we also explain and justify the differences between them. Additionally, we provide information about corpus characteristics, inter-annotator agreement, and our attempts to measure annotation confidence in the absence of adjudication of annotations. Results The resulting corpus consists of 2614 questions (CHQA-email: 1740, CHQA-web: 874). Problems are the most frequent named entities, while treatment and general information questions are the most common question types. Inter-annotator agreement was generally modest: question types and topics yielded highest agreement, while the agreement for more complex frame annotations was lower. Agreement in CHQA-web was consistently higher than that in CHQA-email. Pairwise inter-annotator agreement proved most useful in estimating annotation confidence. Conclusions To our knowledge, our corpus is the first focusing on annotation of uncurated consumer health questions. It is currently used to develop machine learning-based methods for question understanding. We make the corpus publicly available to stimulate further research on consumer health QA.

questions allow the consumers to more fully express their health information needs. Consumers, when they fail to find information using search engines, often turn to online health forums and QA websites, where they can express their health concerns as natural language questions. While such resources are convenient for expressing complex health questions, they also present the additional challenge of question understanding for information retrieval systems. Question understanding can broadly be defined as the task of extracting relevant elements or generating a formal representation of a natural language question, which can then be used to formulate a query to a search engine and retrieve relevant answers (answer retrieval) [6].
Most research in biomedical QA has focused on clinical questions asked by healthcare professionals [7], which are often succinct and well-formed. In contrast, consumer health questions are rife with misspellings, ungrammatical constructions, and non-canonical forms of referring to medical terms, are fairly long with much background information and multiple sub-questions, and are closer to open-domain language than to medical language [8].
To illustrate, consider the following email received by the customer service of the U.S. National Library of Medicine (NLM): (1) pls guide us.
Dear sir/Madam pls guide us recently we found one of woman staying with us ,she coughing and blood coming from mouth so she went to doctor on 2012 they did blood test and sputm test ct scan also they didnt find anything ,recently she went to indonesia [ Three sub-questions can be distinguished: one regarding treatment options for lung tuberculosis, another asking whether this disease is contagious and the third asking how to get tested. The question includes background information that can be considered irrelevant for answer retrieval (that the patient saw a doctor in 2012 and travelled to Indonesia). It also contains spelling and grammar errors that would render it difficult to process with standard NLP tools. For example, the disease in question has a word break error and is abbreviated and contagious is misspelled, making both sub-questions very difficult to parse automatically. Based on such characteristics, it has been suggested that consumer health QA systems should be designed with considerations different from those underlying QA systems targeting healthcare professionals [8].
NLM receives health-related questions and queries from a wide range of consumers worldwide. We have been developing a consumer health QA system to assist various NLM services in answering such questions. Question understanding is a core component of this system. As is typical with other NLP tasks, annotated corpora are needed to develop and evaluate automated tools addressing this task. Our earlier work relied on a small corpus [9], which was sufficient for evaluating the narrow focus of the prototype system developed, but was not large enough to train QA systems. In other work, we relied on curated consumer health questions [10,11]. In the present study, we aimed to address the gap by annotating a corpus of unedited, uncurated consumer health questions. We annotated 2614 consumer health questions with several semantic layers: named entities, question topic, question triggers, and question frames. The majority of the questions (67%) come from consumer health emails received by the the U.S. National Library of Medicine (NLM) customer service and are relatively long questions (Example 1). We refer to this subset as CHQA-email. The rest of the questions (33%) have been harvested from the search query logs of MedlinePlus, a consumer-oriented NLM website for health information, and are generally much shorter (CHQA-web). Despite being posed as search queries, these questions are expressed as natural language questions, rather than keywords. An example question from CHQA-web is provided below: (2) what teeth are more likely to get cavities?
In a previous study, we annotated a subset of the questions in CHQA-email with named entities [12]. The present study builds upon and significantly extends that work. Here, we describe corpus characteristics, annotation schemes used, annotation steps taken, and statistics on the resulting dataset. Additionally, we report a small study to automatically assess confidence of annotations, when annotation adjudication can be impractical. We make the resulting annotated corpus publicly available at https://bionlp.nlm.nih.gov/CHIQAcollections/CHQA-Corpus-1.0.zip. To our knowledge, the corpus is unique for its focus on uncurated consumer health questions and the level and breadth of semantic detail it incorporates. We believe that, with its size and coverage, it can stimulate further research in consumer health QA.
Most biomedical QA research has focused on information needs of healthcare professionals, and several corpora have been used for this purpose. The Clinical Questions Collection, maintained by the NLM, is a repository of 4654 questions collected by Ely et al. [36] and D' Alessandro et al. [37] at the point of care. Original questions, their short and general forms, their topics (e.g., management, diagnosis, epidemiology), keywords (e.g., Polymenorrhea, Menstruation Disorders) as well as physician and patient information are provided. This collection has been used to develop several machine learning-based QA systems, such as AskHERMES [38] and MiPACQ [39]. Other smaller corpora also exist; for example, Parkhurst Exchange and Journal of Family Practice 1 maintain curated question sets that have been used to develop CQA, a clinical QA system based on evidence-based medicine principles [40]. Patrick and Li [41] presented a set of 446 clinical questions from staff in an intensive care unit asking for patient-specific information from their electronic health records and created a question taxonomy based on answering strategies. Text REtrieval Conference (TREC) Genomics track provided a platform to study QA for biological research [42]. A set of 36 questions asking for lists of specific entities (antibodies, pathways, mutations, etc.) was manually developed to represent the information needs of the biologists (e.g., What [MUTATIONS] in the Raf gene are associated with cancer?). More recently, BioASQ challenge also focused on biomedical QA and presented a corpus of 311 questions, categorized by the type of answers they require: yes/no, factoid, list, and summary [43].
Two approaches to modeling and formally representing biomedical questions have been distinguished: empirical and domain-based models [6]. In the empirical approaches, the questions are categorized into a limited number of generic question types [36,41,44]. Domain-based models provide a high-level definition of the main entities and actors in the domain and their relationships. For example, the PICO framework represents the elements of a clinical scenario with four elements, population/problem (P), intervention (I), comparison (C), outcome (O), and has been used as the basis of the CQA system [40]. Other formal approaches have also been used to represent clinical questions; for example, they have been represented as conceptrelation-concept triples [45,46], Description Logic expressions [47], SPARQL queries [48], and λ-calculus expressions [49].
Early research on consumer health information seeking focused on analysis of consumer health queries [4,50]. Based on the finding that such queries are not effective in finding relevant information [4,5], several other small-scale studies specifically analyzed consumer health questions [3,[51][52][53]. For example, Zhang [3] analyzed 276 health-related questions submitted to Yahoo! Answers, a community QA website, on several dimensions, including linguistic style and motivation, and found that these questions primarily described diseases and symptoms (accompanied by some patient information), were fairly long, dense (incorporating more than one question), and contained many abbreviations and misspellings. Slaughter et al. [52] manually annotated the semantics of a small number of consumer health questions and physician answers using Unified Medical Language System (UMLS) Semantic Network and found that both patients and physicians most commonly expressed causal relationships. More recently, Roberts and Demner-Fushman [8] studied how consumer questions differed from those asked by professionals and analyzed 10 online corpora (5 consumer, 5 professional, including some that have been discussed above) at lexical, syntactic, and semantic levels using a variety of NLP techniques. Their findings largely mirror those of Zhang [3]. Additionally, they found substantial differences between consumer corpora and showed that consumer health questions are closer to open-domain language than to medical language. Based on these findings, they suggest that consumer health QA systems need to be designed with a different set of considerations, instead of naively adapting a professional QA system. Liu et al. [54] developed a SVM classifier to distinguish consumer health questions from professional questions, and used questions from Yahoo! Answers and the Clinical Questions Collection for training.
We developed several consumer health corpora in the context of our consumer health QA project. In early work, we used a small set of 83 consumer health questions about diseases/conditions (mostly genetic), which consisted of relatively simple questions received by the NLM customer service and frequently asked questions collected from Genetic and Rare Disease Information Center (GARD) 2 , to develop a rule-based approach to extract question type and question theme [9]. GARD also formed the basis of annotation of 13 question types on 1,467 questions focusing on genetic diseases [10]. These question types include anatomy (location of a disease), cause (etiology of a disease), information (general information about a disease), and management (treatment, management, and prevention of a disease), among others. The average interannotator agreement (Cohen's κ) for question types was found to be 0.79 (substantial agreement). Recognizing that consumer health questions are complex with multiple sub-questions related by a central theme (focus), Roberts et al. [11] also annotated the same GARD dataset with question focus and question decomposition, where individual sub-questions are identified within complex questions, and each sub-question is annotated for several elements, including the central theme (focus), contextual information (background) and the actual question, at sentence and phrase levels. Background sentences were further classified into one of several medical categories, such as diagnosis, symptoms, or family history. The average inter-annotator agreement was found to be 0.91 for focus and 0.72 for background class. These GARD-based datasets have been used to train and evaluate question type classifiers [55] as well as focus recognizers and question decomposition classifiers [56,57]. The questions in the GARD dataset, even though they were asked by consumers, are curated; thus, they are well-formed with few misspellings, and represent an intermediate step between simple questions addressed in [9] and the more typical, complex consumer health questions. Consequently, while good classification performance is reported on the GARD dataset, performance drops significantly on the more complex questions received by NLM [57]. To address this gap, in recent years, we focused on annotation of uncurated consumer health questions received by NLM, without any simplifying assumptions. Considering that misspellings are a major obstacle in question understanding and that off-the-shelf spelling correction tools do not work well on customer health questions, we developed a spelling correction dataset that consists of 472 NLM questions [58]. In this dataset, we annotated non-word, real word, punctuation, as well as word break errors. We also developed an ensemble method that uses edit distance, corpus frequency counts, and contextual similarity to correct misspellings. Finally, we created a dataset of 1548 questions taken from the same resource, de-identified them for personal health information, and annotated them for named entities (CHQ-NER) [12]. Each question was doubleannotated using 16 named entity types (e.g., ANATOMY, DRUG_SUPPLEMENT, PROBLEM, PROCEDURE_DEVICE, PERSON_ORGANIZATION), determined based on a round of practice annotation. The annotation was performed in both manual and assisted mode, in which existing named entity recognition tools were used to preannotate questions. After double-annotation of questions, they were reconciled by one of the expert annotators.
Moderate agreement (0.71 F 1 agreement with exact matching) was achieved, with slightly higher agreement in assisted mode (0.72).

Methods
In this section, we first describe our approach to representing consumer health questions. Next, we discuss corpus construction and the semantic annotation that we performed on this corpus in depth. We conclude this section by providing details on our attempt to automatically estimate confidence scores of semantic annotations, when full adjudication is not feasible due to personnel/time constraints.

Question representation
We briefly discussed several approaches to biomedical question representation above: empirical, domain-based, and formal. Our approach to representing consumer health questions can be viewed as a hybrid method. Like empirical approaches, we begin with existing questions and categorize them into a limited number of generic question templates, and, like formal approaches, we create structured representations of the information in these generic questions that we call question frames. Frame representation is similar to a predicate-argument structure, a semantic representation in which a predicate is linked to its arguments and the semantic roles of the arguments, such as THEME and AGENT, are specified [59]. A question frame consists of a question trigger indicating the question type (similar to predicate), one or more THEME arguments, and a set of optional arguments with other semantic roles, all linked to their textual mentions. All arguments of a frame correspond to named entities. Each sub-question in a consumer health question is represented with a separate frame. To illustrate, consider the question we discussed earlier, reproduced in Example 3. An example representation of this question consists of the frames in Table 1. The first frame indicates that the information request contains a question about the treatment of active lung tuberculosis, triggered by the verb do in what we have to do for her. Note that while a THEME argument is required for each frame, we only specify other arguments if they are likely to be relevant for answer retrieval. For example, in the first frame, one could specify woman as an argument with PATIENT role (i.e., the person to undergo treatment); however, since gender is unlikely to be significant in answering this particular question, its explicit representation is not required. In this sense, our representation is driven more by pragmatic concerns for question answering than by completeness.
Our prior research suggested that when the question topic 3 (often the THEME of a sub-question) and the question types in an information request are known, authoritative answers can be found in 60% of the cases [60]. Taking this into account, we also explicitly indicate the question topic in question representation. The question topic is indicated as a named entity attribute. In the question above, the named entity corresponding to the mention ACTIVE LUNG TB is marked as the question topic. The annotation of this question is illustrated in Fig 1.

Corpus construction and annotation
Our corpus consists of two parts: a set of 1740 questions harvested from consumer health emails received by the NLM customer service between 2013 and 2015 (CHQAemail) and another set of 874 questions harvested from MedlinePlus search query logs (CHQA-web).

CHQA-email
CHQA-email is an extension of an earlier corpus (CHQ-NER), reported in [12], and supersedes it. That corpus consisted of 1548 consumer health information requests received by the NLM customer service in 2014-15 and manually labeled as disease and drug questions by the staff. These requests were annotated for named entities. For details of the question selection procedure, see [12]. Protected health information (PHI), such as names, locations, contact information, has been manually replaced with surrogates in these questions.
As part of the present study, we extended CHQ-NER with 195 additional questions. Furthermore, 2 duplicate questions and 1 non-question request from the original corpus were dropped, bringing the total number of questions to 1740. One hundred fourty six of the new questions come from the set of 300 questions that were considered in a study that investigated whether authoritative answers to consumer health questions can be found in Medline-Plus and other online resources [60]. In that study, question types and topics were marked. Fourty nine questions came from a set which we used for an earlier, experimental annotation of question type, focus, and frames (unpublished). We incorporated existing annotations in these sets as pre-annotations to be used in subsequent steps. Both these additional sets of questions were then deidentified and annotated with named entities, following the guidelines for CHQ-NER [12]. One expert annotator (SES) performed the annotation, she and another expert (DDF) then discussed and adjudicated these named entity annotations. At this stage, all 1740 questions had been fully annotated for named entities.
The rest of annotation of this part of the corpus proceeded as follows: first, all annotators annotated 20 questions for the elements under consideration (question topics, triggers/types, and frames). As the question type categories, we used those that were used in annotating GARD questions [10]. As this step was exploratory and included drug questions which we had not considered before, we allowed the annotators to propose additional question types. Six annotators were involved in this practice step. After a round of discussion and adjudication, we extended the question type categorization, defined three semantic roles (THEME, KEYWORD, and EXCLUDE_KEYWORD), and prepared annotation guidelines. We did not calculate inter-annotator agreement for these practice questions and only make available the final, adjudicated annotations.
Next, we sliced the rest of corpus (1720 questions) such that each annotator was allocated roughly the same number of questions, and each shared with all the other annotators approximately the same number of questions. Seven annotators were involved, although the additional annotator (Ann7) annotated significantly fewer than the others due to scheduling conflicts. The questions initially allocated to her but could not be annotated were redistributed among four annotators in such a way to ensure that each question was still double-annotated and the questions were fairly distributed. The resulting number of questions annotated by each annotator is given in Table 2. At this step, the annotators were allowed to modify the original named entity annotations if they deemed it necessary for topic/frame annotation. After this step was completed, we automatically checked the annotations for consistency, which involved addressing two common problems observed in annotation: first, we ensured that each question contained a question topic annotation and second, we checked that each frame contained a single THEME argument, except when the question type is one that allows multiple THEME arguments, such as COMPARISON. We calculated inter-annotator agreement for question triggers/types, frames with and without optional elements, and question topic. Inter-annotator agreement is calculated as the micro-average F 1 score when one set of annotations is considered the gold standard [61].
Considering the length of questions, the average number of annotations per question, and the number of annotators involved, adjudicating this set is a daunting task 4 . Therefore, instead of adjudication, we chose to investigate whether confidence scores can be automatically estimated for annotations. More details about this process are given at the end of this section.

CHQA-web
In contrast to CHQA-email, the second part of the corpus, CHQA-web, has been fully annotated from scratch in this study. MedlinePlus provides search capabilities that are geared towards traditional keyword searches; however, a significant number of searches are posed as natural language questions. Such searches often fail. To investigate whether we can address such questions automatically and to create a counterpoint to customer service information requests, we harvested a set of short questions from Med-linePlus search query logs. These questions were asked in the period from May 2015 to April 2016. We looked for wh-words to identify these questions in the query logs: how, what, when, where, which, who, why. We also included queries ending with question marks and those that start with an auxiliary (is it, can, could, do, does). For each category, we selected 70 most frequent questions as well as 70 random questions. After removing duplicates and near-duplicates using string matching heuristics, we were left with 1180 questions. In contrast to CHQAemail, we did not assess whether these were all legitimate questions or were answerable; instead, we used two labels, NOT_ANSWERABLE and NOT_QUESTION, during the annotation process to mark such questions. Recognizing that our heuristics for removing duplicates may not be adequate, we used another label, DUPLICATE, to indicate such cases.
In a practice step, the same six annotators annotated and adjudicated 100 of these questions using the guidelines for CHQA-email. During this process, the guidelines were updated: some question types were consolidated, a couple of new types were added, and the types of semantic roles were expanded.
Next, four of the annotators double-annotated the rest of the questions (n=1,080). Instead of pairing all annotators, we only paired those with clinical expertise (n=2) with those without (n=2), leading to four annotator combinations. In the end, each annotator labeled 540 questions. Since these questions were relatively short and there were fewer annotator pairs, we chose to adjudicate these annotations. Each pair adjudicated the questions that they new, latest information Qualitative Concept both annotated. As in CHQA-email, we calculated interannotator agreement using micro-average F 1 score. Fourty eight questions were adjudicated as NOT_QUESTION, 203 were deemed NOT_ANSWERABLE and 55 were found to be DUPLICATE, bringing the total number of annotated questions in this part down to 874.

Annotation
Seven annotators participated in various stages of the annotation, depending on their availability. All annotators had experience in working with biomedical text. Two annotators (LR, DDF) have clinical expertise, two are trained as medical librarians (SES, KM), and four have experience in natural language processing (HK, ABA, YM, and DDF). We used brat annotation tool [62] for all stages of annotation. Named entities and question triggers were annotated as terms, and frames were annotated as events in brat. Question topic was specified as an attribute of the relevant named entity.
In the rest of this subsection, we provide details about annotation of these semantic layers and highlight the differences in their annotation between the two parts of the corpus.
Named entity annotation Named entity annotation in CHQA-email essentially follows the principles of CHQ-NER [12].
In and OTHER). Nested entities and multi-part, noncontiguous annotations were allowed. Annotation was performed both manually and with assistance from several named entity recognizers [63][64][65].
For the present study, recognizing that many of the entities with OTHER type corresponded to physiological functions (e.g. pregnancy), we added a new named entity type ORGANISM_FUNCTION. We use the UMLS [66] definition for Organism Function semantic type for this new category: "A physiologic function of the organism as a whole, of multiple organ systems, or of multiple organs or tissues". To ensure that annotations are consistent, one of the annotators (DDF) examined all strings annotated as OTHER in CHQ-NER and identified those that can be considered organism functions. We then automatically relabeled these mentions, originally labeled OTHER, as ORGANISM_FUNCTION in CHQA-email.
For CHQA-web, we considered 18 categories: 16 categories above in addition to ORGANISM_FUNCTION and RESEARCH_CUE. The latter, though not common, is a category that was devised to indicate that the consumer is interested in information that goes beyond typical consumer health information.
Example RESEARCH_CUE mentions include new and latest information. In contrast to CHQA-email, we did not consider nested entities for CHQA-web, as the annotators generally found the annotation of nested entities challenging, although such entities can provide a more precise evaluation for named entity recognizers, as illustrated in [12].
The full list of named entity categories used in annotation, with brief definitions, examples, and relevant UMLS semantic types, is provided in Table 3.

Question topic annotation
Question topic is the central theme (a disease, a drug, etc.) of a question. Each legitimate question is expected to have at least one topic, but multiple topic elements can be annotated, especially in questions about relationships between two diseases or drugs. In the earlier example, ACTIVE LUNG TB is the question topic, while there are two topics in the following question in bold: (4) I'd like to learn more about megalocytic interstitial nephritis with malakoplakia.
All mentions of the topic term, including synonyms and misspellings, were expected to be annotated. For example, in the following question, synonyms anosmia and loss of smell are both legitimate topics.
This site tells me that ongoing research may solve my problem some day. Please, could I be informed if such result does happen? I live alone, aged 78, and this complete loss of smell has been my great problem for quite a while.
When considering nested entities for question topics, annotators were instructed to label the term as topic at the nesting level that is appropriate for the question. For example, in the following question, both mentions of ropinirole HCl as well as repinirole HCl 0.25 mg and ropinirole HCl 0.5 mg were annotated as named entities. With this instruction, annotators were expected to label mentions of ropinirole HCl as the topic, since the dosage/form of this drug is not relevant to the primary question. Please tell me if these meds are gluten-free.
These principles applied to question topic annotation in CHQA-email. For short questions in CHQA-web, based on the practice annotation, we chose not to explicitly annotate question topic, since this almost always corresponded to the THEME argument of the question frame.
Question type annotation We based our practice annotation of CHQA-email on the 14 question type categories proposed for disease questions in [10]. Discussion/adjudication of practice questions led to an expanded set of 33 categories. These 33 categories can be analyzed in three distinct groups.
• General question types: These categories can apply to any type of entity (COMPARISON, INFORMATION, and OTHER_QUESTION). • Problem question types: These categories apply to diseases/conditions, and other problems (e.g., CAUSE,   We provide some further details in Table 4. The last column indicates whether the question type was among the 14 categories proposed by Roberts et al. [10]. For question type annotation in CHQA-web, this scheme was modified in several ways, resulting in a somewhat simplified categorization that consists of 26 classes. First, some question types that do not occur frequently were merged into other categories that can be considered their supertypes. For example, in CHQA-email annotation, we split the question type SUSCEPTIBILITY We generalized the question type EFFECT to indicate underspecified relations between two or more named entities, and changed its name to ASSOCIATION for clarity. Finally, we added two new question types. DIAG-NOSE_ME identifies questions asking for diagnosis from a set of symptoms. For example, this question type was annotated for the following question (the trigger is in bold). While we specifically avoided annotating such questions in CHQA-email, since they were considered beyond the scope of a QA system provided by a medical library, we added such annotations in CHQA-web to allow further experiments. The other question type, TIME, identifies questions asking for time/duration of an event. An example is given below.

(8) who gets vaccines and when?
The annotators were instructed to annotate the minimal trigger expression, except when two equally valid triggers are conjoined. In the first question in Example 1, the question trigger is take, rather than can take or if I can take. In the second question, since both reduce and remove can trigger the same question type, the question trigger is taken as reduce or remove.

Question frame annotation
In CHQA-email, question frames were annotated using the 17 named entity categories and 33 question types discussed above. A question frame in this context minimally consists of two elements: a trigger indicating the question type and a THEME argument, which corresponds to a named entity. The type of the question frame is inherited from the trigger. Most questions types allow a single THEME argument, with the exceptions of EFFECT, which allows multiple THEME arguments, and COMPARISON, which requires multiple THEME arguments. Restrictions on the type of named entities that can fill the THEME role are specified in Table 5.
In CHQA-web, question frame annotation involved 26 question types and 18 named entity categories. We lifted the type restrictions for theme arguments for this part of the corpus, because the annotators found them too restrictive. However, the cardinality restrictions remained in place. For example, INTER-ACTION still required at least two THEME arguments. Two additional question types, DIAGNOSE_ME and TIME, were allowed multiple themes and a single theme, respectively.
Two parts of the corpus also differ with respect to non-THEME semantic roles. In CHQA-email, two generic semantic roles were defined: KEYWORD, and EXCLUDE_KEYWORD. KEYWORD arguments indicate named entities that are expected to be helpful in finding more specific answers to a question. EXCLUDE_KEYWORD arguments are those that should be used to exclude some potential answers. While KEYWORD can be viewed as a generic semantic role and can correspond to more typical semantic roles, such as AGENT, PATIENT, PURPOSE, EXCLUDE_KEYWORD is unique and is meaningful in a question answering context. Examples in Table 6 illustrate these roles.
In CHQA-web, we used fine-grained roles, more in line with typical roles used in semantic role labeling. On the other hand, EXCLUDE_KEYWORD was not considered. The fine-grained roles used are as follows: • AGENT indicates the entity that performs the action involving the theme. In the clinical domain, inanimate objects, such as drugs, procedures and devices could be agents. For example, in the first example in Table 6, both KEYWORD arguments can be annotated as AGENTs. • LOCATIVE indicates the location where the action involving the theme occurs (primarily, the body parts). leg in the question parkinson's disease best drugs for legs? can be annotated with this role in a TREATMENT frame. • PURPOSE indicates the reason/goal for which the action is performed. eczema in the question how many mg. of zinc daily for eczema patients? can be annotated with this role in a USAGE frame. • PATIENT indicates the experiencer of the action. baby in the question what if a mother quits intravenous A question frame can contain multiple arguments with such optional roles. We did not stipulate any semantic restrictions on the named entities that can fill these roles, even though for example, the PATIENT role is likely to be filled with an entity of PERSON_POPULATION type, and the RESEARCH role should be filled with a RESEARCH_CUE entity.
In CHQA-email, we associated a frame with the attribute RESEARCH to indicate that the person asking the question is interested in information that goes beyond consumer health information. In CHQA-web, as noted above, we specifically annotated terms indicating such information need as named entities (RESEARCH_CUE), in order to allow training of machine learning models that automatically learn such phrases.
When there are multiple mentions of a term that fills a specific role, the annotators were instructed to use the mention that is syntactically linked to the question type trigger, if possible. Otherwise, they select the closest mention that precedes the trigger. Similarly, if there are multiple potential triggers, annotators were expected to select the trigger that is syntactically related to the theme or is the trigger closest to it.
The same question is sometimes asked multiple times in the same information request, especially in long requests. Annotators were expected to annotate all instances of such questions. We also found that some questions can be represented from multiple perspectives. Consider the example below: This can be formalized in one of two ways: • A TREATMENT frame where cancer fills the THEME role and l-leucine is an AGENT (in essence, corresponding to the question does l-leucine treat cancer? ) • An INDICATION frame where l-leucine is the THEME and cancer is a PURPOSE (corresponding to is l-leucine indicated for cancer? ) Annotators were instructed to annotate a single frame in such cases and to take the question topic into consideration in making a decision. In this question, l-leucine is the more salient entity (thus, the question topic) and therefore, the frame in which this entity appears as the THEME element (i.e., the INDICATION frame) would be preferred. If both entities are equally likely as question topic, we asked the annotators to prefer specific question types over others (e.g., TREATMENT over INDICATION, COMPLICA-TION over EFFECT, CAUSE over DIAGNOSE_ME).

Annotation confidence estimation
It became clear early on that adjudicating annotations in CHQA-email would be significantly difficult, due to the number and length of questions, and the size of annotator pool (21 annotator pairs). This is in contrast to CHQA-web with the smaller number of questions, which are shorter on average, and annotators (4 pairs). Thus, we investigated whether we can estimate confidences for the annotations.
For this purpose, we used two annotation confidence modeling methods: MACE (Multi-Annotator Competence Estimation) [67] and another probabilistic approach proposed by Passonneau and Carpenter (which we refer to as P&C here) [68]. Implementations of both methods are freely available and have been successfully applied to crowd-sourced data. They share similarities: they are both generative models based on variations of the itemresponse model [69] and learn annotator competence and correct labels in an unsupervised fashion using expectation-maximization (EM). They mainly differ in priors and the EM flavor used, model paramaters, and goals. We refer the reader to respective papers for further details.
We considered annotations of named entities, topic, triggers, and frames for modeling. To simplify matters, we treated each annotation as an independent data instance for MACE and P&C. Once we modeled the annotation confidences using these methods, we used the confidence scores generated to create a new set of annotations. When Note that questions in the unadjudicated set are counted twice since this set is double-annotated incorporating these scores, we assumed that if two annotators agreed on a specific annotation, that annotation was correct. For the rest of the annotations on which two annotators did not agree, we followed the following procedure: • We calculate the mean average confidence score for each semantic class (i.e., entities, topic, triggers, frames). • If the confidence score associated with an individual item is higher than the mean average for its class, it is added to the new set of annotations as long as it is consistent with the annotations already added. Consistency is defined as follows: -If the item under consideration is named entity or question trigger, it is always considered consistent, as they are atomic and independent. -If it is a question topic, it is considered consistent as long as the underlying named entity has already been added.
-If it is a question frame, it is consistent as long as its trigger and all its arguments have already been added.
We compared these two methods to two other simple methods: the first (AGREE) only considers the subset of annotations that annotators agreed upon for the new set. The second (IAA_RANK) uses inter-annotator agreement results. For this method, we rank annotators by their inter-annotator agreement with others. Then, for a given question, we prefer the annotations of the annotator ranked higher among the two who annotated the question. Note that none of the methods as used here can distinguish cases where both annotators agree but are both incorrect.
To evaluate the performance of these methods, we ran experiments on CHQA-web, since an adjudicated set of annotations was available. Based on the results we obtained, we generated several sets of annotations for CHQA-email incorporating these methods. We make these sets publicly available in addition to raw annotations by each annotator.

Results and discussion
In this section, we present and discuss corpus statistics for each part of the corpus, provide an in-depth analysis of the semantic categories annotated, detail inter-annotator agreement and annotation confidence estimation results. Table 7 shows the basic corpus statistics for each part of the corpus. On average, CHQA-email requests are much longer (55.1 tokens vs. 7.5), confirming findings of earlier research [3,8]. Standard deviation is high, reflecting a large amount of variation in question length.

Named entity annotation
Statistics regarding named entities in the corpus are provided in Table 8. Results for CHQA-email are presented separately for the small practice set and the larger unadjudicated set, because the questions in the latter have been double-annotated and each question is counted twice. The comparison of frequency of the named entity categories is also illustrated in Fig. 2.
As expected, the distribution of named entity categories and the average number of annotations per question in CHQA-email are similar to those reported for CHQ-NER, which CHQA-email supersedes. For the short questions in CHQA-web, the average number of entities per question is much lower (1.6). DRUG_SUPPLEMENT and DIAG-NOSTIC_PROCEDURE terms appear more frequently in CHQA-web, while categories like PERSON_POPULATION, ORGANIZATION, PROFESSION, which often provide background information, appear less frequently than in CHQA-email. The distribution also confirms findings from earlier studies that consumer questions focus mostly on diseases and symptoms (i.e., PROBLEM category).
A small number of named entity annotations were added to CHQA-email (Unadjudicated) or deleted by annotators, increasing the number of annotations in this set from 33,494 to 33,747. PROBLEM category had the highest number of additions (141), and ORGANIZATION had the highest number of deletions (12), though additions/deletions often amounted to a very small percentage of the entities in a given category. Additions included problems like pain in period and drug classes like allopathic medicines.
RESEARCH_CUE category was only annotated in CHQAweb. In CHQA-email, RESEARCH attribute assigned to question frames indicates the same information without explicitly marking the mention. The total number of

Question trigger/type annotation
Statistics regarding question triggers/types in the corpus are provided in Table 9. TREATMENT triggers are most common in CHQA-email by a significant margin. In CHQA-web, INFORMATION triggers are most common, though its margin over the next most frequent type, CAUSE, is not as large. There are differences in the distribution of question types between the two parts: most significant are PERSON_ORGANIZATION and PROGNOSIS question types, which appear much more frequently in CHQA-email, and CAUSE, LOCATION, ACTION, and INDI-CATION question types, which occur more frequently in CHQA-web. Some of these differences can be explained by the question selection methodology. For example, in constructing CHQA-web, the same number of questions were sampled using different wh-words. where questions often contain LOCATION questions, which occurs in large quantity in CHQA-web, but is very rare in CHQA-email. The comparison of frequency of question trigger types in CHQA-email and CHQA-web is illustrated in Fig. 3. The distribution of question types in CHQA-email is similar to that presented for GARD dataset in Roberts et al. [10]. We analyzed the distribution of trigger expressions for each question type. A subset of results is provided in Table 10. Only top 10 triggers, if they occur more than once, are listed. We list the triggers without any lemmatization or spelling correction. Our analysis indicates that trigger lists for question types often have a long tail. In CHQA-email, TREATMENT, COST, and CAUSE question types have the least uniform distribution (i.e., relatively few triggers are used), while in CHQA-web, these question types are LOCATION and CAUSE. There is a slight preference for verbal triggers in CHQA-web as compared with CHQA-email. Light verbs, such as get, help, have, and do are the most heterogeneously annotated triggers, indicating 13, 11, 11, and 11 different question types, respectively, in CHQA-email. In CHQA-web, these triggers are have, why, when, and take, indicating a mix of light verbs and wh-words, to some extent reflecting the question selection process for this part of the corpus.
We asked annotators to indicate their proposed question types when annotating OTHER_QUESTION and DRUG_QUESTION categories in CHQA-email. After the annotations were completed, we manually categorized the small number of proposed types the annotators had specified. These types are provided in Table 11. Reference ranges for diagnostic tests, availability of drugs and other interventions as well as DIAGNOSE_ME (which we adopted later for CHQA-web) were among the most frequent. In general, the annotators were not consistent in proposing new question types; therefore, these results should be interpreted with caution. Also note that the  Numbers in the second and fourth columns are unique counts of triggers used for the corresponding category. Only triggers that occur at least twice are shown. The most frequent trigger for a given category is indicated with its frequency in parentheses (when this frequency is 2, all triggers given occur twice) annotation guidelines sometimes indicate a particular question type for a specific kind of information request annotated with one of these two types (e.g., INFORMA-TION for Manufacturer); however, these were missed by some annotators, indicating potential annotation errors.

Question frame annotation
Statistics regarding question frame types in the corpus are provided in Table 12. For CHQA-email (Practice), the results are the same as those for question triggers, indicating that each trigger in this set was associated with a single frame. In CHQA-email (Unadjudicated) and CHQA-web, a small proportion of triggers were used by several frames. The maximum number of frames indicated by a single trigger was 6. On average, a single SIDE_EFFECT trigger was annotated with highest number of frames, indicating that consumers were often interested in side effects of multiple drugs in the same question. The comparison of frequency of question frame categories in CHQA-email and CHQA-web is illustrated in Fig 4. Table 13 shows the counts of semantic roles in frames. The percentage of frame annotations involving a non-THEME role was approximately 68% in CHQAemail and 46% in CHQA-web. This difference is to be expected, as longer CHQA-email questions often provide more context that may be relevant for answering questions. The most frequently used semantic role in CHQA-web was the unspecific RESTRICTOR. Considering that this role is largely similar to KEYWORD and not very informative, fine-grained annotation of semantic roles (AGENT, PATIENT, etc.) may not provide much advantage over simply using coarse-grained KEYWORD when it comes to question understanding. EXCLUDE_KEYWORD role was rarely used in CHQAemail, suggesting that its utility for answering consumer health questions may be limited, and was later dropped for CHQA-web.

Question topic annotation
We only annotated question topic in CHQA-email, as this element almost always corresponds to the theme element of the question in the short CHQA-web questions. On average, 1.8 question topics were annotated per question. The maximum number of topic mentions annotated for a single question was 17.

Inter-annotator agreement
We calculated pairwise inter-annotator agreement, using micro-average F 1 score when one set of annotations is taken as the gold standard. Agreement was calculated for named entities, question triggers/types, question topics, and frames. Exact span match criterion was used to compare mentions (named entities and question triggers). We calculated frame agreement in several ways. For full frame agreement calculation, we considered the question trigger/type and all semantic roles (KEYWORD, AGENT, etc.). For core frame agreement, we considered only the THEME role in addition to question trigger/type. In addition, frame agreement calculation can be based on question trigger mentions or normalized question types indicated by these mentions. Combinations of these options, consequently, led to the following  . 4 The distribution of question frames in CHQA-email and CHQA-web parts of the corpus. The question frame categories in CHQA-web is used and some of the CHQA-email types are merged with their supertypes frame agreement types: Full frame w/ trigger, Full frame w/ type, Core frame w/ trigger, and Core frame w/ type. We consider Core frame w/ type as the most relevant for QA, as identifying specific words used to express a question precisely is generally not needed for accurately answering it. We did not consider agreement on practice sets for each part of the corpus. We also did not calculate named entity agreement on CHQA-email (Unadjudicated), since this largely overlaps with CHQ-NER and the inter-annotator agreement for named entities in that set were reported earlier [12]. The results of inter-annotator agreement calculation are provided in Table 14 and illustrated in Figs. 5 and 6.
Inter-annotator agreement in CHQA-email is overall low. The highest agreement is for question topics (0.71)   Inter-annotator agreement is calculated as the micro-average F1 score when one set of annotations is taken as the gold standard and the lowest is for full frame agreement (0.22), which considers named entity and trigger mentions, and all semantic roles (Full frame w/ trigger). Frame annotation in general is challenging as the results indicate, but the agreement is somewhat improved when the agreement only focuses on core frame elements, question type and theme (0.41 with Core frame w/ type). The low agreement for frames is not unexpected, as they can involve many parts, each potentially leading to disagreement. In particular, annotators had difficulty in agreeing on the exact boundaries of trigger mentions (0.37), one of the major components of the frames. When the matching criterion is changed to allow mention overlap (approximate match), average question trigger agreement increases  While these figures are still low, they suggest that more strict guidelines may be useful in annotating triggers. In contrast to named entities which are often expressed with specific part-of-speech elements, triggers are lexically and syntactically more heterogeneous and ambiguous, especially in consumer health questions; thus, developing such guidelines may be difficult. Analyzing inter-annotator agreement for triggers of each question type, we find that the agreement is generally higher for triggers of the frequent question types (Table 15). The question types ranked highest in terms of inter-annotator agreement for triggers are CAUSE (0.60), TREATMENT  Compared to question triggers, the agreement on question types is much higher, indicating that annotators can identify the information request in abstract, but have difficulty in annotating the relevant piece of text. We also note that agreement for question types is lower than that reported for GARD (0.58 and 0.74 in this study vs. 0.81) [10]. This is partly due to the fact that the number of question types is significantly higher in our case and also to the fact that GARD questions are in general much shorter and well-formed. On the whole, these results point out that it would be more practical to focus on annotating question types only, instead of attempting to also precisely identify the question triggers. However, we should also note that machine learning approaches are likely to benefit from mention-level trigger annotations.
Agreement in CHQA-web is consistently higher than that in CHQA-email by about 0.2 points. Trends are similar for different semantic classes, confirming our basic intuition that annotating these shorter questions is easier, compared to CHQA-email. Agreement for named entities (0.72) is consistent with that reported for CHQ-NER (0.71) [12], although given the trends, we would have expected it to be higher. This may be attributed to the fact named entity annotation for this part of the corpus was not performed as a separate study, and annotators may not have been as diligent in following the strict guidelines for named entity annotation. In this part of the corpus, interannotator agreement for question triggers is generally higher than that in CHQA-email, and is more evenly distributed between frequent and rare question types: among the top four question types are COST, DIAGNOSE_ME, and COMPARISON, which are relatively rare, as well as TREAT-MENT, the third most frequent trigger type in this subset (Table 15). Agreement on question types follows a similar pattern and, for frequent question types, reaches as high as 0.86 (TREATMENT).

Annotation confidence estimation
The evaluation results of annotation confidence estimation for CHQA-web are provided in Table 16. These results have been obtained by comparing the estimation method results with the adjudicated results. We ran the same estimation methods on CHQA-email (Unadjudicated) and provide the resulting data; however, we were unable to evaluate those results, as no gold standard exists for comparison. The results show that the baseline (AGREE: taking simple agreement of annotators as indication of confidence) already achieves an agreement of 0.79 with the adjudicated results. The precision of 0.99 indicates that a small number of annotations that the annotators agreed upon were discarded later in adjudication. All other methods improve on the low recall of the baseline (0.66), while leading to some precision loss. IAA_RANK uses the ranking of annotators by their average agreement score, and can be based on any of the agreement categories described earlier. Because frame annotations incorporate other atomic annotations (named entities, triggers), we ran IAA_RANK with two frame agreement results. The first ranks the annotators by their average agreement for full frame with triggers, and the second by their agreement for core frame with types. The latter led to best confidence estimation for CHQA-web (0.844).
The results obtained with MACE [67] and P&C [68] methods were similar to each other, although MACE had a slight edge and provided confidence estimation that is very close to the best results (0.841 vs. 0.844). Using the annotator reliability ranking provided by MACE and P&C, we obtained results that are slightly lower than their confidence score-based counterparts (0.83 vs. 0.84 and 0.82 vs. 0.83). Overall, the results obtained by these two methods are lower than the results reported for other corpora [67,68]. We attribute this partly to the fact that crowdsourcing is used in those cases; with a larger pool of annotators who annotate all instances, it is likely that these methods would perform better. Furthermore, compared to the type of annotations they consider (e.g., word senses, textual entailment pairs), our annotations (in particular, frames) are significantly more complex. In fact, when we only focused on the named entities in CHQ-NER (1548 questions) (i.e., less complex annotations), we obtained the best results with MACE. As noted earlier, as input to these two methods, we made the simplifying assumption that each annotation is independent. A representation that better reflects the compositional nature of frames may also lead to better results.
We ran the same methods on CHQA-email (Unadjudicated) and provide this part of the corpus in several versions: AGREE, IAA_RANK (both Full frame w/trigger and Core frame w/ type), MACE, and P&C subsets. In addition, we provide MACE and P&C confidence estimations for each annotation. This can enable further research in developing more advanced methods that estimate annotation confidence. For example, the confidence scores generated by MACE and P&C and the output of other simpler methods can be used as features to learn to predict confidence. Confidence scores can also be used in an active learning framework, where the annotator is presented only with the annotations with moderate confidence, reducing adjudication load. Distinguishing high from low confidence items can also benefit training and evaluation: for example, a system can be penalized less for an incorrect response, when the annotation is labeled with low confidence [68].

Conclusion
We presented a corpus of consumer health questions annotated for various semantic classes: named entities, question triggers/types, question topics, and frames. The questions cover a wide spectrum, from short questions with little to no context to long questions filled with irrelevant information, reflecting all the complexity and difficulty of consumer health questions. With respect to question understanding and answering, they present different challenges and different approaches may be needed. The corpus is the first focusing on annotation of real consumer questions, and with its size, the number and types of annotations included, it represents a rich resource for consumer health QA. The corpus, annotation guidelines and various analyses are publicly available at https://bionlp. nlm.nih.gov/CHIQAcollections/CHQA-Corpus-1.0.zip.
Our corpus has some limitations. Our conceptualization of question understanding and our goals have evolved over time and the corpus reflects this to some extent. Two parts of the corpus are largely similar, but there are also differences. In most cases, these differences can be reconciled with some automatic processing, as in question types that are merged in the other part of the corpus or non-THEME semantic roles. Some aspects are more difficult to reconcile, however. For example, we ignored diagnosis-seeking questions (DIAGNOSE_ME) for a long time, since they were considered outside the scope of our work. More recently, we began considering these questions but annotated them only in CHQA-web. A large portion of the corpus has not been adjudicated due to personnel/time constraints, which may limit its usefulness. To address this, we experimented with several methods of annotation confidence estimation and make these estimates available, which could stimulate research in better understanding annotator behavior and automatic adjudication.
In ongoing and future work, we plan to expand this corpus on several other semantic dimensions. For example, we are currently normalizing named entities in our corpus by mapping them to the corresponding UMLS Metathesaurus concepts. By addressing the lexical variability in named entities, this can improve question understanding. In earlier work, we found that coreference played a major role in consumer health question understanding [9], and we plan to add coreference annotations to a subset of CHQA-email. Similarly, we annotated spelling errors on a smaller set of long questions in an earlier study [58], and we also plan to add such annotations to this corpus. Endnotes 1 http://www.mdedge.com/jfponline/clinical-inquiries 2 https://rarediseases.info.nih.gov/ 3 In previous work, we used the terms focus [11] and topic [57] to refer to this element. In this paper, we use the term topic. 4 Note, however, that named entities are reconciled since they have been mostly inherited from CHQ-NER [12].