Q: what standard dataset were used?
Text: Introduction
Information extraction tasks have become very important not only in the Web, but also for in-house enterprise settings. One of the crucial steps towards understanding natural language is named entity recognition (NER), which aims to extract mentions of entity names in text. NER is necessary for many higher-level tasks such as entity linking, relation extraction, building knowledge graphs, question answering and intent based search. In these scenarios, NER recall is critical, as candidates that are never generated can not be recovered later BIBREF0 .
Robust Contextual Word Labeling
We abstract the task of NER as sequential word labeling problem. Figure FIGREF15 illustrates an example for sequential transformation of a sentence into word labels. We express each sentence in a document as a sequence of words: INLINEFORM0 , e.g. INLINEFORM1 Aspirin. We define a mention as the longest possible span of adjacent tokens that refer to a an entity or relevant concept of a real-world object, such as Aspirin (ASA). We further assume that mentions are non-recursive and non-overlapping. To encode boundaries of the mention span, we adapt the idea of ramshaw1995text, which has been adapted as BIO2 standard in the CoNLL2003 shared task BIBREF15 . We assign labels INLINEFORM2 to each token to mark begin, inside and outside of a mention from left to right. We use the input sequence INLINEFORM3 together with a target sequence INLINEFORM4 of the same length that contains a BIO2 label for each word: INLINEFORM5 , e.g. INLINEFORM6 B. To predict the most likely label INLINEFORM7 of a token regarding its context, we utilize recurrent neural networks.
Robust Word Encoding Methods
We have shown that most common errors for recall loss are misspellings, POS errors, capitalization, unseen words and irregular context. Therefore we generalize our model throughout three layers: robust word encoding, in-sentence word context and contextual sequence labeling.
Dictionary-based word vectorization methods suffer from sparse training sets, especially in the case of non-verbatim mentions, rare words, typing and capitalization errors. For example, the word2vec model of mikolov2013efficient generalizes insufficiently for rare words in idiosyncratic domains or for misspelled words, since for these words no vector representation is learned at training time. In the GENIA data set, we notice 27% unseen words (dictionary misses) in the pretrained word2vec model. As training data generation is expensive, we investigate a generic approach for the generation of word vectors. We use letter-trigram word hashing as introduced by huang2013learning. This technique goes beyond words and generates word vectors as a composite of discriminative three-letter “syllables”, that might also include misspellings. Therefore, it is robust against dictionary misses and has the advantage (despite its name) to group syntactically similar words in similar vector spaces. We compare this approach to word embedding models such as word2vec.
The most important features for NER are word shape properties, such as length, initial capitalization, all-word uppercase, in-word capitalization and use of numbers or punctuation BIBREF16 . Mixed-case word encodings implicitly include capitalization features. However, this approach impedes generalization, as words appear in various surface forms, e.g. capitalized at the beginning of sentences, uppercase in headlines, lowercase in social media text. The strong coherence between uppercase and lowercase characters – they might have identical semantics – is not encoded in the embedding. Therefore, we encode the words using lowercase letter-trigrams. To keep the surface information, we add flag bits to the vector that indicate initial capitalization, uppercase, lower case or mixed case.
Deep Contextual Sequence Learning
With sparse training data in the idiosyncratic domain, we expect input data with high variance. Therefore, we require a strong generalization for the syntactic and semantic representation of language. To reach into the high 80–90% NER F1 performance, long-range context-sensitive information is indispensable. We apply the computational model of recurrent neural networks, in particular long short-term memory networks (LSTMs) BIBREF17 , BIBREF18 to the problem of sequence labeling. Like neural feed-forward networks, LSTMs are able to learn complex parameters using gradient descent, but include additional recurrent connections between cells to influence weight updates over adjacent time steps. With their ability to memorize and forget over time, LSTMs have proven to generalize context-sensitive sequential data well BIBREF19 , BIBREF20 .
Figure FIGREF15 shows an unfolded representation of the steps through a sentence. We feed the LSTM with letter-trigram vectors INLINEFORM0 as input data, one word at a time. The hidden layer of the LSTM represents context from long range dependencies over the entire sentence from left to right. However, to achieve deeper contextual understanding over the boundaries of multi-word annotations and at the beginning of sentences, we require a backwards pass through the sentence. We therefore implement a bidirectional LSTM and feed the output of both directions into a second LSTM layer for combined label prediction.
For the use in the neural network, word encodings INLINEFORM0 and labels INLINEFORM1 are real-valued vectors. To predict the most likely label INLINEFORM2 of a token, we utilize a LSTM with input nodes INLINEFORM3 , input gates INLINEFORM4 , forget gate INLINEFORM5 , output gate INLINEFORM6 and internal state INLINEFORM7 . For the bidirectional case, all gates are duplicated and combined into forward state INLINEFORM8 and backward state INLINEFORM9 . The network is trained using backpropagation through time (BPTT) by adapting weights INLINEFORM10 and bias parameters INLINEFORM11 to fit the training examples. DISPLAYFORM0
We iterate over labeled sentences in mini-batches and update the weights accordingly. The network is then used to predict label probabilities INLINEFORM0 for unseen word sequences INLINEFORM1 .
Implementation of NER Components 
To show the impact of our bidirectional LSTM model, we measure annotation performance on three different neural network configurations. We implement all components using the Deeplearning4j framework. For preprocessing (sentence and word tokenization), we use Stanford CoreNLP BIBREF11 . We test the sequence labeler using three input encodings:
[noitemsep]
DICT: We build a dictionary over all words in the corpus and generate the input vector using 1-hot encoding for each word
EMB: We use the GoogleNews word2vec embeddings, which encodes each word as vector of size 300
TRI: we implement letter-trigram word hashing as described in Section SECREF14 .
During training and test, we group all tokens of a sentence as mini-batch. We evaluate three different neural network types to show the impact of the bidirectional sequence learner.
[noitemsep]
FF: As baseline, we train a non-sequential feed-forward model based on a fully connected multilayer perceptron network with 3 hidden layers of size 150 with relu activation, feeding into a 3-class softmax classifier. We train the model using backpropagation with stochastic gradient descent and a learning rate of 0.005.
LSTM: We use a configuration of a single feed-forward layer of size 150 with two additional layers of single-direction LSTM with 20 cells and a 3-class softmax classifier. We train the model using backpropagation-through-time (BPTT) with stochastic gradient descent and a learning rate of 0.005.
BLSTM: Our final configuration consists of a single feed-forward layer of size 150 with one bidirectional LSTM layer with 20 cells and an additional single-direction LSTM with 20 cells into a 3-class softmax classifier. The BLSTM model is trained the same way as the single-direction LSTM.
Evaluation
We evaluate nine configurations of our model on five gold standard evaluation data sets. We show that the combination of letter-trigram word hashing with bidirectional LSTM yields the best results and outperforms sequence learners based on dictionaries or word2vec. To highlight the generalization of our model to idiosyncratic domains, we run tests on common-typed data sets as well as on specialized medical documents. We compare our system on these data sets with specialized state-of-the-art systems.
Evaluation Set Up
We train two models with identical parameterization, each with 2000 randomly chosen labeled sentences from a standard data set. To show the effectiveness of the components, we evaluate different configurations of this setting with 2000 random sentences from the remaining set. The model was trained using Deeplearning4j with nd4j-x86 backend. Training the TRI+BLSTM configuration on a commodity Intel i7 notebook with 4 cores at 2.8GHz takes approximately 50 minutes.
Table TABREF33 gives an overview of the standard data sets we use for training. The GENIA Corpus BIBREF3 contains biomedical abstracts from the PubMed database. We use GENIA technical term annotations 3.02, which cover linguistic expressions to entities of interest in molecular biology, e.g. proteins, genes and cells. CoNLL2003 BIBREF14 is a standard NER dataset based on the Reuters RCV-1 news corpus. It covers named entities of type person, location, organization and misc.
For testing the overall annotation performance, we utilize CoNLL2003-testA and a 50 document split from GENIA. Additionally, we test on the complete KORE50 BIBREF21 , ACE2004 BIBREF22 and MSNBC data sets using the GERBIL evaluation framework BIBREF23 .
Measurements
We measure precision, recall and F1 score of our DATEXIS-NER system and state-of-the-art annotators introduced in Section SECREF2 . For the comparison with black box systems, we evaluate annotation results using weak annotation match. For a more detailed in-system error analysis, we measure BIO2 labeling performance based on each token.
We measure the overall performance of mention annotation using the evaluation measures defined by cornolti2013framework, which are also used by ling2015design. Let INLINEFORM0 be a set of documents with gold standard mention annotations INLINEFORM1 with a total of INLINEFORM2 examples. Each mention INLINEFORM3 is defined by start position INLINEFORM4 and end position INLINEFORM5 in the source document INLINEFORM6 . To quantify the performance of the system, we compare INLINEFORM7 to the set of predicted annotations INLINEFORM8 with mentions INLINEFORM9 : DISPLAYFORM0
We compare using a weak annotation match: DISPLAYFORM0
We measure micro-averaged precision ( INLINEFORM0 ), recall ( INLINEFORM1 ) and NER-style ( INLINEFORM2 ) score: DISPLAYFORM0
Tuning the model configuration with annotation match measurement is not always feasible. We therefore measure INLINEFORM0 , INLINEFORM1 , INLINEFORM2 , INLINEFORM3 separately for each label class INLINEFORM4 in our classification model and calculate binary classification precision INLINEFORM5 , recall INLINEFORM6 and INLINEFORM7 scores. To avoid skewed results from the expectedly large INLINEFORM8 class, we use macro-averaging over the three classes: DISPLAYFORM0
Evaluation Results
We now discuss the evaluation of our DATEXIS-NER system on common and idiosyncratic data.
Table TABREF35 shows the comparison of DATEXIS-NER with eight state-of-the-art annotators on four common news data sets. Both common and medical models are configured identically and trained on only 2000 labeled sentences, without any external prior knowledge. We observe that DATEXIS-NER achieves the highest recall scores of all tested annotators, with 95%–98% on all measured data sets. Moreover, DATEXIS-NER precision scores are equal or better than median. Overall, we achieve high micro-F1 scores of 84%–94% on news entity recognition, which is slightly better than the ontology-based Entityclassifier.eu NER and reveals a better generalization than the 3-type Stanford NER with distributional semantics. We notice that systems specialized on word-sense disambiguation (Babelfy, DBpedia Spotlight) don't perform well on “raw” untyped entity recognition tasks. The highest precision scores are reached by Stanford NER. We also notice a low precision of all annotators on the ACE2004 dataset and high variance in MSNBC performance, which are probably caused by differing annotation standards.
Table TABREF45 shows the results of biomedical entity recognition compared to the participants of the JNLPBA 2004 bio-entity recognition task BIBREF14 . We notice that for these well-written Medline abstracts, there is not such a strong skew between precision and recall. Our DATEXIS-NER system outperforms the HMM, MEMM, CRF and CDN based models with a micro-F1 score of 84%. However, the highly specialized GENIA chunker for LingPipe achieves higher scores. This chunker is a very simple generative model predictor that is based on a sliding window of two tokens, word shape and dictionaries. We interpret this score as strong overfitting using a dictionary of the well-defined GENIA terms. Therefore, this model will generalize hardly considering the simple model. We can confirm this presumption in the common data sets, where the MUC-6 trained HMM LingPipe chunker performs on average on unseen data.
We evaluate different configurations of the components that we describe in Section SECREF22 . Table TABREF47 shows the results of experiments on both CoNLL2003 and GENIA data sets. We report the highest macro-F1 scores for BIO2 labeling for the configuration of letter-trigram word vectors and bidirectional LSTM. We notice that dictionary-based word encodings (DICT) work well for idiosyncratic medical domains, whereas they suffer from high word ambiguity in the news texts. Pretrained word2vec embeddings (EMB) perform well on news data, but cannot adapt to the medical domain without retraining, because of a large number of unseen words. Therefore, word2vec generally achieves a high precision on news texts, but low recall on medical text. The letter-trigram approach (TRI) combines both word vector generalization and robustness towards idiosyncratic language.
We observe that the contextual LSTM model achieves scores throughout in the 85%–94% range and significantly outperforms the feed-forward (FF) baseline that shows a maximum of 75%. Bidirectional LSTMs can further improve label classification in both precision and recall.
Discussion and Error Analysis
We investigate different aspects of the DATEXIS-NER components by manual inspection of classification errors in the context of the document. For the error classes described in the introduction (false negative detections, false positives and invalid boundaries), we observe following causes:
In dictionary based configurations (e.g. 1-hot word vector encoding DICT), we observe false negative predictions caused by dictionary misses for words that do not exist in the training data. The cause can be rare unseen or novel words (e.g. T-prolymphocytic cells) or misspellings (e.g. strengthnend). These words yield a null vector result from the encoder and can therefore not be distinguished by the LSTM. The error increases when using word2vec, because these models are trained with stop words filtered out. This implicates that e.g. mentions surrounded by or containing a determiner (e.g. The Sunday Telegraph quoted Majorie Orr) are highly error prone towards the detection of their boundaries. We resolve this error by the letter-trigram approach. Unseen trigrams (e.g. thh) may still be missing in the word vector, but only affect single dimensions as opposed to the vector as a whole.
Surface forms encode important features for NER (e.g. capitalization of “new” in Alan Shearer was named as the new England captain / as New York beat the Angels). However, case-sensitive word vectorization methods yield a large amount of false positive predictions caused by incorrect capitalization in the input data. An uppercase headline (e.g. TENNIS - U.S. TEAM ON THE ROAD FOR 1997 FED CUP) is encoded completely different than a lowercase one (e.g. U.S. team on the road for Fed Cup). Because of that, we achieve best results with lowercase word vectors and additional surface form feature flags, as described in Section SECREF14 .
We observe mentions that are composed of co-occurring words with high ambiguity (e.g. degradation of IkB alpha in T cell lines). These groups encode strong syntagmatic word relations BIBREF24 that can be leveraged to resolve word sense and homonyms from sentence context. Therefore, correct boundaries in these groups can effectively be identified only with contextual models such as LSTMs.
Orthogonal to the previous problem, different words in a paradigmatic relation BIBREF24 can occur in the same context (e.g. cyclosporin A-treated cells / HU treated cells). These groups are efficiently represented in word2vec. However, letter-trigram vectors cannot encode paradigmatic groups and therefore require a larger training sample to capture these relations.
Often, synonyms can only be resolved regarding a larger document context than the local sentence context known by the LSTM. In these cases, word sense is redefined by a topic model local to the paragraph (e.g. sports: Tiger was lost in the woods after divorce.). This problem does not heavily affect NER recall, but is crucial for named entity disambiguation and coreference resolution.
The proposed DATEXIS-NER model is restricted to recognize boundaries of generic mentions in text. We evaluate the model on annotations of isolated types (e.g. persons, organizations, locations) for comparison purposes only, but we do not approach NER-style typing. Contrary, we approach to detect mentions without type information. The detection of specific types can be realized by training multiple independent models on a selection of labels per type and nesting the resulting annotations using a longest-span semantic type heuristic BIBREF25 .
Summary
ling2015design show that the task of NER is not clearly defined and rather depends on a specific problem context. Contrary, most NER approaches are specifically trained on fixed datasets in a batch mode. Worse, they often suffer from poor recall BIBREF26 . Ideally, one could personalize the task of recognizing named entities, concepts or phrases according to the specific problem. “Personalizing” and adapting such annotators should happen with very limited human labeling effort, in particular for idiosyncratic domains with sparse training data.
Our work follows this line. From our results we report F1 scores between 84–94% when using bidirectional multi-layered LSTMs, letter-trigram word hashing and surface form features on only few hundred training examples.
This work is only a preliminary step towards the vision of personalizing annotation guidelines for NER BIBREF2 . In our future work, we will focus on additional important idiosyncratic domains, such as health, life science, fashion, engineering or automotive. For these domains, we will consider the process of detecting mentions and linking them to an ontology as a joint task and we will investigate simple and interactive workflows for creating robust personalized named entity linking systems.
The GENIA Corpus , CoNLL2003