Deep learning has yielded state-of-the-art performance on many natural language processing tasks including named entity recognition (NER). However, this typically requires large amounts of labeled data. In this work, we demonstrate that the amount of labeled training data can be drastically reduced when deep learning is combined with active learning. While active learning is sample-efficient, it can be computationally expensive since it requires iterative retraining. To speed this up, we introduce a lightweight architecture for NER, viz., the CNN-CNN-LSTM model consisting of convolutional character and word encoders and a  long short term memory (LSTM) tag decoder. The model achieves nearly state-of-the-art performance on standard datasets for the task while being computationally much more efficient than best performing models. We carry out incremental active learning, during the training process, and are able to nearly match state-of-the-art performance with just 25\% of the original training data. Over the past few years, papers applying deep neural networks (DNNs) to the task of named entity recognition (NER) have successively advanced the state-of-the-art BID7 BID17 BID24 BID6 BID48 . However, under typical training procedures, the advantages of deep learning diminish when working with small datasets. For instance, on the OntoNotes-5.0 English dataset, whose training set contains 1,088,503 words, a DNN model outperforms the best shallow model by 2.24% as measured by F1 score BID6 . However, on the comparatively small CoNLL-2003 English dataset, whose training set contains 203,621 words, the best DNN model enjoys only a 0.4% advantage. To make deep learning more broadly useful, it is crucial to reduce its training data requirements.Generally, the annotation budget for labeling is far less than the total number of available (unlabeled) samples. For NER, getting unlabeled data is practically free, owing to the large amount of content that can be efficiently scraped off the web. On the other hand, it is especially expensive to obtain annotated data for NER since it requires multi-stage pipelines with sufficiently well-trained annotators BID19 BID5 . In such cases, active learning offers a promising approach to efficiently select the set of samples for labeling. Unlike the supervised learning setting, in which examples are drawn and labeled at random, in the active learning setting, the algorithm can choose which examples to label.Active learning aims to select a more informative set of examples in contrast to supervised learning, which is trained on a set of randomly drawn examples. A central challenge in active learning is to determine what constitutes more informative and how the active learner can recognize this based on what it already knows. The most common approach is uncertainty sampling, in which the model preferentially selects examples for which it's current prediction is least confident. Other approaches include representativeness-based sampling where the model selects a diverse set that represent the input space without adding too much redundancy.In this work, we investigate practical active learning algorithms on lightweight deep neural network architectures for the NER task. Training with active learning proceeds in multiple rounds. Traditional active learning schemes are expensive for deep learning since they require complete retraining of the classifier with newly annotated samples after each round. In our experiments, for example, the model must be retrained 54 times. Because retraining from scratch is not practical, we instead carry out incremental training with each batch of new labels: we mix newly annotated samples with the older ones, and update our neural network weights for a small number of epochs, before querying for labels in a new round. This modification drastically reduces the computational requirements of active learning methods and makes it practical to deploy them.We further reduce the computational complexity by selecting a lightweight architecture for NER. We propose a new CNN-CNN-LSTM architecture for NER consisting of a convolutional character-level encoder, convolutional word-level encoder, and long short term memory (LSTM) tag decoder. This model handles out-of-vocabulary words gracefully and, owing to the greater reliance on convolutions (vs recurrent layers), trains much faster than other deep models while performing competitively.We introduce a simple uncertainty-based heuristic for active learning with sequence tagging. Our model selects those sentences for which the length-normalized log probability of the current prediction is the lowest. Our experiments with the Onto-Notes 5.0 English and Chinese datasets demonstrate results comparable to the Bayesian active learning by disagreement method . Moreover our heuristic is faster to compute since it does not require multiple forward passes. On the OntoNotes-5.0 English dataset, our approach matches 99% of the F1 score achieved by the best deep models trained in a standard, supervised fashion despite using only a 24.9% of the data. On the OntoNotes-5.0 Chinese dataset, we match 99% performance with only 30.1% of the data. Thus, we are able to achieve state of art performance with drastically lower number of samples.
We introduce a lightweight architecture for named entity recognition and carry out incremental active learning, which is able to match state-of-the-art performance with just 25% of the original training data.
{"src": "scitldr-AIC_train", "paper_id": "ry018WZAZ"}