Fine-grained Entity Recognition (FgER) is the task of detecting and classifying entity mentions to a large set of types spanning diverse domains such as biomedical, finance and sports.    We  observe  that  when  the  type  set  spans  several  domains,  detection  of  entity mention becomes a limitation for supervised learning models.   The primary reason being lack  of  dataset  where  entity  boundaries  are  properly  annotated  while  covering  a  large spectrum of entity types.   Our work directly addresses this issue.   We propose Heuristics Allied with Distant Supervision (HAnDS) framework to automatically construct a quality dataset suitable for the FgER task.   HAnDS framework exploits the high interlink among Wikipedia  and  Freebase  in  a  pipelined  manner,  reducing  annotation  errors  introduced by naively using distant supervision approach.   Using HAnDS framework,  we create two datasets, one suitable for building FgER systems recognizing up to 118 entity types based on the FIGER type hierarchy and another for up to 1115 entity types based on the TypeNet hierarchy.   Our extensive empirical experimentation warrants the quality of the generated datasets.   Along with this, we also provide a manually annotated dataset for benchmarking FgER systems. In the literature, the problem of recognizing a handful of coarse-grained types such as person, location and organization has been extensively studied BID18 Sekine, 2007, Marrero et al., 2013] . We term this as Coarse-grained Entity Recognition (CgER) task. For CgER, there exist several datasets, including manually annotated datasets such as CoNLL BID28 ] and automatically generated datasets such as WP2 BID21 . Manually constructing a dataset for FgER task is an expensive and time-consuming process as an entity mention could be assigned multiple types from a set of thousands of types.In recent years, one of the subproblems of FgER, the Fine Entity Categorization or Typing (Fine-ET) problem has received lots of attention particularly in expanding its type coverage from a handful of coarse-grained types to thousands of fine-grained types BID17 BID6 . The primary driver for this rapid expansion is exploitation of cheap but fairly accurate annotations from Wikipedia and Freebase BID4 via the distant supervision process BID7 . The Fine-ET problem assumes that the entity boundaries are provided by an oracle.We observe that the detection of entity mentions at the granularity of Fine-ET is a bottleneck. The existing FgER systems, such as FIGER BID12 , follow a two-step approach in which the first step is to detect entity mentions and the second step is to categorize detected entity mentions. For the entity detection, it is assumed that all the fine-categories are subtypes of the following four categories: person, location, organization and miscellaneous. Thus, a model trained on the CoNLL dataset BID28 ] which is annotated with these types can be used for entity detection. Our analysis indicates that in the context of FgER, this assumption is not a valid assumption. As a face value, the miscellaneous type should ideally cover all entity types other than person, location, and organization. However, it only covers 68% of the remaining types of the FIGER hierarchy and 42% of the TypeNet hierarchy. Thus, the models trained using CoNLL data are highly likely to miss a significant portion of entity mentions relevant to automatic knowledge bases construction applications.Our work bridges this gap between entity detection and Fine-ET. We propose to automatically construct a quality dataset suitable for the FgER, i.e, both Fine-ED and Fine-ET using the proposed HAnDS framework. HAnDS is a three-stage pipelined framework wherein each stage different heuristics are used to combat the errors introduced via naively using distant supervision paradigm, including but not limited to the presence of large false negatives. The heuristics are data-driven and use information provided by hyperlinks, alternate names of entities, and orthographic and morphological features of words.Using the HAnDS framework and the two popular type hierarchies available for Fine-ET, the FIGER type hierarchy BID12 and TypeNet BID17 , we automatically generated two corpora suitable for the FgER task. The first corpus contains around 38 million entity mentions annotated with 118 entity types. The second corpus contains around 46 million entity mentions annotated with 1115 entity types. Our extensive intrinsic and extrinsic evaluation of the generated datasets warrants its quality. As compared with existing automatically generated datasets, supervised learning models trained on our induced training datasets perform significantly better (approx 20 point improvement on micro-F1 score). Along with the automatically generated dataset, we provide a manually annotated corpora of around thousand sentences annotated with 117 entity types for benchmarking of FgER models. Our contributions are highlighted as follows:• We analyzed that existing practice of using models trained on CoNLL dataset has poor recall for entity detection in the Fine-ET setting, where the type set spans several diverse domains. (Section 3)• We propose HAnDS framework, a heuristics allied with the distant supervision approach to automatically construct datasets suitable for FgER problem, i.e., both fine entity detection and fine entity typing. (Section 4)• We establish the state-of-the-art baselines on our new manually annotated corpus, which covers 2.7 times more finer-entity types than the FIGER gold corpus, the current de facto FgER evaluation corpus. (Section 5)The rest of the paper is organized as follows. We describe the related work in Section 2, followed by a case study on entity detection problem in the Fine-ET setting, in Section 3. Section 4 describes our proposed HAnDS framework, followed by empirical evaluation of the datasets in Section 5. In Section 6 we conclude our work. In this work, we initiate a push towards moving from CgER systems to FgER systems, i.e., from recognizing entities from a handful of types to thousands of types. We propose the HAnDS framework to automatically construct quality training dataset for different variants of FgER tasks. The two datasets constructed in our work along with the evaluation resource are currently the largest available training and testing dataset for the entity recognition problem. They are backed with empirical experimentation to warrants the quality of the constructed corpora.The datasets generated in our work opens up two new research directions related to the entity recognition problem. The first direction is towards an exploration of sequence labeling approaches in the setting of FgER, where each entity mention can have more than one type. The existing state-of-the-art sequence labeling models for the CgER task, can not be directly applied in the FgER setting due to state space explosion in the multi-label setting. The second direction is towards noise robust sequence labeling models, where some of the entity boundaries are incorrect. For example, in our induced datasets, there are still entity detection errors, which are inevitable in any heuristic approach. There has been some work explored in BID8 assuming that it is a priori known which tokens have noise. This information is not available in our generated datasets.Additionally, the generated datasets are much richer in entity types compared to any existing entity recognition datasets. For example, the generated dataset contains entities from several domains such as biomedical, finance, sports, products and entertainment. In several downstream applications where NER is used on a text writing style different from Wikipedia, the generated dataset is a good candidate as a source dataset for transfer learning to improve domain-specific performance.
We initiate a push towards building ER systems to recognize thousands of types by providing a method to automatically construct suitable datasets based on the type hierarchy. 
{"src": "scitldr-AIC_train", "paper_id": "HylHE-9p6m"}