task_categories:token-classification task_ids:named-entity-recognition annotations_creators:machine-generated language_creators:machine-generated multilinguality:multilingual source_datasets:original language:de language:en language:es language:fr language:it language:nl language:pl language:pt language:ru language:zh license:cc-by-nc-sa-4.0 structure-prediction region:us
---
annotations_creators:
- machine-generated
language_creators:
- machine-generated
language:
- de
- en
- es
- fr
- it
- nl
- pl
- pt
- ru
- zh
license:
- cc-by-nc-sa-4.0
multilinguality:
- multilingual
source_datasets:
- original
task_categories:
- token-classification
task_ids:
- named-entity-recognition
pretty_name: multinerd-dataset
tags:
- structure-prediction
---

## Table of Contents
- [Description](#description)
- [Dataset Structure](#dataset-structure)
- [Additional Information](#additional-information)
  
## Dataset Card for MultiNERD dataset

## Dataset Description

- **Summary:** Training data for fine-grained NER in 10 languages.
- **Repository:** [https://github.com/Babelscape/multinerd](https://github.com/Babelscape/multinerd)
- **Paper:** [https://aclanthology.org/multinerd](https://aclanthology.org/2022.findings-naacl.60/)
- **Point of Contact:** [tedeschi@babelscape.com](tedeschi@babelscape.com)


## Description

- **Summary:** In a nutshell, MultiNERD is the first **language-agnostic** methodology for automatically creating **multilingual, multi-genre and fine-grained annotations** for **Named Entity Recognition** and **Entity Disambiguation**. Specifically, it can be seen an extension of the combination of two prior works from our research group that are [WikiNEuRal](https://www.github.com/Babelscape/wikineural), from which we took inspiration for the state-of-the-art silver-data creation methodology, and [NER4EL](https://www.github.com/Babelscape/NER4EL), from which we took the fine-grained classes and inspiration for the entity linking part. The produced dataset covers: **10 languages** (Chinese, Dutch, English, French, German, Italian, Polish, Portuguese, Russian and Spanish), **15 NER categories** (Person (PER), Location (LOC), Organization (ORG}), Animal (ANIM), Biological entity (BIO), Celestial Body (CEL), Disease (DIS), Event (EVE), Food (FOOD), Instrument (INST), Media (MEDIA), Plant (PLANT), Mythological entity (MYTH), Time (TIME) and Vehicle (VEHI)), and **2 textual genres** ([Wikipedia](https://www.wikipedia.org/) and [WikiNews](https://www.wikinews.org/));
- **Repository:** [https://github.com/Babelscape/multinerd](https://github.com/Babelscape/multinerd)
- **Paper:** [https://aclanthology.org/multinerd](https://aclanthology.org/2022.findings-naacl.60/)
- **Point of Contact:** [tedeschi@babelscape.com](tedeschi@babelscape.com)

## Dataset Structure

The data fields are the same among all splits.
- `tokens`: a `list` of `string` features.
- `ner_tags`: a `list` of classification labels (`int`). 
- `lang`: a `string` feature. Full list of language: Chinese (zh), Dutch (nl), English (en), French (fr), German (de), Italian (it), Polish (pl), Portugues (pt), Russian (ru), Spanish (es).
- The full tagset with indices is reported below:
```python
{
    "O": 0,
    "B-PER": 1,
    "I-PER": 2,
    "B-ORG": 3,
    "I-ORG": 4,
    "B-LOC": 5,
    "I-LOC": 6,
    "B-ANIM": 7,
    "I-ANIM": 8,
    "B-BIO": 9,
    "I-BIO": 10,
    "B-CEL": 11,
    "I-CEL": 12,
    "B-DIS": 13,
    "I-DIS": 14,
    "B-EVE": 15,
    "I-EVE": 16,
    "B-FOOD": 17,
    "I-FOOD": 18,
    "B-INST": 19,
    "I-INST": 20,
    "B-MEDIA": 21,
    "I-MEDIA": 22,
    "B-MYTH": 23,
    "I-MYTH": 24,
    "B-PLANT": 25,
    "I-PLANT": 26,
    "B-TIME": 27,
    "I-TIME": 28,
    "B-VEHI": 29,
    "I-VEHI": 30,
  }
```

## Additional Information

- **Licensing Information**: Contents of this repository are restricted to only non-commercial research purposes under the [Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (CC BY-NC-SA 4.0)](https://creativecommons.org/licenses/by-nc-sa/4.0/). Copyright of the dataset contents belongs to the original copyright holders.

- **Citation Information**: Please consider citing our work if you use data and/or code from this repository.
```bibtex
@inproceedings{tedeschi-navigli-2022-multinerd,
    title = "{M}ulti{NERD}: A Multilingual, Multi-Genre and Fine-Grained Dataset for Named Entity Recognition (and Disambiguation)",
    author = "Tedeschi, Simone  and
      Navigli, Roberto",
    booktitle = "Findings of the Association for Computational Linguistics: NAACL 2022",
    month = jul,
    year = "2022",
    address = "Seattle, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2022.findings-naacl.60",
    doi = "10.18653/v1/2022.findings-naacl.60",
    pages = "801--812",
    abstract = "Named Entity Recognition (NER) is the task of identifying named entities in texts and classifying them through specific semantic categories, a process which is crucial for a wide range of NLP applications. Current datasets for NER focus mainly on coarse-grained entity types, tend to consider a single textual genre and to cover a narrow set of languages, thus limiting the general applicability of NER systems.In this work, we design a new methodology for automatically producing NER annotations, and address the aforementioned limitations by introducing a novel dataset that covers 10 languages, 15 NER categories and 2 textual genres.We also introduce a manually-annotated test set, and extensively evaluate the quality of our novel dataset on both this new test set and standard benchmarks for NER.In addition, in our dataset, we include: i) disambiguation information to enable the development of multilingual entity linking systems, and ii) image URLs to encourage the creation of multimodal systems.We release our dataset at https://github.com/Babelscape/multinerd.",
}
```
- **Contributions**: Thanks to [@sted97](https://github.com/sted97) for adding this dataset.

