task_categories:token-classification size_categories:10K<n<100K language:en license:cc-by-sa-3.0 wikidata wikipedia named-entity-recognition named-entity-linking region:us
---
license: cc-by-sa-3.0
task_categories:
- token-classification
language:
- en
tags:
- wikidata
- wikipedia
- named-entity-recognition
- named-entity-linking
pretty_name: AIDA CoNLL-YAGO Wikidata
size_categories:
- 10K<n<100K
---

# Dataset Card for AIDA CoNLL-YAGO Wikidata

## Table of Contents
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks](#supported-tasks)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
- [Additional Information](#additional-information)
  - [Licensing Information](#licensing-information)

## Dataset Description

- **Repository:** [AIDA CoNLL-YAGO Wikidata repository](https://github.com/cyanic-selkie/aida-conll-yago-wikidata)

### Dataset Summary

The AIDA CoNLL-YAGO Wikidata dataset is the same as the original [AIDA CoNLL-YAGO](https://www.mpi-inf.mpg.de/departments/databases-and-information-systems/research/ambiverse-nlu/aida/downloads) dataset, but with Wikidata QIDs instead of Wikipedia titles as entity identifiers. They are automatically generated (with a few manual corrections) from Wikidata and Wikipedia dumps (March 1, 2023).

The code for generating the dataset can be found [here](https://github.com/cyanic-selkie/aida-conll-yago-wikidata).

### Supported Tasks

- `named-entity-recognition`: The dataset can be used to train a model for Named Entity Recognition.
- `named-entity-linking`: The dataset can be used to train a model for Named Entity Linking.

### Languages

The text in the dataset is in English. The associated BCP-47 code is `en`.

## Dataset Structure

### Data Instances

A typical data point represents a document (news article).

The `text` field contains the original text in an NFC normalized, UTF-8 encoded string.

The `entities` field contains a list of entities, each represented by a struct with the inclusive starting byte `start` field, exclusive ending byte `end` field, a nullable `qid` field, and a nullable `pageid` field.

Additionally, each document has a unique `document_id` field.

An example from the AIDA CoNLL-YAGO Wikidata test set looks as follows:

```
{
  "document_id": 1214,
  "text": "RADIO ROMANIA AFTERNOON HEALINES AT 4 PM . BUCHAREST 1996-12-06 Radio Romania news headlines : * The Democratic Convention signed an agreement on government and parliamentary support with its coalition partners the Social Democratic Union and the Hungarian Democratic Union ( UDMR ) . The ceremony was attended by President Emil Constantinescu . * The three parties in the government coalition have committed themselves to a real reform of Romania 's economy , Constantinescu said after the ceremony . * The UDMR wants to contribute to social reform and economic revival in Romania , union leader Marko Bela said . * The international airport in Timisoara and the domestic airports in Arad , Oradea and Sibiu were closed due to fog . -- Bucharest Newsroom 40-1 3120264",
  "entities": [
    {
      "start": 0,
      "end": 13,
      "tag": "ORG",
      "pageid": null,
      "qid": null,
      "title": null
    },
    {
      "start": 43,
      "end": 52,
      "tag": "LOC",
      "pageid": 36877,
      "qid": 19660,
      "title": "Bucharest"
    },
    {
      "start": 64,
      "end": 77,
      "tag": "ORG",
      "pageid": null,
      "qid": null,
      "title": null
    },
    {
      "start": 101,
      "end": 122,
      "tag": "MISC",
      "pageid": null,
      "qid": null,
      "title": null
    },
    {
      "start": 215,
      "end": 238,
      "tag": "ORG",
      "pageid": null,
      "qid": null,
      "title": null
    },
    {
      "start": 247,
      "end": 273,
      "tag": "ORG",
      "pageid": null,
      "qid": null,
      "title": null
    },
    {
      "start": 276,
      "end": 280,
      "tag": "ORG",
      "pageid": 49749134,
      "qid": 266582,
      "title": "Democratic_Union_of_Hungarians_in_Romania"
    },
    {
      "start": 324,
      "end": 343,
      "tag": "PER",
      "pageid": 393370,
      "qid": 299152,
      "title": "Emil_Constantinescu"
    },
    {
      "start": 440,
      "end": 447,
      "tag": "LOC",
      "pageid": 25445,
      "qid": 218,
      "title": "Romania"
    },
    {
      "start": 461,
      "end": 475,
      "tag": "PER",
      "pageid": 393370,
      "qid": 299152,
      "title": "Emil_Constantinescu"
    },
    {
      "start": 508,
      "end": 512,
      "tag": "ORG",
      "pageid": 49749134,
      "qid": 266582,
      "title": "Democratic_Union_of_Hungarians_in_Romania"
    },
    {
      "start": 574,
      "end": 581,
      "tag": "LOC",
      "pageid": 25445,
      "qid": 218,
      "title": "Romania"
    },
    {
      "start": 597,
      "end": 607,
      "tag": "PER",
      "pageid": 1219345,
      "qid": 897108,
      "title": "Béla_Markó"
    },
    {
      "start": 646,
      "end": 655,
      "tag": "LOC",
      "pageid": 33693389,
      "qid": 83404,
      "title": "Timişoara"
    },
    {
      "start": 685,
      "end": 689,
      "tag": "LOC",
      "pageid": 22537901,
      "qid": 173591,
      "title": "Arad,_Romania"
    },
    {
      "start": 692,
      "end": 698,
      "tag": "LOC",
      "pageid": 2024606,
      "qid": 2102332,
      "title": "Oradea_International_Airport"
    },
    {
      "start": 703,
      "end": 708,
      "tag": "LOC",
      "pageid": 2384413,
      "qid": 946418,
      "title": "Sibiu_International_Airport"
    },
    {
      "start": 737,
      "end": 755,
      "tag": "ORG",
      "pageid": null,
      "qid": null,
      "title": null
    }
  ]
}
```

### Data Fields

- `document_id`: an integer that uniquely identifies the document this sentence belongs to
- `sentence_index`: an integer that uniquely identifies the position of the sentence in its original document
- `text`: an NFC normalized, UTF-8 encoded string representing the sentence
- `entities`: a list of structs representing entities, each entity has:
  - `start`: an integer representing the inclusive starting UTF-8 code point of the entity
  - `end`: an integer representing the exclusive ending UTF-8 code point of the entity
  - `tag`: a string representing the entity type (PER, LOC, ORG or MISC)
  - `qid`: an integer representing the Wikidata QID this entity refers to; it can be null if the entity didn't exist in Wikidata at the time of the creation of the original dataset
  - `pageid`: an integer representing the English Wikipedia's pageID this entity refers to; it can be null if the entity didn't exist in Wikipedia at the time of the creation of the original dataset
  - `title`: an NFC normalized, UTF-8 encoded string representing the English Wikipedia's title this entity refers to; it can be null if the entity didn't exist in Wikipedia at the time of the creation of the original dataset

### Data Splits

The data is split into training, validation and test sets; all of the sentences belonging to an article are in the same split. The final split sizes are as follows:

|                                               | Train     | Validation  | Test    |
| :-----                                        | :------:  | :-----:     | :----:  |
| AIDA CoNLL-YAGO Wikidata - documents          | 946       | 216         | 231     |
| AIDA CoNLL-YAGO Wikidata - entities           | 23,374    | 5,912       | 5,608   |
| AIDA CoNLL-YAGO Wikidata - entities with QIDs | 18,540    | 4,791       | 4,481   |

## Additional Information

### Licensing Information

The licensing status of the dataset is the same as the licensing status of the original [AIDA CoNLL-YAGO](https://www.mpi-inf.mpg.de/departments/databases-and-information-systems/research/ambiverse-nlu/aida/downloads) dataset which is under a [Creative Commons Attribution-ShareAlike 3.0 Unported License](http://creativecommons.org/licenses/by-sa/3.0/deed.en_US).