task_categories:token-classification size_categories:10M<n<100M language:en license:cc-by-sa-3.0 wikidata wikipedia wikification region:us
---
license: cc-by-sa-3.0
task_categories:
- token-classification
language:
- en
tags:
- wikidata
- wikipedia
- wikification
pretty_name: WikiAnc EN
size_categories:
- 10M<n<100M
---

# Dataset Card for WikiAnc EN

## Table of Contents
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks](#supported-tasks)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
- [Additional Information](#additional-information)
  - [Licensing Information](#licensing-information)

## Dataset Description

- **Repository:** [WikiAnc repository](https://github.com/cyanic-selkie/wikianc)

### Dataset Summary

The WikiAnc EN datasets is an automatically generated dataset from Wikipedia (en) and Wikidata dumps (March 1, 2023). 

The code for generating the dataset can be found [here](https://github.com/cyanic-selkie/wikianc).

### Supported Tasks

- `wikificiation`: The dataset can be used to train a model for Wikification.

### Languages

The text in the dataset is in English. The associated BCP-47 code is `en`.

You can find the Croatian version [here](https://huggingface.co/datasets/cyanic-selkie/wikianc-hr).

## Dataset Structure

### Data Instances

A typical data point represents a paragraph in a Wikipedia article.

The `paragraph_text` field contains the original text in an NFC normalized, UTF-8 encoded string.

The `paragraph_anchors` field contains a list of anchors, each represented by a struct with the inclusive starting UTF-8 code point `start` field, exclusive ending UTF-8 code point `end` field, a nullable `qid` field, a nullable `pageid` field, and an NFC normalized, UTF-8 encoded `title` (Wikipedia) field.

Additionally, each paragraph has `article_title`, `article_pageid`, and (nullable) `article_qid` fields referring to the article the paragraph came from.

There is also a nullable, NFC normalized, UTF-8 encoded `section_heading` field, and an integer `section_level` field referring to the heading (if it exists) of the article section, and the level in the section hierarchy that the paragraph came from.

The `qid` fields refers to Wikidata's QID identifiers, while the `pageid` and `title` fields refer to Wikipedia's pageID and title identifiers (there is a one-to-one mapping between pageIDs and titles).

**NOTE:** An anchor will always have a `title`, but that doesn't mean it has to have a `pageid`. This is because Wikipedia allows defining anchors to nonexistent articles.

An example from the WikiAnc EN test set looks as follows:

```
{
  "uuid": "5f74e678-944f-4761-a5e0-b6426f6f61b8",
  "article_title": "Climatius",
  "article_pageid": 5394373,
  "article_qid": 867987,
  "section_heading": null,
  "section_level": 0,
  "paragraph_text": "It was a small fish, at 7.5 cm, and to discourage predators, Climatius sported fifteen sharp spines. There was one spine each on the paired pelvic and pectoral fins, and on the aingle anal and two dorsal fins, and a four pairs without fins on the fish's underside.",
  "paragraph_anchors": [
    {
      "start": 140,
      "end": 146,
      "qid": 3335089,
      "pageid": 56849833,
      "title": "Pelvic_fin"
    },
    {
      "start": 151,
      "end": 159,
      "qid": 4162555,
      "pageid": 331956,
      "title": "Pectoral_fin"
    },
    {
      "start": 184,
      "end": 188,
      "qid": 4162555,
      "pageid": 331958,
      "title": "Anal_fin"
    },
    {
      "start": 197,
      "end": 208,
      "qid": 1568355,
      "pageid": 294244,
      "title": "Dorsal_fin"
    }
  ]
}
```

### Data Fields

- `uuid`: a UTF-8 encoded string representing a v4 UUID that uniquely identifies the example
- `article_title`: an NFC normalized, UTF-8 encoded Wikipedia title of the article; spaces are replaced with underscores 
- `article_pageid`: an integer representing the Wikipedia pageID of the article
- `article_qid`: an integer representing the Wikidata QID this article refers to; it can be null if the entity didn't exist in Wikidata at the time of the creation of the original dataset
- `section_heading`: a nullable, NFC normalized, UTF-8 encoded string representing the section heading
- `section_level`: an integer representing the level of the section in the section hierarchy
- `paragraph_text`: an NFC normalized, UTF-8 encoded string representing the paragraph
- `paragraph_anchors`: a list of structs representing anchors, each anchor has:
  - `start`: an integer representing the inclusive starting UTF-8 code point of the anchors
  - `end`: an integer representing the exclusive ending UTF-8 code point of the anchor
  - `qid`: a nullable integer representing the Wikidata QID this anchor refers to; it can be null if the entity didn't exist in Wikidata at the time of the creation of the original dataset
  - `pageid`: a nullable integer representing the Wikipedia pageID of the anchor; it can be null if the article didn't exist in Wikipedia at the time of the creation of the original dataset
  - `title`: an NFC normalized, UTF-8 encoded string representing the Wikipedia title of the anchor; spaces are replaced with underscores; can refer to a nonexistent Wikipedia article 

### Data Splits

The data is split into training, validation and test sets; paragraphs belonging to the same article aren't necessarily in the same split. The final split sizes are as follows:

|                                   | Train         | Validation   | Test         |
| :-----                            | :------:      | :-----:      | :----:       |
| WikiAnc EN - articles             | 5,883,342     | 2,374,055    | 2,375,830    |
| WikiAnc EN - paragraphs           | 34,555,183    | 4,317,326    | 4,321,613    |
| WikiAnc EN - anchors              | 87,060,158    | 10,876,572   | 10,883,232   |
| WikiAnc EN - anchors with QIDs    | 85,414,610    | 10,671,262   | 10,677,412   |
| WikiAnc EN - anchors with pageIDs | 85,421,513    | 10,672,138   | 10,678,262   |

**NOTE:** The number of articles in the table above refers to the number of articles that have at least one paragraph belonging to the article appear in the split.

## Additional Information

### Licensing Information

The WikiAnc EN dataset is given under the [Creative Commons Attribution 4.0 International](https://creativecommons.org/licenses/by/4.0/) license. 
