task_categories:token-classification size_categories:1M<n<10M language:hr license:cc-by-sa-3.0 wikidata wikipedia wikification region:us
---
license: cc-by-sa-3.0
task_categories:
- token-classification
language:
- hr
tags:
- wikidata
- wikipedia
- wikification
pretty_name: WikiAnc HR
size_categories:
- 1M<n<10M
---

# Dataset Card for WikiAnc HR

## Table of Contents
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks](#supported-tasks)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
- [Additional Information](#additional-information)
  - [Licensing Information](#licensing-information)

## Dataset Description

- **Repository:** [WikiAnc repository](https://github.com/cyanic-selkie/wikianc)

### Dataset Summary

The WikiAnc HR datasets is an automatically generated dataset from Wikipedia (hr) and Wikidata dumps (March 1, 2023). 

The code for generating the dataset can be found [here](https://github.com/cyanic-selkie/wikianc).

### Supported Tasks

- `wikificiation`: The dataset can be used to train a model for Wikification.

### Languages

The text in the dataset is in Croatian. The associated BCP-47 code is `hr`.

You can find the English version [here](https://huggingface.co/datasets/cyanic-selkie/wikianc-en).

## Dataset Structure

### Data Instances

A typical data point represents a paragraph in a Wikipedia article.

The `paragraph_text` field contains the original text in an NFC normalized, UTF-8 encoded string.

The `paragraph_anchors` field contains a list of anchors, each represented by a struct with the inclusive starting UTF-8 code point `start` field, exclusive ending UTF-8 code point `end` field, a nullable `qid` field, a nullable `pageid` field, and an NFC normalized, UTF-8 encoded `title` (Wikipedia) field.

Additionally, each paragraph has `article_title`, `article_pageid`, and (nullable) `article_qid` fields referring to the article the paragraph came from.

There is also a nullable, NFC normalized, UTF-8 encoded `section_heading` field, and an integer `section_level` field referring to the heading (if it exists) of the article section, and the level in the section hierarchy that the paragraph came from.

The `qid` fields refers to Wikidata's QID identifiers, while the `pageid` and `title` fields refer to Wikipedia's pageID and title identifiers (there is a one-to-one mapping between pageIDs and titles).

**NOTE:** An anchor will always have a `title`, but that doesn't mean it has to have a `pageid`. This is because Wikipedia allows defining anchors to nonexistent articles.

An example from the WikiAnc HR test set looks as follows:

```
{
  "uuid": "8a9569ea-a398-4d14-8bce-76c263a8c0ac",
  "article_title": "Špiro_Dmitrović",
  "article_pageid": 70957,
  "article_qid": 16116278,
  "section_heading": null,
  "section_level": 0,
  "paragraph_text": "Špiro Dmitrović (Benkovac, 1803. – Zagreb, 6. veljače 1868.) hrvatski časnik i politički borac u doba ilirizma.",
  "paragraph_anchors": [
    {
      "start": 17,
      "end": 25,
      "qid": 397443,
      "pageid": 14426,
      "title": "Benkovac"
    },
    {
      "start": 27,
      "end": 32,
      "qid": 6887,
      "pageid": 1876,
      "title": "1803."
    },
    {
      "start": 35,
      "end": 41,
      "qid": 1435,
      "pageid": 5903,
      "title": "Zagreb"
    },
    {
      "start": 43,
      "end": 53,
      "qid": 2320,
      "pageid": 496,
      "title": "6._veljače"
    },
    {
      "start": 54,
      "end": 59,
      "qid": 7717,
      "pageid": 1811,
      "title": "1868."
    },
    {
      "start": 102,
      "end": 110,
      "qid": 680821,
      "pageid": 54622,
      "title": "Ilirizam"
    }
  ]
}
```

### Data Fields

- `uuid`: a UTF-8 encoded string representing a v4 UUID that uniquely identifies the example
- `article_title`: an NFC normalized, UTF-8 encoded Wikipedia title of the article; spaces are replaced with underscores 
- `article_pageid`: an integer representing the Wikipedia pageID of the article
- `article_qid`: an integer representing the Wikidata QID this article refers to; it can be null if the entity didn't exist in Wikidata at the time of the creation of the original dataset
- `section_heading`: a nullable, NFC normalized, UTF-8 encoded string representing the section heading
- `section_level`: an integer representing the level of the section in the section hierarchy
- `paragraph_text`: an NFC normalized, UTF-8 encoded string representing the paragraph
- `paragraph_anchors`: a list of structs representing anchors, each anchor has:
  - `start`: an integer representing the inclusive starting UTF-8 code point of the anchors
  - `end`: an integer representing the exclusive ending UTF-8 code point of the anchor
  - `qid`: a nullable integer representing the Wikidata QID this anchor refers to; it can be null if the entity didn't exist in Wikidata at the time of the creation of the original dataset
  - `pageid`: a nullable integer representing the Wikipedia pageID of the anchor; it can be null if the article didn't exist in Wikipedia at the time of the creation of the original dataset
  - `title`: an NFC normalized, UTF-8 encoded string representing the Wikipedia title of the anchor; spaces are replaced with underscores; can refer to a nonexistent Wikipedia article 

### Data Splits

The data is split into training, validation and test sets; paragraphs belonging to the same article aren't necessarily in the same split. The final split sizes are as follows:

|                                   | Train        | Validation | Test      |
| :-----                            | :------:     | :-----:    | :----:    |
| WikiAnc HR - articles             | 192,653      | 116,375    | 116,638   |
| WikiAnc HR - paragraphs           | 2,346,651    | 292,590    | 293,557   |
| WikiAnc HR - anchors              | 8,368,928    | 1,039,851  | 1,044,828 |
| WikiAnc HR - anchors with QIDs    | 7,160,367    | 891,959    | 896,414   |
| WikiAnc HR - anchors with pageIDs | 7,179,116    | 894,313    | 898,692   |

**NOTE:** The number of articles in the table above refers to the number of articles that have at least one paragraph belonging to the article appear in the split.

## Additional Information

### Licensing Information

The WikiAnc HR dataset is given under the [Creative Commons Attribution 4.0 International](https://creativecommons.org/licenses/by/4.0/) license.