task_categories:fill-mask annotations_creators:other language_creators:found multilinguality:multilingual size_categories:10M<n<100M source_datasets:original language:bg language:cs language:da language:de language:el language:en language:es language:et language:fi language:fr language:ga language:hr language:hu language:it language:lt language:lv language:mt language:nl language:pl language:pt language:ro language:sk language:sl language:sv license:cc-by-4.0 region:us
Wikipedia dataset containing cleaned articles of all languages.
The datasets are built from the Wikipedia dump
(https://dumps.wikimedia.org/) with one split per language. Each example
contains the content of one full Wikipedia article with cleaning to strip
markdown and unwanted sections (references, etc.).
@ONLINE {wikidump,
    author = {Wikimedia Foundation},
    title  = {Wikimedia Downloads},
    url    = {https://dumps.wikimedia.org}
}
---
annotations_creators:
- other
language_creators:
- found
language:
- bg 
- cs 
- da 
- de 
- el 
- en
- es 
- et 
- fi 
- fr 
- ga
- hr 
- hu 
- it 
- lt 
- lv 
- mt
- nl 
- pl 
- pt 
- ro 
- sk 
- sl 
- sv
license:
- cc-by-4.0
multilinguality:
- multilingual
paperswithcode_id: null
pretty_name: "EUWikipedias: A dataset of Wikipedias in the EU languages"
size_categories:
- 10M<n<100M
source_datasets:
- original
task_categories:
- fill-mask

---

# Dataset Card for EUWikipedias: A dataset of Wikipedias in the EU languages

## Table of Contents

- [Table of Contents](#table-of-contents)
- [Dataset Description](#dataset-description)
    - [Dataset Summary](#dataset-summary)
    - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
    - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
    - [Data Instances](#data-instances)
    - [Data Fields](#data-fields)
    - [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
    - [Curation Rationale](#curation-rationale)
    - [Source Data](#source-data)
    - [Annotations](#annotations)
    - [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
    - [Social Impact of Dataset](#social-impact-of-dataset)
    - [Discussion of Biases](#discussion-of-biases)
    - [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
    - [Dataset Curators](#dataset-curators)
    - [Licensing Information](#licensing-information)
    - [Citation Information](#citation-information)
    - [Contributions](#contributions)

## Dataset Description

- **Homepage:**
- **Repository:** 
- **Paper:** 
- **Leaderboard:**
- **Point of Contact:** [Joel Niklaus](mailto:joel.niklaus.2@bfh.ch)

### Dataset Summary

Wikipedia dataset containing cleaned articles of all languages.
The datasets are built from the Wikipedia dump
(https://dumps.wikimedia.org/) with one split per language. Each example
contains the content of one full Wikipedia article with cleaning to strip
markdown and unwanted sections (references, etc.).

### Supported Tasks and Leaderboards

The dataset supports the tasks of fill-mask.

### Languages

The following languages are supported: 
bg, cs, da, de, el, en, es, et, fi, fr, ga, hr, hu, it, lt, lv, mt, nl, pl, pt, ro, sk, sl, sv

## Dataset Structure

It is structured in the following format: {date}/{language}_{shard}.jsonl.xz
At the moment only the date '20221120' is supported.

Use the dataset like this:
```python
from datasets import load_dataset

dataset = load_dataset('joelito/EU_Wikipedias', date="20221120", language="de", split='train', streaming=True)
```


### Data Instances

The file format is jsonl.xz and there is one split available (`train`). 

| Source       |   Size (MB) |      Words |   Documents |   Words/Document |
|:-------------|------------:|-----------:|------------:|-----------------:|
| 20221120.all |       86034 | 9506846949 |    26481379 |              359 |
| 20221120.bg  |        1261 |   88138772 |      285876 |              308 |
| 20221120.cs  |        1904 |  189580185 |      513851 |              368 |
| 20221120.da  |         679 |   74546410 |      286864 |              259 |
| 20221120.de  |       11761 | 1191919523 |     2740891 |              434 |
| 20221120.el  |        1531 |  103504078 |      215046 |              481 |
| 20221120.en  |       26685 | 3192209334 |     6575634 |              485 |
| 20221120.es  |        6636 |  801322400 |     1583597 |              506 |
| 20221120.et  |         538 |   48618507 |      231609 |              209 |
| 20221120.fi  |        1391 |  115779646 |      542134 |              213 |
| 20221120.fr  |        9703 | 1140823165 |     2472002 |              461 |
| 20221120.ga  |          72 |    8025297 |       57808 |              138 |
| 20221120.hr  |         555 |   58853753 |      198746 |              296 |
| 20221120.hu  |        1855 |  167732810 |      515777 |              325 |
| 20221120.it  |        5999 |  687745355 |     1782242 |              385 |
| 20221120.lt  |         409 |   37572513 |      203233 |              184 |
| 20221120.lv  |         269 |   25091547 |      116740 |              214 |
| 20221120.mt  |          29 |    2867779 |        5030 |              570 |
| 20221120.nl  |        3208 |  355031186 |     2107071 |              168 |
| 20221120.pl  |        3608 |  349900622 |     1543442 |              226 |
| 20221120.pt  |        3315 |  389786026 |     1095808 |              355 |
| 20221120.ro  |        1017 |  111455336 |      434935 |              256 |
| 20221120.sk  |         506 |   49612232 |      238439 |              208 |
| 20221120.sl  |         543 |   58858041 |      178472 |              329 |
| 20221120.sv  |        2560 |  257872432 |     2556132 |              100 |


### Data Fields

[More Information Needed]

### Data Splits

[More Information Needed]

## Dataset Creation

This dataset has been created by downloading the wikipedias using [olm/wikipedia](https://huggingface.co/datasets/olm/wikipedia) for the 24 EU languages.
For more information about the creation of the dataset please refer to prepare_wikipedias.py

### Curation Rationale

[More Information Needed]

### Source Data

#### Initial Data Collection and Normalization

[More Information Needed]

#### Who are the source language producers?

[More Information Needed]


### Annotations

#### Annotation process

[More Information Needed]

#### Who are the annotators?

[More Information Needed]

### Personal and Sensitive Information

[More Information Needed]

## Considerations for Using the Data

### Social Impact of Dataset

[More Information Needed]

### Discussion of Biases

[More Information Needed]

### Other Known Limitations

[More Information Needed]

## Additional Information

### Dataset Curators

[More Information Needed]

### Licensing Information

[More Information Needed]

### Citation Information

```
TODO add citation
```

### Contributions

Thanks to [@JoelNiklaus](https://github.com/joelniklaus) for adding this dataset.
