task_categories:text-generation task_categories:fill-mask task_ids:language-modeling task_ids:masked-language-modeling annotations_creators:no-annotation language_creators:crowdsourced multilinguality:multilingual size_categories:n<1K size_categories:1K<n<10K size_categories:10K<n<100K size_categories:100K<n<1M size_categories:1M<n<10M source_datasets:original language:bg language:cs language:da language:de language:el language:en language:es language:et language:fi language:fr language:ga language:hr language:hu language:it language:lt language:lv language:mt language:nl language:pl language:pt language:ro language:sk language:sl language:sv license:cc-by-sa-3.0 license:gfdl region:us
Wikipedia dataset containing cleaned articles of all languages.
The datasets are built from the Wikipedia dump
(https://dumps.wikimedia.org/) with one split per language. Each example
contains the content of one full Wikipedia article with cleaning to strip
markdown and unwanted sections (references, etc.).
@ONLINE {wikidump,
    author = {Wikimedia Foundation},
    title  = {Wikimedia Downloads},
    url    = {https://dumps.wikimedia.org}
}
---
annotations_creators:
- no-annotation
language_creators:
- crowdsourced
pretty_name: Wikipedia
paperswithcode_id: null
license:
- cc-by-sa-3.0
- gfdl
task_categories:
- text-generation
- fill-mask
task_ids:
- language-modeling
- masked-language-modeling
source_datasets:
- original
multilinguality:
- multilingual
size_categories:
- n<1K
- 1K<n<10K
- 10K<n<100K
- 100K<n<1M
- 1M<n<10M
language:
   - bg
   - cs
   - da
   - de
   - el
   - en
   - es
   - et
   - fi
   - fr
   - ga
   - hr
   - hu
   - it
   - lt
   - lv
   - mt
   - nl
   - pl
   - pt
   - ro
   - sk
   - sl
   - sv
---

# Dataset Card for Wikipedia

This repo is a wrapper around [olm/wikipedia](https://huggingface.co/datasets/olm/wikipedia) that just concatenates data from the EU languages.
Please refer to it for a complete data card.

The EU languages we include are:
   - bg
   - cs
   - da
   - de
   - el
   - en
   - es
   - et
   - fi
   - fr
   - ga
   - hr
   - hu
   - it
   - lt
   - lv
   - mt
   - nl
   - pl
   - pt
   - ro
   - sk
   - sl
   - sv


As with `olm/wikipedia` you will need to install a few dependencies:


```
pip install mwparserfromhell==0.6.4 multiprocess==0.70.13
```

```python
from datasets import load_dataset

load_dataset("dlwh/eu_wikipedias", date="20221101")
```

Please refer to the original olm/wikipedia for a complete data card.
