size_categories:100M<n<1B region:us
\
Wikipedia dataset containing cleaned articles of all languages.
The datasets are built from the Wikipedia dump
(https://dumps.wikimedia.org/) with one split per language. Each example
contains the content of one full Wikipedia article with cleaning to strip
markdown and unwanted sections (references, etc.).
\
@ONLINE {wikidump,
    author = {Wikimedia Foundation},
    title  = {Wikimedia Downloads},
    url    = {https://dumps.wikimedia.org}
}
---
size_categories:
- 100M<n<1B
---
### Dataset Summary

This dataset is based on [olm/wikipedia](https://huggingface.co/datasets/olm/wikipedia).
The main difference is that we add `Section::::` prefix to each section title to keep the section structure information.
We also use `:` to join the hierarchical section titles.

Following is an example.
```text
Alison Jane Horner (born June 1966) is a British businesswoman, and, until it was sold in 2020, was the CEO of the Asian arm of the Tesco supermarket chain.

Section::::Early life
Alison Jane Horner was born in June 1966. She earned a bachelor's degree in chemistry from the University of Manchester, and an MBA from Manchester Business School.

Section::::Career

Section::::Career:Tesco
Horner joined Tesco as a personnel manager in 1999 and was on Tesco's executive committee from 2011.

In October 2013, Horner became a founding member of The Guardian's Women in Leadership network. in 2015, she became a member of Alliance Manchester Business School's advisory board.

Horner was Tesco' chief people officer (chief human resources officer) of Tesco until May 2018, when she was promoted to be chief executive of Tesco's Asia business in Malaysia and Thailand, until it was sold in late 2020. She was set to step down in February 2021 after 22 years with Tesco.

Section::::Career:Carillion non-executive role
Horner was a non-executive director of Carillion from December 2013, chairing the remuneration committee from June 2014. As of 30 December 2016 her basic compensation was £61,000. After the company went into liquidation in January 2018, Horner was one of the non-executive directors who gave evidence to the House of Commons Business and Work and Pensions select committees on 6 February 2018. In the final report of the Parliamentary Inquiry, published on 16 May 2018, Horner was criticised by MPs; the report concluded:

"... Alison Horner presided over growing salaries and bonuses at the top of the company as its performance faltered. In her evidence to us, she sought to justify her approach by pointing to industry standards, the guidance of advisors, and conversations with shareholders. She failed to demonstrate to us any sense of challenge to the advice she was given, any concern about the views of stakeholders, or any regret at the largesse at the top of Carillion. Ms Horner continues to hold the role of Chief People Officer of Tesco, where she has responsibilities to more than half a million employees. We hope that, in that post, she will reflect on the lessons learned from Carillion and her role in its collapse."

In January 2021, the Insolvency Service said it would seek to ban eight former Carillion directors, including Horner, from holding senior boardroom positions.

Section::::References

Living people
1966 births
British businesspeople in retailing
Tesco people
Alumni of the University of Manchester
Alumni of the Manchester Business School
Carillion people
```

### Data Fields

- `title`: a `string` feature.
- `text`: a `string` feature.

### How to use this dataset

To load this dataset you need to install these first:

```shell
pip install mwparserfromhell==0.6.4 multiprocess==0.70.13
```

Then, you can load any subset of Wikipedia per language and per date this way:

```python
from datasets import load_dataset

dataset = load_dataset("intfloat/wikipedia", language="en", date="20230401")
```

For more information,
please check out [olm/wikipedia](https://huggingface.co/datasets/olm/wikipedia).

## Supported Languages

```
aa
ab
ace
ady
af
ak
als
am
an
ang
ar
arc
arz
as
ast
atj
av
ay
az
azb
ba
bar
bat-smg
bcl
be
be-x-old
bg
bh
bi
bjn
bm
bn
bo
bpy
br
bs
bug
bxr
ca
cbk-zam
cdo
ce
ceb
ch
cho
chr
chy
ckb
co
cr
crh
cs
csb
cu
cv
cy
da
de
din
diq
dsb
dty
dv
dz
ee
el
eml
en
eo
es
et
eu
ext
fa
ff
fi
fiu-vro
fj
fo
fr
frp
frr
fur
fy
ga
gag
gan
gd
gl
glk
gn
gom
gor
got
gu
gv
ha
hak
haw
he
hi
hif
ho
hr
hsb
ht
hu
hy
ia
id
ie
ig
ii
ik
ilo
inh
io
is
it
iu
ja
jam
jbo
jv
ka
kaa
kab
kbd
kbp
kg
ki
kj
kk
kl
km
kn
ko
koi
krc
ks
ksh
ku
kv
kw
ky
la
lad
lb
lbe
lez
lfn
lg
li
lij
lmo
ln
lo
lrc
lt
ltg
lv
mai
map-bms
mdf
mg
mh
mhr
mi
min
mk
ml
mn
mr
mrj
ms
mt
mus
mwl
my
myv
mzn
na
nah
nap
nds
nds-nl
ne
new
ng
nl
nn
no
nov
nrm
nso
nv
ny
oc
olo
om
or
os
pa
pag
pam
pap
pcd
pdc
pfl
pi
pih
pl
pms
pnb
pnt
ps
pt
qu
rm
rmy
rn
ro
roa-rup
roa-tara
ru
rue
rw
sa
sah
sat
sc
scn
sco
sd
se
sg
sh
si
simple
sk
sl
sm
sn
so
sq
sr
srn
ss
st
stq
su
sv
sw
szl
ta
tcy
te
tet
tg
th
ti
tk
tl
tn
to
tpi
tr
ts
tt
tum
tw
ty
tyv
udm
ug
uk
ur
uz
ve
vec
vep
vi
vls
vo
wa
war
wo
wuu
xal
xh
xmf
yi
yo
za
zea
zh
zh-classical
zh-min-nan
zh-yue
zu
```