task_categories:text-generation task_categories:fill-mask size_categories:100K<n<1M language:en license:apache-2.0 language modeling lamguage 2023 data region:us
---
license: apache-2.0
task_categories:
- text-generation
- fill-mask
language:
- en
tags:
- language modeling
- lamguage
- 2023 data
size_categories:
- 100K<n<1M
---

# simple wikipedia

the 'simple' split of Wikipedia, from Sept 1 2023. The train split contains about 65M tokens,

Pulled via:

```python
dataset = load_dataset(
    "wikipedia", language="simple", date="20230901", beam_runner="DirectRunner"
)
```

## stats

### train split

general info 

```
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 226242 entries, 0 to 226241
Data columns (total 4 columns):
 #   Column       Non-Null Count   Dtype 
---  ------       --------------   ----- 
 0   id           226242 non-null  string
 1   url          226242 non-null  string
 2   title        226242 non-null  string
 3   text         226242 non-null  string
dtypes: string(4)
```

token length (NeoX)

![plot](neox_tokencounts_train.png)

|       |   tokens |
|:------|--------------:|
| count |    226242     |
| mean  |       287.007 |
| std   |      1327.07  |
| min   |         1     |
| 25%   |        65     |
| 50%   |       126     |
| 75%   |       243     |
| max   |     60844     |