task_categories:text-generation task_categories:fill-mask size_categories:100K<n<1M source_datasets:pszemraj/simple_wikipedia language:en license:apache-2.0 region:us
---
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
  - split: validation
    path: data/validation-*
  - split: test
    path: data/test-*
dataset_info:
  features:
  - name: id
    dtype: string
  - name: url
    dtype: string
  - name: title
    dtype: string
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 248051733
    num_examples: 226242
  - name: validation
    num_bytes: 6910685
    num_examples: 5954
  - name: test
    num_bytes: 6359625
    num_examples: 5954
  download_size: 152635605
  dataset_size: 261322043
license: apache-2.0
language:
- en
source_datasets: pszemraj/simple_wikipedia
task_categories:
- text-generation
- fill-mask
size_categories:
- 100K<n<1M
---
# Dataset Card for "simple_wikipedia_LM"


A filtered/edited version of [pszemraj/simple_wikipedia](https://huggingface.co/datasets/pszemraj/simple_wikipedia) that removes headings/contents that appear in the `text` column without any relevant text for them (_at least in the `simple` split_).

```python
import re


def split_on_headings(text):
    headings = ["References", "Related pages", "Other websites", "Further reading"]

    for heading in headings:

        parts = re.split(
            r"^\s*" + re.escape(heading) + r".*$", text, flags=re.MULTILINE
        )

        if len(parts) > 1:
            return parts[0].strip()

    return text


text = """
Central Zazaki is a dialect of the Zazaki language. It is spoken in Eastern Anatolia Region of Turkey. 
Related pages
Zazaki
Central Anatolia Region
Other websites
example.com
"""

print(split_on_headings(text))
```