language:en wikipedia region:us
---
language:
- en
tags:
- wikipedia
---

# Dataset Description

This dataset contains the first paragraph of cleaned Wikipedia articles in English. 

It was obtained by transorming the [Wikipedia](https://huggingface.co/datasets/wikipedia) "20220301.en" dataset as follows:
```python
from datasets import load_dataset

dataset = load_dataset("wikipedia", "20220301.en")["train"]

def get_first_paragraph(example):
  example["text"] = example['text'].split('\n\n')[0]
  return example

dataset = dataset.map(get_first_paragraph)
```

# Why use this dataset?
The size of the original English Wikipedia dataset is over 20GB. It takes 20min to load it on a Google Colab notebook and running computations on that dataset can be costly.

If you want to create a use case that mostly needs the information in the first paragraph of a Wikipedia article (which is the paragraph with the most important information), this 'wikipedia-first-paragraph' dataset is for you.

Its size is 1.39GB and it takes 5 min to load it on a Google colab notebook.


# How to load dataset

You can load it by runnning:
```python
from datasets import load_dataset

load_dataset("abokbot/wikipedia-first-paragraph")

```

# Dataset Structure
An example looks as follows:
```
{
  'id': '12',
  'url': 'https://en.wikipedia.org/wiki/Anarchism',
  'title': 'Anarchism',
  'text': 'Anarchism is a political philosophy and movement that is sceptical of authority and rejects \
          all involuntary, coercive forms of hierarchy. Anarchism calls for the abolition of the state, \
          which it holds to be unnecessary, undesirable, and harmful. As a historically left-wing movement, \
          placed on the farthest left of the political spectrum, it is usually described alongside communalism \
          and libertarian Marxism as the libertarian wing (libertarian socialism) of the socialist movement, and \
          has a strong historical association with anti-capitalism and socialism.'
}
```