Add the 800GB Pile dataset?
Hi folks, thanks to some awesome work by @lhoestq and @albertvillanova you can now stream the Pile as follows:

```python
# Install master branch of `datasets`
pip install git+https://github.com/huggingface/datasets.git#egg=datasets[streaming]
pip install zstandard

from datasets import load_dataset

dset = load_dataset("json", data_files="https://the-eye.eu/public/AI/pile/train/00.jsonl.zst", streaming=True, split="train")
next(iter(dset))
# {'meta': {'pile_set_name': 'Pile-CC'},
# 'text': 'It is done, and submitted. You can play “Survival of the Tastiest” on Android, and on the web ... '}
```

Next step is to add the Pile as a "canonical" dataset that can be streamed without specifying the file names explicitly :)
## Adding a Dataset
- **Name:** The Pile
- **Description:** The Pile is a 825 GiB diverse, open source language modelling data set that consists of 22 smaller, high-quality datasets combined together. See [here](https://twitter.com/nabla_theta/status/1345130408170541056?s=20) for the Twitter announcement
- **Paper:** https://pile.eleuther.ai/paper.pdf
- **Data:** https://pile.eleuther.ai/
- **Motivation:** Enables hardcore  (GPT-3 scale!) language modelling

## Remarks
Given the extreme size of this dataset, I'm not sure how feasible this will be to include in `datasets` 🤯  . I'm also unsure how many `datasets` users are pretraining LMs, so the usage of this dataset may not warrant the effort to integrate it.

Add the 800GB Pile dataset? 
 ## Adding a Dataset
- **Name:** The Pile
- **Description:** The Pile is a 825 GiB diverse, open source language modelling data set that consists of 22 smaller, high-quality datasets combined together. See [here](https://twitter.com/nabla_theta/status/1345130408170541056?s=20) for the Twitter announcement
- **Paper:** https://pile.eleuther.ai/paper.pdf
- **Data:** https://pile.eleuther.ai/
- **Motivation:** Enables hardcore  (GPT-3 scale!) language modelling

## Remarks
Given the extreme size of this dataset, I'm not sure how feasible this will be to include in `datasets` 🤯  . I'm also unsure how many `datasets` users are pretraining LMs, so the usage of this dataset may not warrant the effort to integrate it.
 
 Hi folks, thanks to some awesome work by @lhoestq and @albertvillanova you can now stream the Pile as follows:

```python
# Install master branch of `datasets`
pip install git+https://github.com/huggingface/datasets.git#egg=datasets[streaming]
pip install zstandard

from datasets import load_dataset

dset = load_dataset("json", data_files="https://the-eye.eu/public/AI/pile/train/00.jsonl.zst", streaming=True, split="train")
next(iter(dset))
# {'meta': {'pile_set_name': 'Pile-CC'},
# 'text': 'It is done, and submitted. You can play “Survival of the Tastiest” on Android, and on the web ... '}
```

Next step is to add the Pile as a "canonical" dataset that can be streamed without specifying the file names explicitly :)