Add the 800GB Pile dataset?
The pile dataset would be very nice.
Benchmarks show that pile trained models achieve better results than most of actually trained models
## Adding a Dataset
- **Name:** The Pile
- **Description:** The Pile is a 825 GiB diverse, open source language modelling data set that consists of 22 smaller, high-quality datasets combined together. See [here](https://twitter.com/nabla_theta/status/1345130408170541056?s=20) for the Twitter announcement
- **Paper:** https://pile.eleuther.ai/paper.pdf
- **Data:** https://pile.eleuther.ai/
- **Motivation:** Enables hardcore  (GPT-3 scale!) language modelling

## Remarks
Given the extreme size of this dataset, I'm not sure how feasible this will be to include in `datasets` 🤯  . I'm also unsure how many `datasets` users are pretraining LMs, so the usage of this dataset may not warrant the effort to integrate it.

Add the 800GB Pile dataset? 
 ## Adding a Dataset
- **Name:** The Pile
- **Description:** The Pile is a 825 GiB diverse, open source language modelling data set that consists of 22 smaller, high-quality datasets combined together. See [here](https://twitter.com/nabla_theta/status/1345130408170541056?s=20) for the Twitter announcement
- **Paper:** https://pile.eleuther.ai/paper.pdf
- **Data:** https://pile.eleuther.ai/
- **Motivation:** Enables hardcore  (GPT-3 scale!) language modelling

## Remarks
Given the extreme size of this dataset, I'm not sure how feasible this will be to include in `datasets` 🤯  . I'm also unsure how many `datasets` users are pretraining LMs, so the usage of this dataset may not warrant the effort to integrate it.
 
 The pile dataset would be very nice.
Benchmarks show that pile trained models achieve better results than most of actually trained models