arxiv:2305.17266 arxiv:1910.10683 arxiv:1511.02301 region:us
---
dataset_info:
  features:
  - name: TEXT
    dtype: string
  splits:
  - name: train
    num_bytes: 5437652389
    num_examples: 9081490
  - name: validation
    num_bytes: 50107745
    num_examples: 100000
  - name: test
    num_bytes: 50134861
    num_examples: 100000
  download_size: 3732550490
  dataset_size: 5537894995
---

# Dataset Card for unconstrained_language

## Table of Contents
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
- [Citation Information](#additional-information)
  - [Citation Information](#citation-information)


## Dataset Description

- **Paper: https://arxiv.org/abs/2305.17266**
- **Point of Contact: vijeta_deshpande@student.uml.edu**

### Dataset Summary

This dataset is one of the two datasets published by "Honey, I Shrunk the Language: Language Model Behavior at Reduced Scale" (https://arxiv.org/abs/2305.17266). 
The dataset available at this link is the pre-training data **not** constrained by any predefined vocabulary. The other published data i.e. the pre-training data that is constrained by vocabulary is available at https://huggingface.co/datasets/text-machine-lab/constrained_language.

This dataset is curated by randomly sampling text spans (of an approximate length of 128 tokens) from the following corpora,
- C4: https://arxiv.org/abs/1910.10683,
- BookCorpus: https://ieeexplore.ieee.org/document/7410368,
- Wikipedia: https://huggingface.co/datasets/wikipedia,
- Simplified-Wikipedia: https://simple.wikipedia.org/wiki/Main_Page,
- Children's Book Test Corpus: https://arxiv.org/abs/1511.02301

The dataset includes ~9 million contiguous spans, each with approximately 128 tokens. 

### Languages

The dataset contains the English language only.

## Dataset Structure

The dataset is available in the arrow dataset format with three splits namely, train, validation, and test. Every data instance has only one key "Text" that included a text span of approximately 128 tokens.

### Citation Information
If this dataset is useful to you please cite our work.

```sh
@article{deshpande2023honey,
  title={Honey, I Shrunk the Language: Language Model Behavior at Reduced Scale},
  author={Deshpande, Vijeta and Pechi, Dan and Thatte, Shree and Lialin, Vladislav and Rumshisky, Anna},
  journal={arXiv preprint arXiv:2305.17266},
  year={2023}
}
```