license:other region:us
---
license: other
---

# Dataset Summary

This collection is a compilation of long context datasets, specifically designed for tasks requiring extensive comprehension and inference from large text inputs. 
Currently, it encompasses data intended for training a robust base model, which can be found in the pretrain/ directory. Additionally, it includes datasets tailored for specific needs, located in the fine-tune/ directory. These specialized datasets include multi-passage question answering, derived from Natural Questions, and long-context summarization, exemplified by the BookSum dataset.

# Detailed Description

## Pretrain Data

The pretraining data is a collection of diverse datasets utilized to train the AI model. These datasets include a variety of sources that provide a wide range of information, from books to scientific papers, and instruction data. Here's a detailed look at each:

### RedPajama-Book

This dataset is a specific slice of the larger RedPajama-Data-1T. The RedPajama-Book subset specifically focuses on data extracted from books. This broad and diverse range of literary content helps the model to understand and generate text in a wide variety of styles, genres, and topics, and especially in a wide range of context.

### RedPajama-ArXiv

The RedPajama-ArXiv dataset is another specific slice of RedPajama-Data-1T. In this dataset, the abstract corresponding to each paper is appended after the paper, providing a summary of the paper's content. This helps the model to leverage the long-range context.

### UL2 Oscar

This dataset is generated with LAION-AI's Open-Instruction-Generalist, asking the model to fill in missing chunks, or complete the text.

### RedPajama

This is a subset of the RedPajama-Data-1T. The RedPajama dataset is a large and diverse dataset that includes a wide variety of data sources. The specific subset used in this case (togethercomputer/RedPajama-Data-1T-Sample) is a representative sample of the larger dataset, providing a broad overview of the types of data included in RedPajama-Data-1T.

### NI

The Materialized Natural Instruction (NI) data is a dataset that focuses on natural language instructions. This dataset has been decontaminated against HELM core scenarios, meaning any data that matches specific scenarios outlined in the HELM core has been removed to avoid bias or overfitting. This dataset aids the model in understanding and generating instructional text.

### P3

The Materialized Public Pool of Prompts (P3) data is a dataset that includes a wide variety of user-generated prompts. This dataset has also been decontaminated against HELM core scenarios. The P3 dataset helps the model in understanding a broad set of user prompts and generating appropriate responses.

### Pile

The Pile dataset is a large and diverse dataset that includes a wide variety of data sources. The specific subset used in this case is a subsample of the larger Pile dataset.

## Fine-tune Data

### Multi-passage QA from Natural Questions:

This dataset is a multi-passage question answering dataset derived from the original Natural Questions (NQ) dataset by Google. The NQ dataset consists of real user queries issued to Google's search engine, paired with high-quality answers. In this derived version, each example consists of a question along with multiple (10-200) Wiki passages, from which the model must infer the correct answer. This dataset is designed to challenge and evaluate models on their ability to handle complex, multi-passage question answering.

### BookSum:

BookSum is a dataset for long context summarization. It includes a vast collection of books from various genres, and the task is to generate a coherent and concise summary given a long context from the book. This dataset is designed to test and train models on their ability to understand and summarize long, complex narratives.

# Dataset Limitations and Future Work

While these datasets provide a robust platform for training and evaluating models on long context tasks, they may still contain some limitations. For instance, the datasets might be biased towards the types of questions asked in Google's search engine and the genres of books included in the BookSum dataset. In the future, we plan to expand this collection to include more diverse datasets for a wider range of long context tasks.


# Licensing Information

Please refer to the original sources of the datasets for information on their respective licenses.