license:apache-2.0 region:us
---
license: apache-2.0
---

# Dataset Summary

RedPajama-Instruct-Data is curated from a diverse collection of NLP tasks from both [P3 (BigScience)](https://huggingface.co/datasets/bigscience/P3) and [Natural Instruction (AI2)](https://github.com/allenai/natural-instructions), 
and conduct aggressive decontamination against [HELM]((https://crfm.stanford.edu/helm/latest/?group=core_scenarios)), 
in two steps: (1) We first conduct semantic search using each validation example in HELM as the query and get top-100 similar instances from the Instruct data set and check tasks that have any returned instances overlapping (using 10-Gram) with the validation example. 
We remove the entire task if the returned instance and the validation example correspond to the same task 
(In this step, we keep the task in the case that the returned instance happens to use the same Wikipedia article as the validation example, but asks different questions); 
(2) We then remove all instances that have any 10-Gram overlap with any HELM validation example. 
In total, we filtered out 137 tasks and 5.2M instances (out of 1069 tasks and 93.3M instances).

# QuickStart

The materialized version of P3 includes three main fields. The inputs field contains task instructions and data inputs, while the targets field denotes the labels. The third field, meta, provides meta information.
```python
data = load_dataset('togethercomputer/RedPajama-Instruct-Data', data_files='data/P3_decontaminated.jsonl.zst', split='train')
```

For NI, the definition field refers to the task instructions, while inputs represent the input data. The targets field pertains to the labels, and meta provides relevant meta information.
```python
data = load_dataset('togethercomputer/RedPajama-Instruct-Data', data_files='data/NI_decontaminated.jsonl.zst', split='train')
```

# Source Data

RedPajama-Instruct-Data is sourced from two prominent datasets:

- [Public Pool of Prompts](https://huggingface.co/datasets/bigscience/P3): A large dataset featuring various creative tasks obtained from crowdsourcing efforts.
- [Natural-Instructions](https://github.com/allenai/natural-instructions): An instruction-tuning dataset comprising a diverse set of tasks in natural languages.

# Languages

Primarily English.

# Licensing Information

This dataset is released under the licsence of Apache 2.0.

