Commit
•
996f824
1
Parent(s):
21cd91a
Update README.md
Browse files
README.md
CHANGED
|
@@ -7,7 +7,7 @@ datasets:
|
|
| 7 |
language:
|
| 8 |
- en
|
| 9 |
---
|
| 10 |
-
# Instruction Pre-Training: Language Models are Supervised Multitask Learners
|
| 11 |
This repo contains the **general models pre-trained from scratch** (on 100B tokens) in our paper [Instruction Pre-Training: Language Models are Supervised Multitask Learners](https://huggingface.co/papers/2406.14491).
|
| 12 |
|
| 13 |
We explore supervised multitask pre-training by proposing ***Instruction Pre-Training***, a framework that scalably augments massive raw corpora with instruction-response pairs to pre-train language models. The instruction-response pairs are generated by an efficient instruction synthesizer built on open-source models. In our experiments, we synthesize 200M instruction-response pairs covering 40+ task categories to verify the effectiveness of *Instruction Pre-Training*. Instruction Pre-Training* outperforms *Vanilla Pre-training* in both general pre-training from scratch and domain-adaptive continual pre-training. **In pre-training from scratch, *Instruction Pre-Training* not only improves pre-trained base models but also benefits more from further instruction tuning.** In continual pre-training, *Instruction Pre-Training* enables Llama3-8B to be comparable to or even outperform Llama3-70B.
|
|
@@ -17,6 +17,8 @@ We explore supervised multitask pre-training by proposing ***Instruction Pre-Tra
|
|
| 17 |
</p>
|
| 18 |
|
| 19 |
**************************** **Updates** ****************************
|
|
|
|
|
|
|
| 20 |
* 2024/8/29: Updated [guidelines](https://huggingface.co/instruction-pretrain/medicine-Llama3-8B) on evaluating any 🤗Huggingface models on the domain-specific tasks
|
| 21 |
* 2024/7/31: Updated pre-training suggestions in the `Advanced Usage` section of [instruction-synthesizer](https://huggingface.co/instruction-pretrain/instruction-synthesizer)
|
| 22 |
* 2024/7/15: We scaled up the pre-trained tokens from 100B to 250B, with the number of synthesized instruction-response pairs reaching 500M. The performance trend on downstream tasks throughout the pre-training process:
|
|
@@ -75,7 +77,7 @@ accelerate launch -m lm_eval --model hf \
|
|
| 75 |
## Citation
|
| 76 |
If you find our work helpful, please cite us:
|
| 77 |
|
| 78 |
-
Instruction Pre-Training
|
| 79 |
```bibtex
|
| 80 |
@article{cheng2024instruction,
|
| 81 |
title={Instruction Pre-Training: Language Models are Supervised Multitask Learners},
|
|
|
|
| 7 |
language:
|
| 8 |
- en
|
| 9 |
---
|
| 10 |
+
# Instruction Pre-Training: Language Models are Supervised Multitask Learners (EMNLP 2024)
|
| 11 |
This repo contains the **general models pre-trained from scratch** (on 100B tokens) in our paper [Instruction Pre-Training: Language Models are Supervised Multitask Learners](https://huggingface.co/papers/2406.14491).
|
| 12 |
|
| 13 |
We explore supervised multitask pre-training by proposing ***Instruction Pre-Training***, a framework that scalably augments massive raw corpora with instruction-response pairs to pre-train language models. The instruction-response pairs are generated by an efficient instruction synthesizer built on open-source models. In our experiments, we synthesize 200M instruction-response pairs covering 40+ task categories to verify the effectiveness of *Instruction Pre-Training*. Instruction Pre-Training* outperforms *Vanilla Pre-training* in both general pre-training from scratch and domain-adaptive continual pre-training. **In pre-training from scratch, *Instruction Pre-Training* not only improves pre-trained base models but also benefits more from further instruction tuning.** In continual pre-training, *Instruction Pre-Training* enables Llama3-8B to be comparable to or even outperform Llama3-70B.
|
|
|
|
| 17 |
</p>
|
| 18 |
|
| 19 |
**************************** **Updates** ****************************
|
| 20 |
+
* 2024/9/20: Our paper has been accepted by EMNLP 2024 main conference🎉
|
| 21 |
+
* 2024/9/11: Updated [FAQ on continual pre-training from Llama3](https://huggingface.co/instruction-pretrain/instruction-synthesizer)
|
| 22 |
* 2024/8/29: Updated [guidelines](https://huggingface.co/instruction-pretrain/medicine-Llama3-8B) on evaluating any 🤗Huggingface models on the domain-specific tasks
|
| 23 |
* 2024/7/31: Updated pre-training suggestions in the `Advanced Usage` section of [instruction-synthesizer](https://huggingface.co/instruction-pretrain/instruction-synthesizer)
|
| 24 |
* 2024/7/15: We scaled up the pre-trained tokens from 100B to 250B, with the number of synthesized instruction-response pairs reaching 500M. The performance trend on downstream tasks throughout the pre-training process:
|
|
|
|
| 77 |
## Citation
|
| 78 |
If you find our work helpful, please cite us:
|
| 79 |
|
| 80 |
+
[Instruction Pre-Training](https://huggingface.co/papers/2406.14491) (EMNLP 2024)
|
| 81 |
```bibtex
|
| 82 |
@article{cheng2024instruction,
|
| 83 |
title={Instruction Pre-Training: Language Models are Supervised Multitask Learners},
|