Update README.md
Browse files
README.md
CHANGED
@@ -19,7 +19,12 @@ base_model: meta-llama/Llama-2-70b-hf
|
|
19 |
Tulu is a series of language models that are trained to act as helpful assistants.
|
20 |
Tulu 2 QLoRA 70B is a fine-tuned version of Llama 2 that was trained on a mix of publicly available, synthetic and human datasets.
|
21 |
We release the QLoRA modules alone, which can be merged with Llama 2 70b base.
|
22 |
-
We
|
|
|
|
|
|
|
|
|
|
|
23 |
|
24 |
|
25 |
## Model description
|
@@ -77,12 +82,13 @@ The following hyperparameters were used during training:
|
|
77 |
If you find Tulu 2 is useful in your work, please cite it with:
|
78 |
|
79 |
```
|
80 |
-
@misc{
|
81 |
-
|
82 |
-
|
83 |
-
|
84 |
-
|
85 |
-
|
|
|
86 |
}
|
87 |
```
|
88 |
|
|
|
19 |
Tulu is a series of language models that are trained to act as helpful assistants.
|
20 |
Tulu 2 QLoRA 70B is a fine-tuned version of Llama 2 that was trained on a mix of publicly available, synthetic and human datasets.
|
21 |
We release the QLoRA modules alone, which can be merged with Llama 2 70b base.
|
22 |
+
We provide [a merging script](https://github.com/allenai/open-instruct/blob/main/open_instruct/merge_lora.py) in our repository for easy merging.
|
23 |
+
We recommend **dequantising the model before merging**, done using the `--qlora` flag.
|
24 |
+
|
25 |
+
|
26 |
+
For more details, read the paper: [Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2
|
27 |
+
](https://arxiv.org/abs/2311.10702).
|
28 |
|
29 |
|
30 |
## Model description
|
|
|
82 |
If you find Tulu 2 is useful in your work, please cite it with:
|
83 |
|
84 |
```
|
85 |
+
@misc{ivison2023camels,
|
86 |
+
title={Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2},
|
87 |
+
author={Hamish Ivison and Yizhong Wang and Valentina Pyatkin and Nathan Lambert and Matthew Peters and Pradeep Dasigi and Joel Jang and David Wadden and Noah A. Smith and Iz Beltagy and Hannaneh Hajishirzi},
|
88 |
+
year={2023},
|
89 |
+
eprint={2311.10702},
|
90 |
+
archivePrefix={arXiv},
|
91 |
+
primaryClass={cs.CL}
|
92 |
}
|
93 |
```
|
94 |
|