|
--- |
|
license: apache-2.0 |
|
datasets: |
|
- stanfordnlp/SHP |
|
- Anthropic/hh-rlhf |
|
- OpenAssistant/oasst1 |
|
language: |
|
- en |
|
metrics: |
|
- accuracy |
|
tags: |
|
- human feedback |
|
- rlhf |
|
- preferences |
|
- alignment |
|
- HALO |
|
- halos |
|
- dpo |
|
- rl |
|
--- |
|
|
|
 |
|
|
|
This repo contains the model checkpoints for: |
|
- model family <b>pythia6-9b</b> |
|
- optimized with the loss <b>KTO</b> |
|
- aligned using the SHP, Anthropic HH and Open Assistant datasets. |
|
|
|
To prompt archangel models, ensure that the format is consistent with that of TuluV2, i.e. `"<s>\n<|user|>\n" + <prompt> + "\n<|assistant|>\n</s>"`. |
|
Note that the BOS / EOS tokens should be excluded if automatically added by your tokenizer during batch collation. |
|
|
|
Please refer to our [code repository](https://github.com/ContextualAI/HALOs) or [blog](https://contextual.ai/better-cheaper-faster-llm-alignment-with-kto/) which contains intructions for training your own HALOs and links to our model cards. |
|
|
|
If you find this repo or the technical paper useful in your research, please feel free to cite [our work](https://github.com/ContextualAI/HALOs/blob/main/assets/report.pdf): |
|
``` |
|
@techreport{ethayarajh2023halos, |
|
author = {Ethayarajh, Kawin and Xu, Winnie, and Jurafsky, Dan and Kiela, Douwe}, |
|
title = {Human-Centered Loss Functions (HALOs)}, |
|
institution = {Contextual AI}, |
|
note = {https://github.com/ContextualAI/HALOs/blob/main/assets/report.pdf}, |
|
year = {2023}, |
|
} |
|
``` |
