Model Card for LION-LLaMA-3-8b-odpo-v1.0

The LION-series are trained using an empirically optimized pipeline that consists of three stages: SFT, DPO, and online preference learning (online DPO). We find simple techniques such as sequence packing, loss masking in SFT, increasing the preference dataset size in DPO, and online DPO training can significantly improve the performance of language models. Our best models (the LION-series) exceed the performance of the official instruct models tuned with closed-source data and algorithms.

For training datasets, code, and evaluation scripts, please refer to our paper and codebase.

Model description

This model is finetuned from Columbia-NLP/LION-LLaMA-3-8b-dpo-v1.0 using online DPO from the LION pipeline.

• Model type: meta-llama/Meta-Llama-3-8B
• Language(s) (NLP): Primarily English
• License: LLaMa-3 Terms of Use
• Finetuned from model: Columbia-NLP/LION-LLaMA-3-8b-dpo-v1.0

Performance

Model	Method	Size	Arena-Hard	AlpacaEval-2	MT-Bench	OpenLLM
LLaMA-3-8b	-	8B	-	-	-	63.05
LLaMA-3-8b-it	SFT+RS+DPO+PPO	8B	20.6	22.9	8.00	68.28
LION-LLaMA-3-8b-sft-v1.0 (ours)	SFT	8B	11.3	17.9	7.58	68.71
LION-LLaMA-3-8b-dpo-v1.0 (ours)	SFT+DPO	8B	19.1	21.8	8.12	71.28
⮕ LION-LLaMA-3-8b-odpo-v1.0 (ours)	SFT+DPO+ODPO	8B	22.0	26.8	8.19	71.41

Intended uses

To ensure reproducibility, please use the following chat templates:

import torch
from transformers import pipeline

pipe = pipeline(
    "text-generation",
    model="Columbia-NLP/LION-LLaMA-3-8b-odpo-v1.0",
    device_map="auto",
    torch_dtype=torch.bfloat16,
)
messages = [
    # no system message for LLaMa
    {
        "role": "user", 
        "content": "Write a short paragraph where every sentence starts with the letter A."
    },
]
outputs = pipe(
    messages,
    max_new_tokens=128,
    do_sample=False,
    stop_sequence="<|im_end|>",
)
print(outputs[0]["generated_text"][-1]["content"])
# Astonishingly adorable animals actively adventure across amazing areas, artists ardently aim at achieving absolute accuracy.
# An array of activities allure adventurous adventurers.
# Always anticipate amazing adventures awaiting all.
# As autumn arrives, apples abound around ample acreages. Architecture amazes, astonishingly adorned, aching for appreciation.
# Altruistic actions always arouse awe.

to inspect the chat template/manually do generation:

tokenizer = AutoTokenizer.from_pretrained("Columbia-NLP/LION-LLaMA-3-8b-odpo-v1.0")
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
print(prompt)
# tokenize prompt and use model.generate

Training details

Please refer to our codebase.

Citation Information

If you find this model useful in your work, please consider citing our paper:

@misc{yu2024lionsempiricallyoptimizedapproach,
      title={LIONs: An Empirically Optimized Approach to Align Language Models}, 
      author={Xiao Yu and Qingyang Wu and Yu Li and Zhou Yu},
      year={2024},
      eprint={2407.06542},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2407.06542}, 
}

Acknowledgements

We thank the Columbia-NLP group and articulate.ai for providing OpenAI API credits and computational resources to conduct our experiments.