|
--- |
|
license: apache-2.0 |
|
datasets: |
|
- openbmb/UltraFeedback |
|
--- |
|
|
|
Original post: [Snorkel link] |
|
|
|
#### Dataset: |
|
ONLY the prompts from [UltraFeedback](https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized); **no external LLM responses used**. |
|
|
|
#### Methodology: |
|
1. Generate five response variations for each prompt from a subset of 20,000 using the LLM - to start, we used [Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2). |
|
2. Apply [PairRM](https://huggingface.co/llm-blender/PairRM) for response reranking. |
|
3. Update the LLM by applying Direct Preference Optimization (DPO) on the top (chosen) and bottom (rejected) responses. |
|
4. Use this LLM as the base model for the next iteration, repeating three times in total. |
|
|
|
This overview provides a high-level summary of our approach. |
|
We plan to release more detailed results and findings in the coming weeks on the [Snorkel blog](https://snorkel.ai/blog/). |
|
|
|
#### Key Premises: |
|
- **Specialization Requirement**: For most enterprise use cases, using LLMs "off-the-shelf" falls short of production quality, necessitating additional fine-tuning and alignment. |
|
- **Ease of Model Building**: Creating ranking/scoring/classification models is simpler than developing high-quality, manually annotated datasets for long-form responses. |
|
- **Programmatic Alignment**: Using smaller but specialized teacher models (reward models) can incrementally align LLMs towards specific axes. We call this **Programmatic Alignment** - capturing domain knowledge in programmatic forms that can be used to guide LLM improvement. |
|
|
|
#### Contemporary Work and Acknowledgements: |
|
|
|
|
|
#### Applications: |
|
Unlike our customers, who have very specific use cases to align LLMs to, |
|
the AlpacaEval 2.0 leaderboard measures the ability of LLMS to follow general user instructions. |
|
Thus, for this demonstration, we use a general-purpose reward model - the performant [PairRM model](https://huggingface.co/llm-blender/PairRM). |
|
We use the [Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2) model as our base LLM. |
|
|
|
With this demonstration, we focus on the general approach of programmatic alignment. |
|
|
|
For interest in building your **specialized internal reward models |
|
that reflect your enterprises' needs**, please contact the Snorkel AI team or consider attending our |
|
[**Enterprise LLM Summit: Building GenAI with Your Data on January 25, 2024**](https://snorkel.ai/event/enterprise-llm-summit/) |
|
to learn more about "Programmatically scaling human preferences and alignment in GenAI". |
|
|
|
#### Result: |
|
On [**Alpaca-Eval 2.0**](https://tatsu-lab.github.io/alpaca_eval/): |
|
- The base model: [Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2) scored **14.72**. |
|
After applying the above methodology: |
|
- This model scored **30.2** - ranked 3rd and the highest for an open-source base model at the time of publication. |
|
- Utilizing the model with PairRM, which involved generating 16 responses and submitting the highest-scoring response by PairRM, we scored **34.86** - ranked 2nd. |
|
The best model on the leaderboard is "gpt-4-turbo", which is also the judge of optimal responses. |
|
|
|
We recognize that the Alpaca-Eval 2.0 benchmark does not entirely capture the full range of capabilities and performances of LLMs. |
|
However, in our current work, where the goal is to align with general "human preferences," Alpaca-Eval 2.0 serves as a suitable and representative benchmark. |
|
Moving forward, we anticipate further contributions from the community regarding new alignment axes, and conduct evaluations using other appropriate benchmarks. |
|
|
|
## Limitations: |
|
The model is a quick demonstration that the LLMs can be programmatically aligned using smaller specialized reward models. |
|
It does not have any moderation mechanisms. |
|
We look forward to continuing to engage with the research community and our customers exploring optimal methods for gettings models to respect guardrails, |
|
allowing for deployment in environments requiring moderated outputs. |
|
|
|
## Acknowledgments: |
|
- The Mistral AI Team for developing and releasing the advanced Mistral-7B-Instruct-v0.2 model. |
|
- The author of the [Direct Preference Optimization paper](https://arxiv.org/abs/2305.18290) for the innovative approach |
|
- The author of the [Pairwise Reward Model for LLMs paper](https://arxiv.org/abs/2306.02561) for the powerful general-purpose reward model |
|
- The HuggingFace team for the DPO implementation under [The Alignment Handbook](https://github.com/huggingface/alignment-handbook) |
|
- We would also like to acknowledge contemporary work published on arXiv a few days ago by Meta & NYU (Yuan, et al) in a paper called [Self-Rewarding Language Models](https://arxiv.org/abs/2401.10020), |
|
which proposes a similar approach for creating alignment pairs from a larger set of candidate responses, but using the LLM as the reward model. |
|
While this may work for general-purpose models, our experience has shown that task-specific reward models guided by SMEs are necessary for most |
|
enterprise applications of LLMs for specific use cases, which is why we focus on the use of external reward models. |
|
|
|
## The Snorkel AI Team |
|
Hoang Tran, Chris Glaze, Braden Hancock |
