OpenAssistant Conversations - Democratizing Large
Language Model Alignment
Andreas Köpf
andreas.koepf@provisio.comYannic Kilche
OpenAssistant Conversations -- Democratizing Large Language Model Alignment
Aligning large language models (LLMs) with human preferences has proven to
drastically improve usability and has driven rapid adoption as demonstrated by
ChatGPT. Alignment techniques such as supervised fine-tuning (SFT) and
reinforcement learning from human feedback (RLHF) greatly reduce the required
skill and domain knowledge to effectively harness the capabilities of LLMs,
increasing their accessibility and utility across various domains. However,
state-of-the-art alignment techniques like RLHF rely on high-quality human
feedback data, which is expensive to create and often remains proprietary. In
an effort to democratize research on large-scale alignment, we release
OpenAssistant Conversations, a human-generated, human-annotated assistant-style
conversation corpus consisting of 161,443 messages distributed across 66,497
conversation trees, in 35 different languages, annotated with 461,292 quality
ratings. The corpus is a product of a worldwide crowd-sourcing effort involving
over 13,500 volunteers. To demonstrate the OpenAssistant Conversations
dataset's effectiveness, we present OpenAssistant, the first fully open-source
large-scale instruction-tuned model to be trained on human data. A preference
study revealed that OpenAssistant replies are comparably preferred to
GPT-3.5-turbo (ChatGPT) with a relative winrate of 48.3% vs. 51.7%
respectively. We release our code and data under fully permissive licenses.
Andreas Köpf Yannic Kilcher Dimitri von Rütte Sotiris Anagnostidis Zhi-Rui Tam Keith Stevens Abdullah Barhoum Nguyen Minh Duc Oliver Stanley Richárd Nagyfi Shahul ES Sameer Suri David Glushkov Arnav Dantuluri Andrew Maguire Christoph Schuhmann Huu Nguyen Alexander Mattick