Recent AI-assistant agents, such as ChatGPT, predominantly rely on supervised fine-tuning (SFT)  with human annotations and reinforcement learning from human feedback (RLHF)  to align the output of large language models (LLMs)  with human intentions, ensuring they are helpful, ethical, and reliable. However, this dependence can significantly constrain the true potential of AI-assistant agents due to the high cost of obtaining human supervision and the related issues on quality, reliability, diversity, self-consistency, and undesirable biases. To address these challenges, we propose a novel approach calledSelf-Align, which combines principle-driven reasoning and the generative power of LLMs for the self-alignment of the AI agents with minimal human supervision. ApplyingSelf-Alignto theLLaMA-65bbase language model, we develop an AI assistant namedDromedary. With fewer than300 lines of human annotations(includingseed prompts, 16 generic principles, and 5 exemplars for in-context learning) ,Dromedarysignificantly surpasses the performance of several state-of-the-art AI systems, includingText-Davinci-003andAlpaca, on benchmark datasets with various settings.
We have open-sourced the code, LoRA weights ofDromedary, and our synthetic training data to encourage further research into aligning LLM-based AI agents with enhanced supervision efficiency, reduced biases, and improved controllability.