As large language models improve, there is increasing interest in techniques that leverage these models’ capabilities to refine their own outputs.
In this work, we introduceShepherd, a language model specifically tuned to critique model responses and suggest refinements, extending beyond the capabilities of an untuned model to identify diverse errors and provide suggestions to remedy them.
At the core of our approach is a high quality feedback dataset, which we curate from community feedback and human annotations.
Even thoughShepherdis small (7B parameters) , its critiques are either equivalent or preferred to those from established models including ChatGPT. Using GPT-4 for evaluation,Shepherdreaches an average win-rate of 53-87% compared to competitive alternatives. In human evaluation,Shepherdstrictly outperforms other models and on average closely ties with ChatGPT.111We released data in this repo:https://github.com/facebookresearch/Shepherd