Self-supervised language models of increasing scale learn to complete some tasks zero-shot [31] or with few-shot prompts [6; 25; 11]. However, their performance on downstream tasks and alignment with user intent can be significantly improved by fine-tuning on datasets of instructions and human-written completions [23; 36; 13; 39]. This 'instruction-tuning' procedure enables LLMs to generalize to instructions outside of the instruction-tuning set and generally increase their usability [13]. Despite the success of instruction tuning, _relative_ human judgments of response quality are often easier to collect than expert demonstrations, and thus subsequent works have fine-tuned LLMs with datasets of human preferences, improving proficiency in translation [18], summarization [38; 49], story-telling [49], and instruction-following [26; 32].