**Tasks.** Our experiments explore three different open-ended text generation tasks. For all experiments, algorithms learn a policy from a dataset of preferences \(\mathcal{D}=\left\{x^{(i)},y_{w}^{(i)},y_{l}^{(i)}\right\}_{i=1}^{N}\). In **controlled sentiment generation**, \(x\) is a prefix of a movie review from the IMDb dataset [22], and the policy must generate \(y\) with positive sentiment. In order to perform a controlled evaluation, for this experiment we _generate_ preference pairs over generations using a pre-trained sentiment classifier, where \(p\)(positive \(\mid x,y_{w}\rangle>p\)(positive \(\mid x,y_{l}\rangle\). For SFT, we fine-tune GPT-2-large until convergence on reviews from the train split of the IMDB dataset (further details in App C.1). In **summarization**, \(x\) is a forum post from Reddit; the policy must generate a summary \(y\) of the main points in the post. Following prior work, we use the Reddit TL;DR summarization dataset [41] along with human preferences gathered by Stiennon et al.. We use an SFT model fine-tuned on human-written forum post summaries2 with the TRLX [42] framework for RLHF. The human preference dataset was gathered by Stiennon et al. on samples from a different, but similarly-trained, SFT model. Finally, in **single-turn dialogue**, \(x\) is a human query, which may be anything from a question about astrophysics to a request for relationship advice. A policy must produce an engaging and helpful response \(y\) to a user's query; we use the Anthropic Helpful and Harmless dialogue dataset [1], containing 170k dialogues between a human and an automated assistant. Each transcript ends with a pair of responses generated by a large (although unknown) language model along with a preference label denoting the human-preferred response. In this setting, no pre-trained SFT model is available; we therefore fine-tune an off-the-shelf language model on only the preferred completions to form the SFT model.