We conduct a human study to verify the reliability of GPT-4's judgments, using the results of the TL:DR summarization experiment and two different GPT-4 prompts. The **GPT-4 (S)** (simple) prompt simply asks for which summary better-summarizes the important information in the post. The **GPT-4 (C)** (concise) prompt also asks for which summary is more concise; we evaluate this prompt because we find that GPT-4 prefers longer, more repetitive summaries than humans do with the **GPT-4 (S)** prompt. See Appendix C.2 for the complete prompts. We perform three comparisons, using the highest (DPO, temp. middle-performing (SFT, temp. 0.25) method with the aim of covering a diversity of sample qualities; all three methods are compared against greedily-sampled PPO (its best-performing temperature). We find that with both prompts, GPT-4 tends to agree with humans about as often as humans agree with each other, suggesting that GPT-4 is a reasonable proxy for human evaluations (due to limited human raters, we only collect multiple human judgments for the DPO and PPO-1 comparisons). Overall, the **GPT-4 (C)** prompt generally provides win rates more representative of humans; we therefore use this prompt for the main results in Section 6.2. For additional details about the human study, including the web interface presented to raters and the list of human volunteers, see Appendix D.3.