Conclusion:Despite it having the best performance in Table 5, Claude-3-Opus ultimately performs too poorly to be a reliable auto-rater (47.5 F1 when classifying _Unt faithful_ claims). This comes as a surprise as this pattern of decompose-then-verify has been shown to correlate with human judgments in other settings, like Min et al. (2023). Manual analysis of the errors reveals that Claude-3-Opus struggles most with claims involving non-narrative information (23.1%), assessments often based on common sense reasoning (20.5%), and character confusions (12.8%), which often require a deep understanding of the entire book; see confusion matrix in Figure 10 and examples in Figure 3. Qualitatively, we can also gauge from annotator comments (Table 4) the difficulty of this claim verification task as evidence may be difficult to localize (in "needle-in-the-haystack" manner) and require full document reasoning.