Results:We evaluate the performance of each auto-rater configuration by comparing its predictions to the ground-truth labels (_Faithful_ and _Unt faithful_) from our human annotations. Due to the class imbalance, we report separate F1 scores for each label, split across claims generated by different LLMs, in Table 5.9 As a sanity check, the "no evidence" setting performs extremely poorly; more interestingly, human evidence underperforms both retrieval and the entire book setting, suggesting that the LLM requires more context to judge claim validity. The best performing auto-rater is Claude-3-Opus in the entire book setting, which significantly outperforms both GPT-4-Turbo in the same setting as well as BM25.