**Can faithfulness be evaluated automatically? (SS4)** Collecting human annotations on 26 books cost us **SS5.2K**, demonstrating the difficulty of scaling our workflow to new domains and datasets. We thus implement multiple LLM-based raters of faithfulness, following prior work such as BooookScore (Chang et al., 2023b) and FactScore (Min et al., 2023) that achieve high correlation with human judgments. However, all of our metric configurations struggle to reliably identify unfaithful claims. Our best-performing method operates similarly to "needle-in-the-haystack"-style evaluations (Kamradt, 2023; Gemini Team, 2024) by feeding as much of the book as possible into a long-context LLM along with a single claim to verify. We promote this claim-level verification task as both important for book-length summarization evaluation as well as a challenging benchmark for long-context understanding.