Most commercial LLMs, like ChatGPT (OpenAI, 2023), Gemini (Gemini Team, 2023), and Perplexity.ai already employ some version of RAG in their Web interfaces. For example, ChatGPT employs a Bing search whereas Gemini accesses Google Search results. Though RAG has quickly become a default feature of user-facing LLM systems, most evaluations of LLM capabilities are still performed on the non-RAG counterparts (Zheng et al., 2023). This is problematic, as a model's default and RAG-enabled responses can drastically diverge depending on the quality and accuracy of the retrieved content. This problem is compounded when considering that web results constantly change, and can contain outdated, incorrect, or harmful information (Dash et al., 2023; Daws, 2020; Nastasi et al., 2023). Thus, objective evaluations of RAG-enabled LLM behavior are as important as benchmarking their non-RAG counterparts, especially as RAG systems are increasingly relied upon to provide factual information in a myriad of domains.