The issue of hallucination in LLMs has been explored in multiple contexts and models (Ji et al., 2023; Kaddour et al., 2023). As a response, RAG systems have been shown to reduce hallucination (Shuster et al., 2021; Kang et al., 2023). Previous works have explored automated RAG evaluation frameworks in various settings (Es et al., 2023; Hoshi et al., 2023; Saad-Falcon et al., 2023; Zhang et al., 2024). For example, some studies use LLMs to evaluate the faithfulness, answer relevance, and context relevance of RAG systems by using GPT-3.5 as an evaluator (Es et al., 2023; Saad-Falcon et al., 2023). In another study, the authors propose metrics such as noise robustness, negative rejection, information integration, and counterfactual robustness (Chen et al., 2024). Multiple studies have shown that RAG can mislead LLMs in the presence of complex or misleading search results and that such models can still make mistakes even when given the correct response (Foulds et al., 2024; Shuster et al., 2021). In relation to understanding model priors, other works have used log probabilities to assess the LLM's confidence in responses (Mitchell et al., 2023; Zhao et al., 2024). However, so far there has not been a systematic exploration of a model's confidence (via logprobs) and the model's preference for RAG-provided information.