Figure 2: Across six QA datasets using GPT-4, we consistently observe an inverse relationship between the RAG preference rate (y-axis) and two characteristics (x-axes): 1. the model’s prior response probability (lefthand plots), and 2. the amount of deviation from the prior (righthand plots). RAG preference rate is defined as the proportion of responses that align with the information presented in the prompt as context. The model’s prior response probability is computed from the average log probability of the response tokens queried without RAG. The left plot in each pair visualizes the prior probability (grouped into 10 bins) against the RAG preference rate, along with the best-fit trend line and slope. The right plot visualizes absolute deviation from the reference information (for numerical datasets (top), up to two log-fold changes (along with the trendline); for categorical datasets (bottom), a total of four modification categories) against RAG preference rate. Additionally, the upper and lower half percentiles are shown in the right plots to illustrate that lower probability prior responses have monotonically lower RAG preference rates than higher probability prior responses.