Note Claiming that there is a strong bias towards utilizing only the context information to answer the question, while relying minimally on their parametric memory. Checked for Llama and Phi.

Note Evaluating long-context LLMs and RAG systems, by introducing the SummHay benchmark task: summarize large sets of documents - cover and cite insights, in the context of a particular query. 1. RAG systems typically improve citation quality at the cost of insight coverage. 2. Using advanced RAG components (e.g., Cohere’s Rerank3) leads to end-to-end performance boosts. 3. Confirms the "lost in the middle" phenomenon - LLMs are biased towards the top or bottom of the context window.