While RAG is becoming standard practice in commercially available LLMs, the reliability of such systems is still understudied. Our experiments uncover several mechanisms that modulate the degree to which LLMs adhere to RAG systems.