**Simulated Dialogue.** The use of simulated data allowed us to quickly scale training to a broad set of conditions and patient contexts, while the injection of knowledge from search encouraged these dialogues to remain grounded and realistic. Though the simulated patients encompassed a wide range of conditions, they failed to capture the full range of potential patient backgrounds, personalities, and motivations. Through the inner self-play procedure, we were able to iteratively improve the simulated dialogue we generated and used in fine-tuning. However, these improvements were limited by our ability to articulate what makes a good dialogue in the critic instructions, the critic's ability to produce effective feedback, and AMIE's ability to adapt to such feedback. For example, in the simulated environment we impose that AMIE reaches a proposed differential and testing/treatment plan for the patient, but such an endpoint may be unrealistic for some conditions, especially in the virtual chat-based setting.