When prompting language models to complete a task, users often leave important aspects unsaid. While asking questions could resolve this ambiguity, models often struggle to ask good questions. We explore a language model’s ability to self-improveby rewarding the model for generating useful questions—a simple method we dub STaR-GATE. We generate a synthetic dataset of 25,500 unique persona-task prompts to simulate conversations between a pretrained language model—theQuestioner—and aRoleplayerwhose preferences are unknown to theQuestioner. By asking questions, theQuestionerelicits preferences from theRoleplayer. TheQuestioneris iteratively finetuned on questions that increase the probability of high-quality responses to the task, which are generated by anOraclewith access to theRoleplayer’s latent preferences. After two iterations of self-improvement, theQuestionerasks better questions, allowing it to generate responses that are preferred over responses from the initial model on72%of tasks. Our results indicate that teaching a language model to ask better questions leads to better personalized responses.