Although it is not common to evaluate CIR using user simulation, in related areas of conversational AI there are already user simulation methods. As summarized by [1]} simulators for dialog agents can be based on randomly generated scenarios. A scenario has two parts. One is the things that the user does not yet know, for example to book a movie ticket, the user wants to find out the ticket, theater and start time. The other is the user's constraints, that she wants to book for three people, tomorrow and for the movie “batman vs superman”. The simulation can then be carried out using a rule-based approach, where the simulator represents the users “state of mind” using a stack data structure known as the user agenda. The simulation can also use a model-based approach, which still has the user's goal and constraints, but trains a sequence-to-sequence model for simulation. Simulators, particularly rule-based ones, can incorporate domain knowledge in the process. For example, the user must always have a number of people in mind, and that number of people should be greater than zero but not too large.
