Candidates for OEC in Web search were considered in [1]}, confirming that signals relating to clicks and dwell time are still valid. That study also demonstrated that higher quality Web results are associated with shorter sessions. The assumption is that users have a fixed task, and if the results are worse they need to try more queries to get the same results. The discrepancy with CPS, which rewards longer sessions, could be because Web search is often task-oriented, which is different from a conversational chatbot that encourages longer interaction. To resolve which approach works best for CIR, it would be possible to consider the long-term effects of different A/B hypotheses, and see which short-term CIR user behaviors (such as longer or shorter sessions) best correlate with long-term engagement.
