In the area of social chatbots, a study with large-scale A/B experimentation is [1]}. The paper describes the metric Conversation-turns Per Session (CPS). The more turns we see from a user in a session, the more engaged they are with the social chatbot. They observe that this metric is better used as a long-term metric. Specifically, if the goal is to have a short-term increase in CPS, then the system can just generate responses such as “I don't understand, what do you mean?”, which could entice some users to respond, but would make them like the system less, and reduce CPS in the long term. Similarly, adding some new task completion functionality could reduce CPS in the short term, by helping users complete their task more quickly. However, by making the system more useful and trustworthy, this functionality is likely to increase future engagement in the long term.
