In terms of application-oriented research, our understanding of how to measure and optimize long-term engagement and satisfaction in a variety of settings is evolving. In general chatbot settings, there is a need to continue to define long-term evaluation metrics for engagement that go beyond conversational-turns per session (CPS) and Number of Active Users (NAU) and consider task-completion, goal-support, etc. For example, recommendation technology increasingly is explored in aspirational goal-setting (e.g. “I want to lose weight”, “I want to eat healthy”, “I want to learn more about deep learning”) – clearly in these settings achieving the goal while creating engagement to get users to return must be considered (presumably being more likely to return also makes the user more likely to achieve their goal). Similarly when conversational agents are primarily providing social companionship, there are theories of linguistic accommodation or coordination [1]} which describe human-human interactions and may form a key part of metrics to know when AI-based social companions are performing well.
