In the Xiaoice social chatbot [1]} a number of areas were identified for ongoing study. Privacy is a concern, since users reveal information about themselves to the conversational system. This is certainly true in Web search as well, where users reveal their thoughts to the engine in the form of queries, but could happen even more if the conversation with the agent elicits further details or even (in the case of a chatbot) is treated as a friend. Another concern is allowing Xiaoice to suggest not conversing or even refuse a conversation, either because it is late at night or because the topic is inappropriate. Another concern is that the system should make it clear what it can and cannot do, admitting that it is fallible, because users who trust the system too much could make a mistake. The overall concern is how all the possible impacts on human users can be correctly evaluated and optimized in a machine learning based system.
