Relevance is a natural offline quality metric for conversational question suggestions. It measures whether a suggestion is on topic and related to the query. An off-topic suggestion is unlikely to provide a user a meaningful conversational search experience. However, as shown in [1]}, the offline relevance metric does not correlate well with user preferences in question suggestion observed in an online A/B test. A suggested question that is more relevant to a user query is not always preferred by the user, e.g., a suggestion that simply paraphrases the query is certainly relevant but does not provide any new information. A good suggestion should be not only relevant but also useful in that it helps further the users’ information need by bringing them to a fruitful next state in their exploration.
