[1]} propose a new usefulness metric, which goes beyond relevance and measures whether the suggestion for a query brings real value to a user, e.g., new information she needs, the next step to complete a task, or helping her ask the right questions to explore a topic. [1]} define the guidelines of the usefulness metric, label query-suggestion suggestion paired sampled from Bing search logs, as exemplified in Table REF , and report five failure modes as categories of not useful which help diagnose the common challenges of CIR systems.
<TABLE>