Clarification evaluation based on reference clarifications, from forums or log data, has the problem that the CIR system may generate an equally good clarification question that is different from the reference. One solution is to use human annotation, to identify which clarifying questions seem most appropriate and useful [1]}. The approach was to collect clarifying questions relating to different facets of the query, based on crowdsourcing. The task of selecting a clarifying question can then be framed as a question retrieval problem.
