In general, in summarization and also generation of clarifying question, it is possible for the CIR system to do natural language generation. There are three main methods for evaluating such generation [1]}. One is to involve humans, asking them if the generated text is good. The main drawback is that the CIR system may generate new text that a human has never seen before, so additional human input is needed. The second method is to compare the generated texts to a reference text, using a metric based on simple similarity metrics such as string overlap. The reference text may be human-generated and/or human-annotated. The main drawback is that the CIR system may generate text that is too different from the references, but is still valid, which would lead it to being penalized. The third approach is to use machine learned metrics to play the role of the human judge. The main drawback would be if the metric does not do a perfect job of simulating the human.
