[1]} compared search using a text interface where the CIR system is a real system, a human expert or a human posing as a CIR system. They found that users do not have a bias against conversational systems, but the real-world CIR system was not up to the task of searching for complex information needs. Specifically the known human and human CIR system performed similarly on overall satisfaction, ability to find information and topical quiz success, while the automatic system performed significantly worse on all three counts. The automatic system would fail to maintain context and misunderstand partially-stated questions. The automatic system in this study would give a URL with each result, which the users preferred to the human and human CIR system, who did not. When dealing with the human, people felt uncomfortable asking too many questions. Feedback was found to be important.
