There are a variety of situations in which a human operator has to answer a set of discrete questions given a corpus of documents containing information pertaining to the questions. One example of such a situation is that in which a human operator is tasked with associating billing codes with a hospital stay of a patient, based on a collection of all documents containing information about the patient's hospital stay. Such documents may, for example, contain information about the medical procedures that were performed on the patient during the stay and other billable activities performed by hospital staff in connection with the patient during the stay.
This set of documents may be viewed as a corpus of evidence for the billing codes that need to be generated and provided to an insurer for reimbursement. The task of the human operator, a billing coding expert in this example, is to derive a set of billing codes that are justified by the given corpus of documents, considering applicable rules and regulations. Mapping the content of the documents to a set of billing codes is a demanding cognitive task. It may involve, for example, reading reports of surgeries performed on the patient and determining not only which surgeries were performed, but also identifying the personnel who participated in such surgeries, and the type and quantity of materials used in such surgeries (e.g., the number of stents inserted into the patient's arteries), since such information may influence the billing codes that need to be generated to obtain appropriate reimbursement. Such information may not be presented within the documents in a format that matches the requirements of the billing code system. As a result, the human operator may need to carefully examine the document corpus to extract such information.
Because of such difficulties inherent in generating billing codes based on a document corpus, various computer-based support systems have been developed to guide human coders through the process of deciding which billing codes to generate based on the available evidence. Despite such guidance, it can still be difficult for the human coder to identify the information necessary to answer each question.
To address this problem, the above-referenced patent application entitled, “Providing Computable Guidance to Relevant Evidence in Question-Answering Systems” (U.S. patent application Ser. No. 13/025,051) discloses various techniques for pointing the human coder to specific regions within the document corpus that may contain evidence of the answers to particular questions. The human coder may then focus initially or solely on those regions to generate answers, thereby generating such answers more quickly than if it were necessary to review the entire document corpus manually. The answers may themselves take the form of billing codes or may be used, individually or in combination with each other, to select billing codes.
For example, an automated inference engine may be used to generate billing codes automatically based on the document corpus and possibly also based on answers generated manually and/or automatically. The conclusions drawn by such an inference engine may, however, not be correct. What is needed, therefore, are techniques for improving the accuracy of billing codes and other data generated by automated inference engines.