Detecting adverse events for patient safety research: a review of current methodologies <s> Combined modalities <s> Quality screening and management (QSM), developed by United HealthCare Corporation for its own health plans but applicable to other settings, analyzes health care provided to an enrolled population using claims and administrative data supplemented, when necessary, with medical record review. For selected topics/conditions, QSM compares the care received by patients to that recommended by established practice guidelines and also reports and describes quality through incidence rates, condition-specific process and outcome measures, occurrence of adverse events, and use of preventive services. Results of the analyses are linked to appropriate quality management actions and are used to identify strategies for improvement. Data permit health plans to make performance comparisons on individual indicators and longitudinal comparisons on the effectiveness of quality improvement initiatives. <s> BIB001 </s> Detecting adverse events for patient safety research: a review of current methodologies <s> Combined modalities <s> Applying a computerized algorithm to administrative data to help assess the quality of hospital care is intriguing. As Iezzoni and colleagues point out, there are major differences of opinion as to the worth of such efforts. This article significantly advances the state of the art in using administrative data to screen for potential quality-of-care problems. In addition, this work on identifying complications of care goes well beyond the emphasis of many government organizations on hospital mortality rates. One question, however, not raised in the paper is: What is a practical upper limit to the sensitivity and specificity in comparing computerized screen results with the consensus judgments of a group of independent physicians? Advanced statistical techniques (such as bootstrapping) might be used to estimate the stability of consensus judgments by physician groups. When the judgments of two groups of physicians are compared with each other, the resulting sensitivity and specificity will not be .99! In addition, more training of members of the physician panels would probably have increased interrater reliability. While acknowledging this problem, the researchers' detailed analysis of the panel results is intriguing and represents a model for such studies. It is hoped that the authors will follow up on the avenues opened here. Furthermore, what degree of accuracy is necessary to identify facilities with higher-than-expected rates of complications? The authors discuss problems involved in using administrative data to target hospitals and departments for more costly in-depth reviews of quality. It is hoped that the promising findings that are reported here will be validated in other studies. Certainly their algorithms should find a ready audience in insurers and hospitals willing to try them out. Finally, should we expect additional research to lead to improvement in the authors' algorithms? I believe the algorithms will prove difficult to improve upon; but perhaps we should not worry about this. At some point, however, the cost of trying to identify and correct quality problems in "minimally outlier" hospitals will exceed the benefits, particularly given alternative uses for the funds. Might we now be close the the "flat of the curve" in the development of such systems for identification of quality problems? This issue should be discussed much further in future studies. <s> BIB002 </s> Detecting adverse events for patient safety research: a review of current methodologies <s> Combined modalities <s> OBJECTIVE ::: To determine the suitability of insurance claims information for use in clinical outcomes research in ischemic heart disease. ::: ::: ::: DESIGN ::: Concordance study of two databases. ::: ::: ::: SETTING ::: Tertiary care referral center. ::: ::: ::: PATIENTS ::: A total of 12,937 consecutive patients hospitalized for cardiac catheterization for suspected ischemic heart disease between July 1985 and May 1990. ::: ::: ::: INTERVENTIONS ::: Two-by-two tables were used to compute overall and kappa measures of agreement comparing clinical versus claims data for 12 important predictors of prognosis in patients with ischemic heart disease. ::: ::: ::: MEASUREMENTS ::: Kappa statistics (agreement adjusted for chance agreement) were used to quantify agreement rates. ::: ::: ::: RESULTS ::: Agreement rates between the clinical and claims databases ranged from 0.83 for the diagnosis of diabetes to 0.09 for the diagnosis of unstable angina (kappa values). Claims data failed to identify more than one half of the patients with prognostically important conditions, including mitral insufficiency, congestive heart failure, peripheral vascular disease, old myocardial infarction, hyperlipidemia, cerebrovascular disease, tobacco use, angina, and unstable angina, when compared with the clinical information system. ::: ::: ::: CONCLUSIONS ::: Our results suggest that insurance claims data lack important diagnostic and prognostic information when compared with concurrently collected clinical data in the study of ischemic heart disease. Thus, insurance claims data are not as useful as clinical data for identifying clinically relevant patient groups and for adjusting for risk in outcome studies, such as analyses of hospital mortality. <s> BIB003 </s> Detecting adverse events for patient safety research: a review of current methodologies <s> Combined modalities <s> The Complications Screening Program (CSP) is a method using standard hospital discharge abstract data to identify 27 potentially preventable in-hospital complications, such as post-operative pneumonia, hemorrhage, medication incidents, and wound infection. The CSP was applied to over 1.9 million adult medical/surgical cases using 1988 California discharge abstract data. Cases with complications were significantly older and more likely to die, and they had much higher average total charges and lengths of stay than other cases (P < 0.0001). For most case types, 13 chronic conditions, defined using diagnosis codes, increased the relative risks of having a complication after adjusting for patient age. Cases at larger hospitals and teaching facilities generally had higher complication rates. Logistic regression models to predict complications using demographic, administrative, clinical, and hospital characteristics variables, had modest power (C statistics = 0.64 to 0.70). The CSP requires further evaluation before using it for purposes other than research. <s> BIB004 </s> Detecting adverse events for patient safety research: a review of current methodologies <s> Combined modalities <s> The goals of this study were to evaluate the sensitivity and specificity of 15 screening criteria for adverse events, preventable adverse events, and severe adverse events in medical patients, and to evaluate combinations of these criteria, including those available through hospital billing data, to determine whether a small subset of generic screens might efficiently identify adverse events. The authors studied 3,137 consecutive admissions to a medical service over a 4-month period at an urban tertiary care hospital. Chart reviews were performed after discharge by reviewers blinded to the eventual determination of presence of an adverse event. Judgments regarding presence, severity, and preventability of adverse events were made using guided implicit reviews by physicians. Of all admissions, 341 (11%) were judged to include an adverse event, of which 274 were severe and 145 were preventable. Sensitivity and specificity of individual screens varied widely, with prior hospitalization the most sensitive (68%) but least specific (56%). Death was specific (97%) but not sensitive (9%); readmission was intermediate (sensitivity 28%, specificity 80%). In analyses using severe and preventable adverse events as the outcome, results were generally similar. Combinations of screens also were compared, including some using only screens available through billing data; the most sensitive billing strategy detected just 47% of adverse events, but cost only $3 per admission reviewed and $57 per adverse event, versus $13 per admission and $116 per adverse event for a strategy in which all records were reviewed. It is concluded that no small subset of screens identified a high percentage of adverse events. Using screens available through billing data, although insensitive, would be much less costly. <s> BIB005 </s> Detecting adverse events for patient safety research: a review of current methodologies <s> Combined modalities <s> Purpose of this study was to identify variables near hospital admission that could identify patients at risk for placement, death, or readmission. The goals were to assess sensitivity and specificity of screening strategies in predicting adverse outcomes that deter or affect home health care. We evaluated whether generic screens might efficiently identify type of outcome. Criteria reported in the literature were used to predict outcomes for 1,332 admissions. Factors that discriminated type of outcome included comorbidity, mental status, living arrangement, transfer to special care, prior admission, iatrogenic trauma, and pending litigation. We conclude risk scores can accurately predict outcome of hospital treatment, which may be useful in targeting patients for intervention. Using billing data, although rather insensitive, was the most cost-effective strategy. <s> BIB006 </s> Detecting adverse events for patient safety research: a review of current methodologies <s> Combined modalities <s> Objectives. This study examined the validity of the Complications Screening Program (CSP) by testing whether (1) ICD-9-CM codes used to identify a complication are coded completely and accurately and (2) the CSP algorithm successfully separates conditions present on admission from those occurring in the hospital. Methods. We compared diagnosis and procedure codes contained in the Medicare claim with codes abstracted from an independent re-review of more than 1,200 medical records from Connecticut and California. Results. Eighty-nine percent of the surgical cases and 84% of the medical cases had their CSP trigger codes corroborated by re-review of the medical record. For 13% of the surgical cases and 58% of the medical cases, the condition represented by the code was judged to be present on admission rather than occurring in-hospital. The positive predictive value of the claim was greater than 80% for the surgical risk pool, suggesting the value of the CSP as a screening tool. Conclusions. The CSP has validity as a screen for most surgical complications but only for 1 medical complication. The CSP does not have validity as a "stand-alone" tool to identify more than a few in-hospital surgery-related events. The addition of an indicator to the Medicare claim to capture the timing of secondary diagnoses would improve the validity of the CSP for identifying both surgical and medical in-hospital events. <s> BIB007 </s> Detecting adverse events for patient safety research: a review of current methodologies <s> Combined modalities <s> BACKGROUND ::: Hospital discharge diagnoses, coded by use of the International Classification of Diseases, 9th Revision, Clinical Modification (ICD-9-CM), increasingly determine reimbursement and support quality monitoring. Prior studies of coding validity have investigated whether coding guidelines were met, not whether the clinical condition was actually present. ::: ::: ::: OBJECTIVE ::: To determine whether clinical evidence in medical records confirms selected ICD-9-CM discharge diagnoses coded by hospitals. ::: ::: ::: RESEARCH DESIGN AND SUBJECTS ::: Retrospective record review of 485 randomly sampled 1994 hospitalizations of elderly Medicare beneficiaries in Califomia and Connecticut. ::: ::: ::: MAIN OUTCOME MEASURE ::: Proportion of patients with specified ICD-9-CM codes representing potential complications who had clinical evidence confirming the coded condition. ::: ::: ::: RESULTS ::: Clinical evidence supported most postoperative acute myocardial infarction diagnoses, but fewer than 60% of other diagnoses had confirmatory clinical evidence by explicit clinical criteria; 30% of medical and 19% of surgical patients lacked objective confirmatory evidence in the medical record. Across 11 surgical and 2 medical complications, objective clinical criteria or physicians' notes supported the coded diagnosis in >90% of patients for 2 complications, 80% to 90% of patients for 4 complications, 70% to <80% of patients for 5 complications, and <70% for 2 complications. For some complications (postoperative pneumonia, aspiration pneumonia, and hemorrhage or hematoma), a large fraction of patients had only a physician's note reporting the complication. ::: ::: ::: CONCLUSIONS ::: Our findings raise questions about whether the clinical conditions represented by ICD-9-CM codes used by the Complications Screening Program were in fact always present. These findings highlight concerns about the clinical validity of using ICD-9-CM codes for quality monitoring. <s> BIB008 </s> Detecting adverse events for patient safety research: a review of current methodologies <s> Combined modalities <s> INTRODUCTION ::: In inpatients, computer monitors have been used to improve the detection of adverse drug events (ADEs). However, similar programs have not been available in outpatients. ::: ::: ::: OBJECTIVE ::: To describe an approach for detecting incidents suggesting that an ADE may have occurred in outpatients by adapting methods from inpatient computer monitoring and developing terminology searches of electronic medical records. ::: ::: ::: METHODS ::: One year of information from the outpatient electronic medical record (EMR) at one hospital and its clinics was reviewed. Altogether, 23064 patients and 88514 visits were identified. Patient demographics, medical problem lists, ICD-9 claims, patient allergies, medication history and all clinic visit notes were extracted and merged. We then searched for incidents suggesting that an ADE might be present using four methods: ICD-9 claims, new allergies, computer rules linking laboratory data to known medication exposures, and a medical terminology lexicon (M2D2). In this report, we describe how these search methods were developed to allow for ADE identification. ::: ::: ::: CONCLUSION ::: The ability to carry out such quality-related work is an example of the benefits of the outpatient EMR that may not be apparent to those institutions considering adopting it. <s> BIB009 </s> Detecting adverse events for patient safety research: a review of current methodologies <s> Combined modalities <s> BACKGROUND ::: It is uncertain whether lower levels of staffing by nurses at hospitals are associated with an increased risk that patients will have complications or die. ::: ::: ::: METHODS ::: We used administrative data from 1997 for 799 hospitals in 11 states (covering 5,075,969 discharges of medical patients and 1,104,659 discharges of surgical patients) to examine the relation between the amount of care provided by nurses at the hospital and patients' outcomes. We conducted regression analyses in which we controlled for patients' risk of adverse outcomes, differences in the nursing care needed for each hospital's patients, and other variables. ::: ::: ::: RESULTS ::: The mean number of hours of nursing care per patient-day was 11.4, of which 7.8 hours were provided by registered nurses, 1.2 hours by licensed practical nurses, and 2.4 hours by nurses' aides. Among medical patients, a higher proportion of hours of care per day provided by registered nurses and a greater absolute number of hours of care per day provided by registered nurses were associated with a shorter length of stay (P=0.01 and P<0.001, respectively) and lower rates of both urinary tract infections (P<0.001 and P=0.003, respectively) and upper gastrointestinal bleeding (P=0.03 and P=0.007, respectively). A higher proportion of hours of care provided by registered nurses was also associated with lower rates of pneumonia (P=0.001), shock or cardiac arrest (P=0.007), and "failure to rescue," which was defined as death from pneumonia, shock or cardiac arrest, upper gastrointestinal bleeding, sepsis, or deep venous thrombosis (P=0.05). Among surgical patients, a higher proportion of care provided by registered nurses was associated with lower rates of urinary tract infections (P=0.04), and a greater number of hours of care per day provided by registered nurses was associated with lower rates of "failure to rescue" (P=0.008). We found no associations between increased levels of staffing by registered nurses and the rate of in-hospital death or between increased staffing by licensed practical nurses or nurses' aides and the rate of adverse outcomes. ::: ::: ::: CONCLUSIONS ::: A higher proportion of hours of nursing care provided by registered nurses and a greater number of hours of care by registered nurses per day are associated with better care for hospitalized patients. <s> BIB010 </s> Detecting adverse events for patient safety research: a review of current methodologies <s> Combined modalities <s> OBJECTIVE ::: Detecting adverse events is pivotal for measuring and improving medical safety, yet current techniques discourage routine screening. The authors hypothesized that discharge summaries would include information on adverse events, and they developed and evaluated an electronic method for screening medical discharge summaries for adverse events. ::: ::: ::: DESIGN ::: A cohort study including 424 randomly selected admissions to the medical services of an academic medical center was conducted between January and July 2000. The authors developed a computerized screening tool that searched free-text discharge summaries for trigger words representing possible adverse events. ::: ::: ::: MEASUREMENTS ::: All discharge summaries with a trigger word present underwent chart review by two independent physician reviewers. The presence of adverse events was assessed using structured implicit judgment. A random sample of discharge summaries without trigger words also was reviewed. ::: ::: ::: RESULTS ::: Fifty-nine percent (251 of 424) of the discharge summaries contained trigger words. Based on discharge summary review, 44.8% (327 of 730) of the alerted trigger words indicated a possible adverse event. After medical record review, the tool detected 131 adverse events. The sensitivity and specificity of the screening tool were 69% and 48%, respectively. The positive predictive value of the tool was 52%. ::: ::: ::: CONCLUSION ::: Medical discharge summaries contain information regarding adverse events. Electronic screening of discharge summaries for adverse events using keyword searches is feasible but thus far has poor specificity. Nonetheless, computerized clinical narrative screening methods could potentially offer researchers and quality managers a means to routinely detect adverse events. <s> BIB011
Combined modalities are methods for detecting adverse events that rely on both electronic and manual review process. In general, these systems identify an electronically stored ''signal,'' such as a laboratory abnormality, use of a medication along with development of a symptom, as screening criteria to identify charts for further review. These systems generally require less reviewer time and thus are less expensive to operate than manual systems. However, as a result of low specificity, much of these approaches still require some form of manual review. A few of these screening methods have evolved into stand-alone systems that no longer require any form of manual review. We begin with a review of adverse event detection systems that utilize a single data source, such as administrative data, as a signal, followed by more sophisticated systems that use multiple data sources, such as pharmacy and laboratory databases, and rule-based alert logic to detect adverse events. 3.1. Adverse event detection system 3.1.1. Single data source As discussed above, chart reviews have relied on a two-step process, which initially involves a screening review followed by physician review. However, many of the criteria used for the initial screening process could be electronically captured. Several investigators have evaluated the test characteristics of generic screens obtainable through administrative or billing data. In one study, five generic screens were used to attempt to screen for adverse events. Overall these five triggers had a positive predictive value of 20% and were able to detect 50% of all AE occurring in the cohort BIB005 . Removing a poorly performing billing screen resulted in an overall improvement in the positive predictive value (30%) but detected only about half as many AEs as the first method. Other investigators have had similar results with positive predictive values of the generic screens ranging from 12 to 34% BIB006 . Despite these low positive predictive values, strategies for adverse event detection that incorporate these electronically captured generic screens do cost significantly less per adverse event detected BIB005 making them an attractive alternative to pure manual review. The Veterans Affairs Hospital has developed an occurrence-screening program that utilizes similar electronically captured generic screens to search for events that might require further peer review  . Iezzoni developed a computer algorithm called the Complications Screening Program (CSP), which searched discharge abstracts for potential complications, using ICD-9-CM codes as a signal BIB004 BIB002 . For surgical event screens, 73% were corroborated on chart review. However, only 32% corroborated for the medical event screens BIB007 . The major reason for the poor performance of the medical event screens resulted from the toolÕs inability to distinguish diagnoses that were present on admission from those that occurred during the hospitalization. Of the screens detected for the medical cases, 58% were present on hospital admission as opposed to only 13% in the surgical patients. As a result of these findings the authors did not advise using the tool as a ''stand-alone'' test for detecting most complications, though it still may be useful with follow-up review. Regardless, ICD-9-CM codes have been utilized to detect medical adverse events in recent publications BIB010 , and are used by many commercial organizations to detect complications [51, BIB001 . ICD-9-CM codes have also been used to detecting adverse events in outpatient care, but identified only a small number of events, in part because adverse event codes were rarely used in this setting BIB009 . There are several limitations to using ICD-9-CM codes as a search strategy. First ICD-9-CM codes are generated for reimbursement purposes rather than clinical care and may not be appropriate for clinical studies BIB003 . Second, ICD-9-CM codes frequently are erroneously assigned. In one study designed to determine whether clinical evidence could be found within the chart to support the ICD-9-CM codes found in the discharge abstract, only 70% of medical patient and 81% of surgical patients had objective evidence to support the code within their record BIB008 . Finally, as mentioned above, ICD-9-CM codes do not discriminate between pre-existing and newly documented diagnoses. One possibility means for overcoming the limitations of using ICD-9-CM coding to detect possible adverse events is to use clinical narratives as a data source. Clinical narratives contain rich information yet the uncoded nature of this data has made it challenging to develop detection systems. Methods have been developed for both inpatient and outpatient clinical narratives. Initial approaches to detect adverse events within inpatient clinical narrative have focused on discharge summaries. In one study, simple keyword searches were used to screen for adverse events BIB011 . Discharge summaries were parsed into individual words and queried for ''trigger words'' that represented adverse event concepts. An example of a ''trigger word'' was agitation. Of the 424 selected discharge summaries for screening, 3 contained the ''trigger word'' agitation and all three had an adverse event revealed on chart review. A total of 95 trigger words were included. Overall, the electronic tool had a positive predictive value of 52% (95% CI 46-58%) but had poor specificity (48%, 95% CI 42-54%). Specificity could have been markedly improved by adding a manual discharge summary review step to the screening process. However, the addition of this step would offset some of the time gains made through electronic screening. Addressing the negation terms and lexical variants that resulted from keyword queries could eliminate some false positives, however to make substantial improvements in the tool, more sophisticated searching techniques, such as natural language processing, would be required BIB011 .