Network Meta analysis for Clinical Practice Guidelines: A Case Study on First Line Medical Therapies for Primary Open Angle Glaucoma
In 2011, the Institute of Medicine defined clinical practice guidelines as statements that include recommendations intended to optimize patient care, that are informed by a systematic review of evidence and an assessment of the benefits and harms of alternative care options (1) Historically, guidelines primarily represented the opinions of individual authors or the consensus of experts (2) With the advent of evidence based health care, guidelines have increasingly used systematic reviews and meta analyses of randomized, controlled trials (RCTs) to form the basis of recommendations (24) Standard meta analytic techniques can be used if the guideline addresses pairwise comparisonsfor example, treatment A versus treatment B. If a guideline is attempting to address the question of which treatment is best among multiple options, however, standard meta analysis may not be adequate. By contrast, network meta analysisa method that uses information from both direct and indirect comparisons and makes inferences about the comparative effectiveness of all the treatments of interest in a single analysis (5, 6)is particularly suited in such situations. Clinical conditions for which guidelines could benefit from network meta analysis the most are those with numerous treatment options, such as first line medical treatment of primary open angle glaucoma (POAG) In this condition, which is highly prevalent in the United States and worldwide, optic nerve damage leads to gradual and painless visual field reduction over time (7, 8) Because optic nerve damage is difficult to measure and changes in visual field take years to develop, treatment effectiveness is generally determined by reduction in intraocular pressure (IOP) a modifiable risk factor for POAG over a period of a few months (7, 9) The American Academy of Ophthalmology (AAO) POAG Preferred Practice Pattern (PPP) has been particularly influential in the United States (7, 1017) The first version of this guideline was published in 1989, and major revisions have since been published approximately every 3 to 5 years. When the AAO PPP guideline was first developed by AAO's Glaucoma Panel, evidence was gathered on the basis of the panel members' knowledge: Members submitted what they considered seminal works, and these works were distributed among the rest of the panel (18) Since 1996, the panel has been using a more systematic approach, carrying out a formal search of the relevant scientific literature and rating the strength of evidence for recommendations (7, 1317) The objective of this study is to compare the evidence base for first line medical treatments of POAG with the recommendations for each major revision of the AAO PPP by using cumulative network meta analysis (that is, conducting a series of network meta analyses on a systematically assembled set of RCTs published up to several distinct periods) Previously, Antman and Lau demonstrated, by comparing the results from cumulative pairwise meta analyses with recommendations given by experts, that meta analysis can improve the timeliness of guidance (19, 20) Using this previous work as a model, we evaluated whether network meta analysis can provide additional benefit in developing clinical practice guidelines. The data for our cumulative network meta analysis are from a systematic review and network meta analysis we previously published (21) This study is not intended as criticism of guideline developers for not using statistical methods that were undeveloped at times in the past but as an example to show how network meta analysis may be able to benefit future guideline recommendations. Methods Guideline Identification and Extraction We searched the AAO Web site (www.aao.org) and contacted the AAO's librarian to identify all versions of AAO's PPPs in August 2014. One member of the team (B.R. reviewed each version of the guideline, identified statements concerning first line POAG medical treatment (that is, as topical monotherapy for decreasing IOP [22] and identified among them the recommendations. We defined recommendations as statements that used the words recommend, should, appropriate, necessary, must, or other words that suggested a particular practice, such as prescribing a medication. A second author (T.L. verified the abstraction and the classification of whether a statement was really a recommendation. We then categorized recommendations by drug name and class of medical treatment (for example, latanoprost, prostaglandins) and extracted the quantitative estimates of effect (for example, reduction of IOP) when provided. We also extracted the ratings of strength of evidence for each recommendation (for example, level I indicates that the basis is a high quality large RCT or a systematic review, and level III indicates that the basis is consensus of experts) (7, 1317) When 2 or more consecutive guideline versions reported the same literature search years or, in absence of reporting search years, they presented identical recommendations regarding medical treatment, we grouped them together. This was done to facilitate the comparison of the guidelines recommendations with the results from our cumulative network meta analyses. Systematic Review and Cumulative Network Meta analysis We identified all available RCTs from a systematic review our group recently conducted (21) In this review, we searched Cochrane Register of Controlled Trials (CENTRAL) in The Cochrane Library, MEDLINE, and Embase, on 11 March 2014 (Supplement 1, shows the full search strategy) and included RCTs evaluating first line topical monotherapies for POAG or ocular hypertension in comparison with no treatment, placebo, or other topical monotherapies. The process of title and abstract screening, full text screening, data abstraction, and risk of bias evaluation has been described previously (21) All data were extracted into the Systematic Review Data Repository (23, 24) Supplement. Supplementary Material For this article, we used the latest guideline in each set to define eligible studies for each network meta analysis. Eligible studies are those published up to the stopping year for the literature search reported in the guidelines or, if such a point was not reported, the year before the guideline were published, to allow for lag time between publication and inclusion of evidence in the guideline. The primary outcome was the mean IOP at 3 months as a continuous variable in units of mm Hg, which corresponds to the primary effectiveness end point on which guideline recommendations were made (7) We prioritized using mean change in IOP from baseline values, but we also accepted mean IOP at 3 months when the change score was not reported (25) Our analysis did not distinguish between drug concentrations, and comparisons were based on the active ingredient and class of that ingredient. We first examined direct comparisons using random effects model meta analysis assuming comparison specific heterogeneity and a common heterogeneity across all comparisons at both the drug and class level. To assess the statistical heterogeneity, we examined the I2 and 2 values for these models. Analyses for direct comparisons were conducted with Stata software, version 13, using the metan command. We fitted Bayesian random effects network meta analysis models using WinBUGS 1.4.3 (2628) We used a 3 level hierarchical model with components at the following levels: study, individual drug, and drug class. This model accounts for the within study correlation of multigroup trials and also incorporates class effect (26, 27, 29) A valid network analysis requires the assumption of transitivity (that is, there are no systematic differences among the trials other than the treatments being compared) (5) This assumption can be tested by assessing inconsistency, the statistical disagreement between direct and indirect comparisons (5) (Supplement 2) We examined mean differences in IOP (and 95% credible intervals [CrIs] between pairs of individual drugs and drug classes (21) We also ranked each drug or class (for example, the probability of a drug being the most effective treatment or the second best) We examined the hierarchy of treatment rankings by using the surface below the cumulative ranking curve (SUCRA) (30, 31) A SUCRA value (or percentage) gives the probability that a treatment is among the most effective treatments, with a value of 1 (or 100% meaning that a treatment is certain to be the most effective of treatments in the network and a value of 0 (or 0% meaning that a treatment is certain to be the least effective. Rankings based on SUCRA values are considered to take into account uncertainty in estimated treatment effects better than do general ranking probabilities (30, 31) Guidelines and Network Meta analysis Comparison We compared information extracted from each guideline set to the results of the corresponding network meta analysis. We assessed whether the recommended drugs or drug classes and effectiveness estimates in the guideline match with the highest ranking drug or drug class, determined by SUCRA values, from the network meta analysis. Comparison With Published Pairwise Meta analyses To determine whether network meta analysis gives incremental information to guideline developers that cannot be gained from pairwise meta analyses, we examined the results of published, high quality systematic reviews and pairwise meta analyses identified previously (32) We matched the pairwise results to the network meta analysis results on the basis of which interventions were compared and the year of publication. We examined agreement of the findings from the 2 approaches qualitatively. For example, when the 95% confidence interval (CI) covers the null value, we concluded that one drug is not superior to another drug. Role of the Funding Source The project was funded by the National Eye Institute, National Institutes of Health. The sponsor had no role
Annals of Internal Medicine