Early recognition and, where possible, avoidance of risk factors that contribute to the development of poststernotomy mediastinitis (PSM) form the basis for successful prevention. Once the presence of PSM is diagnosed, the known risk factors have been shown to have limited influence on management decisions. Evidence-based knowledge on treatment decisions, which include the extent and type of surgical intervention (other than debridement), timing and others is available but has not yet been incorporated into a classification on management decisions regarding PSM. Ours is a first attempt at developing a classification system for management of PSM, taking the various evidence-based reconstructive options into consideration. The classification is simple to introduce (there are four Types) and relies on the careful establishment of two variables  prior to deciding on the best available reconstructive option. It should allow better insight into why treatment decisions fail or have to be altered and will allow better comparison of treatment outcomes between various institutions. Poststernotomy mediastinitis (PSM) is still one of the most complex and costly infectious processes to treat. Changes in cardiac surgical patient population and contributing pathogens, amongst others, have ensured that the incidence of PSM, despite many advances in prevention, remains significant.Data recently presented by the Netherlands Association for Cardio-Thoracic Surgery, which includes all 16 cardiac surgical centers, showed an increase in open cardiac procedures between 2007 and 2011 from 15 500 to 16 500 per year .The acute mortality rates in some large centers still approach 40% internationally , the adjPSM has a significant impact on both healthcare and hospital budgets. In a recent case-control study performed by Graf and colleagues  from HanThe enormous amount of intellectual investment required from a multidisciplinary team including, among others, microbiologists, intensivists, radiologists, cardiologists, cardiothoracic and plastic surgeons in effectively dealing with these cases has yet to be calculated.Recent developments such as the introduction of topical negative pressure therapy (TNP), new insights into the timing of flap surgery and flap choice, design and methods of harvesting have all become significant in recent years. Further improvement in the treatment and research of PSM requires a targeted approach. A new, evidence-based classification may allow for better comparison between different treatment protocols and may further both research and refinement in the management of PSM.The aim of this paper is to categorize and assess the effectiveness of a range of primary and secondary reconstructive management options for PSM, based on currently available evidence and present an evidence-based classification. See the “Clinical definition of PSM”.Poststernotomy mediastinitis (in adults) must meet the following criteria:Infection occurs within 1 year, regardless of whether an implant is in place or notAND infection appears related to the operative procedurePatient has organisms cultured from mediastinal tissue or fluid obtained during a surgical operation or needle aspiration.Patient has evidence of mediastinitis seen during a surgical operation or histopathologic examination.Patient has at least 1 of the following signs or symptoms with no other recognized cause: fever (>38 0C), chest pain, or sternal instabilityAND at least 1 of the following:Purulent discharge from mediastinal areaOrganisms cultured from blood or spontaneous discharge from mediastinal areaRadiological evidence of an infective process in the mediastinum.AND, at least 1 of the following criteria:A search of the literature from 1990 to March 2014 was conducted without language restrictions using the Cochrane Central Register of Controlled Trials, Ovid Medline, and PubMed and Web of Science databases. Key words and MeSH terms were used to identify and a manual search of the reference lists was done regarding all possible factors  that could influence the current treatment decisions of PSM. Only procedures for which either usefulness or efficacy was claimed were considered. The quality and strength of the evidence was weighed according to the Rating Scheme used by the Society of Thoracic Surgeons (STS) Workforce on Evidence-based Medicine (EBM) for Classification of Recommendation (see below) . Two hunRating Scheme for the Strength of the EvidenceLevels of EvidenceLevel A. Data derived from multiple randomized clinical trials.Level B. Data derived from a single randomized trial or from nonrandomized trials.Level C. Consensus expert opinion.Rating Scheme for the Strength of the RecommendationsClassification of RecommendationsClass I. Conditions for which there is evidence or general agreement, or both, that a given procedure is useful and effective.Class II. Conditions for which there is conflicting evidence or a divergence of opinion, or both, about the usefulness/efficacy of a procedure.Class IIa. Weight of evidence favors usefulness/efficacy.Class IIb. Usefulness/efficacy is less well established by evidence.Class III. Conditions for which there is evidence or general agreement, or both, that the procedure is not useful/effective.The Level of Evidence was independently established by two authors (JJvW & DU) and a Classification of Recommendation was set by mutual consent. In case of disagreement, a third reviewer (either BdM or CvdH) was consulted.From two hundred thirty three records, the search strategy rendered 78 studies, of which 4 were case reports Table  that comper procedure, while procedures for which no evidence existed were discarded.As a result most studies were categorized as class II  The type of study determined the level of evidence for a specific procedure. Systematic reviews of observational studies, controlled clinical trials, or comparable cohort studies were considered to be Level B1 and case series and regression analyses as B2. The Level of Evidence for Case Reports was considered to represent Level C.The outcome of the categorization of the supporting literature is summarized in the Table The new classification was based on the Level of Evidence for the usefulness or effectiveness of a specific management procedure as well as sternal stability, viability, and available bone stock, together with timing of reconstruction. This resulted in a classification presented in Table When minimal bone loss and a relatively stable sternum are found at debridement, available evidence favors wound management through the application of TNP. Microbial identification and antibiotic susceptibility should be established early. In addition, the dosing of antimicrobial agents should be adjusted in obese patients to ensure adequate tissue levels  , as the primary causative pathogen, is not a contraindication for TNP therapy.  Negative-pressure therapy was also successfully applied as a temporizing measure prior to secondary closure in candidal mediastinitis , me, me29], stinitis ,31] .In Type 2 there is sufficient bone stock and the sternum is relatively stable. Direct closure, usually accompanied by advancement of the pectoralis muscle, is done either primarily (without a conservative management bridge such as TNP) (Type 2a) or delayed (Type 2b).A single-stage procedure (Type 2a), combining drainage and debridement with immediate flap reconstruction, was favored by early workers in the field. Although never specified, two possible reasons justified this approach at that time: first, the introduction of flap reconstruction which dramatically decreased mortality related to PSM; and secondly, the high failure rate related to conservative management such as packing and or antibiotic irrigation. This was prior to the introduction of TNP therapy. In the advent of very early diagnosis and referral, in the hemodynamically stable patient or where TNP therapy is not available, immediate flap closure may still be justified, as some have pointed out -36]  has gained strong advocates in recent years for the following reasons:Leaving the chest open between the stages of drainage, debridement and closure has become less of a hazard since the application of, and the experience gained with, TNP therapy. A body of current literature supports the findings from our institution  that earS. aureus – instead of a reduction in bacterial load   reconstruction following debridement of all necrotic and fractured bone improves local vascularity, can obliterate dead space and, on its own , may ensure sufficient chest stability  in theseAs a better understanding regarding angiogenesis ,71], an, an71], A clear preference has been expressed for the use of omentum (instead of muscle) in cases where the primary causative organism is particularly resistant, such as MRSA ,77]-[80-[8077]-,81] , available to the surgical team reaches a level of evidence higher than B. This means that data were derived from either a single randomized trial (RT) or from non-randomized trials, but not from multiple RTs nor through consensus expert opinion. The absence of variation in the level of evidence for each reconstructive option also implies that there is insufficient evidence available at this stage to confirm superiority of one reconstructive method above another. This is the reason why comparison between the various modalities of treatment available to the surgical team is so difficult. Yet, it also suggests that some treatment modalities can be recommended more strongly than others (Class I recommendation) under comparable circumstances. For example both TNP (only) and a delayed flap come with a Class I recommendation in cases where the sternum is found to be fairly stable and viable following an assiduous debridement. This too may complicate determination of the best treatment option. Both difficulties caused by similarities in the level of evidence and classification of recommendation underscore the need for an EB classification as a starting point from which the best decision can be derived. This is the ultimate aim of the proposed Classification. Consistent cognizance of the variables should allow comparison between treatment that are now regarded to be either “apples or pears” (for example TNP (only) vs. flap reconstruction). For example, in our institution our first choice, in cases with a fairly stable and reasonable viable bone, would be the application of TNP following a thorough debridement . In otheAt the other end of the spectrum, the surgical team is left with few options when the sternum is very necrotic and found to be insufficient for closure following debridement. But then, which flap would be preferable and at which stage (primary or delayed) should it be applied? The most difficult cases to decide on (and to compare) are the scenarios in between the two Types I and 4, because of the relatively small number of cases per institution. Comparison could be made easier if some of the variables are standardized. Eventually it would also open a way for meta-analysis, which is necessary in small case series of each institution. The proposed Classification could be a first step in this direction. What this Classification is not primarily intended for is to encourage institutions to alter their approach to each case of PSM. The preferred approach to the further management of PSM is decided on during the first, vigorous debridement that follows immediately upon diagnosing PSM. Patient-related factors  and local circumstances  will influence this first decision and determine the type of management is planned accordingly. It is this first decision that determines the type of management to which the patient is allocated. Further treatment may change according to altering circumstances  but the Type will not change - regardless of what treatment was ultimately given. In this way a prognostic balance generated from the original treatment allocation is ensured. Management approach (type) allocation allows for outcome analysis, which in turn reflects the real-world clinical scenario because it admits treatment-decision alterations and deviations. Insight into the reasons for, and timing of, these decisions will become clearer. A potential weakness of the proposed Classification, some may argue, could be that it is over simplified. This will have to be weighed against a Classification that is so cumbersome that it becomes impractical. Time will tell - at least it is a start.The development of further, evidence-based guidelines and research would benefit from universally adopted classification of the severe infectious process in the wound that may follow a sternotomy.In this paper a classification system is introduced based on Evidence-based principles with the aim of providing the highest level of evidence currently available. This system, due to its simplicity, could easily be adopted provided that there is an early, collaborative approach between cardiothoracic-, and reconstructive plastic surgeons. As more evidence becomes available, undoubtedly the classification proposed here will be refined. For more evidence, larger numbers, from numerous centers internationally, are required and for this, a classification as presented here, may serve as a starting as a starting point.JJvW: Literature search, preparation of the manuscript, revising the manuscript, submission, corresponding author; DU: Literature search, preparation of the manuscript; CvdH: Literature assessment, preparation of the manuscript, revising the manuscript; BdM: Literature evaluation, preparation of the manuscript, revising the manuscript. All authors read and approved the final manuscript.


Concerns about rising health care costs and the often incremental nature of improvements in health outcomes continue to fuel intense debates about ‘progress’ and ‘value’ in cancer research. In times of tightening fiscal constraints, it is increasingly important for patients and their representatives to define what constitutes ’value’ to them. It is clear that diverse stakeholders have different priorities. Harmonisation of values may be neither possible nor desirable. Stakeholders lack tools to visualise or otherwise express these differences and to track progress in cancer treatments based on variable sets of values.The Patient Access to Cancer care Excellence (PACE) Continuous Innovation Indicators are novel, scientifically rigorous progress trackers that employ a three-step process to quantify progress in cancer treatments: 1) mine the literature to determine the strength of the evidence supporting each treatment; 2) allow users to weight the analysis according to their priorities and values; and 3) calculate Evidence Scores (E-Scores), a novel measure to track progress, based on the strength of the evidence weighted by the assigned value.We herein introduce a novel, flexible value model, show how the values from the model can be used to weight the evidence from the scientific literature to obtain E-Scores, and illustrate how assigning different values to new treatments influences the E-Scores.The Indicators allow users to learn how differing values lead to differing assessments of progress in cancer research and to check whether current incentives for innovation are aligned with their value model. By comparing E-Scores generated by this tool, users are able to visualise the relative pace of innovation across areas of cancer research and how stepwise innovation can contribute to substantial progress against cancer over time. Learning from experience and mapping current unmet needs will help to support a broad audience of stakeholders in their efforts to accelerate and maximise progress against cancer. The ongoing ’War on Cancer’ poses a far greater challenge than originally anticipated, largely because solid tumours have been found to be extremely genetically heterogeneous. Experts now estimate that more than 200 distinct types of tumours exist . AdditioIn the meantime, divergent opinions and selective use of indicators continue to enable misunderstandings about advances in cancer research. The public tends to expect progress to occur in great leaps. Once a cure is available (for example in testicular cancer), we tend to forget about the numerous steps it took to achieve that success. Because of space and resource limitations, stakeholders often focus on ‘milestones’ in cancer research, which may augment the flawed view that cancer progress occurs in great leaps. While major advances do occur in irregular, unpredictable intervals, most progress happens more slowly as continuous stepwise improvements accumulate. Educating the public and stakeholders about the true nature and value of stepwise progress against cancer is, therefore, of great relevance .Rising health care costs and the often incremental improvements in health outcomes have fueled intense debates about maximising the ‘value’ of investments in cancer research –7. In tiInstead of trying to homogenise diverse values, we need a better understanding of existing differences to implement flexible policies and account for real differences in values. Stakeholders lack tools to visualise and otherwise express differences in values, which ultimately leads to different decisions about research and health spending priorities.PACE  has responded to these needs by developing objective, scientifically rigorous, and forward-facing indicators that permit tracking and comparing cumulative progress across various cancer types.A central component of progress in cancer treatments is the knowledge that researchers gain from clinical trials. Every time researchers achieve even a small increase in survival—and often when treatments fail entirely—they gain some understanding of the cancer disease process. Many such small steps ultimately lead to substantial gains. Successful and unsuccessful treatment trials often provide important clues regarding the best next steps. Treatments that we take for granted today often took a long time and many trials to establish.The Continuous Innovation Indicators allow users to quantify and visualise how stepwise innovation can contribute to more substantial progress against cancer over time. We envision the final online implementation of the Indicators as a simple user interface that generates summary graphs from which a user can access the supporting individual evidence records and additional detailed information .Evidence Scores (E-Scores) are a novel quantitative measure of cumulative evidence that is further explained below. The user can move up and down between the E-Score summaries and other layers to map the available evidence onto the current Value Matrix, a flexible new platform for weighing the value of treatments based on therapeutic goals. Those interested in the underlying studies can access the exact identifiers of the clinical trials, retrospective studies, systematic reviews, and literature references that comprise the evidence base for a treatment of interest. The first release will include data for 12 anatomical disease sites: breast, colon, endometrium, kidney, liver, lung, pancreas, prostate, skin (melanoma), stomach, rectum, and testes.Accountability in all steps is essential for the success of this effort because stakeholders will expect to be able to scrutinise the data and the computations that led to the summary scores. The layered design of the Indicators ensures that all stakeholders can access information at the desired level.One factor that has likely slowed progress in the past is the lack of a common framework to assess areas in which research has progressed versus those that have lagged behind. PACE has initiated this project to implement data management procedures that can track progress consistently across cancers. To convert data from PubMed references and other sources into quantitative measures, we have developed a standardised data acquisition and management methodology with the following main components:Pieces of Evidence: When reviewing evidence from the literature, one complicating factor is the frequent use of multiple interventions and outcome measures within the same study. Users may have different views of or uses for the findings. We therefore developed a methodology to disaggregate the evidence by splitting each study into distinct Pieces of Evidence. From each study, we derive one or more such pieces, each of which contains one disease state, therapeutic goal, treatment, and outcome. For each outcome, we include only analyses that apply proper correction for multiple testing.Evidence Hierarchy/Quality: Evidence that a new treatment works usually accumulates over time. To derive a quantitative measure, we set thresholds for evidence of sufficient strength to count and to define milestones for achieving stronger evidence. We further developed a system to assign weights to evidence from different kinds of studies , observational studies). This can be done at the level of the type of study or for each individual study.Value Matrix: To measure progress, we need to define ‘success’. Which developments are sufficiently different from existing treatments to be considered ‘novel’? Which are ‘improvements’ of existing treatments? How do we value different treatment goals? We have developed a novel Value Matrix for cancer research, which is further explained below. Analysts working with the system who want to set their priorities differently can easily change the parameters.Relational database with tagged data: Our audiences will be interested to compare outcomes not only between anatomical sites of the primary tumour, but also between different therapeutic approaches (e.g. targeted therapy versus immunotherapy) or even between individual targeted signalling pathways. The Continuous Innovation Indicators use a relational database design and data-tagging methods to provide this flexibility. This approach allows comparisons across anatomical sites, pathways, treatments, etc. It is an important feature to create forward-looking indicators that can reveal important data stratified by not only disease site but also with emerging biomarkers and therapeutic approaches.Output formats include detailed plots of the data onto the Value Matrix. We further supply E-Scores to measure progress over time and summary statistics that can be stratified by a large number of tags .The Indicators approach relies initially on disaggregating the evidence. Each publication, clinical trial record, historical reference, or other source of information usually reports multiple analyses of outcomes. These measurements become separate Pieces of Evidence  in our sOne Piece of Evidence for each treatment arm, for example:Drug A / Drug B versus placeboDrug A versus Drug B, if Drug B is the current standard of careDrug A and Drug B together vs. Drug A and Drug B administered sequentiallyOne Piece of Evidence for each time point One Piece of Evidence for each patient subgroup (e.g. entire population versus subgroup positive for a molecular marker)One Piece of Evidence for each analysis goal Each Piece of Evidence can contain multiple tags that provide additional information about the anatomical site, eligibility criteria, histological subtypes, etc. Additional information about each Piece of Evidence is gathered in a separate data table that is part of a relational database. This ensures that parameters in the Piece of Evidence table are connected to additional tables. The treatment table, for example, contains information about the drug class, the type of surgery or radiotherapy, different approaches of targeted therapies, and eligibility criteria . It further links treatments to other treatments and tracks whether a treatment is considered a new treatment, an improvement of an existing treatment, or a combination of several existing treatments.This flexible approach allows the user to query the database and retrieve all evidence supporting or discounting a certain type of targeted treatment (e.g. ‘anti-angiogenesis treatment’) in a given context . Through the PubMed identifier, each Piece of Evidence is tagged with a date that further allows users to track progress over time and visualise possible lags in innovative developments in one cancer versus another.For the analysts working on the database and populating it with new Pieces of Evidence, the relational design makes the daunting challenge of reviewing the evidence less cumbersome, because it divides it into manageable units of analysis. The analysis team can further work on the Pieces of Evidence and treatment characterisations in parallel. Quality control procedures for data in each table ensure that only valid information becomes connected to other parts of the database and is included in the E-Score calculations. To facilitate this process, each Piece of Evidence carries multiple flags that can assume yes/no/to-do values to help with the systematic review of all data.We have incorporated several safeguards to ensure data and analytical integrity. Data review is based on automated inputs and downloads of electronic records through the National Center for Biotechnology Information (NCBI) Ebot tools, which minimises manual typing errors and saves time for analysts. All analysts use a common standard operating procedure to determine which evidence to include. Finally, multiple analysts conduct blind duplicate reviews to ensure consistent scoring.Identifying the best measure to track progress in cancer research is not trivial. Several seemingly ‘obvious’ candidate measures fail to provide an accurate picture when scrutinised closely. Period survival, for example, is subject to lead-time and length biases .The number of cancer survivors, which is sometimes used as an indicator of success, has increased during the past decades and is projected to continue to increase in the coming years. The definition of ‘survivor’, however, is a person who has been diagnosed with cancer and is still alive . Based oWe therefore decided to create a new measure for progress in cancer research, called an E-Score , which iClinicalTrials.gov within one year of completion, references to clinical trials in the Indicators’ database should be fairly complete. However, a recent study showed that only 22% of studies completed in 2009 met the requirement to report results [All evidence is not equal. Systematic reviews of primary data, which often include meta-analyses of RCTs and other trials, can often reach stronger conclusions than reports of individual clinical trials. This is because meta-analyses are often used for controversial topics where there are no definitive studies, and analysing pooled data from several smaller trials is necessary to confidently draw conclusions . Some sy results .E-Scores account for differences in evidence strength functionally and numerically by using an Evidence Hierarchy/Quality weight. Meta-analyses are the highest functional tier and can, as described below, override evidence produced by individual studies in certain circumstances. Nevertheless, meta-analyses do not receive the highest numerical weight when calculating E-Scores because doing so would artificially increase E-Scores for diseases with many controversies and small trials. Instead, in the current default settings, individual clinical trials receive the highest numerical weight, and both observational studies and meta-analyses receive lower weight. If the authors of the original trial report nominally significant p-values for the treatment effect but note that the side effects were so severe that the trial failed, we record the statistics from the trial in a Piece of Evidence but do not allow it to increment E-scores.Some Pieces of Evidence are considered ‘group evidence’ when the treatment effect cannot be attributed to a single factor, for example when a study examines a drug class rather than individual drugs. We assign a fractional weight to results of these studies because it is unclear how much each individual drug contributes to the overall effect. If these individual effects are identified for pre-planned analyses, then we include the individual results. Some or all of the treatments tested as a drug class in one study might also be singly represented as Pieces of Evidence based on trials that tested the treatment individually.With each data release, the analysts who curate the data revisit the existing evidence to determine how new Pieces of Evidence change their assessment of previous pieces. This procedure is described in detail in In brief, if a systematic review, for example a meta-analysis by the Cochrane Collaboration, concludes that the supporting evidence for a treatment in a certain context is not strong enough to support this treatment any longer, then the E-score algorithm disregards (though does not delete) all prior Pieces of Evidence for that particular treatment until further notice. If Pieces of Evidence prove to be invalid in this way, they become flagged and disqualified in the next data release, rather than decrease the E-Scores ‘in real time.’ This means that the entire cumulative E-Score curve for the corresponding cancer will be lower in the following data release, but the slope will never be negative. Additionally, the analysts can query the database and obtain statistics on how often treatments that showed significant effects in one or more trials were later re-assessed and found to be ineffective.This procedure establishes an audit trail and ensures complete accountability regarding the underlying evidence, because no records are ever deleted from the database. It is, of course, fully possible that additional trials may reverse the assessment once again, or that improvements of a treatment lead to statistically significantly improved outcomes, in which case the analysts will re-assess the existing evidence. This procedure reflects the stepwise nature of progress in cancer science and provides a realistic picture of the many trials and studies that are often necessary to understand the benefit of a new treatment.The Continuous Innovation Indicators seek to combine scientific accuracy with ease of use. Each new therapy (represented by a circle) can map to one or more squares on the matrix. If the goal of a treatment is, for example, to make the patient disease-free after a diagnosis of advanced or metastatic cancer, then it will be represented in the top square labeled ‘A’. If the goal is to stop progression of an advanced or metastatic cancer, then it will be represented in the square labeled ‘D’, whereas a treatment meant to keep patients with resected tumours disease-free will be represented in the square labeled ‘J’ , most patients (medium), or essentially all patients (large). Targeted treatments that have strictly defined molecular eligibility criteria are represented by small circles if they constitute the standard of care for a small, defined group of patients. If additional evidence becomes available indicating that a targeted treatment benefits a larger share of the patient population, the corresponding circle ‘grows’ in the visual representations over time.The first release of the Indicators will focus on the outcome of overall survival. Therefore, only treatments that have been shown to improve overall survival  are plotted onto the matrix and included in the E-Score calculations. One advantage of our model compared to other efforts to track innovation is that it accounts for multiple treatment modalities . It further allows the analysts to create ‘treatment goal paths,’ which are depicted as red arrows. The value of neo-adjuvant treatment, for example, will often be producing a larger subpopulation of operable cancers and increasing survival, not by shrinking the tumour per se but by increasing surgical success rates. Thus, the treatments connected in a path work together to improve outcomes for patients, and including additional treatments in these paths can lead to overall outcome improvements that are larger than the sum of its parts. These synergies are rarely evident when a new treatment is introduced and often require a substantial number of additional studies. It is further possible that additional treatments interact with available treatments, for example when neo-adjuvant treatments might threaten to obscure nodal status and impair pathological staging . The treCustom weights, the final component of E-Scores, include additional parameters to tailor E-Scores according to stakeholder priorities. Custom weights can factor in, for example, effect size , drug class, and other variables of stakeholder interest.The mathematical formula for the E-Scores is: evidence x value weights x custom weights. Once the relevant Pieces of Evidence have been identified, the algorithm calculates E-Scores using the Evidence Hierarchy, value weights from the Value Matrix, and any custom weights selected. The resulting curves go up when the evidence increases. Based on the value weights, some new treatments may lead to considerable increases in E-Scores within a short time period. More often, however, in accordance with the paradigm of continuous innovation in cancer research, gradual increases add up to larger increases over time.Users may modify the weights of the Evidence Hierarchy, Value Matrix, and custom weights that underlie the algorithm to match their own values or priorities. The weights, in particular, are likely to differ among stakeholders. Because values can change over time, the scoring algorithm accepts several such matrices (for different points in time), using the scores from the closest preceding matrix.We recognise that visualisation and quantification of progress in cancer research needs to be systematic and evidence-based to ensure its utility for stakeholders.The approach is detailed below and summarised in The first step is data capture. The Continuous Innovation Indicators allow users to import large sets of records from existing databases with ease. The first release will focus on results of clinical trials  and studies that have been identified by others as key findings, for example, all references cited in the National Cancer Institute (NCI) Physician Data Query (PDQ) guidelines. We also include references from the Cochrane Library. Because we envision this tool as a dynamic resource that improves over time, we will include additional references whenever necessary. To allow comparisons of drug approval dates between countries, we can also import data from Citeline Pharmaprojects® and other sources that contain information on the development, approval, and launch of new therapies. We aim to provide updated versions of the references about every three to four months.The data from the above references are, in their native format, not suitable for quantification, because each study may contain multiple outcome measures. Disaggregating them into distinct Pieces of Evidence facilitates compilation of a common body of knowledge that stakeholders can review, discuss, and revise.When new Pieces of Evidence have been created, they not only add to the existing evidence, but also may modify the net findings derived from the resulting body of evidence. Therefore, the system pulls up all Pieces of Evidence for the same treatment and square every time a new Piece of Evidence has been added so that the analysts maintaining the system can determine whether the E-Score should change and whether the validity and importance of older pieces need to be revised.Once the relevant data have been captured and checked for quality, the user can query the database to retrieve all the Pieces of Evidence that are relevant for a particular clinical context. Based on the available evidence, the Evidence Hierarchy, and optional custom weights, the system then generates raw scores for each year of interest.Finally, the evidence is weighted by the values from the Value Matrix. Experience with test audiences suggests that users may find it useful to plot the existing treatments onto the matrix before assigning the values to increase consistency between different raters. In addition, it may be useful to plot similar scenarios from different cancers so that users can learn from previous developments and set realistic priorities and expectations.Flexibility is one of the greatest strengths of the Continuous Innovation Indicators. Indeed, the Indicators are designed to grow alongside of continuous progress in cancer research. The streamlined approach to integrating large amounts of new evidence and reassessing past evidence in the present context could, for example, be used to trigger alerts when new evidence emerges that substantially alters prior conclusions. Virtually all components of the Indicators can be adjusted and enhanced to serve evolving needs. Different outcome measures, data sources, and refined weighting systems can all be incorporated.The first release of the Continuous Innovation Indicators will focus exclusively on studies that measure overall survival. Overall survival is a broadly accepted, well-defined, and commonly measured outcome and is therefore a logical starting point. The Indicators can easily incorporate additional outcome measures, such as progression-free survival, disease-free survival, or measures of mortality at the population level.One outcome measure of great interest to many stakeholders of cancer treatment research is quality of life (QOL). Unfortunately, QOL is difficult to measure and interpret. A recent American Society of Clinical Oncology (ASCO) cancer research committee working group on pancreatic cancer tasked with defining clinically meaningful outcomes concluded that, while important, improvement in QOL should not be used as a primary endpoint for clinical trials because ‘current global quality of life questionnaires are not considered to be useful’ . InsteadOthers have noted a difference between efficacy and effectiveness of treatments: efficacy refers to the benefit of a treatment under ideal or controlled conditions, such as RCTs, and effectiveness refers to the benefit of a treatment under routine or ‘real-world’ conditions, such as treatment in community oncology settings , 16. In Incorporating new data sources can also mitigate the effects of publication bias. Publication bias refers to the greater likelihood of some studies (e.g. those with positive results) to be published over others (e.g. those with negative results). The Indicators may be somewhat influenced by publication bias because they incorporate broad selections of published literature. Others have identified strategies to control for such biases , some ofMany approaches have been developed to assess the quality of evidence and could be used to refine the Continuous Innovation Indicators Evidence Hierarchy. For example, the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) working group developed a now widely used two-step approach to rate the quality of evidence on a four-tier confidence scale . First, Evidence-based medicine is an important cornerstone of modern health care systems. Informed policymakers are more likely to invest in health than those without exposure to evidence regarding the benefits of existing and novel treatments. Under global fiscal constraints for health care systems, evidence about benefits and harms to individuals and populations become even more critical because it helps to inform investments about innovation and policies mediating access to novel treatments.The Continuous Innovation Indicators can serve as an educational tool to support evidence-based investments by informing policymakers and the public about the stepwise nature and paths of cancer research progress. Publications about advances in research often focus on milestones and ignore the many iterative improvements of existing drugs, combination treatments, and multimodal treatment approaches. This leads to the false impression that cancer research progress occurs in great leaps. It is remarkable that the debate in public media is still dominated by black and white statements on whether or not we have made progress against cancer –22. FocuThe Continuous Innovation Indicators do not offer clinical guidance, and they are not meant to replace existing treatment guideline publications. Further, the Indicators are not meant to replace current efforts by the Cancer Intervention and Surveillance Modeling Network (CISNET)  and otheThe E-Scores computed with our tool are not absolute values that quantify overall survival but instead measure the increasing evidence supporting new treatments against cancers of interest. Similar to other commonly used indices, their primary purpose is to track changes over time and between areas of interest, not to provide an absolute value that can be directly translated into survival estimates. At the core of the Indicators is a large database that allows analysts to carry out many other queries, for example to understand how many treatments have been deemed ineffective after their introduction, whether or not certain countries or regions approve new treatments faster than others, etc.We hope that the disaggregated Pieces of Evidence approach will stimulate interactions among scientists to establish a body of knowledge of accepted evidence. While we do not expect the Continuous Innovation Indicators to ever substitute for carefully conducted systematic reviews, we hope that they will become useful to those in the field searching for a reference framework. By getting a faster overview of the state of the art, possible comparators, and unmet needs, we hope to be able to contribute to accelerating the important work of those who evaluate new treatments.Continuous updates of the Indicators as a central resource hold the promise to create a resource for those in the field similar to the availability of a public genome-browser for those working in genomics . We are We hope to contribute to ongoing discussions about value in cancer care by highlighting the critical importance of stepwise innovation. Furthermore, identifying unmet needs will stimulate discussions of greater societal benefits of new treatments. The Indicators are thus meant to complement, not replace, ongoing efforts to discuss cost-effectiveness of new treatments. Value in cancer care is a very complex concept that will likely require multiple efforts and approaches to define and translate into concrete action by policymakers and others who set priorities for cancer treatment developments. The Continuous Innovation Indicators are meant to become one piece in this complex puzzle.Finally, we hope that components of our approach, such as the Value Matrix, will be useful in supporting policy discussions as outlined in The PACE Continuous Innovation Indicators provide a novel tool to:Establish a relational database that allows execution of queries such as:Number of times we learned more about treatments after they were introducedNumber of times treatments did not work out as plannedKeep track of progress in a flexible framework that allows the analysts to incorporate relevant new evidence in real time and to quickly determine the impact of this new evidence on the assessment of the available body of relevant evidenceGain a better understanding of the complex evolution of value in cancer treatmentVisualise how stepwise progress contributes to significant progress against cancer over time, including the synergies of combination and multimodality therapiesIlluminate the extent of progress and allow for comparisons across anatomical sites, treatment approaches, molecular subtypes, and other stratifying variablesEstablish a map of remaining unmet needs in the treatment of cancerIllustrate the potential impact of cancer-policy reformsDifferentiated analyses based on values provided by various stakeholders will help the cancer policy field to obtain accurate representations of the complex, stepwise progress against different cancers over time. We encourage organisations who want to partner with PACE to contact the corresponding author. We envision partnerships and collaborations to support educational efforts, identification, and illustration of policy goals, and work in the field of health technology assessments. We will not make this tool available to individuals or organisations for the purpose of deriving treatment recommendations.


Coccidioides spp., which afflicts primarily humans and other mammals. It is endemic to areas in the southwestern United States, including the San Joaquin Valley portion of Kern County, California, our region of interest (ROI). Recently, incidence of coccidioidomycosis, also known as valley fever, has increased significantly, and several factors including climate change have been suggested as possible drivers for this observation. Up to date details about the ecological niche of C. immitis have escaped full characterization. In our project, we chose a three-step approach to investigate this niche: 1) We examined Landsat-5-Thematic-Mapper multispectral images of our ROI by using training pixels at a 750 m×750 m section of Sharktooth Hill, a site confirmed to be a C. immitis growth site, to implement a Maximum Likelihood Classification scheme to map out the locations that could be suitable to support the growth of the pathogen; 2) We used the websoilsurvey database of the US Department of Agriculture to obtain soil parameter data; and 3) We investigated soil samples from 23 sites around Bakersfield, California using a multiplex Polymerase Chain Reaction (PCR) based method to detect the pathogen. Our results indicated that a combination of satellite imagery, soil type information, and multiplex PCR are powerful tools to predict and identify growth sites of C. immitis. This approach can be used as a basis for systematic sampling and investigation of soils to detect Coccidioides spp.Coccidioidomycosis is a fungal disease acquired through the inhalation of spores of  Coccidioides immitis and Coccidioides posadasii, the fungi that cause coccidioidomycosis , in contrast to its close relative C. immitis which seems to be restricted to areas in California. Of the two fungal species, it is C. immitis which afflicts the San Joaquin Valley portion of Kern County, California Coccidioides spp. revealed recent hybridization between both species Coccidioides spp. bloom, and windy spells might facilitate the dispersal of its arthroconidia. The “grow and blow” hypothesis has first been introduced by Comrie and Glueck Coccidioides spp. are generally poor competitors Coccidioides immitis in soils of Eastern WA It is reasonable to expect that climatic fluctuations might affect the rate at which humans become infected Coccidioides spp. in more detail C. immitis and C. posadasii do not grow in disturbed soils Coccidioides spp. ecological niche corresponds to the Lower Sonoran Life Zone , microbial antagonism (antibiotic production) and synergism, as well as through selective grazing by microeukaryotes There also have been several attempts to characterize the ecological niche of C. immitis growth sites in Kern County, California Coccidioides spp. from soil and dust samples C. immitis in bulk soil samples from Kern County, predominantly around Bakersfield, have been performed C. immitis is likely to be found in the Bakersfield area at locations that are non-agricultural and have about equal parts of sand, clay, and silt (clay loam), a pH between 7.8 and 8.5, an available water capacity of about 0.15–0.2 cm/cm, a water content of about 30% (1/3 bar), an available water supply (0–25 cm) of 4–5 cm, and a Cation Exchange Capacity (CEC7) of over 20 milliequivalents per 100 grams.There are few published data available about the distribution of C. immitis is not well characterized, we present here a RS technique that allows the mapping of sites around Bakersfield, California, where the pathogen is suspected to grow based on data obtained in a previous study by Lauer et al. C. immitis growth site  as a basis, and then examined satellite images of the ROI to find all locations with similar spectral signatures. This is similar to characterizing the growth sites by the vegetation that tends to grow in the same environment as C. immitis, using the vegetation type as a marker. This is reasonable because the vegetation type closely reflects the co-variation of the relevant physical and chemical parameters such as clay and sand content, temperature, pH, nutrients, water content, etc., and also affects the development of the microbial diversity in the soils The idea of using remote sensing (RS) techniques to piece together environmental characteristics, environmental change, and their relationship to disease transmission has been used extensively in connection with other diseases such as malaria C. immitis, followed by a molecular biological approach to detect the fungus in these soils with a culture independent polymerase chain reaction (PCR) based method To validate our approach, we investigated if a combination of remote sensing and soil parameter information can predict locations which might be suitable to support the growth of No specific permissions were required for the soil sampling. Our field study did also not involve endangered or protected species.http://EarthExplorer.usgs.gov). The satellite relayed a continuous data stream which was then framed into individual scenes each 23.92 sec . The images for path 42, rows 35 and 36: Worldwide Reference System to cover our ROI were downloaded, and then the two images were mosaicked. Most of the analysis that is presented here was performed on a spatial subset of this mosaicked image. This subset corresponds to an area approximately one million hectares that covers the San Joaquin Valley portion of Kern County. Our work mainly focused on a multispectral image taken on April 20, 2008 at 10:23 PM local time. This image was chosen because it was obtained at a date (during spring) were microbial activity and biomass in the soil is generally considered high, because of supportive environmental parameters, such as moderate temperatures and increased water content. Furthermore, this image had 0% cloud cover. Our analysis started by defining a 25 pixel ×25 pixel area centered at latitude 35° 28′ 20.29′′ N, and longitude 118° 54′ 37.04′′ W. This location is at STH, an area where C. immitis has been repeatedly detected (Landsat-5 Thematic Mapper (TM) L1G corrected multispectral images were downloaded from the United States Geological Survey archive . A threshold value was set at 0p, meaning that if a pixel had a probability 0p decreased, the fraction f of pixels in the ROI which belong to the STH-vegetation class increased. This is because pixels which are less and less like the training pixels get included into this class.where 0p needed to be reduced from 1 until the sites which tested positive for C. immitis came into the STH-vegetation class. This served to calibrate the method and as a validation step. Clearly, if the C. immitis-positive sites get included in the STH-vegetation class for C. immitis-positive sites remain unclassified until It was also investigated how much 2) that was characterized by vegetation that belonged into the STH-vegetation class over the sampling period and until early 2014 using landsat images and the software ENVI 5.1+IDL 8.3.Lastly, we determined the area  was used as follows to examine the thermal landscape of our ROI. The same area in STH was taken as training pixels, and their average n = 1,2,3 were put into this spectral class, and we referred to this as the “STH-thermal” class. All other pixels were left unclassified. The image's digital numbers were converted to temperatures by applying the procedure described in the National Aeronautics and Space Administration's Landsat 7 Science Data Users Handbook (http://landsathandbook.gsfc.nasa.gov/). See also Chander & Markham Surface temperature variations across the ROI were of interest as well. In addition to utilizing a vegetation class to assess potential sites for http://cdec.water.ca.gov/snow_rain.html). The cumulative monthly precipitation (inches) over time was assembled from 5 stations , and Huntington Lake (HNT]). A more detailed analysis of the weather data for Bakersfield in particular was not the focus of this study.Precipitation data for the Southern San Joaquin Valley was obtained from the California Data Exchange Center (http://websoilsurvey.nrcs.usda.gov/) was used. Furthermore, all sampling sites were characterized by using the soil series extent mapping tool from the website of the Center of Environmental Informatics (CEI) (http://www.cei.psu.edu/cei_wp/). Thus, through agricultural and environmental support tools available from the USDA and CEI websites, our sampling sites were further characterized in regard to land use and vegetation. Additional geological information was obtained as well, such as the distribution of certain soil types and series in California. By using the soil series extent mapping tool, our soil samples were linked to known soil series and soil groups that are characteristic for the Southern San Joaquin Valley and beyond.To determine physical and chemical soil parameters of all soil samples, the websoilsurvey database of the United States Department of Agriculture  from three different depths , placed on ice during transport to the lab, and frozen at −80°C when not processed immediately. See C. immitis. Also see the first column of table two for the year they were investigated. Our sampling sites were not chosen based on Landsat imagery. They were chosen mainly based on the percentage of clay in the soil as indicated by the USDA websoilsurvey database. About 30% of clay had been indicative of a potential C. immitis positive site based on previous research C. immitis was detected correlate with sites indicated by Landsat imagery to fall into STH vegetation sites.Soil physical and chemical parameters that could likely support the growth of the pathogen based on results of the study by Lauer et al. C. immitis from soil DNA by Lauer et al. C. immitis in all soil samples with two primer pairs. Primer pair ITSC1A/ITS C2 , which is specific for C. immitis, was used in combination with primer pair RDS478/RDS482  which amplifies 18S rDNA from all fungi. The ITS region was chosen due to its high nucleotide variability. Amplified ITS fragments were extracted from the 2% Agarose Gel, extracted with the Zymo Clean Gel DNA Recovery kit , and subsequently sequenced to confirm the presence of C. immitis. Extracted DNA from a C. immitis isolate (M39), obtained from the Laboratory of Medical Mycology at the Universidad Nacional Autónoma de México was used as positive control. Negative controls and positive controls were included in all PCR's to detect contamination and to verify the amplification of a PCR product of the desired size.DNA was extracted from well-mixed soil samples (two replicates) using the MoBio PowerSoil DNA Isolation Kit  following the manufacturer's protocol. The multiplex PCR approach developed by Greene et al. C. immitis was detected at least twice in a deeper soil layer during the late winter/spring (February–May) when the soil is moist and the soil temperature increased were referred to as ‘growth sites’ of the pathogen in this study, assuming that the soil parameters likely supported the growth of the fungus and thus, the pathogen could be detected consecutively over a several year period in the same location, over several growth seasons. Based on the definition provided by Fisher et al. Coccidioides may have been deposited on or near the soil surface after being transported from growth sites by wind, water, organisms, or anthropogenic means. We are aware that we did not investigate the activity of the pathogen in the soil or verify its growth, and that finding the pathogen in the surface layer of the soil does not mean that it cannot grow there at all. Therefore, we have to consider that some of our results might have been false negatives.Sites were C. immitis, were indicated in yellow, whereas sites that are characterized by different vegetation types appeared in various shades of green and red . Results by Landsat imagery indicated large areas west of Bakersfield as potential growth sites of the pathogen, in addition to the STH area east of Bakersfield. The city of Taft southwest of Bakersfield was completely surrounded by vegetation that is similar to the vegetation type that characterizes STH. Landsat imagery furthermore indicated small pockets of potential growth sites of C. immitis scattered throughout the Southern San Joaquin Valley and around and within the city of Bakersfield. Overall, the yellow colors indicate that about 15% of the landscape visible in the satellite image was covered with vegetation that has the same reflection pattern as the STH vegetation.A false color map of our ROI for April 28, 2008 was generated and is presented in C. immitis could be present in the neighboring environments. And indeed, yellow areas in the immediate neighborhood of the prisons, as presented by Landsat imagery, indicated the presence of potential growth sites of the pathogen.We also applied Landsat imagery to an area northwest of Bakersfield where two prisons are located near the cities of Avenal and Coalinga (Kings County and Fresno County) . In this0p is presented. This figure also presents the dependence of the fraction ƒ on 0p. It can be pointed out that when 0p drops from 1, ƒ changes from zero to 0.36 at 0p until 0p would be expected as other surface types get merged into the STH-vegetation class.Overall, 4 false color maps were generated for our ROI showing results for April 2008 to April 2011 , Chanac , and Pleito  series. These soil series are not restricted to the Southern San Joaquin Valley. See C. immitis was never detected were considered negative sites and included areas within Bakersfield city, an area northwest of Bakersfield, and 2 sites near Wasco, California. Fungal DNA could be detected in all soil samples. Of all sites investigated in 2011 , 4 sites (25%) were found negative, and 12 sites (75%) were confirmed as growth sites of the pathogen. No new accumulation sites were discovered in 2011. The site at Bear Mt. Road was the strongest growth site of C. immitis for the 2011 sampling set (positive for C. immitis from Jan–Apr). For an example of multiplex PCR results, see In addition to the two growth sites of the pathogen that were detected in 2008 and 2009  to TM bands 4, 3, and 2 respectively. With this choice, the different depths of red indicated different plant associations. These maps were the result of implementing the MLC method with C. immitis. STH site 2 was confirmed as a growth site by Frank Swatek  was added when 0p was reduced to 0.10. Sampling sites 1, 5, 6, and 11,  never got added for C. immitis ecological niche within our ROI. However, in some occasions the prediction made by satellite imagery to indicate soils that could potentially harbor the pathogen could not be confirmed by multiplex PCR. Of all 25 sites included in this study, only sites 8 and 9 (Cole's Levee Rd. I and II) fell in the STH-vegetation class in each year whenC. immitis by multiplex PCR. Other sites that were confirmed as growth sites of the fungus by our culture independent approach fell in this class at least on one occasion out of four when C. immitis growth sites by multiplex PCR, two sites were never indicated as a potential growth site by the MLC method (site 10 [Olen Ave.] and site 20 [Elementary Lne.]), but the pathogen was present in soil samples from both sites as confirmed by multiplex PCR (see By comparing results obtained by satellite imagery (STH-vegetation class) and multiplex PCR, we found that in ∼74% (17 out of 23 tested sites) the satellite imagery results and the results obtained by multiplex PCR agreed at least in one year out of four years  or 65%  . Tables  PCR see . Further2) that fell into the STH-vegetation class were observed for the sampling period until early 2014 and are displayed in table S2 in Changes in extend of areas (kmC. immitis by combining Landsat imagery (based on vegetation and soil temperature), and soil parameter information (from 25 sites) with a culture independent PCR-based method to detect the pathogen. We showed that satellite imagery, combined with soil parameter information, can provide a map of locations within our ROI, where C. immitis might reasonably be expected to be found. We were able to verify the presence of the pathogen by a multiplex PCR method in about 74%Coccidioides DNA extracted from the soil might have been under the detection limit of our PCR based methods (sites 22 and 23), or 2) the resolution of the satellite imagery might not have been detailed enough , and 3) the distribution of Coccidioides in the soil might have been spotty, and the positive site was missed . A closer look at sites where satellite imagery and soil parameter data indicated potential growth sites for C. immitis also revealed that these sites were not uniform in regard to plant coverage, distribution, and diversity, thus, generating microhabitats for soil microorganisms that most likely would be quiet distinct, especially in and around the rhizosphere C. immitis might be able to persist in soils that have been converted to agricultural fields for an unknown amount of time, but its arthroconidia might never germinate and grow into vegetative hyphae. These sites could be termed dormant sites . To assess these impacts on our results was not the focus of our work, but we are aware of these limitations. In previous research we have investigated the limitation of the multiplex PCR approach to detect C. immitis,  suggest that years with a reduced precipitation (drought) favor plants of the STH-vegetation class, but other factors likely play a role as well, such as development and changes in land use see , which wC. immits. Site 25 was never included  maybe because of the limited resolution of the satellite imagery, as discussed earlier. It is important to keep in mind that the TM-6 image tells us surface temperatures. It may very well be that what matters is the temperature below the surface Compared to the STH-vegetation class data, the STH-thermal data showed considerably more variation for the four different years, as expected. The vegetation on a certain day in each year may be very similar, but soil temperatures might be more variable in different years (data for http://www.africasoils.net/data/ldsf-description) in future studies. A time series analysis could also be considered, if feasible. Phenology development throughout the year can make the analysis more specific to a particular vegetation type. Niche modeling rather than automated classification could be considered as well to obtain a richer output that indicates variables of importance. However, a large dataset would be necessary that would include presence and also absence data of the pathogen in a certain type of soil at a certain time with presence or absence of a certain type of vegetation. Furthermore, it should be considered that broad band signatures over larger geographic areas and ecotones might not be precise enough to be useful in predicting growth sites of a pathogen, especially when the pathogen could be adapted to grow in a variety of different ecosystems.To improve the value of satellite imagery data, the actual spectral reflectance profiles of various soil components could be included to complement the satellite data , which were all loamy sands. Several sites around Bakersfield, California, that fell into one of these map units were indeed growth sites of C. immitis, and thus serve as a tool to further investigate the ecological niche occupied by the pathogen in the Southern San Joaquin Valley and beyond in more detail. Recent advances in computer processing and geographic information system and global positioning system technologies facilitate integration of remote sensing data, such as environmental parameters with disease incidence data, so that models for disease surveillance and control can be developed In conclusion, the combination of the methods used in our research can be used to generate maps that indicate potential growth sites of Figure S1http://cdec.water.ca.gov/snow_rain.html).Cumulative monthly precipitation (inches) over time for the Southern San Joaquin Valley, assembled from 5 stations , and Huntington Lake (HNT]) obtained from the California Data Exchange Center at (TIF)Click here for additional data file.File S1Supporting tables. Table S1. Detailed soil series descriptions of sites which were found to be growth sites of C. immitis. Table S2. Extend of STH-vegetation class in our ROI between 2008 and 2011 based on satellite imagery.(DOCX)Click here for additional data file.


Network meta-analysis may require substantially more resources than does a standard systematic review. One frequently asked question is “how far should I extend the network and which treatments should I include?”To explore the increase in precision from including additional evidence.We assessed the benefit of extending treatment networks in terms of precision of effect estimates and examined how this depends on network structure and relative strength of additional evidence. We introduced a “star”-shaped network. Network complexity is increased by adding more evidence connecting treatments under five evidence scenarios. We also examined the impact of heterogeneity and absence of evidence facilitating a “first-order” indirect comparison.In all scenarios, extending the network increased the precision of the A versus B treatment effect. Under a fixed-effect model, the increase in precision was modest when the existing direct A versus B evidence was already strong and was substantial when the direct evidence was weak. Under a random-effects model, the gain in precision was lower when heterogeneity was high. When evidence is available for all “first-order” indirect comparisons, including second-order evidence has limited benefit for the precision of the A versus B estimate. This is interpreted as a “ceiling effect.”Including additional evidence increases the precision of a “focal” treatment comparison of interest. Once the comparison of interest is connected to all others via “first-order” indirect evidence, there is no additional benefit in including higher order comparisons. This conclusion is generalizable to any number of treatment comparisons, which would then all be considered “focal.” The increase in precision is modest when direct evidence is already strong, or there is a high degree of heterogeneity. STAs typically evaluate a single treatment close to marketing launch, and as such the focal comparison of interest is with standard/usual care options. We note that this is true even when multiple treatments are included in a network and relative rankings reported Indirect comparisons and network meta-analysis (NMA) are increasingly common in the evaluation of multiple competing health technologies when interest lies in the relative rankings of all treatments of clinical interest fixed “decision set” of treatments  of interest) to which additional evidence (a “supplementary set” of treatments identified a priori) may be prospectively included to connect those already in the network. Such an approach has been separately described by Ades et al. NMA may be used in STAs when direct evidence from trials of A versus B is either unavailable or sparse; however, no formal guidelines exist to ensure transparency on which treatments should be included, when to extend a network, or how far it should be extended. In the absence of such guidelines, there are concerns that networks could be defined specifically to favor a particular treatment A recent case study calls for further work to evaluate network size and structure and provide generalizable findings on the added value of extending treatment networks In this article, we explore the effect of combining direct and indirect evidence in an NMA on the precision of a single pairwise comparison in a hypothetical six-treatment network. Our starting point is to assume that a literature search has been conducted and has generated a “star”-shaped starting network. We explore the effects of “extending” the network by including additional evidence situated at different points in the network. The article is structured as follows. First, we define the statistical properties of indirect comparisons. Then, we introduce the network structure and describe the different evidence scenarios considered here. The statistical method is described and findings are reported. We conclude by discussing the practical implications of the findings, make recommendations for the systematic review component of HTA, and discuss implications for NMA, in general.In a three-treatment network, an indirect estimate of the A versus B treatment effect estimate is derived as follows:Our starting point was to assume that a literature search has been conducted and has generated a network with six treatments labeled A, B, …, F, where treatments A and B form the “decision set” of treatments and the effect estimate of interest is is available for hypothetical scenarios are considered under an assumption of consistency . No assumptions are made about the observed treatment effects, and results are general for any outcome measure with our assumed input precisions. Furthermore, our conclusions are based on the relative precision across different parts of the network, rather than on the absolute value. Input precision values for each scenario are reported in Here we concentrate on a network structure in which direct evidence Equation . In eachScenario 1: Equal variance is assumed for each contrast across the network. Here, each contrast VXY= 1, where VXY is the observed variance (SE2) from a meta-analysis of X versus Y. The precision of X versus Y is defined as Scenario 2: A versus B comparison is the “weakest” link in the six-treatment network. Contrasts contributing first-order indirect evidence are also weak (imprecise), and second-order contrasts contribute even weaker evidence for A versus B. This scenario is sometimes seen when fewer trials are conducted for ethical or practical reasons, for example, in pain management for women in labor Scenario 3: The A versus B comparison is the “weakest” link in the six-treatment network, with the contrasts forming both first- and second-order indirect comparisons being stronger. In HTA, this scenario is seen when A versus B are interventions from rival manufacturers that have seldom been compared, or are compared only in a small study Scenario 4: A versus B is the strongest link in the six-treatment network, with the contrasts forming indirect comparisons being weaker. This scenario may be seen in practice when both A and B are older interventions, perhaps the criterion standards for the clinical area, and have been trialed many times Scenario 5: A versus B comparison is the strongest link, with the contrasts contributing to indirect comparisons also being strong. This scenario may be seen in practice with “me-too” pharmaceutical treatments such as selective serotonin reuptake inhibitors In (i) above, we assumed that evidence is available for every contrast in the six-treatment network. This, however, may not be the case in practice. Here, we also explore the impact of absent first-order evidence for An algebraic solution for the posterior precision 2. We based our choice of τ2 = 0.1, 0.5, and 1  on findings from a meta-epidemiological database of 234 meta-analyses that provides a range of τ = 0 to 1.33 on the log-odds scale doi:10.1016/j.jval.2015.03.1792 provides further details on the between-trial variance.Under each of Scenarios 1 to 5 outlined above,  network . The incdoi:10.1016/j.jval.2015.03.1792. Across all five scenarios, The results from the fixed-effect analyses are shown in the top left-hand panel of  network E, howeve2 = 0.1 and is least when τ2 = 1. If one considers the most conservative scenario under considerable heterogeneity , the increase in 2 = 1) than when τ2 = 0.1. When direct evidence is strong (Scenarios 4 and 5), the reverse is observed.doi:10.1016/j.jval.2015.03.1792 reports the findings from a series of sensitivity analyses to investigate the impact of both increasing the size of the variance inputs and varying them within the level of evidence for the six-treatment network. Although the increase in For completeness, doi:10.1016/j.jval.2015.03.1792).Focusing on the fixed-effect model, under Scenario 1, in the absence of evidence facilitating a first-order indirect comparison, there is a small benefit achieved by including second-order evidence; included C.If two first-order comparisons are available D, PABNMAer loops E as PABNThis exploration again illustrates a “ceiling effect” as observed in the “all available” evidence structure—there is no additional increase in doi:10.1016/j.jval.2015.03.1792.A sensitivity analysis was also conducted to investigate the impact of allowing information on the second-order evidence to outweigh that available for the direct and first-order evidence. Under a fixed-effect assumption, we note that it is only when the precision of the second-order evidence outweighs the first-order evidence by 400:1 that the precision gained by a single first-order loop is equivalent to the precision gained by a single second-order  loop. We also note that the increase in precision achieved via both a first- and second-order loop of evidence is the same as that achieved by either two second-order loops or two first-order loops. Complete findings for the six-treatment network are reported in In this article, we explored the effects of extending a six-treatment network under two structures and five different evidence scenarios, in which the amount of information available for direct and indirect evidence was varied. Under both evidence structures, extending the network increased the precision of the treatment comparison of interest, if first-order indirect evidence is unavailable and even then this gain is minimal. This can be termed a “weakest link” effect, and similar findings have been observed in simulation studies For the fixed-effect analyses assuming the absence of first-order indirect evidence, we observed that including second-order evidence affords a small increase in precision only decrease the precision of effect estimates. We did not explore this possibility in our analysis because we assumed that the heterogeneity parameter was unchanged by the addition of extra trials. Here, we also assumed that the heterogeneity parameter is common across treatment comparisons. Relaxing this assumption by introducing a heterogeneous variance model Similar to Song et al. The observed “ceiling effect” occurred when all first-order evidence had been included in the network. This effect was evident under both fixed- and random-effects models, and was not dependent on the strength or position of information in the network. Although we used a different approach, this effect is consistent with König et al.’s least two of the focal treatments, that is, first-order evidence, to form a triangular loop in a network. Beyond this, our results show little merit, in terms of precision, in searching for further higher order evidence—except in the absence of first-order indirect evidence . For further discussion of the Hawkins approach, readers are referred to Dequen et al. Although a limitation of our approach is that we focused on the single comparison A versus B, we note that conclusions will generalize to networks in which multiple treatments are of interest. For example, if the focal treatments of interest are A, B, and C, then all first-order comparisons for A, all first-order comparisons for B, and all first-order comparisons for treatment C should be included. These conclusions have implications for the scoping and searching stages of a systematic review and are supportive of the iterative search strategy proposed by Hawkins et al. Our objective in this article was to explore the increase in precision from including additional evidence. This does not depend on the observed effect estimates .


Evidence-based policy documents that are well developed by senior civil servants and are timely available can reduce the barriers to evidence utilization by health policy makers. This study examined the barriers and facilitators in developing evidence-based health policy documents from the perspective of their producers in a developing country.In a qualitative study with a framework analysis approach, we conducted semi-structured interviews using purposive and snowball sampling. A qualitative analysis software (MAXQDA-10) was used to apply the codes and manage the data. This study was theory-based and the results were compared to exploratory studies about the factors influencing evidence-based health policymaking.18 codes and three main themes of behavioral, normative, and control beliefs were identified. Factors that influence the development of evidence-based policy documents were identified by the participants: behavioral beliefs included quality of policy documents, use of resources, knowledge and innovation, being time-consuming and contextualization; normative beliefs included policy authorities, policymakers, policy administrators, and co-workers; and control beliefs included recruitment policy, performance management, empowerment, management stability, physical environment, access to evidence, policy making process, and effect of other factors.Most of the cited barriers to the development of evidence-based policy were related to control beliefs, i.e. barriers at the organizational and health system levels. This study identified the factors that influence the development of evidence-based policy documents based on the components of the theory of planned behavior. But in exploratory studies on evidence utilization by health policymakers, the identified factors were only related to control behaviors. This suggests that the theoretical approach may be preferable to the exploratory approach in identifying the barriers and facilitators of a behavior. Evidence is experienced or observed facts that support a conclusion. Research evidence is the most convincing type of evidence. It plays an important role in every stage of the policy making process—from setting agendas and developing policies to implementing and evaluating them . Evidencth World Health Assembly, and Bamako Declarations encouraged health systems to incorporate research evidence into health policymaking .Promotion of EBPDs that are well developed by SCSs and are timely available can reduce the barriers to evidence utilization by health policy makers. Studies have focused on the factors influencing the use of evidence in health policy. The present study is based onevidence-informed policymaking. This approach considers scientific evidence as only one of the inputs in the policy making process. It focuses on evidence-informed policy rather than evidence-based policy. Therefore, our focus in this study is on policy documents, an objective product, rather than the policy making which is a subjective process. This research was designed to study barriers and facilitators of developing EBPDs from the perspective of their producers in IMoH.A qualitative study with a framework analysis approach was used for data collection and analysis. Studies of the barriers and facilitators of a phenomenon are either exploratory or theory-based . In thisWe conducted semi-structured interviews with a purposive sample of EBPDs producers, directors of technical offices (D), and their senior Technicians (T) in IMoH. Those with a minimum of five years’ experience in policy development at the national level were interviewed. Snowball sampling was also used to identify key informants. IMoH consists of nine deputies, each deputy consists of a few bureaus/centers, and each bureau/center consists of several technical offices . Technical offices are the starting point for developing an EBPD in IMoH.An interview guide, informed by TPB, was developed and pilot-tested with 5 interviewees . The intAll interviews were conducted face-to-face by the first author (INMH) from January to February 2013. INMH has 12 years of experience in IHS. There was no communication between the interviewer and the participants prior to the study. To ensure confidentiality, all interviews were performed in the participant’s individual office or in the meeting room with nobody else present. Interviews lasted between 32 and 78 minutes (53 min. on average). Verbal informed consent was obtained before recording the interviews by an electronic voice recorder. The data collection continued until the point of data saturation.All of the recorded interviews were transcribed and coded based on framework analysis developed by Ritchie and Spencer. The key steps in framework analysis are: familiarization, identifying a thematic framework, indexing, charting, and mapping and interpretation . Based oThis study was based on theory and the results were compared to the findings of exploratory studies on the factors that affect evidence utilization by health policy makers. To this end, factors affecting evidence utilization by health policy makers were classified based on the TPB components . They believed that using valid evidence leads to correct identification of policy issues, adoption of the most effective policy options, selection of the best implementation strategy, and proper evaluation of policy effects. “I think the best way to properly prioritize community health needs is to refer to evidence” (T14).Behavioral beliefs refer to one’s perception of the consequences of a particular behavior. The positive or negative evaluation of behavioral beliefs determines one’s attitude toward the behavior  Figure . Most of4, 11, 15). Interviewees believed that evidence utilization by policy making organizations results in the optimum use of financial resources in the health sector, especially in developing countries where resources are scarce .Some of the directors and the majority of the technicians believed that appropriate use of resources, both in the supply and the demand side, is a positive outcome of EBPDs. They argued that much money is spent on research, and it will be a waste of financial resources if the results are not put to practice : “What will happen if we don’t support health policies by evidence and they are approved and implemented? Many valuable resources are devoted to them which could have been spent on appropriate policies which would work. This results in the waste of my country’s resources” (T1). They also believed that EBPDs improve the opportunity-cost of financial resources in the health system . “Using of evidence leads to adoption of proper policies which won’t need pilot implementation” .Some technicians argued that they have always been concerned about financial losses caused by the proposed policy options to the health system and society . They consider it a satisfying and motivating experience . Interviewees believed that new learning experiences are the least benefit of evidence review regardless of its use in the policy making process .The technicians mentioned that preparation of EBPDs results in the development of their knowledge and innovation (TDeveloping EBPDs is often so slow that we get the feeling that we’re losing opportunities” (D1).Most of the participants reported that developing EBPD is a time-consuming process. They believed that this may result in losing the chance to swiftly respond to problems. “7 & T4, 5, 9).One director and some technicians believed that developing EBPDs creates a tendency toward policies that are supported by international evidence, but may not be adaptable to the Iranian context .Normative beliefs are perceptions of social pressure to perform or avoid a particular behavior. The positive or negative judgments about normative beliefs determine the prevailing subjective norm  Figure . “The at1 & T3, 13). At times, non-governmental organizations such as patients associations and professional bodies such as Medical Associations oppose the use of evidence in health policy due to their interests . The participants believed that the scientific societies always support EBPDs .Most of the interviewees believed that policy authorities such as the Parliament, the High Council of Health, and the Vice President of Strategy support EBPDs. But sometimes regional interests of some parliamentarians lead to their opposition against evidence that supports national interests . The participants believed that those co-workers with EBPD skills support this behavior (T8).Most of the interviewees believed that policy administrators often resist change. They argued that using evidence to support health policies can decrease the resistance .All of the interviewees noted the weaknesses in the performance measurement system and the poor relationship between compensation and performance. “8 & T1, 15). They also noted that skill in the development of EBPDs is not a “must” criterion for employment in IMoH .Some interviewees believed that appointing faculty members as IMoH policy makers can facilitate the use of evidence in policy development . “Policy makers are now supporting this behavior, but there is no guarantee that future policy makers will have the same attitude” (D4).The participants noted lack of management stability as a barrier to the development of EBPDs . But they did not consider it a common strategy for all IMoH bodies involved in policy making .The majority of interviewees believed that rejection of non-evidence-based policies by senior policy authorities can facilitate the development of EBPDs , but not sufficient: “A few evidence-based health policy workshops are held, but they are not practical and useful” (T8). Moreover, the majority of interviewees mention crowded workplace as a barrier to the development of EBPDs .Most participants believed that empowering courses and workshops are essential to the development of EBPDs . “In the implementation of the family physician policy, we didn’t consider the views of providers and receivers of services; so we couldn’t identify the barriers to policy implementation correctly (E7).”Access to evidence was evaluated from three perspectives: stakeholders’ view, current data on the organization, and research evidence. Lack of facilities such as teleconference and a database to identify researchers leads to inefficient use of researchers’ and stakeholders’ views . Some of the directors and technicians considered insufficient access to the current data of health system, especially the cost of services, as a barrier to developing EBPDs . “To measure the cost-effectiveness of an intervention, we can simply use published evidence; but unfortunately, there are no valid data to estimate the cost of interventions” (D2).Another obstacle was lack of access to the full text of some articles  factors related to other components of TPB do not affect evidence utilization by health policy makers; (2) by not using TPB, these studies tend to neglect factors related to other components; or (3) the most important factors influencing evidence utilization by health policy makers and development of EBPDs are related to control beliefs. So other identified factors in this study do not have significant effects, but asking about them in the interviews resulted in talking about them.Our findings cannot be generalized to other settings because of its design. Although lack of generalizability is the major limitation of qualitative studies, they can provide important insights into the factors that affect a phenomenon through in-depth interviews with key informants .In spite of the participants’ concerns about time-consuming and contextualization, their attitude toward developing EBPDs was positive. We showed that not all the stakeholders of health policies support EBPDs. The main barriers to the development of EBPDs were related to control beliefs . This finding remind us of the 85:15 rule in Total Quality Management (TQM), which claims that 85% of quality problems are related to processes and systems, while employees may cause only 15% of quality problems . Our the