Infants under 6 months (U6M) contribute a significant proportion of the burden and mortality of severe malnutrition globally. Evidence of underlying aetiology in this population is sparse, but it is known that the group includes ex-preterm and low birthweight (LBW) infants. They represent a unique population given their dependence on breastmilk or a safe, secure alternative. Nutrition agencies and health providers struggle to make programming decisions on which interventions should be provided to this group based upon the 2013 WHO Guidelines for the ‘Management of Severe Acute Malnutrition in Infants and Young Children’ since there are no published interventional trial data focussed on this population. Interim guidance for this group might be informed by evidence of safety and efficacy in adjacent population groups.A narrative literature review was performed of systematic reviews, meta-analyses and randomised controlled trials of antimicrobial and micronutrient interventions  for malnutrition) across the population groups of low birthweight/preterm infants, infants under 6 months, infants and children over 6 months with acute malnutrition or through supplementation to breastfeeding mothers. Outcomes of interest were safety and efficacy, in terms of mortality and morbidity.Ninety-four articles were identified for inclusion within this review. None of these studied interventions exclusively in severely malnourished infants U6M. 64% reported on the safety of studied interventions. Significant heterogeneity was identified in definitions of study populations, interventions provided, and outcomes studied. The evidence for efficacy and safety across population groups is reviewed and presented for the interventions listed.The direct evidence base for medical interventions for severely malnourished infants U6M is sparse. Our review identifies a specific need for accurate micronutrient profiling and interventional studies of micronutrients and oral fluid management of diarrhoea amongst infants U6M meeting anthropometric criteria for severe malnutrition. Indirect evidence presented in this review may help shape interim policy and programming decisions as well as the future research agenda for the management of infants U6M identified as malnourished. Malnutrition remains a major problem globally with an estimated 22.4 million children under 5 years old affected by severe wasting , and witThe 2013 update of the World Health Organization (WHO) guidelines on Severe Acute Malnutrition (SAM) was the first to include specific recommendations for infants U6M, accommodating both inpatient and outpatient care of complicated  and uncomplicated  respectively . These g“the same general medical care as infants with severe acute malnutrition (SAM) who are 6 months of age or older  from nutrition agencies and health providers as to which interventions besides breastfeeding support are safe, effective and should be prioritised  in the management of acutely malnourished infants U6M  or nutritional status . By way of background, the current WHO recommendations for micronutrient supplementation in very LBW (VLBW)/LBW infants and for children with SAM are summarised in This literature review aims to summarise evidence for antimicrobial and micronutrient treatments that might inform policies for management of infants U6M identified as severely malnourished. Objectives towards this are: to identify evidence from adjacent paediatric populations  on micronutrient supplementation and antimicrobial interventions; to summarise what this evidence shows in terms of safety and efficacy; and to identify critical areas needing future research.Fig. 1. Whilst our focus is on infants U6M with severe malnutrition, where an article is cited within a malnourished population, that article’s terminology with regard to malnutrition is presented in inverted commas. For ease of interpretation, we present data in the following age categories, LBW/preterm, infants U6M, and children over 6 months (up to 18 years but with the majority of included articles focusing on 6-59months). This is to reflect the structure the current WHO SAM guidelines as U6M and over 6 months, recognising that a significant proportion of the at-risk U6M population may have been born LBW/preterm or both.Key abbreviations and definitions used throughout the article are presented in In this scoping review, PubMed and Google Scholar databases were searched on 1st October 2018 to identify systematic reviews (SRs), meta-analyses (MAs) and randomised-controlled trials (RCTs) where the full-text was available in English. Populations included in the search criteria were infants U6M , infants and children over the age of 6 months , pre-term/LBW/VLBW infants and breastfeeding mothers. Any definition of severe malnutrition was included. The interventions included were those mentioned in the current WHO guidelines for the management of SAM in those over the age of 6 months  and for The following were considered outside the scope of this review: breast milk fortifiers; pre-natal nutritional supplementation; breastfeeding support interventions; and the duration, mechanism of delivery and cost of interventions.Identified articles were screened for suitability and reviewed by the first author. Where the results of an RCT were included and presented in a subsequent MA, the article was not included unless it contained a subgroup population of interest  that was not presented or the primary population of interest in the MA. Similarly, where a study looked primarily at multiple micronutrient interventions, this was included if the effects of including or excluding a single micronutrient. When LBW is mentioned, it refers to studies that defined a population as either SGA, pre-term or LBW (<2.5 kg). When VLBW is mentioned, it refers specifically to a study of <1.5 kg birth weight.A formal GRADE assessment for quality of evidence of all included articles was beyond the scope of this review but where a GRADE classification of quality (Very Low/Low/Moderate/High) has been made in a recent SR or MA, this is presented.We identified a total of 94 articles for inclusion and review. None of these focused exclusively on severely malnourished infants U6M. Four articles (4%) examined infants U6M alongside other malnourished children over 6 months; 15 (16%) RCTs included malnourished children (both U6M and up to age 14 years). No subgroup analysis by age was presented in any of these articles. Of included articles, 60 (64%) documented the presence or absence of adverse effects or addressed safety. Our search identified marked heterogeneity in: age range of the population studied; anthropometric definitions of malnutrition; dosage and duration of intervention; and outcomes studied. This makes summarising overall direction and size of effect challenging. Key findings by intervention are shown in Current WHO guidance for antibiotic use in management of ‘SAM’ in infants U6M is to treat inpatient admissions  with intravenous (IV) antibiotics and outpatients  with oral antibiotics, such as amoxicillin. Our search identified eight key articles for inclusion in this review , one in LBW infants and one published RCT that included malnourished infants U6M). Summarising a literature-base that includes multiple drug classes with different pharmacokinetics and potential side effects is challenging but is addressed in two SRs on this topic in malnourished children .Arguments framing the usage of antibiotics for malnourished infants U6M include the higher mortality rates in this population  and the In the management of ‘SAM’ in children over 6 months, a recent MA and comment on two RCTs assessing the efficacy of oral amoxicillin compared to placebo reported a 3% increase in survival with antibiotic treatment with amoxicillin , whilst The only study to include infants U6M (as well as those over 6 months) with ‘SAM’ examined 12 month mortality with or without six months of co-trimoxazole prophylaxis after inpatient admission for ‘SAM’. This found no evidence of mortality or growth benefits .With regard to inpatient IV therapy for ‘complicated SAM’, Williams & Berkley identified one interventional trial from 1996 and argue that current recommendations are based on no supporting evidence for either the type and duration of antibiotics .In the pre-term population, one study provides evidence of a role for erythromycin in establishing exclusive breastfeeding (EBF) in pre-term infants , likely No study addressed the safety profile of oral amoxicillin or first-line parental antibiotics in malnourished infants U6M. Co-trimoxazole prophylaxis was associated with an increased risk of neutropenia (grade 4), but age-specific effects were not reported .“significant rate of adverse effects documented in antibiotic intervention group(s)”, but they do raise safety concerns about the currently recommended seven day course of gentamicin in the ‘SAM’ population, given its known renal toxicity, uncertain pharmacokinetics in the ‘SAM’ population and inability to monitor renal function and serum concentrations in many low-income settings  and one RCT that studied deworming in breastfeeding mothers.p < 0.0001)   , althoug 0.0001) , whereasctively) . More rectively) .p = 0.003) in length-for-age Z score (LAZ) in infants of treated mothers who had stool smears positive for helminth infections  . No impaIn VLBW infants, a MA shows a similar level of reduction in all-cause mortality but this is not statistically significant (GRADE –Moderate) . MA evidp = 0.029) and in annual MUAC gain  with but there seems to be no benefit of high dose supplementation compared to low dose in other RCTs  . A two tLow-dose supplementation shows the potential for significant benefit in terms of mortality and diarrhoea incidence in deficient populations and in such settings should be given. However, outside of such situations of specific clinical need, routine use cannot be currently recommended given strong evidence of mild to moderate safety concerns.More research on which populations/individuals do and which do not need extra vitamin A would be valuable.Vitamin D has diverse effects, being involved in calcium homeostasis, immune modulation, cell metabolism and growth. Global worldwide estimates of vitamin D deficiency in unsupplemented breastfed infants is 76% (18–82%) .Our search identified three MAs across infants, children and mothers and 13 RCTs across all populations. Six (one MA and five RCTs) of the included studies focused on serum vitamin D sufficiency as the outcome of interest. These were included as evidence for safety . Doses op < 0.05) in infants supplemented with vitamin D orally , and significant improvements across multiple developmental indices  /weight-for-length z score (WLZ)  . In MAs anguage) .In VLBW/LBW infants, vitamin D supplementation resulted in increases in all growth metrics  in two RCTs  but follp < 0.01), but no impact on infant growth  suggesting small benefits in terms of weight-for-age (WAZ) but reductions in some developed-setting subgroups in HAZ and rate of length gain , one MA In LBW infants, the included MA  identifip = 0.04)  . Iron wa = 0.04) .There is a lack of any strong evidence for benefits of iron supplementation in terms of mortality and morbidity but evidence of increased haemoglobin status and some neurodevelopmental benefits across age groups. Concerns, such as those raised in the WHO guidelines on iron supplementation in children , exist aFurther trials are investigating potential alternatives to simple iron salts and ways to target iron therapy.Zinc has received much attention for its role in the management of acute diarrhoea, but a plurality of roles in cell growth, immunity and metabolism continue to be identified . EstimatOur search identified a vast literature on zinc supplementation that proved challenging to summarise. We identified 24 relevant articles: two RCTs in infants and children with ‘SAM’ and four further RCTs in children over 6 months of age; one SR and seven RCTs in LBW infants; and two RCTs and nine MAs in non-malnourished infants and children. No articles were identified that looked at maternal post-natal zinc supplementation. In children, the most common zinc dosage was 20mg/day (range 5-40mg) with high dose usage in some RCTs up to 6mg/kg/day. In infants under 12 months, doses ranged from 1.78–20 mg/day and in LBW from 5–10 mg/day or 2mg/kg/day.p < 0.025), reduction in number of diarrhoeal episodes  at 8 weeks follow-up following admission for diarrhoea, and evidence of slight length gain  but not weight gain  . In combAmong children aged over 6 months, one MA showed an 18% reduction in all-cause mortality in those over 12 months , but twoIn the LBW population, concerning indices of growth, one SR of three trials identified no impact of zinc supplementation on length or weight , but foup = 0.03)  compared to low-dose (1.5 mg/kg for 15 days) zinc supplementation ( = 0.03)  and two  = 0.03) .Consistent evidence across age groups exists of zinc supplementation being associated with reduced morbidity and improved anthropometry, whilst mortality and neurodevelopmental impacts are more unclear. Zinc should not be supplemented as a high-dose, given mortality concerns, and its tolerability should be considered in the context of vomiting risk. Zinc should be supplemented as per diarrhoea guidelines for all severely malnourished infants U6M with diarrhoeal illnesses . In caseResearch is urgently needed to establish zinc requirements for malnourished infants U6M and the fact that there is some evidence of mortality benefit when supplemented over an extended period suggests that randomised trials are warranted in this age group.Folic acid is essential for DNA synthesis and repair, erythropoiesis and cellular metabolism and deficiency is clinically associated with megaloblastic anaemias and foetal neural tube defects . CharactFolic acid has garnered much attention regarding pre-conceptual and pre-natal supplementation, but our search identified comparatively little with regard to child or post-natal supplementation: no MAs in infants and children and 10 RCTs across pre-terms, infants and both well- and malnourished children. Folic acid supplementation varied from 50–250 µg/day among trials in LBW and between 50-150 µg/day in those among children.Folic acid supplementation was shown in two RCTs in infants and children to have no impact on mortality . In chilAmongst LBW infants, of three RCTs included all showed no impact of folic acid on weight gain, one noticed length gains in a subgroup of infants of birthweight >1750g, and one showed no improvement in infectious disease incidence or haemoglobin status .A safety debate around folic acid supplementation has centred on its usage in malaria-endemic settings as malaria parasites can utilise exogenous folate during co-administration with sulfadoxine/pyrimethamine (SP) . In a paired set of RCTs studying folic acid supplementation amongst children 1-36 months, the Tanzanian trial was stopped early because of increased adverse events and hospital admissions in the supplemented arm  but thisFolic acid was well tolerated in other studies with the exception of one RCT amongst Indian children that showed a two times increased incidence of persistent diarrhoea in the supplemented group , somethiThere is limited evidence of benefit in the child population in terms of morbidity and growth and therefore this is not recommended as a routine intervention for malnourished infants U6M. Any benefits that may exist are not consistent across age groups. Safety in a malaria endemic setting remains uncertain but is less concerning given SP is no longer routinely used for the treatment of malaria. Safety of folic acid supplementation should be considered in areas where SP is used for Seasonal Malarial Chemoprophylaxis (SMC) or Intermittent Preventive Treatment in pregnancy (IPTp).Further studies investigating the role of folic acid in malnourished infants U6M are warranted but the evidence presented does not identify this as a priority area for research.Given the nutritive demands on the breastfeeding mother, maternal micronutrient and macronutrient supplementation during breastfeeding is seen as a potentially promising approach to improving both maternal and infant nutritional status and multiple micronutrient and macronutrient supplementation has received increasing attention in recent years.Our search identified four MAs and one RCT covering the topics of post-natal multiple micronutrient and polyunsaturated fatty acid (PUFA) supplementation. Articles pertaining to maternal supplementation with single micronutrients are included in the section relevant to that micronutrient. Within these included studies the post-natal period was inconsistently defined but supplementation typically occurred within 2 weeks of delivery and up to 4 months post-partum.Maternal post-natal multiple micronutrient supplementation was associated with no quantitative evidence of improvement in infant and child mortality or morbidity outcomes . One of No clinically detectable adverse effects were reported in any of the included studies.We found insufficient evidence of the benefits of maternal supplementation to infants to justify routine use in current programming for malnourished infants U6M. However, potential benefits to the mother, not included in this review, should be considered and evaluated in more detail in further research to inform decisions in this area.Diarrhoea affects a large proportion of children with severe malnutrition admitted to inpatient care. Studies in sub-Saharan Africa identify the prevalence of diarrhoea at admission to hospital with ‘SAM’ at 49-67.3%, with its presence being associated with increased inpatient mortality rates  . The cur‘should not be given undiluted F-100 at any time ’ ’ ; a factoration)’ .Our search focused on the evidence for the usage of ReSoMal for oral rehydration in severe malnutrition complicated by diarrhoea. Comparisons of other formulations of ORS, polymer-based ORS and intravenous rehydration strategies were considered beyond the scope of this review.We identified one recent SR  that incp = 0.036). Stool frequency and number of patients requiring IV fluids after attempting oral rehydration was shown to be similar between treatment groups in both studies. Only Both studies reported on time to rehydration, with p < 0.01 and <0.001 respectively) and with one episode of hyponatraemic seizures (n = 130). Similarly, n = 110). The primary outcome of the included SR was the occurrence of hyponatraemia. There are no studies of ReSoMal in infants U6M. Limited evidence from inpatient studies of older children with ‘SAM’ suggests that ReSoMal is of similar efficacy in terms of rehydration to standard ORS but that there are significant safety concerns in terms of risk of hyponatraemia. On the basis of current evidence, and the fact that infants in the first few months of life are at increased risk of water and salt retention due to immature hormonal and renal excretion mechanisms, there is no reason to change current recommendations for use of ReSoMal in malnourished infants U6M. This age group may differ from older children in both risks and responses to treatment and is thus a priority area for clinical trials.This review has collated, reviewed and summarised the evidence-base for a selection of common medical interventions  that may be considered for use in infants U6M identified by current screening criteria as severely malnourished. A key finding is the lack of direct evidence for this population group. In its absence, we have identified evidence in closely-related populations, exploring consistency of effect in terms of both efficacy and safety. Even for these groups, including LBW infants and malnourished children over the age of 6 months, the evidence base is neither strong nor extensive. It does however allow us to make some tentative recommendations and establishes an initial evidence base from which policy discussions about best approaches to managing infants U6M with severe malnutrition can begin.The scope of this review is deliberately broad, particularly in using a non-specific definition of severe malnutrition and broad age group categories to reflect current WHO subdivisions. The utility of this review is derived from it gathering a vast body of evidence of disparate interventions, populations and study outcomes which may help frame policy and programming discussions as well as future research. Given such breadth, it was not possible to perform a full systematic search strategy and MA for each intervention nor was it possible to do a comprehensive assessment of quality of evidence, something that would be warranted in the future. Guidance for programmers is urgently needed, and whilst we have presented tentative recommendations, the weak and disparate evidence-base would benefit from wider expert consultation and consensus beyond those directly engaged in this review.It is notable that of included articles, most showed no effect of the investigated intervention and amongst those that did, the effects were small with broad confidence intervals. There is a lack of accurate global micronutrient profiling, partly due to technical difficulties in measurement and capacity for assessment in low-resource settings, and as a result, the majority of studies look at the impact of mass supplementation without knowledge of whether it is a nutrient deficient or replete population. Studies may therefore be underpowered to detect benefits at a population level and fail to detect important effects among specifically micronutrient deficient infants and children. In addition, on reviewing the doses and dosing regimens of included studies, there is significant heterogeneity in dosing between trials within age groups, let alone across age groups. This makes creating accurate estimates of effect challenging and leaves policy makers with difficulty as to how to interpret these null or marginal effects, and programmers uncertain as to what dosage to implement in the context of their specific population or sub-population.Little is known about micronutrient status and needs of infants U6M identified as malnourished. Although it is likely that they fall somewhere between those of LBW infants and malnourished children over 6 months of age, they may vary between those born LBW or not, or in relation to maternal capacity to develop foetal pre-natal stores, and by exclusivity of breastfeeding. Given this, we have presented the evidence for medical interventions in these two adjacent populations groups; the implications of these findings on the population of malnourished U6M and how this should shape future programming remains to be determined.The review raises several key research and policy questions which warrant urgent discussion and evaluation including: what is the micronutrient status of infants U6M with severe malnutrition and what interventions might be effective in their recovery, what is the optimal oral rehydration protocol for this age group, how might medical interventions practically be delivered alongside supporting/re-establishing EBF and finally what interventions can safely form interim policy in the absence of direct evidence?


To effectively tackle population health challenges, we must address the fundamental determinants of behaviour and health. Among other things, this will entail devoting more attention to the evaluation of upstream intervention strategies. However, merely increasing the supply of such studies is not enough. The pivotal link between research and policy or practice should be the cumulation of insight from multiple studies. If conventional evidence synthesis can be thought of as analogous to building a wall, then we can increase the supply of bricks (the number of studies), their similarity  or the strength of the mortar . However, many contemporary public health challenges seem akin to herding sheep in mountainous terrain, where ordinary walls are of limited use and a more flexible way of combining dissimilar stones (pieces of evidence) may be required. This would entail shifting towards generalising the functions of interventions, rather than their effects; towards inference to the best explanation, rather than relying on binary hypothesis-testing; and towards embracing divergent findings, to be resolved by testing theories across a cumulated body of work. In this way we might channel a spirit of pragmatic pluralism into making sense of complex sets of evidence, robust enough to support more plausible causal inference to guide action, while accepting and adapting to the reality of the public health landscape rather than wishing it were otherwise. The traditional art of dry stone walling can serve as a metaphor for the more ‘holistic sense-making’ we propose. Systematic reviews and guidance development groups frequently conclude that the available evidence about the effects of population health interventions is too diverse, flawed or inconclusive to support a more general conclusion about what should be done.In spite of all the developments in quantitative methods for primary research and evidence synthesis, we struggle to derive meaningful generalisable inferences from the evaluation of interventions in arenas such as the food, transport or welfare systems to guide and support public health action.We respond to a long-standing call for more ‘holistic sense-making’ in this arena by proposing a more eclectic, flexible and reflexive approach to building and interpreting the evidence.We show how a spirit of pragmatic pluralism might be channelled into constructing ‘dry stone walls’ of evidence, robust enough to support more plausible causal inference to guide action, while accepting and adapting to the reality of the public health landscape rather than wishing it were otherwise.We should look beyond simple notions of ‘interventions’, search for patterns and embrace the mess in evidence synthesis in order to better understand what makes for an effective public health strategy.Effectively tackling population and planetary health challenges such as climate change or diabetes requires us to address the fundamental, upstream determinants of behaviour and health in populations.2However, as the ongoing drip-feed of contested and contradictory research findings in respect of coronavirus pandemic control measures has illustrated, merely increasing the supply (and rigour) of primary studies is not enough.2particular settings is also the main apparent impediment to their generalisability. Interventions to change such things as how products are taxed, how cities are laid out or how society supports people in old age inevitably take place in particular places with particular characteristics, which vary widely across the globe and even within countries. How might we do a better job of deriving meaningful generalisable inferences from studies like this to guide and support public health action in other places?One reason for this is that studies conducted in ‘real-world’ settings are often critiqued for a lack of internal validity in comparison with randomised trials in more controlled settings. This may be compensated for by greater external validity—the likelihood of producing practice-based evidence that might be successfully translated to the systems in which others work.Thrombolysis was not routinely used to treat heart attack until the late 1980s. If the available trials had been combined in a meta-analysis, however, its effectiveness would have been established beyond reasonable doubt by 1973.In practice, however, many systematic reviews of population health improvement strategies have been more successful in delineating what we do not know than in identifying unequivocally effective interventions.9 10implementation of truly comparable interventions in multiple jurisdictions and beyond the control of researchers may be even less feasible.If population-level intervention studies were to use a common set of exposure and outcome measures, this would make meta-analysis more feasible. Important progress has been made in this respect, for example in physical activity epidemiology.If empirical intervention studies are so difficult to design, implement or combine in meta-analysis, why not make more use of observational and simulation methods? The growth of ‘big data’ and interest in the ‘quantified self’ now offer unprecedented possibilities to gather enormous quantities of information, whether from surveillance systems such as traffic cameras or portable devices unobtrusively capturing continuous geographical, physiological and other signals from individuals. This torrent of data has led us towards a contemporary version of Bentham’s panopticon, telling observers exactly what people are or have been doing, where, when and even with whom. Datasets of this breadth, depth and precision make it possible to investigate associations with an unprecedented degree of statistical power and analytical complexity, and one might assume that so long as sufficiently rich data are available to populate such analyses, more and more secure causal inference will follow. The results can also be used as inputs to tools such as systems dynamic modelling—to simulate the consequences of altering upstream determinants of health, identify new intervention points and explore to what extent the outcomes observed in one system may be generalisable to others.14exogenous factors  as we do to refining ways of eliciting ‘causal’ evidence from other datasets. Even if well-founded concerns about the representativeness and privacy implications of relying on ‘big data’ can be addressed, the resulting associational cornucopia is unlikely to help much if it contributes merely to producing ‘ever more sophisticated answers to the wrong questions’.16However, enthusiasm for increasing computational complexity in the search for causal inference from observational or simulation data should be tempered with the recognition that design-based inference is generally considered stronger than model-based inference.For example, hundreds of studies now tell us that more walking is reported in areas where it is easier and safer for people to walk and there are places for them to walk to.20The point of evidence synthesis is surely to derive more generalisable causal inference. In spite of the academic language, this is as much an applied problem as an abstract, theoretical problem; to put it another way, what can transport planners in Birmingham learn from what their counterparts did in Bogotá?5If cumulating evidence from multiple studies can be thought of as analogous to building a wall, then the ‘solutions’ outlined above can be regarded as ways of increasing the supply of bricks (the number of pieces of information), the similarity  of the bricks or the strength of the mortar . These are helpful if the aim is to build a larger and stronger conventional wall, formed of neat rows of bricks of roughly the same shape and size.Important and useful as all these approaches are, they have the potential to distract us from the real problem. Conventional brick walls work best on flat, smoothly prepared ground. Many contemporary public health challenges seem more akin to herding sheep in a mountainous landscape characterised by steep slopes, rocky outcrops and boggy ground. In this terrain, the more artisanal, bespoke and traditional solution of the dry stone wall may be more useful . Dry stowork (in what terms?), what they mean by it  and indeed what they mean by does . Sometimes, what people really mean is ‘Will it work?’—a predictive question—or ‘Did it work?’—an empirical but also a particular question, perhaps better formulated as ‘What happened?’10For most public health interventions—even well-established population screening programmes—the only honest answer to the question ‘Does it work?’ is ‘It depends’.in a context, at least not in the sense that a new clinical procedure might be introduced in a certain hospital or healthcare system. Rather than exerting an effect within a ‘moderating’ context, it may be more helpful to see these interventions as targeting and altering the context in which people live and make choices.10 23 24Why is this so difficult? Most public health interventions are at least somewhat unique to their context, which ought to be taken account of in their evaluation, and many can also be seen as interventions in complex systems.forms or their ‘active components’, and turning our attention instead to their functions—the processes and changes they evoke—or indeed their ‘spirit’.12 26But if everything is complex and context ‘is’ the intervention, what exactly might we seek to generalise from one instance to another? We tend to assume that interventions are things that work in and of themselves and might be universally generalisable, like Newton’s laws of motion.To take a well-known clinical example, some reviews have found that the type of psychotherapy offered to a patient with depression makes little difference to the outcome.theories about intervention functions by combining information from studies conducted in different situations, including studies that were not explicitly designed with this in mind.7 14 27 32 33If this idea has traction, we will need to expand the scope and flexibility of our repertoire in evidence synthesis in order to derive more concrete and defensible inferences about what to do for public health. Rather than assessing whether interventions of a particular type ‘work’ in an overall sense, this will entail aggregating evidence for and against understanding.How might we do this? We could accept the limitations of relying so heavily on testing binary statistical hypotheses about singular study outcomes,estimation with a variety of quantitative and qualitative sources of causal explanation such as causal process observations.One might counter that the principle of comparing observed and expected data applies equally to the paradigm of the randomised controlled trial. While this is true, testing theories in the way we describe entails a more radical challenge to established notions of a hierarchy of study design. For example, it is often understood that quantitative methods are for testing hypotheses, whereas qualitative (and some quantitative) methods play more subservient roles such as generating hypotheses, developing interventions or assessing their acceptability.expect confusing, divergent, mixed or unexpected patterns of results.10Some public health strategies will inevitably be more successful than others, and every ‘solution’ has the potential to generate more problems. Such uncertainty is—or at least should be—what drives scientific enquiry in the first place.While the metaphor of the dry stone wall can be applied at the level of the individual study, as shown in 3 9Many readers engaged with conducting, synthesising or applying the findings of population health intervention research are likely to agree with the editors of the Cochrane Handbook, who recently wrote that in spite of all the developments in quantitative methods for evidence synthesis, it is frequently still not possible for these ‘to provide insight beyond a commentary on what evidence has been identified’.7We need to find a better way, otherwise merely piling up more studies may leave us confronting another kind of stack—endlessly circling the runway of a conclusion on which we never seem to have clearance to land. In this paper we have responded to a long-standing call for more ‘holistic sense-making’ in this arena. We have outlined a strategy of constructing ‘dry stone walls’ of evidence: pluralist mosaics whose strength derives from the complementarity of their components rather than being found in spite of it.This will entail facing up to the challenge of working as ‘scholars, rather than just researchers’


Evidence synthesis is a vital part of evidence-informed decision-making, but high growth in the volume of research evidence over recent decades has made efficient evidence synthesis increasingly challenging. As the appreciation and need for timely and rigorous evidence synthesis continue to grow, so too will the need for tools and frameworks to conduct reviews of expanding evidence bases in an efficient and time-sensitive manner. Efforts to future-proof evidence synthesis through the development of new evidence synthesis technology (ESTech) have so far been isolated across interested individuals or groups, with no concerted effort to collaborate or build communities of practice in technology production. We established the evidence synthesis Hackathon to stimulate collaboration and the production of Free and Open Source Software and frameworks to support evidence synthesis. Here, we introduce a special series of papers on ESTech, and invite the readers of environmental evidence to submit manuscripts introducing and validating novel tools and frameworks. We hope this collection will help to consolidate ESTech development efforts and we encourage readers to join the ESTech revolution. In order to future-proof evidence synthesis against the evidence avalanche, we must support community enthusiasm for ESTech, reduce redundancy in tool design, collaborate and share capacity in tool production, and reduce inequalities in software accessibility. Evidence synthesis is a vital part of evidence-informed decision-making, and the substantial increase in the publication of systematic reviews and maps in recent years highlights that rigorous review is increasingly valued by the academic community. At the same time, however, the continuing explosion of evidence—a seemingly exponential growth in the volume of research evidence over the last few decades, referred to be some as an ‘infodemic’ —will makAs the appreciation and need for timely and rigorous evidence synthesis continue to grow, so too will the need for tools and frameworks that support users to conduct reviews of expanding evidence bases in an efficient and time-sensitive manner . Such ‘ewww.eshackathon.org) to act as a community of practice revolving around Open Science principles in evidence synthesis . The miThrough our hackathons—highly interactive workshops for evidence synthesis experts and software programmers—we aim to produce workflows and tools that are open, reproducible, based on the best available technology and methods, and supported by the community. Primarily, we hope to continue to establish and support a community of practice working on ESTech to support collaborative working towards our goals and aims.https://www.eshackathon.org/events.html) in Sweden, Australia, and remotely, across a suite of evidence synthesis methods themes and specific stages , and including both programming and discussion streams. Some 20 projects have been initiated (see https://www.eshackathon.org/projects.html), and several tools are now publicly available and in use [https://wviechtb.github.io/metafor/reference/reporter.html).To date, we have had four hackathons automated extraction of meta-data (descriptive information) and data from full texts in a reliable manner.Based on our experiences across multiple ESH events, we recognise the following areas that are in particular need of technological development:This is by no means an exhaustive list, but it highlights the range of challenges and solutions needed to move towards a more effective and fit-for-purpose ESTech and evidence synthesis landscape.The ESH series has events planned for 2021 and beyond, with a primary emphasis on remote participation and inclusion of low- and middle- income country participants. In response to the COVID-19 pandemic, in 2020 we trialled fully online events for the first time.Finding a balance between integration of existing software and innovation through the production of novel tools;Building for a fit-for-purpose future evidence synthesis environment rather than retrofitting the present and past;Creating and curating an inclusive, collaborative and supportive community of practice of evidence synthesis technologists.The ESH will focus on:https://estechseries.github.io/.We introduce here an ongoing and open special series in environmental evidence, in association with the Collaboration for environmental evidence and the Campbell Collaboration. The series is a joint endeavour across environmental evidence and the Campbell Collaboration journal, Campbell systematic reviews. Authors should direct their presubmission enquiries to the dedicated ESTech special series website, which describes the series in full: Readers are encouraged to submit commentaries , methodologies , and reviews . Authors should think carefully about the legacy of their work in the rapidly changing landscape of ESTech, and are encouraged to make use of online supplementary media to ensure their work remains up-to-date wherever possible; for example, providing a list of ESTech resources for a particular task that can be regularly updated.The series aims to cover all stages of evidence synthesis processes; from planning, through conduct, to communication. We are also interested in issues relevant to ESTech that relate to other forms of evidence synthesis than systematic reviews and systematic maps , although relevance to rigorous evidence synthesis methods must be demonstrated.The subject scope is not limited to environmental evidence synthesis and can span any subject where discipline agnostic ESTech can be discussed. We will publish papers on all aspects of ESTech including but not limited to: technology development; coordination and communities of practice; technology application in evidence syntheses; technology validation; acceptability and uptake of technology.Technologies that fill a real gap: i.e., it should introduce a new tool that did not previously exist; or, make an existing tool much easier to use or make it available to a new audience.Technologies that are broadly accessible, as appropriate for the tool in question: i.e. they should pass standard checks to ensure they work across a range of operating systems or computational contexts; they must be free-to-use  and preferably Open Source.We hope this collection will help to consolidate ESTech development efforts and we encourage readers to join the ESTech revolution. We encourage papers that fulfil the following criteria:All submissions that meet these criteria will be considered, regardless of whether they include software or research from ESH events. Moreover, while many within the evidence synthesis community already share some or all of the goals that we have discussed, we call on the readers of environmental evidence to embrace ESTech and these ideals and goals in their future work.In order to future-proof evidence synthesis against the evidence avalanche, we must support community enthusiasm for ESTech, reduce redundancy in tool design, collaborate and share capacity in tool production, and reduce inequalities in software accessibility.


Patient Blood Management (PBM) aims to optimize the care of patients who might need a blood transfusion. The International Consensus Conference on PBM (ICC‐PBM) aimed to develop evidence‐based recommendations on three topics: preoperative anaemia, red blood cell transfusion thresholds and implementation of PBM programmes. This paper reports how evidence‐based methodologies and technologies were used to enhance shared decision‐making in formulating recommendations during the ICC‐PBM.Systematic reviews on 17 PICO  questions were conducted by a Scientific Committee  according to GRADE  methodology. Evidence‐based recommendations were formulated using Consensus Development Conference methodology.We screened 17 607 references and included 145 studies. The overall certainty in the evidence of effect estimates was generally low or very low. During the ICC, plenary sessions  were followed by closed sessions where multidisciplinary decision‐making panels (>50 experts and patient organizations) formulated recommendations. Two chairs (content‐expert and methodologist) moderated each session and two rapporteurs documented the discussions. The Evidence‐to‐Decision template (GRADEpro software) was used as the central basis in the process of formulating recommendations.This ICC‐PBM resulted in 10 clinical and 12 research recommendations supported by an international stakeholder group of experts in blood transfusion. Systematic, rigorous and transparent evidence‐based methodology in a formal consensus format should be the new standard to evaluate (cost‐) effectiveness of medical treatments, such as blood transfusion. Patient Blood Management (PBM) is a patient‐focused, evidence‐based and multidisciplinary approach to optimize both the management of patients and transfusion of blood products for quality and effective patient care. It is designed to improve patient outcomes through the safe and rational use of blood and blood products and by minimizing unnecessary exposure to blood products Key areas in PBM include diagnosing and treating perioperative anaemia, implementing blood‐saving measures throughout the course of diagnosis and treatment, and transfusing patients according to accepted and evidence‐based transfusion thresholds.To develop rigorous evidence‐based guidelines, according to the AGREE II checklist GRADE is a transparent framework for developing and presenting summaries of evidence and provides a systematic approach for making clinical practice recommendations The National Institutes of Health (NIH) has developed a consensus development conference format to evaluate biomedical technologies and practices and to disseminate these results to health professionals and the public A diverse, international group of policymakers, managers, their support staff and other stakeholders indicated that the systematic consideration of the best available evidence would help to improve health system decision‐making processes Therefore, in order to enhance shared decision‐making and to formulate evidence‐based recommendations in three PBM topics  transfusion thresholds and implementation of PBM programmes), an international consortium of European, American, Canadian and Australian organizations in the field of blood transfusion organized a 2‐day (24 and 25 April 2018) International Consensus Conference on Patient Blood Management in Frankfurt/Main, Germany (ICC‐PBM 2018). The key results and conclusions/recommendations of this meeting were published elsewhere A Scientific Committee was established consisting of 23 members including one methodologist from the Centre for Evidence‐Based Practice, Belgian Red Cross (Belgium) with expertise in conducting systematic reviews and 22 subject matter experts that were appointed by the following sponsoring and participating organizations: German Red Cross Blood Transfusion Services (Germany); Grenoble University Hospital (France); National Health Services Blood & Transplant (United Kingdom); Academic Hospital of Brest (France); European Blood Alliance (The Netherlands); Canadian Blood Services (Canada); Western Health, Melbourne ; Fairview Health Services and Patient Readiness Institute, Minneapolis (USA); Ludwig Boltzmann Institute for Clinical and Experimental Traumatology, Vienna (Austria); One Blood, Orlando (USA); Institute for Immunology and Transfusion Medicine, Greifswald (Germany); Sanquin, Amsterdam (The Netherlands); Duke University School of Medicine, North Carolina (USA); Italian National Institute of Health ; University Hospital Frankfurt, Frankfurt am Main (Germany); Institute for Transfusion Medicine and Immunohaematology, Frankfurt (Germany); Australian Red Cross Blood Service ; Mount Sinai Hospital, Toronto (Canada); Établissement Français du Sang (France); Italian Society for Transfusion Medicine and Immunohaematology , Monash University, Melbourne  ) accordingly: (1) the diagnosis and management of preoperative anaemia (3 PICO questions), (2) the use of RBC transfusion thresholds (11 PICO questions) and (3) the implementation of PBM programmes (3 PICO questions). The 17 PICO questions were related to (1) definition (PICO 1), diagnosis (PICO 2) and management (PICO 3) of preoperative anaemia in adult elective surgery patients, (2) the use of RBC transfusion thresholds in intensive care and acute interventions , haematology and oncology (PICO 10 & PICO 11) and neurology (PICO 12 & PICO 13) and (3) the implementation of ‘comprehensive’ PBM programmes (PICO 15) and behavioural interventions (PICO 16) or decision support systems (PICO 17) to promote the implementation of PBM programmes. Detailed information about the PICO questions, selection criteria and corresponding search strategies are published elsewhere.A stepwise, rigorous and transparent process was used to conduct evidence reviews starting from the formulation of a PICO question and selection criteria followed by developing search strategies in 4 different databases (MEDLINE (PubMed interface), EMBASE (via Embase.com), the Cochrane Library and the Transfusion Evidence Library). The following steps included screening of relevant papers , data extraction, quality assessment and data synthesis . The evidence reviews were performed by one methodologist (per review) from the Centre for Evidence‐Based Practice (Belgian Red Cross), and revisions were done by the Scientific Committee.Detailed methodological principles to conduct the evidence reviews in the context of a clinical guideline project are published in a methodological charter n = 16, 70%), RBC transfusion thresholds  and PBM implementation . Based on the calculation of the mean rating scores and a discussion within the Scientific Committee in case of discrepant results (i.e. large variability in rating scores), final rating scores were provided.Starting from the PICO questions and the selection of outcomes, the Scientific Committee members independently assessed the relative importance of outcomes numerically on a 1–9 scale . Ranking outcomes by their relative importance can help guideline developers to focus attention, when formulating recommendations, on those outcomes that are considered most important https://gradepro.org/) to enable a summary of findings including an estimate of effect for each outcome and a quality rating of the evidence for each outcome according to the 8 GRADE criteria .A standard evidence summary template (in Word) was used to describe the PICO question, the search strategies, the search date, the selection criteria, the characteristics of included studies, the synthesis of findings, the quality of the evidence, the certainty of the body of evidence, the conclusion(s) and the references of the included studies When developing recommendations, the GRADE approach recommends that an expert panel should use the evidence reviews as the fundamental source of information  and recommends that an expert panel should subsequently formulate a conditional/weak of strong recommendation (strength) for or against an intervention (direction). In order to formulate appropriate recommendations, several items need to be considered such as the balance between benefits and harms, the quality of the evidence, values and preferences, resource use and the feasibility, equity and acceptability. These items are collected in an Evidence‐to‐Decision (EtD) framework (available in the GRADEpro software) which facilitates this consideration and enhances the formulation of recommendations in a clear and transparent manner http://www.menti.com, Stockholm, Sweden, application via smartphone or laptop). We summarized the results from the indicative opinion polls by expressing a range (%) of different answers from one or more items. Since these opinion polls were not binding but rather indicative for the decision‐making panels, no consensus definition was used. The rapporteurs used the GRADEpro software to insert the additional considerations by the general audience directly into the EtD framework  sessions (according to the three selected topics) followed by discussion with the general audience. The general audience was divided over these three sessions (according to their preference which was recorded during the registration for the conference): 46 people attended the ‘preoperative anaemia’ session, 68 people attended the session on ‘RBC transfusion thresholds’ and 61 people attended the ‘PBM implementation’ session. Each session was chaired by two experts (one subject matter expert and one methodologist), and two Scientific Committee members (per session) served as the Rapporteurs by recording the key points from the discussion with the general audience. Before the ICC‐PBM 2018, the speakers, chairs and rapporteurs were trained in the GRADE methodology and the Consensus Conference methodology via an online webinar. A detailed list of the speakers, chairs and rapporteurs can be found in Appendix At the end of day 1, further deliberation on the evidence and additional considerations by the general audience was done by the three decision‐making panels (one panel per topic) during three separate and closed (executive) sessions. The decision‐making panels consisted of 10–15 subject matter experts (including 2–3 Scientific Committee members) together with the same chairs (one subject matter expert and one methodologist) as for the open sessions. Key information from the evidence reviews together with important additional considerations of the general audience was presented via the EtD framework (directly in the GRADEpro software) by the chairs after which discussion among the decision‐making panellists was started and judgements on the different EtD items were completed. The two rapporteurs (per session) recorded the judgement decisions into the summary of judgement template of the GRADEpro software. Finally, based on this summary of judgements, a conclusion was drafted including the formulation of a conditional, strong or research recommendation (using standard wording) together with the underlying justifications.At day 2, a general plenary session with all conference attendees was organized and moderated by three chairs Appendix . During At the end of day 2, three separate and final closed (executive) sessions were organized with the decision‐making panels. Based on the plenary session presentations (i.e. GRADE’s EtD framework), discussions with the general audience and indicative opinion poll votes on the draft recommendations, changes to the draft recommendations were made (if needed) and final recommendations and justifications were formulated. Consensus among the decision‐making panellists was defined as a two‐thirds majority (via hand raising).The use of the NIH consensus development conference methodology resulted in the following four consecutive major steps:TM software).A total number of 186 persons from five continents and 35 countries were registered for the 2‐day Conference Appendix . The attThe screening of 17 607 references resulted in the final inclusion of 145 studies The ‘RBC transfusion thresholds’ session started with an introductory keynote presentation by Prof. Dr. Jeffrey L. Carson  followed by the presentation of the evidence from 40 studies that investigated the effectiveness of a more restrictive transfusion threshold compared to a more liberal transfusion threshold in the following adult patient populations: critically ill but clinically stable intensive care , orthopaedic and non‐cardiac surgery , acute  bleeding , symptomatic coronary heart disease , septic shock , cardiac surgery , haematology , oncology , neurology .The mean response rate of the indicative opinion poll was 81% (range 77–82%). About 82% of the respondents decided that there is (probably) important uncertainty or variability in how people/patients value the critical outcomes. Fifty‐four per cent indicated that the use of restrictive RBC transfusion thresholds will result in negligible costs/savings to moderate/large savings. About 60% of the people believe that the use of restrictive RBC transfusion thresholds will probably have no impact or even an increased impact on health equity. The majority of people (70‐90%) said that restrictive RBC transfusion strategies are probably acceptable and feasible to key stakeholders.In the ‘PBM implementation’ session, the evidence of 43 studies regarding the implementation of comprehensive PBM programmes  and potential behavioural interventions  or computerized decision support systems  was presented.Detailed information of all evidence reviews and presentations is available at the ICC‐PBM website During the second part of the three parallel sessions, a closed/executive session with the decision‐making panels took place. Based on the evidence reviews, the additional considerations and the indicative polling of the audience, the decision‐making panels made final judgements on the different EtD criteria. A summary of judgements for the 17 PICO questions can be found in Appendix The draft recommendations and underlying justifications from the decision‐making panels were presented to the general audience. Detailed information of these presentations and the indicative opinion poll results is available at the ICC‐PBM website..At the end of the second day, the decision‐making panels formulated final recommendations. For preoperative anaemia, four clinical and three research recommendations were formulated, including a strong recommendation to detect and manage anaemia sufficiently early before major elective surgery. For RBC transfusion thresholds, four clinical and six research recommendation were developed, including two strong clinical recommendations for critically ill but clinically stable intensive care patients with or without septic shock  as well as for patients undergoing cardiac surgery . For implementation of PBM programmes, two clinical and three research recommendations were developed, including recommendations to implement comprehensive PBM programmes and to use electronic decision support systems  to improve appropriate RBC utilization. Detailed information about the final recommendations is published elsewhere.Numerous consensus conference formats have been used to address issues in medical care For the first time in the field of PBM, and after two years of preparation, clinical bedside experts from anaesthesiology, haemostasis and thrombosis, intensive care medicine, transfusion medicine, internal medicine, neurology, clinical laboratory medicine, haematology, clinical immunology, oncology, neurosurgery, vascular, cardiac and oncological surgery, gynaecology and obstetrics met with nurses, patient and blood banking representatives, representatives of blood transfusion services and government authorities, Evidence‐Based Medicine methodologists as well as epidemiologists for a two‐day intensive exchange guided by the GRADE methodology. Prior to the meeting, significant deliberation was put into the framing of the questions to be addressed. Consistent with the NIH model, the ICC‐PBM relied on a broad‐based independent panel to bring balanced perspectives and relevant knowledge to the process. In addition to subject matter experts, both practitioners and methodologists as well as representatives of blood recipients participated. Information was presented in an open meeting format with opportunity for both panellists and audience to question and discuss. Lastly, draft recommendations (consensus statement) were generated at the end of the first day and presented to the entire group of participants on the second day for discussion and refinement. Because of the use of the GRADE methodology, the ICC‐PBM spent less time on expert presentations than is typical for most NIH Consensus Conferences. Consensus conferences are dependent on acceptance of the scientific method and a robust application of scientific rigour generates trust in the process. By thoroughly training all members of the Scientific Committee, all panellists, chairs, rapporteurs and speakers in the GRADE and EtD methodologies as well as in the use of the GRADEpro software by establishing more than ten webinars in addition to the regular teleconferences before the conference, we were able to facilitate the whole process. All critical partners had a deep and thorough methodology knowledge before attending the ICC. Furthermore, use of GRADE methodology, including a thorough assessment of the relevant literature, provided a solid foundation for consideration of the PICO questions.Together with the active involvement of the audience comprising about another 150 clinical experts, the three panels evaluated the outcome of the literature search performed following the predefined PICO questions. The panellists were able to agree on the above‐mentioned recommendations in the three predefined PBM topics on the diagnosis and treatment of preoperative anaemia, RBC transfusion thresholds and the implementation of PBM programmes The GRADE and EtD methodology enabled the three panels to integrate evidence‐based literature searches into a discussion on practical bedside implementation and guided the drafting of evidence‐based recommendations. Employing the tools described above to maintain academic rigour through the process, the three panels were able to take a holistic view of PBM. One of the major outcomes of this process are the recommendations, which go beyond the simple interpretation of the published literature to give concise advice to the bedside clinician and to patients potentially requiring RBC transfusion, ESA and/or iron support while keeping in line with the published evidence.Several challenges or limitations regarding the use of the Consensus Conference format and the GRADE approach were present. Firstly, although an enormous effort was made to inform and educate all people involved in the evidence‐based methodology (i.e. GRADE approach), the formulation of straightforward PICO questions and selection criteria, including rating the importance of outcomes, was a difficult discussion between the methodologists and the subject matter experts. This is part of the lumping–splitting debate where subject matter experts are more lumping the review questions, that is addressing a wide range of populations, interventions and outcomes. Lumped review questions are time‐consuming but will better inform decisions about which interventions to implement when there may be a range of options. On the other hand, methodologists are more in favour of splitting the review questions, that is addressing a narrow range of populations, interventions and outcomes. These type of questions are less time‐consuming. However, these reviews can only inform decisions about whether or not to implement narrowly focused interventions. Given the tight time schedule (i.e. 1 year) to complete the 17, predominantly lumped evidence reviews, in addition to the technical preparation of the ICC‐PBM 2018, we might have missed some relevant studies (selection bias). Therefore, a face‐to‐face meeting at the beginning of the project with the formulation and final approval of clinically relevant and feasible PICO questions (given the available time and resources) together with the preparation of an a priori data analysis plan should have been better to conduct evidence reviews in the most efficient way. To further improve the engagement of the Scientific Committee members, the decision‐making panels, chairs and rapporteurs, a second online/face‐to‐face meeting with the Scientific Committee, the rapporteurs and the decision‐making panels could be organized after conducting the evidence reviews to discuss the results, critical appraisal and the conclusions of these evidence reviews. Secondly, the formulation of the recommendations is the key task for decision‐making panels. Therefore, a rigorous procedure is required to select panellists that accept the principles of Evidence‐Based Medicine and to manage potential conflicts of interest in a fair, judicious and transparent manner The use of a formal Consensus Conference format in combination with the GRADE approach provides a powerful framework to formulate evidence‐based recommendations based on the best available evidence, patient perspectives and clinical expertise. This first international consensus conference on PBM could open ways to future developments in this field. First, it provides a tool to help evaluate the outcomes from this consensus conference . Second, the emitted recommendations should be reviewed and updated at least every 5 years according to the same methodology. The systematic use of evidence‐based methodologies should be the new standard to evaluate (cost‐) effectiveness of medical treatments in order to avoid recommendations based on expert opinion solely and/or strongly held opinions on religious (e.g. Jehovah Witnesses) or commercial  grounds. Third, future consensus conferences using the same methodology should be organized to elaborate other recommendations in areas not covered by the first ICC‐PBM such as PBM for platelet and plasma transfusion or PBM in paediatrics or obstetrics.Dr Van Remoortel reported receiving financial support/funding to his institution from organizers of the ICC‐PBM 2018 (see list of co‐sponsors listed for Dr Mueller below) during the conduct of the study. Dr Mueller reported receiving grants from the European Blood Alliance (EBA), American Association of Blood Banks (AABB), International Society of Blood Transfusion (ISBT), Deutsche Gesellschaft für Transfusionsmedizin und Immunhämatologie (DGTI), Societe Francaise de Transfusion Sanguine (SFTS) and Societa Italiana di Medicina Trasfusionale e Immunoematologia (SIMTI) during the conduct of the study and receiving personal fees from TerumoBCT outside the submitted work. Dr De Buck reported receiving financial support/funding to her institution from organizers of the ICC‐PBM 2018 (see list of co‐sponsors listed for Dr Mueller above) during the conduct of the study. Dr Devine reported receiving grants from Macopharma, TerumoBCT and Hemanext outside the submitted work. Dr Meybohm reported receiving grants, personal fees and nonfinancial support from B. Braun Melsungen, CSL Behring, Fresenius Kabi, and Vifor Pharma; receiving personal fees from Pharmacosmos outside the submitted work; and receiving research grants from the German Research Foundation , International Anesthesia Research Society, German Society of Anaesthesiology and Intensive Care Medicine, and European Society of Anaesthesiology. Dr Wood reported receiving grants from Celgene Corporation, CSL Behring, Australian Red Cross Blood Service, New Zealand Blood Service, Amgen, Abbvie, Bristol‐Myers Squibb, Janssen, Roche, Sanofi and Takeda outside the submitted work. Dr Seifried received personal fees from Vivor outside the submitted work and reported receiving grants from the European Blood Alliance (EBA), American Association of Blood Banks (AABB), International Society of Blood Transfusion (ISBT), Deutsche Gesellschaft für Transfusionsmedizin und Immunhämatologie (DGTI), Societe Francaise de Transfusion Sanguine (SFTS) and Societea Italiana di Medicina Transfusionale e Immunoematologia (SIMTI) during the conduct of the study. No other authors reported disclosures.Appendix S1 Overview of the 23 Scientific Committee Members.Appendix S2 Rating the importance of outcomes for all PICO questions by the Scientific Committee membersAppendix S3 Chairs, presenters and rapporteurs of the three parallel sessions (day 1) and the general plenary session (day 2)Appendix S4 List of continents and countries included in the participation list of the ICC‐PBM 2018Appendix S5 List of Institutions/Organizations that co‐sponsored or contributed during the ICCPBM 2018Appendix S6 Composition decision‐making panelsAppendix S7 Summary of judgements of the Evidence‐to‐Decision framework items relevant to the 17 PICO questionsAppendix S8 Draft recommendations of the decision‐making panels at the end of day 1Click here for additional data file.


Risk stratification is needed for patients referred to hospital eyeservices by Diabetic Eye Screening Programme UK. This requires a set of candidate predictors. The literature contains a large number of predictors. The objective of this research was to arrive at a small set of clinically important predictors for the outcome of the progression of diabetic retinopathy (DR). They need to be evidence based and readily available during the clinical consultation.Initial list of predictors was obtained from a systematic review of prediction models. We sought the clinical expert opinion using a formal qualitative study design. A series of nominal group technique meetings to shorten the list and to rank the predictors for importance by voting were held with National Health Service hospital-based clinicians involved in caring for patients with DR in the UK. We then evaluated the evidence base for the selected predictors by critically appraising the evidence.The source list was presented at nominal group meetings (n=4), attended by 44 clinicians. Twenty-five predictors from the original list were ranked as important predictors and eight new predictors were proposed. Two additional predictors were retained after evidence check. Of these 35, 21 had robust supporting evidence in the literature condensed into a set of 19 predictors by categorising DR.We identified a set of 19 clinically meaningful predictors of DR progression that can help stratify higher-risk patients referred to hospital eye services and should be considered in the development of an individual risk stratification model.A qualitative study and evidence review.Secondary eye care centres in North East, Midlands and South of England. A large list of already identified predictors, for diabetic retinopathy progression was available but had duplications and overlap.With the clinical experts’ opinion using nominal group technique and evidence evaluation we identified a shorter, evidence-based and pragmatic list of 19 predictors, which was also ranked by clinical importance.This set of candidate predictors can help stratify patients with referable diabetic retinopathy under hospital ophthalmology services through risk stratification.Diabetic retinopathy (DR) can develop in anyone with diabetes mellitus (DM) and is a major cause of blindness due to damage to, and disruption of the retina (the light-sensitive layer at the back of the eye) leading to loss of sight. DR is a consequence of changes to the blood vessels in this part of the eye leading to leakage of blood and fluid and the formation of abnormal blood vessels.1There has been a global increase in the number of adults with DM from an estimated 108 million in 1980 to 422 million in 2014.3In the UK, DR services are organised into the Diabetic Eye Screening Programme (DESP) for lower-risk patients and the hospital eye services (HES) for higher-risk patients with referable DR, with the HES providing treatment, and closer observation of patients. While services may be organised differently in other countries, the care pathways are likely to be similar for these patients. Patients with DR are referred to the HES when they develop clinical signs of the so called sight-threatening retinopathy. The clinical signs based on photographs are the only differentiating features used for this risk stratification. However, approximately 50%–70% of referralsThere have been successful attempts to optimise the diabetic screening services through risk stratification of patients with the help of a prognostic prediction model.7Predictors9A prediction model combines two or more predictors  to predict the likelihood of an outcome, for example, DR progression to a treatment requiring stage or loss of vision, before it occurs.There were two comprehensive and up to date evidence reviews on the subject of predictors. However, their perspective was quite different from ours. They were a rich resource of evidence and very useful for horizon scanning,In this qualitative study, we aimed to deduplicate the list of predictors and through clinicians input and evidence assessment to arrive at a shorter list, more clinically relevant to high-risk patients.To identify all the predictors for DR progression in the literatureTo seek clinicians’ opinion about the most important predictors among them for high-risk patients and to identify any new predictors not yet described.To assess the evidence base from published literature for each predictor identified in objective 2.To finalise a potentially parsimonious set, that is, a minimal number of predictors in a future multivariate model able to give the highest predictive performance.14 15Our recent systematic review of prognostic modelling studies for the development of DR of any type including maculopathy in patients with diabetes10NGT is a qualitative research methodology, where every meeting is a structured small group exercise allowing judgements by individuals to be pooled to arrive at a decision in an uncertain situation.29An information pack with the predictors list from the systematic review was sent to participating unit before the meeting10.1136/bmjophth-2020-000579.supp1Supplementary dataDuring the round-robin recording, the facilitator went around the table inviting one item from each participant at a time  and recording them on the flip chart. It was left open to participants to choose any number of predictors from the list provided or name their own predictors using their personal experience and insight. We requested participants to restrict their final choices to a total of 15–20 predictors.The round-robin cycles (within a meeting and between the meetings) were repeated until saturation had been achieved—a point where no new predictors were being added and thus all new information had been obtained.To ensure a maximum range of views and opinions were collected, keeping purposeful sampling in mind that is, selection of information-rich resource for the research questionReflexivityData extraction in spreadsheet and collation after every meeting were carried out by one of the coauthors involved in that meeting and validated with the paper forms with the help of a second researcher. We collated all new predictors suggested by clinical colleagues participating in these meetings, added them to the list provided and helped participants rank them for importance in prognosis prediction.Voting frequency was calculated for voting scores per predictor (summing of scores) as well as frequency of voting percentage (score achieved for the item/maximum possible score × 100) and tabulated according to their rank.29In this step, we searched the reference lists of the modelling studies identified in the systematic reviewWhile there was no direct patients and public involvement required in this research, we included all the predictors from Individualised Screening for Diabetic Retinopathy (ISDR) model which were chosen by an expert patient panelThe protocol was evaluated by the four NHS trusts where the NGT was carried out. Since the research was essentially a decision exercise based on clinicians’ expert opinion and did not involve patient data, no ethical approval was required.The list of 78 candidate predictors from the systematic review are given in We conducted four NGT meetings, within secondary care eye centres in NHS trusts in the following areas in the UK:  Two Midland trusts, (3) North East England and (4) South of England. After the fourth NGT meeting, saturation was reached. The full set of 33 predictors selected by the clinicians are shown in Following NGT, during the evidence review, two further predictors from the original list of 78 were added back to give a total of 35 predictors . ‘Systol36After NGT, residual duplication/overlap still remained, for example, ‘diabetic nephropathy’ and ‘chronic kidney disease’, ‘hypertension’, ‘diastolic blood pressure’ (DBP) and ‘SBP’, ‘dyslipidaemia’, ‘cholesterol’ and ‘statins’. ‘Diabetic nephropathy’ is the primary cause of ‘chronic kidney disease’ characterised by progressive decline of39 40We did not find any primary studies supporting the association between DR progression and ‘psychiatric illness’ or ‘diabetic neuropathy’. Among the studies found, no proven association of DR with ‘BMI’,Among the new predictors suggested by NGT, primary studies were found confirming the association for ‘early worsening’,The most common primary studies quoted were UKPDS,Only 2 out of 14 modelling studiesMost of the primary studies had multiple publications. Out of 35 predictors, 25 (71%) had good supporting evidence of predictive value from the literature . Four prWe used clinical opinion in the NGT meetings to reduce a list of 78 previous candidate predictors to a list of 25 potential predictors. The study also suggested a further eight potential predictors . After eThe new predictors suggested by the NGT made good clinical/biological sense, but four of them  have not been explored for association with DR progression, although it is possible that the last two may be operational through other variables such as glycated haemoglobin (HbA1c) rise due to uncontrolled blood sugar. The ‘only eye’ situation clearly has a higher risk of blindness because of the absence of function in the affected eye, or higher risk because of the same pathology as the lost eye but has not been investigated for its association with DR progression. The other three predictors, ‘early worsening’, ‘pregnancy’ and ‘preproliferative DR’ have a proven association with our outcome of interest and were therefore included. While specific comorbidities were considered as mentioned above, ‘comorbidities’ presence or numbers as a predictor had no supporting evidence. ‘Frequent DNA’ has been proven to add prognostic value, so has been added to the final list but is in need for further confirmation.The top three ranked predictors from NGT  were the same as the low risk of bias models’ predictor sets  chosen bet alSeveral predictors in previously reported models were associated with diabetic complications other than retinopathy—in multiple outcomes or composite outcome models.We based this study on the full list of predictors that had duplication, overlap and extrapolation of evidence from outcomes other than DR progression. In this situation with a large number of predictors already confirmed and replicated, incorporating clinical insight through a qualitative study design was invaluable to reach a shorter, more manageable list and also generate some new predictors. This is also the first study to evaluate the evidence base of potential predictors used in existing prognostic models for DR progression and to follow the Prognosis Research strategy (PROGRESS) framework recommendations.The ISDR modelThis set of predictors derived will be useful to risk stratify patients, optimise treatment strategies, inform patients about their personal risks and improve research strategies as well as providing the building blocks for future prognostic models.While we have attempted to develop a list of predictors that are useful in predicting patients who progress from referable DR to a stage of needing treatment or vision loss, the list is primarily from patients with diabetes under screening for incident DR and referable DR and as such are generalizable as markers for progression to any stages.Risk of bias assessment did not affect our decision to exclude any predictors as vast majority of the predictors had good evidence base. PROGRESS criteria was used to decide which predictors were not confirmed yet. That has helped decide where there is need for further research, for example, ‘Race’, ‘diet’, ‘exercise’ and so on.We also found the reporting of the evidence base for the predictors selected in the modelling studies suboptimal. Under-reporting has been mentioned by other observers beforeThere have been recent useful reviews and studies on ocular predictors and their natural history use as predictors of DR progressionWe have been able to identify 19 evidence-based predictors for DR progression, using a novel method (NGT) and evidence review inline with PROGRESS recommendations. This smaller and more practical set provides a useful resource for a potential model to stratify patients for risk of DR progression, to aid clinical decision-making and optimise clinical care pathways. This set is ranked in importance by the NGT.


Autism Spectrum Disorder (ASD) is a neuro-developmental disorder that affects communication and behavior with a prevalence of approximately 1% worldwide. Health outcomes of interventions for ASD are largely Participant Reported Outcomes (PROs). Specific guidelines can help support the best care for people with ASD to optimize these health outcomes but they have to adhere to standards for their development to be trustworthy.The goal of this article is to describe the new methodological standards of the Italian National Institute of Health and novel aspects of this guideline development process. This article will serve as a reference standard for future guideline development in the Italian setting.We applied the new standards of the Italian National Institute of Health to the two guidelines on diagnosis and management of children/adolescents and adults with ASD, with a focus on the scoping, panel composition, management of conflict of interest, generation and prioritization of research questions, early stakeholders’ involvement, and PROs. Recommendations are based on the Grading of Recommendations Assessment, Development and Evaluation (GRADE) Evidence-to-Decision frameworks.Following a public application process, the ISS established two multidisciplinary panels including people with ASD and/or their caregivers. Seventy-nine research questions were identified as potentially relevant for the guideline on children and adolescents with ASD and 31 for the one on adults with ASD. Questions deemed to have the highest priority were selected for inclusion in the guidelines. Other stakeholders valued their early involvement in the process which will largely focus on PROs. The panels then successfully piloted the development of recommendations using the methodological standards and process set by the ISS with a focus on PROs.In this article, we describe the development of practice guidelines that focus on PROs for the diagnosis and management of ASD based on novel methods for question prioritization and stakeholder involvement. The recommendations allow for the adoption or adaptation to international settings. Clinical Practice Guidelines (CPGs) are statements containing recommendations for clinical practice or public health policy. A recommendation describes, for the intended end-user of the guideline, what he or she can or should do in specific situations to achieve the best health outcomes possible, individually or collectively . BesidesFollowing novel national regulations on responsibilities of healthcare professionals , the ItaThe essential behavioral features of ASD are persistent impairment in reciprocal social communication and social interaction and restricted, repetitive patterns of behavior, interests, or activities. These core symptoms are present from early childhood and limit or impair everyday functioning , are ext‘The treatment of children and adolescents with ASD’. [These factors represent challenges for the development of evidence-based guidelines in this field. In 2011, the ISS published the first Italian Guideline on ASD entitled th ASD’.  The publth ASD’. . In 2015th ASD’. . Yet, thThis article introduces the methods and approach to guideline development at the ISS, laying out its innovative methods using the example of evidence-based guidelines for the diagnosis and management of ASD with a focus on PROs.https://cebgrade.mcmaster.ca/guidelinechecklistonline.html). It was intended to meet recommendations for trustworthy guidelines by the National Academy of Medicine (U.S.), formerly known as the Institute of Medicine, the World Health Organization and the GIN [The guideline development process was guided by the ISS methodological manual  and deri the GIN , 17, 18.‘Provisions on prevention, treatment, and rehabilitation of people with autism spectrum disorders, and assistance to families’  intends to ensure the health, the improvement of living conditions and the inclusion in social and working environments of individuals with ASD. The two guidelines described here will be developed in observance of the law 134. Their scope includes the diagnosis and management of ASD and requires describing the perspective, objectives, target population, and target audience.The Italian national law The guideline working group benefits from the contribution of several teams. We describe their roles and responsibilities here.The ISS Steering Committee (SC) leads and oversees the development of the guideline, it defines the groups involved  and supports their productive interaction and it is responsible for the development process including budgeting, the definition of a timeline, and the management of COI. The SC, coordinated by a principal investigator and supported by scientific and technical secretariat, selected two chairs for each guideline: one content expert and one methodological expert. The chairs are included in the SC together with a quality assurance team that ensures the compliance of the development process with the ISS methods and regulations.https://gradepro.org), work closely with the guideline panel in prioritizing the relevant questions and outcomes, prepare background documents for the guideline panel and stakeholders, coordinate teleconferences and online voting processes, review comments.The guideline group or panel is responsible for prioritization of questions for the guideline, participation in group meetings and teleconferences, providing input on evidence and contextual factors, reviewing evidence summaries, making judgments and formulating recommendations in final panel meetings, reviewing and writing of final guideline report and support for dissemination . Two sepThe Evidence Review Team (ERT) searches the literature and produces syntheses of the evidence. Following the GRADE approach, the ERT rates the certainty in the evidence, prepares the GRADE evidence tables and Evidence-to-Decision (EtD) frameworks that the panels use in formulating recommendations.The ISS policy on the management of COI follows the GIN principles for disclosure of interests and management of COI in guidelines  and it iThe working group met for the first time in a two-day meeting held at the ISS headquarter. The following activities took place: the SC outlined the scope of the guideline; guideline developers presented the existing guidelines on ASD; the working group discussed the resources and time available and agreed to produce recommendations on 16 research questions for each of the two ASD identified populations over an 18-month time period. The guideline quality assurance team presented the ISS policy on COI and collected COI disclosure forms from participants. The ERT introduced the GRADE methodology in two presentations. The first presentation served as introduction to the GRADE constructs of certainty in the evidence and strength of recommendations , 22. TheWe implemented a two-step approach that allowed panels to identify and agree on the questions to be addressed in the guidelines using the module in GRADEpro that allows for the generation and prioritization of questions and health outcomes .Guideline developers drafted a list of strategies and interventions addressed in existing CPGs on the diagnosis and management of ASD –31. We dOnce the list of candidate questions was finalized, we asked panels to rate the priority of questions on a 1 to 9 scale. We used surveys electronically generated in GRADEpro  to the groups in separate two-hour teleconferences using the mean rating score as a ranking criterion. We invited the groups to critically appraise the list and to evaluate its harmony. In particular, we asked to verify if any of the top-rated questions for inclusion could not be considered as exhaustively informative to the reader if not paired with another question that was not rated for inclusion. To achieve harmony, we also organized questions in sensible units, consisting of the smallest recommendation sets that would be informative or required for readers to avoid gaps and achieve rapid dissemination . We usedhealth outcome descriptors, to reduce the risk of introducing error that could result when panel members have different understanding of the same outcomes. We then sent GRADEpro surveys asking to add, for each question separately, potentially relevant people-important outcomes that were not yet included in the list drafted by the PRUs  over a four-week period. Example of questions used in the questionnaire are available in Additional file www.osservatorionazionaleautismo.it) where recommendations and the underlying evidence will be available for different user profiles, similar to those of the European Commission Breast Guidelines [Guideline panel members reviewed the comments that were collected electronically using a structured questionnaire (idelines .https://www.webex.com/), to allow off-site participation and visualization of content on the screen of panel members’ devices while discussion it. To pilot the online approach, we used the PanelVoice module of GRADEpro (https://gradepro.org/panel-voice/). Through electronic surveys that are integrated in the EtDs, PanelVoice enables guideline developers to facilitate the decision-making process electronically. The process starts with the collection of panel judgments on the EtD criteria  Fig. .Fig. 5CBetween June and July 2018, the steering committee received 158 applications for the two multidisciplinary and multi-professional panels of independent experts. Twenty-six applicants were not eligible because employed in private healthcare facilities or universities, had undocumented declared professional competences, their professional profile was not requested in the public selection announcement, or they applied after the submission deadline. Among the 138 who met the requirements, the SC selected 16 applicants per panel on the basis of their professional and personal experience, expertise, healthcare setting , and geographical representation. Table The SC reviewed the detailed declarations of interest of all the 158 candidate panelists. None of them was prevented from participating in the panel because of relevant COI, since all the interests declared were considered as manageable through measures such as partial exclusion or public disclosure. Afterwards, the SC evaluated the panelists’ declared and non-declared interests, the latter identified through surveillance of research projects or training activities in which experts are engaged. The SC did not identify any relevant COI that would have prevented guideline panelists from participating in the generation and selection of the research questions addressed.We abstracted interventions and management strategies from previous guidelines into 7 macro-areas to create an initial list: diagnosis and assessment of ASD core-symptoms, diagnosis and assessment of ASD associated features, comorbidities, differential diagnosis, pharmacological interventions, psychosocial interventions, other non-pharmacological interventions. Strategies within the same macro-area were categorized and grouped together by the PRU where applicable. Categorization was based on the target population (e.g. people with ASD versus their caregivers) and on the theory underpinning the interventions. Due to the availability of multiple theoretical frameworks related to non-pharmacological interventions for ASD, the latter categorization presented challenges that we solved through discussion. As for the population of children and adolescents with ASD, the process resulted in a list of 79 questions of which 27 were high-priority, 46 questions important, and 6 questions not important. As for adults, we generated a list of 31 questions of which 21 were high-priority and 10 questions important. For each population, we will develop recommendations to answer 16 research questions whereas all other questions will be mentioned as not prioritized in the guideline. Tables The panel responsible for children and adolescents with ASD rated ASD core-symptoms as critical outcomes for all research questions. Impairments in social interaction and communication, and restricted and repetitive behaviors were considered as distinct core-symptoms of ASD and rated separately. Other critical and important outcomes included quality of life, adaptive functioning skills, and parenting stress. The panel responsible for adults with ASD prioritized outcomes related to quality of life and outcomes such as social inclusion, level of independency from the caregivers, overall functioning and professional competencies. Other important outcomes included core-symptoms, behavioral disturbances, psychotic symptoms and treatments’ side effects. All outcomes were PROs.Of the 129 stakeholders that requested to comment on the list of questions identified for inclusion in the guideline, 115 met the eligibility criteria. We excluded stakeholders for the following reasons: the application process was not completed or the relationship with healthcare industries was not declared. Figure The majority of comments pertained to potentially relevant subgroups and outcomes that were not considered in the prioritized guideline questions. Many stakeholders requested clarification regarding the meaning of “standard of care” which was used to phrase some of the questions. Based on the feedback received, we reviewed the comments and improved the wording of research questions and added new sub-groups, where necessary.The questions identified for piloting the process focused on the impact of polyunsaturated fatty acids on PROs in children and adolescents with ASD. The body of evidence consisted of Randomized Controlled Trials (RCTs) and the overall certainty in the estimated effects was rated as very low owing to serious indirectness and very serious imprecision. Based on the very low certainty in the evidence of effects and uncertainty in other judgments on EtD criteria, the panel made conditional recommendations. Further details on the pilot questions, including the EtD framework with panel judgments, are available in Additional file We have described the methods and processes for guideline development at the ISS using the diagnosis and management of people with ASD as an example. It is the first guideline that follows the new ISS standards and has posed a number of methodological challenges that we addressed using novel guideline development approaches . A challThe heterogenous composition of the panels, which includes health professionals and stakeholders across a broad spectrum, reflects the complexity of the condition being addressed. The management of such large guideline groups, which encompass different professionals and potentially heterogeneous viewpoints, requires particular ability by chairpersons to conduct effective meetings. Given the broad interest in this guideline by many and diverse stakeholder groups, the process requires maximum possible transparency and we tackled this challenge through the use of GRADE EtDs, the early involvement of key stakeholders and press releases. Applying the ISS COI policy revealed the need for a cultural change. In fact, experts are often not aware that having published on the topic of interest or carried out research or professional activities in the field constitute an interest to be declared. This is important not only for disclosure purposes but also for allowing an assessment of potential conflicts and for determining measures to manage them. We provided guidance to experts in this process to enable them to recognize and declare any circumstance in which a secondary interest could interfere with the impartial performance of their duties, functions and tasks.We created a large multidisciplinary panel which include people with ASD and their caregivers and operate under a transparent policy on COI. Our process for prioritization, using a structured and transparent approach, granted equal voices to panel members and focused on PROs. Our process is supported by independent systematic reviews by the ERT which include an assessment of the certainty in the evidence according to the GRADE approach. We used health outcome descriptors to minimize the bias and improve the overall transparency of the process. Using the GRADE EtD framework, criteria and judgments that yield recommendations are transparent and allow targeting to different user profiles . ThroughWe used information technology to streamline the development process and improve efficiency. Indeed, web-based decision-making and communication tools such as of GRADEpro, StarLeaf, and Webex facilitated work logistics and decreased costs associated with in-person meetings while increasing panel members’ involvement. We promoted stakeholders’ involvement from early stages of the process. The ISS SNLG web platform ensured a transparent and participative process in which stakeholders are empowered to provide valuable feedback in several phases of the process.Guideline development requires advanced methodological skills and understanding of evidence. Although panel members are not formally required to know the details of methodology, they must get acquainted with the relevant principles in order to understand the process flow; an ability which demands appropriate training. Human and time resources to develop the syntheses of the evidence that are used to inform the guideline are a very relevant component of the development process but these resources are small compared to the cost of treatment and primary research in this area.We have described the new Italian national guideline development process during its first application in recommendations about the diagnosis and management of ASD. The process seems feasible and acceptable to key stakeholders, including guideline panel members, those synthesizing the evidence and the public. The guideline working group is now developing recommendations that will be disseminated and adopted in Italy. This guideline aims to serve as a reference standard for future guideline development in the Italian setting, and it will allow the adoption or adaptation to various settings, including international jurisdictions.Additional file 1. Interpretation of ratings for research questions and outcomes.Additional file 2. Example of questions included in the questionnaire for stakeholders.Additional file 3. Ratings of research questions for the ISS ASD guidelines.Additional file 4. EtD for the research question used to pilot the process in-person.