Injury risk in elite youth soccer players is high. Implementing an optimal training load is of utmost importance to reduce the risk of injuries.To conduct a systematic review and best evidence synthesis to explore the effects of internal and external training load on injury risk in elite youth soccer players.MEDLINE, Embase, Web of Science, CENTRAL, and CINAHL were searched up until 17 January 2020. Each article had to meet all of the following criteria: (1) the study population consisted of male elite youth soccer players aged between 12 and 21 years; (2) a longitudinal, prospective study design was used; (3) soccer-related injuries were registered ; (4) external and/or internal load parameters were described; and (5) the article was published in an English peer-reviewed scientific journal. The quality of the included articles was assessed using the Newcastle–Ottawa Quality Assessment Scale (NOS). A best evidence synthesis was performed to rank the level of evidence.Five studies  were included. Best evidence synthesis highlighted that there was moderate evidence for (1) no association between 2-, 3-, and 4-week cumulative loads for total distance covered; (2) no association between 1-week workloads (sRPE × duration); and (3) no association between A:C workload ratios (4 weeks) and injury risk. For all other comparisons, only insufficient or conflicting evidence was found.There is a paucity of evidence for an association between internal and external training load parameters and injury risk in elite youth soccer players.The online version contains supplementary material available at 10.1186/s40798-020-00296-1. There is a lack of evidence for an association between internal and external training load parameters and injury risk in elite youth soccer players.Future high-quality longitudinal research is required to further assess the role of internal and external training load on the occurrence of injuries in elite youth soccer players, where a multifactorial approach must also be applied.Soccer has evolved and games are played faster and more aggressively than in the past, requiring elevated fitness levels . To meetAdolescence, the period of life between childhood and adulthood, is a critical time in a soccer player’s career , and durImplementing an optimal training load in elite youth soccer training programs is of utmost importance to reduce the risk of injuries and attain peak performance. Training load can be categorized into two components. External load refers to the physical work performed during training or matches, such as distance covered and accelerations , whereasThe systematic review was conducted according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines . SystemaBased on the title and abstract, two reviewers  selected the articles for full-text appraisal. From this pool, the two reviewers  independently selected articles for final inclusion. Each article had to meet all of the following criteria: (1) the study population consisted of male elite youth soccer players aged between 12 and 21 years, (2) a longitudinal prospective study design was used, (3) soccer-related injuries were registered , (4) external and/or internal load parameters were described, and (5) the article was published in an English peer-reviewed scientific journal. Elite youth soccer players were defined as players who are part of an academy of an elite soccer club playing in the highest competition of their country. EndNote X8 software  was used to perform the selection process. Any disagreement over inclusion was resolved via discussion between the reviewers. In the case of disagreement, a third reviewer (JHS) was consulted. The references of all included studies were checked for other relevant articles.The following relevant data from each study were extracted: study details , study population , injury definition, workload , and measure of association . Where possible, these associations were directly extracted from the original article. For articles in which this information was not presented, associations calculated using raw data were provided. Load parameters were classified as external or internal based on the International Olympic Committee consensus statement on load in sport and risk of injury . The firThe methodological quality of each study was assessed using an adapted version of the Newcastle–Ottawa Quality Assessment Scale (NOS) for cohort studies . The oriEach included study was appraised by two authors (SV and RMvR) independently, and all discrepancies in scoring were resolved by arbitration between the two reviewers. In cases where discrepancies could not be resolved, a third reviewer (JHS) assessed the item in question. The reviewed studies were not blinded for reasons of practicality.Strong evidence: consistent findings in multiple (≥ 2) high-quality studies;Moderate evidence: consistent findings in one high-quality study and at least one low-quality study, or consistent findings in multiple low-quality studies;Insufficient evidence: only one study available; andConflicting evidence: inconsistent findings in multiple (≥ 2) studies.A best evidence synthesis was conducted to rate the strength of the evidence. The following ranking of evidence was used :Strong p < 0.05.Results of the studies reporting on a particular relationship were considered consistent when, for at least 75% of the study, results were in the same direction, as defined by An initial 12,501 articles were retrieved from database searches . After removal of 5944 duplicate records, a further 6557 irrelevant articles were removed based on title and abstract, resulting in 44 articles for full-text appraisal. Finally, five articles were included in the review  and three were of low quality (high risk of bias) Table . RevieweInsufficient evidence exists for an association between psychosocial stress and recovery, and traumatic and overuse injury risk . This hiOne high-quality study provided insufficient evidence for a lack of an association between total distance (TD) covered and injury risk (contact/traumatic) . Two stuInsufficient evidence exists for a lack of a relationship between high speed running distance (HSD), accumulated over 1–4 weeks, and the occurrence of contact/traumatic injuries .Two studies, one high quality and one low quality, investigated the association between HSD and non-contact/overuse injury risk , 27. BeiOne high-quality study investigated the relationship between the number of accelerations (ACC) and injury occurrence . InsuffiOne high-quality study evaluated the association between the duration of training and matches  and traumatic and overuse injury risk . Weekly One high-quality study investigated the association between A:C workload ratios (based on GPS and accelerometer-derived variables) and the occurrence of contact/traumatic, non-contact/overuse, and overall injuries . The risFor low chronic high speed running distance (HSD) (< 938 m), non-contact injury risk was increased by a high A:C ratio (1.41–1.96). For high chronic HSDs (> 938 m), non-contact injury occurrence was associated with a moderately high A:C ratio (0.91–1.34). However, a low ratio (0–0.36) for all chronic HSDs was associated with a reduced overall injury risk. Moreover, there is insufficient evidence for (1) an association between high A:C HSD ratios and non-contact injury occurrence (low chronic HSDs), (2) an association between a moderately high A:C HSD ratio and non-contact injury occurrence (high chronic HSDs), (3) an association between low A:C HSD ratios and overall injury occurrence, and (4) no association for all other comparisons.The risk of contact injury was increased when the A:C acceleration (ACC) ratio was > 1.77 (very high). Overall injury risk was reduced when the A:C ACC ratio was low (0–0.33) for low chronic ACC (< 1856). Furthermore, there is insufficient evidence for (1) an association between very high A:C ACC ratios and contact injury occurrence, (2) an association between a low A:C ACC ratio and reduced overall injury occurrence (low chronic ACCs), and (3) no association for all other comparisons.The risk of non-contact injury was increased when the A:C total load (TL) ratio was moderately high (0.88–1.32). A moderately low A:C TL ratio (0.44–0.88) was associated with the occurrence of contact injuries. In sum, there is insufficient evidence for (1) an association between moderately high A:C TL ratios and non-contact injury occurrence, (2) an association between moderately low A:C TL ratios and contact injury occurrence, and (3) no association for all other comparisons.Three studies, one high quality and two low quality, investigated workload as a risk factor for injury occurrence –30. All There was moderate evidence for a lack of an association between 1-week workloads and overall injury risk , 30. DelOne high-quality study described the association between the monotony of training load and the occurrence of traumatic injuries, as well as overuse injuries . There iThere is insufficient evidence for an association between the amount of effort (strain) a player experienced and the occurrence of traumatic injuries, and also insufficient evidence for no association between strain and overuse injuries .Two low-quality studies investigated the association between the acute to chronic (A:C) workload ratio and the occurrence of overall injuries , 30. DelThis systematic review evaluates the evidence concerning the effects of internal and external training load on injury risk in elite youth soccer players. According to the best evidence synthesis, there was insufficient evidence for an association between psychosocial stress and recovery  and the occurrence of injuries.With regard to external load parameters, there was moderate evidence for no association between 2-, 3-, and 4-week cumulative loads for total distance covered and injury risk. Conflicting evidence exists for the association between 2- and 4-week accumulated HSD loads and non-contact/overuse injury risk. Moreover, there was moderate evidence for a lack of an association between 3-weekly accumulated HSD loads and injury occurrence. Furthermore, insufficient evidence existed for an association between the number of accelerations, duration of training/matches, A:C workload ratios (based on GPS and accelerometer-derived variables), and the occurrence of injuries.Combined load parameters, such as monotony and strain, showed insufficient evidence for an association with injury risk. There was moderate evidence for no correlation between 1-week workloads (product of sRPE and the duration) and overall injury risk, alongside insufficient evidence for an association between 3- and 4-week cumulative workloads and injuries. Finally, moderate evidence existed for a lack of an association between A:C workload ratios (4 weeks) and the occurrence of injuries, with insufficient evidence also found for no association for A:C ratios .n = 5), with a maximum of three studies per potential risk factor, which limits the conclusions that can be drawn.Overall, the evidence for associations between internal/external load factors and the occurrence of injuries is based on a limited number of studies  and adequacy of follow-up of cohorts (item 7). None of the studies demonstrated that included soccer players were injury-free at baseline. Moreover, none of the studies reported a follow-up rate or provided a description of those soccer players who were lost between inception and follow-up. These studies are therefore more susceptible to selection and attrition bias, and, as a consequence, this will affect the generalizability of our results.In addition to the shortcomings mentioned above, there are some other explanations for the low evidence found in this systematic review. All included studies used a general injury definition, for example “Any physical complaint sustained by a player that results from a soccer match or training, resulting in time loss or medical attention” or “A non-contact/contact injury was defined as one that caused any absence from future soccer participation” , 28. UsiTo our knowledge, this is the first systematic review summarizing the available evidence for associations between internal and external training load and injury occurrence in elite youth soccer players. A strength of this systematic review is that a comprehensive literature search was conducted in five different databases, which makes it likely that all potential relevant studies have been identified. Alongside this, a best evidence synthesis was applied to summarize and evaluate the existing evidence. This methodology results in transparency in the process of evidence assessment by applying clear criteria that include the quality of the studies. However, this study also has some limitations. First, the studies included in this systematic review only included elite youth soccer players from Europe , resulting in more homogenous groups of soccer players and possibly reducing generalizability to other parts of the world. Therefore, we would like to emphasize that more research is needed regarding internal and external workload and injury risk in elite youth soccer players. Second, different operational definitions of injury are used in the included studies. This variability of definitions can impact injury estimates and, as a result, influence the association between training load and injury risk. Third, there is the potential for a publication and language bias given that only published literature, written in English, was included for the purposes of this review.After summarizing the literature, it can be concluded that there is a paucity of evidence for an association between internal and external training load parameters and injury risk in elite youth soccer players. Future high-quality longitudinal research is required to further assess the role of internal and external training load on the occurrence of injuries in elite youth soccer players, where a multifactorial approach must also be applied. In addition, the shortcomings of the included studies concerning selection and attrition bias should be taken into account.Additional file 1. Search strategy


Health care decision makers require capacity to demand and use research evidence for effective decision making. Capacity to undertake health policy and systems research (HPSR) and teaching is low in developing countries. Strengthening the capacity of producers and users of research is a more sustainable strategy for developing the field of HPSR in Africa, than relying on training in high-income countries.Data were collected from 118 participants who had received the capacity building, using a pre-tested questionnaire. Respondents included health research scientists from institutions (producers) and decision makers (users) in the public health sector, in Anambra and Enugu states, southeast Nigeria. Data were collected on participants’ progress with proposed group activities in their short- term goals; effects of these activities on evidence-informed decision making and constraints to implementing activities. Univariate analysis was done using SPSS version 16.All prioritised activities were carried out. However, responses were low. Highest response for an activity amongst producers was 39.1%, and 44.4% for users. Some of the activities implemented positively influenced changes in practice; like modification of existing policies and programme plans. There was a wide range of responses between producers of evidence (0.0–39.1%) and users (2.7–44.4%) across both study states. Lack of authority to implement activities was the major constraint (42-9-100.0% across activities), followed by financial constraints (70.6%).Capacity building intervention improved skills of a critical mass of research scientists, policymakers and practitioners, towards evidence-based decision making. Participants committed to undertake proposed activities but faced a number of constraints. These need to be addressed, especially the decision space and authority, improving funding to implement activities that influence Getting Research into Policy & Practice (GRIPP). Being at different stages of planning and implementing proposed activities; participants require continuous technical and financial support to successfully implement activities and engage meaningfully within and across professional boundaries and roles, in order to achieve short-, medium- and long- term goals. Sustainable research capacity and evidence-informed policy making require health research professionals and policymakers with in-depth scientific expertise and complementary skills that enable them to conduct independent infectious disease research with global appeal but relevant to the health priorities of their country. Hence, Health Policy and Systems Research (HPSR) is recognized as critical in the success of health interventions and programmes.Historically, the field of HPSR started to emerge a little over two decades ago. At the time, a number of challenges were identified; an agreed definition of the field, dominance of clinical and biomedical research and no demand for HPSR at the time, but a major factor and cross cutting issue was the limited capacity to undertake HPSR, and this was even more so in Low and Middle Income Countries (LMICs). AlthougAs both policymakers and communities increasingly demand better returns on investments in health, HPSR has the potential to enable health system interventions to achieve better value for money. To address this, producers and users of HPSR evidence need to be trained and empowered locally, to be more context useful. The long-term goal is to strengthen individual and institutional capacity to initiate and lead research activities in disease endemic countries, while developing national and international partnerships. HPSR is typically context-specific and to apply research evidence to policy, national-level capacity is needed , 12, andThere is a current need to build the capacity of HPSR in LMICs as this encompasses the processes of decision-making at all levels of the health system., 17, 18 There is lack of demand for research and research findings, especially in HPSR for decision-making in LMICs , 22. SomIn Nigeria, tertiary educational institutions such as universities are important to strengthening and sustaining capacity in HPSR. They, not only produce knowledge through research but are also mandated to teach the next generation of policy-makers, health professionals and researchers. HoweverIn recognition of the need to build capacity for HPSR that will lead to demand and use of HPSR for decision-making, especially for the control of endemic tropical disease in Nigeria, a 12-month implementation research project was undertaken to build the capacity of a critical mass of research scientists, policy/decision makers and practitioners who: (i) know that successful endemic disease control programmes rely on evidence-informed decision making, and that HPSR and economic evaluation are viable tools for producing research evidence; (ii) commit to undertake activities towards improving the use of research evidence in endemic disease control in their respective States; and (iii) are willing to form research and practice networks with other producers and users of evidence.The ultimate goal of the project was to build the requisite capacity that can bridge the gap between researchers, practitioners and policymakers in the study areas, and hence facilitate and enhance getting research into policy & practice (GRIPP), 26.The paper provides new knowledge on how capacity for Health Policy and Systems Research and Analysis (HPSR+A) and GRIPP was built for researchers (producers of evidence) and programme managers (users of evidence) and the level to which they used this capacity in reviewing, re-designing programming, implementing and evaluating strategies and plans for the control of endemic diseases in two states in southeast Nigeria. Hence, it shows how the level of the skills that users and producers of evidence gained from a capacity development project was used by both the users and producers of evidence whose capacities were developed to improve HPSR evidence-based decision making and production of HPSR evidence in their organisations, and how these improve their activities for the control of endemic diseases in the two states. Using two states for the study was not for direct comparison of the outputs, but rather to illustrate the need for context-specificity in planning and undertaking activities, given varying contexts. The paper therefore contributes the experiences of the study context to the growing field of HPSR and knowledge translation.This capacity building project was implemented in Anambra and Enugu states. Both states are located in the south east geo-political zone of Nigeria. Anambra and Enugu states have an estimated population of 4.2 million and 3.3 million respectively, based on an annual growth rate of 2.8% , 28. TheAs with other regions in Nigeria, prevalence of malaria is high, and the disease burden is highest among pregnant women and children under five years of age. Similarly, prevalence of Neglected Tropical Diseases (NTDs) is high in the region, 31. At This was a multi-phased mixed methods study. Phase 1Invited stakeholders included section heads of the Ministries of Health (MoH) with an emphasis on their Departments of Planning Research and Statistics (DPRS), and Departments of Public Health and Disease Control. For producers of research evidence, we invited heads of departments, core HPSR+A or HPSR-related senior researchers in the Public Health departments of the University of Nigeria Nsukka/Enugu-Campus, Enugu, Nigeria, Enugu state University, Enugu and Nnamdi Azikiwe University, Awka, Anambra state. In addition to strengthening capacity for control of NTDs and malaria, the researchers were also keen on a broader health system strengthening (HSS) and this was taken into consideration during the mapping of the users of evidence. Approvals were received from both states’ MoH for their staff to attend the workshops. Also, individual contacts were made by the research team to the different potential participants, to mobilise and sensitise them about the benefits of the project. An information sheet about the capacity building project was initially sent to identified stakeholders either physically or through e-mails. Explanatory invitation letters were physically dropped off in their offices. This was followed by phone calls and text messages, i) to confirm that letters/e-mails were received and ii) to check availability of the proposed participant for the baseline capacity assessment and subsequent workshops. We started by identifying all potential stakeholders and prioritised the invitations based on their position and area of work, i.e. prioritizing NTDs, malaria, MCH and stakeholders who were in a position to influence decisions that would impact on GRIPP. However, participants were also required to step-down the training to other relevant statekholders in their organizations, whom were then included in the evaluation. Workshops included an initial introductory and capacity needs assessment workshop and subsequently, three capacity building workshops . After the initial braod introductory workshop, based on the resources available to the researchers, we invited a total of 60 participants (30 producers and 30 users) from Anambra and Enugu states for capacity building workshops 1 and 2, and for workshop 3, the number was increased to 120 (60 producers and 60 users) in both states. All initial participants for workshop 1 were also expected to attend workshop 2 and then a wider cohort was expected for workshop 3 (step down). Hence, participants who attended the third workshop consisted of some people who attended the first two workshops and some who did not attend either of the first two workshops. The actual attendance by participants is outlined in the Results section.The baseline capacity assessment workshop was used to introduce the project to the participants face to face and assess their HPSR + A capacity needs. The needs assessment was conducted using a self-administered questionnaire. Questionnaires were also given to relevant stakeholders who could not attend the sensitization workshop. In-depth interviews on capacity gaps and needs were also conducted on a sub-set of participants, purposively selected based on their position in their organization or on their roles in various health programmes. Details of the process and findings from the quantitative and qualitative situational analysis have been published and are available as open access., 36In workshops 1 and 2, we built the capacity of participants in HPSR+A and techniques (GRIPP). In the third workshop, we stepped down the training that was provided in the first and second workshops to a wider group. Each workshop was held for 2 days. The first and second workshops were a mix of producers and users of evidence in attendance. The third workshop consisted of one day of parallel sessions on strategies of GRIPP for producers and users of HPSR evidence. On the second day of the third workshop, a combined session for producers and users of evidence was held, during which participants were grouped into four thematic knowledge networks based on their previous area of work or interests, or their current work or interest. It was ensured that there was at least one top level manager/head of department in each thematic group. The thematic networks comprised both producers and users of evidence and are namely, NTDs, malaria, MCH, and HSS. In each workshop, the programme was scheduled to start with the lectures first, followed by interacive sessions for clarifications and then group work, with short breaks in between. During group works, participants were divided into either thematic or topic groups and allocated a faclitator from the research team and a moderator who could be a participant. At the end of group works, participants came back together in the larger group to feed back the group work outputs.Lectures were delivered on HPSR+A and Health Economics using power point presentations, followed by group work, to reinforce understanding using case studies. Prior to the lectures, a test of 15–20 multiple choice questions on each subject was administered to participants and their scores were recorded. After the lectures, the same questions were re-administered to check understanding. The HPSR topics were- Introduction to Health systems, the WHO Building Blocks, Introduction to Policy Analysis and Stakeholder Analysis. For Health Economics, the concepts of Demand and Supply, fundamentals of assessing Cost-Effectiveness and Basic Principles and Importance of Costing Health interventions were taught. These topics were all geared towards improving evidence-based health care decision making for policies and programmes.This workshop was used to introduce the concept of GRIPP. It focused on researcher-user engagement and different models of GRIPP that have worked for the evidence producers in the past. Strategies of communicating research findings were discussed with participants based on previous activities and experiences of the HPRG researchers which had produced positive results.The third workshop consisted of one day of parallel sessions for producers and users of evidence. Participants (producers and users) who had been trained in workshops 1 and 2 were now made trainers to step down the knowledge to a wider audience. On the second day of this third workshop, a combined session for producers and users of evidence was held, during which participants were grouped into four thematic knowledge networks  based on their area of work in their organization or interests. It was ensured that there was at least one top level manager/head of department in each thematic group. In their various groups, they proposed activities they were going to carry out in their various organizations in the short term (three months), medium and long term, after the workshop. Although, three-month post intervention evaluation may be considered a short period, given other competing responsibilities of participants , a longeA total of 118 producers and users of research evidence who attended the workshops were surveyed. Respondents included researchers, academia, policy makers, programme/project managers, M&E officers, directors in health agencies and parastatals, members of civil society organizations and media representatives as earlier outlined.Two structured questionnaires (one for producers and the other for users) were used to collect information on participants’ progress with implementation of proposed short term HPSR+A activities and the effects of these activities on getting research evidence into policy and practice within their respective organizations, units or departments. The questionnaires were designed for this study using information on participants’ planned activities for improving evidence-based decision making for control of endemic tropical diseases. Hence, the questionnaires were developed by the researchers. following literature review and also drawing from a previous capacity needs assessment carried out by the researchers . InformaDate were entered and analysed with the aid of the Statistical Package for Social Sciences (SPSS) Version 16. The participants’ proposed activities for GRIPP, plans, and strategies in the control of endemic diseases were disaggregated by their thematic areas. Response to variables were summarised in percentages and presented separately for producers and users respectively. Because we are not comparing the two states, given that the activities proposed were different, both states are presented on the same table.Ethical approval (NHREC/05/01/2008B-FWA00002458-IRB00002323) was obtained from the Health Research Ethics Committee of the University of Nigeria Teaching Hospital, Enugu, Enugu State, Nigeria. Written informed consent was obtained from respondents and all questionnaires were anonymised using respondent codes.In the results section, we first present the socio-demographic characteristics of participants followed by what progress had been made with implementing identified activities in three months. We then present what GRIPP activities had been carried out as a result of implementing these activities or as a result of applying knowledge and insights gained from the capacity building workshops. We also present reasons given by participants for not implementing the tasks as proposed.All 118 questionnaires administered were retrieved and included in the analysis. These comprised 23 producers of evidence and 37 users of evidence in Anambra state, and 31 producers of evidence and 27 users of evidence in Enugu state. Attendance rate at all workshops was high as outlined in For their short-term goal activities, which were evaluated and reported in this study, participants identified a number of activities . Given the relatively short interval (three months), participants were advised to realistically prioritize one or two activities per thematic group. It is those prioritized activities that we report below. Selected activities which participants identified as key, in their respective groups are outlined below but results of entire list of activities undertaken will be attached as additional tables. Although tasks were identified in groups, participants were encouraged to share updates across groups during the execution period as they were all in the same organization . Based on this, evaluation questionnaires were also administered to individual participants and not as a group, consequently the unit of analysis was also on the participant  level.Results of implementation of activities identified as key, by producers of evidence in both states are presented below . AnambraIn Anambra state, the activity where the most progress had been made was undertaking stakeholder sensitization on evidence informed decision making (32.4%) and step down capacity building of other colleagues in their organization on evidence informed decision making (29.7%). Advocacy visits to women groups for improved MCH decision making was the key activity that been most implemented by users of evidence in Enugu state. Progress with other activities are shown in Tables GRIPP outcomes of implementation of proposed activities were more pronounced among users of evidence in Enugu state compared to producers of evidence; 9.7% of producers of evidence reported they had disseminated information on malaria control program to policymakers, and 3.7% each reported they had disseminated findings on human resource for MCH services to key stakeholders through policy briefs. Among users of evidence in the State, 14.8% stated that advocacy had been paid to decision makers in health and other stakeholders; 22.2% said they had been involved in knowledge translation and use and management within malaria control unit and an equal proportion said they were involved in knowledge network for free MCH; 29.6% reported that they had interacted with HPRG formally since after the workshop. With respect to modification of policies and plans, 33.3%of users of evidence in the State reported that their activities resulted in modifications to health policies, plans and strategies in malaria control, NTD, MCH and health systems strengthening; 37% reported that activities had resulted in modification of M&E frameworks; and 29.6% said the activities had influenced research priority setting in their organization. A particular thematic network that was formed during the workshops, “Maternal and Child Health (MCH) network” was able to launch a social media platform, which has enabled the Enugu Ministry of Health as an organization to be socially visible and get feedback from external customers, with the simultaneous enablement for internal communication between staff.Respondents were also asked for reasons for not having implemented proposed activities, where this was the case. The options were given in the questionnaire were i) time constraints; ii) financial constraints; iii) Lack of authority to implement activity; iv) Lack of support from team; v) Loss of interest in proposed activity after the capacity building workshop; vi) Other. Multiple responses were allowed. Whilst lack of authority to implement was the most common reason for all activities proposed by Anambra producers, most common reasons amongst Enugu producers were more varied, but with lack of authority still being most common reasons shown in This study looked at whether building the capacity of relevant stakeholders can lead to enhanced evidence-based decision making for better control of endemic diseases in Nigeria. The results highlight three main findings i) producer-user engagement which resulted in participants undertaking to change their practice within a short time (3 months); ii) Implementation of activities as identified despite constraints iii) Modification of practice towards improving control of endemic diseases and overall health system strengthening. Reasons for not engaging in proposed activities were also highlighted and these were also qualitatively explored in the larger study of which this is a component.A variety of producers and users of evidence, as seen from the socio-demographics actively participated in the capacity building workshops. This engagement afforded an opportunity to reinforce producer-user dialogue, the absence of which is known in the literature to constrain GRIPP. Weak linkages between producers and users of evidence have been reported severally in literature–39 as a Participants committed and took initiatives to implement activities to change or improve their practices and were reported to have already implemented some activities, while some others were in progress, in the short period of three months following the capacity building exercise. In the last two decades a number of strategies have progressively encouraged these engagements in different country contexts as reported in a recent summary of the progress of HPSR. These were in form of events or publications.Participants were able to influence some GRIPP activities in their organization by applying new skills learnt during the capacity building. As a result of new skills and competencies, participants reported successes in influencing a number of GRIPP activities in both states, namely, revising programme plans and existing M&E frameworks, and also incorporating evidence for better malaria control. These are very encouraging steps towards evidence-based practice, especially as this was just within three months of capacity building. Secondly, this covertly also implies a changing mindset which is key to changing practice, and begins to challenge resistance to change, which has been reported as one of the constraints to achieving GRIPP, amongst others.The study also throws up a number of constraints to achieving these goals in the study contexts. Of the various reasons given by participants for not implementing proposed activities, lack of authority was the major reason amongst producers of evidence, in both states and amongst users of evidence in Enugu state. Lack of authority reported by users, is a concern, because it was felt that, given the higher position of some of the participants, they would be able to implement some proposed changes, even if not all, and for this reason, were purposively allocated to be present in each thematic group during the capacity building workshop. Although, these were not tested for statistical significance because of small sample sizes, the finding forms an initial step to interrogating an implied narrow or non-existent decision space, which is being explored in the qualitative component of this project.Financial constraint was the second most common reason for not implementing proposed activities, while time constraint was the third common reason for not applying evidence from HPSR for decision making. Time constraint may be as a result of the short duration (3 months) between capacity building and evaluation. However, this may also be due to participants not having yet changed mindsets adequately to prioritise these activities, given that there will always be competing activities and deadlines from other activities. Financial constraints along with time constraints have also been reported in previous studies in Nigeria and other LMICs , 43, 44.Another study carried out in a different state in Nigeria recognises that strategies as employed in our study have been shown to improve evidence-based policy making and implementation in developed countries and are likely to produce better outcomes in developing countries like Nigeria, if these constraints are addressed. The stuThese findings from evaluation of participants’ short-term goals reveal that they will require further support in completing their short-term goals, since some of the activities were still in progress, and also for carrying out their medium- and long- term goals. The main cross cutting constraints to the use of evidence from HPSR for decision making were limited decision space/authority to make GRIPP changes, time and funding. Continued support is needed for longevity and sustainability, in order to realise the HPSR mandate of building a critical mass in LMICs , 45.Finally, for sustaining GRIPP, mainstreaming HPSR as a subject into under- and post-graduate teaching in developing countries, building a collective of emerging leaders in HPSR and aligning HPSR capacity strengthening within the wider organizational development are also key strategies to be embarked upon, going forward , 38. A cThe strengths of this study are that it not only created the awareness on the knowledge and capacity gaps for HPSR production and use, but also embarked on filling identified gaps. The previous experience of the researchers in HPSR and the already established linkages and trust that the team has with most of the decision makers (research users) also facilitated the project.A limitation of the study was the short duration between the implementation of the interventions and the evaluation of their effects in terms of the participants’ goals. This was done in order to include at least one evaluation within the project period which was just 12 months. Studies have shown that further support is necessary for achievement of set goals, especially the long term ones , 48. It This study highlighted key findings from a 12 month implementation research project which built capacity of particpants. The key messages are that producers and users of HPSR evidence should engage each other at every opportunity. Getting research evidence into policy and practice can be initiated by making little changes in everyday practices of all stakeholders. Outstanding challenges to these activities, especially decision spaces, time and financial constraints need to be actively addressed, moving forward, not just for improving control of endemic diseases but towards strengthening the overall health system. Undertaking the study in two states helped to show the divergent and convergent outputs from two states that are neighbours and that had the same capacity development activity. Although, there was no direct comparison, it helps to illustrate the need for context-specificity in planning and undertaking activities.


The complexity of public health interventions create challenges in evaluating their effectiveness. There have been huge advancements in quantitative evidence synthesis methods development  for dealing with heterogeneity of intervention effects, inappropriate ‘lumping’ of interventions, adjusting for different populations and outcomes and the inclusion of various study types. Growing awareness of the importance of using all available evidence has led to the publication of guidance documents for implementing methods to improve decision making by answering policy relevant questions.The first part of this paper reviews the methods used to synthesise quantitative effectiveness evidence in public health guidelines by the National Institute for Health and Care Excellence (NICE) that had been published or updated since the previous review in 2012 until the 19th August 2019.The second part of this paper provides an update of the statistical methods and explains how they address issues related to evaluating effectiveness evidence of public health interventions.The proportion of NICE public health guidelines that used a meta-analysis as part of the synthesis of effectiveness evidence has increased since the previous review in 2012 from 23% (9 out of 39) to 31% (14 out of 45). The proportion of NICE guidelines that synthesised the evidence using only a narrative review decreased from 74% (29 out of 39) to 60% (27 out of 45).An application in the prevention of accidents in children at home illustrated how the choice of synthesis methods can enable more informed decision making by defining and estimating the effectiveness of more distinct interventions, including combinations of intervention components, and identifying subgroups in which interventions are most effective.Despite methodology development and the publication of guidance documents to address issues in public health intervention evaluation since the original review, NICE public health guidelines are not making full use of meta-analysis and other tools that would provide decision makers with fuller information with which to develop policy. There is an evident need to facilitate the translation of the synthesis methods into a public health context and encourage the use of methods to improve decision making.The online version contains supplementary material available at (10.1186/s12889-021-10162-8). To make well-informed decisions and provide the best guidance in health care policy, it is essential to have a clear framework for synthesising good quality evidence on the effectiveness and cost-effectiveness of health interventions. There is a broad range of methods available for evidence synthesis. Narrative reviews provide a qualitative summary of the effectiveness of the interventions. Meta-analysis is a statistical method that pools evidence from multiple independent sources . Meta-anMany challenges are faced in the synthesis of public health interventions. There is often increased methodological heterogeneity due to the inclusion of different study designs. Interventions are often poorly described in the literature which may result in variation within the intervention groups. There can be a wide range of outcomes, whose definitions are not consistent across studies. Intermediate, or surrogate, outcomes are often used in studies evaluating public health interventions . In addiA methodological review of NICE public health intervention guidelines by Achana et al. (2014) found that meta-analysis methods were not being used . The firThe second part of this paper aims to illustrate what methods are available to address the challenges of public health intervention evidence synthesis. Synthesis methods that go beyond a pairwise meta-analysis are illustrated through the application to a case study in public health and are discussed to understand how evidence synthesis methods can enable more informed decision making.The third part of this paper presents software, guidance documents and web tools for methods that aim to make appropriate evidence synthesis of public health interventions more accessible. Recommendations for future research and guidance production that can improve the uptake of these methods in a public health context are discussed.The National Institute for Health and Care Excellence (NICE) was established in 1999 as a health authority to provide guidance on new medical technologies to the NHS in England and Wales . Using aThe paper by Achana et al. (2014) explored the use of evidence synthesis methodology in NICE public health intervention guidelines published between 2006 and 2012 . The authttps://www.nice.org.uk/guidance) by searching the ‘Guidance and Advice List’ and filtering by ‘Public Health Guidelines’  be valuable for efficiently targeting preventive strategies to the specific subsets of a population that will derive maximal benefit’ . Tailoring interventions has the potential to reduce the effect of inequalities in social factors that are influencing the health of the population. Identifying which interventions should be targeted to which subgroups can also lead to better public health outcomes and help to allocate scarce NHS resources. Research interest, therefore, lies in identifying participant level covariate-intervention interactions.IPD meta-analysis uses data at the individual level to overcome ecological bias. The interpretation of IPD meta-analysis is more relevant in the case of using participant characteristics as covariates since the interpretation of the covariate-intervention interaction is at the individual level rather than the study level. This means that it can answer the research question: ‘which interventions work best in subgroups of the population?’. IPD meta-analyses are considered to be the gold standard for evidence synthesis since it increases the power of the analysis to identify covariate-intervention interactions and it has the ability to reduce the effect of ecological bias compared to aggregated data alone. IPD meta-analysis can also help to overcome scarcity of data issues and has been shown to have higher power and reduce the uncertainty in the estimates compared to analysis including only summary aggregate data .Despite the advantages of including IPD in a meta-analysis, in reality it is often very time consuming and difficult to collect IPD for all of the studies . AlthougAn NMA including IPD, where it is available, was performed, based on the model presented in Samarago et al. (2012) . The resIn many public health research settings the complex interventions are comprised of a number of components. An NMA can compare all of the interventions in a network as they are implemented in the original trials. However, NMA does not tell us which components of the complex intervention are attributable to this effect. It could be that particular components, or the interacting effect of multiple components, are driving the effectiveness and other components are not as effective. Often, trials have not directly compared every combination of components as there are so many component combination options, it would be inefficient and impractical. Component NMA was developed by Welton et al. (2009) to estimate the effect of each component of the complex interventions and combination of components in a network, in the absence of direct trial evidence and answers the question: ‘are interventions with a particular component or combination of components effective?’ . For exaThe results of the analyses of the main effects, two way effects and full effects models are shown in Table In practice, this method has rarely been used since its development –26. It mThe analysis and methods described in this paper only cover a small subset of the methods that have been developed in meta-analysis in recent years. Methods that aim to assess the quality of evidence supporting a NMA and how to quantify how much the evidence could change due to potential biases or sampling variation before the recommendation changes have been developed , 29. ModThis paper only considered an example with a binary outcome. All of the methods described have also been adapted for other outcome measures. For example, the Technical Support Document 2 proposed a Bayesian generalised linear modelling framework to synthesise other outcome measures. More information and models for continuous and time-to-event data is available elsewhere , 35–38.In the previous section, meta-analytic methods that answer more policy relevant questions were demonstrated. However, as shown by the update to the review, methods such as these are still under-utilised. It is suspected from the NICE public health review that one of the reasons for the lack of uptake of methods in public health could be due to common software choices, such as RevMan, being limited in their flexibility for statistical methods.Table In this paper, the network plot in Figs. WinBUGS is a flexible tool for fitting complex models in a Bayesian framework. The NICE Decision Support Unit produced a series of Evidence Synthesis Technical Support Documents  that proThe software, R2WinBUGS, was used in the analysis of the motivating example. Increasing numbers of researchers are using R and so packages that can be used to link the two softwares by calling BUGS models in R, packages such as R2WinBUGS, can improve the accessibility of Bayesian methods . The newThis paper has reviewed NICE public health intervention guidelines to identify the methods that are currently being used to synthesise effectiveness evidence to inform public health decision making. A previous review from 2012 was updated to see how method utilisation has changed. Methods have been developed since the previous review and these were applied to an example dataset to show how methods can answer more policy relevant questions. Resources and guidelines for implementing these methods were signposted to encourage uptake.The review found that the proportion of NICE guidelines containing effectiveness evidence summarised using meta-analysis methods has increased since the original review, but remains low. The majority of the reviews presented only narrative summaries of the evidence - a similar result to the original review. In recent years, there has been an increased awareness of the need to improve decision making by using all of the available evidence. As a result, this has led to the development of new methods, easier application in standard statistical software packages, and guidance documents. Based on this, it would have been expected that their implementation would rise in recent years to reflect this, but the results of the review update showed no such increasing pattern.A high proportion of NICE guideline reports did not provide a reason for not applying quantitative evidence synthesis methods. Possible explanations for this could be time or resource constraints, lack of statistical expertise, being unaware of the available methods or poor reporting. Reporting guidelines, such as the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA), should be updated to emphasise the importance of documenting reasons for not applying methods, as this can direct future research to improve uptake.Where it was specified, the most common reported reason for not conducting a meta-analysis was heterogeneity. Often in public health, the data is heterogeneous due to the differences between studies in population, design, interventions or outcomes. A common misconception is that the presence of heterogeneity implies that it is not possible to pool the data. Meta-analytic methods can be used to investigate the sources of heterogeneity, as demonstrated in the NMA of the motivating example, and the use of IPD is recommended where possible to improve the precision of the results and reduce the effect of ecological bias. Although caution should be exercised in the interpretation of the results, quantitative synthesis methods provide a stronger basis for making decisions than narrative accounts because they explicitly quantify the heterogeneity and seek to explain it where possible.The review also found that the most common software to perform the synthesis was RevMan. RevMan is very limited in its ability to perform advanced statistical analyses, beyond that of pairwise meta-analysis, which might explain the above findings. Standard software code is being developed to help make statistical methodology and application more accessible and guidance documents are becoming increasingly available.The evaluation of public health interventions can be problematic due to the number and complexity of the interventions. NMA methods were applied to a real Cochrane public health review dataset. The methods that were demonstrated showed ways to address some of these issues, including the use of NMA for multiple interventions, the inclusion of covariates as both aggregated data and IPD to explain heterogeneity, and the extension to component network meta-analysis for guiding future research. These analyses illustrated how the choice of synthesis methods can enable more informed decision making by allowing more distinct interventions, and combinations of intervention components, to be defined and their effectiveness estimated. It also demonstrated the potential to target interventions to population subgroups where they are likely to be most effective. However, the application of component NMA to the motivating example has also demonstrated the issues around uncertainty if there are a limited number of studies observing the interventions and intervention components.The application of methods to the motivating example demonstrated a key benefit of using statistical methods in a public health context compared to only presenting a narrative review – the methods provide a quantitative estimate of the effectiveness of the interventions. The uncertainty from the credible intervals can be used to demonstrate the lack of available evidence. In the context of decision making, having pooled estimates makes it much easier for decision makers to assess the effectiveness of the interventions or identify when more research is required. The posterior distribution of the pooled results from the evidence synthesis can also be incorporated into a comprehensive decision analytic model to determine cost-effectiveness . AlthougAlthough heterogeneity seems to be inevitable within public health interventions due to their complex nature, this review has shown that it is still the main reported reason for not using statistical methods in evidence synthesis. This may be due to guidelines that were originally developed for clinical treatments that are tested in randomised conditions still being applied in public health settings. Guidelines for the choice of methods used in public health intervention appraisals could be updated to take into account the complexities and wide ranging areas in public health. Sophisticated methods may be more appropriate in some cases than simpler models for modelling multiple, complex interventions and their uncertainty, given the limitations are also fully reported . SynthesThere is an evident need to facilitate the translation of the synthesis methods into a public health context and encourage the use of methods to improve decision making. This review has shown that the uptake of statistical methods for evaluating the effectiveness of public health interventions is slow, despite advances in methods that address specific issues in public health intervention appraisal and the publication of guidance documents to complement their application.Additional file 1 Key for the Nice public health guideline codes. Available in NICEGuidelinesKey.xlsx.Additional file 2 NICE public health intervention guideline review flowchart for the inclusion and exclusion of documents. Available in Flowchart.JPG.


Evidence-based decision-making is crucial to leadership in the health sector to identify country-level priorities and generate solutions supported by rigorous research. Barriers and enablers have been explored, but limited evidence about what works to strengthening capacity at individual and institutional levels within countries has been reported, and inconsistent use of evidence to inform policy-making is a persistent challenge and concern.We conducted a framework analysis comparing experiences of nine purposively selected countries . We utilised qualitative case studies developed by in-country teams to explore enablers and barriers described across components of a predefined theory of change and then identified six cross-cutting themes and recommendations for relevant stakeholders associated with each theme.The cross-cutting themes included (1) leadership and political will, (2) incentives and resources, (3) infrastructure and access to health data, (4) designated structures and processes, (5) interaction and relationships, and (6) capacity strengthening and engagement. While each case country’s context and experience was different, common enablers and barriers surfaced across each of these themes, with Ministries of Health and other government agencies having strong roles to play, but also recognising the need for other stakeholders, including researchers, donors and civil society, to serve as essential collaborators in order to strengthen evidence uptake. Substantial and sustained investment in research capacities, able leaders and stronger engagement of civil servants are needed to further this progress and strengthen processes of health decision-making.All countries represented in this study have made commendable progress in increasing evidence uptake and strengthening supportive systems. Establishing and strengthening necessary structures and the relationships that underpin them takes time as well as resources. Going forward, the findings from this study can help guide and support advocacy to increase domestic funding for health research, especially health policy and systems research, and ensure that civil servants as well as researchers have the capacity and support to collaborate and continue to bolster evidence uptake. The need for research uptake in decision-making processes of Ministries of Health (MoHs) and other health authorities as well as among civil servants (CSs) has been acknowledged for some time. Several reviews and a number of case examples have pointed towards the facilitators and barriers for effective evidence-based decision-making, especially within a particular organisation or project. Two systematic reviews have also looked at aspects of use of evidence by policy-makers, including barriers such as mistrust between researchers and policy-makers, power and budget struggles, poor access to research, and a lack of timeliness and synthesis of evidence, and enablers such as personal contact, building relationships, timeliness of evidence, and summaries of evidence with policy recommendations , 2. Howethe ability of individuals, institutions and societies to perform functions, solve problems, and set and achieve objectives in a sustainable manner” , 1817, 1In preparation for a cross-case analysis, we read all the cases in their entirety, including the methods sections and a review of bibliographic information to assess the quality and reliability of each case study’s content. Then, we removed all figures, tables, boxes and other formatting from the cases in order to conduct coding of the main textual content in Dedoose version 8.2.14. We retained copies of all content that was removed and noted in the text where it was referenced in order to ensure that we did not miss any relevant content related to each case study. We developed a codebook based on the ToC with the addition of the PESTLE  analysis  dimensioThrough a series of summarisation and synthesis steps, we reviewed data for each code across country cases and developed matrices with summaries of each country’s data as well as overall summary notes for each code. During the first iterative step, we teased out enablers and barriers within each code for each case study, and during later iterations we explored linkages between codes and developing key messages and recommendations, supported by illustrative examples from the data. A set of six cross-cutting themes with corresponding recommendations were developed for presentation in this paper.This section provides a brief comparison of key contextual features of all of the case countries. We then present our findings related to capacity strengthening for evidence use through the lens of a set of cross-cutting themes and associated recommendations that emerged from the data.Country cases analysed for this study differ across political, economic and social contexts. These contextual factors have direct effects on the trajectories of each country towards strengthening the individual and institutional capacity for the use of evidence in policy- and decision-making. The Additional file material associated with this article provides further details about the context of all nine of the case study countries. In terms of political factors, some countries had worked to decentralise their health system in recent years, while others have maintained more centralised systems and focused rather on strengthening the public sector services and building collaborations with other partners and the private sector, and/or invested in improving primary healthcare services using community health workers and new financing models. All countries faced political challenges — whether related to conflict, corruption or others — but had also been successful in maintaining some political will and focus on improving the health system and using evidence to support this goal.Economic factors included heavy dependence on foreign aid for several countries as well as challenges in mobilising and sustaining domestic resources for health, both of which could substantially affect activities within countries’ MoHs. Countries also had diverse perspectives on payment for public sector services  and what kinds of care received the most budget and focus within the health sector. Relatedly, social factors that were captured in the study included a range of population sizes across countries  and variance in the numbers of different ethnic groups within a country. Many countries reported large proportions of their populations still living in rural areas, though urbanisation is a simultaneous trend. Inequities in access to care as well as outcomes across ethnic and rural/urban populations were a concern in several countries.Finally, each country had a unique trajectory in terms of how the use of evidence became a priority and how that use has evolved and strengthened over time. Post-war, or post-social reformation or revolution, several countries introduced evidence as part of a re-invention of their societies. Other countries have been incentivised and driven towards use of evidence due to donor expectations. Still others have recognised fundamental changes in population demographics and/or pressure from citizens to better understand and respond to emerging challenges. Though original motivators as well as specific approaches have varied across countries, all countries have made exceptional gains in evidence uptake over recent decades and recognise additional opportunities to continue to strengthen and expand the capacity that has been created so far.The six cross-cutting themes presented in Table The first theme on leadership and political will came out clearly across all country case studies. The need for the governmental system as a whole to recognise the value of evidence in policy- and decision-making as well as the need for individual leaders to prioritise evidence generation and use and ensure that adequate motivation, funding and institutionalisation of processes is essential. Without strong skills in agenda-setting and management, when the health system focus is predominantly on service delivery, or when other political challenges arise, leaders can also contribute challenges and constraints to evidence uptake. A relevant example of this theme is the launch in Ethiopia’s Health Sector Development Program (HSDP) III in 2005. Within this programme, Ethiopia instituted the principle of ‘One Plan, One Budget, and One Report’ in order to harness the resources, expertise and evidence across multiple partners. The government of Ethiopia, at national and sub-national levels, was a key player in utilising evidence and leadership to instigate and carry out this national initiative, despite initial donor skepticism and push back.Appropriate incentives and resources are also key to evidence uptake. These include both financial and non-financial incentives and resources, which can be developed and sustained through providing staff with opportunities such as networking with higher-level staff in their MoHs, travel to other institutions or countries to apply skills or observe other systems first-hand, salary top-ups related to research activities, and aligning government policies with international declarations and goals related to health spending and health-related research. Without resources or other incentives, such as career pathways, or when incentives unintentionally de-motivate staff that truly engage in problem-solving and evidence use, evidence uptake can be severely hindered. The history of Rwanda exemplifies this theme in a positive light. In the last 15 years, growth in financial resources paired with recognition of the need for an increase in capacity for evidence generation and use. The national response to these identified needs included a ‘district challenge fund’, a collaboration between the MoH and School of Public Health aiming to train district-level staff to analyse their own data and use research findings to make decisions.Regarding infrastructure and access to health data, being able to access real-time data as well as share data across different agencies and partners are major enablers. Digitisation of datasets, which is underway across case study countries, can facilitate this as well. Lack of training on database use or dysfunctionality of these databases causes frustration and delays or failure to make use of existing data. In addition, language barriers or lack of communication processes within and between the MoH and other institutions, including universities and separate data or statistics units, and practices can limit access to or the usability of data. For example, the Chilean MoH identified the need for an integrated Health Technology Assessment process — one that centralises data collection, storage and access procedures to assist with research appraisal by multiple stakeholders. A central repository and established appraisal process via HTA would facilitate higher-quality evaluations of existing research before being used to inform policy decisions.Designated structures and processes for evidence uptake were also identified across many of the cases. While ensuring that the system as a whole recognises the value of evidence uptake and engages in ensuring that it is happening, not every government agency or unit within a MoH is well-placed to nor needs to be fully responsible for handling the entire process on its own. Many countries found that establishing a designated unit within a MoH or across multiple sectors of government, including health, allowed for more focused and in-depth capacity to be built, provided a more coordinated platform for different partners to contribute, and enabled more coherent national research strategies and availability and use of related tools and resources. Without a designated structure, it was easy for governments to lose focus on evidence generation and use. In addition, when these structures did exist but lacked adequate funding or skills staff, these were also barriers. Well-designed structures for evidence use are exemplified in the Knowledge to Policy Center, a WHO collaborating center in Lebanon that is a key structure established to support the Ministry of Public Health as a capacity strengthening mechanism. Through its collaboration with the Knowledge to Policy Center, the Ministry of Public Health has co-facilitated a number of national policy dialogues that include key policy-makers, researchers and other stakeholders around high-priority issues in order to achieve policy impact.In terms of interaction and relationships, having ongoing, structured and accountable contact between CSs, researchers and a range of other stakeholders was described as key to effectively utilising available capacity and different agencies’ strengths to support evidence uptake. In addition, international organisations as well as standards and best practices for collaboration and engagement can help overcome stalemates or barriers from ideological positions taken by any stakeholder. However, building such relationships and collaborations takes time and without facilitation or structured processes, communication channels and supportive leaders, these interactions can struggle or cease to function. In Kyrgyzstan, for example, in response to rising burden of cardiovascular disease, a process for developing an effective solution was not only informed by extensive research, but also by well-rounded interaction and inclusion of several stakeholders outside of the MoH. With the help of the involved agencies, intensive research served to create the National Strategy on Cardiovascular Disease Control, which in turn outlined policies improving the organisation of care and treatment as well as service delivery throughout the health system.Finally, capacity strengthening and engagement of CSs was identified as another essential component for evidence uptake. This includes regular opportunities for early career and ongoing training and skill-building as well as initiatives and support for the establishment of mentorship models. Without leadership support and funding, such programmes are not possible. In addition, training programmes that do occur may not cover relevant skills, may not help learners link the concept learned in a training to actual policy-making processes, or turnover of CSs may result in skills gained not being able to be applied. The MoH of Sri Lanka has a strong example of this in its Education, Training and Research Unit, which functions as a hub for health personpower training as well as undertaking health research to inform the public sector health service delivery. The MoH has established training requirements in order to strengthen the capacity of CSs on topics related to evidence generation and uptake. Through the Education, Training and Research Unit’s research directorate, research allowances are available to academicians and senior level CS officers in order to motivate CSs to engage in research.We identified a set of key recommendations aligned with the six cross-cutting themes as well as specific actions that different stakeholders could undertake to further these recommendations. These recommendations are presented in Table The recommendations we identified also build on successful actions and progress that have been seen across country contexts, and therefore it is an opportunity to further build on success, and certainly not an indication that efforts are starting from nothing. This is exemplified in Themes 4 and 5, where recommendations focus on MoHs maintaining a national agenda and sustaining a focused entity to support the implementation of that agenda, including the engagement of CSs throughout the process. The roles of academic institutions and development partners include engaging with these MoH-led structures and supporting the national agenda by collaborating across the continuum from problem identification to use of evidence to inform policy and practice. By taking a longer-term view of what capacity is desired and needed by a particular country, many stakeholders can be part of strengthening and facilitating greater evidence uptake by feeding into all of these recommendations.By undertaking this framework analysis, we aimed to synthesise transferable and actionable lessons from across diverse contexts described in the nine country case studies. Each case represents a complex, constantly evolving process that we analysed looking at progress over time as well as areas for ongoing learning and further action. In addition, we recognise that case contexts continue to evolve and may have changed since the data presented in the case studies, or since the case studies were completed. We utilised country case examples as opportunities to learn from remarkable progress and also identify areas for continued work, and not to shed any country in a negative and punitive light.We identified several limitations while undertaking the analysis for this paper. First, we were working with a set of case studies prepared by academic leaders in collaboration with highly experienced colleagues across all countries, which despite the use of a common ToC, did not have an equivalent level of detail or organisation of data on every aspect of the conceptual framework. We utilised rich descriptions from across countries and compared experiences across context when possible. We were not able to collect additional data in order to fill in identified gaps, as the case study development had already been completed and the approach utilised in developing and presenting the findings could not be completely standardised or quality controlled. In some instances, authors of the original cases also noted limitations in terms of not being able to access evidence across some aspects of the conceptual frameworks. In addition, the focus of this study as well as the original charge given to the nine country case study teams, was centered on institutional capacity for evidence uptake within government institutions. Perhaps partly due to this explicit focus from the outset, we did not find a lot of examples or descriptions of initiatives undertaken by or recommendations geared for other stakeholders such as researchers, donors and civil society. However, we know that, particularly within fields like HPSR, researchers have a strong responsibility and large potential for support and impact if they do engage proactively and collaborate with all stakeholders in ensuring evidence uptake , 23–25.Finally, the recommendations gleaned from the case study analysis are general themes, which may not be relevant or timely in every context. Specific recommendations will likely require adaption and careful planning as well as strengthening of political alliance, establishing relationships and considering other contextual factors before implementation in a new setting.Evidence uptake has been an active pursuit for many countries, including the nine case countries analysed in this study. While each country has undertaken the process in different ways, common themes related to leadership, incentives and resources, infrastructure, designated structures, interaction, and capacity strengthening have been identified by comparing experiences from these diverse contexts. Continuing to strengthen capacity for evidence uptake in any country requires time, sustained investment on multiple levels, and active engagement of multiple stakeholders, including civil servants, researchers, development partners and civil society. Going forward, ongoing effort to strengthen both the supply and demand side of evidence uptake is needed across many countries; continuing to move this agenda forward will require the allocation of additional resources — particularly domestic resources — as well as further engagement of researchers to work alongside government leaders and civil servants throughout the evidence generation and uptake process.Additional file 1.


This paper documents an exercise to synthesize and assess the best available scientific knowledge on the effectiveness of different farm practices at enhancing natural pest regulation in agriculture. It demonstrates a novel combination of three approaches to evidence synthesis—systematic literature search, collated synopsis and evidence assessment using an expert panel. These approaches follow a logical sequence moving from a large volume of disparate evidence to a simple, easily understandable answer for use in policy or practice. The example of natural pest regulation in agriculture was selected as a case study within two independent science-policy interface projects, one European and one British. A third funder, a private business, supported the final stage to translate the synthesized findings into a useful, simplified output for agronomists. As a whole, the case study showcases how a network of scientific knowledge holders and knowledge users can work together to improve the use of science in policy and practice. The process identified five practices with good evidence of a benefit to natural pest regulation, with the most beneficial being ‘Combine trap and repellent crops in a push–pull system’. It highlights knowledge gaps, or potential research priorities, by showing practices considered important by stakeholders for which there is not enough evidence to make an assessment of effects on natural pest regulation, including ‘Alter the timing of pesticide application.’ Finally, the process identifies several important practices where the volume of evidence of effects on natural pest regulation was too large  to be summarised with the resources available, and for which focused systematic reviews may be the best approach. These very well studied practices include ‘Reduce tillage’ and ‘Plant more than one crop per field’.The online version of this article (doi:10.1007/s10531-016-1133-7) contains supplementary material, which is available to authorized users. This paper describes an exercise to synthesize and assess the best available scientific knowledge on the effectiveness of different farm practices at enhancing natural pest regulation in agriculture. It demonstrates a novel combination of three different approaches to evidence synthesis—systematic literature search  damage to wheat, and enhance wheat yield by 10 %, in fields not treated with insecticide. To our knowledge, the full breadth of this evidence has not previously been brought together in a format readily accessible to policymakers and agronomists, or analysed in the context of ecosystem service delivery. Reviews and meta-analyses that have been published usually focus on one specific aspect, such as the influence of habitat management  were eager to get a list of possible practices in natural pest control, as well as a synthesis about evidence of their effectiveness. This led the authors to opt for a systematic map or review approach , but must be clearly explained and transparent to maintain rigour and replicability.The collated synopsis approach corresponds to the summary level in the 4S hierarchy of organising evidence Fig. , becauseAs the two projects selected methodological approaches from adjacent levels of the 4S hierarchy Fig. , they arhttp://www7.inra.fr/hyppz/), broad categories of natural enemies, and types of practice and their outcomes . The search terms were chosen by an iterative process of searching and refining. The final search equation was tested for effectiveness against a benchmark list of 83 papers identified as relevant by a scoping exercise run on Scopus (PZ), Biosis (PZ) and Web of Science (BL), and a preliminary list sent by University of Cambridge . The complete list of search terms is provided in Supplementary Information, Part 1. Results of the searches in CAB Abstract and Web of Science were combined and duplicates removed. Titles not mentioning agriculture or ecological topics (e.g. medicine) were discarded  habitat, the focus chosen by the BiodiversityKnowledge project. Seventeen categories of practice (e.g. grazing) or habitat features (e.g. shelterbelts) were identified, as indicated by the ‘action’ search terms (Supplementary Information Part 1). This provided literature on a subset of the full practice list generated by the NERC Knowledge Exchange Programme, as practices unrelated to habitat management (such as ‘reduce pesticide use’ or ‘use crop rotation’) were not explicitly considered in the search equations. These actions were captured by the journal trawl approach described below, but received less search effort.http://www.conservationevidence.com). For published examples and more details of the method, see Dicks et al.  they tested, and compiled into a Microsoft Access database. This final database could be considered an unpublished systematic map  and 100 (high) for:Details of the scoring, and guidance given to the assessors, are provided in Supplementary Information, Appendix 3.During the workshop, participants were presented with the range and median scores of the group. Each practice was discussed in detail and the group members each scored again, anonymously and independently. Important discussion points were recorded and are included in a final guidance document . Medians of the second round scores were used to place the practices into categories of effectiveness, using thresholds shown in Table Figure The systematic literature search returned 33,852 studies  once duplicates were removed. We estimated that these searches obtained approximately 56 % of the relevant literature, based on the percentage of references from the benchmark list that were returned by the searches. This is a relatively low capture percentage, reflecting the difficulty of designing systematic search terms for such a broad question. With more time and resources, it may have triggered further refinement of the search terms, but this was not possible with the resources available. After title screening, a set of 4202 papers where retained to be screened at abstract stage.A set of 92 practices to enhance natural pest regulation in agriculture were identified and 3947 individual studies that tested them were retained. These practices are listed in the Supplementary Information, Appendix 2, which also shows the number of studies testing the effectiveness of each. The number of relevant studies found per practice ranged from 0 to 570, with a strongly positive skewed distribution , ‘Alter timing of sowing or harvesting’ (445), ‘Reduce pesticide use’ (404), ‘Use crop varieties that resist or suppress pests, diseases or weeds’ (383), and ‘Reduce tillage’ (375). Four of these were selected as priorities during stakeholder interaction 4 Fig. . The exchttp://www.conservationevidence.com). These practices have between 1 and 19 relevant studies . In both cases, the decision was well justified. Sets of evidence relevant to practice were obviously missing, due to the constraints and scope of the prior stages of evidence synthesis. Rather than being a limitation of the expert assessment method, stakeholder influence is important, because it generates buy-in to the outcomes from stakeholder groups, and helps to ensure that the outcomes are relevant and understandable. Different stakeholder sets were involved at each of the different stages, creating a broad body of consultees overall and raising awareness of the process among the stakeholder community.The stakeholders involved at the interaction points described in Fig. It is important to guard against introduction of bias through stakeholder influence. The processes described do this by using rigorous formal consensus methods such as the modified Delphi technique where possible to avoid undue influence by specific individuals. Changes such as the removal of practices from the assessment had to be clearly and transparently justified, with agreement from all those involved at that stage. For future iterations of a process like this, the set of stakeholders could be more sharply defined, or remain relatively opportunistic to broaden the consultation as widely as possible, depending on the requirements of a particular issue.This natural pest regulation case study illustrates the value of the Network of Knowledge approach devised by the BiodiversityKnowledge project at linking people and projects together. The pilot synopsis and assessment of evidence were completed using the combined resources of two projects that were initially operating independently, and additional resources from a follow-on project conducted at Cambridge. These efforts were brought together as a result of networking activity.It is interesting to consider how the stakeholders involved in the final stage of the process—the expert assessment panel—felt about the validity of the exercise and the method of assessment. This group were given an opportunity to provide feedback. Two members of the panel felt the process itself needed more explanation in the output (Supplementary Information), if it is to be used by agronomists. In particular, where the evidence was weak on enhancing natural pest regulation, this was often because the pest regulation service, or impacts of the practices on pest damage or yields had not been measured explicitly. Documented increases in numbers of natural enemies in the wider environment, but not active in crop fields, did not provide a very high level of certainty in this assessment, a subtlety that may not be apparent when using the output.As an example of this, two members of the panel were uneasy about the categorisation of ‘Beetle banks’ as ‘Unlikely to be beneficial’, arguing that there is still considerable uncertainty about the effect of beetle banks on pest regulation, despite the practice going through the full three rounds of scoring. As explained in the Agronomist’s Guide , the evidence shows that beetle banks can lead to an increase in natural enemies and a reduction in pests in, or close to, the banks. The six studies that measured natural enemies in the crop found they only penetrated a short distance into the field and only for a limited period of time Thomas . Enhance“Most end users would want to know ‘Does this method mean I get fewer pest problems?’ Or ‘could this method be helpful for reducing my reliance on pesticides or reducing my pest control costs?’ Rather than ‘does this method enhance natural pest regulation?’” Feedback from Expert Panel Member (NGO representative).Two panel members felt the information would be more useful for the farming and agronomy community if it included details of the specific practices, and used more familiar language, as demonstrated in the following quote from a feedback email:There is clearly substantial scope for further work. In the early part of developing the collated synopsis (Step 2), global evidence was mapped for 70 additional practices for which evidence was neither summarised nor assessed Fig. . This inEvidence assessment for selected practices). In cases where effectiveness score was low but certainty quite high, as in the case of beetle banks discussed above , new evidence could shift the category into the beneficial categories, but it would have to carry substantial weight in the judgement of the expert panel members to outweigh the existing body of evidence that led to low effectiveness scores. At least one paper has been published on beetle banks since 2012 Below is the link to the electronic supplementary material.


Use of good quality, evidence-informed and up-to-date clinical practice guidelines (CPGs) has the potential to impact health outcomes. This paper describes the development, implementation and evaluation of a dedicated CPG training course to address the training needs of CPG stakeholders in South Africa.We first reviewed the content and teaching strategies of existing CPG courses. This review consisted of a systematic review of teaching and learning strategies for guideline teams and a document review of existing courses offered by international guideline groups, universities and professional groups. We then strengthened an existing CPG course and evaluated it.We found no studies on teaching and learning strategies for guideline teams. We identified six CPG courses being offered as full courses (part of a postgraduate degree program) by universities or as independent training for continuing professional education by professional groups. Contents focused on new guideline development. One course included alternative methods of guideline approaches such as contextualization and adaptation. The format varied from face-to-face sessions, to online sessions, group exercises and discussions, seminar format and project based activities. The revised CPG four-month long course that we implemented was designed to be pragmatic, reflective and contextually relevant. It used local guideline examples, authentic tasks, and an online forum for discussions and resources. It covered de novo CPG development, alternative methods of development , and implementing CPGs. Course evaluation identified strengths and areas for improvement.Dedicated capacity development has potential to positively influence CPG development and implementation. High quality, evidence-informed clinical practice guidelines (CPGs) offer a way of bridging the gap between policy, best-practice, local contexts and patient choice . Good quSouth Africa has a long history of developing various CPGs. Key role players include the National Department of Health, professional societies, private sector and non-governmental organisations producing guidance for their respective constituents. Our team came together in 2014 to conduct the South African Guidelines Excellence (SAGE) Project . We undeWe first reviewed the content and teaching strategies of existing CPG courses using a systematic review of teaching and learning strategies for guideline teams and a document review of existing courses offered by international guideline groups, universities and professional groups. We then developed, implemented and evaluated a CPG course targeted to the needs of end-users in resource constrained environments.Systematic review: to evaluate the effectiveness of teaching or learning strategies for CPGs, we searched for randomised and non-randomised trials, pre-post experimental studies or primary observational studies. We excluded studies that focussed only on methods for teaching evidence-based practice, and did not include CPGs. Our target participants included CPG writers, methodologists, implementers and evaluators. We developed a comprehensive strategy (Supplementary file 1) to search for all eligible studies available up to 29th November 2017, regardless of language or publication status. For published literature we searched Medline, Cochrane Database of Systematic Reviews (Effective Practice and Organization Group), Database of Abstracts of Reviews of Effects (DARE), Cumulative Index to Nursing and Allied Health Literature (CINAHL), current controlled trials, Educational Resources Information Center (ERIC) and Turning Research into Practice (Trip) database. We used the following search terms and tailored them appropriately to the different databases: ‘guide*’, ‘clinical practice guideline’, ‘teach*’, ‘learn*’. Three reviewers  independently screened titles, abstracts and full texts of potentially eligible articles. To access unpublished literature, we searched websites of 17 guideline groups using a Google scoping search. We also contacted relevant experts in the field of CPGs. We aimed to conduct independent data extraction and risk of bias assessments and then to look at within group differences focusing on outcomes measured between baseline and post training. These outcomes could include short-term measures on teaching methods or programs  and longer-term measures on the quality of the guideline process from development to evaluation.Document review of existing CPG courses: we searched the internet for existing CPG courses using search term “clinical practice guidelines courses” OR “guideline courses”. We searched guideline sites and professional associations/organizations. We contacted 21 content experts via email, professional groups and the networks of Cochrane South Africa, Centre for Evidence Based Health Care and International Centre for Allied Health Evidence (iCAHE) to seek information regarding any CPG course training being offered, and for any evaluation of CPG courses delivered. Eighteen groups responded and we obtained the information from the websites of three groups who did not respond. We collected all possible information about CPG courses and extracted relevant information: course description, objectives, content, teaching and learning strategies, mode of delivery, format, duration, outcomes and evaluation/assessments. We used a descriptive qualitative framework to guide our data collection. Qualitative descriptive approaches are particularly useful in program assessment and development. We used content analysis to synthesize our data  had unique aspects that is going to help me in my job. We had modules on phrasing questions we had modules on searching and finding guidelines which is what I do every day. We had modules on evaluating guidelines, we had modules on implementation the implementation part, maybe not so much, but it is something I am actually getting into it. They were all very helpful, and I'm going to be using the knowledge I have learnt in my everyday work actually.’‘Students appreciated the various forms of engagement, level of expertise and preparation by facilitators. They liked the use of practical exercises alongside didactic teaching, and rated the course as providing high value to their work. Students found the assessment process relevant, fair and appropriate. Some of the students gave recommendations for areas of improvement for future contact sessions-availability of materials before the block contact session, allotting more time for some sessions, early briefing on the assessment structure as well as provision of in-class notes. They also indicated some concern about having too much information covered during the 5-day teaching block.I wish they could have structured it in a way that you speak for an hour, somebody else comes for an hour, and you come back so that you have different people speaking. And they have more interactive slides and you ask a few questions, somebody answers. Now some of the people all they do is sit there in front of us and go through it so fast and you are more confused by the end of the hour than you were before.’‘at the heart of this process is a very mucky set of human interactions which can be managed well and can be managed badly, and can make a huge difference to the outcome.’The CPG content expert reviewer emphasized the importance of conveying concepts simply and ensuring adequate time for practical examples and reflection. He suggested additional relevant content such as the project management and chairing skills for CPG development as he described that ‘With the growth in evidence-informed practices in the African region ,25 and tThe course fills a specific gap for decision-makers from resource constrained settings as we share best practice knowledge for alternative development of CPGs. De novo CPG development is time and resource intensive requiring technical skills for conducting systematic reviews, funding to host meetings and ensure participation and consultation as broadly as possible. For this reason, alternative methods have been developed and our course is unique in generating awareness and discussion of these approaches , 31. We We used an evidence-informed multifaceted approach to enhance a master's level CPG course for the needs of CPG developers and implementers in resource constrained settings. Dedicated, evidence-informed capacity development for CPG activities may empower current and future CPG developers and end-users and has the potential to positively influence CPG development and implementation.High quality, evidence-informed CPGs offer a way of bridging the gap between policy, best-practice, local contexts and patient choice;There is an increasing need, and demand for, building capacity for evidence-informed CPG development and implementation.There are limited robust evaluations of CPG capacity-development initiatives;This paper described the development, implementation and evaluation of a dedicated CPG training course designed to meet specific development needs in South Africa;The course fills a specific gap for decision-makers from resource constrained settings.The authors declare no competing interests.