Introduction: Evidence syntheses, often in the form of systematic reviews, are essential for clinical guideline development and informing changes to health policies. However, clinical guideline development groups (CGDG) are multidisciplinary, and participants such as policymakers, healthcare professionals and patient representatives can face obstacles when trying to understand and use evidence synthesis findings. Summary formats to communicate the results of evidence syntheses have become increasingly common, but it is currently unclear which format is most effective for different stakeholders. This mixed-methods systematic review (MMSR) evaluates the effectiveness and acceptability of different evidence synthesis summary formats for CGDG members.Methods: This protocol follows guidance from the Joanna Briggs Institute on MMSRs and is reported according to the Preferred Reporting Items for Systematic Reviews (PRISMA)-P guideline. A comprehensive search of six databases will be performed with no language restrictions. Primary outcomes are those relating to the effectiveness and preferences for and attitudes towards the different summary formats. We will include qualitative research and randomised controlled trials. Two reviewers will perform title, abstract, and full-text screening. Independent double-extraction of study characteristics and critical appraisal items will be undertaken using a standardised form. We will use a convergent segregated approach to analyse quantitative and qualitative data separately; results will then be integrated.Discussion: The results of this systematic review will provide an overview of the effectiveness and acceptability of different summary formats for evidence synthesis findings. These findings can be helpful for those in or communicating to guideline development groups. The results can also inform the development and pilot-testing of summary formats for evidence summaries. The development of a clinical guideline involves a rigorous synthesis of the best available evidence on a specific clinical topic. It may involve formal consensus methods with a range of multidisciplinary stakeholders. Guideline development groups comprise a range of decision makers, often including healthcare professionals, methodologists, health policymakers, clinicians, and patient representatives – all of whom have varying levels of expertise in evidence synthesis methods. This complicates the consensus process as stakeholders may prioritise and understand the findings of evidence syntheses, such as systematic reviews, differently.Clinical guidelines support decision making to improve patient outcomes and quality of care in a cost-effective manner, there are still barriers to their creation and use. A meta-analysis of nearly 200 systematic reviews registered on the international Prospective Register of Systematic Reviews (PROSPERO) registry found that the average systematic review, from registration to publication date, takes 67.3 weeks, involves an average of five authors, and requires the full-text screening of 63 papers (range: 0–4385). The number of academic papers and systematic reviews being published in recent decades has rapidly increased, further accelerating during the recent COVID-19 (coronavirus disease) pandemic. The expanding evidence base and acceptance of trade-offs in validity in time-sensitive matters, has resulted in the growing popularity of other evidence synthesis methods, such as rapid reviews. This increase in different types of evidence synthesis methods further complicates matters for guideline development groups, who may interpret different types of systematic reviews in different ways based on how familiar they might be with particular approaches.While the methods and recognition of the importance of systematic reviews have advanced in recent decades. Previous work has highlighted that decision makers more easily understand evidence summaries than complete systematic reviews. These summaries can come in a variety of different formats such as policy briefs, one-page reports, abstracts, summary of findings tables, plain language summaries, visual abstracts or infographics, podcasts, and more. While formatting may vary, decision-makers have expressed several key preferences, such as succinct summaries highlighting contextual factors like local applicability and costs.For those using different types of evidence synthesis to inform clinical guideline development and health policy, the amount of included studies, length, and technical nature of evidence syntheses can make it difficult to find answers about the effectiveness of healthcare interventions. It is currently unclear which evidence summary format is most helpful for decision making for different guideline development group stakeholders. For example, Cochrane recommends a ‘summary of findings’ table but testing with users familiar with the Cochrane library and evidence-based practices raised concerns around comprehension and presentation of results and the balance between precision and simplicity. Others have tested the presentation of information using different formats such as an abstract, plain-language summary, podcast or podcast transcription with no clear answer regarding which format was most suited to which stakeholder and resulted in the best understanding. Similarly, infographics, plain-language summaries, and traditional abstracts were found to be equally effective in transmitting knowledge to healthcare providers; however, there were differences in measures of acceptability .Succinctness should be inherent in an evidence summary, but how this distilled information is formatted and presented affects the interpretation and use of systematic reviews. As impacts on decision-making and preferences for formats may be evaluated through different study designs, a comprehensive synthesis of the evidence is needed beyond a typical single method systematic review. Mixed methods systematic reviews (MMSR) can more easily identify discrepancies within available evidence, pinpoint how quantitative or qualitative research has focused on particular interest areas, and offer a deeper understanding of findings. A MMSR is especially useful for this project as it brings together findings of effectiveness and experience so findings are more useful for decision makers. Guideline developers need to consider diverse considerations in their work such as feasibility, priority, cost effectiveness, equity, acceptability, and patient values and preferences. Similarly, a MMSR allows us to consider and integrate data from a variety of different questions and synthesize information in a single project.To better support clinical guideline development groups and decision-makers, it is important to identify which format works best for which stakeholder. Previous reviews have focused on identifying barriers and facilitators to use, or have been solely based on summary of findings tablesThe aim of this mixed methods systematic review is to evaluate the effectiveness of, preferences for, and attitudes towards, different communication formats of evidence summary findings amongst guideline development group members, including healthcare providers, policy makers and patient representatives. To achieve this, the proposed MMSR will answer the following questions:1. How and to what degree do different summary formats  of presenting evidence synthesis findings impact the end user’s understanding of the review findings?2. What are the end users’ preferences for and attitudes towards these formats?.The proposed systematic review will be conducted in accordance with the Joanna Briggs Institute (JBI) Manual for Evidence Synthesis which details the methodology for mixed methods systematic reviews (MMSR) As this is a MMSR, we will include quantitative , qualitative, and mixed methods studies evaluating the effectiveness and/or preferences for and attitudes towards evidence summary formats. We will exclude conference abstracts, case reports, case series, editorials, and letters. Further details regarding eligibility criteria are given within the review-relevant sections below. We are interested in studies involving stakeholders such as policy makers, healthcare providers, and health systems managers, as well as other GDG members such as clinicians, patient representatives, and methodologists such as systematic review authors. We will exclude studies where the sole participants are students, the general population , and journalists as communication to these populations is more complex given a wide variety of confounding factors. We will also exclude studies related to clinical decision-making for individual patients.We have followed the Population, Intervention, Comparison, Outcome (PICO) format for the quantitative review  and the Quantitative systematic review. Due to the complexity of stakeholders, evidence synthesis types, and summary formats, there is a high potential that confounding factors will be extensive. Relatedly, randomised controlled trials (RCTs) are the most appropriate design to evaluate the effectiveness of the interventions in question. Thus, we chose to restrict to RCTs  only in order to focus on the performance and impact of summary formats in optimal settings. We will include studies where the intervention is any summary mode  which communicates the findings from an evidence synthesis study  to policy-makers and decision makers, including guideline development groups (GDGs). We anticipate that included summary formats may encompass visual abstracts, Summary of Findings tables, one-page summaries, podcasts, Graphical Overview of Evidence Reviews (GofER) diagrams, and others. We will not exclude a summary format if it is one that we did not explicitly list in our search strategy  will be excluded. We do not anticipate finding evidence syntheses with no form of summary or abstract as international organisations, journals, and reporting guidelines would consider a summary to be a mandatory component of any report or peer review manuscript produced.Our primary outcomes of interest are:1. Effectivenessa. User understanding and knowledge, and/or beliefs in key findings of evidence synthesis  b. Self-reported impact on decision‐makingc. Intervention metrics 2. Acceptabilitya. Preferences and attitudes . .We will not be including outcomes related to health literacy, numeracy, nor risk communication in patient-centred care. We are aligning our definition of ‘health literacy’ with a recent systematic review on its meaning, which is complex in nature and composed of ‘(1) knowledge of health, healthcare and health systems; (2) processing and using information in various formats in relation to health and healthcare; and (3) ability to maintain health through self-management and working in partnerships with health providers.’ As impacts on one’s individual health or clinical care is not the main focus of this review, we are focusing only on one aspect (2) -- impacts on one’s understanding of knowledge which is constrained to a specific topic that an evidence summary is coveringQualitative evidence synthesis. Primary studies investigating the understanding and acceptability of evidence summary formats will include qualitative studies . Mixed-methods studies with primary qualitative data collection will be included if they meet the inclusion criteria for a randomised controlled trial and where it is possible to extract the findings derived from the qualitative research. We prioritized the inclusion of qualitative data from primary studies over free text from questionnaire surveys as we hypothesized primary data would be richer and thicker and thus more informative. Our primary outcomes of interest relate to participant’s views and experiences with summary formats. This includes their perceptions of the impact of summary formats on their understanding, knowledge, and decision making, and participant’s beliefs, attitudes, and feelings towards usability and readability. . Aligned with the Peer Review of Electronic Search Straggles (PRESS) Statement, we engaged a medical librarian after the MEDLINE search was drafted but before it was translated to the other databases. As we are including a range of study designs, we did not apply study design specific filters. Although we have used a PICO and SPiDER approach for the quantitative and qualitative reviews we used the PICO format to inform the search strategy as previous researchers found that the SPiDER approach for search strategies may be too restrictive and specific. Language and date restrictions will not be applied.The following databases will be searched from inception to May 2021: Ovid MEDLINE, EMBASE, APA PsycINFO, CINAHL , Web of Science, and Cochrane Library. The search strategy for Ovid MEDLINE includes a combination of keywords and medical subject headings (MeSH) terms for GDG members, evidence syntheses, and formats for the communication of findings see. As we acitationchaser Shiny application built in R version 1.4. This application performs backwards citation screening (reviewing reference lists) and internally de-duplicates results. Each step of the search is summarised for transparency and references are given as a downloadable RIS file.Backwards citation identification on all eligible studies will be performed using theZotero reference manager version 5.0. For ease, rather than using Zotero for screening, title and abstract screening will be managed usingCovidence. Two reviewers will independently screen titles and abstracts for inclusion criteria. Disagreements for inclusion will be resolved through discussions. If it is still unclear if the paper should be included, both authors will review the full version of the paper and discuss it again. If there is still disagreement, a third review author will be consulted. The screening process will be documented in the final manuscript using the Preferred Reporting Items for Systematic Reviews (PRISMA) flow diagram and a supplemental file detailing the reason for exclusion for each individual study will be made publicly available.All citations will be downloaded and stored inTwo review authors will independently extract data from each of the included studies using a standardised data-extraction form. If there are disagreements or discrepancies, the two authors will discuss and consult with a third review author if needed. Where possible, qualitative outcomes such as themes and categories will be extracted into the standardized form. In parallel, articles containing qualitative methods will be also imported in NVivo12 for line-by-line coding for information related to outcomes. This separate but parallel data extraction is important for our analytical approach of the qualitative data which is discussed in greater detail in the Qualitative Analysis section. The following information will be extracted using the pilot-tested standardized data-extraction formBibliometric data Study characteristics  to provide detailed information on the why, what, who, how, where, and when of the intervention described.Intervention characteristics will be collected following the structure of the Template for Intervention Description and Replication (TiDieR) checklistPrimary and secondary outcomes JBI critical appraisal tools for qualitative and quantitative studiesData from the domains listed within theFunding sourcesIf information is missing from the study report, we will contact authors to inquire about these gaps. We will provide narrative syntheses in lieu of imputing missing data.JBI critical appraisal checklists will be used to assess the individual randomized controlled trials and qualitative studies. Two review authors will independently complete the critical appraisal checklist for each included study. Differences will be resolved through discussion and consultation with a third review author if necessary. These checklists will provide useful contextual information about the included studies such as information about performance bias. Checklist items cover things like intervention assessors and their reflexivity which are important factors to consider as participant attitudes towards summary formats may be influenced by external factors such as who created the summary .TheAn assessment of the overall certainty of evidence using the GRADE or ConQual approach is not recommended for a JBI MMSR26. This is due to the complexities in the analysis wherein the data from separate quantitative and qualitative evidence is transformed and integrated. as it reduces the false positive rate. Egger’s test for funnel plot asymmetry will be used to investigate small study effects and publication bias.If quantitative data allows for a meta-analysis, a forest plot will be generated using R. If we find a low number of studies, large treatment effects, few events per trial, or all trials are of similar sizes, we will use the Harboard test for publication bias, an estimated I2 of 50–90% represents substantial heterogeneity. We will weigh this against an χ2 test for heterogeneity (<.10). If our results indicate 50% or greater and a low χ2 statistic, this indicates that the heterogeneity may not be due to chance, thus we will not pool results into a meta-analysis. If data can be pooled, effect sizes and accompanying 95% confidence intervals will be reported as either relative risks  or standardized mean differences (for continuous data). Where data is available, we will compare and contrast reported findings on preference and whether or not preference is aligned with improvement of outcomes of impact such as knowledge.A narrative synthesis will be performed, however, if appropriate, quantitative data from randomised control trials will be synthesised using meta-analysis. Heterogeneity will first be explored by assessing pertinent study characteristics that may vary across the included studies . If sufficient data is available, subgroup analyses  will be conducted. Furthermore, statistical heterogeneity will be explored according to statistical guidance on heterogeneity. This approach organises and categorises findings based on similarity in meaning and avoids re-interpretation. Therefore, it does not violate paradigms and approaches used by the original study authors. This approach also enables meaningful generalizable recommendations for practitioners and policy makers. If we are unable to pool findings together , likely due to an insufficient number of studies identified, a narrative summary will be presented.Where possible, qualitative findings will be pooled together using the meta-aggregation approach, which allows a reviewer to present findings of included studies as originally intended by the original authors. The segregated design integrates evidence through a method called configuration which essentially arranges complementary evidence into a single line of reasoning. After the separate quantitative and qualitative analyses are conducted, they will be organized into a coherent whole, as they cannot be directly combined nor can one refute the other. Converging or complementary data assumes that, while the streams of evidence may ask different research questions, they are related to different aspects or dimensions of the same phenomenon of interest. Data will be triangulated during the interpretation stage, comparing quantitative and qualitative findings side-by-side to identify areas where there is convergence, inconsistency, or contradiction in the data. We do not aim to transform the qualitative data into quantitative, nor vice versa. Following JBI guidance for MMSR, we will use a convergent segregated approach that conducts separate quantitative and qualitative syntheses separately but at the same time and then integrates the findings of each in a convergent segregated MMSR. We will use a thematic synthesis method for integration which groups together similar codes, develops descriptive categories (or themes) to create an overall summary of findings. Initial coding will be performed independently by two authors who will meet and discuss similarities and differences in coding to start grouping them into descriptive categories. A drafted summary of findings will be created by one author, reviewed by both, and discussions will be held until a final version is agreed upon. Two authors will discuss the descriptive categories and, as a group, will draft the final analytical categories with accompanying detailed descriptions.There are several methods for integrating qualitative and quantitative evidence synthesesIf we have a sufficient number of included studies for meta-analysis (minimum of three), we will report information according to participant subgroups , and outcomes . PROSPERO. However, we will preregister the study onOpen Science Framework. If an amendment to this protocol is necessary, the date of each amendment will be given alongside the rationale and description of the change(s). This information will be detailed in an appendix accompanying the final systematic review publication. Changes will not be incorporated into the protocol.As the focus of this review is not evaluating health-related interventions nor outcomes, we will not register the protocol onOSF project page.Findings will be disseminated as peer-reviewed publications. Data generated from the work proposed within this protocol will be made available on the aforementionedThis review will summarise the evidence on the effectiveness and acceptability of different evidence synthesis summary formats. By including a variety of evidence summary types and stakeholder participants, results can help tease apart the real-world complexity of guideline development groups and provide an overview of what summary formats work for which stakeholders in what circumstances. It is expected that review findings can support decision-making by policy-makers and GDGs, by establishing the best summary formats for presenting evidence synthesis findings.No data are associated with this article.https://doi.org/10.17605/OSF.IO/SK4NXOSF: PRISMA-P checklist for ‘Evidence synthesis summary formats for clinical guideline development group members: a mixed-methods systematic review protocol’.Creative Commons Attribution 4.0 International license (CC-BY 4.0).Data are available under the terms of the I am happy with the responses of the authors.Is the study design appropriate for the research question?YesIs the rationale for, and objectives of, the study clearly described?YesAre sufficient details of the methods provided to allow replication by others?NoAre the datasets clearly presented in a useable and accessible format?Not applicableReviewer Expertise:NAI confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. Thank you for the opportunity to review this interesting protocol.  Overall, the research plan is sound and described in detail. I have only a few minor suggestions to propose to make the protocol more precise. In Methods sections, under Eligibility criteria, I propose that you list all research designs that will be included in the analysis. Currently, only RCT-s are included, but it is not stated will all types of RCT-s will be included . On top of that, what about non-randomized trials and control before and after studies? Will those studies be included, too? The sentence: "We will not be including outcomes related to health literacy, numeracy, nor risk communication in patient-centred care." is a bit problematic. Based on my understanding, health literacy is the examination of knowledge and/or understanding of health information, which will be the case of most research studies you include in the review. Maybe you wanted to state that you will not include studies that assessed health literacy with specific tests of health literacy? Would you consider including Open Science Framework registry as one of the databases in which you would perform a literature search?Minor points:"If we have a sufficient number of included studies, subgroup analyses will be performed to investigate differences based on participant groups  and outcomes " What would be considered a sufficient number of studies? I hope those comments will help you in the process.Is the study design appropriate for the research question?YesIs the rationale for, and objectives of, the study clearly described?YesAre sufficient details of the methods provided to allow replication by others?PartlyAre the datasets clearly presented in a useable and accessible format?Not applicableReviewer Expertise:Public health; Health communicationI confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. Thank you for the opportunity to review this interesting protocol. Overall, the research plan is sound and described in detail. I have only a few minor suggestions to propose to make the protocol more precise.Author’s response: Thank you for the time spent reviewing our manuscript and for your valuable feedback.  In Methods sections, under Eligibility criteria, I propose that you list all research designs that will be included in the analysis. Currently, only RCT-s are included, but it is not stated will all types of RCT-s will be included . On top of that, what about non-randomized trials and control before and after studies? Will those studies be included, too?Author’s response: Thank you for this question. We have edited the text to explicitly given examples of the types of RCTs that will be included ‘’  The other reviewer had a related question regarding inclusion of different study designs. We have chosen to restrict study designs for this review to randomized controlled trials and qualitative studies as we believe these study designs are best able to either 1) control for confounding/interacting factors that could influence one’s attitude towards a format, such as who is disseminating the message (RCTs) or 2) better able to fully explore multiple factors influencing one’s perception . We chose not to include non-randomised designs or observational studies due to the complexity of factors that influence one’s attitude towards a format. These study designs also relate directly to the objectives of the review .  The sentence: "We will not be including outcomes related to health literacy, numeracy, nor risk communication in patient-centred care." is a bit problematic. Based on my understanding, health literacy is the examination of knowledge and/or understanding of health information, which will be the case of most research studies you include in the review. Maybe you wanted to state that you will not include studies that assessed health literacy with specific tests of health literacy?Author’s response: Thank you for this comment. We believe that health literacy outcomes are quite distinct from knowledge/understanding outcomes. We recognize that the definition of health literacy has evolved over time and can often mean different things to different people so we have added additional information to the end of the outcomes section to clarify the focus of our project and the outcomes we are including. We have added text to the manuscript to clarify ‘We are aligning our definition of ‘health literacy’ with a recent systematic review on its meaning, which is complex in nature and composed of ‘(1) knowledge of health, healthcare and health systems; (2) processing and using information in various formats in relation to health and healthcare; and (3) ability to maintain health through self-management and working in partnerships with health providers.’ As impacts on one’s individual health or clinical care is not the main focus of this review, we are focusing only on one aspect (2) -- impacts on one’s understanding of knowledge which is constrained to a specific topic that an evidence summary is covering. [1] Would you consider including Open Science Framework registry as one of the databases in which you would perform a literature search?Author’s response: Thank you for this suggestion.We did not search Open Science Framework as this registry is more often suited for observational studies and qualitative study protocols are still rare in the published literature. The search functionality of OSF (using Lucene) is also quite limited. For example using ‘evidence synthesis summary formats’ would result in over 23,000 registrations to screen. We did not search any registries for this study and will add this as a limitation in the discussion section of the final manuscript detailing the MMSR findings.    Minor points: "If we have a sufficient number of included studies, subgroup analyses will be performed to investigate differences based on participant groups  and outcomes " What would be considered a sufficient number of studies?Author’s response: Thank you for this question. To our knowledge there is no numeric threshold for subgroup analyses for a MMSR. We edited the text to say ‘If we have a sufficient number of included studies for meta-analysis (minimum of 3), we will report information according to participant subgroups , and outcomes .’ as the analysis will be descriptive anyway.  This is a review underpinned by clear and interesting questions, that warrant a mixed method strategy. There is room for improvement though on the level of motivating choices for concepts and finetuning methods.  The biggest issue I have is on the relation between content and form in the choice for the mediums to be compared. The authors tend to compare forms (?), but attitude towards form is not fully independent to the way messages are framed, nor by whom they are disseminated. How do the authors intend to deal with this in their comparison?  .  I was not fully sure why things like surveys measuring attitudes and opinions of people were not taken along in this review. (I am aware of the fact that guidance on the inclusion of such designs in reviews is scarce though-consider it a free offer to work towards a more comprehensive review type). Particularly, because in the RCTs authors tend to rely on self-reported measures, with LIKERT scales. Here, a comparison could sit with time, rather than comparator. Does people's attitude change when exposed to different formats over time?  Sometimes the arguments are not fully clear. In the introduction, attention is paid to problems complicating the conduct of systematic reviews. This is not the core focus of the project. The project is linked to the translation phase of turning review findings into evidence summaries. This is a matter of content, form and channels rather than the SR pre-track issues. It would therefore be more appropriate to highlight the diversity of actors involved in producing guidelines in relation to how communication and dissemination channels have changed because of that. I am a big fan of the authors discussing things like podcasts and multimedia accounts as examples. Naturally, this is linked to a society moving into multimodality and the limitations of numbers and narratives in reaching out to or working with different publics.  The PICO would perhaps work better if clinicians is changed into professionals. It would make it relevant to a broader target group.  Also, the comparator could also be no summary at all, for example, reading findings from reviews straight into the guideline procedure. Why are these explicitly excluded from the review? Do the authors only include qualitative studies where an intervention is evaluated by participants in terms of meaningfulness or applicability? Or do they also include qualitative studies where attitudes, viewpoints and options for form/channel/content are discussed from a more theoretical point of view, or in pilot cases? It is not clear where the selection sits on the qualitative side, but the data collection part suggests the first option.  Perhaps the authors may want to bring a rationale for why they want to limit the formats to textual and audio-visual productions, as there are many more forms available through which evidence is and can be communicated  Perhaps these forms haven't been studied extensively in the context of guidelines, but in a protocol phase we are not sure what to find yet :-).  I would find it more practical if the authors would discuss their position towards critical appraisal and how to structure this instead of putting it in the data collection part as an extraction category . Also, critical appraisal checklist for primary studies have a different purpose than GRADE and CONQUAL that work on a review level . I think the authors need to spell the two out in more detail, as they are used in different phases of the review process.  Would quality of a study not rather fit a sensitivity analysis rather than a subgroup analysis? What purpose would it have as a subgroup in the review? It has disappeared in the mixed method part as a subgroup analysis, so wouldn't mention it as a subgroup in other parts of the text. What do the authors mean with the phrase "If textual pooling is not available, a narrative summary will be presented." Is a narrative summary not already a textual pooling? Unclear.  While configuration might have different meanings, it is generally not commonly used as a metaphor for a separate type of synthesis. Configuration would make the different parts unrecognizable in the whole. I don't think meta-aggregation would achieve this level of integration. Perhaps integration is better than configuration here.  Also, thematic synthesis is not the same as meta-aggregation, and line by line coding is certainly not a part of meta-aggregation. I believe the authors need to make an informed choice between qualitative evidence synthesis approaches and then follow the guidance through. Is the study design appropriate for the research question?YesIs the rationale for, and objectives of, the study clearly described?YesAre sufficient details of the methods provided to allow replication by others?NoAre the datasets clearly presented in a useable and accessible format?Not applicableReviewer Expertise:MethodsI confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above. This is a review underpinned by clear and interesting questions, that warrant a mixed method strategy. There is room for improvement though on the level of motivating choices for concepts and finetuning methods.Author’s response: Thank you for your review and thoughtful suggestions for our manuscript.  The biggest issue I have is on the relation between content and form in the choice for the mediums to be compared. The authors tend to compare forms (?), but attitude towards form is not fully independent to the way messages are framed, nor by whom they are disseminated. How do the authors intend to deal with this in their comparison?  .Author’s response: Thank you for this comment. This is an interesting point, the studies we include in our review may well demonstrate a disconnect between what formats participants prefer, and what formats may actually produce improvement in outcomes such as knowledge. If included studies present this data we will be able to explore this more detail in our analysis but we suspect from the studies we have seen that this level of detail will not be available. Based on the importance of this suggestion, we will include this as a point of consideration in the discussion in the main manuscript in the protocol, we have made the following amendments to clarify our position on this:We have added additional information to the ‘bias and quality assessments’ section in the methods regarding this point: ‘These checklists will provide useful contextual information about the included studies such as information about performance bias. Checklists items cover things like intervention assessors and their reflexivity which are important factors to consider as participant attitudes towards summary formats may be influenced by external factors such as who created the summary .’ We agree that the issue is complex and there are many confounders and interacting factors that could influence one’s attitude towards a format. Relatedly, we have chosen to restrict study designs for this review to randomized controlled trials and qualitative studies as we believe these study designs are best able to control for factors such as who is disseminating the message (RCTs) and are better able to fully explore multiple factors influencing one’s perception . For example, these attitudes can be more fully explored with probes during semi-structured interviews. We have included the use of assessment tools for bias to further investigate issues such as the potential influence of the study assessors  and their recipients . We have also kept our outcomes broad so as to not take a one-dimensional view towards preference for one format over another. We can describe differences between preferences for formats and quantitative outcomes such as knowledge in the analysis where the data is available in included studies. We have clarified this in the analysis section of the methods as follows: “Where data is available, we will compare and contrast reported findings on preference and whether or not preference is aligned with improvement of outcomes of impact such as knowledge.” I was not fully sure why things like surveys measuring attitudes and opinions of people were not taken along in this review. (I am aware of the fact that guidance on the inclusion of such designs in reviews is scarce though-consider it a free offer to work towards a more comprehensive review type). Particularly, because in the RCTs authors tend to rely on self-reported measures, with LIKERT scales. Here, a comparison could sit with time, rather than comparator. Does people's attitude change when exposed to different formats over time?Author’s response: Thank you for this comment. The aim of this mixed-methods systematic review was to evaluate the effectiveness and acceptability of different evidence synthesis summary formats for CGDG members. For effectiveness we chose to restrict to RCTs as the most appropriate design to evaluate the effectiveness of the interventions in question. Therefore, observational studies like surveys were not included for the effectiveness question as they would not be appropriate to answer the question posed, as described in the methods section. We did not explicitly exclude RCT designs that had interventions with ‘repeated exposures,’ multiple timepoints for outcome assessment or longitudinal follow ups so if identified such studies will be included. However, these design seem unlikely in this context. While policy makers, clinicians, and methodologists could potentially be involved in multiple guideline development groups, this would likely be over quite an extended period of time and would likely have objectives related to the decision-making process and attitude change over time. RCTs exploring this seem unlikely and repeated surveys would not provide effectiveness data.To explore opinions and attitudes, we included qualitative studies and mixed-methods studies with primary qualitative data collection. We choose to focus on primary qualitative data as this data in this context. In keeping with Cochrane guidance, we do not consider a questionnaire survey to be a qualitative study (https://training.cochrane.org/handbook/current/chapter-21). We prioritized inclusion of qualitative data from primary studies over free text from questionnaire surveys as we hypothesized primary data would be richer and thicker and thus more informative. We have clarified this in the methods section as follows: “We prioritized the inclusion of qualitative data from primary studies over free text from questionnaire surveys as we hypothesized primary data would be richer and thicker and thus more informative.”                                                                                Sometimes the arguments are not fully clear. In the introduction, attention is paid to problems complicating the conduct of systematic reviews. This is not the core focus of the project. The project is linked to the translation phase of turning review findings into evidence summaries. This is a matter of content, form and channels rather than the SR pre-track issues. It would therefore be more appropriate to highlight the diversity of actors involved in producing guidelines in relation to how communication and dissemination channels have changed because of that. I am a big fan of the authors discussing things like podcasts and multimedia accounts as examples. Naturally, this is linked to a society moving into multimodality and the limitations of numbers and narratives in reaching out to or working with different publics.Author’s response: Thank you for this feedback. We have edited the introduction to try to truncate the paragraph on systematic reviews and to try to relate more to the core focus of the project. The process of conducting a systematic review is related to the time and resources the team will have to create the summary formats themselves so we have largely kept the information about the ‘pre-track’ problems. Furthermore, we are focusing on multiple types of evidence syntheses and the landscape has recently evolved in this regard. This also directly relates to the search strategy used and inclusion criteria.    The PICO would perhaps work better if clinicians is changed into professionals. It would make it relevant to a broader target group.  Also, the comparator could also be no summary at all, for example, reading findings from reviews straight into the guideline procedure. Why are these explicitly excluded from the review?Author’s response: Thank you for this question. We have changed clinicians to healthcare professionals to be more inclusive. We have excluded no summary at all as a comparator because we do not think it is appropriate to compare the reading experience of a summary format versus none. We have added text to the ‘Quantitative systematic review’ section: Studies where the comparison is no intervention  will be excluded. Producing a full text of an evidence synthesis without some form of summary is not ‘usual care’. Internationally, organisations, journals and reporting guidelines would consider a summary to be a mandatory component of any report/peer review manuscript produced .  Therefore no form of summary or abstract ‘is not an appropriate comparator  Do the authors only include qualitative studies where an intervention is evaluated by participants in terms of meaningfulness or applicability? Or do they also include qualitative studies where attitudes, viewpoints and options for form/channel/content are discussed from a more theoretical point of view, or in pilot cases? It is not clear where the selection sits on the qualitative side, but the data collection part suggests the first option.Author’s response:  Thank you for this question. We have chosen the SPiDER approach for the qualitative evidence synthesis as we are not restricting qualitative studies to those that evaluate an intervention in terms of meaningfulness of applicability. Our evaluation outcomes include the ‘Views, attitudes, opinions, experiences, perceptions, beliefs, feelings, understanding’ of participants. The information in the data collection section regarding the qualitative data is just to provide a summary overview of the study’s main findings (outcomes) to help with descriptive analyses. We edited the text to emphasize that NVivo12 coding will be conducted ‘in parallel’ .  Perhaps the authors may want to bring a rationale for why they want to limit the formats to textual and audio-visual productions, as there are many more forms available through which evidence is and can be communicated  Perhaps these forms haven't been studied extensively in the context of guidelines, but in a protocol phase we are not sure what to find yet :-).Author’s response: Thank you for this suggestion. We did not list any exclusion of formats explicitly, nor did we limit the formats to textual or audio-visual productions. In the list provided, we used e.g. ‘e.g., visual, audio, text based’ (for example) to list potential summary modes we may find rather than ‘i.e.’ (that is) to explicitly list summary modes we are restricting our search to. Our search strategy included specific formats that we thought we might find but we would not exclude other types of communications that weren’t explicitly listed. We have added text to the ‘quantitative systematic review’ section to clarify this ‘We will not exclude a summary format if it is one that we did not explicitly list in our search strategy (Table 3).’  I would find it more practical if the authors would discuss their position towards critical appraisal and how to structure this instead of putting it in the data collection part as an extraction category . Also, critical appraisal checklist for primary studies have a different purpose than GRADE and CONQUAL that work on a review level . I think the authors need to spell the two out in more detail, as they are used in different phases of the review process.Author’s response: Thank you for this suggestion. In reporting this protocol, we have followed the JBI manual and as such the ‘bias and quality assessments’ subheading should be it’s own unique section and does not fall under the data collection section. The JBI critical appraisal tools are mentioned in the data collection section as they are a part of the standard data-extraction form that will be used by the reviewers. These should be two separate sections; apologies if there was a formatting error. We have edited the text within the ‘data collection’ and ‘bias and quality assessments’ sections to try to clarify what will be extracted for what purpose and to clarify the GRADE/ConQual concern. The text changed in the data collection now reads: ‘The following data information will be extracted using the pilot-tested standardized data-extraction form’ and ‘Data from the domains listed within the JBI critical appraisal tools for qualitative and quantitative studies.’ We agree that the critical appraisal checklist for primary studies have a different purpose than GRADE and CONQUAL. We have reworded the information in the ‘bias and quality assessments’ section to clarify that the JBI checklists are for individual studies AND there is no checklist recommended for the final assessment in the entirety of evidence (similar to the GRADE and CONQUAL purposes) for a mixed methods systematic review.  Would quality of a study not rather fit a sensitivity analysis rather than a subgroup analysis? What purpose would it have as a subgroup in the review? It has disappeared in the mixed method part as a subgroup analysis, so wouldn't mention it as a subgroup in other parts of the text.Author’s response: Thank you for this question. We recognize that the text in the quantitative analysis section was unclear. We have edited the text, deleting the for example list and restricting it only to the subgroups of interest (participant group or summary format type). We note subgroup analyses for participant groups or intervention types as these involve comparisons across the different groups. There are no planned sensitivity analyses for study quality as there will likely be far too much heterogeneity in outcomes  to be able to compare the different methods. What do the authors mean with the phrase "If textual pooling is not available, a narrative summary will be presented." Is a narrative summary not already a textual pooling? Unclear.Author’s response: Thank you for this question. Apologies for this issue. We have edited the text accordingly. ‘If we are unable to pool findings together , likely due to an insufficient number of studies identified, a narrative summary will be presented.’  While configuration might have different meanings, it is generally not commonly used as a metaphor for a separate type of synthesis. Configuration would make the different parts unrecognizable in the whole. I don't think meta-aggregation would achieve this level of integration. Perhaps integration is better than configuration here.41, 42’ Meta-aggregation is solely used for the qualitative analysis, not for the mixed methods synthesis which includes the integration.  Author’s response: Thank you for this suggestion. We have reworded the mixed methods synthesis section to make our approach more clear and have added a references to the specific sections of the JBI manual that we are referencing regarding configuration and the thematic analysis . We are following JBI’s guidance for MMSRs, and have chosen to use a convergent segregated approach. We have added additional text to clarify that ‘There are several methods for integrating qualitative and quantitative evidence syntheses26, [2] in a convergent segregated MMSR. We will use a thematic synthesis method for integration which groups together similar codes, develops descriptive categories (or themes) to create an overall summary of findings. Also, thematic synthesis is not the same as meta-aggregation, and line by line coding is certainly not a part of meta-aggregation. I believe the authors need to make an informed choice between qualitative evidence synthesis approaches and then follow the guidance through.Author’s response: Thank you for this comment. Apologies for the confusion but we are not using a meta-aggregation approach for the mixed methods synthesis – only for the qualitative analysis which is done prior to the integration of the two separate streams of evidence. We have edited the mixed methods synthesis section to further clarify our methodological approach which is in accordance with chapter 8 of the JBI manual.


To support evidence-informed decision-making, we created an evidence gap map to characterise the evidence base on the effectiveness of interventions in improving routine childhood immunisation outcomes in low-income and middle-income countries (LMICs).We developed an intervention–outcome matrix with 38 interventions and 43 outcomes. We searched academic databases and grey literature sources for relevant impact evaluations (IEs) and systematic reviews (SRs). Search results were screened on title/abstract. Those included on title/abstract were retrieved for full review. Studies meeting the eligibility criteria were included and data were extracted for each included study. All screening and data extraction was done by two independent reviewers. We analysed these data to identify trends in the geographic distribution of evidence, the concentration of evidence across intervention and outcome categories, and attention to vulnerable populations in the literature.We identified 309 studies, comprising 226 completed IEs, 58 completed SRs, 24 ongoing IEs and 1 ongoing SR. Evidence from IEs is heavily concentrated in a handful of countries in sub-Saharan Africa and South Asia. Among interventions, the most frequently evaluated are those related to education and material incentives for caregivers or health workers. There are gaps in the study of non-material incentives and outreach to vulnerable populations. Among outcomes, those related to vaccine coverage and health are well covered. However, evidence on intermediate outcomes related to health system capacity or barriers faced by caregivers is much more limited.There is valuable evidence available to decision-makers for use in identifying and deploying effective strategies to increase routine immunisation in LMICs. However, additional research is needed to address gaps in the evidence base. We adopt a rigorous and systematic approach to literature mapping.We map a broad swath of the literature on routine childhood immunisation in low-income and middle-income countries.interactive online map.Our analysis is complemented by an Coverage of intermediate outcomes  in this evidence gap map may be incomplete if studies on these outcomes do not target routine vaccination specifically.Immunisation remains one of the most cost-effective interventions to prevent and control life-threatening infectious diseases. A 2016 study found that for every US dollar invested in vaccination in the world’s 94 lowest-income countries, US$16 can be saved in healthcare costs, lost wages and lost productivity due to illness and death.4In the last few decades, interventions have emerged specifically aimed at strengthening national-level immunisation programmes by addressing delivery constraints. However, even with strengthened routine immunisation programmes, marginalised and vulnerable communities are at risk of being overlooked. Thus, a number of international and national policy frameworks are increasingly focusing on multipronged approaches that provide contextualised solutions to address constraints in delivery of immunisation services as well as behavioural, social and practical barriers faced by caregivers.This study presents an evidence gap map (EGM) of impact evaluations (IEs) and systematic reviews (SRs) measuring the effects of interventions on outcomes related to routine child immunisation. An EGM aims to establish what is known and unknown about an evidence base in a thematic area.EGMs highlight both absolute gaps (an empty cell in the framework), which could be filled with new IEs, and synthesis gaps (multiple IEs but no SRs), which could be filled with evidence synthesis. Furthermore, they can be used to highlight potentially over-researched areas, where the effects of a particular intervention have been evaluated against a range of outcomes (or vice versa) and where additional studies may not be necessary.The objectives of this EGM are to:Present a framework of types of interventions and outcomes related to routine child immunisation in LMICs.Using this framework, map available SRs and IEs by their interventions and outcomes, with a summary of findings provided in this paper.Assess the quality of the SRs falling under the framework.Drawing on the literature on routine child immunisation in LMICs, we developed a theory of change in which we identified key factors—from the perspectives of caregivers and communities, vaccinators, health systems and beyond—that influence whether children receive the full course of routine vaccinations.The theory of change suggests how interventions may be oriented to address afore-mentioned key factors and lead to greater immunisation coverage and better health outcomes . It suggWe included IEs and SRs evaluating the effectiveness of interventions to increase immunisation of children under five in LMICs. Interventions did not have to be expressly designed for the purpose of increasing immunisation rates; rather, we searched for all IEs and SRs that provided evidence on the impact of any intervention on routine immunisation coverage, morbidity or mortality from vaccine-preventable diseases, or any intermediate outcome that precedes them in the theoretical causal chain. There were no restrictions by publication date, publication type/status or language.The EGM includes experimental and quasiexperimental IEs that estimate the causal impact of an intervention, as compared with usual practice or an alternative treatment, by establishing a counterfactual.9To develop the intervention–outcome framework for this EGM, we developed an initial matrix by drawing on the research literature on determinants and constraints related to vaccination in LMICs,This resulted in a framework with 38 unique interventions and 43 unique outcomes. These are each grouped together in a three-level hierarchy of categories. The interventions are grouped broadly by who or what is targeted: caregivers, the health system, non-caregiver community members, the community as a whole or policies and institutions beyond the health system. Our outcome taxonomy has three levels. At the highest level, we divide outcomes into three broad types: (a) those related to behavioural, social and practical barriers faced by caregivers and communities, (b) those related to delivery of vaccination services by formal and informal components of the health system and (c) those related to vaccination coverage/timeliness and health outcomes . Within each of these types are a series of mid-level (broad) categories, and within each of these are specific outcome categories. The inclusion of intermediate outcomes in (a) and (b) allows us to assess the extent to which the evidence base contains insights not only on the effectiveness of interventions in improving vaccination coverage/timeliness and health outcomes but also the effectiveness of interventions in improving their important causal antecedents. The outcomes in category (c) are further divided into more specific categories corresponding to various prerequisites for universal vaccination coverage. The framework also includes seven ‘cross-cutting themes’ that capture possible subgroup analyses  as well as other key features of interest . A sketch of the matrix, showing the first-level and second-level intervention and outcome categories, appears in 10.1136/bmjopen-2021-058258.supp1Supplementary dataMany studies evaluated interventions that included multiple components  and sensitisation meetings with women). We determined that the best approach was to capture all major intervention components for each study. That is, some studies have multiple intervention codes applied to them. In this way, the EGM can be considered a map of ‘instances of evidence’ rather than a map of studies. A study provides an ‘instance of evidence’ when it provides evidence about the impact of a particular type of activity on a particular outcome. Because many studies assess multiple interventions and/or multiple outcomes, many studies provide multiple instances of evidence. While mapping intervention components separately has limitations (see Limitations section below), it is well suited to a literature like that covered in this EGM, where multicomponent interventions are common, but there are few ‘standard packages’ of intervention components that are frequently implemented and evaluated together. This approach also has the advantage of allowing users of the map to easily identify studies that evaluate particular intervention components of interest.This EGM employed systematic methods for search, screening and data extraction, following International Initiative for Impact Evaluation recommended guidelines for EGMs.All search results were screened on title/abstract by a team of trained reviewers. Two reviewers independently screened each abstract. Studies included at the title/abstract level were retrieved for full review. Full texts were also independently screened by two reviewers. All screening disagreements were resolved through discussion.Data extraction was conducted independently by two trained coders using a standard template . For eacWhen our search identified multiple publications resulting from the same study , we counted these as a single study and data extraction for each study drew on all the publications identified for that study.When studies evaluated interventions with multiple components, we coded each intervention separately. While this approach captures important detail about the intervention activities that have been included in IEs, it can also overstate the apparent amount of evidence by ‘double-counting’ studies with multicomponent interventions. Hence, we also conducted an analysis with all multicomponent studies grouped into a single ‘multicomponent’ category, rather than having each component coded individually.For SRs, we conducted critical appraisals using a standardised tool for assessing the quality and systematicity of each review’s search, screening, data extraction and analysis methods . Based oAs the unit of analysis for this EGM is research studies rather than human subjects, there was not a need to involve individual patients. However, to ensure the relevance of our research to a broad array of stakeholders, we engaged an advisory group comprising immunisation experts from the research, practitioner and policy-maker domains. This group advised on the development of our framework, search strategy, data analysis and dissemination plans.A total of 85 countries are represented among the IEs included in the EGM. The regions most commonly represented are sub-Saharan Africa (95 IEs) and South Asia (75 IEs). There is comparatively little evidence from Europe and Central Asia (five IEs) or the Middle East and North Africa (three IEs), with somewhat greater numbers in the East Asia/Pacific and Latin America/Caribbean regions. India is by far the most represented country with 45 IEs. Other South Asian countries are also well represented, including Pakistan (12 IEs) and Bangladesh (9 IEs). Evidence is fairly well-distributed across sub-Saharan Africa, though with some notable gaps. Nigeria is the most represented country with 14 IEs, followed by Kenya (12 IEs), Rwanda (6 IEs) and Ethiopia (7 IEs).The intervention type most commonly evaluated is caregiver incentives and motivation, with 79 IEs and 35 SRs  Because yields nThe most commonly examined specific interventions are material/monetary incentives for caregivers , health system strategic planning  and written or pictorial messages for caregivers , which includes SMS reminders or motivational messages. There is comparatively little work on non-material incentives for caregivers , which includes, for example, community recognition as an incentive to vaccinate. Among interventions to motivate HWs, pay-for-performance schemes are by far the most commonly evaluated .Several evidence gaps are evident. Apart from monetary incentives and pay-for-performance schemes, there is very little evidence on incentives, motivation and informational messages for HWs. There are also significant gaps in outreach to migrant populations  and campaigns to vaccinate refugee populations (no IEs or SRs). We did not identify any IEs or SRs with interventions framed specifically as outreach to vaccine-hesitant groups, although there is some evidence on outreach to vulnerable populations , which may overlap with vaccine-hesitant groups to some extent. Cold chain infrastructure improvements present another notable gap, with only two IEs and one low-confidence SR evaluating such interventions. There is a potential synthesis gap with respect to community HW training and education, for which there are 22 IEs but no medium-confidence or high-confidence SRs.Of the 226 IEs in our EGM, 87 evaluated multicomponent interventions . We findResources and financing in this review met our methodological inclusion criteria). A potential synthesis gap exists on community HW supply and capacity, for which we identified seven IEs but no SRs.By far, the most commonly measured outcome categories fall under vaccination coverage, with 210 IEs and 52 SRs examining 1 of these outcomes . Health Regarding outcomes related to behavioural, social and practical barriers faced by caregivers and communities, some outcome categories are relatively well studied, while others have very little evidence. We identified 18 IEs and 8 SRs that considered caregivers’ attitudes about immunisation. The practical factors that influence caregivers’ opportunities to vaccinate  are also comparatively well studied, with 19 IEs and 5 SRs . In contrast, few studies examine the social processes affecting caregivers’ vaccination decisions  or their readiness to vaccinate .Only two specific outcome categories on the caregiver side of our framework were examined in more than 10 studies: caregivers’ knowledge about immunisation  and retention of vaccination cards  . EvidencRelatively little evidence examines specific outcomes related to delivery of immunisation services . The twoVaccination coverage and child health outcomes are fairly well studied and there are limited evidence gaps or synthesis gaps . One gapOur map identified much more evidence on vaccination coverage than on downstream health outcomes. This pattern likely appears in our map because we only included studies measuring morbidity or mortality if they also measured vaccination coverage—that is, if they provided data that could be used to assess whether the intervention affected morbidity and mortality via immunisation coverage, as opposed to other changes produced by the intervention.Commonly examined interventions such as caregiver incentives and motivation and health system governance, policies and financing are frequently evaluated for their effect on vaccination coverage outcomes. In comparison, most interventions have occasionally been evaluated for their effects of intermediate outcomes that precede coverage in the theoretical chain: behavioural, social and practical barriers faced by caregivers and communities or the health system’s capacity to deliver vaccination services. Some exceptions are (1) sustained sensitisation and education campaigns, which are frequently evaluated for their effect on knowledge of immunisation and (2) formal HW training and education interventions, which have been evaluated for their effects on HW motivation and capacity.We examined included studies for the presence of evidence about themes that cut across our intervention and outcome framework. Many of these themes affect whether the study provides evidence about the effects of an intervention on particular vulnerable populations, either via subgroup analysis or by evaluating an intervention specifically targeted to that population.Evidence patterns are quite similar across equity themes, with attention to each of these themes being relatively rare in IEs . The mosWe recorded whether included studies provided any sort of information about the costs of the intervention, including any form of cost-effectiveness or cost–benefit analysis. Including such cost information is relatively rare: only 37 IEs (16%) and 9 SRs (15%) provide any such information.About two-thirds of the 58 included reviews (39 SRs) received ‘low’ confidence ratings, with the remainder split roughly equally between ‘medium’ (9 SRs) and ‘high’ (10 SRs) confidence ratings.Most EGMs serve two primary purposes.Reflecting the first purpose, this map can serve as a tool for policy-makers and practitioners working in the routine immunisation space to find rigorous evidence to inform decisions about which programmes to implement and how to implement them. High-quality syntheses of rigorous evidence are often the best sources of information on the effectiveness of interventions available to policy-makers and practitioners. Therefore, we recommend consulting the high-confidence syntheses identified in this EGM to inform the selection, design and delivery of interventions to improve immunisation outcomes  behavioural, social and practical barriers faced by caregivers or (b) outcomes influencing the delivery of vaccination services such as the supply and capacity of formal HWs, supply chain management and the use of immunisation data in monitoring and planning. There is likely additional evidence on these intermediate outcomes, but our EGM suggests that it is relatively rare for rigorous quantitative studies to assess them in the specific context of promoting vaccination coverage.In general, our EGM did not identify any glaring synthesis gaps in the form of large clusters of IEs on similar topics but no medium-confidence or high-confidence SRs synthesising the results of these evaluations. However, we note that there are 44 IEs on health provider incentives and motivation interventions, but only one high-confidence SR, which is now nearly a decade old.Among SRs in our sample, the majority (67%) received a confidence rating of ‘low’ based on their methods. The SRs in our sample frequently lacked comprehensive search strategies and appropriate reporting of evidence, particularly with respect to the risk of bias of included primary studies . RecogniOur approach to coding multicomponent interventions presents some limitations. First, it risks overstating the volume of available evidence because the same study can appear in multiple rows in the map. Second, it can potentially distort theories of change. This is because, from looking at the map, it can appear that there is rigorous evidence about the effect of an intervention on an outcome, whereas in actuality there is only evidence about how that intervention affects the outcome as a component in a multicomponent intervention. To address this limitation, we provided an alternative analysis wherein all multicomponent interventions are classified under a single ‘multicomponent’ category .Furthermore, it should be emphasised that our search strategy might have missed studies focused on intermediate outcomes, if assessing impact on vaccination-related outcomes was not a central component of the study. Therefore, these data should not be interpreted as showing, for example, that there is virtually no evidence about how to strengthen capacity among health service administrators. Rather, our EGM suggests that this outcome is rarely considered in the context of vaccination specifically.Our identification of evidence gaps can provide direction for researchers and funders of research about high-priority areas for future investigation. The most glaring gap concerns intermediate outcomes on the capacity of the health system to deliver vaccination services, and on behavioural, social and practical barriers faced by caregivers. As noted above, our EGM results do not necessarily suggest an absolute dearth of evidence on health system strengthening. Rather, we find a lack of specific evidence tying health system capacity to the delivery of vaccination services. Thus, going forward researchers may wish to consider assessing the extent to which improvements in health system capacity lead to improvements in immunisation coverage, by including both as outcomes in evaluations. Specific areas requiring additional attention include: caregivers’ attitudes about health providers, the social and practical factors influencing caregivers’ decisions about vaccination and their access to vaccination services, the supply and capacity of formal HWs and administrators responsible for delivering vaccination services, use of data in monitoring and improving vaccination services and coverage of IPV.Future research should pay greater attention to equity in immunisation studies—as emphasised in WHO’s Immunization Agenda 2030High-quality syntheses of rigorous evidence are often the best sources of information available to policy-makers and practitioners on the effectiveness of interventions. Therefore, we recommend consulting the high-confidence syntheses identified in this EGM to inform the selection, design and delivery of interventions to improve immunisation outcomes , one can consult low-confidence SRs (with appropriate caution) and individual IEs, although it is important not to place too much weight on any individual study. For those deciding to launch an intervention where relevant IE evidence is lacking, we suggest considering whether incorporating an IE of the programme would be feasible.


Clinical decisions in human and veterinary medicine should be based on the best available evidence. The results of primary research are an important component of that evidence base. Regardless of whether assessing studies for clinical case management, developing clinical practice guidelines, or performing systematic reviews, evidence from primary research should be evaluated for internal validity i.e., whether the results are free from bias (reflect the truth). Three broad approaches to evaluating internal validity are available: evaluating the potential for bias in a body of literature based on the study designs employed (levels of evidence), evaluating whether key study design features associated with the potential for bias were employed , and applying a judgement as to whether design elements of a study were likely to result in biased results given the specific context of the study (risk of bias assessment). The level of evidence framework for assessing internal validity assumes that internal validity can be determined based on the study design alone, and thus makes the strongest assumptions. Risk of bias assessments involve an evaluation of the potential for bias in the context of a specific study, and thus involve the least assumptions about internal validity. Quality assessment sits somewhere between the assumptions of these two. Because risk of bias assessment involves the least assumptions, this approach should be used to assess internal validity where possible. However, risk of bias instruments are not available for all study designs, some clinical questions may be addressed using multiple study designs, and some instruments that include an evaluation of internal validity also include additional components . Therefore, it may be necessary to embed questions related to risk of bias within existing quality assessment instruments. In this article, we overview the approaches to evaluating internal validity, highlight the current complexities, and propose ideas for approaching assessments of internal validity. Every day in clinical practice, veterinary professionals need to make decisions ranging from a decision as to whether (or not) to use an intervention or to apply a diagnostic test, to decisions about the overall management of complex clinical conditions. Increasingly, it is expected that clinical decisions are evidence-based. Evidence-based veterinary medicine incorporates clinician experience, client preferences, animal needs, and scientific evidence when making clinical decisions . In thisFormal methods have been developed to systematically identify, select, and synthesize the available evidence to assist veterinary professionals in evidence-based decision-making. These include critically appraised topics (CATs) , systemaBox 1Systematic review, meta-analysis, and network meta-analysis: Systematic review is a structured methodology for identifying, selecting and evaluating all relevant research to address a structured question, which may relate to descriptive characteristics such as prevalence, etiology, efficacy of interventions, or diagnostic test accuracy  use the same principles as systematic reviews to address clinical questions but employ a more rapid approach, particularly in relation to the screening and summation of the evidence. They were designed to be employed by clinicians as a way of rapidly gathering and interpreting evidence on clinical questions relating to specific cases  . AlthougThere are several terms used to describe the approaches to assessing internal validity of primary research studies, including evidence hierarchies and levels of evidence, quality assessment, and risk of bias assessment. The use of these terms may be confusing, and it is not uncommon for some of these terms to be used interchangeably , 15. AlsLevels of evidence is an approach to evaluating the internal validity of a body of evidence, based on the potential for bias which is inherent to the employed study designs that were used to address the clinical question. The concept behind levels of evidence is that there is a hierarchy of study designs, with different study designs having different potential for bias. The way evidence hierarchies are used is based on either the name of the design or the description of the design. Readers of a study look for this information, then determine the design and assign a level of evidence. No further differentiation of methodological features or judgment is conducted.Evidence hierarchies were initially introduced in 1979 by the Canadian Task Force on the Periodic Health Examination . A pyramThe concept of evaluating the potential for bias in an individual study based on the study design can be extended to an evaluation of the potential for bias in a body of literature. This approach for evaluating the internal validity of a body of literature is referred to as “levels of evidence”. The approach is applied by identifying research (or other evidence) that pertains to the clinical question, determining the study design used for each of the studies, and then assigning each study to a level of evidence based on that design. For instance, a framework for levels of evidence in veterinary clinical nutrition has been proposed by Roudebush et al. . In thisThe levels of evidence approach may be perceived as a quick and easy approach to assessing internal validity because it requires only a knowledge of the study design employed and not the individual features of a study that may or may not be associated with the potential for bias. However, that ease of use is based on very strong assumptions: 1) that study design maps directly to bias, 2) that authors always correctly label study designs, and 3) that authors execute and report study designs appropriately. The approach also pertains to a body of evidence, implying that there are multiple comparable studies available to address the question of interest.An important critique of levels of evidence is that the approach focuses on the study design, rather than the actual design features that were used or the context of the study. Thus, although this framework illustrates the inherent potential for bias of the different study designs, it does not provide a consideration of the methodological rigor with which any specific individual study was conducted . For insAn additional consideration is that for questions related to aspects of clinical care other than selection of interventions, the framework and positioning of study designs included in https://www.merriam-webster.com/dictionary/quality). Quality generally is understood to be a multi-dimensional concept. While clear definitions are difficult to find in the research literature, the lay literature includes numerous treaties on the dimensions of quality. One example is the eight dimensions of quality delineated by David Gavin, which include performance, features, reliability, conformance, durability, serviceability, aesthetics, and perceived quality .As the name implies, quality assessment represents an evaluation of the quality of a primary research article. However, the term “quality” is difficult to specifically define in the context of evidence-based medicine, in that it does not appear to have been used consistently in the literature. The Merriam-Webster dictionary defines quality as “how good or bad something is” or “a high level of value or excellence”  or may be intended to reduce the potential for information bias (blinding of outcome assessors). Conducting a quality assessment is more complicated and time-consuming than evaluating levels of evidence because the presence or absence of the specific design features needs to be identified and validated within the study report. However, the approach requires only that the person evaluating internal validity can identify whether (or not) a design feature was used. Therefore, this approach requires more technical expertise that the levels of evidence approach, but less than the risk of bias approach.Risk of bias assessments have been developed specifically for evaluating the potential for elements of the design or conduct employed within a study to lead to a biased effect size , 35. TheSome components of a risk of bias assessment are the same as those included in a quality assessment approach . However, the way the assessment is done differs, with quality assessments generally involving present/absent judgements as opposed to assessments as to whether the risk of bias is likely or not. Hartling et al.  applied Although the critical elements for risk of bias are well described for RCTs in human healthcare and to a large extent in veterinary RCTs, these elements are not as well described for non-randomized trials and observational studies where allocation to groups is not under the control of the investigator. There are some risk of bias tools available for assessing risk of bias in non-randomized studies, such as ROBINS-I . HoweverCurrently, the available approaches to assessing internal validity tend to be used for different applications. Levels of evidence have previously been used for creating evidence-based recommendations or clinical practices guidelines , 40, 41,However, the different approaches are not necessarily mutually exclusive, but are nested within each other based on assumptions, and the methodology and use of the different approaches has evolved over time. As previously described, a criticism of the use of levels of evidence is that the potential for bias is based on the study design that was employed, rather than the methodological rigor of a specific study . For thiAnother example of the evolution of approaches to assessing internal validity is from the Cochrane Back review group, who conduct systematic reviews of neck and back pain. The initial methods guidelines, published in 1997, recommended that a quality assessment be performed on each included study, with each item in the quality assessment tool scored based on whether the authors reported their use . UpdatedThe examples from the human medical literature illustrate that assessment of internal validity need not be static, and that modifications to our approach to assessing internal validity can strengthen the evidence base for clinical decision making. When developing or using tools which include an evaluation of internal validity, the assessment of internal validity should use the approach with the least assumptions about bias. This implies that the risk of bias approach, where context specific judgements are made related to the potential for bias, is the preferred approach for assessing internal validity. The risk of bias approach is well developed for RCTs. Therefore, when RCTs are included in the evidence available to address a clinical question, a risk of bias assessment approach should be used. When evaluating internal validity as a component of a SR-MA, the Cochrane ROB2.0 tool  could beHowever, there are circumstances where these recommendations may not be appropriate or sufficient, such as for observational studies where risk of bias assessment instruments do not formally exist, or where a variety of study designs have been identified that answer the clinical question (particularly non-intervention type questions). When observational studies are used as evidence, individuals assessing internal validity may wish to evaluate risks of bias for each study ad hoc by considering the specific risks of bias related to selection bias, information bias, and confounding in the context of the topic area. However, this approach requires considerable methodological expertise. Alternatively, a quality assessment approach could be used to evaluate internal validity for observational studies, recognizing that more assumptions related to the potential for bias are involved. As instruments for evaluating the risk of bias for observational studies are developed and validated, these could replace ad hoc or quality assessment approaches.For situations where the evidence base includes multiple study types, such as clinical practice guidelines, the use of levels of evidence may be useful for framing the potential for bias inherent in the studies identified to address the clinical questions. However, within each evidence level, there still is a need to evaluate the internal validity of each study. The proposed approach for situations where RCTs and observational studies are included in the evidence base was described in the preceding paragraphs. For lower levels of evidence, such as case series, textbooks and narrative reviews, and expert opinion, levels of evidence could be used to emphasize that these types of evidence have high potential for bias based on their design.It should be noted that although this article has focused on approaches to evaluating internal validity of studies, this is only one component of the assessment of evidence. Critical appraisal, CATs, SR-MA, and clinical practice guidelines explicitly incorporate other aspects of decision-making, including a consideration of the magnitude and precision of an intervention effect or the potential clinical impact, the consistency of the research results across studies, the applicability  of the research results, and the directness of the evidence to a clinical situation . However, a discussion of these components for decision-making is beyond the scope of the current study. The interested reader is referred to further details on the components used in evaluating evidence for CATs , for SR-JS drafted the manuscript. All authors contributed equally to the conceptualization of this work. All authors read and approved the final contents.Partial funding support was obtained from the University of Guelph Research Leadership Chair (Sargeant).The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.