RepliesI want to thank all the contributors for their thoughtful comments on the book, Brown .I’ve trFratantonio and Lasonen-Aarnio (henceforth FLA) argue that an infallibilist can evade two of my main objections. The first of these objections was my claim that infallibilism can avoid scepticism about important forms of knowledge only by endorsing SKSS: if S knows that p, then p is part of S’s evidence for p. But, as I pointed out, it is standardly infelicitous to cite a proposition as evidence for itself.Following Fratantonio  FLA suggIn reply, it seems to me that even the second justifying reading of SKSS also gives rise to infelicity data. When one asks for the evidence for p in this second sense, one is asking for evidence which provides justification to believe p, or reason to believe p. But, intuitively, p doesn’t provide justification or reason to believe that p. For instance, I might be interested in the substantive question of whether the government lifted coronavirus restrictions too early. Knowing that my friend believes that’s so, I might ask, “What’s your evidence that the government lifted restrictions too early?” Here it would be infelicitous for my friend to reply to my question by merely restating that the government lifted coronavirus restrictions too early. Thus, whether we are interested in the motivating or justifying sense of “evidence for p” it seems infelicitous to cite p as evidence for p.Setting that aside, FLA suggest that the infallibilist needn’t accept SKSS even understood to involve the notion of justifying evidence for p. Relatedly, they suggest that the formulation of infallibilism I employ doesn’t accurately characterise certain positions. Their argument for these points rests on the idea that one might come to embrace infallibilism as a result of commitment to a recognisably fallibilist view of knowledge and the sufficiency of knowledge for evidence (SKE). The relevant fallibilist view is an explanationist account of evidential support on which S’s evidence e provides evidential support for p if and only if p is the best sufficiently good or plausible explanation for why S has e. Since p might provide the best sufficiently good explanation of why S has e even if e doesn’t entail that p, this looks like a fallibilist position. But, FLA argue that when combined with SKE it entails infallibilism*, so would be wrongly classified as an infallibilist position. Furthermore, they argue that this style of infallibilism wouldn’t entail SKSS for it’s implausible that a proposition p is the best sufficiently good explanation for p.evidence for d entails that d.  But there seems no reason for the fallibilist explanationist to accept that S’s evidence for d has expanded to include d. Indeed, she has reason not to accept this. If whenever a subject knows that p, p is treated as part of the subject’s evidence for p, then we cannot say that some known claims have better or stronger evidential support than others. But given that the explanationist holds that it’s sufficient for S to know that p that p provides the best sufficiently good or plausible explanation for why S has the evidence, it’s part and parcel of the view that some explanations are better than others and so some claims are better evidentially supported than others. If, as I suggest, the explanationist position isn’t plausibly committed to infallibilism*, then its relation to SKSS is irrelevant to whether infallibilism is committed to SKSS.Let’s examine the suggestion more carefully. Suppose that the explanationist grants that S knows by inference to the best explanation that the dog ate the birthday cake, or d, since d is the best explanation of S’s evidence, namely the missing cake (c), the dog’s guilty look (g), together with facts about the location of the cake and the dog (l). So far, we needn’t suppose that S’s evidence for the claim that the dog ate the birthday cake, namely {c, g, l}, entails that the dog ate the birthday cake. Do things change if we add SKE? Of course, if S’s evidence now includes d, then her evidence does entail d. But to show that this position amounts to infallibilism* we need to establish the further claim that her FLA then turn to consider my arguments in defence of defeat. In particular, they question my argument against level-splitting views on which a subject can rationally be in an epistemically akratic state, such as believing that p while also believing that her evidence does not support that p. Against such views, I argue that they sanction certain kinds of bad reasoning. FLA worry that this problem is faced not only by level-splitters but also by fallibilists. They provide an example in which S’s evidence both supports that p and that she has evidence that her evidence does not support that p. If a fallibilist were to endorse the evidentialist view on which one has justification to believe what is likely on one’s evidence, S has justification to believe both p and that she has evidence that her evidence does not support that p. But, part of my argument against level-splitting views is that if such a combination of beliefs is justified, it will lead to blatantly bad reasoning. So, they question how a fallibilist can avoid the kind of bad reasoning I argue affects level-splitting views. In reply, and as I argue in the book, the fallibilist should reject the relevant evidentialist view and endorse certain inter-level coherence requirements which require coherence between the attitudes she takes at first-order and the attitudes she has at higher-levels. While the book mentioned coherence requirements between first-order beliefs and beliefs about what one’s evidence supports, for similar reasons one might defend coherence requirements between first-order states and beliefs about what one’s evidence supports about what one’s evidence supports. This would deal with the problematic examples and would seem natural from the point of view of somebody already committed to inter-level coherence principles.While Goldberg agrees with much of my case against infallibilism, he thinks that the case of testimonial knowledge may not present such a problem for infallibilism as I suggested. I argued that testimony presents a problem because it is one of several kinds of knowledge where what is intuitively one’s evidence for a known claim does not entail what is known. In reply, Goldberg suggests a more generous view of testimonial evidence which is supposed to help infallibilism.Goldberg suggests that when all goes well, testimony constitutes a certain kind of epistemic link to the facts, where a subject S is epistemically linked to the fact that p when she truly believes that p and her belief that p was acquired in a way W and in conditions C such that, in C, true beliefs acquired through W constitute knowledge. The claim that testimony can constitute such an epistemic link doesn’t in itself say anything about testimonial evidence. However, Goldberg suggests that this general approach can be developed in a variety of epistemological frameworks in such a way that it does make claims about testimonial evidence. First, he suggests it might be combined with Dretske’s notion of a conclusive reason, on which a reason r is a conclusive reason for S to believe that p just in case S wouldn’t have r unless p. But even if testimony can, when things go well, provide a conclusive reason in this sense, the reason doesn’t entail what is known. To say that S would not have testified that p unless p, is not to say that there is no possible situation in which S testifies falsely that p.Now consider Goldberg’s other suggestion, an appeal to “extended evidentialism” according to which a hearer’s testimonial evidence for p includes “not just the fact that a (seemingly sincere and competent) speaker says so but also the facts that constitute S’s relevant evidence—the evidence on the basis of which S said so”. On this view, when a hearer, H, gains knowledge that p from the testimony of a speaker, S, whether H has evidence which entails p depends on whether S does. For instance, suppose that Watson gains knowledge that p from the testimony of Holmes, where Holmes doesn’t tell Watson what his evidence for p is. On extended evidentialism, whether Watson has evidence that entails that p depends on whether Holmes does. Holmes might have deduced p from known premises; but, alternatively, he might have come to know p by inference to the best explanation from premises which do not entail p. Either way, testimony doesn’t provide any additional problem for infallibilism. Rather, the core question is whether knowledge other than testimony meets the infallibilist criterion.However, while appeal to extended evidentialism would in this way show that testimony doesn’t provide any additional problem for infallibilism, it’s a highly controversial view. It’s in tension with the widely held view that a proposition is part of S’s evidence only if it is part of her epistemic perspective  > P(h)). Instead, his preferred solution is to combine the probability raising conception with contextualism about evidence according to which, when we raise the question of the evidence for p, p is no longer evidence. Thus, a known proposition is not evidence for itself.evidence for p which entails that p. But, then, in order to avoid scepticism about large swathes of what we ordinarily take ourselves to know, the infallibilist needs to endorse SKSS. By contrast, since the fallibilist allows that one can know that p even if one doesn’t have evidence for p which entails that p, she is not committed to SKSS.In reply, I first want to note that setting aside the relevant probabilistic conception of evidence, fallibilism and infallibilism stand in a different dialectical relationship to SKSS. As I argue in chapter 2, infallibilism is best understood as holding that knowing that p not only requires evidence which entails that p, but also that one has However, even granting that, it may be pointed out that if we do assume the relevant probabilistic conception of evidential support, then a version of the problem of self-support faces any  view of evidence. For instance, restricting one’s evidence to what one knows in some particular kind of way, say what one non-inferentially knows, would still face an analogous problem. I agree that that’s so, and indeed argued at length that restricting what counts as our evidence is no solution to the problem of evidential self-support in an earlier article Reject PR or modify it so that it doesn’t have the consequence that any proposition which is part of one’s evidence is evidence for itself.Greco agrees with the argument of the book that the first option fails. Greco himself favours the second option (2), but I considered and criticised that option in Brown ; SKE ; and SKSS . First, they raise the concern that what they take to be my objection to SKE would apply to any potential fallibilist account of evidence. They take me to object to SKE on the grounds that, even if one knows that p, it’s not felicitous to reply to a request for evidence for p by saying “p”. But that’s to mistake my argument. I use the relevant infelicity data not against SKE, but against SKSS. When one is asked for “evidence for p” one is being asked for evidence which supports that p, not for any part of one’s evidence whatsoever. In itself, an account of evidence, whether fallibilist or not, is silent on when a piece of evidence supports a hypothesis.Turning to SKSS, they suggest that it’s wrong to claim as I do that the infallibilist needs to endorse SKSS to avoid scepticism about important kinds of knowledge, such as knowledge by inference to the best explanation, enumerative induction, and testimony. They suggest that “while data from testimony, inference to the best explanation and enumerative induction may not entail what is known, they may when combined with a sufficiently friendly epistemic environment”. It’s not clear how appeal to a friendly epistemic environment will help. To illustrate, suppose that I acquire knowledge that p from the testimony of a speaker who says that “p”. That she said that p doesn’t entail that p, even with the addition of the claims that speakers are generally reliable, or that she is generally reliable. So, even adding that the testimony takes place in a friendly epistemic environment, the testimony doesn’t entail what’s known. KCS then attempt to defend factivity from the objection that it yields the counterintuitive result that a person and her BIV twin are not equally justified in believing that they have hands. The book criticised one standard infallibilist strategy for dealing with this data, namely to suggest that the BIV twin is merely excused for believing that she has hands and that we confuse excuse and justification. KCS defend this strategy claiming that the infallibilist is able to accommodate the intuitive data. They describe the dialectical situation as follows, “here is what we have so far: (1) the data to be explained – the positive intuition concerning the epistemic status of the BIV’s belief, and (2) the empirical data suggesting that we can’t discriminate between justification-triggered positive intuitions on the one hand and blamelessness-triggered positive intuitions on the other. If so, the data that needs to be accommodated… is that there is either justification or blamelessness present at BIV scenarios. New infallibilism does accommodate this data via the second leg of the disjunction”. KCS don’t offer any support for (2) and it just seems false. We routinely distinguish between morally justified and blameless action. Furthermore, we can distinguish epistemically justified and blameless belief, e.g. consider the case of a scientist who believes a claim which is supported by her evidence and a subject who believes a claim on the basis of a tempting but fallacious inference which she is unable to understand to be fallacious. So their description of the dialectic seems incorrect.Having attempted to defend the key commitments of infallibilism, they raise some questions about the second half of the book in which I defend fallibilism against objections. One of those objections is that fallibilism violates closure for knowledge. In response, I argued that there is reason to suppose that closure fails quite independently of the dispute between fallibilism and infallibilism so that closure cannot be used to motivate infallibilism rather than fallibilism. The book focused on one way in which closure fails independently of that dispute, due to the phenomenon of defeat. In reply to such worries, KCS suggests that closure should be reformulated with a no-defeat condition. I agree that one could attempt to preserve closure in this way, but I suggest that it’s unmotivated to do so. The main motivation for closure is that we can clearly gain knowledge by valid inference from known premises. But, of course that we can often gain knowledge in this way is compatible with the failure of closure. In addition, there are other reasons to think that closure fails even on assumptions congenial to infallibilism. For instance, as Lasonen-Aarnio  and HawtFinally, KCS raise some worries about my discussion of the defeat solution to the dogmatism puzzle. Before addressing those worries, it’s worth noting that whether or not the defeat response to the dogmatism puzzle works is tangential to the main concerns of the book. The argument of the book does involve defending defeat. But I don’t do so by appealing to the defeat solution to the dogmatism puzzle.The dogmatism puzzle arises from the fact that if a subject knows that p, then she also knows that any future evidence against p is misleading. But, then it’s puzzling why she isn’t simply entitled to disregard future evidence against p. One prominent response to this puzzle employs the notion of defeat: once one receives evidence against p, one’s prior knowledge that p is defeated so that one cannot employ that knowledge to dismiss the evidence against p. However, Lasonen-Aarnio points out that this leaves a residual worry in “no-defeat” cases in which one receives weak evidence that not-p which is not strong enough to defeat one’s knowledge that p. Intuitively, one should still reduce one’s credence but this cannot be explained by saying that one has lost one’s knowledge. A fallibilist may be tempted to say that, in such cases, once one receives evidence that not-p, this reduces the probability of p on one’s evidence so that one should reduce one’s credence but to a level compatible with knowing that p. But, Lasonen-Aarnio  argued tIn the book I offer several explanations of why such sentences may be intuitively unacceptable even if true. One factor that may be at work is that to say that something is misleading standardly conveys that one shouldn’t follow its lead. But, it’s a point of contention between defenders and opponents of the defeat solution whether having evidence that not-p when p, is misleading in the sense that one shouldn’t follow its lead. Defenders think that one should follow the lead of such evidence and no longer believe that p; opponents think one should retain the belief that p which still constitutes knowledge. Thus, I argued that in the context of this debate, use of the word “misleading” should be avoided in favour of the following more neutral description that one obtains evidence that not-p despite p being the case. If that’s right, then it’s not to the point to suggest, as KCS do, that “it is true in virtue of the meaning of “misleading” that if something is misleading, then one shouldn’t follow its lead”. I don’t dispute that there is a meaning of “misleading” in which that’s true. Rather, given that, I dispute whether it is legitimate to use “misleading” to describe the evidence in the dogmatism puzzle.Littlejohn’s comments focus on my discussion of the knowledge view of justification, KVJ (a belief is justified if and only if it’s known). He sketches a number of positive arguments for KVJ, saying that he wishes I’d said more about why I think they fail. In addition, he offers an account of when a non-knowledge belief is excused which he hopes avoids the problems I raise for existing accounts. I consider these points in order.One line of argument for KVJ which Littlejohn mentions combines the idea that knowledge is the norm for belief with the idea that justification is norm-conformity. Indeed, he expresses puzzlement how anybody could accept the knowledge norm of belief and yet not accept KVJ. I explained why I find this style of argument for KVJ unpersuasive in the book (93–94). There, I pointed out that even if an epistemologist accepts the controversial claim that knowledge is the norm of belief, she needn’t accept that the notion of justification in play in traditional epistemological debates is the notion of conformity to the norm of belief. Indeed, that doesn’t seem a charitable interpretation: many contemporary epistemologists accept a factive norm for belief (e.g. truth or knowledge) and at the same time have developed accounts of justification which allow that a false belief can be justified.In reply, Littlejohn could stipulate that by “justification” he means conformity to the norm of belief but then a traditional epistemologist will think that he is simply changing the topic. Alternatively, he could suggest that epistemology doesn’t need the traditional notion of justification in addition to the notions of whether a belief conforms to the knowledge norm and whether it is excused. He suggests that if ethics can get by just with the notions of norm-conformity and excuse, perhaps epistemology can. If a defender of KVJ takes this second argumentative strategy, then she needs to show that there is no significant loss if epistemology jettisons the notion of justification. But, in the book, I argue that there would be a significant loss (94–95). The defender of KVJ has a hard time accommodating some of the important traditional roles of the notion of justification, especially its graded and propositional uses. Furthermore, without appeal to the notion of justification we seem to be left unable to make some of the important distinctions we want to make between non-knowledge beliefs. A traditional epistemologist can treat some of these beliefs as justified  and some as merely excused . Notice that the scientist doesn’t need an excuse for her non-knowledge beliefs, and neither the scientist nor the fallacious thinker are exempt from the norms of belief. So, appeal to the knowledge norm, excuse and exemption doesn’t enable us to make all the distinctions we want as epistemologists.A third potential style of argument for KVJ exploits alleged principles unifying the theoretical and practical. For instance, Littlejohn  defends Let’s now turn to Littlejohn’s comments about excuse. Littlejohn questions why an infallibilist need offer an account of the conditions in which a non-knowledge belief is excused. Since the notion of an excusable norm violation is familiar from discussions in ethics, one might think that the infallibilist can simply exploit whatever is the correct account in ethics for her purposes. But, the infallibilist’s options are in fact much more constrained than this. For some popular claims in ethics about when one is excused for morally wrong action are unavailable to the defender of KVJ. For instance, some ethicists claim that justified false belief, or false belief that meets the norm of belief excuses moral wrongdoing. But the defender of KVJ denies that false beliefs can be justified or meet the norm of belief. Other ethicists claim that blameless false belief can excuse moral wrongdoing. But, this presupposes the notion of interest, what it is for a false belief to be blameless. In the book, I point out the unavailability of these options as well as criticise a range of “epistemic” accounts which attempt to provide a sufficient condition for excuse in terms of some epistemic condition (e.g. one isn’t in a position to know that one is violating the norm or has evidence supporting that one’s conforming to the norm). Thus, it’s no surprise that defenders of KVJ have offered dispositional and modal accounts of excuse of the kind I criticise in the book.Littlejohn acknowledges the problems I raise for such dispositional accounts but attempts to overcome them in his expectation account. Dispositional accounts face two central problems. First, doing what most people disposed to conform to a norm would do isn’t sufficient for excuse. As parents frequently say, “That everybody else would do it doesn’t excuse your doing it!”. Second, I might do what most people disposed to conform to the norm would do in the circumstances, but do it for bad reasons (e.g. to hurt someone). Littlejohn attempts to get round both of these problems in his expectation account according to which an agent should be excused for wrongdoing if (1) she meets our expectations where “we meet expectations if our responses align with the responses of someone who is responsive, judicious, and handles risk well” in responding to right and wrong-making features and (2) “the reasoning behind [those responses] doesn’t show us to be unresponsive, less than judicious, or insensitive to risk”. Although this account is supposed to be an improvement on existing dispositional accounts, it still seems to appeal to a dispositional notion. In particular, given that excuse is needed only when a subject does wrong, in those circumstances she is not in fact responding appropriately to the right-and wrong-making features of the situation. So, condition (1) should be understood as saying that the agent’s response aligns with the response of someone who is disposed to be responsive, judicious, and risk-sensitive in responding to right- and wrong- making features. Thus by itself (1) faces the problems for dispositional accounts discussed above. Adding (2) doesn’t help. On similar grounds to (1), in (2) the notion of being responsive must be understood dispositionally. So, (2) says that the reasoning behind the agent’s response doesn’t show that the agent is not disposed to be responsive, judicious, and risk-sensitive in responding to right and wrong making features. But, compatibly with that, the agent might be blameworthy for wrongdoing. As Sher  puts it,Williamson's comments raise some interesting issues about how we should do philosophy, as well as more specific points about some of my key objections to infallibilism. Before addressing the general methodological issues, let me reply to some of the more specific points he raises.Let’s start by considering the problems I raise for the infallibilist’s commitment to SKSS . It’s a problem for SKSS that when asked for one’s evidence for p, it’s infelicitous to cite p even if one knows that p. In his reply Williamson attempts to defend SKSS by appeal to the probability raising conception of evidential support. Like Greco he suggests that the infelicity data isn’t really a problem for infallibilism per se since the probability raising account of evidential support has the consequence that any proposition which is evidence is evidence for itself. And he suggests that to the extent that an alternative account of evidential support is offered which doesn’t have this consequence, the infallibilist could take advantage of it too. But, the latter suggestion ignores the broader dialectical situation. In particular, in the book I show how, on pain of accepting scepticism, infallibilism is committed to holding that if one knows that p, then p is part of one’s evidence for p. Infallibilism is committed to these claims since, without them, the evidence we have in support of a range of important knowledge claims doesn’t entail what’s known, including testimonial knowledge, knowledge by inference to the best explanation and knowledge by enumerative induction. By contrast, since the fallibilist denies that knowledge requires entailing evidence, she needn’t make these commitments. Thus, infallibilism cannot be satisfactorily defended by appeal to a conception of evidential support which doesn’t support SKSS.So, let’s turn to the suggestion that one can support SKSS by appeal to the probability raising conception of evidential support. The probability raising conception is controversial. Furthermore, just appealing to that conception doesn’t offer any explanation of the infelicity of citing a proposition as evidence for itself. So, an infallibilist who endorses the probably raising conception also needs to explain the infelicity of citing a known proposition as evidence for itself. On the face of it, there are three main potential explanations: pragmatic explanation, an error theoretic explanation, or appeal to contextualism. In the book and elsewhere (e.g. 2015) I argue against Williamson’s  pragmatiIn principle, a different kind of reply might seem available, the kind of move standardly used to defend the formal logical definition of “validity” in the face of worries that it has the result that certain arguments get counted as valid even if they would not ordinarily be judged so (e.g. the argument: p therefore p). The logician may simply be interested in a technical sense of “valid” that is not coextensive with its ordinary use. It’s no objection to this technical use that it doesn’t accord with the way we ordinarily speak. Williamson suggests in his reply that a similar “ambiguity” response may be used to defend the probability raising conception of evidential support. While he doesn’t commit to this option, he says “we do not need to decide between the ambiguity and error approaches here, since they serve present purposes equally well”.I agree that it’s open to a formal epistemologist to say that she is interested in notions of “evidence” and “evidential support” which are technically defined and not coextensive with the ordinary senses of these expressions. And I have no complaint against this as such. But, to the extent that this is the way the probability raising conception of evidential support is defended, it’s not clear how it connects with the epistemological controversies concerning infallibilism to which the book is a contribution. As the book explains, the choice between fallibilism and infallibilism lies at the heart of important debates concerning the nature and extent of knowledge and evidence, the connection between knowledge, assertion and practical reasoning, and our ability to extend our knowledge through argument. Furthermore, many of the key arguments draw on our ordinary ways of thinking and talking about knowledge and evidence. In particular, infallibilists have challenged fallibilism by appeal to the infelicity of concessive knowledge attributions, the infelicity of claiming to know lottery propositions, as well as the ordinary ways in which we criticise and defend assertions and practical reasoning. To the extent that the infallibilist says that she is interested in technical notions of evidence, knowledge, and evidential support, it’s not clear to what extent she is engaging with much of the contemporary philosophical literature around infallibilism. Furthermore, it’s not clear to what extent she is entitled to appeal to these ordinary ways of speaking and talking in defending her own position.Now let’s turn to the broad methodological points which Williamson raises. Williamson endorses an abductive methodology according to which we should compare fully fleshed out rival theories with respect to their abductive virtues: fit with evidence, simplicity, elegance, informativeness and explanatory power. He worries that, by contrast, my book instead involves comparing “one worked-out theory, infallibilism… with a much sketchier version of fallibilism”. But, he suggests that this makes my life too easy by appeal to the Kuhnian point that any scientific paradigm at any time has unresolved anomalies. He suggests that it’s illegitimate to press these worries before one has an alternative fully-fleshed out rival theory. I take it that, in a variety of ways, the book does serve to flesh out fallibilism, even though I fully accept that further work needs to be done. But even setting aside the issue of whether the fallibilism I defend is fully-fleshed out, it seems to me, contra Williamson, that one can usefully make progress by pointing out defects in an existing theory, even before having a fully-fleshed out rival theory. Even if one doesn’t have a fully-fleshed out rival theory to hand, that doesn’t make problems for an existing theory any less problematic. Pointing out these defects is often a useful way of either refining and developing the existing theory or developing a rival theory. If evaluation of one theory always needed to wait until we had a fully fleshed-out rival theory, this would be a hindrance rather than a help to investigation, whether philosophical or scientific.


This is a protocol for developing a guideline to establish the evidence ecosystem of acupuncture. It describes all steps that will be followed in line with the World Health Organization Handbook for Guideline Development and the Reporting Items for practice Guidelines in Healthcare (RIGHT). The key steps included guideline protocol development, guideline registration, systematic review of acupuncture evidence issues, systematic review of methods for establishing evidence ecosystem, survey of acupuncture stakeholders regarding potential acupuncture evidence issues, development of potential items for guidelines, Delphi method for guideline item development, consensus meeting, drafting guideline, peer review, approval, and publishing. This future guideline will help to establish evidence ecosystem of acupuncture, which will facilitate the application of acupuncture in clinical practice. The United Nations Educational, Scientific, and Cultural Organization (UNESCO) had placed acupuncture on the Representative List of the Intangible Cultural Heritage of Humanity  and thisAcupuncture clinical practice should be based on high-quality evidence, which could help in decision-making. Thus, acupuncture research should provide sufficient data to enable funders, reviewers, and steering committees to appraise the scientific and methodological rigor of the studies, and for the researchers to replicate and implement these studies . HoweverThe evidence ecosystem , 12–14 wThe guideline will be evidence-based. It will be developed according to the World Health Organization Handbook for Guideline Development and reported according to the Reporting Items for practice Guidelines in Healthcare (RIGHT). It has been registered on the Global Practice Guidelines Registry Platform (registration number: IPGRP-2021CN106).The guideline project was launched by West China Hospital of Sichuan University in April 2021. Chinese Evidence-based Medicine Center or Cochrane China would provide technical support. The guideline working groups will be established in mid-2021 and will comprise four groups: a guideline steering committee, a guideline formulation group, a guideline secretary group, and an external review group.The steering committee group will comprise a clinical chairman, a methodology chairman, an executive chairman, and acupuncture experts. Their responsibilities will be as follows: determine the scope according to PICO strategy; establish a guideline secretary group, evidence evaluation group, and consensus expert group; manage the statement of conflict of interest; supervise the process of guideline formulation; approve the recommendations; and monitor and evaluate the revised guideline.Experts of evidence-based medicine, acupuncturists, editors, policymakers, and computer scientists will be invited as the guideline formulation group. Their responsibilities will be as follows: perform a rapid and high-quality retrieval of the relevant evidence, evaluation, synthesis, and quality classification; generate a summary table and draft recommendations; draft, modify, and improve the guideline; and promote and disseminate the guideline.Students majoring in evidence-based medicine and acupuncture will be selected as the guideline secretary group. Their responsibilities will be as follows: coordinate and organize the work during guideline formulation; investigate the essential clinical problems and outcomes; organize three Delphi expert consensus conferences (online or face-to-face); draft, modify, and improve the guideline; and conduct data analysis and evidence synthesis.The external review group will consist of acupuncture stakeholders, including evidence-based medicine experts, acupuncture experts, TCM experts, editors, policymakers, computer scientists, and patients. Their responsibilities will be as follows: evaluate guideline scope and the issues covered and review the draft guideline.We will use a systematic approach to design the whole evidence ecosystem of acupuncture, including establishing clear descriptions of ecosystem theory, processing evaluations in acupuncture clinical trials, and identifying of key process questions in ecosystem. In addition, each process in the ecosystem will be different, and the guidance facilitates planning and conducting a process ecosystem.All members involved in the guideline development will declare their interests and relevant relationships, and also complete a statement of interest form. If conflicts of interest exist, the expert will be excluded from participation in evidence evaluation and recommendation.It is important to develop a high-quality and useful acupuncture evidence ecosystem. We will define the problems in guideline construction using the Population, Intervention, Comparator, Outcome strategy (PICO).We will search RCTs, systematic reviews, meta-analyses, overviews of systematic reviews and clinical practice guidelines, clinical decision analyses, health technology evaluations, and health policy research methods using the following electronic databases: PubMed, EMbase, CBM, CNKI, VIP, and WanFang Data. We will search the gray literature using the following websites to collect potential guidelines: the National Guideline Clearinghouse , Canadian Medical Association InfoBase, Guidelines International Network, PEDro, Trip Database, American College of Physicians Clinical Recommendations, Australian Government, National Health and Medical Research Council, Health Services/Technology Assessment Texts, Institute for Clinical Systems Improvement, National Institute for Health and Clinical Excellence (NICE) Guidance, NICE Pathways, New Zealand Guidelines Group, Scottish Intercollegiate Guidelines Network (SIGN), and the WHO guidelines approved by the Guidelines Review Committee. Search terms are shown in The following inclusion criteria will be used: RCTs, systematic reviews, meta-analyses, overviews of systematic reviews and clinical practice guidelines, clinical decision analyses, health technology evaluations, and health policy research methods in acupuncture. The following exclusion criteria will be used: non-Chinese and non-English documents; conference abstracts, reviews, and so on; complete unavailable data; old versions of guidelines, consensus, standards, procedures, and specifications formulated by the same institution; organization or management-related guidelines, standards procedures, and specifications formulated and released by non-associated societies and groups; interpretations, translations, plans, reviews, and other articles related to guidelines, consensuses, standards, procedures, and norms; research on compliance with guidelines, consensuses, standards, procedures, and norms; lastly, guidelines, standards, processes, and normative after-effect evaluation research, etc.Two independent reviewers (QW and YH) will screen titles and abstracts to identify potentially eligible documents, which will be retrieved in full text for final review. Duplicate records will be removed. Any disagreements will be discussed and resolved by a third reviewer (YGZ).We will design a data extraction form based on study characteristics and conduct a pre-experiment. The following content will be extracted: author, publication year, published journal, disease, sample size, location, and quality of evidence. If extra information about the study is required, we will contact the author.We will assess the methodological quality of the included studies using several tools . We willA synthesis of qualitative research will be conducted as part of the scope of the guideline and this will also help to define and to refine the questions. A mixed-methods synthesis will combine a meta-analysis of quantitative data and a qualitative analysis. We will summarize the recommendations for the acupuncture evidence ecosystem. The main outcomes will be as follows: author, title and subtitle, type and version, implementation status, target disease, population, user, country or region, literature search method, funding, and patient preferences. The outcome of qualitative research is patient preferences. We will evaluate the effects by using “Preferences about Therapy Questionnaire” on a 5-point Likert scale. We will perform a systematic review of the main principles, basic methods, and procedures for formulating key core technical items of the evidence ecosystem.Once the evidence is retrieved, synthesized, and assessed, the evidence will be used to develop recommendations. The recommendations will be surveyed by 3 rounds of Delphi methods. We will conduct an online-based Delphi process. It is an accepted method in medical research to gain consenting information from a group of experts. The anonymous multi-step survey comprised three rounds, starting with open-ended questions and ending with a consensus-scoring assessment of the guideline. Design and analysis methods will be adopted and modified from a recent guideline for the World Health Organization Handbook for Guideline Development. The online Delphi process is conducted using the survey tool Questionnaire star. We will submit them to the guideline-steering committee for recommendations. Then, the draft of the guideline will be developed. A full draft will recirculate for further review before being revised and approved.The guideline will undergo peer review. The review and response processes will be recorded. The guideline will be revised after discussion with the peer review comments. Peer review opinions will be submitted to the guideline steering committee for further assessment. The next version of the guideline will be developed.Dissemination involves making guidelines accessible, advertising their availability, and distributing them widely. The guidelines will be disseminated through association. The guideline will be submitted to peer review journals, such as the Journal of Evidence-based Medicine or other journals.The guideline is used for establishing the evidence ecosystem of acupuncture. After the approval of the guideline, the evidence ecosystem will be established.This is a protocol for the development of a guideline, which will be used to establish an acupuncture evidence ecosystem. Several challenges of developing the guideline should be considered. First, sustainability is a potential issue in generating the data. Thus, an appropriate strategy should be considered when developing the guideline. Seeking assistance from the central government or industry may be useful in developing the ecosystem. A potentially good example from the Cochrane Collaboration may be helpful. We should move to a long-term, inexpensive, and sustainable structure with a website that is more accessible and useful to end-users. Collaboration with associations or researchers would be useful. In addition, the ecosystem will be used to strengthen clinical practice, which we will provide a guidance for carrying out the process evaluation of acupuncture clinical trials. Therefore, efficient flow between evidence producers, evidence synthesizers, evidence processers and disseminators, and evidence implementers in the evidence ecosystem should be well defined. International scientists should also be invited to contribute to the ecosystem and assist with decision-making. Third, because the quality of evidence ecosystem of acupuncture is still uncertain, the recommendations should be carefully considered , 24. FouThe acupuncture evidence ecosystem model will illustrate how evidence is transferred between different key stages to strengthen health systems and inform care. It will show the importance of “closing the loop” between evidence producers, synthesizers, disseminators, and users. It will also help to process evaluation and to provide potential progress in developing high-quality acupuncture trials. Although the acupuncture evidence ecosystem regarding evidence generation, synthesis, and translation contains thriving areas, others remained unclear. Recognition of these strengths and weaknesses, and the identification of future paths to a more robust system would allow health care and public institutions to focus on the real problems with acupuncture evidence rather than the purported problems highlighted by universal critics. A future priority is to further strengthen initiatives to establish an acupuncture evidence ecosystem, to translate evidence into policy and practice, and to sustain the capacity for continuous technical support for health system policy development and implementation.In summary, this protocol provides a plan for developing a guideline. The future guideline will help to establish evidence ecosystem of acupuncture, which will facilitate the application of acupuncture in clinical practice.YZ and DK: design the study. NL, YH, DZ, XL, JF, and RJ: data collection and analysis. QW and YZ: draft the protocol. YZ and JL: revise the protocol. All authors contributed to the article and approved the submitted version.This protocol was supported by the National Natural Science Foundation of China (Grant No. 82004213) and the Project of Sichuan Provincial Department of Science and Technology (Grant No. 2021YFH0191).The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.


Guidelines are published by international gastroenterology societies regarding the management of ulcerative colitis (UC) and Crohn’s disease (CD) to help clinicians to provide high-quality patient care. We examined the guidelines for the quality and strength of evidence used to develop the recommendations, methods for grading evidence, differences in disease-specific recommendations, conflicts of interest, and plans for guideline updates.A systematic search was performed on PubMed using “ulcerative colitis,” “Crohn’s disease,” and “guidelines” in April 2019. International gastroenterology society websites were searched for UC- and CD-specific guidelines. Guidelines from 12 societies were examined by two authors. Chi-squared tests were used for comparing evidence-level grades, strength of recommendations, and reported conflicts of interest. Linear-regression modeling was used to evaluate the relationship between the number of authors and the number of recommendations in a given guideline.Of 28 guidelines reviewed, 25 (89%) used a total of three different systems to grade the level of evidence and 2 (7%) used an unknown system. Three (11%) reviewed guidelines did not provide a conflict-of-interest statement, while three (11%) provided a timeline for guideline updates. Of 1,265 total statements examined, 246 (19%) reported no grade of evidence quality or explicitly stated that the recommendation was based on “expert opinion.” One hundred and thirty-five (22%) UC recommendations were noted to be “weak/conditional” and 95 (16%) did not have a recommendation strength. Two hundred and forty-two (37%) CD recommendations were noted to be “weak/conditional” and 151 (23%) did not have a recommendation strength.The majority of UC and CD guidelines are based on a low/very low quality of evidence and are further weakened due to the lack of homogeneity in specific aspects of management recommendations as well as conflicts of interest. Clinical practice guidelines (CPGs) were first formally defined by the Institute of Medicine (IOM) in 1990 . The sysIn the 2011 IOM update, the organization further described standards for developing trustworthy CPGs. These include the following: funding transparency, managing/disclosure of conflicts of interest, having a multidisciplinary development group that includes patient representatives, using a systematic-review process, establishing evidence foundations and rating the strength of recommendations, standardizing the articulation of the recommendation, performing an external review, and having a update schedule for recommendations [The first review of inflammatory bowel disease (IBD) international guidelines was performed in 2013 . It founSince 2013, unfortunately not a great deal has changed. A 2019 study of American Heart Association/American College of Cardiology guidelines again found significant variation in levels of evidence supporting various interventions and that there was very little high-quality evidence . SimilarThere has not been a review of more recent IBD guidelines to assess whether there has been a change in compliance with IOM CPG standards. We performed a systematic review of major international gastroenterology and IBD society guidelines specifically on the topic of Crohn’s disease (CD) and ulcerative colitis (UC) that have been published in the literature and on the websites from these societies. Our primary aim was to assess the overall quality of evidence cited in formulating the recommendations and the strength of these recommendations. As secondary aims, we also sought to determine the methods used to grade the evidence, evaluate for differences in recommendations or grades of evidence, gauge potential COI, delineate plans for guideline updates, and highlight opportunities for improvement.A systematic search was performed on PubMed including Mesh terms for UC and guidelines, as well as CD and guidelines, in April 2019. In addition, major international gastroenterological society websites were also examined for the presence of UC- and CD-specific guidelines. Guidelines that were not present in the systematic PubMed search nor posted on the societies’ guidelines webpage were excluded in this analysis. Pediatric guidelines were excluded in an effort to distinguish pediatric IBD guidelines from adult IBD guidelines. Societies with no available English guidelines were excluded in order to prevent errors in translation. Guidelines were also excluded if they were not specific for CD or UC , as these disease states are unique and CPGs differ with regard to disease management. Finally, in the event of duplicate guidelines with the same title, the earlier guideline was excluded from the analysis.The following societies were ultimately included in the final analysis: American College of Gastroenterology (ACG), American Gastroenterological Association (AGA), Asia Pacific Association of Gastroenterology, National Institute for Health and Care Excellence (NICE), Canadian Association of Gastroenterology (CAG), Crohn’s and Colitis Foundation of America (CCFA), European Crohn’s and Colitis Organisation (ECCO), Gastroenterological Society of Australia, Indian Society of Gastroenterology, Japanese Society of Gastroenterology, Korean Association for the Study of Intestinal Diseases, New Zealand Society of Gastroenterology, and the Spanish Working Group (GETECCU) .In the category of UC guidelines, we identified 97 guidelines published by 31 gastroenterology societies. Nineteen societies were excluded based on the exclusion criteria mentioned above. In the category of CD guidelines, we identified 118 guidelines published by 28 international societies. Twenty societies were excluded based on the exclusion criteria mentioned above. Finally 12 societies’ UC guidelines and 9 societies’ CD guidelines were reviewed. Of note, AGA includes both guidelines and a technical review. The technical review was analysed with regard to authorship and COI only (see below) , 17, 19.Guidelines that met inclusion criteria were further examined to determine whether any grading system was used to assess the level of evidence for the recommendations. The quality of the level of evidence supporting the recommendations was evaluated for each individual guideline. In addition, the strength of recommendations was categorized. The guidelines and websites were also reviewed for any comment regarding planned updates to their current guidelines. The guidelines were examined individually, in aggregate by society, between societies, and in an overall analysis of all published IBD practice guidelines.Multiple systems were utilized to the grade level of evidence by the different societies. These included GRADE , Oxford In an effort to standardize the reporting of level of evidence, the following categorizations were used (based on the GRADE and ABC(D) systems) similar to prior publications , 44:High quality of evidence: Oxford level 1, GRADE high quality, A level, I.Moderate quality of evidence: Oxford level 2, GRADE moderate quality, B level, II-1, II-2, II-3.Low and very low quality of evidence: Oxford levels 3, 4, and 5, GRADE low and very low quality, C and D level, III.In addition, recommendations were further categorized as strong, weak/conditional, and no quality provided/expert opinion. If the guideline contained a clear recommendation as indicated by bullet point (or similar), or a recommendation was separated out from the remainder of the text and there was no accompanying grade or level of evidence noted, then it was assigned to this last category.In an effort to further delineate variation between societies’ guidelines, representative recommendations for both UC and CD were analysed in greater detail. For UC, recommendations regarding initial and surveillance screening colonoscopy, as well as colonoscopy technique (i.e. chromoendoscopy), were chosen as representative topics to be reviewed for consistency and discrepancy across guidelines. For CD, this was done using recommendations on the medical management of moderate/severe CD, fistulizing CD, and post-operative CD management. CD guidelines were also assessed for the presence of material on anti-integrins and anti-interleukins. If there were inconsistencies, the recommendations and level of evidence used to support the recommendations were further analysed.All guidelines/websites were evaluated to determine whether potential COI were disclosed. If COI were present, the guideline was reviewed to determine the total number of authors with COI, as well as whether the primary author and principal investigator (last author) had COI. COI that were determined to be relevant included the following: advisory board, speaker's bureau, consulting, and industry-sponsored continuing medical-education activities. Government and non-profit awards were not considered COI and were excluded from analysis. The COI were assessed by individual society, between societies, and in aggregate.All guidelines were reviewed by two authors (A.G. and R.S.) for the use of a grading system for the quality of evidence behind recommendations, the specific system by which the evidence was graded, the clarity of the document layout, the presence of COI, and evaluating similarities and discrepancies between CD and UC recommendations.p-value of 0.05 was considered significant. Analysis was done using SAS.Chi-squared tests were used for comparing evidence-level grades, strength of recommendations, and COI reported between societal guidelines for all international organizations analysed. Linear-regression modeling was used to evaluate the relationship between the number of authors and the number of recommendations in a given guideline. A A total of 215 guideline documents were reviewed for inclusion in this study. After exclusion criteria were applied, 28 guidelines  were further analysed. Twenty-seven (96%) guidelines graded the quality of evidence behind their recommendations. The only group that did not was the New Zealand Society of Gastroenterology .After excluding AGA technical reviews, 21 of the original 28 guidelines were able to be merged into the grading system for evidence quality used in this study. The 21 guidelines had a total of 1,265 recommendations. Of these, 246 (19%) reported no grade of evidence quality or explicitly stated that the recommendation was based on “expert opinion.”P < 0.001).For UC, a total of 604 recommendations were analysed for evidence quality. Ninety (15%) recommendations were supported by high-quality evidence, whereas 170 (28%) were supported by moderate-quality evidence and 273 (45%) by low- or very-low-quality evidence. The breakdown by society on grading of the level of evidence by society is shown in P < 0.001).For CD, a total of 661 recommendations were analysed for evidence quality. Ninety-three (14%) recommendations were supported by high-quality evidence, whereas 157 (24%) were supported by moderate-quality evidence and 353 (53%) by low- or very-low-quality evidence. The breakdown by society on grading of the level of evidence by society is shown in As noted above, the methods used to grade evidence in these documents were variable. Seventeen (61%) guidelines used the GRADE system, five (18%) used the Oxford system, and two (7%) used the ABC(D) method of evidence-quality grading. One society (Japanese Society of Gastroenterology) used an unknown, non-standardized method of evidence grading . NICE noP < 0.001).After excluding societies that did not note the strength of their recommendations in the entirety of their guidelines, the remaining subgroups  were further analysed. There was a statistically significant difference in the strength of recommendations  for both UC and CD (One hundred and sixty-nine 28%) UC recommendations were delineated “strong,” while 135 (22%) were denoted “weak/conditional” and 95 (16%) did not have a recommendation strength. For CD, 106 (16%) recommendations were delineated “strong,” while 242 (37%) were denoted “weak/conditional” and 151 (23%) did not have a recommendation strength. Breakdown by individual society for UC and CD can be seen in 8% UC recP < 0.001). The percentage of guideline authors with COI ranged from 0% to 100%, with a mean of 52% (standard deviation = 32.2%). Fourteen (50%) first authors and 18 (64%) senior authors reported COI  , 38–40. rted COI  and 2.Among the 604 UC recommendations, 169 28%) were delineated “strong,” while 135 (22%) were denoted “weak/conditional” and 95 (16%) did not have a recommendation strength. Among the 661 CD recommendations, 106 (16%) were delineated “strong,” while 242 (37%) were denoted “weak/conditional” and 151 (23%) did not have a recommendation strength. See 8% were dFor UC, there was significant guideline variability on the timing of the initial colon-cancer screening for UC patients, ranging from no recommendation to first screening colonoscopy 8–10 years after diagnosis . Four ouFor CD, there was significant recommendation variability in the medical management of moderate/severe CD . Four ou = 2.6 years). Guideline age ranged from 1 year (written in 2019) to 11 years (written in 2009). Three societies included in the initial analysis included timelines for future updates or reviews of their guidelines .The mean age of guidelines amongst all societies is 5 years  has noted that guidelines are meant to advise physicians on treatments for their patients and to create a safer medical system . In ordePoor evidence quality behind CPGs continues to be a challenge. Our 2013 study, as well as this current study, confirmed similar findings , 9–11. Oet al. [COI in both guideline development and clinical medicine continue to be a prominent issue, as they can lead to distrust of guidelines and prevent adequate and unbiased patient care. Highlighting the importance of COI in medicine, JAMA presented a theme issue in May 2017 on the topic . Althouget al.  found thet al. . The preAs noted previously, one of the IOM standards is the creation of a schedule for guideline updates . In our We note three limitations of this study. First, multiple systems were used to grade the quality of evidence by international societies. To allow uniform analysis of the level of evidence supporting the recommendations, we merged the grading systems into one system, based on the GRADE system of evidence quality. To limit potential bias, this was performed by two authors. Second, guidelines included in this analysis were required to be specifically regarding UC or CD. Thus, guidelines under the general heading “inflammatory bowel disease” were left out, which may be a substantial number. Nonetheless, we feel that these results would likely be applicable to those recommendations given that the same societies included in this study also publish these more “non-specific” guidelines. Furthermore, IBD guidelines published by other societies may exist, but were not considered in this analysis. Lastly, when assigning a low/very low level of evidence to recommendations without any supporting evidence, only one level was assigned even if multiple recommendations were included in one sentence. This potentially underestimates the total number of recommendations with ungraded evidence.In summary, the majority of IBD CPG recommendations are based on low- and very-low-quality evidence, as per our GRADE-based system. This has unfortunately not changed significantly compared to 2013. Reassuringly, COI are reported much more frequently compared to 2013, though this is not a universal practice. Additionally, management recommendations vary between societies for both UC and CD. Few societies report a timeline for review of their guidelines and updates. This study continues to highlight the need for improving the development of IBD CPGs. Recommendations would be improved by stronger supporting evidence, agreement between societies, up-to-date recommendations, and transparency regarding all potential COI in the development process.Study concept and design: J.D.F. Acquisition of data: A.G., R.S., and G.H. Analysis and interpretation of data: A.G., R.S., G.H., and J.D.F. Drafting of the manuscript: A.G., R.S., G.H., and J.D.F. Critical revision of the manuscript for important intellectual content: A.G., R.S., G.H., and J.D.F. Statistical analysis: A.G. Study supervision: J.D.F. Each author has approved the final draft of this manuscript.No financial support was provided for this manuscript.None declared.


The Evidence-Based Policymaking (EBP) process in Nepal is rife with poor practices where often policymakers are portrayed as perpetrators for such practices. However, we need to think of the EBP as a two-sided coin where both research pull and research push play equally significant roles. This study aimed to assess the perception of Nepalese health policymakers and researchers on EBP and identify appropriate mechanisms to integrate evidence into policies.Following the constructivist philosophical paradigm, qualitative research design was used in the study with the grounded theory approach. Purposive sampling was performed, and the 12 in-depth interviews were conducted, where number of interviews was finalized using saturation theory. All interviews were audio-recorded, transcribed, translated to English, coded line by line, and then developed into themes. Thematic analysis technique was used to manually analyze the data.Study participants highlighted that evidence is being utilized during policy formulation but not in the amount it should be, with a preference for anecdotal evidence further reducing the chance. Apart from these barriers, poor credibility of information obtained, poorly targeted dissemination, inadequate policy-based researches, and policymakers and researchers operating within the spheres of their own with a feeble link to channel the flow of information between them were identified by participants. On the other hand, the publication of one-pager research brief, conduction of nationally representative surveys especially quantitative studies, the practice of cost-effectiveness study, and policymaker’s involvement during the research were some facilitators identified.Moreover, the study accentuates that better communication strategies such as the establishment of formal forums with policymakers and researchers, better-targeted dissemination, and identification of priority areas have wide potential to promote a unified front of health policymakers and researchers for EBP. The evolution of utilization of evidence in the policymaking process for any country usually follows a spectrum starting from evidence-free policy (EFP) to policy-based evidence (PBE) and ultimately to evidence-based policymaking (EBP) . EBP is The need for EBP is well established in the global policy environment and the context of Nepal is no different . Even thThus, the study assesses perspectives and attitudes of Nepalese health policymakers and researchers together, regarding the use and impact of evidence in policymaking. Additionally, not only does the research help to identify key barriers and facilitators of EBP but also points out the appropriate mechanisms to integrate research evidence into policymaking.This study took a participatory constructivist philosophical paradigm to seek the understanding and actions of study participants. Following this paradigm, the qualitative research design was used to explore the perception and attitudes of participants on EBP. Further, aspects of grounded theory within qualitative design helped in the systematic generation of theory from a diverse array of information obtained from interviews.The participants of this study were Nepalese health policymakers and researchers. Health policymakers were defined as government employees in the capacity of a section or division chief (9th level officer and above), working currently or in the past, at the Ministry of Health and Population (MoHP) and its departments. This also included health policy advisors from organizations other than MoHP that advocate for policy need and those who prepare briefing notes and policy documents that in turn shape policy determination. Health researchers were defined as individuals working as institutional and/or independent researchers conducting researches related to health and health system.Purposive sampling of both health researchers and policymakers was done to identify the study population. The possible health researchers were identified by visiting health research institutes operating in Kathmandu while possible health policymakers were identified by visiting MoHP and Department of Health Services (DoHS). The possible participants were first approached with a proposal brief of the study to clarify study objectives and get further information on their past policy involvement. Afterwards, principal investigator prepared a prioritized list of participants based on their past policy involvement, free schedule, and willingness to participate in the study. All participants were approached, following the prepared prioritized list, accepted to participate in the research interview. The total number of participants was finalized following the saturation theory which included 5 health policymakers and 7 health researchers.Before all interviews, consent was obtained from participants following a verbal explanation and upon reading and signing an informed consent form. Face-to-face, semi-structured, in-depth interview was conducted with all 12 key informants at their workplace in a separate room with the presence of only researcher and participant. Two interview guides were developed, each for health policymakers and researchers, based on similar studies conducted in Argentina , 7 and wAll the authors were involved during the data analysis and coding process. Interviews were then transcribed in the Nepali language except for one interview which was conducted in English. The interview in English was conducted with participant R06, where the participant preferred English as a form of communication during the interview. The Nepali transcripts were translated into English before further processing the data. The Three-step coding approach by Strauss and Corbin was used during the coding process i.e. open coding, axial coding, and selective coding. Before iFigure As in seen in Fig. Under this theme, we discuss the perception of participants on EBP in terms setting national policy agenda, evidence considered during the policy making process and the term policy itself. During the interviews with participants (participant’s profile is provided in Annex 1), it was revealed that three factors played major roles in setting national policy agendas i.e., government and its entity, political influence, and external developmental partners (EDPs). In Nepal, usually the concerned ministry or its department who realizes the need for a certain policy in their line of work, developed policies using the template provided by National Planning Commission (NPC) . When vi“Entity of ministry that works for the issue and takes the responsibility for implementation of policy and its execution will decide and bring out the policy for example if it is related to maternal health, it will be family health division.” P01.Similarly, most of the health policymakers highlighted that past policy reviews are often used to set the agenda. They identified evidence such as prevalence studies, data from routine information systems, and various surveys along with the international commitment helped to guide national agendas and decision making.When looking at the perception of health researchers on policy, most of them perceived it as a form of commitment from the government and reflected through their actions. In addition, researchers perceived that the policies have a role in every step of research, and few even considered policies increased the scopes for research opportunities.“Policy I think it defines research. For example, there’s a policy addressing child health but some of its components are not being implemented then, research helps to find out the reasons behind it and ways forward. So looking policy from the research aspect, it provides research opportunities and increases its scope.” R02.Under this theme, we discuss the different barriers that are being created through the sides of health policymakers in evidence utilization during policymaking process. As mentioned earlier, political leaders are major influencers of the policy formulation process and ultimately, on how the evidence was to be utilized in the due process. But unfortunately, in recent years, there had been a proliferation of policies with support from policymakers, not as a measure to solve health problems of the country rather, as a means to strengthen their resume and boast about the number of policies developed during their leadership. As a result, these policies did not capture the real need of the population. Moreover, participants highlighted that the relationship between bureaucrats and the vision of the leading political party also influenced the evidence utilization for decision making.“In these recent years, there is a race among ministers to implement the maximum number of policies to show it as their success, rather than developing policies based on actual needs. The policy developed in such a way has no implementation and costing plans. We term this trend as “policy ko kheti” (developing policies as sowing the field).” P01.The impact of that political influence is best described by the common statement used by the participants “Evidence are being utilized but not in the amount it should be”. On top of it, the poor technical competency of policymakers to understand research findings led to a poor culture of scientific evidence utilization and increased preference towards anecdotal evidence.“It’s about our practice of preference towards anecdotal evidence, for example, if a certain policymaker had visited Rukum once and had seen a certain situation and the factors behind it, they tend to generalize the context into national level but in reality, it may or may not be true because in another district some other factors might be creating that very situation.” R02.These scenarios further demotivated researchers to conduct research on priority areas and due to the lack of such researches, anecdotal evidence were preferred again. Furthermore, donor agencies pushed their agendas to the government through partnerships and implemented novel projects prior to conducting impact studies in the local context leading to less promotion for evidence generation.“People get motivated to generate quality data if the generated data is used and vice versa. So, the use of data is one of the barriers to new research. If there is a good practice of utilizing data in decision making, definitely that will drive more research and surveys. Thus, we need to analyze the data availability; where the data gap is, and how to fill the gap.” R06.Moreover in Nepal, research is considered a luxurious commodity so, the level of priority placed on it, allocated resources and future growth plans are poor.“When talking about evidence, we need to look at our system, how it has been running since past, for example, the plans are prepared at the central level and budget is decided before plans are finalized and as research is not a priority, it gets a very low budget.” R01.Here, we investigate the barriers that are being created by health researchers that hindering the process of evidence utilization in policymaking process. Most policymakers and even few researchers felt that small-scaled researches are not preferred for evidence utilization as these raise some serious issues of generalizability. Linked to this issue, is the question on the credibility of the evidence and the researchers producing it, for instance, researcher’s skills, vague recommendations, and most importantly researcher’s tendency to highlight results that prove their research objectives while hiding the ones that do not. This leads to the development of mistrust among policymakers towards researchers, making it even more difficult to convince them. As a result, policymakers preferred large-scale surveys conducted by the institutions working closely with them. It is a good practice by policymakers as they trusted quality evidence but this also means that they overlooked other quality findings obtained from researches, both large and small-scale, conducted by organizations or individuals unfamiliar to them.“Sometimes quality and trust of evidence also play a factor. For example, Global Burden Disease is calculated by both WHO and IHME but Nepalese policymakers prefer WHO’s data than IHME’s data even though these two organizations use the same methodology because WHO has been working as a partner of government for a long time and policymakers have developed trust towards them.” R03.In addition, the poor culture of targeted dissemination by researchers was another barrier to evidence utilization. Every day numerous researches are being conducted at the either organizational or individual level but their potentials are limited in the form of reports collecting dust on shelves, thus leaving policymakers oblivious of the evidence pool.“Dissemination happens and researchers try to disseminate in different forums but targeted dissemination is lacking. Also, researchers disseminate findings using technical terms that are difficult for policymakers to understand.” R07.“Two Culture Hypothesis”. The two culture hypothesis is the concept where researchers and policymakers operate in a different environment with a frail link connecting them i.e. researches being conducted based on the researcher’s self-interest, without consideration of the nation’s priority, and on other hand, to make the condition worse, policymakers formulating national policies based on political influence and anecdotal evidence. This practice further weakened the already frail link between policymakers and researchers leading to the formation of evidence-free policy.In theme three, we look at how all these above-mentioned barriers resulted in two different spheres of health researchers and policymakers or as termed by a participant On a brighter side, in this theme we discuss few facilitators practiced at policymakers’ level to promote evidence-based policymaking. The international commitment was identified by most participants to be a major factor to set the national policy agenda and identify policy areas. Being signatories of these commitments, the government implemented its strategies which may include but are not limited to an increased budget for research, the compulsion to use evidence during policy formulation, and such.“Well, these international commitments give us target such as where to reach in-terms of certain issues such as MMR. Also, these provide us some guidelines, for example, due to SDG commitment a minimum of 2 % of the health budget is allocated for research and due to partnership with WHO, the Nepal government needs to gradually increase the budget in the health sector to 10 %.” P05.Similarly, few Nepalese researchers and policymakers identified experts/ stakeholder’s consultation as one of the factors that promoted evidence utilization.Similarly, in this theme, we discuss the different facilitators practiced by health researchers to promote EBP. Most policymakers, as well as researchers, identified that the researches that are done based on the government’s priority and with the involvement of policymakers increased the chance for the evidence to be utilized in the policymaking process.“Lawrence Green once said for evidence-based practice we need practice-based evidence thus, for the evidence-based policy we first need a practice of policy-based evidence. We think that if we present research evidence, it should guide policy but in reality, it does not occur. It is difficult to go that route from the beginning so first, we need to practice research to be guided by policy and policy priority identified by government and gradually move towards guiding policy by research evidence.” R03.These priority areas are identified by holding close discussions between researchers and policymakers. Furthermore, this nature of collaboration increased ownership among policymakers.“Those researches conducted under guidance from the ministry are easy to translate into policy but those conducted at an individual or organizational level are hard to translate and is time consuming mainly due to the issues of trusting the finding from the research.” R01.These sorts of collaboration are observed in nationally representative surveys such as Nepal Demographic and Health Survey (NDHS), STEPs, and Multiple Cluster Indicator Surveys (MICS) and thus, policymakers are aware of it and most likely utilized it during policy development.In the previous section, lack of targeted dissemination was identified as a barrier, thus an appropriate communication approach such as one-pager policy brief in the Nepali language solved the issue. These briefs provided a short description of the findings and specific recommendations with minimal use of technical terms. This initiative helped policymakers understand the evidence and utilized them during policy development.“One-pager research brief will benefit as well because no policymaker has time to read 40-50 pages report. So one-pager with key findings and recommendations will help them understand the situation and will be easier to discuss as well.” R02.Moreover, there had been few instances in Nepal where findings from cost-effectiveness studies for certain programs are utilized. These effectiveness studies gave insights on the feasibility of the programs and implementation module.“While developing nursing and midwifery policy, we decided to have a nurse in each ward and thus performed cost analysis where we found that cost for producing a single ANM is about 5-6 lakhs and we have about 300 wards so the cost came huge and on top of that we also need to provide the salary. Another example is of placing school health nurse, there are 36,000 schools in Nepal, and the basic salary of staff nurse is 23,500 so the cost for the single year came to about 9 Arba, which we cannot bear every year thus, in these cases we decided to go phase-wise based on data from costing.” P04.Apart from the health researcher’s and policymaker’s practices to promote EBP, in this theme we discuss the different strategies that are in place within the national health care system of Nepal that promotes EBP. First one is the desk review as a compulsory step during policy development, which helped to reduce the gap between evidence not being utilized and their utilization.“The trend of desk reviews is prevalent in the health sector. For example, if the policy is going to be developed on certain themes, then available literature is collected through desk reviews. We do not have a specific guideline for a literature review so we look for published and unpublished researches in Nepalese context that are available to us.” P01.Many policymakers identified policy reviews to be one of the factors used to determine the country’s priority goals. These policy reviews, though often irregular in practice, gave priority areas and there had been instances of researches, large and small-scale, being conducted based on these priorities.“While developing health policies, it’s not like that we don’t have any policies till date, we do have some which are reviewed may be in 5 years, 7 years or 10 years and based on review finding we either update the existing policy or develop new policy. Similarly, we come across new policy issues during the yearly review as well.” P03.“knowledge broker”, a term used by a participant, was rising. A knowledge broker may be an institution or an employee of a certain institution whose job is to make current evidence available at the policymaker’s table. Because of these knowledge brokers, the policymakers easily got their hands on quality and updated information to make evidence-based decisions.At the time of data collection, the concept of evidence facilitators or The practice of regular updates and uptake of the computerized routine information system in Nepal, recently District Health Information System 2 (DHIS 2), helped the very ground level health facility to get quality information in real-time. This ease in accessibility of quality information enabled local government to plan programs accordingly. Moreover, the implementation of a new federal structure provided an opportunity for local government to exercise more power to plan programs based on their context and quality information provided by the routine information system.Along with identifying the barriers, facilitators and strategies in place to promote EBP in Nepal, the study also focused on different approaches that can be taken by both health policymakers and researchers, identified from themselves, for its better practice. These suggested actions are presented in this theme in Table Studies looking at EBP have identified a variety of strategies used by policymakers to set national policy agendas. In this study, the government and its entity, political influence, and EDPs were identified as major influential factors for setting such agendas. A similar study conducted in Nepal  identifiThrough this study, we found that though sometime evidence are utilized, political agendas triumph over it with the only point of consolation being the utilization of evidence, to some extent, in support of that decision. However, in recent years we have seen a positive move towards EBP where policymakers have been giving priority for some national-level researches and their utilization. Unfortunately in Nepal, this trend tends to be limited to the inclusion of evidence as a statement in the background of the policies rather than being utilized to guide the strategies and activities of the policies.Barriers that were identified by Nepalese policymakers and researchers which also appeared in other published studies include political influence during decision making, resource constraints, a poor culture of evidence utilization, issues of generalizability and credibility of research, quality and trust of evidence being available to health policymakers, poorly targeted dissemination, low availability of priority-based research, use of jargons and technical competencies of policymakers , 18–20. Due to the lack of targeted dissemination, the evidence is not reaching the tables of the policymakers and moreover, the less use of evidence demotivates researchers to conduct more studies. As a researcher, we are eager to blame policymakers for not utilizing evidence but we are also not fulfilling our responsibility of disseminating findings. The scale and quality of research primarily focus on generalizability, credibility, trust, validity, and reliability of the research. As most of the small scale researches are done at the local level with few samples, it raises the question on the quality of information generated. Thus, policymakers who make national-level policies do not prefer them. Moreover, it is extremely difficult to convince policymakers to trust research findings generated by sources unfamiliar to them. This might either be because they were not involved in research or due to their poor knowledge regarding research terms such as p values. As a result, policymakers and researchers operate on two different spheres and fail to not perceive each other’s views on the research. For example, we have little to no priority-based evidence, mostly because researches are conducted based on the researcher’s self-interest and also due to lack of promotion of priority areas by policymakers. Policymakers perceive research to be a luxurious commodity and researchers perceive policymakers to be unknown about research concepts thus creating a gap for evidence utilization. Further increasing the already existing gap is the resource constraint where the budget allocated for research is very low. The combination of all these factors has led to the formulation of policies that utilize little to no scientific evidence.Similar to barriers, facilitators to EBP identified by Nepalese policymakers and researchers that also appeared in other published studies include stakeholders and/or expert consultation, conduction of research on nationally identified priority areas, the culture of targeted dissemination, the practice of cost-effectiveness study, and involvement of policymakers during the research , 19–21. As mentioned before, not every policy is based on anecdotal evidence; scientific evidence is also utilized mostly due to the mandatory desk reviews. But the utilization of current and relevant evidence during policy formulation is still a question. As stated by researchers, factors such as large scale nationally representative surveys and the involvement of policymakers during the research help for evidence to get utilized. Similarly, policymakers stated exchange efforts such as targeted dissemination and publication of a one-pager research brief can help policymakers be aware of the findings. Moreover, the development of mass communication technologies especially the increasing use of social media and promotion of findings through it helps to reach policymakers easily. Similarly, the promotion of evidence facilitators in recent times has helped to reduce the gap between research and policy. A new computerized routine information system provides a new opportunity for research evidence to be utilized more at the local level in the new federal structure.Comparing the recommendation provided by Nepalese policymakers and researchers for better EBP with the recommendation mentioned in different published journals, communication is identified as a major tool , 9, 21. The study was based on the sample of researchers and policymakers who were interested in participating and willing to speak with us. Thus, the sample is limited by the non-random selection of participants. Similarly, the study was an academic research project for obtaining an undergraduate degree, thus allocated time of one month for data collection was a constraint and academic researchers were not included as research participants. In addition, due to the time constraint, during the data collection repeat interviews could not be conducted, the transcripts were not returned to participant for comment as well as they were not consulted to provide feedback on the findings.This research is unique in its focus on both the demand and the supply side of evidence utilization conducted in the context of a lower middle-income country like Nepal. It explores how both sides perceive each other and identifies key barriers and facilitators to EBP along with potential measures to lessen the gap that exists between them. It was identified that though the research evidence and policy analysis documents were considered in the policymaking process, political influence played a major role in setting national policy agendas. The research sheds light on importance of research development along with its utilization culture to minimize the gap that exists between policymakers and researchers for implementation of policies. For this, better communication strategies can act as a major potential measure for a comprehensive and united vision of policymakers and researchers to promote EBP. These strategies include the establishment of a formal forum of policymakers and researchers, better-targeted dissemination of findings, and identification of the nation’s priority areas for research. Apart from these strategies, promotion of practice for impact study prior to policy implementation and promotion of systematic reviews and meta-analysis could further help for better EBP.


Clinical practice guidelines (CPGs) are representative methods for promoting healthcare standardization and improving its quality. Previous studies on the CPG (published by 2006) development process in Japan reported that the involvement of experts and patients, efficient evidence collection and appraisal, and paucity of evidence on Japanese patients should be improved for the efficient CPG development. This study aimed to clarify the trends of CPG development process in Japan, focusing on the involvement of experts and patients, efficient evidence collection and appraisal, and paucity of Japanese evidence.A cross-sectional questionnaire survey was conducted for CPG development groups to collect information on the development activities of the CPGs published from 2012 to 2019. These CPGs were identified from the Japanese guideline clearinghouse. The questionnaire included the questions on composing the group, securing funding sources, collecting and appraising the research evidence, and the difficulties in the CPG development process. The questionnaires were distributed to the chairpersons of the CPG development groups through postal mail from November 2020 to January 2021. Combining the data from the current survey with those of previous studies reporting the development process of CPGs published by 2011, we analyzed the trend in the CPG development process.Of the total 265 CPGs included in the analysis, 164 (response rate: 41.4%) were from the current survey and 101 (response rate: 44.5%) were from previous studies. Among these, 40 (15.1%) were published by 2005, 47 (17.7%) in 2006–2010, 77 (29.1%) in 2011–2015, and 101 (38.1%) in 2016–2019. The proportion of CPGs involving methodologists did not increase through the publication periods. The proportion of CPGs involving patients almost doubled from the first period (15.9%) to the fourth period (32.4%). The yield rates of the articles did not change through the publication periods. The difficulty in “Coping with the paucity of Japanese evidence” has been improving consistently (69.2% in the first period to 37.4% in the fourth period).Our results suggest the need for methodological improvement in the efficient collection and appraisal of evidence and in the system assigning experts to the CPG development groups.The online version contains supplementary material available at 10.1186/s12913-022-07492-7. Clinical practice guidelines (CPGs) are statements that include recommendations based on a systematic review of evidence and an assessment of the benefits and harms of alternative care options in order to assist the decision making of practitioners and patients , 2. CPGsNumerous development manuals and more than 40 appraisal tools have been published to ensure the quality of CPGs , 4. The Based on the questionnaire surveys conducted on the development groups of the CPGs published by 2006 –16, HaseThis study aimed to clarify the trends of CPGs development process in Japan, focusing on the involvement of experts and patients, efficient evidence collection and appraisal in systematic review, and paucity of evidence on Japanese patients.A cross-sectional questionnaire survey conducted for Japanese CPG development groups was used to collect information on the development activities of CPGs published from 2012 to 2019.To identify Japanese CPGs, we used a Japanese guideline clearinghouse managed by the Toho University Medical Media Center , which cThe survey was conducted through postal mail, targeting chairpersons of the CPGs identified from the Japanese guideline clearinghouse, from November 2020 to January 2021. The bibliographic information about the CPG for which we requested to answer were enclosed with the questionnaire to minimize the potential discrepancies between CPGs participants answered and those we intended. We also sent reminders in December 2020 to the unresponsive chairpersons. We retrieved information on the chairpersons from the descriptions in the CPGs. To ensure continuity in the analysis, the questionnaire was similar to that of the previous studies regarding the questions on a) the processes for composing the development group , and collecting and appraising the research evidence, and b) the difficulties in the CPG development processes –16, 19. The Ethics Committee of Toho University School of Medicine stated that this study was not applicable for ethical review under the Japanese regulations (No. A20064). All participants were informed about the objective of the research and the policy for keeping their data confidential and anonymous in the survey. There is no published protocol of this study. For the reporting, we followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guideline for observational studies .Information about the publication year, development group, and versions of the CPGs were collected from the descriptions in the CPGs. The publication years of the CPG were divided into four periods: by 2005, from 2006 to 2010, from 2011 to 2015, and from 2016 to 2019. Development groups were grouped into three categories: research group, research group plus academic society, and academic society. “Research groups” were temporary groups for conducting research on specific themes. In Japan, a CPG development handbook explaining the methodology related to MINDS, a business arm associated with the CPGs of the Japan Council for Quality Care, has served as the basis for the development of CPGs for a considerable time , 25. FurP-values < 0.05 were considered statistically significant.We used data of the CPGs published in 2012–2019 from the current survey and those published by 2011 from previous studies –16, 19. Figure The characteristics of the 265 included CPGs are listed in Table Table As shown in Table Table This survey involving CPG development groups in Japan revealed that among the problems noted in CPG development in a previous study , there hPatient involvement has increased across many fields in Japan, such as clinical practice, health policy making, and clinical research –28. WithBasing on the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach , the manThe CPG development methodologies, such as the GRADE approach that was developed by some of the original founders of the evidence-based medicine (EBM) movement, have improved over the last decade or more. Introducing the GRADE approach, the Japanese CPG development manual has emphasized on the comprehensiveness, transparency, and unbiasedness rather than the efficiency of systematic reviews in the CPG development process . HoweverWith regard to expert involvement, some organizations offer support to the CPG development groups. In Japan, The MINDS Guideline Library assigns experts of CPG development to these groups , and sevIn addition to the problems noted in the previous study , “composThis is the first study to reveal the long-term trends in CPG development activities through a cross-sectional questionnaire survey. However, it has several limitations. The response rate was not high, and there was a time lag between the CPG development and the current survey, as well as recall, selection, and unresponsive bias. The results might be biased towards the responses of the CPG development groups which have been involved in the CPG development and dissemination actively. Therefore, the results should be not interpreted as the trends for the general CPG development organizations.Although there has been improvement in the patient involvement and coping with the paucity of Japanese evidence in the CPG development process in Japan, expert involvement and efficiency of evidence collection and appraisal have not. Our results revealed the need for methodological improvement in the efficient collection and appraisal of evidence and in the supporting system assigning experts to the CPG development groups.Additional file 1.