Evidence-based decision on drug list or formulary has been applied worldwide. Although the importance of scientific evidence was emphasized, the decision-making procedures for including medicines into the national reimbursement drug list were often challenged by their process opacity and relying on subjective expert opinion. This study aimed to explore and assess the evidence for the effectiveness of anti-hypertensive medicines included on the Chinese National Reimbursement Drug List (NRDL), and to provide recommendations for further improvement.Three international evidence-based guidelines were selected to serve as reference criteria. The antihypertensive medicines included in NRDL of Urban Employee Basic Medical Insurance (UEBMI) were compared with recommended drugs in three international guidelines. Medicines recommended by at least two guidelines were considered to have sound evidence support for the effectiveness. Otherwise, published literature with high evidence grade, namely systematic review, meta-analysis and randomized controlled trial (RCT), were searched for further assessment. Medicines reported as fairly good effectiveness by literature with high evidence grade can be also considered having sound evidence for the effectiveness. Methodological quality of systematic review or meta-analysis was evaluated by AMSTAR scale and PRISMA statement. Literature quality of RCTs was assessed by Jadad scale.For the 97 antihypertensive medicines in NRDL, there were sound evidence supports for the effectiveness of 56 kinds of medicines. Specifically, twenty-six of them were supported by international evidence-based guidelines, twenty were supported by systematic review or meta-analysis and the other ten by RCT. However, for the rest 41 medicines, there is insufficient evidence for their effectiveness.Some antihypertensive medicines in NRDL did not have sufficient evidence for their effectiveness. Further evaluation and revision were required. It is also recommended to standardize decision-making procedures for inclusion of medicines, set up high quality evidence database to timely provide sound evidence, and so on.The online version of this article (10.1186/s12913-019-3937-0) contains supplementary material, which is available to authorized users. Evidence-based decision on drug list or formulary has been applied worldwide . After pIn China, National Reimbursement Drug List (NRDL) of Urban Employee Basic Medical Insurance (UEBMI) is an important guidance list for production, supply, use, supervision, and management of included medicines. The medicines in this list are categorized into Class A and Class B. The medicines in Class A are often widely used in clinical treatment and have good curative effect, while the medicines in Class B alternatively using in clinical practice also have well treatment efficacy, but higher price . The NRDAlthough the importance of scientific evidence was emphasized, which has also promoted the renewal and improvement in selection of medicines into the NRDL of UEBMI to some extent, the further improvement of the evidence-based evaluation system and working mechanism are still in urgent need for China , 7. For For the wide coverage of UEBMI medicines, it is of great importance to use evidence-based methods to ensure their safety, efficacy, and cost-effectiveness. In consideration of their public health significance, this study takes the antihypertensive medicines as example to examine the situation of evidence support for the effectiveness of the medicines in the NRDL of UEBMI. The findings can provide recommendations for further revision of NRDL and the improvement of evidence-based decision-making.To objectively judge whether the effectiveness of certain medicine was supported by sound evidence, we set a strategy by reviewing international evidence-based medicine lists in combination with published research paper of high evidence grade.Scientific evidence support and representative are two prerequisites for the selection of international evidence-based clinical guidelines or medicine lists as reference criteria. Representativeness implies the generality to a certain extent, while the scientific evidence support means the development of certain guidelines or drug lists following a formal and scientific procedure, such as searching and screening available high-level evidence, extracting, synthesizing and assessing data by rigorous methods, and so on.Comprehensively considering the two elements mentioned above, three international evidence-based clinical guidelines or medicine list were selected: the WHO Essential Medicine List of 2017 version (EML 2017) , the EigFor each drug in the NRDL, a literature search was performed using the following databases: Pubmed, Medline, Web of science, China National Knowledge Infrastructure (CNKI), Wangfang Data, Chinese Bio-Medicine (CBM) database. The search started with broad search terms, such as the genetic name or commonly used brand name of certain medicine included in the current NRDL of UEBMI to treat hypertension, then narrowing down to the field of effectiveness with the time span between Jan 1st, 1995 and March 31st, 2017.Clinical studies focusing on the certain medicines’ effectiveness of the anti-hypertension with the research design as meta-analysis, systematic review, or randomized controlled trials between Jan 1st, 1995 and March 31st, 2017.Studies mainly focusing on economic or cost analysisStudies that were not designed as meta-analysis, systematic review, or randomized controlled trials.Meta-analyses or systematic reviews that were not the latest ones on the effectiveness of certain medicines.Clinical studies not published in English or Chinese.The literature quality of the searched articles were subsequently reviewed and scored by two independent reviewers under the guidance of the scales or checklists mentioned below. If there were differences in the scores between the two reviewers, they were further discussed for consensus purpose.The reporting quality of systematic review and meta-analysis were assessed by using Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement. The checklist has 27 items about the literature integrity, such as objection in the introduction section, data collection process in the methods section, summary of evidence in the discussion section and so on . If certMeanwhile, the methodological quality of systematic review and meta-analysis were assessed by applying A Measurement Tool for the ‘Assessment of multiple systematic reviews’ (AMSTAR). The scale contains 11 items, such as “Was an “a priori” design provided?”, “Was a list of studies (included and excluded) provided?”, and so on. If the answer is “Yes”, “Partly” and “No”, it will get 1 point, 0.5 point and 0 point, respectively. Therefore, the total score ranges from 0 to 11 points . By convThe methodological qualities of the included RCT were assessed by Jadad scale, which contains three items, such as “Randomization”, “Double blinding” and “Withdrawals and dropouts” . For theThere are three steps as follows to assess the evidence for the effectiveness of anti-hypertensive medicines included on the NRDL.Step one, comparison among the drug lists. If certain medicine included in NRDL of UEBMI was also included in at least two international evidence-based clinical guidelines or medicine lists mentioned above, it can be perceived that the effectiveness of this medicine was supported by sound evidence. Since the development of these evidence-based medicine lists following a scientific procedure which includes evidence searching and implementation from academic literature, these medicines supported by the evidence-based medicine lists will be not referred to published research paper for further assessment (No need to next step). However, if certain medicine included in NRDL of UEBMI was only included in one or none of these three lists, for considering many guidelines do not purport to provide an exhaustive list of all effective medicines, it will be referred to some published research paper with high-level evidence grade for further assessment.Step two, collect evidence from systematic review or meta-analysis. For the medicines only included in one or none of these three international evidence-based lists, the systematic review or meta-analysis on their antihypertensive effectiveness will be reviewed. If certain medicine was confirmed effective in treating hypertension with low incidence of adverse events in meta-analysis or systematic review with acceptable literature quality, the effectiveness of this medicine can be also considered as having sound evidence support. Since systematic reviews or meta-analyses are the literature with highest evidence grade for its comprehensively combining the results of previous RCT, these medicines supported by meta-analysis or systematic review will be not referred to published RCT for further assessment (No need to next step). However, for the rest medicines having evidence support from neither international evidence-based medicine lists nor meta-analyses / systematic reviews, they will be referred to published RCT for further assessment.Step three, collect evidence from RCT. For the rest medicines mentioned above, RCTs on their effectiveness in treating hypertension will be reviewed. If certain medicine was reported by RCT with acceptable methodological quality as an effective antihypertensive medication with few adverse events, the effectiveness of this medicine can also be considered as having sound evidence support. Or else, the effectiveness of certain medicine in treating hypertension will be considered as having insufficient evidence.The times of certain medicine included in the three-international evidence-based clinical guidelines on hypertension were calculated. For the medicines only included in one guideline or not included in any one of these three guidelines, the numbers of the kinds of medicines supported by published systematic review, meta-analysis and RCT were calculated respectively.There are 97 antihypertensive medicines included in NRDL of UEBMI, which were divided into Class A and Class B. There are 26 medicines in Class A (26.8%) and 71 included in Class B (73.2%). Based on the mechanism of action, the 97 antihypertensive medicines can be divided into seven categories, such as calcium antagonist, β-blockers, diuretics, angiotensin converting enzyme inhibitors (ACEIs), Angiotensin II receptor antagonist, vasodilators and others, respectively accounting for 24.7, 14.4, 11.4, 15.5, 12.4, 7.2 and 14.4% of all the 97 kinds of medicines.Twenty-six of the 97 surveyed medicines were included in all or two of the three international evidence-based clinical guidelines, which accounted for 26.8%. The numbers of the kinds of medicines included in one of the three clinical guidelines were 16, accounting for 16.5% of the total. Additionally, fifty-five kinds of medicines (approximately 60%) were included in neither of the guidelines  to 73.3% (for ACEIs). Regarding to the different management categories, approximately 70% of the antihypertensive medicines in Class A have sound evidence support for their effectiveness, while the corresponding proportion is about 50% for the antihypertensive medicines in Class B.  of antihypertensive medicines were supported by meta-analysis or systematic review with regard to their effectiveness, while only 10.3% (10/97) of antihypertensive medicines were confirmed by RCT. Besides, it is worth noting that nearly 30% of the included RCTs had unacceptable quality, which implied that the methodological quality of RCT needs further improvement. Some shortcomings, such as inappropriate method in description of withdrawals and dropouts and failure in double blinding, had greatly deteriorated the authenticity and reliability of RCT. For the sake of further improving the methodological or report quality of systematic review, meta-analysis and RCT, it is recommended to provide researchers with methodological trainings on special issues, such as proper quality evaluation methods for meta-analysis, systematic review and RCT, appropriate method for applying randomization and blinding . AdditioWith regard to the overall situation of evidence support for the effectiveness, only a small proportion of the medicines among vasodilators were confirmed by sound evidence, which highlights the need of further adjustment for certain kinds of medicines. Additionally, in comparison with Class B, a larger proportion of antihypertensive medicines in Class A were confirmed by sound evidence, which demonstrates that more deliberate management had been implemented on essential and commonly used medicines.However, it is still noteworthy that there is insufficient evidence for the effectiveness of almost 45% of the antihypertensive medicines in the NRDL of UEBMI. What is more, six of them were demonstrated by meta-analysis or systematic review as having no definitive conclusion on the effectiveness of treating hypertension. Although it can be more prudent to wait for more evidence, these medicines were still included in the NRDL. There are some reasons underlying this fact. One plausible reason may be the price issue even though it is not the focus of this paper. There is no denying that the drugs with less efficacies but lower prices also tended to be included in the insurance drug list for their potential great economic accessibility. Besides, the influence of the decision-making process of medicine inclusion in China also can not be ignored. It is an expert reviews system with hundreds or even thousands of experts involved to provide their opinions on the safety, clinical efficacy and economic cost of certain medicine. And the decision will be made to include or exclude certain medicine from the NRDL by comprehensively taking the experts’ opinion into account . ApparenTo promote scientific and efficient decision-making, some recommendations are also proposed as two folds: on the one hand, we should steadily improve the methodological quality of academic literature, such as meta-analysis, systematic review and RCT. It is also essential to widely implement continuing education and training for evaluation methodology of medicine safety, efficacy and economy. On the other hand, we should set up high quality evidence database and establish specialized evidence evaluation agencies . These dAlthough this research and the findings are China-based, the policy implications probably have wider applicability in the sense that many other countries are also confronted with similar dilemma of decision-making while including medicines into NRDL. Even more important, this study provides a new strategy of comprehensively integrating international and local available high-grade evidence to make a rapid evaluation of NRDL’s evidence-based rationality. Apparently, this strategy is not unique to a particular country or only applicable to single kinds of medicine. It can also be widely applied by many other countries to examine the evidence for certain kinds of medicines, such as antihypertensive medicines, anti-diabetic medicines, anti-cancer medicines and so on.As a preliminary application of a rapid evaluation strategy, this study still has some limitations, which should be noticed when subsequently quoting the results or applying the evaluation strategy. Firstly, this study mainly collected published meta-analysis, systematic review, and RCT to evaluate the effectiveness of the medicines without direct evidence support from international evidence-based drug lists. However, the publication bias is still hard to be completely ruled out. Secondly, by considering scientific evidence support and representativeness, three international evidence-based medicine lists were selected as reference criteria in this study. However, there is still doubt about whether there are other international evidence-based medicine lists more appropriately for the reference criteria. In view of this issue, the intensive comparison between many commonly used international evidence-based medicine lists will be further considered in future studies. And the research outcomes will further improve the rapid evaluation strategy in this study.Nearly 45% of the antihypertensive medicines included in the NRDL of UEBMI lack sound evidence supports for its effectiveness. Some shortcomings in reporting and methodological quality of published meta-analysis, systematic review, and RCT had deteriorated their authenticity and reliability, thus weakening their evidence support for certain antihypertensive medicines. It highlights the importance of providing professional and comprehensive methodological trainings to researchers and reinforcing methodological quality assurance in writing peer-reviewed journal articles. To optimize the selection of medicines into NRDL, it is recommended to establish specialized evidence evaluation agencies and set up high quality evidence database to timely provide sound evidence, as well as standardize the decision-making procedures for inclusion of medicines.Additional file 1:Table S1. The completion of each items in AMSTAR Scale——For the included systematic reviews and meta-analyses. It provides information on whether certain included systematic reviews and meta-analyses had completed each of the 11 items in AMSTAR Scale. (DOCX 27 kb)Additional file 2:Table S2. Jadad score for the included randomized controlled trials. It provides information on whether certain included RCT had completed each of the 3 items, namely “randomization”, “double blinding” and “withdrawals and dropouts”, in the Jadad scale. (DOCX 21 kb)


The elicitation of broadly neutralizing antibodies (bnAbs) is considered crucial for an effective, preventive HIV-1 vaccine. Led by the discovery of a new generation of potent bnAbs, the field has significantly advanced over the past decade. There is a wealth of knowledge about the development of bnAbs in natural infection, their specificity, potency, breadth and function. Yet, devising immunogens and vaccination regimens that evoke bnAb responses has not been successful. Where are the roadblocks in their development? What can we learn from natural infection, where bnAb induction is possible but rare? Herein, we will reflect on key discoveries and discuss open questions that may bear crucial insights needed to move towards creating effective bnAb vaccines. The potency of the neutralization response to HIV-1 has long been underappreciated, as most of the antibodies identified were type-specific or had only limited breadth . This poDiscoveries in the last decade have revealed that such elite neutralizing responses occur during natural infection, but are rare –14. InitFocused efforts to decipher antibody maturation pathways alongside virus evolution have facilitated reconstruction of bnAb development in some individuals , 17–23. Prolonged bnAb maturation does not necessarily lead to improved breadth. The highest frequency of bnAb activity is observed after approximately 3 years of infection , 32–34. Virus escape creates new epitopes that allow bnAb responses to mature , 24. SloThe virus population is not only shaped by the bnAb lineage, but is also subject to pressure of the vigorous type-specific antibody response each patient mounts. This can, in turn, substantially impact bnAb evolution. Early bnAb development may be compromised through competitive exclusion by strain-specific antibodies that target the same epitope , 45. YetA range of factors that promote bnAb development has been implicated and several confirmed across different cohorts , 48–50 . These sTwo other parameters linked to extended antigenic stimulation, virus diversity and length of infection, also increase the likelihood of bnAb elicitation , 48, 49.As evidenced by the unresolved impact of viral diversity in steering bnAb responses, the overall role of the infecting virus remains to be defined. What are the genetic determinants of the virus that trigger a bnAb lineage? As discussed above, it is possible that viral diversity can foster bnAb evolution, but whether diversity at the level of the infecting virus  is decisive or not, has not been resolved.Understanding the role of the T/F viruses may be key . While nA main issue that has not been clarified is whether the capacity of a virus to induce a bnAb is (1) restricted to a specific Env variant that is only transiently present and lost upon Env evolution or (2) preserved over prolonged time periods and may even be stable over multiple transmissions. Despite continuous Env evolution, there is emerging proof that some Env traits are stable and evoke similar responses. Clear evidence for this stems from the comparison of bnAb specificities mounted in subtype B infection compared to non-B infection. Non-B infection proved superior in mounting V2-glycan-directed responses , which iIn principle, bnAb development could be facilitated by a range of phenotypic virus features that influence epitope exposure and accessibility. These include, Env conformation and stability, the degree of shielding, and Env density on the virion surface. Unravelling genetic and/or structural Env features that promote or suppress bnAb development will be crucial for vaccine success.+ T cell levels and a high rate of CD4+ T cell decline appear to be linked with broad neutralization in some cohorts [+ T cell subsets was not assessed directly in most studies, lower CD4+ T cell levels may implicate reduced levels of regulatory CD4+ T cells that may feed into bnAb development. However, a link between low CD4+ T cell levels and neutralization breadth was not observed universally [+ T cell counts and viral load [+ T cell levels requires cohorts with the power to dissect confounding factors. Controlling for a range of factors, including viral load and infection length, the Swiss 4.5k Screen studying 4484 individuals found only a marginal independent impact of lower CD4+ T cell levels in individuals that mounted low levels of breadth and not those who had potent bnAb activity [+ T cell levels were found not to correlate with binding antibody responses to gp120, but a strong inverse effect on MPER IgG1 levels was observed [+ T cells. In fact, a recent study suggests that perturbations in the CD4+ T cell environment are linked with neutralization breadth. bnAb inducers had lower numbers of PD-1high regulatory CD4+ T cells, highlighting a decreased regulatory capacity [FH) CD4+ cells, particularly early in infection [bnAbs often show poly/autoreactivity (reviewed in ) and aut cohorts , 33, 50.versally , 49, 51.ral load . With viactivity . Of noteobserved . This sucapacity . Extensicapacity  and elevnfection , 78–80. nfection –83.What stimulates such beneficial immune environments has not been resolved, but host genetics are implicated by several lines of evidence. A decreased prevalence of the protective allele HLA-B*57  (linked In support of a strong genetic influence, individuals with black ethnicity were found to more frequently induce bnAb activity compared to white study participants  and haveWith increased knowledge about parameters that are linked with bnAb development in natural infection Tables , opportuTiter  and avidThe importance of effector functions in the protective effect of neutralizing antibodies has been long recognized and both activity of non-neutralizing and neutralizing antibodies in the context of effector functions implicated , 91–96. IgG subtypes differ in their capacity to deliver effector functions and differential glycosylation of the Fc also influences the interaction with Fc receptors and the efficacy of the effector responses . ConsideWhile most bnAbs have high levels of SHM and evolved after prolonged maturation pathways, this seems not to be needed in all settings. A proportion of adults develop bnAb activity comparatively rapidly , 22, 32.While antibodies with lower SHM may prove to be common in infants, it is intriguing that bnAbs with low SHM have not been described in adults. Does the adult immune system not favor the development of low SHM bnAbs? A potential scenario could be that cross-reactive antibodies that bind to HIV Env exist in adults. Following the dogma of original antigenic sin , insteadVertical transmission is distinct from all other HIV-1 transmissions, as maternal antibodies are transferred and remain present for prolonged time periods. The role of maternal Abs in influencing transmission risk is still debated –115. IrrHigh viral load—a strong influence on bnAb evolution in adults—is frequent in infant HIV-1 infection in the first 2 years of life , 118 andDetailed analysis of determinants that shape bnAb responses in natural infection have led to the identification of a range of factors that are linked with the development of neutralization breadth. However, so far we lack formal proof for which factors are ultimately causative in bnAb elicitation and which evolve alongside. Delineating the causes and consequences, as well as defining parameters that need to be incorporated into vaccine regimens will be critical. Figure


Diagnostic tests’ impact on patient outcomes and health processes is potentially large, and proper evaluations before widespread adoption are warranted. Such evaluations are challenged by the fact that tests can have multiple purposes, in different clinical pathways, with different roles.The National Institute for Health Research (NIHR) established four Diagnostic Evidence Cooperatives (DEC) in 2013, across England. The aim of these DECs was to facilitate the development and evaluation of clinically relevant in vitro diagnostics, by offering methodological expertise and access to real-life settings for evaluations in patients. In this commentary, we discuss our experience over the past 4 years.The interaction of industry, researchers, and clinicians has proven to be very worthwhile. Diagnostic tests are pivotal to health care and encompass a wide range of modalities from history taking or clinical examination to imaging, blood tests, and pathology. We use tests for a variety of reasons, such as risk prediction, early detection, diagnosis, monitoring, and prognosis. In contrast to treatments, the potential impact of tests is not always appreciated. Tests obviously can have indirect impact by changing treatment decisions, but they can also have direct impact including the psychological and emotional effects of diagnostic labelling (be it positive or negative) and complications of tests such as bowel perforation from colonoscopy or radiation exposure from imaging. Whether or not the test is worthwhile depends on the balance between benefits and harms, which means both need to be assessed and quantified. Several frameworks have been proposed that describe the different types of evidence that is needed for new diagnostic tests , and altAs a result, evidence is required for tests to be allowed onto the market and recommended for routine care. In the past, regulatory approvals focused mainly on safety, with many being able to self-certify for CE marking which secures market access in the EU and beyond. In the new EU Regulation for In Vitro Diagnostics (Regulation 2017/746), published in May 2017 and coming into full effect in 2022, more evidence will be required to demonstrate safety and accuracy depending on the test’s risk level, which will pose additional challenges to in vitro diagnostic (IVD) developers. National bodies such as the National Institute for Health and Care Excellence (NICE) already have raised the bar for diagnostic tests to be recommended, by demanding evidence on cost-effectiveness which includes impact on patient outcome albeit this may be indirect from linking diagnostic accuracy evidence with pre-existing evidence on treatment efficacy. On average, it takes 9 years for evidence to accumulate on a test’s ability to diagnose a condition through to its efficacy and cost-effectiveness . This coMany diagnostic tests have multiple purposes, a feature which complicates test evaluation. For example, HbA1c is used for both monitoring diabetic treatment response and initial diagnosis of diabetes. New regulations demand the intended role be clear, showing accuracy for that role in a relevant population because accuracy may change with changes in the spectrum of disease, an issue that is particularly relevant for primary care , 7. MoreIn routine care, every innovative test will be part of a clinical pathway, and its effect will largely depend on the nature of this clinical pathway and how effective this existing pathway is without additional tests. In the early phases of diagnostic test evaluation, describing this clinical pathway is of the utmost importance. It will clarify the goal of the new test and its associated desired diagnostic properties. It will also help to identify the minimum requirements for a cost-effective test. Clinical pathways are mostly established by expert opinion, clinical guidelines, and best practice. However, real-life pathways can deviate significantly from these ‘ideal’ pathways. Patients undergoing chemotherapy have been shown to follow a large number of pathways, almost equal to the number of patients: in effect, an individual pathway . This inFor all these reasons, it should come as no surprise that the transition from a successful proof of concept to a commercially available test that is ready for implementation in routine care has been shown to be challenging, especially for test developers whose expertise understandably lies in technological development rather than clinical epidemiology. This ‘valley of death’ could be overcome by providing test developers with access to expertise in diagnostic test evaluation, information on clinical needs, and real-life settings for evaluations in patient populations. To this end, the UK Department of Health and the National Institute for Health Research (NIHR) funded four Diagnostic Evidence Cooperatives (DECs) in 2013, based in Newcastle, Leeds, Oxford, and London. Our Oxford DEC has focused on in vitro diagnostics for primary care, where currently diagnostic opportunities are relatively limited or require a referral to a laboratory, imaging facility, or secondary care. Our team consists of general practitioners and researchers with expertise in diagnostic test evaluation, health economists, statisticians, and qualitative researchers.https://www.community.healthcare.mic.nihr.ac.uk/about-us/horizon-scanning-1). However, a review of all of our Horizon Scans showed that for most tests evidence on impact is lacking [Over the past 4 years, we have met with 54 companies to advise them on the test’s potential role in primary health care, and the next steps in the evidence accumulation. We have been amazed at the technological possibilities under development, which hold great promise for the future. Our Horizon Scanning program, which has used a standardized published methodology  to produClinicians have generally shown appetite for new diagnostic tests. An international survey of general practitioners indicated a need for point-of-care tests for acute and potentially serious conditions such as myocardial infarction and pulmonary embolism to guide referrals, for infections to guide treatment in primary care, and for chronic conditions such as diabetes and anticoagulant use to monitor treatment . StrikinIn January 2018, all four DECs have reincarnated as some of the 11 different organizations, which have been designated NIHR Medtech and In Vitro Diagnostics Cooperatives. Our new incarnation, the NIHR Community Healthcare Medtech and IVD Cooperative, will learn from the past 4 years’ work by engaging with companies far earlier in the pipeline of development of diagnostic tests, in addition to the ongoing remit of supporting the generation of evidence for commercially available diagnostics. Working with companies who may not yet have chosen a target pathway for their diagnostic will allow us to support the development of tests which fill a genuine clinical need and to harness the potential of exciting new technologies to improve patient outcomes in community healthcare, with particular focus on chronic disease, infections, and diagnostics which facilitate ambulatory care. In addition, we have expanded our remit from purely IVDs to all diagnostic technologies, recognizing that often new pathways of care will include multiple tests of different modalities.The DECs have been a very worthwhile experiment in asking two very different types of people, industries, and researchers, to explore each other’s perspectives and to develop a shared approach to achieving the goals of introducing new relevant diagnostics to improve patient care. Our DEC has explored clinical needs in current routine practice, collated existing evidence on innovative diagnostic tests, highlighted evidence gaps in the current evidence trajectory, and facilitated evidence accumulation by providing access to expertise, patient populations, and clinical studies. The DECs, with their mandate to support industry in evidence generation, are unique in life sciences. Other fields of industrial development could well benefit from similar approaches.


In many countries, committees make priority-setting decisions in order to control healthcare costs. These decisions take into account relevant criteria, including clinical effectiveness, cost-effectiveness, and need, and are supported by evidence usually drawn from clinical and economic studies. These sources of evidence do not include the specific perspective and information that patients can provide about the condition and treatment.Drawing on arguments from political philosophy and ethics that are the ethical basis for many priority-setting bodies, the authors argue that criteria like need and its effects on patients and caregivers are best supported by evidence generated from patients’ experiences. Social sciences and mixed-methods research support the generation and collection of robust evidence.Patient experience is required for a decision-making process that considers all relevant evidence. For fair priority-setting, decision-makers should consider relevant evidence and reasons, so patient experience evidence should not be ignored. Patient experience must be gathered in a way that generates high quality and methodologically rigorous evidence. Established quantitative and qualitative methods can assure that evidence is systematic, adherent to quality standards, and valid. Patient, like clinical, evidence should be subject to a transparent review process.Considering all relevant evidence gives each person an equal opportunity at having their treatment funded. Patient experience gives context to the clinical evidence and also directly informs our understanding of the nature of the condition and its effects, including patients’ needs, how to meet them, and the burden of illness. Such evidence also serves to contextualise reported effects of the treatment. The requirement to include patient experience as evidence has important policy implications for bodies that make priority-setting decisions since it proposes that new types of evidence reviews are commissioned and considered. Healthcare costs have continued to rise, and healthcare systems increasingly rely on gatekeepers or regulators to control costs. In some countries, these take the form of explicit committees, like the UK’s National Institute for Health and Care Excellence (NICE), Germany’s Institute for Quality and efficiency in Health Care (IQWiG), or the Swedish Council on Health Technology Assessment (SBU). Elsewhere, payers make decisions, like US insurance companies determinations of the coverage offered, or local boards that allocate a health budget. Each “committee” of decision makers does or ought to take into account relevant criteria such as clinical effectiveness, cost-effectiveness, and patients’ need –4. Whethas evidence in healthcare decision-making. In the course of making this argument we distinguish this role for patient experience from other kinds of patient or public involvement. Typically, these do not seek to generate systematic evidence of the kind used in evaluating the criteria for a health technology assessment.In this paper, we will argue that patient experience should play a role Consideration of patient experience as evidence is an important element of the decision-making process for priority setting: its consideration alongside other forms of evidence helps to ensure equality of opportunity by making certain important evidence is not overlooked. Patient experience, considered as evidence, contributes to understanding the nature of the condition, the effect of the treatment, and the effect of the treatment on patients’ and carers’ lives, which is all evidence relevant to the decision. It will therefore be important that not only decision makers but also patients understand their role in contributing to the evidence for a decision.In response to likely concerns about the quality and standards of patient evidence, we suggest research methods that might be used to generate the evidence. Treating patient experience as evidence, in the way we suggest here, uses systematic methodology rather than personal anecdote to generate evidence that can be rigorously considered alongside scientific evidence. Concerns about quality can be met with good methodologies, and should not pose a barrier to using patient evidence in decision-making for priority setting.It seems fair that patients who will be directly affected by the healthcare committee’s decisions are in some way included or involved in the decision-making process.This statement seems to capture something important about the justification of a role for patients in decision-making. The immediate question is how this thought survives reflection on the nature and justification of the decision-making process. For example, it is unclear whether (i) this is about involving patients in the decision, as decision-makers for example or (ii) about the value and role of their experience as relevant evidence for the decision , 8.In what follows we consider the second of the alternatives mentioned: the focus of this paper is to consider the ways in which patient experience can and should act as evidence in the deliberations of healthcare limit-setting or commissioning committees. In subsequent sections, we develop a robust account of the nature of patient experience as evidence and what counts as high quality evidence of this type.Following on from the thought expressed above, we may think patients have a right to be heard—inclusiveness in these processes means allowing patients or patient groups to express their views and have their say –13. HereThese issues of process, however, are crucially distinct from the content of the decision to be made and the obligation on the decision-maker to ensure that the decision-making process gathers all of the relevant evidence and makes its decisions in light of all of that evidence. Evidence is an issue of content, not process. This paper is concerned with evidence as decision-making content and will not directly consider the very important questions concerning how best and most appropriately to involve patients and patient groups in the decision process.This also means that the paper focuses squarely on the nature of the evidence that is required for the decision and ways of ensuring that the evidence is of the highest quality. Therefore, we set aside questions about the right to be heard, the right to have a say and the ways in which these rights might be protected.This is the key question. On the face of it, the experience that patients have in living with a particular condition and with its (attempted) treatments is important in understanding: the nature of the condition; the impact it has on the lives of those with the condition and those around them; the potential for treatments to assist with the condition and the impact it has on life; and the reality of the impact of new treatments for the condition –25. The as evidence alongside the other forms of evidence, such as that from scientific and economic analyses. This section lays out the justification for its inclusion.Beyond the instrumental value of patient experience, the literature has not engaged with the questions of why and how these experiences should feature in the process The claim that patient experience should be gathered and presented as part of the evidence considered in the decision-making process depends on the arguments about the nature of the decision and the justification of the process. The decision-making process should be established in such a way that the best (and fairest) decisions can be made in difficult circumstances.how much each one matters. The pragmatic approach to the theoretical positions of wanting both an equal process and taking into account disagreement on substantive principles is to propose a process that treats participants fairly but also allows for deliberation and debate on how to weight different kinds of values. Four key features of this argument justify a process that includes equal consideration and the use of evidence, including patient evidence, are as follows:i.There is reasonable disagreement about what is the right decision for funding a treatment in a range of cases. This feature of the resource allocation context provides the grounding for the process: it is in the light of this disagreement that the process can be considered to be a fair method of making a decision.One might assume that the fairest way to distribute scarce resources is to hold a lottery since this gives each person an equal chance at receiving scarce resources. While each person is equally likely to have her needs met, it will quickly become apparent that the lottery does not meet other substantive values that are widely held: it does not account for how sick a person is, i.e. how great her need is, it does not consider how effective the treatments are or how much each one costs, some may even object that it does not consider who wins the lottery and that some features, like age, matter. The lottery thought experiment shows that there are substantive considerations, other values, that matter in addition to the fairness of the process by which winners and losers are chosen. However, while there may be general agreement on the sorts of substantive values that matter  there is not agreement on The initial problem of making just decisions about healthcare resources—the problem of allocating these resources in the fairest way—is noticeably defined by a distinct kind of disagreement: people can reasonably disagree about what the right course of action is –30. SomeThis disagreement is reasonable because it can and does occur between individuals who satisfy a number of important characteristics. Reasonable disagreement is disagreement between individuals who (i) understand that a decision needs to be made, (ii) understand that others are likely to disagree, (iii) understand that there are a range of relevant criteria to be considered and to which others may appeal in the defence of their view, (iv) are able to accept that others may value the criteria in different ways for a given decision and (v) recognise that the processes involved in coming to a decision centre on reasons and reasoning.ii.All people should be given an equal opportunity at having the treatments for their condition being funded, which is achieved by following a fair process and giving equal consideration of relevant criteria from available evidence in the decision-making process.The problem of making decisions when there is reasonable disagreement is pervasive in modern, democratic societies that recognize that citizens will hold different views of what is morally right, and this is often the case for making determinations about shared or public goods, like healthcare. If people cannot agree on the right way to distribute healthcare resources, then a fair process leads to decisions that follow a procedural standard rather than applying a specific moral, substantive view to each one.Because people can reasonably disagree about the way in which healthcare resources should be allocated and so whether particular treatments should be funded, a process which enables all parties to have a fair hearing for their cause is one plausible system for helping to ensure that justice is done. “Equal opportunity” here means that each person or patient group has equal access to the fair process, equal opportunity to present their case and evidence, and the case should be judged on its particular merits and on the basis of all the available evidence. Each case equally follows the process even if the same outcome does not result. This is different from a lottery that gives each person an equal chance, which raises other concerns discussed above, and instead emphasizes the procedural consideration of substantive values. Equal opportunity is connected to the idea, familiar from the work of John Rawls, of justice as fairness . It alsoiii.Equal consideration and so equal opportunity requires that all relevant reasons are considered fully and in the light of all of the relevant evidence.An ‘equal opportunity at having their treatment funded’ is a claim about the content of the decision-making and how the decision-making process is constructed. It means that the various stakeholders’ perspectives and reasoning should be included in the decision-making process. This must include patients and carers with the condition in question and also those with an interest in the treatment or who will be responsible for its delivery. Importantly, in each case, an equal opportunity refers to what evidence and reasons are considered as a part of the decision-making process and how they are considered.Giving and considering reasons for and against funding is the key element of the fair process , 36–38. Ultimately there are only two sources of genuine disagreement within a system that is characterised by reasonable disagreement.iv.If all relevant reasons for and against a particular funding decision are to be considered fully and fairly, a full range of evidence in support of those reasons must be also considered.Both of these sources of disagreement are intractable in this context. The process is not designed to resolve this problem but to enable a fair decision in the context of it.Patient and carer experience of the nature of the condition, its effects on their lives, and the effects of the treatment is relevant to the decision-making and so should be formally included within it. This experience, like all other forms of evidence, should feature in the process as evidence in support of the criteria used in the process. Just as health economic evidence about costs and clinical effectiveness can support reasoning about cost-effectiveness, so too, evidence about what it is like to live with a condition can support reasoning about need: the nature of the condition, the effect of the condition on lives of sufferers and their carers, and the way in which the treatment might play a role in their lives. In short, this experience can be important evidence for decision-making and, given our interest for reasons of fairness in the inclusion of the full range of evidence, the experiences of patients and carers should be included.It is one thing to justify that these experiences ought to be a part of the decision-making process as evidence but it is another to give an account of how, in general terms, this evidence might play a role in those decisions.The patient and public involvement literature is replete with examples of the way information provided by patients can bring context and depth to other sources of data collected in clinical research studies. Such context and depth is seen to provide a fuller account of these alternative data sets and to assist in maximising the relevance of such data sets when translated from ‘bench to bedside’. However, this literature invariably fails to make reference either to the role that this added value might play in the actual processes of decision-making or how patient experiences more generally  should feature in funding decisions . In deveThese three examples illustrate the way in which the criteria function to provide reasons for particular decisions. Clearly in actual cases the criteria and so the reasons pull in different directions: perhaps most often, patient need gives decision-makers reason to fund but cost-effectiveness provides a reason not to fund. The process described above requires that each of the criteria are treated and considered equally. Most importantly, it means that decision-makers should not approach the decision-making task with a prejudged value attached to any of the criteria. They should not be predisposed to value one more highly than another: to do so would be to fail to treat all stakeholders equally in the decision-making process. We assume the equal stance of stakeholders since these decisions occur in the context of allocating healthcare resources to those who are participants in the system, either as insurance-premium or tax payers.Evidence functions in a definite way to provide support for reasons. So evidence of patient need provides a reason to fund a treatment whereas evidence of a lack of need provides a reason not to fund. Similarly, evidence of clinical effectiveness provides a reason to fund whereas evidence of no effect provides a reason not to fund. In each of these cases the evidence does not necessarily provide an over-riding reason—what gives us the strongest or an over-riding reason will depend on the other reasons, their relative strength, and the judgement of the decision-maker. Also, clearly, lack of evidence gives no reason either way.So far the argument has shown that fairness requires equal opportunity and that equal opportunity requires equal consideration of relevant criteria. Equal consideration of relevant criteria requires attention to the full range of reasons for and against funding a treatment. Proper attention to all these reasons is possible only when adequate evidence is provided in support of each.In the examples above, we have mentioned only patient need, clinical effectiveness and cost-effectiveness as relevant reasons. As we will see below, there are a number of others that should be included. We will also discuss how the decision-making process and criteria used function as a range of relevant reasons that ought to be considered.When making decisions about new treatments, committees use criteria such as the nature of the condition, the impact of the new treatment, the cost and cost-effectiveness of the treatment, and the treatment’s impact beyond direct health benefits.The idea is that each of these criteria is to be considered by the committee in order to make a decision about funding. In this way, these criteria function as the relevant considerations that play a central role in making the process of decision-making a fair one—it is because these criteria capture the possible range of relevant reasons that the process is able to be fair and procedurally just. Following from the arguments above then, evidence which supports each of these criteria is also crucial in making the process a fair one.The nature of the condition and its effects on the lives of patients and their carers.It is important, therefore, to articulate clearly how accounts of patient and carer experience connect to the decision criteria. There are three ways in which patient and carer experience can function as evidence in support of these criteria—in each case the experience of patients and carers is importantly placed to support or weaken the role the criteria play in the decision-making reasoning. Three of the six criteria are most relevant here: (a) the nature of the condition, (b) the impact of the treatment, and (c) the impact of the treatment beyond direct health benefits. The other criteria depend heavily on knowledge of the costs associated with providing the treatment. Patients and carers are not well placed to contribute to these criteria.i.the lives of patients, family and carers.ii.what is of value in those respective lives.iii.the perceived lack/deficiencies in the lives of patients, family and carers.iv.the ways in which these deficiencies could be met (articulated in terms that do not rely directly on the effectiveness of the treatment).As we have suggested, patients and carers can provide important evidence about the needs that they experience living with the condition , 41, 42.b.Patients’ and carers’ experience of the condition and its effects on their lives means they can contextualise the clinical evidence that is provided to the committee.In each case the evidence provided is able to support reasoning about the needs of patients and carers, which informs a decision.Patients’ and carers’ knowledge and experience of the condition enable them to provide comment on the significance and relevance of the outcome measures used in research, the relevance of the research to the lives of patients with the condition, and the relevance of both clinical research and quality of life assessments. In this way patients’ and carers’ experiences provide evidence about the value of the research that has been conducted on the treatment under consideration , 43–50. An account of the experience of the treatment and its affects on the lives of patients and carers adds important context to the findings of clinical research , 47, 50.In the previous section, the nature and value of understanding patient experience as evidence has been articulated, and the role that such evidence ought to play in decision-making clarified in light of the relevant kinds of criteria.While these arguments clarify why patient evidence ought to be used, there are plausible concerns about the quality and methods of gathering patient experiences together as evidence. These concerns need to be addressed: if decision-makers are not confident of the quality of the evidence presented, the purpose of using patient evidence will be undermined. Moreover, if such evidence is of poor quality there is a danger that decisions made by the decision-makers that are contingent on this evidence will lack an adequate justification. For any kind of evidence, poor quality evidence should not be used to inform decisions. Therefore, it is important to address concerns about patient evidence and demonstrate that it can be systematically collected and evaluated.The first set of concerns focuses on ascertaining whether patient evidence has been obtained systematically. These worries highlight the need for clarification about the overall methodological framework guiding how patient evidence is collected, analysed and presented.In order for this evidence to be treated as being of equivalent status to other forms of evidence assessed in the process, the framework within which the evidence was generated must be appropriate. If this can be guaranteed, then the risk of bias in patient evidence would be reduced and concerns that it is a marketing ploy, mere anecdote, or a single case study that lacks external validity with limited force as evidence generalised to the patient group as a whole.The second set of concerns focuses on the adherence to particular quality standards in the systematic production of the patient evidence. Patient evidence must avoid the incorporation of bias and subjective interpretation. In order to meet evidence standards, it should be generated by researchers using rigorous methods and with appropriate knowledge in this field.robust and valid account of patients’ experiences. Without such standards, decision-makers are likely to be sceptical about the evidence and dismiss it from serious consideration in their evaluations, as they should with poor quality evidence.Adherence to certain quality standards is crucial to ensure that the patient evidence presented at the committee meetings can be taken to be a The concern about quality standards has been raised in the literature on evidence of this kind, and a number of solutions have been proposed by social scientists working in the area of health research –53. BecaBelow, we suggest how patient evidence could be generated in light of research standards articulated within social scientific inquiry. We show how these standards can shape (i) how studies are designed in order to document patient experiences in a methodologically systematic way that is tailored to the requirements of the committee’s evaluation, and (ii) how specific quality considerations are upheld in the process of collecting and analysing this kind of evidence.For data about patient and carer experiences to be counted as valid research evidence it should be gathered in a systematic way. Therefore, it is important that those responsible for generating this evidence have a clear methodological framework within which they are operating, and that this framework is orientated directly and explicitly towards the requirements of the committee’s evaluation process. The framework that we articulate below could underpin a number of different methodologies or methods within the social sciences. The appropriateness of each methodology and method needs to be considered in light of this general framework.Patient experience generates evidence about the nature of the condition and the impact of the technology under evaluation on the person’s health and broader quality of life. For this evidence to be most useful, it is important to capture a rich and in-depth account of patients’ lives so that all salient features about what it means live with the condition—and what it is like to receive an intervention to manage or treat this condition—are documented to the fullest extent.In contrast to other kinds of evidence that the committees review, the depth and detail of understanding required here points towards a qualitative study design. Qualitative inquiry includes prolonged and detailed contact with a person’s life situation, aims for a ‘holistic’ overview of the context under study, and attempts to capture data ‘from the inside’ by being responsive to the participants’ insights rather than being driven solely by the researchers’ priorities .Traditional social research methodologies within the domain of health aim to produce new understandings about some aspect of health care or health conditions. One commonly adopted methodology is ‘grounded theory’ –57, wherThis key observation implies that an appropriate methodology for generating patient evidence will need to be ‘top-down’ in nature, developing, shaping  accounts of patients’ experiences such that they can flesh out the relevant criteria that healthcare committee members are required to take into account when evaluating a specific technology.Sampling strategy Patients, carers and family members who are selected to share their experiences must be chosen on the grounds that (i) they have direct experience living with the condition, (ii) or that they have direct experience caring directly for those living with the condition, or (iii) that they have direct experience of the technology being evaluated (whether this be in their own country or elsewhere). This suggests a systematic though non-probabilistic sampling strategy, such as purposive sampling, should be endorsed.Methods for data collection Interviews with patients, carers and family members that aim to understand the experiences of these individuals must be structured in such a way that they focus solely on the criteria relevant to the particular healthcare decision-making process. This suggests that a structured or semi-structured interview strategy should be adopted in order to keep the elucidation of experience ‘on track’.Methods for data analysis The analysis of these interviews must be undertaken in ways that place the relevant criteria at the heart of the analytic process. For example, a ‘thematic analysis’ of patients’ experiences—where commonly occurring insights emerge in the process of coding interviews [terviews , 59—mustterviews , 61.Presentation of the evidence The evidence must be presented to the healthcare committee in a systematic way, derived explicitly from the structured analytic framework used to capture the different themes within the data and their relationship to the committee’s evaluation criteria. Extracts from interviews with patients, carers and family members should be presented as illustrative of the general insights obtained about patients’ understandings of the nature of the condition, and the experience of the treatment under consideration. They should not be presented as stand-alone examples of particular and unique personal experiences. A pro-forma approach for presenting the patient evidence to the committee would be one strong option, and such an approach would be entirely consistent with the methodological requirements and analytic process outlined above.Adopting a ‘top down’ approach to generating patient evidence has a number of practical implications for the study design and how the outcomes of the study inform the evaluation process:Qualitative inquiry sheds light on the range of relevant patient experiences, but gives limited insight into how these experiences are distributed across the patient group as a whole. It is possible, for example, that a range of experiences about what it is like to live with the relevant condition will be documented from the qualitative study, but that one of these experiences will prove to be common to all patients, whilst other experiences apply only to a minority of patients. Patient evidence can incorporate the extent to which different kinds of experiences are common to all patients, and that may be especially important to a healthcare committee making population-level decisions.For this reason, we recommend a mixed-methods methodological design that incorporates a survey of experiences across the relevant patient, carer and family member groups as a whole. The development of a patient questionnaire to undertake such a survey should be a secondary component of the methodological design, and the content of the questionnaire should be shaped directly by the range of experiences captured in the qualitative study. Existing validated questionnaires to measure health outcomes are unlikely to be fit for the purpose of the committee’s role . As the aim here is to document the commonality of the range of health experiences documented in the interviews, a questionnaire design that incorporates a Likert scale is likely to be sufficient.It is likely that the mixed-methods approach will be more compelling in the evaluation of some technologies rather than others. When the patient group affected by any given technology is relatively small, it might well be possible to conduct interviews with most of these patients, negating the additional value of conducting a survey. In contrast, a large patient group will mean that interviews will only be able to be conducted with a small sample of these patients. In these cases, we strongly advise that a survey is conducted following the interview study. Though we recognise that the feasibility of undertaking this study design will be contingent on surveying potentially large groups of patients affected by a particular health condition.To conclude, for patient experiences to function as evidence, it is important that these experiences are captured within a systematic methodological framework that has been designed to meet the requirements of the particular healthcare technology evaluation process. We recommend a mixed qualitative and quantitative study design that adopts certain key methodological characteristics to capture the relevant experiences in a systematic and reliable way.Once a systematic framework for generating patient evidence has been articulated, we need to turn our attention towards the quality standards that should guide this process. Concerns about the ‘quality’ of social inquiry have been raised over the last two decades, particularly in the context of health services research , 63. SomIn light of the broader methodological concerns, there have been significant attempts within the social research literature to articulate a common set of ‘quality standards’ that enable the validity and robustness of a piece of qualitative research to be judged on its own terms –70. SuchThere are a range of quality standards available in the literature that share some common features. One particular set of quality standards that has been extremely influential in social research is that put forward by Mays and Pope ; anotheri.Triangulation By comparing the outcomes of two different methods for data collection, the researcher can be confident that the evidence produced is comprehensive in its scope, and that an overall interpretation of the data has been corroborated by the different data sources available. This standard can be maintained by combining an interview phase and a survey phase in the generation of patient evidence. For example, if the survey results suggest that some patients who complete the questionnaire do not agree that the kinds of experiences captured within the items of the questionnaire are comparable with their own experiences, this would suggest that not all relevant experiences have been captured in the interview phase of the study, and that further interviews need to be undertaken.ii.Respondent validation The researcher can reduce analytic error by establishing that his/her interpretation of these data corresponds to the experiences that the study participants were trying to convey in the research process. This standard applies to the analysis of the interview data. Respondent validation generally requires the use of ‘member checking’, where a number of the patients interviewed are asked to comment on whether the central themes that are captured in the process of analysis match their experiences. If the patients are not able to validate these interpretations, further analysis of the data is required. In the survey phase of the study, validation of the responses from the patients also needs to be considered.iii.Clear exposition of methods of data collection and analysis The researcher needs to ensure that those reviewing or considering the data know precisely which methods were adopted, and how the decisions to use these methods were made. This standard applies to the presentation of the data, to those tasked with reviewing and critiquing the evidence, and to the committee members who need to review this evidence to evaluate the particular technology. Any critique of the evidence must be able to ‘track back’ to the decisions made by the researcher in designing the study, and a clear justification for why certain techniques or methods were employed needs to be provided for this evidence to be validated as fit for purpose.iv.Reflexivity The researcher needs to be sensitive to the multiple ways in which s/he, or the research process itself, has shaped how the data has been collected and how it has been interpreted. This standard applies to the collection, analysis and presentation of the data. Being sensitive to the researcher’s own positioning in the research process requires that any biases that might have been imported (i) through the researcher’s prior assumptions, (ii) through the degree of ‘distance’ s/he has from the patients interviewed, or (iii) because of the effects of his/her personal characteristics , need to be documented and discussed carefully. Steps should be taken to ensure that such biases are reduced to the greatest possible extent. If, for example, the decision is made that a member of the patient organisation should interview other patients on the grounds that this will increase the quality of the experiences shared by the patient, careful consideration should be paid to ensure that this researcher does not impose his/her own understandings of patient experiences into the dialogue. Equally, if an independent researcher is conducting these interviews, consideration should be given to whether the professional status of this researcher might limit the experiences that the patient feels comfortable sharing, and how rapport between the researcher and the patient in the interview setting could be improved [improved , 74.v.Attention to negative cases The researcher can improve the validity of the evidence generated by focusing on ‘outlier’ or ‘deviant’ cases that can help to refine and refocus the analysis to gain a comprehensive picture of the object of study. This standard has implications for sampling and for data analysis. In deciding who to interview, it is important to follow up interviews with patients whose experiences appear to be contradictory to the general accounts of patients’ experiences that emerge as the interviews progress. Focusing subsequent interviews on these ‘outlier’ or ‘deviant’ cases can help to refine and refocus the analysis to gain a comprehensive picture of the range of patients’ experiences that should be taken into account in the evaluation of a technology. The requirements of data ‘saturation’ ought to be invoked and applied here. If the interviews being conducted with patients continue to give rise to different kinds of experiences that had not been captured in previous interviews, further interviews should be conducted in order to shed light on these different experiences.vi.Fair dealing The researcher must ensure that a wide range of perspectives is incorporated into the study design and that the views of one group are not over-represented. Adopting a non-probabilistic sampling strategy is likely to ensure that this standard is met. Recruiting people for interviews such that the maximum variation in patient experiences is captured will act to ensure that the insights of one single person, or group, can be taken to be generalisable. Again, the strategy of ensuring data ‘saturation’ will assist in meeting this standard, as will the researcher taking steps to ensure that the views of carers and family members are captured, as well as just the experiences of the relevant patient group.The six quality standards articulated by Mays and Pope  are:i.TrEven when these standards have been followed in the generation of patient evidence, it is not clear that healthcare committees are best placed to ascertain whether the evidence presented to them meets these methodological and quality standards . It is important, therefore, that individuals with expertise in social research methodology are able to critique the patient evidence generated, prior to it being made available to the committee.In order to respond to valid concerns about standards and quality, there looks to be an important place for the independent appraisal of the evidence generated from patients’, carers’ and family members’ experiences. We recommend that a patient evidence review group made up of experts with knowledge and experience of social research methodologies is appointed to critique this evidence, such that the committee members can be reassured that the submissions from patient groups can be taken to be high quality evidence about patients’ relevant experiences.Given that the methodological guidance outlined above would support a range of different methodologies and methods, it is important that any review process enables the expert reviewers to enter into a critical dialogue with those responsible for generating this evidence. The researchers should be prepared to be able to articulate and defend the methodological framework within which they have sought to generate accounts of patients’ experiences.The patient evidence review group (or the committee member who possesses this expertise) should then report their views about the quality of this evidence to the healthcare committee. It should be clear to the committee members whether the evidence has been generated in ways that meet the standards of qualitative or mixed-methods inquiry, as it may well be the case that the committee members judge poor quality evidence to have less weight in the evaluation of the particular technology.Once an appropriate methodological framework, a clear set of quality standards, and a transparent evidence review process has been established, the healthcare committee can be confident that the patent evidence generated is valid, robust and relevant. If so, it will be fit for purpose to be considered alongside other clinical evidence necessary to flesh out the relevant criteria in making resource allocation decisions. When committee members are trained to recognise how different methodological approaches and research standards underpin the different kinds of evidence that they are required to consider, the patient evidence generated for their consideration should be able to be incorporated into the healthcare technology evaluation.Priority-setting committees make challenging decisions about the commissioning or provision of healthcare services based on reasoning supported by the best available evidence. The first part of the paper argued that these committees have an obligation to consider relevant criteria in the decision-making process. To support these criteria, evidence provides reasons for making particular decisions. While there is a strong practice of using clinical, epidemiologic, and economic evidence for these decisions, patient experience can provide important, relevant insight into the nature of patients’ need, the condition, and the treatment under consideration. The second half of the paper describes features that should be used to guide the generation of patient evidence in order to ensure its validity.The claim that patient evidence is necessary to support a fair process of decision-making has broad implications for all decisions made that affect patients’ access to treatment. Such decisions are regularly made by formal committees conducting health technology assessments, like those in the UK or Sweden, and by healthcare payers, such as the US insurance companies. Some of these bodies, like the UK’s NICE, have articulated processes to promote fairness in their decision-making, and patient evidence should be added to these decision-making procedures, barring the practical concerns about costs and implementation. For those healthcare committees that fail to meet the criteria for a fair process, it should be clear from the second half of the article that even they can make use of patient experience as a valuable form of evidence to inform decisions. Following through on the arguments presented in this paper requires a policy shift amongst priority-setting bodies towards commissioning evidence reviews of patient experience and including this evidence in their deliberations.The importance of patient evidence stands out particularly when other forms of evidence are scant. Patient evidence may provide a useful evaluation tool for new healthcare technologies that have limited clinical data to support claims of effectiveness. For example, in the case of ultra-orphan conditions, it may be near impossible to run adequately powered clinical trials. Robust patient evidence, generated via the framework we have described, could then be some of the most clearly articulated evidence available regarding the new technology.This paper has offered a new account of the need to include patient evidence in the decision-making process. This claim is not grounded in an argument about inclusiveness or a right of patients to express their views. Rather, the necessity of patient evidence in decision-making is grounded in the obligation to consider evidence relevant to the decision. Patient evidence is one such kind of evidence, and, if generated according to an appropriate methodological framework, it is essential information to the decision-making.