BMC Medical Research Methodology this month. It involves reviewers in choosing a conceptual model likely to be suitable for the question of the review, and using it as the basis of their initial coding framework. This framework is then modified in response to the evidence reported in the studies in the reviews, so that the final product is a revised framework that may include both modified factors and new factors that were not anticipated in the original model. 'Best fit' framework-based synthesis may be especially suitable in addressing urgent policy questions where the need for a more fully developed synthesis is balanced by the need for a quick answer.Framework analysis is a technique used for data analysis in primary qualitative research. Recent years have seen its being adapted to conduct syntheses of qualitative studies. Framework-based synthesis shows considerable promise in addressing applied policy questions. An innovation in the approach, known as 'best fit' framework synthesis, has been published in http://www.biomedcentral.com/1471-2288/11/29.Please see related article:  A practical benefit of doing this is that it enables questions or issues identified in advance by various stakeholders  to be explicitly and systematically considered in the analysis, while also facilitating enough flexibility to detect and characterise issues that emerge from the data.As the appetite for more holistic overviews of research evidence has grown, the last 10-15 years have seen increasing interest and activity directed at developing methods for synthesising qualitative studies . Framewoet al [BMC Medical Research methodology reports an interesting evolution of the framework-based synthesis approach.Framework analysis has become hugely popular as a way of conducting analysis of primary qualitative data, especially in areas of healthcare with policy relevance. A recent study, for example, used framework analysis in a study of mothers' interpretations of dietary recommendations . Becauseet al  publisheet al., including widening the scope of the review to include relevant topics identified by lay people, and the creation of data displays that could be viewed and assessed by people other than the primary analyst. Other examples of the application of the framework approach to the conduct of synthesis have now begun to appear, though it is dismaying that these and other syntheses of qualitative evidence continue to label themselves with the confusing, tautological, and inappropriate term 'meta-synthesis' [Framework analysis has previously been adapted for use in synthesis of qualitative evidence by Oliver and colleagues , who devnthesis' .BMC Medical Research Methodology manuscript [In the recent nuscript , CarrollCarroll and colleagues emphasise the advantages of this kind of approach when time is short and the demand for policy-relevant evidence is urgent. It enables focusing of the research on the priorities of those commissioning the work, while still leaving some room for finding the 'best fit' in the light of what the evidence actually reports. Of course, like framework analysis for primary research  there arFramework-based synthesis is an important advance in conducting reviews of qualitative synthesis. The 'best fit' strategy is a variant of this approach that may be very helpful when policymakers, practitioners or other decision makers need answers quickly, and are able to tolerate some ambiguity about whether the answer is the very best that could be given.The author declares that they have no competing interests.The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1741-7015/9/39/prepub


Prevention of Orthopaedic Implant Infection in Patients Undergoing Dental Procedures.” This evidence-based guideline ,detailed in 325 pages, has three recommendations and substitutes the previous AAOS guideline. The new published clinical guideline is a protocol to prevent patients undertaking dental procedures from orthopaedic implant infection. The guideline is developed on the basis of a collaborative systematic review to provide practical advice for training clinicians, dentists and any qualified physicians who need to consider prevention of orthopaedic implant (prosthesis) infection in their patients. This systematic review found no explicit evidence of cause-and-effect relationship between dental procedures and periprosthetic joint infection (PJI). The American Academy of Orthopaedic Surgeons (AAOS) and the American Dental Association (ADA), along with 10 other academic associations and societies recently (December 2012) published their mutual clinical practice guideline “This LTTE wishes to present a vivid summary of AAOS/ADA clinical practice guideline as a clinical update and an academic implementation to inform and assist Iranian competent clinicians and dentists in the course of their treatment decisions, to enrich the value and quality of health care on the latest international basis. Moreover, it indicates that practitioners should consider changing their customary practice of prescribing prophylactic antibiotics for dental patients inconclusive recommendation or not. The guideline emphasizes that patient preference should have a significant influencing role and practitioners should cautiously wait for future publications that elucidate the existing evidence to determine the balance between benefits and potential risk Consensus, indicating that expert opinion supports the guideline recommendation albeit the fact that no available evidence can encounter the inclusion criteria. The guideline emphasizes on the imperative role of patients’ preference in decision making and also affirms the flexibility of practitioners in deciding whether to follow a recommendation rated as Consensus or not. Consensus recommendations are the weakest form of recommendation and cannot be used to ignore recommendations developed from higher levels of evidence [This recommendation was graded as evidence -3. This recommendation conveys the maintenance of good oral hygiene and apparently, it is the only consensus recommendation in the new guideline. Oral hygiene measures are available and cheap, provide possible benefit, are consistent with current clinical practice and are in concordance with good oral health .Goals and Implications for Clinical PracticeThe rate of OI infection is recorded from 0.3% to 8.3% in the available published literature. Invasion of organisms into the surgical wound during the surgery, haematogenous spread, recurrence of infection in previously involved and infected joints, or propagation from an infective local source may produce such infection .Established on the best existing evidence, the rational for this clinical practice guideline is to assist the related clinicians and dentists to choose a paramount preventing and treatment modality when it is requisite. Contemporary dental practice inevitably depends on evidence-based standards and stipulates physicians and dentists to employ the best available evidence for treatment planning in their clinical practice. That’s why this guideline consists a systematic review of literature, conducted between October 2010 and July 2011 by AAOS and ADA methodologists and the doctor/dentist vocational groups and declared wherever the evidence was adequate or inadequate .They even discussed the gaps in the literature, where future researches are particularly needed [Jevsevar, an orthopaedic surgeon and chairman of the AAOS team, declared that this clinical practice guideline was not supposed to be an impartial document and he confirmed that clinicians should use it as an instructive tool in their treatment planning to improve the quality and efficacy of their health care .In summary, the guideline is deliberated to conduct clinical practice and also to provide a source of information for all qualified practitioners dealing with prevention of OI infection in dental patients. The AAOS and ADA hope that this guideline would also assist to ensure patients regarding the logics behind their treatment planning .http://www.aaos.org/research/guidelines/PUDP/PUDP_guideline.pdf and the ADA website: http://www.ada. org/sections/professionalResources/pdfs/PUDP_guideline.pdf (Therefore, The new guideline replaces the previous AAOS Information Statement and the full guideline with all succeeding credentials and workgroup declarations is available to access on the AAOS website: line.pdf .


Abandoning ineffective medical practices and mitigating the risks of untested practices are important for improving patient health and containing healthcare costs. Historically, this process has relied on the evidence base, societal values, cultural tensions, and political sway, but not necessarily in that order. We propose a conceptual framework to guide and prioritize this process, shifting emphasis toward the principles of evidence-based medicine, acknowledging that evidence may still be misinterpreted or distorted by recalcitrant proponents of entrenched practices and other biases. Divesting from ineffective and harmful medical practices has the potential to improve outcomes for patients, and mitigate the unsustainable rise in healthcare costs. Abandonment (de-implementation) of medical interventions may depend on multiple factors. Empirical evidence from well-designed studies should count, but other considerations such as inertia, financial and professional conflicts, cultural and societal values, knowledge brokering, and lobbying may also be very important eventually. The question is how we can position evidence so as to be more informative and influential in these complex processes. Here we provide a framework to guide the evidence-based de-implementation of interventions, acknowledging how on-the-ground realities can enter these considerations. Broadly, we will consider three categories of healthcare practices: those that are known not to work; those for which the evidence base is uncertain; and those that are in development and where strategic preemptive placement of evidence may help their eventual de-implementation, if needed. While the examples herein draw upon our experience appraising medical practices, the principles are broadly applicable to all healthcare fields.The number of medical practices where the best evidence shows no efficacy or harms outweighing benefits is substantial. One search produced over 150 potentially ineffective or unsafe practices .e.g. focusing on lesser endpoints, highlighting subgroup analyses, or performing additional studies with straw man controls. Expert-based meta-analyses with tailored eligibility criteria and outcome selection to show some benefit [When large, well-done randomized trials have contradicted current medical practice, de-implementation makes sense, but it can meet with fierce tactical resistance. Proponents of contradicted medical practices can procure not only editorials, but also counter-evidence that cuts corners,  benefit , and con benefit .Take for example the 2007 COURAGE study, which found that among patients with coronary artery disease and stable angina, routine percutaneous coronary intervention (PCI) was no better than an initial strategy of optimal medical treatment (OMT). In the month following the study’s publication, PCI and stenting was down 13% nationally; however, by 2010, those numbers returned to prepublication levels . Not onlMoreover, multiple meta-analyses of PCI co-authored by interventional cardiologists and/or sponsored by the industry claimed benefits in PCI by pooling together trials of stable angina and trials with residual ischemia, transferring the benefits of the latter population to the former. Practice guidelines then could follow the same path.i.e. considering de-implementation only after contact precautions are proven to be harmful, not merely ineffective. Such resistance to adhere to the best available evidence inflates healthcare costs, and may distract from alternative strategies with promising early results of efficacy, such as universal decolonization protocols [Another example is the routine use of gown and glove precautions among patients colonized with resistant pathogens, which is supported only by quasi-experimental, before-and-after, studies . Yet, torotocols .In brief, evidence wars can hinder de-implementation, and ‘practice resuscitation’ may be successful at reclaiming lost market shares. Of course, it is entirely possible that some subgroup truly benefits when a practice has ‘negative’ results globally, or that some lesser endpoints are meaningful to patients. Yet, more often such counter-evidence resurrection studies simply create excuses to not abandon the contradicted practice.We propose a simple standard to curb such practice resuscitation: The evidence to revive a contradicted medical practice (whether in part or in whole) should involve endpoints and controls at least as rigorous as the contradictory study. Until such evidence is obtained, payers may offer disincentives by placing restrictions to reimbursement, and regulators may consider revoking or restricting prior approvals.Clearly contradicted practices are less common than unproven ones. Among 1,344 articles assessing a medical practice, 363 27%) tested standard of care, with 146  contradicting it 7% tested, and 48%A rational strategy to de-implement medical practices supported by little to no evidence is to subject them to testing in systematic fashion. Ideally, this assessment would be performed under the auspices of non-conflicted bodies, possibly within existing governmental structures, such as the Agency for Healthcare Research and Quality. Table With multiple novel interventions  being introduced in medical care, a key consideration is to take preemptive steps that would allow efficient de-implementation if the intervention eventually proves inefficient and harmful. While there is increasing pressure to adopt novel interventions before substantial evidence has been obtained on them, one method to curb the spread of ineffective practices is to restrict their use prior to widespread dissemination, as demonstrated by the case of percutaneous transluminal angioplasty and stenting (PTAS) for intracranial stenosis.In 2005, the Wingspan intracranial artery stent was granted humanitarian device exemption from the US Food and Drug Administration, based upon provisional data that it could improve intracranial artery lumen diameter in patients with stenosis refractory to medical therapy . HoweverIn 2011, the only randomized study of the device, the SAMMPRIS trial, found that PTAS among patients with a recent transient ischemic attack (TIA) or stroke and documented stenosis of a major intracranial artery nearly tripled the 30-day risk of stroke or death compared to optimal medical management (14.7% versus 5.8%) .During the years it was approved but trial data was lacking, CMS’s policy dramatically limited off-protocol use of the device and effectively protected the public. Altogether, only a few hundred patients received the device (200+ treated on protocol) in the US  — contraThe lesson of PTAS is that higher upfront standards have potential to protect patients from ultimately flawed care. Unfortunately, regulatory agencies appear to move increasingly in the opposite direction, notably with the creation of the FDA’s ‘breakthrough’ designation , and emeThe opinions that we express here may well be biased. We believe, however, that there is no reason that experimental studies cannot be leveraged to provide clarity for health policies with broad societal repercussions. To date, regulatory policies have been based on theory or scant retrospective observational studies, but at least some policies may be tested creatively with randomized controlled trials . For insDe-implementing practices reflects a recommitment to evidence-based healthcare. This is important for medications, devices, procedures, behavioral or psychological interventions, screening and diagnostic tests, and any other intervention undertaken by people in the health professions. Strategies to eliminate ineffective and harmful practices may help contain healthcare spending and optimize outcomes. Ideally, the majority of medical decisions should be supported by robust data, with ambiguous decisions made only within the confines of ongoing studies. However, as we stated, rational, quantitative evidence may not necessarily be the only or even main factor driving healthcare decisions. Research to understand better the other, cognitive or political factors that facilitate or hinder de-implementation is thus also warranted (see Box 1).of Implementation Science invited this editorial following a consultation with our editorial team and Editorial Board. They identified “de-implementation“ as an important theme, which deserves more attention than it currently receives. We regard de-implementation broadly as “stopping practices that are not evidence-based”. We encourage further papers on this theme and will include these in a special article series in the journal to enhance their visibility. All submissions will be reviewed and handled according to our normal procedures. In addition, we welcome and encourage comments in response to the accompanying editorial, using the comment feature of the journal’s platform. These comments, while moderated, are intended to stimulate discussion and debate within the implementation research community. In both papers and comments, we welcome a range of perspectives and rigorous studies on the theme of de-implementation, including (but not limited to) contributions that cover psychological, organizational or economic factors. We intend to promote other themes in the future.The Editors-in-Chief The authors declare that they have no competing interests.VP and JPAI drafted the manuscript. VP and JPAI edited the manuscript for intellectual content. JPAI provided supervision. Both authors read and approved the final manuscript.


Despite collection of patient reported outcome (PRO) data in clinical trials of antiepileptic drugs (AEDs), PRO results are not being routinely reported on European Medicines Agency (EMA) and Food and Drug Administration (FDA) product labels. This review aimed to evaluate epilepsy-specific PRO instruments against FDA regulatory standards for supporting label claims. Structured literature searches were conducted in Embase and Medline databases to identify epilepsy-specific PRO instruments. Only instruments that could potentially be impacted by pharmacological treatment, were completed by adults and had evidence of some validation work were selected for review. A total of 26 PROs were reviewed based on criteria developed from the FDA regulatory standards. The ability to meet these criteria was classified as either full, partial or no evidence, whereby partial reflected some evidence but not enough to comprehensively address the FDA regulatory standards. Most instruments provided partial evidence of content validity. Input from clinicians and literature was common although few involved patients in both item generation and cognitive debriefing. Construct validity was predominantly compromised by no evidence of a-priori hypotheses of expected relationships. Evidence for test-retest reliability and internal consistency was available for most PROs although few included complete results regarding all subscales and some failed to reach recommended thresholds. The ability to detect change and interpretation of change were not investigated in most instruments and no PROs had published evidence of a conceptual framework. The study concludes that none of the 26 have the full evidence required by the FDA to support a label claim, and all require further research to support their use as an endpoint. The Subjective Handicap of Epilepsy (SHE) and the Neurological Disorders Depression Inventory for Epilepsy (NDDI-E) have the fewest gaps that would need to be addressed through additional research prior to any FDA regulatory submission, although the NDDI-E was designed as a screening tool and is therefore unlikely to be suitable as an instrument for capturing change in a clinical trial and the SHE lacks the conceptual focus on signs and symptoms favoured by the FDA. Epilepsy is defined by the recurrence of spontaneous/unprovoked seizures and covers a range of clinical situations in terms of age of onset, type of seizures, aetiological background, resulting handicap, prognosis, and response to treatment . EpilepsThe assessment of efficacy in clinical studies evaluating AEDs is generally focused on seizure frequency/occurrence in line with regulatory guidelines . HoweverIn order to guide the selection of PROs for epilepsy clinical studies, there have been numerous reviews evaluating PROs, each review with a unique focus: evaluating PROs designed to measure change in seizure severity , overvieDespite almost routine collection of PRO data in clinical trials of AEDs and the wide choice of PRO instruments, PRO results are not being routinely reported on European Medicines Agency (EMA) and Food and Drug Administration (FDA) product labels. A review of the current state of PRO label claims granted for new molecular entities approved 2006–2010  found twThere is limited guidance from regulatory authorities on incorporating PROs into evaluations of AEDs, the recent EMA guideline on clinical investigation of medicinal products in the treatment of epileptic disorders  suggestsStructured Embase and Medline searches were conducted in February 2011 to identify epilepsy studies reporting the use or development of PRO instruments in epilepsy samples. Searches were constructed by combining epilepsy and seizure subject heading search terms with a number of PRO measurement terms. Searches were limited to ‘humans’, ‘English language’ and ‘year 2000 –current’. Full search terms are shown in the Additional file Abstracts of publications identified by these searches were screened to identify PRO instruments used. Full text versions of publications were screened to identify specific instruments where abstracts referred to PRO measurement without naming a specific instrument, or where abstracts specified endpoints that may have been captured by PRO measurement. Further electronic searches were conducted for each instrument identified to retrieve a copy of the instrument and establish its relevance. Given the wide variety of PRO instruments used with epilepsy patients, the following instrument inclusion criteria were used to focus the review on broadly comparable instruments that might be used to evaluate pharmacological treatment:•Epilepsy-specific scale. Based on the FDA’s preference for more proximal endpoints that are specifically relevant to the target population, generic PROs which make no reference to epilepsy were excluded.•Adult patient completed. Most epilepsy-specific PROs have been developed for adult patient completion. Scales developed to assess epilepsy impacts for children are commonly designed for proxy completion  in part or whole and are not directly comparable to PROs designed for adult patient completion. PROs designed for completion by adolescent as well as adult patients were included.•Target a concept that could potentially be impacted by pharmacological treatment. Non-interventional studies or those evaluating non-pharmacological interventions may also use PROs to evaluate concepts that would not be impacted by pharmacological treatment. However, PROs evaluating these concepts would not need to meet FDA requirements for supporting a label claim.Further literature searches were then conducted in Embase and Medline to identify published validation work on each PRO instrument identified. Full text versions of all published reviews of PRO instruments identified by the initial search and instrument focused literature searches were retrieved and checked for any additional instruments or validation work. Any instrument that was found to comprise single items that did not form a scale and/or for which no documentary evidence of validation could be located was excluded from further review.Detailed review criteria were developed from requirements laid out in the FDA guidance for PROs supporting label claims . These cInitial Embase and Medline searches identified a total of 1854 publications (after de-duplication) from which 159 PRO instruments were identified by abstract or full-text manuscript review. Following instrument retrieval and initial review, 133 of these were excluded as they were found to be not epilepsy specific (n = 52), not for adult patient completion (n = 42), not to target a concept that could be potentially impacted by pharmacological treatment (n = 20) or instruments comprising single items that did not form a scale and/or for which no documentary evidence of validation could be located (n = 19). Many of these instruments could be excluded on more than one criterion. Numbers shown here depict the first reason for exclusion.Twenty-six 26) epilepsy-specific PRO instruments were identified and reviewed, Table 6 epilepsnow (EPI-QOL) to six months (SHE), with five PROs referring to multiple recall periods within the same PRO . Response options for most of the reviewed PROs employ a Likert approach with 3–5 options, for example the Epilepsy Psycho-Social Effects Scale has a five point Likert scale ranging from ‘almost always or always’ to ‘never’ and the EPI-QOL has a five point response scale ranging from ‘very frequently’ to ‘not at all’. Less common is a numerical rating scale (NRS) approach with anchors, such as the ESES which has a 11 point scale ranging from 0  to 10 (sure I can do).Instruments vary in terms of length, ranging from five items  to 132 items (WPSI). Nine of the PROs do not have a defined recall period , two refer to the time of an epilepsy attack . Most specify a recall period varying from The PROs provide different levels of information depending on the scoring approach, a minority of instruments provide only a total score based on the scoring of all items in the PRO , whilst most provide a more detailed amount of information through provision of sub-scale scores , with the number of sub-scale scores ranging widely, from two  to 20 ). Most instruments provide between 5–15 subscale scores.Table The authors did not find any published conceptual framework for any of the reviewed instruments.Three of the PROs involved patients in concept elicitation/item generation and in the evaluation of items through a cognitive debrief methodology or similar approach: the EFA, LSSS and SHE. Of these, the LSSS and SHE also had documented evidence that literature had been used to guide the instrument development and/or clinical experts were involved. Most of the PROs (n = 14) had partially involved patients in the development of the instrument, in most cases either concept elicitation OR cognitive debrief were undertaken, but not both methodologies. All but one of these 14 instruments (LAEP) had also reviewed literature and/or involved clinical experts in the instrument development. Of the 26 instruments only five did not involve literature and/or clinical experts in the instrument development process according to published information.Seven instruments had full evidence of the construct validity of the PRO by providing hypotheses of the expected relationships between the PRO under evaluation and other clinical or PRO measures, and the hypotheses being supported by reported results. Most of the PROs only had partial evidence of construct validity (n = 16), in most cases there was no evidence of hypotheses of expected relationships being developed in advance of analysis. For some instruments this was compounded by limited testing e.g. only known groups validity was evaluated or very limited comparisons were made. The Perceived Limitations Scale, Perceived Stigma Scale and SIDAED did not have any available evidence of construct validity.Five PROs had sufficient evidence of test-retest reliability, and four PROs had sufficient evidence of internal consistency reliability, with two PROs providing sufficient evidence of both types of reliability: LAEP and NDDI-E. Thirteen PROs had partial evidence of test-retest reliability and 14 had partial evidence of internal consistency. Most of the PROs that had only partial evidence of these measurement properties had a mix of results for sub-scales in terms of achieving the required criteria , Perceived Stigma Scale, Perceived Limitations Scale and IES. The IES has an internal consistency of 0.65 which increases to 0.82 if one of the 8 items is removed, but later publications are based on either the eight item version or a 10-item version ,87 for wFour PROs had full evidence of ability to detect change and reported the results of analysis undertaken with the specific aim of testing within-group responsiveness for all sub-scales as well as total score (as appropriate): ESI-55, Performance, subjective evaluation and socio-demographic data (PESOS), Quality of Life in Epilepsy 31 (QOLIE-31) and Quality of Life in Epilepsy 89 (QOLIE-89). Three PROs provided partial evidence of ability to detect change: LSSS, QOLIE-10 and WPSI; for example analysis was conducted on a previous version of the PRO (LSSS) or a non-empirically derived scale structure . For most of the PROs (n = 19) this measurement property had not been investigated.The LAEP was the only PRO that had fully documented evidence of the minimally important difference (MID) of the scale. This was evaluated in a study that was designed to assess the magnitude of change in the LAEP and other PROs in order to exclude chance or error at various levels of certainty in patients with medically refractory epilepsy through application of a Reliable Change Index analytic approach . Two PROThis review sought to evaluate epilepsy-specific PRO instruments against FDA regulatory standards and to identify gaps in the development process of the instruments that would need to be addressed prior to regulatory review. Twenty-six (26) epilepsy-specific PRO instruments were identified and reviewed.This review identified that the SHE and NDDI-E met more of the regulatory requirements in terms of measurement properties, with both scales meeting four of the eight measurement properties evaluated. These two PROs lacked a published conceptual framework and require further evidence of ability to detect change and interpretation of change. In addition, the NDDI-E requires further evidence of patient input and the SHE requires further evidence of internal consistency reliability. Internal consistency fell below the required standard (α ≥0.8)  for someThe LAEP, ICI and ESI-55 all met requirements for three of the eight measurement properties. The LAEP in terms of test-retest reliability, internal consistency and interpretation of change; the ICI in terms of literature and clinician input, construct validity and internal consistency; and the ESI-55 in terms of literature and clinician input, construct validity and ability to detect change. Therefore all three instruments lack a published conceptual framework and required further evidence in terms of patient input, as well as specific gaps for each PRO.When considering PROs with at least partial evidence of measurement properties, a different group of PROs come to the forefront: the QOLIE-31 and QOLIE-89 met or partially met requirements for seven out of the eight measurement properties, and the LSSS met or partially met requirements for six of the measurement properties. However, of concern for the QOLIE-89 is that seizure free patients did not score significantly higher than the most severe group on nine of the sub-scales  and testOf particular concern from a regulatory perspective are the Perceived Limitations Scale, Perceived Stigma Scale and the SIDAED, all of which failed to achieve even partial evidence for any of the eight measurement characteristics. Notably, the Perceived Limitations Scale and Perceived Stigma Scale failed to achieve required standards for internal consistency when tested (no evidence of testing for this for the SIDAED). No other psychometric properties were tested for these three PROs.Content validity is defined by the FDA as ‘… the extent to which the instrument measures the concept of interest’, with evidence being supported through the conduct of qualitative studies to demonstrate that the items and domains of an instrument are appropriate and comprehensive relative to its intended measurement concept, population and use . Importaa priori hypotheses are tested concerning logical relationships that should exist with measures of related concepts or scores. This was an area where most PROs failed to meet the criteria in this review. It is entirely possible that hypotheses were set in advance of analysis, but not reported in the published manuscript and it is strongly encouraged that this information is shared in publications reporting the psychometric properties of PROs. In terms of test-retest reliability and internal consistency, it is necessary to provide evidence for all domains and total scores (if applicable). Again this was a common area where PROs failed to meet the criteria in this review.In terms of evidence gathered through psychometric testing, the review identified common pitfalls. In relation to testing for construct validity, the FDA require that Although several PROs had evidence from clinical studies that the PRO measured change in an epilepsy population, few PROs had evidence that ability to detect change had been specifically investigated. One likely reason for this is that this measurement property is harder to evaluate in terms of study design as it means undertaking a longitudinal study of a patient group expected to improve/deteriorate. However, this is an essential measurement property, particularly when considering that the PROs are to be incorporated into clinical studies designed to test the effectiveness of a treatment for epilepsy.Even fewer PROs had evidence for interpreting change or empirically derived responder definitions. The responder definition may vary by target population or other clinical trial design characteristics and therefore the FDA will evaluate a PROs responder definition in the context of each clinical trial. Evidence towards a PROs MID can contribute to the responder definition, and so this review sought to identify any evidence of statistically derived or anchor-based MID and/or responder definition, and found that only the LAEP was able to demonstrate empirically derived evidence of the measurement property.One reoccurring problem found through the course of this review is that of several versions of one PRO instrument. Great care has to be taken when evaluating published evidence on the development and psychometric validation of a PRO instrument that the same version of the PRO is being referred to. By way of example, the ESES was originally developed to be a 25-item instrument, with evidence for content validity, construct validity and internal consistency . HoweverIt is vital to consider these findings in the broader context of PRO label claims in the US. The review of the current state of PRO label claims granted for new molecular entities approved 2006–2010  found thIt is worth considering the extent to which the evaluated PROs would be suitable to support regulatory approval of medicines in Europe through a regulatory review conducted by the EMA. The EMA has been less prescriptive in terms of their requirements for PROs, with one reflection paper published in 2005 for HRQL instruments, but nothing extended more generally to PROs. In the absence of clear guidelines from the EMA, it is difficult to determine which of the evaluated PROs would be well received by the EMA. However, whilst both the FDA and EMA require PROs supporting regulatory approvals to be validated and reliable, the EMA is more likely to accept and encourage the use of well-known, commonly used PROs than the FDA. The EMA also places less emphasis on qualitative evidence of content validity. Therefore PROs that are likely to be well received by the EMA will need to have demonstrated evidence of the psychometric properties of the PROs, particularly where this evidence is published in peer-review publication.A limitation to this research is that the review was conducted on published information. It is frequently the case that documentation on the development process for PRO instruments is not published, particularly for older PRO instruments which were developed at a time where there were less publication options for PRO development manuscripts. It is likely that there are more details on the development of the reviewed instruments that have not been considered in this review. An important step for anyone considering the use of the PRO to support a label claim is to contact the instrument developer to see if further information can be made available to address seeming gaps in the evidence, which if available will reduce the need to conduct further research to gather evidence in support of the PRO instrument.This systematic review of 26 epilepsy-specific PRO instruments, evaluated to the standards set out in the FDA guidance  indicateAEDs: Antiepileptic drugs; ABNAS: Aldenkamp-Baker neuroassessment Schedule; BPSE: Bonner psychsoziale skalen fur epilepsie; EFA: Epilepsy foundation of America; EMA: European Medicines Agency; EPSES: Epilepsy psycho-social effects scale; ESES: Epilepsy self-efficacy scale; ESI-55: Epilepsy surgery inventory 55; FDA: Food and Drug Administration; HRQL: Health related quality of life; ICI: Ictal consciousness inventory; IES: Impact of epilepsy scale; LAEP: Liverpool adverse events profile; LSSS: Liverpool seizure severity scale; MID: Minimally important difference; NDDI-E: The neurological disorders depression inventory for epilepsy; NEWQOL: Quality of life in newly diagnosed epilepsy; NRS: Numerical rating scale; PESOS: Performance, subjective evaluation and socio-demographic data; PNS: Portland neurotoxicity scale; PRO: Patient Reported Outcome; QOL: Quality of life; QOLIE: Quality of life in epilepsy; QOLIE-10: Quality of life in epilepsy 10; QOLIE-31: Quality of life in epilepsy 31; QOLIE-89: Quality of life in epilepsy 89; SEALS: Side effect and life satisfaction; SHE: Subjective handicap of epilepsy; SIDAED: Assessing side effects of AED treatment; SSQ: Seizure severity questionnaire; WHO: World Health Organisation; WPSI: Washington psychosocial seizure inventoryThe authors are employees of Oxford Outcomes, an ICON Plc. company. Oxford Outcomes received payment from Eisai for conducting the review and writing the manuscript. Eisai paid the article-processing charge.AN: Study design, contributed to designing the search strategy, reviewed abstracts, developed the review criteria, reviewed the PRO instruments according to the criteria, led on manuscript development. CK: Study design, designed the search strategy, reviewed abstracts, developed the review criteria, reviewed the PRO instruments according to the criteria, contributed to manuscript development. KB: Contributed to designing the search strategy, reviewed abstracts, developed the review criteria, reviewed the PRO instruments according to the criteria, assisted with manuscript development. DW: Reviewed PRO instruments, reviewed the manuscript content. All authors read and approved the final manuscript. Listing of search terms used through Embase and Medline.Click here for file


The balanced application of the evidence  is the central point of practicing evidence-based orthopedics (EBO) and involves, according to evidence-based medicine principles, integration of our clinical expertise and judgment with patients’ perceptions and societal values, and with the best available research evidence.1EBO involves careful attention to the design, statistical analysis, and critical appraisal of clinical research. The delineation between “outcomes” research and “evidence-based medicine” is vague. Since the term evidence-based medicine was coined first at McMaster University, orthopedic surgeons and researchers have adopted their own style of critical appraisal often coined as “evidence-based orthopedics (EBO)” using a clear delineation of relevant clinical questions, a thorough search of the literature relating to the questions, a critical appraisal of available evidence and its applicability to the clinical situation, and a balanced application of the conclusions to the clinical problem.12Gaining momentum from the global EBM movement, the concepts and ideas attributed to and labelled collectively as EBO have become a part of daily clinical lives, and surgeons increasingly hear about evidence-based guidelines, evidence-based care paths, and evidence-based questions and solutions. The controversy has shifted from whether to implement the new concepts to how to do so sensibly and efficiently, while avoiding potential problems associated with a number of misconceptions about what EBO is and what it is not. The EBO-related concepts of hierarchy of evidence, meta-analyses, confidence intervals, study design are so widespread that health care providers willing to use today’s medical literature with understanding have no choice but to become familiar with principles and methodologies of evidence-based orthopedics.Indian Journal of Orthopaedics is highlighting a new section entitled, “Evidence-Based Orthopaedics: Tips for Clinical Practice” for its readers. This novel section of the journal will publish concise and practical tips to reviewing the orthopedic literature. The aim is, of course, to empower the journal readership with the tools necessary to become efficient in their critical appraisal techniques of the published literature. Simply put, this section will assist readers in generating better evidence as well as understanding the validity of published papers. We use a 3-min checklist approach to allow our readership to critically evaluate the reliability of a given study in a few short minutes. A further enhancement in the Journal includes an “Evidence Scan”, a section aimed to provide an evidence summary of one or two highly important recent publications in our field.Understanding the ongoing need to adapt to evidence-based approaches and the success of our 2008 symposium on evidence-based medicine,Indian Journal of Orthopaedics one step closer to its goal among the top sources of orthopedic evidence.We hope that both additions will serve to engage readers and move the


The rapid advancements in the field of genome sequencing are aiding our understanding on many biological systems. In the last five years, computational biologists and bioinformatics specialists have come up with newer, better and more efficient tools towards the discovery, analysis and interpretation of different genomic variants from high-throughput sequencing data. Availability of reliable simulated dataset is essential and is the first step towards testing any newly developed analytical tools for variant discovery. Although there are tools currently available that can simulate variants, none present the possibility of simulating all the three major types of variations  and can generate reads taking a realistic error-model into consideration. Therefore, an efficient simulator and read generator is needed that can simulate variants taking the error rates of true biological samples into consideration.Snp, Indel and Cnv) an open-source variant simulator and read generator capable of simulating all the three common types of biological variants taking into account a distribution of base quality score from a most commonly used next-generation sequencing instrument from Illumina. SInC is capable of generating single- and paired-end reads with user-defined insert size and with high efficiency compared to the other existing tools. SInC, due to its multi-threaded capability during read generation, has a low time footprint. SInC is currently optimised to work in limited infrastructure setup and can efficiently exploit the commonly used quad-core desktop architecture to simulate short sequence reads with deep coverage for large genomes.We report SInC (http://sourceforge.net/projects/sincsimulator.We have come up with a user-friendly multi-variant simulator and read-generator tools called SInC. SInC can be downloaded from The rapid advancements in the field of genome sequencing is aiding our understanding of genome organisation in many biological systems-3. TheseAlthough, next-generation sequencing (NGS) instruments generate reads of various lengths and with varying error profiles, the most popular source of data remains sequencing instruments from Illumina, which employs a sequencing-by-synthesis chemistrCurrently available tools can either generate platform-specific, error-profile based reads or simulate reads across platforms-28. It iHere we present an evaluation of SInC using commonly used SNV, indel and CNV detection tools. The speed, accuracy and efficiency was compared against other popular simulators and read generators.SInC performs two jobs; first it simulates variants (simulator) and then it generates reads (read generator). SInC simulator consists of three independent modules  that can either be executed independently in a mutually exclusive manner or in any combination.DAvg) between SNVs is calculated . Then the algorithm applies a user-defined transition to transversion (Ti/Tv) metric to maintain the biological significance of the SNVs across the genome. A Ti/Tv ratio of 2.1 was maintained across the population of simulated SNVs with 20% inherent heterozygosity to simulate human genome data as previously reported. The floInsertion and deletion (indel) events have a wide range of size-based variability. SInC simulator simulates short, medium and large indels in the range of 1-10 bp 11-20 bp and 21-100 bp respectively in concordance with earlier studies. The ratThe algorithm first randomly generates the position for indels and then uses a filter to replace any indel within the region of the N-region of the genome  with one in the sequenced region. To remove duplicates, the simulated indels are coordinate sorted and only the unique locations are retained. Usually, a redundancy of 2-5% is filtered out post coordinate sorting (that can result from either duplicates or un-sequenced regions). Hence, an additional 5% of indels are generated at the initial stage of the algorithm to account for the loss of indels at the duplicate removal step. In the next stage of the algorithm, the frequency of short, medium and large indels was factored in based on previous literature evidence for their distribution in human. The indThe CNV simulation constitutes the final step of the simulation algorithm, as it can ply in a sequential manner post indel simulation. Since the input files from indel simulation may contain heterozygous indels, which may be of unequal size, hence the CNV module takes it into consideration and prevents the possibility of boundary overlaps with indels. The flow chart in Figure Unlike the indel module, here the size and location of the variants are both generated dynamically with the flow of the program after obtaining the feed from the user to determine the number of CNVs and their range of size distribution (upper and lower limit). Such simulated data is particularly useful to test the accuracy and sensitivity of a new or existing CNV caller across a wide range of CNV sizes. The algorithm then filters the simulated CNVs based on its coordinates. First the span of each of the CNVs are evaluated to ensure correspondence with chromosomal boundary in either allele and subsequently the CNV boundaries are checked for overlap with neighboring CNVs. The CNV is logged and the next iteration of location and size are generated upon meeting the aforementioned conditions. Unlike the SNV and indel simulation modules, the annotation data for both alleles in the CNV module is stored in the same file in the form of a tabular data. The tool also outputs a simplified results file (similar to a BED file), which can be read easily by any program for visualizing CNVs.SInC has a read generator part that generates short reads using a multi-threaded approach utilizing the parallel processing power of commonly used quad-core desktop/laptop architecture. The process of read generation uses a quality profile distribution-based error-model. We have used publicly available 100 bp read pair data derived by using Illumina instrument from the SRA database to generate the quality profile distribution and assessed the quality distribution for both forward and reverse reads of the training set and data post read generation  without inordinate expense of time. SInC can utilize 1 to 4 threads for optimal function. Our comparison was set up based on default use of 1 core ranging upto a maximum utilization of 4 cores in SInC versus the other tools  and generate sequencing reads taking into account a realistic platform-specific quality-profile of an sequencing instrument is currently lacking. We tried filling this void by designing a versatile and fast tool that can generate multiple types of biological variants  and can run on a minimalistic quad core desktop computer using multi-threaded option. The time advantage obtained in SInC could be attributed to the optimized algorithms and efficient use of C thread functions to manage the I/O streams. This advantage is also obvious in a single core, which delegates the bulk of the data generation to multiple threads to ensure efficient use of memory in line with “divide and conquer” approach. The optimization of multiple core usage is available upto 4 cores in quad-core architecture.Another major functional advantage of this tool is its ability to simulate CNVs. CNVs have been shown to contribute more towards genetic diversity than SNVs and are conspicuous by their pervasiveness in human genome-39. The Production of large amount of heterogeneous data in high-throughput biology requires sophisticated computational tools for efficient analysis, storage, sharing and archiving. This requires resources, both software and hardware, and interoperability of computational resources. A common practice among computational biologist is to use simulated data to test the efficacy of the tools before applying them to real dataset. Although there are many simulators available currently, there is none that suits the need of every computational biologist wanting to make tools for short-read sequence data. Keeping this in mind, we have developed a tool to help computational biologists create simulated datasets using only one simulator that can span across sequencing platforms and variant types . Although, SInC simulator was tested with human genome, it is versatile to address the complexity of any genome, its substitution rate, variant frequency and transition to transversion ratio. Large genomes, like that from many plants, need time to generate simulated reads at high coverage and this is where the multi-threaded capability of SInC scores high in comparison to other tools. By using a standalone quality-score distribution model of real dataset, SInC provides an opportunity to individual user to generate reads at different read lengths but with realistic quality.We report a tool called SInC that can simulate and generate short sequence reads with different types of biological variants. The ability of SInC to generate realistic fastq reads based on Illumina read quality profiles along with its capacity to simulate multiple biological variants and generate reads concurrently makes it a powerful option in a variety of simulation studies and a part of computational biologists’ essential toolkit.Project name: sincsimulatorProject home page:http://sourceforge.net/projects/sincsimulatorOperating system(s): LinuxProgramming Language: COther Requirements: GNU Scientific Library(gsl library), pthreads libraryLicense: Creative Commons Attribution Non-Commercial License V2.0Any restrictions to use by non-academics: License neededSNP: Single nucleotide polymorphism; Indel: Insertions and deletions; CNV: Copy number variations.Both SP and BP are paid by Strand Life Sciences. The authors declare that they have no other competing interests.BP conceived the project and come up with the tool’s parameters. SP designed the analytical workflow and fixed bugs in the code. AAR coded the first version of the tool. SG helped in filling parts of the code, fixed bugs and tested the tool. SP and BP wrote the manuscript. All authors read and approved the manuscript.SInC SNP distribution. A) A Gaussian distribution was implemented to dynamically allocate distance between two SNVs. Under default conditions, which follows a SNV rate of 0.001, the mean distance, DAvg, between two SNVs was set to 1000 as evidenced by studies from 1000 genome project. Also, a lower limit of DAvg was set to 300 based on these studies allowing us to dynamically simulate SNVs of biological relevance. B) Normalized frequency distribution of simulated SNVs per chromosome in hg19 assembly.Click here for fileTime profiles of SInC, and variant re-discovery numbers. Time elapsed to perform one complete simulation with default options using 1–4 cores A) For chromosome 22 at 15X B) For human whole genome (hg19) at 5X. SNPs were re-discovered using GATK and indels with Pindel.Click here for fileSInC indel distribution. A) The size based frequency distribution of indels used in SINC based on literature evidence from Millis et al. B) Normalized frequency distribution of simulated indels per chromosome in hg19 assembly.Click here for fileIllumina-derived base quality score distribution used to generate reads by SInC. Quality score distribution of reads from training sets vs reads simulated using SInC; A) for forward read B) for reverse reads. Top panel: training set, bottom panel: reads simulated using SInC.Click here for fileScripts used to run various tools.Click here for fileCoverage verses SNP re-discovery rate. Effect of coverage on combined process of simulation + read-generation + variant calling and re-discovery.Click here for file


Systematic and transparent approaches to priority setting are needed, particularly in low-resource settings, to produce decisions that are sound and acceptable to stakeholders. The EVIDEM framework brings together Health Technology Assessment (HTA) and multi-criteria decision analysis (MCDA) by proposing a comprehensive set of decision criteria together with standardized processes to support decisionmaking. The objective of the study was to field test the framework for decisionmaking on a screening test by a private health plan in South Africa.Liquid-based cytology (LBC) for cervical cancer screening was selected by the health plan for this field test. An HTA report structured by decision criterion  was produced based on a literature review and input from the health plan. During workshop sessions, committee members 1) weighted each MCDA decision criterion to express their individual perspectives, and 2) to appraise LBC, assigned scores to each MCDA criterion on the basis of the by-criterion HTA report.Committee members then considered the potential impacts of four contextual criteria on the use of LBC in the context of their health plan. Feedback on the framework and process was collected through discussion and from a questionnaire.For 9 of the MCDA matrix decision criteria, 89% or more of committee members thought they should always be considered in decisionmaking. Greatest weights were given to the criteria "Budget impact", "Cost-effectiveness" and "Completeness and consistency of reporting evidence". When appraising LBC for cervical cancer screening, the committee assigned the highest scores to "Relevance and validity of evidence" and "Disease severity". Combination of weights and scores yielded a mean MCDA value estimate of 46% (SD 7%) of the potential maximum value. Overall, the committee felt the framework brought greater clarity to the decisionmaking process and was easily adaptable to different types of health interventions.The EVIDEM framework was easily adapted to evaluating a screening technology in South Africa, thereby broadening its applicability in healthcare decision making. In any country, healthcare resource allocation decisions are complex and involve assessment of the available scientific evidence, clarification of priorities, value judgments and ethical considerations ,2. In deIn the healthcare sector of South Africa, current decisionmaking approaches are centered around evidence-based medicine, affordability and, where available, cost-effectiveness/costutility analysis (CEA/CUA) . IncreasThus, there is a need for a process that supports consideration of all dimensions impacting a decision in a systematic and explicit fashion, and increases transparency and access to the evidence upon which decisions are based. Multi-criteria decision analysis (MCDA) is a tool to support complex decisionmaking which allows a structured, objective consideration of factors that are both measurable and value-based in an open and transparent manner ,13,14.While MCDA has been used historically in sectors such as transport or agriculture, there is a growing interest in using and applying the principles of MCDA, and similar approaches based on multiple decision criteria, to resource allocation decisionmaking in health care -21. The A proof-of-concept study of the EVIDEM framework was performed in Canada involving a diverse panel of stakeholders appraising 10 medicines . The corThe objective of the current study was to expand the scope of field testing both geographically and with respect to type of intervention: i.e., to field test the framework as a support for coverage decisionmaking on a cervical cancer screening test  by a private health plan in South Africa.The healthcare system in South Africa is dichotomous with a small (approximately 7 million lives) but resource-rich private healthcare sector and a growing (approximately 45 million lives) resource-scarce public sector. The private healthcare sector is largely funded by medical insurance companies , while the public sector is currently funded out-of-pocket or by the government. With respect to cervical cancer screening, the Department of Health guidelines stipulate that all women should be screened three times in their lifetime from the age of 30 years . In the The EVIDEM framework was field-tested in the private healthcare sector of South Africa with a clinical policy and decisionmaking committee of a major health plan. This panel of experts included doctors , pharmacists and nurses with at least 12 months experience in decision-making at a health policy level for the health plan. At the time of the study, the committee was already using evidence-based medicine, cost-effectiveness, affordability and some epidemiological principles in a multicriteria-based decisionmaking process.The study design is presented in Figure During workshop sessions, committee members assigned: 1) weights to each criterion of the MCDA matrix to express their perspectives; 2) scores for LBC for each criterion of the MCDA matrix based on data from the by-criterion HTA report; and 3) the qualitative impact of system-related criteria on the appraisal. The adoptability and utility of the EVIDEM approach were explored through a post-testing survey.Relevant data for each decision criterion of the framework was identified by searching PubMed, websites of HTA agencies  and the website of South Africa's Department of Health.Data on the projected budget impact of LBC as well as local costing data was obtained from the health plan. The health plan also provided input for the four contextual criteria, which were appraised qualitatively. Collected data was extracted, analyzed and synthesized to produce a by-criterion HTA report.The quality of the clinical evidence and the economic evaluation was assessed using previously developed instruments according to two quality criteria: "Completeness and consistency or reporting" and "Relevance and validity of evidence" . The HTATo explore their individual values, committee members were asked to assign weights to each MCDA decision criterion on a scale of 1 to 5, where 5 represents the greatest importance to their decisionmaking. Weights were to be assigned from individual perspectives in the context of the health plan, independent of the intervention to be appraised.All committee members were asked to familiarise themselves with the study intervention  using the EVIDEM Synthesised evidence report prior to carrying out the evaluation.To appraise the proposed intervention, committee members were then asked to score  LBC for cervical cancer screening with respect to each MCDA criterion of decision, based on synthesized data from the by-criterion HTA report. Committee members then considered what type of impact each of the four qualitative criteria might have on use of LBC in the context of the health plan.Feedback on the framework and process was elicited during discussion periods at the first workshop and at a follow-up workshop, and from a questionnaire administered during the follow-up workshop. Committee members were surveyed about whether each of the framework's decision criteria should always, sometimes or never be considered in decisionmaking. They were also surveyed about how the EVIDEM approach compared to their current process regarding data provided, deliberative process and communication of decision .Weights, scores and impact obtained from committee members were entered into Excel software. Descriptive statistics were used and mean ± standard deviations (SD) were calculated for weights and scores. The MCDA value estimate for the use of LBC in cervical cancer screening was then obtained by applying an MCDA linear additive model . Thus, talways be considered  committee member expressing the opinion that it should always be considered. No committee member indicated that any criterion should never be considered.When surveyed on whether each of the criteria of the framework should always, sometimes or never be considered in the decisionmaking process, all committee members agreed that "Budget impact on health plan" should ed Table . For eigCommittee members were asked to weight each of the 14 MCDA decision criteria included in this field test according to their importance to the appraisal of an intervention, independently of the intervention that was to be appraised.  The greatest weights  were assigned to the criteria "Budget impact on health plan", "Cost-effectiveness of intervention" and "Completeness and consistency of reporting evidence", followed by "Improvement of efficacy/effectiveness" and "Relevance and validity of evidence" (mean 4.4) Figure . The lowhttp://www.evidem.org/tiki/?page=CCLBC-RecordMenu). Key items included in the assessment are reported here. In the context of cervical cancer screening, LBC refers to a technique in which the cervical sample is suspended into a preservative liquid, rather than, as in conventional cytology , being smeared onto a slide. Cervical cancer causes high morbidity and mortality [The HTA report for LBC was based on 14 references  of the potential maximum value  the committee assigned to the cost-effectiveness of LBC , reflecting minor improvement at a significant cost, and also captured the importance of absolute elements of value such as disease severity, type of medical service and quality of evidence.Non-quantifiable, contextual criteria of decision were also identified and considered. Given the lack of cost-effectiveness and high additional cost of LBC compared to conventional Pap smears, the budget impact would be substantial and therefore a co-payment would be considered. However, health plan members and many other South Africans are used to receiving Pap smears for free as part of health screening programmes and the implementation of co-payments would meet with strong resistance from both patients and healthcare providers. The use of the MCDA framework as a communication device to convey the structure and transparency of the decisionmaking process was thought to be an important tool to improve acceptability of the committee's decision.Adaptation of the EVIDEM framework from medicines evaluation to a screening technology required minimal change to the structure of the framework. Since devices, procedures and technologies form a growing portion of the basket of healthcare interventions that require evaluation it was useful to assess the adaptability of the framework to these types of interventions.Currently, there is no single HTA body (such as NICE or CADTH) in South Africa with the result that evaluations of new and existing health interventions are scattered and often duplicated. Skilled resources to carry out a proper HTA are scarce and a need is recognized for pooling these resources, allowing greater availability to all sectors in the country ,39. HealOn the whole, the committee members favoured the use of the EVIDEM process and were positive about its use. Some committee members rated their feedback of the EVIDEM process as "same" as existing process in the questionnaire, which may be explained by the fact that a more crude and simpler multicriteria approach was already being used by the committee.This field test indicates that the EVIDEM framework may be useful for the evaluation of health technologies in the South African private healthcare sector context. However, further adaptation to a health plan's specific mission, scope and priorities is required to create a "custom-made" decisionmaking framework fully aligned with the local context. The framework's contextual tool  -- a result of a further development of this framework  -- facilThe EVIDEM framework was easily adapted to evaluating a screening technology thereby broadening its applicability in healthcare evaluation. It was found to improve: understanding of intervention; access to quality assessment of evidence; consideration of key elements of decision; transparency of decision; and understandability of decision by stakeholders. Further field testing and instrument validation and development are ongoing to collaboratively advance MCDA approaches and contribute to more transparent and efficient healthcare decisionmaking.AEs: Adverse events; ANOVA: Analysis of variance; CE: Cost-effectiveness; EVIDEM: Evidence and Value: Impact on DEcisionMaking; HTA: Health technology assessment; ICC: Intra-rater correlation coefficients; INAHTA: International Network of Agencies for Heath Technology Assessment; MCDA: Multicriteria decision analysis; PRO: Patient reported outcomes; QALY: Quality-adjusted life-year; QM: Quality Matrix; RCT: Randomized controlled trial; SD: Standard deviation.The authors declare that they have no competing interests.JKM, MMG, and MW designed the study and HK collected and analysed the data. JKM, MMG and MW drafted the manuscript. DR reviewed the manuscript and is one of the developers of the EVIDEM framework. All authors read and approved the final manuscript.


To support the use of research evidence by community-based organizations (CBOs) we have developed 'Synthesized HIV/AIDS Research Evidence' (SHARE), which is an evidence service for those working in the HIV sector. SHARE consists of several components: an online searchable database of HIV-relevant systematic reviews (retrievable based on a taxonomy of topics related to HIV/AIDS and open text search); periodic email updates; access to user-friendly summaries; and peer relevance assessments. Our objective is to evaluate whether this 'full serve' evidence service increases the use of research evidence by CBOs as compared to a 'self-serve' evidence service.We will conduct a two-arm randomized controlled trial (RCT), along with a follow-up qualitative process study to explore the findings in greater depth. All CBOs affiliated with Canadian AIDS Society (n = 120) will be invited to participate and will be randomized to receive either the 'full-serve' version of SHARE or the 'self-serve' version (a listing of relevant systematic reviews with links to records on PubMed and worksheets that help CBOs find and use research evidence) using a simple randomized design. All management and staff from each organization will be provided access to the version of SHARE that their organization is allocated to. The trial duration will be 10 months , the primary outcome measure will be the mean number of logins/month/organization (averaged across the number of users from each organization) between baseline and the end of the intervention period. The secondary outcome will be intention to use research evidence as measured by a survey administered to one key decision maker from each organization. For the qualitative study, one key organizational decision maker from 15 organizations in each trial arm (n = 30) will be purposively sampled. One-on-one semi-structured interviews will be conducted by telephone on their views about and their experiences with the evidence service they received, how helpful it was in their work, why it was helpful (or not helpful), what aspects were most and least helpful and why, and recommendations for next steps.To our knowledge, this will be the first RCT to evaluate the effects of an evidence service specifically designed to support CBOs in finding and using research evidence.NCT01257724ClinicalTrials.gov:  To do this, they need support in finding and using research evidence to help them plan and deliver more effective and cost-effective programs and strengthen health systems.Community-based organizations (CBOs) are important stakeholders in health systems ,2 becausHowever, there are many potential challenges related to research use. Barriers that have been consistently identified across different sectors include the complexity of research evidence, organizational barriers, lack of available time, poor access to current literature, lack of timely research, lack of experience and skills for critical appraisal, unsupportive culture for research, lack of actionable messages in research reports, and limited resources for implementation -7. GivenWhile there are strategies for supporting the use of research evidence by clinicians ,14, and i.e., target audiences incorporating prompts for research evidence in their decision-making processes and developing their capacity to find and use research evidence) or 'exchange' efforts, which focus on the producers and users of researchers building partnerships and working collaboratively in the production and interpretation of research evidence [Efforts to facilitate the use of research evidence often focus on four clusters of knowledge translation activities  , and theevidence .Our objective is to evaluate whether, how, and why this 'full serve' evidence service increases the use research evidence by key decision makers in CBOs as compared to a 'self-serve' evidence service.We will conduct this trial using a sequential explanatory mixed methods design , beginniCommunity-based HIV/AIDS organizations in Canada provide a number of programs and services to people living with or affected by HIV, which may include prevention initiatives, individual or group counseling/support, and community outreach and/or education. In addition, organizations in Canada are situated in diverse geographic settings ranging from dense urban settings to rural, northern, and/or remote settings, with some focused on specific at-risk populations and/or cultural or ethnic groups.e.g., the Ontario AIDS Network), and send an organizational invitation to the executive director and management team (if applicable). The invitation will indicate that if they are interested in having their organization participate, access to SHARE will be provided to all interested staff. Given that SHARE is currently only provided in English, we will exclude organizations that do not have at least one key decision maker who is comfortable participating and corresponding in English.We will draw our sample from those organizations affiliated with the Canadian AIDS Society and from relevant provincial HIV/AIDS networks  on behalf of their organization at baseline and again at the completion of the trial. We will request that the executive director complete the survey, but will indicate that they can delegate to another manager provided the manager has a decision-making role about programs, services, and advocacy, and provided the manager does not include the conduct of research among their core responsibilities. Because the overall intent of the intervention is to support the use of research evidence in decisions about CBOs' programs, services, and advocacy, we deemed it most appropriate for the executive director (or another manager) to complete the survey because they would have the most impact on whether research evidence is used to inform decisions.Based on the membership list provided by the Canadian AIDS Society on their website, there are 120 CBOs available to draw the sample from. Drawing on previous experience with this sector, we expect to achieve an approximate response rate of 70%. To increase our response rate, the Canadian AIDS Society will send out an email to all its members, encouraging them to participate by highlighting the importance of the trial. We will provide additional incentive to enroll in the trial by holding a draw where we will select three organizations to receive prizes (gift cards) worth $500, $250 and $100.We will run a two-arm RCT with a 'full-serve' evidence service (SHARE) as the intervention arm and a 'self-serve' version as the control arm. The components of each version of SHARE are outlined in Table Organizations allocated to this study arm will receive access to a 'full-serve' version of SHARE, which provides:1. an online searchable database of HIV-relevant systematic reviews  and provide a brief overview of the range of topics addressed by the new reviews;2. periodic email updates (at least one per month), which will profile the types of new reviews recently added to the database (3. access to user-friendly summaries produced by us or by others (when available);4. links to scientific abstracts;5. peer relevance assessments, which involves periodic requests (contained in the single record for each review) to complete a brief assessment of how useful the information in the newly added review is ;6. an interface for participants to leave comments (up to 250 characters in length) in the records of systematic reviews in the database (7. links to full-text articles (when publicly available); and8. access to worksheets that help CBOs find and use research evidenceAustralasian Cochrane Centre (AAC) Policy Liaison Initiative, Database of Abstracts of Review of Effects (DARE), Effective Healthcare Research Programme Consortium, Evidence AID, Health Knowledge Network, Health-Evidence.ca, Reproductive Health Library, Rx for Change, and Supporting Policy Relevant Reviews and Trials (SUPPORT)[To provide access to user-friendly summaries (see component three above) we will provide links to user-friendly summaries produced by nine groups (when available) from around the world: (SUPPORT),26-34.Organizations allocated to the control group will only be provided website access to a listing of systematic reviews that are organized by year of publication with links to the record on PubMed (or another publicly available source when not available on PubMed) and access to worksheets that help CBOs find and use research evidence.After consenting to participate in the trial, we will use simple randomization to assign organizations to receive either the 'full-serve' or the 'self-serve' evidence service. The list of participating organizations will be sent to a statistician (TB) who will assign a unique ID number to each organization, conduct the randomization, and keep both the key linking the organizations to their ID and the randomization log in a secure password protected folder at the Ontario HIV Treatment Network to provide a clear audit trail. We will perform simple randomization sampling using the SAS SELECTSURVEY procedure to assign equal numbers of organizations to the 'full-serve' and the 'self-serve' groups. The procedure will be performed with a fixed seed so that the sampling can be replicated if needed. The statistician will then provide the list of unique IDs with the results of the randomization to the SHARE database administrator at the Ontario HIV Treatment Network  who will provide individuals from each participating organization with access to the 'full-serve' or 'self-serve' versions of SHARE. This will require the SHARE database administrator to have access to the key linking the unique IDs to the organizations but it will remain concealed from the research team.Prior to the start of the trial, all organizations will be requested to provide a list of emails of management and staff interested in receiving access to SHARE, which will be provided to the SHARE database administrator at the Ontario HIV Treatment Network. We will then send bi-monthly emails to the executive director (or another delegated staff member for correspondence) to identify any staff that have either joined or left the organization in order to accurately track usage at the organizational level. The SHARE database administrator at the Ontario HIV Treatment Network will send the updates to individuals affiliated with organizations with access to the 'full-serve' version of SHARE (the updates will be written by MGW and checked by the co-investigators). The statistician (TB) is a member of the study team but will only be involved with randomization at the start of the trial and the data analysis upon completion of the trial. Therefore, participants and all investigators except the statistician (TB) and the SHARE database administrator will be blinded to group assignment.Measuring the impact of knowledge transfer and exchange (KTE) interventions, such as the evidence service proposed here, poses significant challenges as there is a long chain of factors between a KTE intervention such as SHARE and the health status of clients of CBOs or of broader populations ,35. For et al. (2006) used in their trial of the McMaster Premium Literature Updating Service (PLUS) [e.g., URLs to abstracts, user-friendly summaries, and/or full-text), and the number of times the email updates to the 'full-serve' group are forwarded.Given these constraints, our primary and secondary outcomes for the trial are proxy measures for research use. The primary outcome will be a measure of utilization that is similar to what Haynes e (PLUS) . SpecifiEach version of the evidence service will be hosted on the Ontario HIV Treatment Network server and for the duration of the trial will require a user login that will be used to link each participant's identification with their usage of the evidence service website and to their organization. SHARE is a new database that is not yet publicly available , which allows us to evaluate it without participants being able to gain access from a publicly available site. In addition, requiring a user login will help protect against contamination of the intervention and control group. However, we cannot protect fully against the possibility of participants from the organizations sharing information given that many may collaborate with each other.i.e., beliefs and judgments), subjective norms , and perceived behavioural control  -- that shape the behaviour intentions of people, which is subsequently a strong predictor of future behaviour [For the secondary outcome, we will use the theory of planned behaviour to measure participants' intention to use research. The theory of planned behaviour proposes a model about how human action is guided ,41 and cehaviour . In FiguThe theory of planned behaviour has been extensively used and tested in the fields of psychology and healthcare. Systematic reviews conducted in the psychology field have demonstrated that the theory explains about 39% of the variance in intention and about 27% of the variance in behaviour ,43. A nuet al. (2010) found moderate test-retest reliability of the instrument using Generalizability Theory (G = 0.50) [et al. (2010), the first administration of the tool immediately followed a McMaster Health Forum stakeholder dialogue, which may have promoted enthusiasm for using research evidence among participants. This likely produced higher measures of intention on the first administration of the tool as compared to the second, resulting in the lower G-score. Given that we won't be administering the tool in a similar atmosphere of enthusiasm for using research evidence, it is likely that the level of reliability of the tool will be sufficient without two administrations at both baseline and follow-up.Using a manual to support health researchers who want to construct measures based on the theory , our col = 0.50)  when sco = 0.50) . In the We have slightly modified the wording in each of the questions of the tool to reflect the different intervention being tested (SHARE) and the target audience (CBOs)  and, during the analysis, all investigators -- except for one of us who is involved in the both the analysis and randomization (TB) -- will be blinded to the key linking the organizations to their unique identifiers.e.g., the amount of staff support an executive director or manager has may determine the extent to which they can spend time finding and using research evidence), and therefore adjusting for them will allow for a better assessment of the effects of the intervention. Moreover, as part of a secondary analysis, we will assess whether there is an interaction between each (entered as fixed factors) and the outcome measures. Given the likelihood that the distribution of the outcomes will be skewed, we will transform the data where necessary and possible, which may include adjusting the time period for which we calculate the mean number of logins/month/organization  if the number of logins is low and there are insufficient data for analysis. We will also qualitatively compare the number of participants in the intervention and control groups that do not complete the follow-up survey, and attempt to assess if there are reasons for why they dropped out based on their baseline characteristics.We will treat both outcome measures as continuous variables and analyze the change in these measures over time using a two-way mixed effects linear repeated measures analysis of variance (ANOVA), which will assess the effects within groups, between groups, and over time with the latter as the main feature of interest. In addition, we will control for four variables -- province the organization is located in, size of organization , number of participants that participated from each organization, and the number of clients served each year by the organization -- using analysis of covariance. For the analysis of the secondary outcome, we will also control for whether the key decision maker is full-time or part-time, and whether they have had research training in the past. Each of these variables may at least partially explain research use  to be significant. For the primary outcome measure (mean logins/month/organization), missing data are irrelevant as they are a naturalistic measure. For the secondary outcome measure (obtained through the survey), missing data can be taken into account through the use of a mixed-effects model.Given a fixed sample size of approximately 85 organizations (70% of 120 organizations) a sample size calculation is not relevant. Instead, we have calculated the level of statistical precision that we can expect given our fixed sample size. To calculate the expected statistical precision in the trial, an estimation of intra-class correlation coefficient (ICC) of measurements for individuals over time for the primary outcome is required. However, we have no mechanism to estimate the ICC due to the fact that no similar study with this population has been conducted (at least to our knowledge). Therefore, we calculated estimates of statistical precision for ICCs of 0.2, 0.3, 0.5, 0.7 and 0.8 based on a six-month trial period with 80% power, an estimated standard deviation of 1.0, significance of 0.05 (two-sided test), and 42 organizations per study group  . Assumine.g., the ease of use of SHARE).Given that this is the first RCT evaluating a KTE intervention for CBOs (at least to our knowledge) and the inherent limitations associated with measuring research use as an outcome, we will conduct a qualitative process study after the completion of the trial to explore the RCT findings in greater depth. The qualitative study will explore how and why the evidence service worked (or didn't work), determine how the 'full-serve' and 'self-serve' evidence services were used, including the degree of contamination between the intervention and control groups, and other factors that may have influenced their use , and with varying size and location within the country. We have assumed a 70% response rate, which means that we should sample approximately 40 organizational key decision makers to achieve a sample size of 30.We will use a mixed method sequential nested sampling procedure whereby a larger sample is analyzed in one study (RCT), and a subset of the larger sample is selected for further inquiry in the second study . SpecifiOne-on-one semi-structured telephone interviews will be conducted with key decision makers about their experiences with the evidence service, including whether and how they used it  and why, whether, and how it was helpful in their work and why, what aspects were most and least helpful and why, and recommendations for next steps. In addition, we will ask participants about any recommendations for how to improve upon our efforts to support the use of research evidence by CBOs. Finally, for the document analysis, we will collect all comments provided in the user forums for each systematic review recordi.e., we will send a brief, structured summary of what we learned from the interviews and invite comment on it). Finally, we will use the document analysis of the comments left in the user forum to help further our understanding of how participants engaged with the 'full-serve' version of SHARE.We will tape and transcribe all interviews, use N-Vivo 8 for data management of both the interview transcripts and document analysis, and use a constant comparative method for analysis -56. SpecTo our knowledge, this will be the first RCT to evaluate the effects of an evidence service specifically designed to help CBOs find and use research evidence. As we have outlined elsewhere , effortsThe main limitation of this trial is the relatively small sample size that we have available to draw upon. However, while the sample size is relatively small, we are still reaching an entire sector of CBOs, which will help provide more generalizable results. In addition, through our partnership with the Canadian AIDS Society and their support with study recruitment, we hope to achieve a high response rate. Another potential limitation is study contamination between the intervention and control groups as some participants may collaborate with each other and share their login and password. To assess contamination we have included a question in the follow-up survey asking if they shared their login and password with anyone else outside their organization.Three of the authors  were involved in the development of the SHARE database and remain involved in its continuous updating. SHARE is the intervention being tested in the trial.MGW conceived of the study, participated in its design, and drafted the protocol. JNL participated in the design of the study and helped draft the protocol. JG and RBH participated in the design of the study and provided feedback on drafts of the protocol. TB participated in the design of the study, performed the sample-size calculations, and provided feedback on drafts of the protocol. SBR provided feedback on drafts of the protocol. All authors read and approved the final manuscript.Appendix 1: SHARE (Synthesized HIV/AIDS Research Evidence) taxonomy of topics. Topics used to categorize systematic reviews contained in SHAREClick here for fileAppendix 2: Peer-relevance assessment question. Each systematic review record in SHARE asks users to answer one question about how useful the information is. The results are displayed to the user after answering the question.Click here for fileAppendix 3: Data collection instrument (secondary outcome measure). A survey measuring participants' intention to use research evidence, which will be administered at baseline and at the end of the trial.Click here for file