Insufficient access to essential medicines is a major health challenge in developing countries. Despite the importance of Standard Treatment Guidelines and National Essential Medicine Lists in facilitating access to medicines, little is known about how they are updated. This study aims to describe the process of updating the Standard Treatment Guidelines and National Essential Medicine List in Tanzania and further examines the criteria and the underlying evidence used in decision-making.This is a qualitative study in which data were collected by in-depth interviews and document reviews. Interviews were conducted with 18 key informants who were involved in updating the Standard Treatment Guidelines and National Essential Medicine List. We used a thematic content approach to analyse the data.The Standard Treatment Guidelines and National Essential Medicine List was updated by committees of experts who were recruited mostly from referral hospitals and the Ministry of Health and Social Welfare. Efficacy, safety, availability and affordability were the most frequently utilised criteria in decision-making, although these were largely based on experience rather than evidence. In addition, recommendations from international guidelines and medicine promotions also influenced decision-making. Cost-effectiveness, despite being an important criterion for formulary decisions, was not utilised.Recent decisions about the selection of essential medicines in Tanzania were made by committees of experts who largely used experience and discretionary judgement, leaving evidence with only a limited role in decision-making process. There may be several reasons for the current limited use of evidence in decision-making, but one hypothesis that remains to be explored is whether training experts in evidence-based decision-making would lead to a better and more explicit use of evidence. Insufficient access to essential medicines is a major health challenge in developing countries; among poor populations more than half have been estimated to lack regular access to medicines The essential medicines programme entails stocking a limited range of efficacious, safe and cost-effective medicines that are sufficient to meet the priority health needs of the people Tanzania, one of the pioneers of the essential medicines programme, produced its first list of essential medicines in the early 1970s Tanzania is categorised as a low-income country with a per capita expenditure on health of about 41 US$ per year Most developed countries have health technology assessment (HTA) systems, such as the National Institute of Clinical Excellence (NICE) in the UK and the Canadian Agency for Drugs and Technology in Health (CADTH), which issue formulary recommendations for reimbursement decisions. Cost-effectiveness evaluation is a mandatory criterion employed to inform decision-making The study was approved by the Ethical Review Committee of the Tanzania National Institute of Medical Research. The study was conducted during a period when there was a medical doctors’ strike in the country and therefore we anticipated challenges in obtaining written consents. This concern was communicated to the ethics committee and authorisation was granted to use verbal consent. After self-introduction, the purpose of the study was explained to each informant and confidentiality was assured. All the informants were nevertheless, cautiously, asked for written consent before commencing the interviews, but they opted to give verbal consent. Each informant was assigned a code number which was entered on a consent form and signed to document that verbal consent had been given. Furthermore, the interviews were recorded with permission from the informants and the digital voice recorder and the transcripts were kept confidential.This qualitative study utilises a descriptive case study design, which is an empirical inquiry that investigates a phenomenon within its real-life context The research team consisted of two doctoral students, a senior researcher and two professors. ATM’s background is in pharmacy, health policy analysis and management. FN’s background is in medicine (MD) and health economics. EAK is a senior researcher (PhD) and has outstanding experience with the ATLAS.ti® data analysis software. OFN and BR are professors with extensive experience of national guidelines and drug reimbursement advisory committees in Norway.We used purposive sampling methods to select 18 information-rich informants from the list of experts who participated in the revision of the STG/NEML and two who did not participate, but who were perceived to possess important information for the study. We obtained this list, which contained the names, professions, specialisations, institutions and phone and email contacts, from the Ministry of Health and Social Welfare (MoHSW). Some informants were contacted through phone calls while others were visited at their work places. All the selected informants agreed to participate in the study. Several other informants were also involved in the study through informal interviews which were conducted in order to broaden our understanding of the inquiry.In selecting the informants, we chose those who had experience of participating in the previous revision process, but we also wanted to have good professional, institutional and speciality representations. Therefore our informants were pharmacists and clinicians with different specialisations from referral, municipal and specialised hospitals. Others were programme and section officers from the MoHSW . The finIn-depth interviews and document reviews were the main methods of data collection and were carried out between June and December 2012, while the revision of the STG/NEML was still ongoing.In-depth interviews were conducted face to face with key informants, in English, using a pre-tested, semi-structured interview guide. All formal interviews were conducted in the offices of our informants and nobody else was present during the conversations. Interviews were digitally recorded and each lasted for 30–45 minutes. The 18 formal interviews, including one repeat interview, with the STG/NEML review group and the two additional in-depth interviews with informants from the TFDA were sufficient for data saturation. In addition, some informal interviews were conducted without the interview guide.Several documents containing information related to the implementation of the essential medicines programme in Tanzania were reviewed to supplement the interview data. This included the STG/NEML of 2007 and 2012, the national drug policy, minutes and proceedings of the review meetings, published reports and research articles. We reviewed the malaria and HIV/AIDS treatment guidelines in order to determine whether they are consistent with international guidelines. The research team was already in possession of some of these documents, which were used for a systematic review study about the use of pharmacoeconomics as a criterion of medicine selection in Tanzania The interview guide contained questions and probes as described below. The flow varied from one participant to another depending on how the discussions unfolded. It was piloted with four participants and, because no major changes were introduced in the guide, we decided to include these interviews in the analysis.Informants were asked to describe how the review process was conducted and probed about how they became involved, how and by whom they were contacted and their personal views about the process. Those from the MoHSW who initiated and co-ordinated the process were in addition probed about how they selected the participants, the rationale for doing the review, composition of the committees etc.Informants were asked about how they selected the medicines, the selection criteria, how strictly the criteria were followed and to rank the criteria based on their importance/strengths. For each criterion they mentioned, they were probed to give the type and source of evidence and how they evaluated such evidence.Informants were asked whether they used economic analysis as a criterion if they had not mentioned it before, and how and to what extent economic evaluation was used (probed to give examples). They were further probed about challenges that hinder the use of economic evaluation and enabling factors for its use. Lastly they were asked if they had received any training in health economics.Verbatim data were transcribed into text using a standardised transcription protocol The coding tree consisted of three branches (themes); process, criteria and evidence. Under process there were two categories; the STG/NEML review and the approval process. Under criteria we did not have categories but only the codes for each criterion. Evidence was categorised as being drawn from experience and scientific study or from official documents. Code names reflected the content of each text segment.Five measures were taken to ensure that the data obtained were valid and trustworthy. Firstly, informants were contacted informally to create a platform for self-introduction, to explain the purpose of the research and assure confidentiality. Secondly, informants were purposively selected to maximise representation of a wide range of perspectives on the subject. Thirdly, interviews were recorded and then transcribed shortly after each interview session to preserve the originality of the data. Fourthly, transcripts were shared with the informants for cross-checking and validation, after which ten of the 18 informants provided feedback, most of them without any changes and a few with minor editing. Fifthly, triangulation of data was performed to enrich and supplement the information collected by interviews and document reviews.This section provides results about the STG/NEML revision process, the criteria applied and the extent to which evidence was utilised during decision-making. To illustrate the findings, supporting verbatim quotes from the informants are provided. Informants are only identified by their institutions in order to avoid any breach of confidentiality.The process of updating the STG/NEML was initiated in early 2012 by the Ministry of Health and Social Welfare (MoHSW). The process was co-ordinated by the Pharmaceutical Service Section (PSS) on behalf of the National Medicines and Therapeutic Committee (NMTC). The sequence of events is shown in The review was caused by two important things: first the number of diseases had increased and the required medicines to manage such diseases were not in the STG/NEML. Also it has been a long time since the existing STG/NEML was revised and there have been some new developments and changes in how certain diseases are managed in clinical practice. Basically, the WHO recommends revision after every 2–3 years. (Participant from the MoHSW)To begin the review process, the PSS convened an internal meeting to establish a committee of experts, known as ‘a guideline review secretariat’ to revise the STG first; this was important because only those medicines recommended in the STG are listed in the NEML. One official explained:First we had an internal meeting where we decided the kind of people to involve in the process; we wanted a mixture of people from primary to tertiary-level facilities, including people from various programmes. Therefore we consulted people from the malaria and HIV/AIDS control programmes who had the experience of reviewing their treatment guidelines and they gave us guidance about who we should involve in the review process. (Participant from the MoHSW)The STG/NEML document has two parts; the STG part contains 25 chapters covering common diseases in Tanzania, their clinical signs and symptoms, how they should be diagnosed and the recommended treatments or supportive care. The NEML part contains the list of all medicines that are recommended in the STG. It uses generic names and the medicines are arranged according to their pharmacological groups. The NEML also specifies the dosage form, its strength and the level of healthcare facilities where each medicine should be made available.The guideline review secretariat was composed of a multidisciplinary team of experts  who wereWhen I was pursuing my masters at […] we were at times being taught by people from foreign universities such as […] so they emphasised the use of evidence, particularly from meta-analyses in decision-making. At its first meeting, the secretariat discussed the approach to update the STG. A consensus was reached to split into groups according to medical specialities to simplify and speed up the revision process. Each group was tasked with revising a specific section of the guidelines pertaining to its speciality. One informant said:We started the review process by going through the old STG first; looking at what was missing, what to add and even what should be removed completely. We went through each disease condition one after another but not as a single panel. We were divided into specialities, people dealing with cancer looked at cancers, cardiology the same etc. The revision process varied between groups; some groups organised discussion meetings and disseminated their recommendations around their respective departments for comment, while in other groups the task was a take-home assignment for group members. The majority of informants said that the organisation, participation and time allocated to the task were not satisfactory.In my opinion the process is not perfect. First of all the time for review is very constrained, in such a way that you cannot have effective and detailed discussions on the management of patients and other important issues. The process was so disorganised, I remember I presented the work of another person who was not there […]. I can say the whole organisation was not good. During the second meeting, group leaders presented the proposed recommendations to the secretariat so that other members could provide comments. Thereafter, these group works were compiled into a first draft of the STG. The PSS then extracted all the recommended medicines from this guideline to formulate the NEML. The draft of the STG/NEML was disseminated to different experts before it was submitted to the National Medicines and Therapeutic Committee (NMTC) for approval. One informant said:The draft of the STG/NEML was sent to the panel of reviewers for comment. Most of our reviewers come from major hospitals; we believe they have experience in research and publications. Then, after receiving their comments, we addressed them before we sent the document to the National Therapeutic Committee for approval. (Participant from the MoHSW)The criteria employed and level of evidence varied between the groups updating different sections of the STG; some groups considered only one criterion while others used several. These criteria, together with their supporting evidence and other factors that influenced decision-making, are described below.A majority of informants said that they used an evidence-based approach in updating the STG/NEML. Efficacy and safety were the most cited criteria to have been used by different informants from the guideline review groups. Two informants said:We were using an evidence-based approach that a recommended drug must have shown that clinically it was more potent and produced more benefits and there is research evidence for that. At the time of the review […] was a hot cake, people were trying to assess whether it was safe for patients. In general the concern was how it fares in the field as compared to […]; there were some discussions and we reached a consensus as you saw in the guidelines. Regarding the use of evidence, all informants acknowledged that evidence summaries for these criteria were not generated to inform decision-making, but claimed that such evidence was known to them through clinical experience. Two informants said:Clinical experience was crucial, because you want to produce a practical guideline. Therefore frankly speaking, a lot of evidence came from my daily practice and this was not scientific evidence but simply my experience. Doctors were the ones who were recommending medicines for the Standard Treatment Guidelines to manage diseases through their clinical practice. Our belief was that as long as the medicine is being used in the hospital then automatically there was clinical evidence for that medicine to be selected. (Participant from MoHSW)The two informants who mentioned being trained in evidence-informed decision-making said that their recommendations were supported by research evidence from clinical trials and meta-analyses. However, during the interviews, as well as saying they did not develop evidence summaries, they were also not able to give sufficient explanation about how they searched and appraised the evidence. Some informants, particularly the pharmacists, recognised the lack of evidence in the decision-making process. One member of the secretariat expressed the following concern:We asked the physicians if the evidence they were giving to support their recommendations was actually based on scientific research! Unfortunately no one said it was scientific evidence. They all said the evidence was observation from their clinical practice and feedback from their patients. In some situations the interviewer challenged the informants with scientific evidence supporting the use of some medicines for a condition other than the one they had recommended. Surprisingly, some informants disagreed with such evidence, which was another indication of how difficult it was for scientific evidence to find its way onto the decision-making table compared with that from clinical experience. One such example is Tranexamic Acid (TXA) injection, a drug which reduces the risk of death in bleeding trauma patients I have never heard of any clinical trials done on tranexamic acid injection. I do not agree or believe in that research, it is not correct. Tranexamic acid is not a treatment, it is prevention and it is contraindicated in massive injuries. How can it help when someone has massive bleeding? All informants said that it is very important that the medicines they select are available on the Tanzanian market so that patients can access them even when they are not available at public healthcare facilities. They said that the existing policy requires all medicines in the STG/NEML to be written with generic names rather than brand names because generics are readily available, relatively cheap and affordable.We asked ourselves whether the medicines we were selecting were actually available in the market. The main question was: if a certain medicine in the list was prescribed will it be available in the Tanzanian market? We thought it would not make sense to have medicines in the STG/NEML which were not readily available in the country. Informants from the Tanzania Food and Drugs Authority (TFDA) said that the availability of medicines depends on whether they are registered in Tanzania or not. They said that in order for any medicine to be allowed to enter the Tanzanian market it has to go through a rigorous registration process in which its quality, efficacy and safety are thoroughly checked. They said that the TFDA registers all medicines that meet a minimum level of prescribed standards and it is from this pool that essential medicines are selected. One official explained:Drug registration actually involves many processes. In summary, the applicant, usually the manufacturer of the drug, must provide detailed information about the active pharmaceutical ingredient, the finished product and good manufacturing process which demonstrates the quality, safety and efficacy of the drug product. (Participant from the TFDA)None of the informants mentioned having used the list of registered medicines from the TFDA. Instead they said they knew the available medicines through their practice. Through the interviews we learned that healthcare workers in Tanzania frequently receive drug information from representatives from pharmaceutical companies. One informant said:We communicate a lot with our colleagues working with pharmaceutical companies, so they tell us if there are new drugs as first lines for certain diseases with better clinical outcomes. A majority of informants said that affordability was an important criterion in medicine selection because economically Tanzania is very poor. Despite this concession, some informants said that there were disagreements between the doctors and pharmacists about the limit on the number of medicines to be added to the STG/NEML and whether expensive medicines should also be selected. Some doctors wanted the STG to have a variety of medicines from different therapeutic classes for each disease, some of which were considered expensive. Pharmacists, as the custodians of medicines, often opposed them because they were concerned about budget implications. There was no consensus about these disputes and the final ruling awaited the approval meeting of the National Medicines and Therapeutic Committee. One informant said:There were disagreements between me and the doctors, I told them essential medicine means to have a limited number of affordable medicines but they said “No! We are the ones who are in the field treating patients”. Therefore there is a need for people to be educated about the meaning of essential medicines. Informants said that they also took into account the total cost of treatment rather than the unit prices of individual medicines, especially for chronic diseases. One informant said:In some cases we looked at costs for a full course of treatment rather than unit prices. A month’s cost of a 50 Tanzanian shilling (Tshs) tablet taken three times a day is 4,500 Tshs compared to 3,000 Tshs for a once-a-day sustained-release tablet which costs 100 Tshs. You see, this is cheaper and increases compliance with treatment. Regarding the evidence for medicine prices, the majority said they knew the prices of most medicines through experience but some said they used price lists from the Medical Stores Department and drug representatives from private medicine suppliers. One informant said:So if two drugs were equally efficacious but one is cheaper then we selected the cheaper one so long as it has acceptable quality. We used the price list from the Medical Stores Department to compare the costs of the drugs. The understanding of cost-effectiveness analysis was poor among the majority of informants. Some confused it with cost comparisons and others were completely unaware of the concept. Those who said they were aware of it said that cost-effectiveness was not used as a criterion for medicine selection. They went on to say that economic evaluation studies are scarce in the country, and even if they were available they could not use them because they lack expertise.Two informants said the following:Economic evaluation was not used at all. I think that means there should have been some studies about economic evaluation of medicines in Tanzania, which I am not sure if there is. Honestly, we have not taken on board such a criterion in the medicine selection process. (Participant from the MoHSW)Well, that one I cannot say much about since it is not part of our expertise. I don’t remember in our group talking anything about economic evaluation of medicines, maybe in other groups but not the one I was with. Only one informant said he had received training about the use of economic evidence. Amongst the others, besides saying this was not an area in which they had expertise, one went further to comment that:Perhaps in the future, use of economic evaluation should be emphasised and the team involved in the review should be given lectures on pharmacoeconomics so that they can have knowledge about other criteria for inclusion of medicines in the Standard Treatment Guidelines besides clinical reasons. Informants said that Tanzania has vertical programmes for HIV/AIDS, malaria, tuberculosis and leprosy. These programmes have their own guidelines, which are updated based on global recommendations. These recommendations were adopted in the STG. In addition, some informants said that they copied their recommendations from textbooks and guidelines from countries such as the USA, South Africa, Ghana and Lesotho. Regarding the use of international guidelines, one informant said:With malaria we follow the global recommendations, for example in 2010, the WHO malaria treatment guidelines were revised and artesunate injection was recommended for severe malaria. So it is the same with other ACTs, vaccines and some other medicines. (Participant from MoHSW)Some informants accused medicine promotion for influencing prescription practices. They were concerned that the medicines recommended for addition in the STG were there because of these types of influence. Two informants said:The second and most important point is that selection was the direct influence of medical representatives […]. They come here and talk to the doctors and they give them some free samples, what they get in return we do not know. With time doctors get used to these medicines and then they force their inclusion in the hospital formularies and later into the STG/NEML. Lobbying by medical representatives is a problem. This hospital has a policy that medical representatives should make presentations in the meetings but sometimes they do not do that, they follow us into our offices to convince us to prescribe their products. They give gifts and some other things. Honestly, this is a common practice. To elaborate more on this practice, one informant said she used her experience and sometimes gathered evidence from journals. When she was asked to give the name of the journal, she said:Oooh! my Goodness I cannot remember the journal, but I can link you with those individuals at […] pharmaceutical companies and they can give you that information. In the informal interviews, drug representatives acknowledged that they persuade healthcare workers to procure and prescribe their products, and they do this by giving them free medicine samples and gifts such as stationery, refrigerators and televisions.The National Medicines and Therapeutic Committee (NMTC) is responsible for the approval of treatment guidelines and formularies in Tanzania and the PSS acts as the secretariat for this committee. The newly formed committee is multidisciplinary and consists of 18 members , with thThe updated draft of the STG/NEML was submitted to the NMTC by the secretariat for approval in September 2012; about 130 new medicines were proposed to be added and five to be deleted. This draft was submitted without a summary of the changes and or the rationales behind them. Therefore the approval process proceeded first with the explanations given by the secretariat about the changes made in each chapter, followed by brief discussions. An excerpt from one of the document reads:The committee was taken through the reviewed STG and NEML, chapter by chapter. In each chapter presentations, the major changes which were made, in comparison to the STG/NEML edition of 2007, were explained to the panel. The NMTC members discussed and made recommendations.There was no specific set of criteria used by the NMTC to approve each of the proposed changes. However, through document reviews, we found that the proposal to add clindamycin injection for the management of malaria in pregnancy was rejected because of safety and affordability concerns. The committee also ordered the secretariat to shorten the NEML, citing budget limitations as the only factor. This shows that to a certain extent some criteria were considered but again were not supported by evidence. One informant who participated in the approval process said:Is an evidence-based process used by the National Medicines and Therapeutic Committee at the moment? I don’t think so. Is the committee applying an evidence-based framework in decision-making processes? I don’t think that’s what is being done at the moment. (Participant from the MoHSW)The fourth edition of the STG/NEML was released in July 2013, and contained nearly all the medicines that were initially proposed for addition. In contrast to the previous editions, the new STG/NEML contains, as appendices, application forms for addition, deletion and change of dosage form, strength and indication of the listed medicines. The guiding criteria include efficacy, safety, cost-effectiveness, cost comparison and budgetary impact. These criteria must be supported by relevant evidence, such as the results of clinical trials conducted in Tanzania. An excerpt from the application form reads:Reasons why the proposed drug is preferred to drugs already in the NEML . State briefly the results of clinical trials conducted in Tanzania […]. If no official trials, state personal experience and/or submit documentary proof.The most important finding derived from this study is that essential medicines in Tanzania were largely selected through an experience-based process, in contrast to the evidence-based approach that was recommended by the WHO Expert Committee on Selection and Use of Essential Medicines in 2002 The review involved experts who were selected mostly from referral hospitals and the Ministry of Health and Social Welfare in a process that can be described as implicit and not sufficiently consultative. Participation by a wide array of stakeholders is important to ensure that the needs existing across all levels of the healthcare system are reflected in the STG/NEML. The effectiveness of guidelines is often compromised by the development process, and those guidelines that are imposed by higher levels have a high probability of being under-utilised or even rejected by healthcare workers We found that efficacy, safety, availability and affordability were the most commonly cited criteria employed in medicine selection. These criteria are to a large extent consistent with those recommended by the WHO The criterion of cost-effectiveness was not used despite being one of the most important criteria employed by medicine management committees in developed countries to inform formulary decisions Experience rather than scientific evidence played a major role in the decision-making processes. In the few cases where scientific evidence was claimed to have been used, there was neither a systematic search nor an appraisal of the evidence and evidence summaries were not generated to aid decision-making. Even for criteria such as availability and affordability, which are relatively easy to apply compared to efficacy and safety, evidence also mainly came from experience and not official sources such as the TFDA, Medical Stores Department or the International Drug Price Indicator Guide. This could be explained by the inexperience of the experts involved in the review process in using an evidence-based approach, the lack of guidelines on how to do the review and time constraints. Commitment to the use of research evidence and the availability of adequate infrastructures, tools and expertise are essential to facilitate evidence-informed decision-making This study employed a qualitative approach, which is suitable for phenomena of which prior knowledge or understanding is limited The expertise of the principle investigator in essential medicine and the functioning of the medicine selection committees in Tanzania was instrumental in conducting this study. The potential influence of his prior experience on the interpretation and discussion of the findings was minimised through observing a well-accepted protocol for qualitative studies, and by involving co-authors in the analysis of the data, as well as in other phases of the study.The committees were faced with many challenges in performing the reviews. Firstly, it appears that they were not given any training or guidelines about how to update the STG/NEML; secondly, the majority were performing a review for the first time; thirdly, they were not given sufficient time or other resources to carry out the review and, lastly, even the organisers were constrained by limited capacity and resources to carry out a smooth review process. Therefore we believe that all those who were involved performed the review of the STG/NEML to the best of their ability. This paper has thus identified areas that can be improved in future reviews.Recent decisions about the selection of essential medicines in Tanzania were made by committees of experts, who largely use experience and discretionary judgement, leaving evidence with only a limited role in decision-making processes. This practice increases the risk of adopting ineffective and costly interventions that may not be worth implementing. Because of this, the health authorities in Tanzania should take the necessary measures to ensure that limited health resources are allocated to proven interventions with the greatest potential to reduce the burden of disease and meet other public health goals. This can be achieved through the systematic application of relevant evidence-based criteria in priority-setting decisions between competing interventions. There may be several reasons for the current limited use of evidence in the decision-making process, but one hypothesis that remains to be explored is whether training experts in evidence-based decision-making would lead to a better and more explicit use of evidence.


Evidence mapping describes the quantity, design and characteristics of research in broad topic areas, in contrast to systematic reviews, which usually address narrowly-focused research questions. The breadth of evidence mapping helps to identify evidence gaps, and may guide future research efforts. The Global Evidence Mapping (GEM) Initiative was established in 2007 to create evidence maps providing an overview of existing research in Traumatic Brain Injury (TBI) and Spinal Cord Injury (SCI).The GEM evidence mapping method involved three core tasks:1. Setting the boundaries and context of the map: Definitions for the fields of TBI and SCI were clarified, the prehospital, acute inhospital and rehabilitation phases of care were delineated and relevant stakeholders  who could contribute to the mapping were identified. Researchable clinical questions were developed through consultation with key stakeholders and a broad literature search.2. Searching for and selection of relevant studies: Evidence search and selection involved development of specific search strategies, development of inclusion and exclusion criteria, searching of relevant databases and independent screening and selection by two researchers.3. Reporting on yield and study characteristics: Data extraction was performed at two levels - 'interventions and study design' and 'detailed study characteristics'. The evidence map and commentary reflected the depth of data extraction.One hundred and twenty-nine researchable clinical questions in TBI and SCI were identified. These questions were then prioritised into high (n = 60) and low (n = 69) importance by the stakeholders involved in question development. Since 2007, 58 263 abstracts have been screened, 3 731 full text articles have been reviewed and 1 644 relevant neurotrauma publications have been mapped, covering fifty-three high priority questions.GEM Initiative evidence maps have a broad range of potential end-users including funding agencies, researchers and clinicians. Evidence mapping is at least as resource-intensive as systematic reviewing. The GEM Initiative has made advancements in evidence mapping, most notably in the area of question development and prioritisation. Evidence mapping complements other review methods for describing existing research, informing future research efforts, and addressing evidence gaps. All stakeholders in health care like to have research evidence to inform their decision-making. Some decisions relate to questions that are narrowly focused, such as the effectiveness of an intervention or the accuracy of a diagnostic test. Others are much broader, such as the relative effectiveness of various treatment options available for a particular condition. The methods of systematic reviews are well developed for the former group, but the latter poses many challenges for locating, organising, collating and synthesising information into a useful, understandable format, as it requires an organised up-to-date reference source of research evidence addressing broad content areas.For this task, existing evidence resources seem generally inadequate. Finding all relevant publications on a topic using PubMed-type databases is a long and laborious task. Systematic reviews are rarely completely up-to-date. Being the product of a largely investigator-driven collaboration, the Cochrane Library doesn't necessarily contain reviews encompassing all questions in a given topic area. Furthermore neither searches for primary studies, nor systematic reviews are well suited to identifying important evidence gaps.Techniques for describing existing evidence in a broad content area are relatively new, and have been called evidence mapping and scoping studies ,2. How t1. Setting the boundaries and context of the topic area in question;2. Searching for and selection of relevant studies; and3. Reporting on yield and study characteristics.In 2007 the Global Evidence Mapping (GEM) Initiative was established with funding from the Victorian Neurotrauma Initiative (VNI), a research funding agency established by the Victorian Government's Transport Accident Commission and Department of Innovation Industry and Regional Development . The GEMThe GEM Initiative aimed to map the available research for the prehospital, acute inhospital and rehabilitation phases of care for TBI and SCI. The breadth of clinical research questions and volume of studies in these areas is considerable. The evidence mapping approach was thought to be the most appropriate method because its main purpose is to provide an overview of a broad range of research questions and identify evidence gaps -5. In crThis article focuses on the GEM Initiative's evidence mapping methods and illustrates how these methods were applied to neurotrauma (TBI and SCI) topics. The results of the mapping process itself are also briefly described.Figure Because evidence mapping outputs have stand-alone merit and application, the GEM evidence mapping process, like previously published evidence mapping methods, is not an 'all or nothing' proposition; the number of steps undertaken and the breadth of the topic area covered can therefore vary according to available resources. An evidence map can form the basis of a scoping review or number of individual systematic reviews, as evidence mapping produces source materials for both these activities Table . FurtherFollowing is an in-depth description of the GEM Initiative evidence mapping methods, illustrated with specific examples from our evidence maps. Emphasis has been placed upon the unique aspects of these methods; the use of a mapping workshop to identify clinical research questions, development of questions, question prioritisation and the two levels of data extraction and presentation.The GEM Initiative approached this task using two distinct steps - question development and question prioritisation.The question development phase involved four broad data collection exercises designed to gain an overview of the scope of the field and gather perceptions of key stakeholders - patients, carers, clinicians, researchers and policymakers - regarding the key clinical issues in the field of interest. The data gathered from these exercises was used to develop searchable clinical questions.Consultation with experts has been identified as a useful strategy for setting the boundaries and context of an evidence map ,5. ExperTo frame the TBI and SCI evidence maps, the GEM Initiative consulted with local and international clinical and research leaders in these fields. These consultations involved introducing the GEM Initiative and the concept of evidence mapping, followed by an unstructured discussion designed to enhance our understanding of specific aspects of TBI and SCI care. For example, our discussion with experts in SCI rehabilitation encompassed common problems experienced by SCI patients in this phase of care and the healthcare disciplines involved in managing these.Although the GEM Initiative focused on clinicians and researchers, the other key stakeholders with expert knowledge in this area - patients, carers and policymakers (who were involved in later stages in the GEM process) - can add different perspectives to this process and could also be involved at this stage.To identify existing relevant reviews and guidelines relating to TBI and SCI in each of the phases of care, a preliminary literature search was conducted by a GEM Initiative project officer who had extensive Cochrane systematic review experience. Major medical databases were searched using relevant Medical Index Subject Headings (MeSH) and other terms based upon the clinical knowledge of the GEM researchers. The focus of the search was on reviews and guidelines rather than primary studies. Relevant reviews and guidelines were used to identify clinical questions, describe the extent to which they had been addressed by existing literature and identify additional key literature search terms. This information further developed the scope of the map.To ensure the evidence maps were comprehensive and useful, key stakeholders involved in all phases of TBI and SCI patient care - patients, carers, clinicians, researchers and policymakers - were invited to participate in mapping workshops. This is an enhancement on the techniques for obtaining expert opinion described by Katz et al. .The primary aim of the mapping workshops was to gather stakeholder perspectives on key clinical questions, outcomes, patient factors and contextual factors affecting TBI and SCI care. A total of four mapping workshops were conducted; one each for the pre-hospital and rehabilitation phase of care in which both SCI and TBI stakeholders were involved; and separate TBI and SCI workshops dealing with the acute inhospital phase of care.The nominal group technique  was usedA professional facilitator with experience in the nominal group technique ran the workshops. The mapping workshops varied in length from one to two hours according to the breadth of the field and participant availability. A consistent workshop format was employed comprising:Introduction: Aims, key definitions, confidentiality/consent procedures.• Brainstorming: Writing of ideas on individual adhesive notes, without discussion. The PICO format  was suggested but not mandatory. Clinical issues could include diagnosis, prognosis, interventions, and service delivery/organisation. The adhesive notes were then placed on large sheets around the room.• Prioritisation: During a short refreshment break, GEM Initiative researchers grouped similar notes. Participants then prioritised the notes by placing a limited number of adhesive dots next to the notes they perceived as most important.• Discussion: Open discussion was conducted on the ideas generated, the results of the prioritisation and any further issues identified. This was a valuable way of gaining insight into other perspectives and elaborating, modifying and developing consensus [• onsensus . Where tConclusion: Participants were thanked and invited to participate in subsequent question development and prioritisation activities and/or receive a copy of the final report.• The mapping workshops resulted in an unstructured collection of ideas and PICO terms of interest as well as a sense of the relative importance of these ideas.An online survey was used to augment the information gathered in the mapping workshops. Only one published evidence mapping method has incorporated the use of a survey. The GEM Initiative evidence mapping survey built on the method described by Katz et al.  by placiSurvey recipients were identified through GEM clinical and research contacts, primary searching of hospital and academic websites and organisational mailing lists, including a mailing list of clinicians and researchers on the VNI database to which GEM staff were blinded. The survey comprised three sections:Introduction and background: Brief background to the project, purpose, confidentiality and contact information.1. Survey of PICO terms of interest: Participants were asked to fill in a minimum of two blank PICO templates identifying a series of clinical problems, therapies and/or outcomes that they considered important in their specialty area.2. Demographic details: Profession/patient group, years experience or years a patient, extent of research/clinical involvement in TBI/SCI, geographical location, gender.3. The survey data were more closely aligned to the PICO format than the mapping workshop data, as the survey responses were mandated under PICO headings.These four activities (1A -1D) generated a very large volume of unstructured ideas including single words, PICO fragments, statements and questions. They varied in format, language, perspective and emphasis because a range of stakeholders was involved in the process.Organisation of this heterogeneous data was required in order to promulgate a key aim of the GEM Initiative - the identification of evidence gaps - by searching for literature pertaining to the identified issues. We addressed this need by transforming the data into a list of 'answerable' clinical research questions. An 'answerable' question seeks specific knowledge, is framed to facilitate literature searching and therefore, follows a semi-standardised structure ,9. The c1. Entering data into an electronic database (Microsoft Excel).2. Coding the clinical problems described using codes from the International Classification of Functioning, Disability and Health (ICF) ,11. This3. Collating coding information to identify the range and frequency of all codes represented in the data.4. Conducting a series of half-day question construction meetings in which the GEM Initiative research team transformed the data into a list of answerable clinical research questions encompassing all ICF codes represented in the data.The example in Table A range of question development strategies have been reported in published evidence mapping projects. Arksey  used quaThe GEM Initiative approach to question development built upon these strategies by using the ICF framework to organise large amounts of unstructured data and detailing the specific tasks involved in transforming this data into answerable clinical research questions. This approach facilitated the development of a large number (129) of answerable research questions across an entire research field. The comprehensive question development approach used by the GEM Initiative ensured that all of the ideas identified by the key stakeholders in steps 1A - 1D - patients, carers, clinicians, researchers and policymakers - were represented in the final list of answerable clinical research questions for each phase.Bates et al.  recognisThe GEM Initiative involved the key stakeholders from the question development process in prioritising the list of clinical research questions from Step 1. An online survey of patients, clinicians and researchers was conducted. Survey recipients included participants from the mapping workshop, the GEM advisory group and other representatives with expert knowledge of TBI and SCI. Survey respondents were asked to rank each searchable clinical question on three domains - clinical importance , novelty  and controversy ('What is the level of variability regarding opinion and/or practice for this question?') - using a scale of 1 to 4, where:1 = not at all2 = a little3 = a moderate amount4 = high.The questions were then ranked into two broad categories:High priority questions: Questions where the majority of respondents ranked 'Clinical Importance' as high AND ('Novelty' OR 'Controversy' as high or moderate).• Low priority questions: All other questions.• This ranking procedure was arbitrary and driven by the imperatives of the GEM Initiative project; other research groups may use other ranking parameters and divisions consistent with the focus of their research activity. For example, cost-effectiveness or feasibility of undertaking primary or secondary research, rather than clinical importance, may be deemed the most important determinants of which questions to pursue.The evidence search and selection methods used by the GEM Initiative were consistent with accepted principles of systematic reviewing. Specifically, these were:1. In-depth search strategies using applicable MeSH headings and keywords ,13.2. Pre-determined inclusion criteria ,13.3. Searching a broad range of medical databases.4. Review of retrieved citations/full-text articles against the question-specific inclusion criteria by two independent reviewers to reduce bias ,13.As such methods are extensively documented in systematic review methodology literature, only a brief description of their specific application to the GEM Initiative follows. Examples are contained in Appendix 1.An information specialist was used to develop and run searches and collate search results. Search strategies were built by combining at least two of:• Core terms representing TBI and SCI• Filter terms, for example to target rehabilitation literature• Intervention terms specific to the clinical questions being mapped.GEM Initiative researchers with a clinical background  contributed their clinical knowledge of the subject areas to search strategy development. Abridged versions of the search strategy were used for the databases that had a simpler search interface or smaller content, such as non-English databases, as these databases could not accommodate a large number of search terms ,14.A priori eligibility criteria were developed regarding study type , participants (explicit definitions of TBI and SCI) and phase of care .Given the breadth of topics covered, question-specific refinements to the inclusion criteria were sometimes required after preliminary database searching and review; for example, to determine which clinical sub-categories of a particular disorder were most relevant. Such post hoc refinements, which can only occur when some literature has been examined , also aiMajor medical databases (including non-English) and trials registries were searched for all phases of care covered by the GEM Initiative. Searching of further databases tailored to the specific needs of each phase/question was also conducted. For example, for the question pertaining to spasticity, three additional databases were searched to reflect the multidisciplinary nature of this intervention in addition to the databases routinely searched (Appendix 1).For all searches there was no restriction on year of publication or language. For each question all of the chosen databases were searched on the same day and a reference library (Endnote) of search results by database was compiled. The reference library was then de-duplicated for the purpose of screening and selection.Retrieved citations were reviewed against the question-specific inclusion criteria by two independent reviewers. Where necessary, a third reviewer was used to resolve disagreements by breaking a tie.Full-text articles identified during screening were procured either directly or via order from university/hospital libraries.Full-text articles were reviewed against inclusion criteria as described above (3D).Records of screening and selection decisions  were systematically kept.As a search of all the major and auxiliary databases may inadequately reflect the total relevant research on any topic ,16, refeThe GEM Initiative undertook two levels of evidence mapping reporting; an overview of interventions (where relevant) and study design of included studies and a more in-depth exploration of study characteristics. In each case, the output tables and text identified and articulated evidence gaps by describing how the relevant literature addressed the research question. These levels of evidence mapping are described and illustrated below with examples from the GEM Initiative evidence maps.A relational data extraction database was created (Microsoft Access), which could be interrogated to explore and query the evidence map .prehospital evidence map, an in-depth data extraction and review of study quality was undertaken. However this process proved to be unfeasible once the volume and breadth of literature in subsequent phases became apparent. Furthermore, it was felt that quality appraisal, summary of findings and specific outcomes of individual studies was better conducted within a formal systematic review, and we have distinguished evidence maps and systematic reviews in part on this level of data extraction and reporting as outlined earlier. This resulted in the development of the two data extraction protocols that were used in the rehabilitation and acute inhospital evidence maps:For the Extracting to this level involved identifying the number of studies by study design for each intervention addressing the question. Where the question did not pertain to an intervention, relevant studies were listed by study design only. For all phases of the project, study design was classified according to the definitions outlined in the Australian National Health and Medical Research Council (NHMRC) levels of evidence  and StraThe following data were extracted: study design, country, sample size, source population, interventions, outcome measures, patient factors . This facilitated a more in-depth evidence map for the clinical question, as reflected by both the output table and the scope of the commentary on evidence Table .Use of these two data extraction methods facilitated a broad coverage of clinical questions in these areas within project resources. The decision regarding which data extraction method to use involved consideration of a range of factors including the nature and complexity of the question and the need to balance consistency, comprehensiveness and question coverage across both TBI and SCI topics.http://www.evidencemap.org. This site contains a searchable database of all TBI and SCI topics for which an evidence map has been completed. For each topic, either the output tables ('maps') alone , acute inhospital  and rehabilitation  phases of TBI and SCI care.The prioritisation procedure undertaken in the GEM Initiative resulted in 60 of the 129 clinical research questions being identified as high priority; 30 in each of TBI and SCI.Since 2007, 58 263 abstracts have been screened, 3 731 full text articles have been reviewed and 1 644 relevant neurotrauma publications have been identified.Data extraction has been completed to produce evidence maps for fifty-three high priority questions. For 11 of the 53 questions , gaps in primary research evidence were identified:• For six questions  no studies were identified.• For a further four questions pertaining to interventions , only case series, case study or cross-sectional study designs were identified. These represent neurotrauma research topics in which comparative studies  are needed, as reflected by the NHMRC levels of evidence for intervention studies .• For a further question pertaining to prognosis in SCI, no prospective cohort studies were identified, which are the highest ranked primary study for investigating prognosis .TBI topics with primary evidence gaps included prehospital therapeutic hypothermia and participation in leisure and social activities. SCI topics with primary evidence gaps included prehospital antiemetic medications and interventions for sleep disordered breathing.A further 12 questions  contained secondary evidence gaps; more than one comparative study were identified, but no systematic review. These are neurotrauma research topics in which research synthesis may be warranted. TBI topics with secondary evidence gaps included prehospital hyperventilation and interventions for minimising drug and alcohol misuse. SCI topics with secondary evidence gaps included interventions for atelectasis/chest infections and interventions for optimising nutrition and body weight.The GEM Initiative has produced evidence maps that may be useful to research funders, researchers, clinicians and other research consumers. An extensive list of prioritised research questions in a topic area, even in the absence of study retrieval and data extraction, is a potential springboard for research, research funding and policy development. Because the question development and prioritisation process is driven by consultation with key stakeholders - patients, carers, clinicians, researchers and policymakers - this list is more likely to be relevant to real-world issues than research questions driven solely by academic researchers. Our experience of the power of engaging with a wide stakeholder group is shared by Arksey et al. , who refFunding agencies can use evidence maps to access completed/ongoing studies in a topic area; for example, studies using intrathecal baclofen for spasticity in TBI. This information can aid in evaluating the need for further funded research when reviewing a grant application. The collection of evidence maps across a whole topic area  also highlights areas of high and low research activity, potentially informing strategic research funding decisions. The VNI exemplified this application of evidence maps through its explicit instructions to funding applicants to consult the GEM evidence maps in the process of preparing their applications . Other rResearchers can use evidence maps to inform research decisions and designs. Evidence maps parallel all steps of a traditional systematic review except in-depth quality appraisal and synthesis. Systematic reviewers can therefore use evidence maps as a resource for updating an existing systematic review, or to evaluate the feasibility and resource requirements of a systematic review based upon the search yield, and proceed to full study appraisal and synthesis using this resource. One example of such a review is a Cochrane review of interventions for managing skeletal muscle spasticity, which built upon the GEM evidence map of this topic . As illuClinicians can use evidence maps to rapidly access research information pertaining to specific populations and interventions. For example, a case manager could access literature about the effectiveness of community integration programs for TBI specifically in paediatric populations, or focusing on computer-based interventions. However, it should be emphasised that evidence mapping does not purport to guide clinical practice, as the studies in the GEM Initiative evidence maps have not been subject to quality appraisal or had their results synthesised.This limitation of evidence mapping necessitates caution especially in cases where a large volume of evidence is identified for a topic. For example, for eleven of the GEM topics  over 50 relevant English articles were identified. These represent neurotrauma research topics in which knowledge translation research, rather than more primary or secondary studies, may be appropriate. However, this cannot be surmised from the yield alone, but by combining this with an examination of the quality of the comparative studies and/or systematic reviews. Should the research be of high quality and consistency, knowledge translation research may be warranted. If this is not the case, further primary or secondary studies are required - just as in cases where there is a low volume of evidence. Avoiding the potential hazard of misreading evidence maps is best facilitated by transparency in the description of the purpose and methods of evidence mapping, in particular by drawing a clear distinction between evidence mapping and systematic reviewing Figure ; ensurinIdentification of and prolonged engagement with the likely end-users  starting early in an evidence mapping project enables tailoring of evidence mapping activities and outputs to their priorities. Although there was provision in the evidence mapping process for the identification and mapping of non-intervention questions, the existing GEM evidence resource predominantly contains maps pertaining to interventions. This could be in part due to the use of PICO to guide question development; it may be a reflection of the priorities of those involved in question development; or it could be a simple illustration that most clinical research questions are intervention-focused. In this context, some tailoring to the evidence mapping process could be necessitated  if outputs focused on non-intervention issues are identified as important during initial consultation with end-users.There are many resource-intensive steps in evidence mapping. As described, the GEM Initiative evidence maps involved the generation of 129 clinical questions and independent review of tens of thousands of abstracts and thousands of full-text articles by the 3.5 EFT project staff. A similar level of staffing has been reported by Arksey et al. , plus thOur experiences in this project combined with the available cost data reinforce the statement by Arksey et al.  that "thSince its inception in August 2007, the GEM Initiative has advanced the field of evidence mapping. The in-depth question development and prioritisation methods - in particular the use of mapping workshops to engage a broad range of key stakeholders - result in evidence maps that are more likely to be relevant to patients, carers, clinicians, researchers and policymakers. This question development method could easily adapt to any situation requiring the generation of clinical questions.The study search and selection methods parallel accepted systematic review approaches, creating potential synergies with systematic reviewing. The possibility of fruitful collaborations between evidence mappers and systematic reviewers has been identified by other evidence mapping authors ,5. GivenTwo levels of data extraction have been explored by the GEM Initiative; an overview of interventions and study design, and a more in-depth description of study characteristics. These highlight the many potential outputs and applications of evidence maps and the ability of the evidence mapping process to be tailored to specific needs. In highlighting the usefulness of evidence maps in scoping research in broad topic areas, it is acknowledged that this breadth of coverage comes at the cost of evidence synthesis and quality appraisal, and therefore evidence maps cannot be used to guide clinical practice. This limitation should be borne in mind when reporting on and disseminating evidence maps.Evidence maps complement other forms of reviews for use by researchers, clinicians, patients, carers and policy-makers with an interest in accessing and using evidence resources.1. Muscle Spasticity/2. spastic*.mp.3. 1 or 24. Botulinum Toxin Type A/5. exp GABA Agonists/6. exp gamma-Aminobutyric Acid/7. "gamma amino butyric acid*".mp.8. or/4-79. "Range of Motion, Articular"/10. stretch*.mp.11. Splints/12. position*.mp.13. exp Physical Therapy Modalities/14. physiotherap*.mp.15. Occupational Therapy/16. or/9-1517. Rhizotomy/or rhizotomy.mp.18. myelotomy.mp.19. Cordotomy/or cordotomy.mp.20. cordectomy.mp.21. or/17-2022. or/8,16,2123. 3 and 22Subacute care and long term care through community management. Subacute care is defined as medical and skilled nursing services provided to patients who are not in an acute phase of an illness but who require a level of care higher than that provided in a long term care setting. Long term care includes community management and is defined as extended care for chronic conditions or disabilities requiring periodic, intermittent, or continuous care .Major medical databases:Cumulative Index to Nursing and Allied Health Literature (CINAHL), OVID Embase, OVID Medline, PsycINFO, Cochrane Library, ISI Web of Science, PubMedNon-English databases:India Med, Korea Med, LILACS, PanteleimonTrials registries:WHO - International Clinical Trials Registry Platform (ICTRP), The UK National Research RegisterFurther databases (to reflect the multidisciplinary nature of this intervention):Physiotherapy Evidence Database (PEDro), OTSeeker, PsycBITE673 titles/abstracts screened152 full-text articles ordered and reviewed61 articles included in evidence map The authors declare that they have no competing interests.RG and AC conceived the GEM Initiative; all authors were involved in undertaking research for the GEM Initiative and developing its evidence mapping methods; PB drafted the manuscript; all authors read and approved the final manuscript.The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2288/11/92/prepub


The arguments presented here are generally relevant to all areas of health care, though mental health applications provide the primary context for this essay.Both scientists and the public routinely refer to randomized controlled trials (RCTs) as being the 'gold standard' of scientific evidence. Although there is no question that placebo-controlled RCTs play a significant role in the evaluation of new pharmaceutical treatments, especially when it is important to rule out placebo effects, they have many inherent limitations which constrain their ability to inform medical decision making. The purpose of this paper is to raise questions about This article first traces the history of RCTs, and then evaluates five of their major limitations: they often lack external validity, they have the potential for increasing health risk in the general population, they are no less likely to overestimate treatment effects than many other methods, they make a relatively weak contribution to clinical practice, and they are excessively expensive . Next, the nine Hill criteria are presented and discussed as a richer approach to the evaluation of health care treatments. Reliance on these multi-faceted criteria requires more analytical thinking than simply examining RCT data, but will also enhance confidence in the evaluation of novel treatments.Excessive reliance on RCTs tends to stifle funding of other types of research, and publication of other forms of evidence. We call upon our research and clinical colleagues to consider additional methods of evaluating data, such as the Hill criteria. Over-reliance on RCTs is similar to resting all of health care evidence on a one-legged stool. In fact, one could argue that most major discoveries in health have evolved from observations first documented in case reports and case series. Randomized controlled trials (RCTs) are often mounted later to validate an intervention, especially in comparison with a placebo, but as is well-recognised, they are rarely major sources of scientific discovery. In spite of the fact that observational data hold a time-honored place in medicine, 21de facto standard by which all forms of evidence are evaluated. No other study design rivals the RCT's ability to eliminate selection bias and reduce the risk of a serious imbalance in known and unknown factors that could influence outcomes (when the randomization procedure is executed properly). Our concern is that evidence based medicine has made a leap from considering RCTs to be a high standard to being the only standard. The primary purpose of this paper is to question the over-valuation of RCTs that Rosenbaum referred to as a form of tyranny [There are many reasons why RCTs have become the  tyranny . Our preth-18th centuries. A dietary factor was suspected in 1601 when Captain James Lancaster administered a tablespoon of lemon juice per day to each sailor on one ship: 0% of those on the ship with lemon juice rations died, while 40% of those on the three ships without lemon juice were dead halfway through the journey to India. Replications and extensions followed: e.g., in the mid-18th century a ship physician named James Lind conducted a study in which early signs of scurvy were effectively treated in those randomly assigned to receive citrus, thereby showing the ability of citrus to reverse the disease in its early stages [The use of randomized controlled trials predates the actual term RCT by several centuries. One of the earliest documented applications to health was the proof that citrus prevented scurvy . Scurvy y stages .It was Sir Austin Bradford Hill , a British statistician and epidemiologist, who promoted the use of randomization for clinical trial research employed to test health care interventions, a position he took prior to World War II . HoweverIn 1962, RCTs were still quite rare, yet they were the norm in 1992 when the Evidence-Based Medicine Working Group published their seminal paper . This clThe new effectiveness criterion had a significant impact on standards of scientific evidence for the next 30 years; however, by the end of the 20th century, even FDA was beginning to question the need for multiple Phase III RCTs as proof of effectiveness to justify market approval . CriticiSince the era of Hill's methodological contributions, various groups have promoted the idea of levels of evidence, but recently, some have questioned the position of RCTs in a hierarchical model. For instance, Ghaemi commented that "...the key feature of levels of evidence to keep in mind is that each level has its own strengths and weaknesses, and as a result, no single level is completely useful or useless" (p.10) . FurtherRCTs bolster confidence in causal claims related to the effects of a treatment by eliminating threats to internal validity. They do this by using the tools of random assignment and experimental control. However, a medical science that relies primarily upon achieving internal validity with a relative neglect of external validity  is at great danger of ignoring the individual and context characteristics that impinge upon treatment outcome. The five criticisms of RCTs reviewed below redirect us to consider a more diverse medical science.RCTs usually lack external validity. For any given study, clinicians should reasonably ask, to whom do these results apply? Particularly in mental health, it has been shown that RCTs tend to employ such strict inclusion and exclusion criteria that the participants are not representative of the general population of individuals with a given disorder. As Concato and colleagues pointed out [1. nted out   and generalizability of RCTs have been questioned for a number of years , and a r2. In the long term, RCTs may actually increase health risks in the general population. As a corollary to the issue of external validity, excessively constrained sampling approaches can have consequences for population health. Health researchers sometimes distinguish between efficacy and effectiveness. One might say that all RCTs are efficacy studies ), demonstrating benefit in ideal conditions - particularly by selecting only relatively 'pure' samples of people with no co-existing problems. The lack of external validity caused by these atypical samples means that the drugs are approved without evidence of effectiveness . Since people with complex health problems  are not usually participants in RCTs evaluating mental health treatments, it is reasonable to consider the possibility that patients prone to adverse effects are not studied. But approval of the drug then has the potential of increasing the risk of those vulnerable individuals to exacerbation of their pre-existing health problems. One could argue that effectiveness studies using designs such as case-control methodology that have more external validity would be more informative about the health impacts of a new drug for the broader population. Without such forms of 'observational' studies, drug approval based only on RCTs may be increasing the health risks to the populace.The premise that RCTs are the only form of evidence capable of providing an unbiased estimate of treatment effects is false. A fundamental reason for the elevated status of RCTs is the conviction held by many that all other forms of evidence, even cohort and case-control studies, overestimate treatment effects. However, some published research does not support this premise. Concato and colleagues [3. lleagues  evaluatelleagues . So evenRCTs are unable to tell clinicians everything they really want to know. Comparisons between groups of individuals can obscure processes operating within individuals. However, RCTs only reveal differences between treatment and control group means - these aggregated results are uninformative of the potential benefit of a treatment for any individual in the study, and more importantly, for the individual who was not in the study but whose treatment decisions will nevertheless be made on the basis of the study [4. he study . What clhe study . There i5. The excessive expense of RCTs leads to vulnerabilities in the quality of evidence. On average, a Phase III clinical drug trial costs > 15 million USD [lion USD , which r• Part of the cost of an RCT in some countries is significant payment to participants, which has led to a practice of 'guinea-pigging' -19 in wh• The expense of RCTs adds to the cost of medications that are eventually approved. Hence, the broader population has the right to ask if they should be paying for research that has questionable generalizabilty to the general populace.• The significant costs of RCTs add pressure to bury negative reports that will prevent a medication from moving to market, thereby removing the possibility of development cost recovery by the pharmaceutical company. The total cost of developing a new drug is now in excess of US$1.3 billion  - a subs• The expense of RCTs biases their implementation toward areas for which commercial funding is possible; i.e., pharmaceutical interventions. Million dollar grants are less available for the evaluation of non-conventional forms of treatment . When the valuation of RCTs as "gold standard" is combined with a systematic bias toward commercial applications, the weight of evidence itself becomes biased in favor of pharmaceutical treatments.• A final adverse side effect of the high costs of RCTs, at least with regard to psychiatric drugs, is the brief intervention period that is typically employed to evaluate efficacy. Most RCTs in mental health now last only 6-8 weeks, which is more economical than a 12-week or longer trial more typical a decade ago . But clThe unfortunate result of these financial pressures is that clinical decisions are made to administer medications to patients for many years, but the decisions themselves may be based on very brief trials conducted with very unusual people , and without evidence for other interventions which may be effective.In conclusion, dismissing or devaluing rigorously collected data obtained with study designs other than RCTs results in a science that is inherently unsupportable - as shaky as a one legged stool. Indeed, as Concato and colleagues argued, the evidence does not support the commonly accepted concept of a hierarchy of study designs employed for clinical research . If RCTsHill was primarily concerned with causation of disease when he outlined nine considerations for determining causal relations in epidemiology, using the association between smoking and lung cancer as his illustration. However, the criteria he defined in his 1965 presidential address to the Section of Occupational Medicine of the Royal Society of Medicine  are alsoStrength. The strength of association between an outcome and a putative causative agent is an important signal of a causal relationship. All things being equal, a strong association is less likely to occur from extraneous than causal effects. Hill did caution that no matter how slight an association may be, it should not be dismissed until argument for or against causality exists, and he used as an illustration the evidence that relatively few persons harboring meningococcus actually become ill with meningococcal meningitis. As discussed above (item 4), the fact that a causal relation between two variables can be detected within persons but are not observed when data are aggregated should caution against methods that rely exclusively upon indices of group differences. Therefore, methodology that relies solely on mean differences is a limited approach, and alternatives that accommodate individual differences may bolster conclusions. Such alternatives might include qualitative methods [1. Defining the strength of association between treatments and symptom severity has been an especially contentious issue in psychiatry, particularly with respect to depression. There has been much concern that publication bias against negative findings, discussed above, results in approval of medications in spite of multiple trials failing to show benefit over placebo. Using Freedom of Information legislation to gain access to unpublished studies of anti-depressant efficacy, Kirsch and colleagues showed very little difference between medication and placebo in 35 RCTs on four SSRIs . They us2. Consistency. Hill used this term to refer to obtaining similar results across different research sites and methodologies, something we might now call replication. As he pointed out, repeating studies is necessary to prove the obtained association is not a result of confounding variables in one setting or group. Similar results from independent researchers using different methods are more convincing than a single study. In the social sciences, this is generally referred to as method triangulation, or multi-methods [-methods .3. Specificity. Hill recognized that two different patients may have varied outcomes from treatment simply based on individual variables. Accordingly, it is not always possible to demonstrate specificity, even when a causal association exists. For example, Hill noted that smokers have a higher death rate than non-smokers for many causes of death. However, the relative increase for other diseases is modest (10-20%) while the increase for lung cancer is 900 - 1000%. Such specificity in the magnitude of the association provides important evidence for a causal association. In contrast, there are multiple determinants of mental disorders , making it very difficult to estimate the specific contribution of any particular predictor variable. Because of this multicausality, this criterion may not always be applicable to evaluating causality in areas such as mental health, where family dynamics and other social issues can play such a prominent role. Qualitative methods may also play a role in the generation of hypotheses in such situations.4. Temporality. Temporality refers to the common sense notion that the cause always precedes the outcome. Temporality is crucial for determining direction of causality: e.g., does the decrease in a factor result in a disease, or does the disease result in the decrease of that factor? In mental health, elucidating this relationship can be difficult, but not impossible. For instance, within-subject crossover study designs  can be useful for investigating the effect of a treatment. Assuming the existence of minimal carryover effects and sufficient time devoted for washout, this methodology can show (a) whether there is an improvement in the patient's condition, and (b) whether the treatment was actually the factor causing the improved outcome and not a confounding variable. We note that when it is possible to randomly allocate the treatment sequence, such cross-over designs can be considered RCTs.5. Biological Gradient. A biological gradient is best described as a dose-response curve: increased treatment would presumably result in a proportionate increase in the effect. Hill realized that this criterion might not be applicable to all research fields, and he recommended that it be considered only when logical. Within psychiatry, outcomes vary largely because of individual differences. For example, it is clear that the optimum dose of a given medication is not necessarily the highest one tested. Even in the Kirsch meta-analysis described above, the relationship between severity and response to medication was not linear [t linear . Where t6. Plausibility. Plausibility refers to whether the cause and effect can be reasonably connected given the current state of knowledge within the discipline. Importantly, Hill states that new findings must not be immediately dismissed if they do not fit in with current knowledge ("dogma") on the subject. As Kuhn observed, considerable evidence disconfirming the accepted view must accumulate before new ways of thinking can emerge from new data [new data . Further7. Coherence. Similar to plausibility, coherence refers to the agreement of a study's findings with what is already known. According to Hill, the cause-effect interpretation of the data should not seriously interfere with current knowledge of the disease. Furthermore, Hill mentioned that basic lab evidence should not be a requirement, primarily because some outcomes would be difficult to demonstrate in a controlled environment. For instance, the search for animal models of human mental disorders has a long history, but no matter how promising some of the achievements, science can never validate an animal model for internal mental functions such as delusions, suicidality, euphoria, or hallucinations. For this reason, the criterion of coherence may not be a requisite standard for evaluating a novel approach to something like mental disorders.8. Experiment. The criterion of experimental evidence can be fulfilled in many ways. One reason for the wide acceptance of RCTs for pharmaceuticals is the well-documented placebo/expectancy effect in psychiatry. As Kirsch et al. commented regarding their meta-analysis, the longer-term improvements for medication observed in those 35 RCTs seemed to be a result of the decreasing placebo response which had been quite high (> 80%) [ (> 80%) . Placebo9. Analogy. The notion that a similar cause results in a similar outcome is referred to as analogy. Hill described this criterion as accepting less evidence based on previous results, citing the case of pregnancy and thalidomide. If a new drug were to demonstrate negative consequences during pregnancy, we would be less hesitant to stop its use even if little evidence of harm exists, because of the tremendous social and personal cost that resulted from thalidomide. This criterion may not be relevant to all areas of health care treatment, particularly mental health. As with the criteria of specificity and biological gradient, the issue comes down to the multicausality of mental disorders and the multifinality of patient outcomes. Two patients presenting with similar illness sometimes respond quite differently to an identical treatment. Likewise, the justification that a past patient responded positively to a drug does not ensure that the current patient will, which introduces uncertainty about the nature of the association between the treatment and the outcome.Hill emphasized that not all nine criteria would be applicable to all situations. For instance, five appear to be applicable for the demonstration of causality in mental health: Strength, Consistency, Temporality, Plausibility, and Experiment. For any application, common sense needs to prevail when considering criteria to evaluate causality.One could argue that over-reliance on RCTs has fostered a less critical form of thinking in the evaluation of health care treatments. Several years ago Smith and Pell wrote a satirical, insightful commentary on the need to do an RCT of the effectiveness of parachutes for the prevention of major trauma caused by gravity . They coThe methods we use constrain the types of observations we can make. Because of this, it is important to use as many different sources of information as possible. Multi-method research can provide converging evidence on treatment effects, where "multi-method" refers to obtaining diverse sources of information that are minimally related to the existing sources. Unfortunately, it is increasingly difficult to fund or publish studies that are not RCTs. While the majority of social scientists fly the multi-method banner, it is RCTs that primarily hold the attention of health researchers. This dependence on RCTs means that the weight of evidence is precariously balanced upon a single method, a clear example of the instability of a one-legged stool. Because inferences from clinical research propagate to clinical practice, failure to consider multiple sources of information compromises the foundation on which medical decisions are based, and on which the fate of lives may rest.In summary, over-reliance on RCTs a) has been influenced in part by market pressures relevant to pharmaceutical companies, b) was stimulated significantly by the 1962 amendments to the American Food, Drug, and Cosmetic Act, and c) is not scientifically sound. As Parker stated (p. 971) [The authors declare that they have no competing interests.BJK, SS, and KM conceived the manuscript and drafted it. GG and BJK provided the major portion of shaping the final content. All authors edited multiple versions and approved the final manuscript.BJK and GG are research psychologists; SS is a clinical and academic psychiatrist; KM is a graduate student in the Health Sciences Program, with a special interest in clinical research methodology.The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2288/11/65/prepub


The FOSB gene is involved in cell proliferation, differentiation and transformation in several tumor types. We investigated whether coding single-nucleotide polymorphisms (cSNPs) and promoter SNPs of FOSB contribute to the development of papillary thyroid cancer (PTC). We also assessed the associations between FOSB SNPs and the clinicopathological characteristics of PTC. One coding SNP  and one promoter SNP  in the FOSB gene were genotyped using direct sequencing in 94 PTC patients and 213 healthy controls. Genetic data were analyzed using SNPStats, HelixTree and SNPAnalyzer. PTC patients were dichotomized and compared with respect to clinicopathological characteristics of PTC. We detected an association between PTC and cSNP (rs2282695) in FOSB . The G allele was a risk allele in the geno-type and allele analyses of cSNP (rs2282695) in the FOSB gene . A promoter SNP (rs12373539) in FOSB was associated with cervical lymph node metastasis of PTC . The A allele was a protective allele in the genotype and allele analyses of SNP (rs12373539) in the FOSB gene promoter . Variation in a FOSB cSNP (rs2282695) may be associated with risk of PTC. The FOSB promoter SNP (rs12373539) may be associated with lymph node metastasis of PTC. The incidence of thyroid cancer has increased more than 2-fold in the US during the past three decades, mainly due to increases in papillary thyroid cancer (PTC), which is the predominant type of malignant thyroid tumor . Due to The development of thyroid cancer is strongly determined by individual genetic background. A genetic predisposition for PTC has been suggested by case-control studies showing a 3- to 8-fold increase in risk in first-degree relatives, which is one of the highest such risks of any cancer type . DespiteFBJ murine osteosarcoma viral oncogene homolog B, also known as FOSB (in humans) or FosB (in other species), is a protein that is encoded by the FOSB gene in humans –9. The FIn this study, we investigated whether SNPs in the FOSB gene contribute to the development of PTC. We also assessed the relationships between the FOSB gene and the clinicopathological characteristics of PTC.We recruited 94 patients with PTC and 213 control patients at Kyung Hee University Medical Center, Seoul, Republic of Korea. The diagnosis of PTC and the presence of cervical regional lymph node metastasis were confirmed by pathological examination. The specimens that were diagnosed as follicular variants, diffuse sclerosing and tall cell variants were excluded. None of the controls were diagnosed with cancer or thyroid disease at the time of enrollment. The male-to-female ratio of the included patients was 27:67 and the mean age was 53.2±12.0 years. The control group of 213 healthy adults  included 108 males and 105 females. This study was approved by the Institutional Review Board of the Medical Research Institute, Kyung Hee University Medical Center. Written informed consent was directly obtained from all subjects.To determine the nature of the relationship between FOSB SNPs and the clinicopathological characteristics of PTC, patients were divided into subgroups according to size (<1 or ≥1 cm), number  and location of cancers (one or both lobes). In addition, PTC patients were also subgrouped into extrathyroidal invasion (+) and (-) groups based on pathological findings. Finally, PTC patients were further subgrouped into lymph node metastasis (+) and (-) groups to evaluate the contribution of FOSB SNPs to cancer metastasis. The demographic characteristics of PTC patients are summarized in www.ncbi.nlm.nih.gov/ SNP, dbSNP BUILD 135). Among the SNPs of FOSB, SNPs with heterozygosity <0.05 or unknown and monogenotype were excluded. For genetic analysis, we selected rs2282695  and rs12373539 . Genomic DNAs were extracted from blood samples collected in EDTA tubes using the Roche DNA Extraction kit . SNP genotyping was conducted by direct sequencing. Polymerase chain reaction (PCR) was performed using specific primers for the FOSB SNPs that were selected for analysis (We searched the oding SNPs (cSNPs) of the FOSB gene. Information related to the SNPs was obtained from the National Center of Biotechnology Information (NCBI) SNP database  in patients and controls, and adjusted for gender and gender. We used HelixTree  and SNPAnalyzer  to analyze genetic data. Multiple logistic regression models  were employed to obtain odds ratios (ORs), 95% confidence intervals (CIs) and P-values. All data analysis was performed using SPSS 18.0 software . Statistical significance was set at P<0.05.Continuous variables are presented as the mean ± SD and were analyzed by independent t-test and Chi-square test. Hardy-Weinberg equilibrium (HWE) was assessed using SNPStats software  . The G aThe genotypic distributions of the two SNPs examined in this study were in Hardy-Weinberg equilibrium . In our analyses of genotype data collected from 94 PTC patients and 213 controls, the synonymous cSNP rs2282695 of FOSB was associated with development of PTC . However, the promoter SNP rs12373539 of FOSB was not associated with development of PTC .When we assessed the genetic relationships between SNPs and subgroups of PTC patients according to cervical lymph node metastasis, FOSB SNP (rs2282695) was not associated with cervical lymph node metastasis (data not shown). However, FOSB SNP (rs12373539) was associated with cervical lymph node metastasis  [codominAssociations between SNPs and the risk of differentiated thyroid cancer have recently been investigated in studies focused on polymorphisms of genes that affect physiological pathways such as DNA repair, cell-cycle control, kinase-dependent signaling, endogenous or exogenous metabolisms and apoptosis –15. HoweFosB is an acidic protein of 338 amino acids that shares structural similarities with the prototype of the Fos family, c-Fos, namely, a proline-rich basic DNA binding region, a leucine zipper required for dimer formation, and a C-terminal transactivation domain . The humet al . At the sites of the FOSB SNP (rs12373539), the A-containing sequences bind with C/EBPdel, Oct-1, and Oct-5 transcription factors, and the G-containing sequences bind with Sp1, Oct-1, and N-Oct-4 transcription factors. Assuming that transcription factor binding varies with promoter SNPs, this promoter SNP may influence gene and protein expression of FOSB SNP (rs12373539).To determine whether the promoter SNP affects transcription factors, we used the online program AliBaba 2.1 , this limitation does not greatly impact on this study. Small sample size was another limitation. To confirm our results, an additional study with a larger sample size should be conducted. Another limitation is the absence of tumor tissue analysis. Genetic changes of FOSB should be examined in cell lines, and immunohistochemical studies with specimen tissues are required to determine biological effects.In conclusion, our results indicate that a synonymous SNP  of the FOSB gene is associated with the development of PTC, and that a promoter SNP  of the FOSB gene may be associated with the nodal metastasis of PTC. The G allele of the FOSB SNP  may be a risk factor for development of PTC in the Korean population.


AbstractCalliostoma tupinamba isa new species from Southeastern Brazil, ranging from southern Rio de Janeiro to northern São Paulo, and found only on coastal islands, on rocks and sessile invertebrates at 3 to 5 meters of depth. Shell and soft part morphology is described here in detail. Calliostoma tupinamba is mainly characterized by a depressed trochoid shell; eight slightly convex whorls; a sharply suprasutural carina starting on the third whorl and forming a peripheral rounded keel; and a whitish, funnel-shaped and deep umbilicus, measuring about 5%–10% of maximum shell width. Calliostoma tupinamba resembles Calliostoma bullisi Clench & Turner, 1960 in shape, but differs from it in being taller and wider, having a smaller umbilicus and lacking a strong and large innermost spiral cord at its base. Finally, an identification key of Brazilian Calliostoma species is presented. Calliostoma Swanson, 1840 has a worldwide distribution, occurring from the intertidal zone to depths of several hundred meters ; Búzios Island, Aquário coast, MZSP 105574, 07 specimens . Calliostoma bullisi. Brazil, Amapá, Cabo Orange holotype USNM 612702. Calliostoma hassler. Brazil, Bahia; Belmonte, MZSP 37264, 03 specimens ; Alcobaça, MZSP 34885, 11 specimens (Coltro leg.). Espírito Santo; Guarapari, 05 specimens (vi/2006); MZSP 69424, 09 specimens ; MZSP 57485, 09 specimens (Coltro leg. iv/1992). Calliostoma depictum. Brazil, Bahia, Salvador, MZSP 66727,  01 specimen. Calliostoma militare. Brazil, Rio de Janeiro, off Saquarema, MZSP 66961, 02 specimens . Calliostoma brunneopictum. Brazil, Rio de Janeiro, off Santana, MZSP 73915, 14 specimens (col. xi/1999). Calliostoma viscardii. Brazil, Rio de Janeiro, off Saquarema, MZSP 68597, 02 specimens . Calliostoma carcellesi. Brazil, Rio de Janeiro, Arraial do Cabo, MZSP 70459, 01 specimen (col. v/2001). Calliostoma adspersum. Brazil, São Paulo, São Sebastião, MZSP 94332, 02 specimens (J. Vaz leg. 03/iii/1998). Calliostoma rota. Brazil, São Paulo, Ubatuba, MZSP 38524, 02 specimens (BIOTA-FAPESP col. 10/vi/2001). Calliostoma jucundum. Brazil, Rio de Janeiro, Niterói, MZSP 66598, 02 specimens (col. xii/2001). Calliostoma gemmosum. Brazil, Bahia, Salvador, MZSP 34850, 02 specimens.Institutional abbreviations: MNRJ, Museu Nacional da Universidade Federal do Rio de Janeiro, Brazil; MOFURG, Museu Oceanográfico "Prof. E.C. Rios", da Universidade Federal do Rio Grande, Brazil; MZSP, Museu de Zoologia da Universidade de São Paulo, São Paulo, Brazil; NMNH, National Museum of Natural History, Smithsonian Institution, Washington, D.C., USA.Type species:Trochus conulus Linné, 1758: 3579 (SD by Herrmannsen 1846: 154).urn:lsid:zoobank.org:act:3AD5A1CD-5634-4E17-995F-77784551943Fhttp://species-id.net/wiki/Calliostoma_tupinambaCalliostoma ilhabelensisnomen nudum) Prado, 2003.  : Calliostoma jujubinumnon Gmelin 1791). : PageBreak23°54'26"S, 45°27'57"W, 10 m (col. 20.i.2012), MZSP 103766, 2 specimens; Alcatrazes Archipelago, Farol Island, 24°05'44.7"S, 45°42'08.0"W, 6–9 m (coll. 05.x.2011), MZSP 102223, 7 specimens, MNRJ 30603, 1 specimen, MOFURG 51661, 1 specimen; Ilhabela, Saco do Eustáquio, 23°50'11"S, 45°14'21"W, 12 m (col. 09.i.2012), MZSP 105118, 1 specimen; Vitória Island, Saco do Paiá, 23°44.658'S, 45°01.343'W,PageBreak 6–9 m (15.v.2012), MZSP 105660, 4 specimens, MNRJ 30602, 1 specimen, MOFURG 51662, 1 specimen.Holotype MZSP 105740 (37.3 mm × 38.2 mm). Paratypes: MZSP 91745, 4 specimens, from type locality. São Paulo state; São Sebastião, Therezinha wreck, 23°13'S, 44°08'W, 5–7 m (col. 08–09.viii.2009).Brazil, Rio de Janeiro state, Angra dos Reis, Jorge Grego Island, Reference to the Ecological Station (Esec) Tupinambás for expeditions along the southeastern coast of Brazil and is an arbitrary combination of letters.Shell reaching 38 mm in height, with sharply suprasutural carina starting on the third whorl and forming a large peripheral rounded keel. Umbilicus deep, white, funnel-shaped, narrow (5% to 10% of maximum shell width). Base convex.Shell , 16. AttHead-foot , 32. TotPageBreakin median region of furrow. Anterior third of epipodium formed by well-developed neck lobes  of left kidney ~1/3 of rectum’s length; oval, with wide base; gradually narrowing towards anterior, ending at left of nephrostome; inner wall with numerous thin and long papillae projecting inward from all surfaces. Right kidney (rk) divided in two regions: anterior region hollow, a tube ~1/3 of rectum’s length and ~1/2 width of papillary sac in male; in female called ampulla (ap), large, oval, hollow, filled by mucus, ~3/4 larger than papillary sac; kidney expanding ventrally, covering right half of surface of adjacent visceral hump; posterior region spreading around visceral mass immediately beneath mantle, encircling inner surface of columellar muscle.m1, series of small muscles connecting buccal mass with adjacent inner surface of snout and haemocoel; m1d, pair PageBreakof small dorsal retractors muscles, originating in postero-lateral region of mouth sphincter .On rocks and sessile invertebrates, at 3–5 meters.Calliostoma tupinamba closely resembles Calliostoma bullisi Clench & Turner, 1960 from northern Brazil. It differs in being taller and wider: the holotype of Calliostoma bullisi , Calliostoma depictum Dall, 1927 and Calliostoma adspersum   . As proti, 1851)  it is haCalliostoma tupinamba differs from Calliostoma hassler Clench & Aguayo, 1939  allocated some species such as Calliostoma jujubinum, Calliostoma bullisi in the subgenus Elmerlinia. This taxon is characterized by the triangular shape of the jaws, with denticles projecting in tufts on the distal edge, and by the two lateromarginal plates in the radula. The same pattern is observed in Calliostoma tupinamba and Calliostoma hassler and, therefore, it seems to be closely related with species of Elmerlinia.In a revision of the Western Atlantic’s