Health care technological evolution through new drugs, implants and other interventions is a key driver of healthcare spending. Policy makers are currently challenged to strengthen the evidence for and cost-effectiveness of reimbursement decisions, while not reducing the capacity for real innovations. This article examines six cases of reimbursement decision making at the national health insurance authority in Belgium, with outcomes that were contested from an evidence-based perspective in scientific or public media.In depth interviews with key stakeholders based on the adapted framework of Davies allowed us to identify the relative impact of clinical and health economic evidence; experience, expertise & judgment; financial impact & resources; values, ideology & political beliefs; habit & tradition; lobbyists & pressure groups; pragmatics & contingencies; media attention; and adoption from other payers & countries.Evidence was not the sole criterion on which reimbursement decisions were based. Across six equivocal cases numerous other criteria were perceived to influence reimbursement policy. These included other considerations that stakeholders deemed crucial in this area, such as taking into account the cost to the patient, and managing crisis scenarios. However, negative impacts were also reported, in the form of bypassing regular procedures unnecessarily, dominance of an opinion leader, using information selectively, and influential conflicts of interest.‘Evidence’ and ‘negotiation’ are both essential inputs of reimbursement policy. Yet, purposely selected equivocal cases in Belgium provide a rich source to learn from and to improve the interaction between both. We formulated policy recommendations to reconcile the impact of all factors identified. A more systematic approach to reimburse new care may be one of many instruments to resolve the budgetary crisis in health care in other countries as well, by separating what is truly innovative and value for money from additional ‘waste’. Innovation of clinical practice has enabled professional care providers to improve population health in a way that would have been impossible before the 21th century. Yet, ironically, the same drive for innovation and the related technology push from industry is currently often cited as one of the main reasons why health systems across Western countries are spinning out of control There is a growing consensus that the rise in spending should be topped off at the level justified by demographic changes and inflation. This, however, might threaten the future capacity for innovation. New drugs, implants and other care interventions enter the market continuously. Nowadays policy makers are faced with the increasingly difficult challenge to decide which new entrants will be reimbursed and which will not. The general public often presumes that this decision making process is done in a systematic manner, guided by evidence of (cost) effectiveness. Evidence based policy implies that policy makers make well informed decisions about policies, programs and projects by putting the best available evidence from research at the heart of policy development and implementation According to the evidence based policy framework formulated by Philip Davies, the decision to include or exclude an innovation in/from reimbursement will, next to clinical and health economic evidence, theoretically also depend on (1) Experience, expertise and judgment of decision makers, (2) Financial impact – including cost-effectiveness – and resources, (3) Values, ideology and political beliefs, (4) Habit and tradition, (5) Lobbyists and pressure groups, and (6) Pragmatics and contingencies In real life reimbursement policy, the relative weight of each of these factors on the final outcome of decision making, and how these factors interact, are both not clear The study was approved by the University of Leuven ethical review board, sub domain of public health. Written informed consent was obtained from all participants.Stakeholders included key decision makers directly involved in the decision to approve or decline reimbursement at a national level in one or more of the six case-studies. The scope of our study is restricted to the Belgian health system, wherein negotiation occurs between federal government and sickness funds as payers, and physician, and pharmacy representatives as care providers, completed with academics. In Belgium, reimbursement decisions for drugs and implants are guided by separate directives of the EU that specify adoption requirements. Decision making takes place in three national councils of the above representatives, as part of the responsibility of the National Institute of Health and Disability Insurance. One council makes reimbursement decisions about medical drugs, another about medical devices such as implants, and a third about medical fees for services unrelated to a specific drug or device (such as breast cancer screening).An overall set up of negotiation is considered to be typical for part of the Western health systems, as illustrated in many health system profiles, published by the Health Systems & Policies Observatory of the World Health Organization The study made use of a series of six qualitative case-studies. Cases, being national reimbursement decisions on new drugs, implants or other care interventions that are known to be contestable from a purely clinical and health economic evidence-based perspective, were selected in the following way: First, both media publications and scientific literature were screened to devise a list of candidate cases, based on expressed doubt about the evidence-based nature of reimbursement decisions made. Secondly, the research team selected six cases that covered drugs, implants and other medical services in a balanced way. The case studies address aortic endovascular replacement, breast cancer screening, oseltamivir, hadron therapy, Alzheimer medication, and trastuzumab. We emphasize that the selection of these cases was not based on a fully-agreed upon lack of evidence; nor was it the result of the subjective opinion of ourselves as investigators. Literature revealed that in the selected cases inconsistencies of study findings or discrepancies of normal procedures were likely at play. Cases were focused upon to learn from, without judgment or allocating blame.Data were collected through in depth semi structured interviews of eight key stakeholders, who were recruited based on their involvement in cases and their central role in the decision making process. At least two stakeholders were interviewed per case. In Thematic analysis was applied: First, themes were extracted according to the categories of the evidence based policy framework. Secondly, relevant text fragments were allocated to themes. We also sought to cover additional themes, if present, which were not addressed in the framework. Thematic analysis was performed by HT, and independently validated by PVH and DR. Disagreement was resolved through discussion until consensus was reached.In this section we first describe the specific context of each case. After separately reporting the results for six cases on the stakeholder perceived impact of each factor, we cross our findings across cases to describe relevant patterns. We use the following codes to label cited text fragments according to stakeholder role: CP  =  care provider/physician; G  =  government; and HF  =  health fund/insurer.In the year 2000, while evidence for this procedure was still lacking, payers in Belgium decided to reimburse aortic endovascular replacement, based on the presumption that this would decrease the need for open surgery with its associated potential complications. A reimbursement procedure meant to temporarily support experimental testing of new techniques in a limited number of hospitals, was extended to the majority of all Belgian hospitals. However, in 2005 mounting evidence – including the EVAR I and II RCT in the UK – showed that the gain of less complications and mortality may be present during 1 to 2 years only, after which this benefit disappears All three respondents recognized that at the time of ‘experimental’ reimbursement insufficient evidence was available. Respondent 3 (HF) sees this as a typical example of a case in care for which evidence is expected to develop later on, as an argument for this kind of temporary reimbursement. In those days (year 2000) evidence was not sought systematically . The supplier firm hired vascular surgeons and paid them to demonstrate the technique to colleagues . This kind of ‘tutoring’ spread wildly across a too high number of centers, with as a result some centers placing only two or three stents a year . According to resp. 6 (CP) some expert opinion leaders in vascular surgery claimed that there was evidence, a statement that as such was taken into account in the reimbursement decision. All three respondents agreed that the growing use of this procedure in practice, despite a lack of evidence, made it very difficult not to reimburse. ‘Regulation was caught up by the facts’ as resp. 6 (CP) noted. A lot of stents were already being placed.Cost-effectiveness was not considered, only the additional cost of the new procedure as such: higher prosthesis costs compensated by a lower length of stay . Resp. 2 (HF) commented that the high cost of procedure, making it almost financially unaffordable for patients to bear, added to the pressure to reimburse. Resp. 6 (CP) agreed with this. Resp. 3 (HF) added the following observation:“If a patient entered with an aneurysm, which can become a life threatening condition, one did not check first whether he/she could afford to pay for the procedure. A prosthesis was placed. This was an important argument to grant reimbursement.”In this case the regular decision making process was not followed: A shortcut was taken by allowing so called temporary experimental reimbursement to spread via a convention with hospitals. Respondent 6 (GP) confirms that such a deviation is not exceptional, even today, whereas respondent 3 (HF) disagrees. In line with the external pressure, additional speed lies often at the foundation of the decision to quicken the process. Up to eight months could be gained in this way .Before the decision making process even started, a popular newsletter had published an article titled ‘I am sitting here, waiting, with a bomb in my body’ . It mentioned that the government refused to reimburse the procedure. The article ended up the same day on the desk of the minister of social affairs. Despite many question marks, this hastened the process up to three weeks . The minister was also invited by the media to explain his position .Resp. 3 (CP) confirms that the implant industry pressured vascular surgeons to use the new procedure. He/she adds that industry and the professional interest group had found each other in this endeavor.“This was the way of the early days, when a technology reached the market because professional interest groups were contacted by the firms. They said: We have a good new device and we will take you out for dinner, or to an international symposium, if you want to try it out. In this way the device was introduced.”All three respondents agreed that external pressure was the single most important determinant of the decision to grant reimbursement. Pressure came from care providers, suppliers and patients, the latter mainly due to the cost size for them to bear without reimbursement.Mammography is used in both breast cancer screening and in diagnostic case-finding. A screening mammography is intended to detect a tumor in the pre-clinical stage. The procedure needs to adhere to strict technical quality criteria and a double reading by two trained radiologists. Diagnostic case-finding on the other hand is intended to be applied if a patient has complaints or if a care provider has detected an anomaly on clinical examination. The quality of this last procedure is less guaranteed than for a screening mammography. If a diagnostic mammography is used as a first screening tool, the risk of detecting false positives will increase Evidence played a crucial part in the decision to reimburse screening mammography. Screening was promoted based on the evidence . Yet, respondent 4 (CP) adds that the evidence is currently the topic of some heated debates, because Belgium does not reach a sufficient screening coverage of the population to reach optimal results. “If, after ten years, such conditions are not met, you lose a large part of the evidence based potential.” Both respondents (CP/G) agreed that the initiative in this case mainly came from the government and policy makers themselves. A broad input of expertise has been taken into account  .The decision to organize screening mammography at a population level and not only after referral by a physician, implied a change from traditions and developing new habits, although this met large resistance of some health care professionals (resp. 1 CP/G). Resp. 1 also mentioned the habits of the patient population as a restraining factor to organize screening more systematically and efficiently: ‘As a patient you go to a physician, you get what you want. You want to have a mammography right away, because you are worried’. Patients do not always follow the rules and regulations. Making patients more responsible in this matter remains a challenge. Respondent 4 (CP) confirms that neither the care provider nor the patient are motivated “to follow the correct way”.The fact that neighboring countries already applied screening mammography boosted public demand, which in turn incited policy makers to take initiative . Adoption by other countries was interpreted by some decision makers as a proof of evidence . The way in which screening was subsequently organized, did however not correspond with the approach used in other countries, such as the Netherlands and Scandinavia, where the screening process was more steered by the state .Jurisdictions of regional versus federal government complicated and delayed matters. Prevention is a jurisdiction of the regions in Belgium, whereas the financing partly had to come from the federal level. Regional responsibility is also perceived to be related to currently observed geographical variability in the use of screening mammography versus diagnostic screening .According to resp. 1 (CP/G), pressure from the medical corps and from radiologists in particular resulted into a large degree of therapeutic freedom in the choice of screening modalities.Resp. 1 (CP/G) reported the influence of cultural differences between the north and the south: the north acting in an Anglo-Saxon style by supporting guidelines, the south focusing on therapeutic freedom as is more the tradition in the south of Europe. In line with this, resp. 4 mentioned the physician interpretation of the decision to involve non-physicians in screening as an ideological choice. He said this as follows: “Physicians thought this choice went towards a form of state driven medicine, a medicine of the masses, without personal contact with the physician”. Both respondents make a distinction between political preferences: socialist parties were proponent of a more directed form of medicine, whereas liberal parties were proponent of a more liberal (and freedom affirmative) medicine.Oseltamivir is an antiviral drug promoted for both therapeutic and profylactic use in influenza. The drug was massively distributed and prescribed in Belgium during the Mexican flu pandemic in 2009. The evidence for the effectiveness of Oseltamivir is however sparse. The findings of the main supportive study for its use, conducted by its pharmaceutical supplier, have been heavily questioned in top ranking medical journals. Whereas proponents argued that Oseltamivir prevented infection in 80% of contacts Resp. 1 (CP/G), resp. 2 (HF) and resp. 5 (CP) all agree that most decision makers were aware of, and regretted, that clear cut evidence was lacking. It was however difficult to question the purchasing decision, because a primary opinion leader, as a scientist, confirmed the need for providing Oseltamivir freely to high risk patients with prescription and building up sufficient stock, although this point of view was not shared by the scientific community as a whole. “If evidence is lacking, as a politician you listen to the most renowned expert” .Resp. 1 (CP/G) wondered whether putting both the scientific and policy responsibility in the hands of one person was not counterproductive. He/she had preferred a clearer cut division of scientific versus policy stakeholder ship, respecting checks and balances. “In this case politicians hid behind the broad back of the scientific expert” (resp. 1).In the heat of emotions, the financial impact was not taken into account . According to resp. 2 (HF), in terms of cost-effectiveness this as such represented a disaster. Belgium spent more resources on the purchase than other countries .Due to the urgent situation, the regular scientific evaluation by an independent evidence-based drug review was not performed, and instead replaced by the opinion of a few experts .The media increased the pressure of public opinion, by instilling fear and reinforcing the – real or artificial – urgency of decision making .Collaboration and consensus building between countries did not work appropriately. Each country implemented its own policy . In the end, many countries stocked up Oseltamivir. Yet, Belgium did this at the double rate of most other countries, to be completely independent from others .Public opinion, fearing a flu pandemic, put a large pressure on politicians to do something, even if the true effect was minimal . Resp. 1 (CP/G) mentioned the negative impact of the pharmaceutical industry, that tried to sell products as much as possible, even when little arguments were available.Hadron therapy uses protons and neutrons to radiate tumors. Theoretically, this would lead to fewer side effects than commonly used electron and foton-based radiotherapy, especially for children. At present, these assumptions are however not yet been confirmed by clinical trials. Two systematic reviews concluded that no definitive conclusions can be drawn for head and neck cancer, gastrointestinal tumors, non small-cell lung cancer, sarcomas, cancer of the uterine cervix, prostate cancer, most CNS tumors, and bladder cancer The respondents  indicate that this decision is being prepared with ups and downs in the degree to which the evidence is taken into account. At first, the evidence was put to the centre of the decision. However, in a second term of legislation, with a new minister on board, the evidence has moved to the background and the decision is more ‘politicized’. Previous studies are ignored, and under the pretence of doing a new study, reimbursement and implementation are being prepared.Despite knowledge of lacking cost-effectiveness, the ministry pursues its plans . Positive financial impact of hadron therapy as a regional investment strategy, for care providers and the local population to benefit from, and for politicians, to defend local voters, has surpassed patient value as one of the most determining factors .Evaluation procedures were followed at first. However, when politicians disagreed with the outcome, regular procedures were being shortcut by an additional study with external budgets controlled by the ministry itself, and not by the normal stakeholder groups at government level .Resp. 3 (HF) also mentioned the habit in health care of wanting to implement everything everywhere, because of the level of prestige associated. Only the investment cost has prevented the development of likely six to seven hadron centra across a small country.Larger countries have a sufficient number of patients eligible for this type of care. This is, however, not the case in Belgium (both resp.).Respondents mentioned that radiotherapy centers of teaching hospitals, which represent both the scientific expertise, and the potential gainer of a decision to reimburse and implement, have no unbiased role in this case . This is one of the reasons that this case is repeatedly put onto the table, despite lacking evidence. In addition, local industry acted as lobbyist to attract additional investments .Politics dominated in this case. One of the respondents referred to “an angry crowd of politicians in power who disagree with scientific recommendations” .Dementia drugs, such as inhibitors of acetylcholinesterase (ChEIs) and memantine, are used to temporarily alleviate the symptoms of the disease. Both types were reimbursed according to severity of cognitive dysfunction in Belgium since 2002 and 2004, respectively. The clinical and health economic evidence showed mixed results at that time. Due to small effect sizes, the clinical relevance of positive findings was questionable from the onset Respondents  reported an evolution from little to a larger degree of evidence-based policy in this case. About a decade ago, it was possible to reimburse something for Alzheimer treatment, even with unreliable evidence. The recent, more stringent reimbursement policy takes evidence much more into account.Whereas external expertise, in terms of providing clinical and health economic evidence, was almost irrelevant at the onset of reimbursement, this factor dominated in recent reform (both resp.).Similarly, a lack of cost-effectiveness, and very high prices demanded by the pharmaceutical industry, were no objection to commence reimbursement . Later on, reimbursement adoptions were not based on a complete cost effectiveness analysis, but moved in that direction. In addition, in line with the evidence, an increase in eligibility based on cognitive functioning, was compensated by a demand to suppliers to decrease prices .This factor was very relevant a decade ago, with impact on decision making . At present, complaints from patient organizations were taken up by the media. This did not influence decision making .Initially, this factor was very relevant since Belgium was one of the last countries in Europe to approve reimbursement of anti-Alzheimer drugs . Recent reform was based on a more systematic review of the evidence, in combination with an international policy analysis .Public demand was high at the time of introduction of anti-Alzheimer medication. This played a large part in the reimbursement decision . After recent reform, some stakeholder groups complained about the more stringent indication setting, backed up by pharmaceutical suppliers. They also involved patient organizations in this matter. Yet, after analysis no argument was found to modify reforms .Women in an early stage of HER2 positive breast cancer (without metastasis), who receive chemotherapy after surgery, live longer if they are treated with Trastuzumab. Two to three year survival increased with 75 to 89%, Reimbursement was arranged for, even before all evidence was critically reviewed . Resp. 7 formulated this as follows: “If evidence is defined from a purely scientific perspective, then it was not yet present. If evidence is defined as women knocking at the door who stated that their physician said this drug would help them, which it also did, subjectively or not, then this other type of evidence was available”.First, the supplier introduced the drug for a narrow group of patients with metastasized breast cancer, with high prices. After the first approval of reimbursement, the supplier broadened the group of eligible patients to early breast cancer, but did not reduce prices, knowing that public pressure would force decision makers to accept . This strategy is typical for pharmaceutical industries, he/she adds. Making this new expensive drug affordable for all eligible patients was also one of the main reasons why this reimbursement was adopted rather quickly .To speed up the process, normal procedures were not followed. According to resp. 7 (G) this was understandable to cope with public pressure. However, pharmaceutical industries build up such public pressure to commercialize their products as quickly as possible.The media increased public pressure, according to both respondents.Examples of reimbursement in other countries heightened public pressure . He adds that this international policy analysis was done in a rather selective way, by focusing on countries with reimbursement, and ignoring countries without.The fact that prescriptions were increasing even before reimbursement was arranged for, reinforced the decision .According to resp. 2 (HF) and 7 (G), trastuzumab was considered to be a ‘wonder drug’. The pressure of society, prescribers and suppliers was therefore significant. Decision makers were swamped with expert opinions, some independently, some sponsored by suppliers.In terms of financial impact & resources (3), with exception of the first phase in the hadron therapy case and the second phase in the Alzheimer drug case, cost to society was not perceived to have a high impact in any case. Cost to the patient did have a high impact in two cases, both in combination with a low impact of evidence. Habit & tradition (4) was interpreted by respondents in two ways: First, upholding reimbursement decision procedures or not. In low impact of evidence cases, procedures were not followed or shortcuts were taken. Secondly, as inducing a change in healthcare traditions as a consequence of a reimbursement decision. The breast cancer screening case showed that healthcare system habits can be changed, according to respondents. Yet, existing habits did temper the extent to which change could be fully leveraged.Media attention (5) was perceived to have a high impact in four cases, each time combined with a low impact of evidence. The impact of adoption by other payers or countries (6) was in all cases considered to be medium to high. This was used as a source of information to examine the evidence basis and, in the Alzheimer medication case, to decide to implement. In some cases with low impact of evidence, comparisons were made selectively or independently of contextual differences, to pursue the objective that one or more stakeholder groups aimed at.Pragmatics & contingencies (7) were reported to have a high impact in three forms: decision making in a public crisis situation, deciding after the care intervention was already widely spread across the market, and having to take into account jurisdictions in decision making of multiple payers, states or regions. The impact of lobbyists & pressure groups (8) was high in all cases, except in the second phase of the Alzheimer medication case. In two cases clear conflicts of interest were reported to impact upon the decision. More specifically, care providers, as co-decision makers, were financial benefiters of a positive reimbursement decision, while they served as scientific experts to objectively examine and report the evidence in a balanced way. Finally, values, ideology & political beliefs (9) had a high perceived impact in two cases, in which both political and cultural differences directed decision making.To the best of our knowledge, this is one of the first studies to analyze in depth which factors were at the basis of reimbursement decisions, for which the evidence was lacking or debatable. Our findings confirm, in line with other publications, that evidence is not the sole criterion on which reimbursement decisions are based We agree with Philip Davies that it is impossible to judge in hindsight whether a good or bad decision has been taken However, the results provide policy makers with conclusions that could guide future reform in reimbursement decision making. Health care across developed countries is in dire need of system improvements that safeguard what works well for the patient– the legitimate impact of many factors we described – and, simultaneously, allow us to cope with ever rising budgetary challenges. Here we formulate nine recommendations for policy makers in Belgium and in other countries that make use of payer-provider reimbursement councils by linking them to the cases in our study.Separate a standard decision procedure from a ‘fast track’ procedure to allow rapid reimbursement of new care interventions with an exceptional value for public health (such as trastuzumab). Use fast track by exception only, based on a fixed set of formalized criteria and with full public transparency. Give no stakeholder group the freedom to decide singularly to deviate from one of both procedures (e.g. hadron therapy).As a first ‘evidence’ stage of both procedures, summarize all available evidence by a full or rapid health technology assessment, providing policy makers with a balanced overview of inconsistencies and remaining gaps in the evidence, with all pro’s and con’s of a positive versus negative decision . Speed up, but do not discard, this stage as part of the fast track procedure . Document cost to the health care system  and cost to the patient  objectively, with cost-effectiveness analysis.During the ‘evidence’ stage, invite an external panel of key experts in the care intervention subject under review to reach consensus. Select members of the panel independently, for example based on the number and quality of contributions to knowledge as authors of peer reviewed papers on the topic, and make their conflicts of interest public. A representation of the scientific community by one or two persons is insufficient (oseltamivir). Because experts often are local care providers , compose the panel with an equally large proportion of experts who will not benefit from any decision outcome. As our cases show, often a professional group as a whole has a financial conflict of interest. Involving international experts, with selection based on publications, could offer a partial solution to address this problem.Evidence should not only address the yes or no decision to reimburse or not. It should also address how to implement a positive reimbursement decision further in practice, based on the methods that have shown most optimal results.Make a country or payer comparison part of the ‘evidence’ stage (including ‘implementation’ evidence), with equal attention to countries or payers who decided to reimburse, as to the ones that decided otherwise (trastuzumab).In times of a public health crisis, time may be lacking for a comprehensive ‘evidence’ stage. Still, consult a sufficient number of local and international scientific experts to inform policy makers in a balanced way (oseltamivir).Although there is a call to involve stakeholder groups even more in policy making, we think that the factors ‘experience, expertise & judgment’, ‘values, ideology, & political beliefs’ and ‘lobbyists & pressure groups’ should become relevant in the ‘negotiation’ stage only, after the ‘evidence’ stage has been terminated. Based on all previously summarized information, stakeholders - each from their own perspective - have the freedom to discuss, argument and reach agreement.Media attention is not controllable. However, a more systematic application of the procedures above is likely to prevent policy makers to move prematurely from the evidence towards the negotiation stage when feeling pressured by the media . Minimize pressure from suppliers (aorta endoprosthesis) through normative regulation and make the financial and non financial compensation to care providers and patient organizations public.Finally, periodically review reimbursement decisions, taking into account new evidence that becomes available (Alzheimer medication). However, base the decision to reevaluate on formalized criteria. This will prevent that a care intervention for which reimbursement has been turned down, is reentered repeatedly into the process to let it pass (hadron therapy).The findings of our study should be interpreted with caution. Being qualitative in nature, they cannot be generalized towards all reimbursement decisions. Future research should examine whether the evidence based policy framework can find similar interactions in other cases. Equal attention should be given to cases in which reimbursement was declined. We interviewed a number of key decision makers only, and not all stakeholders involved. We did not attempt to reach further saturation, because the analysis across cases already revealed a sufficient number of recurring patterns. Since we used a conceptualized framework to detect themes, this might have prevented us from identifying other influential factors. However, all respondents confirmed that our framework provided a sufficient degree of coverage of how each case evolved.Our findings showed that respondents sometimes disagreed with each other or made statements that were contradictory to what we know to be evidence from the literature . Such inconsistencies are likely due to a mixture of inconsistencies in the evidence itself, selective information use from a personal stakeholder perspective, the possibility that one was not aware of certain facts at the time of decision making and/or recall bias several years after a decision was made.Finally, we studied reimbursement policy in Belgium only. As reported above, our findings correspond well with general theories and findings in other health systems. There is no reason to assume that components of the framework as a whole will impact reimbursement decision making differently in other developed countries. Many health systems make use of a similar centralized procedure as the one central to the study. Other countries, such as the United States of America, decentralize the decision making to individual insurance agencies. The same factors are likely to be at play, although the degree of negotiation power will be different in such a context. Some findings, such as the impact of cost to patient, the impact of cultural differences and the impact of decision making jurisdictions within a health system are also likely context dependent.Reimbursement decisions for new drugs, implants and other health care interventions in Belgium are the outcome of a process of interplay between several factors. Clinical and health economic evidence played a role in many of the decisions, but was often not the predominant factor in the decision made. Based on the analysis of six cases, we formulated nine recommendations, aimed at policy makers in Belgium and other countries making use of payer-provider reimbursement councils, that are expected to reconcile the ‘evidence’ and the ‘negotiation’ stage of health policy decision making in a more systematic and balanced way, without hurting the fundamentals of democratic debate. This may be one of many instruments to cope with the increasing budgetary pressure in health care, by separating real, worthwhile innovations from additional ‘waste’. Future research should focus on the relative importance of influential factors on decision making. A mixed methods approach, adding quantitative to qualitative information, is recommended.


The aim of this article is to provide a preliminary estimate of how much CAM is evidence-based. For this purpose, I calculated the percentage of 685 treatment/condition pairings evaluated in the “Desktop Guide to Complementary and Alternative Medicine” which are supported by sound data. The resulting figure was 7.4%. For a range of reasons, it might be a gross over-estimate. Further investigations into this subject are required to arrive at more representative figures. A lively discussion exists about the question as to how much of conventional medicine might be based on sound evidence . One figThe discussion about the evidence-base of CAM is far less lively. Here I present a first attempt to generate some data and hopefully a constructive discussion on this potentially important subject.The Desktop Guide to Complementary and Alternative Medicine [n = 46). For each condition, we compiled a “summary of clinical evidence” table in which the treatments are categorized according to the “weight” and “direction” of the evidence. The “weight” is conceptualized as a composite measure of the quantity, quality and level of the research evidence, which refers to the confidence that can be placed on that evidence [As a basis for my assessment, I used our own book Medicine . In thisevidence . The levFor the purpose of this analysis, I have simply counted the number of treatments which obtained the maximum “weight” and also were rated as “clearly positive” in our “summary of clinical evidence” tables. This provided the number of treatments that are supported by good evidence (if one therapy was effective for two indications it was counted twice). Subsequently, this figure was put in relation to the total number of treatment/condition pairings from all the “summary of clinical evidence” tables in our book .Fifty-one treatments were characterized as having maximum “weight” of evidence as well as being “clearly positive.” The total number of treatment/condition pairings was 685. Consequently, 7.4% of them were based on sound evidence. The estimate that 7.4% of CAM is based on sound evidence may well be over-optimistic. We selected the conditions for inclusion in our book  on the bA glance at Serenoa repens) [Ginkgo biloba [Finally, several cases of “sound” evidence included in  repens) , glucosa repens) –12, Ginko biloba –16 and ao biloba , 18.Another concern is that the present analysis merely relates to the question of how many therapies might be supported by sound research evidence. It does not address the question of how solidly CAM practice is evidence-based. This would require an assessment of which treatments are used and how often. Such a research project would be complex but would certainly be a valuable contribution to the literature.Although my estimate of how much of CAM is evidence-based draws on a critical evaluation of the available evidence, it still presents a rather optimistic view. Further investigations into this subject are required to arrive at more representative figures.


In health care organizations, there is a mutual interest from politicians, managers, practitioners and patients that the best available care is provided. Efforts are made to translate new knowledge and evidence-based practices into routine care, but there are a number of obstacles to this translation process. Factors related to the new practice as well as factors related to the implementation process are important, but there is still a knowledge gap regarding how to achieve effective implementation. The aim of the present study was to assess opinions about practice change among staff in primary health care (PHC), focusing on factors related to a new practice and factors related to the implementation process.A questionnaire was sent to 470 staff members at 22 PHC centres where a new tool for lifestyle intervention had recently been implemented. Thirteen items regarding the characteristics of the new practice and nine items regarding the implementation process were to be judged from not at all important to very important. A factor analysis was performed, and statistical analysis was done using the Kruskal-Wallis nonparametric test.Objective characteristics, followed by Evidence base, Subjectively judged characteristics and Organizational level characteristics. Two factors were identified regarding the implementation process: Bottom-up strategies were judged most important and Top-down strategies less important. The most important single items regarding characteristics were “easy to use” and “respects patient privacy”, and the most important implementation process item was “information about the new practice”. Nurses differed most from the other professionals, and judged the factors Evidence base and Organizational level characteristics more important than the others. Staff with more than 10 years experience in their profession judged the Evidence base factor more important than those who were less experienced.Four factors regarding the characteristics of the new practice were identified. Most important was To incorporate new practices in PHC, objective characteristics of the new practice and the evidence base should be considered. Use of bottom-up strategies for the implementation process is important. Different opinions according to profession, gender and years in practice should be taken into account when planning the implementation. In health care organizations there is a mutual interest from politicians, managers, practitioners and patients that the best available care is provided. Efforts are made to translate new knowledge and evidence-based practices into routine care, but there are a number of obstacles to this translation process. Such obstacles could be contextual factors, factors related to the intended adopters or characteristics of the new practice . To overA number of theories and frameworks for implementation are described in the literature ,3-5. DamStrategies for implementation of evidence-based practices have been described by Nutley et al. . They stIn Sweden, primary health care (PHC) has an obligation to provide health promotion/preventive services, which requires a change from a disease-centred approach to a health promotion approach. A number of barriers to this change have been identified, such as lack of time, skills and resources ,14. To fThe aim of the present study was to assess opinions about practice change among staff members in PHC who recently have experienced the introduction of the CLT, focusing on factors related to a new practice and factors related to the implementation process.The study was conducted among PHC staff in Östergötland County, Sweden, using a cross-sectional survey design. Östergötland County has about 420,000 inhabitants and has a mix of rural and urban communities. The county has been found to be representative of the Swedish population in terms of gender distribution, employment rates and economy . Of the The CLT was introduced at the participating units in 2008-2010. After 2 years, a follow-up evaluation was performed using a survey questionnaire sent by e-mail to staff at the participating units. Staff groups included were general practitioners (GPs), nurses, assistant nurses (ANs) and allied professionals (APs). The questionnaire assessed opinions about using the CLT. It also contained two questions regarding general opinions about what is perceived to be more or less important when a new practice or tool is introduced in PHC. These questions focused on the characteristics of the innovation and factors related to the implementation process, and were based on former research described in the background to this article. The questions were put as “To make you incorporate a new practice, how important are the following characteristics of the new method?” (13 items), and “When a new practice is introduced at your work place, how important are the following factors?” (9 items). The questions were answered using a Likert type scale with the alternatives very important, important, less important, not important at all, for each of the suggested items. All the items from the two questions were included in the analysis.U test. Statistical significance was set at p ≤ 0.05. However, when the four professional groups were compared, two to three analyses were conducted for each item, which might result in mass significance; therefore p values between 0.017 and 0.05 should be interpreted as tendencies rather than significant differences. Possible correlations between profession, years in practice and gender where tested with Spearman’s rho. Statistical analyses were performed using the computer-based analysis program SPSS version 21.0.A factor analysis was performed to identify clusters of items, which could then be compared according to staff group, age and years of practice. The Kaiser-Meyer-Olkin measure of sampling adequacy was used to ensure that the sample size was adequate for these analyses. Direct oblimin rotation was used because the factors were expected to correlate. Factor loadings <0.4 were not reported. In addition, data from the questionnaires were analysed item by item, and answers from the different staff groups were compared using the Kruskall-Wallis or the Mann-Whitney The study was carried out in compliance with the Helsinki Declaration and was approved by the Ethics Board in Linköping, Sweden (Dnr Ö 16-08). The participants were staff members therefore written consent was not required according to Swedish regulations (SFS 2003:460).The questionnaire was sent to 470 staff members. Of those, 239 responded, yielding a 51% response rate. Internal drop-out for the questions used in the present paper resulted in response rates for the specific questions from 40% to 42%. An analysis of the drop-outs showed no differences between responders and the entire sample in terms of profession or gender, except that the proportion of GPs was lower among the responders (12.6%) than in the entire sample (20.5%).r = 0.393, which is regarded as a moderate correlation), but associations between profession and years in practice, or gender and years in practice were very low.The questions about importance of characteristics were analysed according to profession, years in practice and gender. The questions were answered by 27 GPs, 111 nurses, 19 APs and 37 ANs. Most were women (88%), and 83% had more than 10 years of practice in their current profession. Gender was associated with profession 1. Evidence base 2. Subjectively judged characteristics 3. Organizational level characteristics 4. Evidence base and Organizational level characteristics more important than the other professionals . No differences were found between GPs, APs and ANs. Women found Organizationel level characteristics more important than men did (p = 0.040). Staff with more than 10 years experience in their profession found the Evidence base factor more important than those who were less experienced (p = 0.013). Age was not associated with differences regarding the four factors.When the four factors were analysed according to profession, age, gender and years in practice, it was found that nurses found the factors The 13 items representing characteristics of the new practice were also analysed one by one. Figure When groups were compared according to profession, gender and years in practice, significant differences or tendencies were found regarding five of the items, as shown in Table The question about importance of issues regarding the implementation process was answered by 189 individuals . Of these, 87% were women and 82% had more than 10 years of practice in their current profession. The nine issues were– Influence over the decision to introduce the new practice– Information about the new practice– The manager being positive about the new practice– Continuous support and encouragement from the manager (Manager support)– Continuous support and encouragement from peers (Peer support)– Continuous support and encouragement from a change agent outside the organization (Change agent support)– Evaluation of the new practice after a certain amount of time– Patients requesting the new practice– Financial support to the PHC centreThe Kaiser-Meyer-Olkin measure was 0.756, which according to Hutcheson and Sofroniou  can be cBottom-up strategies 1. Top-down strategies 2. Bottom-up strategies was less important to GPs than to the other professions (p < 0.001). Furthermore, women scored higher than men did (p < 0.001). No differences were found according to age or years in practice. Regarding Top-down strategies, there were no differences according to profession, age, gender or years in practice.When the two factors were analysed according to profession, age, gender and years in practice, it was found that the factor The nine issues were also analysed one by one. Figure When groups according to profession, gender or years in practice were compared, significant differences or tendencies regarding eight of the nine items were found, as shown in Table Objective characteristics of the new practice were perceived as most important, and that Bottom-up strategies were given higher importance than Top-down strategies. Substantial differences between the professional groups were found regarding these factors, and some differences were found according to years in practice.Staff opinions about factors important for changing practice, in relation to the characteristics of the new practice, and to the implementation process were assessed. The main finding was that The two characteristics of a new practice that were considered most important regardless of staff category were that it should be easy to use, and that it should respect patient privacy. When Rogers  identifiEvidence base was more important to staff with long experience in their current position than to their less experienced peers. The result, however, is surprising, taking into account the focus on evidence-based practice currently influencing education in medicine and nursing.Differences regarding the characteristics of the new practice that were related to years in practice were whether the new practice was advocated in guidelines, and if it was supported by the immediate manager. Those who had long experience (10 years or more) were more likely to appreciate these features. When perceived competence to develop evidence-based practice was assessed in a Spanish study, it was found that nurses with shorter experience obtained the best scores , which cOrganizational level characteristics more important than the other professionals. This might reflect that nurses put themselves in a position of responsibility for the care delivered by the health care centre, they show loyalty to the manager and want things to work for everybody. Nurses are known to put a high value in the practice of nursing as described by Sellman [Evidence base more important. Current nursing research is focusing extensively on evidence-based medicine or evidence-based practice, which could explain this difference.Another difference revealed from the factor analysis was that nurses find the factor  Sellman , and a w Sellman . Nurses A notable finding is that recommendation from peers was given low importance by the staff members in the present study. Influence from peers has previously been shown to be an important factor for organizational change in health care , but undWith regard to issues most important for the implementation process, the two items “good information about the new practice” and “the manager being positive” were judged most important. Information is known to be important when changes are implemented in organizations , and theThe issue given less importance when studying the entire group was “patients requiring the new practice”. However, GPs found it more important than the other groups, and those with long experience found it more important than the less experienced staff members. It could be that experienced GPs listen to a higher degree to patients’ preferences. This practice is in accordance with the original ideas of evidence-based medicine, stating that individual clinical expertise, the best available external clinical evidence from systematic research and patient preferences all are important to provide effective and efficient treatment .Bottom-up strategies was less important to GPs than to other professions. An explanation to this could be found in the hierarchical structure in health care, whereby GPs have a dominant position [Results from the factor analysis showed that the factor position . The domIn the present study, staff members in PHC were asked about their opinions regarding change in practice, and the answers could be influenced by social desirability. This should be taken into account when interpreting the results. However, as the study assesses opinions and not performance, the influence of social desirability is probably limited. The questionnaire used was developed exclusively for the present study, and face validity was obtained by discussing the questions among the members of the research team. The questions have not been tested for validity, which is a limitation. Another limitation of the study is that the response rates were not very high and GPs were underrepresented among the responders, which affect the generalizability of the results. A strength is that the study was performed shortly after the implementation of a new practice, which means that the responders all had recent experience of an implementation process.To incorporate a new practice in PHC, the objective characteristics of the new practice and the evidence base should be given consideration. It is important to use bottom-up strategies for the implementation process. Different opinions exist according to profession, but also according to gender and years in practice, which should be taken into account when an implementation activity is planned.The authors declare that they have no competing interests.SC contributed to the design of the study, the acquisition and interpretation of the data, and was involved in drafting the manuscript. KF contributed to analysis of the data and was involved in drafting the manuscript. Both authors have read and approved the final manuscript.SC has a PhD in Medical Science, holds a Master’s degree in Public Health and is a registered physiotherapist; KF has a PhD in Social Medicine and Public Health and holds a Master’s degree in Statistics.The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2296/15/2/prepub


Clinical guidelines can improve health-care delivery, but there are a number of challenges in adopting and implementing the current practice guidelines for depression. The aim of this study was to determine clinical experiences and perceived barriers to the implementation of these guidelines in psychiatric care.A web-based survey was conducted with 386 psychiatric specialists to inquire about experiences and attitudes related to the depression guidelines and barriers influencing the use of the guidelines. Quantitative data were analyzed, and qualitative data were transcribed and coded manually.Almost three quarters of the psychiatrists (74.6%) were aware of the clinical guidelines for depression, and over half of participants (55.7%) had had clinical experiences with the guidelines in practice. The main reported advantages of the guidelines were that they helped in clinical decision making and provided informative resources for the patients and their caregivers. Despite this, some psychiatrists were making treatment decisions that were not in accordance with the depression guidelines. Lack of knowledge was the main obstacle to the implementation of guidelines assessed by the psychiatrists. Other complaints addressed difficulties in accessing the guidelines, lack of support for mental health services, and general attitudes toward guideline necessity. Overall, the responses suggested that adding a summary booklet, providing teaching sessions, and improving guidance delivery systems could be effective tools for increasing depression guideline usage.Individual barriers, such as lack of awareness and lack of familiarity, and external barriers, such as the supplying system, can affect whether physicians’ implement the guidelines for the treatment of depression in Korea. These findings suggest that further medical education to disseminate guidelines contents could improve public health for depression. Depression is an enormous health-care problem that is responsible for 11% of disability worldwide.For practicing psychiatrists, the guidelines provide many suggestions for different forms of treatment of the many kinds of depressive patients. ClinicaBy understanding which factors are associated with implementation of evidence-based guidelines for the management of depressive disorders, strategies can be identified to improve physician guideline adherence and to adapt and focus guidelines and thus improve the quality of depression care. There are numerous possible barriers to implementation, from the distribution of guidelines to the use of guidelines in practice by physicians. The literature on determinants of implementation of evidence-based guidelines on depressive disorders has primarily focused on socio-demographic patient characteristics and disease-related factors, such as patient age and severity of the disorder,12. DespIn Korea, since the first Korean Medication Algorithm for Major Depressive Disorder was developed, updatedWe hypothesized that the provision of guideline-recommended care is influenced by characteristics of both the physician and the practice. Understanding experiences and barriers is important for the development of effective implementation strategies. The present study, which was initiated as a quality improvement project, aims to assess the feasibility of the depression guidelines for clinical settings in Korea. In this article, we investigate barriers to the implementation of the guidelines for managing depressive disorders. We used a questionnaire to ask staff about their knowledge and use of these guidelines. Our findings can help developers of guidelines, practice directors, and health-care services researchers to design effective interventions to change physician practice.This survey was performed from August 2011 to January 2012. To recruit expert participants for this study’s cross-sectional, web-based survey measuring met and unmet care needs, we used a modified Dillman method which caQuestionnaire constructs for measuring the participants’ knowledge, experiences, attitudes, and barriers to implementation of the depression guidelines were developed from previous literature. The questionnaire contained three parts. The first part included demographic data for the psychiatrist: age, gender, clinical experience, employment status, and percentage of patients with depressive disorders. In addition, this part of the questionnaire contained items that assessed factors affecting diagnosing and treating depression. In part 2, we asked about the participants’ experiences with and attitudes toward the depression guidelines. This part contained items measuring perceived barriers to health care provision for patients with depressive disorders in terms of implementation of the evidence-based depression guidelines. Questionnaires also addressed the unmet needs in the depression guidelines. The study design was evaluated and approved by the Medical Ethics Review Committee of Korea University.P < 0.05. All statistical procedures were performed using SPSS software version 18.0 0 .Descriptive statistics are presented with frequency and percentage distributions for categorical data and means and standard deviations for continuous data. Chi-squared and independent sample t-tests, paired sample t-tests, and one-way ANOVA were employed for between-group comparisons. For all analyses, the level of statistical significance was set at The descriptive statistics of selected characteristics of the participants are given in Table Identified factors reported to affect diagnosis and treatment decisions were experience of the clinician in 258 (68.1%), treatment guideline in 107 (28.2%), and others such as textbooks, articles, recent reports, web-site information, inertia of previous practice, and other things in 14 (3.8%).Seventy-five percent (n = 283) of psychiatrists reported that they had heard about the depression guidelines. Of these, 74.6% said they had read the depression guidelines. They first heard about the guideline through various sources: formal material from the academy (74.9%), their own reading of articles (11.9%), colleague discussions (4.7%), and the Internet and others (8.6%). The most recent time of referring to the guideline was reported as within the last 6 months in 46.0%, 6 to 12 months in 24.2%, and 12 months in 29.2% of participants. About 90 of the 211 (89.6% of) respondents who said they had read the guidelines also reported applying the depression guidelines in their practice, although most of them said they only applied the guidelines partially. Only 4.7% of psychiatrists reported using the guidelines exactly as written, and 5.7% of psychiatrists reported not using the guidelines although they had read them. About 90% of experts reported feeling slightly or moderately confident in using the guidelines, but only 10% were very confident in using them.Approximately half (48.2%) of the experts reported using the guidelines some or most of the time when making decisions about patient care, while 26.8% reported using the guidelines for decisions about medication. Thus, about 73.9% of experts used the depression guidelines to some degree in their clinical practice. Other ways of using the guidelines included: for education of patients and their caregivers (19.4%), training of other clinical team members (15.79%), understanding depression in general, and diagnosis of depression. The mean rating of helpfulness of the guidelines on a visual analog scale was 50.65, and the factor most associated with assessing the treatment guidelines as reliable was the reliability to the organization that developed the guideline (57.0%).A large proportion of the surveyed experts (88.1%) reported an opinion to recommend the depression treatment guidelines for team members. The reported reasons for recommending the guidelines were their usefulness for the treatment of depression (74.6%) and for agreement of treatment among doctors 24.3%). On the other hand, reasons for objecting to the guidelines were that it can be a limitation for clinical practice (42.2%), the difference between practice and the guidelines (24.4%), and questioning the usefulness of guidelines in general (20.2%)  were knowledge-related, such as lack of awareness 28.2%) and lack of familiarity (21.4%), attitude-related such as disagreement with guidelines and not concerning for experts (18.5%), and systems-related, such as lack of support from the government and difficulty in application with the national health insurance (21.0%). Other reasons included other insurance issues, patients’ misperception of depression and its treatment, the complex nature of some cases for which the guidelines provide no advice, not enough knowledge and understanding of the guidelines, and rejection of the treatments (10.9%). The experts asked for more help in using the depression treatment guidelines, and they stated the usefulness of a summary booklet, web-site, program development, teaching sessions or symposium, and compatibility with national health insurance  to continue prescribing antidepressants more than two months because of the restrictions of reimbursement by the national Health Insurance Review and Assessment Service. Hence, depression patients are referred to psychiatrists when they are acknowledged by the general practitioners.Most psychiatrists in this study reported intending to use the guidelines, but some were concerned about certain inherent limitations. Reasons for reluctance to use the guidelines included the following reported opinions: using the guidelines limits their professional independence, there is a gap between practice and guidelines, and using guidelines does not improve their work. Some were skeptical about the objectivity of guidelines and considered that recommendations could not be applied to their individual patients. Moreover, the most often reported barrier for implementation of the guidelines was little knowledge and experience with them. Health system restrictions were the next more reported barrier. Previous reports have also identified barriers at the clinician, clinical, and service levels.It is important to be aware of these barriers so that they may be overcome. One essential factor in successful implementation is physicians' faith in the guidelines, particularly reflected in the common physician concerns about the relevance of guideline for patients and the loss of autonomy in treatment decision making. Acceptable, built-in, guideline-based treatment alternatives – rather than a single treatment mandate – would increase adherence by permitting physicians some level of treatment choice. Guidelines must also be easy to use and understand, as well as time efficient, in order to be sustainable,19. MorePrevious studies showed a number of strategies to encourage the use of clinical practice guidelines, including multifaceted interventions involving audit and feedback of treatment practices, reminders about appropriate use of guidelines, local consensus processes in adoption of guidelines, and interactive educational meetings. As our There are some limitations in this study that should be acknowledged. First, the survey results may not be generalizable to all the psychiatrists who did not participate, although the response rate of 14.2% is similar with previous studies. The list of potential participants also could include retired psychiatrists. However, we could not reveal that who is retired or not on the list. The response rate might be higher if this is taken into account. Also, annual report of Korean neuropsychiatric Association does not give information on the members’ education, and other demographic data. Authors assumed that the psychiatrists who did not answer would have similar educational background but there could be other issues in generalization of the present results. Clinician have poor responses rates to surveys in general, and webThis study has demonstrated that psychiatric experts providing mental health services frequently applied the depression guidelines for general decision making, for medication selections, and for information transmission to patients and their caregivers. Physicians’ adherence is critical in translating evidence-based recommendations into improved outcomes. However, individual factors such as lack of awareness and lack of familiarity and external barriers such as the reimburse restriction can affect the employment of the guidelines in the real practice. Changes should be made to improve public health by increasing implementation of the guidelines, but proper implementation can be achieved with active promotion of the guideline contents and addiThere are no conflicts of interest for any of the authors. All authors disclose any financial and personal relationships with other people or organizations that could inappropriately influence and/or bias their work.CH was integral in the design and writing of the article. JY conducted analyses and wrote the first draft of the manuscript. S-YP and M-JK recruited the subjects and performed the web-survey. H-KY, C-UP and JA contributed to the interpretation of data and gave their advice throughout the study. All authors contributed to and have approved the final manuscript.The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-244X/13/150/prepub


Szepligetella levipetiolata, Szepligetella deercreeki Deans and Mikó sp. nov., Szepligetella irwini Deans and Mikó sp. nov., and the nearly cosmopolitan Evania appendigaster. A fifth species, Szepligetella sericea, including Szepligetella impressa, syn. nov., has not yet been collected in New Caledonia but can be found on islands throughout the Pacific and so is included in the diagnostic key. Taxonomic descriptions are unparalleled sources of knowledge of life's phenotypic diversity. As natural language prose, these data sets are largely refractory to computation and integration with other sources of phenotypic data. By formalizing taxonomic descriptions using ontology-based semantic representation, we aim to increase the reusability and computability of taxonomists' primary data. Here, we present a revision of the ensign wasp (Hymenoptera: Evaniidae) fauna of New Caledonia using this new model for species description. Descriptive matrices, specimen data, and taxonomic nomenclature are gathered in a unified Web-based application, mx, then exported as both traditional taxonomic treatments and semantic statements using the OWL Web Ontology Language. Character:character-state combinations are then annotated following the entity–quality phenotype model, originally developed to represent mutant model organism phenotype data; concepts of anatomy are drawn from the Hymenoptera Anatomy Ontology and linked to phenotype descriptors from the Phenotypic Quality Ontology. The resulting set of semantic statements is provided in Resource Description Framework format. Applying the model to real data, that is, specimens, taxonomic names, diagnoses, descriptions, and redescriptions, provides us with a foundation to discuss limitations and potential benefits such as automated data integration and reasoner-driven queries. Four species of ensign wasp are now known to occur in New Caledonia:  These phenotypic descriptions are difficult to data-mine . By working with real data early in the development of methodologies employing SPs, we identify and address some of the limitations of this approach. We conclude with a discussion as to both the real and perceived future problems of SPs as used here and highlight potential explorations that may advance the approach in subsequent work.Ensign wasps are charismatic hymenopterans that develop as solitary predators of cockroach (Blattodea) eggs inside oothecae . Despitehttp://www.w3.org/TR/owl2-overview/ last accessed May 13, 2013), was deposited as an Resource Description Framework (RDF)-XML file  in the Dryad repository .Our SPs follow the entity–quality (EQ) approach, meaning we draw “entity” terms from an organism-specific anatomy ontology, and phenotypic quality terms from a taxon-agnostic ontology of qualitative descriptors. EQ is a guiding principle for formulating ontological class expressions (SPs), which represent the class of organisms that a given character state denotes. The phenotype class expressions we created follow the approach advocated by http://morphbank.net last accessed May 13, 2013), as image collection 783132. Verbatim specimen label data and museum coden information are provided in online Appendix 2. Taxonomic nomenclature, specimen data, supporting images, and character matrix-based descriptive statements were compiled in the open-source web application mx  through interactive forms and integrated batch-uploading. NL treatments that include nomenclatural, descriptive, and material-related sections were rendered from these data using automated mechanisms included in mx. NL character descriptions were formulated with ontology annotation in mind, and in some cases revised to facilitate annotation.Diagnostic characters were discovered during direct examination of specimens under an Olympus SZX16 stereomicroscope and by comparing standard view images of specimens. Digital images were made using an Olympus CX41 compound microscope, equipped with a DP71 digital camera. SEM micrographs were taken by Philips XL30 ESEM-FEG (ISU) and FEI Nova 400 NanoSEM (FSU) on Au–Pd-coated specimens. Original images are deposited at Morphbank . These data, along with ontologies listed in http://protege.stanford.edu/ last accessed May 13, 2013). Navigation of the descriptive matrix elements in Protégé was aided by a custom-built plugin, available from the Github source code repository . SP annotations were added to character states within Protégé as OWL class expressions using the built-in Manchester syntax (http://www.w3.org/TR/owl2-manchester-syntax/) last accessed May 13, 2013 editor. These phenotype annotation axioms were created in a separate OWL file which imported the character matrix OWL file exported from mx; this allowed edits to character data within mx to be integrated with in-progress phenotype annotation work in Protégé, by re-exporting and replacing the character matrix file  and the FaCT++ Description Logic reasoner, version 1.5.3 . We performed the same queries over a comparison data set covering another ensign wasp genus, generated in parallel to this one . For example, the “some” keyword in the following expressions signifies an existentially quantified property restriction. For a basic phenotype such as “wing shape: curved,” inheres_in some wing” , and those described from the perspective of the entity, for example, “wing and bearer_of some curved.” We adopted the entity-based form, which can be more conveniently associated with a specimen exhibiting the phenotype, through a has_part relationship:The SP expressions we created build on the OWL representation for EQ explicated by It is not necessary to explicitly include “shape” in the expression; the knowledge that “curved” is a subtype of “shape” is built into the PATO ontology.primary locator, whereas the containing structure is a secondary locator . We used a nested series of has_part restrictions, mapping neatly to Sereno's secondary locator(s), L:In many cases, the structure that bears the quality being described is an instance of a “general” class, which must be further specified. For example, this may be a class of repeated anatomical structure, such as “bristle,” or a class denoting an abstract spatial region; both of these require localization to a specific, containing, anatomical structure, such as “mesosoma.” In the terminology of Context-dependent anatomical entities such as the above are often described as standalone expressions using a “post-composition” approach in other EQ annotation software, such as Phenex . The enthas_part chain, which provides two advantages over the part_of construction: (i) the entity class is more proximately associated with the quality it bears within the Manchester syntax expression, helping the human annotator verify correctness of the expression and (ii) an automated reasoner can infer that the “locator” structure is part of the same organism as the entity structure, a fact that is not implied by the semantics of the part_of -based class expression.However, we found that postcompositions in our annotations were nearly always used to express parthood relationships; these structures could instead be represented using the aforementioned Building upon the basic EQ construct just described, we identified four template EQ expressions which could be used to express the meaning of the various character forms in the matrix:Relative measurement phenotypes highlighted an important limitation of OWL class expressions: without the ability to include variables within the expression, it was impossible to fully represent the intended meaning . Namely,http://www.w3.org/Submission/SWRL/ last accessed May 13, 2013). For the example phenotype above, the corresponding rule would state that if there is an organism which has as parts an antenna and an eye, and the antenna is increased_in_magnitude_relative_to the eye, then that organism is inferred to be a member of the class defining the phenotype. Within our data set, these rules are not actually exercised; they are included only as a clarification of relative measurement semantics within the limitations of the OWL 2 language.Unsatisfyingly, to be an instance of this class, an antenna needs to merely be longer than at least one eye in the world, not necessarily an eye possessed by the same organism. Although the semantics of the above phenotype description are not complete with respect to the meaning of the character, they still provide useful information about character data and specimens by making it clear that the given character describes aspects of antenna and eye size. Indeed, for not only relative measurement phenotypes but in fact all phenotype descriptions we created, the OWL class representing a phenotype for a given character state is defined not as equivalent to the EQ description but rather as a subclass—the provided semantics are necessary aspects of the phenotype but not a wholly sufficient description. To make the intended semantics of relative measurement phenotypes more explicit to consumers of the semantic description, we added a rule for each such character state using Semantic Web Rule Language  significantly departed from the basic templates, usually by incorporating more complicated intersection or union expressions. All SP expressions along with the original natural-language characters can be found in online Appendix 1.A phenotype annotation consists of an OWL class, P, the ontological description of that class using an EQ expression, and the linkage of the phenotype class to a particular character state, CS, by means of a class assertion:denotes only P)CS Type (denotes property is defined within the Information Artifact Ontology (IAO) to signify a reference by an informational entity to a “portion of reality.”The denotes to connect both the character states and the Operational Taxonomic Units (OTUs) within a character matrix to the actual organisms (specimens) being described. So, since in our data model an OTU denotes a particular set of specimens under investigation, it follows that a character state denotes any specimen whose OTU has that state as a matrix value. We encoded this assertion by defining an OWL property chain (a property chain describes a path of links in the RDF graph which imply a new direct link between the nodes at either end of the path) (denotes only P), we can infer that any specimens denoted by OTUs possessing a given character state  are members of that phenotype class. This logical framework allows us to propagate phenotypes to specimens while directly asserting semantic annotations only for character states (We made use of he path) . By connr states .Evaniscus (Szepligetella data set (this study) referenced a markedly broader range of quality descriptors than the Evaniscus data set: 41 concepts across 43 characters versus 30 concepts across 56 characters.As preliminary examples for how ontology-based annotation facilitates cross-data set computation, we performed queries, programmed using the OWL API, over both this data set and a comparison data set for a relatively distantly related genus, vaniscus . Becausevaniscus . AlthougEvaniscus data set characters describing color variation are much more prevalent than in this study .We also used the ontologies to assess the distribution of study characters across selected anatomical and qualitative partitions . By meanEvania appendigaster (Linnaeus 1758).– Antennal shelf presence: absent ; eye colAntennal shelf presence: present ; eye col– Female flagellum color pattern: monocolored; Carinae laterally on frons presence: absent ; lower fFemale flagellum color pattern: banded  and 6f; >2× ocellus diameter) presence: present (Szepligetella irwini Deans and Mikó sp. nov.>2× ocellus diameter) presence: absent  , 9c, dEvania appendigaster (Ichneumon appendigaster: Linnaeus (1758)  [sex unknown], deposited at LSUK, labels: “habitat in America.” Taxonomic history documented by Body length.—7.0–7.8 mm.Head.—Median clypeal projection sharpness: blunt. Lower face texture: foveae absent. Median carina of lower face presence: absent. Malar space length: longer than 0.5 of compound eye height. Carinae laterally on frons presence: absent. Antennal rim shape: not raised laterally. Antennal shelf presence: present. Eye color: blue. Long setae (length >2× ocellus diameter) presence: absent. Mandibular teeth count: 3. Female scape length: greater than eye height. Female flagellum color pattern: monocolored. Female flagellum ventral sensillar patch spatial arrangement: F5–F11.Mesosoma.—Shape of median area of pronotum: not recurved. Sulcus delimiting pronotal lobe presence: present. Pronotal lobe carina presence: absent. Anteromedian carina of the prosternum presence: absent. Mesoscutal humeral sulcus continuity: discontinuous. Notaulus shape: sigmoid. Notaulus lateral margins alignment: parallel. Median mesoscutal area shape: not prominent relative to lateral mesoscutal area. Scutoscutellar suture structure: not foveate. Ventro-lateral region of mesosoma texture: foveate. Anterolateral mesopectal projection 2d shape: scalene triangular. Speculum presence: absent. Epicnemium sculpture: smooth. Epicnemial carina shape: convex medially. Ventral margin of mesopectus length: shorter than ventral margin of metapectus length. Metapleural sulcus position: horizontal. Posterior margin of the propodeum ventrally of the propodeal foramen lateral view shape: convex. Gastral scrobe conspicuousness: inconspicuous. Lateral carina of gastral scrobe presence: absent. Submedian propodeal projection presence: absent. Nucha presence: absent. Female metatibial spines presence: absent. 1M length versus 1CUb length: 1M is distinctly longer than 1CUb. Distal part of 4RS shape: straight or arched proximally. Costal cell coloration: brown in the distal one-tenth.Metasoma.—Petiole texture: smooth. Petiole pilosity: dense. Lateroventral carina of the petiole presence: present. Setiferous patch on dorsal region of abdominal terga 4–7 in female presence: present.Syntype male: {no locality label} LINN 2719 (LSUK). Syntype female: {no locality label} LINN 2720 (LSUK). Other material: FRANCE: New Caledonia: one female. NCSU 49532 (PSUC). USA: TX: Brazos Co.: one female. NCSU 23566 (PSUC).E. appendigaster used in the majority of literature, whereas the female specimen (LINN 2720) is easily diagnosed, based on surface sculpture, mesosoma shape, appendage color, and wing venation, as Prosevania fuscipes (Illiger 1807). We wait to designate a lectotype, however, until these specimens can be observed directly.The two putative syntype specimens deposited at LSUK are not conspecific. The male specimen (LINN 2719) is morphologically consistent with concept of Deans and Mikó sp. nov. , 7e, fSzepligetella deercreeki Diagnosis.—Differs from all known Szepligetella species in the combination of the following character states: notaulus lateral margins alignment: diverging; median mesoscutal area shape: prominent relative to lateral mesoscutal area; setiferous patch on dorsal region of abdominal terga 4–7 in female presence: absent.Body length.—5.3–6.6 mm.Head.—Median clypeal projection sharpness: pointed. Lower face texture: foveate. Median carina of lower face presence: absent. Malar space length: shorter than 0.5 of compound eye height. Carinae laterally on frons presence: present. Antennal rim shape: raised laterally. Antennal shelf presence: absent. Eye color: gray-silver. Long setae (length >2× ocellus diameter) presence: absent. Mandibular teeth count: 4. Female scape length: equal to eye height. Female flagellum color pattern: monocolored. Female flagellum ventral sensillar patch spatial arrangement: F4–F11.Mesosoma.—Shape of median area of pronotum: recurved. Sulcus delimiting pronotal lobe presence: absent. Pronotal lobe carina presence: present. Anteromedian carina of the prosternum presence: present. Mesoscutal humeral sulcus continuity: continuous. Notaulus shape: sigmoid. Notaulus lateral margins alignment: diverging. Median mesoscutal area shape: prominent relative to lateral mesoscutal area. Scutoscutellar suture structure: foveate. Ventro-lateral region of mesosoma texture: areolate. Anterolateral mesopectal projection 2d shape: isosceles triangular. Speculum presence: present. Epicnemium sculpture: wrinkled. Epicnemial carina shape: concave medially. Ventral margin of mesopectus length: longer than ventral margin of metapectus length. Metapleural sulcus position: vertical. Posterior margin of the propodeum ventrally of the propodeal foramen lateral view shape: straight. Gastral scrobe conspicuousness: conspicuous. Lateral carina of gastral scrobe presence: absent. Submedian propodeal projection presence: absent. Nucha presence: present. Female metatibial spines presence: present. 1M length versus 1CUb length: 1M is distinctly longer than 1CUb. Distal part of 4RS shape: straight or arched proximally. Costal cell coloration: brown in the distal one-tenth; brown in the distal one-half–two-third.Metasoma.—Petiole texture: smooth. Petiole pilosity: sparse. Lateroventral carina of the petiole presence: absent. Setiferous patch on dorsal region of abdominal terga 4–7 in female presence: absent.Holotype: : FRANCE: New Caledonia: NCSU 43287 (CNC). Paratypes : FRANCE: New Caledonia: NCSU 41660 (CNC), NCSU 49529–49531 (BMNH).This species is named after Deer Creek High School in Edmond, Oklahoma, USA, for recently contributing to our understanding and appreciation of insects through poetry through the 2011 Hexapod Haiku Challenge, hosted by the NCSU Insect Museum.Deans and Mikó sp. nov. , 4e, 6fSzepligetella irwini Diagnosis.—Szepligetella irwini and S. levipetiolata differ from all known Szepligetella species in the following character states: submedian propodeal projection presence: present. Female flagellum color pattern: banded; Carinae laterally on frons presence: present.Szepligetella irwini differs from S. levipetiolata in the following character states: long setae (length >2× ocellus diameter) presence: present; female metatibial spines presence: absent.Body length.—6.5–6.8 mm.Head.—Median clypeal projection sharpness: pointed. Lower face texture: foveae absent. Median carina of lower face presence: present. Malar space length: shorter than 0.5 of compound eye height. Carinae laterally on frons presence: present. Antennal rim shape: raised laterally. Antennal shelf presence: absent. Eye color: gray-silver. Long setae (length >2× ocellus diameter) presence: present. Mandibular teeth count: 4. Female scape length: greater than eye height. Female flagellum color pattern: banded. Female flagellum ventral sensillar patch spatial arrangement: F6–F11.Mesosoma.—Shape of median area of pronotum: recurved. Sulcus delimiting pronotal lobe presence: absent. Pronotal lobe carina presence: present. Anteromedian carina of the prosternum presence: present. Mesoscutal humeral sulcus continuity: continuous. Notaulus shape: sigmoid. Notaulus lateral margins alignment: parallel. Median mesoscutal area shape: not prominent relative to lateral mesoscutal area. Scutoscutellar suture structure: foveate. Ventro-lateral region of mesosoma texture: areolate. Anterolateral mesopectal projection 2d shape: isosceles triangular. Speculum presence: present. Epicnemium sculpture: wrinkled. Epicnemial carina shape: concave medially. Ventral margin of mesopectus length: longer than ventral margin of metapectus length. Metapleural sulcus position: vertical. Posterior margin of the propodeum ventrally of the propodeal foramen lateral view shape: convex. Gastral scrobe conspicuousness: inconspicuous. Lateral carina of gastral scrobe presence: absent. Submedian propodeal projection presence: present. Nucha presence: present. Female metatibial spines presence: absent. 1M length versus 1CUb length: 1M is distinctly longer than 1CUb. Distal part of 4RS shape: straight or arched proximally. Costal cell coloration: brown in the distal one-half–two-third.Metasoma.—Petiole texture: smooth. Petiole pilosity: sparse. Lateroventral carina of the petiole presence: absent. Setiferous patch on dorsal region of abdominal terga 4–7 in female presence: absent.Holotype female: FRANCE: New Caledonia: NCSU 41664 (MNHN). Paratype female: FRANCE: New Caledonia: 44113 (INHS).The species name refers to the collector of the type specimens, M. E. Irwin. Figs. 4, 9a, bSzepligetella levipetiolata Evania levipetiolataTaxonomic history documented by Body length.—4.6–8.0 mm.Head.—Median clypeal projection sharpness: pointed. Lower face texture: foveae absent. Median carina of lower face presence: present. Malar space length: shorter than 0.5 of compound eye height. Carinae laterally on frons presence: present. Antennal rim shape: raised laterally. Antennal shelf presence: absent. Eye color: gray-silver. Long setae (length >2× ocellus diameter) presence: absent. Mandibular teeth count: 4. Female scape length: greater than eye height. Female flagellum color pattern: banded. Female flagellum ventral sensillar patch spatial arrangement: F6–F11.Mesosoma.—Shape of median area of pronotum: recurved. Sulcus delimiting pronotal lobe presence: absent. Pronotal lobe carina presence: present. Anteromedian carina of the prosternum presence: present. Mesoscutal humeral sulcus continuity: continuous. Notaulus shape: sigmoid. Notaulus lateral margins alignment: parallel. Median mesoscutal area shape: not prominent relative to lateral mesoscutal area. Scutoscutellar suture structure: foveate. Ventro-lateral region of mesosoma texture: areolate. Anterolateral mesopectal projection 2d shape: isosceles triangular. Speculum presence: present. Epicnemium sculpture: wrinkled. Epicnemial carina shape: concave medially. Ventral margin of mesopectus length: longer than ventral margin of metapectus length. Metapleural sulcus position: vertical. Posterior margin of the propodeum ventrally of the propodeal foramen lateral view shape: straight. Gastral scrobe conspicuousness: inconspicuous. Lateral carina of gastral scrobe presence: absent. Submedian propodeal projection presence: present. Nucha presence: present. Female metatibial spines presence: absent; present. 1M length versus 1CUb length: 1M is distinctly longer than 1CUb. Distal part of 4RS shape: straight or arched proximally. Costal cell coloration: brown in the distal one-half–two-third.Metasoma.—Petiole texture: smooth. Petiole pilosity: sparse. Lateroventral carina of the petiole presence: absent. Setiferous patch on dorsal region of abdominal terga 4–7 in female presence: present.Holotype male: FRANCE: New Caledonia, B.M.TYPE.HYM. 3a.287 (BMNH). Other material: FRANCE: New Caledonia: 87 males, 1 female (NCSU 41656). NCSU 51293 (CAS); NCSU 34829, 34842, 34848, 34860, 41650, 41654, 41656, 44107, 44114, 44116, 44119, 44124, 51298 (PSUC); NCSU 33643, 41652–41653, 41655, 41657, 41661–41662, 41665–41666, 41669, 41897–41898, 44100, 44103, 44108, 44110–44112, 44115, 44117, 44120, 44122–44123, 44125–44126, 44129, 51297, 51299 (MNHN); NCSU 34828, 34830–34841, 34843–34847, 34849–34859, 34861–34865, 41658, 41668, 44101–44102, 44104–44106, 44109, 44127, 51296 (INHS); NCSU 51294–51295 (CNC). Figs. 6, 8c, dSzepligetella sericea Evania sericeaEvania impressaBody length.—5.5–9.1 mm.Head.—Median clypeal projection sharpness: pointed. Lower face texture: foveate. Median carina of lower face presence: present. Malar space length: shorter than 0.5 of compound eye height. Carinae laterally on frons presence: present. Antennal rim shape: raised laterally. Antennal shelf presence: absent. Eye color: gray-silver. Long setae (length >2× ocellus diameter) presence: absent. Mandibular teeth count: 4. Female scape length: equal to eye height. Female flagellum color pattern: monocolored. Female flagellum ventral sensillar patch spatial arrangement: F4–F11.Mesosoma.—Shape of median area of pronotum: recurved. Sulcus delimiting pronotal lobe presence: absent. Pronotal lobe carina presence: present. Anteromedian carina of the prosternum presence: present. Mesoscutal humeral sulcus continuity: continuous. Notaulus shape: falciform. Notaulus lateral margins alignment: parallel. Median mesoscutal area shape: not prominent relative to lateral mesoscutal area. Scutoscutellar suture structure: foveate. Ventro-lateral region of mesosoma texture: areolate. Anterolateral mesopectal projection 2d shape: isosceles triangular. Speculum presence: present. Epicnemium sculpture: wrinkled. Epicnemial carina shape: concave medially. Ventral margin of mesopectus length: longer than ventral margin of metapectus length. Metapleural sulcus position: vertical. Posterior margin of the propodeum ventrally of the propodeal foramen lateral view shape: straight. Gastral scrobe conspicuousness: conspicuous. Lateral carina of gastral scrobe presence: present. Submedian propodeal projection presence: absent. Nucha presence: present. Female metatibial spines presence: present. 1M length versus 1CUb length: equal. Distal part of 4RS shape: arched distally. Costal cell coloration: brown in the distal one-tenth.Metasoma.—Petiole texture: furrowed. Petiole pilosity: sparse. Lateroventral carina of the petiole presence: absent. Setiferous patch on dorsal region of abdominal terga 4–7 in female presence: present.Evania sericeaEvania impressa>100 species of cockroaches on New Caledonia that construct egg cases  which could serve as “hosts” for ensign wasp larvae. Yet, we could only find two species of evaniid that are presumably native to the island and one introduced species, whose hosts are well known. The two native ensign wasp species could be (i) generalists that predate on numerous cockroach species .A critical consideration for the application of semantic technology to phenotypic descriptions is how well a logical representation can express the meaning currently conveyed by NL. For example, creating logical statements that encapsulate the full meaning of the single NL statement “aedeagus apical column broad, and flattened apically, ending abruptly with rounded basal angle and narrow, hooked apical lobe, column extending more than half its length beyond penis valve, bending dorsally,” is a much larger challenge relative to the statement “gonocoxa orange” , EQ translation was generally straightforward. However, as described in the “Results” section, one class of characters, “relative measurements,” proved challenging to adequately describe using OWL. Five out of the 43 characters described here involved relative measurements. Although they constitute only a little more than 10% of the characters in this study, relative measurements are a common and important means for describing morphological changes in a size-independent manner. The OWL limitation stems from a requirement that class descriptions exhibit a tree-like structure, a factor in decidability . Althougclassify NL phenotypic descriptions, rather than fully replace them. In contrast, With representational limitations in mind, we suggest that semantic annotation should be primarily considered as a means to Indeed, the ontological classification of characters in this study and in It is clear that widespread adoption of semantic methodologies within systematics will require development of tools that facilitate, rather than complicate, systematists' work. The initial approach demonstrated here requires some familiarity with both OWL and the Protégé application. Our cdao-protege plugin made navigation of characters and associated states within Protégé fairly straightforward; however, the mechanics of manually creating ontology classes for phenotypes and consistently creating all the required links to character states did prove challenging to nonexpert Protégé users. Even so, it is our experience that interested biologists can quickly comprehend and apply OWL class descriptions, particularly using the English-like Manchester Syntax. We have prototyped a fully integrated SP annotation interface directly within mx, building on the approach begun with the Phenex annotation application . Howeverhttp://portal.hymao.org last accessed May 13, 2013) to compare their descriptive text against the HAO and to return ontology concept URIs .Either approach, careful SP description or simple semantic tagging, requires a well-developed multispecies anatomy ontology for maximum utility. The importance of expert morphologists has never been greater in this regard . CommuniAs the ontology language for the Semantic Web, OWL not only provides an ontological reasoning framework but also a means to publish our descriptions as RDF, contributing to the emerging universe of Linked Open Data . The usehttp://dx.doi.org/10.5061/dryad.2gd84.Data files and/or other supplementary information related to this paper have been deposited at Dryad under National Science Foundation ; and the National Evolutionary Synthesis Center and benefited from discussions initiated through the Phenotype Research Coordination Network (in part). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.This material is based on work supported by the