Incorrect classification of outcomes in clinical trials can lead to biased estimates of treatment effect and reduced power. Ensuring appropriate adjudication methods to minimize outcome misclassification is therefore essential. While there are many reported adjudication approaches, there is little consensus over which approach is best.Under the assumption of non-differential assessment , we use simulation and theoretical results to address four different questions about outcome adjudication: (a) How many assessors should be used? (b) When is it better to use onsite or central assessment? (c) Should central assessors adjudicate all outcomes, or only suspected events? (d) Should central assessment with multiple assessors be done independently or through group consensus?No one adjudication approach performs optimally in all settings. The optimal approach depends on the misclassification rates of site and central assessors, and the correlation between assessors. We found: (a) there will generally be little incremental benefit to using more than three assessors and, for outcomes with very high correlation between assessors, using one assessor is sufficient; (b) when choosing between site and central assessors, the assessor with the smallest misclassification rate should be chosen; when these rates are unknown, a combination of one site assessor and two central assessors will provide good results across a range of scenarios; (c) having central assessors adjudicate only suspected events will typically increase bias, and should be avoided, unless the threshold for sending outcomes for central assessment is extremely low; (d) central assessors can adjudicate either independently or in a group, and the preferred option should be dictated by whichever is expected to have the lowest misclassification rate.Outcome adjudication is of critical importance to ensure validity of trial results, although no one approach is optimal across all settings. Investigators should choose the best strategy based on the specific characteristics of their trial. Regardless of the adjudication strategy chosen, assessors should be qualified and receive appropriate training. Many randomized controlled trials involve binary outcomes where either an event occurred or it did not . In some situations it is self-evident whether an event occurred . However, in most situations it is less clear and may involve some subjectivity; thus, the outcome may need to be adjudicated to determine whether an event occurred. Appropriate adjudication of outcomes is of critical importance to the validity of trial results, as poor adjudication can lead to bias and a loss of power . HoweverFor example, clinical trials of ulcerative colitis often use an endoscopic video to determine whether an ulcer or a degree of inflammatory change is present or absent. There are several potential adjudication approaches in this setting. The outcome could be adjudicated either by a site assessor, who is directly involved in the patient’s care, and may be influenced by knowledge of the patient’s symptoms and their clinical history, or by a central assessor, who would only have access to the endoscopic video, and is blinded to all other clinical information. The number of adjudicators could be varied. The type of outcomes sent to the central assessors could vary . Different methods of adjudication could be used . There is currently little evidence to inform these choices.Assuming that there is a ‘true’ but unknown outcome, there are four possible results after adjudication Table ; true-poFalse-positives and false-negatives are forms of misclassification . AssuminThe extent of the bias under a non-null treatment effect depends on the rate of misclassification; the larger the rate of misclassification, the higher the degree of bias. For example, if the treatment effect was a difference in percentage points, the estimated treatment effect in the presence of misclassification will be biased by a factor of:φ is the false-positive rate (the proportion of non-events that were incorrectly classified as events) and θ is the false-negative rate (the proportion of events that were incorrectly classified as non-events) [where -events) . For exa-events) .Given the adverse consequences of misclassification of outcomes, reducing the misclassification rate is of critical importance to ensure the validity of results in randomized controlled trials. A number of case studies have compared results from different adjudication approaches in specific trials –15; howeIn this paper, we focus on the assessment approach for binary outcomes, and assume that there is non-differential assessment , as would typically be the case in a double-blind trial or in trials where all assessors are blinded to treatment allocation.We begin by describing a relevant example from clinical trials in ulcerative colitis. We then discuss the notation to be used, followed by an overview of the statistical properties of several different adjudication approaches. We then describe a simulation study comparing the different adjudication approaches. Key terminology used in this paper is listed in Table Ulcerative colitis is a chronic, relapsing disorder of the colon resulting from an excessive immune response against environmental antigens, although the aetiology is unknown. Patients typically present with symptoms of abdominal pain, bleeding and diarrhoea. Most clinical trials for ulcerative colitis require adjudication of endoscopy results (an examination of the patient’s colon using a fibre-optic camera) to determine whether there is evidence of disease activity . The results of the endoscopy are used to inform the outcome of clinical remission (yes vs. no), either alone or in conjunction with other information, such as patient-reported symptoms and a global physician assessment .The results from the endoscopy could be adjudicated based on a number of different approaches. For example, the results could be adjudicated by the local onsite clinician who performs the endoscopy (site assessor), or by a clinician not based at the site who adjudicates the outcome based on a review of the endoscopic video .Furthermore, a central assessor, if used, could adjudicate either all patients or only a subset of patients. For example, one adjudication strategy could be to have a site assessor perform an initial adjudication, and then have a central assessor to adjudicate suspected events only. Additionally, under central adjudication, any number of assessors could be used. For example, instead of using only one assessor, three assessors could be used. This would typically involve two assessors performing an initial adjudication, and if they disagree, a third assessor acting as a tiebreak. This approach could be generalized to any odd number of assessors, e.g. 3, 5, 7, 9, etc. An additional challenge in using central adjudication is whether to only include the assessments made by the central assessor(s), or to use a hybrid approach, which also includes the assessment made by the site assessor.Finally, under central adjudication with multiple assessors, the adjudication could be done either, independently by each assessor, or all together, as a group . Under the independent assessment approach, each assessor performs the adjudication independently, without knowledge of results from the other assessors; the final classification is then based on a ‘majority rules’ approach. For example, with three assessors, the outcome is classified as an event if the majority of assessors (two or more) adjudicate an event (and vice versa for no event). Conversely, the group approach would typically involve an endpoint review committee, in which all assessors meet and conduct the adjudication together.In the following sections, we outline some of the statistical considerations of these differing adjudication approaches. We then discuss some considerations for choosing an adjudication approach in ulcerative colitis clinical trials later in the paper.Yidenote the true outcome for the ith patient, with Yi = 1 indicating that an event occurred and Yi = 0 indicating that no event occurred. Assume that there are j different assessors; then, let Yij* denote the adjudicated outcome for the ith patient from the jth assessor: Yij* = 1 means that the assessor has adjudicated an event, and Yij* = 0 means that the assessor has adjudicated no event. Finally, let Yi* denote the final adjudicated outcome for the ith patient .Before considering any of the listed adjudication scenarios in detail, we outline some notation to be used. Let Thendenotes the false-positive rate (the probability of a non-event being incorrectly classified as an event), anddenotes the false-negative rate (the probability of an event being incorrectly classified as a non-event).Letβ is the true treatment effect, and p1 and p0 are the true probabilities of an event in the intervention and control arm respectively. Therefore, β represents the difference in proportions between treatment arms. The estimated treatment effect, where where In the presence of misclassification,β = 0 (indicating no difference between treatment groups) then indicating that the estimated treatment effect will always be closer to zero (indicating no difference between treatment groups) than the true difference, and that if Finally, we define the infraclass correlation (ICC) asP(Yij* = Yij′*) denotes the probability that two different assessors make the same classification for the same patient, and P(Yij* = Yi′j′*) denotes the probability that two different assessors make the same classification for different patients [where patients . The ICCpatients , 18, whiWe begin with the simplest question – if we were using only central assessors, how many assessors should we use? For the moment, we assume that adjudication will be done independently by each assessor, and that the final classification will be based on a ‘majority rules’ approach. This approach requires an odd number of assessors .Adjudication requires time, effort, and other resources , 14, 19;The potential advantage of increasing the number of assessors can be seen in the following example. Assume that all assessors have a false-positive rate of 10% . Furthermore, let us assume that results from each assessor are completely independent . Then, using only one assessor will lead to an overall false-positive rate of 10%. However, with three assessors, a false-positive will only occur if two or more assessors get the classification wrong. Therefore, with three assessors, the overall false-positive rate can be calculated as:i.e. using three assessors can reduce the overall false-positive rate from 10% to only 2.8%, which could substantially reduce bias and increase power.However, if we make the alternate assumption that ICC = 1 , then all three assessors would essentially function as one. In this situation, increasing the number of assessors from one to three has no impact, and the overall false-positive rate will be unchanged at 10%. Thus, increasing the number of assessors can reduce misclassification, thereby reducing bias and increasing power, but the extent of this benefit will depend on the correlation between different assessors; the higher the correlation, the less incremental benefit of adding extra assessors.Assuming that adjudication could be performed by either site assessors or central assessors , a natural question is which type of assessor to use. Central assessment typically requires additional resources compared with site assessment. For example, information needs to be collated and sent to the central assessors, who then need to take time out to perform the adjudication, often in a time-sensitive manner. Furthermore, specialized equipment may sometimes be required to record the event. Conversely, site assessment can be done on the spot, and is typically  done routinely. Therefore, using central assessment usually carries a resource burden, and should only be used when the benefit of doing so outweighs the costs.In practice, central assessment will benefit the trial when central assessors have lower misclassification rates than site assessors. Therefore, when central assessors are expected to be better at classifying outcomes, central assessment should be the preferred option. Conversely, when site assessors are expected to have better classification rates, they should be the preferred option. When the classification rates are expected to be approximately the same between site and central assessors, then either option could be used.In practice, however, the comparative misclassification rates of site and central assessors are often unknown and may be difficult to predict. In this situation, it is not clear which adjudication approach to use. We could then guess at which type of assessor is likely to be better, based on the specific trial characteristics. For example, for outcomes that are easier to adjudicate and do not require much experience or specific training , site and central assessors are likely to be comparable, and so using a site assessor to reduce resource burden makes sense. Conversely, for more challenging outcomes, which require experience and specific training , using central assessors may be the best approach, as it will be easier to ensure that a small number of central assessors are sufficiently experienced and have received appropriate and consistent training than to ensure the same for a large number of site assessors.An alternative option, when the comparative misclassification rates of site and central assessors are unknown, is to use a combination approach to adjudication; instead of using either site or central assessors, both could be used. For example, the independent adjudication of both the site and a central assessor could be used, with a second central assessor  used if a tiebreak is required.φSA and φCA denote the false-positive rates of the site and central assessors respectively, and θSA and θCA denote their false-negative rates. Then, the false-positive rate for the approach of using one site assessor and two central assessors (denoted φ1SA + 2CA) is:Under the assumption of independence between assessors (i.e. ICC = 0), we can derive the overall false-positive rate for the approach of using one site assessor and two central assessors as follows. Let The false-negative rate is derived similarly. We can see in Fig. both the site and central assessors adjudicate an event, and it will be classified as no event if either the site or central assessors adjudicate no event. We refer to this adjudication approach as a two-stage approach in this paper.A common adjudication strategy is for site assessors to perform an initial adjudication of all outcomes, and then for central assessors to re-adjudicate suspected events. This strategy is outlined in Fig. The two-stage approach is likely to be used in situations where it is believed that central assessors have better classification rates than site assessors, but central assessors come with a cost or resource burden, and so it is more economical to only have central assessors adjudicate a subset of outcomes in order to reduce costs. Therefore, a relevant question is: Under which circumstances is it acceptable to have central assessors only assess suspected events, rather than all outcomes? In the remainder of this section, we compare the two approaches .φSA and φCA denote the false-positive rates of the site and central assessors respectively, and θSA and θCA denote their false-negative rates. Then, the false-positive and false-negative rates for the two-stage approach  can be calculated asUnder the assumption of independence between site and central assessors (i.e. ICC = 0), we can derive the overall false-positive and false-negative rates under the two-stage approach as follows. Let andφSA, φCA, θSA and θCA are all between 0 and 1, this indicates thatBecause andThat is, the two-stage approach will always have a false-positive rate that is less than or equal to that of a site or central assessor alone; however, it will always have a false-negative rate that is equal or higher. Therefore, the two-stage approach will only be as good as or better than using a central assessor to adjudicate all outcomes when the reduction in the false-positive rate is equal to or greater than the increase in the false-negative rate, i.e. whenThis expression reduces toThis expression relies on several items; however, we can see that a primary factors that will influence whether the two-stage approach is useful is the false-negative rate of the site assessors; when this is very small, using the two-stage approach is likely to be beneficial. However, the assumption that site assessors will have very low false-negative rates is contradictory to the assumption behind the two-stage approach, which is that the site assessor’s misclassification rates are high enough that suspected events require further adjudication by central assessors to correct any mistakes the site assessor has made.This highlights the contradiction in logic behind the two-stage approach; that site assessors cannot be trusted to determine whether an event occurred, but can be trusted to determine whether an event did not occur. This approach assumes that correctly classifying events is more important than correctly classifying non-events, which is not the case; as seen in formula (1), misclassification of both events and non-events has the same effect on bias.The bias of the two-stage approach  is shown in Fig. From Fig. If multiple central assessors are used for adjudication, they could either conduct the adjudication independently , or they could perform the adjudication in a group if trial practicalities permit (e.g. as part of an endpoint review committee), where each patient is discussed, and the final classification is decided together. Under group adjudication, the outcome could either be decided by a vote (with a ‘majority rules’ approach), by consensus, with disagreements resolved by discussion , or through another method .Group adjudication may require additional resources compared with independent adjudication, for example, additional travel, time, or communication costs. Therefore, it is useful to know when we might expect an improvement in classification under the group approach, in order to determine whether the additional resource and logistical requirements are worthwhile.Depending on the dynamics of the group, we might reasonably expect group adjudication to either increase or decrease the classification rates compared with independent adjudication and, potentially, to increase the correlation between assessors. For example, there may be one dominant voice or an opinion leader in the group that the other assessors follow; this would have the effect of increasing correlation between individuals (as the group would effectively act as one voice). Furthermore, if the dominant voice belongs to an assessor with poor classification rates, this could increase the overall misclassification rate compared with independent adjudication. Conversely, if the dynamic is one of collaboration and engagement, a discussion of each patient may allow the group to arrive at the correct classification more often than if the assessors had done the adjudication alone.In practice, group adjudication may be preferred if it is expected that this would lead to better classification rates than achieved for independent adjudication. However, this must be balanced against the potential increase in correlation between assessors from group adjudication; as seen earlier, the higher the correlation, the less benefit there is to using more than one assessor. Therefore, if group adjudication increases the classification rate by a small margin, but leads to a substantial increase in the correlation, it may be that using independent adjudication actually leads to slightly better classification rates overall.Yi represents the true outcome for the ith patient, with Yi = 1 indicating that an event occurred, and Yi = 0 indicating that no event occurred. Yij* represents the adjudicated outcome for the ith patient from the jth assessor. Yij* = 1 indicates that the assessor has adjudicated an event, and Yij* = 0 indicates that the assessor has adjudicated no event. Finally, Yi* represents the final adjudicated outcome for the ith patient .We conducted a simulation study to compare the different adjudication approaches across these scenarios. As denoted, We generated the true patient outcomes from the following model:α is the probability of an event in the control arm, Xi represents the treatment group to which the ith patient was allocated , and, as before, β is the difference in proportions between the intervention and control arm (representing the treatment effect).where n = 1000 (500 patients in each of the two treatment arms), and α = 0.5 (denoting a 50% event rate in the control). We set β = −0.088 . We chose these values of α and β to give 80% power under perfect adjudication.For all simulation scenarios we set We generated the adjudicated outcomes from central assessors using a beta-binomial distribution with a specified ICC , and with a specified false-positive and false-negative rate. For scenarios that included a site assessor, we generated adjudicated outcomes from the site assessor based on the same beta-binomial model as for the central assessor, but using the specified false-positive and false-negative rates for the site assessor. This means that the correlation between site and central assessors will be less than the correlation between different central assessors, owing to the differences in the false-positive and false-negative rates.Yi*) for each scenario are provided next. Once we calculated Yi* for each patient, we used these adjudicated outcomes to estimate Yi* as the outcome, and the treatment arm (Xi) as the only covariate.Details of how we calculated an overall adjudicated outcome . With only one assessor, the final classification of the outcome was set to the one assessor’s adjudicated outcome. Formally:m represents the total number of assessors used.where φ and θ). We set the false-positive and false-negative rate to be equal for each assessor, e.g. if φ was 5% then θ was also set to 5%. We varied the ICC between 0.10, 0.25, 0.50, 0.75 and 0.90, and varied φ and θ between 5%, 10%, 15% and 20%. This led to 5 × 4 = 20 total scenarios.We varied the correlation between assessors (ICC) and the assessors’ false-positive and false-negative rates . For the approaches with one site assessor or one central assessor, the final classification was based on the assessor’s adjudicated outcome. For the approach with one site assessor and two central assessors, the final classification of the outcome was based on the adjudication of the majority of assessors , regardless of whether they were site or central assessors. For example, the final classification was an ‘event’ if two or more assessors adjudicated ‘event’, regardless of whether these were the two central assessors, or one central assessor and one site assessor.φSA = 20%, θSA = 20%○ Site assessor: φCA = 10%, θCA = 10%○ Central assessor: Scenario 1: central assessors have lower false-positive and false-negative rates:φSA = 10%, θSA = 10%○ Site assessor: φCA = 20%, θCA = 20%○ Central assessor: Scenario 2: site assessors have lower false-positive and false-negative rates:φSA = 20%, θSA = 10%○ Site assessor: φCA = 10%, θCA = 20%○ Central assessor: Scenario 3: site assessors have lower false-negative rates, central assessors have lower false-positive rates:We varied the ICC and the false-positive and false-negative rates for the different assessors. We varied the ICC between 0.10, 0.25, 0.50, 0.75 and 0.90. We used three different scenarios for the false-positive and false-negative rates of the assessors:This led to 5 × 3 = 15 total scenarios.One site assessorOne central assessorTwo-stage approachWe evaluated three different approaches to adjudication:We simulated a scenario where the central assessor generally had better classification rates than the site assessor .φCA = θCA = 5%); for the site assessor, we set the false-positive rate to 10% (φSA = 10%). We then varied the false-negative rate for the site assessor (θSA) between 5%, 10%, 15% and 20%. We varied the ICC between 0.10, 0.50 and 0.90 .For the central assessor, we set both the false-positive and false-negative rates to 5% .Central assessors adjudicate the outcome together in a group (group adjudication).We explored the scenario where multiple central assessors will adjudicate the outcome, and evaluated two different approaches:For each method, we used three central assessors. Final classification of the outcome for independent adjudication was based on the majority. For group adjudication, we assumed an approach where assessors would discuss each outcome before voting, with the final classification being based on the majority.We assumed that group adjudication could change the classification rate compared with independent adjudication (either increase or decrease it), and could increase the correlation between different assessors.We set the false-positive and false-negative rates to 20%, and the ICC to 0.50 for independent adjudication. Under group adjudication, we varied the change in the false-positive and false-negative rates between −2%, −1%, 0%, 1% and 2%. A change of −2% indicates that using group adjudication led to a decrease of 2% for both the false-positive and false-negative rate compared with independent adjudication (i.e. the rates were reduced from 20% to 18%). We varied the change in ICC under group adjudication between 0, 0.1, 0.2, 0.3 and 0.4. A change of 0.4 indicates that using group adjudication led to an increase in the ICC of 0.4 compared with independent adjudication (i.e. the ICC was increased from 0.50 to 0.90).This led to 5 × 5 = 25 total scenarios.Results are shown in Figs. Furthermore, the benefit of adding extra assessors depends on how many assessors there are to start with. For example, for false-positive and false-negative rates of 20%, Fig. Results are shown in Figs. Figure Figure Results are shown in Fig. Results are shown in Fig. As discussed previously, the key factors that will influence the choice of adjudication approach are (i) the comparative misclassification rates of the site and central assessors; (ii) the degree of correlation between assessors and (iii) the additional resource burden associated with central assessment.For trials in ulcerative colitis, there is no gold standard method of adjudication, and it is therefore impossible to estimate the exact misclassification rates of the site and central assessors (as we have no way of determining what the ‘true’ outcome is). However, some published data on the rates of agreement between site and central assessors suggest that the two types of assessor disagree in up to 30% of cases , 23. ThiPublished data suggest that there is generally moderate-to-high correlation between central assessors (range 0.50–0.83) , 25.Central assessors would generally adjudicate an endoscopic video through a web portal. The resource burden associated with this is primarily driven by the fact that the endoscopy needs to be videoed at the enrolling clinical site, the video needs to be uploaded into a secure web portal that meets regulatory requirements, a coordinating image management organization must operationalise the review and distribution of images to central assessors, and the central assessors typically need to be paid for their time.Based on the rate of disagreement between site and central assessors, the correlation between assessors, and the resource requirements for central assessment, several adjudication strategies could be proposed. If there is evidence to indicate that the disagreement between site and central assessors is primarily due to misclassification on the part of site assessors, then using only central assessors would be the preferred option. Because the estimated correlation between central assessors is high (>0.50), using more than three assessors would confer little benefit in terms of reducing bias; therefore, using one or three assessors is reasonable. If three assessors were to be used, we would recommend using independent (rather than group) adjudication, as group adjudication would lead to a large increase in resources, without necessarily improving the classification rate.not primarily due to misclassification by the site assessors  then using a combination approach  may be the preferred option. As before, we would use independent (rather than group) adjudication here.If we believe that the disagreement between site and central assessors is Regardless of which adjudication approach is chosen, all assessors should be provided training if necessary, and it may be useful to have them adjudicate a number of test cases before the trial starts to ensure adjudication is performed in a consistent manner.Outcome misclassification can lead to biased treatment effect estimates and reduced power, potentially resulting in an erroneous conclusion regarding treatment efficacy. The implementation of strategies to reduce misclassification is therefore of critical importance in clinical trials. We found that the choice of adjudication approach can have a large impact on trial results, and should be given careful consideration in the planning stages of a clinical trial. We found that no one approach is optimal across all scenarios; instead, the best approach will depend on the specific trial characteristics, notably the misclassification rates of the site and central assessors, and the correlation between assessors. The resource implications of the different approaches is also worth considering.Our conclusions are summarized in Table Outcome adjudication is of critical importance to ensure validity of trial results, although no one approach is optimal across all settings. Investigators should choose the best strategy based on the specific characteristics of their trial. Regardless of the adjudication strategy chosen, assessors should be qualified and receive appropriate training.


Synthetic cannabinoids (SCs), commonly known by the street name “Spice,” are designer drugs of abuse that mimic the psychoactive effects of marijuana. Intentional SC use has resulted in multiple toxicities  were collected over 2 days. Seventeen surface wipe samples were collected from the SC laboratory and the agents’ office. Air and surface wipe samples were analyzed for AB-PINACA and mitragynine using gas chromatography–mass spectrometry at Bureau Veritas North America, Novi, Michigan. No validated method exists for quantitative air sampling of SC, so a modified DEA method using gas chromatography–mass spectrometry and methanol extraction was used. The results obtained should be considered semiquantitative and the reported minimum detectable and minimum quantifiable concentrations should be considered estimates. The ventilation system of the office was evaluated by measuring airflow using a ventilation flow hood.Nine agents completed a questionnaire and, after providing informed consent, submitted urine samples for SC analyses. The questionnaire asked about demographics and work, medical, and smoking history; alcohol use; and pertinent health symptoms ever experienced when handling SC evidence. Each agent provided five urine specimens over 3 consecutive days. The first specimen (baseline) was collected on designated day 1 (the day before the raid); the second was collected at the end of the shift on day 2 (after the raid and evidence collection). The third was collected at bedtime on day 2; the fourth was collected before beginning the shift on day 3, and the fifth was collected at the end of the shift on day 3 (after sorting evidence in the office) . Liquid NIOSH investigators observed agents’ practices during the raid and evidence collection and processing. During the raid (day 2), the agents collected suspect plant material, sealed packets containing SC and mitragynine, baking sheets, and paper records. The evidence was placed in plastic or paper bags and stored in a locked evidence room located in the agency offices. The next day (day 3), the agents took evidence bags from the evidence room to a carpeted conference room, an adjacent equipment room with tiled flooring, and individual offices for sorting and cataloging.During the raid, some agents voluntarily wore disposable protective clothing, and one wore a filtering face-piece respirator during part of the raid. No agents wore disposable shoe coverings. The use of latex or nitrile gloves during the raid and evidence processing was inconsistent. Some agents entered the office after the raid without changing out of the clothing  worn during the raid. Agents were not provided with a designated locker room or showers. Agents were observed to be eating and drinking while handling and cataloging the evidence. The evidence storage and processing rooms were connected to the ventilation system that also supplied the other office areas. The ventilation system, the layout of the offices, and the agents’ work practices were not designed to contain or control forensic hazards.All nine participating agents were men, and their average age was 45 years (range = 35–60 years). The average time in their current job was 9 years (range = 4–19 years). Some agents reported tobacco or alcohol use, exposure to solvents outside of work, chronic medical conditions, and prescription medication use. All reported previously handling SC material evidence from previous drug raids. Approximately half of the nine agents reported ever having cough, eye irritation, throat irritation, and dizziness or lightheadedness  from previous activities handling SC . Four agAll nine agents reported handling or being in the same room with the SC compound during the NIOSH evaluation. Two agents reported that they always wore gloves, five said they sometimes or usually wore gloves, and two said they did not wear gloves when handling SC during this evaluation. Six agents reported that they sometimes wore respirators during their job, including dust masks, filtering face-piece respirators, or half-mask elastomeric respirators .3 and from 0.02 to 0.1 mg/m3, respectively. Only one wipe sample, taken from a baking sheet in the SC laboratory where treated plant material was dried, had a detectable level of AB-PINACA. No wipe samples had detectable levels of mitragynine.No AB-PINACA or mitragynine was found in any of the personal air or area air samples. The estimated minimum detectable concentrations for AB-PINACA and mitragynine were based on laboratory measurements and ranged from 0.03 to 0.1 mg/mLiquid chromatography–tandem mass spectrometry testing did not detect AB-PINACA, its metabolites, or mitragynine in any agent’s baseline urine specimen . After tAcute SC poisonings in the United States appear to be increasing, and several recent outbreaks among users have been reported , opium-like effects (pain relief and sedation), and modulation of opiate withdrawal, but it appears to have addictive properties .Law enforcement agents are at risk for dermal and ingestion exposures to SCs and other contaminants when collecting evidence in the field and processing evidence in the office; the risk for inhalation exposure is not well understood. A final report issued to the agency included the following recommendations to reduce personal exposures to potential contaminants during raids and while handling evidence: 1) properly design the forensic facility, 2) train agents on good hygiene practices, including prohibiting eating, drinking, and smoking while processing evidence, and 3) require a minimum level of PPE  to avoid skin contact or inadvertent ingestion. Other occupations, such as housekeeping staff members in the offices, also might be at risk for exposure. Additional research exploring air exposures from SCs, take-home exposures, laboratory methods, and preventive measures is needed. The related health hazard evaluation was published online by NIOSH in March 2016  have had severe health effects. New SCs continue to be manufactured despite Drug Enforcement Administration efforts. Law enforcement personnel have experienced symptoms during methamphetamine laboratory investigations, but little is known about their symptoms during SC enforcement activities.This is the first reported investigation of occupational SC exposure. SCs or their urinary metabolites or mitragynine were detected in the urine of six of nine law enforcement agents after they were involved in raiding an illegal SC manufacturing facility and collecting, processing, and cataloging SC evidence. No policies were in place regarding the appropriate handling of evidence, such as requirements for gloves and protective clothing, or on prohibiting food and drink in evidence processing areas. Shower and locker areas were not provided for agents to reduce contamination and prevent take-home exposure. The layout and ventilation of the agency’s office did not contain or control potential hazards from the receiving, processing, and storing of evidence.Law enforcement agents are exposed to material containing SCs during raids of illegal SC laboratories, and when collecting, processing, and cataloging SC evidence. A properly designed forensic facility and good hygiene practices can reduce personal exposures to potential contaminants during law enforcement raids and while handling evidence. A minimum level of personal protective equipment is needed, in addition to prohibiting eating, drinking, and smoking while processing SC evidence.


Culicoides biting midges and caused by African horse sickness virus (AHSV). AHS is endemic to sub-Saharan Africa, but devastating outbreaks have been recorded periodically outside this region. The perceived risk of an AHS outbreak occurring in Europe has increased following the frequent epidemics caused in ruminants by bluetongue virus, closely related to AHSV.African horse sickness (AHS) is a lethal equine disease transmitted by Attenuated vaccines for AHS are considered unsuitable for use in non-endemic countries due bio-safety concerns. Further, attenuated and inactivated vaccines are not compatible with DIVA  strategies. All these factors stimulated the development of novel AHS vaccines that are safer, more efficacious and DIVA compatible.We showed previously that recombinant modified Vaccinia Ankara virus (MVA) vaccines encoding the outer capsid protein of AHSV (AHSV-VP2) induced virus neutralising antibodies (VNAb) and protection against AHSV in a mouse model and also in the horse. Passive immunisation studies demonstrated that immunity induced by MVA-VP2 was associated with pre-challenge VNAb titres in the vaccinates. Analyses of the inoculum of these MVA-VP2 experimental vaccines showed that they contained pre-formed AHSV-VP2.We continued studying the influence of pre-formed AHSV-VP2, present in the inoculum of MVA-VP2 vaccines, in the immunogenicity of MVA-VP2 vaccines. Thus, we compared correlates of immunity in challenged mice that were previously vaccinated with: a) MVA-VP2 (live); b) MVA-VP2 (live and sucrose gradient purified); c) MVA-VP2 (UV light inactivated); d) MVA-VP2 (UV light inactivated and diluted); e) MVA-VP2 (heat inactivated); f) MVA-VP2 (UV inactivated) + MVA-VP2 (purified); g) MVA-VP2 (heat inactivated) + MVA-VP2 (purified); and h) wild type-MVA (no insert). The results of these experiments showed that protection was maximal using MVA-VP2 (live) vaccine and that the protection conferred by all other vaccines correlated strongly with the levels of pre-formed AHSV-VP2 in the vaccine inoculum. •MVA-VP2 vaccines (expressing African horse sickness virus VP2) induce protective immunity in mouse model and in horses.•Experimental MVA-VP2 vaccines contain preformed AHSV-VP2 in the inoculum if they are not sucrose-gradient purified.•MVA-VP2 vaccines express AHSV-VP2 in MVA-VP2 infected cells.•de novo’ synthesised AHSV-VP2 in MVA-VP2 vaccinates contribute to MVA-VP2 immunogenicity.Both pre-formed AHSV-VP2 and ‘ The disease is caused by African horse sickness virus (AHSV), a member of the genus Orbivirus, family Reoviridae, closely related to bluetongue virus (BTV). The non-enveloped 55–70 nm AHSV spherical virion consists of a triple-layered capsid surrounding ten double-stranded RNA segments and three proteins involved in viral replication  (African horse sickness (AHS) is an arthropod-borne viral disease of solipeds transmitted by haematophagus insects of the genus ctively) . The innctively)  and contctively) .Culicoides midges. Live attenuated AHS vaccines (LAV) have been in use in Africa for almost 100 years and permitted the subsistence of horses in that part of the world  and Chicken embryo fibroblast (DF-1)  were grown in high glucose Dulbecco's modified Eagle's medium (DMEM) supplemented with 2 mM glutamine, penicillin (100 units/ml), streptomycin (100 μg/ml) and 10% foetal calf serum (FCS). AHSV serotype 4  was grown in Vero cells and MVA viruses (wild type-MVA and MVA-VP2) grown in DF-1 cells. Virus stocks were generated by infection of sub confluent cells using a multiplicity of infection (MOI) of 0.1. When a total cytopathic effect (CPE) was visible, the cells and supernatants were harvested and centrifuged. The virus was released from the cells by three freeze/thaw cycles, sonication and then titrated by plaque assay.2.2a)MVA-VP2 (live): MVA-VP2, expressing AHSV-4 VP2, was previously described  and was b)MVA-VP2 (UV) was obtained by exposure of 1 ml aliquots of MVA-VP2 dispensed on a p100 tissue culture plate to 254 nm wavelength UV light for 40 min. UV irradiation was achieved by placing the plates at 3 cm below a TUV T8 lamp (Philips).c)MVA-VP2 (HI), was heat-inactivated by heating 100 μl MVA-VP2 aliquots in a thermal block for 15 min at 56 °C.d)MVA-VP2 (SGP) is a sucrose-gradient-purified MVA-VP2 preparation obtained as follows. A volume of 20 ml of MVA-VP2 was overlaid carefully on top of 20 ml of a sterile 36% (w/w) solution of sucrose in PBS using 40 ml ultra-clear centrifuge tube (Beckmann). The samples were centrifuged for 1.5 h at 300000 g and 4 °C in a SW 28 rotor. The pellets were re-suspended thoroughly in 0.5 ml sterile PBS and sonicated briefly before titration.Various MVA-VP2 vaccine preparations were used:2.3MVA-VP2 (HI) and MVA-VP2 (UV) were tested for infectivity using MVA-VP2 (live) as a positive control. All three preparations were inoculated (MOI of 1) into DF-1 cell cultures. After an adsorption period of 1.5 h cells were washed and medium was replaced. After 48 h, infected cells were washed three times with PBS and RNA was extracted from the cells using Trizol reagent (Invitrogen) according to the method recommended by the manufacturer.Reverse transcription-PCR (RT-PCR) was used to detect mRNA from AHSV-4 genome segment 2 (coding for VP2 protein) in DF-1 cells incubated with live and inactivated MVA-VP2. Oligonucleotide primers used to amplify a segment of 900 bp were: a) forward primer: 5′-CGCCCGGGATGGCGTCCGAGTTTGGAATATTG-3´; and b) reverse primer: 5′-CGCCCGGGCTACCCCTGCTTATCACCTGCTGA-3´.2 and Reverse Transcriptase (200 U/μL) was added and the reaction was incubated for 1 h at 50 °C. Amplification of the VP2 gene was performed by PCR using PCR Buffer II (Invitrogen), dNTPs, specific primers (forward and reverse primers), MgCl2 solution, AmpliTaq DNA Polymerase (Invitrogen), and cDNA template. Amplification cycle parameters were: 94 °C for 2 min (1×); 94 °C for 45 s, 55 °C for 1 min, and 72 °C for 2 min (30×); 94 °C for 15 min (1×).RNA was denatured in the presence of a reverse VP2 gene-specific primer and dNTP mix by heating to 65 °C for 5 min. A mix of RT buffer, MgCl2.42-KKKEEGEDDTARQEIRKAWC-COOH; NH2-NKGKWKEHIKEVTEKLKKA-COOH; NH2-DMNEKQKPYFEFEYDDFKPC-COOH) were selected to obtain a VP2-specific rabbit polyclonal antibody from a commercial source (GenScript). This antibody was used at a 1:400 dilution. A goat anti-rabbit peroxidase (Bio-Rad) was used at a dilution of 1:10000.Immunoblotting was performed as described previously . Samples2.5Seven-week-old, female, Type I interferon receptor KO A129 IFNAR (−/−) mice were purchased from B&K Universal. The animals were rested for about a week before the experiments were performed in the animal facilities of the Centro de Investigación en Sanidad Animal (INIA-CISA). All protocols for animal use were approved by the Ethical Committee of the Centre for Animal Health Research (CISA-INIA) (Permit number: PROEX 039/15) in strict accordance with the Spanish National Royal Decree (RD1201/2005), EU guidelines 2010/63/UE about protection of animals used for experimentation and other scientific purposes, and the Spanish Animal Welfare Act 32/2007.2.67 pfu/mouse  of each vaccine preparation for the first experiment. For the second experiment, 3 groups of mice (n = 5) were vaccinated with 107, 108 or 5 × 108 pfu/mouse of purified MVA-VP2. All immunisations were performed at day 0 and 21 by the intra-peritoneal route. Sera were collected from the submandibular vein at 2 weeks after the second vaccination for serological analysis.Mice (n = 5) were inoculated twice with 102.750 of AHSV-4 for 1 h at 37 °C. Then, samples were inoculated into one well each of 12-well plates containing confluent monolayers of Vero cells. Following incubation for 1 h at 37 °C in 5% CO2 an agar overlay  was added and plates incubated further for 4 days at 37 °C in 5% CO2. Plaques were visualized with a counter-stain solution . PRNT titre was calculated as the reciprocal (log10) of the highest dilution of serum that neutralised 50% of the control virus input.Serial dilutions of mouse sera were incubated with 100 TCID2.86 TCID50 of AHSV-4 per mouse administered by the sub-cutaneous route. Animals were monitored twice a day after challenge.Two weeks after the second vaccination, mice from the first experiment were infected with 102.9Clinical scoring was performed as described previously . Mice weWhole-EDTA-blood samples were collected from the submandibular vein at different days post infection to analyse viraemia. Blood cells were lysed with water before performing plaque assays on Vero cells as described previously .2.1050 titre was explored using linear regression with log10 PRNT50 titre as the response variable and log10 dose as the explanatory variable. A significance level of P = 0.05 was used in all analyses. All methods were implemented in R version 3.2.1 (R Core Team 2015).In the first experiment, differences in survival amongst groups of mice receiving each vaccine were compared using log-rank tests. Clinical scores in groups of mice receiving each vaccine were compared using Kruskal-Wallis tests. Differences between groups were then explored using post hoc Wilcoxon rank-sum tests. Differences in viraemia amongst groups of mice receiving each vaccine were assessed using linear mixed models with log10 pfu + 1 as the response variable, days post infection (as a factor) and group as fixed effects and mouse as a random effect. Model selection proceeded by stepwise deletion of non-significant terms (as judged by the Akaike information criterion) starting from a model including days post infection and group as main effects. In the second experiment, the relationship between vaccine dose and PRNT37 pfu/mouse; 108 pfu/mouse or 5 × 108 pfu/mouse of MVA-VP2 (SGP). Sera were obtained from these animals after vaccination and VNAb titres determined.In order to investigate the immunogenicity of MVA-VP2 vaccines in relation to pre-formed AHSV-VP2 present in the inoculum, we conducted two vaccination experiments using various MVA-VP2 vaccine preparations. In the first vaccination experiment we vaccinated 8 groups of 5 mice with either of the following: a) MVA-VP2 (live); b) MVA-VP2 (SGP); c) MVA-VP2 (UV); d) MVA-VP2 (UV 1/4) prepared by diluting MVA-VP2 (UV) in DMEM; e) MVA-VP2 (HI); f) MVA-VP2 (UV) + MVA-VP2 (SGP); g) MVA-VP2 (HI) + MVA-VP2 (SGP); and h) wild type-MVA (no insert). Following vaccination, animals were challenged with AHSV-4 and protective immunity assessed. In the second experiment, we vaccinated 3 groups of 5 mice with either 10de novo from MVA-VP2 infected host cells.Some of these MVA-VP2 vaccines were generated by inactivating the infectivity of MVA-VP2 cell lysate stocks by exposure to heat or UV light. Therefore, the immunogenicity of these vaccines would rely only on the presence of pre-formed AHSV-VP2 in the inoculum and not on VP2 protein synthesised 3.1Both inactivated MVA-VP2 vaccines did not produce cytopathic effect in DF-1 cell cultures after an incubation period of 5 days. In order to corroborate that inactivation of infectivity was successful and that live MVA-VP2 virus was not present in the inoculum of MVA-VP2 (HI) and MVA-VP2 (UV) vaccines, fresh DF-1 cell cultures were inoculated with MVA-VP2 (HI) or MVA-VP2 (UV) and 48 h later VP2-mRNA detection was performed by RT-PCR. PCR products of the expected size could only be obtained from MVA-VP2 (SGP) and positive control MVA-VP2 (live) infected cultures . Additio3.2All the vaccine preparations used in the first mouse experiment were analysed for the presence of VP2 in the inoculum by immuno-blotting with VP2-specific rabbit polyclonal antisera. The results indicated that, with the exception of MVA-VP2 (SGP) and MVA (wt), all samples showed a clear band of approximately 116 kDa, which corresponds to monomers of AHSV-VP2 .Fig. 2Pr3.3Protective immunity against AHSV-4 conferred by each of the MVA-VP2 vaccine preparations was assessed in a vaccination and challenge experiment.3.3.150 = 2.10), followed by those obtained from all groups that were vaccinated with preparations containing un-diluted inactivated MVA-VP2 (PRNT50 titres ranging between 1.52 and 1.75). As expected, sera from MVA-VP2 (1/4 UV) vaccinates showed PRNT50 titres just below the positive cut-off value of the assay (PRNT50 = 0.99). Similar values were recorded for MVA-VP2 (SGP) vaccinates (PRNT50 = 1.03). The data collectively suggest that the presence of pre-formed AHSV-VP2 in the inactivated MVA-VP2 vaccines played an important role in their capacity to induce a VNAb response and that the immunogenicity was reduced when the pre-formed VP2 in the inoculum was diluted or removed from the inoculum of live MVA-VP2 vaccines (at the dose of 107 pfu/mouse).Determination of VNAb titres against AHSV-4 of post-vaccination (day 35) sera showed that all MVA-VP2 vaccinates developed VNAb. However, there were significant differences between some groups . Highest3.3.2All vaccinates exhibited survival rates of 100% (five out of five mice surviving), except those receiving MVA-VP2 (SGP) and MVA-VP2 (¼ UV) . As expected, all MVA (wt) vaccinated mice succumbed to AHSV-4 challenge. Survival was significantly (P < 0.001) lower in mice receiving wild-type MVA compared with mice receiving any of the other vaccines.a > MVA-VP2 (1/4 UV) b > MVA-VP2 (SGP) b > MVA-VP2 (HI) c > MVA-VP2 (UV) c > MVA-VP2 (SGP) + MVA-VP2 (UV) cd > MVA-VP2 (SGP) + MVA-VP2 (HI) d > MVA-VP2 (live) d (where groups share a common superscripted letter they do not differ significantly (P > 0.05)).Clinical scores  differedAnalysis of viraemia data  show thade novo synthesis of VP2 in MVA-VP2 infected host cells plays an additional role in immunogenicity since MVA-VP2 (live) vaccines induced the highest protection levels in this study.Collectively the results from the challenge show that groups vaccinated with MVA-VP2 (SGP) and MVA-VP2 (1/4 UV) displayed low levels of clinical protection against AHSV-4, and this correlated with the levels of viraemia displayed by the animals. From the data it can be inferred that the levels of protection observed in the animals also correlated with the presence of pre-formed VP2 in the inoculum of the vaccines. MVA-VP2 (1/4 UV) presented a weak western blot signal corresponding to AHSV-VP2 and this was almost non-existent in the MVA-VP2 (SGP) lane . When th3.4de novo’ from MVA-VP2 (SGP)-infected host cells since the inoculum is virtually free from VP2. Since we have observed that an MVA-VP2 (SGP) vaccination dose of 107 pfu/mouse induced a very poor VNAb response, a dose-response experiment was conducted to determine if serum VNAb titres could be induced at all by increasing the dose of the MVA-VP2 (SGP) vaccine.The previous experiment indicated that the immunogenicity of MVA-VP2 vaccines relies on the total mass of VP2 protein that is presented to the immune system following inoculation of the vaccine. The VP2 protein in the inoculation site of the vaccinated animals originates from two sources: a) pre-formed in the inoculum; and/or b) expressed from MVA-VP2-infected host cells. In MVA-VP2 (SGP) vaccinated animals, VP2 is almost exclusively derived ‘7, 108 or 5 × 108 pfu/mouse and the post-vaccination sera subjected to PRNT testing. The mean PRNT titres of vaccinated mouse sera were 1.05, 1.33 and 1.67 for the 107, 108 and 5 × 108 dose groups, respectively. Statistical analysis of the data demonstrated a strong correlation between dose and serum titres (P < 0.001), with every one log increase in dose resulting in a 0.36  increase in log10 PRNT titre. These data suggest that whilst live MVA-VP2 vaccines containing cell lysates rely on the pre-formed VP2 for inducing a protective VNAb response, de novo synthesis of VP2 from host cells plays also a role in MVA-VP2 immunogenicity. The immunoblotting experiment performed with the MVA-VP2 (SGP) vaccines (8 pfu/mouse), were lower than those of the MVA-VP2 (1/4 UV), which induced a very low VNAb titre in the first vaccination experiment. This indicates that the positive correlation between VNAb titre and the dose of MVA-VP2 (SGP) is probably associated with the increased expression of VP2 in cells of the vaccinated animal. This hypothesis is consistent with results of immunofluorescence experiments performed on MVA-VP2 infected DF-1 cells showing that the numbers of cells expressing VP2 increase proportionally to multiplicity of MVA-VP2 infection (data not shown).Thus, 3 groups of mice were vaccinated with MVA-VP2 (SGP) using doses of 104Recombinant MVA viruses have been used experimentally for the prevention of infectious diseases both in humans and in a large variety of animal species . MVA vacWe have shown previously that recombinant AHSV vaccines based on expression of AHSV-VP2 (MVA-VP2) are able to consistently induce VNAb responses, cell-mediated immunity and protection against virulent challenge, both in a mouse model and in the equine species . Furtherde novo synthesised within cells of the MVA-VP2 vaccinated host influence the induction of VNAb and protection. Our results showed that MVA-VP2 induced immunity relies strongly on the levels of the protein that are delivered to the immune system at the time of vaccination and that both sources of VP2 played a role in the MVA-VP2 vaccine immunogenicity. Thus, all the MVA-VP2 vaccine preparations that contained pre-formed VP2 in their inoculum induced VNAb and protection against disease and viraemia. Protection levels were reduced when the levels of VP2 were reduced or absent in the inoculum of live MVA-VP2 vaccines (at a dose of 107 pfu/mouse). However, the protective capacity was restored when the inoculum of such vaccines was supplemented with pre-formed VP2, as was observed in the case of mice vaccinated with MVA-VP2 (SGP) + MVA-VP2 (UV) or MVA-VP2 (SGP) + MVA-VP2 (HI).In this paper, we studied the factors that determined the immunogenicity of MVA-VP2 vaccines. Specifically, we studied how the pre-formed VP2 that is present in the MVA-VP2 vaccine inoculum and VP2 that is However, sterile immunity was only induced by MVA-VP2 (live) vaccines, which contain pre-formed VP2 and also express VP2 in host cells following vaccination. The maximum level of protection in MVA-VP2 (live) vaccinates could be explained by the highest VNAb titres reached by these animals at the time of challenge. We determined previously  that pro7 pfu of MVA-VP2 (SGP) induced very low levels of VNAb (just below a titre of 10) and it was not clear whether the MVA-VP2 immunogenicity depended exclusively on pre-formed VP2 present in the inoculum. However, in the dose-response experiment we showed that increasing doses of MVA-VP2 (SGP) correlated with increasing VNAb titres in the vaccinates sera. This strongly suggests that VP2 is expressed from within host cells and is effectively processed and presented to the immune system to induce VNAb and possibly cell-mediated immunity. Therefore, it can be concluded that both pre-formed VP2 in the inoculum and de novo expression of VP2 from MVA-VP2 infected cells of the vaccinated host contributed to the induction of the highly protective immune response of MVA-VP2 vaccination, but that if administered in a purified form, MVA-VP2 would need to be administered at higher doses than when it is administered as a crude lysate.Any of the mechanisms alluded to above could only occur if expression of VP2 within cells of the MVA-VP2 vaccinated host actually took place. In the first experiment described in this paper, a dose of 10With regards to the reduced immunogenicity observed in mice vaccinated with MVA-VP2 (SGP) vaccine, it is worth noting that other poxvirus vector vaccines are formulated with an adjuvant . Thus, CThe observations made in our study open new avenues for improving the efficacy of MVA based vaccines for infectious diseases in general and for AHSV in particular. Consequently, it would be interesting to test whether improved antibody responses could be achieved by enhancing the secretion of the antigens from MVA-VP2 infected cells, which would facilitate B-cell receptor binding of the antigen. Targeting the MVA-expressed antigens to the extracellular space might also contribute to preserving the native structure of relevant conformational epitopes. In addition, it would also be equally interesting to examine the importance of the antigen processing and MHC presentation pathways within host cells infected with MVA vaccines to direct more effectively the induction of T-cell responses. In previous experiments we showed that the main effector mechanism of immunity of MVA-VP2 vaccines was mediated by VNAb, though other studies  and the The use of recombinant MVA viruses expressing AHSV-VP2 have been shown to be a suitable approach for developing safe, efficacious AHS vaccines with DIVA  capability . Further


Attempts at measuring quality of rehabilitation care are hampered by a gap in knowledge translation of evidence-based approaches and lack of consensus on best practices. However, adoption of evidence-based best practices is needed to minimize variations and improve quality of care. Therefore, the objective of this project was to describe a process for assessing the quality of evidence of clinical practices in traumatic brain injury (TBI) rehabilitative care.A multidisciplinary team of clinicians developed discipline-specific clinical questions using the Population, Intervention, Control, Outcome process. A systematic review of the literature was conducted for each question using Pubmed, CINAHL, PsychInfo, and Allied Health Evidence databases. Team members assessed the quality, level, and applicability of evidence utilizing a modified Oxford scale, the Agency for Healthcare Research and Quality Methods Guide, and a modified version of the Grading of Recommendations, Assessment, Development, and Evaluation scale.Draft recommendations for best-practice were formulated and shared with a Delphi panel of clinical representatives and stakeholders to obtain consensus.Evidence-based practice guidelines are essential to improve the quality of TBI rehabilitation care. By using a modified quality of evidence assessment tool, we established a process to gain consensus on practice recommendations for individuals with TBI undergoing rehabilitation. For patients who suffer significant neurologic and functional deficits due to traumatic brain injury (TBI), inpatient rehabilitation centers provide comprehensive post-injury care that has been shown to improve functional outcomes and successful reintegration into the community . The strTo further compound lack of a uniform approach to rehabilitation, the current literature supporting care practices in TBI rehabilitation does not allow for the identification of evidence-based care guidelines for optimal dose or intensity of therapy, the ideal timing of therapy in the recovery process, or the necessary modifications for subpopulations. This is due to variability in care practices and a lack of rigorously executed research including randomized controlled trials , 9, 13. A team of research personnel and multidisciplinary clinicians who specialize in treating patients with TBI was established including representatives from the following disciplines: physiatry, neuropsychology, physical therapy, occupational therapy, speech language pathology, and therapeutic recreation. Clinicians created a list of keywords specific to their discipline based on common clinical problems or symptoms that they treat in their day-to-day practices. For example, key words chosen by physical therapists included hemiparesis, spasticity, balance, and ataxia. Research personnel then conducted a preliminary literature search using the keywords combined with “traumatic brain injury” to identify existing literature. Based on the initial evidence collected, clinicians were then asked to form research questions in PICO format  related to their clinical discipline and TBI rehabilitation interests . The goal of this exercise was to generate a list of PICO questions in order to evaluate current literature and develop evidence-based recommendations.We conducted a systematic literature review for each individual PICO question using search terms identified by the clinician in conjunction with the corresponding “brain injury” medical subject heading. PubMed, CINAHL, PSYCInfo, and Allied Health Evidence databases were utilized because they were projected to include the most content related to TBI rehabilitation research across clinical disciplines. Inclusion criteria for articles were limited to clinical trials, randomized controlled trials, comparative studies, observational studies, and case-series. Meta-analyses were also reviewed for relevant papers. Articles were included if they pertained to adults (≥18 years of age) with TBI and were published in English from January 1, 1987 – current date. Additionally, studies with a patient population of mild through severe and acute and chronic TBIs were considered. Relevant articles cited within manuscripts were also reviewed to find additional evidence to support the PICO question. Finally, articles were collated and sent to the clinician to review for applicability.After identifying relevant literature for PICO questions, the next step was to assess the quality of evidence for the purpose of developing clinical practice recommendations . ResearcWe then reviewed other methodologies used to generate evidence-based recommendations and developed a modified ranking methodology. This combined the level of evidence methodology from the 1) Oxford Centre for Evidence-Based Medicine 2011 Levels of Evidence , (2) stu Oxford CThe multidisciplinary team assessed the quality of evidence for one PICO question at a time. The team members individually reviewed the articles and completed the quality of evidence assessment table. Afterwards, the clinician who developed the PICO question led the discussion, and as a group, level of evidence, applicability, and quality of evidence were discussed and scored for each article. Additionally, footnotes were produced to indicate the rationale for how the quality of the articles was assessed .IRB approval was obtained to recruit Delphi panel members consisting of clinical professionals who treat patients with TBI and stakeholders  with a waiver of informed consent. Panel members were recruited through TBI professional organizational membership rosters for each disciplinary field via email invitation. Additionally, TBI survivors and their family members who participate in our local TBI advisory council were recruited to gauge the relevance of the research for real-world application.Based on the results of the quality of evidence assessment table, a draft recommendation was created using GRADE recommendation formatting . Table The quality of evidence assessment table and corresponding articles were shared with Delphi panel members for review. However, footnotes and additional qualifying information by our group of clinicians was excluded to avoid biasing Delphi panel members. A conference call was then held to discuss the impetus of the study, history regarding the TBIMS, background information regarding the quality of evidence classification schemes employed in this study, and the process of draft recommendation development. Panel members were then directed to a HIPAA compliant web-based survey tool where they rated on a 4-point Likert scale the degree to which they agree with the draft recommendation Figure . If 70% Developing evidence-based clinical practice guidelines is critical to improving quality of care and rehabilitation outcomes for TBI survivors. The process described in this paper provides a comprehensive approach to addressing the issue. However, there were several constraints that were noted throughout the process that warrant discussion as a means of refining the process. First, several attempts were made trying to utilize GRADE methodology in its original form for each PICO question before the quality of evidence review process was reorganized. After reorganization, all PICO questions that were reviewed under GRADE methodology were reevaluated using the modified ranking methodology. In addition, due to a lack of research articles for many of the original PICO questions created, several meetings were dedicated to generating new PICO questions. Furthermore, it was difficult for clinical staff to fit literature reviews and meetings times related to the recommendation process into their clinical load. Likewise, the time and process to recruit Delphi panel members was also challenging given personal and professional demands on time.While GRADE is deemed the gold standard for evaluating evidence and generating recommendations, it was not applicable for TBI rehabilitation research due to the lack of published TBI rehabilitation articles that had used robust methodologies  within each discipline of rehabilitation . In addiThere is a need for more evidence-based practice guidelines to improve quality of rehabilitation care for TBI survivors. Recommendations were created across physiatry; neuropsychology; and speech, physical, occupational, and recreational therapy disciplines using the process described above. In creating the process, it became evident that there is a need for higher-quality research for individuals with TBI in the rehabilitation setting. Additionally, it is important to develop and standardize utilization of common data elements and outcome measures for research studies to enable researchers to more adequately compare different therapies and identify best practices. Finally, it is valuable and necessary to include the input of TBI survivors and family members when selecting the best therapies to treat TBI sequelae and help survivors achieve their goals .


There are diverse opinions and confusion about defining and including patient values and preferences  in the guideline development processes. This article aims to provide an overview of a process for systematically incorporating values and preferences in guideline development.In 2013 and 2014, we followed the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach to adopt, adapt and develop 226 recommendations in 22 guidelines for the Ministry of Health of the Kingdom of Saudi Arabia. To collect context-specific values and preferences for each recommendation, we performed systematic reviews, asked clinical experts to provide feedback according to their clinical experience, and consulted patient representatives.We found several types of studies addressing the importance of outcomes, including those reporting utilities, non-utility measures of health states based on structured questionnaires or scales, and qualitative studies. Guideline panels used the relative importance of outcomes based on values and preferences to weigh the balance of desirable and undesirable consequences of alternative intervention options. However, we found few studies addressing local values and preferences.Currently there are different but no firmly established processes for integrating patient values and preferences in healthcare decision-making of practice guideline development. With GRADE Evidence-to-Decision (EtD) frameworks, we provide an empirical strategy to find and incorporate values and preferences in guidelines by performing systematic reviews and eliciting information from guideline panel members and patient representatives. However, more research and practical guidance are needed on how to search for relevant studies and grey literature, assess the certainty of this evidence, and best summarize and present the findings.The online version of this article (doi:10.1186/s12955-017-0621-0) contains supplementary material, which is available to authorized users. According to the World Health Organization (WHO), “a recommendation [in a practice guideline] tells the intended end-user of the guideline what he or she can or should do in specific situations to achieve the best health outcomes possible, individually or collectively…” . A recomAlthough infrequently practiced, ideally this information should be based on evidence from thoroughly conducted research, which is collected through a systematic approach . The maiThe Grading of Recommendation, Assessment, Development and Evaluation (GRADE) working group developed the Evidence-to-Decision (EtD) framework to facilitate the process of integrating the criteria considered necessary in guideline development and documenting such process for different audiences , 5. WithIn fact, the definition of and strategies for determining values and preferences are still under debate. The GRADE approach includes the consideration of patient values and preferences as the relative importance of outcomes or health states of interest , 17, 18.Box 1. Relevant criteria in Evidence-to-Decision FrameworkDespite the increasing importance of practice guidelines in the management of health problems, there is a lack of evidence informing about initiatives using values and preferences in the guideline development process. Therefore, we addressed the challenges of integrating values and preferences in practice guidelines. Generally, we utilized the GRADE system for guideline development that is endorsed by over 100 organizations and applied worldwide  effectiveness because there is a concern that some patients in Saudi Arabia would not accept SLIT with some allergens of animal origin”. Consequently, the recommendation was a weak recommendation suggesting sublingual immunotherapy for treatment of adults with seasonal or intermittent allergic rhinitis based on moderate quality evidence . AlthougThe recommendation comparing ultrasonography versus mammography, as part of the triple assessment of palpable breast masses in women aged 30–40 years, was associated with very low certainty in the evidence of effects. The panel suggested “patients would likely favour the use of ultrasonography” because mammography can be more painful and uncomfortable for patients. In the panels’ view this consideration of values and preferences justified a strong recommendation because ultrasonography showed better diagnostic accuracy (sensitivity and specificity) compared with mammography despite very low certainty in the evidence .Incorporating values and preferences in guideline development required resources on the following levels: literature searches, screening and synthesis, preparation of the GRADE EtD frameworks and consideration of values and preferences in decision-making. During development of the search strategy, we noted that many relevant studies were difficult to identify because of the lack of a validated filter or of standardized keywords  being used to tag eligible studies. With the definition, measurement and methodology of values and preferences for guidelines still under debate, our aim to not miss relevant information was time and resource consuming. We managed this burden by limiting our search strategy through the stepwise use of a geographic search filter when required. For example, in the Migraine Headache guidelines, we first applied a geographic filter. After identifying no eligible studies, we felt it was necessary to spend additional time and resources to do a larger search for indirect evidence outside of the local context.Panels recognized the importance of explicitly incorporating the information in the process and considered it in all of the 226 recommendations. The structured summary and presentation of the values and preferences information for each question in the GRADE EtD framework facilitated the process of considering this type of evidence.We describe an approach for the incorporation of the relative importance of health outcomes in healthcare recommendations. We applied a multi-faceted approach utilizing a systematic review strategy complemented by other information sources. We use illustrative examples to show the usefulness of identifying relevant studies and using their findings in drafting the recommendations.The systematic and transparent approach to identify and summarize published literature on values and preferences is the strength of the proposed strategy. The feedback from experienced panel members suggested that we did not miss important relevant studies. A second strength is our pre-conceived and structured approach to incorporate both published and elicited local values and preferences in the decision making process. Guideline developers can assume an international or national, or, alternatively, a localized or specific perspective. By considering the appropriate setting the recommendations could potentially be more acceptable to stakeholders. While the former strategy would be helpful for international organizations such as WHO, those adapting recommendations to a specific setting should consider locally relevant evidence, as was the case in this project , 38.This study has some weaknesses. While the study is based on the development of over 20 guidelines and over 200 recommendations, it is restricted to one geographic setting. Also, limited local information was identified for patient values and preferences. The one related advantage is identifying the necessity of conducting more research on local values and preferences. Second, our definition and eligibility criteria for values and preferences were broad. The inclusion of a variety of study designs resulted in challenges with determining the eligibility of individual studies and the category they belong to. The time and resources spent on systematic reviews of values and preferences varied across guideline topics. We also did not formally assess the certainty or the quality of the evidence in the values and preferences from published studies. As for information about values and preferences from panel members, the collected information was unsystematic, potentially biased, and sometimes difficult to use. Furthermore, we were not able to assess publication bias due to the nature of the study question, study design and the geographic filter we used. While we identified studies with a variety of designs providing relevant evidence, the lack of standardized methods for reporting and identifying the evidence places additional limitations on current guideline development but not on our work.Although the integration of values and preferences is considered standard for trustworthy guideline development processes, using systematic reviews to identify values and preferences in a structured approach is still uncommon , 39, 40.There are other guideline efforts that consider patient values and preferences in the process of developing recommendations. For example, the National Institute for Health and Clinical Excellence (NICE) also considers the impact of values and preferences on the strength of recommendation. The process includes asking patient representatives to reveal their experience in addition to reviews of qualitative research evidence and cross-sectional surveys. However, NICE does not operationalize values and preferences as the importance of outcomes .Thus, despite recently increasing numbers of available primary studies and systematic reviews on values and preferences –44, theyAlthough considering the relative importance of health outcomes is essential in informing healthcare decision-making, use of this type of information remains a complex area to integrate. Our experience shows that guidelines in general and GRADE EtD frameworks in particular, lend themselves to the incorporation of this aspect in clinical and public health recommendations. To further facilitate this process a methodologically rigorous and consistent approach for reporting, summarizing and interpreting the information is needed due to the great heterogeneity on the definition, perspective and measurement of values and preferences. We provide an empirical approach to address this concern through systematic reviews and panel members’ input.