Electronic health  literacy of consumers is essential in order to improve information and communication technology (ICT) use for health purposes by ordinary citizens. However, performed eHealth literacy is seldom studied. Therefore, the present study assessed perceived and performed eHealth literacy using the recent conceptualization of health literacy skills.The aim of this paper was to examine the association between perceived and performed eHealth literacies.In total, 82 Israeli adults participated in the study, all 50 years and older, with a mean age of 67 (SD 11). Of the participants, 60% (49/82) were women and 72% (59/82) had a post-secondary education. The participants were first surveyed and then tested in a computer simulation of health-related Internet tasks. Performed, perceived , and evaluated eHealth literacy were assessed, and performed eHealth literacy was also recorded and re-evaluated later. Performance was scored for successful completion of tasks, and was also assessed by two researchers for motivation, confidence, and amount of help provided.P=.01) while facets of performance  were highly correlated . Participants low and high in performed eHealth literacy were significantly different: low performers were older and had used the Internet for less time, required more assistance, and were less confident in their conduct than high performers.The skills of accessing, understanding, appraising, applying, and generating new information had decreasing successful completion rates. Generating new information was least correlated with other skills. Perceived and performed eHealth literacies were moderately correlated (r=.34, The moderate association between perceived and performed eHealth literacy indicates that the latter should be assessed separately. In as much, the assessment of performed eHealth literacy in clinical settings should entail the structuring of tasks as well as shortening and automatizing the assessment. Electronic health  services have been rapidly expanding in many directions  yet conneHealth literacy includes the concept of health literacy ,4 as welr=.60; [Though the association between perceived and performed digital literacy has been extensively examined in several studies (for a review see ), few str=.60; ). As conr=.60; , it is pThe current study aimed at examining the association between the eHEALS as a perceived measure of eHealth literacy and eHealth literacy performance on both digital skills and content-related health Internet skills. Health Internet skills were conceived in terms of the following recent conceptualization on health literacy : (1) accThe following research questions were examined in this study: (1) Is successful completion rates of a task higher for relatively simple skills such as accessing and understanding health information and lower for appraising and applying? Is generating new materials the least successful task? (2) Is there an association between perceived and performed eHealth literacy, both at the overall skill level and between the components of the skills? (3) Is there a negative association between assistance provided in the performance tasks and skill level, both perceived and performed? (4) What are the associations between performed eHealth literacy and background characteristics ?A telephone survey and a face-to-face computer simulation (performance test) were conducted. The following sections describe participant recruitment, data collection, the tasks participants were asked to perform, and data analysis.Participants were recruited by a nationally representative random-digital-dial telephone household survey of Israeli adults aged 50 years and older. Calls were placed to 1206 residential households of whom 603 agreed to be interviewed, representing a 50.00% response rate and a sampling error of 2.04%. As there were only 206 participants  who used the Internet for health purposes in the representative sample, the sample was augmented by an additional 236 individuals , resulting in 442 Internet users. Interviews were conducted in Hebrew, Arabic, and Russian by professional interviewers who went through a special training session to familiarize themselves with the questionnaire's terminology. The interviewers conducted the telephone survey using computer-assisted telephone interviewing software. At the end of the survey, participants who used the Internet for health purposes were asked whether they would be willing to participate in a second phase of the study. Those who agreed  were asked to provide contact information.All 101 survey participants who agreed to participate in the second stage of the study were contacted and 28  agreed to take part in the simulation and its recording. An additional 54 participants were recruited in a snowball fashion, using a selective quota to reach a sample as close as possible to the representative survey sample regarding gender, age, education, chronic medical conditions, and income, resulting in a total 82 participants who completed both the survey and the performance simulation .The survey took place first. Respondents to the telephone survey who agreed to be later contacted for the second phase of the study were tested in their homes. Participants who were recruited via snowball were also first contacted by telephone, followed by the survey administration and then the home test. The survey took about 30 minutes to complete whereas the performance simulation took approximately 1.5 hours to complete. The simulation was carried out on a portable computer connected to a cellular modem and was recorded by a TechSmith Morae Recorder, version 2.2. This approach controlled for quality of the hardware, software, and Internet connection, and ensured that the setting was similar for all participants. The advantage of conducting the simulation at the participant's home is that they were in a familiar location; however, the shortcoming is that they were required to use a computer that was configured differently from the device they ordinarily used, which may have affected their performance.The telephone survey was conducted between December 2013 and March 2014. The computer simulations were carried out at the participants' homes between May 2014 and April 2015 and all participants signed an informed consent form and indicated their preferred language in the simulation. The participants who were recruited through the snowball technique responded to the telephone survey a couple of days prior to performing the face-to-face computer simulation. Participants were given a sequence of 15 assignments one at a time. Although there was a time frame allocated for each assignment, participants were not aware of it. When they hesitated or had difficulties completing tasks, the researcher helped them to complete the task and move on to the next. The researcher documented every instance that assistance was provided.Perceived eHealth literacy was measured by the eHEALS tool . The scaPerformed digital and eHealth literacy were measured through the completion of 15 computerized simulation tasks. The tasks were adapted from previous work -18,23,24A researcher performed a detailed and an overall observational judgment on each participant’s performance. The observational judgments pertained to the participants’ motivation to carry out the tasks, confidence, and proficiency level. All observational evaluations ranged from 1 (poor) to 5 (good). The observational evaluations were carried out both immediately after the completion of the tasks and later on the recorded performance. Two such observational evaluations were carried out on each performance, and in cases of disagreement a third observational evaluation took place.Once the time limit for task completion elapsed or a participant said she/he was about to give up on the task, participants were offered assistance. The researchers evaluated the amount of assistance given to participants and the assistance was summed across digital aspects , and health aspects .Demographic and background variables related to health and Internet use  were documented as part of the survey.First, the data for basic descriptive statistics for the key variables of background information, perceived and performed eHealth literacy was analyzed. Second, a series of bivariate tests were conducted to assess the association between the key variables of perceived and performed eHealth literacy and also with assistance provided. The participants were then divided into two groups, based on their performed eHealth literacy, and their scores on perceived eHealth literacy, amount of assistance provided, evaluated performance, and background characteristics compared using the analysis of variance (ANOVA) procedure. All analyses were carried out using SPSS Statistics, version 23.0 .Characteristics of the simulation sample and the survey representative sample are presented in Performance in the 15 tasks comprising the simulation was grouped according to skill type . The descriptive statistics on performance and success rate in completing each skill type and the descriptive statistics for perceived eHealth literacy are shown in 1,74=16.96, P<.001, eta square=0.186). In addition, they also had significantly fewer years of experience using the Internet with mean values of 10.54 (SD 5.81) and 14.13 (SD 6.14), respectively . They reported marginally significantly less education  and perceived themselves as marginally significantly less healthy than the high eHealth performing group with mean values of 2.95 (SD 0.93) and 3.36 (SD 0.89), respectively . There were no significant differences between the high and low eHealth literacy performance groups in perceived income  and the number of chronic medical conditions , nor were there differences in the gender distribution between the groups, for example 43% (17/40) men and 58% (23/40) women in the high performing group .In order to examine the concurrent validity of performed eHealth literacy, participants were assigned to two groups based on their mean score obtained on the performed eHealth literacy scale, similar to an analysis carried out by van der Vaart et al . We usedr=.34 (P<.01), and a similar association was found between performed digital literacy and perceived eHealth literacy . The correlation between the perceived access factor was significant with performed skills of accessing, understanding, appraising, and applying  and the least with performed skill of generating . The correlation between the perceived appraise factor was significant with all performed skills  except generating . Generating information also correlated the least with all other performed skills and overall performance.Pearson correlations between overall perceived and overall performed eHealth literacy was computed, as well as correlations between the perceived eHEALS factors of accessing and appraise in both modalities . The cor1,79 = 16.59, P<.001, eta square=0.174). Participants in the low performed eHealth literacy group were also granted more assistance, but only in the digital aspect of the tasks  and not on the health content aspect  of the tasks. Participants in the low performed eHealth literacy group were consistently evaluated as significantly lower in skill , confidence , and marginally significant in motivation  by the researchers, compared with the high performing eHealth literacy group.Participants in the low and high performed eHealth literacy groups were compared in terms of their perceived eHealth literacy score, the amount of help they received , and the researchers’ judgment on motivation, skill, and confidence . Particir=.67, P<.001) between the two kinds of assistance, so that the more assistance one was given on digital tasks the more assistance they were also given on the eHealth content tasks. Assistance on digital aspects was negatively associated with both perceived  and performed score  assessments suggesting that the more one was given assistance on digital aspects of tasks the lower the performed score and the lower the perceived skill. However, assistance on eHealth content was negatively associated with perceived eHealth literacy  while not significantly associated with performed eHealth literacy .Pearson correlations between assistance provided for digital and eHealth content tasks and perceived and performed eHealth literacy were computed. There was a positive association . Perceived eHealth literacy was assed using the eHEALS tool, whereas the performed eHealth literacy assessment was built on methodology and materials developed previously ,21, whilThe study has several important findings. First, the more complex the skill , the lower the successful completion of tasks. Successful completion rates thus created a gradient made of accessing, understanding, appraising, applying, and generating information. The skill of generating information  is of special interest since the success rates in this task were very low; however, it is unclear whether the task is more cognitively taxing or merely an unfamiliar activity for people in this age group.The second and main finding of this study is that perceived and performed eHealth literacy is significantly associated with each other, though to a moderate degree. The finding suggests that people make a reasonable, though not accurate, evaluation of their skill level. The significant association is in line with findings on perceived and performed numeracy , though A third finding is that participants who performed low and high in performed eHealth literacy were different from each other in other aspects reported in this study , re-iterating previous findings on the digital divide in the health domain ,11. InteThe study possessed several strengths. First, it assessed eHealth literacy through actual performance, not relying on self-perceived assessment. It thus joins few works ,11,21 inThe study has also several limitations. First, the sample is age-skewed to older adults, from 50 years and older. Results could be somewhat different, especially in terms of successful completion rates of task, among a heterogeneously aged sample. Second, the fact that participants were recruited on a voluntary basis implies that they might already have been more interested in using the Internet and searching for information, which could have influenced the results. In addition, the snowball recruitment of some of the participants may have contributed to the relative homogeneity of the sample . Third, the skill of generating was assessed with only one task and future studies will probably enlarge the assessment of this skill in view of the increased prevalence of social media, the different interactive competencies called for , and as Performed eHealth literacy was assessed laboriously in this study: the simulation took about 1.5 hours and a similar amount of time was required to code and evaluate the performance of a single person. This duration is clearly impractical in clinical settings. This calls for the development of a computerized, tailored test for performed eHealth literacy. The results of the present work indicate what this future tool could look like. Specifically, the moderate association between perceived and performed eHealth literacy, the high completion rates of accessing tasks concurrent with low variance, and the low completion rates in the generating task point to several attributes. First, the test needs to be short so that it can be applied in clinical settings. Hence, it could be adaptive so that performance determines the next task which saves time in measuring items an individual is likely to succeed in. Second, the test could contain a few perceived eHealth literacy items; perceptions take little time to measure and in this case are indicative of performance, at least in Web 1.0-related tasks . Third, A better understanding and assessment of eHealth literacy is essential in order to improve ICT use for health purposes by ordinary citizens. Improved understanding and assessment are prerequisites for enhancing eHealth literacy, thereby empowering patients in self-management of their health. This is even more important to those needing this most, such as long-term patients and the elderly. The present study demonstrated that performed eHealth literacy could be validly and reliably measured, that it is related to both human observations of skill, motivation, confidence, provision of help, and background characteristics, on the one hand, and to self-perceived eHealth literacy, on the other hand. The next stage of developing computerized adaptive short testing tools for eHealth literacy is advocated.


Researchers and practitioners working in sports medicine and science require valid tests to determine the effectiveness of interventions and enhance understanding of mechanisms underpinning adaptation. Such decision making is influenced by the supportive evidence describing the validity of tests within current research. The objective of this study is to review the validity of lower body isometric multi-joint tests ability to assess muscular strength and determine the current level of supporting evidence.Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) guidelines were followed in a systematic fashion to search, assess and synthesize existing literature on this topic. Electronic databases such as Web of Science, CINAHL and PubMed were searched up to 18 March 2015. Potential inclusions were screened against eligibility criteria relating to types of test, measurement instrument, properties of validity assessed and population group and were required to be published in English. The Consensus-based Standards for the Selection of health Measurement Instruments (COSMIN) checklist was used to assess methodological quality and measurement property rating of included studies. Studies rated as fair or better in methodological quality were included in the best evidence synthesis.Fifty-nine studies met the eligibility criteria for quality appraisal. The ten studies that rated fair or better in methodological quality were included in the best evidence synthesis. The most frequently investigated lower body isometric multi-joint tests for validity were the isometric mid-thigh pull and isometric squat. The validity of each of these tests was strong in terms of reliability and construct validity. The evidence for responsiveness of tests was found to be moderate for the isometric squat test and unknown for the isometric mid-thigh pull. No tests using the isometric leg press met the criteria for inclusion in the best evidence synthesis.Researchers and practitioners can use the isometric squat and isometric mid-thigh pull with confidence in terms of reliability and construct validity. Further work to investigate other validity components such as criterion validity, smallest detectable change and responsiveness to resistance exercise interventions may be beneficial to the current level of evidence.The online version of this article (doi:10.1186/s40798-017-0091-2) contains supplementary material, which is available to authorized users. Isometric mid-thigh pulls and squats are reliable tests that can differentiate between strength ability of participants. These tests can be utilized by both male and females independent of their training status or experience.The isometric squat test can be used to track progress in muscular strength over time.Test protocols should be carefully considered and documented when using isometric multi-joint tests to ensure repeatability, such as joint angle at which testing occurred.Commonly isometric multi-joint tests are 5 s in duration, with two to three trials allowed and 3–5 min recovery between tests.Testing of specific strength capabilities is a critical aspect of understanding the strategies that best enhance muscular strength , 2. In aIt is suggested that isometric multi-joint tests (IMJT) provide a time efficient assessment of muscular strength , 15 thatValidity of a test refers to the degree to which it measures what it intends. Many different properties of validity can be assessed to examine the efficacy of a test. These properties include face and content validity, based on a judgement that the test is likely to measure the intended construct and that the test adequately represents the construct of interest . CriteriTo determine the appropriateness of IMJTs to evaluate muscular strength and responsiveness to resistance exercise interventions, knowledge of validity properties is essential. Noting previous recommendations that validity is accumulated from multiple studies and cannot be demonstrated ‘once and for all’ by any single study , there iThe Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines were followed . The updThe PICO process  was utilized to develop a search strategy based on the aims of the review. Initially the term ‘isometric multi-joint test’ was searched in Google Scholar, with the first 100 articles sorted by relevance and screened for variations in terminology within their titles, key words and text terms. All variations in terminology were recorded into a key terms list to enable the broadest possible search terms specifically relevant to the research question and aims of this review. A validated search method was adapted , and appI.Construct: studies with a lower body IMJT. Tests were maximal effort trials with the duration of the isometric test greater than 2 s to allow participants sufficient opportunity to achieve a true maximal force [al force , 24.II.Target population: studies involved healthy, uninjured, human participants between the ages of 15–65 years of age. Participants maintained normal nutritional habits throughout the duration of the study and were free from supplementation with any form of ergogenic aid.III.Measurement instrument: testing was performed using verifiable isometric testing instruments, allowing direct measurement of vertical ground reaction force through a force plate. In accordance with the recommendations made by James et al. [s et al. , force pIV.Measurement properties: studies involved the evaluation of any measurement properties relating to the IMJT .V.Full text: Studies published in English. Adequate disclosure of information relating the status of participants and testing methods to enable assessment of methodological quality was required.The following inclusion criteria were established based on the research question and agreed between authors:Relevant full-text articles were initially screened by DD to ensure inclusion criteria was met. The authors were familiar with the existing literature and had no conflicting bias with any of the literature screened for inclusion in the review. Agreement was required by two authors to exclude any study, a process previously outlined by Sampson et al. . Two autIncluded full-text articles underwent the following process of review. Firstly, for data extraction purposes, a study summary table was created outlining the number of participants and their strength training experience, description of performance test, measurement equipment and variables, level of familiarization and instruction, test position, as well as number of trials, duration and processing methods. Data extraction for studies included in the best evidence synthesis is presented in Table p < .05) or effect size moderate or large relating only to isometric peak force measurements [Secondly, the methodological quality of each study was reviewed against the COSMIN four-point scoring criteria  whereby the worst score counts in each subsection. The authors made interpretive adaptations to the protocol and are outline accordingly below. Box A  and box E  of the COSMIN procedure were deemed inappropriate for evaluation within this review as no statistical models were used to analyze data that was derived through the application of a questionnaire. Box F, Hypothesis were evaluated with specific relevance to multi-joint isometric tests only within each study. Box G, Cross-cultural validity was not evaluated, as none of the included studies were relevant to this section. Box H, criterion validity item 4, states can the criterion used is considered as a reasonable gold standard? This item was interpreted by defining the isoinertial repetition maximum (RM) squat as the current gold standard for evaluating resistance exercise. Box I, responsiveness item 7 states, was a proportion of the patients changed (i.e. improvement or deterioration)? This item was interpreted by the authors based on the magnitude of change in the group mean from baseline , measurement error (9 studies), hypothesis testing 26 studies), criterion validity (9 studies) and responsiveness (15 studies) . Wilson et al. [≥14.5% . Eight studies of poor methodological quality [In total, 15 studies were evaluated with only studies of poor methodological quality for isometric mid-thigh pull , 72, 73 n et al.  used the quality , 72, 73  quality , 75–77, Whilst not being rated within methodological quality rating, interpretability is an important characteristic of any measurement instrument across the included studies. Overall, 48 were rated positive and 11 as negative.For the best evidence synthesis studies of fair or better methodological quality were considered and summarized for IMJTs combined. Additionally, the level of evidence for each individual isometric test is reported. No studies were included in the best evidence synthesis for either measurement error or criterion validity. No studies within the best evidence synthesis used the isometric leg press and therefore were classified as unknown in terms of the level of evidence supporting the use of this test.With respect to reliability, there was strong evidence for combined IMJTs. There was moderate evidence for isometric mid-thigh pull and strong for isometric squat test. Evidence for hypothesis testing was found to be strong for combined IMJTs. Isometric mid-thigh pull and isometric squat tests individually have moderate level evidence supporting their use. Finally, there was moderate evidence for responsiveness of the isometric squat test and unknown evidence for the isometric mid-thigh pull and leg press test.Assessment of physiological mechanisms and adaptations associated with resistance exercise is critical to improve understanding and efficacy of interventions . The praStrong evidence was found for reliability of combined IMJTs including strong evidence for the isometric squat test and moderate evidence for the isometric mid-thigh pull independently. ICC values for reliability of peak force in the seven studies included in the best evidence synthesis ranging from ≥0.80 to 0.99, well above the acceptable threshold for ICC >0.7 as discussed in Baumgartner and Chung . ResearcOur results demonstrate strong evidence to support construct validity (hypothesis testing) for combined isometric multi-joint tests, with moderate evidence for the isometric squat test and isometric mid-thigh pull, respectively, as independent tests. Primarily, the experimental design of three studies in the best evidence synthesis were correlational , 43, 54 ≥0.70 with the gold standard test. This demonstrates a likelihood of strong evidence for criterion validity where future research investigating this critical component of validity satisfies key methodological criteria, such as appropriate participant numbers.It was hypothesized that criterion validity would be supported with moderate level evidence based on knowledge of existing literature examining the relationship of isometric tests to dynamic performance tests. Additionally, Juneja et al.  suggestsModerate evidence was found supporting responsiveness for combined IMJTs in keeping with our hypothesis. Surprisingly this evidence was found only in studies using the isometric squat test , 74. TheCommon methodological protocols for IMJTs are apparent amongst the best evidence inclusions Additional file 2:Full text inclusion study table. (DOCX 59 kb)


A thermodynamic model of thermoregulatory huddling interactions between endotherms is developed. The model is presented as a Monte Carlo algorithm in which animals are iteratively exchanged between groups, with a probability of exchanging groups defined in terms of the temperature of the environment and the body temperatures of the animals. The temperature-dependent exchange of animals between groups is shown to reproduce a second-order critical phase transition, i.e., a smooth switch to huddling when the environment gets colder, as measured in recent experiments. A peak in the rate at which group sizes change, referred to as pup flow, is predicted at the critical temperature of the phase transition, consistent with a thermodynamic description of huddling, and with a description of the huddle as a self-organising system. The model was subjected to a simple evolutionary procedure, by iteratively substituting the physiologies of individuals that fail to balance the costs of thermoregulation (by huddling in groups) with the costs of thermogenesis (by contributing heat). The resulting tension between cooperative and competitive interactions was found to generate a phenomenon called self-organised criticality, as evidenced by the emergence of avalanches in fitness that propagate across many generations. The emergence of avalanches reveals how huddling can introduce correlations in fitness between individuals and thereby constrain evolutionary dynamics. Finally, a full agent-based model of huddling interactions is also shown to generate criticality when subjected to the same evolutionary pressures. The agent-based model is related to the Monte Carlo model in the way that a Vicsek model is related to an Ising model in statistical physics. Huddling therefore presents an opportunity to use thermodynamic theory to study an emergent adaptive animal behaviour. In more general terms, huddling is proposed as an ideal system for investigating the interaction between self-organisation and natural selection empirically. self-organised criticality. Criticality is a hallmark of complex systems, and is evidenced here by the emergence of a power-law distribution of thermal properties in the evolving composition of the group. The model therefore reveals how complexity can emerge in a well-defined biological system (thermoregulation), where experiments can be designed to investigate the interaction between self-organisation and natural selection.Huddling is an adaptive behavior that emerges from simple interactions between animals. Huddling is a particularly important self-organising system because the behavior that emerges at the level of the group directly impacts the fitness of the individual. The huddle insulates the group, allowing pups to thermoregulate at a reduced metabolic cost, however a huddle can only self-organise if pups in turn contribute heat. Contributing too much heat is costly but contributing too little compromises the ability of the huddle to self-organise. To investigate how the resulting tension between co-operation and competition in the huddle might affect natural selection, litters of simulated rodents were subjected to a simple evolutionary process. After interacting with its littermates, the individual that incurred the greatest metabolic cost for thermoregulation was iteratively replaced by another with random thermal properties. Simulations resulted in the emergence of a phenomenon called  In cold environments, huddling together is a ‘good trick’ that is exploited by many endothermic species in order to keep warm. Huddling in many species of mammals and birds has been described as a self-organising system –4, whereConverging evidence suggests that thermogenesis plays a central role in balancing competition and cooperation in rodent litters. Non-shivering thermogenesis via brown adipose fat tissue (BAT) is functional in juvenile rats, and in the cold they huddle together. Rats seek cold when BAT is pharmaceutically increased , and hudThe metabolic costs of BAT-thermogenesis to the individual may be offset by the thermoregulatory benefits of huddling to the group. In cold environments individuals must cooperate by contributing heat in order for the huddle to self-organise, but generating too much heat is energetically costly. The price for burning oxygen must ultimately be paid in the consumption of food, e.g., the milk of the dam, and pressure to thermoregulate thus puts individuals in competition for food when energy resources are limited. Understanding the thermodynamics of huddling may shed light on how biological systems are shaped by the tension between cooperation and competition, and thus by the interaction between self-organisation and selection in more general terms.Thermoregulatory huddling behaviours displayed by litters of juvenile rodents give rise to what has been described as a second-order critical phase transition, i.e., a continuous but abrupt decrease in the degree of aggregation as the temperature of the environment is increased . In the The classic system for investigating phase transitions is the Ising spin-glass model , 11, accRodent huddles display a group-level behaviour known as ‘pup flow’, whereby individuals continually cycle between the cool periphery of the huddle and its warm core . HuddlinA thermodynamic description of a complex adaptive animal group behaviour in which metabolic cost can be defined explicitly, could open the door for a thermodynamic theory of the interaction between self-organisation and selection. To this end, in the following section we derive a description of self-organising thermoregulatory huddling in thermodynamic terms. Thermal physiologies in the model are then evolved under pressure to reduce metabolic cost, and the model is shown to display a phenomenon called self-organised criticality . The emeThe approach here is to construct the simplest possible description of the huddle as a self-organising thermoregulatory system. The main criteria for a successful description of self-organising huddling are i) that across a range of intermediate ambient temperatures, the average body temperature, computed over all individuals, should remain approximately constant at a preferred body temperature of 37°C, ii) that a metric of huddling should display a second-order critical phase transition as the ambient temperature is manipulated, and iii) that an appropriate metric of ‘pup flow’ should display a peak at the critical ambient temperature of the phase transition. The simplest model that can generate a second-order phase transition is known to be the Ising spin-glass model, and the dynamics of an Ising model can be simulated using a Monte Carlo algorithm . Our aiThe Monte Carlo algorithm presented here uses the standard method of Gibbs sampling, which is described next in terms of huddling interactions. The description of huddling is based only on a description of the litter in terms of the sizes of its component groups p1, isT is referred to as the ‘temperature parameter’. Outcome 1 is more likely to occur when the temperature parameter is higher. Gibbs sampling involves generating a random number from a uniform distribution r ∈ , and on each iteration selecting outcome 1 if r < p1 and selecting outcome 2 otherwise. The probability of outcome 2 is therefore p2 = 1 − p1.Central to the Gibbs sampling method is to determine the probability of making a decision between two outcomes. Consider the situation where two independent groups of aggregated individuals collide. As a result of a contact between two individuals (from different groups), two outcomes are possible. Outcome 1 is that the individuals may remain in contact, and thus the two groups to which they belong will have merged together into one larger group. Outcome 2 is that an individual may become displaced from its group to form a new group of size 1. Note that all other possibilities that might be observed, for example the possibility of both individuals becoming simultaneously displaced from their groups, of both individuals returning to their original pre-contact groups, or of one individual leaving its group and joining the other, are all possible as consequences of further iterations of the decision between outcomes 1 and 2. Hence, our algorithm is based on iteratively deciding between outcomes 1 and 2, and updating the sizes of the groups after each iteration accordingly. On each iteration, the probability of outcome 1, N individuals are assigned to a group. The number of individuals in a particular group is n. For individuals indexed by i ∈ , the number of individuals in the group to which individual i belongs is defined to be ni. On each iteration, two individuals, labelled a and b, are selected at random from two different groups. Accordingly, the probability that individual i is selected to be individual a is pa = ni/N, and for the remaining N − 1 individuals the probability of being selected as individual b is therefore pb = ni/(N − na). Note that individuals from larger groups are more likely to be selected.The algorithm of the huddling model proceeds as follows. Each of r < p1, then the groups to which a and b belong are joined together to create a single group of size na + nb (outcome 1), otherwise a is removed from its group to form a new group of size 1 (outcome 2).If T is defined in terms of the body temperature, TB, of individuals a and b,TP = 37°C is the preferred (or target) body temperature, and the factor 2 accounts for the combination of temperatures measured from two individuals. According to The temperature parameter TA is the ambient temperature, G is the rate of heat production by thermogenesis, A ∈  is the proportion of the surface area that is exposed to the ambient temperature, and k1 is the area-specific wet thermal conductance. Note that the area-specific wet thermal conductance is dependent on many factors intrinsic and extrinsic to the body, including emittance, thermal conductivity, thickness of the fur, speed of the wind, viscosity and density of the air, and the geometry of the body . For k1 was chosen to be k1 = 0.3, which effectively sets the slope of the phase transition, and was found by experiment to provide a qualitative match to similar results derived from a related model . The validity of this evolutionary procedure is considered in full in Discussion. However, note that this algorithm effectively poses the following question of the system; what configuration of metabolic rates in the group is optimal when selection pressure on individual physiological thermoregulation (pressure to reduce G) is balanced with pressure on behavioural thermoregulation (pressure to reduce |TB − TP|)?On each of 10,000 generations of the evolutionary algorithm, the thermodynamic huddling model was simulated through second-weakest individual. The emergence and evolution of a representative avalanche is shown in Evolution of the distribution of thermal physiologies in the group resulted in extended periods of stasis, sometimes lasting for hundreds of generations, in which the metabolic rates of all in the group remained at constant low values. However, these periods of stasis were interrupted by prolonged bursts in which the fitnesses of several in the group suddenly reached significantly lower values before returning to stasis. In deference to descriptions of self-organised criticality in models of sandpile formation , these sG. The individual with the highest metabolic rate is likely to consistently be the least fit, and hence during periods of stasis the metabolic rate of the least fit varies randomly because it is randomly substituted. However in the generation at which an avalanche is triggered, the random metabolic rate of the individual that has been substituted into the group was observed to be higher than that of the individual subsequently evaluated to be the least fit. This means that the new individual is fitter not because it has a lower metabolic rate, but because it benefits more than the least fit from compensatory savings in thermoregulation. The high metabolic rate G of the new individual is compensated for a small |TB − TP|, which must occur due to effective huddling interactions with the rest of the group.During the period of stasis prior to an avalanche, the metabolic rates of the group are maintained at low values because the fitness function penalises high values of At the onset of an avalanche, the individual that will be replaced in the next generation is therefore not the individual that is contributing the most heat to the group by thermogenesis, hence the total heat available to the group in the next generation increases. In general, the fitness function penalises high metabolic rates and thus the heat from individual thermogenesis tends to reduce as the system evolves, but at the onset of an avalanche the total heat is instead transiently increased. Thus the evolving configuration of the group becomes critically stable, such that the total heat in the group gradually reduces until the overall reduction of heat triggers the next avalanche.F, at each generation for all individuals except for the least fit, which as we have seen in TA = 10°C, hence we expect pressure to reduce G towards 0 to maintain fitnesses at around |TA − TP| = −27 (by assuming TB = TA). Values of F that exceed this baseline indicate where huddling interactions have increased the body temperature and values below the baseline indicate where additional costs of sustaining a high metabolic rate have been incurred.To reveal how the critically stable dynamics revealed by inspection of a single avalanche unfold over the full evolutionary history, second-least fit individual at each generation, which is referred to as Gmin, and the close temporal correspondence between Gmin, by the mechanism explained in relation to H(Gmin), reveals a fall-off in frequency indicative of a power-law distribution. A fitted curve obtained by linear regression of the histogram after a log-log transformation, i.e., a fit to the model log H(Gmin) = m log Gmin + c, is shown in A histogram of the distribution of metabolic rates in the second least fit, TA, hence obtaining a power law fit to the distribution of metabolic rates from simulations conducted through a range of ambient temperatures would be stronger evidence of self-organised criticality. Twenty-four evolutionary simulations were run, each at a constant ambient temperature Ta ∈ . For all simulations at TA < TP, where huddling is expected to improve fitness, the distribution conformed well to a power-law model. The exponent m of the power-law fit obtained at each ambient temperature is shown in m remains approximately constant over a wide range of lower ambient temperatures, and the steepness of the curve increases as the ambient temperature approaches 37°C.A power-law relationship is a strong indicator of self-organised criticality, . A secoThe emergence of self-organised criticality in the thermodynamic huddling model reveals how the tension between cooperative and competitive interactions in animal groups may constrain evolutionary dynamics. Next we ask whether the effects are a specific feature of the abstract thermodynamic description of huddling developed here, or whether self-organised criticality might also emerge in the evolution of a physically realisable model of self-organising huddling behaviours, i.e., one that describes also the movements of the individuals.x is the 2D position of the individual, v1 is its velocity, and θ(t) is its direction of travel at time t. The change in the direction of travel is a function of its interactions with proximal particles.The thermodynamic model of huddling was developed on the basis of an analogy between interacting rodents and the magnetic spins of an Ising model. In an Ising model, spins are fixed in position on a lattice. However, the characteristic thermodynamics of Ising models are preserved in what is known as a Vicsek model . A Vicsedθ/dt is a function of the difference between the body temperature and the preferred temperature.A recent agent-based model of rodent thermoregulatory huddling describes the huddle as a self-organising system, generating a second-order phase transition in huddling and a peak in pup flow at the critical temperature . InteresAccording to the model of , rat pupTLi and TRi, are computed by averaging over many thermal sensors around its circumference, each registering either the ambient temperature or the body temperature of another pup where a contact occurs. The resulting temperatures are mapped to two ‘sensor’ values that incorporate the difference between the preferred temperature and the body temperature, sLi = (1 + exp(−σ(TP − TBi)TLi))−1 and sRi = (1 + exp(−σ(TP − TBi)TRi))−1, which are combined so as to turn the body in the direction that will reduce this difference,v2 sets the turning rate.According to the agent-based huddling model, pups turn using a strategy of ‘homeothermotaxis’, defined as follows. The temperature on the left and right of the body, i changes according toTCi is the average temperature of the thermal sensors of pup i that are in contact with other pups (who are thus reducing its exposed area). The parameter k2 is another thermal conductance constant, which determines the rate at which heat is exchanged between pups that are in contact. The three terms on the right of A = 1, i.e., when an individual is isolated and therefore fully exposed, the solution of The body temperature of pup Given the relationship between the Ising and Vicsek models of particle interactions, and the assumption that our model of rodent huddling is correspondingly related to the model of , we ask G, k1, and k2 to values in the range investigated originally by .The results presented in nally by . Howevert = 1000 timesteps, and the fitness of each pup was again computed using G ∈  and a random thermal conductivity determined by k2 ∈ . Results are shown in TA = 10°C and α = 3.0.The dynamics of the agent-based huddling model were simulated for Evolution of the full agent-based huddling model again generated avalanches in fitness , whose ok2, the group are collectively able to generate higher body temperatures during huddling interactions. As a result, when substitution of a random thermal physiology into the group triggers an avalanche, the fitnesses of those who benefit from huddling with it can be seen to spread out above the baseline fitness in TA < TP, and that the exponent of the fitted curve is stable across a wide range of lower ambient temperatures.The dynamic exchange of heat via huddling interactions manifests as greater variability in the distribution of heat amongst the group in the full agent-based model compared with the thermodynamic model. Moreover, as rapid exchange of heat can occur between individuals with a high conductivity A thermodynamic description of rodent huddling behaviours was developed here, according to which the thermoregulatory properties of the huddle can be understood in terms of a second-order phase transition, as it is defined and studied in statistical physics. In particular, the model equates the phenomenon of pup flow observed in rodent huddles with fluctuations in the heat capacity of a particle system, and as such predicts that a peak in pup flow should occur at the critical temperature of the phase transition between huddling at low temperatures and non-huddling at higher temperatures. If this prediction cannot be confirmed in future experiments, then the thermodynamic model developed here is wrong, and the onset of self-organising huddling behaviours at low ambient temperatures is not evidence of a true second-order phase transition.To investigate how tensions between co-operation and competition within animal groups may influence natural selection, the thermodynamic huddling model was subjected to a simple evolutionary procedure. Simulations revealed that individual thermal physiologies interacting via huddling may together be maintained in a regime of self-organised criticality. In simulation, optimisation of metabolic rates to exploit the metabolic savings of huddling leads to an ultimately unstable set of thermal couplings within the group. Evolved over a wide range of low environment temperatures, these instabilities manifest as a distribution of low metabolic rates that remain stable for long evolutionary periods, but which are periodically interrupted by abrupt decreases in fitness that affect all group members. When huddling interactions were instead represented with a full agent-based simulation of huddling interactions, essentially the same evolutionary dynamics were found to emerge, suggesting that criticality is a robust feature of the evolutionary algorithm, when either model of huddling is used to introduce long-range correlations in fittness between individuals.The present results indicate that under pressure to reduce individual metabolic costs, exploiting interactions with the group causes individuals to become critically dependent upon the group. We often think of evolution as a gradual ascent of a genome towards peak fitness, but the present simulations show how long-range correlations in phenotypic fitness due to interactions within animal groups can carve deep cliffs into a fitness landscape .It is important to acknowledge that the evolutionary procedure implemented here was not expressed in terms of a genetic algorithm, and as such it is abstract with respect to a full population-based account of the exchange of genetic information between generations. By simply replacing the weakest thermal physiology for a random physiology, many generations of agent-based huddling interactions could be simulated efficiently. More importantly, this abstraction expresses in its simplest form how the tension between co-operation and competition in the huddle might impact on natural selection. Substituting one random thermal physiology into the huddle at a time enables the emergence of criticality to be attributed here directly to huddling interactions, rather than to additional correlations introduced if deriving physiologies genetically, i.e., by recombination. This procedure also allows the emergence of criticality in the model to be understood in terms of explanations for criticality in similarly formulated models of complexity . With tFor endotherms, the benefits of huddling are afforded only by groups in which individuals contribute heat, i.e., by thermogenesis. For example juvenile hamsters, who develop non-shivering thermogenesis late relative to rats and do not huddle , will acEfficient thermoregulation is important for growth and ultimately for survival. For example, growth rates in developing mice vary markedly with manipulations of the environment temperature , and difHeavier pups outcompete lighter littermates in the ‘scramble’ for the milk of the mother , 41. Undwithin a co-operating group for shared energy resources, i.e., food, against which metabolic efficiency is ultimately measured. Evolutionary dynamics are doubtless strongly governed by absolute fitness; if an animal is too cold it will die before it reproduces. However, within a group, the animal needing the most energy to stay warm relative to the others will lose out in the competition for shared energy resources. The critical challenge within a litter may therefore be simply to ‘out-thermoregulate’ the runt of the litter. For rats and mice, a direct selection pressure acting on the runt of the litter may derive from the tendency of the dam to eat the runt, particularly in cold environments  similarly captures the competition s . Cohabis . Auto-tuning of the critical temperature at which dynamic group interactions persist would constitute an ‘evolution towards the edge of chaos’, of the kind pursued by the theoretical biologist Stuart Kauffman through his study of random boolean networks . FurtherS1 CodeContains i) a standalone python script for recreating (ZIP)Click here for additional data file.


There is extensive health and public health literature on the ‘evidence-policy gap’, exploring the frustrating experiences of scientists trying to secure a response to the problems and solutions they raise and identifying the need for better evidence to reduce policymaker uncertainty. We offer a new perspective by using policy theory to propose research with greater impact, identifying the need to use persuasion to reduce ambiguity, and to adapt to multi-level policymaking systems.We identify insights from secondary data, namely systematic reviews, critical analysis and policy theories relevant to evidence-based policymaking. The studies are drawn primarily from countries such as the United States, United Kingdom, Canada, Australia and New Zealand. We combine empirical and normative elements to identify the ways in which scientists can, do and could influence policy.We identify two important dilemmas, for scientists and researchers, that arise from our initial advice. First, effective actors combine evidence with manipulative emotional appeals to influence the policy agenda – should scientists do the same, or would the reputational costs outweigh the policy benefits? Second, when adapting to multi-level policymaking, should scientists prioritise ‘evidence-based’ policymaking above other factors? The latter includes governance principles such the ‘co-production’ of policy between local public bodies, interest groups and service users. This process may be based primarily on values and involve actors with no commitment to a hierarchy of evidence.We conclude that successful engagement in ‘evidence-based policymaking’ requires pragmatism, combining scientific evidence with governance principles, and persuasion to translate complex evidence into simple stories. To maximise the use of scientific evidence in health and public health policy, researchers should recognise the tendency of policymakers to base judgements on their beliefs, and shortcuts based on their emotions and familiarity with information; learn ‘where the action is’, and be prepared to engage in long-term strategies to be able to influence policy; and, in both cases, decide how far you are willing to go to persuade policymakers to act and secure a hierarchy of evidence underpinning policy. These are value-driven and political, not just ‘evidence-based’, choices. The health sciences are well represented in studies of the ‘evidence-policy gap’, in which academics describe their attempts to overcome ‘barriers’ between the production of evidence by scientists and its use by policymakers. Oliver et al.’s , 2 systeThere are two key problems with this literature. First, very few studies draw on policy theory or other forms of knowledge of the policy process . Most arThe most-frequently stated solutions to ‘barriers’ highlight the limits to such a theoretical analysis. For example, to address the problem of supply, studies highlight the need for ‘improved dissemination’ to ensure that policymakers pay attention to, and understand, the best evidence . Too fewTo address demand, the most-proposed solution is to develop scientific competence in government. Too many studies assume that it is realistic to produce a captive audience of policymakers willing to invest the time to prioritise and understand scientific evidence. This approach is at odds with the less rigid ways in which policymakers use many forms of evidence , 6–9.Second, too few studies recognise the role of values in politics. Instead, an often implicit and untested assumption is that policymaking should be as ‘evidence based’ as medicine, which is at odds with the more common starting point, in the study of politics, to produce a democratic system which translates competing societal values and preferences into policy solutions. One may desire a political system based on value judgements and evidence, but should recognise and address the trade-offs between these aims, and that the production of evidence is also an inherently value-driven process.How far should scientists go to persuade policymakers to act on their evidence?Policy studies suggest that actors are influential when they ‘frame’ their evidence in simple, manipulative and/or emotional terms to generate policymaker attention. Should scientists also combine evidence with emotional appeals to influence the policy agenda? Should they deny scientific uncertainty and make unequivocal claims about the implications of their evidence? Or, does this strategy produce an ethical dilemma and/or the potential to reduce long-term scientific credibility?How far should scientists go to defend a hierarchy of evidence to deliver policy solutions?Initially, policy studies prompt us to identify where ‘the action’ is, to know where to engage in a complex and multi-level system in which many actors make policy in many venues. However, it does not identify the main aim of engagement. Is it to make sure that all actors prioritise ‘evidence-based’ policymaking above other factors, or to cooperate with other actors with different ideas about ‘good’ policymaking? The latter can include governance principles such as ‘localism’ and the ‘co-production’ of policy between local public bodies, interest groups and service users. This process may be based primarily on values and involve actors with no commitment to a hierarchy of evidence when they implement policy solutions.We address these gaps in the literature by raising two questions designed to promote discussion about pragmatic policy strategies. Each question identifies a central insight from policy studies and prompts scientists to consider how they would combine an attachment to evidence-based policymaking with other values to produce a realistic way forward.Consequently, insights from policy studies provide practical advice and raise new dilemmas  being manipulative and hyperbolic, to generate attention in the short term, and (2) maintaining an image of objectivity, to generate strong relationships in government based on reliability, expertise, trust and ‘independence’. Further, there are many compromise solutions, including a strategy built on framing the implications of evidence according to the stated beliefs of policymakers and building up a reputation for reliability within many policymaking venues.In health science, many models of research impact are built on strategies which make minimal reference to policymaking, namely identifying the research question, developing a research methodology, implementing data collection, analysis and synthesis, interpreting findings, and developing research and then policy/practice recommendations. In this patrician model, the process is owned and controlled by researchers, who then advise or disseminate their work in the general direction of policymakers .There is also the advocacy model, where researchers develop policy solutions and attempt to convince policymakers to adopt them , 32. MosA more collaborative kind of advocacy is ‘facilitational’, in which researchers attempt to share power, responsibility and accountability for research with stakeholders. ‘Co-production’ was originally used to describe a model of public service delivery which rested on the belief that using a broad range of perspectives lead to better, fairer, more useful and more used products and services –38. ThesThere are many models for collaborative interactions between scientists and policymakers e.g. –54). The. The54])Little research has been performed on the strengths, risks and implications for research, researchers and policy of these models of interactions , 43. ProWhat most of these models have in common is that they are modifications of the research process, not responses to the policy process. The disinclination of many scientists to engage with the policy world, even amongst the most collaboratively minded, is revealing. It highlights an unresolved issue about the extent to which a systematic research-driven model can be adapted and improved to reflect the need to compete with advocates of other forms of evidence to secure the attention and support of policymakers.In summary, scientists have developed different answers to the question ‘how far should you go?’ These answers correspond to ways in which scientists and policy actors interact, from disseminator/receiver to equal co-producers. However, these interactions have not yet been examined through the lens of policy theory, nor have empirical comparative descriptions enabled us to determine what types of interactions work best, for which purposes and under which circumstances.Any competition to frame issues takes place in a complex policymaking environment or system . This coIn its place are theories which identify a policy ‘environment’ which relates to the interaction between five factors, namely the many policy actors operating in multi-level systems, pursuing ideas (using their beliefs and knowledge to influence how policymakers think), adapting to institutions , networks , and the impact of socioeconomic context and events , 67–71. This analytical shift from cycles to environments and systems helps us reject the notion that there is a core group of policymakers at the heart of the process, analysing and making policy from the ‘top down’, and that scientists promoting evidence can focus their efforts at a single point of problem definition or policy formulation .Rather, scientists compete with many influential actors to present evidence to secure a policymaker audience at many levels of government. Support for evidence-based solutions varies according to which organisation takes the lead and how its rules encourage policymakers to understand the problem. Some networks are close-knit and difficult to access because bureaucracies have operating procedures that favour some sources of evidence and participants over others. There is a language in institutions and networks that takes time to learn. It indicates which ways of thinking are in good ‘currency’, including a dominant way to ‘frame’ a problem and limit discussion to certain solutions. Well-established beliefs provide the context for policymaking – new evidence on the effectiveness of a policy solution has to be accompanied by a shift of attention and successful persuasion. Social or economic ‘crises’ can prompt lurches of attention from one issue to another, and some evidence can be used to encourage that shift. However, most policy change is minor; it is rare for events to combine with new evidence to produce quick and radical policy change .From such discussions, we can provide a second key message for scientific advocates – learn ‘where the action is’, and be prepared to engage in a long-term strategy to be in a position to influence policy. In other words, identify ‘the action’ at several levels of government, learn the ‘rules of the game’ in institutions and networks (or the ‘venues’ in which policymaking takes place), form coalitions with like-minded actors, and work with the ‘policy entrepreneurs’ possessing the skills to exploit ‘windows of opportunity’ to give policymakers the motive and opportunity to adopt new solutions , 73–78.In most cases, these coalitions will be with many actors with different ways of thinking about policymaking, engaging in politics to turn their beliefs and values into policy, often unaware or unsupportive of EBM’s hierarchy of evidence, and likelier to refer to governance principles other than evidence-based policymaking.Consequently, this second message produces a second practical and ethical dilemma – how far should you go to defend a hierarchy of evidence to deliver policy solutions? It involves the extent to which policy actors should accept and adapt to policymaking complexity and the diffusion of power, or seek to change some aspects and assert their claim for greater power.For example, central governments have some incentives to try to reassert central control because they are held accountable in elections and parliaments for policy outcomes, regardless of their responsibility . Yet, thIn that context of multi-level and delegated policymaking, scientists may support phrases such as ‘co-production of research’ superficially , but also recognise the need for tough choices to ensure the application of evidence-based policy solutions. They need to engage with two debates simultaneously, on EBM and localism.Policymakers may not care that evidential hierarchies exist, preferring a variety of sources of knowledge , 6, 84. Some advocates of RCTs recommend greater reflection on scientific values in politics rather than the assumption that hierarchies of evidence are necessarily valid , 89.EBM contrasts with an adherence to practice-based evidence, which includes the experiential knowledge of practitioners, feedback from service users and a greater focus on supporting different voices , 44)..44]).Some scholars argue that complex policymaking systems defy control and, therefore, are not conducive to the use of RCTs (see ). InsteaFirst, not all actors adhere to the EBM ideal –83. TherSecond, central governments face the trade-off between national and local policymaking . Their nConsequently, there is an inextricable link between debates on the selection of evidence-based solutions/interventions and the correct choice of governance arrangements. This involves a two-part decision in which policymakers combine (1) their chosen method to identify and gather knowledge of successful policy solutions, with (2) their judgement on how to work with other actors to ‘scale up’ success.In this context, consider the scale of the task to ensure the primacy of RCTs and/or a hierarchy of evidence in policymaking. It involves (1) persuading many actors in many central and local venues that (2) there is a well-established best way to produce scientific evidence, and therefore (3) that scientific forms of knowledge should be prioritised in policymaking. This is a two-pronged dilemma – should you devote necessarily high resources to this line of persuasion, and should you go to great lengths to dismiss competing understandings of good policymaking?To demonstrate the issues central to dilemma 2, we outline a spectrum of approaches to answer the ‘how do you scale up success?’ question , with less focus on the delivery system (e.g. the gelatine capsule). However, in health and social care it refers to the interaction between practitioners and service users – the ‘dosage’ of an intervention by a practitioner is harder to distinguish from the ‘delivery system’.Interventions shown in well-designed and implemented randomised controlled trials, preferably conducted in typical community settings, to produce sizable, sustained benefits to participants and/or society”. Trials have been conducted since 1977, producing 18 peer-reviewed articles in elite journals , with at least two non-United States studies [setting out core model elements covering clinical delivery, staff competencies and organisational standards to ensure it is delivered well” . It began in the United States as the Nurse–Family Partnership, designed to engage nurses with first time mothers (deemed to be at relatively high risk of poor life chances) once per month from pregnancy until the child is two. The United States’ Coalition for Evidence-Based Policy  gave it  studies . It was  studies . The FNPd well” . nefits”  aGather evidence of effectiveness with reference to practitioner and service user testimony.‘Scale-up’ best practice by telling stories based on your experience and inviting other people to learn from them.Prioritise key principles such as respect for service user experiences and a conducive environment in which to deliver public services respectfully.A storytelling approach suggests that you:to promote quality of life for those living, dying, visiting and working in care homes for older people through relationship-centred and evidence-based practice” (http://myhomelife.uws.ac.uk/scotland/). With this approach, much derives from individual feedback, with a focus on the richness of experience. The result is a set of principles to inform future practice, not a specific intervention with a correct dosage. The principles are deliberately broad to allow practitioners and service users to make sense of them in specific settings [My Home Life is a key example. It began as an English initiative, also developed in Scotland, “settings , 101. ThThe comparison prompts us to recognise that there are no easy compromises between approaches to policy delivery that have fundamental differences. For example, the distinction between dosage and delivery system allows us to focus on the dilemmas in multi-level policymaking that cannot be solved with reference to evidence alone, because they involve prioritising intervention effectiveness versus governance principles based on practitioner and service user experience and values. There is a clear trade-off in practice between (a) a rigid attachment to EBM during policy delivery, and (b) the governance principles that many governments favour when promoting a storytelling approach .In that context, we should ask how far health scientists should go to ensure the correct delivery of an intervention. Should they be uncompromising, or seek ways to promote evidence-based policy and practices such as co-production in evidence gathering and delivery? At one end of the spectrum, in focusing on the evidence hierarchy, researchers assert their authority to describe reality , claiminGather evidence of effectiveness based on a mix of evidence, from RCTs to experiential data.‘Scale-up’ best practice through local experimentation and learning.Prioritise the use of a specific improvement method, training practitioners before they experiment and decide how best to turn evidence into practice.In reality, few such researchers exist. Co-produced RCTs, amongst other designs, are becoming the norm , 103. Yelies fallow and unused in daily work” ([The Early Years Collaborative (EYC) (Scotland) is a key example. ‘Collaborative’ refers to a group of organisations engaging on a problem, drawing on the ‘sound science’ on how to reduce costs or improve outcomes, which exists but “y work” . y work”  method for single organisations to coordinate a multi-agency project, working with local and health authorities through ‘community planning partnerships’. The general assumption is that, in some areas, there is no set of known, effective interventions. Rather, an important strand of this approach is learning as you go, with a long-term aim to gather comparable data on local practices to aid learning, supplemented by ‘word of mouth’ measures of success and ad hoc decisions to expand projects they feel are successful. This is justified with reference to the poor alternatives, such as the excessive gaps between the RCT evidence on a problem and resultant practice . The staOther improvement science-inspired approaches have become very common in health sciences, especially amongst projects such as the Collaborations for Leadership in Health Research and Care (CLAHRC) , and the political choice to roll it out, based on the assumption that national governments are best placed to spread that success throughout the country.We have largely discussed extremes or ideal types to highlight key issues. In practice, people may be willing to compromise or produce pragmatic responses to the need to adapt research methods to real world situations. In that context, this kind of discussion should help clarify why scientists may need to work with policymakers or practitioners to produce a solution that each actor can support and why that solution may not be ‘evidence based’ in the way that scientists understand that phrase.


The success of the clinical use of sequencing based tests (from single gene to genomes) depends on the accuracy and consistency of variant interpretation. Aiming to improve the interpretation process through practice guidelines, the American College of Medical Genetics and Genomics (ACMG) and the Association for Molecular Pathology (AMP) have published standards and guidelines for the interpretation of sequence variants. However, manual application of the guidelines is tedious and prone to human error. Web-based tools and software systems may not only address this problem but also document reasoning and supporting evidence, thus enabling transparency of evidence-based reasoning and resolution of discordant interpretations.http://calculator.clinicalgenome.org.In this report, we describe the design, implementation, and initial testing of the Clinical Genome Resource (ClinGen) Pathogenicity Calculator, a configurable system and web service for the assessment of pathogenicity of Mendelian germline sequence variants. The system allows users to enter the applicable ACMG/AMP-style evidence tags for a specific allele with links to supporting data for each tag and generate guideline-based pathogenicity assessment for the allele. Through automation and comprehensive documentation of evidence codes, the system facilitates more accurate application of the ACMG/AMP guidelines, improves standardization in variant classification, and facilitates collaborative resolution of discordances. The rules of reasoning are configurable with gene-specific or disease-specific guideline variations . The software is modular, equipped with robust application program interfaces (APIs), and available under a free open source license and as a cloud-hosted web service, thus facilitating both stand-alone use and integration with existing variant curation and interpretation systems. The Pathogenicity Calculator is accessible at By enabling evidence-based reasoning about the pathogenicity of genetic variants and by documenting supporting evidence, the Calculator contributes toward the creation of a knowledge commons and more accurate interpretation of sequence variants in research and clinical care.The online version of this article (doi:10.1186/s13073-016-0391-z) contains supplementary material, which is available to authorized users. Successful clinical diagnostic application of exome and genome sequencing hinges on the ability of clinical laboratories to consistently and accurately assess pathogenicity of genetic variants. Several studies have identified inconsistencies in variant classification across laboratories, reference databases, and even within individual databases –4. InterTo improve the standardization of variant interpretation by clinical testing laboratories, the American College of Medical Genetics (ACMG) and the Association of Molecular Pathologists (AMP) have recently updated guidelines for the interpretation of sequence variants . The newThe formal structuring of evidence-based arguments for or against pathogenicity and their implementation in software provide several opportunities for enhancing the emerging knowledge of genetic variation. One such opportunity is efficient evidence tracking that facilitates critical evaluation of variant classifications. Recent studies , 4 suggeThe Calculator client-side, including the main interface, is implemented using JavaScript and ExtJS. The main Calculator interface Fig.  presentshttp://reg.clinicalgenome.org (manuscript in preparation).As illustrated in Additional file The registry enables linking and viewing of pathogenicity assessments that are independently generated by different users  for each evidence tag within the evidence grid  and aggregated evidence for the variant being pathogenic or benign (encoded as a set of evidence tags) and produces a conclusion Fig. . The reawww.redmine.org) application.GenboreeKB is implemented in Ruby as an enhancement to the Genboree server. Thus, the group-based data sharing is managed by the Genboree access control system. A web-based GenboreeKB management interface employing the ExtJS Javascript framework is available as a plugin of the Redmine Rails . GenboreeKB stores both data models and the data itself as document collections and exposes them via the APIs.Numerous tools and systems for variant curation and evaluation have been developed and used by various laboratories. Recognizing that some of those tools may benefit from the functionalities of the Calculator, all the data and functionality of the Calculator are programmatically accessible via REST APIs. The Calculator may be used as a web service (with authentication and collaborative features such as group-based access control) or by installing it on local hardware.Two adaptations of the original guidelines had to be introduced to facilitate formal algorithmic reasoning. First, several metarules were introduced to capture implicit “common sense” rules practitioners were expected to apply without explicit instruction. For example, if there are two strong and three moderate-strength evidence tags in favor of pathogenicity, then the variant can be pathogenic (≥2 strong evidence tag) or likely pathogenic (≥3 moderate-strength evidence tags). In such a case the metarules specify that the variant is pathogenic. Conversely, evidence tags that can be combined to yield both Likely Benign and Benign are scored as Benign. The Calculator distinguishes between uncertain significance that arises due to insufficient evidence from uncertain significance due to conflicting evidence.Second, the guidelines allow the practitioners to change the strength level originally assigned to certain tags based on perceived strength of supporting data for the tag . To accThe ACMG/AMP guidelines and the Calculator were tested in the context of a multi-site study  involvinGLA) and condition name and mode of inheritance . A new assessment is initiated by providing an HGVS allele identifier  can access the functionalities of the Calculator via Genboree REST-APIs. Using the services, users can upload evidence tags for several variants at a time. The uploaded data become accessible through the interface as well as through REST-APIs. In the future, the bulk upload functionality will be made accessible even for regular users without any background in computer programming.Sharing of pathogenicity assessments of pathogenicity of variants by clinical laboratories via databases such as ClinVar aids in the practice of clinical genome interpretation. However, currently, the format of information by which pathogenicity for a given variant was determined by multiple submitters to ClinVar often appears in a free text unstructured format, if it is provided at all. It would be useful if this knowledge sharing was further extended to include the reasoning and evidence codes that supported conclusions about pathogenicity in a standardized format. For example, evidential support for an early assessment that a variant is pathogenic may not have addressed more recent information about the high frequency of the variant in a specific population, thus explaining discordance with a newer independent assessment that includes this recent information. With the evidential support available for inspection, a user of the database may benefit from being able to rapidly and critically evaluate both assessments based on the evidence provided in a structured way. Toward this goal, the Calculator facilitates the sharing of the reasoning and the evidential support behind every assessment. Access to this information is provided via the interfaces Fig. , Panel 3Recent studies indicate that the initial application of the ACMG/AMP guidelines did not increase concordance in laboratory assessment ; howeverThe Calculator does not share information about pathogenicity of variant publicly without explicit user request. The information entered is protected using a login and password. It provides optional free-text fields summarizing tag assignments and links to supporting evidence. The Calculator does not require or have fields for providing identifying information about patients. Public sharing per user request will be enabled in future releases.Although ACMG/AMP guidelines provide a general framework for Mendelian sequence variant interpretation, it was always recognized that this framework would evolve and be customized to meet requirements for specific genes and disease conditions , 16–19. While each assessment refers to a specific guideline, different guidelines may exist within the same system. For example, for the purpose of assessing variants within genes causing inborn errors of metabolism, metabolite levels could be used as an evidence type in pathogenicity assessments. Because the original ACMG/AMP guidelines do not specify this type of evidence or potentially others that will be added in the future, we have designed flexibility into the Calculator to allow for addition of new evidence types (new rows in the evidence grid) and corresponding evidence tags and of new rules without programming.By automating the formal guideline-based assessment of genetic variants, the ClinGen Pathogenicity Calculator can reduce human error. The Calculator extends functionality of other currently available calculators  along seBy providing an intuitive user interface and by addressing other key computational requirements, the Calculator facilitates formal and collaborative evidence-based reasoning about the pathogenicity of genetic variants, thus paving the way toward more effective and accurate clinical genomic interpretation.


P-values cannot reflect but Bayes factors can. It is argued that appropriate conclusions match the Bayesian inferences, but not those based on significance testing, where they disagree.Inference using significance testing and Bayes factors is compared and contrasted in five case studies based on real research. The first study illustrates that the methods will often agree, both in motivating researchers to conclude that H1 is supported better than H0, and the other way round, that H0 is better supported than H1. The next four, however, show that the methods will also often disagree. In these cases, the aim of the paper will be to motivate the sensible evidential conclusion, and then see which approach matches those intuitions. Specifically, it is shown that a high-powered non-significant result is consistent with no evidence for H0 over H1 worth mentioning, which a Bayes factor can show, and, conversely, that a low-powered non-significant result is consistent with substantial evidence for H0 over H1, again indicated by Bayesian analyses. The fourth study illustrates that a high-powered significant result may not amount to any evidence for H1 over H0, matching the Bayesian conclusion. Finally, the fifth study illustrates that different theories can be evidentially supported to different degrees by the same data; a fact that  Social Psychology.This paper will present case studies from real research that illustrate how significance testing and Bayesian statistics can lead researchers to draw different conclusions. The question will be, which conclusions are most sensible? First, we will discuss the nature of hypothesis testing, then the anatomy of a Bayes factor, focusing on how one models the theory. Finally, the heart of the paper will be a set of five case studies taken from a recent special replication issue of the journal In using inferential statistics to test a theory of scientific interest, the world is typically first divided into H0 (the null hypothesis) and H1 , where one of those hypotheses is a consequence of the theory. Data are then collected in order to evaluate H0 and H1. In evaluating whether the theory survived the test, it would often be useful to say whether the data provided good enough evidence for H0; good enough evidence for H1; or else failed to discriminate the hypotheses. That is, one might like to make a three-way distinction, as indicated in Fig. P-values only allow one to distinguish evidence against H0 from the other two evidential states of affairs In significance testing, one models H0 and not H1. A typical model for H0 is, for example, the model that there is no population difference in means. Assuming in addition a model of the data , the probability of the data given H0 can be calculated. Unfortunately, modelling H0 but not H1 does not allow one to make a three-way distinction. How can one know by which hypothesis the data are better predicted, if one only knows how well the data are predicted by one of the hypotheses? Thus, significance testing only allows a weak form of inference; it tells us something but not all that we want. As shown in Fig. To remedy the problem, it might seem obvious that one needs a model of H1 Dienes, . The hypIn fact, to know how well the hypothesis predicts the data, one needs to know the probability of each effect size given the theory . Figure Generally, in science, predictions are made from a theory via auxiliary assumptions  in general over-estimate effect sizes  is available is illustrated in Fig. Having constructed an H1, for example by the method just described, there is a crucial final step: the judgment that the model is acceptable for the scientific problem Good . While aBH means the Bayes factor obtained when a Half-normal distribution (hence ‘H’) is used to model H1 with a mode of 0  and a standard deviation of S. .As a Bayes factor is relative to the model of H1, we will use a subscript to specify the model of H1 . Jeffreys  the evidence favours H1 over H0 (in the convention used in this paper). As the Bayes factor decreases below 1 (towards 0) the evidence favours H0 over H1. There are no sharp boundaries or necessary thresholds (unlike the fixed significance levels of the Neyman Pearson approach), just a continuous degree of evidence. Nonetheless, rough guidelines can be provided, in much the same way as Cohen  suggesteJeffreys  suggesteSocial Psychology  = 2.02, P = .053. Gibson, Losee, and Vitiello (t(81) = 2.40, P = .02. So there is a significant effect with a raw effect size almost identical to that in the original study. Correspondingly, BH = 4.50. That is, there is substantial evidence for H1 over H0 in the replication.When a significant result is obtained along with an effect size matching that expected in theory, there will be evidence for H1 over H0. Shih, Pittinsky, and Ambady  argued tVitiello  replicatN, it will often be evidence for H0. Williams and Bargh  = 0.04, indicating substantial evidence for the null hypothesis over the hypothesis defined by the effect obtained in the original study.Similarly, when a non-significant result is obtained with large nd Bargh ; study 2nd Bargh  attempteIn sum, we considered a case where a significant result corresponded with the convention for substantial evidence for H1 over H0; and a case where a non-significant result corresponded to the convention for substantial evidence for H0 over H1. Correspondingly, Jeffreys (t(72) = 2.70, P = .01. Brandt, IJzerman, and Blanken  greater than 0.9.Banerjee, Chatterjee, & Sinha , study 2t(119) = 0.17, P = 0.87. That is, it was a high-powered non-significant result. By the canons of classic hypothesis testing one should accept the null hypothesis. Yet Brandt et al. sensibly concluded “… we are hesitant to proclaim the effect a false positive based on our null findings, … Instead we think that scholars interested in how morality is grounded should be hesitant to incorporate the studies reported by BCS into their theories until the effect is further replicated. (p. 251)” Why is this conclusion sensible if the non-significant outcome was high powered? Because a study having high power does not necessitate it has much evidential weight, and researchers should be concerned with evidence  = 0.97.Of course, H1 does not really predict just one value. Using our standard representation of plausible effect sizes, a half-normal scaled by the original effect size , we get d = 0.25 effect is 24%. Given the low power, perhaps it is not surprising that the replication yielded a non-significant effect, t(99) = 1.15, P = 0.25. However, it would be wrong to conclude that the data were not evidential. The mean difference was 4% in the wrong direction according to the theory. When the data go in the wrong direction (by a sufficient amount relative to the standard error), they should carry some evidential weight against the theory. Testing the directional theory by modelling H1 as a half-normal with a standard deviation of 5%, BH = 0.31, substantial evidence for the null relative to the H1.Now we consider a converse case. Shih, Pittinsky, and Ambady  argued teffect, t9 = 1.15,Note that a sample difference going in the wrong direction is not necessarily good evidence against the theory Dienes, . If the Imagine two theories about earthquakes, theory A and theory B, being used to predict whether an earthquake will happen in downtown Tokyo on a certain week. Theory A predicts an earthquake only on Tuesday between 2 pm and 4 pm of a magnitude between 5 and 6. Theory B predicts earthquakes any time between Monday and Saturday of a magnitude anywhere between 1 (non-existent) to 7 (intense). Theory A makes a precise prediction; theory B is vague and allows just about anything. An earthquake of magnitude 5.1 in fact happens on Tuesday around 2:30 pm. These data are in the predicted range of both theories. Nonetheless, does this observation count as stronger evidence for one theory rather than the other? Would you rely on one of those theories for future predictions more than the other in the light of these data?It should be harder to obtain evidence for a vague theory than a precise theory, even when predictions are confirmed. That is, a theory should be punished for being vague. If a theory allows many outcomes, obtaining one of those outcomes should count for less than if the theory allows only some outcomes Popper, . Thus, aP < .05). Lynott et al.  = 1.56, indicating the data are insensitive in discriminating H1 from H0.The issue can be illustrated using Lynott et al.’s  replicatt et al.  obtainedP-value is indifferent to the inferentially relevant feature of a theory being vague. So call this model of H1 the vague model.How can a significant result not count in favour of a theory that predicted a difference? It depends on the theory being tested. The original finding was that 75% of people exposed to cold selected a personal treat (and only 46% exposed to warmth did so); if one could expect an effect size from very small to even larger than this, then a small effect size is not especially probable in itself.P = .049. Now OR = (552/452) = 1.49, and ln OR = 0.40. These data render a ln OR greater than about twice 0.40 as quite unlikely . The theory is more precise . Call the model of H1 based on these counterfactual results the precise model. Finding a replication ln OR of 0.28 (with a standard error of 0.14 as before), falls within the range of predictions of this rather precise theory, just as it fell within the range of predictions of the vague theory. Now BH = 3.81, support for the precise H1 over H0 (the B was 1.56 for the vague H1 over H0). Bayes factors are sensitive to how vague or precise the theory is; P-values are not. But, normatively, precise theories should be favoured over vague ones when data appear within the predicted range.Let us say in the original study, 55% of people exposed to cold chose the personal reward whereas 45% of people exposed to warmth did so, and this was significant Finally, notice that the replication study had less power to distinguish the ln OR of 0.40  from H0 than it had to distinguish the ln OR of 1.26  from H0. In this case, the high powered significant result was less good evidence for the theory than the low powered significant result. A high-powered significant result is not necessarily evidence for a theory. How strong the evidence is for a theory all depends on how well the theory predicted the data.Jeffreys , p vi wrt(41) = 2.57, P = .014. Johnson, Cheung, and Donnellan  = 0.63, P = 0.53. Thus, there is a high-powered non-significant result. But, as is now clear, that still leaves open the question of how much evidence there is, if any, for H0 rather than H1.The issue often comes up as a criticism of Bayes factors . The predictions of H1 could be represented as a uniform distribution from 0 to 6. That claim has the advantage of simplicity, as it can be posited without reference to data. These considerations give BH = 0.37.We also have our half-normal model for representing H1. The original raw effect size was 1.11 rating units; and, So we have one Bayes factor of 0.09 and another of 0.37. Both Bayes factors have a reasonable rationale. Yet they are sufficiently different that they may lead to different conclusions in a Each Bayes factor is an indication of the evidence for the H1 represented as opposed to H0. The H1s are different, and each Bayes factor appropriately answers a different question. Which Bayes factor answers the question we have been asking in this paper for each case study, namely, the extent to which the replication provided evidence for the regularity claimed by the first study? The first Bayes factor is not good at answering this question, because it is not informed by the first study. The second Bayes factor is informed (and is otherwise simply specified). Therefore, the second Bayes factor is the one that should be used to address this question, and thus guide the corresponding discussion and conclusions in the paper.t-distribution described in the P-value testing against H0 would be the same.The first Bayes factor in effect refers to a different theory, and thus a poses a different question of the data. That theory predicted all differences as equally plausible. It is a vague theory and thus was not supported as well as the more precise theory defined by the effect found in the original study. But theories, or models of data, need not differ just in being vague versus precise. Two models could be just as precise but predict different size effects. The half-normal model we have been using does not allow this ; but the A Bayes factor is a method for comparing two models. Thus there is not one Bayes factor that reflects what the data mean. In comparing H1 to H0, the answer depends on what H1 and H0 are. That’s not a problem, any more than in comparing two scientific theories, the answer depends on what the theories are. Further, the use of Bayes factors in no way precludes estimating parameters, or deriving credibility intervals, in order to understand the data. Both model comparison (hypothesis testing) and parameter estimation are important and complementary aspects of the scientific process  Because B = 3 is typically around the borderline of what is worth taking note of, analytic flexibility could push conclusions around when B = 3 is used as a threshold  as a criterion for sufficient evidence to draw a conclusion, we have done so merely because that roughly matches the standard of evidence the psychology community has been using up to now. However, our aim has been to advocate using a genuine measure of evidence, which is different from advocating a particular degree of evidence as sufficient. A conjecture is that the current standard of evidence has arisen for psychological reasons, namely it is a point where researchers typically judge that evidence is just enough to be worth taking notice of. . On the one hand, distinctions would be drawn that are not available to the P-aficionado, and more informed decisions taken. On the other hand, a significant P-value at the 5% level indicates there is some way of specifying H1 such that B > 3 (Royall, B for every P,” we, as a community, would learn to see the relationship between significance testing and Bayes factors—and, crucially, come to debate the optimal Bayesian ways of addressing different research questions.What is the way forward? We suggest a community learning process in which orthodox statistics are reported, but along with the orthodox statistics such as  Royall, , which m