Since the introduction of evidence-based medicine, there have been discussions about the epistemic primacy of randomised controlled trials (RCTs) for establishing causality in medicine and public health. A growing movement within philosophy of science calls instead for evidential pluralism: that we need more than one single method to investigate health outcomes. How should such evidential pluralism look in practice? How useful are the various methods available for causal inquiry? Further, how should different types of causal evidence be evaluated? This paper proposes a constructive answer and introduces a framework aimed at supporting scientists in developing appropriate methodological approaches for exploring causality. We start from the philosophical tradition that highlights intrinsic properties  as essential features of causality. This abstract idea has wide methodological implications. The paper explains how different methods, such as lab experiments, case studies, N-of-1 trials, case control studies, cohort studies, RCTs and patient narratives, all have some strengths and some limitations for picking out intrinsic causal properties. We explain why considering philosophy of causality is crucial for evaluating causality in the health sciences. In our proposal, we combine the various methods in a temporal process, which could then take us from an observed phenomenon  to a causal hypothesis and, finally, to improved theoretical knowledge. The current evidence-based paradigm requires that the most reliable scientific evidence is used to increase patient safety by predicting how a certain intervention might affect health and well-being. This is true both in a strictly clinical context, where doctors and patients need to evaluate the potentiality of interventions for the single case, and in public health, where policy interventions are needed at the population level. How should such evidence best be generated? This is an ongoing debate in medicine and public health, but also in philosophy of science.Recently, there has been some discussions about the epistemic primacy of statistical approaches, such as randomised controlled trials (RCTs), for establishing causality and predict the health outcomes of medical interventions. In the recent article ‘Understanding and Misunderstanding Randomized Controlled Trials’, for instance, Deaton and Cartwright summarise a number of arguments against the dominant reliance on RCTs over every other form of causal evidence . This arScholars who argue that we need plural methods to establish causality use different types of arguments and have different emphasis. Some traditions, such as critical realism, are primarily critical of the ontological bias in scientific methodology; they argue that standard ways to evaluate causal evidence mistakenly rely on a positivist, Humean conception of the nature of causality ,8,9. OthBMJ Evidence Based Medicine, a large group of clinicians, medical researchers and philosophers made a common appeal to evidence-based medicine to expand its notion of ‘evidence’:The rapid dominance of evidence based medicine has sparked a philosophical debate concerning the concept of evidence… The undersigned include 42 clinicians and philosophers from interdisciplinary research networks working specifically on questions related to causality in medicine worldwide. Our research has developed out of a conviction that philosophical analysis ought to have a direct impact on the practice of medicine. In particular, if we are to understand what is meant by ‘evidence’, what is the ‘best available evidence’ and how to apply it in the context of medicine, we need to tackle the problem of causality head on… Establishing causality often requires the use of multiple methods, since no single method will be universally applicable or perfect for this purpose (p. 1)Although the formal approach is still dominant, there is increasing awareness among scientists that the evaluation of causal hypotheses cannot be restricted to purely statistical standards ,13. In aHow should this call for evidential pluralism be understood? How useful are the various methods available for causal inquiry? Further, how should different types of causal evidence be evaluated? Among philosophers of science, various answers to this question have been proposed. For instance, Russo and Williamson (2007) state that a causal claim must be supported both by evidence of difference-making from population studies and from evidence of the underlying mechanism, and that each of these approaches compensate for the weakness of the other . Others In this paper, we propose our own version of evidential pluralism. This version is based on the philosophical idea that any type of scientific claim, including causal claims within medicine and public health, should seek to say something about a system’s intrinsic properties, as well as their mutual influence and causal interaction. Our aim here is to introduce a framework to support scientists in developing appropriate methodological approaches for exploring causality. This is relevant for ensuring that all the evidence necessary for the safe treatment of individuals and populations is considered.stable capacities  are integral to the development of hypotheses of drug safety concerns’ [An example can be found in the practice of pharmacovigilance: the post-marketing monitoring of secondary effects of drugs. The aim of pharmacovigilance is to build a hypothesis of causality between a drug and a symptom, based on spontaneous reports of patients’ experiences. The WHO Collaborating Centre for International Drug Monitoring, Uppsala Monitoring Centre, has the world’s largest database of spontaneous reports of suspected adverse drug reactions. They use this database to assess whether the reported reactions were in fact caused by an intrinsic disposition of the drug. Although pharmacovigilance is largely based on statistical analysis of such databases, a key step in the generation of a causal hypotheses is to assess the likelihood of a causal relationship based on a single or a few patient reports. In the words of pharmacovigilance expert, Rebecca Chandler, ‘oncerns’  (p. 23).The practice of pharmacovigilance is thus an excellent example of how using databases of patient narratives, in combination with theoretical knowledge and clinical expertise, can be effective for detecting causal dispositions.A case study is the detailed description and analysis of one single instance of a certain type of phenomenon. In medicine, it typically consists of the analysis of one medical case that is of particular interest—for instance, if it is new, rare and unusual, or somehow contributes to the advancement of medical understanding. For the purpose of public health, case studies of a particular context are often used to gain insights into the mechanisms underlying observed correlations. For instance, in order to understand the causal relationship between loneliness, social isolation and general health in older adults, case studies were conducted in specific contexts .In medicine and public health, there is disagreement about the epistemic role of such a type of study. One view, which is dominant in the evidence-based paradigm, is that a case study provides qualitative knowledge only about the analysed context and can therefore not be used to derive general causal claims. In contrast, others argue that a detailed case study might have the potential for generalisation across similar instances. According to this view, depending on the research question, generalising from the detailed understanding of one instance can be equally justified  as generalising from the statistical analysis of a few parameters in a larger population ,50. Thisunderstand how the dispositions at place interact with each other. Moreover, it allows the description of rare or unexpected effects in a particular individual or context, suggesting previously unknown dispositions or mutual manifestation partners. This in turn can contribute to theory development of causal mechanisms. For the purpose of uncovering intrinsic dispositions, case studies are important tools. Indeed, they typically include a variety of scientific methodologies to gain a full overview of the context under scrutiny. A well-conducted case study can converge medical examinations, lab analyses, medical history, qualitative interviews, analysis of social context and of lived experiences, and can therefore be analysed from the perspectives of different experts. This detailed and complex analysis aims to On the other hand, case studies have some limitations for the purpose of detecting dispositions. Specifically, the information in the case study is already defined, filtered and interpreted, restricting the causal information gathered from the instance analysed. This is especially relevant in medicine, where case studies translate the narrative and perspective of the patient into medical terms and categories. In this process, the researcher filters the information that is considered relevant, in light of their knowledge, interest and research question. When we miss the plurality of perspectives, we might also miss important information about the dispositions at place in the single case.in collecting surface soil samples, testers had actually sampled new dirt that residents had imported and put over their old, contaminated dirt… ‘They sent out some people to do that testing out here and they scooped a little bit of dirt with spoons on the ground. Hey, I done put dirt on top of dirt trying to get rid of the floods and things we been having out here for years’. … (p. 117)This can be exemplified with a case study in the field of medical anthropology, aimed at investigating the causal relationship between the high incidence of illness in a low income African-American neighbourhood and pollution from surrounding industries . With thThere are, however, also case studies that are more biomedical in nature and that can potentially be of great importance for discovering new and unexpected dispositions. Consider, for instance, the infamous case reports of malformations after embryonic exposure to the drug thalidomide. The study of embryonic malformation in cases of maternal exposure to thalidomide suggested that certain tissues of the developing embryo were, contrary to previous assumptions, manifestation partners for the drug. These reports led to investigations that ultimately resulted in new theories of teratogenicity, safe placental barrier, and sensitivity in different species. This in turn motivated mechanistic studies, uncovering, for instance, the disposition of thalidomide to inhibit angiogenesis and, consequently, to treat myeloma .Case control and retrospective studies are commonly used in medicine and epidemiology. In case control studies, one considers a group of people with a similar condition and tries to find a factor which is common to all the participants’ medical history, but absent in a control group. This means that one is looking for a common cause of social, medical or toxicological type, that might explain the difference in outcome between the test group and the control.The dispositionalist advantage of this study design is that, while looking for a factor that supposedly made a difference between two samples of the same population, it allows a detailed causal study of small samples, such as outlier cases or very rare conditions. In this way, one might detect dispositions and manifestations that are otherwise too rare to show up through other statistical approaches. Such studies also acknowledge the temporal aspect of causality—that dispositions might take a long time to manifest as a detectable outcome. This could in turn contribute to discover a causal mechanism.The downside, on the other hand, is that these studies do not specifically look for manifestation partners that might contribute to the outcome but are not common to all the participants. Indeed, they are designed to find common dispositions across different contexts. We also need to point out that the output of these studies are correlations and not dispositions. Without a further theory of mechanism that can plausibly account for how the common factor brings about the common outcome, we cannot claim to have found an intrinsic property.This point is not only valid for this experimental design, but for all controlled experiments that are primarily designed for detecting correlations between variables at a statistical level. There is, for instance, a correlation between the exposure to paracetamol and incidence of asthma in children . This prAn example of the discovery of a causal disposition from a case control study is the following. Between 1966 and 1969 at the Vincent Memorial Hospital, Massachusetts, eight young women were found to have a rare form of vaginal adenocarcinoma. This observation triggered a retrospective comparative study, which showed that seven of the patients had been exposed to the exogenous hormone stilbestrol during embryonic life . This suIn cohort studies, a large group of people is observed over a period of time, in which relevant medical events and exposures of every participant are recorded in databases. Such data are then used to answer different research questions about existing correlations between possible risk factors or susceptibility factors and health outcomes, at the population level.Like retrospective studies, cohort studies also acknowledge the temporal aspect of causality, which is central if one thinks in terms of intrinsic dispositions and their manifestations. One could argue that, since prospective studies record events as they happen over time, rather than being based on memories and existing records, such studies account better for the temporal dimension than retrospective studies. It is, however, not easy to determine how much time an unknown disposition will take to manifest. In order to make a good estimate of this, one must have a great deal of background knowledge and mechanistic understanding of the processes in place, so that the study is not discontinued before it is possible to make the relevant observations. Some dispositions might take generations to develop, as was the case with stilbestrol and vaginal cancer, while others can manifest within seconds.In addition to the temporal dimension, another dispositional advantage of cohort studies is that they have the potential to address causal complexity and the interactions of different manifestation partners. By broadly monitoring development in a cohort over a long time, one could detect the interdependency of a certain health outcome with numerous factors. In principle, these observations could allow the study of causal complexity by pointing to intrinsic dispositions , extrinsic dispositions  or even their mutual interactions. Again, since these observations are made at the population level, they just indicate causal complexity, but provide per se no understanding of it: of how and why intrinsic properties interact and manifest. Still, in combination with other approaches, such as deep contextual analyses of single cases, the evidence produced in cohort studies can potentially be a good starting-point for the detection of dispositions.The dispositionalist downside of this experimental design is that when resources are limited, priority tends to be given to large numbers of people over large numbers of factors monitored. This takes away a great potential of this study design for uncovering causal complexity of dispositions. Normally, research on databases from cohort studies tends to point to the interrelation of a few rather than many factors, normally just two or three.Cohort studies can be used to discover both biomedical and psychosocial dispositions. For instance, some cohort studies found a positive association between an increased consumption of processed meat and an increased incidence of colorectal cancer . Howeverhttps://www.ntnu.edu/hunt). In this case, a large range of factors were monitored, with an initial aim to study risk factors of cardiovascular conditions, metabolic conditions and quality of life. Because of this unusually broad approach, researchers have been able to use the study database to uncover a considerable correlation between self-reported childhood trauma and adult comorbidity [Another cohort study is the Norwegian Trøndelag Health study (HUNT), which so far has had over 125,000 participants since 1984 . Here, we only consider the strengths and limitations of RCTs for establishing intrinsic dispositions and how they interact with other mutual manifestation partners.The purpose of a randomised controlled trial is to compare the outcome of one or more randomly assigned clinical intervention(s). This is considered the gold standard for causal evidencing in medicine and evidence-based decision making. The reason for this is that it is specifically designed to look for difference makers, while controlling for confounding factors. This means that any difference in outcome between the test group and the control group should be caused by the tested interventions, since all other differences should be homogenously distributed between the two groups. The exclusive reliance on RCTs has been critically debated in recent decades, both from philosophy of science and from the research community (see, for instance, the special issue of An obvious strength of RCTs from the perspective of causal dispositionalism is that dispositions tend to make a difference to the outcome . This meMoreover, RCTs are less suited for identifying mutual manifestation partners for a disposition, since the focus is on one or a few particular interventions for which there can be a control or comparison, and on one or a few outcomes. The experimental design is intended to minimise complexity—for instance, through strict inclusion and exclusion criteria ,59,60. FAn example can illustrate this. Large randomised controlled trials showed a convincing association between the use of statins as a pharmacological treatment of hypercholesterolemia and a lower incidence of cardiovascular mortality . These sIn an N-of-1 clinical trial, several different interventions are tested in random sequence in the same patient or context, and the outcomes are compared. The aim of this type of experiment is to find which treatment is correlated to the best outcome in the single case, especially for complex, context-sensitive and multi-factorial treatments and conditions. For instance, the correlation between type of diet and symptoms of an illness can be tested with N-of-1 trials, with the advantage that the patient works as her own control. Having a perfectly matching control is important, since the response to a dietary change is interrelated to a number of other known and unknown individual variables.N-of-1 trials are good tools for establishing difference making in a unique context. Difference making, we said, can be a good indicator of dispositions, since dispositions tend to make a difference. This is because of their comparative design and because repeatable interventions can be randomly allocated, sometimes even blindly. N-of-1 trials therefore have some epistemic advantages for establishing dispositions, especially because they allow us to monitor the effects of different interventions within the same patient. This means that these studies can help reveal how different dispositions manifest in interaction with a particular patient and their unique combination of dispositions, even if this specific combination of mutual manifestation partners is never repeated in another patient.There are, however, also some epistemic limitations. Further, this type of experiment, taken in isolation, only establishes a correlation. To move from this to claiming that the correlation is an expression of an intrinsic disposition, we will need additional theoretical knowledge and high quality patient-based evidence.An N-of-1 trial could, for instance, be used in cases where we seek to understand which factors could influence the intensity and frequency of symptoms, such as migraine attacks, epileptic seizures, cramps or episodes of memory loss. Although different people might share the same disposition toward migraine, the triggers and the vulnerability to these will typically vary from one individual to another. Such individual variation, which is essential in causal dispositionalism, is not a problem in these trials, since they already consider the most relevant ‘subpopulation’. Still, any such individual outcomes will remain correlations until a further plausible mechanism of action is provided.For example, N-of-1 trials were used in single patients with osteoarthritis to compare the effectiveness of paracetamol and ibuprofen to tailor treatment to the individual patient . While tWe now summarise the advantages and disadvantages of the reviewed research methods for the purpose of detecting dispositions and manifestation partners, as well as indicating the corresponding recommendations, in the following table . Looking at the strengths and limitations of each of these methods, we can see that it is necessary to combine them for the purpose of causal enquiry and evidencing. That said, our analysis also shows why the different methods would have to play different roles at different stages in this process. Specifically, we have seen that, while all types of evidence contribute to detect dispositions and their manifestation partners, some types of evidence play an indispensable role for establishing their intrinsicality. This allows us to support a pluralist methodology while at the same time offer a way to prioritise which methods to use at different stages. In light of the dispositionalist theory, as outlined above, we thus propose the following approach to causal inquiry and evidencing in medicine and public health :In In the process described in From a dispositionalist perspective, establishing that an intervention causes an effect requires three forms of understanding, which should ultimately increase patient safety. First, we need to understand whether the intervention actually has an intrinsic disposition to produce the effect. Second, we need to understand which other dispositions or mutual manifestation partners played or could play a causal role in the process, either contributing to or counteracting the effect. Finally, we need to understand how these different dispositions interacted. This means that causal investigation must aim at theory development and mechanistic knowledge. In conclusion, we wish to anticipate one possible objection to our proposal. Such objection resonates an argument typically used in defence of the current evidence-based paradigm and can be summarised as follows. Is it feasible to prioritise the advance of theoretical knowledge, given the scarcity of resources for medical research, the huge costs and the amount of practical challenges that need to be addressed? Should medicine not be satisfied with finding solutions to problems, rather than using resources to investigate how and why these solutions work? This reasoning is built on the assumption that it is possible to make good medical research, aimed to answer the ‘whether’ and ‘how often’ questions, without the knowledge of ‘how’ and ‘why’. Such an assumption has been extensively addressed and challenged ,10,13. T


Undergraduate clinical assessors make expert, multifaceted judgements of consultation skills in concert with medical school OSCE grading rubrics. Assessors are not cognitive machines: their judgements are made in the light of prior experience and social interactions with students. It is important to understand assessors’ working conceptualisations of consultation skills and whether they could be used to develop assessment tools for undergraduate assessment. To identify any working conceptualisations that assessors use while assessing undergraduate medical students’ consultation skills and develop assessment tools based on assessors’ working conceptualisations and natural language for undergraduate consultation skills. In semi-structured interviews, 12 experienced assessors from a UK medical school populated a blank assessment scale with personally meaningful descriptors while describing how they made judgements of students’ consultation skills (at exit standard). A two-step iterative thematic framework analysis was performed drawing on constructionism and interactionism. Five domains were found within working conceptualisations of consultation skills: Application of knowledge; Manner with patients; Getting it done; Safety; and Overall impression. Three mechanisms of judgement about student behaviour were identified: observations, inferences and feelings. Assessment tools drawing on participants’ conceptualisations and natural language were generated, including ‘grade descriptors’ for common conceptualisations in each domain by mechanism of judgement and matched to grading rubrics of Fail, Borderline, Pass, Very good. Utilising working conceptualisations to develop assessment tools is feasible and potentially useful. Work is needed to test impact on assessment quality. Consultation skills such as obtaining a medical history and performing a physical examination are core elements of undergraduate medical education  and summative objective structured clinical examinations (OSCEs)  with their own descriptors of differing levels of performance. During five pilot interviews , we determined that some assessors could not work with a blank scale so we developed a scale with reference points of ‘Clear pass’, ‘Borderline’ and ‘Clear fail’ “” to enabIn interview, participants were asked to describe the ‘global scale’ they used when judging a medical student to the standard of being ready to enter first year of training as a doctor [intern] (exit standard) by populating a scale with words and phrases. Participants were initially offered a completely blank scale. If they struggled, they were given the assessment scale developed in the pilots “”. ParticParticipants were asked to describe specific student performances to illustrate their conceptualisations, drawing on cognitive interviewing Willis , criticaAll authors contributed to the thematic analysis and critical review at each level of analysis  developed during the pilot interviews so that all research team members could review the evolving analysis “”. Words After 12 interviews there was consensus that no new domains or judgement mechanisms were emerging, and the final interviews had added little. Data from all interviews were combined in table format and all researchers re-analysed the interviews seeking data extracts which confirmed or challenged provisional findings of domains and judgement mechanisms (charting). A second researcher reviewed each domain table critically for alternative explanations.Data extracts were integrated into short descriptions drawing on participants’ natural language and conceptualisations to create ‘word pictures’ (stage 1 mapping and interpretation) which could be used to place students on a scale. These ‘word pictures’ were summarised drawing on the raw data to identify key conceptualisations in the form of ‘word summaries’ (stage 2 mapping and interpretation). These ‘word summaries’ permitted a global overview of the data and were discussed and critiqued at a round-table meeting. The terms ‘word picture’ and ‘word summary’ evolved during conception of the study and analysis of the data. In the final stage (stage 3 interpretation) ‘grade descriptors’ were developed to synthesise all three judgement mechanisms for each of the four grades for each domain. These final ‘grade descriptors’ drew on the ‘word pictures’ and ‘word summaries’, as well as the raw data and participants’ comments about how they graded students. ‘Grade descriptors’ were reviewed and critiqued by a second researcher, then discussed at a round-table meeting. In the case of ‘overall impression’ a second round of reviewing and critique was performed to capture this domain’s complexity in the ‘grade descriptors’. At each stage of the analysis we checked back to the previous stage and the original data to ensure consistency with the language used by assessors. This ensured the natural language was used to create the products of our analysis and drew on it in generating the descriptors. This process of developing ‘grade descriptors’ is further described in “12  experienced clinician assessors were recruited from 11 different clinical specialties. Each had assessed students in at least 10 OSCEs. They were 39 to 56 years old, had 4 to 29 year’s teaching experience and 7 had experience in completing formal workplace-based assessments on students. As well as being undergraduate assessors, all participants had other postgraduate teaching or assessment experience “”. Of theKey findings of the research are described below: participants’ three judgement mechanisms and three examples of the five cross-cutting skill domains are presented first. Assessors’ working conceptualisations identified in the iterative analysis are highlighted within the descriptions of the domains and illustrated by the ‘word summaries’ Table . We founAssessors used three judgement mechanisms: observations of students’ behaviour, inferences and feelings about the student’s behaviour Box . Within Application of knowledgeManner with patientsGetting it doneSafetyOverall impressionFive domains of working conceptualisations emerged in participants’ interviews:These are conceptually different from current discrete sequential or task-based domain categorisations of skills currently used in our medical school assessment rubrics ; “The patient doesn’t feel listened to and starts to switch off from the doctor; having forgotten what’s already been said” (Assessor 6); “little conversation, conversing only the clinical bit, focusing on the task” (Assessor 10).In contrast, with a ‘very good’ student, participants ‘felt’ ‘reassured (about skills to work with patients)’. This conceptualisation emerged from the raw data and the ‘word picture’: ‘Reassurance that student knows what they are doing. Able to be human and warm as well as professional. Creates the beginnings of a doctor-patient relationship.’ This ‘word picture’ closely relates to data with exemplifying extracts that the student “Conveys a degree of reassurance that they know what they’re doing” (Assessor 1) and have the “beginnings of patient doctor relationship” (Assessor 9).‘Safety’ was a prominent feature of participants’ discourse. Working conceptualisations of the ‘safety’ domain were underpinned by the three judgement mechanisms. Key conceptualisations identified in ‘word summaries’ drew on raw data from participants as described below .Awareness: If participants inferred that candidates were unaware of the hurts and harms they caused or may have caused; the candidate was considered to be failing while those who exhibited awareness were considered to be borderline  and if students changed their approach to reduce hurt or harm they were considered to be of passing standard (Assessor 1).Potential for remediation: If participants inferred that students’ deficits were remediable, participants were likely to judge them borderline .worrying’ , ‘dangerous’ , ‘cannot be trusted’ (Assessor 6) or ‘scary on their own’ (Assessor 3)) led to a fail. Conversely if the participant felt the student had demonstrated ‘honesty in mistakes’ (Assessor 10) this led to a borderline judgement.Trust: Any feelings of distrust : assessors feel happy to have them as a foundation doctor [intern] and feel “you almost forget that they’re a medical student” (Assessor 9).Being a professional: with very good students participants described feeling like they are ‘petulant’’ they are unable to continue (Assessor 10), whereas borderline students may be perceived as ‘’nervous’’, or demonstrate ‘’panic’’ or ‘’inappropriate emotion’’ with some impact  but are able to continue.Managing emotions: participants inferred failing students may get so angry, upset or ‘‘lack insight or don’t know they are wrong” (Assessor 4).Insight: with failing students, participants inferred they “not accepting responsibility for own learning or for care of the patient” (Assessor 4). Whereas a student who a participant inferred was “conscientious” (Assessor 12) was graded ‘very good’.Taking responsibility for their actions: with a failing student participants may infer students are “exam seriously or acting’  or being: overconfident or arrogant” (Assessor 7).Attitude: participants inferred that borderline students may have attitudinal problems: not taking the ‘‘Overall impression’ denotes a set of descriptions of ‘the impression the student made on me’ with which these participants informed their assessment. In these descriptions, participants’ judgements were more abstract, often based on inferences and feelings than descriptions of what students did “”. Across‘Grade descriptors’ encapsulate participants’ descriptions of students drawing on one or more of the three judgement mechanisms in each domain. They were developed from ‘word summaries’, ‘word pictures’ and the raw data for all five domains Table . For exaAcross domains, ‘borderline’ grades were described using a mixture of ‘fail’ and ‘pass’ characteristics and being able to respond to feedback or improve.Across domains, ‘very good’ grades were described as exceeding expectations and showing flexibility and adaptability to situations with some participants reflecting that a student’s consultation skills were better than his/hers at that stage.The core of our findings describes assessors’ idiosyncratic reasoning thus highlighting the need to pay more attention to this in the design of assessment tools. Participating assessors used their working conceptualisations when forming exit standard consultation skills assessments based on three mechanisms of judgement  across four skills domains, ‘Application of Knowledge’, ‘Manner with patients’, ‘Getting it done’ and ‘Safety’ and one more abstract skills domain of ‘Overall impression’. While some of the domains identified correlate with those commonly present in rubrics generated using expert consensus, this study provides novel data on how these domains are operationalised in practice through working conceptualisations of assessors. Furthermore, expert consensus rubrics don’t address how assessors variably choose to draw on observation, interference and feelings in qualitatively evidencing their judgements and making choices about how to weigh these different mechanisms in different domains.The five domains identified have some resonance with findings in postgraduate training assessment studies but do not match completely. Domains described for postgraduate assessment tend to be broader; for example, clinical skills and professional behaviour . Or it may be the data is evidence of assessors applying stereotypes or ‘person models’  was not appropriate in this study. Some qualitative methodologists may disagree with this although limitations of member checking have been described (Mays and Pope If one was to take a purely cognitive view on assessment it might be questioned whether our findings are a product of poor assessor training. As indicated in our study rationale and data, examiners do not take a purely cognitive approach to assessment and this is regardless of training. All our participants were experienced examiners who had engaged with the training requirements of the medical school, and these were comparable to training requirements commonly used as best practice elsewhere.Finally, the assessment rubric is of necessity brief: it needs to be a document which is usable by assessors undertaking a cognitively challenging task (Tavares and Eva We have shown that it is in the application of judgement that working conceptualisations come to the fore of assessors thinking and hence training in knowledge of assessment rubrics may always be ‘trumped’ by assessors’ prior experience and intuition when interacting with students. Our data suggests assessors who deviate from standard rubrics may be doing so in the belief that overlooked significant factors are at play, rather than because they do not understand how to apply the rubric consistently.That working conceptualisations are identifiable is an exciting finding and encouraging for this field of research. Remaining questions include: are these conceptualisations shared by a larger, multi-institutional cohort of assessors within different contexts? Could assessment tools using working conceptualisations and natural language descriptors reduce the potential error in translation between assessors’ conceptualisations and an external rubric (Gingerich et al. Assessment tools aligned to clinician assessors’ working conceptualisations may help students understand for example, professional concerns around safety, the need to respond constructively to errors, the mismatch between checklist and global scores (Hodges and McIlroy Our findings demonstrate that experienced clinicians use identifiable working conceptualisations when assessing undergraduate medical students’ consultation skills. We have also demonstrated that assessment tools drawing on participants’ conceptualisations and natural language can be generated, including ‘grade descriptors’ for common conceptualisations in each domain by mechanism and matched to the commonly used grading rubric of Fail, Borderline, Pass, Very good. These tools are aligned to the ‘real life’ approach taken by clinicians in assessing undergraduate consultations skills. Further work is needed to explore application of the research findings including prospective utility for assessors and institutions, and the impact on assessment quality.


In its current form, evidence-based practice in psychology gives scientific research precedence in psychotherapy practice. However, psychotherapy practice’s complexity warrants reflection beyond the limits of science. The importance of clinical expert is not recognised in the current policy statement. The clinical expert is necessary to translate psychological research into clinical practice. It is also crucial to identify, clarify and include patient preferences in psychotherapy practice. This paper argues that virtue ethics is a useful theoretical framework for conceptualising clinical expertise. Clinical expertise is conceptualised as the meta-capacity of practical wisdom ( Evidence-based practice in psychology is the most important regulatory policy-statement in current psychotherapy. Its historical background includes the work of Archie Cochrane . CochranThe earliest definitions of evidence-based medicine emulated Cochrane’s ideals. Over the course of some decades, however, evidence-based medicine scholars have revised their models several times. A significant change was to expand Cochrane’s model into a tripartite model. In the tripartite model, evidence-based medicine is defined as the integration of best research evidence , clinical expertise and patient values. In a subsequent revision, There is a similar genesis in the regulation of psychotherapy practice. The American Psychological Association launched the criteria for empirically validated treatments  . In empiHowever, Berg has shown that the part ‘best available research’ define and legitimate the two remaining parts ‘clinical expertise’ and ‘patient characteristics, culture and preferences.’ Thus, evidence-based practice in psychology only consists of one part and not of three parts. Nonetheless, there are good reasons for re-defining evidence-based practice in psychology as a tripartite model consisting of best available evidence, clinical expertise and patient characteristics, culture and preferences .phronesis or practical wisdom. In addition, there are three classes of virtues corresponding to the three parts: best available research, clinical expertise and patient preferences. These classes of virtues are epistemic virtues, relational virtues, self-reflective virtues. In each class, non-exhaustive examples of specific virtues are given.Virtue theory originated in Ancient Greek with the works of First, virtues are deep-rooted excellences that exceed mere skill. Like skills, they are cultivated through repetition, but unlike skills they become habitual dispositions which shape the person’s character and outlook on the world . Virtuesphronesis which can be translated into practical wisdom  and techne or mere practical skill . Phronesis is the kind of knowledge and skill that enable human beings to understand and do the right thing in practical matters. It is ‘the link between the intellectual and the moral life […] habitually [disposing us] to attain truth for the sake of action as opposed to truth for its own sake’   . These e‘[…] the anxiety of choice in complex circumstances […] enabl[ing] us to assess the complexities as accurately as possible and to approximate, as closely as the circumstances permit, what would be right and good’ .In clinical practice, phronesis is the indispensable , perception (in order to notice facts in a situation) and finally, understanding (noûs), a common and practical good sense [.Within the evidence-based practice in psychology framework, it means the ability to integrate all of the relevant complexity in clinical action. Evidence-based practice in psychology (implicitly) defines the most relevant complexity through its three parts. These three parts have three corresponding classes of virtues:(i)Best available research: Epistemic virtues(ii)Patient culture characteristics and preferences: Relational virtues.(iii)Clinical expertise: Self-reflexive virtues.A virtuous clinical expert knows what the right means for the right ends are. This includes questions of both fact (means) and value (ends). Moreover, it indicates that theory of knowledge is ethically relevant. Virtue epistemology telescope questions of knowledge with questions of ethical character. The overarching emphasis is the question of being truthful or knowing the truth .There are two prominent traditions in virtue epistemology. These are called reliabilism and responsibilism . ReliabiDue to psychotherapy’s complexity, the clinical expert must possess the epistemic virtues to understand the merits of different kinds of knowledge. This entails assessing the epistemic quality in different kinds of knowledge . The risA classic virtue ethical doctrine states that one should treat only like cases as alike. In psychotherapy, every patient differs in clinically significant matters and, presumably, different patients have different notions of the good life. As noted by phronesis. However, in practice the two traditions may be less contradictory. The question of what it means for a given patient to live well invoke the patients’ autonomy to decide over her own life. In order to obtain patient autonomy, patients must be willing to explore and share their  preferences. This makes trustworthiness a basic relational virtue . Anotherhow the therapist understands is often as informative as what the therapist knows about a patient  necessitating extra-scientific deliberations (Emphasising the role of erations . VirtuouThe author confirms being the sole contributor of this work and has approved it for publication.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


While systematic review (SR) methods are gaining traction as a method for providing a reliable summary of existing evidence for health risks posed by exposure to chemical substances, it is becoming clear that their value is restricted to a specific range of risk management scenarios - in particular, those which can be addressed with tightly focused questions and can accommodate the time and resource requirements of a systematic evidence synthesis.The concept of a systematic evidence map (SEM) is defined and contrasted to the function and limitations of systematic review (SR) in the context of risk management decision-making. The potential for SEMs to facilitate evidence-based decision-making are explored using a hypothetical example in risk management priority-setting. The potential role of SEMs in reference to broader risk management workflows is characterised.SEMs are databases of systematically gathered research which characterise broad features of the evidence base. Although not intended to substitute for the evidence synthesis element of systematic reviews, SEMs provide a comprehensive, queryable summary of a large body of policy relevant research. They provide an evidence-based approach to characterising the extent of available evidence and support forward looking predictions or trendspotting in the chemical risk sciences. In particular, SEMs facilitate the identification of related bodies of decision critical chemical risk information which could be further analysed using SR methods, and highlight gaps in the evidence which could be addressed with additional primary studies to reduce uncertainties in decision-making.SEMs have strong and growing potential as a high value tool in resource efficient use of existing research in chemical risk management. They can be used as a critical precursor to efficient deployment of high quality SR methods for characterising chemical health risks. Furthermore, SEMs have potential, at a large scale, to support the sort of evidence summarisation and surveillance methods which would greatly increase the resource efficiency, transparency and effectiveness of regulatory initiatives such as EU REACH and US TSCA. The methodology was developed in response to medical practitioners’ need to distill clear and reliable conclusions about the efficacy of clinical interventions from an evidence base seemingly full of contradiction, heterogeneity and bias . This neConsequently, interest in the application of systematic review to regulatory decision-making contexts within chemicals policy and wider environmental health is growing. This is evidenced by the increasing number of systematic reviews published in the field , the estin vivo toxicity testing. Although vital for compiling a more complete understanding of a chemical’s toxicity, the broad scope and increasing availability of such data presents challenges for decision-makers tasked with handling, appraising and interpreting this data for risk assessment. Failure to have a transparent structure for considering all relevant data appropriate to risk assessment  reduces stakeholder confidence and has the potential to bias regulatory decisions. Studies reporting results amenable to the observer bias of independent assessors, or to the vested interests of non-independent assessors, may be cherry picked from the wider evidence base. Even where all relevant studies are considered, the role that scientific judgement plays in the process of appraisal and interpretation of data can lead to conflicting conclusions between different regulatory bodies  aligns well with the needs of chemicals policy, conflicts between the practicalities associated with the methodology and those associated with regulatory frameworks hinder their wider uptake, and/or the production of reviews that are of sufficient quality to produce trustworthy results . Key areIn this manuscript, we briefly outline systematic review methodology to illustrate its strengths and highlight the transferable barriers which have been suggested as preventing its wider uptake in other fields . We disc2.The utility and advantages of systematic review methods for advancing chemical risk assessment have been extensively documented elsewhere . SystemaThe key features of a systematic review  are:a clearly specified research objective - usually captured in a Population-Exposure-Comparator-Outcome (PECO) statementa comprehensive search strategyscreening of the search results - for evidence relevant to addressing the research objectiveextraction of data from included studies - using a prespecified data extraction frameworkcritical appraisal of included studies - according to a prespecified set of quality criteria, usually targeting risk of biassynthesis of findings from the included studies - using suitable quantitative statistical methods and otherwise qualitative methods as appropriatecharacterisation of confidence in the evidence for the results of the synthesis - according to a prespecified set of criteriastatement of conclusions - including an assessment of limitations in design and conduct of the review itself.Specific methodological decisions concerning each of these key features, from definition of the PECO statement to the chosen synthesis approach, are specified in a pre-published protocol.However, with the methodology’s pursuit of rigor and comprehensiveness comes a significant demand for time and resources. Evidence from medical systematic reviews indicates it takes on average approximately 70 weeks to progress a systematic review from protocol registration in the PROSPERO registry  to publiSystematic review management software has been developed e.g. “ to assisAlso at conflict with the demands of regulatory decision-making is the narrow scope of systematic reviews, which are designed to address a specific and clearly defined objective or research question. To ensure a manageable, relevant and focused review, suitable research questions are typically closed framed, such that the review can synthesise a single, coherent answer. These closed-framed questions are well suited to the decision-making contexts of medicine (the field from which systematic reviews originate), but may be difficult to apply to chemical risk assessment. The web of interlinked endpoints, potential variation in sensitive populations, uncharacterised low dose effects, and unknown behaviour of a chemical in the environment or in contact with other chemicals can mean that the decision-critical information which can be supplied by a tightly focused research question is often not readily apparent in chemical risk assessment contexts. Even where such a question can be devised, and the answer reached through systematic review, the specificity of the research problem and its resolution are likely to comprise only part of the much broader range of unaddressed decisions and information requirements faced by risk managers.3.In light of the time and resource intensity of current systematic review practice, identifying the most informative research questions is important for maximising the value and efficiency of systematic reviews in regulatory decision-making. Investing resources in systematic review as a means of addressing specific research questions is inefficient if there is a lack of data available for answering those questions. Devising specific research questions therefore becomes a reactive process, rather than a proactive one. This is at odds with the goals of chemicals policy, which aims to predict and prevent harm as a result of exposure to chemical substances.Decision-makers therefore need to monitor and understand the evidence base as a whole – such that emerging trends or issues of potential concern can be identified and investigated in a timely manner. Identifying trends in the evidence base, including evidence clusters and evidence gaps, facilitates the formulation of proactive research questions by relevant stakeholders. Reviewers need not rely on environmental health outcomes becoming infamous or epidemic as an indicator of sufficient evidence for an efficient and valuable synthesis. Instead, trends in the availability of evidence ensure prevention of synthesis attempts for which there is insufficient data  and promote the targeting of primary research efforts at evidence gaps. This kind of evidence surveillance has traditionally been the domain of scoping reviews. These reviews are often narrowly focused precursors to systematic reviews. Thus a specific systematic review question has already begun to be framed, and the literature scoped for sufficient data to address/focus it – rather than vice versa e.g. . ScopingInstead, the introduction of systematic evidence mapping, a methodology recently adapted from the social sciences  for enviThe methodological steps involved in constructing a systematic evidence map are similar to those involved in the initial stages of producing a systematic review see  whereby To facilitate this exploration, the output of a SEM takes the form of a queryable database  as opposWhereas systematic reviews present users with select information from included studies (i.e. data relevant to addressing the research question), SEMs aim to extract a broader range of data from included studies and aim to maintain the native format of these data. In this sense, the search and screening process are the steps of SEM methodology most affected by its research objective or context, as the focus of data extraction remains broad regardless. This is in contrast to systematic review, where all steps are heavily influenced by its research question. The data extracted for inclusion in a SEM database can then be flexibly categorised, or “coded” to facilitate comparison of an otherwise heterogeneous evidence base.Resolution of coding can be adapted to suit the needs of regulators. For example, coding the species under investigation in a study might use categories such as “Sprague-Dawley”, “Rat”, “Rodent” or “Mammal”; or may use all of these categories such that the data can be interrogated in successively deeper levels of detail. As well as facilitating variably resolved interrogation of the evidence base, coding plays a significant role in systematic mapping’s amenability to updating. Use of universal, standardised ontologies for coding, such as the Unified Medical Language System (UMLS) , offers only coded data may offer a saving in the resource intensity of the process. However, in maintaining a transparent link between raw extracted data and the code used to categorise it, SEMs offer a gateway to automation – whereby controlled vocabulary ontologies can be used to train machine learning algorithms to automatically identify, extract and code data from the literature.In current practice it is common to present users with SEMs that house only coded information for simplicity and ease of access e.g. . HoweverPending such advances, the time required to conduct a fit for purpose systematic map in environmental health is uncharacterised. Evidence from the wider environmental sciences  suggestsAs the purpose of a SEM is to characterise the evidence base, there is no risk of allocating resources to the production of an inconclusive output, as is the case for “empty” systematic reviews (systematic reviews which ask research questions for which there is too little included evidence for them to reach a conclusion or be supportive of a decision). In fact, systematic evidence maps may reduce the resource strain associated with systematic reviews. A SEM’s broad overview of the evidence base allows fast identification of topics for which there is sufficient data to warrant a full systematic review. The SEM itself, if conducted to sufficiently rigorous standards, can even replace the literature search and screening process of a systematic review. As SEMs present all available relevant evidence on a broader topic such as the “health effects of bisphenol-A” (obtained through a systematic but less specific search strategy), filtering this information according to the PECO statement of a systematic review may act in an equivalent manner to approaching the literature with a more focused search strategy in the first instance. The pre-screened nature of this subset is likely to reduce the number of false positive results, facilitating faster syntheses.As advances in machine learning facilitate more highly resolved data extraction processes, future SEMs may even store enough detail for them to form the basis of meta-analytical syntheses. If all data contained within study reports is extracted and indexed within a SEM, there would be no data required specifically for syntheses which could not be found in the SEM. This would allow SEMs to form the dataset on which meta-analytical and predictive toxicological models are based, the results of which may additionally be incorporated into the SEM itself – facilitating more transparent, resource-efficient and easily updated syntheses.4.Systematic evidence mapping facilitates identification of trends which are informative for many risk management scenarios. To illustrate the flexibility and potential utility of SEMs’ trendspotting capacity, this section highlights the type of data visualization and exploration possible through querying subsets of information in a SEM database. Specifically, “priority setting” , the proSeveral factors are relevant to prioritizing individual chemicals for assessment, broadly ranging from recorded levels of exposure to evidence for toxicity. Underlying these broad considerations are several more specific factors such as the bio-accessibility of the chemical, the relevance of its toxicity evidence for predicting health risks in human populations etc. In order to make the most efficient use of resources and the systematic review process, decision-makers require access to a means of comparing these features to justify prioritization of a particular chemical for review/risk assessment.This is the role of a SEM, which may be constructed with the aim of identifying and characterising the risk assessment relevant evidence for a broader group of legacy chemicals, e.g. flame retardants. Once data has been extracted and coded from the literature, the SEM can be explored with a succession of queries of increasingly narrow focus, each considering a narrower subset of the evidence base than the last, such that a research question appropriate for more detailed synthesis is resolved at the end of a process which begins with a very broad research objective. This is illustrated in Queries 1 and 2 depicted in Based on  existing evidence, Queries 1 and 2 indicate flame retardants A and B as potential candidates for full assessment. Resolving which to prioritize involves accessing more study-specific information through a series of queries which consider a successively narrow subset of the evidence base. Despite availability of toxicity data, observing flame retardants in human relevant locations might not be concerning if the concentrations observed are negligible. Thus Query 3 examines the range of concentrations reported in the literature for FRs A and B in human blood and breast milk. Visualization of Query 3 indicates a wider range of lower concentrations reported for FR A, compared to a narrower range of higher concentrations for FR B. Query 4 then examines the relevance of these concentrations against the current estimated tolerable daily intake (TDI) for FR B, indicating several observations of toxicity below the current TDI and supporting prioritization of FR B for assessment. Further, the relatively large volume of observations of neurotoxicity may indicate sufficient data available to conduct a systematic review on FR B’s relationship with neurotoxicity.However, it is important to distinguish the results of SEM queries from synthesis. SEMs only present what has been studied in the literature – they cannot present what has not been studied, and do not always assess the risk of bias of the findings they report. Thus, while a high number of observations of flame retardants A and B in human relevant locations is a valid trend to explore further, it does not necessarily mean that there are fewer of the other flame retardants present in human relevant locations, but rather that there may simply be fewer of these flame retardants studied at all. Identification of such evidence gaps is equally valid for focusing primary research. For example, the relatively high number of observations of reproductive toxicity for FR F, but comparatively low number of observations of this flame retardant in any exposure locations might warrant re-analysis of samples or new exposure studies to verify whether exposure to this substance is of concern.The SEM is also sufficiently flexible that different trends can be investigated, and different research questions formulated, based on the priorities of regulators. For example, the number of observations in the literature which found FR D in aquatic environments might spur further investigation into the ecotoxicity of this compound. A single SEM exercise therefore makes efficient use of resources in its potential to meet the varied needs of several end users.5.In addition to priority setting, SEMs have the potential to fill several roles within wider workflows.5.1.Although evidence synthesis methodology can be considered costly in terms of time and resources, this cost can be dwarfed by the equivalent resource demands associated with conducting primary research relevant to assessing the hazards associated with exposure to a chemical, as illustrated with more established examples in the field of medicine . In an eSEMs have the potential to provide this much needed transparency. The nature of a SEM’s output being a collection of relevant search results, and specific information coded from those results, introduces a greater level of accountability for registrants. Studies are identified by registrants as “key”, “supporting” etc. based on the perceived relevance, adequacy and reliability of the evidence they provide for a specific endpoint, assessed using “sound scientific judgement” . These a5.2.Beyond offering improvements in transparency during the data gathering phase, SEMs may be of particular value to the problem formulation stage of regulatory decision-making. Problem formulation is a prerequisite to conducting a chemical risk assessment, identifying an issue of regulatory relevance around which the assessment will be focused . These i5.3.Identifying trends in the evidence base may also play a significant role in read-across applications. Read-across allows the toxicologically relevant properties of a chemical to be inferred by comparison with a structurally similar chemical of known toxicological behaviour . Read-ac5.4.Once regulation is in place, it is vital that it is kept up to date. Such is the role of the ongoing, evidence surveillance phase of regulatory decision-making. Within REACH, registrants are required to update their registration dossiers “whenever new information is available” , such thOpenly accessible and easily updated SEMs may serve to address such obstacles. As the population of a SEM database does not require detailed analysis or complex interpretation of the raw data, SEMs could be amenable to automation. Technological advances in text-mining and artificial intelligence might assist the automatic screening, extraction and coding of new information as it is published, based on the data fields and coding ontologies used to populate the original SEM. Although some years away from implementation, application of SEM methodology in the interim will promote fast uptake of such technological advances.6.Systematic evidence mapping presents a transparent and robust methodological framework with which to assess the evidence landscape at the level of individual chemical risk management and innovation, to regulatory decision-making in chemicals policy. The broad scope of SEMs lowers the barrier to evidence synthesis in chemical risk assessment through more efficient use of resources. Future developments in text mining and machine learning are likely to further reduce the resource intensity of the methodology, and of chemical risk assessment in general. These advances will enable the automatic production of highly resolved SEMs capable of synthesising evidence or feeding predictive models.In the interim pursuit of a more evidence-based approach to chemicals policy, the resource strain associated with producing a SEM can be managed through adaptation of the methodology to present day limitations. Depending on the needs of the user and the constraints of their use case, SEM methodology is sufficiently flexible that it may be adapted  without compromising the utility of the end product in the same way as the results of a synthesis might be adversely affected by modification of systematic review methodology. By working closely with stakeholders to define objectives, the scope of the SEM  can be adjusted as appropriate to objectives. For example, critical appraisal of studies may not be imperative to the aim of the SEM and may therefore be omitted or might be planned as part of a stepwise approach after the SEM identifies pockets of evidence of interest to stakeholders. Although designed to reduce the resource strain of SEM exercises, such flexible adaptation of the methodology does not compromise the fitness-for-purpose of SEMs as a means of identifying and comparing trends in the availability of evidence in a vast and heterogeneous information landscape.Consequently, examples of research activities producing fit-for-purpose SEM outputs and/or developing aspects of SEM methodology specific to chemicals policy contexts are beginning to emerge , with re


A common feature of performance assessments is the use of human assessors to render judgements on student performance. From a measurement perspective, variability among assessors when assessing students may be viewed as a concern because it negatively impacts score reliability and validity. However, from a contextual perspective, variability among assessors is considered both meaningful and expected. A qualitative examination of assessor cognition when assessing student performance can assist in exploring what components are amenable to improvement through enhanced rater training, and the extent of variability when viewing assessors as contributing their individual expertise. Therefore, the purpose of this study was to explore assessor cognition as a source of score variability in a performance assessment of practice-based competencies.A mixed-method sequential explanatory study design was used where findings from the qualitative strand assisted in the interpretation of results from the quantitative strand. Scores from one objective structured clinical examination (OSCE) were obtained for 95 occupational therapy students. Two Generalizability studies were conducted to examine the relative contribution of assessors as a source of score variability and to estimate the reliability of domain and holistic scores. Think-aloud interviews were conducted with eight participants assessing a subset of student performances from the OSCE in which they participated. Findings from the analysis of think-aloud data and consideration of assessors’ background characteristics were used to assist in the interpretation of variance component estimates involving assessors, and score reliability.Results from two generalizability analyses indicated the highest-order interaction-error term involving assessors accounted for the second-highest proportion of variance, after student variation. Score reliability was higher in the holistic vs. analytic scoring framework. Verbal analysis of assessors' think-aloud interviews provided evidential support for the quantitative results.This study provides insight into the nature and extent of assessor variability during a performance assessment of practice-based competencies. Study findings are interpretable from the measurement and contextual perspectives on assessor cognition. An integrated understanding is important to elucidate the meaning underlying the numerical score because the defensibility of inferences made about students’ proficiencies rely on score quality, which in turn relies on expert judgements. Performance assessments  are one method for assessing practice-based competencies within a simulated context in a direct manner in health professions education . PerformAssessor cognition is a growing area of research to investigate how assessors’ cognitive processes affect decision making and assessment quality. Previous studies have identified different perspectives on assessor cognition by which to view the causes of assessor variability, including “assessor as trainable”  and “assessor as meaningfully idiosyncratic”  . The meaThe variability in assessors’ decision making could be considered problematic for measurement quality. The differing cognitive processes of assessors in formulating judgements may result in inaccurate assessment outcomes due to factors other than the intended measured construct. Therefore, an examination of assessor cognition can provide insights into its contribution to score variability, reliability, and validity. To examine assessors’ thinking processes in performance assessments, prior research has employed qualitative or mixed-method approaches. By using quantitative methods, researchers have compared scoring between assessors or between groups to ascertain the source of score variability . By usinThe purpose of this study was to explore assessor cognition as a source of variability in a performance assessment of practice-based competencies within an occupational therapy program. Quantitatively, assessor variability was estimated from observed ratings of performance. Qualitatively, assessor cognition as a source of variability was examined using think-aloud interviews. Together, these perspectives provide an integrated picture of the nature and variability of assessor cognition when rendering judgments on student performance.A mixed-method sequential explanatory study design was used where findings from the qualitative strand assisted in the interpretation of results from the quantitative strand , 10. EthNinety-five students within a course-based master’s program in occupational therapy participated in an OSCE at the end of their first year, or two semesters, of coursework. OSCEs occur four times within the 26-month occupational therapy program, one at the end of each semester of coursework. The OSCE aimed to assess the practice-based competency domain of “Expert in Enabling Occupation”. Enacting the role of an expert in enabling occupation requires the student to demonstrate competencies such as clinical reasoning, professionalism, and effective communication. The OSCE entailed the student communicating a clinical summary of a client case to their supervisor. This scenario simulated an authentic task that occupational therapy students would experience when participating in case conferences with the healthcare team or debriefing client cases with their clinical supervisor. The clinical summary typically included a description of the client’s occupational profile, the client’s concerns related to participation in everyday activities, initial assessment findings, and proposed intervention strategies.A rubric  was usedFourteen members from the occupational therapy department assessed the students during the OSCE. Thirteen assessors were female; one was male. All assessors held a practice-to-entry degree in occupational therapy representing varied practice backgrounds including mental health, neurological rehabilitation, occupational rehabilitation, pediatrics, and chronic pain, across the continuum of care. Ten were academic faculty with between 3 and 25+ years of experience in their role. Three assessors were clinical faculty with between 1 and 4 years in the role, and one assessor was a PhD student in their fifth year. On average, participants had participated in between two and three OSCEs in its current form. Assessors were paired by the OSCE coordinator; all but two taught within the first year of the program and were familiar with the students and curriculum completed at this stage in the program.Within a group setting, the assessors were provided with an orientation to the OSCE by the OSCE coordinator one week prior to the event. Participants were briefed on the objectives of the OSCE and the purpose of the present study. To facilitate assessors’ development of a shared understanding of expected performance at this stage in the program, assessors were presented the rubric and directed to consider the performance descriptors and distinguishing features between categories for each domain. During the orientation session, assessors were given an opportunity to discuss and ask questions to clarify the rubric content, exam procedures, and assessor expectations. Assessors were instructed to score each student along each of the domains. Then, considering their performance overall, assessors were instructed to assign a holistic score. Each student was assigned to a pair of assessors , who assessed the student independently. Seven non-overlapping pairs of raters assessed between 8 and 14 students each. To keep the OSCE administration within the available time and resources, student assessments were conducted concurrently among the seven rater pairs. Due to this arrangement, it was not possible to use the same standardized patient across rater pairs. All rater pairs were instructed to score the student independently both at the domain level and holistic level. At the end of the encounter, students were provided with brief verbal feedback by the raters, typically focusing on one area that was done well and one suggestion for improvement. As per standard practice, for the purposes of review in the event of an appeal, all OSCE performances were video-recorded, saved on an external hard drive and stored in a secure location for later viewing.Domain and holistic scores were summarized through descriptive statistics. We first calculated the mean scores and standard deviations within each assessor and domain combination, then within each domain collapsed across assessors. A Pearson correlation analysis was completed using JMetrik  to explop x d x r), and 2) one-facet G-study with students fully crossed with raters . Different groups of students were assessed by different raters  resulting in a nested design and sparse data. The rater approach was used to generate variance component estimates for each rater pair using a publicly available SAS macro  address a little bit more on maybe physical and social aspect and cognitive aspect as well. (Assessor 1)As another example, Assessor 3’s expectations appeared to be informed from the dual perspectives of a clinician familiar with physical interventions and as an educator within the program. This working knowledge of clinical practice and students’ knowledge and skills at this stage in the curriculum then framed their expectations regarding the adequacy of clinical reasoning.So she’s linking the vacuuming to the shoulder and that’s going to be difficult. I know they [the students] don’t have a lot of experience yet and that part of this could be a lack of experience in the level that they are at but then again it’s that clinical reasoning behind that. So right now, I’m missing a little bit of link between, how are you not causing pain in that shoulder? (Assessor 3)For some assessors, expectations regarding the student’s responses were not directly linked to the rubric criteria, perhaps reflecting more personal expectations on how the student’s responses should unfold. These expectations could likely influence assessors’ evaluations potentially contributing a source of idiosyncratic variance in the observed scores. Assessor 3 illustrated this kind of expectation regarding any assumptions made by the student when designing a treatment plan. For Assessor 3, identifying and discussing assumptions provided a clearer window into the students’ clinical reasoning process:So why are you making that assumption but also, what are some of the other assumptions that you could make and that you don’t think are relevant in this aspect … ? So to really give me some insight in why you’re thinking what you’re thinking, why are you making that assumption and not a different assumption? (Assessor 3)As another example, the language used in one student’s response set up an expectation for Assessor 5 regarding communication of the client’s intervention plan to the clinical supervisor.So I’ve had to give a more definitive description on what progression of a treatment is. I would’ve expected that to have been a little more spontaneous, in that we would start here and then we moved here and what types of things we would add or take away to test acquisition of strength and mobility or what, given that it is a wrist injury. (Assessor 5)The salient observations made that would later inform scoring were not the same across all students for each assessor. We interpreted this finding as assessors reading into the rubric different things for different students. In this context, we had the rubric domains, assessor, and student coming together in an idiosyncratic way. To further explore this variability, we examined the verbalizations of two pairs of assessors, who were paired together during the original OSCE and also participated in think-aloud interviews, assessing the same student. The following examples demonstrated variability of thought processes among assessors while scoring the same student performance in the domains of Communication, Evidence-Based Practice, and Knowledge of Client.Communication-wise, but I think she needs to be more concise in her communication, so she can get more information, more useful information, in there. (Assessor 2’s judgement on Student R)Her communication is very clear and she's able to answer all questions. She has a very good body posture and very clear communication. (Assessor 7’s judgement on Student R)Her evidence-based practice, she was able to give quite a bit of detail about the different things they looked at, but maybe not as much detail about what was the evidence about those specific components of the group program. (Assessor 2’s judgement on Student P)Evidence-based priorities, she's good at this part. She can demonstrate very good knowledge about evidence based. (Assessor 7’s judgement on Student P)She was giving a summary of the case and talking about the environment of the client, the social aspect of the client (was) pretty okay... Somehow, she’s able to explain like the client and show a little bit of understanding of the client. She talks a lot about the social aspect and the environment, which we think she has built an understanding of that. (Assessor 1’s judgement on Student S)So she's giving a summary of her client. I felt that the summary was very weak... Knowledge with the client again didn't really know enough or go into enough depth on the client, so I'd give her intermediate for that.(Assessor 8’s judgement on Student S)These examples reflected that assessors had varying perspectives and interpretations even though they were assessing the same performance. While scoring the Communication domain, Assessor 2 considered the student’s delivery of their answers and concluded that there was room for improvement; whereas Assessor 7 considered the student’s body posture and ability to answer questions and concluded that the student demonstrated clear communication. In the second example, Assessor 2 thought that the student could provide more details regarding evidence; however, Assessor 7 thought that the student performed well in the domain of Evidence-Based Practice. In the last example, while judging the domain of Knowledge of Client, Assessor 1 focused on the student’s clarification on the social and environmental components of the client, and Assessor 8 seemed to reflect on the student’s overall knowledge of the client and concluded that the student did not dig deep to understand the client.From the contextual perspective, differences in judgement between the paired assessors may represent legitimate differences instead of errors. Each assessor is an expert in the field of occupational therapy, and although the area of expertise may differ, each perspective is considered valid. Assessors’ prior knowledge and experience within the curriculum may have influenced their expectations, interpretation, and evaluation of performance against the rubric criteria for a specific domain. Given this, it seems plausible that the demonstration of complex or multi-faceted competencies could be interpreted in multiple, valid ways due to the interaction of the assessor and their expertise, the student, and the domain in which they are assessed.The pattern of correlations between domain and holistic scores varied across the assessors suggesting differential emphasis of the domains in the generation of the holistic score. Assessor verbalizations and assessor backgrounds together provided some explanation of the correlation results. For example, Assessor 2 had a low correlation between the domain of Theory, Models, and/or Frame of Reference and the holistic score (r = 0.03). Assessor 2, who was early in their role as clinical faculty, verbalized not feeling confident in their scores within the domain of Theory, Models, and/or Frame of Reference. Assessor 2 did not teach the course on theory, models, and frames of reference within the curriculum which may explain the low correlation between this domain with the holistic score.Interestingly, the most experienced instructors (25+ years) in this study  demonstrated variable scoring patterns among each other. For example, Assessor 5 had a weak correlation between the domain of Professionalism (r = 0.15) and the holistic score, but a high correlation between Communication and the holistic score (r = 0.83). In a comment during the think-aloud interview, Assessor 5 remarked that they had their own “perception” for assessing students, comparing their judgement of one student against the rest, and with a large proportion of comments related to non-verbal communication and verbal explanations. This finding was also highlighted in a previous study by Govaerts et al., where they reported that experienced assessors had more distinctive judging schemas compared to novice assessors during a performance assessment. They concluded that experienced assessors were significantly triggered by task-specific cues linked to prior experience when judging performance .G = 0.93 and a phi coefficient of ϕ = 0.89 obtained, respectively.The generalizability analysis results within the holistic scoring framework showed a high proportion of variance attributable to student performance (81%) and a relatively lower proportion of variance attributable to rater (7%). The findings from the think-aloud interviews support the generalizability analysis. Findings from the think-aloud interviews showed that assessors were generally consistent in discriminating between good and weaker performances with agreement on observable indicators of performance. This relative consistency across assessors is reflected in the lower variability components attributable to raters. However, differences in how holistic scores were assigned were also noted in assessor verbalizations and aligned with the results from the correlation analysis supporting the variance component estimate attributable to the interaction-error term at 12%. The contributions of the individual domains to the overall holistic score differed across assessors or assessors took an impressionistic approach in assigning the holistic score. Taken together, findings from the think-aloud interviews support the high reliability value reported with a generalizability coefficient of p x d x r, e) at 29%. The presence of this sizeable interaction term indicates that the relative ranking of student performance by scores was dependent to some degree on the domain in which they were assessed and the assessor involved. The variability among assessors’ performance expectations likely influenced the observations and interpretations made during the assessment task facilitating potentially idiosyncratic application of the scoring rubric across students. This interaction between student, assessor, and rubric application is captured in the presence of the non-negligible interaction variance components  from the generalizability analysis. Taken together, findings from the analysis of assessors’ verbalizations support the reported reliability values for analytic scoring as lower in comparison to holistic scoring with a generalizability coefficient of G = 0.80 and a phi coefficient of ϕ = 0.77.The picture for analytic scoring is more complicated. Student variation accounted for the largest proportion of variance at 40% and rater variation (collapsed across domains and students) was 4%. Similar to the findings of holistic scoring, the qualitative findings support some level of consistency across assessors in the general understanding of the domains with an apparently shared understanding of performance indicating effective performance. The second-largest contributor of variance after student performance was attributed to the highest order interaction-error term . Assessor verbalizations demonstrated general understanding of the domains despite rarely referencing the rubric criteria explicitly in their assessment and scoring of performance. Additionally, assessors had variable performance expectations, influenced by their frame of reference or educational perspective. Variability was also noted in analytic scoring, where verbalizations of assessor pairs viewing the same student performance resulted in different judgements and assigned scores. Taken together, the qualitative findings supported the non-negligible variance component attributable to the highest-order interaction-error term, suggesting that despite participation in assessor training, variability remains in assessors’ cognition during the assessment and scoring of student performance.Verbal analysis of assessors’ cognitions provided insight into the quantitative results, specifically the reliability estimates obtained and explanation for the assessor (rater) interaction terms. For holistic scoring, assessors as a group were generally consistent in their characterizations of stronger and weaker performances and their ability to distinguish between the two. The generation of the holistic score varied between assessors with differential emphasis of the domains to the holistic score, as supported by the correlation results, or the use of a global, impressionistic approach. These findings supported the relatively high reliability of holistic scores with the largest proportion of variance attributable to student differences on the assessed competency. In comparison to holistic scores, the reliability for analytic scores was lower. Student differences accounted for the largest proportion of variance but was then followed by the highest-order interaction-error term (i.e., The results of this study are consistent with previous studies examining assessor cognition. In a similar study design, Naumann et al.  examinedThe use of concurrent and retrospective think-aloud interviews to investigate assessor cognition was also used in a study by Yeates et al.  investigThe findings of this study are interpretable from the measurement and contextual perspectives on assessor cognition . ExaminiInteractions involving assessors may suggest potential issues with fairness, where a student’s score may be dependent on the assessor observing them and the domain in which they are being assessed. The verbalizations indicated assessors approached the assessment task with variable frames of reference or perspectives which influenced expectations for performance. It may be worthwhile to examine the current approach to assessor orientation and implement procedures where assessors discuss, or are shown, what student performance “looks like” with regards to each of the domains as described by the performance descriptors. This approach is aligned with the “frame of reference” rater training as described by Holmboe .The contextual perspective on assessor cognition is where assessors are viewed as experts where differences represent legitimate and valid judgments . In thisNotably, in the context of the occupational therapy program within which this study is situated, different perspectives among assessors were seen positively by some. Occupational therapy models of practice embrace a contextual perspective. A client’s lived experience of health and participation results from a unique interaction between themselves, their everyday activities, and the environment in which they function. Tensions may arise with occupational therapy faculty as assessors, espousing a situated worldview and engaging in forms of assessment that prioritize standardization and reduced measurement error. Further exploration of assessment beliefs and faculty attitudes toward assessment would contribute to an understanding of acceptability and use of performance assessment in occupational therapy education.A limitation of this study is the recruitment of eight assessors from the original 14 to participate in the think-aloud interviews. Additionally, the procedures for the think-aloud interviews simulated the conditions of the original OSCE where assessors viewed interactions in which they participated. Due to administration and resource considerations for the OSCE, this resulted in assessor pairs viewing a different standardized patient for their assigned group of students. Therefore, the results of this study are limited to the conclusions drawn from the eight assessors under realistic conditions. Moreover, the institutional background or type of program in which the assessor was educated and its impact on assessment was not examined. The study results may not generalize to OSCEs in other professional programs. However, professional programs might find the results useful for understanding assessors’ thinking and its influence on the scoring of performance against criteria within a rubric in real-time.This study contributes to health professions education by providing insight into the nature and extent of variability among assessors in performance assessment from two perspectives. In particular, this study investigated assessors’ cognition, though the use of think-aloud interviews, as a contributing source of variation to observed score variation within the context of an OSCE within occupational therapy education. An integrated understanding is important to elucidate the meaning underlying the numerical score and because the defensibility of inferences made about students’ proficiencies rely on score quality, which in turn relies on expert judgements. This study contributes to scholarly discourse on seemingly incompatible perspectives on assessor variability – as error, consistent with the measurement perspective, or as expected, consistent with the contextual perspective. In the context of occupational therapy education, the suggested next stage of inquiry is to explore relationships between assessors’ own professional values and their beliefs and attitudes towards performance assessment, particularly the acceptability and use of standardized forms.