For effective virtual realities, “presence,” the feeling of “being there” in a virtual environment (VR), is deemed an essential prerequisite. Several studies have assessed the effect of the (non-)availability of auditory stimulation on presence, but due to differences in study design , generalizing the results and estimating the effect of the auditory component is difficult. In two experiments, the influence of an ambient nature soundscape and movement-triggered step sounds were investigated regarding their effects on presence. In each experiment, approximately forty participants walked on a treadmill, thereby strolling through a virtual park environment reproduced via a stereoscopic head-mounted display (HMD), while the acoustical environment was delivered via noise-canceling headphones. In Experiment 1, conditions with the ambient soundscape and the step sounds either present or absent were combined in a 2 × 2 within-subjects design, supplemented with an additional “no-headphones” control condition. For the synchronous playback of step sounds, the probability of a step being taken was estimated by an algorithm using the HMD's sensor data. The results of Experiment 1 show that questionnaire-based measures of presence and realism were influenced by the soundscape but not by the reproduction of steps, which might be confounded with the fact that the perceived synchronicity of the sensor-triggered step sounds was rated rather low. Therefore, in Experiment 2, the step-reproduction algorithm was improved and judged to be more synchronous by participants. Consequently, large and statistically significant effects of both kinds of audio manipulations on perceived presence and realism were observed, with the effect of the soundscape being larger than that of including footstep sounds, possibly due to the remaining imperfections in the reproduction of steps. Including an appropriate soundscape or self-triggered footsteps had differential effects on subscales of presence, in that both affected overall presence and realism, while involvement was improved and distraction reduced by the ambient soundscape only. This study evaluates the effect of auditory stimuli and their influence on presence, the feeling of “being there” in the virtual environment, while the user can actually experience walking (by using a treadmill) in the virtual world. The effect of the (non-)availability of auditory stimuli on presence has already been assessed by several studies  these feelings? Or is exposing ourselves to a virtual reality rather deliberately handing over our senses, knowing it will be short-term and even the absence of sound might be acceptable?In the following, we will briefly review the few studies that have assessed the influence of sounds being reproduced in contrast to silence in their influence on presence in virtual realities.M = 3.45 (SD = 0.82) compared to M = 2.73 (SD = 0.90) in the condition with sound. Curiously thus, presence [1–5] and realism decreased when sound was added.Hendrix and Barfield  visuallyDinh et al.  visuallyLarsson et al.  visuallyThe methodological diversity of the studies reviewed and their somewhat inconsistent results make it difficult to draw any further conclusions beyond stating that it is likely that sound enhances perceived presence in VR somehow. Therefore, we decided to further investigate the issue using a modern, stereoscopic head-mounted VR display and stereophonic sound drawing on two sources: (a) environmental sounds, and (b) the self-generated sound of footsteps produced by walking in the VR.One of the first decisions to be made was how to design the sounds so they would lead to findings adding to the already existing body of research. We wanted to benefit from the “temporal nature of sound” implying that “something is happening”  than in the unimodal condition . Since this suggests that reproducing one's own step-sounds should increase presence in virtual realities, we chose step sounds as the “impulse-driven” sound events.In a study by Nordahl , 20 partRecognizing the participants' actual steps and playing back appropriate sounds synchronously poses technical challenges. Visell et al.  suggest Similar efforts had been made by Slater et al. : they trFor the present investigation, the principal research question was which sounds would enhance presence most. Would it be enough to just cancel the noises from the lab, particularly the treadmill-sounds, i.e., to isolate the user from the environment in order to increase presence in the VR? Would it be helpful, if sounds triggered by the user's footsteps on the treadmill were presented in addition to noise canceling? Would a soundscape which corresponded to and enhanced the virtual environment have a greater effect? Or would the combination of virtual steps and soundscape produce the highest degree of presence?To answer these questions we investigated participants' sense of presence while moving through a VR by walking on a treadmill. Its audio component was varied in a within-subjects design by which the presentation of footstep sounds triggered by the participant and of an environmental soundscape were factorially combined in an on/off fashion and—in Experiment 1—supplemented with a no-headphones control condition.Thirty-six participants  took part in Experiment 1, mostly psychology students with ages ranging from 18 to 47 years . All participants claimed not to have any impairment of hearing or movement. Visual impairment was tolerated up to one diopter, larger refractive errors only, if corrected by glasses or contact lenses. Due to the technical requirements of the treadmill, the participants were allowed a maximum height of 185 cm, as not to touch the front of the treadmill while holding onto the handle, and a maximum weight of 150 kg. This study was carried out in accordance with the recommendations of the Ethics Code of the American Psychological Association. The protocol was approved by the central ethics commission of TU Darmstadt. All subjects gave written informed consent.Experiment 1 was conducted as a one-factorial, within-subjects design with five levels. The type of background sound was varied see  to assesExperiment 1 was conducted in a robotics lab with other people being present and at times talking in low voices. During each trial, the participants wore an Oculus Rift DK2 HMD allowing for stereoscopical presentation of visual stimuli on two OLED displays with a resolution of 960 × 1,080 pixels per eye and a refresh rate of up to 75 Hz  and the soundscape (7 min) were played when called for and then looped, the footstep-sounds (between 0.626 and 0.783 s) were only reproduced when triggered.For timing the reproduction of the step-sounds, Caserman et al.  developeFor Experiment 1, items measuring presence were eclectically chosen from published questionnaires. Two questions assessing presence and realism were 1-item-scales taken from Hendrix and Barfield . In addiIn each experimental condition, the participant walked on the treadmill with a velocity of 2.5 km/h while seeing a virtual reality presented by an Oculus Rift HMD and wearing headphones (except in the “No Headphones”-condition). The duration of exposure to the virtual reality for a given condition was varied between 3:40 and 5:00 min to permit varying estimations of elapsed time as a dependent variable assessing a potential loss of the participant's sense of time. During the exposure, the participant was asked to count fly agarics aiming to capture the participant's attention. After each exposure to a given VR/audio condition, the participants filled out the presence questionnaire before entering the next experimental condition. The order of the experimental conditions and the duration of the trials were independently randomized by using multiple 5 × 5 Latin squares.The subjectively perceived sense of presence had been assessed by two questions from Hendrix and Barfield , the firRealism had been assessed by one question from Hendrix and Barfield  on how rt-tests were conducted to test whether the presence and realism-ratings in the condition without headphones significantly differed from those in the no-sound, noise-canceling condition, but no significant effects were found, suggesting that presence did not depend on whether the participant wore headphones or not.Welch two-sided F = 48.95, p < 0.001, with a large effect size (η2 = 0.583), while the factor “Steps” had no significant effect, F = 0.003, p = 0.96, nor did their interaction, F = 0.531, p = 0.471. The results of the presence-ratings from 1 to 7 paint a similar picture. With respect to realism, the factor “Soundscape” also resulted in highly significant differences, F = 42.48, p < 0.001, with a large effect size (η2= 0.548), while the factor “Steps” had no significant effect, F = 1.52, p = 0.226, nor had their interaction, F = 0.173, p = 0.680. These results imply that Presence and realism are increased by playing back an appropriate soundscape, but not by adding self-triggered sounds of footsteps.In order to assess the auditory effects of playing back step-sounds or adding an ambient “Soundscape,” two-way analyses of variance (excluding the “no Headphones” condition) were conducted, thereby analyzing the effects of including “Steps” (played back or not) or an ambient soundscape (on or off). For the presence-ratings from 1 to 100, the factor “Soundscape” was highly significant, t(69.96) = −2.342, p < 0.05. Playback of a soundscape increased both involvement and realism, and simultaneously decreased distraction by the lab environment . Including the sound of the participants' footsteps in the VR produced an effect of its own on the realism scale only, F = 10.409, p < 0.001; η2 = 0.229. The significant interaction observed between the presence of a soundscape and the playback of steps on the realism scale, F = 7.235, p < 0.05, η2 = 0.171, confirms the impression that providing a soundscape and adding step sounds does not increase presence in an additive fashion.The Witmer and Singer  involvemFor the Witmer and Singer  visual aF = 33.9, p < 0.001, with a large effect size (η2 = 0.478). Pairwise comparisons using paired t-tests with Bonferroni corrections clarified that there are significant differences between the Steps condition and the two conditions involving a soundscape, the latter ones being rated significantly higher in audio quality. In conclusion, the differences are due to adding the soundscape whereas adding step sounds did not result in significant effects.For the Witmer and Singer  auditoryM = 2.33 (SD = 1.57), in the “Steps & Soundscape” condition it was M = 2.81 (SD = 1.56), thus in both cases only slightly below the midpoint of the response scale. To evaluate, whether the shortcomings in the playback of step sounds contributed to reducing presence, correlations between the synchronicity ratings and various measures of presence and its sub-dimensions were computed. The strongest correlation was observed between synchronicity and the Hendrix and Barfield  = 3.03, p < 0.01, suggesting that the more the step sounds were perceived as synchronous with one's own gait, the more realistic the environment was perceived. Slightly weaker correlations were obtained with involvement in the VR, r = 0.25, t(70) = 2.19, p = 0.03, and with distraction by extraneous events r = −0.24, t(70) = −2.0814, p = 0.04.To evaluate the step reproduction algorithm, participants also assessed whether they felt their actual steps and the steps they heard in the virtual reality to be synchronous by making a rating on a seven-point scale labeled at the extremes from “not at all” (synchronous) to “very” (synchronous). Coding the response categories as 0–6, in the “Steps” condition, the average synchrony rating was Barfield  single-ir = 0.224, t(178) = 3.068, p < 0.01, suggesting that the more participants felt they lost track of time, the longer they estimated their time spent in the VR.Many studies assess whether the participant has the feeling of losing track of time, so we asked the participants to estimate the time they had spent in the VR in a given condition and calculated the difference to the actual time elapsed. In general, the results show that there was a tendency to underestimate the time spent in the virtual reality. The means differed considerably by condition, but so did their standard deviations, thus failing to produce significant differences in time estimation accuracy between conditions. Interestingly, however, a small but significant Pearson product-moment correlation between the Witmer and Singer  questionTwo questions assessed if the participants heard sounds from the laboratory environment or the virtual environment, respectively. Participants reported hearing the noise of the treadmill, their own steps, technical sounds , or human-produced sounds . While there were only few hearing nothing at all in the conditions “No Headphones” and “Noise-Canceling,” that number increased to 12 participants in the “Steps” condition, 20 in the “Soundscape” condition and 18 in the “Steps and Soundscape” condition. That suggests that reproducing sounds like steps or a soundscape in itself serves to block out auditory distractors from the actual surrounding.With respect to hearing sounds in the virtual environment, the perception was quite rich, including wind noises, wings flapping, distant thunder, the humming of flies and insects and the chirping of crickets. One participant even reported hearing a chainsaw, although there was no such sound in the soundscape.Experiment 1 demonstrates that the reproduction of an appropriate soundscape along with the visual VR has highly significant effects on Presence, Realism, Involvement, Distraction and  the “Auditory aspects” of exposure to the virtual world, all with large effect sizes. Note, that the baseline condition without headphones produced an average presence rating  of 44 in comparison to a mean of about 70 for the conditions with soundscapes, thereby indicating that presence is considerably enhanced if the sensory information about the virtual environment is delivered in more than one modality. Of course, that also implies that the presence in the virtual reality is only about 70% of the presence which is perceived when interacting with a comparable real-world scenario.Reproducing self-generated footstep sounds, however, neither produced significant effects on overall measures of presence, nor on its subscales. The only exception is the realism scale, suggesting that providing proprioceptive audio feedback may increase the authenticity of the virtual world.In general, whether (noise-canceling) headphones are worn or not reduces distraction by the actual (laboratory) environment. That is consistent with responses to the open question on whether sounds were heard or not: reproducing sound like steps or a soundscape in itself helped to block out auditory distractors from the actual surrounding. Presence, involvement and realism, however, are not influenced by just wearing headphones. Thus, the active noise canceling only diminished distraction.The time estimation task showed a general tendency to underestimate the time spent in the VR, but notably, although we offered spaces for entering minutes and seconds, nearly all participants rounded their estimates up or down to full or half minutes which is problematic because it diminishes resolution. That suggests looking for better ways to investigate the sense of losing track of time.As to the authenticity of playing back user-triggered step sounds in the VR, the ratings of synchronicity of the steps were rather disappointing, ranging between 2 and 3 on the 0–6 scale, although it is evident that the reproduction of a soundscape enhanced the perceived synchronicity of the steps. Still, the algorithm calculating the steps should be improved in further studies.Although adding sounds to the visual virtual reality did greatly enhance presence for the conditions involving soundscapes, there was no significant increase in presence due to the presentation of step-sounds synchronized with the participant's gait on the treadmill. That might be due to the fact that the algorithm reproducing the steps is still suboptimal, generating gaps in the step sound presentation which are perceived to be asynchronous. Therefore, the step-reproduction algorithm was modified in order to re-assess the interaction of the availability of an ambient soundscape and the presentation of self-generated footsteps with an improved experimental setup.Furthermore, the eclectic use of items and subscales from existing questionnaires may be problematic due to the unknown reliability and validity of the selection. Therefore, in Experiment 2, a full-length, psychometrically evaluated questionnaire was employed  with ages ranging from 18 to 34 years . The participants were all students, most of them of psychology. With respect to their usage of HMD's, 50% had never used one, another 45% had used an HMD between one and three times and only two participants had worn it 10 or 30 times, respectively. All participants signed an informed consent form explaining the study and informing them about their right to end participation at any time. The protocol was approved by the central ethics commission of TU Darmstadt. All subjects gave written informed consent.The second experiment was designed as a two-factorial, within-subjects design with two levels per factor see . The twoThe procedure, the Virtual Reality and the sounds used (Steps and Soundscape) were the same as in Experiment 1. The experimental setup was changed insofar as the second experiment was conducted in a windowless laboratory room with hardly any background noise, except for the constant hiss of the air condition, which was almost completely suppressed by the noise-canceling headphones.Furthermore, small changes were made to the step-sound algorithm as to enhance user perception. The position was now weighted 100% which means, the data from the position detector alone were used to estimate the execution of a step. The other detectors  were no longer taken into account. Furthermore, a step was always triggered when the momentary duration of a step exceeded the medium duration of previous steps in order to avoid gaps in step reproduction.The focus of Experiment 2 was to measure changes in presence and realism using an established full-length presence questionnaire. We chose to use the IPQ  = 5.821, p = 0.021, η2= 0.130, and the factor Soundscape, respectively, F = 24.512, p < 0.001, η2 = 0.386. The interaction of both factors was not statistically significant, F = 0.831, p = 0.368.Responses to the IPQ single-item question show that presence, the impression of “being there” in the computer-generated environment, rated on a scale from −3 to +3 increases from a minimum in the “Noise-canceling” control condition, over the “Steps” condition up to the maximum reached in the “Soundscape” and “Steps & Soundscape” conditions see . A two-fF = 11.797, p = 0.001, η2 = 0.232, as well as of soundscape, F = 41.518, p < 0.001, η2 = 0.516. The interaction of steps and soundscape, F = 5.361, p = 0.026, η2 = 0.121, was also statistically significant, suggesting that the introduction of step sounds had a greater effect on presence when it was the only auditory enrichment provided, as compared to when it was combined with an ambient soundscape.The single-item question from Hendrix and Barfield  1: “no . A two-fThe mean of four items each was calculated for the IPQ-subscales “Realism,” “Spatial Presence,” and “Involvement.”F = 12.939, p < 0.001, η2 = 0.249, and the factor Soundscape, respectively, F = 20.185, p < 0.001, η2 = 0.341. The interaction of both factors was not statistically significant, F = 0.854, p = 0.361.With respect to the Realism of the VR, the ranking is the same as for the presence scales see : the lowAs F = 0.430, p = 0.516, while revealing a significant main effect for the factor Soundscape, F = 36.983, p < 0.001, η2 = 0.487. The interaction of both factors was not statistically significant, F = 0.026, p = 0.874.On the involvement scale the means for the conditions without soundscape are about the same, as are the means for the conditions providing a soundscape see . ConsistAs F = 18.6, p < 0.001, η2 = 0.323. A pairwise t-test with Bonferroni correction clarifies that the difference is between the Steps-condition and the two conditions with Soundscape, the latter not differing significantly from each other. That suggests that replaying a soundscape affects auditory perception whereas replaying steps does not.By contrast, on the auditory aspects scale the mean ratings tend to increase when a soundscape is added. A one-factorial analysis of variance shows that the auditory aspects ratings distinguish the three sound conditions, M = 3.225 (SD = 2.069) on the 7-point Likert scale, in the “Steps & Soundscape” condition as M = 3.675 (SD = 1.760), slightly above the midpoint of the scale and approximately half a response category higher than in Experiment 1. A Welch two-sample t-test confirmed that difference to be statistically significant, t(148.7) = −3.103; p < 0.01. As in Experiment 1, a significant correlation between the rated synchronicity of the step sounds and the perceived realism of the VR was found, r = 0.40, t(78) = 3.89, p < 0.001, but in Experiment 2, synchronicity was also related to overall presence: A significant correlation was found with the 1–100 presence scale, r = 0.28, t(78) = 2.61, p < 0.05, and marginal significance when correlating synchrony ratings with the single-item presence scale of the IPQ.As in Experiment 1, participants rated whether they felt their actual steps and the steps they heard in the VR to be synchronous. In the “Steps” condition, on average, the synchrony was rated as In Experiment 2, the variation of essentially the same auditory features as in Experiment 1  but with an improved step-reproduction algorithm produced highly significant perceptual effects of both features, where in Experiment 1 only the soundscape had been influential.That was true for the sense of presence experienced in the VR as well as for its perceived realism, while the IPQ subscale of spatial presence was not affected, and the IPQ subscale of involvement only reflected the auditory enrichment by the soundscape.The latter is somewhat surprising, since hearing one's own steps in the VR is intuitively highly likely to contribute to involvement. Note, however, that the step algorithm is still far from perfect as reflected in the low synchronicity ratings. That the spatial presence subscale was not sensitive to our manipulations of the auditory environment may be due to the fact that the sounds, though being somewhat localizable by the mechanisms provided in Unity (as with the sound of church bells waxing and waning as the listener passed by), were not specifically designed to provide directional audio cues, nor did the participants' task require the use of spatial auditory information. Rather, the soundscape used in the present research delivered diffuse, ambient information on sound sources , much less than on their specific location.In two experiments, participants moved on a straight gravel path through a park-like virtual reality presented by a HMD while actually walking on a treadmill. During that experience the visual VR was supplemented with different types of audio exposure in order to study to what extent auditory stimulation enhances perceived presence.An initial question for this research was, whether pure noise-canceling without reproduction of sound would lead to mild feelings of isolation or “deadness” as reported by Ramsdell's . As showThe variation of the sounds presented via headphones to complement the VR had a strong influence on perceived presence, realism and the auditory aspects scale, so presenting a plausible soundscape or self-triggered step sounds evidently made participants feel more present and the virtual reality appear more real. Notably, the effect of including the soundscape was generally larger than the effect of adding step sounds which, for some measures, even turned out to be insignificant in Experiment 1.As regards retrospective ratings of presence, in conditions with the visual VR only, and without sound, people experienced between 40 (Experiment 2) and 54 (Experiment 1) percent of real-world presence in our virtual “walk-in-the-park” scenario. By contrast, in both experiments, presence ratings increased by some 15–20 percentage points when both step sounds and the ambient soundscape were added. In general, ratings on all quality-related scales in the Steps & Soundscape condition were a little lower in the second experiment compared to the first which might indicate that the second sample was more skeptical, though that is not supported by discrepancies in their previous exposure to VR, for example.Notably, spatial presence, the feeling that one is really located and acting in the virtual world and that it is more than just watching a movie, was not influenced by the sounds at all in Experiment 2. Further research might investigate whether achieving high spatial presence is simply difficult, or whether our failure to provide it is due to the particular sounds chosen.The perceived synchronicity of the steps was disappointingly low in both experiments, though a simplification of the step-reproduction algorithm slightly improved synchronicity ratings in Experiment 2. Interestingly, in both experiments, the perceived synchrony of the steps appeared to have an effect on participants' sense of presence, particularly as measured by the “realism” subscales. It is conceivable that the mechanism by which ill-fitted step sounds reduce presence is the participant's sense of agency, the perception of “being in action or of exerting power”  effect sizes, there is not much published evidence yet, but in the present study, reproducing a soundscape influenced different aspects of presence with medium to large effect sizes ranging from about η2 = 0.40 to 0.72, whereas reproducing step-sounds alone resulted in effect sizes ranging from about η2 = 0.13–0.25. It should be noted, though, that due to the imperfections of the step reproduction procedure, the effect of adding self-triggered step sounds may be underestimated by the present research.For integrating our study into the already existing body of research, we compared our results regarding presence and realism to those of earlier studies: mean scale values and effects sizes  the research design employed, (2) the method of measuring presence, and (3) the implementation of audio enrichment chosen.In comparing the different audio conditions, we chose to use a within-subjects design, where each participant experiences and rates each of the 4–5 audio conditions. That will undoubtedly reduce error variance due to inter-individual differences in scale usage, it is likely to enhance contrast between conditions , but on the downside exposure to all audio conditions will increase the transparence as to what is being investigated, entailing the risk of encouraging participants to comply with the presumed research goals. That the results were replicated with two different samples, may serve to somewhat weaken that potential criticism. Nevertheless, replicating the present study using a between-subjects design and/or objective measurements might be called for.As to the latter point, note that for all dependent measures, we used questionnaires to be completed retrospectively, after the actual experience. Therefore, less subjective and more continuous measurement methods are highly desirable, for example continuous measurement of presence while the participant is exposed to the virtual world. A better way to measure the phenomenon of losing track of time might also be of interest, as are objective/psychophysiological indices of presence  Sensor-triggered, extraneous footstep sounds synchronized to the actual gait of the participant, and (b) a diffuse, globally enveloping nature soundscape providing a background “sound texture” rather than discrete auditory events. The null effect of our audio manipulations on the “spatial presence” scale and the rather small effects on “involvement” (Experiment 2) still have to be explained. Future studies might explore sounds that are more useful for orientation in the VR, or even instrumental in solving a spatial task, potentially including the “signal” or even “social” levels of usage of hearing postulated by Ramsdell  over andLikewise, in the present study, the placement of sounds in the environment was accomplished employing the basic methods supplied by the Unity software used to construct the VR. Future research might explore more sophisticated methods of rendering audio in VR, such as wave field synthesis . The patients/participants provided their written informed consent to participate in this study.AK and WE contributed jointly to conception and design of the study, contributed to the statistical analysis and contributed to manuscript revisions, and read and approved the submitted version. AK performed the literature search, set up the visual and auditory VR, and collected the data and wrote the first draft of the manuscript. WE revised and complemented the manuscript.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


The principle of workplace based assessment (WBA) is to assess trainees at work with feedback integrated into the program simultaneously. A student driven WBA model was introduced and perception evaluation of this teaching method was done subsequently by taking feedback from the faculty as well as the postgraduate trainees (PGs) of a residency program.Descriptive multimethod study was conducted. A WBA program was designed for PGs in Chemical Pathology on Moodle and forms utilized were case-based discussion (CBD), direct observation of practical skills (DOPS) and evaluation of clinical events (ECE). Consented assessors and PGs were trained on WBA through a workshop. Pretest and posttest to assess PGs knowledge before and after WBA were conducted. Every time a WBA form was filled, perception of PGs and assessors towards WBA, time taken to conduct single WBA and feedback were recorded. Faculty and PGs qualitative feedback on perception of WBA was taken via interviews. WBA tools data and qualitative feedback were used to evaluate the acceptability and feasibility of the new tools.n = 7 and PGs n = 6), 12 ECEs (assessors n = 6 and PGs n = 5), and 20 DOPS (assessors n = 6 and PGs n = 6) were documented. PGs average pretest score was 55.6%, which was improved to 96.4% in posttest; p value< 0.05. Scores of annual assessment before and after implementation of WBA also showed significant improvement, p value 0.039, Overall mean time taken to evaluate PG’s was 12.6 ± 9.9 min and feedback time 9.2 ± 7.4 min. Mean WBA process satisfaction of assessors and PGs on Likert scale of 1 to 10 was 8 ± 1 and 8.3 ± 0.8 respectively.Six eligible PGs and seventeen assessors participated in this study. A total of 79 CBDs (assessors Both assessors and fellows were satisfied with introduction and implementation of WBA. It gave the fellows opportunity to interact with assessors more often and learn from their rich experience. Gain in knowledge of PGs was identified from the statistically significant improvement in PGs’ assessment scores after WBA implementation. Chemical pathology encompasses both practical laboratory and clinical skills. According to the guidelines of the College of Physicians and Surgeons of Pakistan (CPSP) 4 years’ training is required to satisfactorily complete the chemical pathology curriculum to the required depth and breadth. By the completion of training, the postgraduate (PG) trainees are expected to develop the desired competencies in the six major domains; medical expert, inter-professional communication, system based practice, professionalism, practice based learning and improvement and communication skills for independent practice. Interpretation and reporting is an important aspect of PG learning in chemical pathology –3. This A virtual learning environment (VLE) is a distance learning platform for synchronous or asynchronous teaching that integrates course materials, assignments, assessments, and other tools –15. For A descriptive multi methods study was conducted in the section of chemical pathology, Department of Pathology and Laboratory Medicine, Aga Khan University (AKU), Karachi, Pakistan. The study was conducted from January to December 2019 and was introduced in the training process of chemical pathology residency program. The chemical pathology training program is a 5 year residency at Aga Khan University; accredited by CPSP. The program is a self-directed adult learning, systemically organized in the form of week by week structured bench rotations for learning analytical skills and rotations for enhancing their clinical skills in practice of biochemical laboratory medicine to diagnose disease and to manage patients. The PG trainees must acquire a detailed understanding of biochemical processes and the changes that occur in various diseases while rotating through the benches/ subsections of chemical pathology which include the following: routine chemistries and enzymes; therapeutic drug monitoring and toxicology, immunoassays, biochemical genetics laboratory, urolithiasis laboratory service, point of care testing, electrophoresis and protein laboratory. A major part of fellows’ time is spent in advising other clinicians about the appropriate tests for the investigation of a particular clinical problem, the interpretation of results and follow-up, performing procedures, provocative tests and evaluating new tests for introduction in service, assay optimization and validation and research.Before initiating the study the university’s ethical review committee’s permission was sought. All PGs registered at CPSP, including fellows of chemical pathology training program at AKU and fellows rotating in section of chemical pathology from other pathology specialties from January to December 2019 were eligible to participate in the study. Faculty, senior technologists and managers at the section of chemical pathology were taken as assessors. Seventeen assessors and six eligible PGs were contacted for written informed consent and all agreed.To ensure that the outcomes achieved at the end of each phase meet their purpose, and that WBA program team members are properly prepared for the next phase we divided the WBA project into phases. To deliver the WBA program effectively the entire study process was divided into following seven phases:The chemical pathology residency program has a different teaching learning and training environment which includes report interpretations and correlating with clinical data, clinical audits, clinical consultations, and laboratory procedures, provocative testing, quality assurance measures and complaint handling. Therefore it was essential to know that the selected WBAs tools were fit for purpose. Using Norcini AMEE guide, validated WBA tools were chosen and modified according to the needs of chemical pathology residency program by WBA project team . These tThe process of WBA was clearly delineated with the goals of making it student driven and making it instrumental in the provision of feedback to PGs to navigate their learning towards desired outcomes. The PG was supposed to select a case (for CBD), any laboratory procedure (for DOPS) or complaint/ clinical audit (for ECE) which he/she has recently been involved with. The WBA process was to be initiated by PGs and they were supposed to select the assessor of their choice. For DOPS assessor was supposed to be present during the demonstration of the procedure by the PG. The discussion will start from and be centered on the PG’s record in the notes or reports and is designed to assess clinical decision-making and the application or use of medical knowledge in the care of patients. The discussion of assessors and PG would be followed by immediate feedback on assessment form uploaded on Moodle in another 5–10 min in the presence of the PG trainee. The assessments were to be performed against the standard expected at the end of the PG’s current stage of training and according to the level of complexity for each assessment. Guide for assessing complexity of cases, procedures or clinical events were prepared for the assessors. Low complexity WBAs were those that were uneventful and straight-forward with few demands made on the PG trainees, average complexity WBAs were those with manageable complications that most likely occur on a regular basis and high complexity WBAs were those that were difficult or unusual due to demanding encounters or unusual findings.The primary goal of the assessment was to provide relevant comprehensive feedback to the PGs. Prior to submitting any of the three WBA tools PGs and assessors’ satisfaction level with the overall WBA process was recorded on VLE. Satisfaction level of both PGs and assessors was gauged separately on a Likert scale of 1–10 every time a WBA tool was filled. To maximize the educational impact of WBA, assessors were invited to provide qualitative comments for PGs on their overall performance in CBD, DOPS or ECE.For WBA program structure, activities, management and application Moodle (free of charge resource) was selected as the VLE platform. With support from the department of I.T. Academic and Computing, the WBA tools finalized in phase I of the study were developed on the VLE and all essential documents including WBA process and assessors’ guide were uploaded Table . DescripA half day hands-on WBA workshop to prepare the assessors to use the VLE for WBA was designed and executed face to face to 15 participants. Thorough hands on training through this workshop of all assessors was done with added objective of controlling rater bias. In-depth briefing regarding the entire process of WBA was given by the principal investigator (PI) and CO-PIs through various activities to determine the strengths, weaknesses, opportunities and threats (SWOT) of introducing WBA, along with hands-on use of WBA tools  by role play. The SWOT analysis on WBA project was done in flipchart activity format with assessors by dividing them into four groups. Preliminary results from SWOT analysis were then detailed and the session invited comments and feedback using post-it’s to elaborate as described in Table A full hands-on workshop was conducted by experts from department of I.T. Academic and Computing to prepare the assessors to use the VLE for WBA seven master trainers. The workshop was a complete training session with hands on practice of using Moodle on individual laptops. Following the TOT methodology the master trainers, given the title of ‘Moodle Champs’, then trained the rest of the assessors in the Section by small group discussions. They demonstrated how to use VLE for WBA in multiple small group discussions.PG trainees were introduced to WBA with a presentation and eight to ten small group discussions on WBA and the use of VLE for teaching and learning. The whole WBA theory was explained and they were demonstrated the overall process of WBA. It was explained that it was for their benefit and to provide regular timely feedback to them. It was reiterated that in order to close the gap between actual performance and desired performance they need to actively take action and improve. Blueprint of the course curriculum was made available on VLE for them with the desired outcomes.Once WBA tools were developed, validated and tested on VLE, all assessors and PGs were trained and the process was clear to all participating the WBA process was implemented. Pretest of all PGs participating were conducted before WBA implementation and posttest was taken after 3 months of participating in WBA program. Being convenient, cost effective with the ability to cover broad content, multiple choice questions (MCQs) were chosen as the method of assessment for both pre and posttests. The MCQs based on CPSP chemical pathology residency program curriculum were developed by two content experts and uploaded on VLE. All 20 questions developed were scenario based and were structured for a higher cognition level of analyzing and interpretation. Data gathered during WBA execution were PGs performance through WBA tools, time spent in each WBA feedback and discussion and pretest and post test scores. Results of annual assessment before (year 2018) and after implementation (year 2019) of WBA were also evaluated to see the effect of WBA in PGs knowledge. Both type of assessments involved the same batch of PG trainees. The annual assessments of 2018 and 2019 were similar in content and structure using same table of specifications of chemical pathology course by paper setters. The assessments were blueprinted to the chemical pathology curriculum, ensuring that there was extensive assessment coverage of the curriculum content.Semi-structured individual interviews were conducted with randomly selected five assessors and PG trainees to gauge their response about the advantages of WBA, challenges faced and how they can be resolved. A quiet and convenient venue was chosen for the interviews. The interviews, which lasted 30 min to 1 h each, were conducted by one interviewer using a semi-structured interview format to promote a discussion and further questions. The interviewer, guided by the interview format, could direct questions in such a way as to facilitate more details and insight on the PG trainees’ and assessors’ responses. All questions were open ended. The main questions asked were as follows: how was their experience of using VLE for WBA and how could WBA program be improved? Data via interviews was being collected simultaneously from two sources; PG trainees and assessors. Data collection and analysis occur simultaneously so that the analyzed data guided subsequent data collection. The interviewer did not record the interviews but made written notes which were supplemented by extensions on those notes immediately following the end of the interview.P value < 0.05 was considered significant.All data from recorded WBA on VLE were extracted into excel spreadsheets. Mean ± SD were generated for quantitative data (pre and posttests scores). Pre and posttests score comparison and comparison of results of annual assessment and after implementation of WBA were done using paired sample t test. For qualitative data frequencies were calculated. The feedback interviews were conducted and analyzed using the constructive grounded theory approach . The notn = 7 and PGs n = 6), 12 ECEs (assessors n = 6 and PGs n = 5), and 20 DOPS (assessors n = 6 and PGs n = 6) were successfully recorded.Six eligible PGs and 17 assessors participated in this study. A total of 79 CBDs (assessors n = 79), 7 assessing components were marked ‘borderline’ while 2 components were found to be ‘below expectations’ at various events. PGs were at par in most of the discussions. Suggestions for development were provided in 49.3% (n = 39) of CBDs  recorded dealt with laboratory data interpretation followed by analytical  and preanalytical  issues. From the total cases in CBDs, 22.4% (n = 13) were of high complexity, 72.4% (n = 42) moderate and 5.2% (n = 3) of low complexity. Case mix included electrolyte imbalance, proteinuria, renal stones and renal failure, endocrinopathies , autoimmune disorders, hypervitaminosis D, dyslipidemia, liver and pancreatic insufficiency, sepsis, malignancies and rare diseases . The average time taken for CBDS’ discussion and feedback were 11.2 ± 5.3 min and 9.2 ± 7 min respectively. Out of the total CBDs conducted 41.7% (n = 33) got over within 20 min.The CBDs most frequently focused on clinical and laboratory data on an outpatient record  or inpatient record , and internal and external quality control summaries were also used. During the CBDs . Out of the total ECEs (n = 12) conducted, 50% (n = 6) were regarding consultation on laboratory data, 42% (n = 5) on complaint handling while no clinical audits were documented. All complaints or consultations discussed in ECE were of moderate complexity  to high complexity . Seven components of ECEs were marked ‘borderline’ at several times. Topics discussed in ECE were autoimmune disorders, anemia, malignancy, endocrinopathies, quality control and proficiency testing survey data and prenatal screening panel. Average time taken for discussion and feedback of ECEs were 8.7 ± 3.2 min and 7.7 ± 4.2 min respectively. Majority  of the time assessors and PGs got over with the discussion and feedback over within 20 min.The ECEs were most frequently conducted patients with outpatient record (n = 14) were on biochemical analysis and instrument handling while 30% (n = 6) were on procedural skills. Six components of DOPS were marked ‘borderline’ several times. None of these WBA demonstrated evidence of ‘below expectation’ by the assessors. Instruments on which DOPS was demonstrated were ion selective electrode analyzer, point of care testing, centrifuges, Fourier transformed infrared spectroscopy and biochemical methods discussed were stone analysis, arterial blood gases analysis, and protein electrophoresis. Additionally there were DOPS carried out on safety measures to be taken when there is a biological and chemical spill. Majority  of the time assessors and PGs got over with the observation of DOPS, discussion and feedback within 20 min.Out of the 20 recorded DOPS, 70%  indicates that WBA can be successfully integrated in a chemical pathology postgraduate training program. Further steps are to be taken to align WBA with the chemical pathology curriculum, EPAs and make the process acceptable at a national level. In short WBA implemented in chemical pathology had a catalytic effect creating more learning opportunities for the PG trainees and improving teaching and learning environment in the section.Additional file 1. Annexure 1 a. WBA form for Case-based discussion (CBD). b. WBA form for Direct observation of practical skills (DOPS). c. WBA form for Evaluation of clinical events (ECE).


Healthcare policy-makers are expected to develop ‘evidence-based’ policies. Yet, studies have consistently shown that, like clinical practitioners, they need to combine many varied kinds of evidence and information derived from divergent sources. Working in the complex environment of healthcare decision-making, they have to rely on forms of  knowledge quite different from that produced by researchers. It is therefore important to understand how and why they transform research-based evidence into the knowledge they ultimately use.We purposively selected four healthcare-commissioning organisations working with external agencies that provided research-based evidence to assist with commissioning; we interviewed a total of 52 people involved in that work. This entailed 92 interviews in total, each lasting 20–60 minutes, including 47 with policy-making commissioners, 36 with staff of external agencies, and 9 with freelance specialists, lay representatives and local-authority professionals. We observed 25 meetings (14 within the commissioning organisations) and reviewed relevant documents. We analysed the data thematically using a constant comparison method with a coding framework and developed structured summaries consisting of 20–50 pages for each case-study site. We iteratively discussed and refined emerging findings, including cross-case analyses, in regular research team meetings with facilitated analysis. Further details of the study and other results have been described elsewhere.The commissioners’ role was to assess the available care provision options, develop justifiable arguments for the preferred alternatives, and navigate them through a tortuous decision-making system with often-conflicting internal and external opinion. In a multi-transactional environment characterised by interactive, pressurised, under-determined decisions, this required repeated, contested sensemaking through negotiation of many sources of evidence. Commissioners therefore had to subject research-based knowledge to multiple ‘knowledge behaviours’/manipulations as they repeatedly re-interpreted and recrafted the available evidence while carrying out their many roles. Two key ‘incorporative processes’ underpinned these activities, namely contextualisation of evidence and engagement of stakeholders. We describe five Active Channels of Knowledge Transformation – Interpersonal Relationships, People Placement, Product Deployment, Copy, Adapt and Paste, and Governance and Procedure – that provided the organisational spaces and the mechanisms for commissioners to constantly reshape research-based knowledge while incorporating it into the eventual policies that configured local health services.Our new insights into the ways in which policy-makers and practitioners inevitably transform research-based knowledge, rather than simply translate it, could foster more realistic and productive expectations for the conduct and evaluation of research-informed healthcare provision. We have long known that knowledge developed on the ‘high-ground’ of scientific research differs markedly from that used in the ‘swampy lowlands’ of practice . Terms sThe evidence-based healthcare movement has heightened the problem of ensuring that scientifically derived evidence is reliably passed from the researchers who produce it to the policy-makers, managers, clinicians or service users who apply it. Yet, there is a growing recognition that it is naive to envisage this as the simple transfer of knowledge across a research–practice ‘gap’ . Hence, the messy engagement of multiple players with diverse sources of knowledge”. This statement encapsulates two crucial principles. Firstly, the uptake of new evidence is not a linear journey but a multidirectional, multifaceted one – a phenomenon that has been ignored, denied or obscured by over-simplified linear models of knowledge translation  “Views varied:But, you know, what do you do with that data? We know that it must be saying something to us, the fact that a little old lady has had three falls ... But it doesn’t tell us that she’s somebody that necessarily needs to be assessed by the team. … We’ve now got a waiting list for people to be assessed, and half of them, I suspect, will be assessed and it will be decided that they hadn’t really got a big problem at all.” (GP commissioner)“Contextualisation repeatedly appeared in the Product Deployment ACKT, as the knowledge and assumptions built into software tools were often based on scientifically derived evidence or expert consensus from abroad, which needed ‘Anglicising’. Without contextualisation, whether by the external agency or the commissioners themselves, decision-makers tended to dismiss such tools – as happened in PL-Audit (Box 3), where the application of the embedded evidence needed not only major reconfiguration for the United Kingdom context but also specifically for the local hospital . Even then, commissioners needed advice on applying the local data outputs to their decisions; ideally, they needed access to known and trusted ‘interpreters’ . In one commissioning organisation, an external analyst regularly attended meetings (People Placement ACKT) to present a dashboard and work through the local implications of the data with committee members. However, elsewhere, a commissioner worried that without a trustworthy interpreter the wrong conclusions could be drawn:Contextualisation was central to the Copy, Adapt and Paste ACKT but also manifested itself through the Interpersonal Relationships and People Placement ACKTs, where through relationship-based mingling of knowledge and expertise, research-based evidence could be transformed into a more readily acceptable form. Training on decision-support tools alone did not suffice. The outputs had to be interpreted and contextualised by those who understood the tools  and passed on to the ‘decision-makers’ who could digest and apply local information, usually after a further round of contextualisation that could include demographic, clinical, political, organisational or financial considerations. When applying national and regional mandates to local circumstances, contextualisation within the Governance and Procedure ACKT entitled local stakeholders to object; this could result in modification or rejection of the mandates, regardless of their basis in scientifically derived evidence.‘Engagement’ meant promulgating and refining knowledge by informing, training or enthusing those who might either have important information or perspectives or be well positioned to instigate behavioural change (or show good reason not to) and/or might tap into local or national networks to help the initiative succeed. Naturally, the more those individuals were engaged, and the more powerful they were, the more the knowledge was subjected to the behaviours and negotiations discussed above. The Governance and Procedure ACKT was rich in engagement, not least because it required stakeholders to be explicitly consulted and to accept responsibility for decisions. This was often given visibly high priority, but took many different forms with varying success; which stakeholders became engaged and when, and how much power and influence they wielded – or not – could greatly alter the degree and direction of the transformation of the research-based knowledge. Engagement was essential to the successful uptake of the scientifically derived evidence conveyed by the second round of PL-Audit work (Box 3) following the ‘disastrous’ impact of the first audit where there had been no local clinical involvement.  However, clinical engagement could also thwart naive attempts at evidence-based policy-making; in another case study, hostile clinicians within the commissioning organisation initially disparaged the evidence in national guidance supporting the introduction of telehealth. A compromise policy, buttressed by alternative research-based evidence that backed the clinicians’ scepticism, was eventually agreed. The irony was that, in the face of the two sets of contradictory research-based evidence, the compromise was based on relatively weak local data.Our case studies have amply confirmed, as reported in the literature described in our introduction, that commissioning took place in a multi-transactional milieu resulting in interactive, pressurised, underdetermined and rapid decisions – an environment not of contemplative and logical appraisal of research-based evidence but of continual contested sensemaking through negotiation that involved many other sources of evidence. Our work has enabled us to provide a finer description of the dynamics of this environment and its impact on knowledge utilisation. All types of evidence, including research-based evidence, were repeatedly (re)transformed as commissioners inevitably and incessantly re-interpreted and recrafted it while carrying out their many roles. At every stage, the analysts, managers, consultants, clinicians and others who contributed to commissioning decisions  had their particular motivations, demands and constraints arising from their various specific roles and activities. Therefore, each actor (or group of actors) had to find their own solutions for the best ways to handle such pressures and tensions. The resultant negotiated iteration of the evidence depended on their context-based needs, their experience and tacit knowledge, and their individual and/or collective power.In analysing where and how that process occurs, we have postulated ‘active channels of knowledge transformation’ that provided both the organisational spaces and the mechanisms for it to happen, fundamentally affecting how and why the evidence was processed, and by whom. Working within those ACKTs, commissioners repeatedly subjected knowledge, including research-based knowledge, to numerous behaviours , as they negotiated and melded it with other, often conflicting, sources and types of knowledge, beliefs and evidence. This enabled them to maintain two key ‘incorporative processes’, contextualisation and engagement, by which the knowledge was transformed as it was absorbed into the eventual clinical policies that shaped local health and social services Fig. .Fig. 1 We were able as a deliberately reflexive, multidisciplinary team to draw upon a number of theoretical perspectives and compare across multiple data sources  to develop a rich picture of the ways in which managers used research-based evidence. We were also able to iteratively test our emerging ideas with commissioners on our advisory group.A possible limitation was that the original focus of our study was on the way commissioners exchanged knowledge with external agencies. Therefore, although that aim resulted in a great deal of data about how they subsequently processed that knowledge – on which we have drawn here – our methods were not designed to explore all the factors. This limits our capacity to analyse in any detail, for example, the power dynamics of the ways in which different actors were able to treat and negotiate knowledge or the differences in the way that they took up different types and presentations of research-based evidence. Crucially, we were rarely able to follow a given item of evidence, e.g. a particular best-practice patient pathway, through its various mutations. That would have required extensive shadowing (which the commissioning organisations declined) coupled with detailed discourse analysis, which was beyond the scope of the study.Another potential weakness may have been the timing of the study, which was disrupted by the 2012 Health and Social Care Act that restructured the NHS, changing the commissioning arrangements. However, we have no evidence to suggest that those changes materially affected the generic structures and processes we report here. Moreover, several of the authors have been in positions that allow us to maintain contact with a range of NHS commissioning organisations, and those contacts give no cause to suppose that the key elements of our findings have altered since our fieldwork.Finally, we recognise that the five ACKTs that we have identified in these case studies constitute are a beginning to the exploration of knowledge transformation processes rather than the last word. Other organisations will undoubtedly have different types of ACKTs.pluralism and opportunism”  may be used as evidence when identifying solutions to a problem, whereas financial data may be used to understand feasibility. Certain types of evidence relevant in one episode may also be questioned or disregarded in another.” [“nother.” A series of recent studies by a group based at the University of Warwick has also strongly pointed in this direction. They found commissioning managers using “practice . Hence, journey” .“EvidencOur detailed results confirm and reinforce this suggestion. Our commissioners similarly co-produced their policy deliberations and decisions through multiple interactive processes; they mainly did so within organisational spaces that we have delineated as active channels of knowledge transformation. Crucially, however, we found that this led to them continually reshaping the knowledge that underpinned their decisions, often doing so concurrently, not just episodically as suggested in the quotation above. Each (inter)action resulted in a re-negotiated co-construction – a reformulation – of that knowledge.situated expertise” based on the varying experience, local knowledge, “workarounds” and so on that they have accumulated. One way to frame our findings, then, is to posit that commissioners were using what Gabbay and le May [contextual adroitness” that arises from the internalised guidance – the mindlines – that they had collectively and individually accumulated. This would have allowed them to flex their distinctive professional knowledge, skills and techniques, their varied organisational norms, expectations and experiences, their alternative additional evidence, assumptions and values, and enabled them to efficiently negotiate complex decisions. Thus, for any piece of knowledge at any given moment in the process, each actor could formulate their particular rendition of the knowledge-in-practice-in-context necessary to satisfice their contextually distinctive goals and needs [Each actor or group of actors has their own set of goals – often at variance with others’ – in trying to solve a particular commissioning problem and move it to the next stage of the process. The pressures and demands on a clinical adviser, for example, would be different from those on a public health analyst or contract manager or indeed subtly divergent from another clinical adviser. In addition to their technical and professional knowledge and skills, each actor has what Swan et al.  refer tod le May  have calnd needs . It is tdialogical model” of research utilisation proposes that knowledge is contested, negotiated and emergent [propositional knowledge” to the complex “narrative knowledge” that Tsoukas has described in organisational knowledge management [Besides mindlines, one could frame this shifting of the underlying knowledge in other ways. The notion of organisational sensemaking , which oemergent . Finallynagement .Whatever theoretical frame one chooses, the important point is that it may indeed be worth examining more closely how policy-makers and practitioners interactively transform the nature and content of the evidence itself, and what that means for the conduct and evaluation of research-informed policy-making. Ethnographic techniques such as shadowing will help to unpick exactly how the evidence mutates through the repeated interactions and to explore and test the influence of the active channels of knowledge transformation. These and other methods may help elicit the social psychology of organisational decision-making, not just among policy-makers but also operational managers and clinicians. This will inevitably lead to some difficult epistemological questions about the nature of the different forms of knowledge, including scientifically derived knowledge, that inform policy and practice. However, it is not simply a question of ‘further research needed’. The practical implications of taking seriously the phenomenon of knowledge transformation are paramount.respectful critical conversations” [The processes we have described can have very different outcomes; they may end up badly corrupting research-based evidence, pragmatically adapting it or contextually enhancing it. Once one recognises and accepts that those possibilities always exist, it will become easier to develop ways to discern and realistically monitor the balance between these possible outcomes. Moreover, by understanding the processes by which knowledge is necessarily transformed, one might, rather than naively trying to insist on fidelity, foster the art of discerning when an adaptation maintains the essence of the research findings in a contextually appropriate way, and when it has become so attenuated or distorted as to no longer reflect them. Moreover, rather than trying forlornly to supress the transformation processes, one might be better placed to nurture them so as to augment their benefits, iron out their dysfunctional aspects and minimise their harmful effects. For example, by identifying and enhancing the workings of key communities of practice, or indeed artificially creating them, one might ensure that they have the appropriate membership, shared goals, mutual trust and respect to allow a constructive discussion of the various types of evidence.  Linked to this is the possibility of facilitating more “sations” , which wsations”  for ACKTsations” . Deeper Recognising and describing the inevitable phenomenon of transformation may also help us evaluate the impact of research on practice. Nearly 40 years ago, Weiss established that policy-makers can rarely point to research that directly informs their decisions since, as she put it, knowledge creeps into their deliberations as decisions gradually accrete . HoweverBox 1 The commissioning context at the time of our study (2012–14)In the United Kingdom, local healthcare policy-makers, known as NHS commissioners, are responsible for planning, contracting and evaluating their local healthcare. Their role is to optimise health outcomes by using the best available information to assess local needs, decide priorities and strategies, and then negotiate contracts for the appropriate hospital and community services. From 2012, this function in England was executed by clinical commissioning groups (CCGs) working in a complex environment with senior policy-makers, clinicians and local communities. During our study, there were over 200 local CCGs, each employing general practitioners (GPs) working part-time alongside non-clinical commissioners. CCGs often relied on support from public sector Commissioning Support Units and local government public health departments contributing, for example, evidence reviews, help with analyses of service activity data, project management or contract negotiation. Some CCGs also engaged commercial and not-for-profit providers of software tools and consultancy. At a national level, the National Institute for Health and Care Excellence (NICE) was a key source of guidance about the effectiveness or otherwise of treatments, and an ‘NHS Improving Quality’ initiative offered advice, guidance and tools. NHS England and the Department of Health provided strategic direction by issuing frequent guidance and directives. These and other support agencies, such as think-tanks and health-professional organisations, provided much of the research-based evidence that CCGs were expected to incorporate into their decisions. However, commissioners were also expected to fit these recommendations into their local contexts through analysis of local data and widespread consultation with local health professionals, patients, politicians and public.Box 2 How EpiTech’s evidence was promulgated to clinical commissioning groups and clinical staff‘EpiTech’ was a small international company originally from a university in the United States, marketing a software tool, developed through decades of international academic research, that enabled the use of GP practice-level data to generate information about patients with a high likelihood of using health services to a substantial degree. To provide the software, training and the user interfaces, EpiTech relied on local branches of large consultancy companies, whose advantage was their United Kingdom local knowledge. However, the United Kingdom consultancies were selective about the knowledge that filtered through to commissioners, e.g. an informatics company emphasised the delivery of useable patient data, whereas a management consultancy prioritised its managerial consequences, both of which affected the use and usefulness of the knowledge that the tool gave the commissioners. In this case study, keen local analysts, commissioning managers and project managers  were selected to learn advanced analytical skills via webinars where EpiTech provided technical training originally oriented to American healthcare but with some attempt to contextualise it for the NHS. The super-users’ roles were to encourage GP practice staff to use the software when planning patient care or to design algorithms., e.g. to help commissioners identify the right people to attend drop-in clinics for older people.Box 3 How the evidence from PL-Audit was contested and usedPL-Audit had used international standards, developed from evidence-based expert consensus on ‘best place of care’, to create a tool with which auditors could review patients’ notes to assess whether they ‘qualified’ to be in their current setting . The commissioners engaged commercial analysts to use the PL-Audit tool in an acute trust to identify hospital admissions that the research-based evidence and international expert consensus groups would have deemed unnecessary. The results suggested that 28% of patients need not have been in hospital, a finding hotly contested by the hospital and placing further strain on their already difficult relations with the commissioners. Over the next 9 months, after the serious shortcomings of the first audit were agreed and its methods revised, the commercial consultants were now to be simply kept on hand in an advisory capacity. In their place, five reviewers from the local hospital, community provider and commissioning agency were trained to use the tool but encouraged to continue augmenting and consolidating the audit methods through the experience of conducting the audit. This involved daily discussions with senior clinicians. The second audit concluded that the proportion of patients ‘not qualifying’ was 24%. Although similar to the earlier, rejected, results, these findings were accepted and readily acted upon within the hospital. The commissioners and hospital trust did not use the tool again but adopted its underlying methods (without the built-in evidence base) in their future audits elsewhere in the hospital.


The scientific data to guide the management of Peutz–Jeghers syndrome (PJS) are sparse. The available evidence has been reviewed and discussed by diverse medical specialists in the field of PJS to update the previous guideline from 2010 and formulate a revised practical guideline for colleagues managing PJS patients. Methods: Literature searches were performed using MEDLINE, Embase, and Cochrane. Evidence levels and recommendation strengths were assessed using the Grading of Recommendations Assessment, Development and Evaluation (GRADE). A Delphi process was followed, with consensus being reached when ≥80% of the voting guideline committee members agreed. Recommendations and statements: The only recent guidelines available were for gastrointestinal and pancreatic management. These were reviewed and endorsed after confirming that no more recent relevant papers had been published. Literature searches were performed for additional questions and yielded a variable number of relevant papers depending on the subject addressed. Additional recommendations and statements were formulated. Conclusions: A decade on, the evidence base for recommendations remains poor, and collaborative studies are required to provide better data about this rare condition. Within these restrictions, multisystem, clinical management recommendations for PJS have been formulated. Peutz–Jeghers syndrome (PJS) is a rare hereditary condition characterized by mucocutaneous pigmentation and Peutz–Jeghers hamartomatous polyps, predominantly affecting the small intestine  1,2]. T,2. T1,2]STK11/LKB1 gene) and follows an autosomal dominant inheritance pattern [PJS is caused by heterozygous germline pathogenic variants (PV) in the serine threonine kinase 11 tumor suppressor gene  commissioned this guideline (chair GM) and appointed a guideline leader (AW), who invited the listed authors to participate in guideline development. Two to three members around each key topic formulated key questions that were approved by the other members see . To devehttp://www.ehtg.org/guidelines/.This guideline was issued in 2020 and will be considered for  updating if indicated. Updates will be noted on the EHTG website: In 2000, the study by Giardello et al. clearly indicated that PJS patients are at considerable risk of developing cancer . This waBased on the reviewed literature and expert opinion, recommendations and statements were formulated on (1) clinical genetic, (2) gastrointestinal, (3) surgical, (4) pancreatic, (5) breast, and (6) gynecological management.STK11 (LKB1) (MIM 602216) is the only gene that is known to be associated with PJS. It is located on 19p13.3 and was identified by linkage analysis followed by cloning of the gene and identification of heterozygous PVs in affected individuals [ividuals ,8,20,21.STK11 PVs in cohorts of PJS patients varies. Without the use of methods to identify large deletions and duplications such as Multiplex Ligation dependent Probe Amplification (MLPA), detection rates of 10–70% were reported. With the introduction of MLPA, it became clear that larger deletions account for a significant proportion of PVs, and higher detection rates of 60–100% were reported  become available once a PV in STK11 gene is a useful tool to clarify the risk of PJS, especially in children and younger adults.If the clinical diagnostic criteria for PJS are met, genetic germline screening of the STK11 gene is warranted regardless of age. A patient meeting the clinical criteria should be regarded as having PJS, even if an underlying causative germline variant is not identified.Level of evidence: moderateStrength of recommendation: strongThe detection rate of pathogenic STK11 variants in patients with a clinical diagnosis of PJS is high (up to 100%), using techniques that detect single nucleotide changes as well as larger deletions and duplications in the STK11 gene. Currently, there is no evidence for genetic heterogeneity in patients fulfilling the diagnostic clinical criteria for PJS without a germline STK11 PV. Thus pathogenic variants that cannot be identified by up-to-date methods in routine diagnostics should be considered in these cases.Level of evidence: moderateStrength of recommendation: strongBased on recent recommendations of the European Society for Pediatric Gastroenterology Hepatology and Nutrition (ESPHGAN), genetic germline screening of the STK11 gene is warranted in children and adolescents with typical perioral pigmentation. Genetic screening may be considered in adults with isolated typical perioral pigmentation but is less likely to yield a pathogenic variant with increasing age.Level of evidence: lowStrength of recommendation: moderateGenetic germline screening of the STK11 gene is warranted in children and adolescents with one PJ polyp. Genetic screening may be considered in adults with a confident diagnosis of solitary polyp but is less likely to yield a pathogenic variant with increasing age.Level of evidence: lowStrength of recommendation: moderateIf no pathogenic variant in STK11 can be identified in a patient not fulfilling the clinical diagnostic criteria for PJS, the patient should not be considered as having PJS.                    Level of evidence: lowStrength of recommendation: moderateInitial genetic screening in patients who do not meet the clinical criteria for PJS is recommended by the European Society for Pediatric Gastroenterology Hepatology and Nutrition (ESPGHAN) in children and adolescents with lip and mucosal freckling suggestive of PJS . No new Based on recent recommendations of the European Society of Gastrointestinal Endoscopy (ESGE), a baseline oesophagogastroduodenoscopy and colonoscopy is recommended at the age of 8 years in asymptomatic individuals with PJS. If polyps are detected at the baseline endoscopy, a 1–3 yearly interval based on phenotype for oesophagogastroduodenoscopy and/or colonoscopy is recommended. Routine oesophagogastroduodenoscopy and colonoscopy surveillance is recommended at the age of 18 if the baseline endoscopy is negative.Level of evidence: lowStrength of recommendation: strongBased on recent recommendations of the European Society of Gastrointestinal Endoscopy (ESGE), small bowel surveillance is recommended from the age of 8 years in asymptomatic individuals with PJS. A 1–3 yearly interval is recommended based on phenotype for small-bowel surveillance. Either MRI studies or video capsule enteroscopy is recommended for small-bowel surveillance.Level of evidence: moderateStrength of recommendation: strongBased on recent recommendations of the European Society of Gastrointestinal Endoscopy (ESGE) elective polypectomy should be performed for small-bowel polyps > 15–20 mm to prevent intussusception. In a symptomatic patient, smaller polyps causing obstructive symptoms should be removed.Level of evidence: lowStrength of recommendation: strongBased on recent recommendations of the European Society of Gastrointestinal Endoscopy (ESGE), device-assisted enteroscopy for the removal of polyps is recommended. Based on phenotype, intraoperative enteroscopy could be considered.Level of evidence: moderateStrength of recommendation: strongIn case of symptoms, an oesophagogastroduodenoscopy, small bowel investigation, or colonoscopy should be performed earlier rather than waiting for routine surveillance.Level of evidence: lowStrength of recommendation: strongIn 2019, the European Society of Gastrointestinal Endoscopy (ESGE) and European Society for Pediatric Gastroenterology Hepatology and Nutrition (ESPHAN) published guidelines including the luminal gastrointestinal management of PJS ,49. AfteSolitary PJ polyps are rare, but there are numerous case reports describing the identification of a solitary PJ polyp, which may occur at all sites of the gastrointestinal tract, with the exception of the esophagus. There are only two published case series of patients with solitary PJ polyps, which, although imperfect, represent the best data available. Oncel and colleagues described eight patients managed at the Cleveland Clinic . Six of More recently, a larger case series from Japan has been reported . This muSTK11 gene is a useful tool to diminish the risk of PJS, especially in children and younger adults.For patients with a confident diagnosis of a solitary PJ polyp, routine endoscopic surveillance is not recommended.                     Level of evidence: low                     Strength of recommendation: strong                     Although the data above represent weak evidence, they suggest that there is no increased risk of metachronous PJ polyps or cancer, which in addition to the profiling data suggests that routine surveillance is not required once a diagnosis of solitary PJ polyp has been made. However, it is key that an appropriate assessment has been made to exclude PJS on clinical grounds; physical inspection for the typical mucocutaneous pigmented lesions, a full family history, and pan-enteric assessment  are required before a diagnosis of solitary PJ polyp can be made. There are no data reporting on the utility of genetic testing in the setting of a solitary PJ polyp, but as stated above, genetic testing of the Routine haemoglobin testing in children with PJS is not recommended, as there are no data reporting on its utility and outcome. Haemoglobin testing may be useful in the symptomatic setting.Level of evidence: lowStrength of recommendation: weakThere are no data to address the question whether there is a role for haemoglobin testing in children with PJS. There is a wide literature of anemia (with or without overt GI bleeding) as a mode of presentation for a subsequent diagnosis of PJS, but there are no data regarding routine haemoglobin testing in children with PJS, let alone children under 8 years of age, which is the advised age to start gastrointestinal endoscopy. Therefore, we are unable to recommend that haemoglobin testing should routinely be performed as part of the surveillance of children with PJS. However, if there is any clinical suspicion regarding significant polyps , checking the haemoglobin level may be a useful adjunct to guide us to the need for investigation with standard surveillance.Intussusception occurs when a proximal segment of bowel and its mesentery slides into the lumen of the adjacent distal segment. In PJS, a polyp typically forms the hypomochlion that subsequently leads to intussusception due to bowel peristalsis. Intussusception is a surgical emergency leading to bowel ischemia, necrosis, and perforation when untreated .PJS patients with an episode of acute severe abdominal pain and/or suspicion of intussusception should urgently be referred to a surgical unit, preferably a dedicated center. If, after clinical and diagnostic evaluation the event of small bowel intussusception is not ruled out, emergency surgery (even in diagnostic intent) is recommended.Level of evidence: moderate/lowStrength of recommendation: strongAt surgery, the preferred strategy of treating an intussusception is to dessuscept, if safe to do so. If successful, the polyp that acts as a hypomochlion should be removed by enterotomy with resection of the (pedunculated) polyp at the base. In addition, the entire small bowel should be critically inspected for further relevant polyps, and all polyps > 15 mm should be removed by enterotomy or by intraoperative enteroscopy. Depending on the distance between the polyps, an enterotomy in between polyps allowing for removal of multiple polyps via one enterotomy is preferred.Level of evidence: moderate/lowStrength of recommendation: strongIn PJS, the risk of intussusception is estimated to be 44% by the age of 10 and about 50% by the age of 20% . The risIn 2019, the International Cancer of the Pancreas (CAPS) Consortium formulated guidelines on pancreatic surveillance . After rPancreatic cancer (PDAC) is the third most common tumor affecting PJS patients with a lifetime risk of 11–55%  12,13,1,17,18. IAlthough PJS is considered a hereditary condition that carries some of the highest lifetime risks for developing pancreatic cancer, it should be discussed with patients that the benefits and harms of pancreatic cancer surveillance are not well established yet and under investigation. Therefore, it is recommended that surveillance is conducted at centers of expertise in the framework of a study or registry.Level of evidence: moderate/lowStrength of recommendation: strongBased on recent recommendations of the International Cancer of the Pancreas (CAPS) Consortium, patients with PJS are eligible for pancreatic surveillance in the framework of a study or registry, irrespective of patients’ family history of pancreatic cancer (PDAC), because of an estimated lifetime risk to develop PDAC of 11–55%.Level of evidence: moderate/lowStrength of recommendation: strongThe recommendations for pancreatic surveillance of patients with PJS of the International CAPS Consortium are endorsed and should be followed.Level of evidence: moderate/lowStrength of recommendation: weakFew studies compared the diagnostic yield of EUS and MRI/CPRM in pancreatic surveillance. A high concordance of clinically relevant lesions’ detection between the two methods was described by Canto et al. . Conversp < 0.0001), even after adjusting for other factors . In particular, the presence of a solid mass, mural nodule, thickened cyst wall, rapid cyst growth rate, and an MPD (main pancreatic duct) dilated to >5 mm at any time during surveillance were associated with the development of PDAC or high-grade precursor neoplasm, both at univariate and multivariate analysis.Prevailing regional pancreatic cyst surveillance guidelines should be carried out for cyst follow-up and management in PJS patients.Level of evidence: moderate/lowStrength of recommendation: weakAny significant abnormal finding during surveillance should be discussed in a multidisciplinary panel.Level of evidence: lowStrength of recommendation: strongIn a multicenter prospective study, Konings et al. reported a very high incidence of cystic lesions both in individuals with FPC (61%) and PV carriers (47%), including 11 PJS patients . They alTwo surgical approaches have been proposed for HRIs with pathologic findings identified during surveillance: the radical approach  and the conservative  surgical therapy. The main advantage of total pancreatectomy is radical removal of all pancreatic high-risk parenchyma, given the multifocality of precancerous pancreatic lesions in HRIs ,66. Howep < 0.001, I2 = 0%), while the pooled proportion of overall surgery was 6% , and that of unnecessary surgery was 68.1% . These results suggest that the probability of proceeding to surgery during surveillance is non-negligible, and unnecessary surgery is a potential negative outcome. Another meta-analysis was performed on 13 studies, including 90 HRIs (PJS patients in seven out of 13 studies) by de Mestier et al. and demonstrated that the surgical resection specimen revealed a pre/malignant lesion in 38 HRIs (42.2%), including 20 PDAC (22.2%) [In a meta-analysis of 16 studies by Paiella et al. including a total of 1551 FPC patients (syndromic HRIs were excluded), 30 subjects (1.82%) received a diagnosis of PDAC, PanIN3, or HGD-IPMNs . Therefo (22.2%) .A recent multicenter international study was conducted through the CAPS Consortium Registry to examine the diagnostic yield and outcomes of HRIs who underwent surgical resection or progressed to invasive cancer under surveillance and the characteristics of patients who developed high-risk neoplastic precursor lesions or PDAC . Of 76 hp = 0.02), in particular delayed gastric emptying . Postoperative diabetes developed in 20% HRIs who underwent partial pancreatectomy with no difference between distal and Whipple surgery, while it developed in 100% of HRIs receiving total pancreatectomy. No intra-abdominal hemorrhage was observed.Another recent study carried out by Canto et al. evaluated HRIs  outcomes after pancreatic resection during surveillance: 354 asymptomatic HRIs enrolled prospectively in CAPS studies from 1998 to 2014 and underwent surveillance for at least 6 months . The autAccording to the recent recommendations of the International CAPS Consortium, a  pancreatectomy should be performed in case of detection of: (i) a solid lesion ≥ 10 mm ; (ii) IPMN in case of a mural nodule, an enhanced solid component, symptoms , thickened/enhanced cyst walls, abrupt change in pancreatic duct with distal pancreatic atrophy, or a main pancreatic duct ≥ 10 mm.Level of evidence: moderate/lowStrength of recommendation: strongDue to its significant morbidity and potential mortality even in experienced hands, a total pancreatectomy is not recommended for a localized lesion.Level of evidence: lowStrength of recommendation: strongProphylactic pancreatectomy is not recommended because of the significant associated morbidity and potential mortality, even in experienced hands.Level of evidence: lowStrength of recommendation: strongWhen screening is negative in HRIs, prophylactic pancreatectomy is not indicated in view of its significant morbidity and the potential mortality even in experienced hands, mainly with pancreatoduodenectomy ,76.AccoSTK11 and in 5/8 women suspected of having PJS (based on mucocutaneous pigmentation) [Estimates for the lifetime risk of breast cancer (BC) in women with PJS vary widely from 19.3% to 54%, which is probably due to the small sample sizes in most studies  12,13,1,17,18. Ontation) . Multiplntation) ,78,79. Antation) ,17,25,80ntation) ,17. In mThe following breast surveillance is recommended in female PJS patients: Raising awareness at age 18 years e.g., by starting breast self-examination; Clinical breast exam every 6–12 months starting at age of 25 years; Annual breast contrast MRI screening (or breast ultrasound if MRI contraindication or unavailability) at age 25–30 years; Annual mammogram with consideration of tomosynthesis and ultrasound for dense breast and annual breast contrast MRI at age 30–50 years; Annual mammogram with consideration of annual breast contrast MRI for dense breast pattern at age 50–75 years; Management should be considered on an individual basis from age > 75 years.Level of evidence: lowStrength of recommendation: moderateThe optimal breast surveillance strategy in female PJS patients remains debated and the benefits of surveillance remain to be established. Therefore, it is recommended that surveillance is conducted at centers of expertise in the framework of a study or registry.Level of evidence: lowStrength of recommendation: strongAs evidence for its benefit is lacking, prophylactic mastectomy is currently not recommended for female PJS patients. Risk reducing mastectomy should be discussed in a multidisciplinary setting also taking into account family history and other clinical factors.Level of evidence: lowStrength of recommendation: moderateNo clinical trials on breast surveillance protocols for women with PJS have been published. Although PJS is mentioned in guidelines on breast cancer surveillance for individuals at high risk of developing cancer, there are limited recommendations for PJS patients specifically. MRI starting at the age of 25–30 years is most often recommended, and several authors refer to the NCCN guidelines for hereditary breast and ovarian cancer, which recommend mammogram and breast MRI annually and clinical breast evaluation every six months, all starting at age 25 ,82. SeveLKB1 germline mutations from four different cancer institutions [STK11 germline PV carriers from 16 institutes. Seven gynecological cancers were detected [The risk of gynecological cancer is increased in women with PJS with current estimations ranging from 18% to 50% by the age of 50 years  12,13,1,18. In titutions . Seven odetected . Of the detected . Their rThere are no data on prospective surveillance programs with a systematic approach to gynecological cancer surveillance. Van Lier et al. published results from their program, which included all PJS patients from two Dutch hospitals . The patExpert gynecological surveillance should be offered to female patients with PJS, irrespective of their family history of gynecological cancer, because of an estimated lifetime risk of specific gynecological tumors of 18–50%.Level of evidence: lowStrength of recommendation: moderateIt is recommended that female PJS patients are counseled regarding specific gynecological cancer risks, red flag symptoms, contraceptive choices, and family planning by a PJS specialist at 18–20 years of age.Level of evidence: lowStrength of recommendation: moderateIt is recommended that female PJS patients have annual gynecological examinations from the age of 25 years. In addition to cervical screening as performed in population-based screening programs that run in many countries, gynecological surveillance in female PJS patients should be focused on the detection of cervical adenocarcinomas, in particularly minimal deviation adenocarcinoma, and rare non-epithelial ovarian tumors. Surveillance for cervical adenocarcinomas should involve speculum examination and cervical screening ("Pap smear") including cytology even in an HPV-negative sample. Surveillance for non-epithelial ovarian cancers should involve bimanual pelvic examination with a transvaginal ultrasound in case of suspicion of a pelvic mass. CA125 testing is not indicated.Level of evidence: lowStrength of recommendation: moderateThe optimal gynecological surveillance strategy in female PJS patients remains debated and the benefits of surveillance remain to be established. Therefore, it is recommended that surveillance is conducted by a gynecologist who is experienced in the particular cancer risks that PJS patients face in the framework of a study or registry.Level of evidence: lowStrength of recommendation: strongIn conclusion, cervical adenocarcinoma, in particular minimal deviation adenocarcinoma , is the most frequently reported gynecological cancer in women with PJS. The risk for ovarian cancer is also increased, but the histology of the ovarian cancers is not well reported. Based on the literature, the ovarian cancer risk seems to apply to non-epithelial ovarian cancer (SCTAT), with the risk of the more common epithelial ovarian cancer not increased. The reports do not suggest that the risk of endometrial cancer is increased in women with PJS. In the absence of evidence-based data on gynecological surveillance, our recommendation is based on expert opinion and current knowledge of gynecological cancer risks in women with PJS. The detection of minimal deviation adenocarcinoma from Pap smears or from clinical features is difficult and requires a high index of suspicion. These tumors are not caused by high-risk human papillomavirus (HPV), and therefore, routine cervical screening triaged by the presence of HPV may fail to detect them. Except for vaginal ultrasound examination of the ovaries, there are no good screening tests for non-epithelial ovarian tumors, which constitute the major ovarian cancer risk. There are no data on the usefulness of tumor markers  for ovarian cancer surveillance in PJS patients.There is no literature on experience with prenatal genetic diagnosis (PND) or preimplantation genetic diagnosis (PGD) in PJS. Wang et al. describe a positive prenatal genetic test for PJS with continuation of the pregnancy; PGD for a subsequent pregnancy of the PJS patient was suggested . Woo et Peutz-Jeghers syndrome can be an indication for Prenatal Genetic Diagnosis (PND) and Preimplantation Genetic Diagnosis (PGD) and these options should be discussed with PJS patients in whom a STK11 pathogenic variant has been identified.Level of evidence: lowStrength of recommendation: strongThe evidence base for recommendations regarding the management of PJS remains poor, and collaborative studies are required to provide better data on this rare condition. Within these restrictions, multisystem, clinical management recommendations for PJS have been reviewed and updated. EHTG supports the concept of continuous revision of these recommendations as a concept of “dynamic” guidance. In the event of relevant literature providing evidence for a better management strategy, the corresponding recommendation will be revised accordingly. In a multidisciplinary management program, all parties (including patients) are welcome to approach EHTG and request a revision.