Different measures to prevent and control the spread of SARS-CoV-2 have been implemented in German schools. Decisions regarding such measures should be informed by evidence regarding their effectiveness, and their unintended consequences for health and society. A multi-stakeholder panel was convened to develop an evidence- and consensus-based guideline for school measures, using the novel WHO-INTEGRATE framework. Developing a guideline to inform decision-making outside of the clinical realm during a public health emergency was unprecedented in Germany. This study aims to identify lessons learnt for similar endeavours by addressing the following research question: What were the strengths and weaknesses of the guideline development process as perceived by the different groups involved?Fifteen semi-structured interviews were conducted virtually. We recruited participants aiming to include the perspectives of all groups contributing to the guideline development, including both panel members  and the guideline secretariat. For analysis, we carried out deductive-inductive thematic qualitative text analysis according to Kuckartz, structuring findings using a category system.Due to time pressure, the guideline secretariat was heavily involved not only in synthesising the evidence but also in developing and drafting recommendations. Participants critically reflected on certain methods-related decisions, including the development of draft recommendations and application of the WHO-INTEGRATE framework by scientists only. The full potential of the framework might not have been harnessed. Participants’ understanding of relevant and valid evidence varied, and the available evidence base was limited. Participants represented different types of expertise, notably expertise informed by scientific evidence and expertise grounded in lived experience, influencing their involvement in the guideline development process and discussions during meetings.Developing an evidence- and consensus-based public health guideline in only three months was challenging, notably because of the involvement of a broad range of stakeholders and the use of a novel Evidence-to-Decision framework, both unprecedented in Germany. Learning from this process with a view to “institutionalising” the development of public health guidelines and refining methods can contribute to more evidence-informed public health decision-making in Germany and beyond, in general and during a public health emergency. During the COVID-19 pandemic, public health and social measures (PHSM), also referred to as non-pharmacological interventions, have been implemented around the world across a range of settings. Measures to prevent and control the spread of SARS-CoV-2 in schools (ff. school measures) include interventions to reduce the opportunity for contacts such as cohorting; measures to make contacts safer such as mask mandates and surveillance and response measures such as testing  and reduThe Appraisal of Guidelines for Research & Evaluation Instrument (AGREE II)  as well as public health practitioners, set out to develop an evidence- and consensus-based guideline for school measures . The guiGREE II) .In Germany, the Association of the Scientific Medical Societies  coordinates the development of clinical guidelines in adherence with the AGREE II quality criteria. These are classified according to the extent to which “elements of systematic development”  are applDeveloping an S3-guideline to inform decision-making outside of the clinical realm during a public health emergency was unprecedented in Germany. Based on an understanding of schools as complex systems and of any intervention to prevent or control SARS-CoV-2 infections taking place within such systems, a consideration of the potential effects of these interventions beyond health outcomes was emphasised during guideline development. Consequently, the WHO-INTEGRATE Evidence-to-Decision framework (EtD), developed with a view to aid decision-making “about complex interventions implemented in complex systems” , was appIt is important to have a thorough understanding of the usefulness and impact of different processes and instruments seeking to aid evidence-based decision-making, especially in the context of the COVID-19 pandemic, which has required timely and potentially incisive measures. The S3-guideline represents such an instrument and evaluating its impact as well as the strengths and weaknesses of its development process can contribute novel insights to the research fields of evidence-based policymaking, as well as help identify lessons learnt and develop best practices for future health emergencies.This study represents one sub-study of a multi-pronged approach to evaluating the S3-guideline as an instrument for providing evidence-based support for decision-making. The focus of the present sub-study is on the guideline development process from the first constitutional meeting on 16th December 2020 until publication of the first long guideline version on 26th November 2021, notably it addresses the strengths and weaknesses of the guideline development process as perceived by the different groups involved. The impact of this guideline on political and practical decision-making as well as experiences with implementing measures in schools according to this guideline are not the subject of the present paper but the subject of further studies.We chose a qualitative research approach as we were interested in the subjective experiences and perspectives of those who participated in the guideline development process . We had n = 61) of the guideline panel and secretariat and informed them about the purpose of the study. Willingness to participate was expressed by the following number of participants within each of the five groups: (i) 9 scientists , (ii) 4 public health practitioners , (iii) 4 members of the guideline secretariat , (iv) 6 members of the school family  and (v) 1 observer . The only inclusion criterion was having served as a member of the guideline panel or guideline secretariat during the period indicated above. To choose amongst those who expressed willingness to participate in each group, we used an online random generator. In each group, we aimed to interview at least two individuals and included further participants according to the randomly generated order, with particular attention paid to achieving saturation regarding the perspectives of teachers, parents and students from within the school family group. The only exceptions to this approach concerned the representative of the AWMF as well as the pilot interview participant who we explicitly invited to participate and categorised as guideline secretariat.To ensure representation of all relevant perspectives, we categorised members of the guideline panel in four groups plus the guideline secretariat. We sent invitations to all members (via a web-based tele-conferencing tool. Questions concerned the following topic areas: (a) participants’ perception of the process, (b) their understanding of evidence and of the role of evidence in the process, (c) their understanding of expertise and of the role of expertise in the process and (d) views on the consideration of societal implications and unintended consequences. We also investigated participants’ views on the impact of the guideline on political decision-making, on lessons learnt for decision-making in public health (emergencies) as well as roles and communication during the guideline development process, which are not reported in this article. The interviews were recorded using a linear PCM recorder (OLYMPUS LS-P1). KW and MR independently wrote memos following each interview. All data were securely stored on an encrypted device and anonymised, with KW and MR having exclusive access to the primary, non-anonymised data to maintain confidentiality given the other authors’ active involvement in the guideline development process.All prospective participants were provided with study information sheets and consent forms. Two researchers  conducted semi-structured interviews in German from 12th November to 22nd December 2021, using an interview guide which contained open-ended questions in relation to the study aim see . This waTranscription of all interviews was carried out by Audiotranskription , a GermaEthical approval for this study was obtained prior to recruitment of participants from the Ethics Committee of the Medical Faculty, Ludwig-Maximilians-Universität München (No. 21-0944).In total, 15 members of the guideline panel and secretariat were interviewed. The secretariat was described as “the structural guards of the guideline” . Members of the secretariat were heavily involved in the guideline development process, not just in a coordinating and methods-related supporting role but also in helping with drafting and revising recommendations. Whilst one participant pointed out that “actually I understand that a secretariat always assists in guideline [development] […]. And that decisions are then made separately” , it was also noted “that a lot was then left up to the staff [of the secretariat]” . Even more explicitly, this participant said: “And later I also directly worked on the recommendations and supported the [working] groups. But partly I also wrote a lot myself” .Participants emphasised that they appreciated the transparent, democratic and anonymous nature of the consensus-building procedures. They also perceived it as useful and time-efficient that the scientific literature was being systematically identified and appraised for them:“I found the scientific preparation VERY good. [..] For us as participants in the guideline panel, everything was prepared and presented very well and in a comprehensible manner” .The iterative process of working in smaller groups and consensus voting with the full panel was generally described as efficient and goal-oriented. One member of the school family pointed out that they perceived the whole panel to be committed to the shared goal of keeping schools open and maintaining the best possible education even under pandemic circumstances. However, the process also involved some polarisation within the panel:“[There were] players [..] who [..] suddenly put in a veto and said ‘We're definitely not going to go along with this and we have to start everything from scratch again’” .Certain methods-related decisions taken during the guideline development process were critically reflected upon by participants. One participant reported that, after voting on a few options for recommendations or wordings, the option with the highest agreement was immediately chosen instead of voting again on that option; this participant pointed out that the recorded votes may “not depict what a final voting might have resulted in” . Furthermore, not having formally prioritised endpoints for outcomes of interest to consider during the process was pointed out as a challenge by several participants, for example:“We did not undertake a formal prioritisation of the endpoint[s]. [..] So we kind of presented and discussed them in the kick-off meeting […]. But we did not formally prioritise them” .One participant  was particularly vocal about methods-related decisions and their implementation. For this analysis, this perspective ought to be seen as unique due to a particular interest and expertise in processes of evidence-based decision-making. One aspect criticised was the very high number of topic areas for recommendations, which in this person’s view deprived the panel of time and resources to discuss fewer recommendations in a more comprehensive manner. Similarly, the same participant noted that allocating more time to a thorough and comprehensive process of prioritising and then adapting the generic criteria of the WHO-INTEGRATE framework at the beginning of the process might have been beneficial, as using the generic criteria “later led to the fact that we simply needed a lot of coordination again, the teams needed a lot more support because they could not do anything with a lot of the stuff […] if we had simply stopped [earlier] at: Which endpoints are important to us? What should we look at through INTEGRATE sub-criteria? […] We would have saved a lot more time in the course [of the guideline development process]” . This was partially echoed by another participant who said that they would have wanted to “actually go through these criteria again in detail, see if we have evaluated them correctly? Are there any problems in the area of implementation, for example, which should also inform the formulation of the guideline? And THAT could not take place, due to the time pressure” . This participant even suggested that the assessment of criteria was based on a few individuals’ opinions:“de facto, we did not make enough room for this process [consideration of societal implications], or perhaps we did not have the tools to do it particularly well. So we actually WANTED to attach so much importance to these criteria, but DIDN’T. Because two or three people somehow simply came up with what they thought about these criteria” .Similarly, it was mentioned that “the point about conformity with fundamental and human rights was like a box that had to be ticked once, […] yes or no. Yes, that wasn‘t a question for discussion at all” , pointing to further aspects which influenced the application of the WHO-INTEGRATE framework as discussed in more detail under Consideration of societal implications and unintended consequences.Furthermore, some participants perceived the composition of the guideline panel as imbalanced. In relation to this, the following comment regarding the extent to which legitimate representation was achieved is interesting:“For example, it was not clear to me what criteria were used to include people or institutions from the education sector in the panel. I found that rather random. […] For example, the Federal Parents' Council is not strictly a participatory, democratically elected body […]” .Methods-related decisions for the systematic review, which represented the main—although not the only—source of peer-reviewed evidence to inform the guideline development process, were also critically discussed. This concerned, for example, the decision to include modelling studies of varying degrees of quality alongside intervention studies based on empirical data, but to exclude basic research studies (e.g. lab-based studies). Consequently, a certain body of evidence was available and considered by the panel and this “did also frame the process, when saying ‘Here we have some evidence. Here we have no evidence’” .During the interviews, it emerged hat “The group was, so to speak, […] divided into those who had already thought things through, pre-formulated things in the specific working groups. And then, so to speak, as a reflecting team, then us as school experts” . This points to a procedural decision regarding participants’ distinct roles in the process, evoking diverging opinions. One participant’s comment shows that some may have perceived this approach as derogatory:“So [the parents] were basically included […] in the formulation, but not in the development [of recommendations]. There were only a few institutions. For instance, the parents are not considered […] as experts” .It was also questioned whether certain issues (e.g. feasibility or acceptability) might have been addressed differently, if members of the school family had been involved in the development of recommendations from the very beginning. Contrastingly, another participant questioned the value of having debates on an eye-level between all guideline panel members:“But I dare to doubt whether all people then have to vote on everything on equal terms. So to put it very concretely, it has sometimes been difficult for me to discuss with people what the scientific evidence in favour or against a recommendation is. And conversely, as a scientist, I don't want to make presumptions about the implementation in practice. […] involving them [those affected by or tasked with implementation of measures] right at the beginning […] I found difficult and tedious” .Lastly, it was pointed out that the guideline fell short of considering how the recommendations would interact with each other and whether following through on some of them might render others redundant, e.g. whether a mask mandate would still be warranted if distancing and ventilation measures were fully implemented.Operating under time pressure was identified as a big challenge by participants. As drafts for the recommendations were developed by small working groups only, these might have been perceived as a “done deal”  because: “It was certainly the case, and we also regretted that […] there wasn’t enough time to explain everything in detail to the others” . This suggests that there was not enough time to fully address all concerns within the panel and to create a shared understanding of the rationale for recommendations before they were voted upon. This participant pointed out that “in my view, a more intensive debate was missing in the aftermath or in the whole implementation [of the process] […] And that takes time and therefore, given the dynamics of COVID, we simply do not have that time, so for me it is questionable to what extent such a guideline is purposeful” .Sharing a lot of material for information and comment shortly before meetings with the panel “also caused discontent” , as it constituted the only option for those who were not part of the working groups to familiarise themselves with and comment on the draft recommendations before voting on them. It was suspected that “given that this was roundabout a 100-page document, only very few people did that [reviewed the material]. Simply for capacity reasons” . Related to this, participants stated that they felt overwhelmed with having too little time to prepare for the panel meetings. Slightly contrary to the general notion of time pressure, one participant noted that they had enough time to make up their mind about their voting decision: “These coordination processes were often such that one had time […] to think it through afterwards and to deliberate at home on the computer how I should decide now. […] So, in this respect, no one can really complain that they were somehow forced to make decisions too spontaneously, too suddenly” .There were discrepancies in how scientific evidence was interpreted and perceived by different groups represented in the guideline panel, as well as the secretariat. Those with a background in epidemiology and public health emphasised a need to focus on population-level outcomes when developing recommendations for this guideline. They also mentioned other defining aspects for scientific evidence, notably the systematic consideration of all available evidence and the assessment of its quality, as well as a notion of usefulness and reliability of certain types of evidence compared to others. For members of the school family, the notion of scientific evidence as well as scientific methods to be applied to assess certain phenomena was probably inherently different from those with a background in science:“I would say that perhaps scientific disciplines like medicine […] have an easier time. Because you can measure a lot of things quite clearly and then it's clear that I'll do so and so many studies with certain parameters and determine the evidence from them. This is much more difficult in the pedagogical field because, for example, you cannot work so much with control groups or with placebos […] So I think that the question of ‚based on evidence ‘[…] is perhaps not so common among people who work in the pedagogical field” .It was also mentioned that certain types of evidence should have received more weight during the process, for example basic research studies:“There were [..] different interpretations of what evidence really is, and of what direct and indirect evidence is. So, the majority […] they are epidemiologists somehow and they have very clear standards of what evidence means […] for me, evidence from basic research counted just as much […] and I would give that a little more weight […] than it actually happened in the group, because the very clear, hard epidemiological criteria were applied there” .Thus, the understanding of evidence and the extent to which certain types of evidence could or should be used to base conclusions on, appears to have varied between the different interview participant groups. Not only were there differences amongst those with a background in science but also between this group and members of the school family. For example, an understanding of the importance of standard scientific practice appears not to have been equally shared by all members of the guideline panel and secretariat:“Especially among non-medical persons or [..] non-scientific persons who participated, […] they often simply sent us statements by some expert that somehow did not even list any references [..] Which for me would be a reason to simply exclude any studies from a guideline. And that was not accepted by others” .The following statement suggests that the separation of participants in those actively involved with drafting the recommendations and those commenting on the outputs could be linked to a different understanding of evidence and its role in the process:“I suspect that the understanding varies greatly within the group. So those, especially the scientific societies, who developed the recommendations in close coordination with [the secretariat], I would assume they HAVE this understanding. I suspect that many of the others, who actually only came to the guideline meetings, do not have this distinction so clearly in their minds” .The following quote alludes to the importance of providing more opportunities to develop a shared understanding of evidence and its role:“And there were some very strong misunderstandings at the beginning, which we, namely I, was able to resolve through conversation and also through an exchange about what evidence actually is. I found that very helpful. But these were background discussions, nothing that the whole group took part in” .In principle, participants agreed that evidence had to and did play a critical role in the process.“a very big [role] and it also has to be that way at the AWMF […] That’s what the guidelines are based on” .Participants referred to the systematic review on the effectiveness of school measures  as the p“A disadvantage of the guideline is precisely that it IS evidence-based, and when people then come and argue, there's no point, I don't have any studies, although common sense says that it/they are aerosols, and if I sit with 30 children in too small a room without ventilation and masks, then they accumulate [..] the call for evidence is not always helpful, yes. Because that sometimes also blows wind into the sails of counter-arguments by simply claiming that there is no evidence” .The three notions of expertise present in the data with supporting quotes are displayed in Expert opinion was considered crucial for the guideline development process, given the lack of evidence for many questions at hand, as well as to contextualise, interpret and, where appropriate, apply weightings to the available evidence. The perspectives of affected groups were also considered important, especially with regard to feasibility and practical implementation.“I definitely need [..] the expertise of those who are somehow involved in the exact conceptualisation and implementation of measures. For example, the perspective from the school family: What can actually work? Or the perspective from the local health authority: How can I properly implement quarantine regulations in schools? In other words, scientific evidence must be coupled with anecdotal evidence or precisely with the expertise of the different affected groups” .The diversity of perspectives and opinions represented in the guideline panel was described as providing legitimacy for the guideline. However, one member of the school family pointed out that an even more diverse panel, e.g. with regard to age, would have reflected the diversity of the school family and school setting even better. An interesting reflection on the role of their own expertise was provided by this member of the school family: “I therefore knew what role we have, or what it meant that it is a medical guideline, where we as [professional title] can perform an advisory function. But also hold ourselves back concerning medical questions” .Participants pointed out that they perceived hierarchies based on seniority and academic credentials, with some scientists’ statements being more contested than others.“I noticed that some of the statements made by [the person], although [the person] belongs to a scientific society that should actually have expertise in the field, were not so strongly accepted. On the contrary, they were often questioned, critically questioned. [..] in general, titles alone, no title, I perceived that to be making a difference in the process. [..] even if [..] a person is referred to as professor or doctor, I think that makes a difference in the discussion” .Whilst these differences in the recognition of expertise of members of different scientific societies were observed, even more striking perceived differences with regard to scientists versus members of the school family are reflected in the following statements.“The parents [..] had to acquire such a comprehensive expertise [..] and yet we are not seen as experts because we might not belong to an association [..] are you an expert if you are a doctor or scientist and belong to an association [..] You are labelled, you are only such a small light and you do not belong to this big institution” .“In a context with […] who all speak very scientifically, it sometimes is an effort to say something” .Related to this, one participant argued that without proof of expertise in the form of relevant professional experience, a member of the school family should not be accepted as an expert: “[The person] also did not have anything in their CV based on which [the person] could have said I have expertise regarding this topic” .It was pointed out that a lack of experience with developing guidelines or similar products, with systematically appraising evidence as well as with using scientific language, not only, but specifically on the part of members of the school family, might have led to reduced possibilities for fully participating in the process.“For me it was the first time that I took part in something like this and at the beginning it was perhaps also a little confusing and alienating [..] I mentioned at some point that many technical terms, abbreviations are very confusing for us because I then had the feeling that ‘Unfortunately I do not know what you are talking about’” .Some also mentioned that they perceived the discourse to be strongly influenced by those members of the panel who were particularly “eloquent and in need of talking” , which left another participant rather “disillusioned [..] because I had the feeling that decisions were not primarily made on the basis of what was really known, but that in the discussion, for example, individuals who simply had a very high social standing due to their background or their position dominated very much [..], and that many people then simply followed this” . Specifically, participants reflected on the role and influence of one institution during the process which was perceived as having a strong voice. Some participants even suggested that the guideline was therefore aligned with already existing recommendations:“In various decisions, [the institution] has simply asserted itself. So many of the things that are currently in the guideline are actually points that [the institution] has in its school recommendations. And when the guideline wanted to deviate a bit from that, a veto was used until this compromise was reached again” .The weight of the students’ voices was described differently by different participants.“It was not the case that one had the feeling that some scientists were dominating [..] the younger people from the school sector, for example. [..] there is a hierarchy gap; a chance, let's say, for this [the hierarchy gap] to have an effect. But in this process I had the feeling that it was actually going quite well, I have to say” .“The students who were very withdrawn. [..] I experienced them as very mute in the process [..] possibly because of these hierarchies that were not actively broken up” .A certain difference in self-perceived legitimacy amongst members of the school family versus those with a background in science could also be observed. This member of the school family was “very surprised when I was appointed. […] Because, as I said, S3 guideline did not mean anything to me. But I think it’s good that in this case, the three truly biggest players from the school sector were involved” . Conversely, this researcher explained that their organisation “was involved in this leading role from the very beginning. And we also discussed this internally in the board of directors that we [pseudonymised] also want to take on a leading role in the process” . Contrastingly, some participants did not perceive any differences between themselves and others or within the panel at large and indeed emphasised that their specific expertise was sought and appreciated.“In the beginning, what was new for me from the medical perspective was new for those from the practical school perspective. There was a process of convergence. But I did not feel that we were somehow second-class members of the expert council” .“It is always mandated […] when it comes to medical guidelines, that there are also patients in it. But honestly, it is often a marginal contribution. And here it was not a marginal contribution. […] Everything that the academics contributed was countered a little bit by views from the world of the students, teachers, teachers' associations, parents” .When participants were asked to recall how societal implications and unintended consequences of recommendations were considered, those who were part of the working groups or had a coordinating role elaborated on these in relation to the criteria laid out in the WHO-INTEGRATE framework. Participants noted, however, that unintended health consequences as well as social outcomes received a lot more attention relative to economic, ecological or legal aspects. It can also be assumed that the foci of consideration in the working groups varied, as pointed out by this participant:“It happened in such a way that small groups suggested wordings, and these were then discussed. And that was very different. On the one hand, the level of wordings was different, the level of consideration of the different levels of the framework was very, very different” .Others, particularly members of the school family who were not part of the working groups, where the framework was explicitly applied, recalled that the weighing up of feasibility and further aspects happened in discussion during the full panel meetings (as opposed to a systematic examination of criteria). Crucially, it was remarked that “[the weighing up] was included through discussion. Although this is/ partly it is dependent on which persons/ who gets to talk” . From the perspective of members of the school family, the main tension to address when agreeing on recommendations was what was stipulated with regard to infectious disease control and what was desirable from an educational perspective . Participants noted that institutional interests likely also played a role in influencing participants’ lines of argumentation during these discussions.It was noted by participants that both evidence and specific expertise for assessing societal implications and unintended consequences beyond direct health impacts was largely missing.“As we didn't even have time to really discuss the advantages and disadvantages for health in such a way that I think everyone had their say as they should have done [..] the expertise was of course not in the group either. So [..] we didn't have anyone from that field. [..] should have been strengthened in any case by some expertise within the group” .Another participant noted that due to the lack of evidence, “good common sense”  played a big role in assessing any implications beyond health impacts as well as in agreeing on the strength of recommendations. Similarly, the lack of qualitative research regarding values or preferences of those affected by the recommendations was highlighted as a limitation.Participants reflected on the relevance of EtD frameworks, especially in the context of developing guidelines during a public health emergency. In principle, and especially in the absence of conclusive evidence regarding the effectiveness of measures, they were perceived as being important to aid decision-making and to give a comprehensible structure to the process of arriving at recommendations, as it “helps us to create transparency […] in the justification of recommendations” . Using an EtD framework was described to have helped the panel to ground their recommendations in reality and to explicitly consider any potential side effects of recommendations in a transparent manner according to this participant: “The appeal of the criteria [was] that we had a scheme that we worked with and that enabled us to make such considerations transparent, whereas in guidelines that do not use criteria, such things often feature in rather intuitively, implicitly, and are not made explicit” .Another participant, despite recognising the value of increased transparency, questioned the added value of using EtD frameworks in guideline development processes. This person questioned the usability for practitioners, as they had previously noticed “that they will not follow any more at some point. That they then say, this is actually too complex, too complicated for me” . Based on this observation, they further questioned: “Are those really better guidelines, are they really better suited for practical considerations? Or are they better suited for the publication? […] I do not think we know that […] I am so far not aware that this results in the guidelines being better accepted and implemented” . Another participant similarly questioned whether the working groups’ assessments of the WHO-INTEGRATE criteria were directly relevant for panel members’ voting behaviour or for the target group of the guideline.The development of an S3-guideline to provide evidence- and consensus-based support for decision-makers regarding school measures during a public health emergency was unprecedented in Germany. The present study provides insights into the strengths and weaknesses of the guideline development process as perceived by the different groups involved.In addition to its overall coordinating role, the guideline secretariat systematically identified and appraised the evidence, provided methods support, notably by being involved in the application of the WHO-INTEGRATE framework in the working groups, and directly contributed to interpreting the evidence and drafting recommendations. It can be argued that this is justified by the considerable time pressure and the mixed levels of experience in guideline development and use of EtD frameworks, which warranted intensive support. As such, the secretariat’s work was appreciated by participants. Conversely, uniting too many tasks in one group can overburden this group and potentially lead to undue influence on the panel’s product. The tasks of interpreting the evidence and drafting recommendations are usually separate from the tasks of searching and synthesising evidence in well-established guideline development processes at the World Health Organization or the UK National Institute for Health and Care Excellence –17. Our The application of established AWMF procedures for clinical guidelines to the rapid development of a public health guideline happened during the ongoing process. Some methods-related decisions by the guideline secretariat in consultation with the scientific societies who formally registered the guideline were critically reflected upon by participants, suggesting that the pros and cons of these different decisions had not been made transparent. These included the selection of institutions invited to join the panel, the missing prioritisation of endpoints and the process of and lack of time allocated to the application of the WHO-INTEGRATE framework. As recommendations were developed in working groups composed only of scientists, the perspectives of those affected by or tasked with implementation of recommended measures were only included later in the process. Some participants perceived this separation of tasks as appropriate, whilst others felt that having all perspectives – representing different types of expertise as discussed below - included in the full process of recommendation development would add to their quality. Which of these approaches would be more time-efficient for a rapid guideline is unclear. This indicates that further methods-related reflection, nationally or internationally, and ideally agreement on which shortcuts are acceptable in guideline development during a public health emergency, is warranted with a particular focus on sources of bias .In line with AWMF procedures, the development of an S3-guideline required systematic searches for and quality assessment of scientific studies ; the fraPanel members included those whose expertise should arguably be grounded in an understanding and critical assessment of scientific evidence, and those whose expertise was grounded in their lived experience. In the absence of a comprehensive and conclusive evidence base and under time pressure, those with a scientific background also had to draw from their own lived experiences during the pandemic. Nonetheless, whose expertise was accepted as such seems to have been contingent on proven professional experience, academic credentials as well as eloquence. To avoid undue influence of “convincing opinions”  especialHealth and societal implications of the recommendations were to be considered systematically by applying the WHO-INTEGRATE framework; they were further attended to by including representatives of those groups who would be concerned with or affected by the implementation of school measures in the panel. The expertise or “anecdotal evidence” of panel members regarding these aspects was the main source for these considerations given the lack of peer-reviewed studies as well as lack of professional expertise within the panel for some of the framework criteria which were also not formally prioritised and adapted to this guideline. However, the framework was mostly applied in the working groups of scientists with little time dedicated to the criteria in full group meetings that included practitioners and school family members; whilst health and educational consequences were discussed in these meetings, the framework’s criteria beyond these two areas were not usually systematically considered. The full potential of using this EtD framework might not have been harnessed given these limitations. It has been previously argued that EtD frameworks such as the GRADE EtD framework for health system and public health decisions can be readily used in rapid guidelines, too . Furtherper se but examined the strengths and weaknesses of taking this approach to developing public health recommendations during a global health emergency.Making use of scientific evidence and following formal consensus processes amongst scientists and affected stakeholders has been established as the gold standard for producing guidelines by the World Health Organization as well as many national institutions across the world, including the AWMF in Germany , 21. ThiThree of the authors  were members of the guideline secretariat and therefore potentially eligible to participate in this study; the two remaining authors  work in the same institution. The fact that this process evaluation was not conducted fully externally represents a main limitation of the study, as it may lead to social desirability bias in participants’ responses, or to selection bias in that participants holding strong positive or negative views regarding the guideline development process might be more likely to participate. Due to lack of dedicated funding, a truly independent evaluation of the guideline development process was not possible. We have been fully aware of this limitation from the start of the process and have sought to address any potential biases arising from it in the design and execution of the study. Importantly, recruitment, data collection and all but the very final steps of data analysis were solely carried out by KW and MR who also had exclusive access to the primary data.During recruitment we were actively seeking to reduce any selection bias that may arise from KW’s and MR’s potential knowledge of internal dynamics within the guideline development process; recruitment processes were exclusively performed by KW and MR. Instead of sampling participants purposively, all panel and secretariat members were approached and asked to express their interest in participating in the study. This process of self-selection might have led to the exclusion of potentially interesting perspectives of less extrovert or proactive individuals. Subsequently, random selection of participants amongst those who expressed their interest to participate was carried out. Representation of perspectives from all groups, including members of the school family, was achieved; however, the sample was not quantitatively balanced between the school family and those with a scientific background.During the interviews, acquiescence or desirability bias could have been present to not discredit the process or its output retrospectively. To counter this, we reiterated before the start of each interview that manuscripts would be fully anonymised, that potentially compromising statements would not be included in the analysis and that participants would be offered to review the transcript before analysis. Thus, we are relatively confident that participants did not refrain from disclosing their honest opinions and describing the process and interpersonal dynamics in rich enough detail to render this analysis meaningful. Recall bias might have been present as participants were asked to recount their experiences almost a year after the process had been initiated. Most of their answers referred to the first 3 months of developing recommendations for the short version of the guideline, although the interview questions did not stipulate this. No reflections on the difficulties and opportunities of developing “living guidelines” were elicited, however, we had also not included any probing questions about this in the interview guide.Whilst the development of main categories was jointly undertaken by MR and KW, the development of sub-categories was carried out by one researcher (KW) only, therefore inter-coder reliability was not established for this step. However, KW established intra-coder reliability by reviewing and critically examining the sub-categories repeatedly over a period of 3 months. Data were rich and saturation in terms of included perspectives and the content of each category was likely achieved. Translating quotes from German into English, undertaken by KW and checked by EAR, was necessary to present our findings to an international audience, however this might have led to some loss or change of meaning. The researchers’ own background in public health and medical science might have influenced data analysis, particularly regarding the categories ‘Understanding of evidence’ and ‘Understanding of expertise’ as a certain shared understanding of these concepts is central to their epistemological standpoint and scientific socialisation. However, a process of reflecting about their own positionality accompanied the process of data collection and analysis and subjective interpretations were minimised by constantly scrutinising findings and their interpretation. Being close to the subject matter of qualitative research can also be seen as an advantage as it supports interpretation of the data in a way that is practically relevant. KW developed a first draft of the manuscript with a focus on choosing quotes that would not compromise confidentiality and yet provide a holistic account of the primary data. During review and subsequent refinement of the results, some aspects of the data presented were interpreted differently by the co-authors closely involved in the guideline process . In these cases, KW presented further text passages and elaborated on their interpretation to ensure the analysis and conclusions were grounded in the data.The aim of producing an evidence- and consensus-based guideline on infection prevention and control in schools was to provide decision-makers and politicians with a salient basis for their decisions, rooted in scientific evidence and the expertise of a broad range of stakeholders, including scientists from across multiple disciplines, those tasked with the implementation of school measures and those affected by them. The rapid development of this public health guideline during a public health emergency was challenging, mostly due to enormous time pressure, with the first version of the guideline developed over the course of three  months only. More specific challenges were the limited evidence base, the lack of adequate financial and human resources as well as heterogeneity within the guideline panel and secretariat, notably regarding individuals’ understanding of reliable and valid evidence and experience with guideline development. The guideline development process was also characterised by some methods-related challenges, notably the use of a novel decision-making framework and the consultation of different types of expertise represented by a broad range of stakeholders beyond scientific societies. Learning from this process with a view to “institutionalising” the development of public health guidelines and refining public health-specific methodological approaches can contribute to more evidence-informed public health decision-making in Germany and beyond, in general and during a public health emergency.The raw data supporting the conclusions of this article will be made available by the authors upon request, without undue reservation.The studies involving human participants were reviewed and approved by Ethics Committee of the Medical Faculty, Ludwig-Maximilians-Universität München (No. 21-0944). Written informed consent to participate in this study was provided by the participants or their legal guardian/next of kin.EAR, BS and LMP conceived the overall research project. LMP, KW and MR jointly developed the protocol with substantial input from EAR and BS. Recruitment, data collection and all but the very final steps of data analysis were solely carried out by KW and MR who contributed equally to this research. All authors discussed the results and their implications. KW wrote the first draft of the manuscript. All authors critically reviewed the different versions of the manuscript, suggested revisions, and approved the version to be published.This research was undertaken without dedicated funding. Staff time was financially supported by the Chair of Public Health and Health Services Research at the LMU Munich.EAR, BS, and LMP are members of the guideline secretariat as well as initiators of this process evaluation, which represents a conflict of interest (see Methods and Discussion with regard to how this research sought to minimise any undue influence as a result of this conflict of interest).The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.


The spleen, often referred to as the “forgotten organ”, plays numerous important roles in various diseases. Recently, there has been an increased interest in the application of radiomics in different areas of medical imaging. This systematic review aims to assess the current state of the art and evaluate the methodological quality of radiomics applications in spleen imaging. A systematic search was conducted on PubMed, Scopus, and Web of Science. All the studies were analyzed, and several characteristics, such as year of publication, research objectives, and number of patients, were collected. The methodological quality was evaluated using the radiomics quality score (RQS). Fourteen articles were ultimately included in this review. The majority of these articles were published in non-radiological journals (78%), utilized computed tomography (CT) for extracting radiomic features (71%), and involved not only the spleen but also other organs for feature extraction (71%). Overall, the included papers achieved an average RQS total score of 9.71 ± 6.37, corresponding to an RQS percentage of 27.77 ± 16.04. In conclusion, radiomics applications in spleen imaging demonstrate promising results in various clinical scenarios. However, despite all the included papers reporting positive outcomes, there is a lack of consistency in the methodological approaches employed. The spleen, though considered for many years the “forgotten organ”, is well visualized in imaging of the left hypochondriac region of the abdomen . The splRadiomics, defined as the high-throughput extraction of a huge number of quantitative features from medical images, presents an innovative approach to capture these subtle variations and overcome the limitations of traditional imaging ,8. The rImage segmentation is the process of identifying and delineating regions of interest within the image, such as tumors or organs. A wide range of quantitative features is extracted from the segmented regions, which can include shape, texture, and intensity-based features . FinallyThe escalating interest in radiomics research has led to an increase in its applications across various medical imaging domains, particularly in the field of oncology ,13,14,15The radiomics quality score (RQS) has been proposed by Lambin et al. to quantify the quality of radiomic research by assessing all the steps of the radiomic pipeline .This systematic review aims to evaluate the state of the art and to assess the methodological quality of radiomics applications in spleen imaging.To identify all the relevant papers about radiomics applications in spleen imaging, a systematic literature search was independently carried out by two reviewers (S.C.F. and M.F.), accordingly to the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) guidelines . The exaAll the included articles were examined to extract the following data: year of publication, journal topic , number of patients, disease (primarily splenic or not) imaging technique, targeted organs (spleen or spleen and other organs), feature type (first-order or more), and study design. A methodological quality assessment of the included articles was carried out using the RQS tool  by two rAfter the automated deletion of duplicates (68) and the exclusion of non-inherent reports (15), 15 articles were finally included. The flowchart of the systematic search leading to the selection of the articles is shown in The majority of the articles were published in non-radiological journals  starting in 2020. In particular, two articles were published in 2020 (13%), seven in 2021 (46%), four in 2022 (26%), and two in 2023 (13%). In slightly more than half the studies, radiomic features of the spleen were extracted from CT images , and the remainder from MR images  and from 18F-FDG PET/CT . Only two papers addressed a disease primarily involving the spleen (13%), and in 11 studies (73%), radiomic features were extracted also from other organs, particularly the liver and the esophagus. The mean patient number was 231.66 ± 173.60 (range 70–326), and all the studies featured a retrospective design. The characteristics of the included articles are given in Overall, the included papers achieved an RQS total mean of 10.06 ± 6.29 (range −4–16), with a corresponding RQS percentage of 28.70 ± 15.87 (range 0.00–44.44). In the first domain, the majority of the studies  described the acquisition protocol, while only in a minority  were multiple segmentations carried out, and none collected images at additional time points. Regarding the second domain, feature reduction or adjustment for multiple testing was properly adopted in 14 studies (94%). Of fifteen studies, only in one (6%) was an external validation of the results carried out, while five (33%) did not conduct a definite validation without retraining. In the remainder , the validation was based on an internal dataset. Moving to the third domain, all the papers discussed the potential biological correlates of their results, and almost half  included a decision curve analysis to assess the potential clinical utility of the developed models. In the fourth domain, addressing statistical analysis, almost all the papers  reported discrimination statistics, including three also applying resampling methods. Regarding the fifth domain, none of the included articles reported a cost-effectiveness analysis. Finally, none of the studies made code and data publicly available. The detailed RQS assessment is reported in In recent years, there has been a growing interest in developing radiomic models for implementation in clinical practice, and spleen imaging has not been exempted from this trend. Radiomics has the potential to enhance spleen imaging by providing a quantitative and more objective perspective to the diagnostic process of primary splenic disorders. Enke et al. investigated the role of CT-based radiomics in differentiating malignant lymphoma of the spleen from non-lymphomatous lesions and found an area under the curve (AUC) of 0.86 . In addiHowever, what sets the spleen apart is its central role in immune function and regulation, which consequently leads to its involvement in systemic inflammatory diseases and even in the progression of primary tumors in other organs ,2,3,4,5.Yang et al. developed a radiomics score (Rad-score) to predict the 6-month survival of patients with Hemophagocytic Lymphohistiocytosis (HLH), a rare, life-threatening disorder of immune regulation that can potentially lead to end-organ damage and death . The autThe potential role of splenic features and characteristics was also investigated by Wang et al. with a different aim, namely the prediction of the survival rate of patients with early gastric cancer . The radAnother chronic condition where the spleen plays a significant role in the pathophysiology and complications is liver cirrhosis. Cirrhosis is the final endpoint of multiple liver diseases, such as viral infections and alcoholic or non-alcoholic steatohepatitis, and it has become one of the most common causes of death worldwide . Given tYin et al. found out that machine learning models incorporating CT splenic features outperformed models solely considering CT hepatic features when detecting the liver fibrosis stage . A similTo assess portal hypertension, Tseng et al. proposed a noninvasive predictive model of portal venous pressure based on liver and spleen CT-extracted radiomic features . The samEsophagogastric variceal bleeding is the most fatal of the consequences of portal hypertension in cirrhotic patients, associated with a high mortality rate . Among tFinally, another clinical scenario where radiomic features extracted from the spleen were helpful in clinical decision making is COVID-19. The value of lung parenchymal quantitative imaging biomarkers for COVID-19 diagnosis and severity assessment has been already widely proven in the literature ,40,41. HThe methodological quality of the included articles was assessed using the RQS . To our No specific thresholds have been established to categorize work as high or low quality based on this score. However, given the maximum score of 36 (100%), the average score of the reviewed papers, which is 10.06 (28.70%), may indicate suboptimal quality. It is worth noting that this result is not novel. Spadarella et al. already reported a similar overall mean of 18.87% in a paper reviewing all the RQS applications in the literature . Though In conclusion, this systematic review highlighted the growing body of literature addressing the potential role of radiomics application in spleen imaging, showcasing promising results in various clinical scenarios. However, it is evident that there is a lack of consistency in methodological approaches across the studies examined. While all the included papers demonstrated positive results, the methodological drawbacks hinder the ability to draw definitive conclusions. By establishing consistent methodologies, we can finally unlock the full potential of radiomics in enhancing our understanding and clinical management of spleen-related pathologies.


The living systematic review (LSR) approach is based on ongoing surveillance of the literature and continual updating. Most currently available guidance documents address the conduct, reporting, publishing, and appraisal of systematic reviews (SRs), but are not suitable for LSRs per se and miss additional LSR-specific considerations. In this scoping review, we aim to systematically collate methodological guidance literature on how to conduct, report, publish, and appraise the quality of LSRs and identify current gaps in guidance.A standard scoping review methodology was used. We searched MEDLINE (Ovid), EMBASE (Ovid), and The Cochrane Library on August 28, 2021. As for searching gray literature, we looked for existing guidelines and handbooks on LSRs from organizations that conduct evidence syntheses. The screening was conducted by two authors independently in Rayyan, and data extraction was done in duplicate using a pilot-tested data extraction form in Excel. Data was extracted according to four pre-defined categories for (i) conducting, (ii) reporting, (iii) publishing, and (iv) appraising LSRs. We mapped the findings by visualizing overview tables created in Microsoft Word.Of the 21 included papers, methodological guidance was found in 17 papers for conducting, in six papers for reporting, in 15 papers for publishing, and in two papers for appraising LSRs. Some of the identified key items for (i) conducting LSRs were identifying the rationale, screening tools, or re-revaluating inclusion criteria. Identified items of (ii) the original PRISMA checklist included reporting the registration and protocol, title, or synthesis methods. For (iii) publishing, there was guidance available on publication type and frequency or update trigger, and for (iv) appraising, guidance on the appropriate use of bias assessment or reporting funding of included studies was found. Our search revealed major evidence gaps, particularly for guidance on certain PRISMA items such as reporting results, discussion, support and funding, and availability of data and material of a LSR.Important evidence gaps were identified for guidance on how to report in LSRs and appraise their quality. Our findings were applied to inform and prepare a PRISMA 2020 extension for LSR.The online version contains supplementary material available at 10.1186/s13643-023-02396-x. Systematic reviews (SRs) are essential to provide evidence-based answers to clinical and public health-related questions. Due to the continuous publishing of relevant primary studies in some areas, it is important to keep these SRs up-to-date . One couFew guidance documents address the conduct, reporting, publishing, and appraisal of LSRs. The Living Evidence Network developed in 2019 the “Guidance for the production and publication of Cochrane living systematic reviews” . HoweverTherefore, it is of high interest to summarize the literature evaluating methods of conducting, reporting, publishing, and appraising LSRs, as well as any guidance on those methods. Scoping reviews are particularly useful in the context of emerging evidence and act as a precursor for other topic-related projects . This scThe main objective is to systematically collate methodological literature on guidance on how to conduct, report, publish, and appraise the quality of LSRs and to systematically map how much and what kind of evidence is currently available.A protocol elaborating on the detailed methodology of this scoping review was already published . The maiIdentification of the research questionIdentification of relevant studiesStudy selectionCharting the dataCollating, summarizing, and reporting of the results To achieve the objective, we conducted a scoping review to identify and evaluate existing evidence and map the availability of methods papers, evidence gaps, and associated primary research gaps . We follMoreover, we adhered to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) checklist  for undertaking LSRs, issued by organizations that conduct evidence syntheses, and commentaries or editorials that discuss methods for LSR.We excluded from our search, LSRs themselves, LSR protocols, and non-LSR-specific papers.We searched MEDLINE (Ovid), EMBASE (Ovid), and The Cochrane Library. All searches were completed on August 28, 2021, and we searched from database inception. The search strategy was initially developed by a researcher experienced in developing literature search strategies with support from an information specialist (LH), as part of a larger project to develop an extension of the PRISMA 2020 statement for LSRs , 11. TheAs for searching the “gray literature,” we looked for existing guidelines and handbooks on LSRs from organizations that conduct evidence syntheses  using the Lens.org website. Additionally, we conducted an ancestry search to identify relevant LSR handbooks and guidance documents from the reference list of published LSRs. We performed a descendency search, using certain seminal documents , and tracked their citations via Google Scholar.Two authors  contributed to screening independently and in duplicate titles and abstracts. We used a web-based systematic review software Rayyan (RRID:SCR_017584) for the screening process. To ensure a consistent screening procedure and optimize agreement, we developed and used a detailed written instruction form. We then screened for full text assessing eligibility, based on our predefined eligibility criteria. Disagreements and conflicts were solved by consulting a third author.Two review authors  extracted and cataloged the data on LSR-specific methodological aspects into a standardized and pilot-tested data extraction form in Microsoft Excel (RRID:SCR_016137). We extracted the main article characteristics and LSR-specific guidance data according to our predefined categories on (i) conducting, (ii) reporting, (iii) publishing, and (iv) appraising LSRs. The identified evidence was mapped by visualizing overview tables created in Microsoft Word. The items of the conducting category are based on the standard process of conducting a systematic review from the Cochrane Handbook , includiWe identified 4590 references, potentially relevant to our research question. After having removed 1171 duplicates, we screened 3436 records on title and abstract and excluded 3379 records that did not meet the pre-defined eligibility criteria. We screened the full text of the remaining 57 records and included 17 papers from the database search in the scoping review. We also searched for “gray literature” and identified 49 potential records, from which we included five papers in the scoping review. In total, 21 articles from both searches were included in the scoping review. The detailed selection process and results are reported in the PRISMA flow diagram , data items (10), effect measure (12), reporting bias assessment (14), and certainty assessment (15). Further, there was no guidance identified for the reporting of results, including study characteristics (17), presenting the risk of bias in studies (18), results of individual studies (19), results of synthesis (20), reporting bias (21), and certainty of evidence (22). No data was found on reporting the three items  of the discussion, on the item reporting support and funding (25), and on the availability of data and material (27).From the 15 papers including guidance on publishing LSRs, we mapped the available data for our pre-defined items and sub-items and summarized the identified guidance  Checklist. Table S3. List of extracted study characteristics and extracted items of the categories.


Background: Individuals born preterm  and/or at low birthweight (<2500 g) are at increased risk of health impairments from birth to adulthood. This review aimed to evaluate the psychometric performance of generic childhood-specific or childhood-compatible multi-attribute utility instruments (MAUIs) in preterm and/or low birthweight (PLB) populations. Methods: Searches covered seven databases, including studies that targeted childhood (aged < 18 years) and/or adult (≥18 years) PLB populations; provided psychometric evidence for generic childhood-specific or compatible MAUI(s) (any language version); and published in English. Eighteen psychometric properties were evaluated using a four-part criteria rating system. Data syntheses identified psychometric evidence gaps and summarised the psychometric assessment methods/results. Results: A total of 42 studies were included, generating 178 criteria rating outputs across four MAUIs: 17D, CHSCS-PS, HUI2, and HUI3. Moreover, 64.0% of outputs concerned the HUI3 MAUI, and 38.2% related to known-group validity. There was no evidence for five psychometric properties. Only 6.7% of outputs concerned reliability and proxy–child agreement. No MAUI outperformed others across all properties. The frequently applied HUI2 and HUI3 lacked content validity evidence. Conclusions: This psychometric evidence catalogue should inform the selection of MAUI(s) suited to the specific aims of applications targeting PLB populations. Further psychometric research is warranted to address the gaps in psychometric evidence. Individuals born preterm  and/or at low birthweight <2500 g) are at an increased risk of a range of physical, motor, cognitive, psychosocial, and behavioural problems that persist from the neonatal period into adulthood 500 g are,5,6,7. TThe health utilities generated using MAUIs can serve as inputs into economic evaluations that compare the costs and consequences of alternative interventions targeting PLB populations . SpecifiThere are nevertheless key challenges for measuring health utilities in childhood populations in general. First, rapid biopsychosocial development during childhood means that the relevant dimensions of HRQoL are likely to vary by childhood age . Second,Psychometrics concerns the performance of measurement scales and is applied in healthcare to develop scientifically rigorous patient-reported outcome measures (PROMs) of health, including MAUIs ,31,32. KA recent systematic review identified and synthesised evidence for the psychometric performance of 14 generic MAUIs specific to or compatible with childhood . The aim(1)Create a catalogue of evaluated psychometric evidence that can aid in the selection of generic childhood-specific or childhood-compatible MAUIs for application in PLB populations;(2)Identify gaps in psychometric evidence to inform future psychometric research in this population;(3)Summarise the commonly used psychometric assessment methods and the relative psychometric performance of instruments by property.Notably, the aggregated reporting of the identified psychometric evidence by the above review across all childhood population groups precluded any judgment on the relative performance of the MAUIs for specific clinical populations such as PLB populations. This specific focus is important given that the minimum standard set for psychometric performance likely varies across populations. To that end, the current study aims to conduct a systematic review of the published psychometric evidence of the 14 childhood-specific or compatible MAUIs in PLB populations. These MAUIs are potentially applicable to both children and adults; hence, studies that applied them in adult PLB populations were also included. This allows an evaluation of whether the psychometric performance of the MAUIs varies across childhood and adulthood. Previous systematic reviews of HRQoL in the PLB population focused on HRQoL differences between PLB groups and term-born and/or normal BW controls and not on the psychometric aspects of the MAUIs used ,36. The A pre-specified protocol outlining the systematic review methods was developed and registered with the Prospective Register of Systematic Reviews (CRD42023428176). The PRISMA 2020 guideline was followed : see the16D: 16-dimensional health-related measure 17D: 17-dimensional health-related measure AHUM: Adolescent Health Utility Measure AQoL-6D Adolescent: Assessment of Quality of Life, 6-Dimensional, Adolescent CH-6D: Child Health—6 Dimensions CHSCS-PS: Comprehensive Health Status Classification System—Preschool CHU9D: Child Health Utility—9 Dimensions ,45EQ-5D-Y-3L: EuroQoL 5 Dimensional questionnaire for Youth 3 Levels EQ-5D-Y-5L: EQ-5D-Y 5 Levels HUI2: Health Utilities Index 2 HUI3: Health Utilities Index 3 IQI: Infant health-related Quality of life Instrument QWB: Quality of Well-Being scale ,52TANDI: Toddler and Infant health related quality of life instrument ,54.The database searches aimed to identify studies targeting PLB populations that provide evidence for the psychometric performance of one or more of the following 14 generic childhood-specific or childhood-compatible MAUIs identified and evaluated in a systematic review :16D: 16-Of the above, the HUI2, HUI3, and QWB are childhood-compatible, and the rest are childhood-specific .An information specialist (NR) guided the database choice and designed the search strategy to maximise the coverage of studies that applied MAUIs in general and clinical populations. Seven databases were searched from the database’s inception to 26 April 2023: Medline, Embase, PsycInfo, EconLit, CINAHL, Scopus, and Science Citation Index. The search strategies are shown by database in There were three main inclusion criteria: (1) The study contained evidence for at least one psychometric property (see The exclusion criteria were: (i) the study used one of the MAUIs as a criterion standard to validate a new instrument; (ii) the study targeted patients with specific diseases common in preterm birth  rather than the PLB population more generally; (iii) the study developed and validated value sets for health utility derivation without assessing or providing evidence of the psychometric properties of the health utilities.Data from the included studies were extracted by JK, and 20% was independently extracted by OO and CB. The following data fields were extracted and stored in Excel: (i) bibliography; (ii) study country(ies); (iii) study design—e.g., RCT; (iv) direct or indirect assessment psychometric properties; (v) psychometric property(ies) assessed; (vi) methodological issues for psychometric assessment; (vii) MAUI(s) assessed—any language version(s); (xiii) MAUI component(s) assessed—e.g., utility score after valuation, dimension level response; (ix) value set derivation country and population; (x) respondent type(s)—self-report and/or proxy report; (xi) administration mode(s)—e.g., with interviewer; (xii) study population clinical characteristics including gestational age and birthweight; (xiii) target and sample age—e.g., mean, range—and proportion of females; (xiv) target and actual sample size; (xv) intervention(s) assessed (if relevant).This review evaluated the psychometric assessments conducted by the included primary studies in terms of their assessment methods and the resulting psychometric evidence. Comprehensive evaluation required context-specific judgements which are reported in the online Excel (1)Create a catalogue of evaluated psychometric evidence. The online Excel file serves as the main catalogue wherein the criteria rating outputs are tabulated and the main rationale for each rating. More condensed catalogues are presented in this manuscript.(2)Identify gaps in psychometric evidence. Two aspects were defined as evidence gaps: (i) no criteria rating output available for an MAUI for a property and (ii) no criteria rating output available or where available no ‘+’ output. The number of these cases was computed for the whole evidence base and for a subset of evidence involving PLB adult populations.(3)Summarise the psychometric assessment methods and performance of instruments by property. The psychometric assessment methods used by the included studies were described by property. The relative performance of MAUIs was compared by property, using the proportion of ‘+’ as the performance metric and also considering the absolute number of outputs.The criteria rating outputs were synthesised to address the three review objectives:Eight studies were judged by the review authors as having a direct aim of assessing the psychometric performance of MAUI(s) ,72,73,74The 42 included studies, in total, conducted 178 psychometric assessments. For the second review objective, This section addresses the third review objective and describes the psychometric assessment methods and performance of MAUIs for the properties with at least one criteria rating output. Only one criteria rating output was available for internal consistency. Verrips and colleagues  estimateThe only assessment of test–retest reliability was conducted by Saigal and colleagues  for the The CHSCS-PS was again the only MAUI with inter-rater reliability evidence concerning the level of agreement between parental and clinician responses . Percentp < 0.05) differences in the mean HUI3 utility score and in the mean HUI3 unweighted sum of dimension levels between mail and interviewer administrations.Verrips and colleagues  providedSaigal and colleagues  found hiContent validity evidence was available for the HUI2 and CHSCS-PS. Saigal and colleagues (1994)  perceiveBy contrast, Saigal and colleagues (2005)  providedA large proportion (38.2%) of identified psychometric evidence concerned known-group validity. When comparing the utility scores and/or dimension-level responses between the PLB group and full-term/normal birthweight controls, studies that stated their a priori hypothesis generally expected to find significantly lower HRQoL for the PLB group ,95,96,97Hypothesised subgroup differences were found (output ‘+’) in 50% of known-group validity assessments for the 17D, 100% for the CHSCS-PS, 63.6% for the HUI2, and 58.5% for the HUI3. However, the numbers of assessments were smaller for the 17D (n = 4) and CHSCS-PS (n = 1) than for the HUI2 (n = 22) and HUI3 (n = 41), precluding any straightforward judgement on the best performing MAUI.Evidence on hypothesis testing mostly comprised results of multivariate regression analyses. For example, Selman and colleagues  hypothesThree criteria rating outputs were available for convergent validity, one for the HUI2 and two for the HUI3. Feeny and colleagues  assessedSaigal and colleagues (2005)  conducteNo study conducted a longitudinal assessment of the effectiveness of a specific healthcare intervention targeting the PLB population. All six criteria rating outputs for responsiveness, therefore, concerned the natural history of the PLB population’s HRQoL measured by the HUI3. Ni and colleagues (2021)  found stTwelve studies assessed acceptability but only concerning one criterion , which meant that their assessments (n = 15) received the output ‘?’ ,83,96,97higher HRQoL than that observed in the general childhood population. No study derived the population norm or MID for HRQoL de novo; hence, none received an output of ‘+’.Evidence on interpretability mainly consisted of using the minimal clinically important difference (MID) sourced from the literature. For the HUI2 and HUI3, a change or difference of 0.03 in utility score and 0.05 in single-attribute utility score were cited to be the MID ,101. ForThis study is the first review of the psychometric performance of childhood-specific or compatible MAUIs in the PLB population. The psychometric evidence base developed from the 42 included studies should facilitate the selection of scientifically rigorous MAUI(s) for clinical research and health economic evaluations in this population, as well as motivate further psychometric research to fill the identified gaps in the current evidence base. The review also summarised the psychometric assessment methods and performance of the four MAUIs applied in this population  by psychometric property. No MAUI consistently outperformed the others across all properties for which evidence was available. This suggests that selection should depend on which properties are most relevant for the research and clinical practice setting in which the MAUI is applied. The CHSCS-PS had the greatest number of properties for which there was at least one ‘+’ output but had the lowest number of outputs from a single study and targeted a narrow age group of 2–4 years. The HUI3 was the most commonly applied childhood-compatible MAUI in PLB populations but had mixed psychometric performance across the properties and lacked any evidence for content validity.Other major gaps in the psychometric evidence base were identified for this population. First, the range of psychometric properties covered was narrow: five properties  lacked any evidence, and another seven  lacked any positive rating of evidence. The review revealed that known-group validity was the property with the greatest psychometric evidence, reflected in 38.2% of assessments. Evidence on reliability  and proxy–child agreement were particularly lacking, with only 12 outputs in total, representing just 6.7% of the 178 outputs. In comparison, for all childhood populations as identified in the previous psychometric review , evidencSecond, the range of MAUIs covered was similarly narrow, comprising only four. Moreover, the evidence volume was skewed towards the HUI system, with the HUI3 being applied in 64.0% of the identified assessments and the HUI2/3 in 89.9%. In comparison, the respective proportions of assessments by these measures were 28.9% and 50.9% for all childhood populations . The freContent validity is particularly important for the measurement of HRQoL in PLB populations, which typically adapt to disabilities such that their self-reported levels on the ‘subjective’ dimensions of HRQoL  are broadly comparable to those of their full term/normal birthweight peers, this phenomenon being labelled the ‘disability paradox’ . The subA key strength of this study is its focus on the psychometric performance of MAUIs in a specific population. The previous psychometric review covering all childhood populations provided top-line evidence only , while tThis study nevertheless has several limitations. First, although the coverage of adult PLB populations was a strength, the non-coverage of adult-specific MAUIs such as the SF-6D in adult PLB populations curtailed the range of psychometric evidence. Second, the assumption that the studies that did not state their a priori hypothesis were expecting a lower HRQoL for the PLB group than controls may have underestimated the psychometric performance of the MAUIs: a lack of significant HRQoL difference was interpreted as poor psychometric performance when it could have accurately reflected comparable HRQoL between the two groups . That saThis review points to several avenues of further research, most importantly those addressing the identified psychometric evidence gaps. First, there is a significant paucity of evidence from low- and middle-income countries. Second, there is a particular need for empirical validity evidence concerning the degree to which the MAUI utility values reflect people’s preferences over health, often measured by self-reported health status ,105. GivThis systematic review provides comprehensive and up-to-date evidence on the psychometric performance of generic childhood-specific or compatible MAUIs that have been applied in preterm and/or low birthweight populations. The catalogue of evaluated psychometric evidence provides a valuable resource for researchers and policymakers—particularly those involved in cost-effectiveness analysis, modelling, and decision-making—in selecting MAUI(s) for applications targeting this population as well as in interpreting the results of studies that applied the MAUI(s). No MAUI consistently outperformed others across all properties, meaning that selection would depend on which properties are most relevant for further application. Important psychometric evidence gaps were identified, which should motivate further psychometric research, such as the paucity of evidence around reliability and proxy–child agreement and the lack of evidence on content validity for the HUI3. The commonly observed issues in psychometric assessment design and methods, such as the clear statement of the a priori hypothesis for testing associations and changes, should likewise inform future psychometric studies.