Background: This is a review and analysis of the questionnaires most used in empirical research on psychological phenomena labelled as “presence,” “flow,” and “narrative absorption,” mostly for experiences mediated by technology . Overlapping concepts have been formulated in different fields according to specific disciplinary interests and based on knowledge within each field.Objectives: This review focuses on how language is actually used in questionnaire items, rather than on how concepts are formulated top-down and associated with corresponding linguistic expressions that become items of a questionnaire. The goal is to highlight similarities and overlaps in order to show a possible interdisciplinary agreement about the core aspects of the psychological states elicited by mediated experiences.Eligibility criteria: Questionnaires developed or used for research about VR, video games, films, or books have been selected for analysis. They should be available in English and used in empirical research since the year 2000.Sources of evidence: A search has been performed through Google Scholar and two other disciplinary bibliographies edited by international learned societies.Charting methods: The items of each questionnaire are categorized based on their wordings, and thus independently from the conceptual models within which they have been developed.  Based on this categorization, various domains to which the items can be ascribed are identified  and psychological phenomena are linked to them .Results: 308 items in 23 questionnaires have been found to have overlapping of wordings.Conclusions: A list of the core aspects of presence, social presence, flow, and narrative absorption is presented, together with a critical selection of items suitable to measure each construct. Experiences mediated by technology  are studied across a variety of disciplines, often with little cooperation. Different theorizations, models, and empirical tools have been developed, resulting in a fuzzy agglomerate of related and overlapping concepts, like presence  and I associated them to the respective psychological phenomena for which they are more frequently used .I followed Arksey and O’Malley’s framework for scoping reviews , refinedThe sources considered are questionnaires available in English, no year limit has been used. To be included in the review, questionnaires need to have been developed or used for research about one of the following media: VR, video game, film, book. I only included questionnaires measuring psychological states, not those measuring personality traits or broader psychological concepts . Validation and statistical reliability were not necessary criteria.International Society for the Empirical Study of Literature (IGEL), and themeasurement guides provided by the International Society for Presence Research (ISPR). Additional useful comparisons of presence-related concepts can be found inI performed the search in May 2020, using three sources: the aggregator Google Scholar, the bibliography of theThe queries used in Google Scholar are: “presence questionnaire,” “immersion questionnaire,” “flow questionnaire,” “narrative questionnaire,” “narrative engagement,” “narrative absorption,” “narrative transportation.”I obtained information about questionnaires directly from published articles and also from reviews included in Master theses or PhD dissertations. The criterion used to consider a questionnaire eligible as a source of evidence is its application in recent years: once I identified a questionnaire, I checked its use in research starting from the year 2000. I made this selection also with the help of a review of the questionnaires most used in VR research in the years 2016–17 , which can be considered subdimensions of narrative absorption. Given their metaphorical nature, items explicitly asking whether an experience elicited involvement, engagement, immersion, or absorption are not particularly useful for describing the psychological processes activated during the experiences they aim at qualifying. Moreover, the adjective “immersive” is used in VR research as a technical attribute of the medium – consistently with Sheridan seminal definition  – whereaAnother popular but quite heterogeneous group of questions concerns the emotional impact of mediated experiences. Ten questionnaires investigate this aspect in slightly different ways, so it is hard to say whether emotional impact is a component of any of the presence-related phenomena or a secondary effect elicited by them.The recognition presented can be used to reflect on the extent to which wording similarities among items from different questionnaires actually result from similarities between the underlying conceptualizations. One possible outcome is a cross-disciplinary systematization of concepts, suggesting viable options for an interdisciplinary agreement about the core aspects of the psychological states elicited by mediated experiences. To sum up, attention and time distortion are common to all the considered phenomena, and spatial presence (space and agency) is the phenomenon with the narrowest scope, the core. Social presence and narrative absorption are phenomena of increasingly broader scope, each of them including the listed phenomena of narrower scope. Flow is a concept transversal to the other three, being more related to the balance between a person’s skills and the complexity of the stimulus, rather than to a specific psychological dimension.Following the above-mentioned strategy, inCategorizing only 308 items, out of the total 484 found in the sampled questionnaires, this scoping review may have missed some aspects of presence and related concepts that are important to grasp the nuances of the phenomenal experience that may be specific to certain media. However, by focusing on items showing a recurring intersubjective agreement between researchers and disciplines, I think I have successfully identified and summarized the core aspects of the surveyed phenomena. However, it is worth remembering that the employment of measurement tools should always be justified by theoretical reflection and empirical validation. A scoping review is an aid for the systematization of knowledge, but it also produces new knowledge that requires further scrutiny and methodological testing before it can be deployed into experimental settings.The categorization proposed here can be used to further refine existing questionnaires and possibly encourage a convergence of different disciplines towards a use of the same items, so that insight coming from different fields could be used for the advancement of knowledge in specific areas. For instance, empirical research on narrative could benefit from using existing items for presence and social presence, without “reinventing the wheel” and focusing rather on refining how to measure dimensions like suspense and imagery. Moreover, a shared agreement on basic items will enable better and more informative meta-analyses, as well as comparative media studies, a kind of research that is strongly relevant for all the disciplines that I mentioned here, since only a comparison between experiences with different media can help to account for the specificity of presence and related phenomena.OSF: Presence, flow, and narrative absorption questionnaires: a scoping reviewhttps://doi.org/10.17605/OSF.IO/RBZ8G (O/RBZ8G This project contains the following underlying data:scoping_review_data_2021-02-26.xlsx (Human-readable version containing the 23 selected questionnaires with color coding of the items and summary model)scoping_review_data_2021-02-26.csv (Machine-readable version containing the 23 selected questionnaires with the respective annotations for each item)Data are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).Reporting guidelinesOSF: PRISMA-ScR checklist for ‘Presence, flow, and narrative absorption questionnaires: a scoping review’.https://doi.org/10.17605/OSF.IO/RBZ8G (O/RBZ8G Creative Commons Zero “No rights reserved” data waiver (CC0 1.0 Public domain dedication).Data are available under the terms of the This article represents an attempt to survey and synthesize the conceptually related but practically disparate fields of immersion, flow, presence and narrative absorption. The focus is on an analysis of the use of language in questionnaire development for these constructs, in order to better understand their shared higher order structure. This is such a fantastic article for its ability to synthesize the base information needed to determine the kind of questionnaire that may be required, or could be useful, for a lot of basic research in the articulated fields. I wish that this was available when I first started my research in narrative absorption! I hope I can add something meaningful beyond the other two excellent reviews.There’s a slight contradiction in the eligibility criteria section, “available in English, no year limit has been used.” You do mention a year limit of 2000 in the abstract.I’m also interested in the data charting process. For the use of the shortest version of the questionnaire, is there any documentation for what may be lost? There is almost certainly something lost in many questionnaires for deciding to use the short version over the long version, is there anything of note here?For “data items”, was “close similarities” determined through inter-rater standards, or just what looked “close”? Perhaps this is just a nuance of the scoping method and can be ignored.For “critical appraisal of individual sources of evidence”, is there any affordance for bivalent or complex experiences? Might this be included in a future study?Were all removed articles (or questionnaires - n=24) removed because of complexity? It would be useful to have a better sense of why these articles were removed.This is a fantastic first step, but I would love to see this article have at least one follow-up article – a tracing of theory in relation to linguistic use, as Christoph Klimmt points out, but it would also be useful to see how these concepts align with other pragmatic considerations . This would not only make it easier for researchers to know which questionnaire might work best for their needs, but it would also allow for an easier way to branch out into related, but perhaps slightly different methodologies. The only reason I say, "partly" for "Are the conclusions drawn adequately supported by the results presented in the review?" is that I would like to see a little more discussion regarding future studies for how this work may be built upon and made into a study that makes that extra connection with practical use and theory. I just have a few notes for minor revisions or future considerations:Are sufficient details of the methods and analysis provided to allow replication by others?YesAre the rationale for, and objectives of, the Systematic Review clearly stated?YesIs the statistical analysis and its interpretation appropriate?YesAre the conclusions drawn adequately supported by the results presented in the review?PartlyReviewer Expertise:Narrative absorption; metaphor; psychometrics.I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. The motivation behind Federico Pianzola’s analysis is the enormous number of questionnaires and scales that have been developed across various fields in which scholars investigate media users' state of being 'captured‘, 'absorbed‘, 'transported‘ or 'involved‘ by a message and / or its delivery technology. Because much media innovation aims at improving or intensifying such user experiences, thematic research is thriving and very productive. As a consequence, a hard-to-overview landscape of empirical measures has evolved, and oftentimes, these measures obviously utilize similar-sounding questions and items for assessing (presumably) different concepts. Pianzola offers a scoping review of the items used in some 20+ such measures to investigate the extent to which such "overlaps" occur, to gain a better orientation of the measures available, and to derive recommendations about which measure to use for which concept. The author’s goal of serving an interdisciplinary research community that is confused by the many different published questionnaires with orientation knowledge is relevant and laudable, and the overview of measures, items and linguistic similarities among items is highly valuable. However, from a psychological-methodological point of view, the idea of scoping the items of different questionnaires *across* and *independent of their* theoretical concepts, is a kind of "reverse engineering“ – because Pianzola tacitly tries to build an integrative *theory* of media absorption by synthesizing the *measures* used for different concepts. This does, in my humble view, not work at all. First, the key reason why there are so many different questionnaires and measures out there is that there are so many concepts and different elaborations of the same concepts , and various author teams have tried to establish their conceptual understanding and an according measure. The large number of measures available on the market is thus a consequence of theoretical diversity  – a normal situation in the social sciences in which there is no standard definition of entities, variables, or phenomena as it can be found in natural (hard) sciences. For instance, in social psychology, a huge number of "theories" is circulating, many of which differ from each other in similarly small nuances as conceptualizations of "Presence" differ in the current context. The multiplicity of similar concepts and of elaborations of the same concept is a notorious problem, and Pianzola is right that in the case of media absorption, in which many scholars from diverse fields are interested, the resulting complexity and chaos is particularly undesirable. – But this chaos cannot be resolved by distilling some 'optimal' choice of questionnaire items or scales from the abundance of available measures. This is because measures in psychology are developed *based on their theoretical foundation*. Using a specific conceptualization of, say, Spatial Presence, one author team has created their items to measure Presence and has done  research to validate their measure (against the background of their theory). It is certainly NOT the case that all research teams who have developed a questionnaire of Spatial Presence had the SAME basic conceptual understanding of Spatial Presence in mind when they crafted their items. It is therefore misleading that Pianzola writes "this review focuses on how language is actually used in questionnaire items, rather than on how concepts are formulated top-down and arbitrarily associated with corresponding linguistic expressions that become items of a questionnaire" – no author team would accept the assumption that they have "arbitrarily" linked item wordings to a concept; on the contrary, most authors have invested much thinking in finding item wordings that match their conceptual understanding. As a consequence, Pianzola’s (implicit) idea that we can reshuffle items from different measures to come up with best-choice approaches to assess certain concepts or conceptual dimensions disconnects each questionnaire from its individual theoretical base. This is not how social-scientific measurement works – it must always be "theory first – measurement second". Pianzola, however, disregards theoretical differences and suggests a kind of meta-measures based on wording similarities. Such wording similarities among items from different questionaires, however, actually result from similarities between the underlying conceptualizations, and maybe in some cases, from poor operationalizations . So I express a warning not to step over the necessity of theoretical integration and synthesis by simply searching for "good measures" based on linguistic similarity analysis. Measures only "work" within the context of their underyling theoretical substance, and this substance differs between all questionnaires, which cannot be ignored when comparing (or combining) measures. Hence, for epistemological reasons, the main objective that Pianzola is pursuing, cannot (and should not) be achieved. But the author’s review of the many similar-yet-different measures of the many similar-yet-(maybe?)-different concepts is nevertheless of great utility! First, the mere listing of available measures and the accompanying descriptions help to maintain an orientation of the existing diversity. And second, this scoping review must be understood by many scholars active in the field as a reminder that the notorious diversity and chaos that results from competing conceptualizations and individual desires to establish one’s own theory and measure (in spite of the existence of many other similar theories and measures) is a huge problem. The precise analysis of which items and questionnaires display which kind of linguistic overlap in spite of  conceptual discrepancies is a great contribution by Pianzola, because it helps to detect those spots in media absorption research that seem to require particular effort of theoretical re-thinking, integration, and synthesis. Practically speaking, scholars may refer to this scoping review when articulating (and justifying) their individual decision to use certain conceptualizations (and according measures) in their specific study, thus acknowledging the undesirable plurality of available concepts and measures and making an informed selection decision at the same time. So I encourage the author of this paper and all readers to reflect on the theory-measurement-relationship and to develop conclusions on how the highly diverse, cross-disciplinary research on media absorption phenomena can cope with the theoretical-methodological diversity and still maintain strong standards of excellence in social-scientific (self-report) measurement. Clearly, this question is a key challenge that hopefully will inspire many young scholars to work towards greater theoretical integration, harmony, and parsimony. Based on my own experience with measuring user states that occur during media exposure but that are only assessed cumulatively after exposure, I end with the recommendation of not being too ambitious regarding the ability of assessing  fine-grained differences in experience this way. Language that is comprehensible to laypersons is hardly capable to make experiential nuances  distinguishable. So one issue that Pianzola inspired me to reflect on is this specific aspect of the theory-measurement-relationship: Small theoretical discrepancies may simply not be possible to translate validly and precisely into item wording differences. So maybe media absorption research is in need of a new pragmatism in measuring concepts *in spite of* conceptual diversities. Authors of (slightly) differing concepts should thus feel invited to team up and debate whether they can agree on an integrated measure that would fit *both* their concepts. Now that would be a pathway of "bottom-up" integration of measures and hence a potentially viable way to move forward into the direction that Pianzola has shown to us.Are sufficient details of the methods and analysis provided to allow replication by others?YesAre the rationale for, and objectives of, the Systematic Review clearly stated?YesIs the statistical analysis and its interpretation appropriate?YesAre the conclusions drawn adequately supported by the results presented in the review?PartlyReviewer Expertise:NAI confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above. The precise analysis of which items and questionnaires display which kind of linguistic overlap in spite of  conceptual discrepancies is a great contribution by Pianzola, because it helps to detect those spots in media absorption research that seem to require particular effort of theoretical re-thinking, integration, and synthesis. Practically speaking, scholars may refer to this scoping review when articulating (and justifying) their individual decision to use certain conceptualizations (and according measures) in their specific study, thus acknowledging the undesirable plurality of available concepts and measures and making an informed selection decision at the same time. Klimmt also rightfully points out that the main objective of my work may be mistaken for a ready-to-use optimal choice of questionnaire items to be employed to measure presence, social presence, and narrative absorption. In light of this, I have now reworded the presentation of my suggestions, stressing that they can be a useful schema to foster more awareness about advancements in other disciplinary fields. Hopefully, this will also lead scholars to aim for epistemic comparability, in the spirit of open science. Indeed, participating in a collective scientific effort to improve the understanding of what presence and narrative absorption are is more valuable than pursuing a finely nuanced grasping of human experience via the use of rigidly structured proxies like psychometric questionnaires. I agree that social-scientific measurement must always be “theory first – measurement second” and I hope this scoping review will help to refine theories before a selection of questionnaire items is made. I have now made this explicit in the Discussion and Conclusion.I am very grateful to Christoph Klimmt for his thorough commentary. I am very glad to see that we agree on the main epistemological goal of my scoping review: The article provides a systematic overview of presence, flow, and narrative absorption questionnaires available in the literature. I believe this is an important work towards understanding how these concepts - which are commonly confused - are dealt with in the scientific literature. The study shows that, although there is a diversity of aspects that comprise the complex concepts of presence, flow, and narrative absorption, some themes are more frequently related to these concepts than others. This work is especially important in a context in which new VR tools are being increasingly used. Perhaps, in the future, we will focus less on trying to find a consensus towards the meaning of "flow" or "presence" and will focus on more pragmatic aspects such as "losing track of time"? One aspect I particularly like about this article is that all conclusions are based on data, and I could not find any overstatement regarding the study. It is clearly written. Used data is available for replication. The discussion session is especially exciting.In the Rationale (if there is space): the text states: "Overlapping concepts have been formulated in different fields according to specific disciplinary interests and based on knowledge within each field". This is absolutely true, and it is the main problem tackled in this work. Could the author provide one or two examples of these different definitions?In the third paragraph of the discussion, change "and the comprehension of the content of the story" to "the understanding of the story's narrative and characters".The idea of "reality non mediated by technology", which appears in the synthesis of results, refers to "actual, physical reality", as opposed to its virtual counterpart. Is this correct? Could the author include one or two sentences explaining the difference between mediated and non-mediated reality, and why is this different from differentiating "virtual" from "real"? Or, at least, provide a reference for further reading? None of these notes take away the relevance and quality of this work. I congratulate the author and have no further comments on this article. Some minor observations:Are sufficient details of the methods and analysis provided to allow replication by others?YesAre the rationale for, and objectives of, the Systematic Review clearly stated?YesIs the statistical analysis and its interpretation appropriate?YesAre the conclusions drawn adequately supported by the results presented in the review?YesReviewer Expertise:I have worked with presence and flow, although I currently do not use these terms anymore (due to the exact reasons stated in this article). Currently, I work with technology-mediated musical interactions and this article is highly interesting to my research.I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. I will add examples of how different definitions of the same concept are based on specific disciplinary knowledge.I agree that understanding characters is often considered an important aspect of subjective-phenomenal states related to narrative.Open Research Europe, 1(28), 1–25. https://doi.org/10.12688/openreseurope.13193.1I will elaborate on the difference between mediated and non-mediated experiences, a topic which I treated more at lengths in a complementary article based on this scoping review: Pianzola, F., Riva, G., Kukkonen, K., & Mantovani, F. (2021). Presence, flow, and narrative absorption: An interdisciplinary theoretical exploration with a new spatiotemporal integrated model based on predictive processing. Thank you very much for your positive feedback. Regarding your comments:


Laboratory Life Latour and Woolgar present an account of how scientific “facts” are formed through a process of microsocial interactions among individuals and “inscription devices” in the lab initially described as social construction. The process moves through a series of steps during which the details and nature of the object become more and more certain until all qualifications are dropped, and the “fact” emerges as secure scientific knowledge. An alternative to this account is described based on a Bayesian probabilistic framework which arrives at the same end point. The motive force for the constructivist approach appears to involve social processes of convincing colleagues while the Bayesian approach relies on the consistency of theory and evidence as judged by the participants. The role of social processes is discussed in Bayesian terms, the acquisition and asymmetry of information, and its analogy to puzzle solving. Some parallels between the Bayesian and constructivist accounts are noted especially in relation to information theory.In the book  Laboratory Life based on Latour's experience as a “participant observer” at the Salk Institute. Chapter 3 of that book is a case study of the “social construction of a scientific fact.” The “fact” in question is the existence and structure of thyrotropin releasing factor, TRF, for which Roger Guillemin and Andrew Schally were awarded the Nobel Prize in 1977. Latour and Woolgar claim to show how this chemical entity was brought into existence and assumed a factual status as the result of the process of “social construction.” The authors assert that its existence is due to social processes independent of its physical existence.In 1979 Bruno Latour and Steve Woolgar published their iconic book Laboratory Life that such a probabilistic perspective is not that far removed from the fact construction process that the authors endorse. However, the Bayesian and constructivist processes prove to be incompatible.The purpose of this paper is to show that an alternative and more plausible account of the taken-for-granted or “fact” status of TRF is possible based on a probabilistic treatment of hypotheses and evidence in a Bayesian framework. What emerges from this analysis is that the factual status of a biological substance such as TRF or DNA Small,  is due tLaboratory Life. A second edition appeared in 1986 to which a postscript was added but otherwise without major revisions. One notable change was dropping the word “social” from “social construction” which was possibly motivated by Latour's objections to the Strong Program in sociology . Wade's book is a detailed technical history of the search for various releasing factors whereas Latour and Woolgar's book is an extended argument for the “social constructivist” interpretation of this history. Wade, by contrast, focuses more on the competition for priority between Guillemin and Schally taking a more traditional, Mertonian view of scientific activity. In developing a Bayesian account of the TRF story, we rely on the Latour and Woolgar and Wade books, along with historical accounts by Guillemin and Schally themselves. However, the Bayesian account which we offer is an alternative to both the constructivist and Mertonian viewpoints.Another excellent source for the Guillemin and Schally pursuit of hormone releasing factors is Nicholas Wade's Laboratory Life which is entitled “The construction of a fact: the case of TRF(H).” Work on this “hormone releasing factor” was background to the research program of the Salk Institute lab under Roger Guillemin at the time of Latour's stay at the lab from 1975 to 1977  for special attention in Chapter 3 of Nevertheless, Latour and Woolgar distanced themselves from typical historical treatments which they claim take for granted the factual status and “reality” of the knowledge they are tracing historically. They were interested in the “process by which scientists make sense of their observations” . Theory and evidence can cohere or not cohere to varying degrees if, for example, a consequence of a theory agrees or disagrees with an experimental result, for example, by deriving it from theory. If the evidence is consistent with the theory, then by Bayes' theorem the probability of the theory given the evidence increases by induction unless an alternative theory is more consistent with the same evidence. Alternative theories can be either explicit or implicit.In assessing theories against evidence what needs to be determined is their degree of consistency or compatibility. That is, how securely is the evidence a consequence of the theory? This is expressed as a conditional probability of the evidence given the hypothesis or theory, expressed as P(E|T) which is called a likelihood. For example, in the history of hormone releasing factors, the anatomical fact that portal blood vessels connect the hypothalamus to the anterior pituitary was weakly consistent with the idea that some substance was being transmitted from one location to the other. However, this was circumstantial evidence and only weakly consistent. Subsequently it was also shown experimentally that cutting these vessels prevented the pituitary from releasing a hormone required for sexual reproduction, and this provided stronger evidence at a higher probability or likelihood.For historical reconstructions, the procedure is to estimate these probabilities from the point of view of the historical participants. For example, evidence generated by experiment or direct observation usually carries more weight than evidence based on rules of thumb or analogy to other systems. As shown in a previous study Small, , the proi, n) is the posterior probability of the theory given n forms of evidence, P(T) is the prior probability of theory T, and P(Ei|T) are the likelihoods for the various forms of evidence i. P(~T), the probability of “not T,” is the probability of the alternative hypotheses. P(~T) equals 1 – P(T) because the set of all possible theories must sum to 1. If the posterior is greater than the prior, the hypothesis T is regarded as confirmed and disconfirmed if it is less than the prior.When multiple lines of evidence are taken into account and those different types of evidence are conditionally independent  indicates that the likelihoods for each form of evidence are multiplied together. It is possible by simple algebra to show that this formulation is equivalent to each line of evidence considered individually provided the posterior of applying one form of evidence is used as the prior for the next form. The numerator is the product of the likelihoods and the prior for the theory being tested and expresses how well the theory fits the evidence. The first term in the dominator is the same as the numerator and the second term is the product of the likelihoods for the alternative theories times the prior probability of the alternative theories, 1 minus the prior. This second term expresses how well the alternative theories fit the evidence. If the posterior equals the prior, the theory is not confirmed or disconfirmed which occurs, for example, if the product of likelihoods of T equals the likelihoods of ~T. Setting P(T) and P(~T) to 0.5 is an example of uniform priors and corresponds to the state of the highest uncertainty and maximum entropy as will be shown later. This is sometimes referred to as the “principle of indifference” Howson, .We will present each model or theory as a table with the prior probabilities, P(T) and P(~T), given at the top of the table followed by the conditional probabilities for each form of evidence with respect to both T and ~T (the likelihoods), and the computed posterior at the bottom of the table. The posterior can then be computed using the formula (1) above from the data in the table. For each new model or theory, the prior probability is set to the posterior probability of the previous model in the series provided the object of inquiry, or what the model is supposed to explain, has not changed. This can be interpreted psychologically as meaning that the participants confidence in the new hypothesis is dependent on their previous experience with other models.Latour and Woolgar divide the releasing factor story into two parts. The first part was establishing that “releasing factors” exist in the hypothalamus which control the secretion of various hormones by the pituitary. The chief proponent of this idea was the British physiologist Geoffrey Harris whose 1955 book was entitled “Neural control of the pituitary gland.” This early work was carried out primarily on a proposed factor CRF (corticotropin releasing factor) which was thought to cause the pituitary to release ACTH (corticotropin). ACTH in turn stimulated the adrenal glands to secrete cortisone. Hans Selye considered ACTH part of the stress response and called the hypothetical CRF the “first mediator” because it initiated the response. The idea that the hypothalamus released different “factors” for each pituitary hormone was countered by the physiologist Samuel McCann at the University of Pennsylvania who thought that the hormone vasopressin (or oxytocin) from the posterior pituitary controlled the release of ACTH Wade, , p. 71.The second part of the releasing factor story was to figure out the chemical composition for one of the factors. By about 1960, work had shifted away from the CRF-ACTH system to the TRF-TSH system (thyrotropin releasing factor—thyroid stimulating hormone) due to the difficulties encountered in purifying enough CRF Schally, , p. 68.The quest for the chemical structure of TRF went through several phases. The first phase, up to about 1965, dealt with circumstantial evidence that the factor was some kind of peptide. The second phase, covering the years 1965 to 1968, focused on the idea that the factor was a simple peptide consisting of three amino acids. This claim was eventually rejected, and as shown below, leads to Bayesian disconfirmation. The third phase, which occurred during the single year, 1969, was that TRF is a modified peptide that was somehow blocked or protected at both of its ends. This phase consisted of three steps: first postulating that TRF was blocked at one of its terminals by an acetyl group, second that it was blocked instead by a pyro ring, and third that it was also modified at its other end by an amide group. The reasoning that led to this final form can be interpreted as a Bayesian process leading to a high posterior probability for the structure which accords with Latour and Woolgar's conclusion that a scientific fact had been constructed.Beginning with the first question, whether releasing factors from the hypothalamus exist, The prior probability P(T) is set to 0.5 to reflect the initial uncertainty regarding the theory prior to the evidence being considered. This automatically fixes the value of the alternative hypothesis to 1 – P(T) = 0.5 since the sum of all hypotheses cannot exceed 1. The conditional probabilities show the consistency or inconsistency of each line of evidence with both the target and alternative theory.In this early period up to about 1955, the evidence for hormone releasing factors from the hypothalamus was modest and somewhat contradictory. The evidence in In 1956 Andrew Schally took a position with Guillemin at Baylor College of Medicine in Houston where they worked together on CRF. After 5 years and no clear results, Schally decided to move on and accepted a position at the Veterans Administration Hospital and Tulane University School of Medicine in New Orleans, Louisiana where he could set up his own lab. In 1960 Guillemin had taken on an additional appointment at College de France in Paris and divided his time between Houston and Paris. He decided to set the CRF problem aside and look for other postulated releasing factors such as LRF (luteinizing releasing factor) and TRF (thyrotropin releasing factor). He located a slaughterhouse in France that could provide sheep hypothalamic tissue and used that material to pursue those other factors. Schally, on the other hand, decided to switch to another species for hypothalamic tissue, namely the pig, and arranged with Oscar Mayer and Company to provide the tissues Schally, , p. 94.An important development in focusing attention on TRF was Guillemin's development of an assay for TSH (thyroid stimulating hormone) which was the hormone released by the pituitary supposedly in response to TRF. The assay involves injecting the thyroid of rats with radioactive iodine which is incorporated into the animal's thyroid hormones. The rats can then be injected with a TRF-like test substance, and the presence of radioactive TSH in their blood indicated if the test material was acting as real TRF. A Geiger counter measured the amount of radioactivity in the blood samples and indicated the purity of the TRF test sample Wade, , p. 149.In Based on the principle of indifference, as in the case of the existence of releasing factors, we assign a prior probability of 0.5 (even odds) that releasing factors are peptides making the alternative hypothesis (factors are not peptides) also 0.5. The three lines of evidence are all indirect and weakly consistent. E1 is based on an analogy to the peptide structures of vasopressin and oxytocin from the posterior pituitary discovered by Vincent du Vigneaud Hofmann,  for whicAs research efforts focused on TRF, evidence accumulated that both supported and contradicted TRF's peptide make up. At the same time purification methods advanced, for example, using Sephadex columns for separation and the development of other chromatographic methods Schally, . The fraWhile it was possible to detect the amino acids glutamic acid, histidine, and proline , three of the four lines of evidence pointed to the non-peptide nature of TRF and a strong disconfirmation of the simple peptide hypothesis.In 1964 Guillemin, together with a chemist Darrell Ward at nearby M. D. Anderson Hospital, had concluded that TRF was a peptide consisting of 11 amino acid units. In 1965 this was revised to 18 units consisting of his, glu and pro (E1) whose order was unknown. In 1965 Schally found TRF to consist of 23 units of the same three amino acids. However, the peptide portion of the TRF molecule comprised only 8 percent by weight of the purified material in Guillemin and Ward's experiments, while in Schally's case, the amino acids accounted for 30 percent (E2). Thus, most of the purified material was non-peptidic in nature for both investigators. In addition, if TRF were a peptide it would be degraded by proteolytic enzymes, but this was not the case. Some biological activity was still observed after treatment with proteases (E3). Finally, Schally asked colleagues at Merck to synthesize all six permutations of the three amino acids , but none of the permutations showed any biological activity (E4). By 1966 both teams had declared that TRF was not a simple peptide  as Schally had done earlier with Merck. Six weeks later Burgus determined, as Schally had earlier, that none of the six showed TRF activity. However, he then acetylated each form and found to his surprise that only one of the permutations, acetyl-glu-his-pro, showed activity. According to Wade this was the most thrilling moment of Burgus' career. However, the activity of the acetylated tripeptide was less than the activity of the purified natural material Wade, , p. 150.2 at the N-terminal was consistent with the presence of another chemical entity such as an acetyl group (E1). This idea is also consistent by analogy with other natural peptides having this structure (E2). The unexpected activity of one of the acetylated sequences (acetyl-glu-his-pro) was consistent with acetyl-glu-his-pro being similar to actual TRF (E3). Finally, the acetyl hypothesis is inconsistent with its lower biological activity than purified TRF (E4). Nevertheless, taking all the evidence into consideration, the hypothesis is confirmed, a reversal from the disconfirmation of the previous simple tripeptide model in The ironic twist is that Burgus was only partly correct. The activity he thought was due to acetyl-glu-his-pro was a chimera. Back in Houston, Burgus heard from chemists at Hoffmann-La Roche who had provided the tripeptides that glu-his-pro might have formed an internal ring at the N-terminal during acetylation. The ring was called pyro-glutamic acid or pyro-glu and it was known that several natural proteins had pyro-glu terminals Wade, , p. 154.2) is not clear. Once again analogy may have played a role because in an article published in June, 1969 they stated “…there exist a number of biologically active polypeptides of which the terminal amino acid has been amidated”  with diverse types of evidence rather than social processes. “Consistency” is a two-part relation involving a theory component and an evidence component where the theory leads in some manner to the evidence. But what then is the role of social processes in constructing this knowledge?Of course, millions of microsocial interactions occurred in the 15-year history of work on TRF from the early postulation of hypothalamic “factors” by Harris to Guillemin and Schally's breakthroughs of 1969. What we need to ask, though, is whether the theory and evidence at different stages were dependent on social processes? For example, if we examine Turning to 2 group but was somehow protected, an idea that was first broached at the meeting of the endocrinology study section at Tucson. Guillemin's call to an acquaintance at Hoffmann-La-Roche in Switzerland to see if they could provide synthetic tripeptides paralleled Schally's earlier communication with Merck. Hoffmann-La-Roche also provided Burgus with the key suggestion that acetylation could have caused a pyro ring at the N-terminal to form, and as well as synthesizing the pure materials that led to pyro as the active form. It is doubtful that the Hoffmann-La-Roche researchers realized the theoretical implications of their information. Burgus, as Guillemin's chemist, would have realized the implications of these findings for TRF before Guillemin did, but, of course, would have shared them with Guillemin as the lab head. It was known that Guillemin was not actively involved at the lab bench during this period. Burgus' report that he was elated at finding only one of the six acetylated tripeptides was active shows that he realized its implications for TRF structure.Prominent social interactions for TRF as a protected peptide , 6 were For Burgus and Guillemin, support for the amide form  appears Social process such as reading prior literature, formal presentations, the sharing of materials, and direct communication and collaboration with colleagues facilitated “fact construction” and knowledge formation, providing at times important pieces of theory or evidence. However, like relevance judgments in linguistics  or P(E|~T) which express how well the theory T or ~T (not T) fit the evidence E. We need only assign these conditionals approximately taking the points of views of the participants. P(E|T) is equivalent to the conditional expression if T then E Log2P(Ti) where P is the probability of an event, the log is base 2, and the summation is over all possible states of the system P(E|T)Log2P(E|T) where the double summation is over T, ~T, E and ~E. This expression has a maximum value of 1.0, maximum uncertainty, when P(E|T) and P(E|~T) = 0.5 in which case the theory is neither confirmed nor disconfirmed. If either P(E|T) or P(E|~T) is 0.5 then the entropy has the highest average value when the other conditional is at 0.5. Thus, a conditional probability of 0.5, designated as “neutral” in Another result from information theory that has a bearing on Bayesian analysis is the concept of conditional entropy , the probability of evidence E being consistent with theory T, where E can be an empirical observation or a theoretical claim.Social processes in science, such as formal or informal communication, act as a suggestion box for potential changes in theory or evidence, and either facilitate or retard the acquisition of theoretical ideas and new forms of relevant evidence. Most of the suggestions communicated prove to be dead ends but a small fraction turn out to be decisive for advancing confirmation or disconfirmation. However, the fit between theory and evidence can only be perceived by individuals in possession of both components because confirmation is driven by two-place conditionals. In many cases of social interaction in the TRF case, knowledge by the parties involved was asymmetric, with one party knowing both the theory and evidence and able to assess the fit, and the other party knowing only one side of the story. Controlling the flow of information—who knows what—is important in the competition for priority but can impede confirmation. Thus, social processes can accelerate or decelerate new knowledge or “fact” formation but do not do the work of confirmation.The Bayesian account of the TRF story arrives at the same end point as Latour and Woolgar's constructivist account, namely highly probable knowledge. But the two accounts arrive at that end point in quite different ways and are ultimately incompatible. The motive force for the constructivist approach involves microsocial processes and convincing colleagues, while the Bayesian approach is driven by the consistency of theory and evidence as judged by each participant. The actors may agree with one another and achieve a consensus, but it is a consensus of individuals who have each separately perceived the theory-evidence fit. In the Bayesian approach the judgement of consistency must precede consensus and the convincing of colleagues.While we have used the terms model, theory, and hypothesis interchangeably, it is not difficult to see how the Bayesian methods could be extended to other types of discourse, for example, scientific methods, instruments, and even technologies. A method or technology, like a scientific theory or model, is directed toward an end. One such end is the structure of a hormone, and another is a method to assay a hormone or to separate a complex mixture. A method can be compared with competing methods or techniques according to their advantages and disadvantages using the same types of conditional probabilities we have used to assess theories and models. These Bayesian extensions into technology remain to be explored.The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.HS conceived the study, collected and analyzed the data, and wrote the paper.


Low health literacy is widely reported in people with chronic kidney disease (CKD) and has been associated with reduced disease self-management, poor health outcomes, increased mortality and poorer quality of life. However, these associations are still not well understood.Electronic-based systematic searches were performed to identify studies examining associations between health literacy and self-management behaviours and/or health outcomes in patients with CKD. A tabular and narrative synthesis of the data was performed. Meta-analysis was not appropriate due to the heterogeneity of study designs and methods.Searches identified 48 studies that met the inclusion criteria. A total of 41 published articles, six conference abstracts, and one thesis were included. Of the 48 studies, 11 were cohort and 37 were cross-sectional. In total there were 25,671 patients; 16,952 from cohort studies. Median study sample size was 159 (IQR 92–275). Study quality was high (5), moderate (24) and poor (19). Thirteen measures of health literacy were used. Despite the limitations of the available evidence, there appear to be consistent relationships between higher health literacy and favourable self-management behaviours for patients with CKD. Definitive relationships between health literacy and patient outcomes are far less clear and remain incompletely understood.Conclusive evidence describing a causal link between health literacy and patient outcomes remains limited, but for many outcomes, a consistent association is described. In addition to associations with mortality, hospitalisation and clinical events, there were consistent associations between health literacy and favourable self-management behaviours which could support the development of patient education aimed at improving health literacy.The online version contains supplementary material available at 10.1007/s40620-022-01537-0. The term health literacy refers to the cognitive and social skills, which establish the motivation and ability of an individual to gain access to, and to understand and use information in ways that promote good health, prevention of disease and improve quality of life (QoL) . Early cSelf-management behaviours refer to the ability to learn and practice skills and behaviours which allow people to better manage their day-to-day health, and include behaviours related to medication adherence, physical activity, managing symptoms, information seeking and interaction with healthcare providers . Health A previous systematic review evaluated the relationship between health literacy, mortality and patient outcomes in individuals with CKD . Whilst This systematic review aimed to update and expand the previous review by examining the relationships between health literacy and self-management behaviours and health outcomes. Better defining these relationships will allow better planning of prospective studies and the design of complex interventions that seek to improve health literacy and self-management behaviours to improve patient experience and outcomes for people with CKD.The PRISMA flow diagram is shown in Fig. (i)Were retrospective, prospective, cross-sectional, longitudinal, case-control, cohort studies or randomised controlled trials (RCTs). Conference abstracts were included where sufficient data was present and data were not published elsewhere. Review articles, drug intervention studies, trial protocols, qualitative studies, and case reports were excluded, along with studies that were not in the English language.(ii)2) including those requiring renal replacement therapy . Studies containing participants with CKD stages 1 and 2 exclusively were excluded.Included adults, children, and adolescents with CKD stages three to five (estimated glomerular filtration rate (eGFR) less than 60 mL/min/1.73 m(iii)Used an accepted tool for the assessment of health literacy .(iv)Reported associations between health literacy and self-management behaviours and/or health outcomes as part of study outcomes.Studies were considered eligible if they:Electronic-based searches from the date of inception to December 2021 were performed to identify studies examining associations between health literacy and self-management behaviours and/or health outcomes in patients with CKD. The databases searched were: MEDLINE (Ovid), EMBASE (HDAS), CINAHL Plus (EBSCO), The Cochrane Library, PsycINFO (EBSCO), OpenGrey and Scopus. Trial registers searched included: ISRCTN Registry; ClinicalTrials.gov; WHO International Clinical Trials Registry Platform (ICTRP). Database searches were supplemented with internet searches (Google Scholar), contact with trial authors, experts and research groups, and identification of key citations from included trials and review articles. Search terms were adapted to database requirements and the full search strategy for the MEDLINE database is shown in supplementary material (appendix S1).Search results were managed using Endnote  referencing software. Full texts were obtained and reviewed for articles meeting the inclusion criteria. Two reviewers independently reviewed two articles using the data extraction tool (an adapted version of the Cochrane Data Extraction Template) to ensure suitability. From therein, reviewers extracted data from each article including study design, sample-size, recruitment method, inclusion/exclusion criteria, primary aims, reported associations between health literacy and outcomes, and statistical testing. Crosschecking was performed by a second reviewer (R.E.B).k = 0.354; 95% CI 0.292–0.416). Discrepancies were resolved with the inclusion of a third reviewer (R.E.B).Titles and abstracts were screened by two reviewers (A.T and S.F.A) based on the criteria below. Agreement between the two reviewers was 86.28%  using a modified Newcastle Ottawa Scale, and final grading was decided following discussion. Quality scoring is presented within the supplementary material (appendix S2). Statistical significance was accepted as Data are presented in tabular form including study characteristics, sample sizes, demographics of participants, health literacy measurement tools, outcome measures, associations tested between health literacy and self-management behaviours and health outcomes in univariate and multivariate analyses, covariates used, and any significant associations that were found. A narrative synthesis of findings, detailing the association between health literacy and self-management behaviours and/or health outcomes was also performed. Meta-analysis was not appropriate due to the heterogeneity of study designs and methods.Searches identified 48 studies that met the inclusion criteria Fig. . A totalTable Three cohort and 24 cross-sectional studies including a total of 5367 patients, explored elements of self-management. Study qualities were: 2 high, 15 moderate, and 10 low. Several studies reported on self-management behaviours as a whole, and others reported individual behaviours.In 410 patients with CKD stages 1–5, health literacy was independently associated with self-management behaviours . HoweverIn a cohort study of 99 KTRs, limited health literacy was independently associated with higher odds of medication non-adherence . Health One study of 452 patients on haemodialysis reported that lower health literacy was associated with lower adherence to dietary and fluid intake recommendations . Higher No association between health literacy and physical activity was shown in 137 patients with CKD stages 3–5 . HoweverHealth literacy was not associated with tobacco use in 72 patients on dialysis , 137 patSeven cohort studies with a median sample size of 480 IQR 260–2274) and follow-up times between 12 and 35 months examined the association between health literacy and mortality. One study was high quality and six studies were moderate quality. Six studies were of patients on haemodialysis and one study was of kidney transplant candidates. Three studies showed no association between health literacy and mortality in 2820 patients on haemodialysis 0–2274 an. One stuEight studies  explored the association between health literacy and hospitalisation . Study quality was: three low and five moderate. Lower hospitalisation rates were associated with the HLQ domain ‘actively managing my health’ in 221 patients on haemodialysis after adjustment for age, comorbidity, and education . Lower hSix studies  totalling 3096 patients  explored the association between health literacy and kidney function. One study was low quality and five studies were moderate quality. In a cohort study of 74 patients with CKD stages 3–4, patients with inadequate or marginal health literacy had lower eGFR than those with adequate health literacy . LimitedCross-sectional analysis of 2340 patients with CKD stages 2–4 showed those with limited health literacy were more likely to report a history of cardiovascular disease after adjusting for age, gender, race, BMI, and education . In unadTen cross-sectional studies explored QOL and/or outcomes relating to cognition with a total of 1643 patients . Four studies were moderate quality and six were low quality. In 187 patients with CKD stages 3–5, significantly better QOL was found in patients with higher health literacy . This stFour studies  evaluated the association between health literacy and aspects of the transplantation process. Study quality was: one high, two moderate, and one low. In a cohort study of 2274 patients receiving dialysis, limited health literacy was associated with a reduced likelihood of receiving a kidney transplant of any type but was not associated with pre-emptive transplant listing . In a coSeven studies  explored the association of health literacy and vascular access. Study quality was: two high, three moderate, and two low. Patients with lower health literacy were more likely to choose a catheter over arteriovenous fistula (AVF) in a cross-sectional analysis of 56 patients on haemodialysis . Similarn = 37) than cohort studies (n = 11), but cohort studies tended to be larger accounting for 16,952 of the patients included. Study quality was highly variable with only five studies classified as high quality. This variability, the heterogeneity of study designs and the limitations of statistical analyses mean it is not possible to define causal links between measures of health literacy and outcome measures assessed in this systematic review. The heterogeneity of the study designs and the wide variety of tools used to assess health literacy and outcomes meant it was also not possible to meta-analyse the available data. Fewer than one-quarter of studies assessed the relationships between health literacy and ‘hard’ outcomes such as mortality or hospitalisation, and there were limited numbers of studies that included KTRs or patients on peritoneal dialysis. No studies included paediatric patients. Whilst 27 studies explored the relationship between health literacy and self-management behaviours, only three of these were cohort studies. In a previous review of the relationships between health literacy and patient outcomes in patients with CKD [This systematic review synthesised data from 48 studies and 25,671 patients. There were more cross-sectional studies Below is the link to the electronic supplementary material.


The combination of KD and physical activity can significantly reduce body weight and increase the levels of LDL-C and cortisol. In addition, KD was associated with seizure reduction in children, which can be explained by the ketosis state as induced by the diet. Furthermore, KD demonstrated a better alleviation effect in refractory childhood epilepsy, in terms of median effective rates for seizure reduction of ≥50%, ≥90%, and seizure freedom. However, the strength of evidence supporting the aforementioned associations was generally weak, thereby challenging their credibility. Consequently, future studies should prioritize stringent research protocols to ascertain whether KD interventions with longer intervention periods hold promise as a viable treatment option for various diseases.Numerous studies have examined the effects of ketogenic diets (KD) on health-related outcomes through meta-analyses. However, the presence of biases may compromise the reliability of conclusions. Therefore, we conducted an umbrella review to collate and appraise the strength of evidence on the efficacy of KD interventions. We conducted a comprehensive search on PubMed, EMBASE, and the Cochrane Database until April 2023 to identify meta-analyses that investigated the treatment effects of KD for multiple health conditions, which yielded 23 meta-analyses for quantitative analyses. The evidence suggests that KD could increase the levels of low-density lipoprotein cholesterol (LDL-C), total cholesterol (TC) and high-density lipoprotein cholesterol (HDL-C), the respiratory exchange rate (RER), and could decrease total testosterone and testosterone levels (all  The ketogenic diet (KD) was defined as a diet that induces ketosis, which is characterized by a very low carbohydrate content, limited to 5–10% of total daily calorie intake . KD was In the present day, there exist four primary types of ketogenic dietary therapies (KDTs): the classic ketogenic diet, the modified Atkins diet (MAD), the medium-chain triglyceride KD (MCTKD), and the low glycemic index treatment (LGIT) . AdditioStudies have shown that, under a low-carbohydrate and high-fat diet, glycerol produced from triglyceride (TG) hydrolysis could generate 16–60% glucose in the liver to directly supply a small amount of energy to the central nervous system in the early stage ,8. In adAnother concern about the use of KD is its impact on blood lipids. Animal studies have shown that KD can lead to lipid abnormalities ,21, but To date, numerous meta-analyses have been conducted to assess the effects of KD intervention on health-related outcomes. However, there are various biases . Therefore, this umbrella review aims to systematically collect meta-analyses on the efficacy of KD in managing diverse health problems, and to evaluate the reliability and strength of evidence for these interventions.Protocol and registration details for this umbrella review have been pre-registered with PROSPERO  under the registration number CRD42023413116. The study followed the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) reporting guideline . Mo. Mon = 6s: 0.01)  , population characteristics, and the design of original studies appear to be the sources of heterogeneity in intervention outcomes. Finally, the majority of articles included in our review assessed the short-term health outcomes of the KD intervention, but limited long-term follow-up data were available, which could reflect the effect of KD on cardiovascular, renal, gastrointestinal disease, and so on. As a result, there is a lack of evidence regarding the long-term efficacy of the KD, such as its ability to sustain weight loss over time or the potential for increased adverse health outcomes. Based on current findings, there is weak evidence that KD can be used as an effective treatment for epilepsy and effectively improve blood lipids among patients with T2DM. Additionally, the strength of evidence that KD combined with exercise reduced body weight in overweight and obese individuals and reduced respiratory exchange rates in athletes is weak. However, most interventions only examine short-term effects, and there is a need for more comprehensive investigations that examine the long-term implications, adherence, and potential side effects associated with KD. Furthermore, individual variations in responses to KD should be taken into account when considering its effectiveness in different populations.


Comments in PubMed are usually short papers for supporting or refuting claims, or discussing methods and findings in original articles. This study aims to explore whether they can be used as a quick and reliable evidence appraisal instrument for promoting research findings into practice, especially in emergency situations such as COVID-19 in which only missing, incomplete or uncertain evidence is available.Evidence–comment networks (ECNs) were constructed by linking COVID-19-related articles to the commentaries  they received. PubTator Central was used to extract entities with a high volume of comments from the titles and abstracts of the articles. Among them, six drugs were selected, and their evidence assertions were analysed by exploring the structural information in the ECNs as well as the sentiment of the comments . Recommendations in WHO guidelines were used as the gold standard control to validate the consistency, coverage and efficiency of comments in reshaping clinical knowledge claims.The overall positive/negative sentiments of comments were aligned with recommendations for/against the corresponding treatments in the WHO guidelines. Comment topics covered all significant points of evidence appraisal and beyond. Furthermore, comments may indicate the uncertainty regarding drug use for clinical practice. Half of the critical comments emerged 4.25 months earlier on average than the guideline release.Comments have the potential as a support tool for rapid evidence appraisal as they have a selection effect by appraising the benefits, limitations and other clinical practice issues of concern in existing evidence. We suggest as a future direction an appraisal framework based on the comment topics and sentiment orientations to leverage the potential of scientific commentaries supporting evidence appraisal and decision-making.The online version contains supplementary material available at 10.1186/s12961-023-00969-9. We revealed that the sentiment orientation (positive/negative) of comments is largely aligned with the recommendations for specific interventions for COVID-19 in WHO guidelines.Comment topics were found to cover all significant points of evidence appraisal and beyond.Half of the critical comments emerged 4.25 months on average earlier than the guideline release.We show the consistency, coverage and time-efficiency of comments as an evidence appraisal tool.We suggest that a rigorous comment-based evidence appraisal from the perspective of comment sentiment and comment points can leverage the potential of scientific commentaries in evidence appraisal and decision-making.Implementation science seeks to promote the uptake of research and other evidence-based findings into practice . EvidencEvidence appraisal, the critical evaluation of published studies, plays an important role in differentiating rigorous science from weak science , 11. To Published research commentaries are formal and short communications such as letters to the editor and editorials that reflect commenters’ viewpoints by neutrally commenting on, supporting or challenging research publications –17. SuchAre the sentiment orientations in commentaries consistent with the strength of recommendations in clinical practice guidelines?Are the topics in commentaries aligned with the core concerns of evidence appraisal in developing clinical guidelines ?Do critical comments provide a faster approach to shaping the evidence than the released guideline recommendations?We pose the following research questions:A workflow diagram of our approach is shown in Fig. N = 5379)Keyword “(Covid-19[MeSH] OR Covid-19[Title/Abstract]) and hascommentin”: identifying publications (evidence) that have COVID-19 in titles or abstracts and include at least one comment (N = 5863)Keyword “(Covid-19[MeSH] OR Covid-19[Title/Abstract]) and hascommenton”: identifying published comments that have COVID-19 in titles or abstracts and comment on at least one publication  first read all 56 full texts of comments regarding corticosteroids and labelled the topics of interest and sentiment orientations for each comment separately. After the initial annotation of corticosteroids was completed, a group meeting was held with a third reviewer (JD), who reconciled disagreements. Finally, an annotation guideline was completed, and a reviewer (SW) labelled the remaining 320 comments.Comment topics were categorized based on Kastner et al.’s categorization frame of “letter to the editor” . In geneOnce the comment topics were determined, we identified the sentiment orientation of the overall comment: supportive, critical or neutral. After going through the comments’ full texts in the above section, reviewers located the sentences with clear sentiment and then manually evaluated the document sentiment orientation Table .Table 2EAfter labelling all full-text articles, 21 comments were excluded because they were in Spanish and could not be translated accurately or because an evidence–comment pair was matched incorrectly in PubMed. Finally, 168 evidence articles  and 376 accompanying comments were included in this study. Two groups of pairs were included for a total of 427 pairs: evidence–comment pairs (354 pairs representing comments on primary research articles) and comment–comment pairs (73 pairs representing comments on previous comments). In this study, we analysed both groups and used “evidence–comment” to refer to all of them. Figure Cytoscape 3.9.0 software was utilized to draw evidence–comment networks (ECNs). In an ECN, each publication  is represented as a node. Edges represent the relations of evidence–comment pairs, with the direction of the edge pointing from one comment publication to the article on which it comments.For each drug, the largest connected subgraph was analysed in depth for elaborating comment-driven evidence assertions. Specifically, first, each primary article was manually reviewed for a research claim. Next, comments for each article were read to identify the comment sentiments  and comment points  towards the given claim. For example, if a randomized controlled trial (RCT) research claim that “HCQ is effective on COVID-19” received five critical comments, then a comment-driven assertion here would be that “HCQ is effective on COVID-19 is negated”. Such assertions indicated the evidence appraisal results by leveraging the rich information provided by comments towards the primary evidence. Third, at the subgraph level, all appraised evidence results were aggregated to conclude an integrated assertion on the specific topic. Comment-driven evidence assertions of each drug’s largest one or two subgraphs were then summarized and compared with the final recommendations in the WHO guidelines.Overall directed ECNs of all six drugs were drawn as shown in Fig. For example, in Fig. Significantly, when Alizargar commented on two articles claiming the efficacy of HCQ + A, he also commented on Rosenberg et al.’s research, which found no evidence of efficacy for either HCQ or A or the combination of both (HCQ/A/both), to integrate conflicting studies and evidence , 35. In A relevant absolute conclusion was not reached from the above ECN. However, a more conclusive claim could be developed as more evidence–comment pairs joined together with consistent sentiments. In Fig. Except for LPV/r, the detected effectiveness from the top subgraphs of the five drugs  were consistent with the respective WHO guidelines. Specifically, for HCQ and tocilizumab (IL-6 receptor blockers), we looked at the largest subgraph, and for the other four drugs we went through the largest two subgraphs. For a more in-depth analysis of the ECNs for the other five drugs, see Additional file Comments can also reflect uncertainties in evidence. Research commentaries may publish original research and case reports that may be insufficient to produce an original research article but cannot be neglected as vital evidence to supplement the field literature . In the After a detailed analysis of the largest two subgraphs, the overall sentiment orientation for all the subgraphs was computed for each drug. The overall sentiment could help further identify the overall effectiveness propensity of each drug for researchers and clinicians. We found that the overall sentiment orientation fully aligned with the recommendation propensity of WHO guidelines.We calculated the overall sentiment in the commentaries for each drug group. Without considering specific topics that were discussed  and only considering the distribution of comment sentiment orientations, we found that the results were aligned with recommendations in WHO guidelines. Specifically, WHO guidelines recommended using IL-6 receptor blockers (tocilizumab/sarilumab) and corticosteroids for patients with severe or critical COVID-19. Only IL-6 receptor blockers  and corticosteroids  received more supportive comments than critical comments Table . HCQ . These were consistent with the conditional recommendations for/against the use of the drug  in WHO guidelines, suggesting the reliability of comment-driven evidence assertion. Detailed evidence and comment examples for these three drugs are listed below:To probe the coverage of comments, we compared the comment topics with the concerns of WHO guidelines and other evidence appraisal systems. We aimed to uncover whether comment topics were in accord with the core concerns of evidence appraisal criteria in the development of clinical practice guidelines and whether they went beyond these topics.The quality of evidence, especially the research methodology, is the most crucial factor in developing guidelines and recommendations. Nevertheless, having high-quality evidence alone does not entail a strong recommendation. Other factors affecting recommendations include clinical applications , patient values and preferences, and costs, among other factors. , 47The distribution of comment topics is plotted in Fig. Specifically, the top three subcategories of methodology topics were analysis (14.04%), intervention (10.38%) and study design (8.46%); in clinical theme topics, these were clinical practice-related (11.54%), biological mechanisms (5.00%) and clinical evidence-related (3.65%); in other topics, these included just mentioned (8.08%), ethical issues (2.12%) and evidence-based medicine or experience-based medicine (0.96%).As the results revealed, the points that commentaries focused on were in accord with the concerns of current grading systems . As an overall clinical evidence appraisal system, GRADE mainly focuses on two aspects, namely factors that impact evidence quality (study design) and factors that impact the strength of recommendations, which are both linked to comment topics . When evThough strongly related, GRADE concentrates on RCT or other observational trials for the certainty of evidence. Comments can contribute to various aspects of decision-making beyond science, such as politics, economics, availability and feasibility. In the “other” category of our comment topics, especially in the subcategory of “just mentioned”, commentators mention the research evidence without detailed appraisal but discuss other related topics. For example, Self et al.  demonstrConsistency and coverage are prerequisites to enabling comment-driven evidence appraisal to aid clinical policymaking. However, what makes it the most competitive is the timeliness. We analysed the comment time span for each drug. Specifically, we extracted the date of the first published critical comment for each drug and the first half of critical comments published compared to the date of WHO guidelines publication, in order to determine to what extent assertions shaped by critical comments are faster than the final released recommendations in WHO guidelines.We plotted the comment time span of each sentiment orientation for each drug, as shown in Fig. Interestingly, for corticosteroids and IL-6 receptor blockers, all critical comments happened before the publication of guideline recommendations, which shows the acceptance of these two drug candidates. Since corticosteroids and IL-6 receptor blockers both fight virus-associated cytokine release syndrome in severe or critical cases instead of directly suppressing viral replication, as the safety concerns are solved, the controversy may gradually dissipate. By contrast, HCQ and remdesivir, two controversial candidates, continued receiving new critical comments even after the recommendations against their use in the WHO guidelines.If an informatics approach could help detect critical signs or assertions faster, it would provide strong clues for clinical guideline development. Furthermore, the timeliness of the comment-driven methodology makes it a powerful approach for evidence appraisal when fast decisions are needed in urgent situations.The present study validated how research commentaries can appraise clinical evidence and impact the shaping of knowledge, focusing on COVID-19 and six well-known drugs that have been used to treat it. Our results revealed the effectiveness of the largest subgraphs of five drugs (the exception being LPV/r) derived from relevant comments for predicting the subsequent recommendation in the WHO guidelines. The overall sentiment orientations derived from comments for each drug were fully aligned with guidelines, showing the consistency of comment sentiment. Further, comment coverage analysis revealed that methodology, clinical themes and ethical issues are core topics discussed in comments. This was well matched with the core concerns of WHO in their guideline recommendations and even went beyond these, including political and societal issues. Finally, for efficiency analysis, half of the critical comments appeared on average 4.25 months earlier than the release of the guidelines, which makes comment-driven assertions a timely appraisal tool. For clinical research, the median and mean time from publication of an article to the publication of a comment is 4 months and 6 months, respectively .Generally, a short time window of the formal publication between evidence and comment adapts to one of the essential components—keeping updated—in the development of living guidelines. Compared to 2–3 years required to develop traditional guidelines , the emeScientific commentaries are an important method of scholarly communication, but they remain underutilized. Horton firstly pointed out that “failure to recognize the critical footprint of primary research weakens the validity of guidelines and distorts clinical knowledge” . He streHow to take advantage of this support tool in a simple but powerful way becomes a worthy discussion point. Here, based on our research findings, we suggest a comment-based appraisal framework based on the comment topics and sentiment orientations, as shown in Table For a given article, each comment would be analysed to identify its comment topics and comment sentiment. Under a specific comment theme, all sentiment orientations would be synthesized for a cumulative score. All scores of all themes would then be further combined to obtain the final assessment score. For different scientific questions, comment themes of interest could be filtered in producing the most relevant theme scope and weights, to help prioritize the most relevant comment topics.The Lancet by using controlled vocabulary, such as “lacks equipoise”, to represent journal articles (comments) [Based on the proposed framework, automatic identification of comment topics and sentiments for evidence appraisal automation would also be essential in future work. Therefore, to take advantage of commentaries for evidence appraisal, we suggest that medical journals label comment sentiment and topics once comments are accepted for publishing. This could help create structured data for calculating comment-driven evidence assertions. Sahin et al. expressed a similar view in omments) . This coomments)  (includiomments) , 54) as omments)  can be dOur study has some limitations. Firstly, we only included six drugs and one WHO baseline living guideline in this study to show the feasibility of the approach. Considering the small sample and the inconsistency among guidelines of different countries, the generalizability of this approach could be further validated by including more drugs and more guidelines in the analysis. Second, although crucial information of specific subgroups or phenotypes are involved in comments, we did not represent COVID-19 at finer granularity regarding the severity of the disease in annotation. Thus, the overall comment sentiment was towards COVID-19 as a whole rather than a specific severity and failed to align with the conditional recommendation for severe cases in the current version of the guideline . Third, Based on the consistency, coverage and efficiency performance, we conclude that research commentaries could be used to support evidence appraisal by providing clues that indicate the importance and validity of evidence to support evidence-based decision-making, especially in emergency situations like the COVID-19 pandemic. Scientific commentaries have a selection effect by appraising the benefits, limitations and other clinical practice issues of concern for existing evidence. It is notable that negative comments could provide a more detailed understanding regarding specific diseases/interventions not by only criticizing prior studies but also by introducing new viewpoints or evidence. Comments also have the potential to inform decision-making regarding both therapeutic efficacy and topics such as economics, politics and ethical issues, which are crucial aspects in health policy-making.Additional file 1. Additional materials.Additional file 2. Adherence to reporting guidelines.


While the burgeoning researcher and practitioner interest in physical literacy has stimulated new assessment approaches, the optimal tool for assessment among school-aged children remains unclear.The purpose of this review was to: (i) identify assessment instruments designed to measure physical literacy in school-aged children; (ii) map instruments to a holistic construct of physical literacy ; (iii) document the validity and reliability for these instruments; and (iv) assess the feasibility of these instruments for use in school environments.This systematic review  was conducted in accordance with the Preferred Reporting Items for Systematic Review and Meta-Analysis (PRISMA) statement.Reviews of physical literacy assessments in the past 5 years (2017 +) were initially used to identify relevant assessments. Following that, a search  in six databases  was conducted for assessments that were missed/or published since publication of the reviews. Each step of screening involved evaluation from two authors, with any issues resolved through discussion with a third author. Nine instruments were identified from eight reviews. The database search identified 375 potential papers of which 67 full text papers were screened, resulting in 39 papers relevant to a physical literacy assessment.Instruments were classified against the Australian Physical Literacy Framework and needed to have assessed at least three of the Australian Physical Literacy Framework domains .Instruments were assessed for five aspects of validity . Feasibility in schools was documented according to time, space, equipment, training, and qualifications.Assessments with more validity/reliability evidence, according to age, were as follows: for children, the Physical Literacy in Children Questionnaire (PL-C Quest) and Passport for Life (PFL). For older children and adolescents, the Canadian Assessment for Physical Literacy (CAPL version 2). For adolescents, the Adolescent Physical Literacy Questionnaire (APLQ) and Portuguese Physical Literacy Assessment Questionnaire (PPLA-Q). Survey-based instruments were appraised to be the most feasible to administer in schools.This review identified optimal physical literacy assessments for children and adolescents based on current validity and reliability data. Instrument validity for specific populations was a clear gap, particularly for children with disability. While survey-based instruments were deemed the most feasible for use in schools, a comprehensive assessment may arguably require objective measures for elements in the physical domain. If a physical literacy assessment in schools is to be performed by teachers, this may require linking physical literacy to the curriculum and developing teachers’ skills to develop and assess children’s physical literacy.The online version contains supplementary material available at 10.1007/s40279-023-01867-4. Physical literacy is lifelong holistic learning acquired and applied in movement and physical activity contexts. It reflects ongoing changes integrating physical, psychological, social and cognitive capabilities. It is vital in helping us lead healthy and fulfilling lives through movement and physical activity. A physically literate person is able to draw on their integrated physical, psychological, social and cognitive capabilities to support health promoting and fulfilling movement and physical activity — relative to their situation and context — throughout the lifespan,” as described in the Australian Physical Literacy Framework (APLF)  were used to identify instruments (subjective or objective) specifically designed for the purpose of assessing physical literacy in school-aged children in the school setting. In this review, including ‘physical literacy’ in the name of an instrument may not necessarily meet the review inclusion requirements. As the aim was to identify instruments designed to measure a holistic construct of physical literacy, instruments needed to assess at least three domains of the APLF . Instruments that met these criteria and addressed additional elements outside of the APLF were also included.Searches were conducted by health faculty librarians on 20 July, 2022 for physical literacy instruments in school-aged children (not preschool or early years) that may have been missed/or published in (or since) the existing reviews in the past 5 years in six databases  . The search strategy, including all identified keywords and relevant subject headings , was adapted for each included information source. The key concepts and search terms were Concept 1: ‘Child’, Concept 2: ‘School’, and Concept 3: ‘Physical literacy’. Please see the Electronic Supplementary Material for the final search plan including alternative terms for the concepts. Table motivation might be assessed by more than one survey item within an instrument/assessment. The converse could also apply if the item was assessed as meeting more than one of the APLF elements. For example, the item might measure psychological aspects of engagement and enjoyment and social aspects of collaboration. If instruments assessed additional elements to those assessed in the APLF, they were mapped to the appropriate domain or new domains were created.Instruments that met the included criteria were classified against the APLF (by one author and then checked with a second author) in terms of which of the 30 elements they assessed. Within the coding process, it was possible for two (or more) items in an instrument to be matched to only one element in the APLF. For example, test content , response processes , internal structure , relations with other variables, and the consequences of testing . Specifically, for relations with other variables, age, sex, motor skill competence, physical literacy, and physical literacy over time were considered and reported on. Physical activity was not included, as this was not always considered part of the definition of physical literacy. The included instruments were assessed for each of these validity aspects (by one author and then checked with a second author) and then the evidence categorized as: supporting (✓), partially supportive ( ~), not supported (x), or not yet tested/reported (–). Please see Table The Standards for Educational and Psychological Testing  providedtest content evidence was assessed using a modified matrix developed previously  (n = 8), or the cognitive (n = 5) domains. The social domain was typically not assessed [Eight systematic or narrative reviews were identified Fig. . These rcal] n = , or the assessed –18 was tfun, PLAYbasic (a shortened version of PLAYfun), PLAYself, and PLAYcoach (counted as four instruments) . Six were from Canada, three from China, one from Australia, one each from Iran, Portugal, the UK, and the USA. There were seven self-report instruments, one designed for children (PL-C Quest), five designed for adolescents , and one without an age specification (IPLA). One proxy-report instrument was designed for coaches (PLAYcoach). A further four had mixed assessment approaches including self-report and observation .A total of 14 instruments were included in our synthesis (nine from prior reviews and five from the updated search) . Additionally, both these PLAY instruments had large positive correlations with the Canadian Agility and Movement Skill Assessment (CAMSA) motor skill obstacle course  and small-to-moderate correlations with a self-reported measure of physical activity . A suite of PLAY instruments was tested in children aged 8–13 years  and initial partial support for construct validity  [relations with other variables, as authors reported predictive consistency between scales and components over the testing period of 2 years for the different year groups [There was support for the iations) . For stuiations) . Finallyr groups .content validity was based on a construct map and literature review All scale items were aligned with the National Standards and grade level outcomes for K-12 PE published by SHAPE and theories of physical literacy prescribed by Whitehead Expert feedback was provided by academic staff [internal structure. There was good internal consistency reliability for each subscale and the total 22-item instrument (0.93) and adequate construct validity . All items loaded greater than 0.40 in the final model [For the PPLMS (one PhD thesis located ), evidenic staff . There wal model .consequences evidence [. No study reported on using any of the included instruments in children with disability.Only one study published evidence . That stevidence . While tbasic, 8. PLAYcoach, 9. PLAYfun, 10. PLAYself, 11. PL-C Quest, 12. PPLA-Q, 13. PPLI, and 14. PPLMS). Please see Table Only the instruments with more than one aspect of validity evidence were considered for feasibility. , followed by the PL-C Quest (median 11.5 min), and then the PPLA-Q (27 min). The remainder did not report a completion time .fun or basic) taking 5–10 min, the seated component (PLAYself) also taking 5–10 min to administer and PLAYcoach does not have an administrative time reported. However, one study noted that the PLAY tools were time consuming as a whole package [The PLAY instruments seem to take the least time with the objective components  also require space, equipment for the objective components, and a level of training for administering these sections. The CAMSA requires two staff to administer and while the number of staff is not reported for PLAYfun/basic and the PFL, it is likely that two staff would also be needed for a class, i.e., one to administer and one to supervise the remaining children.A recent paper documents an R analysis package  that autThis review identified 14 tools, mainly from Canada, designed to measure physical literacy in children and adolescents. Overall, the assessment approaches with more validity evidence  were thEven though these assessments had more validity and reliability evidence than other assessments, there was little evidence for consequences evidence. A recent paper has begun to question the consequential validity of physical literacy assessment instrument use  in physical education settings . It is aA clear gap for all assessments is validity and reliability evidence for instruments when the population includes children with disability. For example, one of the studies on the PFL noted that a gap was understanding students with special needs . Instrumand/or antecedent of physical literacy, but not as an essential domain of physical literacy [When considering instrument breadth in terms of domain, the PL-C Quest was designed to map to the APLF and therefore assessed four domains (and 30 elements) of physical literacy. Other instruments that assessed more than ten elements across all four APLF domains were the PFL, PLAQ, PPLA-Q, CAEPL, CAPL version 2, and APLQ. Some instruments added additional domains and/or elements to those included in the APLF, potentially adding to a holistic mapping of physical literacy. Eight assessments incorporated physical activity (including sedentary behavior for two instruments) as an additional domain to the APLF. The position of the expert panel during the initial development of the APLF was that physical activity can be considered a consequence literacy . What thspecific sports skills) was added to three instruments , with these instruments designed for adolescent populations. The addition of sports may make the instruments more relevant to adolescents, as the context of skill performance is then acknowledged. This supports the psychological theory that as children cognitively develop, their capacity to self-report in the physical domain changes to one that is more differentiated [power and body image were each added to one instrument and body composition was added to one instrument. Power could be a relevant addition to a holistic framework, although this would increase the number of physical elements and this domain already outweighs the other domains. Body image may be an important psychological element to consider including in a holistic physical literacy framework, as a scoping review identified positive body image as linked to physical activity and sport behaviors in adolescents (30% of the study samples) [Another aspect of instrument breadth or holism is the range of elements assessed. An additional element in the physical domain (entiated . Other asamples) . IncludiSurvey-based instruments are the most feasible to administer in school settings and they can potentially reach larger populations/samples as a result, with the shortest being the PPLI and PL-C Quest. However, a key reason these instruments are shorter is that they do not provide an objective assessment of movement skills or fitness and therefore do not require more than one teacher to administer. Some instruments included an objective assessment of motor skill , with the CAMSA (part of the CAPL version 2) reasonably efficient to administer as it is done as a class group . Motor skill competence is an important component of physical literacy , and objA broader consideration of feasibility (seven articles located) is whether school personnel have the capacity, interest, and requirement to implement a physical literacy assessment. This discussion goes beyond the choosing of assessments for the school setting . The neeNot having an explicit link to the curriculum is likely to be a primary barrier to physical literacy assessment in schools . The insThe strengths of this review include a thorough search, a comprehensive approach to validity assessment, and broad coverage of feasibility. Applying instruments to the APLF may be seen as a limitation depending on what definition of physical literacy the reader subscribes to, but even so, for those interested in physical literacy assessments that span multiple domains, this process should still have value. It also provides a template approach for others wishing to follow a similar process with other frameworks. It is important for transparency to anchor any physical literacy paper within the definition subscribed to. For example, an earlier paper conducted a conceptual critique of three Canadian physical literacy assessment instruments for school-aged children in terms of how well they related to Whitehead’s conception of physical literacy . ReportiA total of 14 physical literacy assessment instruments were identified, with at least five  having evidence for at least three validity aspects. Three instruments assessed four domains of the APLF and more than half the elements . Survey-based instruments were the most feasible to administer in schools, although a comprehensive assessment may arguably include some objective assessments.Supplementary file1 (DOCX 65 KB)Below is the link to the electronic supplementary material.