This systematic review is one of the three which sought to identify measures commonly implemented in parenting program research, and to assess the level of psychometric evidence available for their use with this age group. This review focuses specifically on measures of child social–emotional and behavioral outcomes. Two separate searches of the same databases were conducted; firstly to identify eligible instruments, and secondly to identify studies reporting on the psychometric properties of the identified measures. Five commercial platforms hosting 19 electronic databases were searched from their inception to conducted search dates. Twenty-four measures were identified from Search 1: a systematic search of randomized controlled trial evaluations of parenting programs. For Search 2, inclusion/exclusion criteria were applied to 21,329 articles that described the development and/or validation of the 24 measures identified in Search 1. Thirty articles met the inclusion criteria. resulting in 11 parent report questionnaires and three developmental assessment measures for review. Data were extracted and synthesized to describe the methodological quality of each article using the COSMIN checklist alongside the overall quality rating of the psychometric property reported for each measure. Measure reliability was categorized into four domains . Measure validity was categorized into four domains . Results indicated that supporting evidence for included measures is weak. Further work is required to improve the evidence base for those measures designed to assess children’s social–emotional and behavioral development in this age group. PROSPERO Registration number: CRD42016039600.The online version of this article (10.1007/s10567-019-00277-1) contains supplementary material, which is available to authorized users. Social–emotional and behavioral problems in infancy are common  as well as a lack of agreement as to which measure should be accepted as the single standard  evaluations of parenting programs delivered antenatally and/or for parents of children up to and including 5 years. Specifically, we were interested in measures that provided an assessment of the child’s behavior, social and emotional development, and cognitive outcomes. Aim 2 was to identify and synthesize the current evidence base for each of the included measures psychometric properties via a second systematic search of the scientific literature. The rationale for focusing specifically on commonly used measures within RCTs of parenting programs was twofold. Firstly, we wished to identify the breadth of child outcome measures being commonly adopted for evaluation purposes, and secondly, we sought to recommend a small battery of reliable and valid outcome measures that could be used by both researchers and practitioners seeking to evaluate change. Throughout the remainder of this review, evidence for each of the included measures psychometric standing will be conceptually organized according to their reliability and validity using the terms and definitions applied by the COSMIN checklist  undertook a domain-mapping exercise as recommended by Vaughn et al. . The intSearch 1 focused solely on identifying high-quality parent program evaluations, i.e., RCTs; consequently, the literature searches were restricted only to peer-reviewed items. Included studies (1) presented primary research relating to the evaluation of a parenting program using an RCT design. Studies reported a randomly allocated treatment and comparison group ; (2) included samples that included expectant parents, mothers and/or fathers, or other types of primary carer, of children up to and including the age of 5 years ; (3) described a parenting program that was structured, manualized, delivered by trained facilitator, and designed to improve some aspect of child social and emotional wellbeing or behavior; (4) reported on at least one relevant parent–child outcome which had been developed and validated independently of the RCT; (5) were published in the English language within the period 1995–2015. Papers were excluded if they met the inclusion criteria but (1) there was insufficient information to determine eligibility (where a scan of full text could not provide missing information), and (2) the manuscript was not available to download in full-text format from host Universities library, Endnote, Paperpile, or Google Scholar.A total of five commercial platforms hosting 19 scientific databases were searched in November 2015, with only studies published after January 1995 included because of increasing prevalence of RCTs. Databases were searched in English. An example of the search strategy used for retrieving relevant papers from each of the 19 databases is as follows:parent* training* OR parent* program* OR parent* education OR parent* intervention* AND toddler OR infant OR pre*school OR bab*y OR child* OR pregnancy OR antenatal AND experimental OR randomi?ed controlled trial.See online resource Fig. 1 for a flowchart depicting article retrievals. The databases that were searched were Arts and Humanities Citation Index, ASSIA, British Nursing Index, CINAHL plus, Cochrane Library, Conference Proceedings Index, DARE, Econlit, EMBASE, ERIC, HTA, Maternity and Infant care Database [MIDIRS], MEDLINE Journal articles, NHS EED, Psycarticles, PsycInfo, Social Policy and Practice database [SOPP], Social Science Citation Index expanded, and Social Sciences Citation Index.All retrieved articles were downloaded into an Endnote database and duplicates removed. Three reviewers  independently performed a title and abstract screen of the remaining articles before performing a full-text screen applying the inclusion and exclusion criteria outlined above. Prior to data extraction, inter-rater reliability checks were performed on a 20% random selection of all identified and included articles, and a 20% random selection of all excluded articles by two of the three reviewers. There were no recorded disagreements between reviewers.Three reviewers  independently extracted data from the remaining articles using a google form to enable consistency. Data that were extracted were study authors, study design , parenting program name and type , country of study, sample size and characteristics , the reported measures, and their defined constructs according to our initial domain-mapping exercise, i.e., attachment, bonding, maternal sensitivity, parent–child interaction.The data were then synthesized by two reviewers (SB and NG). This process sought to identify each individual measure and the number of times it occurred as an outcome in the included RCTs. The measures were then grouped within the domains, i.e., parent, child, dyadic by their format . As the objective of Search 1 was to identify the most commonly reported measures used in RCT evaluation, it was important that measures included in Search 2 were widely used in the evaluation of parenting program research. To avoid bias that may occur by applying strict criteria, the optimal threshold of appearances was explored. Across all three domains  inclusion in at least three or more independent trials proved to be the optimum cut-off, and subsequently this threshold was applied to identify the most relevant measures of interest.Inclusion criteria were papers which (1) described the development or evaluation of a questionnaire or developmental test identified in Search 1; (2) reported on a sample of expectant parents, mothers and/or fathers, and other types of primary carer of children up to and including the age of five (where the study population spanned a wider age range at least 80% of the participants had to meet this criterion); (3) was published in the English language; (4) was published as a full-text article; and (5) related to the most recent/short version of the measure. The exclusion criterion was that the population were a clinical subpopulation unrelated to the outcome .Databases were the same as for Search 1, with the exception of Centre for Reviews and Dissemination  and the Cochrane Library, which were not searched. Research indicates that it can be difficult to identify articles reporting the development or evaluation of measures due to inconsistencies in the indexing and keywords used by different databases .Search 2 data were extracted and entered onto pre-determined data extraction forms using Qualtrics software. A systematic approach was taken to capture both the quality and evaluation of findings reported in eligible articles according to the structure of two sources; (1) the COSMIN  pooled the methodological quality ratings  from the COSMIN with the ratings applied for their reported psychometric evidence , indefinite (?), or negative (-) ratings) using the Terwee checklist. To ensure that no measure was unfairly disadvantaged during the data synthesis stage, the following rules were applied to account for differences in the number of studies providing supporting evidence for each of the 10-psychometric domains:Strong Level of Evidence (+++ or −−−): This rating was applied when the evidence for the target psychometric property of a measure was supported by consistently positive or negative findings in multiple studies (two or more) rated good in methodological quality, or in one study of excellent methodology quality.Moderate Level of Evidence (++ or −−): This rating was applied when the evidence for the target psychometric property of a measure was supported by consistently positive or negative findings in multiple studies (two or more) rated fair in methodological quality, or in one study of good methodological quality.Limited Level of Evidence (+ or −): This rating was applied when the evidence for the target psychometric property of a measure was supported by positive or negative findings from one study rated fair in methodological quality.Conflicting Level of Evidence (+/): This rating was applied when the evidence for the target psychometric property of a measure was supported by studies with conflicting findings.Unknown (?): This rating was applied when the evidence for the target psychometric property of a measure was supported only by studies of poor methodological quality, or the criteria were not met for a positive or negative rating in the majority of reviewed studies.N = 24 to 5563), target caregiver , ethnicity and country of study, indicating that this pool provided an adequate representation of the available literature. A total of 480 measures were reported across the 279 studies including questionnaires (N = 268), developmental tests (N = 55), observational tools (N = 106), and other formats (N = 51) such as clinical interview schedules. Assessment of the varying frequencies of use/occurrence of measures across independent RCTs  was conducted to determine the optimal criteria that best represented the term ‘commonly used.’ Application of these thresholds across all three domains  indicated that ≥1 and ≥2, yielded too many measures for the review to be manageable and meaningful, whilst the difference between the ≥3 and ≥4 criteria was minimal. Subsequently, three or more appearances were deemed appropriate for all domains and these criteria were applied leaving 17 parent report questionnaires and seven developmental tests eligible for progression to Search 2.Search 1 yielded 16,761 articles, with 279 articles progressing to the data extraction stage . The 279 articles comprised peer-reviewed and published RCTs describing the evaluation of 113 parenting programs delivered within clinics or communities as one-to-one or group-based programs. Sample characteristics varied across individual studies in terms of size . Following a title and abstract screen, 5,669 duplicates were removed and a further 15,117 papers were found to be ineligible. Of the remaining 543 articles sent for full-text screen, 513 were excluded leaving 30 articles representing 11 questionnaires and three developmental tests for data extraction. Characteristics of the 14 measures are described in Table Six measures of parent-reported measures of child behavior were identified and reviewed, namely, the Child Behavior Checklist  over a ten-month period yielding a negative rating with limited evidence for this psychometric property. Finally, inter-rater reliability estimates were reported for the IBQ-R and the SDQ 3–16 years by comparing primary and secondary caregiver reports , and rated poor in methodological quality rendered an unknown rating for this psychometric property. Neither study that reported data for the SDQ 3–16 years met Terwee criteria, resulting in a moderate level of evidence of inter-rater reliability with negative findings for this measure.Test–re-test reliability was only reported for the ECBI in one study of fair methodological quality  yielding a limited level of evidence with negative findings for this psychometric property  for moderate levels of evidence with a positive rating  meaning that all measures yielded moderate levels of negative evidence for this psychometric property.All studies for the BIQ  social–emotional and behavior outcome measures reported in RCT evaluations of parenting programs, in order to assess the quality and strength of their psychometric standing. The objective of this was to be able to inform the development of a small battery of recommended measures to monitor change following intervention. The review finds that despite their popularity, there is a lack of consistent evidence published by independent researchers to support the use of these measures with young children. Consequently, we were unable to propose a list of measures that could be considered for recommendation. There is a need for further assessment of the psychometric properties of child outcomes in this area to ascertain their appropriateness with this age group.The synthesized evidence of the included measures indicates that none performed consistently well across multiple measurement properties. Evidence for the behavior measures suggests that the strongest support was found for the ECBI, SDQ, and CBRS across different psychometric domains; however, there are costs attached to the use of the ECBI which may limit its widespread use by practitioners. Conversely, the SDQ can be downloaded and used freely, whilst items from the CBRS can be obtained via published articles. The BITSEA and the PAS-R, a measure of child anxiety, appeared to have the most robust psychometric evidence for those measures representing the social–emotional domain. Usefully, the PAS-R is available in the public domain at no cost, complete with scoring instructions, whilst the BITSEA can be obtained from the publishers at a cost. The only parent-reported measure of child cognitive outcomes was the MCDI, a specific measure of child language development. Whilst lacking evidence to support all psychometric domains, those properties assessed indicated positive results. Moreover, its availability with a one-off fee makes it more feasible to researchers and practitioners for use as a language-screening measure. In terms of practitioner-administered measures of cognitive development, the findings indicate little evidence to support their psychometric standing. Consequently, further research is needed to draw definitive conclusions about the performance of each instrument against key measurement properties. This is particularly important given that the costs associated with these measures are the highest of all those included in this review.The general lack of evidence across all psychometric domains for the included measures supports previous reviews in this area


The World Health Organization (WHO) is a leading source of trustworthy guidelines in public health, including in emergencies. In addition to standard guidelines produced in preparation for emergency response, WHO has processes and methods for issuing guidelines in the context of urgent public health need, including rapid advice guidelines (production time 2 to 3 months) and health emergency interim guidelines (days to weeks). There are numerous challenges to producing guidelines in response to an emergency in addition to the compressed timeline which necessitates truncating or modifying standard processes. There is frequently a lack of scientific data on the disease or situation at hand, especially early in the event timeline. Resources are limited, particularly the availability of WHO staff and external experts, and disease and emergency response experts may lack knowledge and experience in developing guidelines. Finally, the rapid production of new information and the resultant short shelf‐life of recommendations pose a significant challenge to keeping guidelines up to date. In order to better meet end‐users’ needs, WHO must anticipate areas of uncertainty in emergency response and proactively develop relevant guidelines, explore optimal ways of communicating gaps in knowledge in the field to guideline developers, and promote and participate in research on the sources of bias in guideline development within compressed timeframes. WHO guidelines are of several types . Although categorizations and labels are useful, these information products represent a spectrum from standard to “ultrafast” guidelines, as the development of every document is tailored to Member States’ and end‐user needs, and to the evidence that is available.WHO handbook for guideline development ,Rapid advice guidelines can be completed within 2 to 3 months using abbreviated and accelerated methods. These types of guidelines have well‐development processes, procedures and standards as described in the HEIGs are information products focused on knowledge gaps identified during an emergency response for which no relevant, up‐to‐date guidelines exist and the guidance is needed within a few days to several weeks. The GRC Secretariat developed steps and a toolkit for developing such guidelines, including linkages to WHO's Incident Management System to ensure appropriate prioritization of guidelines being produced; tips for focusing on the key areas of uncertainty in the field; interlinked templates for planning, development, approval and production such that information and text are only entered once, with all steps leading directly to the final information product; input from technical experts, field staff and end‐users, all of whose declarations of interests have been collected and any conflicts appropriately managed; targeted peer review; and GRC review  with 24‐hour turn‐around.Despite the wide variations in development time, all WHO guidelines adhere to the principles of trustworthy information products: use of explicit and transparent processes and methods; the application of as rigorous an approach as possible to minimize bias in evidence collection, appraisal and synthesis, and in the resultant recommendations; a focus on end‐users’ needs; and the production of usable, impactful guidance.2The production of high‐quality guidelines in response to public health emergencies poses a number of challenges for WHO technical units. First, the short time periods available for rapid advice guidelines and most particularly for HEIGs impose difficult decisions regarding truncating, omitting, or accelerating steps in the standard guideline development process. Each decision can potentially jeopardize the trustworthiness and accuracy of the resultant guideline. The approaches used to minimize the risks include standardized techniques to minimize bias in evidence collection and synthesis, collection and management of declaration of interests, early and continuous consultation with experts to identify relevant data, peer review, and involvement of a methodologist who is not a content expert and who has no pre‐conceived ideas about the “right” recommendation. However, there is little information in the guideline methods literature on the most important sources of potential bias to guide decisions on which corners to cut.ad hoc, expert opinion‐based (but transparent) manner.Second, there may be a paucity of structured scientific data on emerging diseases or novel situations, especially early in the event timeline, hindering the formulation of evidence‐informed recommendations. Unlike many standard guidelines which are often engendered by a plethora of evidence or significant new data, guidelines in emergencies are often solely driven by uncertainty and needs in the field. This situation makes it all the more important to adhere to structured, transparent and explicit processes for evidence identification and appraisal, and for the formulation of recommendations. On the other hand, the lack of research evidence simplifies the evidence collection and appraisal phase, and the expert group can move quickly to decision‐making, taking issues such as feasibility, acceptability, equity, and resource considerations into account in a more Third, resource limitations, particularly the availability of WHO staff and external experts is a critical issue. As we found in the Ebola outbreak of 2014‐2016, content experts, programme managers and field workers as well as WHO staff were focused on operational issues, emergency response management, and care delivery. Spending time contributing to a guideline was understandably of lower priority for these individuals.Fourth, emergency response managers and staff may not be versed in standard approaches to evidence‐based decision‐making and guideline development. As experts in their field, they may not understand or appreciate the value of additional processes and procedures required to develop evidence‐informed, trustworthy and impactful guidelines.Fifth, many of the information products required and issued in emergency response do not lend themselves well to standard guideline processes, even to processes adapted to compressed timelines. Each document may contain hundreds of very specific statements on clinical care  which are best considered good practice statements or operational guidance, which may not benefit from reviews of the evidence or elaborate development processes.Finally, guidelines developed rapidly in response to emergencies need to be reassessed regularly for continued validity and for the addition of any new information. Thus, these guidelines are labeled as “interim” at WHO and a “review‐by” date is required in the guideline publication. This short shelf‐life poses challenges as resources may continue to be severely constrained if the outbreak or other public health emergency continues or if there are competing concurrent emergencies. Data may not have been collected in a rigorous manner and trials may not have been implemented, thus expert opinion and programmatic experiences may be the only basis for an update. In addition, needs change over the course of emergencies: for example, in the Ebola outbreak in West Africa in 2014‐2016, the health needs of survivors emerged during the course of the outbreak, necessitating new guidance, which had to be prioritized along with updates of existing guidance.3As WHO staff, we have learned much from recent emergencies about how to prioritize, manage, develop, and issue guidelines in this challenging context, and have made tremendous progress in this regard. Nonetheless, both WHO and the international guideline community have significant work to do to improve guidelines produced in response to emergencies.First, most acute health needs requiring technical guidance in emergency response can be anticipated and guidelines planned and developed prior to an event as part of risk governance, resilience, and preparedness priorities. In this ideal scenario, technical units use established processes and methods for standard guidelines to produce trustworthy, high‐quality documents. WHO is working to focus its guideline efforts more on this preparation phase, although this is challenging given our resource constraints.uncertainty. This critical step must be carefully considered, with input from the affected Member States, and from managers and field workers who are on the ground dealing with the event. Care must be taken to craft the questions such that they not only reflect the uncertainty and need, but are answerable .Second, one of the most challenging steps in guideline development, and arguably the most important, is getting the questions right. Whether for standard guidelines, rapid advice guidelines or HEIGs, guidelines processes and methods are designed to address a key area of Third, further research is needed on the sources of bias in guideline development within compressed timeframes, in order to work toward the optimal balance between rigor (and development time) and production of a valid, impactful guideline. Finally, WHO staff responsible for guidelines in emergencies need to be trained in the principles and methods for evidence‐informed decision‐making and need to recognize the value of these approaches even in the most challenging of response settings. Equally importantly, guideline methodologists need to respect the constraints faced by real‐world managers and frontline responders, and develop and implement processes and methods that add value, while not obstructing the urgent need for guidance in pursuit of the perfect guideline according to standards set for nonemergency situations.WHO, as a trusted source of technical normative guidance in healthcare, produces a large number of hugely impactful information products in the context of public health emergencies. We have made tremendous progress towards optimizing efficient production, transparency, validity, and impact on health outcomes. However, challenges remain which the Organization and the international guideline development community need to continue to work diligently to address.


However, physical (aerobic and/or resistance) exercise-induced release of heat shock protein, immune function alteration or reduced tissue hypoxia, have been highlighted as possible mechanisms by which increasing physical activity may reduce plasma pro-inflammatory cytokines in uninfected individuals and should be appraised in the literature for evidence of similar benefits in people living with HIV (PLWH). Therefore, we evaluated the effects of physical exercises on 1) inflammatory biomarkers and 2) cardiopulmonary function  for a period of at least 4 weeks among adults living with HIV, were included. Two independent reviewers determined the eligibility of the studies. Data were extracted and risk of bias (ROB) was assessed with the Cochrane Collaboration ROB tool. Meta-analyses were conducted with random effect models using the Review Manager (RevMan) computer software.n = 1073 participants at study completion) comprising male and female with age range 18–65 years. Three meta-analyses across three sub-groups comparisons were performed. The result showed no significant change in biomarkers of inflammation (IL-6 and IL-1β) unlike a significant  improvement in VO2 Max. Overall, the GRADE evidence for this review was of moderate quality.Twenty-three studies met inclusion criteria . However, this should not be interpreted as “No evidence of effect” because the individual trial studies did not attain sufficient power to detect treatment effects. The moderate grade evidence for this review suggests that further research may likely have an important impact on our confidence in the estimate of effects and may change the estimate.The online version of this article (10.1186/s12879-019-3960-0) contains supplementary material, which is available to authorized users. For decades, the Human Immunodeficiency Virus (HIV), which induces cell death (CD4- helper cells) , has rem1) . The findings from this study are very important because available evidence [There is evidence that cardiopulmonary function is significantly improved in patients with HIV infection by engaging in either aerobic exercises Search strategy in PubMed for inflammatory biomarkers. The Additional file 2:MESH terms used to search the Pubmed database for evidence of the effects of physical exercises on cardiopulmonary function in HIV conditions. (DOCX 14 kb)Search strategy in PubMed for cardiopulmonary function. The Additional file 3:MESH terms used to search the Cochrane library database for evidence of the effects of physical exercises on inflammatory biomarkers in HIV conditions. (DOCX 13 kb)Search strategy in Cochrane library for inflammatory biomarkers. The Additional file 4:MESH terms used to search the Cochrane library database for evidence of the effects of physical exercises on cardiopulmonary function in HIV conditions. (DOCX 13 kb)Search strategy in Cochrane library for cardiopulmonary function. The Additional file 5:Quality of evidence and definition. A definition of the different categories of quality of evidence applied in grading the studies included in the research. (DOCX 19 kb)


Despite the emergence of curated app libraries for mental health apps, personal searches by consumers remain a common method for discovering apps. App store descriptions therefore represent a key channel to inform consumer choice. This study examined the claims invoked through these app store descriptions, the extent to which scientific language is used to support such claims, and the corresponding evidence in the literature. Google Play and iTunes were searched for apps related to depression, self-harm, substance use, anxiety, and schizophrenia. The descriptions of the top-ranking, consumer-focused apps were coded to identify claims of acceptability and effectiveness, and forms of supporting statement. For apps which invoked ostensibly scientific principles, a literature search was conducted to assess their credibility. Seventy-three apps were coded, and the majority (64%) claimed effectiveness at diagnosing a mental health condition, or improving symptoms, mood or self-management. Scientific language was most frequently used to support these effectiveness claims (44%), although this included techniques not validated by literature searches (8/24 = 33%). Two apps described low-quality, primary evidence to support the use of the app. Only one app included a citation to published literature. A minority of apps (14%) described design or development involving lived experience, and none referenced certification or accreditation processes such as app libraries. Scientific language was the most frequently invoked form of support for use of mental health apps; however, high-quality evidence is not commonly described. Improved knowledge translation strategies may improve the adoption of other strategies, such as certification or lived experience co-design. Reviews of the quality of the content within publicly available health apps4 and specifically mental health apps7 support this disparity, reporting that the majority of consumer-available apps are not evidence-based and can contain harmful content.Recent reviews have found mobile health  apps to be effective in reducing symptoms of depression8 app libraries10 and frameworks to support clinicians in recommending mental health apps,11 personal searches on commercial app stores operated by the major smartphone platform providers remain a common method for discovering mental health apps.12 In this setting, marketing materials provided by developers are a principal source of information to inform consumer or clinician choice. The format of this material is standardised for commercial app stores, consisting of a written app description and, optionally, screenshots or videos of app functions.Although there is an increasing interest in accreditation processes,13Within this restricted context, the extent to which scientific evidence is presented as a potential marker of quality for health apps is unclear. A preliminary investigation by the authors previously reported that, for apps clinically relevant for depression, 38% of app store descriptions included wording related to claims of effectiveness, whereas only 2.6% provided evidence to substantiate such claims.This study aims to extend this preliminary analysis to further understand how scientific evidence is currently used to market and sell mental health apps by (i) examining the types of claims made by mental health apps and, specifically, estimating the proportion of apps that invoke claims of effectiveness; (ii) describing the types of supporting statements used to justify claims and, specifically, estimating the proportion of apps which invoke scientific principles; and (iii) assessing the credibility of scientific principles that are used as supporting statements. Insight into methods used to present apps on commercial stores has the potential to inform government and professional efforts to establish curated libraries for health apps, as well as develop our understanding of translational gaps between mHealth research and developer practices.A total of 1435 apps were identified through searches of the app stores  described a single mental health-related functionality; fewer apps described two  or three  discrete functions. Three apps did not clearly describe any specific functionality . The types of functionality described by the apps are summarised in Table Just over four-fifths of apps  made a positive claim in their online app store description, including claims related to effectiveness  or acceptability  – see Table Forty-seven apps  also provided some form of statement supporting use of the app . The most common form of support was the use of scientific language , although eight of these apps used general terms (e.g. “evidence-based treatment”); specific scientific methods or techniques were identified for 24 apps  – full details of the annotated techniques are described later. Notably only two apps  described direct evidence associated with the app , and only one app  provided citation details to scientific literature . A post-hoc analysis identified that five apps  mentioned research or clinical trials underway.The second most common type of support was the description of technical expertise . This was predominantly through descriptions of the credibility of the app developer , and less commonly through inclusion of expert endorsements  or awards and prizes . No apps referred to formal accreditation or certification schemes (§4.b.i).Ten apps  referred to lived experience perspectives, either in their design or development process  or in the development team itself . App descriptions invoked the “wisdom of the crowd” in just under one-fifth of cases , referring to download, usage, or popularity metrics , user testimonials and reviews , or press endorsements .n = 26), of which just over half  also did not include supporting statements. However, where supporting statements were included, these were evenly distributed across the categories. The small number of apps which made claims related to diagnosis or detection of a mental health condition exclusively invoked supporting statements related to scientific language .Apps were grouped together based on the type of effectiveness claims made, and the associated supporting statements were examined – see Fig. n = 6/8, 75%) and to the exclusion of statements about lived experience involvement.Approximately half of the apps included a single type of claim related to improvements in symptoms or self-monitoring. In this set of apps, scientific language and descriptions of technical expertise were invoked equally. For the set of apps that claimed improvements in both symptoms and self-management, supporting statements were predominantly related to scientific statements  of these apps provided no supporting statements. Scientific language was frequently used in apps for treatment or therapy  or self-assessment . Apps involving peer-support or community support included the highest proportion of support involving technical expertise , lived experience perspectives  and the wisdom of the crowd .Apps were also grouped together based on the functionality of the app, and the types of supporting statements invoked were examined – see Fig. n = 7 and n = 6, respectively), for which positive evidence was found.14 The second most common combination was the use of “binaural beats” in relation to depression and anxiety , for which no evidence could be found in the scientific literature. Other combinations described more than once were generally associated with positive evidence, including dialectical behaviour therapy for self-harm (n = 3),15 the use of the Patient Health Questionnaire (PHQ-9) for the assessment of depression (n = 3),16 the Generalised Anxiety Disorder (GAD-7) questionnaire for the assessment of anxiety (n = 2),17 and a harm reduction approach in substance use (n = 2).18 Active listening was also mentioned in reference to a range of conditions, and although no specific evidence could be identified in the literature searches, it is acknowledged that this is considered a key clinical skill.19 From the remaining combinations of techniques and conditions which were described once, the majority were also associated with positive evidence (n = 20), and the remainder with unclear evidence (n = 9) or no found evidence (n = 8).From the descriptions of the 24 apps which mentioned a specific scientific technique, 11 unique conditions (§1) and 38 unique methods (§3.a.i) were identified, resulting in 49 unique literature searches being conducted – the results of which are presented in Supplementary Information Overall, from the 49 combinations of conditions and methods, 26 (53%) were associated with positive evidence, 13 (27%) were associated with unclear evidence, and evidence could not be found for 10 (20%). Aggregating at an app-level, a third of the apps described at least one technique for which evidence could not be found .Seventy-three mental health apps, representing the most highly ranked apps from the two major app stores, were examined in this study. Sixty-four percent of these apps made positive claims about their effectiveness, and 45% claimed acceptability. Statements supporting the use of the apps were presented through scientific descriptions (44%), technical expertise (32%), appeals to the “wisdom of the crowd” (19%), or lived experience involvement (14%). Of the scientific methods described, just over a half (53%) were associated with evidence in academic literature; of the apps describing specific scientific techniques, a third referred to techniques for which no evidence could be found (33%).From a research perspective, it is perhaps reassuring that scientific language was the leading form of support employed by developers; however, this was present in fewer than half of the apps. Importantly, only two apps (2.7%) provided direct evidence associated with app use – results from a pilot study, and user-reported changes in mood after app use. One app description (1.3%) cited a validation paper for a self-report questionnaire. While these cases represent the best evidence provided by apps in this study, they still fall short of high-quality evidence obtained, for example, from randomised controlled trials.20 Furthermore, a third of apps whose descriptions included scientific techniques referred to principles that had no evidence available in the scientific literature. Together with those apps which cited principles with conflicting evidence, and those which used general scientific language without reference to specific methods, this suggests that developers are using scientific language to appeal to consumers, regardless of the accuracy of the claims. Sector engagement with app developers and consumers may help improve the reporting and understanding of the science associated with mental health apps.Although there may be a lack of published evidence directly supporting the use of the mental health apps examined in this review, when apps described scientific techniques more broadly, in just over half the cases these techniques were associated with good evidence from the literature. This raises the hope that apps are evidence-informed, if not necessarily evidence-based. Caution, however, is still required as apps claiming to deliver, for example, cognitive behavioural therapy for depression may have minimal concordance with the actual principles of CBT.8 Given the variable quality of evidence identified in this study, this suggests there may be an opportunity for researchers to work with developers to identify how high-quality evidence and real-world performance data could best be captured.These results are also important in the context of new efforts to regulate health apps. The United States Food and Drug Administration (FDA) is exploring a Software Precertification (Pre-Cert) Pilot Program that will shift regulation towards the app manufacturers themselves and rely on “monitoring real-world performance” of apps in the wild.22 and conversely a lack of involvement is often associated with poor uptake and engagement of digital interventions.23 These factors highlight the potential for increased lived experience involvement in the development of mental health apps.Of the categories of supporting statements identified in this study, the least frequently described was the involvement of those with lived experience (14%). It is acknowledged that consumer involvement and co-design of interventions can be a key factor for their success,11 Although the APA app evaluation framework does not offer direct recommendations or marks of approval, it focuses on informed decision making and helps clinicians and individuals consider the risks and benefits of app use on a case-by-case basis. Such an approach supports selection of an app based upon the individual needs of a user, with clinician consideration of the scientific claims and evidence associated with the app. Future reviews may be warranted to examine whether references to accreditation schemes increase with the adoption of schemes such as the APA framework and the implementation of FDA pre-cert.It was also noted that despite increasing interest in app accreditation frameworks and curated libraries, no apps described these in their app store descriptions. For the apps in this study, which already have good visibility through high search result rankings, this may reflect a lack of perceived need for such processes. It may alternatively reflect a lack of awareness of these schemes outside academic or clinical communities, or that accreditation could be used as a marker of credibility in a commercial marketplace. Regardless of the underlying reason, further knowledge translation activities appear to be warranted to increase the profile of such accreditation schemes. One such scheme attempting to identify quality apps for clinical and individual use is the American Psychiatric Association (APA) app evaluation scheme.13 as well as other indicators such as compliance with quality frameworks and lived experience involvement.In the future, app stores could include standardised data fields allowing developers to provide additional details to support their apps. There has been progress towards mandating that apps include a privacy policy, and this could be extended for health apps by allowing developers to include a PubMed identifier, offering users the opportunity to click through to published articles related to the app,13 Nevertheless, these results provide a broad indication of the nature and credibility of claims associated with mental health apps. The study did not examine either the content of ancillary marketing material presented alongside app descriptions, such as screenshots or user comments. These elements were excluded partly for reasons of standardisation  and partly because it was considered unlikely that either imagery or user comments would reference scientific principles, which was a key purpose of this study. Previous research indicates that while users provide a range of positive or negative ratings, there is only minimal mention of scientific quality or evidence.24It is acknowledged that this study provides only a snapshot of a subset of mental health apps, and that the app stores represent a rapidly evolving ecosystem for distribution of health apps.At the outset of this study, we initially aimed to differentiate between claims related to improvements in mood and improvements in symptoms, as a means of differentiating, for example, feelings of depression vs symptoms of clinical depression. However, it became apparent that such distinctions were not clearly articulated within the app store descriptions, so these coding categories were combined.25This study included appeals to the “wisdom of the crowd” and lived experience involvement as markers of credibility which can be used to support claims made in app store descriptions. It should be noted, however, that user ratings do not necessarily correlate well with clinical utility or quality.26 – however, such a rigorous approach was not possible here due to the number of literature searches required (n = 49). Nevertheless, the three-point scale incorporated existing systematic reviews, where available, to differentiate techniques for a particular mental health condition for which there is clear evidence in the literature, mixed or unclear evidence, or no evidence found. Further inspection of in-app content, by multiple stakeholders including those with lived experience and clinical expertise, would be required to obtain a complete understanding of the quality of an app. This would also include an assessment of whether the scientific principles cited are actually used within the app, and to what degree of fidelity.Scientific methods are reported in this study using a three-point evidence scale. More rigorous evidence evaluation schemes exist, for example, through formal systematic review, meta-analysis and the OCEBM Levels of Evidence27 and the APA framework,11 amongst others, and serve as a best-practice guide in terms of app development standards and important information to be provided to allow individuals, clinicians or app library providers to make informed decisions about app adoption.This review has examined a set of markers of quality which can be derived from app store descriptions, with a particular focus on the description of scientific techniques and evidence. However, these are not the only important markers of quality. Additional factors such as usability, data privacy and security, and integration with clinical workflows and systems are also of importance, and may not be discernible from just the app store description. These domains are included in guidelines for the relaunched NHS Apps LibraryThis study examined 73 of the top ranked mental health apps publicly available to map the nature of claims and the type of supporting statements employed in app descriptions presented in app stores. Scientific language was the most frequently employed strategy for supporting effectiveness claims. However, direct evidence from app-specific studies was lacking, and many apps described techniques for which there was not clear evidence in the literature. Lived experience involvement and engagement with formal accreditation processes were limited, suggesting further knowledge translation activities may be required to raise the awareness of these critical aspects of mental health app development.28 Chronic pain syndrome was not included due to the uncertainty associated with the allocation of DALYs between mental health and musculoskeletal conditions. Searches for these five conditions  were performed on 21 November 2017. Searches for Android apps were performed on the US Google Play store website, and for iOS apps through the iTunes search application programming interface (API) set to the US store. For each search term on each platform, the app title and description were extracted  for the top 40 search results.Apps were selected for mental health conditions based upon the greatest global burden of disease. Based upon estimates of disability-adjusted life years (DALYs) provided by Vigo et al., the five greatest burdens of disease were considered to be depression, self-harm, substance use disorders , anxiety disorders and schizophrenia.After extracting the search result data, the title and descriptions of each app were reviewed to assess eligibility, using the criteria in Table Two coders annotated each app description using the coding scheme described below. Disagreements were resolved by a third coder. The broad coding categories were defined in advance, with iterative refinements to sub-categories following pilot testing with a subset of the apps. Screenshots and other materials presented on the app stores (e.g. user comments) were not reviewed and are not included in this analysis.§1. Target condition(s): any mood state or mental health condition identified in the description text was annotated as a target condition for the app. If no conditions were explicitly mentioned, the search term (or terms) which identified the app was used.§2. App functionality: the described function of the app was coded as providing (i) self-assessment; (ii) symptom or mood monitoring; (iii) information or psychoeducation; (iv) therapy or treatment; or (v) peer-support or community support. Zero, one, or more functionalities could be coded.§3. Positive claims: two broad, non-mutually exclusive, categories of positive claims were identified from the app store descriptions:Claims of effectiveness. Specifically, text was coded as a claim if it linked the use of the app to any of the following outcomes: (i) the detection or diagnosis of a condition; (ii) improvement in symptoms or mood; or (iii) improvement in the individual’s ability to self-manage their condition .Claims of acceptability, such as statements focusing on the usability or acceptability of the app, rather than the app’s impact on health and wellbeing.§4. Supporting statements: to identify the types of statements used to support the use of the app or the claims made, the following categories were identified:Support invoking scientific language, specifically: (i) mentions or use of a specific scientific technique, method, or principle; (ii) evidence from a study evaluating use of the app; or (iii) citations to scientific literature. Specific scientific techniques were coded and the perceived credibility or evidence associated with these methods was later evaluated .Support based on technical expertise, specifically: (i) any formal quality assessment framework, or certification or accreditation programmes related to the developer or app; (ii) prizes or awards for the developer or the app; (iii) the credibility of the app developer or other professionals associated with the app; or (iv) endorsements from credible or trustworthy professionals or organisations.Support based on design informed by lived experience, specifically: (i) involvement of individuals with lived experience in the design or development of the app (including focus group feedback); or (ii) developers with lived experience.Support based on “the wisdom of the crowd”, specifically: (i) download, usage, or popularity statistics; (ii) testimonials from users; or (iii) endorsements from the press or media.§5. Negative claims: within app store descriptions two types of disclaimers were identified (a) medical disclaimers, such as not being a replacement for medical care, and (b) legal disclaimers.§6. Evidence base: coded as either: (a) positive evidence from at least one systematic review or randomised controlled trial, with consensus amongst the reviewers; (b) unclear evidence, where some evidence was found but there was also contradictory evidence identified, concerns about the quality of the evidence, or there was not a clear consensus; or (c) no evidence found, where evidence from a systematic review or randomised controlled trial could not be found. For details of the method used to identify evidence, see the Evidence search section, below.After initial coding, each combination of identified target condition (§1) and scientific technique (§4.a.i) were enumerated. A literature search was conducted to try to establish the state of the evidence, if any, supporting the application of each technique to each identified condition. Given the large number of combinations of techniques and conditions, it was not feasible to conduct a full systematic review or meta-analysis for each. Therefore, for pragmatic reasons, searches were conducted using the MEDLINE, Embase, and PsycINFO databases for articles including the combination of technique and condition, limited to either systematic reviews or randomised controlled trials. Two researchers independently performed each search, and reviewed the titles and abstracts, and full-texts where necessary, to determine whether there was evidence found from at least one systematic review or randomised control trial to support the application of a method for a specific condition. Coding disagreements were resolved by a third reviewer. As this may result in a permissive stance, where a positive single randomised controlled trial could be coded as evidence supporting a technique, the resolved coding decisions were then reviewed by an additional two expert coders to identify any relevant literature supporting, or contradicting, the coding. Evidence was summarised using the three-point coding scale described previously in the coding schema.Descriptive statistics were used to summarise the results of coding. Sub-group analyses were performed to examine the types of supporting statements invoked for different categories of effectiveness claims, and for different app functionalities.Supplementary Information 1.Coded app data


Evidence-based practice is now widely recognized as the key to improving healthcare quality and patient outcomes. Although the purposes of nursing research (conducting research to generate new knowledge) and evidence-based nursing practice (utilizing best evidence as basis of nursing practice) seem quite different, an increasing number of research studies have been conducted with the goal of translating evidence effectively into practice. Clearly, evidence from research (effective innovation) must be accompanied by effective implementation, and an enabling context to achieve significant outcomes.The Journal of Nursing Research must provide an open forum for all kind of research in order to help bridge the gap between research-generated evidence and clinical nursing practice and education.As mentioned by Professor Rita Pickler, “nursing science needs to encompass all manner of research, from discovery to translation, from bench to bedside, from mechanistic to holistic” . I feel In this issue, an article by professor Ying-Ju Chang and colleagues at National Cheng Kung University presents an evidence-based practice curriculum for undergraduate nursing students developed using an action research-based model. This “evidence-based practice curriculum” spans all four academic years, integrates coursework and practicums, and sets different learning objectives for students at different grade levels. Also in this issue, Yang et al. apply a revised standard care procedure to increase the ability of critical care nurses to verify the placement of nasogastric tubes. After appraising the evidence, the authors conclude that the aspirate pH test is the most reliable and economical method for verifying nasogastric tube placement at the bedside. They subsequently develop a revised standard care procedure and a checklist for auditing the procedure, conduct education for nurses, and examine the effectiveness of the revised procedure.I hope that these two studies help us all better appreciate that, in addition to innovation and new breakthrough discoveries, curriculum development and evidence-based quality improvement projects, though may not seem so novel, are also important areas of nursing research. Translating evidence into practice is sound science and merits more research.


Clinical predictive tools quantify contributions of relevant patient characteristics to derive likelihood of diseases or predict clinical outcomes. When selecting predictive tools for implementation at clinical practice or for recommendation in clinical guidelines, clinicians are challenged with an overwhelming and ever-growing number of tools, most of which have never been implemented or assessed for comparative effectiveness. To overcome this challenge, we have developed a conceptual framework to Grade and Assess Predictive tools (GRASP) that can provide clinicians with a standardised, evidence-based system to support their search for and selection of efficient tools.A focused review of the literature was conducted to extract criteria along which tools should be evaluated. An initial framework was designed and applied to assess and grade five tools: LACE Index, Centor Score, Well’s Criteria, Modified Early Warning Score, and Ottawa knee rule. After peer review, by six expert clinicians and healthcare researchers, the framework and the grading of the tools were updated.GRASP framework grades predictive tools based on published evidence across three dimensions: 1) Phase of evaluation; 2) Level of evidence; and 3) Direction of evidence. The final grade of a tool is based on the highest phase of evaluation, supported by the highest level of positive evidence, or mixed evidence that supports a positive conclusion. Ottawa knee rule had the highest grade since it has demonstrated positive post-implementation impact on healthcare. LACE Index had the lowest grade, having demonstrated only pre-implementation positive predictive performance.GRASP framework builds on widely accepted concepts to provide standardised assessment and evidence-based grading of predictive tools. Unlike other methods, GRASP is based on the critical appraisal of published evidence reporting the tools’ predictive performance before implementation, potential effect and usability during implementation, and their post-implementation impact. Implementing the GRASP framework as an online platform can enable clinicians and guideline developers to access standardised and structured reported evidence of existing predictive tools. However, keeping GRASP reports up-to-date would require updating tools’ assessments and grades when new evidence becomes available, which can only be done efficiently by employing semi-automated methods for searching and processing the incoming information. Modern healthcare is building upon information technology to improve the effectiveness, efficiency, and safety of healthcare and clinical processes –7. In paClinical predictive tools (here referred to simply as predictive tools) belong to the third level of CDS and include various applications ranging from the simplest manually applied clinical prediction rules to the most sophisticated machine learning algorithms , 11. ThrTraditionally, the selection of predictive tools for implementation in clinical practice has been conducted based on subjective evaluation of and exposure to particular tools , 15. ThiMore recently, there has been an increase in the mention and recommendation of selected predictive algorithms in clinical guidelines. However, this represents only a small proportion of the amount and variety of proposed clinical predictive tools, which have been designed for various clinical contexts, target many different patient populations and comprise a wide range of clinical inputs and techniques –18. UnliAlthough most reported tools have been internally validated, only some have been externally validated and very few have been implemented and studied for their post-implementation impact on healthcare , 24. VarIn addition, decision makers need to consider the usability of a tool, which depends on the specific healthcare and IT settings in which it is embedded, and on users’ priorities and perspectives . The usaSeveral methods have been proposed to evaluate predictive tools , 46–60. On the other hand, looking beyond predictive tools, the GRADE framework, grades the quality of published scientific evidence and strength of clinical recommendations, in terms of their post-implementation impact. GRADE has gained a growing consensus, as an objective and consistent method to support the development and evaluation of clinical guidelines, and has increasingly been adopted worldwide , 65. AccThe aim of this study is to develop a conceptual framework for evidence-based grading and assessment of predictive tools. This framework is based on the critical appraisal of information provided in the published evidence reporting the evaluation of predictive tools. The framework should provide clinicians with standardised objective information on predictive tools to support their search for and selection of effective tools for their intended tasks. It should support clinicians’ informed decision making, whether they are implementing predictive tools at their clinical practices or recommending such tools in clinical practice guidelines to be used by other clinicians.Guided by the work of Friedman and Wyatt, and their suggested three phases approach, which became an internationally acknowledged standard for evaluating health informatics technologies , 41, 69,We started with a focused review of the literature in order to examine and collect the evaluation criteria, and measures proposed for the appraisal of predictive tools along these three phases. The concepts used in the search included “clinical prediction”, “tools”, “rules”, “models”, “algorithms”, “evaluation”, and “methods”. The search was conducted using four databases; MEDLINE, EMBASE, CINAHL and Google Scholar, with no specific timeframe. This literature review was then extended to include studies describing methods evaluating CDS systems and more generally, health information systems and technology. Following the general concepts of the PRISMA guidelines , the duphttps://www.mdcalc.com) [Using the extracted information, we designed an initial version of the framework and applied it to asses and grade five predictive tools. We reviewed the complete list of 426 tools published by the MDCalc medical reference website for decision support tools and applications and calculators  . We exclalc.com) , from a A comprehensive and systematic search for the published evidence, on each of the five predictive tools, was conducted, using MEDLINE, EMBASE, CINAHL and Google Scholar, and refined in four steps. 1) The primary studies, describing the development of the tools, were first identified and retrieved. 2) All secondary studies that cited the primary studies or that referred to the tools’ names or to any of their authors, anywhere in the text, were retrieved. 3) All tertiary studies that cited the secondary studies or that were used as references by the secondary studies were retrieved. 4) Secondary and tertiary studies were examined to exclude non-relevant studies or those not reporting the validation, implementation or evaluation of the tools. After the four steps, eligible evidence was examined and grades were assigned to the predictive tools. Basic information about the tool, such as year of publication, intended use, target population, target outcome, and source and type of input data were extracted, from the primary studies, to inform the first “Tool Information” section of the framework. Eligible studies were then examined in detail for the reported evaluations of the predictive tools. Additional file The framework and its application to the selected five predictive tools were then peer reviewed by six expert healthcare professionals. Three of these professionals are clinicians, who work in hospitals and have over 10 years of experience using CDS systems, while the other three are healthcare researchers, who work in research organisations and have over 20 years of experience in developing, implementing or evaluating CDS systems. The reviewers were provided with the framework’s concept design and its detailed report template. They were also provided with the summarised and detailed grading of the five predictive tools, as well as the justification and published evidence underpinning the grade assignment. After a brief orientation session, reviewers were asked to feedback on how much they agreed with each of the framework’s dimensions and corresponding evaluation criteria, the ‘Tool information’ section as well as the grading of the five exemplar predictive tools. The framework was then refined and the grading of the five predictive tools was updated based on the reviewers’ feedback. Figure The search in the four databases, after removing the duplicates, identified a total of 831 studies. After screening the titles and abstracts, 647 studies were found not relevant to the topic. The full text of the remaining 184 studies were then examined to exclude non-eligible studies, which were 134 studies, based on the inclusion criteria. Only 50 studies were identified as eligible. Twenty three of the 50 studies described methods for the evaluation of predictive tools , 63, 73,During the development phase, the internal validation of the predictive performance of a tool is the first step to make sure that the tool is doing what it is intended to do , 55. PreBefore wide implementation, it is important to learn about the estimated potential effect of a predictive tool, when used in the clinical practice, on three main categories of measures: 1) Clinical effectiveness, such as improving patient outcomes, estimated through clinical effectiveness studies, 2) healthcare efficiency, including saving costs and resources, estimated through feasibility and cost-effectiveness studies, and 3) patient safety, including minimising complications, side effects, and medical errors. These categories are defined by the Institute of Medicine as objectives for improving healthcare performance and outcomes, and are differently prioritised by clinicians, healthcare professionals and health administrators , 100. ThIn addition, it is important to learn about the usability of predictive tools. Usability is defined as the extent to which a system can be used by the specified users to achieve specified and quantifiable objectives in a specified context of use , 109. ThSome predictive tools have been implemented and used in the clinical practice for years, such as the PECARN head injury rule or the Ottawa knee and ankle rules –122. In Phase of Evaluation,Level of EvidenceandDirection of Evidence.Our suggested GRASP framework (abbreviated from Grading and Assessment of Predictive Tools) is illustrated in Table Assigns a letter A, B and/or C based on the highest phase of evaluation reported in the published literature. A tool is assigned the lowest phase, C, if its predictive performance has been tested and reported for validity; phase B if its usability and/or potential impact have been validated; and the highest phase, A, if it has been implemented in clinical practice and its post-implementation impact has been reported.Assigns a numerical score within each phase of evaluation based on the level of evidence associated with the evaluation process. Tools in phase C of evaluation can be assigned three levels. A tool is assigned the lowest level, C3, if it has been tested for internal validity; C2 if it has been tested for external validity once; and C1 if it has been tested for external validity multiple times. Similarly, tools in phase A of evaluation can be assigned the lowest level of evidence, A3, if their post-implementation impact has been evaluated only through subjective or descriptive studies; A2 if it has been evaluated via observational studies; and A1 if post-implementation impact has been measured using experimental evidence. Tools in phase B of evaluation are assigned grade B2 if they have been tested for potential impact and B1 if they have been tested for usability. Effect sizes for each outcome of interest together with study type, clinical settings and patient populations are also reported.Due to the large heterogeneity in study design, outcome measures and patient subpopulations contained in the studies, synthesising measures of predictive performance, usability, potential effect or post-implementation impact into one quantitative value is not possible. Furthermore, acceptable values of predictive performance or post-implementation impact measures depend on the clinical context and the task at hand. For example, tools like Ottawa knee rule  and WellPositive evidence is assigned when all studies reported positive conclusions while negative evidence is assigned when all studies reported negative or equivocal conclusions. In the presence of mixed evidence, studies are farther ranked according to their quality as well as their degree of matching with the orginal tool specifications . Mixed evidence is then classified considering this ranking as supporting an overall positive conclusion or supporting an overall negative conclusion. Details and illustration of this protocol can be found in Additional file The final grade of a predictive tool is based on the highest phase of evaluation, supported by the highest level of positive evidence, or mixed evidence that supports an overall positive conclusion. Figure In order to show how GRASP works, we applied it to grade five randomly selected predictive tools; LACE Index for Readmission , Centor LACE Index is a prognostic tool designed to predict 30 days readmission or death of patients after discharge from hospitals. It uses multivariable logistic regression analysis of four administrative data elements; length of stay, admission acuity, comorbidity (Charlson Comorbidity Index) and emergency department (ED) visits in the last 6 months, to produce a risk score [sk score . The toosk score , 130.Two studies examined the predictive performance of LACE Index on small sub-population samples; 507 geriatric patients in the United Kingdom and 253 congestive heart failure patients in the United States, and found that the index performed poorly , 132. TwCentor Score is a diagnostic tool that uses a rule-based algorithm on clinical data to estimate the probability that pharyngitis is streptococcal in adults who present to the ED complaining of sore throat [e throat . The scoe throat –142. Thie throat . This quEvidence of the post-implementation impact of Centor score is mixed. One RCT conducted in Canada reported a clinically important 22% reduction in overall antibiotic prescribing . Four otWells’ Criteria is a diagnostic tool used in the ED to estimate pre-test probability of pulmonary embolism [embolism , 124. Usembolism . The tooembolism –153 and embolism –156. In embolism . This quembolism . TherefoThe Modified Early Warning Score (MEWS) is a prognostic tool for early detection of inpatients’ clinical deterioration and potential need for higher levels of care. The tool uses a rule-based algorithm on clinical data to calculate a risk score [sk score . The MEWsk score –165. Allsk score . Using tThe MEWS has been implemented in different healthcare settings. One observational before-and-after intervention study failed to prove positive post-implementation impact of the MEWS on patient safety in acute medical admissions . HoweverOttawa Knee Rule is a diagnostic tool used to exclude the need for an X-ray for possible bone fracture in patients presenting to the ED, using a simple five items manual check list [eck list . It is oeck list . Furthereck list . Anothereck list . AccordiIn the Additional file On peer-review, experts found the GRASP framework logical, helpful and easy to use. The reviewers strongly agreed to all criteria used for evaluation. The reviewers suggested adding more specific information about each tool, such as the author’s name, the intended user of the tool and the recommended action based on the tool’s findings. The reviewers showed a clear demand for knowledge regarding the applicability of tools to their local context. Two aspects were identified and discussed with the reviewers. Firstly, the operational aspect of how easy it would be to implement a particular tool and if the data required to use the tool is readily available in their clinical setting. Secondly, the validation aspect of adopting a tool developed using local predictors, such as life expectancy (which is location specific) or information based on billing codes . Following this discussion, elements related to data sources and context were added to the information section of the framework. One reviewer suggested assigning grade C0 to the reported predictive tools that did not meet C3 criteria, i.e. those tools which were tested for internally validity but were either insufficiently internally validated or the internal validation was insufficiently reported in the study, in order to differentiate them from those tools for which neither predictive performance nor post-implementation impact have been reported in the literature.It is challenging for clinicians to critically evaluate the growing number of predictive tools proposed to them by colleagues, administrators and commercial entities. Although most of these tools have been assessed for predictive performance, only a few have been implemented or evaluated for comparative predictive performance or post-implementation impact. In this manuscript, we present GRASP, a framework that provides clinicians with an objective, evidence-based, standardised method to use in their search for, and selection of tools. GRASP builds on widely accepted concepts, such as Friedman and Wyatt’s evaluation approach , 41, 69 The GRASP framework is composed of two parts; 1) the GRASP framework concept, which shows the grades assigned to predictive tools based on the three dimensions: Phase of Evaluation, Level of Evidence and Direction of Evidence, and 2) the GRASP framework detailed report, which shows the detailed quantitative information on each predictive tool and justifies how the grade was assigned. The GRASP framework is designed for two levels of users: 1) Expert users, who will use the framework to assign grades to predictive tools and report their details, through the critical appraisal of published evidence about these tools. This step is essential to provide decision making clinicians with grades and detailed reports on predictive tools. Expert users include healthcare researchers who specialise in evidence-based methods and have experience in developing, implementing or evaluating predictive tools. 2) End users, who will use the GRASP framework detailed report of tools and their final grades, produced by expert users, to compare existing predictive tools and select the most suitable tool(s) for their predictive tasks, target objectives, and desired outcomes. End users include clinicians and other healthcare professionals involved in the decision making and selection of predictive tools for implementation at their clinical practice or for recommendation in clinical practice guidelines to be used by other clinicians and healthcare professionals. The two described processes; the grading of predictive tools by expert users and the selection decisions made by end users, should occur before the recommended tools are made available for use in the hands of the practicing clinicians.Previous approaches to the appraisal of predictive tools from the published literature, namely the TRIPOD statement , 62 and Walker and Habboushe, at the MDCalc website; classified and reported the most commonly used medical calculators and other clinical decision support applications. However, the website does not provide users with a structured grading system or an evidence-based method for the assessment of the presented tools. Therefore, we believe that our proposed framework can be adopted and used by MDCalc, and similar clinical decision support resources, to grade their tools.One of the main challenges in assessing and comparing clinical predictive tools is dealing with the large variability in the quality and type of studies in the published literature. This heterogeneity makes it impractical to quantitatively synthesise measures of predictive performance, usability, potential effect or post-implementation impact into single numbers. Furthermore, as discussed earlier, a particular value of a predictive performance metric that is considered good for some tasks and clinical settings may be considered insufficient for others. In order to avoid complex decisions regarding the quality and strength of reported measures, we chose to assign a direction of evidence, based on positive or negative conclusions as reported in the studies under consideration, since synthesizing qualitative conclusions is the only available option which adds some value. We then provide the end user with the option to look at a summary of the reported measures, in the GRASP detailed report, for further details.It is not uncommon to encounter conflicting conclusions when a tool has been validated in different patient subpopulations. For example, LACE Index for readmission showed positive external validity when tested in adult medical inpatients , 130, buAs is the case with other healthcare interventions, examining the post-implementation impact of predictive tools is challenging, since it is confounded by co-occurrent socio-technical factors –176. ThiLikewise, well designed post-implementation impact evaluation studies are required in order to explore the influence of organisational factors and local differences on the success or failure of predictive tools , 181. DaThere is a challenging trade-off between the generalisability and the customisation of a given predictive tool. Some algorithms are developed using local data. For example, Bouvy’s prognostic model, for mortality risk in patients with heart failure, uses life quality and expectancy scores from the Netherlands . SimilarGRASP evaluates predictive tools based on the critical appraisal of the existing published evidence. Therefore, it is subject to publication bias, since statistically positive results are more likely to be published than negative or null results , 185. ThThe grade assigned to a tool provides relevant evidence-based information to guide the selection of predictive tools for clinical decision support, but it is not prescriptive. An A1 tool is not always better than an A2 tool. A user may prefer an A2 tool showing improved patient safety in two observational studies rather than an A1 tool showing reduced cost in one experimental study. The grade is a code  that provides information on three relevant dimensions: phase of evaluation, level of evidence, and direction of evidence as reported in the literature.One of the limitations of the GRASP framework is that the Direction of Evidence dimension is based on the conclusions of the considered studies on each predictive tool, which confers some subjectivity to this dimension. However, the end-user clinicians are provided with the full details of all available studies on each tool, through the GRASP detailed report, where they can access the required objective information to support their decisions. In addition, we applied the GRASP framework to only five predictive tools and consulted a small number of healthcare experts for their feedback. This could have limited the conclusions about the framework’s coverage and/or validity. Although GRASP framework is not a predictive tool, it could be thought of as a technology of Grade C3, since it has only been internally validated after development. However, conducting a large-scale validation study of the framework, extending the application of the framework to a larger number of predictive tools, and studying its effect on end-users’ decisions is out of the scope of this study and is left for future work.To validate, update, and evaluate the GRASP framework, the authors are currently working on three more studies. The first study should validate the design and content of the framework, through seeking the feedback of a wider international group of healthcare experts, who have published work on developing, implementing or evaluating predictive tools. This study should help to update the criteria used, by the framework, to grade predictive tools and improve the details provided, by the framework, to the end users. The second study should evaluate the impact of using the framework on improving the decisions made by clinicians, regarding evaluating and selecting predictive tools. The experiment should compare the performance and outcomes of clinicians’ decisions with and without using the framework. Finally, the third study aims to apply the framework to a larger consistent group of predictive tools, used for the same clinical task. This study should show how the framework provides clinicians with an evidence-based method to compare, evaluate and select predictive tools, through reporting and grading tools based on the critical appraisal of published evidence.The GRASP framework builds on widely accepted concepts to provide standardised assessment and evidence-based grading of predictive tools. Unlike other methods, GRASP is based on the critical appraisal of published evidence reporting the tools’ predictive performance before implementation, potential effect and usability during implementation, and their post-implementation impact. Implementing the GRASP framework as an online platform can enable clinicians and guideline developers to access standardised and structured reported evidence of existing predictive tools. However, keeping the GRASP framework reports up-to-date would require updating tools’ assessments and grades when new evidence becomes available. Therefore, a sustainable GRASP system would require employing automated or semi-automated methods for searching and processing the incoming published evidence. Finally, we recommend that GRASP framework be applied to predictive tools by working groups of professional organisations, in order to provide consistent results and increase reliability and credibility for end users. These professional organisations should also be responsible for making their associates aware of the availability of such evidence-based information on predictive tools, in a similar way of announcing and disseminating clinical practice guidelines.Additional file 1: Table S1. Phases, Criteria, and Measures of Evaluating Predictive Tools. Table S2. Evaluating Evidence Direction Based on the Conclusions of Studies. Table S3. LACE Index for Readmission – Grade C1. Table S4. Centor Score for Streptococcal Pharyngitis – Grade B1. Table S5. Wells’ Criteria for Pulmonary Embolism – Grade A2. Table S6. Modified Early Warning Score (MEWS) – Grade A2. Table S7. Ottawa Knee Rule – Grade A1. Table S8. Predictive Performance of the Five Tools – Before Implementation. Table S9. Usability of Two Predictive Tools – During Implementation. Table S10. Post-Implementation Impact of Three Predictive Tools. Figure S1. Study Selection for the Focused Review of Literature. Figure S2. Searching the Literature for Published Evidence on Predictive Tools. Figure S3. The Mixed Evidence Protocol. Figure S4. Reported C-Statistic of LACE Index, Centor Score, Wells Criteria and MEWS [and MEWS –192.


Despite smaller effect sizes, interventions delivered at population level to prevent non-communicable diseases generally have greater reach, impact and equity than those delivered to high-risk groups. Nevertheless, how to shift population behaviour patterns in this way remains one of the greatest uncertainties for research and policy. Evidence about behaviour change interventions that are easier to evaluate tends to overshadow that for population-wide and system-wide approaches that generate and sustain healthier behaviours. Population health interventions are often implemented as natural experiments, which makes their evaluation more complex and unpredictable than a typical randomised controlled trial (RCT). We discuss the growing importance of evaluating natural experiments and their distinctive contribution to the evidence for public health policy. We contrast the established evidence-based practice pathway, in which RCTs generate ‘definitive’ evidence for particular interventions, with a practice-based evidence pathway in which evaluation can help adjust the compass bearing of existing policy. We propose that intervention studies should focus on reducing critical uncertainties, that non-randomised study designs should be embraced rather than tolerated and that a more nuanced approach to appraising the utility of diverse types of evidence is required. The complex evidence needed to guide public health action is not necessarily the same as that which is needed to provide an unbiased effect size estimate. The practice-based evidence pathway is neither inferior nor merely the best available when all else fails. It is often the only way to generate meaningful evidence to address critical questions about investing in population health interventions. Governments around the world are committed to tackling the growing burden of non-communicable diseases.The aetiological associations between behaviours and many chronic disease outcomes are sufficiently well established to justify efforts to ameliorate behavioural risk factors. However, how population behaviour patterns might most effectively be shifted remains one of the greatest uncertainties for public health research and policy. To date, effort has largely been directed at developing and evaluating interventions to change the behaviours of individuals at higher risk, sometimes successfully. However, it is doubtful that merely scaling up these approaches to reach more and more people would be affordable, effective or equitable as a global disease prevention strategy.10.1136/jech-2019-213085.supp1Supplementary dataIn this paper, we discuss the growing importance of evaluating natural experiments in primordial prevention and their distinctive contribution to generating evidence for public health policy. We identify some of the obstacles to this type of research and suggest greater effort and investment in this area to ensure that research more effectively supports public health action.goal on the one hand, that is, the outcome we ultimately wish to achieve; and on the other, ‘behaviour change’ as an intervention strategy or moral imperative, that is, a way of framing both the problem  and the solution (they need to make different choices). An approach too narrowly focused on people and their behaviour or lifestyle as the problem, and thereby on interventions that often seek ‘to persuade the poor to change their behaviour’,We should reflect on the extent to which our research is aligned with the societal processes it is intended to inform.systems that generate and sustain less healthy behaviour patterns.The evidence available to guide policy has long been subject to an ‘evaluative bias’ in favour of behavioural interventions targeting people at higher risk because such interventions are generally easier to evaluate and, in particular, easier to randomise.The lack of evidence for effective primordial prevention strategies may be traced to one of three types of obstacle.political obstacles encountered by researchers who are willing but unable to produce the evidence. Researchers seeking to evaluate environmental or policy interventions—such as improving access to green space or taxing particular foods—depend on governments or other agencies to implement evaluable strategies.The first is a set of cultural obstacles in research, manifested by a research community that is able in principle to produce the evidence, but rendered somewhat unwilling by circumstances. Nearly two decades ago, it was pointed out that only a small fraction of UK public health research expenditure was directed towards ‘solutions’.The second is a set of practical obstacles. Primordial preventive strategies are generally implemented as ‘natural’ or ‘quasi-’ experiments rather than ‘true’ experiments  for more widespread implementation, subject to broader contextual considerations such as affordability and political acceptability. This is shown in the upper pathway of Much has been written about the bench-to-bedside translational medicine pathway linking basic science with clinical practice. That concept has also strongly influenced thinking about evidence to support public health action, for example, in discourse that refers to institutionalising ‘proven’, ‘evidence-based’ interventions.However, this implied linear, rational way in which new knowledge is converted into ‘evidence-based practice’ has limited empirical support or practical utility for ‘upstream’ public health interventions.effectiveness to justify action, the latter depends more on multiple instances of action from which to develop at least preliminary evidence of effectiveness. This may in turn support taking further evaluable action and the consequent, cumulative reduction of uncertainty about its effects.Public health research is concerned with ‘generating discoveries and new knowledge within the public health field itself’.most appropriate methods  to systematically rule out alternative explanations for observed effects and demonstrate credible causal pathways leading to those effects.The application of each step of the practice-based evidence pathway, and how it differs from the converse pathway, can be illustrated with a worked example based on the published protocol for the ongoing evaluation of the UK soft drinks industry levy,How, then, might effort and investment in developing the published evidence base more effectively support the kinds of policy intervention required for primordial prevention? Our analysis suggests three main implications; examples of potential actions arising from these are given in behaviour patterns—just as we judge smoking cessation services not on their direct impact on heart disease or lung cancer, but on whether they help people quit smoking.Studies of interventions to change upstream determinants of disease risk, such as population dietary or commuting patterns, are sometimes criticised because they have not followed participants to ultimate, ‘hard’ physiological or clinical endpoints. However, there is no reason to expect that all parts of a putative causal chain should be directly proved within a single study. It has been argued that to inform action, public health intervention research should be guided as much by a decision-theoretical approach  as by thas if’ it were random, although not within a planned RCT. Although randomisation eliminates important sources of potential confounding, an expectation that intervention studies should entail a comparable allocation process (such as a lottery) would further entrench existing evaluative biases because many interventions relevant to public health are never likely to fulfil this criterion.relevance of the intervention to current, real-world policy questions. A key advantage of natural experimental studies is that they ‘do not interfere in the natural data generation process’,plausibility of the causal inference.how an intervention achieves its effects.Although natural experimental study designs have important theoretical underpinnings in common with the RCT, their worth does not reside solely in the extent to which they emulate an RCT design. Dunning proposes three criteria for assessing the utility of natural experimental studies.because) it falls into the implicitly disparaging category of ‘non-randomised’ studies.Of course, this may appear to sit uneasily within a research funding system based on a biomedical paradigm that privileges the RCT above all other methods for establishing effectiveness.This is not to deny that many non-randomised studies do have major limitations and are reported in ways that lack rigour or transparency. For example, systematic reviews of studies linking changes in the built environment with changes in diet, physical activity and adiposity have noted multiple potential sources of bias and that ‘studies with weaker designs were more likely to report associations in the positive direction’.We have well-established, and continually developing, catechisms for assessing the internal validity of intervention studies, and groups of studies, in health research.We are more likely to halt the rise in the global prevalence of non-communicable diseases by taking and evaluating new, more ambitious or radical actions to address the underlying causes than by merely applying existing preventive approaches—even if these are effective—with greater intensity. Even apparently simple questions about effectiveness in this arena cannot be answered without action—although based on the best available evidence at the time—that necessarily precedes evaluation. The practice-based evidence pathway can be regarded as an essential, and currently under-resourced and undervalued, complement to the more established evidence-based practice pathway. It is neither inferior nor merely the best available when all else fails. On the contrary, it is often the only way to generate meaningful evidence to address critical questions about investing in population health interventions.The two pathways for generating evidence described in this paper do not represent mutually exclusive approaches. Some policy and practice innovations could and should be evaluated in RCTs, and many more would benefit from more planned evaluation using a wider range of study designs. Nevertheless, the public health research community and those who fund and publish their work have key roles to play in supporting the development and credibility of researchers in this field, and the more thoughtful conduct, appraisal and synthesis of natural experimental studies, to populate critical missing pieces of the evidence base to support more effective public health action.There are well-established associations between behaviour and chronic disease, which justify government efforts to reduce behavioural risk factors. However, the question of how population behaviour patterns might most effectively be shifted remains one of the greatest uncertainties for research and policy. This reflects the substantial challenges of evaluating population preventive strategies, interpreting the findings and translating them into action. Greater effort and investment in this area may help ensure that research more effectively supports public health action.We discuss the growing importance of evaluating natural experiments and their distinctive contribution to the evidence for public health policy. We contrast the established evidence-based practice pathway, in which randomised controlled trials generate ‘definitive’ evidence for particular interventions, with a practice-based evidence pathway in which evaluation can help adjust the compass bearing of existing policy. We propose that intervention studies should focus on reducing critical uncertainties, that non-randomised study designs should be embraced rather than tolerated and that a more nuanced approach to appraising the utility of diverse types of evidence is required.