The American College of Medical Genetics and Genomics (ACMG)/Association for Molecular Pathology (AMP) clinical variant interpretation guidelines established criteria for different types of evidence. This includes the strong evidence codes PS3 and BS3 for “well-established” functional assays demonstrating a variant has abnormal or normal gene/protein function, respectively. However, they did not provide detailed guidance on how functional evidence should be evaluated, and differences in the application of the PS3/BS3 codes are a contributor to variant interpretation discordance between laboratories. This recommendation seeks to provide a more structured approach to the assessment of functional assays for variant interpretation and guidance on the use of various levels of strength based on assay validation.The Clinical Genome Resource (ClinGen) Sequence Variant Interpretation (SVI) Working Group used curated functional evidence from ClinGen Variant Curation Expert Panel-developed rule specifications and expert opinions to refine the PS3/BS3 criteria over multiple in-person and virtual meetings. We estimated the odds of pathogenicity for assays using various numbers of variant controls to determine the minimum controls required to reach moderate level evidence. Feedback from the ClinGen Steering Committee and outside experts were incorporated into the recommendations at multiple stages of development.The SVI Working Group developed recommendations for evaluators regarding the assessment of the clinical validity of functional data and a four-step provisional framework to determine the appropriate strength of evidence that can be applied in clinical variant interpretation. These steps are as follows: (1) define the disease mechanism, (2) evaluate the applicability of general classes of assays used in the field, (3) evaluate the validity of specific instances of assays, and (4) apply evidence to individual variant interpretation. We found that a minimum of 11 total pathogenic and benign variant controls are required to reach moderate-level evidence in the absence of rigorous statistical analysis.The recommendations and approach to functional evidence evaluation described here should help clarify the clinical variant interpretation process for functional assays. Further, we hope that these recommendations will help develop productive partnerships with basic scientists who have developed functional assays that are useful for interrogating the function of a variety of genes. The American College of Medical Genetics (ACMG) and the Association for Molecular Pathology (AMP) jointly developed standards and guidelines for the assessment of evidence to increase consistency and transparency in clinical variant interpretation . One typIn response to calls to further standardize variant interpretation , 4, the In this manuscript, we detail additional guidance developed by the SVI regarding the assessment of the clinical validity of functional studies and a provisional framework for the determination of suitable evidence strength levels, with the goal that experimental data cited as evidence in clinical variant interpretation meets a baseline quality level. We expect to further refine these approaches in collaboration with VCEPs as they apply these recommendations moving forward.In November 2018, during the monthly SVI Working Group conference call, we first outlined our goals of defining what constitutes a well-established functional assay and how functional assay evidence should be structured for computation and curation. In this meeting, we presented a preliminary approach to curating functional evidence and important considerations for assay validation. This process was subsequently presented at the ClinGen Steering Committee in-person meeting in Seattle, WA, in December 2018 for comments and further refinement. The proposed PS3/BS3 evaluation process was then discussed on the SVI Working Group call in March 2019 and again in-person at the American College of Medical Genetics and Genomics (ACMG) meeting in April 2019. Subsequently, a smaller subgroup developed a final version of these recommendations, incorporating the feedback from ClinGen biocurators and VCEPs, which were then approved by the SVI Working Group.We used curated functional evidence from VCEP-developed rule specifications  and expe1) and the proportion of pathogenic variants in the groups with functionally abnormal or functionally normal readouts as posterior probabilities (P2). The stringency of the thresholds determining an abnormal versus normal readout is related to the confidence in the assay result. We initially estimated an optimistic OddsPath that could be achieved by a perfect binary classifier, where the readout for all control variants tested is consistent with the variant interpretation ]/[(1 − P2) × P1]  . Each Od2) × P1] .in cis and in trans, epigenetic effects, cell type, assay conditions, etc.). For conditions inherited in an autosomal recessive pattern, biallelic variants are required, often in a loss-of-function mechanism where the penetrance and expressivity of disease manifestations may depend on the thresholds of overall protein activity that reflect the cellular/biochemical phenotype arising from a combination of variants and potentially other cellular gene products. In this case, it will be important to distinguish the overall protein activity levels that cause different phenotypes (severe versus mild disease) from the functional assay results that would qualify for variant-level evidence toward a pathogenic or benign interpretation. If a variant is known to be homozygous , and can be evaluated in multiple unrelated individuals, functional assay evidence from patient-derived material can be interpreted with greater confidence.The genetic construct and context being evaluated in an assay are important considerations for determining appropriateness for clinical variant interpretation. The assay material being utilized  should be taken into account when evaluating the validity of a functional assay. When using patient-derived samples, a functional assay evaluates a broader genetic and physiologic background  and to delineate the expected disease phenotype in patients with certain combinations of variants or homozygous variants with known pathogenicity. If the curator decides to proceed with evaluating an assay performed in patient-derived material, the level of strength applied should be determined based on validation parameters (see below). In the context of a VCEP, gene-specific guidance should include the required number of unrelated individuals in whom the variant has been tested, in order for the evidence to qualify for variant interpretation.Typically, model organisms are used to implicate the role of a gene in a disease . Issues related to cost and throughput have typically limited the generation of extensive allelic series intended for the purpose of clinical variant interpretation. In addition, it can be challenging to assess how well the model organism reflects human anatomy/physiology/genetic context, or whether the full phenotype must necessarily be recapitulated in order to satisfy the functional evidence criteria. The genome of the organism may include an orthologous gene , or the model organism may lack relevant homologs that affect the phenotype in humans, thus affecting the degree to which an artificially introduced genetic variant can cause a relevant phenotype. Even within a given species, measurable phenotypes can vary depending on the genetic background of the organism , and therefore, studies using more than one strain or line would be preferable, further increasing the cost of such assays. Therefore, the recommendations herein will primarily focus on cellular and biochemical in vivo or in vitro assays, which are commonly encountered in laboratory evaluations of variants implicated in human disease.Recommendation 2: From the point of view of clinical variant interpretation, evaluation of functional evidence from model organisms should take a nuanced approach, considering the caveats described above. If model organism data are to be used in variant interpretation, the strength of evidence should be adjusted based on the rigor and reproducibility of the overall data provided.The nature of the variant and the context in which it is studied can significantly affect the assay readout. The effect of the variant on the expressed gene product must be carefully considered when determining the clinical validity of an assay that utilizes an artificially engineered variant. For example, CRISPR-introduced genetic variants in an otherwise normal genomic context will use the endogenous cellular transcriptional and splicing machinery, although off-target effects must be carefully considered. In contrast, transient expression of cDNA constructs, which usually contain artificial promoters and other regulatory sequences that can result in variant overexpression, should be carefully standardized using controls to ensure that the overexpression does not mask the true effects of variants. Nonsense and frameshift variants that result in premature termination codons before the 3′-most 50 nucleotides of the penultimate exon are expected to undergo nonsense-mediated decay (NMD) and eliminate the mRNAs , 20; theRecommendation 3: While testing variants in a more natural genomic context is preferable, it is not a requirement of a well-validated assay. Instead, one should consider how the approach impacts the interpretation of the results and take into account whether the study controls for these limitations when assigning the strength of evidence.Since an individual functional assay may not fully capture all gene or protein functions relevant to disease pathogenesis, a “normal” result in a laboratory assay may simply reflect that the functional effect of the specific variant was not suitably assayed in the experiment. Therefore, in order to determine when, and at what strength, to apply the BS3 criterion, it is essential to understand how well the assay captures the molecular consequence of the variant and its impact on the expressed protein or functional domain. A more complete assessment of protein function permits scoring the result as having a benign effect, whereas an assay that is limited to a specific domain or functional readout may provide less strong evidence for having a benign effect. It should also be noted that a missense or synonymous variant that does not affect protein function might still have a negative impact by introducing a cryptic splice site . These cMessenger RNA splicing is a complex process, and clinical variant interpretation can take into account both predictive and laboratory evidence. RNA splicing assays, developed using the endogenous genomic context or using artificial mini-gene assays, can be useful to determine the impact of variants on splicing integrity. However, unlike protein assays, the readout  does not necessarily correlate with protein function. For example, abnormal splicing of the last exon might lead to a truncated protein whose function is still intact. In general, abnormal splicing can have heterogenous outcomes with respect to mRNA fate and the protein-reading frame. Abnormally spliced transcripts might undergo NMD, while other abnormal transcripts can lead to a shortened or truncated protein with or without functional consequences . The relBecause RNA splicing assays do not provide a direct measure of protein function, additional recommendations are needed to determine the applicability of splicing assays to satisfy PS3/BS3 versus PVS1 (loss-of-function). For canonical ± 1,2 splice variants, PVS1 application is based on the predicted impact of a variant on mRNA stability and protein-reading frame whereas a functional assay may conclusively demonstrate abnormal splicing and confirm a loss-of-function impact. Additional data and considerations are needed to determine the appropriate aggregate strength of PVS1 and PS3 in the scenario that functional data is present and supports PVS1 application. Similarly, splicing assays could be used to bolster support for in silico predictions for variants outside the canonical ± 1,2 splice sites. An SVI subgroup is currently working on recommendations for the incorporation of predictive and functional evidence of altered splicing into the ACMG/AMP framework. For variants impacting protein length that are not predicted to lead to loss-of-function, such as in-frame exon skipping due to abnormal splicing or a large in-frame deletion, the change in protein length alone could be used to justify application of PM4, while application of PS3/BS3 could also be appropriate if a functional assay examined the protein function of the resulting product.Standardized, structured language can improve communication and transparency across clinical laboratories, physicians, and patients. Uniform terminology should be used to describe the readout of a laboratory assay of protein function and document the curation of functional evidence. As such, the variant-level results of functional assays should not be categorized as “pathogenic” or “benign,” since these falsely equate functional impact with a clinical determination that involves a number of other evidence lines. In addition, terms describing assay results as “deleterious” or “damaging” can be confusing since their meanings are greatly context-dependent and generally only apply when the loss-of-function is the mechanism of disease. For example, in conditions where the mechanism involves gain-of-function, a variant may be damaging or deleterious to the organism but not to protein activity as measured in a functional assay. Establishing standardized language to describe assay readout is an important step to prevent the misinterpretation of published data and to reduce inter-laboratory discordance with respect to PS3/BS3 application .Recommendation 4: The terms “functionally normal” or “functionally abnormal” should be used to describe the functional impact of a variant as measured in a given assay. Further granular specifications should be used to describe the “functionally abnormal” impact  as outlined by Spurdle et al. . The finThe 2015 ACMG/AMP guidelines assert that “functional studies that have been validated and shown to be reproducible and robust in a clinical diagnostic laboratory setting are considered to be the most well established”  may not Recommendation 5: The entity performing a functional assay should not govern whether PS3/BS3 criteria are satisfied or at what strength. This determination should be based primarily on the validation of the assay, including the use of appropriate laboratory controls as well as clinical validation controls (as described below).Good laboratory practice is essential for the application of functional evidence in clinical variant interpretation. Every experiment should include internal controls that demonstrate the dynamic range of the assay . In some cases, the readout may be normalized to a wild-type value, which should generally be run in the same conditions as the variants being tested to avoid a batch effect. Well-conducted experiments typically use technical replicates that control for the random differences associated with a protocol or instrument-associated variation, to demonstrate the reproducibility of the result within a given experiment. Similarly, biological replicates  are included to control for random biological variation in parallel measures of unique biological samples and to demonstrate the reproducibility of the result between instances of the same experiment. Biological replicates are more important for understanding the variance within a population, while technical replicates can reduce measurement error .Furthermore, well-validated assays are benchmarked by including known pathogenic and known benign variants that establish the ranges of assay readout for these classes of variants and define the thresholds beyond which the result can be considered functionally abnormal, indeterminate, or functionally normal. It is important to note that the clinical interpretation of these validation control variants should reach a pathogenic/likely pathogenic or benign/likely benign interpretation using lines of evidence independent of functional data, so as to avoid circularity in defining the assay’s predictive value. The number of controls required depends on the dynamic range of the assay and the variance of each replicate; controls should also be relevant to the disease mechanism (such as gain-of-function or loss-of-function) and the type of variant under consideration . For genes associated with multiple disorders through different mechanisms, an assay validated for one disorder may not necessarily be applied universally to analyze the variant effect in other disorders if the mechanisms of the disease are different. Variants in the Genome Aggregation Database (gnomAD)  that havMany previously published assays do not identify known benign or known pathogenic variant controls or may have only tested a few variant controls in the same assay. To address this, it may be possible for analysts to assemble these controls from multiple specific instances of the same general class of assay. Any tested variant that could be classified as likely benign/benign (LB/B) or likely pathogenic/pathogenic (LP/P) without functional criteria would qualify as a control for the determination of evidence strength. The assay readout for each of these variants, as tested across multiple instances of the same general class of assay, can be plotted together in order to set thresholds for normal, intermediate, and abnormal function Fig. .Fig. 1A1. Define the disease mechanismIn order for functional assays to be useful in clinical variant interpretation, the underlying gene-disease mechanism must be reasonably well understood. The VCEP or individual interpreting variants in a given gene should first delineate this mechanism to determine what functional assays can be considered applicable. This is an important first step since some genes are associated with different diseases depending on the mechanism . A structured narrative using ontologies or other specific terms can be used to describe the gene-disease mechanism Table .Table 22. Evaluate the applicability of general classes of assays used in the fieldNext, the general types or classes of assays used in the field should be defined and documented, including the model system, experimental method, and functional outcome being evaluated. The defined gene-disease mechanism should guide an evaluation of how well a general class of assay models pathogenesis . Relative strengths and weaknesses of the model system should be assessed, and disease-specific assertions regarding the appropriateness of animal, cellular, and in vitro models should be addressed  and positive  controlsAssays that do not include technical and/or biological replicatesFunctional evidence should not be applied in the following scenarios unless the dynamic range of the assay and the thresholds for defining a functionally normal, indeterminate, or functionally abnormal result are extremely well understood:Assays that include experimental controls and replicates but have 10 or fewer validation controls to assess the ability of the assay readout to distinguish pathogenic from benign variants  or benign interpretation (BS3_supporting) may be applied in the following scenarios:Assays with at least 11 total validation controls including a mix of benign and pathogenic variants, but no formal statistical analysis of the ability to discriminate between pathogenic and benign variants  or benign interpretation (BS3_moderate) may be applied in the following scenarios:Any level of evidence in favor of pathogenicity may be applied when rigorous statistical analysis enables a formal OddsPath to be calculated, with the strength of evidence correlating to the calculated OddsPath Table .Evidence in favor of a benign interpretation up to a strong level BS3) may be applied when rigorous statistical analysis enables a formal OddsPath to be calculated, with the strength of evidence correlating to the calculated OddsPath  and the specific instances of assays that do not qualify (and why). Documentation should include PMID or other universal reference to the source of the assay evaluated , the type of assay readout  and units, the range of assay results that qualify for a given strength of evidence according to level of validation as above, and the range in which the assay result is indetBS3/BS3.4. Apply evidence to individual variant interpretationIf the assay demonstrates a functionally abnormal result consistent with the mechanism of disease, the PS3 criterion can be applied at a level of strength based on the degree of validation detailed above.If the assay demonstrates a functionally normal result, the BS3 criterion can be applied at a level of strength based on the degree of validation detailed above.Variants demonstrating an intermediate level of impact on function merit special consideration, as this could be because the assay does not fully reflect the protein function (decreasing strength applied to the assertion), or may provide evidence supporting a hypomorphic or partial loss-of-function effect, such as in a condition with incomplete penetrance and/or less severe expressivity. Consideration of disease mechanism should help guide the appropriate level of strength to be applied for these types of variants.Once the specific instance of an assay has been evaluated as a whole, the results from that assay for a given variant can be applied as evidence in variant interpretation.When PS3 and BS3 are applied by any variant analyst, the documentation of the supporting evidence should reference the strength of the clinical validation of the functional assay.The SVI Working Group recommends that evaluators use a four-step process to determine the applicability and strength of evidence of functional assays for use in clinical variant interpretation: 1) define the disease mechanism, (2) evaluate the applicability of general classes of assays used in the field, (3) evaluate the validity of specific instances of assays, and (4) apply evidence to individual variant interpretation. Unlike the ACMG/AMP guidelines [ define tIf the results are consistent , apply PS3/BS3 at the level of strength appropriate for the most well-validated assay.If the results are conflicting, the assay that most closely reflects the disease mechanism and is more well-validated can override the conflicting result of the other, and evidence should be applied at the strength indicated by the assay’s validation parameters. If the assays are essentially at the same level of validation, conflicting functional evidence should not be used in the interpretation of the variant.For a variant analyzed by multiple assays (belonging to the same or different class):The committee did not reach consensus on whether results from different classes of functional assay could be combined . The primary concern with this approach is that it is extremely difficult to ascertain that two assays are measuring independent functions and that this would lead to double-counting the same evidence regarding variant function. Another concern is that stacking evidence from multiple assays could lead to a conflated interpretation of the disease risk for a particular variant . On the other hand, if the assays are measuring different functions, the evidence may be complementary and increase confidence in the overall result, especially for the assertion of BS3 criteria. Variant curators and expert groups will need to decide how to best proceed, keeping in mind the cautions regarding the double-counting of evidence.When multiple functional assay results are available for a single variant , the evaluator should apply evidence from the assay that is the most well-validated and best measures the disease mechanism.This provisional framework for the evaluation and application of functional evidence in clinical variant interpretation represents the first important steps toward reducing discordance in the use of PS3/BS3 criteria. Moving forward, this approach will be tested with a range of diverse disorders in collaboration with ClinGen VCEPs. We recognize that many historical publications may not meet the specifications outlined here, which will limit our ability to apply these assays as strong evidence in the ACMG/AMP variant interpretation framework, though they may still qualify for supporting-level evidence if performed rigorously and with appropriate laboratory controls. The applicability of these recommendations for rare diseases with small numbers of known pathogenic variants will also need to be tested. As always, clinical laboratories will need to make a judgment call about the rigor, reproducibility, and clinical validation of any piece of available functional evidence and determine its strength, hopefully adhering to the spirit of these recommendations even if the specific circumstances do not permit the analytic process suggested here. Undoubtedly, many other kinds of evidence will be re-weighted as the ACMG/AMP guidelines are revised and this provisional framework will evolve alongside these updates.As the field moves to develop assays with sufficient controls and validation to permit the calculation of an OddsPath, more quantitative approaches for stacking evidence and assigning evidence strength may be adopted, as outlined in the Bayesian adaptation of the ACMG/AMP variant interpretation framework . This quWhile typical functional assays cited as evidence in variant curations analyze relatively few variants , new mulBRCA2 VUS would be reclassified as likely pathogenic with functional evidence, leading to increased early surveillance and recommendations regarding cancer prophylaxis and management).As it is time-consuming and expensive to develop and sufficiently validate novel functional assays, effort and resources should be directed to have the greatest clinical benefit. One could prioritize assays that would examine genes with the greatest number of genetic tests performed or individuals tested annually or focus on the genes with the greatest proportion of VUS that could be adjudicated with functional evidence . AlternaWe hope that these recommendations will help develop productive partnerships with basic scientists who have developed functional assays that are useful for interrogating the function of a variety of different genes . RealistAdditional file 1: Odds of Pathogenicity (OddsPath) by performance of classified variant controls. Contains two tables modeling the Odds of Pathogenicity and strength of evidence based on the number of known benign and known pathogenic variants tested and the ability of the assay to correctly classify those variants, either as a perfect binary  readout (Table S1) or permitting one variant control (either benign or pathogenic) to have an indeterminate readout (Table S2).


The UK National Screening Committee (UK NSC) reviews evidence about existing or potential population screening programmes using rapid review products called evidence summaries. We provide a case report as an example of how rapid reviews are developed within the UK NSC’s process, consider how the quality of rapid reviews should be assessed and ask whether the rapid review was an appropriate tool to inform the UK NSC’s decision-making process.We present the rapid review approach taken by the commissioner and the reviewers to develop an evidence summary for vasa praevia (VP), which the UK NSC reappraised as part of its 3-yearly cycle for conditions where screening is currently not recommended. We apply the AMSTAR 2 quality appraisal checklist for systematic reviews, the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) checklist and a published checklist of items to consider with a rapid review approach. As UK NSC evidence summaries do not include meta-analyses, any related AMSTAR 2 or PRISMA checklist items were considered inapplicable.The evidence summary was available within the required timelines and highlighted little change from the previous review in terms of key evidence gaps relating to the epidemiology of VP, the screening test and the management pathway. Therefore, the UK NSC concluded that there was insufficient evidence to support a change in its previous recommendation against screening. The evidence summary scored moderately against the applicable AMSTAR 2 and PRISMA checklist items. Against the published checklist of items to consider with a rapid review approach, the evidence summary performed well.In this case report, the use of a rapid review as part of the UK NSC’s process enabled a pragmatic approach to assessing the overall volume, quality and direction of literature on key questions relating to the viability of a population screening programme for VP. Based on our assessments of this single evidence summary, systematic review quality appraisal tools may undervalue rapid reviews. The validity of the methods used in this case report, as well as the wider generalisability of our insights relating to rapid review practice, reporting and quality assessment, requires analysis of a larger sample of rapid reviews. Screening is the process of identifying healthy people who may be at increased risk of a disease or condition. In the UK, the National Screening Committee (NSC) is responsible for reviewing evidence about existing or potential population screening programmes. Evidence reviews are used to advise government ministers and the National Health Service (NHS) on the implementation, continuation and cessation of screening programmes.Systematic literature reviews (SLRs) are regarded as the gold standard for evidence-informed policy-making because they provide robust, comprehensive and trustworthy appraisals of the evidence on a topic . HoweverFor conditions where the current recommendation is not to offer screening, the UK NSC reappraises the evidence base every 3 years using rapid review products called evidence summaries .1 EvidenThe UK NSC assesses all potential screening programmes against a formal list of 20 criteria for appraising their viability, effectiveness and appropriateness. These criteria are structured in line with the principles proposed by Wilson and Jungner  and consThe UK NSC conducted an appraisal of screening for vasa praevia (VP) as part of the 3-yearly review cycle. VP is a rare but serious obstetric condition where exposed umbilical vessels lie across the cervical opening during pregnancy. If VP is undiagnosed, the blood vessels can rupture during the natural labour process. This can lead to fetal exsanguination, which can be lethal. Ultrasound screening for VP during the second trimester has been proposed , 10, usiHere, we present the approach taken to develop a UK NSC evidence summary for VP, providing a case report of how rapid reviews are developed within the UK NSC’s evidence review process. We also report the person-time required to produce the evidence summary, discuss the use of systematic review quality appraisal and reporting checklists when applied to this example of a rapid review and consider whether the evidence summary was an appropriate tool to be used by the UK NSC in their decision-making process.The approach taken to developing the evidence summary is detailed below, along with details of the published checklists that we applied to the review product.The review was commissioned by the UK NSC  and conducted by an external review team who specialise in evidence synthesis . A summary of the respective roles of the commissioning and review teams on this project is provided in Table A scoping exercise (including scoping literature searches) and discussion with experts on the previous review within the UK NSC’s reference group structures led to the identification of 9 questions relating to VP and VCI. These were presented to the review team in “Population, Intervention, Comparator and Outcomes” (PICO) format in a brief that also provided background on the topic, details of previous UK NSC reviews and practical points such as the proposed timescale for the evidence summary.Once the project was underway, regular meetings between the two teams enabled the commissioning team to contribute to discussions on the review protocol and drafts of the report. With regard to study selection, the commissioning team provided the review team with clarifications on the eligibility criteria, but the review team made the final decisions on the eligibility of studies for inclusion in the review. The commissioning team also contributed to discussions with the review team when making judgements as to whether the evidence identified met the UK NSC evidence criteria. Topic area expertise was available as required, and this was most frequently accessed during the protocol development stage and when the review document was being processed within the UK NSC reference group structures.The rapid review aimed to identify developments in the evidence base relating to VP since the previous UK NSC review on VP was conducted in 2012, establish whether the current recommendation against screening should be reconsidered and determine whether further research was required.The 2012 review and subsequent public consultation found that detection of VCI would be an important component of a screening strategy for VP. Most cases of VP are associated with VCI; however, only a minority of pregnancies affected by VCI are also affected by VP, and detection of VCI as part of a VP screening programme would represent a departure from the current approach in UK practice with the potential for over-detection. A set of separate questions relating to VCI were therefore introduced in the current review to explore these issues in more depth.The previous review on screening for VP was conducted in 2012, before the UK NSC’s evidence summary process was standardised, and the results were not structured in the same way. To enable consistent synthesis of the older and newer evidence and to ensure that relevant evidence on VCI was also identified, the current review included studies published from 2000 onwards. Studies identified in the previous review were therefore re-included and re-analysed in the current review.The specific, focused questions considered in the review are detailed in Table Scoping searches were carried out during the development of the review brief. At the protocol development stage, several decisions were made to streamline the searching process. The overall volume of published literature on VP and VCI is very limited, with fewer than 1000 records identified across key databases  even when terms for VP and VCI were used without any filters for dates, study design or outcomes. The reviewers judged that it would be more efficient to search using terms for VP and VCI only rather than developing more specific search strategies for each review question. Taking this approach allowed the database searches and abstract review to be completed within 2 days, in contrast to potentially several weeks if more extensive protocol development had been required.Hand searching of some excluded SLRs was undertaken, but it was planned that no further supplementary searches would be conducted, such as manual searches of key congress proceedings. Authors were not contacted for further information on studies.For each question, eligibility criteria were developed to identify studies that could provide the most robust and relevant information. It was prespecified that studies would be prioritised for inclusion in the evidence synthesis based on study design. For example, if a relevant SLR or meta-analysis was identified on a particular question, it would not be considered necessary to include lower quality evidence such as that obtained from retrospective studies. If there were no relevant SLRs or meta-analyses, but an abundance of primary studies on a particular topic, it was planned that the primary studies would be prioritised by study design. For epidemiology studies, longitudinal observational studies would be prioritised over cross-sectional studies. For studies on the performance of screening methods and the effectiveness of management pathways, randomised controlled trials (RCTs) would be given highest priority, followed by interventional non-RCTs, prospective cohort studies, retrospective cohort studies, case control studies and cross-sectional studies. During the rapid review, however, it was not necessary to prioritise any study designs over others; on one topic, an SLR was identified along with all of the primary studies included within it, while on other topics, all primary studies identified were included.Sifting of each record was performed by a single reviewer; a second independent reviewer provided input in cases of uncertainty and validated a random 20% of the first reviewer’s screening decisions. A single reviewer extracted relevant data from included studies into prespecified extraction tables, which were included as appendices in the evidence summary. A second reviewer then independently verified the extracted information and checked that no relevant information had been missed.Quality assessment was performed for each study using prespecified checklists suitable for each study design. In SLRs, checklists are sometimes adapted to suit the purposes of the review, but for this rapid review, published checklists were used without adaptation to minimise the time required for protocol development. The published checklists included the following: for epidemiology and prognostic studies, the JBI Critical Appraisal Checklist for Studies Reporting Prevalence Data and the Centre for Evidence Based Medicine Prognostic Studies Critical Appraisal Worksheet, and for diagnostic accuracy studies, the Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) tool.The rapid review was written up into a template document developed in line with the UK NSC’s evidence summary reporting checklist . The cheUK NSC evidence summaries do not extend to the conduct of a quantitative meta-analysis. Instead, for each question, relevant studies identified in the review were grouped by outcome reported and summarised narratively. Where epidemiology study results were consistent, summary ranges were presented, while the results of diagnostic test accuracy studies  were tabulated. Reference was made to existing SLRs, and the results of a review addressing the epidemiology of VP were summarised. However, an SLR of ultrasound testing for VP was not used in this way because it did not present the results of the included studies in sufficient detail to inform an analysis of second-trimester transabdominal sonography specifically, which was required to answer the review question. Equally, an SLR relating to VCI was hand-searched but not included, as the outcomes investigated within were not searched for systematically.The quality of the identified studies fed into a narrative analysis, which informed the conclusion of the review regarding each question. The analysis for each question was structured using themes from the Grading of Recommendations Assessment, Development and Evaluation (GRADE) framework, taking into account the volume, quality, applicability and consistency of the evidence. In some cases, appraisal of the volume of the evidence alone (in terms of the number of studies and the number of participants in each study) was sufficient to determine that a UK NSC criterion could not be met.As well as discussing the limitations of the available evidence, e.g. in terms of evidence gaps, a key section of the report discussed the limitations of the review methodology, e.g. pertaining to eligibility criteria  and the record screening approach (such as the use of a single reviewer screening records for relevance in the first instance). An assessment was made of the likely impact of any such methodological choices in terms of the likelihood of pivotal studies being missed . A tabulTo inform a discussion in this paper on different aspects of the VP evidence summary, we applied a checklist of practical items to consider when choosing a rapid review approach, as proposed by Kaltenthaler et al. , and theThe evidence summary reported that little had changed since the previous review in terms of key evidence gaps relating to the epidemiology of VP, the screening test and the management pathway. As a result, the UK NSC concluded that there was insufficient evidence to support a change in its previous recommendation against screening. For VCI, there were some outcomes where the direction of the evidence could not be established due to the limitations of the evidence summary methods.The rapid review was commissioned in May 2016, protocol development commenced in late June 2016, and a first draft of the evidence summary relating to the 9 review questions was prepared in August 2016. The interval from the start of protocol development to delivery of the first draft of the evidence summary was 8 calendar weeks; over this period, approximately 30 person-days of time from the review team were required.An assessment of the review was performed against a checklist of “items to consider when determining a rapid review approach”, developed by Kaltenthaler et al. . This chA summary of the methodological quality assessment using the AMSTAR 2 checklist is presented in Table Finally, detailed results of an assessment of the original review report against the PRISMA reporting checklist are provided in the Supplementary Materials  did not attempt meta-analysis. In the current review, narrative synthesis enabled the identification of a subset of outcomes for which focused meta-analyses may help to address conflicting reports of associations between VCI and fetal/neonatal mortality, pre-eclampsia and low Apgar score. The narrative synthesis also highlighted the paucity of prospective research into the benefits and harms of defined screening, management and intervention pathways. As a result of this, two related projects have been commissioned. These are a meta-analysis to explore the subset of issues relating to VCI and a modelling exercise to estimate the potential impact of screening and help gauge the viability of primary research. In this sense, the review met the commissioning requirement to highlight further work that could be done before the next scheduled review of this topic.As the current rapid review was considered to be an adequate tool for the UK NSC decision-making mechanism but did not meet all relevant AMSTAR 2 checklist items, this poses the question of how to measure the quality of rapid review products. SLR methods provide a valuable reference point for the conduct and reporting of rapid reviews and serve to identify factors potentially increasing the risk of bias in rapid reviews as compared with SLRs. However, SLR quality appraisal tools may undervalue rapid reviews if the purpose and context of these products are not factored in to the assessment of quality. None of the checklists applied in the current case report considered these elements, which may be beneficial in any future quality assessment and reporting checklists being developed and validated for use specifically on rapid reviews.In this case report, the rapid review results were available within the required timelines and enabled the UK NSC to make an informed decision about whether there was sufficient evidence to reconsider the established recommendation regarding population screening for VP. On this occasion, use of a rapid review as part of the UK NSC’s process enabled a pragmatic approach to assessing the overall volume, quality and direction of literature on key questions relating to the viability of a population screening programme for VP. Confirmation of the validity of the methods used in this case report, as well as the wider generalisability of our insights relating to rapid review practice, reporting and quality assessment, would require analysis of a larger sample of UK NSC reviews.Additional file 1: Table S1. UK NSC reporting checklist for evidence summaries, completed for the VP review. This table contains a completed version of the reporting checklist for the rapid review.Additional file 2: Table S2. Quality assessment against checklist adapted from Kaltenthaler et al. This table contains full details of the results of the quality assessment of the rapid review using the checklist developed by Kaltenthaler et al.Additional file 3: Table S3. AMSTAR 2 quality assessment of the evidence summary. This table contains full details of the results of the quality assessment of the rapid review using the AMSTAR 2 checklist.Additional file 4: Table S4. Completed PRISMA checklist for the rapid review.


Due to the complexityof research objects, theoretical concepts, and stimuli in media research, researchers in psychology and communications presumably need sophisticated measures beyond self-report scales to answer research questions on media use processes. The present study evaluates stimulus-dependent structure in spontaneous eye-blink behavior as an objective, corroborative measure for the media use phenomenon of spatial presence. To this end, a mixed methods approach is used in an experimental setting to collect, combine, analyze, and interpret data from standardized participant self-report, observation of participant behavior, and content analysis of the media stimulus. T-pattern detection is used to analyze stimulus-dependent blinking behavior, and this structural data is then contrasted with self-report data. The combined results show that behavioral indicators yield the predicted results, while self-report data shows unpredicted results that are not predicted by the underlying theory. The use of a mixed methods approach offered insights that support further theory development and theory testing beyond a traditional, mono-method experimental approach. Experimental research in media psychology and communications focuses on complex human experiencing and behavior, and often uses complex theorizing to explain respective phenomena. To examine these theories, researchers regularly use complex stimuli, such as movies, video games, or virtual reality environments. In addition to this inherent complexity, further challenges can arise if researchers are not only interested in outcomes of media use, but also in the dynamic process aspects of media use phenomena. In this case, summative, post-session self-report scales may not be sufficient to describe the dynamic nature of media use phenomena. Instead, more sophisticated measurement instruments, such as continuously recorded observational measures can be used to corroborate self-report measures.In this article, we describe a study on the media use phenomenon of spatial presence, or “the conviction of being located in a mediated environment” , p. 495,To address this research question, it is necessary to consider subjective viewer experience, objectively observable viewer behavior, and aspects of the media stimulus. The present study collects, integrates, and analyzes data from all three involved domains: First, self-report data on presence experiences is collected with an established, standardized questionnaire instrument   . The movnorance) . Before,M = 20.26°C, SD = 0.27, range: 19.50 – 20.90; relative humidity: M = 39.79%, SD = 1.82, range: 38.20 – 44.00).The experiment was conducted in a cinema-like, acoustically optimized, darkened laboratory with identical technical setup for all sessions. The stimulus video was available in Blu-ray quality, and was presented on a 150″ projection screen at a viewing distance of 4.3 m in upscaled 4 K resolution. Audio was presented with a room-calibrated 7.2 channels surround sound system. Synchronized observations of the screen and of participants were recorded with unobtrusively placed near-infrared pan-tilt-zoom dome cameras and the Noldus Media Recorder 2 software. Before each session, temperature and relative humidity were recorded with a consumer-grade digital thermometer and hygrometer. Environmental conditions in the laboratory were relatively stable  with normal or corrected to normal vision participated in the study, and were granted course credit, if desired. Twenty participants or 32.8% of the sample were already familiar with the stimulus; however, due to existing findings on intra-subject eye-blink synchronization during repeated viewing of a stimulus .The study was carried out in accordance with the research institution’s ethical guidelines. All participants provided written informed consent in accordance with the Declaration of Helsinki. Sixty-one participants . Because very low kappa has been reported to possibly occur under certain conditions despite high agreement percentage . Average inter-rater agreement with Cohen’s kappa was very low . Four participants were excluded from further analyses because their blink rate was more than two standard deviations above the coded sample’s mean.OpenCV toolbox  = −1.69, p = 0.951, d = −0.48). In fact, participants in the form condition reported higher presence scores . A follow-up one-tailed independent samples t-test showed that this difference was statistically significant (t(47) = −1.69, p = 0.049, d = −0.48; see We hypothesized that participants in the high presence condition, who were instructed to view the movie, report higher presence scores than participants in the low presence condition, who were explicitly instructed to focus on form aspects of the movie. The four item SPSL scale showed good internal consistency with a Cronbach’s alpha of t-test requirements were not met, one-tailed Mann-Whitney U tests were calculated, and showed that participants in the form condition reported significantly higher attention to technical aspects than the content group , but did not differ significantly from the content group on the other measures .Compliance with instructions was tested for both groups using the additional, corroborative manipulation check items. Since n = 24, M = 24.17, SD = 5.72; low presence: n = 25, M = 21.64, SD = 4.76). The requirements for a one-tailed, independent samples t-test were met, and the test showed that significantly more T-patterns with blinks and media events were identified in participants in the content group (t(47) = 1.68, p = 0.050, d = 0.48; see T-pattern analysis with consecutively numbered peaks in optical flow as media events, and blinks as user events identified 43 different T-patterns in the content group, with 580 combined occurrences, and 42 different T-patterns in the form group, with 541 occurrences. Comparison of detected T-patterns of length 2 in real data and randomized data showed that considerably more T-patterns were detected in real data see , thus sun = 24, M = 73.25, SD = 15.05; low presence: N = 25, M = 62.24, SD = 15.76). The requirements for a one-tailed, independent samples t-test were fulfilled. The test showed that significantly more T-patterns with blinks and media events were identified in participants in the content group [t(47) = 2.50, p = 0.008, d = 0.71; see T-pattern analysis with consecutively numbered, scene-related changes as media events, and blinks as user events identified 139 different T-patterns with 1,758 combined occurrences in the content group, and 134 different T-patterns with 1,556 occurrences in the form group. Comparison of detected T-patterns of length 2 in real data and randomized data showed that considerably more T-patterns were detected in real data see , thus suThe present study aimed at evaluating stimulus-dependent temporal structure in spontaneous eye-blink behavior as a possible alternative measure for spatial presence experiences. To this end, data from self-report, observation, and content analysis have been collected in an experiment with externally valid stimulus and setting, and two conditions with low and high presence potential. The various data have been integrated and analyzed to assess the degree of convergence between the objective indicator and an established self-report measure for presence. While analysis of data from observation and content analysis showed the predicted result of more stimulus-dependent structure in blinking behavior in the high presence condition, the different self-report measures showed divergent and unpredicted results. Participants in the low presence condition reported a stronger focus on the medium itself, but, at the same time, higher levels of presence experiences.Thus, the mixed methods approach revealed an overall pattern of results that contradicts predictions of the underlying theory. According to the manipulation check items, participants in the low-presence condition paid more attention to stimulus form features, but according to the standardized SPES questionnaire, they also reported higher presence experiences. If all measures are valid, then this pattern of results is in conflict with the underlying presence theory, since presence should emerge from not paying attention to the medium itself, and should be diminished when viewers focus on the medium itself e.g., . Even whAside from the study’s specific results, structured blinking behavior as a measurement approach for presence has limitations in itself due to its theoretical and conceptual foundations. First, structured blinking behavior is not directly influenced by presence experiences. It rather reflects the activity of attentional and other cognitive processes that should engage in processing of mediated information when recipients are in a state of presence, such as orienting responses on media content . Like otIn the present study, two approaches have been used to identify relevant media events for T-pattern detection: optical flow as a basic, perception-related property of the stimulus, and media events related to the cinematic presentation of stimulus content. Future studies could extend the focus on media features further to the domain of media content by considering, for example, suspenseful media events and their relation to attentional focus, as has been addressed for the media use phenomenon of narrative transportation , or by cDifferent from a mono-method study design, the present study included data from several sources. Standardized self-report measures and computationally defined media events were included as purely quantitative measures. Theoretically derived media events were included as data that has been obtained by considering qualitative aspects during content analysis of the stimulus. Above all, our procedures of collecting observational data on viewer behavior, and the subsequent analysis of behavioral structure, can be regarded as an important component of a mixed methods approach . DetaileWhile the overall, conflicting pattern of results does not allow for a conclusive answer to the original research question, the mixed methods approach nevertheless allowed for a level of insight into the research topic that could hardly be obtained in a mono-method study. Theory development on presence, and probably on other media use phenomena, as well, can benefit from more complex studies and measures that offer researchers a process perspective on the dynamic nature of media use phenomena.The datasets collected for this study are available upon request from the corresponding author.This study was carried out in accordance with the ethical guidelines of the German Psychological Society , the American Psychological Association (APA), and the Declaration of Helsinki. All participants provided written informed consent. The stimulus movie has been released for audiences as of 12 years of age by the German movie rating board , and was thus assessed as appropriate. Due to the use of standard stimulus material and measurement methods, no approval of the local ethics committee had been requested.The major part of the study was conducted as part of the doctoral thesis of MB, with FS as thesis supervisor. MB and FS contributed conception and design of the study’s extension with content-analysis-defined media events, and MB contributed the additional analyses in this part. All authors contributed to manuscript drafting and revision, and have read and approved the submitted version.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


Ankle fractures are painful and debilitating injuries that pose a significant burden to society and healthcare systems. Patient reported outcome measures (PROMs) are commonly used outcome measures in clinical trials of interventions for ankle fracture but there is little evidence on their validity and reliability. This systematic review aims to identify and appraise evidence for the measurement properties of ankle specific PROMs used in adults with an ankle fracture using Consensus Based Standards for the Selection of Health Measurement Instrument (COSMIN) methodology.We searched MEDLINE, Embase and CINAHL online databases for evidence of measurement properties of ankle specific PROMs. Articles were included if they assessed or described the development of the PROM in adults with ankle fracture. Articles were ineligible if they used the PROM to assess the measurement properties of another instrument. Abstracts without full articles and conference proceedings were ineligible, as were articles that adapted the PROM under evaluation without any formal justification of the changes as part of a cross-cultural validation or translation process. Two reviewers completed the screening. To assess methodological quality we used COSMIN risk of bias checklist and summarised evidence using COSMIN quality criteria and a modified Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach. Two reviewers assessed the methodological quality and extracted the data for a sample of articles.The searches returned a total of 377 articles. From these, six articles were included after application of eligibility criteria. These articles evaluated three PROMs: A-FORM, OMAS and AAOS. The A-FORM had evidence of a robust development process within the patient population, however lacks post-formulation testing. The OMAS showed sufficient levels of reliability, internal consistency and construct validity. The AAOS showed low quality evidence of sufficient construct validity.There is insufficient evidence to support the recommendation of a particular PROM for use in adult ankle fracture research based on COSMIN methodology. Further validation of these outcome measures is required in order to ensure PROMs used in this area are sufficiently valid and reliable to assess treatment effects. This would enable high quality, evidenced-based management of adults with ankle fracture. Ankle fractures cause significant pain, reduced mobility and subsequent limitation of usual activities . The injClinical trials of interventions for fractures of the lower limb often utilise Patient Reported Outcome Measures (PROMs) as primary outcomes –13. It iConducting a randomised controlled trial is expensive, time consuming and relies on the good will of participants to be randomised to an intervention and complete questionnaires. If the PROM used in a clinical trial does not measure the treatment effects of the interventions in a valid and reliable way, this places the unnecessary burden of randomisation and trial processes onto participants. Using PROMs with insufficient measurement properties in randomised controlled trial is therefore a waste of resource and unethical . A systeThe aim of this review is to identify and critically appraise the available evidence for the measurement properties of foot and ankle specific PROMs for use in adults with an ankle fracture. The results of this review will aim to determine the most appropriate instrument for use in evaluating change resulting from interventions in the context of randomised controlled trials in this research area.We prospectively registered this review with PROSPERO International Prospective Register of Systematic Reviews (Reference CRD42018103112). Consensus Based Standards for the Selection of Health Measurement Instrument (COSMIN) Methodology for Systematic Reviews of Measurement properties of PROMs was adhered to  and thisThis review was completed following a previous systematic review looking to assess all outcome measures collected in clinical trials of interventions for ankle fracture . The outIncluded articles assessed the measurement properties, development or interpretability of one or more of the PROMs included in the pre-specified list in a majority patient population of adults with ankle fracture. Here, majority is defined as equal to or greater than 50% of the sample. In articles which did not reach the criteria of 50% but performed a separate analysis on the ankle fracture sub-sample of patients, these articles were included and only the analyses performed on the single sub-sample of individuals with ankle fracture were included; any analyses on the sample as a whole or comparing the two clinical groups were not included.Articles were ineligible for inclusion if they use the PROM/s only for outcome measurement in an experimental study, where no formal evaluation of a measurement property is completed. Articles which use the PROM in question to validate another PROM (not on the pre-specified list here) were also ineligible for inclusion. Studies were excluded if the authors adapted the PROM in any way without formal justification of the changes as part of a translation or cross-cultural validation process. Abstracts without full articles and conference proceedings were not eligible for inclusion.A systematic search of the literature was completed using the MEDLINE, EMBASE and CINAHL databases on 16/04/2019 up to the present date with no date limits applied using search strategies developed by the COSMIN group specifically for this type of review . AdditioThe lead author and a second reviewer (AR) independently screened the articles by title and abstract for possible inclusion. The reviewers selected any articles which were potentially eligible from title and abstract review and retrieved the full text. If it was unclear at the initial title and abstract review, the full text was retrieved and reviewed for purposes of completeness. If at least one of the reviewers felt that a study might be eligible based upon the initial title and abstract screening, then both researchers independently reviewed the full text to assess eligibility for inclusion. The reviewers then discussed findings and reached consensus on inclusion of articles. In instances of disagreement, a third reviewer (RSK) was consulted for a final decision.The methodological quality of the articles included in this review was assessed using the COSMIN risk of bias checklist . EvidencA decision was made that the criteria and box for criterion validity was not to be completed as there is no accepted gold-standard measure for assessing outcome in adults with ankle fracture, therefore this measurement property does not apply in this particular case. If reported, data on the interpretability and feasibility of the PROMs were also extracted and reviewed. We contacted developers of the PROMs where possible to obtain a copy of the user manual (if available) and to ensure that, to their knowledge, there were no further validation studies on the scores which may not have been identified in the database searches.A weak correlation is defined as < 0.30A weak to moderate correlation is defined as > 0.20 - < 0.40A moderate correlation is defined as > 0.30 - < 0.70A moderate to high correlation is defined as > 0.60 to < 0.80A high correlation is defined as > 0.70Hypotheses for assessing the construct validity evidence in the instances that this was assessed in the included articles was pre-defined . The folThe hypotheses tested during this review for construct validity are outlined in Table The searches produced a total of 377 returns. Following initial screening of the titles and abstracts, 353 records were excluded, leaving 24 articles for full text review. Following full-text review of the 24 articles, six articles were included in this review –34 and dTable All of the PROMs included in this review are paper based questionnaires self-administered by the patient either in a clinical or research context. The AAOS consists of 25 questions including stiffness (one item), swelling (one item), pain (nine items), giving way (three items), function (six items) and footwear (five items). The score consists of a core score (AAOS-CS) comprising of 20 items and a shoe comfort scale (AAOS-SCS) comprising of five items. The scores are calculated to a normative score for each of these two scales, which is then converted to a summative mean for both the AAOS-CS and AAOS-SCS. The summative score for each subscale ranges between 0 and 100 with higher scores indicating a more favourable outcome.The A-FORM consists of 15 items including pain, swelling, stiffness, anxiety regarding footwear, sleeping, jumping, waking, social aspects, anxiety related to future ankle function, depression and fatigue. The raw score is converted to a summary score which ranges between 0 and 100, with lower scores indicating more favourable outcomes. The footwear item is not included in the summary score conversion, so users are asked to omit this item from the summary score conversion process. The summary score conversion table is found in the user manual which can be requested from the developers at no cost to users. The summary score conversion was based on the Rasch analysis presented in the development article included in this review .The OMAS is a nine-item questionnaire including pain, stiffness, swelling, stairs, squatting, supports, jumping, running and usual activities. Final scores range between 0 and 100 with higher scores indicating more favourable outcomes. The score is totalled using the scoring system provided in the development paper included in this review . DiffereTable Following the COSMIN guidance for PROM development, an article encountered in the reference list of the A-FORM development articles  was takeTable The AAOS demonstrated low levels of evidence for sufficient construct validity. Zelle et al.  correlatMcPhail et al.  detailedAuthors of the included studies assessed the translated versions of the OMAS for structural validity in Norwegian and internal consistency, reliability and construct validity in both Norwegian and Turkish languages. The OMAS Norwegian version achieved high level evidence for sufficient construct validity; Garratt et al.  correlatTable There was no information reported in any of the included studies on response shift or minimal importance difference of the measures therefore these facets of interpretability have not been included in Table COSMIN methodology advises that in order to recommend a PROM, it should demonstrate any level of content validity and a minimum of low level evidence for internal consistency . None ofThis review demonstrates that at the time this review was undertaken, none of the PROMs used in clinical trials of interventions for ankle fracture had adequate evidence of measurement properties and we are therefore unable to recommend a particular PROM for use in this context and patient population. Furthermore, there were four additional PROMs  which have been or are currently being used in clinical trials of interventions for ankle fracture for which the current review did not find any evidence of their measurement properties within the patient population. Whilst the OMAS demonstrates sufficient internal consistency, structural validity and construct validity, the PROM development scored poorly against COSMIN criteria used in this review. In contrast, the A-FORM demonstrates some evidence for PROM development within the patient population, but there is limited post-formulation testing of this PROM.This review updates the one completed in 2016 by Ng et al. .Strengths of this review include the use of a well-developed, thorough and consensus based methodology and search filters for finding and reviewing the evidence for development and measurement properties of PROMs. Limitations of the review include the inherent difficulty in defining the construct under analysis; there is little research into the experiences of individuals recovering from an ankle fracture and further research into the construct of interest would be beneficial. The construct of outcome in ankle fracture recovery may vary depending on several individual factors, such as age, gender and whether the fracture is treated operatively or non-operatively. When considering the varied distributions of the different ankle fracture patterns which has been demonstrated in the epidemiological literature , one couWe encountered difficulty in applying the COSMIN methodology and assessment criteria to older articles such as the development of the OMAS instrument . We acknThis review shows that currently there is no PROM that can be recommended for use for the purpose of assessing outcome in clinical trials of interventions for ankle fracture. Further validation work should focus on ascertaining the acceptability, relevance and comprehensiveness of commonly used questionnaires such as OMAS in a population of adults with ankle fracture. Future research studies in this area should make use of COSMIN based standards for designing and reporting validation research to ensure that the appropriate evidence base is acquired for a PROM to be recommended. As this review demonstrates, there is no evidence that this PROM was formulated with the input of individuals who have ankle fractures and understanding the content validity of this widely used instrument would enable an understanding of whether it is fit for purpose in the patient population or whether the use of this outcome measure should be discontinued. Furthermore, the OMAS demonstrated ceiling effects in excess of the widely recognised acceptable level of 15% , 38, whiFuture exploratory research should aim to understand the patient experience of ankle fracture and the factors of most importance to individuals with this injury, with an understanding that this may differ between age group of the individuals and possibly fracture management. It might well be that the construct between these groups differs so much that it is not appropriate for the same PROM to be used between these populations. Exploring the relevance and comprehensiveness of PROMs such as the OMAS which were not developed with input from the patient population would be beneficial to ascertain the appropriateness of the ongoing use of these outcome measure. None of the articles here assessed the responsiveness of the PROMs and future research should seek to ensure that the instruments are suitably responsive to detect treatment effects in resource-intensive clinical trials. Furthermore, validation of the A-FORM questionnaire to ascertain the measurement properties of this PROM in its final format would be advantageous. Further validation research of the PROMs used in ankle fracture is warranted here to ensure that randomised controlled trials in this clinical area answer the questions needed to manage these individuals most effectively. Furthermore, the preparation of an agreed core outcome set for use in this patient population would be advantageous, enabling the conduct of high quality trials using an appropriate and standardised set of outcome measures for this important injury.Additional file 1. Search strategies.Additional file 2. PRISMA Checklist.


The rapid pace of invention in technology and the evolution of network communication has produced a new lifestyle with variety of opportunities and challenges. Remotely Piloted Aerial Systems (RPAS) technology, which includes drones, is one example of a recently invented technology that requires the collection of a solid body of defensible and admissible evidence to help eliminate potential real-world threats posed by their use. With the advent of smartphones, there has been an increase in digital forensic investigation processes developed to assist specialized digital forensic investigators in presenting forensically sound evidence in the courts of law. Therefore, it is necessary to apply digital forensic techniques and procedures to different types of RPASs in order to create a line of defense against new challenges, such as aerial-related incidents, introduced by the use of these technologies. Drone operations by bad actors are rapidly increasing and these actors are constantly developing new approaches. These criminal operations include invasion of privacy, drug smuggling, and terrorist activities. Additionally, drone crashes and incidents raise significant concerns. In this paper, we propose a technical forensic process consisting of ten technical phases for the analysis of RPAS forensic artifacts, which can reduce the complexity of the identification and investigation of drones. Using the proposed technical process, we analyze drone images using the Computer Forensics Reference Datasets (CFReDS) and present results for the Typhoon H aerial vehicle manufactured by Yuneec, Inc. Furthermore, this paper explores the availability and value of digital evidence that would allow a more practical digital investigation to be able to build an evidence-based experience. Therefore, we particularly focus on developing a technical drone investigation process that can be applied to various types of drones. According to the National Institute of Standards and Technology (NIST) , criminaThis paper examines digital evidence and artifacts using the benchmark drone forensic images of the Yuneec, Inc. Typhoon H drone, which was recently involved in delivering contraband content to a prison . Flying Previous research introduced a forensic analysis of the DJI Phantom 3 . This paThis paper introduces an enhanced Drone Technical Forensic Investigation Process based on acquired drone data from different components of the Typhoon H.This paper creates a standardized approach to conduct a digital forensic analysis of a drone using the outcome of this research by proposing a digital forensic investigation into technical processes for drones.This paper utilizes validation and verification methods for the forensics analysis of drones. It is important to note that the validation process is critical to meet the standards of a legitimate digital forensics investigation. Therefore, several tools were tested in relation to the analysis of digital evidence found in drone components.This paper is structured as follows. In A recent work  highlighOne of the most relevant works to our research is presented in . The autIn addition, Jain et al.  analyzedClark et al. presented the analysis of the DJI Phantom III in  where thThe primary focus of this research is to generate a technical process that can be implemented by forensic examiners and law enforcement investigators. Additionally, in the event that all drone components are available, a complete report of findings can be achieved. For instance, Phase 4 of the proposed technical process draws the importance of investigating media file storage and flight logs to extract relevant digital evidence and validate them accordingly, as described in Phase 5. Validation of digital evidence is crucial to make sure that data have not been altered. This can be achieved by comparing timestamps and meta-data found in media files with flight activity logs, which includes information found in media files, such as flight duration, and GPS metadata. Therefore, the analysis of both media files and flight activity logs lead to more accurate findings. However, if the collection of digital evidence is only on the controller of the drone, then investigating media file storage is not guaranteed because media files are stored in the memory cards.Furthermore, the availability of the remote controller makes the analysis smoother. Similarly, withdrawing the flight activity log provides an extra layer to the visualization and analysis of GPS data, as shown in Phase 6; however, the storage mechanism of an RPAS stores flight logs on the remote controlled storage device. Therefore, it is difficult to recover flight activity logs from components other than the remote control. Consequently, Phase 7 demonstrates the possibility of recovering encrypted flight logs. For instance, DJI drones use encryption mechanisms to store flight logs which require decryption of these files in order to visualize GPS data. As illustrated in Phase 8, however, if decryption is not successful  then that will move the process to the “reporting” phase (Phase 10). On the other hand, Phase 9 is an important phase in the proposed technical process, which has not been addressed in previous work or even non-traditional digital forensic process. This phase deals with the analysis of flight status leading to a behavioral analysis of the drone’s activity. Phase 9 is described as the “interpretation of drone black-box”, which is discussed in more details in On the other hand, if a remote controller is not available, then the investigator could apply different approaches in order to analyze media files. The file system structure of media storage is illustrated in Memory Card of CameraDrone chipController chipThis research employed the Typhoon H drone as a case study to better understand the structure of digital evidence available on drones and provides ten technical phases to support the investigation process of drone forensics. This paper divides the forensic analysis of drones into three parts, as shown in y-axis and x-axis to identify relevant information about the whole flight activity, so as to reach a better conclusion before reporting digital forensic findings. For instance, interpreting the altitude, strength of signals, and the three principal axes of an RPAS  would aid in determining if an incident was intentional (possible attack) or accidental.The Autopsy 4.6.0 forensic tool was mainly used for data recovery, alongside some keyword searches where we analyzed the file system of each component and searched for relevant files that could aid in investigative reporting. Tools such as ExifTool 11.10, Winhex 16.7, MediaInfo 18.8.1, and 010 Editor 8.0.1 were helpful in analyzing the content of exported media files recovered from drone components. Some of the important information retrieved from these files are partially depicted in The file path location is important, because an image acquired from a MicroSD card of the camera of the drone is used in identifying that the file system is a FAT32 type, which contains all videos in the “DCIM” folder. Then, it is straightforward from that point onward. For instance, to extract all media files of type “MP4” under the following path “\1\vol_419430\DCIM\100MEDIA\YUN00001.mp4”, the first step is identifying the timestamps of any media files. Since the focus of this paper is to develop a technical process for drone forensics, the “YUN0001.MP4” media file was analyzed as a scenario. We investigated the timestamps of the media file “YUN00001.mp4” using ExifTool, written by Phil Harvey, and subsequently discussed further analysis regarding the investigation.The MP4 file format consists of multiple atoms, and our research focuses exclusively on those atoms that are relevant to forensic investigators, especially for RPAS digital evidence. Boxes of the MP4 file are identified in a 4-byte sequence, which starts with the “ftyp” box that indicates the file type  atom belongs to the top level and is an invaluable spot for digital forensic investigators. In our case, the “moov” atom starts at offset 1481989013 with a size of 20,405 bytes. The structure of the stored MP4 video contains multiple timestamps under the “moov” atom. Interestingly, during the video streaming of the drone, the timestamps are recorded in a sequence of three-second updates. The actual timestamps of the created video can be extracted using Exiftool. The command “Exiftool.exe—CreateDate—Duration YUN000001.MP4” can be used to extract metadata from the video. The above metadata extraction would help digital forensic investigators to link appropriate digital evidence in the next phase of the analysis. Our initial plan was to extract GPS metadata information from the stored videos in the SD card. However, we concluded that the Typhoon H drone employs a different storage structure and GPS metadata are not stored in the camera memory. This is an unlikely case for other types of drones such as the DJI Phantom, where media files contain GPS metadata . We now http://www.gpsvisualizer.com.Particularly for the studied drone, the controller has a significant amount of information that can aid in a forensics investigation. The controller of the Typhoon H drone is equipped with an Android operating system ground station and a screen attached to it. Note that most of the other known drones require the use of mobile device applications instead of an additional hardware. This feature enables the digital forensic investigator to access GPS related metadata without the need for file decryption. For instance, the GPS metadata of the DJI drone requires an investigator to decrypt the DAT file in order to collect relevant information . HoweverIn The process of drone forensics involves three main evidentiary categories. Each and every drone has propriety structures. However, the large number of file systems involved in a particular case is considered to be a technical investigation issue. This also applies to the number of acquired forensic images related to a single case. Therefore, the outcome of the analysis presented in this paper is to assist in the development of a technical process for drone forensics.Media files include the extraction and analysis of videos and images.Flight logs include sensor metadata and GPS coordinates .Event logs include controller commands, connection errors, and system events.It is crucial to take the issue of anti-forensic techniques into consideration when dealing with such technology. Our preliminary results in  present In this paper, we propose ten technical investigative phases for forensic analysis of RPASs, where the Yuneec, Typhoon H is used as a case study. In addition, the technical process identifies “preparation” as the first phase, which describes the preparation of digital evidence to include a collection of electronic devices at the crime scene. The next phase is the identification of both physical and digital evidence. This phase identifies the types of digital evidence including the classification of the drone. The third phase is where the forensics investigation commences, either on all components of the drone or on some of them. Based on the third phase, there will be subsequent technical steps that will need to be followed in order to achieve a complete and forensically sound report. Phases 4–10 primarily concentrate on the analysis of outcomes during the investigation, which, if adhered to, can result in a successful, and forensically sound collection of evidence and reporting.The proposed work in this research was applied in our initial phase in generating a drone forensic tool to aid in reporting illegal flight activities. Upcoming works can be done on other types of drones based on the proposed technical process in order to develop a complete structure of drone forensics. In addition, we are currently developing open source tools which can analyze different atoms of media files as well as analyze digital evidence found on drones to enhance the validation procedure of digital evidence.Based on the methodology discussed in acquiring forensically sound digital evidence, we deliberated more by developing a forensic tool to examine a hypothetical drone case automatically. This case deals with a specific type of incident of drones such as flying in a restricted area. Here, we assume that a drone has flown in a no-fly zone for any purpose, and law enforcement officers have seized the drone body only. The question we will answer is: How can we prove flight activities in no-fly zones in a forensically sound manner, and reach the admissibility level against criminals? Therefore, a tool that can acquire the extracted media files from the drone’s memory storage and process its GPS metadata to match it with no-fly zone location would aid in investigating the present response to such incidents. More so, in the availability of the drone control, it is important for an investigator to conduct behavioral analysis by analyzing the flight activity to interpret the “drone black-box” and reach a classification of certain measures to reflect the category of the pilot intention. On the other hand, considering drone anti-forensic and customization techniques are important aspects in drone incident response. In addition, an analysis of the secure encryption mechanism of flight logs is crucial on all commercial RPASs to enhance admissibility of drone digital evidence.


Single Hub and Access point for pediatric Rheumatology in Europe (SHARE) was launched to optimise and disseminate diagnostic and management regimens in Europe for children and young adults with rheumatic diseases. Juvenile idiopathic arthritis (JIA) is the most common rheumatic disease in children and uveitis is possibly its most devastating extra-articular manifestation. Evidence-based guidelines are sparse and management is mostly based on physicians’ experience. Consequently, treatment practices differ widely, within and between nations.In 2012, a European initiative called To provide recommendations for the diagnosis and treatment of JIA-associated uveitis.Recommendations were developed by an evidence-informed consensus process using the European League Against Rheumatism standard operating procedures. A committee was constituted, consisting of nine experienced paediatric rheumatologists and three experts in ophthalmology from Europe. Recommendations derived from a validated systematic literature review were evaluated by an Expert Committee and subsequently discussed at two consensus meetings using nominal group techniques. Recommendations were accepted if >80% agreement was reached .In total, 22 recommendations were accepted (with >80% agreement among experts): 3 on diagnosis, 5 on disease activity measurements, 12 on treatment and 2 on future recommendations.The SHARE initiative aims to identify best practices for treatment of patients suffering from JIA-associated uveitis. Within this remit, recommendations for the diagnosis and treatment of JIA-associated uveitis have been formulated by an evidence-informed consensus process to suggest a standard of care for JIA-associated uveitis patients throughout Europe. In 2012, Single Hub and Access point for pediatric Rheumatology in Europe (SHARE) was launched with the aim of optimising and disseminating diagnostic and management regimens for children and adolescents with rheumatic diseases. The European League against Rheumatism (EULAR) has produced a number of recommendations in the area of paediatric rheumatology such as juvenile dermatomyositisJuvenile idiopathic arthritis (JIA) is the most common rheumatic disease in children, and uveitis is the most frequent and potentially most devastating extra-articular manifestation. There are no international consensus statements specifically relating to the diagnosis and treatment of JIA-associated uveitis although there are national guidelines from Germany and Spain.With the rapid development of novel therapies for JIA, clear recommendations based on available best evidence and expert opinion  will help physicians in the care of patients with JIA-associated uveitis. The uveitis seen with JIA is usually chronic anterior uveitis which is asymptomatic, but acute anterior uveitis can also be seen in the enthesitis-related arthritis subtype.prevent or reduce the likelihood of JIA-associated uveitis from occurring and minimise the damage at the time of diagnosisrecommend treatments and management strategies that would reduce inflammation and prevent the development of those ocular complications most likely to cause irreversible visual loss.The primary aims of the recommendations forwarded by the Expert Group were to develop agreed strategies toA committee of 12 experts  in paediatric rheumatology (n=9) or ophthalmology (n=3) was established to develop recommendations for JIA-associated uveitis based on consensus, but evidence informed, using EULAR standard operating procedures for developing best practice.The electronic databases PubMed/MEDLINE, Embase and Cochrane were searched independently by two researchers for eligible articles in February 2015. All synonyms of JIA were searched in Medical Subject Headings/Emtree terms, titles and abstracts . ReferenThe selected articles were randomly allocated to the expert group, and two members per paper independently assessed the methodological quality of those papers meeting the inclusion criteria . Data weAs part of the EULAR standard operating procedure, experts described the main results and conclusions of each paper, along with their validity and level of evidence. These descriptions were collated by five experts  and used to formulate provisional recommendations which were reviewed by a panel of three experts . A summary of the evidence was presented along with each provisional recommendation to the expert committee. The recommendations were revised and discussed at a face-to-face meeting in March 2014 (Genova: 12 participants), using a Nominal Group Technique to reach a consensus.The literature search yielded a total of 1323 papers which, after deduplication, left 1259 unique articles. Evaluation of inclusion/exclusion criteria in titles and/or abstracts resulted in a core reference database of 176 articles for which full-text copies were obtained for quality screening by the expert committee. Of these, 117 were selected to support the development of consensus-based recommendations by the expert group .The sections that follow report the recommendations of the expert committee based on the supporting literature.JIA is the most common chronic rheumatic disease in children with an incidence of 8.2 (7.5–9.0)/100 000 of the population under 16 and an annual prevalence of approximately 70.2 [16 – 140]/100 000.Structural complications, some of them leading to irreversible visual loss, include cataracts, glaucoma, band keratopathy, macular oedema, retinal detachment and sequelae associated with chronic hypotony. The uveitis usually has an insidious time course and can be chronic or recurrent but, most frequently, JIA-associated uveitis is a chronic relapsing condition lasting several years. It almost universally starts as an anterior uveitis, but in rare instances can become a panuveitis. JIA-associated uveitis is usually asymptomatic in the age group in which it most commonly develops (age 3–7 years), but severe inflammation may cause symptomatic pain and redness and these symptoms and signs, as well as pupil distortion, may be noticed by carers and lead to early referral. Reduced visual function is an uncommon cause of presentation in small children unless it is severe and usually secondary to irreversible structural damage. The presence of antinuclear antibodies (ANA), oligoarthritis and early onset of arthritis are predominant risk factors for chronic anterior uveitis in those with JIA, and improved identification of children at risk is a key priority for targeted screening programmes. This exercise only looked at chronic anterior uveitis associated with JIA and not at acute anterior uveitis seen in human leukocyte antigen (HLA) B27-positive children with enthesitis-related arthritis.Timely and aggressive treatment is clearly needed where there is significant damage at the time of diagnosis. However, the variable course of the disease means that some significant events may occur years after therapy commences and there remains considerable disagreement among expert practitioners about the timing and indications for treatment escalation.All patients with JIA should be screened for uveitis according to contemporary and audited screening protocols which should be implemented in all centres in which patients with JIA are being managed.Advances in genetics have not found any more specific diagnostic markers than the HLA types originally reported as being associated with both oligoarticular disease and uveitis.Despite the advances in subtyping ANAs in other rheumatic disorders, there has been little advance in our understanding of the association of ANAs with the risk of JIA-associated uveitisThere is a clear unmet need to adapt screening policies to the contemporary usage of early systemic immunosuppressant treatment of arthritisThe risk of visual symptoms and potential for relapse in patients initially responding to treatment highlights the necessity for maintained regular close ophthalmological scrutiny. The expert group recommended that the frequency of ophthalmological follow-up should be based on ocular disease severity and decided upon in conjunction with an expert ophthalmologist.Methotrexate (MTX) is the immunosuppressive therapy of choice in patients with JIA-related uveitis .There are no established biomarkers currently available for predicting disease activity or guiding treatment in JIA-associated uveitis and research efforts in this area need to be intensified .24 32 34In a small group of children with JIA-related anterior uveitis, serum interleukin-2R levels were significantly increased,Active uveitis in JIA usually requires immediate treatment. In a comparison of two cohorts of patients with new-onset JIA, separated by a 10-year interval, the more recently treated patients received early intensive treatment and close monitoring, and reported fewer complications and milder uveitis with visual loss avoided in most cases.Based on past usage, the expert group recommend that topical corticosteroids (preferably prednisolone or dexamethasone) are the first-line treatments of choice for both acute and chronic anterior uveitis.One study found that the increased risk of cataract formation with high-dose topical steroids was independent of active uveitis or presence of posterior synechiae.In a retrospective study, the adjunctive use of non-steroidal anti-inflammatory drugs (NSAIDs) for the treatment of chronic iridocyclitis was evaluated in 14 patients, 8 with JIA and 6 with idiopathic iridocyclitis.Systemic immunosuppression for active uveitis is recommended to reduce complications in those cases where topical steroids are insufficient to eliminate ocular inflammation short term, or such high doses are required that treatment risks outweigh the beneficial effects. However, the threshold for introducing systemic treatment is lower in those with multiple risks for visual loss as discussed earlier.Treatment failure should lead to a change in treatment dose, route or modality taking into consideration the fact that many of the drugs require different times to achieve their optimal effect and there are sometimes only a limited number of drugs available. Inappropriate drug changes may result from difficulties in achieving consensus on the definition of treatment failure. In a retrospective study, the clinical outcome of 23 patients with JIA-associated uveitis unresponsive to corticosteroids was ascertained. Uveitis was controlled using immunosuppressive therapy in all cases.Based on findings of efficacy and tolerability/safety from a number of studies, the expert group recommended that MTX is the immunosuppressive therapy of choice in patients with JIA-related uveitis.As noted above, the evidence with other forms of immunosuppressive therapy in patients with JIA-related uveitis is low,All of the biological therapies investigated produced some benefit in patients with uveitis refractory to conventional therapy.The use of TNF inhibitors in uveitis was hypothesised based on their proven efficacy in a range of systemic inflammatory disorders including JIA, rheumatoid arthritis and Crohn’s disease.et al reported preliminary evidence that in seven cases of refractory uveitis switching between biological agents (over a period of 5–24 months) helped to control intraocular inflammation.Findings from studies including small numbers of patients provide evidence that if treatment with one anti-TNF agent becomes ineffective switching to a different anti-TNF agent could prove to be clinically beneficial.While the literature search revealed no direct evidence of the effects of low drug trough levels or the development of anti-drug antibodies (ADAs) on the clinical efficacy of biological agents in patients with uveitis, the expert group considered findings from other clinical settings. They concluded that in cases of loss of efficacy over time consideration should be given to testing for ADAs and drug trough levels.http://www.aptitude-trial.org.uk/).In keeping with an earlier recommendation 18, , there aThe optimum time for surgery in children with complications from refractory uveitis has not been addressed as a recommendation due to paucity of evidence. Recent literature does demonstrate that a significant number of children with uveitis still require surgery for complications.A MIWGUC identified the need for clinical trials and longitudinal studies to determine the benefits and costs of health interventions in this setting.The expert group indicated a need for more well-controlled clinical trials in children with JIA-related uveitis to provide the scientific best evidence in the areas of diagnosis, screening, disease activity and treatment to enable the optimal care of these patients.Following a systematic review of the literature and Nominal Group Technique methodology, under the auspices of SHARE and EULAR operating procedures, 22 recommendations for the screening, diagnosis, disease activity monitoring, treatment and future plans for children with JIA-associated uveitis were accepted with at least 80% agreement. In a disease setting where the evidence base is limited by small numbers of patients, and which is developing rapidly, these expert recommendations should help specialists with the evidence-based advice to provide optimal care for their patients.It should be noted that, in general, the level of evidence was quite low with 13 of 22 recommendations being level 3 or 4, seven level 2 and only two level 1. This highlights the need for more research in this clinical setting where a number of new therapies, particularly biological agents, have been introduced in recent years. At the time that the data search for this article took place, the expert group noted a need for more well-controlled clinical trials in children with JIA-related uveitis. The goal being to ensure that scientific best evidence is used to support optimal treatment. In the interim period, but clearly outside the search time frame of this review, a well-designed randomised controlled trial (RCT) has been published which compared adalimumab with placebo in children with JIA-related uveitis who were taking a stable dose of MTX.