Biocuration has become a cornerstone for analyses in biology, and to meet needs, the amount of annotations has considerably grown in recent years. However, the reliability of these annotations varies; it has thus become necessary to be able to assess the confidence in annotations. Although several resources already provide confidence information about the annotations that they produce, a standard way of providing such information has yet to be defined. This lack of standardization undermines the propagation of knowledge across resources, as well as the credibility of results from high-throughput analyses. Seeded at a workshop during the Biocuration 2012 conference, a working group has been created to address this problem. We present here the elements that were identified as essential for assessing confidence in annotations, as well as a draft ontology—the Confidence Information Ontology—to illustrate how the problems identified could be addressed. We hope that this effort will provide a home for discussing this major issue among the biocuration community.Tracker URL:https://github.com/BgeeDB/confidence-information-ontologyOntology URL:https://raw.githubusercontent.com/BgeeDB/confidence-information-ontology/master/src/ontology/cio-simple.obo Curation in biology has become essential for capturing information from publications or results from experiments, and for making these data available through public repositories. Whether to allow efficient data retrieval , or to make sense of the overwhelming amount of data produced by current technologies , this curation work provides us with standardized datasets that are essential for downstream analyses , 2. HoweAs the volume of biological data has grown, so has the amount of annotations available . This grThe type of evidence used to support the assignment of an annotation is often used as a proxy for judging its quality, in large part owing to the extensive use of the evidence ontology (ECO) . The ECOFor example, ‘microarray evidence’ (ECO:0000058) may report results from a high quality experiment with several biological replicates, or from a single low quality experiment. Or a ‘protein BLAST evidence’ (ECO:0000208) could correspond to a weak similarity over part of the protein, or 99% identity over the whole length of the protein. Another example is the use of annotations automatically assigned by computational methods, without curator supervision, tagged with the related evidence term (ECO:0000501 or GO evidence code IEA); they have often been considered the least reliable, whereas after evaluation, these annotations appeared to be as reliable as curated non-experimental annotations . Since the CIO is still in relatively early stages of development, we anticipate that there might be significant changes to the terms, as the community begins to explore the use of CIO. With this publication, we hope to expand the set of active CIO developers and users, so as to build an optimal ontology for this domain. We then provide suggestions of implementation, to highlight how the use of such an ontology could address the problem of standardizing confidence information about annotations.Annotations are created on the basis of available evidence lines. In this work, evidence is considered to be any scientific evidence obtained from laboratory experiments, computational methods, or manual curation, as described in the ECO. Confidence in a particular assertion can then be defined, based on the set of related evidence lines.Although confidence and quality are related concepts, they are nevertheless distinct. Quality refers to the value of the source or of the annotation, whereas confidence refers to the level of certainty that an assertion is correct. For instance, several evidence lines of low quality, produced by methods known to be noisy (e.g. yeast two-hybrid), could yield an assertion of high confidence, because they are repeatedly confirmed (e.g. with different reporter genes); or, an annotation could be of poor quality because of missing information, but capture a high confidence assertion.In this manuscript, we aim at defining how to provide global confidence about annotations, and not at defining criteria to assess quality of evidence.To convey confidence information in a standard way, an obvious solution is to use an ontology, to provide standard terms for assessing confidence, in the same way the ECO provides standard terms to annotate evidence types. Although it is neither feasible nor desirable for different annotation groups to use a single definition of quality metrics, it should be feasible and desirable for each group to map their quality assessments to terms from a common ontology.supplementary material.In order to demonstrate how such an ontology could address the need for a standard method to provide confidence information, we present here a new ontology that we have developed: the CIO . This onParameters for determining confidence in annotations are highly heterogeneous among groups working on different data types. Yet it is possible to summarize the confidence information using a basic rating system. Each group working on a specific data type could define clear parameters to assign these levels of confidence, and several groups have already implemented such a rating system (see Introduction).P-value threshold, or a percentage of identity.Three CIO terms corresponding to basic levels of confidence have been defined in the ontology: ‘low confidence from single evidence, medium confidence from single evidence, and high confidence from single evidence’ . These cAn additional term, ‘rejected’, allows to tag assertions that were retracted, for instance, following paper retraction, author misinterpretation or curator misinterpretation. This term is used to circumvent the fact that, when results are retracted, associated annotations are often deleted. Consequently, end-users might not be aware that a result was annotated, and then shown to be incorrect. Annotating an assertion with this confidence term would allow to keep this information available, and keep track of the invalidation. Note that this is different from negative annotations, used to negate annotation interpretation. It is also different from conflicting multiple evidence lines, where each single evidence has not been directly invalidated. Rejection is a stronger assertion about an annotation source than conflict, capturing that this evidence should no longer be used.These basic confidence statements should be used for annotating each individual evidence. While it might seem an added burden to annotate an assertion with yet another ontology term, we believe that curators already informally assess evidence sources they use. Thus in many cases a basic rating system could be applied with little added effort (see Discussion).Although some evidence types can have numerical estimates of confidence (e.g. BLAST score), others can only be judged qualitatively (e.g. non-traceable author statement in GO). Confidence information should always be examined in the light of the evidence source used. For this reason, we believe that annotations using the CIO should always be provided along with terms from the ECO. We also believe that for those evidence sources that allow numerical estimates, annotation providers should always publish the parameters used to assign a confidence statement.We are hopeful that, in the long run, agreement on these parameters might be reached by different groups working in the same field. For example, it might be possible to define a certain level of sequence identity that would be a good predictor for a particular class of GO terms; or, certain experimental setup that would provide more trustworthy protein-protein interaction data. It is the standardization of annotation assignment in ontologies such as ECO and CIO that facilitates this.The strongest point of agreement among all workshop participants was that assertions supported by multiple evidence lines are more reliable than assertions supported by a single line of evidence. This means that a global confidence in an assertion can arise from taking all available evidence into account. This led us to create two branches in the ontology: a ‘confidence statement from single evidence’ branch’ and a ‘confidence statement from multiple evidence lines’ branch . The forThe best practice would be that for assertions supported by multiple evidence lines, each individual evidence would still be assigned a term from the ‘confidence statement from single evidence’ branch. The overall confidence in the assertion, assessed from all available evidence lines, would then be summarized using a term from the ‘confidence statement from multiple evidence lines’ branch.Another consensual point was that different, non redundant, experimental or computational methods provide a stronger support for an annotation. For this reason, we created the terms ‘confidence statement from multiple evidence lines of same type’ and ‘confidence statement from multiple evidence lines of multiple types’ , where ehttps://github.com/BgeeDB/anatomical-similarity-annotations/wiki/Similarity-annotations). For instance, an assertion states that the urinary bladder is homologous among Tetrapoda. Some evidence lines provided (Tetrapoda species). These individual statements are of medium confidence, and if we were to integrate these two lines of evidence of the same type, the confidence would be unchanged. However, the assertion is also supported by an evidence of type ‘developmental similarity evidence’  consortium in the field of protein interactions , with thSuch a summary annotation, taking all evidence lines into account, could be automatically produced, with rules specifying how individual confidence values should contribute to the overall confidence (see section Suggestions of implementation for an example). It would also be possible to decide to not formally assess confidence for each individual evidence, but to manually annotate only the overall assertion, which would represent an advantage for curation teams with limited resources.Finally, the information provided by multiple evidence lines needs to be reconciled. When all evidence lines supporting an assertion are congruent, the assertion should be trusted with higher confidence than when some evidence lines are conflicting. Congruent assertions could be, for instance, annotations between the same gene product and GO term, based on different evidence sources. Conflicting assertions are annotations yielding opposite interpretation, for instance, a positive annotation between a gene product and a GO term, and a negative annotation between the same gene product and GO term, using the ‘NOT’ qualifier; or incompatible GO terms assigned to the same gene product, for instance the GO term ‘DNA replication’ from the ‘Biological Process’ branch, and the GO term ‘cytoplasm’ from the ‘Cellular Component’ branch; in any case, it is the resources implementing the CIO that should define what a conflict is, for the type of data that they annotate.When evidence lines are contradictory, we can distinguish ‘weak’ contradictions , from ‘strong’ contradictions .For these reasons, the ‘same type’ and ‘multiple types’ terms each have two subclasses, e.g. when evidence lines are of the same type: ‘confidence statement from congruent evidence lines of same type’ and ‘confidence statement from conflicting evidence lines of same type’. Such a ‘conflicting evidence lines’ term also has two subclasses, e.g.: ‘confidence statement from strongly conflicting evidence lines of same type’ and ‘confidence statement from weakly conflicting evidence lines of same type’ .Figure The term ‘confidence statement from congruent evidence lines of same type’ has three subclasses, based on the overall confidence obtained from the associated evidence lines . For ins’; ‘confidence statement from weakly conflicting evidence lines of same type, overall confidence “medium”’; ‘confidence statement from weakly conflicting evidence lines of same type, overall confidence “low”’.The term ‘confidence statement from weakly conflicting evidence lines of same type’ has similar subclasses. Indeed, when using weakly contradicting evidence lines, the implication is that the overall assertion is believed to be true. It is then possible to provide an overall confidence from the set of available evidence lines. The following three subclasses were thus created: ‘confidence statement from weakly conflicting evidence lines of same type, overall confidence “high”Such subclasses were not deemed relevant for the term ‘confidence statement from strongly conflicting evidence lines of same type’. In that case, the supported assertion is believed to need further validation, as evidence lines yield different conclusions of similar confidence. Assertions tagged with this term should be targeted in priority when integrating new findings, as they are more likely to evolve, following the development of new methods or technologies. The individual single-evidence confidence annotations associated to strongly conflicting evidence lines would be in that case useful to define directions for further analyses.Note that the rationales are the same when using terms from the branch ‘confidence statement from multiple evidence lines of multiple types’. Also, assertions annotated with the term ‘rejected’ should not be considered when aggregating multiple evidence lines.http://geneontology.org/page/go-annotation-conventions). Please note that this is an illustrative example, and that the implementation of such conventions is pending further discussion within the GO consortium. We provide this example only to present how the CIO could be used. For an example of an actual implementation, see next section.An example of use would be to extend the GO annotation conventions , the product of the Candidaglabrata gene ERG9 (UniProtKB ID Q9HGZ6) is associated to the GO term GO:0051996 ‘squalene synthase activity’, with two annotations:Arabidopsisthaliana, to 71.3% identity in Saccharomyces cerevisiae, and identified three conserved predicted kinase motifs. This protein sequence is thus highly conserved on a significant length, even in distant species, with relevant motifs found; this annotation could be assigned the term CIO:0000003 ‘high confidence from single evidence’. Of note, the GO consortium does not define numerical cutoffs for the extent or percentage identity of sequence similarity . These cutoffs should vary, depending on, e.g. the organism studied, or the protein function captured. Again, each annotation team should define and communicate clear parameters to assign confidence.One of them is supported by the evidence term ECO:0000200 ‘sequence alignment evidence’ (GO evidence code ISA). It was produced based on a publication  providinC. glabrata strains where ERG9 was under the control of the tetracycline-regulatable promoter. The correct replacement of the endogenous ERG9 promoter was verified by PCR, and the ERG9-controllable strains exhibited a severe growth defect in medium with DOX; the growth defect caused by DOX was suppressed by the addition of serum containing exogenous cholesterol. Overall, as this experiment was carefully designed, and provided a clear result, this annotation could also be assigned the term CIO:0000003 ‘high confidence from single evidence’.The other annotation is supported by the evidence term ECO:0000015 ‘mutant phenotype evidence’ (GO evidence code IMP), and was produced on the basis of the same publication. The authors generated For instance, in the GO annotation file gene_association.goa_uniprot_noiea . The summary line and the summary confidence term could also be automatically produced from the set of individual assertions, as it is already the case in the Bgee project (see next section).To retrieve the evidence terms and references used to produce the additional summary line, either the corresponding single-evidence annotation lines (notably with the same gene product identifier and GO identifier) could be used if provided, or the multiple evidence terms and references could be integrated in the summary line as a list . Such negative assertions can be in conflict with other, positive, assertions, a conflict that could be captured in the summary line, with the use of terms from the branches ‘confidence statement from conflicting evidence lines of same type’ and ‘confidence statement from conflicting evidence lines of multiple types’. This would allow a simple and clear overview of the status of a particular assertion.In the GO annotation file format, it is also possible to provide negative assertions, using the NOT qualifier . The format of this annotation file is inspired from the GO annotation file format (see http://www.geneontology.org/GO.format.gaf-2_0.shtml), and the procedure to provide supporting information is inspired from the guide to GO evidence codes (see http://www.geneontology.org/GO.evidence.shtml); additionally, the procedure follows the guidelines described here.An example of use can be found in the annotation of homology between anatomical entities, provided by the Bgee team .For example, to annotate the homology of the autopod among volution . These hAn overall assertion, taking into account each line of evidence, is then automatically produced, summarizing whether the evidence lines are conflicting or congruent, and using the ECO to decide whether they are of a same or different types . As the Finally, the terms from the CIO that are the most informative and likely to be used are described in https://github.com/BgeeDB/confidence-information-ontology), as well as a practical solution for those who wish to rapidly implement it. Once the principle and design of the CIO are approved by the community, the formalization of this draft ontology could then be improved, by properly defining the semantics of the terms created, using, e.g. the Ontology for Biomedical Investigations (https://code.google.com/p/information-artifact-ontology/).The aim of this work is to show how confidence in annotations can be provided in a standard way; it is not to impose one specific practice in assessing confidence. One purpose of the draft CIO described in this manuscript is to invite feedback and comments from the community. Whatever solution is eventually adopted, the problem of assessing confidence in annotations must be addressed. We hope that this project will provide a home for discussing this major issue . Summary confidence annotations could then be automatically generated and assigned a confidence term from the ‘confidence statement from multiple evidence lines’ branch, as it is the case, e.g. in the Bgee homology annotations, as long as individual assertions are provided with confidence information and ECO terms. Alternatively, annotation teams with limited resources could choose to provide annotations only at the global summary level. However, the latter solution has the disadvantage of masking the evaluation of confidence at the level of each evidence, which limits the transparency of annotations.Even when it is impossible to provide confidence in each statement, whether because of lack of manpower or because of methodological limitations (e.g. in case of many electronic annotation methods), it is still very relevant to record whether one evidence line or several supported an assertion, whether they were of the same type or not, and whether they were contradictory or consistent. Such an approach can also be used to integrate the CIO with legacy annotations, by imposing an arbitrary confidence statement to all single evidence lines (e.g. medium confidence from single evidence), and then automatically generating terms from the multiple evidence lines branch, thus providing confidence information at least according to the multiplicity and consistency of evidence lines.Assessing the quality of the data being captured is one of the most difficult, yet essential, tasks of biocurators. Until now, this has not been done systematically, or even explicitly. There are several reasons for this: first, all papers go through a process of peer-review, and published data is usually assumed to be trustworthy. Second, due to the scale of the task, the resources available are not sufficient to capture the data from all published papers; a selection must be done as to which papers provide the most relevant information for the users of the resource being developed. As curated databases’ usage increases, biocurators have an editorial role that effectively filters published articles into biological databases. A careful selection of the data is thus essential.This opens the question of defining what makes a ‘high confidence’ evidence. Many biocurators are accustomed to estimating the confidence in evidence sources that they use, yet it can be difficult to transform such subjective estimates into standardized levels. This issue is akin to inter-curator agreement for GO annotations , that deIndeed, biocuration is a translation problem: the language of biologists must be translated into a structured vocabulary suitable for computational analyses. Ideally, it would be done without losing any of the original meaning, but that is hardly possible. An important aspect which is often missing from annotations is their biological context. For example, a protein may be found in the nucleus in one article by some immunocytochemistry approach, but may be known to have a function that is more consistent with a mitochondrial localization. Ideally at some point, one should be able to integrate all information and try to reconstruct the biological meaning of the annotations. Having a confidence in the different annotations that describe a proteins’ role will certainly help to resolve some of the apparent discrepancies in the annotations (and in the literature).One important feature we are proposing is to systematically provide a summary annotation when several evidence sources are available. We believe that the use of a summary annotation would be of great benefit, by allowing to have a clear overview of an assertion, taking into account all evidence lines. This can often be difficult when many sources are available, especially when they are contradictory.Another advantage of the guidelines proposed here is the ability of maintaining erroneous assertions, for informing users about retracted results, while being able to discard them to produce summary interpretations. Indeed, while resources providing a global overview of annotations about an entry, such as UniProtKB/Swiss-Prot, can remove erroneous information, and provide comments to warn about incorrect information, this is hardly possible for resources presenting data based on individual assertions. For instance, for GO annotations, while the presence of the NOT qualifier allows to track conflicting information, it is not sufficient since information from dubious publications remains. Moreover, when an annotation is removed, e.g. following a paper retraction, no trace of this annotation is maintained. A user coming across the original article, unaware of the retraction, might conclude that the publication has just not yet been annotated.et al. , the reevaluation of an incorrect statement could be easily propagated to the summary annotation.An example of this issue was described by Poux et al. , who shoAn aspect that is not addressed by the guidelines provided here is the different levels at which confidence in assertions can be estimated: at the level of the experimental procedure; at the level of the author interpretation, as authors might have selected results not allowing an unbiased interpretation; and at the level of the annotator interpretation. For now, the basic terms from the ‘confidence statement from single evidence’ branch should be used to take into account these different layers all together. The CIO could be expanded if that turned out to be a need of the community. Possible solutions could be to capture the confidence at these different levels independently, or to modify the branch ‘confidence statement from single evidence’ for this purpose. We hope that the current work will promote discussions towards this aim, possibly through the associated tracker.http://www.w3.org/TR/prov-overview/). However, in the current state, documenting the full provenance of annotations might be overwhelming for most curation teams. The CIO is designed to be easily used in daily curation work, and to be as close as possible to the rating systems already adopted by several resources. Capturing provenance of an annotation in accordance with W3C and other standards (e.g. linked data frameworks) should be a long term goal; our current work represents a first step towards such structured capture.This issue is related to the definition of the provenance of an annotation. Data provenance aims at documenting origin of data, but also annotation steps or task workflow  while it is difficult to standardize parameters to define confidence in annotations across resources, it is possible to use a common ontology language to provide this information; (ii) in the same way that the GO guidelines recommend to provide annotations at the level of each individual evidence, a confidence statement might also be assigned to each single evidence, using a basic rating system; (iii) when several evidence lines are available relative to an assertion, it is desirable to provide a global summary assertion, taking all evidence into account.We created the CIO in order to illustrate these principles. We hope it might be a trigger for the biocuration community to address this need for standardizing confidence information. Whether this ontology undergoes major changes in the near future, following feedback from the community, or whether it is used ‘as is’ by several resources, we hope that annotation confidence will be increasingly available in biocuration efforts.Supplementary data are available at Database Online.


The level of RCT support for common orthopaedic procedures compares unfavourably with other fields of medicine.We investigated the proportion of orthopaedic procedures supported by evidence from randomised controlled trials comparing operative procedures to a non-operative alternative. Orthopaedic procedures conducted in 2009, 2010 and 2011 across three metropolitan teaching hospitals were identified, grouped and ranked according to frequency. Searches of the Cochrane Central Register of Controlled Trials (CENTRAL), the Cochrane Database of Systematic Reviews (CDSR) and the Database of Abstracts of Reviews of Effects (DARE) were performed to identify RCTs evaluating the most commonly performed orthopaedic procedures. Included studies were categorised as “supportive” or “not supportive” of operative treatment. A risk of bias analysis was conducted for included studies using the Cochrane Collaboration's Risk of Bias tool. A total of 9,392 orthopaedic procedures were performed across the index period. 94.6% (8886 procedures) of the total volume, representing the 32 most common operative procedure categories, were used for this analysis. Of the 83 included RCTs, 22.9% (19/83) were classified as supportive of operative intervention. 36.9% (3279/8886) of the total volume of procedures performed were supported by at least one RCT showing surgery to be superior to a non-operative alternative. 19.6% (1743/8886) of the total volume of procedures performed were supported by at least one  Operative procedures are often difficult to standardise Almost two decades ago, surgical research was ridiculed as a “comic opera” as most peer reviewed publications were found to be case series or professional opinion any evidence from randomised trials (when compared to non-operative interventions) is relevant when deciding whether resources are being used appropriately.While empirical research has been conducted on the quantity and quality of orthopaedic and surgical randomised trials types was supported by RCT evidence in favour of the operative approach over a non-operative approach.Our aims were: i) to determine the proportion of the total number of commonly performed orthopaedic procedures that are supported by RCT evidence (indicating operative treatment may be superior to non-operative treatment), and ii) to investigate the risk of bias in RCTs that have compared commonly performed orthopaedic procedures to non-operative alternatives. A secondary aim was to establish what proportion of procedure We reviewed the RCT evidence for commonly performed orthopaedic procedures in three major metropolitan teaching hospitals in southwest Sydney, Australia. The hospitals were chosen as they are from a single large health district, but have different roles: one general hospital (mixed minor elective and trauma), one hospital with a large elective orthopaedic caseload, and one with a large trauma orthopaedic workload. Data pertaining to orthopaedic procedures conducted in 2009, 2010 and 2011 were extracted from an electronic clinical database recording details of all procedures undertaken in the operating theatres of these hospitals. All data pertaining to patient records and/or information was de-identified prior to extraction and analysis. Operative procedures - defined as any procedure conducted in an operating theatre by an orthopaedic team that involved penetration of the skin - were identified, grouped under general headings and ranked according to frequency . ProceduA search of the Cochrane Central Register of Controlled Trials (CENTRAL), the Cochrane Database of Systematic Reviews (CDSR) and the Database of Abstracts of Reviews of Effects (DARE) was performed to identify RCTs examining each operative procedure. CENTRAL, CDSR and DARE were chosen as these databases index the vast majority of published (and many unpublished) RCTs any RCT evidence may be found for commonly performed orthopaedic procedures, we crudely categorized each RCT as being either 'supportive’ or 'not supportive’ of operative treatment using the following criteria:Included RCTs were independently assessed in their entirety by two reviewers, and the sample size and population, the specific interventions being compared, primary outcome, primary outcome findings, secondary outcomes, findings of secondary outcomes and the authors' conclusions were identified. Given that our objective was to determine whether For RCTs where a primary outcome was stated, and the findings demonstrated a statistically significant result in favour of operative treatment for the stated primary outcome – regardless of the results of any stated secondary outcome(s) – the study was categorised as being 'supportive of operative treatment'.For RCTs where a primary outcome was stated, and the findings demonstrated a statistically significant result in favour of the non-operative treatment or where no significant difference was shown for the stated primary outcome, the study was categorised as being 'not supportive of operative treatment'.For RCTs where a primary outcome was not stated but several outcomes were measured, and where statistically significant results were demonstrated across all measured outcomes in favour of operative treatment, the study was categorized as “supportive of operative treatment”.For RCTs where a primary outcome was not stated but several outcomes were measured, and where the measured outcomes did not consistently demonstrate statistically significant results in favour of operative treatment, the study was categorised as 'not supportive of operative treatment'".Discrepancies were resolved by discussion and arbitrated by a third author if necessary.A risk of bias analysis was conducted using the Cochrane Collaboration's Risk of Bias tool Results were analysed and presented as:The proportion of the total volume of procedures that were assessed by RCTs;The proportion of the total volume of procedures that were:supported by at least one RCTsupported by at least one low risk of bias RCTsupported by at least one RCT with sample size greater than or equal to the median sample size of included RCTsPrimary analyses were conducted using total procedure volume, as this was a more accurate reflection of surgical volume. Secondary analyses were conducted using procedure type, rather than total procedure volume; this analysis would highlight which procedure types were supported by high-level evidence.A total volume of 9,392 orthopaedic procedures were performed across the three hospitals in the years 2009, 2010 and 2011 comprising 91 operative procedure categories. 94.6%  of the total volume was represented by 32 operative procedure categories. This group was used as the basis for this study. The 10 most commonly performed procedures comprised 80.6% of the 8,886 procedures included in the study . 32 diffOf the 83 RCTs included, 22.9% (19/83) were classified as supportive of operative intervention. The median sample size of the included RCTs was 72. A list of included and excluded studies is found in 52.6%  of the total volume of procedures were subjected to an RCT comparing operative to non-operative treatment. 36.9% (3279/8886) of total volume of procedures had at least one RCT supporting operative intervention. When analysed according to risk of bias assessments, 19.6% (1743/8886) of total volume of procedures had at least one low risk of bias RCT supporting operative intervention. When analysed against sample size, 47.5% (4223/8886) of total volume of procedures had at least one RCT with sample size greater than or equal to 72 supporting operative intervention .Of the 32 procedure types comprising 94.6% of the total volume of procedures, 16 (50%) had RCTs comparing the procedure to non-operative treatment. 28.1% (13/32) procedure types had at least one RCT supporting operative intervention. When analysed according to risk of bias, 9.3% (3/32) procedure types had at least one low risk of bias RCT supporting the operative intervention. 34.3% (11/32) procedure types had at least one RCT with sample size greater or equal to 72 supporting operative intervention .12 of the 83 included RCTs were assessed as having low risk of bias across the two domains: allocation concealment and blinding of outcome assessment. 26 RCTs were assessed as having unclear risk of bias and 45 as having high risk of bias across the two domains. Four of the 12 RCTs assessed as having low risk of bias  were supportive of operative treatment.A risk of bias summary for each domain is depicted in We reviewed the available RCT evidence in relation to commonly performed orthopaedic procedures in the years 2009, 2010 and 2011 to establish what proportion of these procedures was supported by RCT evidence. We found that 52.6% of the total volume of orthopaedic procedures performed had been compared to non-operative treatments in at least one RCT, and that 36.9% of the total volume of procedures performed was supported by at least one RCT in favour of the operative treatment. Where RCT evidence was available, the quality of that evidence was largely deemed sub-optimal due to inherent biases in the trial methodology.While the proportion of RCTs in surgical publications has increased over recent decades, our findings suggest that the proportion of surgical interventions supported by quality RCT-level evidence remains low. In general, this is in keeping with the findings of earlier studies.Howes et al conducted a prospective study looking at the evidence-base for surgical decisions made on 100 patients admitted to a general surgical ward in a tertiary teaching hospital. The literature concerning the efficacy of each treatment was reviewed and it was found that while 95% of the surgical interventions studied were evidence-based, only 24% were based on RCT-level evidence, defined as at least one RCT in favour of the surgical approach For surgical procedures performed commonly, it is not unreasonable to expect decision-making to be based on a high level of scientific evidence. Our findings, however, show that despite 52.6% of the total procedure volume being subjected to at least one RCT, only 19.6% of performed procedures had at least one low risk of bias RCT supporting the operative treatment over non-operative alternatives. Similarly, only a minority of the procedure types were supported by higher quality RCT evidence in favour of the surgical procedure. These comparatively low figures are in part explained by our methodology. In contrast to the three studies highlighted above, our study analysed the results according to the quality of RCT evidence. If the figures obtained in the current study were interpreted using the same definition as that of the previous studies, this would result in 36.9% of total volume of procedures having support from at least one RCT. While this figure may seem comparatively favourable, we consider our assessment to be a more accurate reflection of available evidence.idual patient Despite the methodological differences between studies, the common finding is that there is a paucity of high-level evidence to support a large volume of surgical procedures. This disconnect between evidence and practice may be explained by several factors, including clinician bias. Katz, in his ethnography on the culture of surgeons, contended that “surgeons have been resistant to accepting new scientific findings and applying them to their practice" The 83 included RCTs had variable levels of bias. The risk of bias summary for each surgical procedure consistently showed poor scores in the performance and outcome detection categories of bias due to difficulties with blinding the surgical intervention. Furthermore, a quarter of studies reported poorly on their randomisation strategy while only a handful performed sample size calculations for power. The only category that was consistently adequate across the included RCTs was follow-up. Of the two included domains, RCTs performed most poorly in blinding of outcome assessment. Just 17 studies specifically attempted to address this potential source of detection bias in their methodology and was the largest avoidable contributor to the overall high risk of bias across the included studies.When RCTs were analysed against sample size, less than half the total volume of procedures were supported by RCTs with a sample size greater than the median of 72. Though inadequate sample size does not directly influence study bias, underpowered studies are more likely to have a greater rates of type II errors in which studies fail to detect statistically significant treatment effects. An investigation by Lochner et al into the rates of type II errors of randomised trials involving orthopaedic fracture care showed higher than accepted levels of type II errors and, low mean level of study power We acknowledge the limitations of our study. Not subdividing the RCT evidence by indication for the specific procedures in our analysis  is a potential limitation of this study as the RCT evidence support for a procedure may be restricted to specific indications. Given the primary aim of the study was to assess potential differences between available RCT evidence and current clinical practice, an analysis subdividing each of the included procedure groups for indication would have contributed little value to the primary outcome. As our criteria allowed inclusion of supportive RCTs for any indication, however, it is likely that there was a potential overestimation of RCT support for any procedure group. Our study investigated RCT-level evidence exclusively as this study design provides the least biased evidence for the efficacy and safety of an intervention though its ability to deal with confounding While improvements have occurred in the quantity and to a lesser extent, the quality of surgical research over the past several decades, this study confirms that, consistent with other surgical specialities, the majority of orthopaedic surgical interventions are not based on RCT evidence. The findings of this study are reflections of the current disconnect between trial evidence and orthopaedic surgical practice and its comparison to other specialities. These findings support the need for better quality RCTs to evaluate the indications for orthopaedic procedures and stakeholder discussions about the lack of support for many procedures currently being performed.Appendix S1List of procedures performed ordered according to frequency.(DOCX)Click here for additional data file.Appendix S2Syntax of search strategies for each operative procedure category.(DOCX)Click here for additional data file.Appendix S3List of included and excluded RCTs after full-text assessment.(DOCX)Click here for additional data file.Appendix S4Characteristics of included RCTs.(DOCX)Click here for additional data file.Appendix S5Risk of Bias assessment of 83 included RCTs using the Cochrane Collaboration's Risk of Bias tool.(PDF)Click here for additional data file.Checklist S1(DOC)Click here for additional data file.


Recognizing the need for evidence to inform US Government and governments of the low- and middle-income countries on efficient, effective maternal health policies, strategies, and programmes, the US Government convened the Evidence Summit on Enhancing Provision and Use of Maternal Health Services through Financial Incentives in April 2012 in Washington, DC, USA. This paper summarizes the background and methods for the acquisition and evaluation of the evidence used for achieving the goals of the Summit. The goal of the Summit was to obtain multidisciplinary expert review of literature to inform both US Government and governments of the low- and middle-income countries on evidence-informed practice, policies, and strategies for financial incentives. Several steps were undertaken to define the tasks for the Summit and identify the appropriate evidence for review. The process began by identifying focal questions intended to inform governments of the low-and middle-income countries and the US Government about the efficacy of supply- and demand-side financial incentives for enhanced provision and use of quality maternal health services. Experts were selected representing the research and programme communities, academia, relevant non-governmental organizations, and government agencies and were assembled into Evidence Review Teams. This was followed by a systematic process to gather relevant peer-reviewed literature that would inform the focal questions. Members of the Evidence Review Teams were invited to add relevant papers not identified in the initial literature review to complete the bibliography. The Evidence Review Teams were asked to comply with a specific evaluation framework for recommendations on practice and policy based on both expert opinion and the quality of the data. Details of the search processes and methods used for screening and quality reviews are described. Over their lifetime, women are about 100 times more likely to die as a result of pregnancy in sub-Saharan Africa than in developed regions of the world  is leading a series of evidence summits focused on important development challenges. The aim of these summits is to provide evidence-based expert recommendations on how to achieve some of the world's most difficult development goals, for example, caring for children living outside families  and suppTo address the problem of financial barriers to maternal healthcare utilization, the US Government convened the Evidence Summit on Enhancing Provision and Use of Maternal Health Services through Financial Incentives on 24-25 April 2012 in Washington, DC. The Summit brought together leading researchers, development experts, and those involved in implementing programmes in the field to assess the evidence that will ultimately inform policies, strategies, and programmes relevant to financial incentives and maternal health services in LMICs and, in so doing, simultaneously identify gaps in evidence that could shape the future research agenda. This Evidence Summit reflects USAID's commitment to evidence-based, innovative, efficacious, effective and sustainable development efforts of the US Government in partnership with other governments. The rapid application of knowledge and scale-up of novel discoveries and innovations to populations needing them most requires a continuum of learning from basic to operational research and a broad coalition of expertise and contributors across the US Government, academia, and policy- and practice-related leaderships from developing countries. The Eunice Kennedy Shriver National Institute of Child Health and Human Development at the National Institutes of Health joined USAID in the organization and support of this Evidence Summit. This paper describes the Evidence Summit process from its inception to the post-summit activities. Other papers in this Supplement of the Journal present the findings and recommendations.http://www.ghi.gov/), which calls for interagency collaboration, innovation, and doing more of what works as guiding principles for US Government's assistance in global health. Like other evidence summits of the US Government, this Evidence Summit was not a single event, but rather a year-long process that led up to the April 2012 Summit and continues with the implementation of recommendations developed during the Summit process (http://www.usaid.gov/sites/default/files/documents/1864/mh_summit_interagency_planning_group.pdf). The formation of the Core Group was followed by a scoping exercise consisting of an external consultation, which included experts in maternal health, economics, and implementation to review the topic and identify key questions. Leveraging this external advice, the Core Group created focal questions to guide the evidence review process, the selection and appointment of external and internal experts to Evidence Review Teams (ERTs), a systematic literature search for documents relevant to the focal questions, and the screening and evaluation of these documents and the identification of additional relevant material by members of the ERTs. A pre-summit meeting was held to organize and motivate the ERTs, which subsequently prepared Evidence Synthesis Papers and drafted recommendations for presentation at the Summit. The final step of the Evidence Summit process—the revision of materials based on feedback at the Summit and the implementation of an Evidence to Action Plan to act upon recommendations—is still underway. The following sections provide more details on these steps in the Evidence Summit process, focusing primarily on the gathering of evidence and review phases.The initial planning for the Summit originated in the Bureau for Global Health at USAID and was tied to President Obama's Global Health Initiative  and/or improved implementation of interventions;There is a strong demand from the field for evidence-based solutions to pressing policy/programme challenges or questions represented by the topic;The topic informs an important global health issue;The evidence on the topic can be pooled, synthesized, shared, and discussed at reasonable cost; andEvidence-based guidance is not available on the topic or needs to be updated.The underutilization of maternal health services was identified as a general topic for an evidence summit. Additional information on the scoping exercise and rationale for the topic can be found in the introductory paper to this Supplement . RecogniThe Core Group met over a period of several months to draft a concept paper describing the background for the Summit, the goals and anticipated outcomes, and a process for accomplishing the summit objectives. On 30 August 2011, the Core Group also convened an external consultation with experts from the fields of maternal health and performance-based financing to review the concept paper and identify key questions for the Evidence Summit. Participants in the consultation included experts from the Population Council, Agency for Health Research and Quality, Broad Branch Associates, Center for Global Development, Abt Associates/Health Systems 20/20, and the Bill & Melinda Gates Institute for Population and Reproductive Health/Johns Hopkins Bloomberg School of Public Health. During the external consultation, the experts advised that supply- and demand-side interventions and outcomes were interdependent and should not be separated in the Evidence Summit. This advice resulted in additional refinements of the concept paper, including a decision to focus on financial incentives that would increase both demand for and supply of maternal healthcare services.Following extended internal discussions and consideration of recommendations provided during the external consultation, the Core Group selected two key focal questions to be addressed during the summit process. These focal questions emerged from a set of beliefs about the roles of financial incentives, the contexts that led to those beliefs, and the formation of explicit development hypotheses about what the evidence review might find. These are as follows:Focal Question 1: What financial incentives, if any, are linked positively or negatively to maternal and neonatal health outcomes, the provision or utilization of maternal health services, or to care-seeking behaviour by women?Belief: Financial incentives can influence users’ and providers’ behaviours, including the utilization and provision of services and can potentially alter maternal and neonatal health outcomes positively and, in some cases, negatively. Some incentives will be more influential than others and interaction of incentives in various combinations will produce different results.Context: In recent years, financial incentives in the form of vouchers, waivers, conditional cash transfers, variations of pay-for-performance, and so forth, have galvanized tremendous interest in the public health community. Considerable documentation of financial incentives for health, in general, has been compiled. To date, there is less information relating to the effect of financial incentives on maternal health behaviours, including the use of services, providers’ behaviours, and maternal and neonatal health outcomes. Many governments and donors are supporting, with substantial investments, implementation of financial incentives for maternal and newborn health but this is based on limited evidence. Because of the significant potential to affect the use and provision of services, there is a need to identify, synthesize, and analyze the available evidence to determine positive and negative effects for maternal and newborn health.Development hypothesis: A review of evidence of financial incentives and their effects on maternal and neonatal health behaviours, service delivery, and outcomes will increase understanding of available interventions and lead to more effective and efficient policies, programmes, and strategies.Focal Question 2: What are the contextual factors that impact the effectiveness of the financial incentives?Belief: Numerous contextual factors, including household income and wealth, providers’ compensation, geography, health workers’ quality and access, availability of transport, capacity of services to accommodate more clients effectively, management of the financial incentive programme, quality of the health management information system, the political situation, and so forth, are critical to implementation of the incentive programmes for maternal health and for their results.Context: While there is potential for significant positive changes for health behaviours and health outcomes, those with experience in implementing and evaluating financial incentive programmes to date advise that, without understanding the context in which the financial incentives are applied, it is difficult to generalize from results in any one setting. For example, a programme that quickly increases service utilization but that cannot provide quality services could result in fewer clients accessing services over time and/or yield negative health outcomes. Furthermore, supervision and support, social norms, household support (from husband/partner/mother/mother-in-law), community wealth , and infrastructure are some of the other contextual issues that can influence the results of financial incentive programmes.Development hypothesis: A review of various levels of evidence about the wide range of contextual variants in financial incentive programmes will aid in understanding the nuances of designing and implementing policies and programmes for effective results in different settings.To facilitate the organization of the Evidence Summit and the literature search process, the Core Group subdivided the evidence by the type of financial incentive and development measure assessed in the study. As shown in Finally, the careful examination of the context in which the research was conducted was critical to answering Focal Question 2. There are several dimensions along which context might be considered. Geographical region, nation, urban versus rural, or culture may be an important contextual factor influencing the outcomes of different financial incentives. It may be that certain characteristics of the patient, such as socioeconomic status or membership in a stigmatized group, could modify the effectiveness of incentives. The ERTs were asked to identify the contextual elements that should be examined to answer Focal Question 2.Central to the Evidence Summit process was the selection and organization of experts into three ERTs to assess the evidence on financial incentives and maternal health services and make recommendations on policies, strategies, and programmes. Experts were nominated by Core Group members or, in some cases, by outside experts who were initially contacted about participating but who nominated someone else in their place. Many members of the Core Group also served on ERTs. The challenge in selecting ERT members was to achieve balance along several dimensions, such as area and level of expertise, programmatic and research experience, and affiliation. ERT members had expertise in maternal health, health economics, health systems, development, and other related topics, and many had experience in implementing programmes or conducting research involving the use of financial incentives. They represented a mix of senior and mid-level managers, practitioners, and researchers. ERT members were affiliated with the following institutions:Non-governmental organizations: Abt Associates; Broad Branch Associates; Center for Global Development; Futures Institute; JHPIEGO; John Snow, Inc.; Population Council; Results for Development; RTI International; Save the Children; and University Research Co.Academic institutions: Johns Hopkins Bloomberg School of Public Health, Harvard School of Public Health, London School of Hygiene & Tropical Medicine, University of North Carolina, Gillings School of Global Public Health, and Virginia Commonwealth UniversityUS Government agencies: Centers for Disease Control and Prevention, National Institutes of Health, and USAIDPrivate foundation: Bill & Melinda Gates FoundationBilateral and multilateral institutions: Department for International Development (United Kingdom), World Bank, and World Health Organization.Although the number of ERT members based in LMICs was relatively small, many experts had extensive experience working and residing in these settings. It was decided to bring a greater LMIC perspective into the process during the Summit itself by inviting participants from some of these countries. Finally, the Core Group selected ERT members based on their knowledge and expertise in maternal health services, financial incentives, health systems, health economics, development, and other related topics. A large majority of persons invited to become ERT members agreed to serve in this capacity, which was a significant effort commitment, given the long-term nature of the Evidence Summit process and the work required for these teams.The three ERTs were organized around three categories of financial incentives: supply-side financial incentives, conditional cash transfers, and other demand-side financial incentives (excluding cash transfers) . The ERTTwo strategies were employed to acquire evidence for the Evidence Summit process: (i) a formal literature search conducted by public health professionals, and (ii) a call for evidence issued to members of the ERTs. Although it was recognized that high-quality evidence existed in both published and unpublished literature, the Core Group opted to limit the formal literature search to articles published in peer-reviewed scholarly journals, anticipating that relevant documents from the grey literature would be submitted by experts in the field in the subsequent call for evidence. The Core Group also decided to exclude documents based on research carried out in high-income countries. This decision reflected the thinking that research in high-income countries may have limited relevance to LMICs; if relevant research from high-income countries was identified by experts in the ERTs, it could be submitted during the call for evidence process.The databases searched included ABI Inform, Cochrane Library, PubMed, SCOPUS, EconLit, Embase, SocioAbstracts, and CABI . Where pDocuments not published in EnglishDocuments published before 1 January 1990Research carried out in high-income countriesMagazine or newspaper articlesLetters-to-the-EditorObituariesCommentaries/recommendations not based on thorough literature reviewsBook reviewsJob postingsHistorical accountshttp://www.usaid.gov/node/7186. The latter goal was achieved by including papers that described research on an intervention involving the use of one or more financial incentive(s) and included measurement of at least one outcome related to the demand for or utilization of maternal/neonatal health services by women, the performance of health professionals or organizations providing maternal/neonatal health services, or health-related maternal or neonatal outcomes that resulted from changes in the behaviour of patients or providers. Maternal/neonatal health services were defined as routine antenatal visits, special programmes for pregnant women , care for pregnant women suffering from an illness, intrapartum care, and other neonatal services through the first 28 days of life. Abortion and family planning services were excluded. The screening process was completed through a review of abstracts and resulted in the retention of 65 papers. In addition to excluding documents that did not meet the screening criteria, this screening step was also used in sorting documents into the type(s) of financial incentive(s) that were studied  and into the type(s) of outcome(s) measures that were studied , effectively sorting documents into the matrix shown in Following the pre-screen and removal of duplicates, the initial search of the peer-reviewed literature yielded a total of 470 papers. KMS then undertook a manual screening of the papers with two goals: (i) to exclude documents according to the basic exclusion criteria shown before, in case any such document had not been previously eliminated via the search limits and pre-screen, and (ii) to include only those papers with interventions directly related to the Focal Questions and that contained outcome data . The scrDuring the course of the evidence reviews, the ERT that was asked to examine the literature on supply-side incentives decided to include interventions that utilized insurance, necessitating a renewed literature search for documents on this topic that otherwise met the above criteria for inclusion . The inihttp://www.usaid.gov/node/7186. In selecting papers for submission, the ERT members were encouraged to review the exclusion criteria that had been applied to the initial literature search but they were not required to adhere to those criteria to enable the experts to include papers they deemed highly relevant to the focal questions. They were also advised that purely descriptive papers had not proven useful in previous USAID-supported Evidence Summits and that priority should be given to the following types of papers:To supplement the papers identified through the literature search, members of the ERTs were invited to submit documents they felt would help address the focal questions. This took place through a formal call for evidence prior to the Evidence Summit. It utilized an online document submission process in which ERT members were asked a series of questions about the documents they were submitting. The Call for Evidence submission protocol can be found on the Evidence Summit website: Papers with primary data of high scientific quality with maternal or neonatal health outcomes, maternal healthcare-seeking behavioural outcomes, or provider behavioural outcomes that result from the application of a financial incentive.Papers summarizing interventions or evaluationsDocuments of relevance to low- and middle-income countries, even though these may describe work done in high-income countriesDocuments that had undergone peer reviewSystematic reviewsGrey literature in the form of studies, reviews, or evaluationsA total of 25 documents were formally submitted through the call for evidence. Eight duplicates were removed, resulting in 17 additional documents . During http://www.usaid.gov/node/7186 for the text of the quality review questionnaire). ERT members were assigned sets of papers to review and asked eight questions concerning the types of equivalence of the comparator group, if any; the adequacy of the study design; the fidelity of the intervention; the validity and relevance of the outcome measures; the data analysis; the generalizability of the results; and evidence for the sustainability of the intervention. Each document was assigned to two ERT members for review. Of the 82 papers, 70 underwent a quality review with 28 papers reviewed by one reviewer, 33 papers reviewed by two reviewers, and 9 papers reviewed by three reviewers. Summary quality scores between 0 and 1  were derived for use by the ERT members in the later evaluation of evidence. The summary score did not penalize documents if reviewers skipped a question.After the screening and sorting processes were completed, ERT members were asked to assess the quality of the 65 papers derived from literature search and screening process and the 17 papers obtained through the Call for Evidence. A complex quality review framework for both empirical studies and programme evaluation documents had been developed previously for the Evidence Summit on Protecting Children Outside of Family Care . The agenda included panel discussions on important issues that were identified in multiple ERTs, such as unintended consequences of financial incentives, contextual issues, and a discussion on ethical issues involved in FI studies. Participants at the Evidence Summit included members of all of the ERTs as well as many additional experts in both science and practice of enhancing the provision and use of maternal health services and in using financial incentives. Notably, an effort was made to expand the involvement of LMIC representatives as participants in the Evidence Summit. The LMIC experts were included in roundtable discussions and included representatives, including His Excellency Mam Bunheng, Minister of Health, Cambodia; Dr. Fidele Ngabo, Director of the Maternal and Child Health Unit, Ministry of Health, Rwanda; Francis-Xavier Andoh-Adjei, Deputy Director of Strategy and Corporate Affairs, National Health Insurance Authority, Ghana; and Nazrin Oriakhail, Director, Malalai Maternity Hospital, Kabul, Afghanistan.ERTs were asked to prepare two presentations for the Evidence Summit: (i) a narrative evidence synthesis which contained a review of the evidence on the focal questions that pertained to the financial mechanism they were assigned to consider and (ii) a set of recommendations. The Evidence Summit was held in Washington, DC on 24-25 April 2012, and the agenda and list of participants are available on the Summit website , insurance, conditional cash transfers, vouchers, and other demand-side interventions. These five papers form the core of the reports on the Evidence Summit that are published in this Supplement of the Journal.The commitment to evidence-informed practice, policies, and strategies to informing global health decisions is the first step in a long road to doing so, especially when demand for effective, efficient, global health services requires action now. Ideally, evidence-informed policy for global health necessitates the quality and standardized expert evaluation of the evidence for the critical challenges faced by governments of the low- and middle-income countries. These challenges, such as effective use of financial incentives to increase demand for and supply of quality maternal health services, are complex, multisectoral, and transministerial problems. Ideal literature to inform these problems would focus on the critical questions needed to inform government policies in the LMICs on sustainable, effective practices and strategies. The expertise in designing and conducting studies would parallel the multidisciplinary nature of the problem. The evidence would address relevant contextual issues so countries could know what works in the context of settings similar to theirs. Finally, studies on implementation science would follow standard guidelines for reporting similar to the consort guidelines.The Evidence Summit process was developed to acquire and evaluate the available evidence to make recommendations for global health practice, policy, research prioritization, and strategy. Despite a thorough acquisition process, the identified literature for this Evidence Summit fell far short of ideal. Collectively, the literature did not address critical issues, such as sustainability, contextual issues, specificity around quality of care measures, or details on implementation that would enable reproducibility. Further, study designs usually fell short of needed information, such as causal attribution of the financial incentives to health outcomes, longitudinal neonatal and maternal health outcomes, and continued impact of FI; sustainability assessments, such as health value, cost-effectiveness, resource requirements, etc. A common challenge involved lack of coordinated multidisciplinary teams of economists and maternal health experts in planning and conducting studies. The majority of studies neglected to see the financial incentive in the paradigm of a dynamic system and, therefore, did not measure the impact on both supply and demand side of interventions. Few studies assessed unintended consequences resulting from financial incentive schemes. Further, it is quite clear that the research agenda must be informed, in part, by the policy-makers so that the research community can address the research around the key issues faced by policy-makers. Global health issues would benefit from a standardized evidence evaluation framework to guide the development of evidence and its evaluation and reporting. As part of the evidence to action strategy, USAID developed a concept for a consortium to bring economists and maternal health experts together to address the research needs in the area of financial incentives and maternal health. The idea for the formation of consortium was discussed with some of the organizations participating in the Evidence Summit including but not limited to USAID, NORAD, World Bank, Bill & Melinda Gates Foundation, USAID, and NICHD. Most opinions of those present were that an additional consortium was not needed to address the research needs because existing maternal health consortia could do so. These concerns are addressed in another paper in this Supplement .This Evidence Summit drew lessons from the previous US Government Evidence Summit on Protecting Children Outside of Family Care , e.g. itLack of ideal or robust evidence or an ideal framework in which to evaluate evidence should not deter the global health development and research communities from reviewing evidence around critical global health challenges. Doing so creates a transparent understanding of what is known enabling better decision-making processes, more targeted relevant research agenda while leveraging the tremendous resident expertise in the development, academic and multilateral communities.This work was carried out by the United States Federal Government Employees and persons supported through contractual mechanisms by the federal government. This is an open-access article under the terms of the Creative Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided that the original authors and source are credited. Among the authors, ESH is supported by the National Institute of Allergy and Infectious Diseases and was on assignment with the Bureau for Global Health at USAID during the planning and execution of the Evidence Summit. ES is paid by Knowledge Management Services in Washington, DC. This organization received contractual funding from USAID for their work on the Evidence Summit. RR is currently paid by USAID. During the planning of the Evidence Summit, she was paid by the National Institutes of Health and was on assignment with the Bureau for Global Health at USAID. RLB is paid by Virginia Commonwealth University in Richmond, VA, USA. During the planning of the Evidence Summit, he was a Jefferson Science Fellow assigned to USAID and received supplemental support from the National Academy of Sciences. He currently receives part of his salary through an Intergovernmental Personnel Act arrangement with USAID. Under the direction of USAID staff, Knowledge Management Services designed the search and screening process, and USAID facilitated the publication of the manuscripts resulting from the Evidence Summit process. No funding bodies had any role in the data analysis or conclusions drawn from the literature reviews. The views and opinions expressed in this paper are those of the authors and not necessarily of the National Institutes of Health, the Department of Health and Human Services, or the Government of the United States.


Currently, there are gaps in the allergic rhinitis (AR) guidelines (and the evidence upon which they are based) that impede their function. Firstly, there is a scarcity of high quality evidence supporting AR treatment decisions . Randomi*Dymista


GraphoGame (GG) is originally a technology-based intervention method for supporting children with reading difficulties. It is now known that children who face problems in reading acquisition have difficulties in learning to differentiate and manipulate speech sounds and consequently, in connecting these sounds to corresponding letters. GG was developed to provide intensive training in matching speech sounds and larger units of speech to their written counterparts. GG has been shown to benefit children with reading difficulties and the game is now available for all Finnish school children for literacy support. Presently millions of children in Africa fail to learn to read despite years of primary school education. As many African languages have transparent writing systems similar in structure to Finnish, it was hypothesized that GG-based training of letter-sound correspondences could also be effective in supporting children’s learning in African countries. In this article we will describe how GG has been developed from a Finnish dyslexia prevention game to an intervention method that can be used not only to improve children’s reading performance but also to raise teachers’ and parents’ awareness of the development of reading skill and effective reading instruction methods. We will also provide an overview of the GG activities in Zambia, Kenya, Tanzania, and Namibia, and the potential to promote education for all with a combination of scientific research and mobile learning. The widespread endurance of illiteracy across the world deprives millions of citizens of economic and political opportunities to secure their basic human rights. It is estimated that almost one in six people over the age of 15 cannot read and write, and most of them live in low-income countries where only a minority of children have access to quality basic schooling . In ordeThe national education system in Finland has received international recognition. Literacy has always been highly valued in Finnish culture. Anecdotally, almost from the time of establishment in the 16th century of the Finnish orthography, a good incentive for learning to read was the fact that people were not permitted to marry without first passing a public literacy examination administered by the local priest. In present day Finland, the majority of school children learn the basic reading skill within the first months of school entry. Failure to learn to read at the same time as classmates can be a personal tragedy that affects self-esteem and leads to long-term negative consequences, such as depression, behavior problems, and other mental health issues . Since iJyväskylä Longitudinal study of Dyslexia project taught us that the key to achieving basic literacy skill in the alphabetic script of Finnish language is to learn how letters (graphemes) and sounds correspond to each other in the writing system e.g., , 2009. Cd = 0.19) in supporting the instruction of beginning reading skills. A more recent review , but not necessarily in terms of motivation . Because of lack of systematic research concerning the use of serious games in beginning/remedial reading instruction, conclusions as to the overall effectiveness of games in this area of learning can be drawn.Earlier research has shown that technology has the potential to prevent and remediate reading difficulties. A research synthesis by t review  with mort review  suggest 1 is a web-based scientifically validated, non-profit internet service for reading acquisition that is already used widely in Finland on multiple platforms. GG teaches the learner basic literacy skills and at the same time, collects data on the learning process for research and education purposes  at the end of kindergarten (6 years of age) and at children who are in Grades 1 or 2 at school and require support with basic decoding skills or reading fluency. Since the launch of the LukiMat service in 2008, GG has acquired approximately 270,000 registered players in Finland, and every day, around 6000 children log on to play. As each age cohort has about 60,000 children in Finland, it seems that the game is in great demand, even in the country with one of the top literacy scores in the OECD countries (The Finnish versions of GG (Ekapeli) are provided via the LukiMat serviceountries .A major obstacle to increasing our understanding of learning difficulties is that the vast majority of information on the topic comes from Western societies and describes learning as it happens within high-income societies. What is currently known is that from the cognitive and neurobiological perspectives, the most important predictive factors emanate from the development of spoken language skills from very early on. The JLD results show that such individual factors based on genetic starting points reveal the first observable indications during the first days of life . Such inNonetheless, on a global level, the role of biological factors is insufficiently studied. Many environmental factors environmental factors, such as malnutrition and early childhood illness may have an effect on learning abilities. The poor quality of education itself is a major risk factor. Inadequate teacher training and negative attitudes in the community play a role, and most of all, the language of instruction may be an underestimated cause of poor learning outcomes. While the use of indigenous languages as media of instruction has been defended on the basis of human rights and preservation of local cultures, the impact of language goes much deeper. The relationship between the spoken language and the writing system has a large effect on the literacy learning rate. The amount of written symbols in the writing system and the rules of connecting them to the speech sounds vary across languages. The easiness of learning to read is related to the amount of letter-sound pairs learner needs to memorize. One cannot learn to connect spoken to written language without that the learner having at least implicit awareness of the speech sounds of the language. In certain languages, such as English, sounding out some items  is difficult since these words violate spelling to sound rules and thus pronouncing them correctly requires access to the meaning and/or derivation of the word. In contrast languages such as Finnish can be fully transparent: letter to sound associations always produce the correct pronunciation. These differences between languages have profound effects on learning e.g., .Understanding how different languages affect the rate of reading acquisition has only recently come to the attention of researchers. A cross-linguistic comparison study by 14 European countries showed that it takes significantly longer to learn the foundations of reading in English in comparison to languages with more transparent orthographies e.g., . This stInternational organizations, such as USAID, have already taken into account the information learned about the impact of specific orthographies on reading acquisition and emphasized in the guidelines of In addition to biological risk factors and compromised learning environment, a third factor that affects reading acquisition, social support, needs to be considered. One of the most important ways in which parents and teachers can support their children’s literacy acquisition, is to find ways to motivate the child to become interested in reading-related activities and to start using reading skills early on. Maintaining the learner’s interest in reading is the most efficient protective factor among children at risk for reading difficulties . ChildreIn Sub-Saharan Africa, one in five children were out of school in 2011  and acroThese dramatic numbers tell us that the full scale and gravity of the global illiteracy crisis has only recently gained public attention. It is not just about how many children cannot go to school in Africa, it is the fact that millions of children are failing to learn despite years of school attendance . To prevIt is worth considering whether the poor literacy results in Africa are related to the fact that it is the only continent where the majority of children still start school using a foreign language . In sub-). Therefore, it is not an exaggeration to say that poor learning outcomes in Africa are intimately linked to the language factor, which seems relatively neglected in the discourse of international development , conducted by five Zambian Ph.D. students. The Project started in 2010 and by then, GG was already available for mobile phones and the intervention was provided to both the teachers and the learners. The medium of initial literacy instruction in the government schools of Lusaka was the widely spoken, local Bantu language, ciNyanja. Since the pilot study had suggested that teachers were not consistently teaching the correct letter-sounds in ciNyanja, a sub-study was conducted which assessed teachers’ knowledge of the sounds of the letters of local languages. The results revealed that teachers’ literacy skills varied significantly and some teachers had difficulties in letter-sound correspondences of ciNyanja, even though they were teaching literacy to first graders . It was therefore necessary to find a method that would support children’s literacy acquisition and also provide literacy instruction training to teachers.The RESUZ intervention study was conducted with a representative sample of 573 Grade 1 learners in 42 Lusaka schools, and comprised a control group and various intervention combinations with the purpose of ascertaining the most efficient intervention model . ResultsMoreover, the RESUZ project revealed the importance of parents’ attitude toward reading by studying a sub-sample of 72 low-income families with children participating in GG interventions . The stuThe most recent literacy assessments from Zambia indicate that the literacy skills levels continue to be low. The empirical validation of GG in the local context of Zambian Government schools points to a viable opportunity to address this challenge. The Early Grade Reading Assessment (EGRA) baseline test conducted in 2012  by Read Figure 2 as an example). It has not yet been possible to see what kind of learning benefits could be achieved if children had access to this technology every week throughout a complete semester, or used it regularly at home.The establishment of CAPOLSA and the fast progress of ICT becoming widely available have prompted new research questions. Now that we know that GG supports both teachers and children, we now need to know how best to distribute it to the learners in need. This was studied in Lusaka in February to August 2014 with the goal of finding the most effective method for providing GG exposure to the maximum number of first grade pupils in urban areas . With siIn 2013, the Ministry of Education revised the primary curriculum in order to expand the use of Zambian local language as the medium of instruction for Grades 2–4 and published a New Literacy Framework that emphasizes the use of phonics in early reading instruction . It is important that information concerning the new policy and available pedagogical guidelines is effectively disseminated to 1000s of teachers in Zambia, both in teacher training colleges and as in-service training to those already working in schools. With improved internet connections and high usage of mobile phones, it is expected that CAPOLSA can provide the required training by using the GG learning environment. In 2012, 76% of Zambians used mobile phone and 13.5% used internet . Since tEncouraged by the success of the Zambian GG team, the same approach has been introduced in Kenya, Tanzania, and Namibia. The first step of the GG adaptation process is the analysis of the existing language environment and current curriculum for literacy instruction. Local experts design the contents for the game and ensure the game’s effectiveness through controlled trials within an authentic environment. In Kenya, a validation study conducted in 2011 assessed the early literacy skills of Grade 1 children in Kikuyu and Kiswahili . The chaEven though Tanzania has been celebrated for its success in promoting a national language and literate culture in an African language and literacy among children and adults , there iAdaptation of GG in Namibia has proceeded to expand the use of the GG learning game in teacher education units as a method to instruct effective use of phonics. Furthermore, the GG content has also been extended to train basic mathematics in Namibia. Similar to many other African countries, Namibia is plagued by a large portion of its learner population being unable to read according to standard. According to SACMEQ reports I–III and EGRA pilots , NamibiaImproving literacy in Africa is not only about the need to build academic capacity and scientific expertise around learning-related sciences. This knowledge needs to be disseminated to the correct stakeholders. The central role that teacher educators play in shaping teachers’ skills is often the most neglected aspect of teacher preparation systems . The UniIn 2012, the Niilo Mäki Institute started a literacy training program in Zambia, Namibia, Kenya, and Tanzania for teacher educators. The main goal of this program was to train literacy specialists who have the latest evidence-based knowledge of learning to read and reading difficulties. GG technology was utilized in this project as the main tool to support learning to read. The 3-years “GraphoLearn Diploma Training Program” was funded by the Ministry of Foreign Affairs of Finland. The 25 trainees and their four local trainers participated in two annual workshops. Each workshop focused on different aspects of emergent literacy, for example, grapheme–phoneme correspondences in transparent languages, phonological awareness, decoding skills, reading fluency and reading comprehension. Assessment and literacy case-study interventions using GG were also included in the program. In every workshop, the trainees presented the results of the case studies that they had carried out in their own countries between the workshops. In these small intervention studies, they practiced how to teach literacy and how to use GG as a part of the training. The local trainers supervised these interventions and reported them to the Finnish trainers. The trainees have learned to identify, assess, and train reading difficulties and as teacher educators, they can pass on these new skills to their students and to the curricula of their universities and teacher training colleges.As mentioned in Quality of education and ensuring literacy acquisition in early grades are fundamental themes in the global post-2015 agenda. One key area is the need for knowledge on inter-dynamics between language and literacy. Conducting interdisciplinary research, consensus building and awareness-raising campaigns to update knowledge on language in education and development is recommended . This wo4 tablets and phones.In September–October 2014, CAPOLSA included a teacher-orientation component in the design of the latest GG field test in rural areas  and condEven though we are currently able to provide mobile-based literacy intervention to teachers, children, and parents, and thus increase the knowledge on reading acquisition in Africa, there are still many challenges. Some studies report that, although ICT seems to be a promising tool in an educational context, many teachers are reluctant to integrate it in their daily practice, even in the developed countries . In deveAnother major challenge is to ensure that learners practice for sufficient time. By all means, this should mean long enough to attain reading fluency, which is required before the learner is ready to transition to the use of new languages . FluencyDistribution of GG and related eLearning content is another long-term process which will most likely involve collaboration with governments of the local countries, as well as support from non-governmental international organizations and private companies. The preparation of GG commercialization has been supported by Tekes, The Finnish Funding Agency for Innovation, in a research project called “Grapho Learning International Development and Exports” which was conducted during 2012–2015. While the adaptations of the new game versions can be conducted as research projects, or within international development projects, the provision of user support in the long run requires other solutions. The next challenge is to expand the GG technology-enhanced support globally. Currently, GG utilizes a project-based delivery model consisting of public-private partnership and donor funding to enable development of the evidence based learning tool in different languages and parts of the world. While approaching new countries and languages, development of a new kind of business model is necessary to ensure financial sustainability and scalability of the initiative. These plans are under development but the aim is to complement the long-term public procurement and donation model by establishing a company and contracting with local distributors. This could mean for example the development of new language versions in collaboration with GraphoWORLD member universities, licensing and distributing the GG service and analytics by the company, and certifying local partners to operate the related training and support services.The aim of GG activities should be to provide services free of charge to the end user in the developing countries. From the business model perspective, the customer could be the adult caretaker of the learner, school, or government, depending on the market situation in a given partner country. GG related research has laid the foundation for the next step for improvement of global literacy. This step, purporting to support dissemination, includes all the processes necessary to make the public procurement or other distribution mechanism possible in each country that wants to make the GG service available in one form or another. Such a step can be taken as the IPR protection (patenting), branding (trademarking), and stakeholder management (communication plan) have been carefully prepared to include the GraphoWORLD network of researchers in the world’s top-universities, ministries, funding partners and sponsors, as well as the general public, in order to raise awareness of the literacy levels and potentially to make a difference by providing inclusive literacy learning for all.This new information on learning needs then to be disseminated to the right stakeholders. Policy makers and teacher educators need to be informed about the new evidence-based methodologies. GG teams in Africa have been active in discussions with their Ministry representatives and have provided expert consultation for curriculum development in teacher training colleges and formulation of new literacy instruction guidelines. If and when the teacher training curriculum or literacy instruction materials change, there is a need to inform 1000s of in-service teachers about these changes. Creating eLearning environments for teachers is a way of disseminating this information very quickly and GG technology allows monitoring of teacher’s professional capacity. One way to do this is to monitor teachers’ literacy and language skills from the GG game logs and a certificate can be awarded when they have reached adequate skill level. Certificate can be given if the teacher shows adequate knowledge of local language literacy by playing GG and is able to successfully organize a GG literacy intervention in her or his classroom and increase the children’s literacy skills with sufficient amount of practice. Performance in the game can be monitored over the internet. This GG instruction certification design was developed by the University of Zambia–CAPOLSA research team in close consultation with the Zambian Ministry of Education, Directorate of Teacher Education and Specialized Services TESS;  and it iIn summary, GG can contribute toward addressing the global literacy crisis on multiple levels. It is apparent that any changes in teacher training, curriculum development, policy making, and production of learning materials needs to be based on solid science and local academic expertise. If the international community wants Sub-Saharan countries to improve their quality of education, it is necessary to provide African scholars and academic institutions with adequate resources and support to conduct research on language, literacy, and learning related topics. For the scientific community, GG is a research laboratory in a mobile phone, enabling much needed research on the process of literacy acquisition. For the teachers, GG can provide relief in their daily struggle with multiple problems. For policymakers, GG provides an opportunity to articulate concrete connections among language, orthography, curriculum, teaching materials, and effective literacy instruction for the next generation of children. While there are many long-term, macrosocial, intertwined challenges for developing education in Africa, there is something that we can do immediately and with little cost: scientific information on learning needs to be available and accessible to all teachers. Without scientific knowledge, all other attempts to improve education are likely to fail expectations. Brand new school buildings and higher salaries have little effect if the teachers never receive the pedagogical training for their work.Sustainable improvement in the quality of education and literacy learning outcomes is a process which needs to happen on multiple levels at the same time: planning education policy, developing learning materials for schools, providing teachers with evidence-based pedagogical training, engaging parents in literacy practices and supporting children with reading difficulties. All this requires scientific research on literacy learning process in authentic environments in local languages by local people. GG and GraphoWORLD network can facilitate this process, not only in Africa, but elsewhere in the world. The latest development of this process is the establishment of the UNESCO Chair on Inclusive Literacy Learning for All, under the leadership of Professor Heikki Lyytinen, the inventor of GG. The purpose of the UNESCO Chair is to promote evidence-based education policies, scientifically valid instruction methods, and effective teacher training programs. Mobile phones can provide this information for teachers and other education professionals around the world. GG can both support the children with learning difficulties and collect data for further research on language, literacy, and learning. Thus, a simple learning game designed for a small minority of Finnish children can now provide learning opportunities for struggling readers across the globe.EO is a project researcher at the Agora Center, University of Jyväskylä and is writing her Ph.D. in Psychology on learning letter-sound correspondences in Zambian languages.MR has worked as a post-doctoral researcher at the Agora Center, University of Jyväskylä and has studied the motivational and learning effects of GraphoGame since early 2000s.TA is Professor at the Department of Psychology, University of Jyväskylä and has been working in Niilo Mäki Foundation international cooperation projects in Africa since 1990s.TC-K is a lecturer at the Department of Psychology, University of Zambia, was part of the Reading support to Zambia-project and wrote her Ph.D. on the effect of home environment on children’s literacy skills.PF is a lecturer at the Department of Psychology, University of Namibia and she is writing her Ph.D. on the use of GraphoGame in practice of literacy skills in Afrikaans.JJ-F is a lecturer at the Department of Psychology, University of Zambia, was part of the Reading support to Zambia-project and wrote her Ph.D. on teaching practices of literacy and use of GraphoGame in Zambia.K-PK worked at the Agora Center, University of Jyväskylä and wrote his Master’s thesis in Psychology on the letter-sound error patterns of the 1st grade GraphoGame players in Zambia.RK is a lecturer at the Department of Teacher Education, University of Helsinki, has been working in Niilo Mäki Foundation international cooperation projects in Africa since 2001 and is part of the original Finnish Ekapeli development team.DN is a lecturer at the Sebastian Kolowa Memorial University, Tanzania and has been been involved with GraphoGame research since 2010.MP is the head of the project management at the Agora Center and has been working in Africa GraphoGame projects since 2008.SP is a project researcher at the Agora Center, University of Jyväskylä and wrote her Ph.D. on the use of GraphoGame in Kiswahili and Kikuyu languages in Kenya.FS is working at the USAID/Read to Succeed project in Zambia, was part of the Reading support to Zambia-project and writing his Ph.D. on children’s early literacy skills in Zambia.GW is a lecturer at Department of Educational Psychology, Sociology and Special Education, University of Zambia, and has conducted research on the use of GraphoGame on tablets in Zambian classrooms.CY is working at the Zambian National Union of Teachers, he was part of the Reading support to Zambia-project and writing his Ph.D. on teachers’ literacy skills and knowledge of literacy instruction methodology for Zambian languages.KP is a Head of the Haskins Laboratories, Professor at Yale University and a member of the GraphoWORLD network of excellence board.UR is a Professor at the Agora Center, University of Jyväskylä and is the head of the GraphoGame unit.RS is a Professor at the Department of Psychology, University of Zambia and is the head of the CAPOLSA.HL is the inventor of the GraphoGame and holds the UNESCO Chair on Inclusive Literacy Learning for all, coordinated by the Agora Center, University of Jyväskylä.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.