The ‘Sponsoring National Processes for Evidence-Informed Policy Making in the Health Sector of Developing Countries’ program was launched by the Alliance for Health Policy and Systems Research, WHO, in July 2008. The program aimed to catalyse the use of evidence generated through health policy and systems research in policymaking processes through (1) promoting researchers and policy advocates to present their evidence in a manner that is easy for policymakers to understand and use, (2) creating mechanisms to spur the demand for and application of research evidence in policymaking, and (3) increased interaction between researchers, policy advocates, and policymakers. Grants ran for three years and five projects were supported in Argentina, Bangladesh, Cameroon, Nigeria and Zambia. This paper seeks to understand why projects in some settings were perceived by the key stakeholders involved to have made progress towards their goals, whereas others were perceived to have not done so well. Additionally, by comparing experiences across five countries, we seek to illustrate general learnings to inform future evidence-to-policy efforts in low- and middle-income countries.J Health Serv Res Policy 8(2):94–9, 2003) as a framing device to reflect on project experiences across the five cases. Using data from the projects’ external evaluation reports, which included information from semi-structured interviews and quantitative evaluation surveys of those involved in projects, and supplemented by information from the projects’ individual technical reports, we applied the theoretical framework with a partially grounded approach to analyse each of the cases and make comparisons.We adopted the theory of knowledge translation developed by Jacobson et al. (There was wide variation across projects in the type of activities carried out as well as their intensity. Based on our findings, we can conclude that projects perceived as having made progress towards their goals were characterized by the coming together of a number of domains identified by the theory. The domains of Jacobson’s theoretical framework, initially developed for high-income settings, are of relevance to the low- and middle-income country context, but may need modification to be fully applicable to these settings. Specifically, the relative fragility of institutions and the concomitantly more significant role of individual leaders point to the need to look at leadership as an additional domain influencing the evidence-to-policy process. A heightened awareness of the mismatch between our ever increasing wants and scarce resources has brought to the forefront questions of resource allocation and priority setting, and the need to justify decisions taken at every level of the health system. This is all the more important in low- and middle-income countries (LMICs) where there are major health challenges accompanied by limited resources to address these. It is against this background that the use of evidence-based and evidence-informed decision-making have gained ground, both in the field of medicine and, more recently, in health policymaking .However, clinical decision-making is a fundamentally different process from policymaking. Clinical medicine, steeped in the positivist paradigm, seeks evidence of generalizable cause and effect relationships, which are universally applicable as the basis of decision-making. However, evidence is only one of the many inputs that policymakers consider. Policymaking is complex and context dependent, influenced by ethical values, interest group and party politics, as well as social and economic factors. Recognizing this complexity, evidence-informed policymaking has been described as an approach that seeks to ‘ensure that decision-making is influenced by the best available research evidence’, even while it acknowledges the multiplicity of factors that influence policymaking .This paper reflects on the practical experience of a multi-country program  programme) aimed at enhancing evidence-informed policymaking in five LMICs supported by the Alliance for Health Policy and Systems Research (AHPSR), an international partnership based within WHO. Over the past decade, WHO has played an important role in promoting evidence-informed policymaking, from the 2004 Ministerial Summit on Health Research, held in Mexico , to the The approaches used in this programme included (1) encouraging researchers and policy advocates to present their evidence succinctly and vividly, (2) increasing interaction between researchers, policy advocates and policymakers, and (3) strengthening the capacity of Ministries of Health (MOHs) to demand and utilize evidence. While reflections on single projects as well as on specific components of this program have been previously published -9, this There are a number of models that seek to explain the incorporation of research evidence into decision-making. These include (1) ‘push’ models, which emphasize the role of producers of research to provide information to decision-makers, (2) ‘pull models’, which give primacy to efforts by research users such as decision-makers to ‘extract information’ from the world of research, (3) ‘exchange models’, which prioritize the creation of linkages or partnerships between producers and users of research, and (4) integrated approaches combining elements of the three aforementioned approaches that make use of knowledge translation platforms that bring together research consumers and producers and encourage both push and pull efforts towards clearly defined goals, often established through priority setting mechanisms . Given, There exists a significant literature largely centred around high-income countries that examines the use of research evidence in the health policymaking process in an attempt to identify factors influencing the incorporation of evidence into decision-making. These include systematic reviews by Innvaer et al.  and by OThere is also a growing literature in this area from LMICs. Evidence-informed decision-making faces additional hurdles in these settings. Reflecting funding availability and institutional research capacity, there is often a paucity of locally relevant research evidence available to inform decision-making. This is often accompanied by a shortage of skilled human resources in MOHs to demand, evaluate, synthesize and adapt available research evidence. Frequent transfers of officials within MOHs, something that is not unusual in these settings, further exacerbates this problem. Examples of studies based in LMICs include a literature review and case study on Thailand by Sauerborn et al. , a studyhttp://www.who.int/alliance-hpsr/projects/national_processes/en/). The second section discusses our preferred theoretical framework, whereas the third describes data sources and methods used to analyse the data. Section four presents the findings and examines these in terms of the theoretical framework. The final section discusses the implications of our study in terms of lessons learned for evidence-informed policymaking for health systems in LMICs.The remainder of this paper is divided into five parts. We begin with a description of projects carried out under the AHPSR’s SNP program  promoting researchers and policy advocates to present their evidence in a manner that is easy for policymakers to understand and use, (2) creating mechanisms to spur the demand for and application of research evidence in policymaking, and (3) increased interaction between researchers, policy advocates and policymakers. A number of strategies were suggested including the creation of platforms to produce and communicate research to policymakers in an accessible manner, training policymakers and establishing units within MOHs to strengthen MOH capacity to demand and use research evidence, developing policy briefs (defined as documents that bring together the best available evidence on a problem and viable solutions to address the problem) , and sup promotinThe SNP program is thus best viewed as a joint effort involving national stakeholders (including researchers and former policymakers) who designed and implemented individual projects, and an international collaboration, namely the AHPSR, which conceptualized and funded this program as well as mandated the evaluation carried out by a team of international researchers.We use the theory of knowledge translation developed by Jacobson et al.  as a fraJacobson et al.  give greA document review of reports produced by the AHPSR SNP program was conducted at the outset. The projects’ formal evaluation report was the main source of data for this paper . This waThe project evaluation report was produced by two AHPSR appointed external researchers who between them carried out the evaluation across the five projects. Country visits were carried out to monitor project progress, review documents and hold interviews with project leaders, team members, and selected decision-makers, as discussed below. Visits were made to all project sites except Argentina, where this was difficult due to logistic reasons. Additionally, five meetings were held with project teams taking advantage of other events including SURE  meetings as well as the McMaster Health Forum policy dialogues . The evaSemi-structured interview guides were developed by the Knowledge Translation Platform Evaluation Team at McMaster University . InitialThe evaluation report also included results from surveys on (1) policy briefs produced, (2) policy dialogues held, and (3) project outcomes. Potential respondents for these surveys were identified by project teams based on their likely involvement in policymaking around issues that were being examined by the project teams. Respondents included national and sub-national level policymakers, managers at healthcare institutions, staff members of donor agencies, non-governmental organizations, or professional health associations and researchers at universities and other research institutions ,24.For the policy brief survey, identified individuals were posted a package containing policy briefs prepared by the country projects and an invitation to take part in a policy dialogue. Along with the brief and invitation, respondents were provided with a questionnaire to assess the policy briefs that they were required to complete prior to attending the policy dialogue. The questionnaire included questions on respondent perceptions of the process by which the brief was designed and developed as well as an overall assessment of the policy brief itself examined through a 7-point Likert scale ,24. A toImmediately after the policy dialogue, participants were provided with a questionnaire to assess the policy dialogue process and were required to complete and return the questionnaire on the spot . The queFinally, for both the Nigeria and Zambia projects, the evaluation report included results from an outcomes evaluation survey conducted after more than 2 years of project implementation to examine views of these same groups of stakeholders on (1) research availability for issues with high priority in health policy, (2) relationships between researchers, government officials and other policymakers, and (3) the capacity of both researchers and policymakers to promote evidence-informed policymaking in the health sector. The outcomes evaluation survey contained a mix of questions, some of which used a 7-point Likert scale while others were open ended questions. The target sample size was up to 50 respondents in each setting, the final sample size was 66 respondents in Nigeria and 48 respondents in Zambia .Information from the evaluation report was supplemented by findings from individual final project technical reports submitted by project teams to the AHPSR -32. ThesUsing the sources above, we first summarized information on each project in terms of outputs and activities, respondents’ perceptions of policy briefs and policy dialogues, and their perceptions of outcomes, enabling us to categorize projects according to their performance in (1) producing policy relevant evidence, (2) fostering interactions between policymakers and researchers, and (3) building capacity of policymakers to demand and use research.We then applied the theoretical framework within a partially grounded approach to analyse each of the five cases. This involved coding the data from each of the five projects in terms of the chosen framework as well as allowing for new categories to emerge from the data in order to not force the data to fit the theory . We emplInformation on project areas and activities is provided in Table The research team in Argentina appears to have focused its attention solely on the production of policy briefs and holding policy dialogues with little evidence of sustained advocacy and dissemination efforts. There was no information on the impact of the policy briefs or dialogues on the policymaking process nor any evidence of the creation of structural-links across researchers and policymakers or of capacity-creation or strengthening efforts ,25,30.had a huge impact on specific decisions taken by policymakers at the national level” by the project leader during the semi-structured interview [In Bangladesh, policy briefs regarding the use of the H1N1 vaccine and non-communicable diseases were assessed to have “nterview ,31. The nterview ,31.In the Zambia case, while the impact of the policy briefs on policymaking itself remains unclear, the great majority of respondents felt that the policy briefs and dialogue both achieved their desired objectives Tables  and 4. TTwo of the policy briefs produced by the Cameroon project, on community health insurance and on scaling up malaria control in the country, were asserted by the project leader to have influenced national policies in these areas . RespondOver 85% of respondents felt that the policy briefs and dialogues undertaken as part of the Nigerian project had largely achieved their goals Tables  and 5. TFrom the above information and Table Next, we used the theoretical framework to frame team experiences and explain variations in project performance. We recognize that the relative importance of each of the five domains is context specific and will vary across settings, and that interactions among the domains are critical. For example, interested user-groups (domain: characteristic of the user-group), are more likely to be concerned about the research produced on the area of work that interests them and would thus tend to commission research that is relevant to them (domain: issue characteristic). They would also tend to make clear to the researcher the type of output that they would find useful in their work (domain: characteristic of dissemination strategy). Conversely, in the presence of an indifferent user group, researchers are more likely to pick issues that they think important and produce outputs that they deem relevant; a situation that is far less likely to lead to evidence incorporation into decision-making. Table The findings from Argentina clearly illustrate the importance of the coming together of domains. While we lack much information on user-group perceptions of research and researchers, the project technical report explicitly mentioned the difficulty in getting together experts and policymakers to hold policy dialogues as an obstacle to the projects’ functioning, suggesting a generally low level of policymaker enthusiasm on the topic. Additionally, the groups had never worked together before and there was a frequent turnover of MOH staff, which made the establishment of relationships between researchers and policymakers difficult. The need for the project to actively work to ‘spur political interest’ in the issue of provincial health insurance (the main focus of the researchers’ effort), as mentioned in the project technical report, suggests that this was not an issue that was on the policy agenda at the time . ResearcThe experience from Bangladesh, on the other hand, demonstrates how critical perceived relevance of research on a particular issue at a particular instance of time is to explaining the success of the evidence-to-policy process. While we lack quantitative information on user group perceptions, data from the merit review indicates that policymakers were keen to increase their interactions with researchers. In addition to forming the basis of a trusted relationship, the pre-existing links between an established research institution International Centre for Diarrhoeal Disease Research, Bangladesh and health policymakers in Bangladesh facilitated the latter’s early involvement in the project and potentially played an important role in enabling the project to overcome the challenge posed by frequent staff changes in the MOH . It alsoUser groups in Zambia tended to have a positive view of research and researchers, with 97% of respondents mentioning that the use of research evidence was extremely beneficial to their work. An overwhelming 86% of the respondents also felt that they were expected to use research evidence in their work Table . In commIn common with the information from other African countries, members of user groups in Cameroon appeared to have experience in carrying out research, with 67% of respondents who did not describe themselves as researchers indicating that they had research experience. The project’s activities were facilitated by pre-existing institutional structures, being based at a research institution that had been set up to promote the cause of evidence-informed policymaking in Cameroon’s health sector . This stThe Nigeria project followed a similar pattern to Cameroon. User groups wanted to and were expected to use research evidence, with 98% and 87% of respondents, respectively, strongly agreeing with these assertions Table . In commReflecting on our experiences from these AHPSR-supported projects in five countries across three continents, we can conclude that the successful incorporation of evidence into policymaking is indeed greatly aided by the coming together of a number of domains . The comThe results of this study are supported by findings from a systematic review by Innvaer et al.  that ideHowever, we caution against an understanding of the evidence-to-policy process as a simple coming together of domains. First, as explained above, the domains have the potential to reinforce each other and reflect an interactive spectrum where the whole can be greater or lesser than the sum of the parts. The Argentinian case of apparent user group indifference, apparently weaker pre-existing user group-researcher relationships, and lack of issue salience lies at one end of the spectrum. The Nigerian project, with political interest and involvement, systematic priority-setting exercises and capacity building initiatives, represents the other. Second, the relative contribution of each of these domains varied greatly across the different projects suggesting the importance of other contextual factors in shaping observed effects. For example, it would seem that, in Bangladesh, the issue, including the urgency surrounding it, its technical nature and potential to affect key interest groups, determined whether or not evidence was considered in policy around that issue. On the other hand, in the Zambian experience, the project’s ability to reach out to policymakers determined whether or not research evidence was considered in policymaking around the issue.Additionally, in line with Harries et al. , we findThese findings suggest that, though relevant to a large extent, Jacobson’s theoretical framework, initially developed for high-income settings, may need modification to be applicable to LMICs. The relative fragility of institutions and concomitantly more significant role of individual leaders points to the need to look at leadership as an additional domain influencing the evidence-to-policy process.The evidence from Nigeria, Zambia and Cameroon (where we have this information) also suggests that the separation between the ‘two communities’ of researchers and policymakers discussed above in the theoretical framework  is not aFinally, the findings from this study point to the need for comprehensive mechanisms to facilitate the incorporation of evidence in policymaking and overcome what has been termed as the false dichotomy of the ‘know-do’ gap in health policy and systems research . EmbeddiThere are some limitations to this study. The most significant being that the paper is based primarily on the perceptions of stakeholders including researchers, policymakers, and a few civil society and development partner groups who were in turn identified by the country project teams. It is these perceptions as opposed to well-defined policy outcomes that have been used to assess the role of evidence in informing health policymaking. Additionally, individuals identified by the project teams may feel a need to report positively about project processes and outcomes, which is a potential source of bias in the data. The inclusion of a wider range of stakeholders, such as the media as well as politicians, in the sample would have further strengthened and potentially enhanced the generalizability of the findings, in addition to removing this potential conflict of interest. Second, data availability varied by project. Having identical data sources across all the projects would have enhanced our ability to draw inferences by directly comparing results across sites. Third, the project evaluation was led by a team of two outside researchers from major universities in the Middle East and Latin America. While, on the one hand, this reduces the chance of biased analysis and reporting, on the other, it has the potential to lead to a loss of insider perspective that critical self-evaluation can provide and that an outsider may not be able to access through the interview process. Notwithstanding these limitations, we do believe that this study, by comparing the five countries as cases, has enabled us to at least answer some of the ‘why’ questions behind each project’s achievements, to contribute to further build on the chosen theoretical framework and significantly inform efforts towards evidence-informed health policy in LMICs.


Software assurance is an important part of the software development process to reduce risks and ensure that the software is dependable and trustworthy. Software defects and weaknesses can often lead to software errors and failures and to exploitation by malicious users. Testing, certification and accreditation have been traditionally used in the software assurance process to attempt to improve software trustworthiness.In this paper, we examine a methodology known as a structured assurance model, which has been widely used for assuring system safety, for its potential application to software assurance. We describe the structured assurance model and examine its application and use for software assurance. We identify strengths and weaknesses of this approach and suggest areas for further investigation and testing. Software for Dependable Systems—Sufficient Evidence?” by the Committee on Certifiably Dependable Software Systems of the National Research Council, [evidence-based approach for assessing and assuring dependability in software systems that argues for and justifies dependability claims based on explicit evidence supporting such arguments and claims.Software assurance is an important part of the software development process to reduce risks and ensure that the software is trustworthy. The critical importance of establishing and assuring dependability and trustworthiness  of systems and/or software in avionics, industrial control systems and other safety and mission-critical systems has long been recognized.Council,  the CommAn evidence-based approach to software system assurance can be made through use of an assurance case methodology based on a structured assurance case model. Structured assurance case models have been used widely in the United Kingdom , 3 and UIn this paper, we describe the structured assurance case model and examine its application to software assurance with a simple software-based system example.NIST/SAMATE project:– Trustworthiness—no exploitable vulnerabilities exist either of malicious or unintended origin, and– Predictable execution—justifiable confidence that software, when executed, functions as intended.”“The planned and systematic set of activities that ensures that software processes and products conform to requirements, standards and procedures in order to help achieve:DHS SwA Program: “Software assurance (SwA) is the level of confidence that software is free from vulnerabilities, either intentionally designed into the software or accidentally inserted at any time during its life cycle, and that the software functions in the intended manner.” .Various definitions of software assurance have been given. For example, Wikipedia  referenca documented body of evidence that provides a convincing and valid argument that a specified set of critical claims regarding a system ’s properties are adequately justified for a given application in a given environment [A structured assurance case is “ironment .” Much lThe structured assurance case model is represented as a directed graph whose nodes consist of claims, arguments, and evidence elements. Currently, the Object Management Group (OMG) System Assurance Platform Task Force (PTF)  is including the structured assurance case model as part of a common framework of standards for analysis and exchange of information related to systems assurance and trustworthinesss . This FrThe standards group, ISO/IEC JTC1 Sub-Committee 7 WG7, is also including the structured assurance model as a revision to Part 2 of the ISO/IEC 15026, Systems and Software Assurance standard [A claim is a statement asserting some characteristic, property, or behavior of the software or system that can be evaluated for truthfulness, is demonstrable, and is supported by arguments based on objective evidence. A claim may be further decomposed into sub-claims, and expressed either as a positive or negative statement.For example, one can declare positive claims about the requirements-based, quality properties of software, such as its dependability or availability, or one can make negative claims about the same software by claiming that the code does not contain specific weaknesses and vulnerabilities in the design and implementation that could be exploited to break or compromise the system.Arguments are logical propositions intended to support a claim through reasoning or logic that links evidence to a claim. Arguments define the relationships directly linking each claim and/or sub-claims, and piece of evidence, used by an argument to the claims immediately supported by the argument. An argument is the explanation of how the evidence can be interpreted as supporting a claim or sub-claim.Arguments can also include any unusual events or conditions that are within the context of the claim. The argument can contain considerations of potential causes of failure and appropriate corrective actions if failure occurs. Hence, an argument may include conditions, assumptions, and judgments about the system, its use, and its operational environment, threats, and likelihood of occurrence, for which the claims and evidence are being marshaled as part of an overall assurance case.Evidence is information used to support a claim. Ideally, evidence should be objective, reproducible, repeatable, and non-disputable. Evidence is key to making a credible assurance case. Without evidence, there is no way to substantiate the claim.The sources of evidence will depend in part on the availability of artifacts. The evidence data collection may be conducted formally, informally or semiformally.Evidence comes in many different forms, so it is impossible to dictate what kind of evidence or argument is appropriate for every situation. Evidence may be in the form of an artifact which could be automatically, semi-automatically or manually produced and demonstrated. Evidence must be traceable to its source and method of origination. The evidence may consist of test results, formal analyses, simulation results, hazard analyses, modeling, inspections, and can include deterministic, probabilistic, and qualitative data or information . ExampleThe structured assurance case model defined previously has potential in providing a framework for an effective software assurance case. The model may be developed in a top-down approach, bottom-up approach, or a mix of both top-down and bottom-up approaches.– Define or assert a top level claim about a software or system property which is to be shown.– Consider decomposing the top level claim into smaller related sub-claims.– Identify or provide the supporting evidence for the sub-claims.– Develop a set of arguments that link claims/subclaims to evidence to support the claims/subclaims.– State any assumptions, judgments, and conditions underlying the claims, arguments, and evidence.– Evaluate the strength and sufficiency of the assurance case evidence and arguments in substantiating the claims and sub-claims.The basic steps in implementing a structured assurance case are to:sufficiency will often be decided by some combination of objective evidence  and expert opinion that collectively provides strong and plausible evidence supporting an argument and claim.The process is both cumulative and iterative. Assurance claims and sub-claims may be decomposed to any level of granularity until necessary and sufficient evidence is obtained in supporting the satisfaction of a claim or a sub-claim. The structured assurance case then rests upon the aggregation of all sub-claims and arguments, each supported by evidence, which collectively satisfies a top-level claim. Ideally, objective measures of whether the evidence is of high quality and sufficient are desirable. However, in practice, this may be difficult and – Goal Structuring Notation (GSN) – Claims-Arguments-Evidence (CAE) Structuring assurance cases so they can be understood is a challenge. Due to the massive amount of evidence that may be needed to demonstrate an assurance case for moderately-sized software, and to improve human capability for reviewing and visualizing an assurance case, automated tools have been developed. Examples of notations for which tools have been developed, include:Both use a graphical notation for representing the structure of an assurance case. There are similarities and differences in notations among different tools. For example, the GSN notation defines node types for Goals (claims), Strategy (argument), and Solution (evidence), with supporting nodes that include Assumptions, Justifications, Context, Models, and Notes. CAE defines nodes for Claims, Arguments, and Evidence. GSN has a goal-oriented view that supports a top-down approach in developing the structured assurance case beginning with claims, while the CAE supports a bottom-up view that uses evidence to determine which claims can be made. Currently, within the Object Management Group (OMG) there is an effort to produce a standard that encompasses concepts from both of these notations .The sources of evidence will depend in part on the availability of artifacts. The evidence data collection may be conducted formally, informally or semiformally.Data facts, as evidence, are collected to support the argument that the software will satisfy particular claims for software assurance.As an illustration of a structured assurance case model approach for a software-based product, a simple example is shown for an automated teller machine (ATM). This assurance-related claim is derived from a presumed security specification requirement for an ATM that states that the ATM must not allow un-authorized access to a bank account.The structured assurance case model has been extensively used and shown to be an effective approach for assuring safety in avionics and other complex systems . Other aThe structured assurance case model, applied to software assurance, can support various stakeholder roles and needs, including those of the developer, acquirer, and certifier throughout the system life-cycle. The structured assurance case model provides a framework for identifying critical properties of a software system, such as, safety, security, and dependability, and ensuring that these are addressed during development, implementation, and testing. Furthermore, an assurance framework enables capturing lifecycle artifacts that provide the evidence needed to support claims about these requirements. The structure and hierarchy of the structured assurance case model can help identify gaps between claims, arguments, and evidence, and provide a consistent approach for software assurance.The structured assurance case model offers an additional approach to software assurance that has traditionally been provided through certification and accreditation activities by providing traceability. Thus, use of this approach can improve the overall software assurance process.– Measuring the effectiveness of the structured assurance case model for software assurance.– Determining what and how much evidence is sufficient for verifying a claim/sub-claim.– Ensuring that the quality of evidence is satisfactory.– Ensuring an appropriate level of detail or granularity of sub-claims.– Ensuring that relationships among claims, arguments, and evidence are clear and explicit.– Managing large, complex structured assurance case models.– Improving guidance on how to efficiently gather, merge, and review arguments and evidence.– Developing automated tools to analyze structured assurance cases.However, the use of structured assurance case models for software assurance is an on-going topic of research and case-studies. There are still open issues surrounding the use of the structured assurance method for software assurance, including:Difficulty in transforming existing safety and security requirements into the structure of an evidence-based assurance model. Standards for defining safety and security requirements for application domains often specify a structure and format that do not easily translate into a structured assurance case model. The result can be an assurance model that is incomplete, contradictory, and not aligned with the requirements.Existing assurance models in safety and security rely heavily on evidence of compliance to standards for lower levels of assurance  and evaluation assurance levels (EAL) respectively), with the assumption that adherence to those standards validates the overall assurance claim. Use of an evidence-based model can facilitate the use of artifacts generated by tools as evidence against the actual system itself, providing a stronger claim of safety, security or other property for lower levels of assurance.Assurance modeling of “system of systems” adds another layer of complexity to the assurance case. While a system may be deemed safe or secure by itself in a particular environment, the introduction of other systems into that environment increases the complexity of the assurance model and must be considered and evaluated as part of a larger system. Assurance case models today do not address the assurance of systems of systems.Some ongoing issues with the structured assurance case model approach include:Use of a structured assurance case method shows promise for use in assurance of software properties, such as, safety, security, reliability, and others. This model provides an organized, structured approach to software assurance based on claims, arguments and evidence, and provides a means of traceability among these elements.The model appears useful throughout the software development lifecycle by providing a framework where intended product claims can be identified early in the development cycle and used to identify system requirements upon which these claims can be based, and for which arguments and evidence can be established during development to support these claims.quality” of the model in providing a strong software system assurance case.However, further work is needed in developing models for different software system properties and examining relationships and patterns that may exist within and among these models. Further work is also needed to develop automated methods to handle and process potentially large and complex assurance models, and support definition, maintenance, and revision of large assurance models, amounts of evidence, and to develop methods for objective measurements for evaluating the “


A fundamental concern of a theory of statistical inference is how one should measure statistical evidence. Certainly the words “statistical evidence,” or perhaps just “evidence,” are much used in statistical contexts. It is fair to say, however, that the precise characterization of this concept is somewhat elusive. Our goal here is to provide a definition of how to measure statistical evidence for any particular statistical problem. Since evidence is what causes beliefs to change, it is proposed to measure evidence by the amount beliefs change from a priori to a posteriori. As such, our definition involves prior beliefs and this raises issues of subjectivity versus objectivity in statistical analyses. This is dealt with through a principle requiring the falsifiability of any ingredients to a statistical analysis. These concerns lead to checking for prior-data conflict and measuring the a priori bias in a prior. Given that statistical reasoning is used throughout science, it is important that such a theory be sound, in the sense that it is free from illogicalities and counterexamples, and be complete, in the sense that it produces unambiguous answers to all properly expressed statistical problems.It is our contention that any such theory must deal explicitly with the concept of statistical evidence. Statistical evidence is much referred to in the literature, but most theories fail to address the topic by prescribing how it should be measured and how inferences should be based on this. The purpose of this paper is to provide an outline of a theory based on an explicit measure of statistical evidence.Example 1The Archetypal Statistical Problem.Before describing this, there are several preliminary issues that need to be discussed. To start, we are explicit about what could be seen as the most basic problem in statistics and to which all others are related.X : Ω → χ. As such X(ω) ∈ χ is the measurement of object ω ∈ Ω.Suppose there is a population Ω with #(Ω) < ∞. So Ω is just a finite set of objects. Furthermore, suppose that there is a measurement X over Ω or, equivalently, the relative frequency function fX(x) = # ({ω : X(ω) = x})/ # (Ω) for χ is. Typically, only a subset {ω1, … , ωn} ⊂ Ω can be observed giving the data xi = X(ωi) for i = 1 , … , n where n ≪ # (Ω), so there is uncertainty about fX.This leads to the fundamental object of interest in a statistical problem, namely, the relative frequency distribution of fX is to propose that fX ∈ {fθ : θ ∈ Θ}, a collection of possible distributions, and referred to as the statistical model. Due to the finiteness of Ω, and the specific accuracy with which X(ω) is measured, the parameter space Θ is also finite.The standard approach to dealing with the uncertainty concerning Note that in fθ(x) = limϵ → 0Pθ(Nϵ(x))/Vol(Nϵ(x)) where Nϵ(x) is a set that shrinks nicely to x, as described in Rudin Pθ(Nϵ(x)) ≈ fθ(x)Vol(Nϵ(x)) for small ϵ.So the position taken here is that in statistical problems there are essentially no infinities and there are no continuous distributions. Infinity and continuity are employed as simplifying approximations to a finite reality. This has a number of consequences, for example, any counterexample or paradox that depends intrinsically on infinity is not valid. Also, densities must be defined as limits as in θ ∈ Θ and as such is subjective and so unscientific. Our answer to this is that all the ingredients to a statistical analysis are subjective with the exception, at least when it is collected correctly through random sampling, of the observed data. For example, a model {fθ : θ ∈ Θ} is chosen and there is typically no greater foundation for this than it is believed to be reasonable, for example, this could be a set of normal distributions with unknown mean and variance.To define a measure of evidence we need to add one more ingredient, namely, a prior probability distribution as represented by density π on Θ. For some, the addition of the prior will seem immediately objectionable as it is supposed to reflect beliefs about the true value of Principle of empirical criticism: Every ingredient chosen by a statistician as part of a statistical analysis must be checked against the observed data to determine whether or not it makes sense.The subjective nature of any statistical analysis is naturally of concern in scientific contexts as it is reasonable to worry about the possibility of these choices distorting what the data is saying through the introduction of bias. We cope with this, in part, through the following principle.x, has been collected appropriately and so can be considered as being objective.This supposes that the data, which hereafter is denoted by fθ in the model, is a familiar process and so the model satisfies this principle. It is less well-known that it is possible to provide a consistent check on the prior by assessing whether or not the true value of θ is a surprising value for π. Such a check is carried out by computing a tail probability based on the prior predictive distribution of a minimal sufficient statistic  factors as Π × Pθ = Π(⋅| T) × MT × P(⋅| T) where P(⋅| T) is conditional probability of the data given T, MT is the prior predictive for T and Π(⋅| T) is the posterior for θ. These probability measures are used respectively for model checking, checking the prior and for inference about θ and, as such, these activities are not confounded. Hereafter, it is assumed that the model and prior have passed their checks so we focus on inference. It is not at all clear that any other ingredients, such as loss functions, can satisfy the principle of empirical criticism but, to define a measure of evidence nothing beyond the model and the prior is required, so this is not a concern.It is to be noted that, for any minimal sufficient statistic fθ : θ ∈ Θ}, a prior π and data x, we pose the basic problems of statistical inference as follows. There is a parameter of interest Ψ : Θ → Ψ (we do not distinguish between the function and its range to save notation) and there are two basic inferences.ψ = Ψ(θ) together with an assessment of the accuracy of the estimate.Estimation: Provide an estimate of the true value of H0 : Ψ(θ) = ψ0 is either true or false together with an assessment of the strength of this evidence.Hypothesis assessment: Provide a statement of the evidence that the hypothesis Given a model {p-values are often cited as corresponding to strong evidence. In fact, separating the measure of evidence from a measure of its strength helps to resolve various difficulties.Some of the statement concerning hypothesis assessment is in italics because typically the measure of the strength of the evidence is not separated from the statement of the evidence itself. For example, large values for Bayes factors and very small p-values, Bayesian theories and Bayes factors, and fiducial inference. Some of the salient points are presented in the following paragraphs together with further references.There are of course many discussions in the statistical literature concerning the measurement of evidence. Chapter 3 of Evans θ. A likelihood ratio, however, is a measure of relative evidence between two values of θ and is not a measure of the evidence that a particular value θ is true. The relative belief ratio for θ, defined in θ is true and furthermore a calibration of this measure of evidence is provided. While these are significant differences in the two approaches, there are also similarities between the pure likelihood approach and relative belief approach to evidence. For example, it is easily seen that the relative belief ratio for θ gives the same ratios between two values as the likelihood function. Another key difference arises, however, when considering measuring evidence for an arbitrary ψ = Ψ(θ). Pure likelihood theory does not deal with such marginal parameters in a satisfactory way and the standard recommendation is to use a profile likelihood. A profile likelihood is generally not a likelihood and so the basic motivating idea is lost. By contrast the relative belief ratio for such a ψ is defined in a consistent way as a measure of change in belief.Edwards p-values are commonly used as measures of evidence. A basic issue that arises with the p-value is that a large value of such a quantity cannot be viewed as evidence that a hypothesis is true. This is because in many examples, a p-value is uniformly distributed when the hypothesis is true. It seems clear that any valid measure of evidence must be able to provide evidence for something being true as well as evidence against and this is the case for the relative belief ratio. Another key problem for p-values arises with so-called “data snooping” as discussed in Cornfield p-values as measures of evidence. A much bigger issue for a frequency theory of evidence is concerned with the concept of ancillary statistics and the conditionality principle. The lack of a unique maximal ancillary leads to ambiguities in the characterization of evidence as exemplified by the discussion in Birnbaum In frequency theory In a Bayesian formulation the Bayes factor is commonly used as a measure of evidence. The relationship between the Bayes factor and the relative belief ratio is discussed in The Dempster–Shafer theory of belief functions, as presented in Shafer 2ω with associated probability measure P.A , C ⊂ Ω with P(C) > 0, if told that the event C has occurred, then replace P(A) by The principle of conditional probability: For events To determine inferences three simple principles are needed. First is the principle of conditional probability that tells us how beliefs should change after receiving evidence bearing on the truth of an event. We let Ω denote a general sample space for response P(A | C) > P(A), then there is evidence in favor of A being true because the belief in A has increased. If P(A | C) < P(A), then there is evidence A is false because the belief in A has decreased. If P(A | C) = P(A), then there isn't evidence either in favor of A or against A as belief in A has not changed.Principle of evidence: If This leads to a very simple characterization of evidence.P(A), P(A | C)). A number of such measures have been discussed in the literature and Crupi et al. A is true, having observed C, is measured by the relative belief ratio RB(A | C) = P(A | C)/P(A) when P(A) > 0.Principle of relative belief: The evidence that This principle suggests that any valid measure of the quantity of evidence is a function of (RB(A | C) > 1 implies that observing C is evidence in favor of A and the bigger RB(A | C) is, the more evidence in favor.So, for example, BF(A | C) in favor of A being true is the ratio of the posterior to prior odds in favor of A. It is easily shown that BF(A | C) = RB(A | C)/B(Ac | C), namely, from the point of view of the relative belief ratio, the Bayes factor is a comparison between the evidence in favor of A and the evidence in favor of its negation. The relative belief ratio satisfies RB(A | C) = BF(A | C)/(1 − P(A) + P(A)BF(A | C)) and so cannot be expressed in terms of the Bayes factor itself. From this it is concluded that the relative belief ratio is a somewhat more elemental measure of evidence. As discussed in Baskurt and Evans The Bayes factor is also used as a measure of evidence. The Bayes factor ψ = Ψ(θ). Let πΨ(⋅| x) and πΨ denote the posterior and prior densities of ψ. Then the three principles imply that the relative belief ratioψ is the true value and this holds as a limit in the continuous case, see Evans RBΨ(ψ | x) so the measures agree in that context. Given RBΨ(⋅| x), this prescribes a total order for the ψ values as ψ1 is not preferred to ψ2 whenever RBΨ(ψ1 | x) ≤ RBΨ(ψ2 | x) since there is at least as much evidence for ψ2 as there is for ψ1. This in turn leads to unambiguous solutions to the inference problems.For the statistical context suppose interest is in 2.1ψ is the value for which the evidence is greatest, namely,cΨ , γ(x) = inf {k : ΠΨ(RBΨ(ψ | x) ≤ k | x) ≥ 1 − γ}. Notice that ψ(x) ∈ CγΨ , (x) for every γ ∈  and so, for selected γ, the size of CΨ , γ(x) can be taken as a measure of the accuracy of the estimate ψ(x). Given the interpretation of RBΨ(ψ | x) as the evidence for ψ, we are forced to use the sets CΨ , γ(x) for the credible regions. For if ψ1 is in such a region and RBΨ(ψ2 | x) ≥ RBΨ(ψ1 | x), then ψ2 must be in the region as well as there is at least as much evidence for ψ2 as for ψ1. This presents the relative belief solution to the Estimation problem.The best estimate of 2.2H0 : Ψ(θ) = ψ0, the evidence is given by RBΨ(ψ0 | x). One problem that both the relative belief ratio and the Bayes factor share as measures of evidence, is that it is not clear how they should be calibrated. Certainly the bigger RBΨ(ψ0 | x) is than 1, the more evidence there is in favor of ψ0 while the smaller RBΨ(ψ0 | x) is than 1, the more evidence there is against ψ0. But what exactly does a value of RBΨ(ψ0 | x) = 20 mean? It would appear to be strong evidence in favor of ψ0 because beliefs have increased by a factor of 20 after seeing the data. But what if other values of ψ have even larger increases?For the assessment of the hypothesis RBΨ(ψ0 | x) can be calibrated, however, by comparing it to the other possible values RBΨ(⋅| x) through its posterior distribution. For example, one possible measure of the strength isψ has a relative belief ratio no greater than that of the hypothesized value ψ0. While Eq. p-value, it has a very different interpretation. For when RBΨ(ψ0 | x) < 1, so there is evidence against ψ0, then a small value for Eq. RBΨ(ψ0 | x) and there is strong evidence against ψ0. If RBΨ(ψ0 | x) > 1, so there is evidence in favor of ψ0, then a large value for Eq. RBΨ(ψ0 | x) and so there is strong evidence in favor of ψ0. Notice that, in the set {ψ : RBΨ(ψ | x) ≤ RBΨ(ψ0 | x)}, the “best” estimate of the true value is given by ψ0 simply because the evidence for this value is the largest in this set.The value RBΨ(ψ0 | x), as the measure of the evidence, and Eq. Various results have been established in Baskurt and Evans RBΨ(ψ0 | x) > 1 and ΠΨ({RBΨ(ψ0 | x)}| x) is large, there is strong evidence in favor of ψ0 while, if RBΨ(ψ0 | x) < 1 is very small, then there is immediately strong evidence against ψ0.So if x is a sample of n from a N distribution, with μ ∈ R1 unknown and σ2 known, and the prior is given by a N distribution. It is common to take τ02 very large to reflect the lack of much prior information about the true value of μ. But it is easily shown that  → ∞ as τ02 → ∞ and this is also true of the Bayes factor as it equals RB(μ | x) in this case. So by being appropriately uninformative about the true value of μ, one can make the evidence in favor of a particular value of μ as large as one likes. This example also produces the Jeffreys–Lindley paradox because it is possible that the classical frequentist p-value is very small when assessing the hypothesis that μ0 is the true value, while the corresponding relative belief ratio/Bayes factor is large in favor of this hypothesis and so these measures contradict each other. When the relative belief ratio is calibrated, however, the classical p-value is seen to arise as a measure of the strength of the evidence and so this says that, while there may be evidence in favor of μ0, it may be weak evidence. It is clear that by choosing the prior to be very diffuse a bias in favor of the hypothesis is being introduced and the final resolution of the paradox is accomplished by computing what is referred to as bias in favor, as is discussed in the following section. This example makes it clear that the value of a relative belief ratio or Bayes factor cannot be interpreted generally as a measure of the strength of the evidence.To see more clearly the issue concerning calibration consider the following basic example. Suppose that the data 2.3RBΨ(ψ0 | x) to assess the evidence that ψ0 is the true value. One of the key concerns with Bayesian inference methods is that the choice of the prior can bias the analysis in various ways. An approach to dealing with the bias issue is discussed in Baskurt and Evans ψ0 is true is based on RBΨ(ψ0 | x), the solution is to measure a priori whether or not the chosen prior induces bias either in favor of or against ψ0. To see how to do this, note first the Savage–Dickey ratio result (see Dickey x given that Ψ(θ) = ψ0 and x.There is another issue associated with using ψ0 can be measured by computingM(⋅| ψ0) is the prior probability measure of the data given that ψ0 is the true value. Therefore, Eq. ψ0 will not be obtained when ψ0 is true. So when Eq. ψ0 and subsequently reporting that there is evidence against ψ0 is not convincing. To measure the bias in favor of ψ0, choose values ψ0' ≠ ψ0 such that the difference between ψ0 and ψ0' represents the smallest difference of practical importance. Then computeψ0 will not be obtained when ψ0 is false. Note that Eq. ψ0' moves away from ψ0. When Eq. ψ0 and so subsequently reporting that evidence in favor of ψ0 being true has been found, is not convincing. For a fixed prior, both Eqs. From Eq. 3Example 2Inferences for a proportion.Consider now examples of applying relative belief inferences. The first example is concerned with making inferences about an unknown proportion.x =  ∈ {0, 1}n is observed where the xi are assumed to be i.i.d. Bernoulli(θ) with θ ∈ . This could arise from tossing a coin n times where 1 denotes a head and 0 a tail and θ is the probability of obtaining a head. A beta distribution, where α0 and β0 are specified, is taken for the prior. Let the parameter of interest be Ψ(θ) = θ. The the posterior of θ is a betaα0 = β0 = 4 provides an appropriate prior so the posterior is a betaSuppose that Suppose the data is given byθ.This data was actually generated from a Bernoulli(1/2) so indeed procedures for model checking and checking for prior-data conflict do not find any issues with the choices made. For this situationθ we always have θ(x) = θMLE(x) which in this case is θ. Here C0.95(x) =  and its length 0.593 − 0.227 = 0.366 indicates that there is a reasonable degree of uncertainty about the true value of θ. Note that, while relative belief inferences for θ take the same form as likelihood inferences for θ, it is not correct to consider RB(⋅| x) as a likelihood function as multiplying it by a positive constant destroys its interpretation as a measure of evidence. For a general Ψ(θ), the relative belief ratio RBΨ(⋅| x) is not proportional to a profile likelihood function.When making inference about the full model parameter H0 : θ = θ0 compute RB(θ0 | x). In this case, when θ0 = 1/2, then RB(1/2 | x) = 1.421, and since this is greater than 1, there is evidence in favor of H0. For the strength of this evidence we obtain,H0 is only moderate as there is a posterior probability of 0.691 that the true value of θ has a larger relative belief ratio. It is wrong, however, to conclude from the value 0.691 that there is evidence against θ0 = 1/2 because indeed the data have lead to an increase in belief that this is the true value. At the same time it is reasonable to have some concern about the reliability of this inference since the strength is not large. To see what the strength represents graphically consider RB(⋅| x) at two points which, when projected onto the θ-axis, gives an interval of θ values. The strength is then the posterior content of the two tails that form the complement of this interval together with the end-points. This geometric interpretation generalizes in an obvious way to the situation where θ is multidimensional.To assess the hypothesis H0 : θ = 1/2, compute the prior probability, when H0 is true, that evidence against H0 will be obtained, namely,To assess the bias against θ0. Bias in favor of H0 : θ = 1/2 is measured by the prior probability, when θ = θ⁎ ∈ {0.45, 0.55} is true, that there is evidence in favor of H0, namely,This indicates only modest bias against H0 = {1/2} induced by the beta prior, at least when a deviation of 0.05 from the null is considered as meaningful. A smaller deviation considered as meaningful would result in more bias in favor of H0. As previously mentioned, both biases can be controlled, namely, made as small as desired, by choosing the sample size n appropriately.So there is some bias in favor of Example 3Prosecutor's fallacy.The following example is very simple but nevertheless it has produced considerable confusion concerning the role of measuring evidence as opposed to taking a decision-theoretic approach to statistical inference. It emphasizes the importance of being very clear about how to measure evidence.In general, the prosecutor's fallacy refers to any kind of error in probabilistic reasoning made by a prosecutor when arguing for the conviction of a defendant. The paper Thompson and Schumann 1 of those innocent and a proportion ψ2 of those who are guilty. The overall proportion in the population possessing the trait is then (1 − ϵ)ψ1 + ϵψ2 and this will be small whenever ϵ and ψ1 are small. The values ϵ and ψ1 being small correspond to the proportion of guilty being very small and the trait being very rare in the population. The prosecutor notes that the defendant has this trait and, because (1 − ϵ)ψ1 + ϵψ2 is very small, concludes the defendant is guilty. Actually, as cited in Thompson and Schumann 1 + ϵψ2} as the probability of guilt! In any case, our concern here is the fallacious reasoning concerning the smallness of (1 − ϵ)ψ1 + ϵψ2 and what it implies about the guilt of the defendant.Suppose a population is split into two classes where a proportion ϵ are guilty of a crime and a proportion 1 − ϵ are not guilty. Suppose further that a particular trait is held by a proportion ψTreating ϵ as the prior probability that the defendant is guilty, without observing whether or not they have the trait, it is seen immediately that the posterior probability that the defendant is guilty, given that they have the trait, isRB (“guilty” | “defendant has the trait”) > 1 if and only if ψ2 > ψ1 and this occurs if and only if RB (“innocent” |“defendant has the trait”) < 1. If the trait is at all useful in terms of determining guilt, it is sensible to suppose ψ2 > ψ1 and, under these circumstances, it is certainly reasonable to say there is evidence in favor of guilt as the probability of guilt has increased from a priori to a posteriori.Now Ψ , γ(“defendant has the trait”) → {“guilty,” “not guilty”} as ϵ → 0 for any γ > 0. So for small ϵ the estimate has no accuracy at all! Furthermore, if we elected instead to assess the hypothesis H0: “guilty,” then the strength of this evidence is best assessed, since there are only two possible values, using the posterior probability P (“guilty” | “defendant has the trait”) and this converges to 0 as ϵ → 0 and again there is only very weak evidence in favor of guilt. So using the relative belief ratio to assess evidence, together with a measure of the strength of the evidence, protects against the prosecutor's fallacy as we will surely not convict based upon evidence in favor of guilt that is considered weak.The question now is: does relative belief commit a prosecutor's fallacy? It might seem so as there will always be evidence of guilt when the trait is observed. Recall, however, that there are two parts to a relative belief inference whether estimation or hypothesis assessment, namely, we must also say something about the accuracy of the inference. Under these circumstances we have that ψ(“defendant has the trait”) = “guilty” but it is clear that Cψ1 , ψ2 correspond to the probabilities of a test for infection being positive in the noninfected and infected populations, respectively. A good test will of course have ψ2 > ψ1 and so we are in exactly the same situation as, for a patient with a positive test, relative belief will record that there is evidence the patient is infected. Even if this is weak evidence, however, it would seem somewhat foolhardy to simply ignore the evidence.But the situation is more complicated than this yet and exposes a clear distinction between taking a decision-based approach and an evidential one. For consider the problem where ϵ corresponds to the proportion of individuals infected with a deadly infectious disease and ψ using the value that maximizes the posterior, called the MAP (maximum a posteriori) estimate. For ϵ small enough, this will declare the defendant innocent and the patient noninfected. In the former case this is reasonable but surely not in the latter case. It would seem that a categorical statement is not what is wanted from a statistical procedure in such problems. Undoubtedly decisions will be ultimately be made and these decisions may, for good reasons, ignore what the evidence says, but the additional criteria that come into play in making decisions are not statistical in nature. What is wanted from a theory of statistics is a statement concerning what the evidence indicates and, in addition, how strong that evidence is.A standard approach in this simple classification problem is to estimate 4A broad outline of relative belief theory has been described here. The inferences have many nice properties like invariance under reparameterizations and a wide variety of optimal properties in the class of all Bayesian inferences. The papers Evans fθ : θ ∈ Θ} and the prior π. Neither of these ingredients is necessarily determined by the application. In the end they are choices made by the practitioner which hopefully represent good judgment. In the event that these are poor choices, then it can be expected that the inferences may be erroneous and this is why the activities of model checking and checking for prior-data conflict are so important. If after these checks there is no reason to reject the choices made, then inference can proceed and relative belief gives an unambiguous approach to this. This lack of ambiguity is important as the failure of theories of inference to effectively solve inference problems leads to doubts as to the validity of inferences drawn on an ad hoc basis. The validity of relative belief inferences, once the basic principles are accepted, then rests with the choices made for the model and prior. Of course, it can never be said that these choices are “correct” only that they are not substantially wrong. These choices are essentially subjective in nature but the theory gives us tools for assessing any bias that the choices may have introduced into the analysis. This is the most we can expect from any theory of statistical inference.It is worth emphasizing that for practitioners there are two ingredients that need to be specified to apply the theory of relative belief to statistical analyses, namely, the model {


Clinical practice guidelines (CPGs) are used to standardize care according to evidence-based recommendations. However, the implementation of CPGs in the real world varies; a possible reason for this is the lack of a systematic approach to the implementation of CPG. We, therefore, aimed to develop an intervention to improve the implementation of a local hypertension CPG based on the needs of healthcare professionals (HCPs), best available evidence, and theories.This study was conducted in 2013 at an urban hospital-based primary care clinic in Malaysia which used a paper-based medical record system. The 2008 national CPG on hypertension was used for the research. The intervention was developed based on: (1) findings from an audit and needs assessment study involving the PCPs, pharmacists, nurses and the administrators; (2) literature review of the effectiveness of various CPG implementation strategies in primary care; and (3) the theory of planned behaviour.The research team convened to summarise the key findings from the needs assessment study and reached a consensus on two major issues: lack of accessibility to the CPG and collaboration among the HCPs. The literature review highlighted the need for a multi-faceted strategy which should include dissemination and implementation.A need-, evidence- and theory-based intervention was developed after a few iterations and feedback from the users. It comprised several components. Firstly, two training sessions were conducted by two senior academicians to increase the PCPs’ knowledge on the assessments and treatment of hypertension based on the CPG. Secondly, a quick reference guide summarising key recommendations from the CPG was placed on table tops as well as on the computer screen of each consultation room for easy access and reference. Thirdly, prior to the consultation with the doctor, the patient completed a self-assessment form and this was followed by the measurement of anthropometric parameters by the nurses. Requisition forms for investigations recommended by the CPG were placed together with patient medical records for ease of access and to facilitate doctors in making a treatment decision. Finally, a personalized checklist was placed in the patient medical record to serve as a quick reference and reminder to the doctor and it included items such as cardiovascular risk assessment.This study showed that it is feasible to develop a CPG implementation strategy based on needs, evidence and theories. We are in the process of evaluating the intervention.


The aim of this conference paper was to examine the evidence base for primary and secondary prevention of dental caries, erosions and dentin hypersensitivity through professional and self-care measures.A mapping of systematic reviews (SR) of literature was carried out in PubMed and the Cochrane library through April 2014 using established MeSH-terms and disease-related search words in various combinations. The search was restricted to SR's published in English or Scandinavian and all age groups were considered. The reference lists of the selected papers were hand-searched for additional review articles of potential interest. Meta-analyses, guidelines and treatment recommendations were considered only when SR's were lacking. In the event of updates or multiple systematic reviews covering the same topic, only the most recent article was included. No quality assessment of the systematic reviews was carried out. The quality of evidence was rated in four levels according to the GRADE scale.In total, 39 SR were included. For primary caries prevention, the quality of evidence was high for the use of fluoride toothpaste (with and without triclosan) and moderate for fluoride varnish and fissure sealants. The quality of evidence for fluoride gel, fluoride mouth rinse, xylitol gums and silver diamine fluoride (SDF) was rated as low. For secondary caries prevention and caries arrest, only fluoride interventions and SDF proved consistent benefits, although the quality of evidence was low. Likewise, the GRADE score for preventing erosions located in the enamel with fluoride supplements was low. The quality of evidence for various professional and self-care methods to prevent and manage dentine hypersensitivity was very low.There are knowledge gaps in many domains of cariology and preventive dentistry that must be addressed and bridged through clinical research of good quality. Dental caries, dental erosion and dentin hypersensitivity are prevalent oral conditions occurring at all ages. They are all multifactorial; caries is a biofilm-mediated disease resulting from a complex interaction between the commensal microbiota, host susceptibility and environmental factors such as diet , dental A broad search for systematic reviews of literature was carried out in PubMed  and in the Cochrane library through April 2014 using the main search words “caries”, “dental decay”, “early childhood caries”, “prevention”, “oral health promotion”, “antimicrobials”, “sugar”, “sugar substitutes”, “diet”, “fluoride”, “oral hygiene”, “dental erosion”, “dentin hypersensitivity” and “tooth sensitivity” in various combinations. Literature published in English, or in any of the Scandinavian languages, was considered and the reference lists of the selected reviews were hand-searched for additional review articles of potential interest. All age groups were of interest. If more than one systematic review on the same subject was identified, only the most recent date was included. However, no quality assessment of the systematic reviews was carried out. In the absence of systematic reviews, meta-analyses, guidelines and treatment recommendations were appraised, but only when based on a thorough and well-defined search strategy.The caries prevented fraction (PF) was extracted from the systematic reviews when possible. PF, expressed as percent, was defined as the difference between caries increment in control group and the experimental group, divided by the increment in the control group. For dental erosions and sensitivity, no such clear-cut endpoint of efficacy could be found and a narrative synthesis was made. The quality of evidence was rated in four levels according to the GRADE scale :• High (⊕⊕⊕⊕). Based on high or moderate quality studies containing no factors that weaken the overall judgement. The true effect lies close to that of the estimate of the effect.• Moderate (⊕⊕⊕O). Based on high or moderate quality studies containing isolated factors that weaken the overall judgement. The true effect is likely to be close to the estimate of the effect, but there is a possibility that it is substantially different.• Low (⊕⊕OO). Based on high or moderate quality studies containing factors that weaken the overall judgement. The true effect may be substantially different from the estimate of the effect.• Very low (⊕OOO). The evidence base is insufficient when scientific evidence is lacking, quality of available studies is poor or studies of similar quality are contradictory. Our confidence in the effect estimate is limited: The true effect is likely to be substantially different from the estimate of the effect.In systematic reviews relying on other ratings or quality systems, the author's statements and conclusions were subjectively transformed to GRADE by the author of this paper: GRADE High=Strong; GRADE Moderate=medium, good; GRADE Low=limited; fair; GRADE Very Low=weak, insufficient, inconclusive; some evidence.A total number of 39 systematic reviews were included to form the current evidence base; 24 were on primary caries prevention with various fluoride and non-fluoride agents, 4 dealt with secondary caries prevention, 3 compiled the evidence for prevention of erosions and 8 were about dentin hypersensitivity.Dental caries is one of the most prevalent diseases on the globe. It is almost a paradox to find strong evidence for selected caries-preventive measures when a recent report has calculated that 2.5 billion people (35%) on the planet have untreated caries in their permanent dentition [The prevented fraction and quality of evidence for the fluoride-based technologies are summarized in Table The runner-up was fluoride varnish with a moderate quality of evidence and a PF of 43% following 2-4 professional applications per year in permanent teeth compared with placebo/no treatment. The prevented fraction seemed to be higher in vulnerable populations compared to populations with low caries risk. Concerning primary teeth, the caries preventive effect of fluoride varnish was ambiguous. Most studies were in favor for professional applications in vulnerable preschool populations (PF18-24%), but displayed an unclear risk of bias ,34. The The quality of evidence concerning the non-fluoride methods for professional and self-care caries prevention is shown in Table In recent years, the addition of antibacterial (triclosan) or pH-rising (arginine) supplements to fluoride toothpastes has been introduced. There was evidence of high quality that triclosan and copolymers slightly may increase the caries preventive effect on coronal caries (PF 5%) in comparison with standard fluoride toothpaste while the support to reduce root caries was weaker . No systin vivo. Although findings suggested CPP-ACP to work better than placebo on early enamel lesions, the effect was not significantly different from that of fluoride. The quality of evidence, based on current clinical trials, was graded as very low [As stated above, there are few systematic reviews available on secondary caries prevention or caries control, which means re-mineralization or arrestment of existing early, non-cavitated lesions. The present quality of evidence is presented in Table very low .The use of 38% antibacterial silver diamine fluoride (SDF) is controversial in secondary caries prevention. Two available systematic reviews ,22 were The relative knowledge gap for secondary prevention is a drawback because the early “whitish”, demineralized enamel with rough structure is often the first detectable clinical sign of caries activity, calling for individualized and targeted prevention. Thus, and needless to say, there is an urgent need of high-quality clinical research on caries lesion control to elucidate this topic.in situ studies, displayed low quality evidence that fluoride may have a re-mineralizing role to play in reducing dental erosions, when located in the enamel [Dental erosion is the progressive and irreversible loss of dental hard tissue caused by acids from non-bacterial compounds. Consequently, the primary prevention of this condition would simply be to avoid any contact with erosive drinks and natural acid or acid-flavored food items. When eating-disorders, vomiting or gastric reflux are parts of the risk complex, psychological counselling, lifestyle changes and medication are essential components for the management of the condition. Dentists are however more commonly facing already established dental erosions, or tooth wear, of various stages and being an irreversible condition, there is an indistinct zone between secondary prevention and therapy. There are several systematic reviews available on the prevalence and etiology of dental erosions but very few dealing with its prevention or management. Among the non-invasive preventive measures to arrest this chemical process, diet changes, water rinses, fluoride-containing products, calcium-additives, matrix metalloproteinases and laser therapy have been commonly suggested but hardly evaluated in a scientific way. A recent systematic review  has grade enamel . No evide enamel . The useThe primary preventive measures for dentin hypersensitivity are common-sense based and almost identical to those advocated for dental erosions; avoid erosive (acidic) drinks and dietary sources and execute gentle but efficient tooth brushing with low abrasive toothpaste, separated from the acid intake. There are however no systematic reviews covering such counselling. For dental professionals, obvious measures to prevent the risk of tooth sensitivity is to avoid multiple in-office bleaching sessions with prolonged applications of high-concentrated hydrogen peroxide and to make gap-free fillings. For the management of established dentin hypersensitivity, treatment is perhaps a more adequate term than prevention. There are two main approaches to treat hypersensitive teeth ; i) blocMapping of systematic reviews offer is an established transparent way to describe research within a field and to identify gaps in the research base . The proAt a first glance, it may be disappointing to find that so much of the preventive care relies on low or very low quality of evidence. However, this is not the same as saying “do not do” because evidence-based dentistry is not a cook-book approach to clinical dentistry. A certain intervention may be useful even though the evidence is weak or even lacking. The quality grade “very low” is in fact in most cases a knowledge gap due to a shortage of good primary studies or, non-existent studies. The quality of evidence should exclusively be based on relevant, scientifically sound studies with low or moderate risk of bias. Treatment recommendations and guidelines on the other hand, are a result of merging the evidence with clinical experience, expert opinions and good clinical practice. This means that a technology rated as “very low” might work well in a skilled hand and under certain conditions. At contrary, methods with high quality of evidence might be of no or very small clinical importance . FurtherA shortcoming with the present paper was the language restriction and that only one major database was searched and therefore, relevant systematic reviews may have been missed. Another limitation was that no structured quality assessment of the systematic reviews was conducted, for example with the AMSTAR-tool . FactorsA striking fact was that health-economic evaluations and ethical considerations in connection with primary and secondary prevention were almost non-existent in the systematic reviews  and patiIn conclusion, there was high and moderate quality of evidence for the use of fluoride technologies and fissure sealants for primary prevention of dental caries in the young permanent dentition. For secondary caries prevention, the quality of evidence was rated as low or very low. In elderly, systematic reviews of literature were unable to establish definite conclusions regarding the effectiveness of adjunct methods for caries prevention. The quality of evidence for prevention of dental erosions and dentin hypersensitivity was very low. Thus, there is an urgent need of clinical research of good quality in many domains of cariology and preventive dentistry.The author received funding from Colgate Palmolive to attend and present at the “Prevention in Practice” conference. This was provided in the form of flights, hotel accommodation and subsistence. No one from the Colgate Palmolive Company was involved in the production, assessment or peer review of the manuscript nor was it submitted to them for approval prior to publication.


Health literacy (HL) is seen as an increasingly relevant issue for global public health and requires a reliable and comprehensive operationalization. By now, there is limited evidence on how the development of tools measuring HL proceeded in recent years and if scholars considered existing methodological guidance when developing an instrument.We performed a systematic review of generic measurement tools developed to assess HL by searching PubMed, ERIC, CINAHL and Web of Knowledge (2009 forward). Two reviewers independently reviewed abstracts/ full text articles for inclusion according to predefined criteria. Additionally we conducted a reporting quality appraisal according to the survey reporting guideline SURGE.We identified 17 articles reporting on the development and validation of 17 instruments measuring health literacy. More than two thirds of all instruments are based on a multidimensional construct of health literacy. Moreover, there is a trend towards a mixed measurement (self-report and direct test) of health literacy with 41% of instruments applying it, though results strongly indicate a weakness of coherence between the underlying constructs measured. Overall, almost every third instrument is based on assessment formats modeled on already existing functional literacy screeners such as the REALM or the TOFHLA and 30% of the included articles do not report on significant reporting features specified in the SURGE guideline.Scholars recently developing instruments that measure health literacy mainly comply with recommendations of the academic circle by applying multidimensional constructs and mixing up measurement approaches to capture health literacy comprehensively. Nonetheless, there is still a dependence on assessment formats, rooted in functional literacy measurement contradicting the widespread call for new instruments. All things considered, there is no clear “consensus” on HL measurement but a convergence to more comprehensive tools. Giving attention to this finding can help to offer direction towards the development of comparable and reliable health literacy assessment tools that effectively respond to the informational needs of populations.The online version of this article (doi:10.1186/1471-2458-14-1207) contains supplementary material, which is available to authorized users. Health literacy is an important determinant of public and individual health and is seen as a core element of patient centered care . In receOverall health literacy denotes “people’s knowledge, motivation and competences to access, understand, appraise and apply health information in order to make judgments and take decisions in everyday life concerning health care to maintain or improve quality of life during the life course” . ImproviTherefore the call for action regarding the development of an internationally comparable and reliable population based measure of health literacy is increasing .By now there are several definitions and theoretical frameworks of health literacy in place serving as a foundation to operationalize health literacy by developing framework based measures . These iIn this systematic review, we evaluate the status quo of health literacy measurement by providing insights in the currently applied measurement approaches and modes. Further, we appraise the reporting quality of publications dealing with the development and validation of instruments measuring health literacy. The review will help to verify if currently developed tools aiming to measure health literacy consider methodological critiques in the academic literature and contribute to the improvement of health literacy measurement.We conducted a systematic review of generic measurement instruments developed and validated to assess health literacy. Our review is in accordance with the recently extended guidelines of the PRISMA statement for reporting systematic reviews  see Fig. The useth April, 2013 and was limited to fully available English language publications developing and validating  generic instruments to measure health literacy. We decided on the specific period of time to cover literature left out in previous reviews on health literacy measurement [The review was completed by using PubMed, the Educational Resources Information Center (ERIC), the Cumulative Index to Nursing and Allied Health Literature (CINAHL) and Web of Knowledge databases. Additionally references in already published reviews and other publications were screened and a manual search on websites and print sources dealing with health literacy measurement was conducted. The search strategies encompassed key words as well as MESH terms depending on the database and were supplemented by synonyms and thesaurus terms as described in Additional file Articles were included by screening titles and abstracts of all unique publications and formulating full-text reports of all records passing the title/abstract screen by two independent reviewers. All health literacy instruments were categorized according to their characteristics including their purpose, applied health literacy taxonomy and construct, instrument design, availability, scoring method, validation-study sample characteristics as well as psychometric properties such as reliability and validity of the instrument. Additionally, a quality assessment according to the specifications of the reporting guidelines for survey research (SURGE) was performed. It encompasses reporting items in eight subdomains: article background, methods used, sample selection criteria, research tool characteristics, response rate as well as presentation of results, interpretation and discussions of findings and requirements of ethics and disclosure . CategorAfter extracting the instrument characteristics these were entered into an evidence table and critically assessed for reporting quality by two independent raters, followed by a third rater checking the extracted information for accuracy. Disagreements were resolved by a consensus process between the three raters.The PRISMA flow chart in Figure Among the 17 included publications on the development/validation of a generic health literacy measurement tools certain patterns can be identified. As depicted in Table The generic instruments identified here consider the multi-dimensional measurement approach by applying print literacy in all instruments and measuring quantitative abilities in almost half of the identified tools. In contrast, only three instruments are considering communication skills of individuals when measuring an individual’s health literacy. Following the theoretical framework of health literacy being a multidimensional, dynamic construct  with an (METER) developed in the United States is a brief self-administered screening tool (2 min administration time) for the clinical setting and includes 40 medical words and 40 words without an actual meaning (non-words) while aiming the identification of the medical words [(REALM)[(SAHL-S&E) also uses a word recognition approach as applied in the REALM and combines these with a comprehension test using multiple choice questions designed by an expert panel [health and financial literacy addresses the link between literacy and decision making in the context of health related and financial factors. It examines health literacy by using 9 items dealing with health knowledge regarding health insurance, burden of disease as well as medication skills [(CHC-Test) consists of 72 items presented in 4 scenarios dealing with skills such as the understanding of medical concepts, searching literature, basic statistics and the design of experiments and samples [ focuses on building a novel item pool in accordance with items used in the Test of Functional Health Literacy in Adults (TOFHLA). It measures prose, document and quantitative literacy in the field of certain lifestyle diseases as well as insurance related issues and patient rights administering these items with a multimedia gadget [The direct testing of competencies related to the health literacy construct is used frequently in the academic literature and five novel instruments were published in the search period. The Medical Term Recognition Test al words . The for [(REALM). Thus, t [(REALM). The Shon skills . The tes samples . The bila gadget , 44. A d(MAHL) assesses health literacy as a dynamic construct by addressing several domains: patient-provider encounter, interaction with the health care system, rights and responsibilities and health information. These are developed by analyzing items of numerous already existing instruments, identifying relevant items and modifying as well as supplementing them by new items [(HELMS) consists of 8 scales with 4–5 items and aims to assess health literacy by using a comprehensive approach. It encompasses multiple domains such as patient attitudes towards health and their proactivity as well as access, understanding and use of health information and access and communication with healthcare professionals [(HLS-CH) also addresses numerous domains such as information and  decision making, cognitive and interpersonal skills as well as problem solving. In this regard health literacy is rather a package of competencies interacting with each other [(AAHLS) measures health literacy based on the framework developed by Nutbeam [(MHLS) also captures health literacy as a multi-domain construct encompassing obtaining, understanding and processing health related information related to health promotion, disease symptoms, diagnosis, and treatment and using them in decision making [All identified instruments measuring health literacy by a self-report use a multidimensional concept of health literacy by integrating several domains and factors associated with health literacy. The self-report approach was applied in five instruments published in the search period. The Multidimensional Measure of Adolescent Health Literacy ew items . The Heassionals . The 127ch other . The All Nutbeam  and meas Nutbeam . Seemingn making . A furth(HLSI) as well as the short form (HLSI-SF) are 25/10 item tools that use real life health stimuli to assess an individual’s health literacy addressing print, oral, quantitative and internet based information seeking skills. The short form is derived by analyzing the psychometric properties of the HLSI and selecting best performing items. Additionally an 8 item self-report of the perceived performance among the skills addressed in the direct assessment of health literacy is conducted. Both approaches assess print literacy, numeracy and oral literacy as well as media literacy in a different manner demonstrating an acceptable internal consistency reliability of a Cronbach’s alpha of 0.86 for the HLSI and 0.70 for the HLSI-SF [(HLS-EU) carried out in eight European countries (Germany (NRW), Bulgaria, Austria, Greece, Spain, Ireland, Netherlands, Poland) also uses a mixed assessment approach measuring functional health literacy with the Newest Vital Sign (NVS) and using a self-report survey with 47 items. It defines health literacy in three domains  and 4 modes . Though the HLS-EU demonstrates a robust reliability of a Cronbach’s alpha of 0.97 for general health literacy the Spearman’s rho correlation between the NVS and HLS-EU with r = .245 is comparatively low indicating different constructs of health literacy [The combination of a direct testing and a self-report of health literacy skills is practiced frequently among indices, thus seven instruments identified in the search period use this approach. It enables to combine the methodological advantages of both approaches by diminishing possible straits . The Hea HLSI-SF , 50. Theliteracy , 51. Simliteracy . A furthliteracy . The briliteracy  and the literacy  with numliteracy . The healiteracy , 57–59. literacy , 53. A fThe application of reporting guidelines is a useful way to facilitate transparency and gauge the reliability of an instrument used in a survey. However the compliance with reporting guidelines such as the “reporting guideline for survey research” recently compiled by Bennett and colleges  is limitIn our review, we identified recently published (2009 forward) publications dealing with novel instruments developed and validated to measure health literacy. The review followed two main objectives. In the first place, we examined how the measurement of health literacy proceeded in recent years particularly emphasizing whether novel instruments consider existing recommendations of the scientific community on features an instrument measuring health literacy should cover. In addition, we analyzed the reporting quality of the identified papers dealing with the development of health literacy measurement tools.Our analysis resulted in six major findings, which extend the prior knowledge on health literacy measurement.First of all, we examined an increasing use of multidimensional constructs to measure health literacy. Especially instruments with a subjective measurement format address numerous domains of health literacy such as patient-provider encounter; interaction with the health care system; rights and responsibilities; health information-seeking; understanding, processing, and using healthcare information as well as communication with healthcare professionals , 48, 50.Furthermore, we found that almost all instruments apply a multidimensional measurement of health literacy by principally assessing print literacy and numeracy and in some cases adding oral literacy. Previous reviews dealing with health literacy measurement tools emphasized the lack of instruments integrating communication skills  in the health literacy construct . To fillIn addition we identified that there is a trend towards the use of objective (task based) and subjective (self-report based) measurement approaches in a mixed manner. Scholars using this mixed measurement approach often apply already existing health literacy screeners  and develop additional item batteries , 56, 59.A further striking finding is that regardless of the used measurement approach, scholars do not sufficiently explain why they are choosing a certain type of measurement. According to Abel, the first issue in the circle of instrument development is to determine the purpose of the instrument by answering the “what for” question. As soon as the given theoretical context and setting is clear, ideas on the way of measurement can be developed systematically . If the Finally, there is an extensive use of assessment formats modeled on already existing instruments such as the REALM or the TOFHLA inserting mostly straightforward additions , 44, 47.The appraisal of the reporting quality of publications dealing with the development and validation of health literacy indices has yielded mixed findings. Some domains such as the description of the article background and presentation and interpretation of results are reported thoroughly, while other domains addressing methodological properties have received less consideration. Overall, the identified papers included in the review demonstrate a lack of compliance with reporting guidelines especially for methodological issues such as psychometric properties of the developed instruments, sample selection strategy and response rate presentation. These findings are in line with previous research stating that key survey characteristics in health care literature in general , 64 and From an overall perspective, almost all identified instruments apply a multi-dimensional measurement (often print and numeracy literacy) and the majority utilizes a mixed measurement approach (objective and subjective measurement) with a multidimensional construct enhancing the comprehensiveness of tools measuring health literacy. Nevertheless, there is no clear indication of the demanded “consensus” on health literacy measurement. This is mainly because there have been only minor developments among the measurement formats, as can be seen in the increased use of earlier instruments, even though the academic world is calling for new instruments .To continuously advance the field of health literacy measurement work should proceed on several fronts. Though there is currently a huge effort to improve the more comprehensive measurement of health literacy, the format of measurement generally relies on already existing approaches such as the cloze technique ) or word recognition (used in “Rapid estimate of adult literacy in Medicine” (REALM)) , 44, 47.Apart from the issue of originality, it would be necessary to reflect more closely on the combination of objective and subjective measurement instruments, thus current studies show less coherence. Though the limited reporting guideline compliance of health literacy instruments was identified by Jordan and colleges before  our analIn considering such recommendations, certain limitations should be noticed regarding our review. Although we followed the PRISMA guidelines when performing our systematic review and used MESH terms and key words, we may have missed relevant literature. Furthermore, there was no reporting guideline available that provided a scoring scheme for the reporting quality. As a consequence we could not grade the reporting quality of the identified articles resulting in a descriptive description of the results. Finally, the appraisal of health literacy instruments was limited as the item batteries and scoring methods were not always available despite a direct request to the authors.Apart from this, our review exhibits certain strengths such as the compliance to guidelines when performing the literature search, data selection, analysis and appraisal of the reporting quality of the identified articles.Our review offers insights in the status quo of health literacy measurement. It critically appraises applied measurement approaches and analyses reporting qualities by commenting on current developments and their value for the further evolution of health literacy measurement. Giving attention to the evidence presented here can help to offer direction towards the development of comparable and reliable health literacy assessment tools that effectively respond to the informational needs of populations.All authors are affiliated to the Institute for Health Economics and Clinical Epidemiology, University Hospital of Cologne. and primarily deal with health systems and outcomes research focusing on chronic care and disease management. Mrs. Prof. Dr. med. Stephanie Stock is the chairwoman of the German Health Literacy Network and coordinates the network activities in Germany.Prisma Checklist.(DOC 66 KB)Additional file 1:Overview of the chosen search strategy.(DOCX 12 KB)Additional file 2: