An extended systematic literature review on provision of evidence for safety certification <s> Data Extraction and Quality Criteria <s> This paper presents the results of work performed by Lawrence Livermore National Laboratory to assist the U.S. Nuclear Regulatory Commission in understanding the state of the art in software reliability for computer-based reactor protection systems. The activities reported upon summarize advice from technical experts in software reliability and safety, and identify the best current software development practices used in industry for safety-critical software. The research reported here has identified a number of positive and negative design factors that can serve as the basis for a safety assessment. The results of the interviews and discussions were combined into a set of principles which were termed "design factors". Although the areas of emphasis among the three sources of information (standards, experts and organizations) tend to be quite different, no substantial areas of disagreement were found. Many of the factors contributing to the success or failure of software may be attributed to the knowledge, understanding, intelligence, and care of the individuals and companies involved in the development of safety-critical software. By combining the best from theory and practice it is possible to isolate a number of factors that distinguish the good from the bad. > <s> BIB001 </s> An extended systematic literature review on provision of evidence for safety certification <s> Data Extraction and Quality Criteria <s> Commercial-Off-The-Shelf (COTS) software components are being used within complex safety-critical applications. However, to use them with confidence, it is necessary to ensure that potential failures of the COTS component does not contribute to system level hazards. To this end, we have established a contract-based approach to capture the application-specific safety requirements, and corresponding assurance requirements, derived for a potential COTS component. This "contract" can be used to form the basis of a packaged safety argument (i.e. a safety case) for the component. This COTS component safety case (or safety case module) can then be used to form part of an overall system safety case. Using the previously developed concepts of compositional safety case construction (Kelly 2003), we describe the activity of matching application level safety objectives and assurance requirements to those claims and levels of assurance that can be established for the COTS component. The role of argument mitigation strategies is described for those situations where direct matches cannot be achieved. An example derived from an industrial COTS-based application is used to illustrate the approach. <s> BIB002 </s> An extended systematic literature review on provision of evidence for safety certification <s> Data Extraction and Quality Criteria <s> Several safety-related standards exist for developing and certifying safety-critical systems. System safety assessments are common practice and system certification according to a standard requires submitting relevant system safety information to appropriate authorities. The RTCA DO-178B standard is a software quality assurance, safety-related standard for the development of software aspects of aerospace systems. This research introduces an approach to improve communication and collaboration among safety engineers, software engineers, and certification authorities in the context of RTCA DO-178B. This is achieved by utilizing a Unified Modeling Language (UML) profile that allows software engineers to model safety-related concepts and properties in UML, the de facto software modeling standard. A conceptual meta-model is defined based on RTCA DO-178B, and then a corresponding UML profile, which we call SafeUML, is designed to enable its precise modeling. We show how SafeUML improves communication by, for example, allowing monitoring implementation of safety requirements during the development process, and supporting system certification per RTCA DO-178B. This is enabled through automatic generation of safety and certification-related information from UML models. We validate this approach through a case study on developing an aircraft's navigation controller subsystem. <s> BIB003 </s> An extended systematic literature review on provision of evidence for safety certification <s> Data Extraction and Quality Criteria <s> We discuss the construction of an assurance case for the pacemaker software. The software is developed following a model-based technique that combined formal modeling of the system, systematic code generation from the formal model, and measurement of timing behavior of the implementation. We show how the structure of the assurance case reflects our development approach. <s> BIB004
We designed a data extraction template (a spreadsheet) to collect the information needed to answer the research questions. Apart from the bibliographic information (title, authors, year, and publisher), we extracted from each study the application domain in which the system under assessment or certification was used, the underlying safety standard(s) used to show compliance, the information, artefact, tool, or technique contributing to evidence, techniques for evidence structuring, techniques for assessing confidence on the evidence collected, and the needs and challenges addressed about provision of evidence. Appendix A provides a table with some sample data extracted from the studies. All the information about the data extracted from all the studies can be found in  . We further extracted data for publication quality assessment. For this, we defined three criteria: Evidence abstraction level, which was assigned on the basis of the specificity of the evidence instances presented in a given study. The levels allow us to weight the quality of evidence items identified from the analysis of the primary studies. The abstraction levels defined, from the most abstract to the most specific, were: generic, domain level, safety standard level, system type level, and specific system level. Using the evidence types from our evidence classification (Section 4.1), example instances of evidence for the non-generic abstraction levels are: Hazard specification for domain level (e.g., nuclear domain) BIB001 , Source code for safety standard level (e.g., for DO-178B BIB003 ), System Historical Service Data Specification for system type level (e.g., COTS-based systems BIB002 ), and Model Checking Results for specific system level, e.g., instantiated for a specific pacemaker software BIB004 . The "generic" abstraction level refers to instances of evidence mentioned in a primary study that are not presented within the scope of any specific domain, standard, system type, or specific system. Generally, we consider lower abstraction levels and thus more specific evidence to be more useful since it is more likely for those studies to contain some practical advice. Validation method, which was assigned based on how a given study had been validated. The studies were classified as: case study (validated during projects by practitioners different from the authors), field study (validated with data from real projects, but not during the execution of the projects), action research (validated during real projects by the authors themselves), survey (validated on the basis of practitioners' opinion and perspectives), or none. It is important to note that we use the term "validation" in a broad sense. In particular, validation does not necessarily imply validation in a controlled environment such as a controlled experiment. Indeed, we did not find any primary studies reporting a controlled experiment. Nonetheless, we consider information gathered from validated work to be more useful as they better reflect the state of practice. Tool support, which assists in the provision of evidence (collection, structuring, and assessment) for certification or safety assurance purposes. We consider the availability of tool support to be an important maturity factor for the underlying technique and a necessary step for its industrial application.