In many domains, software systems cannot be deployed until authorities judge them fit for use in an intended operating environment. Certification standards and processes have been devised and deployed to regulate operations of software systems and prevent their failures. However, practitioners are often unsatisfied with the efficiency and value proposition of certification efforts. In this study, we compare two certification standards, Common Criteria and DO-178C, and collect insights from literature and from interviews with subject-matter experts to identify design options relevant to the design of standards. The results of the comparison of certification efforts---leading to the identification of design dimensions that affect their quality---serve as a framework to guide the comparison, creation, and revision of certification standards and processes. This paper puts software engineering research in context and discusses key issues around process and quality assurance and includes observations from industry about relevant topics such as recertification, timely evaluations, but also technical discussions around model-driven approaches and formal methods. Our initial characterization of the design space of certification efforts can be used to inform technical discussions and to influence the directions of new or existing certification efforts. Practitioners, technical commissions, and government can directly benefit from our analytical framework.
There are also papers that perform standards comparison, but much more focused and fine-grained. compared both IEC 60880 and IEC 61508 standards, aiming at identifying differences in strictness and scope between the shall'' requirements as defined by the two standards. The results indicate that such fine-grained comparison is difficult, but they could still identify that even though the two standards are fairly similar, they complement each other and should be used together to achieve higher safety assurance. @cite_18 investigated differences in certifying a wheel braking system when using a prescriptive approach (DO-178C) and when using assurance cases. The authors suggest linking assurance cases and DO-178C processes to increase safety assurance, since using assurance cases alone might be challenging, especially for vendors without experience in developing cases. Our study complements these papers by contrasting two distinct standards, which contributes to a broader discussion about certification efforts across communities. We are not aware of any other papers that compare standards across communities, focusing on contrasting cases to confront community-specific culture.
@cite_18
2157607087 611011698 1532757718 2495715420
In safety–critical applications, it is necessary to justify, prior to deployment, why software behaviour is to be trusted. This is normally referred to as software safety assurance. Within certification standards, developers demonstrate this by appealing to the satisfaction of objectives that the safety assurance standards require for compliance. In some standards the objectives can be very detailed in nature, prescribing specific processes and techniques that must be followed. This approach to certification is often described as prescriptive or process-based certification. Other standards set out much more high-level objectives and are less prescriptive about the particular processes and techniques to be used. These standards instead explicitly require the submission of an assurance argument which communicates how evidence, generated during development (for example from testing, analysis and review) satisfies claims concerning the safety of the software. There has been much debate surrounding the relative merits of prescriptive and safety assurance argument approaches to certification. In many ways this debate can lead to confusion. There can in fact be seen to be a role for both approaches in a successful software assurance regime. In this paper, we provide a comparative examination of these two approaches, and seek to identify the relative merits of each. We first introduce the concepts of assurance cases and prescriptive software assurance. We describe how an assurance case could be generated for the software of an aircraft wheel braking system. We then describe how prescriptive certification guidelines could be used in order to gain assurance in the same system. Finally, we compare the results of the two approaches and explain how these approaches may complement each other. This comparison highlights the crucial role that an assurance argument can play in explaining and justifying how the software evidence supports the assurance argument, even when a prescriptive safety standard is being followed. Nowadays, road vehicles, including trucks, are characterized by an increasedcomplexity due to a greater variety of software, and a greater number of sensorsand actuators. As a consequence, there is an increased risk in termsof software or hardware failures that could lead to unacceptable hazards.Thus safety, more precisely functional safety, is a crucial property that mustbe ensured to avoid or mitigate these potential unacceptable hazards. Inthe automotive domain, recently (November 2011), the ISO-26262 safetystandard has been introduced to provide appropriate requirements and processes.More specically, the standard denes the system development processthat must be carried out to achieve a system that can be consideredacceptably safe. To be released on the market, systems must be certied,proofs that the systems are acceptably safe must be provided in terms of astructured argument, known as safety case, which inter-relates evidence andclaims. Certication authorities are in charge of evaluating the validity ofsuch safety cases. In the automotive domain, certication and compliancewith the standard ISO-26262 is becoming mandatory. By now, trucks donot have to be compliant with the standard. However, it is likely that by2016 they will have to. Scania is one of the leading companies in trucksdevelopment. To be ready by 2016, Scania is interested in investigatingISO-26262 as well as safety case provision. Thus this thesis focuses on theprovision of a safety case in the context of ISO-26262 for Fuel Level Estimationand Display System (FLEDS), which is one of the safety-criticalsystems in Scania.1 A safety case should provide a clear, comprehensible and defensible argument, supported by evidence, that a system is acceptably safe to operate in a particular environment. This approach is not new. For example, in the nuclear industry, safety cases are approaching their 50th birthday. In stark contrast, the automotive industry has never been required to produce a safety case. Instead, it has relied on compliance with extensive regional and national regulation. With the imminent introduction of the automotive safety standard ISO 26262, the production of a safety case is now explicitly required by the standard for electrical and electronic systems. This presents both opportunities and challenges to safety practitioners and researchers within that industry. This paper sets out to look at the issues of what a safety case might look like for a complete vehicle and how the ISO 26262 fits into the existing framework of automotive safety. Using the ideas of modular safety case construction, this approach is developed into a number of reusable safety arguments to form an automotive safety case pattern catalogue. Evaluation of the approach is carried out through an industrial case study. In engineering of safety critical systems, regulatory standards often put requirements on both traceable specification-based testing, and structural coverage on program units. Automated test generation techniques can be used to generate inputs to cover the structural aspects of a program. However, there is no conclusive evidence on how automated test generation compares to manual test design, or how testing based on the program implementation relates to specification-based testing. In this paper, we investigate specification -- and implementation-based testing of embedded software written in the IEC 61131-3 language, a programming standard used in many embedded safety critical software systems. Further, we measure the efficiency and effectiveness in terms of fault detection. For this purpose, a controlled experiment was conducted, comparing tests created by a total of twenty-three software engineering master students. The participants worked individually on manually designing and automatically generating tests for two IEC 61131-3 programs. Tests created by the participants in the experiment were collected and analyzed in terms of mutation score, decision coverage, number of tests, and testing duration. We found that, when compared to implementation-based testing, specification-based testing yields significantly more effective tests in terms of the number of faults detected. Specifically, specification-based tests more effectively detect comparison and value replacement type of faults, compared to implementation-based tests. On the other hand, implementation-based automated test generation leads to fewer tests (up to 85 improvement) created in shorter time than the ones manually created based on the specification.