Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> Evidence-based Software Engineering <s> Describes an ongoing research project conducted jointly by Siemens and the Software Engineering Institute. Siemens software-development organizations in Germany and the United States are case-study sites at which we measure the effect of methods to improve the software-development process. To observe and quantify the impact of software-process improvement, we must measure the performance of a software-development organization over time. Comparison of performance across organizations is very difficult, since organizations define measures and collect performance data in different ways. However, we can separately track performance improvement in each organization if it defines measures consistently and develops similar products. We have defined basic measures for performance of a software-development organization. We limited ourselves to a small number of simple measures to reduce the complexity of collecting, analyzing, and maintaining the performance data. Improving the software-development process improves the quality of software products and the overall performance of the software-development organization. However, process is only one of several controllable factors in improving software quality and organization performance. Others include the skills and experience of the people developing the software, the technology used (e.g. CASE tools), product complexity, and environmental characteristics such as schedule pressure and communications. > <s> BIB001 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> Evidence-based Software Engineering <s> This paper covers both the development of BS 7925-2 and compliance with the finished standard. The process used to produce BS 7925-2 is described by presenting a history of its development, showing how initial chaotic development evolved with experience into a systematic process. This process was found to bear a strong resemblance to the software engineering process, and borrowed techniques from it, drawing particularly from those that support verification and validation. The knowledge gained from this development should be applicable to many areas of standardisation. The most important clauses of BS 7925-2 describe a generic test process and definitions of test case design techniques and test measurement techniques. These are explained in detail and compliance with BS 7925-2 is described in the context of following the test process, which, in turn, requires the user to select techniques from the other two clauses. <s> BIB002 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> Evidence-based Software Engineering <s> Software testing is a core activity in quality assurance. To improve the test process, we can use best practice models which describe in detail what to do in organizational test processes. The improvement activities using best practice models are performed as follows: checking the current status of test processes, suggesting and planning new actions, and implementing the actions. However, it is difficult to apply all of these actions to the organization due to the limitation of resources. In this paper, we suggest a strategy for optimizing test process action plans. The background of this research is TMM (Testing Maturity Model), which is the most representative test process model. By applying design of experiments to the TMM assessment procedure, we can accept the actions selectively by statistical significance and find the best solution. <s> BIB003 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> Evidence-based Software Engineering <s> Existing test process assessment and improvement models intend to raise maturity of an organization with reference to testing activities. Such process assessments are based on ldquowhatrdquo testing activities are being carried out, and thus implicitly evaluate process quality. Other test process measurement techniques attempt to directly assess some partial quality attribute such as efficiency or effectiveness some test metrics. There is a need for a formalized method of test process quality evaluation that addresses both implicitly and partially of these current evaluations. This paper describes a conceptual framework to specify and explicitly evaluate test process quality aspects. The framework enables provision of evaluation results in the form of objective assessments, and problem-area identification to improve the software testing processes. <s> BIB004 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> Evidence-based Software Engineering <s> It is well known that most of mission critical systems like defense systems demand higher quality than general software systems. In order to assure the high quality of the systems, those software systems must be thoroughly tested with well defined testing processes. As software systems evolve, software testing should also be improved with continual-base. Even if there are some previous researches on test process model, those models have some limitations and problems to be applied for software testing process improvement in a specific domain like military weapon systems. In this paper, we propose a new testing maturity model, called MND-TMM (Ministry Of National Defense-Testing Maturity Model), which can be adopted effectively for the weapon software system development. We also present OWL ontology for the MND-TMM, called MTO(MND-TMM ontology) to help software organizations set up their effective testing strategies and improve their testing processes, which can lead to high quality product developments. <s> BIB005 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> Evidence-based Software Engineering <s> Producing high-quality software has been one of the greatest challenges of the development market in the last few years. The software testing is the essence of the quality assurance, but the implementation of this activity is still difficult due to many factors. This paper presents some factors that influence on the adoption of a testing process. Some of these factors were outlined in the literature, and others were selected empirically based on the work experience obtained in a large company. Besides, this paper presents the results of a survey carried out to identify factors that make difficult the implementation of software testing process in small-sized software organizations and a proposed approach of a testing process to them. <s> BIB006 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> Evidence-based Software Engineering <s> As one of the most challenging and costly processes, the software testing provides strong support for the high quality software. The popular process improvement models do not adequately address testing application issues, nor has the knowledge management (KM) implementation in testing been well defined. The plan-do-check-action (PDCA)-based testing improvement framework is proposed to address these issues. The framework contains the adaptive processes definition, the processes implementation and measurement analysis, and benchmarking-based assessment approach. Experiments show that: the framework can stimulate team member's subjective initiative, promote the software testing process and improve the testing service quality. <s> BIB007 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> Evidence-based Software Engineering <s> Practitioners come across many defects that impede the rapid progress of software process development, which is critical to the Organization's operation and growth [11]. In order to improve the quality of software development process, a phase-based Defect Removal Model (DRM) is used in this study to judge the Inspection/Test effectiveness and to find out the count of defects that remains at the exit of each phase. The aim of this study is to reduce the defects at the exit of a phase so that defects do not penetrate to subsequent stages, and to improve the Inspection/Test effectiveness. This has been achieved in this study by proposing a Strategy for Process Improvement (SPI). SPI insists that reviews and inspections of software artefacts should be done by a special independent team called Q-GoD (Quality-Guard of Defects) in addition to the Internal Review Team throughout the development life cycle for identifying defects that escape review/inspection process thereby improving software quality. <s> BIB008 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> Evidence-based Software Engineering <s> Context: Testing is one of the most important phases of software development. However, in industry this phase is usually compromised by the lack of planning and resources. Due to it, the adoption of a streamlined testing process can lead to the construction of software products with desirable quality levels. Objective: Presenting the results of a survey conducted to identify a set of key practices to support the definition of a generic, streamlined software testing process, based on the practices described in the TMMi (Test Maturity Model integration). Method: Based on the TMMi, we have performed a survey among software testing professionals who work in both academia and industry. Their responses were analyzed quantitatively and qualitatively in order to identify priority practices to build the intended generic process. Results: The analysis enabled us to identify practices that were ranked as mandatory, those that are essential and should be implemented in all cases. This set of practices (33 in total) represents only 40% of the TMMis full set of practices, which sums up to 81 items related to the steps of a testing process. Conclusion: The results show that there is a consensus on a subset of practices that can guide the definition of a lean testing process when compared to a process that includes all TMMi practices. It is expected that such a process encourages a wider adoption of testing activities within the software industry. <s> BIB009 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> Evidence-based Software Engineering <s> In agile software development, industries are becoming more dependent on automated test suites. Thus, the test code quality is an important factor for the overall system quality and maintainability. We propose a Test Automation Improvement Model (TAIM) defining ten key areas and one general area. Each area should be based on measurements, to fill the gap of existing assessments models. The main contribution of this paper is to provide the outline of TAIM and present our intermediate results and some initial metrics to support our model. Our initial target has been the key area targeting implementation and structure of test code. We have used common static measurements to compare the test code and the source code of a unit test automation suite being part of a large complex telecom subsystem. Our intermediate results show that it is possible to outline such an improvement model and our metrics approach seems promising. However, to get a generic useful model to aid test automation evolution and provide for comparable measurements, many problems still remain to be solved. TAIM can as such be viewed as a framework to guide the research on metrics for test automation artifacts. <s> BIB010 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> Evidence-based Software Engineering <s> Model-based testing (MBT) provides a systematic and automated way to facilitate rigorous testing of software systems. MBT has been an intense area of research and a large number of MBT techniques have been developed in the literature and in the practice. However, all of the techniques have been developed using their own concepts and terminology of MBT, which are very often different than other techniques and at times have conflicting semantics. Moreover, while working on MBT projects with our industrial partners in the last several years, we were unable to find a unified way of defining MBT techniques based on standard terminology. To precisely define MBT concepts with the aim of providing common understanding of MBT terminology across techniques, we formalize a small subset of the recently released ISO/IEC/IEEE 29119 Software Testing Standard as a conceptual model (UML class diagrams) together with OCL constraints. The conceptual model captures all the necessary concepts based on the standard terminology that are mandatory or optional in the context of MBT techniques and can be used to define new MBT tools and techniques. To validate the conceptual model, we instantiated its concepts for various MBT techniques previously developed in the context of our industrial partners. Such instantiation automatically enforces the specified OCL constraints. This type of validation provided us feedback to further refine the conceptual model. Finally, we also provide our experiences and lessons learnt for such formalization and validation. <s> BIB011
Evidence-based Software Engineering based models [TM14] focus mainly on mapping and gathering evidence through systematic reviews. The process is performed in three steps. First, the need for the evidence and review question are formulated. Subsequently, the evidence is critically evaluated and the answer to the question is provided. In the third step, value stream mapping is used to map the challenges of the testing process to the value stream  . Each testing model is specified by a different set of characteristics, domain, and modality of application. The testing models are divided into two big categories based on whether they are supported by a maturity model (Table 7) : overall, ten out of seventeen testing models have a maturity structure, which defines several improvement stages. The maturity models and their levels differ by goals and key elements used in the testing process. On the other hand, models which are not based on a maturity structure use completely different approaches such as processes divided into three levels [TM8], gathering evidence by performing reviews [TM14] or focusing on knowledge management [TM12]. Based on the empirical studies, we identified the domains in which the models can be applied ( Number of models 4.2 Which are the reported aspects of a testing process the testing models help to improve? Empirical studies report aspects that the application of testing models helps to improve ( Table 8 ). The most reported aspect in the papers is an improvement of the whole testing process by introducing more formalization. Organizations focus particularly on the application of a testing model to improve and standardize testing which can have subsequently positive impact also on other aspects such as product quality BIB009 or risk management BIB003 . Eight case studies focus mainly on quality improvement by adopting a testing model. As it is mentioned in the case study of Farooq et al. BIB004 , the quality of a developed product depends mostly on an efficient testing process. For this reason, the authors developed a conceptual framework which defines and evaluates test process quality aspects. When designing a concept of process evaluation, authors were inspired by the Evaluation Theory, which defines six core elements of process evaluation  . Another example of the quality improvement is the study of Ryu et al. BIB005 , where the authors had to deal with the needs of high quality in the defense domain. They proposed the MND-TMM model [TM7], which can help organizations to implement effective testing and improve testing processes for high quality products. Moreover, they suggested an ontology called MTO (MND-TMM Ontology) for effective use and execution of model processes, which also can lead to high quality product developments BIB005 . Two papers BIB010 BIB001 describe and use sets of metrics and measurements as a part of software development improvement. Authors suggest measurements for defects, effort or product size, which can be expressed for example by lines of code (LOC) count. The study of Eldh et al. BIB010 defines an evaluation process based on validated metrics, which can help to perform test automation in an objective and mature way. In the second case study of Paulish and Carleton BIB001 , the authors measure the effect of methods and design several types of metrics based on the subject of measurement (e.g. defects, effort). The empirical study of Ali and Yue BIB011 designs a conceptual model, which is based on Model-based Testing (MBT) and the ISO/IEC/IEEE 29119 standard, to provide a starting point for developing new MBT techniques. In the paper by Kumaresh and Baskaran BIB008 , authors applied Defect Removal Model to reduce the cost by early removal of the defects. Reid BIB002 focuses on component testing by applying a testing model, which consists of several process phases such as component test planning, specification or execution. The paper of Rodrigues et al. BIB006 describes how important it is to meet clients' requirements by providing high-quality products. The authors primarily deal with small-sized companies, because not many of them execute test software activities. Authors perform a survey and provide factors, which influence the adoption of a testing process and thus obstruct the final product improvement BIB006 . The case study of Xu-Xiang and Zhang BIB007 applies the PDCA-based testing improvement framework [TM12] based on the analysis of agile processes, benchmarking process and knowledge management. These three areas are parts of the model concept, which address testing application issues and improve the testing service quality. The agile process helps to keep balance between flexibility and stability, and can respond to market changes rapidly. Execution of the benchmarking process improves cooperation, trust and communication, which leads to the creation of an organizational knowledge-oriented culture. Knowledge management includes all the activities that ensure individuals obtain the needed knowledge and subsequently the organization gains competitive advantages BIB007 .